scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-11-11 08:37:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

feat: add the test scenario map and a Vitest 5 mutation runner (#3686)

Browse source 
* feat(ci): add advisory duplicate-test scan to the PR report

Adds `npm run test:dupes` (gitnexus/scripts/test-dupes.ts) and a
comment-only CI job that reports, for the lines a PR changes:

- same-test: same body and same setup as an existing test
- assertions-pinned: every assertion is a literal fn(...) -> matcher
  that another test already makes
- same-fixture-setup: a before* hook builds a fixture another describe
  already builds
- copied-helper: a top-level test helper identical to one elsewhere
- orphan-bench-gate: a bench script with --check that nothing runs

A case's identity includes its hooks, describe-level statements,
wrapper arguments, loop heads, each tables, referenced file-local
declarations and resolved imports, with fixture paths compared by
value. Extra context can only split a group, so identical bodies that
run against different fixtures, mocks or env are not reported.

The ci.yml job is continue-on-error and outside the CI Gate.
ci-report.yml renders its JSON into the sticky CI comment, keeping
only whitelisted kinds and strict path/line shapes because the
artifact comes from PR code. `// dupe-ok: <reason>` silences an
intentional repeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mutation): record Vitest 5 and Stryker 10 probe findings

Go/no-go probe for the Stryker Vitest 5 runner: name-filter separator, per-spec task ids, static mutant activation, TypeScript plugin loading in child processes, setup-file meta and hook ordering.

* feat(mutation): prepare a mutable copy and a Vitest mutation config

Prepared copy blanks TypeScript as-assertions (Stryker's instrumenter skips every TSAsExpression) with equal-length spaces, checked by AST equality, and relocates gitnexus-shared/src inside the package so it can be mutated. The mutation Vitest config resolves gitnexus-shared to the relocated sources and takes its test list from a scope file.

* feat(mutation): add a Stryker test runner for Vitest 5

Stryker's vitest-runner 10.0.0 runs zero tests per mutant under Vitest 5 because Vitest 5 matches testNamePattern against ' > '-joined names. This runner selects tests by Vitest task id per file, keeps the configured fork pool, shares the dry-run manifest across Stryker worker processes, attributes hook-time hits to the suite's tests, and records a per-test outcome class in sidecar files. Adds exact devDependencies @stryker-mutator/core and @stryker-mutator/api 10.0.0.

* feat(mutation): add global mutant identities and a span-map check

A global mutant identity (original repo path, original content hash, span, mutator, replacement) makes mutants comparable across Stryker runs. The span-map check proves every mutant of a prepared copy sits on the same span and text in the original source, apart from blanked as-assertions.

* feat(mutation): add a scope driver for Stryker runs

run-scope turns a scope file into a prepared copy, a generated Stryker config (vitest5 or command runner), a Stryker run and normalised results: global mutant ids, runner test ids, sidecar outcome classes, consistency flags and the span-map check.

* fix(mutation): count assertion kills only for killed runs

A run classified timeout, inconclusive, invalid or infrastructure can still carry assertion rows (a test failing after a swallowed hit-limit error, or under a retry). Those kills are not trustworthy, so assertionKilledBy is now empty unless the run's status is killed. Found by the conformance suite.

* feat(mutation): add the runner conformance suite (gate 2)

A fixture project under test/fixtures/mutation-conformance pins 23 runner behaviours (static and collection-time mutants, beforeAll attribution, disableBail, it.each row ids, outcome classes for runtime, hook, collection, timeout, infrastructure, retry and zero-test runs, NoCoverage, environment isolation, runner reuse and manifest sharing) in two scheduling profiles. The path map and scope driver accept an explicit gitnexus/test/fixtures/mutation-* root for it.

* feat(mutation): classify pure-JS unit tests for mutation runs

Walks each unit test's import closure (static imports plus statically resolvable dynamic imports, require and createRequire) and rejects tests that reach native packages, worker threads or child processes, read source or fixture files a prepared copy would break, or contain unresolvable dynamic imports. 247 of 911 unit test files are pure JS.

* feat(mutation): add gate 1 execution-path fidelity and the envelope record

Gate 1 compares a normal Vitest 5 run of the pure-JS category with the runner's unmutated Stryker run: test identities with locations, outcomes, effective config against an allowed-difference list, resolved module paths (everything inside the active sandbox) and a runtime load check for native bindings, worker construction and child processes. It passes on 246 files / 4541 tests and writes test/fixtures/mutation-oracle/envelope.json. The runtime check found commander loading child_process, so the classifier now rejects tests importing commander.

* feat(mutation): add the oracle's command-runner run and classifier

The Vitest 5 cross-check oracle runs its own Stryker run with the command runner (cleanTempDir false). A wrapper runs the scope's tests per mutant with an oracle reporter and writes a per-mutant record, and the oracle classifies every mutant into the KTD17 outcome classes with its own classifier. Oracle code imports no runner code.

* feat(mutation): add the oracle single-test launcher and cost benchmark

launch-single runs one test with one mutant in a fresh process inside the oracle's kept sandbox, selected by file:line plus an anchored name pattern, with the mutant installed before any import, an oracle setup file that proves the installed id and buckets hits (load and collection, hook, per-test), a wall-clock limit and a process-group kill. A launch is valid only when exactly the intended test ran. The benchmark projects the three-scope oracle sample's cost on 4 vCPU. A boundary test keeps oracle code from importing runner code.

* feat(mutation): add the oracle freeze and gate 3 compare

freeze samples mutants per scope with a recorded seed, tops up strata, counts stratum populations from the oracle's own coverage probe, launches every sampled mutant against every scope test (resumable), and writes a manifest keyed by content hashes of the scope's import closure, Vitest configs and prepare tool. compare checks freshness, inventories (tests and global mutant ids) before outcomes, every mutant's class and sampled per-test outcomes, allowing only the four KTD13 normalisations.

* feat(mutation): add gate 4 positive controls and determinism

Nine planted controls, selected from the scopes' own Stryker mutants by global id (three in relocated gitnexus-shared as-const tables), must be killed by exactly their expected killers. Two scheduling profiles (concurrency 1 with unlimited runner reuse, concurrency 3 with one mutant per runner process) must give identical outcome classes and kill sets.

* ci(mutation): add an advisory mutation-runner validation workflow

Runs the runner conformance suite (gate 2) on pull requests that touch the mutation tooling, its fixtures, the gitnexus manifest, lockfile or Vitest config. workflow_dispatch also runs gate 1, the gate 3 oracle compare (or a per-scope re-freeze whose manifests are uploaded, never committed), gate 4 and the oracle cost benchmark. All jobs are advisory. Documents the tooling in TESTING.md and the residual deltas against upstream StrykerJS.

* test(mutation): freeze the gate 3 oracle manifests and record Phase 0 results

Frozen Vitest 5 cross-check oracle manifests for the three pure-JS scopes (seed u7-gate3-1): shared.ts, schema-constants.ts and middleware.ts. Gate 3 passes: all 1,606 mutant classes and all 20,455 sampled per-test outcomes agree with the runner, 0 differences. Adds the approved Test Scenario Map design (updated with Phase 0 findings and gate results) and the Phase 0 implementation plan.

* refactor(mutation): simplify the Phase 0 tooling

Shared CLI/fs helpers for the non-oracle tools, an oracle-local util for isObject and atomic writes, formatMutateEntry and formatDuration reused, a typed isUnusableClass, and hot-path fixes: per-sidecar index in the gate 3 compare, lazy suite-chain in the runner's dry-run hit attribution, one Classifier per freeze, memoised display paths and Vitest lookups. Behaviour unchanged: unit tests, gate 3 compare (byte-identical report) and the conformance suite (23/23) pass.

* fix(review): apply review findings

- Move the duplicate-test scanner out of the published scripts/ directory to gitnexus/tools/test-dupes/ so it no longer ships in the npm package.
- Scope the orphan bench gate to touched lines, like every other scanner finding.
- Let the conformance environment case reach Stryker through an explicit childEnv override, so it exercises the runner's own __STRYKER_ACTIVE_MUTANT__ deletion (a negative check now fails it).
- Do not raise the zero-tests flag when a timeout outside-test row explains the zero.
- Run Stryker in its own process group and signal the group on cancellation.
- Fingerprint conformance inputs and refuse stale artefacts in --compare-only.
- Path-gate the mutation workflow on gitnexus-shared/package.json and the setup action; anchor the PR-report path filter with \A...\z.

* chore(security): mark the runner's provide/inject key as a gitleaks false positive

gitleaks' generic-api-key rule matched PROTOCOL_KEY = 'gitnexusStrykerVitest5' (a Vitest provide/inject key name). Allow the line inline and ignore the fingerprint in the commit that introduced it.

* test(mutation): re-freeze the schema-constants oracle manifest

Merging main changed the scope's import closure (template-arguments.ts),
so gate 3 reported re-freeze required. Re-frozen with seed u7-gate3-2:
72 sampled mutants, 3024 valid observations, 0 invalid. Gate 3 now
passes on all three scopes (shared 1232, schema-constants 72,
middleware 302 mutants) with 0 differences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): add the Test Scenario Map Phase 1 plan

* chore(security): stop the gitleaks allowlist comment from quoting the key

The previous comment quoted the provide/inject key assignment and gitleaks
flagged the comment line itself (generic-api-key). Reword it and allowlist
that historical finding by fingerprint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(mutation): select single tests in the vitest5 runner and keep per-test classes

Phase 1 U1 (KTD2, KTD9, KTD16):
- driver scope accepts optional testIds, passed as the vitest5.testIds
  runner option; the dry run collects, maps ids to task ids, runs only
  those tests and fails before any mutant run on unknown ids
- the dry-run sidecar (schema 2) carries the hook marker: per mutant,
  the tests a suite hook attributed it to
- results.json (schema 2) keeps every sidecar attempt's per-test rows
  (perTest), attemptsDisagree and hookAttributedTests
- gate 2 gains a selection scope and hook-marker cases (26 cases hold)

Gate 3 hashed inputs are untouched.

* refactor(test-dupes): export Layer 0's case extraction for the scenario map

Phase 1 U2 (KTD6): export extract, extractCases, buildFrame, closure,
staticValue, suppressed and dupeOkReason, plus a per-case record with
the describe path, title, context hash, body, each-table node and
enclosing loop head. Layer 0's --all scan output is byte-identical
before and after (1464 files, 114 findings).

* test(mutation): refresh the gate 1 envelope after the runner extensions

Gate 1 re-ran over the current pure-JS category after U1 and passed with
0 differences: 251 files, 4667 tests. The classifier now rejects
conformance-compare.test.ts (imports the child_process-spawning
conformance runner) and php-namespace-extraction.test.ts (now reaches
tree-sitter through main), and accepts the three oracle unit tests.

* feat(scenario-map): add the eligible-test inventory, identity join and anchors

Phase 1 U3 (R1, R2, R15; KTD3, KTD7):
- eligibility = gate 1 envelope intersect a fresh classifier run, with the
  delta; controls mode enrols a fixture's *.case.ts files
- static expansion of it.each/test.each/test.for tables and literal for-of
  loops into per-row instances; anything not statically evaluable is
  unresolved (fail closed)
- a pure join of static instances to runtime records through Phase 0
  test ids; any count or name mismatch makes the tests unsupported
- continuity anchors (no line; row and body fingerprints) and the one
  production-file predicate shared by every later stage
- test-dupes: additive exports (shapeOf, roleOf, eachTableOf, ASTs, an
  isTest option); Layer 0 output byte-identical

Over the repo: 4667 runtime tests, 4597 joined one-to-one, 70 unsupported
(5 tables built at runtime), 0 mismatches.

* test(scenario-map): add the controls fixture project with expected verdicts

Phase 1 U17: one pair per design §8 pure-JS adversarial case (Math.round,
parseInt radix, localeCompare, normalize, Date boundary, regex
alternatives, numeric boundary with a constants-only module, an
expected-only difference, a derived value reaching toFixed, an
enum-member-only difference), an order-dependent pair, an object-key row,
and positive controls (verbatim duplicate, repo arithmetic, Set table
reads). expected.json was written from the design's stated outcomes.
The fixture has its own Vitest project and tsconfig; no repo project
collects it.

* feat(scenario-map): extract per-instance checks and context

Phase 1 U5 (R5, KTD6):
- checks: the shape of each expect(...).matcher(...) with literals
  abstracted and selector literals in the subject kept exact; snapshots
  hashed from inline text or the .snap entry; a documented supported
  matcher list, anything else unsupported; per-row for it.each/test.for
- context: Layer 0's context hash plus body-level mocks/env writes and
  the test call's own modifiers and options
- checksContextKey for the second candidate source

Over the 251 eligible files: 4600 instances in 2.9 s, 183 with an
unsupported form, 37 with an unresolved selector.

* feat(scenario-map): add Layer 1 per-test V8 coverage and candidate retrieval

Phase 1 U4 (R3, R4; KTD4, KTD5):
- a Vitest setup file records per-test V8 block coverage in a prepared copy
  under the mutation config (gitnexus-shared runs from source), with design
  §5.2 attribution: module load discarded, a suite's beforeAll delta taken
  at its first beforeEach, concurrent tests flagged unattributable
- provenance per test: executed files, native-package entries and worker or
  child-process spawn call sites
- candidates: exact Jaccard >= 0.99 (size-ratio and prefix filters) plus
  identical checks-and-context buckets; connected components schedule
  Layer 3 only

Over the 251 eligible files: 4667 tests in 1:55 (82.8 s Vitest), 6281
candidate pairs in 611 groups, largest group 36 tests. Controls: 16/16
pairs are candidates, 0 cross-pair candidates. The directory is
path-coverage/ because coverage/ is ignored by git, prettier and eslint.

* feat(scenario-map): compute the per-instance input facet

Phase 1 U6 (R5, R9; KTD6), porting the input-facet spike without its
hard-coded inputs:
- rules 1-8: tree-sitter shapes of inline source, the COBOL profile,
  best-fit production tables (executed path, 2-hop import closure,
  property columns; never pooled), selector literals (reusing the checks
  predicate), object keys, source order, shared fixtures, fixture files
- numeric literals with 0/1/-1/negatives exact, and the +-1 rule against
  production numeric constants from executed files and the static import
  closure (folded initialisers included)
- discarded values per position with source locations for value flow
- the residual-difference guard: a body difference the facet misses makes
  the pair facet-unequal and both instances unresolved

Controls: all 16 pairs match expected.json. The spike's untouched
holdout-4 still scores 23/23 distinct. Eligible run: 4667 instances in
about 33 s, 245 unresolved.

* feat(scenario-map): run Layer 3 per candidate group and merge raw observations

Phase 1 U7 (R6, R7, R11; KTD7, KTD8, KTD14, KTD17):
- whole-file mutate scope per group from Layer 1 provenance and the
  static closure; executed-but-unmutated repo files mark tests excluded
- a run manifest of the prepared inputs that every shard verifies
- two plain Vitest 5 baselines (fresh instance per run) plus the Stryker
  dry run must agree; the generated config must carry perTest coverage,
  disableBail and static mutants before mutating
- raw observations from every sidecar attempt, keyed by group, run kind,
  lineage, attempt, test and mutant; class disagreement is inconclusive
  and a missing result is unknown, never a non-kill
- deterministic shard planning that never splits a group, with
  group-over-budget and start-only-if-it-fits rules

Controls: all 16 pairs ran as real groups; every kill set matches
expected.json.

* feat(scenario-map): add the full isolation pass for repeat-stage pairs

Phase 1 U8 (R7; KTD8, KTD9):
- select repeat-stage pairs (same coveredBy path, checks, context, facet
  without a residual difference, and the same non-empty kill set)
- per pair one together run and one alone run per test (shared across
  pairs), mutate restricted to the covered universe, a fresh runner
  process per mutant, all behind the same KTD8 baseline and config checks
  (runGroup's steps moved into runMutationRun)
- coverage and outcome class per test and mutant must agree across main,
  together and alone; a missing rerun result is unknown; hook-attributed
  mutants cap the pair at weak

Controls: order-dependent ends disagrees; verbatim-duplicate,
repo-arithmetic and set-non-member agree, matching expected.json.
Measured fresh-process cost: about 5.5 s per mutant.

* feat(scenario-map): add the row-deletion operator

Phase 1 U9 (R8, KTD10):
- find the one supported production table (object literal with plain
  keys, literal Set/Map) holding both differing values; spreads, computed
  keys, Object.assign, writes, getters, proxies, duplicate keys or an
  ambiguous table make the outcome inconclusive
- splice one key's text in a throwaway prepared copy, run the pair with a
  fresh Vitest instance per run after a control run that proves both pass
  and the edited file loaded
- symmetric / asymmetric (Row) / not applicable / inconclusive, with
  RowDeletion global mutant ids
- vitest-run: an opt-in load probe (probeFiles, loadedFiles)

Controls: all 16 pairs match expected.json (object-key-row and
set-membership asymmetric).

* feat(scenario-map): add the static value-flow check

Phase 1 U10 (R9, KTD11): for facet-equal pairs whose discarded values
differ, all sources must come back clean or the pair is vetoed to Review:
- Babel binds each differing value to its entry (call argument, fixture
  field, table key); an expected-only difference is unknown
- the TypeScript 7.0.2 unstable checker resolves entries and callees
  through aliases; a version mismatch degrades the whole run
- a walk stitched across functions over GitNexus PDG REACHING_DEF and
  CALL_SUMMARY edges (pdg_query flows are intra-procedural): opaque
  sinks (lib/@types/node_modules, any regex) veto, truncation, dynamic
  dispatch, a depth cap or a missing PDG layer are unknown or degraded;
  repo code is clean only inside the pair's mutated files
- row-deletion exports scanSupportedTables so table reads mean exactly
  the tables row deletion supports

Controls: 0 mismatches against expected.json. Whole-repo analyze --pdg:
6:49 wall, 8.4 GB peak RSS.

* feat(scenario-map): add the value-flow trace run

Phase 1 U11 (R9, KTD11): one traced Vitest 5 run over the tests of every
facet-equal candidate pair, with a setup file that wraps value-sensitive
built-ins (Math, string transforms and comparisons, RegExp, number
parsing and formatting, JSON, Intl, Date, sort, plus U10's opaque
callees) and records, per test and production call site, the receiver,
arguments and count. Storage and table reads stay unwrapped.
A shared site with a different receiver or argument, or asymmetric
sites or counts, vetoes the pair; a test whose outcome changes under
tracing makes it inconclusive.

Controls: all 13 pairs with a trace expectation match expected.json,
including receivers 24 and 25 reaching toFixed. Overhead on 40 eligible
files: 1.19x pooled, no outcome changes.

* feat(scenario-map): add the verdict engine

Phase 1 U12 (R10, R11, R15; KTD5, KTD8, KTD9, KTD12): a pure function
over merged observations and every stage's evidence. Design §2
precedence as an ordered rule list (Inconclusive, Out of scope, Weak,
Repeat strong, Repeat weak, Row, Review, Distinct), with the value-flow
and trace vetoes as a gate before any repeat category and the residual
difference ending in Review. One verdict per pair with its deciding rule
and every weak disqualifier; per-test Weak, Inconclusive and Subsumed;
per-file gaps recomputed from all observations with the §5.4 wording;
dupe-ok pairs kept with their evidence. Output is deterministic.

Controls: all 16 verdicts and rules match expected.json.

* feat(scenario-map): build the report, issue body and week-over-week diff

Phase 1 U13 (R12, R13; KTD13, KTD15):
- the JSON report: run provenance (the design §7 cache-key inputs),
  completeness and degraded flags, the eligibility delta, Layer 0
  findings, every verdict with its evidence, per-test properties, gaps,
  a seeded spot audit of 10 strong candidates, N of M evaluated and the
  precision-not-measured statement
- continuity: mutant fingerprints without revision or span, universe ids
  from the pair's common universe, evidence hashes per design §5.5
- a markdown rendering and the tracking-issue body (strong repeats, weak
  tests, gaps; truncated at 60,000 characters; a run-id marker so a
  stale run never overwrites a newer body)
- baseline comparison: new, changed, unchanged, resolved, with first-run
  and incompatible-baseline banners

* feat(scenario-map): add the CLI and the end-to-end controls run

Phase 1 U14 (R14, R16; KTD1, KTD14, KTD17): npm run test:scenario-map
with plan, shard, value-flow, merge, all and controls stages exchanging
JSON in a work directory, so CI jobs and a local run produce the same
report. Shards batch groups into one mutation run when their combined
estimate fits, and persist row-deletion and isolation results as they
finish. Exit codes are set explicitly.

controls: all 16 fixture pairs match expected.json (exit 0, 76 min).

* feat(scenario-map): fit the scheduled run into the CI budget

Phase 1 U14 follow-up (KTD14, KTD17), measured on a seeded slice of the
real category; no evidence rule is relaxed:
- value flow runs before the shards and re-plans them, so isolation is
  spent only on pairs that can still be strong (a value-flow veto or
  unknown ends Review, a trace inconclusive ends Inconclusive); a
  missing, stale or degraded value-flow result falls back to the
  provisional plan
- a shards stage prints the final shard count for the CI matrix
- --concurrency and --speed-factor feed the estimate; the main-run cost
  form is fixed + mutated bytes (measured; it does not grow with tests)
- a group whose isolation does not fit still runs its main run and row
  deletion; its pairs get isolation not-run (weak cap), and a shard
  isolates deferred pairs only when their measured universe fits
- groups that execute no production file are not run; their pairs are
  out of scope

Predicted CI run: 9 shards at concurrency 4, 1658-1829 min in total,
largest shard 196-253 min (budget 300), 0 groups over budget.
controls: 16/16 pairs match, 28 min.

* ci(scenario-map): add the weekly Test Scenario Map workflow and a PR controls job

Phase 1 U15 (R13, R14; KTD13, KTD14, KTD17):
- test-scenario-map.yml (weekly + dispatch, advisory): plan, value-flow,
  shards (sizes the matrix from the value-flow re-plan), a shard matrix
  (fail-fast off, max-parallel 10, 300-minute deadline), merge (always,
  against the newest scenario-map-baseline artifact) and upsert-issue
  (default branch only; the only job with issues: write; replaces the
  tracking issue's body only for a complete run that is not older than
  the body's run marker, comments on partial runs, never closes it)
- mutation-runner-validation.yml: a path-gated scenario-map-controls job
  that runs the controls fixture end to end on PRs touching the tooling

actionlint (with shellcheck), zizmor 1.24.1 and the concurrency check
pass. Neither workflow can run before the merge.

* docs(scenario-map): record Phase 1 in the design and the testing guide

Phase 1 U16: design corrections (the stitched PDG walk, the trace's
receivers and scope, row deletion and baselines on plain Vitest 5, the
isolation pass on testIds after value flow), the demote-only additions,
the refreshed gates, the measured Phase 1 costs and the CI topology
change; TESTING.md gains a Test Scenario Map section (stages, controls
fixture, supported matchers, wrapped built-ins, weekly workflow,
dupe-ok).

* fix(scenario-map): store observations compactly so large batches fit in memory

The first full run crashed a shard with "Invalid string length":
observations were materialised for every collected test x every mutant,
including a valid not-covered survival for each pair (1.6M objects for a
batch of 73 selected tests over 21,965 mutants; the batch file needed
about 700 MB of JSON).

- only a run's selected tests produce observations
- each run records its universe (selected tests, mutants with their
  status, reasons and hook marker); only rows, covered or static
  mutants without a row, and hook-named tests are stored explicitly; a
  not-covered survival is derived when read
- every consumer (pair comparison, isolation, verdict engine, report,
  batch split) reads through one index; group observation and isolation
  run files move to schema 2, and old files are rejected with a message
- merge lists an unreadable group result as incomplete instead of
  dropping it; repeat-stage decisions keep comparison summaries only

On the crashing batch: 141,308 explicit observations instead of 1.6M,
12.1 MB instead of an overflow, and a byte-identical verdict report.
controls: 16/16 pairs match.

* docs(scenario-map): record the Phase 1 measured slice; give the real-repo classify test headroom

- design §9: the full-category report comes from the first weekly
  workflow run; the local full run was stopped because it saturated the
  shared machine. Records the measured slice and the whole-category plan
  and value-flow figures.
- classify.test.ts: classify the real repo once in beforeAll with a
  120 s timeout; it takes about 30 s on a busy host and timed out at
  the default 30 s.

* fix(review): apply review findings and simplify the scenario-map tool

Review (ce-code-review, validated findings):
- trace: a baseline or traced run that is invalid, inconclusive, retried
  or has outside-test failures, unhandled errors, unwrapped entries or
  recorder errors makes every pair inconclusive (trace-run-invalid), never
  clean; the value-flow stage reports it and merge marks the run partial
- verdict: tests pin that a not-required static check for differing
  values reads as unknown, plus the worker-spawn, missing-provenance,
  unresolved-input and unknown-hook-marker weak caps
- workflow: the baseline comes only from this workflow's own scheduled or
  dispatched default-branch runs of this repository; an unreadable
  baseline shows the incompatible-baseline banner instead of failing merge
- controls: the oracle fails on an empty expected list, on any unlisted
  repeat verdict and on a case file without an expected pair
- value flow: the PDG analyze budget (25 min) sits below the 50-minute
  CI step; the TypeScript checker server is closed when opening fails
- --concurrency now also bounds every in-process Vitest run (maxWorkers),
  so local runs stay within the requested cores
- merge reads isolation runs tolerantly (an unreadable file makes the run
  partial); a duplicate test title is renamed; a stale TESTING.md heading
  reference is updated

Simplification (behavior-preserving; merge and plan outputs byte-identical
on fixed inputs): one util.ts for shared helpers, a run-level row-deletion
cache, one classify() per plan, memoised universe ids and single-pass
pair/subsumption/report loops.

* fix(test-dupes): keep beforeAll and beforeEach fixture builds apart

A same-fixture-setup key now includes the hook name. A beforeAll fixture is
built once per suite and a beforeEach one per test, so sharing one result
between them would drop per-test isolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): temporary label-gated pull_request run for PR #3686

schedule and workflow_dispatch only run from the default branch, so the
Test Scenario Map workflow cannot run before this PR merges. Adding the
run-ci label to the PR starts one full run. A pull_request run writes no
tracking issue and no baseline. Remove the trigger before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): remove the temporary pull_request trigger

The one full run on PR #3686 finished (run 40183982111, complete, 9 of 9
shards). This reverts 5d06d731c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(scenario-map): decide pairs past unknown mutants on a kill witness

A mutant both tests ran cleanly, killed by assertion by exactly one, proves
the kill sets differ whatever the unknown and runtime-class mutants hold.
kills-unknown and non-assertion-outcome now leave such a pair Inconclusive
no longer; it goes on to Row or Distinct. A pair's empty kill set only
decides Weak on a fully known run. Verdict engine 2.

Re-merged from the artifacts of run 40183982111: Inconclusive 4532 -> 1469;
every pair engine 1 decided keeps its verdict, rule and details, and the
repeat candidates are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(scenario-map): record a test file's import failure instead of a missing result

A static mutant that breaks a module at import makes its test files fail to
collect: the runner writes one file-level killed-outside-test:collection row
and no per-test rows, so every test of the file was a missing result. Those
were every missing-result observation of run 40183982111 (597 mutants, 675
same-file pairs left Inconclusive by them alone).

Extraction now records each test of a failed file with that class. When both
tests of a pair failed to import, neither ran, so the mutant cannot tell them
apart: comparePair counts it as a non-kill for both rather than another
class. Verdict engine 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): temporary label-gated pull_request run for PR #3686

schedule and workflow_dispatch only run from the default branch, so the
Test Scenario Map workflow cannot run before this PR merges. Adding the
run-ci label to the PR starts one full run. A pull_request run writes no
tracking issue and no baseline. Remove the trigger before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): remove the temporary pull_request trigger again

The engine 3 run on PR #3686 finished (run 40208819019, complete, 9 of 9
shards). This reverts 0e76acc09.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(scenario-map): record the engine 3 run's results

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): scenario fingerprint verdicts (same path, input, expectation)

Two tests are duplicates when they run the same execution path on the same
input data and check the same expectation. The fingerprint engine decides
that from Layer 1 block sets, Layer 2 facets and discarded values, the
checks-and-context key and the trace run, with no mutation: duplicate,
possible-duplicate (anything unresolved, fail-closed), table-row (other
input values), different-checks, distinct.

On run 40208819019 the fully resolved triple names 26 pairs: the run's 13
strong repeats and 13 more mutation could not decide, none contradicted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): lead the report with the scenario fingerprint

The report (schema 2) carries every pair's fingerprint verdict with its
continuity key and the run's mutation scope. The markdown and the tracking
issue lead with duplicates and same-scenario-different-checks pairs; the
mutation sections (strong repeats, weak tests, gaps) render only when the
run gathered mutation evidence. The spot audit samples duplicates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): mutation scope (none, possible-duplicates, all)

plan and all take --mutation. The plan computes every pair's fingerprint
before the trace and packs only the scope's target: no group (none), every
group (all), or the groups with a possible duplicate, trimmed to the tests
of those pairs and the files they executed. Value flow re-plans a narrower
scope after the trace (a usable trace only narrows it) and builds the PDG
index for scope all only. Merge reports every pair's fingerprint and the
mutation verdicts of the pairs whose tests ran in a mutated group. A plan
with nothing to mutate has no shard; all skips the shard stage then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): check fingerprint verdicts in the controls oracle

expected.json gives every control its fingerprint verdict and rule, written
from the fingerprint rules before any run: verbatim-duplicate is the one
duplicate, enum-member a possible duplicate (residual difference),
numeric-boundary distinct, the other 13 table rows. The controls stage
compares both verdicts per pair and fails on an unlisted fingerprint
duplicate. Local controls run: all 16 pairs match on both.

value-flow checks for dist/ only when it builds the PDG index (scope
all); controls, and all with scope all, still check before any stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): fingerprint weekly, mutation on dispatch or label

The weekly run is the fingerprint only (mutation scope none: no build, no
shards). Workflow dispatch takes a mutation input (none, possible-duplicates,
all). Adding the scenario-map label to a pull request starts the deep run
over the possible duplicates; it writes no issue and no baseline, and every
run puts the issue-body view in the job summary. Pull request runs get their
own concurrency slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(scenario-map): the scenario fingerprint as the primary verdict

The design defines a duplicate as the same execution path, input data and
expectation, records the maintainer decisions of 2026-10-25 (fingerprint
weekly, mutation on dispatch or label) and the local measurement: the
fingerprint-only run took 8.2 min over all 4,632 eligible tests and named
the 26 duplicates the evidence predicted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ci): place a single shard artifact in the scenario-map merge

download-artifact unpacks a pattern that matches one artifact into the
path itself, not into a directory named after it. With a one-shard plan
(the scenario-map label's possible-duplicates run) the placement step found
no shard directory, so merge reported the shard missing and every mutated
pair Inconclusive (run 40268120818). The step now uses the download path
when no artifact directory exists. A local re-merge of that run's artifacts
gives the complete result, recorded in the design doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): fingerprint every unit and integration test

The fingerprint needs no Stryker run, so its scope is no longer the
mutation runner's validated envelope. Eligibility now carries
fingerprintFiles (every *.test.ts under test/unit and test/integration)
next to the mutation-eligible files; a verdict input carries eligible
(mutation), fingerprintEligible and passed.

- Layer 1 and the trace run the base config's lbug-db files in a
  partition of their own with one worker (partitions.ts), the rest
  with --concurrency workers, and merge both runs.
- New fingerprint rule not-passing: a test that did not pass in
  Layer 1 makes its pairs Inconclusive.
- The mutation target keeps only pairs whose two tests are
  mutation-eligible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): run Layer 1 in CI slices; load vendor and dist in integration runs

A sample of the integration tests showed two problems with fingerprinting
every unit and integration test:

- Half the sampled files failed to load: the prepare step (gate-3 hashed)
  leaves vendor/ and dist/ out of the prepared copy, and integration tests
  load leiden, the parse worker and the built CLI. Layer 1 and the trace now
  copy both from the checkout in normal mode, and every stage that runs tests
  (coverage, plan without --coverage, value-flow, all, controls) requires a
  built dist/.
- One plan job cannot hold Layer 1: 44 of 435 integration files ran past
  35 minutes. A new coverage stage (--index i --of n) runs one slice, split
  like the CI test shards (test/helpers/shard-balance.ts), and
  plan --coverage <dir> merges the slice files: one block table, indices
  remapped, files and tests in file order. A slice that did not report leaves
  its files unrecorded. The workflow runs 12 coverage slices before the plan.

The sequential partition now reads every fileParallelism: false project of
the base config (lbug-db and cli-e2e), not lbug-db alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(scenario-map): order-dependent controls pair is Inconclusive on the fingerprint

The not-passing fingerprint rule (a test that did not pass in Layer 1 holds
its pairs Inconclusive) came after expected.json was written. Layer 1 runs
every test twice in a row (repeats 1, design §5.2); the order-dependent
fixture's shared counter makes each test's second attempt fail, and Vitest
reports a test failed when any repetition fails. So rule 2, not-passing,
decides the pair before input-values-differ. The expectation follows from
the rule and the fixture's code; the CI controls run that reported the
difference is run 40278026899.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scenario-map): skip Layer 1 for files no fingerprint finding can come from

Past the path rule, a pair is a duplicate, a possible duplicate or a table
row only when both checks-and-context keys are equal or one is unknown, and
the key is static (facet/checks.ts over the static inventory). Layer 1 now
runs only the files that hold a test whose key another test shares or that
has none, plus every file the static view cannot vouch for: a call whose
instance count is not static, a file snapshot read, a call at registration
level that is not the test API (a helper such as defineRenameFixtureTests or
describeGrammarPresence, or an it alias), or a parse failure. Mutation scope
all and the controls still run every file; the coverage jobs get the run's
--mutation so they slice the same files as the plan.

On the current checkout this skips 449 of 1,399 test files, about 24% of the
weighted Layer 1 cost. On the 250 unit files of the last full run, the static
keys reproduce all 3,313 runtime keys and select the same 172 files; runtime
collection (vitest list --no-static-parse) of the skipped files finds no test
the static inventory misses. Given up: different-checks pairs with a test of
a skipped file (0 on the last full unit run). The report says how many files
Layer 1 ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scenario-map): diff-scope the labelled pull request run

A labelled pull request asks whether its own tests duplicate anything, yet
the run fingerprinted the whole suite (12 coverage slices, hours of runner
time). plan --changed-since <ref> now scopes it to the diff:

- the changed test files are the fingerprint's files the diff since the
  merge base of ref and HEAD added or modified;
- Layer 1 runs the changed files the prefilter keeps plus every file with
  a test whose checks-and-context key a changed test shares;
- candidate pairs are kept only when they have a test of a changed file,
  before the groups form, so the mutation target, the trace and the report
  follow;
- a diff that changes no test file writes an empty work file and an empty,
  complete report.

The workflow skips the coverage matrix for pull requests and passes the
merge commit's first parent (fetch-depth 2); the plan job builds dist/ and
runs Layer 1 itself. The report names the changed files. Locally, over the
two test files of the previous commit, the run took 4 min 41 s end to end
(Layer 1 over 2 files 1.3 min, trace 1.1 min) on a loaded shared host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scenario-map): size Layer 1 at 4 coverage slices, not 12

Twelve slices came from an estimate of three times the CI coverage time,
built on a local integration sample that other suites on the shared host
had slowed down. The test workflow runs the whole unit and integration
suite with coverage in three 7-8 minute shards (ci.yml run 30014110469),
and an instrumented Layer 1 run shows its overhead is small: an
integration file's work sits in beforeAll, which Layer 1 neither repeats
nor snapshots per test (resolvers/rust: 10.1 s of coverage snapshots
against about 190 s for the file). With each slice paying about five
minutes of setup, twelve slices spent more on setup than on tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): report the Layer 1 tests that got no fingerprint

Tests that Layer 1 saw but did not run dropped out of the candidates
without a word: skipped ones (skip, skipIf, todo; the GITNEXUS_BENCH
benchmark suites, which run only in the test workflow's benchmarks job)
and unfinished ones (pending, because their file's run ended first). The
report's eligibility section now counts both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(scenario-map): close gaps a code review found in the Layer 1 work

- The registration guard let a suite whose callback is not written
  inline (describe('x', defineCases)) through, so a file whose tests the
  inventory cannot see could be skipped. Such a suite now keeps its file.
- An empty coverage slice exited 2 and uploaded nothing; it now writes
  an empty slice file.
- --changed-since was accepted with the controls fixture, whose run must
  cover every file; that is now a usage error.
- The report counted the files of a coverage slice that did not report
  as run. It now says how many files wrote no Layer 1 record.
- A diff that only changes a snapshot file scoped nothing; a changed
  __snapshots__/<name>.snap now counts as a change of its test file.
- The diff-scope report claimed no other file can hold a changed test's
  duplicate. Files whose tests have no static key are not run, so it now
  says a possible duplicate there is not looked for.

Not changed: mutation shards never get integration tests, because
mutationTarget keeps only pairs of two mutation-eligible tests; a slice
that disagrees with the plan stops the merge on purpose (the slices
header now says so).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scenario-map): cut setup cost and duplication in the Layer 1 work

A four-angle cleanup review (reuse, simplification, efficiency,
altitude) of the Layer 1 slices, prefilter and diff scope.

Fixed along the way: a shard's row-deletion batch ran every planned pair
of a complete group, including pairs whose tests the mutation target had
trimmed away (an integration test, or a pair outside the scope); it now
runs only pairs whose two tests the trimmed group holds (shardRowPlans).

Setup cost:
- Coverage jobs ran the Phase 0 classifier twice (about 33 s each) just
  to list the fingerprint's files; fingerprintScope lists them directly.
- The plan recomputed the prefilter (about 30 s) to rebuild the slice
  split. Slice files now carry Layer 1's file list, the mutation scope
  and the revision; the plan checks those against its own run.
- The prefilter re-parsed every test file. The unseen-registration check
  now lives in the static inventory (StaticInventory.incomplete), on the
  ASTs it holds and test-dupes' roleOf, which also catches
  test.describe with a named callback and it('x', namedFn).

Simplification: runCoverage writes the empty work file itself; reuse
partitionFiles, codeList, writeJson and path.matchesGlob (the hand-made
glob matcher read ** differently from Vitest); sameList moves to util;
copyBuildOutput owns the controls rule; slices are read before the work
directory is emptied, so --coverage no longer has to lie outside --work;
a dead sort is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(scenario-map): record the Layer 1 cleanup review in the fingerprint plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(scenario-map): full recall for diff-scoped pull request runs

A labelled pull request's run left out unchanged files whose tests have
no static checks-and-context key (helper-registered tests, dynamic it.each
counts, file snapshots, no assertion key), so a changed test's possible
duplicate in one was never looked for. Such a test can pair with any
changed test, so the diff scope now runs every one of those files too.

That floor is 230 of 1,407 test files, about 59% of the suite's weighted
Layer 1 cost with 59 sequential files: too much for the plan job alone.
A labelled pull request therefore runs the coverage slices as well:
coverage --changed-since records the changed files in each slice, and the
plan takes the diff scope from the slices (plan --changed-since stays for
a local all run). The plan job no longer builds dist/ or fetches history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(scenario-map): temporary owner-only test run on PR #3686

TEMPORARY, remove before merge. The scenario-map-test label starts the
whole-suite run (no diff scope) with mutation possible-duplicates, so
every stage (coverage slices, plan, value flow, mutation shards, merge)
can be tested on CI before the workflow reaches the default branch, where
workflow_dispatch would serve. It runs only when magyargergo adds the
label on PR #3686; the label on any other pull request, or by anyone
else, leaves every job skipped. The scenario-map label keeps its
diff-scoped run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: stabilize scenario-map test selection

* fix: exclude retry tests from scenario traces

* fix: keep captured loop inputs unresolved

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Gergő Magyar 2026-10-27 16:46:58 +00:00 • committed by GitHub
parent b2c72997ea
commit bad2422aa1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
214 changed files with 79422 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
.github/workflows/ci-report.yml vendored
View file

@ -69,6 +69,7 @@ jobs:
const temp = process.env.RUNNER_TEMP;
await downloadArtifact('pr-meta', path.join(temp, 'dl'));
await downloadArtifact('test-reports', path.join(temp, 'dl'));
await downloadArtifact('test-dupes', path.join(temp, 'dl'));
- name: Extract artifacts
shell: bash
@ -456,6 +457,34 @@ jobs:
echo ""
fi
# ── Possible duplicate tests (advisory, ci.yml test-dupes job) ──
# The JSON is produced by PR code, so only whitelisted kinds and
# strict path/line shapes reach the comment; free text is dropped.
DUPES_FILE=$(find_first "$DIR/test-dupes" "test-dupes.json")
if [ -n "$DUPES_FILE" ]; then
jq -r '
def path_ok: type == "string" and test("\\A[A-Za-z0-9._/@+-]{1,300}\\z");
def line_ok: type == "number" and . >= 1 and . <= 1000000 and floor == .;
def loc: "`\(.file):\(.line)`";
[ .findings[]?
| select(.kind | IN("same-test", "assertions-pinned", "same-fixture-setup", "copied-helper", "orphan-bench-gate"))
| select((.file | path_ok) and (.line | line_ok))
| .matches = ([(.matches // [])[]? | select((.file | path_ok) and (.line | line_ok))][:3])
]
| if length == 0 then empty else
"### Possible duplicate tests",
"",
"Advisory only, never blocks merge: \(length) finding(s) on lines this PR changes. Run `npm run test:dupes` in `gitnexus/` for details; mark an intentional repeat with `// dupe-ok: <reason>`.",
"",
"| Kind | Changed | Already at |",
"|------|---------|------------|",
(.[:20][] | "| `\(.kind)` | \(loc) | \(if (.matches | length) == 0 then "no workflow runs it" else (.matches | map(loc) | join("<br>")) end) |"),
(if length > 20 then "", "…and \(length - 20) more." else empty end),
""
end
' "$DUPES_FILE" 2>/dev/null || true
fi
echo "---"
echo "<sub>📋 [View full run](${RUN_URL}) · Generated by CI</sub>"
echo "GITNEXUS_CI_REPORT_EOF_7f3a"

31
.github/workflows/ci.yml vendored
View file

@ -48,6 +48,37 @@ jobs:
permissions:
contents: read
# ── Advisory duplicate-test scan ─────────────────────────────────
# Comment-only: ci-report.yml renders this JSON into the sticky PR
# comment. Kept out of `ci-status.needs` and marked continue-on-error,
# so neither a finding nor a scanner crash can block a merge.
# fetch-depth 2: HEAD is the PR merge commit, HEAD^1 the base tip.
test-dupes:
name: Duplicate test scan
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 10
continue-on-error: true
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 2
- uses: ./.github/actions/setup-gitnexus
with:
lifecycle-scripts: 'false'
- name: Scan lines this PR changes
run: npm run test:dupes -- --base HEAD^1 --json ../test-dupes/test-dupes.json
working-directory: gitnexus
- name: Upload scan result
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: test-dupes
path: test-dupes/
retention-days: 1
if-no-files-found: ignore
# ── Save PR metadata for the reporting workflow ─────────────────
# The ci-report.yml workflow (triggered by workflow_run) needs the
# PR number and job results to post a comment. We save them as an

394
.github/workflows/mutation-runner-validation.yml vendored Normal file
View file

@ -0,0 +1,394 @@
name: Mutation Runner Validation
# Validates the repo-local Stryker test runner for Vitest 5 and its validation gates
# (gitnexus/tools/mutation/, TESTING.md "Mutation testing tooling"), and the test
# scenario map built on it (gitnexus/tools/scenario-map/).
#
# pull_request gate 2: the conformance suite, one job per profile; and the scenario-map
# controls run. Runs only when the tooling (tools/mutation, tools/scenario-map,
# tools/test-dupes), its fixtures (mutation-scenario-map among them), the
# gitnexus package manifest or lockfile (Vitest and Stryker versions), the base
# Vitest config, the gitnexus-shared package manifest (its exports drive the
# mutation config's aliases) or the setup-gitnexus action (the jobs' install
# path) change. The paths are workflow-level, so a scenario-map change also runs
# gate 2, an accepted extra cost.
# workflow_dispatch gate 2 plus gate 1 (fidelity), gate 3 (oracle), gate 4 (controls) and
# the oracle cost benchmark, each selectable by an input.
#
# Advisory: every job is continue-on-error and none is in ci.yml's ci-status.needs. Nothing here
# blocks a merge. Gate 3's full sample takes hours, so it never runs on pull requests.
#
# Gate 3 (`oracle` input):
# compare one job: a fresh vitest5 runner run per frozen manifest in
# gitnexus/test/fixtures/mutation-oracle/, then one compare over all three, so the
# strata are counted across the scopes combined, as the freeze counted them. Cheap:
# one Stryker run per scope.
# refreeze re-freezes manifests from scratch, one job per scope (`refreeze_scope`). The
# full three-scope sample does not fit one 6-hour job on 4 vCPU (benchmark: 6.18 h
# at best), so it is sharded by scope: shared about 4.5 h, schema-constants about
# 45 min, middleware about 1 h. The new manifest is uploaded as an artifact for a
# maintainer to review and commit; CI never commits it. Local equivalent, from
# gitnexus/ (the work dir must lie outside the checkout; a freeze is resumable):
# npm run test:mutation:oracle:freeze -- \
# --from-manifest test/fixtures/mutation-oracle/<scope>.json \
# --seed <new seed> --work <dir outside the checkout>
#
# Gate 4 runs as one job: both profiles (A: three concurrency-1 runs in parallel, B: three
# concurrency-3 runs in turn) took about 81 min locally under load. Split jobs would have to
# ship every run directory, prepared copies included, to a `--compare-only` job at the same
# absolute paths, because the staleness check hashes the prepared inputs.
#
# Every tool writes outside the checkout (`--out`/`--work` under $RUNNER_TEMP): a prepared copy
# must not lie inside it. setup-gitnexus installs Node 22, the Node major recorded in the frozen
# manifests; another major makes gate 3 report "re-freeze required". Timings were measured on a
# loaded WSL2 machine with the checkout on a 9p mount; ubuntu-latest (4 vCPU, local disk) will
# differ, so the timeouts leave margin.
on:
pull_request:
branches: [main]
paths:
- '.github/workflows/mutation-runner-validation.yml'
- 'gitnexus/tools/mutation/**'
- 'gitnexus/tools/scenario-map/**'
- 'gitnexus/tools/test-dupes/**'
- 'gitnexus/test/fixtures/mutation-*/**'
- 'gitnexus/package.json'
- 'gitnexus/package-lock.json'
- 'gitnexus/vitest.config.ts'
- 'gitnexus-shared/package.json'
- '.github/actions/setup-gitnexus/**'
workflow_dispatch:
inputs:
fidelity:
description: 'Gate 1: execution-path fidelity on the pure-JS category'
type: boolean
default: true
oracle:
description: 'Gate 3: compare the frozen manifests, re-freeze them, or skip'
type: choice
options: [compare, refreeze, skip]
default: compare
refreeze_scope:
description: 'Oracle scope to re-freeze (oracle = refreeze)'
type: choice
options: [all, shared, schema-constants, middleware]
default: all
seed:
description: 'New sample seed for a re-freeze (required when oracle = refreeze)'
type: string
required: false
default: ''
controls:
description: 'Gate 4: planted controls and determinism'
type: boolean
default: true
bench:
description: 'Oracle cost benchmark'
type: boolean
default: true
permissions:
contents: read
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
# A new push supersedes a PR run; manual dispatches (hours long) are never cancelled.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
defaults:
run:
shell: bash
working-directory: gitnexus
jobs:
# ── Gate 2: runner conformance suite ───────────────────────────────
conformance:
name: Gate 2 conformance (${{ matrix.profile }})
runs-on: ubuntu-latest
timeout-minutes: 30
continue-on-error: true
permissions:
contents: read
strategy:
fail-fast: false
matrix:
# The profile names of gitnexus/test/fixtures/mutation-conformance/expected.json.
profile: [concurrency-2-reuse-1, concurrency-1-reuse-unlimited]
steps:
# persist-credentials: false — the job uploads artifacts and never pushes.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Full install, as the test jobs do: tsx, Stryker and Vitest are devDependencies.
- uses: ./.github/actions/setup-gitnexus
- name: Run the conformance suite
env:
PROFILE: ${{ matrix.profile }}
run: npm run test:mutation:conformance -- --profile "$PROFILE" --out "$RUNNER_TEMP/mutation/conformance"
- name: Upload conformance results
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-conformance-${{ matrix.profile }}
path: |
${{ runner.temp }}/mutation/conformance/observed.json
${{ runner.temp }}/mutation/conformance/*/*/results.json
${{ runner.temp }}/mutation/conformance/*/*/stryker.log
retention-days: 14
# ── Gate 1: execution-path fidelity and envelope ───────────────────
fidelity:
name: Gate 1 fidelity
if: github.event_name == 'workflow_dispatch' && inputs.fidelity
runs-on: ubuntu-latest
timeout-minutes: 60
continue-on-error: true
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
# The envelope goes to the artifact, not over the committed
# test/fixtures/mutation-oracle/envelope.json.
- name: Run gate 1
run: |
npm run test:mutation:fidelity -- \
--out "$RUNNER_TEMP/mutation/fidelity" \
--envelope "$RUNNER_TEMP/mutation/envelope/envelope.json"
- name: Upload gate 1 report and envelope
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-gate1
path: |
${{ runner.temp }}/mutation/fidelity/*.json
${{ runner.temp }}/mutation/envelope/envelope.json
retention-days: 14
# ── Gate 3: compare a fresh runner run with the frozen manifests ───
oracle-compare:
name: Gate 3 oracle compare
if: github.event_name == 'workflow_dispatch' && inputs.oracle == 'compare'
runs-on: ubuntu-latest
timeout-minutes: 120
continue-on-error: true
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Run the runner on every oracle scope and compare
run: |
work="$RUNNER_TEMP/mutation/gate3"
mkdir -p "$work"
pairs=()
for name in shared schema-constants middleware; do
manifest="test/fixtures/mutation-oracle/$name.json"
if [ ! -f "$manifest" ]; then
echo "::error::$manifest is missing; freeze it first"
exit 2
fi
jq '.scope' "$manifest" > "$work/$name.scope.json"
# Exit 1 means results with integrity flags; gate 3 reports those as differences.
status=0
npm run test:mutation:run -- --scope "$work/$name.scope.json" \
--out "$work/$name" --concurrency 4 || status=$?
if [ "$status" -gt 1 ]; then exit "$status"; fi
pairs+=(--scope "$manifest,$work/$name/results.json")
done
npm run test:mutation:oracle -- "${pairs[@]}" --out "$work/gate3-report.json"
- name: Upload gate 3 report
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-gate3
path: |
${{ runner.temp }}/mutation/gate3/gate3-report.json
${{ runner.temp }}/mutation/gate3/*.scope.json
${{ runner.temp }}/mutation/gate3/*/results.json
${{ runner.temp }}/mutation/gate3/*/stryker.log
retention-days: 14
# ── Gate 3: re-freeze manifests, sharded by scope ──────────────────
oracle-refreeze:
name: Gate 3 re-freeze (${{ matrix.scope }})
if: github.event_name == 'workflow_dispatch' && inputs.oracle == 'refreeze'
runs-on: ubuntu-latest
# Projected: shared ~4.5 h with Stryker runs and setup; the others ~1 h. 6 h job cap.
timeout-minutes: ${{ matrix.scope == 'shared' && 355 || 150 }}
continue-on-error: true
permissions:
contents: read
strategy:
fail-fast: false
matrix:
scope: ${{ fromJSON(inputs.refreeze_scope == 'all' && '["shared","schema-constants","middleware"]' || format('["{0}"]', inputs.refreeze_scope)) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Check the seed
env:
SCOPE: ${{ matrix.scope }}
SEED: ${{ inputs.seed }}
run: |
manifest="test/fixtures/mutation-oracle/$SCOPE.json"
if [ -z "$SEED" ]; then
echo "::error::a re-freeze needs a new sample seed (the seed input)"
exit 2
fi
frozen="$(jq -r '.sample.seed' "$manifest")"
if [ "$SEED" = "$frozen" ]; then
echo "::error::$manifest was frozen with seed $frozen; give a new seed"
exit 2
fi
# --manifest lies outside the checkout, so the manifest records the default
# test/fixtures/mutation-oracle/<scope>.json path, and only a freeze without problems
# writes it. <work>/manifest.json is written either way and lists any problems.
- name: Re-freeze the scope
env:
SCOPE: ${{ matrix.scope }}
SEED: ${{ inputs.seed }}
run: |
out="$RUNNER_TEMP/mutation/refreeze"
mkdir -p "$out"
npm run test:mutation:oracle:freeze -- \
--from-manifest "test/fixtures/mutation-oracle/$SCOPE.json" \
--seed "$SEED" --work "$out/work" --manifest "$out/$SCOPE.json"
- name: Upload the re-frozen manifest
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-oracle-refreeze-${{ matrix.scope }}
path: |
${{ runner.temp }}/mutation/refreeze/${{ matrix.scope }}.json
${{ runner.temp }}/mutation/refreeze/work/manifest.json
${{ runner.temp }}/mutation/refreeze/work/freeze-work.json
${{ runner.temp }}/mutation/refreeze/work/command/oracle-command-run.json
${{ runner.temp }}/mutation/refreeze/work/*/stryker.log
retention-days: 30
# ── Gate 4: planted controls and determinism ───────────────────────
controls:
name: Gate 4 controls
if: github.event_name == 'workflow_dispatch' && inputs.controls
runs-on: ubuntu-latest
timeout-minutes: 240
continue-on-error: true
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Run gate 4
run: npm run test:mutation:controls -- --work "$RUNNER_TEMP/mutation/controls"
- name: Upload gate 4 report
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-gate4
path: |
${{ runner.temp }}/mutation/controls/controls-report.json
${{ runner.temp }}/mutation/controls/runs/*/*/results.json
${{ runner.temp }}/mutation/controls/runs/*/*/stryker.log
retention-days: 14
# ── Oracle cost benchmark ──────────────────────────────────────────
bench:
name: Oracle cost benchmark
if: github.event_name == 'workflow_dispatch' && inputs.bench
runs-on: ubuntu-latest
timeout-minutes: 150
continue-on-error: true
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
# Per oracle scope: the oracle's command-runner run and a vitest5 runner run, then the
# benchmark over all three. Concurrency 4 matches the recorded numbers.
- name: Run the oracle cost benchmark
run: |
work="$RUNNER_TEMP/mutation/bench"
mkdir -p "$work"
scopes=()
for name in shared schema-constants middleware; do
manifest="test/fixtures/mutation-oracle/$name.json"
if [ ! -f "$manifest" ]; then
echo "::error::$manifest is missing; freeze it first"
exit 2
fi
jq '.scope' "$manifest" > "$work/$name.scope.json"
npm run test:mutation:oracle:command -- --scope "$work/$name.scope.json" \
--out "$work/$name-command" --concurrency 4
# Exit 1 means results with integrity flags; the benchmark only reads which
# mutants are static and which tests cover the timed-out ones.
status=0
npm run test:mutation:run -- --scope "$work/$name.scope.json" \
--out "$work/$name-vitest5" --concurrency 4 || status=$?
if [ "$status" -gt 1 ]; then exit "$status"; fi
scopes+=(--scope "$work/$name-command/oracle-command-run.json,$work/$name-vitest5/results.json")
done
npm run test:mutation:bench -- "${scopes[@]}" --out "$work/report" --concurrency 4
- name: Upload benchmark report
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: mutation-oracle-bench
path: |
${{ runner.temp }}/mutation/bench/report/oracle-cost.json
${{ runner.temp }}/mutation/bench/report/oracle-cost.txt
${{ runner.temp }}/mutation/bench/*-command/oracle-command-run.json
${{ runner.temp }}/mutation/bench/*/stryker.log
retention-days: 14
# ── Scenario map controls ──────────────────────────────────────────
# The PR check of test-scenario-map.yml's pipeline (plan, value flow, one shard, merge) over the
# controls fixture (gitnexus/test/fixtures/mutation-scenario-map/), mutation scope `all`, then
# every expected.json mutation and fingerprint verdict and rule compared; exit 1 means a pair
# differs. build: 'true' because the value-flow stage's PDG index
# spawns analyze workers from dist/. About 30 min locally.
scenario-map-controls:
name: Scenario map controls
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 60
continue-on-error: true
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
- name: Run the scenario-map controls
env:
GITNEXUS_WORKER_READY_TIMEOUT_MS: '120000'
run: npm run test:scenario-map -- controls --work "$RUNNER_TEMP/scenario-map-controls"
- name: Upload the controls report
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-controls-${{ github.run_attempt }}
path: |
${{ runner.temp }}/scenario-map-controls/scenario-map-report.json
${{ runner.temp }}/scenario-map-controls/scenario-map-report.md
${{ runner.temp }}/scenario-map-controls/issue-body.md
${{ runner.temp }}/scenario-map-controls/plan.json
${{ runner.temp }}/scenario-map-controls/value-flow.json
${{ runner.temp }}/scenario-map-controls/shard-0.json
${{ runner.temp }}/scenario-map-controls/value-flow/pdg/analyze.log
retention-days: 14

755
.github/workflows/test-scenario-map.yml vendored Normal file
View file

@ -0,0 +1,755 @@
name: Test Scenario Map
# Advisory report on the validated pure-JS unit tests of gitnexus/ (R13, R14, KTD13, KTD14,
# KTD17; docs/plans/2026-10-25-1700-feat-scenario-fingerprint-plan.md): duplicate tests (the same
# execution path, input data and expectation: the scenario fingerprint), and, with mutation
# evidence, tests that kill no evaluated mutant and per-file gaps. The CLI is
# gitnexus/tools/scenario-map/scenario-map.ts; its header documents the stages, flags,
# work-directory layout and exit codes. Local equivalent, from gitnexus/ (the work dir must lie
# outside the checkout; `npm run build` first: the fingerprint's integration tests load dist/):
# npm run test:scenario-map -- all --work <dir> --mutation none
#
# Triggers and mutation scope (`--mutation`, types.ts `MutationScope`):
# schedule weekly, `none`: the fingerprint only, minutes, no shard;
# workflow_dispatch input `mutation`: `none` (default), `possible-duplicates` or `all` (the
# full Phase 1 run, hours over up to 20 shards);
# pull_request TEMPORARY, remove before merge: the `scenario-map-test` label, added by
# magyargergo on PR #3686 only, runs the whole suite (no diff scope) with
# mutation `possible-duplicates`, to test every stage on CI before merge.
# When a maintainer adds the `scenario-map` label to a pull request:
# `possible-duplicates`, the deep run for a suspicious change, diff-scoped
# (`--changed-since` on the coverage slices: only pairs with a test of a changed
# test file; Layer 1 over those files, the files sharing their
# checks-and-context keys and every file with a test of no static key). A PR
# run writes no tracking issue and no baseline; its report is an artifact and
# the job summary.
#
# Jobs, each one CLI stage over one work directory ($RUNNER_TEMP/scenario-map) exchanged as
# artifacts:
# coverage one matrix job per Layer 1 slice (4, fail-fast off): the fingerprint's unit and
# integration test files, split like the CI test shards.
# plan eligibility, Layer 0, the prepared copy and run manifest, Layer 1 merged from the
# slices (a slice that did not report leaves its files unrecorded), Layer 2,
# candidates and the provisional shard plan.
# value-flow for scope `all` the PDG index (`analyze --pdg` spawns workers from dist/) and the
# static value-flow check; the trace run and the final shard plan
# (a narrower scope re-plans its target after the trace). It runs
# before the shards, so a shard isolates only the pairs value flow leaves able to be
# strong (the CLI's deviation from KTD14's parallel value-flow job).
# shards prints the final shard count (the re-plan's, or the provisional plan's when value
# flow left no usable result); it sizes the shard matrix, and 0 skips the shards.
# shard one matrix job per shard (fail-fast off): Layer 3 group runs, row deletion and
# isolation until a 300-minute deadline (KTD17). Results upload even on failure.
# merge always runs once a plan exists: merges whatever the shards and value flow left
# (a missing shard leaves the run partial and its pairs Inconclusive, R11), compares
# against the newest `scenario-map-baseline` artifact of an earlier completed run of
# this workflow on the default branch, and writes the report. Only a complete
# default-branch run uploads a new baseline (KTD13).
# upsert-issue default branch only: a complete run replaces the tracking issue's body unless
# the issue already records a newer run; a partial run comments and leaves the body
# alone. The issue is found by label in any state and never closed.
#
# Advisory: every job is continue-on-error, so nothing here fails the run or blocks a merge. Exit
# code 1 of a stage means "finished with incomplete results"; it is a warning, the job carries on
# and merge reports the run as partial. Exit code 2 (usage or setup error) fails the step.
# The full matrix can hold up to 20 runners for about 5.5 hours.
#
# Artifact names carry the run attempt. A downstream job takes the plan and value-flow artifact
# names from those jobs' outputs, so re-running one job reuses the earlier attempt's inputs; merge
# copies shard results oldest attempt first, so a later attempt of a shard wins.
#
# The controls run (the U17 fixture, every expected mutation and fingerprint verdict) is a PR job
# in mutation-runner-validation.yml.
on:
schedule:
# Mondays 04:17 UTC, outside the busy CI hours.
- cron: '17 4 * * 1'
workflow_dispatch:
inputs:
mutation:
description: 'Mutation evidence: none (fingerprint only), possible-duplicates, or all'
type: choice
options: [none, possible-duplicates, all]
default: none
seed:
description: 'Seed of every sampled choice, the spot audit included (default: the run id)'
type: string
required: false
default: ''
shards:
description: 'Number of mutation shards (default: the fewest that fit, at most 20)'
type: string
required: false
default: ''
# The deep run of one pull request: only the `scenario-map` label starts it (the plan job's
# `if`), and the TEMPORARY `scenario-map-test` label on PR #3686 by magyargergo; any other
# label event leaves every job skipped.
pull_request:
types: [labeled]
permissions:
contents: read
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
# Weekly and manual runs share the tracking issue and the baseline, so they share one slot and run
# one at a time; each pull request has its own slot. A run that holds hours of shard work is never
# cancelled. (GitHub keeps one pending run: a third run queued behind a pending one replaces it.)
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || 'default' }}
cancel-in-progress: false
defaults:
run:
shell: bash
working-directory: gitnexus
env:
# Stryker concurrency: the plan's cost estimate and the shards' runs (ubuntu-latest: 4 vCPUs).
SCENARIO_MAP_CONCURRENCY: '4'
# One shard's budget in the plan and its deadline in the shard job (KTD17). The shard job's
# timeout-minutes (330) leaves room for setup and the last uploads.
SCENARIO_MAP_DEADLINE_MINUTES: '300'
jobs:
# ── Coverage: Layer 1 in slices ────────────────────────────────────
coverage:
name: Coverage ${{ matrix.index }}
# TEMPORARY: the `scenario-map-test` clause (PR #3686 testing); remove before merge.
if: github.event_name != 'pull_request' || github.event.label.name == 'scenario-map' || (github.event.label.name == 'scenario-map-test' && github.event.sender.login == 'magyargergo' && github.event.pull_request.number == 3686)
runs-on: ubuntu-latest
# Four slices balanced like the CI test shards (test/helpers/shard-balance.ts): the whole unit
# and integration suite runs with coverage in three 7-8 minute shards of the test workflow
# (ci.yml run 30014110469). Layer 1 repeats only test bodies and snapshots per test (about 5%
# of an integration file whose work sits in beforeAll), and the prefilter drops a quarter.
timeout-minutes: 75
continue-on-error: true
permissions:
contents: read
strategy:
fail-fast: false
matrix:
index: [0, 1, 2, 3]
steps:
# A pull request's merge commit needs its first parent (the base) for the diff scope.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 1 }}
# Integration tests load dist/ (the built CLI and its worker scripts).
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
- name: Run the slice
timeout-minutes: 60
env:
INDEX: ${{ matrix.index }}
OF: ${{ strategy.job-total }}
# The plan's scope (its "Choose the mutation scope" step): unless `all`, Layer 1 skips the
# files whose tests all have a checks-and-context key no other test shares.
MUTATION: ${{ github.event_name == 'pull_request' && 'possible-duplicates' || inputs.mutation || 'none' }}
EVENT: ${{ github.event_name }}
LABEL: ${{ github.event.label.name }}
run: |
args=(--index "$INDEX" --of "$OF" --mutation "$MUTATION"
--work "$RUNNER_TEMP/scenario-map" --concurrency "$SCENARIO_MAP_CONCURRENCY")
# The merge commit's first parent is the base: the pull request's own changes. The
# TEMPORARY `scenario-map-test` run covers the whole suite.
if [ "$EVENT" = "pull_request" ] && [ "$LABEL" = "scenario-map" ]; then
args+=(--changed-since 'HEAD^1')
fi
status=0
npm run test:scenario-map -- coverage "${args[@]}" || status=$?
if [ "$status" -eq 1 ]; then
echo "::warning::a test file of slice $INDEX failed or wrote no record; the report counts it"
elif [ "$status" -ne 0 ]; then
exit "$status"
fi
- name: Upload the slice
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-coverage-${{ github.run_attempt }}-${{ matrix.index }}
path: ${{ runner.temp }}/scenario-map/coverage-slice-${{ matrix.index }}.json
if-no-files-found: error
retention-days: 14
# ── Plan: Layers 0-2, candidates, shard plan, run manifest ─────────
plan:
name: Plan
needs: coverage
# TEMPORARY: the `scenario-map-test` clause (PR #3686 testing); remove before merge.
if: ${{ !cancelled() && (github.event_name != 'pull_request' || github.event.label.name == 'scenario-map' || (github.event.label.name == 'scenario-map-test' && github.event.sender.login == 'magyargergo' && github.event.pull_request.number == 3686)) }}
runs-on: ubuntu-latest
timeout-minutes: 60
continue-on-error: true
permissions:
contents: read
outputs:
artifact: ${{ steps.artifact.outputs.name }}
mutation: ${{ steps.scope.outputs.mutation }}
# Scheduled runs always run on the default branch; their event payload has no repository.
default_branch: ${{ github.event_name == 'schedule' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
steps:
# persist-credentials: false — the jobs upload artifacts and never push.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Layer 1 comes from the slices: the plan runs no test and needs no dist/.
- uses: ./.github/actions/setup-gitnexus
- name: Choose the mutation scope
id: scope
env:
MUTATION: ${{ github.event_name == 'pull_request' && 'possible-duplicates' || inputs.mutation || 'none' }}
run: echo "mutation=$MUTATION" >> "$GITHUB_OUTPUT"
# No slice at all fails the plan (nothing to merge); a missing slice leaves its files
# unrecorded in the report.
- name: Download the coverage slices
continue-on-error: true
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: scenario-map-coverage-*
path: ${{ runner.temp }}/scenario-map-coverage
- name: Place the coverage slices
run: |
shopt -s nullglob
root="$RUNNER_TEMP/scenario-map-coverage"
dest="$RUNNER_TEMP/scenario-map-slices"
mkdir -p "$dest"
found=("$root"/scenario-map-coverage-*)
# download-artifact unpacks a pattern that matches ONE artifact into the path itself.
if [ "${#found[@]}" -eq 0 ] && [ -d "$root" ]; then found=("$root"); fi
# scenario-map-coverage-<attempt>-<index>: oldest attempt first, so a later attempt wins.
while IFS= read -r dir; do
[ -n "$dir" ] || continue
for file in "$dir"/coverage-slice-*.json; do cp "$file" "$dest/"; done
done < <(printf '%s\n' "${found[@]}" | sort -V)
ls "$dest"
- name: Plan the run
env:
SEED: ${{ inputs.seed || github.run_id }}
SHARDS: ${{ inputs.shards }}
MUTATION: ${{ steps.scope.outputs.mutation }}
run: |
# The slices carry Layer 1, and a pull request's diff scope with it.
args=(--work "$RUNNER_TEMP/scenario-map" --seed "$SEED" --mutation "$MUTATION"
--concurrency "$SCENARIO_MAP_CONCURRENCY"
--deadline-minutes "$SCENARIO_MAP_DEADLINE_MINUTES"
--coverage "$RUNNER_TEMP/scenario-map-slices")
if [ -n "$SHARDS" ]; then args+=(--shards "$SHARDS"); fi
npm run test:scenario-map -- plan "${args[@]}"
- name: Upload the plan
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-plan-${{ github.run_attempt }}
path: |
${{ runner.temp }}/scenario-map/plan.json
${{ runner.temp }}/scenario-map/scenario-run-manifest.json
${{ runner.temp }}/scenario-map/layer0.json
${{ runner.temp }}/scenario-map/layer2.json
${{ runner.temp }}/scenario-map/coverage/coverage.json
if-no-files-found: error
retention-days: 14
- name: Name the plan artifact
id: artifact
env:
NAME: scenario-map-plan-${{ github.run_attempt }}
run: echo "name=$NAME" >> "$GITHUB_OUTPUT"
# ── Value flow: PDG static check, trace run, final shard plan ──────
value-flow:
name: Value flow
needs: plan
if: ${{ !cancelled() && needs.plan.outputs.artifact != '' }}
runs-on: ubuntu-latest
# Locally: PDG index about 3 min (peak RSS about 8.4 GB of the runner's 16 GB), the whole
# stage about 5 min.
timeout-minutes: 60
continue-on-error: true
permissions:
contents: read
outputs:
artifact: ${{ steps.artifact.outputs.name }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# The trace runs integration tests, and the PDG index of scope `all` spawns workers: dist/.
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
- name: Download the plan
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.plan.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
- name: Run value flow
timeout-minutes: 50
env:
GITNEXUS_WORKER_READY_TIMEOUT_MS: '120000'
run: |
status=0
npm run test:scenario-map -- value-flow --work "$RUNNER_TEMP/scenario-map" || status=$?
if [ "$status" -eq 1 ]; then
echo "::warning::value flow is degraded, or its trace did not run or was invalid; the report says so"
elif [ "$status" -ne 0 ]; then
exit "$status"
fi
- name: Upload the value-flow result
id: upload
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-value-flow-${{ github.run_attempt }}
path: |
${{ runner.temp }}/scenario-map/value-flow.json
${{ runner.temp }}/scenario-map/value-flow/pdg/analyze.log
retention-days: 14
# Named only when value-flow.json was uploaded: without it, shards and merge fall back to
# the provisional plan and merge reports value flow missing.
- name: Name the value-flow artifact
id: artifact
if: always() && steps.upload.outcome == 'success'
env:
NAME: scenario-map-value-flow-${{ github.run_attempt }}
run: |
if [ -f "$RUNNER_TEMP/scenario-map/value-flow.json" ]; then
echo "name=$NAME" >> "$GITHUB_OUTPUT"
fi
# ── Shard count: sizes the matrix ──────────────────────────────────
shards:
name: Shard count
needs: [plan, value-flow]
# Scope `none` plans no shard.
if: ${{ !cancelled() && needs.plan.outputs.artifact != '' && needs.plan.outputs.mutation != 'none' }}
runs-on: ubuntu-latest
timeout-minutes: 20
continue-on-error: true
permissions:
contents: read
outputs:
count: ${{ steps.count.outputs.count }}
matrix: ${{ steps.count.outputs.matrix }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Download the plan
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.plan.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
- name: Download the value-flow result
if: needs.value-flow.outputs.artifact != ''
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.value-flow.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
# --silent: the stage prints the count and nothing else; npm's banner would precede it.
- name: Read the final shard count
id: count
run: |
count="$(npm run --silent test:scenario-map -- shards --work "$RUNNER_TEMP/scenario-map")"
if ! [[ "$count" =~ ^[0-9]+$ ]]; then
echo "::error::the shards stage printed '$count', not a shard count"
exit 1
fi
echo "count=$count" >> "$GITHUB_OUTPUT"
echo "matrix=$(jq -cn --argjson n "$count" '[range($n)]')" >> "$GITHUB_OUTPUT"
echo "Final shard plan: $count shard(s)"
# ── Layer 3 shards ─────────────────────────────────────────────────
shard:
name: Shard ${{ matrix.index }}
needs: [plan, value-flow, shards]
if: ${{ !cancelled() && needs.shards.outputs.matrix != '' && needs.shards.outputs.count != '0' }}
runs-on: ubuntu-latest
timeout-minutes: 330
continue-on-error: true
permissions:
contents: read
strategy:
fail-fast: false
# Leave runner slots for PR CI on the default plan (about 9 shards at concurrency 4).
max-parallel: 10
matrix:
index: ${{ fromJSON(needs.shards.outputs.matrix) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Download the plan
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.plan.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
- name: Download the value-flow result
if: needs.value-flow.outputs.artifact != ''
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.value-flow.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
# The shard verifies its own prepared copy against the run manifest; a mismatch runs
# nothing and leaves its groups' pairs Inconclusive. The step timeout sits inside the job's,
# so a shard that overruns its deadline still uploads what it finished.
- name: Run the shard
timeout-minutes: 315
env:
INDEX: ${{ matrix.index }}
COUNT: ${{ needs.shards.outputs.count }}
run: |
status=0
npm run test:scenario-map -- shard --work "$RUNNER_TEMP/scenario-map" \
--index "$INDEX" --of "$COUNT" \
--deadline-minutes "$SCENARIO_MAP_DEADLINE_MINUTES" \
--concurrency "$SCENARIO_MAP_CONCURRENCY" || status=$?
if [ "$status" -eq 1 ]; then
echo "::warning::shard $INDEX finished with incomplete results; the report says which"
elif [ "$status" -ne 0 ]; then
exit "$status"
fi
# Every group result is written as soon as it exists: upload whatever the shard finished,
# even after a failure.
- name: Upload the shard results
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-shard-${{ github.run_attempt }}-${{ matrix.index }}
path: |
${{ runner.temp }}/scenario-map/shard-${{ matrix.index }}.json
${{ runner.temp }}/scenario-map/row-deletion-shard-${{ matrix.index }}.json
${{ runner.temp }}/scenario-map/observations/
${{ runner.temp }}/scenario-map/batches/
${{ runner.temp }}/scenario-map/isolation/
${{ runner.temp }}/scenario-map/groups/*/stryker/stryker.log
${{ runner.temp }}/scenario-map/groups/*/*/stryker/stryker.log
retention-days: 14
# ── Merge: verdicts and the report ─────────────────────────────────
merge:
name: Merge and report
needs: [plan, value-flow, shard]
if: ${{ always() && needs.plan.outputs.artifact != '' }}
runs-on: ubuntu-latest
timeout-minutes: 30
continue-on-error: true
permissions:
contents: read
actions: read # list and download an earlier run's scenario-map-baseline artifact
outputs:
artifact: ${{ steps.artifact.outputs.name }}
complete: ${{ steps.merge.outputs.complete }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
- name: Download the plan
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.plan.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
- name: Download the value-flow result
if: needs.value-flow.outputs.artifact != ''
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.value-flow.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map
# No shard result at all is a valid, partial outcome: merge reports every shard missing.
- name: Download the shard results
continue-on-error: true
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: scenario-map-shard-*
path: ${{ runner.temp }}/scenario-map-shards
- name: Place the shard results
run: |
shopt -s nullglob
work="$RUNNER_TEMP/scenario-map"
root="$RUNNER_TEMP/scenario-map-shards"
found=("$root"/scenario-map-shard-*)
# download-artifact unpacks a pattern that matches ONE artifact into the path itself, not
# into a directory named after it (a one-shard plan): the path is then that artifact.
if [ "${#found[@]}" -eq 0 ] && [ -d "$root" ]; then found=("$root"); fi
# scenario-map-shard-<attempt>-<index>: oldest attempt first, so a later attempt wins.
while IFS= read -r dir; do
[ -n "$dir" ] || continue
echo "Using $(basename "$dir")"
cp -R "$dir/." "$work/"
done < <(printf '%s\n' "${found[@]}" | sort -V)
# KTD13: the newest non-expired `scenario-map-baseline` of an earlier completed run of THIS
# workflow on the default branch; only complete default-branch runs upload one. Never a
# repository-wide artifact listing: it holds fork pull_request runs too, and a fork branch
# may carry the default branch's name. Only schedule and workflow_dispatch runs count (a
# labelled pull request run never does), and each run's head repository is checked as well. None found: "first run" banner. A
# failure here must not cost the run its report: merge then runs without a baseline, and a
# baseline merge cannot read gets the incompatible-baseline banner.
- name: Fetch the baseline
id: baseline
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
DEFAULT_BRANCH: ${{ github.event.repository.default_branch || github.ref_name }}
run: |
dir="$RUNNER_TEMP/scenario-map-baseline"
mkdir -p "$dir"
if ! gh api --method GET \
"repos/$GITHUB_REPOSITORY/actions/workflows/test-scenario-map.yml/runs" \
-f branch="$DEFAULT_BRANCH" -f status=completed -f per_page=20 > "$dir/runs.json"; then
echo "::warning::could not list the earlier runs; the report shows no baseline"
exit 0
fi
id=""
# Newest first: this workflow's scheduled or manual runs other than this one, whose
# repository and head repository are this repository.
while IFS= read -r run_id; do
[ -n "$run_id" ] || continue
if ! gh api "repos/$GITHUB_REPOSITORY/actions/runs/$run_id/artifacts?name=scenario-map-baseline" \
< /dev/null > "$dir/artifacts.json"; then
continue
fi
id="$(jq -r '[.artifacts[] | select(.expired | not)]
| sort_by(.created_at) | last | .id // empty' "$dir/artifacts.json")"
if [ -n "$id" ]; then
echo "Baseline run: $run_id"
break
fi
done < <(jq -r --argjson run "$GITHUB_RUN_ID" --argjson repo "$GITHUB_REPOSITORY_ID" '
[.workflow_runs[]
| select(.id != $run
and (.event == "schedule" or .event == "workflow_dispatch")
and .repository.id == $repo
and .head_repository.id == $repo)]
| sort_by(.created_at) | reverse | .[].id' "$dir/runs.json")
if [ -z "$id" ]; then
echo "No earlier scenario-map-baseline artifact: first run, no baseline."
exit 0
fi
if ! gh api "repos/$GITHUB_REPOSITORY/actions/artifacts/$id/zip" > "$dir/baseline.zip" \
|| ! unzip -q -o "$dir/baseline.zip" -d "$dir/report" \
|| [ ! -f "$dir/report/scenario-map-report.json" ]; then
echo "::warning::could not fetch baseline artifact $id; the report shows no baseline"
exit 0
fi
echo "Baseline: artifact $id"
echo "report=$dir/report/scenario-map-report.json" >> "$GITHUB_OUTPUT"
- name: Merge
id: merge
env:
BASELINE: ${{ steps.baseline.outputs.report }}
ON_DEFAULT_BRANCH: ${{ needs.plan.outputs.default_branch }}
RUN_ATTEMPT: ${{ github.run_attempt }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
REF: ${{ github.ref }}
run: |
work="$RUNNER_TEMP/scenario-map"
args=(--work "$work" --run-id "$GITHUB_RUN_ID" --run-attempt "$RUN_ATTEMPT"
--run-url "$RUN_URL" --ref "$REF")
if [ -n "$BASELINE" ]; then args+=(--baseline "$BASELINE"); fi
if [ "$ON_DEFAULT_BRANCH" = 'true' ]; then args+=(--default-branch); fi
status=0
npm run test:scenario-map -- merge "${args[@]}" || status=$?
if [ "$status" -eq 1 ]; then
echo "::warning::partial report: some shard or value-flow result is missing"
elif [ "$status" -ne 0 ]; then
exit "$status"
fi
complete="$(jq -r '.completeness.complete == true' "$work/scenario-map-report.json")"
echo "complete=$complete" >> "$GITHUB_OUTPUT"
- name: Upload the report
id: upload
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-report-${{ github.run_attempt }}
path: |
${{ runner.temp }}/scenario-map/scenario-map-report.json
${{ runner.temp }}/scenario-map/scenario-map-report.md
${{ runner.temp }}/scenario-map/issue-body.md
retention-days: 90
# The tracking-issue view on the run page: a pull request run's only rendered report.
- name: Summarise the report
if: always()
run: |
body="$RUNNER_TEMP/scenario-map/issue-body.md"
if [ -f "$body" ]; then tail -n +2 "$body" >> "$GITHUB_STEP_SUMMARY"; fi
- name: Name the report artifact
id: artifact
if: always() && steps.upload.outcome == 'success'
env:
NAME: scenario-map-report-${{ github.run_attempt }}
run: |
if [ -f "$RUNNER_TEMP/scenario-map/scenario-map-report.json" ]; then
echo "name=$NAME" >> "$GITHUB_OUTPUT"
fi
# overwrite: a re-run attempt replaces this run's own baseline instead of failing on it.
- name: Upload the baseline
if: steps.merge.outputs.complete == 'true' && needs.plan.outputs.default_branch == 'true'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: scenario-map-baseline
path: ${{ runner.temp }}/scenario-map/scenario-map-report.json
if-no-files-found: error
retention-days: 90
overwrite: true
# ── Tracking issue (R13) ───────────────────────────────────────────
# The only job with a write scope, and only on the default branch. No install scripts run here
# (lifecycle-scripts: 'false'): tsx alone runs the decision.
upsert-issue:
name: Upsert tracking issue
needs: [plan, merge]
if: ${{ !cancelled() && needs.plan.outputs.default_branch == 'true' && needs.merge.outputs.artifact != '' }}
runs-on: ubuntu-latest
timeout-minutes: 15
continue-on-error: true
permissions:
contents: read
issues: write # find, open, update or comment on the tracking issue (and its label)
env:
LABEL: test-scenario-map
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/setup-gitnexus
with:
lifecycle-scripts: 'false'
- name: Download the report
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: ${{ needs.merge.outputs.artifact }}
path: ${{ runner.temp }}/scenario-map-report
- name: Find the tracking issue
id: find
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const fs = require('fs');
const path = require('path');
const issues = await github.paginate(github.rest.issues.listForRepo, {
owner: context.repo.owner,
repo: context.repo.repo,
labels: process.env.LABEL,
state: 'all',
per_page: 100,
});
// Any state; the oldest one when several carry the label.
const tracking = issues
.filter((issue) => !issue.pull_request)
.sort((a, b) => a.number - b.number)[0];
if (tracking === undefined) {
core.info('No tracking issue yet.');
return;
}
const issueDir = path.join(process.env.RUNNER_TEMP, 'scenario-map-issue');
fs.mkdirSync(issueDir, { recursive: true });
fs.writeFileSync(path.join(issueDir, 'existing-body.md'), tracking.body ?? '');
core.setOutput('number', String(tracking.number));
core.info(`Tracking issue #${tracking.number} (${tracking.state}).`);
# The decision is issue-body.ts's own R13/KTD13 rule (shouldUpdateIssue): a complete
# default-branch run updates unless the body's first-line run marker records a newer run.
- name: Decide the update
id: decide
env:
ISSUE_NUMBER: ${{ steps.find.outputs.number }}
run: |
node --import tsx --input-type=module <<'EOF'
import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import path from 'node:path';
import { shouldUpdateIssue } from './tools/scenario-map/report/issue-body.ts';
import { completenessLine, degradedLine, runLink } from './tools/scenario-map/report/markdown.ts';
const { ISSUE_NUMBER, GITHUB_OUTPUT, RUNNER_TEMP } = process.env;
const reportDir = path.join(RUNNER_TEMP, 'scenario-map-report');
const issueDir = path.join(RUNNER_TEMP, 'scenario-map-issue');
const report = JSON.parse(
readFileSync(path.join(reportDir, 'scenario-map-report.json'), 'utf8'),
);
const existing = ISSUE_NUMBER
? readFileSync(path.join(issueDir, 'existing-body.md'), 'utf8')
: null;
let action = 'skip';
if (report.completeness.complete) {
if (shouldUpdateIssue(report, existing)) action = existing === null ? 'create' : 'update';
else console.log('The issue already records a newer run: body left alone.');
} else if (existing !== null) {
action = 'comment';
const lines = [
`Partial ${runLink(report.run)}: the issue body is left unchanged; only a complete run updates it.`,
'',
`- ${completenessLine(report)}`,
`- ${degradedLine(report)}`,
'',
"The run's report artifact holds every result it produced.",
];
mkdirSync(issueDir, { recursive: true });
writeFileSync(path.join(issueDir, 'comment.md'), `${lines.join('\n')}\n`);
} else {
console.log('Partial run and no tracking issue yet: nothing to comment on.');
}
appendFileSync(GITHUB_OUTPUT, `action=${action}\n`);
console.log(`Tracking issue action: ${action}`);
EOF
# Bodies are read from files, never passed through env or outputs. Never closes the issue.
- name: Apply the update
if: steps.decide.outputs.action != 'skip'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
ACTION: ${{ steps.decide.outputs.action }}
ISSUE_NUMBER: ${{ steps.find.outputs.number }}
with:
script: |
const fs = require('fs');
const path = require('path');
const { owner, repo } = context.repo;
const label = process.env.LABEL;
const read = (dir, file) =>
fs.readFileSync(path.join(process.env.RUNNER_TEMP, dir, file), 'utf8');
switch (process.env.ACTION) {
case 'create': {
try {
await github.rest.issues.getLabel({ owner, repo, name: label });
} catch (error) {
if (error.status !== 404) throw error;
await github.rest.issues.createLabel({
owner,
repo,
name: label,
color: 'c5def5',
description: 'Tracking issue of the weekly test scenario map',
});
}
const { data } = await github.rest.issues.create({
owner,
repo,
title: 'Test scenario map: repeat candidates, weak tests and gaps',
body: read('scenario-map-report', 'issue-body.md'),
labels: [label],
});
core.info(`Opened tracking issue #${data.number}.`);
break;
}
case 'update':
await github.rest.issues.update({
owner,
repo,
issue_number: Number(process.env.ISSUE_NUMBER),
body: read('scenario-map-report', 'issue-body.md'),
});
core.info(`Updated the body of #${process.env.ISSUE_NUMBER}.`);
break;
case 'comment':
await github.rest.issues.createComment({
owner,
repo,
issue_number: Number(process.env.ISSUE_NUMBER),
body: read('scenario-map-issue', 'comment.md'),
});
core.info(`Commented on #${process.env.ISSUE_NUMBER}.`);
break;
default:
throw new Error(`unknown action ${process.env.ACTION}`);
}

4
.gitignore vendored
View file

@ -34,6 +34,10 @@ coverage/
.vitest/
.tmp-test/
gitnexus/.tmp-test/
# Mutation testing (gitnexus/tools/mutation). Prepared copies and run directories live
# outside the checkout; these catch a Stryker run started inside it.
.stryker-tmp/
gitnexus/reports/mutation/
# Misc
*.local

4
.gitleaksignore
View file

@ -2,3 +2,7 @@
c9fdab17f25ebaf332fba6e6ba55ee328f20fe66:README.md:curl-auth-header:348
# Synthetic Kotlin Actuator fixture value from PR #3107; no credential was present.
3951079300a18b14e79f5b5f5dd778ae19ced6e3:gitnexus/test/integration/spring-actuator-kotlin-runtime-pipeline.test.ts:generic-api-key:8
# Vitest provide/inject key name in the mutation runner protocol from PR #3686; no credential was present.
bf4fe7666212c1e2a82aea8309095f63e4b23dcc:gitnexus/tools/mutation/runner/protocol.ts:generic-api-key:11
# The previous wording of the comment above quoted that key name and was itself flagged; no credential was present.
02989dd688f863248993122879818ffc4cc95e11:.gitleaksignore:generic-api-key:5

156
TESTING.md
View file

@ -26,6 +26,7 @@ From `gitnexus/`:
| `npm run test:parity` | Scope-resolution parity for all migrated languages | After changing resolver or scope code |
| `npm run test:cross-platform` | Platform-sensitive subset only | Debugging a Windows/macOS issue |
| `npm run test:watch` | Vitest in watch mode | Active development |
| `npm run test:dupes` | Duplicate-test scan of lines changed vs `main` | Before adding tests or benchmarks |
Vitest transpiles TypeScript without checking types. Run `npm run typecheck:tests`
in `gitnexus/` to check test code and helpers against the production types. This
@ -159,12 +160,167 @@ The PR report shows the reconciled result as **Unverified**. Zero means every te
has execution evidence; individual OS logs still show tests that require another
OS as skipped. Failed executions remain failures even if another job passes.
### Duplicate-test scan (advisory)
The `Duplicate test scan` job in `ci.yml` runs `npm run test:dupes` on the lines a PR
changes, and the PR report lists what it finds under **Possible duplicate tests**. It
never fails a build. It reports:
| Kind | Meaning |
| -------------------- | --------------------------------------------------------------------------- |
| `same-test` | Same body and same setup as an existing test |
| `assertions-pinned` | Every assertion is a literal `fn(...)` → matcher another test already makes |
| `same-fixture-setup` | A `before*` hook builds a fixture another describe already builds |
| `copied-helper` | A top-level helper identical to one in another file or `test/helpers` |
| `orphan-bench-gate` | A bench script with `--check` that no workflow or npm script runs |
"Same setup" includes enclosing hooks, describe-level statements, wrapper arguments,
loop heads, `each` tables, referenced file-local declarations and resolved imports, so
identical bodies that run against different fixtures are not reported. Mark an
intentional repeat with `// dupe-ok: <reason>` on the reported line or the line above.
Locally, `npm run test:dupes` compares against `origin/main`; pass `--base <ref>` to
change that, or `--all` to scan every test. Python tests are not scanned.
`npm run test:benchmarks` discovers all tests gated by `GITNEXUS_BENCH` and runs
them serially. Keep timing measurements out of parallel coverage workers. The
`eval-tests` job installs the locked Python dependencies and runs the workflow
preflight Vitest tests as well as pytest; those tests must exercise real Python
validation, never a stubbed success.
## Mutation testing tooling
`gitnexus/tools/mutation/` holds a Stryker test runner for Vitest 5 (`vitest5`) and the tools
that validate it. The runner supports Vitest `5.0.x` only and refuses any other version. Its
validated scope is the pure-JS category: 251 unit test files (4,667 tests) that load no native
code, workers or child processes and read no fixtures. The gates stay out of the main suite
(only the tools' unit tests in `test/unit/mutation/` run there), and nothing ships in the npm
package.
Every run works in a prepared copy outside the checkout (`--out`/`--work` must lie outside it).
The copy blanks each `as` type assertion in the mutated files with spaces of equal length,
because Stryker never mutates code under one, and relocates `gitnexus-shared/src` into the
package. From `gitnexus/` (each tool documents its options in its file header):
| Command | What it does |
| -------------------------------------- | --------------------------------------------------------------------- |
| `npm run test:mutation:run` | One Stryker run over a scope file with the `vitest5` runner |
| `npm run test:mutation:classify` | Lists the pure-JS unit test files |
| `npm run test:mutation:fidelity` | Gate 1: the unmutated run matches plain Vitest; records the envelope |
| `npm run test:mutation:conformance` | Gate 2: pinned outcomes of the conformance fixture project |
| `npm run test:mutation:oracle` | Gate 3: compares runner runs with the frozen oracle manifests |
| `npm run test:mutation:oracle:freeze` | Freezes (or re-freezes) one oracle manifest |
| `npm run test:mutation:oracle:command` | The oracle's own Stryker run, with Stryker's `command` runner |
| `npm run test:mutation:controls` | Gate 4: planted positive controls and determinism across two profiles |
| `npm run test:mutation:bench` | Oracle cost benchmark |
| `npm run test:mutation:prepare` | Builds a prepared copy on its own |
Frozen artifacts live in `gitnexus/test/fixtures/`: `mutation-conformance/` (the gate 2
fixture project and `expected.json`), `mutation-oracle/` (the gate 1 envelope and one gate 3
manifest per oracle scope: `shared`, `schema-constants`, `middleware`) and
`mutation-controls/planted.json` (gate 4).
Re-freeze a manifest when gate 3 reports "re-freeze required", that is after a change to a
scope's tests or their imports, the Vitest configs, the prepare tool, or the Stryker, Vitest or
Node major version. Use the `refreeze` command recorded in the manifest, with a new seed.
Re-select the controls when gate 4 reports "re-select required". After a Vitest, Stryker or Node
upgrade, re-run all four gates. Refresh `expected.json` from the conformance `observed.json`
only for an intended change.
`.github/workflows/mutation-runner-validation.yml` runs gate 2 on PRs that touch the tooling,
its fixtures, `gitnexus/package.json`, the lockfile or `gitnexus/vitest.config.ts`. A manual
dispatch also runs gates 1, 3 and 4 and the benchmark, or re-freezes oracle manifests sharded
by scope and uploads them for a maintainer to commit. The workflow is advisory.
### Test Scenario Map
`gitnexus/tools/scenario-map/` builds a report on the eligible pure-JS tests: tests that repeat
a scenario, tests that kill no evaluated mutant, and per-file gaps. It runs Layers 0–3 of the
design (`docs/plans/2026-10-23-test-scenario-map-design.md`) on the `vitest5` runner. It only
reports. It never edits, deletes or folds a test.
From `gitnexus/`, run `npm run test:scenario-map -- <stage> --work <dir>`. The work directory
must lie outside the checkout. Every stage reads and writes JSON there, so a local `all` run and
CI produce the same report. `value-flow`, `all` and `controls` need a built `dist/`
(`npm run build`). The CLI header (`tools/scenario-map/scenario-map.ts`) documents every flag
and file.
| Stage | What it does | Writes (under `--work`) |
| ------------ | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `plan` | Eligibility, Layers 0–2, the run manifest, candidates, shard plan | `plan.json`, `scenario-run-manifest.json`, `layer0.json`, `layer2.json`, `coverage/` |
| `value-flow` | PDG index, static value-flow check, trace run, final shard plan | `value-flow.json` |
| `shards` | Prints the final shard count (the CI matrix size) | Nothing |
| `shard` | `--index <i> --of <n>`: Layer 3 runs, row deletion, isolation pass | `observations/`, `batches/`, `isolation/`, `row-deletion-shard-<i>.json`, `shard-<i>.json` |
| `merge` | Merges every result, computes verdicts, compares with `--baseline` | `scenario-map-report.json`, `scenario-map-report.md`, `issue-body.md` |
| `all` | `plan` (one shard), `value-flow`, `shard` 0 of 1 and `merge` | All of the above |
| `controls` | `all` over the controls fixture, then compares every expected result | All of the above |
Exit code 0 means complete results. Exit code 1 means the stage finished with incomplete results,
or a controls verdict differs. Exit code 2 is a usage or setup error.
The controls fixture is `gitnexus/test/fixtures/mutation-scenario-map/`: one pair per
`*.case.ts` file (16 pairs). It holds the pure-JS adversarial pairs of design §8, an
order-dependent pair, an object-key row and positive controls. `expected.json` was written from
the design before any pipeline run. Never regenerate it from observed output: a mismatch is a
finding. The fixture has its own Vitest config and tsconfig, and no repo project collects it.
The `scenario-map-controls` job in `.github/workflows/mutation-runner-validation.yml` runs
`controls` on the same PRs as gate 2.
**Supported matchers** (`SUPPORTED_MATCHERS` in `tools/scenario-map/facet/checks.ts`), after
`expect(…)` or `expect.soft(…)` and an optional `.not`, `.resolves` or `.rejects`:
`toBe`, `toEqual`, `toStrictEqual`, `toMatchObject`, `toBeTruthy`, `toBeFalsy`, `toBeNull`,
`toBeUndefined`, `toBeDefined`, `toBeNaN`, `toBeNullable`, `toBeTypeOf`, `toBeInstanceOf`,
`toBeGreaterThan`, `toBeGreaterThanOrEqual`, `toBeLessThan`, `toBeLessThanOrEqual`,
`toBeCloseTo`, `toBeOneOf`, `toContain`, `toContainEqual`, `toHaveLength`, `toHaveProperty`,
`toMatch`, `toThrow`, `toThrowError`, `toHaveBeenCalled`, `toHaveBeenCalledOnce`,
`toHaveBeenCalledTimes`, `toHaveBeenCalledWith`, `toHaveBeenCalledExactlyOnceWith`,
`toHaveBeenLastCalledWith`, `toHaveBeenNthCalledWith`, `toHaveBeenCalledBefore`,
`toHaveBeenCalledAfter`, `toBeCalled`, `toBeCalledTimes`, `toBeCalledWith`, `toHaveReturned`,
`toHaveReturnedTimes`, `toHaveReturnedWith`, `toHaveLastReturnedWith`, `toHaveNthReturnedWith`,
`toReturn`, `toReturnTimes`, `toReturnWith`, `toHaveResolved`, `toHaveResolvedTimes`,
`toHaveResolvedWith`, `toHaveLastResolvedWith`, `toHaveNthResolvedWith`, and the snapshot
matchers `toMatchSnapshot`, `toMatchInlineSnapshot`, `toMatchFileSnapshot`,
`toThrowErrorMatchingSnapshot`, `toThrowErrorMatchingInlineSnapshot`. `expect.assertions(n)` and
`expect.hasAssertions()` count as checks. Any other form is unsupported and blocks strong
evidence: `assert.*`, chai-style chains, custom matchers, `toSatisfy`, and an `expect` inside a
loop, a conditional, a catch clause, a nested function or a helper.
**Wrapped built-ins** of the trace run (`BASE_WRAP_LIST` in
`tools/scenario-map/value-flow/builtins.ts`), by group:
- `math`: every `Math` function except `random`.
- `string`: `String.prototype` `toUpperCase`, `toLowerCase`, `toLocaleUpperCase`,
`toLocaleLowerCase`, `normalize`, `localeCompare`, `trim`, `trimStart`, `trimEnd`, `padStart`,
`padEnd`, `slice`, `substring`, `substr`, `replace`, `replaceAll`, `split`, `at`, `charAt`,
`charCodeAt`, `codePointAt`, `startsWith`, `endsWith`, `includes`, `indexOf`, `lastIndexOf`,
`repeat`.
- `regexp`: `RegExp.prototype.test` and `exec`; `String.prototype.match`, `matchAll` and
`search`.
- `number`: `parseInt`, `parseFloat`, the `Number` constructor, and `Number.prototype.toFixed`,
`toPrecision`, `toExponential`, `toString`, `toLocaleString`.
- `json`: `JSON.parse`, `JSON.stringify`.
- `intl`: the `Intl` constructors `Collator`, `DateTimeFormat`, `NumberFormat`, `PluralRules`,
`RelativeTimeFormat`, `ListFormat`, `Segmenter`, `DisplayNames`; `Collator.prototype.compare`,
`DateTimeFormat.prototype.format` and `NumberFormat.prototype.format`; and the methods
`formatToParts`, `formatRange`, `formatRangeToParts`, `select`, `selectRange`, `segment` and
`of`.
- `date`: the `Date` constructor, `Date.UTC`, `Date.parse`, every `Date.prototype` getter and
setter, and its formatters.
- `sort`: `Array.prototype.sort` and `toSorted`, recorded with the receiver and the comparator.
The trace also wraps the opaque callees that the static value-flow check finds. Storage and
table reads stay unwrapped: array `push` and `unshift`, and every method of `Map`, `Set`,
`WeakMap` and `WeakSet`.
`.github/workflows/test-scenario-map.yml` runs the pipeline every Monday and on manual dispatch
(inputs `seed` and `shards`). Its jobs are `plan`, `value-flow`, `shards`, a `shard` matrix,
`merge` and `upsert-issue`. Every job is advisory (`continue-on-error`). Only a complete run from
the default branch uploads the `scenario-map-baseline` artifact and replaces the body of the
tracking issue (label `test-scenario-map`), unless the issue already records a newer run. A
partial run adds a comment and leaves the body alone. The workflow never closes the issue.
Mark a human keep with `// dupe-ok: <reason>` on either test of a pair, on the test call's line
or the line above. The report then lists the pair as kept, with its evidence.
## Regression testing
Re-run the full relevant suite when:

749
docs/plans/2026-10-23-1823-feat-stryker-vitest5-runner-phase0-plan.md Normal file
View file

@ -0,0 +1,749 @@
---
title: Stryker Vitest 5 Runner, Phase 0 Pure-JS Slice - Plan
type: feat
date: 2026-10-23
origin: docs/plans/2026-10-23-test-scenario-map-design.md
artifact_contract: ce-unified-plan/v1
artifact_readiness: implementation-ready
product_contract_source: ce-plan-bootstrap
execution: code
deepened: 2026-10-23
---
# Stryker Vitest 5 Runner, Phase 0 Pure-JS Slice - Plan
## Goal Capsule
- **Objective:** maintainers can run Stryker mutation testing on `gitnexus/` pure-JS unit tests under the repo's own Vitest 5, and trust each per-test kill inside the validated envelope because four validation gates passed. Gate 3's evidence is a frozen, stratified sample, so its claim is a bounded difference rate per scope, not a check of every mutant. This is the precondition for every strong verdict in the Test Scenario Map (see origin: `docs/plans/2026-10-23-test-scenario-map-design.md` §5.4a).
- **Means:** a repo-local Stryker test-runner plugin for Vitest 5 (KTD1), a prepared-copy step that makes type-asserted code mutable (KTD7, KTD8), and an independent Vitest 5 cross-check oracle with its own Stryker run (KTD11, KTD18).
- **Authority:** this plan's R-IDs and KTDs, then the origin design (§5.4, §5.4a, §9 Phase 0), then repo rules in `AGENTS.md`, `GUARDRAILS.md`, `TESTING.md`.
- **Stop conditions:** stop and report if the Vitest 5 runner cannot reach per-test kill attribution on the pure-JS slice, or if gate 3 leaves an unresolved difference. An exclusion must name the runner mechanism behind the difference and remove every pure-JS test that shows that trait, not only the sampled tests; after any fix or exclusion, gate 3 re-freezes with a new seed. Never weaken a gate to pass it.
- **Execution profile:** tooling work in `gitnexus/tools/mutation/`, fixtures under `gitnexus/test/fixtures/mutation-*`, unit tests, one CI workflow, and supporting edits to `gitnexus/package.json`, `gitnexus/package-lock.json`, `gitnexus/tsconfig.test.json`, ignore and lint configs, `TESTING.md` and the plan and design docs. No production code under `gitnexus/src` changes. No test under `gitnexus/test/unit` or `gitnexus/test/integration` is edited or deleted.
- **Tail ownership:** the runner, oracle manifests and CI workflow are owned by this PR (#3686). Phase 1 (Layers 1-3 reporting) consumes them.
---
## Product Contract
### Summary
Build a Stryker 10 test-runner plugin that works with Vitest 5.0.x. Validate it on the pure-JS test category through the four gates in the origin design: execution-path fidelity, a conformance suite, a Vitest 5 cross-check oracle, and positive controls with determinism. Include the prepared-copy step that makes `as`-asserted code mutable and resolves `gitnexus-shared` to source. Benchmark the oracle's cost.
### Problem Frame
Stryker's official `@stryker-mutator/vitest-runner` 10.0.0 runs 0 tests per mutant under Vitest 5, so no mutant is ever killed. Root cause, confirmed in source: Vitest 5 matches `testNamePattern` against `task.fullTestName`, joined with `' > '`, while the runner builds space-joined names. Every test inside a `describe` is filtered out. Upstream merged a fix (StrykerJS PR #6214, 2026-10-23) but has not released it; `latest` is still 10.0.0. Without a working, validated runner there is no mutation evidence, and the Test Scenario Map cannot issue any strong verdict.
### Key Decisions
- **Vitest 5 only, for the runner and its oracle** (session-settled: user-directed — chosen over a Vitest 4 copy harness and Vitest 4 spike results as oracle: the repo uses Vitest 5). Governs R1, R8, R13.
- **Build the runner in this effort and run mutation on the repo itself** (session-settled: user-directed — chosen over extending the scratch Vitest 4 harness project: the user asked for a Vitest 5 runner). Governs R1, R2, R3.
- **Scope is `gitnexus/` plus `gitnexus-shared/src` as a CLI test dependency** (session-settled: user-directed — chosen over including `gitnexus-web/`: web is deferred). Governs R5, R10.
- **Reports only; CI stays advisory** (session-settled: user-directed — chosen over automatic deletion or blocking CI: the user chose comment-only). Governs R12.
- **Phase 0 validates the pure-JS category only** (session-settled: user-approved — chosen over also validating native-backed and fixture-reading categories in Phase 0: Codex round 7 showed Phase 0 could not meet its gates at that scope). Governs R6, R9, R10.
- **Strictness over recall: anything unvalidated is out of scope or weak, never strong** (session-settled: user-approved — chosen over facet or path agreement alone as strong evidence: the top anti-goal is calling distinct scenarios the same). Governs R3, R8.
### Requirements
**Runner**
- R1. Stryker runs mutation on `gitnexus/` tests under Vitest 5.0.x with per-test coverage, `disableBail`, static mutants, hit limits and timeouts.
- R2. Every mutant run records an outcome class per test from the KTD17 table: assertion, runtime, hook, collection, timeout, infrastructure, inconclusive or invalid.
- R3. Test identities are stable between the dry run and every mutant run, including `it.each`/`test.for` rows. A run that executes zero tests, or collects a different test set, is invalid.
**Prepared sources**
- R4. Code inside TypeScript `as` assertions, including `as const` tables, is mutated, and every mutant's identity sits on the original source span.
- R5. `gitnexus-shared` resolves to source in the mutation configuration, and its mutants map back to `gitnexus-shared/src` paths.
**Validation gates (origin §5.4a)**
- R6. Gate 1: a normal Vitest 5 run and the runner's unmutated run agree on the pure-JS category, and the validated envelope is recorded.
- R7. Gate 2: a committed conformance suite pins runner behaviour for the cases in U4.
- R8. Gate 3: a frozen Vitest 5 cross-check oracle, independent of the runner, agrees with the runner on every supported observation.
- R9. Gate 4: planted positive controls behave as expected, and two runs with different scheduling give identical kill sets.
- R10. A static classifier decides which test files belong to the pure-JS category.
**Operations**
- R11. The oracle's fresh-process cost is measured and recorded.
- R12. A path-gated CI workflow runs the conformance suite when the runner, Vitest, Stryker or the lockfile change. The oracle runs on demand.
- R13. The runner refuses to run on a Vitest version outside the supported minor.
- R14. Residual improvements over upstream StrykerJS are written up in `gitnexus/tools/mutation/UPSTREAM.md`, not opened upstream, in this run.
### Success Criteria
- The pure-JS category passes gates 1-4 with zero unresolved differences.
- On the three KTD11 scopes, the runner's per-test outcomes match the oracle on every sampled mutant, and every required stratum (KTD11) is represented in the frozen sample.
- The oracle benchmark, run by the `workflow_dispatch` job on `ubuntu-latest` (4 vCPU), reports per-launch cost and a projected full-sample wall clock that fits one 6-hour job.
### Scope Boundaries
- Out: Layers 1-3 reporting, the value-flow check, the kill-validity protocol and the scheduled job (origin §9 Phase 1).
- Out: native-backed, fixture-reading, LadybugDB and worker-backed categories (origin §9 Phases 2-3).
- Out: `gitnexus-web/` and Python tests.
- Out: any edit to existing tests or production code.
#### Deferred to Follow-Up Work
- TypeScript `enum` members are never mutated: the instrumenter skips `TSEnumDeclaration`, and enums emit runtime code, so they cannot be blanked. Record as a known blind spot; revisit in Phase 1.
- Opening the upstream StrykerJS PR or issues (R14 prepares notes only).
- A Dependabot ignore rule for the Vitest minor. The runtime guard (KTD2) and the lockfile CI trigger cover Phase 0.
---
## Planning Contract
### Key Technical Decisions
- KTD1. **Own runner plugin, written in TypeScript, not a vendored copy of the upstream runner.** The upstream fix (#6214) covers only the separator. This plan also needs failure classes, a shared run manifest, task-id selection, no related mode and the repo's own pool, which change most of the upstream file. A fresh implementation avoids vendoring Apache-2.0 code into a PolyForm-licensed repo. Drop it once an upstream release passes the same four gates (origin §5.4a Maintenance). No upstream release can pass gates 2 and 3 until upstream exposes per-test failure classes, so until then the plugin is a standing commitment. (session-settled: user-directed — chosen over extending the scratch Vitest 4 harness project: the user asked for a Vitest 5 runner.) Conflict call-out: origin §9 Phase 0 says "open the upstream PR"; upstream already merged the root-cause fix, so R14 narrows this to residual-delta notes.
- KTD2. **Version guard.** At init the runner reads the Vitest version resolved from the project root and refuses anything outside `5.0.x`. A second Vitest copy resolved from the plugin's own location is an error, because `inject` then returns undefined. Results produced on a Vitest or Stryker version not recorded in the gate 1 envelope are unvalidated until gates 2 and 3 re-pass on that version. (session-settled: user-directed — chosen over a Vitest 4 harness: the repo uses Vitest 5.)
- KTD3. **Test identity.** The Stryker test id is `<relative file>#<fullTestName>`. Parse ids on the first `#`, because titles contain `#` (for example `(#2415)`). Tests that share a full name (duplicate titles, or `it.each` rows whose titles truncate at 40 characters) get a `:<line>:<column>` suffix. Rows of one `it.each`/`test.for` call share the call-site location, so a remaining collision adds the row's ordinal within that call (`:<line>:<column>:<row>`). Rows that needed the ordinal are not oracle-validated (origin §5.4a gate 3). Each id maps to its Vitest task id in the run manifest (KTD16). Per-test coverage keys use exactly these ids.
- KTD4. **Mutant runs select tests by Vitest task id.** For each file, build one test specification carrying the task ids of the filtered tests, and run the specifications directly. Never use related mode or name patterns. Task ids are positional and stay stable for runtime mutants, because any hit during collection makes a mutant static. The setup file marks the end of collection in its file-level `beforeAll`. A hit with no current test after that point is a hook hit and is recorded as per-test coverage for every test of the suite whose hook ran. Only load-time and collection-time hits stay static. Static mutants run exactly the manifest's full test set, never the config's whole include. With forks and default isolation, each file gets a fresh module graph and the setup file runs before the test file is imported, so `reloadEnvironment: true` needs no extra reload logic. The Stryker `testFiles` option is forbidden: with it, static mutants receive file paths as their filter and runtime activation, and become false survivors. The runner rejects any filter entry that is not a manifest test id.
- KTD5. **Isolation.** Keep the configured pool (`forks`) instead of forcing threads. Each run starts fresh workers. Each run, including every oracle process (U12, U13), uses its own Vitest `cacheDir` inside the sandbox, never the shared `node_modules/.vite`. The runner deletes `__STRYKER_ACTIVE_MUTANT__` from its own environment and its workers' environment. It never serialises the Vitest config.
- KTD6. **Failure classes travel in sidecar files.** Stryker's result type has no field for them. The runner takes an absolute sidecar directory as a plugin option, declared in an exported validation schema. Each mutant run writes one file per local mutant id, attempt and worker (`STRYKER_MUTATOR_WORKER`), written to a temporary name and then renamed. The dry run writes a record too. A missing record means timeout or unknown, because Stryker kills a timed-out process and retries crashed runs. Records use the classes in KTD17.
- KTD7. **Prepared copy blanks `as` assertions with equal-length spaces.** The instrumenter skips every `TSAsExpression` subtree (`tsTypeAnnotationNodeTypes`), not only `as const`; there are 317 `as const` nodes in `gitnexus/src` alone. Replace the ` as <type>` text with spaces of the same length, keeping newlines, so every line and column is unchanged and the span map is an identity on columns. The check is AST equality: the blanked file's AST must equal the original AST with every `TSAsExpression` unwrapped. "Still parses" is not enough, because ASI before `(`, `[` or a template literal, precedence (`a + b as T * c`) and a parenthesised `(x) as T` can parse to a different tree. After instrumentation, no `TSAsExpression` may remain in mutate-scope files. `satisfies` and the non-null `!` are not on the skip list and stay. Set Stryker `disableTypeChecks: false`, because its default prepends `// @ts-nocheck` and shifts every test line by one. This extends origin §5.4, which named only table-level `as const`; the mechanism and the span-map check are the same.
- KTD8. **`gitnexus-shared/src` is relocated inside the prepared `gitnexus/` tree.** Stryker cannot mutate files outside its working directory. The relocation path never contains `node_modules`. The mutation config derives its regex aliases from the `gitnexus-shared` package `exports` (today `.` and `./test-helpers`, which cover all 811 imports) and computes them relative to the config file, so they resolve inside whichever sandbox runs. `gitnexus-shared/src` uses only package-relative imports. A path map translates relocated paths back to `gitnexus-shared/src/...`.
- KTD9. **Mutation Vitest config.** A committed `vitest.mutation.config.ts` reuses the root test settings of `gitnexus/vitest.config.ts`, replaces its projects with one `mutation` project whose include list comes from the run's scope, adds the KTD8 aliases, and sets `dangerouslyIgnoreUnhandledErrors: false`. Every difference from the base config is on gate 1's allowed list (U6).
- KTD10. **Tooling lives in `gitnexus/tools/mutation/`.** `gitnexus/scripts/` ships in the npm package (`files` includes `scripts`), so it is the wrong home. Entry points run with `node --import tsx`, exposed as `test:mutation:*` npm scripts. Stryker's test-runner child processes receive only `testRunnerNodeArgs`, so the generated Stryker config sets `testRunnerNodeArgs: ["--import", "tsx"]`. Never use `NODE_OPTIONS`, which leaks into Vitest fork workers. Tests for pure functions live in `gitnexus/test/unit/mutation/`.
- KTD11. **Oracle scopes.** Three pure-JS scopes: `src/core/ingestion/type-extractors/shared.ts` (143 tests, static mutants, no hooks), `gitnexus-shared/src/lbug/schema-constants.ts` via `test/unit/schema.test.ts` (`as const` tables, the alias, one `it.each`), and one hook-bearing pure-JS scope chosen with the classifier (U5). Freezing counts every stratum across the three scopes combined. Within one scope, a stratum is required only when that scope's tests contain the construct (`beforeAll` hooks, `it.each`/`test.for`). An empty required stratum fails freezing unless the affected tests are marked not oracle-validated. This augments origin §5.4a gate 3, which named only `shared.ts`, because that scope exercises neither the pre-step, the alias, `beforeAll` nor `it.each`.
- KTD12. **Frozen manifests use content hashes.** Each manifest records content hashes of the full static import closure of the scope's test files (from the U5 walk), `gitnexus/vitest.config.ts`, the mutation config and the prepare tool, plus the operator set, the Stryker and Vitest versions, the Node major version, test ids and global mutant ids. A hash or version mismatch fails with "re-freeze required", and a documented re-freeze command exists. Content hashes survive squash-merges and shallow CI checkouts, which git SHAs do not.
- KTD13. **Normalisations allowed by gate 3.** Exactly four: the timeout rows of KTD17 are one class, NoCoverage and survived are one class (the raw Stryker status is kept for the report), title escaping, and path separators. Everything else must agree (origin §5.4a gate 3 acceptance).
- KTD14. **Conformance and oracle are tools plus a path-gated workflow, not tests in the main suite.** They run Stryker and many Vitest processes, which would slow every CI shard. The conformance fixture lives under `gitnexus/test/fixtures/` with `*.case.ts` files, so neither the default project, the test-completeness gate nor the PR-time dupe scanner collects it.
- KTD15. **The oracle installs its own probes and classifier.** An oracle setup file confirms the installed mutant id before any test-file import, records hits separately from activation, and sorts hits into the same three buckets as the runner: load and collection, hook, and per-test. An oracle reporter records the executed test identity and outcome class with the oracle's own failure classifier. Each launch has a wall-clock limit and a process-group kill. Code under `oracle/` never imports code under `runner/`; a unit test enforces it.
- KTD16. **Run directory and run manifest.** Each Stryker run gets an absolute run directory. Only one pool process runs the dry run; the others, and replacements started after crashes, timeouts or `maxTestRunnerReuse`, only call `init` and `mutantRun`. So the dry-run process writes a run manifest atomically: test files, Stryker id to Vitest task id and location, and the expected test set per file. Every runner process loads it before its first mutant run and refuses to run without it.
- KTD17. **One outcome-class table for runner and oracle.** Stryker's status is Killed for every killed row below, so Stryker's own report stays correct. The kill sets used by gates 3 and 4, the success criteria and Phase 1 count only `killed-by-test` rows with class assertion, as origin §2 requires.
| Observation | Class | Runner reports to Stryker |
| ---------------------------------------------------------------------- | --------------------------------------- | --------------------------------------- |
| Test fails with `AssertionError` | killed-by-test, assertion | Killed |
| Test or hook fails with Vitest's own `testTimeout`/`hookTimeout` error | timeout | Timeout |
| Test fails with another error | killed-by-test, runtime | Killed |
| `beforeEach`/`afterEach` fails for a test | killed-by-test, hook | Killed |
| `beforeAll` fails; suite tests skipped with a suite error | killed-outside-test, hook | Killed, attributed to the suite's tests |
| `afterAll` fails; tests stay passing | killed-outside-test, hook | Killed, no test attribution |
| Test file fails to import or collect | killed-outside-test, collection | Killed, no test attribution |
| Unhandled error during the run | infrastructure | Error (inconclusive) |
| A selected test has `retry` above zero | inconclusive | Error, sidecar records the reason |
| Hit limit, Stryker timer or oracle wall clock | timeout | Timeout |
| Zero tests, or a test set different from the manifest | invalid | Error, sidecar records the reason |
| No selected test hits the mutant | survived (NoCoverage is the same class) | Survived or NoCoverage |
The command runner sees only an exit code, so U13 wraps its command to write Vitest JSON per active mutant. The oracle then classifies its own runs into the same table.
- KTD18. **The oracle runs its own Stryker run.** One Stryker run cannot use both `vitest5` and `command`. The oracle's run uses `testRunner: command` with `cleanTempDir: false`. Its single-test launches run in that kept, instrumented sandbox. Local mutant ids of both runs map to global ids through U2. The runner and the oracle share Stryker core (planning and instrumentation), the prepare step and the mutation config. They share no runner code. The oracle derives its own test inventory with a Vitest list run in the kept sandbox and compares it with the runner's run manifest. The mutant inventories of runs A and B must be identical sets of global ids. Any inventory difference blocks gate 3 before outcomes are compared.
- KTD19. **Independence is enforced, not assumed.** In both runs, every production module must resolve inside the active sandbox; U6 checks it. Gate 4 plants controls in relocated `gitnexus-shared` code and in one `as const` table, so a mis-pointed alias shows up as a failed control instead of two runs agreeing that every shared mutant survived. Gate 1 also records the modules each pure-JS test loads at runtime; a native binding, worker or `child_process` load fails gate 1, so a classifier mistake cannot shrink the category silently.
### High-Level Technical Design
Components and data flow. Two Stryker runs share the prepared copy, Stryker core and the mutation config; they share no runner code (KTD18).
```mermaid
flowchart TB
SRC[gitnexus/ and gitnexus-shared/src] --> PREP[Prepare: copy, relocate shared src, blank as assertions]
PREP --> PMAP[Path map]
PREP --> PCOPY[Prepared copy]
PCOPY --> RUNA[Stryker run A: vitest5 runner plugin]
RUNA --> RESA[mutation.json A + sidecar files + run manifest]
PCOPY --> RUNB[Stryker run B: command runner + Vitest JSON wrapper, cleanTempDir false]
RUNB --> RESB[mutation.json B + per-mutant Vitest JSON]
RUNB --> KEPT[Kept instrumented sandbox]
KEPT --> SINGLE[Single-test launches: oracle setup file + oracle reporter]
SINGLE --> OKILL[Oracle per-test outcomes]
RESA --> IDS[Global mutant ids]
RESB --> IDS
PMAP --> IDS
FROZEN[Frozen oracle manifest] --> CMP[Gate 3 comparison and acceptance]
IDS --> CMP
OKILL --> CMP
```
Validation gates. Gate 2 validates the runner itself; gates 1, 3 and 4 validate one category. A category is out of scope until all pass.
```mermaid
flowchart TB
G2{Gate 2: runner conformance suite} -->|any pinned case fails| STOP[Runner not trusted]
G2 -->|pass| C[Classifier: pure-JS test files]
C --> G1{Gate 1 fidelity: identities, outcomes, config diffs, resolved paths}
G1 -->|difference outside allowed list| OUT[Category stays out of scope]
G1 -->|agree| G3{Gate 3 oracle: every supported observation agrees}
G3 -->|unresolved difference| STOP2[Stop and report]
G3 -->|pass| G4{Gate 4 positive controls + determinism}
G4 -->|fail| OUT
G4 -->|pass| IN[Category validated; envelope recorded]
```
Mutant run inside the runner, directional only:
```text
mutantRun(activeMutant, testFilter?, hitLimit, mutantActivation):
manifest = load run manifest from the run directory; refuse if absent
reject filter entries that are not manifest test ids
specs = per-file specs with the Vitest task ids of testFilter,
or the manifest's full test set for a static mutant
provide(mode=mutant, activeMutant, mutantActivation, hitLimit) # fresh forks
run specs; collect tasks for these modules only
classify every observation with the KTD17 table
write the sidecar file for (mutant, attempt, worker), then rename
return Killed, Survived, Timeout or Error per KTD17
```
### Output Structure
```text
gitnexus/tools/mutation/
runner/ Stryker plugin: plugin entry, setup file, ids, run manifest, classification, sidecar
prepare/ prepared copy, as-blanking, shared relocation, path map, mutant identity, span-map check
driver/ scope file to Stryker config, Stryker run, normalised results
classify/ pure-JS category classifier
fidelity/ gate 1 comparison and envelope record
conformance/ gate 2 driver
oracle/ command-runner wrapper, single-test launcher, freeze, compare; oracle setup file and reporter
controls/ gate 4 planted controls and determinism run
bench/ oracle cost benchmark
UPSTREAM.md probe findings and residual deltas against upstream
vitest.mutation.config.ts
gitnexus/test/fixtures/mutation-conformance/ fixture project (*.case.ts) and expected.json
gitnexus/test/fixtures/mutation-oracle/ frozen oracle manifests and the gate 1 envelope record
gitnexus/test/fixtures/mutation-controls/ planted.json (gate 4 controls)
gitnexus/test/unit/mutation/ unit tests for pure functions
.github/workflows/mutation-runner-validation.yml
```
### Assumptions
- Vitest 5.0.x keeps `fullTestName`, per-specification test ids, task `meta` propagation and `afterAll((ctx, suite))` as read in 5.0.5. These are marked experimental; KTD2's guard limits exposure, and U11 confirms them before anything is built.
- `test/unit/schema.test.ts` is pure JS by the classifier's rules. The spike ran it in a pure-JS scratch project; U5's acceptance test confirms it.
- A hook-bearing pure-JS scope exists among unit tests. If U5 finds none, the `beforeAll` stratum is covered only by the conformance fixture, and real tests with `beforeAll` stay not oracle-validated.
- The oracle's full sample fits one 6-hour `workflow_dispatch` job on `ubuntu-latest`. It is about 14,300 single-test launches for `shared.ts` alone (100 mutants times 143 tests, origin §5.4a), plus 100 sampled mutants times the test count of each other KTD11 scope, plus strata top-ups, plus one command-runner pass per mutant. If U12's projection exceeds the job limit, sampling changes go back to the maintainer as a design change.
### Risks and Dependencies
| Risk | Mitigation |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Vitest marks `state.getFiles`, task `meta` and specification APIs experimental; a minor release can change them. | KTD2 version guard; the conformance workflow runs on every lockfile change (U9). |
| Dependabot floats `vitest ^5.0.0`; a lockfile-only bump to 5.1 disables the runner. | The guard fails loudly with the supported range. Re-validate through the four gates, then widen the range. |
| Stryker 10.0.1+ ships the upstream fix and changes replacement printing, so global mutant ids shift. | Exact pins (U3); manifests record the Stryker version and fail with "re-freeze required" (KTD12). |
| Blanking `as` assertions exposes code the instrumenter never mutated before; some mutants may produce invalid code, and many tables become static mutants. | The prepare step checks AST equality (KTD7), and U12 counts static mutants. Compile-error mutants are recorded by Stryker as `CompileError` and never count as kills (origin §2). |
| The oracle's full sample may be too slow for CI. | U12 benchmarks before U7's full freeze. The oracle stays `workflow_dispatch`-only until a budget is set (origin §5.4a). |
| Fork pool plus parallel Stryker workers oversubscribes 4 vCPU. | `maxWorkers` per Vitest run is on the allowed-difference list (U6). Stryker concurrency is set per run and recorded in the envelope. |
| PR #3686 grows from an advisory scanner into tooling. | All Phase 0 code sits in `gitnexus/tools/mutation/`, `gitnexus/test/unit/mutation/`, fixtures and one workflow. Nothing in `gitnexus/src` changes. |
### System-Wide Impact
- **npm package:** `gitnexus/tools/` is not in `files`, so nothing ships. `@stryker-mutator/core` and `@stryker-mutator/api` are devDependencies only.
- **CI:** one new advisory workflow. It is outside `ci-status.needs` and does not run on PRs that touch none of its paths.
- **Main test suite:** the conformance fixtures use `*.case.ts` under `test/fixtures/`, so the `default` project, the test-completeness gate and the PR-time dupe scanner ignore them (KTD14). New unit tests in `test/unit/mutation/` join the normal shards.
- **Repo hygiene:** `.gitignore`, `.prettierignore` and ESLint ignores gain the prepared-copy directory, `.stryker-tmp/` and mutation reports (U1).
### Sources and Research
- Root cause and runner internals: `@stryker-mutator/vitest-runner` 10.0.0 `dist/src/vitest-test-runner.js` (`run` builds space-joined names); Vitest 5.0.5 `dist/task-utils.js` (`fullTestName` match), `chunks/cli-api.*.js` (`parseFilter`, JSON reporter). Vitest 5 migration guide: "`testNamePattern` Matches the `>`-Joined Full Name".
- Upstream: StrykerJS PR #6214 (separator fix, merged, unreleased), #6246 (end anchoring), #6236 (forced threads pool), #6256 (NoCoverage reported as Survived).
- Stryker internals: `core/dist/src/mutants/mutant-test-planner.js` (static vs runtime activation, `reloadEnvironment`), `core/dist/src/test-runner/command-test-runner.js` (exit-code status, `__STRYKER_ACTIVE_MUTANT__`), `instrumenter/dist/src/util/syntax-helpers.js` (`tsTypeAnnotationNodeTypes` skip list), `instrumenter/dist/src/disable-type-checks.js` (`// @ts-nocheck` prepend), `core/dist/src/utils/object-utils.js` (`toSchemaLocation`, 1-based, end exclusive).
- Repo: `gitnexus/vitest.config.ts` (projects, `pool: 'forks'`, `includeTaskLocation`, `dangerouslyIgnoreUnhandledErrors`), `gitnexus/scripts/test-completeness.ts` (test key and every-file gate), `gitnexus/test/unit/test-completeness.test.ts` (subprocess Vitest pattern), `.github/workflows/impact-pdg-mutation-report.yml` (scheduled tool-workflow pattern), `gitnexus-shared/package.json` (dist exports).
- Spike prior art (scratch, not committed): `gitnexus/vitest.stryker-bake.config.ts`, `/tmp/test-audit/v4proj/vitest.confirm.config.ts` (regex aliases), `/tmp/test-audit/spike5/row-delete.mjs`.
---
## Implementation Units
| U-ID | Title | Key files | Depends on |
| ---- | ----------------------------------------- | ---------------------------------------------------------------- | -------------- |
| U11 | Go/no-go probe | `gitnexus/tools/mutation/UPSTREAM.md` | none |
| U1 | Prepared copy, path map, mutation config | `gitnexus/tools/mutation/prepare/`, `vitest.mutation.config.ts` | U11 |
| U3 | Vitest 5 runner plugin | `gitnexus/tools/mutation/runner/` | U11, U1 |
| U2 | Global mutant identity, span-map check | `gitnexus/tools/mutation/prepare/mutant-identity.ts` | U1, U3 |
| U10 | Scope driver | `gitnexus/tools/mutation/driver/` | U1, U2, U3 |
| U4 | Conformance suite (gate 2) | `gitnexus/test/fixtures/mutation-conformance/`, `conformance/` | U10 |
| U5 | Pure-JS classifier | `gitnexus/tools/mutation/classify/` | none |
| U6 | Gate 1 fidelity and envelope | `gitnexus/tools/mutation/fidelity/` | U5, U10 |
| U13 | Oracle command-runner run and wrapper | `gitnexus/tools/mutation/oracle/command-*` | U10 |
| U12 | Oracle single-test launcher and benchmark | `gitnexus/tools/mutation/oracle/launch-*`, `bench/` | U13 |
| U7 | Oracle freeze and compare (gate 3) | `gitnexus/tools/mutation/oracle/freeze.ts`, `compare.ts` | U5, U12, U13 |
| U8 | Gate 4 controls and determinism | `gitnexus/tools/mutation/controls/` | U7 |
| U9 | CI workflow and docs | `.github/workflows/mutation-runner-validation.yml`, `TESTING.md` | U4, U6, U7, U8 |
### U11. Go/no-go probe
**Goal:** confirm the root cause and the Vitest 5 and Stryker behaviours the plan depends on, before anything is built. This is the plan's first stop condition.
**Requirements:** R1, R14; KTD1, KTD4, KTD10.
**Dependencies:** none.
**Files:**
- `gitnexus/tools/mutation/UPSTREAM.md` (create: probe findings, one line each, with the Vitest and Stryker versions)
**Approach:** probe in a scratch location outside the repo, on the `shared.ts` scope (no `as const`, so no prepare step is needed). Record each answer:
1. In Vitest 5.0.5, `-t 'Describe test'` runs 0 tests and `-t 'Describe > test'` runs them.
2. A test specification with per-spec test ids runs exactly those tests.
3. An unfiltered run with a mutant active at module load executes tests and can fail them.
4. A TypeScript plugin loads in Stryker's child processes with `testRunnerNodeArgs: ["--import", "tsx"]`.
5. Task `meta` written in a setup file reaches the main process, and `afterAll((ctx, suite))` receives the suite.
6. During a suite's `beforeAll`, no current test is set, and a file-level setup `beforeAll` runs before it.
**Test expectation:** none -- throwaway probe; its findings are recorded in `UPSTREAM.md` and no probe code is committed.
**Verification:** all six answers are recorded. If any is negative, stop and report before U1.
### U1. Prepared copy, path map and mutation config
**Goal:** produce a prepared copy of `gitnexus/` in which `as` assertions are blank and `gitnexus-shared/src` is relocated, plus the mutation Vitest config that runs in it.
**Requirements:** R4, R5; KTD7, KTD8, KTD9, KTD10.
**Dependencies:** U11.
**Files:**
- `gitnexus/tools/mutation/prepare/prepare.ts` (create)
- `gitnexus/tools/mutation/prepare/blank-assertions.ts` (create)
- `gitnexus/tools/mutation/prepare/path-map.ts` (create)
- `gitnexus/tools/mutation/vitest.mutation.config.ts` (create)
- `gitnexus/package.json` (modify: `test:mutation:prepare` script)
- `.gitignore`, `.prettierignore`, `eslint.config.*` (modify: ignore the prepared-copy directory, `.stryker-tmp/`, mutation run directories and reports)
- `gitnexus/tsconfig.test.json` (modify: include `tools/**/*` so entry points no test imports are typechecked)
- `gitnexus/test/unit/mutation/blank-assertions.test.ts` (create)
- `gitnexus/test/unit/mutation/path-map.test.ts` (create)
**Approach:**
1. Copy the `gitnexus/` sources into a prepared directory outside the checkout, symlinking `node_modules`. Leave out `vendor/` (native prebuilds, about 224 MB), `dist/` and other build output; the pure-JS slice does not load them.
2. Copy `gitnexus-shared/src` into the prepared tree at a fixed relative location without `node_modules` in its path, and record the path map (KTD8).
3. For each `.ts` file in the mutate scope, blank every `TSAsExpression` per KTD7 and run the AST-equality check.
4. The mutation config derives root test settings from `vitest.config.ts`, derives aliases from the `gitnexus-shared` package `exports`, and takes its include list from a scope file written by the caller.
**Patterns to follow:** `gitnexus/tools/test-dupes/test-dupes.ts` (exported pure functions plus a main guard); `/tmp/test-audit/v4proj/vitest.confirm.config.ts` (regex aliases).
**Test scenarios:**
- `const T = ['a', 'b'] as const;` blanks to the same length with `]` and `;` at unchanged columns.
- `] as const satisfies Record<string, number>` keeps `satisfies Record<string, number>` and blanks only `as const`.
- Nested `x as unknown as T` blanks both assertions.
- A multi-line type in `as { a: string;\n b: number }` keeps its newline, and later lines keep their line numbers.
- `(a as T).b` and `() => x as T` pass the AST-equality check.
- A synthetic `a + b as T * c` fails the AST-equality check, and the file is reported instead of written.
- A synthetic case where blanking would leave a line starting with `(` after an expression without a semicolon fails the AST-equality check.
- A file with no assertions is byte-identical after the step.
- The path map translates a relocated path back to `gitnexus-shared/src/lbug/schema-constants.ts` and leaves `gitnexus/src/...` paths unchanged.
- A path outside both roots is rejected, not passed through.
- Aliases are generated for every key in the `gitnexus-shared` `exports` map and nothing else.
**Verification:** the prepared copy runs `test/unit/schema.test.ts` and the two `shared.ts` test files green under `vitest.mutation.config.ts`, and every module loaded from `gitnexus-shared` resolves inside the relocated copy.
### U3. Vitest 5 runner plugin
**Goal:** a Stryker `vitest5` test runner that gives correct per-test coverage, kills and failure classes under Vitest 5.0.x.
**Requirements:** R1, R2, R3, R13; KTD1-KTD6, KTD16, KTD17.
**Dependencies:** U11, U1.
**Files:**
- `gitnexus/tools/mutation/runner/plugin.ts` (create: plugin declaration, options schema, runner class)
- `gitnexus/tools/mutation/runner/setup.ts` (create: worker setup file for `__stryker__` globals, coverage, hit counts, static vs runtime activation)
- `gitnexus/tools/mutation/runner/test-ids.ts` (create)
- `gitnexus/tools/mutation/runner/run-manifest.ts` (create)
- `gitnexus/tools/mutation/runner/classify-outcome.ts` (create)
- `gitnexus/tools/mutation/runner/sidecar.ts` (create)
- `gitnexus/tools/mutation/runner/version-guard.ts` (create)
- `gitnexus/package.json`, `gitnexus/package-lock.json` (modify: exact pins of `@stryker-mutator/core` and `@stryker-mutator/api` 10.0.0)
- `gitnexus/test/unit/mutation/test-ids.test.ts` (create)
- `gitnexus/test/unit/mutation/run-manifest.test.ts` (create)
- `gitnexus/test/unit/mutation/classify-outcome.test.ts` (create)
- `gitnexus/test/unit/mutation/sidecar.test.ts` (create)
- `gitnexus/test/unit/mutation/version-guard.test.ts` (create)
**Approach:**
1. Dry run: run the scope's files, record per-test coverage keyed by KTD3 ids in `afterAll` metadata, write the run manifest (KTD16) and a dry-run sidecar record, and report one result per test.
2. Mutant run: follow the KTD4 selection, KTD5 isolation, KTD6 sidecar and KTD17 classes.
3. Report Timeout when the summed hit count reaches the hit limit, including when production code swallows the hit-limit error.
**Patterns to follow:** the upstream runner's structure for provide/inject and setup-file injection, re-implemented per KTD1; `gitnexus/scripts/test-completeness.ts` for test keys.
**Test scenarios:**
- An id for `describe A > it b (#2415)` round-trips, and parsing splits on the first `#` only.
- A title containing `' > '` produces an id that still round-trips.
- Two tests with the same full name get location-suffixed ids. Two rows with identical name and location are flagged not oracle-validated.
- A run manifest written by one process loads in another and maps every Stryker id to one Vitest task id.
- A mutant run without a run manifest refuses to run with a clear error.
- A filter containing a file path instead of a test id is rejected.
- Each KTD17 row classifies as stated, from synthetic task results: `AssertionError`, `TypeError`, a failing `beforeEach`, a `beforeAll` suite error with skipped tests, a failing `afterAll` with passing tests, a file with zero tasks and an import error, an unhandled error, a hit-limit stop.
- Sidecar files from two workers for the same mutant and attempt do not overwrite each other, and a half-written file is never visible under the final name.
- A hit recorded after the file-level setup `beforeAll` with no current test is attributed to every test of the suite whose hook ran. A hit during collection stays static.
- Rows of one `it.each` call with identical truncated names get distinct ids by row ordinal and are flagged not oracle-validated.
- The guard accepts 5.0.5, rejects 5.1.0 and 4.1.12, and rejects a second Vitest copy resolved from the plugin location.
**Verification:** the unit tests pass, and the plugin loads in a Stryker child process. The end-to-end kill check runs in U10, which owns the generated Stryker config.
### U2. Global mutant identity and span-map check
**Goal:** give every mutant a global identity on original source spans, and prove the span map is correct for every mutant.
**Requirements:** R4, R5; KTD7, KTD12.
**Dependencies:** U1, U3 (a real `mutation.json` for verification).
**Files:**
- `gitnexus/tools/mutation/prepare/mutant-identity.ts` (create)
- `gitnexus/tools/mutation/prepare/span-map-check.ts` (create)
- `gitnexus/test/unit/mutation/mutant-identity.test.ts` (create)
- `gitnexus/test/unit/mutation/span-map-check.test.ts` (create)
**Approach:**
1. Read Stryker's `mutation.json` (`files[path].mutants[].location`, 1-based, end exclusive).
2. Identity is original path (via the path map), content hash of the original file, span, mutator name and replacement (origin §2 mutant identity, with a content hash per KTD12). It also returns the local id map for one run.
3. The check applies each replacement at the same span in the original file and compares with the prepared file's mutated text, masking blanked ranges.
**Test scenarios:**
- Two mutants on the same span with different replacements get different identities.
- The same mutant read from two runs, with different local ids, gets the same global identity.
- A mutant inside a blanked-assertion line maps to the same line and column in the original.
- A replacement whose original span text differs from the prepared span text, outside blanked ranges, fails the check with the mutant's identity in the message.
- 1-based end-exclusive locations convert correctly at line start and line end.
**Verification:** the check passes for every mutant of the three oracle scopes (KTD11).
### U10. Scope driver
**Goal:** one entry point that turns a scope file into a prepared copy, a generated Stryker config, a Stryker run and normalised results, for U4, U6, U7, U8 and U13.
**Requirements:** R1, R4, R5; KTD4, KTD6, KTD10, KTD16, KTD18.
**Dependencies:** U1, U2, U3.
**Files:**
- `gitnexus/tools/mutation/driver/run-scope.ts` (create)
- `gitnexus/tools/mutation/driver/stryker-config.ts` (create)
- `gitnexus/package.json` (modify: `test:mutation:run` script)
- `gitnexus/test/unit/mutation/stryker-config.test.ts` (create)
**Approach:**
1. A scope file names test files and mutate globs, which may include relocated `gitnexus-shared` paths.
2. The generated Stryker config sets: `appendPlugins`, the test runner (`vitest5` or `command`), `coverageAnalysis: "perTest"`, `disableBail: true`, `disableTypeChecks: false`, a non-existent `tsconfigFile`, `testRunnerNodeArgs`, the run directory, the sidecar directory, `cleanTempDir` and `ignorePatterns` that keep `vendor/` and build output out of the sandbox. It never sets `testFiles`.
3. Results are normalised through U2: global mutant ids, statuses, killers, and sidecar classes joined per mutant.
**Test scenarios:**
- A scope with a relocated shared path produces mutate globs inside the prepared tree.
- The `command` variant sets `cleanTempDir: false`. The `vitest5` variant sets the run and sidecar directories as absolute paths.
- `testFiles` never appears in a generated config.
- Normalisation joins a sidecar record to its mutant through the local id map, and flags a `killedBy` entry with no sidecar record.
**Verification:** `test:mutation:run` on the `shared.ts` scope produces non-empty per-test kill sets, no mutant run executes zero tests, and the normalised results carry global ids and sidecar classes for every mutant.
### U4. Conformance fixture project and conformance check (gate 2)
**Goal:** pin runner behaviour on a small fixture project, so regressions in the runner, Vitest or Stryker show up as concrete failures.
**Requirements:** R7, R2, R3; KTD6, KTD14, KTD16, KTD17.
**Dependencies:** U10.
**Files:**
- `gitnexus/test/fixtures/mutation-conformance/` (create: tiny production modules, `*.case.ts` test files, own Vitest config)
- `gitnexus/tools/mutation/conformance/run-conformance.ts` (create)
- `gitnexus/test/fixtures/mutation-conformance/expected.json` (create: pinned per-test coverage, kills and outcome classes; under `test/fixtures/` so tool-written JSON is outside `format:check`)
- `gitnexus/package.json` (modify: `test:mutation:conformance` script)
- `gitnexus/test/unit/mutation/conformance-compare.test.ts` (create: comparison logic only)
**Approach:** the driver runs the fixture project through U10, reads normalised results and compares them with `expected.json`. Every case below is a planted mutant or test with a pinned expectation. The suite runs twice: at Stryker concurrency 2 with `maxTestRunnerReuse: 1`, and at concurrency 1 with Stryker's default reuse, so one runner process handles many mutants.
**Test scenarios (pinned cases):**
- A static mutant in a module-level table is killed with the module re-imported fresh.
- A mutant hit while tests are collected (in a `describe` body, or in a production-computed `it.each` table) is recorded as static. A changed test set under that mutant is invalid, not a non-kill.
- A mutant executed only in a suite's `beforeAll` is attributed to every test of that suite.
- Under `disableBail`, two tests kill the same mutant and both are reported.
- An `it.each` table whose row titles truncate to the same name keeps distinct ids by row ordinal, and those rows are flagged not oracle-validated.
- A mutant that throws a `TypeError` is Killed in Stryker's report and runtime in the sidecar.
- A failing `beforeAll` is killed-outside-test, hook, attributed to the suite's tests.
- A failing `afterAll` leaves its tests passing and is killed-outside-test, hook.
- A static mutant that makes a test file fail to import is killed-outside-test, collection, never invalid and never Survived.
- An infinite loop inside a production `try/catch` that swallows the hit-limit error is a timeout through the hit count.
- An unhandled rejection during a mutant run is infrastructure.
- A test with `retry: 1` is inconclusive.
- Exporting `__STRYKER_ACTIVE_MUTANT__` in the parent environment changes no result.
- A mutant covered by no test has the raw Stryker status NoCoverage and the normalised class survived.
- A test failing with Vitest's own `testTimeout` error is class timeout.
- One runner process handles two consecutive mutants, and the second run shows no state from the first.
- A mutant run that selects zero tests is invalid.
- Mutants handled by processes that never ran the dry run still select the right tests through the run manifest.
**Verification:** `test:mutation:conformance` passes, and changing any pinned expectation makes it fail with the case named.
### U5. Pure-JS category classifier
**Goal:** decide which `gitnexus/test/unit` files belong to the pure-JS category.
**Requirements:** R10; KTD11.
**Dependencies:** none.
**Files:**
- `gitnexus/tools/mutation/classify/classify.ts` (create)
- `gitnexus/test/unit/mutation/classify.test.ts` (create)
**Approach:**
1. Walk each test file's import closure: static imports, plus `import()`, `require()` and `createRequire` calls whose specifier is a string literal or a `const` bound to one. Resolve relative imports and `gitnexus-shared` specifiers.
2. Reject a file whose closure reaches a native package (tree-sitter grammars, `@ladybugdb/core`, onnxruntime, any `.node` binding), a worker-spawning module, or `child_process`.
3. Reject a test that reads files as text from `src/`, from `test/fixtures/`, or from a path outside `gitnexus/` (for example `../../../gitnexus-shared/src`). Production modules that import `fs` do not cause rejection by themselves.
4. Reject a file whose closure contains a dynamic import, `require` or `createRequire` call with a specifier that cannot be resolved this way (reason: "unresolvable dynamic import").
5. Emit the accepted list and a reason for every rejection.
6. Report which accepted files have `beforeAll` hooks, for KTD11's third scope.
**Test scenarios:**
- `test/unit/shared-type-extractors.test.ts`, `test/unit/extract-element-type-from-string.test.ts` and `test/unit/schema.test.ts` are accepted.
- `test/unit/receiver-twin-list-drift.test.ts` is rejected with reason "reads source outside gitnexus/".
- `test/unit/type-env.test.ts` is rejected with reason "native package: tree-sitter".
- A synthetic test that reads `src/x.ts` as text is rejected.
- A synthetic test whose production closure imports `fs` but reads no files in the test is accepted.
- `test/unit/node-module-compat.test.ts` is rejected: it reaches `node:child_process` through `await import(...)` of `src/core/embeddings/runtime-install.ts`.
- A module loaded through `createRequire` that resolves to a native package causes rejection.
- A dynamic import with a computed, non-constant specifier causes rejection with reason "unresolvable dynamic import".
**Verification:** the accepted list is identical across two runs, and every rejection carries a reason.
### U6. Gate 1 execution-path fidelity and envelope
**Goal:** prove that the runner's unmutated run behaves like a normal Vitest 5 run on the pure-JS category, and record the validated envelope.
**Requirements:** R6; KTD9, KTD19.
**Dependencies:** U5, U10.
**Files:**
- `gitnexus/tools/mutation/fidelity/fidelity.ts` (create)
- `gitnexus/tools/mutation/fidelity/allowed-config-diffs.ts` (create)
- `gitnexus/test/unit/mutation/fidelity-compare.test.ts` (create)
**Approach:**
1. Rebuild `gitnexus-shared/dist`, then run the pure-JS list with the base config.
2. Run the same list through the runner's unmutated dry run via U10.
3. Compare test identities as `[file, titles, line, column]`, outcomes, effective config and resolved production module paths.
4. Allowed config differences: `setupFiles`, `maxWorkers`, `maxConcurrency`, `bail`, coverage, the KTD8 aliases, `dangerouslyIgnoreUnhandledErrors`, `cacheDir`, `NODE_ENV`/`VITEST`, `disableTypeChecks`, the project name, the include and exclude lists, the projects array, `sequence.groupOrder`, and `root` compared after mapping sandbox paths to checkout paths.
5. Path rule (KTD19): every production module of the Stryker run resolves inside the active sandbox. `gitnexus-shared/dist` paths in the base run map to relocated `src` paths.
6. Runtime load check (KTD19): in the base run, record the modules each pure-JS test loads, and fail on any native binding, worker or `child_process` load.
7. Write the envelope record to `gitnexus/test/fixtures/mutation-oracle/envelope.json`: Vitest and Stryker versions, Stryker concurrency and runner reuse, config, Node version, OS, platform-dependent skips, environment variables read at load (for example `GITNEXUS_EMBEDDING_DIMS`).
**Test scenarios:**
- Identical identity lists with one line shifted by one fail with the shifted test named.
- A config difference outside the allowed list fails. An allowed difference passes and is listed in the envelope.
- A Stryker run that loads a module from the prepared copy or the original checkout, rather than the sandbox, fails the path rule.
- A test passing in one run and skipped in the other fails.
- A pure-JS test that loads `node:child_process` at runtime fails the runtime load check.
- The envelope record lists the Vitest and Stryker versions of the run.
**Verification:** gate 1 passes on the pure-JS list, and the envelope record is written.
### U13. Oracle command-runner run and wrapper
**Goal:** the oracle's own Stryker run, which gives a per-mutant outcome class for every mutant and keeps the instrumented sandbox for single-test launches.
**Requirements:** R8; KTD17, KTD18.
**Dependencies:** U10.
**Files:**
- `gitnexus/tools/mutation/oracle/command-wrapper.ts` (create: runs Vitest with a JSON reporter and writes the report under the active mutant id)
- `gitnexus/tools/mutation/oracle/command-run.ts` (create)
- `gitnexus/tools/mutation/oracle/oracle-classify.ts` (create: the oracle's own KTD17 classifier)
- `gitnexus/test/unit/mutation/oracle-classify.test.ts` (create)
**Approach:**
1. Run the scope through U10 with the `command` runner and `cleanTempDir: false`. The command is the wrapper.
2. The wrapper writes one Vitest JSON report per active mutant, so zero-test runs, infrastructure failures and kills outside tests are distinguishable from test kills.
3. Classify every mutant with the oracle's own classifier, and record the kept sandbox path.
**Test scenarios:**
- A JSON report with one failing test and an `AssertionError` stack classifies as killed-by-test, assertion.
- A JSON report with a file-level error and no tests classifies as killed-outside-test, collection.
- A mutant with Stryker status Timeout and no report classifies as timeout, read from `mutation.json`.
- A missing report for a mutant with a non-zero exit and no Timeout status classifies as infrastructure.
- A report with zero executed tests classifies as invalid.
**Verification:** every mutant in the `shared.ts` scope has an oracle class, and the kept sandbox exists after the run.
### U12. Oracle single-test launcher and benchmark
**Goal:** run one test with one mutant in a fresh process inside the kept sandbox, and measure what the oracle costs.
**Requirements:** R8, R11; KTD13, KTD15.
**Dependencies:** U13.
**Files:**
- `gitnexus/tools/mutation/oracle/launch-single.ts` (create)
- `gitnexus/tools/mutation/oracle/oracle-setup.ts` (create)
- `gitnexus/tools/mutation/oracle/oracle-reporter.ts` (create)
- `gitnexus/tools/mutation/bench/oracle-cost.ts` (create)
- `gitnexus/package.json` (modify: `test:mutation:bench` script)
- `gitnexus/test/unit/mutation/launch-single.test.ts` (create: observation validity rules)
- `gitnexus/test/unit/mutation/oracle-boundary.test.ts` (create: no import from `runner/` under `oracle/`)
**Approach:**
1. Select the test by `<file>:<line>` plus an escaped, anchored `-t`, with the mutant installed through the environment before any import.
2. A launch is valid only when exactly one test executed with the intended identity and the installed mutant id matches.
3. Enforce a wall-clock limit per launch with a process-group kill, set above the base config's `testTimeout` plus `hookTimeout` so a hang surfaces as Vitest's own timeout first. Both are class timeout (KTD17).
4. The benchmark times launches on a small sample, counts static mutants after blanking, times the command-runner pass, and projects the three-scope full-sample wall clock on 4 vCPU against the 6-hour job limit.
**Test scenarios:**
- A launch that executes zero tests, or two tests, is an invalid observation.
- A passing launch that never hits the mutant is a valid non-covering observation.
- A launch whose installed mutant id differs from the intended one is invalid.
- A launch killed by the wall-clock limit is a timeout observation.
- The cost projection from a fixed list of timings is deterministic.
- A module under `oracle/` importing from `runner/` fails the boundary test.
**Verification:** the benchmark report lists per-launch cost, static-mutant count, command-runner cost and the projected full-sample wall clock.
### U7. Oracle freeze and compare (gate 3)
**Goal:** freeze expected outcomes for the three oracle scopes and compare the runner with them.
**Requirements:** R8; KTD11, KTD12, KTD13, KTD17.
**Dependencies:** U5, U12, U13.
**Files:**
- `gitnexus/tools/mutation/oracle/freeze.ts` (create)
- `gitnexus/tools/mutation/oracle/compare.ts` (create)
- `gitnexus/test/fixtures/mutation-oracle/*.json` (create: one frozen manifest per scope)
- `gitnexus/package.json` (modify: `test:mutation:oracle` and `test:mutation:oracle:freeze` scripts)
- `gitnexus/test/unit/mutation/oracle-compare.test.ts` (create)
- `gitnexus/test/unit/mutation/oracle-sample.test.ts` (create)
**Approach:**
1. Freeze: select the three scopes, sample 100 mutants per scope with a recorded seed, and top up the strata: static/module-load, `beforeAll`, several killers, survivors, NoCoverage, `it.each`. Count every stratum, and write the manifest per KTD12.
2. Baseline: every selected test passes alone with no mutant, and the oracle's own test inventory matches the runner's run manifest (KTD18).
3. Per-mutant classes come from U13. Per-test outcomes for sampled mutants come from U12 launches.
4. Compare with the runner's normalised results from U10. Every supported observation must agree, under KTD13's normalisations only.
**Execution note:** run U12's benchmark before the full freeze, so the cost is known first.
**Test scenarios:**
- The same seed gives the same sample, and a different seed changes it.
- Strata are counted across the three scopes. The `shared.ts` scope alone, with no hooks and no `it.each`, freezes without a `beforeAll` or `it.each` stratum.
- A scope whose tests contain `beforeAll` hooks but whose sample has none fails freezing unless those tests are marked not oracle-validated.
- A test present in the oracle's own inventory but missing from the runner's run manifest blocks gate 3 before outcomes are compared.
- A global mutant id present in run A but not run B blocks gate 3.
- Runner Timeout against an oracle wall-clock kill is accepted as one class.
- Runner NoCoverage against oracle survived is accepted as one class, and the raw status is kept in the report.
- Runner killed-by-test assertion against oracle runtime blocks, even with a recorded cause.
- A manifest whose content hashes or versions no longer match fails with "re-freeze required".
**Verification:** the oracle passes on all three scopes with zero unresolved differences, and the manifests are committed.
### U8. Gate 4 positive controls and determinism
**Goal:** finish the pure-JS validation with planted positive controls and a determinism check.
**Requirements:** R9; KTD19.
**Dependencies:** U7.
**Files:**
- `gitnexus/tools/mutation/controls/run-controls.ts` (create)
- `gitnexus/test/fixtures/mutation-controls/planted.json` (create: frozen global mutant ids with expected killers and expected non-killers)
- `gitnexus/package.json` (modify: `test:mutation:controls` script)
- `gitnexus/test/unit/mutation/controls.test.ts` (create)
**Approach:**
1. Controls are selected from the run's own Stryker mutants by global id; no source is edited. Freezing picks them, records their expected killers and non-killers, and regenerates them on re-freeze. At least one control sits in relocated `gitnexus-shared` code and one in an `as const` table (KTD19).
2. Each control is killed by the tests that depend on it and survives under tests that do not touch it.
3. Determinism: profile A runs at Stryker concurrency 1 with default runner reuse; profile B runs at concurrency 3 with `maxTestRunnerReuse: 1`. Kill sets, keyed by global mutant id and test id, must be identical.
**Test scenarios:**
- A planted mutant killed by a test outside its expected killers fails the controls check.
- A planted mutant that survives its expected killer fails the controls check.
- Two kill-set maps that differ in one test fail the determinism comparison with the mutant and test named.
**Verification:** gate 4 passes on the pure-JS category, including the shared-code and `as const` controls.
### U9. CI workflow and documentation
**Goal:** run the conformance suite automatically on relevant changes, and document the tooling and the Phase 0 findings.
**Requirements:** R12, R14; KTD14.
**Dependencies:** U4, U6, U7, U8.
**Files:**
- `.github/workflows/mutation-runner-validation.yml` (create)
- `TESTING.md` (modify: mutation tooling section)
- `docs/plans/2026-10-23-test-scenario-map-design.md` (add to git with force, since `docs/` is gitignored; update §5.4, §5.4a, §7 with Phase 0 findings: `as` assertions, `disableTypeChecks`, upstream #6214, oracle scopes, the second Stryker run, benchmark numbers)
- `docs/plans/2026-10-23-1823-feat-stryker-vitest5-runner-phase0-plan.md` (add to git with force)
- `gitnexus/tools/mutation/UPSTREAM.md` (modify: residual deltas against upstream, for a later upstream PR)
**Approach:**
1. Workflow triggers: `pull_request` path-gated on `gitnexus/tools/mutation/**`, `gitnexus/test/fixtures/mutation-*/**`, `gitnexus/package.json`, `gitnexus/package-lock.json` and `gitnexus/vitest.config.ts`, running the conformance suite. `workflow_dispatch` also runs gates 1, 3 and 4 and the oracle benchmark.
2. Workflow conventions: top-level `concurrency` block, SHA-pinned actions, `persist-credentials: false`, `permissions: contents: read`, advisory (`continue-on-error: true`), outside `ci-status.needs`.
**Patterns to follow:** `.github/workflows/impact-pdg-mutation-report.yml` (tool workflow with dispatch, setup action, artifact upload); the `test-dupes` job in `.github/workflows/ci.yml` (advisory job conventions).
**Test expectation:** none -- CI configuration and documentation; verified by the workflow linter and a dispatch run.
**Verification:** the workflow passes `check-workflow-concurrency.py`, a PR touching `gitnexus/tools/mutation/**` triggers it, and the design doc reflects Phase 0 findings.
---
## Verification Contract
| Check | Command (from `gitnexus/` unless noted) | Applies to |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | -------------------- |
| Unit tests for tooling | `npx vitest run test/unit/mutation` | U1-U8, U10, U12, U13 |
| Existing dupe-scanner tests still pass | `npx vitest run test/unit/test-dupes.test.ts` | all |
| Typecheck tests and the tools they import | `npm run typecheck:tests` | all |
| Lint and format (repo root) | `npm run lint` and `npm run format:check` | all |
| Prepared copy runs the oracle scopes green | `npm run test:mutation:prepare`, then the scopes under `tools/mutation/vitest.mutation.config.ts` | U1 |
| Runner kills on `shared.ts` | `npm run test:mutation:run` on the `shared.ts` scope | U3, U10 |
| Gate 2 | `npm run test:mutation:conformance` | U4 |
| Oracle benchmark | `npm run test:mutation:bench` | U12 |
| Gate 3 | `npm run test:mutation:oracle` | U7 |
| Gate 4 | `npm run test:mutation:controls` | U8 |
| Workflow concurrency rule (repo root) | `python3 .github/scripts/check-workflow-concurrency.py` | U9 |
| GitNexus change analysis before each commit (repo root) | `node .gitnexus/run.cjs detect-changes --scope all --repo .` | all |
---
## Definition of Done
- Gates 1-4 pass on the pure-JS category, and the envelope record and frozen oracle manifests are committed.
- The runner produces non-empty per-test kill sets on `shared.ts` that match the oracle.
- The oracle benchmark numbers are recorded in the design doc.
- The conformance workflow runs on a PR touching `gitnexus/tools/mutation/**`.
- No existing test or production file under `gitnexus/src`, `gitnexus/test/unit` or `gitnexus/test/integration` changed.
- The untracked spike files (`gitnexus/spike-input-facet.mts`, `gitnexus/test/spike-coverage.setup.ts`, `gitnexus/vitest.spike-cov.config.ts`, `gitnexus/vitest.stryker-bake.config.ts`) are not committed.
- Abandoned-attempt code and debug probes are removed from the diff.
- `CHANGELOG.md` is unchanged.

715
docs/plans/2026-10-23-test-scenario-map-design.md Normal file
View file

@ -0,0 +1,715 @@
# Test Scenario Map — design
Date: 2026-10-23 · Status: design — Codex round 7 (fresh full review, `gpt-6-astra` xhigh): rework, 9 findings, all applied; blocker resolved with a semantic value-flow check (§5.3a). Approved 2026-10-23 ("we can proceed"). Phase 0 (§5.4a, pure-JS runner slice) built 2026-10-24: gates 1-4 pass on the pure-JS category. Phase 1 (§9, pure-JS report) built 2026-10-25 in `gitnexus/tools/scenario-map/`; its deviations from this design are applied in place and listed in §12 · Related: PR #3686 (advisory duplicate-test scan; Phase 0 tooling in `gitnexus/tools/mutation/`)
## 1. Intent
GitNexus has about 22,700 test cases (1,350 files) plus 42 benchmark directories. **Scope of this effort: the CLI package `gitnexus/`** (about 1,280 test files: `test/unit`, `test/integration`, `bench/`). `gitnexus-web/` is deferred. We want three things:
1. **No repeated scenarios.** Many tests may exercise the same code path, but each must exercise it in a _different scenario_. Two tests that run the same path, check the same thing, and differ only in data the code treats identically are one scenario written twice.
2. **Less clutter.** Repeated scenarios should be removed, and same-logic table rows folded into `it.each`, without losing any protection.
3. **High quality.** The tool should also surface tests that catch none of the evaluated faults and code that the evaluated tests do not protect.
The tool reports. It never deletes or edits tests. Nothing it emits is a removal decision: every removal needs strong evidence (§2), a human scenario review, and the removal check in a cleanup PR (§6).
## 2. Definitions
| Term | Meaning |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Test identity** | Test file, describe path, title, declaration location, and parameter-row index for `it.each`/`test.for` rows. Duplicate titles are disambiguated by declaration order. Stable across runs and shards. |
| **Mutant identity** | Source revision + file + span + operator + replacement. Globally comparable across runs and shards (Stryker's per-run mutant IDs are not). |
| **Execution provenance** | Per test (from the V8 hook, §5.2): the set of files it executed, including files excluded from mutation, `node_modules` entry points of native packages (tree-sitter, `@ladybugdb/core`, onnxruntime), and worker-spawn call sites. It covers what the canonical path cannot see. **Strong-eligible provenance** (the one predicate used everywhere): provenance is known, and contains no mutation-excluded file and no native-package entry point, and no worker spawn unless the worker code is inside a validated instrumented-worker envelope (§5.4). |
| **Canonical path** | For verdicts: the set of mutant identities a test covers (Stryker `coveredBy`), including mutants executed by its suite's `beforeAll`. V8 block coverage (§5.2) is used **only to retrieve candidates**, never for a verdict. |
| **Checks** | The shape of each `expect(subject).matcher(…)` with literals abstracted. Selector literals inside the subject (`getNodesByLabel(r, 'Function')`) stay exact. Expected values are not lost: they are part of the input facet (§5.3 rule 3). Snapshot assertions are compared by snapshot content hash. Supported assertion forms are listed in the implementation; anything else makes the test _unsupported_ for strong evidence. |
| **Context** | Enclosing hooks, `vi.mock`/`vi.spyOn`/`vi.stubEnv`, env writes and wrapper-call arguments (reusing #3686's context closure). Must be exactly equal for any repeat or row verdict. |
| **Input facet** | A signature of what the test feeds the code (§5.3). It also keeps the concrete values it discarded, per position, for the value-flow check (§5.3a). |
| **Kill** | A mutant for which the test fails **by assertion failure**. Timeouts, runtime errors, crashes, hook/fixture failures and infrastructure failures are **not** kills; they make the comparison inconclusive. |
| **Kill set** | The test's kills over the **evaluated mutant universe** of that run (`disableBail`). |
### Scenario fingerprint (primary verdict since 2026-10-25)
Two tests are **duplicates** when they run the **same execution path** (identical V8 block sets) on the **same input data** (equal input facets and discarded values, every input resolved, the runtime trace clean) and check the **same expectation** (equal checks-and-context keys). This needs no mutation run. A pair with an unresolved input is at most a **possible duplicate**; other input values make a **table-row** candidate; the same path and input with other checks is **same scenario, different checks**. Rules: `gitnexus/tools/scenario-map/verdict/fingerprint.ts`; plan: `docs/plans/2026-10-25-1700-feat-scenario-fingerprint-plan.md`. The mutation verdicts below remain as optional evidence (§6: dispatch, or the `scenario-map` label on a pull request).
### Verdict categories
Precedence, highest first: **Inconclusive → Out of scope → Weak → Repeat (strong) → Repeat (weak) → Row → Review → Distinct.** Subsumed is a per-test property that is reported alongside these verdicts. A test with an empty kill set can never be part of a repeat.
| Verdict | Rule | Action |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| **Inconclusive** | Any of: a timeout, error or flaky result on a mutant relevant to the pair, or missing results, unless a kill witness already rules out a repeat (a mutant both tests ran cleanly, killed by assertion by exactly one; the pair then goes on to Row or Distinct); differing outcomes on rerun; an isolation check that disagrees (§5.4); unattributable path (concurrent tests) | Rerun or investigate; no other verdict |
| **Out of scope** | Covers no evaluated mutant, or its test category has not passed runner validation (§5.4a) | Not judged |
| **Weak** | Covers evaluated mutants but **kills no evaluated mutants** | Strengthen or delete (human) |
| **Repeat candidate (strong evidence)** | All of: exact canonical path; same checks; same context; same input facet; identical non-empty kill sets over one complete common mutant universe; kills are assertion failures and stable on rerun; the full isolation pass agrees (§5.4); row deletion is _symmetric_ or _not applicable_; value-flow check clean (§5.3a); no unsupported assertion or table construct; strong-eligible provenance for both tests; no unresolved facet input | Human scenario review; removal only through a cleanup PR with the removal check (§6) |
| **Repeat candidate (weak evidence)** | Path, checks, context, facet and kills match as above, but one disqualifier applies: provenance is not strong-eligible, an unresolved facet input, inconclusive row deletion, an unsupported construct, the full isolation pass was not run (§5.4), or a test is not oracle-validated (§5.4a). A value-flow veto never yields a repeat verdict; it yields Review | Review queue only |
| **Row** | Same canonical path, checks, context and facet, but kill sets differ or row deletion is _asymmetric_ | Fold into `it.each`; keep every row |
| **Review** | Same canonical path, checks, context and kill set, but a different input facet, **or** the same facet with a value-flow veto or unknown (§5.3a) | Human decides; mutation cannot tell them apart |
| **Distinct** | Anything else | Keep |
| _Subsumed_ (property) | Non-empty kill set strictly inside another test's, over the common universe | Reported as "no unique protection among evaluated mutants"; never a removal suggestion alone |
Phase 1 adds two weak-evidence disqualifiers. Both can only demote: V8 block sets that are not identical (§5.2), and a hook-attributed mutant on the pair's path or kills (§5.4 kill validity). A residual body difference makes two equal facets unequal (§5.3), so such a pair with equal kills ends in Review, never in a repeat.
## 3. Goals, non-goals, anti-goals
**Goals**
- Report repeat candidates, rows, weak tests and evaluated coverage gaps for the whole `gitnexus/` suite. Unit tests first, then the integration suite (tree-sitter, resolver and LadybugDB tests) through the Vitest 5 runner on the repo itself (§5.4, §5.4a).
- Every verdict carries its evidence: canonical paths, checks, context, facet differences, kill-set differences, row-deletion outcome, exclusions, and inconclusive causes.
- Report precision per stage, with sample sizes and confidence intervals (§8), and never claim more than those numbers support.
- Re-runnable from the CLI and on a schedule in CI.
**Non-goals**
- Automatic deletion, rewriting or folding of tests.
- Per-PR execution of the expensive layers (coverage and mutation). PR-time stays the structural scanner from #3686.
- `gitnexus-web/` (deferred, maintainer decision 2026-10-23) and Python tests (`eval/`). This effort covers the **CLI package `gitnexus/`** only: CLI, MCP server and core, with their tests and benches.
- Proving semantic equivalence. Identical kill sets mean "no evaluated mutant tells these apart", not "equivalent under every bug".
- Whole-suite protection claims from candidate-limited runs.
**Anti-goals** (failure even if it technically works)
- Any verdict that calls two genuinely distinct scenarios the same with nothing flagging it. This is the error that destroys protection.
- Similarity scores (embeddings, text similarity, path similarity) presented as evidence for a verdict.
- A report so noisy that people stop reading it.
## 4. Evidence this design rests on
All measured on this repo on 2026-10-22/23. Scratch artifacts were in `/tmp/test-audit/`; the labeled sets are to be committed (§8).
**Limits of this evidence (read first).**
- One author produced all the labels and also tuned the rules on them.
- The holdouts were drawn from a filtered slice: "same path + checks, different data" groups over 47 coverage files.
- 23/23 on the one untouched holdout gives a two-sided 95% exact lower bound of about 85% for distinct-scenario recognition. It measures **facet discrimination only**, not end-to-end precision of repeat candidates.
§8 adds independent labels and prospective holdouts before any precision claim is made.
### 4.1 Audit
1,350 test files, 22,678 cases, 42 bench dirs. Hand-verified:
- about 160 removable cases (~0.7%)
- 22 redundant pipeline runs
- 3 bench scripts and 7 tripwire tests
- defects: a fail-open bench guard, 5 orphan bench gates, tests that test nothing
### 4.2 Tool bake-off
83 labeled duplicate pairs (48 found by manual audit) and 15 look-alikes.
| Tool | Audit-found pairs caught | Look-alikes wrongly flagged |
| ------------------ | ----------------------------------------------------------- | --------------------------- |
| #3686 scanner | 39/48 | 0/15 |
| vitest-dedup 0.7.0 | 26/48 (skips 24 oversized files and 1,482 unresolved tests) | 0/15 |
| jscpd 5.4.2 | 14/48 | 5/15 |
saidvsdid 0.1.0 (deletion guard) caught a skip, deletions and a weakened matcher, but missed a dropped `it.each` row.
### 4.3 Embeddings (rejected)
arctic-embed-xs over all 22,769 test bodies:
- Retrieval was perfect: every known pair was in the top 5.
- Discrimination failed: duplicates vs look-alikes scored AUC 0.69 on bodies and 0.48 with titles.
- 0 of 16 sampled high-similarity pairs were redundant.
- GitNexus test cases are not graph nodes, and the indexes carry no embeddings.
### 4.4 Path layer (V8 per-test block coverage)
- AUC 0.876 for duplicates vs look-alikes; 48/55 known duplicates share path, checks and data.
- Pipeline tests drift slightly between runs (0.997 similarity). That is why similarity is used only for retrieval, and verdicts require exact canonical paths, which costs some recall in the safe direction.
- Blind to native tree-sitter parsing, regex alternatives and lookup-table rows.
### 4.5 Input facet
- Tuned on 4 labeled sets of 30 pairs.
- On a fifth, untouched set: **23/23 distinct scenarios** recognised (95% lower bound about 85%) and 4/7 repeats/rows. Every miss was on the safe side.
### 4.6 Mutation layer (Stryker 10, per-test)
_Historical: these numbers come from the spike, which ran a temporary Vitest 4 copy harness. The design uses Vitest 5 only (§5.4a)._
- **Runtimes:**
- `shared.ts` alone: 1,232 mutants in 26 s.
- 166 self-contained unit files (2,980 tests): 55,878 mutants in 65 min on 12 workers, static mutants skipped.
- 48 candidate files with static mutants: 12–14 min.
- **Blind spots found and handled:**
- Static mutants are required: 26 of 84 "repeats" were table rows.
- `as const` array entries are never mutated: the spike stripped them in its copy (§5.4 keeps this as a pre-step).
- Object-literal keys are never mutated: added a row-deletion operator.
- Per-test results must be grouped by path and checks first.
- **Result in the 48-file scope:** 57 pairs passed all layers.
- 23 are verbatim copies.
- Of the 34 non-identical pairs, 33 were judged genuine repeats by the same author. 1 is debatable and is counted as not genuine.
- This is a single-author judgement, not an adjudicated precision figure.
- **Weak tests:** about 16 kill no evaluated mutants, e.g. the 2048-character guard tests, which pass without the guard.
## 5. Architecture
Four layers. Layers 0–2 run over every test in scope, and Layer 0 agreement is **not** required for any verdict (Layer 0 misses real repeats, e.g. pairs whose `afterEach` differs). Layer 3 runs on candidate groups produced by Layers 1–2. The value-flow check (§5.3a) runs on candidate pairs whose concrete values differ.
### 5.1 Layer 0 — structural scan (exists, PR-time)
`gitnexus/tools/test-dupes/test-dupes.ts` from #3686 checks:
- same body + same setup
- literal-assertion pins
- repeated fixture builds
- copied helpers
- orphan bench gates
It stays the per-PR advisory comment. Planned small fixes (the bot findings on #3686, renamed locals, `afterEach` relaxation) are tracked there.
### 5.2 Layer 1 — candidate retrieval (path + checks + context)
V8 per-test coverage hook from the spike: a Vitest setup file using `node:inspector/promises`, run with `--repeats 1` (keep the warm run). Attribution rules:
- Coverage between a test's `beforeEach` and `afterEach` belongs to the test.
- A root `beforeAll` discards module-load coverage.
- `beforeAll` work is attributed per suite level: the coverage taken at the first `beforeEach` of each nested suite belongs to that suite.
- Tests under `describe.concurrent`/`test.concurrent` cannot be attributed. Their provenance is _unknown_, so any pair involving them is **Inconclusive** (reason: path unattributable).
Retrieval:
- **Metric:** Jaccard similarity over block identities. Pairs at ≥ 0.99 become candidates, together with pairs with identical checks + context.
- **Not transitive:** candidate groups are connected components and are used only to schedule Layer 3, never as verdicts.
- **Block identity (Phase 1, demote only):** a pair whose V8 block sets are not identical caps at weak evidence. The 0.99 tolerance would otherwise hide one test that reaches extra mutant-free code.
### 5.3 Layer 2 — input facet
Computed statically per test. Each rule was added to fix a measured miss; removing any of them reintroduces that miss. An input the rules cannot resolve is recorded as _unresolved_ and blocks strong evidence.
1. **Inline source code** is detected by language (`SupportedLanguages.X`, grammar identifiers, file-name literals, test path) and parsed with GitNexus's `createParserForLanguage`. Its anonymised tree shape is hashed:
- **Locally declared names become `ID`.** Only declaration nodes count, including for/range/enhanced-for/assignment targets and Go `type_spec`.
- **External types and GitNexus builtin names stay**, even when a fixture stubs them.
- **Leaf rules apply only to leaves.** Composites such as `qualified_identifier` and `compound_literal_expression` keep their structure.
- **Literal values are dropped; literal kinds are kept.**
- **Comment text is shaped** for inline inputs and ignored in fixture files.
2. **COBOL** gets a reserved-word profile.
3. **Data literals, including expected values,** map to their production table by best fit. Search order:
1. tables on the executed path
2. the import closure, up to 2 hops
3. property columns
Never pool these. A table is a string array, Set/Map keys, object keys, switch cases or a regex alternation. A literal in no table is reduced to a boundary-aware shape: `0`, `1`, `-1`, negatives and empty strings stay exact.
4. **Selector literals** in an `expect(...)` subject call stay exact.
5. **Object-literal keys** count as input.
6. **Literals and builder calls keep their source order,** as shapes plus called function names.
7. **Shared-fixture tests** resolve their literals, and word tokens inside them, against the describe's fixture. The facet takes:
- the matched definition's shape
- the shapes of the fixture types it references, with their bases
- the declared types of the fields it uses
Presence-only assertions use the declaration kind instead.
8. **Fixture files** named anywhere under `test/fixtures/` contribute their shape. Only synthetic paths fall back to extension tables.
**Phase 1 additions (demote only):**
- **Numeric literals** follow rule 3 too: `0`, `1`, `-1` and negatives stay exact.
- **±1 rule:** a numeric literal within ±1 of a production numeric constant stays exact. Stryker has no numeric-literal mutator, so a boundary value such as `LIMIT + 1` would otherwise collapse with an interior value. The constants come from the test's executed production files and its static import closure, because a constants-only module runs only during the discarded module load. Statically evaluable initialisers such as `2 * 1024` count.
- **Residual-difference guard:** for two instances with equal facets, the check diffs the instance bodies. Any differing node that is neither a facet part nor a recorded discarded value marks both instances unresolved and makes the two facets unequal. This catches pairs that differ only in an enum member, a regex, a boolean, `null` or an operator.
### 5.3a Value-flow check (semantic veto)
**Why:** the facet drops literal values (rules 1 and 3). Two tests can then share facet, path and kills while protecting different behaviour inside code mutation cannot see. Example: a wrapper around `Math.round` tested with `2.4 → 2` and `2.5 → 3`. Non-empty kills show some sensitivity, not sensitivity to the erased difference.
**Scope:** every candidate pair whose discarded values differ (§2 input facet). Phase 1 also runs step 4 on every facet-equal candidate pair, as a cheap backstop for differences the facet cannot see. The check can only send a pair to **Review**; it never promotes. It uses four sources, and all must come back clean:
1. **Test side (Babel).** Bind each differing value to where it enters production code: a call argument, a fixture field or a table key. A differing expected value follows the input that produces it. A value that cannot be bound is _unknown_.
2. **Types and resolution (TypeScript 7 checker).** Through `typescript/unstable/sync` on `gitnexus/tsconfig.test.json`, resolve each entry call through aliases (`getAliasedSymbol`) to its declaration and read the parameter and contextual types.
- An `any`/`unknown`-typed or unresolved entry is _unknown_.
- Literal-union or enum members are recorded for the reviewer.
- Probe (2026-10-23, TypeScript 7.0.2): the 2,401-file project loads in about 9 s; `Math.round` resolves to `lib.es5.d.ts`. The API is unstable: pin the TypeScript version, and treat any API failure as _unknown_.
3. **Static value flow (GitNexus PDG).** From each entry parameter, follow the value inter-procedurally to its sinks, and classify each sink with the checker:
- **repo code** (branch, comparison, arithmetic, call into a repo function): coverage and mutation see it. Clean only inside the pair's mutated production files (§5.4 mutate scope). Repo code outside them is _unknown_.
- **supported table read** (§5.4 row deletion): row deletion tests it. Clean.
- **pass-through** (stored, returned unchanged, compared for identity only): clean.
- **opaque:** a call declared in `lib.*.d.ts`, `@types/node` or `node_modules`, or any regex use. Veto.
- **unknown:** truncated flow, dynamic dispatch, or a file without a PDG layer. Veto (fail closed).
- Classes apply in that order. Reads from supported tables and storage into built-in containers (`push`, `Map.set`, `Set.add`) are table reads and pass-through, even though they are declared in `lib.*.d.ts`. The flow continues from wherever the stored value is read back; if the PDG cannot follow it, it is _unknown_.
- **Stitched walk (Phase 1, `gitnexus/tools/scenario-map/value-flow/pdg-walk.ts`).** GitNexus `pdg_query` flows are intra-procedural, so the check stitches the walk itself:
- It reads each function's REACHING_DEF and `CALL_SUMMARY` edges through LocalBackend's `cypher` tool, scoped to the function's `BasicBlock:<file>:<fnLine>:` id prefix.
- The TypeScript checker resolves each callee. The argument position in the production AST gives the parameter. `CALL_SUMMARY` covers return values.
- A binding derived from the value (`const { ratio } = opts`, `x = v`) is walked under its own name. A store into a property or module state is _unknown_.
- Any opaque or unknown use in a statement vetoes the whole statement.
- A truncated per-function query, an ambiguous binding or the depth cap (4 functions, entry included) is _unknown_.
4. **Dynamic trace (Vitest 5).** One extra run of the pair with a setup file that wraps value-sensitive built-ins: `Math`, string transforms and comparisons, `RegExp` methods, number parsing and formatting, `JSON`, `Intl`, `Date`, sort comparators, plus every opaque sink step 3 found. Storage and table reads (`push`, `Map`/`Set` methods) are not wrapped. Each call from a production frame (`gitnexus/src`, `gitnexus-shared/src`) records its call site, its receiver (`this`), its arguments and a call count per site.
- If the two tests reach the same call site with a different receiver or different arguments, the differing data reached opaque code. Veto. This also catches derived values (`x * 10`) and flows step 3 missed.
- Asymmetric call sites, or asymmetric call counts at a site, veto too.
- Phase 1 runs the trace for every facet-equal candidate pair, not only for value-differing ones (demote only).
- The trace run is separate from mutation runs. If a test's outcome changes under tracing, the pair is Inconclusive.
**Lifting a veto:** only an explicit equivalence rule for that built-in (e.g. `Math.round` on two integers), each with its own test. The list starts empty; add a rule only when measured Review noise justifies it.
**Cost:** one checker load per run, PDG queries per value-differing candidate pair, and one traced run over the tests of every facet-equal candidate pair. Phase 1 runs value flow before the mutation shards, so the isolation pass (§5.4) is spent only on pairs that can still be strong.
### 5.4 Layer 3 — mutation evidence
**Configuration**
- Stryker `perTest`, `disableBail: true`, **static mutants included**.
- Runs **on the repo itself** (Stryker's sandbox copy of the checkout), through the Vitest 5 runner (§5.4a). There is no separate harness project.
- Phase 0 runs Stryker in a **prepared copy** of `gitnexus/` outside the checkout. Its `node_modules` links to the checkout's. It leaves out `vendor/`, `prebuilds/` and `dist/`, which the pure-JS category never loads.
- Phase 1: baselines, row deletion and the trace run (§5.3a) run on plain Vitest 5 under the mutation config, with a fresh Vitest instance per run. Row deletion cannot run through Stryker, because Stryker applies only its own mutators. It edits a throwaway copy of the prepared tree instead. This still honours Vitest 5 only.
- Mutate scope: the candidates' production closure **and** `gitnexus-shared/src`. `gitnexus-shared` resolves to its source via a Vitest alias in the Stryker-only config, so its tables are mutated.
- Phase 1: the mutate scope of a candidate group is whole production files: every production file in either test's V8 provenance or static import closure, with no line ranges.
- Phase 0 finding: Stryker cannot mutate files outside its working directory. The prepared copy therefore relocates `gitnexus-shared/src` inside the `gitnexus/` tree. The mutation config derives its aliases from the package `exports`, and a path map translates relocated paths back.
- **Type assertions** (Phase 0 finding, replaces the `as const`-only pre-step). Stryker's instrumenter skips every `TSAsExpression` subtree, not only `as const` tables (`gitnexus/src` alone has 317 `as const` nodes).
- The prepared copy blanks every ` as <type>` in the mutated files with spaces of the same length, keeping newlines. Lines and columns do not move, so the span map is an identity.
- The check is AST equality: the blanked file must parse to the original AST with every `TSAsExpression` unwrapped. "Still parses" is not enough: ASI, precedence and a parenthesised `(x) as T` can give a different tree.
- `satisfies` and the non-null `!` are not skipped, and stay.
- The edit is type-only, and Vitest does not typecheck. An upstream instrumenter option is the long-term fix.
- **`disableTypeChecks: false`.** Stryker's default prepends `// @ts-nocheck` to matching files, test files included. That shifts every test line by one and breaks test identities.
**Kill validity**
- Before mutating, the baseline must pass twice with identical test identities.
- Phase 1: two plain Vitest 5 baselines (a fresh instance per run) and Stryker's dry run must agree. Otherwise the group is _inconclusive_.
- Only assertion failures count (§2). Stryker `Timeout`, runtime and compile errors are recorded per test, and make any pair they touch _inconclusive_.
- A test file that fails to import under a static mutant reports no per-test result (`killed-outside-test:collection`). Every test of that file is recorded with that class, not as a missing result. When it holds for both tests of a pair, neither test ran, so the mutant cannot tell them apart and counts as a non-kill for both.
- **Full isolation pass**, required for strong evidence: rerun every mutant covered by either test (the pair's covered universe), both tests together and then each alone, comparing coverage and outcome class. Static mutants and mutants executed in `beforeAll` rerun in a fresh process with modules reset. Any disagreement makes the pair _inconclusive_. This catches order dependence and leaking mocks that hide a distinguishing mutant.
- Phase 1: the pass selects tests through the runner's `testIds` scope key (§5.4a). Each pair gets one Stryker run with both tests, and each test one run alone, shared across pairs. Mutate is restricted to the pair's covered universe. `maxTestRunnerReuse` 1 gives every mutant a fresh process. Coverage and outcome class per test and mutant must agree across the main, together and alone runs.
- Phase 1: the pass runs only for pairs that can still be strong after value flow, because value flow (§5.3a) runs before the mutation shards. A pair whose isolation does not fit the shard's time stays isolation not-run, which caps it at weak.
- **Hook-stratum cap (Phase 1, demote only):** a pair whose path or kills touch a hook-attributed mutant caps at weak. The cap holds until a gate 3 scope with a non-empty hook stratum passes (§5.4a). Today that stratum is empty in all three scopes.
- A sampled isolation pass is allowed as a cheap screen, but a pair without the full pass caps at weak evidence.
- Two passing baselines show observed stability, not absence of flakiness. The report says so.
**Comparability and sharding**
- Shard by **candidate group**, so both members of every compared pair are evaluated in the same run against the same mutant universe.
- Pairs are compared only over their common evaluated universe, keyed by global mutant identity. A missing result is _unknown_, never a non-kill.
- Shards merge **raw test–mutant observations**: test identity, mutant identity, selected-test scope, coverage, outcome class, validity and universe membership.
- Weak, gaps and subsumption are **recomputed** from the merged observations. Computed verdicts and counts are never merged or summed: two groups can mutate the same file, so a mutant can survive one group and be killed by another.
- Two observations of the same test and mutant with different outcome classes make it _inconclusive_.
- **Subsumption** of A by B is claimed only if B was evaluated on **every** mutant in A's evaluated universe, whether in the same shard or in a merged result covering that universe. Otherwise subsumption is _unknown_.
- Every comparison records its universe identifier in the report.
**Row-deletion operator** (Stryker never mutates object keys)
- Applies to candidate pairs whose differing literals are both keys of one production object literal.
- Delete each key in turn and run the pair.
- Phase 1: the operator splices one key's text out of a throwaway copy of the prepared tree and runs the pair on plain Vitest 5, a fresh instance per run. A reused instance serves cached transforms and would turn a no-op deletion into a false "neither fails". A control run comes first: both tests must pass, and the edited file must load.
- Outcomes:
- **symmetric:** both tests fail, or neither does
- **asymmetric:** exactly one fails, which means _row_
- **not applicable:** no such table
- **inconclusive:** errors or timeouts
- **Supported tables:** object literals with plain identifier or string keys, read through ordinary lookups (`table.key`, `table['key']`, `table[variable]`, `key in table`, `Object.keys/entries(table)`); and `new Set([...])`/`new Map([...])` built from literal entries, read through `has`/`get`. Deletion removes one key or one entry.
- **Unsupported tables:** any table that is built or changed at runtime (spreads, computed keys, `Object.assign`, assignments into it, getters, proxies, duplicate keys). These make the outcome _inconclusive_, which blocks strong evidence.
**Execution environment** _(maintainer decision 2026-10-23: Stryker runs on Vitest 5, the version `gitnexus/` uses, through a runner built in this effort)_
- Stryker runs on the repo checkout with its real `node_modules`, fixtures and config, so native packages, `test/fixtures/` and repo-relative paths behave as in normal test runs.
- Worker scripts are built (`node scripts/build.js`) before the run, with `GITNEXUS_WORKER_READY_TIMEOUT_MS=60000`.
- LadybugDB tests keep `lbug-db` sequential isolation **and** get a per-Stryker-process database and temp namespace, or are globally serialised across Stryker workers.
- **Runner validation** (§5.4a) replaces the harness fidelity checks.
- **Native code is never mutated.** Loading it is not the same as testing it.
**Worker threads**
- Parse workers load the built `dist/` script, not the instrumented sources.
- The builder makes workers load instrumented code, and proves it with **planted worker-side mutants as positive controls**: they must be killed by a test known to depend on them.
- Until that is proven, worker-only files are mutation _exclusions_.
- Strong evidence is enforced through **execution provenance**. V8 coverage cannot see inside native addons or other worker isolates, and module-load coverage is discarded, so a clean V8 run is **not** proof of no native or excluded execution. Two checks always apply together:
1. **V8 provenance:** excluded files, native-package entry points or worker spawns seen at runtime cap the test at weak evidence.
- **Excluded files (Phase 1):** any executed repo file under `gitnexus/` that is neither a test file (`test/**`, `*.case.ts`) nor mutated, such as `tools/` or `scripts/` code.
2. **Static closure:** if the test's static import closure reaches any native package (tree-sitter included), any worker-spawning module, or **any mutation-excluded file**, the cap applies too. Excluded files' module initialisation runs during the discarded load phase, so V8 cannot vouch for it.
- **No other exceptions.**
- Tests depending on tree-sitter, LadybugDB, onnxruntime or any other native package top out at **weak evidence**: review queue, never a strong repeat candidate.
- Worker execution stays capped until instrumentation is proven by positive controls. A proven, validated instrumented-worker envelope removes **only** the worker-spawn disqualifier from strong-eligible provenance (§2); worker code then appears in the canonical path. The native-package, excluded-file and static-closure caps still apply.
- Consequence: most parser, resolver and LadybugDB repeats will be weak-evidence candidates. That is intended; it is the price of the top anti-goal.
- Unknown or incomplete provenance always caps at weak evidence.
- Unexecuted worker mutants never count as survived.
**Integration cost**
- Unknown. Benchmark three cases before scheduling: one resolver file, one parser-worker-heavy file and one LadybugDB file. Record wall clock, CPU time, peak memory and timeout rate.
- Do not extrapolate linearly from the 12-worker numbers.
**Wording of per-test results**
- Use "kills no evaluated mutants" and "no unique protection among evaluated mutants".
- Gaps are "uncovered by selected tests" (`NoCoverage`) and "survived the selected tests" (`Survived`), counted per production file. `Survived` includes equivalent mutants.
- A whole-suite run with static mutants is an optional manual job.
### 5.4a Vitest 5 test runner for Stryker (Phase 0, critical path)
**Why:** `@stryker-mutator/vitest-runner` 10.0.0 supports Vitest ≤4. Under the repo's Vitest 5.0.5, the dry run succeeds but every mutant run executes **0 tests**, so nothing can be killed. Without a working runner there is no Layer 3.
**Root cause (Phase 0 diagnosis):** Vitest 5 matches `testNamePattern` against the full name joined with `' > '`. The runner builds space-joined names, so every test inside a `describe` is filtered out. Upstream merged the separator fix (StrykerJS PR #6214, 2026-10-23) but had not released it; `latest` was still 10.0.0.
**Built (Phase 0):** a new runner plugin, `vitest5`, written in TypeScript in `gitnexus/tools/mutation/runner/`. It is not a fork: the upstream fix covers only the separator, and the needs below change most of the upstream file. Versions: Vitest 5.0.5, Stryker 10.0.0, Node 22. The runner:
- selects tests by **Vitest task id**, one test specification per file, never by name pattern or related mode. It checks that the executed tests equal the requested ones, because a specification with only unknown ids silently skips every test.
- keeps the configured `forks` pool and starts fresh workers per run. It never forces threads.
- writes a **run manifest** from the dry run: test id to Vitest task id and location, and the expected test set per file. Only one Stryker process runs the dry run, so every runner process loads the manifest before its first mutant run.
- writes per-test failure classes to **sidecar files**, because Stryker's result type has no field for them.
- counts a hit in a suite's `beforeAll` as coverage for every test of that suite. Only load- and collection-time hits make a mutant static.
- refuses any Vitest outside `5.0.x`.
**Extended (Phase 1):**
- an optional `testIds` key in the driver scope, passed as the `vitest5.testIds` runner option. The dry run runs only the requested tests, checks the executed set against the request, and fails on an unknown id before any mutant run.
- per-test outcome classes from every sidecar attempt in `results.json` (`perTest`, `attemptsDisagree`).
- a per-mutant hook marker (`hookAttributedTests`): the tests a suite hook attributed the mutant to.
What upstream would need before the plugin can be dropped is listed in `gitnexus/tools/mutation/UPSTREAM.md`.
**Required behaviour:**
- per-test coverage (`perTest`), mutant activation per run, `disableBail`, static mutants (full re-run), hit limits and timeouts
- test filtering by test identity
- **per-test failure classification:** assertion failure vs runtime error vs hook failure vs timeout, so §2's kill definition can be enforced
- stable test identities including `it.each` rows
**Validation, before Layer 3 is trusted.** Validation is staged **per test category**: pure JS, native-backed, fixture-reading, LadybugDB, worker-backed. A category is out of scope until it passes all four gates.
1. **Execution-path fidelity and envelope.** For each category, compare a normal Vitest 5 run with the runner's unmutated Stryker run, including the prepared sources (blanked `as` assertions) and the `gitnexus-shared` alias. Both must agree on:
- collected test identities
- outcomes
- effective configuration
- resolved paths of production modules (inside the sandbox)
Record the validated envelope (config, environment, pool, aliases). Anything outside it is out of scope.
**Span map check:** for every mutant in the prepared copy, applying its replacement at the mapped span in the original source must give the same code, apart from the blanked `as` assertions.
**Phase 0 result:** pass on the pure-JS category, 246 files and 4,541 tests, 0 differences. The runtime load check found `commander` loading `child_process` at runtime; the classifier now rejects tests that import it. The span map check passed on 1,510 mutants with 0 failures.
**Phase 1 result:** after the runner extensions, gate 1 passes on 251 files and 4,667 tests, with 0 differences. The envelope was refreshed: the classifier now rejects `conformance-compare.test.ts` and `php-namespace-extraction.test.ts`, and accepts the three oracle unit tests.
2. **Runner conformance suite.** A small, committed fixture project pins expected per-test coverage and outcomes for:
- static and module-load mutants
- mutants executed in a suite's `beforeAll`
- several killers of one mutant under `disableBail`
- `it.each` row filtering by test identity
- mutant activation and reset between runs
- assertion vs hook vs runtime vs timeout failures
A mutant run that unexpectedly executes zero tests invalidates that mutant's evidence. Worker-category controls must prove the workers ran the instrumented sandbox build.
**Phase 0 result:** all 23 pinned cases hold in two profiles, about 4 min each: Stryker concurrency 2 with a fresh runner process per mutant, and concurrency 1 with unlimited runner reuse. Only this fixture exercises `beforeAll` hit attribution: the hook-hit populations of all three oracle scopes are empty.
**Phase 1 result:** the fixture gains a `testIds` selection scope and hook-marker cases. All 26 pinned cases hold.
3. **Vitest 5 cross-check oracle.** It is built entirely on Vitest 5, independently of the new runner. Every category gets its own frozen cross-check scopes before it is promoted:
- **Scopes (Phase 0):** the pure-JS category has three, because `shared.ts` alone exercises neither the blanking, the alias, `beforeAll` nor `it.each`:
- `gitnexus/src/core/ingestion/type-extractors/shared.ts`: 1,232 mutants, 143 tests, no hooks.
- `gitnexus-shared/src/lbug/schema-constants.ts` through `gitnexus/test/unit/schema.test.ts`: 72 mutants, all static, 42 tests. It covers the blanked `as const` tables, the alias and `it.each`.
- `gitnexus/src/server/middleware.ts` through `gitnexus/test/unit/middleware-public-origin.test.ts`: 302 mutants, 31 tests, with a `beforeAll`. The hook runs no mutated code.
- **Independence:** both oracle paths share only Stryker's instrumented sandbox. They never use the runner's test filtering, state, kill attribution or result conversion. Each run is a fresh process that installs one frozen mutant before any module import and starts clean, so nothing carries over between mutants. **Activation and execution are recorded separately:** the run must independently confirm that the intended mutant ID was installed (else it is invalid), and records whether the mutant executed. A correctly executed, passing test that never hits the mutant is a valid non-covering observation. A hit is required only where a control predicts coverage.
- **Per-mutant status:** Stryker's built-in `command` runner, which works on Vitest 5 but cannot attribute kills to tests, gives failed vs passed for every mutant. The runner's per-mutant outcome must agree on every mutant.
- Phase 0: one Stryker run cannot use two test runners, so the oracle runs **its own Stryker run** with the `command` runner and keeps its instrumented sandbox for the single-test launches. Runner and oracle share Stryker core (planning, instrumentation), the prepare step and the mutation config, and no runner code. The two runs' mutant inventories must be identical sets of global ids.
- On `shared.ts`, the command run agreed with the runner on all 1,232 mutant classes, and on the assertion kill sets of all 379 assertion-killed mutants.
- **Per-test kill sets:** each test in the scope runs alone with the mutant active, selected by `vitest run <file>:<line>` plus an escaped, anchored `-t "^<full name>$"`. A run must report exactly one executed test, with the intended identity; zero or several invalidate the observation. Tests that still cannot be told apart (identical expanded `it.each` titles, duplicate full titles) are listed as not oracle-validated, and their evidence caps at weak.
- **Outcome classes:** keep Vitest's JSON failure details and classify each failure as assertion, hook, runtime, timeout or infrastructure. Compare the runner and the oracle class by class, so the §2 assertion-only kill rule is checked, not just pass/fail.
- **Sample:** a seeded random 100 mutants, topped up so every stratum is present: static/module-load mutants, mutants executed in `beforeAll`, mutants with several killers, survivors and `it.each` tests. Zero differences in 100 random mutants bounds the difference rate at roughly 3% (95% upper bound). That is the claim, and it covers this category only.
- Phase 0: strata are counted across the three scopes combined. Within one scope, a stratum is required only when the scope's tests contain the construct and its population is not empty. Empty populations are recorded, never required.
- **Cost (Phase 0 benchmark):** one single-test launch takes about 3.3 s wall and 1.8 s CPU at concurrency 4, about 0.87 s of throughput.
- The sample is about 120 mutants (100 random plus top-ups) times the scope's tests: 17,160 launches for `shared.ts`, 3,024 for `schema-constants.ts` (all 72 mutants), 3,720 for `middleware.ts`. Each scope adds one command-runner pass.
- Projected on 4 vCPU: 6.18 h at best (5.52 h launches, 0.41 h command-runner passes, 0.25 h setup). That exceeds the 6 h job limit.
- So CI shards the freeze by scope: about 4.3 h for `shared.ts`, 44 min for `schema-constants.ts`, 56 min for `middleware.ts`. Comparing a runner run with the frozen manifests costs one Stryker run per scope.
- Gate 3 stays on demand (`workflow_dispatch`); it does not run on pull requests.
- **Freezing:** the oracle's source revision, configuration, operator set, test identities and global mutant identities are frozen with it.
- Phase 0 records content hashes instead of a revision, since hashes survive squash merges and shallow checkouts: the scope's import closure, the Vitest configs and the prepare tool, plus the Stryker and Vitest versions and the Node major version. Any mismatch fails with "re-freeze required".
- **Acceptance:** every supported observation must agree. The only allowed differences are an enumerated list of meaning-preserving representation normalisations. Phase 0 allows exactly four: every timeout row is one class, NoCoverage and survived are one class (the raw status stays in the report), title escaping, and path separators. Any execution, attribution or failure-class difference blocks validation **even when its cause is known**: fix it, or exclude the affected tests or category from the validated envelope.
- **Not covered by the oracle:** it shares Stryker's instrumentation, so it cannot vouch for instrumentation or the blanking span map. Gate 1 checks the span map separately.
- The spike's Vitest 4 results are history only, not an oracle.
- **Phase 0 result:** gate 3 passes on all three scopes (seed `u7-gate3-1`, frozen 2026-10-24 in `gitnexus/test/fixtures/mutation-oracle/`): every mutant class agrees (`shared.ts` 1,232, `schema-constants.ts` 72, `middleware.ts` 302; NoCoverage read as survived for 621, timeouts 5), and every sampled per-test observation agrees (14,300 + 3,024 + 3,131 = 20,455 over 273 sampled mutants), with 0 differences and 0 invalid launches. Inventories match (tests and global mutant ids). Strata across the scopes: static, multi-killer, survived, no-coverage and each-row met; hook-hit population empty. The freezes took 2 h 55 min, 1 h 25 min and 1 h 36 min locally at launch concurrency 8, 3 and 3.
- **Re-freeze (Phase 1, 2026-10-24):** merging `main` changed the import closure of the `schema-constants.ts` scope (`template-arguments.ts`), so gate 3 reported "re-freeze required". The manifest was re-frozen with seed `u7-gate3-2`: 72 sampled mutants, 3,024 valid observations, 0 invalid. Gate 3 passes on all three scopes again (1,232, 72 and 302 mutants), with 0 differences. The runner extensions changed no gate 3 hashed input, so they caused no re-freeze.
4. **Positive controls and determinism.** Per category, a fixed list of planted mutants must be killed by the tests that depend on them and must survive under tests that don't touch them. Two runs must give identical kill sets.
**Phase 0 result:** pass. Nine controls, three per oracle scope, picked from each run's own mutants by global id; three sit in relocated `gitnexus-shared` `as const` tables. The `shared.ts` scope uses lines 729-904 (199 mutants) to bound the run time. Profile A runs at Stryker concurrency 1 with unlimited runner reuse, profile B at concurrency 3 with a fresh runner process per mutant. Every control held in every run, and A and B gave 0 differences in outcome classes and kill sets. About 81 min locally.
**Phase 1 result:** after the runner extensions, gate 4 passes with 0 failures.
**Maintenance:**
- Pin the Vitest minor version that the runner supports. Done: the runner refuses anything outside `5.0.x`, and Stryker is pinned exactly.
- A CI check runs the conformance suite and the Vitest 5 cross-check oracle whenever Vitest, Stryker or the local runner changes, so an upgrade or edit cannot silently break kill evidence.
- Phase 0: `.github/workflows/mutation-runner-validation.yml` runs the conformance suite on pull requests that touch the tooling, its fixtures, `gitnexus/package.json`, the lockfile or `gitnexus/vitest.config.ts`. The oracle, gates 1 and 4 and the benchmark run on manual dispatch, because the oracle takes hours. The workflow is advisory.
- Offer the patch upstream to `stryker-mutator/stryker-js`. Drop the local copy only once the upstream runner passes the same four gates.
- Phase 0: upstream already merged the root-cause fix (#6214), so no PR was opened. The residual deltas are listed in `gitnexus/tools/mutation/UPSTREAM.md`.
### 5.5 Noise control and decisions
- **Tracking issue:** strong repeat candidates, weak tests and the per-file gap summary. Weak-evidence repeats, Review pairs and Inconclusive items live in the artifact report.
- **A human "keep"** is recorded with `// dupe-ok: <reason>`, as in Layer 0. Fold and remove decisions are recorded by the PRs that make them.
- **Two identities.** Execution identity is revision-pinned (§2) and kept as provenance. **Report continuity** uses stable anchors instead, so unrelated edits do not resurface findings:
- a test anchor: file, describe path, title, row index and a body fingerprint, without the declaration line. A moved test is matched by body fingerprint; an unmatched test is new.
- a mutant fingerprint: file, operator, replacement and the normalised enclosing code, without revision or span.
- **Week over week:** the report shows items that are new, **or whose evidence hash changed**, keyed by the continuity anchors. The hash uses fingerprints, not revisions or line positions, and covers:
- canonical path and provenance
- checks and context
- facet
- kills and universe identifier
- row-deletion outcome
- validity status and runner-validation status
- exclusions
- the computed verdict, its deciding rule or disqualifiers, and the verdict-engine version (so a rule change resurfaces every pair it reclassifies)
### 5.6 Verdict engine
Applies §2 with its precedence. Emits evidence for each pair and each test, including which rule or disqualifier decided the verdict. A pair has exactly one verdict.
## 6. Where it runs
| Surface | Layers | Trigger | Output |
| ------------- | -------------------------------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| PR | 0 | every PR | Sticky CI comment (exists) |
| Scheduled job | 0–2 and the scenario fingerprint on `gitnexus/test/unit` and `gitnexus/test/integration` (Layer 1 in 4 CI slices) | weekly (mutation `none`) | Artifact (JSON + markdown); one tracking issue updated in place |
| Manual run | as scheduled, plus 3 at the chosen mutation scope | workflow dispatch | As the scheduled job; mutation `possible-duplicates` or `all` adds kill sets, weak tests and gaps |
| Labelled PR | as scheduled but diff-scoped (pairs with a changed test file), plus 3 over the possible duplicates | `scenario-map` label | Artifact and job summary; no issue, no baseline |
| Local CLI | any | developer | Same report |
| Cleanup PR | 3 on the touched tests and their partners | cleanup PR author | **Removal check:** mutation run over the pair's common universe, before and after the removal. The remaining suite's kill set must be unchanged, and there must be no new inconclusive results. |
## 7. Constraints
- **Vitest version.** The repo uses Vitest 5. The stock Stryker runner supports only Vitest ≤4, so this effort builds the Vitest 5 runner (§5.4a). It is the critical path: Layer 3, and every strong verdict, waits for it.
- **Vitest 5 only**, the version pinned in `gitnexus/package.json`. No Vitest 4 anywhere: no harness, no oracle, no `--legacy-peer-deps`.
- The runner refuses anything outside Vitest `5.0.x`. A Vitest, Stryker or Node upgrade needs the four §5.4a gates again before its results count.
- **TypeScript 7:** the stable JS API is not ready, and Stryker's tsconfig preprocessor crashes, so point `tsconfigFile` at a non-existent file. Set `disableTypeChecks: false` (§5.4). The value-flow check uses the shipped unstable API (`typescript/unstable/sync`), pinned to the repo's TypeScript version and fail-closed (§5.3a).
- **Native code and fixtures** work as in normal test runs, because Stryker runs on the repo. The spike reached only pure-JS tests (170/894 unit files) through its copy harness, so integration-scale cost is still unmeasured (§5.4). Phase 0 validated the pure-JS category only: 246 unit files, chosen by a static classifier and checked by gate 1's runtime load check. Phase 1 refreshed that envelope to 251 files (§5.4a gate 1).
- **Compute:**
- Shard by candidate group (§5.4), and merge raw test–mutant observations by global identity (§5.4).
- Must fit `ubuntu-latest` (4 vCPU, 6 h) through sharding. Size the job from the §5.4 benchmarks, not by extrapolating.
- The gate 3 oracle's full pure-JS sample does not fit one job (6.18 h projected on 4 vCPU), so its freeze is sharded by scope (§5.4a).
- **Measured budgets** (12 workers, unit scope):
- about 65 min per 55k mutants without static mutants
- 12–14 min per 30k-mutant candidate pass with them
- about 54 min of coverage for 47 integration-heavy files
- **Measured Phase 0 times** (Vitest 5 runner; a loaded 24-CPU WSL2 machine, checkout on a 9p mount; CI will differ):
- `shared.ts`, 1,232 mutants with static mutants: 5 min 12 s at Stryker concurrency 4; the oracle's command-runner pass 18 min
- gate 1 over the pure-JS category: 2 min 39 s
- gate 2: about 4 min per profile
- gate 4: about 81 min for both profiles
- **Measured Phase 1 costs** (local runs; CI will differ):
- Layer 1: 4,667 tests in about 2 min (1 min 55 s)
- Layer 2: checks for 4,600 instances in 2.9 s; input facets for 4,667 instances in about 33 s
- Layer 3 main run: about 1.5 min fixed per group, plus about 13 min per MB of mutated code, at Stryker concurrency 2. The cost does not grow with the group's tests.
- isolation pass: about 130 s fixed per run, plus about 5.5 s per mutant (fresh process per mutant)
- row deletion: about 48 s per batch, plus 2.5–5 s per pair run
- whole-repo `analyze --pdg` for value flow: 6 min 49 s wall, 8.4 GB peak RSS
- trace run: 1.19× pooled overhead on 40 eligible files, with no outcome change
- controls fixture end to end: 16 of 16 pairs match, 28 min
- predicted CI run: 9 shards at Stryker concurrency 4, 1,658–1,829 min in total, largest shard 196–253 min (budget 300 min), 0 groups over budget
- **Caching:** key results on everything that changes them:
- test and production file contents
- fixtures read
- the evaluated mutant universe identifier
- the Vitest, Stryker and runner versions, and the operator set
- the dependency lockfile
- the source revision
- relevant env vars and runtime: Node version, OS, `GITNEXUS_*` settings
- the Stryker and Vitest config used for the run
- **Determinism:** stable report ordering. Any sampling (reruns, spot audits) uses a recorded seed.
## 8. Validation
**Labeled data**
- Commit the raw evidence of the 150 labeled pairs (5 sets) and the 20-pair mutation-scope set, frozen at commit `caacc3bac`:
- both test bodies
- describe paths
- fixture snippets and production table excerpts
- kill sets, for the mutation set
- Signatures are **re-derived** from this raw evidence in CI, so extraction regressions are caught. A full re-derivation at the pinned commit is a manual job.
- Rejected tools are not re-scored.
**Independent adjudication**
- Before any precision claim, a human who did not tune the rules relabels at least the four tuning sets, and all disagreements are resolved.
- Report agreement (Cohen's κ).
- **Strong-repeat precision** is measured on a seeded random sample of strong candidates from each scheduled run class (unit, integration): 50 per class, or all of them if fewer. Each is independently labeled by a human who did not tune the rules. The denominator is the sample size.
**Prospective holdouts**
- **Grouping:** holdouts are split by fixture and candidate group, so related pairs always land in the same set and cannot leak tuning information.
- **Size:** each fresh holdout has at least 30 distinct pairs and 30 repeat pairs. 30/30 gives a two-sided 95% lower bound of about 88%; 50/50 gives about 93%. These are bounds, not claims of near-zero risk.
- **Adversarial set:** pairs built to differ only where the facet or mutation layer is weakest. All must end in Review or Distinct:
- opaque built-ins: `Math.round` on `2.4`/`2.5`, `parseInt` radix, `toFixed`, `localeCompare`, Unicode `normalize`, `Date` boundaries
- regex alternatives
- Unicode
- numeric boundaries
- ordering
- native-parsed input
- worker-only code
- positive controls that must stay eligible: repo-only arithmetic (`add(1, 2)` vs `add(5, 5)`), membership in a supported `Set` table
- **Integration set:** pairs drawn once the runner reaches integration tests.
**Metrics per stage**, each with sample size and 95% CI:
- facet discrimination
- strong-repeat precision, adjudicated
- row/review split
- weak-test claim validity, audited as two separate questions:
- is the scoped mutation claim true? Reruns confirm no evaluated mutant is killed, and the share of likely-equivalent mutants is recorded.
- assertion quality, judged by a human. Meaningful assertions can kill no evaluated mutant, so this is reported separately.
**Gates for changing facet or verdict rules**
- Distinct recognition on a **fresh** untouched holdout stays at 100%. Each holdout gates one change and then retires into the tuning pool. The next gate draws a new holdout.
- **Paired precision check:** old and new rules run on the same fresh sample. Any false strong repeat that the new rule introduces blocks the change. The precision difference is reported with its 95% CI.
- Every new rule cites a newly labeled miss and is re-scored on a fresh holdout.
**Spot audit:** each scheduled report includes 10 strong repeat candidates, selected with a recorded seed, for human verification.
## 9. Rollout
Each category becomes eligible for Layer 3 only after it passes the §5.4a gates.
0. **Vitest 5 runner, pure-JS vertical slice (§5.4a):**
- diagnose and patch
- the `as const` pre-step with span mapping and its span map check, and the `gitnexus-shared` alias
- commit the conformance suite and the `shared.ts` cross-check oracle
- validate the **pure-JS** category only
- benchmark the oracle's fresh-process cost
- write the residual deltas against upstream in `gitnexus/tools/mutation/UPSTREAM.md` (upstream merged the root-cause fix, StrykerJS #6214; no PR is opened in Phase 0)
1. **Pure-JS unit tests:** Layers 1–3, the value-flow check, the kill-validity protocol, the report and the scheduled job.
- **Built 2026-10-25:** `gitnexus/tools/scenario-map/`, the `npm run test:scenario-map` CLI and `.github/workflows/test-scenario-map.yml`. The plan is `docs/plans/2026-10-24-0902-feat-test-scenario-map-phase1-plan.md`.
- The controls fixture (`gitnexus/test/fixtures/mutation-scenario-map/`) matches all 16 expected verdicts and rules.
- The first full run ran on PR #3686 through a temporary label-gated trigger (run 40183982111, 2026-10-25): complete, 9 of 9 shards, about 4 h. 4,632 eligible tests, 2,327 evaluated, 6,264 candidate pairs. All 11 strong repeat candidates call `extractElementTypeFromString` with the same input and expectation, in two files or under different titles.
- Verdict engine 1 left 4,532 pairs Inconclusive (`kills-unknown` 3,141, `non-assertion-outcome` 1,391): any unknown or runtime-class mutant decided the pair, even when the kill sets already differed. Engine 2 adds the kill witness. Re-merged from the same artifacts, Inconclusive falls to 1,469 (Distinct 1,053 to 3,952, Row 46 to 165, Weak 117 to 162). Every pair engine 1 decided keeps its verdict, rule and details, and the repeat candidates are unchanged.
- The remaining Inconclusive pairs have no witness: `kills-unknown` 977 (`missing-result` alone 675, all same-file pairs; timeouts alone 245), `non-assertion-outcome` 345, `member-inconclusive` 147. Every `missing-result` came from a static mutant whose test file failed to import (no per-test rows), not from a timeout; 966 of the 1,020 run-timeout mutants had a test past Vitest's 30 s testTimeout. Engine 3 records the import failure (§5.4). Its run (40208819019, complete, same 6,264 pairs) left 833 pairs Inconclusive (`non-assertion-outcome` 382, `kills-unknown` 302, of which 266 are timeouts, `member-inconclusive` 149). The 596 pairs import failures alone had blocked now reach the repeat stage: 595 are `csv-escaping.test.ts` table rows the trace vetoes to Review, because the arguments differ. Strong repeat candidates rose from 11 to 13. Both new ones were weak repeats that now had isolation, and both repeat a call with the same input and expectation. A timeout retry was tried and dropped: re-running the six timed-out mutants of one group locally resolved none (loop mutants that hang every time), and 214 of the 244 timeout-only pairs time out in both tests. Those pairs stay Inconclusive: a Stryker timer timeout marks every covered test, and a hung test can starve the next one, so a timeout of both tests does not show that the mutant cannot tell them apart. 2,305 eligible tests have no candidate partner, so they are never mutation-run; their Weak status is not evaluated.
- Scenario fingerprint (2026-10-25), measured locally (shared 24-core WSL2 host, concurrency 4, `nice -n 19`): the fingerprint-only run over all 4,632 eligible tests took 8.2 min (plan 5.4, of which Layer 1 3.0; trace 2.6; merge under a second; 2 GB peak RSS), against about 4 h and 30 runner-hours for the full mutation run. It names 26 duplicates, exactly the 26 pairs the evidence above predicts (the 13 strong repeats and the 13 mutation could not decide), 109 possible duplicates (all with an unresolved input) in 17 groups, 945 table rows and 5,184 distinct pairs. The controls fixture matches all 16 fingerprint expectations and all 16 mutation expectations. The deep run over the possible duplicates (the `scenario-map` label on PR #3686, run 40268120818) took 5 min 42 s on CI: plan 2.1, value flow 1.0, one shard of 5 groups, 25 tests and 844 mutants in 1.3, merge 0.5. Of the 109 possible duplicates, mutation settles 35 of the 38 it can reach: 31 are Row (the `embedding-recovery` table rows: kills differ), 1 Distinct, 3 weak-evidence repeats; 3 stay Inconclusive. The other 71 test `tools/mutation/` code, which is outside the mutation scope (Out of scope). That run's merge found no shard result: with one shard, `download-artifact` unpacks the only matching artifact into the path itself, and the placement step looked for a subdirectory; the merge above is a local re-merge of its artifacts, and the step now handles both layouts.
2. **Rest of the unit suite:**
- a frozen cross-check scope, then validation, for the **native-backed** and **fixture-reading** categories
- LadybugDB namespaces, then validation of the **LadybugDB** category
3. **Integration:**
1. Benchmark the three cases (§5.4).
2. Prove worker instrumentation and validate the **worker-backed** category.
3. Shortlist by candidate group and shard.
## 10. Decided
- Embeddings, vitest-dedup in CI and jscpd are out (measured, §4).
- PR-time stays Layer 0. The expensive layers run on a schedule or on demand.
- Strong evidence requires all of §2's conditions, including mutation evidence. Facet agreement alone is never enough, and nothing is removed without human review plus the removal check.
- Similarity of any kind is retrieval only.
- The canonical path for verdicts is mutant-granular `coveredBy`; V8 block coverage is for retrieval.
- Mutation runs on the repo itself on Vitest 5, through a Stryker runner built in this effort. Vitest 4 is not used anywhere, including validation.
- Scope: the `gitnexus/` CLI package. `gitnexus-web/` is deferred.
- The unit suite comes first.
- Data that differs but collapses to one facet is judged by semantic tooling: Babel, the TypeScript 7 checker, GitNexus PDG and a Vitest 5 trace run (§5.3a). It can only veto a repeat.
## 11. Left to the builder
- Package layout and language: TypeScript under `gitnexus/scripts/` or a separate tool directory.
- Report format details and tracking-issue formatting.
- Where the Vitest 5 runner lives (an in-repo tooling package, or a pinned fork) until upstream ships support. Also the cache implementation (cache keys are fixed in §7).
- The list of supported assertion forms, as long as it is documented and anything else is reported as unsupported.
- The exact list of wrapped built-ins in the trace run (§5.3a), as long as storage and table reads stay unwrapped and the list is documented.
Phase 1 choices: the tool lives in `gitnexus/tools/scenario-map/`. The supported matcher list and the wrapped built-ins list are documented in `TESTING.md` and in the headers of `facet/checks.ts` and `value-flow/builtins.ts`. There is no result cache yet: the report records the §7 cache-key inputs as run provenance.
## 12. Decisions log
Maintainer (2026-10-26):
- **Fingerprint scope:** the fingerprint covers every unit and integration test of `gitnexus/`; mutation stays on the validated pure-JS category. Layer 1 runs in CI slices (fingerprint plan E5, E6, E7).
Maintainer (2026-10-25):
- **Duplicate definition:** a duplicate is the same execution path, the same input data and the same expectation (the scenario fingerprint, §2). It is the primary verdict; mutation kill sets are optional evidence.
- **Where it runs:** weekly fingerprint only; mutation on manual dispatch; a pull request gets the deep run (fingerprint plus mutation over the possible duplicates) when the `scenario-map` label is added, not automatically.
- **Cost:** optimise the deep run as far as possible and measure locally, not on CI.
Maintainer (2026-10-23):
- **Tests outside mutation reach:** bring native-backed, fixture-reading and integration tests into mutation scope (§5.4), rather than adding an unconfirmed verdict or waiting. Delivered through the Vitest 5 runner on the repo itself, not a harness.
- **Vitest 5 only (2026-10-23):** use Vitest 5, as `gitnexus/` does, for the runner and its validation oracle. Vitest 4 is gone from the design.
- **CLI focus (2026-10-23):** this effort covers `gitnexus/` only. `gitnexus-web/` is deferred.
- **Semantic value flow (2026-10-23):** for data that collapses to one facet, use whatever gives real semantic information (Babel, TypeScript itself, the Vitest engine) rather than a blanket rule (§5.3a).
- **Vitest 5 runner (2026-10-23):** build a Stryker test runner for Vitest 5 as part of this effort, and run mutation on the repo itself. This supersedes the earlier inference of a Vitest 4 harness: no harness, no `--legacy-peer-deps`, no cross-version fidelity checks.
Independent review (Codex, 2026-10-23, round 1 verdict "rework"). All 11 findings were applied:
1. The repeat verdict is split into strong and weak evidence, and is never "confirmed" without a human.
2. Kill-validity protocol: assertion-only kills, reruns, isolation checks, Inconclusive.
3. Global mutant and test identities; comparisons only over a common universe; sharding by candidate group.
4. Similarity is used for retrieval only; the canonical path is decided; attribution rules are specified.
5. Explicit context equality, snapshot handling, supported assertion forms, row-deletion outcomes.
6. Evidence limits and per-stage metrics with confidence intervals; independent adjudication; prospective holdouts.
7. Differential Vitest 4/5 fidelity checks and resolution-path verification _(superseded: Vitest 5 only; fidelity is now checked per category in §5.4a)_.
8. Worker positive controls; out-of-reach code caps a test at weak evidence.
9. LadybugDB namespaces; three-case benchmark before scheduling.
10. Scoped wording for weak tests and gaps; verdict precedence; empty kill sets never form a repeat.
11. Raw-evidence re-derivation, complete cache keys, evidence-hash resurfacing, seeded audits.
Codex round 2 (2026-10-23, "rework": 5 resolved, 6 partially resolved, 4 new). Applied:
- Execution provenance enforces the exclusion cap (blocker).
- Supported table reads are defined for row deletion.
- Strong-repeat evaluation sample and holdout retirement.
- Fault-positive controls and the supported harness envelope.
- Full cache and evidence-hash inputs.
- Concurrent tests go to Inconclusive.
- Cross-shard subsumption eligibility.
- Weak-test validation split into a mutation-claim audit and an assertion-quality review.
- The Vitest 4 decision is marked pending _(superseded: Vitest 5 only, decided)_.
Codex round 3 (2026-10-23): 8 of 10 open items resolved, no new problems. Applied:
- Provenance completeness: the static-closure check always applies alongside V8, and other natives and workers stay capped.
- The resurfacing hash includes the verdict, deciding rules and verdict-engine version.
Codex round 4 (2026-10-23): **approve with changes**, both applied:
- The tree-sitter exception is removed as unsound: lossy shapes cannot vouch for native lexer and parser execution, and it contradicted §2. Native-dependent tests cap at weak evidence.
- The static-closure check also covers mutation-excluded files, since their module initialisation is invisible to V8.
Codex round 5 (2026-10-23), reviewing the Vitest 5 runner revision: **approve with changes**, all applied:
- per-category execution-path fidelity and envelope checks
- a runner conformance suite (coverage attribution, zero-test runs, failure classes, worker build)
- a frozen, mapped regression oracle with a blocking acceptance rule _(now the Vitest 5 cross-check oracle)_
- CI on runner changes
- rollout prerequisites reconciled: categories are validated in stages, and the `as const` pre-step and shared alias move to Phase 1
Codex round 6 (2026-10-23), reviewing the Vitest-5-only oracle and the CLI scope: **approve with changes**, all applied to gate 3:
- oracle independence from the runner's execution adapter; activation before import, proven per run
- exact test selection (`file:line` plus anchored `-t`, exactly one executed test); indistinguishable titles are not oracle-validated and cap at weak
- outcome classes compared, not just pass/fail; unexplained differences can never be accepted
- stratified sample on top of the random 100; the roughly 3% bound stated as the claim
- cost estimate, with a benchmark and CI budget before the check becomes mandatory
- CLI scope change: consistent; `gitnexus-shared/src` stays in scope as a CLI test dependency
Codex round 7 (2026-10-23, fresh full review, `gpt-6-astra` xhigh): **rework**, 1 blocker, 6 major, 2 minor. All applied:
1. Blocker, lossy facets: the value-flow check (§5.3a) vetoes pairs whose differing data reaches opaque code or flows unknown; the facet keeps the discarded values.
2. Full isolation pass over the pair's covered universe for strong evidence; sampled isolation caps at weak.
3. Oracle activation is confirmed separately from execution; non-covering runs are valid.
4. Known-cause defects block oracle acceptance; only enumerated normalisations pass.
5. Phase 0 is a pure-JS vertical slice with its prerequisites; per-category cross-check scopes; a separate span map check.
6. Shards merge raw observations; weak, gaps and subsumption are recomputed.
7. Holdouts are grouped by fixture and candidate group, with sizes and a paired precision check.
8. One strong-eligible provenance predicate; a validated worker envelope lifts only the worker disqualifier.
9. Report continuity anchors are separate from revision-pinned execution identity.
Settled by the author (reviewable):
- **Deletion guard:** tracked separately.
- **Report home:** artifact plus one tracking issue.
- §5 layer flow, §5.5 noise control, §7 sharding, §8 frozen labels.
Phase 1 implementation (2026-10-24/25, `docs/plans/2026-10-24-0902-feat-test-scenario-map-phase1-plan.md`). Deviations from this design, all applied in place above:
1. **Stitched value-flow walk (§5.3a step 3).** GitNexus `pdg_query` flows are intra-procedural. The check reads per-function REACHING_DEF and `CALL_SUMMARY` edges through LocalBackend's `cypher` tool. The TypeScript checker resolves callees, and AST argument positions map to parameters. Truncation, ambiguity and the depth cap of 4 are unknown. Repo code is clean only inside the pair's mutated files. Any opaque or unknown use in a statement vetoes it.
2. **Trace (§5.3a step 4).** It records receivers as well as arguments. It runs for every facet-equal candidate pair. Asymmetric call sites or call counts veto.
3. **Plain Vitest 5 outside Stryker (§5.4).** Baselines and row deletion run on plain Vitest 5 with a fresh instance per run. Row deletion cannot run through Stryker, because Stryker applies only its own mutators. It edits a throwaway copy of the prepared tree.
4. **Isolation pass (§5.4).** It selects tests through the runner's new `testIds` scope key and gives every mutant a fresh process. It runs only for pairs that can still be strong after value flow.
5. **Demote-only additions:**
- non-identical V8 block sets cap a pair at weak (§5.2);
- numeric literals keep `0`, `1`, `-1` and negatives exact (§5.3 rule 3);
- the ±1 numeric-constant rule, with constants from executed files and the static import closure and folded initialisers (§5.3);
- the residual-difference guard: a body difference the facet misses makes the facets unequal, so a pair with equal kills ends in Review (§5.3);
- excluded files: any executed repo file under `gitnexus/` that is neither a test nor mutated caps the test at weak (§5.4);
- the hook-stratum cap (§5.4);
- the trace on every facet-equal pair (§5.3a).
6. **Runner extensions (§5.4a).** `testIds` selection, per-test classes from every attempt and a per-mutant hook marker. Gates 1–4 pass after them, with no oracle re-freeze caused by them.
7. **Schema-constants re-freeze (§5.4a gate 3).** Merging `main` changed the scope's import closure. The manifest was re-frozen with seed `u7-gate3-2`, and gate 3 passes on all three scopes.
8. **CI topology (§6, §7).** Value flow runs before the mutation shards, not in parallel with them. The shard plan is rebuilt after value flow, so it counts isolation only for the pairs value flow leaves able to be strong. A group whose isolation does not fit still runs its main run and row deletion; its pairs get isolation not-run (weak cap). A group that executes no production file is not run, and its pairs are Out of scope.

992
docs/plans/2026-10-24-0902-feat-test-scenario-map-phase1-plan.md Normal file
View file

@ -0,0 +1,992 @@
---
title: Test Scenario Map, Phase 1 Pure-JS Report - Plan
type: feat
date: 2026-10-24
origin: docs/plans/2026-10-23-test-scenario-map-design.md
artifact_contract: ce-unified-plan/v1
artifact_readiness: implementation-ready
product_contract_source: ce-plan-bootstrap
execution: code
---
# Test Scenario Map, Phase 1 Pure-JS Report - Plan
## Goal Capsule
- **Objective:** maintainers get a weekly report on the validated pure-JS unit tests of `gitnexus/`. It lists tests that repeat a scenario, tests that kill no evaluated mutant, and per-file gaps. No strong repeat candidate merges two distinct scenarios without a flag, and the report never claims more than its evidence supports.
- **Means:** a four-layer pipeline in a new `gitnexus/tools/scenario-map/` (KTD1). It runs on the Phase 0 vitest5 runner, extended for single-test selection (KTD2). A fail-closed value-flow veto (KTD11) and a pure verdict engine (KTD12) sit on top.
- **Authority:** this plan's R-IDs and KTDs, then the origin design (`docs/plans/2026-10-23-test-scenario-map-design.md` §2, §5, §6, §7, §9 Phase 1), then the Phase 0 plan's KTDs for runner behaviour (`docs/plans/2026-10-23-1823-feat-stryker-vitest5-runner-phase0-plan.md`), then `AGENTS.md`, `GUARDRAILS.md`, `TESTING.md`.
- **Stop conditions:** stop and report in any of these cases:
- A runner extension breaks gate 2, 3 or 4 and cannot be fixed without weakening the gate.
- The controls fixture ends any adversarial pair as a repeat candidate.
- The eligible category cannot fit the 6 h job through sharding. Report the measured costs instead of narrowing the scope silently.
Never relax a strictness rule to make a pair strong.
- **Execution profile:**
- New tool code under `gitnexus/tools/scenario-map/`, with unit tests under `gitnexus/test/unit/scenario-map/`.
- A controls fixture under `gitnexus/test/fixtures/mutation-scenario-map/`.
- New pinned cases in `gitnexus/test/fixtures/mutation-conformance/`.
- Extensions to `gitnexus/tools/mutation/{runner,driver}` and `gitnexus/tools/test-dupes/test-dupes.ts`.
- The `test:scenario-map` script in `gitnexus/package.json`.
- One new workflow and one job in an existing workflow, plus docs.
- No change under `gitnexus/src`.
- No edit to any existing test of production code. The tooling's own tests (`gitnexus/test/unit/mutation/`, `gitnexus/test/unit/test-dupes.test.ts`) may gain cases.
- **Tail ownership:** PR #3686 (branch `feat/test-dupe-detector`) owns all of it. Phase 2 (the other test categories) consumes it.
---
## Product Contract
### Summary
Build the Phase 1 pipeline of the Test Scenario Map for the pure-JS unit category that Phase 0 validated:
- Layer 1: per-test V8 coverage retrieval.
- Layer 2: checks, context and input facet.
- Layer 3: per-test mutation evidence with the kill-validity protocol and the row-deletion operator.
- The value-flow check, the verdict engine, the report and one tracking issue.
- A weekly CI job.
The tool reports. It never changes a test.
### Problem Frame
Phase 0 made per-test kill evidence trustworthy on Vitest 5, but nothing turns that evidence into findings yet. Layer 0 (`test-dupes`) runs on every PR. It misses real repeats and cannot tell distinct scenarios apart. The spikes showed the layered method works: 57 confirmed pairs, 23/23 distinct scenarios recognised on an untouched holdout. But they ran as scratch scripts over Vitest 4 copies with hard-coded `/tmp` inputs. Without a repeatable pipeline the suite keeps its repeated scenarios, and nobody can tell which tests protect nothing.
### Key Decisions
- **Vitest 5 only** (session-settled: user-directed — chosen over a Vitest 4 harness or a cross-version oracle: the repo runs Vitest 5). Governs R6, R17.
- **Scope is the CLI package `gitnexus/`, with `gitnexus-shared/src` as a mutated dependency** (session-settled: user-directed — chosen over including `gitnexus-web/`: the maintainer deferred it). Governs R1.
- **The tool reports only, and CI is advisory** (session-settled: user-approved — chosen over automatic deletion or folding, or a blocking gate: losing protection is the top anti-goal). Governs R14, R15.
- **Phase 1 lands on PR #3686** (session-settled: user-directed — chosen over a new branch or a stacked PR: the user asked for it on this PR). Governs R15.
- **Similarity is retrieval only; the verdict path is mutant-granular `coveredBy`** (session-settled: user-approved — chosen over V8 coverage similarity as verdict evidence: measured run-to-run drift would produce false repeats). Governs R3, R10.
- **One artifact plus one tracking issue** (session-settled: user-approved — chosen over per-finding issues or PR comments: noise is an anti-goal). Governs R12, R13.
- **Only the pure-JS category validated in Phase 0 is eligible** (session-settled: user-approved — chosen over running native-backed, fixture-reading, LadybugDB or integration tests now: they have not passed the four gates). Governs R1.
- **Embeddings, vitest-dedup and jscpd stay out** (session-settled: user-approved — chosen over embedding similarity: AUC 0.69, and 0 of 16 sampled high-similarity pairs were redundant). Governs R3.
### Requirements
**Scope and identity**
- R1. Eligible tests are the files in the Phase 0 envelope (`gitnexus/test/fixtures/mutation-oracle/envelope.json`, `accepted.files`) that the current classifier still accepts. Every other test is Out of scope. The report lists the delta: newly accepted files not yet validated, and envelope files now rejected.
- R2. Every test instance has an execution identity and a continuity anchor (design §2, §5.5). A static-to-runtime join that is not one-to-one makes the test unsupported for strong evidence.
**Layer 1: candidate retrieval**
- R3. Per-test V8 block coverage, attributed per design §5.2, yields candidate pairs. A pair is a candidate at Jaccard ≥ 0.99, or when both tests have identical checks and context. Connected components schedule Layer 3 and never decide a verdict.
- R4. Each test records execution provenance: executed files, native-package entry points, worker spawns, and concurrent mode. Strong-eligible provenance follows design §2.
**Layer 2: checks, context, facet**
- R5. Each test instance records checks, context and input facet per design §2 and §5.3. These are computed per runtime instance (an `it.each`/`test.for` row, a loop iteration), and the discarded values are kept per position. An unresolved input or an unsupported assertion form blocks strong evidence. For two instances with equal facets, any body difference that is neither a facet part nor a recorded discarded value marks both instances unresolved.
**Layer 3: mutation evidence**
- R6. Per-test kill sets over the evaluated universe come from the Phase 0 runner with `perTest`, `disableBail` and static mutants included. Only the outcome class `killed-by-test:assertion` counts as a kill.
- R7. Kill validity follows design §5.4:
- two passing baselines with identical identities;
- agreement across every attempt;
- the full isolation pass for strong evidence.
A timeout, runtime, hook or infrastructure outcome on a mutant relevant to a pair makes the pair Inconclusive.
- R8. The row-deletion operator classifies a pair as symmetric, asymmetric, not applicable or inconclusive (design §5.4).
**Value flow**
- R9. A pair whose discarded values differ must pass the value-flow check (design §5.3a). A veto or an unknown sends the pair to Review and never yields a repeat verdict.
**Verdicts**
- R10. The verdict engine applies design §2 with its precedence. It gives exactly one verdict per pair, plus the rule or disqualifiers that decided it. It reports Weak tests, the Subsumed property and gaps, using the scoped wording of design §5.4.
- R11. Shards merge raw test–mutant observations by global identity. Verdicts, weak tests, gaps and subsumption are recomputed from the merged observations. A pair whose tests never ran together in one Layer 3 run is Inconclusive with the reason "not scheduled".
**Report and CI**
- R12. The report is one JSON artifact plus a markdown rendering. It carries:
- every verdict's evidence;
- the run's provenance and limits;
- a seeded spot audit of 10 strong candidates;
- new, changed and resolved items against the last complete run.
- R13. One tracking issue holds strong repeat candidates, weak tests and per-file gaps. Only complete runs from the default branch update it in place.
- R14. A weekly and manually dispatched GitHub Actions job runs Layers 0–3 on the eligible tests. Through sharding, it fits `ubuntu-latest` limits. It never blocks a merge.
- R15. The tool never edits, deletes or folds a test. A `// dupe-ok: <reason>` comment on either member of a pair records a human keep (design §5.5).
- R16. A local CLI produces the same report as CI.
**Runner integrity**
- R17. After the Phase 0 runner extensions, gates 1, 2, 3 and 4 still pass.
### Success Criteria
- On the controls fixture, every pure-JS adversarial pair of the design §8 set ends in Review or Distinct, and every positive control stays eligible for strong evidence. The verbatim duplicate becomes a strong repeat candidate, and the object-key pair becomes a Row. The native-parsed and worker-only adversarial pairs wait for the phases that make those categories eligible (design §9 Phases 2–3).
- A complete local run over the eligible category writes the report with no unexplained error. Its Inconclusive count and causes are recorded in the PR.
- Gates 1, 2, 3 and 4 pass after the runner extensions.
- The new workflow passes actionlint, zizmor and the repo's concurrency check.
### Scope Boundaries
- Native-backed, fixture-reading, LadybugDB and worker-backed test categories are out of scope until they pass the Phase 0 gates (design §9 Phases 2–3).
- `gitnexus-web/` and the Python `eval/` tests are out of scope.
- No precision claim. The report says that strong-repeat precision is not yet measured (design §8).
#### Deferred to Follow-Up Work
- The removal check for cleanup PRs (design §6, "Cleanup PR" row).
- Design §8 labeled-data commit, independent adjudication, prospective holdouts and the paired precision check.
- A result cache keyed on the design §7 inputs. Phase 1 records those inputs as provenance only (KTD15).
- An optional whole-suite mutation run with static mutants.
- Equivalence rules that lift a value-flow veto. The list starts empty (design §5.3a).
- Upstreaming the runner's `testIds` scope extension to StrykerJS. Add it to `gitnexus/tools/mutation/UPSTREAM.md` only.
---
## Planning Contract
### Key Technical Decisions
- KTD1. **New tool package `gitnexus/tools/scenario-map/`, one CLI with stages.**
- The CLI imports Phase 0 modules from `gitnexus/tools/mutation/` and adds no new dependency. Already installed: Babel 8 parser/types/traverse, TypeScript 7.0.2 `typescript/unstable/sync`, the tree-sitter loader in `gitnexus/src/core/tree-sitter/parser-loader.ts`, the Vitest node API, and Stryker 10.
- Stages: `plan`, `shard`, `value-flow`, `merge`, `all`, `controls`. They map one-to-one onto CI jobs, so a local `all` run and CI produce the same report (R16).
- KTD2. **Mutation evidence runs through the Phase 0 vitest5 runner, with three extensions** (session-settled: user-approved — chosen over the stock `@stryker-mutator/vitest-runner`: it runs 0 tests per mutant on Vitest 5). The extensions:
1. An optional `testIds` key in the driver scope (`driver/stryker-config.ts`), passed to the plugin as a `vitest5.testIds` runner option in the generated Stryker config. The dry run first collects the scope files, assigns Phase 0 ids with `assignTestIds`, and maps the requested ids to Vitest task ids. It then runs one `createSpecification(file, { testIds })` per file and checks the executed set against the request.
2. Per-test outcome classes from every sidecar attempt in `results.json`.
3. A per-test marker for mutants attributed through a suite hook.
The extensions never touch gate 3's hashed inputs (`tools/mutation/scope-file.ts`, `tools/mutation/vitest.mutation.config.ts`, `tools/mutation/prepare/*`). A change there forces a re-freeze of all three oracle manifests, which took about 6 h.
Baselines, row deletion and the trace run use plain Vitest 5 through `createVitest` and `tools/mutation/vitest.mutation.config.ts`, as in `tools/mutation/fidelity/fidelity.ts`.
**Conflict call-out:** the brief says mutation runs through the Phase 0 runner. Row deletion cannot, because Stryker applies only its own mutators. It runs on plain Vitest 5 in a throwaway copy of the prepared tree, which still honours Vitest 5 only. The isolation pass also needs extension 1 before it can select single tests.
- KTD3. **Identities.**
- The execution identity is the Phase 0 test id (`tools/mutation/runner/test-ids.ts`) plus the manifest location plus the row ordinal.
- The continuity anchor (design §5.5) comes from static extraction: file, describe path, title, a row fingerprint over the row's values, and a body fingerprint. A body fingerprint matches a moved test only when it is unique. Identical bodies are exactly the repeats and the rows of one `it.each`.
- KTD4. **Layer 1 runs in a prepared copy under the mutation config.** That way `gitnexus-shared` executes from source and every path maps back through `tools/mutation/prepare/path-map.ts`.
- The coverage hook ports `gitnexus/test/spike-coverage.setup.ts` to the runner's hook pattern: file-level `aroundAll` with the `{}` parameter, and `TestRunner.getCurrentTest()`.
- It records the task id, full name and location so that `assignTestIds` produces Phase 0 ids.
- Block ids are keyed by repo path.
- KTD5. **Candidate retrieval uses exact Jaccard with size and prefix filters**, plus buckets of identical checks-and-context hashes.
- A pair whose V8 block sets are not identical caps at weak. This is stricter than the design, and it can only demote.
- It closes the hole where one test reaches extra mutant-free code that the 0.99 tolerance hides.
- KTD6. **Layer 2 reuses Layer 0's AST walk.**
- `tools/test-dupes/test-dupes.ts` exports its case extraction, context closure and `dupe-ok` lookup. Layer 0's PR output must not change.
- Checks support a documented matcher list in `tools/scenario-map/facet/checks.ts`. Any other form is unsupported.
- The input facet ports `gitnexus/spike-input-facet.mts`. It covers rules 1–8, with rule 6 (source order) taken from the spike's `analyze-slice.mjs`. It adds numeric literals, per-instance evaluation of `each` tables and static loops, discarded values per position, and an `unresolved` marker.
- A numeric literal within ±1 of a production numeric constant stays exact. Stryker has no numeric-literal mutator, so a boundary value such as `LIMIT + 1` would otherwise collapse with an interior value. Constants come from the test's executed production files and its static import closure, because a constants-only module runs only during the discarded module load. Statically evaluable initialisers such as `2 * 1024` count, through Babel's `evaluate()`.
- Residual-difference guard (R5): for two instances with equal facets, diff the instance body ASTs. Any differing node that is neither a facet part nor a recorded discarded value marks both instances unresolved. This catches pairs that differ only in an enum member, a regex, a boolean or `null`, or an operator.
- KTD7. **The mutate scope of a candidate group is whole production files.**
- Production files are defined once, in `tools/scenario-map/types.ts`, as path-map-mapped files outside test files. That means `gitnexus/src/`, relocated `gitnexus-shared/src/` and, for an extra root, its `src/` subtree. The Layer 1 block filter, provenance, the mutate scope and the trace frame filter all use this one definition.
- The mutate scope is every production file in either test's V8 provenance or static import closure, with no line ranges.
- An excluded file is any executed repo file under `gitnexus/` that is neither a test file (`test/**`, `*.case.ts`) nor mutated, such as `tools/` or `scripts/` code. It caps the test at weak (design §5.4 worker threads).
- The mutated universe is never truncated silently.
- KTD8. **Kill validity mechanics.**
- Two plain Vitest 5 baseline runs of the group's tests must pass with identical identities, and so must Stryker's dry run. Otherwise the group is Inconclusive.
- Observations come from every sidecar attempt, not the latest. Any class disagreement between attempts or runs is inconclusive.
- A missing result is unknown, never a non-kill (design §5.4).
- Every generated Stryker config must carry `perTest` coverage, `disableBail` and static mutants. The run fails before mutating if one is missing.
- A test with no assertion kills but some runtime, hook or timeout kills is per-test Inconclusive, never Weak.
- KTD9. **The full isolation pass runs per pair, with alone runs shared.**
- It applies to each pair that reaches the repeat stage. That means one Stryker run with `testIds` set to both tests, and one run per test alone.
- Mutate is restricted to the pair's covered universe: Stryker ranges, then filtered by global id.
- `--max-test-runner-reuse 1` gives each mutant a fresh process with modules reset (design §5.4).
- Coverage and outcome class per test and mutant must agree across the main, together and alone runs.
- A pair whose path or kills touch a hook-attributed mutant caps at weak. The cap holds until a gate 3 scope with a non-empty hook stratum passes; today that stratum is empty in all three scopes.
- KTD10. **Row deletion.**
- Supported tables follow design §5.4. For each of the two differing keys in turn, the key is spliced out of the source text in a throwaway copy of the prepared tree.
- A control run comes first: both tests must pass, and V8 or `evaluatedModules` must show that the edited file was loaded.
- Outcomes are classified with `tools/mutation/runner/classify-outcome.ts`.
- Each edited run uses a fresh Vitest instance, because a reused instance serves cached transforms and would turn a no-op deletion into a false "neither fails".
- The global id comes from `formatGlobalId`, with mutator name `RowDeletion` and an empty replacement.
- KTD11. **Value-flow check** (session-settled: user-directed — chosen over a blanket rule that sends every value-differing pair to Review or ignores the discarded values: the user asked for real semantic information). The four sources of design §5.3a must all come back clean. Local deltas:
- **Test side (Babel).** A differing expected value with no differing bound input is unknown.
- **Checker (TypeScript).** The checker takes its tsconfig path as an input: `tsconfig.test.json` by default, the fixture's own tsconfig for `controls`. It refuses to run unless `typescript` resolves to exactly 7.0.2. A version mismatch marks the whole run degraded, like a missing PDG layer, because Dependabot can bump the lockfile. Any other API failure is unknown for that pair.
- **PDG.** The layer comes from `gitnexus analyze --pdg`, run with the checkout's own build. If it is missing or unreadable, the run is marked degraded, and every value-differing pair is vetoed with the reason "no PDG layer". Every degraded flag appears in the report's completeness block and in the issue header.
- **Trace.** It records the receiver (`this`) as well as the arguments of every wrapped call. It vetoes when the receiver or any argument differs at a shared call site, and on asymmetric sets or counts of opaque call sites. Production frames are recognised through the KTD7 definition. The trace runs for every facet-equal candidate pair, not only value-differing ones. This is a cheap backstop for differences that the facet cannot see.
**Conflict call-out:** GitNexus `pdg_query` flows are intra-procedural (`gitnexus/src/mcp/tools.ts`, `_pdgQueryImpl` in `gitnexus/src/mcp/local/local-backend.ts`). Its output also loses the exact def/use lines, caps every file-wide query at 200 edges, and never exposes `CALL_SUMMARY`. No edge maps an argument to a parameter. Step 3 therefore stitches the walk itself:
1. It reads REACHING_DEF and `CALL_SUMMARY` edges through LocalBackend's `cypher` tool, scoped to the function's `BasicBlock:<file>:<fnLine>:` id prefix. It decodes the def/use line pairs with `decodeReachingDefReason`.
2. It classifies each use at the consuming expression. Any opaque or unknown use in a statement vetoes the whole statement.
3. A new binding derived from the value (`const { ratio } = opts`, `this.x = v`) is walked under its own name, or marked unknown.
4. Repo code is a clean sink only inside the pair's mutated production files (KTD7).
5. The checker resolves each callee, and the argument position in the production AST gives the parameter. `CALL_SUMMARY` covers return values.
6. Truncation of the per-function query, ambiguity or a depth cap is unknown.
The check stays fail-closed.
- KTD12. **The verdict engine is a pure function over merged observations.**
- It applies the value-flow veto as a gate before any repeat category, because design §2 ranks Repeat (weak) above Review.
- It carries a `VERDICT_ENGINE_VERSION` constant into the evidence hash.
- It reads unscheduled pairs as Inconclusive.
- KTD13. **Report identity.**
- The universe identifier of a pair is a hash of the sorted mutant fingerprints of its common universe. Regrouping alone does not resurface a pair.
- The evidence hash follows design §5.5.
- The baseline is a dedicated artifact, `scenario-map-baseline`. The merge job uploads it only for complete default-branch runs, with `retention-days: 90`. The next merge job takes the newest artifact of that name through the Actions artifacts REST API, excluding its own run. A run that finds none shows a "first run, no baseline" banner. A baseline with a different report schema version shows an "incompatible baseline" banner, and no delta is computed.
- The issue body records the run id it came from. An upsert skips the update when the issue already records a newer run.
- The spot audit picks 10 strong candidates with a recorded seed.
- KTD14. **CI topology.** A new workflow, `.github/workflows/test-scenario-map.yml`, runs on a weekly cron and on dispatch. One workflow-wide concurrency group with `cancel-in-progress: false` serialises weekly and manual runs. Jobs:
1. `plan`: Layer 0 scan over the eligible files, classify, prepare, Layers 1–2, candidates, shard plan and the run manifest.
2. Mutation shards: a matrix with `fail-fast: false`.
3. `value-flow`, in parallel with the shards: static check and trace run for the facet-equal candidate pairs.
4. `merge`, with `if: always()`.
5. `upsert-issue`: default branch only, complete runs only. A partial run adds a comment and leaves the body alone.
Every job is `continue-on-error`, with a timeout and SHA-pinned actions. Artifact names carry the run attempt.
The run manifest from `plan` holds the eligible tests, their ids and the prepared inputs' content hashes. Each shard and the `value-flow` job verify their own prepared copy against it. A mismatch makes that job's results invalid, and its groups' pairs Inconclusive.
The controls run is a PR job in `.github/workflows/mutation-runner-validation.yml`, with `build: 'true'`. It is path-gated on `gitnexus/tools/scenario-map/**`, `gitnexus/tools/mutation/**`, `gitnexus/tools/test-dupes/**` and the fixture. The workflow's paths are workflow-level, so scenario-map changes also run the gate 2 conformance jobs, which is an accepted extra cost.
- KTD15. **No result cache in Phase 1.** The report records the design §7 cache-key inputs as run provenance, so a later cache has its keys.
- KTD16. **Runner extensions re-run gates 1, 2, 3 and 4.** Gate 1 covers the changed dry run. Gate 2 gains pinned cases for `testIds` selection, per-test classes across attempts, and the hook marker.
- KTD17. **Shard planning is deterministic.**
- A candidate group is never split across shards, so every pair runs in one run.
- Groups are bin-packed by estimated cost, ordered by group id. The estimate is a fixed per-run overhead, plus mutated file bytes times group tests, plus an isolation term: candidate pairs × covered mutants × 3 runs at the measured fresh-process cost.
- A group whose estimate exceeds one shard's budget is not run. Its pairs are Inconclusive with reason `group-over-budget`, and the report lists the group.
- A shard starts a group only when its estimate fits the remaining time before a 300-minute deadline. A group it never ran leaves its pairs Inconclusive (not scheduled, R11).
- Each finished group's observations are written to the work directory at once and uploaded with `if: always()`.
### High-Level Technical Design
Pipeline data flow:
```mermaid
flowchart TB
E[Eligible tests: envelope intersect classifier] --> S[Static extraction per runtime instance]
E --> P[Prepared copy]
P --> L1[Layer 1: V8 per-test blocks + provenance]
S --> L2[Layer 2: checks, context, facet, discarded values]
L1 --> C[Candidate pairs + groups]
L2 --> C
C --> SH[Shard plan]
SH --> L3[Layer 3 per group: baselines, Stryker perTest run]
L3 --> ISO[Isolation pass for repeat-stage pairs]
L3 --> RD[Row deletion for key-table pairs]
C --> VF[Value-flow: Babel, checker, PDG walk, trace]
L3 --> M[Merge raw observations]
ISO --> M
RD --> M
VF --> M
M --> V[Verdict engine]
V --> R[Report JSON + markdown + issue body]
```
Verdict decision for one pair (design §2 precedence, KTD12 veto gate):
```mermaid
flowchart TB
A{Any inconclusive cause?} -->|yes| I[Inconclusive]
A -->|no| B{Both tests eligible and covering evaluated mutants?}
B -->|no| O[Out of scope]
B -->|yes| W{Either test kills nothing?}
W -->|yes| WK[Weak per test; pair Distinct]
W -->|no| PC{Same coveredBy path, checks, context?}
PC -->|no| D[Distinct]
PC -->|yes| F{Same facet?}
F -->|no| K1{Same kill set?}
K1 -->|yes| RV[Review]
K1 -->|no| D
F -->|yes| K2{Same kill set and row deletion symmetric or n/a?}
K2 -->|no| RW[Row]
K2 -->|yes| TR{Trace clean?}
TR -->|veto| RV
TR -->|clean| DV{Discarded values differ?}
DV -->|no| DQ
DV -->|yes| VG{Static value-flow clean?}
VG -->|veto or unknown| RV
VG -->|clean| DQ{Any weak disqualifier?}
DQ -->|yes| RWK[Repeat, weak evidence]
DQ -->|no| RS[Repeat, strong evidence]
```
Directional data sketch of one raw observation, the unit that shards merge (R11):
```text
Observation {
run: { kind: main | together | alone-A | alone-B, group, shard, attempt }
test: executionId, mutant: globalId, mutantFingerprint
covered: boolean, hookAttributed: boolean, static: boolean
outcome: OutcomeClass // from runner/classify-outcome.ts
valid: boolean, reason?
}
```
Value-flow check for one value-differing pair (KTD11). The trace step also runs alone for facet-equal pairs whose discarded values are equal:
```mermaid
flowchart TB
B1[Babel: bind each differing value to an entry call argument] -->|unbound or expected-only| U[unknown: veto]
B1 --> T1[Checker: resolve entry through aliases, read parameter types]
T1 -->|any, unknown, unresolved, version mismatch| U
T1 --> P1[PDG walk: per-function flows, stitched across calls]
P1 -->|opaque sink or regex| X[veto]
P1 -->|truncated, dynamic, no layer, depth cap| U
P1 --> TR[Trace run: wrapped built-ins + opaque sinks]
TR -->|different receiver or args at a shared site, asymmetric sites, outcome changed| X
TR --> OK[clean]
```
CI job topology (KTD14):
```mermaid
flowchart TB
PL[plan: Layer 0, classify, prepare, Layer 1, Layer 2, candidates, shard plan, run manifest] --> SH1[shard 1..N: Layer 3, isolation, row deletion]
PL --> VF[value-flow: build, analyze --pdg, static check, trace]
SH1 --> MG[merge if always: verdicts, report, completeness]
VF --> MG
MG --> UP{default branch and complete?}
UP -->|yes| IS[update issue body]
UP -->|no, default branch| CM[comment: partial run]
```
### Output Structure
```text
gitnexus/tools/scenario-map/
scenario-map.ts CLI and stage orchestration
types.ts shared record types and schema version
inventory/ eligible set, static case extraction, identity join, anchors
coverage/ Layer 1 setup file, coverage run, candidate retrieval
facet/ checks, context, input facet, production tables
mutation/ shard plan, group runs, observations, isolation, row deletion, plain Vitest runs
value-flow/ Babel binding, checker, PDG walk, trace setup and run, wrapped built-ins
verdict/ verdict engine
report/ JSON, markdown, issue body, baseline diff, spot audit
gitnexus/test/unit/scenario-map/ unit tests per module
gitnexus/test/fixtures/mutation-scenario-map/ controls fixture project and expected verdicts
.github/workflows/test-scenario-map.yml
```
### Assumptions
These are unconfirmed bets made in pipeline mode.
- The weekly cron runs Monday 04:17 UTC, outside the busy CI hours.
- The tracking issue is never closed automatically. It is found by label in any state and updated in place. A complete run with no findings says so in the body.
- `gitnexus analyze --pdg` on this repo fits a 4 vCPU job well under its timeout. Measure it in U10. If it does not fit, the run is marked degraded under KTD11.
- Eight mutation shards are enough for the eligible category. U7 measures the cost and adjusts the count.
- The ±1 numeric-constant rule and residual-difference guard (KTD6), the block-identity cap (KTD5), the excluded-file definition (KTD7), the hook-stratum cap (KTD9) and the trace on every facet-equal pair (KTD11) are additions that only demote. They are not in the design text; U16 records them there.
- Running `together` per pair, with `alone` runs shared across pairs, satisfies design §5.4's isolation pass.
### Risks and Dependencies
| Risk | Mitigation |
| --- | --- |
| Whole-file mutate scopes (KTD7) make static mutants expensive: each one re-runs every selected test | Selecting group tests with `testIds` bounds the run; U7 measures per-group cost. Narrowing a scope only adds weak caps, never false strong results |
| The TypeScript unstable API changes | Exact version guard; any failure is unknown (KTD11) |
| The PDG layer is slow or missing in CI | Degraded run with an explicit veto reason; never silent (KTD11) |
| A high Inconclusive rate leaves few strong verdicts | Report the rate and causes; this is the intended price of the top anti-goal |
| Runner extensions regress Phase 0 evidence | Gates 1, 2, 3 and 4 re-run (KTD16, R17); gate 3's hashed inputs stay untouched (KTD2) |
| The issue body exceeds GitHub's 65,536-character limit | Truncate at 60,000 characters in a fixed order, with counts and a run link (U13) |
| A candidate group chains into a component too large for one shard | `group-over-budget` Inconclusive with the group listed (KTD17); the largest group size is measured in U4 |
| Wrapping string built-ins slows string-heavy tests into trace timeouts | A changed outcome under tracing is Inconclusive, never clean; U11 measures the overhead on the eligible category |
| A dispatch from a feature branch overwrites main's findings | Issue upsert only from the default branch (KTD14) |
| Scheduled and dispatch workflows cannot run from this PR branch before merge | The local `all` run and the PR-gated controls job carry the proof; the first scheduled run follows the merge |
### System-Wide Impact
- **Phase 0 runner and gates:** U1 changes `tools/mutation/driver/stryker-config.ts`, `runner/plugin.ts`, `runner/coverage-attribution.ts`, `runner/setup.ts`, `runner/protocol.ts`, `runner/sidecar.ts` and `driver/normalise-results.ts`. These are the gate 1–4 surfaces, re-validated by KTD16.
- **Layer 0 PR job:** U2 changes only exports of `tools/test-dupes/test-dupes.ts`. Its findings on the repo must stay byte-identical.
- **CI load:** one weekly job of up to about 6 h wall time across shards, plus a short path-gated PR job.
- **Contributors:** the tracking issue changes no workflow. A human keep is still a `// dupe-ok:` comment.
### Sources and Research
- Phase 0 outputs:
- `results.json` schema: `gitnexus/tools/mutation/driver/normalise-results.ts` (`RunResults`, `NormalisedMutant`, `NormalisedTest`).
- Outcome classes: `gitnexus/tools/mutation/runner/classify-outcome.ts`.
- Scope keys, which reject unknown keys today: `gitnexus/tools/mutation/driver/stryker-config.ts`.
- Dry-run spec globbing: `runner/plugin.ts`.
- Hook-hit folding: `runner/coverage-attribution.ts` (`attributeHookHits`).
- Per-test selection pattern: `project.createSpecification(file, { testIds })` in `runner/plugin.ts` mutant runs, documented in `gitnexus/tools/mutation/UPSTREAM.md`.
- Plain Vitest 5 runs: `createVitest` plus `runTestSpecifications` in `gitnexus/tools/mutation/fidelity/fidelity.ts`.
- Identity helpers:
- `gitnexus/tools/mutation/prepare/mutant-identity.ts` (`formatGlobalId`, `contentHash`);
- `prepare/path-map.ts` (`toRepoPath`, `toPreparedPath`, `buildPathMap`, which accepts only `gitnexus/test/fixtures/mutation-<name>/` extra roots).
- Spikes to port, which hard-code `/tmp` and Vitest 4 paths:
- `gitnexus/test/spike-coverage.setup.ts`;
- `gitnexus/spike-input-facet.mts`;
- `/tmp/test-audit/spike5/analyze-slice.mjs` (grouping, subsumption, rows);
- `/tmp/test-audit/spike5/row-delete.mjs` (object tables only, regenerates files: do not port as-is).
- PDG access:
- `LocalBackend.callTool('pdg_query', …)`, with the script pattern in `gitnexus/bench/impact-pdg/real-code.mjs`.
- Layer status: `pdgLayerStatus` in `gitnexus/src/mcp/local/pdg-impact.ts`.
- There is no CLI equivalent. No local index has a PDG layer today.
- TypeScript checker: the probe at `/tmp/test-audit/tsprobe/probe2.mjs`. `getAliasedSymbol` and `getResolvedSignature` exist in the API typings but were not probed.
- Issue upsert template: `.github/workflows/tree-sitter-upgrade-readiness.yml` (`actions/github-script`, a separate `issues: write` job).
- Advisory workflow shape: `.github/workflows/mutation-runner-validation.yml`. Nightly `--pdg` precedent: `.github/workflows/impact-pdg-mutation-report.yml`.
- External research was skipped. Local patterns are strong: Phase 0 tooling, the spikes and the probes.
---
## Implementation Units
| U-ID | Title | Key files | Depends on |
| --- | --- | --- | --- |
| U1 | Runner single-test selection and per-test classes | `tools/mutation/driver/stryker-config.ts`, `runner/plugin.ts`, `runner/coverage-attribution.ts`, `runner/protocol.ts`, `runner/sidecar.ts`, `driver/normalise-results.ts` | none |
| U2 | Layer 0 extraction API | `tools/test-dupes/test-dupes.ts` | none |
| U3 | Inventory, identity and anchors | `tools/scenario-map/inventory/*`, `tools/scenario-map/types.ts` | U2 |
| U17 | Controls fixture project | `test/fixtures/mutation-scenario-map/*` | none |
| U5 | Layer 2 checks and context | `tools/scenario-map/facet/checks.ts`, `facet/context.ts` | U2, U3 |
| U4 | Layer 1 coverage and candidate retrieval | `tools/scenario-map/coverage/*` | U3, U5 |
| U6 | Layer 2 input facet | `tools/scenario-map/facet/input-facet.ts`, `facet/tables.ts` | U3, U4 |
| U7 | Layer 3 group runs and observations | `tools/scenario-map/mutation/*` | U1, U3, U4, U17 |
| U8 | Full isolation pass | `tools/scenario-map/mutation/isolation.ts` | U1, U5, U6, U7, U17 |
| U9 | Row-deletion operator | `tools/scenario-map/mutation/row-deletion.ts` | U6, U7, U17 |
| U10 | Value-flow static check | `tools/scenario-map/value-flow/{bind,checker,pdg-walk,sinks}.ts` | U5, U6, U17 |
| U11 | Value-flow trace run | `tools/scenario-map/value-flow/{trace.setup,trace,builtins}.ts` | U4, U10, U17 |
| U12 | Verdict engine | `tools/scenario-map/verdict/verdict.ts` | U5–U11 |
| U13 | Report and issue body | `tools/scenario-map/report/*` | U12 |
| U14 | CLI orchestration and controls run | `tools/scenario-map/scenario-map.ts` | U1–U13, U17 |
| U15 | Scheduled workflow and PR controls job | `.github/workflows/test-scenario-map.yml`, `mutation-runner-validation.yml` | U14 |
| U16 | Documentation and design corrections | design doc, `TESTING.md`, `UPSTREAM.md` | U14 |
All `tools/` and `test/` paths are under `gitnexus/`.
### U1. Runner single-test selection and per-test classes
- **Goal:** the Phase 0 runner can run a chosen subset of tests, and `results.json` carries every per-test class and the hook marker that Phase 1 needs.
- **Requirements:** R6, R7, R17. KTD2, KTD9, KTD16.
- **Dependencies:** none.
- **Files:**
- Modify:
- `gitnexus/tools/mutation/driver/stryker-config.ts`
- `gitnexus/tools/mutation/runner/plugin.ts`
- `gitnexus/tools/mutation/runner/coverage-attribution.ts`
- `gitnexus/tools/mutation/runner/setup.ts`
- `gitnexus/tools/mutation/runner/protocol.ts` (`FileMeta` carries hook-attributed tests per mutant)
- `gitnexus/tools/mutation/runner/sidecar.ts` (the dry-run record carries them; bump `SIDECAR_SCHEMA_VERSION`)
- `gitnexus/tools/mutation/driver/normalise-results.ts`
- `gitnexus/tools/mutation/UPSTREAM.md`
- Never modify: `gitnexus/tools/mutation/scope-file.ts`, `gitnexus/tools/mutation/vitest.mutation.config.ts`, `gitnexus/tools/mutation/prepare/*` (gate 3 hashed inputs, KTD2).
- Fixture: `gitnexus/test/fixtures/mutation-conformance/` (new `*.case.ts` and `expected.json` entries).
- Tests:
- `gitnexus/test/unit/mutation/stryker-config.test.ts`
- `gitnexus/test/unit/mutation/normalise-results.test.ts`
- `gitnexus/test/unit/mutation/coverage-attribution.test.ts`
- **Approach:**
1. Accept an optional `testIds` array in the driver scope. Each id must be a Phase 0 test id of one of the scope's `testFiles`. Pass it to the plugin as the `vitest5.testIds` runner option (KTD2).
2. In the dry run, collect the scope files, assign ids with `assignTestIds`, map the requested ids to task ids, run one specification per file, and check the executed set equals the request.
3. Keep the raw hook bucket in the per-file metadata, next to the attributed per-test coverage. Persist it through `FileMeta` and the dry-run sidecar record. Expose `hookAttributedTests` per mutant.
4. In normalisation, keep every sidecar attempt's per-test rows as `perTest: { testId, outcome, attempt }[]`. Mark a mutant `attemptsDisagree` when attempts differ.
5. Bump the results `schemaVersion`, and keep gate 3 compare reading the fields it uses today.
- **Execution note:** run GitNexus impact analysis on `parseDriverScope`, `attributeHookHits`, `normaliseResults`, the `FileMeta` and sidecar record types, and the plugin's dry-run method before editing. These are gate surfaces.
- **Patterns to follow:**
- The mutant-run specification building in `runner/plugin.ts`.
- The scope validation in `stryker-config.ts`.
- The conformance case style in `gitnexus/test/fixtures/mutation-conformance/`.
- **Test scenarios:**
- A scope file with `testIds` naming two of five tests in one file: the parsed scope keeps them; the generated dry-run selection contains exactly those two.
- A `testIds` entry not in any scope file: the parser fails with a message naming the id.
- An empty `testIds` array: rejected, so that "all tests" never happens by accident.
- Normalising a mutant with two sidecar attempts, `killed-by-test:assertion` then `killed-by-test:runtime` for the same test: both rows are kept, and `attemptsDisagree` is true.
- Normalising a mutant whose coverage came only from a `beforeAll` hit: `hookAttributedTests` lists every test of that suite.
- New conformance case: a dry run with `testIds` set to one of two tests reports coverage and outcomes for that test only.
- New conformance case: a mutant hit in `beforeAll` carries the hook marker for each test below the suite.
- A `testIds` entry naming a test the collection did not find: the dry run fails before any mutant run, naming the id.
- Covers R17: gate 3 compare on fresh runner runs of the three oracle scopes still passes with 0 differences, with no re-freeze required.
- **Verification:** unit tests pass; gate 1 passes over the pure-JS category with 0 differences; gate 2 passes with the new pinned cases; gate 3 compare passes on all three scopes; gate 4 passes.
### U2. Layer 0 extraction API
- **Goal:** Phase 1 reuses Layer 0's case extraction, context closure and `dupe-ok` lookup without changing Layer 0's findings.
- **Requirements:** R5, R15. KTD6.
- **Dependencies:** none.
- **Files:**
- Modify: `gitnexus/tools/test-dupes/test-dupes.ts`.
- Test: `gitnexus/test/unit/test-dupes.test.ts`.
- **Approach:**
1. Export the case extraction, frame building, context closure and the `suppressed` lookup.
2. Add a per-case record with file, line, describe path, title, context hash, body text, `each` table node and enclosing loop head. Today's `recordCase` folds context into one hash; keep that hash for Layer 0 and expose the parts.
3. Change no detection rule.
- **Execution note:** run impact analysis on `extract`, `buildFrame`, `closure` and `scan` before editing. `recordCase` is nested in `extract` and covered by its analysis. Capture `npm run test:dupes -- --all --json <file>` before and after, with the same set of test files on disk, and require the two files to be identical.
- **Patterns to follow:** the existing exported `scan` and `orphanBenchGates` signatures and their tests.
- **Test scenarios:**
- Extracting a file with a nested describe and two tests returns two records. Each has the full describe path, the title, the declaration line and distinct context hashes when the `beforeEach` differs.
- A test with `// dupe-ok: reason` on the line above is reported as suppressed with that reason.
- An `it.each` call returns one record, with the table node and the title template.
- Layer 0 scan output on the existing test fixtures is unchanged.
- **Verification:** the `test-dupes` unit tests pass, and the `--all` scan JSON is byte-identical before and after.
### U3. Inventory, identity and anchors
- **Goal:** one inventory of eligible test instances, with execution identities joined one-to-one to the static records, and continuity anchors.
- **Requirements:** R1, R2, R15. KTD3.
- **Dependencies:** U2.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/inventory/eligible.ts`
- `gitnexus/tools/scenario-map/inventory/cases.ts`
- `gitnexus/tools/scenario-map/inventory/identity.ts`
- `gitnexus/tools/scenario-map/types.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/eligible.test.ts`
- `gitnexus/test/unit/scenario-map/cases.test.ts`
- `gitnexus/test/unit/scenario-map/identity.test.ts`
- **Approach:**
1. The eligible set is the envelope's accepted files intersected with a fresh `classify()` run. The delta goes into the inventory (R1). In `controls` mode, the fixture's test files are enrolled instead, and its `src/` modules are production files through `extraRoots` (KTD7). Normal runs always apply R1.
2. Static cases expand `it.each`/`test.each`/`test.for` tables and `for…of` loops over literal arrays into one instance per row, with the row's bound values. A table that cannot be evaluated statically gives instances marked `unresolved` (design §5.3).
3. The join takes the runtime records from Layer 1 (task id, full name, location, row order within a location) through `assignTestIds`. It matches them to static instances by file, location and row order. Any count mismatch makes every instance at that location unsupported.
4. Anchors per KTD3.
- **Patterns to follow:** `classify()` in `gitnexus/tools/mutation/classify/classify.ts`; `assignTestIds` in `runner/test-ids.ts`.
- **Test scenarios:**
- The envelope lists files A, B and C, and the classifier now accepts A, C and D: eligible is A and C; the delta lists D as not validated and B as rejected.
- An `it.each([[1, 2], [3, 4]])` test expands to two instances, with bound values `[1, 2]` and `[3, 4]` and row ordinals 0 and 1.
- An `it.each(buildRows())` test gives instances marked `unresolved`.
- Three runtime rows at one location against two static instances: all three are unsupported, with reason `join-mismatch`.
- Two tests with identical bodies in different files: the body fingerprints are equal and the anchors stay distinct by file. A moved test with a unique body keeps its anchor.
- `controls` mode with the fixture root: the fixture's case files are eligible, and the envelope is not consulted.
- The production-file predicate: `gitnexus/src/a.ts` and relocated `gitnexus-shared/src/b.ts` are production; `gitnexus/tools/x.ts`, `gitnexus/test/unit/y.test.ts` and a fixture `*.case.ts` are not; a fixture `src/` module is production only in `controls` mode.
- **Verification:** unit tests pass; an inventory run over the repo's eligible files completes and reports the join-mismatch count.
### U17. Controls fixture project
- **Goal:** a committed fixture project holds one pair per pure-JS adversarial and positive-control case, with expected verdicts written before any run, so U7–U11 can verify against it.
- **Requirements:** Success Criteria. KTD7.
- **Dependencies:** none.
- **Files:**
- Create: `gitnexus/test/fixtures/mutation-scenario-map/` with `src/` production modules, `*.case.ts` tests, `vitest.config.ts`, `tsconfig.json` (extends `gitnexus/tsconfig.test.json` and includes the fixture's `src/` and `*.case.ts`), and `expected.json`.
- **Approach:**
1. The fixture holds one pair per design §8 pure-JS adversarial and positive-control case:
- `Math.round` 2.4/2.5;
- `parseInt` radix;
- `localeCompare`;
- Unicode `normalize`;
- `Date` boundaries;
- regex alternatives;
- numeric boundary ±1, with the constant in a separate constants-only module;
- expected-only difference;
- derived value reaching `toFixed`;
- an enum-member-only difference;
- an order-dependent pair;
- an object-key row;
- a verbatim duplicate;
- `add(1, 2)` against `add(5, 5)`;
- membership in a supported `Set` table.
2. `expected.json` maps each pair to its verdict: every adversarial pair Review or Distinct, the order-dependent pair Inconclusive, the object-key pair Row, the duplicate and positive controls strong repeats.
3. No repo Vitest project collects the fixture, as with `mutation-conformance`.
- **Execution note:** write `expected.json` from the design's stated outcomes, not from observed output.
- **Patterns to follow:** `gitnexus/test/fixtures/mutation-conformance/` (own `vitest.config.ts` project, `*.case.ts`, `expected.json`).
- **Test scenarios:** Test expectation: none -- fixture data. U14's `controls` stage exercises it end to end.
- **Verification:** the fixture's own Vitest project passes unmutated; its tsconfig type-checks; the main suite does not collect it.
### U5. Layer 2 checks and context
- **Goal:** checks and context per test instance, with unsupported forms marked.
- **Requirements:** R5. KTD6.
- **Dependencies:** U2, U3.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/facet/checks.ts`
- `gitnexus/tools/scenario-map/facet/context.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/checks.test.ts`
- `gitnexus/test/unit/scenario-map/context.test.ts`
- **Approach:**
1. Checks are the shape of each `expect(subject)[.not|.resolves|.rejects].matcher(args)` with literals abstracted. Selector literals inside the subject call stay exact (design §2).
2. Snapshot assertions hash the inline snapshot text or the `.snap` entry.
3. The supported matcher list is a documented constant. Anything else is `unsupported`: `assert.*`, custom matchers, `expect` inside helpers not inlined in the body.
4. Context is the Layer 0 closure: hooks, `vi.mock`/`vi.spyOn`/`vi.stubEnv`, env writes, wrapper-call arguments. It is compared by exact hash.
- **Patterns to follow:** `assertionSignatures`, `closure` and `buildFrame` in `tools/test-dupes/test-dupes.ts`.
- **Test scenarios:**
- `expect(fn('a')).toBe(1)` and `expect(fn('b')).toBe(2)`: same checks shape.
- `expect(getNodesByLabel(r, 'Function')).toHaveLength(1)` and the same call with `'Class'`: different checks, because the selector literal stays exact.
- `expect(x).not.toBe(1)` and `expect(x).toBe(1)`: different checks.
- `toMatchInlineSnapshot` with different snapshot text: different checks.
- A body using `assert.equal`: marked unsupported, with the form named.
- Two tests that differ only in an `afterEach` body: different context.
- **Verification:** unit tests pass; the supported matcher list appears in the module header and in `TESTING.md` (U16).
### U4. Layer 1 coverage and candidate retrieval
- **Goal:** per-test V8 blocks and provenance for every eligible test, then candidate pairs and groups.
- **Requirements:** R3, R4. KTD4, KTD5.
- **Dependencies:** U3, U5 (checks-and-context hashes for the second candidate source).
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/coverage/coverage.setup.ts`
- `gitnexus/tools/scenario-map/coverage/run-coverage.ts`
- `gitnexus/tools/scenario-map/coverage/candidates.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/candidates.test.ts`
- `gitnexus/test/unit/scenario-map/coverage-records.test.ts`
- **Approach:**
1. Prepare a copy with `prepare()` and run the eligible files with `createVitest` under the mutation config. Prepend the coverage setup file.
2. Attribute per design §5.2. A root `beforeAll` discards module-load coverage. A suite's `beforeAll` delta is taken at its first `beforeEach`. Mark `concurrent` tests.
3. Keep the blocks of production files (KTD7), translated to repo paths.
4. Provenance lists every executed script URL, flags known native-package entry points, and records worker spawns through wrapped `worker_threads.Worker` and `child_process` spawn functions.
5. Candidates come from exact Jaccard ≥ 0.99 with a size-ratio filter and a rare-block prefix filter, plus pairs sharing a checks-and-context hash from U5. Groups are connected components with a deterministic group id.
6. Record the largest group's size and test count for KTD17.
- **Patterns to follow:**
- The hook pattern in `gitnexus/tools/mutation/runner/setup.ts` and `fidelity/recorder.setup.ts`.
- The spike `gitnexus/test/spike-coverage.setup.ts`.
- **Test scenarios:**
- Jaccard of two block sets that differ by 1 in 200: a candidate at the 0.99 threshold.
- Two sets that differ by 3 in 200: not a candidate.
- Two tests with Jaccard 0.5 but an identical checks-and-context hash: a candidate.
- Pairs A–B and B–C with A–C below threshold: one group of three containing exactly two candidate pairs.
- A coverage record from a `concurrent` test: its provenance is unknown, and its pairs are flagged path-unattributable.
- A record whose executed URLs include `node_modules/tree-sitter`: native-package entry flagged.
- Integration: run the hook on the controls fixture. A test whose `beforeAll` calls production code shows that code in its suite blocks, not in module-load blocks.
- **Verification:** unit tests pass; a coverage run over the eligible files finishes. Record its wall time and candidate counts in the PR.
### U6. Layer 2 input facet
- **Goal:** an input facet per test instance, with discarded values per position and unresolved inputs marked.
- **Requirements:** R5, R9. KTD6.
- **Dependencies:** U3, U4.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/facet/input-facet.ts`
- `gitnexus/tools/scenario-map/facet/tables.ts`
- `gitnexus/tools/scenario-map/facet/source-shape.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/input-facet.test.ts`
- `gitnexus/test/unit/scenario-map/tables.test.ts`
- **Approach:**
1. Port the spike's rules 1–8 (design §5.3). Production tables come from the test's executed files (U4), then its 2-hop import closure, then property columns. Tables are never pooled.
2. Add rule 6 (source order of literal shapes and builder-call names) and numeric literals.
3. Add the ±1 numeric-constant rule (KTD6), with constants from executed production files and the static import closure.
4. Keep each discarded concrete value per position, with its source location, for U10.
5. Add the residual-difference guard (KTD6) over each facet-equal candidate pair.
6. Remove every hard-coded path from the spike. The spike file stays untracked and out of the commit.
- **Patterns to follow:** `gitnexus/spike-input-facet.mts` (`syntaxProfile`, `declaredIn`, `shapeOf`, `tableOf`, `fixturesOfTest`); `createParserForLanguage` from `gitnexus/src/core/tree-sitter/parser-loader.ts`.
- **Test scenarios:**
- Two tests feeding `'Function'` and `'Class'`, both members of one production string-array table: equal facets, and discarded values `['Function']` and `['Class']` at position 0.
- Two tests feeding inline Rust and inline Go source: different facets.
- Two tests feeding inline TypeScript that differs only in a local variable name: equal facets.
- `parse(101)` and `parse(150)` against a production `LIMIT = 100`: different facets, because 101 stays exact under the ±1 rule.
- `parse(150)` and `parse(170)` against the same constant: equal facets.
- `LIMIT = 100` declared in a separate constants-only module that the test never executes per test: `parse(101)` and `parse(150)` still get different facets.
- `LIMIT = 2 * 50`: the folded constant 100 applies, and `parse(101)` stays exact.
- Two tests that differ only in an enum-member argument (`NodeLabel.Function` against `NodeLabel.Method`) outside the expect subject: equal facets, both instances unresolved by the residual-difference guard.
- Two tests that differ only in a regex-literal argument: both instances unresolved.
- Two textually identical test bodies: no residual difference, nothing marked unresolved.
- A literal `0` against a literal `5`: different facets (boundary shape).
- A fixture lookup the rules cannot resolve: the facet is marked unresolved.
- Literal order `[a, b]` against `[b, a]` with different shapes: different facets.
- Reproduce the spike's untouched holdout-4: 23/23 distinct pairs keep different facets. Holdout data comes from `/tmp/test-audit/spike3` if present. This is a one-off local check, not a committed test.
- **Verification:** unit tests pass; a facet run over the eligible files completes and reports the unresolved count.
### U7. Layer 3 group runs and observations
- **Goal:** per candidate group, valid baselines, one runner run over the group's tests, and raw observations ready to merge across shards.
- **Requirements:** R6, R7, R11. KTD7, KTD8, KTD14, KTD17.
- **Dependencies:** U1, U3, U4, U17.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/mutation/plan-shards.ts`
- `gitnexus/tools/scenario-map/mutation/vitest-run.ts`
- `gitnexus/tools/scenario-map/mutation/run-group.ts`
- `gitnexus/tools/scenario-map/mutation/observations.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/plan-shards.test.ts`
- `gitnexus/test/unit/scenario-map/observations.test.ts`
- **Approach:**
1. The mutate scope follows KTD7. Static closure files come from the classifier's closure output.
2. `vitest-run.ts` runs given test ids in a given prepared tree, with a fresh `createVitest` instance per call. Baselines, row deletion and the trace run share it.
3. A shard first verifies its prepared copy against the run manifest (KTD14).
4. A group run is two baselines, then `runScope` with `testFiles`, `testIds` and the whole-file `mutate` list. Before mutating, it checks that the generated config carries `perTest`, `disableBail` and static mutants (KTD8). Then it extracts observations from every per-test row (U1).
5. A raw observation is keyed by group, run kind, attempt, test execution id and global mutant id. Two observations of one test and mutant in one run lineage with different classes become inconclusive (design §5.4). Observations from different groups stay separate.
6. A pair is compared only inside one group run's common universe (R11). A covered mutant with no observation for one test is unknown, never a non-kill.
7. Each finished group's observations are written out at once (KTD17).
- **Patterns to follow:** `runScope` in `gitnexus/tools/mutation/driver/run-scope.ts`; `RunResults` in `driver/normalise-results.ts`.
- **Test scenarios:**
- Shard planning over five groups with costs 10, 9, 5, 3 and 1 into two shards: deterministic assignment, identical across two calls, and no group split across shards.
- A group whose estimate exceeds one shard's budget: not assigned; its pairs are Inconclusive with reason `group-over-budget`.
- A shard with 20 minutes left and a next group estimated at 30 minutes: the group is not started; its pairs are `not-scheduled`.
- A shard result that omits group G: every pair in G is Inconclusive with reason `not-scheduled`.
- A shard whose prepared-copy hashes differ from the run manifest: its results are invalid, and its groups' pairs are Inconclusive.
- A generated Stryker config without `disableBail`: the group run fails before mutating.
- Test A has an observation for covered mutant M and test B has none: M is unknown for the pair, and the kill sets are not compared as equal.
- Baselines with different identity sets: the group is Inconclusive with reason `baseline-unstable`, and no observations are emitted.
- Baseline 2 has a failing test: the group is Inconclusive.
- Merging two observations of test T and mutant M from two attempts, `killed-by-test:assertion` and `survived`: one inconclusive observation.
- Mutant M evaluated in group X (killed by T1) and in group Y (survived T2): both observations are kept, and each pair compares only within its own group.
- A test executed `src/a.ts` while the mutate list lacks it: the test is marked `excluded-file`.
- **Verification:** unit tests pass; one group run on the controls fixture produces observations for every test and mutant pair in its universe.
### U8. Full isolation pass
- **Goal:** strong evidence requires that per-test outcomes agree when the pair runs together and when each test runs alone, each mutant in a fresh process.
- **Requirements:** R7. KTD9.
- **Dependencies:** U1, U5, U6, U7, U17.
- **Files:**
- Create: `gitnexus/tools/scenario-map/mutation/isolation.ts`.
- Test: `gitnexus/test/unit/scenario-map/isolation.test.ts`.
- **Approach:**
1. Select pairs with the same `coveredBy` path, checks, context, facet and kill set from the main run.
2. Build the covered universe of each pair, and the Stryker ranges for its mutants.
3. Run `together` once per pair. Run `alone` once per test, over the union of its pairs' universes.
4. Compare per test and mutant: covered flag and outcome class across main, together and alone. Any difference makes the pair Inconclusive with reason `isolation-disagrees`.
5. A hook-attributed mutant in the pair's path or kills caps the pair at weak (KTD9).
- **Patterns to follow:** gate 3 per-test comparison in `gitnexus/tools/mutation/oracle/compare.ts`.
- **Test scenarios:**
- A pair whose main, together and alone observations all agree: isolation `agrees`.
- Test A kills M in main and together, but survives M alone: Inconclusive, `isolation-disagrees`, naming M.
- Coverage of M by B appears in main but not alone: Inconclusive.
- A pair that never reached the repeat stage: no isolation run is scheduled.
- Pairs A–B and A–C: `alone(A)` is scheduled once, over the union universe.
- A pair touching a hook-attributed mutant: capped at weak with disqualifier `hook-stratum-unvalidated`.
- **Verification:** unit tests pass; on the controls fixture, the order-dependent control pair ends Inconclusive.
### U9. Row-deletion operator
- **Goal:** pairs whose differing literals are keys of one supported production table get a symmetric, asymmetric, not-applicable or inconclusive outcome.
- **Requirements:** R8. KTD10.
- **Dependencies:** U6, U7, U17.
- **Files:**
- Create: `gitnexus/tools/scenario-map/mutation/row-deletion.ts`.
- Test: `gitnexus/test/unit/scenario-map/row-deletion.test.ts`.
- **Approach:**
1. Find the table from U6's table index.
2. Reject unsupported constructs: spreads, computed keys, `Object.assign`, writes into the table, getters, proxies, duplicate keys. The outcome is `inconclusive`, with the construct named.
3. Splice one key's property or entry text, including its trailing comma. Never regenerate the file.
4. Run the control, then one run per deleted key, through `vitest-run.ts` in a throwaway copy. Restore the file after each run.
- **Patterns to follow:**
- The identity build in `prepare/mutant-identity.ts`.
- The spike `/tmp/test-audit/spike5/row-delete.mjs` for the table search only.
- **Test scenarios:**
- An object table `{ a: 1, b: 2, c: 3 }`, with tests reading `a` and `b`: two deletions planned, with global ids that carry mutator `RowDeletion`.
- A `new Set(['x', 'y'])` table: deleting `'x'` splices only that element, and the file still parses.
- A table with a spread: `inconclusive`, reason `unsupported-table: spread`.
- Differing literals in no common table: `not-applicable`.
- Classification: only test A fails when `a` is deleted, giving `asymmetric`. Both fail, or neither fails, giving `symmetric`. A runtime error gives `inconclusive`.
- The control run has a failing test: `inconclusive`, reason `control-failed`.
- The control run never loads the edited file: `inconclusive`, reason `edit-not-loaded`.
- Two deletion runs in a row: each uses a new Vitest instance, so the second run sees the second edit, not a cached transform.
- **Verification:** unit tests pass; on the controls fixture, the object-key pair ends as Row.
### U10. Value-flow static check
- **Goal:** for each value-differing pair, the test-side binding, checker resolution and PDG walk give a clean, veto or unknown result with reasons.
- **Requirements:** R9. KTD11.
- **Dependencies:** U5, U6, U17.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/value-flow/bind.ts`
- `gitnexus/tools/scenario-map/value-flow/checker.ts`
- `gitnexus/tools/scenario-map/value-flow/pdg-walk.ts`
- `gitnexus/tools/scenario-map/value-flow/sinks.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/bind.test.ts`
- `gitnexus/test/unit/scenario-map/sinks.test.ts`
- `gitnexus/test/unit/scenario-map/pdg-walk.test.ts`
- **Approach:**
1. `bind.ts` maps each differing discarded value to an entry call argument, fixture field or table key, with Babel and `@babel/traverse` scope bindings. An expected-only difference is unknown (KTD11).
2. `checker.ts` opens the given tsconfig (KTD11) through `typescript/unstable/sync`, checks the version, and resolves each entry through `getAliasedSymbol` to its declaration path. It records parameter types and literal-union members. A version mismatch marks the run degraded.
3. `pdg-walk.ts` reads REACHING_DEF and `CALL_SUMMARY` edges through LocalBackend's `cypher` tool, scoped to the function's block-id prefix, and decodes them with `decodeReachingDefReason`. It applies the KTD11 walk rules. A repo callee recurses with the parameter at the argument's position. Depth is capped at 4, and the cap is unknown.
4. `pdgLayerStatus` gates the walk: a missing layer marks the run degraded (KTD11).
5. `sinks.ts` holds the pure classification, the statement rule and the storage/table-read exemptions, so it is unit-testable without a PDG.
6. The static check covers every facet-equal candidate pair whose discarded values differ. It runs in the `value-flow` job, in parallel with the shards (KTD14).
- **Execution note:** measure `gitnexus analyze --pdg` on the repo at the checkout's build, and record wall time and peak memory in the PR before sizing the CI job.
- **Patterns to follow:**
- `gitnexus/bench/impact-pdg/real-code.mjs` (backend construction under tsx).
- The probe `/tmp/test-audit/tsprobe/probe2.mjs`.
- Write output with `writeSync(1, …)`, because LadybugDB captures stdout (`gitnexus/src/cli/tool.ts`).
- **Test scenarios:**
- Covers design §8 adversarial "Math.round": the differing values reach `Math.round`, declared in `lib.es5.d.ts`. The result is `veto`, sink `opaque: Math.round`.
- Values reach `a + b` inside a repo function only: `clean`, sink `repo-code`.
- A value stored with `Map.set` and read back with `Map.get` into a repo comparison: `clean`, classified as storage and table read.
- A value used in `new RegExp(v)` or `v.match(...)`: `veto`, regex.
- Only the expected values differ (`toContain('X')` against `toContain('Y')`) and the inputs are equal: `unknown`, reason `expected-only`.
- The entry parameter is typed `any`: `unknown`.
- The PDG returns `truncated: true`: `unknown`.
- Recursion depth 5: `unknown`, reason `depth-cap`.
- `if (path.extname(v) === '.ts')`: `veto`, because the opaque `path.extname` use vetoes the whole statement even though the comparison is repo code.
- `const { ratio } = opts` followed by `Math.round(ratio)`: `veto`, because the derived binding is walked under its own name.
- The value reaches a repo function in `gitnexus/tools/`, outside the pair's mutated files: `unknown`, not `clean`.
- The value is returned unchanged and the caller compares it: `CALL_SUMMARY` continues the walk in the caller.
- The checker reports TypeScript 7.0.3: the run is degraded, and every value-differing pair is `unknown` with reason `checker-version`.
- `pdgLayerStatus` returns `no-layer`: the run is degraded, and every value-differing pair is vetoed with reason `no-pdg-layer`.
- **Verification:** unit tests pass; on the controls fixture, indexed with `--pdg` and checked with the fixture's tsconfig, each adversarial value pair gets the expected veto or clean result.
### U11. Value-flow trace run
- **Goal:** one traced run per candidate group catches differing data that reaches opaque built-ins, including derived values and flows that static analysis missed.
- **Requirements:** R9. KTD11.
- **Dependencies:** U4, U10, U17.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/value-flow/builtins.ts`
- `gitnexus/tools/scenario-map/value-flow/trace.setup.ts`
- `gitnexus/tools/scenario-map/value-flow/trace.ts`
- Test: `gitnexus/test/unit/scenario-map/trace.test.ts`.
- **Approach:**
1. `builtins.ts` is the documented wrap list (design §11): `Math`, string transforms and comparisons, `RegExp` methods, number parsing and formatting, `JSON`, `Intl`, `Date` and sort comparators. Add every opaque sink that U10 found. Storage and table reads stay unwrapped.
2. The setup file wraps each listed built-in, prototype methods included. It records call site, receiver, arguments and count, per current test, only for production frames (KTD7).
3. The trace run uses `vitest-run.ts` with the tests of every facet-equal candidate pair (KTD11). A test whose outcome differs from the baseline under tracing makes its pairs Inconclusive.
4. Measure the trace overhead on the eligible category and record it in the PR.
- **Patterns to follow:** the setup-file prepend in `runner/plugin.ts`; `TestRunner.getCurrentTest()` attribution in `runner/setup.ts`.
- **Test scenarios:**
- Two traces reach `Math.round` at `src/x.ts:12`, with arguments `2.4` and `2.5`: `veto`.
- Same site with equal arguments: `clean`.
- `(x * 10).toFixed(1)` reached with receivers 24 and 25 and equal arguments: `veto`, reason `receiver-differs`.
- Two facet-equal pairs with equal discarded values and identical traces: `clean`.
- Test A calls `toFixed` at a site that test B never reaches: `veto`, reason `asymmetric-sites`.
- Same site, A calls it twice and B once: `veto`, reason `asymmetric-counts`.
- A call from a `node_modules` frame only: ignored.
- A test passes at baseline and fails under tracing: the pair is Inconclusive, reason `trace-changed-outcome`.
- `Map.prototype.set` calls are never recorded.
- **Verification:** unit tests pass; on the controls fixture, the derived-value control (`x * 10` reaching `toFixed`) is vetoed.
### U12. Verdict engine
- **Goal:** one verdict per pair, and per-test Weak, Inconclusive and Subsumed properties, from merged observations and the Layer 1/2 and value-flow evidence.
- **Requirements:** R10, R11, R15. KTD5, KTD8, KTD9, KTD12.
- **Dependencies:** U5–U11.
- **Files:**
- Create: `gitnexus/tools/scenario-map/verdict/verdict.ts`.
- Test: `gitnexus/test/unit/scenario-map/verdict.test.ts`.
- **Approach:**
1. Encode design §2 precedence as an ordered list of rules. Each rule returns a verdict or passes. The first decisive rule names itself, and every weak disqualifier is listed.
2. Weak disqualifiers: provenance not strong-eligible, an unresolved facet input, inconclusive row deletion, an unsupported construct, isolation not run, a test not oracle-validated, non-identical V8 blocks (KTD5), an excluded file (KTD7), a hook stratum (KTD9).
3. Subsumption holds only when B was evaluated on every mutant in A's universe within one group run. Otherwise it is unknown.
4. Gaps are recomputed per production file from all observations. A mutant is killed if any assertion kill exists for it; it is "survived the selected tests" when covered somewhere and never killed; it is "uncovered by selected tests" otherwise.
5. A `dupe-ok` comment on either member moves the pair to a kept list and keeps its evidence.
- **Patterns to follow:** the grouping and subsumption logic in `/tmp/test-audit/spike5/analyze-slice.mjs`.
- **Test scenarios:**
- Covers design §8 positive control: `add(1, 2)` against `add(5, 5)` with identical path, checks, context, facet and kills, clean value flow, isolation agreeing, and all strong conditions met: Repeat (strong evidence).
- The same pair with isolation not run: Repeat (weak evidence), disqualifier `isolation-not-run`.
- Same facet, kill sets differing by one mutant: Row.
- Same facet and kills, row deletion `asymmetric`: Row.
- Same kills, different facet: Review.
- Same facet and kills, value-flow veto: Review, never a repeat.
- Same facet and kills, equal discarded values, trace veto (asymmetric sites): Review.
- Same facet and kills, equal discarded values, clean trace: the static value-flow check is not required, and the pair can be a strong repeat.
- A test that executed `gitnexus/tools/x.ts`: the pair caps at weak, disqualifier `excluded-file`.
- Both kill sets empty: never a repeat; each test is Weak.
- Test A has no assertion kills but has a runtime kill: per-test Inconclusive, not Weak.
- A timeout on a mutant covered by B: the pair is Inconclusive.
- The pair is `not-scheduled`: Inconclusive.
- A's kills are a strict subset of B's, and B was evaluated on all of A's universe: A is Subsumed. When B misses one mutant: subsumption unknown.
- Mutant M survives group X and is killed in group Y: the gap list counts M as killed.
- A pair with `dupe-ok` on test B: listed as kept, absent from the findings.
- Running the engine twice on the same input gives byte-identical output.
- **Verification:** unit tests pass with one test per precedence rule and per disqualifier.
### U13. Report and issue body
- **Goal:** the JSON artifact, its markdown rendering and the issue body, with continuity and week-over-week change.
- **Requirements:** R12, R13. KTD13, KTD15.
- **Dependencies:** U12.
- **Files:**
- Create:
- `gitnexus/tools/scenario-map/report/report.ts`
- `gitnexus/tools/scenario-map/report/markdown.ts`
- `gitnexus/tools/scenario-map/report/issue-body.ts`
- `gitnexus/tools/scenario-map/report/baseline.ts`
- Tests:
- `gitnexus/test/unit/scenario-map/report.test.ts`
- `gitnexus/test/unit/scenario-map/issue-body.test.ts`
- `gitnexus/test/unit/scenario-map/baseline.test.ts`
- **Approach:**
1. The JSON carries `schemaVersion`, run provenance (KTD15 inputs, seed, engine version), completeness (shards complete, degraded flags), eligibility delta, the Layer 0 findings over the eligible files, every verdict with its evidence, per-test properties, gaps, spot audit and change status.
2. The mutant fingerprint is file, operator, replacement and normalised enclosing statement (design §5.5).
3. `baseline.ts` compares against a previous report: new, changed (evidence hash differs), unchanged, resolved. With no baseline, a "first run" banner appears. With a different schema version, an "incompatible baseline" banner appears and no delta is computed (KTD13).
4. The issue body has a header: run link and run id marker, completeness, every degraded flag, "N of M eligible tests evaluated", and a precision-not-measured statement. Then come strong repeats, weak tests and gaps. It is truncated at 60,000 characters with counts.
- **Patterns to follow:** the upsert body shape in `.github/workflows/tree-sitter-upgrade-readiness.yml`; the markdown rendering in `tools/test-dupes/test-dupes.ts`.
- **Test scenarios:**
- A previous report has pair P with evidence hash h1, and the current report has P with h1: unchanged, absent from "new or changed".
- Same P with h2: changed.
- P only in previous: counted as resolved.
- No previous report: every item is new, and the banner reads "first run, no baseline".
- A previous report with an older schema version: the "incompatible baseline" banner, and no new/changed/resolved counts.
- A run degraded by `no-pdg-layer`: the issue header names the flag.
- The issue body carries a run id marker that a later upsert can compare.
- A verdict-engine version bump alone changes every evidence hash.
- A regrouping that leaves a pair's common universe unchanged keeps its evidence hash.
- The spot audit with seed `s1` picks the same 10 strong candidates on two runs. With fewer than 10 strong candidates, it picks all of them.
- An issue body with 5,000 gap rows: truncated below 60,000 characters, keeping all strong repeats first, then weak tests, and ending with a count line.
- A complete run with no findings: the body states "no strong repeat candidates, weak tests or gaps found".
- **Verification:** unit tests pass; the markdown rendered from the controls run reads correctly.
### U14. CLI orchestration and controls run
- **Goal:** one CLI runs every stage locally or as CI jobs, and the controls run proves the pipeline end to end.
- **Requirements:** R14, R16; Success Criteria. KTD1, KTD17.
- **Dependencies:** U1–U13, U17.
- **Files:**
- Create: `gitnexus/tools/scenario-map/scenario-map.ts`.
- Modify: `gitnexus/package.json` (script `test:scenario-map`).
- Test: `gitnexus/test/unit/scenario-map/cli.test.ts`.
- **Approach:**
1. Stages per KTD1. Each stage reads and writes JSON files in a work directory, so CI jobs exchange them as artifacts.
2. The `plan` stage also runs the Layer 0 scan over the eligible files and writes the run manifest (KTD14).
3. `controls` runs every stage on the U17 fixture through `extraRoots`, with the fixture's tsconfig for the checker, and compares the verdicts with `expected.json`. Its exit codes: 0 holds, 1 differs, 2 usage.
4. The fixture's PDG index follows `gitnexus/bench/impact-pdg/measure.mjs`: copy the fixture to a temp directory and run `node dist/cli/index.js analyze <copy> --pdg --skip-git --index-only` under a temporary `GITNEXUS_HOME`. It needs `dist/` built, and leaves the developer's `~/.gitnexus` untouched.
- **Patterns to follow:** `gitnexus/tools/mutation/conformance/run-conformance.ts` (fixture project, compare-only mode, exit codes); `gitnexus/bench/impact-pdg/measure.mjs` (fixture PDG index); the npm script form in `gitnexus/package.json`.
- **Test scenarios:**
- Parsing `shard --index 2 --of 8 --work dir` gives the stage options.
- `--index 9 --of 8`: usage error, exit 2.
- `merge` with a missing shard result: the run is marked partial, and that shard's pairs are Inconclusive.
- Covers the Success Criteria: the `controls` stage on the fixture matches `expected.json`. Every adversarial pair is Review or Distinct; the duplicate is a strong repeat; the object-key pair is a Row; the order-dependent pair is Inconclusive.
- `controls` without a built `dist/`: exits 2 with a message naming the build step, rather than running degraded.
- **Verification:** `controls` exits 0. A local `all` run over the eligible category completes. Its counts, Inconclusive causes and wall time are recorded in the PR.
### U15. Scheduled workflow and PR controls job
- **Goal:** a weekly advisory job produces the artifact and updates the tracking issue, and PRs touching the tooling run the controls.
- **Requirements:** R13, R14. KTD14, KTD17.
- **Dependencies:** U14.
- **Files:**
- Create: `.github/workflows/test-scenario-map.yml`.
- Modify: `.github/workflows/mutation-runner-validation.yml` (`scenario-map-controls` job with `build: 'true'`; paths per KTD14).
- **Approach:**
1. Triggers: `schedule` (Assumptions) and `workflow_dispatch`, with inputs `shards` and `seed`.
2. Top level: `contents: read`, one workflow-wide concurrency group with `cancel-in-progress: false` (KTD14), and `defaults.run` in `gitnexus`. Every job has `timeout-minutes` and `continue-on-error: true`, and uses `./.github/actions/setup-gitnexus`.
3. Jobs per KTD14. `value-flow` uses `build: 'true'`, because `analyze --pdg` spawns workers from `dist/`.
4. Merge fetches the newest `scenario-map-baseline` artifact through the Actions artifacts REST API, under `actions: read`. It uploads a new one only for complete default-branch runs, with `retention-days: 90` (KTD13).
5. Upsert runs `actions/github-script` with `issues: write`. It finds the issue by label in any state, skips when the issue records a newer run id, and passes the body as a file.
6. Artifact names carry `github.run_attempt`. Shard artifacts upload with `if: always()`.
- **Execution note:** this is CI configuration. Verify with actionlint, zizmor and `python3 .github/scripts/check-workflow-concurrency.py`; the job cannot be dispatched before the merge.
- **Patterns to follow:**
- `.github/workflows/tree-sitter-upgrade-readiness.yml` (upsert job).
- `.github/workflows/mutation-runner-validation.yml` (pinned SHAs, artifacts, advisory jobs).
- `.github/workflows/impact-pdg-mutation-report.yml` (`--pdg` with a build).
- **Test scenarios:** Test expectation: none -- workflow configuration. Its logic lives in U13 and U14, which carry the tests; workflow lint covers the YAML.
- **Verification:** actionlint, zizmor (high severity) and the concurrency check pass; the PR's controls job runs green.
### U16. Documentation and design corrections
- **Goal:** the design, the testing guide and the upstream notes match what Phase 1 built.
- **Requirements:** R12, R16, R17.
- **Dependencies:** U14.
- **Files:**
- Modify:
- `docs/plans/2026-10-23-test-scenario-map-design.md`
- `TESTING.md`
- `gitnexus/tools/mutation/UPSTREAM.md`
- **Approach:** in the design:
- correct §5.3a step 3 (intra-procedural PDG, stitched walk; KTD11);
- correct §5.4 (row deletion and the isolation pass on plain Vitest 5 and `testIds`; KTD2, KTD9);
- record the demote-only additions (KTD5 block identity, KTD6 ±1 rule and residual-difference guard, KTD7 excluded files, KTD9 hook cap, KTD11 trace on every facet-equal pair);
- record the schema-constants re-freeze already committed on this branch (merging `main` changed the scope's import closure; re-frozen with seed `u7-gate3-2`, gate 3 passing on all three scopes);
- add the Phase 1 results: the full-run counts, Inconclusive causes and costs from U14.
In `TESTING.md`: the `test:scenario-map` stages, the supported matcher list, the wrapped built-ins list, the controls fixture. In `UPSTREAM.md`: the `testIds` scope delta.
- **Test scenarios:** Test expectation: none -- documentation only.
- **Verification:** the docs name only files and flags that exist; prettier check passes.
---
## Verification Contract
Commands run from `gitnexus/` unless noted.
| Check | Command | When |
| --- | --- | --- |
| Types | `npm run typecheck:tests` | every unit |
| Unit tests | `npx vitest run test/unit/scenario-map test/unit/mutation test/unit/test-dupes.test.ts` | every unit |
| Lint and format | `npx eslint tools/scenario-map tools/mutation tools/test-dupes test/unit/scenario-map` and `npx prettier --check` on changed files (repo root config) | every unit |
| Layer 0 unchanged | `npm run test:dupes -- --all --json <file>` before and after, same test files on disk, files identical | U2 |
| Gate 1 | `npm run test:mutation:fidelity` | U1 |
| Gate 2 | `npm run test:mutation:conformance` | U1 |
| Gate 3 | fresh runner runs of the three oracle scopes, then `npm run test:mutation:oracle -- --scope <manifest>,<results>` ×3 | U1 |
| Gate 4 | `npm run test:mutation:controls` | U1 |
| Controls end to end | `npm run test:scenario-map -- controls` | U14 onward |
| Full local run | `npm run test:scenario-map -- all --work <dir>` | U14 |
| Workflow lint | `actionlint`, `zizmor` and `python3 .github/scripts/check-workflow-concurrency.py` (repo root) | U15 |
| Graph safety | GitNexus `impact` before editing existing symbols; `node .gitnexus/run.cjs detect-changes --scope all --repo .` before each commit (repo root) | U1, U2, every commit |
The full unit suite (`npm run test:unit`) runs before the PR is pushed. The known load flake in `test/unit/cli-index-help.test.ts` is rerun alone before it is treated as a failure.
---
## Definition of Done
- Every unit's Verification holds, and every command in the Verification Contract passes.
- Gates 1, 2, 3 and 4 pass after U1 (R17), with no oracle re-freeze caused by U1.
- `controls` matches `expected.json`, and one full local run over the eligible category completes. Its counts, Inconclusive causes and costs are in the PR description and in design §9.
- No file under `gitnexus/src` changes. No existing test of production code is edited; only the tooling's own tests gain cases.
- The spike files (`gitnexus/spike-input-facet.mts`, `gitnexus/test/spike-coverage.setup.ts`, `gitnexus/vitest.spike-cov.config.ts`, `gitnexus/vitest.stryker-bake.config.ts`) stay uncommitted.
- No `any` in new code. No `if` inside test bodies; table cases use `it.each` with unique titles. Every sampled choice uses a recorded seed.
- Abandoned-attempt code, debug output and scratch scripts are removed from the diff.

192
docs/plans/2026-10-25-1700-feat-scenario-fingerprint-plan.md Normal file
View file

@ -0,0 +1,192 @@
---
title: Scenario fingerprint — mutation-free duplicate verdicts for the test scenario map
date: 2026-10-25
status: implemented
origin: docs/plans/2026-10-23-test-scenario-map-design.md
branch: feat/test-dupe-detector (PR #3686)
---
# Scenario fingerprint
## Problem
Two tests are duplicates when they run the **same execution path** on the **same input data** and
check the **same expectation** (maintainer definition, 2026-10-25). The scenario map decides
repeats from mutation kill sets: about 4 h and 30 runner-hours per full run, and most pairs end
Inconclusive on runtime-class kills and hangs.
## Evidence (run 40208819019, verdict engine 3, mutation as reference)
- Pairs whose V8 block sets are identical, whose checks-and-context keys match, whose input facets
and discarded values are equal with nothing unresolved, and whose trace is clean: **26**.
Mutation calls 13 of them strong repeats (all 13 of the run); the other 13 it could not decide
(timeouts, runtime-class kills), and their titles show the same input and expectation. **None**
is Distinct or Row.
- The static `same-test` scan flags 25 of the 26; the one it misses is a renamed clone
(`method-props.test.ts` lines 47 and 185, `findString` renamed `findStr`).
- Every triple-equal pair whose kills differ carries an unresolved input (`it.each` rows whose
value only shapes filesystem state): the fail-closed rule already holds them out.
- Mutation is only informative where inputs are unresolved: 1,066 pairs in 38 groups, about
1.4 h of the 42.9 h estimated cost.
## Decisions
- **F1 (maintainer):** the fingerprint is the primary verdict; mutation is optional evidence.
- **F2 (maintainer):** the weekly run is fingerprint only; mutation runs on manual dispatch.
- **F3 (maintainer):** a labelled pull request (`scenario-map`) gets the deep run (fingerprint
plus mutation over the possible duplicates); no automatic PR run.
- **F4 (maintainer):** optimise the deep run as far as possible and measure it locally, not on CI.
- **F5:** different input data is not a duplicate. A pair with equal shape and different values
is a table-row candidate, decided without mutation.
- **F6:** an input the static extraction cannot resolve, a missing facet or checks key, a residual
body difference, an unsupported construct, or a trace that is not clean makes a pair at most a
possible duplicate, never a duplicate (fail-closed, design R2, D1).
- **F7:** the deep run mutates only groups with a possible duplicate, and in each such group only
the tests of those pairs, over the files those tests executed.
## Fingerprint verdicts (pure, `verdict/fingerprint.ts`)
Per candidate pair, first rule that holds:
| Verdict | Rule | When |
| ------------------ | ---------------------------------- | -------------------------------------------------------------------- |
| out-of-scope | `not-eligible` | a test is outside the eligible category |
| inconclusive | `path-unattributable` | a test ran concurrently |
| distinct | `path-differs` | the V8 block sets differ |
| possible-duplicate | `checks-unknown` / `facet-unknown` | a checks key or facet is missing and nothing known differs |
| different-checks | `same-input-different-checks` | same path and input (equal, resolved), checks differ |
| distinct | `checks-differ` | checks differ otherwise |
| distinct | `input-shape-differs` | facet fingerprints differ |
| table-row | `input-values-differ` | same shape, discarded values differ |
| table-row | `runtime-values-differ` | equal static values, trace veto |
| possible-duplicate | `input-unresolved` | equal, with an unresolved input (F6) |
| possible-duplicate | `trace-unknown` | equal and resolved, trace not clean (inconclusive, missing, not run) |
| duplicate | `same-path-input-expectation` | everything equal, resolved, trace clean |
A `// dupe-ok:` reason on either test moves the pair to `kept`.
## Units
- **U1** `verdict/fingerprint.ts` and its tests (one case per rule).
- **U2** report: `fingerprint` section and `mutation` scope; markdown and issue body lead with
duplicates and possible duplicates; weak tests and gaps only when mutation ran. Report schema
bump.
- **U3** CLI: `--mutation none|possible-duplicates|all` on `plan` and `all`; value flow skips the
PDG static check unless `all`; the re-plan keeps only the groups F7 names, trimmed; merge
computes fingerprints always and mutation verdicts only for the mutated pairs; zero shards is a
valid plan.
- **U4** controls: fingerprint expectations in `expected.json`, written from the rules above
before any run, and checked by the controls stage.
- **U5** workflow: weekly fingerprint; dispatch input `mutation`; `pull_request: labeled`
(`scenario-map`) with mutation `possible-duplicates`; per-ref concurrency for PR runs; shard jobs
skipped at zero shards; PR runs write no issue and no baseline.
- **U6** docs: design doc, workflow and CLI headers.
- **U7** local measurement (bounded, `nice -n 19`, concurrency at most 4): the full fingerprint
run, and the deep run over the possible duplicates.
## Verification
`tsc` (src and tests), eslint, prettier, the scenario-map unit tests, the local controls run, and
U7's timings recorded in the design doc.
## Results (2026-10-25)
- U1-U5 committed on PR #3686. The controls fixture matches all 16 fingerprint and all 16 mutation
expectations (local run, 43 min).
- Fingerprint-only run, all 4,632 eligible tests, locally: 8.2 min (plan 5.4, trace 2.6), 2 GB
peak RSS. 26 duplicates (the 26 the evidence predicted), 109 possible duplicates in 17 groups,
945 table rows, 5,184 distinct.
- Deep run over the possible duplicates, on CI (run 40268120818, the `scenario-map` label): 5 min 42 s
end to end, the shard 1.3 min for 5 groups and 844 mutants. It settles 35 of the 38 possible
duplicates mutation can reach (31 Row, 1 Distinct, 3 weak repeats; 3 Inconclusive); 71 test
`tools/mutation/` code outside the mutation scope. Its merge missed the single shard artifact
(unpacked into the download path itself); fixed in the workflow's placement step.
## Extension: every unit and integration test (maintainer, 2026-10-26)
The fingerprint needs no Stryker run, so its scope is no longer bound to the mutation runner's
validated category. **F8:** the fingerprint covers every `*.test.ts` under `gitnexus/test/unit`
and `gitnexus/test/integration`; mutation stays on the gate-validated pure-JS envelope.
- **E1** Eligibility carries `fingerprintFiles` besides the mutation-eligible `files`; a test's
verdict input carries `eligible` (mutation), `fingerprintEligible` and `passed`.
- **E2** Layer 1 runs the files of the base config's `fileParallelism: false` projects (`lbug-db`,
`cli-e2e`) with one worker, the rest with `--concurrency` workers, and joins both into one work
file.
- **E3** The trace run uses the same two partitions.
- **E4** Fingerprint rule `not-passing`: a test that did not pass in Layer 1 makes its pairs
Inconclusive. The mutation target keeps only pairs whose two tests are mutation-eligible.
- **E5** Measure Layer 1 on a sample of integration files locally; shard Layer 1 on CI when the
whole set does not fit one plan job.
- Result (2026-10-26): a sample of 44 integration files (every tenth) ran past 35 min at
concurrency 4 on the shared host, against 8.2 min for all 4,632 unit tests. That host was
loaded by other suites (see E7), so the sample overstated the cost; Layer 1 still runs in
slices, which keeps the plan job small and the wall clock short. The `coverage` stage runs one
slice (`--index i --of n`,
`path-coverage/slices.ts`: split like the CI test shards by `test/helpers/shard-balance.ts`),
and `plan --coverage <dir>` merges the slice files; a slice that did not report leaves its
files unrecorded. The workflow runs 4 slices before the plan (first sized at 12 from an
over-pessimistic estimate; corrected under E7: the whole suite with coverage runs in three
7-8 minute shards of the test workflow, ci.yml run 30014110469).
- **E6** (found by E5) The prepare step (gate-3 hashed) leaves `vendor/` and `dist/` out of the
prepared copy, so half the sampled integration files failed to load (leiden, the parse worker,
the built CLI). Layer 1 and the trace copy both from the checkout into their prepared copy in
normal mode; every stage that runs tests (plan, value-flow, all, controls) requires a built
`dist/`, and the workflow's plan and value-flow jobs build it.
- **E7** (optimisation, maintainer 2026-10-26: "can we optimise the tool?") Layer 1 skips the files
no fingerprint finding can come from. Past the path rule a pair is a duplicate, a possible
duplicate or a table row only when the two checks-and-context keys are equal or one is unknown,
and the key is static, so `path-coverage/prefilter.ts` keeps a file only when a test's key is
shared or unknown, a call's instance count is not static, a body reads a file snapshot, the file
may register tests the inventory cannot see (a call at registration level that is not the test
API, such as `defineRenameFixtureTests`, `describeGrammarPresence` or an `it` alias), or it did
not parse. Given up: `different-checks` between a test of a skipped file and another test (0 on
the last full unit run). `--mutation all` and the controls run every file.
- Evidence: on the 250 unit files of the last full fingerprint run, the static keys reproduce
all 3,313 runtime keys and select exactly the 172 files the runtime keys select. On the current
checkout the prefilter skips 449 of 1,399 files (149 integration), about 24% of the weighted
Layer 1 cost (`test/helpers/shard-balance.ts`); Vitest's runtime collection
(`vitest list --no-static-parse`) of the skipped files finds no test the static inventory
misses (434 equal counts, 15 with fewer runtime tests: benchmarks skipped at collection).
- Layer 1 overhead, measured as ratios inside one instrumented run (absolute times on the shared
host are meaningless: a plain run of `resolvers/rust` skipped all 255 tests after 798 s): the
coverage snapshots of `resolvers/rust` (1,022 takes) cost 10.1 s against about 190 s for the
file, whose work sits in `beforeAll` (not repeated, not snapshotted per test), and its repeat
attempts re-ran 0.1 s of test bodies; a unit file of many tiny tests (`scenario-map/cli`)
spends 2.5 s on snapshots against 0.3 s of test bodies. So Layer 1 costs little more than a
plain run, and the slices are sized from the test workflow's shards: 4.
- Visible gaps: the report counts the Layer 1 tests without a fingerprint, those skipped at run
time (`skip`, `skipIf`, `todo`; the `GITNEXUS_BENCH` benchmark suites run only in the test
workflow's benchmarks job, as wall-clock and heap guards) and those that never finished
(their file's run ended first).
- **E8** (optimisation) A labelled pull request's run is diff-scoped: `--changed-since <ref>` (the
workflow passes the merge commit's first parent to the coverage slices; a local `all` takes it
directly) keeps only candidate pairs with a test of a test file the diff added or modified, and
Layer 1 runs only those files (when the prefilter would keep them), the files sharing their
tests' checks-and-context keys and the files below. A changed `__snapshots__/<name>.snap` counts as a change of its test file;
changed helper modules and fixtures do not. Full recall (maintainer, 2026-10-26): every file
with a test of no static key (helper-registered, dynamic `it.each` counts, file snapshots, no
assertion key, unparsed) runs too, since such a test can pair with any changed test. That floor
is 230 of 1,407 files, about 59% of the suite's weighted Layer 1 cost with 59 sequential
files, too much for the plan job alone, so a labelled pull request runs the coverage slices as
well: `coverage --changed-since` records the changed files in each slice, and the plan takes
the diff scope from the slices. A pull request that changes no test file still runs the floor;
the report names the changed files.
- Review (2026-10-26, eight-angle code review of E1-E8, findings verified against the code):
fixed a suite callback passed by name (`describe('x', defineCases)`) slipping past the
registration guard, an empty coverage slice failing its job, `--changed-since` accepted with the
controls fixture, the report counting unreported slices' files as run (it now counts unrecorded
files), snapshot-only diffs scoping nothing, and the diff-scope report claiming more than it
checks. Not changed: mutation shards never get integration tests (`mutationTarget` keeps only
pairs of two mutation-eligible tests, the pure-JS envelope); a slice that disagrees with the
plan stops the merge on purpose.
- Cleanup review (2026-10-26, reuse, simplification, efficiency and altitude): a shard's
row-deletion batch ran every planned pair of a complete group, including pairs whose tests the
mutation target trimmed away; it now runs only pairs of tests the trimmed group holds. Setup
cost: the coverage jobs list the fingerprint's files without the Phase 0 classifier (about
33 s each, twice per job); slice files carry Layer 1's file list, mutation scope and revision,
so the plan checks them instead of recomputing the prefilter (about 30 s); the
unseen-registration check moved into the static inventory (`incomplete`), which parses each
file once and uses test-dupes' `roleOf`. Not changed: `fingerprintEligible` is always true at
its only producer, so the fingerprint's `not-eligible` rule cannot fire (kept for a future scope
narrower than Layer 1's files).

2
eslint.config.mjs
View file

@ -52,6 +52,8 @@ export default [
'**/dist/**',
'**/node_modules/**',
'**/coverage/**',
'**/.stryker-tmp/**',
'gitnexus/reports/mutation/**',
'gitnexus/vendor/**',
'gitnexus-web/src/vendor/**',
'gitnexus/test/fixtures/**',

1657
gitnexus/package-lock.json generated
View file

File diff suppressed because it is too large Load diff

14
gitnexus/package.json
View file

@ -62,6 +62,18 @@
"test:coverage": "vitest run --coverage",
"test:cross-platform": "tsx scripts/run-cross-platform.ts",
"test:benchmarks": "node --import tsx scripts/run-benchmarks.ts",
"test:dupes": "node --import tsx tools/test-dupes/test-dupes.ts",
"test:mutation:prepare": "node --import tsx tools/mutation/prepare/prepare.ts",
"test:mutation:run": "node --import tsx tools/mutation/driver/run-scope.ts",
"test:mutation:conformance": "node --import tsx tools/mutation/conformance/run-conformance.ts",
"test:mutation:classify": "node --import tsx tools/mutation/classify/classify.ts",
"test:mutation:fidelity": "node --import tsx tools/mutation/fidelity/fidelity.ts",
"test:mutation:oracle": "node --import tsx tools/mutation/oracle/compare.ts",
"test:mutation:oracle:command": "node --import tsx tools/mutation/oracle/command-run.ts",
"test:mutation:oracle:freeze": "node --import tsx tools/mutation/oracle/freeze.ts",
"test:mutation:bench": "node --import tsx tools/mutation/bench/oracle-cost.ts",
"test:mutation:controls": "node --import tsx tools/mutation/controls/run-controls.ts",
"test:scenario-map": "node --import tsx tools/scenario-map/scenario-map.ts",
"postinstall": "node scripts/prepare-tree-sitter-bundle.cjs && node scripts/build-tree-sitter-grammars.cjs",
"assert-publish-coverage": "node scripts/assert-publish-grammar-coverage.cjs",
"prepare": "node scripts/build.js",
@ -125,6 +137,8 @@
"@babel/traverse": "^8.0.0",
"@babel/types": "^8.0.0",
"@mermaid-js/tiny": "11.17.2",
"@stryker-mutator/api": "10.0.0",
"@stryker-mutator/core": "10.0.0",
"@types/busboy": "^1.5.4",
"@types/cli-progress": "^3.11.6",
"@types/cors": "^2.8.17",

985
gitnexus/test/fixtures/mutation-conformance/expected.json vendored Normal file
View file

@ -0,0 +1,985 @@
{
"schemaVersion": 1,
"profiles": [
{
"name": "concurrency-2-reuse-1",
"concurrency": 2,
"maxTestRunnerReuse": 1
},
{
"name": "concurrency-1-reuse-unlimited",
"concurrency": 1,
"maxTestRunnerReuse": 0
}
],
"scopes": [
{
"name": "main",
"testFiles": [
"static.case.ts",
"import-failure.case.ts",
"hooks.case.ts",
"runtime.case.ts",
"rows.case.ts"
],
"mutate": [
"src/static-table.ts",
"src/boot.ts",
"src/hooks.ts",
"src/runtime.ts",
"src/counter.ts",
"src/rows.ts"
],
"testIds": null,
"tests": [
{
"id": "hooks.case.ts#records loaded in beforeAll > loads one record",
"skipped": false,
"notOracleValidated": false
},
{
"id": "hooks.case.ts#records loaded in beforeAll > names the record",
"skipped": false,
"notOracleValidated": false
},
{
"id": "hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check",
"skipped": false,
"notOracleValidated": false
},
{
"id": "hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"skipped": false,
"notOracleValidated": false
},
{
"id": "hooks.case.ts#a suite whose afterAll checks the connection > runs before the closing check",
"skipped": false,
"notOracleValidated": false
},
{
"id": "hooks.case.ts#runs outside every suite with hooks",
"skipped": false,
"notOracleValidated": false
},
{
"id": "import-failure.case.ts#boots in strict mode",
"skipped": false,
"notOracleValidated": false
},
{
"id": "rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"skipped": false,
"notOracleValidated": true
},
{
"id": "rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1",
"skipped": false,
"notOracleValidated": true
},
{
"id": "runtime.case.ts#doubles two",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#doubles three",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#reads the first character of nothing",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#counts three items",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#schedules no audit when quiet",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#settles without waiting",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#issues the first ticket",
"skipped": false,
"notOracleValidated": false
},
{
"id": "runtime.case.ts#runs while the feature is enabled",
"skipped": false,
"notOracleValidated": false
},
{
"id": "static.case.ts#reads the unit table",
"skipped": false,
"notOracleValidated": false
},
{
"id": "static.case.ts#a suite whose body calls production code > keeps the label computed during collection",
"skipped": false,
"notOracleValidated": false
},
{
"id": "static.case.ts#row alpha",
"skipped": false,
"notOracleValidated": false
},
{
"id": "static.case.ts#row beta",
"skipped": false,
"notOracleValidated": false
}
]
},
{
"name": "retry",
"testFiles": [
"retry.case.ts"
],
"mutate": [
"src/retry.ts"
],
"testIds": null,
"tests": [
{
"id": "retry.case.ts#waits ten milliseconds per attempt",
"skipped": false,
"notOracleValidated": false
},
{
"id": "retry.case.ts#uses the linear policy",
"skipped": false,
"notOracleValidated": false
}
]
},
{
"name": "selection",
"testFiles": [
"selection.case.ts"
],
"mutate": [
"src/selection.ts"
],
"testIds": [
"selection.case.ts#scaling > scales two"
],
"tests": [
{
"id": "selection.case.ts#scaling > scales one",
"skipped": true,
"notOracleValidated": false
},
{
"id": "selection.case.ts#scaling > scales two",
"skipped": false,
"notOracleValidated": false
}
]
}
],
"cases": [
{
"kind": "mutant",
"name": "static: a mutant in a module-level table is killed with the module re-imported fresh",
"scope": "main",
"mutant": {
"file": "src/static-table.ts",
"location": "7:62-7:76",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "Killed",
"static": true,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [],
"selected": "all",
"killedBy": [
"static.case.ts#reads the unit table"
],
"assertionKilledBy": [
"static.case.ts#reads the unit table"
],
"tests": {
"static.case.ts#reads the unit table": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "static: a mutant hit in a describe body during collection is static",
"scope": "main",
"mutant": {
"file": "src/static-table.ts",
"location": "12:10-12:21",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "Killed",
"static": true,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [],
"selected": "all",
"killedBy": [
"static.case.ts#a suite whose body calls production code > keeps the label computed during collection"
],
"assertionKilledBy": [
"static.case.ts#a suite whose body calls production code > keeps the label computed during collection"
],
"tests": {
"static.case.ts#a suite whose body calls production code > keeps the label computed during collection": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "static: a mutant hit in a production-computed it.each table is static, and its changed test set is invalid, not a non-kill",
"scope": "main",
"mutant": {
"file": "src/static-table.ts",
"location": "20:5-20:11",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "RuntimeError",
"static": true,
"outcome": "invalid",
"outcomeSource": "sidecar",
"coveredBy": [],
"selected": "all",
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [],
"reason": "has full name \"row \", the run manifest expects \"row beta\"",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "hooks: a mutant executed only in a suite's beforeAll is attributed to every test of that suite",
"scope": "main",
"mutant": {
"file": "src/hooks.ts",
"location": "7:11-7:19",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"hooks.case.ts#records loaded in beforeAll > loads one record",
"hooks.case.ts#records loaded in beforeAll > names the record"
],
"selected": [
"hooks.case.ts#records loaded in beforeAll > loads one record",
"hooks.case.ts#records loaded in beforeAll > names the record"
],
"killedBy": [
"hooks.case.ts#records loaded in beforeAll > names the record"
],
"assertionKilledBy": [
"hooks.case.ts#records loaded in beforeAll > names the record"
],
"tests": {
"hooks.case.ts#records loaded in beforeAll > names the record": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [
"hooks.case.ts#records loaded in beforeAll > loads one record",
"hooks.case.ts#records loaded in beforeAll > names the record"
],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: under disableBail two tests kill the same mutant and both are reported",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "7:10-7:19",
"mutatorName": "ArithmeticOperator",
"replacement": "value / 2"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#doubles three",
"runtime.case.ts#doubles two"
],
"selected": [
"runtime.case.ts#doubles three",
"runtime.case.ts#doubles two"
],
"killedBy": [
"runtime.case.ts#doubles three",
"runtime.case.ts#doubles two"
],
"assertionKilledBy": [
"runtime.case.ts#doubles three",
"runtime.case.ts#doubles two"
],
"tests": {
"runtime.case.ts#doubles three": "killed-by-test:assertion",
"runtime.case.ts#doubles two": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "rows: it.each rows whose titles truncate to one name keep distinct ids by row ordinal",
"scope": "main",
"mutant": {
"file": "src/rows.ts",
"location": "6:40-6:51",
"mutatorName": "ArithmeticOperator",
"replacement": "sum - value"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1"
],
"selected": [
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1"
],
"killedBy": [
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1"
],
"assertionKilledBy": [
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1"
],
"tests": {
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0": "killed-by-test:assertion",
"rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: a mutant that throws a TypeError is Killed in Stryker's report and runtime in the sidecar",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "13:10-13:22",
"mutatorName": "OptionalChaining",
"replacement": "text.charAt"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:runtime",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#reads the first character of nothing"
],
"selected": [
"runtime.case.ts#reads the first character of nothing"
],
"killedBy": [
"runtime.case.ts#reads the first character of nothing"
],
"assertionKilledBy": [],
"tests": {
"runtime.case.ts#reads the first character of nothing": "killed-by-test:runtime"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "hooks: a failing beforeAll is killed-outside-test:hook, attributed to the suite's tests",
"scope": "main",
"mutant": {
"file": "src/hooks.ts",
"location": "13:7-13:12",
"mutatorName": "BooleanLiteral",
"replacement": "open"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-outside-test:hook",
"outcomeSource": "sidecar",
"coveredBy": [
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check"
],
"selected": [
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check"
],
"killedBy": [
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check"
],
"assertionKilledBy": [],
"tests": {
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check": "killed-outside-test:hook",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check": "killed-outside-test:hook"
},
"outsideTest": [
{
"file": "hooks.case.ts",
"outcome": "killed-outside-test:hook",
"attributedTo": [
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check"
]
}
],
"reason": null,
"flags": [],
"hookAttributedTests": [
"hooks.case.ts#a suite whose beforeAll checks the connection > runs a second test after the opening check",
"hooks.case.ts#a suite whose beforeAll checks the connection > runs after the opening check"
],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "hooks: a failing afterAll leaves its tests passing and is killed-outside-test:hook",
"scope": "main",
"mutant": {
"file": "src/hooks.ts",
"location": "19:7-19:14",
"mutatorName": "BooleanLiteral",
"replacement": "closed"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-outside-test:hook",
"outcomeSource": "sidecar",
"coveredBy": [
"hooks.case.ts#a suite whose afterAll checks the connection > runs before the closing check"
],
"selected": [
"hooks.case.ts#a suite whose afterAll checks the connection > runs before the closing check"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [
{
"file": "hooks.case.ts",
"outcome": "killed-outside-test:hook",
"attributedTo": []
}
],
"reason": null,
"flags": [],
"hookAttributedTests": [
"hooks.case.ts#a suite whose afterAll checks the connection > runs before the closing check"
],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "static: a mutant that makes a test file fail to import is killed-outside-test:collection, never invalid or Survived",
"scope": "main",
"mutant": {
"file": "src/boot.ts",
"location": "10:38-10:46",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "Killed",
"static": true,
"outcome": "killed-outside-test:collection",
"outcomeSource": "sidecar",
"coveredBy": [],
"selected": "all",
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [
{
"file": "import-failure.case.ts",
"outcome": "killed-outside-test:collection",
"attributedTo": []
}
],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: an infinite loop whose try/catch swallows the hit-limit error is a timeout through the hit count",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "21:47-21:54",
"mutatorName": "UpdateOperator",
"replacement": "index--"
},
"expect": {
"status": "Timeout",
"static": false,
"outcome": "timeout",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#counts three items"
],
"selected": [
"runtime.case.ts#counts three items"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {
"runtime.case.ts#counts three items": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": "Hit limit reached (301/300)",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: an unhandled rejection during a mutant run is infrastructure",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "33:7-33:13",
"mutatorName": "BooleanLiteral",
"replacement": "quiet"
},
"expect": {
"status": "RuntimeError",
"static": false,
"outcome": "infrastructure",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#schedules no audit when quiet"
],
"selected": [
"runtime.case.ts#schedules no audit when quiet"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [],
"reason": "unhandled error during the run: Error: runtime: the audit failed",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: a mutant covered by no test is NoCoverage in Stryker and survived when normalised",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "52:10-52:21",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "NoCoverage",
"static": false,
"outcome": "survived",
"outcomeSource": "no-coverage",
"coveredBy": [],
"selected": null,
"killedBy": [],
"assertionKilledBy": [],
"tests": null,
"outsideTest": null,
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: a test failing with Vitest's own testTimeout error is class timeout",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "39:7-39:12",
"mutatorName": "BooleanLiteral",
"replacement": "fast"
},
"expect": {
"status": "Timeout",
"static": false,
"outcome": "timeout",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#settles without waiting"
],
"selected": [
"runtime.case.ts#settles without waiting"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {
"runtime.case.ts#settles without waiting": "timeout"
},
"outsideTest": [],
"reason": "Test timed out in 2000ms",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "state: the first of two consecutive mutants in one process is killed",
"scope": "main",
"mutant": {
"file": "src/counter.ts",
"location": "10:3-10:14",
"mutatorName": "AssignmentOperator",
"replacement": "issued -= 1"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#issues the first ticket"
],
"selected": [
"runtime.case.ts#issues the first ticket"
],
"killedBy": [
"runtime.case.ts#issues the first ticket"
],
"assertionKilledBy": [
"runtime.case.ts#issues the first ticket"
],
"tests": {
"runtime.case.ts#issues the first ticket": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "state: the second of two consecutive mutants shows no state from the first and survives",
"scope": "main",
"mutant": {
"file": "src/counter.ts",
"location": "12:15-12:33",
"mutatorName": "StringLiteral",
"replacement": "``"
},
"expect": {
"status": "Survived",
"static": false,
"outcome": "survived",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#issues the first ticket"
],
"selected": [
"runtime.case.ts#issues the first ticket"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "runtime: a mutant run that selects tests but runs zero of them is invalid",
"scope": "main",
"mutant": {
"file": "src/runtime.ts",
"location": "46:10-46:15",
"mutatorName": "BooleanLiteral",
"replacement": "true"
},
"expect": {
"status": "RuntimeError",
"static": false,
"outcome": "invalid",
"outcomeSource": "sidecar",
"coveredBy": [
"runtime.case.ts#runs while the feature is enabled"
],
"selected": [
"runtime.case.ts#runs while the feature is enabled"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {},
"outsideTest": [],
"reason": "did not run (state skipped); zero tests ran",
"flags": [
"zero-tests"
],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "retry: a test with retry: 1 makes the run of a mutant it covers inconclusive",
"scope": "retry",
"mutant": {
"file": "src/retry.ts",
"location": "7:10-7:22",
"mutatorName": "ArithmeticOperator",
"replacement": "attempt / 10"
},
"expect": {
"status": "RuntimeError",
"static": false,
"outcome": "inconclusive",
"outcomeSource": "sidecar",
"coveredBy": [
"retry.case.ts#waits ten milliseconds per attempt"
],
"selected": [
"retry.case.ts#waits ten milliseconds per attempt"
],
"killedBy": [],
"assertionKilledBy": [],
"tests": {
"retry.case.ts#waits ten milliseconds per attempt": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": "retry.case.ts#waits ten milliseconds per attempt has retry 1",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "retry: a static mutant selects every test, the retry test too, so its run is inconclusive",
"scope": "retry",
"mutant": {
"file": "src/retry.ts",
"location": "11:29-11:37",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "RuntimeError",
"static": true,
"outcome": "inconclusive",
"outcomeSource": "sidecar",
"coveredBy": [],
"selected": "all",
"killedBy": [],
"assertionKilledBy": [],
"tests": {
"retry.case.ts#uses the linear policy": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": "retry.case.ts#waits ten milliseconds per attempt has retry 1",
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "selection: with testIds set to one of two covering tests, only the selected test covers, runs and kills the mutant",
"scope": "selection",
"mutant": {
"file": "src/selection.ts",
"location": "8:10-8:19",
"mutatorName": "ArithmeticOperator",
"replacement": "value / 3"
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"selection.case.ts#scaling > scales two"
],
"selected": "all",
"killedBy": [
"selection.case.ts#scaling > scales two"
],
"assertionKilledBy": [
"selection.case.ts#scaling > scales two"
],
"tests": {
"selection.case.ts#scaling > scales two": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [],
"attemptsDisagree": false
}
},
{
"kind": "mutant",
"name": "selection: a mutant hit only in the suite beforeAll carries the hook marker for the selected test only",
"scope": "selection",
"mutant": {
"file": "src/selection.ts",
"location": "14:10-14:13",
"mutatorName": "StringLiteral",
"replacement": "\"\""
},
"expect": {
"status": "Killed",
"static": false,
"outcome": "killed-by-test:assertion",
"outcomeSource": "sidecar",
"coveredBy": [
"selection.case.ts#scaling > scales two"
],
"selected": "all",
"killedBy": [
"selection.case.ts#scaling > scales two"
],
"assertionKilledBy": [
"selection.case.ts#scaling > scales two"
],
"tests": {
"selection.case.ts#scaling > scales two": "killed-by-test:assertion"
},
"outsideTest": [],
"reason": null,
"flags": [],
"hookAttributedTests": [
"selection.case.ts#scaling > scales two"
],
"attemptsDisagree": false
}
},
{
"kind": "tests",
"name": "rows: it.each rows whose titles truncate to one name are flagged not oracle-validated",
"scope": "main",
"tests": [
{
"id": "rows.case.ts#totals a row label long enough for Vitest to t…:11:3:0",
"skipped": false,
"notOracleValidated": true
},
{
"id": "rows.case.ts#totals a row label long enough for Vitest to t…:11:3:1",
"skipped": false,
"notOracleValidated": true
}
]
},
{
"kind": "tests",
"name": "selection: a dry run with testIds runs the selected test and lists the other one as skipped",
"scope": "selection",
"tests": [
{
"id": "selection.case.ts#scaling > scales one",
"skipped": true,
"notOracleValidated": false
},
{
"id": "selection.case.ts#scaling > scales two",
"skipped": false,
"notOracleValidated": false
}
]
},
{
"kind": "processes",
"name": "processes: mutants handled by processes that never ran the dry run still select the right tests through the run manifest",
"profile": "concurrency-2-reuse-1",
"rule": "one-mutant-run-per-process",
"consecutive": null
},
{
"kind": "processes",
"name": "processes: one runner process handles two consecutive mutants, and the second run shows no state from the first",
"profile": "concurrency-1-reuse-unlimited",
"rule": "every-run-in-the-dry-run-process",
"consecutive": [
"state: the first of two consecutive mutants in one process is killed",
"state: the second of two consecutive mutants shows no state from the first and survives"
]
},
{
"kind": "environment",
"name": "environment: exporting __STRYKER_ACTIVE_MUTANT__ in the parent environment changes no result",
"profile": "concurrency-1-reuse-unlimited",
"activeMutantOf": "static: a mutant in a module-level table is killed with the module re-imported fresh",
"localId": "60"
}
]
}

47
gitnexus/test/fixtures/mutation-conformance/hooks.case.ts vendored Normal file
View file

@ -0,0 +1,47 @@
// Conformance fixture (gate 2): mutants hit only in suite hooks.
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { assertClosed, assertOpen, loadRecords } from './src/hooks.js';
describe('records loaded in beforeAll', () => {
let records: string[] = [];
beforeAll(() => {
records = loadRecords();
});
it('loads one record', () => {
expect(records).toHaveLength(1);
});
it('names the record', () => {
expect(records[0]).toBe('record');
});
});
describe('a suite whose beforeAll checks the connection', () => {
beforeAll(() => {
assertOpen(true);
});
it('runs after the opening check', () => {
expect(true).toBe(true);
});
it('runs a second test after the opening check', () => {
expect(false).toBe(false);
});
});
describe('a suite whose afterAll checks the connection', () => {
afterAll(() => {
assertClosed(true);
});
it('runs before the closing check', () => {
expect(true).toBe(true);
});
});
it('runs outside every suite with hooks', () => {
expect(loadRecords).toBeTypeOf('function');
});

7
gitnexus/test/fixtures/mutation-conformance/import-failure.case.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Conformance fixture (gate 2): the static mutant of src/boot.ts makes this file fail to import.
import { expect, it } from 'vitest';
import { BOOT_MODE } from './src/boot.js';
it('boots in strict mode', () => {
expect(BOOT_MODE).toBe('strict');
});

11
gitnexus/test/fixtures/mutation-conformance/retry.case.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Conformance fixture (gate 2), retry scope: one test declares `retry: 1`.
import { expect, it } from 'vitest';
import { RETRY_POLICY, retryDelay } from './src/retry.js';
it('waits ten milliseconds per attempt', { retry: 1 }, () => {
expect(retryDelay(2)).toBe(20);
});
it('uses the linear policy', () => {
expect(RETRY_POLICY).toBe('linear');
});

13
gitnexus/test/fixtures/mutation-conformance/rows.case.ts vendored Normal file
View file

@ -0,0 +1,13 @@
// Conformance fixture (gate 2): `it.each` rows whose titles truncate to the same name (Vitest
// truncates `$label` at 40 characters), so their ids need the row ordinal.
import { expect, it } from 'vitest';
import { total } from './src/rows.js';
const PREFIX = 'a row label long enough for Vitest to truncate it: ';
it.each([
{ label: `${PREFIX}first`, values: [1, 2], sum: 3 },
{ label: `${PREFIX}second`, values: [2, 3], sum: 5 },
])('totals $label', ({ values, sum }) => {
expect(total(values)).toBe(sum);
});

52
gitnexus/test/fixtures/mutation-conformance/runtime.case.ts vendored Normal file
View file

@ -0,0 +1,52 @@
// Conformance fixture (gate 2): runtime mutants, each covered by the tests that call it.
import { expect, it } from 'vitest';
import { nextTicket } from './src/counter.js';
import {
countItems,
double,
featureDisabled,
firstChar,
scheduleAudit,
settleDelay,
} from './src/runtime.js';
const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => {
setTimeout(resolve, ms).unref();
});
it('doubles two', () => {
expect(double(2)).toBe(4);
});
it('doubles three', () => {
expect(double(3)).toBe(6);
});
it('reads the first character of nothing', () => {
expect(firstChar(null)).toBe('');
});
it('counts three items', () => {
expect(countItems([1, 2, 3])).toBe(3);
});
it('schedules no audit when quiet', async () => {
scheduleAudit(true);
await sleep(20);
expect(true).toBe(true);
});
it('settles without waiting', async () => {
await sleep(settleDelay(true));
expect(true).toBe(true);
});
it('issues the first ticket', () => {
expect(nextTicket()).toBe(1);
});
it('runs while the feature is enabled', (context) => {
context.skip(featureDisabled(), 'the feature is disabled');
expect(true).toBe(true);
});

21
gitnexus/test/fixtures/mutation-conformance/selection.case.ts vendored Normal file
View file

@ -0,0 +1,21 @@
// Conformance fixture (gate 2), selection scope: the scope's testIds select "scales two" only, so
// the dry run skips "scales one" and every coverage, outcome and hook marker names only the
// selected test.
import { beforeAll, describe, expect, it } from 'vitest';
import { scale, unit } from './src/selection.js';
describe('scaling', () => {
let suffix = '';
beforeAll(() => {
suffix = unit();
});
it('scales one', () => {
expect(`${scale(1)}${suffix}`).toBe('3x');
});
it('scales two', () => {
expect(`${scale(2)}${suffix}`).toBe('6x');
});
});

10
gitnexus/test/fixtures/mutation-conformance/src/boot.ts vendored Normal file
View file

@ -0,0 +1,10 @@
// Stryker disable all
// Conformance fixture (gate 2): a module whose mutated form throws while it is imported.
function requireMode(mode: string): string {
if (mode === '') throw new Error('boot: the mode must not be empty');
return mode;
}
// Stryker restore next-line StringLiteral
export const BOOT_MODE = requireMode('strict');

18
gitnexus/test/fixtures/mutation-conformance/src/counter.ts vendored Normal file
View file

@ -0,0 +1,18 @@
// Stryker disable all
// Conformance fixture (gate 2): module-level state that would show carry-over between two
// mutant runs in one runner process.
let issued = 0;
let lastLabel = '';
export function nextTicket(): number {
// Stryker restore next-line AssignmentOperator
issued += 1;
// Stryker restore next-line StringLiteral
lastLabel = `ticket ${issued}`;
return issued;
}
export function lastTicketLabel(): string {
return lastLabel;
}

20
gitnexus/test/fixtures/mutation-conformance/src/hooks.ts vendored Normal file
View file

@ -0,0 +1,20 @@
// Stryker disable all
// Conformance fixture (gate 2): production code called only from suite hooks.
/** Called only in a suite's `beforeAll`. */
export function loadRecords(): string[] {
// Stryker restore next-line StringLiteral
return ['record'];
}
/** Called in a `beforeAll`: the mutant makes the hook throw. */
export function assertOpen(open: boolean): void {
// Stryker restore next-line BooleanLiteral
if (!open) throw new Error('hooks: the connection is not open');
}
/** Called in an `afterAll`: the mutant makes the hook throw after the tests passed. */
export function assertClosed(closed: boolean): void {
// Stryker restore next-line BooleanLiteral
if (!closed) throw new Error('hooks: the connection is still open');
}

11
gitnexus/test/fixtures/mutation-conformance/src/retry.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Stryker disable all
// Conformance fixture (gate 2), retry scope: a test with `retry: 1` makes every run that selects
// it inconclusive, static runs (which select every test) included.
export function retryDelay(attempt: number): number {
// Stryker restore next-line ArithmeticOperator
return attempt * 10;
}
// Stryker restore next-line StringLiteral
export const RETRY_POLICY = 'linear';

7
gitnexus/test/fixtures/mutation-conformance/src/rows.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Stryker disable all
// Conformance fixture (gate 2): summed by `it.each` rows whose titles truncate to one name.
export function total(values: readonly number[]): number {
// Stryker restore next-line ArithmeticOperator
return values.reduce((sum, value) => sum + value, 0);
}

53
gitnexus/test/fixtures/mutation-conformance/src/runtime.ts vendored Normal file
View file

@ -0,0 +1,53 @@
// Stryker disable all
// Conformance fixture (gate 2): runtime mutants, each hit only while a test runs.
/** Two tests kill its mutant. */
export function double(value: number): number {
// Stryker restore next-line ArithmeticOperator
return value * 2;
}
/** The mutant drops the optional chain, so a null argument throws a TypeError. */
export function firstChar(text: string | null): string {
// Stryker restore next-line OptionalChaining
return text?.charAt(0) ?? '';
}
/** The mutant never ends the loop; the catch swallows the hit-limit error too. */
export function countItems(items: readonly number[]): number {
let counted = 0;
try {
// Stryker restore next-line UpdateOperator
for (let index = 0; index < items.length; index++) {
counted += 1;
}
} catch {
// Swallowed on purpose: the run must still end as a timeout through the hit count.
}
return counted;
}
/** The mutant leaves a rejected promise without a handler. */
export function scheduleAudit(quiet: boolean): void {
// Stryker restore next-line BooleanLiteral
if (!quiet) void Promise.reject(new Error('runtime: the audit failed'));
}
/** The mutant turns the delay into one far above the fixture's testTimeout. */
export function settleDelay(fast: boolean): number {
// Stryker restore next-line BooleanLiteral
if (!fast) return 10_000;
return 0;
}
/** The mutant makes the only test that covers it skip itself. */
export function featureDisabled(): boolean {
// Stryker restore next-line BooleanLiteral
return false;
}
/** No test calls it. */
export function neverCalled(): string {
// Stryker restore next-line StringLiteral
return 'unreached';
}

15
gitnexus/test/fixtures/mutation-conformance/src/selection.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Stryker disable all
// Conformance fixture (gate 2), selection scope: both tests of selection.case.ts cover this code,
// and the scope's testIds select one of them (Phase 1 KTD2).
/** Called in each test. */
export function scale(value: number): number {
// Stryker restore next-line ArithmeticOperator
return value * 3;
}
/** Called only in the suite's `beforeAll`. */
export function unit(): string {
// Stryker restore next-line StringLiteral
return 'x';
}

22
gitnexus/test/fixtures/mutation-conformance/src/static-table.ts vendored Normal file
View file

@ -0,0 +1,22 @@
// Stryker disable all
// Conformance fixture (gate 2). Every mutant except the ones restored below is disabled, so each
// planted mutant is one pinned case of expected.json.
// Static: the table is built while the module loads.
// Stryker restore next-line StringLiteral
export const UNITS: Readonly<Record<string, string>> = { ms: 'milliseconds' };
/** Called from a `describe` body, so its mutant is hit while tests are collected. */
export function suiteLabel(): string {
// Stryker restore next-line StringLiteral
return 'collected';
}
/** Rows of an `it.each` table: a mutant here changes the collected test set. */
export function rowNames(): string[] {
return [
'alpha',
// Stryker restore next-line StringLiteral
'beta',
];
}

19
gitnexus/test/fixtures/mutation-conformance/static.case.ts vendored Normal file
View file

@ -0,0 +1,19 @@
// Conformance fixture (gate 2): static mutants, hit while modules load or tests are collected.
import { describe, expect, it } from 'vitest';
import { UNITS, rowNames, suiteLabel } from './src/static-table.js';
it('reads the unit table', () => {
expect(UNITS.ms).toBe('milliseconds');
});
describe('a suite whose body calls production code', () => {
const label = suiteLabel();
it('keeps the label computed during collection', () => {
expect(label).toBe('collected');
});
});
it.each(rowNames())('row %s', (name) => {
expect(['alpha', 'beta']).toContain(name);
});

37
gitnexus/test/fixtures/mutation-conformance/vitest.config.ts vendored Normal file
View file

@ -0,0 +1,37 @@
/**
* Vitest config of the gate 2 conformance fixture (U4). The conformance driver
* (tools/mutation/conformance/run-conformance.ts) runs it through the scope driver with the
* vitest5 runner: `configFile` is this file and the project is `conformance`. Only the settings
* the runner requires (forks, isolation, task locations) and the fixture's short timeouts are set,
* so a change to gitnexus/vitest.config.ts cannot move the pinned results.
*
* Like tools/mutation/vitest.mutation.config.ts, the include list is the `testFiles` of the scope
* file named by GITNEXUS_MUTATION_SCOPE. The fixture's tests are `*.case.ts` files, which no repo
* Vitest project collects.
*/
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { defineConfig } from 'vitest/config';
import {
parseMutationScope,
resolveMutationScopePath,
} from '../../../tools/mutation/scope-file.js';
const packageDir = fileURLToPath(new URL('../../../', import.meta.url));
const scope = parseMutationScope(
readFileSync(resolveMutationScopePath(process.env, packageDir), 'utf8'),
);
export default defineConfig({
test: {
pool: 'forks',
isolate: true,
includeTaskLocation: true,
// Short limits keep the timeout cases quick; every passing test finishes in milliseconds.
testTimeout: 2000,
hookTimeout: 2000,
teardownTimeout: 1000,
dangerouslyIgnoreUnhandledErrors: false,
projects: [{ extends: true, test: { name: 'conformance', include: scope.testFiles } }],
},
});

245
gitnexus/test/fixtures/mutation-controls/planted.json vendored Normal file
View file

@ -0,0 +1,245 @@
{
"schemaVersion": 1,
"kind": "gitnexus-mutation-controls",
"description": "Gate 4 positive controls (U8, KTD19): Stryker mutants of the scopes, named by global id. In every determinism profile each must be killed with an assertion by exactly its expected killers, and every listed covering non-killer must run with it and pass. Regenerate with `reselect` when a control or test goes missing (tools/mutation/controls/run-controls.ts).",
"reselect": "npm run test:mutation:controls -- --select --work <dir outside the checkout>",
"profiles": [
{
"name": "A",
"concurrency": 1,
"maxTestRunnerReuse": 0
},
{
"name": "B",
"concurrency": 3,
"maxTestRunnerReuse": 1
}
],
"timeoutMS": 10000,
"selection": {
"seed": "u8-gate4-1",
"perScope": 3,
"maxKillers": 3,
"maxNonKillers": 3,
"selectedFrom": [
"A"
]
},
"scopes": [
{
"name": "shared",
"scope": {
"testFiles": [
"test/unit/shared-type-extractors.test.ts",
"test/unit/extract-element-type-from-string.test.ts"
],
"mutate": [
"gitnexus/src/core/ingestion/type-extractors/shared.ts:729-904"
]
},
"mutants": 199,
"tests": 143,
"controls": [
{
"id": "gitnexus/src/core/ingestion/type-extractors/shared.ts@02e17f5524a110b2:849:23-849:43:Regex:5e06ba59d5c022fc",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > strips Rust mutable reference prefix: &mut User → User"
],
"expectedNonKillers": {
"ran": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns base type for non-wrapper generic: List<User> → List",
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns undefined for bare wrapper: Option",
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns undefined for primitive \"int\""
],
"notRun": [
"test/unit/extract-element-type-from-string.test.ts#extractElementTypeFromString > Go slice prefix > []string → string",
"test/unit/shared-type-extractors.test.ts#getContainerDescriptor > descriptor for Vec has get in valueMethods",
"test/unit/shared-type-extractors.test.ts#methodToTypeArgPosition > returns last for get on Map (value method)"
]
}
},
{
"id": "gitnexus/src/core/ingestion/type-extractors/shared.ts@02e17f5524a110b2:856:19-861:8:MethodExpression:16c687aaf4579f1a",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > strips nullable union: User | null → User"
],
"expectedNonKillers": {
"ran": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns undefined for genuine union: User | Repo"
],
"notRun": [
"test/unit/extract-element-type-from-string.test.ts#extractElementTypeFromString > cross-bracket nesting (bracket depth fix) > Dict[str, List[int]] → str (pos=first: key arg)",
"test/unit/extract-element-type-from-string.test.ts#extractElementTypeFromString > edge cases — return undefined > empty array suffix → undefined",
"test/unit/shared-type-extractors.test.ts#methodToTypeArgPosition > returns last for non-key method with no container"
]
}
},
{
"id": "gitnexus/src/core/ingestion/type-extractors/shared.ts@02e17f5524a110b2:862:9-862:27:ConditionalExpression:b5bea41b6c623f7c",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns undefined for genuine union: User | Repo"
],
"expectedNonKillers": {
"ran": [
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > strips nullable union: User | null → User"
],
"notRun": [
"test/unit/shared-type-extractors.test.ts#extractElementTypeFromString > returns undefined for just brackets: []",
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > returns undefined for primitive \"void\"",
"test/unit/shared-type-extractors.test.ts#extractReturnTypeName > skips Rust lifetime parameters in Result: Result<'_, User, Error> is unusual but lifetime in first pos is skipped"
]
}
}
]
},
{
"name": "schema-constants",
"scope": {
"testFiles": [
"test/unit/schema.test.ts"
],
"mutate": [
"gitnexus-shared/src/lbug/schema-constants.ts"
]
},
"mutants": 72,
"tests": 42,
"controls": [
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:12:3-12:9:StringLiteral:12ae32cb1ec02d01",
"tags": [
"relocated-shared",
"as-const"
],
"static": true,
"expectedKillers": [
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > includes all core node types"
],
"expectedNonKillers": {
"ran": [
"test/unit/schema.test.ts#LadybugDB Schema > embedding schema > creates CodeEmbedding table",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > FILE_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE File (\n id STRING,\n name STRING,\n filePath STRING,\n content STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Function schema has startLine and endLine"
],
"notRun": []
}
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:40:3-40:16:StringLiteral:12ae32cb1ec02d01",
"tags": [
"relocated-shared",
"as-const"
],
"static": true,
"expectedKillers": [
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > includes multi-language node types"
],
"expectedNonKillers": {
"ran": [
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes the DI collection-injection edge type (#2200)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > has type, confidence, reason, step properties",
"test/unit/schema.test.ts#LadybugDB Schema > schema query ordering > NODE_SCHEMA_QUERIES has correct count"
],
"notRun": []
}
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:95:3-95:18:StringLiteral:12ae32cb1ec02d01",
"tags": [
"relocated-shared",
"as-const"
],
"static": true,
"expectedKillers": [
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes the control-dependence edge types (issue #2085 M5)"
],
"expectedNonKillers": {
"ran": [
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > INTERFACE_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Interface (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Property schema preserves declaredType",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects Function to Function (CALLS)"
],
"notRun": []
}
}
]
},
{
"name": "middleware",
"scope": {
"testFiles": [
"test/unit/middleware-public-origin.test.ts"
],
"mutate": [
"gitnexus/src/server/middleware.ts"
]
},
"mutants": 302,
"tests": 31,
"controls": [
{
"id": "gitnexus/src/server/middleware.ts@424f95b28e60eab6:14:10-14:35:ConditionalExpression:fcbcf165908dd18a",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — bound host is matched on its port > treats an elided default port as that port"
],
"expectedNonKillers": {
"ran": [
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — bound host is matched on its port > rejects the bound host on the default port when it is not the bound one"
],
"notRun": [
"test/unit/middleware-public-origin.test.ts#assertServeAuthForPublicOrigin > is a no-op when no public origin is configured",
"test/unit/middleware-public-origin.test.ts#assertServeAuthForPublicOrigin > names the variable and the value in the failure, and points at both remedies",
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — public origin is matched on its port > admits an exact scheme/host/port match on a wildcard bind"
]
}
},
{
"id": "gitnexus/src/server/middleware.ts@424f95b28e60eab6:51:33-51:58:MethodExpression:faa1c3dbc55de26e",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — public origin is matched on its port > admits a bracketed IPv6 literal, on any port when configured without one"
],
"expectedNonKillers": {
"ran": [
"test/unit/middleware-public-origin.test.ts#logOriginPolicy > still warns on a wildcard bind with a valid public origin, and names it"
],
"notRun": [
"test/unit/middleware-public-origin.test.ts#assertServeAuthForPublicOrigin > treats a blank value (\"\") as unset",
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — bound host is matched on its port > keeps loopback port-agnostic — the dev UI runs on its own port",
"test/unit/middleware-public-origin.test.ts#logOriginPolicy > names the resolved hostname at info when a public origin is configured"
]
}
},
{
"id": "gitnexus/src/server/middleware.ts@424f95b28e60eab6:93:18-93:35:Regex:0d604fc840a74414",
"tags": [],
"static": false,
"expectedKillers": [
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — public origin is matched on its port > admits http for a bare host only when http:// is spelled out"
],
"expectedNonKillers": {
"ran": [
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — public origin is matched on its port > admits nothing extra when the configured value is not one reachable host",
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — public origin is matched on its port > leaves the Origin-less passthrough alone — the CLI sends no Origin",
"test/unit/middleware-public-origin.test.ts#logOriginPolicy > does not report a trailing-dot hostname as a working origin"
],
"notRun": [
"test/unit/middleware-public-origin.test.ts#assertServeAuthForPublicOrigin > treats a blank value (\" \") as unset",
"test/unit/middleware-public-origin.test.ts#createWriteOriginGuard — bound host is matched on its port > matches any port when no bound port is given",
"test/unit/middleware-public-origin.test.ts#logOriginPolicy > warns on a wildcard bind with no public origin, pointing at both remedies"
]
}
}
]
}
]
}

642
gitnexus/test/fixtures/mutation-oracle/envelope.json vendored Normal file
View file

@ -0,0 +1,642 @@
{
"schemaVersion": 1,
"description": "Gate 1 (U6): the vitest5 runner’s unmutated run matched a normal Vitest 5 run on the pure-JS category under these versions and settings. Results from other Vitest or Stryker versions are unvalidated until gates 2 and 3 pass again (KTD2). Regenerate with `npm run test:mutation:fidelity` (from gitnexus/).",
"versions": {
"vitest": "5.0.5",
"stryker": "10.0.0",
"node": "v22.23.3"
},
"platform": {
"os": "linux",
"release": "6.6.114.1-microsoft-standard-WSL2",
"arch": "x64"
},
"stryker": {
"concurrency": 2,
"maxTestRunnerReuse": 0,
"testRunner": "vitest5",
"coverageAnalysis": "perTest",
"disableTypeChecks": false,
"timeoutMS": 10000
},
"config": {
"base": {
"file": "gitnexus/vitest.config.ts",
"project": "default"
},
"run": {
"file": "gitnexus/tools/mutation/vitest.mutation.config.ts",
"project": "mutation"
},
"settings": {
"pool": "forks",
"isolate": true,
"maxWorkers": 23,
"globals": true,
"environment": "node",
"testTimeout": 30000,
"hookTimeout": 120000,
"teardownTimeout": 3000,
"clearMocks": false,
"mockReset": false,
"restoreMocks": false,
"unstubEnvs": false,
"unstubGlobals": false,
"includeTaskLocation": true,
"env": {
"GITNEXUS_ATOMIC_INCREMENTAL": "",
"GITNEXUS_LANG": "en",
"GITNEXUS_MEMORY": "off"
},
"sequence": {
"groupOrder": 3,
"hooks": "stack",
"sequencer": "[function PerfSequencer]"
}
},
"allowedDifferences": [
{
"name": "setupFiles",
"paths": [
"project.setupFiles"
],
"reason": "The runner prepends its own setup file (tools/mutation/runner/setup.ts, loaded from the prepared copy next to the plugin) for coverage and mutant activation (KTD4). The gate 1 recorder is installed in both runs the same way."
},
{
"name": "coverage",
"paths": [
"global.coverage.exclude",
"project.coverage.exclude"
],
"reason": "Vitest coverage is off in both runs; its resolved exclude list names the config file. Stryker records its own per-test coverage through the runner setup file."
},
{
"name": "KTD8 aliases",
"paths": [
"vite.resolve.alias"
],
"reason": "The mutation config resolves `gitnexus-shared` imports to the relocated sources inside the sandbox (KTD8); the base run resolves the package to its built `dist`. The module comparison checks both resolve to the same modules."
},
{
"name": "dangerouslyIgnoreUnhandledErrors",
"paths": [
"global._rawTestConfig.dangerouslyIgnoreUnhandledErrors",
"global.dangerouslyIgnoreUnhandledErrors",
"project.dangerouslyIgnoreUnhandledErrors"
],
"reason": "Off in the mutation config so an unhandled error fails the run (KTD9); gate 1 reports any unhandled error of the base run separately."
},
{
"name": "cacheDir",
"paths": [
"global.cache",
"global.cliOptions.cache",
"global.cliOptions.fsModuleCache",
"global.fsModuleCache",
"global.providedOptions.fsModuleCache",
"global.viteOverrides.cacheDir",
"project.cache.dir",
"project.fsModuleCache",
"vite.cacheDir"
],
"reason": "The runner uses a per-process Vite cache directory inside the sandbox and turns the results and fs module caches off (KTD5), so no run reuses another run’s transforms."
},
{
"name": "project name",
"paths": [
"global.cliOptions.project",
"global.project",
"project.name"
],
"reason": "The runner runs the single `mutation` project; Vitest task ids depend on the name, which is why tests are compared by file, titles and location instead."
},
{
"name": "include and exclude lists",
"paths": [
"env.GITNEXUS_MUTATION_SCOPE",
"project.exclude",
"project.include"
],
"reason": "The mutation project includes exactly the scope file’s test files (named by GITNEXUS_MUTATION_SCOPE); gate 1 runs the same file list on both sides."
},
{
"name": "projects array",
"paths": [
"global.projects",
"global.resolvedProjects"
],
"reason": "The mutation config replaces the three repo projects with one (KTD9); the resolved project list follows it. The run project’s own config is compared key by key."
},
{
"name": "sequence.groupOrder",
"paths": [
"project.sequence.groupOrder"
],
"reason": "Orders projects against each other; a single-project run has nothing to order."
},
{
"name": "config file (addition)",
"paths": [
"global.cliOptions.config",
"global.config"
],
"reason": "Addition to the plan’s list: the name of the config file. The Stryker run loads tools/mutation/vitest.mutation.config.ts (KTD9) through the gate 1 wrapper tools/mutation/fidelity/vitest.fidelity.config.ts; their settings are compared key by key."
}
],
"ignoredVolatileValues": [
{
"name": "sequence.seed",
"paths": [
"global.sequence.seed",
"project.sequence.seed"
],
"reason": "Defaults to the start time when unset and only drives shuffling; ignored only while shuffling is off in both runs."
}
]
},
"envReadsAtLoad": [
{
"name": "DEBUG",
"set": false,
"value": null,
"readers": [
"node_modules/debug/src/node.js"
]
},
{
"name": "FORCE_COLOR",
"set": false,
"value": null,
"readers": [
"node_modules/supports-color/index.js"
]
},
{
"name": "GITNEXUS_CONTENT_RETENTION",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/content-retention.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_API_KEY",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_DIMS",
"set": false,
"value": null,
"readers": [
"gitnexus/src/core/lbug/schema.ts",
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_MAX_ATTEMPTS",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_MIN_INTERVAL_MS",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_MODEL",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_RETRY_CAP_MS",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_EMBEDDING_URL",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/http-embedding-body-retry.test.ts"
]
},
{
"name": "GITNEXUS_HOME",
"set": true,
"value": "/home/node/.gitnexus",
"readers": [
"gitnexus/test/unit/storage-resolver.test.ts"
]
},
{
"name": "GITNEXUS_MAX_CALLABLE_VALUE_TARGETS",
"set": false,
"value": null,
"readers": [
"gitnexus/src/core/ingestion/scope-resolution/passes/callable-value-flow.ts"
]
},
{
"name": "GITNEXUS_MAX_FILE_SIZE",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/max-file-size.test.ts"
]
},
{
"name": "GITNEXUS_MAX_INTERFACE_DISPATCH_FANOUT",
"set": false,
"value": null,
"readers": [
"gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts"
]
},
{
"name": "GITNEXUS_MAX_PROPERTY_DISPATCH_FANOUT",
"set": false,
"value": null,
"readers": [
"gitnexus/src/core/ingestion/scope-resolution/passes/property-dispatch.ts"
]
},
{
"name": "GITNEXUS_PUBLIC_ORIGIN",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/middleware-public-origin.test.ts"
]
},
{
"name": "GITNEXUS_RD_FUZZ_N",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/cfg/reaching-defs-equivalence.test.ts"
]
},
{
"name": "GITNEXUS_STORAGE_PATH",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/storage-resolver.test.ts"
]
},
{
"name": "GITNEXUS_STORAGE_ROOT",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/storage-resolver.test.ts"
]
},
{
"name": "GITNEXUS_TRUST_PROXY",
"set": false,
"value": null,
"readers": [
"gitnexus/test/unit/middleware-trust-proxy.test.ts"
]
},
{
"name": "NODE_ENV",
"set": true,
"value": null,
"readers": [
"gitnexus/src/core/ingestion/utils/env.ts",
"gitnexus/test/unit/scope-resolution/reconcile-ownership.test.ts"
]
},
{
"name": "NODE_V8_COVERAGE",
"set": false,
"value": null,
"readers": [
"node_modules/thread-stream/index.js"
]
},
{
"name": "VALIDATE_SEMANTIC_MODEL",
"set": false,
"value": null,
"readers": [
"gitnexus/src/core/ingestion/utils/env.ts",
"gitnexus/test/unit/scope-resolution/reconcile-ownership.test.ts"
]
}
],
"accepted": {
"fileCount": 251,
"testCount": 4667,
"ran": 4666,
"skipped": 1,
"files": [
"test/unit/ai-context-read-path-must.test.ts",
"test/unit/ai-context-unknown-risk-policy.test.ts",
"test/unit/analysis-features.test.ts",
"test/unit/analyze-job.test.ts",
"test/unit/analyze-worker-ipc.test.ts",
"test/unit/binding-accumulator.test.ts",
"test/unit/c-static-linkage-side-channel.test.ts",
"test/unit/call-processor-routes.test.ts",
"test/unit/call-routing/ruby.test.ts",
"test/unit/cfg-callees-of-block.test.ts",
"test/unit/cfg/cfg-builder.test.ts",
"test/unit/cfg/opt-in.test.ts",
"test/unit/cfg/post-dominators.test.ts",
"test/unit/cfg/reaching-def-reason-codec.test.ts",
"test/unit/cfg/reaching-defs-equivalence.test.ts",
"test/unit/cfg/synthetic-escape.test.ts",
"test/unit/cjk-segmentation.test.ts",
"test/unit/cli-group-status-format.test.ts",
"test/unit/cli-i18n.test.ts",
"test/unit/cli-int-option.test.ts",
"test/unit/cli-message.test.ts",
"test/unit/clone-safety-cloneable.test.ts",
"test/unit/clone-safety-payloads.test.ts",
"test/unit/clone-safety.test.ts",
"test/unit/cobol-copy-expander.test.ts",
"test/unit/cobol-copy-external-imports.test.ts",
"test/unit/cobol-preprocessor.test.ts",
"test/unit/compatible-stdio-transport.test.ts",
"test/unit/conn-lock.test.ts",
"test/unit/constant-resolver.test.ts",
"test/unit/content-retention.test.ts",
"test/unit/csharp-namespace-extraction.test.ts",
"test/unit/csv-escaping.test.ts",
"test/unit/cypher-escape.test.ts",
"test/unit/dart-package-dependencies.test.ts",
"test/unit/definition-properties.test.ts",
"test/unit/django-root-discovery.test.ts",
"test/unit/embedding-checkpoint.test.ts",
"test/unit/embedding-count.test.ts",
"test/unit/embedding-dims.test.ts",
"test/unit/embedding-recovery.test.ts",
"test/unit/embedding-restore-spill.test.ts",
"test/unit/env.test.ts",
"test/unit/eval-server-bind-restriction.test.ts",
"test/unit/expo-routes.test.ts",
"test/unit/extension-load-error.test.ts",
"test/unit/extract-element-type-from-string.test.ts",
"test/unit/fastapi-router-bindings.test.ts",
"test/unit/fetch-reason-parsing.test.ts",
"test/unit/fetch-wrapper-index.test.ts",
"test/unit/format-elapsed.test.ts",
"test/unit/format-path.test.ts",
"test/unit/framework-route-paths.test.ts",
"test/unit/fts-rss-verdict.test.ts",
"test/unit/fts-schema.test.ts",
"test/unit/git-ref.test.ts",
"test/unit/gitnexus-rc-embeddings.test.ts",
"test/unit/go-route-handler.test.ts",
"test/unit/graph.test.ts",
"test/unit/group/config-parser.test.ts",
"test/unit/group/fs-utils.test.ts",
"test/unit/group/group-path-utils.test.ts",
"test/unit/group/group-tools.test.ts",
"test/unit/group/manifest-extractor.test.ts",
"test/unit/group/matching.test.ts",
"test/unit/group/service-boundary-detector.test.ts",
"test/unit/group/storage.test.ts",
"test/unit/group/types.test.ts",
"test/unit/hf-env.test.ts",
"test/unit/http-client-safe-url.test.ts",
"test/unit/http-embedding-body-retry.test.ts",
"test/unit/impact-risk.test.ts",
"test/unit/import-cycles.test.ts",
"test/unit/import-path-candidates.test.ts",
"test/unit/import-resolvers/csharp-csproj-parity.test.ts",
"test/unit/import-resolvers/suffix-index-lazy-dir-map.test.ts",
"test/unit/incremental-derived-writeback.test.ts",
"test/unit/incremental-escalation-gate.test.ts",
"test/unit/incremental-file-hash.test.ts",
"test/unit/incremental-parse-cache.test.ts",
"test/unit/incremental-shadow-candidates.test.ts",
"test/unit/incremental-subgraph-extract.test.ts",
"test/unit/index-freshness-graph-collapse.test.ts",
"test/unit/ingestion/di.test.ts",
"test/unit/ingestion/heritage-marker.test.ts",
"test/unit/ingestion/non-value-definition-keys.test.ts",
"test/unit/ingestion/qualified-name-utils.test.ts",
"test/unit/integrations/circuit-breaker.test.ts",
"test/unit/integrations/resilient-fetch.test.ts",
"test/unit/integrations/retry.test.ts",
"test/unit/ipynb-extractor.test.ts",
"test/unit/java-module-resolution.test.ts",
"test/unit/jcl-parser.test.ts",
"test/unit/kotlin-module-resolution.test.ts",
"test/unit/language-provider-route-constants.test.ts",
"test/unit/lbug-query-result-utils.test.ts",
"test/unit/lbug/graph-emit-sink.test.ts",
"test/unit/lbug/pdg-emit-sink.test.ts",
"test/unit/line-base-conversion.test.ts",
"test/unit/local-symbol-pruner.test.ts",
"test/unit/logger.test.ts",
"test/unit/max-file-size.test.ts",
"test/unit/mcp-output-budget.test.ts",
"test/unit/mcp-response-filter.test.ts",
"test/unit/mcp-stdout-sentinel.test.ts",
"test/unit/mcp-tool-arguments.test.ts",
"test/unit/mcp/query-process-attaches.test.ts",
"test/unit/method-props.test.ts",
"test/unit/middleware-public-origin.test.ts",
"test/unit/middleware-trust-proxy.test.ts",
"test/unit/model/field-registry.test.ts",
"test/unit/model/method-registry.test.ts",
"test/unit/model/owned-members-lookup.test.ts",
"test/unit/model/registration-table.test.ts",
"test/unit/model/semantic-model.test.ts",
"test/unit/model/type-registry.test.ts",
"test/unit/mutation/blank-assertions.test.ts",
"test/unit/mutation/classify-outcome.test.ts",
"test/unit/mutation/conformance-fingerprint.test.ts",
"test/unit/mutation/coverage-attribution.test.ts",
"test/unit/mutation/mutant-identity.test.ts",
"test/unit/mutation/normalise-results.test.ts",
"test/unit/mutation/oracle-boundary.test.ts",
"test/unit/mutation/oracle-classify.test.ts",
"test/unit/mutation/oracle-util.test.ts",
"test/unit/mutation/scope-file.test.ts",
"test/unit/mutation/sidecar.test.ts",
"test/unit/mutation/span-map-check.test.ts",
"test/unit/mutation/stryker-config.test.ts",
"test/unit/mutation/test-ids.test.ts",
"test/unit/mutation/version-guard.test.ts",
"test/unit/objective-c-macro-marker-preprocess.test.ts",
"test/unit/package-version.test.ts",
"test/unit/parsedfile-store.test.ts",
"test/unit/pdg-callee-id-capture.test.ts",
"test/unit/phase-timer.test.ts",
"test/unit/pipeline-runner.test.ts",
"test/unit/private-ip.test.ts",
"test/unit/process-detection-budget.test.ts",
"test/unit/python-relative-imports.test.ts",
"test/unit/query-batch.test.ts",
"test/unit/query-params.test.ts",
"test/unit/receiver-chain-codec.test.ts",
"test/unit/rel-pair-routing.test.ts",
"test/unit/rename-family.test.ts",
"test/unit/resolve-route-handler-symbols.test.ts",
"test/unit/result-merge.test.ts",
"test/unit/route-tool-detection.test.ts",
"test/unit/schema-fingerprint.test.ts",
"test/unit/schema-pair-coverage.test.ts",
"test/unit/schema.test.ts",
"test/unit/scope-extraction-failures.test.ts",
"test/unit/scope-index-store.test.ts",
"test/unit/scope-resolution/answered-receiver-drops.test.ts",
"test/unit/scope-resolution/c/c-header-scan.test.ts",
"test/unit/scope-resolution/c3-linearize-strategy.test.ts",
"test/unit/scope-resolution/callable-value-flow-env.test.ts",
"test/unit/scope-resolution/callable-value-flow-worklist.test.ts",
"test/unit/scope-resolution/callable-value-target-index.test.ts",
"test/unit/scope-resolution/cobol-import-target-parity.test.ts",
"test/unit/scope-resolution/cpp/cpp-adl-seq-coverage.test.ts",
"test/unit/scope-resolution/cpp/cpp-overload-ranking.test.ts",
"test/unit/scope-resolution/cpp/cpp-qualified-ns-index.test.ts",
"test/unit/scope-resolution/csharp/csharp-hooks.test.ts",
"test/unit/scope-resolution/csharp/csharp-interpret.test.ts",
"test/unit/scope-resolution/csharp/csharp-nullable-field-interpret.test.ts",
"test/unit/scope-resolution/def-index.test.ts",
"test/unit/scope-resolution/definition-id.test.ts",
"test/unit/scope-resolution/emit-references.test.ts",
"test/unit/scope-resolution/finalize-algorithm.test.ts",
"test/unit/scope-resolution/finalize-orchestrator.test.ts",
"test/unit/scope-resolution/free-call-fallback-guess-taint.test.ts",
"test/unit/scope-resolution/go/go-build-constraints.test.ts",
"test/unit/scope-resolution/go/go-workspace-owners-warning.test.ts",
"test/unit/scope-resolution/graph-bridge-label-split.test.ts",
"test/unit/scope-resolution/imports-to-edges-deferred.test.ts",
"test/unit/scope-resolution/imports-to-edges-type-only.test.ts",
"test/unit/scope-resolution/java/java-interpret.test.ts",
"test/unit/scope-resolution/lexical-name-claims.test.ts",
"test/unit/scope-resolution/method-dispatch-index.test.ts",
"test/unit/scope-resolution/module-scope-index.test.ts",
"test/unit/scope-resolution/name-fallback-summary.test.ts",
"test/unit/scope-resolution/namespace-channel-lookup.test.ts",
"test/unit/scope-resolution/namespace-target-cache.test.ts",
"test/unit/scope-resolution/namespace-targets-import-path.test.ts",
"test/unit/scope-resolution/overload-narrowing.test.ts",
"test/unit/scope-resolution/parse-worker-scope-integration.test.ts",
"test/unit/scope-resolution/pick-implicit-this-overload.test.ts",
"test/unit/scope-resolution/pick-unique-global-callable.test.ts",
"test/unit/scope-resolution/pick-unique-global-class.test.ts",
"test/unit/scope-resolution/position-index.test.ts",
"test/unit/scope-resolution/property-dispatch-fanout-env.test.ts",
"test/unit/scope-resolution/python/python-generic-annotation-reduction.test.ts",
"test/unit/scope-resolution/qualified-name-index.test.ts",
"test/unit/scope-resolution/reconcile-ownership.test.ts",
"test/unit/scope-resolution/references-to-edges-call-form.test.ts",
"test/unit/scope-resolution/registries.test.ts",
"test/unit/scope-resolution/resolve-ambiguous-inheritance-base.test.ts",
"test/unit/scope-resolution/resolve-references.test.ts",
"test/unit/scope-resolution/resolve-same-namespace-inheritance-base.test.ts",
"test/unit/scope-resolution/resolve-type-ref.test.ts",
"test/unit/scope-resolution/rust/rust-call-return-path.test.ts",
"test/unit/scope-resolution/rust/rust-dyn-type-normalization.test.ts",
"test/unit/scope-resolution/rust/rust-module-path.test.ts",
"test/unit/scope-resolution/scope-extractor.test.ts",
"test/unit/scope-resolution/scope-id.test.ts",
"test/unit/scope-resolution/scope-tree.test.ts",
"test/unit/scope-resolution/strip-cast-wrappers.test.ts",
"test/unit/scope-resolution/swift/callable-visibility.test.ts",
"test/unit/scope-resolution/typescript/nuxt-auto-imports.test.ts",
"test/unit/scope-resolution/undecided-satisfaction.test.ts",
"test/unit/scope-resolution/unresolved-receiver-files.test.ts",
"test/unit/scope-resolution/validate-bindings-immutability.test.ts",
"test/unit/scope-resolution/value-ref-namespace-precedence.test.ts",
"test/unit/scope-resolution/walkers-augmentations.test.ts",
"test/unit/scope-resolution/wildcard-collision-export-evidence.test.ts",
"test/unit/semantic-chunk-search.test.ts",
"test/unit/server-validation.test.ts",
"test/unit/shape-check.test.ts",
"test/unit/shard-arg.test.ts",
"test/unit/shared-type-extractors.test.ts",
"test/unit/spring-annotation-arguments.test.ts",
"test/unit/spring-aop.test.ts",
"test/unit/spring-config-drift.test.ts",
"test/unit/spring-destinations.test.ts",
"test/unit/spring-non-http-handlers.test.ts",
"test/unit/storage-resolver.test.ts",
"test/unit/storage/fs-atomic.test.ts",
"test/unit/structure-processor.test.ts",
"test/unit/suffix-index-ambiguity.test.ts",
"test/unit/swift-conditional-directive-preprocess.test.ts",
"test/unit/symbol-table.test.ts",
"test/unit/taint/call-summary-codec.test.ts",
"test/unit/taint/interproc-solver.test.ts",
"test/unit/taint/path-codec.test.ts",
"test/unit/taint/source-sink-registry.test.ts",
"test/unit/taint/summary-model.test.ts",
"test/unit/temp-dir-pool-cleanup.test.ts",
"test/unit/text-generator.test.ts",
"test/unit/topological-sort.test.ts",
"test/unit/trpc-route-extractor.test.ts",
"test/unit/utils.test.ts",
"test/unit/v8-sidecar.test.ts",
"test/unit/vendored-extension-path.test.ts",
"test/unit/vue-sfc-extractor.test.ts",
"test/unit/watch-queue.test.ts",
"test/unit/wiki-mermaid-sanitizer.test.ts",
"test/unit/windows-long-path-prefix.test.ts",
"test/unit/worker-entry.test.ts",
"test/unit/workers/quarantine.test.ts",
"test/unit/write-reconciliation.test.ts"
]
},
"skippedOnThisPlatform": [
"test/unit/storage-resolver.test.ts > storage resolver > matches a registered missing repository through a Windows extended-length path"
],
"runtimeLoads": {
"workerThreadsLoadedBy": [
"gitnexus/test/unit/ai-context-read-path-must.test.ts",
"gitnexus/test/unit/ai-context-unknown-risk-policy.test.ts",
"gitnexus/test/unit/cfg/opt-in.test.ts",
"gitnexus/test/unit/cli-message.test.ts",
"gitnexus/test/unit/cobol-copy-expander.test.ts",
"gitnexus/test/unit/cobol-copy-external-imports.test.ts",
"gitnexus/test/unit/cobol-preprocessor.test.ts",
"gitnexus/test/unit/group/manifest-extractor.test.ts",
"gitnexus/test/unit/incremental-parse-cache.test.ts",
"gitnexus/test/unit/ingestion/di.test.ts",
"gitnexus/test/unit/local-symbol-pruner.test.ts",
"gitnexus/test/unit/logger.test.ts",
"gitnexus/test/unit/max-file-size.test.ts",
"gitnexus/test/unit/middleware-public-origin.test.ts",
"gitnexus/test/unit/middleware-trust-proxy.test.ts",
"gitnexus/test/unit/parsedfile-store.test.ts",
"gitnexus/test/unit/pipeline-runner.test.ts",
"gitnexus/test/unit/scope-index-store.test.ts",
"gitnexus/test/unit/scope-resolution/cobol-import-target-parity.test.ts",
"gitnexus/test/unit/scope-resolution/finalize-orchestrator.test.ts",
"gitnexus/test/unit/scope-resolution/free-call-fallback-guess-taint.test.ts",
"gitnexus/test/unit/scope-resolution/name-fallback-summary.test.ts",
"gitnexus/test/unit/scope-resolution/namespace-target-cache.test.ts",
"gitnexus/test/unit/scope-resolution/parse-worker-scope-integration.test.ts",
"gitnexus/test/unit/scope-resolution/wildcard-collision-export-evidence.test.ts",
"gitnexus/test/unit/v8-sidecar.test.ts"
]
},
"gate": {
"result": "pass",
"differences": []
}
}

1427
gitnexus/test/fixtures/mutation-oracle/middleware.json vendored Normal file
View file

File diff suppressed because it is too large Load diff

820
gitnexus/test/fixtures/mutation-oracle/schema-constants.json vendored Normal file
View file

@ -0,0 +1,820 @@
{
"schemaVersion": 1,
"kind": "gitnexus-mutation-oracle-manifest",
"name": "schema-constants",
"description": "Gate 3 (U7): the oracle's frozen outcomes for the schema-constants scope: a seeded sample of mutants with strata top-ups, the oracle class of every mutant (U13 command run) and the per-test outcome of every sampled mutant with every test, each test launched alone (U12). Check a vitest5 runner run with `npm run test:mutation:oracle -- --scope test/fixtures/mutation-oracle/schema-constants.json,<results.json>` (from gitnexus/). When gate 3 reports \"re-freeze required\", run the refreeze command.",
"refreeze": "npm run test:mutation:oracle:freeze -- --from-manifest test/fixtures/mutation-oracle/schema-constants.json --seed <new seed> --work <dir outside the checkout>",
"scope": {
"testFiles": ["test/unit/schema.test.ts"],
"mutate": ["gitnexus-shared/src/lbug/schema-constants.ts"]
},
"versions": {
"stryker": "10.0.0",
"vitest": "5.0.5",
"nodeMajor": 22
},
"operators": {
"mutator": null,
"names": ["ArrayDeclaration","StringLiteral"]
},
"inputs": {
"closure": {
"gitnexus-shared/src/impact-risk.ts": "1a2233d9b0bc8fb5b9bcc0293703fd14047709a3df973519e7b049fd1b725946",
"gitnexus-shared/src/index.ts": "9c7c37cd2627aeed468df6a7587afdb623407fb875326ec7a15e0d8ef09d377d",
"gitnexus-shared/src/integrations/circuit-breaker.ts": "55d930f5823a35ade27ecb478b6698dad131f0bc68c69dd35820b0dd79a7d25d",
"gitnexus-shared/src/integrations/resilient-fetch.ts": "f40ca1c4bef5ddabbbf04a283b341a019cbbeaf657f67b3f31631f63a9c98c3b",
"gitnexus-shared/src/integrations/retry.ts": "b4692a9a7c38a5ec0191ed50e29e5b85bbb6cca831f7e10cddecf1010946c227",
"gitnexus-shared/src/integrations/understand-quickly.ts": "17704a5eeb2c270ca4bd6955a4828dfe65a080254a861a6c33d6128f7eea442d",
"gitnexus-shared/src/language-detection.ts": "b3b909c5b7cd07d794cd499d07cc3e8c8439a90af47941825be8896ca2324e6c",
"gitnexus-shared/src/languages.ts": "1ae6948327d84d1fd13439eb21bf842ee7f960f60f7430f07d1c84c47940a29d",
"gitnexus-shared/src/lbug/schema-constants.ts": "edf2130f7f8f5f90078f55ddae29ceefd8d05ddb447bf229d06ac77cdbc18c7f",
"gitnexus-shared/src/rename/edits.ts": "b345e588bb9f7dd266d45b7420e1a67bb952bf3f59a7a3f56445250384d3bd5a",
"gitnexus-shared/src/rename/family.ts": "25f980bbd4ec302c78372bfae3ab25962232a78e4883a215369258801aea33e7",
"gitnexus-shared/src/rename/positions.ts": "c17c2d7f17bfb900c086c9607e84234082b1bc148da6c05ece25800805503efa",
"gitnexus-shared/src/rename/results.ts": "ee289c0b68cf075e83aee3896d9b657525410c3ea9ab904ad161b5b2856d1a8b",
"gitnexus-shared/src/rename/types.ts": "c39c5cc43e2f6f40c32febac142d1557c6266e088381b13202993be812167146",
"gitnexus-shared/src/scope-resolution/def-index.ts": "92c884d9c6b393940353980a104a3293fca97b75dac85d03b1724499ac08ab88",
"gitnexus-shared/src/scope-resolution/evidence-weights.ts": "13e775c1a4cbe36263c952251aeed67774df67dfdd3487cdf2fd4bb508da7be7",
"gitnexus-shared/src/scope-resolution/finalize-algorithm.ts": "f92fc086c103e8f0a5ff17175132c57494eff12db38a619592c5bd7f7e1455f2",
"gitnexus-shared/src/scope-resolution/language-classification.ts": "d9f4a7da95c6b39ef91298c481d6b0a3a4e419a55b845438e3c8305236fe2e84",
"gitnexus-shared/src/scope-resolution/method-dispatch-index.ts": "90d3003848324661ad52fe5244ad2302de5b5cbb15910a4e46da4cad6b2972dd",
"gitnexus-shared/src/scope-resolution/module-scope-index.ts": "2205ff236a2aebd74346ae1a9e3d8ff24fbdbc13d99063bac05ddd92ebdf0850",
"gitnexus-shared/src/scope-resolution/name-claims.ts": "01a5e40b6d507e75117db5566292135f3e36ebae55f2a46732137b3354f0d741",
"gitnexus-shared/src/scope-resolution/origin-priority.ts": "0f334497f2d999d895d03f73d30c26f3e484306da6d2c11518b3a9d5f24b59ef",
"gitnexus-shared/src/scope-resolution/position-index.ts": "298bebc52fec08469db72952fb299cd46445cba5556ac5001bf81f6020169f21",
"gitnexus-shared/src/scope-resolution/qualified-name-index.ts": "e5c84085142cb978a5635dd753ca7750e6626574990a1f9232ee81a657753f17",
"gitnexus-shared/src/scope-resolution/registries/class-registry.ts": "e8d80638503ba9c3d9d1eada38708b19faa71a40cf7a85912cf41d84830139c8",
"gitnexus-shared/src/scope-resolution/registries/context.ts": "7544e9e62add75a7469d4167094bebb62bee43abd1e6fcb2956386aba89220f7",
"gitnexus-shared/src/scope-resolution/registries/evidence.ts": "b40951e441460720e1ded5c400d07b66f2e4f46ab61e734dad07a0817ea733a2",
"gitnexus-shared/src/scope-resolution/registries/field-registry.ts": "7f6f879da23b802fb8c2556ca22d9269ec2ddc07b19bdc6d064123c3e35f222b",
"gitnexus-shared/src/scope-resolution/registries/lookup-core.ts": "1f9871447c6cf0c3090d1626f0b4ad543c52a30a863ef49d71b16506e92803c7",
"gitnexus-shared/src/scope-resolution/registries/lookup-qualified.ts": "0a59cbc7c5fb3d1dd117209994b01f9b8fa625fe9d7421c9aae2928ac42f0c52",
"gitnexus-shared/src/scope-resolution/registries/macro-registry.ts": "6051812b34519649d5fdbf7540da2b1cb71cd139ed29f673262ec2b76ed1b421",
"gitnexus-shared/src/scope-resolution/registries/method-registry.ts": "ece8e14b03376f1f5a7c88d035b80ddeb631996db4801e04022859c0ab42ef60",
"gitnexus-shared/src/scope-resolution/registries/tie-breaks.ts": "61e86fe8abe75585fdbc3e6a2c5c3bb1935ca868bfc69ec5f046e63e451295ab",
"gitnexus-shared/src/scope-resolution/resolve-type-ref.ts": "087882b52ba380e8137d018e79b06d351a4407d381e0916748948c64148c949a",
"gitnexus-shared/src/scope-resolution/scope-id.ts": "9c942d4397ab620e335ed3b9b5458c30f328bdd6d1205b880600250a0c0bf743",
"gitnexus-shared/src/scope-resolution/scope-tree.ts": "66347a2b1bd61ff2089ace80c494747a87ec3cde2823019be693ef554acfb536",
"gitnexus/src/core/ingestion/model/symbol-table.ts": "773f44217a3805098c07157eab87f1a11be1a685b5d050b5a668b32073843005",
"gitnexus/src/core/ingestion/scope-resolution/graph-bridge/node-lookup.ts": "b94d6612a30ec6314791f0bb2fc28612fb4b152e6ab430c7fabf837f5833ee80",
"gitnexus/src/core/ingestion/utils/callable-labels.ts": "a81d5b15bb810632d7f04550f5203ae6e15fa0c00c9b1a2a50cf299e5988dcb8",
"gitnexus/src/core/ingestion/utils/method-props.ts": "827ede5ea1f9fb79bbbe85ad51064a36d74d1c3d01243658a904f777d97c1fa6",
"gitnexus/src/core/ingestion/utils/template-arguments.ts": "1cd981b2666c68e3215df9f3b29ea5edefce6bab85b5c5d2afea939404ffcbda",
"gitnexus/src/core/lbug/rel-pair-routing.ts": "83a6d374d129eaa2685a3e98c94d8da24818321e6f86626727af00f2a468741e",
"gitnexus/src/core/lbug/schema.ts": "289aa7b015e5c9a40fe5a58b7cef06c9bdb2d9165373e3ecae0ffe0e9f46dfdf",
"gitnexus/src/lib/utils.ts": "5c0f6b00074f239adf99d4a760a62efba80b6ba6ff309de616c99d0a36d09276",
"gitnexus/test/unit/schema.test.ts": "b077372fe9e016c4488840f8f1079aa7234ccd28ab000cc5822a3bf77da763ba"
},
"vitestConfig": {
"gitnexus/test/helpers/perf-sequencer.ts": "3aeeedffc8cf940af43c54a28c142da859d78eded2c2c7881b8094175dca94f3",
"gitnexus/test/helpers/shard-balance.ts": "5f27cc347b560e40dd6e03d7c299afaddf02975964476187ab97a2ade816c140",
"gitnexus/vitest.config.ts": "bea4e1ccad48d4742325095b53b5819a480671f01a753d5cbcc65deb2fc0e609"
},
"mutationConfig": {
"gitnexus-shared/package.json": "a0100a5b0858d80622846788097a2e596723fdf2b86602ffd93d1076f82dee13",
"gitnexus/test/helpers/perf-sequencer.ts": "3aeeedffc8cf940af43c54a28c142da859d78eded2c2c7881b8094175dca94f3",
"gitnexus/test/helpers/shard-balance.ts": "5f27cc347b560e40dd6e03d7c299afaddf02975964476187ab97a2ade816c140",
"gitnexus/tools/mutation/prepare/path-map.ts": "d595cbebc2058cfbf09597b0889ae7f7cf10f46ccbc2e2900ed063fdacb5dbe7",
"gitnexus/tools/mutation/scope-file.ts": "2582a4071c46d7f412013cd7036bd5fbbd46d341e0f268a8fe4a93ceeaa69195",
"gitnexus/tools/mutation/vitest.mutation.config.ts": "9e234252881e0cd5347a3f413320135c59226ac8923465a57f58290079906585",
"gitnexus/vitest.config.ts": "bea4e1ccad48d4742325095b53b5819a480671f01a753d5cbcc65deb2fc0e609"
},
"prepareTool": {
"gitnexus/tools/mutation/prepare/blank-assertions.ts": "895eb697d123cbd915543be411945055655e39963e6abb88e221ca3b570939f1",
"gitnexus/tools/mutation/prepare/mutant-identity.ts": "8887c9290cca144b6e547f5f0b74bc593a9e56ba989598c09cd573f787db2943",
"gitnexus/tools/mutation/prepare/path-map.ts": "d595cbebc2058cfbf09597b0889ae7f7cf10f46ccbc2e2900ed063fdacb5dbe7",
"gitnexus/tools/mutation/prepare/prepare.ts": "a8a703718cdb99ac9af6347009a068de26b199b976b9806061f031a2982ba0bf",
"gitnexus/tools/mutation/prepare/span-map-check.ts": "30b768d4386631ebeb9547ca624162a034dc8fdb0bb7cbe4bbca8a86b52fa4b4"
}
},
"sample": {
"seed": "u7-gate3-2",
"method": "sha256(seed + \"\\n\" + globalId) ascending; the size lowest ranks, then strata top-ups",
"size": 100,
"topUpMin": 3,
"random": 72,
"topUps": []
},
"strata": [
{
"name": "static",
"construct": null,
"constructPresent": null,
"population": 72,
"sampled": 72,
"toppedUp": 0,
"affectedTests": 0,
"required": false,
"status": "combined"
},
{
"name": "hook-hit",
"construct": "beforeAll",
"constructPresent": false,
"population": 0,
"sampled": 0,
"toppedUp": 0,
"affectedTests": 0,
"required": false,
"status": "construct-absent"
},
{
"name": "multi-killer",
"construct": null,
"constructPresent": null,
"population": 5,
"sampled": 5,
"toppedUp": 0,
"affectedTests": 0,
"required": false,
"status": "combined"
},
{
"name": "survived",
"construct": null,
"constructPresent": null,
"population": 22,
"sampled": 22,
"toppedUp": 0,
"affectedTests": 0,
"required": false,
"status": "combined"
},
{
"name": "no-coverage",
"construct": null,
"constructPresent": null,
"population": 0,
"sampled": 0,
"toppedUp": 0,
"affectedTests": 0,
"required": false,
"status": "empty-population"
},
{
"name": "each-row",
"construct": "each",
"constructPresent": true,
"population": 0,
"sampled": 0,
"toppedUp": 0,
"affectedTests": 9,
"required": false,
"status": "empty-population"
}
],
"baseline": {
"tests": 42,
"passedAlone": 42
},
"tests": [
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > includes all core node types",
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > includes multi-language node types",
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > includes the BasicBlock taint/PDG substrate node (issue #2080)",
"test/unit/schema.test.ts#LadybugDB Schema > NODE_TABLES > has expected total count",
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes all expected relationship types",
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes the taint/PDG substrate edge types (issue #2080)",
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes the control-dependence edge types (issue #2085 M5)",
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes the DI collection-injection edge type (#2200)",
"test/unit/schema.test.ts#LadybugDB Schema > REL_TYPES > includes Spring condition and auto-configuration edge types (#2415)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > FILE_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE File (\n id STRING,\n name STRING,\n filePath STRING,\n content STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > FOLDER_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Folder (\n id STRING,\n name STRING,\n filePath STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > FUNCTION_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Function (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n convexEndpointFactory STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > CLASS_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Class (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n frameworkAnnotations STRING[],\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > INTERFACE_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Interface (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > METHOD_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Method (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n parameterCount INT32,\n returnType STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > CODE_ELEMENT_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE CodeElement (\n id STRING,\n name STRING,\n filePath STRING,\n startLine INT64,\n endLine INT64,\n isExported BOOLEAN,\n content STRING,\n description STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > COMMUNITY_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Community (\n id STRING,\n label STRING,\n heuristicLabel STRING,\n keywords STRING[],\n description STRING,\n enrichedBy STRING,\n cohesion DOUBLE,\n symbolCount INT32,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > PROCESS_SCHEMA contains CREATE NODE TABLE for \nCREATE NODE TABLE Process (\n id STRING,\n label STRING,\n heuristicLabel STRING,\n processType STRING,\n stepCount INT32,\n communities STRING[],\n entryPointId STRING,\n terminalId STRING,\n PRIMARY KEY (id)\n)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Function schema has startLine and endLine",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Function schema has isExported",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Property schema preserves declaredType",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > BasicBlock schema is wired into SCHEMA_QUERIES (issue #2080, F1 guard)",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Community schema has heuristicLabel and cohesion",
"test/unit/schema.test.ts#LadybugDB Schema > node schema DDL > Process schema has processType and stepCount",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > creates a single REL TABLE named CodeRelation",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > has type, confidence, reason, step properties",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects Function to Function (CALLS)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects File to Function (CONTAINS/DEFINES)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects symbols to Community (MEMBER_OF)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects symbols to Process (STEP_IN_PROCESS)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > connects BasicBlock to BasicBlock (taint/PDG substrate edges, #2080)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > persists class-like framework edges to synthetic declarations (#2416)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > declares the Swift enum/property member-containment pairs (#2769)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > has all FROM/TO pairs needed for HAS_METHOD edges",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > has FROM/TO pairs for HAS_METHOD edges whose member is minted as Function, not Method (#2769)",
"test/unit/schema.test.ts#LadybugDB Schema > relation schema > has FROM/TO pairs for HAS_METHOD edges owned by a JS/TS object-literal binding (#2769)",
"test/unit/schema.test.ts#LadybugDB Schema > embedding schema > creates CodeEmbedding table",
"test/unit/schema.test.ts#LadybugDB Schema > embedding schema > has vector index query",
"test/unit/schema.test.ts#LadybugDB Schema > schema query ordering > NODE_SCHEMA_QUERIES has correct count",
"test/unit/schema.test.ts#LadybugDB Schema > schema query ordering > REL_SCHEMA_QUERIES has one relation table",
"test/unit/schema.test.ts#LadybugDB Schema > schema query ordering > SCHEMA_QUERIES includes all node + rel + embedding schemas",
"test/unit/schema.test.ts#LadybugDB Schema > schema query ordering > node schemas come before relation schemas in SCHEMA_QUERIES"
],
"notOracleValidated": [],
"mutants": [
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:100:37-100:52:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:11:28-49:2:ArrayDeclaration:4f53cda18c2baa0c","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:12:3-12:9:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:13:3-13:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:14:3-14:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:15:3-15:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:16:3-16:13:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:17:3-17:13:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:18:3-18:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:19:3-19:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:20:3-20:16:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:21:3-21:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:22:3-22:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:23:3-23:12:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:24:3-24:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:25:3-25:9:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:26:3-26:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:27:3-27:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:28:3-28:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:29:3-29:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:30:3-30:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:31:3-31:9:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:32:3-32:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:33:3-33:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:34:3-34:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:35:3-35:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:36:3-36:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:37:3-37:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:38:3-38:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:39:3-39:15:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:40:3-40:16:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:41:3-41:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:42:3-42:11:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:43:3-43:10:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:44:3-44:9:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:46:3-46:16:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:48:3-48:15:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:53:31-53:45:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:55:26-96:2:ArrayDeclaration:4f53cda18c2baa0c","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:56:3-56:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:57:3-57:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:58:3-58:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:59:3-59:10:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:60:3-60:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:61:3-61:15:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:62:3-62:15:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:63:3-63:17:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:64:3-64:13:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:65:3-65:21:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:66:3-66:14:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:67:3-67:22:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:68:3-68:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:69:3-69:20:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:70:3-70:18:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:71:3-71:17:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:72:3-72:18:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:73:3-73:12:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:74:3-74:17:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:75:3-75:19:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:76:3-76:10:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:77:3-77:12:StringLiteral:12ae32cb1ec02d01","survived"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:78:3-78:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:79:3-79:19:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:80:3-80:13:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:81:3-81:15:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:85:3-85:8:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:86:3-86:17:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:87:3-87:12:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:88:3-88:14:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:89:3-89:15:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:94:3-94:8:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"],
["gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:95:3-95:18:StringLiteral:12ae32cb1ec02d01","killed-by-test:assertion"]
],
"sampled": [
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:80:3-80:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[8,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:13:3-13:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:18:3-18:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:70:3-70:18:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:64:3-64:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:25:3-25:9:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:30:3-30:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:65:3-65:21:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:20:3-20:16:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:26:3-26:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:86:3-86:17:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[5,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:23:3-23:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:22:3-22:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:62:3-62:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:17:3-17:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:57:3-57:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:94:3-94:8:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[6,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:63:3-63:17:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:69:3-69:20:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:56:3-56:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:72:3-72:18:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:60:3-60:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:48:3-48:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[2,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:85:3-85:8:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[5,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:24:3-24:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:16:3-16:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:28:3-28:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:35:3-35:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:39:3-39:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:68:3-68:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:81:3-81:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[8,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:19:3-19:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","multi-killer"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"],[29,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:11:28-49:2:ArrayDeclaration:4f53cda18c2baa0c",
"strata": ["static","multi-killer"],
"covering": 42,
"outcomes": [
[0,"killed-by-test:assertion"],
[1,"killed-by-test:assertion"],
[2,"killed-by-test:assertion"],
[3,"killed-by-test:assertion"],
[28,"killed-by-test:assertion"],
[29,"killed-by-test:assertion"]
],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:33:3-33:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:38:3-38:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:89:3-89:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[5,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:14:3-14:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","multi-killer"],
"covering": 42,
"outcomes": [
[0,"killed-by-test:assertion"],
[28,"killed-by-test:assertion"],
[29,"killed-by-test:assertion"]
],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:21:3-21:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:40:3-40:16:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:71:3-71:17:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:100:37-100:52:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:37:3-37:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:66:3-66:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:27:3-27:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:53:31-53:45:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:75:3-75:19:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:76:3-76:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:77:3-77:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:44:3-44:9:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:34:3-34:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:46:3-46:16:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:41:3-41:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:59:3-59:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:73:3-73:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:79:3-79:19:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[8,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:67:3-67:22:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:42:3-42:11:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:29:3-29:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:55:26-96:2:ArrayDeclaration:4f53cda18c2baa0c",
"strata": ["static","multi-killer"],
"covering": 42,
"outcomes": [
[4,"killed-by-test:assertion"],
[5,"killed-by-test:assertion"],
[6,"killed-by-test:assertion"],
[7,"killed-by-test:assertion"],
[8,"killed-by-test:assertion"]
],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:88:3-88:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[5,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:61:3-61:15:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:87:3-87:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[5,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:36:3-36:13:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:31:3-31:9:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:74:3-74:17:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:15:3-15:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","multi-killer"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"],[28,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:78:3-78:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[7,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:43:3-43:10:StringLiteral:12ae32cb1ec02d01",
"strata": ["static","survived"],
"covering": 42,
"outcomes": [],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:12:3-12:9:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[0,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:32:3-32:14:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[1,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:58:3-58:12:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[4,"killed-by-test:assertion"]],
"invalid": []
},
{
"id": "gitnexus-shared/src/lbug/schema-constants.ts@edf2130f7f8f5f90:95:3-95:18:StringLiteral:12ae32cb1ec02d01",
"strata": ["static"],
"covering": 42,
"outcomes": [[6,"killed-by-test:assertion"]],
"invalid": []
}
],
"launches": {
"total": 3066,
"valid": 3024,
"invalid": 0,
"wallClockLimitMs": 180000
},
"warnings": [],
"problems": []
}

2523
gitnexus/test/fixtures/mutation-oracle/shared.json vendored Normal file
View file

File diff suppressed because it is too large Load diff

15
gitnexus/test/fixtures/mutation-scenario-map/date-boundary.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `date-boundary` (design §8): the
// end of February in a leap year and in a common year. The leap-year rule lives inside Date. Only
// the value-flow check separates the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { daysInMonth } from './src/date-boundary.js';
describe('date-boundary', () => {
it('counts the days of February in a leap year', () => {
expect(daysInMonth(2024, 1)).toBe(29);
});
it('counts the days of February in a common year', () => {
expect(daysInMonth(2023, 1)).toBe(28);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/derived-to-fixed.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `derived-to-fixed` (design §5.3a,
// §8): 2.4 and 2.5 are scaled to 24 and 25 by repo arithmetic before they reach toFixed. Only the
// value-flow check separates the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { formatRating } from './src/derived-to-fixed.js';
describe('derived-to-fixed', () => {
it('formats a 2.4-star rating', () => {
expect(formatRating(2.4)).toBe('24.0');
});
it('formats a 2.5-star rating', () => {
expect(formatRating(2.5)).toBe('25.0');
});
});

20
gitnexus/test/fixtures/mutation-scenario-map/enum-member.case.ts vendored Normal file
View file

@ -0,0 +1,20 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `enum-member` (KTD6 residual-
// difference guard): the bodies differ only in a const-object member passed outside the expect
// subject. No literal differs and no mutant tells the members apart. Expected: Review
// (expected.json, which records the plan conflict for this pair).
import { describe, expect, it } from 'vitest';
import { MiniGraph, NodeLabel } from './src/enum-member.js';
describe('enum-member', () => {
it('finds a callable after adding a function node', () => {
const graph = new MiniGraph();
graph.addNode(NodeLabel.Function);
expect(graph.hasCallableNode()).toBe(true);
});
it('finds a callable after adding a method node', () => {
const graph = new MiniGraph();
graph.addNode(NodeLabel.Method);
expect(graph.hasCallableNode()).toBe(true);
});
});

17
gitnexus/test/fixtures/mutation-scenario-map/expected-only.case.ts vendored Normal file
View file

@ -0,0 +1,17 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `expected-only` (KTD11 test side):
// the same input from the describe's shared setup, asserted against two different expected
// values. Only the value-flow check separates the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { formatTags } from './src/expected-only.js';
describe('expected-only', () => {
const tags = ['alpha', 'beta'];
it('renders the first tag', () => {
expect(formatTags(tags)).toContain('alpha');
});
it('renders the second tag', () => {
expect(formatTags(tags)).toContain('beta');
});
});

453
gitnexus/test/fixtures/mutation-scenario-map/expected.json vendored Normal file
View file

@ -0,0 +1,453 @@
{
"$comment": [
"Expected verdicts of the scenario-map controls fixture (Phase 1 plan U17, docs/plans/2026-10-24-0902-feat-test-scenario-map-phase1-plan.md). They were written from the design (docs/plans/2026-10-23-test-scenario-map-design.md §2, §5.3, §5.3a, §5.4, §8) and the plan's KTD5-KTD12 before any pipeline run. Never regenerate them from observed output; a mismatch is a finding, and an adversarial pair that ends as a repeat candidate is a plan stop condition.",
"Test ids: `<case file relative to this directory>#<Vitest full name>`, the Phase 0 id scheme of tools/mutation/runner/test-ids.ts with the file relative to the fixture root, as in mutation-conformance/expected.json. Prefix `test/fixtures/mutation-scenario-map/` for the package-relative Phase 0 id. Every full name is unique, so no id needs a location suffix.",
"Layout: one pair per case file, and each case file imports one production module under src/ (numeric-boundary imports src/page-size.ts, which imports the constants-only src/limits.ts). No two pairs share V8 blocks, checks or a mutate scope, so no cross-pair candidate is expected. `mutate` is the pair's KTD7 mutate scope: whole production files, relative to this directory.",
"`verdict` uses the ids of `verdicts` (design §2 labels). `rule` names the deciding design rule. `stages` gives the design's expected result for each stage on the pair's decision path (Verdict decision diagram of the plan); a stage not listed is off the path and does not affect the verdict. Stage values: facet equal | differs | equal-unresolved; discardedValues equal | differ; killSets equal (always non-empty here) | differ; rowDeletion not-applicable | symmetric | asymmetric; trace clean | veto; staticValueFlow clean | veto | unknown | not-required (discarded values equal); isolation agrees | disagrees.",
"Every test passes unmutated in file order. No production mutant throws, loops or times out in any test, so every kill is an assertion failure and no pair is Inconclusive by accident; only order-dependent is built to be Inconclusive.",
"Plan conflicts, recorded on the pairs in `planConflict`: enum-member (the residual-difference guard alone yields repeat-weak; expected review), set-membership (the design yields row, not the strong repeat U17 lists for positive controls), repo-arithmetic (design §5.3 rule 3 keeps the literal 1 exact). set-non-member is a positive control added beyond the U17 list: it is the only pair that checks a table read through Set.prototype.has is not vetoed as opaque.",
"`fingerprint` is the scenario fingerprint verdict and rule (tools/scenario-map/verdict/fingerprint.ts; docs/plans/2026-10-25-1700-feat-scenario-fingerprint-plan.md), written from its rules before any run, like the rest: every pair runs one path in both tests (no expected mutation rule is path-differs), so a pair whose discarded values differ is a table row whatever the values feed (the opaque built-ins, the expected-only literal 'alpha' against 'beta', the table keys and Set members); order-dependent is Inconclusive (not-passing): Layer 1 runs every test twice in a row (repeats 1, design §5.2), the shared counter makes each test's second attempt fail, and Vitest reports a test failed when any repetition fails; numeric-boundary's facets differ (distinct); enum-member's residual difference holds it at possible-duplicate; verbatim-duplicate is the one duplicate. A fingerprint duplicate of a pair this file does not list fails the oracle."
],
"schemaVersion": 1,
"verdicts": {
"inconclusive": "Inconclusive",
"out-of-scope": "Out of scope",
"weak": "Weak",
"repeat-strong": "Repeat candidate (strong evidence)",
"repeat-weak": "Repeat candidate (weak evidence)",
"row": "Row",
"review": "Review",
"distinct": "Distinct"
},
"pairs": [
{
"id": "math-round",
"kind": "adversarial",
"mechanism": "opaque built-in: Math.round on 2.4 and 2.5",
"tests": [
"math-round.case.ts#math-round > rounds 2.4 down to 2",
"math-round.case.ts#math-round > rounds 2.5 up to 3"
],
"mutate": [
"src/math-round.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "Both tests take the same path through roundScore (the `raw < 0` guard is false) with the same check and context. 2.4/2.5 and 2/3 are in no table and not within ±1 of the module's only constant 0, so the facets are equal and the values are discarded. Stryker has no mutator inside Math.round: both tests kill exactly the guard forced true, `raw >= 0` and the removed body. No table, so row deletion does not apply. The trace sees Math.round called from src/math-round.ts with 2.4 against 2.5 (veto); the static check finds `raw` flowing into Math.round, declared in lib.es5.d.ts (opaque, veto). Design §2: same path, checks, context and kills with a value-flow veto is Review."
},
{
"id": "parse-int-radix",
"kind": "adversarial",
"mechanism": "opaque built-in: parseInt with radix 8 and radix 16",
"tests": [
"parse-int-radix.case.ts#parse-int-radix > reads 17 as an octal code",
"parse-int-radix.case.ts#parse-int-radix > reads 17 as a hexadecimal code"
],
"mutate": [
"src/parse-int-radix.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "The text '17' is the same; the radix 8/16 and the results 15/23 are discarded (not within ±1 of the module's constant -1). Both tests skip the NaN branch and kill the same mutants: the NaN guard forced true and the removed body. The trace sees parseInt('17', 8) against parseInt('17', 16) at one call site, and Number.isNaN(15) against Number.isNaN(23) (veto); the static check finds `radix` flowing into parseInt, declared in lib.es5.d.ts (veto). Review."
},
{
"id": "locale-compare",
"kind": "adversarial",
"mechanism": "opaque built-in: String.prototype.localeCompare, difference in argument position",
"tests": [
"locale-compare.case.ts#locale-compare > orders apple before Banana",
"locale-compare.case.ts#locale-compare > orders apple before Damson"
],
"mutate": [
"src/locale-compare.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "The receiver 'apple' is the same; only the argument differs, and 'Banana'/'Damson' have the same shape, so the facets are equal. Both comparisons are negative, so both tests take the same branch and kill the same mutants: first guard forced false, `order >= 0`, first guard body removed, `-1` to `+1`, body removed (the second guard is not covered). The trace sees localeCompare from src/locale-compare.ts with the same receiver but different arguments (veto); the static check finds `right` flowing into String.prototype.localeCompare, declared in lib (veto). Review."
},
{
"id": "unicode-normalize",
"kind": "adversarial",
"mechanism": "opaque built-in: String.prototype.normalize, difference in receiver position",
"tests": [
"unicode-normalize.case.ts#unicode-normalize > composes e and a combining acute into one code unit",
"unicode-normalize.case.ts#unicode-normalize > keeps a combining acute that has no composed form"
],
"mutate": [
"src/unicode-normalize.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "'Jose' + U+0301 composes to 4 code units; 'Josq' + U+0301 has no composed form and stays 5. Both inputs have the same shape (a capitalised word, then U+0301) and 4/5 are discarded, so the facets are equal. Both kill the same mutants: the empty-name guard forced true, `name !== ''` and the removed body. The differing value is the receiver (`this`) of normalize, which the trace records (KTD11): different receivers at one call site (veto). The static check finds `name` as the receiver of String.prototype.normalize, declared in lib (veto). Review."
},
{
"id": "date-boundary",
"kind": "adversarial",
"mechanism": "opaque built-in: Date month-end rollover in a leap and a common year",
"tests": [
"date-boundary.case.ts#date-boundary > counts the days of February in a leap year",
"date-boundary.case.ts#date-boundary > counts the days of February in a common year"
],
"mutate": [
"src/date-boundary.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "The month literal 1 is exact and equal in both tests; the years 2024/2023 and the results 29/28 are discarded (not within ±1 of the module's constants 0 and 1). The leap-year rule lives inside Date, so both tests kill the same mutants: `month + 1` to `month - 1` (31 days) and the removed body. The trace sees Date.UTC(2024, 2, 0) against Date.UTC(2023, 2, 0), and different Date and getUTCDate receivers (veto); the static check finds `year` flowing into Date.UTC, declared in lib (veto). Review."
},
{
"id": "regex-alternatives",
"kind": "adversarial",
"mechanism": "regex: two inputs matching different alternatives of one regex",
"tests": [
"regex-alternatives.case.ts#regex-alternatives > recognises a getter name",
"regex-alternatives.case.ts#regex-alternatives > recognises a setter name"
],
"mutate": [
"src/regex-alternatives.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "'getName' and 'setName' match different alternatives of /^(?:get|set)[A-Z]/ and have the same shape; whether the facet maps them to the alternation table or to shapes, they stay equal. Weapon-regex level 1 has no operator that drops one alternative, so both kill the same mutants: the character-class negation and the removed body (removing `^` survives both). A regex alternation is not a row-deletion table (design §5.4, KTD10), so row deletion does not apply. The trace sees RegExp.prototype.test with different arguments (veto); the static check treats any regex use as opaque (veto). Review."
},
{
"id": "numeric-boundary",
"kind": "adversarial",
"mechanism": "numeric boundary: PAGE_LIMIT + 1 against an interior value, the constant in a constants-only module",
"tests": [
"numeric-boundary.case.ts#numeric-boundary > cuts a page size one above the limit",
"numeric-boundary.case.ts#numeric-boundary > cuts a page size far above the limit"
],
"mutate": [
"src/page-size.ts",
"src/limits.ts"
],
"verdict": "review",
"rule": "facet-differs-same-kills",
"fingerprint": {
"verdict": "distinct",
"rule": "input-shape-differs"
},
"stages": {
"facet": "differs",
"killSets": "equal"
},
"reason": "PAGE_LIMIT = 100 lives in src/limits.ts, a constants-only module that runs only while it loads, so the facet must read it through the static import closure (KTD6). 101 is within ±1 of 100 and stays exact; 150 collapses to a shape; the expected 100 is exact in both. The facets differ. Both inputs take the clamp branch and kill the same mutants: guard forced false, `requested <= PAGE_LIMIT`, clamp branch emptied, body removed (`>=` survives both; Stryker has no numeric-literal mutator). Design §2: same path, checks, context and kill set with a different facet is Review. Without the ±1 rule the facets would be equal, the trace and the static check would be clean (a comparison and a pass-through only) and the pair would wrongly become a strong repeat."
},
{
"id": "expected-only",
"kind": "adversarial",
"mechanism": "expected-only difference: one shared input, toContain('alpha') against toContain('beta')",
"tests": [
"expected-only.case.ts#expected-only > renders the first tag",
"expected-only.case.ts#expected-only > renders the second tag"
],
"mutate": [
"src/expected-only.ts"
],
"verdict": "review",
"rule": "value-flow-unknown",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "unknown"
},
"reason": "Both tests render the describe's shared `tags`, so path, calls and kills are identical: the `#` template emptied, the map callback removed and the removed body (each leaves no 'alpha' and no 'beta'; the separator mutant survives both). Only the expected values 'alpha'/'beta' differ; they are in no production table and have the same shape, so the facets are equal and the values are discarded. The trace sees identical calls (clean). KTD11 test side: a differing expected value with no differing bound input is unknown, so the static check vetoes. Review."
},
{
"id": "derived-to-fixed",
"kind": "adversarial",
"mechanism": "derived value reaching an opaque built-in: (x * 10).toFixed(1) with x 2.4 and 2.5",
"tests": [
"derived-to-fixed.case.ts#derived-to-fixed > formats a 2.4-star rating",
"derived-to-fixed.case.ts#derived-to-fixed > formats a 2.5-star rating"
],
"mutate": [
"src/derived-to-fixed.ts"
],
"verdict": "review",
"rule": "value-flow-veto",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "veto",
"staticValueFlow": "veto"
},
"reason": "2.4/2.5 and '24.0'/'25.0' are discarded (not within ±1 of the constants 10 and 1). Both kill the same mutants: `*` to `/` ('0.2'/'0.3') and the removed body. `stars * 10` is repo arithmetic, but its result is the receiver of Number.prototype.toFixed: the trace sees receivers 24 and 25 at one call site (veto; this is the derived-value case of design §5.3a step 4), and the static check finds the product of `stars` reaching toFixed, declared in lib (veto). Review."
},
{
"id": "enum-member",
"kind": "adversarial",
"mechanism": "enum-member-only difference outside the expect subject (KTD6 residual-difference guard)",
"tests": [
"enum-member.case.ts#enum-member > finds a callable after adding a function node",
"enum-member.case.ts#enum-member > finds a callable after adding a method node"
],
"mutate": [
"src/enum-member.ts"
],
"verdict": "review",
"rule": "residual-difference",
"fingerprint": {
"verdict": "possible-duplicate",
"rule": "input-unresolved"
},
"stages": {
"facet": "equal-unresolved",
"discardedValues": "equal",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "not-required"
},
"reason": "The bodies differ only in NodeLabel.Function against NodeLabel.Method, a member expression passed to addNode outside the expect subject. No literal differs, so the facets are equal and nothing is discarded; the residual-difference guard (KTD6, R5) marks both instances unresolved. Mutants of NodeLabel (object emptied, member strings emptied) change the argument and the CALLABLE_LABELS Set together and survive both; both kill the same mutants: the Set's entry array emptied (static), addNode and hasCallableNode bodies removed, the `some` callback removed (`every` survives a one-element list). The differing nodes are not literals, so row deletion does not apply (design §5.4). Set.has, push and some are not wrapped, so the trace is clean. Mutation cannot tell the tests apart and the facet did not capture their difference, so the inputs are not shown equal: Review (fail closed). Removing NodeLabel.Method from the Set would fail only the second test.",
"planConflict": "As written, U6 (residual difference marks both instances unresolved) plus U12 (an unresolved facet input is only a weak disqualifier) would give repeat-weak for this pair, a repeat candidate, which U17 and the Goal Capsule stop condition forbid for an adversarial pair. Resolved by decision D1: the verdict engine treats a residual-difference pair as not facet-equal (rule residual-difference, design §2 Review: same path, checks, context and kill set with a different facet), so the pair ends Review."
},
{
"id": "order-dependent",
"kind": "special",
"mechanism": "order dependence: the second test relies on module state the first leaks",
"tests": [
"order-dependent.case.ts#order-dependent > labels the first ticket",
"order-dependent.case.ts#order-dependent > labels the second ticket"
],
"mutate": [
"src/order-dependent.ts"
],
"verdict": "inconclusive",
"rule": "isolation-disagrees",
"fingerprint": {
"verdict": "inconclusive",
"rule": "not-passing"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "clean",
"isolation": "disagrees"
},
"reason": "nextTicketLabel reads a module-level counter. In file order the first test leaves it at 1 and the second gets 'beta-2'; alone, the second gets 'beta-1' and fails. Run together (main run, baselines, dry run, trace run), the pair looks like a repeat: same path, checks and context; 'alpha'/'beta' and 'alpha-1'/'beta-2' have the same shapes; both kill `+=` to `-=`, the template emptied and the removed body; no table; no wrapped built-in; the value flows only into a template literal and the return. So it reaches the repeat stage, and the full isolation pass (design §5.4, KTD9) runs the second test alone: its baseline and Stryker dry run fail, and its outcomes disagree with the together run. Design §2: an isolation check that disagrees is Inconclusive, which takes precedence over every other verdict."
},
{
"id": "object-key-row",
"kind": "special",
"mechanism": "row deletion: two keys of one three-key object-literal table",
"tests": [
"object-key-row.case.ts#object-key-row > converts seconds to milliseconds",
"object-key-row.case.ts#object-key-row > converts minutes to milliseconds"
],
"mutate": [
"src/object-key-row.ts"
],
"verdict": "row",
"rule": "row-deletion-asymmetric",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"killSets": "equal",
"rowDeletion": "asymmetric"
},
"reason": "UNIT_FACTORS = { ms, s, min } is an object literal with plain identifier keys read through `UNIT_FACTORS[unit]`, a supported table (design §5.4). 's' and 'min' are keys of it, so both map to the table and the facets are equal; the amount 2 is exact (within ±1 of the constant 1) and equal; 2000/120000 have the same shape. Stryker never mutates object keys, and both tests kill the same mutants: the table emptied (static), the unknown-unit guard forced true, `factor !== undefined`, `*` to `/`, the removed body. Row deletion: deleting `s` fails only the seconds test and deleting `min` only the minutes test, so it is asymmetric. Design §2: same path, checks, context and facet with asymmetric row deletion is Row."
},
{
"id": "verbatim-duplicate",
"kind": "positive-control",
"mechanism": "verbatim duplicate: byte-identical bodies, different titles",
"tests": [
"verbatim-duplicate.case.ts#verbatim-duplicate > clamps a percentage above 100",
"verbatim-duplicate.case.ts#verbatim-duplicate > caps an oversized percentage"
],
"mutate": [
"src/verbatim-duplicate.ts"
],
"verdict": "repeat-strong",
"rule": "all-strong-conditions",
"fingerprint": {
"verdict": "duplicate",
"rule": "same-path-input-expectation"
},
"stages": {
"facet": "equal",
"discardedValues": "equal",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "not-required",
"isolation": "agrees"
},
"reason": "The two bodies are byte-identical and share the describe, so path, V8 blocks, checks, context, facet and kills are identical by construction (both kill the first guard forced true, `value >= 0`, the second guard forced false, `value <= 100`, the clamp branch emptied and the removed body). The discarded values are equal, so only the trace is needed, and clampPercent calls no built-in. The module is a pure production file of the fixture, so provenance is strong-eligible, and the pure function lets isolation agree. Every design §2 strong condition holds."
},
{
"id": "repo-arithmetic",
"kind": "positive-control",
"mechanism": "repo-only arithmetic: add(2, 3) against add(5, 5) (design §8; operands avoid 0, 1 and -1, which design §5.3 rule 3 keeps exact)",
"tests": [
"repo-arithmetic.case.ts#repo-arithmetic > adds 2 and 3",
"repo-arithmetic.case.ts#repo-arithmetic > adds 5 and 5"
],
"mutate": [
"src/repo-arithmetic.ts"
],
"verdict": "repeat-strong",
"rule": "all-strong-conditions",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "clean",
"isolation": "agrees"
},
"reason": "add uses only repo arithmetic. Both tests kill exactly `+` to `-` and the removed body. The inputs and sums are discarded: none is 0, 1 or -1 (design §5.3 rule 3 keeps those exact) and the module has no numeric constant, so the ±1 rule keeps none of them exact. The static check finds `left` and `right` flowing only into `+` (repo arithmetic) and the return (pass-through): clean. No wrapped built-in is called: the trace is clean. Plan U12 lists this pair as its strong-repeat scenario."
},
{
"id": "set-membership",
"kind": "positive-control",
"mechanism": "membership in a supported Set table: one test per member, identical kill sets (design §8)",
"tests": [
"set-membership.case.ts#set-membership > accepts 200 as a success status",
"set-membership.case.ts#set-membership > accepts 204 as a success status"
],
"mutate": [
"src/set-membership.ts"
],
"verdict": "row",
"rule": "row-deletion-asymmetric",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"killSets": "equal",
"rowDeletion": "asymmetric"
},
"reason": "SUCCESS_STATUSES = new Set([200, 204]) is built from literal entries and read through `has`, a supported table (design §5.4). The entries are numbers, which Stryker never mutates one by one, so the kill sets are identical: the entry array emptied (static) and the removed body. 200 and 204 are entries of the table, so both map to it and the facets are equal (Set entries are not named constants, so the ±1 rule does not apply). No weak disqualifier applies. But row deletion applies to exactly this table: deleting 200 fails only the first test and deleting 204 only the second, which is asymmetric, and design §2 ranks Row before any Repeat. A Set table with one test per member is therefore a Row (fold into it.each, keep every row), the verdict the design implies.",
"planConflict": "U17 and the Success Criteria list Set membership among positive controls that end as strong repeats. With two members tested by two tests, the design can only reach Row: row deletion exists to separate them. set-non-member covers the eligibility property instead (a Set.has table read is not vetoed)."
},
{
"id": "set-non-member",
"kind": "positive-control",
"mechanism": "table read through Set.prototype.has is clean, not opaque (design §5.3a step 3)",
"tests": [
"set-non-member.case.ts#set-non-member > rejects banana as a reserved word",
"set-non-member.case.ts#set-non-member > rejects cherry as a reserved word"
],
"mutate": [
"src/set-non-member.ts"
],
"verdict": "repeat-strong",
"rule": "all-strong-conditions",
"fingerprint": {
"verdict": "table-row",
"rule": "input-values-differ"
},
"stages": {
"facet": "equal",
"discardedValues": "differ",
"killSets": "equal",
"rowDeletion": "not-applicable",
"trace": "clean",
"staticValueFlow": "clean",
"isolation": "agrees"
},
"reason": "Added beyond the U17 list: it is the pair that keeps the Set-membership eligibility property under test. 'banana' and 'cherry' are not entries of RESERVED_WORDS, so they collapse to the same shape and row deletion does not apply. Both tests kill only the removed body (an emptied entry array or an emptied entry string still answers false). Their differing values reach Set.prototype.has, declared in lib.es2015.collection.d.ts, which design §5.3a step 3 classifies as a supported table read (clean), and Set methods are not wrapped by the trace (clean). Every design §2 strong condition holds. A value-flow check that vetoes this pair over-vetoes table reads."
}
]
}

16
gitnexus/test/fixtures/mutation-scenario-map/locale-compare.case.ts vendored Normal file
View file

@ -0,0 +1,16 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `locale-compare` (design §8): the
// same receiver compared with two different arguments (argument position). Both orders hold in
// every Latin collation, and both fail under code-unit order. Only the value-flow check separates
// the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { compareLabels } from './src/locale-compare.js';
describe('locale-compare', () => {
it('orders apple before Banana', () => {
expect(compareLabels('apple', 'Banana')).toBe(-1);
});
it('orders apple before Damson', () => {
expect(compareLabels('apple', 'Damson')).toBe(-1);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/math-round.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `math-round` (design §5.3a, §8):
// 2.4 rounds down and 2.5 rounds up inside Math.round. Path, checks, context, facet and kills
// match; only the value-flow check separates the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { roundScore } from './src/math-round.js';
describe('math-round', () => {
it('rounds 2.4 down to 2', () => {
expect(roundScore(2.4)).toBe(2);
});
it('rounds 2.5 up to 3', () => {
expect(roundScore(2.5)).toBe(3);
});
});

16
gitnexus/test/fixtures/mutation-scenario-map/numeric-boundary.case.ts vendored Normal file
View file

@ -0,0 +1,16 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `numeric-boundary` (design §8,
// KTD6): 101 is PAGE_LIMIT + 1, 150 is an interior value. PAGE_LIMIT lives in a constants-only
// module two imports away. Only the facet's ±1 rule separates the two tests. Expected: Review
// (expected.json).
import { describe, expect, it } from 'vitest';
import { clampPageSize } from './src/page-size.js';
describe('numeric-boundary', () => {
it('cuts a page size one above the limit', () => {
expect(clampPageSize(101)).toBe(100);
});
it('cuts a page size far above the limit', () => {
expect(clampPageSize(150)).toBe(100);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/object-key-row.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), special pair `object-key-row` (design §5.4 row
// deletion, KTD10): two tests read two different keys of one three-key object-literal table.
// Deleting either key fails only the test that reads it. Expected: Row (expected.json).
import { describe, expect, it } from 'vitest';
import { toMillis } from './src/object-key-row.js';
describe('object-key-row', () => {
it('converts seconds to milliseconds', () => {
expect(toMillis(2, 's')).toBe(2000);
});
it('converts minutes to milliseconds', () => {
expect(toMillis(2, 'min')).toBe(120000);
});
});

16
gitnexus/test/fixtures/mutation-scenario-map/order-dependent.case.ts vendored Normal file
View file

@ -0,0 +1,16 @@
// Scenario-map controls fixture (Phase 1 U17), special pair `order-dependent` (design §5.4 full
// isolation pass, KTD9): the second test relies on the counter the first test leaves behind. In
// file order both pass; run alone, the second gets `beta-1` and fails. Expected: Inconclusive
// (expected.json).
import { describe, expect, it } from 'vitest';
import { nextTicketLabel } from './src/order-dependent.js';
describe('order-dependent', () => {
it('labels the first ticket', () => {
expect(nextTicketLabel('alpha')).toBe('alpha-1');
});
it('labels the second ticket', () => {
expect(nextTicketLabel('beta')).toBe('beta-2');
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/parse-int-radix.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `parse-int-radix` (design §8): the
// same text read in radix 8 and radix 16. Only the value-flow check separates the two tests.
// Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { readCode } from './src/parse-int-radix.js';
describe('parse-int-radix', () => {
it('reads 17 as an octal code', () => {
expect(readCode('17', 8)).toBe(15);
});
it('reads 17 as a hexadecimal code', () => {
expect(readCode('17', 16)).toBe(23);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/regex-alternatives.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `regex-alternatives` (design §8):
// two inputs that match different alternatives of one regex. Only the value-flow check separates
// the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { isAccessorName } from './src/regex-alternatives.js';
describe('regex-alternatives', () => {
it('recognises a getter name', () => {
expect(isAccessorName('getName')).toBe(true);
});
it('recognises a setter name', () => {
expect(isAccessorName('setName')).toBe(true);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/repo-arithmetic.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), positive control `repo-arithmetic` (design §8):
// add(2, 3) against add(5, 5) on repo-only arithmetic. Expected: Repeat, strong evidence
// (expected.json).
import { describe, expect, it } from 'vitest';
import { add } from './src/repo-arithmetic.js';
describe('repo-arithmetic', () => {
it('adds 2 and 3', () => {
expect(add(2, 3)).toBe(5);
});
it('adds 5 and 5', () => {
expect(add(5, 5)).toBe(10);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/set-membership.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), positive control `set-membership` (design §8): two
// members of one supported Set table, one test each, with identical kill sets. Expected: Row, the
// verdict the design implies (expected.json explains why it is not a repeat).
import { describe, expect, it } from 'vitest';
import { isSuccessStatus } from './src/set-membership.js';
describe('set-membership', () => {
it('accepts 200 as a success status', () => {
expect(isSuccessStatus(200)).toBe(true);
});
it('accepts 204 as a success status', () => {
expect(isSuccessStatus(204)).toBe(true);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/set-non-member.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), positive control `set-non-member`: two different
// values reach Set.prototype.has on a supported table, a table read the value-flow check must not
// veto. Expected: Repeat, strong evidence (expected.json).
import { describe, expect, it } from 'vitest';
import { isReservedWord } from './src/set-non-member.js';
describe('set-non-member', () => {
it('rejects banana as a reserved word', () => {
expect(isReservedWord('banana')).toBe(false);
});
it('rejects cherry as a reserved word', () => {
expect(isReservedWord('cherry')).toBe(false);
});
});

8
gitnexus/test/fixtures/mutation-scenario-map/src/date-boundary.ts vendored Normal file
View file

@ -0,0 +1,8 @@
// Scenario-map controls fixture (Phase 1 U17), pair `date-boundary`. Day 0 of the next month rolls
// back to the last day of this one inside Date, so the leap-year rule is Date's, not this module's.
/** Days in a month (0-based, as Date counts months) of a year, in UTC. */
export function daysInMonth(year: number, month: number): number {
const lastDay = new Date(Date.UTC(year, month + 1, 0));
return lastDay.getUTCDate();
}

7
gitnexus/test/fixtures/mutation-scenario-map/src/derived-to-fixed.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Scenario-map controls fixture (Phase 1 U17), pair `derived-to-fixed`. The input is scaled by repo
// arithmetic first, so the value that reaches Number.prototype.toFixed is derived, not the input.
/** Formats a star rating (0 to 5) on a 0 to 50 scale with one decimal. */
export function formatRating(stars: number): string {
return (stars * 10).toFixed(1);
}

28
gitnexus/test/fixtures/mutation-scenario-map/src/enum-member.ts vendored Normal file
View file

@ -0,0 +1,28 @@
// Scenario-map controls fixture (Phase 1 U17), pair `enum-member`. The label constants feed both
// the argument a test passes and the callable-label Set, so a mutant of a constant changes both
// sides at once and no mutant tells the two members apart.
/** Graph node labels, as a const object (an enum-like table of string members). */
export const NodeLabel: Readonly<{ Function: 'Function'; Method: 'Method'; Class: 'Class' }> = {
Function: 'Function',
Method: 'Method',
Class: 'Class',
};
export type NodeLabelName = (typeof NodeLabel)[keyof typeof NodeLabel];
const CALLABLE_LABELS: ReadonlySet<string> = new Set([NodeLabel.Function, NodeLabel.Method]);
/** The node labels a graph builder has collected. */
export class MiniGraph {
readonly #labels: NodeLabelName[] = [];
addNode(label: NodeLabelName): void {
this.#labels.push(label);
}
/** True when any collected node is a callable (a function or a method). */
hasCallableNode(): boolean {
return this.#labels.some((label) => CALLABLE_LABELS.has(label));
}
}

7
gitnexus/test/fixtures/mutation-scenario-map/src/expected-only.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Scenario-map controls fixture (Phase 1 U17), pair `expected-only`. One rendering of a shared tag
// list; the two tests assert different parts of the same output.
/** Renders tags as a comma-separated list of `#`-prefixed names. */
export function formatTags(tags: readonly string[]): string {
return tags.map((tag) => `#${tag}`).join(', ');
}

6
gitnexus/test/fixtures/mutation-scenario-map/src/limits.ts vendored Normal file
View file

@ -0,0 +1,6 @@
// Scenario-map controls fixture (Phase 1 U17), pair `numeric-boundary`. A constants-only module:
// it declares no function, so its code runs only while the module loads, a phase V8 per-test
// coverage discards. The facet must find PAGE_LIMIT through the static import closure (KTD6).
/** Largest page size a listing request may ask for. */
export const PAGE_LIMIT = 100;

15
gitnexus/test/fixtures/mutation-scenario-map/src/locale-compare.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), pair `locale-compare`. The ordering is decided
// inside String.prototype.localeCompare, which Stryker cannot mutate. No locale literal is passed:
// a mutated (empty) locale string would make the call throw instead of fail an assertion.
/** Orders two display labels with the runtime's collation: -1, 0 or 1. */
export function compareLabels(left: string, right: string): number {
const order = left.localeCompare(right);
if (order < 0) {
return -1;
}
if (order > 0) {
return 1;
}
return 0;
}

11
gitnexus/test/fixtures/mutation-scenario-map/src/math-round.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Scenario-map controls fixture (Phase 1 U17), pair `math-round`. The rounding happens inside
// Math.round, which Stryker cannot mutate, so inputs on either side of the half kill the same
// mutants of this module.
/** Rounds a raw score to whole points; a negative score counts as zero. */
export function roundScore(raw: number): number {
if (raw < 0) {
return 0;
}
return Math.round(raw);
}

14
gitnexus/test/fixtures/mutation-scenario-map/src/object-key-row.ts vendored Normal file
View file

@ -0,0 +1,14 @@
// Scenario-map controls fixture (Phase 1 U17), pair `object-key-row`. A supported row-deletion
// table (design §5.4): an object literal with plain keys, read through `table[variable]`. Stryker
// never mutates object keys.
const UNIT_FACTORS: Readonly<Record<string, number>> = { ms: 1, s: 1000, min: 60000 };
/** Converts an amount of a time unit to milliseconds; an unknown unit converts to 0. */
export function toMillis(amount: number, unit: string): number {
const factor = UNIT_FACTORS[unit];
if (factor === undefined) {
return 0;
}
return amount * factor;
}

10
gitnexus/test/fixtures/mutation-scenario-map/src/order-dependent.ts vendored Normal file
View file

@ -0,0 +1,10 @@
// Scenario-map controls fixture (Phase 1 U17), pair `order-dependent`. Module-level state: every
// label depends on how many tickets earlier code in the same module instance issued.
let issued = 0;
/** Labels the next ticket of a queue with a process-wide running number. */
export function nextTicketLabel(queue: string): string {
issued += 1;
return `${queue}-${issued}`;
}

11
gitnexus/test/fixtures/mutation-scenario-map/src/page-size.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Scenario-map controls fixture (Phase 1 U17), pair `numeric-boundary`. The limit comes from the
// constants-only module ./limits.ts; Stryker has no numeric-literal mutator.
import { PAGE_LIMIT } from './limits.js';
/** Clamps a requested page size to PAGE_LIMIT. */
export function clampPageSize(requested: number): number {
if (requested > PAGE_LIMIT) {
return PAGE_LIMIT;
}
return requested;
}

11
gitnexus/test/fixtures/mutation-scenario-map/src/parse-int-radix.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Scenario-map controls fixture (Phase 1 U17), pair `parse-int-radix`. The radix is applied inside
// parseInt, which Stryker cannot mutate.
/** Reads a code written in the given radix; text that is not a number reads as -1. */
export function readCode(text: string, radix: number): number {
const value = parseInt(text, radix);
if (Number.isNaN(value)) {
return -1;
}
return value;
}

7
gitnexus/test/fixtures/mutation-scenario-map/src/regex-alternatives.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Scenario-map controls fixture (Phase 1 U17), pair `regex-alternatives`. Stryker's regex mutator
// (weapon-regex, level 1) has no operator that drops one alternative of an alternation.
/** True when a method name reads as an accessor: `get` or `set` followed by a capital letter. */
export function isAccessorName(name: string): boolean {
return /^(?:get|set)[A-Z]/.test(name);
}

7
gitnexus/test/fixtures/mutation-scenario-map/src/repo-arithmetic.ts vendored Normal file
View file

@ -0,0 +1,7 @@
// Scenario-map controls fixture (Phase 1 U17), pair `repo-arithmetic` (design §8 positive control).
// Repo-only arithmetic: every operation is visible to coverage and mutation.
/** Adds two numbers. */
export function add(left: number, right: number): number {
return left + right;
}

11
gitnexus/test/fixtures/mutation-scenario-map/src/set-membership.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Scenario-map controls fixture (Phase 1 U17), pair `set-membership` (design §8 positive control).
// A supported row-deletion table (design §5.4): `new Set([...])` of literal entries, read through
// `has`. The entries are numbers, which Stryker never mutates one by one, so the only table mutant
// empties the whole Set.
const SUCCESS_STATUSES: ReadonlySet<number> = new Set([200, 204]);
/** True when an HTTP status code is one this client treats as success. */
export function isSuccessStatus(status: number): boolean {
return SUCCESS_STATUSES.has(status);
}

10
gitnexus/test/fixtures/mutation-scenario-map/src/set-non-member.ts vendored Normal file
View file

@ -0,0 +1,10 @@
// Scenario-map controls fixture (Phase 1 U17), pair `set-non-member`. A supported Set table read
// through `has`: a table read, not an opaque call, although Set is declared in lib.*.d.ts (design
// §5.3a step 3), and not wrapped by the trace.
const RESERVED_WORDS: ReadonlySet<string> = new Set(['class', 'enum', 'super']);
/** True when a word is reserved and cannot name a symbol. */
export function isReservedWord(word: string): boolean {
return RESERVED_WORDS.has(word);
}

11
gitnexus/test/fixtures/mutation-scenario-map/src/unicode-normalize.ts vendored Normal file
View file

@ -0,0 +1,11 @@
// Scenario-map controls fixture (Phase 1 U17), pair `unicode-normalize`. The composition happens
// inside String.prototype.normalize, called on the input itself (receiver position). The form is
// left at its NFC default: a mutated (empty) form string would make the call throw.
/** Length in UTF-16 code units of a name after canonical composition (NFC); empty names are 0. */
export function composedLength(name: string): number {
if (name === '') {
return 0;
}
return name.normalize().length;
}

13
gitnexus/test/fixtures/mutation-scenario-map/src/verbatim-duplicate.ts vendored Normal file
View file

@ -0,0 +1,13 @@
// Scenario-map controls fixture (Phase 1 U17), pair `verbatim-duplicate`. Repo-only logic, no
// built-in calls.
/** Clamps a percentage to the range 0 to 100. */
export function clampPercent(value: number): number {
if (value < 0) {
return 0;
}
if (value > 100) {
return 100;
}
return value;
}

5
gitnexus/test/fixtures/mutation-scenario-map/tsconfig.json vendored Normal file
View file

@ -0,0 +1,5 @@
{
"extends": "../../../tsconfig.test.json",
"include": ["src/**/*.ts", "*.case.ts"],
"exclude": []
}

16
gitnexus/test/fixtures/mutation-scenario-map/unicode-normalize.case.ts vendored Normal file
View file

@ -0,0 +1,16 @@
// Scenario-map controls fixture (Phase 1 U17), adversarial pair `unicode-normalize` (design §8):
// the differing value is the receiver of String.prototype.normalize. `e` + U+0301 composes to one
// code unit; `q` + U+0301 has no composed form and stays two. Only the value-flow check separates
// the two tests. Expected: Review (expected.json).
import { describe, expect, it } from 'vitest';
import { composedLength } from './src/unicode-normalize.js';
describe('unicode-normalize', () => {
it('composes e and a combining acute into one code unit', () => {
expect(composedLength('Jose\u0301')).toBe(4);
});
it('keeps a combining acute that has no composed form', () => {
expect(composedLength('Josq\u0301')).toBe(5);
});
});

15
gitnexus/test/fixtures/mutation-scenario-map/verbatim-duplicate.case.ts vendored Normal file
View file

@ -0,0 +1,15 @@
// Scenario-map controls fixture (Phase 1 U17), positive control `verbatim-duplicate`: two
// byte-identical test bodies with different titles. Expected: Repeat, strong evidence
// (expected.json).
import { describe, expect, it } from 'vitest';
import { clampPercent } from './src/verbatim-duplicate.js';
describe('verbatim-duplicate', () => {
it('clamps a percentage above 100', () => {
expect(clampPercent(150)).toBe(100);
});
it('caps an oversized percentage', () => {
expect(clampPercent(150)).toBe(100);
});
});

52
gitnexus/test/fixtures/mutation-scenario-map/vitest.config.ts vendored Normal file
View file

@ -0,0 +1,52 @@
/**
* Vitest config of the scenario-map controls fixture (Phase 1 plan U17). The `controls` stage of
* tools/scenario-map (U14) runs every layer on this project, with the fixture's `src/` modules as
* production files, and compares the verdicts with expected.json. Like
* test/fixtures/mutation-conformance/vitest.config.ts, only the settings the vitest5 runner
* requires (forks, isolation, task locations) are set, so a change to gitnexus/vitest.config.ts
* cannot move the controls. Test order is pinned: the `order-dependent` pair passes only in file
* order.
*
* The fixture's tests are `*.case.ts` files, which no repo Vitest project collects. When
* GITNEXUS_MUTATION_SCOPE names a scope file (tools/mutation/scope-file.ts), its `testFiles` are
* the include list, as in tools/mutation/vitest.mutation.config.ts; without it every case file of
* the fixture runs. Include paths are relative to the `gitnexus/` package, the working directory
* of every caller.
*
* Usage, from gitnexus/:
* npx vitest run --config test/fixtures/mutation-scenario-map/vitest.config.ts
*/
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { defineConfig } from 'vitest/config';
import {
MUTATION_SCOPE_ENV,
parseMutationScope,
resolveMutationScopePath,
} from '../../../tools/mutation/scope-file.js';
const packageDir = fileURLToPath(new URL('../../../', import.meta.url));
/** Every case file of the fixture, relative to the package. */
const ALL_CASES = 'test/fixtures/mutation-scenario-map/*.case.ts';
const include: string[] =
(process.env[MUTATION_SCOPE_ENV] ?? '') === ''
? [ALL_CASES]
: parseMutationScope(readFileSync(resolveMutationScopePath(process.env, packageDir), 'utf8'))
.testFiles;
export default defineConfig({
test: {
pool: 'forks',
isolate: true,
includeTaskLocation: true,
sequence: { shuffle: false, concurrent: false },
// Every test finishes in milliseconds; the limits only bound a mutant that loops.
testTimeout: 5000,
hookTimeout: 5000,
teardownTimeout: 1000,
dangerouslyIgnoreUnhandledErrors: false,
projects: [{ extends: true, test: { name: 'scenario-map-controls', include } }],
},
});

180
gitnexus/test/unit/mutation/blank-assertions.test.ts Normal file
View file

@ -0,0 +1,180 @@
/**
* Locks the `as` assertion blanking of the mutation prepare step (KTD7): every
* `TSAsExpression` is replaced by spaces of equal length so line and column positions are
* unchanged, and a file is accepted only when its blanked AST equals the original AST with
* every assertion unwrapped.
*/
import assert from 'node:assert/strict';
import { describe, expect, it } from 'vitest';
import { blankAssertions, blankScope } from '../../../tools/mutation/prepare/blank-assertions.js';
const spaces = (text: string): string => ' '.repeat(text.length);
/** The blanked text of a source that must pass the AST-equality check. */
const blankedText = (source: string): string => {
const result = blankAssertions(source);
assert.ok(result.ok, 'expected the AST-equality check to pass');
return result.text;
};
describe('blankAssertions: equal-length blanking', () => {
it('blanks `as const` and keeps `]` and `;` at their original columns', () => {
const source = "const T = ['a', 'b'] as const;";
const text = blankedText(source);
expect(blankAssertions(source)).toEqual({
ok: true,
assertions: 1,
text: `const T = ['a', 'b']${spaces(' as const')};`,
});
expect([text.length, text.indexOf(']'), text.indexOf(';')]).toEqual([
source.length,
source.indexOf(']'),
source.indexOf(';'),
]);
});
it('keeps `satisfies` and blanks only the `as const` it wraps', () => {
const source = 'const T = [1, 2] as const satisfies Record<string, number>;';
expect(blankAssertions(source)).toEqual({
ok: true,
assertions: 1,
text: `const T = [1, 2]${spaces(' as const')} satisfies Record<string, number>;`,
});
});
it('blanks both assertions of a nested `x as unknown as T`', () => {
expect(blankAssertions('const y = x as unknown as T;')).toEqual({
ok: true,
assertions: 2,
text: `const y = x${spaces(' as unknown as T')};`,
});
});
it('keeps the newlines of a multi-line asserted type so later lines keep their numbers', () => {
const source = [
'const v = x as {',
' a: string;',
' b: number;',
'};',
'const after = 1;',
].join('\n');
expect(blankedText(source).split('\n')).toEqual([
`const v = x${spaces(' as {')}`,
spaces(' a: string;'),
spaces(' b: number;'),
' ;',
'const after = 1;',
]);
});
it('keeps CRLF line endings inside a blanked type', () => {
expect(blankAssertions('const v = x as {\r\n a: 1 };\r\nf();')).toEqual({
ok: true,
assertions: 1,
text: `const v = x${spaces(' as {')}\r\n${spaces(' a: 1 }')};\r\nf();`,
});
});
it('keeps the non-null `!` operator', () => {
expect(blankAssertions('const n = (m as M)!.k;')).toEqual({
ok: true,
assertions: 1,
text: `const n = (m${spaces(' as M')})!.k;`,
});
});
it('keeps the parentheses around a parenthesised operand and a parenthesised type', () => {
expect(blankAssertions("const p = (['a']) as const, q = x as (A | B);")).toEqual({
ok: true,
assertions: 2,
text: `const p = (['a'])${spaces(' as const')}, q = x${spaces(' as (A | B)')};`,
});
});
it('blanks a comment that sits inside the assertion', () => {
expect(blankAssertions('const c = x as /* why */ T;')).toEqual({
ok: true,
assertions: 1,
text: `const c = x${spaces(' as /* why */ T')};`,
});
});
});
describe('blankAssertions: AST-equality check', () => {
it.each([
[
'a parenthesised member access',
'const b = (a as T).b;',
`const b = (a${spaces(' as T')}).b;`,
],
['an arrow body', 'const f = () => x as T;', `const f = () => x${spaces(' as T')};`],
['an assignment target', '(x as T) = y;', `(x${spaces(' as T')}) = y;`],
])('accepts %s', (_label, source, text) => {
expect(blankAssertions(source)).toEqual({ ok: true, assertions: 1, text });
});
it.each([
['operator precedence', 'const r = a + b as T * c;'],
['a next line starting with `(` after no semicolon', 'const f = g as Fn\n(h as H).call();'],
['a next line starting with `[` after no semicolon', 'const a = b as B\n[1].forEach(f);'],
['a next line starting with a template literal', 'const s = t as Tag\n`x`;'],
])('rejects blanking that changes the tree: %s', (_label, source) => {
expect(blankAssertions(source)).toMatchObject({
ok: false,
reason: expect.stringMatching(/^AST differs/),
});
});
it('rejects a source that does not parse', () => {
expect(blankAssertions('const = ;')).toMatchObject({
ok: false,
assertions: 0,
reason: expect.stringMatching(/^original does not parse/),
});
});
});
describe('blankAssertions: files without `as` assertions', () => {
it.each([
['plain code', 'export function f(a: number): number {\n return a + 1;\n}\n'],
['`satisfies` and non-null `!` only', 'const c = { a: 1 } satisfies R;\nconst n = m!.k;\n'],
[
'`as` in strings, comments and import renames',
"import { a as b } from 'm';\n// x as y\nconst s = 'p as q';\n",
],
['an angle-bracket assertion', 'const n = <number>v;\n'],
])('leaves %s byte-identical', (_label, source) => {
expect(blankAssertions(source)).toEqual({ ok: true, assertions: 0, text: source });
});
});
describe('blankScope', () => {
it('reports a file that fails the AST check instead of writing it', () => {
const scope = blankScope([
{ path: 'gitnexus/src/good.ts', source: 'export const T = [1] as const;\n' },
{ path: 'gitnexus/src/bad.ts', source: 'export const r = a + b as T * c;\n' },
]);
expect(scope.written).toEqual([
{
path: 'gitnexus/src/good.ts',
assertions: 1,
text: `export const T = [1]${spaces(' as const')};\n`,
},
]);
expect(scope.failed).toEqual([
{ path: 'gitnexus/src/bad.ts', reason: expect.stringMatching(/^AST differs/) },
]);
});
it('writes a file with no assertions unchanged', () => {
const source = 'export const x = 1;\n';
expect(blankScope([{ path: 'gitnexus/src/plain.ts', source }])).toEqual({
written: [{ path: 'gitnexus/src/plain.ts', assertions: 0, text: source }],
failed: [],
});
});
});

365
gitnexus/test/unit/mutation/classify-outcome.test.ts Normal file
View file

@ -0,0 +1,365 @@
/**
* Locks the KTD17 outcome-class table of the vitest5 runner: every observation row maps to its
* class and to the status the runner reports to Stryker, from synthetic task results.
*/
import { describe, expect, it } from 'vitest';
import {
classifyRun,
type ObservedError,
type ObservedModule,
type ObservedRun,
type ObservedTest,
type RequestedTest,
} from '../../../tools/mutation/runner/classify-outcome.js';
const FILE = 'test/unit/a.test.ts';
const requested = (index: number, file = FILE): RequestedTest => ({
id: `${file}#S > t${index}`,
taskId: `${file}_${index}`,
fullName: `S > t${index}`,
file,
});
const error = (name: string, message: string): ObservedError => ({ name, message });
const test = (index: number, overrides: Partial<ObservedTest> = {}, file = FILE): ObservedTest => ({
taskId: `${file}_${index}`,
fullName: `S > t${index}`,
state: 'passed',
errors: [],
failedHooks: [],
retry: 0,
...overrides,
});
const module = (
tests: ObservedTest[],
overrides: Partial<ObservedModule> = {},
): ObservedModule => ({
file: FILE,
collected: true,
errors: [],
failedHooks: [],
suites: [
{ taskId: 'suite-S', failedHooks: [], errors: [], testTaskIds: tests.map((t) => t.taskId) },
],
tests,
hitCount: 0,
...overrides,
});
const run = (modules: ObservedModule[], overrides: Partial<ObservedRun> = {}): ObservedRun => ({
requested: [requested(0), requested(1)],
modules,
unhandledErrors: [],
hitLimit: 1000,
...overrides,
});
const assertionError = error('AssertionError', 'expected 1 to be 2');
describe('classifyRun: killed-by-test rows', () => {
it.each([
[
'an AssertionError',
test(1, { state: 'failed', errors: [assertionError] }),
'killed-by-test:assertion',
],
[
'another error',
test(1, { state: 'failed', errors: [error('TypeError', 'x is not a function')] }),
'killed-by-test:runtime',
],
[
'a failing beforeEach',
test(1, { state: 'failed', errors: [assertionError], failedHooks: ['beforeEach'] }),
'killed-by-test:hook',
],
[
'a failing afterEach after a passing body',
test(1, { state: 'failed', errors: [error('Error', 'cleanup')], failedHooks: ['afterEach'] }),
'killed-by-test:hook',
],
[
'a failing body followed by a failing afterEach',
test(1, {
state: 'failed',
errors: [assertionError, error('Error', 'cleanup')],
failedHooks: ['afterEach'],
}),
'killed-by-test:assertion',
],
])('classifies a test that fails with %s', (_label, failing, outcome) => {
const result = classifyRun(run([module([test(0), failing])]));
expect(result).toMatchObject({ status: 'killed', outcome, killedBy: [`${FILE}#S > t1`] });
expect(result.tests).toEqual([
{ id: `${FILE}#S > t0`, outcome: 'survived', message: undefined },
{ id: `${FILE}#S > t1`, outcome, message: failing.errors[0].message },
]);
});
it('reports every failing test as a killer', () => {
const failing = { state: 'failed', errors: [assertionError] } as const;
expect(classifyRun(run([module([test(0, failing), test(1, failing)])])).killedBy).toEqual([
`${FILE}#S > t0`,
`${FILE}#S > t1`,
]);
});
});
describe('classifyRun: killed-outside-test rows', () => {
it('attributes a suite beforeAll failure to the suite tests that it skipped', () => {
const skipped = { state: 'skipped' } as const;
const result = classifyRun(
run([
module([test(0, skipped), test(1, skipped)], {
suites: [
{
taskId: 'suite-S',
failedHooks: ['beforeAll'],
errors: [error('Error', 'connect failed')],
testTaskIds: [`${FILE}_0`, `${FILE}_1`],
},
],
}),
]),
);
expect(result).toMatchObject({
status: 'killed',
outcome: 'killed-outside-test:hook',
killedBy: [`${FILE}#S > t0`, `${FILE}#S > t1`],
outsideTest: [
{
file: FILE,
suiteTaskId: 'suite-S',
outcome: 'killed-outside-test:hook',
attributedTo: [`${FILE}#S > t0`, `${FILE}#S > t1`],
message: 'connect failed',
},
],
});
});
it('attributes a file-level beforeAll failure to every requested test of the file', () => {
const skipped = { state: 'skipped' } as const;
const result = classifyRun(
run([
module([test(0, skipped), test(1, skipped)], {
failedHooks: ['beforeAll'],
errors: [error('Error', 'file setup failed')],
}),
]),
);
expect(result).toMatchObject({
status: 'killed',
killedBy: [`${FILE}#S > t0`, `${FILE}#S > t1`],
});
});
it('kills without test attribution when afterAll fails and the tests stay passing', () => {
const result = classifyRun(
run([
module([test(0), test(1)], {
suites: [
{
taskId: 'suite-S',
failedHooks: ['afterAll'],
errors: [error('Error', 'teardown failed')],
testTaskIds: [`${FILE}_0`, `${FILE}_1`],
},
],
}),
]),
);
expect(result).toMatchObject({
status: 'killed',
outcome: 'killed-outside-test:hook',
killedBy: [],
outsideTest: [{ suiteTaskId: 'suite-S', attributedTo: [], message: 'teardown failed' }],
});
});
it('kills without test attribution when a test file fails to import', () => {
const result = classifyRun(
run([
module([], {
collected: false,
suites: [],
errors: [error('SyntaxError', 'Unexpected token')],
hitCount: undefined,
}),
]),
);
expect(result).toMatchObject({
status: 'killed',
outcome: 'killed-outside-test:collection',
killedBy: [],
nrOfTests: 0,
outsideTest: [{ file: FILE, outcome: 'killed-outside-test:collection', attributedTo: [] }],
});
});
});
describe('classifyRun: timeout rows', () => {
it('reports Timeout when the summed hit count passes the hit limit, even with passing tests', () => {
const result = classifyRun(
run(
[
module([test(0)], { hitCount: 60 }),
module([test(1)], { file: 'test/unit/b.test.ts', hitCount: 50 }),
],
{
requested: [requested(0), { ...requested(1), file: 'test/unit/b.test.ts' }],
hitLimit: 100,
},
),
);
expect(result).toMatchObject({
status: 'timeout',
outcome: 'timeout',
reason: 'Hit limit reached (110/100)',
});
});
it('reports Timeout when the hit-limit error stopped collection', () => {
const result = classifyRun(
run([
module([], {
collected: false,
suites: [],
errors: [error('Error', 'Stryker: Hit count limit reached (101)')],
hitCount: undefined,
}),
]),
);
expect(result).toMatchObject({ status: 'timeout', outcome: 'timeout' });
});
it.each([
['testTimeout', 'Test timed out in 5000ms.\nIf this is a long-running test, ...'],
['hookTimeout', 'Hook timed out in 10000ms.\nIf this is a long-running hook, ...'],
])('reports Timeout for a test that fails with Vitest %s', (_label, message) => {
const result = classifyRun(
run([module([test(0), test(1, { state: 'failed', errors: [error('Error', message)] })])]),
);
expect(result).toMatchObject({ status: 'timeout', outcome: 'timeout', killedBy: [] });
expect(result.tests[1]).toEqual({ id: `${FILE}#S > t1`, outcome: 'timeout', message });
});
it('reports Timeout for a beforeAll that hits Vitest hookTimeout', () => {
const skipped = { state: 'skipped' } as const;
const result = classifyRun(
run([
module([test(0, skipped), test(1, skipped)], {
suites: [
{
taskId: 'suite-S',
failedHooks: ['beforeAll'],
errors: [error('Error', 'Hook timed out in 120000ms.')],
testTaskIds: [`${FILE}_0`, `${FILE}_1`],
},
],
}),
]),
);
expect(result).toMatchObject({ status: 'timeout', outcome: 'timeout', killedBy: [] });
});
it('prefers a kill over a Vitest timeout in another test', () => {
const result = classifyRun(
run([
module([
test(0, { state: 'failed', errors: [error('Error', 'Test timed out in 5000ms.')] }),
test(1, { state: 'failed', errors: [assertionError] }),
]),
]),
);
expect(result).toMatchObject({ status: 'killed', killedBy: [`${FILE}#S > t1`] });
});
});
describe('classifyRun: error rows', () => {
it('reports an unhandled error as infrastructure', () => {
const result = classifyRun(
run([module([test(0), test(1)])], {
unhandledErrors: [error('Error', 'socket hang up')],
}),
);
expect(result).toMatchObject({ status: 'error', outcome: 'infrastructure' });
expect(result.reason).toContain('socket hang up');
});
it('reports a selected test with retry above zero as inconclusive', () => {
const result = classifyRun(run([module([test(0), test(1, { retry: 2 })])]));
expect(result).toMatchObject({ status: 'error', outcome: 'inconclusive' });
expect(result.reason).toContain('retry');
});
it.each([
['zero requested tests', run([module([test(0), test(1)])], { requested: [] }), /zero tests/],
[
'a requested test that did not run',
run([module([test(0), test(1, { state: 'skipped' })])]),
/did not run/,
],
[
'a test that ran without being requested',
run([module([test(0), test(1), test(2)])]),
/not requested/,
],
[
'a task id whose full name changed',
run([module([test(0), test(1, { fullName: 'S > renamed' })])]),
/full name/,
],
['a requested file that produced no module', run([]), /did not run/],
])('reports %s as invalid', (_label, observed, reason) => {
const result = classifyRun(observed);
expect(result).toMatchObject({ status: 'error', outcome: 'invalid' });
expect(result.reason).toMatch(reason);
});
});
describe('classifyRun: survived row', () => {
it('reports Survived when every requested test passes', () => {
expect(classifyRun(run([module([test(0), test(1)])]))).toEqual({
status: 'survived',
outcome: 'survived',
reason: undefined,
killedBy: [],
failureMessage: undefined,
nrOfTests: 2,
hitCount: 0,
tests: [
{ id: `${FILE}#S > t0`, outcome: 'survived', message: undefined },
{ id: `${FILE}#S > t1`, outcome: 'survived', message: undefined },
],
outsideTest: [],
});
});
it('ignores tests that the selection skipped', () => {
const result = classifyRun(
run([module([test(0), test(1), test(2, { state: 'skipped' })])], {
requested: [requested(0), requested(1)],
}),
);
expect(result).toMatchObject({ status: 'survived', nrOfTests: 2 });
});
});

574
gitnexus/test/unit/mutation/classify.test.ts Normal file
View file

@ -0,0 +1,574 @@
/**
* Locks the pure-JS category classifier (U5, KTD11): which `test/unit` files a mutation run may
* use without native code, workers, child processes, or file reads that a prepared copy would
* break. Real-repo cases run against the checkout; synthetic cases build a small repo in a temp
* directory (`gitnexus/`, `gitnexus-shared/`, `node_modules/`) and classify one test file in it.
*/
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { afterAll, describe, expect, it } from 'vitest';
import {
classify,
renderSummary,
type Classification,
type FileVerdict,
} from '../../../tools/mutation/classify/classify.js';
const REPO_ROOT = fileURLToPath(new URL('../../../../', import.meta.url));
// ── Real repo ──────────────────────────────────────────────────────────────
const REAL_ACCEPTED = [
'test/unit/shared-type-extractors.test.ts',
'test/unit/extract-element-type-from-string.test.ts',
'test/unit/schema.test.ts',
];
const REAL_REJECTED: readonly (readonly [file: string, reason: string])[] = [
['test/unit/receiver-twin-list-drift.test.ts', 'reads source outside gitnexus/'],
['test/unit/type-env.test.ts', 'native package: tree-sitter'],
['test/unit/node-module-compat.test.ts', 'spawns processes: child_process'],
// Found by gate 1's runtime load check: commander loads node:child_process when imported.
['test/unit/cli-command-banner.test.ts', 'spawns processes: child_process'],
];
let realResult: Classification | undefined;
const real = (): Classification =>
(realResult ??= classify({
repoRoot: REPO_ROOT,
testFiles: [...REAL_ACCEPTED, ...REAL_REJECTED.map(([file]) => file)],
}));
const verdictIn = (result: Classification, file: string): FileVerdict | undefined =>
result.files.find((verdict) => verdict.file === file);
describe('classify on the real repo', () => {
// Walking the real import closures takes about 30 s on a busy host; do it once, with headroom.
beforeAll(() => {
real();
}, 120_000);
it.each(REAL_ACCEPTED)('accepts %s', (file) => {
expect(verdictIn(real(), file)).toMatchObject({ status: 'accepted', reasons: [] });
});
it.each(REAL_REJECTED)('rejects %s with "%s"', (file, reason) => {
expect(verdictIn(real(), file)?.status).toBe('rejected');
expect(verdictIn(real(), file)?.reasons).toContain(reason);
});
it('finds child_process in node-module-compat through the dynamic import of runtime-install', () => {
const evidence = verdictIn(real(), 'test/unit/node-module-compat.test.ts')?.evidence.find(
(entry) => entry.reason === 'spawns processes: child_process',
);
expect(evidence).toMatchObject({
module: 'gitnexus/src/core/embeddings/runtime-install.ts',
via: [
'gitnexus/test/unit/node-module-compat.test.ts',
'gitnexus/src/core/embeddings/runtime-install.ts',
],
});
});
it('resolves the gitnexus-shared alias into gitnexus-shared/src for schema.test.ts', () => {
expect(verdictIn(real(), 'test/unit/schema.test.ts')?.closure).toContain(
'gitnexus-shared/src/lbug/schema-constants.ts',
);
});
});
// ── Synthetic repos ────────────────────────────────────────────────────────
const BASE_FILES: Readonly<Record<string, string>> = {
'gitnexus/package.json': '{ "name": "gitnexus", "type": "module" }\n',
'gitnexus-shared/package.json': JSON.stringify({
name: 'gitnexus-shared',
type: 'module',
exports: {
'.': { types: './dist/index.d.ts', default: './dist/index.js' },
'./test-helpers': { types: './dist/test-helpers.d.ts', default: './dist/test-helpers.js' },
},
}),
'gitnexus-shared/src/index.ts': "export { SHARED } from './constants.js';\n",
'gitnexus-shared/src/constants.ts': "export const SHARED = 'shared';\n",
'gitnexus-shared/src/test-helpers.ts': 'export const helper = (): number => 1;\n',
'gitnexus/src/pure.ts': 'export const add = (a: number, b: number): number => a + b;\n',
'gitnexus/src/x.ts': 'export const x = 1;\n',
'gitnexus/src/spawn.ts':
"import { execFileSync } from 'node:child_process';\nexport const run = () => execFileSync('true');\n",
'gitnexus/node_modules/native-addon/package.json':
'{ "name": "native-addon", "main": "index.js" }',
'gitnexus/node_modules/native-addon/index.js':
"module.exports = require('./build/Release/a.node');\n",
'gitnexus/node_modules/native-addon/build/Release/a.node': 'binary',
'gitnexus/node_modules/gyp-addon/package.json': '{ "name": "gyp-addon", "main": "index.js" }',
'gitnexus/node_modules/gyp-addon/binding.gyp': '{}',
'gitnexus/node_modules/gyp-addon/index.js': 'module.exports = {};\n',
'gitnexus/node_modules/wrapper/package.json':
'{ "name": "wrapper", "main": "index.js", "dependencies": { "native-addon": "1.0.0" } }',
'gitnexus/node_modules/wrapper/index.js': 'module.exports = {};\n',
'gitnexus/node_modules/plain-js/package.json':
'{ "name": "plain-js", "main": "index.js", "dependencies": { "missing-optional": "1.0.0" } }',
'gitnexus/node_modules/plain-js/index.js': 'module.exports = {};\n',
};
const TEST_FILE = 'test/unit/a.test.ts';
const TEST_PATH = `gitnexus/${TEST_FILE}`;
const roots: string[] = [];
afterAll(() => {
for (const root of roots) rmSync(root, { recursive: true, force: true });
});
function makeRepo(files: Readonly<Record<string, string>>): string {
const root = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-classify-'));
roots.push(root);
for (const [relative, text] of Object.entries({ ...BASE_FILES, ...files })) {
const absolute = path.join(root, relative);
mkdirSync(path.dirname(absolute), { recursive: true });
writeFileSync(absolute, text);
}
return root;
}
/** Classifies the single test file `gitnexus/test/unit/a.test.ts` of a synthetic repo. */
function classifyTest(
testSource: string,
extra: Readonly<Record<string, string>> = {},
): FileVerdict {
const root = makeRepo({ ...extra, [TEST_PATH]: testSource });
return classify({ repoRoot: root, testFiles: [TEST_FILE] }).files[0];
}
const NODE_IMPORTS = [
"import { readFileSync, mkdtempSync } from 'node:fs';",
"import os from 'node:os';",
"import path from 'node:path';",
"import { fileURLToPath } from 'node:url';",
"import { it, expect, vi } from 'vitest';",
].join('\n');
const withTest = (body: string): string =>
`${NODE_IMPORTS}\n${body}\nit('t', () => expect(1).toBe(1));\n`;
const CREATE_REQUIRE_LOADER = (specifier: string, base = 'import.meta.url'): string =>
[
"import { createRequire } from 'node:module';",
`const require = createRequire(${base});`,
`export const binding = require(${specifier});`,
].join('\n');
const REJECTIONS: readonly (readonly [
label: string,
testSource: string,
extra: Readonly<Record<string, string>>,
reason: string,
])[] = [
[
'reads src/x.ts as text through __dirname',
withTest("const text = readFileSync(path.join(__dirname, '../../src/x.ts'), 'utf8');"),
{},
'reads source under src/',
],
[
'reads src/x.ts as text through import.meta.url',
withTest("const text = readFileSync(new URL('../../src/x.ts', import.meta.url), 'utf8');"),
{},
'reads source under src/',
],
[
'reads src/x.ts by a cwd-relative literal path',
withTest("const text = readFileSync('src/x.ts', 'utf8');"),
{},
'reads source under src/',
],
[
'reads src/x.ts through const-bound roots',
withTest(
[
'const here = path.dirname(fileURLToPath(import.meta.url));',
"const ROOT = path.resolve(here, '..', '..');",
"const text = readFileSync(path.join(ROOT, 'src', 'x.ts'), 'utf8');",
].join('\n'),
),
{},
'reads source under src/',
],
[
'reads src/x.ts through a function-local require of path behind a type assertion',
withTest(
[
"it('reads', () => {",
" const p = require('node:path') as typeof import('node:path');",
" const fsLocal = require('node:fs') as typeof import('node:fs');",
" const source = fsLocal.readFileSync(p.resolve(__dirname, '../../src/x.ts'), 'utf8');",
' expect(source).toBeTruthy();',
'});',
].join('\n'),
),
{},
'reads source under src/',
],
[
'reads src/x.ts through a name that another function binds differently',
withTest(
[
'export function temp() {',
' const here = os.tmpdir();',
" return readFileSync(path.join(here, 'a.txt'), 'utf8');",
'}',
'export function source() {',
' const here = path.dirname(fileURLToPath(import.meta.url));',
" return readFileSync(path.resolve(here, '../../src/x.ts'), 'utf8');",
'}',
].join('\n'),
),
{},
'reads source under src/',
],
[
'names a file under src/ with an unknown name',
withTest(
"export const read = (name: string) => readFileSync(path.join(__dirname, '../../src', name));",
),
{},
'reads source under src/',
],
[
'reads test fixtures',
withTest("const text = readFileSync(path.join(__dirname, '../fixtures/repo/a.ts'), 'utf8');"),
{},
'reads test fixtures',
],
[
'reads a file outside gitnexus/',
withTest(
"const text = readFileSync(path.join(__dirname, '../../../gitnexus-shared/src/index.ts'), 'utf8');",
),
{},
'reads source outside gitnexus/',
],
[
'names the package root with an unknown suffix',
withTest(
"export const read = (name: string) => readFileSync(path.join(__dirname, '../..', name));",
),
{},
'unresolvable file path',
],
[
'concatenates an unknown suffix onto the package root',
withTest("export const read = (name: string) => readFileSync(process.cwd() + name, 'utf8');"),
{},
'unresolvable file path',
],
[
'loads a native package through createRequire in a production module',
withTest("import { binding } from '../../src/loader.js';\nconsole.log(binding);"),
{ 'gitnexus/src/loader.ts': CREATE_REQUIRE_LOADER("'native-addon'") },
'native package: native-addon',
],
[
'loads a local .node binding through createRequire',
withTest("import { binding } from '../../src/loader.js';\nconsole.log(binding);"),
{
'gitnexus/src/loader.ts': CREATE_REQUIRE_LOADER("'./addon.node'"),
'gitnexus/src/addon.node': 'binary',
},
'native binding: gitnexus/src/addon.node',
],
[
'imports a package that depends on a native package',
withTest("import wrapper from 'wrapper';\nconsole.log(wrapper);"),
{},
'native package: native-addon',
],
[
'imports a package with binding.gyp',
withTest("import gyp from 'gyp-addon';\nconsole.log(gyp);"),
{},
'native package: gyp-addon',
],
[
'imports a package known to load child_process when imported',
withTest("import { Command } from 'commander';\nconsole.log(new Command());"),
{},
'spawns processes: child_process',
],
[
'imports a known native package that is not installed',
withTest("import Parser from 'tree-sitter';\nconsole.log(new Parser());"),
{},
'native package: tree-sitter',
],
[
'mocks a native package with a factory that receives importOriginal',
withTest(
"vi.mock('tree-sitter', async (importOriginal) => ({ ...(await importOriginal<object>()) }));",
),
{},
'native package: tree-sitter',
],
[
'imports a computed specifier',
withTest('export const load = (name: string) => import(name);'),
{},
'unresolvable dynamic import',
],
[
'imports a template specifier with an expression',
withTest('export const load = (name: string) => import(`../../src/${name}.js`);'),
{},
'unresolvable dynamic import',
],
[
'requires a computed specifier in a production module',
withTest("import { binding } from '../../src/loader.js';\nconsole.log(binding);"),
{ 'gitnexus/src/loader.ts': 'export const binding = (name: string) => require(name);\n' },
'unresolvable dynamic import',
],
[
'requires a relative specifier through createRequire with an unknown base',
withTest("import { binding } from '../../src/loader.js';\nconsole.log(binding);"),
{ 'gitnexus/src/loader.ts': CREATE_REQUIRE_LOADER("'./x.js'", 'process.argv[1]') },
'unresolvable dynamic import',
],
[
'loads a computed specifier through vi.importActual',
withTest('export const load = (name: string) => vi.importActual(name);'),
{},
'unresolvable dynamic import',
],
[
'reaches child_process through a const-bound dynamic import',
withTest("const MOD = '../../src/spawn.js';\nexport const load = () => import(MOD);"),
{},
'spawns processes: child_process',
],
[
'imports Worker from worker_threads',
withTest("import { Worker } from 'node:worker_threads';\nconsole.log(Worker);"),
{},
'spawns workers: worker_threads',
],
[
'imports a module outside gitnexus/ by a relative path',
withTest(
"import { SHARED } from '../../../gitnexus-shared/src/index.js';\nconsole.log(SHARED);",
),
{},
'imports outside gitnexus/',
],
[
'imports a module from test/fixtures',
withTest("import { fixture } from '../fixtures/data.js';\nconsole.log(fixture);"),
{ 'gitnexus/test/fixtures/data.ts': 'export const fixture = 1;\n' },
'imports from ignored dir: test/fixtures/',
],
[
'imports a missing relative module',
withTest("import { missing } from '../../src/missing.js';\nconsole.log(missing);"),
{},
'unresolved import: ../../src/missing.js',
],
];
const ACCEPTANCES: readonly (readonly [
label: string,
testSource: string,
extra: Readonly<Record<string, string>>,
])[] = [
[
'a production closure that imports fs while the test reads no files',
withTest("import { add } from '../../src/io.js';\nconsole.log(add(1, 2));"),
{
'gitnexus/src/io.ts': [
"import { readFileSync } from 'node:fs';",
"export const read = (file: string) => readFileSync(file, 'utf8');",
'export const add = (a: number, b: number) => a + b;',
].join('\n'),
},
],
[
'a test that reads files in its own temp directory',
withTest(
[
"const dir = mkdtempSync(path.join(os.tmpdir(), 'x-'));",
"export const text = () => readFileSync(path.join(dir, 'a.txt'), 'utf8');",
].join('\n'),
),
{},
],
[
'a test that reads a data file next to itself',
withTest("const data = readFileSync(path.join(__dirname, 'data.json'), 'utf8');"),
{ 'gitnexus/test/unit/data.json': '{}' },
],
[
'a path concatenated from process.cwd() and path.sep',
withTest("const pkg = readFileSync(process.cwd() + path.sep + 'package.json', 'utf8');"),
{},
],
[
'type-only and type-position imports of native packages',
withTest(
[
"import type Parser from 'tree-sitter';",
"import { type Language } from 'tree-sitter-typescript';",
"import { SyntaxNode } from 'tree-sitter';",
'let node: SyntaxNode | Parser | Language | undefined;',
'console.log(node);',
].join('\n'),
),
{},
],
[
'worker_threads imported without Worker',
withTest("import { isMainThread } from 'node:worker_threads';\nconsole.log(isMainThread);"),
{},
],
[
'a native package mocked with a factory that does not load the original',
withTest("vi.mock('tree-sitter', () => ({ default: class {} }));"),
{},
],
[
'a const-bound dynamic import of a pure module',
withTest("const MOD = '../../src/pure.js';\nexport const load = () => import(MOD);"),
{},
],
[
'a package whose missing dependency is not installed',
withTest("import plain from 'plain-js';\nconsole.log(plain);"),
{},
],
[
'the gitnexus-shared aliases',
withTest(
[
"import { SHARED } from 'gitnexus-shared';",
"import { helper } from 'gitnexus-shared/test-helpers';",
'console.log(SHARED, helper);',
].join('\n'),
),
{},
],
];
describe('classify on synthetic repos', () => {
it.each(REJECTIONS)('rejects a test that %s', (_label, testSource, extra, reason) => {
const verdict = classifyTest(testSource, extra);
expect(verdict.status).toBe('rejected');
expect(verdict.reasons).toContain(reason);
});
it.each(ACCEPTANCES)('accepts %s', (_label, testSource, extra) => {
expect(classifyTest(testSource, extra)).toMatchObject({ status: 'accepted', reasons: [] });
});
it('records the local closure, packages and builtins of an accepted file', () => {
const verdict = classifyTest(
withTest(
[
"import { SHARED } from 'gitnexus-shared';",
"import plain from 'plain-js';",
"const MOD = '../../src/pure.js';",
'export const load = () => import(MOD);',
'console.log(SHARED, plain, path.sep);',
].join('\n'),
),
);
expect(verdict.closure).toEqual([
'gitnexus-shared/src/constants.ts',
'gitnexus-shared/src/index.ts',
'gitnexus/src/pure.ts',
TEST_PATH,
]);
expect(verdict.packages).toEqual(['plain-js', 'vitest']);
expect(verdict.builtins).toEqual(['path']);
});
it('gives each rejection reason its module, line and import chain', () => {
const verdict = classifyTest(
withTest("import { load } from '../helpers/load.js';\nconsole.log(load);"),
{ 'gitnexus/test/helpers/load.ts': "export { run as load } from '../../src/spawn.js';\n" },
);
expect(verdict.evidence).toEqual([
{
reason: 'spawns processes: child_process',
module: 'gitnexus/src/spawn.ts',
line: 1,
detail: 'imports node:child_process',
via: [TEST_PATH, 'gitnexus/test/helpers/load.ts', 'gitnexus/src/spawn.ts'],
},
]);
});
it('counts hooks, each-calls and test call sites of the test file', () => {
const verdict = classifyTest(
[
"import { beforeAll, describe, it, expect } from 'vitest';",
'let n = 0;',
'beforeAll(() => { n = 1; });',
"describe('d', () => {",
' beforeAll(() => { n += 1; });',
" it('one', () => expect(n).toBe(2));",
" it.each([1, 2])('row %i', (v) => expect(v).toBeGreaterThan(0));",
'});',
].join('\n'),
);
expect(verdict).toMatchObject({
status: 'accepted',
hooks: { beforeAll: 2, beforeEach: 0, afterAll: 0, afterEach: 0 },
eachCalls: 1,
testCalls: 2,
});
});
});
describe('classify over a test root', () => {
const root = (): string =>
makeRepo({
'gitnexus/test/unit/b.test.ts': withTest(
"import { add } from '../../src/pure.js';\nadd(1, 2);",
),
'gitnexus/test/unit/nested/c.test.ts': withTest(
"import { run } from '../../../src/spawn.js';\nbeforeAll(() => run());",
),
'gitnexus/test/unit/hooks.test.ts': withTest(
"import { add } from '../../src/pure.js';\nbeforeAll(() => add(1, 2));",
),
'gitnexus/test/unit/helper.ts': 'export const h = 1;\n',
});
it('lists every *.test.ts file under the root, sorted, and groups the verdicts', () => {
const result = classify({ repoRoot: root(), testRoot: 'test/unit' });
expect(result.files.map((verdict) => verdict.file)).toEqual([
'test/unit/b.test.ts',
'test/unit/hooks.test.ts',
'test/unit/nested/c.test.ts',
]);
expect(result.accepted).toEqual(['test/unit/b.test.ts', 'test/unit/hooks.test.ts']);
expect(result.rejected).toEqual([
{ file: 'test/unit/nested/c.test.ts', reasons: ['spawns processes: child_process'] },
]);
expect(result.acceptedWithBeforeAll).toEqual(['test/unit/hooks.test.ts']);
expect(result.summary).toEqual({
total: 3,
accepted: 2,
rejected: 1,
byReason: { 'spawns processes: child_process': 1 },
byCategory: { 'spawns processes': 1 },
});
});
it('is deterministic across runs and renders a summary', () => {
const repo = root();
const first = classify({ repoRoot: repo, testRoot: 'test/unit' });
expect(classify({ repoRoot: repo, testRoot: 'test/unit' })).toEqual(first);
expect(renderSummary(first)).toContain('3 test files: 2 accepted, 1 rejected');
});
});

579
gitnexus/test/unit/mutation/conformance-compare.test.ts Normal file
View file

@ -0,0 +1,579 @@
/**
* Locks the comparison logic of the gate 2 conformance check (U4): a run that matches every
* pinned case passes, a change to any pinned fact fails naming its case, and a missing or an
* unexpected mutant, a broken process rule or a wrong exported mutant id are reported. Pure
* functions only; the Stryker runs are exercised by `npm run test:mutation:conformance`.
*/
import { describe, expect, it } from 'vitest';
import {
compareConformance,
fixtureRelative,
formatDifference,
observeScopeRun,
parseExpected,
type ExpectedCase,
type ExpectedConformance,
type MutantFacts,
type MutantSelector,
type ObservedMutant,
type ObservedScopeRun,
type ObserveInput,
type ScopeSpec,
} from '../../../tools/mutation/conformance/compare.js';
import { driverScope } from '../../../tools/mutation/conformance/run-conformance.js';
import type {
NormalisedMutant,
NormalisedTest,
} from '../../../tools/mutation/driver/normalise-results.js';
import type { RunClassification } from '../../../tools/mutation/runner/classify-outcome.js';
import {
SIDECAR_SCHEMA_VERSION,
type SidecarRecord,
} from '../../../tools/mutation/runner/sidecar.js';
const TWO = 'runtime.case.ts#doubles two';
const THREE = 'runtime.case.ts#doubles three';
const ROOT = 'test/fixtures/mutation-conformance';
const PROFILE = 'concurrency-1-reuse-unlimited';
const CASE = 'two tests kill the same mutant';
const DOUBLE: MutantSelector = {
file: 'src/runtime.ts',
location: '7:10-7:19',
mutatorName: 'ArithmeticOperator',
replacement: 'value / 2',
};
const TICKET: MutantSelector = {
file: 'src/counter.ts',
location: '10:3-10:14',
mutatorName: 'AssignmentOperator',
replacement: 'issued -= 1',
};
const FACTS: MutantFacts = {
status: 'Killed',
static: false,
outcome: 'killed-by-test:assertion',
outcomeSource: 'sidecar',
coveredBy: [TWO, THREE],
selected: [TWO, THREE],
killedBy: [TWO, THREE],
assertionKilledBy: [TWO, THREE],
tests: { [TWO]: 'killed-by-test:assertion', [THREE]: 'killed-by-test:assertion' },
outsideTest: [],
reason: null,
flags: [],
hookAttributedTests: [],
attemptsDisagree: false,
};
const TICKET_FACTS: MutantFacts = {
...FACTS,
coveredBy: ['runtime.case.ts#issues the first ticket'],
selected: ['runtime.case.ts#issues the first ticket'],
killedBy: ['runtime.case.ts#issues the first ticket'],
assertionKilledBy: ['runtime.case.ts#issues the first ticket'],
tests: { 'runtime.case.ts#issues the first ticket': 'killed-by-test:assertion' },
};
const TESTS = [
{ id: TWO, skipped: false, notOracleValidated: false },
{ id: THREE, skipped: false, notOracleValidated: false },
];
const mutantCase = (name: string, mutant: MutantSelector, facts: MutantFacts): ExpectedCase => ({
kind: 'mutant',
name,
scope: 'main',
mutant,
expect: facts,
});
/** The expected document, round-tripped through the parser like expected.json. */
const MAIN_SCOPE: ScopeSpec = {
name: 'main',
testFiles: ['runtime.case.ts'],
mutate: ['src/runtime.ts'],
testIds: null,
tests: TESTS,
};
const expectedWith = (
cases: readonly ExpectedCase[],
scopes: readonly ScopeSpec[] = [MAIN_SCOPE],
): ExpectedConformance =>
parseExpected(
JSON.stringify({
schemaVersion: 1,
profiles: [{ name: PROFILE, concurrency: 1, maxTestRunnerReuse: 0 }],
scopes,
cases,
}),
);
const BASE_CASES: readonly ExpectedCase[] = [
mutantCase(CASE, DOUBLE, FACTS),
mutantCase('counter carry-over', TICKET, TICKET_FACTS),
{
kind: 'processes',
name: 'one process runs every mutant, back to back',
profile: PROFILE,
rule: 'every-run-in-the-dry-run-process',
consecutive: ['counter carry-over', CASE],
},
{
kind: 'environment',
name: 'exported active mutant',
profile: PROFILE,
activeMutantOf: CASE,
localId: '32',
},
];
const observedMutant = (
selector: MutantSelector,
localId: string,
facts: MutantFacts,
writtenAtMs: number,
): ObservedMutant => ({ ...selector, localId, facts, runs: [{ pid: 7, writtenAtMs }] });
const observedRun = (overrides: Partial<ObservedScopeRun> = {}): ObservedScopeRun => ({
profile: PROFILE,
scope: 'main',
mutants: [observedMutant(TICKET, '10', TICKET_FACTS, 1), observedMutant(DOUBLE, '32', FACTS, 2)],
tests: TESTS,
dryRunPid: 7,
runFlags: [],
spanMapFailures: 0,
exportedActiveMutant: '32',
...overrides,
});
const cases = (differences: ReturnType<typeof compareConformance>): string[] =>
differences.map((difference) => difference.case);
describe('compareConformance', () => {
it('reports nothing when every pinned case holds', () => {
expect(compareConformance(expectedWith(BASE_CASES), [observedRun()])).toEqual([]);
});
const ALTERATIONS: readonly [keyof MutantFacts, Partial<MutantFacts>][] = [
['status', { status: 'Survived' }],
['static', { static: true }],
['outcome', { outcome: 'killed-by-test:runtime' }],
['outcomeSource', { outcomeSource: 'no-coverage' }],
['coveredBy', { coveredBy: [TWO] }],
['selected', { selected: 'all' }],
['killedBy', { killedBy: [THREE] }],
['assertionKilledBy', { assertionKilledBy: [] }],
['tests', { tests: { [TWO]: 'killed-by-test:assertion' } }],
['outsideTest', { outsideTest: null }],
['reason', { reason: 'Hit limit reached' }],
['flags', { flags: ['zero-tests'] }],
['hookAttributedTests', { hookAttributedTests: [TWO] }],
['attemptsDisagree', { attemptsDisagree: true }],
];
it('has an alteration for every pinned fact', () => {
expect(ALTERATIONS.map(([key]) => key).sort()).toEqual(Object.keys(FACTS).sort());
});
it.each(ALTERATIONS)('fails naming the case when the pinned %s changes', (key, patch) => {
const altered = BASE_CASES.map((entry) =>
entry.name === CASE ? mutantCase(CASE, DOUBLE, { ...FACTS, ...patch }) : entry,
);
const differences = compareConformance(expectedWith(altered), [observedRun()]);
expect(differences.map((d) => [d.profile, d.scope, d.case, d.detail.split(':')[0]])).toEqual([
[PROFILE, 'main', CASE, key],
]);
});
it('fails naming the case when no mutant matches its selector', () => {
const run = observedRun({ mutants: [observedMutant(TICKET, '10', TICKET_FACTS, 1)] });
const differences = compareConformance(expectedWith(BASE_CASES.slice(0, 2)), [run]);
expect(differences).toEqual([
{
profile: PROFILE,
scope: 'main',
case: CASE,
detail: `0 mutants match src/runtime.ts:7:10-7:19 ArithmeticOperator "value / 2", expected exactly one`,
},
]);
});
it('reports a mutant that no case selects as unexpected', () => {
const extra = observedMutant({ ...DOUBLE, replacement: 'value + 2' }, '33', FACTS, 3);
const run = observedRun({ mutants: [...observedRun().mutants, extra] });
const differences = compareConformance(expectedWith(BASE_CASES), [run]);
expect(cases(differences)).toEqual(['unexpected mutant']);
expect(differences[0].detail).toContain('local id 33');
});
it('reports a run that is missing altogether', () => {
expect(compareConformance(expectedWith(BASE_CASES), [])).toEqual([
{ profile: PROFILE, scope: 'main', case: 'run', detail: 'no observed run' },
]);
});
it('fails the test inventory when a pinned test flag differs', () => {
const run = observedRun({ tests: [TESTS[0], { ...TESTS[1], notOracleValidated: true }] });
expect(cases(compareConformance(expectedWith(BASE_CASES), [run]))).toEqual([
'test inventory of scope main',
]);
});
it('fails a tests case when one of its tests is missing', () => {
const pinned: ExpectedCase = {
kind: 'tests',
name: 'row ordinals',
scope: 'main',
tests: TESTS,
};
const run = observedRun({ tests: [TESTS[0]] });
expect(cases(compareConformance(expectedWith([...BASE_CASES, pinned]), [run]))).toEqual([
'test inventory of scope main',
'row ordinals',
]);
});
it('fails the process case when a mutant ran in another process than the dry run', () => {
const moved = { ...observedMutant(DOUBLE, '32', FACTS, 2), runs: [{ pid: 8, writtenAtMs: 2 }] };
const run = observedRun({ mutants: [observedMutant(TICKET, '10', TICKET_FACTS, 1), moved] });
const differences = compareConformance(expectedWith(BASE_CASES), [run]);
expect(differences.map((d) => d.detail)).toEqual([
"1 mutant run(s) came from other processes than the dry run's (32)",
`mutant 32 (${CASE}) did not run right after mutant 10 (counter carry-over) in process 7`,
]);
expect(new Set(cases(differences))).toEqual(
new Set(['one process runs every mutant, back to back']),
);
});
it('fails the process case when the consecutive mutants ran in the other order', () => {
const run = observedRun({
mutants: [
observedMutant(TICKET, '10', TICKET_FACTS, 3),
observedMutant(DOUBLE, '32', FACTS, 2),
],
});
expect(cases(compareConformance(expectedWith(BASE_CASES), [run]))).toEqual([
'one process runs every mutant, back to back',
]);
});
it('checks a consecutive pair only in the scope of its mutant cases', () => {
const other: ScopeSpec = {
name: 'other',
testFiles: ['o.case.ts'],
mutate: ['src/o.ts'],
testIds: null,
tests: [],
};
const otherRun = observedRun({ scope: 'other', mutants: [], tests: [] });
const differences = compareConformance(expectedWith(BASE_CASES, [MAIN_SCOPE, other]), [
observedRun(),
otherRun,
]);
expect(differences).toEqual([]);
});
it('fails a one-run-per-process rule when a process handled two mutant runs', () => {
const rule: ExpectedCase = {
kind: 'processes',
name: 'fresh processes',
profile: PROFILE,
rule: 'one-mutant-run-per-process',
consecutive: null,
};
const differences = compareConformance(expectedWith([...BASE_CASES.slice(0, 2), rule]), [
observedRun(),
]);
expect(differences.map((d) => d.detail)).toEqual([
'process 7 handled 2 mutant runs (10, 32)',
'no mutant run came from a process that never ran the dry run',
]);
});
it('fails the environment case when the exported id names another mutant', () => {
const run = observedRun({ exportedActiveMutant: '10' });
const environment: ExpectedCase = {
kind: 'environment',
name: 'exported active mutant',
profile: PROFILE,
activeMutantOf: CASE,
localId: '10',
};
const differences = compareConformance(expectedWith([...BASE_CASES.slice(0, 3), environment]), [
run,
]);
expect(differences.map((d) => [d.case, d.detail])).toEqual([
[
'exported active mutant',
`the exported local id 10 is not the mutant of "${CASE}" (its local id is "32")`,
],
]);
});
it('reports run-level flags and span-map failures', () => {
const run = observedRun({ runFlags: ['dry-run-missing: no record'], spanMapFailures: 2 });
expect(cases(compareConformance(expectedWith(BASE_CASES), [run]))).toEqual([
'span map',
'run flags',
]);
});
it('formats a difference with its profile, scope and case', () => {
expect(
formatDifference({ profile: PROFILE, scope: 'main', case: CASE, detail: 'status: x' }),
).toBe(`[${PROFILE} / main] case "${CASE}": status: x`);
});
});
describe('parseExpected', () => {
const valid = {
schemaVersion: 1,
profiles: [{ name: PROFILE, concurrency: 1, maxTestRunnerReuse: 0 }],
scopes: [
{ name: 'main', testFiles: ['a.case.ts'], mutate: ['src/a.ts'], testIds: null, tests: [] },
],
cases: [
{ kind: 'mutant', name: CASE, scope: 'main', mutant: DOUBLE, expect: FACTS },
] satisfies readonly ExpectedCase[],
};
it('accepts a valid document', () => {
expect(parseExpected(JSON.stringify(valid)).cases.map((entry) => entry.name)).toEqual([CASE]);
});
it.each([
[
'an unknown fact key',
{ ...valid, cases: [{ ...valid.cases[0], expect: { ...FACTS, outcom: 'x' } }] },
/unknown key "outcom"/,
],
[
'an unknown outcome class',
{ ...valid, cases: [{ ...valid.cases[0], expect: { ...FACTS, outcome: 'killed' } }] },
/outcome: must be one of/,
],
[
'a case in an unknown scope',
{ ...valid, cases: [{ ...valid.cases[0], scope: 'other' }] },
/unknown scope other/,
],
['a repeated case name', { ...valid, cases: [valid.cases[0], valid.cases[0]] }, /listed twice/],
[
'a consecutive pair naming a missing case',
{
...valid,
cases: [
...valid.cases,
{
kind: 'processes',
name: 'p',
profile: PROFILE,
rule: 'one-mutant-run-per-process',
consecutive: [CASE, 'nope'],
},
],
},
/"nope" is not a mutant case/,
],
[
'a malformed location',
{ ...valid, cases: [{ ...valid.cases[0], mutant: { ...DOUBLE, location: '7' } }] },
/location: must look like/,
],
['another schema version', { ...valid, schemaVersion: 2 }, /schemaVersion: must be 1/],
[
'an empty test selection',
{ ...valid, scopes: [{ ...valid.scopes[0], testIds: [] }] },
/testIds: must be null or a non-empty array/,
],
[
'a scope without its testIds key',
{ ...valid, scopes: [{ name: 'main', testFiles: ['a.case.ts'], mutate: [], tests: [] }] },
/missing key "testIds"/,
],
])('rejects %s', (_label, document, message) => {
expect(() => parseExpected(JSON.stringify(document))).toThrow(message);
});
});
describe('observeScopeRun', () => {
const ID_TWO = `${ROOT}/${TWO}`;
const ID_THREE = `${ROOT}/${THREE}`;
const classification: RunClassification = {
status: 'killed',
outcome: 'killed-by-test:assertion',
reason: undefined,
killedBy: [ID_TWO],
failureMessage: 'expected 3 to be 4',
nrOfTests: 2,
hitCount: 1,
tests: [
{ id: ID_TWO, outcome: 'killed-by-test:assertion', message: 'expected 3 to be 4' },
{ id: ID_THREE, outcome: 'survived', message: undefined },
],
outsideTest: [],
};
const record: SidecarRecord = {
schemaVersion: SIDECAR_SCHEMA_VERSION,
kind: 'mutant',
mutantId: '32',
worker: '0',
attempt: 1,
pid: 7,
mutantActivation: 'static',
hitLimit: 100,
requestedTestIds: [ID_THREE, ID_TWO],
hookAttributedTests: null,
result: classification,
};
const normalised = (overrides: Partial<NormalisedMutant>): NormalisedMutant => ({
globalId: `gitnexus/${ROOT}/src/runtime.ts@0:7:10-7:19:ArithmeticOperator:0`,
localId: '32',
file: `gitnexus/${ROOT}/src/runtime.ts`,
location: { start: { line: 7, column: 10 }, end: { line: 7, column: 19 } },
mutatorName: 'ArithmeticOperator',
replacement: 'value / 2',
static: true,
status: 'Killed',
statusReason: null,
testsCompleted: 2,
outcome: 'killed-by-test:assertion',
outcomeSource: 'sidecar',
killedBy: [ID_TWO],
coveredBy: [],
assertionKilledBy: [ID_TWO],
sidecar: {
file: '/out/sidecars/mutant-32.json',
worker: '0',
attempt: 1,
pid: 7,
writtenAtMs: 5,
status: 'killed',
outcome: 'killed-by-test:assertion',
nrOfTests: 2,
reason: null,
failureMessage: 'expected 3 to be 4',
killedBy: [ID_TWO],
hitCount: 1,
mutantActivation: 'static',
outsideTest: [],
},
attempts: [
{
file: '/out/sidecars/mutant-32.json',
worker: '0',
attempt: 1,
pid: 7,
writtenAtMs: 5,
status: 'killed',
outcome: 'killed-by-test:assertion',
nrOfTests: 2,
},
],
perTest: [
{ testId: ID_TWO, outcome: 'killed-by-test:assertion', attempt: 0 },
{ testId: ID_THREE, outcome: 'survived', attempt: 0 },
],
attemptsDisagree: false,
hookAttributedTests: [ID_THREE, ID_TWO],
flags: [],
...overrides,
});
const test = (id: string): NormalisedTest => ({
id,
file: `${ROOT}/runtime.case.ts`,
fullName: id.split('#')[1],
taskId: 'h_0',
skipped: false,
notOracleValidated: false,
killed: [],
assertionKilled: [],
});
const input: ObserveInput = {
profile: PROFILE,
scope: 'main',
results: {
mutants: [normalised({}), normalised({ localId: '33', status: 'Ignored', sidecar: null })],
tests: [test(ID_TWO), test(ID_THREE)],
flags: [{ kind: 'zero-tests', localId: '32', globalId: null, detail: 'per mutant' }],
spanMap: { checked: 2, failures: [] },
},
records: new Map([['/out/sidecars/mutant-32.json', record]]),
dryRunPid: 7,
exportedActiveMutant: null,
fixtureRoot: ROOT,
};
const run = observeScopeRun(input);
it('leaves Ignored mutants out and keeps only run-level flags', () => {
expect([run.mutants.map((mutant) => mutant.localId), run.runFlags]).toEqual([['32'], []]);
});
it('makes paths and test ids fixture-relative', () => {
expect([run.mutants[0].file, run.tests.map((entry) => entry.id)]).toEqual([
'src/runtime.ts',
[TWO, THREE],
]);
});
it('records a selection of every manifest test as "all" and only non-survived test outcomes', () => {
expect(run.mutants[0].facts).toMatchObject({
selected: 'all',
tests: { [TWO]: 'killed-by-test:assertion' },
killedBy: [TWO],
});
expect(run.mutants[0].location).toBe('7:10-7:19');
});
it('reports the hook marker fixture-relative and sorted, and whether attempts disagree', () => {
expect([
run.mutants[0].facts.hookAttributedTests,
run.mutants[0].facts.attemptsDisagree,
]).toEqual([[THREE, TWO], false]);
});
it('keeps a missing hook marker as null', () => {
const unknown = observeScopeRun({
...input,
results: { ...input.results, mutants: [normalised({ hookAttributedTests: null })] },
});
expect(unknown.mutants[0].facts.hookAttributedTests).toBeNull();
});
it('throws when the latest sidecar record was not read', () => {
expect(() => observeScopeRun({ ...input, records: new Map() })).toThrow(/was not read/);
});
it('strips the fixture directory from package- and repo-relative values only', () => {
expect(
[`${ROOT}/a.case.ts#x`, `gitnexus/${ROOT}/src/a.ts`, 'test/unit/a.test.ts#y'].map((value) =>
fixtureRelative(value, ROOT),
),
).toEqual(['a.case.ts#x', 'src/a.ts', 'test/unit/a.test.ts#y']);
});
});
describe('driverScope (run-conformance.ts)', () => {
const scope: ScopeSpec = {
name: 'selection',
testFiles: ['selection.case.ts'],
mutate: ['src/selection.ts'],
testIds: ['selection.case.ts#scales two'],
tests: [],
};
it('passes a scope test selection to the driver with package-relative ids', () => {
expect(driverScope(scope)).toEqual({
testFiles: [`${ROOT}/selection.case.ts`],
mutate: [`gitnexus/${ROOT}/src/selection.ts`],
testIds: [`${ROOT}/selection.case.ts#scales two`],
});
});
it('writes no testIds key for a scope that runs every test', () => {
expect(driverScope({ ...scope, testIds: null })).not.toHaveProperty('testIds');
});
});

99
gitnexus/test/unit/mutation/conformance-fingerprint.test.ts Normal file
View file

@ -0,0 +1,99 @@
/**
* Locks the input fingerprint of gate 2 conformance runs: `--compare-only` must refuse results
* that an earlier run left behind once the runner, driver, prepare step, fixture or the installed
* Stryker and Vitest changed. Pure functions only; the file walk is exercised by
* `npm run test:mutation:conformance`.
*/
import { describe, expect, it } from 'vitest';
import {
STALE_ARTEFACTS,
fingerprintOf,
isFingerprintInput,
staleConformance,
} from '../../../tools/mutation/conformance/fingerprint.js';
const VERSIONS = { stryker: '10.0.0', vitest: '5.0.5' };
const FILES = {
'tools/mutation/runner/plugin.ts': 'a'.repeat(64),
'test/fixtures/mutation-conformance/src/rows.ts': 'b'.repeat(64),
};
const current = fingerprintOf(FILES, VERSIONS);
describe('isFingerprintInput', () => {
it.each([
['the runner', 'tools/mutation/runner/plugin.ts', true],
['the driver', 'tools/mutation/driver/run-scope.ts', true],
['the prepare step', 'tools/mutation/prepare/prepare.ts', true],
['the shared helpers', 'tools/mutation/shared/fs.ts', true],
['the scope file reader', 'tools/mutation/scope-file.ts', true],
['the mutation Vitest config', 'tools/mutation/vitest.mutation.config.ts', true],
['a fixture case file', 'test/fixtures/mutation-conformance/rows.case.ts', true],
['a fixture production file', 'test/fixtures/mutation-conformance/src/rows.ts', true],
['the expected file', 'test/fixtures/mutation-conformance/expected.json', false],
['the comparison itself', 'tools/mutation/conformance/compare.ts', false],
['the oracle', 'tools/mutation/oracle/compare.ts', false],
['another fixture', 'test/fixtures/mutation-oracle/shared.json', false],
['a sibling with a shared prefix', 'tools/mutation/runner-notes/x.ts', false],
])('treats %s as %s', (_label, file, input) => {
expect(isFingerprintInput(file)).toBe(input);
});
});
describe('fingerprintOf', () => {
it('does not depend on the order the files were listed in', () => {
const reversed = Object.fromEntries(Object.entries(FILES).reverse());
expect(fingerprintOf(reversed, VERSIONS).digest).toBe(current.digest);
});
it.each([
['a file content', { ...FILES, 'tools/mutation/runner/plugin.ts': 'c'.repeat(64) }, VERSIONS],
['a new file', { ...FILES, 'tools/mutation/driver/x.ts': 'd'.repeat(64) }, VERSIONS],
['the Stryker version', FILES, { ...VERSIONS, stryker: '10.1.0' }],
['the Vitest version', FILES, { ...VERSIONS, vitest: '5.1.0' }],
])('changes the digest with %s', (_label, files, versions) => {
expect(fingerprintOf(files, versions).digest).not.toBe(current.digest);
});
});
describe('staleConformance', () => {
it('accepts a run recorded with the current inputs', () => {
expect(staleConformance(JSON.parse(JSON.stringify(current)), current)).toBeNull();
});
it.each([
['no fingerprint', undefined, 'the run records no input fingerprint'],
['a malformed fingerprint', { digest: 1 }, 'the run records no input fingerprint'],
[
'a changed file',
fingerprintOf({ ...FILES, 'tools/mutation/runner/plugin.ts': 'e'.repeat(64) }, VERSIONS),
'changed since the run: tools/mutation/runner/plugin.ts',
],
[
'a file the run did not have',
fingerprintOf({ 'tools/mutation/runner/plugin.ts': 'a'.repeat(64) }, VERSIONS),
'changed since the run: test/fixtures/mutation-conformance/src/rows.ts',
],
[
'an older Stryker',
fingerprintOf(FILES, { ...VERSIONS, stryker: '9.6.0' }),
'Stryker 9.6.0 -> 10.0.0',
],
[
'a digest its own files do not produce',
{ ...current, digest: '0'.repeat(64) },
'the recorded input digest does not match',
],
[
'edited files under the current digest',
{
...fingerprintOf({ ...FILES, 'tools/mutation/shared/fs.ts': 'f'.repeat(64) }, VERSIONS),
digest: current.digest,
},
'changed since the run: tools/mutation/shared/fs.ts',
],
])('reports stale artefacts for %s', (_label, recorded, detail) => {
const message = staleConformance(recorded, current);
expect(message).toContain(STALE_ARTEFACTS);
expect(message).toContain(detail);
});
});

523
gitnexus/test/unit/mutation/controls.test.ts Normal file
View file

@ -0,0 +1,523 @@
/**
* Locks gate 4's rules (U8, KTD17, KTD19): a planted control must be killed with an assertion by
* exactly its expected killers and pass under every other test, a control or test missing from
* the run means "re-select", the two determinism profiles must agree on every mutant's outcome
* class and kill sets, and selection is deterministic and covers the required tags (relocated
* `gitnexus-shared` code, an `as const` table). Every input is synthetic.
*/
import { describe, expect, it } from 'vitest';
import {
EXIT_FAILURES,
EXIT_PASS,
EXIT_RESELECT,
EXIT_USAGE,
PLANTED_KIND,
asConstRanges,
checkControls,
checkRequirements,
compareDeterminism,
formatPlanted,
gateExitCode,
mutantTags,
parsePlanted,
runBounded,
selectScopeControls,
type ControlFailure,
type ControlTag,
type DeterminismDifference,
type GateInput,
type PlantedControl,
type PlantedControls,
type PlantedScope,
type RunMutantView,
type RunView,
} from '../../../tools/mutation/controls/run-controls.js';
const FILE = 'test/unit/a.test.ts';
const K = `${FILE}#S > kills`;
const K2 = `${FILE}#S > also kills (#2415)`;
const N = `${FILE}#S > runs and passes`;
const U = `${FILE}#S > unrelated`;
const TESTS = [K, K2, N, U];
const SRC = 'gitnexus/src/a.ts';
const M = `${SRC}@0123456789abcdef:3:1-3:9:StringLiteral:aaaaaaaaaaaaaaaa`;
const M2 = `${SRC}@0123456789abcdef:5:1-5:9:ConditionalExpression:bbbbbbbbbbbbbbbb`;
const mutant = (overrides: Partial<RunMutantView> = {}): RunMutantView => ({
globalId: M,
file: SRC,
location: { start: { line: 3, column: 1 }, end: { line: 3, column: 9 } },
mutatorName: 'StringLiteral',
static: false,
status: 'Killed',
outcome: 'killed-by-test:assertion',
killedBy: [K],
assertionKilledBy: [K],
ran: [
{ id: K, outcome: 'killed-by-test:assertion' },
{ id: N, outcome: 'survived' },
],
attempts: 1,
flags: [],
...overrides,
});
const run = (mutants: RunMutantView[], overrides: Partial<RunView> = {}): RunView => ({
profile: 'A',
tests: TESTS.map((id) => ({ id, skipped: false })),
mutants,
...overrides,
});
const control = (overrides: Partial<PlantedControl> = {}): PlantedControl => ({
id: M,
tags: [],
static: false,
expectedKillers: [K],
expectedNonKillers: { ran: [N], notRun: [U] },
...overrides,
});
const scope = (controls: PlantedControl[] = [control()]): PlantedScope => ({
name: 's',
scope: { testFiles: [FILE], mutate: [SRC] },
mutants: 2,
tests: TESTS.length,
controls,
});
const noTags = (): ControlTag[] => [];
const kinds = (failures: readonly ControlFailure[]): [string, string | null][] =>
failures.map((failure) => [failure.kind, failure.test]);
const differences = (
diffs: readonly DeterminismDifference[],
): [string, string | null, string | null][] =>
diffs.map((diff) => [diff.kind, diff.mutant, diff.test]);
describe('checkControls', () => {
it('passes a control killed by exactly its expected killers whose covering non-killer passed', () => {
expect(checkControls(scope(), run([mutant()]), noTags)).toEqual([]);
});
it.each([
[
'a listed covering non-killer',
mutant({
killedBy: [K, N],
assertionKilledBy: [K, N],
ran: [
{ id: K, outcome: 'killed-by-test:assertion' },
{ id: N, outcome: 'killed-by-test:assertion' },
],
}),
N,
],
[
'an unlisted test, with a runtime error',
mutant({
killedBy: [K, K2],
ran: [
{ id: K, outcome: 'killed-by-test:assertion' },
{ id: K2, outcome: 'killed-by-test:runtime' },
{ id: N, outcome: 'survived' },
],
}),
K2,
],
])('fails a control killed by a test outside its expected killers: %s', (_name, killed, test) => {
expect(kinds(checkControls(scope(), run([killed]), noTags))).toEqual([
['unexpected-killer', test],
]);
});
it.each([
[
'the mutant survived',
mutant({
status: 'Survived',
outcome: 'survived',
killedBy: [],
assertionKilledBy: [],
ran: [
{ id: K, outcome: 'survived' },
{ id: N, outcome: 'survived' },
],
}),
],
[
'the killer failed with a runtime error, not an assertion',
mutant({
outcome: 'killed-by-test:runtime',
assertionKilledBy: [],
ran: [
{ id: K, outcome: 'killed-by-test:runtime' },
{ id: N, outcome: 'survived' },
],
}),
],
])('fails a control that survives its expected killer: %s', (_name, survived) => {
expect(kinds(checkControls(scope(), run([survived]), noTags))).toEqual([['killer-missed', K]]);
});
it('reports a control missing from a run of a changed source file as stale (re-select)', () => {
const changed = `${SRC}@fedcba9876543210:3:1-3:9:StringLiteral:aaaaaaaaaaaaaaaa`;
const failures = checkControls(scope(), run([mutant({ globalId: changed })]), noTags);
expect(kinds(failures)).toEqual([['stale-mutant', null]]);
expect(gateExitCode({ staleRuns: 0, problems: 0, failures, differences: 0 })).toBe(
EXIT_RESELECT,
);
});
it('fails a control missing from a run whose source file is unchanged (not generated)', () => {
const failures = checkControls(scope(), run([mutant({ globalId: M2 })]), noTags);
expect(kinds(failures)).toEqual([['mutant-missing', null]]);
expect(gateExitCode({ staleRuns: 0, problems: 0, failures, differences: 0 })).toBe(
EXIT_FAILURES,
);
});
it('reports a planted test missing from the run as stale and checks nothing else of it', () => {
const renamed = `${FILE}#S > renamed`;
const stale = control({ expectedKillers: [renamed] });
expect(kinds(checkControls(scope([stale]), run([mutant()]), noTags))).toEqual([
['stale-test', renamed],
]);
});
it('fails a listed covering non-killer that did not run with the mutant active', () => {
const notRun = mutant({ ran: [{ id: K, outcome: 'killed-by-test:assertion' }] });
expect(kinds(checkControls(scope(), run([notRun]), noTags))).toEqual([
['non-killer-not-run', N],
]);
});
it('fails a control whose recorded tags differ from the facts of its source', () => {
const tagged = control({ tags: ['as-const', 'relocated-shared'] });
const facts = (): ControlTag[] => ['relocated-shared'];
expect(kinds(checkControls(scope([tagged]), run([mutant()]), facts))).toEqual([['tags', null]]);
});
it('names the scope, profile and control of every failure', () => {
const [failure] = checkControls(scope(), run([], { profile: 'B' }), noTags);
expect(failure).toMatchObject({ scope: 's', profile: 'B', control: M });
});
});
describe('compareDeterminism', () => {
it('passes two runs with identical outcome classes and kill sets', () => {
expect(compareDeterminism(run([mutant()]), run([mutant()], { profile: 'B' }))).toEqual([]);
});
it('names the mutant and the test when the kill sets differ in one test', () => {
const b = run([mutant({ killedBy: [K, K2], assertionKilledBy: [K, K2] })], { profile: 'B' });
expect(differences(compareDeterminism(run([mutant()]), b))).toEqual([
['killed-by', M, K2],
['assertion-killed-by', M, K2],
]);
});
it('fails when the outcome class of a mutant differs between the profiles', () => {
const b = run(
[mutant({ status: 'Timeout', outcome: 'timeout', killedBy: [], assertionKilledBy: [] })],
{ profile: 'B' },
);
expect(differences(compareDeterminism(run([mutant()]), b))).toEqual([
['outcome', M, null],
['status', M, null],
['killed-by', M, K],
['assertion-killed-by', M, K],
]);
});
it('fails when a mutant is in one profile only', () => {
const b = run([mutant()], { profile: 'B' });
expect(differences(compareDeterminism(run([mutant(), mutant({ globalId: M2 })]), b))).toEqual([
['mutant-inventory', M2, null],
]);
});
it('fails when a test is in one profile only', () => {
const b = run([mutant()], {
profile: 'B',
tests: [K, K2, N].map((id) => ({ id, skipped: false })),
});
expect(differences(compareDeterminism(run([mutant()]), b))).toEqual([
['test-inventory', null, U],
]);
});
});
const TABLE_SOURCE = [
"export const NAME = 'Code';",
'export const TABLES = [',
" 'File',",
" 'Folder',",
'] as const;',
'const n = 1 as number;',
'',
].join('\n');
describe('asConstRanges and mutantTags', () => {
it('finds the operand of each `as const` assertion and ignores other `as` assertions', () => {
const ranges = asConstRanges(TABLE_SOURCE);
expect(ranges.map((range) => TABLE_SOURCE.slice(range.start, range.end))).toEqual([
"[\n 'File',\n 'Folder',\n]",
]);
});
it.each([
[
'inside an `as const` table of relocated shared code',
'gitnexus-shared/src/lbug/t.ts',
{ start: { line: 3, column: 3 }, end: { line: 3, column: 9 } },
['relocated-shared', 'as-const'],
],
[
'outside the table, in relocated shared code',
'gitnexus-shared/src/lbug/t.ts',
{ start: { line: 1, column: 21 }, end: { line: 1, column: 27 } },
['relocated-shared'],
],
[
'inside a plain `as` assertion under gitnexus/src',
SRC,
{ start: { line: 6, column: 11 }, end: { line: 6, column: 12 } },
[],
],
])('tags a mutant %s', (_name, file, location, tags) => {
expect(mutantTags({ file, location }, TABLE_SOURCE)).toEqual(tags);
});
});
describe('selectScopeControls', () => {
const MT = `${SRC}@0123456789abcdef:7:1-7:9:ArrayDeclaration:cccccccccccccccc`;
const MS = `${SRC}@0123456789abcdef:9:1-9:9:BooleanLiteral:dddddddddddddddd`;
const MD = `${SRC}@0123456789abcdef:11:1-11:9:EqualityOperator:eeeeeeeeeeeeeeee`;
const MR = `${SRC}@0123456789abcdef:13:1-13:9:BlockStatement:ffffffffffffffff`;
const MW = `${SRC}@0123456789abcdef:15:1-15:9:MethodExpression:1111111111111111`;
const tagsOf = (entry: { file: string; location: unknown }): ControlTag[] =>
entry.location === TAGGED ? ['as-const'] : [];
const TAGGED = { start: { line: 7, column: 1 }, end: { line: 7, column: 9 } };
const mutants = (variant: 'A' | 'B'): RunMutantView[] => [
mutant(),
mutant({
globalId: MT,
location: TAGGED,
mutatorName: 'ArrayDeclaration',
static: true,
killedBy: [K, K2],
assertionKilledBy: [K, K2],
ran: [
{ id: K, outcome: 'killed-by-test:assertion' },
{ id: K2, outcome: 'killed-by-test:assertion' },
{ id: N, outcome: 'survived' },
{ id: U, outcome: 'survived' },
],
}),
mutant({
globalId: MS,
status: 'Survived',
outcome: 'survived',
killedBy: [],
assertionKilledBy: [],
}),
mutant({
globalId: MD,
mutatorName: 'EqualityOperator',
killedBy: [variant === 'A' ? K : K2],
assertionKilledBy: [variant === 'A' ? K : K2],
}),
mutant({
globalId: MR,
mutatorName: 'BlockStatement',
outcome: 'killed-by-test:runtime',
assertionKilledBy: [],
}),
mutant({
globalId: MW,
mutatorName: 'MethodExpression',
killedBy: [K2],
assertionKilledBy: [K2],
ran: [{ id: K2, outcome: 'killed-by-test:assertion' }],
}),
];
const runs = [run(mutants('A')), run(mutants('B'), { profile: 'B' })];
const options = { seed: 'seed-1', perScope: 3, maxKillers: 3, maxNonKillers: 3 };
it('picks the required tag first, then stable assertion kills with small killer sets', () => {
const selection = selectScopeControls(runs, options, tagsOf);
expect(selection.candidates).toBe(3);
// Sorted by global id: line 15 sorts before lines 3 and 7.
expect(selection.controls).toEqual([
{
id: MW,
tags: [],
static: false,
expectedKillers: [K2],
expectedNonKillers: { ran: [], notRun: [K, N, U] },
},
{
id: M,
tags: [],
static: false,
expectedKillers: [K],
expectedNonKillers: { ran: [N], notRun: [K2, U] },
},
{
id: MT,
tags: ['as-const'],
static: true,
expectedKillers: [K2, K],
expectedNonKillers: { ran: [N, U], notRun: [] },
},
]);
});
it('selects the same controls for the same seed whatever the order of the mutants', () => {
const reversed = runs.map((entry) => ({ ...entry, mutants: [...entry.mutants].reverse() }));
expect(selectScopeControls(reversed, options, tagsOf)).toEqual(
selectScopeControls(runs, options, tagsOf),
);
});
it('caps the listed non-killers at maxNonKillers', () => {
const selection = selectScopeControls(runs, { ...options, maxNonKillers: 1 }, tagsOf);
expect(selection.controls.map((entry) => entry.expectedNonKillers.notRun.length)).toEqual([
1, 1, 0,
]);
});
});
const planted = (scopes: PlantedScope[]): PlantedControls => ({
schemaVersion: 1,
kind: PLANTED_KIND,
description: 'synthetic',
reselect: 'npm run test:mutation:controls -- --select --work <dir>',
profiles: [
{ name: 'A', concurrency: 1, maxTestRunnerReuse: 0 },
{ name: 'B', concurrency: 3, maxTestRunnerReuse: 1 },
],
timeoutMS: 10000,
selection: { seed: 's', perScope: 3, maxKillers: 3, maxNonKillers: 3, selectedFrom: ['A', 'B'] },
scopes,
});
describe('planted file', () => {
it('parses what it formats', () => {
const file = planted([scope([control({ tags: ['relocated-shared', 'as-const'] })])]);
expect(parsePlanted(formatPlanted(file))).toEqual(file);
});
it.each([
['another kind', { kind: 'gitnexus-mutation-oracle-manifest' }, /kind/],
['another schema version', { schemaVersion: 2 }, /schemaVersion/],
['an unknown tag', { scopes: [scope([control({ tags: [] })])].map(withTag('nope')) }, /tag/],
])('rejects a file with %s', (_name, overrides, message) => {
const text = JSON.stringify({ ...planted([scope()]), ...overrides });
expect(() => parsePlanted(text)).toThrow(message);
});
it.each([
['no tagged control', [control()], ['as-const', 'relocated-shared']],
['a relocated shared control only', [control({ tags: ['relocated-shared'] })], ['as-const']],
['both tags on one control', [control({ tags: ['relocated-shared', 'as-const'] })], []],
])('lists the required tags no control carries: %s', (_name, controls, missing) => {
expect(checkRequirements(planted([scope(controls)]))).toEqual(missing);
});
});
function withTag(tag: string): (entry: PlantedScope) => unknown {
return (entry) => ({
...entry,
controls: entry.controls.map((item) => ({ ...item, tags: [tag] })),
});
}
describe('gateExitCode', () => {
const failure = (kind: ControlFailure['kind']): ControlFailure => ({
scope: 's',
profile: 'A',
control: M,
kind,
test: null,
detail: '',
});
const clean: GateInput = { staleRuns: 0, problems: 0, failures: [], differences: 0 };
it.each([
['a clean gate', clean, EXIT_PASS],
['a stale run, before everything else', { ...clean, staleRuns: 1, differences: 1 }, EXIT_USAGE],
['a determinism difference', { ...clean, differences: 1 }, EXIT_FAILURES],
['a run problem', { ...clean, problems: 1 }, EXIT_FAILURES],
[
'a failed control, before a stale one',
{ ...clean, failures: [failure('stale-mutant'), failure('killer-missed')] },
EXIT_FAILURES,
],
['only stale controls', { ...clean, failures: [failure('stale-test')] }, EXIT_RESELECT],
])('%s', (_name, input, expected) => {
expect(gateExitCode(input)).toBe(expected);
});
});
describe('runBounded', () => {
interface Item {
name: string;
cost: number;
}
/** Each item takes one macrotask; returns who was running when each item started. */
const schedule = async (
items: Item[],
budget: number,
): Promise<{ starts: [string, string[]][]; peak: number }> => {
const active = new Map<string, number>();
const starts: [string, string[]][] = [];
let peak = 0;
await runBounded(items, budget, async (item) => {
starts.push([item.name, [...active.keys()].sort()]);
active.set(item.name, item.cost);
peak = Math.max(
peak,
[...active.values()].reduce((sum, cost) => sum + cost, 0),
);
await new Promise<void>((resolve) => setImmediate(resolve));
active.delete(item.name);
});
return { starts, peak };
};
it('starts items while the running costs fit the budget', async () => {
const items = [
{ name: 'a1', cost: 1 },
{ name: 'a2', cost: 1 },
{ name: 'a3', cost: 1 },
{ name: 'b1', cost: 3 },
{ name: 'b2', cost: 3 },
];
expect(await schedule(items, 3)).toEqual({
starts: [
['a1', []],
['a2', ['a1']],
['a3', ['a1', 'a2']],
['b1', []],
['b2', []],
],
peak: 3,
});
});
it('runs an item costlier than the budget alone, after the items that fit', async () => {
const items = [
{ name: 'x', cost: 1 },
{ name: 'big', cost: 5 },
{ name: 'y', cost: 1 },
];
expect((await schedule(items, 3)).starts).toEqual([
['x', []],
['y', ['x']],
['big', []],
]);
});
});

192
gitnexus/test/unit/mutation/coverage-attribution.test.ts Normal file
View file

@ -0,0 +1,192 @@
/**
* Locks how the vitest5 runner's setup file buckets instrumenter hits (KTD4): load- and
* collection-time hits stay static, hits inside a test are per-test coverage, and a hit with no
* current test after collection is a hook hit, attributed to every test of the suite whose hook
* ran. Per-test coverage keys are then translated from Vitest task ids to Stryker ids (KTD3). The
* hook bucket is also kept apart, as the per-mutant hook marker (Phase 1 KTD2).
*/
import { describe, expect, it } from 'vitest';
import {
CoverageRecorder,
attributeHookHits,
bucketForHit,
hookAttributedTests,
hookCoverage,
mergeCoverage,
rekeyCoverage,
type HitContext,
} from '../../../tools/mutation/runner/coverage-attribution.js';
/** Mirrors the instrumenter's `stryCov_9fa48` with no `currentTestId` set: it always counts into `static`. */
const instrumenterHit = (recorder: CoverageRecorder, ...mutantIds: string[]): void => {
const coverage = recorder.instrumenterView;
for (const id of mutantIds) coverage.static[id] = (coverage.static[id] ?? 0) + 1;
};
describe('bucketForHit', () => {
it.each([
['a collection-time hit', { phase: 'collection' }, { kind: 'static' }],
[
'a hit inside a test',
{ phase: 'run', testId: 'f_0_1', runningSuiteIds: ['f', 'f_0'] },
{ kind: 'test', testId: 'f_0_1' },
],
[
'a hit with no current test after collection',
{ phase: 'run', testId: undefined, runningSuiteIds: ['f', 'f_0'] },
{ kind: 'hook', suiteId: 'f_0' },
],
] satisfies [string, HitContext, ReturnType<typeof bucketForHit>][])(
'buckets %s',
(_label, context, bucket) => {
expect(bucketForHit(context)).toEqual(bucket);
},
);
it('refuses a run-phase hit with no running suite', () => {
expect(() => bucketForHit({ phase: 'run', testId: undefined, runningSuiteIds: [] })).toThrow(
/no running suite/,
);
});
});
describe('CoverageRecorder with attributeHookHits', () => {
const testsBySuite = {
f: ['f_0_0', 'f_0_1', 'f_1_0'],
f_0: ['f_0_0', 'f_0_1'],
f_1: ['f_1_0'],
};
it('attributes a hook hit after collection to every test of the suite whose hook ran, and keeps collection hits static', () => {
let context: HitContext = { phase: 'collection' };
const recorder = new CoverageRecorder(() => context);
instrumenterHit(recorder, '1', '1');
context = { phase: 'run', testId: undefined, runningSuiteIds: ['f', 'f_0'] };
instrumenterHit(recorder, '2');
context = { phase: 'run', testId: 'f_1_0', runningSuiteIds: ['f', 'f_1'] };
instrumenterHit(recorder, '3');
expect(attributeHookHits(recorder.snapshot(), testsBySuite)).toEqual({
static: { '1': 2 },
perTest: { f_0_0: { '2': 1 }, f_0_1: { '2': 1 }, f_1_0: { '3': 1 } },
});
});
it('attributes a file-level hook hit to every test of the file', () => {
const recorder = new CoverageRecorder(() => ({
phase: 'run',
testId: undefined,
runningSuiteIds: ['f'],
}));
instrumenterHit(recorder, '7');
expect(attributeHookHits(recorder.snapshot(), testsBySuite).perTest).toEqual({
f_0_0: { '7': 1 },
f_0_1: { '7': 1 },
f_1_0: { '7': 1 },
});
});
it('adds hook hits to the per-test hits of the same test', () => {
let context: HitContext = { phase: 'run', testId: 'f_1_0', runningSuiteIds: ['f', 'f_1'] };
const recorder = new CoverageRecorder(() => context);
instrumenterHit(recorder, '4');
context = { phase: 'run', testId: undefined, runningSuiteIds: ['f', 'f_1'] };
instrumenterHit(recorder, '4');
expect(attributeHookHits(recorder.snapshot(), testsBySuite).perTest).toEqual({
f_1_0: { '4': 2 },
});
});
it('refuses a hook hit for a suite it does not know', () => {
const recorder = new CoverageRecorder(() => ({
phase: 'run',
testId: undefined,
runningSuiteIds: ['unknown'],
}));
instrumenterHit(recorder, '1');
expect(() => attributeHookHits(recorder.snapshot(), testsBySuite)).toThrow(/unknown/);
});
});
describe('hookCoverage and hookAttributedTests (the hook marker, KTD2)', () => {
const testsBySuite = {
f: ['f_0_0', 'f_0_1', 'f_1_0'],
f_0: ['f_0_0', 'f_0_1'],
f_1: ['f_1_0'],
};
/** Mutant 2 is hit only in suite f_0's beforeAll, mutant 3 only inside test f_1_0. */
const recorded = (): CoverageRecorder => {
let context: HitContext = { phase: 'collection' };
const recorder = new CoverageRecorder(() => context);
instrumenterHit(recorder, '1');
context = { phase: 'run', testId: undefined, runningSuiteIds: ['f', 'f_0'] };
instrumenterHit(recorder, '2');
context = { phase: 'run', testId: 'f_1_0', runningSuiteIds: ['f', 'f_1'] };
instrumenterHit(recorder, '3');
context = { phase: 'run', testId: undefined, runningSuiteIds: ['f', 'f_1'] };
instrumenterHit(recorder, '3');
return recorder;
};
it('keeps the hook bucket apart: only hook hits, attributed to every test of the suite', () => {
expect(hookCoverage(recorded().snapshot(), testsBySuite)).toEqual({
f_0_0: { '2': 1 },
f_0_1: { '2': 1 },
f_1_0: { '3': 1 },
});
});
it('lists, per mutant, every test that a suite hook attributed it to', () => {
expect(hookAttributedTests(hookCoverage(recorded().snapshot(), testsBySuite))).toEqual({
'2': ['f_0_0', 'f_0_1'],
'3': ['f_1_0'],
});
});
it('leaves the attributed per-test coverage unchanged', () => {
expect(attributeHookHits(recorded().snapshot(), testsBySuite)).toEqual({
static: { '1': 1 },
perTest: { f_0_0: { '2': 1 }, f_0_1: { '2': 1 }, f_1_0: { '3': 2 } },
});
});
it('refuses a hook-bucket hit for a suite it does not know', () => {
const recorder = new CoverageRecorder(() => ({
phase: 'run',
testId: undefined,
runningSuiteIds: ['unknown'],
}));
instrumenterHit(recorder, '1');
expect(() => hookCoverage(recorder.snapshot(), testsBySuite)).toThrow(/unknown/);
});
});
describe('rekeyCoverage and mergeCoverage', () => {
it('translates per-test keys from task ids to Stryker ids and drops tests outside the map', () => {
expect(
rekeyCoverage(
{ static: { '1': 1 }, perTest: { f_0: { '2': 1 }, f_skipped: { '3': 1 } } },
new Map([['f_0', 'test/a.test.ts#A > b']]),
),
).toEqual({ static: { '1': 1 }, perTest: { 'test/a.test.ts#A > b': { '2': 1 } } });
});
it('sums the coverage of several test files', () => {
const into = { static: { '1': 1 }, perTest: { a: { '2': 1 } } };
mergeCoverage(into, { static: { '1': 2, '5': 1 }, perTest: { a: { '2': 3 }, b: { '6': 1 } } });
expect(into).toEqual({
static: { '1': 3, '5': 1 },
perTest: { a: { '2': 4 }, b: { '6': 1 } },
});
});
});

412
gitnexus/test/unit/mutation/fidelity-compare.test.ts Normal file
View file

@ -0,0 +1,412 @@
/**
* Locks the comparison logic of gate 1 (U6): test identities and outcomes, the effective config
* against the allowed-difference list, the KTD19 path rule and module comparison, the runtime
* load check, and the envelope record. Pure functions on synthetic inputs; the real runs are
* exercised by `npm run test:mutation:fidelity`.
*/
import { describe, expect, it } from 'vitest';
import {
ALLOWED_CONFIG_DIFFS,
compareConfigs,
normalisePaths,
type ConfigView,
} from '../../../tools/mutation/fidelity/allowed-config-diffs.js';
import {
buildEnvelope,
checkPathRule,
checkRuntimeLoads,
compareModuleSets,
compareTests,
locateModule,
runPathRoots,
basePathRoots,
type EnvelopeInput,
type ObservedTest,
type Roots,
} from '../../../tools/mutation/fidelity/fidelity.js';
import type {
JsonValue,
LoadEvent,
WorkerRecord,
} from '../../../tools/mutation/fidelity/record.js';
const REPO = '/repo';
const PREPARED = '/out/stryker/prepared';
const SANDBOX = `${PREPARED}/gitnexus/.stryker-tmp/sandbox-AbC123`;
const ROOTS: Roots = { repoRoot: REPO, preparedRoot: PREPARED, sandbox: SANDBOX };
const FILE = 'test/unit/schema.test.ts';
const test = (fullName: string, line: number, outcome: ObservedTest['outcome']): ObservedTest => ({
file: FILE,
fullName,
location: { line, column: 5 },
outcome,
});
const BASE_TESTS: ObservedTest[] = [
test('schema > has a table', 10, 'passed'),
test('schema > has an index', 14, 'passed'),
test('schema > is skipped on win32', 20, 'skipped'),
];
describe('compareTests', () => {
it('passes identical identity lists', () => {
expect(compareTests(BASE_TESTS, BASE_TESTS)).toEqual([]);
});
it('fails a line shifted by one and names the shifted test', () => {
const run = [
BASE_TESTS[0],
{ ...BASE_TESTS[1], location: { line: 15, column: 5 } },
BASE_TESTS[2],
];
const differences = compareTests(BASE_TESTS, run);
expect(differences).toHaveLength(1);
expect(differences[0]).toMatchObject({
kind: 'location',
subject: `${FILE} > schema > has an index`,
});
expect(differences[0].detail).toContain('14:5');
expect(differences[0].detail).toContain('15:5');
});
it('fails a test that passes in one run and is skipped in the other', () => {
const run = [BASE_TESTS[0], { ...BASE_TESTS[1], outcome: 'skipped' as const }, BASE_TESTS[2]];
expect(compareTests(BASE_TESTS, run)).toEqual([
{
kind: 'outcome',
subject: `${FILE} > schema > has an index`,
detail: 'base passed, run skipped',
},
]);
});
it('fails a test that only one run collected, and a file only one run has', () => {
const other: ObservedTest = { ...BASE_TESTS[0], file: 'test/unit/other.test.ts' };
const kinds = compareTests(BASE_TESTS, [BASE_TESTS[0], BASE_TESTS[2], other]).map(
(difference) => [difference.kind, difference.subject],
);
expect(kinds).toEqual([
['test-missing', `${FILE} > schema > has an index`],
['file-extra', 'test/unit/other.test.ts'],
]);
});
it('pairs tests that share a full name by location order', () => {
const rows = [test('each row', 30, 'passed'), test('each row', 30, 'failed')];
const swapped = [rows[1], rows[0]];
expect(compareTests(rows, swapped).map((difference) => difference.kind)).toEqual([
'outcome',
'outcome',
]);
});
});
const view = (project: { [key: string]: JsonValue }, root: string): ConfigView => ({
project: { root, testTimeout: 30000, sequence: { shuffle: false, seed: 1 }, ...project },
global: { root },
vite: {},
env: { VITEST: 'true' },
});
describe('compareConfigs', () => {
const base = view({ dangerouslyIgnoreUnhandledErrors: true }, '<package>');
it('passes an allowed difference and lists it', () => {
const result = compareConfigs(
base,
view({ dangerouslyIgnoreUnhandledErrors: false }, '<package>'),
);
expect(result.failures).toEqual([]);
expect(result.allowed).toEqual([
{
name: 'dangerouslyIgnoreUnhandledErrors',
paths: ['project.dangerouslyIgnoreUnhandledErrors'],
},
]);
});
it('fails a difference outside the allowed list', () => {
const run = view({ dangerouslyIgnoreUnhandledErrors: true, testTimeout: 5000 }, '<package>');
expect(compareConfigs(base, run).failures).toEqual([
{ path: 'project.testTimeout', base: 30000, run: 5000 },
]);
});
it.each([
['ignored while shuffling is off', false, []],
['a failure when shuffling is on', true, ['project.sequence.seed']],
])('treats a different sequence.seed as %s', (_label, shuffle, failures) => {
const sequence = { shuffle, seed: 2 };
const left = view(
{ dangerouslyIgnoreUnhandledErrors: true, sequence: { shuffle, seed: 1 } },
'<package>',
);
const right = view({ dangerouslyIgnoreUnhandledErrors: true, sequence }, '<package>');
expect(compareConfigs(left, right).failures.map((failure) => failure.path)).toEqual(failures);
});
it.each([
['the sandbox maps to the checkout', SANDBOX, []],
['a root in the prepared copy still differs', `${PREPARED}/gitnexus`, ['project.root']],
['a root in the checkout still differs', `${REPO}/gitnexus`, ['project.root']],
])('compares root after mapping paths: %s', (_label, runRoot, failures) => {
const left = normalisePaths(`${REPO}/gitnexus`, basePathRoots(REPO));
const right = normalisePaths(runRoot, runPathRoots(ROOTS));
const result = compareConfigs(
{ ...base, project: { root: left, sequence: null } },
{ ...base, project: { root: right, sequence: null } },
);
expect(result.failures.map((failure) => failure.path)).toEqual(failures);
});
it('documents every allowed entry with a reason', () => {
expect(ALLOWED_CONFIG_DIFFS.filter((entry) => entry.reason.length < 40)).toEqual([]);
});
});
const load = (
source: LoadEvent['source'],
module: string,
parent: string | null,
phase: LoadEvent['phase'] = 'run',
): LoadEvent => ({
source,
module,
parent,
phase,
test: phase === 'run' ? 'schema > has a table' : null,
});
const record = (file: string, root: string, loads: LoadEvent[]): WorkerRecord => ({
schemaVersion: 1,
pid: 1,
ppid: 0,
file,
root,
project: 'default',
loads,
envReadsAtLoad: [],
env: {},
});
const runRecord = (loads: LoadEvent[]): WorkerRecord =>
record(`${SANDBOX}/${FILE}`, SANDBOX, [
load('module-runner', `${SANDBOX}/${FILE}`, null),
...loads,
]);
const baseRecord = (loads: LoadEvent[]): WorkerRecord =>
record(`${REPO}/gitnexus/${FILE}`, `${REPO}/gitnexus`, [
load('module-runner', `${REPO}/gitnexus/${FILE}`, null),
...loads,
]);
describe('path rule (KTD19)', () => {
it.each([
['the sandbox', `${SANDBOX}/src/core/lbug/schema.ts`, []],
[
'the sandbox relocated shared sources',
`${SANDBOX}/relocated/gitnexus-shared/src/index.ts`,
[],
],
[
'the prepared copy outside the sandbox',
`${PREPARED}/gitnexus/src/core/lbug/schema.ts`,
['prepared copy'],
],
['the original checkout', `${REPO}/gitnexus/src/core/lbug/schema.ts`, ['checkout']],
['the checkout shared build', `${REPO}/gitnexus-shared/dist/index.js`, ['checkout']],
[
'the checkout, a tool module a test imports',
`${REPO}/gitnexus/tools/mutation/runner/test-ids.ts`,
['checkout'],
],
[
'the prepared copy, the runner setup file next to the plugin (harness)',
`${PREPARED}/gitnexus/tools/mutation/runner/setup.ts`,
[],
],
])('a Stryker-run local module from %s', (_label, module, origins) => {
const differences = checkPathRule(
runRecord([load('module-runner', module, `${SANDBOX}/${FILE}`)]),
ROOTS,
);
expect(differences.map((difference) => difference.kind)).toEqual(
origins.map(() => 'path-rule'),
);
expect(differences.map((difference) => difference.detail.includes(origins[0] ?? '-'))).toEqual(
origins.map(() => true),
);
});
it('maps the base run shared build to the relocated sources', () => {
expect(
locateModule(`${REPO}/gitnexus-shared/dist/lbug/schema-constants.js`, 'base', ROOTS),
).toEqual({
key: 'gitnexus-shared/src/lbug/schema-constants.ts',
kind: 'production',
origin: 'checkout',
});
});
it('finds the same production modules when the base loads dist and the run the relocated src', () => {
const base = baseRecord([
load('module-runner', `${REPO}/gitnexus/src/core/lbug/schema.ts`, `${REPO}/gitnexus/${FILE}`),
load(
'module-runner',
`${REPO}/gitnexus-shared/dist/index.js`,
`${REPO}/gitnexus/src/core/lbug/schema.ts`,
),
]);
const run = runRecord([
load('module-runner', `${SANDBOX}/src/core/lbug/schema.ts`, `${SANDBOX}/${FILE}`),
load(
'module-runner',
`${SANDBOX}/relocated/gitnexus-shared/src/index.ts`,
`${SANDBOX}/src/core/lbug/schema.ts`,
),
]);
expect(compareModuleSets(base, run, ROOTS)).toEqual([]);
});
it('leaves out what the harness imports, such as the runner setup file importing vitest', () => {
const vitestEntry = `${REPO}/gitnexus/node_modules/vitest/dist/index.js`;
const run = runRecord([
load('module-runner', `${PREPARED}/gitnexus/tools/mutation/runner/setup.ts`, null),
load('module-runner', vitestEntry, `${PREPARED}/gitnexus/tools/mutation/runner/setup.ts`),
]);
expect(compareModuleSets(baseRecord([]), run, ROOTS)).toEqual([]);
});
it('reports a production module only one run loads', () => {
const base = baseRecord([
load('module-runner', `${REPO}/gitnexus/src/core/lbug/schema.ts`, `${REPO}/gitnexus/${FILE}`),
]);
const differences = compareModuleSets(base, runRecord([]), ROOTS);
expect(differences.map((difference) => [difference.kind, difference.subject])).toEqual([
['module-set', `gitnexus/${FILE}`],
]);
expect(differences[0].detail).toContain('gitnexus/src/core/lbug/schema.ts');
});
});
describe('runtime load check (KTD19)', () => {
const testModule = `${REPO}/gitnexus/${FILE}`;
const pkg = `file://${REPO}/gitnexus/node_modules/some-pkg/index.js`;
const harness = `file://${REPO}/gitnexus/node_modules/vitest/dist/module-evaluator.js`;
it.each([
[
'the test imports node:child_process',
load('module-runner', 'child_process', testModule),
['child_process'],
],
[
'a package requires child_process',
load('node-resolve', 'node:child_process', pkg),
['child_process'],
],
[
'a package loads cluster at load time',
load('node-resolve', 'node:cluster', pkg, 'load'),
['cluster'],
],
[
'a native addon is opened',
load('dlopen', `${REPO}/gitnexus/node_modules/x/x.node`, null),
['native binding'],
],
[
'a .node file is resolved',
load('node-resolve', `file://${REPO}/a/b.node`, pkg),
['native binding'],
],
['a Worker is constructed', load('worker', 'file:///w.js', testModule), ['worker']],
['the harness resolves child_process', load('node-resolve', 'node:child_process', harness), []],
[
'a package loads worker_threads without a Worker',
load('node-resolve', 'node:worker_threads', pkg),
[],
],
])('%s', (_label, event, kinds) => {
const result = checkRuntimeLoads(baseRecord([event]), 'base', ROOTS);
expect(result.failures.map((failure) => failure.kind)).toEqual(kinds.map(() => 'runtime-load'));
expect(result.failures.map((failure) => failure.detail.split(':')[0])).toEqual(kinds);
});
it('names the test that was running', () => {
const result = checkRuntimeLoads(
baseRecord([load('module-runner', 'node:child_process', testModule)]),
'base',
ROOTS,
);
expect(result.failures[0].subject).toBe(`gitnexus/${FILE} > schema > has a table`);
});
it('notes worker_threads loads without failing', () => {
const result = checkRuntimeLoads(
baseRecord([load('node-resolve', 'node:worker_threads', pkg, 'load')]),
'base',
ROOTS,
);
expect(result.workerThreadsLoaded).toBe(true);
});
});
const ENVELOPE_INPUT: EnvelopeInput = {
versions: { vitest: '5.0.5', stryker: '10.0.0', node: 'v22.23.3' },
platform: { os: 'linux', release: '6.6.0', arch: 'x64' },
stryker: { concurrency: 2, maxTestRunnerReuse: 0 },
config: {
base: { file: 'gitnexus/vitest.config.ts', project: 'default' },
run: { file: 'gitnexus/tools/mutation/vitest.mutation.config.ts', project: 'mutation' },
settings: { pool: 'forks' },
comparison: {
failures: [],
allowed: [{ name: 'setupFiles', paths: ['project.setupFiles'] }],
volatile: [],
},
},
envReadsAtLoad: [],
files: [FILE],
tests: BASE_TESTS,
differences: [],
runtimeNotes: { workerThreadsLoadedBy: [] },
};
describe('buildEnvelope', () => {
it('records the Vitest and Stryker versions of the run', () => {
expect(buildEnvelope(ENVELOPE_INPUT).versions).toEqual({
vitest: '5.0.5',
stryker: '10.0.0',
node: 'v22.23.3',
});
});
it('lists the allowed config differences that occurred, with their reasons', () => {
expect(buildEnvelope(ENVELOPE_INPUT).config.allowedDifferences).toEqual([
{
name: 'setupFiles',
paths: ['project.setupFiles'],
reason: ALLOWED_CONFIG_DIFFS[0].reason,
},
]);
});
it('counts tests and lists the skips of this platform', () => {
const envelope = buildEnvelope(ENVELOPE_INPUT);
expect(envelope.accepted).toMatchObject({ fileCount: 1, testCount: 3, ran: 2, skipped: 1 });
expect(envelope.skippedOnThisPlatform).toEqual([`${FILE} > schema > is skipped on win32`]);
});
it.each([
['pass', [], 'pass'],
[
'fail',
[{ kind: 'outcome' as const, subject: FILE, detail: 'base passed, run failed' }],
'fail',
],
])('reports the gate result: %s', (_label, differences, result) => {
expect(buildEnvelope({ ...ENVELOPE_INPUT, differences }).gate.result).toBe(result);
});
});

586
gitnexus/test/unit/mutation/launch-single.test.ts Normal file
View file

@ -0,0 +1,586 @@
/**
* Locks the oracle's single-test launch rules (U12, KTD15, KTD17): which launches are valid
* observations, how a launch's outcome is classified, how the test is selected (`<file>:<line>`
* plus an escaped, anchored `-t`), the launch environment, the wall-clock default, and the
* benchmark's cost projection. Everything here runs on synthetic launch results; no process is
* spawned.
*/
import { describe, expect, it } from 'vitest';
import {
BASE_HOOK_TIMEOUT_MS,
BASE_TEST_TIMEOUT_MS,
DEFAULT_WALL_CLOCK_MS,
buildLaunchEnv,
classifyLaunch,
escapeRegExp,
locationFilter,
testNamePattern,
type LaunchEvidence,
type LaunchTarget,
} from '../../../tools/mutation/oracle/launch-single.js';
import {
ORACLE_RECORD_SCHEMA_VERSION,
toOracleError,
type OracleError,
type OracleModuleRecord,
type OracleRunRecord,
type OracleTestRecord,
} from '../../../tools/mutation/oracle/oracle-classify.js';
import {
PROBE_SCHEMA_VERSION,
checkActivation,
parseOracleProbe,
type OracleProbe,
type ProbeHits,
} from '../../../tools/mutation/oracle/probe.js';
import {
evenlySpaced,
planLaunches,
projectOracleCost,
summarizeTimings,
type CostProjectionInput,
} from '../../../tools/mutation/bench/oracle-cost.js';
import type { CommandRunMutant } from '../../../tools/mutation/oracle/launch-single.js';
const FILE = 'test/unit/a.test.ts';
const TARGET: LaunchTarget = {
id: `${FILE}#S > one`,
file: FILE,
fullName: 'S > one',
location: { line: 10, column: 3 },
};
const OTHER_ID = `${FILE}#S > two`;
const assertionError = toOracleError({
message: 'expected 1 to be 2',
stack: 'AssertionError: expected 1 to be 2\n at test/unit/a.test.ts:11:15',
});
const testTimeoutError: OracleError = {
name: 'Error',
message:
'Test timed out in 30000ms.\nIf this is a long-running test, pass a timeout value as the ' +
'last argument or configure it globally with "testTimeout".',
};
const testRecord = (
fullName: string,
line: number,
overrides: Partial<OracleTestRecord> = {},
): OracleTestRecord => ({
fullName,
location: { line, column: 3 },
state: 'skipped',
errors: [],
failedHooks: [],
retry: 0,
retryCount: 0,
...overrides,
});
/** `S > one` (the target, line 10) and `S > two` (line 20), skipped unless overridden. */
const twoTests = (
one: Partial<OracleTestRecord> = {},
two: Partial<OracleTestRecord> = {},
): OracleTestRecord[] => [testRecord('S > one', 10, one), testRecord('S > two', 20, two)];
const moduleRecord = (
tests: OracleTestRecord[],
overrides: Partial<OracleModuleRecord> = {},
): OracleModuleRecord => ({
file: FILE,
state: tests.some((test) => test.state === 'failed') ? 'failed' : 'passed',
errors: [],
failedHooks: [],
suites: [{ fullName: 'S', errors: [], failedHooks: [], tests: tests.map((_, index) => index) }],
tests,
...overrides,
});
/** A launch record whose Vitest JSON totals and exit code agree with its tests. */
const runRecord = (module: OracleModuleRecord, mutantId: string | null = '7'): OracleRunRecord => {
const { tests } = module;
const failed =
tests.some((test) => test.state === 'failed') ||
module.errors.length > 0 ||
module.suites.some((suite) => suite.errors.length > 0);
return {
schemaVersion: ORACLE_RECORD_SCHEMA_VERSION,
kind: mutantId === null ? 'dry-run' : 'mutant',
mutantId,
attempt: 1,
pid: 4242,
root: '/sandbox',
configFile: '/sandbox/tools/mutation/vitest.mutation.config.ts',
vitestVersion: '5.0.5',
exitCode: failed ? 1 : 0,
wallClockMs: 1500,
wrapperError: null,
run: { reason: failed ? 'failed' : 'passed', unhandledErrors: [], modules: [module] },
vitestJson: {
numTotalTests: tests.length,
numPassedTests: tests.filter((test) => test.state === 'passed').length,
numFailedTests: tests.filter((test) => test.state === 'failed').length,
},
vitestJsonFile: '/out/mutant-7.attempt-1.vitest.json',
};
};
const NO_HITS: ProbeHits = { loadAndCollection: 0, hook: 0, perTest: [] };
const probe = (
hits: ProbeHits | null,
activation: { intended: string | null; installed: string | null } = {
intended: '7',
installed: '7',
},
): OracleProbe => ({
schemaVersion: PROBE_SCHEMA_VERSION,
pid: 4243,
activation: checkActivation({
...activation,
namespaceExists: false,
namespaceActiveMutant: null,
}),
hits,
complete: hits !== null,
});
const targetHits = (count: number): ProbeHits => ({
loadAndCollection: 0,
hook: 0,
perTest: [{ taskId: 'abc_0_0', fullName: 'S > one', location: { line: 10, column: 3 }, count }],
});
const evidence = (overrides: Partial<LaunchEvidence>): LaunchEvidence => ({
target: TARGET,
intendedMutantId: '7',
wallClockLimitMs: DEFAULT_WALL_CLOCK_MS,
killedByWallClock: false,
exitCode: 0,
signal: null,
record: null,
probe: null,
...overrides,
});
describe('classifyLaunch: observation validity', () => {
it('a launch that executes zero tests is invalid', () => {
const observation = classifyLaunch(
evidence({ record: runRecord(moduleRecord(twoTests())), probe: probe(NO_HITS) }),
);
expect(observation).toMatchObject({ validity: 'invalid', outcome: 'invalid', executed: [] });
expect(observation.reason).toMatch(/zero tests executed/);
});
it('a launch that executes two tests is invalid', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'passed' }, { state: 'passed' })));
const observation = classifyLaunch(evidence({ record, probe: probe(NO_HITS), exitCode: 0 }));
expect(observation).toMatchObject({
validity: 'invalid',
outcome: 'invalid',
executed: [TARGET.id, OTHER_ID],
});
expect(observation.reason).toMatch(/2 tests executed/);
});
it('a launch that executes one test other than the intended one is invalid', () => {
const record = runRecord(moduleRecord(twoTests({}, { state: 'passed' })));
const observation = classifyLaunch(evidence({ record, probe: probe(NO_HITS) }));
expect(observation).toMatchObject({ validity: 'invalid', executed: [OTHER_ID] });
expect(observation.reason).toMatch(/is not the intended test/);
});
it('a location filter that matched no test is invalid, not a collection kill', () => {
const module = moduleRecord(twoTests(), {
state: 'failed',
errors: [{ name: 'Error', message: `No test found in ${FILE} in line 11` }],
});
const observation = classifyLaunch(
evidence({ record: runRecord(module), probe: probe(null), exitCode: 1 }),
);
expect(observation).toMatchObject({ validity: 'invalid', outcome: 'invalid' });
expect(observation.reason).toMatch(/location filter/);
});
it('a passing launch that never hits the mutant is a valid non-covering observation', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'passed' })));
const observation = classifyLaunch(evidence({ record, probe: probe(NO_HITS) }));
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'survived',
executed: [TARGET.id],
covering: false,
hits: { loadAndCollection: 0, hook: 0, test: 0, otherTests: 0 },
killedBy: [],
});
});
it('a passing launch that hits the mutant is a valid covering survivor', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'passed' })));
const observation = classifyLaunch(evidence({ record, probe: probe(targetHits(3)) }));
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'survived',
covering: true,
hits: { loadAndCollection: 0, hook: 0, test: 3, otherTests: 0 },
});
});
it('a launch whose installed mutant id differs from the intended one is invalid', () => {
const module = moduleRecord([], {
state: 'failed',
suites: [],
errors: [{ name: 'Error', message: 'oracle setup: installed mutant "8" differs from "7"' }],
});
const observation = classifyLaunch(
evidence({
record: runRecord(module),
probe: probe(null, { intended: '7', installed: '8' }),
exitCode: 1,
}),
);
expect(observation).toMatchObject({
validity: 'invalid',
outcome: 'invalid',
installedMutantId: '8',
});
expect(observation.reason).toMatch(/installed mutant "8" differs from the intended "7"/);
});
it('a launch whose probe names another intended id than the launcher is invalid', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'passed' })));
const observation = classifyLaunch(
evidence({ record, probe: probe(NO_HITS, { intended: '9', installed: '9' }) }),
);
expect(observation).toMatchObject({ validity: 'invalid', outcome: 'invalid' });
});
it('a launch without a record is infrastructure', () => {
const observation = classifyLaunch(evidence({ exitCode: 1, probe: probe(null) }));
expect(observation).toMatchObject({ validity: 'invalid', outcome: 'infrastructure' });
});
});
describe('classifyLaunch: outcome classes', () => {
it('a launch killed by the wall-clock limit is a timeout observation', () => {
const observation = classifyLaunch(
evidence({ killedByWallClock: true, exitCode: null, signal: 'SIGKILL', probe: probe(null) }),
);
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'timeout',
killedBy: [],
covering: null,
});
expect(observation.reason).toMatch(/wall-clock limit/);
});
it('a wall-clock kill before the setup file ran is infrastructure', () => {
const observation = classifyLaunch(
evidence({ killedByWallClock: true, exitCode: null, signal: 'SIGKILL', probe: null }),
);
expect(observation).toMatchObject({ validity: 'invalid', outcome: 'infrastructure' });
});
it('a Vitest testTimeout failure is a timeout observation', () => {
const record = runRecord(
moduleRecord(twoTests({ state: 'failed', errors: [testTimeoutError] })),
);
const observation = classifyLaunch(
evidence({ record, probe: probe(targetHits(1)), exitCode: 1 }),
);
expect(observation).toMatchObject({ validity: 'valid', outcome: 'timeout', killedBy: [] });
});
it('an assertion failure is killed-by-test assertion', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'failed', errors: [assertionError] })));
const observation = classifyLaunch(
evidence({ record, probe: probe(targetHits(2)), exitCode: 1 }),
);
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'killed-by-test:assertion',
killedBy: [TARGET.id],
assertionKilledBy: [TARGET.id],
covering: true,
});
});
it('a failing beforeAll that skips the intended test is killed outside the test', () => {
const module = moduleRecord(twoTests(), {
state: 'failed',
failedHooks: ['beforeAll'],
errors: [{ name: 'TypeError', message: 'boom' }],
});
const observation = classifyLaunch(
evidence({ record: runRecord(module), probe: probe({ ...NO_HITS, hook: 1 }), exitCode: 1 }),
);
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'killed-outside-test:hook',
killedBy: [TARGET.id],
assertionKilledBy: [],
hits: { hook: 1 },
});
});
it('an unmutated baseline launch that passes is a valid survivor without hits', () => {
const record = runRecord(moduleRecord(twoTests({ state: 'passed' })), null);
const observation = classifyLaunch(
evidence({
intendedMutantId: null,
record,
probe: probe(NO_HITS, { intended: null, installed: null }),
}),
);
expect(observation).toMatchObject({
validity: 'valid',
outcome: 'survived',
mutantId: null,
covering: false,
});
});
});
describe('test selection', () => {
const META_TITLE = 'a|b (c) [d] e? f. g* h+ ^i$ {j} \\k';
it.each([
['every metacharacter', META_TITLE, `${META_TITLE}!`],
['a dot', 'S > a.c', 'S > abc'],
['a plus', 'S > a+', 'S > aa'],
['parentheses', 'S > (x)', 'S > x'],
['an alternation', 'S > a|b', 'S > b'],
['a character class', 'S > [ab]', 'S > a'],
['a quantifier brace', 'S > a{2}', 'S > aa'],
['a trailing backslash', 'S > back\\', 'S > back\\\\'],
['a prefix of a longer name', 'S > one', 'S > one two'],
])('the anchored pattern for %s matches only the exact name', (_label, name, other) => {
const pattern = new RegExp(testNamePattern(name));
expect([pattern.test(name), pattern.test(other), pattern.test(`x${name}`)]).toEqual([
true,
false,
false,
]);
});
it('escapes each regex metacharacter with a backslash', () => {
expect(escapeRegExp('|()[]?.*+^${}\\')).toBe('\\|\\(\\)\\[\\]\\?\\.\\*\\+\\^\\$\\{\\}\\\\');
});
it('anchors the escaped full name', () => {
expect(testNamePattern('S > a.b')).toBe('^S > a\\.b$');
});
it('filters by the test file and the line of the test', () => {
expect(locationFilter(TARGET)).toBe(`${FILE}:10`);
});
it('refuses a target without a location', () => {
expect(() => locationFilter({ ...TARGET, location: null })).toThrow(/has no location/);
});
});
describe('launch environment and wall clock', () => {
const base = {
PATH: '/usr/bin',
__STRYKER_ACTIVE_MUTANT__: '99',
STRYKER_MUTATOR_WORKER: '3',
GITNEXUS_MUTATION_SCOPE: '/old/scope.json',
};
it('installs the mutant and the probe settings and drops stale Stryker variables', () => {
const env = buildLaunchEnv(base, {
scopeFile: '/run/scope.json',
probeFile: '/launch/probe.json',
intendedMutantId: '7',
installedMutantId: '7',
});
expect(env).toEqual({
PATH: '/usr/bin',
__STRYKER_ACTIVE_MUTANT__: '7',
GITNEXUS_MUTATION_SCOPE: '/run/scope.json',
GITNEXUS_ORACLE_PROBE_FILE: '/launch/probe.json',
GITNEXUS_ORACLE_INTENDED_MUTANT: '7',
});
});
it('installs no mutant for a baseline launch', () => {
const env = buildLaunchEnv(base, {
scopeFile: '/run/scope.json',
probeFile: '/launch/probe.json',
intendedMutantId: null,
installedMutantId: null,
});
expect([env.__STRYKER_ACTIVE_MUTANT__, env.GITNEXUS_ORACLE_INTENDED_MUTANT]).toEqual([
undefined,
'',
]);
});
it('defaults the wall-clock limit above testTimeout plus hookTimeout', () => {
expect(DEFAULT_WALL_CLOCK_MS).toBeGreaterThan(BASE_TEST_TIMEOUT_MS + BASE_HOOK_TIMEOUT_MS);
});
});
describe('probe file', () => {
it('round-trips through JSON', () => {
const value = probe(targetHits(4));
expect(parseOracleProbe(JSON.stringify(value))).toEqual(value);
});
it('rejects another schema version', () => {
expect(() => parseOracleProbe(JSON.stringify({ ...probe(null), schemaVersion: 2 }))).toThrow(
/schemaVersion/,
);
});
it.each([
['matching ids', '7', '7', false, true],
['differing ids', '7', '8', false, false],
['instrumented code before the setup file', '7', '7', true, false],
['a baseline launch', null, null, false, true],
])('activation check with %s', (_label, intended, installed, namespaceExists, matches) => {
expect(
checkActivation({ intended, installed, namespaceExists, namespaceActiveMutant: null })
.matches,
).toBe(matches);
});
});
describe('oracle cost projection', () => {
const input: CostProjectionInput = {
model: {
vcpus: 4,
launchWallMs: 4000,
launchCpuMs: 2000,
wallClockLimitMs: 180000,
jobLimitMs: 6 * 3600 * 1000,
fixedOverheadMs: 600000,
},
scopes: [
{
name: 'a',
population: 1000,
tests: 100,
sampleSize: 100,
topUps: 20,
timeoutMutants: 10,
hangingTestsPerTimeoutMutant: 5,
commandRunnerMs: 1200000,
commandRunnerConcurrency: 4,
},
{
name: 'b',
population: 50,
tests: 10,
sampleSize: 100,
topUps: 20,
timeoutMutants: 0,
hangingTestsPerTimeoutMutant: 0,
commandRunnerMs: 60000,
commandRunnerConcurrency: 2,
},
],
};
it('is deterministic for a fixed list of timings', () => {
expect(projectOracleCost(input)).toEqual(projectOracleCost(structuredClone(input)));
});
it('projects launches, hangs, command passes and the job-limit verdict', () => {
const projection = projectOracleCost(input);
// a: 120 mutants x 100 tests; 120 x (10 / 1000) x 5 = 6 hanging launches.
// launches: 11994 x 4000 / 4 + 6 x 180000 / 4 = 11994000 + 270000.
// b: the whole population of 50 mutants x 10 tests = 500 launches; command 60000 x 2 / 2.
expect(projection.scopes).toEqual([
{
name: 'a',
sampledMutants: 120,
launches: 12000,
hangingLaunches: 6,
launchSerialMs: 11994 * 4000 + 6 * 180000,
launchMs: 12264000,
commandMs: 1200000,
totalMs: 13464000,
},
{
name: 'b',
sampledMutants: 50,
launches: 500,
hangingLaunches: 0,
launchSerialMs: 2000000,
launchMs: 500000,
commandMs: 60000,
totalMs: 560000,
},
]);
expect(projection).toMatchObject({
launches: 12500,
totalMs: 13464000 + 560000 + 600000,
cpuBoundFloorMs: (11994 + 500) * 500 + 270000 + 1260000 + 600000,
fits: true,
// (6 h - 600000 overhead - 1260000 command - 270000 hanging) x 4 vCPU / 12494 launches.
breakEvenLaunchMs: Math.round(((21600000 - 600000 - 1260000 - 270000) * 4) / 12494),
});
});
it('reports a projection above the job limit as not fitting', () => {
const slow = { ...input, model: { ...input.model, launchWallMs: 8000 } };
expect(projectOracleCost(slow).fits).toBe(false);
});
it('summarises timings with nearest-rank percentiles', () => {
expect(summarizeTimings([5, 1, 4, 2, 3, 10, 6, 7, 9, 8])).toEqual({
count: 10,
meanMs: 5.5,
medianMs: 5,
p90Ms: 9,
minMs: 1,
maxMs: 10,
});
});
});
describe('benchmark launch sample', () => {
const inventory: LaunchTarget[] = ['t0', 't1', 't2', 't3', 't4', 't5'].map((name, index) => ({
id: `${FILE}#S > ${name}`,
file: FILE,
fullName: `S > ${name}`,
location: { line: 10 + index, column: 3 },
}));
const mutant = (
localId: string,
oracleClass: CommandRunMutant['oracleClass'],
killers: string[] = [],
): CommandRunMutant => ({
localId,
globalId: `g${localId}`,
oracleClass,
killedBy: killers,
assertionKilledBy: killers,
wallClockMs: 1000,
});
it('picks evenly spaced entries in order', () => {
expect(evenlySpaced([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 3)).toEqual([1, 5, 8]);
});
it('plans killer, non-killer, survivor and baseline launches deterministically', () => {
const mutants = [
mutant('3', 'killed-by-test:assertion', [`${FILE}#S > t2`]),
mutant('1', 'survived'),
mutant('2', 'timeout'),
];
const plan = planLaunches(mutants, inventory, { mutants: 1, testsPerMutant: 2, baselines: 1 });
expect(
plan.map((launch) => [launch.role, launch.mutantId, launch.target.fullName, launch.expected]),
).toEqual([
['killer', '3', 'S > t2', 'killed-by-test:assertion'],
['non-killer', '3', 'S > t3', 'survived'],
['survivor', '1', 'S > t1', 'survived'],
['survivor', '1', 'S > t4', 'survived'],
['baseline', null, 'S > t3', 'survived'],
]);
});
});

268
gitnexus/test/unit/mutation/mutant-identity.test.ts Normal file
View file

@ -0,0 +1,268 @@
/**
* Locks the global mutant identity (origin §2, KTD12): a mutant is named by its original repo
* path, the content hash of the original file, its 1-based end-exclusive span, its mutator and
* its replacement, so the same mutant gets the same id in Stryker runs whose local ids differ.
*/
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import {
contentHash,
identifyMutants,
parseMutationReport,
type MutationReport,
type ReportMutant,
} from '../../../tools/mutation/prepare/mutant-identity.js';
import { buildPathMap } from '../../../tools/mutation/prepare/path-map.js';
const PACKAGE_DIR = path.join(path.sep, 'prepared', 'gitnexus');
const SHARED_KEY = 'relocated/gitnexus-shared/src/lbug/schema-constants.ts';
const SHARED_REPO = 'gitnexus-shared/src/lbug/schema-constants.ts';
const SRC_KEY = 'src/core/example.ts';
const SRC_REPO = 'gitnexus/src/core/example.ts';
const ORIGINAL = "export const KINDS = ['a', 'b'] as const;\n";
/** What the prepare step writes for {@link ORIGINAL}: ` as const` blanked to spaces. */
const PREPARED = "export const KINDS = ['a', 'b'] ;\n";
const ORIGINALS: Readonly<Record<string, string>> = {
[SHARED_REPO]: ORIGINAL,
[SRC_REPO]: 'const a = 1;\n',
};
const missing = (repoPath: string): never => {
throw new Error(`no original for ${repoPath}`);
};
const readOriginal = (repoPath: string): string => ORIGINALS[repoPath] ?? missing(repoPath);
/** StringLiteral on `'a'` of {@link ORIGINAL}: columns 23-26, 1-based and end exclusive. */
const mutant = (overrides: Partial<ReportMutant> = {}): ReportMutant => ({
id: '0',
mutatorName: 'StringLiteral',
replacement: '""',
status: 'Survived',
location: { start: { line: 1, column: 23 }, end: { line: 1, column: 26 } },
...overrides,
});
const report = (files: Record<string, readonly ReportMutant[]>): MutationReport => ({
schemaVersion: '1.0',
files: Object.fromEntries(
Object.entries(files).map(([file, mutants]) => [
file,
{ language: 'typescript', source: PREPARED, mutants: [...mutants] },
]),
),
});
const identify = (files: Record<string, readonly ReportMutant[]>) =>
identifyMutants(report(files), { packageDir: PACKAGE_DIR, readOriginal });
describe('identifyMutants', () => {
it('gives two mutants on the same span with different replacements different identities', () => {
const run = identify({
[SHARED_KEY]: [mutant({ id: '0' }), mutant({ id: '1', replacement: '"Stryker was here!"' })],
});
const [first, second] = run.mutants;
expect(first.globalId).not.toBe(second.globalId);
expect({ ...first.identity, replacement: '' }).toEqual({ ...second.identity, replacement: '' });
});
it('gives the same mutant the same global id in two runs with different local ids', () => {
const runA = identify({ [SHARED_KEY]: [mutant({ id: '3' })] });
const runB = identify({
[SRC_KEY]: [mutant({ id: '16', mutatorName: 'BooleanLiteral' })],
[SHARED_KEY]: [mutant({ id: '17' })],
});
expect(runA.localToGlobal.get('3')).toBe(runB.localToGlobal.get('17'));
expect(runB.localToGlobal.get('16')).not.toBe(runB.localToGlobal.get('17'));
});
it('maps every local id of a run to its global id', () => {
const run = identify({
[SHARED_KEY]: [mutant({ id: '0' }), mutant({ id: '9', replacement: 'x' })],
});
expect([...run.localToGlobal]).toEqual(run.mutants.map((m) => [m.localId, m.globalId]));
});
it('names the mutant by its original repo path and the hash of the original file', () => {
const [identified] = identify({ [SHARED_KEY]: [mutant()] }).mutants;
expect(identified.identity).toEqual({
file: SHARED_REPO,
fileHash: contentHash(ORIGINAL),
location: { start: { line: 1, column: 23 }, end: { line: 1, column: 26 } },
mutatorName: 'StringLiteral',
replacement: '""',
});
expect(identified.identity.fileHash).not.toBe(contentHash(PREPARED));
expect(identified.reportFile).toBe(SHARED_KEY);
});
it('formats the global id as path@file-hash:span:mutator:replacement-hash', () => {
const [identified] = identify({ [SRC_KEY]: [mutant()] }).mutants;
expect(identified.globalId).toBe(
`${SRC_REPO}@${contentHash('const a = 1;\n').slice(0, 16)}:1:23-1:26:StringLiteral:` +
contentHash('""').slice(0, 16),
);
});
it('changes the global id when the original file content changes', () => {
const before = identify({ [SRC_KEY]: [mutant()] }).mutants[0].globalId;
const after = identifyMutants(report({ [SRC_KEY]: [mutant()] }), {
packageDir: PACKAGE_DIR,
readOriginal: () => 'const a = 2;\n',
}).mutants[0].globalId;
expect(after).not.toBe(before);
});
it('maps an absolute report path under the package dir like a relative one', () => {
const absolute = identify({ [path.join(PACKAGE_DIR, SHARED_KEY)]: [mutant()] });
const relative = identify({ [SHARED_KEY]: [mutant()] });
expect(absolute.mutants[0].globalId).toBe(relative.mutants[0].globalId);
});
it.each([
['a test file', 'test/unit/schema.test.ts'],
['a tooling file', 'tools/mutation/scope-file.ts'],
['a file above the package dir', '../gitnexus-shared/src/index.ts'],
])('throws for a mutant in %s, outside the mapped roots', (_label, file) => {
expect(() => identify({ [file]: [mutant()] })).toThrow(
/outside the mapped roots|not a relative path/,
);
});
it('throws when two local mutants share one global identity', () => {
expect(() => identify({ [SHARED_KEY]: [mutant({ id: '0' }), mutant({ id: '1' })] })).toThrow(
/local ids 0 and 1 have the same global identity/,
);
});
it('throws when a local id appears twice', () => {
expect(() =>
identify({ [SHARED_KEY]: [mutant({ id: '4' }), mutant({ id: '4', replacement: 'x' })] }),
).toThrow(/local id 4 appears twice/);
});
});
describe('identifyMutants: extra roots', () => {
const FIXTURE_KEY = 'test/fixtures/mutation-conformance/src/runtime.ts';
const FIXTURE_REPO = `gitnexus/${FIXTURE_KEY}`;
const fixtureReport = report({ [FIXTURE_KEY]: [mutant()] });
const readFixture = (repoPath: string): string =>
repoPath === FIXTURE_REPO ? ORIGINAL : missing(repoPath);
it('names a fixture mutant by its repo path when the path map has the fixture root', () => {
const [identified] = identifyMutants(fixtureReport, {
packageDir: PACKAGE_DIR,
readOriginal: readFixture,
pathMap: buildPathMap(['gitnexus/test/fixtures/mutation-conformance']),
}).mutants;
expect(identified.globalId.startsWith(`${FIXTURE_REPO}@`)).toBe(true);
});
it('rejects the fixture mutant under the default path map', () => {
expect(() =>
identifyMutants(fixtureReport, { packageDir: PACKAGE_DIR, readOriginal: readFixture }),
).toThrow(/outside the mapped roots/);
});
});
describe('contentHash', () => {
it('is the sha256 hex digest of the UTF-8 text', () => {
expect(contentHash('abc')).toBe(
'ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad',
);
});
});
/** A report in Stryker's JSON shape, as `JSON.parse` sees it. */
const rawReport = (mutantOverrides: Record<string, unknown> = {}): Record<string, unknown> => ({
schemaVersion: '1.0',
projectRoot: PACKAGE_DIR,
thresholds: { high: 80, low: 60, break: null },
files: {
[SHARED_KEY]: {
language: 'typescript',
source: PREPARED,
mutants: [
{
id: '7',
mutatorName: 'StringLiteral',
replacement: '""',
status: 'Killed',
static: true,
statusReason: 'expected',
testsCompleted: 3,
coveredBy: [],
killedBy: ['2', '5'],
location: { end: { column: 26, line: 1 }, start: { column: 23, line: 1 } },
...mutantOverrides,
},
],
},
},
});
describe('parseMutationReport', () => {
it('reads the fields the identity and the outcome comparison consume', () => {
const parsed = parseMutationReport(JSON.stringify(rawReport()));
expect(parsed.projectRoot).toBe(PACKAGE_DIR);
expect(parsed.files[SHARED_KEY]).toEqual({
language: 'typescript',
source: PREPARED,
mutants: [
{
id: '7',
mutatorName: 'StringLiteral',
replacement: '""',
status: 'Killed',
static: true,
statusReason: 'expected',
testsCompleted: 3,
coveredBy: [],
killedBy: ['2', '5'],
location: { start: { line: 1, column: 23 }, end: { line: 1, column: 26 } },
},
],
});
});
const position = (line: unknown, column: unknown) => ({ line, column });
it.each([
['a missing replacement', { replacement: undefined }, /replacement/],
['a numeric id', { id: 7 }, /\.id/],
['a mutator name with a colon', { mutatorName: 'String:Literal' }, /mutatorName/],
['an unknown status', { status: 'Exploded' }, /status/],
['a non-string killedBy entry', { killedBy: [2] }, /killedBy/],
[
'a zero column',
{ location: { start: position(1, 0), end: position(1, 26) } },
/location\.start/,
],
[
'a fractional line',
{ location: { start: position(1.5, 23), end: position(1, 26) } },
/location\.start/,
],
[
'an end before the start',
{ location: { start: position(2, 1), end: position(1, 26) } },
/ends before it starts/,
],
])('rejects a mutant with %s', (_label, overrides, message) => {
expect(() => parseMutationReport(JSON.stringify(rawReport(overrides)))).toThrow(message);
});
it.each([
['schema version 2', { ...rawReport(), schemaVersion: '2' }, /schemaVersion/],
['files that are not an object', { ...rawReport(), files: [] }, /files/],
[
'a file without source',
{ schemaVersion: '1.0', files: { a: { language: 'typescript', mutants: [] } } },
/source/,
],
])('rejects a report with %s', (_label, raw, message) => {
expect(() => parseMutationReport(JSON.stringify(raw))).toThrow(message);
});
});

634
gitnexus/test/unit/mutation/normalise-results.test.ts Normal file
View file

@ -0,0 +1,634 @@
/**
* Locks the scope driver's result normalisation (U10, KTD6, KTD16, KTD17): report test ids map to
* the runner's Stryker ids through the run manifest, sidecar records join their mutant through the
* local id map (latest attempt wins), and integrity problems are flagged instead of hidden. Every
* attempt's per-test rows are kept and a class change between attempts is marked (Phase 1 KTD8),
* and the dry run's hook marker is exposed per mutant (Phase 1 KTD2, KTD9).
*/
import { describe, expect, it } from 'vitest';
import {
mapReportTestIds,
normaliseRun,
orderAttempts,
parseReportTestFiles,
type NormaliseInput,
type ReportTestFiles,
type SidecarInput,
} from '../../../tools/mutation/driver/normalise-results.js';
import {
identifyMutants,
type MutationReport,
type ReportMutant,
} from '../../../tools/mutation/prepare/mutant-identity.js';
import type { RunClassification } from '../../../tools/mutation/runner/classify-outcome.js';
import type { RunManifest } from '../../../tools/mutation/runner/run-manifest.js';
import {
CoverageRecorder,
hookAttributedTests,
hookCoverage,
rekeyCoverage,
type HitContext,
} from '../../../tools/mutation/runner/coverage-attribution.js';
import {
SIDECAR_SCHEMA_VERSION,
type SidecarRecord,
} from '../../../tools/mutation/runner/sidecar.js';
const PACKAGE_DIR = '/prepared/gitnexus';
const SOURCE = 'export const a = 1 + 2;\nexport const b = a > 0;\n';
const FILE = 'src/x.ts';
const T_ONE = 'test/unit/a.test.ts#A > one';
const T_TWO = 'test/unit/a.test.ts#A > two';
const manifest: RunManifest = {
schemaVersion: 1,
project: 'mutation',
vitestVersion: '5.0.5',
files: [
{
file: 'test/unit/a.test.ts',
tests: [
{
id: T_ONE,
taskId: 'h_0',
fullName: 'A > one',
location: null,
notOracleValidated: false,
},
{
id: T_TWO,
taskId: 'h_1',
fullName: 'A > two',
location: null,
notOracleValidated: false,
},
],
skipped: [],
},
],
};
const testFiles: ReportTestFiles = {
'test/unit/a.test.ts': {
tests: [
{ id: '0', name: 'A > one' },
{ id: '1', name: 'A > two' },
],
},
};
const mutant = (id: string, overrides: Partial<ReportMutant>): ReportMutant => ({
id,
mutatorName: 'ArithmeticOperator',
replacement: `r${id}`,
location: { start: { line: 1, column: 18 }, end: { line: 1, column: 23 } },
status: 'Survived',
...overrides,
});
const identities = (mutants: ReportMutant[]) => {
const report: MutationReport = {
schemaVersion: '1.0',
projectRoot: PACKAGE_DIR,
files: { [FILE]: { language: 'typescript', source: SOURCE, mutants } },
};
return identifyMutants(report, { packageDir: PACKAGE_DIR, readOriginal: () => SOURCE });
};
const classification = (overrides: Partial<RunClassification>): RunClassification => ({
status: 'survived',
outcome: 'survived',
reason: undefined,
killedBy: [],
failureMessage: undefined,
nrOfTests: 2,
hitCount: 1,
tests: [],
outsideTest: [],
...overrides,
});
const killedByOne = classification({
status: 'killed',
outcome: 'killed-by-test:assertion',
killedBy: [T_ONE],
failureMessage: 'expected 3 to be 4',
tests: [
{ id: T_ONE, outcome: 'killed-by-test:assertion', message: 'expected 3 to be 4' },
{ id: T_TWO, outcome: 'survived', message: undefined },
],
});
const sidecar = (
mutantId: string | null,
result: RunClassification,
where: {
worker?: string;
attempt?: number;
at?: number;
hookAttributedTests?: Record<string, string[]>;
} = {},
): SidecarInput => {
const record: SidecarRecord = {
schemaVersion: SIDECAR_SCHEMA_VERSION,
kind: mutantId === null ? 'dry-run' : 'mutant',
mutantId,
worker: where.worker ?? '1',
attempt: where.attempt ?? 1,
pid: 100,
mutantActivation: mutantId === null ? null : 'runtime',
hitLimit: null,
requestedTestIds: [T_ONE, T_TWO],
hookAttributedTests: mutantId === null ? (where.hookAttributedTests ?? {}) : null,
result,
};
return {
record,
file: `/out/sidecars/${mutantId ?? 'dry-run'}.json`,
writtenAtMs: where.at ?? 0,
};
};
const dryRun = sidecar(null, classification({ nrOfTests: 2 }));
const run = (mutants: ReportMutant[], sidecars: SidecarInput[]): NormaliseInput => ({
identities: identities(mutants),
testFiles,
manifest,
sidecars: [dryRun, ...sidecars],
});
describe('normaliseRun: sidecar join through the local id map', () => {
const ids = identities([
mutant('7', { status: 'Killed', killedBy: ['0'], coveredBy: ['0', '1'] }),
]);
const result = normaliseRun({
identities: ids,
testFiles,
manifest,
sidecars: [dryRun, sidecar('7', killedByOne, { worker: '2' })],
});
const [only] = result.mutants;
it('carries the global id of the local mutant', () => {
expect(only.globalId).toBe(ids.localToGlobal.get('7'));
expect(only.localId).toBe('7');
expect(only.file).toBe(`gitnexus/${FILE}`);
});
it('takes the outcome class from the sidecar record of that local id', () => {
expect(only).toMatchObject({
outcome: 'killed-by-test:assertion',
outcomeSource: 'sidecar',
sidecar: { worker: '2', attempt: 1, status: 'killed', killedBy: [T_ONE], nrOfTests: 2 },
assertionKilledBy: [T_ONE],
});
});
it('maps report killers and coverage to runner test ids', () => {
expect(only.killedBy).toEqual([T_ONE]);
expect(only.coveredBy).toEqual([T_ONE, T_TWO]);
});
it('builds per-test kill sets keyed by runner test id', () => {
expect(result.tests.find((test) => test.id === T_ONE)).toMatchObject({
killed: [only.globalId],
assertionKilled: [only.globalId],
});
expect(result.tests.find((test) => test.id === T_TWO)?.killed).toEqual([]);
});
it('raises no flag', () => {
expect(result.flags).toEqual([]);
});
});
describe('normaliseRun: assertion kills come only from killed runs (KTD17)', () => {
const assertionRow = { id: T_ONE, outcome: 'killed-by-test:assertion', message: 'x' } as const;
// Found by the gate 2 conformance fixture (U4): a hit-limit timeout whose swallowed error made
// the test fail its assertion, and inconclusive runs of a scope with a `retry: 1` test.
const notKilled = [
['a hit-limit timeout', 'Timeout', classification({ status: 'timeout', outcome: 'timeout' })],
[
'an inconclusive run',
'RuntimeError',
classification({ status: 'error', outcome: 'inconclusive' }),
],
['an invalid run', 'RuntimeError', classification({ status: 'error', outcome: 'invalid' })],
] as const;
it.each(notKilled)('lists no assertion killer for %s', (_label, status, result) => {
const normalised = normaliseRun(
run([mutant('5', { status })], [sidecar('5', { ...result, tests: [assertionRow] })]),
);
expect(normalised.mutants[0].assertionKilledBy).toEqual([]);
expect(normalised.tests.find((test) => test.id === T_ONE)?.assertionKilled).toEqual([]);
});
it('still lists the assertion killers of a killed run', () => {
const normalised = normaliseRun(
run([mutant('6', { status: 'Killed', killedBy: ['0'] })], [sidecar('6', killedByOne)]),
);
expect(normalised.mutants[0].assertionKilledBy).toEqual([T_ONE]);
});
});
describe('normaliseRun: integrity flags', () => {
it('flags a killedBy entry with no sidecar record, and the Killed mutant without one', () => {
const result = normaliseRun(run([mutant('3', { status: 'Killed', killedBy: ['1'] })], []));
expect(result.flags.map((flag) => [flag.kind, flag.localId])).toEqual([
['killed-by-without-sidecar', '3'],
['killed-without-sidecar', '3'],
]);
expect(result.flags[0].detail).toContain(T_TWO);
expect(result.mutants[0]).toMatchObject({ outcome: null, outcomeSource: 'none' });
});
it('flags a killedBy entry the sidecar does not list', () => {
const result = normaliseRun(
run([mutant('3', { status: 'Killed', killedBy: ['0', '1'] })], [sidecar('3', killedByOne)]),
);
expect(result.flags.map((flag) => flag.kind)).toEqual(['killed-by-not-in-sidecar']);
expect(result.flags[0].detail).toContain(T_TWO);
});
it('flags a mutant run that executed zero tests', () => {
const invalid = classification({ status: 'error', outcome: 'invalid', nrOfTests: 0 });
const result = normaliseRun(
run([mutant('4', { status: 'RuntimeError' })], [sidecar('4', invalid)]),
);
expect(result.flags.map((flag) => flag.kind)).toEqual(['zero-tests']);
expect(result.summary.zeroTestRuns).toBe(1);
});
it('does not flag zero tests when the file failed to collect', () => {
const collection = classification({
status: 'killed',
outcome: 'killed-outside-test:collection',
nrOfTests: 0,
});
const result = normaliseRun(
run([mutant('4', { status: 'Killed', static: true })], [sidecar('4', collection)]),
);
expect(result.flags).toEqual([]);
});
// A loop turned endless at module load or in a beforeAll fixture hits the instrumenter's limit
// (or the hook timeout) before any requested test runs: a timeout, explained by its row.
it.each([
[
'a hit-limit timeout while collecting the file',
undefined,
'Stryker: Hit count limit reached',
],
['a beforeAll hook timeout', 'h_s0', 'Hook timed out in 2000ms'],
])('does not flag zero tests explained by %s', (_label, suiteTaskId, message) => {
const timeout = classification({
status: 'timeout',
outcome: 'timeout',
nrOfTests: 0,
outsideTest: [
{ file: 'test/unit/a.test.ts', suiteTaskId, outcome: 'timeout', attributedTo: [], message },
],
});
const result = normaliseRun(run([mutant('4', { status: 'Timeout' })], [sidecar('4', timeout)]));
expect(result.flags).toEqual([]);
expect(result.summary.zeroTestRuns).toBe(0);
});
it.each([
['an invalid run', 'RuntimeError', classification({ status: 'error', outcome: 'invalid' })],
[
'a timeout with no outside-test row',
'Timeout',
classification({ status: 'timeout', outcome: 'timeout' }),
],
] as const)('still flags zero tests for %s', (_label, status, result) => {
const normalised = normaliseRun(
run([mutant('4', { status })], [sidecar('4', { ...result, nrOfTests: 0 })]),
);
expect(normalised.flags.map((flag) => flag.kind)).toEqual(['zero-tests']);
});
it('flags a sidecar status that disagrees with the Stryker status', () => {
const result = normaliseRun(
run([mutant('5', { status: 'Survived' })], [sidecar('5', killedByOne)]),
);
expect(result.flags.map((flag) => flag.kind)).toEqual(['status-mismatch']);
});
it('flags a Survived mutant without a sidecar record', () => {
const result = normaliseRun(run([mutant('5', { status: 'Survived' })], []));
expect(result.flags.map((flag) => flag.kind)).toEqual(['missing-sidecar']);
});
it('flags a sidecar record whose local id is not in the report', () => {
const result = normaliseRun(run([], [sidecar('99', killedByOne)]));
expect(result.flags).toEqual([
expect.objectContaining({ kind: 'orphan-sidecar', localId: '99', globalId: null }),
]);
});
it('flags a vitest5 run without a dry-run record', () => {
const result = normaliseRun({ ...run([], []), sidecars: [] });
expect(result.flags.map((flag) => flag.kind)).toEqual(['dry-run-missing']);
expect(result.dryRun).toBeNull();
});
it('flags a report test id that maps to no runner test', () => {
const result = normaliseRun({
...run(
[mutant('6', { status: 'Survived', coveredBy: ['42'] })],
[sidecar('6', classification({}))],
),
});
expect(result.flags.map((flag) => [flag.kind, flag.localId])).toEqual([
['unmapped-test-id', '6'],
]);
expect(result.mutants[0].coveredBy).toEqual(['42']);
});
});
describe('normaliseRun: outcomes without a sidecar record', () => {
it.each([
['NoCoverage', 'survived', 'no-coverage'],
['Timeout', 'timeout', 'stryker-timeout'],
['Ignored', null, 'none'],
] as const)('%s -> %s (%s), no flag', (status, outcome, outcomeSource) => {
const result = normaliseRun(run([mutant('8', { status })], []));
expect(result.mutants[0]).toMatchObject({ outcome, outcomeSource });
expect(result.flags).toEqual([]);
});
});
describe('normaliseRun: attempts', () => {
it('records every attempt and lets the latest one win', () => {
const timeout = classification({ status: 'timeout', outcome: 'timeout', nrOfTests: 1 });
const result = normaliseRun(
run(
[mutant('9', { status: 'Killed', killedBy: ['0'] })],
[
sidecar('9', killedByOne, { worker: '3', at: 20 }),
sidecar('9', timeout, { worker: '1', at: 10 }),
],
),
);
expect(result.mutants[0].attempts.map((attempt) => [attempt.worker, attempt.outcome])).toEqual([
['1', 'timeout'],
['3', 'killed-by-test:assertion'],
]);
expect(result.mutants[0].sidecar?.worker).toBe('3');
expect(result.flags).toEqual([]);
});
it('orders attempts by write time, then numeric worker, then attempt number', () => {
const records = [
sidecar('1', killedByOne, { worker: '10', attempt: 1, at: 5 }),
sidecar('1', killedByOne, { worker: '9', attempt: 2, at: 5 }),
sidecar('1', killedByOne, { worker: '9', attempt: 1, at: 5 }),
sidecar('1', killedByOne, { worker: '2', attempt: 1, at: 1 }),
];
expect(
orderAttempts(records).map(({ record }) => `${record.worker}/${record.attempt}`),
).toEqual(['2/1', '9/1', '9/2', '10/1']);
});
});
describe('normaliseRun: per-test classes of every attempt (KTD8)', () => {
const runtimeByOne = classification({
status: 'killed',
outcome: 'killed-by-test:runtime',
killedBy: [T_ONE],
tests: [
{ id: T_ONE, outcome: 'killed-by-test:runtime', message: 'x is undefined' },
{ id: T_TWO, outcome: 'survived', message: undefined },
],
});
it('keeps the rows of both attempts and marks a class change between them', () => {
const result = normaliseRun(
run(
[mutant('9', { status: 'Killed', killedBy: ['0'] })],
[
sidecar('9', killedByOne, { worker: '1', at: 10 }),
sidecar('9', runtimeByOne, { worker: '2', at: 20 }),
],
),
);
expect([result.mutants[0].perTest, result.mutants[0].attemptsDisagree]).toEqual([
[
{ testId: T_ONE, outcome: 'killed-by-test:assertion', attempt: 0 },
{ testId: T_TWO, outcome: 'survived', attempt: 0 },
{ testId: T_ONE, outcome: 'killed-by-test:runtime', attempt: 1 },
{ testId: T_TWO, outcome: 'survived', attempt: 1 },
],
true,
]);
});
it.each([
['one attempt', [sidecar('9', killedByOne)]],
[
'two attempts with the same classes',
[sidecar('9', killedByOne, { at: 1 }), sidecar('9', killedByOne, { worker: '2', at: 2 })],
],
])('does not mark %s as disagreeing', (_label, records) => {
const result = normaliseRun(run([mutant('9', { status: 'Killed', killedBy: ['0'] })], records));
expect(result.mutants[0].attemptsDisagree).toBe(false);
});
it('keeps no rows and no disagreement for a mutant without a record', () => {
const result = normaliseRun(run([mutant('8', { status: 'NoCoverage' })], []));
expect([result.mutants[0].perTest, result.mutants[0].attemptsDisagree]).toEqual([[], false]);
});
});
describe('normaliseRun: hook marker from the dry-run record (KTD2, KTD9)', () => {
/** The dry run's hook bucket: mutant 7 is hit only in suite `h`'s beforeAll. */
const dryRunMarker = (): Record<string, string[]> => {
let context: HitContext = { phase: 'collection' };
const recorder = new CoverageRecorder(() => context);
context = { phase: 'run', testId: undefined, runningSuiteIds: ['f', 'h'] };
const coverage = recorder.instrumenterView;
coverage.static['7'] = (coverage.static['7'] ?? 0) + 1;
const byTask = hookCoverage(recorder.snapshot(), { f: ['h_0', 'h_1'], h: ['h_0', 'h_1'] });
const byId = rekeyCoverage(
{ static: {}, perTest: byTask },
new Map([
['h_0', T_ONE],
['h_1', T_TWO],
]),
);
return hookAttributedTests(byId.perTest);
};
it('lists every test of the suite for a mutant covered only through a beforeAll hit', () => {
const result = normaliseRun({
identities: identities([
mutant('7', { status: 'Killed', killedBy: ['0'], coveredBy: ['0', '1'] }),
mutant('8', { status: 'NoCoverage' }),
]),
testFiles,
manifest,
sidecars: [
sidecar(null, classification({}), { hookAttributedTests: dryRunMarker() }),
sidecar('7', killedByOne),
],
});
expect(result.mutants.map((entry) => [entry.localId, entry.hookAttributedTests])).toEqual([
['7', [T_ONE, T_TWO]],
['8', []],
]);
});
it('has no marker when no dry-run record names one', () => {
const result = normaliseRun({
...run([mutant('8', { status: 'NoCoverage' })], []),
sidecars: [],
});
expect(result.mutants[0].hookAttributedTests).toBeNull();
});
});
describe('normaliseRun: summary and dry run', () => {
const result = normaliseRun(
run(
[
mutant('1', { status: 'Killed', killedBy: ['0'] }),
mutant('2', { status: 'NoCoverage' }),
mutant('3', { status: 'Survived' }),
],
[sidecar('1', killedByOne), sidecar('3', classification({}))],
),
);
it('counts statuses and outcome classes', () => {
expect(result.summary).toMatchObject({
mutants: 3,
byStatus: { Killed: 1, NoCoverage: 1, Survived: 1 },
byOutcome: { 'killed-by-test:assertion': 1, survived: 2 },
mutantsWithSidecar: 2,
mutantsWithOutcome: 3,
testsWithKills: 1,
testsWithAssertionKills: 1,
zeroTestRuns: 0,
flags: 0,
});
});
it('summarises the dry-run record', () => {
expect(result.dryRun).toMatchObject({ outcome: 'survived', nrOfTests: 2, attempts: 1 });
});
});
describe('normaliseRun: command variant (no manifest, no sidecars)', () => {
const commandFiles: ReportTestFiles = { '': { tests: [{ id: '0', name: 'All tests' }] } };
const result = normaliseRun({
identities: identities([
mutant('1', { status: 'Survived' }),
mutant('2', { status: 'Killed', killedBy: ['0'] }),
]),
testFiles: commandFiles,
manifest: null,
sidecars: null,
});
it('maps report test ids to report test names and joins no sidecar', () => {
expect(result.mutants.map((m) => [m.status, m.killedBy, m.outcomeSource])).toEqual([
['Survived', [], 'none'],
['Killed', ['All tests'], 'none'],
]);
expect(result.tests).toEqual([
expect.objectContaining({
id: 'All tests',
taskId: null,
killed: [result.mutants[1].globalId],
}),
]);
});
it('raises no sidecar flag', () => {
expect(result.flags).toEqual([]);
expect(result.dryRun).toBeNull();
});
});
describe('mapReportTestIds', () => {
const twin = (id: string, taskId: string, skipped = false) => ({
id,
taskId,
fullName: 'T > row',
location: { line: 3, column: 5 },
notOracleValidated: skipped,
});
it('pairs tests that share a full name by collection order, across ran and skipped tests', () => {
const twins: RunManifest = {
...manifest,
files: [
{
file: 'test/unit/t.test.ts',
tests: [
twin('test/unit/t.test.ts#T > row:3:5:0', 'h_0'),
twin('test/unit/t.test.ts#T > row:3:5:2', 'h_2'),
],
skipped: [{ ...twin('test/unit/t.test.ts#T > other', 'h_1'), fullName: 'T > other' }],
},
],
};
const files: ReportTestFiles = {
'test/unit/t.test.ts': {
tests: [
{ id: '0', name: 'T > row' },
{ id: '1', name: 'T > other' },
{ id: '2', name: 'T > row' },
],
},
};
const mapped = mapReportTestIds(files, twins);
expect([...mapped.ids]).toEqual([
['0', 'test/unit/t.test.ts#T > row:3:5:0'],
['1', 'test/unit/t.test.ts#T > other'],
['2', 'test/unit/t.test.ts#T > row:3:5:2'],
]);
expect(mapped.problems).toEqual([]);
});
it('reports a report test that the manifest does not list, and a manifest test the report lacks', () => {
const files: ReportTestFiles = {
'test/unit/a.test.ts': {
tests: [
{ id: '0', name: 'A > one' },
{ id: '1', name: 'A > ghost' },
],
},
'test/unit/z.test.ts': { tests: [{ id: '2', name: 'Z' }] },
};
const mapped = mapReportTestIds(files, manifest);
expect([...mapped.ids]).toEqual([['0', T_ONE]]);
expect(mapped.problems).toEqual([
expect.stringContaining('"A > ghost"'),
expect.stringContaining(T_TWO),
expect.stringContaining('test/unit/z.test.ts'),
]);
});
});
describe('parseReportTestFiles', () => {
it('reads the testFiles section of a Stryker report', () => {
const text = JSON.stringify({ schemaVersion: '1.0', files: {}, testFiles });
expect(parseReportTestFiles(text)).toEqual(testFiles);
});
it('returns no files when the report has no testFiles section', () => {
expect(parseReportTestFiles(JSON.stringify({ schemaVersion: '1.0', files: {} }))).toEqual({});
});
it.each([
['a non-object section', { testFiles: [] }, /testFiles must be an object/],
['a file without tests', { testFiles: { a: {} } }, /tests must be an array/],
['a test without a name', { testFiles: { a: { tests: [{ id: '0' }] } } }, /id and name/],
])('rejects %s', (_label, value, message) => {
expect(() => parseReportTestFiles(JSON.stringify(value))).toThrow(message);
});
});

189
gitnexus/test/unit/mutation/oracle-boundary.test.ts Normal file
View file

@ -0,0 +1,189 @@
/**
* Enforces KTD15: code under tools/mutation/oracle/ never imports code under
* tools/mutation/runner/ (the vitest5 runner), so gate 3 compares two independent
* implementations. The scanner reads every direct import specifier of each oracle module with
* @babel/parser: `import` (type-only included), `export ... from`, `import x = require()`,
* `import()` and `require()`; a dynamic import or require with a non-literal argument fails,
* because it cannot be checked. Relative specifiers resolve against the module's directory.
*
* Allowed: the scope driver (tools/mutation/driver/) and the prepare step
* (tools/mutation/prepare/), which KTD18 lets runner and oracle share. command-run.ts (U13)
* imports driver/run-scope.ts, which imports runner/run-manifest.ts and runner/sidecar.ts; the
* oracle reaches those runner modules only transitively through the driver, and only this
* direct-import rule is enforced.
*/
import { readFileSync, readdirSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { parse } from '@babel/parser';
import { describe, expect, it } from 'vitest';
const PACKAGE_DIR = fileURLToPath(new URL('../../../', import.meta.url));
const ORACLE_DIR = path.join(PACKAGE_DIR, 'tools/mutation/oracle');
const RUNNER_DIR = path.join(PACKAGE_DIR, 'tools/mutation/runner');
interface ImportSite {
specifier: string | null;
line: number;
}
interface Violation {
file: string;
line: number;
detail: string;
}
interface AstNode {
type: string;
loc?: { start: { line: number } } | null;
[key: string]: unknown;
}
const isNode = (value: unknown): value is AstNode =>
typeof value === 'object' && value !== null && typeof Reflect.get(value, 'type') === 'string';
/** The cooked text of a template element (`{ value: { raw, cooked } }`). */
const cooked = (quasi: unknown): string => {
const value: unknown = isNode(quasi) ? quasi.value : undefined;
const text: unknown =
typeof value === 'object' && value !== null ? Reflect.get(value, 'cooked') : undefined;
return typeof text === 'string' ? text : '';
};
const literal = (node: unknown): string | null =>
isNode(node) && node.type === 'StringLiteral' && typeof node.value === 'string'
? node.value
: isNode(node) &&
node.type === 'TemplateLiteral' &&
Array.isArray(node.expressions) &&
node.expressions.length === 0 &&
Array.isArray(node.quasis)
? node.quasis.map(cooked).join('')
: null;
/** Every direct import site of a module's source, in source order. */
function importSites(source: string): ImportSite[] {
const ast = parse(source, { sourceType: 'module', plugins: ['typescript'] });
const sites: ImportSite[] = [];
const visit = (node: unknown): void => {
if (Array.isArray(node)) {
for (const child of node) visit(child);
return;
}
if (!isNode(node)) return;
const line = node.loc?.start.line ?? 0;
const declares =
node.type === 'ImportDeclaration' ||
node.type === 'ExportAllDeclaration' ||
(node.type === 'ExportNamedDeclaration' && node.source !== null);
if (declares) sites.push({ specifier: literal(node.source), line });
if (node.type === 'TSExternalModuleReference') {
sites.push({ specifier: literal(node.expression), line });
}
if (node.type === 'ImportExpression') sites.push({ specifier: literal(node.source), line });
const callee = node.type === 'CallExpression' && isNode(node.callee) ? node.callee : null;
const calls =
callee !== null &&
(callee.type === 'Import' || (callee.type === 'Identifier' && callee.name === 'require'));
if (calls && Array.isArray(node.arguments)) {
sites.push({ specifier: literal(node.arguments[0]), line });
}
for (const [key, value] of Object.entries(node)) {
if (key !== 'loc' && key !== 'start' && key !== 'end') visit(value);
}
};
visit(ast.program);
return sites;
}
const inside = (dir: string, file: string): boolean => {
const relative = path.relative(dir, file);
return relative === '' || (!relative.startsWith('..') && !path.isAbsolute(relative));
};
/** Import sites of `source` (a module at `file`) that reach the runner or cannot be checked. */
function boundaryViolations(file: string, source: string): Violation[] {
return importSites(source).flatMap((site): Violation[] => {
const at = {
file: path.relative(PACKAGE_DIR, file).split(path.sep).join('/'),
line: site.line,
};
if (site.specifier === null) return [{ ...at, detail: 'non-literal dynamic import' }];
const relative = site.specifier.startsWith('.') || path.isAbsolute(site.specifier);
if (!relative) return [];
const target = path.resolve(path.dirname(file), site.specifier);
return inside(RUNNER_DIR, target) ? [{ ...at, detail: `imports ${site.specifier}` }] : [];
});
}
const oracleModules = (): string[] =>
readdirSync(ORACLE_DIR, { recursive: true, encoding: 'utf8' })
.filter((name) => /\.[cm]?[jt]s$/.test(name))
.map((name) => path.join(ORACLE_DIR, name))
.sort();
const SYNTHETIC = path.join(ORACLE_DIR, 'synthetic.ts');
describe('oracle boundary (KTD15)', () => {
it('scans the oracle modules', () => {
const names = oracleModules().map((file) => path.basename(file));
expect(names).toEqual(
expect.arrayContaining([
'command-run.ts',
'command-wrapper.ts',
'compare.ts',
'freeze.ts',
'launch-single.ts',
'oracle-classify.ts',
'oracle-reporter.ts',
'oracle-setup.ts',
'probe.ts',
]),
);
});
it('no module under oracle/ imports from runner/', () => {
const violations = oracleModules().flatMap((file) =>
boundaryViolations(file, readFileSync(file, 'utf8')),
);
expect(violations).toEqual([]);
});
it('command-run.ts reaches the runner only through the scope driver', () => {
const file = path.join(ORACLE_DIR, 'command-run.ts');
const specifiers = importSites(readFileSync(file, 'utf8')).map((site) => site.specifier);
expect(specifiers).toEqual(expect.arrayContaining(['../driver/run-scope.js']));
});
it.each([
['a static import', 1, "import { RUNNER_NAME } from '../runner/plugin.js';"],
['a type-only import', 1, "import type { FileMeta } from '../runner/protocol.js';"],
['a re-export', 1, "export { readSidecars } from '../runner/sidecar.js';"],
['a star re-export', 1, "export * from '../runner/test-ids.js';"],
['a dynamic import', 1, "const m = await import('../runner/setup.js');"],
['a template-literal import', 1, 'const m = await import(`../runner/setup.js`);'],
['a require', 1, "const m = require('../runner/version-guard.js');"],
['an import-equals', 1, "import m = require('../runner/run-manifest.js');"],
['a longer path into the runner', 1, "import '../../mutation/runner/plugin.js';"],
['the runner directory itself', 1, "import '../runner';"],
['a non-literal dynamic import', 1, 'const m = await import(name);'],
['the scope driver (allowed)', 0, "import { runScope } from '../driver/run-scope.js';"],
['the prepare step (allowed)', 0, "import { prepare } from '../prepare/prepare.js';"],
['another oracle module (allowed)', 0, "import { classifyRun } from './oracle-classify.js';"],
['a package (allowed)', 0, "import { startVitest } from 'vitest/node';"],
['a sibling named like the runner (allowed)', 0, "import x from '../runner-notes/x.js';"],
])('a synthetic oracle module with %s has %i violation(s)', (_label, count, source) => {
expect(boundaryViolations(SYNTHETIC, source)).toHaveLength(count);
});
it('reports the module and line of a violation', () => {
const source = "import path from 'node:path';\n\nimport { x } from '../runner/plugin.js';\n";
expect(boundaryViolations(SYNTHETIC, source)).toEqual([
{
file: 'tools/mutation/oracle/synthetic.ts',
line: 3,
detail: 'imports ../runner/plugin.js',
},
]);
});
});

520
gitnexus/test/unit/mutation/oracle-classify.test.ts Normal file
View file

@ -0,0 +1,520 @@
/**
* Locks the oracle's own KTD17 classifier (U13): synthetic wrapper records (Vitest observations
* from the oracle reporter plus Vitest's JSON totals) and Stryker statuses read from a
* `mutation.json` text map to the outcome classes, per-test rows and kill sets.
*/
import { describe, expect, it } from 'vitest';
import {
parseMutationReport,
type MutantStatus,
} from '../../../tools/mutation/prepare/mutant-identity.js';
import {
ORACLE_CLASSES,
assignTestIds,
classifyMutant,
classifyRun,
isUnusableClass,
parseOracleRunRecord,
parseRecordFileName,
recordFileName,
strykerStatuses,
toOracleError,
type OracleError,
type OracleModuleRecord,
type OracleObservedRun,
type OracleRunRecord,
type OracleSuiteRecord,
type OracleTestRecord,
type VitestJsonTotals,
} from '../../../tools/mutation/oracle/oracle-classify.js';
const FILE = 'test/unit/a.test.ts';
const T0 = `${FILE}#S > t0`;
const T1 = `${FILE}#S > t1`;
const EXPECTED: readonly string[] = [T0, T1];
const error = (name: string, message: string): OracleError => ({ name, message });
const assertionError = toOracleError({
message: 'expected 1 to be 2',
stack: 'AssertionError: expected 1 to be 2\n at test/unit/a.test.ts:12:15',
});
const typeError = error('TypeError', "Cannot read properties of undefined (reading 'x')");
const testTimeout = error(
'Error',
'Test timed out in 30000ms.\nIf this is a long-running test, pass a timeout value as the last ' +
'argument or configure it globally with "testTimeout".',
);
const hookTimeout = error(
'Error',
'Hook timed out in 120000ms.\nIf this is a long-running hook, pass a timeout value as the last ' +
'argument or configure it globally with "hookTimeout".',
);
const test = (index: number, overrides: Partial<OracleTestRecord> = {}): OracleTestRecord => ({
fullName: `S > t${index}`,
location: { line: 10 + index, column: 3 },
state: 'passed',
errors: [],
failedHooks: [],
retry: 0,
retryCount: 0,
...overrides,
});
const suite = (tests: readonly number[], overrides: Partial<OracleSuiteRecord> = {}) => ({
fullName: 'S',
errors: [],
failedHooks: [],
tests: [...tests],
...overrides,
});
const module = (
tests: OracleTestRecord[],
overrides: Partial<OracleModuleRecord> = {},
): OracleModuleRecord => ({
file: FILE,
state: 'passed',
errors: [],
failedHooks: [],
suites: [suite(tests.map((_, index) => index))],
tests,
...overrides,
});
const observed = (
modules: OracleModuleRecord[],
overrides: Partial<OracleObservedRun> = {},
): OracleObservedRun => ({ reason: 'passed', unhandledErrors: [], modules, ...overrides });
/** What Vitest's own JSON reporter counts for the same run. */
const totals = (run: OracleObservedRun): VitestJsonTotals => {
const all = run.modules.flatMap((entry) => entry.tests);
return {
numTotalTests: all.length,
numPassedTests: all.filter((entry) => entry.state === 'passed').length,
numFailedTests: all.filter((entry) => entry.state === 'failed').length,
};
};
const record = (
run: OracleObservedRun,
exitCode: number,
overrides: Partial<OracleRunRecord> = {},
): OracleRunRecord => ({
schemaVersion: 1,
kind: 'mutant',
mutantId: '7',
attempt: 1,
pid: 4242,
root: '/sandbox',
configFile: '/sandbox/tools/mutation/vitest.mutation.config.ts',
vitestVersion: '5.0.5',
exitCode,
wallClockMs: 1500,
wrapperError: null,
run,
vitestJson: totals(run),
vitestJsonFile: '/reports/mutant-7.attempt-1.vitest.json',
...overrides,
});
const classify = (run: OracleObservedRun, exitCode: number) =>
classifyMutant({
strykerStatus: exitCode === 0 ? 'Survived' : 'Killed',
record: record(run, exitCode),
expected: EXPECTED,
});
/** A one-file Stryker report holding a mutant with local id 7 and the given status. */
const mutationJson = (status: MutantStatus): string =>
JSON.stringify({
schemaVersion: '1.0',
files: {
'src/x.ts': {
language: 'typescript',
source: 'export const x = 1;\n',
mutants: [
{
id: '7',
mutatorName: 'BooleanLiteral',
replacement: 'false',
location: { start: { line: 1, column: 18 }, end: { line: 1, column: 19 } },
status,
},
],
},
},
});
const statusOf = (status: MutantStatus): MutantStatus =>
strykerStatuses(parseMutationReport(mutationJson(status))).get('7') ?? 'Pending';
describe('oracle classifier: KTD17 rows from a wrapper record', () => {
it('classifies one failing test with an AssertionError stack as killed-by-test, assertion', () => {
const result = classify(
observed([module([test(0, { state: 'failed', errors: [assertionError] }), test(1)])]),
1,
);
expect(assertionError.name).toBe('AssertionError');
expect(result).toMatchObject({
outcome: 'killed-by-test:assertion',
source: 'report',
killedBy: [T0],
assertionKilledBy: [T0],
executed: [T0, T1],
});
expect(result.tests).toEqual([
{
id: T0,
file: FILE,
fullName: 'S > t0',
outcome: 'killed-by-test:assertion',
message: 'expected 1 to be 2',
},
{ id: T1, file: FILE, fullName: 'S > t1', outcome: 'survived', message: null },
]);
});
it('classifies a file-level error with no tests as killed-outside-test, collection', () => {
const result = classify(
observed([
module([], {
state: 'failed',
errors: [error('SyntaxError', 'Unexpected token')],
suites: [],
}),
]),
1,
);
expect(result).toMatchObject({
outcome: 'killed-outside-test:collection',
killedBy: [],
assertionKilledBy: [],
executed: [],
});
expect(result.outsideTest).toEqual([
{
file: FILE,
suite: null,
outcome: 'killed-outside-test:collection',
attributedTo: [],
message: 'SyntaxError: Unexpected token',
},
]);
});
it('classifies a missing report with a non-zero exit and no Timeout status as infrastructure', () => {
expect(
classifyMutant({ strykerStatus: 'Killed', record: null, expected: EXPECTED }),
).toMatchObject({
outcome: 'infrastructure',
source: 'missing-report',
killedBy: [],
});
});
it('classifies a Stryker Timeout with no report as timeout, read from mutation.json', () => {
const strykerStatus = statusOf('Timeout');
expect(strykerStatus).toBe('Timeout');
expect(classifyMutant({ strykerStatus, record: null, expected: EXPECTED })).toMatchObject({
outcome: 'timeout',
source: 'stryker-timeout',
killedBy: [],
});
});
it('classifies a Stryker Timeout as timeout even when a report shows a kill', () => {
const run = observed([
module([test(0, { state: 'failed', errors: [assertionError] }), test(1)]),
]);
expect(
classifyMutant({ strykerStatus: 'Timeout', record: record(run, 1), expected: EXPECTED }),
).toMatchObject({ outcome: 'timeout', source: 'stryker-timeout', assertionKilledBy: [] });
});
it('classifies a report with zero executed tests as invalid', () => {
const skipped = { state: 'skipped' as const };
const result = classify(observed([module([test(0, skipped), test(1, skipped)])]), 0);
expect(result.outcome).toBe('invalid');
expect(result.reason).toContain('zero tests ran');
});
it('classifies a TypeError as killed-by-test, runtime', () => {
const result = classify(
observed([module([test(0, { state: 'failed', errors: [typeError] }), test(1)])]),
1,
);
expect(result).toMatchObject({
outcome: 'killed-by-test:runtime',
killedBy: [T0],
assertionKilledBy: [],
});
});
it('classifies a beforeEach failure as killed-by-test, hook', () => {
const failed = test(0, { state: 'failed', errors: [typeError], failedHooks: ['beforeEach'] });
const result = classify(observed([module([failed, test(1)])]), 1);
expect(result).toMatchObject({
outcome: 'killed-by-test:hook',
killedBy: [T0],
assertionKilledBy: [],
});
});
it('attributes a beforeAll failure to the skipped suite tests as killed-outside-test, hook', () => {
const skipped = { state: 'skipped' as const };
const result = classify(
observed([
module([test(0, skipped), test(1, skipped)], {
state: 'failed',
suites: [suite([0, 1], { failedHooks: ['beforeAll'], errors: [typeError] })],
}),
]),
1,
);
expect(result).toMatchObject({
outcome: 'killed-outside-test:hook',
killedBy: [T0, T1],
assertionKilledBy: [],
executed: [],
});
expect(result.outsideTest).toEqual([
{
file: FILE,
suite: 'S',
outcome: 'killed-outside-test:hook',
attributedTo: [T0, T1],
message: `TypeError: ${typeError.message}`,
},
]);
expect(result.tests.map((row) => row.outcome)).toEqual([
'killed-outside-test:hook',
'killed-outside-test:hook',
]);
});
it("classifies Vitest's own testTimeout error as timeout", () => {
const result = classify(
observed([module([test(0, { state: 'failed', errors: [testTimeout] }), test(1)])]),
1,
);
expect(result).toMatchObject({ outcome: 'timeout', killedBy: [] });
});
it('classifies a beforeAll hookTimeout error as timeout without attribution', () => {
const skipped = { state: 'skipped' as const };
const result = classify(
observed([
module([test(0, skipped), test(1, skipped)], {
state: 'failed',
suites: [suite([0, 1], { failedHooks: ['beforeAll'], errors: [hookTimeout] })],
}),
]),
1,
);
expect(result).toMatchObject({ outcome: 'timeout', killedBy: [] });
expect(result.outsideTest[0]?.attributedTo).toEqual([]);
});
it('classifies an all-passing report as survived', () => {
expect(classify(observed([module([test(0), test(1)])]), 0)).toMatchObject({
outcome: 'survived',
reason: null,
killedBy: [],
executed: [T0, T1],
});
});
it('classifies NoCoverage as survived, read from mutation.json', () => {
expect(
classifyMutant({ strykerStatus: statusOf('NoCoverage'), record: null, expected: EXPECTED }),
).toMatchObject({ outcome: 'survived', source: 'no-coverage' });
});
});
describe('oracle classifier: further KTD17 rows and integrity checks', () => {
it('classifies an unhandled error as infrastructure', () => {
const run = observed([module([test(0), test(1)])], {
reason: 'failed',
unhandledErrors: [error('Error', 'socket hang up')],
});
expect(classify(run, 1)).toMatchObject({ outcome: 'infrastructure', killedBy: [] });
});
it('classifies an executed test with retry above zero as inconclusive', () => {
const result = classify(observed([module([test(0, { retry: 2 }), test(1)])]), 0);
expect(result.outcome).toBe('inconclusive');
expect(result.reason).toContain(`${T0} has retry 2`);
});
it('classifies a test set different from the baseline as invalid', () => {
const result = classify(observed([module([test(0), test(2)])]), 0);
expect(result.outcome).toBe('invalid');
expect(result.reason).toContain(`baseline test ${T1} did not run`);
expect(result.reason).toContain(`test ${FILE}#S > t2 ran but is not in the baseline`);
});
it('classifies an afterAll failure with passing tests as killed-outside-test, hook, unattributed', () => {
const result = classify(
observed([
module([test(0), test(1)], {
state: 'failed',
suites: [suite([0, 1], { failedHooks: ['afterAll'], errors: [typeError] })],
}),
]),
1,
);
expect(result).toMatchObject({
outcome: 'killed-outside-test:hook',
killedBy: [],
executed: [T0, T1],
});
});
it.each([
['a passing body and a failing afterEach', [typeError], 'killed-by-test:hook'],
[
'a failing body and a failing afterEach',
[assertionError, typeError],
'killed-by-test:assertion',
],
] as const)('classifies %s by the first failure', (_label, errors, expected) => {
const failed = test(0, { state: 'failed', errors: [...errors], failedHooks: ['afterEach'] });
expect(classify(observed([module([failed, test(1)])]), 1).outcome).toBe(expected);
});
it('prefers an assertion kill over a runtime kill in the same run', () => {
const result = classify(
observed([
module([
test(0, { state: 'failed', errors: [typeError] }),
test(1, { state: 'failed', errors: [assertionError] }),
]),
]),
1,
);
expect(result).toMatchObject({
outcome: 'killed-by-test:assertion',
killedBy: [T0, T1],
assertionKilledBy: [T1],
});
});
it.each([
['exit 0 with a failed test', 0, test(0, { state: 'failed', errors: [assertionError] })],
['a non-zero exit with every test passing', 1, test(0)],
] as const)('classifies %s as infrastructure', (_label, exitCode, first) => {
const run = observed([module([first, test(1)])]);
const result = classifyRun(run, { exitCode, expected: EXPECTED, vitestJson: totals(run) });
expect(result.outcome).toBe('infrastructure');
expect(result.reason).toContain(`Vitest exited ${exitCode}`);
});
it('classifies a disagreement with the Vitest JSON totals as infrastructure', () => {
const run = observed([module([test(0), test(1)])]);
const result = classifyRun(run, {
exitCode: 0,
expected: EXPECTED,
vitestJson: { numTotalTests: 2, numPassedTests: 1, numFailedTests: 1 },
});
expect(result.outcome).toBe('infrastructure');
expect(result.reason).toContain('Vitest JSON report');
});
it.each([
['Killed', 0],
['Survived', 1],
] as const)(
'classifies Stryker status %s with wrapper exit %i as infrastructure',
(strykerStatus, exitCode) => {
const run = observed([module([test(0), test(1)])]);
expect(
classifyMutant({ strykerStatus, record: record(run, exitCode), expected: EXPECTED })
.outcome,
).toBe('infrastructure');
},
);
it('classifies a record whose wrapper could not start Vitest as infrastructure', () => {
const run = observed([]);
const broken = record(run, 1, {
run: null,
vitestJson: null,
wrapperError: 'config failed to load',
});
expect(
classifyMutant({ strykerStatus: 'Killed', record: broken, expected: EXPECTED }),
).toMatchObject({
outcome: 'infrastructure',
reason: 'config failed to load',
});
});
it('classifies a mutant Stryker never ran as invalid', () => {
expect(
classifyMutant({ strykerStatus: 'Ignored', record: null, expected: EXPECTED }),
).toMatchObject({
outcome: 'invalid',
source: 'not-run',
});
});
});
describe('oracle test identity and record files', () => {
it('suffixes duplicate full names with the location, then the row at that location', () => {
const at = (line: number) => ({ line, column: 5 });
expect(
assignTestIds([
{ file: FILE, fullName: 'S > unique', location: at(1) },
{ file: FILE, fullName: 'S > dup', location: at(2) },
{ file: FILE, fullName: 'S > dup', location: at(3) },
{ file: FILE, fullName: 'S > row', location: at(4) },
{ file: FILE, fullName: 'S > row', location: at(4) },
{ file: FILE, fullName: 'S > other', location: at(4) },
{ file: 'test/unit/b.test.ts', fullName: 'S > unique', location: at(1) },
]),
).toEqual([
`${FILE}#S > unique`,
`${FILE}#S > dup:2:5`,
`${FILE}#S > dup:3:5`,
`${FILE}#S > row:4:5:0`,
`${FILE}#S > row:4:5:1`,
`${FILE}#S > other`,
'test/unit/b.test.ts#S > unique',
]);
});
it.each([
['dry-run', null, 1, 'dry-run.attempt-1.json'],
['mutant', '42', 3, 'mutant-42.attempt-3.json'],
] as const)('names a %s record and parses the name back', (kind, mutantId, attempt, name) => {
expect(recordFileName(kind, mutantId, attempt)).toBe(name);
expect(parseRecordFileName(name)).toEqual({ kind, mutantId, attempt });
});
it.each(['mutant-42.attempt-3.vitest.json', '.mutant-42.attempt-3.json.tmp', 'notes.txt'])(
'ignores the file name %s',
(name) => {
expect(parseRecordFileName(name)).toBeNull();
},
);
it('round-trips a record through JSON and rejects a malformed one', () => {
const original = record(observed([module([test(0), test(1)])]), 0);
expect(parseOracleRunRecord(JSON.stringify(original))).toEqual(original);
expect(() => parseOracleRunRecord(JSON.stringify({ ...original, schemaVersion: 2 }))).toThrow(
/schemaVersion/,
);
});
});
describe('oracle classes without a verdict', () => {
it('counts exactly infrastructure, inconclusive and invalid as unusable', () => {
expect(ORACLE_CLASSES.filter(isUnusableClass)).toEqual([
'infrastructure',
'inconclusive',
'invalid',
]);
});
});

493
gitnexus/test/unit/mutation/oracle-compare.test.ts Normal file
View file

@ -0,0 +1,493 @@
/**
* Locks gate 3's comparison rules (U7, KTD12, KTD13, KTD17, KTD18): freshness ("re-freeze
* required"), the inventory check that blocks before outcomes are compared, per-mutant classes
* and per-test outcomes under exactly the four KTD13 normalisations, and the gate's exit codes.
* Every input is synthetic: a frozen manifest and the parsed views of a runner run.
*/
import { describe, expect, it } from 'vitest';
import {
EXIT_DIFFERENCES,
EXIT_PASS,
EXIT_REFREEZE,
EXIT_USAGE,
checkFreshness,
compareScope,
gateExitCode,
runnerPerTest,
runnerTestClass,
type CurrentState,
type Freshness,
type RunnerPerTest,
type ScopeInput,
type SidecarView,
} from '../../../tools/mutation/oracle/compare.js';
import {
diffInventory,
normaliseTestId,
type CombinedStratum,
type OracleManifest,
type RunnerMutantView,
type RunnerRunView,
} from '../../../tools/mutation/oracle/freeze.js';
import type { OracleClass } from '../../../tools/mutation/oracle/oracle-classify.js';
const FILE = 'test/unit/a.test.ts';
const T0 = `${FILE}#S > one`;
const T1 = `${FILE}#S > two (#2415)`;
const M0 = 'gitnexus/src/a.ts@0123456789abcdef:3:1-3:9:StringLiteral:aaaaaaaaaaaaaaaa';
const M1 = 'gitnexus/src/a.ts@0123456789abcdef:5:1-5:9:ConditionalExpression:bbbbbbbbbbbbbbbb';
const M2 = 'gitnexus/src/a.ts@0123456789abcdef:7:1-7:9:BlockStatement:cccccccccccccccc';
const manifest = (overrides: Partial<OracleManifest> = {}): OracleManifest => ({
schemaVersion: 1,
kind: 'gitnexus-mutation-oracle-manifest',
name: 'a',
description: 'synthetic',
refreeze: 'npm run test:mutation:oracle:freeze -- --from-manifest a.json --seed <s> --work <w>',
scope: { testFiles: [FILE], mutate: ['gitnexus/src/a.ts'] },
versions: { stryker: '10.0.0', vitest: '5.0.5', nodeMajor: 22 },
operators: { mutator: null, names: ['BlockStatement', 'ConditionalExpression', 'StringLiteral'] },
inputs: {
closure: { 'gitnexus/src/a.ts': 'h-src', [`gitnexus/${FILE}`]: 'h-test' },
vitestConfig: { 'gitnexus/vitest.config.ts': 'h-config' },
mutationConfig: { 'gitnexus/tools/mutation/vitest.mutation.config.ts': 'h-mutation' },
prepareTool: { 'gitnexus/tools/mutation/prepare/prepare.ts': 'h-prepare' },
},
sample: { seed: 's', method: 'sha256', size: 100, topUpMin: 3, random: 2, topUps: [] },
strata: [],
baseline: { tests: 2, passedAlone: 2 },
tests: [T0, T1],
notOracleValidated: [],
mutants: [
[M0, 'killed-by-test:assertion'],
[M1, 'survived'],
[M2, 'survived'],
],
sampled: [
{
id: M0,
strata: ['multi-killer'],
covering: 2,
outcomes: [[1, 'killed-by-test:assertion']],
invalid: [],
},
{ id: M1, strata: ['survived'], covering: 1, outcomes: [], invalid: [] },
],
launches: { total: 6, valid: 4, invalid: 0, wallClockLimitMs: 180000 },
warnings: [],
problems: [],
...overrides,
});
const runnerMutant = (
globalId: string,
outcome: OracleClass | null,
status: string,
overrides: Partial<RunnerMutantView> = {},
): RunnerMutantView => ({
globalId,
localId: globalId.slice(-1),
status,
outcome,
outcomeSource: 'sidecar',
sidecarFile: `/sidecars/${globalId.slice(-1)}.json`,
...overrides,
});
const runner = (mutants: RunnerMutantView[]): RunnerRunView => ({
file: '/run/results.json',
scope: { testFiles: [FILE], mutate: ['gitnexus/src/a.ts'] },
runDir: '/run/run',
preparedRoot: '/run/prepared',
configFile: '/run/stryker.config.json',
mutants,
flags: [],
spanMapFailures: 0,
});
const sidecar = (tests: [string, OracleClass][]): SidecarView => ({
tests: tests.map(([id, outcome]) => ({ id, outcome })),
outsideTest: [],
});
const FRESH: Freshness = { fresh: true, mismatches: [], refreeze: 'refreeze' };
const AGREEING_RUNNER = runner([
runnerMutant(M0, 'killed-by-test:assertion', 'Killed'),
runnerMutant(M1, 'survived', 'Survived'),
runnerMutant(M2, 'survived', 'Survived'),
]);
const AGREEING_SIDECARS: Record<string, SidecarView> = {
[M0]: sidecar([
[T0, 'survived'],
[T1, 'killed-by-test:assertion'],
]),
[M1]: sidecar([[T0, 'survived']]),
};
const scopeInput = (overrides: Partial<ScopeInput> = {}): ScopeInput => ({
manifest: manifest(),
manifestFile: 'a.json',
freshness: FRESH,
staleness: [],
runner: AGREEING_RUNNER,
runnerTests: [T0, T1],
perTest: (mutant): RunnerPerTest =>
runnerPerTest(mutant, (file) => {
const found = Object.entries(AGREEING_SIDECARS).find(([id]) =>
file.endsWith(`${id.slice(-1)}.json`),
);
return found?.[1] ?? sidecar([]);
}),
...overrides,
});
describe('test id normalisation (KTD13: title escaping, path separators)', () => {
it.each([
['path separators', 'test\\unit\\a.test.ts#S > one', T0],
['an escaped regex metacharacter', `${FILE}#S > two \\(#2415\\)`, T1],
['an escaped quote', `${FILE}#it's \\"quoted\\"`, `${FILE}#it's "quoted"`],
['NFD title', `${FILE}#café`, `${FILE}#café`],
['an unchanged id', T0, T0],
])('normalises %s', (_label, raw, expected) => {
expect(normaliseTestId(raw)).toBe(normaliseTestId(expected));
});
it('keeps distinct titles distinct', () => {
expect(normaliseTestId(`${FILE}#a > b`)).not.toBe(normaliseTestId(`${FILE}#a > c`));
});
it('reports ids one inventory lacks and ids that collapse after normalisation', () => {
expect(
diffInventory([T0, T1, `${FILE}#x(`, `${FILE}#x\\(`], [T0, `${FILE}#y`], normaliseTestId),
).toEqual({
onlyA: [T1, `${FILE}#x(`].sort(),
onlyB: [`${FILE}#y`],
ambiguousA: [`${FILE}#x\\(`],
ambiguousB: [],
});
});
});
describe('gate 3 comparison', () => {
it('passes a runner run that agrees with the manifest', () => {
const report = compareScope(scopeInput());
expect({
result: report.result,
classes: report.classes?.differences,
perTest: [report.perTest?.compared, report.perTest?.agree, report.perTest?.differences],
}).toEqual({ result: 'pass', classes: [], perTest: [4, 4, []] });
});
it('blocks before outcomes when a test of the oracle inventory is missing from the run manifest', () => {
const report = compareScope(scopeInput({ runnerTests: [T0] }));
expect({
result: report.result,
blocked: report.inventory?.blocked,
onlyOracle: report.inventory?.tests.onlyA,
classes: report.classes,
perTest: report.perTest,
}).toEqual({
result: 'differences',
blocked: true,
onlyOracle: [T1],
classes: null,
perTest: null,
});
});
it('blocks when a global mutant id of run A is missing from run B', () => {
const report = compareScope(
scopeInput({ runner: runner(AGREEING_RUNNER.mutants.filter((m) => m.globalId !== M2)) }),
);
expect([report.result, report.inventory?.mutants.onlyA, report.classes]).toEqual([
'differences',
[M2],
null,
]);
});
it('blocks when run B has a global mutant id that run A lacks', () => {
const extra = `${M2.slice(0, -1)}d`;
const report = compareScope(
scopeInput({
runner: runner([...AGREEING_RUNNER.mutants, runnerMutant(extra, 'survived', 'Survived')]),
}),
);
expect([report.inventory?.blocked, report.inventory?.mutants.onlyB]).toEqual([true, [extra]]);
});
it('accepts a runner Timeout against an oracle wall-clock kill as one class', () => {
const timedOut = manifest({
mutants: [
[M0, 'timeout'],
[M1, 'survived'],
[M2, 'survived'],
],
sampled: [{ id: M0, strata: [], covering: 2, outcomes: [[1, 'timeout']], invalid: [] }],
});
const hitLimit = runner([
runnerMutant(M0, 'timeout', 'Timeout'),
...AGREEING_RUNNER.mutants.slice(1),
]);
const report = compareScope(
scopeInput({
manifest: timedOut,
runner: hitLimit,
perTest: () => ({
kind: 'record',
sidecar: sidecar([
[T0, 'survived'],
[T1, 'timeout'],
]),
}),
}),
);
expect({
result: report.result,
normalised: report.classes?.normalised.map((row) => [row.globalId, row.rule]),
perTest: report.perTest?.differences,
}).toEqual({ result: 'pass', normalised: [[M0, 'timeout']], perTest: [] });
});
it("lists the per-test rows of a mutant Stryker's timer killed as unsupported", () => {
const timedOut = manifest({
mutants: [
[M0, 'timeout'],
[M1, 'survived'],
[M2, 'survived'],
],
sampled: [{ id: M0, strata: [], covering: 2, outcomes: [[1, 'timeout']], invalid: [] }],
});
const strykerTimer = runner([
runnerMutant(M0, 'timeout', 'Timeout', {
outcomeSource: 'stryker-timeout',
sidecarFile: null,
}),
...AGREEING_RUNNER.mutants.slice(1),
]);
const report = compareScope(scopeInput({ manifest: timedOut, runner: strykerTimer }));
expect([report.result, report.perTest?.unsupported, report.perTest?.compared]).toEqual([
'pass',
[{ globalId: M0, reason: "Stryker's timer killed the run; no record", observations: 2 }],
0,
]);
});
it('accepts runner NoCoverage against oracle survived and keeps the raw status', () => {
const noCoverage = runner([
AGREEING_RUNNER.mutants[0],
runnerMutant(M1, 'survived', 'NoCoverage', {
outcomeSource: 'no-coverage',
sidecarFile: null,
}),
AGREEING_RUNNER.mutants[2],
]);
const report = compareScope(scopeInput({ runner: noCoverage }));
expect({
result: report.result,
normalised: report.classes?.normalised,
notRunAsSurvived: report.perTest?.notRunAsSurvived,
}).toEqual({
result: 'pass',
normalised: [
{
globalId: M1,
oracle: 'survived',
runner: 'survived',
runnerStatus: 'NoCoverage',
runnerSource: 'no-coverage',
rule: 'no-coverage',
},
],
notRunAsSurvived: 2,
});
});
it('blocks a runner assertion kill against an oracle runtime kill, even with a recorded cause', () => {
const runtime = manifest({
mutants: [
[M0, 'killed-by-test:runtime'],
[M1, 'survived'],
[M2, 'survived'],
],
sampled: [
{ id: M0, strata: [], covering: 2, outcomes: [[1, 'killed-by-test:runtime']], invalid: [] },
],
warnings: ['known cause: the mutant throws a TypeError inside expect()'],
});
const report = compareScope(scopeInput({ manifest: runtime }));
expect({
result: report.result,
classes: report.classes?.differences.map((row) => [row.globalId, row.oracle, row.runner]),
perTest: report.perTest?.differences.map((row) => [row.testId, row.oracle, row.runner]),
exit: gateExitCode([report], []),
}).toEqual({
result: 'differences',
classes: [[M0, 'killed-by-test:runtime', 'killed-by-test:assertion']],
perTest: [[T1, 'killed-by-test:runtime', 'killed-by-test:assertion']],
exit: EXIT_DIFFERENCES,
});
});
it('blocks a runner run with flags', () => {
const flagged = { ...AGREEING_RUNNER, flags: ['status-mismatch: Killed vs survived'] };
expect(compareScope(scopeInput({ runner: flagged })).result).toBe('differences');
});
it('blocks a sampled mutant the runner left without a record', () => {
const missing = runner([
runnerMutant(M0, 'killed-by-test:assertion', 'Killed', { sidecarFile: null }),
...AGREEING_RUNNER.mutants.slice(1),
]);
const report = compareScope(scopeInput({ runner: missing }));
expect(report.perTest?.differences.map((row) => [row.testId, row.runnerSource])).toEqual([
[T0, 'missing'],
[T1, 'missing'],
]);
});
it('skips per-test outcomes of tests that are not oracle-validated', () => {
const marked = manifest({ notOracleValidated: [T1] });
const killedElsewhere: Record<string, SidecarView> = {
[M0]: sidecar([[T1, 'killed-by-test:runtime']]),
};
const report = compareScope(
scopeInput({
manifest: marked,
perTest: (mutant) => ({
kind: 'record',
sidecar: killedElsewhere[mutant.globalId] ?? sidecar([]),
}),
}),
);
expect([report.result, report.perTest?.compared, report.perTest?.notOracleValidated]).toEqual([
'pass',
2,
2,
]);
});
});
describe("the runner's per-test class", () => {
const record: SidecarView = {
tests: [{ id: `${FILE}#ran`, outcome: 'killed-by-test:hook' }],
outsideTest: [
{
file: FILE,
suiteTaskId: 'suite-1',
outcome: 'killed-outside-test:hook',
attributedTo: [`${FILE}#skipped by beforeAll`],
},
{
file: 'test/unit/b.test.ts',
suiteTaskId: null,
outcome: 'killed-outside-test:collection',
attributedTo: [],
},
],
};
it.each([
['its row', `${FILE}#ran`, { outcome: 'killed-by-test:hook', source: 'row' }],
[
'an attributing beforeAll row',
`${FILE}#skipped by beforeAll`,
{ outcome: 'killed-outside-test:hook', source: 'attributed' },
],
[
'a collection failure of its file',
'test/unit/b.test.ts#anything',
{ outcome: 'killed-outside-test:collection', source: 'collection' },
],
[
'a test the runner did not run',
`${FILE}#not run`,
{ outcome: 'survived', source: 'not-run' },
],
])('comes from %s', (_label, testId, expected) => {
expect(runnerTestClass(record, testId)).toEqual(expected);
});
});
describe('freshness (KTD12)', () => {
const current = (overrides: Partial<CurrentState> = {}): CurrentState => ({
inputs: manifest().inputs,
versions: manifest().versions,
mutator: null,
...overrides,
});
it('is fresh when hashes, versions and operators match', () => {
expect(checkFreshness(manifest(), current())).toEqual({
fresh: true,
mismatches: [],
refreeze: manifest().refreeze,
});
});
it('requires a re-freeze when a closure hash or a version changed', () => {
const changed = current({
inputs: {
...manifest().inputs,
closure: { ...manifest().inputs.closure, 'gitnexus/src/a.ts': 'h-new' },
},
versions: { ...manifest().versions, vitest: '5.0.6' },
});
const freshness = checkFreshness(manifest(), changed);
const report = compareScope(scopeInput({ freshness }));
expect({
mismatches: freshness.mismatches.map((entry) => [entry.kind, entry.subject]),
result: report.result,
compared: [report.inventory, report.classes, report.perTest],
exit: gateExitCode([report], []),
}).toEqual({
mismatches: [
['input', 'closure: gitnexus/src/a.ts'],
['version', 'vitest'],
],
result: 'refreeze-required',
compared: [null, null, null],
exit: EXIT_REFREEZE,
});
});
it('requires a re-freeze when a closure file appears or the operator set changes', () => {
const changed = current({
inputs: {
...manifest().inputs,
closure: { ...manifest().inputs.closure, 'gitnexus/src/new.ts': 'h' },
},
mutator: { excludedMutations: ['StringLiteral'] },
});
expect(checkFreshness(manifest(), changed).mismatches.map((entry) => entry.kind)).toEqual([
'input',
'operators',
]);
});
});
describe('gate exit codes', () => {
const pass = compareScope(scopeInput());
const stale = compareScope(scopeInput({ staleness: ['the runner run used Vitest 5.0.4'] }));
const unmet: CombinedStratum = {
name: 'no-coverage',
population: 3,
sampled: 0,
status: 'unmet',
scopes: [],
};
it.each([
['every scope passes', [pass], [], EXIT_PASS],
['a stale runner run', [pass, stale], [], EXIT_USAGE],
['an unmet stratum across the scopes', [pass], [unmet], EXIT_DIFFERENCES],
] as const)('%s', (_label, scopes, strata, expected) => {
expect(gateExitCode(scopes, strata)).toBe(expected);
});
it('compares nothing for a stale runner run', () => {
expect([stale.result, stale.inventory]).toEqual(['stale-runner-run', null]);
});
});

584
gitnexus/test/unit/mutation/oracle-sample.test.ts Normal file
View file

@ -0,0 +1,584 @@
/**
* Locks the oracle freeze's sampling and strata rules (U7, KTD11, KTD12): the seeded sample and
* its top-ups, the per-scope strata requirements, strata counted across scopes, the strata
* populations from baseline coverage, the resumable launch work dir and the manifest format.
* Launches are synthetic: the injected launcher only writes launch.json files into a temporary
* work dir; no process is spawned.
*/
import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { afterAll, describe, expect, it } from 'vitest';
import {
combineStrata,
computePopulations,
eachRowTests,
evaluateStrata,
formatJson,
launchKey,
ordinalTests,
parseManifest,
runLaunches,
sampleRank,
selectSample,
type OracleManifest,
type PlannedLaunch,
type Populations,
type StrataInput,
type StratumName,
type StratumRecord,
} from '../../../tools/mutation/oracle/freeze.js';
import type { LaunchOptions, LaunchTarget } from '../../../tools/mutation/oracle/launch-single.js';
const FILE = 'test/unit/a.test.ts';
const MUTANTS = Array.from(
{ length: 40 },
(_, i) =>
`gitnexus/src/a.ts@0123456789abcdef:${i + 1}:1-${i + 1}:9:StringLiteral:fedcba9876543210`,
);
const byRank =
(seed: string) =>
(a: string, b: string): number =>
sampleRank(seed, a).localeCompare(sampleRank(seed, b));
const populations = (entries: Partial<Record<StratumName, readonly string[]>>): Populations =>
new Map(Object.entries(entries).map(([name, ids]) => [name as StratumName, new Set(ids)]));
const target = (name: string, line: number, column = 3, file = FILE): LaunchTarget => ({
id: `${file}#${name}`,
file,
fullName: name,
location: { line, column },
});
describe('seeded sample (U7 sampling)', () => {
const sample = (seed: string, mutants: readonly string[] = MUTANTS): string[] =>
selectSample({ mutants, seed, sampleSize: 10, topUpMin: 0, populations: new Map() }).selected;
it('gives the same sample for the same seed', () => {
expect(sample('seed-1')).toEqual(sample('seed-1'));
});
it('gives a different sample for a different seed', () => {
expect(sample('seed-2')).not.toEqual(sample('seed-1'));
});
it('does not depend on the input order', () => {
expect(sample('seed-1', [...MUTANTS].reverse())).toEqual(sample('seed-1'));
});
it('takes the lowest ranks of SHA-256(seed + newline + global id)', () => {
expect(sample('seed-1')).toEqual([...MUTANTS].sort(byRank('seed-1')).slice(0, 10));
});
it('takes every mutant when the scope has fewer than the sample size', () => {
const few = MUTANTS.slice(0, 4);
expect(sample('seed-1', few).sort()).toEqual([...few].sort());
});
it('tops a stratum up with its lowest-ranked members outside the random sample', () => {
const lastThree = [...MUTANTS].sort(byRank('s')).slice(-3);
const selection = selectSample({
mutants: MUTANTS,
seed: 's',
sampleSize: 10,
topUpMin: 2,
populations: populations({ static: lastThree }),
});
expect(selection.topUps).toEqual([
{ globalId: lastThree[0], stratum: 'static' },
{ globalId: lastThree[1], stratum: 'static' },
]);
expect(selection.selected).toHaveLength(12);
});
it('adds no top-ups with --top-up-min 0', () => {
const lastThree = [...MUTANTS].sort(byRank('s')).slice(-3);
const selection = selectSample({
mutants: MUTANTS,
seed: 's',
sampleSize: 10,
topUpMin: 0,
populations: populations({ static: lastThree }),
});
expect(selection.topUps).toEqual([]);
});
it('counts random members toward a stratum quota', () => {
const firstTwo = [...MUTANTS].sort(byRank('s')).slice(0, 2);
const selection = selectSample({
mutants: MUTANTS,
seed: 's',
sampleSize: 10,
topUpMin: 2,
populations: populations({ survived: firstTwo }),
});
expect(selection.topUps).toEqual([]);
});
});
const strataInput = (overrides: Partial<StrataInput>): StrataInput => ({
populations: populations({}),
selected: [],
topUps: [],
constructs: { beforeAll: false, each: false },
affectedTests: { beforeAll: [], each: [] },
notOracleValidated: new Set(),
...overrides,
});
const statusOf = (records: readonly StratumRecord[], name: StratumName): string | undefined =>
records.find((record) => record.name === name)?.status;
describe('per-scope strata (KTD11)', () => {
it('freezes a scope without hooks or it.each without a beforeAll or it.each stratum', () => {
const records = evaluateStrata(
strataInput({
populations: populations({ static: [MUTANTS[0]], survived: [MUTANTS[1]] }),
selected: [MUTANTS[0]],
}),
);
expect(records.map((record) => [record.name, record.status, record.required])).toEqual([
['static', 'combined', false],
['hook-hit', 'construct-absent', false],
['multi-killer', 'empty-population', false],
['survived', 'combined', false],
['no-coverage', 'empty-population', false],
['each-row', 'construct-absent', false],
]);
});
it('fails a scope with beforeAll and a hook-hit population whose sample has none', () => {
const records = evaluateStrata(
strataInput({
populations: populations({ 'hook-hit': [MUTANTS[2]] }),
selected: [MUTANTS[0]],
constructs: { beforeAll: true, each: false },
affectedTests: { beforeAll: [`${FILE}#a`, `${FILE}#b`], each: [] },
}),
);
expect(records.find((record) => record.name === 'hook-hit')).toMatchObject({
required: true,
status: 'unmet',
population: 1,
sampled: 0,
affectedTests: 2,
});
});
it('waives that stratum when every affected test is marked not oracle-validated', () => {
const records = evaluateStrata(
strataInput({
populations: populations({ 'hook-hit': [MUTANTS[2]] }),
constructs: { beforeAll: true, each: false },
affectedTests: { beforeAll: [`${FILE}#a`, `${FILE}#b`], each: [] },
notOracleValidated: new Set([`${FILE}#a`, `${FILE}#b`]),
}),
);
expect(statusOf(records, 'hook-hit')).toBe('waived');
});
it('keeps it unmet when only some affected tests are marked', () => {
const records = evaluateStrata(
strataInput({
populations: populations({ 'hook-hit': [MUTANTS[2]] }),
constructs: { beforeAll: true, each: false },
affectedTests: { beforeAll: [`${FILE}#a`, `${FILE}#b`], each: [] },
notOracleValidated: new Set([`${FILE}#a`]),
}),
);
expect(statusOf(records, 'hook-hit')).toBe('unmet');
});
it('records an empty population of a present construct without failing', () => {
const records = evaluateStrata(
strataInput({
constructs: { beforeAll: true, each: true },
affectedTests: { beforeAll: [`${FILE}#a`], each: [`${FILE}#r1`, `${FILE}#r2`] },
}),
);
expect(
records
.filter((record) => record.construct !== null)
.map((record) => [record.name, record.status, record.required, record.population]),
).toEqual([
['hook-hit', 'empty-population', false, 0],
['each-row', 'empty-population', false, 0],
]);
});
it('meets an it.each stratum with one sampled row mutant and counts its top-ups', () => {
const records = evaluateStrata(
strataInput({
populations: populations({ 'each-row': [MUTANTS[3], MUTANTS[4]] }),
selected: [MUTANTS[0], MUTANTS[3]],
topUps: [{ globalId: MUTANTS[3], stratum: 'each-row' }],
constructs: { beforeAll: false, each: true },
affectedTests: { beforeAll: [], each: [`${FILE}#r1`, `${FILE}#r2`] },
}),
);
expect(records.find((record) => record.name === 'each-row')).toMatchObject({
status: 'met',
required: true,
sampled: 1,
toppedUp: 1,
});
});
});
const record = (
name: StratumName,
status: StratumRecord['status'],
population: number,
sampled: number,
): StratumRecord => ({
name,
construct: null,
constructPresent: null,
population,
sampled,
toppedUp: 0,
required: status === 'met' || status === 'unmet' || status === 'waived',
status,
affectedTests: 0,
});
describe('strata across the scopes combined (KTD11)', () => {
const scopes = [
{
name: 'shared',
strata: [
record('static', 'combined', 245, 20),
record('hook-hit', 'construct-absent', 0, 0),
record('multi-killer', 'combined', 90, 0),
record('each-row', 'construct-absent', 0, 0),
],
},
{
name: 'schema-constants',
strata: [
record('static', 'combined', 72, 72),
record('hook-hit', 'construct-absent', 0, 0),
record('multi-killer', 'combined', 3, 2),
record('each-row', 'empty-population', 0, 0),
],
},
{
name: 'middleware',
strata: [
record('static', 'combined', 3, 3),
record('hook-hit', 'empty-population', 0, 0),
record('multi-killer', 'empty-population', 0, 0),
record('each-row', 'met', 12, 4),
],
},
];
const combined = combineStrata(scopes);
const byName = (name: StratumName) => combined.find((entry) => entry.name === name);
it('sums populations and sampled members over the three scopes', () => {
expect(byName('static')).toMatchObject({ population: 320, sampled: 95, status: 'met' });
expect(byName('multi-killer')).toMatchObject({ population: 93, sampled: 2, status: 'met' });
});
it('records a stratum whose population is empty everywhere', () => {
expect(byName('hook-hit')).toMatchObject({
population: 0,
sampled: 0,
status: 'empty-population',
});
});
it('meets a construct stratum one scope samples', () => {
expect(byName('each-row')).toMatchObject({ population: 12, sampled: 4, status: 'met' });
});
it('lists every scope of a stratum', () => {
expect(byName('static')?.scopes.map((entry) => entry.scope)).toEqual([
'shared',
'schema-constants',
'middleware',
]);
});
it('fails a stratum with a population but no sampled member anywhere', () => {
const unmet = combineStrata([
{ name: 'a', strata: [record('no-coverage', 'combined', 5, 0)] },
{ name: 'b', strata: [record('no-coverage', 'empty-population', 0, 0)] },
]);
expect(unmet.find((entry) => entry.name === 'no-coverage')?.status).toBe('unmet');
});
it('fails a stratum one scope leaves unmet even when another samples it', () => {
const unmet = combineStrata([
{ name: 'a', strata: [record('hook-hit', 'unmet', 2, 0)] },
{ name: 'b', strata: [record('hook-hit', 'met', 3, 1)] },
]);
expect(unmet.find((entry) => entry.name === 'hook-hit')?.status).toBe('unmet');
});
});
describe('strata populations from baseline coverage', () => {
const ids = ['m0', 'm1', 'm2', 'm3', 'm4', 'm5'].map((id) => `gitnexus/src/a.ts@x:${id}`);
const mutants = [
{ globalId: ids[0], localId: '0', oracleClass: 'survived' as const, killedBy: [] },
{
globalId: ids[1],
localId: '1',
oracleClass: 'killed-by-test:assertion' as const,
killedBy: [`${FILE}#a`, `${FILE}#b`],
},
{
globalId: ids[2],
localId: '2',
oracleClass: 'killed-by-test:runtime' as const,
killedBy: [`${FILE}#a`],
},
{ globalId: ids[3], localId: '3', oracleClass: 'survived' as const, killedBy: [] },
{ globalId: ids[4], localId: '4', oracleClass: 'survived' as const, killedBy: [] },
{ globalId: ids[5], localId: '5', oracleClass: 'timeout' as const, killedBy: [] },
];
const baseline = [
{ testId: `${FILE}#a`, coverage: { loadAndCollection: ['0'], hook: [], test: ['1', '2'] } },
{ testId: `${FILE}#r1`, coverage: { loadAndCollection: ['0'], hook: ['3'], test: ['4'] } },
];
const result = computePopulations(mutants, baseline, new Set([`${FILE}#r1`]));
const members = (name: StratumName): string[] => [...(result.get(name) ?? [])].sort();
it.each([
['static', [ids[0]]],
['hook-hit', [ids[3]]],
['multi-killer', [ids[1]]],
['survived', [ids[0], ids[3], ids[4]]],
['no-coverage', [ids[5]]],
['each-row', [ids[4]]],
] as const)('%s', (name, expected) => {
expect(members(name)).toEqual([...expected].sort());
});
});
describe('test inventory shapes (KTD3)', () => {
const inventory = [
target('S > one', 10),
target('row 1', 20),
target('row 2', 20),
{ ...target('dup', 30), id: `${FILE}#dup:30:3:0` },
{ ...target('dup', 30), id: `${FILE}#dup:30:3:1` },
{ ...target('title ends like 1:2:3', 40), id: `${FILE}#title ends like 1:2:3` },
];
it('finds it.each rows by their shared file and location', () => {
expect(eachRowTests(inventory)).toEqual([
`${FILE}#row 1`,
`${FILE}#row 2`,
`${FILE}#dup:30:3:0`,
`${FILE}#dup:30:3:1`,
]);
});
it('finds the tests whose id needed the row ordinal', () => {
expect(ordinalTests(inventory)).toEqual([`${FILE}#dup:30:3:0`, `${FILE}#dup:30:3:1`]);
});
});
describe('resumable launches (U7 work dir)', () => {
const root = mkdtempSync(path.join(os.tmpdir(), 'oracle-freeze-test-'));
afterAll(() => rmSync(root, { recursive: true, force: true }));
const plans: PlannedLaunch[] = ['a', 'b', 'c', 'd'].map((name, i) => ({
key: launchKey('7', `${FILE}#${name}`),
target: target(name, 10 + i),
mutantId: '7',
}));
const fakeLaunch =
(calls: string[], validity: 'valid' | 'invalid' = 'valid') =>
async (options: LaunchOptions): Promise<void> => {
calls.push(path.relative(root, options.launchDir));
mkdirSync(options.launchDir, { recursive: true });
writeFileSync(
path.join(options.launchDir, 'launch.json'),
JSON.stringify({
target: { id: options.target.id },
mutantId: options.mutantId,
observation: { validity, outcome: 'survived', reason: null, covering: false },
}),
);
};
const common = {
sandboxDir: '/nowhere',
scopeFile: '/nowhere/scope.json',
wallClockMs: 1000,
concurrency: 1,
};
it('stops on abort, then skips the done launches and redoes an unfinished one', async () => {
const launchRoot = path.join(root, 'resume');
const controller = new AbortController();
const firstCalls: string[] = [];
const writeLaunch = fakeLaunch(firstCalls);
// The first launch finishes, then the freeze is interrupted.
const first = await runLaunches(plans, {
...common,
launchRoot,
retryInvalid: false,
signal: controller.signal,
launch: async (options) => {
await writeLaunch(options);
controller.abort();
},
});
// A launch killed mid-way leaves an attempt dir without launch.json.
mkdirSync(path.join(launchRoot, plans[2].key, 'attempt-1'), { recursive: true });
writeFileSync(path.join(launchRoot, plans[2].key, 'attempt-1', 'request.json'), '{}');
const secondCalls: string[] = [];
const second = await runLaunches(plans, {
...common,
launchRoot,
retryInvalid: false,
launch: fakeLaunch(secondCalls),
});
expect({
first: { aborted: first.aborted, launched: first.launched, calls: firstCalls },
second: { aborted: second.aborted, launched: second.launched, reused: second.reused },
secondCalls,
done: second.completed.size,
leftover: existsSync(path.join(launchRoot, plans[2].key, 'attempt-1', 'request.json')),
}).toEqual({
first: {
aborted: true,
launched: 1,
calls: [path.join('resume', plans[0].key, 'attempt-1')],
},
second: { aborted: false, launched: 3, reused: 1 },
secondCalls: [1, 2, 3].map((i) => path.join('resume', plans[i].key, 'attempt-1')),
done: 4,
leftover: false,
});
});
it('gives a done invalid launch a new attempt only with --retry-invalid', async () => {
const launchRoot = path.join(root, 'retry');
const one = plans.slice(0, 1);
await runLaunches(one, {
...common,
launchRoot,
retryInvalid: false,
launch: fakeLaunch([], 'invalid'),
});
const skipped: string[] = [];
await runLaunches(one, {
...common,
launchRoot,
retryInvalid: false,
launch: fakeLaunch(skipped),
});
const retried: string[] = [];
const result = await runLaunches(one, {
...common,
launchRoot,
retryInvalid: true,
launch: fakeLaunch(retried),
});
expect({
skipped,
retried,
attempts: readdirSync(path.join(launchRoot, one[0].key)).sort(),
validity: result.completed.get(one[0].key)?.validity,
}).toEqual({
skipped: [],
retried: [path.join('retry', one[0].key, 'attempt-2')],
attempts: ['attempt-1', 'attempt-2'],
validity: 'valid',
});
});
it('refuses a work dir whose launch holds another test', async () => {
const launchRoot = path.join(root, 'foreign');
const dir = path.join(launchRoot, plans[0].key, 'attempt-1');
mkdirSync(dir, { recursive: true });
writeFileSync(
path.join(dir, 'launch.json'),
JSON.stringify({
target: { id: `${FILE}#other` },
mutantId: '7',
observation: { validity: 'valid', outcome: 'survived', reason: null, covering: null },
}),
);
await expect(
runLaunches(plans.slice(0, 1), {
...common,
launchRoot,
retryInvalid: false,
launch: fakeLaunch([]),
}),
).rejects.toThrow(/use a new work dir/);
});
});
describe('manifest format (KTD12)', () => {
const manifest: OracleManifest = {
schemaVersion: 1,
kind: 'gitnexus-mutation-oracle-manifest',
name: 'tiny',
description: 'a tiny manifest',
refreeze: 'npm run test:mutation:oracle:freeze -- --from-manifest x --seed <s> --work <w>',
scope: { testFiles: [FILE], mutate: ['gitnexus/src/a.ts'] },
versions: { stryker: '10.0.0', vitest: '5.0.5', nodeMajor: 22 },
operators: { mutator: null, names: ['StringLiteral'] },
inputs: {
closure: { 'gitnexus/src/a.ts': 'aa', [`gitnexus/${FILE}`]: 'bb' },
vitestConfig: { 'gitnexus/vitest.config.ts': 'cc' },
mutationConfig: { 'gitnexus-shared/package.json': 'dd' },
prepareTool: { 'gitnexus/tools/mutation/prepare/prepare.ts': 'ee' },
},
sample: {
seed: 's',
method: 'sha256',
size: 100,
topUpMin: 3,
random: 2,
topUps: [{ globalId: MUTANTS[2], stratum: 'static' }],
},
strata: [record('static', 'combined', 3, 1)],
baseline: { tests: 2, passedAlone: 2 },
tests: [`${FILE}#a`, `${FILE}#b`],
notOracleValidated: [],
mutants: [
[MUTANTS[0], 'survived'],
[MUTANTS[1], 'killed-by-test:assertion'],
],
sampled: [
{
id: MUTANTS[1],
strata: ['multi-killer'],
covering: 2,
outcomes: [[1, 'killed-by-test:assertion']],
invalid: [],
},
],
launches: { total: 4, valid: 2, invalid: 0, wallClockLimitMs: 180000 },
warnings: [],
problems: [],
};
it('round-trips through its JSON format', () => {
expect(parseManifest(formatJson(manifest))).toEqual(manifest);
});
it('keeps each mutant row on one line', () => {
expect(formatJson(manifest).split('\n')).toContain(
` ${JSON.stringify([MUTANTS[0], 'survived'])},`,
);
});
it('rejects an outcome row that names a test outside the inventory', () => {
const broken = {
...manifest,
sampled: [{ ...manifest.sampled[0], outcomes: [[5, 'killed-by-test:assertion']] }],
};
expect(() => parseManifest(JSON.stringify(broken))).toThrow(/no test 5/);
});
});

39
gitnexus/test/unit/mutation/oracle-util.test.ts Normal file
View file

@ -0,0 +1,39 @@
/**
* Locks the oracle's shared helpers (oracle/util.ts): the object guard of its JSON readers and the
* temp-file-and-rename write that freeze.ts, launch-single.ts, command-wrapper.ts and
* oracle-setup.ts use for manifests, launch files, wrapper records and probe files.
*/
import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { isObject, writeFileAtomicSync } from '../../../tools/mutation/oracle/util.js';
describe('isObject', () => {
it.each([
{ label: 'a plain object', value: {}, expected: true },
{ label: 'null', value: null, expected: false },
{ label: 'an array', value: [], expected: false },
{ label: 'a string', value: 'x', expected: false },
])('$label is an object: $expected', ({ value, expected }) => {
expect(isObject(value)).toBe(expected);
});
});
describe('writeFileAtomicSync', () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-oracle-util-'));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it('replaces the file and leaves no temp file beside it', () => {
const file = path.join(dir, 'probe.json');
writeFileSync(file, 'old\n');
writeFileAtomicSync(file, '{"complete":true}\n');
expect(readFileSync(file, 'utf8')).toBe('{"complete":true}\n');
expect(readdirSync(dir)).toEqual(['probe.json']);
});
});

210
gitnexus/test/unit/mutation/path-map.test.ts Normal file
View file

@ -0,0 +1,210 @@
/**
* Locks the prepared-copy path map and the `gitnexus-shared` aliases (KTD8): relocated
* shared sources translate back to `gitnexus-shared/src/...`, `gitnexus/src/...` is the
* identity, anything else is rejected, and one alias exists per package `exports` key.
*/
import { readFileSync } from 'node:fs';
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import {
PATH_MAP,
buildPathMap,
deriveSharedAliases,
parseSharedPackageJson,
toPreparedPath,
toRepoPath,
toRepoPathFromPackageDir,
} from '../../../tools/mutation/prepare/path-map.js';
const RELOCATED_SCHEMA = 'gitnexus/relocated/gitnexus-shared/src/lbug/schema-constants.ts';
const SHARED_DIR = path.join(path.sep, 'prepared', 'gitnexus', 'relocated', 'gitnexus-shared');
describe('toRepoPath', () => {
it('translates a relocated shared path back to gitnexus-shared/src', () => {
expect(toRepoPath(RELOCATED_SCHEMA)).toBe('gitnexus-shared/src/lbug/schema-constants.ts');
});
it('leaves a gitnexus/src path unchanged', () => {
expect(toRepoPath('gitnexus/src/core/ingestion/type-extractors/shared.ts')).toBe(
'gitnexus/src/core/ingestion/type-extractors/shared.ts',
);
});
it('accepts Windows separators and a leading ./', () => {
expect(toRepoPath('.\\gitnexus\\relocated\\gitnexus-shared\\src\\index.ts')).toBe(
'gitnexus-shared/src/index.ts',
);
});
it.each([
['a test file', 'gitnexus/test/unit/schema.test.ts'],
['an unrelocated shared path', 'gitnexus-shared/src/index.ts'],
['a parent-relative path', '../gitnexus/src/index.ts'],
['a path that climbs out of src', 'gitnexus/src/../../etc/passwd'],
['an absolute path', '/abs/gitnexus/src/index.ts'],
['the src root itself', 'gitnexus/src'],
])('rejects %s', (_label, input) => {
expect(() => toRepoPath(input)).toThrow(/outside the mapped roots|not a relative path/);
});
});
describe('toPreparedPath', () => {
it('maps gitnexus-shared/src into the relocated directory', () => {
expect(toPreparedPath('gitnexus-shared/src/lbug/schema-constants.ts')).toBe(RELOCATED_SCHEMA);
});
it('round-trips a gitnexus/src path', () => {
expect(toRepoPath(toPreparedPath('gitnexus/src/core/lbug/schema.ts'))).toBe(
'gitnexus/src/core/lbug/schema.ts',
);
});
it('rejects a path outside both roots', () => {
expect(() => toPreparedPath('gitnexus/test/unit/schema.test.ts')).toThrow(
/outside the mapped roots/,
);
});
});
describe('toRepoPathFromPackageDir', () => {
const sandbox = path.join(path.sep, 'tmp', 'sandbox-1');
it('maps an absolute relocated path under a sandbox back to gitnexus-shared/src', () => {
expect(
toRepoPathFromPackageDir(
sandbox,
path.join(sandbox, 'relocated', 'gitnexus-shared', 'src', 'lbug', 'schema-constants.ts'),
),
).toBe('gitnexus-shared/src/lbug/schema-constants.ts');
});
it('rejects an absolute path outside the package directory', () => {
expect(() =>
toRepoPathFromPackageDir(sandbox, path.join(path.sep, 'tmp', 'other', 'src', 'a.ts')),
).toThrow(/outside the mapped roots|not a relative path/);
});
});
describe('buildPathMap: extra roots for mutation fixture projects', () => {
const FIXTURE = 'gitnexus/test/fixtures/mutation-conformance';
const FIXTURE_FILE = `${FIXTURE}/src/static-table.ts`;
const map = buildPathMap([FIXTURE]);
it('is the default map when no extra root is given', () => {
expect(buildPathMap()).toEqual(PATH_MAP);
});
it('maps a file under an extra root to itself, in both directions', () => {
expect([toPreparedPath(FIXTURE_FILE, map), toRepoPath(FIXTURE_FILE, map)]).toEqual([
FIXTURE_FILE,
FIXTURE_FILE,
]);
});
it('maps an absolute sandbox path under an extra root back to its repo path', () => {
const sandbox = path.join(path.sep, 'tmp', 'sandbox-2');
const absolute = path.join(sandbox, 'test', 'fixtures', 'mutation-conformance', 'src', 'a.ts');
expect(toRepoPathFromPackageDir(sandbox, absolute, map)).toBe(`${FIXTURE}/src/a.ts`);
});
it('keeps the default roots in an extended map', () => {
expect(toPreparedPath('gitnexus-shared/src/lbug/schema-constants.ts', map)).toBe(
RELOCATED_SCHEMA,
);
});
it('still rejects the fixture path under the default map', () => {
expect(() => toPreparedPath(FIXTURE_FILE)).toThrow(/outside the mapped roots/);
});
it('still rejects paths outside every root under an extended map', () => {
expect(() => toPreparedPath('gitnexus/test/unit/schema.test.ts', map)).toThrow(
/outside the mapped roots/,
);
});
it('accepts a trailing slash, Windows separators and a repeated root once', () => {
expect(buildPathMap([`${FIXTURE}/`, 'gitnexus\\test\\fixtures\\mutation-conformance'])).toEqual(
map,
);
});
it.each([
['another test directory', 'gitnexus/test/unit'],
['a fixture that is not a mutation fixture', 'gitnexus/test/fixtures/lang-resolution'],
['a file inside a mutation fixture', 'gitnexus/test/fixtures/mutation-x/src'],
['the fixtures directory itself', 'gitnexus/test/fixtures'],
['a production root', 'gitnexus/src'],
['a parent-relative path', '../gitnexus/test/fixtures/mutation-x'],
['an absolute path', '/gitnexus/test/fixtures/mutation-x'],
])('rejects %s as an extra root', (_label, root) => {
expect(() => buildPathMap([root])).toThrow(/path map/);
});
});
describe('deriveSharedAliases', () => {
it('creates exactly one alias per key of the gitnexus-shared exports map', () => {
const text = readFileSync(
new URL('../../../../gitnexus-shared/package.json', import.meta.url),
'utf8',
);
const { exports: exportsMap }: { exports: Record<string, unknown> } = JSON.parse(text);
const aliases = deriveSharedAliases(parseSharedPackageJson(text), SHARED_DIR);
expect(aliases.map((a) => a.specifier)).toEqual(
// '.' -> 'gitnexus-shared', './test-helpers' -> 'gitnexus-shared/test-helpers'
Object.keys(exportsMap).map((key) => `gitnexus-shared${key.slice(1)}`),
);
});
it('points each alias at the matching source file under the relocated src', () => {
const aliases = deriveSharedAliases(
{
name: 'gitnexus-shared',
exports: {
'.': { types: './dist/index.d.ts', default: './dist/index.js' },
'./test-helpers': {
types: './dist/test-helpers.d.ts',
default: './dist/test-helpers.js',
},
},
},
SHARED_DIR,
);
expect(aliases.map(({ specifier, replacement }) => ({ specifier, replacement }))).toEqual([
{ specifier: 'gitnexus-shared', replacement: path.join(SHARED_DIR, 'src', 'index.ts') },
{
specifier: 'gitnexus-shared/test-helpers',
replacement: path.join(SHARED_DIR, 'src', 'test-helpers.ts'),
},
]);
});
it('anchors each alias so it matches only its own specifier', () => {
const [root] = deriveSharedAliases(
{ name: 'gitnexus-shared', exports: { '.': './dist/index.js' } },
SHARED_DIR,
);
expect(
[
'gitnexus-shared',
'gitnexus-shared/test-helpers',
'gitnexus-shared-x',
'x/gitnexus-shared',
].map((specifier) => root.find.test(specifier)),
).toEqual([true, false, false, false]);
});
it.each([
['a wildcard export', { './*': './dist/*.js' }],
['a target outside dist', { '.': './lib/index.js' }],
['a target without a default condition', { '.': { types: './dist/index.d.ts' } }],
['a non-subpath key', { import: './dist/index.js' }],
])('rejects %s', (_label, exportsMap) => {
expect(() =>
deriveSharedAliases({ name: 'gitnexus-shared', exports: exportsMap }, SHARED_DIR),
).toThrow(/gitnexus-shared exports/);
});
});

37
gitnexus/test/unit/mutation/prepare-mutate-paths.test.ts Normal file
View file

@ -0,0 +1,37 @@
/**
* Locks how the prepare step resolves its `--mutate` files: canonical repo paths, each once,
* through the default path map, and through an extended map (a mutation fixture project's root,
* U4) only when the caller passes one.
*/
import { describe, expect, it } from 'vitest';
import { canonicalMutatePaths } from '../../../tools/mutation/prepare/prepare.js';
import { buildPathMap } from '../../../tools/mutation/prepare/path-map.js';
const FIXTURE_ROOT = 'gitnexus/test/fixtures/mutation-conformance';
const FIXTURE_FILE = `${FIXTURE_ROOT}/src/runtime.ts`;
describe('canonicalMutatePaths', () => {
it('canonicalises separators and a leading ./ and lists each file once', () => {
expect(
canonicalMutatePaths([
'./gitnexus-shared/src/lbug/schema-constants.ts',
'gitnexus-shared\\src\\lbug\\schema-constants.ts',
'gitnexus/src/core/lbug/schema.ts',
]),
).toEqual(['gitnexus-shared/src/lbug/schema-constants.ts', 'gitnexus/src/core/lbug/schema.ts']);
});
it('accepts a fixture file under the extended path map', () => {
expect(canonicalMutatePaths([`./${FIXTURE_FILE}`], buildPathMap([FIXTURE_ROOT]))).toEqual([
FIXTURE_FILE,
]);
});
it.each([
['a fixture file under the default map', FIXTURE_FILE, /outside the mapped roots/],
['a declaration file', 'gitnexus/src/types.d.ts', /not a \.ts source file/],
['a non-TypeScript file', 'gitnexus/src/index.js', /not a \.ts source file/],
])('rejects %s', (_label, file, message) => {
expect(() => canonicalMutatePaths([file])).toThrow(message);
});
});

189
gitnexus/test/unit/mutation/run-manifest.test.ts Normal file
View file

@ -0,0 +1,189 @@
/**
* Locks the vitest5 runner's run manifest (KTD16): the dry-run process writes it atomically to the
* run directory, every other runner process loads it before its first mutant run, and test
* selection (KTD4) accepts only manifest test ids.
*/
import { execFileSync } from 'node:child_process';
import { mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import {
RUN_MANIFEST_FILE,
RunManifestMissingError,
TestFilterError,
parseRunManifest,
readRunManifest,
selectTests,
writeRunManifest,
type RunManifest,
} from '../../../tools/mutation/runner/run-manifest.js';
const MODULE_URL = new URL('../../../tools/mutation/runner/run-manifest.ts', import.meta.url).href;
const manifest: RunManifest = {
schemaVersion: 1,
project: 'mutation',
vitestVersion: '5.0.5',
files: [
{
file: 'test/unit/a.test.ts',
tests: [
{
id: 'test/unit/a.test.ts#A > one (#2415)',
taskId: 'h1_0_0',
fullName: 'A > one (#2415)',
location: { line: 4, column: 5 },
notOracleValidated: false,
},
{
id: 'test/unit/a.test.ts#A > two',
taskId: 'h1_0_1',
fullName: 'A > two',
location: { line: 8, column: 5 },
notOracleValidated: false,
},
],
skipped: [
{
id: 'test/unit/a.test.ts#A > later',
taskId: 'h1_0_2',
fullName: 'A > later',
location: { line: 12, column: 5 },
notOracleValidated: false,
},
],
},
{
file: 'test/unit/b.test.ts',
tests: [
{
id: 'test/unit/b.test.ts#row:3:1:0',
taskId: 'h2_0',
fullName: 'row',
location: { line: 3, column: 1 },
notOracleValidated: true,
},
],
skipped: [],
},
],
};
let runDir: string;
beforeEach(() => {
runDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-run-manifest-'));
});
afterEach(() => {
rmSync(runDir, { recursive: true, force: true });
});
describe('writeRunManifest and readRunManifest', () => {
it('loads, in this process, a manifest written by another process', () => {
const script = [
`const { writeRunManifest } = await import(${JSON.stringify(MODULE_URL)});`,
`writeRunManifest(process.argv[1], JSON.parse(process.argv[2]));`,
].join('\n');
execFileSync(
process.execPath,
['--import', 'tsx', '--input-type=module', '-e', script, runDir, JSON.stringify(manifest)],
{ stdio: 'pipe' },
);
const loaded = readRunManifest(runDir);
const taskIdByStrykerId = new Map(
loaded.files.flatMap((file) => file.tests.map((test) => [test.id, test.taskId] as const)),
);
expect(loaded).toEqual(manifest);
expect(Object.fromEntries(taskIdByStrykerId)).toEqual({
'test/unit/a.test.ts#A > one (#2415)': 'h1_0_0',
'test/unit/a.test.ts#A > two': 'h1_0_1',
'test/unit/b.test.ts#row:3:1:0': 'h2_0',
});
});
it('leaves only the final manifest file in the run directory', () => {
writeRunManifest(runDir, manifest);
writeRunManifest(runDir, manifest);
expect(readdirSync(runDir)).toEqual([RUN_MANIFEST_FILE]);
});
it('refuses to load when the dry run left no manifest', () => {
expect(() => readRunManifest(runDir)).toThrow(RunManifestMissingError);
expect(() => readRunManifest(runDir)).toThrow(/no run manifest .*dry run/);
});
it('refuses a relative run directory', () => {
expect(() => writeRunManifest('relative/run', manifest)).toThrow(/absolute/);
});
it('refuses a corrupt manifest file', () => {
writeFileSync(path.join(runDir, RUN_MANIFEST_FILE), '{"schemaVersion":1');
expect(() => readRunManifest(runDir)).toThrow(/not valid JSON/);
});
});
describe('parseRunManifest', () => {
const withFiles = (files: unknown): string => JSON.stringify({ ...manifest, files });
it.each([
['an unknown schema version', JSON.stringify({ ...manifest, schemaVersion: 2 }), /schema/],
['a missing project', JSON.stringify({ ...manifest, project: '' }), /project/],
[
'a Stryker id listed twice',
withFiles([manifest.files[0], { ...manifest.files[0], file: 'test/unit/c.test.ts' }]),
/listed twice/,
],
[
'an absolute test file',
withFiles([{ ...manifest.files[1], file: '/abs/b.test.ts' }]),
/package-relative/,
],
[
'a test without a task id',
withFiles([{ ...manifest.files[1], tests: [{ ...manifest.files[1].tests[0], taskId: '' }] }]),
/taskId/,
],
])('rejects %s', (_label, text, message) => {
expect(() => parseRunManifest(text, 'run-manifest.json')).toThrow(message);
});
});
describe('selectTests', () => {
it('selects the filtered tests per file by Vitest task id', () => {
expect(
selectTests(manifest, ['test/unit/b.test.ts#row:3:1:0', 'test/unit/a.test.ts#A > two']),
).toEqual({
files: [
{ file: 'test/unit/a.test.ts', tests: [manifest.files[0].tests[1]] },
{ file: 'test/unit/b.test.ts', tests: [manifest.files[1].tests[0]] },
],
});
});
it("selects the manifest's full test set, without skipped tests, when there is no filter", () => {
expect(selectTests(manifest, undefined)).toEqual({
files: [
{ file: 'test/unit/a.test.ts', tests: manifest.files[0].tests },
{ file: 'test/unit/b.test.ts', tests: manifest.files[1].tests },
],
});
});
it('rejects a filter that holds a test file path instead of a test id', () => {
expect(() => selectTests(manifest, ['test/unit/a.test.ts'])).toThrow(TestFilterError);
expect(() => selectTests(manifest, ['/sandbox/test/unit/a.test.ts'])).toThrow(/testFiles/);
});
it.each([
['an unknown test id', 'test/unit/a.test.ts#A > three'],
['a test that was skipped in the dry run', 'test/unit/a.test.ts#A > later'],
])('rejects %s', (_label, id) => {
expect(() => selectTests(manifest, [id])).toThrow(/not a test id of the run manifest/);
});
});

133
gitnexus/test/unit/mutation/run-scope.test.ts Normal file
View file

@ -0,0 +1,133 @@
/**
* Locks the scope driver's pure helpers (U10): CLI parsing, the Stryker command line, the child
* environment, which must name the scope file and must not inherit Stryker's own worker and
* active-mutant variables (KTD5, KTD10) unless a caller sets them explicitly, and the pid that
* signals reach the whole Stryker process tree through.
*/
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import { runLayout } from '../../../tools/mutation/driver/stryker-config.js';
import {
findSandboxes,
parseCliArgs,
signalTarget,
strykerChildEnv,
strykerCommand,
} from '../../../tools/mutation/driver/run-scope.js';
const CWD = path.resolve('/work/gitnexus');
describe('parseCliArgs', () => {
it('applies the defaults and resolves paths against the working directory', () => {
expect(parseCliArgs(['--scope', 'scope.json', '--out', '/tmp/run'], CWD)).toEqual({
scopeFile: path.join(CWD, 'scope.json'),
outDir: path.resolve('/tmp/run'),
variant: 'vitest5',
concurrency: 2,
maxTestRunnerReuse: 0,
timeoutMS: 10000,
command: undefined,
});
});
it('reads the command variant and the scheduling profile', () => {
const args = [
'--scope=/s.json',
'--out=/o',
'--variant=command',
'--command=true',
'--concurrency=4',
'--max-test-runner-reuse=10',
'--timeout-ms=20000',
];
expect(parseCliArgs(args, CWD)).toMatchObject({
variant: 'command',
command: 'true',
concurrency: 4,
maxTestRunnerReuse: 10,
timeoutMS: 20000,
});
});
it.each([
['no --scope', ['--out', '/o'], /--scope is required/],
['no --out', ['--scope', '/s.json'], /--out is required/],
['an unknown variant', ['--scope=/s', '--out=/o', '--variant=jest'], /unknown variant "jest"/],
['a non-numeric concurrency', ['--scope=/s', '--out=/o', '--concurrency=two'], /integer/],
['an unknown flag', ['--scope=/s', '--out=/o', '--testFiles=a'], /Unknown option/],
])('rejects %s', (_label, args, message) => {
expect(() => parseCliArgs(args, CWD)).toThrow(message);
});
});
describe('strykerChildEnv', () => {
const inherited = {
PATH: '/bin',
__STRYKER_ACTIVE_MUTANT__: '12',
STRYKER_MUTATOR_WORKER: '3',
GITNEXUS_MUTATION_SCOPE: 'stale.json',
};
const env = strykerChildEnv(inherited, '/out/scope.json');
it('names the run scope file', () => {
expect(env.GITNEXUS_MUTATION_SCOPE).toBe('/out/scope.json');
});
it('scrubs inherited Stryker worker and active-mutant variables', () => {
expect(Object.keys(env).sort()).toEqual(['GITNEXUS_MUTATION_SCOPE', 'PATH']);
});
// Gate 2's environment case hands Stryker an active mutant the runner itself must remove.
it('sets an explicit child environment after the scrub', () => {
expect(
strykerChildEnv(inherited, '/out/scope.json', { __STRYKER_ACTIVE_MUTANT__: '7' }),
).toEqual({
PATH: '/bin',
__STRYKER_ACTIVE_MUTANT__: '7',
GITNEXUS_MUTATION_SCOPE: '/out/scope.json',
});
});
it('keeps the run scope file over a child environment that names another', () => {
const child = strykerChildEnv(inherited, '/out/scope.json', {
GITNEXUS_MUTATION_SCOPE: 'other.json',
});
expect(child.GITNEXUS_MUTATION_SCOPE).toBe('/out/scope.json');
});
});
describe('signalTarget', () => {
it.each([
['linux', -4242],
['darwin', -4242],
['win32', 4242],
] as const)('signals the Stryker process tree on %s through pid %i', (platform, target) => {
expect(signalTarget(4242, platform)).toBe(target);
});
});
describe('strykerCommand', () => {
it('runs the prepared copy Stryker binary under tsx from the prepared package', () => {
const layout = runLayout(path.resolve('/out'));
expect(strykerCommand(layout, '/usr/bin/node')).toEqual({
command: '/usr/bin/node',
args: [
'--import',
'tsx',
path.join(layout.packageDir, 'node_modules/@stryker-mutator/core/bin/stryker.js'),
'run',
layout.configFile,
],
cwd: layout.packageDir,
});
});
});
describe('findSandboxes', () => {
it('keeps sandbox directories only, sorted', () => {
expect(findSandboxes('/t', ['sandbox-b', 'other', 'sandbox-a', '.lock'])).toEqual([
path.join('/t', 'sandbox-a'),
path.join('/t', 'sandbox-b'),
]);
});
});

56
gitnexus/test/unit/mutation/scope-file.test.ts Normal file
View file

@ -0,0 +1,56 @@
/**
* Locks the scope-file contract between the caller of a mutation run and
* tools/mutation/vitest.mutation.config.ts (KTD9): a JSON object whose `testFiles` lists
* package-relative test paths.
*/
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import {
MUTATION_SCOPE_ENV,
parseMutationScope,
resolveMutationScopePath,
} from '../../../tools/mutation/scope-file.js';
describe('parseMutationScope', () => {
it('returns the listed test files in order', () => {
expect(
parseMutationScope(
JSON.stringify({ testFiles: ['test/unit/schema.test.ts', 'test/unit/a/b.test.ts'] }),
),
).toEqual({ testFiles: ['test/unit/schema.test.ts', 'test/unit/a/b.test.ts'] });
});
it.each([
['invalid JSON', '{', /not valid JSON/],
['a non-object', '[]', /must be a JSON object/],
['a missing testFiles', '{}', /testFiles must be a non-empty array/],
['an empty testFiles', '{"testFiles":[]}', /testFiles must be a non-empty array/],
['a non-string entry', '{"testFiles":[1]}', /must be a string/],
['an absolute path', '{"testFiles":["/abs/x.test.ts"]}', /package-relative/],
['a parent segment', '{"testFiles":["test/../../x.test.ts"]}', /package-relative/],
['a backslash path', '{"testFiles":["test\\\\unit\\\\x.test.ts"]}', /package-relative/],
['a duplicate entry', '{"testFiles":["t/x.test.ts","t/x.test.ts"]}', /listed twice/],
])('rejects %s', (_label, text, message) => {
expect(() => parseMutationScope(text)).toThrow(message);
});
});
describe('resolveMutationScopePath', () => {
const packageDir = path.join(path.sep, 'prepared', 'gitnexus');
it('resolves a relative scope path against the package directory', () => {
expect(resolveMutationScopePath({ [MUTATION_SCOPE_ENV]: 'scope.json' }, packageDir)).toBe(
path.join(packageDir, 'scope.json'),
);
});
it('keeps an absolute scope path', () => {
const absolute = path.join(path.sep, 'runs', 'scope.json');
expect(resolveMutationScopePath({ [MUTATION_SCOPE_ENV]: absolute }, packageDir)).toBe(absolute);
});
it('fails with the variable name when no scope file is set', () => {
expect(() => resolveMutationScopePath({}, packageDir)).toThrow(MUTATION_SCOPE_ENV);
});
});

155
gitnexus/test/unit/mutation/sidecar.test.ts Normal file
View file

@ -0,0 +1,155 @@
/**
* Locks the vitest5 runner's sidecar files (KTD6): one record per local mutant id, attempt and
* worker, written under a temporary name and published under its final name only when complete.
*/
import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import type { RunClassification } from '../../../tools/mutation/runner/classify-outcome.js';
import {
SIDECAR_SCHEMA_VERSION,
readSidecars,
sidecarFileName,
writeSidecar,
type SidecarDraft,
} from '../../../tools/mutation/runner/sidecar.js';
const survived: RunClassification = {
status: 'survived',
outcome: 'survived',
reason: undefined,
killedBy: [],
failureMessage: undefined,
nrOfTests: 1,
hitCount: 0,
tests: [{ id: 'test/unit/a.test.ts#A > one', outcome: 'survived', message: undefined }],
outsideTest: [],
};
const draft = (worker: string): SidecarDraft => ({
kind: 'mutant',
mutantId: '42',
worker,
mutantActivation: 'runtime',
hitLimit: 100,
requestedTestIds: ['test/unit/a.test.ts#A > one'],
hookAttributedTests: null,
result: survived,
});
let dir: string;
beforeEach(() => {
dir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-sidecar-'));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
describe('sidecarFileName', () => {
it.each([
['a mutant run', 'mutant', '42', '3', 1, 'mutant-42.worker-3.attempt-1.json'],
['the dry run', 'dry-run', null, '1', 2, 'dry-run.worker-1.attempt-2.json'],
['an unsafe worker id', 'mutant', '7', '../x', 1, 'mutant-7.worker-___x.attempt-1.json'],
] as const)('names %s', (_label, kind, mutantId, worker, attempt, expected) => {
expect(sidecarFileName(kind, mutantId, worker, attempt)).toBe(expected);
});
});
describe('writeSidecar', () => {
it('keeps the records of two workers for the same mutant and attempt apart', () => {
const first = writeSidecar(dir, draft('1'));
const second = writeSidecar(dir, draft('2'));
expect([first.attempt, second.attempt]).toEqual([1, 1]);
expect(readdirSync(dir).sort()).toEqual([
'mutant-42.worker-1.attempt-1.json',
'mutant-42.worker-2.attempt-1.json',
]);
expect(readSidecars(dir).map((record) => [record.worker, record.attempt])).toEqual([
['1', 1],
['2', 1],
]);
});
it('gives a repeated run of the same mutant in the same worker the next attempt', () => {
writeSidecar(dir, draft('1'));
const retry = writeSidecar(dir, draft('1'));
expect(retry.attempt).toBe(2);
expect(readdirSync(dir).sort()).toEqual([
'mutant-42.worker-1.attempt-1.json',
'mutant-42.worker-1.attempt-2.json',
]);
});
it('never overwrites a record already published under the final name', () => {
const taken = path.join(dir, 'mutant-42.worker-1.attempt-1.json');
writeFileSync(taken, 'published by another process');
const written = writeSidecar(dir, draft('1'));
expect(readFileSync(taken, 'utf8')).toBe('published by another process');
expect(path.basename(written.file)).toBe('mutant-42.worker-1.attempt-2.json');
});
it('never shows a half-written record under the final name', () => {
const finalName = path.join(dir, 'mutant-42.worker-1.attempt-1.json');
const seenDuringWrite: boolean[] = [];
expect(() =>
writeSidecar(dir, draft('1'), {
writeFile: (file, data) => {
writeFileSync(file, data.slice(0, 10));
seenDuringWrite.push(existsSync(finalName));
throw new Error('disk full');
},
}),
).toThrow(/disk full/);
expect(seenDuringWrite).toEqual([false]);
expect(readdirSync(dir)).toEqual([]);
});
it('writes the complete record, with pid and schema version, as JSON', () => {
const written = writeSidecar(dir, draft('5'));
expect(JSON.parse(readFileSync(written.file, 'utf8'))).toEqual({
schemaVersion: SIDECAR_SCHEMA_VERSION,
attempt: 1,
pid: process.pid,
...draft('5'),
});
});
it('keeps the dry-run hook marker: per mutant, the tests a suite hook attributed it to', () => {
const marker = { '3': ['test/unit/a.test.ts#A > one', 'test/unit/a.test.ts#A > two'] };
const written = writeSidecar(dir, {
...draft('1'),
kind: 'dry-run',
mutantId: null,
mutantActivation: null,
hitLimit: null,
hookAttributedTests: marker,
});
expect(readSidecars(dir)).toEqual([
expect.objectContaining({ schemaVersion: 2, kind: 'dry-run', hookAttributedTests: marker }),
]);
expect(path.basename(written.file)).toBe('dry-run.worker-1.attempt-1.json');
});
it('refuses a relative sidecar directory', () => {
expect(() => writeSidecar('relative/sidecars', draft('1'))).toThrow(/absolute/);
});
});
describe('readSidecars', () => {
it('ignores temporary files that a crashed writer left behind', () => {
writeSidecar(dir, draft('1'));
writeFileSync(path.join(dir, '.mutant-42.worker-9.attempt-1.json.123.tmp'), '{"half":');
expect(readSidecars(dir).map((record) => record.worker)).toEqual(['1']);
});
});

218
gitnexus/test/unit/mutation/span-map-check.test.ts Normal file
View file

@ -0,0 +1,218 @@
/**
* Locks the span-map check (KTD7): Stryker's locations refer to the prepared file, whose only
* difference from the original is `as` assertions blanked to spaces of equal length. A mutant
* passes when its span means the same text in both files outside the blanked ranges; any other
* difference fails with the mutant's global identity in the message.
*/
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import { blankAssertions } from '../../../tools/mutation/prepare/blank-assertions.js';
import {
identifyMutants,
type MutationReport,
type ReportLocation,
type ReportMutant,
} from '../../../tools/mutation/prepare/mutant-identity.js';
import {
assertSpanMap,
checkSpanMap,
indexSource,
locationToOffsets,
} from '../../../tools/mutation/prepare/span-map-check.js';
const PACKAGE_DIR = path.join(path.sep, 'prepared', 'gitnexus');
const SHARED_KEY = 'relocated/gitnexus-shared/src/lbug/schema-constants.ts';
const TABLE = "export const KINDS = ['a', 'b'] as const;\n";
const TABLE_PREPARED = "export const KINDS = ['a', 'b'] ;\n";
const CALL = 'export function f(x: unknown) {\n return g(x as string);\n}\n';
const CALL_PREPARED = 'export function f(x: unknown) {\n return g(x );\n}\n';
const loc = (
startLine: number,
startColumn: number,
endLine: number,
endColumn: number,
): ReportLocation => ({
start: { line: startLine, column: startColumn },
end: { line: endLine, column: endColumn },
});
const mutant = (id: string, location: ReportLocation, replacement = '""'): ReportMutant => ({
id,
mutatorName: 'StringLiteral',
replacement,
status: 'Survived',
location,
});
/** Runs the check on one file: `original` in the repo, `prepared` as Stryker read it. */
const check = (original: string, prepared: string, mutants: readonly ReportMutant[]) => {
const report: MutationReport = {
schemaVersion: '1.0',
files: { [SHARED_KEY]: { language: 'typescript', source: prepared, mutants: [...mutants] } },
};
const run = identifyMutants(report, { packageDir: PACKAGE_DIR, readOriginal: () => original });
return { run, result: checkSpanMap(report, run) };
};
describe('fixtures', () => {
it.each([
['an as-const table', TABLE, TABLE_PREPARED],
['an assertion inside a call', CALL, CALL_PREPARED],
])('match what the prepare step writes for %s', (_label, original, prepared) => {
expect(blankAssertions(original)).toEqual({ ok: true, assertions: 1, text: prepared });
});
});
describe('locationToOffsets', () => {
it.each([
['a span at the start of a line', 'ab\ncd\n', loc(2, 1, 2, 2), 'c'],
['a span that ends at the end of a line', 'ab\ncd\n', loc(2, 2, 2, 3), 'd'],
['the last line without a terminator', 'ab\ncd', loc(2, 1, 2, 3), 'cd'],
['an empty span at the end of a line', 'ab\n', loc(1, 3, 1, 3), ''],
['a multi-line span', 'ab\ncd\nef', loc(1, 2, 3, 2), 'b\ncd\ne'],
['a span on a CRLF line', 'ab\r\ncd\r\n', loc(2, 1, 2, 3), 'cd'],
['a multi-line span over CRLF', 'ab\r\ncd', loc(1, 2, 2, 2), 'b\r\nc'],
['a span after a lone CR', 'ab\rcd', loc(2, 1, 2, 3), 'cd'],
['a span after a line separator', 'ab
cd', loc(2, 2, 2, 3), 'd'],
])('converts %s', (_label, text, location, expected) => {
const { start, end } = locationToOffsets(indexSource(text), location);
expect(text.slice(start, end)).toBe(expected);
});
it('returns 0-based offsets with an exclusive end', () => {
expect(locationToOffsets(indexSource('ab\r\ncd\r\n'), loc(2, 1, 2, 3))).toEqual({
start: 4,
end: 6,
});
});
it.each([
['line 0', 'ab\n', loc(0, 1, 1, 2)],
['a line past the end', 'ab\n', loc(3, 1, 3, 1)],
['column 0', 'ab\n', loc(1, 0, 1, 2)],
['a column past the end of the line', 'ab\ncd', loc(1, 1, 1, 4)],
['a column inside a CRLF terminator', 'ab\r\ncd', loc(1, 1, 1, 4)],
])('rejects %s', (_label, text, location) => {
expect(() => locationToOffsets(indexSource(text), location)).toThrow(RangeError);
});
});
describe('checkSpanMap', () => {
it('maps a mutant inside a blanked-assertion line to the same line and column', () => {
const { run, result } = check(TABLE, TABLE_PREPARED, [mutant('5', loc(1, 28, 1, 31))]);
expect(result).toEqual({
checked: 1,
failures: [],
mapped: [
{
localId: '5',
globalId: run.mutants[0].globalId,
start: 27,
end: 30,
originalText: "'b'",
preparedText: "'b'",
},
],
});
});
it('passes a mutant whose span contains a whole blanked assertion', () => {
const { result } = check(CALL, CALL_PREPARED, [mutant('0', loc(1, 31, 3, 2), '{}')]);
expect(result.failures).toEqual([]);
expect(result.mapped.map((m) => [m.originalText, m.preparedText])).toEqual([
['{\n return g(x as string);\n}', '{\n return g(x );\n}'],
]);
});
it('fails a mutant whose span text differs outside blanked ranges, naming its global id', () => {
const { run, result } = check('const a = 1;\n', 'const a = 2;\n', [
mutant('3', loc(1, 11, 1, 12), '0'),
]);
expect(result.mapped).toEqual([]);
expect(result.failures).toEqual([
{
localId: '3',
globalId: run.mutants[0].globalId,
message:
`span map: mutant ${run.mutants[0].globalId} (local id 3): the original span text ` +
'differs from the prepared span text at 1:11 outside blanked assertions ' +
'(original "1", prepared "2")',
},
]);
});
it('quotes only a window around the difference of a long span', () => {
const literal = (digit: string) => `const s = '${'a'.repeat(40)}${digit}${'b'.repeat(40)}';\n`;
const { result } = check(literal('1'), literal('2'), [mutant('6', loc(1, 11, 1, 94))]);
const window = (digit: string) => `…"${'a'.repeat(24)}${digit}${'b'.repeat(23)}"…`;
expect(result.failures.map((f) => f.message.replace(/^.*?\): /, ''))).toEqual([
'the original span text differs from the prepared span text at 1:52 outside blanked ' +
`assertions (original ${window('1')}, prepared ${window('2')})`,
]);
});
it('fails a mutant when the files differ outside its span', () => {
const { run, result } = check('const a = 1;\nconst b = 2;\n', 'const a = 1;\nconst b = 3;\n', [
mutant('8', loc(1, 11, 1, 12), '0'),
]);
expect(result.failures.map((f) => f.message)).toEqual([
`span map: mutant ${run.mutants[0].globalId} (local id 8): the original file differs ` +
'from the prepared file at 2:11 outside blanked assertions',
]);
});
it('fails every mutant when the prepared file has a different length', () => {
const { run, result } = check(TABLE, `// @ts-nocheck\n${TABLE_PREPARED}`, [
mutant('0', loc(2, 23, 2, 26)),
mutant('1', loc(2, 28, 2, 31)),
]);
expect(result.failures.map((f) => [f.globalId, f.message])).toEqual(
run.mutants.map((m) => [
m.globalId,
`span map: mutant ${m.globalId} (local id ${m.localId}): the prepared file has 57 ` +
'characters and the original 42, so columns do not map',
]),
);
});
it('fails a mutant whose span boundary cuts a blanked assertion', () => {
const { run, result } = check(TABLE, TABLE_PREPARED, [mutant('2', loc(1, 34, 1, 41), '')]);
expect(result.failures.map((f) => f.message)).toEqual([
`span map: mutant ${run.mutants[0].globalId} (local id 2): the span start cuts a ` +
'blanked assertion at 1:34',
]);
});
it('fails a mutant whose location lies outside the file', () => {
const { run, result } = check(TABLE, TABLE_PREPARED, [mutant('4', loc(1, 23, 9, 1))]);
expect(result.failures.map((f) => f.message)).toEqual([
`span map: mutant ${run.mutants[0].globalId} (local id 4): line 9 is outside the ` +
'prepared file (2 lines)',
]);
});
});
describe('assertSpanMap', () => {
it('accepts a result without failures', () => {
const { result } = check(TABLE, TABLE_PREPARED, [mutant('0', loc(1, 23, 1, 26))]);
expect(() => assertSpanMap(result)).not.toThrow();
});
it('throws with the count and every failing global id', () => {
const { run, result } = check('const a = 1;\n', 'const a = 2;\n', [
mutant('0', loc(1, 11, 1, 12), '0'),
mutant('1', loc(1, 1, 1, 6), ''),
]);
const ids = run.mutants.map((m) => m.globalId);
expect(() => assertSpanMap(result)).toThrow(
new RegExp(`2 of 2 mutants failed[\\s\\S]*${ids.map(escape).join('[\\s\\S]*')}`),
);
});
});
function escape(text: string): string {
return text.replace(/[.*+?^${}()|[\]\\/-]/g, '\\$&');
}

469
gitnexus/test/unit/mutation/stryker-config.test.ts Normal file
View file

@ -0,0 +1,469 @@
/**
* Locks the scope driver's generated Stryker config (U10, KTD4, KTD6, KTD10, KTD16, KTD18): mutate
* patterns point inside the prepared tree (relocated gitnexus-shared included), the vitest5 variant
* carries absolute run and sidecar directories, the command variant keeps its sandbox, and no
* generated config ever sets `testFiles`. A scope's `testIds` (Phase 1 KTD2) are validated against
* its test files, reach the runner as `vitest5.testIds`, and select exactly those tests in the dry
* run; an empty selection is rejected everywhere.
*/
import { existsSync } from 'node:fs';
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import {
DEFAULT_IGNORED_DIRS,
assertTestFilesInSandbox,
buildStrykerConfig,
formatMutateEntry,
parseDriverScope,
parseMutateEntry,
parseVariant,
runLayout,
strykerMutatePattern,
type StrykerConfigInput,
} from '../../../tools/mutation/driver/stryker-config.js';
import { buildPathMap } from '../../../tools/mutation/prepare/path-map.js';
import { parseRunnerOptions } from '../../../tools/mutation/runner/plugin.js';
import {
assignTestIds,
selectRequestedTests,
type CollectedTest,
} from '../../../tools/mutation/runner/test-ids.js';
const OUT = path.resolve('/tmp/gitnexus-u10-config-test/out');
const SHARED = 'gitnexus-shared/src/lbug/schema-constants.ts';
const LOCAL = 'gitnexus/src/core/ingestion/type-extractors/shared.ts';
const input = (overrides: Partial<StrykerConfigInput> = {}): StrykerConfigInput => ({
variant: 'vitest5',
outDir: OUT,
mutate: [parseMutateEntry(LOCAL), parseMutateEntry(SHARED)],
concurrency: 3,
maxTestRunnerReuse: 25,
timeoutMS: 10000,
...overrides,
});
const commandInput = (overrides: Partial<StrykerConfigInput> = {}): StrykerConfigInput =>
input({ variant: 'command', command: 'node wrapper.mjs', ...overrides });
describe('runLayout', () => {
it('places every run artefact under the absolute output directory', () => {
const layout = runLayout(OUT);
expect(layout).toMatchObject({
outDir: OUT,
preparedRoot: path.join(OUT, 'prepared'),
packageDir: path.join(OUT, 'prepared', 'gitnexus'),
pluginFile: path.join(OUT, 'prepared', 'gitnexus', 'tools/mutation/runner/plugin.ts'),
scopeFile: path.join(OUT, 'scope.json'),
configFile: path.join(OUT, 'stryker.config.json'),
reportFile: path.join(OUT, 'mutation.json'),
runDir: path.join(OUT, 'run'),
sidecarDir: path.join(OUT, 'sidecars'),
resultsFile: path.join(OUT, 'results.json'),
});
});
it('rejects a relative output directory', () => {
expect(() => runLayout('tmp/out')).toThrow(/must be an absolute path/);
});
});
describe('parseMutateEntry and strykerMutatePattern', () => {
it('maps a relocated gitnexus-shared path to a pattern inside the prepared tree', () => {
const layout = runLayout(OUT);
const pattern = strykerMutatePattern(parseMutateEntry(SHARED));
expect(pattern).toBe('relocated/gitnexus-shared/src/lbug/schema-constants.ts');
expect(path.resolve(layout.packageDir, pattern)).toBe(
path.join(
layout.preparedRoot,
'gitnexus/relocated/gitnexus-shared/src/lbug/schema-constants.ts',
),
);
});
it.each([
[`${LOCAL}`, 'src/core/ingestion/type-extractors/shared.ts'],
[`${LOCAL}:545-660`, 'src/core/ingestion/type-extractors/shared.ts:545-660'],
[`${SHARED}:3:5-9:1`, 'relocated/gitnexus-shared/src/lbug/schema-constants.ts:3:5-9:1'],
])('keeps a Stryker line range: %s', (entry, pattern) => {
expect(strykerMutatePattern(parseMutateEntry(entry))).toBe(pattern);
});
it('splits the file from its range', () => {
expect(parseMutateEntry(`${LOCAL}:545-660`)).toEqual({ file: LOCAL, range: '545-660' });
});
it.each([LOCAL, `${LOCAL}:545-660`, `${SHARED}:3:5-9:1`])(
'formats the parsed entry %s back to its text',
(entry) => {
expect(formatMutateEntry(parseMutateEntry(entry))).toBe(entry);
},
);
it.each([
['README.md', /\.ts source file/],
['gitnexus/src/types.d.ts', /\.ts source file/],
['gitnexus/test/unit/schema.test.ts', /outside the mapped roots/],
['/abs/gitnexus/src/a.ts', /not a relative path inside the root/],
[`${LOCAL}:0-4`, /line range/],
[`${LOCAL}:9-4`, /line range/],
[`${LOCAL}:4-`, /\.ts source file/],
])('rejects %s', (entry, message) => {
expect(() => parseMutateEntry(entry)).toThrow(message);
});
});
describe('mutate entries under an extra root (fixture projects)', () => {
const FIXTURE_ROOT = 'gitnexus/test/fixtures/mutation-conformance';
const FIXTURE = `${FIXTURE_ROOT}/src/static-table.ts`;
const map = buildPathMap([FIXTURE_ROOT]);
it('accepts a fixture file and maps it to its package-relative pattern', () => {
expect(strykerMutatePattern(parseMutateEntry(`${FIXTURE}:3-9`, map), map)).toBe(
'test/fixtures/mutation-conformance/src/static-table.ts:3-9',
);
});
it('rejects the fixture file under the default map', () => {
expect(() => parseMutateEntry(FIXTURE)).toThrow(/outside the mapped roots/);
});
it('parses a driver scope with fixture files under the extended map only', () => {
const text = JSON.stringify({
testFiles: ['test/fixtures/mutation-conformance/a.case.ts'],
mutate: [FIXTURE],
});
expect(parseDriverScope(text, map).mutate).toEqual([{ file: FIXTURE, range: null }]);
expect(() => parseDriverScope(text)).toThrow(/outside the mapped roots/);
});
});
describe('parseDriverScope', () => {
it('parses package-relative test files and repo-relative mutate entries', () => {
const scope = parseDriverScope(
JSON.stringify({ testFiles: ['test/unit/schema.test.ts'], mutate: [`${SHARED}:1-40`] }),
);
expect(scope).toEqual({
testFiles: ['test/unit/schema.test.ts'],
mutate: [{ file: SHARED, range: '1-40' }],
});
});
it.each([
['missing testFiles', { mutate: [LOCAL] }, /testFiles must be a non-empty array/],
['missing mutate', { testFiles: ['test/unit/a.test.ts'] }, /mutate must be a non-empty array/],
['empty mutate', { testFiles: ['test/unit/a.test.ts'], mutate: [] }, /mutate must be/],
[
'a non-string mutate entry',
{ testFiles: ['test/unit/a.test.ts'], mutate: [3] },
/mutate entries must be strings/,
],
[
'a repeated mutate entry',
{ testFiles: ['test/unit/a.test.ts'], mutate: [LOCAL, LOCAL] },
/listed twice/,
],
[
'an unknown key',
{ testFiles: ['test/unit/a.test.ts'], mutate: [LOCAL], testFile: [] },
/unknown key "testFile"/,
],
['a non-object', [LOCAL], /must be a JSON object/],
])('rejects %s', (_label, value, message) => {
expect(() => parseDriverScope(JSON.stringify(value))).toThrow(message);
});
it('rejects text that is not JSON', () => {
expect(() => parseDriverScope('{')).toThrow(/not valid JSON/);
});
});
describe('parseDriverScope: testIds (single-test selection, KTD2)', () => {
const FILE = 'test/unit/a.test.ts';
const OTHER = 'test/unit/b.test.ts';
/** Five collected tests of one file, in collection order. */
const FIVE: readonly CollectedTest[] = ['one', 'two', 'three', 'four', 'five'].map(
(name, index) => ({
taskId: `h_${index}`,
fullName: `A > ${name}`,
location: { line: index + 3, column: 3 },
}),
);
const PICKED = [`${FILE}#A > two`, `${FILE}#A > four`];
const scopeText = (testIds: unknown): string =>
JSON.stringify({ testFiles: [FILE, OTHER], mutate: [LOCAL], testIds });
it('keeps the requested ids of two of five tests in one file', () => {
expect(parseDriverScope(scopeText(PICKED)).testIds).toEqual(PICKED);
});
it('leaves testIds out when the scope file has none', () => {
const scope = parseDriverScope(JSON.stringify({ testFiles: [FILE], mutate: [LOCAL] }));
expect('testIds' in scope).toBe(false);
});
it('carries the ids into the generated config, whose dry-run selection holds exactly those two tests', () => {
const scope = parseDriverScope(scopeText(PICKED));
const config = buildStrykerConfig(input({ testIds: scope.testIds }));
const options = parseRunnerOptions(config.vitest5);
const selection = selectRequestedTests(
new Map([[FILE, assignTestIds(FILE, FIVE)]]),
options.testIds ?? [],
);
expect([config.vitest5?.testIds, selection]).toEqual([
PICKED,
{ files: [{ file: FILE, taskIds: ['h_1', 'h_3'] }], unknown: [] },
]);
});
it.each([
['an empty array', [], /testIds must be a non-empty array/],
['a non-array', `${FILE}#A > two`, /testIds must be a non-empty array/],
['a non-string entry', [3], /testIds entries must be strings/],
['a repeated id', [PICKED[0], PICKED[0]], /listed twice/],
['an id without a file part', ['A > two'], /"A > two" is not a test id/],
])('rejects %s', (_label, testIds, message) => {
expect(() => parseDriverScope(scopeText(testIds))).toThrow(message);
});
it('rejects an id of a file outside the scope, naming the id', () => {
const stray = 'test/unit/c.test.ts#C > three';
expect(() => parseDriverScope(scopeText([PICKED[0], stray]))).toThrow(
`test id "${stray}" names no test file of the scope`,
);
});
});
describe('buildStrykerConfig: testIds', () => {
const ids = ['test/unit/a.test.ts#A > two'];
it('passes no testIds to the runner when the scope selects none', () => {
expect(buildStrykerConfig(input()).vitest5).not.toHaveProperty('testIds');
});
it.each([
['an empty selection', input({ testIds: [] }), /testIds must not be empty/],
['a selection for the command variant', commandInput({ testIds: ids }), /only for the vitest5/],
])('rejects %s', (_label, configInput, message) => {
expect(() => buildStrykerConfig(configInput)).toThrow(message);
});
});
describe('parseRunnerOptions: testIds', () => {
const base = { runDir: '/run', sidecarDir: '/sidecars' };
it('reads no selection as null', () => {
expect(parseRunnerOptions(base).testIds).toBeNull();
});
it.each([
['an empty array', []],
['a non-array', 'test/unit/a.test.ts#A > two'],
['a non-string entry', [7]],
['an id without a file part', ['A > two']],
['a repeated id', ['test/unit/a.test.ts#A', 'test/unit/a.test.ts#A']],
])('rejects %s', (_label, testIds) => {
expect(() => parseRunnerOptions({ ...base, testIds })).toThrow(
/vitest5\.testIds must be a non-empty array of distinct test ids/,
);
});
});
describe('parseVariant', () => {
it.each(['vitest5', 'command'] as const)('accepts %s', (variant) => {
expect(parseVariant(variant)).toBe(variant);
});
it('rejects an unknown variant', () => {
expect(() => parseVariant('jest')).toThrow(/unknown variant "jest"/);
});
});
describe('buildStrykerConfig: vitest5 variant', () => {
const layout = runLayout(OUT);
const config = buildStrykerConfig(input());
it('loads the runner plugin from the prepared copy and runs it under tsx', () => {
expect(config).toMatchObject({
testRunner: 'vitest5',
appendPlugins: [layout.pluginFile],
testRunnerNodeArgs: ['--import', 'tsx'],
checkerNodeArgs: ['--import', 'tsx'],
coverageAnalysis: 'perTest',
disableBail: true,
disableTypeChecks: false,
});
});
it('sets the run and sidecar directories as absolute paths', () => {
expect(config.vitest5).toEqual({ runDir: layout.runDir, sidecarDir: layout.sidecarDir });
expect([config.vitest5?.runDir, config.vitest5?.sidecarDir].map(path.isAbsolute)).toEqual([
true,
true,
]);
});
it('points tsconfigFile at a file that does not exist', () => {
expect(path.isAbsolute(config.tsconfigFile)).toBe(true);
expect(existsSync(config.tsconfigFile)).toBe(false);
});
it('passes the scheduling profile through', () => {
expect(config).toMatchObject({ concurrency: 3, maxTestRunnerReuse: 25, timeoutMS: 10000 });
});
it('writes the JSON report to the layout report file', () => {
expect(config.reporters).toContain('json');
expect(config.jsonReporter).toEqual({ fileName: layout.reportFile });
});
it('maps every mutate entry inside the prepared tree', () => {
expect(config.mutate).toEqual([
'src/core/ingestion/type-extractors/shared.ts',
'relocated/gitnexus-shared/src/lbug/schema-constants.ts',
]);
});
it('keeps vendor, native and build output out of the sandbox, but not the relocated tree', () => {
expect(config.ignorePatterns).toEqual(
expect.arrayContaining(['/vendor', '/prebuilds', '/dist']),
);
expect(config.ignorePatterns).toEqual(DEFAULT_IGNORED_DIRS.map((dir) => `/${dir}`));
expect(config.ignorePatterns.some((pattern) => pattern.includes('relocated'))).toBe(false);
});
it('cleans the sandbox after a successful run', () => {
expect(config.cleanTempDir).toBe(true);
});
});
describe('buildStrykerConfig: command variant', () => {
const config = buildStrykerConfig(commandInput());
it('keeps the sandbox: cleanTempDir is false', () => {
expect(config.cleanTempDir).toBe(false);
});
it('runs the given command with coverage analysis off and without the runner plugin', () => {
expect(config).toMatchObject({
testRunner: 'command',
commandRunner: { command: 'node wrapper.mjs' },
coverageAnalysis: 'off',
});
expect(config.appendPlugins).toBeUndefined();
expect(config.vitest5).toBeUndefined();
});
it('sets no test-runner node args, which the command runner ignores, but keeps the checker args', () => {
expect(config.testRunnerNodeArgs).toBeUndefined();
expect(config.checkerNodeArgs).toEqual(['--import', 'tsx']);
});
});
describe('buildStrykerConfig: never sets testFiles', () => {
it.each([
['vitest5', input()],
['command', commandInput()],
])('%s', (_variant, configInput) => {
const config = buildStrykerConfig(configInput);
expect(Object.keys(config)).not.toContain('testFiles');
expect(JSON.stringify(config)).not.toContain('testFiles');
});
});
describe('buildStrykerConfig: input errors', () => {
it.each([
['a relative output directory', input({ outDir: 'relative/out' }), /absolute path/],
['no mutate entry', input({ mutate: [] }), /at least one mutate entry/],
['concurrency 0', input({ concurrency: 0 }), /concurrency must be a positive integer/],
['a fractional concurrency', input({ concurrency: 1.5 }), /concurrency must be/],
[
'a negative maxTestRunnerReuse',
input({ maxTestRunnerReuse: -1 }),
/maxTestRunnerReuse must be a non-negative integer/,
],
['timeoutMS 0', input({ timeoutMS: 0 }), /timeoutMS must be a positive integer/],
['a command for vitest5', input({ command: 'true' }), /only for the command variant/],
['no command for the command variant', commandInput({ command: undefined }), /needs a command/],
['an empty command', commandInput({ command: '' }), /needs a command/],
])('rejects %s', (_label, configInput, message) => {
expect(() => buildStrykerConfig(configInput)).toThrow(message);
});
it('rejects an unknown variant that bypassed the type', () => {
const variant = parseVariantUnchecked('karma');
expect(() => buildStrykerConfig({ ...input(), variant })).toThrow(/unknown variant "karma"/);
});
});
describe('buildStrykerConfig: fixture projects', () => {
const FIXTURE_ROOT = 'gitnexus/test/fixtures/mutation-conformance';
const map = buildPathMap([FIXTURE_ROOT]);
const fixtureMutate = [parseMutateEntry(`${FIXTURE_ROOT}/src/runtime.ts`, map)];
const fixtureIgnored = DEFAULT_IGNORED_DIRS.filter((dir) => dir !== 'test/fixtures');
const vitest = {
configFile: 'test/fixtures/mutation-conformance/vitest.config.ts',
project: 'conformance',
};
it('passes the Vitest config file and project to the vitest5 runner', () => {
const config = buildStrykerConfig(input({ vitest }));
expect(config.vitest5).toEqual({
runDir: runLayout(OUT).runDir,
sidecarDir: runLayout(OUT).sidecarDir,
...vitest,
});
});
it('maps fixture mutate entries through the given path map', () => {
const config = buildStrykerConfig(
input({ mutate: fixtureMutate, pathMap: map, ignoredDirs: fixtureIgnored }),
);
expect(config.mutate).toEqual(['test/fixtures/mutation-conformance/src/runtime.ts']);
});
it('rejects a mutate file that an ignored directory keeps out of the sandbox', () => {
expect(() => buildStrykerConfig(input({ mutate: fixtureMutate, pathMap: map }))).toThrow(
/mutate file .* lies under the ignored directory test\/fixtures\//,
);
});
it.each([
['a Vitest config for the command variant', commandInput({ vitest }), /only for the vitest5/],
[
'an absolute Vitest config path',
input({ vitest: { ...vitest, configFile: '/abs/vitest.config.ts' } }),
/package-relative POSIX path/,
],
[
'a Vitest config path that climbs out of the package',
input({ vitest: { ...vitest, configFile: '../vitest.config.ts' } }),
/package-relative POSIX path/,
],
['an empty project name', input({ vitest: { ...vitest, project: '' } }), /project name/],
])('rejects %s', (_label, configInput, message) => {
expect(() => buildStrykerConfig(configInput)).toThrow(message);
});
});
describe('assertTestFilesInSandbox', () => {
it('accepts test files outside the ignored directories', () => {
expect(() =>
assertTestFilesInSandbox(['test/unit/schema.test.ts'], DEFAULT_IGNORED_DIRS),
).not.toThrow();
});
it.each(['test/fixtures/mutation-x/a.case.ts', 'test/integration/b.test.ts', 'bench/c.test.ts'])(
'rejects %s, which the sandbox would not contain',
(file) => {
expect(() => assertTestFilesInSandbox([file], DEFAULT_IGNORED_DIRS)).toThrow(
/lies under the ignored directory/,
);
},
);
});
/** Simulates an unvalidated CLI value reaching the generator. */
function parseVariantUnchecked(value: string): StrykerConfigInput['variant'] {
return value as StrykerConfigInput['variant'];
}

241
gitnexus/test/unit/mutation/test-ids.test.ts Normal file
View file

@ -0,0 +1,241 @@
/**
* Locks the Stryker test identity of the vitest5 runner (KTD3): `<relative file>#<fullTestName>`,
* parsed on the first `#`, with a `:<line>:<column>` suffix for duplicate full names and a
* `:<line>:<column>:<row>` suffix for rows of one `it.each`/`test.for` call that still collide.
* The dry run's `testIds` selection maps requested ids to task ids and checks the executed set.
*/
import { describe, expect, it } from 'vitest';
import {
assignTestIds,
assignPortableTestIds,
formatTestId,
parseTestId,
selectRequestedTests,
selectionProblems,
type CollectedTest,
} from '../../../tools/mutation/runner/test-ids.js';
const FILE = 'test/unit/shared-type-extractors.test.ts';
const collected = (
taskId: string,
fullName: string,
line: number,
column: number,
): CollectedTest => ({ taskId, fullName, location: { line, column } });
describe('portable test identity', () => {
it('matches coverage, trace and Stryker copies while retaining the runtime title', () => {
const roots = [
'/work/coverage/prepared/gitnexus',
'/work/trace/prepared/gitnexus',
'/work/.stryker-tmp/sandbox-1',
];
const runs = roots.map((root) => {
const fullName = `hooks > reads ${root}/hooks/registry.cjs`;
const [test] = assignPortableTestIds(FILE, [collected('task', fullName, 20, 3)], root);
expect(test.fullName).toBe(fullName);
return test.id;
});
expect(new Set(runs).size).toBe(1);
});
it('keeps sibling plugin paths and package paths distinct across prepared copies', () => {
const runs = ['/coverage/prepared', '/trace/prepared'].map((root) =>
assignPortableTestIds(
FILE,
['gitnexus', 'gitnexus-claude-plugin'].map((dir, i) =>
collected(`task-${i}`, `reads ${root}/${dir}/hooks/registry.cjs`, 20, 3),
),
`${root}/gitnexus`,
).map((test) => test.id),
);
expect(runs[0]).toEqual(runs[1]);
expect(new Set(runs[0]).size).toBe(2);
});
it('does not rewrite other absolute paths or a root prefix in a sibling directory', () => {
const tests = [collected('a', 'reads /outside/a and /copycat/gitnexus/a', 1, 1)];
expect(assignPortableTestIds(FILE, tests, '/copy/gitnexus')).toEqual(
assignTestIds(FILE, tests),
);
});
it('retains collision suffixes when normalization makes titles equal', () => {
const tests = [
collected('a', 'reads /copy/gitnexus/a', 1, 1),
collected('b', 'reads <package>/a', 2, 1),
];
const ids = assignPortableTestIds(FILE, tests, '/copy/gitnexus');
expect(ids.map((test) => test.id)).toEqual([
`${FILE}#reads <package>/a:1:1`,
`${FILE}#reads <package>/a:2:1`,
]);
});
});
describe('formatTestId and parseTestId', () => {
it('joins the file and the full name with the first #', () => {
expect(formatTestId(FILE, 'describe A > it b (#2415)')).toBe(
`${FILE}#describe A > it b (#2415)`,
);
});
it.each([
['a title with an issue reference', 'describe A > it b (#2415)'],
['a title that contains the separator twice', 'a # b > c#d'],
['a title containing " > "', 'outer > inner > maps a > b to c'],
['a title that ends with a location-like suffix', 'parses x:12:3'],
])('round-trips %s', (_label, name) => {
const id = formatTestId(FILE, name);
expect(parseTestId(id)).toEqual({ file: FILE, name });
});
it('splits on the first # only', () => {
expect(parseTestId('test/a.test.ts#A > b (#1) and #2')).toEqual({
file: 'test/a.test.ts',
name: 'A > b (#1) and #2',
});
});
it('rejects an id without a separator', () => {
expect(() => parseTestId('test/a.test.ts')).toThrow(/no "#"/);
});
it('rejects a file path that contains the separator', () => {
expect(() => formatTestId('test/a#b.test.ts', 'x')).toThrow(/contains "#"/);
});
});
describe('assignTestIds', () => {
it('uses the plain full name when it is unique in the file', () => {
expect(
assignTestIds(FILE, [collected('f_0', 'A > one', 3, 5), collected('f_1', 'A > two', 4, 5)]),
).toEqual([
{
taskId: 'f_0',
fullName: 'A > one',
location: { line: 3, column: 5 },
id: `${FILE}#A > one`,
notOracleValidated: false,
},
{
taskId: 'f_1',
fullName: 'A > two',
location: { line: 4, column: 5 },
id: `${FILE}#A > two`,
notOracleValidated: false,
},
]);
});
it('suffixes duplicate full names with their location', () => {
const ids = assignTestIds(FILE, [
collected('f_0', 'A > same', 3, 5),
collected('f_1', 'A > same', 9, 7),
collected('f_2', 'A > other', 12, 5),
]);
expect(ids.map((test) => [test.id, test.notOracleValidated])).toEqual([
[`${FILE}#A > same:3:5`, false],
[`${FILE}#A > same:9:7`, false],
[`${FILE}#A > other`, false],
]);
});
it('adds the row ordinal of the call when name and location collide, and flags those rows', () => {
const ids = assignTestIds(FILE, [
collected('f_0', 'each > truncated row title', 20, 3),
collected('f_1', 'each > distinct row', 20, 3),
collected('f_2', 'each > truncated row title', 20, 3),
]);
expect(ids.map((test) => [test.id, test.notOracleValidated])).toEqual([
[`${FILE}#each > truncated row title:20:3:0`, true],
[`${FILE}#each > distinct row`, false],
[`${FILE}#each > truncated row title:20:3:2`, true],
]);
});
it('keeps every assigned id parseable back to its file and suffixed name', () => {
const [first] = assignTestIds(FILE, [
collected('f_0', 'A > b (#2415)', 3, 5),
collected('f_1', 'A > b (#2415)', 4, 5),
]);
expect(parseTestId(first.id)).toEqual({ file: FILE, name: 'A > b (#2415):3:5' });
});
it('refuses a duplicate full name without a location', () => {
expect(() =>
assignTestIds(FILE, [
{ taskId: 'f_0', fullName: 'A > same', location: undefined },
{ taskId: 'f_1', fullName: 'A > same', location: undefined },
]),
).toThrow(/includeTaskLocation/);
});
it('refuses ids that still collide after suffixing', () => {
expect(() =>
assignTestIds(FILE, [
collected('f_0', 'A > x:3:5', 1, 1),
collected('f_1', 'A > x', 3, 5),
collected('f_2', 'A > x', 4, 5),
]),
).toThrow(/not unique/);
});
it('refuses a task id listed twice', () => {
expect(() =>
assignTestIds(FILE, [collected('f_0', 'A > one', 3, 5), collected('f_0', 'A > two', 4, 5)]),
).toThrow(/task id f_0/);
});
});
describe('selectRequestedTests (dry-run selection by testIds)', () => {
const OTHER = 'test/unit/other.test.ts';
const identified = new Map([
[
FILE,
assignTestIds(FILE, [
collected('f_0', 'A > one', 3, 5),
collected('f_1', 'A > two', 4, 5),
collected('f_2', 'A > three', 5, 5),
]),
],
[OTHER, assignTestIds(OTHER, [collected('g_0', 'B > one', 3, 5)])],
]);
it('maps the requested ids to task ids per file, in collection order, and leaves out files with none', () => {
expect(selectRequestedTests(identified, [`${FILE}#A > three`, `${FILE}#A > one`])).toEqual({
files: [{ file: FILE, taskIds: ['f_0', 'f_2'] }],
unknown: [],
});
});
it('reports every requested id that no collected test has', () => {
expect(
selectRequestedTests(identified, [`${OTHER}#B > one`, `${FILE}#A > four`, 'x.test.ts#y']),
).toEqual({
files: [{ file: OTHER, taskIds: ['g_0'] }],
unknown: [`${FILE}#A > four`, 'x.test.ts#y'],
});
});
});
describe('selectionProblems (the executed set must equal the request)', () => {
const ONE = `${FILE}#A > one`;
const TWO = `${FILE}#A > two`;
it('accepts a run that executed exactly the requested tests, in any order', () => {
expect(selectionProblems([ONE, TWO], [TWO, ONE])).toEqual([]);
});
it('names a requested test that did not run and a test that ran unrequested', () => {
expect(selectionProblems([ONE], [TWO])).toEqual([
`requested test ${JSON.stringify(ONE)} did not run (vitest5.testIds)`,
`test ${JSON.stringify(TWO)} ran but vitest5.testIds does not request it`,
]);
});
});

105
gitnexus/test/unit/mutation/version-guard.test.ts Normal file
View file

@ -0,0 +1,105 @@
/**
* Locks the vitest5 runner's version guard (KTD2): the Vitest resolved from the project root must
* be 5.0.x, and the plugin location must resolve the same Vitest copy.
*/
import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import {
assertSupportedVitest,
checkVitestInstalls,
resolveVitestInstall,
type VitestInstall,
} from '../../../tools/mutation/runner/version-guard.js';
const install = (version: string, packageJsonPath = '/repo/node_modules/vitest/package.json') =>
({ version, packageJsonPath }) satisfies VitestInstall;
describe('checkVitestInstalls', () => {
it('accepts Vitest 5.0.5 resolved to one copy from both locations', () => {
expect(checkVitestInstalls(install('5.0.5'), install('5.0.5'))).toEqual(install('5.0.5'));
});
it.each(['5.1.0', '4.1.12', '5.0.5-beta.1', '6.0.0'])('rejects Vitest %s', (version) => {
expect(() => checkVitestInstalls(install(version), install(version))).toThrow(/5\.0\.x/);
});
it('rejects a second Vitest copy resolved from the plugin location', () => {
expect(() =>
checkVitestInstalls(
install('5.0.5'),
install('5.0.5', '/plugin/node_modules/vitest/package.json'),
),
).toThrow(/second Vitest copy/);
});
it('rejects a project root that resolves no Vitest', () => {
expect(() => checkVitestInstalls(undefined, install('5.0.5'))).toThrow(/project root/);
});
it('rejects a plugin location that resolves no Vitest', () => {
expect(() => checkVitestInstalls(install('5.0.5'), undefined)).toThrow(/plugin location/);
});
});
describe('resolveVitestInstall and assertSupportedVitest', () => {
let root: string;
const fakeVitest = (dir: string, version: string): void => {
const pkgDir = path.join(dir, 'node_modules', 'vitest');
mkdirSync(pkgDir, { recursive: true });
writeFileSync(path.join(pkgDir, 'package.json'), JSON.stringify({ name: 'vitest', version }));
};
beforeEach(() => {
root = realpathSync(mkdtempSync(path.join(os.tmpdir(), 'gitnexus-version-guard-')));
});
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
it('reads the version and real path of the Vitest a directory resolves', () => {
const project = path.join(root, 'project');
fakeVitest(project, '5.0.5');
expect(resolveVitestInstall(path.join(project, 'test'))).toEqual({
version: '5.0.5',
packageJsonPath: path.join(project, 'node_modules', 'vitest', 'package.json'),
});
});
it('returns undefined when no Vitest resolves', () => {
const empty = path.join(root, 'empty');
mkdirSync(empty);
expect(resolveVitestInstall(empty)).toBeUndefined();
});
it('accepts a project whose node_modules is a symlink to the plugin copy', () => {
const checkout = path.join(root, 'checkout');
const prepared = path.join(root, 'prepared');
fakeVitest(checkout, '5.0.5');
mkdirSync(prepared);
symlinkSync(path.join(checkout, 'node_modules'), path.join(prepared, 'node_modules'), 'dir');
expect(assertSupportedVitest(prepared, path.join(checkout, 'tools')).version).toBe('5.0.5');
});
it('rejects a plugin that resolves its own Vitest copy', () => {
const project = path.join(root, 'project');
const plugin = path.join(root, 'plugin');
fakeVitest(project, '5.0.5');
fakeVitest(plugin, '5.0.5');
expect(() => assertSupportedVitest(project, plugin)).toThrow(/second Vitest copy/);
});
it('rejects a project that resolves Vitest 5.1.0', () => {
const project = path.join(root, 'project');
fakeVitest(project, '5.1.0');
expect(() => assertSupportedVitest(project, project)).toThrow(/5\.1\.0/);
});
});

177
gitnexus/test/unit/scenario-map/baseline.test.ts Normal file
View file

@ -0,0 +1,177 @@
/**
* Locks the week-over-week comparison of the scenario map report (U13, R12, KTD13, design §5.5):
* new, changed, unchanged and resolved items keyed by continuity anchors, the unique-only body
* fallback for moved tests, the "first run, no baseline" and "incompatible baseline" banners (a
* baseline that could not be read included), and
* a report compared with itself and with a copy whose one evidence hash differs.
*/
import { describe, expect, it } from 'vitest';
import {
BANNERS,
UnreadableBaseline,
compareWithBaseline,
itemKey,
type ContinuityItem,
type ContinuitySection,
} from '../../../tools/scenario-map/report/baseline.js';
import { buildReport } from '../../../tools/scenario-map/report/report.js';
import { SCENARIO_MAP_SCHEMA_VERSION, type TestAnchor } from '../../../tools/scenario-map/types.js';
import { reportInput } from './report-fixtures.js';
const anchorOf = (key: string, body: string): TestAnchor => ({
key,
file: 'test/unit/p.test.ts',
describePath: ['p'],
title: key,
rowFingerprint: 'row-0',
bodyFingerprint: body,
});
const X = anchorOf('x', 'body-x');
const Y = anchorOf('y', 'body-y');
const pairItem = (members: readonly string[], evidenceHash: string): ContinuityItem => ({
kind: 'pair',
key: itemKey('pair', members),
members: [...members].sort(),
evidenceHash,
label: members.join(' + '),
});
const P = (evidenceHash: string): ContinuityItem => pairItem(['x', 'y'], evidenceHash);
const section = (items: ContinuityItem[], anchors: TestAnchor[] = [X, Y]): ContinuitySection => ({
anchors,
items,
});
const previousReport = (
continuity: ContinuitySection,
schemaVersion: number = SCENARIO_MAP_SCHEMA_VERSION,
): unknown => ({ schemaVersion, run: { id: '1000' }, continuity });
describe('compareWithBaseline', () => {
it('keeps a pair with the same evidence hash unchanged, out of "new or changed"', () => {
const result = compareWithBaseline(section([P('h1')]), previousReport(section([P('h1')])));
expect(result.changes).toEqual([
{ kind: 'pair', key: P('h1').key, status: 'unchanged', previous: P('h1').key },
]);
expect(result.changes.filter((change) => change.status !== 'unchanged')).toEqual([]);
expect(result.counts).toEqual({ new: 0, changed: 0, unchanged: 1, resolved: 0 });
});
it('marks a pair whose evidence hash differs as changed', () => {
const result = compareWithBaseline(section([P('h2')]), previousReport(section([P('h1')])));
expect(result.changes.map((change) => change.status)).toEqual(['changed']);
expect(result.counts).toEqual({ new: 0, changed: 1, unchanged: 0, resolved: 0 });
});
it('counts a pair found only in the previous report as resolved', () => {
const result = compareWithBaseline(section([]), previousReport(section([P('h1')])));
expect(result.resolved).toEqual([
{ kind: 'pair', key: P('h1').key, label: 'x + y', evidenceHash: 'h1' },
]);
expect(result.counts).toEqual({ new: 0, changed: 0, unchanged: 0, resolved: 1 });
});
it('marks every item new under the "first run, no baseline" banner without a baseline', () => {
const gap: ContinuityItem = {
kind: 'gap',
key: itemKey('gap', [], 'gitnexus/src/a.ts'),
members: [],
evidenceHash: 'h3',
label: 'gitnexus/src/a.ts',
};
const result = compareWithBaseline(section([P('h1'), gap]), null);
expect(result).toMatchObject({
state: 'first-run',
banner: 'first run, no baseline',
counts: { new: 2, changed: 0, unchanged: 0, resolved: 0 },
});
expect(result.changes.map((change) => change.status)).toEqual(['new', 'new']);
});
it('shows the "incompatible baseline" banner and no delta for an older schema version', () => {
const older = previousReport(section([P('h1')]), SCENARIO_MAP_SCHEMA_VERSION - 1);
expect(compareWithBaseline(section([P('h2')]), older)).toMatchObject({
state: 'incompatible',
banner: BANNERS.incompatible,
counts: null,
changes: [],
resolved: [],
});
});
it('treats a baseline without a readable continuity section as incompatible', () => {
const broken = { schemaVersion: SCENARIO_MAP_SCHEMA_VERSION, continuity: { anchors: 'x' } };
expect(compareWithBaseline(section([P('h1')]), broken)).toMatchObject({
state: 'incompatible',
reason: 'baseline has no readable continuity',
counts: null,
});
});
it('treats a baseline that could not be read as incompatible, with the read error', () => {
const unreadable = new UnreadableBaseline('Unexpected end of JSON input');
expect(compareWithBaseline(section([P('h1')]), unreadable)).toMatchObject({
state: 'incompatible',
banner: BANNERS.incompatible,
reason: 'unreadable baseline: Unexpected end of JSON input',
baseline: { runId: null, schemaVersion: null },
counts: null,
changes: [],
});
});
it('follows a moved test through its unique body fingerprint', () => {
const moved = anchorOf('x-moved', 'body-x');
const current = section([pairItem(['x-moved', 'y'], 'h1')], [moved, Y]);
const result = compareWithBaseline(current, previousReport(section([P('h1')])));
expect(result.changes).toEqual([
{
kind: 'pair',
key: itemKey('pair', ['x-moved', 'y']),
status: 'unchanged',
previous: P('h1').key,
},
]);
expect(result.counts?.resolved).toBe(0);
});
it('never follows a body that two current tests share', () => {
const twinA = anchorOf('x-a', 'body-x');
const twinB = anchorOf('x-b', 'body-x');
const current = section(
[pairItem(['x-a', 'y'], 'h1'), pairItem(['x-b', 'y'], 'h1')],
[twinA, twinB, Y],
);
const result = compareWithBaseline(current, previousReport(section([P('h1')])));
expect(result.counts).toEqual({ new: 2, changed: 0, unchanged: 0, resolved: 1 });
});
});
describe('a built report against a baseline', () => {
it('finds every item unchanged against itself', () => {
const first = buildReport(reportInput());
const again = buildReport(reportInput({ overrides: { baseline: first } }));
expect(again.changes.counts).toEqual({
new: 0,
changed: 0,
unchanged: first.continuity.items.length,
resolved: 0,
});
expect(again.pairs.map((pair) => pair.change)).toEqual(['unchanged', 'unchanged']);
});
it('finds exactly the item whose evidence hash was altered changed', () => {
const first = buildReport(reportInput());
const altered = structuredClone(first);
altered.continuity.items[0] = { ...altered.continuity.items[0], evidenceHash: 'e-altered' };
const again = buildReport(reportInput({ overrides: { baseline: altered } }));
expect(again.changes.counts).toEqual({
new: 0,
changed: 1,
unchanged: first.continuity.items.length - 1,
resolved: 0,
});
expect(again.changes.changes.filter((change) => change.status === 'changed')).toEqual([
expect.objectContaining({ key: first.continuity.items[0].key }),
]);
});
});

241
gitnexus/test/unit/scenario-map/bind.test.ts Normal file
View file

@ -0,0 +1,241 @@
/**
* Locks the test side of the value-flow check (U10, R9, KTD11 "Test side (Babel)", design §5.3a
* step 1): each differing discarded value of a pair (real U6 facets) is bound to where it enters
* production code, a call argument (callee and argument index), a field of a fixture object or
* array, or a table key (an `it.each` row value), following `const` bindings. A differing expected
* value follows the input that produced it; when only expected values differ the pair is unknown
* (`expected-only`). A value transformed by test code, used nowhere, or bound differently by the
* two tests is unknown.
*/
import { describe, expect, it } from 'vitest';
import {
computeFacet,
createFacetContext,
type FacetSources,
type FacetSubject,
} from '../../../tools/scenario-map/facet/input-facet.js';
import { expandCases } from '../../../tools/scenario-map/inventory/cases.js';
import {
bindPair,
bindSideOf,
type BindResult,
type BindSide,
} from '../../../tools/scenario-map/value-flow/bind.js';
const TEST_FILE = 'gitnexus/test/unit/a.test.ts';
const PACKAGE_FILE = 'test/unit/a.test.ts';
const M = 'gitnexus/src/m.ts';
const PRODUCTION = [
'export function roundScore(raw: number): number { return Math.round(raw); }',
'export function formatTags(tags: readonly string[]): string { return tags.join(); }',
'export function configure(options: { ratio: number }): number { return options.ratio; }',
'export class Box { constructor(readonly size: number) {} }',
'export function pick(first: number, second: number): number { return first + second; }',
].join('\n');
const IMPORTS = `import { Box, configure, formatTags, pick, roundScore } from '../../src/m.js';`;
const memorySources = (files: Record<string, string>): FacetSources => ({
read: (file) => files[file] ?? null,
list: (dir) =>
Object.keys(files)
.filter((file) => file.startsWith(`${dir}/`))
.sort(),
});
/** The two bind sides of the instances written in `body`, with real U6 facets. */
async function sidesOf(body: string): Promise<[BindSide, BindSide]> {
const source = [IMPORTS, `describe('d', () => {`, body, `});`].join('\n');
const sites = expandCases([{ path: TEST_FILE, source }]).sites;
const ctx = createFacetContext({
sources: memorySources({ [M]: PRODUCTION, [TEST_FILE]: source }),
productionFiles: [M],
closures: new Map([[PACKAGE_FILE, [TEST_FILE, M]]]),
builtinFiles: [],
});
const subjects: FacetSubject[] = sites.flatMap((site) =>
(site.instances ?? []).map((instance) => ({
id: `${PACKAGE_FILE}#d > ${site.record.title}#${instance.row}`,
file: PACKAGE_FILE,
record: site.record,
instance,
executedFiles: [M],
})),
);
expect(subjects).toHaveLength(2);
const sides = await Promise.all(
subjects.map(async (subject) =>
bindSideOf(subject.record, (await computeFacet(subject, ctx)).discarded, source),
),
);
return [sides[0], sides[1]];
}
/** The result without AST-dependent detail, for comparison. */
function summary(result: BindResult): unknown {
return {
status: result.status,
reasons: result.reasons,
entries: result.entries.map(
(entry) =>
`${entry.kind} ${entry.isNew ? 'new ' : ''}${entry.callee}#${entry.argIndex}${entry.path.length > 0 ? ` ${entry.path.join('.')}` : ''} @${entry.position}`,
),
roles: result.positions.map((position) => `${position.position}:${position.role}`),
};
}
const twoTests = (first: string, second: string): string =>
`it('a', () => { ${first} });\nit('b', () => { ${second} });`;
describe('bindPair: entries', () => {
it.each([
{
title: 'a call argument, with the expected value following its input',
body: twoTests('expect(roundScore(2.4)).toBe(2);', 'expect(roundScore(2.5)).toBe(3);'),
want: {
status: 'bound',
reasons: [],
entries: ['call-argument roundScore#0 @0'],
roles: ['0:input', '1:expected'],
},
},
{
title: 'a call argument through a const binding',
body: twoTests(
'const raw = 2.4; expect(roundScore(raw)).toBe(2);',
'const raw = 2.5; expect(roundScore(raw)).toBe(3);',
),
want: {
status: 'bound',
reasons: [],
entries: ['call-argument roundScore#0 @0'],
roles: ['0:input', '1:expected'],
},
},
{
title: 'a fixture field',
body: twoTests(
'expect(configure({ ratio: 2.4 })).toBeGreaterThan(2);',
'expect(configure({ ratio: 2.5 })).toBeGreaterThan(2);',
),
want: {
status: 'bound',
reasons: [],
entries: ['fixture-field configure#0 ratio @0'],
roles: ['0:input'],
},
},
{
title: 'a constructor argument',
body: twoTests(
'expect(new Box(2.4).size).toBe(2.4);',
'expect(new Box(2.5).size).toBe(2.5);',
),
want: {
status: 'bound',
reasons: [],
entries: ['call-argument new Box#0 @0'],
roles: ['0:input', '1:expected'],
},
},
{
title: 'a table key: an it.each row value read by the body',
body: "it.each([[2.4, 2], [2.5, 3]])('rounds %s', (raw, want) => { expect(roundScore(raw)).toBe(want); });",
want: {
status: 'bound',
reasons: [],
entries: ['table-key roundScore#0 @0'],
roles: ['0:input', '1:expected'],
},
},
{
title: 'a table key read through a shorthand property of a fixture',
body: "it.each([{ ratio: 2.4 }, { ratio: 2.5 }])('ratio $ratio', ({ ratio }) => { expect(configure({ ratio })).toBeGreaterThan(2); });",
want: {
status: 'bound',
reasons: [],
entries: ['table-key configure#0 ratio @0'],
roles: ['0:input'],
},
},
])('$title', async ({ body, want }) => {
const [a, b] = await sidesOf(body);
expect(summary(bindPair(a, b))).toEqual(want);
});
});
describe('bindPair: unknown and not required', () => {
it.each([
{
title: 'only the expected values differ (expected-only)',
body: [
"const tags = ['x', 'y'];",
twoTests(
"expect(formatTags(tags)).toContain('alpha');",
"expect(formatTags(tags)).toContain('beta');",
),
].join('\n'),
want: { status: 'unknown', reasons: ['expected-only'], entries: [], roles: ['0:expected'] },
},
{
title: 'a value transformed by test code',
body: twoTests(
'expect(roundScore(2.4 * 2)).toBeGreaterThan(2);',
'expect(roundScore(2.5 * 2)).toBeGreaterThan(2);',
),
want: {
status: 'unknown',
reasons: ['position 0: derived in the test (BinaryExpression)'],
entries: [],
roles: ['0:unbound'],
},
},
{
title: 'a value that reaches no call',
body: twoTests(
'const unused = 2.4; expect(roundScore(3)).toBe(3);',
'const unused = 2.5; expect(roundScore(3)).toBe(3);',
),
want: {
status: 'unknown',
reasons: ['position 0: unbound (never read)'],
entries: [],
roles: ['0:unbound'],
},
},
{
title: 'equal discarded values need no check',
body: twoTests('expect(roundScore(2.4)).toBe(2);', 'expect(roundScore(2.4)).toBe(2);'),
want: { status: 'not-required', reasons: [], entries: [], roles: [] },
},
])('$title', async ({ body, want }) => {
const [a, b] = await sidesOf(body);
expect(summary(bindPair(a, b))).toEqual(want);
});
it('is unknown when the two tests bind a position to different arguments', async () => {
const [a, b] = await sidesOf(
twoTests(
'const x = 2.4; expect(pick(x, 3)).toBe(1);',
'const x = 2.5; expect(pick(3, x)).toBe(1);',
),
);
expect(summary(bindPair(a, b))).toEqual({
status: 'unknown',
reasons: ['position 0: the two tests bind it differently'],
entries: [],
roles: ['0:input'],
});
});
it('is unknown when the discarded positions do not align', async () => {
const [a, b] = await sidesOf(
twoTests('expect(roundScore(2.4)).toBe(2);', 'expect(roundScore(2.5)).toBe(3);'),
);
expect(summary(bindPair(a, { ...b, discarded: b.discarded.slice(0, 1) }))).toEqual({
status: 'unknown',
reasons: ['discarded values do not align'],
entries: [],
roles: [],
});
});
});

Some files were not shown because too many files have changed in this diff Show more