mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-11-16 09:22:00 +00:00
* fix: recover missed test scenarios Run the full core test inventory and retrieve identical callback bodies across setup contexts. Separate assertion evidence from context and retain conservative input, path and runtime provenance checks. Restore prepared test inputs and report missing runtime collection. Add the audited recall corpus and inventory regressions. * Address PR review feedback (#3743) Correct counted nouns in the inventory report, document fully skipped modules, and include unhandled runner errors in the incomplete-collection warning. * Address PR review feedback (#3743) Force-stage explicit temporary fixtures despite inherited Git ignores, and isolate recorded-file collection-error accounting in the inventory tests. * fix(review): simplify selection and guard mixed artifacts * feat(test): detect fixture consolidation opportunities Capture source inputs used by integration tests and compare fixture clones, executed blocks, and assertion subjects and values. Report partial overlaps with the unique checks and fixture spans that consolidation must preserve. Keep cross-language structural suggestions separate from observed equality. Use pinned jscpd with direct pair verification when longer representatives hide partial clones. Retain capture gaps and native/worker limitations. Validate with 1,245 detector regressions, 28 pipeline pair controls, and 12 manually labeled comparisons. Scan 30,698 core cases and find 3,111 opportunity groups; exclude the capture's 103 failed and 89 skipped cases. * Fix scenario-map review findings (#3743) Resolve assertion aliases by lexical binding and normalize duplicate static object keys. Capture fixture files through checked descriptors with bounded reads and guaranteed cleanup. * fix: address test duplicate audit false signals * refactor(test): reuse assertion binding analysis Reuse lexical bindings within each evidence extraction while retaining assertion-specific offsets and expansion limits. Deduplicate fixture coverage endpoints numerically without string conversion. Preserve complete assertion evidence across 29,286 static instances and weighted coverage across 10,000 generated interval cases. * Fix scenario-map review findings (#3743) Bind consolidation evidence to the assertion call accepted by the check extractor, so unsupported same-line calls cannot supply incorrect subjects or expectations. Keep source provenance private and serialized checks intact. Exercise omitted plugins alongside instrumented Vitest runs. * fix: retain input-shaped scenario controls (#3743) Compare assertion subject structure separately from exact input-bearing assertion keys, so changing literal inputs retains table-row and boundary classification. Preserve selectors, callee identity and regex values. Keep the control expectations unchanged. Add regressions for literal inputs, queried subjects and nested regex arguments; bump the fingerprint engine version so reclassified findings resurface. * test: give scenario cases unique titles Include the parameterized case index so repeated input expressions and truncated audit paths cannot collide in the CI execution inventory. * fix: scope assertion subjects to their bindings Prevent same-named declarations in different files or scopes from becoming exact assertion matches. Preserve same-binding consolidation and advance the fingerprint engine to version 5. * fix: raise full-suite planning heap budget The 30,953-case replay aborts at the default 4 GiB heap. Give only the Plan step 8 GiB and document the same local invocation. The identical four coverage slices complete in 4m34s at 8.2 GiB peak RSS, retaining all 116,409 candidate pairs. * fix: stream full-suite scenario reports The full inventory produces a 663 MiB report, exceeding V8's single-string limit. Write records incrementally and stream only required fields when reading baselines and publishing issue updates. Replay preserves all 116,398 pairs from 30,953 collected cases. The report remains partial when the trace run fails. * fix: preserve test lifecycles during capture Forced test repeats changed cache and file state. Capture ordinary runs, retain runnable siblings during tracing, and keep retry-bearing files inconclusive. Preserve failure details and mark failed captures partial. Use the original JSON serializer so test mocks cannot break fixture reads. * fix: limit trace capture to candidates Keep sibling tests in both passes for suite state, while recording only candidate task IDs. This prevents unrelated timing checks and large corpus tests from failing under tracing. * ci: budget full-suite trace collection * fix: address scenario-map feedback (#3743) --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
762 lines
37 KiB
YAML
762 lines
37 KiB
YAML
name: Test Scenario Map
|
|
|
|
# Advisory report on the validated pure-JS unit tests of gitnexus/ (R13, R14, KTD13, KTD14,
|
|
# KTD17; docs/plans/2026-10-25-1700-feat-scenario-fingerprint-plan.md): duplicate tests (the same
|
|
# execution path, input data and expectation: the scenario fingerprint), and, with mutation
|
|
# evidence, tests that kill no evaluated mutant and per-file gaps. The CLI is
|
|
# gitnexus/tools/scenario-map/scenario-map.ts; its header documents the stages, flags,
|
|
# work-directory layout and exit codes. Local equivalent, from gitnexus/ (the work dir must lie
|
|
# outside the checkout; `npm run build` first: the fingerprint's integration tests load dist/):
|
|
# npm run test:scenario-map -- all --work <dir> --mutation none
|
|
#
|
|
# Triggers and mutation scope (`--mutation`, types.ts `MutationScope`):
|
|
# schedule weekly, `none`: the fingerprint only, minutes, no shard;
|
|
# workflow_dispatch input `mutation`: `none` (default), `possible-duplicates` or `all` (the
|
|
# full Phase 1 run, hours over up to 20 shards);
|
|
# pull_request TEMPORARY, remove before merge: the `scenario-map-test` label, added by
|
|
# magyargergo on PR #3686 only, runs the whole suite (no diff scope) with
|
|
# mutation `possible-duplicates`, to test every stage on CI before merge.
|
|
# When a maintainer adds the `scenario-map` label to a pull request:
|
|
# `possible-duplicates`, the deep run for a suspicious change, diff-scoped
|
|
# (`--changed-since` on the coverage slices: only pairs with a test of a changed
|
|
# test file; Layer 1 over those files, the files sharing their
|
|
# checks-and-context keys and every file with a test of no static key). A PR
|
|
# run writes no tracking issue and no baseline; its report is an artifact and
|
|
# the job summary.
|
|
#
|
|
# Jobs, each one CLI stage over one work directory ($RUNNER_TEMP/scenario-map) exchanged as
|
|
# artifacts:
|
|
# coverage one matrix job per Layer 1 slice (4, fail-fast off): the fingerprint's unit and
|
|
# integration test files, split like the CI test shards.
|
|
# plan eligibility, Layer 0, the prepared copy and run manifest, Layer 1 merged from the
|
|
# slices (a slice that did not report leaves its files unrecorded), Layer 2,
|
|
# candidates and the provisional shard plan.
|
|
# value-flow for scope `all` the PDG index (`analyze --pdg` spawns workers from dist/) and the
|
|
# static value-flow check; the trace run and the final shard plan
|
|
# (a narrower scope re-plans its target after the trace). It runs
|
|
# before the shards, so a shard isolates only the pairs value flow leaves able to be
|
|
# strong (the CLI's deviation from KTD14's parallel value-flow job).
|
|
# shards prints the final shard count (the re-plan's, or the provisional plan's when value
|
|
# flow left no usable result); it sizes the shard matrix, and 0 skips the shards.
|
|
# shard one matrix job per shard (fail-fast off): Layer 3 group runs, row deletion and
|
|
# isolation until a 300-minute deadline (KTD17). Results upload even on failure.
|
|
# merge always runs once a plan exists: merges whatever the shards and value flow left
|
|
# (a missing shard leaves the run partial and its pairs Inconclusive, R11), compares
|
|
# against the newest `scenario-map-baseline` artifact of an earlier completed run of
|
|
# this workflow on the default branch, and writes the report. Only a complete
|
|
# default-branch run uploads a new baseline (KTD13).
|
|
# upsert-issue default branch only: a complete run replaces the tracking issue's body unless
|
|
# the issue already records a newer run; a partial run comments and leaves the body
|
|
# alone. The issue is found by label in any state and never closed.
|
|
#
|
|
# Advisory: every job is continue-on-error, so nothing here fails the run or blocks a merge. Exit
|
|
# code 1 of a stage means "finished with incomplete results"; it is a warning, the job carries on
|
|
# and merge reports the run as partial. Exit code 2 (usage or setup error) fails the step.
|
|
# The full matrix can hold up to 20 runners for about 5.5 hours.
|
|
#
|
|
# Artifact names carry the run attempt. A downstream job takes the plan and value-flow artifact
|
|
# names from those jobs' outputs, so re-running one job reuses the earlier attempt's inputs; merge
|
|
# copies shard results oldest attempt first, so a later attempt of a shard wins.
|
|
#
|
|
# The controls run (the U17 fixture, every expected mutation and fingerprint verdict) is a PR job
|
|
# in mutation-runner-validation.yml.
|
|
|
|
on:
|
|
schedule:
|
|
# Mondays 04:17 UTC, outside the busy CI hours.
|
|
- cron: '17 4 * * 1'
|
|
workflow_dispatch:
|
|
inputs:
|
|
mutation:
|
|
description: 'Mutation evidence: none (fingerprint only), possible-duplicates, or all'
|
|
type: choice
|
|
options: [none, possible-duplicates, all]
|
|
default: none
|
|
seed:
|
|
description: 'Seed of every sampled choice, the spot audit included (default: the run id)'
|
|
type: string
|
|
required: false
|
|
default: ''
|
|
shards:
|
|
description: 'Number of mutation shards (default: the fewest that fit, at most 20)'
|
|
type: string
|
|
required: false
|
|
default: ''
|
|
# The deep run of one pull request: only the `scenario-map` label starts it (the plan job's
|
|
# `if`), and the TEMPORARY `scenario-map-test` label on PR #3686 by magyargergo; any other
|
|
# label event leaves every job skipped.
|
|
pull_request:
|
|
types: [labeled]
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
|
|
# Weekly and manual runs share the tracking issue and the baseline, so they share one slot and run
|
|
# one at a time; each pull request has its own slot. A run that holds hours of shard work is never
|
|
# cancelled. (GitHub keeps one pending run: a third run queued behind a pending one replaces it.)
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || 'default' }}
|
|
cancel-in-progress: false
|
|
|
|
defaults:
|
|
run:
|
|
shell: bash
|
|
working-directory: gitnexus
|
|
|
|
env:
|
|
# Stryker concurrency: the plan's cost estimate and the shards' runs (ubuntu-latest: 4 vCPUs).
|
|
SCENARIO_MAP_CONCURRENCY: '4'
|
|
# One shard's budget in the plan and its deadline in the shard job (KTD17). The shard job's
|
|
# timeout-minutes (330) leaves room for setup and the last uploads.
|
|
SCENARIO_MAP_DEADLINE_MINUTES: '300'
|
|
|
|
jobs:
|
|
# ── Coverage: Layer 1 in slices ────────────────────────────────────
|
|
coverage:
|
|
name: Coverage ${{ matrix.index }}
|
|
# TEMPORARY: the `scenario-map-test` clause (PR #3686 testing); remove before merge.
|
|
if: github.event_name != 'pull_request' || github.event.label.name == 'scenario-map' || (github.event.label.name == 'scenario-map-test' && github.event.sender.login == 'magyargergo' && github.event.pull_request.number == 3686)
|
|
runs-on: ubuntu-latest
|
|
# Four slices balanced like the CI test shards (test/helpers/shard-balance.ts): the whole unit
|
|
# and integration suite runs with coverage in three 7-8 minute shards of the test workflow
|
|
# (ci.yml run 30014110469). Layer 1 repeats only test bodies and snapshots per test (about 5%
|
|
# of an integration file whose work sits in beforeAll), and the prefilter drops a quarter.
|
|
timeout-minutes: 75
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
index: [0, 1, 2, 3]
|
|
steps:
|
|
# A pull request's merge commit needs its first parent (the base) for the diff scope.
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
fetch-depth: ${{ github.event_name == 'pull_request' && 2 || 1 }}
|
|
# Integration tests load dist/ (the built CLI and its worker scripts).
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
with:
|
|
build: 'true'
|
|
- name: Run the slice
|
|
timeout-minutes: 60
|
|
env:
|
|
INDEX: ${{ matrix.index }}
|
|
OF: ${{ strategy.job-total }}
|
|
# The plan's scope (its "Choose the mutation scope" step): unless `all`, Layer 1 skips the
|
|
# files whose tests all have a checks-and-context key no other test shares.
|
|
MUTATION: ${{ github.event_name == 'pull_request' && 'possible-duplicates' || inputs.mutation || 'none' }}
|
|
EVENT: ${{ github.event_name }}
|
|
LABEL: ${{ github.event.label.name }}
|
|
run: |
|
|
args=(--index "$INDEX" --of "$OF" --mutation "$MUTATION"
|
|
--work "$RUNNER_TEMP/scenario-map" --concurrency "$SCENARIO_MAP_CONCURRENCY")
|
|
# The merge commit's first parent is the base: the pull request's own changes. The
|
|
# TEMPORARY `scenario-map-test` run covers the whole suite.
|
|
if [ "$EVENT" = "pull_request" ] && [ "$LABEL" = "scenario-map" ]; then
|
|
args+=(--changed-since 'HEAD^1')
|
|
fi
|
|
status=0
|
|
npm run test:scenario-map -- coverage "${args[@]}" || status=$?
|
|
if [ "$status" -eq 1 ]; then
|
|
echo "::warning::a test file of slice $INDEX failed or wrote no record; the report counts it"
|
|
elif [ "$status" -ne 0 ]; then
|
|
exit "$status"
|
|
fi
|
|
- name: Upload the slice
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-coverage-${{ github.run_attempt }}-${{ matrix.index }}
|
|
path: ${{ runner.temp }}/scenario-map/coverage-slice-${{ matrix.index }}.json
|
|
if-no-files-found: error
|
|
retention-days: 14
|
|
|
|
# ── Plan: Layers 0-2, candidates, shard plan, run manifest ─────────
|
|
plan:
|
|
name: Plan
|
|
needs: coverage
|
|
# TEMPORARY: the `scenario-map-test` clause (PR #3686 testing); remove before merge.
|
|
if: ${{ !cancelled() && (github.event_name != 'pull_request' || github.event.label.name == 'scenario-map' || (github.event.label.name == 'scenario-map-test' && github.event.sender.login == 'magyargergo' && github.event.pull_request.number == 3686)) }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 60
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
artifact: ${{ steps.artifact.outputs.name }}
|
|
mutation: ${{ steps.scope.outputs.mutation }}
|
|
# Scheduled runs always run on the default branch; their event payload has no repository.
|
|
default_branch: ${{ github.event_name == 'schedule' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
|
|
steps:
|
|
# persist-credentials: false — the jobs upload artifacts and never push.
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
# Layer 1 comes from the slices: the plan runs no test and needs no dist/.
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
- name: Choose the mutation scope
|
|
id: scope
|
|
env:
|
|
MUTATION: ${{ github.event_name == 'pull_request' && 'possible-duplicates' || inputs.mutation || 'none' }}
|
|
run: echo "mutation=$MUTATION" >> "$GITHUB_OUTPUT"
|
|
# No slice at all fails the plan (nothing to merge); a missing slice leaves its files
|
|
# unrecorded in the report.
|
|
- name: Download the coverage slices
|
|
continue-on-error: true
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
pattern: scenario-map-coverage-*
|
|
path: ${{ runner.temp }}/scenario-map-coverage
|
|
- name: Place the coverage slices
|
|
run: |
|
|
shopt -s nullglob
|
|
root="$RUNNER_TEMP/scenario-map-coverage"
|
|
dest="$RUNNER_TEMP/scenario-map-slices"
|
|
mkdir -p "$dest"
|
|
found=("$root"/scenario-map-coverage-*)
|
|
# download-artifact unpacks a pattern that matches ONE artifact into the path itself.
|
|
if [ "${#found[@]}" -eq 0 ] && [ -d "$root" ]; then found=("$root"); fi
|
|
# scenario-map-coverage-<attempt>-<index>: oldest attempt first, so a later attempt wins.
|
|
while IFS= read -r dir; do
|
|
[ -n "$dir" ] || continue
|
|
for file in "$dir"/coverage-slice-*.json; do cp "$file" "$dest/"; done
|
|
done < <(printf '%s\n' "${found[@]}" | sort -V)
|
|
ls "$dest"
|
|
- name: Plan the run
|
|
env:
|
|
# Full-suite Layer 2 exceeds Node's default ~4 GiB heap. Budget 8 GiB
|
|
# for this step, leaving room on the 16 GB runner for native memory.
|
|
NODE_OPTIONS: --max-old-space-size=8192
|
|
SEED: ${{ inputs.seed || github.run_id }}
|
|
SHARDS: ${{ inputs.shards }}
|
|
MUTATION: ${{ steps.scope.outputs.mutation }}
|
|
run: |
|
|
# The slices carry Layer 1, and a pull request's diff scope with it.
|
|
args=(--work "$RUNNER_TEMP/scenario-map" --seed "$SEED" --mutation "$MUTATION"
|
|
--concurrency "$SCENARIO_MAP_CONCURRENCY"
|
|
--deadline-minutes "$SCENARIO_MAP_DEADLINE_MINUTES"
|
|
--coverage "$RUNNER_TEMP/scenario-map-slices")
|
|
if [ -n "$SHARDS" ]; then args+=(--shards "$SHARDS"); fi
|
|
npm run test:scenario-map -- plan "${args[@]}"
|
|
- name: Upload the plan
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-plan-${{ github.run_attempt }}
|
|
path: |
|
|
${{ runner.temp }}/scenario-map/plan.json
|
|
${{ runner.temp }}/scenario-map/scenario-run-manifest.json
|
|
${{ runner.temp }}/scenario-map/layer0.json
|
|
${{ runner.temp }}/scenario-map/layer2.json
|
|
${{ runner.temp }}/scenario-map/consolidation.json
|
|
${{ runner.temp }}/scenario-map/consolidation.html
|
|
${{ runner.temp }}/scenario-map/fixtures.json
|
|
${{ runner.temp }}/scenario-map/coverage/coverage.json
|
|
if-no-files-found: error
|
|
retention-days: 14
|
|
- name: Name the plan artifact
|
|
id: artifact
|
|
env:
|
|
NAME: scenario-map-plan-${{ github.run_attempt }}
|
|
run: echo "name=$NAME" >> "$GITHUB_OUTPUT"
|
|
|
|
# ── Value flow: PDG static check, trace run, final shard plan ──────
|
|
value-flow:
|
|
name: Value flow
|
|
needs: plan
|
|
if: ${{ !cancelled() && needs.plan.outputs.artifact != '' }}
|
|
runs-on: ubuntu-latest
|
|
# Full-suite tracing runs candidate files twice, including their runnable siblings.
|
|
# The 31k-case inventory exceeded 50 minutes; leave room for both passes and uploads.
|
|
timeout-minutes: 100
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
artifact: ${{ steps.artifact.outputs.name }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
# The trace runs integration tests, and the PDG index of scope `all` spawns workers: dist/.
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
with:
|
|
build: 'true'
|
|
- name: Download the plan
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.plan.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
- name: Run value flow
|
|
timeout-minutes: 90
|
|
env:
|
|
GITNEXUS_WORKER_READY_TIMEOUT_MS: '120000'
|
|
run: |
|
|
status=0
|
|
npm run test:scenario-map -- value-flow --work "$RUNNER_TEMP/scenario-map" || status=$?
|
|
if [ "$status" -eq 1 ]; then
|
|
echo "::warning::value flow is degraded, or its trace did not run or was invalid; the report says so"
|
|
elif [ "$status" -ne 0 ]; then
|
|
exit "$status"
|
|
fi
|
|
- name: Upload the value-flow result
|
|
id: upload
|
|
if: always()
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-value-flow-${{ github.run_attempt }}
|
|
path: |
|
|
${{ runner.temp }}/scenario-map/value-flow.json
|
|
${{ runner.temp }}/scenario-map/value-flow/pdg/analyze.log
|
|
retention-days: 14
|
|
# Named only when value-flow.json was uploaded: without it, shards and merge fall back to
|
|
# the provisional plan and merge reports value flow missing.
|
|
- name: Name the value-flow artifact
|
|
id: artifact
|
|
if: always() && steps.upload.outcome == 'success'
|
|
env:
|
|
NAME: scenario-map-value-flow-${{ github.run_attempt }}
|
|
run: |
|
|
if [ -f "$RUNNER_TEMP/scenario-map/value-flow.json" ]; then
|
|
echo "name=$NAME" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
# ── Shard count: sizes the matrix ──────────────────────────────────
|
|
shards:
|
|
name: Shard count
|
|
needs: [plan, value-flow]
|
|
# Scope `none` plans no shard.
|
|
if: ${{ !cancelled() && needs.plan.outputs.artifact != '' && needs.plan.outputs.mutation != 'none' }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
count: ${{ steps.count.outputs.count }}
|
|
matrix: ${{ steps.count.outputs.matrix }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
- name: Download the plan
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.plan.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
- name: Download the value-flow result
|
|
if: needs.value-flow.outputs.artifact != ''
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.value-flow.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
# --silent: the stage prints the count and nothing else; npm's banner would precede it.
|
|
- name: Read the final shard count
|
|
id: count
|
|
run: |
|
|
count="$(npm run --silent test:scenario-map -- shards --work "$RUNNER_TEMP/scenario-map")"
|
|
if ! [[ "$count" =~ ^[0-9]+$ ]]; then
|
|
echo "::error::the shards stage printed '$count', not a shard count"
|
|
exit 1
|
|
fi
|
|
echo "count=$count" >> "$GITHUB_OUTPUT"
|
|
echo "matrix=$(jq -cn --argjson n "$count" '[range($n)]')" >> "$GITHUB_OUTPUT"
|
|
echo "Final shard plan: $count shard(s)"
|
|
|
|
# ── Layer 3 shards ─────────────────────────────────────────────────
|
|
shard:
|
|
name: Shard ${{ matrix.index }}
|
|
needs: [plan, value-flow, shards]
|
|
if: ${{ !cancelled() && needs.shards.outputs.matrix != '' && needs.shards.outputs.count != '0' }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 330
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
# Leave runner slots for PR CI on the default plan (about 9 shards at concurrency 4).
|
|
max-parallel: 10
|
|
matrix:
|
|
index: ${{ fromJSON(needs.shards.outputs.matrix) }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
- name: Download the plan
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.plan.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
- name: Download the value-flow result
|
|
if: needs.value-flow.outputs.artifact != ''
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.value-flow.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
# The shard verifies its own prepared copy against the run manifest; a mismatch runs
|
|
# nothing and leaves its groups' pairs Inconclusive. The step timeout sits inside the job's,
|
|
# so a shard that overruns its deadline still uploads what it finished.
|
|
- name: Run the shard
|
|
timeout-minutes: 315
|
|
env:
|
|
INDEX: ${{ matrix.index }}
|
|
COUNT: ${{ needs.shards.outputs.count }}
|
|
run: |
|
|
status=0
|
|
npm run test:scenario-map -- shard --work "$RUNNER_TEMP/scenario-map" \
|
|
--index "$INDEX" --of "$COUNT" \
|
|
--deadline-minutes "$SCENARIO_MAP_DEADLINE_MINUTES" \
|
|
--concurrency "$SCENARIO_MAP_CONCURRENCY" || status=$?
|
|
if [ "$status" -eq 1 ]; then
|
|
echo "::warning::shard $INDEX finished with incomplete results; the report says which"
|
|
elif [ "$status" -ne 0 ]; then
|
|
exit "$status"
|
|
fi
|
|
# Every group result is written as soon as it exists: upload whatever the shard finished,
|
|
# even after a failure.
|
|
- name: Upload the shard results
|
|
if: always()
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-shard-${{ github.run_attempt }}-${{ matrix.index }}
|
|
path: |
|
|
${{ runner.temp }}/scenario-map/shard-${{ matrix.index }}.json
|
|
${{ runner.temp }}/scenario-map/row-deletion-shard-${{ matrix.index }}.json
|
|
${{ runner.temp }}/scenario-map/observations/
|
|
${{ runner.temp }}/scenario-map/batches/
|
|
${{ runner.temp }}/scenario-map/isolation/
|
|
${{ runner.temp }}/scenario-map/groups/*/stryker/stryker.log
|
|
${{ runner.temp }}/scenario-map/groups/*/*/stryker/stryker.log
|
|
retention-days: 14
|
|
|
|
# ── Merge: verdicts and the report ─────────────────────────────────
|
|
merge:
|
|
name: Merge and report
|
|
needs: [plan, value-flow, shard]
|
|
if: ${{ always() && needs.plan.outputs.artifact != '' }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
actions: read # list and download an earlier run's scenario-map-baseline artifact
|
|
outputs:
|
|
artifact: ${{ steps.artifact.outputs.name }}
|
|
complete: ${{ steps.merge.outputs.complete }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
- name: Download the plan
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.plan.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
- name: Download the value-flow result
|
|
if: needs.value-flow.outputs.artifact != ''
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.value-flow.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map
|
|
# No shard result at all is a valid, partial outcome: merge reports every shard missing.
|
|
- name: Download the shard results
|
|
continue-on-error: true
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
pattern: scenario-map-shard-*
|
|
path: ${{ runner.temp }}/scenario-map-shards
|
|
- name: Place the shard results
|
|
run: |
|
|
shopt -s nullglob
|
|
work="$RUNNER_TEMP/scenario-map"
|
|
root="$RUNNER_TEMP/scenario-map-shards"
|
|
found=("$root"/scenario-map-shard-*)
|
|
# download-artifact unpacks a pattern that matches ONE artifact into the path itself, not
|
|
# into a directory named after it (a one-shard plan): the path is then that artifact.
|
|
if [ "${#found[@]}" -eq 0 ] && [ -d "$root" ]; then found=("$root"); fi
|
|
# scenario-map-shard-<attempt>-<index>: oldest attempt first, so a later attempt wins.
|
|
while IFS= read -r dir; do
|
|
[ -n "$dir" ] || continue
|
|
echo "Using $(basename "$dir")"
|
|
cp -R "$dir/." "$work/"
|
|
done < <(printf '%s\n' "${found[@]}" | sort -V)
|
|
# KTD13: the newest non-expired `scenario-map-baseline` of an earlier completed run of THIS
|
|
# workflow on the default branch; only complete default-branch runs upload one. Never a
|
|
# repository-wide artifact listing: it holds fork pull_request runs too, and a fork branch
|
|
# may carry the default branch's name. Only schedule and workflow_dispatch runs count (a
|
|
# labelled pull request run never does), and each run's head repository is checked as well. None found: "first run" banner. A
|
|
# failure here must not cost the run its report: merge then runs without a baseline, and a
|
|
# baseline merge cannot read gets the incompatible-baseline banner.
|
|
- name: Fetch the baseline
|
|
id: baseline
|
|
continue-on-error: true
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
DEFAULT_BRANCH: ${{ github.event.repository.default_branch || github.ref_name }}
|
|
run: |
|
|
dir="$RUNNER_TEMP/scenario-map-baseline"
|
|
mkdir -p "$dir"
|
|
if ! gh api --method GET \
|
|
"repos/$GITHUB_REPOSITORY/actions/workflows/test-scenario-map.yml/runs" \
|
|
-f branch="$DEFAULT_BRANCH" -f status=completed -f per_page=20 > "$dir/runs.json"; then
|
|
echo "::warning::could not list the earlier runs; the report shows no baseline"
|
|
exit 0
|
|
fi
|
|
id=""
|
|
# Newest first: this workflow's scheduled or manual runs other than this one, whose
|
|
# repository and head repository are this repository.
|
|
while IFS= read -r run_id; do
|
|
[ -n "$run_id" ] || continue
|
|
if ! gh api "repos/$GITHUB_REPOSITORY/actions/runs/$run_id/artifacts?name=scenario-map-baseline" \
|
|
< /dev/null > "$dir/artifacts.json"; then
|
|
continue
|
|
fi
|
|
id="$(jq -r '[.artifacts[] | select(.expired | not)]
|
|
| sort_by(.created_at) | last | .id // empty' "$dir/artifacts.json")"
|
|
if [ -n "$id" ]; then
|
|
echo "Baseline run: $run_id"
|
|
break
|
|
fi
|
|
done < <(jq -r --argjson run "$GITHUB_RUN_ID" --argjson repo "$GITHUB_REPOSITORY_ID" '
|
|
[.workflow_runs[]
|
|
| select(.id != $run
|
|
and (.event == "schedule" or .event == "workflow_dispatch")
|
|
and .repository.id == $repo
|
|
and .head_repository.id == $repo)]
|
|
| sort_by(.created_at) | reverse | .[].id' "$dir/runs.json")
|
|
if [ -z "$id" ]; then
|
|
echo "No earlier scenario-map-baseline artifact: first run, no baseline."
|
|
exit 0
|
|
fi
|
|
if ! gh api "repos/$GITHUB_REPOSITORY/actions/artifacts/$id/zip" > "$dir/baseline.zip" \
|
|
|| ! unzip -q -o "$dir/baseline.zip" -d "$dir/report" \
|
|
|| [ ! -f "$dir/report/scenario-map-report.json" ]; then
|
|
echo "::warning::could not fetch baseline artifact $id; the report shows no baseline"
|
|
exit 0
|
|
fi
|
|
echo "Baseline: artifact $id"
|
|
echo "report=$dir/report/scenario-map-report.json" >> "$GITHUB_OUTPUT"
|
|
- name: Merge
|
|
id: merge
|
|
env:
|
|
BASELINE: ${{ steps.baseline.outputs.report }}
|
|
ON_DEFAULT_BRANCH: ${{ needs.plan.outputs.default_branch }}
|
|
RUN_ATTEMPT: ${{ github.run_attempt }}
|
|
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
REF: ${{ github.ref }}
|
|
run: |
|
|
work="$RUNNER_TEMP/scenario-map"
|
|
args=(--work "$work" --run-id "$GITHUB_RUN_ID" --run-attempt "$RUN_ATTEMPT"
|
|
--run-url "$RUN_URL" --ref "$REF")
|
|
if [ -n "$BASELINE" ]; then args+=(--baseline "$BASELINE"); fi
|
|
if [ "$ON_DEFAULT_BRANCH" = 'true' ]; then args+=(--default-branch); fi
|
|
status=0
|
|
npm run test:scenario-map -- merge "${args[@]}" || status=$?
|
|
if [ "$status" -eq 1 ]; then
|
|
echo "::warning::partial report: some shard or value-flow result is missing"
|
|
elif [ "$status" -ne 0 ]; then
|
|
exit "$status"
|
|
fi
|
|
complete="$(jq -r '.completeness.complete == true' "$work/scenario-map-report.json")"
|
|
echo "complete=$complete" >> "$GITHUB_OUTPUT"
|
|
- name: Upload the report
|
|
id: upload
|
|
if: always()
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-report-${{ github.run_attempt }}
|
|
path: |
|
|
${{ runner.temp }}/scenario-map/scenario-map-report.json
|
|
${{ runner.temp }}/scenario-map/scenario-map-report.md
|
|
${{ runner.temp }}/scenario-map/issue-body.md
|
|
retention-days: 90
|
|
# The tracking-issue view on the run page: a pull request run's only rendered report.
|
|
- name: Summarise the report
|
|
if: always()
|
|
run: |
|
|
body="$RUNNER_TEMP/scenario-map/issue-body.md"
|
|
if [ -f "$body" ]; then tail -n +2 "$body" >> "$GITHUB_STEP_SUMMARY"; fi
|
|
- name: Name the report artifact
|
|
id: artifact
|
|
if: always() && steps.upload.outcome == 'success'
|
|
env:
|
|
NAME: scenario-map-report-${{ github.run_attempt }}
|
|
run: |
|
|
if [ -f "$RUNNER_TEMP/scenario-map/scenario-map-report.json" ]; then
|
|
echo "name=$NAME" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
# overwrite: a re-run attempt replaces this run's own baseline instead of failing on it.
|
|
- name: Upload the baseline
|
|
if: steps.merge.outputs.complete == 'true' && needs.plan.outputs.default_branch == 'true'
|
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
with:
|
|
name: scenario-map-baseline
|
|
path: ${{ runner.temp }}/scenario-map/scenario-map-report.json
|
|
if-no-files-found: error
|
|
retention-days: 90
|
|
overwrite: true
|
|
|
|
# ── Tracking issue (R13) ───────────────────────────────────────────
|
|
# The only job with a write scope, and only on the default branch. No install scripts run here
|
|
# (lifecycle-scripts: 'false'): tsx alone runs the decision.
|
|
upsert-issue:
|
|
name: Upsert tracking issue
|
|
needs: [plan, merge]
|
|
if: ${{ !cancelled() && needs.plan.outputs.default_branch == 'true' && needs.merge.outputs.artifact != '' }}
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
continue-on-error: true
|
|
permissions:
|
|
contents: read
|
|
issues: write # find, open, update or comment on the tracking issue (and its label)
|
|
env:
|
|
LABEL: test-scenario-map
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
with:
|
|
lifecycle-scripts: 'false'
|
|
- name: Download the report
|
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
with:
|
|
name: ${{ needs.merge.outputs.artifact }}
|
|
path: ${{ runner.temp }}/scenario-map-report
|
|
- name: Find the tracking issue
|
|
id: find
|
|
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
|
with:
|
|
script: |
|
|
const fs = require('fs');
|
|
const path = require('path');
|
|
const issues = await github.paginate(github.rest.issues.listForRepo, {
|
|
owner: context.repo.owner,
|
|
repo: context.repo.repo,
|
|
labels: process.env.LABEL,
|
|
state: 'all',
|
|
per_page: 100,
|
|
});
|
|
// Any state; the oldest one when several carry the label.
|
|
const tracking = issues
|
|
.filter((issue) => !issue.pull_request)
|
|
.sort((a, b) => a.number - b.number)[0];
|
|
if (tracking === undefined) {
|
|
core.info('No tracking issue yet.');
|
|
return;
|
|
}
|
|
const issueDir = path.join(process.env.RUNNER_TEMP, 'scenario-map-issue');
|
|
fs.mkdirSync(issueDir, { recursive: true });
|
|
fs.writeFileSync(path.join(issueDir, 'existing-body.md'), tracking.body ?? '');
|
|
core.setOutput('number', String(tracking.number));
|
|
core.info(`Tracking issue #${tracking.number} (${tracking.state}).`);
|
|
# The decision is issue-body.ts's own R13/KTD13 rule (shouldUpdateIssue): a complete
|
|
# default-branch run updates unless the body's first-line run marker records a newer run.
|
|
- name: Decide the update
|
|
id: decide
|
|
env:
|
|
ISSUE_NUMBER: ${{ steps.find.outputs.number }}
|
|
run: |
|
|
node --import tsx --input-type=module <<'EOF'
|
|
import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
|
import path from 'node:path';
|
|
import { shouldUpdateIssue } from './tools/scenario-map/report/issue-body.ts';
|
|
import { readReportFields } from './tools/scenario-map/report/json.ts';
|
|
import { completenessLine, degradedLine, runLink } from './tools/scenario-map/report/markdown.ts';
|
|
|
|
const { ISSUE_NUMBER, GITHUB_OUTPUT, RUNNER_TEMP } = process.env;
|
|
const reportDir = path.join(RUNNER_TEMP, 'scenario-map-report');
|
|
const issueDir = path.join(RUNNER_TEMP, 'scenario-map-issue');
|
|
const report = await readReportFields(
|
|
path.join(reportDir, 'scenario-map-report.json'), ['run', 'completeness'],
|
|
);
|
|
const existing = ISSUE_NUMBER
|
|
? readFileSync(path.join(issueDir, 'existing-body.md'), 'utf8')
|
|
: null;
|
|
let action = 'skip';
|
|
if (report.completeness.complete) {
|
|
if (shouldUpdateIssue(report, existing)) action = existing === null ? 'create' : 'update';
|
|
else console.log('The issue already records a newer run: body left alone.');
|
|
} else if (existing !== null) {
|
|
action = 'comment';
|
|
const lines = [
|
|
`Partial ${runLink(report.run)}: the issue body is left unchanged; only a complete run updates it.`,
|
|
'',
|
|
`- ${completenessLine(report)}`,
|
|
`- ${degradedLine(report)}`,
|
|
'',
|
|
"The run's report artifact holds every result it produced.",
|
|
];
|
|
mkdirSync(issueDir, { recursive: true });
|
|
writeFileSync(path.join(issueDir, 'comment.md'), `${lines.join('\n')}\n`);
|
|
} else {
|
|
console.log('Partial run and no tracking issue yet: nothing to comment on.');
|
|
}
|
|
appendFileSync(GITHUB_OUTPUT, `action=${action}\n`);
|
|
console.log(`Tracking issue action: ${action}`);
|
|
EOF
|
|
# Bodies are read from files, never passed through env or outputs. Never closes the issue.
|
|
- name: Apply the update
|
|
if: steps.decide.outputs.action != 'skip'
|
|
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
|
env:
|
|
ACTION: ${{ steps.decide.outputs.action }}
|
|
ISSUE_NUMBER: ${{ steps.find.outputs.number }}
|
|
with:
|
|
script: |
|
|
const fs = require('fs');
|
|
const path = require('path');
|
|
const { owner, repo } = context.repo;
|
|
const label = process.env.LABEL;
|
|
const read = (dir, file) =>
|
|
fs.readFileSync(path.join(process.env.RUNNER_TEMP, dir, file), 'utf8');
|
|
switch (process.env.ACTION) {
|
|
case 'create': {
|
|
try {
|
|
await github.rest.issues.getLabel({ owner, repo, name: label });
|
|
} catch (error) {
|
|
if (error.status !== 404) throw error;
|
|
await github.rest.issues.createLabel({
|
|
owner,
|
|
repo,
|
|
name: label,
|
|
color: 'c5def5',
|
|
description: 'Tracking issue of the weekly test scenario map',
|
|
});
|
|
}
|
|
const { data } = await github.rest.issues.create({
|
|
owner,
|
|
repo,
|
|
title: 'Test scenario map: repeat candidates, weak tests and gaps',
|
|
body: read('scenario-map-report', 'issue-body.md'),
|
|
labels: [label],
|
|
});
|
|
core.info(`Opened tracking issue #${data.number}.`);
|
|
break;
|
|
}
|
|
case 'update':
|
|
await github.rest.issues.update({
|
|
owner,
|
|
repo,
|
|
issue_number: Number(process.env.ISSUE_NUMBER),
|
|
body: read('scenario-map-report', 'issue-body.md'),
|
|
});
|
|
core.info(`Updated the body of #${process.env.ISSUE_NUMBER}.`);
|
|
break;
|
|
case 'comment':
|
|
await github.rest.issues.createComment({
|
|
owner,
|
|
repo,
|
|
issue_number: Number(process.env.ISSUE_NUMBER),
|
|
body: read('scenario-map-issue', 'comment.md'),
|
|
});
|
|
core.info(`Commented on #${process.env.ISSUE_NUMBER}.`);
|
|
break;
|
|
default:
|
|
throw new Error(`unknown action ${process.env.ACTION}`);
|
|
}
|