mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-09 03:17:54 +00:00
fix(ci): align tree-sitter readiness + grammar-update workflows on a shared manifest (#858) (#2187)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* chore(ci): add shared vendored-grammars manifest; monitor reads it .github/vendored-grammars.json is the single source of truth for the vendored tree-sitter grammars (c/swift/kotlin/dart/proto): name, upstream coords, and policy holds. update-vendored-grammars.mjs now builds its GRAMMARS map from the manifest (behavior-preserving — same exported shape). Adds manifest-agreement tests so the loader can't silently skew from the file. * fix(ci): classify vendored grammars from manifest, drop bare "?" (#858) The readiness report decided "is this vendored?" via is_vendored_pin (a file: package.json spec) — but the 5 vendored grammars aren't in package.json, so they were misrouted through the npm path and rendered bare "?" for ABI (read from an empty node_modules), plus a spurious "? (fetch failed)" for github-only proto. Now vendored grammars are classified by membership in the shared manifest and their ABI is read from gitnexus/vendor/<name>/src/parser.c (always in a checkout). github-only vendored grammars skip the npm peer-dep fetch; the tree-sitter-c hold is surfaced from the manifest (held, not plain "Ready"); and every remaining unintrospectable value renders a labeled token, never a bare "?". --assert-current now covers the vendored grammars too instead of skipping them. Adds a stdlib unittest suite incl. a manifest⇄vendor-dir consistency guard. * docs(ci): document the shared vendored-grammars manifest Both tree-sitter workflow headers now point at .github/vendored-grammars.json as the shared source of truth; the readiness workflow gains a PR-path trigger on the manifest + test, and runs the readiness unit tests on validation events. CONTRIBUTING.md documents the manifest contract under CI automation contracts. * fix(review): apply autofix feedback - Guard manifest reads in both scripts with a clear error (was an opaque module-import traceback that crashed the script and test collection). - Never render a bare "?": relabel the npm-path ABI/version/peer sentinels and the vendored upstream-ABI miss to labeled tokens; the report is now ?-free regardless of node_modules/network, and the test is hermetic. - Add a VENDORED_NAMES ⊆ GRAMMARS guard + manifest-missing error test. - Drop now-dead is_vendored_pin/is_vendored/_(vendored)_. - Compose held + out-of-range vendored blocker reasons instead of overwriting. - Reword the shared-manifest docs to not over-claim shared upstream coords. * fix(ci): apply root prettier formatting to mjs + ts test The quality/format gate runs root `prettier --check .` (printWidth 100, the gitnexus-local config differs and falsely passed locally). * test(ci): make both tree-sitter scripts testable offline The scripts hit live npm/GitHub, which makes the report run flaky and the monitor's detect/apply logic untestable. Add hermetic seams: - readiness: --offline flag (+ GITNEXUS_TS_READINESS_OFFLINE env) no-ops the npm registry + upstream fetches; the report renders deterministically (vendored ABIs from the repo, npm columns marked 'offline', no bare '?'). 3 tests assert an offline run touches ZERO network (urlopen patched to raise). - monitor: detect() and apply() accept injected deps (vendoredVersion/ resolveUpstream/fetchSource/readAbi) so the newer/ABI/hold gating runs offline with fixtures; apply gains --dry-run (validates but writes nothing). 6 tests cover newer/same-version/held-c/ABI-15/applicable + a no-mutation dry-run. * fix(review): keep --assert-current hermetic + harden the no-bare-? invariant Tri-review findings (PR #2187): - P2 REGRESSION: --assert-current (documented 'hermetic and offline', run in CI without --offline) routed the 5 vendored grammars through vendored_drift_summary, which fetches upstream parser.c + commit sha — 10 discarded network calls per run. Fix: read the vendored ABI locally via a new vendored_abi_from_repo() helper (also used by vendored_drift_summary). Now verifiably network-free. - Unify the upstream-ABI miss sentinel: prose said 'n/a (generated at build)' while the matrix said 'n/a' — and 'generated at build' is a wrong cause (swift HAS a committed parser.c). Both now render neutral 'n/a'. - Fix the stale assert_current docstring claiming swift is prebuilt-only/no parser.c. - Guard the last latent bare-? path (vendor package.json missing 'version'). Tests: AssertCurrent (network-free guard + out-of-range via the new injection point), malformed-JSON manifest, detect() error-path, explicit npm/github undefined assertions. 17 Python + 15 vitest, all hermetic. * fix(review): use a single unittest import style (CodeQL 753) CodeQL py/import-and-import-from flagged `import unittest` + `from unittest import mock`. Collapse to `from unittest import TestCase, main, mock`. * fix(review): explicit raise in _matrix_row (CodeQL 754) CodeQL py/mixed-returns flagged the implicit fall-through after self.fail() (which it doesn't model as NoReturn). End with an explicit raise AssertionError. * test(review): replace non-null assertions with a must() guard @typescript-eslint/no-non-null-assertion flagged 4 `!` operators. Add a narrowing must<T>(value, message) helper (throws on undefined) and a named baseResolveUpstream, removing every non-null assertion. * fix(review): unguessable heredoc delimiter for the report output The report embeds the manifest `hold` field (fork-PR-editable); a fixed DRIFT_EOF delimiter in a hold value could close the $GITHUB_OUTPUT heredoc early and inject output keys. Use DRIFT_EOF_$(openssl rand -hex 16) — a value the report cannot contain. (Randomized delimiter over base64: keeps REPORT raw markdown, no consumer-side decode.) * fix(review): scope issues:write to scheduled runs (two-job split) GitHub Actions has no step-level permissions, so the only way to keep PR runs (incl. forks) from receiving `issues: write` is to split the job. A `report` job (contents:read, all events) renders the report + the PR `:⚠️:` and exposes report/exit_code as job outputs; a schedule-only `upsert-issue` job (needs: report, issues:write, no checkout) consumes them for the issue upsert + close. The 'Check upgrade readiness' check name is preserved. * fix(review): launder npm-version '?' in disposition prose The disposition bucket prose interpolated r['npm_version'] raw, so a successful 200 npm /latest response lacking a 'version' key would render a bare '?' (the matrix cell already laundered it). Add npm_version_label ('unknown' for '?') and use it in all five bucket renderers. Test a version-less npm response. * refactor(review): load_vendored_manifest returns only the consumed 'hold' The readiness script reads only the grammar names + 'hold'; the 'key' and 'upstream' fields were phantom data (upstream-drift coords live in the script's own GRAMMARS map). Narrow the return to {hold}. * fix(review): unify detect()/apply() 'newer' check for github grammars detect() compared the bare sha7 while apply() compared up.version (the full <base>-g<sha7> provenance string apply() also writes). After the bot re-vendored a github grammar once, detect() reported a perpetual false 'update available' while apply() correctly saw 'already current' — a noisy job summary + wasted --apply subprocess (the PR-exists guard absorbed it before any duplicate PR). Extract a shared isNewer(up, have) helper used by both. Tests cover equal- provenance (false), first-vendoring plain-version (true, not suppressed), and sha-advanced (true). Coupled with U12 (the detect⇄apply agreement assertion lives there once apply()'s not-newer path returns instead of process.exit). * test(review): cover main()'s out-of-range + prebuilt-only vendored ABI branches main()'s vendored-ABI classification reads through vendored_abi_from_repo (the local-read seam --assert-current uses), so patching it drives the 'Vendored (ABI out of range)' blocker branch and the prebuilt-only (vendored_abi None → 'prebuilt' cell, not '?') branch — neither reachable today since all 5 vendor dirs ship parser.c at ABI 14. * test(review): monitor-side manifest⇄vendor-dir consistency guard Mirror the Python consistency guard on the monitor side — the monitor consumes the same manifest and is the side that WRITES files from manifest `name`, so manifest/vendor-dir drift must fail CI here too. * fix(review): validate grammar names at manifest load (path-traversal guard) The manifest `name` is joined into gitnexus/vendor/<name> paths in both scripts (and apply() WRITES there), so reject any name not matching tree-sitter-[a-z0-9-]+ at the single load chokepoint — defense-in-depth even though the live trust boundary already prevents exploitation. loadManifestGrammars gains an injectable `raw` arg + export for testing; tests reject a '../etc' name in both scripts. * refactor(review): apply() throws ApplyExit; CLI maps to exit codes apply()'s 4 process.exit calls killed the vitest worker, blocking in-process tests of its error branches. Replace them with a thrown ApplyExit{code}; the not-newer (already-current) path returns `have` instead of exit(0). The isMain CLI block try/catches and maps ApplyExit.code → process.exit, so the monitor's subprocess contract (exit 0/2/3) is byte-identical (verified via subprocess smoke). Tests cover unknown-key=2, held=3, ABI-reject=3, and not-newer (returns current, no throw, no write). * refactor(review): extract vendored render helper; trim docstrings (<1000 lines) Extract the 'Vendored parsers' prose render into _render_vendored_section() so main() coordinates named phases rather than inlining a ~450-line monolith, and condense the most verbose docstrings/comments. The script drops from 1092 to 999 lines (under the 1000 bar the maintainability review flagged). Behavior-preserving: the deterministic --offline render is byte-identical before/after (verified in-place), --assert-current still passes, and the full unit suite is green. * fix(review): row-diff regex captures only the Status cell The change-detection regex captured the whole row tail as group 2, so any non-status cell drift (e.g. an upstream-ABI bump) emitted a false-positive 'change' line. Capture only the Status cell ([^|]+? before the final |$). The workflow parseRows regex and the Python _ROW_DIFF_RE stay byte-identical; the stability test now asserts group 2 is the status string (e.g. c → 'Vendored — held') and contains no pipe. * fix(ci): hoist intro string out of the list literal (CodeQL 755) The U13 extraction moved the 'Vendored parsers' intro paragraph (implicitly concatenated string literals) INTO a list literal, tripping CodeQL py/implicit-string-concatenation-in-list (reads as a possibly-missing comma between elements). Hoist it into a parenthesized `intro` variable. Render is byte-identical.
This commit is contained in:
parent
89ffa71a52
commit
96dc368d96
8 changed files with 1203 additions and 268 deletions
|
|
@ -1,27 +1,14 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Monitor tree-sitter 0.25 upgrade readiness.
|
||||
"""Monitor tree-sitter 0.25 upgrade readiness — two things Dependabot can't see:
|
||||
|
||||
Tracks two things Dependabot cannot see:
|
||||
1. Peer-dep compatibility: when every grammar's *latest npm release* accepts
|
||||
tree-sitter@0.25.0 (so we can upgrade without --legacy-peer-deps).
|
||||
2. Vendored upstream drift: whether a vendored grammar's upstream parser.c moved.
|
||||
|
||||
1. Peer-dep compatibility. Each tree-sitter-* grammar declares a peer
|
||||
dependency on the tree-sitter runtime. We want to know when every
|
||||
grammar's *latest npm release* satisfies tree-sitter@0.25.0 so we
|
||||
can upgrade without --legacy-peer-deps.
|
||||
|
||||
2. Vendored upstream drift. vendor/tree-sitter-proto/ is a snapshot of
|
||||
coder3101/tree-sitter-proto's parser.c. When upstream moves, we want
|
||||
to know whether we can pick it up.
|
||||
|
||||
Invoked from .github/workflows/tree-sitter-upgrade-readiness.yml daily.
|
||||
Runs locally too:
|
||||
|
||||
python3 .github/scripts/check-tree-sitter-upgrade-readiness.py
|
||||
|
||||
Outputs Markdown to stdout. Exit 0 when every grammar is upgrade-ready
|
||||
and the vendored proto is in sync. Exit 1 when blockers remain (the
|
||||
workflow uses this to open or update a tracking issue).
|
||||
|
||||
No external deps -- stdlib only, so it runs on any vanilla runner.
|
||||
Invoked daily from tree-sitter-upgrade-readiness.yml; runs locally too. Outputs
|
||||
Markdown to stdout; exit 1 when blockers remain (the workflow upserts a tracking
|
||||
issue). stdlib-only — runs on any vanilla runner.
|
||||
python3 .github/scripts/check-tree-sitter-upgrade-readiness.py [--offline | --assert-current]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -38,6 +25,11 @@ import urllib.request
|
|||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
GITNEXUS_DIR = REPO_ROOT / "gitnexus"
|
||||
|
||||
# Offline mode (--offline flag or GITNEXUS_TS_READINESS_OFFLINE=1): skip ALL network
|
||||
# so the script + tests run hermetically. npm columns render "n/a (offline)";
|
||||
# vendored ABIs are still read from the repo. The read-path mirror of --assert-current.
|
||||
OFFLINE = os.environ.get("GITNEXUS_TS_READINESS_OFFLINE", "") not in ("", "0", "false")
|
||||
|
||||
# ── Upgrade target ──────────────────────────────────────────────────────
|
||||
# The runtime version we want to upgrade TO. Update this when the goal
|
||||
# changes (e.g. once 0.25 lands and we target 0.26).
|
||||
|
|
@ -78,15 +70,11 @@ GRAMMARS: dict[str, tuple[str, str, str]] = {
|
|||
"tree-sitter-proto": ("coder3101/tree-sitter-proto", "main", "src/parser.c"),
|
||||
}
|
||||
|
||||
# Grammars deliberately held below npm latest. The readiness report surfaces
|
||||
# these so reviewers can tell intentional pins apart from drift, and so the
|
||||
# context for each pin (which issue motivated it) is visible at a glance.
|
||||
# Add an entry whenever you pin a grammar below npm latest.
|
||||
# npm-installed grammars deliberately held below npm latest (surfaced so reviewers
|
||||
# can tell intentional pins from drift). Add an entry when you pin an npm grammar.
|
||||
# VENDORED grammars carry their hold in .github/vendored-grammars.json instead, so a
|
||||
# vendored grammar's hold lives in one place — tree-sitter-c's is there, not here.
|
||||
INTENTIONAL_PINS: dict[str, str] = {
|
||||
"tree-sitter-c": (
|
||||
"#1242 — last release built against the tree-sitter@0.21 ABI; "
|
||||
"tree-sitter-c@0.23.x prebuilds segfault on Windows under tree-sitter@0.21.1"
|
||||
),
|
||||
"tree-sitter-cpp": (
|
||||
"#1242 — last 0.23.x release before tree-sitter-cpp added a runtime "
|
||||
"dep on the broken-ABI tree-sitter-c@^0.23.1; pinning here removes "
|
||||
|
|
@ -95,6 +83,56 @@ INTENTIONAL_PINS: dict[str, str] = {
|
|||
}
|
||||
|
||||
|
||||
def load_vendored_manifest() -> dict[str, dict]:
|
||||
"""Load the shared vendored-grammar manifest (.github/vendored-grammars.json).
|
||||
|
||||
The single source of truth — shared with update-vendored-grammars.mjs — for
|
||||
which grammars are *vendored* (shipped from gitnexus/vendor/<name>, not npm)
|
||||
and any policy ``hold`` (e.g. tree-sitter-c, #1242/#858). Membership routes a
|
||||
grammar to the vendored branch, which reads its ABI from the repo instead of
|
||||
node_modules (the #858 source of the old bare ``?``). Returns
|
||||
``{ name: {"hold": str | None} }``; upstream-drift coords stay in ``GRAMMARS``.
|
||||
"""
|
||||
manifest_path = REPO_ROOT / ".github" / "vendored-grammars.json"
|
||||
# Fail loud with a pointer, not a bare traceback: this runs at module import,
|
||||
# so a missing/corrupt manifest would otherwise crash both the script and any
|
||||
# test that imports it with an opaque FileNotFoundError/JSONDecodeError.
|
||||
try:
|
||||
data = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
raise SystemExit(
|
||||
f"vendored-grammars manifest not found at {manifest_path}. "
|
||||
f"It is the shared source of truth for vendored grammars "
|
||||
f"(see CONTRIBUTING.md → CI automation contracts)."
|
||||
) from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise SystemExit(
|
||||
f"vendored-grammars manifest at {manifest_path} is not valid JSON: {exc}."
|
||||
) from exc
|
||||
out: dict[str, dict] = {}
|
||||
for key, g in (data.get("grammars") or {}).items():
|
||||
name = g.get("name")
|
||||
if not name:
|
||||
raise SystemExit(
|
||||
f"vendored-grammars manifest entry {key!r} is missing a 'name' field "
|
||||
f"({manifest_path})."
|
||||
)
|
||||
# Defense-in-depth (#2187): `name` is joined into gitnexus/vendor/<name>, so
|
||||
# reject anything not a plain grammar name before it can traverse ("../etc").
|
||||
if not re.fullmatch(r"tree-sitter-[a-z0-9-]+", name):
|
||||
raise SystemExit(
|
||||
f"vendored-grammars manifest entry {key!r} has an invalid grammar "
|
||||
f"name {name!r} (must match tree-sitter-[a-z0-9-]+)."
|
||||
)
|
||||
out[name] = {"hold": g.get("hold")}
|
||||
return out
|
||||
|
||||
|
||||
# Vendored set + holds, keyed by full grammar name (e.g. "tree-sitter-c").
|
||||
VENDORED: dict[str, dict] = load_vendored_manifest()
|
||||
VENDORED_NAMES: frozenset[str] = frozenset(VENDORED)
|
||||
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _load_package_json() -> dict:
|
||||
|
|
@ -134,6 +172,8 @@ def npm_view_json(pkg: str) -> dict | None:
|
|||
being available (it's a batch file on Windows which complicates
|
||||
subprocess calls).
|
||||
"""
|
||||
if OFFLINE:
|
||||
return None
|
||||
url = f"https://registry.npmjs.org/{pkg}/latest"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||
|
|
@ -190,6 +230,8 @@ def fetch_text(url: str, timeout: int = 8) -> str | None:
|
|||
Adds an Authorization header for github.com URLs when GITHUB_TOKEN is
|
||||
set (raises the rate limit from 60 to 5 000 requests/hour).
|
||||
"""
|
||||
if OFFLINE:
|
||||
return None
|
||||
headers: dict[str, str] = {}
|
||||
# Parse the URL and check the hostname rather than substring-matching
|
||||
# on the full URL string (CodeQL py/incomplete-url-substring-sanitization).
|
||||
|
|
@ -231,14 +273,8 @@ def md_h(text: str, level: int = 2) -> str:
|
|||
|
||||
|
||||
def _first_sentence(text: str) -> str:
|
||||
"""Return the leading sentence of a free-form rationale string.
|
||||
|
||||
Vendor package.json `_vendoredBy` fields often look like
|
||||
"<reason>. <install-script breadcrumb>. Do NOT <warning>." — the
|
||||
first sentence is what reviewers actually want to read; the rest is
|
||||
noise in this context. Match a sentence-ending '.' followed by
|
||||
whitespace; fall back to the whole string if nothing matches.
|
||||
"""
|
||||
"""Return the leading sentence of a `_vendoredBy` rationale (the rest tails off
|
||||
into install-script breadcrumbs); fall back to the whole string."""
|
||||
text = text.strip()
|
||||
match = re.search(r"\.\s+[A-Z]", text)
|
||||
return text[: match.start() + 1] if match else text
|
||||
|
|
@ -262,21 +298,26 @@ def range_includes(spec: str | None, version: str) -> bool:
|
|||
return spec.strip() == version.strip()
|
||||
|
||||
|
||||
def is_vendored_pin(spec: str | None) -> bool:
|
||||
return bool(spec) and spec.startswith(("file:", "git", "http"))
|
||||
def vendored_abi_from_repo(name: str, parser_path: str) -> int | None:
|
||||
"""Read a vendored grammar's ABI directly from gitnexus/vendor/<name>.
|
||||
|
||||
Local-only (no network) — the offline half of ``vendored_drift_summary``,
|
||||
factored out so the hermetic ``--assert-current`` gate can introspect vendored
|
||||
ABIs without triggering the upstream-drift fetches it never uses (#858 review).
|
||||
"""
|
||||
vendor_dir = GITNEXUS_DIR / "vendor" / name
|
||||
vendored_parser = vendor_dir / parser_path
|
||||
if not vendored_parser.is_file():
|
||||
vendored_parser = vendor_dir / "src" / "parser.c"
|
||||
return extract_language_version(vendored_parser)
|
||||
|
||||
|
||||
def vendored_drift_summary(
|
||||
name: str, upstream_repo: str, upstream_branch: str, parser_path: str
|
||||
) -> dict:
|
||||
"""Inspect a vendored grammar under gitnexus/vendor/<name>.
|
||||
|
||||
Returns the vendored package.json's ``version`` and ``_vendoredBy``
|
||||
fields (which carry the human rationale for vendoring), the vendored
|
||||
parser's ABI, and a comparison against upstream main. We deliberately
|
||||
rely on ``_vendoredBy`` rather than a parallel registry in this
|
||||
script: the rationale belongs next to the vendored sources, not in
|
||||
a daily-running CI script.
|
||||
"""Inspect a vendored grammar under gitnexus/vendor/<name>: returns its
|
||||
package.json ``version`` + ``_vendoredBy`` (the rationale, kept next to the
|
||||
sources), the vendored ABI, and a comparison against upstream main.
|
||||
"""
|
||||
vendor_dir = GITNEXUS_DIR / "vendor" / name
|
||||
pkg: dict = {}
|
||||
|
|
@ -290,7 +331,7 @@ def vendored_drift_summary(
|
|||
vendored_parser = vendor_dir / parser_path
|
||||
if not vendored_parser.is_file():
|
||||
vendored_parser = vendor_dir / "src" / "parser.c"
|
||||
vendored_abi = extract_language_version(vendored_parser)
|
||||
vendored_abi = vendored_abi_from_repo(name, parser_path)
|
||||
|
||||
upstream_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
|
|
@ -302,10 +343,13 @@ def vendored_drift_summary(
|
|||
sha_text = fetch_text(
|
||||
f"https://api.github.com/repos/{upstream_repo}/commits/{upstream_branch}"
|
||||
)
|
||||
upstream_sha = "?"
|
||||
# Labeled fallback rather than a bare "?": in CI this fetch succeeds, but
|
||||
# offline (or on a transient API miss) the report should say *why* it's
|
||||
# blank instead of leaving a placeholder (#858).
|
||||
upstream_sha = "unknown"
|
||||
if sha_text:
|
||||
try:
|
||||
upstream_sha = json.loads(sha_text).get("sha", "?")[:12]
|
||||
upstream_sha = json.loads(sha_text).get("sha", "unknown")[:12]
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
|
|
@ -321,7 +365,9 @@ def vendored_drift_summary(
|
|||
|
||||
return {
|
||||
"name": name,
|
||||
"vendored_version": pkg.get("version", "?"),
|
||||
# Labeled fallback, never a bare "?": a vendor package.json should always
|
||||
# carry a version, but if one is missing the report says so plainly (#858).
|
||||
"vendored_version": pkg.get("version") or "unknown",
|
||||
"vendored_by": pkg.get("_vendoredBy"),
|
||||
"vendored_abi": vendored_abi,
|
||||
"upstream_repo": upstream_repo,
|
||||
|
|
@ -336,27 +382,14 @@ def vendored_drift_summary(
|
|||
|
||||
|
||||
def assert_current() -> int:
|
||||
"""Assert every grammar's ABI is loadable by the CURRENT runtime.
|
||||
|
||||
Unlike the readiness report (which probes the npm registry + upstream
|
||||
main for the *target* runtime), this mode is hermetic and offline: it
|
||||
reads only what's checked out / installed locally and asserts each
|
||||
grammar's compiled ABI lies within the current runtime's
|
||||
``RUNTIME_ABI_RANGES`` window. It is the static half of the #1922 ABI
|
||||
gate; the runtime load-smoke (`parser-loader-abi.test.ts`) is the
|
||||
dynamic half.
|
||||
|
||||
Coverage, reusing the existing helpers:
|
||||
- npm-installed grammars: ABI from node_modules/<name>/<parser.c>.
|
||||
- vendored grammars (dart/proto/swift): ABI via ``vendored_drift_summary``.
|
||||
- Swift is prebuilt-only (no parser.c) → not introspectable here;
|
||||
treated as "covered by the runtime load-smoke", not asserted.
|
||||
- INTENTIONAL_PINS are honored: a pinned grammar is expected to sit at
|
||||
an ABI the current runtime loads (that's *why* it's pinned), so it is
|
||||
asserted like any other rather than skipped.
|
||||
"""Assert every grammar's compiled ABI loads on the CURRENT runtime.
|
||||
|
||||
The hermetic/offline static half of the #1922 ABI gate (the runtime
|
||||
load-smoke is the dynamic half): reads only local files — npm ABIs from
|
||||
node_modules/<name>, vendored ABIs from gitnexus/vendor/<name> via
|
||||
``vendored_abi_from_repo`` (no network). A prebuilt-only vendor (no
|
||||
parser.c) is skipped; INTENTIONAL_PINS are asserted like any other grammar.
|
||||
Returns 0 when every introspectable grammar is in range, 1 otherwise.
|
||||
Prints a plain-text (non-Markdown) report so CI logs stay readable.
|
||||
"""
|
||||
current_runtime = read_current_runtime()
|
||||
abi_range = RUNTIME_ABI_RANGES.get(current_runtime)
|
||||
|
|
@ -380,13 +413,20 @@ def assert_current() -> int:
|
|||
|
||||
for name, (upstream_repo, upstream_branch, parser_path) in sorted(GRAMMARS.items()):
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
pin_note = f" [intentional pin: {pinned_spec}]" if name in INTENTIONAL_PINS else ""
|
||||
if name in VENDORED_NAMES and VENDORED[name].get("hold"):
|
||||
pin_note = " [vendored, held]"
|
||||
elif name in INTENTIONAL_PINS:
|
||||
pin_note = f" [intentional pin: {pinned_spec}]"
|
||||
else:
|
||||
pin_note = ""
|
||||
|
||||
if is_vendored_pin(pinned_spec):
|
||||
v = vendored_drift_summary(name, upstream_repo, upstream_branch, parser_path)
|
||||
abi = v["vendored_abi"]
|
||||
# Vendored grammars: ABI read locally from the repo via vendored_abi_from_repo
|
||||
# (NOT vendored_drift_summary, which fetches upstream — this gate is hermetic),
|
||||
# so the offline #1922 gate covers them instead of skipping them (#858/#2187).
|
||||
if name in VENDORED_NAMES:
|
||||
abi = vendored_abi_from_repo(name, parser_path)
|
||||
if abi is None:
|
||||
# Prebuilt-only vendor (e.g. tree-sitter-swift): no parser.c to
|
||||
# Prebuilt-only vendor (e.g. a binary-only grammar): no parser.c to
|
||||
# introspect. The runtime load-smoke covers it instead.
|
||||
skipped.append(f"{name} (vendored, prebuilt — covered by load-smoke)")
|
||||
continue
|
||||
|
|
@ -453,32 +493,18 @@ def _classify_grammar(
|
|||
) -> dict:
|
||||
"""Decide a single primary disposition + a separate bump-now hint.
|
||||
|
||||
Buckets are mutually exclusive and ordered by what a reviewer should
|
||||
look at first:
|
||||
- fetch_failed : npm registry fetch failed (treat as blocker, but
|
||||
surface separately so reviewers don't confuse it
|
||||
with an upstream block)
|
||||
- intentional : pinned in INTENTIONAL_PINS — explicit choice
|
||||
- ready : npm-latest peer dep already accepts the target
|
||||
runtime; nothing to do
|
||||
- waiting : main has a fix (ABI 15 or relaxed peer) but no
|
||||
published npm release yet
|
||||
- blocked : peer dep too tight on both npm and main
|
||||
|
||||
Independently of bucket, `bump_now` reports whether reviewers can
|
||||
move the pin forward today without touching the runtime — we only
|
||||
suggest it when npm-latest's peer dep also accepts our *current*
|
||||
runtime, otherwise the bump would break `npm install`.
|
||||
Mutually-exclusive buckets, ordered by reviewer priority: ``fetch_failed``
|
||||
(npm fetch failed — surfaced apart from upstream blocks), ``intentional``
|
||||
(in INTENTIONAL_PINS), ``ready`` (npm-latest peer accepts the target),
|
||||
``waiting`` (a fix on main, unpublished), ``blocked`` (peer too tight on
|
||||
both). ``bump_now`` is independent: True only when npm-latest's peer also
|
||||
accepts our *current* runtime (else the bump would break ``npm install``).
|
||||
"""
|
||||
is_vendored = is_vendored_pin(pinned_spec)
|
||||
behind_latest = (
|
||||
not is_vendored
|
||||
and npm_version != "?"
|
||||
and not range_includes(pinned_spec, npm_version)
|
||||
)
|
||||
# Intentional pins must never appear as actionable bumps — by definition
|
||||
# we're holding them back on purpose. The pin can only be lifted by
|
||||
# editing INTENTIONAL_PINS and package.json together.
|
||||
# Only npm-path grammars reach this function — vendored grammars are routed
|
||||
# to the vendored branch in main() and `continue` before classification.
|
||||
behind_latest = npm_version != "?" and not range_includes(pinned_spec, npm_version)
|
||||
# Intentional pins are never actionable bumps (held on purpose; lifted only by
|
||||
# editing INTENTIONAL_PINS + package.json together).
|
||||
bump_now = behind_latest and current_compat and name not in INTENTIONAL_PINS
|
||||
|
||||
if fetch_failed:
|
||||
|
|
@ -496,6 +522,9 @@ def _classify_grammar(
|
|||
"name": name,
|
||||
"pinned_spec": pinned_spec or "—",
|
||||
"npm_version": npm_version,
|
||||
# Display form for the disposition prose, laundering a "?" (a malformed 200
|
||||
# npm response lacking `version`) so it never shows bare, like the matrix cell.
|
||||
"npm_version_label": "unknown" if npm_version == "?" else npm_version,
|
||||
"peer_range": peer_range,
|
||||
"target_compat": target_compat,
|
||||
"current_compat": current_compat,
|
||||
|
|
@ -503,15 +532,105 @@ def _classify_grammar(
|
|||
"behind_latest": behind_latest,
|
||||
"bump_now": bump_now,
|
||||
"bucket": bucket,
|
||||
"is_vendored": is_vendored,
|
||||
}
|
||||
|
||||
|
||||
def _render_vendored_section(
|
||||
vendored_grammars: list[dict],
|
||||
target_abi_range: tuple[int, int],
|
||||
blockers: dict[str, str],
|
||||
) -> list[str]:
|
||||
"""Render the 'Vendored parsers' prose block. Appends any runtime-side blocker
|
||||
(upstream ABI beyond the target range) to ``blockers`` in place; returns the
|
||||
markdown lines (empty when nothing is vendored). Extracted from main() so that
|
||||
function coordinates named render phases rather than inlining them (#2187)."""
|
||||
if not vendored_grammars:
|
||||
return []
|
||||
# Hoisted out of the list literal below: an implicit string concatenation
|
||||
# inside a list display trips CodeQL py/implicit-string-concatenation-in-list
|
||||
# (it reads as a possibly-missing comma between elements).
|
||||
intro = (
|
||||
"These grammars ship from `gitnexus/vendor/` rather than the npm "
|
||||
"registry. Their compatibility is governed by the **vendored "
|
||||
"ABI** (must lie in the target runtime's range), not by a peer-"
|
||||
"dep negotiation. The rationale for each vendored copy lives in "
|
||||
"its own `package.json` `_vendoredBy` field."
|
||||
)
|
||||
lines = [md_h(f"Vendored parsers ({len(vendored_grammars)})", 2), intro, ""]
|
||||
for v in sorted(vendored_grammars, key=lambda v: v["name"]):
|
||||
sync_label = "in sync with upstream" if v["in_sync"] else "diverged from upstream"
|
||||
if v["abi_state"] == "in_range":
|
||||
abi_label = f"ABI `{v['vendored_abi']}` (in target range)"
|
||||
elif v["abi_state"] == "prebuilt":
|
||||
abi_label = "ABI `prebuilt` (binary-only vendor, source not introspectable)"
|
||||
else:
|
||||
abi_label = (
|
||||
f"ABI `{v['vendored_abi']}` (**outside** target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]})"
|
||||
)
|
||||
# Never a bare "?": when upstream parser.c can't be read (generated at build,
|
||||
# or a transient fetch miss), use the neutral `n/a` token (#858).
|
||||
upstream_abi_str = (
|
||||
f"ABI `{v['upstream_abi']}`" if v["upstream_abi"] is not None else "ABI `n/a`"
|
||||
)
|
||||
lines.append(
|
||||
f"- **`{v['name']}`** `{v['vendored_version']}` — {abi_label}, "
|
||||
f"upstream `{v['upstream_repo']}@{v['upstream_sha']}` "
|
||||
f"{upstream_abi_str} · {sync_label}"
|
||||
)
|
||||
if v.get("hold"):
|
||||
lines.append(f" - **Held:** {v['hold']}")
|
||||
if v["vendored_by"]:
|
||||
# First sentence only — vendor _vendoredBy fields tail off into noise.
|
||||
lines.append(f" - **Why vendored:** {_first_sentence(v['vendored_by'])}")
|
||||
# Action: regen iff upstream ABI exceeds vendored AND stays within target;
|
||||
# beyond target is a runtime-side blocker. Prebuilt-only vendors get a
|
||||
# manual-refresh action driven by the in-sync flag instead.
|
||||
if v["abi_state"] == "prebuilt":
|
||||
if not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** check whether upstream has shipped a new "
|
||||
"prebuilt release; this vendor ships binary-only artefacts."
|
||||
)
|
||||
elif v["upstream_abi"] and v["vendored_abi"] and v["upstream_abi"] > v["vendored_abi"]:
|
||||
if v["upstream_abi"] <= target_abi_range[1]:
|
||||
lines.append(
|
||||
f" - **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, "
|
||||
f"regenerate `parser.c` from upstream `{v['upstream_sha']}`."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f" - **Action:** wait for a runtime supporting ABI "
|
||||
f"{v['upstream_abi']}; current target ({TARGET_RUNTIME}) only "
|
||||
f"goes up to ABI {target_abi_range[1]}."
|
||||
)
|
||||
blockers[f"vendored-{v['name']}-abi"] = (
|
||||
f"vendored {v['name']}: upstream ABI {v['upstream_abi']} outside target range"
|
||||
)
|
||||
elif not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** review upstream changes; vendored copy may "
|
||||
"need a refresh (no ABI bump required)."
|
||||
)
|
||||
lines.append("")
|
||||
return lines
|
||||
|
||||
|
||||
def main() -> int:
|
||||
blockers: dict[str, str] = {}
|
||||
lines: list[str] = []
|
||||
# Label for npm/upstream values we couldn't determine: in --offline mode the
|
||||
# fetch was deliberately skipped (not "failed"), so say so honestly.
|
||||
miss_label = "offline" if OFFLINE else "fetch failed"
|
||||
lines.append(md_h("Tree-sitter 0.25 upgrade readiness", 1))
|
||||
lines.append("")
|
||||
if OFFLINE:
|
||||
lines.append(
|
||||
"> **Offline mode** — npm registry + upstream GitHub checks were skipped. "
|
||||
"npm-installed grammars show as unverified; vendored-grammar ABIs are read "
|
||||
"from `gitnexus/vendor/`."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
current_runtime = read_current_runtime()
|
||||
current_abi_range = RUNTIME_ABI_RANGES.get(current_runtime, (0, 0))
|
||||
|
|
@ -525,10 +644,9 @@ def main() -> int:
|
|||
)
|
||||
lines.append("")
|
||||
|
||||
# First pass: gather raw data + classification per grammar. We render
|
||||
# the human-friendly buckets first, then the raw matrix in a <details>
|
||||
# block at the end. Status text in the matrix is preserved verbatim
|
||||
# so the workflow's row-diff change-detection keeps working.
|
||||
# First pass: gather + classify per grammar. Human buckets render first, then
|
||||
# the raw matrix in a <details> block (Status text preserved verbatim so the
|
||||
# workflow's row-diff change-detection keeps working).
|
||||
grammar_rows: list[dict] = []
|
||||
raw_matrix: list[str] = [
|
||||
"| Grammar | Pinned | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |",
|
||||
|
|
@ -540,17 +658,17 @@ def main() -> int:
|
|||
for name, (upstream_repo, upstream_branch, parser_path) in sorted(GRAMMARS.items()):
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
|
||||
# Vendored grammars don't have an "npm latest" we install from —
|
||||
# we ship our own copy under gitnexus/vendor/<name>. Treat them
|
||||
# as a separate kind of artefact: their readiness for the runtime
|
||||
# upgrade depends on the vendored ABI being in the target range,
|
||||
# not on a peer-dep negotiation.
|
||||
if is_vendored_pin(pinned_spec):
|
||||
# Vendored grammars are classified by manifest membership (NOT a file: pin
|
||||
# heuristic — they aren't in package.json at all, the #858 misrouting bug).
|
||||
# Their readiness is governed by the vendored ABI, read from the repo, not a
|
||||
# peer-dep negotiation. npm-latest columns get sentinels.
|
||||
if name in VENDORED_NAMES:
|
||||
v = vendored_drift_summary(name, upstream_repo, upstream_branch, parser_path)
|
||||
v["pinned_spec"] = pinned_spec
|
||||
# Three-state classification: in-range, out-of-range, or
|
||||
# not-introspectable (e.g. tree-sitter-swift ships only
|
||||
# prebuilt .node binaries, no parser.c — assume compatible).
|
||||
hold = VENDORED[name].get("hold")
|
||||
v["hold"] = hold
|
||||
# Three-state ABI classification: in-range, out-of-range, or
|
||||
# not-introspectable (e.g. a prebuilt-only vendor with no parser.c).
|
||||
if v["vendored_abi"] is None:
|
||||
v["target_compat"] = True
|
||||
v["abi_state"] = "prebuilt"
|
||||
|
|
@ -567,13 +685,37 @@ def main() -> int:
|
|||
f"vendored `{name}`: ABI {v['vendored_abi']} outside target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]}"
|
||||
)
|
||||
# A held vendored grammar (e.g. tree-sitter-c, #1242/#858) is frozen below
|
||||
# a runtime upgrade: in-range ABI or not, keep it a blocker until the hold
|
||||
# (from the manifest) is lifted — same treatment as npm INTENTIONAL_PINS.
|
||||
if hold:
|
||||
v["target_compat"] = False
|
||||
status = "Vendored — held"
|
||||
# Compose with any out-of-range reason rather than overwriting it:
|
||||
# both share the blockers[name] key, and the ABI-out-of-range
|
||||
# detail would otherwise be lost from the blockers summary.
|
||||
hold_reason = f"vendored `{name}` held: {hold}"
|
||||
prior = blockers.get(name)
|
||||
blockers[name] = f"{prior}; {hold_reason}" if prior else hold_reason
|
||||
# Cell sentinels: never emit a bare "?". A vendored grammar's ABI is
|
||||
# the real LANGUAGE_VERSION when introspectable, else a labeled token.
|
||||
vendored_abi_cell = (
|
||||
str(v["vendored_abi"]) if v["vendored_abi"] is not None else "prebuilt"
|
||||
)
|
||||
# A None upstream ABI means the upstream parser.c couldn't be read —
|
||||
# either it is generated at build time (e.g. swift) or the fetch
|
||||
# missed. We can't tell which here, so use a neutral label rather
|
||||
# than asserting "generated at build". Never a bare "?".
|
||||
upstream_abi_cell = (
|
||||
str(v["upstream_abi"]) if v["upstream_abi"] is not None else "n/a"
|
||||
)
|
||||
# Keep vendored grammars in the raw matrix so the workflow's
|
||||
# row-diff change-detection picks up status transitions on
|
||||
# them too. npm-only columns get sentinels.
|
||||
# row-diff change-detection picks up status transitions on them too.
|
||||
# npm-only columns get sentinels.
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | (vendored) | (vendored) | "
|
||||
f"{'Yes' if v['target_compat'] else '**No**'} | "
|
||||
f"{v['vendored_abi'] or '?'} | {v['upstream_abi'] or '?'} | {status} |"
|
||||
f"{vendored_abi_cell} | {upstream_abi_cell} | {status} |"
|
||||
)
|
||||
vendored_grammars.append(v)
|
||||
continue
|
||||
|
|
@ -593,7 +735,7 @@ def main() -> int:
|
|||
peer_optional = ts_meta.get("optional", False) if peer_range else True
|
||||
|
||||
if fetch_failed:
|
||||
peer_display = "? (fetch failed)"
|
||||
peer_display = f"n/a ({miss_label})"
|
||||
target_compat = False
|
||||
current_compat = False
|
||||
else:
|
||||
|
|
@ -609,7 +751,9 @@ def main() -> int:
|
|||
# Fallback to default location.
|
||||
installed_parser = GITNEXUS_DIR / "node_modules" / name / "src" / "parser.c"
|
||||
installed_abi = extract_language_version(installed_parser)
|
||||
abi_display = str(installed_abi) if installed_abi else "?"
|
||||
# Labeled sentinel, never a bare "?": CI's `npm ci` populates node_modules,
|
||||
# but if it's absent say so plainly rather than leaving a placeholder (#858).
|
||||
abi_display = str(installed_abi) if installed_abi else "n/a (not installed)"
|
||||
|
||||
# Check upstream (main/master branch) ABI for unreleased work.
|
||||
upstream_url = (
|
||||
|
|
@ -618,23 +762,19 @@ def main() -> int:
|
|||
)
|
||||
upstream_text = fetch_text(upstream_url)
|
||||
upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None
|
||||
upstream_abi_display = str(upstream_abi) if upstream_abi else "?"
|
||||
upstream_abi_display = str(upstream_abi) if upstream_abi else "n/a"
|
||||
|
||||
# Status text + upstream-progress detection. The Status column
|
||||
# values are preserved as-is to keep the workflow's row-diff
|
||||
# change-detection working on the raw matrix below.
|
||||
upstream_progress: str | None = None
|
||||
if fetch_failed:
|
||||
status = "Unknown (fetch failed)"
|
||||
blockers[name] = f"`{name}`: npm registry fetch failed — could not verify peer dep"
|
||||
status = f"Unknown ({miss_label})"
|
||||
reason = "checks skipped (offline)" if OFFLINE else "npm registry fetch failed"
|
||||
blockers[name] = f"`{name}`: {reason} — could not verify peer dep"
|
||||
elif name in INTENTIONAL_PINS:
|
||||
# An intentional pin is, by definition, a held-back grammar:
|
||||
# whatever npm-latest's peer dep says, our shipped version is
|
||||
# the one whose ABI/peer must accept the target runtime, and
|
||||
# the pin entry exists precisely because it does not. Treat
|
||||
# it as a blocker until the pin is lifted (entry removed from
|
||||
# INTENTIONAL_PINS), at which point this grammar falls back
|
||||
# to standard classification on the next run.
|
||||
# A held-back grammar: treated as a blocker until the pin is lifted
|
||||
# (entry removed from INTENTIONAL_PINS), then reclassified next run.
|
||||
status = "Intentionally pinned"
|
||||
blockers[name] = (
|
||||
f"`{name}` intentionally pinned at `{pinned_spec}` "
|
||||
|
|
@ -675,8 +815,11 @@ def main() -> int:
|
|||
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
compat_icon = "Yes" if target_compat else "**No**"
|
||||
# "?" stays the internal fetch-failed sentinel (compared above); render a
|
||||
# labeled token in the matrix so the report never shows a bare "?" (#858).
|
||||
npm_version_cell = f"n/a ({miss_label})" if npm_version == "?" else npm_version
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | {npm_version} | {peer_display} | "
|
||||
f"| `{name}` | {pinned_spec} | {npm_version_cell} | {peer_display} | "
|
||||
f"{compat_icon} | {abi_display} | {upstream_abi_display} | {status} |"
|
||||
)
|
||||
|
||||
|
|
@ -726,7 +869,8 @@ def main() -> int:
|
|||
lines.append(f"- {len(by_bucket['waiting'])} waiting on an upstream npm release")
|
||||
lines.append(f"- {len(by_bucket['blocked'])} blocked on upstream (no fix even on main)")
|
||||
if by_bucket['fetch_failed']:
|
||||
lines.append(f"- {len(by_bucket['fetch_failed'])} could not be checked (npm registry unreachable)")
|
||||
why = "checks skipped in offline mode" if OFFLINE else "npm registry unreachable"
|
||||
lines.append(f"- {len(by_bucket['fetch_failed'])} could not be checked ({why})")
|
||||
if bump_now:
|
||||
lines.append(
|
||||
f"- **{len(bump_now)} bump candidate(s) you can take TODAY** (npm-latest "
|
||||
|
|
@ -745,7 +889,7 @@ def main() -> int:
|
|||
lines.append("")
|
||||
for r in sorted(bump_now, key=lambda r: r["name"]):
|
||||
lines.append(
|
||||
f"- `{r['name']}`: `{r['pinned_spec']}` → `{r['npm_version']}` "
|
||||
f"- `{r['name']}`: `{r['pinned_spec']}` → `{r['npm_version_label']}` "
|
||||
f"(peer `{r['peer_range'] or 'none'}`)"
|
||||
)
|
||||
lines.append("")
|
||||
|
|
@ -768,7 +912,7 @@ def main() -> int:
|
|||
"These grammars' npm-latest peer dep already accepts the target runtime. No action needed for the upgrade.",
|
||||
by_bucket["ready"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}` — pinned `{r['pinned_spec']}`, npm latest `{r['npm_version']}`"
|
||||
f"- `{r['name']}` — pinned `{r['pinned_spec']}`, npm latest `{r['npm_version_label']}`"
|
||||
+ (" _(also a bump candidate — see above)_" if r["bump_now"] else "")
|
||||
),
|
||||
)
|
||||
|
|
@ -784,7 +928,7 @@ def main() -> int:
|
|||
reason = INTENTIONAL_PINS.get(r["name"], "(no rationale recorded)")
|
||||
lines.append(
|
||||
f"- `{r['name']}` pinned at `{r['pinned_spec']}` "
|
||||
f"(npm latest `{r['npm_version']}`)\n {reason}"
|
||||
f"(npm latest `{r['npm_version_label']}`)\n {reason}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
|
|
@ -794,7 +938,7 @@ def main() -> int:
|
|||
"We can move forward as soon as upstream cuts a release.",
|
||||
by_bucket["waiting"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`. "
|
||||
f"- `{r['name']}@{r['npm_version_label']}` — peer `{r['peer_range'] or 'none'}`. "
|
||||
f"_{r['upstream_progress']}_"
|
||||
),
|
||||
)
|
||||
|
|
@ -804,90 +948,23 @@ def main() -> int:
|
|||
"Peer dep is too tight on both the latest npm release and on upstream main. "
|
||||
"These need an upstream issue/PR before we can proceed.",
|
||||
by_bucket["blocked"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`"
|
||||
+ (" _(vendored)_" if r["is_vendored"] else "")
|
||||
),
|
||||
lambda r: f"- `{r['name']}@{r['npm_version_label']}` — peer `{r['peer_range'] or 'none'}`",
|
||||
)
|
||||
|
||||
_emit_bucket(
|
||||
"Could not check",
|
||||
"npm registry fetch failed for these grammars. Re-run the workflow to retry.",
|
||||
(
|
||||
"Checks were skipped because the report ran in `--offline` mode. "
|
||||
"Re-run online to verify these grammars."
|
||||
if OFFLINE
|
||||
else "npm registry fetch failed for these grammars. Re-run the workflow to retry."
|
||||
),
|
||||
by_bucket["fetch_failed"],
|
||||
lambda r: f"- `{r['name']}` (pinned `{r['pinned_spec']}`)",
|
||||
)
|
||||
|
||||
# ── Vendored parsers ────────────────────────────────────────────
|
||||
if vendored_grammars:
|
||||
lines.append(md_h(f"Vendored parsers ({len(vendored_grammars)})", 2))
|
||||
lines.append(
|
||||
"These grammars ship from `gitnexus/vendor/` rather than the npm "
|
||||
"registry. Their compatibility is governed by the **vendored "
|
||||
"ABI** (must lie in the target runtime's range), not by a peer-"
|
||||
"dep negotiation. The rationale for each vendored copy lives in "
|
||||
"its own `package.json` `_vendoredBy` field."
|
||||
)
|
||||
lines.append("")
|
||||
for v in sorted(vendored_grammars, key=lambda v: v["name"]):
|
||||
sync_label = (
|
||||
"in sync with upstream" if v["in_sync"] else "diverged from upstream"
|
||||
)
|
||||
if v["abi_state"] == "in_range":
|
||||
abi_label = f"ABI `{v['vendored_abi']}` (in target range)"
|
||||
elif v["abi_state"] == "prebuilt":
|
||||
abi_label = "ABI `prebuilt` (binary-only vendor, source not introspectable)"
|
||||
else:
|
||||
abi_label = (
|
||||
f"ABI `{v['vendored_abi']}` (**outside** target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]})"
|
||||
)
|
||||
upstream_abi_str = (
|
||||
f"ABI `{v['upstream_abi']}`" if v["upstream_abi"] else "ABI `?`"
|
||||
)
|
||||
lines.append(
|
||||
f"- **`{v['name']}`** `{v['vendored_version']}` — {abi_label}, "
|
||||
f"upstream `{v['upstream_repo']}@{v['upstream_sha']}` "
|
||||
f"{upstream_abi_str} · {sync_label}"
|
||||
)
|
||||
if v["vendored_by"]:
|
||||
# Show the first sentence — vendor package.json fields tend
|
||||
# to start with the rationale and tail off into install-
|
||||
# script breadcrumbs that aren't useful in this report.
|
||||
rationale = _first_sentence(v["vendored_by"])
|
||||
lines.append(f" - **Why vendored:** {rationale}")
|
||||
# Action computation: needs regen iff upstream ABI exceeds
|
||||
# vendored AND is still within target range. If upstream ABI
|
||||
# exceeds the target, that's a runtime-side blocker. For
|
||||
# prebuilt-only vendors we can't drive this from source ABI;
|
||||
# the action is a manual upstream-binary refresh, surfaced
|
||||
# via the in-sync flag instead.
|
||||
if v["abi_state"] == "prebuilt":
|
||||
if not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** check whether upstream has shipped a new "
|
||||
"prebuilt release; this vendor ships binary-only artefacts."
|
||||
)
|
||||
elif v["upstream_abi"] and v["vendored_abi"] and v["upstream_abi"] > v["vendored_abi"]:
|
||||
if v["upstream_abi"] <= target_abi_range[1]:
|
||||
lines.append(
|
||||
f" - **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, "
|
||||
f"regenerate `parser.c` from upstream `{v['upstream_sha']}`."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f" - **Action:** wait for a runtime supporting ABI "
|
||||
f"{v['upstream_abi']}; current target ({TARGET_RUNTIME}) only "
|
||||
f"goes up to ABI {target_abi_range[1]}."
|
||||
)
|
||||
blockers[f"vendored-{v['name']}-abi"] = (
|
||||
f"vendored {v['name']}: upstream ABI {v['upstream_abi']} outside target range"
|
||||
)
|
||||
elif not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** review upstream changes; vendored copy may "
|
||||
"need a refresh (no ABI bump required)."
|
||||
)
|
||||
lines.append("")
|
||||
lines.extend(_render_vendored_section(vendored_grammars, target_abi_range, blockers))
|
||||
|
||||
# ── Raw matrix (for completeness + workflow row-diff) ────────────
|
||||
lines.append(md_h("Full grammar matrix", 2))
|
||||
|
|
@ -911,6 +988,11 @@ if __name__ == "__main__":
|
|||
sys.stdout.reconfigure(encoding="utf-8") # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
# `--offline` skips all network so the readiness report renders hermetically
|
||||
# (vendored ABIs from the repo; npm columns marked unverified). Useful for
|
||||
# air-gapped runs and deterministic tests.
|
||||
if "--offline" in sys.argv[1:]:
|
||||
OFFLINE = True
|
||||
# `--assert-current` is the offline CI gate (#1922): assert every grammar's
|
||||
# ABI loads on the CURRENT runtime. Bare invocation keeps the original
|
||||
# target-runtime readiness report behaviour.
|
||||
|
|
|
|||
394
.github/scripts/test_check_tree_sitter_upgrade_readiness.py
vendored
Normal file
394
.github/scripts/test_check_tree_sitter_upgrade_readiness.py
vendored
Normal file
|
|
@ -0,0 +1,394 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Tests for check-tree-sitter-upgrade-readiness.py.
|
||||
|
||||
Stdlib-only (``unittest`` + ``unittest.mock``) to match the script under test,
|
||||
which is deliberately dependency-free so it runs on any vanilla runner. Run with:
|
||||
|
||||
python3 -m unittest .github/scripts/test_check_tree_sitter_upgrade_readiness.py
|
||||
|
||||
(pytest also discovers ``unittest.TestCase`` classes, so a future pytest CI job
|
||||
picks these up unchanged.)
|
||||
|
||||
These tests lock in the #858 fix: the 5 vendored grammars
|
||||
(c/swift/kotlin/dart/proto) are classified from the shared manifest
|
||||
(.github/vendored-grammars.json), their ABI is read from gitnexus/vendor/<name>,
|
||||
and the report never renders a bare ``?`` placeholder. All network is mocked.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
from unittest import TestCase, main, mock
|
||||
|
||||
# ── Load the hyphenated script as a module ───────────────────────────────
|
||||
_SCRIPTS_DIR = pathlib.Path(__file__).resolve().parent
|
||||
_SCRIPT = _SCRIPTS_DIR / "check-tree-sitter-upgrade-readiness.py"
|
||||
_REPO_ROOT = _SCRIPTS_DIR.parents[1]
|
||||
_MANIFEST = _REPO_ROOT / ".github" / "vendored-grammars.json"
|
||||
|
||||
_spec = importlib.util.spec_from_file_location("readiness_under_test", _SCRIPT)
|
||||
readiness = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(readiness) # type: ignore[union-attr]
|
||||
|
||||
# The exact row-diff regex the workflow's change-detection bot uses
|
||||
# (.github/workflows/tree-sitter-upgrade-readiness.yml) — byte-identical so a matrix
|
||||
# format change that would silently break change-detection fails here. Group 2 is
|
||||
# ONLY the Status cell ([^|]+? before the final `|$`).
|
||||
_ROW_DIFF_RE = re.compile(r"\| `(tree-sitter-[^`]+)` \|.*\| ([^|]+?) \|$", re.M)
|
||||
|
||||
|
||||
def _physical_vendor_grammars() -> set[str]:
|
||||
vendor = _REPO_ROOT / "gitnexus" / "vendor"
|
||||
return {
|
||||
p.name
|
||||
for p in vendor.iterdir()
|
||||
if p.is_dir() and p.name.startswith("tree-sitter-")
|
||||
}
|
||||
|
||||
|
||||
def _render_report() -> tuple[str, int]:
|
||||
"""Run main() with network mocked to mirror PRODUCTION; return (md, exit_code).
|
||||
|
||||
- npm grammars resolve to a permissive "Ready" peer dep, so the ONLY blocker
|
||||
left is the held vendored tree-sitter-c — letting us assert the hold is
|
||||
load-bearing (exit code stays non-zero because of it).
|
||||
- npm_view_json records its calls so we can prove vendored grammars are never
|
||||
npm-queried.
|
||||
- fetch_text mirrors the real workflow: upstream parser.c resolves to a real
|
||||
ABI (committed upstream), commit endpoints return a sha — EXCEPT swift's
|
||||
upstream, whose parser.c is generated at build time and so is unreachable
|
||||
(None). That single miss exercises the labeled-sentinel path; every other
|
||||
cell must be a real value, never a bare '?'.
|
||||
"""
|
||||
npm_calls: list[str] = []
|
||||
|
||||
def fake_npm_view_json(pkg: str):
|
||||
npm_calls.append(pkg)
|
||||
return {"version": "9.9.9", "peerDependencies": {"tree-sitter": "^0.25.0"}}
|
||||
|
||||
def fake_fetch_text(url: str, timeout: int = 8):
|
||||
if "parser.c" in url:
|
||||
# swift's upstream parser.c is generated at build time → unreachable;
|
||||
# the others ship a committed parser.c.
|
||||
if "alex-pinkus" in url:
|
||||
return None
|
||||
return "#define LANGUAGE_VERSION 14\n#define STATE_COUNT 1\n"
|
||||
if "/commits/" in url:
|
||||
return json.dumps({"sha": "0123456789abcdef"})
|
||||
# package.json (relaxed-peer probe) etc. — not needed for these assertions.
|
||||
return None
|
||||
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(readiness, "npm_view_json", side_effect=fake_npm_view_json), \
|
||||
mock.patch.object(readiness, "fetch_text", side_effect=fake_fetch_text), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
code = readiness.main()
|
||||
report = buf.getvalue()
|
||||
_render_report.last_npm_calls = npm_calls # type: ignore[attr-defined]
|
||||
return report, code
|
||||
|
||||
|
||||
class ManifestClassification(TestCase):
|
||||
def test_manifest_matches_physical_vendor_dirs(self):
|
||||
"""Consistency guard: the manifest set == the gitnexus/vendor/tree-sitter-*
|
||||
dirs. Vendoring a grammar without a manifest entry (or vice-versa) fails —
|
||||
this is what keeps the two tree-sitter workflows aligned (#858)."""
|
||||
manifest_names = {
|
||||
g["name"]
|
||||
for g in json.loads(_MANIFEST.read_text())["grammars"].values()
|
||||
}
|
||||
self.assertEqual(manifest_names, _physical_vendor_grammars())
|
||||
|
||||
def test_vendored_names_loaded_from_manifest(self):
|
||||
self.assertEqual(set(readiness.VENDORED_NAMES), _physical_vendor_grammars())
|
||||
# npm-installed grammars must NOT be classified vendored.
|
||||
self.assertNotIn("tree-sitter-cpp", readiness.VENDORED_NAMES)
|
||||
self.assertNotIn("tree-sitter-go", readiness.VENDORED_NAMES)
|
||||
|
||||
def test_c_carries_a_hold_cpp_does_not(self):
|
||||
self.assertTrue(readiness.VENDORED["tree-sitter-c"]["hold"])
|
||||
self.assertNotIn("tree-sitter-c", readiness.INTENTIONAL_PINS)
|
||||
# cpp stays an npm intentional pin.
|
||||
self.assertIn("tree-sitter-cpp", readiness.INTENTIONAL_PINS)
|
||||
|
||||
def test_vendored_names_are_a_subset_of_GRAMMARS(self):
|
||||
# The report + --assert-current iterate the hardcoded GRAMMARS dict for
|
||||
# upstream-drift coords. A vendored grammar present in the manifest but
|
||||
# missing from GRAMMARS would be silently dropped from both — re-creating
|
||||
# the cross-workflow divergence the manifest exists to kill (#858). Guard it.
|
||||
missing = set(readiness.VENDORED_NAMES) - set(readiness.GRAMMARS)
|
||||
self.assertEqual(missing, set(), f"manifest grammars missing from GRAMMARS: {missing}")
|
||||
|
||||
def test_missing_manifest_raises_a_clear_error(self):
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
with mock.patch.object(readiness, "REPO_ROOT", pathlib.Path(d)):
|
||||
with self.assertRaises(SystemExit) as ctx:
|
||||
readiness.load_vendored_manifest()
|
||||
self.assertIn("vendored-grammars manifest", str(ctx.exception))
|
||||
|
||||
def test_malformed_manifest_raises_a_clear_error(self):
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
gh = pathlib.Path(d) / ".github"
|
||||
gh.mkdir()
|
||||
(gh / "vendored-grammars.json").write_text("{ not valid json", encoding="utf-8")
|
||||
with mock.patch.object(readiness, "REPO_ROOT", pathlib.Path(d)):
|
||||
with self.assertRaises(SystemExit) as ctx:
|
||||
readiness.load_vendored_manifest()
|
||||
self.assertIn("not valid JSON", str(ctx.exception))
|
||||
|
||||
def test_path_traversal_grammar_name_is_rejected(self):
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
bad = '{"grammars": {"evil": {"name": "../etc"}}}'
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
gh = pathlib.Path(d) / ".github"
|
||||
gh.mkdir()
|
||||
(gh / "vendored-grammars.json").write_text(bad, encoding="utf-8")
|
||||
with mock.patch.object(readiness, "REPO_ROOT", pathlib.Path(d)):
|
||||
with self.assertRaises(SystemExit) as ctx:
|
||||
readiness.load_vendored_manifest()
|
||||
self.assertIn("invalid grammar name", str(ctx.exception))
|
||||
|
||||
|
||||
class AssertCurrent(TestCase):
|
||||
"""The offline #1922 ABI gate (--assert-current) must stay hermetic — it reads
|
||||
vendored ABIs from the repo, never the network. (Regression guard: a prior
|
||||
revision routed vendored grammars through vendored_drift_summary, which fetches
|
||||
upstream parser.c + commit sha, silently breaking the 'hermetic and offline'
|
||||
contract — #858 review.)"""
|
||||
|
||||
def _run_assert_current(self):
|
||||
import urllib.request
|
||||
|
||||
def explode(*a, **k):
|
||||
raise AssertionError("--assert-current attempted a network call")
|
||||
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(urllib.request, "urlopen", side_effect=explode), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
code = readiness.assert_current()
|
||||
return buf.getvalue(), code
|
||||
|
||||
def test_assert_current_is_network_free_and_passes(self):
|
||||
report, code = self._run_assert_current() # raises if any urlopen fires
|
||||
self.assertEqual(code, 0)
|
||||
# All 5 vendored grammars are introspected from the repo (ABI 14), not skipped.
|
||||
for name in readiness.VENDORED_NAMES:
|
||||
self.assertIn(f"{name}: vendored ABI", report)
|
||||
|
||||
def test_assert_current_fails_an_out_of_range_vendored_abi(self):
|
||||
# vendored_abi_from_repo is the local-read injection point: force one
|
||||
# grammar out of the current runtime's ABI window and assert the gate trips.
|
||||
real = readiness.vendored_abi_from_repo
|
||||
|
||||
def fake(name, parser_path):
|
||||
return 99 if name == "tree-sitter-dart" else real(name, parser_path)
|
||||
|
||||
import urllib.request
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(readiness, "vendored_abi_from_repo", side_effect=fake), \
|
||||
mock.patch.object(urllib.request, "urlopen", side_effect=AssertionError("network")), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
code = readiness.assert_current()
|
||||
self.assertEqual(code, 1)
|
||||
self.assertIn("tree-sitter-dart", buf.getvalue())
|
||||
self.assertIn("outside current runtime range", buf.getvalue())
|
||||
|
||||
|
||||
class ReportRendering(TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.report, cls.code = _render_report()
|
||||
cls.rows = dict(_ROW_DIFF_RE.findall(cls.report))
|
||||
|
||||
def test_no_bare_question_mark_anywhere(self):
|
||||
# The only legitimate '?' is the "Satisfies 0.25?" column header.
|
||||
sanitized = self.report.replace("Satisfies 0.25?", "Satisfies 0.25")
|
||||
self.assertNotIn("?", sanitized, "report still contains a bare '?' placeholder")
|
||||
|
||||
def test_malformed_npm_version_renders_unknown_in_prose_not_bare_question(self):
|
||||
# A successful (200) npm /latest response that omits `version` leaves
|
||||
# npm_version == "?"; the grammar is still bucketed (fetch did not fail), so
|
||||
# its disposition PROSE line must show the labeled sentinel, never a bare '?'.
|
||||
def fake_npm(pkg: str):
|
||||
if pkg == "tree-sitter-go":
|
||||
return {"peerDependencies": {"tree-sitter": "^0.25.0"}} # no 'version'
|
||||
return {"version": "9.9.9", "peerDependencies": {"tree-sitter": "^0.25.0"}}
|
||||
|
||||
def fake_fetch(url: str, timeout: int = 8):
|
||||
if "parser.c" in url and "alex-pinkus" not in url:
|
||||
return "#define LANGUAGE_VERSION 14\n"
|
||||
if "/commits/" in url:
|
||||
return json.dumps({"sha": "0123456789abcdef"})
|
||||
return None
|
||||
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(readiness, "npm_view_json", side_effect=fake_npm), \
|
||||
mock.patch.object(readiness, "fetch_text", side_effect=fake_fetch), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
readiness.main()
|
||||
report = buf.getvalue()
|
||||
sanitized = report.replace("Satisfies 0.25?", "Satisfies 0.25")
|
||||
self.assertNotIn("?", sanitized)
|
||||
# The Ready bucket prose line for go shows the labeled 'unknown', not '?'.
|
||||
self.assertRegex(report, r"`tree-sitter-go`.*npm latest `unknown`")
|
||||
|
||||
def test_every_vendored_grammar_shows_numeric_abi_not_question_mark(self):
|
||||
for name in readiness.VENDORED_NAMES:
|
||||
row = self._matrix_row(name)
|
||||
cells = [c.strip() for c in row.strip().strip("|").split("|")]
|
||||
abi_cell = cells[5] # Grammar|Pinned|npm|Peer|Satisfies|ABI|UpstreamABI|Status
|
||||
self.assertRegex(
|
||||
abi_cell, r"^\d+$",
|
||||
f"{name} ABI cell is '{abi_cell}', expected a number (read from vendor/)",
|
||||
)
|
||||
|
||||
def test_proto_is_never_npm_queried(self):
|
||||
# github-only vendored grammars must skip the npm peer-dep path entirely,
|
||||
# which is what removes the old "? (fetch failed)" for tree-sitter-proto.
|
||||
self.assertNotIn("tree-sitter-proto", _render_report.last_npm_calls)
|
||||
self.assertNotIn("tree-sitter-dart", _render_report.last_npm_calls)
|
||||
self.assertNotIn("Could not check", self.report)
|
||||
self.assertNotIn("fetch failed", self.report)
|
||||
|
||||
def test_held_c_renders_held_and_keeps_exit_nonzero(self):
|
||||
# Status is the last matrix cell (the row-diff regex captures the whole
|
||||
# tail, not just status, so read the cell directly).
|
||||
cells = [c.strip() for c in self._matrix_row("tree-sitter-c").strip().strip("|").split("|")]
|
||||
self.assertEqual(cells[-1], "Vendored — held")
|
||||
self.assertIn("**Held:**", self.report)
|
||||
# With every npm grammar mocked to "Ready", the ONLY remaining blocker is
|
||||
# the held c — so a non-zero exit proves the hold is treated as a blocker.
|
||||
self.assertEqual(self.code, 1)
|
||||
|
||||
def test_upstream_abi_miss_uses_labeled_sentinel(self):
|
||||
# swift's upstream parser.c is unreachable (mocked None), so its
|
||||
# upstream-ABI cell is the labeled 'n/a' token, never a bare '?'.
|
||||
cells = [c.strip() for c in self._matrix_row("tree-sitter-swift").strip().strip("|").split("|")]
|
||||
self.assertEqual(cells[6], "n/a") # Upstream ABI column
|
||||
|
||||
def test_row_diff_regex_captures_all_fifteen_grammar_statuses(self):
|
||||
# The change-detection bot keys on this regex: group 1 = grammar name,
|
||||
# group 2 = the Status cell ONLY (not the whole tail). It must match every
|
||||
# row after the format change so status transitions keep being detected.
|
||||
self.assertEqual(len(self.rows), 15)
|
||||
for name in readiness.VENDORED_NAMES:
|
||||
self.assertIn(name, self.rows)
|
||||
# group 2 is the Status cell — held c renders exactly "Vendored — held",
|
||||
# and no captured status contains a pipe (proves cell-scoped capture).
|
||||
self.assertEqual(self.rows["tree-sitter-c"], "Vendored — held")
|
||||
for status in self.rows.values():
|
||||
self.assertNotIn("|", status)
|
||||
|
||||
def _matrix_row(self, name: str) -> str:
|
||||
for line in self.report.splitlines():
|
||||
if line.startswith(f"| `{name}` |"):
|
||||
return line
|
||||
# Explicit terminating raise (not self.fail, which CodeQL doesn't model as
|
||||
# NoReturn) so the function has no implicit fall-through return (CodeQL 754).
|
||||
raise AssertionError(f"no matrix row for {name}")
|
||||
|
||||
|
||||
class OfflineMode(TestCase):
|
||||
"""--offline must render the report touching ZERO network — vendored ABIs come
|
||||
from the repo, npm columns are marked unverified. This is what makes the
|
||||
network-dependent report deterministically testable in air-gapped CI."""
|
||||
|
||||
def _render_offline(self):
|
||||
import urllib.request
|
||||
|
||||
def explode(*a, **k):
|
||||
raise AssertionError("network call attempted in --offline mode")
|
||||
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(readiness, "OFFLINE", True), \
|
||||
mock.patch.object(urllib.request, "urlopen", side_effect=explode), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
code = readiness.main()
|
||||
return buf.getvalue(), code
|
||||
|
||||
def test_offline_touches_no_network_and_still_renders(self):
|
||||
report, code = self._render_offline() # raises if any urlopen fires
|
||||
self.assertIn("Offline mode", report)
|
||||
# Vendored grammars are introspected from the repo → real ABI 14, not a miss.
|
||||
for name in readiness.VENDORED_NAMES:
|
||||
row = next(l for l in report.splitlines() if l.startswith(f"| `{name}` |"))
|
||||
cells = [c.strip() for c in row.strip().strip("|").split("|")]
|
||||
self.assertRegex(cells[5], r"^\d+$", f"{name} vendored ABI missing offline")
|
||||
|
||||
def test_offline_marks_npm_grammars_offline_not_fetch_failed(self):
|
||||
report, _ = self._render_offline()
|
||||
self.assertIn("(offline)", report)
|
||||
self.assertNotIn("fetch failed", report) # honest: skipped, not failed
|
||||
|
||||
def test_offline_report_has_no_bare_question_mark(self):
|
||||
report, _ = self._render_offline()
|
||||
sanitized = report.replace("Satisfies 0.25?", "Satisfies 0.25")
|
||||
self.assertNotIn("?", sanitized)
|
||||
|
||||
|
||||
class VendoredAbiBranches(TestCase):
|
||||
"""main()'s vendored-ABI classification reads through vendored_abi_from_repo
|
||||
(the same local-read seam --assert-current uses), so a single patch drives the
|
||||
out-of-range and prebuilt-only branches that no real vendor dir can trigger
|
||||
today (all ship parser.c at ABI 14)."""
|
||||
|
||||
def _render_with_vendored_abi(self, override):
|
||||
"""Render main() with the standard production-faithful network mock plus a
|
||||
vendored_abi_from_repo override (dict: name -> int|None; others read real)."""
|
||||
real = readiness.vendored_abi_from_repo
|
||||
|
||||
def abi_seam(name, parser_path):
|
||||
return override[name] if name in override else real(name, parser_path)
|
||||
|
||||
def fake_npm(pkg):
|
||||
return {"version": "9.9.9", "peerDependencies": {"tree-sitter": "^0.25.0"}}
|
||||
|
||||
def fake_fetch(url, timeout=8):
|
||||
if "parser.c" in url and "alex-pinkus" not in url:
|
||||
return "#define LANGUAGE_VERSION 14\n"
|
||||
if "/commits/" in url:
|
||||
return json.dumps({"sha": "0123456789abcdef"})
|
||||
return None
|
||||
|
||||
buf = io.StringIO()
|
||||
with mock.patch.object(readiness, "vendored_abi_from_repo", side_effect=abi_seam), \
|
||||
mock.patch.object(readiness, "npm_view_json", side_effect=fake_npm), \
|
||||
mock.patch.object(readiness, "fetch_text", side_effect=fake_fetch), \
|
||||
contextlib.redirect_stdout(buf):
|
||||
code = readiness.main()
|
||||
return buf.getvalue(), code
|
||||
|
||||
def _row(self, report, name):
|
||||
line = next(l for l in report.splitlines() if l.startswith(f"| `{name}` |"))
|
||||
return [c.strip() for c in line.strip().strip("|").split("|")]
|
||||
|
||||
def test_out_of_range_vendored_abi_is_a_blocker(self):
|
||||
# Force tree-sitter-dart's vendored ABI outside the target range (13–15).
|
||||
report, code = self._render_with_vendored_abi({"tree-sitter-dart": 99})
|
||||
cells = self._row(report, "tree-sitter-dart")
|
||||
self.assertEqual(cells[-1], "Vendored (ABI out of range)")
|
||||
self.assertEqual(cells[5], "99")
|
||||
self.assertEqual(code, 1) # out-of-range vendored grammar is a blocker
|
||||
|
||||
def test_prebuilt_only_vendored_abi_renders_prebuilt_not_question(self):
|
||||
# vendored_abi None (a future binary-only vendor with no parser.c).
|
||||
report, _ = self._render_with_vendored_abi({"tree-sitter-dart": None})
|
||||
cells = self._row(report, "tree-sitter-dart")
|
||||
self.assertEqual(cells[5], "prebuilt") # labeled, never a bare '?'
|
||||
self.assertEqual(cells[4], "Yes") # prebuilt is assumed target-compatible
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
171
.github/scripts/update-vendored-grammars.mjs
vendored
171
.github/scripts/update-vendored-grammars.mjs
vendored
|
|
@ -42,17 +42,52 @@ const COMPATIBLE_ABI = new Set([13, 14]); // tree-sitter@0.21.1 LANGUAGE_VERSION
|
|||
// github grammars (no usable npm release) track the default branch HEAD. A `hold`
|
||||
// reason makes a grammar report-only: updates are detected + surfaced but never
|
||||
// auto-applied (c is ABI-pinned and must not move without a runtime upgrade).
|
||||
const GRAMMARS = {
|
||||
c: {
|
||||
name: 'tree-sitter-c',
|
||||
npm: 'tree-sitter-c',
|
||||
hold: 'ABI-pinned at 0.21.4 (#1242/#858) — needs a tree-sitter runtime upgrade before bumping',
|
||||
},
|
||||
swift: { name: 'tree-sitter-swift', npm: 'tree-sitter-swift' },
|
||||
kotlin: { name: 'tree-sitter-kotlin', npm: 'tree-sitter-kotlin' },
|
||||
dart: { name: 'tree-sitter-dart', github: 'UserNobody14/tree-sitter-dart' },
|
||||
proto: { name: 'tree-sitter-proto', github: 'coder3101/tree-sitter-proto' },
|
||||
};
|
||||
//
|
||||
// The vendored set lives in .github/vendored-grammars.json — the SHARED source of
|
||||
// truth this monitor and .github/scripts/check-tree-sitter-upgrade-readiness.py both
|
||||
// read, so the two tree-sitter workflows can never disagree about which grammars are
|
||||
// vendored or where their upstream lives. We reshape the manifest's
|
||||
// `{ upstream: { npm | github } }` form into the flat `{ npm? , github? }` shape the
|
||||
// rest of this script consumes. This is a local file read (import-safe, no network).
|
||||
const MANIFEST = path.join(REPO_ROOT, '.github', 'vendored-grammars.json');
|
||||
// `raw` is injectable for testing; production reads the manifest file.
|
||||
function loadManifestGrammars(raw = null) {
|
||||
if (raw === null) {
|
||||
// Fail loud with a pointer, not a bare ENOENT/SyntaxError: this runs at import.
|
||||
try {
|
||||
raw = JSON.parse(fs.readFileSync(MANIFEST, 'utf8'));
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`Could not load the vendored-grammars manifest at ${MANIFEST} ` +
|
||||
`(shared source of truth — see CONTRIBUTING.md → CI automation contracts): ${e.message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return Object.fromEntries(
|
||||
Object.entries(raw.grammars || {}).map(([key, g]) => {
|
||||
if (!g.name)
|
||||
throw new Error(`manifest entry '${key}' is missing a 'name' field (${MANIFEST})`);
|
||||
// Defense-in-depth: `name` is joined into gitnexus/vendor/<name> paths (and
|
||||
// apply() WRITES there), so reject anything that isn't a plain grammar name
|
||||
// before it can traverse the filesystem (#2187).
|
||||
if (!/^tree-sitter-[a-z0-9-]+$/.test(g.name))
|
||||
throw new Error(
|
||||
`manifest entry '${key}' has an invalid grammar name '${g.name}' ` +
|
||||
`(must match tree-sitter-[a-z0-9-]+)`,
|
||||
);
|
||||
return [
|
||||
key,
|
||||
{
|
||||
name: g.name,
|
||||
...(g.upstream?.npm ? { npm: g.upstream.npm } : {}),
|
||||
...(g.upstream?.github ? { github: g.upstream.github } : {}),
|
||||
...(g.hold ? { hold: g.hold } : {}),
|
||||
},
|
||||
];
|
||||
}),
|
||||
);
|
||||
}
|
||||
const GRAMMARS = loadManifestGrammars();
|
||||
|
||||
const sh = (cmd, args, opts = {}) =>
|
||||
execFileSync(cmd, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], ...opts }).trim();
|
||||
|
|
@ -62,6 +97,26 @@ const clean = (v) =>
|
|||
.replace(/^[v^~]/, '')
|
||||
.trim();
|
||||
|
||||
// Shared "is the candidate newer than what we ship?" check, used by BOTH detect()
|
||||
// and apply() so they can never disagree. up.version is the comparable identity for
|
||||
// both kinds: a plain semver for npm, and the `<base>-g<sha7>` provenance string for
|
||||
// github (which apply() also writes to package.json). detect() previously compared
|
||||
// the bare sha7 for github, so after the bot re-vendored a github grammar once it
|
||||
// reported a perpetual false "update available" while apply() saw "already current"
|
||||
// (#2187 review). Comparing up.version on both sides removes that asymmetry.
|
||||
const isNewer = (up, have) => !have || up.version !== have;
|
||||
|
||||
// apply() throws this (instead of calling process.exit) so its error branches are
|
||||
// exercisable in-process by tests; the CLI entrypoint maps `.code` back to the
|
||||
// original exit code, keeping the monitor's subprocess contract identical (#2187).
|
||||
class ApplyExit extends Error {
|
||||
constructor(message, code) {
|
||||
super(message);
|
||||
this.name = 'ApplyExit';
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
function vendoredVersion(g) {
|
||||
const p = path.join(VENDOR, g.name, 'package.json');
|
||||
return clean(JSON.parse(fs.readFileSync(p, 'utf8')).version);
|
||||
|
|
@ -136,22 +191,30 @@ function readAbi(srcRoot) {
|
|||
return null; // unknown (e.g. parser.c only generated at build time)
|
||||
}
|
||||
|
||||
function detect() {
|
||||
// `deps` injects the network/filesystem seams (vendoredVersion / resolveUpstream /
|
||||
// fetchSource / readAbi) so the classification logic — newer-detection, the ABI
|
||||
// gate, and the policy-hold gate — can be unit-tested offline with fixtures, never
|
||||
// touching live npm/GitHub. Production passes nothing and gets the real functions.
|
||||
function detect(deps = {}) {
|
||||
const getVendored = deps.vendoredVersion || vendoredVersion;
|
||||
const resolveUp = deps.resolveUpstream || resolveUpstream;
|
||||
const fetchSrc = deps.fetchSource || fetchSource;
|
||||
const readAbiFn = deps.readAbi || readAbi;
|
||||
const report = [];
|
||||
for (const [key, g] of Object.entries(GRAMMARS)) {
|
||||
const have = vendoredVersion(g);
|
||||
const have = getVendored(g);
|
||||
let up;
|
||||
try {
|
||||
up = resolveUpstream(g);
|
||||
up = resolveUp(g);
|
||||
} catch (err) {
|
||||
report.push({ grammar: key, error: String(err.message || err) });
|
||||
continue;
|
||||
}
|
||||
const newer = up.kind === 'npm' ? up.version !== have : !have || up.ref.slice(0, 7) !== have;
|
||||
const newer = isNewer(up, have);
|
||||
let abi = null;
|
||||
if (newer) {
|
||||
try {
|
||||
abi = readAbi(fetchSource(g, up.ref));
|
||||
abi = readAbiFn(fetchSrc(g, up.ref));
|
||||
} catch {
|
||||
/* fetch/abi best-effort; null = unknown */
|
||||
}
|
||||
|
|
@ -190,34 +253,48 @@ const copyFile = (srcRoot, dest, rel) => {
|
|||
* notice), LICENSE, and prebuilds/ (the build workflow refreshes those). Bumps the
|
||||
* stripped vendor package.json version + provenance — never re-introduces
|
||||
* scripts/dependencies (#836/#1728). Returns the new version.
|
||||
*
|
||||
* opts.dryRun resolves + ABI-validates the candidate but writes NOTHING — it logs
|
||||
* what it would re-vendor and returns the version, so the flow can be rehearsed
|
||||
* (locally or in CI) without mutating gitnexus/vendor/. opts.deps injects the
|
||||
* network/fs seams for offline testing (same shape as detect()).
|
||||
*/
|
||||
function apply(key) {
|
||||
function apply(key, opts = {}) {
|
||||
const dryRun = opts.dryRun || false;
|
||||
const deps = opts.deps || {};
|
||||
const getVendored = deps.vendoredVersion || vendoredVersion;
|
||||
const resolveUp = deps.resolveUpstream || resolveUpstream;
|
||||
const fetchSrc = deps.fetchSource || fetchSource;
|
||||
const readAbiFn = deps.readAbi || readAbi;
|
||||
const g = GRAMMARS[key];
|
||||
if (!g) {
|
||||
console.error(`unknown grammar '${key}'`);
|
||||
process.exit(2);
|
||||
}
|
||||
if (g.hold) {
|
||||
console.error(
|
||||
if (!g) throw new ApplyExit(`unknown grammar '${key}'`, 2);
|
||||
if (g.hold)
|
||||
throw new ApplyExit(
|
||||
`${key}: report-only (${g.hold}); not auto-applied. Re-vendor manually if intended.`,
|
||||
3,
|
||||
);
|
||||
process.exit(3);
|
||||
}
|
||||
const have = vendoredVersion(g);
|
||||
const up = resolveUpstream(g);
|
||||
const newer = up.kind === 'npm' ? up.version !== have : !have || up.version !== have;
|
||||
const have = getVendored(g);
|
||||
const up = resolveUp(g);
|
||||
const newer = isNewer(up, have);
|
||||
if (!newer) {
|
||||
// Already current: nothing to apply. Return (exit 0 via the CLI) — NOT an error.
|
||||
console.error(`${key}: already current (${have}); nothing to apply.`);
|
||||
process.exit(0);
|
||||
return have;
|
||||
}
|
||||
const srcRoot = fetchSource(g, up.ref);
|
||||
const abi = readAbi(srcRoot);
|
||||
if (abi == null || !COMPATIBLE_ABI.has(abi)) {
|
||||
console.error(
|
||||
const srcRoot = fetchSrc(g, up.ref);
|
||||
const abi = readAbiFn(srcRoot);
|
||||
if (abi == null || !COMPATIBLE_ABI.has(abi))
|
||||
throw new ApplyExit(
|
||||
`${key}: candidate ${up.version} is ABI ${abi ?? 'unknown'} — not tree-sitter@0.21.1 ` +
|
||||
`compatible (need 13/14); refusing to re-vendor. Handle manually.`,
|
||||
3,
|
||||
);
|
||||
process.exit(3);
|
||||
|
||||
if (dryRun) {
|
||||
console.log(
|
||||
`${key}: [dry-run] would re-vendor ${g.name} → ${up.version} (ABI ${abi}); no files written.`,
|
||||
);
|
||||
return up.version;
|
||||
}
|
||||
|
||||
const dest = path.join(VENDOR, g.name);
|
||||
|
|
@ -256,11 +333,31 @@ function apply(key) {
|
|||
// makes live network calls, so importing must be side-effect-free.
|
||||
const isMain = process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href;
|
||||
if (isMain) {
|
||||
if (process.argv[2] === '--apply') {
|
||||
apply(process.argv[3]);
|
||||
const args = process.argv.slice(2);
|
||||
const dryRun = args.includes('--dry-run');
|
||||
if (args[0] === '--apply') {
|
||||
// `--apply <grammar> [--dry-run]` — --dry-run previews without writing.
|
||||
// Map apply()'s thrown ApplyExit back to the original exit codes (0/2/3) so
|
||||
// the monitor workflow's subprocess (which only distinguishes zero vs non-zero)
|
||||
// sees identical behavior.
|
||||
try {
|
||||
apply(args[1], { dryRun });
|
||||
} catch (e) {
|
||||
console.error(e.message);
|
||||
process.exit(e instanceof ApplyExit ? e.code : 1);
|
||||
}
|
||||
} else {
|
||||
process.stdout.write(JSON.stringify(detect(), null, 2) + '\n');
|
||||
}
|
||||
}
|
||||
|
||||
export { detect, apply, resolveUpstream, readAbi, vendoredVersion, GRAMMARS, COMPATIBLE_ABI };
|
||||
export {
|
||||
detect,
|
||||
apply,
|
||||
resolveUpstream,
|
||||
readAbi,
|
||||
vendoredVersion,
|
||||
loadManifestGrammars,
|
||||
GRAMMARS,
|
||||
COMPATIBLE_ABI,
|
||||
};
|
||||
|
|
|
|||
26
.github/vendored-grammars.json
vendored
Normal file
26
.github/vendored-grammars.json
vendored
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"_comment": "Single source of truth for the VENDORED SET + policy holds, read by BOTH .github/scripts/update-vendored-grammars.mjs (weekly auto-PR bot) and .github/scripts/check-tree-sitter-upgrade-readiness.py (daily readiness report -> issue #858). The monitor also resolves each grammar's upstream from the `upstream` field here; the readiness report reads vendored ABIs from gitnexus/vendor/<name>/src/parser.c and keeps its own upstream-drift coords. A consistency-guard test asserts this set equals the gitnexus/vendor/tree-sitter-* directories. See CONTRIBUTING.md.",
|
||||
"grammars": {
|
||||
"c": {
|
||||
"name": "tree-sitter-c",
|
||||
"upstream": { "npm": "tree-sitter-c" },
|
||||
"hold": "ABI-pinned at 0.21.4 (#1242/#858) — needs a tree-sitter runtime upgrade before bumping"
|
||||
},
|
||||
"swift": {
|
||||
"name": "tree-sitter-swift",
|
||||
"upstream": { "npm": "tree-sitter-swift" }
|
||||
},
|
||||
"kotlin": {
|
||||
"name": "tree-sitter-kotlin",
|
||||
"upstream": { "npm": "tree-sitter-kotlin" }
|
||||
},
|
||||
"dart": {
|
||||
"name": "tree-sitter-dart",
|
||||
"upstream": { "github": "UserNobody14/tree-sitter-dart" }
|
||||
},
|
||||
"proto": {
|
||||
"name": "tree-sitter-proto",
|
||||
"upstream": { "github": "coder3101/tree-sitter-proto" }
|
||||
}
|
||||
}
|
||||
}
|
||||
7
.github/workflows/grammar-update-monitor.yml
vendored
7
.github/workflows/grammar-update-monitor.yml
vendored
|
|
@ -14,6 +14,13 @@ name: Vendored grammar update monitor
|
|||
# never auto-bumped — a maintainer re-vendors it deliberately after a runtime
|
||||
# upgrade.
|
||||
#
|
||||
# The vendored set + per-grammar upstream coords + the tree-sitter-c hold live in
|
||||
# .github/vendored-grammars.json — the SHARED source of truth this monitor and
|
||||
# tree-sitter-upgrade-readiness.yml both read, so the two workflows can never
|
||||
# disagree about which grammars are vendored (#858). This monitor additionally
|
||||
# resolves each grammar's upstream from it; the readiness report reads vendored
|
||||
# ABIs from gitnexus/vendor/ and keeps its own upstream-drift coords.
|
||||
#
|
||||
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
|
||||
|
||||
on:
|
||||
|
|
|
|||
|
|
@ -1,12 +1,21 @@
|
|||
name: Tree-sitter Upgrade Readiness
|
||||
|
||||
# Monitors readiness for upgrading tree-sitter to 0.25.x. Tracks:
|
||||
# 1. Peer-dep compatibility — can each grammar install cleanly with
|
||||
# tree-sitter@0.25.0 without --legacy-peer-deps?
|
||||
# 2. Vendored proto drift — has coder3101/tree-sitter-proto moved
|
||||
# ahead of our vendored snapshot?
|
||||
# 1. Peer-dep compatibility — can each NPM-installed grammar install cleanly
|
||||
# with tree-sitter@0.25.0 without --legacy-peer-deps?
|
||||
# 2. Vendored grammars — each grammar in .github/vendored-grammars.json
|
||||
# (c/swift/kotlin/dart/proto) is classified by its vendored ABI, read
|
||||
# straight from gitnexus/vendor/<name>/src/parser.c (NOT node_modules,
|
||||
# which is never populated for vendored grammars — that mismatch is why
|
||||
# the report used to render bare "?" placeholders, #858).
|
||||
# See .github/scripts/check-tree-sitter-upgrade-readiness.py for the logic.
|
||||
#
|
||||
# .github/vendored-grammars.json is the SHARED source of truth for the vendored
|
||||
# SET + policy holds: this readiness report and grammar-update-monitor.yml both
|
||||
# read it, so the two workflows can never disagree about which grammars are
|
||||
# vendored. (The monitor also resolves upstreams from it; this report keeps its
|
||||
# own upstream-drift coords and reads vendored ABIs from gitnexus/vendor/.)
|
||||
#
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
|
||||
on:
|
||||
|
|
@ -18,6 +27,8 @@ on:
|
|||
pull_request:
|
||||
paths:
|
||||
- '.github/scripts/check-tree-sitter-upgrade-readiness.py'
|
||||
- '.github/scripts/test_check_tree_sitter_upgrade_readiness.py'
|
||||
- '.github/vendored-grammars.json'
|
||||
- '.github/workflows/tree-sitter-upgrade-readiness.yml'
|
||||
|
||||
concurrency:
|
||||
|
|
@ -28,14 +39,18 @@ permissions:
|
|||
contents: read
|
||||
|
||||
jobs:
|
||||
readiness:
|
||||
report:
|
||||
name: Check upgrade readiness
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
# Least privilege: rendering the report needs no write. The issue mutation
|
||||
# lives in the schedule-only `upsert-issue` job below, so PR runs (incl. forks)
|
||||
# never receive `issues: write` (#2187 review).
|
||||
permissions:
|
||||
contents: read
|
||||
# Needed to open/update the tracking issue on scheduled runs.
|
||||
issues: write
|
||||
outputs:
|
||||
report: ${{ steps.readiness.outputs.report }}
|
||||
exit_code: ${{ steps.readiness.outputs.exit_code }}
|
||||
steps:
|
||||
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
|
||||
|
||||
|
|
@ -43,6 +58,17 @@ jobs:
|
|||
with:
|
||||
build: 'false'
|
||||
|
||||
# Guard the readiness script's logic (vendored classification, no bare "?",
|
||||
# the manifest⇄vendor-dir consistency guard). Stdlib-only, so no extra deps;
|
||||
# node_modules is populated by setup-gitnexus above, which the npm-path ABI
|
||||
# reads need. Runs only on validation events (PR / manual), not the daily
|
||||
# scheduled report.
|
||||
- name: Run readiness script unit tests
|
||||
if: github.event_name != 'schedule'
|
||||
shell: bash
|
||||
working-directory: .github/scripts
|
||||
run: python3 -m unittest test_check_tree_sitter_upgrade_readiness -v
|
||||
|
||||
- name: Run upgrade readiness check
|
||||
id: readiness
|
||||
shell: bash
|
||||
|
|
@ -54,10 +80,15 @@ jobs:
|
|||
code=$?
|
||||
set -e
|
||||
echo "exit_code=$code" >> "$GITHUB_OUTPUT"
|
||||
# Unguessable per-run heredoc delimiter: the report includes the manifest's
|
||||
# `hold` field, which a fork PR can edit — a fixed delimiter (e.g. DRIFT_EOF)
|
||||
# in a hold value could close the heredoc early and inject $GITHUB_OUTPUT keys.
|
||||
# A random hex delimiter the report cannot contain neutralizes that.
|
||||
DELIM="DRIFT_EOF_$(openssl rand -hex 16)"
|
||||
{
|
||||
echo 'report<<DRIFT_EOF'
|
||||
echo "report<<${DELIM}"
|
||||
cat drift-report.md
|
||||
echo 'DRIFT_EOF'
|
||||
echo "${DELIM}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo "=== Report ==="
|
||||
cat drift-report.md
|
||||
|
|
@ -69,13 +100,22 @@ jobs:
|
|||
run: |
|
||||
echo "::warning::Tree-sitter 0.25 upgrade has blockers. See job output for the full readiness report."
|
||||
|
||||
- name: Upsert tracking issue on scheduled runs
|
||||
if: >
|
||||
github.event_name == 'schedule' &&
|
||||
steps.readiness.outputs.exit_code != '0'
|
||||
# Issue mutation is isolated here so `issues: write` is only ever granted on the
|
||||
# scheduled run (never on PRs). Consumes the report + exit_code via job outputs.
|
||||
upsert-issue:
|
||||
name: Upsert tracking issue
|
||||
needs: report
|
||||
if: github.event_name == 'schedule'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
issues: write
|
||||
steps:
|
||||
- name: Upsert tracking issue on blockers
|
||||
if: needs.report.outputs.exit_code != '0'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
REPORT: ${{ steps.readiness.outputs.report }}
|
||||
REPORT: ${{ needs.report.outputs.report }}
|
||||
with:
|
||||
script: |
|
||||
const title = 'Tree-sitter 0.25 upgrade readiness';
|
||||
|
|
@ -105,7 +145,11 @@ jobs:
|
|||
// | `tree-sitter-foo` | ... | Blocking |
|
||||
const parseRows = (md) => {
|
||||
const map = {};
|
||||
for (const m of md.matchAll(/\| `(tree-sitter-[^`]+)` \|.*?\| (\S+(?:\s\S+)*?) \|$/gm)) {
|
||||
// Group 2 captures ONLY the Status cell ([^|]+? before the final
|
||||
// `|$`), so change-detection fires on status transitions, not on
|
||||
// unrelated cell drift (e.g. an upstream-ABI bump). Mirror this in
|
||||
// _ROW_DIFF_RE in test_check_tree_sitter_upgrade_readiness.py.
|
||||
for (const m of md.matchAll(/\| `(tree-sitter-[^`]+)` \|.*\| ([^|]+?) \|$/gm)) {
|
||||
map[m[1]] = m[2].trim();
|
||||
}
|
||||
return map;
|
||||
|
|
@ -152,10 +196,8 @@ jobs:
|
|||
core.info(`Opened issue #${created.number}`);
|
||||
}
|
||||
|
||||
- name: Close tracking issue on clean scheduled runs
|
||||
if: >
|
||||
github.event_name == 'schedule' &&
|
||||
steps.readiness.outputs.exit_code == '0'
|
||||
- name: Close tracking issue on clean runs
|
||||
if: needs.report.outputs.exit_code == '0'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
|
|
|
|||
|
|
@ -144,6 +144,15 @@ Re-invoking `/autofix` after a successful apply is a safe no-op — the workflow
|
|||
|
||||
**Sensitive paths.** The apply workflow refuses any patch that touches `.github/` (workflow files, CODEOWNERS, dependabot config). A malicious PR could ship a custom prettier or ESLint config that reformats workflow YAML; if accepted, those edits would be pushed under `contents: write` without human review. Apply formatter changes to files under `.github/` manually in a normal commit so they get the same review every other workflow change gets.
|
||||
|
||||
### Vendored tree-sitter grammars
|
||||
|
||||
`.github/vendored-grammars.json` is the **single source of truth** for the vendored tree-sitter grammar **set** and each grammar's policy `hold` (the ones shipped from `gitnexus/vendor/<name>` rather than installed from npm). It lists each grammar's name, upstream coords (`npm` or `github`), and any `hold`. The monitor resolves upstreams from it; the readiness report keeps its own upstream-drift coords and reads vendored ABIs from `gitnexus/vendor/`. Two workflows read it:
|
||||
|
||||
- `grammar-update-monitor.yml` (`.github/scripts/update-vendored-grammars.mjs`) — weekly; opens auto-PRs re-vendoring ABI-compatible upstream updates.
|
||||
- `tree-sitter-upgrade-readiness.yml` (`.github/scripts/check-tree-sitter-upgrade-readiness.py`) — daily; renders the tree-sitter-0.25 readiness report (issue #858), reading each vendored grammar's ABI from `gitnexus/vendor/<name>/src/parser.c`.
|
||||
|
||||
Sharing the manifest keeps the two aligned: a consistency-guard test asserts the manifest set equals the `gitnexus/vendor/tree-sitter-*` directories. **When you vendor a new grammar (or remove one), update `.github/vendored-grammars.json` in the same change** — otherwise that guard fails CI and the readiness report regresses to `?` placeholders.
|
||||
|
||||
## AI-assisted contributions
|
||||
|
||||
If you use coding agents, follow project context files (e.g. `AGENTS.md`, `CLAUDE.md`) and avoid drive-by refactors unrelated to the issue. Prefer incremental, test-backed changes.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readFileSync, readdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
|
|
@ -20,10 +20,21 @@ const MOD = pathToFileURL(
|
|||
),
|
||||
).href;
|
||||
|
||||
type Grammar = { name: string; npm?: string; github?: string; hold?: string };
|
||||
type Upstream = { version: string; ref: string; kind: 'npm' | 'github' };
|
||||
type DetectDeps = {
|
||||
vendoredVersion?: (g: Grammar) => string;
|
||||
resolveUpstream?: (g: Grammar) => Upstream;
|
||||
fetchSource?: (g: Grammar, ref: string) => string;
|
||||
readAbi?: (root: string) => number | null;
|
||||
};
|
||||
let mod: {
|
||||
readAbi: (root: string) => number | null;
|
||||
COMPATIBLE_ABI: Set<number>;
|
||||
GRAMMARS: Record<string, { name: string; npm?: string; github?: string; hold?: string }>;
|
||||
GRAMMARS: Record<string, Grammar>;
|
||||
detect: (deps?: DetectDeps) => Array<Record<string, unknown>>;
|
||||
apply: (key: string, opts?: { dryRun?: boolean; deps?: DetectDeps }) => string;
|
||||
loadManifestGrammars: (raw?: unknown) => Record<string, Grammar>;
|
||||
};
|
||||
let tmp: string;
|
||||
|
||||
|
|
@ -76,3 +87,270 @@ describe('GRAMMARS registry', () => {
|
|||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('shared vendored-grammars manifest', () => {
|
||||
// The vendored set is sourced from .github/vendored-grammars.json — the single
|
||||
// source of truth shared with check-tree-sitter-upgrade-readiness.py. This guards
|
||||
// against the loader silently skewing from the manifest file (#858 alignment).
|
||||
const manifestPath = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
'../../../.github/vendored-grammars.json',
|
||||
);
|
||||
const manifest: {
|
||||
grammars: Record<
|
||||
string,
|
||||
{ name: string; upstream: { npm?: string; github?: string }; hold?: string }
|
||||
>;
|
||||
} = JSON.parse(readFileSync(manifestPath, 'utf8'));
|
||||
|
||||
it('reshapes every manifest entry into the GRAMMARS shape, losing no information', () => {
|
||||
expect(Object.keys(mod.GRAMMARS).sort()).toEqual(Object.keys(manifest.grammars).sort());
|
||||
for (const [key, g] of Object.entries(manifest.grammars)) {
|
||||
const entry = mod.GRAMMARS[key];
|
||||
expect(entry.name).toBe(g.name);
|
||||
expect(entry.hold).toBe(g.hold);
|
||||
// Assert the absent upstream field is explicitly undefined, not just
|
||||
// matching the manifest's absent property (avoids an undefined===undefined
|
||||
// pass that would miss the loader mis-mapping a github coord into `npm`).
|
||||
if (g.upstream.npm) {
|
||||
expect(entry.npm).toBe(g.upstream.npm);
|
||||
expect(entry.github).toBeUndefined();
|
||||
} else {
|
||||
expect(entry.github).toBe(g.upstream.github);
|
||||
expect(entry.npm).toBeUndefined();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('each grammar has exactly one upstream source (npm xor github)', () => {
|
||||
for (const g of Object.values(mod.GRAMMARS)) {
|
||||
expect(Boolean(g.npm) !== Boolean(g.github)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('the manifest grammar set equals the physical vendor/tree-sitter-* dirs (#858)', () => {
|
||||
// Monitor-side mirror of the Python consistency guard. The monitor is the side
|
||||
// that WRITES files from manifest `name`, so vendoring a grammar (or removing
|
||||
// one) without updating the manifest must fail CI here too.
|
||||
const vendorDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../vendor');
|
||||
const physical = readdirSync(vendorDir, { withFileTypes: true })
|
||||
.filter((d) => d.isDirectory() && d.name.startsWith('tree-sitter-'))
|
||||
.map((d) => d.name)
|
||||
.sort();
|
||||
const manifestNames = Object.values(manifest.grammars)
|
||||
.map((g) => g.name)
|
||||
.sort();
|
||||
expect(manifestNames).toEqual(physical);
|
||||
});
|
||||
|
||||
it('rejects a path-traversal grammar name at load (defense-in-depth)', () => {
|
||||
// `name` is joined into vendor/<name> paths and apply() writes there.
|
||||
expect(() => mod.loadManifestGrammars({ grammars: { evil: { name: '../etc' } } })).toThrow(
|
||||
/invalid grammar name/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// Narrowing accessor: throws a clear error instead of a non-null assertion (`!`),
|
||||
// which @typescript-eslint/no-non-null-assertion forbids.
|
||||
function must<T>(value: T | undefined, message: string): T {
|
||||
if (value === undefined) throw new Error(message);
|
||||
return value;
|
||||
}
|
||||
|
||||
describe('detect() classification (offline, injected deps)', () => {
|
||||
// Drive the real detect() loop with faked network/fs seams so the load-bearing
|
||||
// gates — newer-detection, the ABI gate, and the policy hold — are exercised
|
||||
// deterministically without touching live npm/GitHub.
|
||||
const baseResolveUpstream = (g: Grammar): Upstream =>
|
||||
g.npm
|
||||
? { version: '9.9.9', ref: '9.9.9', kind: 'npm' }
|
||||
: { version: '1.0.0-gabc1234', ref: 'abc1234def0', kind: 'github' };
|
||||
const deps: DetectDeps = {
|
||||
vendoredVersion: (g) => (g.name === 'tree-sitter-kotlin' ? '9.9.9' : '0.0.0'),
|
||||
resolveUpstream: baseResolveUpstream,
|
||||
fetchSource: (g) => g.name, // pass the name through to the fake readAbi
|
||||
readAbi: (name) => (name === 'tree-sitter-swift' ? 15 : 14),
|
||||
};
|
||||
let report: Array<Record<string, unknown>>;
|
||||
const byKey = (k: string) =>
|
||||
must(
|
||||
report.find((r) => r.grammar === k),
|
||||
`no detect row for ${k}`,
|
||||
);
|
||||
beforeAll(() => {
|
||||
report = mod.detect(deps);
|
||||
});
|
||||
|
||||
it('flags newer npm + github grammars as updates', () => {
|
||||
expect(byKey('swift').update).toBe(true); // npm 9.9.9 != vendored 0.0.0
|
||||
expect(byKey('dart').update).toBe(true); // github sha differs from vendored
|
||||
});
|
||||
|
||||
it('does not flag a same-version grammar, and skips its ABI fetch', () => {
|
||||
expect(byKey('kotlin').update).toBe(false); // vendored == upstream 9.9.9
|
||||
expect(byKey('kotlin').abi).toBeNull();
|
||||
expect(byKey('kotlin').applicable).toBe(false);
|
||||
});
|
||||
|
||||
it('holds tree-sitter-c: update detected, ABI-compatible, but never applicable', () => {
|
||||
const c = byKey('c');
|
||||
expect(c.update).toBe(true);
|
||||
expect(c.abi).toBe(14);
|
||||
expect(c.abiCompatible).toBe(true);
|
||||
expect(c.hold).toBeTruthy();
|
||||
expect(c.applicable).toBe(false); // policy-hold gate
|
||||
});
|
||||
|
||||
it('refuses an ABI-incompatible candidate (15) — not applicable', () => {
|
||||
const s = byKey('swift');
|
||||
expect(s.abi).toBe(15);
|
||||
expect(s.abiCompatible).toBe(false);
|
||||
expect(s.applicable).toBe(false); // ABI gate
|
||||
});
|
||||
|
||||
it('marks a newer, un-held, ABI-14 grammar applicable', () => {
|
||||
const d = byKey('dart');
|
||||
expect(d.abi).toBe(14);
|
||||
expect(d.applicable).toBe(true);
|
||||
});
|
||||
|
||||
it('records a per-grammar error entry when resolveUpstream throws, without skipping siblings', () => {
|
||||
const report2 = mod.detect({
|
||||
...deps,
|
||||
resolveUpstream: (g) => {
|
||||
if (g.name === 'tree-sitter-dart') throw new Error('gh api 503');
|
||||
return baseResolveUpstream(g);
|
||||
},
|
||||
});
|
||||
const dart = must(
|
||||
report2.find((r) => r.grammar === 'dart'),
|
||||
'no detect row for dart',
|
||||
);
|
||||
expect(dart.error).toContain('gh api 503');
|
||||
expect(dart.update).toBeUndefined(); // error entry, not a classification
|
||||
// The throw on one grammar must not drop the rest.
|
||||
const swift = must(
|
||||
report2.find((r) => r.grammar === 'swift'),
|
||||
'no detect row for swift',
|
||||
);
|
||||
expect(swift.update).toBe(true);
|
||||
expect(report2).toHaveLength(Object.keys(mod.GRAMMARS).length);
|
||||
});
|
||||
});
|
||||
|
||||
describe('detect()/apply() agree on "newer" for github grammars', () => {
|
||||
// github grammars carry up.version = `<base>-g<sha7>` (the provenance string apply()
|
||||
// writes). detect() must compare the same up.version (not the bare sha7) so it stops
|
||||
// reporting a false "update" once the bot has re-vendored once (#2187 review).
|
||||
const PROV = '1.0.0-gabc1234';
|
||||
// deps for dart (github); other grammars get a harmless npm-shaped upstream so the
|
||||
// detect() loop completes — we only inspect dart.
|
||||
const dartDeps = (vendored: string): DetectDeps => ({
|
||||
vendoredVersion: (g) => (g.name === 'tree-sitter-dart' ? vendored : '0.0.0'),
|
||||
resolveUpstream: (g) =>
|
||||
g.name === 'tree-sitter-dart'
|
||||
? { version: PROV, ref: 'abc1234def0', kind: 'github' }
|
||||
: { version: '9.9.9', ref: '9.9.9', kind: 'npm' },
|
||||
fetchSource: (g) => g.name,
|
||||
readAbi: () => 14,
|
||||
});
|
||||
const dartRow = (vendored: string) =>
|
||||
must(
|
||||
mod.detect(dartDeps(vendored)).find((r) => r.grammar === 'dart'),
|
||||
'no detect row for dart',
|
||||
);
|
||||
|
||||
it('equal provenance → update:false (the asymmetry that is fixed)', () => {
|
||||
expect(dartRow(PROV).update).toBe(false);
|
||||
});
|
||||
|
||||
it('first-vendoring (plain version vs provenance) → update:true (not suppressed)', () => {
|
||||
// vendored is the plain pre-bot version; up.version is `<base>-g<sha7>` → still newer.
|
||||
expect(dartRow('1.0.0').update).toBe(true);
|
||||
});
|
||||
|
||||
it('upstream sha advanced → update:true', () => {
|
||||
expect(dartRow('1.0.0-g0000000').update).toBe(true);
|
||||
});
|
||||
// The detect⇄apply agreement on the equal-provenance (already-current) case is
|
||||
// asserted in U12's apply() tests — apply()'s not-newer path currently calls
|
||||
// process.exit(0), which can't be exercised in-process until U12 makes it return.
|
||||
});
|
||||
|
||||
describe('apply(--dry-run): resolves + validates but writes nothing', () => {
|
||||
it('returns the candidate version without mutating the vendored package.json', () => {
|
||||
const pkgPath = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
'../../vendor/tree-sitter-dart/package.json',
|
||||
);
|
||||
const before = readFileSync(pkgPath, 'utf8');
|
||||
const version = mod.apply('dart', {
|
||||
dryRun: true,
|
||||
deps: {
|
||||
vendoredVersion: () => '0.0.0',
|
||||
resolveUpstream: () => ({ version: '9.9.9', ref: '9.9.9abc', kind: 'github' }),
|
||||
fetchSource: () => 'unused',
|
||||
readAbi: () => 14,
|
||||
},
|
||||
});
|
||||
expect(version).toBe('9.9.9');
|
||||
expect(readFileSync(pkgPath, 'utf8')).toBe(before); // untouched
|
||||
});
|
||||
});
|
||||
|
||||
describe('apply() error branches throw ApplyExit (CLI maps to exit codes)', () => {
|
||||
const dartPkg = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
'../../vendor/tree-sitter-dart/package.json',
|
||||
);
|
||||
// catch + return the thrown error's exit code (apply() throws instead of calling
|
||||
// process.exit, so the error branches are exercisable in-process).
|
||||
const codeOf = (fn: () => unknown): number => {
|
||||
try {
|
||||
fn();
|
||||
} catch (e) {
|
||||
return (e as { code?: number }).code ?? -1;
|
||||
}
|
||||
throw new Error('expected apply() to throw');
|
||||
};
|
||||
|
||||
it('unknown grammar key → exit code 2', () => {
|
||||
expect(codeOf(() => mod.apply('nope', { deps: {} }))).toBe(2);
|
||||
});
|
||||
|
||||
it('held grammar (c) → exit code 3 (short-circuits before the newer check)', () => {
|
||||
expect(codeOf(() => mod.apply('c', { deps: {} }))).toBe(3);
|
||||
});
|
||||
|
||||
it('ABI-incompatible candidate (15) → exit code 3', () => {
|
||||
const code = codeOf(() =>
|
||||
mod.apply('dart', {
|
||||
deps: {
|
||||
vendoredVersion: () => '0.0.0', // newer than upstream → reaches the ABI gate
|
||||
resolveUpstream: () => ({ version: '9.9.9', ref: '9.9.9abc', kind: 'github' }),
|
||||
fetchSource: () => 'unused',
|
||||
readAbi: () => 15,
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(code).toBe(3);
|
||||
});
|
||||
|
||||
it('not-newer (already current) → returns the current version, no throw, no write', () => {
|
||||
const before = readFileSync(dartPkg, 'utf8');
|
||||
const deps = {
|
||||
vendoredVersion: () => '9.9.9-gabc1234',
|
||||
resolveUpstream: () => ({
|
||||
version: '9.9.9-gabc1234',
|
||||
ref: 'abc1234',
|
||||
kind: 'github' as const,
|
||||
}),
|
||||
fetchSource: () => 'unused',
|
||||
readAbi: () => 14,
|
||||
};
|
||||
// No dryRun: the not-newer path returns `have` before any fetch/copy.
|
||||
expect(mod.apply('dart', { deps })).toBe('9.9.9-gabc1234');
|
||||
expect(readFileSync(dartPkg, 'utf8')).toBe(before); // untouched
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue