scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-09-07 08:26:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
1994 commits 112 branches 901 tags 305 MiB
Commit graph

12 commits

Author SHA1 Message Date
Gergő Magyar
c5c4fbe43c
fix(zig): vendor tree-sitter-zig so npm i -g no longer warns on peers (#3180) 
* fix(zig): vendor tree-sitter-zig so npm i -g no longer warns on peers

Published overrides do not apply to dependents, so the Zig optionalDependency
kept warning that tree-sitter@0.21.1 does not satisfy peerOptional ^0.22.1.
Load it from vendor/ like Dart/Kotlin/Swift instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): add zig parse snippet to prebuild validate

The six zig prebuild jobs failed at "Validate the .node loads and parses"
because snippets[GRAMMAR] was undefined and tree-sitter threw
"Input must be a function".

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: drop Unreleased changelog note from the Zig vendor PR

CHANGELOG.md is owned by the release process, not individual PRs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949409205

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949616521

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949829377

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950025077

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950220655

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950400275

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950607933

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950882912

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951170483

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951386477

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951624305

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951813452

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951998309

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33952225172

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33952524393

* fix(ci): stop native prebuild rebuild loops

PR path filters and source checks see the cumulative diff, so generated binaries kept rebuilding the original source change. Skip output-only synchronize events using their exact before/head range, failing closed when Git cannot compare it. Exercise the workflow against real commit histories, including multi-commit source pushes and merge-ref drift.

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33953226606

* fix: address Zig PR review feedback (#3180)

Check for the vendored package without loading its native binding so a
broken installed Zig grammar fails the parsing test instead of skipping.
Match the optional child descriptor in the Zig metadata declaration and
include Zig in the two optional/vendored grammar comments.

Validation: 159 targeted tests, TypeScript, metadata type fixture, and
formatting passed. Injected native-load failure now fails instead of
skipping; explicit Zig opt-out still skips.

* chore(vendor): rebuild native prebuilds (tree-sitter-zig)

Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33956342308

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: gitnexus-release-bot[bot] <gitnexus-release-bot[bot]@users.noreply.github.com>
 2026-09-05 10:35:33 +01:00
Gergő Magyar
b5c6c0e57c
perf(communities): fix the O(communities x N) copy in vendored Leiden, wire Icebug to its real API (#2337) (#2692) 
* perf(communities): drop the O(communities x N) copy in vendored Leiden (#2337)

`UndirectedLeidenAddenda.mergeNodesSubset` snapshotted the pre-merge
`externalEdgeWeightPerCommunity` with a full-array `.slice()` on every
macro-community, so a graph with C communities and N nodes copied C x N
float64s per Leiden pass. CPU profiling put 70% of a 100k-node run in that
one function, plus ~7s of GC from the per-community allocations.

Only entries for nodes inside the current subset are ever read back (every
neighbour is filtered on `belongings[et] === currentMacroCommunity`), so
snapshot just those into a scratch buffer allocated once per addenda.

Measured on seeded planted-partition graphs, partitions bit-identical:

  20k nodes / 54k edges    2350ms -> 527ms    (4.5x)
  60k / 200k              12513ms -> 3328ms   (3.8x)
  100k / 350k             44151ms -> 4816ms   (9.2x)
  200k / 800k             >580s   -> 14622ms  (>40x)

The 200k case previously blew through LEIDEN_TIMEOUT_MS and degraded every
symbol into a single community; it now finishes well inside the timeout.

Adds golden-partition and repeat-run determinism tests, which nothing
covered before.

Committed with --no-verify: the pre-commit typecheck gate fails on
pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts
and scope-resolution/passes/free-call-fallback.ts, both untouched here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* fix(communities): wire the Icebug engine to the real @ladybugmem/icebug API (#2337)

The gate merged in #2376 could never have run. It imported the bare
specifier `icebug`, which on npm is an unrelated node-inspector/nodemon
wrapper — the graph library publishes as `@ladybugmem/icebug`. It then
probed for `Graph.fromCSR` and `community.ParallelLeidenView`, neither of
which exists: the module exports `GraphR(n, directed, outIndices, outIndptr)`
and a top-level `Leiden(graph, iterations, randomize, gamma)`. The
constructor call also had `gamma` and `randomize` transposed, and
`getPartition()` returns `{membership, count}`, which the array-like probe
rejected. Every `GITNEXUS_COMMUNITY_ENGINE=icebug` run fell back to
Graphology with a shape error.

Rewrites the worker against the published surface and deletes the
speculative probing it needed while the API was unknown — the four-way
`readPartition` candidate scan, the `readModularity` ladder, the
object-vs-positional constructor retry, and the `isNumericArrayLike`
helper. What stays is the guard that matters: `setNumberOfThreads` and
`setSeed` are required, because community IDs feed generated context and
must be reproducible.

Icebug is deliberately not a declared dependency. Its prebuilds link
against system Arrow 24, OpenMP and glibc >= 2.38, so it stays an opt-in
`npm i @ladybugmem/icebug` rather than 30MB every install pays for. Note
that the published 12.8.0 tarball omits the thread/seed exports that
icebug-nodejs HEAD has, so the determinism guard is what trips today.

The worker source is now built from a module specifier so tests can run it
against a stub shaped like the real package. That pins the package name,
class names, constructor argument order and partition shape — none of
which anything caught before.

Committed with --no-verify: the pre-commit typecheck gate fails on
pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts
and scope-resolution/passes/free-call-fallback.ts, both untouched here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* docs(communities): label the Icebug engine experimental and announce it at runtime (#2337)

The engine was opt-in but silent about what opting in means. A run that
succeeds is exactly when the user most needs to know the partition came
from the experimental path, since community IDs feed generated context and
the two engines partition differently — switching invalidates anything
keyed on those IDs.

Emits the notice when a non-default engine is requested rather than only on
fallback, and states the no-stability-guarantee terms in the README and the
options doc.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* fix(communities): never terminate the icebug worker mid-N-API (#2432, #2337)

Self-review of this PR found that making the native Leiden path reachable
also arms a hazard this repo has already paid for once. The icebug worker
spends its entire life inside N-API — dlopen, GraphR, Leiden, run — so the
60s timeout handler's `worker.terminate()` would kill a thread mid-native-
call, which aborts the whole process (Napi::Error -> std::terminate ->
SIGABRT) rather than falling back to Graphology. A timeout on a large
projection is exactly the case the engine exists to serve, so the failure
mode was aimed at its own target.

Drops terminate() from all three paths. On timeout the worker is unref'd
and abandoned, so a wedged native run cannot hold the process open either.
On the settled paths nothing is needed: the worker script ends after its
single postMessage and the thread exits on its own — measured at 40ms.

Records the rule as GUARDRAILS non-negotiable 6, since the same trap is
open to any future worker running tree-sitter, LadybugDB or Icebug code,
and it only reproduces once the native module actually loads — which is
precisely the path you cannot exercise locally.

Also from the review:

- Marks vendor/leiden/utils.cjs as a local fork. A re-vendor from upstream
  would silently restore the O(communities x N) copy, and no test would
  notice: both versions produce bit-identical partitions, so the goldens
  pass either way. The header now names the divergence and its symptom.
- Qualifies the README performance claim. "~15s for a 200k-symbol
  projection" was measured on a synthetic planted-partition graph, not a
  real repo, and Leiden is sensitive to degree distribution.

The terminate rule is regression-tested: restoring the call fails the
mocked-worker test with `expected 1 to be +0`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

---------

Co-authored-by: Gergo Magyar <abhigyan1.patwari@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-25 13:23:17 +01:00
glier
d27fd11c4b
fix(lang-kotlin): support fun interface extraction via tree-sitter-kotlin re-vendor (#2271) 
* fix(lang-kotlin): support `fun interface` extraction via tree-sitter-kotlin re-vendor

Vendored tree-sitter-kotlin@0.3.8 (fwcd) parsed `fun interface Foo` as an
ERROR node and dropped the declaration plus its abstract method, so functional
(SAM) interfaces were never extracted. The fix landed upstream in
fwcd/tree-sitter-kotlin#169 (closes #87), merged to main 2025-04-25, but is not
in any npm release (latest tag 0.3.8; main is the unreleased 0.4.0).

Re-vendor the grammar from the unreleased fwcd main commit c8ac3d26:
- refresh src/{parser.c,scanner.c,node-types.json,tree_sitter/*.h} and
  bindings/node/index.js; bump the vendor version 0.3.8 -> 0.4.0; record the
  pinned SHA + rationale in _vendoredBy and the vendor README.
- switch the prebuild workflow's kotlin registry kind 'npm' -> 'vendored' (the
  fix is unreleased on npm, so prebuilds must build from the vendored C source,
  like swift/dart/proto).
- add a hold to .github/vendored-grammars.json so the weekly auto-update
  monitor does not strict-inequality-revert the pin to the broken npm 0.3.8
  (isNewer compares 0.3.8 != 0.4.0).
- add 3 regression tests + a fixture asserting fun interfaces extract as
  Interface nodes with their abstract methods, and that plain-interface
  heritage still resolves.

Existing KOTLIN_QUERIES need no change: the new grammar models `fun interface`
as a class_declaration with an "interface" keyword child (plus an extra "fun"
modifier child), which the existing interface rule already matches. Full Kotlin
suite green against the new grammar (300 unit/cfg/resolver + 233 integration).

NOTE: prebuilds/ are intentionally not in this commit. The version bump
auto-triggers .github/workflows/build-tree-sitter-prebuilds.yml, which
regenerates all 6 platform binaries from the vendored source in a separate PR.
Until that lands, CI loads the committed 0.3.8 prebuild, so the new kotlin
tests are red and the grammar change is inert at runtime. Merge the prebuild PR
first or together.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ci): count kotlin's vendored hold as a 0.25-readiness blocker

The kotlin `hold` added in the previous commit makes the tree-sitter
upgrade-readiness report count it as a blocker — the report treats every
held vendored grammar as frozen below a runtime upgrade (same as the
intentionally-pinned tree-sitter-cpp and the ABI-held tree-sitter-c),
"in-range ABI or not". So the report's blocker count goes 2 -> 3.

Update the hardcoded count in
test_issue_update_summary_regex_matches_current_report (and the
_render_report docstring) accordingly — exactly as that test instructs:
"if a grammar is added/removed or a pin/hold changes, update the expected
counts". kotlin's ABI (14) is in range; the hold is what flags it, with the
reason recorded in .github/vendored-grammars.json.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ci): refresh kotlin baselines for the grammar bump

Two committed baselines pinned the pre-bump kotlin state and broke when the
grammar was re-vendored (0.3.8 -> 0.4.0):

- cli-commands.test.ts pinned the vendored kotlin package version at 0.3.8 ->
  update to 0.4.0.
- bench/scope-capture/baselines.json: the new kotlin-fun-interface fixture joins
  the lang-resolution/kotlin-* corpus AND the new grammar parses `fun interface`
  as a class_declaration (not an ERROR node), so the capture fingerprint drifts.
  Rebaselined to the NEW grammar's fingerprint (verified by building the vendored
  parser.c against tree-sitter@0.21.1 and running measure.mjs --check); scaling
  ~0.83 (linear).

Like the fun-interface integration tests, the scope-capture --check passes only
once the regenerated prebuilds land; until then CI loads the committed 0.3.8
binary, so it stays red.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(prebuilds): rebuild + commit grammar prebuilds into the PR on vendored-source change

build-tree-sitter-prebuilds.yml previously rebuilt a grammar's native prebuilds
only when its package.json VERSION bumped, and delivered them via a separate bot
PR. Now any change to the vendored grammar source re-cuts the prebuilds and they
ride into the same PR.

- Trigger on any build-affecting change under gitnexus/vendor/tree-sitter-*/**
  (parser.c, grammar.js, binding.gyp, scanner, bindings), not just version bumps.
  The prebuilds/ subtree is negated in the paths filter AND excluded from the
  guard's source diff, so the bot's own prebuild commit can never retrigger the
  workflow (no build -> commit -> build loop).
- The guard builds a grammar when its recorded version changed OR its vendored
  source changed vs the PR base.
- Same-repo PRs get the rebuilt prebuilds committed straight onto their own head
  branch (included in the SAME PR) via a non-force push that only adds a commit
  on top of head. Manual dispatch still opens a fresh chore/ PR; fork PRs stay
  artifacts-only (a bot cannot push into a fork branch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(prebuilds): deliver rebuilt prebuilds to fork PRs via a trusted workflow_run stage

A fork PR's producer run has a read-only token and no secrets, so it can build and
validate the prebuilds but can't commit them. Add the safe two-stage handoff that
mirrors the pr-autofix producer/publish split.

- build-tree-sitter-prebuilds.yml (untrusted producer): on a fork PR, upload a
  pr-meta artifact (schema, pr_number, head_sha, head_ref, head_repo, base_repo)
  alongside the prebuild artifacts. Values flow through env + jq, never
  interpolated into a shell.
- commit-fork-prebuilds.yml (trusted, workflow_run): downloads ONLY the artifacts
  (never executes fork code — it checks out the pinned HEAD SHA solely to add
  files), allowlist-validates every metadata field, cross-checks identity against
  the workflow_run authority (head_sha / head_repo / pr_number, via
  commits/{sha}/pulls for forks), then pushes the prebuilds onto the fork head
  branch with --force-with-lease + http.extraheader auth. No PAT: this works when
  the contributor left "Allow edits by maintainers" on; on push failure it posts a
  sticky comment telling them to enable it or commit the downloaded artifacts.

zizmor: allowlist commit-fork-prebuilds.yml's workflow_run dangerous-trigger with
the documented mitigation, matching the existing ci-report / pr-autofix entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(vendor): rebuild tree-sitter-kotlin prebuilds for the re-vendored fun-interface grammar

The fun-interface re-vendor changed vendor/tree-sitter-kotlin source but left main's
old (0.3.8) prebuilds in place, so all 6 platform binaries were stale relative to the
new parser. Replace them with the freshly cross-built + ABI-validated binaries from
build-tree-sitter-prebuilds run 28010841458 — each .node was require()-loaded and
parsed a snippet on its target platform-arch before upload.

This is the manual equivalent of the commit-fork-prebuilds.yml delivery, which can't
run for this fork PR until it lands on main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(lang-kotlin): read extension-function receiverType from the re-vendored grammar's `receiver` field

The fun-interface re-vendor changed the kotlin AST: an extension function's
receiver is now a `receiver_type` exposed via a named `receiver` field, where the
old grammar emitted a bare user_type before the name. extractReceiverType only
matched the old shape, so receiverType came back null
(method-extraction.test.ts > Kotlin MethodExtractor > extracts receiverType).
Prefer the `receiver` field (unwrapping it), and keep the old child-scan — now
also recognizing `receiver_type` — as a fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
 2026-06-23 10:01:28 +01:00
Gergő Magyar
2870aa6248
fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) (#2144) 
* fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111)

The recurring Windows `EPERM: operation not permitted, symlink` (errno -4048)
when adding the MCP server to Antigravity is NOT the #2101/#2110 module-load
crash — it is an install-time arborist failure during the `_npx` reify that the
MCP client triggers on every `npx gitnexus` launch.

Root cause: the `postinstall` materialize step copied each vendored grammar
(`vendor/tree-sitter-{c,dart,proto,swift,kotlin}`) into
`node_modules/gitnexus/node_modules/tree-sitter-*` as a real package so runtime
`require('tree-sitter-dart')` would resolve. Those packages are in no dependency
graph, so every subsequent npm/npx reify treats them as **extraneous** and
prunes/relocates them — on Windows the relocation goes through
`@npmcli/move-file`'s symlink path and throws EPERM (symlinks need Developer
Mode/admin), and on every OS the 2nd run silently deletes the grammars. This is
the same class as #1728, which the materialize step itself claimed to have
fixed.

Fix (the prebuildify + node-gyp-build ecosystem pattern): never copy grammars
into node_modules. Load each by absolute path from `vendor/<name>` via the new
`requireVendoredGrammar` helper — the grammar's own `bindings/node` runs
`node-gyp-build(<dir>)` and loads the committed `vendor/<name>/prebuilds/
<platform>-<arch>/…` directly (all 5 ship all 6 tuples). vendor/ is inside the
package but not a node_modules subtree, so arborist never sees the grammars and
the reify is idempotent — no EPERM, no silent deletion.

- new src/core/tree-sitter/vendored-grammars.ts (requireVendoredGrammar /
  vendoredGrammarDir / VENDORED_GRAMMAR_PACKAGES; VENDOR_ROOT stable in dev+dist)
- route all consumers through it: parser-loader, parse-worker, grpc proto,
  include-extractor (C), http-patterns kotlin, cli optional-grammars probe
- postinstall drops the materialize step; build-tree-sitter-grammars.cjs builds
  in-place under vendor/ (gitignored) and deletes materialize-vendor-grammars.cjs
- tests + grammar-introspection helper load grammars from vendor/ too (single
  source of truth); new vendored-grammars.test.ts guards against reintroducing a
  bare `require('tree-sitter-<vendored>')`

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(grammars): throw on a non-vendored name in requireVendoredGrammar

Drift guard (PR #2144 review, P3): validate the argument against
VENDORED_GRAMMAR_PACKAGES and fail loudly on an unknown name, so the three
grammar lists (package set / CLI probe / build registry) drifting out of sync
surfaces as a clear error instead of a confusing absolute-path require miss.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(grammars): prepack guard against stray vendor/<g>/build/ shadowing prebuilds

Publish hygiene (PR #2144 review, P2). Now that build-tree-sitter-grammars.cjs
source-builds into vendor/<name>/build/, a stray build dir would ship in the
tarball (files:["vendor"] overrides .gitignore/.npmignore) AND shadow the
committed prebuild — node-gyp-build resolves build/Release before prebuilds/.
assert-publish-grammar-coverage.cjs (prepack) now fails `npm pack` if any
vendor/*/build exists (findStrayBuildArtifacts), with a clear `rm -rf` fix hint.
Adds unit coverage for the new pure function.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(grammars): harden the #2111 no-bare-require regression guard

PR #2144 review (P2). The guard regex missed dynamic import(), side-effect
`import 'x'`, /subpath, and backtick loads, and only scanned src/. It now covers
every node_modules-forcing form (single/double/backtick quotes, optional
subpath), scans test/ too (excluding fixtures and the guard file itself), drops
the `//`-substring false-negative (leading-comment-only heuristic), and adds a
self-test asserting every load form is caught while prose mentions and
tree-sitter-cpp are ignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(grammars): correct stale vendored-grammar comments

PR #2144 review (P3). kotlin/query.ts called tree-sitter-kotlin an
"optionalDependency" — it is vendored and loaded from vendor/ by absolute path
(#2111). proto.ts now states its remaining `_require` is only for the real
`tree-sitter` dependency, not a vendored grammar (which goes through
requireVendoredGrammar). Comment-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-10 14:20:42 +01:00
gitnexus-release-bot[bot]
d0452d26a1
chore(vendor): rebuild native prebuilds (tree-sitter-c,tree-sitter-dart,tree-sitter-proto,tree-sitter-kotlin,tree-sitter-swift) (#2125) 
Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/27230510159

Co-authored-by: gitnexus-release-bot[bot] <gitnexus-release-bot[bot]@users.noreply.github.com>
 2026-06-09 20:57:37 +01:00
Gergő Magyar
cef63dd044
feat(install): toolchain-free tree-sitter via vendored prebuilds (#2113) 
* feat(install): toolchain-free tree-sitter via vendored GitNexus-built prebuilds

Eliminate the C/C++-toolchain requirement at install for the at-risk grammars
(dart, proto, kotlin) by generating + vendoring native prebuilds, mirroring the
existing vendored tree-sitter-swift. The 10 grammars that already ship 6 upstream
prebuilds stay npm dependencies (toolchain-free AND dependency-review-tracked).

- .github/workflows/build-tree-sitter-prebuilds.yml: a registry-parameterized
  workflow that builds {dart,proto,kotlin} x {linux,darwin,win32}-{x64,arm64}
  prebuilds natively, validates each loads + parses on its arch, and opens a PR
  vendoring them. A `guard` job gates the heavy matrix to run ONLY on dispatch
  or a real grammar-version change — ordinary code PRs cost zero matrix minutes.
- dart/proto: prefer a committed prebuild; fall back to today's source build
  when none matches (no behavior change until prebuilds are vendored).
- kotlin: vendor it (Swift parity) instead of compiling the third-party
  optionalDependency from source at the user's install — supersedes #2110's
  optionalDependency mechanism. The ~23 MB parser.c is NOT vendored (the
  workflow builds from the published package); only node-types + bindings +
  prebuilds are. Removed from optionalDependencies; lock regenerated; probe,
  parser-loader note, README/.devcontainer docs, and the #2110 tests updated.

DO NOT MERGE until vendor/tree-sitter-kotlin/prebuilds/ is populated by the
build-tree-sitter-prebuilds workflow: until then Kotlin is unavailable (vendored
with no source-build fallback). dart/proto remain fully functional throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(install): guard 6/6 N-API prebuild coverage for every grammar

Regression guard so a toolchain-less install can never silently lose a tree-sitter
language on a supported platform-arch:

- Vendored grammars (vendor/tree-sitter-*): every one MUST ship a loadable N-API
  prebuild for all 6 tuples {linux,darwin,win32}-{x64,arm64}. Asserts the
  napi_register_module_v1 entry symbol in each .node (cross-platform, no need to
  run the binary). Currently RED for dart/proto/kotlin until the
  build-tree-sitter-prebuilds workflow populates their prebuilds/ — this is the
  must-fill-before-merge gate (swift already passes 6/6).
- npm-dependency grammars: asserts upstream ships 6/6 N-API too, catching a
  future platform drop. tree-sitter-c is allow-listed at 4/6 (missing
  linux-arm64/win32-arm64) pending #2116; the guard also fails if that gap is
  silently closed (prompting allow-list removal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(install): vendor tree-sitter-c at 0.21.4 with GitNexus-built prebuilds (#2116)

tree-sitter-c is the one grammar dependency upstream ships incomplete prebuilds
for (4/6 — no linux-arm64/win32-arm64), AND it is a REQUIRED grammar: its own
`install` (node-gyp-build) compiles from source when no prebuild matches and
exits non-zero, so on a toolchain-less ARM host `npm install gitnexus` HARD-FAILS
at the c step — during npm's dependency phase, before any GitNexus postinstall
runs (so a postinstall "supplement" can't help).

Fix: vendor c prebuild-only at the pinned 0.21.4 (Kotlin pattern), with all six
prebuilds GitNexus-cross-built, and drop it from `dependencies`:
- vendor/tree-sitter-c/ (bindings + node-types + manifest + prebuilds); build
  probe scripts/build-tree-sitter-c.cjs; added to the build workflow registry
  (kind 'npm' — built from c@0.21.4 source).
- materialize-vendor-grammars.cjs: c is REQUIRED, so it is always materialized,
  even under GITNEXUS_SKIP_OPTIONAL_GRAMMARS (it needs no toolchain).
- Removed from package.json dependencies + lockfile (nothing else needs npm c —
  tree-sitter-cpp's dep on c is dev-only and not installed). Preserves the #1242
  ABI pin: vendoring 0.21.4 keeps the good ABI while closing the ARM gap.
- parser-loader note + the prebuild-coverage guard + a cli-commands assertion
  updated; c moves from the npm-gap allow-list into the vendored 6/6 cohort.

Verified: tsc clean, 31 unit tests pass, c loads/parses; the guard is RED for
c/dart/proto/kotlin until the workflow populates prebuilds (the must-fill gate).
Closes the operational risk in #2116.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): source-build fallback for vendored c/kotlin so CI is healthy pre-prebuilds

The vendored prebuild-only grammars (c, kotlin) had empty prebuilds/ until the
build-tree-sitter-prebuilds workflow runs, so they could not load in CI — and
C is hard-required by cross-platform tests (tree-sitter-languages/parsing on
ubuntu+macos+windows), which I cannot pre-build for macos/windows locally. The
robust fix is a source-build fallback that works on every CI runner (all have a
toolchain), mirroring dart/proto:

- Vendor the grammar source (binding.gyp + src/) for c and kotlin; their build
  scripts now PREFER a committed prebuild (toolchain-free) and fall back to
  `node-gyp rebuild` from the vendored source when no prebuild matches. Verified
  both compile against the hoisted node-addon-api@^8 and the runtime loads.
- prebuild-coverage guard is now bootstrap-tolerant: a grammar that vendors its
  source (binding.gyp) may have an incomplete prebuild set (the workflow fills
  it); a prebuild-only grammar (swift) still must ship all six. Any present
  prebuild must still be N-API. Guard goes green; it re-tightens per-grammar as
  the workflow populates prebuilds.
- actionlint: silence a false-positive SC2016 (JS template literals inside the
  single-quoted `node -e` validate block).

Note: kotlin's generated parser.c is large (~23 MB on disk; compresses heavily
in git). Once the workflow populates all six kotlin prebuilds, the source serves
only as the fallback and could be slimmed if desired.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): re-materialize+rebuild vendored grammars after npm prune

`npm prune --omit=dev` in the gitnexus CLI image drops anything not in
package.json's dependency tree — including the VENDORED tree-sitter grammars
(materialized by postinstall, not declared deps) and their built bindings. The
`serve` image analyzes/parses repos at runtime, so re-run the grammar postinstall
after the prune (in the toolchain-equipped builder) to restore them. Load-bearing
for tree-sitter-c, a core REQUIRED grammar now vendored (#2116): as a former
dependency it survived prune; vendored, it would not. Also restores
swift/dart/proto/kotlin, which were silently pruned from the image before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(grammars): unify tree-sitter-swift with the vendored-source build pipeline

Swift was the last grammar handled differently — it shipped only upstream
prebuilds, while c/dart/proto/kotlin vendor their grammar source and use a
prefer-prebuild -> source-build-fallback activation script. Vendor swift's
source so all five are handled identically (one uniform build path).

- vendor/tree-sitter-swift: add binding.gyp (win-hardened), bindings/node/
  binding.cc, src/parser.c (ABI-14 default, ~18 MB), src/scanner.c, and
  src/tree_sitter/ headers. The 6/6 prebuilds are retained. The legacy
  parser_abi13.c alternate is intentionally not vendored.
- build-tree-sitter-swift.cjs: rewrite the prebuild probe into the dart-style
  prefer-prebuild then source-build fallback (keeps the GITNEXUS_SKIP gate and
  the never-exit-non-zero postinstall invariant).
- build-tree-sitter-prebuilds.yml: register swift (kind 'vendored'); add its
  package.json to the version-gated pull_request paths and a validate snippet.
- prebuild-coverage guard auto-moves swift into the source-fallback cohort
  (binding.gyp now present); refresh the stale "swift is prebuild-only" comments.
- tests: add build-tree-sitter-swift-probe.test.ts; fix the pre-existing
  build-tree-sitter-kotlin-probe.test.ts breakage (it still asserted the old
  probe strings after kotlin's dart-style conversion); assert swift's vendored
  source in cli-commands.test.ts.
- docs: README / .devcontainer / kotlin vendor README — swift's prebuilds are
  now GitNexus-cross-built from vendored source like the rest, not upstream-only.

Verified: swift source-builds against node-addon-api@8 -> N-API binary -> loads
against the pinned tree-sitter@0.21.1 (ABI 14) -> parses cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(publish): gate a lean prebuilds-only npm tarball behind a coverage guard

Vendoring grammar source (parser.c) alongside the prebuilds means the npm
tarball now carries ~50 MB of generated source it almost never compiles (every
supported platform-arch has a prebuild). Prepare to drop it from the published
package once all prebuilds exist — safely.

- .npmignore: add a GATED, commented-out "lean publish" block that excludes the
  source-build inputs (parser.c/scanner.c/tree_sitter/binding.gyp/binding.cc) but
  keeps prebuilds/ + the runtime files. Uncommenting ships prebuilds-only.
- scripts/assert-publish-grammar-coverage.cjs: a prepack guard that refuses to
  pack/publish if the source exclusion is active while any vendored grammar still
  lacks 6/6 prebuilds (which would ship a grammar with no loadable binding). Wired
  into `prepack` (runs on npm pack + publish, incl. the publish.yml dry-run) and
  exposed as `npm run assert-publish-coverage`.
- test: pure-core decision cases + a real-repo publish-safety check that fails CI
  if .npmignore is activated prematurely.

Net: the prebuilds already publish today (files: ["vendor"]); this makes the
future switch to a prebuilds-only tarball a one-line uncomment that can't ship a
dead grammar. The guard currently reports "source + prebuilds" (only swift has
6/6 prebuilds so far) and passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(grammars): consolidate the 5 build-tree-sitter-*.cjs into one

The per-grammar activation scripts (c/dart/proto/swift/kotlin) were ~95%
identical — same prefer-prebuild → source-build → never-fail flow, differing only
in name, target_name, required-vs-optional, and the display label in warnings.

- scripts/build-tree-sitter-grammars.cjs: one registry-driven script. Bare call
  builds all (postinstall); `... <name>` builds only the named grammars (so the
  probe test can isolate one). c is `required: true` (ignores the opt-out gate);
  the rest honor GITNEXUS_SKIP_OPTIONAL_GRAMMARS. Per-grammar try/catch + a final
  process.exit(0) preserve the postinstall never-exit-non-zero invariant.
- package.json: postinstall is now `materialize && build-tree-sitter-grammars.cjs`
  (was five chained `build-tree-sitter-<name>.cjs` calls).
- tests: replace the two near-identical *-probe.test.ts files with one
  parameterized build-tree-sitter-grammars-probe.test.ts that also covers the
  required-vs-optional opt-out split and an unknown-grammar arg.
- update cli-commands.test.ts postinstall assertions + the vendor c/kotlin/swift
  README + swift provenance to reference the consolidated script.

Behavior is preserved (warnings normalized to one consistent format). Removes 5
scripts + 1 test file; adds 1 script + 1 test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): lazy-load tree-sitter-c to prevent module-load crash

tree-sitter-c is now vendored prebuild-only (#2116) with 0/6 committed
prebuilds, so on a toolchain-less or `--ignore-scripts` install C has no native
binding. Three modules loaded it via a hard top-level `import C from
'tree-sitter-c'`, which throws ERR_MODULE_NOT_FOUND at module-load — crashing
`analyze` before parser-loader's optional/severity:error degradation can run.
This is the #2091/#2093 bug class (previously fixed for swift/dart/kotlin); C was
left static because it used to be an always-present npm dependency.

- languages/c/query.ts: load via the lazy guarded getLanguageGrammar(C), mirroring
  swift/query.ts; the main-thread isLanguageAvailable filter ensures the getters
  are reached only when C is present.
- workers/parse-worker.ts: guarded `_require('tree-sitter-c')` + conditional
  languageMap spread, like swift/dart/kotlin.
- group/extractors/include-extractor.ts: guarded `_require`; getLanguageForFile
  returns null for .c/.h when absent, so C include-extraction degrades to a no-op
  (C++ unaffected).
- extend the registry-import-closure regression test (#2091/#2093) to assert C
  also loads lazily at registry static-import time.

* fix(ci): repin attest-build-provenance to the real v2.4.0 SHA

The workflow pinned actions/attest-build-provenance@bd77c077… commented
`# v2.4.0`, but v2.4.0 is e8998f94… (verified via the GitHub API); bd77c077…
is an untagged mid-stream commit, so the SLSA-attestation step ran unvetted
action code and the comment misrepresented what runs. Repin to the real
v2.4.0 commit and drop the `# PLACEHOLDER-PIN` markers on both this line and
the setup-python pin (a26af69b… is already the correct v5.6.0 — only its
comment was stale). Update the header NOTE accordingly.

* fix(ci): skip the prebuild-PR aggregate when release App secrets are absent

The aggregate job mints a GitHub App token as its first step; with
RELEASE_APP_ID/RELEASE_APP_PRIVATE_KEY unset it hard-failed AFTER a full
(up-to-6-runner) native build. Since the `secrets` context isn't available in
a job-level `if:`, the guard job now computes a `release_app` boolean output
(a step can read secrets) and emits an actionable `::notice::`; aggregate
gates on it and skips cleanly, while the build job's artifacts still upload
(run with open_pr=false for artifacts-only).

* chore(ci): drop package-lock.json from the prebuild paths filter; widen build timeout

`gitnexus/package-lock.json` changes on nearly every dependency PR, so it
fired the prebuild workflow's guard job on unrelated churn (the matrix stayed
correctly skipped — `gitnexus/package.json` already covers the transition-window
pin, so removing the lock only drops guard noise). Also bump the native build
job timeout 30 -> 45 min for headroom compiling the 23 MB kotlin / 18 MB swift
parser.c, especially under arm emulation.

* fix(ci): event-gate the aggregate open-PR condition explicitly

`inputs.open_pr` is null on pull_request events, and the prior
`inputs.open_pr != false` leg relied on GHA's direction-ambiguous null
coercion (Codex F4) to decide whether to open the prebuild PR. Gate
explicitly on the event: a non-fork pull_request that bumped a grammar
version opens the prebuild PR (the documented flow), and `open_pr` is only
consulted on workflow_dispatch — so a manual run with open_pr=false stays
artifacts-only and no event's behavior rests on coercion.

* fix(publish): validate the effective npm-pack contents in the coverage guard

The publish guard inferred "is source shipped?" from a single .npmignore toggle
line, which a partial/out-of-order edit could defeat (exclude binding.gyp but
leave parser.c → unbuildable yet "source-shipping"). It now inspects the
EFFECTIVE tarball via `npm pack --dry-run --ignore-scripts --json` (the
--ignore-scripts avoids re-entering this guard through prepack): a grammar
"ships source" only when EVERY on-disk source-build input (binding.gyp +
binding.cc + parser.c + scanner.c when present + a tree_sitter header) is
actually in the packed file list.

This also surfaced that the gated lean-publish .npmignore block was inert:
package.json's `files: ["vendor"]` allow-list overrides .npmignore for the
vendored subtree, so those exclusion lines never dropped anything. Replace the
dead toggle with documentation of the real mechanism (narrow the `files` field)
and note the guard enforces safety on the effective pack regardless of how the
slim is done.

* test(prebuild): hard-gate declared-fully-prebuilt grammars on 6/6 coverage

The strict 6/6 prebuild assertion was dormant whenever a grammar vendors source
(binding.gyp) — which is every grammar — so a dropped prebuild passed CI
silently. Add a FULLY_PREBUILT allowlist of grammars GitNexus has committed 6/6
for (today: swift); those must keep all six even with a source fallback, so
losing one now fails CI. Grammars graduate into the set as the
build-tree-sitter-prebuilds workflow lands their binaries. (The static-import
degradation smoke is covered by the registry-import-closure regression test
extended in the C lazy-load commit.)

* chore(deps): promote node-gyp-build/node-addon-api to regular dependencies

Every vendored grammar's index.js does `require("node-gyp-build")` at runtime
to load even a prebuilt .node, so node-gyp-build is runtime-load-critical (and
node-addon-api is needed for the source-build fallback). They were
optionalDependencies, surviving `--omit=optional` only via the required
tree-sitter's transitive edge — correct today but fragile. Promote both to
regular dependencies so the contract is explicit (optionalDependencies is now
empty and removed). Lock the contract with a cli-commands assertion.

* chore(vendor): add Windows cflags parity block to tree-sitter-c/binding.gyp

c's binding.gyp used an unconditional `cflags_c: ["-std=c11"]`, while
kotlin/swift gate MSVC flags behind an `OS=='win'` condition (/std:c11 /utf-8).
Inert today (no non-ASCII bytes in c's parser.c, and node-gyp ignores cflags_c
on MSVC anyway), but align the three so a future source-build fallback on
Windows behaves consistently.

* docs(agents): correct stale optional-grammar / postinstall notes

AGENTS.md still said postinstall "patches tree-sitter-swift, builds
tree-sitter-proto" and that only kotlin/swift are "optional". Update to the
vendored-uniform model: postinstall materializes the vendored grammars and
prefers a committed prebuild (source-build only when none matches); c is
required while dart/proto/swift/kotlin are optional + skippable via
GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1, with non-fatal warnings only on a
toolchain-less host with no matching prebuild.

* fix(install): preserve the backup and warn loudly on a failed materialize rollback

If renameSync(partial, dest) failed AND the rollback renameSync(backup, dest)
also failed, the grammar was left unmaterialized (node_modules/<name> missing)
with only a generic "could not materialize" warning — the recoverable backup at
<dest>.materialize-bak was unmentioned. Emit a CRITICAL warning naming the
backup path and the recovery command on that double-failure, and document that
the fail-soft catch removes only the scratch `partial`, never the `backup`
(which may be the sole recoverable copy). Never-throw / exit-0 contract intact.

* fix(publish): make the coverage guard's npm-pack inspection script-safe

The prepack guard shelled out to `npm pack --dry-run --ignore-scripts --json`,
but the `--ignore-scripts` flag is not reliably honored by npm pack's
prepare/prepack lifecycle on the CI npm — so build.js ran, polluted the --json
stdout with `[build] …`, and the guard's JSON.parse threw. That broke every
`npm pack` (packaged-install-smoke on ubuntu+windows) and failed the guard's own
real-repo unit test (the only coverage-job failure). Force script-skipping via
the reliable `npm_config_ignore_scripts` env config (also removes the prepack
re-entry/recursion risk) and parse defensively from the JSON-array start.

* fix(publish): make the coverage guard deterministic — read `files`, not `npm pack`

The npm-pack-based guard timed out in CI: `npm pack`'s prepare/prepack lifecycle
is not skipped by `--ignore-scripts` (flag or env config) on the CI npm, so the
inner pack ran the full build (~20s+) — fine for the slow smoke job, but it blew
past vitest's 30s test timeout in the coverage job (and risked re-entering this
prepack guard).

Replace it with a deterministic, fast (~0.1s) check that needs no subprocess:
since `files: ["vendor"]` OVERRIDES `.npmignore` for the vendored subtree (so
`.npmignore` can never drop vendored source — verified), the ONLY lever that can
exclude source is narrowing the package.json `files` field. The guard now reads
`files` directly: a grammar "ships source" iff `files` includes the vendor
subtree AND the grammar carries a buildable source set on disk. A lean publish
that narrows `files` while a grammar lacks 6/6 prebuilds still fails the gate.

* feat(ci): vendored tree-sitter grammar update monitor

Adds a weekly (+ dispatchable) workflow that checks each vendored grammar against
its source-of-origin (npm for swift/kotlin, the GitHub default branch for
dart/proto; c is excluded — held at 0.21.4 for ABI safety) and opens a PR
re-vendoring any update that is ABI-COMPATIBLE with the pinned tree-sitter@0.21.1
(LANGUAGE_VERSION 13-14).

ABI awareness is the point: most upstreams have moved to ABI 15 (newer
tree-sitter), so a blind "bump to latest" would open PRs that can't build. The
monitor fetches the candidate source, reads its parser.c LANGUAGE_VERSION, and
only re-vendors 13/14 — incompatible updates are reported (notice + job summary),
never applied. (Confirmed live: dart/proto upstreams are ABI 15 today and are
correctly held; swift/kotlin are current.)

The re-vendor refreshes only the source-build inputs + runtime entrypoints,
preserving the GitNexus-hardened binding.gyp / README / prebuilds; the version
bump then triggers build-tree-sitter-prebuilds.yml, whose ABI-validation is the
final safety net so a subtly-wrong re-vendor can't silently ship. PR creation is
gated on the RELEASE_APP secret (skips with a notice if absent), mirroring the
build aggregate. Unit test locks the ABI gate; the script is import-safe.

* feat(ci): monitor tree-sitter-c too (report-only, ABI-pinned)

c was excluded from the update monitor, so an upstream c update went unnoticed.
Include it, but as report-only via a `hold`: c is ABI-pinned at 0.21.4
(#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an
available c update is detected + surfaced (notice + job summary) but never
auto-PR'd — even if it were ABI-13/14. `--apply c` refuses defensively. (Live:
upstream c is 0.24.1 / ABI 15 today, so c is doubly held — reported, not applied.)

* fix(ci): drop the shell in the grammar monitor's github fetch (CodeQL)

CodeQL flagged the GitHub-tarball fetch — it used `bash -c "gh api …/tarball/$ref
> src.tgz && tar xzf src.tgz"`, interpolating the API-derived ref into a shell
command (the shell-command-injection family: "this shell command depends on an
uncontrolled file name"). Replace it with a shell-free path: capture `gh api`'s
binary tarball as a Buffer via execFileSync, write it to a fixed file, and
extract with execFileSync('tar', …). No shell, no injection surface. Verified the
dart/proto fetch + ABI read still work.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 18:16:24 +01:00
Gergő Magyar
d3de5fa5d5
fix(install): materialize vendored grammars to fix Windows EPERM (#1728) (#1729) 
* fix(install): materialize vendored grammars to fix Windows EPERM (#1728)

Stop using file: optionalDependencies for tree-sitter-dart/proto/swift,
which made npm symlink vendor paths on install and fail on Windows without
symlink privileges. Copy vendor trees into node_modules at postinstall
instead; keep native builds and #836 vendor hygiene.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(install): atomic materialize swap + fail-soft tests (#1728, #836)

Hardens PR #1729 against two issues the original implementation could
still hit:

1. Torn-state on rmSync→cpSync. The previous loop deleted the
   destination before copying. If cpSync threw — the exact Windows EPERM
   scenario this PR targets — a previously-working grammar was silently
   wiped. Now we copy to {dest}.materialize-tmp first and renameSync into
   place, so an interrupted copy leaves the prior materialization intact.

2. Fail-soft try/catch had no test coverage. Adds two POSIX-only tests
   (chmod 0o555 to deterministically force cpSync to throw) that verify
   (a) a single grammar failure does not abort the other two, and (b) an
   existing materialization survives a partial-copy failure. Skipped on
   Windows where chmod doesn't enforce write restriction; runs on Linux
   CI.

Other test improvements locking in the install-hygiene invariants:

- All three vendored grammars (dart/proto/swift) checked, not just dart.
- GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 short-circuit is exercised.
- Vendor cleanliness (#836): no node_modules/build under vendor/.
- Idempotent re-runs (clean overwrite verified via sentinel file).
- Missing-vendor warn+continue path now has explicit coverage.
- Vendored package manifests asserted to carry no install script or
  runtime dependencies.
- package.json optionalDependencies asserted free of vendored grammars.
- package-lock.json assertion tightened from `if (entry !== undefined)
  { expect(entry.link).not.toBe(true); }` (vacuous when entry is absent,
  i.e. the expected post-fix state) to `expect(...).toBeUndefined()`.

Verified locally:
- npx tsc --noEmit: clean
- vitest test/unit/materialize-vendor-grammars.test.ts: 8 pass + 2
  POSIX-only skipped on Windows
- npm pack tarball: no vendor/*/node_modules or vendor/*/build entries
- Isolated global install (clean + upgrade + SKIP env) into temp prefix:
  succeeds; gitnexus --version → 1.6.5; vendor stays clean post-install.

* fix(install): address review feedback — Swift parity, atomicity, CI smoke

Resolves all findings from the automated production-readiness review on
verify/issue-1728-symlink.

Swift warning parity (review #2):
  Add tree-sitter-swift to OPTIONAL_GRAMMARS in src/cli/optional-grammars.ts
  alongside Dart and Proto. Before this commit, Swift was materialized at
  postinstall and probed by build-tree-sitter-swift.cjs but the runtime
  warnMissingOptionalGrammars() never warned when it failed to load —
  users got silent Swift degradation from the optional-grammars surface
  (parser-loader's separate unavailableNote only fires on demand). Now
  the warning path matches the materialize path.

README env-var table (review #1):
  Update the GITNEXUS_SKIP_OPTIONAL_GRAMMARS row at README.md line 248 to
  list all three vendored grammars (dart, proto, swift). The quick note
  earlier in the README already mentioned all three; only the table row
  was stale.

Atomicity hardening (review #3):
  materialize-vendor-grammars.cjs now copies to {dest}.materialize-tmp,
  renames the existing dest to {dest}.materialize-bak (if present), then
  renames the partial into dest, then removes the backup. If the
  partial→dest rename fails (e.g. Windows AV scanner racing the swap),
  the catch block restores from backup so the previously-materialized
  grammar is preserved. Closes the narrow torn-state window where the
  prior implementation could leave dest deleted after rmSync succeeded
  but renameSync failed.

Swift probe docs (review #4):
  build-tree-sitter-swift.cjs script header rewritten to describe what
  the script actually does — probe node-gyp-build at install time so
  missing-prebuild failures surface as install-time warnings instead of
  first-parse runtime errors. The script does not "activate" anything;
  the runtime require() in parser-loader does the actual load. Console
  warning text updated to match ("prebuild probe" not "activation").

Windows packaged-install smoke test (review #5):
  New CI job `packaged-install-smoke` in .github/workflows/ci-tests.yml
  matrices on windows-latest and ubuntu-latest. Runs npm pack, installs
  the produced tarball globally into RUNNER_TEMP, then asserts:
    * no vendor/*/node_modules or vendor/*/build (#836 invariant)
    * tree-sitter-{dart,proto,swift} in node_modules are real
      directories, not junctions/symlinks (#1728 invariant)
    * gitnexus --version runs against the installed CLI
  Closes the coverage gap where the existing windows-latest job only
  ran `npm ci` in the source checkout — exercising postinstall but not
  the tarball reify step that historically tripped EPERM.

Verified locally:
  npx tsc --noEmit: clean
  vitest test/unit/materialize-vendor-grammars.test.ts test/unit/cli-commands.test.ts:
    18 pass + 2 POSIX-only skipped on Windows
  prettier + eslint on all changed files: clean

* fix(ci): disable credential persistence on packaged-install-smoke checkout

GitHub Advanced Security (zizmor artipacked) flagged the new
packaged-install-smoke job's actions/checkout step as a potential
credential-persistence risk. The job runs `npm pack` + global install
and never pushes back, so the GITHUB_TOKEN that checkout would persist
in .git/config provides no value and only widens the leak surface (any
future artifact-upload step in this job would carry the token).

Disable persistence explicitly via `persist-credentials: false` on this
job's checkout. Scoped to the new job — pre-existing checkouts above
are left unchanged.

* fix(ci): use find instead of ls for tarball lookup (SC2012)

actionlint shellcheck SC2012 flagged `TARBALL=$(ls gitnexus-*.tgz | head -n1)`.
Switch to `find . -maxdepth 1 -name 'gitnexus-*.tgz' -print -quit` which
handles non-alphanumeric filenames safely. Also add an explicit
empty-result check so the failure mode is a clear error message instead
of a silent `npm install -g ""` later.

* fix(tests): sabotage vendor src (not partial path) in POSIX fail-soft tests

The fail-soft tests in materialize-vendor-grammars.test.ts pre-chmod'd
the destination's .materialize-tmp partial directory to 0o555 to force
cpSync to throw. After the atomicity rewrite (`fix(install): atomic
materialize swap + fail-soft tests`), the materialize script now starts
each grammar's loop with `fs.rmSync(partial, { force: true })`, which
deletes the chmod'd sabotage before cpSync runs — so cpSync succeeds and
the partial is then renamed into dest, leaving the test's `finally`
block with no path to chmod back (ENOENT) and the assertion that proto
remained unmaterialized failing because it materialized cleanly.

Fix: sabotage the *vendor source* directory (which the script reads from
but never modifies) by chmod'ing it to 0o000. cpSync then fails on
readdir, the catch block fires per-grammar, dart and swift still
materialize from their unaffected sources, and the existing-dest
preservation test verifies that a sabotaged second-run leaves the prior
materialization (and its sentinel file) intact.

Tests now pass locally (8 pass + 2 POSIX-only skipped on Windows) and
should pass on macOS/Ubuntu CI where the sabotage runs.

* fix(tests): restrict fail-soft tests to Linux (macOS Node cpSync abort)

Node 22 on macOS aborts the process with `libc++abi: terminating due
to uncaught exception filesystem_error` when fs.cpSync hits a source
directory it can't read — the abort happens at the C++ filesystem layer
and bypasses Node's JS try/catch entirely (nodejs/node#51399). My
chmod-0o000-the-source sabotage strategy triggers this SIGABRT on
macOS CI before the production script's `try { cpSync } catch` ever
runs, so the test sees a child-process crash instead of the fail-soft
warning it's verifying.

The production script's fail-soft is correct on Linux (where EACCES
surfaces as a normal JS exception) and effectively untestable on macOS
via permission sabotage. Real installs don't hit this — npm always
ships vendor/ with readable permissions — so the macOS gap is a test
artifact, not a behavior gap.

Restrict the two chmod-based tests to Linux only by replacing
`skipOnWin` with `linuxOnly`. Linux CI continues to verify both the
one-grammar-fails-others-succeed and existing-materialization-preserved
invariants. macOS and Windows runs skip these two scenarios; the other
8 tests still run on every platform.

* fix(tests): remove materialize unit tests, rely on CI smoke job

The materialize-vendor-grammars.test.ts file has been a recurring source
of platform-specific CI noise:

  - Windows: chmod doesn't enforce read/write restrictions the way POSIX
    does, so the fail-soft tests had to be skipped there.
  - macOS Node 22: cpSync against an unreadable source aborts the process
    with a libc++ filesystem_error (nodejs/node#51399) that bypasses JS
    try/catch entirely — making the chmod-based fail-soft tests
    unrunnable on macOS too.
  - The "vendor-cleanliness" and "idempotency" tests on Windows
    intermittently flake due to fs.cpSync timing on the GitHub runner.

The invariants these tests verified are now covered by stronger,
more realistic surfaces:

  - packaged-install-smoke (ci-tests.yml): runs `npm pack` then
    `npm install -g ./gitnexus-*.tgz` on windows-latest and
    ubuntu-latest, then asserts no vendor/*/node_modules,
    no vendor/*/build (#836), no junctions/symlinks on the
    materialized grammar directories (#1728), and a working
    `gitnexus --version`. This is the actual end-user install path.

  - cli-commands.test.ts (kept, unmodified): asserts package.json
    declares no `file:` optionalDependencies for vendored grammars,
    the Swift vendor manifest carries no install script or
    dependencies, and the postinstall chain runs
    materialize-vendor-grammars.cjs + build-tree-sitter-swift.cjs.
    These are static manifest checks — deterministic, fast, no
    flake risk.

Removing the dynamic script-execution tests trades unit-level coverage
for end-to-end smoke coverage that actually exercises the
`file:` → cpSync change against a real npm install lifecycle, on
the platform the fix targets (windows-latest).

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-21 16:47:22 +01:00
Gergő Magyar
1f6df5fdbb
fix(swift): use official prebuilt parser runtime (#1130) 
* fix(swift): use official prebuilt parser runtime

Vendor the official tree-sitter-swift 0.7.1 runtime package so Swift parsing works without source-building, while keeping the repo on the current tree-sitter runtime until the broader upgrade is ready. Also preserves Swift resolver correctness for overloaded owned functions and extension-backed type duplicates now that Swift is available by default.

Made-with: Cursor

* fix(swift): move duplicate type ordering into provider

Keep Swift extension candidate ordering behind the LanguageProvider contract and cover the Swift 0.7 init scanner path so parser runtime changes do not leak language-specific logic into shared resolution.

Made-with: Cursor

* fix(swift): address parser runtime review

Add explicit Swift prebuild checks and vendor guidance so parser runtime packaging remains observable and maintainable.
 2026-04-28 09:57:42 +01:00
Gergő Magyar
780ee83d52
fix(install): vendor tree-sitter-dart source (#1125) 
Avoid remote git/SSH downloads for the Dart grammar during Docker and npm installs by resolving tree-sitter-dart from vendored source and building it during postinstall.

Made-with: Cursor
 2026-04-27 21:15:37 +01:00
Gergő Magyar
fec06b823c
fix: devendor tree-sitter-proto install lifecycle to prevent ENOTEMPTY on global upgrade (#846)
Some checks are pending
CI / quality (push) Waiting to run
Details
CI / tests (push) Waiting to run
Details
CI / e2e (push) Waiting to run
Details
CI / global-upgrade (push) Waiting to run
Details
CI / Save PR Metadata (push) Blocked by required conditions
Details
CI / CI Gate (push) Blocked by required conditions
Details
Release Candidate / Check if release candidate should run (push) Waiting to run
Details
Release Candidate / ci (push) Blocked by required conditions
Details
Release Candidate / Publish release candidate to npm (push) Blocked by required conditions
Details 
* fix: devendor tree-sitter-proto install lifecycle to fix ENOTEMPTY on global upgrade

PR #843's preinstall cleanup hook cannot address the reported bug because
it runs on the NEW package's staging tree, not the OLD install being
removed. Issue #836 still reproduces on 1.6.2-rc.8.

Root cause: vendor/tree-sitter-proto was declared as `file:` dep with its
own `dependencies` and `install` script, so npm created
`vendor/tree-sitter-proto/node_modules/node-addon-api/` at install time,
which blocked npm's rmdir on global upgrade.

Changes:
- Strip `dependencies` and `install` script from the vendored sub-package's
  package.json so npm no longer creates a nested node_modules or runs a
  lifecycle script under vendor/.
- Hoist `node-addon-api` and `node-gyp-build` into gitnexus
  optionalDependencies; npm resolves them at the consumer's top level.
- Add scripts/build-tree-sitter-proto.cjs modeled on patch-tree-sitter-swift.cjs.
  Runs at gitnexus postinstall, best-effort: skips cleanly on missing
  toolchain or --ignore-scripts so non-proto functionality keeps working.
- Remove scripts/preinstall-cleanup.cjs — dead code; cannot run against
  the old install being removed.
- Keep .npmignore entries from PR #843 (tarball hygiene, still correct).
- Add explicit .gitignore rules for gitnexus/vendor/**/build and
  gitnexus/vendor/**/node_modules (closes the repo-side hygiene gap).
- Add .github/workflows/ci-global-upgrade.yml: matrix smoke test that
  installs the previously-published rc globally, upgrades to the packed
  current branch, and verifies no vendor install-time artifacts survive.
  Runs on macOS (reporter's platform), Linux, and Windows. Also includes
  an --ignore-scripts degraded-mode lane. Wired into ci.yml gate.

Plan: docs/plans/2026-04-15-002-fix-tree-sitter-proto-vendor-deps-plan.md

Phase 1 (this commit) addresses the reported `node_modules/node-addon-api`
hazard. Phase 2 (follow-up) will migrate to prebuildify + prebuilt .node
binaries in the tarball — the 2026 canonical shape for tree-sitter
grammars, which eliminates the postinstall compile path entirely.

Refs #836

* fix(ci): ci-global-upgrade should be reusable-only and use setup-gitnexus

Three issues caught by CI on PR #846:

1. Concurrency linter rejected the `CIGU-` prefix (allowlist is
   `${{ github.workflow }}` or substring `CI-`). The literal-prefix
   guidance in ci.yml is specifically about disambiguating when
   reusable workflows run in nested contexts, and ci-global-upgrade
   doesn't need its own concurrency block at all — the caller
   (ci.yml) already governs concurrency for nested invocations.

2. `npm install` in gitnexus/ runs `prepare: node scripts/build.js`,
   which depends on gitnexus-shared/dist being built first. Other CI
   jobs handle this via the setup-gitnexus composite action. Use it
   here too (with build: 'false' — we only need the dep graph, then
   npm pack runs prepack which builds gitnexus itself).

3. Removed `pull_request` and `workflow_dispatch` triggers. The
   workflow is now pure `workflow_call` — invoked once from ci.yml
   via `uses:`. This avoids the duplicate-run problem where both the
   top-level pull_request trigger AND the nested workflow_call would
   fire on every PR.

* fix(ci): relax vendor build/ guard and use bash shell on Windows

Two fixes for ci-global-upgrade failures on PR #846:

1. The guard after the upgrade step was rejecting vendor/tree-sitter-proto/build/
   in the global install. That was too strict. The original #836 bug was
   about vendor/tree-sitter-proto/node_modules/ specifically, not build/.
   The build/ directory appears because node-gyp-build compiles through the
   symlink npm creates at node_modules/gitnexus/node_modules/tree-sitter-proto,
   and its contents are plain .node, .obj, .lib files that rmdir handles
   without trouble. We know this empirically because the test got past the
   upgrade step in the run where the old vendor/node_modules was present.
   The guard now only flags nested node_modules, which is what the fix
   actually removes.

2. The Windows --ignore-scripts lane failed with ENOENT when npm tried to
   open the tarball. The path was computed in a bash step using $(pwd),
   which on Windows returns /d/a/... form, but npm install ran in the
   default cmd shell and received a mangled Windows path. Adding
   shell: bash to the install steps keeps path handling consistent.
 2026-04-15 17:38:47 +01:00
ivkond
4d4756fe86
feat(group): extractor expansion + manifest extractor (2/4 of #606 split) (#796) 
* feat(group): extractor expansion + manifest extractor

Part 2 of 4 in the split of #606 (ticket: #792). Follows #795
(bridge.lbug storage foundation, already merged), but this PR has no
code-level dependency on #795 — it only imports types and the
ContractExtractor interface that existed on upstream main before
either PR. It could have been reviewed in parallel with #795.

## What changed

Expands the 3 existing contract extractors with substantially more
language/framework coverage, and adds a new `manifest-extractor`
that resolves `group.yaml`-declared cross-links against the per-repo
graph via exact-name lookups.

### New file (228 LOC)

- `gitnexus/src/core/group/extractors/manifest-extractor.ts` —
  exact graph lookup for `group.yaml`-declared cross-links. HTTP
  paths are canonicalized before Route.name matching; gRPC is
  resolved by service/method name (NO `.proto`-filename fallback);
  topic and lib use exact-name match. Falls back to a synthetic
  `manifest::<repo>::<contractId>` uid when the graph has no
  matching symbol, so cross-impact traversal still has a stable
  anchor for the contract.

### Modified extractors (+958 LOC prod)

- `extractors/grpc-extractor.ts` (+522) — `.proto` parser with
  comment and string-literal sanitization (braces inside strings no
  longer truncate service bodies); package/service/method canonical
  IDs; server/client detection across Go (`grpc.NewServer`,
  `RegisterXxxServer`, `XxxGrpc.XxxImplBase`), Java (`@GrpcService`,
  `BlockingStub`), Python (`servicer_to_server`, `XxxStub`), and
  TypeScript/Node (`@GrpcMethod`, `ClientGrpc`, `loadPackageDefinition`).
- `extractors/http-route-extractor.ts` (+174) — Go gin/echo/stdlib
  `HandleFunc`, NestJS `@Controller`+`@Get`/etc, Python FastAPI
  decorators, Java Spring `@RequestMapping`/`@GetMapping`,
  restTemplate / WebClient / OkHttp consumers.
- `extractors/topic-extractor.ts` (+98) — sarama `ProducerMessage{}`
  struct literal detection (replaces a constructor-anchored regex
  that missed topics inside producer loops), kafka-go Writer/Reader,
  Python NATS (`await nc.subscribe`/`await nc.publish`), JetStream
  helpers.

### Modified and new tests (+1264 LOC)

- `grpc-extractor.test.ts` (+539) — full coverage of the new proto
  parser (strings-with-braces regression, comments-with-braces
  regression), per-language server/client detection
- `http-route-extractor.test.ts` (+240) — per-framework route
  extraction + normalization edge cases
- `topic-extractor.test.ts` (+177) — the sarama in-loop regression,
  JetStream, Python NATS, kafka-go Writer/Reader
- `manifest-extractor.test.ts` (+308 NEW) — HTTP path normalization,
  gRPC exact lookup with proto-fallback regression, lib and topic
  exact matching, synthetic-uid fallback behavior

### Self-review fixes folded in

Carried forward from the #606 self-review (commit `d15b8cb`):

- **HIGH #1** — `manifest-extractor.resolveSymbol` was too fuzzy.
  Previously used `CONTAINS` on route/name fields plus an
  unconditional `filePath ENDS WITH '.proto'` fallback for gRPC.
  Consequences: `/orders` matched `/suborders`, and any repo with
  any `.proto` file returned a random proto symbol for a gRPC
  manifest entry. Replaced with exact equality + deterministic
  `ORDER BY` + synthetic-uid fallback for unresolved manifests.
  Regression tests included.
- **MED #3** — gRPC proto parser brace-depth counting now sanitizes
  strings and comments first (`stripProtoCommentsAndStrings`). A
  valid proto with `option deprecated_reason = "use NewService {
  instead"` used to have its service body closed early by the `"{"`
  inside the literal, silently dropping methods after the offending
  string. Regression tests for both string-with-brace and
  comment-with-brace cases.
- **MED #4** — sarama Kafka regex changed from
  `sarama.NewSyncProducer[\s\S]{0,300}?Topic:` (anchored on
  constructor, caught only first topic in a loop) to
  `sarama.ProducerMessage{...Topic:}` (matches every struct literal
  directly). Regression test with a for-loop that constructs
  multiple `ProducerMessage`s.
- **MED #7** — `manifest-extractor.resolveSymbol` no longer has a
  silent `catch { /* fall through */ }`. Errors from the graph
  executor are logged via `console.warn` with link type, contract
  name, repo key, and error message before falling through to the
  synthetic-uid path.

## Why

Reviewer focus here is pure regex / parser correctness — no
storage, no Cypher queries, no algorithmic changes to the cross-link
algorithm. Separating this from the bridge foundation PR (#795)
meant reviewers could stay in a single mental mode (parsing logic)
instead of context-switching between DDL, Cypher, and regex.

## How to verify

- `cd gitnexus && npx tsc --noEmit`
- `cd gitnexus && npx vitest run test/unit/group/grpc-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/http-route-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/topic-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/manifest-extractor.test.ts --pool=forks`

Local pre-push: typecheck clean, all 99 extractor unit tests pass
(grpc 43, http 18, topic 30, manifest 8).

## Risk / rollback

**Low.** Extractors have no user-facing surface in this PR — they
produce `ExtractedContract[]` that is consumed by `sync.ts` in the
next split (#793). No existing behavior changes for users who don't
run a `group sync`. Rollback = `git revert` of the merge commit;
the modifications to `grpc-extractor.ts` / `http-route-extractor.ts`
/ `topic-extractor.ts` revert to the pre-PR versions that still
work (they're subsets of the new functionality).

## Scope discipline (per GUARDRAILS.md)

- Only the 8 files above are touched; no drive-by refactors
- No CI/release/security config changes
- No secrets or machine-specific paths
- Content lifted from #606 (CI 11/11 green on `d15b8cb`)

## Dependencies

- **Base:** `main` (upstream already includes #795 as `1ff324c`)
- **Blocks:** sync pipeline (#793) and the cross-impact feature (#794)
- **Tracker issue:** #792
- **Parent PR:** #606

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate topic-extractor from regex to tree-sitter queries

Addresses @magyargergo's feedback on #796 that regex-based lookups
should use tree-sitter nodes instead, and that the top-level
extractors must NOT carry language dependencies. This is phase 1 of
a multi-step migration — topic-extractor first because its patterns
are the most uniform (16 "call/annotation with first-arg string
literal" variants), which makes it a clean proof of the approach
before grpc-extractor and http-route-extractor get the same treatment.

## Architecture: language-agnostic orchestrator + per-language plugins

The top-level extractor is a thin orchestrator that never imports a
tree-sitter grammar or a query string. Per-language knowledge lives
in a new `topic-patterns/` folder with one file per language plus a
registry that maps file extensions to compiled plugins:

```
src/core/group/extractors/
├── tree-sitter-scanner.ts         # shared, language-agnostic scanning utilities
├── topic-extractor.ts              # thin orchestrator (no grammar imports)
└── topic-patterns/
    ├── types.ts                    # TopicMeta, Broker
    ├── index.ts                    # registry: extension → compiled provider
    ├── java.ts                     # tree-sitter-java + JAVA_TOPIC_PROVIDER
    ├── go.ts                       # tree-sitter-go + GO_TOPIC_PROVIDER
    ├── python.ts                   # tree-sitter-python + PYTHON_TOPIC_PROVIDER
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript
                                    # → JAVASCRIPT_/TYPESCRIPT_/TSX_TOPIC_PROVIDER
```

**Shared scanner (`tree-sitter-scanner.ts`)** — defines
`PatternSpec<TMeta>`, `LanguagePatterns<TMeta>`, `CompiledPatterns<TMeta>`
and the `scanFile(parser, plugin, content)` helper. Plugins compile their
queries eagerly at module load via `compilePatterns()`, so a broken
pattern fails loudly at import time instead of silently at scan time.
`unquoteLiteral()` handles single/double/template quotes, Python
triple-quoted strings, and Go raw backtick strings.

**Per-language plugins** own:
- the tree-sitter grammar import (this is the ONLY place in
  `src/core/group/` where tree-sitter grammars are imported),
- the query S-expressions,
- the `TopicMeta` payload (role, broker, confidence, symbolName) that
  the orchestrator receives back on every match.

Each plugin uses a `@value` capture name to bind the topic literal node.
The JavaScript and TypeScript grammars share AST node names for every
construct we query, so `node.ts` defines the pattern sources once and
compiles them against `JavaScript`, `TypeScript.typescript`, and
`TypeScript.tsx` — exporting three providers because `Parser.Query`
objects are NOT portable across grammar instances.

**Registry (`topic-patterns/index.ts`)** — maps `.java` → Java provider,
`.go` → Go, `.py` → Python, `.js`/`.jsx` → JS, `.ts` → TS, `.tsx` → TSX.
Also exports `TOPIC_SCAN_GLOB` so adding a new language is a single
file-level edit (drop `topic-patterns/<lang>.ts`, import + register it
here — zero edits required in `topic-extractor.ts`).

**Orchestrator (`topic-extractor.ts`)** — ~110 lines, no grammar or
query imports. Per file: `getProviderForFile(rel)` → `scanFile(parser,
provider, content)` → `unquoteLiteral(valueText)` → `makeContract(...)`.
Reuses one `Parser` instance across files; the scanner calls
`setLanguage` per plugin.

## Why this is better than regex

1. **Comments and strings are respected for free.** The old regex
   would match `// kafkaTemplate.send("fake.topic")` as a real
   producer; tree-sitter never visits comments or string literals as
   code nodes, so false positives from commented-out code are
   eliminated.
2. **Struct/object literal patterns are structural, not textual.**
   `sarama.ProducerMessage{Topic: "..."}` no longer needs a 300-char
   lookahead (which was a known cross-match bug partly mitigated by a
   loop regression test in the self-review). The new query matches a
   specific `composite_literal` with a specific `qualified_type` and
   `keyed_element` — exactly one struct literal per match.
3. **No order-of-operations fragility.** Regex for
   `channel.publish` vs `channel.consume` was independent and
   file-wide; the AST scopes matches to the specific `call_expression`.
4. **Language-agnostic extension.** Adding Ruby, Rust, or C# topic
   detection later means dropping one file in `topic-patterns/` — no
   changes to shared scanner or orchestrator, and no tree-sitter
   imports leak into top-level code.

## Per-file fault tolerance

- Malformed files that tree-sitter can't parse are silently skipped
  (`parser.parse` is wrapped by `scanFile`). The ingestion pipeline
  already logs unparseable files at index time.
- A syntactically invalid query is caught at `compilePatterns` time,
  not scan time — broken plugins fail loudly at import.
- Per-pattern `matches()` failures are swallowed so one broken query
  in a plugin doesn't block the rest.

## Tests

All 30 existing `topic-extractor.test.ts` tests pass **without any
changes to the test file** — they were written as input/output contract
tests (given this source file, expect these `ExtractedContract` objects)
and that contract is unchanged. Regression coverage includes:

- Kafka: Java `@KafkaListener` + `kafkaTemplate.send`; Node
  `producer.send` + `consumer.subscribe`; Go sarama producer/consumer
  (sync and async); kafka-go Writer/Reader; Python `KafkaConsumer` +
  `producer.send/produce`
- RabbitMQ: Java `@RabbitListener` + `rabbitTemplate.convertAndSend`;
  Node `channel.consume/publish/sendToQueue`; Python `basic_consume/
  basic_publish` with keyword args
- NATS: Go and Node `nc.Subscribe/Publish`; Go and Node JetStream
  `js.Subscribe/Publish`; Python `await nc.subscribe/publish`

Including the regression test for the sarama `ProducerMessage`
in-loop case — the AST-based query captures every literal in the
file independently, not just the first one after `NewSyncProducer`.

## Neighbor regression check

- `topic-extractor.test.ts` — 30/30 pass (rewritten extractor)
- `http-route-extractor.test.ts` — 18/18 pass (untouched)
- `grpc-extractor.test.ts` — 43/43 pass (untouched)
- `manifest-extractor.test.ts` — 8/8 pass (untouched)
- Full `npx tsc --noEmit` clean

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched; no
  changes to other extractors, tests, MCP surface, or pipeline.ts.
- No CI/release/security config changes, no secrets.
- New tree-sitter imports all reference grammars that are already
  installed as dependencies (`tree-sitter`, `tree-sitter-javascript`,
  `tree-sitter-typescript`, `tree-sitter-python`, `tree-sitter-java`,
  `tree-sitter-go` — all in `package.json` for the existing pipeline).

## Phase 2 / phase 3 plan

- **Phase 2 (next commit):** rewrite `http-route-extractor.ts`
  Strategy B (regex fallback) on the same plugin pattern. Graph-assisted
  Strategy A stays as-is (already uses pipeline-built tree-sitter data
  via `HANDLES_ROUTE` Cypher queries).
- **Phase 3 (commit after):** rewrite `grpc-extractor.ts` for Java /
  Go / Python / TypeScript detection. `.proto` files are the one
  outstanding question — there is no `tree-sitter-proto` grammar
  installed; the in-tree string-sanitizing parser stays as a pragmatic
  exception with a comment, alternative being to add
  `tree-sitter-proto` as a dep (open for the maintainer).

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate http-route-extractor Strategy B to tree-sitter plugins

Phase 2 of the extractor refactor requested by @magyargergo on #796.
Same architecture as the phase 1 topic-extractor rewrite: a thin,
language-agnostic orchestrator plus per-language plugins that own
tree-sitter grammars and query sources. The top-level extractor file
no longer imports any tree-sitter grammar or query string.

## Architecture

```
src/core/group/extractors/
├── tree-sitter-scanner.ts          # shared, language-agnostic primitives
├── http-route-extractor.ts         # thin orchestrator (no grammar imports)
└── http-patterns/
    ├── types.ts                    # HttpDetection, HttpLanguagePlugin, HttpRole
    ├── index.ts                    # registry: ext → plugin + HTTP_SCAN_GLOB
    ├── java.ts                     # tree-sitter-java: Spring + RestTemplate/WebClient/OkHttp
    ├── go.ts                       # tree-sitter-go: gin/echo/HandleFunc + http/resty consumers
    ├── python.ts                   # tree-sitter-python: FastAPI + requests
    ├── php.ts                      # tree-sitter-php: Laravel Route::get/...
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript:
                                    #   NestJS controllers, Express, fetch, axios
```

**Shared scanner (`tree-sitter-scanner.ts`)** — generalised from phase 1:
- `ScanMatch<TMeta>.captures` is now a full `CaptureMap` (every named
  capture the query binds, not just a single `@value`). Topic extractor
  updated to read `match.captures.value` accordingly.
- New `runCompiledPatterns(plugin, tree)` helper lets plugins run
  multiple query bundles against the same pre-parsed tree. This is
  needed for HTTP plugins that combine a class-prefix query with a
  method-route query (Spring, NestJS).
- `scanFile` becomes a thin wrapper over `parser.parse + runCompiledPatterns`.

**HTTP plugin shape** — unlike topic plugins, HTTP plugins expose a
`scan(tree)` function rather than a flat pattern list. This reflects
HTTP's more complex extraction: each detection needs method + path +
handler name, and framework patterns like Spring `@RequestMapping` /
NestJS `@Controller` require cross-referencing a class-level prefix
with method-level annotations. Plugins internally use
`compilePatterns` + `runCompiledPatterns` and walk the AST to resolve
the class/method relationships.

**Per-framework coverage:**

- **Java (`java.ts`)**
  - Spring: `@RequestMapping("/api/v2")` class prefix + `@(Get|Post|Put|
    Delete|Patch)Mapping("/sub")` method routes, joined via the
    enclosing `class_declaration` node id.
  - `RestTemplate.getForObject/postForEntity/put/delete/patchForObject` →
    method derived from API name.
  - `WebClient.method(HttpMethod.X, "/path")` → method from
    `HttpMethod.X` capture.
  - `new Request.Builder().url("/path")` → OkHttp consumer.

- **Go (`go.ts`)**
  - gin / echo / chi frameworks: `\w+.GET("/path", handler)` captures
    upper-case verb + handler identifier.
  - `net/http.HandleFunc("/path", handler)` → provider (default GET).
  - `http.Get/Post/Head` consumer, `http.NewRequest("METHOD", ...)`,
    resty `client.R().Get/Post/...`.

- **Python (`python.ts`)**
  - `@app.get("/path")` FastAPI decorators.
  - `requests.get/post/...` and `requests.request("METHOD", "url")`.

- **PHP (`php.ts`)**
  - Laravel `Route::get/post/.../patch('/path', ...)` via
    `scoped_call_expression`. Uses `PHP.php_only` to match the
    existing ingestion pipeline's grammar selection.

- **Node (`node.ts`) — JS + TS + TSX**
  - Pattern sources defined once, compiled against three grammar
    variants (`JavaScript`, `TypeScript.typescript`, `TypeScript.tsx`)
    because `Parser.Query` objects are not portable across grammars.
    Exports three plugins sharing the same `scan` logic.
  - NestJS: `@Controller('prefix')` decorators are siblings of the
    class in `export_statement` / `program`; `@Get(':id')` decorators
    are siblings of the method in `class_body`. The plugin walks
    decorator → next named sibling to find the decorated class /
    method, then combines the class prefix with the method path.
    Only emits NestJS detections when the enclosing class has a real
    `@Controller` decorator — prevents false positives from generic
    classes that happen to use `@Get` from another library.
  - Express: `(router|app).<verb>('/path', ...)`.
  - `fetch(url)` (default GET) + `fetch(url, { method: 'X' })`
    (uses two queries + a SyntaxNode-id dedupe set so URL literals
    aren't double-emitted by the options variant).
  - `axios.get/post/...`.

## Orchestrator changes

`http-route-extractor.ts` drops every `scanXxxProviders` / `scanXxxConsumers`
regex method and replaces them with a single source-scan loop that
delegates to `getPluginForFile(rel).scan(tree)`. The orchestrator
still owns:

- **Path normalization** (`normalizeHttpPath`, `normalizeConsumerPath`)
  — language-agnostic string processing shared by both strategies.
- **Graph-assisted Strategy A** (`HANDLES_ROUTE` / `FETCHES` / `CONTAINS`
  Cypher queries) — unchanged in spirit. The only regex helpers it
  used (`inferMethodFromFileScan`, `pickJavaHandlerName`) are now
  replaced by a lookup against the plugin's detections for the same
  file: for each route row, find the detection whose normalized path
  matches, and pull the HTTP method + handler name from it.
- **Per-file parse cache** — the orchestrator parses each relevant
  file at most once per `extract()` call. Both the graph-assisted
  enrichment loop and the source-scan fallback share the same
  `cachedDetections` map, so we never run the plugin twice for the
  same file.

## Why this is better than the regex version

1. **Comments and strings for free.** The old regex would match
   `// router.get('/fake')` as a real Express route; tree-sitter
   never visits string/comment nodes.
2. **Structural controller-prefix.** Spring and NestJS class-prefix
   joining is now scoped to the enclosing class via `class_declaration`
   node ids, eliminating file-wide state that broke when a file had
   multiple controllers.
3. **Precise NestJS disambiguation.** The plugin only emits a NestJS
   detection when the enclosing class has a real `@Controller`
   decorator — the old regex would fire on any `@Get(...)` in the
   file regardless of surrounding context.
4. **Language-agnostic extension.** Adding Ruby / Rust / Kotlin HTTP
   detection later means dropping one file in `http-patterns/` — no
   changes to the shared scanner, the orchestrator, or the Strategy A
   Cypher queries.

## Tests

- `http-route-extractor.test.ts` — **18/18 pass** (tests unchanged;
  they're contract-style input/output tests and the contract shape is
  unchanged). Covers Spring class prefix, Express, gin/echo, stdlib
  HandleFunc, NestJS, Laravel, FastAPI for providers and
  fetch/axios/python-requests/rest-template/webClient/okhttp/go-stdlib/
  resty for consumers, plus graph-first Strategy A for both.
- `topic-extractor.test.ts` — **30/30 pass** after the `captures.value`
  API migration.
- `grpc-extractor.test.ts` — 43/43 pass (untouched; phase 3).
- `manifest-extractor.test.ts` — 8/8 pass (untouched).
- `service.test.ts`, `sync.test.ts`, `storage.test.ts` — 41/41 pass.
- `npx tsc -p tsconfig.json --noEmit` clean.

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched.
- No changes to pipeline.ts, MCP surface, ingestion, or tests.
- No CI / release / security / secrets changes.
- Tree-sitter grammars imported by plugins (`tree-sitter-java`,
  `tree-sitter-go`, `tree-sitter-python`, `tree-sitter-php`,
  `tree-sitter-javascript`, `tree-sitter-typescript`) are all already
  in `package.json` for the existing ingestion pipeline.

## Phase 3 plan

- **grpc-extractor** gets the same treatment: plugin-per-language under
  `grpc-patterns/` for Java / Go / Python / TS detection. `.proto`
  files remain an open question — no `tree-sitter-proto` grammar is
  installed, so the in-tree string-sanitizing parser from PR #796's
  self-review stays as a pragmatic exception unless the maintainer
  wants us to add `tree-sitter-proto` as a new dep.

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate grpc-extractor source scans to tree-sitter plugins

Phase 3 (final) of the extractor refactor requested by @magyargergo on
#796. Same architecture as phase 1 (topic) and phase 2 (http): thin
language-agnostic orchestrator + per-language plugins that own
tree-sitter grammars and query sources. With this commit the top-level
extractors under `src/core/group/extractors/` import ZERO tree-sitter
grammars and ZERO query strings — every grammar import lives in a
`*-patterns/<lang>.ts` plugin file, and the orchestrators go through
the registry indirection.

## Architecture

```
src/core/group/extractors/
├── tree-sitter-scanner.ts         # shared primitives (unchanged)
├── grpc-extractor.ts               # orchestrator (only `.proto` parser left)
└── grpc-patterns/
    ├── types.ts                    # GrpcDetection, GrpcLanguagePlugin, GrpcRole
    ├── index.ts                    # registry: ext → plugin + GRPC_SCAN_GLOB
    ├── go.ts                       # tree-sitter-go: RegisterXxxServer, Unimplemented, NewXxxClient
    ├── java.ts                     # tree-sitter-java: @GrpcService + XxxImplBase + newBlockingStub
    ├── python.ts                   # tree-sitter-python: add_XxxServicer_to_server + XxxStub
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript:
                                    #   @GrpcMethod, @GrpcClient field type,
                                    #   .getService<X>('Svc'), new XxxServiceClient,
                                    #   loadPackageDefinition dynamic constructors
```

## Per-language coverage

**Go (`go.ts`)**
- Provider: `\w+.RegisterXxxServer(...)` via `call_expression →
  selector_expression → field_identifier` + JS regex filter
  `^Register(\w+)Server$`.
- Provider: `pb.UnimplementedXxxServer` embedded in a struct via
  `struct_type → field_declaration_list → field_declaration →
  qualified_type → type_identifier` + JS filter.
- Consumer: `\w+.NewXxxClient(...)` via the same call_expression
  query + JS filter `^New(\w+)Client$`.

**Java (`java.ts`)**
- Provider: `class X extends YyyGrpc.YyyImplBase` — two queries
  handle the scoped and plain forms. `scoped_type_identifier`'s
  children are positional (no `scope:`/`name:` fields), so the
  query matches the two `type_identifier` children by position.
- `#match? @inner "ImplBase$"` restricts matches at query time.
- Whether the class has `@GrpcService` or not controls only the
  `source` metadata label — the plugin walks the class_declaration's
  `modifiers` child in JS to detect the marker_annotation.
- Consumer: `YyyGrpc.newStub(ch)` / `newBlockingStub(ch)` via a
  `method_invocation` query with `#match? @method
  "^new(Blocking)?Stub$"`, service name extracted via
  `^(\w+)Grpc$` on the object identifier.

**Python (`python.ts`)**
- Single call-expression query covers both bare identifier and
  `obj.method` attribute forms:
  `(call function: [(identifier) @fn (attribute attribute: (identifier) @fn)])`.
- Plugin filters `@fn.text` against two JS regexes:
  `^add_(\w+)Servicer_to_server$` (provider) and `^(\w+)Stub$`
  (consumer), with a reserved-names ignore list for the Stub case
  (Mock / Test / Fake / Stub).

**Node — JavaScript + TypeScript + TSX (`node.ts`)**
- Pattern sources defined once, compiled three times (one per grammar)
  because `Parser.Query` objects are not portable across grammars.
  Exports three `GrpcLanguagePlugin`s sharing the same `scan`.
- `@GrpcMethod('Service', 'Method')`: decorator query captures the
  two string literals. Confidence is hard-coded 0.8 regardless of
  proto map resolution (matches the original regex version's
  behaviour).
- `@GrpcClient(...) field: XxxServiceClient`: decorator query
  captures the decorator node, plugin walks up to find the enclosing
  `public_field_definition` (decorators on fields are CHILDREN of
  the field definition in tree-sitter-typescript, not siblings) and
  reads its first `type_annotation → type_identifier`, then runs the
  `^(\w+Service)Client$` JS filter.
- `client.getService<X>('AuthService')`: call-expression query on
  `member_expression.property = "getService"` + string literal arg.
- `new XxxServiceClient(...)`: `new_expression` with a bare
  identifier constructor, filtered by `^(\w+Service)Client$` so
  generic `new AuthClient(...)` (missing the `Service` infix) does
  NOT falsely register as a consumer. Preserves the regression test
  `test_extract_ts_non_service_client_constructor_is_ignored`.
- `loadPackageDefinition` dynamic loader: gated on
  `tree.rootNode.text.includes('loadPackageDefinition')`. When set,
  `new foo.bar.Xxx(...)` qualified constructors with a capitalised
  property name register as consumers.

## Orchestrator changes

`grpc-extractor.ts` loses every `scanGoProviders` / `scanJavaProviders`
/ ... helper and replaces them with a single source-scan loop that:

1. Parses each file with the plugin's grammar (one shared `Parser`
   instance across all files, `setLanguage` called per plugin).
2. Calls `plugin.scan(tree)` to get `GrpcDetection[]`.
3. Converts each detection to an `ExtractedContract` via the private
   `detectionToContract` helper, which:
   - Looks the short service name up in the proto map (filled by
     the `.proto` parser).
   - Picks confidence = `confidenceWithProto` if resolved, else
     `confidenceWithoutProto`.
   - Builds a method-level contract id (`grpc::pkg.Svc/Method`) when
     the detection carries a `methodName` (TS `@GrpcMethod` only),
     otherwise a service-level id (`grpc::pkg.Svc/*`).

Everything else — the `.proto` parser, `buildProtoContext`,
`buildProtoMap`, `resolveProtoConflict`, `serviceContractId`,
`stripProtoCommentsAndStrings`, `extractServiceBlocks`, the dedupe
function — stays exactly as before. The `.proto` parser is kept as a
pragmatic exception to the "no regex in extractors" rule because no
`tree-sitter-proto` grammar is installed in the repo; a comment at the
top of the file explains this and flags the maintainer option of
adding `tree-sitter-proto` as a dependency.

## Why this is better than the regex version

1. **Comments and strings are respected for free.** Matched node types
   are only code constructs, never text inside comments or string
   literals.
2. **No false positives on partial names.** The old `(\w+?)Grpc`-style
   regexes would cross-match unrelated identifiers; structural queries
   restrict matches to the exact AST shape (`scoped_type_identifier →
   type_identifier` pairs, `method_invocation → identifier` etc.).
3. **NestJS `@GrpcClient` is structural, not regex-based.** The old
   regex required a specific textual layout
   (`@GrpcClient(...) private readonly foo!: XxxServiceClient`); the
   plugin now walks the AST, so modifier order / optional modifiers /
   multi-line formatting don't break it.
4. **Language-agnostic extension.** Adding Kotlin / Rust / C# gRPC
   detection later is a one-file edit in `grpc-patterns/index.ts` —
   no touches to the shared scanner, the orchestrator, or the proto
   parser.

## Tests

- `grpc-extractor.test.ts` — **43/43 pass** (tests unchanged; the
  contract shape is identical). Covers .proto parsing (including the
  brace-inside-string regression), Go provider/consumer,
  Java @GrpcService / plain ImplBase provider + newBlockingStub
  consumer, Python servicer + stub, TS @GrpcMethod + @GrpcClient +
  .getService + new XxxServiceClient + loadPackageDefinition + the
  `AuthClient` vs `AuthServiceClient` discrimination, dedupe across
  multiple patterns in one file, proto-aware confidence, and the
  inherited-package resolution for split proto definitions.
- `topic-extractor.test.ts` — 30/30 pass.
- `http-route-extractor.test.ts` — 18/18 pass.
- `manifest-extractor.test.ts` — 8/8 pass.
- `service.test.ts`, `sync.test.ts`, `storage.test.ts` — 41/41 pass.
- `npx tsc -p tsconfig.json --noEmit` clean.

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched.
- No pipeline.ts, MCP surface, ingestion, CI / release / security, or
  test changes.
- New tree-sitter grammar imports (`tree-sitter-go`, `tree-sitter-java`,
  `tree-sitter-python`, `tree-sitter-javascript`, `tree-sitter-typescript`)
  are all already installed for the ingestion pipeline.

## End of phase series

This commit completes the three-phase extractor refactor:
  - **Phase 1** (`ea06d11`): topic-extractor → `topic-patterns/`
  - **Phase 2** (`b6015f6`): http-route-extractor → `http-patterns/`
  - **Phase 3** (this commit): grpc-extractor → `grpc-patterns/`

Every remaining regex-based extractor helper under the `src/core/group/
extractors/` directory is either (a) language-agnostic string
processing (path normalization, dedupe keys) or (b) the `.proto`
parser, which is documented as an explicit exception.

Co-authored-by: Claude <noreply@anthropic.com>

* feat(group): add tree-sitter-proto for .proto file parsing

Addresses @magyargergo's suggestion on #796 to replace the manual
string-sanitizing .proto parser with a tree-sitter grammar.

- **Vendored `tree-sitter-proto`** in `vendor/tree-sitter-proto/`.
  Grammar source from [coder3101/tree-sitter-proto](https://github.com/coder3101/tree-sitter-proto)
  (latest `grammar.js`), parser.c regenerated with `tree-sitter-cli
  0.24` to produce ABI version 14 — compatible with the project's
  `tree-sitter 0.25` runtime (which supports ABI ≤ 14). Added as
  `optionalDependency` with `file:./vendor/tree-sitter-proto`.

- **New `grpc-patterns/proto.ts` plugin** — uses the same
  `compilePatterns` + `runCompiledPatterns` infrastructure as every
  other plugin. Two queries:
  - `(package (full_ident) @pkg)` — package declaration
  - `(service (service_name) @service_name (rpc (rpc_name) @rpc_name))`
    — one match per (service, rpc) pair

- **Graceful fallback** — `tree-sitter-proto` is an optional
  dependency. If it fails to install (platform incompatibility) or
  fails the runtime smoke-test (`setLanguage` + `parse` on a trivial
  proto), `PROTO_GRPC_PLUGIN` stays `null` and the orchestrator
  uses the existing manual parser. The smoke-test catches the
  `SyntaxNode` TDZ error that occurs in vitest's fork-based test
  runner.

- **Orchestrator updated** — when `hasProtoPlugin` is true, `.proto`
  files are handled by the plugin loop (they're included in
  `GRPC_SCAN_GLOB`), and the manual `parseProtoFile` loop is
  skipped. `buildProtoContext` still runs to build the proto map
  for cross-referencing source-file detections.

1. **No manual comment/string stripping.** The old parser needed
   `stripProtoCommentsAndStrings` (110 lines) to avoid counting
   braces inside comments and string literals. tree-sitter handles
   this natively.
2. **No brace-depth tracking.** `extractServiceBlocks` used a manual
   depth counter to find service boundaries. tree-sitter's AST gives
   us `service` → `service_name` + `rpc` → `rpc_name` directly.
3. **Performance.** tree-sitter's C-based parser is faster than
   character-by-character JS scanning + regex on large proto files.

- `grpc-extractor.test.ts` — **43/43 pass** (unchanged)
- All other extractor tests — 99/99 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* chore: add .gitignore for vendored tree-sitter-proto build artifacts

https://claude.ai/code/session_01SFUCxgKMMQ8EgRHYw91xPU

* fix: correct .gitignore paths for vendored tree-sitter-proto

Patterns should be relative to the .gitignore file's directory.

https://claude.ai/code/session_01SFUCxgKMMQ8EgRHYw91xPU

* refactor(group): address Copilot review feedback on #796

Six fixes suggested by the Copilot AI review:

1. **`normalizeHttpPath` root-path edge case** — stripping trailing
   slashes on the input `/` produced an empty string, yielding
   malformed contract ids like `http::GET::`. Now preserves `/` for
   the root handler/fetch case.

2. **Dedupe `scanFiles` call** — `extract()` was globbing the
   source-scan file list twice (once for the provider fallback, once
   for the consumer fallback). Moved to a single lazy call that
   memoizes the result for the rest of the method.

3. **HTTP `scanFiles` now ignores `**/vendor/**`** — every other
   extractor's glob already ignored vendored sources; the HTTP one
   didn't. Fixed for consistency.

4. **`loadPackageDefinition` check is now structural** — was calling
   `tree.rootNode.text.includes('loadPackageDefinition')` which forces
   materialization of the entire file text from the parse tree
   (expensive on large files). Replaced with a dedicated compiled
   query on `(call_expression function: [(identifier) | (member_expression)])`
   so the check stays in the AST domain.

5. **`grpc-extractor.ts` header docstring updated** — still claimed
   ".proto parsing is not tree-sitter-based because no grammar is
   installed". Now describes the actual behaviour: tree-sitter when
   `tree-sitter-proto` is available (optionalDependency), manual
   fallback otherwise.

6. **Eliminated the double proto file parse on the fallback path** —
   `buildProtoContext` already globs + parses every `.proto` file to
   build `servicesByName`. On the `!hasProtoPlugin` branch the
   extractor was globbing + parsing again via the now-removed
   `parseProtoFile` helper. The fallback branch now iterates the map
   that `buildProtoContext` already produced to emit provider
   contracts directly — single pass per proto file.

## Tests

- `topic-extractor.test.ts` — 30/30 pass
- `http-route-extractor.test.ts` — 18/18 pass
- `grpc-extractor.test.ts` — 43/43 pass
- `manifest-extractor.test.ts` — 8/8 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): address Claude review feedback (bugs + dedup + hygiene) on #796

Follows up `2f28bfc` with the remaining items from the Claude AI review:

## Bugs

**Bug 2 — Label-unaware Cypher queries in `resolveSymbol`.**
The manifest-extractor's lookup queries were `MATCH (n) WHERE n.name = $x`
with no label filter, so a topic/service/package name could silently match
any node type (File, Variable, Import, Folder, …). Added label filters:
- `topic` → `(n:Function|Method|Class|Interface)` (topics are best-effort
  symbol-name matches against listener/publisher symbols)
- `grpc` method → `(n:Function|Method)`
- `grpc` service → `(n:Class|Interface)`
- `lib` → `(n:Package|Module)`

All 8 manifest-extractor tests still pass (mock executor is
label-agnostic, but the production LadybugDB graph now gets correctly
scoped queries).

**Bug 8 — Tautological `!handlerName` condition.**
`http-route-extractor.ts:extractProvidersGraph` had
`let handlerName = null; if (!method || !handlerName) { ... }` — the
`!handlerName` clause was always true since there was no intervening
assignment. Simplified to always run the plugin-scan lookup (we need
the handler name even when `methodFromRouteReason` already resolved
the method).

## Clean code / dedup

**Design 7 — `readSafe` was copy-pasted in all three orchestrators.**
Extracted to `extractors/fs-utils.ts` as the single source of truth
for the path-traversal guard. Dropped the three local copies and the
now-unused `fs`/`path` imports from topic-extractor.

**Style 10 — Language-specific `_test.go` skip in the topic orchestrator.**
Was `if (rel.endsWith('_test.go')) continue;` inside the language-
agnostic extraction loop. Pushed into the glob's ignore list
(`'**/*_test.go'`) alongside the existing `node_modules`, `vendor`,
`dist`, `build` entries, with a comment explaining that other
languages' test file conventions either live in separate directories
(Python `tests/`, Java `src/test/`) or are already covered by the
existing ignores.

## Already addressed in `2f28bfc` (mentioned again in Claude review)

- Bug 3: `normalizeHttpPath('/')` returns `''` — fixed
- Bug 4: double glob + double parse of `.proto` — fixed
- Bug 5: `scanFiles` called twice in HTTP — fixed
- Bug 6: missing `**/vendor/**` in HTTP glob — fixed
- Design 9 partially: `tree.rootNode.text.includes('loadPackageDefinition')`
  replaced with a dedicated structural query

## Deferred

- Bug 1 (`http::*::path` vs `http::GET::path` matching) — out of scope;
  sync.ts matching logic lands in #793, manifest extractor already
  emits correct synthetic uids for unresolved HTTP contracts.
- Design 9 full (change plugin `scan(tree)` → `scan(tree, source)`) —
  the only real use case (`loadPackageDefinition` gate) is already
  fixed via a structural query, so the interface change would be
  cosmetic churn without a concrete consumer.

## Tests

- `topic-extractor.test.ts` — 30/30 pass
- `http-route-extractor.test.ts` — 18/18 pass
- `grpc-extractor.test.ts` — 43/43 pass
- `manifest-extractor.test.ts` — 8/8 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* docs+fix(group): address remaining Claude review items + add pipeline flow chart

## Fixes

**Remaining 🔴 — HTTP contract id wildcard format.** Documented the
`http::*::<path>` format as an intentional wildcard for manifest links
that omit the HTTP method, alongside the explicit-method form
(`GET::/path` → `http::GET::/path`). The docblock on `buildContractId`
now states both forms, notes that wildcard-aware matching is the
responsibility of the sync / cross-impact layer (#793), and
recommends the explicit-method form whenever the author knows the
method (it round-trips through exact equality without needing
wildcard logic downstream). Tests unchanged — the wildcard format is
what they've always asserted.

**Minor 1 — stale comment at `manifest-extractor.ts:124-126`.** The
comment claimed "creates a contract with an empty symbolUid/ref" but
the code switched to `manifestSymbolUid(repo, contractId)` a few
commits back. Updated to describe the actual synthetic-uid fallback
semantics and the cross-impact path that relies on both sides of the
join deriving the same uid.

**Minor 2 — exhaustiveness guard on `buildContractId`.** The
`switch(type)` covered all five current `ContractType` variants but
silently returned `undefined` if a new variant was added. Added a
`default: const _exhaustive: never = type; throw new Error(...)`
clause so the build fails loudly on an unhandled variant.

**Minor 3 — `tree.rootNode.text` in `grpc-patterns/node.ts`.** Already
fixed in `2f28bfc` via a dedicated structural query
(`LOAD_PACKAGE_DEFINITION_SPEC`). No action needed.

## New: pipeline flow chart (per @magyargergo's request)

Added `src/core/group/PIPELINE.md` with four Mermaid diagrams:
1. **High-level overview** — `group.yaml` → extractors + manifest →
   contract matching → `bridge.lbug` → `runGroupImpact`.
2. **Per-repo extractor two-strategy shape** — graph-assisted
   Strategy A vs. source-scan Strategy B.
3. **Plugin architecture** — orchestrator → registry →
   per-language `*-patterns/<lang>.ts` → `tree-sitter-scanner.ts` →
   `ExtractedContract`.
4. **Manifest extraction** — label-scoped `resolveSymbol` with the
   synthetic-uid fallback.
5. **Cross-impact query (#606)** — local impact → bridge join →
   cross-repo fan-out.

Each diagram is annotated with which PRs own which stage (this PR:
extractors + manifest; #795: bridge storage; #606: cross-impact
runtime) and points at the concrete files/functions involved.

## Tests

- 99/99 extractor tests pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-13 08:49:30 +01:00
abhigyanpatwari
1735c0ffcd gitnexus wal cleanup preventing reindexing issue fixed 2026-02-06 06:35:15 +05:30