21 commits

Author	SHA1	Message	Date
azizur100389	f236be05e0	feat: gate Icebug community engine prototype (#2376)	2026-07-09 05:43:49 +01:00
Gergő Magyar	8402963198	fix(ci): shard platform-sensitive matrix + spawn built CLI to fix Windows cross-platform timeout (#2394) ... * fix(ci): shard platform-sensitive matrix + spawn built CLI to fix Windows cross-platform timeout The `windows-latest (platform-sensitive)` job was hitting its 15-min internal vitest watchdog in run-cross-platform.ts. It's cumulative slowness, not a hang: the fixed 72-file suite is dominated by ~50 CLI/worker process spawns, and Windows is ~5x slower than macOS at process startup (macOS ran the same set in ~3min of tests). Two complementary changes bring it back under the watchdog with headroom, without touching any test assertion: - Shard the platform-sensitive matrix (windows/macos × shard [1,2]) and forward `--shard=i/2` through run-cross-platform.ts to vitest, which partitions the fixed file list deterministically (sha1, equal file-count) — halving each runner. macOS/Ubuntu were already under budget. - New test/helpers/cli-entry.ts (`CLI_SPAWN_PREFIX`): spawn the built `dist/cli/index.js` when `GITNEXUS_E2E_CLI=dist` (set on the cross-platform job, which already builds) instead of `node --import tsx src/cli/index.ts`, which re-transpiles the whole CLI on every spawn. Defaults to tsx-on-source so local runs always reflect current source; `GITNEXUS_E2E_CLI=dist` on an unbuilt tree throws an actionable "run npm run build" error. dist is opt-in only — never inferred from a generic `CI` env — so an ambient `CI=1` can't silently run a stale build. Converted 8 spawn-based e2e suites; added test/unit/cli-entry.test.ts. The Ubuntu coverage job leaves `GITNEXUS_E2E_CLI` unset, so the tsx-on-source path stays exercised in CI too (both entry points covered). Measured on Linux: cli-limit-e2e 121.5s→91s, cli-e2e 289s→217s (~25%); larger on Windows where the transpile is a bigger share of each spawn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ci): derive platform-sensitive shard count from one source (#2394) The shard total was hardcoded in three coupled, unenforced places (matrix length, job-name suffix, --shard denominator); editing one without the others silently dropped a shard's tests with green CI. Add a checkout-free shard-plan job whose single TOTAL generates both the shard index list (consumed via fromJSON) and the /N denominator (job name + --shard arg), so they cannot drift. Asserts TOTAL>=1 to rule out an empty-matrix silent skip. No behavior change — still 2 shards per OS. Addresses PR #2394 tri-review finding F2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): 3 shards for real Windows headroom + honest sharding comments (#2394) vitest shards by file COUNT, not runtime, so the heaviest spawn suites cluster into one shard: live CI showed Windows shard 1/2 at 12m12s (~81% of the 15-min watchdog) vs shard 2/2 at 3m0s. The old comments claimed "comfortable/generous headroom", which the count-based split doesn't deliver at 2 shards. Bump TOTAL to 3 (one line, single source) so even the busiest Windows shard clears the watchdog, and reword the comments to describe count-based (not time-based) sharding. Addresses PR #2394 tri-review finding F1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): extract testable parseShardArg from run-cross-platform (#2394) The --shard parse/forward glue had no unit test. Extract it into a pure scripts/shard-arg.ts (mirroring the computeSpawnPrefix extraction precedent) so the branch logic is lockable without the script's top-level execFileSync, and add test/unit/shard-arg.test.ts (absent -> undefined, valid token -> passed through, found amid other args). Behavior unchanged; U4 adds the malformed fail-loud on top. Addresses PR #2394 tri-review finding F3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): fail loud on a malformed --shard arg (#2394) A shard-shaped-but-malformed arg (--shard=1, --shard, --shard=abc) was silently ignored, dropping the shard flag so both legs ran the full unsharded ~50-spawn suite — re-arming the Windows watchdog timeout with no signal. parseShardArg now throws an actionable error on any --shard/--shard=… arg that fails the strict regex (unrelated flags like --shardx= pass through), and the call site in run-cross-platform.ts catches it into console.error + exit 1, kept outside the execFileSync try so the message isn't swallowed by that catch's watchdog-only branch. Addresses PR #2394 tri-review finding F4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): fail loud on an unknown GITNEXUS_E2E_CLI value (#2394) computeSpawnPrefix silently degraded any unknown GITNEXUS_E2E_CLI value to tsx-on-source, so a typo (e.g. `dsit`) would make CI believe it tests the dist entry point while actually running src. Throw on any value other than 'dist'/'src'/unset (the safe tsx default is preserved for unset/''/'src', so it still never selects dist without an explicit opt-in). Flip the unknown-mode unit test to assert the throw and add the missing {mode:undefined, distExists:true} case. Only ci-tests.yml sets the var (=dist), so no existing suite is affected. Addresses PR #2394 tri-review findings minor-a/b. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): run cli-entry.test.ts on the cross-platform matrix (#2394) cli-entry.test.ts resolves CLI_SPAWN_PREFIX from a real path, and its last assertion (cli[/\\]index) has a Windows backslash branch that only Ubuntu exercised. Register it in PLATFORM_LOGIC so it runs on the Windows/macOS matrix too. (shard-arg.test.ts stays out — pure string logic, OS-independent.) List grows 73 -> 74; the generated shard matrix keeps coverage complete. Addresses PR #2394 tri-review finding minor-c. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(test): share tsxLoaderUrl(), dedup the last tsx-loader boilerplate (#2394) bridge-cache-reopen.test.ts carried its own copy of the tsx-loader-resolution boilerplate (createRequire -> resolve('tsx/package.json') -> pathToFileURL) — the one site the PR's CLI_SPAWN_PREFIX migration didn't cover (it spawns a seed script, not the CLI). Export the existing tsxLoaderUrl() from cli-entry.ts and reuse it here; the resolved loader URL is byte-identical. Addresses PR #2394 tri-review finding minor-d. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): make skipUnlessFtsAvailable install FTS on miss so shards are self-sufficient (#2394) Sharding the platform-sensitive suite into 3 exposed a latent test-isolation bug: load-only FTS primitives (test/integration/lbug-core-adapter.test.ts) only passed because a sibling installer test happened to co-locate in the same shard and install FTS into the shared ~/.lbdb first. At 3 shards, lbug-core-adapter landed in a shard with no installer sibling, so its load-only loadFTSExtension() failed deterministically on macOS+Windows shard 2/3 under GITNEXUS_REQUIRE_FTS=1. Make the gate self-sufficient: on a load-only miss under REQUIRE_FTS, install FTS with `auto` (LOAD-first, then one bounded network INSTALL) before treating it as a hard failure — mirroring withTestIndexedDB. A pre-installed extension still costs no network (auto is LOAD-first); offline/local runs (no env var) still skip gracefully. Verified: with a fresh HOME (no pre-installed FTS) + REQUIRE_FTS=1, lbug-core-adapter now passes 15/15 (previously threw). Addresses the 3-shard CI failure surfaced while validating PR #2394's F1 fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: warm-cache the LadybugDB FTS extension across platform shards (#2394) Follow-up to the FTS self-install fix: cache ~/.lbdb/extension per OS + lockfile so a warm run skips the network install entirely and the parallel shards share one download across runs. Pure reliability/speed — on a cache miss the tests still self-install FTS on demand (test/helpers/fts-availability.ts), so this is never a correctness dependency, just a way to cut the network-install surface that made the sharded FTS tests flaky. Keyed by lockfile hash (a LadybugDB version bump re-installs); per-OS since the extension is a native binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): pass shard via env to clear zizmor template-injection (#2394) Interpolating ${{ matrix.shard }} (now sourced from the shard-plan job output) directly into the run: shell tripped zizmor's template-injection audit (code-scanning alert #824, ci-tests.yml:147). Move the value into a SHARD env var — assigned via ${{ }} but referenced as "$SHARD" in the shell, which is not an injection sink — and set shell: bash so the expansion is uniform across the windows + macOS matrix (the default run shell is pwsh on Windows, where $SHARD would be empty and trip the new malformed-shard fail-loud). Verified locally with zizmor: the :147 template-injection finding is gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: shard the ubuntu coverage job and merge blobs before the threshold gate (#2394) The coverage job ran the full suite unsharded (~16 min). Shard it like the cross-platform matrix, then merge the per-shard coverage before enforcing the threshold gate: - shard-plan now also single-sources the coverage shard count (cov_total / cov_shards), so the coverage matrix + /N denominator can't drift. - The `tests` job becomes a coverage shard matrix: each shard runs `vitest run --shard --coverage --reporter=blob` with thresholds forced to 0 (a single shard's partial coverage can never meet the gate) and uploads its blob. FTS self-installs per shard, so sharding the full suite is safe. - New `coverage-merge` job (needs: tests) reduces the blobs with `vitest --mergeReports`, enforcing the REAL config thresholds on the combined ('new') coverage — this is the gate. It also emits the merged test-results.json and runs the unsharded web + docker suites, so the `test-reports` artifact keeps the exact shape ci-report.yml consumes for its base-branch ('baseline') vs new coverage delta. The shard arg goes through a SHARD env var + shell: bash (no template-injection). Validated locally: shard blobs write and merge into a coverage-summary.json + merged test-results.json; the merge enforces thresholds on the union. CI Gate still aggregates the coverage-merge result via the reusable-workflow call. Note: the coverage check names change (ubuntu / coverage 1/3 … + merge) — update any pinned branch-protection required checks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): include hidden files when uploading the coverage blob (#2394) The coverage shards write their blob to gitnexus/.vitest-reports/ (a dotdir). actions/upload-artifact excludes hidden files by default, so the coverage-blob-* artifacts uploaded empty — the merge job then downloaded 0 artifacts and vitest --mergeReports failed with ENOENT scandir '.vitest-reports'. Set include-hidden-files: true on the blob upload so the blobs actually ship. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): group shard-plan GITHUB_OUTPUT writes to satisfy shellcheck SC2129 (#2394) Adding the coverage shard outputs (cov_shards/cov_total) made the shard-plan gen step write four individual `>> "$GITHUB_OUTPUT"` redirects, which shellcheck (run by the actionlint check) flags as SC2129. Group the echoes into a single `{ …; } >> "$GITHUB_OUTPUT"` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(test): cost-balanced shard sequencer to cut CPU contention (#2394) vitest's default --shard hashes file paths and splits by file COUNT, which clustered the spawn-heavy suites onto one runner (Windows platform shard 1 ran ~4x the others). Add a custom sequence.sequencer that overrides only shard() and balances by estimated WORK instead: - specWeight() weights the fileParallelism:false spawn-heavy suites (cli-e2e, lbug-db — already isolated to run sequentially) far above the parallel default files, plus file size as a cheap finer signal. Deterministic per checkout. - assignShards() does greedy longest-processing-time bin-packing (heaviest file into the currently-lightest shard). The partition stays complete and disjoint — verified: on the 74-file cross-platform set the three shards weigh 7611/7610/8064 (the sequential-heavy files spread ~7/7/8) with zero overlap and no file dropped, vs the hash split's count-only balance. sort() is left to the base sequencer so project groupOrder / duration-cache ordering is untouched. Pure logic split into shard-balance.ts with a unit test locking the disjoint+complete, balance, and determinism properties. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): install + cache FTS up front on the coverage (and cross-platform) shards (#2394) coverage 3/3 failed on extension-binary-real.test.ts: it uses the file-path FTS gate (requireFtsResourceOrSkip), which resolves ~/.lbdb/extension at MODULE LOAD and cannot self-install the way the load-path gate (skipUnlessFtsAvailable, U8) does. The coverage job had no FTS cache and relied on an installer test running first in the shard — the balancing sequencer reshuffled the shards and dropped extension-binary-real into a shard with no installer, so FTS was absent. Remove the ordering dependency: add scripts/ensure-fts.ts (init a throwaway lbug db, loadFTSExtension with policy:auto → LOAD-first, INSTALL on miss) and run it up front on every coverage AND cross-platform shard, after restoring the per-OS FTS cache. The coverage job now shares that same cache key (it previously had none — this is the "share the cached FTS with coverage" the failure pointed at). Cold cache installs once; warm cache is a no-network load. Verified locally: ensure-fts installs FTS into a fresh HOME and is a no-op when already present. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-08 09:09:11 +01:00
Gergő Magyar	cdad478c96	fix: proxy-blocked installs survive onnxruntime-node postinstall and self-heal embeddings (#2370) (#2372)	2026-07-05 16:15:10 +01:00
Gergő Magyar	576e81442e	fix(search): index description field for FTS so doc comments are keyword-searchable (#2300) ... * fix(search): index description column for FTS so doc comments are keyword-searchable Closes #2299. descriptionExtractor (#2286) populates the `description` column for every symbol table, but FTS only indexed name+content on 5 tables, so doc-comment keywords (Javadoc/KDoc/godoc/Rust ///) were invisible to BM25 keyword search. - Add `description` to the Function/Class/Method/Interface FTS indexes (File has no description column, left as name+content). - Add FTS indexes for the remaining EMBEDDABLE_LABELS symbol tables (Struct, Enum, Trait, Impl, Macro, Namespace, Constructor, TypeAlias, Typedef, Const, Property, Record, Union, Static, Variable). - createSearchFTSIndexes now drops-then-creates each index so the schema change reaches existing DBs on incremental re-analyze and --repair-fts (createFTSIndex is idempotent-by-name and would otherwise skip stale indexes). Tests: fts-schema column-subset + coverage guards; drop-before-create order; e2e doc-comment keyword search (Java class + Rust struct found by description-only terms). bm25-search assertions derive from FTS_INDEXES. * fix(review): apply autofix feedback - Guard the --repair-fts path on FTS-extension availability before createSearchFTSIndexes drops-then-creates indexes (P1 regression: without the gate, an unavailable extension could drop existing indexes then fail to recreate them, leaving the DB index-less). Mirrors the analyze path's ftsAvailable gate and fails loudly first. - Add a re-analyze upgrade integration test: seed an old name+content-only DB (no Struct index), run the real createSearchFTSIndexes(), and assert description keyword search + the previously un-indexed Struct now resolve. Proves drop-then-create upgrades a live stale index end-to-end. * fix(ci): add loadFTSExtension to --repair-fts test mocks The R3 review fix added a loadFTSExtension availability gate to the --repair-fts path, but run-analyze-fts-repair.test.ts mocked the lbug adapter without that export, so both repair tests threw `No "loadFTSExtension" export`. Add loadFTSExtension to the two mocks (returning true to preserve their original intent) and add a dedicated test proving the guard fails loudly — and does NOT drop any index — when the extension is unavailable. * test(fts): run fts-description-search in the sequential lbug-db project It was the only FTS-index-creating integration test left in the parallel `default` vitest project; every other ftsIndexes-using test (search-core, search-pool, augmentation, …) runs in the `lbug-db` project, which forces fileParallelism: false to avoid LadybugDB native mmap file-lock conflicts in parallel forks (Windows). Add it to the lbug-db include list and the default exclude list to match the convention and remove the flake risk. * test(ci): fail loudly when FTS extension is unavailable, never silently skip FTS-dependent lbug integration suites (search-core, search-pool, augmentation, fts-description-search, …) self-skip via ctx.skip() when the LadybugDB FTS extension can't load, emitting only a console.warn while the job stays green. That means a broken/missing FTS extension in CI would make these integration tests silently vanish with no signal — false confidence. withTestLbugDB now honors GITNEXUS_REQUIRE_FTS=1: when set and the extension is unavailable, setup() throws instead of skipping, so the suite fails loudly. The CI test jobs (ubuntu coverage + windows/macOS cross-platform) set the flag; local/offline runs leave it unset and keep skipping gracefully. (Verified the extension currently loads on all three runners, so this is a guard against regression, not a behavior change today.) * test(ci): run fts-description-search on macOS/Windows cross-platform jobs The new FTS description-search suite was registered in the sequential lbug-db vitest project (ubuntu/coverage) but absent from LBUG_NATIVE, so the macOS/Windows platform-sensitive jobs (which run only the explicit ALL_CROSS_PLATFORM allowlist via run-cross-platform.ts) never executed it. The GITNEXUS_REQUIRE_FTS=1 hardening on those jobs guarded the old FTS fixtures but not the new 20-index/description path. Add the suite to LBUG_NATIVE so the new path is validated cross-platform too. Refs #2299. * fix(search): verify FTS indexes cover description, not just queryability verifySearchFTSIndexes probed each index with QUERY_FTS_INDEX and treated 'queryable' as 'present'. A stale name+content-only index left on a pre-#2299 DB stays queryable yet silently misses the description column, so verification would pass green while doc-comment search stayed broken. Switch to a single CALL SHOW_INDEXES() that exposes property_names per index, and report an index as missing when it is absent OR does not cover its configured columns. Return contract (string[] of table.indexName) is unchanged, so both run-analyze.ts call sites are untouched. The per-index string interpolation is gone, so the now-dead safeIdentifier helper is removed. The real caller of the live function in tests is bm25-search.test.ts (the repair test mocks verifySearchFTSIndexes wholesale); its two probe-shaped cases are rewritten to feed SHOW_INDEXES rows and now assert column coverage, plus an absent-index case. Refs #2299. * test(search): assert description search via the public query surface The #2299 integration suite only exercised the searchFTSFromLbug helper. Add a third block that drives the public LocalBackend.callTool('query') path — which resolves the repo via the registry and routes BM25 through the pool adapter (a different connection context than the core-adapter helper) — and asserts a description-only keyword returns the seeded class. Reuses the existing description-only SEED and production FTS_INDEXES; partial-mocks repo-manager so listRegisteredRepos points at the test DB while cleanupOldKuzuFiles and the rest stay real. Refs #2299. * test(search): make lbug-core-adapter FTS gate honor GITNEXUS_REQUIRE_FTS lbug-core-adapter.test.ts has its own per-test FTS gate (skipUnlessFtsAvailable) that called ctx.skip() whenever the extension could not load — bypassing the GITNEXUS_REQUIRE_FTS=1 hardening that withTestLbugDB already honors. Since this file is in LBUG_NATIVE it runs on the ubuntu/macOS/windows jobs that all set GITNEXUS_REQUIRE_FTS=1, so an FTS regression on a runner would have let these FTS-primitive tests silently vanish from a green run — the exact gap #2299's test-infra hardening set out to close. Make the helper mirror withTestLbugDB: when GITNEXUS_REQUIRE_FTS=1 and the extension is unavailable, throw (hard fail) instead of skipping. Offline/local runs (no env var) still skip gracefully. Refs #2299.	2026-06-25 14:21:44 +01:00
dependabot[bot]	5f667c32a3	chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 (#2292) ... * chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.3 to 7.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](df4cb1c069...9c091bb21b) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * ci: set persist-credentials: false on read-only checkouts (zizmor artipacked) Adds `persist-credentials: false` to the 9 checkout steps flagged by zizmor's credential-persistence (artipacked) rule on PR #2292. All are read-only CI/test/quality jobs that never use the git token afterward, so not persisting it removes the leak surface. Checkouts that push (publish, pr-autofix, commit-fork-prebuilds, etc.) keep credentials and are untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gergo Magyar <gergomagyar@icloud.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 06:53:47 +01:00
Gergő Magyar	3c82361b66	perf(cfg): streaming/chunked PDG graph emit for full-kernel-scale repos (#2202) (#2216)	2026-06-16 05:04:10 +01:00
Gergő Magyar	df08ecc397	perf(lbug): cut graph-DB emit/persistence wall time (#2203) (#2215) ... * perf(lbug): add PROF_LBUG_LOAD persistence-path timing breakdown (#2203 U1) loadGraphToLbug is un-timed today; the analyze 'emit' number is the scope-resolution emit bucket, not the CSV->COPY persistence path. Add a zero-cost-when-off per-stage breakdown (csv-emit/copy-nodes/rel-split/ copy-rels/fallback/total + node/rel counts) gated by PROF_LBUG_LOAD=1, mirroring the PROF_SCOPE_RESOLUTION pattern. Document the flag in README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(lbug): route relationships to per-pair CSVs in the emit pass (#2203 U2) Relationships were written once to a monolithic relations.csv, then re-read line-by-line (regex per edge) and re-split into per-FROM->TO-label-pair files before COPY — writing and reading the entire ~1M-edge set twice. Route each edge to its pair file directly during the single emit pass via a shared RelPairRouter, eliminating the monolithic write + re-read + per-edge regex. The router applies the SAME getNodeLabel + validTables filter as the legacy splitRelCsvByLabelPair, which is retained as a differential oracle. A new differential test asserts the direct-emit per-pair files are byte-for-byte identical to the oracle's, with identical skip/total accounting. The prof line (U1) drops its rel-split stage (routing now folds into csv-emit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(lbug): skip per-row microtask tick in BufferedCSVWriter (#2203 U3) addRow awaited an already-resolved promise on every buffered row, scheduling a microtask per node even when nothing flushed (millions at scale). It now returns a promise ONLY when it flushes; the node-emit loop awaits once per iteration after the switch. Flush/drain semantics are unchanged, so backpressure on the rows that actually write is preserved and the emitted CSV bytes are byte-identical (covered by the determinism + differential tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * bench(lbug): emit throughput + byte-identity gate for the persistence path (#2203 U4) Build-free bench (bench/emit-persistence/measure.mjs) times streamAllCSVsToDisk on a synthetic graph at two scales and gates: (1) an order-independent sha256 fingerprint over every emitted CSV line — the byte-identity guard for the U2/U3 emit optimisations — and (2) a scaling-ratio budget catching an O(n^2) emit re-regression. Wired into ci-tests.yml alongside the cfg/scope-capture benches. The LadybugDB COPY half needs a real DB, so its timing stays in PROF_LBUG_LOAD + the integration round-trip tests (documented in the bench README, with the deferred COPY-parallelism follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): apply autofix feedback (#2203) - P1: router backpressure drain-await rejected with a generic AbortError, masking the real EMFILE/disk-full error. Expose RelPairRouter.lastError and rethrow it in the emit catch — mirrors the oracle's throw streamError ?? err. - P1: cover RelPairRouter error + backpressure + teardown paths with a new unit test (test/unit/rel-pair-routing.test.ts) using an injected mock stream. - P2: wrap streamAllCSVsToDisk body in try/finally so the setMaxListeners bump is always restored (the U2 rel-routing throw path could leak it). - P2: dedup WriteStreamFactory — re-export the canonical type from rel-pair-routing instead of a second identical declaration. - P2: annotate splitRelCsvByLabelPair @internal as the retained differential oracle so a future dead-code sweep doesn't delete the byte-identity guard. - P3: differential test now covers the proc_ prefix + clears GITNEXUS_SORT_GRAPH_OUTPUT to prevent env-leak desync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(lbug): scope byte-identity to quote-free ids + lock the quote-in-id divergence (#2215 review) The 'byte-identical' claim was unconditional, but the router derives labels from the raw id while the retained splitRelCsvByLabelPair oracle re-derives them via a regex over the escaped row — so for an id containing a double-quote they diverge (the router is the more-correct path). Soften the wording in rel-pair-routing.ts, the bench README, and the differential-test comment to document the exception, and add a differential test asserting the intended divergence (router routes the quote-in-id edge; oracle drops it) so a future change can't silently revert to the buggy regex semantics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * bench(lbug): per-file fingerprint so the gate catches pair-file mis-routing (#2215 review) fingerprintEmit flattened every line of every per-pair file into one array, sorted globally, and hashed — losing file boundaries, so a row routed to the WRONG pair file produced an identical fingerprint. Hash a per-file digest (filename + sha256(file bytes)) and combine the sorted entry list, so mis-routing (and within-file row reordering) now changes the fingerprint. Baseline regenerated; the new scheme yields a different hash on byte-identical emit, confirming it is sensitive to file structure the old flatten ignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * bench(lbug): add absolute large-scale wall-time backstop to the emit gate (#2215 review) The scaling-ratio gate only compares large/small, so a uniform Nx slowdown at both scales passes with ratio ~1.0. Add an opt-in max_ms_large ceiling (1000ms vs observed ~200ms — generous, host-noise-tolerant) that --check enforces alongside the ratio, catching a gross absolute regression the ratio misses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(lbug): cover the sorted-output path in the byte-identity differential (#2215 review) The differential test only exercised the default insertion-order emit path. Add a case under GITNEXUS_SORT_GRAPH_OUTPUT=1 that feeds the oracle the same id-sorted order orderedRelationships() uses and asserts per-pair byte-identity, so within-pair row reordering on the sorted path can't slip past the gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(lbug): cover the invalid-TO-label skip branch (#2215 review) Only an invalid-FROM label was exercised; the validTables skip is an OR over both endpoints, so the invalid-TO branch was untested (an inverted && would have slipped through). Add a valid-FROM/invalid-TO edge to the differential test and the router unit test, asserting it's skipped identically by router and oracle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(lbug): exercise the BufferedCSVWriter FLUSH_EVERY boundary in vitest (#2215 review) The U3 addRow change (returns a flush promise only on flush; undefined when buffered) and the loop's `if (pending) await pending` were only crossed by the bench, never vitest (all fixtures are <500 nodes). Add a 600-node graph through streamAllCSVsToDisk asserting all rows land exactly once across the 500-row flush boundary — no drops, dups, or corruption. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(lbug): drop redundant step cast in buildRelRow (#2215 review) GraphRelationship.step is already typed number?, so (rel as { step?: number }).step was a no-op structural cast that obscured the shared-type coupling. Use rel.step directly. Byte-identical — bench fingerprint unchanged, differential test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(lbug): make the unknown-label node drop explicit (#2215 review) With the U3 `let pending` switch idiom, a node whose label matches neither codeWriterMap nor multiLangWriters left `pending` undefined and was silently dropped — a footgun for a future node type. Add an explicit else with a comment documenting that unknown labels are intentionally not persisted and that a new type must be wired into a writer map. No behavior change (byte-identity + tests unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(lbug): drop the unused WriteStreamFactory re-export (#2215 review) The type was re-exported from lbug-adapter 'to preserve this module's surface,' but no external code imports it by name from here (the only test reference is a comment). Keep the import from rel-pair-routing.ts (its canonical home, still used by splitRelCsvByLabelPair's signature) and drop the dead re-export. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-15 19:40:59 +01:00
dependabot[bot]	1150eea98f	chore(deps): bump actions/checkout from 6.0.2 to 6.0.3 (#2152) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.2 to 6.0.3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](de0fac2e45...df4cb1c069) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-06-10 21:58:49 +01:00
Gergő Magyar	5bf8a17cd5	feat(ingestion): add control-flow-graph layer for TS/JS (#2081) (#2099) ... * feat(cfg): language-agnostic CFG construction core (#2081) U1 of M1 (CFG layer). Plain JSON-serializable CFG data model (BasicBlockData/ CfgEdgeData/FunctionCfg — must survive the worker→main boundary + ParsedFile store), a CfgBuilder accumulator (leaders→blocks→edges, synthetic ENTRY/EXIT, idempotent edges), a ControlFlowContext (break/continue/switch + labeled-jump target stacks), and a TraversalResult ({entry, dangling exits}). AST-agnostic and unit-tested on the classic control-flow topologies (if/else, while back-edge, mid-block return, labeled break/continue) the S2 spike validated; reachability helper backs the R9 property test. * feat(ingestion): U2 — TS/JS CFG visitor over tree-sitter AST (#2081) Add the TS/JS CfgVisitor that walks a function's tree-sitter AST and drives the U1 CfgBuilder to produce a serializable FunctionCfg. One visitor covers both languages (shared grammar family). Handles the classic CFG hazards explicitly (R2, R10): - loops allocate a dedicated loop-exit block so `break` has a concrete target before the loop's successor is known; `continue`/back-edge close the loop (while, do-while, C-for with init-once + increment-as-continue-target, for-in, for-of) - switch fallthrough falls out naturally: a non-breaking case yields exits we wire to the next case as `fallthrough`; a breaking case wires to the switch exit via ControlFlowContext - try/catch/finally: normal completion AND exceptional flow both route through finally (post-domination); a conservative exceptional edge models that the protected region may raise to its handler (not just explicit `throw`) - labeled break/continue resolve against the labeled loop's frame - early return/throw wire to EXIT/handler and terminate their block 19 hazard tests (one per construct) + AC1 10-function fixture; all green. No change to the committed U1 core or ControlFlowContext. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U3 — worker CFG build + cfgSideChannel + cache coherence (#2081) Run the CFG visitor in the parse worker (where the AST lives), serialize the per-function CFG onto a new ParsedFile.cfgSideChannel, and keep it coherent across the disk-backed store and the warm/durable parse cache (R3, R4). - gitnexus-shared parsed-file.ts: add `cfgSideChannel?: unknown` as a DISTINCT field from captureSideChannel (different producer/consumer/lifecycle; plain JSON data — blocks/edges deliberately lack the `nodeId` the store's interning reviver keys on, so no mis-interning). - cfg/types.ts + visitors/typescript.ts: add CfgVisitor.isFunction so the worker enumerates functions (and applies the line budget) by a cheap node-type test. - cfg/collect.ts (new): collectFunctionCfgs walks the tree, builds one CFG per function (nested included), applies maxFunctionLines (over-cap = skipped). - language-provider.ts: add `cfgVisitor?: CfgVisitor<SyntaxNode>` hook; typescript.ts attaches it to both the TS and JS providers (shared grammar). - parse-worker.ts: read pdg + pdgMaxFunctionLines from workerData (read once at init — the worker never sees PipelineOptions), gate the build, attach cfgSideChannel alongside captureSideChannel. - parse-cache.ts: bump SCHEMA_BUMP 4→5 (ParsedFile shape changed) and fold the pdg flag into computeChunkHash so a pdg-off cached chunk is NOT reused on a --pdg run (the #2038-class warm-cache trap). Default path keeps its keys. - worker-pool.ts + parse-impl.ts + pipeline.ts: thread pdg/pdgMaxFunctionLines PipelineOptions → WorkerPoolOptions → workerData, and into the chunk-hash key. 9 boundary tests: collect contract, JSON round-trip identity (no AST leakage), the pdg cache-key guard, the line-cap skip, and the no-visitor gate. Full CFG suite (U1+U2+U3) green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U4 — emit BasicBlock + CFG within scope-resolution (#2081) Emit persisted BasicBlock nodes + CFG edges from each ParsedFile's worker-built cfgSideChannel, INSIDE scope-resolution's Phase-4 graph emission — the last point where the worker-built CFGs are loaded (emitParsedFiles carries the channel; the disk store is cleared right after the orchestrator returns). This is the architecture the doc-review corrected to: a standalone post-`mro` phase (the issue's literal subtask) provably reads empty data (KTD1). - cfg/emit.ts (new): pure emitFileCfgs(graph, cfgs, maxEdgesPerFunction, onWarn). BasicBlock id = `BasicBlock:<filePath>:<functionStartLine>:<blockIndex>` (KTD3 — funcStart disambiguates blocks across functions in one file; no `name` column). CFG edge = CodeRelation type 'CFG' with the edge KIND (seq/cond-true/…) in `reason` (kinds can't be their own edge type). Per- function edge cap stops at the cap and warns with the dropped count — no silent truncation (R6/KTD6). - run.ts: pdg-gated emit pass over emitParsedFiles after emitPostResolutionEdges (store still live); RunScopeResolutionInput gains pdg + pdgMaxEdgesPerFunction. - phase.ts: thread ctx.options.pdg / pdgMaxEdgesPerFunction into the call. - pipeline.ts: PipelineOptions.pdgMaxEdgesPerFunction. 6 tests: node/edge shape (KTD3 id, no name, type='CFG', kind in reason), cross-function id uniqueness, AC2 reachability-from-ENTRY property, the edge cap's no-silent-truncation contract, and empty-input no-op. Flag-off byte-identity + full runPipelineFromRepo round-trip land in U7. Build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): U5 — `--pdg` opt-in plumbing (CLI + .gitnexusrc → both sinks) (#2081) Expose the CFG/PDG substrate as an opt-in and thread it from CLI/.gitnexusrc to the single source of truth (PipelineOptions.pdg), which fans out to BOTH sinks already wired in U3/U4: the worker build gate (workerData.pdg) and the scope-resolution emit gate. Off by default (R7). - cli/index.ts: `--pdg` commander flag. - cli/analyze.ts: AnalyzeOptions.pdg + pass `pdg` into runFullAnalysis options. - cli/analyze-config.ts: KEY_SPECS `pdg` (boolean) so `.gitnexusrc { "pdg": true }` normalizes and a non-boolean value fails closed with GitNexusRcError. - core/run-analyze.ts: AnalyzeOptions.pdg → runPipelineFromRepo({ pdg }). (The internal PipelineOptions/WorkerPoolOptions/workerData fields + the parse-cache key fold landed in U3/U4; this unit adds the user-facing surface. The budget knobs stay at internal defaults for M1.) Tests: analyze-config pdg normalization + non-boolean rejection; opt-in.test.ts covers the CLI/file merge precedence and that pdg perturbs the chunk-dispatch key. The full worker-build + main-emit round-trip is the U7 integration test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): U7 — CFG acceptance fixtures, parity, end-to-end + docs (#2081) Acceptance criteria for the M1 CFG layer: - AC1: a 10-function TS fixture's CFG node/edge set matches a committed snapshot (cfg-snapshot.test.ts). - AC2: every BasicBlock is reachable from its function ENTRY (property test over the emitted graph; the fixture has no dead code). - AC3: hazard fixtures lock the classic-bug coverage — try/throw/finally post-domination + labeled break/continue resolution. - AC4: the existing pipeline-graph-golden test stays byte-identical with --pdg off (verified; no UPDATE_GOLDEN), proving the opt-in adds zero default-run drift. - End-to-end (pipeline-pdg.test.ts): runPipelineFromRepo({ pdg: true }) on a tiny repo emits BasicBlock nodes + CFG edges with both endpoints present — the true both-sinks proof (worker builds → store → scope-resolution emits); the default run emits zero. Docs: CHANGELOG M1 entry, ARCHITECTURE "Optional CFG/PDG emission" subsection (why emit is in-phase, not post-mro), README CFG language-support note. Full CFG suite (U1–U7): 56 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): drop unused helper in cfg-snapshot test (#2081) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): apply ce-code-review autofix feedback (#2081) Review (10 reviewers) confirmed OFF-path byte-identity (adversarial + golden) and found defects all within the --pdg path. Fixes: - P1 same-line BasicBlock id collision: add a start-column disambiguator to FunctionCfg + the id (`BasicBlock:<file>:<line>:<col>:<idx>`) so two functions sharing a start line no longer collide under first-writer-wins addNode. - P1 worker crash-cascade: per-file try/catch around collectFunctionCfgs so a CFG-build throw cannot escape to the language-group catch and silently drop every remaining file in the group. - P2 edge-cap drop now logs unconditionally (input.onWarn is validator-gated/ silent in prod) — upholds the no-silent-truncation guarantee. - P2 Array.isArray guard before the cfgSideChannel cast in run.ts. - P2 maxFunctionLines default: worker applies DEFAULT_PDG_MAX_FUNCTION_LINES=2000 when unset; caps forwarded through run-analyze AnalyzeOptions (closes the server-path drop). - P3 README duplicate paragraph removed; `0`-vs-default docstrings corrected; CLI --pdg flag made language-neutral; reachableBlocks JSDoc corrected. - Documented the break-through-finally + stacked-label CFG limitations. - Tests: same-line id-collision regression, standalone throw→EXIT, dead-code- after-return, async/generator/method coverage, strengthened labeled-continue. Refuted: the HTTP-500 getNodeQuery finding — M0 already shipped the BasicBlock branch + name-floor (R12/web-safety handled). CFG + analyze-config suites: 95 tests green; golden parity (AC4) byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): benchmark CFG construction + O(n) block-text accumulation (#2081) Closes the M1 review's requires_verification perf gap ("no benchmark for collectFunctionCfgs; a wall-time + cfgSideChannel byte-size regression gate would catch the extendBlock concatenation before kernel scale"). - bench/cfg/measure.mjs (new): build-free tsx harness timing collectFunctionCfgs (parse once, reuse the tree) across three scaling scenarios — straight-line (extendBlock path), many-functions (collect walk), branchy (block/edge growth) — at 500→2000. Reports a wall-time scaling ratio AND a cfgSideChannel byte-size ratio, plus an order-independent sha256 over the emitted blocks/edges as the behavior gate. `--check` compares both ratios + the fingerprint against bench/cfg/baselines.json; mirrors the scope-capture / python-scope harnesses. - .github/workflows/ci-tests.yml: run the gate on every test job (build-free, alongside the existing scope-capture guards) so an O(n^2) re-regression fails CI. - cfg-builder.ts: structural fix for the one real hotspot the bench surfaced — accumulate basic-block text as fragments joined once in finish(), instead of concatenating onto a growing string per coalesced statement (O(n^2) → O(n)). Behavior-identical (the CFG fingerprint + the AC1 snapshot are unchanged). Measured (post-fix): time ratios straight-line ~1.3, many-functions ~1.0, branchy ~1.1 (all sub-quadratic; a true O(n^2) would be ~4.0). cfgSideChannel bytes scale linearly (~1.0-1.04). 60 CFG tests green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): add memory + disk growth gates to the CFG benchmark (#2081) Extend bench/cfg/measure.mjs beyond wall-time to the two other scalability dimensions that matter at kernel scale: - DISK growth: utf8 byte size of the serialized cfgSideChannel — exactly what a --pdg run writes onto every ParsedFile shard (durable store + parse cache). - MEMORY growth: retained JS heap of the cfgSideChannel payload, measured by the release-delta method (heap held minus heap after dropping it) — robust to pre-existing garbage and dead-stable run-to-run. Needs `node --expose-gc`; without it the heap metric is null and its gate is skipped (local runs still work). ci-tests.yml now passes --expose-gc so the heap gate runs in CI. Both gated on linear scaling in baselines.json (disk_bytes_budget / heap_budget 1.2-1.3). Measured: disk ~1.0-1.04, retained heap ~0.87-1.0 — both linear (~1KB/function each; ~2MB heap / 1.6MB disk at 2000 functions, --pdg only). Bumped REPS 7->15 to stabilize the noisier time signal and widened the coarse time tripwire budgets (the disk/heap gates carry the tight regression detection). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): address tri-review + CFG-expert findings (#2081) Corroborated findings from the tri-review (Codex + CE personas + GitNexus swarm + a CFG/program-analysis domain-expert lane). The OFF-path stays byte-identical; all fixes are within the --pdg path or the benchmark. - [Codex+CFG-expert] Exceptional `throw` edges now wire EVERY block in a try's protected region to the handler, not just the body ENTRY. A branched try body (`try { if (x) { use(t); } } catch`) previously left interior blocks with no path to `catch` — a taint false-negative into the handler for the M2 PDG pass. - [Codex+CFG-expert] An unresolved labeled jump (a stacked outer label or a labeled non-loop block) now routes to the function EXIT instead of leaving a dangling sink — restores the single-exit invariant post-dominator/PDG computation needs. - [Codex] computeChunkHash now folds pdgMaxFunctionLines/pdgMaxEdgesPerFunction into the chunk key (not just the pdg boolean), so a warm cache built under one cap is never served to a run with a different cap (#2038 class, extended to the budgets). Adds PdgCacheKey; boolean form kept for back-compat. - [perf] visitTry resolves catch/finally in a single namedChild pass (the double `namedChildren.find` allocated two throwaway arrays). - [adversarial] The bench `straight-line` scenario now runs at 2000->8000: output is a constant 4 blocks so disk/heap can't see the concat path, and at the old N a genuine O(n²) was masked by V8 cons-strings. Verified at the new N: the array-join impl ~1.0, a rope-optimized `+=` ~1.0 (correctly not flagged), a real O(n²) (re-join-every-append) ~3.8 — budget tightened 2.0->1.5. - [adversarial+Codex] The bench `--check` now FAILS LOUDLY when run without `--expose-gc` instead of silently skipping the retained-heap gate. - Doc: re-labeled the finally-bypass as a SOUNDNESS (false-negative) limitation tracked for M2, not mere "precision." 3 new regression tests (branched-try interior→handler, stacked-label→EXIT, cap-fold key). 99 CFG tests pass; build clean; bench gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(parse-cache): clarify that SCHEMA_BUMP still invalidates caches once (#2099 F6) The computeChunkHash comment claimed pdg-off warm caches "survive this change untouched" — true for the key FORMAT, but misleading as an upgrade-behavior promise: SCHEMA_BUMP 4→5 changes PARSE_CACHE_VERSION and both stores hard-invalidate on it. Separate the two facts so the next cache change isn't reasoned about from a false premise. Review finding F6 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): correct for-loop back-edge kinds when no increment clause (#2099 F5) A for with a body but no increment emitted an unconditional header→header 'loop-back' self-edge (a path that never executes the body) while the real back-edge body→header was labeled 'seq'. Any consumer identifying loops via reason='loop-back' picked the phantom edge and excluded the body from the natural loop. Gate the self-edge on the body being absent (the one case where the header genuinely re-tests itself) and carry 'loop-back' on the body's exits when they ARE the back-edge, matching visitWhile/visitForIn. Review finding F5 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): treat an empty catch clause as a real handler (#2099 F2) visitTry keyed handler semantics off the traversal result — null for an empty body, since visitSeq([]) returns null — instead of the syntactic clause. An empty `catch {}` was therefore treated as NO catch: the swallowed exception escaped to the outer handler/EXIT, the no-catch re-propagation misfired past finally, and code after a try whose body always throws became unreachable from ENTRY — a hard false-negative source for the M2 taint pass, on an extremely common pattern. Synthesize one empty block spanning the clause (entry == sole exit) when the catch body traverses to null, before the protected region is walked. Exception flow lands in it and rejoins the normal continuation; all downstream wiring (handler selection, finally routing, the !catchRes re-propagation gate) operates on the syntactically-correct shape. Review finding F2 (P2, reproduced) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): guard CFG emission per element, not just per outer array (#2099 F4) The cfgSideChannel guard checked only Array.isArray before casting to FunctionCfg[] — its own comment promised a wrong-shape value would 'skip emission, not throw a TypeError mid-graph-build', but a malformed ELEMENT sailed through. Worse, the obvious-looking failure shape never throws at all: emitFileCfgs string-templates any edge endpoint into the BasicBlock id and graph inserts are no-throw, so a non-integer endpoint silently became a dangling 'BasicBlock:…:undefined' edge that degrades the DB rel-pair COPY to row-by-row fallback inserts much later. Layered fix matching house precedents (parsedfile-store reviver, worker-side per-file catch): a per-element shape+content predicate (arrays + integer edge endpoints) that warns and skips malformed elements while valid siblings still emit, plus a per-file try/catch backstop for shapes that genuinely throw (e.g. a null inside blocks). Review finding F4 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(parse-cache): drop emit-time edge cap from the pdg chunk key (#2099 F3) pdgMaxEdgesPerFunction is applied exclusively in emitFileCfgs during scope-resolution on the main thread — the worker never receives it (workerData carries only pdg + pdgMaxFunctionLines), so the cached worker output is byte-identical across cap values. Folding it into the chunk key (added by a prior review round) only converted a free knob into a repo-sized cost: every cap change forced a full re-parse and a durable-store rewrite of unchanged data. Keep pdg + maxFunctionLines (genuinely worker-visible, shape the cached cfgSideChannel) and document the classification test in the PdgCacheKey doc comment so the next option gets sorted deliberately: worker-shard inputs go in this key; persisted-graph-only inputs belong in the RepoMeta pdg stamp (F1). Chunks written under the old ns string miss once and prune — no migration needed. Review finding F3 (P2) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(analyze): record pdg config in RepoMeta; force full writeback on mode flip (#2099 F1) Running --pdg against an already-indexed repo silently persisted ~zero CFG: incremental eligibility had no pdg term, RepoMeta recorded no mode, and extractChangedSubgraph keeps only changed-file nodes — on a no-change --pdg re-run every freshly built BasicBlock was dropped from the written subgraph ('Incremental: changed=0', run succeeds, zero rows). The converse flip left zombie mixed-coverage blocks only --force could clean. Worse, a clean-tree flip hit the alreadyUpToDate fast path and never ran the pipeline at all. - RepoMeta gains an additive-optional pdg stamp ({maxFunctionLines, maxEdgesPerFunction}, resolved values; absent ≡ pdg-off, which covers every legacy meta). No INCREMENTAL_SCHEMA_VERSION bump — that would force a one-time full rebuild for everyone. The end-of-run meta is a fresh literal, so omitting the field on a pdg-off run is what clears the stamp after an on→off flip. - pdgModeMismatch (pure, exported) compares the resolved triple; the flip check sits before the fast path and always logs its notice (not gated on options.force — --skills implies force with no message of its own), naming the .gitnexusrc pdg key that pins the mode. - The full-rebuild branch now writes the incrementalInProgress dirty flag (toWriteCount: 0 sentinel) before the wipe whenever a prior meta exists, mirroring the incremental branch. This closes the crash window where a rebuild dying between the bulk load and saveMeta left meta/DB inconsistent and the fast path certified zombie (or missing) CFG rows indefinitely — and incidentally closes the same pre-existing hole for user --force runs. Recovery log reworded accordingly. Tests: pdg-mode-flip.test.ts (real git + LadybugDB; primary assertion is a direct BasicBlock table count — meta.stats aggregates nondeterministic Community/Process rows) covering off→on, steady-state fast path, on→off zombie cleanup, cap-change rebuild, and dirty-flag + flip composition; pure-helper tests for default resolution and the 0=unlimited carve-out. Review finding F1 (P1) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 19:26:45 +01:00
Gergő Magyar	cef63dd044	feat(install): toolchain-free tree-sitter via vendored prebuilds (#2113) ... * feat(install): toolchain-free tree-sitter via vendored GitNexus-built prebuilds Eliminate the C/C++-toolchain requirement at install for the at-risk grammars (dart, proto, kotlin) by generating + vendoring native prebuilds, mirroring the existing vendored tree-sitter-swift. The 10 grammars that already ship 6 upstream prebuilds stay npm dependencies (toolchain-free AND dependency-review-tracked). - .github/workflows/build-tree-sitter-prebuilds.yml: a registry-parameterized workflow that builds {dart,proto,kotlin} x {linux,darwin,win32}-{x64,arm64} prebuilds natively, validates each loads + parses on its arch, and opens a PR vendoring them. A `guard` job gates the heavy matrix to run ONLY on dispatch or a real grammar-version change — ordinary code PRs cost zero matrix minutes. - dart/proto: prefer a committed prebuild; fall back to today's source build when none matches (no behavior change until prebuilds are vendored). - kotlin: vendor it (Swift parity) instead of compiling the third-party optionalDependency from source at the user's install — supersedes #2110's optionalDependency mechanism. The ~23 MB parser.c is NOT vendored (the workflow builds from the published package); only node-types + bindings + prebuilds are. Removed from optionalDependencies; lock regenerated; probe, parser-loader note, README/.devcontainer docs, and the #2110 tests updated. DO NOT MERGE until vendor/tree-sitter-kotlin/prebuilds/ is populated by the build-tree-sitter-prebuilds workflow: until then Kotlin is unavailable (vendored with no source-build fallback). dart/proto remain fully functional throughout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): guard 6/6 N-API prebuild coverage for every grammar Regression guard so a toolchain-less install can never silently lose a tree-sitter language on a supported platform-arch: - Vendored grammars (vendor/tree-sitter-*): every one MUST ship a loadable N-API prebuild for all 6 tuples {linux,darwin,win32}-{x64,arm64}. Asserts the napi_register_module_v1 entry symbol in each .node (cross-platform, no need to run the binary). Currently RED for dart/proto/kotlin until the build-tree-sitter-prebuilds workflow populates their prebuilds/ — this is the must-fill-before-merge gate (swift already passes 6/6). - npm-dependency grammars: asserts upstream ships 6/6 N-API too, catching a future platform drop. tree-sitter-c is allow-listed at 4/6 (missing linux-arm64/win32-arm64) pending #2116; the guard also fails if that gap is silently closed (prompting allow-list removal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(install): vendor tree-sitter-c at 0.21.4 with GitNexus-built prebuilds (#2116) tree-sitter-c is the one grammar dependency upstream ships incomplete prebuilds for (4/6 — no linux-arm64/win32-arm64), AND it is a REQUIRED grammar: its own `install` (node-gyp-build) compiles from source when no prebuild matches and exits non-zero, so on a toolchain-less ARM host `npm install gitnexus` HARD-FAILS at the c step — during npm's dependency phase, before any GitNexus postinstall runs (so a postinstall "supplement" can't help). Fix: vendor c prebuild-only at the pinned 0.21.4 (Kotlin pattern), with all six prebuilds GitNexus-cross-built, and drop it from `dependencies`: - vendor/tree-sitter-c/ (bindings + node-types + manifest + prebuilds); build probe scripts/build-tree-sitter-c.cjs; added to the build workflow registry (kind 'npm' — built from c@0.21.4 source). - materialize-vendor-grammars.cjs: c is REQUIRED, so it is always materialized, even under GITNEXUS_SKIP_OPTIONAL_GRAMMARS (it needs no toolchain). - Removed from package.json dependencies + lockfile (nothing else needs npm c — tree-sitter-cpp's dep on c is dev-only and not installed). Preserves the #1242 ABI pin: vendoring 0.21.4 keeps the good ABI while closing the ARM gap. - parser-loader note + the prebuild-coverage guard + a cli-commands assertion updated; c moves from the npm-gap allow-list into the vendored 6/6 cohort. Verified: tsc clean, 31 unit tests pass, c loads/parses; the guard is RED for c/dart/proto/kotlin until the workflow populates prebuilds (the must-fill gate). Closes the operational risk in #2116. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): source-build fallback for vendored c/kotlin so CI is healthy pre-prebuilds The vendored prebuild-only grammars (c, kotlin) had empty prebuilds/ until the build-tree-sitter-prebuilds workflow runs, so they could not load in CI — and C is hard-required by cross-platform tests (tree-sitter-languages/parsing on ubuntu+macos+windows), which I cannot pre-build for macos/windows locally. The robust fix is a source-build fallback that works on every CI runner (all have a toolchain), mirroring dart/proto: - Vendor the grammar source (binding.gyp + src/) for c and kotlin; their build scripts now PREFER a committed prebuild (toolchain-free) and fall back to `node-gyp rebuild` from the vendored source when no prebuild matches. Verified both compile against the hoisted node-addon-api@^8 and the runtime loads. - prebuild-coverage guard is now bootstrap-tolerant: a grammar that vendors its source (binding.gyp) may have an incomplete prebuild set (the workflow fills it); a prebuild-only grammar (swift) still must ship all six. Any present prebuild must still be N-API. Guard goes green; it re-tightens per-grammar as the workflow populates prebuilds. - actionlint: silence a false-positive SC2016 (JS template literals inside the single-quoted `node -e` validate block). Note: kotlin's generated parser.c is large (~23 MB on disk; compresses heavily in git). Once the workflow populates all six kotlin prebuilds, the source serves only as the fallback and could be slimmed if desired. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(docker): re-materialize+rebuild vendored grammars after npm prune `npm prune --omit=dev` in the gitnexus CLI image drops anything not in package.json's dependency tree — including the VENDORED tree-sitter grammars (materialized by postinstall, not declared deps) and their built bindings. The `serve` image analyzes/parses repos at runtime, so re-run the grammar postinstall after the prune (in the toolchain-equipped builder) to restore them. Load-bearing for tree-sitter-c, a core REQUIRED grammar now vendored (#2116): as a former dependency it survived prune; vendored, it would not. Also restores swift/dart/proto/kotlin, which were silently pruned from the image before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(grammars): unify tree-sitter-swift with the vendored-source build pipeline Swift was the last grammar handled differently — it shipped only upstream prebuilds, while c/dart/proto/kotlin vendor their grammar source and use a prefer-prebuild -> source-build-fallback activation script. Vendor swift's source so all five are handled identically (one uniform build path). - vendor/tree-sitter-swift: add binding.gyp (win-hardened), bindings/node/ binding.cc, src/parser.c (ABI-14 default, ~18 MB), src/scanner.c, and src/tree_sitter/ headers. The 6/6 prebuilds are retained. The legacy parser_abi13.c alternate is intentionally not vendored. - build-tree-sitter-swift.cjs: rewrite the prebuild probe into the dart-style prefer-prebuild then source-build fallback (keeps the GITNEXUS_SKIP gate and the never-exit-non-zero postinstall invariant). - build-tree-sitter-prebuilds.yml: register swift (kind 'vendored'); add its package.json to the version-gated pull_request paths and a validate snippet. - prebuild-coverage guard auto-moves swift into the source-fallback cohort (binding.gyp now present); refresh the stale "swift is prebuild-only" comments. - tests: add build-tree-sitter-swift-probe.test.ts; fix the pre-existing build-tree-sitter-kotlin-probe.test.ts breakage (it still asserted the old probe strings after kotlin's dart-style conversion); assert swift's vendored source in cli-commands.test.ts. - docs: README / .devcontainer / kotlin vendor README — swift's prebuilds are now GitNexus-cross-built from vendored source like the rest, not upstream-only. Verified: swift source-builds against node-addon-api@8 -> N-API binary -> loads against the pinned tree-sitter@0.21.1 (ABI 14) -> parses cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(publish): gate a lean prebuilds-only npm tarball behind a coverage guard Vendoring grammar source (parser.c) alongside the prebuilds means the npm tarball now carries ~50 MB of generated source it almost never compiles (every supported platform-arch has a prebuild). Prepare to drop it from the published package once all prebuilds exist — safely. - .npmignore: add a GATED, commented-out "lean publish" block that excludes the source-build inputs (parser.c/scanner.c/tree_sitter/binding.gyp/binding.cc) but keeps prebuilds/ + the runtime files. Uncommenting ships prebuilds-only. - scripts/assert-publish-grammar-coverage.cjs: a prepack guard that refuses to pack/publish if the source exclusion is active while any vendored grammar still lacks 6/6 prebuilds (which would ship a grammar with no loadable binding). Wired into `prepack` (runs on npm pack + publish, incl. the publish.yml dry-run) and exposed as `npm run assert-publish-coverage`. - test: pure-core decision cases + a real-repo publish-safety check that fails CI if .npmignore is activated prematurely. Net: the prebuilds already publish today (files: ["vendor"]); this makes the future switch to a prebuilds-only tarball a one-line uncomment that can't ship a dead grammar. The guard currently reports "source + prebuilds" (only swift has 6/6 prebuilds so far) and passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(grammars): consolidate the 5 build-tree-sitter-*.cjs into one The per-grammar activation scripts (c/dart/proto/swift/kotlin) were ~95% identical — same prefer-prebuild → source-build → never-fail flow, differing only in name, target_name, required-vs-optional, and the display label in warnings. - scripts/build-tree-sitter-grammars.cjs: one registry-driven script. Bare call builds all (postinstall); `... <name>` builds only the named grammars (so the probe test can isolate one). c is `required: true` (ignores the opt-out gate); the rest honor GITNEXUS_SKIP_OPTIONAL_GRAMMARS. Per-grammar try/catch + a final process.exit(0) preserve the postinstall never-exit-non-zero invariant. - package.json: postinstall is now `materialize && build-tree-sitter-grammars.cjs` (was five chained `build-tree-sitter-<name>.cjs` calls). - tests: replace the two near-identical *-probe.test.ts files with one parameterized build-tree-sitter-grammars-probe.test.ts that also covers the required-vs-optional opt-out split and an unknown-grammar arg. - update cli-commands.test.ts postinstall assertions + the vendor c/kotlin/swift README + swift provenance to reference the consolidated script. Behavior is preserved (warnings normalized to one consistent format). Removes 5 scripts + 1 test file; adds 1 script + 1 test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): lazy-load tree-sitter-c to prevent module-load crash tree-sitter-c is now vendored prebuild-only (#2116) with 0/6 committed prebuilds, so on a toolchain-less or `--ignore-scripts` install C has no native binding. Three modules loaded it via a hard top-level `import C from 'tree-sitter-c'`, which throws ERR_MODULE_NOT_FOUND at module-load — crashing `analyze` before parser-loader's optional/severity:error degradation can run. This is the #2091/#2093 bug class (previously fixed for swift/dart/kotlin); C was left static because it used to be an always-present npm dependency. - languages/c/query.ts: load via the lazy guarded getLanguageGrammar(C), mirroring swift/query.ts; the main-thread isLanguageAvailable filter ensures the getters are reached only when C is present. - workers/parse-worker.ts: guarded `_require('tree-sitter-c')` + conditional languageMap spread, like swift/dart/kotlin. - group/extractors/include-extractor.ts: guarded `_require`; getLanguageForFile returns null for .c/.h when absent, so C include-extraction degrades to a no-op (C++ unaffected). - extend the registry-import-closure regression test (#2091/#2093) to assert C also loads lazily at registry static-import time. * fix(ci): repin attest-build-provenance to the real v2.4.0 SHA The workflow pinned actions/attest-build-provenance@bd77c077… commented `# v2.4.0`, but v2.4.0 is e8998f94… (verified via the GitHub API); bd77c077… is an untagged mid-stream commit, so the SLSA-attestation step ran unvetted action code and the comment misrepresented what runs. Repin to the real v2.4.0 commit and drop the `# PLACEHOLDER-PIN` markers on both this line and the setup-python pin (a26af69b… is already the correct v5.6.0 — only its comment was stale). Update the header NOTE accordingly. * fix(ci): skip the prebuild-PR aggregate when release App secrets are absent The aggregate job mints a GitHub App token as its first step; with RELEASE_APP_ID/RELEASE_APP_PRIVATE_KEY unset it hard-failed AFTER a full (up-to-6-runner) native build. Since the `secrets` context isn't available in a job-level `if:`, the guard job now computes a `release_app` boolean output (a step can read secrets) and emits an actionable `::notice::`; aggregate gates on it and skips cleanly, while the build job's artifacts still upload (run with open_pr=false for artifacts-only). * chore(ci): drop package-lock.json from the prebuild paths filter; widen build timeout `gitnexus/package-lock.json` changes on nearly every dependency PR, so it fired the prebuild workflow's guard job on unrelated churn (the matrix stayed correctly skipped — `gitnexus/package.json` already covers the transition-window pin, so removing the lock only drops guard noise). Also bump the native build job timeout 30 -> 45 min for headroom compiling the 23 MB kotlin / 18 MB swift parser.c, especially under arm emulation. * fix(ci): event-gate the aggregate open-PR condition explicitly `inputs.open_pr` is null on pull_request events, and the prior `inputs.open_pr != false` leg relied on GHA's direction-ambiguous null coercion (Codex F4) to decide whether to open the prebuild PR. Gate explicitly on the event: a non-fork pull_request that bumped a grammar version opens the prebuild PR (the documented flow), and `open_pr` is only consulted on workflow_dispatch — so a manual run with open_pr=false stays artifacts-only and no event's behavior rests on coercion. * fix(publish): validate the effective npm-pack contents in the coverage guard The publish guard inferred "is source shipped?" from a single .npmignore toggle line, which a partial/out-of-order edit could defeat (exclude binding.gyp but leave parser.c → unbuildable yet "source-shipping"). It now inspects the EFFECTIVE tarball via `npm pack --dry-run --ignore-scripts --json` (the --ignore-scripts avoids re-entering this guard through prepack): a grammar "ships source" only when EVERY on-disk source-build input (binding.gyp + binding.cc + parser.c + scanner.c when present + a tree_sitter header) is actually in the packed file list. This also surfaced that the gated lean-publish .npmignore block was inert: package.json's `files: ["vendor"]` allow-list overrides .npmignore for the vendored subtree, so those exclusion lines never dropped anything. Replace the dead toggle with documentation of the real mechanism (narrow the `files` field) and note the guard enforces safety on the effective pack regardless of how the slim is done. * test(prebuild): hard-gate declared-fully-prebuilt grammars on 6/6 coverage The strict 6/6 prebuild assertion was dormant whenever a grammar vendors source (binding.gyp) — which is every grammar — so a dropped prebuild passed CI silently. Add a FULLY_PREBUILT allowlist of grammars GitNexus has committed 6/6 for (today: swift); those must keep all six even with a source fallback, so losing one now fails CI. Grammars graduate into the set as the build-tree-sitter-prebuilds workflow lands their binaries. (The static-import degradation smoke is covered by the registry-import-closure regression test extended in the C lazy-load commit.) * chore(deps): promote node-gyp-build/node-addon-api to regular dependencies Every vendored grammar's index.js does `require("node-gyp-build")` at runtime to load even a prebuilt .node, so node-gyp-build is runtime-load-critical (and node-addon-api is needed for the source-build fallback). They were optionalDependencies, surviving `--omit=optional` only via the required tree-sitter's transitive edge — correct today but fragile. Promote both to regular dependencies so the contract is explicit (optionalDependencies is now empty and removed). Lock the contract with a cli-commands assertion. * chore(vendor): add Windows cflags parity block to tree-sitter-c/binding.gyp c's binding.gyp used an unconditional `cflags_c: ["-std=c11"]`, while kotlin/swift gate MSVC flags behind an `OS=='win'` condition (/std:c11 /utf-8). Inert today (no non-ASCII bytes in c's parser.c, and node-gyp ignores cflags_c on MSVC anyway), but align the three so a future source-build fallback on Windows behaves consistently. * docs(agents): correct stale optional-grammar / postinstall notes AGENTS.md still said postinstall "patches tree-sitter-swift, builds tree-sitter-proto" and that only kotlin/swift are "optional". Update to the vendored-uniform model: postinstall materializes the vendored grammars and prefers a committed prebuild (source-build only when none matches); c is required while dart/proto/swift/kotlin are optional + skippable via GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1, with non-fatal warnings only on a toolchain-less host with no matching prebuild. * fix(install): preserve the backup and warn loudly on a failed materialize rollback If renameSync(partial, dest) failed AND the rollback renameSync(backup, dest) also failed, the grammar was left unmaterialized (node_modules/<name> missing) with only a generic "could not materialize" warning — the recoverable backup at <dest>.materialize-bak was unmentioned. Emit a CRITICAL warning naming the backup path and the recovery command on that double-failure, and document that the fail-soft catch removes only the scratch `partial`, never the `backup` (which may be the sole recoverable copy). Never-throw / exit-0 contract intact. * fix(publish): make the coverage guard's npm-pack inspection script-safe The prepack guard shelled out to `npm pack --dry-run --ignore-scripts --json`, but the `--ignore-scripts` flag is not reliably honored by npm pack's prepare/prepack lifecycle on the CI npm — so build.js ran, polluted the --json stdout with `[build] …`, and the guard's JSON.parse threw. That broke every `npm pack` (packaged-install-smoke on ubuntu+windows) and failed the guard's own real-repo unit test (the only coverage-job failure). Force script-skipping via the reliable `npm_config_ignore_scripts` env config (also removes the prepack re-entry/recursion risk) and parse defensively from the JSON-array start. * fix(publish): make the coverage guard deterministic — read `files`, not `npm pack` The npm-pack-based guard timed out in CI: `npm pack`'s prepare/prepack lifecycle is not skipped by `--ignore-scripts` (flag or env config) on the CI npm, so the inner pack ran the full build (~20s+) — fine for the slow smoke job, but it blew past vitest's 30s test timeout in the coverage job (and risked re-entering this prepack guard). Replace it with a deterministic, fast (~0.1s) check that needs no subprocess: since `files: ["vendor"]` OVERRIDES `.npmignore` for the vendored subtree (so `.npmignore` can never drop vendored source — verified), the ONLY lever that can exclude source is narrowing the package.json `files` field. The guard now reads `files` directly: a grammar "ships source" iff `files` includes the vendor subtree AND the grammar carries a buildable source set on disk. A lean publish that narrows `files` while a grammar lacks 6/6 prebuilds still fails the gate. * feat(ci): vendored tree-sitter grammar update monitor Adds a weekly (+ dispatchable) workflow that checks each vendored grammar against its source-of-origin (npm for swift/kotlin, the GitHub default branch for dart/proto; c is excluded — held at 0.21.4 for ABI safety) and opens a PR re-vendoring any update that is ABI-COMPATIBLE with the pinned tree-sitter@0.21.1 (LANGUAGE_VERSION 13-14). ABI awareness is the point: most upstreams have moved to ABI 15 (newer tree-sitter), so a blind "bump to latest" would open PRs that can't build. The monitor fetches the candidate source, reads its parser.c LANGUAGE_VERSION, and only re-vendors 13/14 — incompatible updates are reported (notice + job summary), never applied. (Confirmed live: dart/proto upstreams are ABI 15 today and are correctly held; swift/kotlin are current.) The re-vendor refreshes only the source-build inputs + runtime entrypoints, preserving the GitNexus-hardened binding.gyp / README / prebuilds; the version bump then triggers build-tree-sitter-prebuilds.yml, whose ABI-validation is the final safety net so a subtly-wrong re-vendor can't silently ship. PR creation is gated on the RELEASE_APP secret (skips with a notice if absent), mirroring the build aggregate. Unit test locks the ABI gate; the script is import-safe. * feat(ci): monitor tree-sitter-c too (report-only, ABI-pinned) c was excluded from the update monitor, so an upstream c update went unnoticed. Include it, but as report-only via a `hold`: c is ABI-pinned at 0.21.4 (#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an available c update is detected + surfaced (notice + job summary) but never auto-PR'd — even if it were ABI-13/14. `--apply c` refuses defensively. (Live: upstream c is 0.24.1 / ABI 15 today, so c is doubly held — reported, not applied.) * fix(ci): drop the shell in the grammar monitor's github fetch (CodeQL) CodeQL flagged the GitHub-tarball fetch — it used `bash -c "gh api …/tarball/$ref > src.tgz && tar xzf src.tgz"`, interpolating the API-derived ref into a shell command (the shell-command-injection family: "this shell command depends on an uncontrolled file name"). Replace it with a shell-free path: capture `gh api`'s binary tarball as a Buffer via execFileSync, write it to a fixed file, and extract with execFileSync('tar', …). No shell, no injection surface. Verified the dart/proto fetch + ABI read still work. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-09 18:16:24 +01:00
Zander Raycraft	fca30c7e26	fix(audit): Centralize heritage supertype matching (#1921/#1922) (#1940) ... * fix(audit): Centralizes heritage supertype matching so qualified, generic, scoped, and interface bases produce inheritance edges across all OO languages, with per-language configs and fixtures. * fix(audit): Harden parsing for #1922 with per-parse timeouts, ERROR/partial parse flags, tree-sitter pinned to 0.21.1, and CI ABI checks for every grammar. * fix: action lint passing * fix: feedback from triage review --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-06-01 13:16:17 +01:00
Gergő Magyar	d1d2a64d0f	perf(ingestion): linearize scope-capture across all languages + Python import resolution (O(n²)→O(n)) (#1918) ... * bench(python-scope): build-free measure harness + baseline fingerprint for emitPythonScopeCaptures ce-optimize scaffolding for the python-scope-capture run. Mirrors the Go scope-capture harness (#1848): imports the .ts hotpath via tsx, times emitPythonScopeCaptures on a synthetic DAO source at 250/800 entities, and pins an order-independent sha256 capture fingerprint over the whole lang-resolution/python-* corpus + a fixed 20-entity DAO as the correctness gate. Baseline (current code) is O(n^2): 250->800 entities (3.2x) -> 10.7x time (1062->11343ms), scaling_ratio 3.34. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * optimize(python-scope-capture): thread captured nodes to kill O(n^2) findNodeAtRange re-walks emitPythonScopeCaptures re-derived each tree-sitter match's AST node via findNodeAtRange(tree.rootNode, ...) on every match, scanning all of root's named children per call -> O(matches x rootChildren) ~ O(n^2). The same #1848 bug Go had (fixed in eaf0a305), mirrored in Python's captures.ts. Thread the query-captured SyntaxNode (c.node) through a parallel tag->node map and use it directly for all three sites (import / @scope.function / @declaration.function). The Python scope query captures the full statement/definition node, so the captured node IS the one the old code re-derived by range — no ancestor walk needed (simpler than Go's import case). Output is byte-identical: an order-independent sha256 capture fingerprint over all 188 lang-resolution/python-* fixtures + a 20-entity DAO is unchanged. 800 entities: 11343ms -> 319ms (35.5x); 250: 1063ms -> 95ms (11.2x); scaling_ratio 3.34 -> 1.05 (quadratic -> linear). tsc clean; 291 python scope-resolution + resolver tests pass. Adds a golden capture-parity test (forward-drift guard across the python-* corpus + DAO shape) and a non-gated O(n^2) regression tripwire (400-entity source, 346ms vs a 10s budget). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * optimize(python-scope-capture): index Python import resolution to kill O(imports x files) scans resolvePythonImportTarget's fallback path scanned the entire repo file set on every unresolved/external dotted import — once in hasRepoCandidate (package gate) and once in resolveAbsoluteFromFiles (suffix match) — giving O(imports x files) ~ O(n^2) in the resolution phase (audit follow-up to the capture-phase #1848 mirror). Add a per-file-set index (byBasename buckets + .py dir-prefix set + normalized path set), memoized on the allFilePaths Set via a WeakMap so it is built once per run and reused across every import. The two O(files) scans become O(1)/O(bucket) lookups. The shared buildSuffixIndex is deliberately NOT reused: it keeps only a single path per suffix (longest wins) and cannot reproduce Python's exact fewest-segments-then-lexicographic tie-break across all candidates (see the import-target.ts:72 rationale) — so a purpose-built index is used instead. Output is identical: a resolver-output fingerprint over 10,021 cases (exhaustive branch matrix — tie-breaks, gating, collisions, windows paths — plus a 400-repo deterministic fuzz) is byte-for-byte unchanged (e6ec1a59...). Worst-case scaling (k imports x k files): 500/1000/2000/4000 went 25/62/231/899ms -> 1.2/2.9/6.7/10.7ms (84x at 4000, quadratic -> linear). tsc clean; 303 python scope-resolution + resolver tests pass; adds a 10-case parity guard pinning the tie-break / gating / collision semantics the index must preserve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(python): land the import-index reuse on the registry-primary path (PR #1918 P1) The PythonFileIndex WeakMap is keyed on allFilePaths Set identity, but pythonScopeResolver.resolveImportTarget wrapped the orchestrator's stable run-level set in `new Set(allFilePaths)` per import, handing a fresh key to every import — so the index rebuilt on every import and the O(imports x files) cost this index removed persisted on the production path (PR #1918 review P1). Thread ReadonlySet<string> through the resolver chain (PythonResolveContext, getPythonFileIndex, the WeakMap key, resolveAbsoluteFromFiles, hasRepoCandidate, resolvePythonImportInternal, tryResolveWithExtensions — all read-only) and drop the per-import copy so the stable set reaches the WeakMap key. Mirrors the C# counterpart (csharp/import-target.ts), which already keys on ReadonlySet. Guard it deterministically: an ungated index-build counter (index-stats.ts) + a production-path integration test that drives pythonScopeResolver over 300 imports on a stable set and asserts the index is built ONCE (was 300 pre-fix). tsc clean; resolver-output fingerprint unchanged (e6ec1a59); 369 python scope-resolution + resolver tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(python): index only .py files in the import-resolution index (PR #1918 P3b) getPythonFileIndex pushed every workspace file into byBasename (and normSet), but Python import resolution only ever queries .py paths — module <seg>.py, package <seg>/__init__.py, and .py directory prefixes. Non-.py files (.ts, .go, …) could never match any lookup, so they were pure dead weight in the index on polyglot monorepos. Skip non-.py files at the top of the index builder. dirPrefixes was already .py-gated; this extends the same guard to byBasename and normSet (both also .py-only consumers), so it is behavior-preserving. Resolver fingerprint unchanged (e6ec1a59); adds a polyglot parity case proving .ts/.go siblings never affect resolution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(python): parent-key the __init__ bucket to kill package-count skew (PR #1918 P2b) The suffix fallback's package form looked up byBasename.get('__init__.py'), which holds every __init__.py in the repo — so every multi-segment package import (pkg.sub) iterated all N packages to find the one ending /sub/__init__.py. Add byInitParent: __init__.py files keyed by their last two components (<parentDir>/__init__.py). The package lookup now targets only same-named package dirs (typically O(1)) and confirms the full suffix, so the final candidate set and tie-break are unchanged. __init__.py files stay in byBasename too, so the rarer explicit "pkg.__init__" import still resolves via the module (<lastSeg>.py) lookup. Resolver fingerprint unchanged (e6ec1a59); adds parity cases for a nested package (same-parent noise filtered by the suffix confirm) and an explicit pkg.__init__ import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(python): reproduce old startsWith gating for absolute paths + re-baseline (PR #1918 P3a) getPythonFileIndex built dirPrefixes by split('/')+filter(Boolean), which drops the leading empty component of an absolute path: "/repo/svc/x.py" yielded {repo/, repo/svc/}. The old full-scan gate compared the whole normalized path, where "/repo/svc/x.py".startsWith("repo/svc/") is false — so the index gate PASSED where the old gate BLOCKED, an absolute-path-only divergence (production paths are repo-relative, so this never fired in production). Build dirPrefixes from every slash-terminated prefix of the full path instead (including the leading "/" for absolute paths), so dirPrefixes.has(X) matches exactly when the old f.startsWith(X) did. For repo-relative paths the prefix set is identical, so production behavior is unchanged. This is NOT cosmetic. Extending the fingerprint harness with absolute-path file sets surfaced 12 fuzz cases (out of ~4000 new absolute cases) where the pre-fix index resolved an import the old code left unresolved — e.g. `pkg.thing` over {/repo/pkg/__init__.py, /repo/vendor/pkg/thing.py} from /repo/app/main.py resolved to /repo/vendor/pkg/thing.py under the buggy gate but is null (old and fixed). The fix removes those absolute-path false positives. Re-baseline justification: the committed resolver fingerprint moves e6ec1a59 -> d51ea9ed because the harness now adds ~4000 absolute-path cases (branch matrix incl. the reviewer's exact case + a 200-repo absolute fuzz). The relative-path subset is unchanged: the original 10,021-case relative corpus still hashes to e6ec1a59 after the dirPrefixes fix (the fix only alters absolute-path prefixes). The new baseline encodes the old-startsWith-equivalent (correct) behavior, verified by diffing the fixed vs. pre-fix harness output. Adds parity cases pinning the absolute false-positive (now null) and a repo-relative control of the same shape (still resolves). tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(python-bench): add --check mode + REPS=7 to the scope-capture harnesses (PR #1918 P2a) The bench harnesses were dev-only — nothing compared the committed fingerprints or guarded the scaling, so an O(n^2) regression (or a P1-style cache miss) could land silently. Add a --check mode to both: - measure.mjs: assert the capture fingerprint == baseline-fingerprint.txt AND scaling_ratio < 1.5 (linear), exit non-zero on either. REPS bumped 3 -> 7 to stabilize the median on shared CI runners. - import-target-fingerprint.mjs: assert the resolver fingerprint == baseline-import-target-fingerprint.txt, exit non-zero on drift. Without --check both still print JSON for dev use / deliberate re-baselining. Verified: --check passes on the current tree (capture f2b4376f / scaling 1.04; resolver d51ea9ed) and exits 1 with a clear message on a corrupted baseline. Wired into CI by the dedicated benchmark job (next commit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(bench): add a dedicated benchmark job wiring in the gated cross-language suites The cobol/csharp/rust/php/ruby *-pipeline-benchmark.test.ts suites are gated behind GITNEXUS_BENCH, so the main coverage job skips them — their O(n^2) scaling guards never actually ran in CI. Add a dedicated "benchmarks" job to the Tests reusable workflow that runs them with GITNEXUS_BENCH=1, plus the Python scope-capture and import-resolution fingerprint + scaling guards (measure.mjs --check, import-target-fingerprint.mjs --check) from PR #1918. Runs with --no-file-parallelism: the suites measure wall-clock and peak heap, so parallel forks both skew the timings and OOM the worker pool (reproduced locally: the parallel run crashes a worker; serial passes 5/5 in ~80s). The job is part of the Tests workflow, so it gates the existing CI Gate required check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(bench): exclude go-pipeline-benchmark from the gated job (fork-pool instability) Validation surfaced that go-pipeline-benchmark.test.ts's worker-pool (#1848) suite spins a real worker pool that exits unexpectedly under vitest's fork pool, crashing the run (1 of 3 tests, repeated). Including it would make the new benchmark gate flaky. The other five language pipeline benchmarks (cobol/csharp/rust/php/ruby) run clean serially (5/5, ~84s). Go is already guarded by its non-gated O(n^2) tripwire (main coverage job) + golden parity test, so coverage is preserved. Documented inline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(security): set persist-credentials false on all ci-tests checkouts (zizmor artipacked) The new benchmarks job (and the pre-existing tests / cross-platform jobs) used actions/checkout with the default persist-credentials, leaving the token in .git/config. The tests job uploads a test-reports artifact, so that is the literal credential-persistence-through-artifacts case zizmor's artipacked audit flags; the others persist creds needlessly. None of these jobs push — they run npm + vitest only — so persist-credentials: false is safe (the packaged-install-smoke job already runs setup-gitnexus this way). All four ci-tests.yml checkouts are now consistent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * bench(scope-capture): unified build-free measure harness for all benchmarked languages Adds a single tsx harness that measures emit<Lang>ScopeCaptures for every language with a pipeline benchmark (go, csharp, rust, php, ruby, cobol): per-language synthetic-DAO scaling (250/800 entities) + an order-independent sha256 fingerprint over each <lang>-* fixture corpus, with a --check mode gating both against baselines.json. It immediately surfaced that csharp, rust, php and ruby still carry the O(matches x rootChildren) findNodeAtRange(tree.rootNode,...) root-walk that was fixed for go (#1915) and python (#1918): scaling ratios 3.13 / 3.31 / 3.04 / 3.07 (vs ~1.0 for the fixed go and cobol). They are flagged known_quadratic in baselines.json so CI guards drift + worsening until each gets the threaded-node fix (following commits). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ruby): linearize scope-capture (thread captured nodes + dedup set) emitRubyScopeCaptures re-derived each match's node via findNodeAtRange(tree. rootNode,...) per match (import / scope.function / declaration.function / heritage / attr / call-arity), and the constructor-return pass ran out.some(...) once per method over the growing output array — two O(n^2) shapes (measured scaling 3.07). Thread the query's captured node (c.node) through a nodeMap and resolve each anchor with a type-guarded lookup (nodeIfType), and precompute the YARD-return dedup keys into a Set. Output byte-identical (capture fingerprint over the ruby-* fixture corpus + DAO unchanged); scaling 3.07 -> 1.11 (linear). 127 ruby resolver tests pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(php): linearize scope-capture (thread captured nodes) emitPhpScopeCaptures re-derived each match's node via findNodeAtRange(tree. rootNode,...) per match (import / scope.function / declaration / call-arity), giving O(matches x rootChildren) ~ O(n^2) (measured scaling 3.04). Thread the query's captured node (c.node) through a nodeMap and resolve each anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python #1918. Output byte-identical (capture fingerprint over the php-* fixture corpus + DAO unchanged); scaling 3.04 -> 1.03 (linear). 205 php resolver tests pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(rust): linearize scope-capture (thread captured nodes) emitRustScopeCaptures re-derived each match's node via findNodeAtRange(tree. rootNode,...) per match (import / scope.function / declaration / type-binding return-hoist / call-arity), giving O(matches x rootChildren) ~ O(n^2) (measured scaling 3.31 — the worst of the four). Thread the query's captured node (c.node) through a nodeMap and resolve each anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python #1918. Output byte-identical (capture fingerprint over the rust-* fixture corpus + DAO unchanged, incl. the impl-block return-type hoist path); scaling 3.31 -> 1.05 (linear). Rust resolver tests pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(csharp): linearize scope-capture (thread captured nodes) emitCsharpScopeCaptures re-derived each match's node via findNodeAtRange(tree. rootNode,...) per match at 7 sites (import / read.member / scope.function / declaration / call-arity / primary-constructor class+record), giving O(matches x rootChildren) ~ O(n^2) (measured scaling 3.13). Thread the query's captured node (c.node) through a nodeMap and resolve each anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python #1918. Output byte-identical (capture fingerprint over the csharp-* fixture corpus + DAO unchanged); scaling 3.13 -> 0.99 (linear). C# resolver tests pass; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(bench): tighten scope-capture budgets to linear + gate all 6 languages in CI All six benchmarked languages now thread the captured node, so update baselines.json: drop known_quadratic and set scaling_budget 1.5 (linear) for csharp/rust/php/ruby (go/cobol already linear). Fingerprints are unchanged — every fix was byte-identical. Wire the unified build-free guard into the benchmarks job: 'node --import tsx bench/scope-capture/measure.mjs --check' asserts the capture fingerprint and linear scaling for go/csharp/rust/php/ruby/cobol on every run. Build-free (no worker pool), so unlike the go pipeline benchmark it is stable in CI. measure --check passes locally for all six (scaling 0.86-1.10). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ingestion): address PR #1918 tri-review — shared nodeIfType, duck-typed guard, docs Tri-review follow-ups (no behavior change — all capture fingerprints + the resolver fingerprint are byte-identical, verified via the bench --check gates): - maintainability (M1): extract the `nodeIfType` helper (copy-pasted into 4 captures.ts files) to ast-helpers.ts as a generic `nodeIfType<T extends SyntaxNode>`. csharp/php keep their local SyntaxNode aliases (used elsewhere); the generic signature accepts them. - P2 (latent): duck-type the `resolvePythonImportTarget` shape-guard instead of `instanceof Set`. The context type was widened to ReadonlySet<string>; an `instanceof Set` check would reject a legitimate non-Set ReadonlySet and silently drop all Python import edges. Now checks `.has` + `[Symbol.iterator]`. - P3 (ruby dedup): document the snapshot-vs-live `out.some`→Set behavior — the one narrow corner (two same-named methods one row apart, both ending in Const.new) where output differs from the pre-PR code, and why the new behavior (emit both) is intended. - harness cross-ref: note in python-scope/measure.mjs that Python's capture scaling is guarded there (not the unified scope-capture harness) so neither is removed assuming the other covers Python. tsc clean; scope-capture --check passes (6 languages, unchanged + linear); resolver fingerprint unchanged; 300 python/ruby/rust tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): golden + O(n^2) tripwire tests for ruby/rust/php/csharp scope-capture Addresses the PR #1918 tri-review test-gap consensus (testing + adversarial + maintainability): the four newly-linearized languages had no committed correctness/scaling lock in the standard unit-test job — only the bench/scope-capture/measure.mjs --check fingerprint, which runs in the separate benchmarks CI job. Per language, mirroring the existing go/python tests: - test/unit/scope-resolution/<lang>/<lang>-captures-golden.test.ts — ORDER- SENSITIVE golden (modeled on go-captures-golden.test.ts; catches emission reordering the order-independent bench fingerprint misses) over the whole lang-resolution/<lang>-* corpus + a 20-entity synthetic DAO, with UPDATE_GOLDEN regeneration. Runs in the normal unit-test job (fast-fail). - test/integration/<lang>-scope-capture-tripwire.test.ts — non-gated O(n^2) regression tripwire (400-entity source, <10s budget), like python's. The ruby golden also pins the snapshot-dedup behavior (two same-named methods both ending in Const.new emit BOTH @type-binding.return bindings — PR #1918 P3), and the rust golden exercises the impl-block return-type hoist path. 41 tests pass; tsc clean. Goldens generated against the (byte-identical) current output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-30 19:44:22 +01:00
Gergő Magyar	ac9a2ee12f	chore(ci): consolidate parity shards and narrow cross-platform matrix (#1798) ... * chore(ci): reduce CI runner-minutes by consolidating parity and narrowing cross-platform Scope-resolution parity previously spawned 9 separate GitHub Actions jobs (one per migrated language), each doing full checkout + npm ci + build for a single test file. Consolidate into one job running scripts/run-parity.ts which loops through all migrated languages sequentially — same coverage, ~45 fewer runner-minutes of redundant setup per PR. Cross-platform (Windows/macOS) previously ran the full 373-file test suite. Narrow to 45 platform-sensitive files (native LadybugDB, process spawning, path separators, worker threads, filesystem behavior). Full suite still runs on Ubuntu with coverage. Also adds 2 missing lbug integration tests (lbug-orphan-sidecar-recovery, lbug-readonly-init) to the sequential lbug-db vitest project where they belong, and rewrites TESTING.md to document all test lanes. * fix: address code review findings on parity and cross-platform scripts - Capture stderr in run-parity.ts (vitest writes diagnostics to stderr) - Lower per-invocation timeout from 5min to 60s to stay within CI job limit - Add --language flag validation (error on missing value) - Add timeout diagnostic to run-cross-platform.ts catch block - Add analyze-wal-checkpoint-failure.test.ts to lbug-db sequential project - Expand cross-platform list: parser-loader, pipeline, pipeline-graph-golden, setup-skills, cli/tool-no-index-stderr (51 files, was 45) * fix: add shell:true for Windows npx resolution and simplify fs import execFileSync('npx', ...) fails with ENOENT on Windows because npx is npx.cmd — shell:true resolves this. Also replaces dynamic await import('fs') with static import, and fixes timeout detection to use err.killed instead of err.code. * fix(ci): raise parity per-invocation timeout to 120s and job timeout to 30min TypeScript and C++ resolver tests take 60-90s on CI runners, exceeding the 60s per-invocation timeout. Raise to 120s. Also bump the job-level timeout from 25 to 30 minutes for margin (realistic total is ~11 min). * fix(ci): raise parity per-invocation timeout to 180s for C++ resolver C++ resolver tests take 130-150s on CI runners due to template metaprogramming, ADL, and SFINAE fixture volume. 120s was still too tight. Realistic total across all 9 languages is ~12 min, well under the 30-min job timeout. * fix(ci): use stdio inherit for parity — no per-invocation timeout Switch from piped stdio with per-invocation timeouts to stdio: 'inherit'. Vitest output streams to CI console in real time, making failures immediately visible. The CI job-level timeout (30 min) is the only guard — no more artificial per-invocation timeouts that cut off slow resolver tests like C++ (which genuinely takes 3+ minutes). --------- Co-authored-by: Test <test@example.com>	2026-05-24 12:10:10 +01:00
Gergő Magyar	d3de5fa5d5	fix(install): materialize vendored grammars to fix Windows EPERM (#1728) (#1729) ... * fix(install): materialize vendored grammars to fix Windows EPERM (#1728) Stop using file: optionalDependencies for tree-sitter-dart/proto/swift, which made npm symlink vendor paths on install and fail on Windows without symlink privileges. Copy vendor trees into node_modules at postinstall instead; keep native builds and #836 vendor hygiene. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(install): atomic materialize swap + fail-soft tests (#1728, #836) Hardens PR #1729 against two issues the original implementation could still hit: 1. Torn-state on rmSync→cpSync. The previous loop deleted the destination before copying. If cpSync threw — the exact Windows EPERM scenario this PR targets — a previously-working grammar was silently wiped. Now we copy to {dest}.materialize-tmp first and renameSync into place, so an interrupted copy leaves the prior materialization intact. 2. Fail-soft try/catch had no test coverage. Adds two POSIX-only tests (chmod 0o555 to deterministically force cpSync to throw) that verify (a) a single grammar failure does not abort the other two, and (b) an existing materialization survives a partial-copy failure. Skipped on Windows where chmod doesn't enforce write restriction; runs on Linux CI. Other test improvements locking in the install-hygiene invariants: - All three vendored grammars (dart/proto/swift) checked, not just dart. - GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 short-circuit is exercised. - Vendor cleanliness (#836): no node_modules/build under vendor/. - Idempotent re-runs (clean overwrite verified via sentinel file). - Missing-vendor warn+continue path now has explicit coverage. - Vendored package manifests asserted to carry no install script or runtime dependencies. - package.json optionalDependencies asserted free of vendored grammars. - package-lock.json assertion tightened from `if (entry !== undefined) { expect(entry.link).not.toBe(true); }` (vacuous when entry is absent, i.e. the expected post-fix state) to `expect(...).toBeUndefined()`. Verified locally: - npx tsc --noEmit: clean - vitest test/unit/materialize-vendor-grammars.test.ts: 8 pass + 2 POSIX-only skipped on Windows - npm pack tarball: no vendor/*/node_modules or vendor/*/build entries - Isolated global install (clean + upgrade + SKIP env) into temp prefix: succeeds; gitnexus --version → 1.6.5; vendor stays clean post-install. * fix(install): address review feedback — Swift parity, atomicity, CI smoke Resolves all findings from the automated production-readiness review on verify/issue-1728-symlink. Swift warning parity (review #2): Add tree-sitter-swift to OPTIONAL_GRAMMARS in src/cli/optional-grammars.ts alongside Dart and Proto. Before this commit, Swift was materialized at postinstall and probed by build-tree-sitter-swift.cjs but the runtime warnMissingOptionalGrammars() never warned when it failed to load — users got silent Swift degradation from the optional-grammars surface (parser-loader's separate unavailableNote only fires on demand). Now the warning path matches the materialize path. README env-var table (review #1): Update the GITNEXUS_SKIP_OPTIONAL_GRAMMARS row at README.md line 248 to list all three vendored grammars (dart, proto, swift). The quick note earlier in the README already mentioned all three; only the table row was stale. Atomicity hardening (review #3): materialize-vendor-grammars.cjs now copies to {dest}.materialize-tmp, renames the existing dest to {dest}.materialize-bak (if present), then renames the partial into dest, then removes the backup. If the partial→dest rename fails (e.g. Windows AV scanner racing the swap), the catch block restores from backup so the previously-materialized grammar is preserved. Closes the narrow torn-state window where the prior implementation could leave dest deleted after rmSync succeeded but renameSync failed. Swift probe docs (review #4): build-tree-sitter-swift.cjs script header rewritten to describe what the script actually does — probe node-gyp-build at install time so missing-prebuild failures surface as install-time warnings instead of first-parse runtime errors. The script does not "activate" anything; the runtime require() in parser-loader does the actual load. Console warning text updated to match ("prebuild probe" not "activation"). Windows packaged-install smoke test (review #5): New CI job `packaged-install-smoke` in .github/workflows/ci-tests.yml matrices on windows-latest and ubuntu-latest. Runs npm pack, installs the produced tarball globally into RUNNER_TEMP, then asserts: * no vendor/*/node_modules or vendor/*/build (#836 invariant) * tree-sitter-{dart,proto,swift} in node_modules are real directories, not junctions/symlinks (#1728 invariant) * gitnexus --version runs against the installed CLI Closes the coverage gap where the existing windows-latest job only ran `npm ci` in the source checkout — exercising postinstall but not the tarball reify step that historically tripped EPERM. Verified locally: npx tsc --noEmit: clean vitest test/unit/materialize-vendor-grammars.test.ts test/unit/cli-commands.test.ts: 18 pass + 2 POSIX-only skipped on Windows prettier + eslint on all changed files: clean * fix(ci): disable credential persistence on packaged-install-smoke checkout GitHub Advanced Security (zizmor artipacked) flagged the new packaged-install-smoke job's actions/checkout step as a potential credential-persistence risk. The job runs `npm pack` + global install and never pushes back, so the GITHUB_TOKEN that checkout would persist in .git/config provides no value and only widens the leak surface (any future artifact-upload step in this job would carry the token). Disable persistence explicitly via `persist-credentials: false` on this job's checkout. Scoped to the new job — pre-existing checkouts above are left unchanged. * fix(ci): use find instead of ls for tarball lookup (SC2012) actionlint shellcheck SC2012 flagged `TARBALL=$(ls gitnexus-*.tgz | head -n1)`. Switch to `find . -maxdepth 1 -name 'gitnexus-*.tgz' -print -quit` which handles non-alphanumeric filenames safely. Also add an explicit empty-result check so the failure mode is a clear error message instead of a silent `npm install -g ""` later. * fix(tests): sabotage vendor src (not partial path) in POSIX fail-soft tests The fail-soft tests in materialize-vendor-grammars.test.ts pre-chmod'd the destination's .materialize-tmp partial directory to 0o555 to force cpSync to throw. After the atomicity rewrite (`fix(install): atomic materialize swap + fail-soft tests`), the materialize script now starts each grammar's loop with `fs.rmSync(partial, { force: true })`, which deletes the chmod'd sabotage before cpSync runs — so cpSync succeeds and the partial is then renamed into dest, leaving the test's `finally` block with no path to chmod back (ENOENT) and the assertion that proto remained unmaterialized failing because it materialized cleanly. Fix: sabotage the *vendor source* directory (which the script reads from but never modifies) by chmod'ing it to 0o000. cpSync then fails on readdir, the catch block fires per-grammar, dart and swift still materialize from their unaffected sources, and the existing-dest preservation test verifies that a sabotaged second-run leaves the prior materialization (and its sentinel file) intact. Tests now pass locally (8 pass + 2 POSIX-only skipped on Windows) and should pass on macOS/Ubuntu CI where the sabotage runs. * fix(tests): restrict fail-soft tests to Linux (macOS Node cpSync abort) Node 22 on macOS aborts the process with `libc++abi: terminating due to uncaught exception filesystem_error` when fs.cpSync hits a source directory it can't read — the abort happens at the C++ filesystem layer and bypasses Node's JS try/catch entirely (nodejs/node#51399). My chmod-0o000-the-source sabotage strategy triggers this SIGABRT on macOS CI before the production script's `try { cpSync } catch` ever runs, so the test sees a child-process crash instead of the fail-soft warning it's verifying. The production script's fail-soft is correct on Linux (where EACCES surfaces as a normal JS exception) and effectively untestable on macOS via permission sabotage. Real installs don't hit this — npm always ships vendor/ with readable permissions — so the macOS gap is a test artifact, not a behavior gap. Restrict the two chmod-based tests to Linux only by replacing `skipOnWin` with `linuxOnly`. Linux CI continues to verify both the one-grammar-fails-others-succeed and existing-materialization-preserved invariants. macOS and Windows runs skip these two scenarios; the other 8 tests still run on every platform. * fix(tests): remove materialize unit tests, rely on CI smoke job The materialize-vendor-grammars.test.ts file has been a recurring source of platform-specific CI noise: - Windows: chmod doesn't enforce read/write restrictions the way POSIX does, so the fail-soft tests had to be skipped there. - macOS Node 22: cpSync against an unreadable source aborts the process with a libc++ filesystem_error (nodejs/node#51399) that bypasses JS try/catch entirely — making the chmod-based fail-soft tests unrunnable on macOS too. - The "vendor-cleanliness" and "idempotency" tests on Windows intermittently flake due to fs.cpSync timing on the GitHub runner. The invariants these tests verified are now covered by stronger, more realistic surfaces: - packaged-install-smoke (ci-tests.yml): runs `npm pack` then `npm install -g ./gitnexus-*.tgz` on windows-latest and ubuntu-latest, then asserts no vendor/*/node_modules, no vendor/*/build (#836), no junctions/symlinks on the materialized grammar directories (#1728), and a working `gitnexus --version`. This is the actual end-user install path. - cli-commands.test.ts (kept, unmodified): asserts package.json declares no `file:` optionalDependencies for vendored grammars, the Swift vendor manifest carries no install script or dependencies, and the postinstall chain runs materialize-vendor-grammars.cjs + build-tree-sitter-swift.cjs. These are static manifest checks — deterministic, fast, no flake risk. Removing the dynamic script-execution tests trades unit-level coverage for end-to-end smoke coverage that actually exercises the `file:` → cpSync change against a real npm install lifecycle, on the platform the fix targets (windows-latest). --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-21 16:47:22 +01:00
Copilot	6f1cfffdd7	fix(security): Harden CI permissions (#1454) ... * Initial plan * chore(security): harden workflow permissions and pin Docker base image digests Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2ddc8f2b-7355-48cf-9a0b-c06df66c3f47 * fix(security): restore permissions: {} on publish + release-candidate workflows These two release-publishing workflows had permissions: {} (the strictest valid form) before PR #1454, which replaced it with permissions: read-all. Every job in both files already declares its own permissions block, so the workflow-level default is only the safety net for future jobs added without one — read-all weakens that net for no benefit. Restore {} and the explanatory comment. Scorecard's TokenPermissions check accepts both forms, so this preserves U9 compliance. * fix(security): narrow permissions: read-all to contents: read on 13 workflows PR #1454 added permissions: read-all to 13 workflows that previously had no top-level permissions block. read-all is Scorecard-compliant but unnecessarily broad — every job in scope only needs contents:read at the workflow level (job-level blocks already grant the writes that any job actually performs). Snapshot of every job in the 13 workflows confirms contents:read is sufficient: - ci.yml: quality/tests/scope-parity have explicit contents:read job blocks; save-pr-meta uses upload-artifact only (no token scopes needed); ci-status is pure shell. - ci-e2e.yml, ci-quality.yml, ci-scope-parity.yml, ci-tests.yml: all jobs do checkout + npm + tsc/vitest/playwright/upload-artifact only; no API token scopes required. - claude.yml, codeql.yml, dependency-review.yml, docker.yml, gitleaks.yml, pr-labeler.yml, trivy.yml, workflow-lint.yml: all jobs already declare their own job-level blocks (security-events:write, pull-requests:write, packages:write, etc.) so the workflow-level default does not gate them. zizmor (--min-severity high) is clean on the resulting tree. Pre-existing medium findings (secrets-inherit, artipacked) are in unrelated workflows and untouched by this commit. scorecard.yml also uses read-all but pre-existed PR #1454 and is deferred to a follow-up PR per the plan's scope boundary. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-05-09 17:58:22 +01:00
Kritik Bangera	040bb7a489	feat: add docker support (#848) ... * feat: add docker support * feat: move docker files to root * feat: add docker build and push workflow * fix: pin docker action SHAs to verified commits Made-with: Cursor * fix: remove redundant --platform=$TARGETPLATFORM from runtime stage Made-with: Cursor * fix: upgrade docker actions to Node.js 24-compatible versions Made-with: Cursor * docs: updated readme * fix: update docker references * fix(docker-server): reject null bytes in resolvePath Defensively harden the path traversal guard by returning null early when the URL contains a null byte, before normalization runs. Made-with: Cursor * fix(docker-server): handle createReadStream errors Attach an error listener before piping so mid-flight read errors (truncated file, permission change) cleanly destroy the response instead of being silently swallowed. Made-with: Cursor * fix(docker-server): replace existsSync with async stat Eliminates the TOCTOU race between the initial stat call and the subsequent existsSync check. Reuses the async stat pattern already in place and removes the now-unused existsSync import. Made-with: Cursor * test(docker-server): add integration tests; fix %00 null-byte bypass Decode the URL before the null-byte check so percent-encoded null bytes (%00) are also rejected with 400 instead of falling through to the SPA fallback. Adds 5 node:test integration tests covering valid assets, SPA fallback, path traversal, null bytes, and 404. Made-with: Cursor * style: fix prettier formatting in docker-server files Made-with: Cursor * fix(docker): wire tests into CI, fix resolvePath separator, correct image namespace - Add `node --test docker-server.test.mjs` step to ci-tests.yml so the path-traversal guard tests run in every CI pass instead of being silently skipped. - Fix resolvePath containment check: `startsWith(root)` would allow sibling directories like `/app/dist-evil/`; now guards with `root + sep` or exact match. - Update docker-compose.yaml default image from `abhigyanpatwari` namespace to `brainifii` to match what docker.yml publishes to GHCR. * fix(docker): update apt-get commands and set user permissions - Modify Dockerfile and Dockerfile.test to include options for apt-get to bypass validity checks during updates. - Set ownership of the /app directory to the 'node' user in the runtime stage for improved security and proper permission handling. * fix(docker): switch to Alpine base images for smaller footprint - Update Dockerfile to use Alpine-based Node.js images for both builder and runtime stages, reducing image size and improving performance. - Replace apt-get commands with apk for package installation in the runtime stage. * fix(docker): update Node.js version in Dockerfile - Change base image from node:20-alpine to node:22-alpine * fix(docker): update Node.js version in Dockerfile to 22-alpine for runtime --------- Co-authored-by: kritik.b <kritik.b@media.net>	2026-04-18 08:39:18 +01:00
dependabot[bot]	3fd4346bcb	chore(deps): bump actions/checkout from 4.3.1 to 6.0.2 (#842) ... Bumps [actions/checkout](https://github.com/actions/checkout) from 4.3.1 to 6.0.2. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.3.1...de0fac2e4500dabe0009e67214ff5f5447ce83dd) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-04-15 13:52:01 +01:00
dependabot[bot]	c2734cd25e	chore(deps): bump actions/upload-artifact from 4.6.2 to 7.0.1 (#838) ... Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.6.2 to 7.0.1. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](ea165f8d65...043fb46d1a) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-04-15 13:37:10 +01:00
Gergő Magyar	fd7fb5bf1f	feat: unify web and cli ingestion pipeline (#536) ... * feat: add server-side ingestion API (POST /api/analyze, SSE progress) Extract core analysis orchestration from CLI into shared run-analyze.ts module. Add server-side analyze endpoints so the web app can trigger ingestion via HTTP instead of running the full pipeline in-browser. New files: - src/core/run-analyze.ts — shared runFullAnalysis() orchestrator - src/server/analyze-job.ts — job manager (single-slot, dedup, SSE events) - src/server/analyze-worker.ts — forked child process (8GB heap, IPC) - src/server/git-clone.ts — shallow clone/pull with SSRF protection API endpoints: - POST /api/analyze — start analysis (returns 202 + jobId) - GET /api/analyze/:jobId — poll job status - GET /api/analyze/:jobId/progress — SSE progress stream Security: URL validation blocks private IPs and non-HTTP schemes. Path validation requires absolute paths. Git stderr not leaked to API. * feat(web): add server-side analyze UI (Phase 2) Add "Analyze on Server" flow to the web app's Server tab so users can trigger server-side ingestion from the browser. On completion, the graph is automatically loaded via the existing connectToServer flow. New files: - AnalyzeProgress.tsx — progress bar with phase label, elapsed time, cancel Modified files: - backend.ts — startAnalyze(), streamAnalyzeProgress() SSE client - DropZone.tsx — analyze URL input + button below Connect section - App.tsx — onServerAnalyze handler wires analyze -> connect flow * feat: add job cancellation, timeout, and child process tracking (Phase 3) - DELETE /api/analyze/:jobId — cancel running analysis (SIGTERM to worker) - 30-minute timeout kills long-running workers automatically - Child process refs tracked in JobManager for cleanup on shutdown - dispose() kills all active children on SIGINT/SIGTERM - Web cancel button now calls server DELETE endpoint - cancelAnalyze() added to web backend client * refactor(web): remove browser ingestion pipeline (Phase 4) Delete 16 duplicated ingestion files, 2 unused service files (git-clone, zip), and tree-sitter parser-loader from gitnexus-web. All ingestion now runs server-side via POST /api/analyze. Deleted (18 files, ~5,000 lines): - core/ingestion/*.ts (16 pipeline processors) - core/tree-sitter/parser-loader.ts (WASM tree-sitter loader) - services/git-clone.ts (isomorphic-git client-side clone) - services/zip.ts (JSZip extraction) Simplified: - DropZone.tsx — server-only (removed ZIP/GitHub tabs) - ingestion.worker.ts — removed runPipeline/runPipelineFromFiles - useAppState.tsx — removed pipeline callbacks - App.tsx — removed handleFileSelect/handleGitClone - main.tsx — removed Buffer polyfill for isomorphic-git - types/pipeline.ts — removed PipelineResult/serialize helpers Kept: cluster-enricher.ts (LLM enrichment, still used by worker) Dependencies now removable: web-tree-sitter, isomorphic-git, @isomorphic-git/lightning-fs, jszip (estimated 3-4MB bundle savings) * refactor(web): sync graph schema from CLI + delete WASM grammars Sync graph/types.ts and lbug/schema.ts from the CLI (source of truth) to the web module so the browser LadybugDB can handle all node and relationship types the server pipeline produces. Synced types: Route, Tool, Section node labels; HANDLES_ROUTE, FETCHES, HANDLES_TOOL, ENTRY_POINT_OF, WRAPS, QUERIES relationship types; description fields on Function/Class/Interface/Method/CodeElement. Deleted: public/wasm/ directory (14 tree-sitter WASM grammars + core). Removed deps: web-tree-sitter, isomorphic-git, @isomorphic-git/lightning-fs, jszip, buffer, @types/jszip (~3-4MB bundle savings). * feat: create gitnexus-shared package for unified type definitions Create a new gitnexus-shared package that is the single source of truth for types shared between the CLI and web modules: - SupportedLanguages enum (15 languages) - Graph types: NodeLabel, NodeProperties, RelationshipType, GraphNode, GraphRelationship - Schema constants: NODE_TABLES, REL_TYPES, REL_TABLE_NAME, EMBEDDING_TABLE_NAME - Pipeline types: PipelinePhase, PipelineProgress Both gitnexus (CLI) and gitnexus-web import from gitnexus-shared via file: dependency. Each package re-exports and extends with platform-specific additions (CLI: KnowledgeGraph with mutation methods; Web: simpler KnowledgeGraph). This ensures types can never drift between packages — adding a new language, node type, or relationship type in gitnexus-shared automatically propagates to both consumers. * refactor: import shared types directly from gitnexus-shared at call sites Replace all re-export patterns with direct imports from gitnexus-shared. 72 files updated across CLI and web: - SupportedLanguages: 49 CLI files now import from 'gitnexus-shared' instead of '../config/supported-languages.js' - GraphNode, GraphRelationship, NodeLabel: 22 CLI + 10 web files now import from 'gitnexus-shared' instead of local re-export wrappers - NODE_TABLES: api.ts imports from 'gitnexus-shared' - PipelineProgress: useAppState.tsx imports from 'gitnexus-shared' Local types.ts files now only define platform-specific KnowledgeGraph (CLI has mutation methods, web has add-only). No more re-exports. * fix: update lock files for gitnexus-shared, remove stale vite polyfills Add gitnexus-shared@1.0.0 to lock files so npm ci succeeds in CI. Remove buffer polyfill and global define from vite.config.ts (isomorphic-git was removed). * fix(security): add write guard to HTTP /api/query, fix CORS proxy bypass - Add isWriteQuery() check to POST /api/query handler — blocks CREATE, DELETE, SET, MERGE, DROP, etc. via HTTP API (guard was only in MCP pool adapter and browser-side, not the HTTP server path) - Extend CYPHER_WRITE_RE with CALL, INSTALL, LOAD keywords - Fix CORS proxy subdomain bypass: endsWith('github.com') allowed 'evil-github.com'. Now requires exact match or '.github.com' suffix * feat(server): enhance /api/search with enrichment, add /api/grep, strip graph content - POST /api/search: add mode param (hybrid|semantic|bm25), server-side enrichment returns connections/cluster/processes per result in one call (collapses 31 sequential HTTP calls to 1 for the agent search tool) - GET /api/grep: regex search across indexed file contents, eliminates need to transfer all file contents to browser - GET /api/graph: strip content field by default (80-95% payload reduction). Use ?includeContent=true for backward compat - Add LRU cache invalidation hook point for future caching * feat(server): add /api/embed endpoint for server-side embedding generation - POST /api/embed: triggers embedding pipeline via onnxruntime-node with JobManager for single-slot concurrency, timeout, and dedup - GET /api/embed/:jobId: poll job status - GET /api/embed/:jobId/progress: SSE stream with heartbeat, event IDs, and X-Accel-Buffering:no header for proxy compatibility - DELETE /api/embed/:jobId: cancel running embedding job - Maps embedding pipeline phases (ready→complete, error→failed) to JobManager status conventions * feat(web): create consolidated BackendClient module Single HTTP client replacing backend.ts, server-connection.ts, and worker HTTP helpers. Includes: - Typed methods: runQuery, search (enriched), grep, readFile, connect - Generic streamSSE<T> utility extracted from analyze progress pattern - BackendError with discriminated code field (network/server/client/timeout) - Embed API: startEmbeddings, streamEmbeddingProgress, cancelEmbeddings - Search with mode param (hybrid|semantic|bm25) and enrichment * refactor(web): rewrite Graph RAG tools for backend-only HTTP queries - Search tool: uses enriched /api/search (1 call replaces 31 sequential queries) - Cypher tool: removes browser-side embedding; {{QUERY_VECTOR}} routes to /api/search with mode:'semantic' instead of local transformers.js - Grep tool: uses /api/grep instead of in-memory fileContents map - Read tool: uses /api/file instead of fileContents map lookup - Impact tool: getCallSiteSnippet now async via /api/file - createGraphRAGTools now accepts GraphRAGBackend interface instead of 7 separate function params + fileContents map - createGraphRAGAgent simplified to (config, backend, context?) - Removed imports: embedder, lbug/schema (replaced with gitnexus-shared) - Net: -205 lines * refactor(web): delete WASM infrastructure, remove 7 packages (-5242 lines) Delete browser-side LadybugDB, embeddings, search, and worker: - gitnexus-web/src/core/lbug/ (adapter, csv-generator, schema, query-result) - gitnexus-web/src/core/embeddings/ (embedder, pipeline, text-gen, types) - gitnexus-web/src/core/search/ (bm25-index, hybrid-search) - gitnexus-web/src/workers/ingestion.worker.ts (828 lines) - gitnexus-web/src/services/server-connection.ts (merged into backend-client) - gitnexus-web/src/types/lbug-wasm.d.ts Remove packages: @ladybugdb/wasm-core, @huggingface/transformers, comlink, minisearch, vite-plugin-wasm, vite-plugin-top-level-await, vite-plugin-static-copy Update vite.config.ts: remove WASM plugins, COOP/COEP headers, worker config, optimizeDeps exclude Update imports: App.tsx, DropZone, Header, AnalyzeProgress, BackendRepoSelector, useBackend → backend-client * refactor(web): replace Worker/Comlink with direct BackendClient calls - useAppState: remove Worker instantiation, Comlink.wrap, apiRef. All queries now go through BackendClient HTTP functions directly. - Agent runs on main thread (I/O-bound LLM streaming, not CPU-bound) - initializeAgent: creates GraphRAGAgent with GraphRAGBackend interface bound to BackendClient methods (runQuery, search, grep, readFile) - startEmbeddings: calls POST /api/embed + SSE progress instead of running browser-side transformers.js pipeline - switchRepo: no longer loads graph into WASM DB or extracts fileContents - App.tsx: handleServerConnect simplified (no fileContents, no loadServerGraph) - Delete old backend.ts (replaced by backend-client.ts) - Net: -396 lines * fix(web): fix await-in-map build error in agent streaming Move dynamic import of AIMessage outside .map() callback to avoid "await can only be used inside an async function" build error. * fix(web): remove stale apiRef references that broke chat functionality sendChatMessage referenced apiRef.current (deleted Worker ref) which would throw TypeError. Replaced with agentRef.current guard since agent now runs on main thread. * fix(server): dispose embedJobManager on shutdown, fix job mutation - Add embedJobManager.dispose() to shutdown handler (was missing, causing cleanup timer to keep Node process alive) - Replace direct job.repoName/status mutation with updateJob() to ensure SSE event emission for initial status change * fix(server): parameterize Cypher, harden grep, unify SSE endpoints - Search enrichment: replace string interpolation with executePrepared() using $nid parameter binding to prevent Cypher injection - Add executePrepared() to core lbug-adapter (prepare/execute pattern) - /api/grep: add 200-char pattern length limit (ReDoS protection), search files on disk instead of loading entire corpus into memory (constant memory usage regardless of repo size) - Extract mountSSEProgress() shared helper for SSE streaming — both analyze and embed endpoints now have consistent heartbeat (30s), event IDs (reconnection support), and X-Accel-Buffering header * refactor(web): remove dead code from Worker-era architecture - Remove loadServerGraph no-op function, interface member, and all consumers - Remove testArrayParams stub and interface member - Remove fileContents state from GraphStateProvider (never populated in server-side architecture) - Remove forceDevice parameter from startEmbeddings (server-side, no device choice) - Replace phantom EmbeddingProgress type with inline { phase, percent } - Replace resolvePathFromContents (needed fileContents Map) with graph-based file path resolution using filePathIndex built from graph nodes - Fix: AI citation grounding ([[file.ts:10]]) now works via graph node lookup instead of broken fileContents-based resolution * fix(web): use streamAgentResponse for full tool_call/reasoning streaming Replace naive agent.stream() loop that only handled content chunks with streamAgentResponse() generator from agent.ts. This properly routes: - reasoning tokens (before/between tool calls) - tool_call events (name, args, status) - tool_result events (completed tool output) - content tokens (final answer after all tools done) Previously the onChunk handler for tool_call/tool_result/reasoning was dead code since the streaming loop only emitted content events. * fix(web): resolve CI type errors from dead code removal - Import GraphNode/GraphRelationship from gitnexus-shared in graph.ts (not re-exported from local types.ts) - Add Route, Tool entries to NODE_COLORS and NODE_SIZES constants - Add PipelineResult type to web types/pipeline.ts - Remove fileContents from CodeReferencesPanel and RightPanel - Remove testArrayParams and forceDevice from EmbeddingStatus - Remove forceDevice args from startEmbeddings() calls in App.tsx - Fix embeddingProgress property accesses for simplified type * fix(ci): add setup-gitnexus-web action, build shared once per job - Remove prepare script from gitnexus-shared (tsc not available during npm ci of consuming packages) - Create .github/actions/setup-gitnexus-web composite action: builds gitnexus-shared then runs npm ci for gitnexus-web - setup-gitnexus action: already builds gitnexus-shared for CLI jobs - ci-quality typecheck-web: uses setup-gitnexus-web (DRY) - ci-e2e: uses setup-gitnexus-web (DRY) - ci-tests: gitnexus-shared already built by setup-gitnexus, just install web deps without rebuilding * fix(ci): use prepare script so gitnexus-shared builds during npm ci Move typescript from devDependencies to dependencies in gitnexus-shared so the prepare script (tsc) works when npm resolves file: deps during npm ci. No GHA modifications needed — npm handles the build lifecycle automatically. Remove manual gitnexus-shared build steps from setup-gitnexus and setup-gitnexus-web actions. * fix(ci): build gitnexus-shared explicitly in setup actions The file: dependency protocol doesn't reliably run prepare scripts because devDependencies aren't installed first. Instead of fragile lifecycle hacks, build gitnexus-shared explicitly in both setup actions: - setup-gitnexus: npm install && npm run build in gitnexus-shared/ - setup-gitnexus-web: same, before npm ci in gitnexus-web/ - ci-tests: shared already built by setup-gitnexus, web just npm ci No prepare script, no dist in git, no typescript as a prod dependency. * fix: remove CALL from CYPHER_WRITE_RE — breaks FTS and vector search CALL is used by read-only procedures: CALL QUERY_FTS_INDEX(...) and CALL QUERY_VECTOR_INDEX(...). Adding it to the write guard blocked all FTS search, causing 3 test failures. The database is opened in read-only mode as defense-in-depth against write procedures via CALL. Keep INSTALL and LOAD in the blocklist (genuinely dangerous). * fix(web): update vercel.json for gitnexus-shared, remove COOP/COEP - Add installCommand that builds gitnexus-shared before installing web deps (Vercel doesn't know about the monorepo file: dependency) - Remove Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers (no longer needed — WASM LadybugDB removed) * fix(web): update tests for deleted modules - Delete csv-generator.test.ts (tests deleted WASM-only csv-generator) - Update security-guards.test.ts: import NODE_TABLES/REL_TYPES from gitnexus-shared instead of deleted src/core/lbug/schema - Update server-connection.test.ts: import normalizeServerUrl from backend-client, remove extractFileContents tests (function deleted) * fix(e2e): remove Server tab click — UI is now server-only The DropZone no longer has ZIP/GitHub/Server tabs (browser ingestion was removed). The server URL input is directly visible on the landing page. Update e2e test to skip the tab click and go straight to input. All 5 e2e tests pass locally. * refactor: use gitnexus-shared for PipelinePhase/PipelineProgress types CLI was duplicating PipelinePhase and PipelineProgress locally instead of importing from gitnexus-shared. Updated all consumers to import directly. Also removed dead code: SerializablePipelineResult, serializePipelineResult(), deserializePipelineResult(). * fix(server): address PR #536 review — security, race conditions, dead code - Fix path traversal in POST /api/analyze: split into isAbsolute + normalize check - Add shared repo lock (activeRepoPaths) preventing concurrent analyze+embed on same repo - Fix 202 response returning actual job.status instead of hardcoded 'queued' - Add 30-minute timeout for embedding jobs (was missing unlike analyze jobs) - Fix DropZone calling startAnalyze without setting backend URL first - Add SSE reconnect with exponential backoff (3 retries) and Last-Event-ID - Fix normalizeServerUrl to return base URL (no /api suffix) — clear contract - Delete dead code: proxy.ts, server-graph-hydration.ts, pipeline.ts re-export barrel - Update LoadingOverlay to import PipelineProgress directly from gitnexus-shared * fix(server): fix repo lock key mismatch and embed cancel race - Use getStoragePath(targetPath) as lock key in analyze handler to match embed handler's entry.storagePath — keys now always align - Guard embed completion: don't overwrite 'failed' with 'complete' when job was cancelled while pipeline was still running - Remove unused jobType parameter from acquireRepoLock - Log backend.init() errors instead of silently swallowing * fix: add gitnexus-shared as a local dependency in package-lock.json * refactor: move language detection to gitnexus-shared, add syntax highlighting for all 15 languages Move getLanguageFromFilename() from CLI to gitnexus-shared with COBOL support added. Add getSyntaxLanguageFromFilename() for Prism-compatible syntax highlighting covering all 15 code languages plus auxiliary formats (json, yaml, markdown, html, css, bash, sql, xml). Refactor CodeReferencesPanel to use shared function instead of a local 30-line switch. Delete dead gitnexus-web/src/config/supported-languages.ts (web already imports SupportedLanguages from gitnexus-shared). * feat(web): add first-time user onboarding with auto server detection Replace the manual "Connect to Server" panel with an automatic onboarding flow that guides first-time users through starting the GitNexus server. Server detection: - useBackend hook polls via setTimeout chain (3s, no overlap) - Page Visibility API pauses polling when tab is hidden - SSE heartbeat (/api/heartbeat) for instant disconnect detection Onboarding UI (OnboardingGuide.tsx): - Step-by-step flow: copy command → run → auto-connect - Smart command: shows `gitnexus serve` in dev, `npx gitnexus@latest serve` in prod - Node.js version auto-detected from package.json via Vite define - Faux terminal windows with copy-to-clipboard, platform tabs, polling indicator Transitions (DropZone.tsx): - Crossfade wrapper with snapshot pattern for smooth phase transitions - Three phases: onboarding → success (1.2s hold) → loading → graph - Auto-recovery: falls back to onboarding if server dies or connect fails Server changes: - GET /api/heartbeat: SSE endpoint for liveness detection - GET /api/info: version, launch context, Node.js version - npm run serve script for local development - app.disable('x-powered-by') hardening * feat(web): add repo analysis UI, SSE heartbeat, and review fixes Repo analysis: - AnalyzeOnboarding: empty-state card when server has zero repos - RepoAnalyzer: GitHub URL + Local Folder tabs with browse button - Header repo dropdown: click project badge to switch repos or analyze new - DropZone 'analyze' phase integrated into Crossfade transitions Reliability fixes from 5-agent review: - Polling: stop scheduling timers when tab hidden, restart on visibility return - Heartbeat: exponential backoff (1s/2s/4s, 3 retries) prevents graph loss on blip - RepoAnalyzer: completion timer tracked in ref, cleaned up on unmount - DropZone: standardized card padding (p-7), heading sizes (text-lg) Accessibility: - prefers-reduced-motion global CSS rule (WCAG 2.3.3) - focus-visible rings on CopyButton - cursor-pointer on all Header buttons - Consistent rounded-xl on all dropdowns Cleanup: - Deleted dead AnalyzeSheet.tsx (219 LOC) and BackendRepoSelector.tsx (89 LOC) - Fixed AnalyzeProgress lucide import (lucide-react → @/lib/lucide-icons) * fix(server): resolve analyze worker fork crash in dev mode The forked analyze worker was crashing immediately with exit code 1 when running via `npm run serve` (tsx). Two issues: 1. Worker path resolved to `analyze-worker.js` but only `.ts` exists in the source directory — the `.js` file is only in `dist/`. 2. On Windows, bare `--import tsx` in execArgv fails because Node's ESM resolver for --import uses the child's CWD, not the parent's node_modules. Windows also rejects raw paths as `d:` is not a valid URL scheme. Fix: detect dev vs prod via `import.meta.url` extension. In dev mode, resolve `tsx/esm` to an absolute `file://` URL via `pathToFileURL()` anchored to the parent's `createRequire` context. This works on all platforms and doesn't depend on the child's CWD or PATH. Also captures child stderr for better crash diagnostics. Verified: `POST /api/analyze` with GitHub URL completes successfully in dev mode (tsx) — status goes from cloning → analyzing → complete. * fix(server): add worker auto-retry, error handling, and crash diagnostics Worker resilience: - Auto-retry up to 2 times with exponential backoff (1s, 2s) on crash - SSE progress shows "Retrying after crash (1/2)..." during retry - Captures child stderr for crash diagnostics in failure message - AnalyzeJob tracks retryCount per job Server error handling: - app.listen wrapped in Promise so EADDRINUSE/EACCES propagate cleanly - serve.ts catches startup errors with friendly messages and exit code 1 - EADDRINUSE gets actionable guidance (stop other process or --port flag) - Global uncaughtException/unhandledRejection handlers prevent silent exits - DEBUG=1 env var shows full stack traces * feat: add e2e tests for onboarding flows, worker retry, and error handling E2E tests (onboarding.spec.ts — 11 tests): - Flow 1: OnboardingGuide shown when server unreachable (6 tests) - Flow 2: Auto-connect with success card, analyze phase for zero repos - Flow 3: Analyze form — GitHub URL validation, Local Folder tab, tab switching - Flow 4: Repo dropdown in exploring view (skipped without live server) Updated server-connect.spec.ts: - Replaced manual Connect button flow with auto-connect waitForGraphLoaded Server resilience: - Worker auto-retry (2 attempts with exponential backoff) on crash - Friendly error messages for serve startup failures (EADDRINUSE etc.) - Global uncaughtException/unhandledRejection handlers prevent silent exits - app.listen wrapped in Promise for proper error propagation * refactor(shared): enforce exhaustive language coverage via Record types Replace the if/else chain in getLanguageFromFilename with two exhaustive Record<SupportedLanguages, ...> maps: - EXTENSION_MAP: every language → its file extensions - SYNTAX_MAP: every language → its Prism syntax identifier Adding a new member to the SupportedLanguages enum without adding it to both maps now produces a TypeScript compile error: Property '[SupportedLanguages.NewLang]' is missing in type... This matches the existing pattern in languages/index.ts (providers table) which already uses `satisfies Record<SupportedLanguages, LanguageProvider>`. Three compile-time enforcement points now exist: 1. EXTENSION_MAP in language-detection.ts (file extensions) 2. SYNTAX_MAP in language-detection.ts (Prism syntax identifiers) 3. providers in languages/index.ts (LanguageProvider instances) * feat(web): load source code from server and scroll to selected line CodeReferencesPanel now fetches file content via GET /api/file when a node is selected, instead of showing "Code not available in memory". - Fetches via readFile() from backend-client when selectedFilePath changes - Shows loading spinner while fetching - After content loads, auto-scrolls to the selected node's startLine - Highlights the selected line range with a cyan left border - Cancels in-flight fetch if selection changes before it completes Also: refactored language-detection.ts to use exhaustive Record types (EXTENSION_MAP and SYNTAX_MAP) so adding a new SupportedLanguages enum member without implementing extensions/syntax is a compile error. * feat: buffered file reading for Code Inspector Server: GET /api/file now supports ?startLine=N&endLine=M for reading a line range instead of the entire file. Returns { content, startLine, endLine, totalLines }. Client: readFile() returns ReadFileResult with metadata. When selecting a symbol (function, class, method), fetches only ±50 lines around the symbol's startLine/endLine instead of the full file. File nodes still fetch the entire file. SyntaxHighlighter startingLineNumber set from the buffer offset so line numbers are correct even for partial reads. * fix: adapt readFile callers to new ReadFileResult return type tools.ts: readFile comes from GraphRAGBackend interface which returns Promise<string> (the adapter in useAppState extracts .content), so revert the { content } destructuring back to plain string assignment. useAppState.tsx: wrap backendReadFile with { repo } options object and extract .content to satisfy the GraphRAGBackend interface. * fix(web): ensure new repos appear in list immediately after analysis Two fixes: 1. DropZone: handleAnalyzeComplete now passes the repoName through to connectToServer so the specific newly-analyzed repo loads — not the server's default first repo. 2. App.tsx: fetchRepos() is now awaited BEFORE handleServerConnect in both the DropZone and Header flows. This ensures the repo list is populated before the exploring view renders, so the new repo appears in the header dropdown immediately without a page reload. * feat: delete repos, re-analyze with force, select after analysis Server — DELETE /api/repo: - Acquires repo lock first (409 if analyze/embed in flight) - Closes LadybugDB, deletes index + clone dir, unregisters, re-inits - Lock released in finally block Server — analyze complete: - backend.init() must succeed before SSE complete fires - If backend.init() fails, job is marked failed (not complete) Web — Header repo dropdown: - Re-analyze: calls POST /api/analyze with force=true, shows spinning icon + inline progress bar via SSE - Delete: acquires lock, aborts any running re-analysis SSE for same repo, refreshes list, switches to next repo - After analysis completes: refreshes repo list, connects to the specific repo by name, loads graph, shows in explorer - Retry with 1.5s backoff on 404 (server may still be reinitializing) Type safety: - err: any → err: unknown + instanceof BackendError in retry loop - Added missing BackendRepo + BackendError imports in App.tsx	2026-03-28 14:07:11 +00:00
John R. Eakin	f0540b33fb	ci: E2E workflow, web typecheck job, pre-commit hook, test suite (#486)	2026-03-24 06:01:08 +00:00
Abhigyan Patwari	60c93d7d4a	feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds (#374) ... * feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds The upstream fix (ladybug-nodejs#1) resolves the child QueryResult lifetime segfault, making .close() safe on all platforms. This removes 6 workaround sites: - Remove `dangerouslyIgnoreUnhandledErrors` from vitest config - Remove platform-conditional .close() guards in global-setup and test helper - Delete test/setup.ts (process._getActiveHandles unref hack) - Replace no-op cleanup in test-indexed-db.ts with real adapter close - Fix pool adapter closeOne() to properly close connections with shared Database refcount guard and orphaned connection handling in checkin() - Update segfault-related comments across the codebase Also bumps @ladybugdb/wasm-core to ^0.15.2 in gitnexus-web for consistency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: keep dangerouslyIgnoreUnhandledErrors for macOS N-API exit crash The N-API destructor ordering crash during worker fork exit on macOS is independent of the QueryResult lifetime fix in 0.15.2. Tests pass, but the exit triggers a crash. Keep the flag with an updated comment explaining the actual cause. Can be removed once LadybugDB fixes all destructor ordering issues upstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: unify test run for single-pass coverage - Update `npm test` to run all tests (unit + integration + lbug-db) via `vitest run` instead of `vitest run test/unit` - Add `test:unit` script for running unit tests only - Remove `ci-integration.yml` — the per-file lbug-db process isolation is no longer needed with `dangerouslyIgnoreUnhandledErrors` and `fileParallelism: false` handling fork exit issues - Update `ci-unit-tests.yml` to run all tests with build + coverage - Simplify `ci.yml` gate (two jobs: quality + tests) - Simplify `ci-report.yml` (single coverage artifact, no merge step) * fix: update cli-commands test for renamed test:all → test:unit script * fix: set USERPROFILE in setup-skills test for Windows compatibility os.homedir() checks USERPROFILE on Windows, not HOME. * fix: add isolate: false to lbug-db project to prevent fork crashes On macOS, N-API destructors crash fork workers on exit. With isolate: true (default), vitest recycles the fork between files, triggering the crash after each file. After several crashes, the remaining lbug-db files never execute. isolate: false keeps all 8 lbug-db files in a single fork — the fork only exits once after all files complete, and that single exit crash is caught by dangerouslyIgnoreUnhandledErrors. * fix: add unique sequence.groupOrder to vitest projects Vitest v4 requires unique groupOrder when projects have different maxWorkers (lbug-db has fileParallelism: false → maxWorkers: 1). * fix: await async close() in global-setup and remove isolate: false global-setup.ts called conn.close() and db.close() without await — these return Promise<void> in @ladybugdb/core 0.15.2. The setup function returned before the DB was fully closed, so vitest forks hit a stale file lock when opening the same DB path, crashing the lbug-db worker before any test ran. isolate: false caused native state corruption after 2-3 open/close cycles in the same fork (vitest-specific, not reproducible in plain Node.js). Without it, each file gets its own module scope and the N-API destructor crash at fork exit is caught by dangerouslyIgnoreUnhandledErrors. Also fixes fire-and-forget close() calls in the pool adapter — try/catch around an async close() never catches rejections; changed to .catch(() => {}) for proper unhandled-rejection prevention. Before: 0/8 lbug-db files ran on macOS CI (fork crash). After: 8/8 pass, 84 files, 3077 tests, zero errors. * fix: update project index references in AGENTS.md and CLAUDE.md to reflect correct symbol counts and relationships * feat: enhance lbug adapter with external database support and write operation validation * feat: create ci-tests workflow for comprehensive test coverage across platforms * ci: move PR report inline to ci.yml, delete ci-report.yml The old ci-report.yml used workflow_run which always runs code from the default branch (main). This meant the PR comment used main's stale report template that still referenced the old unit/integration split architecture — causing "Merge coverage reports" failures. Moving the report inline to ci.yml means it runs from the PR branch and uses the current report template. The report now shows: - per-platform status (Ubuntu/Windows/macOS columns) - unified test counts from the single vitest run - coverage with base branch (main) delta comparison - commit SHA for traceability Also removes the save-pr-meta job since the report no longer needs a separate workflow_run trigger. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>	2026-03-19 08:25:43 +00:00

Renamed from .github/workflows/ci-unit-tests.yml (Browse further)