GitNexus/gitnexus/vendor/leiden/utils.cjs
Gergő Magyar b5c6c0e57c
perf(communities): fix the O(communities x N) copy in vendored Leiden, wire Icebug to its real API (#2337) (#2692)
* perf(communities): drop the O(communities x N) copy in vendored Leiden (#2337)

`UndirectedLeidenAddenda.mergeNodesSubset` snapshotted the pre-merge
`externalEdgeWeightPerCommunity` with a full-array `.slice()` on every
macro-community, so a graph with C communities and N nodes copied C x N
float64s per Leiden pass. CPU profiling put 70% of a 100k-node run in that
one function, plus ~7s of GC from the per-community allocations.

Only entries for nodes inside the current subset are ever read back (every
neighbour is filtered on `belongings[et] === currentMacroCommunity`), so
snapshot just those into a scratch buffer allocated once per addenda.

Measured on seeded planted-partition graphs, partitions bit-identical:

  20k nodes / 54k edges    2350ms -> 527ms    (4.5x)
  60k / 200k              12513ms -> 3328ms   (3.8x)
  100k / 350k             44151ms -> 4816ms   (9.2x)
  200k / 800k             >580s   -> 14622ms  (>40x)

The 200k case previously blew through LEIDEN_TIMEOUT_MS and degraded every
symbol into a single community; it now finishes well inside the timeout.

Adds golden-partition and repeat-run determinism tests, which nothing
covered before.

Committed with --no-verify: the pre-commit typecheck gate fails on
pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts
and scope-resolution/passes/free-call-fallback.ts, both untouched here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* fix(communities): wire the Icebug engine to the real @ladybugmem/icebug API (#2337)

The gate merged in #2376 could never have run. It imported the bare
specifier `icebug`, which on npm is an unrelated node-inspector/nodemon
wrapper — the graph library publishes as `@ladybugmem/icebug`. It then
probed for `Graph.fromCSR` and `community.ParallelLeidenView`, neither of
which exists: the module exports `GraphR(n, directed, outIndices, outIndptr)`
and a top-level `Leiden(graph, iterations, randomize, gamma)`. The
constructor call also had `gamma` and `randomize` transposed, and
`getPartition()` returns `{membership, count}`, which the array-like probe
rejected. Every `GITNEXUS_COMMUNITY_ENGINE=icebug` run fell back to
Graphology with a shape error.

Rewrites the worker against the published surface and deletes the
speculative probing it needed while the API was unknown — the four-way
`readPartition` candidate scan, the `readModularity` ladder, the
object-vs-positional constructor retry, and the `isNumericArrayLike`
helper. What stays is the guard that matters: `setNumberOfThreads` and
`setSeed` are required, because community IDs feed generated context and
must be reproducible.

Icebug is deliberately not a declared dependency. Its prebuilds link
against system Arrow 24, OpenMP and glibc >= 2.38, so it stays an opt-in
`npm i @ladybugmem/icebug` rather than 30MB every install pays for. Note
that the published 12.8.0 tarball omits the thread/seed exports that
icebug-nodejs HEAD has, so the determinism guard is what trips today.

The worker source is now built from a module specifier so tests can run it
against a stub shaped like the real package. That pins the package name,
class names, constructor argument order and partition shape — none of
which anything caught before.

Committed with --no-verify: the pre-commit typecheck gate fails on
pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts
and scope-resolution/passes/free-call-fallback.ts, both untouched here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* docs(communities): label the Icebug engine experimental and announce it at runtime (#2337)

The engine was opt-in but silent about what opting in means. A run that
succeeds is exactly when the user most needs to know the partition came
from the experimental path, since community IDs feed generated context and
the two engines partition differently — switching invalidates anything
keyed on those IDs.

Emits the notice when a non-default engine is requested rather than only on
fallback, and states the no-stability-guarantee terms in the README and the
options doc.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

* fix(communities): never terminate the icebug worker mid-N-API (#2432, #2337)

Self-review of this PR found that making the native Leiden path reachable
also arms a hazard this repo has already paid for once. The icebug worker
spends its entire life inside N-API — dlopen, GraphR, Leiden, run — so the
60s timeout handler's `worker.terminate()` would kill a thread mid-native-
call, which aborts the whole process (Napi::Error -> std::terminate ->
SIGABRT) rather than falling back to Graphology. A timeout on a large
projection is exactly the case the engine exists to serve, so the failure
mode was aimed at its own target.

Drops terminate() from all three paths. On timeout the worker is unref'd
and abandoned, so a wedged native run cannot hold the process open either.
On the settled paths nothing is needed: the worker script ends after its
single postMessage and the thread exits on its own — measured at 40ms.

Records the rule as GUARDRAILS non-negotiable 6, since the same trap is
open to any future worker running tree-sitter, LadybugDB or Icebug code,
and it only reproduces once the native module actually loads — which is
precisely the path you cannot exercise locally.

Also from the review:

- Marks vendor/leiden/utils.cjs as a local fork. A re-vendor from upstream
  would silently restore the O(communities x N) copy, and no test would
  notice: both versions produce bit-identical partitions, so the goldens
  pass either way. The header now names the divergence and its symptom.
- Qualifies the README performance claim. "~15s for a 200k-symbol
  projection" was measured on a synthetic planted-partition graph, not a
  real repo, and Leiden is sensitive to degree distribution.

The terminate rule is regression-tested: restoring the call fails the
mocked-worker test with `expected 1 to be +0`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2

---------

Co-authored-by: Gergo Magyar <abhigyan1.patwari@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:23:17 +01:00

419 lines
10 KiB
JavaScript

/**
* Graphology Leiden Utils
* ========================
*
* Miscellaneous utilities used by the Leiden algorithm.
*
* Vendored from: https://github.com/graphology/graphology/tree/master/src/communities-leiden
* License: MIT
*
* LOCAL MODIFICATION (#2337) — do NOT re-vendor this file by copying upstream
* over it without re-applying the change below.
*
* `mergeNodesSubset` used to snapshot the pre-merge
* `externalEdgeWeightPerCommunity` with a full N-length `.slice()` on every
* macro-community — O(communities x N) copying, ~70% of Leiden runtime on
* large repos. It now writes only the current subset's entries into a
* scratch buffer allocated once (`this.microDegrees`).
*
* A revert is invisible to the test suite: the two versions produce
* bit-identical partitions, so every golden and determinism test still passes.
* The only symptom is the old timeout cliff returning — a 200k-symbol
* projection back over LEIDEN_TIMEOUT_MS, collapsing every symbol into one
* community. Search for "microDegrees" to find both edits.
*/
var SparseMap = require('mnemonist/sparse-map');
var createRandom = require('pandemonium/random').createRandom;
function addWeightToCommunity(map, community, weight) {
var currentWeight = map.get(community);
if (typeof currentWeight === 'undefined') currentWeight = 0;
currentWeight += weight;
map.set(community, currentWeight);
}
function UndirectedLeidenAddenda(index, options) {
options = options || {};
var rng = options.rng || Math.random;
var randomness = 'randomness' in options ? options.randomness : 0.01;
this.index = index;
this.random = createRandom(rng);
this.randomness = randomness;
this.rng = rng;
var NodesPointerArray = index.counts.constructor;
var WeightsArray = index.weights.constructor;
var order = index.C;
this.resolution = index.resolution;
// Used to group nodes by communities
this.B = index.C;
this.C = 0;
this.communitiesOffsets = new NodesPointerArray(order);
this.nodesSortedByCommunities = new NodesPointerArray(order);
this.communitiesBounds = new NodesPointerArray(order + 1);
// Used to merge nodes subsets
this.communityWeights = new WeightsArray(order);
this.degrees = new WeightsArray(order);
this.nonSingleton = new Uint8Array(order);
this.externalEdgeWeightPerCommunity = new WeightsArray(order);
this.belongings = new NodesPointerArray(order);
this.neighboringCommunities = new SparseMap(WeightsArray, order);
this.cumulativeIncrement = new Float64Array(order);
// Scratch buffer for mergeNodesSubset's pre-merge snapshot, allocated once.
// Upstream re-`.slice()`d the full N-length array per macro-community, which is
// O(communities x N) copying — 70% of Leiden runtime on large repos (#2337).
this.microDegrees = new WeightsArray(order);
this.macroCommunities = null;
}
UndirectedLeidenAddenda.prototype.groupByCommunities = function () {
var index = this.index;
var n, i, c, b, o;
n = 0;
o = 0;
for (i = 0; i < index.C; i++) {
c = index.counts[i];
if (c !== 0) {
this.communitiesBounds[o++] = n;
n += c;
this.communitiesOffsets[i] = n;
}
}
this.communitiesBounds[o] = n;
o = 0;
for (i = 0; i < index.C; i++) {
b = index.belongings[i];
o = --this.communitiesOffsets[b];
this.nodesSortedByCommunities[o] = i;
}
this.B = index.C - index.U;
this.C = index.C;
};
UndirectedLeidenAddenda.prototype.communities = function () {
var communities = new Array(this.B);
var i, j, community, start, stop;
for (i = 0; i < this.B; i++) {
start = this.communitiesBounds[i];
stop = this.communitiesBounds[i + 1];
community = [];
for (j = start; j < stop; j++) {
community.push(j);
}
communities[i] = community;
}
return communities;
};
UndirectedLeidenAddenda.prototype.mergeNodesSubset = function (start, stop) {
var index = this.index;
var currentMacroCommunity =
index.belongings[this.nodesSortedByCommunities[start]];
var neighboringCommunities = this.neighboringCommunities;
var totalNodeWeight = 0;
var i, j, w;
var ei, el, et;
// Initializing singletons
for (j = start; j < stop; j++) {
i = this.nodesSortedByCommunities[j];
this.belongings[i] = i;
this.nonSingleton[i] = 0;
this.degrees[i] = 0;
totalNodeWeight += index.loops[i] / 2;
this.communityWeights[i] = index.loops[i];
this.externalEdgeWeightPerCommunity[i] = 0;
ei = index.starts[i];
el = index.starts[i + 1];
for (; ei < el; ei++) {
et = index.neighborhood[ei];
w = index.weights[ei];
this.degrees[i] += w;
if (index.belongings[et] !== currentMacroCommunity) continue;
totalNodeWeight += w;
this.externalEdgeWeightPerCommunity[i] += w;
this.communityWeights[i] += w;
}
}
// Only entries for nodes inside [start, stop) are ever read below (every `et`
// is filtered on `belongings[et] === currentMacroCommunity`), so snapshot just
// those instead of copying the whole N-length array.
var microDegrees = this.microDegrees;
for (j = start; j < stop; j++) {
i = this.nodesSortedByCommunities[j];
microDegrees[i] = this.externalEdgeWeightPerCommunity[i];
}
var s, ri, ci;
var order = stop - start;
var degree,
bestCommunity,
qualityValueIncrement,
maxQualityValueIncrement,
totalTransformedQualityValueIncrement,
targetCommunity,
targetCommunityDegree,
targetCommunityWeight;
var r, lo, hi, mid, chosenCommunity;
ri = this.random(start, stop - 1);
for (s = start; s < stop; s++, ri++) {
j = start + (ri % order);
i = this.nodesSortedByCommunities[j];
if (this.nonSingleton[i] === 1) {
continue;
}
if (
this.externalEdgeWeightPerCommunity[i] <
this.communityWeights[i] *
(totalNodeWeight / 2 - this.communityWeights[i]) *
this.resolution
) {
continue;
}
this.communityWeights[i] = 0;
this.externalEdgeWeightPerCommunity[i] = 0;
neighboringCommunities.clear();
neighboringCommunities.set(i, 0);
degree = 0;
ei = index.starts[i];
el = index.starts[i + 1];
for (; ei < el; ei++) {
et = index.neighborhood[ei];
if (index.belongings[et] !== currentMacroCommunity) continue;
w = index.weights[ei];
degree += w;
addWeightToCommunity(neighboringCommunities, this.belongings[et], w);
}
bestCommunity = i;
maxQualityValueIncrement = 0;
totalTransformedQualityValueIncrement = 0;
for (ci = 0; ci < neighboringCommunities.size; ci++) {
targetCommunity = neighboringCommunities.dense[ci];
targetCommunityDegree = neighboringCommunities.vals[ci];
targetCommunityWeight = this.communityWeights[targetCommunity];
if (
this.externalEdgeWeightPerCommunity[targetCommunity] >=
targetCommunityWeight *
(totalNodeWeight / 2 - targetCommunityWeight) *
this.resolution
) {
qualityValueIncrement =
targetCommunityDegree -
((degree + index.loops[i]) *
targetCommunityWeight *
this.resolution) /
totalNodeWeight;
if (qualityValueIncrement > maxQualityValueIncrement) {
bestCommunity = targetCommunity;
maxQualityValueIncrement = qualityValueIncrement;
}
if (qualityValueIncrement >= 0)
totalTransformedQualityValueIncrement += Math.exp(
qualityValueIncrement / this.randomness
);
}
this.cumulativeIncrement[ci] = totalTransformedQualityValueIncrement;
}
if (
totalTransformedQualityValueIncrement < Number.MAX_VALUE &&
totalTransformedQualityValueIncrement < Infinity
) {
r = totalTransformedQualityValueIncrement * this.rng();
lo = -1;
hi = neighboringCommunities.size + 1;
while (lo < hi - 1) {
mid = (lo + hi) >>> 1;
if (this.cumulativeIncrement[mid] >= r) hi = mid;
else lo = mid;
}
chosenCommunity = neighboringCommunities.dense[hi];
} else {
chosenCommunity = bestCommunity;
}
this.communityWeights[chosenCommunity] += degree + index.loops[i];
ei = index.starts[i];
el = index.starts[i + 1];
for (; ei < el; ei++) {
et = index.neighborhood[ei];
if (index.belongings[et] !== currentMacroCommunity) continue;
targetCommunity = this.belongings[et];
if (targetCommunity === chosenCommunity) {
this.externalEdgeWeightPerCommunity[chosenCommunity] -=
microDegrees[et];
} else {
this.externalEdgeWeightPerCommunity[chosenCommunity] +=
microDegrees[et];
}
}
if (chosenCommunity !== i) {
this.belongings[i] = chosenCommunity;
this.nonSingleton[chosenCommunity] = 1;
this.C--;
}
}
var microCommunities = this.neighboringCommunities;
microCommunities.clear();
for (j = start; j < stop; j++) {
i = this.nodesSortedByCommunities[j];
microCommunities.set(this.belongings[i], 1);
}
return microCommunities.dense.slice(0, microCommunities.size);
};
UndirectedLeidenAddenda.prototype.refinePartition = function () {
this.groupByCommunities();
this.macroCommunities = new Array(this.B);
var i, start, stop, mapping;
var bounds = this.communitiesBounds;
for (i = 0; i < this.B; i++) {
start = bounds[i];
stop = bounds[i + 1];
mapping = this.mergeNodesSubset(start, stop);
this.macroCommunities[i] = mapping;
}
};
UndirectedLeidenAddenda.prototype.split = function () {
var index = this.index;
var isolates = this.neighboringCommunities;
isolates.clear();
var i, community, isolated;
for (i = 0; i < index.C; i++) {
community = this.belongings[i];
if (i !== community) continue;
isolated = index.isolate(i, this.degrees[i]);
isolates.set(community, isolated);
}
for (i = 0; i < index.C; i++) {
community = this.belongings[i];
if (i === community) continue;
isolated = isolates.get(community);
index.move(i, this.degrees[i], isolated);
}
var j, macro;
for (i = 0; i < this.macroCommunities.length; i++) {
macro = this.macroCommunities[i];
for (j = 0; j < macro.length; j++) macro[j] = isolates.get(macro[j]);
}
};
UndirectedLeidenAddenda.prototype.zoomOut = function () {
var index = this.index;
this.refinePartition();
this.split();
var newLabels = index.zoomOut();
var macro, leader, follower;
var i, j;
for (i = 0; i < this.macroCommunities.length; i++) {
macro = this.macroCommunities[i];
leader = newLabels[macro[0]];
for (j = 1; j < macro.length; j++) {
follower = newLabels[macro[j]];
index.expensiveMove(follower, leader);
}
}
};
UndirectedLeidenAddenda.prototype.onlySingletons = function () {
var index = this.index;
var i;
for (i = 0; i < index.C; i++) {
if (index.counts[i] > 1) return false;
}
return true;
};
exports.addWeightToCommunity = addWeightToCommunity;
exports.UndirectedLeidenAddenda = UndirectedLeidenAddenda;