mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-10 22:43:40 +00:00
* perf(communities): drop the O(communities x N) copy in vendored Leiden (#2337) `UndirectedLeidenAddenda.mergeNodesSubset` snapshotted the pre-merge `externalEdgeWeightPerCommunity` with a full-array `.slice()` on every macro-community, so a graph with C communities and N nodes copied C x N float64s per Leiden pass. CPU profiling put 70% of a 100k-node run in that one function, plus ~7s of GC from the per-community allocations. Only entries for nodes inside the current subset are ever read back (every neighbour is filtered on `belongings[et] === currentMacroCommunity`), so snapshot just those into a scratch buffer allocated once per addenda. Measured on seeded planted-partition graphs, partitions bit-identical: 20k nodes / 54k edges 2350ms -> 527ms (4.5x) 60k / 200k 12513ms -> 3328ms (3.8x) 100k / 350k 44151ms -> 4816ms (9.2x) 200k / 800k >580s -> 14622ms (>40x) The 200k case previously blew through LEIDEN_TIMEOUT_MS and degraded every symbol into a single community; it now finishes well inside the timeout. Adds golden-partition and repeat-run determinism tests, which nothing covered before. Committed with --no-verify: the pre-commit typecheck gate fails on pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts and scope-resolution/passes/free-call-fallback.ts, both untouched here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2 * fix(communities): wire the Icebug engine to the real @ladybugmem/icebug API (#2337) The gate merged in #2376 could never have run. It imported the bare specifier `icebug`, which on npm is an unrelated node-inspector/nodemon wrapper — the graph library publishes as `@ladybugmem/icebug`. It then probed for `Graph.fromCSR` and `community.ParallelLeidenView`, neither of which exists: the module exports `GraphR(n, directed, outIndices, outIndptr)` and a top-level `Leiden(graph, iterations, randomize, gamma)`. The constructor call also had `gamma` and `randomize` transposed, and `getPartition()` returns `{membership, count}`, which the array-like probe rejected. Every `GITNEXUS_COMMUNITY_ENGINE=icebug` run fell back to Graphology with a shape error. Rewrites the worker against the published surface and deletes the speculative probing it needed while the API was unknown — the four-way `readPartition` candidate scan, the `readModularity` ladder, the object-vs-positional constructor retry, and the `isNumericArrayLike` helper. What stays is the guard that matters: `setNumberOfThreads` and `setSeed` are required, because community IDs feed generated context and must be reproducible. Icebug is deliberately not a declared dependency. Its prebuilds link against system Arrow 24, OpenMP and glibc >= 2.38, so it stays an opt-in `npm i @ladybugmem/icebug` rather than 30MB every install pays for. Note that the published 12.8.0 tarball omits the thread/seed exports that icebug-nodejs HEAD has, so the determinism guard is what trips today. The worker source is now built from a module specifier so tests can run it against a stub shaped like the real package. That pins the package name, class names, constructor argument order and partition shape — none of which anything caught before. Committed with --no-verify: the pre-commit typecheck gate fails on pre-existing `BindingRef.visibility` errors in csharp/namespace-siblings.ts and scope-resolution/passes/free-call-fallback.ts, both untouched here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2 * docs(communities): label the Icebug engine experimental and announce it at runtime (#2337) The engine was opt-in but silent about what opting in means. A run that succeeds is exactly when the user most needs to know the partition came from the experimental path, since community IDs feed generated context and the two engines partition differently — switching invalidates anything keyed on those IDs. Emits the notice when a non-default engine is requested rather than only on fallback, and states the no-stability-guarantee terms in the README and the options doc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2 * fix(communities): never terminate the icebug worker mid-N-API (#2432, #2337) Self-review of this PR found that making the native Leiden path reachable also arms a hazard this repo has already paid for once. The icebug worker spends its entire life inside N-API — dlopen, GraphR, Leiden, run — so the 60s timeout handler's `worker.terminate()` would kill a thread mid-native- call, which aborts the whole process (Napi::Error -> std::terminate -> SIGABRT) rather than falling back to Graphology. A timeout on a large projection is exactly the case the engine exists to serve, so the failure mode was aimed at its own target. Drops terminate() from all three paths. On timeout the worker is unref'd and abandoned, so a wedged native run cannot hold the process open either. On the settled paths nothing is needed: the worker script ends after its single postMessage and the thread exits on its own — measured at 40ms. Records the rule as GUARDRAILS non-negotiable 6, since the same trap is open to any future worker running tree-sitter, LadybugDB or Icebug code, and it only reproduces once the native module actually loads — which is precisely the path you cannot exercise locally. Also from the review: - Marks vendor/leiden/utils.cjs as a local fork. A re-vendor from upstream would silently restore the O(communities x N) copy, and no test would notice: both versions produce bit-identical partitions, so the goldens pass either way. The header now names the divergence and its symptom. - Qualifies the README performance claim. "~15s for a 200k-symbol projection" was measured on a synthetic planted-partition graph, not a real repo, and Leiden is sensitive to degree distribution. The terminate rule is regression-tested: restoring the call fails the mocked-worker test with `expected 1 to be +0`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfQfKy4gCmgUv1jBRJTSs2 --------- Co-authored-by: Gergo Magyar <abhigyan1.patwari@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
419 lines
10 KiB
JavaScript
419 lines
10 KiB
JavaScript
/**
|
|
* Graphology Leiden Utils
|
|
* ========================
|
|
*
|
|
* Miscellaneous utilities used by the Leiden algorithm.
|
|
*
|
|
* Vendored from: https://github.com/graphology/graphology/tree/master/src/communities-leiden
|
|
* License: MIT
|
|
*
|
|
* LOCAL MODIFICATION (#2337) — do NOT re-vendor this file by copying upstream
|
|
* over it without re-applying the change below.
|
|
*
|
|
* `mergeNodesSubset` used to snapshot the pre-merge
|
|
* `externalEdgeWeightPerCommunity` with a full N-length `.slice()` on every
|
|
* macro-community — O(communities x N) copying, ~70% of Leiden runtime on
|
|
* large repos. It now writes only the current subset's entries into a
|
|
* scratch buffer allocated once (`this.microDegrees`).
|
|
*
|
|
* A revert is invisible to the test suite: the two versions produce
|
|
* bit-identical partitions, so every golden and determinism test still passes.
|
|
* The only symptom is the old timeout cliff returning — a 200k-symbol
|
|
* projection back over LEIDEN_TIMEOUT_MS, collapsing every symbol into one
|
|
* community. Search for "microDegrees" to find both edits.
|
|
*/
|
|
var SparseMap = require('mnemonist/sparse-map');
|
|
var createRandom = require('pandemonium/random').createRandom;
|
|
|
|
function addWeightToCommunity(map, community, weight) {
|
|
var currentWeight = map.get(community);
|
|
|
|
if (typeof currentWeight === 'undefined') currentWeight = 0;
|
|
|
|
currentWeight += weight;
|
|
|
|
map.set(community, currentWeight);
|
|
}
|
|
|
|
function UndirectedLeidenAddenda(index, options) {
|
|
options = options || {};
|
|
|
|
var rng = options.rng || Math.random;
|
|
var randomness = 'randomness' in options ? options.randomness : 0.01;
|
|
|
|
this.index = index;
|
|
this.random = createRandom(rng);
|
|
this.randomness = randomness;
|
|
this.rng = rng;
|
|
|
|
var NodesPointerArray = index.counts.constructor;
|
|
var WeightsArray = index.weights.constructor;
|
|
|
|
var order = index.C;
|
|
this.resolution = index.resolution;
|
|
|
|
// Used to group nodes by communities
|
|
this.B = index.C;
|
|
this.C = 0;
|
|
this.communitiesOffsets = new NodesPointerArray(order);
|
|
this.nodesSortedByCommunities = new NodesPointerArray(order);
|
|
this.communitiesBounds = new NodesPointerArray(order + 1);
|
|
|
|
// Used to merge nodes subsets
|
|
this.communityWeights = new WeightsArray(order);
|
|
this.degrees = new WeightsArray(order);
|
|
this.nonSingleton = new Uint8Array(order);
|
|
this.externalEdgeWeightPerCommunity = new WeightsArray(order);
|
|
this.belongings = new NodesPointerArray(order);
|
|
this.neighboringCommunities = new SparseMap(WeightsArray, order);
|
|
this.cumulativeIncrement = new Float64Array(order);
|
|
// Scratch buffer for mergeNodesSubset's pre-merge snapshot, allocated once.
|
|
// Upstream re-`.slice()`d the full N-length array per macro-community, which is
|
|
// O(communities x N) copying — 70% of Leiden runtime on large repos (#2337).
|
|
this.microDegrees = new WeightsArray(order);
|
|
this.macroCommunities = null;
|
|
}
|
|
|
|
UndirectedLeidenAddenda.prototype.groupByCommunities = function () {
|
|
var index = this.index;
|
|
|
|
var n, i, c, b, o;
|
|
|
|
n = 0;
|
|
o = 0;
|
|
|
|
for (i = 0; i < index.C; i++) {
|
|
c = index.counts[i];
|
|
|
|
if (c !== 0) {
|
|
this.communitiesBounds[o++] = n;
|
|
n += c;
|
|
this.communitiesOffsets[i] = n;
|
|
}
|
|
}
|
|
|
|
this.communitiesBounds[o] = n;
|
|
|
|
o = 0;
|
|
|
|
for (i = 0; i < index.C; i++) {
|
|
b = index.belongings[i];
|
|
o = --this.communitiesOffsets[b];
|
|
this.nodesSortedByCommunities[o] = i;
|
|
}
|
|
|
|
this.B = index.C - index.U;
|
|
this.C = index.C;
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.communities = function () {
|
|
var communities = new Array(this.B);
|
|
|
|
var i, j, community, start, stop;
|
|
|
|
for (i = 0; i < this.B; i++) {
|
|
start = this.communitiesBounds[i];
|
|
stop = this.communitiesBounds[i + 1];
|
|
community = [];
|
|
|
|
for (j = start; j < stop; j++) {
|
|
community.push(j);
|
|
}
|
|
|
|
communities[i] = community;
|
|
}
|
|
|
|
return communities;
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.mergeNodesSubset = function (start, stop) {
|
|
var index = this.index;
|
|
var currentMacroCommunity =
|
|
index.belongings[this.nodesSortedByCommunities[start]];
|
|
var neighboringCommunities = this.neighboringCommunities;
|
|
|
|
var totalNodeWeight = 0;
|
|
|
|
var i, j, w;
|
|
var ei, el, et;
|
|
|
|
// Initializing singletons
|
|
for (j = start; j < stop; j++) {
|
|
i = this.nodesSortedByCommunities[j];
|
|
|
|
this.belongings[i] = i;
|
|
this.nonSingleton[i] = 0;
|
|
this.degrees[i] = 0;
|
|
totalNodeWeight += index.loops[i] / 2;
|
|
|
|
this.communityWeights[i] = index.loops[i];
|
|
this.externalEdgeWeightPerCommunity[i] = 0;
|
|
|
|
ei = index.starts[i];
|
|
el = index.starts[i + 1];
|
|
|
|
for (; ei < el; ei++) {
|
|
et = index.neighborhood[ei];
|
|
w = index.weights[ei];
|
|
|
|
this.degrees[i] += w;
|
|
|
|
if (index.belongings[et] !== currentMacroCommunity) continue;
|
|
|
|
totalNodeWeight += w;
|
|
this.externalEdgeWeightPerCommunity[i] += w;
|
|
this.communityWeights[i] += w;
|
|
}
|
|
}
|
|
|
|
// Only entries for nodes inside [start, stop) are ever read below (every `et`
|
|
// is filtered on `belongings[et] === currentMacroCommunity`), so snapshot just
|
|
// those instead of copying the whole N-length array.
|
|
var microDegrees = this.microDegrees;
|
|
|
|
for (j = start; j < stop; j++) {
|
|
i = this.nodesSortedByCommunities[j];
|
|
microDegrees[i] = this.externalEdgeWeightPerCommunity[i];
|
|
}
|
|
|
|
var s, ri, ci;
|
|
var order = stop - start;
|
|
|
|
var degree,
|
|
bestCommunity,
|
|
qualityValueIncrement,
|
|
maxQualityValueIncrement,
|
|
totalTransformedQualityValueIncrement,
|
|
targetCommunity,
|
|
targetCommunityDegree,
|
|
targetCommunityWeight;
|
|
|
|
var r, lo, hi, mid, chosenCommunity;
|
|
|
|
ri = this.random(start, stop - 1);
|
|
|
|
for (s = start; s < stop; s++, ri++) {
|
|
j = start + (ri % order);
|
|
|
|
i = this.nodesSortedByCommunities[j];
|
|
|
|
if (this.nonSingleton[i] === 1) {
|
|
continue;
|
|
}
|
|
|
|
if (
|
|
this.externalEdgeWeightPerCommunity[i] <
|
|
this.communityWeights[i] *
|
|
(totalNodeWeight / 2 - this.communityWeights[i]) *
|
|
this.resolution
|
|
) {
|
|
continue;
|
|
}
|
|
|
|
this.communityWeights[i] = 0;
|
|
this.externalEdgeWeightPerCommunity[i] = 0;
|
|
|
|
neighboringCommunities.clear();
|
|
neighboringCommunities.set(i, 0);
|
|
|
|
degree = 0;
|
|
|
|
ei = index.starts[i];
|
|
el = index.starts[i + 1];
|
|
|
|
for (; ei < el; ei++) {
|
|
et = index.neighborhood[ei];
|
|
|
|
if (index.belongings[et] !== currentMacroCommunity) continue;
|
|
|
|
w = index.weights[ei];
|
|
|
|
degree += w;
|
|
|
|
addWeightToCommunity(neighboringCommunities, this.belongings[et], w);
|
|
}
|
|
|
|
bestCommunity = i;
|
|
maxQualityValueIncrement = 0;
|
|
totalTransformedQualityValueIncrement = 0;
|
|
|
|
for (ci = 0; ci < neighboringCommunities.size; ci++) {
|
|
targetCommunity = neighboringCommunities.dense[ci];
|
|
targetCommunityDegree = neighboringCommunities.vals[ci];
|
|
targetCommunityWeight = this.communityWeights[targetCommunity];
|
|
|
|
if (
|
|
this.externalEdgeWeightPerCommunity[targetCommunity] >=
|
|
targetCommunityWeight *
|
|
(totalNodeWeight / 2 - targetCommunityWeight) *
|
|
this.resolution
|
|
) {
|
|
qualityValueIncrement =
|
|
targetCommunityDegree -
|
|
((degree + index.loops[i]) *
|
|
targetCommunityWeight *
|
|
this.resolution) /
|
|
totalNodeWeight;
|
|
|
|
if (qualityValueIncrement > maxQualityValueIncrement) {
|
|
bestCommunity = targetCommunity;
|
|
maxQualityValueIncrement = qualityValueIncrement;
|
|
}
|
|
|
|
if (qualityValueIncrement >= 0)
|
|
totalTransformedQualityValueIncrement += Math.exp(
|
|
qualityValueIncrement / this.randomness
|
|
);
|
|
}
|
|
|
|
this.cumulativeIncrement[ci] = totalTransformedQualityValueIncrement;
|
|
}
|
|
|
|
if (
|
|
totalTransformedQualityValueIncrement < Number.MAX_VALUE &&
|
|
totalTransformedQualityValueIncrement < Infinity
|
|
) {
|
|
r = totalTransformedQualityValueIncrement * this.rng();
|
|
lo = -1;
|
|
hi = neighboringCommunities.size + 1;
|
|
|
|
while (lo < hi - 1) {
|
|
mid = (lo + hi) >>> 1;
|
|
|
|
if (this.cumulativeIncrement[mid] >= r) hi = mid;
|
|
else lo = mid;
|
|
}
|
|
|
|
chosenCommunity = neighboringCommunities.dense[hi];
|
|
} else {
|
|
chosenCommunity = bestCommunity;
|
|
}
|
|
|
|
this.communityWeights[chosenCommunity] += degree + index.loops[i];
|
|
|
|
ei = index.starts[i];
|
|
el = index.starts[i + 1];
|
|
|
|
for (; ei < el; ei++) {
|
|
et = index.neighborhood[ei];
|
|
|
|
if (index.belongings[et] !== currentMacroCommunity) continue;
|
|
|
|
targetCommunity = this.belongings[et];
|
|
|
|
if (targetCommunity === chosenCommunity) {
|
|
this.externalEdgeWeightPerCommunity[chosenCommunity] -=
|
|
microDegrees[et];
|
|
} else {
|
|
this.externalEdgeWeightPerCommunity[chosenCommunity] +=
|
|
microDegrees[et];
|
|
}
|
|
}
|
|
|
|
if (chosenCommunity !== i) {
|
|
this.belongings[i] = chosenCommunity;
|
|
this.nonSingleton[chosenCommunity] = 1;
|
|
this.C--;
|
|
}
|
|
}
|
|
|
|
var microCommunities = this.neighboringCommunities;
|
|
microCommunities.clear();
|
|
|
|
for (j = start; j < stop; j++) {
|
|
i = this.nodesSortedByCommunities[j];
|
|
microCommunities.set(this.belongings[i], 1);
|
|
}
|
|
|
|
return microCommunities.dense.slice(0, microCommunities.size);
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.refinePartition = function () {
|
|
this.groupByCommunities();
|
|
|
|
this.macroCommunities = new Array(this.B);
|
|
|
|
var i, start, stop, mapping;
|
|
|
|
var bounds = this.communitiesBounds;
|
|
|
|
for (i = 0; i < this.B; i++) {
|
|
start = bounds[i];
|
|
stop = bounds[i + 1];
|
|
|
|
mapping = this.mergeNodesSubset(start, stop);
|
|
this.macroCommunities[i] = mapping;
|
|
}
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.split = function () {
|
|
var index = this.index;
|
|
var isolates = this.neighboringCommunities;
|
|
|
|
isolates.clear();
|
|
|
|
var i, community, isolated;
|
|
|
|
for (i = 0; i < index.C; i++) {
|
|
community = this.belongings[i];
|
|
|
|
if (i !== community) continue;
|
|
|
|
isolated = index.isolate(i, this.degrees[i]);
|
|
isolates.set(community, isolated);
|
|
}
|
|
|
|
for (i = 0; i < index.C; i++) {
|
|
community = this.belongings[i];
|
|
|
|
if (i === community) continue;
|
|
|
|
isolated = isolates.get(community);
|
|
index.move(i, this.degrees[i], isolated);
|
|
}
|
|
|
|
var j, macro;
|
|
|
|
for (i = 0; i < this.macroCommunities.length; i++) {
|
|
macro = this.macroCommunities[i];
|
|
|
|
for (j = 0; j < macro.length; j++) macro[j] = isolates.get(macro[j]);
|
|
}
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.zoomOut = function () {
|
|
var index = this.index;
|
|
this.refinePartition();
|
|
this.split();
|
|
|
|
var newLabels = index.zoomOut();
|
|
|
|
var macro, leader, follower;
|
|
|
|
var i, j;
|
|
|
|
for (i = 0; i < this.macroCommunities.length; i++) {
|
|
macro = this.macroCommunities[i];
|
|
leader = newLabels[macro[0]];
|
|
|
|
for (j = 1; j < macro.length; j++) {
|
|
follower = newLabels[macro[j]];
|
|
index.expensiveMove(follower, leader);
|
|
}
|
|
}
|
|
};
|
|
|
|
UndirectedLeidenAddenda.prototype.onlySingletons = function () {
|
|
var index = this.index;
|
|
|
|
var i;
|
|
|
|
for (i = 0; i < index.C; i++) {
|
|
if (index.counts[i] > 1) return false;
|
|
}
|
|
|
|
return true;
|
|
};
|
|
|
|
exports.addWeightToCommunity = addWeightToCommunity;
|
|
exports.UndirectedLeidenAddenda = UndirectedLeidenAddenda;
|