GitNexus/gitnexus/scripts/assert-publish-fts-coverage.cjs
Gergő Magyar 21a52af1d4
fix(lbug): ship FTS per-platform and recover in-place native aborts (#3274)
* fix(lbug): pin Ladybug core so Dependabot cannot ship a skewed FTS artifact

The extension version is a separate upstream constant. Ignore daily core bumps and fail the pairing gate when the committed manifest does not name the installed core.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): make doctor and CI FTS gates resolve the packaged artifact

Doctor and the REQUIRE_FTS file gates still treated an empty ~/.lbdb as
unavailable, which would turn three CI jobs red once analyze stops
installing into that tree.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name native-abort and tuple-missing so analyze cannot mis-advise

The CLI summary's trailing else treated every unknown skip reason as a
missing extension. New crash and platform causes must get their own
remedies, not a network-install hint.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): delete the dead read-path FTS index create

ensureFTSIndex had no production callers and swallowed read-only
CREATE_FTS_INDEX failures, which hid the only signal that a reader
tried to write.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): vendor per-platform FTS artifacts so analyze needs no host install

Keyword search depended on a CDN fetch into ~/.lbdb. Shipping the five
published tuples inside the package makes air-gapped and ignore-scripts
installs load the same artifact the publish gate checksums.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): load the packaged FTS artifact before any network install

Analyze still required a CDN fetch into ~/.lbdb even when the package
already shipped the file. FTS now path-loads the vendored tuple first
and records source labels so a later truncated home copy cannot steal
the diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): diagnose a core/extension version skew instead of a missing runtime

A structurally valid FTS artifact whose path version disagrees with the
packaged pin must name both versions, not prescribe VC++ or OpenSSL.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): stamp an FTS phase so repair stays usable after an in-place abort

A native CREATE_FTS_INDEX abort leaves no skip reason; the next run infers
it from the dirty flag, and --repair-fts must not treat that phase as a
half-written graph.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): park an in-place FTS crash WAL without wiping the graph

An FTS abort after a successful checkpoint must reopen the live index on
macOS, Windows, and Linux. Staging never parks the live WAL; readers keep
today's large-WAL refusal.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): refuse read-only opens of an FTS-poisoned WAL

MCP and serve cannot repair a leftover in-place abort. Fail before the
native open and name --repair-fts, on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name a vendor-neutral Windows OpenSSL prerequisite

OQ1 is unanswered here so GitNexus does not ship OpenSSL DLLs. Windows
FTS now asks for a system OpenSSL 3 runtime instead of Git Bash PATH.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): inject the FTS vendor root and redact it on HTTP and MCP

Path-loaded artifacts no longer vary with HOME. Tests pass an injected
vendor tree and assert search warnings never leak a filesystem path.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): document load-only as the global FTS install default

Analyze still overrides to auto. Packaged per-platform artifacts load
before any network install on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): format the FTS install-policy README table

Prettier does not run on Markdown in pre-commit, so the U10 table wrap
needs its own formatting commit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): skip FTS CREATE after a persisted native abort

A recovered analyze run was retrying CREATE_FTS_INDEX from skipReason
alone. Keep that skip until --repair-fts, fail closed on unsupported
tuples, and honor the checkpoint warrant for park/repair.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): honor checkpoint flushed warrant and align FTS tests with packaged vendor

A no-op CHECKPOINT must not satisfy the FTS park warrant, and CI still asserted HOME-only FTS isolation after analyze started path-LOADing the packaged artifact.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): accept a nonempty incremental write set in the #2790 recovery check

FTS-phase recovery can incremental-add files (changed=0, added=1). That is not the #2790 empty-diff wipe skip.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): compare FTS home versions to the core pin and tighten the publish filename gate

Ladybug's ~/.lbdb/extension directory is the runtime/core version; treating it as the artifact version false-diagnosed skew. The publish guard now rejects a path-escaping filename the same way the fetch script does.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): seed FTS e2e fixtures from the packaged vendor artifact

A machine with no ~/.lbdb copy should still run the vendor-survivorship cases; the seed no longer depends on HOME or a network install.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Keep in-place FTS abort evidence after persist so a second CREATE abort
cannot fail-open readers, and close the CLI, loader, embed, and e2e gaps
the review called out.

Note: full npm test hit Ladybug worker-pool startup failures under memory
pressure; tsc and 180 targeted unit tests passed.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Run the vendored-path symlink guard on the OS matrix, put e2e HOME
fixtures on Ladybug's real extension layout, pin the embed crash-WAL
gate before the writable open, and let analyze writers park through
missing-shadow recovery.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): keep --repair-fts CI green after vendored-first FTS

Never-installed warning fixtures must not inspect a packaged vendor binary, and a failed dirty restamp must not abort an otherwise successful --repair-fts run.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(cli): give the #1169 analyze e2e the same 90s Windows budget as its sibling

The first #1169 persist-meta case was still on a 60s spawn/it budget and was killed banner-only on windows-latest after the FTS warning fixture no longer failed the shard first.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ci): reweight Windows shards after the FTS e2e grew

Vendored-first HOME fixtures pushed fts-extension-e2e to ~6 minutes on windows-latest, so the old 146s weight packed it with skills-e2e and blew the 20-minute watchdog.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-14 08:52:24 +01:00

246 lines
7.4 KiB
JavaScript

#!/usr/bin/env node
/**
* Publish guard: core↔extension pairing plus vendored FTS artifact integrity.
*
* Does not shell out to `npm pack` (prepack re-entrancy; see the grammar gate).
* Checksums and the `files` allow-list are asserted as pure predicates so a
* future lean-publish narrowing cannot drop the artifacts silently.
*/
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const WIN32_ARM64 = 'win32-arm64';
const SAFE_FILENAME = /^[\w.-]+\.lbug_extension$/;
const REQUIRED_SUPPORTED_TUPLES = [
'linux-x64',
'linux-arm64',
'darwin-x64',
'darwin-arm64',
'win32-x64',
];
/**
* Pure pairing core (exported for tests). Returns human-readable problem
* strings; an empty array means the core↔extension pin is consistent.
*/
const EXACT_CORE_PIN = /^\d+\.\d+\.\d+$/;
function findPairingProblems({
installedCoreVersion,
manifestCoreVersion,
manifestExtensionVersion,
}) {
const problems = [];
const installed = String(installedCoreVersion ?? '');
const pinned = String(manifestCoreVersion ?? '');
if (!EXACT_CORE_PIN.test(installed) || !EXACT_CORE_PIN.test(pinned)) {
problems.push(
`core pin must be exact x.y.z: installed '${installedCoreVersion ?? ''}' vs manifest '${manifestCoreVersion ?? ''}'`,
);
return problems;
}
if (installed !== pinned) {
problems.push(
`core pin mismatch: installed ${installed} vs manifest ${pinned}` +
(manifestExtensionVersion ? ` (extension ${manifestExtensionVersion})` : ''),
);
}
return problems;
}
function readInstalledCoreVersion(pkg) {
const raw = pkg?.dependencies?.['@ladybugdb/core'];
return raw == null ? '' : String(raw);
}
function normalizeFilesEntry(value) {
return String(value ?? '')
.replace(/\\/g, '/')
.replace(/\/+$/, '')
.replace(/\/\*\*?$/, '');
}
/** True when package.json `files` still ships the FTS prebuild tree. */
function filesCoverFtsArtifacts(filesField) {
return (filesField || []).some((entry) => {
const n = normalizeFilesEntry(entry);
return (
n === 'vendor' ||
n === 'vendor/lbug-fts' ||
n === 'vendor/lbug-fts/prebuilds' ||
n === 'vendor/**/prebuilds'
);
});
}
function supportedTuplesFromManifest(manifest) {
return (manifest?.tuples ?? []).map((entry) => entry.tuple);
}
function unsupportedTuplesFromManifest(manifest) {
return (manifest?.unsupportedTuples ?? []).map((entry) => entry.tuple);
}
function parseSha256Sums(text) {
const out = {};
for (const line of String(text ?? '').split(/\r?\n/)) {
const m = /^([a-fA-F0-9]{64})\s+\.\/(\S+)$/.exec(line.trim());
if (!m) continue;
out[m[2]] = m[1].toLowerCase();
}
return out;
}
function sha256File(filePath) {
return crypto.createHash('sha256').update(fs.readFileSync(filePath)).digest('hex');
}
/**
* Integrity + coverage predicates. `artifactByTuple` is injected so tests
* never invoke pack and never need the real binaries.
*/
function findArtifactProblems({
tuples,
unsupportedTuples,
filesField,
checksumByRelPath,
artifactByTuple,
filename,
}) {
const problems = [];
const filenameSafe = filename || 'libfts.lbug_extension';
if (!SAFE_FILENAME.test(filenameSafe)) {
problems.push(`invalid FTS artifact filename: ${filenameSafe}`);
}
const listed = new Set(tuples || []);
for (const required of REQUIRED_SUPPORTED_TUPLES) {
if (!listed.has(required)) {
problems.push(`manifest.tuples is missing required ${required}`);
}
}
if (!tuples || tuples.length === 0) {
problems.push('manifest.tuples is empty — refusing to publish with 0 artifacts');
}
if (!filesCoverFtsArtifacts(filesField)) {
problems.push('package.json files no longer covers vendor/lbug-fts/prebuilds');
}
if (!(unsupportedTuples || []).includes(WIN32_ARM64)) {
problems.push('win32-arm64 must be declared unsupported (no upstream artifact)');
}
if ((tuples || []).includes(WIN32_ARM64)) {
problems.push('win32-arm64 is listed as supported but has no upstream artifact');
}
for (const tuple of tuples || []) {
const rel = `${tuple}/${filenameSafe}`;
const artifact = artifactByTuple?.[tuple];
if (!artifact?.exists) {
problems.push(`missing artifact for ${tuple} (${rel})`);
continue;
}
const expected = checksumByRelPath?.[rel];
if (!expected) {
problems.push(`missing SHA-256 for ${rel}`);
continue;
}
if (artifact.hash !== expected) {
problems.push(
`checksum mismatch for ${tuple}: expected ${expected} got ${artifact.hash}` +
(artifact.sizeBytes != null ? ` (${artifact.sizeBytes} bytes)` : ''),
);
}
}
return problems;
}
function readDiskArtifacts(prebuildsDir, tuples, filename) {
const artifactByTuple = {};
for (const tuple of tuples) {
const filePath = path.join(prebuildsDir, tuple, filename);
if (!fs.existsSync(filePath)) {
artifactByTuple[tuple] = { exists: false };
continue;
}
const buf = fs.readFileSync(filePath);
artifactByTuple[tuple] = {
exists: true,
hash: crypto.createHash('sha256').update(buf).digest('hex'),
sizeBytes: buf.byteLength,
};
}
return artifactByTuple;
}
function main() {
const gitnexusRoot = path.join(__dirname, '..');
const pkg = JSON.parse(fs.readFileSync(path.join(gitnexusRoot, 'package.json'), 'utf8'));
const vendorDir = path.join(gitnexusRoot, 'vendor', 'lbug-fts');
const manifestPath = path.join(vendorDir, 'manifest.json');
let manifest;
try {
manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
} catch (err) {
console.error(
`[fts-pairing] Refusing to publish — cannot read ${manifestPath}: ${err.message}`,
);
process.exit(1);
}
const installedCoreVersion = readInstalledCoreVersion(pkg);
const pairing = findPairingProblems({
installedCoreVersion,
manifestCoreVersion: manifest.coreVersion,
manifestExtensionVersion: manifest.extensionVersion,
});
const tuples = supportedTuplesFromManifest(manifest);
const unsupportedTuples = unsupportedTuplesFromManifest(manifest);
const filename = manifest.filename || 'libfts.lbug_extension';
const prebuildsDir = path.join(vendorDir, 'prebuilds');
let checksumByRelPath = {};
try {
checksumByRelPath = parseSha256Sums(
fs.readFileSync(path.join(prebuildsDir, 'SHA256SUMS'), 'utf8'),
);
} catch (err) {
pairing.push(`cannot read SHA256SUMS: ${err.message}`);
}
const artifacts = findArtifactProblems({
tuples,
unsupportedTuples,
filesField: pkg.files,
checksumByRelPath,
artifactByTuple: readDiskArtifacts(prebuildsDir, tuples, filename),
filename,
});
const problems = [...pairing, ...artifacts];
if (problems.length > 0) {
console.error('[fts-pairing] Refusing to publish — FTS artifact coverage failed:');
for (const p of problems) console.error(` - ${p}`);
console.error(
'\nFix: refresh vendor/lbug-fts via .github/scripts/fetch-lbug-fts-artifacts.mjs, ' +
'or restore the core pin / files allow-list.',
);
process.exit(1);
}
console.log(
`[fts-pairing] OK — core ${installedCoreVersion} ↔ extension ${manifest.extensionVersion}; ` +
`${tuples.length} artifacts.`,
);
}
if (require.main === module) main();
module.exports = {
findPairingProblems,
findArtifactProblems,
filesCoverFtsArtifacts,
parseSha256Sums,
readInstalledCoreVersion,
supportedTuplesFromManifest,
unsupportedTuplesFromManifest,
sha256File,
};