GitNexus/.github/scripts/fetch-lbug-fts-artifacts.mjs
Gergő Magyar 21a52af1d4
fix(lbug): ship FTS per-platform and recover in-place native aborts (#3274)
* fix(lbug): pin Ladybug core so Dependabot cannot ship a skewed FTS artifact

The extension version is a separate upstream constant. Ignore daily core bumps and fail the pairing gate when the committed manifest does not name the installed core.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): make doctor and CI FTS gates resolve the packaged artifact

Doctor and the REQUIRE_FTS file gates still treated an empty ~/.lbdb as
unavailable, which would turn three CI jobs red once analyze stops
installing into that tree.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name native-abort and tuple-missing so analyze cannot mis-advise

The CLI summary's trailing else treated every unknown skip reason as a
missing extension. New crash and platform causes must get their own
remedies, not a network-install hint.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): delete the dead read-path FTS index create

ensureFTSIndex had no production callers and swallowed read-only
CREATE_FTS_INDEX failures, which hid the only signal that a reader
tried to write.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): vendor per-platform FTS artifacts so analyze needs no host install

Keyword search depended on a CDN fetch into ~/.lbdb. Shipping the five
published tuples inside the package makes air-gapped and ignore-scripts
installs load the same artifact the publish gate checksums.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): load the packaged FTS artifact before any network install

Analyze still required a CDN fetch into ~/.lbdb even when the package
already shipped the file. FTS now path-loads the vendored tuple first
and records source labels so a later truncated home copy cannot steal
the diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): diagnose a core/extension version skew instead of a missing runtime

A structurally valid FTS artifact whose path version disagrees with the
packaged pin must name both versions, not prescribe VC++ or OpenSSL.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): stamp an FTS phase so repair stays usable after an in-place abort

A native CREATE_FTS_INDEX abort leaves no skip reason; the next run infers
it from the dirty flag, and --repair-fts must not treat that phase as a
half-written graph.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): park an in-place FTS crash WAL without wiping the graph

An FTS abort after a successful checkpoint must reopen the live index on
macOS, Windows, and Linux. Staging never parks the live WAL; readers keep
today's large-WAL refusal.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): refuse read-only opens of an FTS-poisoned WAL

MCP and serve cannot repair a leftover in-place abort. Fail before the
native open and name --repair-fts, on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): name a vendor-neutral Windows OpenSSL prerequisite

OQ1 is unanswered here so GitNexus does not ship OpenSSL DLLs. Windows
FTS now asks for a system OpenSSL 3 runtime instead of Git Bash PATH.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): inject the FTS vendor root and redact it on HTTP and MCP

Path-loaded artifacts no longer vary with HOME. Tests pass an injected
vendor tree and assert search warnings never leak a filesystem path.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): document load-only as the global FTS install default

Analyze still overrides to auto. Packaged per-platform artifacts load
before any network install on macOS, Windows, and Linux.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(lbug): format the FTS install-policy README table

Prettier does not run on Markdown in pre-commit, so the U10 table wrap
needs its own formatting commit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): skip FTS CREATE after a persisted native abort

A recovered analyze run was retrying CREATE_FTS_INDEX from skipReason
alone. Keep that skip until --repair-fts, fail closed on unsupported
tuples, and honor the checkpoint warrant for park/repair.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): honor checkpoint flushed warrant and align FTS tests with packaged vendor

A no-op CHECKPOINT must not satisfy the FTS park warrant, and CI still asserted HOME-only FTS isolation after analyze started path-LOADing the packaged artifact.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): accept a nonempty incremental write set in the #2790 recovery check

FTS-phase recovery can incremental-add files (changed=0, added=1). That is not the #2790 empty-diff wipe skip.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): compare FTS home versions to the core pin and tighten the publish filename gate

Ladybug's ~/.lbdb/extension directory is the runtime/core version; treating it as the artifact version false-diagnosed skew. The publish guard now rejects a path-escaping filename the same way the fetch script does.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(lbug): seed FTS e2e fixtures from the packaged vendor artifact

A machine with no ~/.lbdb copy should still run the vendor-survivorship cases; the seed no longer depends on HOME or a network install.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Keep in-place FTS abort evidence after persist so a second CREATE abort
cannot fail-open readers, and close the CLI, loader, embed, and e2e gaps
the review called out.

Note: full npm test hit Ladybug worker-pool startup failures under memory
pressure; tsc and 180 targeted unit tests passed.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3274)

Run the vendored-path symlink guard on the OS matrix, put e2e HOME
fixtures on Ladybug's real extension layout, pin the embed crash-WAL
gate before the writable open, and let analyze writers park through
missing-shadow recovery.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lbug): keep --repair-fts CI green after vendored-first FTS

Never-installed warning fixtures must not inspect a packaged vendor binary, and a failed dirty restamp must not abort an otherwise successful --repair-fts run.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(cli): give the #1169 analyze e2e the same 90s Windows budget as its sibling

The first #1169 persist-meta case was still on a 60s spawn/it budget and was killed banner-only on windows-latest after the FTS warning fixture no longer failed the shard first.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(ci): reweight Windows shards after the FTS e2e grew

Vendored-first HOME fixtures pushed fts-extension-e2e to ~6 minutes on windows-latest, so the old 146s weight packed it with skills-e2e and blew the 20-minute watchdog.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-14 08:52:24 +01:00

144 lines
5.7 KiB
JavaScript

#!/usr/bin/env node
/**
* Fetch Ladybug FTS artifacts into gitnexus/vendor/lbug-fts/prebuilds/.
*
* Lives outside the published package (`files` includes `scripts` wholesale).
* Reads versions, filename, and tuple→upstream-platform mapping from
* vendor/lbug-fts/manifest.json so the gate and runtime cannot drift.
*
* Usage: node .github/scripts/fetch-lbug-fts-artifacts.mjs
*/
import { createHash } from 'node:crypto';
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
const VENDOR = path.join(REPO_ROOT, 'gitnexus', 'vendor', 'lbug-fts');
const PREBUILDS = path.join(VENDOR, 'prebuilds');
const MANIFEST_PATH = path.join(VENDOR, 'manifest.json');
/** Only the Ladybug official extension host — never a manifest-supplied origin. */
const OFFICIAL_REPO = 'https://extension.ladybugdb.com/';
const EXACT_VERSION = /^\d+\.\d+\.\d+$/;
const SAFE_UPSTREAM = /^(linux_amd64|linux_arm64|osx_amd64|osx_arm64|win_amd64)$/;
/**
* Build the official artifact URL from allowlisted fields only.
* `officialRepo` in the manifest must match {@link OFFICIAL_REPO}; the
* origin itself is a constant so an edited manifest cannot redirect the fetch.
*/
export function officialArtifactUrl(manifest, upstreamPlatform) {
const officialRepo = String(manifest?.officialRepo ?? '');
if (officialRepo !== OFFICIAL_REPO) {
throw new Error(`refusing unofficial FTS repo: '${officialRepo}'`);
}
const version = String(manifest?.extensionVersion ?? '');
if (!EXACT_VERSION.test(version)) {
throw new Error(`unsafe extensionVersion: '${version}'`);
}
if (!SAFE_UPSTREAM.test(String(upstreamPlatform ?? ''))) {
throw new Error(`unsafe upstream platform: '${upstreamPlatform}'`);
}
const filename = String(manifest?.filename ?? '');
if (!SAFE_FILENAME.test(filename)) {
throw new Error(`unsafe FTS artifact filename: '${filename}'`);
}
return `${OFFICIAL_REPO}v${version}/${upstreamPlatform}/fts/${filename}`;
}
const sha256 = (buf) => createHash('sha256').update(buf).digest('hex');
const readExistingHash = (filePath) => {
if (!existsSync(filePath)) return null;
return sha256(readFileSync(filePath));
};
export const supportedTuples = (manifest) => manifest.tuples.map((entry) => entry.tuple);
const SAFE_TUPLE = /^(darwin|linux|win32)-(x64|arm64)$/;
const SAFE_FILENAME = /^[\w.-]+\.lbug_extension$/;
/** Relative-path containment — not a prefix match (rejects `prebuilds-evil`). */
const isPathInsideRoot = (root, candidate) => {
const relative = path.relative(root, candidate);
if (path.isAbsolute(relative)) return false;
return relative !== '' && !relative.startsWith(`..${path.sep}`) && relative !== '..';
};
export function assertSafeArtifactDest({ prebuildsDir, tuple, filename }) {
if (!SAFE_TUPLE.test(String(tuple ?? ''))) {
throw new Error(
`unsafe FTS artifact tuple: '${tuple}' (expected (darwin|linux|win32)-(x64|arm64))`,
);
}
if (!SAFE_FILENAME.test(String(filename ?? ''))) {
throw new Error(`unsafe FTS artifact filename: '${filename}' (expected *.lbug_extension)`);
}
const dest = path.join(prebuildsDir, tuple, filename);
if (!isPathInsideRoot(prebuildsDir, dest)) {
throw new Error(`FTS artifact dest is not inside prebuildsDir: ${dest}`);
}
return dest;
}
async function fetchBuffer(url) {
// codeql[js/request-forgery] — origin is OFFICIAL_REPO; path segments are allowlisted.
// lgtm[js/request-forgery]
// codeql[js/file-access-to-http] — versions/platforms are regex-pinned, not raw file bytes.
const res = await fetch(url, { signal: AbortSignal.timeout(120_000) });
if (!res.ok) {
throw new Error(`GET ${url} → ${res.status} ${res.statusText}`);
}
return Buffer.from(await res.arrayBuffer());
}
const writeAllowlistedArtifact = (prebuildsDir, dest, buf) => {
if (!isPathInsideRoot(prebuildsDir, dest)) {
throw new Error(`FTS artifact dest is not inside prebuildsDir: ${dest}`);
}
// codeql[js/http-to-file-access] — dest is assertSafeArtifactDest + containment-checked.
writeFileSync(dest, buf);
};
export async function refreshArtifacts({
manifest = JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')),
prebuildsDir = PREBUILDS,
download = fetchBuffer,
} = {}) {
mkdirSync(prebuildsDir, { recursive: true });
const lines = [];
for (const { tuple, upstreamPlatform } of manifest.tuples) {
const dest = assertSafeArtifactDest({
prebuildsDir,
tuple,
filename: manifest.filename,
});
mkdirSync(path.dirname(dest), { recursive: true });
const url = officialArtifactUrl(manifest, upstreamPlatform);
const previousHash = readExistingHash(dest);
const previousSize = previousHash ? readFileSync(dest).byteLength : 0;
const buf = await download(url);
const nextHash = sha256(buf);
writeAllowlistedArtifact(prebuildsDir, dest, buf);
const changed = previousHash !== nextHash;
console.log(
changed
? `[fts-fetch] ${tuple}: ${previousHash ?? '(new)'} (${previousSize} B) → ${nextHash} (${buf.byteLength} B)`
: `[fts-fetch] ${tuple}: unchanged ${nextHash} (${buf.byteLength} B)`,
);
lines.push(`${nextHash} ./${tuple}/${manifest.filename}`);
}
lines.sort();
writeFileSync(path.join(prebuildsDir, 'SHA256SUMS'), `${lines.join('\n')}\n`);
return lines;
}
const invokedDirectly =
process.argv[1] && pathToFileURL(path.resolve(process.argv[1])).href === import.meta.url;
if (invokedDirectly) {
refreshArtifacts().catch((err) => {
console.error(`[fts-fetch] ${err instanceof Error ? err.message : err}`);
process.exit(1);
});
}