mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-01 02:01:24 +00:00
* fix(lbug): pin Ladybug core so Dependabot cannot ship a skewed FTS artifact The extension version is a separate upstream constant. Ignore daily core bumps and fail the pairing gate when the committed manifest does not name the installed core. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): make doctor and CI FTS gates resolve the packaged artifact Doctor and the REQUIRE_FTS file gates still treated an empty ~/.lbdb as unavailable, which would turn three CI jobs red once analyze stops installing into that tree. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): name native-abort and tuple-missing so analyze cannot mis-advise The CLI summary's trailing else treated every unknown skip reason as a missing extension. New crash and platform causes must get their own remedies, not a network-install hint. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): delete the dead read-path FTS index create ensureFTSIndex had no production callers and swallowed read-only CREATE_FTS_INDEX failures, which hid the only signal that a reader tried to write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): vendor per-platform FTS artifacts so analyze needs no host install Keyword search depended on a CDN fetch into ~/.lbdb. Shipping the five published tuples inside the package makes air-gapped and ignore-scripts installs load the same artifact the publish gate checksums. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): load the packaged FTS artifact before any network install Analyze still required a CDN fetch into ~/.lbdb even when the package already shipped the file. FTS now path-loads the vendored tuple first and records source labels so a later truncated home copy cannot steal the diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): diagnose a core/extension version skew instead of a missing runtime A structurally valid FTS artifact whose path version disagrees with the packaged pin must name both versions, not prescribe VC++ or OpenSSL. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): stamp an FTS phase so repair stays usable after an in-place abort A native CREATE_FTS_INDEX abort leaves no skip reason; the next run infers it from the dirty flag, and --repair-fts must not treat that phase as a half-written graph. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): park an in-place FTS crash WAL without wiping the graph An FTS abort after a successful checkpoint must reopen the live index on macOS, Windows, and Linux. Staging never parks the live WAL; readers keep today's large-WAL refusal. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): refuse read-only opens of an FTS-poisoned WAL MCP and serve cannot repair a leftover in-place abort. Fail before the native open and name --repair-fts, on macOS, Windows, and Linux. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): name a vendor-neutral Windows OpenSSL prerequisite OQ1 is unanswered here so GitNexus does not ship OpenSSL DLLs. Windows FTS now asks for a system OpenSSL 3 runtime instead of Git Bash PATH. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): inject the FTS vendor root and redact it on HTTP and MCP Path-loaded artifacts no longer vary with HOME. Tests pass an injected vendor tree and assert search warnings never leak a filesystem path. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(lbug): document load-only as the global FTS install default Analyze still overrides to auto. Packaged per-platform artifacts load before any network install on macOS, Windows, and Linux. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(lbug): format the FTS install-policy README table Prettier does not run on Markdown in pre-commit, so the U10 table wrap needs its own formatting commit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): skip FTS CREATE after a persisted native abort A recovered analyze run was retrying CREATE_FTS_INDEX from skipReason alone. Keep that skip until --repair-fts, fail closed on unsupported tuples, and honor the checkpoint warrant for park/repair. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): honor checkpoint flushed warrant and align FTS tests with packaged vendor A no-op CHECKPOINT must not satisfy the FTS park warrant, and CI still asserted HOME-only FTS isolation after analyze started path-LOADing the packaged artifact. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): accept a nonempty incremental write set in the #2790 recovery check FTS-phase recovery can incremental-add files (changed=0, added=1). That is not the #2790 empty-diff wipe skip. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): compare FTS home versions to the core pin and tighten the publish filename gate Ladybug's ~/.lbdb/extension directory is the runtime/core version; treating it as the artifact version false-diagnosed skew. The publish guard now rejects a path-escaping filename the same way the fetch script does. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): seed FTS e2e fixtures from the packaged vendor artifact A machine with no ~/.lbdb copy should still run the vendor-survivorship cases; the seed no longer depends on HOME or a network install. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3274) Keep in-place FTS abort evidence after persist so a second CREATE abort cannot fail-open readers, and close the CLI, loader, embed, and e2e gaps the review called out. Note: full npm test hit Ladybug worker-pool startup failures under memory pressure; tsc and 180 targeted unit tests passed. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3274) Run the vendored-path symlink guard on the OS matrix, put e2e HOME fixtures on Ladybug's real extension layout, pin the embed crash-WAL gate before the writable open, and let analyze writers park through missing-shadow recovery. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): keep --repair-fts CI green after vendored-first FTS Never-installed warning fixtures must not inspect a packaged vendor binary, and a failed dirty restamp must not abort an otherwise successful --repair-fts run. Co-authored-by: Cursor <cursoragent@cursor.com> * test(cli): give the #1169 analyze e2e the same 90s Windows budget as its sibling The first #1169 persist-meta case was still on a 60s spawn/it budget and was killed banner-only on windows-latest after the FTS warning fixture no longer failed the shard first. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ci): reweight Windows shards after the FTS e2e grew Vendored-first HOME fixtures pushed fts-extension-e2e to ~6 minutes on windows-latest, so the old 146s weight packed it with skills-e2e and blew the 20-minute watchdog. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
144 lines
5.7 KiB
JavaScript
144 lines
5.7 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Fetch Ladybug FTS artifacts into gitnexus/vendor/lbug-fts/prebuilds/.
|
|
*
|
|
* Lives outside the published package (`files` includes `scripts` wholesale).
|
|
* Reads versions, filename, and tuple→upstream-platform mapping from
|
|
* vendor/lbug-fts/manifest.json so the gate and runtime cannot drift.
|
|
*
|
|
* Usage: node .github/scripts/fetch-lbug-fts-artifacts.mjs
|
|
*/
|
|
import { createHash } from 'node:crypto';
|
|
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
|
import path from 'node:path';
|
|
import { fileURLToPath, pathToFileURL } from 'node:url';
|
|
|
|
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..');
|
|
const VENDOR = path.join(REPO_ROOT, 'gitnexus', 'vendor', 'lbug-fts');
|
|
const PREBUILDS = path.join(VENDOR, 'prebuilds');
|
|
const MANIFEST_PATH = path.join(VENDOR, 'manifest.json');
|
|
|
|
/** Only the Ladybug official extension host — never a manifest-supplied origin. */
|
|
const OFFICIAL_REPO = 'https://extension.ladybugdb.com/';
|
|
const EXACT_VERSION = /^\d+\.\d+\.\d+$/;
|
|
const SAFE_UPSTREAM = /^(linux_amd64|linux_arm64|osx_amd64|osx_arm64|win_amd64)$/;
|
|
|
|
/**
|
|
* Build the official artifact URL from allowlisted fields only.
|
|
* `officialRepo` in the manifest must match {@link OFFICIAL_REPO}; the
|
|
* origin itself is a constant so an edited manifest cannot redirect the fetch.
|
|
*/
|
|
export function officialArtifactUrl(manifest, upstreamPlatform) {
|
|
const officialRepo = String(manifest?.officialRepo ?? '');
|
|
if (officialRepo !== OFFICIAL_REPO) {
|
|
throw new Error(`refusing unofficial FTS repo: '${officialRepo}'`);
|
|
}
|
|
const version = String(manifest?.extensionVersion ?? '');
|
|
if (!EXACT_VERSION.test(version)) {
|
|
throw new Error(`unsafe extensionVersion: '${version}'`);
|
|
}
|
|
if (!SAFE_UPSTREAM.test(String(upstreamPlatform ?? ''))) {
|
|
throw new Error(`unsafe upstream platform: '${upstreamPlatform}'`);
|
|
}
|
|
const filename = String(manifest?.filename ?? '');
|
|
if (!SAFE_FILENAME.test(filename)) {
|
|
throw new Error(`unsafe FTS artifact filename: '${filename}'`);
|
|
}
|
|
return `${OFFICIAL_REPO}v${version}/${upstreamPlatform}/fts/${filename}`;
|
|
}
|
|
|
|
const sha256 = (buf) => createHash('sha256').update(buf).digest('hex');
|
|
|
|
const readExistingHash = (filePath) => {
|
|
if (!existsSync(filePath)) return null;
|
|
return sha256(readFileSync(filePath));
|
|
};
|
|
|
|
export const supportedTuples = (manifest) => manifest.tuples.map((entry) => entry.tuple);
|
|
|
|
const SAFE_TUPLE = /^(darwin|linux|win32)-(x64|arm64)$/;
|
|
const SAFE_FILENAME = /^[\w.-]+\.lbug_extension$/;
|
|
|
|
/** Relative-path containment — not a prefix match (rejects `prebuilds-evil`). */
|
|
const isPathInsideRoot = (root, candidate) => {
|
|
const relative = path.relative(root, candidate);
|
|
if (path.isAbsolute(relative)) return false;
|
|
return relative !== '' && !relative.startsWith(`..${path.sep}`) && relative !== '..';
|
|
};
|
|
|
|
export function assertSafeArtifactDest({ prebuildsDir, tuple, filename }) {
|
|
if (!SAFE_TUPLE.test(String(tuple ?? ''))) {
|
|
throw new Error(
|
|
`unsafe FTS artifact tuple: '${tuple}' (expected (darwin|linux|win32)-(x64|arm64))`,
|
|
);
|
|
}
|
|
if (!SAFE_FILENAME.test(String(filename ?? ''))) {
|
|
throw new Error(`unsafe FTS artifact filename: '${filename}' (expected *.lbug_extension)`);
|
|
}
|
|
const dest = path.join(prebuildsDir, tuple, filename);
|
|
if (!isPathInsideRoot(prebuildsDir, dest)) {
|
|
throw new Error(`FTS artifact dest is not inside prebuildsDir: ${dest}`);
|
|
}
|
|
return dest;
|
|
}
|
|
|
|
async function fetchBuffer(url) {
|
|
// codeql[js/request-forgery] — origin is OFFICIAL_REPO; path segments are allowlisted.
|
|
// lgtm[js/request-forgery]
|
|
// codeql[js/file-access-to-http] — versions/platforms are regex-pinned, not raw file bytes.
|
|
const res = await fetch(url, { signal: AbortSignal.timeout(120_000) });
|
|
if (!res.ok) {
|
|
throw new Error(`GET ${url} → ${res.status} ${res.statusText}`);
|
|
}
|
|
return Buffer.from(await res.arrayBuffer());
|
|
}
|
|
|
|
const writeAllowlistedArtifact = (prebuildsDir, dest, buf) => {
|
|
if (!isPathInsideRoot(prebuildsDir, dest)) {
|
|
throw new Error(`FTS artifact dest is not inside prebuildsDir: ${dest}`);
|
|
}
|
|
// codeql[js/http-to-file-access] — dest is assertSafeArtifactDest + containment-checked.
|
|
writeFileSync(dest, buf);
|
|
};
|
|
|
|
export async function refreshArtifacts({
|
|
manifest = JSON.parse(readFileSync(MANIFEST_PATH, 'utf8')),
|
|
prebuildsDir = PREBUILDS,
|
|
download = fetchBuffer,
|
|
} = {}) {
|
|
mkdirSync(prebuildsDir, { recursive: true });
|
|
const lines = [];
|
|
for (const { tuple, upstreamPlatform } of manifest.tuples) {
|
|
const dest = assertSafeArtifactDest({
|
|
prebuildsDir,
|
|
tuple,
|
|
filename: manifest.filename,
|
|
});
|
|
mkdirSync(path.dirname(dest), { recursive: true });
|
|
const url = officialArtifactUrl(manifest, upstreamPlatform);
|
|
const previousHash = readExistingHash(dest);
|
|
const previousSize = previousHash ? readFileSync(dest).byteLength : 0;
|
|
const buf = await download(url);
|
|
const nextHash = sha256(buf);
|
|
writeAllowlistedArtifact(prebuildsDir, dest, buf);
|
|
const changed = previousHash !== nextHash;
|
|
console.log(
|
|
changed
|
|
? `[fts-fetch] ${tuple}: ${previousHash ?? '(new)'} (${previousSize} B) → ${nextHash} (${buf.byteLength} B)`
|
|
: `[fts-fetch] ${tuple}: unchanged ${nextHash} (${buf.byteLength} B)`,
|
|
);
|
|
lines.push(`${nextHash} ./${tuple}/${manifest.filename}`);
|
|
}
|
|
lines.sort();
|
|
writeFileSync(path.join(prebuildsDir, 'SHA256SUMS'), `${lines.join('\n')}\n`);
|
|
return lines;
|
|
}
|
|
|
|
const invokedDirectly =
|
|
process.argv[1] && pathToFileURL(path.resolve(process.argv[1])).href === import.meta.url;
|
|
if (invokedDirectly) {
|
|
refreshArtifacts().catch((err) => {
|
|
console.error(`[fts-fetch] ${err instanceof Error ? err.message : err}`);
|
|
process.exit(1);
|
|
});
|
|
}
|