mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-11 03:38:07 +00:00
* fix(cobol): prefer copybook dirs so COPY EXTERNAL does not hit vendor decoys
COPY of an out-of-repo member first-won any same-named .cpy, so
vendor/EXTERNAL.cpy became a live cobol-copy IMPORTS edge. Share one
resolver between census and the regex processor: prefer copybooks/cpy/copy
plus the importer dir when present, else fail-open. Drop COBOL KNOWN_GAPS.
Fixes #2967
* bench(cobol): update depth budget for copybook-dir preference (#2967)
COBOL resolver now prefers well-known copybook directories (copybooks/,
cpy/, copy/, plus importer dir) over vendor paths when resolving COPY
statements. This intentional behavior change moves the resolver from
depth-free (prior measured ~0.885) to depth-sensitive (measured 1.751
on CI run 34394116972), because the new preferredCopybookDirs check
walks path components.
- Raise depth_budget from 1.6 to 2.4 (~1.37x the measured ratio)
- Update _measured.depth_ratio from 0.885 to 1.751
- Add _cobol_copybook_dir_preference_2967 note documenting the change
The COBOL fingerprints already reflect the new target set behavior
(vendor/EXTERNAL.cpy correctly returns null when a copybook dir is
present) per commit fc9e8270.
Co-authored-by: Kevin Rajan <kvnloo@users.noreply.github.com>
* bench(cobol): update baselines for copybook-dir preference (#2967)
COBOL preferred-dir filtering now affects resolution outcomes. The
unique-arm layouts (mixed copybooks/src dirs) drop from 1153 to 442
resolved as files outside preferred directories are correctly filtered.
The collide arm (all files in svc${d}/copybooks) keeps 1153 resolved
because ALL files remain in the preferred class.
- Update small/deep resolved: 1153 → 442
- Update small/large/deep fingerprints for new target set
- Raise heap_bound_bytes.cobol: 3500000 → 6200000 (1.5x measured 4112464 B)
- Add measure.mjs exception: collide legitimately differs from small
- Update _heap_bound_note with new cobol measurement context
- Expand _cobol_copybook_dir_preference_2967 note to explain collide delta
The collide arm now measures collision behavior within the preferred
class rather than across mixed layouts — an intentional outcome of the
preferred-dir semantics rather than a corpus defect.
Refs #2967
Co-authored-by: Kevin Rajan <kvnloo@users.noreply.github.com>
* fix(cobol): P1-A stem key with uppercase extensions, P1-B polyglot preferred-class latch
P1-A: Use raw extension for path.basename so CUSTREC.CPY keys as CUSTREC
- Before: path.basename('CUSTREC.CPY', '.cpy') -> 'CUSTREC.CPY' (no strip)
- After: path.basename('CUSTREC.CPY', '.CPY') -> 'CUSTREC' (stripped)
- Processor used raw extension; census now matches
P1-B: Latch preferred-class only from copybook-tier paths
- Before: docs/copy/README.md triggered hasPreferredDir = true
- After: check preferred-dir only after extension filter
- Processor receives polyglot allPathSet; test added
Test coverage:
- cobol-copy-external-imports.test.ts: processor pins for both fixes
- cobol-import-target-parity.test.ts: resolver parity updated to new behavior
- All existing COBOL tests pass
Co-authored-by: Kevin Rajan <kvnloo@users.noreply.github.com>
* test(cobol): fix Mixed.CPY integration test for P1-A behavior
The integration test in cobol-import-index-reuse.test.ts had outdated
expectations from before P1-A. With P1-A, path.basename uses the raw
extension, so Mixed.CPY is now keyed as MIXED (extension stripped).
Before (pre-P1-A):
- Mixed.CPY → keyed as MIXED.CPY (uppercase ext not stripped)
- COPY MIXED.CPY → found, COPY MIXED → null
After (P1-A):
- Mixed.CPY → keyed as MIXED (raw ext stripped)
- COPY MIXED → found, COPY MIXED.CPY → null
Updated test expectations to match P1-A behavior. All three validation
tests now pass:
- cobol-import-index-reuse.test.ts (integration, index reuse)
- cobol-copy-external-imports.test.ts (processor P1-A/P1-B pins)
- cobol-import-target-parity.test.ts (resolver parity)
Fixes PER-994 CI failure on abhigyanpatwari/GitNexus#3240.
Co-authored-by: Kevin Rajan <kvnloo@users.noreply.github.com>
---------
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Kevin Rajan <kvnloo@users.noreply.github.com>
389 lines
15 KiB
TypeScript
389 lines
15 KiB
TypeScript
/**
|
||
* Differential harness for the COBOL `COPY`-target index hoist (#2908).
|
||
*
|
||
* `cobolScopeResolver.resolveImportTarget` used to answer every `COPY` with TWO
|
||
* full `allFilePaths` scans — copybooks first, then COBOL sources — each calling
|
||
* `path.extname` + `path.basename` + `toUpperCase` on every entry, so resolution
|
||
* cost O(copies × files) and a `COPY` of a member that is not in the repo (the
|
||
* common case) ran both scans to completion. Replacing them with a per-run
|
||
* two-tier index is a pure performance change ONLY if every implicit tie-break
|
||
* survives, and none of them is visible to the type system:
|
||
*
|
||
* - TIER ORDER: a `.cpy`/`.copybook` hit beats a `.cbl`/`.cob`/`.cobol` hit
|
||
* even when the source file comes FIRST in Set-iteration order. Collapsing
|
||
* the two tiers into one first-wins map is the "obvious" rewrite and it
|
||
* silently inverts this;
|
||
* - WITHIN A TIER: the first file in Set-iteration order wins, because the
|
||
* scans returned on first match;
|
||
* - CASE: the extension is compared LOWER-cased while the basename is
|
||
* compared UPPER-cased, and `path.basename(fp, ext)` strips the suffix only
|
||
* on an exact, case-sensitive match — so `Foo.CPY` is indexed under
|
||
* `FOO.CPY`, not `FOO`, and is unreachable by a `COPY FOO`;
|
||
* - `path` SEMANTICS: Node's `path.extname`/`path.basename` are what decide
|
||
* where the stem starts, and on POSIX they do not treat `\` as a separator.
|
||
* Hand-rolled slicing on `/` would start resolving backslash paths that
|
||
* previously returned null.
|
||
*
|
||
* So this file keeps a VERBATIM copy of the pre-change resolver body
|
||
* (`git show HEAD~:…/languages/cobol/scope-resolver.ts`) and asserts the new
|
||
* implementation agrees with it on a deterministic generated corpus plus a
|
||
* hand-built layout per tie-break. The copy is the specification; if an arm here
|
||
* fails, the resolver's OUTPUT moved and COBOL's IMPORTS edges move with it.
|
||
*
|
||
* Mutation-tested against the new implementation — each of these was inserted,
|
||
* confirmed RED here, and reverted: tiers collapsed into one map; within-tier
|
||
* first-wins flipped to last-wins; `targetRaw.toUpperCase()` dropped;
|
||
* `path.extname(fp).toLowerCase()` left un-lowercased.
|
||
*
|
||
* This file calls the resolver directly, which for COBOL is also the
|
||
* orchestrator adapter — but the arms below say nothing about the Set being
|
||
* passed THROUGH, and a defensive `new Set(allFilePaths)` would leave them all
|
||
* green while restoring the per-import rebuild. That failure is guarded by
|
||
* `test/integration/cobol-import-index-reuse.test.ts`.
|
||
*/
|
||
import path from 'node:path';
|
||
import { describe, expect, it } from 'vitest';
|
||
|
||
import { cobolScopeResolver } from '../../../src/core/ingestion/languages/cobol/scope-resolver.js';
|
||
|
||
const { resolveImportTarget } = cobolScopeResolver;
|
||
|
||
/** COBOL takes no `resolutionConfig` and ignores `fromFile`; both are pinned. */
|
||
const FROM_FILE = 'src/PROG.cbl';
|
||
|
||
function resolve(targetRaw: string, files: ReadonlySet<string>): string | readonly string[] | null {
|
||
return resolveImportTarget(targetRaw, FROM_FILE, files, undefined);
|
||
}
|
||
|
||
// ─── verbatim pre-change implementation ──────────────────────────────────────
|
||
|
||
const LEGACY_COPYBOOK_EXTENSIONS = new Set(['.cpy', '.copybook']);
|
||
|
||
function legacyResolveCobolImportTarget(
|
||
targetRaw: string,
|
||
allFilePaths: ReadonlySet<string>,
|
||
): string | null {
|
||
const upper = targetRaw.toUpperCase();
|
||
// Check copybook files first
|
||
for (const fp of allFilePaths) {
|
||
const extRaw = path.extname(fp);
|
||
const extLower = extRaw.toLowerCase();
|
||
if (!LEGACY_COPYBOOK_EXTENSIONS.has(extLower)) continue;
|
||
const basename = path.basename(fp, extRaw).toUpperCase();
|
||
if (basename === upper) return fp;
|
||
}
|
||
// Also search COBOL source files (.cbl, .cob, .cobol)
|
||
const COBOL_SOURCE_EXTS = new Set(['.cbl', '.cob', '.cobol']);
|
||
for (const fp of allFilePaths) {
|
||
const extRaw = path.extname(fp);
|
||
const extLower = extRaw.toLowerCase();
|
||
if (!COBOL_SOURCE_EXTS.has(extLower)) continue;
|
||
const basename = path.basename(fp, extRaw).toUpperCase();
|
||
if (basename === upper) return fp;
|
||
}
|
||
return null;
|
||
}
|
||
|
||
// ─── deterministic corpus ────────────────────────────────────────────────────
|
||
|
||
/** Murmur3 finalizer — a reproducible stand-in for `Math.random()`. */
|
||
function mix(n: number): number {
|
||
let x = n >>> 0;
|
||
x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0;
|
||
x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0;
|
||
return (x ^ (x >>> 16)) >>> 0;
|
||
}
|
||
|
||
/**
|
||
* Directory shapes of a typical mainframe checkout, including one whose
|
||
* segments are separated by BACKSLASHES — on POSIX that is one long filename,
|
||
* which is precisely the `path` semantic the index must not "simplify" away.
|
||
*/
|
||
const DIRS = [
|
||
'',
|
||
'copybooks',
|
||
'COPYBOOKS',
|
||
'src',
|
||
'src/copy',
|
||
'legacy/cpy',
|
||
'jcl/proclib',
|
||
'win\\dir',
|
||
];
|
||
|
||
/** Member names in the case mixture a real repo has. */
|
||
const STEMS = ['CUSTREC', 'custrec', 'AcctRec', 'PAYROLL', 'BOOK', 'COMMON', 'TAXCALC', 'ERRDEMO'];
|
||
|
||
/**
|
||
* Both tiers, both cases, plus two extensions in NEITHER tier: `.txt` (a
|
||
* non-COBOL file that must never answer a `COPY`) and `''` (a file with no
|
||
* extension at all, which `path.extname` reports as the empty string and which
|
||
* therefore falls out of both extension sets).
|
||
*/
|
||
const EXTS = ['.cpy', '.copybook', '.CPY', '.cbl', '.cob', '.cobol', '.CBL', '.txt', ''];
|
||
|
||
/** Directories with no well-known copybook segment — fail-open parity only. */
|
||
const FAIL_OPEN_DIRS = ['', 'src', 'jcl/proclib', 'win\\dir'];
|
||
|
||
function corpusFromDirs(seed: number, fileCount: number, dirs: readonly string[]): Set<string> {
|
||
const files = new Set<string>();
|
||
for (let i = 0; i < fileCount; i++) {
|
||
const a = mix(seed * 7919 + i);
|
||
const b = mix(a ^ 0x9e3779b9);
|
||
const c = mix(b ^ 0x85ebca6b);
|
||
const dir = dirs[a % dirs.length];
|
||
const stem = STEMS[b % STEMS.length];
|
||
const rel = `${stem}${EXTS[c % EXTS.length]}`;
|
||
files.add(dir === '' ? rel : `${dir}/${rel}`);
|
||
}
|
||
// Backslash-separated paths, which `path` reads differently per platform and
|
||
// a `/`-slicing rewrite would read differently from `path` on POSIX.
|
||
files.add('win\\dir\\BOOK.cpy');
|
||
files.add('win\\dir\\PAYROLL.cbl');
|
||
return files;
|
||
}
|
||
|
||
function corpus(seed: number, fileCount: number): Set<string> {
|
||
return corpusFromDirs(seed, fileCount, DIRS);
|
||
}
|
||
|
||
/**
|
||
* `COPY` operands as they appear in source, plus the spellings that reach the
|
||
* corpus's awkward files. Lower-case entries are what breaks if the target
|
||
* stops being upper-cased; the `.CPY`/`.CBL` suffixed entries are what reaches
|
||
* a file whose mixed-case extension `path.basename` refused to strip.
|
||
*/
|
||
const TARGETS = [
|
||
'',
|
||
'CUSTREC',
|
||
'custrec',
|
||
'CustRec',
|
||
'ACCTREC',
|
||
'PAYROLL',
|
||
'payroll',
|
||
'BOOK',
|
||
'COMMON',
|
||
'TAXCALC',
|
||
'ERRDEMO',
|
||
'MISSING',
|
||
'BOOK.CPY',
|
||
'CUSTREC.CPY',
|
||
'PAYROLL.CBL',
|
||
'win\\dir\\BOOK',
|
||
'WIN\\DIR\\BOOK',
|
||
'win/dir/BOOK',
|
||
];
|
||
|
||
const REPOS = 40;
|
||
|
||
function hasPreferredCopyDir(files: ReadonlySet<string>): boolean {
|
||
for (const fp of files) {
|
||
const parts = fp.split('/');
|
||
for (let i = 0; i < parts.length - 1; i++) {
|
||
if (
|
||
parts[i] === 'copybooks' ||
|
||
parts[i] === 'COPYBOOKS' ||
|
||
parts[i] === 'cpy' ||
|
||
parts[i] === 'copy'
|
||
) {
|
||
return true;
|
||
}
|
||
}
|
||
}
|
||
return false;
|
||
}
|
||
|
||
describe('COBOL COPY-target index hoist — output parity with the pre-change scans (#2908)', () => {
|
||
it('agrees with the verbatim pre-change resolver over fail-open corpora (no well-known copybook dir)', () => {
|
||
let checked = 0;
|
||
for (let repo = 0; repo < REPOS; repo++) {
|
||
const files = corpusFromDirs(repo, 6 + (repo % 25), FAIL_OPEN_DIRS);
|
||
expect(hasPreferredCopyDir(files), `repo=${repo} accidentally grew a copybook dir`).toBe(
|
||
false,
|
||
);
|
||
for (const target of TARGETS) {
|
||
expect(resolve(target, files), `cobol "${target}" repo=${repo}`).toEqual(
|
||
legacyResolveCobolImportTarget(target, files),
|
||
);
|
||
checked++;
|
||
}
|
||
}
|
||
expect(checked).toBe(REPOS * TARGETS.length);
|
||
});
|
||
|
||
it('COPY EXTERNAL misses vendor/EXTERNAL.cpy when a copybooks/ dir is present (#2967)', () => {
|
||
const files = new Set(['copybooks/CUSTREC.cpy', 'vendor/EXTERNAL.cpy', 'src/PROG.cbl']);
|
||
expect(resolve('EXTERNAL', files)).toBeNull();
|
||
expect(resolve('CUSTREC', files)).toBe('copybooks/CUSTREC.cpy');
|
||
});
|
||
|
||
it('fail-open: without a copybook dir, COPY EXTERNAL still basename-matches (#2967)', () => {
|
||
const files = new Set(['vendor/EXTERNAL.cpy', 'src/PROG.cbl']);
|
||
expect(resolve('EXTERNAL', files)).toBe('vendor/EXTERNAL.cpy');
|
||
});
|
||
|
||
it('P1-A: uppercase .CPY extension does not break stem extraction', () => {
|
||
const files = new Set(['copybooks/CUSTREC.CPY', 'src/PROG.cbl']);
|
||
expect(resolve('CUSTREC', files)).toBe('copybooks/CUSTREC.CPY');
|
||
});
|
||
|
||
it('P1-B: polyglot file set with copy/cpy segments does not latch preferred-class', () => {
|
||
const files = new Set([
|
||
'docs/copy/README.md',
|
||
'src/copy/clipboard.ts',
|
||
'copybooks/CUSTREC.cpy',
|
||
'vendor/EXTERNAL.cpy',
|
||
'src/PROG.cbl',
|
||
]);
|
||
expect(resolve('CUSTREC', files)).toBe('copybooks/CUSTREC.cpy');
|
||
expect(resolve('EXTERNAL', files)).toBeNull();
|
||
});
|
||
|
||
it('the corpus actually resolves things (the parity arm is not vacuous)', () => {
|
||
// A corpus that resolved nothing would make the arm above pass on
|
||
// `null === null` forever. Measured on this corpus: 390 hits.
|
||
let hits = 0;
|
||
for (let repo = 0; repo < REPOS; repo++) {
|
||
const files = corpus(repo, 6 + (repo % 25));
|
||
for (const target of TARGETS) {
|
||
hits += legacyResolveCobolImportTarget(target, files) === null ? 0 : 1;
|
||
}
|
||
}
|
||
expect(hits).toBeGreaterThan(300);
|
||
});
|
||
});
|
||
|
||
// ─── hand-built tie-breaks ───────────────────────────────────────────────────
|
||
|
||
/**
|
||
* `path` decides where the stem of a backslash path starts: on POSIX the whole
|
||
* `dir\sub\BOOK` is the stem, on Windows only `BOOK`. Deriving the target
|
||
* through the SAME call keeps the two arms below a hit and a miss respectively
|
||
* on both platforms, so what they pin is the `path` semantics rather than the
|
||
* host — and a rewrite that replaces `path` with slicing on `/` changes what
|
||
* they resolve to on Windows.
|
||
*/
|
||
const BACKSLASH_FILE = 'dir\\sub\\BOOK.cpy';
|
||
const BACKSLASH_STEM = path.basename(BACKSLASH_FILE, '.cpy').toUpperCase();
|
||
/** The stem's last backslash-delimited segment — a HIT only where `path` splits on `\`. */
|
||
const BACKSLASH_LEAF = 'BOOK';
|
||
const BACKSLASH_LEAF_EXPECTED = BACKSLASH_STEM === BACKSLASH_LEAF ? BACKSLASH_FILE : null;
|
||
|
||
interface HandBuilt {
|
||
readonly why: string;
|
||
/** Insertion order IS the Set-iteration order, and for most arms it IS the tie-break. */
|
||
readonly files: readonly string[];
|
||
readonly target: string;
|
||
/** The one path (or `null`) both implementations must return. */
|
||
readonly expected: string | null;
|
||
}
|
||
|
||
const HANDBUILT: readonly HandBuilt[] = [
|
||
{
|
||
why: 'a copybook beats a COBOL source that comes FIRST in Set order (tier order)',
|
||
files: ['src/BOOK.cbl', 'copybooks/BOOK.cpy'],
|
||
target: 'BOOK',
|
||
expected: 'copybooks/BOOK.cpy',
|
||
},
|
||
{
|
||
why: '.copybook is tier 1 too, and beats an earlier .cob',
|
||
files: ['src/BOOK.cob', 'copybooks/BOOK.copybook'],
|
||
target: 'BOOK',
|
||
expected: 'copybooks/BOOK.copybook',
|
||
},
|
||
{
|
||
why: 'the source tier answers only when every copybook has missed',
|
||
files: ['copybooks/OTHER.cpy', 'src/BOOK.cbl'],
|
||
target: 'BOOK',
|
||
expected: 'src/BOOK.cbl',
|
||
},
|
||
{
|
||
why: 'within the copybook tier, first in Set order wins',
|
||
files: ['a/BOOK.cpy', 'b/BOOK.cpy'],
|
||
target: 'BOOK',
|
||
expected: 'a/BOOK.cpy',
|
||
},
|
||
{
|
||
why: 'within the source tier, first in Set order wins',
|
||
files: ['b/BOOK.cbl', 'a/BOOK.cob', 'c/BOOK.cobol'],
|
||
target: 'BOOK',
|
||
expected: 'b/BOOK.cbl',
|
||
},
|
||
{
|
||
why: 'the basename is compared UPPER-cased, so a lower-case file answers an upper-case COPY',
|
||
files: ['copybooks/custrec.cpy'],
|
||
target: 'CUSTREC',
|
||
expected: 'copybooks/custrec.cpy',
|
||
},
|
||
{
|
||
why: 'the TARGET is upper-cased too, so a lower-case COPY reaches an upper-case file',
|
||
files: ['copybooks/CUSTREC.cpy'],
|
||
target: 'custrec',
|
||
expected: 'copybooks/CUSTREC.cpy',
|
||
},
|
||
{
|
||
why: 'the extension is matched LOWER-cased for tier, stem stripped with lowercase ext, so `Foo.CPY` is reachable as FOO',
|
||
files: ['copybooks/Foo.CPY'],
|
||
target: 'FOO',
|
||
expected: 'copybooks/Foo.CPY',
|
||
},
|
||
{
|
||
why: 'a `.CPY` file (uppercase ext) is keyed without suffix, hits in tier 1',
|
||
files: ['x/BOOK.cbl', 'y/BOOK.CPY'],
|
||
target: 'BOOK',
|
||
expected: 'y/BOOK.CPY',
|
||
},
|
||
{
|
||
why: 'an uppercase source extension is a source file (`.CBL` → tier 2, keyed without suffix)',
|
||
files: ['src/Pay.CBL'],
|
||
target: 'PAY',
|
||
expected: 'src/Pay.CBL',
|
||
},
|
||
{
|
||
why: 'a file with NO extension never answers a COPY',
|
||
files: ['copybooks/BOOK'],
|
||
target: 'BOOK',
|
||
expected: null,
|
||
},
|
||
{
|
||
why: 'a non-COBOL extension never answers a COPY',
|
||
files: ['copybooks/BOOK.txt', 'docs/BOOK.md'],
|
||
target: 'BOOK',
|
||
expected: null,
|
||
},
|
||
{
|
||
why: 'a target matching nothing resolves to null',
|
||
files: ['copybooks/BOOK.cpy', 'src/PROG.cbl'],
|
||
target: 'NOSUCHBOOK',
|
||
expected: null,
|
||
},
|
||
{
|
||
why: 'an empty target matches nothing (no file has an empty stem)',
|
||
files: ['copybooks/BOOK.cpy', 'src/PROG.cbl'],
|
||
target: '',
|
||
expected: null,
|
||
},
|
||
{
|
||
why: 'a backslash path is addressed by the stem `path` reports for it',
|
||
files: [BACKSLASH_FILE],
|
||
target: BACKSLASH_STEM,
|
||
expected: BACKSLASH_FILE,
|
||
},
|
||
{
|
||
why: 'its trailing segment is a hit only where `path` treats `\\` as a separator',
|
||
files: [BACKSLASH_FILE],
|
||
target: BACKSLASH_LEAF,
|
||
expected: BACKSLASH_LEAF_EXPECTED,
|
||
},
|
||
];
|
||
|
||
describe('COBOL COPY-target index hoist — hand-built tie-breaks (#2908)', () => {
|
||
it.each(HANDBUILT)('$why', ({ files, target, expected }) => {
|
||
const set = new Set(files);
|
||
// Two assertions, not one: agreeing with the legacy copy proves the hoist
|
||
// preserved the behaviour, and pinning the literal proves the behaviour
|
||
// being preserved is the one the case is named for — `toEqual(null)` on
|
||
// both sides would otherwise satisfy an arm that stopped resolving.
|
||
expect(legacyResolveCobolImportTarget(target, set), `legacy: ${target}`).toBe(expected);
|
||
expect(resolve(target, set), `new: ${target}`).toBe(expected);
|
||
});
|
||
});
|