scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-08-28 05:25:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

feat: add COBOL language support with regex extraction pipeline (#498)

Browse source 
* feat: add COBOL language support with regex extraction pipeline

Standalone COBOL processor following the markdown-processor.ts pattern:
- No LanguageProvider modification — COBOL uses regex, not tree-sitter
- No SupportedLanguages enum change — standalone processor pattern

New files:
- cobol-processor.ts — orchestrator (processCobol, isCobolFile, isJclFile)
- cobol/cobol-preprocessor.ts — regex state machine extraction (~888 LOC)
- cobol/cobol-copy-expander.ts — COPY statement expansion with circular detection
- cobol/jcl-parser.ts — JCL job/step/DD extraction
- cobol/jcl-processor.ts — JCL graph node creation

Extraction produces:
- Module nodes (PROGRAM-ID)
- Function nodes (paragraphs)
- Namespace nodes (sections)
- Property nodes (data items)
- CALLS edges (PERFORM intra-file, CALL cross-program)
- IMPORTS edges (COPY statements)
- CONTAINS edges (section → paragraph hierarchy)

Pipeline integration: single processCobol() call in Phase 2.6

54 new tests (33 COBOL + 21 JCL), all 3889 tests pass.

* docs: document custom processor pattern in pipeline.ts

Add comment block at the custom processor integration point
documenting the pattern for future non-tree-sitter language additions.

* feat(cobol): enrich graph with EXEC SQL/CICS, ENTRY points, MOVE data flow, PERFORM THRU

Maps the remaining 60% of CobolRegexResults to the graph:
- EXEC SQL blocks → CodeElement nodes + ACCESSES edges to DB tables
- EXEC CICS LINK/XCTL → CodeElement nodes + cross-program CALLS edges
- ENTRY points → Constructor nodes (registered for cross-program resolution)
- MOVE statements → ACCESSES edges (read/write data flow tracking)
- PERFORM THRU → expanded CALLS edges for range targets
- File declarations → Record nodes with assignment metadata
- Cross-program CALL 2nd pass: resolves unresolved targets after all programs processed

* test(cobol): add 26 integration tests with exact assertions + fix CICS resolution bug

Integration tests (test/integration/resolvers/cobol.test.ts):
- 26 tests covering full COBOL system extraction
- ALL assertions use exact toBe(N) — zero fuzzy assertions
- Fixtures: CUSTUPDT.cbl, AUDITLOG.cbl, CUSTDAT.cpy, RPTGEN.cbl, RUNJOBS.jcl

Bug fix (cobol-processor.ts):
- CICS LINK/XCTL cross-program resolution was broken — edges were
  created with "resolved" reason but pointing to <unresolved> targets
- Fix: use cics-link-unresolved / cics-xctl-unresolved suffix pattern
  matching the existing cobol-call-unresolved pattern
- Second-pass resolver now patches both CALL and CICS unresolved edges

All 3915 tests pass, 0 failures.

* test(cobol): exhaustive 57-test suite with strict exact assertions

Complete rewrite of COBOL integration tests using ground-truth approach:
dump the full graph, then assert EVERY node and EVERY edge.

57 tests across 9 sections:
- Node completeness: Module(3), Function(13), Namespace(2), Property(21),
  Record(1), CodeElement(8), Constructor(1) — exact sorted arrays
- Edge completeness: 22 tests covering every type+reason combination
  with exact source→target pairs
- Cross-program resolution: 6 tests verifying CALL, CICS LINK/XCTL, JCL
- COPY expansion: copybook data items in RPTGEN
- Section hierarchy: exact paragraph membership per section
- Data item ownership: exact per-module breakdown
- MOVE data flow: exact read/write pairs
- JCL integration: job/step/dataset containment
- Grand totals: CALLS(22), CONTAINS(48), IMPORTS(1), ACCESSES(7)

Fixture enhancements:
- CUSTUPDT.cbl: added INIT-SECTION + PROCESSING-SECTION, PERFORM THRU
- AUDITLOG.cbl: added ENTRY "AUDITLOG-BATCH"
- RPTGEN.cbl: added EXEC CICS XCTL

Zero fuzzy assertions — every expect uses toBe(N) or toEqual([...sorted]).

* fix(cobol): add removeRelationship API + single-quote CALL/COPY/ENTRY, PERFORM keyword skip

Phase 0A: Add removeRelationship(id) to KnowledgeGraph interface and
implementation (trivial Map.delete wrapper). Required for orphan edge
cleanup in next commit.

Phase 1A (from PR #500 review, modified):
- RE_CALL and RE_COPY_QUOTED now match both "double" and 'single' quotes
- parseSingleCopyStatement in copy-expander updated for single quotes
- PERFORM_KEYWORD_SKIP set prevents UNTIL/VARYING/WITH/TEST/FOREVER
  from being stored as false-positive perform targets
- Sequence number stripping uses /[^0-9 ]/ (preserves numeric seq numbers
  unlike PR #500's /\S/ which stripped them)
- Normalized || to ?? for regex group extraction in copy-expander

5 new graph unit tests, all 57 COBOL integration tests pass.

* fix(cobol): RE_ENTRY single-quote + remove orphan unresolved CALLS edges

Phase 1B: RE_ENTRY regex now supports both "double" and 'single' quoted
ENTRY targets. Uses named intermediates (entryName, usingClause) with ??
operator. USING capture group shifted from [2] to [3].

Phase 1C: Second-pass resolution now collects resolved orphan edge IDs
during iteration and removes them after the loop completes, using the new
graph.removeRelationship() API. Graph no longer contains phantom
<unresolved>: edges alongside their resolved replacements. CALLS count
drops from 22 to 18 (4 orphan edges removed).

* fix(cobol): Property ID collisions + O(1) Map lookup for MOVE edges

Phase 1D+3C (atomic): Property node IDs now use composite key
filePath:section:level:name instead of filePath:name. This prevents
duplicate data item names in different sections (e.g., STATUS in both
WORKING-STORAGE and LINKAGE) from silently colliding.

New generatePropertyId() helper ensures both node creation and MOVE
edge lookup use the identical key formula. buildDataItemMap() replaces
the O(n) findDataItemNode linear scan with O(1) Map lookup, built once
per file before MOVE processing.

* feat(cobol): MOVE multi-target extraction with OF/IN qualifier filtering

MOVE X TO A B C now produces write edges for all targets, not just the
first. extractMoveTargets() helper handles OF/IN qualified names
(WS-NAME OF WS-RECORD -> target is WS-NAME), subscript stripping
(WS-TABLE(I) -> WS-TABLE), and MOVE_SKIP filtering on targets.

Data model: CobolRegexResults.moves.to:string -> targets:string[]
MOVE CORRESPONDING stays single-target per COBOL standard.
Processor MOVE loop now iterates move.targets.

* feat(cobol): COPY IN/OF library, pseudotext REPLACING, dynamic CALL, PERFORM TIMES, CICS MAP unquoted

Phase 2B: COPY ... IN/OF library-name now captured as metadata in
CopyResolution (IN and OF are synonyms per COBOL-85 standard).

Phase 2C: COPY REPLACING ==pseudotext== support. Tokenizer handles
==...== delimiters alongside "quoted" strings. Pseudotext forces EXACT
type. Two-pass applyReplacing: first pass handles space-containing/
non-identifier pseudotext via global string replace; second pass handles
identifier-level LEADING/TRAILING/EXACT. New test file
cobol-copy-expander.test.ts with 10 tests.

Phase 2E: PERFORM WS-COUNT TIMES no longer produces a false-positive
perform target (checks for TIMES keyword after captured identifier).

Phase 2F: Dynamic CALL via data item (CALL WS-PROG-NAME without quotes)
now emits a CodeElement annotation node with description 'dynamic-call'
instead of silently ignoring. Adds isQuoted:boolean to call results.

Phase 3A: CICS MAP(WS-MAP-NAME) unquoted identifiers now captured.
Phase 3B: Normalized || to ?? in copy-expander (done in Phase 1A).

* feat(cobol): nested program support — capture multiple PROGRAM-IDs per file

Phase 2D: The state machine now captures all PROGRAM-IDs, not just the
first. The primary program name stays in programName; additional nested
programs go into nestedPrograms[]. The processor creates separate Module
nodes for each nested program, contained by the outer module, and
registers them in moduleNodeIds for cross-program CALL resolution.

Paragraphs/data items are not yet scoped per-program (attributed to the
outer module) — full per-program scoping is a future enhancement that
requires END PROGRAM boundary tracking in the state machine.

* test(cobol): expand integration tests for all new language features

New fixtures:
- NESTED.cbl — two PROGRAM-IDs (OUTER-PROG, INNER-PROG) for nested
  program support testing
- COPYLIB.cpy — copybook for pseudotext REPLACING test target

Modified fixtures:
- CUSTUPDT.cbl — single-quoted ENTRY 'ALTENTRY', multi-target MOVE
  (WS-AMT TO FIELD-A FIELD-B), dynamic CALL WS-PROG-NAME, COPY COPYLIB
  with pseudotext REPLACING, LINKAGE SECTION with LS-PARAM
- RPTGEN.cbl — PERFORM WS-COUNT TIMES (false-positive guard), unquoted
  MAP(WS-MAP-NAME), additional data items WS-COUNT WS-MAP-NAME

Integration test rewritten with 62 exact assertions covering:
- 5 Module, 17 Function, 33 Property, 9 CodeElement, 2 Constructor nodes
- Nested program containment (OUTER-PROG -> INNER-PROG)
- Dynamic CALL annotation (CodeElement with cobol-dynamic-call)
- Multi-target MOVE (UPDATE-BALANCE: 2 reads, 3 writes)
- Single-quoted ENTRY (ALTENTRY under CUSTUPDT)
- PERFORM TIMES guard (WS-COUNT not in CALLS)
- Orphan unresolved edge removal (zero -unresolved edges)
- Grand totals: 21 CALLS, 68 CONTAINS, 2 IMPORTS, 10 ACCESSES

* fix(cobol): pseudotext REPLACING now applies correctly via isPseudotext flag

Root cause: ==PREFIX-== matched /^[A-Z][A-Z0-9-]*$/i (trailing hyphens
allowed), routing it to the second-pass EXACT identifier match where
PREFIX-RECORD !== PREFIX- failed silently.

Fix: Propagate isPseudotext from parseReplacingClause to CopyReplacing
interface, then use it in applyReplacing first-pass condition to force
global string replacement for all pseudotext entries regardless of
whether the content looks like an identifier.

Result: COPY COPYLIB REPLACING ==PREFIX-== BY ==WS-==. now correctly
transforms PREFIX-RECORD → WS-RECORD, PREFIX-CODE → WS-CODE, etc.

* refactor(cobol): per-program scoping via boundary tracking + line-range grouping

State machine changes (minimal, ~30 lines):
- Add RE_END_PROGRAM regex for END PROGRAM program-name. detection
- Replace nestedPrograms[] with programs[] containing startLine/endLine/
  nestingDepth metadata for each PROGRAM-ID in the file
- Reset division/section/paragraph state on new PROGRAM-ID boundary
- EOF finalization flushes remaining stack entries (single-program files)
- Programs sorted by startLine (outer before inner)

Processor changes:
- Uses programs[] with line-range containment to find enclosing parent
  Module for nested programs (replaces hardcoded nestedParent logic)
- programModuleIds Map tracks Module node IDs per program name

Fixture: NESTED.cbl now includes END PROGRAM lines for both programs.

Integration test: PREFIX-* Property nodes now correctly appear as WS-*
after the pseudotext REPLACING fix from the previous commit.

* feat(cobol): free-format COBOL support (>>source free)

Auto-detects >>SOURCE FREE directive in the first 500 chars and switches
to free-format line processing:
- No column-position rules (cols 1-6 are program text, not sequence area)
- Comments use *> prefix instead of col 7 indicator
- No continuation line indicator
- Strip inline *> comments
- Skip >>SOURCE directive lines

preprocessCobolSource() skips col-1-6 stripping for free-format files.

Paragraph/section regexes relaxed from fixed 7-space prefix to flexible
whitespace with case-insensitivity (/^\s*([A-Z][A-Z0-9-]+)\.\s*$/i).
EXCLUDED_PARA_NAMES expanded with COBOL verbs (GOBACK, END-READ, etc.)
to prevent false-positive paragraph detection in free-format.

Also fixes: entry-point-scoring.ts crash when language is 'cobol'
(MERGED_ENTRY_POINT_PATTERNS[language] was undefined → optional chaining).

Benchmark on ACAS 3.01 (268 GnuCOBOL free-format programs, 10MB):
- Before: 407 nodes, 393 edges (near-empty, only file nodes)
- After:  4,297 nodes, 3,612 edges, 542 clusters, 11 flows

* fix(cobol): relax data item regexes for free-format (^\s+ to ^\s*)

RE_FD, RE_DATA_ITEM, RE_ANONYMOUS_REDEFINES, and RE_88_LEVEL all used
^\s+ which requires at least 1 leading space. In free-format mode, lines
are trimmed before processing, so data items like "01 WS-FIELD PIC X."
have no leading whitespace after trimming.

Changed to ^\s* (zero or more spaces) which works for both fixed-format
(indented lines still have spaces) and free-format (trimmed lines).

ACAS benchmark (268 GnuCOBOL programs):
- Before: 4,297 nodes, 3,612 edges (paragraphs only)
- After:  13,832 nodes, 8,615 edges (+ data items, FDs, 88-levels)

* feat(cobol): 100% structural feature coverage — GO TO, SCREEN, SD/RD, SORT, SEARCH, CANCEL, Level 66

New extractions: GO TO (CALLS edges), SCREEN SECTION data items,
SD/RD alongside FD (Record nodes), SORT/MERGE USING/GIVING (ACCESSES),
SEARCH (ACCESSES), CANCEL (CALLS), Level 66 RENAMES (Property),
IS EXTERNAL/IS GLOBAL (Property description enrichment).

ACAS: 13,951 nodes | 13,193 edges | 685 clusters | 150 flows
(+53% edges from new GO TO/SORT/SEARCH/CANCEL extractions)

* feat(cobol): enriched CICS extraction — file I/O, dynamic PROGRAM, queues, HANDLE ABEND

EXEC CICS blocks now extract:
- FILE/DATASET clause: captures VSAM file name (literal or data item ref)
  for READ/WRITE/REWRITE/DELETE/STARTBR/READNEXT/READPREV → ACCESSES edges
- PROGRAM clause: now handles unquoted variable references (dynamic CICS
  program transfer) → CodeElement annotation with cics-dynamic-program reason
- QUEUE clause: captures TS/TD queue names from WRITEQ/READQ → ACCESSES edges
- LABEL clause: captures HANDLE ABEND error handler targets → CALLS edges
- TRANSID: now handles unquoted variable references

CodeElement descriptions enriched with all captured fields (map, program,
transid, file, queue, label).

CardDemo benchmark: +49 nodes, +33 edges from enriched CICS extraction.

* feat(cobol): complete CICS command extraction — all 7 expert recommendations

From COBOL expert agent analysis:
1. ENDBR added to isRead file command list
2. LOAD added to PROGRAM edge commands (alongside LINK/XCTL)
3. Two-word commands expanded: WRITEQ/READQ/DELETEQ TS/TD, HANDLE
   ABEND/AID/CONDITION, START TRANSID
4. Queue reason differentiated: cics-queue-read/-write/-delete
5. RETURN/START TRANSID → CALLS edges to synthetic <transid> target
6. MAP → ACCESSES edges for screen traceability
7. INTO/FROM data fields extracted → ACCESSES edges to data items

Also: dataItemMap built before CICS block processing (was declared after),
CodeElement descriptions enriched with all captured CICS fields.

* test(cobol): strict exhaustive integration tests with exact edgeSet assertions

Every edge reason has exact sorted pair assertions via edgeSet(), not
just counts. Any change to extraction that adds, removes, or reorders
edges will produce a precise, descriptive failure.

Updated RPTGEN.cbl fixture with:
- GO TO EXIT-PARAGRAPH, SORT USING/GIVING, SEARCH table
- EXEC CICS READ FILE INTO, WRITEQ TS QUEUE FROM, SEND MAP FROM
- EXEC CICS HANDLE ABEND LABEL, RETURN TRANSID, XCTL PROGRAM(variable)
- ABEND-HANDLER and EXIT-PARAGRAPH paragraphs

46 tests covering 24 CALLS + 79 CONTAINS + 18 ACCESSES + 2 IMPORTS edges
across 15 distinct edge reason codes, all with exact sorted pair lists.

* fix(cobol): address 5 findings from second Claude review (compiler front-end perspective)

Finding #2: Numeric sequence numbers now stripped (changed /[^0-9 ]/ to
/\S/ in preprocessCobolSource). Lines like "000100 MAIN-PARAGRAPH." now
have cols 1-6 blanked so paragraph regex matches correctly.

Finding #11: JCL in-stream PROC ordering fixed — pre-register all PROCs
into moduleNames before step processing. Steps that EXEC a PROC defined
later in the same file now get CALLS edges.

Finding #A: PROCEDURE DIVISION USING no longer captures calling-convention
keywords (BY, VALUE, REFERENCE, CONTENT, ADDRESS, OF) as parameter names.

Finding #C: SORT/MERGE USING/GIVING now captures ALL file references
(multi-file), not just the first. Changed from single-match to section
extraction with split.

Finding #D: Section headers no longer set currentParagraph, preventing
PERFORM caller misattribution to Namespace instead of Function nodes.

* fix(cobol): address code review findings — ReDoS fix, perf, cleanup

P1 CRITICAL — ReDoS in SORT USING/GIVING:
Replaced nested-quantifier regex with safe indexOf+substring+split
approach. No backtracking possible on crafted input.

P2 — readCopy O(M) linear scan:
Added copybookByPath reverse Map for O(1) path-to-content lookup.

P3 — Dead code removal:
Deleted unused RE_SORT_USING and RE_SORT_GIVING constants.

P3 — EXCLUDED_PARA_NAMES simplification:
Replaced 20 END-* entries with startsWith('END-') prefix check.
Auto-covers future END-* verbs.

P3 — Misplaced JSDoc on removeRelationship:
Fixed comment that described removeNodesByFile instead.
Added missing JSDoc to removeNodesByFile.

Review agents: architecture-strategist, performance-oracle,
security-sentinel, code-simplicity-reviewer.

* refactor: add Cobol to SupportedLanguages with parseStrategy: standalone

New languages/cobol.ts — standalone regex processor provider with no-op
tree-sitter fields. Declares parseStrategy: 'standalone' to distinguish
from tree-sitter-based languages.

Added parseStrategy: 'tree-sitter' | 'standalone' to LanguageProviderConfig
for languages that use their own processor instead of tree-sitter.

Removed all 11 'cobol' as any casts — now uses SupportedLanguages.Cobol.
Added empty Cobol entries to entry-point-scoring and framework-detection.

* fix(cobol): 5 fixes from third Claude review + 3 regression tests

Fixes:
- Line numbers now 1-indexed in fixed-format (was 0-indexed, off-by-one
  in jump-to-definition links)
- Copybook content preprocessed before COPY expansion (sequence numbers
  and patch markers in copybooks no longer survive into expanded source)
- ENTRY USING filters calling-convention keywords (BY, VALUE, REFERENCE,
  CONTENT, ADDRESS, OF) — same fix as PROCEDURE DIVISION USING
- SORT/MERGE trailing period stripped from USING/GIVING file tokens
- Paragraph exclusion uses exact match for SECTION/DIVISION (was substring
  match that excluded valid names like CROSS-SECTION-ANALYSIS)

USING_KEYWORDS moved to module scope for reuse by both PROCEDURE DIVISION
USING and ENTRY USING handlers.

New unit tests:
- ENTRY USING BY VALUE filtering
- Paragraph names containing SECTION not excluded
- Numeric sequence numbers stripped enabling paragraph detection

* fix(cobol): address 6 findings from fourth Claude review + tests

Fourth review findings fixed:
- New #IV: PERFORM TIMES guard uses perfMatch.index instead of
  line.indexOf (prevents wrong match when target appears earlier in line)
- New #V: 88-level condition values now handle single-quoted literals
  ('Y' no longer stored with embedded quotes)
- New #I: CANCEL edges use two-pass resolution like CALL (no longer
  silently dropped when target indexed after source)
- New #3: Multi-line SORT/MERGE accumulation — sortAccum state variable
  accumulates lines until period, then extracts USING/GIVING from full
  statement (95% of production SORT statements span multiple lines)
- New #II: PROCEDURE DIVISION USING on split lines — pendingProcUsing
  flag defers parameter capture to next line if USING not on same line
- New #6 (prior): EXCLUDED_PARA_NAMES exact match for SECTION/DIVISION

Updated fixture: RPTGEN.cbl SORT now uses multi-line format with GIVING
on separate line (period-terminated). New sort-giving integration test.
ACCESSES total: 18 → 19 (new sort-giving edge from multi-line capture).

* fix(cobol): address 4 findings from fifth Claude review

Finding #B (5 reviews old): Section/paragraph node IDs now include
enclosing program name to prevent collision when nested programs share
section/paragraph names. New findOwningProgramName() helper uses
programs[] line ranges to find the innermost enclosing program.

Finding #α: pendingProcUsing now reset in the if(procUsingMatch) branch
(was only set in else branch, could leak across nested programs).

Finding #β: RE_CALL_DYNAMIC uses negative lookbehind (?<![A-Z0-9-]) to
prevent false-positive on compound identifiers like WS-CALL OCCURS.

Finding #γ: sortAccum flushed at EOF (parallel to flushSelect and
pendingFdName EOF cleanup). Prevents silent loss of SORT USING/GIVING
relationships in truncated files.

* fix(cobol): address findings from reviews 5+6 with full test coverage

Review 5 fixes:
- #α: pendingProcUsing reset in if(procUsingMatch) branch
- #β: RE_CALL_DYNAMIC negative lookbehind prevents WS-CALL false positive
- #γ: sortAccum flushed at EOF for truncated files
- #B: Section/paragraph IDs include owning program name

Review 6 fixes:
- #P: sectionNodeIds/paraNodeIds maps use program-scoped keys
  (PROGNAME:NAME). New scopedParaLookup/scopedCallerLookup helpers.
  findContainingSection updated with programs parameter.
- #Q: RETURNING added to USING_KEYWORDS for COBOL 2002+
- #R: RE_PERFORM matches both THRU and THROUGH via alternation

New unit tests (6):
- PERFORM THROUGH captures thruTarget
- PROCEDURE DIVISION USING RETURNING filters keyword
- RE_CALL_DYNAMIC no false-match on WS-CALL compound identifier
- Multi-line SORT captures USING/GIVING from continuation lines
- PROCEDURE DIVISION USING on split line via pendingProcUsing
- Copybook preprocessing strips sequence numbers

* fix(cobol): address findings from seventh Claude review + 3 tests

Review 7 fixes:
- #i: findContainingSection only updates best when lookup succeeds
  (prevents undefined overwriting valid parent section)
- #ii: RE_PROC_SECTION handles segment numbers (SECTION 30.)
- #III: procedureUsing now stored per-program on boundary stack
  entries, propagated to programs[] output. Inner programs no longer
  overwrite outer program's parameters.
- #δ: Dynamic CANCEL (CANCEL variable) now creates CodeElement
  annotation node, matching dynamic CALL behavior. RE_CANCEL_DYNAMIC
  with negative lookbehind. cancels[] gains isQuoted field.
- #Q: RETURNING added to USING_KEYWORDS (already in prev commit)
- #R: PERFORM THROUGH already fixed (THRU|THROUGH alternation)

New unit tests:
- Nested programs carry per-program procedureUsing
- SECTION with segment number detected
- Dynamic CANCEL via data item captured with isQuoted=false

* feat(cobol): link PROCEDURE DIVISION USING to LINKAGE data items + close 4 findings

Finding #10 FIXED: procedureUsing parameters now create ACCESSES edges
with reason 'cobol-procedure-using' from Module to matching LINKAGE
SECTION Property nodes. This exposes the program's parameter contract
in the graph (e.g., AUDITLOG → LS-CUST-ID, AUDITLOG → LS-AMOUNT).

Findings closed by expert agent consensus:
- #6 COPY IN library: WONTFIX — captured metadata, no universal
  library-to-directory mapping exists. Field costs nothing and is useful
  for library queries.
- #14 SQL DELETE: WONTFIX — DB2 requires FROM; existing FROM pattern
  handles it. Bare DELETE would risk false positives.
- #E OCCURS DEPENDING ON: WONTFIX — runtime sizing concern, not
  structural. The static occurs count is sufficient for indexing.

All 39 findings from 7 Claude reviews now resolved or closed.

* fix(cobol): resolve 48 review findings across 9 review cycles

Ninth deep review resolved all remaining COBOL parser gaps identified
by 5 specialist agents (COBOL expert, architecture strategist,
TypeScript reviewer, security sentinel, code simplicity reviewer).

Fixes (P1 — critical):
- SELECT OPTIONAL now correctly skips OPTIONAL keyword (C1)
- RETURNING params excluded from PROCEDURE DIVISION USING list (C7)
- SORT GIVING no longer captures clause keywords as file names (C5)
- Extract flushSort() helper eliminating 40-line duplication (S2)
- Flush unclosed EXEC blocks at EOF matching SORT/SELECT pattern (S3)
- Guard undefined map key in jcl-processor moduleNames (S1)
- Add MAX_TOTAL_EXPANSIONS=500 to prevent exponential COPY breadth (S4)

Fixes (P2 — important):
- Quote-aware stripInlineComment for | and *> in string literals (C2+C3)
- Fixed-format literal continuation now handles quoted strings (C6)
- PROGRAM-ID detected regardless of division state for siblings (C9)

Fixes (P3 — cleanup):
- EXEC SQL INTO restricted to INSERT INTO to avoid FETCH false-pos (C8)
- Copy expander line numbers fixed from 0-based to 1-based (C11)
- Remove dead code: inInStreamProc, fileIsLiteral, expansionDepth (S7-S10)

Also fixes 8th-review findings: nested program CONTAINS attribution,
multi-PERFORM on same line, INPUT/OUTPUT PROCEDURE IS in SORT,
GO TO DEPENDING ON multi-target, MOVE CORR abbreviation, per-program
procedureUsing ACCESSES edges.

Tests: 145 COBOL tests passing (59 integration + 86 unit)
Benchmarks: CardDemo 12,323 nodes/8,893 edges (7.4s)
            ACAS 14,016 nodes/15,452 edges (9.3s, -9% faster)

* docs(cobol): update documentation for ninth review cycle fixes

Update all 4 COBOL documentation files to reflect the 16 fixes
from the ninth review cycle:

- regex-extraction.md: quote-aware comment stripping, SELECT OPTIONAL,
  RETURNING exclusion, SORT_CLAUSE_NOISE filter, flushSort() helper,
  GO TO multi-target, PROGRAM-ID division-independent detection
- copy-expansion.md: MAX_TOTAL_EXPANSIONS=500 breadth guard, 1-based
  line numbers, removed expansionDepth/warnedCircular param
- deep-indexing.md: GO TO DEPENDING ON, INPUT/OUTPUT PROCEDURE IS,
  MOVE CORR edge reasons, INSERT INTO restriction, literal continuation
- performance.md: updated benchmarks (CardDemo 12,323n/8,893e/7.4s,
  ACAS 14,016n/15,452e/9.3s), COPY breadth guard

* fix(cobol): resolve 10th review findings — nested program edge attribution

Fix 6 findings from the 10th review (PR #498 comment #4132201110):

#A+#F: All CALL/CANCEL/CICS/ENTRY/SQL/SEARCH/file-declaration edges
now use owningModuleId() for nested program attribution instead of
the outer program's parentId. Added helper function owningModuleId()
to centralize the pattern.

#B: Added USING and GIVING to SORT_CLAUSE_NOISE set to prevent MERGE
USING + OUTPUT PROCEDURE from capturing clause keywords as file names.

#C: INPUT/OUTPUT PROCEDURE regex now captures optional THRU/THROUGH
range end paragraph, mirroring RE_PERFORM's THRU support.

#D: scopedCallerLookup fallback now uses programModuleIds.get(pgm)
instead of parentId, so PERFORM/MOVE/GOTO in nested programs with
unresolvable paragraphs fall back to the correct inner module.

#E: pendingProcUsing only set when PROCEDURE DIVISION line is NOT
period-terminated, preventing false USING expectation.

Tests: 145 passing | TypeScript clean

* fix(cobol): resolve 10th review findings — nested program edge attribution

Fix 6 findings from the 10th review (PR #498 comment #4132201110):

#A+#F: All CALL/CANCEL/CICS/ENTRY/SQL/SEARCH/file-declaration edges
now use owningModuleId() for nested program attribution instead of
the outer program's parentId. Added helper function owningModuleId()
to centralize the pattern.

#B: Added USING and GIVING to SORT_CLAUSE_NOISE set to prevent MERGE
USING + OUTPUT PROCEDURE from capturing clause keywords as file names.

#C: INPUT/OUTPUT PROCEDURE regex now captures optional THRU/THROUGH
range end paragraph, mirroring RE_PERFORM's THRU support.

#D: scopedCallerLookup fallback now uses programModuleIds.get(pgm)
instead of parentId, so PERFORM/MOVE/GOTO in nested programs with
unresolvable paragraphs fall back to the correct inner module.

#E: pendingProcUsing only set when PROCEDURE DIVISION line is NOT
period-terminated, preventing false USING expectation.

Tests: 145 passing | TypeScript clean

* fix(cobol): resolve 11th review findings — final nested program + multi-CALL gaps

#1: scopedCallerLookup(null) now uses owningModuleId(lineNum) instead
of parentId, fixing PERFORM/MOVE/GOTO before first paragraph in nested
programs.

#2+#3: CALL and CANCEL extraction now uses matchAll (global flag) to
capture multiple occurrences on the same line. Dynamic CALL/CANCEL
checked independently instead of in else branch.

#4: SORT/MERGE ACCESSES edge IDs now use owningModuleId(sort.line)
instead of parentId for nested program correctness.

#5: preprocessCobolSource free-format detection now uses first 10 lines
(consistent with extractCobolSymbolsWithRegex threshold).

#6: EXCLUDED_PARA_NAMES expanded with DISPLAY, ACCEPT, WRITE, READ,
REWRITE, DELETE, OPEN, CLOSE, RETURN, RELEASE, SORT, MERGE to prevent
false-positive paragraph detection on isolated verbs.

Also removed unused GraphNode import from cobol-processor.ts.

Tests: 145 passing | TypeScript clean

* docs(cobol): deepened full language coverage plan with research findings

3 research agents analyzed Phase 1-2 features and graph value ranking.

Key findings: cobol-call-using is #1 edge type (9.2/10); multi-line
accumulation is dominant challenge; DECLARATIVES is lowest-risk Phase 2
item; SET TO TRUE covers 80-90% of SET usage.

* feat(cobol): implement Phase 1 — high-value data flow edges

4 new extraction features that create new ACCESSES and IMPORTS edges:

1.1: EXEC SQL INCLUDE -> IMPORTS edges with reason 'sql-include'
     Handles unquoted (SQLCA), quoted ('DBRMLIB.MEMBER'), and
     underscored (CUST_TBL_DCL) member names.

1.2: CALL USING parameter extraction -> ACCESSES edges
     Extracts parameters from CALL USING clause, filtering BY/REFERENCE/
     CONTENT/VALUE/ADDRESS/OF/LENGTH/OMITTED keywords. Creates
     'cobol-call-using' ACCESSES edges (graph value: 9.2/10).

1.4: OCCURS DEPENDING ON -> ACCESSES edges with reason 'cobol-depends-on'
     Extended OCCURS regex captures DEPENDING ON field with subscript
     stripping. Creates dependency edge from table to controlling field.

1.5: VALUE clause for standard data items
     Extracts VALUE from data item clauses: quoted strings with type
     prefix (X/N/G/B), ALL literals, numerics (incl negative/decimal),
     and figurative constants. Populates Property node values.

Tests: 145 passing (+2 ACCESSES from CALL USING) | TypeScript clean

* feat(cobol): implement Phase 2 — DECLARATIVES, SET, INSPECT, EXEC DLI

4 new extraction features for error handling, data flow, and IMS/DB:

2.1: EXEC DLI (IMS/DB) -> CodeElement + ACCESSES edges
     Accumulates EXEC DLI blocks like EXEC SQL. Parses DLI verbs
     (GU, GN, ISRT, REPL, DLET, CHKP, SCHD, TERM). Extracts
     SEGMENT, PCB, INTO/FROM, PSB. Creates dli-{verb} ACCESSES
     edges to <ims>:segment Record nodes.

2.2: DECLARATIVES / USE AFTER EXCEPTION -> ACCESSES edges
     Tracks inDeclaratives state. Detects USE AFTER STANDARD
     EXCEPTION ON file-name. Creates cobol-error-handler ACCESSES
     edge from handler section to file Record.

2.3: SET statement -> ACCESSES edges
     Detects SET TO TRUE (80-90% of SET usage) and SET index
     TO/UP BY/DOWN BY. Creates cobol-set-condition / cobol-set-index
     write edges + cobol-set-read for identifier values.

2.4: INSPECT -> ACCESSES edges with multi-line accumulator
     Accumulates INSPECT until period (like SORT). Extracts inspected
     field + tally counters. Creates cobol-inspect-read/write/tally
     edges. Form detection: tallying/replacing/converting/combined.

Preprocessor: 1398 -> 1597 LOC (+199). Tests: 145 passing.

* feat(cobol): implement Phase 3 — completeness fixes

6 partial features fixed to first-class support:

3.1: CALL RETURNING -> ACCESSES write edge (cobol-call-returning)
3.2: SELECT OPTIONAL flag preserved in FileDeclaration + Record node
3.3: ALTERNATE RECORD KEY extraction (matchAll for multiple keys)
3.4: COMMON attribute on nested programs (RE_PROGRAM_ID extended)
3.5: IS EXTERNAL / IS GLOBAL as first-class boolean properties
     (removed usage string hack)
3.6: AUTHOR / DATE-WRITTEN mapped to Module node description

Tests: 145 passing | TypeScript clean

* feat(cobol): implement Phase 4 — INITIALIZE + metadata completeness

4.1: INITIALIZE statement -> ACCESSES write edge (cobol-initialize)
4.2: DATE-COMPILED and INSTALLATION paragraphs extracted and mapped
     to Module node description alongside existing AUTHOR/DATE-WRITTEN

All 4 plan phases complete. Coverage: ~95% (up from 71.9%).
Tests: 145 passing | TypeScript clean

* test(cobol): add 24 unit tests for Phase 1-4 features

Coverage for all new extraction features:

Phase 1 (8 tests):
- EXEC SQL INCLUDE (unquoted, quoted, underscored)
- CALL USING (simple, mixed modes, ADDRESS OF, OMITTED)
- CALL RETURNING
- OCCURS DEPENDING ON
- VALUE clause (string, numeric, figurative constant)

Phase 2 (10 tests):
- EXEC DLI GU/ISRT/SCHD (verb, segment, PCB, INTO, FROM, PSB)
- DECLARATIVES USE AFTER EXCEPTION (single + multiple sections)
- SET TO TRUE, SET index UP BY
- INSPECT TALLYING, INSPECT REPLACING

Phase 3-4 (6 tests):
- SELECT OPTIONAL flag
- ALTERNATE RECORD KEY
- PROGRAM-ID IS COMMON
- IS EXTERNAL / IS GLOBAL booleans
- INITIALIZE extraction
- Full programMetadata (AUTHOR, DATE-WRITTEN, DATE-COMPILED, INSTALLATION)

Total: 168 tests passing (145 + 24 - 1 removed duplicate)

* fix(cobol): use /\r?\n/ split for Windows CRLF compatibility

All 4 COBOL source files now split on /\r?\n/ instead of '\n' to
handle CRLF line endings on Windows. Previously, trailing \r in
lines caused RE_GOTO's $ anchor to fail on multi-line GO TO
DEPENDING ON statements, producing only 1 goto edge instead of 4.

Files fixed: cobol-preprocessor.ts (2 sites), cobol-processor.ts,
jcl-parser.ts, cobol-copy-expander.ts

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 12th review — dynamic CALL/CANCEL dedup + trailing anchors

#1+#2: Removed incorrect hasQuotedCall/hasQuotedCancel deduplication
guards. RE_CALL_DYNAMIC and RE_CANCEL_DYNAMIC require [A-Z] after
CALL/CANCEL, so they CANNOT match quoted targets — the guards were
both unnecessary and actively harmful, suppressing dynamic CALL/CANCEL
in ON EXCEPTION patterns.

#3+#5: Changed RE_CALL_DYNAMIC and RE_CANCEL_DYNAMIC trailing anchor
from (?:\s|\.) to (?=\s|\.|$) (lookahead). The consuming anchor
failed when the identifier was the last token on a physical line.

Tests: 168 passing | TypeScript clean

* feat(cobol): add CALL accumulator + fix SORT double-statement (#4, #6)

Finding #4: Multi-line CALL USING accumulator
Added callAccum state variable that accumulates CALL statements
spanning multiple physical lines until period or END-CALL is found.
Uses flushCallAccum() to re-extract CALL target + USING parameters
from the full accumulated statement. This fixes the silent loss of
ACCESSES parameter edges when USING appears on lines after CALL.

Finding #6: SORT double-statement on same line
After flushSort(), the code now falls through to re-check the
current line for a new SORT/MERGE start (was previously blocked
by the sortAccum === null check evaluating before flushSort ran).

Also fixed: used non-global regex for CALL detection test to avoid
the classic global regex .test() lastIndex bug.

Tests: 168 passing (+1 ACCESSES from multi-line CALL USING)

* fix(cobol): resolve 13th review — CICS LOAD, USING extraction, file scoping

#1: CICS LOAD unresolved edge no longer silently deleted in second pass.
    Changed narrow cics-link/cics-xctl check to catch-all pattern:
    rel.reason?.startsWith('cics-') && rel.reason.endsWith('-unresolved')

#2: flushCallAccum USING extraction now stops before COBOL statement
    verbs (INSPECT, SEARCH, SORT, MERGE, DISPLAY, ACCEPT, MOVE, PERFORM,
    GO TO, CALL, IF, EVALUATE). Prevents absorbing adjacent statements
    as false USING parameters in legacy pre-COBOL-85 code without END-CALL.

#3: CICS FILE Record nodes now globally-scoped (<cics-file>:FILENAME)
    instead of per-file-scoped. Enables cross-program CICS file access
    analysis, consistent with SQL table scoping (<db>:TABLE).

#4: callAccum pre-check regex now has (?<![A-Z0-9-]) lookbehind to
    prevent false activation on compound identifiers like WS-CALL-FLAG.

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 14th review — callAccum false paragraph + Area A guard

#1: callAccum continuation lines now check for COBOL statement verb
    starts (GO TO, PERFORM, MOVE, etc.) and paragraph/section headers.
    If detected, the CALL is flushed as-is and the line processed
    normally — prevents false paragraph detection and currentParagraph
    corruption from lines like "WS-ADDR." being treated as paragraphs.

#4: callAccum pre-check now guarded by currentDivision === 'procedure'
    to prevent unnecessary activations in DATA DIVISION.

#5: Fixed-format paragraph detection now rejects lines with >7 leading
    spaces (Area B indentation) as paragraph candidates. Paragraph
    names in fixed-format must start in Area A (col 8-11, max 7 spaces).
    Free-format mode is unaffected.

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 15th review — callAccum Area A + verb boundary fixes

#A: Column-position-aware paragraph detection in callAccum flush.
#B: inspectAccum early-flush on paragraph/section/verb headers.
#C: Verb boundary \b → (?:\s|$) prevents MOVE-COUNT false flush.

* test(cobol): add 17 edge-case regression tests + fix USING verb boundary

17 new tests covering all recurring review patterns:

Multi-line CALL USING (7 tests):
- Parameters on separate continuation lines (IBM mainframe style)
- No absorption of INSPECT/GO TO/paragraphs following CALL
- END-CALL scope terminator
- Hyphenated identifiers (MOVE-COUNT) not triggering false flush
- Dual quoted+dynamic CALL on same line (ON EXCEPTION)

Nested program attribution (2 tests):
- CALL in inner program within inner line range
- PERFORM before first paragraph has null caller

CRLF compatibility (1 test):
- GO TO DEPENDING ON with \r\n line endings

Area A paragraph detection (2 tests):
- Area B (>7 spaces) rejected; Area A (7 spaces) accepted

SORT/MERGE (1 test): COLLATING SEQUENCE keywords not captured
PROCEDURE USING (2 tests): RETURNING excluded, period-terminated
Comment stripping (1 test): pipe in quoted string preserved
SELECT OPTIONAL (1 test): correct file name, not OPTIONAL keyword

Bug fix: USING extraction regex verb terminators changed from
\bVERB\b to \bVERB(?=\s|$) in flushCallAccum — prevents truncation
on hyphenated identifiers like MOVE-COUNT, PERFORM-LIMIT.

Total: 185 tests passing

* test(cobol): add 32 comprehensive edge-case regression tests

13 new describe blocks covering all extraction features:

- EXEC DLI: no-SEGMENT, multi-line accumulation (2 tests)
- SET: multiple targets, DOWN BY, TO numeric (3 tests)
- INSPECT: CONVERTING, multiple counters, tallying-replacing,
  paragraph flush during accumulation (4 tests)
- DECLARATIVES: no-STANDARD keyword, I-O mode, post-END paragraphs (3)
- COPY REPLACING: pseudotext deletion ==OLD== BY ==== (1 test)
- VALUE: hex literal, negative numeric, ALL literal (3 tests)
- OCCURS: TO range, fixed-size without DEPENDING ON (2 tests)
- Dynamic CALL/CANCEL: end-of-line, multiple CANCELs (3 tests)
- EXEC SQL: INCLUDE skips tables, SELECT INTO host vars, host
  variable extraction (3 tests)
- INITIALIZE: target and caller context (1 test)
- Nested programs: sibling scoping, PROGRAM-ID without ID DIV (2)
- EXEC EOF flush: unclosed EXEC SQL flushed (1 test)
- Multi-PERFORM: IF/ELSE dual PERFORM on single line (1 test)
- IS EXTERNAL: USAGE not polluted by external flag (1 test)

Total: 215 tests passing

* fix(cobol): resolve 16th review — CANCEL in CALL block + USING boundary

#1: flushCallAccum now extracts CANCEL statements from within CALL
    ON EXCEPTION blocks. Adds RE_CANCEL + RE_CANCEL_DYNAMIC matchAll
    passes alongside existing CALL extraction.

#2: Added \bCANCEL(?=\s|$) to USING lookahead regex to prevent CANCEL
    keyword being captured as false USING parameter.

#3: Multi-line CALL start now returns immediately to prevent the CALL
    start line from simultaneously feeding sortAccum/inspectAccum.

#6: Division transitions now flush all active accumulators (callAccum,
    sortAccum, inspectAccum) to prevent state leakage across programs.

Also added CANCEL to callAccum flush trigger verb list.

Tests: 215 passing | TypeScript clean

* refactor(cobol): extract shared verb constants + resolve 17th review

Extract COBOL_STATEMENT_VERBS, RE_STATEMENT_VERB_START, and
RE_USING_PARAMS as shared constants — eliminates 4 duplicated
25-verb regex patterns.

17th review: #1 flushCallAccum before EXEC entry, #2 inspectAccum
verb parity via shared constant.

Tests: 215 passing | TypeScript clean

* test(cobol): replace all fuzzy assertions with exact toBe checks

Replaced 7 toBeGreaterThan/toBeLessThan/toBeGreaterThanOrEqual
assertions with exact toBe values:

- dataItems.length: >= 3 → toBe(3)
- calls.length: >= 1 → toBe(1)
- calls[0].line: range check → toBe(10)
- programs[].startLine/endLine: comparison → exact values
- innerA.endLine/innerB.startLine: comparison → exact values

Also added 11 new edge-case tests (accumulator flush on EXEC/division
transitions, free-format, CANCEL in CALL block, SORT THRU, verb
flush, integration).

226 tests passing — zero fuzzy assertions remain.

* fix(cobol): resolve 19th review + 15 accumulator flush tests

Fixes:
#1: END PROGRAM flushes callAccum/sortAccum/inspectAccum
#2: PROGRAM-ID sibling path flushes all accumulators
#3: Added COMPUTE/ADD/SUBTRACT/MULTIPLY/DIVIDE/STRING/UNSTRING
    to COBOL_STATEMENT_VERBS (now 32 verbs)

Tests (15 new):
- END PROGRAM flush: single + nested programs (2)
- PROGRAM-ID sibling flush (1)
- Arithmetic verb flush: COMPUTE/ADD/SUBTRACT/MULTIPLY/DIVIDE (5)
- String verb flush: STRING/UNSTRING (2)
- Arithmetic not captured as false USING params (1)
- SORT flushed at END PROGRAM (1)
- INSPECT flushed at END PROGRAM (1)
- All with exact toBe assertions (2)

Total: 239 tests passing | Zero fuzzy assertions

* fix(cobol): resolve 20th review — INITIALIZE multi-target + 2 tests

Finding 1: INITIALIZE now captures multiple targets with REPLACING
clause keyword filtering. Regex changed to lazy match stopping at
REPLACING/WITH/period boundary. Targets split on whitespace and
filtered against INITIALIZE_CLAUSE_KEYWORDS set.

Tests (2 new):
- INITIALIZE multi-target: WS-CUSTOMER WS-ORDER WS-LINE-ITEM → 3
- INITIALIZE with REPLACING: only WS-RECORD captured, not keywords

Total: 241 tests passing | TypeScript clean
This commit is contained in:
Gergő Magyar 2026-03-26 14:03:48 +00:00 • committed by GitHub
parent 3c896cdbcd
commit d2cd0b676f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
35 changed files with 9937 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

100
docs/code-indexing/cobol/README.md Normal file
View file

@ -0,0 +1,100 @@
# COBOL Code Indexing
GitNexus indexes COBOL codebases using a **regex-only extraction** strategy, bypassing tree-sitter entirely. This document explains why, how the pipeline works, and links to detailed sub-documents.
## Why Regex-Only?
The tree-sitter-cobol grammar (v0.0.1) has three critical limitations that make it unusable for production indexing:
| Issue | Impact | Severity |
|-------|--------|----------|
| External scanner hangs on ~5% of files | No timeout mechanism exists for the C scanner; the process blocks indefinitely | **Blocking** |
| Only ~15% of paragraph headers detected | Most procedure-division paragraphs are invisible to the grammar | High |
| Patch markers in cols 1-6 cause parse errors | Enterprise COBOL uses non-standard sequence area content (e.g., `mzADD`, `estero`, `#FIX`) | High |
Because the external scanner hang cannot be interrupted (there is no `setTimeoutMicros` equivalent for tree-sitter), using tree-sitter-cobol would hang the indexing pipeline on a non-trivial fraction of real-world files.
The regex-only approach provides:
- **Speed**: ~1ms per file average extraction time
- **Reliability**: zero hangs, zero crashes across 13,000+ files
- **Coverage**: captures all critical symbols -- program name, paragraphs, sections, CALL, PERFORM, COPY, data items (01-77, 88-level), file declarations, FD entries, EXEC SQL/CICS blocks, ENTRY points, and MOVE statements
## Architecture
```mermaid
flowchart TD
A[Repository Scan] --> B{File Detection}
B -->|Extension match| C[COBOL file]
B -->|GITNEXUS_COBOL_DIRS match| C
B -->|No match| Z[Skip]
C --> D{Copybook?}
D -->|Yes| E[Add to Copybook Map]
D -->|No| F[Source Program]
E --> G[COPY Expansion Engine]
F --> G
G -->|Inline copybook content| H[Expanded Source]
H --> I[Patch Marker Cleanup]
I --> J[Regex State Machine]
J --> K[Extracted Symbols]
K --> L[Graph Model Builder]
L --> M[Knowledge Graph]
subgraph "Per-Chunk Processing"
G
H
I
J
K
L
end
subgraph "Post-Processing"
M --> N[Community Detection]
M --> O[Process Detection]
M --> P[Contract Detection]
end
style J fill:#e8f5e9,stroke:#2e7d32
style G fill:#e3f2fd,stroke:#1565c0
```
## COBOL vs Tree-Sitter Languages
| Feature | COBOL (Regex) | Tree-Sitter Languages |
|---------|--------------|----------------------|
| Parser | Single-pass regex state machine | tree-sitter grammar + queries |
| Speed | ~1ms/file | ~5ms/file |
| AST available | No | Yes |
| COPY expansion | Yes (pre-processing step) | N/A |
| Deep indexing | Data items, SQL, CICS, FD, ENTRY | Type annotations, generics, etc. |
| Call extraction | PERFORM (intra-file) + CALL (cross-program) | AST-based call site detection |
| Import extraction | COPY statements | `import`/`require`/`use`/`#include` |
| Coverage | All critical symbols | Language-dependent query coverage |
| Failure mode | Never hangs | External scanner can hang (COBOL only) |
## Sub-Documents
| Document | Description |
|----------|-------------|
| [File Detection](./file-detection.md) | Extension mapping, `GITNEXUS_COBOL_DIRS`, copybook classification |
| [COPY Expansion](./copy-expansion.md) | Copybook inlining, REPLACING transformations, cycle detection |
| [Regex Extraction](./regex-extraction.md) | State machine, regex patterns, line processing |
| [Deep Indexing](./deep-indexing.md) | Data items, EXEC SQL/CICS, file declarations, FD, ENTRY, MOVE |
| [Graph Model](./graph-model.md) | COBOL-specific node types, edge types, full annotated example |
| [Performance](./performance.md) | Benchmarks, worker pool tuning, caps, troubleshooting |
## Key Source Files
| File | Purpose |
|------|---------|
| `gitnexus/src/core/ingestion/cobol-preprocessor.ts` | Patch marker cleanup + regex extraction engine |
| `gitnexus/src/core/ingestion/cobol-copy-expander.ts` | COPY statement expansion with REPLACING |
| `gitnexus/src/core/ingestion/utils.ts` | `getLanguageFromPath`, `getLanguageFromFilename` |
| `gitnexus/src/core/ingestion/pipeline.ts` | `isCobolCopybook`, `expandCobolCopies`, `detectCrossProgamContracts` |
| `gitnexus/src/core/ingestion/workers/parse-worker.ts` | `processCobolRegexOnly` -- graph model builder |
| `gitnexus/src/core/ingestion/workers/worker-pool.ts` | Configurable sub-batch size for COBOL |

157
docs/code-indexing/cobol/copy-expansion.md Normal file
View file

@ -0,0 +1,157 @@
# COBOL COPY Expansion
The COPY statement is COBOL's include mechanism -- analogous to `#include` in C or `import` in modern languages. GitNexus expands COPY statements **before** regex extraction so that symbols defined inside copybooks (data items, paragraphs, etc.) are visible in the program's extracted graph.
## Supported Syntax
### Basic COPY
```cobol
COPY CPSESP.
COPY "WORKGRID.CPY".
```
Inlines the content of the named copybook, replacing the COPY line(s).
### COPY with REPLACING
```cobol
COPY CPSESP REPLACING "ANAZI-KEY" BY "LK-KEY".
COPY CPSESP REPLACING LEADING "ESP-" BY "LK-ESP-"
LEADING "KPSESPL" BY "LK-KPSESPL".
COPY LINKAGE REPLACING TRAILING "-IN" BY "-OUT".
```
Three REPLACING types are supported:
| Type | Syntax | Behavior | Example |
| ------------ | ------------------------------------ | --------------------------------------- | -------------------------------- |
| **EXACT** | `REPLACING "OLD" BY "NEW"` | Replace exact identifier matches | `ANAZI-KEY` becomes `LK-KEY` |
| **LEADING** | `REPLACING LEADING "PFX-" BY "NEW-"` | Replace prefix on all COBOL identifiers | `ESP-NAME` becomes `LK-ESP-NAME` |
| **TRAILING** | `REPLACING TRAILING "-IN" BY "-OUT"` | Replace suffix on all COBOL identifiers | `DATA-IN` becomes `DATA-OUT` |
Multiple REPLACING clauses can appear in a single COPY statement. They are applied in order to each COBOL identifier in the copybook content.
### Multi-Line COPY
COPY statements can span multiple lines (standard COBOL continuation rules apply):
```cobol
COPY CPSESP REPLACING
- LEADING "ESP-" BY "LK-ESP-"
- LEADING "KPSESPL" BY "LK-KPSESPL".
```
Continuation lines (indicator `-` in column 7) are merged before COPY statement scanning.
## Expansion Flow
```mermaid
sequenceDiagram
participant Pipeline
participant Expander as COPY Expander
participant Resolver
participant Reader
Pipeline->>Pipeline: Identify all COBOL files
Pipeline->>Pipeline: Classify copybooks vs programs
Pipeline->>Reader: Read all copybook content upfront
Reader-->>Pipeline: Copybook content map (name -> content)
loop For each source file in chunk
Pipeline->>Expander: expandCopies(content, filePath, resolveFile, readFile)
Expander->>Expander: Merge continuation lines
Expander->>Expander: Detect COPY statements via regex
loop For each COPY statement (reverse order)
Expander->>Resolver: resolveFile(copyTarget)
Resolver-->>Expander: Copybook key or null
alt Resolved successfully
Expander->>Reader: readFile(resolvedKey)
Reader-->>Expander: Copybook content
Expander->>Expander: Apply REPLACING transformations
Expander->>Expander: Recurse for nested COPYs (depth + 1)
Expander->>Expander: Splice expanded content into output
else Not resolved
Expander->>Expander: Keep original COPY line
end
end
Expander-->>Pipeline: Expanded content + resolution metadata
Pipeline->>Pipeline: Replace file content with expanded content
end
```
The return type `CopyExpansionResult` contains `expandedContent` and `copyResolutions`. The `expansionDepth` field has been removed from the return type (it was unused by callers).
COPY statement line numbers in `CopyResolution` are 1-based (consistent with the preprocessor's line numbering). The splice operation that replaces COPY lines with expanded content adjusts for 0-based array indexing internally.
## Cycle Detection
Circular COPY references (e.g., copybook A includes copybook B which includes copybook A) are detected and handled:
1. Each expansion chain maintains a `visited` set of resolved copybook paths
2. If a copybook path is already in the visited set, the expansion is skipped
3. A `warnedCircular` set (internal to `expandCopies()`, not a parameter) deduplicates warning messages within a single file expansion
Known circular copybooks in PROJECT-NAME: `ANAZI`, `ANDIP`, `QDIPE` (self-referential includes).
## Max Depth
Nested COPY expansion is limited to **10 levels** (`DEFAULT_MAX_DEPTH`). If a COPY chain exceeds this depth, a warning is logged and the remaining COPY statements are left unexpanded.
## Max Total Expansions
A breadth amplification guard caps the total number of COPY expansions across all branches within a single file to **500** (`MAX_TOTAL_EXPANSIONS`). This prevents exponential blowup from diamond-shaped COPY graphs where N copybooks each include N other copybooks. Once the limit is reached, further COPY statements in that file are left unexpanded and a single warning is logged.
## REPLACING Application Detail
The REPLACING engine works by scanning all COBOL identifiers (matching `\b[A-Z][A-Z0-9-]*\b`) in the copybook content and applying each replacement rule:
```
Original copybook content:
05 ESP-NAME PIC X(30).
05 ESP-CODE PIC X(10).
05 KPSESPL-FLAG PIC X(01).
After REPLACING LEADING "ESP-" BY "LK-ESP-" LEADING "KPSESPL" BY "LK-KPSESPL":
05 LK-ESP-NAME PIC X(30).
05 LK-ESP-CODE PIC X(10).
05 LK-KPSESPL-FLAG PIC X(01).
```
For LEADING replacements, the engine checks if each identifier starts with the `from` prefix (case-insensitive) and replaces only the prefix portion, preserving the rest of the identifier.
For TRAILING replacements, the same logic applies to suffixes.
For EXACT replacements, only identifiers that match the `from` value exactly (case-insensitive) are replaced.
## Copybook Resolution
The resolver tries multiple strategies to match a COPY target name to a copybook file:
1. **Exact match**: `COPY CPSESP` resolves to copybook named `CPSESP`
2. **Strip extension**: `COPY WORKGRID.CPY` strips `.CPY` and resolves to `WORKGRID`
3. **Add extension**: `COPY CPSESP` tries `CPSESP.CPY` and `CPSESP.COPY`
If no match is found, the COPY statement is left in place (unexpanded) and a resolution record with `resolvedPath: null` is created.
## Pipeline Integration
The expansion runs **per chunk**, after file content is read but before dispatch to worker threads:
1. All copybook files are read upfront (they are typically small, collectively under 100MB)
2. Per chunk, the copybook map is merged with chunk content (in case a chunk contains copybooks)
3. Only programs (not copybooks themselves) undergo expansion
4. The expanded content replaces the original content in-place before worker dispatch
## Inline Comment Handling
The copy expander's `stripInlineComment()` helper is quote-aware: pipe characters (`|`) inside single- or double-quoted strings are preserved. This matches the same quote-aware logic used by the preprocessor.
## Source Files
- `gitnexus/src/core/ingestion/cobol-copy-expander.ts` -- `expandCopies()`, `parseReplacingClause()`, `applyReplacing()`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `expandCobolCopies()`, copybook map construction, chunk integration

312
docs/code-indexing/cobol/deep-indexing.md Normal file
View file

@ -0,0 +1,312 @@
# COBOL Deep Indexing
Beyond basic symbol extraction (program name, paragraphs, CALL, PERFORM, COPY), GitNexus performs deep indexing of COBOL-specific constructs: data items, EXEC SQL/CICS blocks, file declarations, FD entries, ENTRY points, and MOVE statements.
## Data Items
### Level Numbers
| Level Range | Meaning | Graph Node Type |
|-------------|---------|-----------------|
| 01 | Record (group item) | `Record` |
| 02-49 | Elementary/group items | `Property` |
| 66 | RENAMES | `Property` |
| 77 | Independent item | `Property` |
| 88 | Condition name | `Const` |
FILLER items are skipped (no useful name for the graph).
### Clauses Parsed
The `parseDataItemClauses()` function extracts these clauses from the trailing text of a data item declaration:
| Clause | Pattern | Example |
|--------|---------|---------|
| `PIC` / `PICTURE` | `\bPIC(?:TURE)?\s+(?:IS\s+)?(\S+)` | `PIC X(30)`, `PICTURE IS 9(5)V99` |
| `USAGE` | `\bUSAGE\s+(?:IS\s+)?(COMP\|BINARY\|...)` | `USAGE IS COMP-3`, `BINARY` |
| `REDEFINES` | `\bREDEFINES\s+([A-Z][A-Z0-9-]+)` | `REDEFINES WK-DATE-NUM` |
| `OCCURS` | `\bOCCURS\s+(\d+)` | `OCCURS 12 TIMES` |
Standalone COMP variants (without the `USAGE` keyword) are also detected: `COMP`, `COMP-1` through `COMP-6`, `COMP-X`, `BINARY`, `PACKED-DECIMAL`.
### Data Hierarchy
Data items form a hierarchical structure based on level numbers. The extractor uses a **stack algorithm**:
```
Processing order:
01 WK-RECORD -> push {01, WK-RECORD} -> parent: Module
05 WK-NAME -> push {05, WK-NAME} -> parent: WK-RECORD (01 < 05)
10 WK-FIRST -> push {10, WK-FIRST} -> parent: WK-NAME (05 < 10)
10 WK-LAST -> pop WK-FIRST, push -> parent: WK-NAME (05 < 10)
05 WK-CODE -> pop WK-LAST, WK-NAME -> parent: WK-RECORD (01 < 05)
88 WK-ACTIVE -> (88 handled separately) -> parent: WK-CODE
```
The stack maintains items where each entry's level is strictly less than the next. When a new item arrives with a level <= the top of stack, items are popped until the stack top has a smaller level. A `CONTAINS` edge is created from the stack top to the new item.
For 88-level condition names, the parent is the immediately preceding non-88 data item (found by scanning backwards).
### Annotated Example
```cobol
01 WK-EMPLOYEE.
05 WK-EMP-ID PIC 9(6).
05 WK-EMP-NAME PIC X(30).
05 WK-EMP-STATUS PIC X(01).
88 WK-ACTIVE VALUE "A".
88 WK-INACTIVE VALUE "I".
05 WK-SALARY PIC 9(7)V99 COMP-3.
05 WK-DEPT PIC X(04) OCCURS 3 TIMES.
```
Produces:
- `Record` node: `WK-EMPLOYEE` (level 01, section: working-storage)
- `Property` nodes: `WK-EMP-ID`, `WK-EMP-NAME`, `WK-EMP-STATUS`, `WK-SALARY`, `WK-DEPT`
- `Const` nodes: `WK-ACTIVE` (values: `A`), `WK-INACTIVE` (values: `I`)
- `CONTAINS` edges: `WK-EMPLOYEE -> WK-EMP-ID`, `WK-EMPLOYEE -> WK-EMP-NAME`, etc.
- `CONTAINS` edges: `WK-EMP-STATUS -> WK-ACTIVE`, `WK-EMP-STATUS -> WK-INACTIVE`
### Data Item Cap
A maximum of **500 data items per file** (`MAX_DATA_ITEMS_PER_FILE`) are processed. Some COBOL programs (especially after COPY expansion) can have 10,000+ data items, which would cause graph bloat and push the V8 relationship Map past its 16.7M entry limit across thousands of files.
The cap applies after extraction: the first 500 items in source order are kept. Since 01-level records appear first, critical top-level structure is preserved.
## EXEC SQL
EXEC SQL blocks are accumulated across lines between `EXEC SQL` and `END-EXEC`, then parsed as a unit.
### Operation Classification
The first SQL keyword determines the operation:
| First Keyword | Operation |
|---------------|-----------|
| `SELECT` | SELECT |
| `INSERT` | INSERT |
| `UPDATE` | UPDATE |
| `DELETE` | DELETE |
| `DECLARE` | DECLARE |
| `OPEN` | OPEN |
| `CLOSE` | CLOSE |
| `FETCH` | FETCH |
| *(anything else)* | OTHER |
### Table Extraction
Tables are extracted from SQL clauses:
| Clause Pattern | Example |
|----------------|---------|
| `FROM <table>` | `SELECT * FROM EMPLOYEES` |
| `INSERT INTO <table>` | `INSERT INTO EMPLOYEES` |
| `UPDATE <table>` | `UPDATE EMPLOYEES SET ...` |
| `JOIN <table>` | `LEFT JOIN DEPARTMENTS ON ...` |
Note: The `INTO` pattern is restricted to `INSERT INTO` to avoid false positives from `FETCH ... INTO :host-var` and `SELECT ... INTO :host-var` statements, where `INTO` introduces host variables rather than table names.
### Cursor Detection
```cobol
EXEC SQL
DECLARE C-EMPLOYEES CURSOR FOR
SELECT EMP-ID, EMP-NAME FROM EMPLOYEES
WHERE DEPT = :WK-DEPT
END-EXEC
```
Extracts: cursor `C-EMPLOYEES`, table `EMPLOYEES`, host variable `WK-DEPT`.
### Host Variables
Host variables are COBOL variables referenced in SQL with a `:` prefix. The colon is stripped:
```sql
WHERE EMP-ID = :WK-EMP-ID AND DEPT = :WK-DEPT
```
Extracts: `WK-EMP-ID`, `WK-DEPT`.
### Graph Output
- `CodeElement` node per table, with description `sql-table op:{OP}`
- `CodeElement` node per cursor, with description `sql-cursor`
- `ACCESSES` edge from Module to each CodeElement
- Deduplication: if the same table appears in multiple SQL blocks, only one node is created
## EXEC CICS
EXEC CICS blocks are accumulated and parsed similarly to SQL blocks.
### Command Detection
Two-word commands are detected first (matched against the block start):
```
SEND MAP, RECEIVE MAP, SEND TEXT, SEND CONTROL, READ NEXT, READ PREV
```
If no two-word command matches, the first word is used (e.g., `LINK`, `XCTL`, `RETURN`, `READ`, `WRITE`).
### Extraction
| Element | Pattern | Example |
|---------|---------|---------|
| MAP name | `MAP('name')` or `MAP("name")` | `EXEC CICS SEND MAP('EMPMENU')` |
| PROGRAM name | `PROGRAM('name')` or `PROGRAM("name")` | `EXEC CICS LINK PROGRAM('BGTABUP')` |
| TRANSID | `TRANSID('name')` or `TRANSID("name")` | `EXEC CICS START TRANSID('EMP1')` |
### Graph Output
- MAP: `CodeElement` node with description `cics-map cmd:{CMD}` + `ACCESSES` edge from Module
- PROGRAM: `CALLS` edge (cross-program call via CICS LINK/XCTL)
- TRANSID: `CodeElement` node with description `cics-transid cmd:{CMD}` + `ACCESSES` edge from Module
### Annotated Example
```cobol
EXEC CICS
SEND MAP('EMPMENU')
MAPSET('EMPSET')
FROM(WK-MAP-DATA)
ERASE
END-EXEC
```
Produces:
- `CodeElement` node: `EMPMENU` (description: `cics-map cmd:SEND MAP`)
- `ACCESSES` edge: Module -> `EMPMENU`
## File Declarations
SELECT statements in the INPUT-OUTPUT SECTION are accumulated across multiple lines (until a period terminator) and parsed for:
| Clause | Pattern | Example |
|--------|---------|---------|
| SELECT | `SELECT <name>` | `SELECT MASTER-FILE` |
| ASSIGN | `ASSIGN TO <file>` | `ASSIGN TO "MASTER.DAT"` |
| ORGANIZATION | `ORGANIZATION IS <type>` | `ORGANIZATION IS INDEXED` |
| ACCESS | `ACCESS MODE IS <mode>` | `ACCESS MODE IS DYNAMIC` |
| RECORD KEY | `RECORD KEY IS <field>` | `RECORD KEY IS WK-EMP-ID` |
| FILE STATUS | `FILE STATUS IS <field>` | `FILE STATUS IS WK-FILE-STATUS` |
### Graph Output
- `CodeElement` node with description containing all parsed clauses (e.g., `select org:INDEXED access:DYNAMIC key:WK-EMP-ID status:WK-FILE-STATUS assign:MASTER.DAT`)
- `RECORD_KEY_OF` edge: from Property node to CodeElement (confidence 0.8)
- `FILE_STATUS_OF` edge: from Property node to CodeElement (confidence 0.8)
## FD Entries
FD (File Description) entries associate a file name with its record layout:
```cobol
FD MASTER-FILE.
01 MASTER-RECORD.
05 MR-EMP-ID PIC 9(6).
05 MR-EMP-NAME PIC X(30).
```
The extractor tracks `pendingFdName` state: when an `FD` line is seen, the next 01-level data item becomes its record.
### Graph Output
- `CodeElement` node with description `fd record:{recordName}`
- `CONTAINS` edge: FD CodeElement -> Record node
- `CONTAINS` edge: SELECT CodeElement -> FD CodeElement (linking file declaration to file description)
## ENTRY Points
The `ENTRY` statement defines additional entry points into a COBOL program (in addition to the main program entry):
```cobol
ENTRY "SUBPROG" USING WK-PARAM-1 WK-PARAM-2.
```
### Graph Output
- `Constructor` node with description `entry params:{param1},{param2}` (or just `entry` if no parameters)
- `CONTAINS` edge: Module -> Constructor
- Symbol table entry (so the entry point is discoverable by name)
## PROCEDURE DIVISION USING
```cobol
PROCEDURE DIVISION USING WK-INPUT-REC WK-OUTPUT-REC.
```
The USING clause identifies parameters received by the program from its caller.
### Graph Output
- `RECEIVES` edge: Module -> Property (for each parameter name, confidence 0.8)
## MOVE Statements
MOVE statements produce `ACCESSES` edges in the graph:
```cobol
MOVE WK-NAME TO OUT-NAME.
MOVE CORRESPONDING WK-INPUT TO WK-OUTPUT.
MOVE CORR WK-IN TO WK-OUT.
```
### Extraction Details
- Source and target identifiers are captured
- `CORRESPONDING` and its abbreviation `CORR` are both recognized (bulk field-by-field move)
- Figurative constants (SPACES, ZEROS, LOW-VALUES, HIGH-VALUES, QUOTES, ALL) are skipped
- The enclosing paragraph (`caller`) is tracked for context
### MOVE CORRESPONDING / CORR Edge Reasons
MOVE CORRESPONDING (and CORR) produces distinct edge reasons to differentiate from simple MOVE:
| Edge | Reason (simple MOVE) | Reason (CORRESPONDING/CORR) |
|------|---------------------|-----------------------------|
| Read (source) | `cobol-move-read` | `cobol-move-corresponding-read` |
| Write (target) | `cobol-move-write` | `cobol-move-corresponding-write` |
This distinction allows queries to find bulk field-by-field moves separately from simple variable assignments.
## GO TO DEPENDING ON
The `GO TO` statement with multiple targets and a `DEPENDING ON` clause is a computed branch:
```cobol
GO TO PARA-1 PARA-2 PARA-3
DEPENDING ON WK-SELECTOR.
```
All target paragraph names are extracted and emitted as separate `gotos` entries. Each target produces a `CALLS` edge in the graph (same semantics as PERFORM). The `DEPENDING ON` variable is not currently tracked as a data-flow dependency.
## SORT INPUT/OUTPUT PROCEDURE
SORT and MERGE statements can specify procedural entry points instead of file-based I/O:
```cobol
SORT SORT-FILE ON ASCENDING KEY SORT-KEY
INPUT PROCEDURE IS PREPARE-INPUT
OUTPUT PROCEDURE IS FORMAT-OUTPUT.
```
`INPUT PROCEDURE IS` and `OUTPUT PROCEDURE IS` targets are extracted as control-flow targets (same as PERFORM). They produce `performs` entries and corresponding `CALLS` edges in the graph.
## Fixed-Format Literal Continuation
In fixed-format COBOL, string literals can span multiple lines using the continuation indicator (`-` in column 7). When a continuation line starts with a quote character, the extractor joins it with the predecessor by removing the trailing quote from the previous line and the opening quote from the continuation:
```
Line N: MOVE "THIS IS A LONG STRI
Line N+1 (cont): - "NG VALUE" TO WK-FIELD.
Merged: MOVE "THIS IS A LONG STRING VALUE" TO WK-FIELD.
```
The trailing `"` on line N and the opening `"` on line N+1 are both removed, producing a seamless literal. If no matching quote is found on the predecessor line, the continuation is appended as-is.
## Source Files
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- All extraction logic, clause parsers, EXEC block parsers
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, graph node/edge emission
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback with same `MAX_DATA_ITEMS_PER_FILE` cap

126
docs/code-indexing/cobol/file-detection.md Normal file
View file

@ -0,0 +1,126 @@
# COBOL File Detection
GitNexus detects COBOL files through two mechanisms: extension-based mapping and directory-based override for extensionless files. This document covers both, plus the copybook/program classification logic.
## Extension Mapping
### Program Extensions
| Extension | Type |
|-----------|------|
| `.cbl` | COBOL program |
| `.cob` | COBOL program |
| `.cobol` | COBOL program |
### Copybook Extensions
| Extension | Type | Notes |
|-----------|------|-------|
| `.cpy` | Copybook | Standard |
| `.copy` | Copybook | Standard |
| `.gnm` / `.GNM` | Copybook | Enterprise (GnuCOBOL naming) |
| `.fd` / `.FD` | Copybook | File Description fragment |
| `.wrk` / `.WRK` | Copybook | Working-Storage fragment |
| `.sel` / `.SEL` | Copybook | SELECT clause fragment |
| `.open` / `.OPEN` | Copybook | File OPEN fragment |
| `.close` / `.CLOSE` | Copybook | File CLOSE fragment |
| `.ini` / `.INI` | Copybook | Initialization fragment |
| `.def` / `.DEF` | Copybook | Definition fragment |
All extension matching is case-sensitive in `getLanguageFromFilename` (the extensions above are matched as written, including uppercase variants like `.GNM`).
## Extensionless File Detection: `GITNEXUS_COBOL_DIRS`
Many enterprise COBOL repositories use extensionless files -- the filename alone identifies the program (e.g., `s/BGTABFL` is the source for program `BGTABFL`). GitNexus handles this via the `GITNEXUS_COBOL_DIRS` environment variable.
### Configuration
Set `GITNEXUS_COBOL_DIRS` to a comma-separated list of directory names:
```bash
# Files in s/, c/, and wfproc/ directories (at any depth) are treated as COBOL
export GITNEXUS_COBOL_DIRS=s,c,wfproc
```
The matching is **case-insensitive** and checks all path segments:
- `/repo/s/BGTABFL` -- matches segment `s` -- COBOL
- `/repo/src/c/CPSESP` -- matches segment `c` -- COBOL
- `/repo/wfproc/WF001` -- matches segment `wfproc` -- COBOL
- `/repo/docs/README` -- no matching segment -- skipped
### Decision Tree
```mermaid
flowchart TD
A[getLanguageFromPath] --> B[getLanguageFromFilename]
B --> C{Known extension?}
C -->|Yes .cbl/.cob/.cobol/.cpy/...| D[Return COBOL]
C -->|Yes .ts/.py/.java/...| E[Return other language]
C -->|No match| F{Has extension?}
F -->|"Has dot in basename"| G[Return null]
F -->|"No dot = extensionless"| H{GITNEXUS_COBOL_DIRS set?}
H -->|No| G
H -->|Yes| I{Any path segment<br/>matches a configured dir?}
I -->|Yes| D
I -->|No| G
style D fill:#e8f5e9,stroke:#2e7d32
style G fill:#ffebee,stroke:#c62828
```
### Implementation Detail
The `GITNEXUS_COBOL_DIRS` value is parsed once (on first call) and cached in a `Set<string>`:
```typescript
// From gitnexus/src/core/ingestion/utils.ts
const getCobolDirs = (): Set<string> => {
if (_cobolDirs) return _cobolDirs;
const raw = process.env.GITNEXUS_COBOL_DIRS;
_cobolDirs = raw
? new Set(raw.split(',').map(d => d.trim().toLowerCase()))
: new Set();
return _cobolDirs;
};
```
The path segment check splits the full path on `/` and tests each segment against the cached set.
## Copybook vs Program Classification
After a file is identified as COBOL, it must be classified as either a **program** (to be parsed for symbols) or a **copybook** (to be loaded into the copybook map for COPY expansion).
### Classification Rules
A COBOL file is classified as a **copybook** if ANY of these conditions is true:
1. It has a recognized copybook extension (`.cpy`, `.copy`, `.gnm`, `.fd`, `.wrk`, `.sel`, `.open`, `.close`, `.ini`, `.def`)
2. It is an extensionless file whose path contains a directory segment matching one of: `c`, `copy`, `copybooks`, `copylib`, `cpy`
A file is classified as a **program** if:
1. It has a program extension (`.cbl`, `.cob`, `.cobol`), OR
2. It is extensionless and does NOT match any copybook directory pattern
### Copybook Name Resolution
Copybook names are derived from the filename:
- Strip the extension (if any)
- Convert to uppercase
Examples:
- `c/CPSESP` -- name: `CPSESP`
- `copy/workgrid.cpy` -- name: `WORKGRID`
- `c/ANAZI.GNM` -- name: `ANAZI`
This name is used to resolve `COPY CPSESP.` statements during expansion.
## Source Files
- `gitnexus/src/core/ingestion/utils.ts` -- `getLanguageFromPath()`, `getLanguageFromFilename()`, `getCobolDirs()`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `isCobolCopybook()`, `getCopybookName()`, `COPYBOOK_EXTENSIONS`, `COBOL_PROGRAM_EXTENSIONS`

193
docs/code-indexing/cobol/graph-model.md Normal file
View file

@ -0,0 +1,193 @@
# COBOL Graph Model
This document describes the graph nodes and edges that GitNexus creates for COBOL codebases. The COBOL graph model is richer than most tree-sitter languages because it captures domain-specific constructs: file declarations, FD entries, data hierarchies, SQL tables, CICS maps, and cross-program contracts.
## Entity-Relationship Diagram
```mermaid
erDiagram
File ||--o{ Module : DEFINES
File ||--o{ Function : DEFINES
File ||--o{ Namespace : DEFINES
File ||--o{ Record : DEFINES
File ||--o{ Property : DEFINES
File ||--o{ Const : DEFINES
File ||--o{ CodeElement : DEFINES
File ||--o{ Constructor : DEFINES
File }o--o{ File : IMPORTS
Module ||--o{ Record : CONTAINS
Module ||--o{ Constructor : CONTAINS
Module }o--o{ CodeElement : ACCESSES
Module }o--o{ Module : CALLS
Module }o--o{ Module : CONTRACTS
Module }o--o{ Property : RECEIVES
Record ||--o{ Property : CONTAINS
Record ||--o{ Const : CONTAINS
Record }o--o{ Record : REDEFINES
Property ||--o{ Property : CONTAINS
Property ||--o{ Const : CONTAINS
Property }o--o{ Property : REDEFINES
Property }o--o{ CodeElement : RECORD_KEY_OF
Property }o--o{ CodeElement : FILE_STATUS_OF
CodeElement ||--o{ CodeElement : CONTAINS
CodeElement ||--o{ Record : CONTAINS
Function }o--o{ Function : CALLS
```
## Node Types
| Node Type | COBOL Concept | Created From | Example |
|-----------|--------------|--------------|---------|
| `Module` | PROGRAM-ID | `PROGRAM-ID. BGTABFL` | Name: `BGTABFL`, description may include author and date |
| `Function` | Paragraph | `PROCESS-RECORD.` at column 8 | Name: `PROCESS-RECORD` |
| `Namespace` | Procedure section | `MAIN-LOGIC SECTION.` at column 8 | Name: `MAIN-LOGIC` |
| `Record` | 01-level data item | `01 WK-EMPLOYEE.` | Description: `level:01 section:working-storage` |
| `Property` | 02-49/66/77 data item | `05 WK-NAME PIC X(30).` | Description: `level:05 pic:X(30) section:working-storage` |
| `Const` | 88-level condition | `88 WK-ACTIVE VALUE "A".` | Description: `level:88 values:A` |
| `CodeElement` | SELECT, FD, SQL table, CICS map, cursor, transid | Various | Description varies by subtype |
| `Constructor` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` | Description: `entry params:WK-DATA` |
### CodeElement Subtypes
CodeElement is used for multiple COBOL constructs, distinguished by their description prefix:
| Subtype | ID Pattern | Description Format | Example |
|---------|-----------|-------------------|---------|
| File SELECT | `CodeElement:{path}:SELECT:{name}` | `select org:INDEXED access:DYNAMIC ...` | `SELECT MASTER-FILE` |
| FD entry | `CodeElement:{path}:FD:{name}` | `fd record:{recordName}` | `FD MASTER-FILE` |
| SQL table | `CodeElement:{path}:sql-table:{name}` | `sql-table op:SELECT` | Table `EMPLOYEES` |
| SQL cursor | `CodeElement:{path}:sql-cursor:{name}` | `sql-cursor` | Cursor `C-EMPLOYEES` |
| CICS map | `CodeElement:{path}:cics-map:{name}` | `cics-map cmd:SEND MAP` | Map `EMPMENU` |
| CICS transid | `CodeElement:{path}:cics-transid:{name}` | `cics-transid cmd:START` | Transid `EMP1` |
## Edge Types
| Edge Type | Source | Target | Created By | Confidence | Example |
|-----------|--------|--------|-----------|------------|---------|
| `DEFINES` | File | any node | File defines its symbols | 1.0 | File -> Module `BGTABFL` |
| `CALLS` | Function | Function | `PERFORM X [THRU Y]` | (via call-processor) | `PROCESS-RECORD` -> `CALC-TAX` |
| `CALLS` | Module | Module | `CALL "BGTABUP"` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
| `CALLS` | Module | Module | `EXEC CICS LINK PROGRAM('X')` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
| `IMPORTS` | File | File | `COPY copybook` | (via import-processor) | Source file -> Copybook file |
| `CONTAINS` | Module | Record | Data hierarchy root | 1.0 | `BGTABFL` -> `WK-EMPLOYEE` |
| `CONTAINS` | Record | Property | Data hierarchy | 1.0 | `WK-EMPLOYEE` -> `WK-NAME` |
| `CONTAINS` | Property | Property | Nested data items | 1.0 | `WK-ADDRESS` -> `WK-CITY` |
| `CONTAINS` | Record/Property | Const | 88-level parent | 1.0 | `WK-STATUS` -> `WK-ACTIVE` |
| `CONTAINS` | CodeElement (FD) | Record | FD record link | 1.0 | `FD:MASTER-FILE` -> `MASTER-RECORD` |
| `CONTAINS` | CodeElement (SELECT) | CodeElement (FD) | SELECT-FD link | 0.9 | `SELECT:MASTER-FILE` -> `FD:MASTER-FILE` |
| `CONTAINS` | Module | Constructor | ENTRY in module | 1.0 | `BGTABFL` -> `SUBPROG` |
| `REDEFINES` | Record | Record | `01 X REDEFINES Y` | 1.0 | `WK-DATE-NUM` -> `WK-DATE-ALPHA` |
| `REDEFINES` | Property | Property | `05 X REDEFINES Y` | 1.0 | `WK-CODE-NUM` -> `WK-CODE-ALPHA` |
| `RECORD_KEY_OF` | Property | CodeElement (SELECT) | `RECORD KEY IS field` | 0.8 | `WK-EMP-ID` -> `SELECT:MASTER-FILE` |
| `FILE_STATUS_OF` | Property | CodeElement (SELECT) | `FILE STATUS IS field` | 0.8 | `WK-FS` -> `SELECT:MASTER-FILE` |
| `ACCESSES` | Module | CodeElement | EXEC SQL/CICS | 0.9 | `BGTABFL` -> `sql-table:EMPLOYEES` |
| `RECEIVES` | Module | Property | `PROCEDURE USING` | 0.8 | `BGTABFL` -> `WK-INPUT-REC` |
| `CONTRACTS` | Module | Module | Shared copybook detection | 0.9 | `BGTABFL` -> `BGTABUP` (via `CPSESP`) |
## Full Annotated Example
Given this COBOL program:
```cobol
IDENTIFICATION DIVISION.
PROGRAM-ID. EMPMAINT.
AUTHOR. Development Team.
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT EMP-FILE
ASSIGN TO "EMPLOYEE.DAT"
ORGANIZATION IS INDEXED
ACCESS MODE IS DYNAMIC
RECORD KEY IS EMP-ID
FILE STATUS IS WS-FILE-STATUS.
DATA DIVISION.
FILE SECTION.
FD EMP-FILE.
01 EMP-RECORD.
05 EMP-ID PIC 9(6).
05 EMP-NAME PIC X(30).
WORKING-STORAGE SECTION.
01 WS-FLAGS.
05 WS-FILE-STATUS PIC X(02).
05 WS-EOF-FLAG PIC X(01).
88 WS-EOF VALUE "Y".
LINKAGE SECTION.
01 LK-SEARCH-KEY PIC 9(6).
PROCEDURE DIVISION USING LK-SEARCH-KEY.
MAIN-LOGIC SECTION.
MAIN-START.
PERFORM OPEN-FILE
PERFORM PROCESS-RECORDS
PERFORM CLOSE-FILE
STOP RUN.
OPEN-FILE.
OPEN I-O EMP-FILE.
PROCESS-RECORDS.
MOVE LK-SEARCH-KEY TO EMP-ID
EXEC SQL
SELECT EMP_SALARY INTO :WS-SALARY
FROM EMPLOYEES
WHERE EMP_ID = :EMP-ID
END-EXEC
CALL "EMPREPORT".
CLOSE-FILE.
CLOSE EMP-FILE.
```
The graph produced contains:
**Nodes:**
- `Module`: EMPMAINT (description: `author:Development Team`)
- `Namespace`: MAIN-LOGIC
- `Function`: MAIN-START, OPEN-FILE, PROCESS-RECORDS, CLOSE-FILE
- `Record`: EMP-RECORD, WS-FLAGS, LK-SEARCH-KEY
- `Property`: EMP-ID, EMP-NAME, WS-FILE-STATUS, WS-EOF-FLAG
- `Const`: WS-EOF (values: Y)
- `CodeElement`: SELECT:EMP-FILE, FD:EMP-FILE, sql-table:EMPLOYEES
- (COPY imports, if any, would produce File IMPORTS edges)
**Edges:**
- `DEFINES`: File -> all nodes
- `CONTAINS`: EMPMAINT -> EMP-RECORD, EMPMAINT -> WS-FLAGS, EMPMAINT -> LK-SEARCH-KEY
- `CONTAINS`: EMP-RECORD -> EMP-ID, EMP-RECORD -> EMP-NAME
- `CONTAINS`: WS-FLAGS -> WS-FILE-STATUS, WS-FLAGS -> WS-EOF-FLAG
- `CONTAINS`: WS-EOF-FLAG -> WS-EOF
- `CONTAINS`: FD:EMP-FILE -> EMP-RECORD
- `CONTAINS`: SELECT:EMP-FILE -> FD:EMP-FILE
- `CALLS`: MAIN-START -> OPEN-FILE, MAIN-START -> PROCESS-RECORDS, MAIN-START -> CLOSE-FILE
- `CALLS`: EMPMAINT -> EMPREPORT (external CALL)
- `ACCESSES`: EMPMAINT -> sql-table:EMPLOYEES
- `RECEIVES`: EMPMAINT -> LK-SEARCH-KEY (PROCEDURE USING)
- `RECORD_KEY_OF`: EMP-ID -> SELECT:EMP-FILE
- `FILE_STATUS_OF`: WS-FILE-STATUS -> SELECT:EMP-FILE
## How COBOL Differs from Tree-Sitter Languages
| Aspect | COBOL | Tree-Sitter Languages |
|--------|-------|----------------------|
| Node variety | 8 types (Module, Function, Namespace, Record, Property, Const, CodeElement, Constructor) | Typically 4-6 (Function, Class, Method, Interface, Module, Const) |
| Domain edges | RECORD_KEY_OF, FILE_STATUS_OF, ACCESSES, RECEIVES, CONTRACTS, REDEFINES | Primarily CALLS, IMPORTS, EXTENDS, IMPLEMENTS |
| Data hierarchy | Deep CONTAINS chains (01 -> 05 -> 10 -> 88) | Flat class members |
| Cross-program calls | CALL "name" + CICS LINK PROGRAM | Import-based resolution |
| Contract detection | Shared COPY copybook between caller/callee | Not applicable |
| Metadata | AUTHOR, DATE-WRITTEN on Module | JSDoc/docstring (not indexed) |
## Source Files
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, node/edge emission logic
- `gitnexus/src/core/ingestion/pipeline.ts` -- `detectCrossProgamContracts()` for CONTRACTS edges
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `CobolRegexResults` interface (all extracted data)

261
docs/code-indexing/cobol/performance.md Normal file
View file

@ -0,0 +1,261 @@
# COBOL Performance and Tuning
This document covers real-world benchmarks, worker pool configuration, memory management, known limitations, and troubleshooting for COBOL indexing.
## PROJECT-NAME Benchmark
The PROJECT-NAME project is a large Italian payroll system written in COBOL. It serves as the primary benchmark for COBOL indexing performance.
### Input
| Metric | Value |
| --------------------------- | ---------------------------------------------------------------------------- |
| Paths scanned | 14,217 |
| Parseable files | 13,129 |
| Total source size | 224 MB |
| Chunks | 12 (at 20 MB budget) |
| Copybooks loaded | 2,976 |
| Copybooks used in expansion | 2,955 |
| Key directories | `s/` (7773 programs), `c/` (3036 copybooks), `wfproc/` (1973 workflow files) |
### Output
| Metric | Value |
| ---------------------- | ------ |
| Graph nodes | 2.79M |
| Graph edges | 5.67M |
| Clusters (communities) | 16,679 |
| Execution flows | 300 |
### Timing
| Phase | Duration |
| ------------------------------- | ----------------- |
| Total | ~251s |
| KuzuDB write | 132s |
| Full-text search indexing | 6.7s |
| Regex extraction (avg per file) | ~1ms |
| COPY expansion + deep indexing | Remainder (~112s) |
### Indexing Command
```bash
cd /path/to/PROJECT-NAME
GITNEXUS_COBOL_DIRS=s,c,wfproc GITNEXUS_VERBOSE=1 node --max-old-space-size=8192 \
/path/to/gitnexus/dist/cli/index.js analyze --force
```
## Open-Source Benchmarks
### CardDemo (AWS)
| Metric | Value |
| ------ | ----- |
| Graph nodes | 12,323 |
| Graph edges | 8,893 |
| Total time | 7.4s |
### ACAS
| Metric | Value |
| ------ | ----- |
| Graph nodes | 14,016 |
| Graph edges | 15,452 |
| Total time | 9.3s |
### Micro-Benchmark (Single-File Extraction)
| Metric | Value |
| ------ | ----- |
| Per-iteration | 0.65ms |
| Throughput | ~382K lines/sec |
## Worker Pool Tuning
### Sub-Batch Size
The worker pool splits each worker's chunk into sub-batches to bound peak memory per `postMessage` serialization. COBOL repos use a smaller sub-batch size than the default:
| Parameter | Default | COBOL Mode |
| --------------------- | ----------- | ------------------- |
| Sub-batch size | 1,500 files | 200 files |
| Per sub-batch timeout | 120s | 120s (configurable) |
**Why 200?** COBOL regex extraction + preprocessing takes ~1ms per file on average, but with COPY expansion and deep indexing the effective time is ~150ms per file. At sub-batch size 1500, that would be ~225s per sub-batch, exceeding the 120s timeout.
COBOL mode is activated automatically when `GITNEXUS_COBOL_DIRS` is set:
```typescript
// From pipeline.ts
const cobolSubBatch = process.env.GITNEXUS_COBOL_DIRS ? 200 : undefined;
workerPool = createWorkerPool(workerUrl, undefined, cobolSubBatch);
```
### Worker Count
Workers default to `min(8, cpus - 1)`. For COBOL repos, this is usually sufficient since regex extraction is CPU-bound but fast. The bottleneck is typically KuzuDB write, not extraction.
### Timeout Configuration
| Environment Variable | Default | Purpose |
| ------------------------------------ | --------------- | --------------------------------------------------- |
| `GITNEXUS_WORKER_TIMEOUT_MS` | 120,000 (2 min) | Per sub-batch processing timeout |
| `GITNEXUS_WORKER_STARTUP_TIMEOUT_MS` | 60,000 (1 min) | Worker initialization timeout (tree-sitter loading) |
For COBOL-only repos, worker startup is faster because tree-sitter native modules are loaded lazily (skipped entirely if only COBOL files are present).
## Data Item Cap
### Configuration
```typescript
const MAX_DATA_ITEMS_PER_FILE = 500;
```
This constant appears in both `parse-worker.ts` (worker path) and `parsing-processor.ts` (sequential fallback).
### Rationale
Some COBOL programs, especially after COPY expansion, can have 10,000+ data items. At that scale:
- The in-memory relationship Map (for CONTAINS, REDEFINES, etc.) approaches the V8 16.7M entry limit across thousands of files
- KuzuDB write time increases linearly with edge count
- Most deep-nested items (level 20+) are rarely queried individually
### Impact
The cap truncates data items beyond the 500th in source order. Since 01-level Records appear first in COBOL source, the cap preserves:
- All 01-level record definitions
- The most important 02-49 level items (those closest to the record root)
- 88-level conditions associated with early items
To increase the cap for specific needs, modify the `MAX_DATA_ITEMS_PER_FILE` constant in both files.
## Memory Management
### COPY Expansion Breadth Guard
A per-file `MAX_TOTAL_EXPANSIONS = 500` limit prevents exponential blowup from diamond-shaped COPY graphs (e.g., N copybooks each containing N COPY statements). Once the limit is reached, further COPY statements in that file are left unexpanded. See [copy-expansion.md](copy-expansion.md) for details.
### COPY Expansion Memory
All copybook content is loaded upfront into a Map before chunk processing begins. For PROJECT-NAME:
- 2,976 copybooks, typically under 100MB total
- The Map is shared (read-only) across chunk iterations
- Per-chunk, the copybook map is merged with chunk file content (in case a chunk contains copybooks not in the pre-loaded set)
- After all chunks are processed, the copybook map is freed (`cobolCopybookContents = undefined`)
### Chunk Budget
Source files are grouped into chunks of max 20MB (`CHUNK_BYTE_BUDGET`). Each chunk's lifecycle:
1. Read file content into memory
2. Expand COPY statements (mutates content in-place)
3. Dispatch to workers for extraction
4. Workers return serialized results
5. Merge results into graph
6. Chunk content goes out of scope (GC reclaims)
This ensures only ~20MB of source + ~200-400MB of working memory (ASTs, extracted records, serialization) is active at any time.
### Shared Warning Deduplication
The `warnedCircular` set (used by the COPY expansion engine) is shared across all files in a chunk. This prevents the same circular copybook warning (e.g., `ANAZI includes itself`) from being logged thousands of times.
## Known Limitations
| Limitation | Impact | Workaround |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| tree-sitter-cobol hangs on ~5% of files | Cannot use tree-sitter for COBOL | Regex-only extraction (current approach) |
| Data item cap (500/file) | May miss deeply nested items in large programs | Increase `MAX_DATA_ITEMS_PER_FILE` in source |
| Circular copybooks (ANAZI, ANDIP, QDIPE) | Self-referential includes cannot be expanded | Detected and skipped with warning |
| wfproc/ files may not be pure COBOL | Workflow files may produce extraction noise | Exclude `wfproc` from `GITNEXUS_COBOL_DIRS` if problematic |
| No MOVE DATA_FLOW edges yet | Data flow between variables not in graph | Reserved for future release |
| Continuation line handling | Some complex multi-line continuations (especially in string literals spanning 3+ lines) may not merge correctly | Known edge case; affects <0.1% of lines |
| Single-line EXEC blocks | `EXEC SQL SELECT ... END-EXEC` on one line is handled, but pathological nesting is not | Extremely rare in practice |
| Extension case sensitivity | `.GNM` and `.gnm` are matched differently | Use the exact case from the codebase |
## Troubleshooting
### "COPY expansion failed"
```
[pipeline] COPY expansion failed for s/BGTABFL: Cannot read properties of null
```
**Cause:** A copybook referenced by a COPY statement cannot be found.
**Fix:**
1. Verify `GITNEXUS_COBOL_DIRS` includes the directory containing copybooks (typically `c`)
2. Check that copybook filenames match the COPY target (case-insensitive, after stripping extensions)
3. Ensure copybook files are not in `.gitignore`
### Worker sub-batch timeout
```
Worker 3 sub-batch timed out after 120s (chunk: 200 items)
```
**Cause:** A sub-batch took longer than the timeout. Typically happens when one file is extremely large (50,000+ lines after COPY expansion).
**Fix:** Increase the timeout:
```bash
GITNEXUS_WORKER_TIMEOUT_MS=300000 gitnexus analyze
```
### Memory errors (heap out of memory)
```
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
```
**Fix:** Increase Node.js heap size:
```bash
node --max-old-space-size=16384 /path/to/gitnexus/dist/cli/index.js analyze
```
For very large repos (>500MB source), consider `--max-old-space-size=32768`.
### Concurrent analyze corruption
**Rule:** Only ONE `gitnexus analyze` process should run at a time per repository. Concurrent writes to KuzuDB corrupt the database.
If corruption occurs:
```bash
# Remove the KuzuDB directory and re-index
rm -rf .gitnexus/kuzu
gitnexus analyze --force
```
### Slow KuzuDB write phase
The KuzuDB write phase (132s for PROJECT-NAME) is the bottleneck for large COBOL repos. This is proportional to the number of nodes and edges being written. Reducing `MAX_DATA_ITEMS_PER_FILE` or excluding non-essential directories from `GITNEXUS_COBOL_DIRS` can help.
### Verbose output
Enable verbose logging to see per-phase timing and statistics:
```bash
GITNEXUS_VERBOSE=1 gitnexus analyze
```
This outputs:
- Scan statistics (paths, parseable files, chunk count)
- Worker pool configuration (worker count, sub-batch size)
- COPY expansion statistics (copybooks loaded, files expanded)
- Community and process detection results
- Contract detection results
## Source Files
- `gitnexus/src/core/ingestion/workers/worker-pool.ts` -- `DEFAULT_SUB_BATCH_SIZE`, `SUB_BATCH_TIMEOUT_MS`, `WORKER_STARTUP_TIMEOUT_MS`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `CHUNK_BYTE_BUDGET`, COBOL sub-batch configuration, chunk lifecycle
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `MAX_DATA_ITEMS_PER_FILE`, `processCobolRegexOnly()`
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback `MAX_DATA_ITEMS_PER_FILE`

206
docs/code-indexing/cobol/regex-extraction.md Normal file
View file

@ -0,0 +1,206 @@
# COBOL Regex Extraction
The `extractCobolSymbolsWithRegex()` function in `cobol-preprocessor.ts` performs single-pass, state-machine-driven extraction of all COBOL symbols. This document describes the state machine, line processing flow, and every regex pattern used.
## State Machine: Division Tracking
The extractor tracks which COBOL division is currently being processed. Division transitions are detected by the `RE_DIVISION` pattern.
```mermaid
stateDiagram-v2
[*] --> null : Start of file
null --> identification : IDENTIFICATION DIVISION
identification --> environment : ENVIRONMENT DIVISION
environment --> data : DATA DIVISION
data --> procedure : PROCEDURE DIVISION
note right of identification
Extracts: PROGRAM-ID, AUTHOR, DATE-WRITTEN
end note
note right of environment
Extracts: SELECT ... ASSIGN ... (file declarations)
end note
note right of data
Extracts: FD entries, data items (01-77, 88), COPY
end note
note right of procedure
Extracts: paragraphs, sections, PERFORM, CALL,
ENTRY, MOVE, EXEC SQL/CICS
end note
```
## State Machine: Data Section Tracking
Within the DATA DIVISION, a secondary state machine tracks the current section to tag data items with their origin.
```mermaid
stateDiagram-v2
[*] --> unknown : DATA DIVISION entered
unknown --> working_storage : WORKING-STORAGE SECTION
unknown --> linkage : LINKAGE SECTION
unknown --> file : FILE SECTION
unknown --> local_storage : LOCAL-STORAGE SECTION
working_storage --> linkage : LINKAGE SECTION
working_storage --> file : FILE SECTION
linkage --> working_storage : WORKING-STORAGE SECTION
file --> working_storage : WORKING-STORAGE SECTION
file --> linkage : LINKAGE SECTION
local_storage --> working_storage : WORKING-STORAGE SECTION
```
Within the ENVIRONMENT DIVISION, the `currentEnvSection` tracks whether we are in `INPUT-OUTPUT` or `CONFIGURATION` section. SELECT statement accumulation only occurs in `INPUT-OUTPUT`.
## Line Processing Flow
Each raw source line goes through this pipeline:
```
Raw line
|
v
Length < 7? ---------> Skip (flush pending if any)
|
v
Indicator col 7
|
+-- '*' or '/' -----> Comment: skip entirely
|
+-- '-' ------------> Continuation: append to pending line
|
+-- other ----------> Normal: flush pending, strip inline comments (|),
buffer as new pending logical line
```
After all lines are processed, the final pending line is flushed, along with any accumulated SELECT statement, SORT/MERGE accumulator, and any open EXEC block (truncated file without `END-EXEC`).
### Inline Comment Stripping
Enterprise COBOL (particularly Italian dialect) uses the pipe character `|` as an inline comment marker. The `stripInlineComment()` helper is **quote-aware**: it tracks whether the scan position is inside a single- or double-quoted string and only treats `|` as a comment marker when outside quotes. Pipe characters inside string literals are preserved.
Free-format `*>` inline comment stripping uses the same quote-aware approach: the scanner walks character by character, toggling quote state, and only recognizes `*>` as a comment marker when not inside a quoted string.
### Patch Marker Handling
The `preprocessCobolSource()` function (run before extraction in the worker) replaces non-standard content in columns 1-6. Standard COBOL expects spaces or digit sequence numbers in this area. If any letter or `#` character is found, the entire sequence area is replaced with 6 spaces:
```
Before: mzADD MOVE WK-AMT TO WK-TOTAL
After: MOVE WK-AMT TO WK-TOTAL
```
This preserves exact line count for position mapping.
## Regex Pattern Reference
All patterns are compiled once as module-level constants and reused across calls.
### Division and Section Detection
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_DIVISION` | `\b(IDENTIFICATION\|ENVIRONMENT\|DATA\|PROCEDURE)\s+DIVISION\b` | Division boundary | `PROCEDURE DIVISION` |
| `RE_SECTION` | `\b(WORKING-STORAGE\|LINKAGE\|FILE\|LOCAL-STORAGE\|INPUT-OUTPUT\|CONFIGURATION)\s+SECTION\b` | Section boundary | `WORKING-STORAGE SECTION` |
### IDENTIFICATION DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_PROGRAM_ID` | `\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]*)` | Program name | `PROGRAM-ID. BGTABFL` |
| `RE_AUTHOR` | `^\s+AUTHOR\.\s*(.+)` | Author metadata | `AUTHOR. D. Smith` |
| `RE_DATE_WRITTEN` | `^\s+DATE-WRITTEN\.\s*(.+)` | Date metadata | `DATE-WRITTEN. 2024-01-15` |
### ENVIRONMENT DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_SELECT_START` | `\bSELECT\s+(?:OPTIONAL\s+)?([A-Z][A-Z0-9-]+)` | File SELECT start (with optional `SELECT OPTIONAL` support) | `SELECT MASTER-FILE`, `SELECT OPTIONAL TRANS-FILE` |
SELECT statements are accumulated across multiple lines until a period terminator is found, then parsed for ASSIGN, ORGANIZATION, ACCESS, RECORD KEY, and FILE STATUS clauses.
### DATA DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_FD` | `^\s+FD\s+([A-Z][A-Z0-9-]+)` | File description | `FD MASTER-FILE` |
| `RE_DATA_ITEM` | `^\s+(\d{1,2})\s+([A-Z][A-Z0-9-]+)\s*(.*)` | Data item (01-77) | `05 WK-NAME PIC X(30)` |
| `RE_ANONYMOUS_REDEFINES` | `^\s+(\d{1,2})\s+REDEFINES\s+([A-Z][A-Z0-9-]+)` | Anonymous REDEFINES | `01 REDEFINES WK-REC` |
| `RE_88_LEVEL` | `^\s+88\s+([A-Z][A-Z0-9-]+)\s+VALUES?\s+(?:ARE\s+)?(.+)` | Condition name | `88 WK-ACTIVE VALUE "Y"` |
The trailing clauses of `RE_DATA_ITEM` are parsed by `parseDataItemClauses()` for PIC, USAGE, OCCURS, and REDEFINES.
### PROCEDURE DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_PROC_SECTION` | `^ ([A-Z][A-Z0-9-]+)\s+SECTION\.\s*$` | Procedure section header | ` MAIN-LOGIC SECTION.` |
| `RE_PROC_PARAGRAPH` | `^ ([A-Z][A-Z0-9-]+)\.\s*$` | Paragraph header | ` PROCESS-RECORD.` |
| `RE_PERFORM` | `\bPERFORM\s+([A-Z][A-Z0-9-]+)(?:\s+THRU\s+([A-Z][A-Z0-9-]+))?` | PERFORM call | `PERFORM CALC-TAX THRU CALC-TAX-EXIT` |
| `RE_PROC_USING` | `\bPROCEDURE\s+DIVISION\s+USING\s+([\s\S]*?)(?:\.\|$)` | USING parameters | `PROCEDURE DIVISION USING WK-PARAM` |
| `RE_ENTRY` | `\bENTRY\s+"([^"]+)"(?:\s+USING\s+([\s\S]*?))?(?:\.\|$)` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` |
| `RE_MOVE` | `\bMOVE\s+((?:CORRESPONDING\|CORR)\s+)?([A-Z][A-Z0-9-]+)\s+TO\s+(.+)` | MOVE statement (supports CORR abbreviation and multi-target) | `MOVE WK-NAME TO OUT-NAME`, `MOVE CORR WK-IN TO WK-OUT` |
The USING parameter list (`RE_PROC_USING`) is split on `\bRETURNING\b` before tokenization -- any RETURNING clause and everything after it is excluded from the parameter list (`.split(/\bRETURNING\b/i)[0]`).
Note: `RE_PROC_SECTION` and `RE_PROC_PARAGRAPH` require exactly 7 spaces of leading indentation (COBOL area A starting at column 8). This is the standard COBOL paragraph indentation.
### All-Division Patterns
These patterns are checked regardless of current division:
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_CALL` | `\bCALL\s+"([^"]+)"` | External program call | `CALL "BGTABUP"` |
| `RE_COPY_UNQUOTED` | `\bCOPY\s+([A-Z][A-Z0-9-]+)(?:\s\|\.)` | COPY (unquoted) | `COPY CPSESP.` |
| `RE_COPY_QUOTED` | `\bCOPY\s+"([^"]+)"(?:\s\|\.)` | COPY (quoted) | `COPY "WORKGRID.CPY".` |
### SORT/MERGE Support
| Constant | Purpose |
|----------|---------|
| `SORT_CLAUSE_NOISE` | Set of SORT/MERGE clause keywords filtered from USING/GIVING file lists: `ON`, `ASCENDING`, `DESCENDING`, `KEY`, `WITH`, `DUPLICATES`, `IN`, `ORDER`, `COLLATING`, `SEQUENCE`, `IS`, `THROUGH`, `THRU`, `INPUT`, `OUTPUT`, `PROCEDURE` |
SORT and MERGE statements are accumulated across multiple lines (like SELECT) until a period terminator is found, then parsed for USING/GIVING file lists and INPUT/OUTPUT PROCEDURE targets. The `flushSort()` helper encapsulates the flush-and-parse logic, mirroring the existing `flushSelect()` pattern. Both helpers are called at EOF to handle truncated files.
### GO TO Multi-Target
`RE_GOTO` captures all paragraph names in a `GO TO` statement, including the multi-target form `GO TO p1 p2 p3 DEPENDING ON x`. The captured group contains all target names (space-separated), which are split into individual targets. Each target produces a separate `gotos` entry.
### PROGRAM-ID Detection
PROGRAM-ID is detected regardless of the current division state. This handles sibling programs that appear after `END PROGRAM` and omit the `IDENTIFICATION DIVISION` header -- the extractor will still capture the PROGRAM-ID and push a new program boundary.
### EXEC Block Patterns
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_EXEC_SQL_START` | `\bEXEC\s+SQL\b` | Start of EXEC SQL block | `EXEC SQL` |
| `RE_EXEC_CICS_START` | `\bEXEC\s+CICS\b` | Start of EXEC CICS block | `EXEC CICS` |
| `RE_END_EXEC` | `\bEND-EXEC\b` | End of EXEC block | `END-EXEC` |
EXEC blocks accumulate all lines between `EXEC SQL/CICS` and `END-EXEC`, then delegate to `parseExecSqlBlock()` or `parseExecCicsBlock()` for detailed extraction.
## Excluded Paragraph Names
The following names are excluded from paragraph detection to avoid false positives from division/section headers:
```
DECLARATIVES, END, PROCEDURE, IDENTIFICATION,
ENVIRONMENT, DATA, WORKING-STORAGE, LINKAGE,
FILE, LOCAL-STORAGE, COMMUNICATION, REPORT,
SCREEN, INPUT-OUTPUT, CONFIGURATION
```
Additionally, paragraph candidates containing `DIVISION` or `SECTION` as substrings are excluded.
## MOVE Skip List (Figurative Constants)
MOVE statements where the source is a figurative constant are skipped:
```
SPACES, ZEROS, ZEROES, LOW-VALUES, LOW-VALUE,
HIGH-VALUES, HIGH-VALUE, QUOTES, QUOTE, ALL
```
## Source Files
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `preprocessCobolSource()`, `extractCobolSymbolsWithRegex()`, all regex constants

326
docs/plans/2026-03-26-feat-cobol-full-language-coverage-plan.md Normal file
View file

@ -0,0 +1,326 @@
---
title: "feat: Complete COBOL language feature coverage for maximum knowledge graph value"
type: feat
status: active
date: 2026-03-26
origin: Feature audit from v3-integration-architect agent (session 8642401e)
---
## Enhancement Summary
**Deepened on:** 2026-03-26
**Research agents used:** COBOL expert (Phase 1+2), graph value analyst, codebase explorer
**Sections enhanced:** Phase 1 (5 features), Phase 2 (4 features), graph value ranking
### Key Improvements from Research
1. **CALL USING** is the #1 highest-value edge type (9.2/10) — fixes ~40% of missing caller references
2. **EXEC DLI** requires dual-interface support (EXEC DLI + CBLTDLI CALL) for full IMS coverage
3. **DECLARATIVES** is lowest-risk Phase 2 item — existing section/paragraph detection already captures structure
4. **SET TO TRUE** accounts for 80-90% of all SET statements — prioritize this form
5. **INSPECT** needs multi-line accumulator (like SORT) — can span 5+ continuation lines
6. **Graph value ranking**: cobol-call-using (9.2) > cobol-error-handler (9.0) > dli-gu (8.2) > cobol-string (6.2)
### New Edge Cases Discovered
- CALL USING supports mixed modes: `USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
- CALL USING `ADDRESS OF` and `OMITTED` must be filtered from parameter lists
- EXEC DLI can have multiple SEGMENT levels in hierarchical retrieval (use matchAll)
- DECLARATIVES can have multiple USE sections (one per file + catch-all for INPUT/OUTPUT/I-O/EXTEND)
- INSPECT TALLYING can have multiple counters in a single statement
- STRING/UNSTRING can span multiple lines (need accumulator pattern)
---
# Complete COBOL Language Feature Coverage
## Overview
Implement the remaining 25 unhandled COBOL language features and fix 10 partial features to achieve ~95% coverage (up from 71.9%). The goal is to build the richest possible knowledge graph from COBOL codebases, enabling a future `modernize` MCP command (out of scope for this plan) that would use the graph to assist with COBOL-to-modern-language migration.
## Problem Statement
The COBOL processor currently handles 54 of 89 applicable language features (71.9%). The 25 unhandled features represent real data loss in the knowledge graph:
- **Cross-program data flow** is invisible (CALL ... USING parameters not extracted)
- **IMS/DB programs** produce empty graphs (EXEC DLI not recognized)
- **String transformation logic** is invisible (STRING/UNSTRING/INSPECT not tracked)
- **SQL copybook dependencies** are missing (EXEC SQL INCLUDE not mapped)
- **Error handling flows** are lost (DECLARATIVES/USE AFTER not captured)
## Proposed Solution
Implement features in 4 phases, ordered by graph value density (edges created per LOC of implementation). Each phase is independently shippable and testable.
## Technical Approach
### Phase 1: High-Value Data Flow Edges (~150 LOC, ~8 new edge types)
The highest-ROI features: they create new ACCESSES and IMPORTS edges that directly improve impact analysis.
**Critical research finding**: Multi-line statement accumulation is the dominant challenge. CALL USING, STRING/UNSTRING, and multi-line data item clauses all span multiple lines in production COBOL. The free-format path processes each line independently — these features need statement accumulators (like SORT/SELECT) or the free-format path needs multi-line awareness. Estimated LOC increased from 110 to 150 to account for accumulator infrastructure.
#### 1.1 EXEC SQL INCLUDE -> IMPORTS edges
- **File:** `cobol-preprocessor.ts` (parseExecSqlBlock)
- **What:** Detect `INCLUDE` as the operation, extract member name, emit as a `copies[]` entry
- **Graph:** IMPORTS edge from File to included copybook/SQLCA with reason `sql-include`
- **Tests:** Unit test for `EXEC SQL INCLUDE SQLCA END-EXEC` and `EXEC SQL INCLUDE CUSTCOPY END-EXEC`
**Research insights (EXEC SQL INCLUDE):**
- DB2 member names can contain underscores: `EXEC SQL INCLUDE CUST_TBL_DCL END-EXEC` — regex must use `[A-Z][A-Z0-9_-]+`
- Quoted literal form: `EXEC SQL INCLUDE 'DBRMLIB.MEMBER' END-EXEC` (z/OS PDS qualified name)
- SQLCA/SQLDA are DB2 builtins — won't resolve to repo files. Emit unresolved IMPORTS edge (still valuable)
- No REPLACING support on EXEC SQL INCLUDE (unlike COPY)
- Add `INCLUDE` to `OP_MAP` in `parseExecSqlBlock`; extract member via `RE_SQL_INCLUDE = /^INCLUDE\s+(?:'([^']+)'|"([^"]+)"|([A-Z][A-Z0-9_-]+))/i`
#### 1.2 CALL ... USING parameter extraction -> ACCESSES edges (Graph value: 9.2/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine CALL section)
- **What:** After capturing CALL target, scan for USING clause. Extract parameter names (reuse USING_KEYWORDS filter). Store as `calls[].parameters: string[]`
- **Interface:** Add `parameters?: string[]` to calls array type in CobolRegexResults
- **File:** `cobol-processor.ts` (CALL edge block)
- **Graph:** For each USING parameter, create ACCESSES edge from caller to data item Property node with reason `cobol-call-using`
- **Tests:** `CALL 'AUDITLOG' USING CUST-ID WS-AMOUNT` -> 2 ACCESSES edges
**Research insights (CALL USING forms):**
- Mixed modes: `CALL 'PGM' USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
- Pointer passing: `CALL 'PGM' USING ADDRESS OF WS-A`
- Placeholder: `CALL 'PGM' USING OMITTED WS-B`
- Filter keywords: add `ADDRESS`, `OMITTED`, `LENGTH` to USING_KEYWORDS (already has BY/VALUE/REFERENCE/CONTENT)
- **Impact tool enhancement:** CALL-USING edges enable BFS traversal through parameter data flow — single most impactful edge type for COBOL impact analysis
#### 1.3 STRING/UNSTRING data flow -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (new section in extractProcedure)
- **What:** Accumulate multi-line STRING/UNSTRING until period or END-STRING/END-UNSTRING. Extract sources and INTO targets.
- **Interface:** Add `strings: Array<{ sources: string[]; target: string; type: 'string' | 'unstring'; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** read-ACCESSES on sources, write-ACCESSES on INTO target with reason `cobol-string-read` / `cobol-string-write`
- **Tests:** 2 unit tests + integration test assertions
**Research insights (STRING/UNSTRING):**
- **Needs statement accumulator** — STRING/UNSTRING always span multiple lines in production
- Terminate accumulation at: period, END-STRING/END-UNSTRING, or start of next COBOL verb
- STRING sources: identifiers before each `DELIMITED BY`. Filter: STRING, DELIMITED, BY, SIZE, ALL, INTO, WITH, POINTER, ON, OVERFLOW, NOT, END-STRING
- UNSTRING: source is first identifier after UNSTRING; INTO targets are identifiers after INTO. Filter: DELIMITER, IN, COUNT, TALLYING, OR
- WITH POINTER field is both read AND written (starting position updated)
- TALLYING IN / COUNT IN fields are write targets
- Literal sources (`'text'`) must be filtered — quote-aware tokenization needed
- **Edge case**: STRING terminated by next verb, not period — existing fixture has `STRING ... DISPLAY` without period between them
#### 1.4 OCCURS DEPENDING ON -> ACCESSES edge
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
- **What:** Extend OCCURS regex to capture DEPENDING ON field, KEY fields, and INDEXED BY names
- **Interface:** Add `dependingOn?: string`, `occursMax?: number`, `occursKeys?: Array<{direction: string; fields: string[]}>`, `indexedBy?: string[]` to data items
- **Graph:** ACCESSES edge from table item to controlling field with reason `cobol-depends-on`
- **Tests:** `05 WS-TABLE OCCURS 100 DEPENDING ON WS-COUNT` -> edge
**Research insights (OCCURS):**
- IBM allows `OCCURS 0 TO n DEPENDING ON` (zero minimum) and `OCCURS UNBOUNDED DEPENDING ON` (V6.4)
- Subscripted controlling fields: `DEPENDING ON WS-COUNT(WS-IDX)` — strip subscripts before storing
- **Pre-existing gap**: Multi-line data item clauses without continuation indicator are NOT captured. `05 WS-TABLE\n OCCURS 100\n DEPENDING ON WS-COUNT.` — the current RE_DATA_ITEM only gets the first line, `rest` is empty. Fixing properly requires a data item accumulator (like SELECT). **Defer full fix to Phase 3; implement same-line capture now.**
- KEY IS fields: `ASCENDING KEY IS WS-KEY-1 WS-KEY-2` — capture for SEARCH ALL resolution
- INDEXED BY: `INDEXED BY IDX-1 IDX-2` — capture for SET/SEARCH context
#### 1.5 VALUE clause for standard data items
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
- **What:** Extract VALUE using a pragmatic function that handles quoted strings, numerics, figurative constants, hex/national literals
- **Interface:** Already exists as `values?: string[]` on data items (currently only populated for 88-level)
- **Graph:** Stored in Property node description (no new edges)
- **Tests:** `01 WS-STATUS PIC X VALUE 'A'` -> values: ['A']
**Research insights (VALUE forms):**
- Hex literals: `VALUE X'F1F2F3F4'`, National: `VALUE N'text'`, DBCS: `VALUE G'text'`
- Figurative constants: SPACES, ZEROS, ZEROES, LOW-VALUES, HIGH-VALUES, QUOTES, NULL, NULLS
- ALL literal: `VALUE ALL '*'`
- Numeric with sign/decimal: `VALUE -123.45`, `VALUE +1`
- `VALUE IS` optional — both `VALUE 'A'` and `VALUE IS 'A'` valid
- **Decimal vs period ambiguity**: `VALUE 100.` — is `.` decimal or terminator? `parseDataItemClauses` already strips trailing period, so this is handled
- IBM V6.4: floating-point `VALUE 1.0E5` — extend numeric regex if needed
- Implementation: use a pragmatic `extractValue(rest)` function, not a single complex regex
### Phase 2: EXEC DLI + DECLARATIVES (~90 LOC, ~4 new edge types)
IMS/DB support and error handling flows.
#### 2.1 EXEC DLI (IMS/DB) -> ACCESSES edges (Graph value: 8.2/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine — add RE_EXEC_DLI_START check alongside SQL/CICS)
- **What:** Accumulate EXEC DLI blocks like EXEC SQL. Parse DLI verbs (GU, GN, GNP, GHU, GHN, GHNP, ISRT, DLET, REPL, CHKP, SCHD, TERM). Extract segment name, PCB number, INTO/FROM areas, WHERE fields, PSB name.
- **Interface:** Add `execDliBlocks: Array<{ line: number; verb: string; pcbNumber?: number; segmentName?: string; intoField?: string; fromField?: string; whereField?: string; psbName?: string }>` to CobolRegexResults
- **Graph:** CodeElement node + ACCESSES edge to `<ims>:<segmentName>` Record node with reason `dli-{verb}`; ACCESSES edges to INTO/FROM data areas; PSB ACCESSES for SCHD
- **Tests:** `EXEC DLI GU USING PCB(1) SEGMENT(CUSTOMER) INTO(WS-CUST) END-EXEC`
**Research insights (dual IMS interface):**
- **EXEC DLI**: Embedded command interface for CICS-DL/I programs only
- **CBLTDLI CALL**: Batch interface via `CALL 'CBLTDLI' USING function-code PCB io-area SSA1..SSA15`
- CBLTDLI is already captured as a CALL to 'CBLTDLI' — enrich with USING parameter semantics later
- Multiple SEGMENT levels in hierarchical retrieval — use `matchAll` on segment regex
- DLI verbs: GU (most common), GN, GNP, GHU, GHN, GHNP, ISRT, REPL, DLET, CHKP, SCHD, TERM, ROLL, ROLB
- **Edge case**: DLET/REPL have no SEGMENT clause (operate on current position)
- **Recommended order**: Implement AFTER DECLARATIVES and SET (lower risk, higher frequency)
#### 2.2 DECLARATIVES / USE AFTER STANDARD EXCEPTION (Graph value: 9.0/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine — detect DECLARATIVES keyword, track USE AFTER blocks)
- **What:** When `DECLARATIVES.` is encountered, switch to declaratives mode. Extract USE statements binding sections to files/modes.
- **Interface:** Add `declaratives: Array<{ sectionName: string; useType: 'error' | 'debug' | 'label' | 'reporting'; target: string; line: number }>` to CobolRegexResults
- **Graph:** ACCESSES edge from declarative Namespace to file Record with reason `cobol-declarative-error-handler`
- **Tests:** Unit test with DECLARATIVES section, integration test for error flow
**Research insights (DECLARATIVES syntax):**
- `USE AFTER STANDARD {EXCEPTION|ERROR} ON {file-name|INPUT|OUTPUT|I-O|EXTEND}`
- EXCEPTION and ERROR are synonymous; STANDARD is optional in IBM dialects
- Multiple USE sections allowed (one per file + catch-all for I/O modes)
- `END DECLARATIVES.` must NOT reset PROCEDURE DIVISION state
- `DECLARATIVES` is already in EXCLUDED_PARA_NAMES — no false paragraph risk
- Existing section/paragraph detection already captures structural elements — just need USE binding
- **Lowest risk Phase 2 item** — implement first
#### 2.3 SET statement -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (extractProcedure — new RE_SET regex)
- **Interface:** Add `sets: Array<{ targets: string[]; form: 'to-true'|'to-value'|'up-by'|'down-by'|'address-of'|'to-null'|'to-entry'; value?: string; entryTarget?: string; entryIsLiteral?: boolean; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** ACCESSES write edge with reason `cobol-set-condition` (TO TRUE), `cobol-set-index` (TO/UP/DOWN), `cobol-set-address` (ADDRESS OF). SET ENTRY with literal -> CALLS edge.
- **Tests:** `SET WS-EOF TO TRUE`, `SET IDX-1 TO 5`, `SET IDX-1 UP BY 1`
**Research insights (SET forms by frequency):**
- `SET condition TO TRUE` — 80-90% of all SET usage. Multiple targets: `SET COND-A COND-B TO TRUE`
- `SET index TO/UP BY/DOWN BY` — ~8%. Multiple indices: `SET IDX-1 IDX-2 UP BY 1`
- `SET pointer TO ADDRESS OF data-item` / `SET ADDRESS OF data-item TO pointer` — ~2%
- `SET proc-ptr TO ENTRY "PROGNAME"` — rare but creates CALLS edge (like dynamic CALL)
- Filter OF/IN qualifiers: `SET COND-A OF WS-RECORD TO TRUE` (strip OF WS-RECORD)
- **Prioritize**: SET TO TRUE alone covers 80-90% — implement this form first
#### 2.4 INSPECT -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (extractProcedure — new `inspectAccum` accumulator like SORT)
- **What:** Accumulate multi-line INSPECT until period. Extract inspected field + tally counters.
- **Interface:** Add `inspects: Array<{ inspectedField: string; counters: string[]; form: 'tallying'|'replacing'|'converting'|'tallying-replacing'; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** ACCESSES read on inspected field always; write if REPLACING/CONVERTING. Write edges for tally counters. Reason: `cobol-inspect-read`/`cobol-inspect-write`/`cobol-inspect-tally`
- **Tests:** `INSPECT WS-FIELD TALLYING WS-COUNT FOR ALL 'A'` -> read on WS-FIELD, write on WS-COUNT
**Research insights (INSPECT forms by frequency):**
- REPLACING (~60%): `INSPECT WS-STR REPLACING ALL 'A' BY 'B'`
- TALLYING (~25%): `INSPECT WS-STR TALLYING WS-CNT FOR ALL 'A'` — multiple counters possible
- CONVERTING (~10%): `INSPECT WS-STR CONVERTING 'abc' TO 'ABC'`
- Combined (~5%): TALLYING + REPLACING in single statement
- **Needs multi-line accumulator** — INSPECT frequently spans 3-5 lines in production
- Extract tally counters with `([A-Z][A-Z0-9-]+)\s+FOR\b` matchAll pattern
- Filter figurative constants (SPACES, ZEROS) using existing MOVE_SKIP set
### Phase 3: Completeness Fixes (~60 LOC)
Fix the 10 partial features and small gaps.
#### 3.1 CALL ... RETURNING extraction
- Extend RE_CALL processing to capture RETURNING target after the USING clause
- Store as `calls[].returning?: string`
- Graph: ACCESSES write edge with reason `cobol-call-returning`
#### 3.2 SELECT OPTIONAL flag preservation
- Store `isOptional: boolean` in FileDeclaration interface
- Include in Record node description
#### 3.3 ALTERNATE RECORD KEY extraction
- Add regex in parseSelectStatement: `/\bALTERNATE\s+RECORD\s+KEY\s+(?:IS\s+)?([A-Z][A-Z0-9-]+)/i`
- Store as `alternateKeys?: string[]`
#### 3.4 COMMON attribute on nested programs
- Extend RE_PROGRAM_ID: `/\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]+)(?:\s+IS\s+COMMON)?/i`
- Store `isCommon: boolean` on Module node
- Affects cross-program CALL resolution scope
#### 3.5 IS EXTERNAL / IS GLOBAL as first-class properties
- Change from usage string hack to proper boolean fields on data items
- Add `isExternal?: boolean`, `isGlobal?: boolean` to data item interface
#### 3.6 AUTHOR / DATE-WRITTEN mapped to Module node
- Already extracted as programMetadata — map to Module node properties
- `graph.addNode({ ..., properties: { ..., author, dateWritten } })`
#### 3.7 REPLACE statement
- Track REPLACE / REPLACE OFF state in preprocessor
- Apply text substitutions during preprocessing (before regex extraction)
- Complex: requires careful scoping rules
### Phase 4: Niche Features (~30 LOC)
Low-priority but nice for completeness.
#### 4.1 INITIALIZE statement -> write ACCESSES
- `/\bINITIALIZE\s+([A-Z][A-Z0-9-]+)/i`
- ACCESSES write edge with reason `cobol-initialize`
#### 4.2 Remaining IDENTIFICATION DIVISION paragraphs
- DATE-COMPILED, INSTALLATION, SECURITY, REMARKS
- Map to Module node description properties
#### 4.3 EXEC SQL INCLUDE -> IMPORTS edge (expansion)
- For EXEC SQL INCLUDE inside EXEC blocks that reference copybooks containing SQL
- Create IMPORTS edge similar to COPY
## Acceptance Criteria
### Functional Requirements
- [ ] Phase 1: All 5 features implemented with unit + integration tests
- [ ] Phase 2: All 4 features implemented with unit + integration tests
- [ ] Phase 3: All 7 partial features fixed
- [ ] Phase 4: At least 2 of 3 niche features implemented
- [ ] All existing 145 tests continue to pass
- [ ] TypeScript compiles cleanly
### Non-Functional Requirements
- [ ] No performance regression: CardDemo benchmark stays under 8s
- [ ] No file exceeds 1500 LOC (preprocessor currently 1326)
- [ ] ACAS benchmark shows increased node/edge counts (more data extracted)
- [ ] CardDemo benchmark shows increased edge counts (CALL USING, STRING, etc.)
### Quality Gates
- [ ] Each phase has its own commit
- [ ] Integration test assertions updated with exact counts per phase
- [ ] Benchmark run after each phase to track graph growth
## Dependencies & Risks
### Dependencies
- None. All changes are additive to existing COBOL processor code.
- No LanguageProvider changes needed.
- No graph schema changes needed (all new constructs map to existing node labels + edge types).
### Risks
- **preprocessor.ts size**: Currently 1326 LOC. Phase 1+2 adds ~200 LOC -> 1526 LOC. May need to extract helpers into a separate `cobol-data-flow.ts` module if it exceeds 1500.
- **REPLACE statement** (Phase 3.7) is the most complex feature — requires tracking text substitution state across logical lines. Consider deferring to a separate PR if it takes >100 LOC.
- **EXEC DLI** (Phase 2.1) is only testable against IMS codebases. Need fixture data or synthetic test cases.
## Graph Value Ranking by MCP Tool Impact
Research agent analyzed all 5 MCP tools (query, context, impact, detect_changes, rename) against planned edge types:
| Edge Type | QUERY | CONTEXT | IMPACT | DETECT | RENAME | **Overall** |
|-----------|-------|---------|--------|--------|--------|-------------|
| `cobol-call-using` | 4/5 | 5/5 | 5/5 | 4/5 | 4/5 | **9.2/10** |
| `cobol-error-handler` | 5/5 | 4/5 | 5/5 | 5/5 | 2/5 | **9.0/10** |
| `dli-*` (IMS verbs) | 4/5 | 4/5 | 5/5 | 4/5 | 2/5 | **8.2/10** |
| `cobol-string-*` | 4/5 | 3/5 | 3/5 | 3/5 | 2/5 | **6.2/10** |
**Key finding**: `cobol-call-using` alone would fix ~40% of missing caller references in COBOL graphs.
## Future Considerations
This plan provides the graph data foundation for a future `modernize` MCP command (out of scope) that would:
- Use CALL USING edges to map data contracts between programs
- Use STRING/UNSTRING edges to identify data transformation logic
- Use EXEC SQL/DLI edges to map database access patterns
- Use DECLARATIVES to understand error handling architecture
- Use the complete knowledge graph to generate migration plans
**MCP tool enhancements needed** (after this plan ships):
- Add `cobol-call-using`, `cobol-error-handler`, `dli-*` to IMPACT tool's default `relationTypes` for COBOL repos
- Add confidence floors for new edge types in `IMPACT_RELATION_CONFIDENCE`
- Register new edge types in `VALID_RELATION_TYPES` set (`local-backend.ts:52`)
## Sources & References
### Internal References
- Feature audit: session 8642401e (COBOL expert agent, 123 features audited)
- Prior plans: `docs/plans/2026-03-25-feat-cobol-100-percent-feature-coverage-plan.md`
- Architecture: `docs/code-indexing/cobol/` (7 documentation files)
### External References
- COBOL features reference: mainframestechhelp.com/tutorials/cobol/features.htm
- COBOL-85 standard: ISO/IEC 1989:1985
- IBM Enterprise COBOL reference

2
gitnexus/src/config/supported-languages.ts
View file

@ -42,4 +42,6 @@ export enum SupportedLanguages {
Kotlin = 'kotlin',
Swift = 'swift',
Dart = 'dart',
/** Standalone regex processor — no tree-sitter, no LanguageProvider. */
Cobol = 'cobol',
}

11
gitnexus/src/core/graph/graph.ts
View file

@ -34,7 +34,15 @@ export const createKnowledgeGraph = (): KnowledgeGraph => {
};
/**
* Remove all nodes (and their relationships) belonging to a file
* Remove a single relationship by id.
* Returns true if the relationship existed and was removed, false otherwise.
*/
const removeRelationship = (relationshipId: string): boolean => {
return relationshipMap.delete(relationshipId);
};
/**
* Remove all nodes (and their relationships) belonging to a file.
*/
const removeNodesByFile = (filePath: string): number => {
let removed = 0;
@ -75,6 +83,7 @@ export const createKnowledgeGraph = (): KnowledgeGraph => {
addRelationship,
removeNode,
removeNodesByFile,
removeRelationship,
};
};

1
gitnexus/src/core/graph/types.ts
View file

@ -141,4 +141,5 @@ export interface KnowledgeGraph {
addRelationship: (relationship: GraphRelationship) => void,
removeNode: (nodeId: string) => boolean,
removeNodesByFile: (filePath: string) => number,
removeRelationship: (relationshipId: string) => boolean,
}

1308
gitnexus/src/core/ingestion/cobol-processor.ts Normal file
View file

File diff suppressed because it is too large Load diff

501
gitnexus/src/core/ingestion/cobol/cobol-copy-expander.ts Normal file
View file

@ -0,0 +1,501 @@
/**
* COBOL COPY statement expansion engine.
*
* Expands COPY statements by inlining copybook content, applying REPLACING
* transformations (LEADING, TRAILING, EXACT), and handling nested copies
* with cycle detection.
*
* This is a preprocessing step that runs BEFORE extractCobolSymbolsWithRegex.
* The caller should run preprocessCobolSource first to clean patch markers.
*
* Supported syntax:
* COPY CPSESP.
* COPY "WORKGRID.CPY".
* COPY CPSESP REPLACING LEADING "ESP-" BY "LK-ESP-"
* LEADING "KPSESPL" BY "LK-KPSESPL".
* COPY ANAZI REPLACING "ANAZI-KEY" BY "LK-KEY".
*/
// ---------------------------------------------------------------------------
// Public interfaces
// ---------------------------------------------------------------------------
export interface CopyReplacing {
type: 'LEADING' | 'TRAILING' | 'EXACT';
from: string;
to: string;
isPseudotext?: boolean;
}
export interface CopyResolution {
copyTarget: string;
resolvedPath: string | null;
line: number;
replacing: CopyReplacing[];
library?: string;
}
export interface CopyExpansionResult {
expandedContent: string;
copyResolutions: CopyResolution[];
}
// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------
export const DEFAULT_MAX_DEPTH = 10;
/** COBOL identifier pattern: starts with letter, contains letters, digits, hyphens. */
const RE_COBOL_IDENTIFIER = /\b([A-Z][A-Z0-9-]*)\b/gi;
// ---------------------------------------------------------------------------
// Private helpers
// ---------------------------------------------------------------------------
/**
* Strip inline comments (Italian-style `|` comments).
* Only strips if `|` appears in the code area (col 7+).
*/
function stripInlineComment(line: string): string {
let inQuote: string | null = null;
for (let i = 0; i < line.length; i++) {
const ch = line[i];
if (inQuote) {
if (ch === inQuote) inQuote = null;
} else if (ch === '"' || ch === "'") {
inQuote = ch;
} else if (ch === '|') {
return line.substring(0, i);
}
}
return line;
}
/**
* Check if a line is a COBOL comment (indicator in col 7 is `*` or `/`).
*/
function isCommentLine(line: string): boolean {
return line.length >= 7 && (line[6] === '*' || line[6] === '/');
}
/**
* Check if a line is a continuation line (indicator in col 7 is `-`).
*/
function isContinuationLine(line: string): boolean {
return line.length >= 7 && line[6] === '-';
}
/**
* Merge continuation lines into their predecessors.
* Returns an array of logical lines with their original starting line numbers.
*/
function mergeLogicalLines(
rawLines: string[],
): Array<{ text: string; lineNum: number }> {
const logical: Array<{ text: string; lineNum: number }> = [];
for (let i = 0; i < rawLines.length; i++) {
const raw = rawLines[i];
// Skip comment lines
if (isCommentLine(raw)) {
logical.push({ text: '', lineNum: i + 1 });
continue;
}
// Continuation: merge into previous logical line
if (isContinuationLine(raw)) {
if (logical.length > 0) {
const prev = logical[logical.length - 1];
const continuation = raw.length > 7 ? raw.substring(7).trimStart() : '';
prev.text += continuation;
}
// Push empty placeholder to preserve line count
logical.push({ text: '', lineNum: i + 1 });
continue;
}
// Normal line: strip inline comments
const cleaned = stripInlineComment(raw);
logical.push({ text: cleaned, lineNum: i + 1 });
}
return logical;
}
// ---------------------------------------------------------------------------
// COPY statement parsing
// ---------------------------------------------------------------------------
interface ParsedCopyStatement {
startLine: number;
endLine: number;
target: string;
replacing: CopyReplacing[];
library?: string;
}
/**
* Parse REPLACING clause text into structured replacements.
*
* Input examples:
* LEADING "ESP-" BY "LK-ESP-" LEADING "KPSESPL" BY "LK-KPSESPL"
* "ANAZI-KEY" BY "LK-KEY"
* TRAILING "-IN" BY "-OUT"
* ==CUST-== BY ==WS-CUST-==
* ==OLD-TEXT== BY ====
*/
export function parseReplacingClause(text: string): CopyReplacing[] {
const replacings: CopyReplacing[] = [];
if (!text || text.trim().length === 0) return replacings;
// Tokenize: ==pseudotext==, "quoted strings", or bare words.
// Pseudotext can contain spaces and single = chars but not ==.
interface TokenInfo { value: string; isPseudotext: boolean; }
const tokens: TokenInfo[] = [];
const tokenRe = /==((?:[^=]|=[^=])*)==|"([^"]*)"|(\S+)/g;
let tm: RegExpExecArray | null;
while ((tm = tokenRe.exec(text)) !== null) {
if (tm[1] !== undefined) {
// Pseudotext: trim leading/trailing whitespace
tokens.push({ value: tm[1].trim(), isPseudotext: true });
} else if (tm[2] !== undefined) {
tokens.push({ value: tm[2], isPseudotext: false });
} else {
tokens.push({ value: tm[3], isPseudotext: false });
}
}
// Parse token stream: [LEADING|TRAILING]? <from> BY <to>
let i = 0;
while (i < tokens.length) {
let type: CopyReplacing['type'] = 'EXACT';
// Check for type modifier (only on non-pseudotext tokens)
if (!tokens[i].isPseudotext) {
const upper = tokens[i].value.toUpperCase();
if (upper === 'LEADING') {
type = 'LEADING';
i++;
} else if (upper === 'TRAILING') {
type = 'TRAILING';
i++;
}
}
if (i >= tokens.length) break;
const fromToken = tokens[i];
i++;
// Pseudotext always forces EXACT type
if (fromToken.isPseudotext) type = 'EXACT';
// Expect BY keyword
if (i >= tokens.length) break;
if (tokens[i].value.toUpperCase() !== 'BY') {
// Malformed — skip this token and try to resync
continue;
}
i++; // skip BY
if (i >= tokens.length) break;
const toToken = tokens[i];
i++;
replacings.push({ type, from: fromToken.value, to: toToken.value, isPseudotext: fromToken.isPseudotext || undefined });
}
return replacings;
}
/**
* Scan logical lines for COPY statements.
* COPY statements can span multiple lines and terminate with a period.
*/
function parseCopyStatements(
logicalLines: Array<{ text: string; lineNum: number }>,
): ParsedCopyStatement[] {
const results: ParsedCopyStatement[] = [];
let accumulator: string | null = null;
let startLine = 0;
let endLine = 0;
for (let i = 0; i < logicalLines.length; i++) {
const { text, lineNum } = logicalLines[i];
if (text.length === 0) continue;
// Check for COPY keyword start (not inside a string context)
const copyStart = text.match(/\bCOPY\b/i);
if (accumulator === null) {
if (!copyStart) continue;
// Start accumulating from the COPY keyword onwards
const copyIdx = copyStart.index!;
accumulator = text.substring(copyIdx);
startLine = lineNum;
endLine = lineNum;
} else {
// Continue accumulating
accumulator += ' ' + text.trim();
endLine = lineNum;
}
// Check if statement terminates (period at end of accumulated text)
if (accumulator !== null && /\.\s*$/.test(accumulator)) {
const parsed = parseSingleCopyStatement(accumulator, startLine, endLine);
if (parsed) {
results.push(parsed);
}
accumulator = null;
}
}
// If there's an unterminated COPY (missing period), try to parse what we have
if (accumulator !== null) {
const parsed = parseSingleCopyStatement(accumulator, startLine, endLine);
if (parsed) {
results.push(parsed);
}
}
return results;
}
/**
* Parse a single complete COPY statement string.
*
* Formats:
* COPY target.
* COPY "target".
* COPY target REPLACING ... .
*/
function parseSingleCopyStatement(
stmt: string,
startLine: number,
endLine: number,
): ParsedCopyStatement | null {
// Strip terminating period
const text = stmt.replace(/\.\s*$/, '').trim();
// Extract target: COPY <target> or COPY "<target>" or COPY '<target>'
// Optionally followed by IN/OF <library-name> (COBOL-85 standard: IN and OF are synonyms)
const targetMatch = text.match(
/^COPY\s+(?:"([^"]+)"|'([^']+)'|([A-Z][A-Z0-9-]*))(?:\s+(?:IN|OF)\s+([A-Z][A-Z0-9-]*))?/i,
);
if (!targetMatch) return null;
const target = targetMatch[1] ?? targetMatch[2] ?? targetMatch[3];
const library = targetMatch[4] || undefined;
// Extract REPLACING clause if present
let replacing: CopyReplacing[] = [];
const replacingIdx = text.search(/\bREPLACING\b/i);
if (replacingIdx >= 0) {
const replacingText = text.substring(replacingIdx + 'REPLACING'.length);
replacing = parseReplacingClause(replacingText);
}
return { startLine, endLine, target, replacing, library };
}
// ---------------------------------------------------------------------------
// REPLACING application
// ---------------------------------------------------------------------------
/**
* Apply REPLACING transformations to copybook content.
*
* LEADING: replace prefix in COBOL identifiers.
* TRAILING: replace suffix in COBOL identifiers.
* EXACT: replace exact token matches.
*/
function applyReplacing(content: string, replacings: CopyReplacing[]): string {
if (replacings.length === 0) return content;
// First pass: handle EXACT replacements that contain spaces or non-identifier
// characters (pseudotext). These cannot be handled by identifier-level matching.
let result = content;
for (const r of replacings) {
if (r.type === 'EXACT' && (r.isPseudotext || r.from.includes(' ') || !/^[A-Z][A-Z0-9-]*$/i.test(r.from))) {
const escaped = r.from.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const re = new RegExp(escaped, 'gi');
result = result.replace(re, r.to);
}
}
// Second pass: identifier-level replacements (LEADING, TRAILING, single-word EXACT)
const identifierReplacings = replacings.filter(
r => !(r.type === 'EXACT' && (r.isPseudotext || r.from.includes(' ') || !/^[A-Z][A-Z0-9-]*$/i.test(r.from))),
);
if (identifierReplacings.length === 0) return result;
return result.replace(RE_COBOL_IDENTIFIER, (match) => {
for (const r of identifierReplacings) {
const upper = match.toUpperCase();
const from = r.from.toUpperCase();
const to = r.to.toUpperCase();
switch (r.type) {
case 'LEADING':
if (upper.startsWith(from)) {
return to + match.substring(from.length);
}
break;
case 'TRAILING':
if (upper.endsWith(from)) {
return match.substring(0, match.length - from.length) + to;
}
break;
case 'EXACT':
if (upper === from) {
return to;
}
break;
}
}
return match;
});
}
// ---------------------------------------------------------------------------
// Main expansion engine
// ---------------------------------------------------------------------------
/**
* Expand COBOL COPY statements by inlining copybook content.
*
* @param content - Source COBOL content (after preprocessCobolSource)
* @param filePath - Path of the source file (for diagnostics)
* @param resolveFile - Maps a COPY target name to a filesystem path, or null if not found
* @param readFile - Reads file content by path, or null if unreadable
* @param maxDepth - Maximum nesting depth for recursive expansion (default: 10)
* @returns Expanded content and resolution metadata
*/
export function expandCopies(
content: string,
filePath: string,
resolveFile: (name: string) => string | null,
readFile: (path: string) => string | null,
maxDepth: number = DEFAULT_MAX_DEPTH,
): CopyExpansionResult {
const allResolutions: CopyResolution[] = [];
const warnedCircular = new Set<string>();
let totalExpansions = 0;
const MAX_TOTAL_EXPANSIONS = 500;
const expanded = expandRecursive(content, filePath, 0, new Set<string>());
return {
expandedContent: expanded,
copyResolutions: allResolutions,
};
/**
* Recursively expand COPY statements in content.
*
* @param src - Source content to expand
* @param srcPath - Path of the file being expanded (for cycle detection logging)
* @param depth - Current recursion depth
* @param visited - Set of already-visited copybook paths (cycle detection)
*/
function expandRecursive(
src: string,
srcPath: string,
depth: number,
visited: Set<string>,
): string {
const rawLines = src.split(/\r?\n/);
const logicalLines = mergeLogicalLines(rawLines);
const copyStatements = parseCopyStatements(logicalLines);
// No COPY statements — return as-is
if (copyStatements.length === 0) return src;
// Process COPY statements in reverse order so line numbers stay valid
// as we splice content
const outputLines = [...rawLines];
for (let ci = copyStatements.length - 1; ci >= 0; ci--) {
const cs = copyStatements[ci];
// Resolve the copybook path
const resolvedPath = resolveFile(cs.target);
// Record resolution metadata
allResolutions.push({
copyTarget: cs.target,
resolvedPath,
line: cs.startLine,
replacing: cs.replacing,
library: cs.library,
});
// Cannot resolve — keep original lines
if (resolvedPath === null) {
continue;
}
// Cycle detection
if (visited.has(resolvedPath)) {
if (!warnedCircular.has(resolvedPath)) {
warnedCircular.add(resolvedPath);
console.warn(
`[cobol-copy-expander] Circular COPY detected: ${cs.target} (${resolvedPath}) ` +
`includes itself. Skipping expansion.`,
);
}
continue;
}
// Max depth exceeded — keep unexpanded
if (depth >= maxDepth) {
console.warn(
`[cobol-copy-expander] Max expansion depth (${maxDepth}) reached for ` +
`COPY ${cs.target} in ${srcPath}. Skipping expansion.`,
);
continue;
}
// Guard against exponential breadth amplification (N copybooks each with N COPYs)
if (++totalExpansions > MAX_TOTAL_EXPANSIONS) {
if (!warnedCircular.has('__max_total__')) {
warnedCircular.add('__max_total__');
console.warn(
`[cobol-copy-expander] Max total expansions (${MAX_TOTAL_EXPANSIONS}) reached ` +
`in ${srcPath}. Skipping further expansions.`,
);
}
continue;
}
// Read the copybook content
const copybookContent = readFile(resolvedPath);
if (copybookContent === null) {
continue;
}
// Apply REPLACING transformations
const replaced = applyReplacing(copybookContent, cs.replacing);
// Recurse into the copybook for nested COPYs
const nestedVisited = new Set(visited);
nestedVisited.add(resolvedPath);
const expandedCopybook = expandRecursive(
replaced,
resolvedPath,
depth + 1,
nestedVisited,
);
// Splice: replace the COPY statement lines with expanded content
// startLine/endLine are 1-based; convert to 0-based array index
const expansionLines = expandedCopybook.split('\n');
const removeCount = cs.endLine - cs.startLine + 1;
outputLines.splice(cs.startLine - 1, removeCount, ...expansionLines);
}
return outputLines.join('\n');
}
}

1771
gitnexus/src/core/ingestion/cobol/cobol-preprocessor.ts Normal file
View file

File diff suppressed because it is too large Load diff

263
gitnexus/src/core/ingestion/cobol/jcl-parser.ts Normal file
View file

@ -0,0 +1,263 @@
/**
* JCL Parser — Regex single-pass extraction.
*
* Extracts JCL constructs from mainframe job streams:
* - JOB statements (job name, CLASS, MSGCLASS)
* - EXEC statements (step -> program or proc)
* - DD statements (dataset references, DISP)
* - PROC definitions (in-stream and catalogued)
* - INCLUDE MEMBER= directives
* - SET symbolic parameters
* - IF/ELSE/ENDIF conditional execution
* - JCLLIB ORDER= search paths
*
* Pattern follows cobol-preprocessor.ts — regex-only, no tree-sitter.
*/
export interface JclParseResults {
jobs: Array<{ name: string; line: number; class?: string; msgclass?: string }>;
steps: Array<{ name: string; jobName: string; program?: string; proc?: string; line: number }>;
ddStatements: Array<{ ddName: string; stepName: string; dataset?: string; disp?: string; line: number }>;
procs: Array<{ name: string; line: number; isInStream: boolean }>;
includes: Array<{ member: string; line: number }>;
sets: Array<{ variable: string; value: string; line: number }>;
jcllib: Array<{ order: string[]; line: number }>;
conditionals: Array<{ type: 'IF' | 'ELSE' | 'ENDIF'; condition?: string; line: number }>;
}
// ── JCL statement patterns ─────────────────────────────────────────────
// JCL continuation: line ends with a non-blank in col 72, next line starts with //
// We handle continuations by joining lines before matching.
/** Match //jobname JOB ... */
const JOB_RE = /^\/\/(\w{1,8})\s+JOB\s+(.*)/i;
/** Match //stepname EXEC PGM=program or //stepname EXEC procname */
const EXEC_RE = /^\/\/(\w{1,8})\s+EXEC\s+(.*)/i;
/** Match //ddname DD ... */
const DD_RE = /^\/\/(\w{1,8})\s+DD\s+(.*)/i;
/** Match // JCLLIB ORDER=(lib1,lib2,...) */
const JCLLIB_RE = /^\/\/\s+JCLLIB\s+ORDER=\(([^)]+)\)/i;
/** Match // IF condition THEN */
const IF_RE = /^\/\/\s+IF\s+(.+)\s+THEN/i;
/** Match // ELSE */
const ELSE_RE = /^\/\/\s+ELSE\b/i;
/** Match // ENDIF */
const ENDIF_RE = /^\/\/\s+ENDIF\b/i;
/** Match // INCLUDE MEMBER=name */
const INCLUDE_RE = /^\/\/\s+INCLUDE\s+MEMBER=(\w+)/i;
/** Match // SET var=value */
const SET_RE = /^\/\/\s+SET\s+(\w+)=(.+)/i;
/** Match // PROC or //name PROC */
const PROC_RE = /^\/\/(\w*)\s+PROC\b/i;
/** Match // PEND */
const PEND_RE = /^\/\/\s+PEND\b/i;
// ── Parameter extractors ───────────────────────────────────────────────
function extractParam(params: string, key: string): string | undefined {
// Match KEY=VALUE or KEY='VALUE' in JCL parameter string
const re = new RegExp(`${key}=(?:'([^']*)'|(\\S+?))(?:[,\\s]|$)`, 'i');
const m = params.match(re);
return m ? (m[1] ?? m[2]) : undefined;
}
function extractPgm(params: string): string | undefined {
return extractParam(params, 'PGM');
}
function extractProc(params: string): string | undefined {
// If no PGM= keyword, the first positional parameter is the proc name
if (/PGM=/i.test(params)) return undefined;
const cleaned = params.replace(/,.*/, '').trim();
// Proc name is the first token (no = sign)
if (cleaned && !cleaned.includes('=')) {
return cleaned.replace(/[,\s].*/s, '').toUpperCase();
}
return undefined;
}
function extractDsn(params: string): string | undefined {
return extractParam(params, 'DSN') ?? extractParam(params, 'DSNAME');
}
function extractDisp(params: string): string | undefined {
const m = params.match(/DISP=\(?\s*([^),\s]+)/i);
return m ? m[1] : undefined;
}
/**
* Parse a JCL file and extract all constructs.
*
* @param content - Raw JCL file content
* @param filePath - Path for diagnostics (not used in extraction)
* @returns Parsed JCL results
*/
export function parseJcl(content: string, filePath: string): JclParseResults {
const results: JclParseResults = {
jobs: [],
steps: [],
ddStatements: [],
procs: [],
includes: [],
sets: [],
jcllib: [],
conditionals: [],
};
const rawLines = content.split(/\r?\n/);
// Join continuation lines: a line ending with non-blank in col 71 (0-indexed)
// followed by a line starting with // is a continuation.
const lines: Array<{ text: string; lineNum: number }> = [];
let i = 0;
while (i < rawLines.length) {
let line = rawLines[i];
const lineNum = i + 1;
// JCL continuation: if line is exactly 72+ chars and col 72 is non-blank
// and the next line starts with //, join them.
while (
i + 1 < rawLines.length &&
line.length >= 72 &&
line[71] !== ' ' &&
rawLines[i + 1].startsWith('//')
) {
i++;
// Continuation text starts after // and leading spaces
const contText = rawLines[i].substring(2).replace(/^\s+/, ' ');
// Remove the continuation marker (col 72+) from current line
line = line.substring(0, 71).trimEnd() + contText;
}
lines.push({ text: line, lineNum });
i++;
}
let currentJobName = '';
let currentStepName = '';
let inStreamProcName = '';
for (const { text, lineNum } of lines) {
// Skip JCL comments (starting with //* )
if (text.startsWith('//*')) continue;
// Skip non-JCL lines (don't start with //)
if (!text.startsWith('//')) continue;
// PROC definition (in-stream)
const procMatch = text.match(PROC_RE);
if (procMatch) {
const procName = procMatch[1] || inStreamProcName;
if (procName) {
results.procs.push({ name: procName.toUpperCase(), line: lineNum, isInStream: true });
}
inStreamProcName = procName?.toUpperCase() || '';
continue;
}
// PEND (end of in-stream proc)
if (PEND_RE.test(text)) {
inStreamProcName = '';
continue;
}
// JCLLIB ORDER=
const jcllibMatch = text.match(JCLLIB_RE);
if (jcllibMatch) {
const libs = jcllibMatch[1].split(',').map(s => s.trim().replace(/'/g, ''));
results.jcllib.push({ order: libs, line: lineNum });
continue;
}
// IF/ELSE/ENDIF
const ifMatch = text.match(IF_RE);
if (ifMatch) {
results.conditionals.push({ type: 'IF', condition: ifMatch[1].trim(), line: lineNum });
continue;
}
if (ELSE_RE.test(text)) {
results.conditionals.push({ type: 'ELSE', line: lineNum });
continue;
}
if (ENDIF_RE.test(text)) {
results.conditionals.push({ type: 'ENDIF', line: lineNum });
continue;
}
// INCLUDE MEMBER=
const includeMatch = text.match(INCLUDE_RE);
if (includeMatch) {
results.includes.push({ member: includeMatch[1].toUpperCase(), line: lineNum });
continue;
}
// SET var=value
const setMatch = text.match(SET_RE);
if (setMatch) {
results.sets.push({
variable: setMatch[1].toUpperCase(),
value: setMatch[2].trim().replace(/,\s*$/, ''),
line: lineNum,
});
continue;
}
// JOB statement
const jobMatch = text.match(JOB_RE);
if (jobMatch) {
currentJobName = jobMatch[1].toUpperCase();
const params = jobMatch[2];
results.jobs.push({
name: currentJobName,
line: lineNum,
class: extractParam(params, 'CLASS'),
msgclass: extractParam(params, 'MSGCLASS'),
});
continue;
}
// EXEC statement
const execMatch = text.match(EXEC_RE);
if (execMatch) {
currentStepName = execMatch[1].toUpperCase();
const params = execMatch[2];
const pgm = extractPgm(params);
const proc = pgm ? undefined : extractProc(params);
results.steps.push({
name: currentStepName,
jobName: currentJobName,
program: pgm?.toUpperCase(),
proc: proc?.toUpperCase(),
line: lineNum,
});
continue;
}
// DD statement
const ddMatch = text.match(DD_RE);
if (ddMatch) {
const ddName = ddMatch[1].toUpperCase();
const params = ddMatch[2];
results.ddStatements.push({
ddName,
stepName: currentStepName,
dataset: extractDsn(params)?.toUpperCase(),
disp: extractDisp(params)?.toUpperCase(),
line: lineNum,
});
continue;
}
}
return results;
}

274
gitnexus/src/core/ingestion/cobol/jcl-processor.ts Normal file
View file

@ -0,0 +1,274 @@
/**
* JCL Processor — Converts JCL parse results into graph nodes and edges.
*
* Maps JCL entities to existing graph types (no new tables):
* - Job -> CodeElement (description: "jcl-job class:A msgclass:X")
* - Step -> CodeElement (description: "jcl-step pgm:PROGRAMNAME")
* - Dataset -> CodeElement (description: "jcl-dataset disp:SHR")
* - PROC -> Module
*
* Edges:
* - Job CONTAINS Step
* - Step CALLS Module (when PGM= matches an indexed program)
* - Step references Dataset (CALLS edge with reason "jcl-dd")
* - Job/Step IMPORTS PROC
*
* Pattern follows detectCrossProgamContracts() in pipeline.ts.
*/
import { parseJcl, type JclParseResults } from './jcl-parser.js';
import type { KnowledgeGraph } from '../../graph/types.js';
import { generateId } from '../../../lib/utils.js';
export interface JclProcessResult {
jobCount: number;
stepCount: number;
datasetCount: number;
programLinks: number;
}
/**
* Process JCL files and integrate into the knowledge graph.
*
* @param graph - The in-memory knowledge graph
* @param jclPaths - File paths of JCL files
* @param jclContents - Map of path -> file content
* @returns Summary of what was added
*/
export function processJclFiles(
graph: KnowledgeGraph,
jclPaths: string[],
jclContents: Map<string, string>,
): JclProcessResult {
let jobCount = 0;
let stepCount = 0;
let datasetCount = 0;
let programLinks = 0;
// Collect all Module names for step -> program linking
const moduleNames = new Map<string, string>(); // uppercase name -> node id
graph.forEachNode(node => {
if (node.label === 'Module') {
const nodeName = node.properties.name;
if (typeof nodeName === 'string') {
moduleNames.set(nodeName.toUpperCase(), node.id);
}
}
});
for (const filePath of jclPaths) {
const content = jclContents.get(filePath);
if (!content) continue;
const parsed = parseJcl(content, filePath);
const result = integrateJclResults(graph, parsed, filePath, moduleNames);
jobCount += result.jobCount;
stepCount += result.stepCount;
datasetCount += result.datasetCount;
programLinks += result.programLinks;
}
return { jobCount, stepCount, datasetCount, programLinks };
}
function integrateJclResults(
graph: KnowledgeGraph,
parsed: JclParseResults,
filePath: string,
moduleNames: Map<string, string>,
): JclProcessResult {
let jobCount = 0;
let stepCount = 0;
let datasetCount = 0;
let programLinks = 0;
// Track step node IDs for DD -> step linking
const stepNodeIds = new Map<string, string>(); // stepName -> nodeId
// 1. Create Job nodes
for (const job of parsed.jobs) {
const jobId = generateId('CodeElement', `${filePath}:job:${job.name}`);
const classPart = job.class ? ` class:${job.class}` : '';
const msgPart = job.msgclass ? ` msgclass:${job.msgclass}` : '';
graph.addNode({
id: jobId,
label: 'CodeElement',
properties: {
name: job.name,
filePath,
startLine: job.line,
endLine: job.line,
description: `jcl-job${classPart}${msgPart}`,
},
});
// Link File -> Job (CONTAINS)
const fileId = generateId('File', filePath);
graph.addRelationship({
id: `${fileId}_contains_${jobId}`,
type: 'CONTAINS',
sourceId: fileId,
targetId: jobId,
confidence: 1.0,
reason: 'jcl-job',
});
jobCount++;
}
// 1.5 Pre-register in-stream PROCs so steps can reference them
// (fixes ordering bug: steps processed before PROCs were registered)
for (const proc of parsed.procs) {
const procId = generateId('Module', `${filePath}:proc:${proc.name}`);
moduleNames.set(proc.name.toUpperCase(), procId);
}
// 2. Create Step nodes and link to programs
for (const step of parsed.steps) {
const stepId = generateId('CodeElement', `${filePath}:step:${step.jobName}:${step.name}`);
const pgmPart = step.program ? ` pgm:${step.program}` : '';
const procPart = step.proc ? ` proc:${step.proc}` : '';
graph.addNode({
id: stepId,
label: 'CodeElement',
properties: {
name: step.name,
filePath,
startLine: step.line,
endLine: step.line,
description: `jcl-step${pgmPart}${procPart}`,
},
});
stepNodeIds.set(step.name, stepId);
// Link Job -> Step (CONTAINS)
if (step.jobName) {
const jobId = generateId('CodeElement', `${filePath}:job:${step.jobName}`);
graph.addRelationship({
id: `${jobId}_contains_${stepId}`,
type: 'CONTAINS',
sourceId: jobId,
targetId: stepId,
confidence: 1.0,
reason: 'jcl-step',
});
}
// Link Step -> Module (CALLS) when PGM= matches an indexed program
if (step.program) {
const moduleId = moduleNames.get(step.program.toUpperCase());
if (moduleId) {
graph.addRelationship({
id: `${stepId}_calls_${moduleId}`,
type: 'CALLS',
sourceId: stepId,
targetId: moduleId,
confidence: 0.95,
reason: 'jcl-exec-pgm',
});
programLinks++;
}
}
// Link Step -> PROC (CALLS) — PROC as Module
if (step.proc) {
const procModuleId = moduleNames.get(step.proc.toUpperCase());
if (procModuleId) {
graph.addRelationship({
id: `${stepId}_calls_proc_${procModuleId}`,
type: 'CALLS',
sourceId: stepId,
targetId: procModuleId,
confidence: 0.9,
reason: 'jcl-exec-proc',
});
}
}
stepCount++;
}
// 3. Create Dataset nodes from DD statements
const seenDatasets = new Set<string>();
for (const dd of parsed.ddStatements) {
if (!dd.dataset) continue;
// Create dataset node (deduplicated per file)
const datasetKey = `${filePath}:dataset:${dd.dataset}`;
const datasetId = generateId('CodeElement', datasetKey);
if (!seenDatasets.has(dd.dataset)) {
const dispPart = dd.disp ? ` disp:${dd.disp}` : '';
graph.addNode({
id: datasetId,
label: 'CodeElement',
properties: {
name: dd.dataset,
filePath,
startLine: dd.line,
endLine: dd.line,
description: `jcl-dataset${dispPart}`,
},
});
seenDatasets.add(dd.dataset);
datasetCount++;
}
// Link Step -> Dataset (CALLS with reason jcl-dd)
const stepId = stepNodeIds.get(dd.stepName);
if (stepId) {
graph.addRelationship({
id: `${stepId}_dd_${dd.ddName}_${datasetId}`,
type: 'CALLS',
sourceId: stepId,
targetId: datasetId,
confidence: 0.85,
reason: `jcl-dd:${dd.ddName}`,
});
}
}
// 4. Create PROC nodes (in-stream procs as Module)
for (const proc of parsed.procs) {
if (!proc.isInStream) continue;
const procId = generateId('Module', `${filePath}:proc:${proc.name}`);
graph.addNode({
id: procId,
label: 'Module',
properties: {
name: proc.name,
filePath,
startLine: proc.line,
endLine: proc.line,
description: 'jcl-proc-instream',
},
});
// Register for step linking
moduleNames.set(proc.name.toUpperCase(), procId);
}
// 5. INCLUDE directives -> IMPORTS edges
for (const inc of parsed.includes) {
const moduleId = moduleNames.get(inc.member.toUpperCase());
if (moduleId) {
const fileId = generateId('File', filePath);
graph.addRelationship({
id: `${fileId}_includes_${moduleId}`,
type: 'IMPORTS',
sourceId: fileId,
targetId: moduleId,
confidence: 0.9,
reason: 'jcl-include',
});
}
}
return { jobCount, stepCount, datasetCount, programLinks };
}

3
gitnexus/src/core/ingestion/entry-point-scoring.ts
View file

@ -226,6 +226,7 @@ export const ENTRY_POINT_PATTERNS = {
/^onEvent$/, // BLoC event handler
/^mapEventToState$/, // Legacy BLoC pattern
],
[SupportedLanguages.Cobol]: [], // Standalone regex processor — no tree-sitter entry points
} satisfies Record<SupportedLanguages, RegExp[]>;
/** Pre-computed merged patterns (universal + language-specific) to avoid per-call array allocation. */
@ -325,7 +326,7 @@ export function calculateEntryPointScore(
// Check positive patterns
const allPatterns = MERGED_ENTRY_POINT_PATTERNS[language];
if (allPatterns.some(p => p.test(name))) {
if (allPatterns?.some(p => p.test(name))) {
nameMultiplier = 1.5; // Bonus for matching entry point pattern
reasons.push('entry-pattern');
}

1
gitnexus/src/core/ingestion/framework-detection.ts
View file

@ -601,6 +601,7 @@ export const AST_FRAMEWORK_PATTERNS_BY_LANGUAGE = {
{ framework: 'flutter', entryPointMultiplier: 2.5, reason: 'flutter-widget', patterns: FRAMEWORK_AST_PATTERNS.flutter },
{ framework: 'riverpod', entryPointMultiplier: 2.8, reason: 'riverpod-pattern', patterns: FRAMEWORK_AST_PATTERNS.riverpod },
],
[SupportedLanguages.Cobol]: [], // Standalone regex processor — no AST framework patterns
} satisfies Record<SupportedLanguages, AstFrameworkPatternConfig[]>;
/** Pre-lowercased patterns for O(1) pattern matching at runtime */

7
gitnexus/src/core/ingestion/language-provider.ts
View file

@ -40,7 +40,12 @@ interface LanguageProviderConfig {
readonly extensions: readonly string[];
// ── Parser ────────────────────────────────────────────────────────
/** Tree-sitter query strings for definitions, imports, calls, heritage */
/** Parse strategy: 'tree-sitter' (default) uses AST parsing via tree-sitter.
* 'standalone' means the language has its own regex-based processor and
* should be skipped by the tree-sitter pipeline (e.g., COBOL, Markdown). */
readonly parseStrategy?: 'tree-sitter' | 'standalone';
/** Tree-sitter query strings for definitions, imports, calls, heritage.
* Required for tree-sitter languages; empty string for standalone processors. */
readonly treeSitterQueries: string;
// ── Core (required) ───────────────────────────────────────────────

27
gitnexus/src/core/ingestion/languages/cobol.ts Normal file
View file

@ -0,0 +1,27 @@
/**
* COBOL Language Provider
*
* Standalone regex-based processor — no tree-sitter grammar.
* COBOL files (.cbl, .cob, .cobol, .cpy, .copybook) are detected and
* processed by cobol-processor.ts in pipeline Phase 2.6, not by the
* tree-sitter pipeline.
*
* This provider exists to satisfy the SupportedLanguages exhaustiveness
* checks and to declare parseStrategy: 'standalone'.
*/
import { SupportedLanguages } from '../../../config/supported-languages.js';
import { defineLanguage } from '../language-provider.js';
export const cobolProvider = defineLanguage({
id: SupportedLanguages.Cobol,
parseStrategy: 'standalone',
extensions: [], // COBOL files detected by cobol-processor's isCobolFile/isJclFile
treeSitterQueries: '',
typeConfig: {
declarationNodeTypes: new Set(),
extractDeclaration: () => null,
extractParameter: () => null,
},
exportChecker: () => false,
importResolver: () => null,
});

2
gitnexus/src/core/ingestion/languages/index.ts
View file

@ -23,6 +23,7 @@ import { phpProvider } from './php.js';
import { rubyProvider } from './ruby.js';
import { swiftProvider } from './swift.js';
import { dartProvider } from './dart.js';
import { cobolProvider } from './cobol.js';
export const providers = {
[SupportedLanguages.JavaScript]: javascriptProvider,
@ -39,6 +40,7 @@ export const providers = {
[SupportedLanguages.Ruby]: rubyProvider,
[SupportedLanguages.Swift]: swiftProvider,
[SupportedLanguages.Dart]: dartProvider,
[SupportedLanguages.Cobol]: cobolProvider,
} satisfies Record<SupportedLanguages, LanguageProvider>;
/** Get provider by language enum (always succeeds for SupportedLanguages). */

29
gitnexus/src/core/ingestion/pipeline.ts
View file

@ -1,6 +1,7 @@
import { createKnowledgeGraph } from '../graph/graph.js';
import { processStructure } from './structure-processor.js';
import { processMarkdown } from './markdown-processor.js';
import { processCobol, isCobolFile, isJclFile } from './cobol-processor.js';
import { processParsing } from './parsing-processor.js';
import {
processImports,
@ -464,6 +465,14 @@ async function runScanAndStructure(
stats: { filesProcessed: totalFiles, totalFiles, nodesCreated: graph.nodeCount },
});
// ── Custom (non-tree-sitter) processors ─────────────────────────────
// Each custom processor follows the pattern in markdown-processor.ts:
// 1. Export a process function: (graph, files, allPathSet) => result
// 2. Export a file detection function: (path) => boolean
// 3. Filter files by extension, write nodes/edges directly to graph
// To add a new language: create a new processor file, import it here,
// and add a filter-read-call-log block following the pattern below.
// ── Phase 2.5: Markdown processing (headings + cross-links) ────────
const mdScanned = scannedFiles.filter(f => f.path.endsWith('.md') || f.path.endsWith('.mdx'));
if (mdScanned.length > 0) {
@ -478,6 +487,26 @@ async function runScanAndStructure(
}
}
// ── Phase 2.6: COBOL processing (regex extraction, no tree-sitter) ──
const cobolScanned = scannedFiles.filter(f => isCobolFile(f.path) || isJclFile(f.path));
if (cobolScanned.length > 0) {
const cobolContents = await readFileContents(repoPath, cobolScanned.map(f => f.path));
const cobolFiles = cobolScanned
.filter(f => cobolContents.has(f.path))
.map(f => ({ path: f.path, content: cobolContents.get(f.path)! }));
const allPathSet = new Set(allPaths);
const cobolResult = processCobol(graph, cobolFiles, allPathSet);
if (isDev) {
console.log(` COBOL: ${cobolResult.programs} programs, ${cobolResult.paragraphs} paragraphs, ${cobolResult.sections} sections from ${cobolFiles.length} files`);
if (cobolResult.execSqlBlocks > 0 || cobolResult.execCicsBlocks > 0 || cobolResult.entryPoints > 0) {
console.log(` COBOL enriched: ${cobolResult.execSqlBlocks} SQL blocks, ${cobolResult.execCicsBlocks} CICS blocks, ${cobolResult.entryPoints} entry points, ${cobolResult.moves} moves, ${cobolResult.fileDeclarations} file declarations`);
}
if (cobolResult.jclJobs > 0) {
console.log(` JCL: ${cobolResult.jclJobs} jobs, ${cobolResult.jclSteps} steps`);
}
}
}
return { scannedFiles, allPaths, totalFiles };
}

1
gitnexus/src/core/ingestion/tree-sitter-queries.ts
View file

@ -1164,4 +1164,5 @@ export const LANGUAGE_QUERIES: Record<SupportedLanguages, string> = {
[SupportedLanguages.Ruby]: RUBY_QUERIES,
[SupportedLanguages.Swift]: SWIFT_QUERIES,
[SupportedLanguages.Dart]: DART_QUERIES,
[SupportedLanguages.Cobol]: '', // Standalone regex processor — no tree-sitter queries
};

25
gitnexus/test/fixtures/lang-resolution/cobol-app/AUDITLOG.cbl vendored Normal file
View file

@ -0,0 +1,25 @@
IDENTIFICATION DIVISION.
PROGRAM-ID. AUDITLOG.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-LOG-MESSAGE PIC X(80).
01 WS-TIMESTAMP PIC X(26).
LINKAGE SECTION.
01 LS-CUST-ID PIC 9(8).
01 LS-AMOUNT PIC 9(7)V99.
PROCEDURE DIVISION USING LS-CUST-ID LS-AMOUNT.
MAIN-PARAGRAPH.
PERFORM WRITE-LOG
GOBACK.
WRITE-LOG.
STRING 'Customer ' LS-CUST-ID ' amount ' LS-AMOUNT
DELIMITED BY SIZE INTO WS-LOG-MESSAGE
DISPLAY WS-LOG-MESSAGE.
ENTRY "AUDITLOG-BATCH" USING LS-CUST-ID.
DISPLAY 'Batch audit for ' LS-CUST-ID
GOBACK.

3
gitnexus/test/fixtures/lang-resolution/cobol-app/COPYLIB.cpy vendored Normal file
View file

@ -0,0 +1,3 @@
01 PREFIX-RECORD.
05 PREFIX-CODE PIC X(10).
05 PREFIX-NAME PIC X(30).

6
gitnexus/test/fixtures/lang-resolution/cobol-app/CUSTDAT.cpy vendored Normal file
View file

@ -0,0 +1,6 @@
01 WS-CUSTOMER-DATA.
05 WS-CUST-CODE PIC X(10).
05 WS-CUST-TYPE PIC X(3).
88 PREMIUM-CUSTOMER VALUE 'PRM'.
88 REGULAR-CUSTOMER VALUE 'REG'.
05 WS-CUST-ADDR PIC X(50).

74
gitnexus/test/fixtures/lang-resolution/cobol-app/CUSTUPDT.cbl vendored Normal file
View file

@ -0,0 +1,74 @@
IDENTIFICATION DIVISION.
PROGRAM-ID. CUSTUPDT.
AUTHOR. TEST.
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT CUSTOMER-FILE ASSIGN TO 'CUSTFILE'
ORGANIZATION IS INDEXED
ACCESS IS DYNAMIC
RECORD KEY IS CUST-ID
FILE STATUS IS WS-FILE-STATUS.
DATA DIVISION.
FILE SECTION.
FD CUSTOMER-FILE.
01 CUSTOMER-RECORD.
05 CUST-ID PIC 9(8).
05 CUST-NAME PIC X(30).
05 CUST-BALANCE PIC 9(7)V99.
WORKING-STORAGE SECTION.
01 WS-FILE-STATUS PIC XX.
01 WS-CUSTOMER-NAME PIC X(30).
01 WS-AMOUNT PIC 9(7)V99.
01 WS-EOF PIC 9 VALUE 0.
88 END-OF-FILE VALUE 1.
01 WS-AMT PIC 9(5)V99.
01 WS-PROG-NAME PIC X(8).
01 FIELD-A PIC 9(5)V99.
01 FIELD-B PIC 9(5)V99.
COPY COPYLIB REPLACING ==PREFIX-== BY ==WS-==.
LINKAGE SECTION.
01 LS-PARAM PIC X(20).
PROCEDURE DIVISION.
INIT-SECTION SECTION.
MAIN-PARAGRAPH.
PERFORM INIT-PARAGRAPH
PERFORM PROCESS-PARAGRAPH
PERFORM CLEANUP-PARAGRAPH
STOP RUN.
INIT-PARAGRAPH.
OPEN I-O CUSTOMER-FILE
MOVE SPACES TO WS-CUSTOMER-NAME.
PROCESSING-SECTION SECTION.
PROCESS-PARAGRAPH.
PERFORM READ-CUSTOMER THRU WRITE-CUSTOMER
CALL "AUDITLOG" USING CUST-ID WS-AMOUNT
CALL WS-PROG-NAME.
READ-CUSTOMER.
READ CUSTOMER-FILE
NOT AT END
MOVE CUST-NAME TO WS-CUSTOMER-NAME
END-READ.
UPDATE-BALANCE.
ADD WS-AMOUNT TO CUST-BALANCE
MOVE WS-AMOUNT TO CUST-BALANCE
MOVE WS-AMT TO FIELD-A FIELD-B.
WRITE-CUSTOMER.
REWRITE CUSTOMER-RECORD.
CLEANUP-PARAGRAPH.
CLOSE CUSTOMER-FILE.
ENTRY 'ALTENTRY' USING LS-PARAM.
DISPLAY 'ALTERNATE ENTRY POINT'
GOBACK.

33
gitnexus/test/fixtures/lang-resolution/cobol-app/NESTED.cbl vendored Normal file
View file

@ -0,0 +1,33 @@
IDENTIFICATION DIVISION.
PROGRAM-ID. OUTER-PROG.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-OUTER-FLAG PIC 9 VALUE 0.
PROCEDURE DIVISION.
OUTER-MAIN.
PERFORM OUTER-PROCESS
CALL "INNER-PROG"
STOP RUN.
OUTER-PROCESS.
DISPLAY 'OUTER PROCESSING'.
IDENTIFICATION DIVISION.
PROGRAM-ID. INNER-PROG.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-INNER-CODE PIC X(5).
PROCEDURE DIVISION.
INNER-MAIN.
PERFORM INNER-PROCESS
GOBACK.
INNER-PROCESS.
DISPLAY 'INNER PROCESSING'.
END PROGRAM INNER-PROG.
END PROGRAM OUTER-PROG.

94
gitnexus/test/fixtures/lang-resolution/cobol-app/RPTGEN.cbl vendored Normal file
View file

@ -0,0 +1,94 @@
IDENTIFICATION DIVISION.
PROGRAM-ID. RPTGEN.
DATA DIVISION.
WORKING-STORAGE SECTION.
COPY CUSTDAT.
01 WS-REPORT-LINE PIC X(132).
01 WS-SQL-CODE PIC S9(9) COMP.
01 WS-COUNT PIC 9(4).
01 WS-MAP-NAME PIC X(8).
01 WS-SORT-FILE PIC X(8).
01 WS-QUEUE-NAME PIC X(16).
01 WS-NEXT-PGM PIC X(8).
PROCEDURE DIVISION.
MAIN-PARAGRAPH.
PERFORM FETCH-DATA
PERFORM FORMAT-REPORT
PERFORM SEND-SCREEN
CALL "CUSTUPDT"
GO TO EXIT-PARAGRAPH.
FETCH-DATA.
EXEC SQL
SELECT CUST_NAME, CUST_BALANCE
FROM CUSTOMER
WHERE CUST_ID = :WS-CUST-CODE
END-EXEC.
FORMAT-REPORT.
PERFORM WS-COUNT TIMES
MOVE WS-CUST-CODE TO WS-REPORT-LINE
END-PERFORM
PERFORM MAIN-PARAGRAPH THRU FORMAT-REPORT
IF WS-COUNT > 0 PERFORM FETCH-DATA
ELSE PERFORM SEND-SCREEN
END-IF
SORT WS-SORT-FILE USING CUSTOMER-DATA
GIVING WS-REPORT-LINE.
SORT WS-SORT-FILE ON ASCENDING KEY WS-COUNT
INPUT PROCEDURE IS BUILD-SORT-INPUT
OUTPUT PROCEDURE IS WRITE-SORTED.
MOVE CORR WS-CUSTOMER-DATA TO WS-REPORT-LINE
SEARCH WS-CUSTOMER-DATA
GO TO FETCH-DATA FORMAT-REPORT SEND-SCREEN
DEPENDING ON WS-COUNT.
SEND-SCREEN.
EXEC CICS
SEND MAP(WS-MAP-NAME) MAPSET('CUSTSET')
FROM(WS-REPORT-LINE)
END-EXEC.
EXEC CICS
LINK PROGRAM('AUDITLOG')
END-EXEC.
EXEC CICS
XCTL PROGRAM('CUSTUPDT')
END-EXEC.
EXEC CICS
READ FILE('CUSTFILE')
INTO(WS-CUSTOMER-DATA)
END-EXEC.
EXEC CICS
WRITEQ TS QUEUE('RPTQUEUE')
FROM(WS-REPORT-LINE)
END-EXEC.
EXEC CICS
HANDLE ABEND LABEL(ABEND-HANDLER)
END-EXEC.
EXEC CICS
RETURN TRANSID('RPTG')
END-EXEC.
EXEC CICS
XCTL PROGRAM(WS-NEXT-PGM)
END-EXEC.
BUILD-SORT-INPUT.
DISPLAY 'BUILDING SORT INPUT'.
WRITE-SORTED.
DISPLAY 'WRITING SORTED OUTPUT'.
ABEND-HANDLER.
DISPLAY 'ABEND OCCURRED'.
EXIT-PARAGRAPH.
STOP RUN.

5
gitnexus/test/fixtures/lang-resolution/cobol-app/RUNJOBS.jcl vendored Normal file
View file

@ -0,0 +1,5 @@
//CUSTJOB JOB (ACCT),'CUSTOMER UPDATE',CLASS=A,MSGCLASS=X
//STEP1 EXEC PGM=CUSTUPDT
//CUSTFILE DD DSN=PROD.CUSTOMER.MASTER,DISP=SHR
//STEP2 EXEC PGM=RPTGEN
//SYSOUT DD SYSOUT=*

608
gitnexus/test/integration/resolvers/cobol.test.ts Normal file
View file

@ -0,0 +1,608 @@
/**
* COBOL: Exhaustive strict integration test.
*
* Every single node and edge produced by the COBOL/JCL pipeline is asserted
* with exact counts AND exact sorted edge-pair lists. No fuzzy assertions.
*
* Ground truth captured from the cobol-app fixture:
* CUSTUPDT.cbl, AUDITLOG.cbl, RPTGEN.cbl, NESTED.cbl,
* CUSTDAT.cpy, COPYLIB.cpy, RUNJOBS.jcl
*/
import { describe, it, expect, beforeAll } from 'vitest';
import path from 'path';
import {
FIXTURES, getRelationships, getNodesByLabel, edgeSet,
runPipelineFromRepo, type PipelineResult,
} from './helpers.js';
describe('COBOL full system extraction', () => {
let result: PipelineResult;
beforeAll(async () => {
result = await runPipelineFromRepo(
path.join(FIXTURES, 'cobol-app'),
() => {},
{ skipGraphPhases: true },
);
}, 60000);
// =====================================================================
// NODE COMPLETENESS — exact count + exact sorted name list per label
// =====================================================================
describe('node completeness', () => {
it('produces exactly 5 Module nodes', () => {
const nodes = getNodesByLabel(result, 'Module');
expect(nodes.length).toBe(5);
expect(nodes).toEqual(['AUDITLOG', 'CUSTUPDT', 'INNER-PROG', 'OUTER-PROG', 'RPTGEN']);
});
it('produces exactly 21 Function nodes', () => {
const nodes = getNodesByLabel(result, 'Function');
expect(nodes.length).toBe(21);
expect(nodes).toEqual([
'ABEND-HANDLER', 'BUILD-SORT-INPUT', 'CLEANUP-PARAGRAPH',
'EXIT-PARAGRAPH', 'FETCH-DATA', 'FORMAT-REPORT', 'INIT-PARAGRAPH',
'INNER-MAIN', 'INNER-PROCESS',
'MAIN-PARAGRAPH', 'MAIN-PARAGRAPH', 'MAIN-PARAGRAPH',
'OUTER-MAIN', 'OUTER-PROCESS',
'PROCESS-PARAGRAPH', 'READ-CUSTOMER', 'SEND-SCREEN',
'UPDATE-BALANCE', 'WRITE-CUSTOMER', 'WRITE-LOG', 'WRITE-SORTED',
]);
});
it('produces exactly 2 Namespace nodes', () => {
expect(getNodesByLabel(result, 'Namespace')).toEqual(['INIT-SECTION', 'PROCESSING-SECTION']);
});
it('produces exactly 36 Property nodes', () => {
const nodes = getNodesByLabel(result, 'Property');
expect(nodes.length).toBe(36);
expect(nodes).toEqual([
'CUST-BALANCE', 'CUST-ID', 'CUST-NAME', 'CUSTOMER-RECORD',
'END-OF-FILE', 'FIELD-A', 'FIELD-B',
'LS-AMOUNT', 'LS-CUST-ID', 'LS-PARAM',
'PREMIUM-CUSTOMER', 'REGULAR-CUSTOMER',
'WS-AMOUNT', 'WS-AMT', 'WS-CODE', 'WS-COUNT',
'WS-CUST-ADDR', 'WS-CUST-CODE', 'WS-CUST-TYPE',
'WS-CUSTOMER-DATA', 'WS-CUSTOMER-NAME', 'WS-EOF',
'WS-FILE-STATUS', 'WS-INNER-CODE', 'WS-LOG-MESSAGE',
'WS-MAP-NAME', 'WS-NAME', 'WS-NEXT-PGM', 'WS-OUTER-FLAG',
'WS-PROG-NAME', 'WS-QUEUE-NAME', 'WS-RECORD',
'WS-REPORT-LINE', 'WS-SORT-FILE', 'WS-SQL-CODE', 'WS-TIMESTAMP',
]);
});
it('produces exactly 1 Record node', () => {
expect(getNodesByLabel(result, 'Record')).toEqual(['CUSTOMER-FILE']);
});
it('produces exactly 15 CodeElement nodes', () => {
const nodes = getNodesByLabel(result, 'CodeElement');
expect(nodes.length).toBe(15);
expect(nodes).toEqual([
'CALL WS-PROG-NAME', 'CICS XCTL WS-NEXT-PGM', 'CUSTJOB',
'EXEC CICS HANDLE ABEND', 'EXEC CICS LINK', 'EXEC CICS READ',
'EXEC CICS RETURN', 'EXEC CICS SEND MAP', 'EXEC CICS WRITEQ TS',
'EXEC CICS XCTL', 'EXEC CICS XCTL', 'EXEC SQL SELECT',
'PROD.CUSTOMER.MASTER', 'STEP1', 'STEP2',
]);
});
it('produces exactly 2 Constructor nodes', () => {
expect(getNodesByLabel(result, 'Constructor')).toEqual(['ALTENTRY', 'AUDITLOG-BATCH']);
});
});
// =====================================================================
// CALLS EDGES — exact count + exact sorted pairs per reason
// =====================================================================
describe('CALLS edge completeness', () => {
it('produces exactly 15 CALLS edges with reason cobol-perform', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cobol-perform');
expect(edges.length).toBe(15);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 BUILD-SORT-INPUT',
'FORMAT-REPORT \u2192 FETCH-DATA',
'FORMAT-REPORT \u2192 MAIN-PARAGRAPH',
'FORMAT-REPORT \u2192 SEND-SCREEN',
'FORMAT-REPORT \u2192 WRITE-SORTED',
'INNER-MAIN \u2192 INNER-PROCESS',
'MAIN-PARAGRAPH \u2192 CLEANUP-PARAGRAPH',
'MAIN-PARAGRAPH \u2192 FETCH-DATA',
'MAIN-PARAGRAPH \u2192 FORMAT-REPORT',
'MAIN-PARAGRAPH \u2192 INIT-PARAGRAPH',
'MAIN-PARAGRAPH \u2192 PROCESS-PARAGRAPH',
'MAIN-PARAGRAPH \u2192 SEND-SCREEN',
'MAIN-PARAGRAPH \u2192 WRITE-LOG',
'OUTER-MAIN \u2192 OUTER-PROCESS',
'PROCESS-PARAGRAPH \u2192 READ-CUSTOMER',
]);
});
it('produces exactly 2 CALLS edges with reason cobol-perform-thru', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cobol-perform-thru');
expect(edges.length).toBe(2);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 FORMAT-REPORT',
'PROCESS-PARAGRAPH \u2192 WRITE-CUSTOMER',
]);
});
it('produces exactly 3 CALLS edges with reason cobol-call', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cobol-call');
expect(edges.length).toBe(3);
expect(edgeSet(edges)).toEqual([
'CUSTUPDT \u2192 AUDITLOG',
'OUTER-PROG \u2192 INNER-PROG',
'RPTGEN \u2192 CUSTUPDT',
]);
});
it('produces exactly 4 CALLS edges with reason cobol-goto', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cobol-goto');
expect(edges.length).toBe(4);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 FETCH-DATA',
'FORMAT-REPORT \u2192 FORMAT-REPORT',
'FORMAT-REPORT \u2192 SEND-SCREEN',
'MAIN-PARAGRAPH \u2192 EXIT-PARAGRAPH',
]);
});
it('produces exactly 1 CALLS edge with reason cics-link', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cics-link');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['RPTGEN \u2192 AUDITLOG']);
});
it('produces exactly 1 CALLS edge with reason cics-xctl', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cics-xctl');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['RPTGEN \u2192 CUSTUPDT']);
});
it('produces exactly 1 CALLS edge with reason cics-handle-abend', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cics-handle-abend');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['RPTGEN \u2192 ABEND-HANDLER']);
});
it('produces exactly 1 CALLS edge with reason cics-return-transid', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'cics-return-transid');
expect(edges.length).toBe(1);
});
it('produces exactly 2 CALLS edges with reason jcl-exec-pgm', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'jcl-exec-pgm');
expect(edges.length).toBe(2);
expect(edgeSet(edges)).toEqual(['STEP1 \u2192 CUSTUPDT', 'STEP2 \u2192 RPTGEN']);
});
it('produces exactly 1 CALLS edge with reason jcl-dd:CUSTFILE', () => {
const edges = getRelationships(result, 'CALLS').filter(e => e.rel.reason === 'jcl-dd:CUSTFILE');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['STEP1 \u2192 PROD.CUSTOMER.MASTER']);
});
it('produces zero unresolved CALLS edges', () => {
expect(getRelationships(result, 'CALLS').filter(e => e.rel.reason.endsWith('-unresolved')).length).toBe(0);
});
});
// =====================================================================
// CONTAINS EDGES — exact count + exact sorted pairs per reason
// =====================================================================
describe('CONTAINS edge completeness', () => {
it('produces exactly 4 CONTAINS edges with reason cobol-program-id', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-program-id');
expect(edges.length).toBe(4);
expect(edgeSet(edges)).toEqual([
'AUDITLOG.cbl \u2192 AUDITLOG',
'CUSTUPDT.cbl \u2192 CUSTUPDT',
'NESTED.cbl \u2192 OUTER-PROG',
'RPTGEN.cbl \u2192 RPTGEN',
]);
});
it('produces exactly 1 CONTAINS edge with reason cobol-nested-program', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-nested-program');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['OUTER-PROG \u2192 INNER-PROG']);
});
it('produces exactly 2 CONTAINS edges with reason cobol-section', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-section');
expect(edges.length).toBe(2);
expect(edgeSet(edges)).toEqual([
'CUSTUPDT \u2192 INIT-SECTION',
'CUSTUPDT \u2192 PROCESSING-SECTION',
]);
});
it('produces exactly 21 CONTAINS edges with reason cobol-paragraph', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-paragraph');
expect(edges.length).toBe(21);
expect(edgeSet(edges)).toEqual([
'AUDITLOG \u2192 MAIN-PARAGRAPH',
'AUDITLOG \u2192 WRITE-LOG',
'INIT-SECTION \u2192 INIT-PARAGRAPH',
'INIT-SECTION \u2192 MAIN-PARAGRAPH',
'INNER-PROG \u2192 INNER-MAIN',
'INNER-PROG \u2192 INNER-PROCESS',
'OUTER-PROG \u2192 OUTER-MAIN',
'OUTER-PROG \u2192 OUTER-PROCESS',
'PROCESSING-SECTION \u2192 CLEANUP-PARAGRAPH',
'PROCESSING-SECTION \u2192 PROCESS-PARAGRAPH',
'PROCESSING-SECTION \u2192 READ-CUSTOMER',
'PROCESSING-SECTION \u2192 UPDATE-BALANCE',
'PROCESSING-SECTION \u2192 WRITE-CUSTOMER',
'RPTGEN \u2192 ABEND-HANDLER',
'RPTGEN \u2192 BUILD-SORT-INPUT',
'RPTGEN \u2192 EXIT-PARAGRAPH',
'RPTGEN \u2192 FETCH-DATA',
'RPTGEN \u2192 FORMAT-REPORT',
'RPTGEN \u2192 MAIN-PARAGRAPH',
'RPTGEN \u2192 SEND-SCREEN',
'RPTGEN \u2192 WRITE-SORTED',
]);
});
it('produces exactly 36 CONTAINS edges with reason cobol-data-item', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-data-item');
expect(edges.length).toBe(36);
expect(edgeSet(edges)).toEqual([
'AUDITLOG \u2192 LS-AMOUNT',
'AUDITLOG \u2192 LS-CUST-ID',
'AUDITLOG \u2192 WS-LOG-MESSAGE',
'AUDITLOG \u2192 WS-TIMESTAMP',
'CUSTUPDT \u2192 CUST-BALANCE',
'CUSTUPDT \u2192 CUST-ID',
'CUSTUPDT \u2192 CUST-NAME',
'CUSTUPDT \u2192 CUSTOMER-RECORD',
'CUSTUPDT \u2192 END-OF-FILE',
'CUSTUPDT \u2192 FIELD-A',
'CUSTUPDT \u2192 FIELD-B',
'CUSTUPDT \u2192 LS-PARAM',
'CUSTUPDT \u2192 WS-AMOUNT',
'CUSTUPDT \u2192 WS-AMT',
'CUSTUPDT \u2192 WS-CODE',
'CUSTUPDT \u2192 WS-CUSTOMER-NAME',
'CUSTUPDT \u2192 WS-EOF',
'CUSTUPDT \u2192 WS-FILE-STATUS',
'CUSTUPDT \u2192 WS-NAME',
'CUSTUPDT \u2192 WS-PROG-NAME',
'CUSTUPDT \u2192 WS-RECORD',
'INNER-PROG \u2192 WS-INNER-CODE',
'OUTER-PROG \u2192 WS-OUTER-FLAG',
'RPTGEN \u2192 PREMIUM-CUSTOMER',
'RPTGEN \u2192 REGULAR-CUSTOMER',
'RPTGEN \u2192 WS-COUNT',
'RPTGEN \u2192 WS-CUST-ADDR',
'RPTGEN \u2192 WS-CUST-CODE',
'RPTGEN \u2192 WS-CUST-TYPE',
'RPTGEN \u2192 WS-CUSTOMER-DATA',
'RPTGEN \u2192 WS-MAP-NAME',
'RPTGEN \u2192 WS-NEXT-PGM',
'RPTGEN \u2192 WS-QUEUE-NAME',
'RPTGEN \u2192 WS-REPORT-LINE',
'RPTGEN \u2192 WS-SORT-FILE',
'RPTGEN \u2192 WS-SQL-CODE',
]);
});
it('produces exactly 8 CONTAINS edges with reason cobol-exec-cics', () => {
const edges = getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-exec-cics');
expect(edges.length).toBe(8);
expect(edgeSet(edges)).toEqual([
'RPTGEN \u2192 EXEC CICS HANDLE ABEND',
'RPTGEN \u2192 EXEC CICS LINK',
'RPTGEN \u2192 EXEC CICS READ',
'RPTGEN \u2192 EXEC CICS RETURN',
'RPTGEN \u2192 EXEC CICS SEND MAP',
'RPTGEN \u2192 EXEC CICS WRITEQ TS',
'RPTGEN \u2192 EXEC CICS XCTL',
'RPTGEN \u2192 EXEC CICS XCTL',
]);
});
it('produces exactly 1 CONTAINS edge with reason cobol-exec-sql', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-exec-sql')))
.toEqual(['RPTGEN \u2192 EXEC SQL SELECT']);
});
it('produces exactly 1 CONTAINS edge with reason cics-dynamic-program', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cics-dynamic-program')))
.toEqual(['RPTGEN \u2192 CICS XCTL WS-NEXT-PGM']);
});
it('produces exactly 1 CONTAINS edge with reason cobol-dynamic-call', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-dynamic-call')))
.toEqual(['CUSTUPDT \u2192 CALL WS-PROG-NAME']);
});
it('produces exactly 2 CONTAINS edges with reason cobol-entry-point', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-entry-point')))
.toEqual(['AUDITLOG \u2192 AUDITLOG-BATCH', 'CUSTUPDT \u2192 ALTENTRY']);
});
it('produces exactly 1 CONTAINS edge with reason cobol-file-declaration', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'cobol-file-declaration')))
.toEqual(['CUSTUPDT \u2192 CUSTOMER-FILE']);
});
it('produces exactly 1 CONTAINS edge with reason jcl-job', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'jcl-job')))
.toEqual(['RUNJOBS.jcl \u2192 CUSTJOB']);
});
it('produces exactly 2 CONTAINS edges with reason jcl-step', () => {
expect(edgeSet(getRelationships(result, 'CONTAINS').filter(e => e.rel.reason === 'jcl-step')))
.toEqual(['CUSTJOB \u2192 STEP1', 'CUSTJOB \u2192 STEP2']);
});
});
// =====================================================================
// ACCESSES EDGES — exact count + exact sorted pairs per reason
// =====================================================================
describe('ACCESSES edge completeness', () => {
it('produces exactly 4 ACCESSES edges with reason cobol-move-read', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cobol-move-read');
expect(edges.length).toBe(4);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 WS-CUST-CODE',
'READ-CUSTOMER \u2192 CUST-NAME',
'UPDATE-BALANCE \u2192 WS-AMOUNT',
'UPDATE-BALANCE \u2192 WS-AMT',
]);
});
it('produces exactly 5 ACCESSES edges with reason cobol-move-write', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cobol-move-write');
expect(edges.length).toBe(5);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 WS-REPORT-LINE',
'READ-CUSTOMER \u2192 WS-CUSTOMER-NAME',
'UPDATE-BALANCE \u2192 CUST-BALANCE',
'UPDATE-BALANCE \u2192 FIELD-A',
'UPDATE-BALANCE \u2192 FIELD-B',
]);
});
it('produces exactly 1 ACCESSES edge with reason cics-file-read', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cics-file-read').length).toBe(1);
});
it('produces exactly 1 ACCESSES edge with reason cics-map', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cics-map').length).toBe(1);
});
it('produces exactly 1 ACCESSES edge with reason cics-queue-write', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cics-queue-write').length).toBe(1);
});
it('produces exactly 1 ACCESSES edge with reason cics-receive-into', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cics-receive-into');
expect(edges.length).toBe(1);
expect(edges[0].target).toBe('WS-CUSTOMER-DATA');
});
it('produces exactly 2 ACCESSES edges with reason cics-send-from', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cics-send-from');
expect(edges.length).toBe(2);
expect(edgeSet(edges)).toEqual([
'EXEC CICS SEND MAP \u2192 WS-REPORT-LINE',
'EXEC CICS WRITEQ TS \u2192 WS-REPORT-LINE',
]);
});
it('produces exactly 1 ACCESSES edge with reason cobol-search', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cobol-search');
expect(edges.length).toBe(1);
expect(edgeSet(edges)).toEqual(['RPTGEN \u2192 WS-CUSTOMER-DATA']);
});
it('produces exactly 1 ACCESSES edge with reason sort-using', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'sort-using').length).toBe(1);
});
it('produces exactly 1 ACCESSES edge with reason sort-giving (multi-line SORT)', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'sort-giving').length).toBe(1);
});
it('produces exactly 2 ACCESSES edges with reason cobol-procedure-using', () => {
const edges = getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'cobol-procedure-using');
expect(edges.length).toBe(2);
expect(edgeSet(edges)).toEqual([
'AUDITLOG \u2192 LS-AMOUNT',
'AUDITLOG \u2192 LS-CUST-ID',
]);
});
it('produces exactly 1 ACCESSES edge with reason sql-select', () => {
expect(getRelationships(result, 'ACCESSES').filter(e => e.rel.reason === 'sql-select').length).toBe(1);
});
});
// =====================================================================
// IMPORTS EDGES — exact pairs
// =====================================================================
describe('IMPORTS edge completeness', () => {
it('produces exactly 2 IMPORTS edges with reason cobol-copy', () => {
const edges = getRelationships(result, 'IMPORTS').filter(e => e.rel.reason === 'cobol-copy');
expect(edges.length).toBe(2);
});
});
// =====================================================================
// FEATURE-SPECIFIC ASSERTIONS — validates all review findings resolved
// =====================================================================
describe('multi-PERFORM on same line (Finding #III)', () => {
it('captures both PERFORMs in IF/ELSE on a single logical line', () => {
// IF WS-COUNT > 0 PERFORM FETCH-DATA ELSE PERFORM SEND-SCREEN
const edges = getRelationships(result, 'CALLS').filter(
e => e.rel.reason === 'cobol-perform' && e.source === 'FORMAT-REPORT',
);
const targets = edges.map(e => e.target).sort();
expect(targets).toContain('FETCH-DATA');
expect(targets).toContain('SEND-SCREEN');
});
});
describe('INPUT/OUTPUT PROCEDURE IS in SORT (Finding #iii)', () => {
it('creates CALLS edges for INPUT PROCEDURE and OUTPUT PROCEDURE targets', () => {
const edges = getRelationships(result, 'CALLS').filter(
e => e.rel.reason === 'cobol-perform' && e.source === 'FORMAT-REPORT',
);
const targets = edges.map(e => e.target).sort();
expect(targets).toContain('BUILD-SORT-INPUT');
expect(targets).toContain('WRITE-SORTED');
});
it('creates paragraph nodes for INPUT/OUTPUT PROCEDURE targets', () => {
const nodes = getNodesByLabel(result, 'Function');
expect(nodes).toContain('BUILD-SORT-INPUT');
expect(nodes).toContain('WRITE-SORTED');
});
});
describe('GO TO DEPENDING ON multi-target (Finding #iv)', () => {
it('captures all three targets from GO TO ... DEPENDING ON', () => {
// GO TO FETCH-DATA FORMAT-REPORT SEND-SCREEN DEPENDING ON WS-COUNT
const edges = getRelationships(result, 'CALLS').filter(
e => e.rel.reason === 'cobol-goto' && e.source === 'FORMAT-REPORT',
);
expect(edges.length).toBe(3);
expect(edgeSet(edges)).toEqual([
'FORMAT-REPORT \u2192 FETCH-DATA',
'FORMAT-REPORT \u2192 FORMAT-REPORT',
'FORMAT-REPORT \u2192 SEND-SCREEN',
]);
});
});
describe('MOVE CORR abbreviation (Finding #IV)', () => {
it('produces ACCESSES edges for MOVE CORR with corresponding reason', () => {
const readEdges = getRelationships(result, 'ACCESSES').filter(
e => e.rel.reason === 'cobol-move-corresponding-read',
);
expect(readEdges.length).toBe(1);
expect(edgeSet(readEdges)).toEqual(['FORMAT-REPORT \u2192 WS-CUSTOMER-DATA']);
const writeEdges = getRelationships(result, 'ACCESSES').filter(
e => e.rel.reason === 'cobol-move-corresponding-write',
);
expect(writeEdges.length).toBe(1);
expect(edgeSet(writeEdges)).toEqual(['FORMAT-REPORT \u2192 WS-REPORT-LINE']);
});
});
describe('nested program CONTAINS attribution (Finding #I, #II)', () => {
it('attributes INNER-PROG paragraphs to INNER-PROG, not OUTER-PROG', () => {
const edges = getRelationships(result, 'CONTAINS').filter(
e => e.rel.reason === 'cobol-paragraph' && e.target === 'INNER-MAIN',
);
expect(edges.length).toBe(1);
expect(edges[0].source).toBe('INNER-PROG');
});
it('attributes INNER-PROG data items to INNER-PROG, not OUTER-PROG', () => {
const edges = getRelationships(result, 'CONTAINS').filter(
e => e.rel.reason === 'cobol-data-item' && e.target === 'WS-INNER-CODE',
);
expect(edges.length).toBe(1);
expect(edges[0].source).toBe('INNER-PROG');
});
it('attributes OUTER-PROG data items to OUTER-PROG', () => {
const edges = getRelationships(result, 'CONTAINS').filter(
e => e.rel.reason === 'cobol-data-item' && e.target === 'WS-OUTER-FLAG',
);
expect(edges.length).toBe(1);
expect(edges[0].source).toBe('OUTER-PROG');
});
});
describe('per-program PROCEDURE DIVISION USING (Finding #III partial)', () => {
it('creates ACCESSES edges from AUDITLOG, not from wrong program', () => {
const edges = getRelationships(result, 'ACCESSES').filter(
e => e.rel.reason === 'cobol-procedure-using',
);
expect(edges.length).toBe(2);
// Both edges should source from AUDITLOG (the program that declares USING)
for (const e of edges) {
expect(e.source).toBe('AUDITLOG');
}
});
});
describe('PERFORM THRU edge correctness', () => {
it('captures FORMAT-REPORT PERFORM THRU from MAIN-PARAGRAPH', () => {
const edges = getRelationships(result, 'CALLS').filter(
e => e.rel.reason === 'cobol-perform-thru',
);
expect(edgeSet(edges)).toContain('FORMAT-REPORT \u2192 FORMAT-REPORT');
});
});
describe('nested program CALLS attribution', () => {
it('attributes INNER-PROG PERFORM edges to INNER-PROG paragraphs', () => {
const edges = getRelationships(result, 'CALLS').filter(
e => e.rel.reason === 'cobol-perform' && e.source === 'INNER-MAIN',
);
expect(edges.length).toBe(1);
expect(edges[0].target).toBe('INNER-PROCESS');
});
});
// =====================================================================
// GRAND TOTALS — catch any unexpected edge leakage
// =====================================================================
describe('grand totals', () => {
it('produces exactly 31 total CALLS edges', () => {
// 15 perform + 2 perform-thru + 3 call + 4 goto + 1 link + 1 xctl
// + 1 handle-abend + 1 return-transid + 2 jcl-exec-pgm + 1 jcl-dd
expect(getRelationships(result, 'CALLS').length).toBe(31);
});
it('produces exactly 81 total CONTAINS edges', () => {
// 4 program-id + 1 nested-program + 2 section + 21 paragraph
// + 36 data-item + 8 exec-cics + 1 exec-sql + 1 dynamic-call
// + 1 cics-dynamic-program + 2 entry-point + 1 file-declaration
// + 1 jcl-job + 2 jcl-step
expect(getRelationships(result, 'CONTAINS').length).toBe(81);
});
it('produces exactly 2 total IMPORTS edges', () => {
expect(getRelationships(result, 'IMPORTS').length).toBe(2);
});
it('produces exactly 25 total ACCESSES edges', () => {
// 4 move-read + 5 move-write + 1 move-corresponding-read + 1 move-corresponding-write
// + 1 file-read + 1 map + 1 queue-write
// + 1 receive-into + 2 send-from + 1 search + 1 sort-using + 1 sort-giving
// + 2 procedure-using + 1 sql-select + 2 call-using
expect(getRelationships(result, 'ACCESSES').length).toBe(25);
});
});
});

69
gitnexus/test/unit/cobol-copy-expander.test.ts Normal file
View file

@ -0,0 +1,69 @@
/**
* Unit Tests: COBOL Copy Expander — pseudotext REPLACING support
*/
import { describe, it, expect } from 'vitest';
import { parseReplacingClause } from '../../src/core/ingestion/cobol/cobol-copy-expander.js';
describe('parseReplacingClause', () => {
// Existing quoted-string behavior preserved
it('parses quoted EXACT replacement', () => {
const result = parseReplacingClause(' "OLD-NAME" BY "NEW-NAME" ');
expect(result).toEqual([{ type: 'EXACT', from: 'OLD-NAME', to: 'NEW-NAME' }]);
});
it('parses LEADING replacement', () => {
const result = parseReplacingClause(' LEADING "ESP-" BY "LK-ESP-" ');
expect(result).toEqual([{ type: 'LEADING', from: 'ESP-', to: 'LK-ESP-' }]);
});
it('parses TRAILING replacement', () => {
const result = parseReplacingClause(' TRAILING "-IN" BY "-OUT" ');
expect(result).toEqual([{ type: 'TRAILING', from: '-IN', to: '-OUT' }]);
});
// Pseudotext ==...== support (isPseudotext flag propagated)
it('parses basic pseudotext: ==OLD== BY ==NEW==', () => {
const result = parseReplacingClause(' ==WS-OLD== BY ==WS-NEW== ');
expect(result).toEqual([{ type: 'EXACT', from: 'WS-OLD', to: 'WS-NEW', isPseudotext: true }]);
});
it('parses empty pseudotext (deletion): ==TEXT== BY ====', () => {
const result = parseReplacingClause(' ==REMOVE-ME== BY ==== ');
expect(result).toEqual([{ type: 'EXACT', from: 'REMOVE-ME', to: '', isPseudotext: true }]);
});
it('parses pseudotext with spaces: ==SOME TEXT== BY ==OTHER TEXT==', () => {
const result = parseReplacingClause(' ==WORKING STORAGE== BY ==LOCAL STORAGE== ');
expect(result).toEqual([{ type: 'EXACT', from: 'WORKING STORAGE', to: 'LOCAL STORAGE', isPseudotext: true }]);
});
it('parses pseudotext with single = inside: ==A=B== BY ==C=D==', () => {
const result = parseReplacingClause(' ==A=B== BY ==C=D== ');
expect(result).toEqual([{ type: 'EXACT', from: 'A=B', to: 'C=D', isPseudotext: true }]);
});
it('parses mixed quoted + pseudotext in one clause', () => {
const result = parseReplacingClause(
' "OLD-NAME" BY "NEW-NAME" ==DEL-PREFIX== BY ==== ',
);
expect(result).toEqual([
{ type: 'EXACT', from: 'OLD-NAME', to: 'NEW-NAME' },
{ type: 'EXACT', from: 'DEL-PREFIX', to: '', isPseudotext: true },
]);
});
it('LEADING modifier works alongside pseudotext', () => {
const result = parseReplacingClause(
' LEADING "ESP-" BY "LK-ESP-" ==OLD-EXACT== BY ==NEW-EXACT== ',
);
expect(result).toEqual([
{ type: 'LEADING', from: 'ESP-', to: 'LK-ESP-' },
{ type: 'EXACT', from: 'OLD-EXACT', to: 'NEW-EXACT', isPseudotext: true },
]);
});
it('returns empty array for empty input', () => {
expect(parseReplacingClause('')).toEqual([]);
expect(parseReplacingClause(' ')).toEqual([]);
});
});

2746
gitnexus/test/unit/cobol-preprocessor.test.ts Normal file
View file

File diff suppressed because it is too large Load diff

57
gitnexus/test/unit/graph.test.ts
View file

@ -186,4 +186,61 @@ describe('createKnowledgeGraph', () => {
g.forEachRelationship(r => types.push(r.type));
expect(types).toEqual(['CALLS']);
});
// ─── removeRelationship ─────────────────────────────────────────────
it('removes a relationship by id', () => {
const g = createKnowledgeGraph();
g.addNode(makeNode('fn:a', 'a'));
g.addNode(makeNode('fn:b', 'b'));
g.addRelationship(makeRel('fn:a', 'fn:b'));
expect(g.relationshipCount).toBe(1);
const removed = g.removeRelationship('fn:a-CALLS-fn:b');
expect(removed).toBe(true);
expect(g.relationshipCount).toBe(0);
});
it('removeRelationship returns false for unknown id', () => {
const g = createKnowledgeGraph();
expect(g.removeRelationship('nonexistent')).toBe(false);
});
it('removeRelationship returns false on second call with same id', () => {
const g = createKnowledgeGraph();
g.addNode(makeNode('fn:a', 'a'));
g.addNode(makeNode('fn:b', 'b'));
g.addRelationship(makeRel('fn:a', 'fn:b'));
expect(g.removeRelationship('fn:a-CALLS-fn:b')).toBe(true);
expect(g.removeRelationship('fn:a-CALLS-fn:b')).toBe(false);
});
it('removeRelationship does not affect nodes', () => {
const g = createKnowledgeGraph();
g.addNode(makeNode('fn:a', 'a'));
g.addNode(makeNode('fn:b', 'b'));
g.addRelationship(makeRel('fn:a', 'fn:b'));
g.removeRelationship('fn:a-CALLS-fn:b');
expect(g.nodeCount).toBe(2);
expect(g.getNode('fn:a')).toBeDefined();
expect(g.getNode('fn:b')).toBeDefined();
});
it('removeRelationship leaves other relationships intact', () => {
const g = createKnowledgeGraph();
g.addNode(makeNode('fn:a', 'a'));
g.addNode(makeNode('fn:b', 'b'));
g.addNode(makeNode('fn:c', 'c'));
g.addRelationship(makeRel('fn:a', 'fn:b'));
g.addRelationship(makeRel('fn:b', 'fn:c'));
expect(g.relationshipCount).toBe(2);
g.removeRelationship('fn:a-CALLS-fn:b');
expect(g.relationshipCount).toBe(1);
const remaining = [...g.iterRelationships()];
expect(remaining[0].sourceId).toBe('fn:b');
expect(remaining[0].targetId).toBe('fn:c');
});
});

338
gitnexus/test/unit/jcl-parser.test.ts Normal file
View file

@ -0,0 +1,338 @@
import { describe, it, expect } from 'vitest';
import { parseJcl } from '../../src/core/ingestion/cobol/jcl-parser.js';
import type { JclParseResults } from '../../src/core/ingestion/cobol/jcl-parser.js';
describe('parseJcl', () => {
// ── JOB statements ──────────────────────────────────────────────────
describe('JOB statements', () => {
it('extracts job name', () => {
const jcl = `//MYJOB JOB (ACCT),'MY JOB'`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.jobs).toHaveLength(1);
expect(r.jobs[0].name).toBe('MYJOB');
expect(r.jobs[0].line).toBe(1);
});
it('extracts CLASS and MSGCLASS parameters', () => {
const jcl = `//PAYJOB JOB (ACCT),'PAYROLL',CLASS=A,MSGCLASS=X`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.jobs).toHaveLength(1);
expect(r.jobs[0].name).toBe('PAYJOB');
expect(r.jobs[0].class).toBe('A');
expect(r.jobs[0].msgclass).toBe('X');
});
it('handles job with no CLASS or MSGCLASS', () => {
const jcl = `//BAREJOB JOB (ACCT),'BARE'`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.jobs).toHaveLength(1);
expect(r.jobs[0].name).toBe('BAREJOB');
expect(r.jobs[0].class).toBeUndefined();
expect(r.jobs[0].msgclass).toBeUndefined();
});
});
// ── EXEC statements ─────────────────────────────────────────────────
describe('EXEC statements', () => {
it('extracts step with PGM=program', () => {
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC PGM=IEFBR14',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.steps).toHaveLength(1);
expect(r.steps[0].name).toBe('STEP1');
expect(r.steps[0].program).toBe('IEFBR14');
expect(r.steps[0].proc).toBeUndefined();
});
it('extracts step with proc name (no PGM= keyword)', () => {
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC MYPROC',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.steps).toHaveLength(1);
expect(r.steps[0].name).toBe('STEP1');
expect(r.steps[0].program).toBeUndefined();
expect(r.steps[0].proc).toBe('MYPROC');
});
it('associates step with current job', () => {
const jcl = [
'//JOB1 JOB (ACCT)',
'//STEPA EXEC PGM=PROG1',
'//JOB2 JOB (ACCT)',
'//STEPB EXEC PGM=PROG2',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.steps).toHaveLength(2);
expect(r.steps[0].jobName).toBe('JOB1');
expect(r.steps[1].jobName).toBe('JOB2');
});
});
// ── DD statements ───────────────────────────────────────────────────
describe('DD statements', () => {
it('extracts DD name and dataset (DSN=)', () => {
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC PGM=IEFBR14',
'//INPUT DD DSN=MY.DATA.SET,DISP=SHR',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.ddStatements).toHaveLength(1);
expect(r.ddStatements[0].ddName).toBe('INPUT');
expect(r.ddStatements[0].dataset).toBe('MY.DATA.SET');
});
it('extracts DISP parameter', () => {
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC PGM=IEFBR14',
'//OUTPUT DD DSN=MY.OUT,DISP=(NEW,CATLG,DELETE)',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.ddStatements).toHaveLength(1);
expect(r.ddStatements[0].disp).toBe('NEW');
});
it('associates DD with current step', () => {
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC PGM=PROG1',
'//DD1 DD DSN=DS1,DISP=SHR',
'//STEP2 EXEC PGM=PROG2',
'//DD2 DD DSN=DS2,DISP=SHR',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.ddStatements).toHaveLength(2);
expect(r.ddStatements[0].stepName).toBe('STEP1');
expect(r.ddStatements[1].stepName).toBe('STEP2');
});
});
// ── PROC definitions ────────────────────────────────────────────────
describe('PROC definitions', () => {
it('extracts in-stream PROC with name', () => {
const jcl = [
'//MYPROC PROC',
'//STEP1 EXEC PGM=IEFBR14',
'// PEND',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.procs).toHaveLength(1);
expect(r.procs[0].name).toBe('MYPROC');
expect(r.procs[0].isInStream).toBe(true);
});
it('handles PROC/PEND pairs', () => {
const jcl = [
'//PROC1 PROC',
'//S1 EXEC PGM=PROG1',
'// PEND',
'//PROC2 PROC',
'//S2 EXEC PGM=PROG2',
'// PEND',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.procs).toHaveLength(2);
expect(r.procs[0].name).toBe('PROC1');
expect(r.procs[1].name).toBe('PROC2');
});
});
// ── INCLUDE / SET ───────────────────────────────────────────────────
describe('INCLUDE and SET', () => {
it('extracts INCLUDE MEMBER=name', () => {
const jcl = `// INCLUDE MEMBER=MYINCL`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.includes).toHaveLength(1);
expect(r.includes[0].member).toBe('MYINCL');
expect(r.includes[0].line).toBe(1);
});
it('extracts SET variable=value', () => {
const jcl = `// SET ENV=PROD`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.sets).toHaveLength(1);
expect(r.sets[0].variable).toBe('ENV');
expect(r.sets[0].value).toBe('PROD');
});
});
// ── Conditionals ────────────────────────────────────────────────────
describe('Conditionals', () => {
it('extracts IF condition THEN', () => {
const jcl = `// IF STEP1.RC = 0 THEN`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.conditionals).toHaveLength(1);
expect(r.conditionals[0].type).toBe('IF');
expect(r.conditionals[0].condition).toBe('STEP1.RC = 0');
});
it('extracts ELSE and ENDIF', () => {
const jcl = [
'// IF STEP1.RC = 0 THEN',
'//GOOD EXEC PGM=GOODPGM',
'// ELSE',
'//BAD EXEC PGM=BADPGM',
'// ENDIF',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.conditionals).toHaveLength(3);
expect(r.conditionals[0].type).toBe('IF');
expect(r.conditionals[1].type).toBe('ELSE');
expect(r.conditionals[1].condition).toBeUndefined();
expect(r.conditionals[2].type).toBe('ENDIF');
expect(r.conditionals[2].condition).toBeUndefined();
});
});
// ── JCLLIB ──────────────────────────────────────────────────────────
describe('JCLLIB', () => {
it('extracts JCLLIB ORDER=(lib1,lib2)', () => {
const jcl = `// JCLLIB ORDER=(SYS1.PROCLIB,USER.PROCLIB)`;
const r = parseJcl(jcl, 'test.jcl');
expect(r.jcllib).toHaveLength(1);
expect(r.jcllib[0].order).toEqual(['SYS1.PROCLIB', 'USER.PROCLIB']);
expect(r.jcllib[0].line).toBe(1);
});
});
// ── Continuation lines ──────────────────────────────────────────────
describe('Continuation lines', () => {
it('joins continuation lines (col 72 non-blank + next line starts with //)', () => {
// Build a DD line that is exactly 72 chars with non-blank at col 72 (index 71).
// The continuation line provides the DISP parameter.
// "//DD1 DD DSN=MY.VERY.LONG.DATASET.NAME.THAT.KEEPS.GOING," is 60 chars.
// Pad to 71 then add non-blank at col 72.
const base = '//DD1 DD DSN=MY.VERY.LONG.DATASET.NAME.THAT.KEEPS.GOING,';
const padding = ' '.repeat(71 - base.length);
const line1 = base + padding + 'X'; // col 72 is 'X' (non-blank) -> continuation
const line2 = '// DISP=SHR';
const jcl = [
'//MYJOB JOB (ACCT)',
'//STEP1 EXEC PGM=IEFBR14',
line1,
line2,
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
// The continuation should join the DD line so both DSN and DISP are parsed
expect(r.ddStatements).toHaveLength(1);
expect(r.ddStatements[0].ddName).toBe('DD1');
expect(r.ddStatements[0].dataset).toBe('MY.VERY.LONG.DATASET.NAME.THAT.KEEPS.GOING');
expect(r.ddStatements[0].disp).toBe('SHR');
});
});
// ── Edge cases ──────────────────────────────────────────────────────
describe('Edge cases', () => {
it('skips JCL comments (//*)', () => {
const jcl = [
'//* This is a comment',
'//MYJOB JOB (ACCT)',
'//* Another comment',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.jobs).toHaveLength(1);
expect(r.jobs[0].name).toBe('MYJOB');
});
it('skips non-JCL lines', () => {
const jcl = [
'This is not a JCL line',
'//MYJOB JOB (ACCT)',
' Some data',
'//STEP1 EXEC PGM=IEFBR14',
].join('\n');
const r = parseJcl(jcl, 'test.jcl');
expect(r.jobs).toHaveLength(1);
expect(r.steps).toHaveLength(1);
});
it('empty input returns empty results', () => {
const r = parseJcl('', 'test.jcl');
expect(r.jobs).toEqual([]);
expect(r.steps).toEqual([]);
expect(r.ddStatements).toEqual([]);
expect(r.procs).toEqual([]);
expect(r.includes).toEqual([]);
expect(r.sets).toEqual([]);
expect(r.jcllib).toEqual([]);
expect(r.conditionals).toEqual([]);
});
it('complete JCL job with multiple steps and DDs', () => {
const jcl = [
'//* Complete payroll job',
'//PAYJOB JOB (ACCT123),\'PAYROLL RUN\',CLASS=A,MSGCLASS=X',
'// JCLLIB ORDER=(PAY.PROCLIB,SYS1.PROCLIB)',
'// SET ENV=PROD',
'// INCLUDE MEMBER=STDPARMS',
'//*',
'// IF 1 = 1 THEN',
'//STEP01 EXEC PGM=PAYEXT',
'//INPUT DD DSN=PAY.MASTER,DISP=SHR',
'//OUTPUT DD DSN=PAY.EXTRACT,DISP=(NEW,CATLG,DELETE)',
'//SYSPRINT DD SYSOUT=*',
'//*',
'//STEP02 EXEC PAYCALC',
'//INFILE DD DSN=PAY.EXTRACT,DISP=SHR',
'// ELSE',
'//STEP03 EXEC PGM=IEFBR14',
'// ENDIF',
].join('\n');
const r = parseJcl(jcl, 'payroll.jcl');
// Jobs
expect(r.jobs).toHaveLength(1);
expect(r.jobs[0]).toEqual({
name: 'PAYJOB',
line: 2,
class: 'A',
msgclass: 'X',
});
// JCLLIB
expect(r.jcllib).toHaveLength(1);
expect(r.jcllib[0].order).toEqual(['PAY.PROCLIB', 'SYS1.PROCLIB']);
// SET
expect(r.sets).toHaveLength(1);
expect(r.sets[0]).toEqual({ variable: 'ENV', value: 'PROD', line: 4 });
// INCLUDE
expect(r.includes).toHaveLength(1);
expect(r.includes[0].member).toBe('STDPARMS');
// Conditionals
expect(r.conditionals).toHaveLength(3);
expect(r.conditionals[0].type).toBe('IF');
expect(r.conditionals[1].type).toBe('ELSE');
expect(r.conditionals[2].type).toBe('ENDIF');
// Steps
expect(r.steps).toHaveLength(3);
expect(r.steps[0]).toMatchObject({ name: 'STEP01', program: 'PAYEXT', jobName: 'PAYJOB' });
expect(r.steps[1]).toMatchObject({ name: 'STEP02', proc: 'PAYCALC', jobName: 'PAYJOB' });
expect(r.steps[2]).toMatchObject({ name: 'STEP03', program: 'IEFBR14', jobName: 'PAYJOB' });
// DD statements
expect(r.ddStatements).toHaveLength(4);
expect(r.ddStatements[0]).toMatchObject({ ddName: 'INPUT', stepName: 'STEP01', dataset: 'PAY.MASTER', disp: 'SHR' });
expect(r.ddStatements[1]).toMatchObject({ ddName: 'OUTPUT', stepName: 'STEP01', disp: 'NEW' });
expect(r.ddStatements[2]).toMatchObject({ ddName: 'SYSPRINT', stepName: 'STEP01' });
expect(r.ddStatements[3]).toMatchObject({ ddName: 'INFILE', stepName: 'STEP02', dataset: 'PAY.EXTRACT' });
});
});
});