GitNexus/gitnexus-shared
Copilot ed5a4220dd
feat(ingestion): language-agnostic variable extractor with config+factory pattern (#878)
* Initial plan

* feat(ingestion): add variable extraction types, factory, configs, and wire into language providers

- Create variable-types.ts with VariableInfo, VariableExtractionConfig, VariableExtractor interfaces
- Create variable-extractors/generic.ts with createVariableExtractor() factory
- Add variableExtractor field to LanguageProvider interface
- Create per-language variable extraction configs for all 16 languages
- Wire variableExtractor into all language providers
- Add variable metadata enrichment to parse-worker for Const/Static/Variable labels

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* feat(ingestion): add variable extraction tests and fix Python/TS config issues

- Create test/unit/variable-extraction.test.ts with 29 tests covering
  TypeScript, JavaScript, Python, Go, Rust, C, C++, Ruby, and factory behavior
- Fix isConst in generic factory to use config.isConst over node-type membership
  (TS let/const both use lexical_declaration)
- Fix Python type extraction for annotated assignments at module scope
- Fix Python dunder name visibility (e.g., __name__ is public, not protected)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address code review feedback — move imports, clarify scope comment, use shared test context

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address review comments, fix prettier formatting and lint errors

- Fix prettier formatting in 5 files (c-cpp, jvm, swift configs, test file)
- Remove unused SyntaxNode imports in php.ts and ruby.ts (lint errors)
- Remove unused constNodeSet/variableNodeSet variables in generic.ts (warnings)
- Remove semantically wrong `methodProps.isReadonly = varInfo.isConst` (review)
- Remove dead `nodeLabel === 'Variable'` guard in parse-worker (review)
- Fix test guard: replace `if (declNode)` with `expect(declNode).toBeDefined()` (review)
- Add comment about Python expression_statement broadness (review)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/040edbbf-65b5-40e1-80c8-e98f7c4bb54a

* feat(ingestion): add block-scoped variable extraction via tree-sitter queries

Add @definition.const and @definition.variable tree-sitter query patterns
for TypeScript, JavaScript, Python, Go, Java, C, C++, C#, PHP, Ruby, and
Dart. Add parse-worker dedup logic to avoid duplicate nodes when variable
captures overlap with existing function/property captures. Add 'Variable'
label support in getLabelFromCaptures and DEFINITION_CAPTURE_KEYS.

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* test: add block-scoped variable extraction tests and query capture tests

Add 6 tests for block-scoped variable extraction (TypeScript, Go, Rust, C,
Python). Add 14 tests verifying @definition.const/@definition.variable
query patterns exist in all language query strings. Import RUBY_QUERIES
in test file.

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* test: add Python non-assignment expression statement rejection test

Addresses code review feedback: verify that the Python variable extractor
returns null for expression_statement nodes that contain function calls
rather than assignments (e.g. `print("hello")`).

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: Dart query node type, add Variable schema, update schema counts

- Change `top_level_variable_declaration` → `declaration` in DART_QUERIES
  (the former doesn't exist in tree-sitter-dart grammar, causing all
  Dart integration tests to fail with TSQueryErrorNodeType)
- Add VARIABLE_SCHEMA to schema.ts and register in initLbug() so that
  Variable-labeled nodes are persisted to LadybugDB (not silently dropped)
- Add 'Variable' to MULTI_LANG_TYPES in csv-generator.ts
- Update Dart variable config to remove invalid node type
- Update schema test counts (30→31 node schemas, 32→33 total)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/f79931d1-207f-4fbb-91da-259d44f7fd88

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address code review comment improvements

- Clarify processedDefinitionNodes tracks start indices, not nodes
- Improve Python variableNodeTypes comment wording

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/f79931d1-207f-4fbb-91da-259d44f7fd88

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: add Variable to NODE_TABLES, RELATION_SCHEMA, update golden snapshot

- Add 'Variable' to NODE_TABLES in gitnexus-shared so validTables.has('Variable')
  returns true and Variable graph edges are not silently dropped
- Add FROM File TO Variable, FROM Variable TO Community, FROM Variable TO Process
  to RELATION_SCHEMA so KuzuDB can represent edges connecting Variable nodes
- Update schema.test.ts: add Variable to multiLang list, fix count 30→31
- Regenerate pipeline-graph-golden snapshot for mini-repo fixture

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/e3aad558-e7bb-40d1-b53f-0a2c0132ca96

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: isolate golden test from cli-e2e fixture pollution

The pipeline-graph-golden test was non-deterministic because cli-e2e.test.ts
creates AGENTS.md, CLAUDE.md, .claude/skills/, and .gitignore in the shared
mini-repo fixture during analyze. These leftover files caused the golden test
to find 9 files instead of 7 when tests ran in parallel.

Fixes:
- Golden test now copies the fixture to a temp dir before running, making it
  immune to concurrent test pollution
- cli-e2e afterAll cleanup now removes ALL generated files (AGENTS.md,
  CLAUDE.md, .claude/, .gitignore) not just .git/ and .gitnexus/
- Golden snapshot regenerated from clean 7-file fixture

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/bd378e73-6f37-49c6-aed6-7fabf4dc6183

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
2026-04-16 13:57:25 +01:00
..
src feat(ingestion): language-agnostic variable extractor with config+factory pattern (#878) 2026-04-16 13:57:25 +01:00
package-lock.json feat: unify web and cli ingestion pipeline (#536) 2026-03-28 14:07:11 +00:00
package.json feat: unify web and cli ingestion pipeline (#536) 2026-03-28 14:07:11 +00:00
tsconfig.json feat: unify web and cli ingestion pipeline (#536) 2026-03-28 14:07:11 +00:00