GitNexus/gitnexus/test/unit/schema.test.ts
Copilot ed5a4220dd
feat(ingestion): language-agnostic variable extractor with config+factory pattern (#878)
* Initial plan

* feat(ingestion): add variable extraction types, factory, configs, and wire into language providers

- Create variable-types.ts with VariableInfo, VariableExtractionConfig, VariableExtractor interfaces
- Create variable-extractors/generic.ts with createVariableExtractor() factory
- Add variableExtractor field to LanguageProvider interface
- Create per-language variable extraction configs for all 16 languages
- Wire variableExtractor into all language providers
- Add variable metadata enrichment to parse-worker for Const/Static/Variable labels

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* feat(ingestion): add variable extraction tests and fix Python/TS config issues

- Create test/unit/variable-extraction.test.ts with 29 tests covering
  TypeScript, JavaScript, Python, Go, Rust, C, C++, Ruby, and factory behavior
- Fix isConst in generic factory to use config.isConst over node-type membership
  (TS let/const both use lexical_declaration)
- Fix Python type extraction for annotated assignments at module scope
- Fix Python dunder name visibility (e.g., __name__ is public, not protected)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address code review feedback — move imports, clarify scope comment, use shared test context

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/3cb85c68-1792-473e-9a46-ea2588da0e5e

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address review comments, fix prettier formatting and lint errors

- Fix prettier formatting in 5 files (c-cpp, jvm, swift configs, test file)
- Remove unused SyntaxNode imports in php.ts and ruby.ts (lint errors)
- Remove unused constNodeSet/variableNodeSet variables in generic.ts (warnings)
- Remove semantically wrong `methodProps.isReadonly = varInfo.isConst` (review)
- Remove dead `nodeLabel === 'Variable'` guard in parse-worker (review)
- Fix test guard: replace `if (declNode)` with `expect(declNode).toBeDefined()` (review)
- Add comment about Python expression_statement broadness (review)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/040edbbf-65b5-40e1-80c8-e98f7c4bb54a

* feat(ingestion): add block-scoped variable extraction via tree-sitter queries

Add @definition.const and @definition.variable tree-sitter query patterns
for TypeScript, JavaScript, Python, Go, Java, C, C++, C#, PHP, Ruby, and
Dart. Add parse-worker dedup logic to avoid duplicate nodes when variable
captures overlap with existing function/property captures. Add 'Variable'
label support in getLabelFromCaptures and DEFINITION_CAPTURE_KEYS.

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* test: add block-scoped variable extraction tests and query capture tests

Add 6 tests for block-scoped variable extraction (TypeScript, Go, Rust, C,
Python). Add 14 tests verifying @definition.const/@definition.variable
query patterns exist in all language query strings. Import RUBY_QUERIES
in test file.

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* test: add Python non-assignment expression statement rejection test

Addresses code review feedback: verify that the Python variable extractor
returns null for expression_statement nodes that contain function calls
rather than assignments (e.g. `print("hello")`).

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/9fa828c1-87b7-4482-8f26-d2079fb4c58a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: Dart query node type, add Variable schema, update schema counts

- Change `top_level_variable_declaration` → `declaration` in DART_QUERIES
  (the former doesn't exist in tree-sitter-dart grammar, causing all
  Dart integration tests to fail with TSQueryErrorNodeType)
- Add VARIABLE_SCHEMA to schema.ts and register in initLbug() so that
  Variable-labeled nodes are persisted to LadybugDB (not silently dropped)
- Add 'Variable' to MULTI_LANG_TYPES in csv-generator.ts
- Update Dart variable config to remove invalid node type
- Update schema test counts (30→31 node schemas, 32→33 total)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/f79931d1-207f-4fbb-91da-259d44f7fd88

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: address code review comment improvements

- Clarify processedDefinitionNodes tracks start indices, not nodes
- Improve Python variableNodeTypes comment wording

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/f79931d1-207f-4fbb-91da-259d44f7fd88

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: add Variable to NODE_TABLES, RELATION_SCHEMA, update golden snapshot

- Add 'Variable' to NODE_TABLES in gitnexus-shared so validTables.has('Variable')
  returns true and Variable graph edges are not silently dropped
- Add FROM File TO Variable, FROM Variable TO Community, FROM Variable TO Process
  to RELATION_SCHEMA so KuzuDB can represent edges connecting Variable nodes
- Update schema.test.ts: add Variable to multiLang list, fix count 30→31
- Regenerate pipeline-graph-golden snapshot for mini-repo fixture

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/e3aad558-e7bb-40d1-b53f-0a2c0132ca96

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: isolate golden test from cli-e2e fixture pollution

The pipeline-graph-golden test was non-deterministic because cli-e2e.test.ts
creates AGENTS.md, CLAUDE.md, .claude/skills/, and .gitignore in the shared
mini-repo fixture during analyze. These leftover files caused the golden test
to find 9 files instead of 7 when tests ran in parallel.

Fixes:
- Golden test now copies the fixture to a temp dir before running, making it
  immune to concurrent test pollution
- cli-e2e afterAll cleanup now removes ALL generated files (AGENTS.md,
  CLAUDE.md, .claude/, .gitignore) not just .git/ and .gitnexus/
- Golden snapshot regenerated from clean 7-file fixture

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/bd378e73-6f37-49c6-aed6-7fabf4dc6183

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
2026-04-16 13:57:25 +01:00

225 lines
6.8 KiB
TypeScript

import { describe, it, expect } from 'vitest';
import {
NODE_TABLES,
REL_TABLE_NAME,
REL_TYPES,
EMBEDDING_TABLE_NAME,
NODE_SCHEMA_QUERIES,
REL_SCHEMA_QUERIES,
SCHEMA_QUERIES,
FILE_SCHEMA,
FOLDER_SCHEMA,
FUNCTION_SCHEMA,
CLASS_SCHEMA,
INTERFACE_SCHEMA,
METHOD_SCHEMA,
CODE_ELEMENT_SCHEMA,
COMMUNITY_SCHEMA,
PROCESS_SCHEMA,
RELATION_SCHEMA,
EMBEDDING_SCHEMA,
CREATE_VECTOR_INDEX_QUERY,
} from '../../src/core/lbug/schema.js';
describe('LadybugDB Schema', () => {
describe('NODE_TABLES', () => {
it('includes all core node types', () => {
const core = [
'File',
'Folder',
'Function',
'Class',
'Interface',
'Method',
'CodeElement',
'Community',
'Process',
];
for (const t of core) {
expect(NODE_TABLES).toContain(t);
}
});
it('includes multi-language node types', () => {
const multiLang = [
'Struct',
'Enum',
'Macro',
'Typedef',
'Union',
'Namespace',
'Trait',
'Impl',
'TypeAlias',
'Const',
'Static',
'Variable',
'Property',
'Record',
'Delegate',
'Annotation',
'Constructor',
'Template',
'Module',
];
for (const t of multiLang) {
expect(NODE_TABLES).toContain(t);
}
});
it('has expected total count', () => {
// 9 core + 19 multi-language + Route + Tool = 31
expect(NODE_TABLES).toHaveLength(31);
});
});
describe('REL_TYPES', () => {
it('includes all expected relationship types', () => {
const expected = [
'CONTAINS',
'DEFINES',
'IMPORTS',
'CALLS',
'EXTENDS',
'IMPLEMENTS',
'MEMBER_OF',
'STEP_IN_PROCESS',
];
for (const t of expected) {
expect(REL_TYPES).toContain(t);
}
});
});
describe('node schema DDL', () => {
it.each([
['FILE_SCHEMA', FILE_SCHEMA, 'File'],
['FOLDER_SCHEMA', FOLDER_SCHEMA, 'Folder'],
['FUNCTION_SCHEMA', FUNCTION_SCHEMA, 'Function'],
['CLASS_SCHEMA', CLASS_SCHEMA, 'Class'],
['INTERFACE_SCHEMA', INTERFACE_SCHEMA, 'Interface'],
['METHOD_SCHEMA', METHOD_SCHEMA, 'Method'],
['CODE_ELEMENT_SCHEMA', CODE_ELEMENT_SCHEMA, 'CodeElement'],
['COMMUNITY_SCHEMA', COMMUNITY_SCHEMA, 'Community'],
['PROCESS_SCHEMA', PROCESS_SCHEMA, 'Process'],
])('%s contains CREATE NODE TABLE for %s', (_, schema, tableName) => {
expect(schema).toContain('CREATE NODE TABLE');
expect(schema).toContain(tableName);
expect(schema).toContain('PRIMARY KEY');
});
it('Function schema has startLine and endLine', () => {
expect(FUNCTION_SCHEMA).toContain('startLine INT64');
expect(FUNCTION_SCHEMA).toContain('endLine INT64');
});
it('Function schema has isExported', () => {
expect(FUNCTION_SCHEMA).toContain('isExported BOOLEAN');
});
it('Community schema has heuristicLabel and cohesion', () => {
expect(COMMUNITY_SCHEMA).toContain('heuristicLabel STRING');
expect(COMMUNITY_SCHEMA).toContain('cohesion DOUBLE');
});
it('Process schema has processType and stepCount', () => {
expect(PROCESS_SCHEMA).toContain('processType STRING');
expect(PROCESS_SCHEMA).toContain('stepCount INT32');
});
});
describe('relation schema', () => {
it('creates a single REL TABLE named CodeRelation', () => {
expect(RELATION_SCHEMA).toContain(`CREATE REL TABLE ${REL_TABLE_NAME}`);
});
it('has type, confidence, reason, step properties', () => {
expect(RELATION_SCHEMA).toContain('type STRING');
expect(RELATION_SCHEMA).toContain('confidence DOUBLE');
expect(RELATION_SCHEMA).toContain('reason STRING');
expect(RELATION_SCHEMA).toContain('step INT32');
});
it('connects Function to Function (CALLS)', () => {
expect(RELATION_SCHEMA).toContain('FROM Function TO Function');
});
it('connects File to Function (CONTAINS/DEFINES)', () => {
expect(RELATION_SCHEMA).toContain('FROM File TO Function');
});
it('connects symbols to Community (MEMBER_OF)', () => {
expect(RELATION_SCHEMA).toContain('FROM Function TO Community');
expect(RELATION_SCHEMA).toContain('FROM Class TO Community');
});
it('connects symbols to Process (STEP_IN_PROCESS)', () => {
expect(RELATION_SCHEMA).toContain('FROM Function TO Process');
expect(RELATION_SCHEMA).toContain('FROM Method TO Process');
});
it('has all FROM/TO pairs needed for HAS_METHOD edges', () => {
// HAS_METHOD sources: Class, Interface, Struct, Trait, Impl, Record
// HAS_METHOD targets: Method, Constructor (Property is now HAS_PROPERTY)
const sources = ['Class', 'Interface'];
const backtickSources = ['Struct', 'Trait', 'Impl', 'Record'];
const targets = ['Method'];
const backtickTargets = ['Constructor'];
// Non-backtick source → non-backtick target
for (const src of sources) {
for (const tgt of targets) {
expect(RELATION_SCHEMA).toContain(`FROM ${src} TO ${tgt}`);
}
for (const tgt of backtickTargets) {
expect(RELATION_SCHEMA).toContain(`FROM ${src} TO \`${tgt}\``);
}
}
// Backtick source → all targets
for (const src of backtickSources) {
for (const tgt of targets) {
expect(RELATION_SCHEMA).toContain(`FROM \`${src}\` TO ${tgt}`);
}
for (const tgt of backtickTargets) {
expect(RELATION_SCHEMA).toContain(`FROM \`${src}\` TO \`${tgt}\``);
}
}
});
});
describe('embedding schema', () => {
it('creates CodeEmbedding table', () => {
expect(EMBEDDING_SCHEMA).toContain(`CREATE NODE TABLE ${EMBEDDING_TABLE_NAME}`);
expect(EMBEDDING_SCHEMA).toContain('embedding FLOAT[384]');
});
it('has vector index query', () => {
expect(CREATE_VECTOR_INDEX_QUERY).toContain('CREATE_VECTOR_INDEX');
expect(CREATE_VECTOR_INDEX_QUERY).toContain('cosine');
});
});
describe('schema query ordering', () => {
it('NODE_SCHEMA_QUERIES has correct count', () => {
expect(NODE_SCHEMA_QUERIES).toHaveLength(31);
});
it('REL_SCHEMA_QUERIES has one relation table', () => {
expect(REL_SCHEMA_QUERIES).toHaveLength(1);
});
it('SCHEMA_QUERIES includes all node + rel + embedding schemas', () => {
// 31 node + 1 rel + 1 embedding = 33
expect(SCHEMA_QUERIES).toHaveLength(33);
});
it('node schemas come before relation schemas in SCHEMA_QUERIES', () => {
const relIndex = SCHEMA_QUERIES.indexOf(RELATION_SCHEMA);
const lastNodeIndex = SCHEMA_QUERIES.indexOf(
NODE_SCHEMA_QUERIES[NODE_SCHEMA_QUERIES.length - 1],
);
expect(relIndex).toBeGreaterThan(lastNodeIndex);
});
});
});