mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-11 03:38:07 +00:00
* feat(ingestion): respect .gitignore and .gitnexusignore during file discovery Add support for excluding files from indexing based on .gitignore and .gitnexusignore patterns. Previously, GitNexus used only a hardcoded ignore list, causing significant index pollution in repositories with git-ignored directories containing code (e.g., Docker-mounted volumes). Changes: - Add `ignore` package for gitignore-spec pattern matching - Add `loadIgnoreRules()` to parse .gitignore + .gitnexusignore - Add `createIgnoreFilter()` returning glob-compatible IgnoreLike object - Integrate filter into glob's `ignore` option for directory-level pruning - Remove post-glob `.filter()` call (now handled during traversal) The hardcoded DEFAULT_IGNORE_LIST remains as fallback for non-git repos. Closes #228 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ingestion): address review feedback on ignore filtering - Distinguish ENOENT vs EACCES in loadIgnoreRules (warn on permission errors) - Add GITNEXUS_NO_GITIGNORE env var to bypass .gitignore parsing - Fix bare-name pattern matching in childrenIgnored (check both with/without trailing slash) - Rename isIgnoredDirectory to isHardcodedIgnoredDirectory for clarity - Add clarifying comments for design decisions (D2 negation, D3 dot:false redundancy) - Add tests for bare-name patterns, file-glob patterns, EACCES handling, env var Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ingestion): address second round of review feedback - G1: Document GITNEXUS_NO_GITIGNORE in `analyze --help` and log when active - G2: Add comment clarifying path-scurry POSIX normalization contract - G3: Add IgnoreOptions interface — env var now falls back, callers can pass `noGitignore` explicitly for testability and future CLI flag - G4: Add integration test verifying walkRepositoryPaths respects the env var Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(ingestion): gracefully skip files with unavailable tree-sitter grammars Port unsupported language resilience from PR #301 by @jecanore. - Make Kotlin import optional (like Swift) in parser-loader and parse-worker - Add worker-local isLanguageAvailable() with filePath param for tsx distinction - Track and log skipped files per language in both sequential and worker paths - Add skippedLanguages to ParseWorkerResult for worker→main aggregation - Add isLanguageAvailable unit tests Refs: #301, #155, #228 Co-Authored-By: jecanore <juan@housingbase.io> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(e2e): add ignore + language-skip end-to-end test with fixture repo Add a fixture repo (test/fixtures/ignore-and-skip-repo/) with .gitignore, .gitnexusignore, TypeScript source files, and a Swift file to exercise all three features end-to-end: - File discovery: verifies .gitignore excludes data/ and *.log, .gitnexusignore excludes vendor/, source files are discovered - Parsing: verifies TypeScript files produce Function nodes and DEFINES relationships, Swift files are skipped gracefully when grammar is unavailable Add the test to the standalone group in ci-integration.yml and coverage job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): move ignore-and-skip-e2e test to e2e group per review feedback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): use temp directory instead of fixture for e2e ignore test The fixture's .gitignore prevented data/seed.json and debug.log from being committed — these files would be missing after checkout in CI. Switch to creating the entire test structure in a temp directory via beforeAll (matching filesystem-walker.test.ts pattern). This ensures all files exist regardless of git ignore rules. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): correct graph API usage in e2e ignore test Use graph.nodes property getter instead of graph.getNodes(), and check Function node filePath instead of non-existent File nodes (File nodes are created by processStructure, not processParsing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: add workflows permission to ci-integration.yml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: change workflows permission to write per review Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: move workflows permission from ci-integration.yml to ci.yml caller Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): fix Claude workflows for fork PRs, remove misplaced workflows perm Three issues prevented Claude from running on fork PRs: 1. claude-code-review.yml lacked workflows:write — push failed when fork PRs modify .github/workflows/ files 2. claude.yml had no fork PR support — checked out main and couldn't fetch the fork's branch from origin 3. Cleanup step unconditionally deleted branches even when push failed, breaking the concurrent claude.yml workflow Also removes workflows:write from ci.yml's integration job — CI tests don't need that permission. The permission belongs on the claude workflows that push fork branches. Changes: - Add workflows:write to both claude workflow permissions blocks - Add fork PR detection + branch push/cleanup to claude.yml - Add step id to push-fork; cleanup only runs if push succeeded - Pass branch names via env vars to prevent shell injection (security) - Add concurrency groups to prevent race conditions between workflows - Remove misplaced workflows:write from ci.yml integration job * fix(ci): use GitHub API for fork branch refs instead of git push GITHUB_TOKEN cannot have 'workflows' permission — it's only valid for PATs and GitHub Apps. This means git push fails whenever a fork PR modifies .github/workflows/ files. Replace git push with the GitHub REST API (POST/PATCH /git/refs) to create temporary branch refs. The API creates a pointer to the already-existing PR head commit without triggering the workflow file push protection. Similarly, cleanup uses DELETE /git/refs instead of git push --delete. Also removes the invalid 'workflows: write' from permissions blocks. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: jecanore <juan@housingbase.io> Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
198 lines
8.1 KiB
YAML
198 lines
8.1 KiB
YAML
name: Integration Tests
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
collect-coverage:
|
|
description: 'Whether to run the coverage collection job (only needed for PR reports)'
|
|
required: false
|
|
default: true
|
|
type: boolean
|
|
|
|
jobs:
|
|
# ── Integration test matrix ─────────────────────────────────────────
|
|
# Each test-group runs on a SEPARATE runner per OS, giving full process
|
|
# isolation for the LadybugDB native C++ addon.
|
|
# 3 OS x 4 groups = 12 parallel jobs.
|
|
#
|
|
# Groups:
|
|
# lbug-db — 7 files using withTestLbugDB / lbug-adapter (native addon)
|
|
# Each file runs as its own `vitest run` invocation for full
|
|
# process isolation. LadybugDB's native N-API addon registers
|
|
# persistent handles that prevent fork workers from exiting
|
|
# on Linux, and its C++ destructors segfault during
|
|
# process.exit(). Running each file in its own process lets
|
|
# the OS reclaim all resources cleanly.
|
|
# pipeline — 12 files: ingestion pipeline + csv + 9 resolver tests
|
|
# e2e — 4 files: child-process only (spawnSync), no in-process lbug
|
|
# standalone — 4 files: pure logic, no lbug, no child processes
|
|
test-matrix:
|
|
name: integration (${{ matrix.os }} / ${{ matrix.test-group }})
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
test-group: [lbug-db, pipeline, e2e, standalone]
|
|
include:
|
|
- test-group: lbug-db
|
|
# Marker — actual files are listed in the run step below
|
|
test-glob: ''
|
|
- test-group: pipeline
|
|
test-glob: >-
|
|
test/integration/pipeline.test.ts
|
|
test/integration/csv-pipeline.test.ts
|
|
test/integration/parsing.test.ts
|
|
test/integration/resolvers/typescript.test.ts
|
|
test/integration/resolvers/csharp.test.ts
|
|
test/integration/resolvers/cpp.test.ts
|
|
test/integration/resolvers/java.test.ts
|
|
test/integration/resolvers/python.test.ts
|
|
test/integration/resolvers/rust.test.ts
|
|
test/integration/resolvers/go.test.ts
|
|
test/integration/resolvers/kotlin.test.ts
|
|
test/integration/resolvers/php.test.ts
|
|
test/integration/resolvers/ruby.test.ts
|
|
test/integration/resolvers/swift.test.ts
|
|
- test-group: e2e
|
|
test-glob: >-
|
|
test/integration/cli-e2e.test.ts
|
|
test/integration/hooks-e2e.test.ts
|
|
test/integration/skills-e2e.test.ts
|
|
test/integration/ignore-and-skip-e2e.test.ts
|
|
- test-group: standalone
|
|
test-glob: >-
|
|
test/integration/filesystem-walker.test.ts
|
|
test/integration/enrichment.test.ts
|
|
test/integration/tree-sitter-languages.test.ts
|
|
test/integration/worker-pool.test.ts
|
|
runs-on: ${{ matrix.os }}
|
|
timeout-minutes: 25
|
|
steps:
|
|
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
with:
|
|
build: 'true'
|
|
|
|
# lbug-db: run each file in its own vitest process for full isolation.
|
|
# LadybugDB's native addon hangs fork workers on Linux — process isolation
|
|
# is the only reliable fix boundary.
|
|
- name: Run integration tests — lbug-db (process-isolated)
|
|
if: matrix.test-group == 'lbug-db'
|
|
working-directory: gitnexus
|
|
shell: bash
|
|
run: |
|
|
set -e
|
|
files=(
|
|
test/integration/lbug-core-adapter.test.ts
|
|
test/integration/lbug-pool.test.ts
|
|
test/integration/local-backend.test.ts
|
|
test/integration/local-backend-calltool.test.ts
|
|
test/integration/search-core.test.ts
|
|
test/integration/search-pool.test.ts
|
|
test/integration/augmentation.test.ts
|
|
)
|
|
exit_code=0
|
|
for f in "${files[@]}"; do
|
|
echo "::group::$f"
|
|
if ! npx vitest run --reporter=verbose --pool=forks "$f"; then
|
|
exit_code=1
|
|
echo "::error::Test file failed: $f"
|
|
fi
|
|
echo "::endgroup::"
|
|
done
|
|
exit $exit_code
|
|
|
|
# Non-lbug groups: run all files in a single vitest invocation
|
|
- name: Run integration tests — ${{ matrix.test-group }}
|
|
if: matrix.test-group != 'lbug-db'
|
|
shell: bash
|
|
env:
|
|
TEST_GLOB: ${{ matrix.test-glob }}
|
|
run: npx vitest run --reporter=verbose $TEST_GLOB
|
|
working-directory: gitnexus
|
|
|
|
# ── Coverage collection (ubuntu only) ─────────────────────────────────
|
|
# Runs non-lbug integration tests with coverage enabled so the PR report
|
|
# can merge integration + unit coverage for a combined view.
|
|
# lbug-db tests are excluded because each file must run in its own vitest
|
|
# process (native addon isolation) which prevents single-run coverage merge.
|
|
coverage:
|
|
name: integration (ubuntu / coverage)
|
|
if: inputs.collect-coverage
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
- uses: ./.github/actions/setup-gitnexus
|
|
with:
|
|
build: 'true'
|
|
|
|
- name: Run integration tests with coverage
|
|
working-directory: gitnexus
|
|
run: >-
|
|
npx vitest run
|
|
--reporter=default
|
|
--reporter=json
|
|
--outputFile=integration-results.json
|
|
--coverage
|
|
--coverage.reporter=json-summary
|
|
--coverage.reporter=json
|
|
--coverage.reporter=text
|
|
--coverage.thresholdAutoUpdate=false
|
|
--coverage.reportOnFailure=true
|
|
--coverage.thresholds.statements=0
|
|
--coverage.thresholds.branches=0
|
|
--coverage.thresholds.functions=0
|
|
--coverage.thresholds.lines=0
|
|
test/integration/pipeline.test.ts
|
|
test/integration/csv-pipeline.test.ts
|
|
test/integration/parsing.test.ts
|
|
test/integration/cli-e2e.test.ts
|
|
test/integration/hooks-e2e.test.ts
|
|
test/integration/filesystem-walker.test.ts
|
|
test/integration/enrichment.test.ts
|
|
test/integration/tree-sitter-languages.test.ts
|
|
test/integration/worker-pool.test.ts
|
|
test/integration/ignore-and-skip-e2e.test.ts
|
|
test/integration/resolvers/typescript.test.ts
|
|
test/integration/resolvers/csharp.test.ts
|
|
test/integration/resolvers/cpp.test.ts
|
|
test/integration/resolvers/java.test.ts
|
|
test/integration/resolvers/python.test.ts
|
|
test/integration/resolvers/rust.test.ts
|
|
test/integration/resolvers/go.test.ts
|
|
test/integration/resolvers/kotlin.test.ts
|
|
test/integration/resolvers/php.test.ts
|
|
test/integration/resolvers/ruby.test.ts
|
|
test/integration/resolvers/swift.test.ts
|
|
|
|
- name: Upload integration coverage
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
|
with:
|
|
name: integration-reports
|
|
path: |
|
|
gitnexus/coverage/coverage-summary.json
|
|
gitnexus/coverage/coverage-final.json
|
|
gitnexus/integration-results.json
|
|
retention-days: 5
|
|
|
|
# ── Unified status gate ──────────────────────────────────────────────
|
|
# Branch protection should require THIS job, not the matrix jobs directly.
|
|
# ci.yml's needs.integration.result aggregates through this gate.
|
|
status:
|
|
name: integration (all groups)
|
|
needs: test-matrix
|
|
if: always()
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
steps:
|
|
- name: Check all matrix jobs passed
|
|
shell: bash
|
|
env:
|
|
RESULT: ${{ needs.test-matrix.result }}
|
|
run: |
|
|
if [[ "$RESULT" != "success" ]]; then
|
|
echo "::error::Integration matrix failed or cancelled: $RESULT"
|
|
exit 1
|
|
fi
|