mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-06 02:49:56 +00:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gergő Magyar
|
0561d24efd
|
feat: METHOD_IMPLEMENTS edges, overload disambiguation, MethodExtractor unification (#574) (#642) | ||
|
Gergő Magyar
|
bf09eab95b
|
feat: configure prettier with pre-commit hook (#563)
* feat: configure prettier with pre-commit hook integration Add prettier, lint-staged, and prettier-plugin-tailwindcss at the repo root with husky pre-commit hook integration. Moves husky from gitnexus/ to root package.json for reliable hook installation. - Root package.json with prepare/format/format:check scripts - .prettierrc with endOfLine:lf and tailwindStylesheet for TW v4 - .prettierignore excluding fixtures, vendor, generated, *.d.ts, *.md - .gitattributes enforcing LF line endings for Windows consistency - Pre-commit hook uses direct node_modules/.bin/ paths (no npx) * style: apply prettier formatting to entire codebase One-time bulk format. No logic changes. Use .git-blame-ignore-revs to skip this commit in git blame. * chore: add .git-blame-ignore-revs for prettier format commit * perf: pre-commit hook runs only tests related to staged files Use vitest --related to scope test execution to tests that import the changed files, instead of running the full suite on every commit. * perf: remove vitest from pre-commit hook, keep in CI only Pre-commit now runs lint-staged + tsc only. Tests run in CI (ci-tests.yml) where they belong — keeps commits fast. * ci: add prettier format check to quality workflow PRs will now fail if code isn't formatted with prettier. |
||
|
Gergő Magyar
|
fd7fb5bf1f
|
feat: unify web and cli ingestion pipeline (#536)
* feat: add server-side ingestion API (POST /api/analyze, SSE progress)
Extract core analysis orchestration from CLI into shared run-analyze.ts
module. Add server-side analyze endpoints so the web app can trigger
ingestion via HTTP instead of running the full pipeline in-browser.
New files:
- src/core/run-analyze.ts — shared runFullAnalysis() orchestrator
- src/server/analyze-job.ts — job manager (single-slot, dedup, SSE events)
- src/server/analyze-worker.ts — forked child process (8GB heap, IPC)
- src/server/git-clone.ts — shallow clone/pull with SSRF protection
API endpoints:
- POST /api/analyze — start analysis (returns 202 + jobId)
- GET /api/analyze/:jobId — poll job status
- GET /api/analyze/:jobId/progress — SSE progress stream
Security: URL validation blocks private IPs and non-HTTP schemes.
Path validation requires absolute paths. Git stderr not leaked to API.
* feat(web): add server-side analyze UI (Phase 2)
Add "Analyze on Server" flow to the web app's Server tab so users
can trigger server-side ingestion from the browser. On completion,
the graph is automatically loaded via the existing connectToServer flow.
New files:
- AnalyzeProgress.tsx — progress bar with phase label, elapsed time, cancel
Modified files:
- backend.ts — startAnalyze(), streamAnalyzeProgress() SSE client
- DropZone.tsx — analyze URL input + button below Connect section
- App.tsx — onServerAnalyze handler wires analyze -> connect flow
* feat: add job cancellation, timeout, and child process tracking (Phase 3)
- DELETE /api/analyze/:jobId — cancel running analysis (SIGTERM to worker)
- 30-minute timeout kills long-running workers automatically
- Child process refs tracked in JobManager for cleanup on shutdown
- dispose() kills all active children on SIGINT/SIGTERM
- Web cancel button now calls server DELETE endpoint
- cancelAnalyze() added to web backend client
* refactor(web): remove browser ingestion pipeline (Phase 4)
Delete 16 duplicated ingestion files, 2 unused service files
(git-clone, zip), and tree-sitter parser-loader from gitnexus-web.
All ingestion now runs server-side via POST /api/analyze.
Deleted (18 files, ~5,000 lines):
- core/ingestion/*.ts (16 pipeline processors)
- core/tree-sitter/parser-loader.ts (WASM tree-sitter loader)
- services/git-clone.ts (isomorphic-git client-side clone)
- services/zip.ts (JSZip extraction)
Simplified:
- DropZone.tsx — server-only (removed ZIP/GitHub tabs)
- ingestion.worker.ts — removed runPipeline/runPipelineFromFiles
- useAppState.tsx — removed pipeline callbacks
- App.tsx — removed handleFileSelect/handleGitClone
- main.tsx — removed Buffer polyfill for isomorphic-git
- types/pipeline.ts — removed PipelineResult/serialize helpers
Kept: cluster-enricher.ts (LLM enrichment, still used by worker)
Dependencies now removable: web-tree-sitter, isomorphic-git,
@isomorphic-git/lightning-fs, jszip (estimated 3-4MB bundle savings)
* refactor(web): sync graph schema from CLI + delete WASM grammars
Sync graph/types.ts and lbug/schema.ts from the CLI (source of truth)
to the web module so the browser LadybugDB can handle all node and
relationship types the server pipeline produces.
Synced types: Route, Tool, Section node labels; HANDLES_ROUTE, FETCHES,
HANDLES_TOOL, ENTRY_POINT_OF, WRAPS, QUERIES relationship types;
description fields on Function/Class/Interface/Method/CodeElement.
Deleted: public/wasm/ directory (14 tree-sitter WASM grammars + core).
Removed deps: web-tree-sitter, isomorphic-git, @isomorphic-git/lightning-fs,
jszip, buffer, @types/jszip (~3-4MB bundle savings).
* feat: create gitnexus-shared package for unified type definitions
Create a new gitnexus-shared package that is the single source of truth
for types shared between the CLI and web modules:
- SupportedLanguages enum (15 languages)
- Graph types: NodeLabel, NodeProperties, RelationshipType, GraphNode, GraphRelationship
- Schema constants: NODE_TABLES, REL_TYPES, REL_TABLE_NAME, EMBEDDING_TABLE_NAME
- Pipeline types: PipelinePhase, PipelineProgress
Both gitnexus (CLI) and gitnexus-web import from gitnexus-shared via
file: dependency. Each package re-exports and extends with platform-specific
additions (CLI: KnowledgeGraph with mutation methods; Web: simpler KnowledgeGraph).
This ensures types can never drift between packages — adding a new
language, node type, or relationship type in gitnexus-shared automatically
propagates to both consumers.
* refactor: import shared types directly from gitnexus-shared at call sites
Replace all re-export patterns with direct imports from gitnexus-shared.
72 files updated across CLI and web:
- SupportedLanguages: 49 CLI files now import from 'gitnexus-shared'
instead of '../config/supported-languages.js'
- GraphNode, GraphRelationship, NodeLabel: 22 CLI + 10 web files now
import from 'gitnexus-shared' instead of local re-export wrappers
- NODE_TABLES: api.ts imports from 'gitnexus-shared'
- PipelineProgress: useAppState.tsx imports from 'gitnexus-shared'
Local types.ts files now only define platform-specific KnowledgeGraph
(CLI has mutation methods, web has add-only). No more re-exports.
* fix: update lock files for gitnexus-shared, remove stale vite polyfills
Add gitnexus-shared@1.0.0 to lock files so npm ci succeeds in CI.
Remove buffer polyfill and global define from vite.config.ts (isomorphic-git was removed).
* fix(security): add write guard to HTTP /api/query, fix CORS proxy bypass
- Add isWriteQuery() check to POST /api/query handler — blocks CREATE,
DELETE, SET, MERGE, DROP, etc. via HTTP API (guard was only in MCP
pool adapter and browser-side, not the HTTP server path)
- Extend CYPHER_WRITE_RE with CALL, INSTALL, LOAD keywords
- Fix CORS proxy subdomain bypass: endsWith('github.com') allowed
'evil-github.com'. Now requires exact match or '.github.com' suffix
* feat(server): enhance /api/search with enrichment, add /api/grep, strip graph content
- POST /api/search: add mode param (hybrid|semantic|bm25), server-side
enrichment returns connections/cluster/processes per result in one call
(collapses 31 sequential HTTP calls to 1 for the agent search tool)
- GET /api/grep: regex search across indexed file contents, eliminates
need to transfer all file contents to browser
- GET /api/graph: strip content field by default (80-95% payload
reduction). Use ?includeContent=true for backward compat
- Add LRU cache invalidation hook point for future caching
* feat(server): add /api/embed endpoint for server-side embedding generation
- POST /api/embed: triggers embedding pipeline via onnxruntime-node
with JobManager for single-slot concurrency, timeout, and dedup
- GET /api/embed/:jobId: poll job status
- GET /api/embed/:jobId/progress: SSE stream with heartbeat, event IDs,
and X-Accel-Buffering:no header for proxy compatibility
- DELETE /api/embed/:jobId: cancel running embedding job
- Maps embedding pipeline phases (ready→complete, error→failed) to
JobManager status conventions
* feat(web): create consolidated BackendClient module
Single HTTP client replacing backend.ts, server-connection.ts, and
worker HTTP helpers. Includes:
- Typed methods: runQuery, search (enriched), grep, readFile, connect
- Generic streamSSE<T> utility extracted from analyze progress pattern
- BackendError with discriminated code field (network/server/client/timeout)
- Embed API: startEmbeddings, streamEmbeddingProgress, cancelEmbeddings
- Search with mode param (hybrid|semantic|bm25) and enrichment
* refactor(web): rewrite Graph RAG tools for backend-only HTTP queries
- Search tool: uses enriched /api/search (1 call replaces 31 sequential queries)
- Cypher tool: removes browser-side embedding; {{QUERY_VECTOR}} routes to
/api/search with mode:'semantic' instead of local transformers.js
- Grep tool: uses /api/grep instead of in-memory fileContents map
- Read tool: uses /api/file instead of fileContents map lookup
- Impact tool: getCallSiteSnippet now async via /api/file
- createGraphRAGTools now accepts GraphRAGBackend interface instead of
7 separate function params + fileContents map
- createGraphRAGAgent simplified to (config, backend, context?)
- Removed imports: embedder, lbug/schema (replaced with gitnexus-shared)
- Net: -205 lines
* refactor(web): delete WASM infrastructure, remove 7 packages (-5242 lines)
Delete browser-side LadybugDB, embeddings, search, and worker:
- gitnexus-web/src/core/lbug/ (adapter, csv-generator, schema, query-result)
- gitnexus-web/src/core/embeddings/ (embedder, pipeline, text-gen, types)
- gitnexus-web/src/core/search/ (bm25-index, hybrid-search)
- gitnexus-web/src/workers/ingestion.worker.ts (828 lines)
- gitnexus-web/src/services/server-connection.ts (merged into backend-client)
- gitnexus-web/src/types/lbug-wasm.d.ts
Remove packages: @ladybugdb/wasm-core, @huggingface/transformers,
comlink, minisearch, vite-plugin-wasm, vite-plugin-top-level-await,
vite-plugin-static-copy
Update vite.config.ts: remove WASM plugins, COOP/COEP headers,
worker config, optimizeDeps exclude
Update imports: App.tsx, DropZone, Header, AnalyzeProgress,
BackendRepoSelector, useBackend → backend-client
* refactor(web): replace Worker/Comlink with direct BackendClient calls
- useAppState: remove Worker instantiation, Comlink.wrap, apiRef.
All queries now go through BackendClient HTTP functions directly.
- Agent runs on main thread (I/O-bound LLM streaming, not CPU-bound)
- initializeAgent: creates GraphRAGAgent with GraphRAGBackend interface
bound to BackendClient methods (runQuery, search, grep, readFile)
- startEmbeddings: calls POST /api/embed + SSE progress instead of
running browser-side transformers.js pipeline
- switchRepo: no longer loads graph into WASM DB or extracts fileContents
- App.tsx: handleServerConnect simplified (no fileContents, no loadServerGraph)
- Delete old backend.ts (replaced by backend-client.ts)
- Net: -396 lines
* fix(web): fix await-in-map build error in agent streaming
Move dynamic import of AIMessage outside .map() callback to avoid
"await can only be used inside an async function" build error.
* fix(web): remove stale apiRef references that broke chat functionality
sendChatMessage referenced apiRef.current (deleted Worker ref) which
would throw TypeError. Replaced with agentRef.current guard since agent
now runs on main thread.
* fix(server): dispose embedJobManager on shutdown, fix job mutation
- Add embedJobManager.dispose() to shutdown handler (was missing,
causing cleanup timer to keep Node process alive)
- Replace direct job.repoName/status mutation with updateJob() to
ensure SSE event emission for initial status change
* fix(server): parameterize Cypher, harden grep, unify SSE endpoints
- Search enrichment: replace string interpolation with executePrepared()
using $nid parameter binding to prevent Cypher injection
- Add executePrepared() to core lbug-adapter (prepare/execute pattern)
- /api/grep: add 200-char pattern length limit (ReDoS protection),
search files on disk instead of loading entire corpus into memory
(constant memory usage regardless of repo size)
- Extract mountSSEProgress() shared helper for SSE streaming — both
analyze and embed endpoints now have consistent heartbeat (30s),
event IDs (reconnection support), and X-Accel-Buffering header
* refactor(web): remove dead code from Worker-era architecture
- Remove loadServerGraph no-op function, interface member, and all consumers
- Remove testArrayParams stub and interface member
- Remove fileContents state from GraphStateProvider (never populated in
server-side architecture)
- Remove forceDevice parameter from startEmbeddings (server-side, no device choice)
- Replace phantom EmbeddingProgress type with inline { phase, percent }
- Replace resolvePathFromContents (needed fileContents Map) with graph-based
file path resolution using filePathIndex built from graph nodes
- Fix: AI citation grounding ([[file.ts:10]]) now works via graph node lookup
instead of broken fileContents-based resolution
* fix(web): use streamAgentResponse for full tool_call/reasoning streaming
Replace naive agent.stream() loop that only handled content chunks with
streamAgentResponse() generator from agent.ts. This properly routes:
- reasoning tokens (before/between tool calls)
- tool_call events (name, args, status)
- tool_result events (completed tool output)
- content tokens (final answer after all tools done)
Previously the onChunk handler for tool_call/tool_result/reasoning was
dead code since the streaming loop only emitted content events.
* fix(web): resolve CI type errors from dead code removal
- Import GraphNode/GraphRelationship from gitnexus-shared in graph.ts
(not re-exported from local types.ts)
- Add Route, Tool entries to NODE_COLORS and NODE_SIZES constants
- Add PipelineResult type to web types/pipeline.ts
- Remove fileContents from CodeReferencesPanel and RightPanel
- Remove testArrayParams and forceDevice from EmbeddingStatus
- Remove forceDevice args from startEmbeddings() calls in App.tsx
- Fix embeddingProgress property accesses for simplified type
* fix(ci): add setup-gitnexus-web action, build shared once per job
- Remove prepare script from gitnexus-shared (tsc not available during
npm ci of consuming packages)
- Create .github/actions/setup-gitnexus-web composite action: builds
gitnexus-shared then runs npm ci for gitnexus-web
- setup-gitnexus action: already builds gitnexus-shared for CLI jobs
- ci-quality typecheck-web: uses setup-gitnexus-web (DRY)
- ci-e2e: uses setup-gitnexus-web (DRY)
- ci-tests: gitnexus-shared already built by setup-gitnexus, just
install web deps without rebuilding
* fix(ci): use prepare script so gitnexus-shared builds during npm ci
Move typescript from devDependencies to dependencies in gitnexus-shared
so the prepare script (tsc) works when npm resolves file: deps during
npm ci. No GHA modifications needed — npm handles the build lifecycle
automatically.
Remove manual gitnexus-shared build steps from setup-gitnexus and
setup-gitnexus-web actions.
* fix(ci): build gitnexus-shared explicitly in setup actions
The file: dependency protocol doesn't reliably run prepare scripts
because devDependencies aren't installed first. Instead of fragile
lifecycle hacks, build gitnexus-shared explicitly in both setup actions:
- setup-gitnexus: npm install && npm run build in gitnexus-shared/
- setup-gitnexus-web: same, before npm ci in gitnexus-web/
- ci-tests: shared already built by setup-gitnexus, web just npm ci
No prepare script, no dist in git, no typescript as a prod dependency.
* fix: remove CALL from CYPHER_WRITE_RE — breaks FTS and vector search
CALL is used by read-only procedures: CALL QUERY_FTS_INDEX(...) and
CALL QUERY_VECTOR_INDEX(...). Adding it to the write guard blocked all
FTS search, causing 3 test failures. The database is opened in read-only
mode as defense-in-depth against write procedures via CALL.
Keep INSTALL and LOAD in the blocklist (genuinely dangerous).
* fix(web): update vercel.json for gitnexus-shared, remove COOP/COEP
- Add installCommand that builds gitnexus-shared before installing
web deps (Vercel doesn't know about the monorepo file: dependency)
- Remove Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy
headers (no longer needed — WASM LadybugDB removed)
* fix(web): update tests for deleted modules
- Delete csv-generator.test.ts (tests deleted WASM-only csv-generator)
- Update security-guards.test.ts: import NODE_TABLES/REL_TYPES from
gitnexus-shared instead of deleted src/core/lbug/schema
- Update server-connection.test.ts: import normalizeServerUrl from
backend-client, remove extractFileContents tests (function deleted)
* fix(e2e): remove Server tab click — UI is now server-only
The DropZone no longer has ZIP/GitHub/Server tabs (browser ingestion
was removed). The server URL input is directly visible on the landing
page. Update e2e test to skip the tab click and go straight to input.
All 5 e2e tests pass locally.
* refactor: use gitnexus-shared for PipelinePhase/PipelineProgress types
CLI was duplicating PipelinePhase and PipelineProgress locally instead
of importing from gitnexus-shared. Updated all consumers to import
directly. Also removed dead code: SerializablePipelineResult,
serializePipelineResult(), deserializePipelineResult().
* fix(server): address PR #536 review — security, race conditions, dead code
- Fix path traversal in POST /api/analyze: split into isAbsolute + normalize check
- Add shared repo lock (activeRepoPaths) preventing concurrent analyze+embed on same repo
- Fix 202 response returning actual job.status instead of hardcoded 'queued'
- Add 30-minute timeout for embedding jobs (was missing unlike analyze jobs)
- Fix DropZone calling startAnalyze without setting backend URL first
- Add SSE reconnect with exponential backoff (3 retries) and Last-Event-ID
- Fix normalizeServerUrl to return base URL (no /api suffix) — clear contract
- Delete dead code: proxy.ts, server-graph-hydration.ts, pipeline.ts re-export barrel
- Update LoadingOverlay to import PipelineProgress directly from gitnexus-shared
* fix(server): fix repo lock key mismatch and embed cancel race
- Use getStoragePath(targetPath) as lock key in analyze handler to match
embed handler's entry.storagePath — keys now always align
- Guard embed completion: don't overwrite 'failed' with 'complete' when
job was cancelled while pipeline was still running
- Remove unused jobType parameter from acquireRepoLock
- Log backend.init() errors instead of silently swallowing
* fix: add gitnexus-shared as a local dependency in package-lock.json
* refactor: move language detection to gitnexus-shared, add syntax highlighting for all 15 languages
Move getLanguageFromFilename() from CLI to gitnexus-shared with COBOL
support added. Add getSyntaxLanguageFromFilename() for Prism-compatible
syntax highlighting covering all 15 code languages plus auxiliary
formats (json, yaml, markdown, html, css, bash, sql, xml).
Refactor CodeReferencesPanel to use shared function instead of a local
30-line switch. Delete dead gitnexus-web/src/config/supported-languages.ts
(web already imports SupportedLanguages from gitnexus-shared).
* feat(web): add first-time user onboarding with auto server detection
Replace the manual "Connect to Server" panel with an automatic onboarding
flow that guides first-time users through starting the GitNexus server.
Server detection:
- useBackend hook polls via setTimeout chain (3s, no overlap)
- Page Visibility API pauses polling when tab is hidden
- SSE heartbeat (/api/heartbeat) for instant disconnect detection
Onboarding UI (OnboardingGuide.tsx):
- Step-by-step flow: copy command → run → auto-connect
- Smart command: shows `gitnexus serve` in dev, `npx gitnexus@latest serve` in prod
- Node.js version auto-detected from package.json via Vite define
- Faux terminal windows with copy-to-clipboard, platform tabs, polling indicator
Transitions (DropZone.tsx):
- Crossfade wrapper with snapshot pattern for smooth phase transitions
- Three phases: onboarding → success (1.2s hold) → loading → graph
- Auto-recovery: falls back to onboarding if server dies or connect fails
Server changes:
- GET /api/heartbeat: SSE endpoint for liveness detection
- GET /api/info: version, launch context, Node.js version
- npm run serve script for local development
- app.disable('x-powered-by') hardening
* feat(web): add repo analysis UI, SSE heartbeat, and review fixes
Repo analysis:
- AnalyzeOnboarding: empty-state card when server has zero repos
- RepoAnalyzer: GitHub URL + Local Folder tabs with browse button
- Header repo dropdown: click project badge to switch repos or analyze new
- DropZone 'analyze' phase integrated into Crossfade transitions
Reliability fixes from 5-agent review:
- Polling: stop scheduling timers when tab hidden, restart on visibility return
- Heartbeat: exponential backoff (1s/2s/4s, 3 retries) prevents graph loss on blip
- RepoAnalyzer: completion timer tracked in ref, cleaned up on unmount
- DropZone: standardized card padding (p-7), heading sizes (text-lg)
Accessibility:
- prefers-reduced-motion global CSS rule (WCAG 2.3.3)
- focus-visible rings on CopyButton
- cursor-pointer on all Header buttons
- Consistent rounded-xl on all dropdowns
Cleanup:
- Deleted dead AnalyzeSheet.tsx (219 LOC) and BackendRepoSelector.tsx (89 LOC)
- Fixed AnalyzeProgress lucide import (lucide-react → @/lib/lucide-icons)
* fix(server): resolve analyze worker fork crash in dev mode
The forked analyze worker was crashing immediately with exit code 1
when running via `npm run serve` (tsx). Two issues:
1. Worker path resolved to `analyze-worker.js` but only `.ts` exists
in the source directory — the `.js` file is only in `dist/`.
2. On Windows, bare `--import tsx` in execArgv fails because Node's
ESM resolver for --import uses the child's CWD, not the parent's
node_modules. Windows also rejects raw paths as `d:` is not a
valid URL scheme.
Fix: detect dev vs prod via `import.meta.url` extension. In dev mode,
resolve `tsx/esm` to an absolute `file://` URL via `pathToFileURL()`
anchored to the parent's `createRequire` context. This works on all
platforms and doesn't depend on the child's CWD or PATH.
Also captures child stderr for better crash diagnostics.
Verified: `POST /api/analyze` with GitHub URL completes successfully
in dev mode (tsx) — status goes from cloning → analyzing → complete.
* fix(server): add worker auto-retry, error handling, and crash diagnostics
Worker resilience:
- Auto-retry up to 2 times with exponential backoff (1s, 2s) on crash
- SSE progress shows "Retrying after crash (1/2)..." during retry
- Captures child stderr for crash diagnostics in failure message
- AnalyzeJob tracks retryCount per job
Server error handling:
- app.listen wrapped in Promise so EADDRINUSE/EACCES propagate cleanly
- serve.ts catches startup errors with friendly messages and exit code 1
- EADDRINUSE gets actionable guidance (stop other process or --port flag)
- Global uncaughtException/unhandledRejection handlers prevent silent exits
- DEBUG=1 env var shows full stack traces
* feat: add e2e tests for onboarding flows, worker retry, and error handling
E2E tests (onboarding.spec.ts — 11 tests):
- Flow 1: OnboardingGuide shown when server unreachable (6 tests)
- Flow 2: Auto-connect with success card, analyze phase for zero repos
- Flow 3: Analyze form — GitHub URL validation, Local Folder tab, tab switching
- Flow 4: Repo dropdown in exploring view (skipped without live server)
Updated server-connect.spec.ts:
- Replaced manual Connect button flow with auto-connect waitForGraphLoaded
Server resilience:
- Worker auto-retry (2 attempts with exponential backoff) on crash
- Friendly error messages for serve startup failures (EADDRINUSE etc.)
- Global uncaughtException/unhandledRejection handlers prevent silent exits
- app.listen wrapped in Promise for proper error propagation
* refactor(shared): enforce exhaustive language coverage via Record types
Replace the if/else chain in getLanguageFromFilename with two exhaustive
Record<SupportedLanguages, ...> maps:
- EXTENSION_MAP: every language → its file extensions
- SYNTAX_MAP: every language → its Prism syntax identifier
Adding a new member to the SupportedLanguages enum without adding it to
both maps now produces a TypeScript compile error:
Property '[SupportedLanguages.NewLang]' is missing in type...
This matches the existing pattern in languages/index.ts (providers table)
which already uses `satisfies Record<SupportedLanguages, LanguageProvider>`.
Three compile-time enforcement points now exist:
1. EXTENSION_MAP in language-detection.ts (file extensions)
2. SYNTAX_MAP in language-detection.ts (Prism syntax identifiers)
3. providers in languages/index.ts (LanguageProvider instances)
* feat(web): load source code from server and scroll to selected line
CodeReferencesPanel now fetches file content via GET /api/file when a
node is selected, instead of showing "Code not available in memory".
- Fetches via readFile() from backend-client when selectedFilePath changes
- Shows loading spinner while fetching
- After content loads, auto-scrolls to the selected node's startLine
- Highlights the selected line range with a cyan left border
- Cancels in-flight fetch if selection changes before it completes
Also: refactored language-detection.ts to use exhaustive Record types
(EXTENSION_MAP and SYNTAX_MAP) so adding a new SupportedLanguages enum
member without implementing extensions/syntax is a compile error.
* feat: buffered file reading for Code Inspector
Server: GET /api/file now supports ?startLine=N&endLine=M for reading
a line range instead of the entire file. Returns { content, startLine,
endLine, totalLines }.
Client: readFile() returns ReadFileResult with metadata. When selecting
a symbol (function, class, method), fetches only ±50 lines around the
symbol's startLine/endLine instead of the full file. File nodes still
fetch the entire file.
SyntaxHighlighter startingLineNumber set from the buffer offset so line
numbers are correct even for partial reads.
* fix: adapt readFile callers to new ReadFileResult return type
tools.ts: readFile comes from GraphRAGBackend interface which returns
Promise<string> (the adapter in useAppState extracts .content), so
revert the { content } destructuring back to plain string assignment.
useAppState.tsx: wrap backendReadFile with { repo } options object
and extract .content to satisfy the GraphRAGBackend interface.
* fix(web): ensure new repos appear in list immediately after analysis
Two fixes:
1. DropZone: handleAnalyzeComplete now passes the repoName through to
connectToServer so the specific newly-analyzed repo loads — not the
server's default first repo.
2. App.tsx: fetchRepos() is now awaited BEFORE handleServerConnect in
both the DropZone and Header flows. This ensures the repo list is
populated before the exploring view renders, so the new repo appears
in the header dropdown immediately without a page reload.
* feat: delete repos, re-analyze with force, select after analysis
Server — DELETE /api/repo:
- Acquires repo lock first (409 if analyze/embed in flight)
- Closes LadybugDB, deletes index + clone dir, unregisters, re-inits
- Lock released in finally block
Server — analyze complete:
- backend.init() must succeed before SSE complete fires
- If backend.init() fails, job is marked failed (not complete)
Web — Header repo dropdown:
- Re-analyze: calls POST /api/analyze with force=true, shows spinning
icon + inline progress bar via SSE
- Delete: acquires lock, aborts any running re-analysis SSE for same
repo, refreshes list, switches to next repo
- After analysis completes: refreshes repo list, connects to the
specific repo by name, loads graph, shows in explorer
- Retry with 1.5s backoff on 404 (server may still be reinitializing)
Type safety:
- err: any → err: unknown + instanceof BackendError in retry loop
- Added missing BackendRepo + BackendError imports in App.tsx
|
||
|
Wojciech Guziak
|
546128cdcb
|
refactor: split global BUILT_IN_NAMES into per-language provider fields (#523)
* refactor: make isBuiltInOrNoise provider-aware, remove global BUILT_IN_NAMES Add builtInNames field to LanguageProviderConfig. Rewrite noise-filter.ts to accept a LanguageProvider and check provider.builtInNames instead of a global Set. Update all 3 call sites to pass their existing provider. Built-in entries will be added per-language in subsequent commits. * refactor(js/ts): add per-language builtInNames to JS/TS providers * refactor(python): add per-language builtInNames * refactor(kotlin): add per-language builtInNames * refactor(c/cpp): add per-language builtInNames * refactor(csharp): add per-language builtInNames * refactor(php): add per-language builtInNames * refactor(swift): add per-language builtInNames * refactor(rust): add per-language builtInNames * refactor(ruby): add per-language builtInNames * refactor(dart): add per-language builtInNames * test: update noise-filter tests for per-language API, add isolation tests - Update ingestion-utils.test.ts to pass provider to isBuiltInOrNoise - Add noise-filter.test.ts with 15 cross-language isolation tests - Fix Java heritage test: serialize() is now correctly unfiltered for Java (was false-positive noise from global PHP serialize entry) * refactor: remove noise-filter.ts, add provider.isBuiltInName() method Per review feedback: delete noise-filter.ts entirely and move the check into LanguageProvider as isBuiltInName(name) method, generated by defineLanguage() from the builtInNames set. Call sites now use provider.isBuiltInName(calledName) directly. |
||
|
Gergő Magyar
|
7999b6ba7b
|
refactor: SICP-informed LanguageProvider architecture (#488)
* refactor: SICP-informed LanguageProvider architecture for ingestion pipeline Consolidate 16 scattered dispatch surfaces into a single LanguageProvider Strategy interface per language. Processors are now fully language-agnostic — zero SupportedLanguages.X enum access, zero dispatch table imports. Architecture (5-layer DAG, zero circular dependencies): L0: Capability modules (dispatch tables, single source of truth) L1: LanguageProvider interface + createLanguageProvider factory L2: 13 per-language provider files (Strategy objects) L3: Registry with satisfies Record<SL, LP> + pre-built lookup maps L4: Processors (language-agnostic, all behavior via provider.*) Key changes: - Add LanguageProvider interface with 15 properties (6 required, 9 optional) - Create 13 provider files in languages/ + php-helpers.ts - Migrate all processors to getProvider(language) — cached once per scope - Replace heritage if-checks with provider.interfaceNamePattern/heritageDefaultEdge - Replace MRO switch(language) with switch(provider.mroStrategy) - Replace isNodeExported with provider.exportChecker - Move PHP description extraction behind provider.descriptionExtractor - Move Swift implicit imports behind provider.implicitImportWirer - Move PHP route detection behind provider.isRouteFile - Move Kotlin wildcard append behind provider.importPathPreprocessor - Remove deprecated TypeEnvironment.env, add fileScope()/allScopes() - De-export TypeEnv type (module-private) - Pre-build extensionMap, WILDCARD_LANGUAGES, SYNTHESIS_LANGUAGES at load - Remove dead entryPointPatterns/frameworkPatterns from interface - Derive createLanguageProvider config type via Pick/Partial/Omit - Tighten callback types from any to SyntaxNode - Migrate 270+ test call sites from .env to TypeEnvironment API Adding a new language: 3 files (enum + provider + registry line). No processor file touched. Ever. * refactor: clean architecture for LanguageProvider with O(1) AST cache Address all PR #488 review comments and achieve pristine SICP layer separation: Interface redesign: - Split LanguageProvider into Config (input) + Provider (runtime with defaults) - Rename createLanguageProvider → defineLanguage with explicit DEFAULTS constant - Add MroStrategy, ImportSemantics named type aliases for better IDE tooltips - Tighten labelOverride signature: string|null → NodeLabel|null (compile-time safety) - Tighten descriptionExtractor nodeLabel: string → NodeLabel - Un-export LanguageProviderConfig (internal to defineLanguage) CI fixes (all 4 failures resolved): - isNodeExported: add null guard for unknown languages - preprocessImportPath tests: pass getProvider() instead of raw enum - MRO tests: update expected strings to match language-agnostic prefixes Code deduplication: - Extract findDescendant/extractStringContent to ast-helpers.ts (single source of truth) - Unify Kotlin method detection: remove duplicate from extractFunctionName, use provider.labelOverride as single source of truth via findEnclosingFunctionId - extractFunctionName return type: string → NodeLabel Performance (O(1) AST node access): - Add per-file Map-based memoization in parse-worker for parent-chain walks - Cache enclosingClassId, enclosingFunctionId, exportStatus per SyntaxNode - Clear caches before each file parse (not after — handles parse failures) Architecture (pristine languages/ folder): - Move php-helpers.ts → helpers/php.ts (L0 capability, not L2 config) - Create helpers/swift.ts from extracted Swift provider logic - Extract cppLabelOverride AST walk → isCppInsideClassOrStruct in ast-helpers.ts - Extract isPhpRouteFile → helpers/php.ts - All 13 provider files are now pure configuration — zero implementation logic - Ruby: remove no-op namedBindingExtractor assignment (undefined from dispatch table) * refactor: eliminate LANGUAGE_QUERIES, typeConfigs, namedBindingExtractors dispatch tables Phase 1 of L0 dispatch table elimination. Providers now import capabilities directly instead of indexing into redundant Record<SL, T> dispatch tables: - LANGUAGE_QUERIES: providers import named query constants directly (TYPESCRIPT_QUERIES, PYTHON_QUERIES, etc.). Table kept in tree-sitter-queries.ts for call-processor.ts dynamic lookup + test consumers. - typeConfigs: providers import from individual type-extractor files (typescriptConfig from typescript.ts, javaTypeConfig from jvm.ts, etc.). Dispatch table fully removed from type-extractors/index.ts. - namedBindingExtractors: providers import extractors directly from named-binding-extraction.ts (extractTsNamedBindings, etc.). Dispatch table fully removed from import-resolution.ts. Net: -48 LOC of dispatch table indirection. L3 satisfies Record<SL, LP> remains the single exhaustiveness check. * refactor: eliminate exportCheckers, callRouters, importResolvers dispatch tables Phase 2 of L0 dispatch table elimination. All 6 dispatch tables are now gone: - exportCheckers: individual checkers exported directly (tsExportChecker, pythonExportChecker, etc.). isNodeExported uses a local checkersByLanguage map to avoid circular dependency with languages/index.ts. - callRouters: table removed. Providers import noRouting or routeRubyCall directly. noRouting now exported. Dead import removed from call-processor.ts. - importResolvers: resolver functions exported with clean names (resolveTypescriptImport, resolveJavaImport, etc.). Inline lambdas extracted to named exports. Dispatch functions renamed from *Dispatch suffix to clean resolve*Import pattern. Combined with Phase 1, all 6 L0 dispatch tables have been eliminated. L3 satisfies Record<SL, LanguageProvider> is the single exhaustiveness check. Providers are now fully self-contained — each imports its capabilities directly. * perf+refactor: type-env caching, sequential fallback caching, utils.ts split Phase 3 — performance optimizations and barrel cleanup: Type-env parent-walk caching: - Memoize findEnclosingClassName and findEnclosingParentClassName with per-file Map<SyntaxNode, string|undefined> caches - Eliminates O(n*m) repeated child scanning in extractParentClassFromNode - Caches cleared in buildTypeEnv before each file's walk phase Sequential fallback caching: - Add classIdCache + exportCache Maps to parsing-processor.ts - Mirrors the O(1) memoization pattern from parse-worker.ts - Both paths now have identical caching for parent-chain walks Split utils.ts barrel into focused modules: - noise-filter.ts: BUILT_IN_NAMES + isBuiltInOrNoise (167 LOC) - language-detection.ts: getLanguageFromFilename (58 LOC) - utils.ts slimmed to re-exports + yieldToEventLoop + isVerboseIngestionEnabled - Backward compatible — existing imports from utils.ts still work * refactor: rename resolvers/ → import-resolvers/, restructure tests per-concern Directory renames (git mv — history preserved): - src/core/ingestion/resolvers/ → import-resolvers/ (10 files) - test/unit/call-routing.test.ts → call-routing/ruby.test.ts - test/unit/named-binding-extraction.test.ts → named-bindings/csharp.test.ts - test/unit/import-resolution.test.ts → import-resolution/preprocessing.test.ts All 11 import paths updated to reference new import-resolvers/ location. Test imports updated for new subdirectory depth. Note: test/integration/resolvers/ NOT renamed — those tests cover the full ingestion pipeline per-language, not just import resolution. * refactor: eliminate utils.ts barrel — all 33 consumers now import directly Migrated 65 import sites across 33 files to import from the focused source module instead of the utils.ts barrel: - ast-helpers.js: SyntaxNode, extractFunctionName, findEnclosingClassId, etc. - call-analysis.js: inferCallForm, extractReceiverName, countCallArguments, etc. - noise-filter.js: BUILT_IN_NAMES, isBuiltInOrNoise - language-detection.js: getLanguageFromFilename utils.ts reduced to 2 original functions only: - yieldToEventLoop - isVerboseIngestionEnabled Zero re-exports remain. Every import is now direct to its source module. * refactor: create utils/ folder, move all shared utilities, delete utils.ts barrel Final phase of module structure migration: - git mv ast-helpers.ts, call-analysis.ts, noise-filter.ts, language-detection.ts → utils/ subdirectory (history preserved) - Extract yieldToEventLoop → utils/event-loop.ts - Extract isVerboseIngestionEnabled → utils/verbose.ts - Delete utils.ts (zero re-exports, zero functions remain) - Update 38 import paths across source and test files The ingestion/ root is now clean — only processors, capability modules, and the pipeline orchestrator live at the top level. All shared utilities are in utils/, all language-specific helpers in helpers/, all import resolvers in import-resolvers/. * refactor: move findChild from import-resolvers/utils.ts to utils/ast-helpers.ts findChild is a generic AST helper (find first named child by type) — it belongs with the other AST traversal utilities, not in the import resolver module. 4 consumers updated to import from utils/ast-helpers.js. * refactor: split named-binding-extraction.ts into per-language files Rename named-binding-extraction.ts → named-binding-processor.ts (git mv, history preserved), keeping only walkBindingChain for re-export chain resolution. 7 per-language extractor functions moved to named-bindings/ subdirectory: - named-bindings/typescript.ts (extractTsNamedBindings — TS + JS) - named-bindings/python.ts (extractPythonNamedBindings) - named-bindings/kotlin.ts (extractKotlinNamedBindings) - named-bindings/rust.ts (extractRustNamedBindings + collectRustBindings) - named-bindings/php.ts (extractPhpNamedBindings) - named-bindings/csharp.ts (extractCsharpNamedBindings) - named-bindings/java.ts (extractJavaNamedBindings) Each provider now imports its binding extractor from the per-language file. * refactor: eliminate import-resolution.ts — distribute to natural homes Split per-language resolvers into import-resolvers/ per-language files and eliminate the import-resolution.ts catch-all module entirely: Per-language resolvers moved to import-resolvers/: - standard.ts: resolveStandard, resolveJavascriptImport, resolveTypescriptImport, resolveCImport, resolveCppImport - jvm.ts: resolveJavaImport, resolveKotlinImport - go.ts: resolveGoImport - csharp.ts: resolveCSharpImport (helper renamed to Internal) - php.ts, python.ts, ruby.ts, rust.ts: same pattern - swift.ts: new file for resolveSwiftImport Types distributed to their concern directories: - import-resolvers/types.ts: ImportResult, ImportConfigs, ResolveCtx, ImportResolverFn - named-bindings/types.ts: NamedBinding, NamedBindingExtractorFn preprocessImportPath moved to import-processor.ts (its primary consumer). import-resolution.ts deleted — zero catch-all modules remain. * refactor: tighten SPR — eliminate re-exports, dead code, type holes, and redundant patterns 12 review findings resolved across the ingestion layer: Type safety: - CallRouter callNode: any → SyntaxNode (closes type hole) - CaptureMap type alias replaces Record<string, any> - providersWithImplicitWiring filter now type-narrowed (removes ! assertions) - Ruby exportChecker: unnecessary as-cast removed, named export created Architecture: - Circular type dependency eliminated (ImportResolutionContext moved to types.ts) - LANGUAGE_QUERIES residual dispatch replaced with provider.treeSitterQueries - noRouting sentinel deleted — callRouter now properly optional on 12 providers - All 6 re-exports from import-processor/pipeline/languages eliminated Pattern cleanup: - Dead checkersByLanguage table + isNodeExported removed from export-detection - 4 duplicated config interfaces consolidated to language-config.ts - extractCsharpNamedBindings → extractCSharpNamedBindings (casing consistency) Simplification: - import-resolvers/index.ts barrel deleted (dead re-exports) - helpers/ inlined into languages/ (php.ts, swift.ts) — 1 directory removed Verified: tsc --noEmit clean, 3837 tests pass, 0 failures. * refactor: address review — remove LANGUAGE_QUERIES table, type-extractors barrel, fix Windows timeout Review comment fixes (github.com/abhigyanpatwari/GitNexus/pull/488#issuecomment-4117817648): 1. LANGUAGE_QUERIES dispatch table removed from tree-sitter-queries.ts — 5 test files migrated to getProvider(lang).treeSitterQueries — eliminates last parallel dispatch surface 2. type-extractors/index.ts barrel deleted — type-env.ts now imports TYPED_PARAMETER_TYPES from shared.js directly 3. Windows CI timeout fix: afterAll cleanup hook in test-indexed-db.ts now passes explicit 120s timeout to prevent KuzuDB C++ destructor hang from hitting vitest's default 30s testTimeout on Windows Verified: tsc --noEmit clean, 3835 tests pass, 0 failures. * refactor: eliminate chained getProvider property access — assign to variable first All getProvider(lang).property calls now follow the pattern: const provider = getProvider(language); const x = provider.property; 5 source files + 4 test files updated (~35 occurrences). This ensures consistent provider variable usage and avoids repeated lookups in hot paths. * refactor: remove last 4 re-exports from import-resolvers, fix stale CaptureMap comment - Remove `export type { TsconfigPaths }` from standard.ts - Remove `export type { GoModuleConfig }` from go.ts - Remove `export type { ComposerConfig }` from php.ts - Remove `export type { CSharpProjectConfig }` from csharp.ts All 4 types are canonically defined in language-config.ts; zero consumers imported via the resolver re-exports. - Fix stale CaptureMap JSDoc: said "Uses any" but type is SyntaxNode | undefined |
||
|
Candido Sales Gomes
|
0999595444
|
feat(ruby): Add Ruby language support for CLI and web (#111) | ||
|
Gergő Magyar
|
7376e92063
|
fix: consolidate C/C++/C#/Rust language support from 6 overlapping PRs (#237)
* fix: consolidate C/C++/C#/Rust language support from 6 overlapping PRs Merges fixes from PRs #163, #170, #178, #216, #227, #234 into a single coherent changeset with shared modules and deduplication. Phase 0 — Pre-merge consolidation: - Extract isNodeExported to shared export-detection.ts module - Extract TREE_SITTER_BUFFER_SIZE to shared constants.ts with adaptive sizing - Consolidate FUNCTION_NODE_TYPES, extractFunctionName, isBuiltInOrNoise from duplicated call-processor.ts and parse-worker.ts into shared utils.ts - Add query compilation smoke tests for all 12 languages Language fixes: - fix(c/cpp): isExported checks static linkage instead of returning false - fix(c/cpp): .h files parsed as C++ (tree-sitter-cpp is superset of C) - fix(c/cpp): expanded entry point patterns (~30 new for C, ~18 for C++) - fix(cpp): add typedef, union, macro, prototype, inline method queries - fix(c#): isExported scans sibling modifiers instead of parent walk - fix(c#): heritage queries use correct base_list AST structure - fix(c#): add framework detection, import resolution, entry point scoring - fix(rust): isExported scans sibling visibility_modifier in declaration - fix(builtins): remove open/read/write/close (real C POSIX syscalls) - fix(buffer): adaptive bufferSize (2x fileSize, 512KB-32MB range) - feat(ts/js): add call_expression query patterns for const assignments Deduplication: - call-processor.ts: -226 lines (uses shared utils) - parse-worker.ts: -320 lines (uses shared utils) - parsing-processor.ts: -156 lines (uses shared export-detection) * perf: fix review findings — hoist Sets, deduplicate DEFINITION_CAPTURE_KEYS - Hoist CSHARP_DECL_TYPES and RUST_DECL_TYPES to module-level constants in export-detection.ts (was allocating new Set on every isNodeExported call) - Extract DEFINITION_CAPTURE_KEYS and getDefinitionNodeFromCaptures to shared utils.ts (was duplicated in parsing-processor.ts and parse-worker.ts) - Pre-compute merged entry point patterns to avoid per-call array spread in calculateEntryPointScore * test: add C, C++, and Tree-sitter buffer size tests * fix: C/C++/Rust review findings + comprehensive test coverage (+72 tests) Source fixes: - Add Rust built-in noise (unwrap, clone, into, collect, panic, etc.) - C++ anonymous namespace → internal linkage (not exported) - Replace .text regex with storage_class_specifier child scan (perf) - Raise file skip threshold from 512KB to 32MB (TREE_SITTER_MAX_BUFFER) - Export TREE_SITTER_MAX_BUFFER from constants.ts - Add C++ double pointer query patterns to CPP_QUERIES - Add C#: record_struct, record_class, file_scoped_namespace to decl types - Add Rust: union_item to visibility scanning set Tests (214 → 286): - ingestion-utils: +24 (Rust/C# noise, pointer/ref/destructor extraction, buffer) - parsing: +36 (real AST C/C++ static/namespace, Rust/C#/Java/PHP/Swift edge cases) - tree-sitter-languages: +12 (query accuracy for C/C++/C#/Rust captures) |
||
|
abhigyanpatwari
|
3431edcea0 |
fix(test): update ingestion-utils test for Kotlin support
Move .kt from unsupported list to supported, add Kotlin test case for .kt and .kts extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
abhigyanpatwari
|
8a100a76d3 |
test: add test suite with vitest (unit + integration + fixtures)
- 59 test files covering unit and integration tests - vitest config with coverage thresholds and fork pooling - Test fixtures (mini-repo + multi-language sample code) - Add vitest + coverage-v8 to devDependencies - Add test scripts (test, test:integration, test:all, test:watch, test:coverage) - Move typescript to devDependencies where it belongs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |