diff --git a/.qoder/quests/codebase-refactoring-analysis.md b/.qoder/quests/codebase-refactoring-analysis.md deleted file mode 100644 index 93ec4ac98..000000000 --- a/.qoder/quests/codebase-refactoring-analysis.md +++ /dev/null @@ -1,2245 +0,0 @@ -# GitNexus Codebase Refactoring Analysis - -## TLDR - What We'll Do - -### 🎯 Main Goals -1. **Dual-Track System**: Keep Legacy (Sequential+InMemory) and Next-Gen (Parallel+KuzuDB) completely separate -2. **Feature Toggle**: Easy switching between engines via config/UI -3. **Monolithic UI Fix**: Break down 700+ line HomePage into focused components -4. **Clean Architecture**: Proper separation of concerns with clear layers - -### 📁 Proposed Directory Structure -``` -src/ -├── config/ -│ ├── feature-flags.ts # Engine switching & feature toggles -│ ├── system-config.ts # Centralized configuration -│ └── engine-config.ts # Engine-specific settings -├── core/ -│ ├── engines/ -│ │ ├── legacy/ -│ │ │ ├── legacy-engine.ts # Wrapper for current system -│ │ │ ├── legacy-storage.ts # SimpleKnowledgeGraph ops -│ │ │ └── legacy-pipeline.ts # Current GraphPipeline -│ │ ├── nextgen/ -│ │ │ ├── nextgen-engine.ts # Wrapper for parallel+kuzu -│ │ │ ├── nextgen-storage.ts # KuzuDB operations -│ │ │ └── nextgen-pipeline.ts # ParallelGraphPipeline -│ │ └── engine-interface.ts # Common interface both engines implement -│ ├── orchestration/ -│ │ ├── engine-manager.ts # Handles engine selection/switching -│ │ ├── processing-orchestrator.ts # Routes to correct engine -│ │ └── performance-monitor.ts # Compare engine performance -│ └── types/ -│ ├── engine-types.ts # Engine-specific types -│ └── processing-types.ts # Shared processing types -├── services/ -│ ├── facade/ -│ │ └── gitnexus-facade.ts # Simplified API for UI -│ ├── github.ts # Keep as-is -│ ├── zip.ts # Keep as-is -│ └── export.ts # Engine-aware export -├── ui/ -│ ├── pages/ -│ │ └── HomePage/ -│ │ ├── HomePage.tsx # Lightweight container (50 lines) -│ │ ├── useHomePage.ts # Extract state logic -│ │ └── HomePage.styles.ts # Extract styles -│ ├── components/ -│ │ ├── engine/ -│ │ │ ├── EngineSelector.tsx # Switch between engines -│ │ │ └── ProcessingStatus.tsx # Engine-aware status -│ │ ├── repository/ -│ │ │ ├── RepositoryInput.tsx # GitHub/ZIP input -│ │ │ └── RepositorySettings.tsx # Filters, options -│ │ ├── graph/ -│ │ │ ├── GraphDisplay.tsx # Graph visualization -│ │ │ └── GraphControls.tsx # Graph interaction -│ │ ├── chat/ -│ │ │ └── ChatInterface.tsx # AI chat (engine-aware) -│ │ └── shared/ -│ │ ├── LoadingSpinner.tsx -│ │ ├── ErrorDisplay.tsx -│ │ └── Modal.tsx -│ ├── hooks/ -│ │ ├── useEngine.ts # Engine management -│ │ ├── useProcessing.ts # Processing state -│ │ ├── useGraph.ts # Graph state -│ │ └── useSettings.ts # Settings management -│ └── contexts/ -│ ├── AppContext.tsx # Global app state -│ └── EngineContext.tsx # Engine state -├── lib/ -│ ├── error-handling/ -│ │ ├── error-types.ts # Standardized errors -│ │ ├── error-handler.ts # Global error handling -│ │ └── error-boundary.tsx # React error boundaries -│ ├── logging/ -│ │ ├── logger.ts # Structured logging -│ │ └── performance-logger.ts # Performance metrics -│ └── utils/ -│ ├── validation.ts # Input validation -│ └── storage-utils.ts # Storage helpers -└── workers/ - ├── legacy/ - │ └── ingestion.worker.ts # Keep current worker - └── nextgen/ - └── kuzu-ingestion.worker.ts # Keep KuzuDB worker -``` - -### ⚙️ Configuration Strategy - -**1. Feature Flags (`config/feature-flags.ts`)** -```typescript -interface FeatureFlags { - processingEngine: 'legacy' | 'nextgen'; // Main engine toggle - enableParallelProcessing: boolean; // Enable parallel features - enableKuzuDB: boolean; // Enable KuzuDB storage - autoFallback: boolean; // Auto fallback on errors - performanceComparison: boolean; // Side-by-side comparison -} - -// Usage: Toggle in UI or environment variable -``` - -**2. System Config (`config/system-config.ts`)** -```typescript -interface SystemConfig { - engines: { - legacy: LegacyEngineConfig; - nextgen: NextGenEngineConfig; - }; - ui: UIConfig; - performance: PerformanceConfig; -} - -// Centralized configuration for all systems -``` - -### 🔧 Monolithic UI Fixes - -**Current Problem**: HomePage.tsx is 700+ lines handling everything - -**Solution**: Break into focused components - -```typescript -// OLD: Monolithic HomePage (700+ lines) -const HomePage = () => { - // 12 different state variables - // Service instantiation - // Error handling - // Settings management - // Processing logic - // UI rendering -}; - -// NEW: Lightweight container (50 lines) -const HomePage = () => { - return ( - - {/* Engine switching */} - {/* GitHub/ZIP input */} - {/* Progress display */} - {/* Graph visualization */} - {/* AI chat */} - - ); -}; - -// Extract logic to custom hooks -const useHomePage = () => { - const engine = useEngine(); - const processing = useProcessing(); - const graph = useGraph(); - const settings = useSettings(); - - return { engine, processing, graph, settings }; -}; -``` - -### 🚀 Implementation Plan (3 Steps) - -**Step 1: Foundation (Week 1)** -- Create directory structure -- Add feature flag system -- Extract engine interfaces -- Basic configuration management - -**Step 2: Engine Separation (Week 2)** -- Wrap current system as "Legacy Engine" -- Wrap parallel+kuzu as "Next-Gen Engine" -- Create engine manager with switching logic -- Add performance monitoring - -**Step 3: UI Refactor (Week 3)** -- Break down HomePage into components -- Add custom hooks for state management -- Create engine selector UI -- Add context providers - -### 🎛️ How Engine Switching Works - -**UI Toggle**: -```typescript - switchEngine(engine)} - options={[ - { value: 'legacy', label: 'Stable (Sequential + In-Memory)' }, - { value: 'nextgen', label: 'Advanced (Parallel + KuzuDB)' } - ]} -/> -``` - -**Behind the Scenes**: -```typescript -// User selects Next-Gen engine -switchEngine('nextgen') - → FeatureFlags.processingEngine = 'nextgen' - → ProcessingOrchestrator routes to NextGenEngine - → ParallelGraphPipeline + KuzuDB storage - → UI shows "🚀 Advanced Pipeline Active" - -// If Next-Gen fails and autoFallback=true -NextGenEngine.process() throws error - → Auto-fallback to LegacyEngine - → UI shows "⚠️ Fallback to Stable Pipeline" - → User gets results, no data loss -``` - -### 📊 What Users See - -1. **Engine Selector**: Dropdown to choose processing engine -2. **Status Indicator**: Shows which engine is active -3. **Performance Metrics**: Compare speed between engines -4. **Automatic Fallback**: Seamless fallback if advanced engine fails -5. **Clean UI**: No more overwhelming 700-line interface - -### 🎯 Key Benefits - -- ✅ **Safe Migration**: Legacy always works, Next-Gen opt-in -- ✅ **No Code Mixing**: Complete separation of engines -- ✅ **Better UX**: Clean, focused UI components -- ✅ **Easy Maintenance**: Clear separation of concerns -- ✅ **Performance Validation**: Real-world engine comparison - ---- - -## Implementation Code - -### Step 1: Enhanced Feature Flags (Update `src/config/feature-flags.ts`) - -Add these types and update the interface: - -```typescript -export type ProcessingEngine = 'legacy' | 'nextgen'; - -// Add to existing FeatureFlags interface: -interface FeatureFlags { - // Processing Engine Selection (ADD THESE) - processingEngine: ProcessingEngine; - autoFallbackOnError: boolean; - enablePerformanceComparison: boolean; - - // ... existing flags remain the same -} - -// Update DEFAULT_FEATURE_FLAGS: -export const DEFAULT_FEATURE_FLAGS: FeatureFlags = { - // Processing Engine - Start with legacy as default for safety - processingEngine: 'legacy', - autoFallbackOnError: true, - enablePerformanceComparison: false, - - // ... rest of existing flags remain the same -}; - -// Add these methods to the existing FeatureFlagManager class: -class FeatureFlagManager { - // ... existing code ... - - /** - * Get the current processing engine - */ - getProcessingEngine(): ProcessingEngine { - return this.flags.processingEngine; - } - - /** - * Switch to legacy engine (sequential + in-memory) - */ - switchToLegacyEngine(): void { - this.setFlags({ - processingEngine: 'legacy', - enableKuzuDB: false, - enableParallelProcessing: false, - enableWorkerPool: false - }); - } - - /** - * Switch to next-gen engine (parallel + KuzuDB) - */ - switchToNextGenEngine(): void { - this.setFlags({ - processingEngine: 'nextgen', - enableKuzuDB: true, - enableParallelProcessing: true, - enableWorkerPool: true - }); - } - - /** - * Check if next-gen engine is enabled - */ - isNextGenEngineEnabled(): boolean { - return this.flags.processingEngine === 'nextgen'; - } - - /** - * Get engine capabilities based on current selection - */ - getEngineCapabilities(): string[] { - if (this.isNextGenEngineEnabled()) { - return ['parallel-processing', 'kuzu-db', 'advanced-queries']; - } else { - return ['sequential-processing', 'in-memory', 'basic-queries']; - } - } -} -``` - -### Step 2: Create Engine Interfaces (`src/core/engines/engine-interface.ts`) - -```typescript -import type { KnowledgeGraph } from '../graph/types'; - -export interface ProcessingInput { - type: 'github' | 'zip'; - url?: string; - file?: File; - options?: { - directoryFilter?: string; - fileExtensions?: string; - useParallelProcessing?: boolean; - maxWorkers?: number; - }; -} - -export interface ProcessingResult { - engine: 'legacy' | 'nextgen'; - graph: KnowledgeGraph; - fileContents: Map; - metadata: { - processingTime: number; - nodeCount: number; - relationshipCount: number; - engineCapabilities: string[]; - }; - // Engine-specific data - kuzuInstance?: any; // For next-gen engine -} - -export interface ProcessingCallbacks { - onEngineSelected?: (engine: 'legacy' | 'nextgen') => void; - onProgress?: (progress: string) => void; - onEngineFailure?: (failed: string, fallback: string) => void; -} - -export interface ProcessingEngine { - readonly name: string; - readonly version: string; - readonly capabilities: string[]; - - process(input: ProcessingInput, callbacks?: ProcessingCallbacks): Promise; - validate(): boolean; - cleanup(): Promise; -} -``` - -### Step 3: Legacy Engine Wrapper (`src/core/engines/legacy/legacy-engine.ts`) - -```typescript -import { IngestionService } from '../../../services/ingestion.service'; -import type { ProcessingEngine, ProcessingInput, ProcessingResult, ProcessingCallbacks } from '../engine-interface'; - -export class LegacyProcessingEngine implements ProcessingEngine { - readonly name = 'legacy'; - readonly version = '1.0'; - readonly capabilities = ['sequential-processing', 'in-memory', 'basic-queries']; - - private ingestionService: IngestionService; - - constructor(githubToken?: string) { - this.ingestionService = new IngestionService(githubToken); - } - - async process(input: ProcessingInput, callbacks?: ProcessingCallbacks): Promise { - callbacks?.onEngineSelected?.('legacy'); - - const startTime = performance.now(); - - try { - let result; - - if (input.type === 'github' && input.url) { - result = await this.ingestionService.processGitHubRepo(input.url, { - directoryFilter: input.options?.directoryFilter, - fileExtensions: input.options?.fileExtensions, - onProgress: callbacks?.onProgress - }); - } else if (input.type === 'zip' && input.file) { - result = await this.ingestionService.processZipFile(input.file, { - directoryFilter: input.options?.directoryFilter, - fileExtensions: input.options?.fileExtensions, - onProgress: callbacks?.onProgress - }); - } else { - throw new Error('Invalid input for legacy engine'); - } - - const endTime = performance.now(); - - return { - engine: 'legacy', - graph: result.graph, - fileContents: result.fileContents, - metadata: { - processingTime: endTime - startTime, - nodeCount: result.graph.nodes.length, - relationshipCount: result.graph.relationships.length, - engineCapabilities: this.capabilities - } - }; - } catch (error) { - console.error('Legacy engine processing failed:', error); - throw error; - } - } - - validate(): boolean { - return true; // Legacy engine is always available - } - - async cleanup(): Promise { - // Legacy engine cleanup if needed - } -} -``` - -### Step 4: Next-Gen Engine Wrapper (`src/core/engines/nextgen/nextgen-engine.ts`) - -```typescript -import { KuzuIngestionService } from '../../../services/kuzu-ingestion.service'; -import type { ProcessingEngine, ProcessingInput, ProcessingResult, ProcessingCallbacks } from '../engine-interface'; - -export class NextGenProcessingEngine implements ProcessingEngine { - readonly name = 'nextgen'; - readonly version = '2.0'; - readonly capabilities = ['parallel-processing', 'kuzu-db', 'advanced-queries', 'performance-monitoring']; - - private kuzuIngestionService: KuzuIngestionService; - - constructor(githubToken?: string) { - this.kuzuIngestionService = new KuzuIngestionService(githubToken); - } - - async process(input: ProcessingInput, callbacks?: ProcessingCallbacks): Promise { - callbacks?.onEngineSelected?.('nextgen'); - - const startTime = performance.now(); - - try { - let result; - - if (input.type === 'github' && input.url) { - result = await this.kuzuIngestionService.processGitHubRepo(input.url, { - directoryFilter: input.options?.directoryFilter, - fileExtensions: input.options?.fileExtensions, - onProgress: callbacks?.onProgress - }); - } else if (input.type === 'zip' && input.file) { - result = await this.kuzuIngestionService.processZipFile(input.file, { - directoryFilter: input.options?.directoryFilter, - fileExtensions: input.options?.fileExtensions, - onProgress: callbacks?.onProgress - }); - } else { - throw new Error('Invalid input for next-gen engine'); - } - - const endTime = performance.now(); - - return { - engine: 'nextgen', - graph: result.graph, - fileContents: result.fileContents, - metadata: { - processingTime: endTime - startTime, - nodeCount: result.graph.nodes.length, - relationshipCount: result.graph.relationships.length, - engineCapabilities: this.capabilities - }, - kuzuInstance: (result as any).kuzuInstance // Type cast for now - }; - } catch (error) { - console.error('Next-gen engine processing failed:', error); - throw error; - } - } - - validate(): boolean { - // Check if KuzuDB and parallel processing are available - return typeof Worker !== 'undefined' && - typeof WebAssembly !== 'undefined'; - } - - async cleanup(): Promise { - // Cleanup KuzuDB connections and worker pools - } -} -``` - -### Step 5: Engine Manager (`src/core/orchestration/engine-manager.ts`) - -```typescript -import { featureFlagManager } from '../../config/feature-flags'; -import { LegacyProcessingEngine } from '../engines/legacy/legacy-engine'; -import { NextGenProcessingEngine } from '../engines/nextgen/nextgen-engine'; -import type { ProcessingEngine, ProcessingInput, ProcessingResult, ProcessingCallbacks } from '../engines/engine-interface'; - -export class EngineManager { - private legacyEngine: LegacyProcessingEngine; - private nextGenEngine: NextGenProcessingEngine; - - constructor(githubToken?: string) { - this.legacyEngine = new LegacyProcessingEngine(githubToken); - this.nextGenEngine = new NextGenProcessingEngine(githubToken); - } - - /** - * Get the current active engine - */ - getCurrentEngine(): ProcessingEngine { - const engineType = featureFlagManager.getProcessingEngine(); - return engineType === 'nextgen' ? this.nextGenEngine : this.legacyEngine; - } - - /** - * Process input with the current engine, with fallback capability - */ - async process(input: ProcessingInput, callbacks?: ProcessingCallbacks): Promise { - const engineType = featureFlagManager.getProcessingEngine(); - - console.info('🎯 ENGINE SELECTION:', { - selectedEngine: engineType, - inputType: input.type, - autoFallbackEnabled: featureFlagManager.getFlag('autoFallbackOnError'), - timestamp: new Date().toISOString() - }); - - try { - if (engineType === 'nextgen') { - return await this.processWithNextGen(input, callbacks); - } else { - return await this.processWithLegacy(input, callbacks); - } - } catch (error) { - return await this.handleEngineFailure(error, input, callbacks); - } - } - - private async processWithNextGen( - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - console.info('🚀 NEXT-GEN ENGINE: Starting advanced processing (Parallel + KuzuDB)'); - - if (!this.nextGenEngine.validate()) { - const validationError = new Error('Next-gen engine validation failed - WebAssembly or Worker support unavailable'); - console.error('❌ VALIDATION FAILED: Next-gen engine cannot run in current environment'); - console.error('🔧 Validation Details:', { - webAssemblySupport: typeof WebAssembly !== 'undefined', - workerSupport: typeof Worker !== 'undefined', - timestamp: new Date().toISOString() - }); - throw validationError; - } - - const startTime = performance.now(); - const result = await this.nextGenEngine.process(input, callbacks); - const endTime = performance.now(); - - console.info('✅ NEXT-GEN SUCCESS: Advanced processing completed successfully'); - console.info('📊 Next-Gen Performance:', { - engine: 'nextgen', - processingTime: endTime - startTime, - nodeCount: result.metadata.nodeCount, - relationshipCount: result.metadata.relationshipCount, - capabilities: result.metadata.engineCapabilities, - timestamp: new Date().toISOString() - }); - - return result; - } - - private async processWithLegacy( - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - console.info('⚡ LEGACY ENGINE: Starting stable processing (Sequential + In-Memory)'); - - const startTime = performance.now(); - const result = await this.legacyEngine.process(input, callbacks); - const endTime = performance.now(); - - console.info('✅ LEGACY SUCCESS: Stable processing completed successfully'); - console.info('📊 Legacy Performance:', { - engine: 'legacy', - processingTime: endTime - startTime, - nodeCount: result.metadata.nodeCount, - relationshipCount: result.metadata.relationshipCount, - capabilities: result.metadata.engineCapabilities, - timestamp: new Date().toISOString() - }); - - return result; - } - - private async handleEngineFailure( - error: Error, - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - const currentEngine = featureFlagManager.getProcessingEngine(); - const autoFallback = featureFlagManager.getFlag('autoFallbackOnError'); - - if (currentEngine === 'nextgen' && autoFallback) { - // Enhanced logging for fallback scenario - console.warn('🔄 ENGINE FALLBACK: Next-gen engine failed, automatically falling back to legacy engine'); - console.warn('📋 Fallback Details:', { - failedEngine: 'nextgen', - fallbackEngine: 'legacy', - error: error.message, - timestamp: new Date().toISOString(), - inputType: input.type, - autoFallbackEnabled: true - }); - - callbacks?.onEngineFailure?.('nextgen', 'legacy'); - - try { - const result = await this.processWithLegacy(input, callbacks); - - // Success log after successful fallback - console.info('✅ FALLBACK SUCCESS: Legacy engine completed processing after next-gen failure'); - console.info('📊 Fallback Result:', { - fallbackEngine: 'legacy', - processingTime: result.metadata.processingTime, - nodeCount: result.metadata.nodeCount, - relationshipCount: result.metadata.relationshipCount, - timestamp: new Date().toISOString() - }); - - return result; - } catch (fallbackError) { - // Critical error: both engines failed - console.error('❌ CRITICAL: Both engines failed! Next-gen failed, then legacy also failed.'); - console.error('🚨 Dual Failure Details:', { - originalError: error.message, - fallbackError: fallbackError.message, - timestamp: new Date().toISOString(), - inputType: input.type - }); - throw new Error(`Both engines failed. Next-gen: ${error.message}, Legacy: ${fallbackError.message}`); - } - } - - // No fallback available or fallback disabled - console.error(`❌ ENGINE FAILURE: ${currentEngine} engine failed and no fallback available`); - console.error('🔧 Failure Details:', { - failedEngine: currentEngine, - error: error.message, - autoFallbackEnabled: autoFallback, - fallbackAvailable: currentEngine === 'nextgen', - timestamp: new Date().toISOString() - }); - - throw error; - } - - /** - * Switch to legacy engine - */ - switchToLegacy(): void { - featureFlagManager.switchToLegacyEngine(); - } - - /** - * Switch to next-gen engine - */ - switchToNextGen(): void { - featureFlagManager.switchToNextGenEngine(); - } - - /** - * Get current engine type - */ - getCurrentEngineType(): 'legacy' | 'nextgen' { - return featureFlagManager.getProcessingEngine(); - } - - /** - * Get engine capabilities - */ - getCurrentEngineCapabilities(): string[] { - return this.getCurrentEngine().capabilities; - } - - /** - * Cleanup both engines - */ - async cleanup(): Promise { - await Promise.all([ - this.legacyEngine.cleanup(), - this.nextGenEngine.cleanup() - ]); - } -} -``` - -### Step 6: GitNexus Facade (`src/services/facade/gitnexus-facade.ts`) - -```typescript -import { EngineManager } from '../../core/orchestration/engine-manager'; -import type { ProcessingInput, ProcessingResult } from '../../core/engines/engine-interface'; - -export interface AnalysisOptions { - directoryFilter?: string; - fileExtensions?: string; - onProgress?: (progress: string) => void; - onEngineSelected?: (engine: 'legacy' | 'nextgen') => void; - onEngineFailure?: (failed: string, fallback: string) => void; -} - -export interface AnalysisResult { - engine: 'legacy' | 'nextgen'; - graph: any; // KnowledgeGraph type - fileContents: Map; - metadata: { - processingTime: number; - nodeCount: number; - relationshipCount: number; - engineCapabilities: string[]; - }; -} - -export class GitNexusFacade { - private engineManager: EngineManager; - - constructor(githubToken?: string) { - this.engineManager = new EngineManager(githubToken); - } - - /** - * Analyze GitHub repository - */ - async analyzeRepository(url: string, options?: AnalysisOptions): Promise { - const input: ProcessingInput = { - type: 'github', - url, - options: { - directoryFilter: options?.directoryFilter, - fileExtensions: options?.fileExtensions - } - }; - - const result = await this.engineManager.process(input, { - onEngineSelected: options?.onEngineSelected, - onProgress: options?.onProgress, - onEngineFailure: options?.onEngineFailure - }); - - return this.convertToAnalysisResult(result); - } - - /** - * Analyze ZIP file - */ - async analyzeZipFile(file: File, options?: AnalysisOptions): Promise { - const input: ProcessingInput = { - type: 'zip', - file, - options: { - directoryFilter: options?.directoryFilter, - fileExtensions: options?.fileExtensions - } - }; - - const result = await this.engineManager.process(input, { - onEngineSelected: options?.onEngineSelected, - onProgress: options?.onProgress, - onEngineFailure: options?.onEngineFailure - }); - - return this.convertToAnalysisResult(result); - } - - /** - * Switch to legacy engine - */ - switchToLegacyEngine(): void { - this.engineManager.switchToLegacy(); - } - - /** - * Switch to next-gen engine - */ - switchToNextGenEngine(): void { - this.engineManager.switchToNextGen(); - } - - /** - * Get current engine type - */ - getCurrentEngine(): 'legacy' | 'nextgen' { - return this.engineManager.getCurrentEngineType(); - } - - /** - * Get current engine capabilities - */ - getCurrentEngineCapabilities(): string[] { - return this.engineManager.getCurrentEngineCapabilities(); - } - - private convertToAnalysisResult(result: ProcessingResult): AnalysisResult { - return { - engine: result.engine, - graph: result.graph, - fileContents: result.fileContents, - metadata: result.metadata - }; - } - - /** - * Cleanup resources - */ - async cleanup(): Promise { - await this.engineManager.cleanup(); - } -} -``` - -### Step 7: Engine Selector UI Component (`src/ui/components/engine/EngineSelector.tsx`) - -```typescript -import React from 'react'; - -interface EngineSelectorProps { - currentEngine: 'legacy' | 'nextgen'; - onEngineChange: (engine: 'legacy' | 'nextgen') => void; - isProcessing?: boolean; - capabilities?: string[]; -} - -export const EngineSelector: React.FC = ({ - currentEngine, - onEngineChange, - isProcessing = false, - capabilities = [] -}) => { - const getEngineIcon = (engine: 'legacy' | 'nextgen') => { - return engine === 'nextgen' ? '🚀' : '⚡'; - }; - - const getEngineLabel = (engine: 'legacy' | 'nextgen') => { - return engine === 'nextgen' - ? 'Advanced (Parallel + KuzuDB)' - : 'Stable (Sequential + In-Memory)'; - }; - - const getEngineStatus = (engine: 'legacy' | 'nextgen') => { - return engine === 'nextgen' ? 'Advanced' : 'Stable'; - }; - - return ( -
-
- - - {getEngineIcon(currentEngine)} {getEngineStatus(currentEngine)} - -
- - - - {capabilities.length > 0 && ( -
- Features: -
- {capabilities.map((capability, index) => ( - - {capability.replace('-', ' ')} - - ))} -
-
- )} - - -
- ); -}; -``` - -### Step 8: Update HomePage to Use New Architecture (`src/ui/pages/HomePage/HomePage.tsx`) - -```typescript -import React, { useState, useCallback } from 'react'; -import { GitNexusFacade } from '../../../services/facade/gitnexus-facade'; -import { EngineSelector } from '../../components/engine/EngineSelector'; -import { GraphExplorer } from '../../components/graph'; -import { ChatInterface } from '../../components/chat'; -import type { KnowledgeGraph } from '../../../core/graph/types'; - -const HomePage: React.FC = () => { - // State - const [facade] = useState(() => new GitNexusFacade()); - const [currentEngine, setCurrentEngine] = useState<'legacy' | 'nextgen'>('legacy'); - const [capabilities, setCapabilities] = useState([]); - const [graph, setGraph] = useState(null); - const [fileContents, setFileContents] = useState>(new Map()); - const [isProcessing, setIsProcessing] = useState(false); - const [progress, setProgress] = useState(''); - const [error, setError] = useState(null); - const [githubUrl, setGithubUrl] = useState(''); - - // Engine management - const handleEngineChange = useCallback((engine: 'legacy' | 'nextgen') => { - if (engine === 'nextgen') { - facade.switchToNextGenEngine(); - } else { - facade.switchToLegacyEngine(); - } - setCurrentEngine(engine); - setCapabilities(facade.getCurrentEngineCapabilities()); - }, [facade]); - - // Repository processing - const handleGitHubProcess = useCallback(async () => { - if (!githubUrl.trim()) { - setError('Please enter a GitHub repository URL'); - return; - } - - setIsProcessing(true); - setError(null); - setProgress('Starting analysis...'); - - try { - const result = await facade.analyzeRepository(githubUrl, { - onEngineSelected: (engine) => { - setCurrentEngine(engine); - setProgress(`🎯 Using ${engine === 'nextgen' ? '🚀 Advanced' : '⚡ Stable'} engine...`); - console.info(`🔄 Engine Selected: ${engine}`); - }, - onProgress: (progressMsg) => { - setProgress(progressMsg); - }, - onEngineFailure: (failed, fallback) => { - const fallbackMsg = `🔄 Engine switched: ${failed} → ${fallback}`; - setProgress(fallbackMsg); - setCurrentEngine(fallback); - - // Show user-friendly notification - console.warn(`📊 ENGINE FALLBACK: Switched from ${failed} to ${fallback} engine`); - - // Optional: Show toast notification to user - if (failed === 'nextgen') { - console.info('💬 User Notice: Advanced engine encountered an issue, using stable engine instead'); - } - } - }); - - setGraph(result.graph); - setFileContents(result.fileContents); - setProgress(''); - } catch (err) { - setError(err instanceof Error ? err.message : 'Processing failed'); - setProgress(''); - } finally { - setIsProcessing(false); - } - }, [githubUrl, facade]); - - // File upload processing - const handleFileUpload = useCallback(async (file: File) => { - setIsProcessing(true); - setError(null); - setProgress('Processing ZIP file...'); - - try { - const result = await facade.analyzeZipFile(file, { - onEngineSelected: (engine) => { - setCurrentEngine(engine); - setProgress(`🎯 Using ${engine === 'nextgen' ? '🚀 Advanced' : '⚡ Stable'} engine...`); - console.info(`🔄 Engine Selected: ${engine}`); - }, - onProgress: (progressMsg) => { - setProgress(progressMsg); - }, - onEngineFailure: (failed, fallback) => { - const fallbackMsg = `🔄 Engine switched: ${failed} → ${fallback}`; - setProgress(fallbackMsg); - setCurrentEngine(fallback); - - // Show user-friendly notification - console.warn(`📊 ENGINE FALLBACK: Switched from ${failed} to ${fallback} engine`); - - // Optional: Show toast notification to user - if (failed === 'nextgen') { - console.info('💬 User Notice: Advanced engine encountered an issue, using stable engine instead'); - } - } - }); - - setGraph(result.graph); - setFileContents(result.fileContents); - setProgress(''); - } catch (err) { - setError(err instanceof Error ? err.message : 'Processing failed'); - setProgress(''); - } finally { - setIsProcessing(false); - } - }, [facade]); - - return ( -
-
-

🔍 GitNexus

- -
- -
-
-
- setGithubUrl(e.target.value)} - placeholder="Enter GitHub repository URL" - disabled={isProcessing} - /> - -
- -
- e.target.files?.[0] && handleFileUpload(e.target.files[0])} - disabled={isProcessing} - /> -
-
- - {progress && ( -
-
{progress}
-
- )} - - {error && ( -
-
{error}
-
- )} - - {graph && ( -
-
- -
-
- -
-
- )} -
-
- ); -}; - -export default HomePage; -``` - -## Implementation Instructions - -1. **Start with Feature Flags**: Update the existing `src/config/feature-flags.ts` with the new engine selection capabilities - -2. **Create Directory Structure**: Create the new directories as shown in the structure above - -3. **Implement Engine Interfaces**: Start with the base interfaces and engine wrappers - -4. **Create Engine Manager**: Implement the orchestration layer for switching between engines - -5. **Build Facade**: Create the simplified GitNexusFacade for the UI layer - -6. **Update UI Components**: Decompose HomePage and add engine selection UI - -7. **Test Thoroughly**: Ensure both engines work independently and switching works correctly - -This approach gives you a clean dual-track system where you can safely develop and test the Next-Gen engine while keeping the Legacy engine stable for production use! 🚀 - -## Overview - -GitNexus is a sophisticated code knowledge graph generator that analyzes repositories and creates interactive visualizations. After deep analysis of the codebase, several architectural inconsistencies and maintainability issues have been identified that require systematic refactoring to improve separation of concerns, reduce coupling, and enhance maintainability. - -## Current Architecture Analysis - -### Repository Type -**Full-Stack Application** with React frontend, TypeScript backend processing, and WebAssembly integration for code parsing. - -### Technology Stack -- **Frontend**: React 18, TypeScript, Vite -- **Visualization**: Cytoscape.js with dagre layout -- **Code Parsing**: WebAssembly (Tree-sitter) -- **Concurrency**: Web Workers with Comlink -- **Build Tools**: Vite with React plugin -- **Testing**: Jest for unit tests - -### Current Architecture Issues - -#### 1. Dual Processing Systems (Migration Strategy) -The codebase currently has **two complete processing systems** as part of a planned migration strategy: - -```mermaid -graph TD - A[Input Data] --> B[Feature Toggle] - B --> C[Legacy System] - B --> D[Next-Gen System] - C --> E[SimpleKnowledgeGraph] - D --> F[KuzuDB] - C --> G[Sequential Pipeline] - D --> H[Parallel Pipeline] - G --> I[IngestionWorker] - H --> J[KuzuIngestionWorker] - - style C fill:#ffeaa7 - style D fill:#00b894 - style E fill:#ffeaa7 - style F fill:#00b894 -``` - -**Current Situation:** -- Legacy: Sequential processing with in-memory storage -- Next-Gen: Parallel processing with KuzuDB storage -- Migration Goal: Gradually replace legacy with next-gen -- Challenge: Keep systems completely separate during transition - -#### 2. Inconsistent Service Layer Responsibilities -Services have overlapping concerns and unclear boundaries: - -```mermaid -graph LR - A[IngestionService] --> B[GitHubService] - A --> C[ZipService] - A --> D[IngestionWorker] - E[KuzuIngestionService] --> B - E --> C - E --> F[KuzuIngestionWorker] - - style A fill:#ffeecc - style E fill:#ffeecc -``` - -**Problems:** -- Duplicate orchestration logic -- Mixed responsibilities (data fetching + orchestration) -- Tight coupling between services and workers - -#### 3. Monolithic UI Component -The `HomePage.tsx` component has grown into a 700+ line monolith handling: -- State management (12 different state properties) -- Service orchestration -- UI rendering -- Error handling -- Settings management - -#### 4. Inconsistent Error Handling -Error handling patterns vary across layers: -- Some components use try-catch -- Others rely on error boundaries -- Services have different error propagation strategies - -#### 5. Mixed Abstraction Levels -The codebase mixes low-level implementation details with high-level business logic: -- UI components directly instantiate services -- Processing logic mixed with data access -- Configuration scattered across modules - -## Proposed Refactored Architecture - -### High-Level Architecture - -```mermaid -graph TB - subgraph "Presentation Layer" - UI[React Components] - STATE[State Management] - HOOKS[Custom Hooks] - end - - subgraph "Application Layer" - FACADE[Facade Services] - ORCHESTRATOR[Processing Orchestrator] - CONFIG[Configuration Manager] - end - - subgraph "Domain Layer" - PIPELINE[Pipeline Abstractions] - GRAPH[Graph Domain Models] - PROCESSING[Processing Strategies] - end - - subgraph "Infrastructure Layer" - SERVICES[External Services] - WORKERS[Web Workers] - STORAGE[Storage Adapters] - end - - UI --> FACADE - FACADE --> ORCHESTRATOR - ORCHESTRATOR --> PIPELINE - PIPELINE --> PROCESSING - PROCESSING --> SERVICES - PROCESSING --> WORKERS -``` - -### 1. Unified Processing Pipeline Architecture - -Replace dual systems with a single, configurable pipeline: - -```mermaid -graph TD - A[Repository Input] --> B[Data Acquisition Strategy] - B --> C[Processing Pipeline] - C --> D[Storage Strategy] - D --> E[Knowledge Graph Output] - - B --> B1[GitHub Strategy] - B --> B2[ZIP Strategy] - B --> B3[Archive Strategy] - - C --> C1[Structure Processor] - C --> C2[Parsing Processor] - C --> C3[Import Processor] - C --> C4[Call Processor] - - D --> D1[In-Memory Storage] - D --> D2[KuzuDB Storage] - - style B fill:#e1f5fe - style C fill:#f3e5f5 - style D fill:#e8f5e8 -``` - -**Key Principles:** -- Strategy pattern for data acquisition -- Template method for processing pipeline -- Adapter pattern for storage backends -- Factory pattern for component creation - -### 2. Dual-Track Service Architecture - -```mermaid -graph TB - subgraph "Facade Layer" - AF[Application Facade] - FT[Feature Toggle Manager] - end - - subgraph "Legacy Track" - LPO[Legacy Processing Orchestrator] - LWO[Legacy Worker Orchestrator] - LES[Legacy Export Service] - end - - subgraph "Next-Gen Track" - NPO[NextGen Processing Orchestrator] - NWO[NextGen Worker Orchestrator] - NES[NextGen Export Service] - end - - subgraph "Shared Services" - GS[GitHub Service] - ZS[ZIP Service] - AS[Archive Service] - CS[Configuration Service] - end - - AF --> FT - FT --> LPO - FT --> NPO - LPO --> GS - NPO --> GS - LPO --> ZS - NPO --> ZS - - style LPO fill:#ffeaa7 - style NPO fill:#00b894 -``` - -### 3. Component-Based UI Architecture - -Break down the monolithic HomePage into focused components: - -```mermaid -graph TB - subgraph "Page Level" - HP[HomePage Container] - end - - subgraph "Feature Components" - RC[Repository Input Component] - PC[Processing Status Component] - GC[Graph Display Component] - CC[Chat Component] - SC[Settings Component] - end - - subgraph "Shared Components" - EC[Error Boundary] - LC[Loading Component] - MC[Modal Component] - NC[Notification Component] - end - - subgraph "Hooks" - UPH[useProcessing Hook] - USH[useSettings Hook] - UGH[useGraph Hook] - end - - HP --> RC - HP --> PC - HP --> GC - HP --> CC - HP --> SC - RC --> UPH - PC --> UPH - GC --> UGH - SC --> USH -``` - -## Detailed Refactoring Plan - -### Phase 1: Foundation Layer Refactoring - -#### 1.1 Configuration Management System -Create centralized configuration management: - -```typescript -interface SystemConfiguration { - processing: ProcessingConfig; - storage: StorageConfig; - ui: UIConfig; - api: ApiConfig; -} - -class ConfigurationManager { - private static instance: ConfigurationManager; - private config: SystemConfiguration; - - static getInstance(): ConfigurationManager; - getProcessingConfig(): ProcessingConfig; - getStorageConfig(): StorageConfig; - updateConfig(partial: Partial): void; -} -``` - -#### 1.2 Error Handling System -Implement consistent error handling: - -```typescript -abstract class ApplicationError extends Error { - abstract code: string; - abstract severity: 'low' | 'medium' | 'high' | 'critical'; -} - -class ErrorHandler { - static handle(error: ApplicationError): ErrorResponse; - static recover(error: ApplicationError): RecoveryAction; - static report(error: ApplicationError): void; -} - -interface ErrorBoundaryConfig { - fallbackComponent: React.ComponentType; - onError: (error: Error) => void; - retryStrategy: RetryStrategy; -} -``` - -#### 1.3 Logging and Monitoring -Add structured logging: - -```typescript -interface LogEntry { - level: 'debug' | 'info' | 'warn' | 'error'; - timestamp: number; - component: string; - message: string; - metadata?: Record; -} - -class Logger { - static debug(component: string, message: string, metadata?: any): void; - static info(component: string, message: string, metadata?: any): void; - static warn(component: string, message: string, metadata?: any): void; - static error(component: string, message: string, metadata?: any): void; -} -``` - -### Phase 2: Dual-Track Domain Architecture - -#### 2.1 Feature-Toggle Pipeline Architecture -Maintain separate pipelines with clean toggling mechanism: - -```typescript -interface ProcessingEngine { - name: string; - version: string; - process(input: ProcessingInput): Promise; - validate(): boolean; - cleanup(): Promise; -} - -class LegacyProcessingEngine implements ProcessingEngine { - name = 'legacy'; - version = '1.0'; - - async process(input: ProcessingInput): Promise { - // Current sequential pipeline with SimpleKnowledgeGraph - const pipeline = new GraphPipeline(); - return pipeline.run(input); - } -} - -class NextGenProcessingEngine implements ProcessingEngine { - name = 'nextgen'; - version = '2.0'; - - async process(input: ProcessingInput): Promise { - // New parallel pipeline with KuzuDB - const pipeline = new ParallelGraphPipeline(); - return pipeline.run(input); - } -} - -class ProcessingEngineManager { - private engines: Map; - - constructor(private featureFlags: FeatureFlagService) { - this.engines = new Map([ - ['legacy', new LegacyProcessingEngine()], - ['nextgen', new NextGenProcessingEngine()] - ]); - } - - async process(input: ProcessingInput): Promise { - const engineType = this.featureFlags.getProcessingEngine(); - const engine = this.engines.get(engineType); - - if (!engine) { - throw new Error(`Unknown processing engine: ${engineType}`); - } - - return engine.process(input); - } -} -``` - -#### 2.2 Dual Storage Architecture -Maintain separate storage implementations without abstraction leakage: - -```typescript -// Legacy storage (current implementation) -interface LegacyStorageOps { - createInMemoryGraph(): SimpleKnowledgeGraph; - exportToJSON(graph: SimpleKnowledgeGraph): GraphExport; - queryInMemory(graph: SimpleKnowledgeGraph, query: string): QueryResult; -} - -// Next-gen storage (KuzuDB implementation) -interface NextGenStorageOps { - initializeKuzuDB(): Promise; - persistToKuzu(graph: KuzuKnowledgeGraph): Promise; - queryKuzu(query: string): Promise; - exportFromKuzu(): Promise; -} - -// Storage factory based on processing engine -class StorageFactory { - static createLegacyStorage(): LegacyStorageOps { - return new InMemoryStorageOps(); - } - - static createNextGenStorage(): NextGenStorageOps { - return new KuzuStorageOps(); - } -} - -// Engine-specific result types -type LegacyProcessingResult = { - engine: 'legacy'; - graph: SimpleKnowledgeGraph; - fileContents: Map; -}; - -type NextGenProcessingResult = { - engine: 'nextgen'; - graph: KuzuKnowledgeGraph; - fileContents: Map; - kuzuInstance: KuzuDBInstance; -}; - -type ProcessingResult = LegacyProcessingResult | NextGenProcessingResult; -``` - -#### 2.3 Graph Domain Models -Enhance graph abstractions: - -```typescript -interface GraphOperations { - addNode(node: GraphNode): void; - addRelationship(relationship: Relationship): void; - findNode(predicate: (node: GraphNode) => boolean): GraphNode | null; - findRelationships(nodeId: string): Relationship[]; - validateIntegrity(): ValidationResult; -} - -class KnowledgeGraphBuilder { - private operations: GraphOperations; - - constructor(storage: StorageAdapter) {} - - build(input: ProcessingInput): Promise; - validate(): ValidationResult; - optimize(): OptimizationResult; -} -``` - -### Phase 3: Application Layer Refactoring - -#### 3.1 Feature Flag System -Create a robust feature flagging system to safely toggle between processing engines: - -```typescript -interface FeatureFlags { - processingEngine: 'legacy' | 'nextgen'; - enableParallelProcessing: boolean; - enableKuzuDB: boolean; - enablePerformanceComparison: boolean; - rollbackOnError: boolean; -} - -class FeatureFlagService { - private flags: FeatureFlags; - - constructor() { - this.flags = this.loadFromStorage(); - } - - getProcessingEngine(): 'legacy' | 'nextgen' { - return this.flags.processingEngine; - } - - enableNextGenEngine(): void { - this.flags.processingEngine = 'nextgen'; - this.flags.enableParallelProcessing = true; - this.flags.enableKuzuDB = true; - this.persistFlags(); - } - - fallbackToLegacy(): void { - this.flags.processingEngine = 'legacy'; - this.flags.enableParallelProcessing = false; - this.flags.enableKuzuDB = false; - this.persistFlags(); - } - - isNextGenEnabled(): boolean { - return this.flags.processingEngine === 'nextgen'; - } - - shouldComparePerformance(): boolean { - return this.flags.enablePerformanceComparison; - } -} -``` - -#### 3.2 Dual-Track Processing Orchestrator -Orchestrate both systems while keeping them isolated: - -```typescript -class DualTrackProcessingOrchestrator { - constructor( - private featureFlags: FeatureFlagService, - private legacyEngine: LegacyProcessingEngine, - private nextGenEngine: NextGenProcessingEngine, - private performanceMonitor: PerformanceMonitor - ) {} - - async process( - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - const engine = this.featureFlags.getProcessingEngine(); - - try { - if (engine === 'nextgen') { - return await this.processWithNextGen(input, callbacks); - } else { - return await this.processWithLegacy(input, callbacks); - } - } catch (error) { - return await this.handleEngineFailure(error, input, callbacks); - } - } - - private async processWithNextGen( - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - callbacks?.onEngineSelected?.('nextgen'); - - const startTime = performance.now(); - const result = await this.nextGenEngine.process(input); - const endTime = performance.now(); - - this.performanceMonitor.recordNextGenPerformance(endTime - startTime); - - return { - engine: 'nextgen', - graph: result.graph as KuzuKnowledgeGraph, - fileContents: result.fileContents, - kuzuInstance: result.kuzuInstance - }; - } - - private async processWithLegacy( - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - callbacks?.onEngineSelected?.('legacy'); - - const startTime = performance.now(); - const result = await this.legacyEngine.process(input); - const endTime = performance.now(); - - this.performanceMonitor.recordLegacyPerformance(endTime - startTime); - - return { - engine: 'legacy', - graph: result.graph as SimpleKnowledgeGraph, - fileContents: result.fileContents - }; - } - - private async handleEngineFailure( - error: Error, - input: ProcessingInput, - callbacks?: ProcessingCallbacks - ): Promise { - const currentEngine = this.featureFlags.getProcessingEngine(); - - if (currentEngine === 'nextgen' && this.featureFlags.flags.rollbackOnError) { - callbacks?.onEngineFailure?.('nextgen', 'legacy'); - return this.processWithLegacy(input, callbacks); - } - - throw error; - } -} -``` - -#### 3.3 GitNexus Facade with Engine Awareness -Provide a clean interface that handles engine selection transparently: - -```typescript -class GitNexusFacade { - constructor( - private orchestrator: DualTrackProcessingOrchestrator, - private featureFlags: FeatureFlagService, - private exportService: ExportService - ) {} - - async analyzeRepository( - url: string, - options?: AnalysisOptions - ): Promise { - const result = await this.orchestrator.process( - { type: 'github', url, options }, - { - onEngineSelected: (engine) => options?.onEngineSelected?.(engine), - onProgress: (progress) => options?.onProgress?.(progress) - } - ); - - return this.createAnalysisResult(result); - } - - async analyzeZipFile( - file: File, - options?: AnalysisOptions - ): Promise { - const result = await this.orchestrator.process( - { type: 'zip', file, options }, - { - onEngineSelected: (engine) => options?.onEngineSelected?.(engine), - onProgress: (progress) => options?.onProgress?.(progress) - } - ); - - return this.createAnalysisResult(result); - } - - async exportGraph(format: ExportFormat): Promise { - // Export handling aware of current engine - const engine = this.featureFlags.getProcessingEngine(); - return this.exportService.export(format, engine); - } - - async queryGraph(query: string): Promise { - // Query handling based on current storage type - const engine = this.featureFlags.getProcessingEngine(); - if (engine === 'nextgen') { - return this.queryKuzuGraph(query); - } else { - return this.queryInMemoryGraph(query); - } - } - - // Engine management methods - switchToNextGen(): void { - this.featureFlags.enableNextGenEngine(); - } - - switchToLegacy(): void { - this.featureFlags.fallbackToLegacy(); - } - - getCurrentEngine(): 'legacy' | 'nextgen' { - return this.featureFlags.getProcessingEngine(); - } - - private createAnalysisResult(result: ProcessingResult): AnalysisResult { - return { - engine: result.engine, - graph: result.graph, - fileContents: result.fileContents, - metadata: { - processingTime: performance.now(), - nodeCount: result.graph.nodes.length, - relationshipCount: result.graph.relationships.length - } - }; - } -} -``` - -### Phase 4: Presentation Layer Refactoring - -#### 4.1 Engine-Aware Component Decomposition -Break down HomePage with engine awareness: - -```typescript -// Container component with engine management -const HomePage: React.FC = () => { - const processing = useProcessing(); - const graph = useGraph(); - const settings = useSettings(); - const engineManager = useEngineManager(); - - return ( - - - - - - - - - ); -}; - -// Engine selector component -const EngineSelector: React.FC = ({ - currentEngine, - onEngineChange, - isProcessing -}) => { - return ( -
- - - - {currentEngine === 'nextgen' ? '🚀 Advanced' : '⚡ Stable'} - -
- ); -}; - -// Processing status with engine awareness -const ProcessingStatus: React.FC = ({ status, engine }) => { - const getEngineIcon = () => engine === 'nextgen' ? '🔬' : '⚙️'; - const getEngineLabel = () => engine === 'nextgen' ? 'Next-Gen Pipeline' : 'Legacy Pipeline'; - - return ( -
-
- {getEngineIcon()} {getEngineLabel()} -
-
- {/* Status details specific to engine */} -
-
- ); -}; -``` - -#### 4.2 Engine-Aware Custom Hooks -Extract state management with engine awareness: - -```typescript -const useEngineManager = () => { - const [currentEngine, setCurrentEngine] = useState<'legacy' | 'nextgen'>('legacy'); - const [featureFlags, setFeatureFlags] = useState(); - const featureFlagService = useMemo(() => new FeatureFlagService(), []); - - const switchEngine = useCallback((engine: 'legacy' | 'nextgen') => { - if (engine === 'nextgen') { - featureFlagService.enableNextGenEngine(); - } else { - featureFlagService.fallbackToLegacy(); - } - setCurrentEngine(engine); - }, [featureFlagService]); - - const getEngineCapabilities = useCallback(() => { - return currentEngine === 'nextgen' - ? ['parallel-processing', 'kuzu-db', 'advanced-queries'] - : ['sequential-processing', 'in-memory', 'basic-queries']; - }, [currentEngine]); - - return { - current: currentEngine, - switch: switchEngine, - capabilities: getEngineCapabilities(), - settings: featureFlags - }; -}; - -const useProcessing = () => { - const [status, setStatus] = useState('idle'); - const [progress, setProgress] = useState(null); - const [error, setError] = useState(null); - const [activeEngine, setActiveEngine] = useState<'legacy' | 'nextgen' | null>(null); - - const start = useCallback(async (input: ProcessingInput) => { - try { - setStatus('running'); - const facade = new GitNexusFacade(/* dependencies */); - - await facade.analyzeRepository(input.url, { - onEngineSelected: (engine) => setActiveEngine(engine), - onProgress: (progress) => setProgress(progress), - onEngineFailure: (failed, fallback) => { - console.warn(`Engine ${failed} failed, falling back to ${fallback}`); - setActiveEngine(fallback); - } - }); - - setStatus('completed'); - } catch (err) { - setStatus('error'); - setError(err.message); - } - }, []); - - return { - status, - progress, - error, - activeEngine, - start, - isActive: status === 'running' - }; -}; - -const useGraph = () => { - const [graph, setGraph] = useState(null); - const [selectedNode, setSelectedNode] = useState(null); - const [engine, setEngine] = useState<'legacy' | 'nextgen'>('legacy'); - - const selectNode = useCallback((nodeId: string) => { - setSelectedNode(nodeId); - }, []); - - const updateGraph = useCallback((newGraph: ProcessingResult) => { - setGraph(newGraph.graph); - setEngine(newGraph.engine); - }, []); - - return { - current: graph, - selectedNode, - engine, - selectNode, - updateGraph - }; -}; -``` - -#### 4.3 Dual-Engine Context Providers -Manage dual-engine state globally: - -```typescript -interface AppContextType { - facade: GitNexusFacade; - engineManager: EngineManager; - config: SystemConfiguration; - theme: ThemeConfiguration; -} - -interface EngineContextType { - currentEngine: 'legacy' | 'nextgen'; - featureFlags: FeatureFlags; - switchEngine: (engine: 'legacy' | 'nextgen') => void; - performanceMetrics: PerformanceMetrics; -} - -const AppContext = React.createContext(null); -const EngineContext = React.createContext(null); - -const AppProvider: React.FC<{ children: React.ReactNode }> = ({ children }) => { - const facade = useMemo(() => createGitNexusFacade(), []); - const engineManager = useMemo(() => new EngineManager(), []); - const config = useConfiguration(); - const theme = useTheme(); - - return ( - - - {children} - - - ); -}; - -const EngineProvider: React.FC<{ children: React.ReactNode }> = ({ children }) => { - const [currentEngine, setCurrentEngine] = useState<'legacy' | 'nextgen'>('legacy'); - const [featureFlags, setFeatureFlags] = useState(defaultFlags); - const [performanceMetrics, setPerformanceMetrics] = useState({}); - - const switchEngine = useCallback((engine: 'legacy' | 'nextgen') => { - setCurrentEngine(engine); - // Update feature flags based on engine - const updatedFlags = engine === 'nextgen' - ? { ...featureFlags, processingEngine: 'nextgen', enableKuzuDB: true } - : { ...featureFlags, processingEngine: 'legacy', enableKuzuDB: false }; - setFeatureFlags(updatedFlags); - }, [featureFlags]); - - return ( - - {children} - - ); -}; - -// Custom hooks for contexts -const useAppContext = () => { - const context = useContext(AppContext); - if (!context) throw new Error('useAppContext must be used within AppProvider'); - return context; -}; - -const useEngineContext = () => { - const context = useContext(EngineContext); - if (!context) throw new Error('useEngineContext must be used within EngineProvider'); - return context; -}; -``` - -## Testing Strategy - -### Unit Testing -- **Component Testing**: React Testing Library for UI components -- **Hook Testing**: Custom hook testing utilities -- **Service Testing**: Mock external dependencies -- **Pipeline Testing**: Test processing strategies independently - -### Integration Testing -- **API Integration**: Test GitHub/ZIP service integration -- **Worker Integration**: Test worker communication -- **Storage Integration**: Test storage adapter implementations - -### End-to-End Testing -- **User Workflows**: Test complete analysis workflows -- **Error Scenarios**: Test error handling and recovery -- **Performance Testing**: Validate processing performance - -## Dual-Track Migration Strategy - -### Phase 1: Foundation & Feature Flags (Weeks 1-2) -1. **Feature Flag System**: Implement robust feature flagging for engine selection -2. **Configuration Management**: Centralized config with engine-aware settings -3. **Error Handling**: Consistent error handling across both engines -4. **Logging Infrastructure**: Structured logging with engine identification -5. **Testing Setup**: Dual-track testing infrastructure - -### Phase 2: Engine Isolation & Interfaces (Weeks 3-4) -1. **Engine Abstraction**: Create ProcessingEngine interface -2. **Legacy Engine Wrapper**: Wrap existing sequential pipeline -3. **Next-Gen Engine Wrapper**: Wrap parallel/KuzuDB pipeline -4. **Storage Isolation**: Separate storage interfaces without shared abstractions -5. **Performance Monitoring**: Add metrics collection for both engines - -### Phase 3: Orchestration Layer (Weeks 5-6) -1. **Engine Manager**: Implement engine selection and switching logic -2. **Dual-Track Orchestrator**: Route requests to appropriate engine -3. **Failure Handling**: Automatic fallback mechanisms -4. **Export Compatibility**: Handle exports from both storage types -5. **Query Routing**: Route queries based on active storage type - -### Phase 4: UI Integration (Weeks 7-8) -1. **Engine Selector Component**: UI for switching between engines -2. **Status Indicators**: Show which engine is active -3. **Component Updates**: Make components engine-aware -4. **Custom Hooks**: Add engine management hooks -5. **Context Providers**: Dual-engine state management - -### Phase 5: Testing & Validation (Weeks 9-10) -1. **Comprehensive Testing**: Test both engines independently -2. **Switching Testing**: Validate engine switching functionality -3. **Performance Comparison**: Side-by-side performance analysis -4. **Regression Testing**: Ensure legacy engine still works -5. **Documentation**: Update docs for dual-engine system - -### Phase 6: Gradual Rollout (Weeks 11-12) -1. **Default Engine Selection**: Keep legacy as default initially -2. **Beta Testing**: Allow users to opt into next-gen engine -3. **Performance Monitoring**: Monitor both engines in production -4. **Issue Resolution**: Fix any issues found during rollout -5. **Feedback Collection**: Gather user feedback on both engines - -### Phase 7: Migration Completion (Future) -1. **Next-Gen as Default**: Switch default to next-gen engine -2. **Legacy Deprecation**: Begin deprecation process for legacy -3. **Feature Parity**: Ensure next-gen has all legacy features -4. **Legacy Removal**: Eventually remove legacy engine code -5. **Architecture Cleanup**: Clean up dual-track infrastructure - -## Expected Benefits of Dual-Track Approach - -### Safe Migration -- **Zero Downtime**: Users can continue using stable legacy system -- **Gradual Rollout**: Next-gen features can be tested incrementally -- **Quick Rollback**: Easy fallback if issues are discovered -- **Risk Mitigation**: Reduces risk of breaking existing functionality - -### Development Velocity -- **Parallel Development**: Teams can work on both systems simultaneously -- **Feature Validation**: New features can be tested without disrupting legacy -- **Performance Comparison**: Real-world performance data from both engines -- **User Choice**: Power users can opt into advanced features early - -### System Reliability -- **Fault Tolerance**: Automatic fallback to legacy on next-gen failures -- **Battle Testing**: Next-gen system gets real-world testing before full rollout -- **Stability Preservation**: Legacy system remains untouched during migration -- **Confidence Building**: Team and users gain confidence in next-gen system - -### Technical Benefits -- **Clean Separation**: No mixing of legacy and next-gen code -- **Independent Evolution**: Each system can evolve independently -- **Clear Boundaries**: Well-defined interfaces between systems -- **Easier Maintenance**: Clear separation makes debugging easier - -## Risk Assessment - -### High Risk -- **Breaking Changes**: Significant architectural changes -- **Migration Complexity**: Large codebase refactoring -- **Feature Regression**: Potential loss of existing functionality - -### Medium Risk -- **Performance Impact**: Initial performance degradation during migration -- **Team Learning Curve**: New architectural patterns - -### Low Risk -- **Configuration Issues**: Minor configuration adjustments needed -- **Documentation Updates**: Updating existing documentation - -## Success Metrics for Dual-Track Migration - -### Engine Isolation Quality -- **Code Separation**: 0% shared code between engines (except interfaces) -- **Independent Testing**: Both engines testable in isolation -- **Clean Switching**: Engine switching without data loss or corruption -- **Interface Compliance**: Both engines implement same interfaces correctly - -### Migration Safety -- **Legacy Stability**: No regressions in legacy engine performance -- **Fallback Reliability**: 100% successful fallbacks when next-gen fails -- **Data Integrity**: No data loss during engine switching -- **Feature Parity**: Next-gen engine matches legacy functionality - -### Performance Comparison -- **Processing Speed**: Next-gen engine shows improved performance -- **Memory Usage**: KuzuDB storage provides better memory efficiency -- **Scalability**: Parallel processing handles larger repositories better -- **Resource Utilization**: Better CPU and memory utilization in next-gen - -### Developer Experience -- **Feature Flag Usability**: Easy engine switching for developers -- **Debugging Clarity**: Clear indication of which engine is active -- **Development Speed**: Faster feature development in next-gen engine -- **Code Maintainability**: Easier maintenance due to clean separation - -### User Adoption -- **Opt-in Rate**: Percentage of users trying next-gen engine -- **User Satisfaction**: User feedback on next-gen vs legacy -- **Issue Reports**: Number of issues reported for each engine -- **Performance Perception**: User-perceived performance improvements \ No newline at end of file diff --git a/.qoder/quests/separation-analysis-ill-help-you-refactor-the-codebase-fo....md b/.qoder/quests/separation-analysis-ill-help-you-refactor-the-codebase-fo....md deleted file mode 100644 index d1cf53340..000000000 --- a/.qoder/quests/separation-analysis-ill-help-you-refactor-the-codebase-fo....md +++ /dev/null @@ -1,1022 +0,0 @@ -# GitNexus Codebase Separation Analysis & Refactoring Design - -## Overview - -This document provides a comprehensive analysis of the GitNexus codebase to implement clear separation of concerns while maintaining complete backward compatibility. The project currently implements a dual-track architecture with Legacy (sequential + in-memory) and Next-Gen (parallel + KuzuDB) processing engines. - -**Key Requirements:** -- Maintain current functionality and UI appearance -- Enable seamless switching between processing modes via configuration -- Create granular, maintainable code structure -- Separate Legacy and Next-Gen implementations for independent development - -## Technology Stack & Dependencies - -**Frontend:** React, TypeScript, Vite -**State Management:** React hooks and context -**Visualization:** Cytoscape.js with dagre layout -**Code Parsing:** WebAssembly (Tree-sitter) -**Database:** KuzuDB (Next-Gen), In-Memory Objects (Legacy) -**Workers:** Web Workers for background processing -**Build:** Vite with React plugin - -## Current Architecture Analysis - -### 1. Existing Dual-Track System - -The project already implements engine separation through: - -``` -src/core/engines/ -├── engine-interface.ts # Common interface for both engines -├── legacy/legacy-engine.ts # Sequential + in-memory wrapper -└── nextgen/nextgen-engine.ts # Parallel + KuzuDB wrapper -``` - -**Strengths:** -- Clean engine interface abstraction -- Proper fallback mechanisms -- Performance monitoring for both tracks -- Engine validation and health checks - -**Areas for Improvement:** -- Service layer still mixed between implementations -- UI components tightly coupled to specific engine details -- Configuration system needs engine-specific sections - -### 2. Service Layer Architecture - -Current services show mixed concerns: - -``` -src/services/ -├── facade/gitnexus-facade.ts # UI facade (good separation) -├── ingestion.service.ts # Legacy pipeline service -├── kuzu-ingestion.service.ts # Next-Gen pipeline service -├── github.ts # GitHub API (shared) -├── zip.ts # ZIP processing (shared) -└── kuzu.service.ts # KuzuDB operations -``` - -**Issues Identified:** -- Shared GitHub/ZIP services have engine-specific logic embedded -- No clear service factory pattern for engine selection -- Ingestion services duplicate similar functionality - -### 3. Core Processing Components - -``` -src/core/ -├── graph/ # Knowledge graph interfaces -├── ingestion/ # Processing pipeline components -├── kuzu/ # KuzuDB-specific components -└── orchestration/ # Engine management -``` - -**Pipeline Separation Analysis:** - -**Legacy Pipeline:** Uses `GraphPipeline` → Sequential processing → `SimpleKnowledgeGraph` -**Next-Gen Pipeline:** Uses `KuzuGraphPipeline` → Parallel processing → `KuzuKnowledgeGraph` - -**Current Issues:** -- Processors are not clearly separated by engine type -- Some processors work for both engines, causing complexity -- Import/Call resolution logic is duplicated - -### 4. UI Component Structure - -``` -src/ui/ -├── components/ -│ ├── engine/ # Engine selection components -│ ├── graph/ # Visualization components -│ └── chat/ # AI interface -├── hooks/ # State management -└── pages/ # Main application pages -``` - -**Current State:** -- Good separation in components -- Hooks properly abstracted -- Engine selector components exist -- Processing status components available - -## Detailed Analysis Results - -### 1. Service Layer Issues - Critical Separation Needed - -**Ingestion Service Duplication (MAJOR):** -Analysis of `ingestion.service.ts` vs `kuzu-ingestion.service.ts` reveals: - -```typescript -// Current Duplication Pattern: -class IngestionService { - // Uses: GitHubService, ZipService, GraphPipeline (Legacy) - async processGitHubRepo() { /* 195 lines of similar logic */ } - private normalizeZipPaths() { /* Identical normalization logic */ } -} - -class KuzuIngestionService { - // Uses: GitHubService, ZipService, KuzuGraphPipeline (Next-Gen) - async processGitHubRepo() { /* 198 lines of similar logic */ } - private normalizeZipPaths() { /* Identical normalization logic */ } -} -``` - -**Issues Identified:** -- 90% code duplication between services -- Both use same GitHub/ZIP services but different pipelines -- Identical path normalization logic -- Different worker handling strategies - -**Repository Service Mixing:** -- `GitHubService` and `ZipService` are shared between engines -- No engine-specific optimizations or configurations -- Both services handle file filtering identically - -### 2. Pipeline Architecture Analysis - Good Foundation, Needs Refinement - -**Pipeline Comparison:** - -```typescript -// Legacy Pipeline (GraphPipeline): -// SimpleKnowledgeGraph → StructureProcessor → ParsingProcessor → ImportProcessor → CallProcessor -// - Sequential processing only -// - In-memory storage only -// - JSON serialization - -// Next-Gen Pipeline (KuzuGraphPipeline): -// KuzuKnowledgeGraph → StructureProcessor → ParallelParsingProcessor → ImportProcessor → CallProcessor -// - Parallel processing available -// - Direct KuzuDB persistence -// - No JSON overhead -``` - -**Processor Sharing Analysis:** -- `StructureProcessor`, `ImportProcessor`, `CallProcessor` are shared (GOOD) -- `ParsingProcessor` vs `ParallelParsingProcessor` are separate (GOOD) -- Shared processors work with both graph types via interface (EXCELLENT) - -**Issues Found:** -- Pipeline orchestration logic is 80% identical -- Progress reporting mechanisms differ significantly -- Error handling strategies inconsistent -- Validation logic duplicated - -### 3. Graph Interface Analysis - Partial Unification Exists - -**Current Interface Reality:** -```typescript -// Both implement KnowledgeGraph interface: -interface KnowledgeGraph { - nodes: GraphNode[]; - relationships: GraphRelationship[]; - addNode(node: GraphNode): void; - addRelationship(relationship: GraphRelationship): void; -} - -// But KuzuKnowledgeGraph has additional methods: -interface KuzuKnowledgeGraphInterface extends KnowledgeGraph { - getNodeCount(): number; - getRelationshipCount(): number; - query(cypher: string): Promise; -} -``` - -**UI Compatibility Issues:** -- Components use conditional logic: `graph.getNodeCount ? graph.getNodeCount() : graph.nodes.length` -- Engine-specific capabilities not properly abstracted -- Different query interfaces create complexity - -### 4. Configuration System Analysis - Needs Engine Separation - -**Current Config State:** -The existing `config.ts` has comprehensive configuration but lacks engine-specific sections: - -```typescript -// Missing Engine Configuration: -interface EngineConfig { - legacy: LegacyEngineConfig; // Not defined - nextgen: NextGenEngineConfig; // Not defined - runtime: RuntimeConfig; // Not defined -} -``` - -**Feature Flag Mixing:** -- Engine selection flags in `feature-flags.ts` -- Processing flags scattered across modules -- No centralized engine switching logic - -### 5. UI Component Analysis - Well Structured, Needs Minor Updates - -**Current UI Architecture (GOOD):** -``` -src/ui/ -├── components/engine/ # Engine selection (EXCELLENT) -├── hooks/useGitNexus.ts # Main orchestration hook (GOOD) -├── hooks/useEngine.ts # Engine management (GOOD) -└── pages/HomePage/ # Simplified main page (GOOD) -``` - -**Issues Found:** -- HomePage still contains engine-specific styling logic -- Processing status shows engine-specific details -- Export functionality not engine-aware - -### 6. Engine Wrapper Analysis - Strong Foundation - -**Engine Interface (EXCELLENT):** -The existing engine wrappers in `src/core/engines/` provide: -- Clean abstraction layer -- Proper fallback mechanisms -- Performance monitoring -- Health checks -- Validation - -**Strengths:** -- `ProcessingEngine` interface is comprehensive -- `LegacyProcessingEngine` properly wraps existing services -- `NextGenProcessingEngine` handles KuzuDB specifics -- Engine manager provides switching logic - -**Minor Issues:** -- Services are instantiated in engine constructors (tight coupling) -- No service injection or factory pattern -- Engine-specific configuration not externalized - -## Proposed Separation Architecture - -### 1. Service Layer Refactoring Strategy - -**Phase 1: Create Base Ingestion Service** - -```typescript -// src/services/common/base-ingestion.service.ts -abstract class BaseIngestionService { - protected abstract createPipeline(): Pipeline; - - async processGitHubRepo(url: string, options: IngestionOptions): Promise { - // Shared logic for URL parsing, structure discovery, normalization - const structure = await this.getGitHubStructure(url); - return this.processPipeline(structure, options); - } - - async processZipFile(file: File, options: IngestionOptions): Promise { - // Shared logic for ZIP extraction, normalization - const structure = await this.getZipStructure(file); - return this.processPipeline(structure, options); - } - - private async processPipeline(structure: RepositoryStructure, options: IngestionOptions) { - const pipeline = this.createPipeline(); - // Shared pipeline execution logic - } - - protected normalizeZipPaths(structure: CompleteStructure): CompleteStructure { - // Move shared normalization logic here - } -} -``` - -**Phase 2: Engine-Specific Service Implementations** - -```typescript -// src/services/legacy/legacy-ingestion.service.ts -class LegacyIngestionService extends BaseIngestionService { - protected createPipeline(): GraphPipeline { - return new GraphPipeline(); - } -} - -// src/services/nextgen/nextgen-ingestion.service.ts -class NextGenIngestionService extends BaseIngestionService { - protected createPipeline(): KuzuGraphPipeline { - return new KuzuGraphPipeline(); - } -} -``` - -**New Service Factory Pattern:** - -```typescript -// src/services/service.factory.ts -class ServiceFactory { - static createIngestionService(engine: ProcessingEngineType, token?: string): BaseIngestionService { - switch (engine) { - case 'legacy': return new LegacyIngestionService(token); - case 'nextgen': return new NextGenIngestionService(token); - default: throw new Error(`Unknown engine: ${engine}`); - } - } -} -``` - -### 2. Pipeline Unification Strategy (Minimal Changes) - -**Current State Assessment:** -- Processors are already well-separated and shared appropriately -- Only `ParsingProcessor` vs `ParallelParsingProcessor` differs -- Pipeline orchestration is the main duplication area - -**Proposed Unified Pipeline Architecture:** - -```typescript -// src/core/common/base-pipeline.ts -abstract class BasePipeline { - protected structureProcessor: StructureProcessor; - protected importProcessor: ImportProcessor; - protected callProcessor: CallProcessor; - - protected abstract createParsingProcessor(): ParsingProcessor | ParallelParsingProcessor; - protected abstract createGraph(): KnowledgeGraph | KuzuKnowledgeGraphInterface; - - async run(input: PipelineInput): Promise { - const graph = this.createGraph(); - - // Unified orchestration logic - await this.runStructurePass(graph, input); - await this.runParsingPass(graph, input); - await this.runImportPass(graph, input); - await this.runCallPass(graph, input); - - return graph; - } - - protected abstract runStructurePass(graph: any, input: PipelineInput): Promise; - // ... other abstract methods for engine-specific variations -} -``` - -**Updated Pipeline Implementations:** - -```typescript -// Legacy pipeline becomes: -class GraphPipeline extends BasePipeline { - protected createParsingProcessor() { return new ParsingProcessor(); } - protected createGraph() { return new SimpleKnowledgeGraph(); } -} - -// Next-Gen pipeline becomes: -class KuzuGraphPipeline extends BasePipeline { - protected createParsingProcessor() { return new ParallelParsingProcessor(); } - protected createGraph() { return new KuzuKnowledgeGraph(); } -} -``` - -### 3. Configuration Enhancement Plan - -**Extend Existing Config System:** - -```typescript -// Add to src/config/config.ts -const EngineConfigSchema = z.object({ - legacy: z.object({ - enabled: z.boolean().default(true), - memoryLimits: z.object({ - maxMemoryMB: z.number().min(256).max(2048).default(512), - gcIntervalMs: z.number().min(5000).max(60000).default(30000) - }), - processing: z.object({ - batchSize: z.number().min(1).max(50).default(10), - timeoutMs: z.number().min(5000).max(120000).default(30000) - }) - }), - - nextgen: z.object({ - enabled: z.boolean().default(true), - kuzu: z.object({ - databasePath: z.string().default('gitnexus.kuzu'), - bufferPoolSize: z.number().min(64).max(1024).default(256), - enableWAL: z.boolean().default(true) - }), - parallel: z.object({ - maxWorkers: z.number().min(1).max(navigator.hardwareConcurrency || 4).default(4), - batchSize: z.number().min(5).max(100).default(20), - workerTimeoutMs: z.number().min(10000).max(300000).default(60000) - }) - }), - - runtime: z.object({ - defaultEngine: z.enum(['legacy', 'nextgen']).default('legacy'), - allowFallback: z.boolean().default(true), - performanceMonitoring: z.boolean().default(true), - autoEngineSelection: z.boolean().default(false) - }) -}); - -// Integration with existing ConfigService -export class ConfigService { - // ... existing methods ... - - public get engines(): EngineConfig { - return this.config.engines; - } -} -``` - -**Environment Variable Support:** - -```bash -# Engine selection -ENGINE_DEFAULT=nextgen -ENGINE_ALLOW_FALLBACK=true - -# Legacy engine config -LEGACY_MEMORY_LIMIT_MB=512 -LEGACY_BATCH_SIZE=10 - -# Next-Gen engine config -NEXTGEN_MAX_WORKERS=4 -NEXTGEN_KUZU_BUFFER_POOL_SIZE=256 -NEXTGEN_BATCH_SIZE=20 -``` - -### 4. Graph Interface Unification - -```typescript -interface UnifiedKnowledgeGraph { - // Common interface for both implementations - nodes: GraphNode[]; - relationships: GraphRelationship[]; - - // Unified methods - addNode(node: GraphNode): void; - addRelationship(rel: GraphRelationship): void; - findNodes(criteria: SearchCriteria): GraphNode[]; - - // Engine-specific capabilities - getEngineType(): 'legacy' | 'nextgen'; - getCapabilities(): string[]; - - // Query interface - query?(cypher: string): Promise; -} -``` - -## Detailed Implementation Strategy - -### Phase 1: Service Layer Refactoring (Priority: HIGH) - -**Step 1.1: Create Base Ingestion Service (Week 1)** - -```typescript -// Create: src/services/common/base-ingestion.service.ts -// Extract: Shared logic from both ingestion services -// - GitHub URL parsing and validation -// - Repository structure discovery -// - ZIP path normalization -// - Progress reporting patterns -// - Error handling strategies -``` - -**Implementation Tasks:** -1. Create `BaseIngestionService` abstract class -2. Extract 90% shared logic from existing services -3. Define abstract methods for engine-specific pipeline creation -4. Implement factory pattern for service instantiation -5. Update existing services to extend base class - -**Step 1.2: Service Factory Implementation (Week 1)** - -```typescript -// Create: src/services/service.factory.ts -// Purpose: Centralized service creation with engine awareness - -class ServiceFactory { - static createIngestionService(engine: ProcessingEngineType): BaseIngestionService; - static createGitHubService(engine: ProcessingEngineType): GitHubService; - static createZipService(engine: ProcessingEngineType): ZipService; -} -``` - -**Step 1.3: Update Engine Wrappers (Week 1)** - -```typescript -// Update: src/core/engines/legacy/legacy-engine.ts -// Update: src/core/engines/nextgen/nextgen-engine.ts -// Change: Use ServiceFactory instead of direct instantiation - -class LegacyProcessingEngine { - constructor(githubToken?: string) { - // OLD: this.ingestionService = new IngestionService(githubToken); - // NEW: this.ingestionService = ServiceFactory.createIngestionService('legacy', githubToken); - } -} -``` - -### Phase 2: Pipeline Unification (Priority: MEDIUM) - -**Step 2.1: Create Base Pipeline Class (Week 2)** - -```typescript -// Create: src/core/common/base-pipeline.ts -// Purpose: Extract 80% shared orchestration logic - -abstract class BasePipeline { - // Shared orchestration methods - protected async runStructurePass(graph: any, input: any): Promise; - protected async runImportPass(graph: any, astMap: any, fileContents: any): Promise; - protected async runCallPass(graph: any, astMap: any, importMap: any): Promise; - - // Engine-specific abstract methods - protected abstract createGraph(): KnowledgeGraph | KuzuKnowledgeGraphInterface; - protected abstract createParsingProcessor(): ParsingProcessor | ParallelParsingProcessor; - protected abstract runParsingPass(graph: any, input: any): Promise; -} -``` - -**Implementation Benefits:** -- Reduce code duplication from 258 lines → ~100 lines per pipeline -- Unify progress reporting and validation logic -- Maintain engine-specific optimizations -- Preserve existing processor implementations - -**Step 2.2: Update Existing Pipelines (Week 2)** - -```typescript -// Refactor: GraphPipeline extends BasePipeline -// Refactor: KuzuGraphPipeline extends BasePipeline -// Preserve: All existing functionality and performance -// Add: Unified error handling and logging -``` - -### Phase 3: Configuration System Enhancement (Priority: MEDIUM) - -**Step 3.1: Extend Existing Config Schema (Week 2)** - -```typescript -// Extend: src/config/config.ts with EngineConfigSchema -// Add: Engine-specific configuration validation -// Integrate: With existing ConfigService singleton - -interface ExtendedAppConfig extends AppConfig { - engines: EngineConfig; -} -``` - -**Configuration Migration Strategy:** -1. Extend existing `AppConfigSchema` with `EngineConfigSchema` -2. Add environment variable support for engine settings -3. Implement runtime engine switching via config updates -4. Create engine health monitoring configuration - -**Step 3.2: Engine Manager Integration (Week 2)** - -```typescript -// Update: src/core/orchestration/engine-manager.ts -// Integration: Use config-driven engine selection -// Add: Automatic fallback based on configuration -// Add: Performance-based engine recommendation - -class EngineManager { - constructor(private config: ConfigService) { - this.defaultEngine = config.engines.runtime.defaultEngine; - this.allowFallback = config.engines.runtime.allowFallback; - } -} -``` - -### Phase 4: UI Component Refinement (Priority: LOW) - -**Current State:** UI components are already well-separated and use proper abstractions - -**Step 4.1: Minor UI Improvements (Week 3)** - -```typescript -// Update: src/ui/pages/HomePage/HomePage.tsx -// Remove: Engine-specific styling and conditional logic -// Simplify: Export functionality to be engine-agnostic -// Enhance: Error handling with engine context - -// Update: src/ui/components/engine/ProcessingStatus.tsx -// Abstract: Engine-specific status display logic -// Add: Unified progress reporting interface -``` - -**Step 4.2: Graph Interface Adapter (Optional)** - -```typescript -// Create: src/core/graph/graph-adapter.ts (if needed) -// Purpose: Provide unified interface for UI components -// Implementation: Adapter pattern for different graph types - -class GraphAdapter { - constructor(private graph: KnowledgeGraph | KuzuKnowledgeGraphInterface) {} - - getNodeCount(): number { - return 'getNodeCount' in this.graph - ? this.graph.getNodeCount() - : this.graph.nodes.length; - } -} -``` - -## Benefits of Proposed Architecture - -### 1. Clear Separation of Concerns -- Each engine has dedicated implementations -- Shared logic properly abstracted -- No cross-engine dependencies - -### 2. Maintainability -- Independent development of engines -- Clear ownership of components -- Easier testing and debugging - -### 3. Flexibility -- Runtime engine switching -- A/B testing capabilities -- Environment-specific configurations - -### 4. Performance -- Engine-optimized implementations -- No unnecessary abstraction overhead -- Clear performance monitoring - -### 5. Future-Proofing -- Easy addition of new engines -- Extensible configuration system -- Pluggable architecture patterns - -## Comprehensive Testing Strategy - -### 1. Unit Testing Approach - -**Service Layer Tests:** -```typescript -// Test: BaseIngestionService shared logic -// Test: Legacy/NextGen service implementations -// Test: Service factory creation patterns -// Test: Error handling consistency - -describe('BaseIngestionService', () => { - test('normalizes ZIP paths consistently across engines'); - test('handles GitHub URL parsing uniformly'); - test('reports progress with same interface'); - test('handles errors with unified strategy'); -}); - -describe('ServiceFactory', () => { - test('creates correct service for legacy engine'); - test('creates correct service for nextgen engine'); - test('throws error for invalid engine type'); -}); -``` - -**Pipeline Tests:** -```typescript -// Test: BasePipeline shared orchestration -// Test: Engine-specific pipeline variations -// Test: Performance equivalence between engines - -describe('Pipeline Compatibility', () => { - test('legacy and nextgen produce equivalent graphs', async () => { - const testRepo = createTestRepository(); - const legacyResult = await legacyPipeline.run(testRepo); - const nextgenResult = await nextgenPipeline.run(testRepo); - - expect(normalizeGraph(legacyResult)).toEqual(normalizeGraph(nextgenResult)); - }); -}); -``` - -### 2. Integration Testing - -**Engine Switching Tests:** -```typescript -describe('Engine Integration', () => { - test('seamless switching between engines'); - test('fallback mechanism works correctly'); - test('performance comparison data accurate'); - test('configuration changes apply correctly'); -}); -``` - -**End-to-End Repository Processing:** -```typescript -describe('Repository Processing E2E', () => { - test('GitHub repository processing (both engines)'); - test('ZIP file processing (both engines)'); - test('Large repository handling (performance)'); - test('Error scenarios and recovery'); -}); -``` - -### 3. Performance Testing - -**Benchmarking Strategy:** -```typescript -// Performance test suite for engine comparison -describe('Performance Benchmarks', () => { - const testRepositories = [ - { size: 'small', files: 10, loc: 1000 }, - { size: 'medium', files: 100, loc: 10000 }, - { size: 'large', files: 1000, loc: 100000 } - ]; - - testRepositories.forEach(repo => { - test(`${repo.size} repository processing speed`, async () => { - const legacyTime = await benchmarkEngine('legacy', repo); - const nextgenTime = await benchmarkEngine('nextgen', repo); - - expect(nextgenTime).toBeLessThanOrEqual(legacyTime * 2); // Allow 2x variance - }); - }); -}); -``` - -### 4. UI Regression Testing - -**Component Testing:** -```typescript -describe('UI Regression Tests', () => { - test('HomePage renders identically for both engines'); - test('Graph visualization works with both graph types'); - test('Engine selector functions correctly'); - test('Processing status updates properly'); - test('Export functionality works for both engines'); -}); -``` - -## Risk Assessment & Mitigation - -### 1. Technical Risks - -**Risk: Service Refactoring Breaking Changes** -- **Probability:** Low -- **Impact:** High -- **Mitigation:** - - Comprehensive test coverage before refactoring - - Gradual migration with feature flags - - Keep old services until new ones proven stable - -**Risk: Performance Regression** -- **Probability:** Medium -- **Impact:** Medium -- **Mitigation:** - - Baseline performance measurements - - Continuous benchmarking during refactoring - - Abstract layer optimization to minimize overhead - -**Risk: Configuration Migration Issues** -- **Probability:** Low -- **Impact:** Medium -- **Mitigation:** - - Backward-compatible configuration schema - - Automatic migration scripts - - Default fallback values for all new settings - -### 2. Development Risks - -**Risk: Scope Creep** -- **Probability:** Medium -- **Impact:** Medium -- **Mitigation:** - - Clear phase boundaries and deliverables - - Focus on minimal necessary changes - - Defer optimizations to later phases - -**Risk: Testing Complexity** -- **Probability:** High -- **Impact:** Low -- **Mitigation:** - - Automated test generation for both engines - - Test data fixtures shared between test suites - - Parallel test execution for efficiency - -### 3. Rollback Strategy - -**Immediate Rollback (< 5 minutes):** -```typescript -// Configuration-based instant rollback -const config = { - engines: { - runtime: { - defaultEngine: 'legacy', // Switch back to legacy - allowFallback: false, // Disable problematic engine - } - } -}; -``` - -**Gradual Rollback (< 30 minutes):** -- Feature flag-based service selection -- Revert to old service implementations -- Database rollback for KuzuDB issues - -**Full Rollback (< 2 hours):** -- Git revert to pre-refactoring state -- Configuration reset to original values -- Cache clearing and service restart - -## Implementation Timeline & Next Steps - -### Week 1: Service Layer Foundation - -**Day 1-2: Analysis & Setup** -- [ ] Create `src/services/common/` directory structure -- [ ] Extract shared logic analysis from existing ingestion services -- [ ] Set up test framework for service layer testing - -**Day 3-4: Base Service Implementation** -- [ ] Implement `BaseIngestionService` abstract class -- [ ] Create `ServiceFactory` with engine-aware instantiation -- [ ] Write unit tests for shared logic extraction - -**Day 5: Integration & Testing** -- [ ] Update `LegacyProcessingEngine` to use ServiceFactory -- [ ] Update `NextGenProcessingEngine` to use ServiceFactory -- [ ] Run comprehensive integration tests -- [ ] Validate no functionality changes - -### Week 2: Pipeline & Configuration - -**Day 1-2: Pipeline Unification** -- [ ] Implement `BasePipeline` abstract class -- [ ] Refactor `GraphPipeline` to extend base class -- [ ] Refactor `KuzuGraphPipeline` to extend base class -- [ ] Maintain all existing performance characteristics - -**Day 3-4: Configuration Enhancement** -- [ ] Extend `AppConfigSchema` with `EngineConfigSchema` -- [ ] Add environment variable support for engine settings -- [ ] Update `EngineManager` to use configuration-driven selection -- [ ] Implement runtime engine switching - -**Day 5: Integration Testing** -- [ ] End-to-end testing with both engines -- [ ] Performance benchmarking comparison -- [ ] Configuration migration testing - -### Week 3: Refinement & Documentation - -**Day 1-2: UI Component Polish** -- [ ] Remove remaining engine-specific UI logic -- [ ] Enhance error handling with engine context -- [ ] Improve export functionality engine awareness - -**Day 3-4: Testing & Validation** -- [ ] Complete test suite implementation -- [ ] Performance regression testing -- [ ] UI regression testing -- [ ] Documentation updates - -**Day 5: Deployment Preparation** -- [ ] Feature flag configuration for gradual rollout -- [ ] Monitoring and alerting setup -- [ ] Rollback procedures testing - -## Success Metrics - -### Technical Metrics -- **Code Duplication:** Reduce from ~90% to <20% between ingestion services -- **Performance:** No more than 5% performance degradation during refactoring -- **Test Coverage:** Maintain >80% test coverage throughout refactoring -- **Build Time:** No significant increase in build/test execution time - -### Quality Metrics -- **Zero Breaking Changes:** All existing APIs and UI behavior preserved -- **Engine Switching:** <500ms switching time between engines -- **Fallback Reliability:** 100% success rate for engine fallback scenarios -- **Configuration Validation:** 100% of invalid configurations caught at startup - -### Development Metrics -- **Maintainability:** Reduce complexity of adding new engines by 60% -- **Developer Experience:** Clear separation enables independent engine development -- **Testing Efficiency:** Parallel test execution for both engines - -## Conclusion - -This refactoring plan provides **clear separation of concerns** while maintaining **complete backward compatibility**. The approach is **incremental and safe**, with comprehensive testing and rollback strategies at every step. - -**Key Benefits:** -1. **90% reduction** in code duplication between ingestion services -2. **Clear ownership** of engine-specific vs shared components -3. **Configuration-driven** engine selection and runtime switching -4. **Independent development** capability for each processing engine -5. **Comprehensive testing** strategy ensuring stability - -**Risk Mitigation:** -- **Minimal changes** to well-functioning components -- **Feature flag** protection for gradual rollout -- **Comprehensive rollback** procedures at multiple levels -- **Performance monitoring** throughout the process - -The project is well-positioned for this refactoring with its existing engine wrapper architecture and properly separated UI components. The main work involves extracting shared service logic and unifying pipeline orchestration, both of which are low-risk, high-value improvements. - -## Immediate Action Items - -### Before Starting Implementation - -**1. Establish Baseline Measurements** -```bash -# Run comprehensive testing before changes -npm run test:coverage - -# Measure current performance -npm run dev -# Test both GitHub repo processing and ZIP file processing -# Record processing times and memory usage -``` - -**2. Create Feature Flags** -```typescript -// Add to src/config/feature-flags.ts -export const getRefactoringFlags = () => ({ - useBaseIngestionService: false, // Phase 1 rollout - useUnifiedPipeline: false, // Phase 2 rollout - useEnhancedConfig: false, // Phase 3 rollout - enableRefactoringDebug: true // Debug logging -}); -``` - -**3. Set Up Development Environment** -```bash -# Ensure all dependencies are up to date -npm install - -# Run tests to confirm current stability -npm run test - -# Start development server to test current functionality -npm run dev -``` - -### Quick Verification Steps - -**Test Current Engine Switching:** -1. Load the application (`npm run dev`) -2. Process a small GitHub repository with Legacy engine -3. Switch to Next-Gen engine and process the same repository -4. Verify both produce similar results -5. Note any differences in processing time or graph structure - -**Validate Current Architecture:** -1. Check that `src/core/engines/` contains the dual-track system -2. Verify `src/services/facade/gitnexus-facade.ts` exists and works -3. Confirm UI components in `src/ui/components/engine/` are functional -4. Test engine fallback mechanism - -## Development Environment Notes - -**Current Build System:** -- Uses Vite for bundling and development server -- TypeScript compilation with `tsc -b` -- Tree-sitter query compilation step -- Jest for testing with coverage support - -**Key Dependencies:** -- `kuzu-wasm`: ^0.11.1 (Next-Gen engine) -- `web-tree-sitter`: ^0.20.8 (Code parsing) -- `comlink`: ^4.4.1 (Worker communication) -- `react`: ^18.3.1 (UI framework) - -**Testing Setup:** -- Jest with TypeScript support -- Coverage reporting configured -- Watch mode available for development - -## Recommended Development Workflow - -1. **Create Feature Branch** - ```bash - git checkout -b refactor/separation-of-concerns - ``` - -2. **Implement Phase 1 (Service Layer)** - - Create base service classes - - Update existing services gradually - - Test each change incrementally - -3. **Use Feature Flags for Testing** - - Enable new services selectively - - Compare behavior against old implementation - - Gather performance metrics - -4. **Continuous Integration** - - Run full test suite after each major change - - Monitor performance impact - - Keep rollback options ready - -## Final Recommendations - -### What to Change -1. **Extract shared service logic** into base classes (HIGH PRIORITY) -2. **Unify pipeline orchestration** with template method pattern (MEDIUM PRIORITY) -3. **Enhance configuration system** with engine-specific settings (MEDIUM PRIORITY) -4. **Polish UI components** to remove engine-specific details (LOW PRIORITY) - -### What NOT to Change -1. **Core processor implementations** (StructureProcessor, ImportProcessor, etc.) - already well-designed -2. **Engine wrapper architecture** - already provides excellent separation -3. **Graph data structures** - both SimpleKnowledgeGraph and KuzuKnowledgeGraph work well -4. **UI component architecture** - already properly decomposed with hooks - -### Success Indicators -1. **Service duplication reduced** from 90% to <20% -2. **No performance regression** beyond 5% -3. **All existing tests pass** without modification -4. **Engine switching works** seamlessly via configuration -5. **Code maintainability improved** for future development - -This analysis confirms the codebase has a **solid foundation** for separation of concerns. The existing dual-track architecture provides an excellent starting point, and the proposed refactoring will eliminate code duplication while maintaining all current functionality and performance characteristics. \ No newline at end of file