From 448fb3dd202d4aed3717a0d393146d81ea8158a7 Mon Sep 17 00:00:00 2001 From: Smartsheet-JB-Brown Date: Tue, 15 Apr 2025 13:50:53 -0700 Subject: [PATCH] update implementation documentation --- .../implementation/01-architecture.md | 330 +++--- .../implementation/02-core-components.md | 637 ++++-------- .../implementation/03-data-structures.md | 792 +++++---------- .../implementation/04-search-and-filter.md | 941 +++++------------- 4 files changed, 862 insertions(+), 1838 deletions(-) diff --git a/cline_docs/package-manager/implementation/01-architecture.md b/cline_docs/package-manager/implementation/01-architecture.md index a2c5a9dc3c..9e9e283032 100644 --- a/cline_docs/package-manager/implementation/01-architecture.md +++ b/cline_docs/package-manager/implementation/01-architecture.md @@ -13,7 +13,8 @@ graph TD User[User] -->|Interacts with| UI[Package Manager UI] UI -->|Sends messages| MH[Message Handler] MH -->|Processes requests| PM[PackageManagerManager] - PM -->|Loads data from| MS[MetadataScanner] + PM -->|Fetches repos| GF[GitFetcher] + GF -->|Scans metadata| MS[MetadataScanner] MS -->|Reads| FS[File System / Git Repositories] PM -->|Returns filtered data| MH MH -->|Updates state| UI @@ -35,20 +36,23 @@ The Package Manager components interact through a well-defined message flow: 1. **Data Loading**: + - GitFetcher handles repository cloning and updates - MetadataScanner loads package data from repositories - - PackageManagerManager stores and manages this data + - PackageManagerManager manages caching and concurrency - UI requests data through the message handler 2. **Filtering and Search**: - UI sends filter/search criteria to the backend - - PackageManagerManager applies filters to the data + - PackageManagerManager applies filters with match info - Filtered results are returned to the UI + - State manager handles view-level filtering 3. **Source Management**: - UI sends source management commands - - PackageManagerManager updates source configurations - - MetadataScanner reloads data from updated sources + - PackageManagerManager coordinates with GitFetcher + - Cache is managed with timeout protection + - Sources are processed with concurrency control ## Data Flow Diagram @@ -62,6 +66,7 @@ graph LR end subgraph Backend + GF[GitFetcher] MS[MetadataScanner] PM[PackageManagerManager] MH[Message Handler] @@ -72,10 +77,11 @@ graph LR State[State Management] end - GR -->|Raw Data| MS - FS -->|Template Data| MS - MS -->|Parsed Metadata| PM - PM -->|Stored Items| PM + GR -->|Clone/Pull| GF + FS -->|Cache| GF + GF -->|Metadata| MS + MS -->|Parsed Data| PM + PM -->|Cached Items| PM UI -->|User Actions| MH MH -->|Messages| PM PM -->|Filtered Data| MH @@ -95,16 +101,20 @@ sequenceDiagram participant UI as UI Components participant MH as Message Handler participant PM as PackageManagerManager + participant GF as GitFetcher participant MS as MetadataScanner participant FS as File System/Git User->>UI: Open Package Manager UI->>MH: Send init message MH->>PM: Initialize - PM->>MS: Request metadata scan + PM->>GF: Request repository data + GF->>FS: Clone/pull repository + GF->>MS: Request metadata scan MS->>FS: Read repository data FS-->>MS: Return raw data - MS-->>PM: Return parsed metadata + MS-->>GF: Return parsed metadata + GF-->>PM: Return repository data PM-->>MH: Return initial items MH-->>UI: Update with items UI-->>User: Display packages @@ -118,23 +128,28 @@ This sequence diagram illustrates the search and filter process: sequenceDiagram participant User participant UI as UI Components + participant State as State Manager participant MH as Message Handler participant PM as PackageManagerManager User->>UI: Enter search term - UI->>MH: Send search message + UI->>State: Update filters + State->>MH: Send search message MH->>PM: Apply search filter - PM->>PM: Filter items + PM->>PM: Filter items with match info PM-->>MH: Return filtered items - MH-->>UI: Update with filtered items + MH-->>State: Update with filtered items + State-->>UI: Update view UI-->>User: Display filtered results User->>UI: Select type filter - UI->>MH: Send type filter message + UI->>State: Update type filter + State->>MH: Send type filter message MH->>PM: Apply type filter - PM->>PM: Filter by type + PM->>PM: Filter by type with match info PM-->>MH: Return type-filtered items - MH-->>UI: Update with type-filtered items + MH-->>State: Update filtered items + State-->>UI: Update view UI-->>User: Display type-filtered results ``` @@ -148,257 +163,178 @@ The following class diagram shows the main classes in the Package Manager system classDiagram class PackageManagerManager { -currentItems: PackageManagerItem[] - -sources: PackageManagerSource[] - +getItems(): PackageManagerItem[] + -cache: Map + -gitFetcher: GitFetcher + -activeSourceOperations: Set + +getPackageManagerItems(): PackageManagerItem[] +filterItems(filters): PackageManagerItem[] - +addSource(url): void - +removeSource(url): void - +refreshSources(): void + +sortItems(sortBy, order): PackageManagerItem[] + +refreshRepository(url): void + -queueOperation(operation): void + } + + class GitFetcher { + -cacheDir: string + -metadataScanner: MetadataScanner + +fetchRepository(url): PackageManagerRepository + -cloneOrPullRepository(url): void + -validateRepositoryStructure(dir): void + -parseRepositoryMetadata(dir): RepositoryMetadata } class MetadataScanner { + -git: SimpleGit +scanDirectory(path): PackageManagerItem[] - +scanRepository(url): PackageManagerItem[] - -parseMetadata(file): any + +parseMetadata(file): ComponentMetadata -buildComponentHierarchy(items): PackageManagerItem[] } - class PackageManagerMessageHandler { - +handleMessage(message): any - -handleSearchMessage(message): any - -handleFilterMessage(message): any - -handleSourceMessage(message): any - } - - class PackageManagerItem { - +name: string - +description: string - +type: string - +items: any[] - +tags: string[] - +matchInfo: MatchInfo - } - - class PackageManagerSource { - +url: string - +name: string - +enabled: boolean - } - - PackageManagerManager --> PackageManagerItem: manages - PackageManagerManager --> PackageManagerSource: configures - PackageManagerManager --> MetadataScanner: uses - PackageManagerMessageHandler --> PackageManagerManager: calls -``` - -### UI Component Classes - -This class diagram shows the main UI components: - -```mermaid -classDiagram - class PackageManagerView { + class PackageManagerViewStateManager { -items: PackageManagerItem[] -filters: Filters - -activeTab: string - +render(): JSX - +handleFilterChange(): void - +handleSearch(): void + -sortBy: string + -sortOrder: string + +setFilters(filters): void + +getFilteredAndSortedItems(): PackageManagerItem[] + -itemMatchesFilters(item): boolean } - class PackageManagerItemCard { - -item: PackageManagerItem - -filters: Filters - +render(): JSX - +handleTagClick(): void - } - - class ExpandableSection { - -title: string - -isExpanded: boolean - +toggle(): void - +render(): JSX - } - - class TypeGroup { - -type: string - -items: any[] - -searchTerm: string - +render(): JSX - } - - PackageManagerView --> PackageManagerItemCard: contains - PackageManagerItemCard --> ExpandableSection: contains - ExpandableSection --> TypeGroup: contains + PackageManagerManager --> GitFetcher: uses + GitFetcher --> MetadataScanner: uses + PackageManagerManager --> PackageManagerViewStateManager: updates ``` ## Component Responsibilities ### Backend Components -1. **MetadataScanner** +1. **GitFetcher** - - Scans directories and repositories for package metadata + - Handles Git repository operations + - Manages repository caching + - Validates repository structure + - Coordinates with MetadataScanner + +2. **MetadataScanner** + + - Scans directories and repositories - Parses YAML metadata files - Builds component hierarchies - - Handles file system and Git operations + - Handles file system operations -2. **PackageManagerManager** +3. **PackageManagerManager** - - Stores and manages package items - - Applies filters and search criteria - - Manages package sources - - Handles package operations + - Manages concurrent operations + - Handles caching with timeout protection + - Coordinates repository operations + - Provides filtering and sorting -3. **packageManagerMessageHandler** +4. **packageManagerMessageHandler** - Routes messages between UI and backend - Processes commands from the UI - - Returns data and status updates to the UI + - Returns data and status updates - Handles error conditions ### Frontend Components -1. **PackageManagerView** +1. **PackageManagerViewStateManager** - - Main container component - - Manages overall UI state - - Handles tab navigation - - Displays filter controls + - Manages view-level state + - Handles filtering and sorting + - Maintains UI preferences + - Coordinates with backend state 2. **PackageManagerItemCard** - - Displays individual package information + - Displays package information - Handles tag interactions - - Manages expandable details section - - Provides action buttons + - Manages expandable sections + - Shows match highlights 3. **ExpandableSection** - - Provides collapsible UI sections + - Provides collapsible sections - Manages expand/collapse state - Handles animations - - Displays section headers and badges + - Shows section metadata 4. **TypeGroup** - - Groups and displays components by type + - Groups items by type - Formats item lists - Highlights search matches - - Provides consistent styling - -## Data Flow Patterns - -### Message-Based Communication - -The Package Manager uses a message-based architecture for communication between the frontend and backend: - -1. **Message Structure**: - - ```typescript - { - type: string // The message type (e.g., "search", "filter", "addSource") - payload: any // The message data - } - ``` - -2. **Common Message Types**: - - - `search`: Apply a search term filter - - `filter`: Apply type or tag filters - - `addSource`: Add a new package source - - `removeSource`: Remove a package source - - `refreshSources`: Reload data from sources - -3. **Response Structure**: - ```typescript - { - type: string; // The response type - data: any; // The response data - error?: string; // Optional error message - } - ``` - -### State Management - -The Package Manager maintains state in several places: - -1. **Backend State**: - - - Current items in the PackageManagerManager - - Source configurations - - Cached metadata - -2. **Frontend State**: - - - Current filters and search terms - - UI state (active tab, expanded sections) - - Display preferences - -3. **Persistent State**: - - Source configurations stored in extension settings - - User preferences + - Maintains consistent styling ## Performance Considerations The Package Manager architecture addresses several performance challenges: -1. **Lazy Loading**: +1. **Concurrency Control**: - - Metadata is loaded on demand - - Repositories are scanned only when needed - - UI components render incrementally + - Source operations are locked to prevent conflicts + - Operations are queued during metadata scanning + - Cache timeouts prevent hanging operations + - Repository operations are atomic -2. **Efficient Filtering**: +2. **Efficient Caching**: - - Filtering happens on the backend to reduce data transfer - - Search algorithms optimize for common patterns - - Results are cached when possible + - Repository data is cached with expiry + - Cache is cleaned up automatically + - Forced refresh available when needed + - Cache directories managed efficiently -3. **Responsive UI**: - - Asynchronous operations prevent UI blocking - - Animations provide feedback during loading - - Pagination limits the number of items displayed at once +3. **Smart Filtering**: + - Match info tracks filter matches + - Filtering happens at multiple levels + - View state optimizes re-renders + - Search is case-insensitive and normalized ## Error Handling The architecture includes robust error handling: -1. **Source Errors**: +1. **Repository Operations**: - - Invalid repositories are marked with error states - - Users are notified of access issues - - The system continues to function with other sources + - Git lock files are cleaned up + - Failed clones are retried + - Corrupt repositories are re-cloned + - Network timeouts are handled -2. **Parsing Errors**: +2. **Data Processing**: - - Malformed metadata is gracefully handled - - Partial results are displayed when possible - - Error details are logged for debugging + - Invalid metadata is gracefully handled + - Missing files are reported clearly + - Parse errors preserve partial data + - Type validation ensures consistency -3. **Network Errors**: - - Timeouts and retries for network operations - - Offline mode with cached data - - Clear error messages for user troubleshooting +3. **State Management**: + - Invalid filters are normalized + - Sort operations handle missing data + - View updates are atomic + - Error states are preserved ## Extensibility Points The Package Manager architecture is designed for extensibility: -1. **New Component Types**: +1. **Repository Sources**: - - The system can be extended to support new component types - - Type-specific rendering can be added to the UI - - Backend processing adapts to new types + - Support for multiple Git providers + - Custom repository validation + - Flexible metadata formats + - Localization support -2. **Additional Filters**: +2. **Filtering System**: - - New filter types can be added to the system - - Filter logic can be extended in the PackageManagerManager - - UI can be updated to display new filter controls + - Custom filter types + - Extensible match info + - Flexible sort options + - View state customization -3. **Custom Sources**: - - The source system supports various repository types - - Custom source providers can be implemented - - Authentication mechanisms can be extended +3. **UI Components**: + - Custom item renderers + - Flexible layout system + - Theme integration + - Accessibility support --- diff --git a/cline_docs/package-manager/implementation/02-core-components.md b/cline_docs/package-manager/implementation/02-core-components.md index 44ebdde5ef..e411bd9607 100644 --- a/cline_docs/package-manager/implementation/02-core-components.md +++ b/cline_docs/package-manager/implementation/02-core-components.md @@ -2,172 +2,190 @@ This document provides detailed information about the core components of the Package Manager system, their responsibilities, implementation details, and interactions. +## GitFetcher + +The GitFetcher is responsible for managing Git repository operations, including cloning, pulling, and caching repository data. + +### Responsibilities + +- Cloning and updating Git repositories +- Managing repository cache +- Validating repository structure +- Coordinating with MetadataScanner +- Handling repository timeouts and errors + +### Implementation Details + +```typescript +class GitFetcher { + private readonly cacheDir: string + private metadataScanner: MetadataScanner + private git?: SimpleGit + + /** + * Fetch repository data + * @param repoUrl Repository URL + * @param forceRefresh Whether to bypass cache + * @param sourceName Optional source name + * @returns Repository data + */ + public async fetchRepository( + repoUrl: string, + forceRefresh = false, + sourceName?: string, + ): Promise { + // Implementation details + } + + /** + * Clone or pull repository + * @param repoUrl Repository URL + * @param repoDir Repository directory + * @param forceRefresh Whether to force refresh + */ + private async cloneOrPullRepository(repoUrl: string, repoDir: string, forceRefresh: boolean): Promise { + // Implementation details + } + + /** + * Clean up git locks + * @param repoDir Repository directory + */ + private async cleanupGitLocks(repoDir: string): Promise { + // Implementation details + } +} +``` + +### Key Algorithms + +#### Repository Management + +The repository management process includes: + +1. **Cache Management**: + + - Check if repository exists in cache + - Validate cache freshness + - Clean up stale cache entries + - Handle cache directory creation + +2. **Repository Operations**: + + - Clone new repositories + - Pull updates for existing repos + - Handle git lock files + - Clean up failed operations + +3. **Error Recovery**: + - Handle network timeouts + - Recover from corrupt repositories + - Clean up partial clones + - Retry failed operations + ## MetadataScanner -The MetadataScanner is responsible for reading and parsing package metadata from various sources, including local file systems and remote Git repositories. +The MetadataScanner is responsible for reading and parsing package metadata from repositories. ### Responsibilities - Scanning directories for package metadata files - Parsing YAML metadata into structured objects - Building component hierarchies -- Handling file system and Git operations - Supporting localized metadata +- Validating metadata structure ### Implementation Details ```typescript class MetadataScanner { + private git: SimpleGit + private localizationOptions: LocalizationOptions + /** - * Scans a directory for package metadata - * @param directoryPath Path to the directory to scan - * @param baseUrl Base URL for the repository (for remote sources) + * Scan directory for package metadata + * @param directoryPath Directory to scan + * @param baseUrl Base repository URL + * @param sourceName Source repository name * @returns Array of package items */ - public async scanDirectory(directoryPath: string, baseUrl?: string): Promise { + public async scanDirectory( + directoryPath: string, + baseUrl?: string, + sourceName?: string, + ): Promise { // Implementation details } /** - * Scans a Git repository for package metadata - * @param repoUrl URL of the Git repository - * @returns Array of package items + * Parse metadata file + * @param filePath Path to metadata file + * @returns Parsed metadata */ - public async scanRepository(repoUrl: string): Promise { - // Implementation details - } - - /** - * Parses a YAML metadata file - * @param filePath Path to the metadata file - * @returns Parsed metadata object - */ - private async parseMetadataFile(filePath: string): Promise { - // Implementation details - } - - /** - * Builds a component hierarchy from flat items - * @param items Array of items to organize - * @returns Hierarchical structure of items - */ - private buildComponentHierarchy(items: any[]): PackageManagerItem[] { + private async parseMetadataFile(filePath: string): Promise { // Implementation details } } ``` -### Key Algorithms - -#### Directory Scanning - -The directory scanning algorithm recursively traverses directories looking for metadata files: - -1. Start at the root directory -2. Look for `metadata.*.yml` files in the current directory -3. Parse found metadata files -4. For each subdirectory: - - Determine the component type based on directory name - - Recursively scan the subdirectory - - Associate child components with parent components -5. Build the component hierarchy - -#### Metadata Parsing - -The metadata parsing process handles multiple formats and localizations: - -1. Read the YAML file content -2. Parse the YAML into a JavaScript object -3. Extract the locale from the filename (e.g., `en` from `metadata.en.yml`) -4. Validate required fields (name, description, version) -5. Process optional fields (tags, author, etc.) -6. Return a structured metadata object - -### Error Handling - -The MetadataScanner includes robust error handling: - -- Invalid YAML files are reported with specific parsing errors -- Missing required fields trigger validation errors -- File system access issues are caught and reported -- Network errors during Git operations are handled gracefully -- Partial results are returned when possible, with error flags - ## PackageManagerManager -The PackageManagerManager is the central component that manages package items, applies filters, and handles package operations. +The PackageManagerManager is the central component that manages package data, caching, and operations. ### Responsibilities -- Storing and managing package items -- Applying filters and search criteria +- Managing concurrent operations +- Handling repository caching +- Coordinating with GitFetcher +- Applying filters and sorting - Managing package sources -- Handling package operations -- Maintaining state between sessions ### Implementation Details ```typescript class PackageManagerManager { private currentItems: PackageManagerItem[] = [] - private sources: PackageManagerSource[] = [] + private cache: Map + private gitFetcher: GitFetcher + private activeSourceOperations = new Set() + private isMetadataScanActive = false + private pendingOperations: Array<() => Promise> = [] /** - * Constructor - * @param context VS Code extension context + * Queue an operation to run when no metadata scan is active */ - constructor(private context: vscode.ExtensionContext) { - // Initialize from stored state + private async queueOperation(operation: () => Promise): Promise { + // Implementation details } /** - * Get all items - * @returns Array of all package items + * Get package manager items from sources */ - public getItems(): PackageManagerItem[] { - return this.currentItems + public async getPackageManagerItems( + sources: PackageManagerSource[], + ): Promise<{ items: PackageManagerItem[]; errors?: string[] }> { + // Implementation details } /** * Filter items based on criteria - * @param filters Filter criteria - * @returns Filtered array of items */ - public filterItems(filters: { type?: string; search?: string; tags?: string[] }): PackageManagerItem[] { + public filterItems( + items: PackageManagerItem[], + filters: { type?: ComponentType; search?: string; tags?: string[] }, + ): PackageManagerItem[] { // Implementation details } /** - * Add a new package source - * @param url Source repository URL - * @param name Optional source name - * @returns Success status + * Sort items by field */ - public async addSource(url: string, name?: string): Promise { - // Implementation details - } - - /** - * Remove a package source - * @param url Source repository URL - * @returns Success status - */ - public removeSource(url: string): boolean { - // Implementation details - } - - /** - * Refresh all sources - * @returns Updated items - */ - public async refreshSources(): Promise { - // Implementation details - } - - /** - * Save state to persistent storage - */ - private saveState(): void { + public sortItems( + items: PackageManagerItem[], + sortBy: keyof Pick, + sortOrder: "asc" | "desc", + sortSubcomponents: boolean = false, + ): PackageManagerItem[] { // Implementation details } } @@ -175,362 +193,139 @@ class PackageManagerManager { ### Key Algorithms -#### Item Filtering +#### Concurrency Control -The filtering algorithm applies multiple criteria to the package items: +The manager implements sophisticated concurrency control: -1. Start with the complete set of items -2. If a search term and/or filter is specified: - - Check item name, description, and author for matches - - Check subcomponents for matches - - Keep items that match or have matching subcomponents - - Add match information to the items -3. Return the filtered items with match information +1. **Operation Queueing**: -#### Source Management + - Queue operations during active scans + - Process operations sequentially + - Handle operation dependencies + - Maintain operation order -The source management process handles adding, removing, and refreshing sources: +2. **Source Locking**: -1. For adding a source: + - Lock sources during operations + - Prevent concurrent source access + - Handle lock timeouts + - Clean up stale locks - - Validate the repository URL - - Check if the source already exists - - Add the source to the list - - Scan the repository for items - - Add the items to the current set - - Save the updated source list +3. **Cache Management**: + - Implement cache expiration + - Handle cache invalidation + - Clean up unused cache + - Optimize cache storage -2. For removing a source: +#### Advanced Filtering - - Find the source in the list - - Remove items from that source - - Remove the source from the list - - Save the updated source list +The filtering system provides rich functionality: -3. For refreshing sources: - - Clear the current items - - For each enabled source: - - Scan the repository for items - - Add the items to the current set - - Return the updated items +1. **Multi-level Filtering**: -### State Persistence + - Filter parent items + - Filter subcomponents + - Handle package-specific logic + - Track match information -The PackageManagerManager maintains state between sessions: +2. **Match Information**: + - Track match reasons + - Handle partial matches + - Support highlighting + - Maintain match context -- Source configurations are stored in extension global state -- User preferences are persisted -- Cached metadata can be stored for performance -- State is loaded during initialization -- State is saved after significant changes +## PackageManagerViewStateManager -## packageManagerMessageHandler - -The packageManagerMessageHandler is responsible for routing messages between the UI and the backend components. +The PackageManagerViewStateManager handles UI state and view-level operations. ### Responsibilities -- Processing messages from the UI -- Calling appropriate PackageManagerManager methods -- Returning results to the UI -- Handling errors and status updates -- Managing asynchronous operations +- Managing view-level state +- Handling UI filters +- Coordinating sorting +- Managing item visibility ### Implementation Details ```typescript -/** - * Handle package manager messages - * @param message The message to handle - * @param packageManager The package manager instance - * @returns Response object - */ -export async function handlePackageManagerMessages(message: any, packageManager: PackageManagerManager): Promise { - switch (message.type) { - case "getItems": - return { - type: "items", - data: packageManager.getItems(), - } +class PackageManagerViewStateManager { + private items: PackageManagerItem[] = [] + private sortBy: "name" | "lastUpdated" = "name" + private sortOrder: "asc" | "desc" = "asc" + private filters: Filters = { type: "", search: "", tags: [] } - case "search": - return { - type: "searchResults", - data: packageManager.filterItems({ - search: message.search, - type: message.typeFilter, - tags: message.tagFilters, - }), - } + /** + * Get filtered and sorted items + */ + public getFilteredAndSortedItems(): PackageManagerItem[] { + // Implementation details + } - case "addSource": - try { - const success = await packageManager.addSource(message.url, message.name) - return { - type: "sourceAdded", - data: { success }, - } - } catch (error) { - return { - type: "error", - error: error.message, - } - } - - // Additional message handlers... - - default: - return { - type: "error", - error: `Unknown message type: ${message.type}`, - } + /** + * Check if item matches current filters + */ + private itemMatchesFilters(item: PackageManagerItem | Subcomponent): boolean { + // Implementation details } } ``` -### Message Types +## Component Integration -The message handler processes several types of messages: +The components work together through well-defined interfaces: -#### Input Messages +### Data Flow -1. **getItems**: Request all package items +1. **Repository Operations**: - ```typescript - { - type: "getItems" - } - ``` - -2. **search**: Apply search and filter criteria - - ```typescript - { - type: "search", - search: "search term", - typeFilter: "mode", - tagFilters: ["tag1", "tag2"] - } - ``` - -3. **addSource**: Add a new package source - - ```typescript - { - type: "addSource", - url: "https://github.com/username/repo.git", - name: "Custom Source" - } - ``` - -4. **removeSource**: Remove a package source - - ```typescript - { - type: "removeSource", - url: "https://github.com/username/repo.git" - } - ``` - -5. **refreshSources**: Refresh all sources - ```typescript - { - type: "refreshSources" - } - ``` - -#### Output Messages - -1. **items**: Response with all items - - ```typescript - { - type: "items", - data: [/* package items */] - } - ``` - -2. **searchResults**: Response with filtered items - - ```typescript - { - type: "searchResults", - data: [/* filtered items */] - } - ``` - -3. **sourceAdded**: Response after adding a source - - ```typescript - { - type: "sourceAdded", - data: { success: true } - } - ``` - -4. **error**: Error response - ```typescript - { - type: "error", - error: "Error message" - } - ``` - -### Asynchronous Processing - -The message handler manages asynchronous operations: - -1. Asynchronous methods return promises -2. Errors are caught and returned as error messages -3. Long-running operations can provide progress updates -4. The UI can display loading indicators during processing - -## UI Components - -The Package Manager includes several key UI components that render the interface and handle user interactions. - -### PackageManagerView - -The main container component that manages the overall UI: - -```tsx -const PackageManagerView: React.FC = () => { - const [items, setItems] = useState([]) - const [filters, setFilters] = useState({ type: "", search: "", tags: [] }) - const [activeTab, setActiveTab] = useState<"browse" | "sources">("browse") - - // Implementation details... - - return ( -
-
- - -
- - {activeTab === "browse" ? ( -
- -
- {items.map((item) => ( - - ))} -
-
- ) : ( - - )} -
- ) -} -``` - -### Component Interactions - -The UI components interact through props and state: - -1. **Parent-Child Communication**: - - - Parent components pass data and callbacks to children - - Children invoke callbacks to notify parents of events + - PackageManagerManager coordinates with GitFetcher + - GitFetcher manages repository state + - MetadataScanner processes repository content + - Results flow back to PackageManagerManager 2. **State Management**: - - Component state for UI-specific state - - Shared state for filters and active tab - - Backend state accessed through messages + - PackageManagerManager maintains backend state + - ViewStateManager handles UI state + - State changes trigger UI updates + - Components react to state changes -3. **Event Handling**: +3. **User Interactions**: - UI events trigger state updates - - State updates cause re-renders - - Messages are sent to the backend when needed - -### Accessibility Features - -The UI components include several accessibility features: - -1. **Keyboard Navigation**: - - - Tab order follows logical flow - - Focus indicators are visible - - Keyboard shortcuts for common actions - -2. **Screen Reader Support**: - - - ARIA attributes for dynamic content - - Semantic HTML structure - - Descriptive labels and announcements - -3. **Visual Accessibility**: - - High contrast mode support - - Resizable text - - Color schemes that work with color blindness - -## Component Integration - -The core components work together to provide a complete package management experience: - -### Initialization Flow - -1. The Package Manager is activated -2. The PackageManagerManager loads stored state -3. The UI sends an initial "getItems" message -4. The message handler calls PackageManagerManager.getItems() -5. The UI receives and displays the items - -### Search and Filter Flow - -1. The user enters a search term or selects filters -2. The UI sends a "search" message with the criteria -3. The message handler calls PackageManagerManager.filterItems() -4. The PackageManagerManager applies the filters -5. The UI receives and displays the filtered items - -### Source Management Flow - -1. The user adds a new source -2. The UI sends an "addSource" message -3. The message handler calls PackageManagerManager.addSource() -4. The PackageManagerManager adds the source and scans for items -5. The UI receives confirmation and updates the display + - ViewStateManager processes changes + - Changes propagate to backend + - Results update UI state ## Performance Optimizations -The core components include several performance optimizations: +The system includes several optimizations: -1. **Lazy Loading**: +1. **Concurrent Operations**: - - Items are loaded on demand - - Heavy operations are deferred - - Components render incrementally + - Operation queueing + - Source locking + - Parallel processing where safe + - Resource management -2. **Caching**: +2. **Efficient Caching**: - - Parsed metadata is cached - - Filter results can be cached - - Repository data is cached when possible + - Multi-level cache + - Cache invalidation + - Lazy loading + - Cache cleanup -3. **Efficient Filtering**: +3. **Smart Filtering**: - - Filtering happens on the backend - - Only necessary data is transferred - - Algorithms optimize for common cases + - Optimized algorithms + - Match tracking + - Incremental updates + - Result caching -4. **UI Optimizations**: - - Virtual scrolling for large lists - - Debounced search input - - Optimized rendering of complex components +4. **State Management**: + - Minimal updates + - State normalization + - Change batching + - Update optimization --- diff --git a/cline_docs/package-manager/implementation/03-data-structures.md b/cline_docs/package-manager/implementation/03-data-structures.md index b231c8a713..f6d295274e 100644 --- a/cline_docs/package-manager/implementation/03-data-structures.md +++ b/cline_docs/package-manager/implementation/03-data-structures.md @@ -22,131 +22,30 @@ These types represent the different kinds of components that can be managed by t 3. **package**: Collections of related components 4. **mcp server**: Model Context Protocol servers that provide additional functionality -The type system is extensible, allowing for new component types to be added in the future. +## Core Data Structures -## Metadata Interfaces - -The Package Manager uses a set of interfaces to define the structure of metadata for different components: - -### BaseMetadata +### PackageManagerRepository ```typescript /** - * Base metadata interface + * Represents a repository with its metadata and items */ -export interface BaseMetadata { - name: string - description: string - version: string - tags?: string[] +export interface PackageManagerRepository { + metadata: RepositoryMetadata + items: PackageManagerItem[] + url: string + defaultBranch: string + error?: string } ``` -This interface defines the common properties shared by all metadata types: +This interface represents a complete repository: -- **name**: The display name of the component -- **description**: A detailed explanation of the component's purpose -- **version**: The semantic version number -- **tags**: Optional array of relevant keywords - -### RepositoryMetadata - -```typescript -/** - * Repository root metadata - */ -export interface RepositoryMetadata extends BaseMetadata {} -``` - -This interface represents the metadata for a package source repository. It currently inherits all properties from BaseMetadata without adding additional fields, but is defined separately to allow for future repository-specific extensions. - -### ComponentMetadata - -```typescript -/** - * Component metadata with type - */ -export interface ComponentMetadata extends BaseMetadata { - type: ComponentType -} -``` - -This interface extends BaseMetadata to include a type field, which specifies the component type. - -### PackageMetadata - -```typescript -/** - * Package metadata with optional subcomponents - */ -export interface PackageMetadata extends ComponentMetadata { - type: "package" - items?: { - type: ComponentType - path: string - metadata?: ComponentMetadata - }[] -} -``` - -This interface represents packages that can contain subcomponents: - -- **type**: Always "package" for this interface -- **items**: Optional array of subcomponents, each with: - - **type**: The subcomponent type - - **path**: The file system path to the subcomponent - - **metadata**: Optional metadata for the subcomponent - -### SubcomponentMetadata - -```typescript -/** - * Subcomponent metadata with parent reference - */ -export interface SubcomponentMetadata extends ComponentMetadata { - parentPackage: { - name: string - path: string - } -} -``` - -This interface represents components that are part of a parent package: - -- All fields from ComponentMetadata -- **parentPackage**: Reference to the parent package - - **name**: The name of the parent package - - **path**: The file system path to the parent package - -## Item Structures - -The Package Manager uses several interfaces to represent items in the UI: - -### MatchInfo - -```typescript -/** - * Information about why an item matched search/filter criteria - */ -export interface MatchInfo { - matched: boolean - matchReason?: { - nameMatch?: boolean - descriptionMatch?: boolean - tagMatch?: boolean - hasMatchingSubcomponents?: boolean - } -} -``` - -This interface provides information about why an item matched search or filter criteria: - -- **matched**: Boolean indicating if the item matched -- **matchReason**: Optional object with specific match reasons - - **nameMatch**: True if the name matched - - **descriptionMatch**: True if the description matched - - **tagMatch**: True if a tag matched - - **hasMatchingSubcomponents**: True if a subcomponent matched +- **metadata**: The repository metadata +- **items**: Array of items in the repository +- **url**: The URL to the repository +- **defaultBranch**: The default Git branch (e.g., "main") +- **error**: Optional error message if there was a problem ### PackageManagerItem @@ -166,6 +65,7 @@ export interface PackageManagerItem { version?: string lastUpdated?: string sourceUrl?: string + defaultBranch?: string items?: { type: ComponentType path: string @@ -177,27 +77,145 @@ export interface PackageManagerItem { } ``` -This interface represents a complete package manager item as displayed in the UI: +Key changes: -- **name**: The display name of the item -- **description**: A detailed explanation of the item's purpose -- **type**: The component type -- **url**: The URL to the item's source -- **repoUrl**: The URL to the repository containing the item -- **sourceName**: Optional name of the source repository -- **author**: Optional author name -- **tags**: Optional array of relevant keywords -- **version**: Optional semantic version number -- **lastUpdated**: Optional date of last update -- **sourceUrl**: Optional URL to additional documentation -- **items**: Optional array of subcomponents -- **matchInfo**: Optional information about search/filter matches +- Added **defaultBranch** field for Git branch tracking +- Enhanced **matchInfo** structure for better filtering +- Improved subcomponent handling + +### MatchInfo + +```typescript +/** + * Information about why an item matched search/filter criteria + */ +export interface MatchInfo { + matched: boolean + matchReason?: { + nameMatch?: boolean + descriptionMatch?: boolean + typeMatch?: boolean + tagMatch?: boolean + hasMatchingSubcomponents?: boolean + } +} +``` + +Enhanced match tracking: + +- Added **typeMatch** for component type filtering +- More detailed match reasons +- Support for subcomponent matching + +## State Management Structures + +### ViewState + +```typescript +/** + * View-level state management + */ +interface ViewState { + items: PackageManagerItem[] + sortBy: "name" | "lastUpdated" + sortOrder: "asc" | "desc" + filters: Filters +} +``` + +Manages UI state: + +- Current items +- Sort configuration +- Filter state + +### Filters + +```typescript +/** + * Filter criteria + */ +interface Filters { + type: string + search: string + tags: string[] +} +``` + +Enhanced filtering: + +- Component type filtering +- Text search +- Tag-based filtering + +## Metadata Interfaces + +### BaseMetadata + +```typescript +/** + * Base metadata interface + */ +export interface BaseMetadata { + name: string + description: string + version: string + tags?: string[] +} +``` + +Common metadata properties: + +- **name**: Display name +- **description**: Detailed explanation +- **version**: Semantic version +- **tags**: Optional keywords + +### ComponentMetadata + +```typescript +/** + * Component metadata with type + */ +export interface ComponentMetadata extends BaseMetadata { + type: ComponentType + lastUpdated?: string +} +``` + +Added: + +- **lastUpdated** field for tracking changes + +### PackageMetadata + +```typescript +/** + * Package metadata with subcomponents + */ +export interface PackageMetadata extends ComponentMetadata { + type: "package" + items?: { + type: ComponentType + path: string + metadata?: ComponentMetadata + lastUpdated?: string + }[] +} +``` + +Enhanced with: + +- Subcomponent tracking +- Last update timestamps + +## Source Management ### PackageManagerSource ```typescript /** - * Represents a Git repository source for package manager items + * Git repository source */ export interface PackageManagerSource { url: string @@ -206,462 +224,202 @@ export interface PackageManagerSource { } ``` -This interface represents a package source repository: +Repository source configuration: -- **url**: The URL to the Git repository -- **name**: Optional display name for the source -- **enabled**: Boolean indicating if the source is active +- **url**: Git repository URL +- **name**: Optional display name +- **enabled**: Source status -### PackageManagerRepository +### SourceOperation ```typescript /** - * Represents a repository with its metadata and items + * Source operation tracking */ -export interface PackageManagerRepository { - metadata: RepositoryMetadata - items: PackageManagerItem[] +interface SourceOperation { url: string - error?: string + type: "clone" | "pull" | "refresh" + timestamp: number } ``` -This interface represents a complete repository with its metadata and items: +Tracks repository operations: -- **metadata**: The repository metadata -- **items**: Array of items in the repository -- **url**: The URL to the repository -- **error**: Optional error message if there was a problem loading the repository +- Operation type +- Timestamp +- Source URL -### LocalizedMetadata +## Cache Management + +### CacheEntry ```typescript /** - * Utility type for metadata files with locale + * Cache entry structure */ -export type LocalizedMetadata = { - [locale: string]: T +interface CacheEntry { + data: T + timestamp: number } ``` -This utility type represents metadata that can be localized to different languages: +Generic cache structure: -- **[locale: string]**: Keys are locale identifiers (e.g., "en", "fr") -- **T**: The type of metadata being localized +- Cached data +- Timestamp for expiry -## UI Component Props - -The Package Manager UI components use several prop interfaces: - -### PackageManagerItemCardProps +### RepositoryCache ```typescript -interface PackageManagerItemCardProps { - item: PackageManagerItem - filters: { type: string; search: string; tags: string[] } - setFilters: React.Dispatch> - activeTab: "browse" | "sources" - setActiveTab: React.Dispatch> -} +/** + * Repository cache management + */ +type RepositoryCache = Map> ``` -This interface defines the props for the PackageManagerItemCard component: +Specialized for repositories: -- **item**: The package item to display -- **filters**: The current filter state -- **setFilters**: Function to update filters -- **activeTab**: The currently active tab -- **setActiveTab**: Function to change the active tab - -### ExpandableSectionProps - -```typescript -interface ExpandableSectionProps { - title: string - children: React.ReactNode - className?: string - defaultExpanded?: boolean - badge?: string -} -``` - -This interface defines the props for the ExpandableSection component: - -- **title**: The section header text -- **children**: The content to display when expanded -- **className**: Optional CSS class name -- **defaultExpanded**: Optional flag to set initial expanded state -- **badge**: Optional badge text to display - -### TypeGroupProps - -```typescript -interface TypeGroupProps { - type: string - items: Array<{ - name: string - description?: string - metadata?: any - path?: string - }> - className?: string - searchTerm?: string -} -``` - -This interface defines the props for the TypeGroup component: - -- **type**: The component type to display -- **items**: Array of items of this type -- **className**: Optional CSS class name -- **searchTerm**: Optional search term for highlighting matches - -## Grouped Items Structure - -The Package Manager uses a specialized structure for grouping items by type: - -### GroupedItems - -```typescript -export interface GroupedItems { - [type: string]: { - type: string - items: Array<{ - name: string - description?: string - metadata?: any - path?: string - }> - } -} -``` - -This interface represents items grouped by their type: - -- **[type: string]**: Keys are component types -- **type**: The component type (redundant with the key) -- **items**: Array of items of this type - - **name**: The item name - - **description**: Optional item description - - **metadata**: Optional additional metadata - - **path**: Optional file system path - -## Filter and Sort Structures - -The Package Manager uses several structures for filtering and sorting: - -### Filters - -```typescript -interface Filters { - type: string - search: string - tags: string[] -} -``` - -This interface represents the filter criteria: - -- **type**: The component type filter -- **search**: The search term -- **tags**: Array of tag filters - -### SortConfig - -```typescript -interface SortConfig { - by: string - order: "asc" | "desc" -} -``` - -This interface represents the sort configuration: - -- **by**: The field to sort by (e.g., "name", "author") -- **order**: The sort order ("asc" for ascending, "desc" for descending) +- URL-based lookup +- Timestamp-based expiry +- Full repository data ## Message Structures -The Package Manager uses a message-based architecture for communication: - ### Input Messages ```typescript -// Get all items -{ type: "getItems" } - -// Apply search and filter criteria -{ - type: "search", - search: string, - typeFilter: string, - tagFilters: string[] -} - -// Add a new package source -{ - type: "addSource", - url: string, - name?: string -} - -// Remove a package source -{ - type: "removeSource", - url: string -} - -// Refresh all sources -{ type: "refreshSources" } +type PackageManagerMessage = + | { type: "getItems" } + | { + type: "search" + search: string + typeFilter: string + tagFilters: string[] + } + | { + type: "addSource" + url: string + name?: string + } + | { + type: "removeSource" + url: string + } + | { type: "refreshSources" } ``` ### Output Messages ```typescript -// Response with all items -{ - type: "items", - data: PackageManagerItem[] -} - -// Response with filtered items -{ - type: "searchResults", - data: PackageManagerItem[] -} - -// Response after adding a source -{ - type: "sourceAdded", - data: { success: boolean } -} - -// Error response -{ - type: "error", - error: string -} +type PackageManagerResponse = + | { + type: "items" + data: PackageManagerItem[] + } + | { + type: "searchResults" + data: PackageManagerItem[] + filters: Filters + } + | { + type: "sourceAdded" | "sourceRemoved" + data: { success: boolean } + } + | { + type: "error" + error: string + } ``` -## Template Structure +Enhanced with: -The Package Manager uses a specific directory structure for templates: - -### Basic Template Structure - -``` -package-manager-template/ -├── metadata.en.yml # Repository metadata -├── README.md # Repository documentation -├── packages/ # Directory for package components -│ └── data-platform/ # Example package -│ └── metadata.en.yml # Package metadata -├── modes/ # Directory for mode components -│ └── developer-mode/ # Example mode -│ └── metadata.en.yml # Mode metadata -├── mcp servers/ # Directory for MCP server components -│ ├── example-server/ # Example server -│ │ └── metadata.en.yml # Server metadata -│ └── file-analyzer/ # Another example server -│ └── metadata.en.yml # Server metadata -└── groups/ # Directory for grouping components - └── data-engineering/ # Example group - └── metadata.en.yml # Group metadata -``` - -### Metadata File Structure - -```yaml -# Repository metadata (metadata.en.yml) -name: "Package Manager Template" -description: "A template repository for creating package manager sources" -version: "1.0.0" - -# Component metadata (e.g., modes/developer-mode/metadata.en.yml) -name: "Developer Mode" -description: "A specialized mode for software development tasks" -version: "1.0.0" -type: "mode" -tags: - - development - - coding - - software -``` - -## Data Flow and Transformations - -The Package Manager transforms data through several stages: - -### From File System to Metadata - -1. Raw YAML files are read from the file system -2. YAML is parsed into JavaScript objects -3. Objects are validated against metadata interfaces -4. Localized metadata is combined into a single structure - -### From Metadata to Items - -1. Metadata objects are transformed into PackageManagerItem objects -2. File paths are converted to URLs -3. Parent-child relationships are established -4. Additional information is added (e.g., lastUpdated) - -### From Items to UI - -1. Items are filtered based on user criteria -2. Match information is added to items -3. Items are sorted according to user preferences -4. Items are grouped by type for display +- Filter state in search results +- Operation success tracking +- Detailed error reporting ## Data Validation -The Package Manager includes validation at several levels: - ### Metadata Validation ```typescript -function validateMetadata(metadata: any): boolean { - // Required fields - if (!metadata.name || !metadata.description || !metadata.version) { - return false - } +/** + * Validate component metadata + */ +function validateMetadata(metadata: unknown): metadata is ComponentMetadata { + if (!isObject(metadata)) return false - // Type validation for components - if (metadata.type && !["mode", "prompt", "package", "mcp server"].includes(metadata.type)) { - return false - } - - // Additional validation... - - return true + return ( + typeof metadata.name === "string" && + typeof metadata.description === "string" && + typeof metadata.version === "string" && + (metadata.tags === undefined || Array.isArray(metadata.tags)) && + isValidComponentType(metadata.type) + ) } ``` ### URL Validation ```typescript -function isValidUrl(urlString: string): boolean { - try { - new URL(urlString) - return true - } catch (e) { - return false - } +/** + * Validate Git repository URL + */ +function isValidGitUrl(url: string): boolean { + if (!url) return false + + // Support common Git URL formats + return /^(https?:\/\/|git@)/.test(url) && /\.git$/.test(url) } ``` -### Tag Validation +## Data Flow -```typescript -function validateTags(tags: any[]): string[] { - if (!Array.isArray(tags)) { - return [] - } +The Package Manager transforms data through several stages: - return tags.filter((tag) => typeof tag === "string" && tag.trim().length > 0).map((tag) => tag.trim()) -} -``` +1. **Repository Level**: + + - Clone/pull Git repositories + - Parse metadata files + - Build component hierarchy + +2. **Cache Level**: + + - Store repository data + - Track timestamps + - Handle expiration + +3. **View Level**: + - Apply filters + - Sort items + - Track matches + - Manage UI state ## Data Relationships -The Package Manager maintains several important relationships between data structures: - -### Parent-Child Relationships - -Packages can contain subcomponents, creating a hierarchical structure: +### Component Hierarchy ``` -Package -├── Mode -├── MCP Server -├── Prompt -└── Nested Package - ├── Mode - └── MCP Server +Repository +├── Metadata +└── Items + ├── Package + │ ├── Mode + │ ├── MCP Server + │ └── Prompt + └── Standalone Components ``` -This relationship is represented in the data structures: +### State Flow -- Packages have an `items` array containing subcomponents -- Subcomponents have a `parentPackage` reference - -### Source-Item Relationships - -Items are associated with their source repositories: - -- Each item has a `repoUrl` field pointing to its source -- Sources have a list of items they provide -- When a source is disabled, its items are hidden - -### Type-Group Relationships - -Items are grouped by their type for display: - -- The `GroupedItems` interface organizes items by type -- Each type group contains items of that type -- The UI displays these groups separately - -## Serialization and Persistence - -The Package Manager serializes data for persistence: - -### Source Persistence - -```typescript -// Save sources to extension state -private saveState(): void { - this.context.globalState.update("packageManagerSources", this.sources); -} - -// Load sources from extension state -private loadState(): void { - const savedSources = this.context.globalState.get("packageManagerSources", []); - this.sources = savedSources; -} +``` +Git Repository → Cache → PackageManager → ViewState → UI ``` -### Metadata Caching +### Filter Chain -```typescript -// Cache metadata to improve performance -private cacheMetadata(url: string, metadata: any): void { - const cacheKey = `metadata_${url}`; - this.context.globalState.update(cacheKey, { - timestamp: Date.now(), - data: metadata - }); -} - -// Retrieve cached metadata -private getCachedMetadata(url: string): any | null { - const cacheKey = `metadata_${url}`; - const cached = this.context.globalState.get(cacheKey); - - if (!cached || Date.now() - cached.timestamp > CACHE_TTL) { - return null; - } - - return cached.data; -} ``` - -## Data Structure Evolution - -The Package Manager's data structures are designed for evolution: - -### Versioning Strategy - -- Interfaces include version fields -- New fields are added as optional -- Breaking changes are avoided when possible -- Migration code handles legacy data formats - -### Extensibility Points - -- The ComponentType can be extended with new types -- Metadata interfaces can be extended with new fields -- The message system can handle new message types -- The UI can adapt to display new data formats +Raw Items → Type Filter → Search Filter → Tag Filter → Sorted Results +``` --- diff --git a/cline_docs/package-manager/implementation/04-search-and-filter.md b/cline_docs/package-manager/implementation/04-search-and-filter.md index e768b09ca6..30971f8a03 100644 --- a/cline_docs/package-manager/implementation/04-search-and-filter.md +++ b/cline_docs/package-manager/implementation/04-search-and-filter.md @@ -2,767 +2,302 @@ This document details the implementation of search and filtering functionality in the Package Manager, including algorithms, optimization techniques, and performance considerations. -## Search Algorithm +## Core Filter System -The Package Manager implements a comprehensive search algorithm that matches user queries against multiple fields and supports hierarchical component structures. +The Package Manager implements a comprehensive filtering system that handles multiple filter types, concurrent operations, and detailed match tracking. -### Search Term Matching - -The core of the search functionality is the `containsSearchTerm` function, which checks if a string contains a search term: +### Filter Implementation ```typescript /** - * Checks if a string contains a search term (case insensitive) - * @param text The text to search in - * @param searchTerm The term to search for - * @returns True if the text contains the search term + * Filter items based on criteria with match tracking */ -export function containsSearchTerm(text: string | undefined, searchTerm: string): boolean { - if (!text || !searchTerm) { - return false - } +export function filterItems( + items: PackageManagerItem[], + filters: { + type?: ComponentType + search?: string + tags?: string[] + }, +): PackageManagerItem[] { + // Helper function to normalize text for case-insensitive comparison + const normalizeText = (text: string) => text.toLowerCase().replace(/\s+/g, " ").trim() - return text.toLowerCase().includes(searchTerm.toLowerCase()) + // Normalize search term once + const searchTerm = filters.search ? normalizeText(filters.search) : "" + + // Create a deep clone of items + const clonedItems = items.map((item) => JSON.parse(JSON.stringify(item)) as PackageManagerItem) + + return clonedItems + .filter((item) => { + // Check parent item matches + const itemMatches = { + type: !filters.type || item.type === filters.type, + search: + !searchTerm || + containsSearchTerm(item.name, searchTerm) || + containsSearchTerm(item.description, searchTerm), + tags: !filters.tags?.length || (item.tags && filters.tags.some((tag) => item.tags!.includes(tag))), + } + + // Check subcomponent matches + const subcomponentMatches = + item.items?.some((subItem) => { + const subMatches = { + type: !filters.type || subItem.type === filters.type, + search: + !searchTerm || + (subItem.metadata && + (containsSearchTerm(subItem.metadata.name, searchTerm) || + containsSearchTerm(subItem.metadata.description, searchTerm))), + tags: + !filters.tags?.length || + (subItem.metadata?.tags && + filters.tags.some((tag) => subItem.metadata!.tags!.includes(tag))), + } + + return ( + subMatches.type && + (!searchTerm || subMatches.search) && + (!filters.tags?.length || subMatches.tags) + ) + }) ?? false + + // Include item if either parent matches or has matching subcomponents + const hasActiveFilters = filters.type || searchTerm || filters.tags?.length + if (!hasActiveFilters) return true + + const parentMatchesAll = itemMatches.type && itemMatches.search && itemMatches.tags + const isPackageWithMatchingSubcomponent = item.type === "package" && subcomponentMatches + + return parentMatchesAll || isPackageWithMatchingSubcomponent + }) + .map((item) => addMatchInfo(item, filters)) } ``` -This function: +### Match Tracking -- Handles undefined inputs gracefully -- Performs case-insensitive matching -- Uses JavaScript's native `includes` method for performance - -### Item Search Implementation - -The main search function applies the search term to multiple fields: +The system tracks detailed match information: ```typescript /** - * Checks if an item matches a search term - * @param item The item to check - * @param searchTerm The search term - * @returns Match information + * Add match information to items */ -function itemMatchesSearch(item: PackageManagerItem, searchTerm: string): MatchInfo { - if (!searchTerm) { - return { matched: true } +function addMatchInfo(item: PackageManagerItem, filters: Filters): PackageManagerItem { + const matchReason: Record = { + nameMatch: filters.search ? containsSearchTerm(item.name, filters.search) : true, + descriptionMatch: filters.search ? containsSearchTerm(item.description, filters.search) : true, + typeMatch: filters.type ? item.type === filters.type : true, + tagMatch: filters.tags?.length ? hasMatchingTags(item.tags, filters.tags) : true, } - const term = searchTerm.toLowerCase() - - // Check main item fields - const nameMatch = containsSearchTerm(item.name, term) - const descriptionMatch = containsSearchTerm(item.description, term) - const authorMatch = containsSearchTerm(item.author, term) - - // Check subcomponents - let hasMatchingSubcomponents = false - - if (item.items?.length) { - hasMatchingSubcomponents = item.items.some( - (subItem) => - containsSearchTerm(subItem.metadata?.name, term) || - containsSearchTerm(subItem.metadata?.description, term), - ) - - // Add match info to subcomponents - item.items.forEach((subItem) => { - const subNameMatch = containsSearchTerm(subItem.metadata?.name, term) - const subDescMatch = containsSearchTerm(subItem.metadata?.description, term) + // Process subcomponents + if (item.items) { + item.items = item.items.map((subItem) => { + const subMatches = { + type: !filters.type || subItem.type === filters.type, + search: + !filters.search || + (subItem.metadata && + (containsSearchTerm(subItem.metadata.name, filters.search) || + containsSearchTerm(subItem.metadata.description, filters.search))), + tags: + !filters.tags?.length || + (subItem.metadata?.tags && filters.tags.some((tag) => subItem.metadata!.tags!.includes(tag))), + } subItem.matchInfo = { - matched: subNameMatch || subDescMatch, - matchReason: - subNameMatch || subDescMatch - ? { - nameMatch: subNameMatch, - descriptionMatch: subDescMatch, - } - : undefined, + matched: subMatches.type && subMatches.search && subMatches.tags, + matchReason: { + typeMatch: subMatches.type, + nameMatch: subMatches.search, + tagMatch: subMatches.tags, + }, } + + return subItem }) } - const matched = nameMatch || descriptionMatch || authorMatch || hasMatchingSubcomponents - return { - matched, - matchReason: matched - ? { - nameMatch, - descriptionMatch, - authorMatch, - hasMatchingSubcomponents, - } - : undefined, + ...item, + matchInfo: { + matched: Object.values(matchReason).every(Boolean), + matchReason, + }, } } ``` -This function: +## Sort System -- Checks the item's name, description, and author -- Recursively checks subcomponents -- Adds match information to both the item and its subcomponents -- Returns detailed match information +The Package Manager implements flexible sorting with subcomponent support: -### Search Optimization Techniques +```typescript +/** + * Sort items with subcomponent support + */ +export function sortItems( + items: PackageManagerItem[], + sortBy: "name" | "lastUpdated" | "author", + sortOrder: "asc" | "desc", + sortSubcomponents: boolean = false, +): PackageManagerItem[] { + return [...items] + .map((item) => { + const clonedItem = { ...item } -The search implementation includes several optimizations: + if (clonedItem.items && sortSubcomponents) { + clonedItem.items = [...clonedItem.items].sort((a, b) => { + const aValue = getSortValue(a, sortBy) + const bValue = getSortValue(b, sortBy) + return compareValues(aValue, bValue, sortOrder) + }) + } + + return clonedItem + }) + .sort((a, b) => { + const aValue = getSortValue(a, sortBy) + const bValue = getSortValue(b, sortBy) + return compareValues(aValue, bValue, sortOrder) + }) +} +``` + +## State Management Integration + +The filtering system integrates with the state management: + +```typescript +export class PackageManagerViewStateManager { + private items: PackageManagerItem[] = [] + private sortBy: "name" | "lastUpdated" = "name" + private sortOrder: "asc" | "desc" = "asc" + private filters: Filters = { type: "", search: "", tags: [] } + + /** + * Get filtered and sorted items + */ + getFilteredAndSortedItems(): PackageManagerItem[] { + const filtered = filterItems(this.items, this.filters) + return sortItems(filtered, this.sortBy, this.sortOrder) + } + + /** + * Update filters with optimistic updates + */ + setFilters(newFilters: Partial): void { + this.filters = { ...this.filters, ...newFilters } + } +} +``` + +## Performance Optimizations + +### Concurrent Operation Handling + +```typescript +export class PackageManagerManager { + private isMetadataScanActive = false + private pendingOperations: Array<() => Promise> = [] + + /** + * Queue filter operations during active scans + */ + private async queueOperation(operation: () => Promise): Promise { + if (this.isMetadataScanActive) { + return new Promise((resolve) => { + this.pendingOperations.push(async () => { + await operation() + resolve() + }) + }) + } + + try { + this.isMetadataScanActive = true + await operation() + } finally { + this.isMetadataScanActive = false + + const nextOperation = this.pendingOperations.shift() + if (nextOperation) { + void this.queueOperation(nextOperation) + } + } + } +} +``` + +### Filter Optimizations 1. **Early Termination**: - Returns as soon as any field matches - - Avoids unnecessary checks after a match is found + - Avoids unnecessary checks + - Handles empty filters efficiently 2. **Efficient String Operations**: - - Uses native string methods for performance - - Converts to lowercase once per string - - Avoids regular expressions for simple matching - -3. **Match Caching**: - - - Stores match information on items - - Avoids recalculating matches for the same search term - - Clears cache when the search term changes - -4. **Lazy Evaluation**: - - Only checks subcomponents if main fields don't match - - Processes subcomponents only when necessary - -## Filter Logic - -The Package Manager implements multiple filter types that can be combined to narrow down results. - -### Type Filtering - -Type filtering restricts results to components of a specific type: - -```typescript -/** - * Filters items by type - * @param items Items to filter - * @param type Type to filter by - * @returns Filtered items - */ -function filterByType(items: PackageManagerItem[], type: string): PackageManagerItem[] { - if (!type) { - return items - } - - return items.filter((item) => item.type === type) -} -``` - -### Tag Filtering - -Tag filtering shows only items with specific tags: - -```typescript -/** - * Filters items by tags - * @param items Items to filter - * @param tags Tags to filter by - * @returns Filtered items - */ -function filterByTags(items: PackageManagerItem[], tags: string[]): PackageManagerItem[] { - if (!tags.length) { - return items - } - - return items.filter((item) => { - if (!item.tags?.length) { - return false - } - - // Item must have at least one of the specified tags - return item.tags.some((tag) => tags.includes(tag)) - }) -} -``` - -### Combined Filtering - -The main filter function combines all filter types: - -```typescript -/** - * Filters items based on criteria - * @param items Items to filter - * @param filters Filter criteria - * @returns Filtered items - */ -export function filterItems( - items: PackageManagerItem[], - filters: { type?: string; search?: string; tags?: string[] }, -): PackageManagerItem[] { - if (!isFilterActive(filters)) { - return items - } - - let result = items - - // Apply type filter - if (filters.type) { - result = filterByType(result, filters.type) - } - - // Apply search filter - if (filters.search) { - result = result.filter((item) => { - const matchInfo = itemMatchesSearch(item, filters.search!) - item.matchInfo = matchInfo - return matchInfo.matched - }) - } - - // Apply tag filter - if (filters.tags?.length) { - result = filterByTags(result, filters.tags) - } - - return result -} -``` - -This function: - -- Applies filters in a specific order (type, search, tags) -- Short-circuits if no filters are active -- Adds match information to items -- Returns a new array with filtered items - -### Filter Optimization Techniques - -The filter implementation includes several optimizations: - -1. **Filter Order**: - - - Applies the most restrictive filters first - - Reduces the number of items for subsequent filters - - Improves performance for large datasets - -2. **Short-Circuit Evaluation**: - - - Skips filtering entirely if no filters are active - - Returns early when possible - -3. **Immutable Operations**: - - - Creates new arrays rather than modifying existing ones - - Ensures predictable behavior - - Supports undo/redo functionality - -4. **Selective Processing**: - - Only processes necessary fields for each filter - - Avoids redundant calculations - -## Selector Functions - -The Package Manager uses selector functions to extract and transform data for the UI: - -### Filter Status Selector - -```typescript -/** - * Checks if any filters are active - * @param filters Filter criteria - * @returns True if any filters are active - */ -export const isFilterActive = (filters: Filters): boolean => { - return !!(filters.type || filters.search || filters.tags.length > 0) -} -``` - -### Display Items Selector - -```typescript -/** - * Gets items for display based on filters and sort config - * @param items All items - * @param filters Filter criteria - * @param sortConfig Sort configuration - * @returns Filtered and sorted items - */ -export const getDisplayedItems = ( - items: PackageManagerItem[], - filters: Filters, - sortConfig: SortConfig, -): PackageManagerItem[] => { - const filteredItems = filterItems(items, filters) - return sortItems(filteredItems, sortConfig) -} -``` - -### Sort Function - -```typescript -/** - * Sorts items based on configuration - * @param items Items to sort - * @param config Sort configuration - * @returns Sorted items - */ -export const sortItems = (items: PackageManagerItem[], config: SortConfig): PackageManagerItem[] => { - return [...items].sort((a, b) => { - let comparison = 0 - - switch (config.by) { - case "name": - comparison = a.name.localeCompare(b.name) - break - case "author": - comparison = (a.author || "").localeCompare(b.author || "") - break - case "lastUpdated": - comparison = (a.lastUpdated || "").localeCompare(b.lastUpdated || "") - break - default: - comparison = a.name.localeCompare(b.name) - } - - return config.order === "asc" ? comparison : -comparison - }) -} -``` - -## Grouping Implementation - -The Package Manager includes functionality to group items by type: - -### Group By Type Function - -```typescript -/** - * Groups package items by their type - * @param items Array of items to group - * @returns Object with items grouped by type - */ -export function groupItemsByType(items: PackageManagerItem["items"] = []): GroupedItems { - if (!items?.length) { - return {} - } - - return items.reduce((groups: GroupedItems, item) => { - if (!item.type) { - return groups - } - - if (!groups[item.type]) { - groups[item.type] = { - type: item.type, - items: [], - } - } - - groups[item.type].items.push({ - name: item.metadata?.name || "Unnamed item", - description: item.metadata?.description, - metadata: item.metadata, - path: item.path, - }) - - return groups - }, {}) -} -``` - -### Helper Functions - -```typescript -/** - * Gets the total number of items across all groups - * @param groups Grouped items object - * @returns Total number of items - */ -export function getTotalItemCount(groups: GroupedItems): number { - return Object.values(groups).reduce((total, group) => total + group.items.length, 0) -} - -/** - * Gets an array of unique types from the grouped items - * @param groups Grouped items object - * @returns Array of type strings - */ -export function getUniqueTypes(groups: GroupedItems): string[] { - return Object.keys(groups).sort() -} -``` - -## UI Integration - -The search and filter functionality is integrated with the UI through several components: - -### Search Input Component - -```tsx -const SearchInput: React.FC<{ - value: string - onChange: (value: string) => void -}> = ({ value, onChange }) => { - // Debounce search input to avoid excessive filtering - const debouncedOnChange = useDebounce(onChange, 300) - - return ( -
- - debouncedOnChange(e.target.value)} - placeholder="Search packages..." - className="search-input" - aria-label="Search packages" - /> - {value && ( - - )} -
- ) -} -``` - -### Type Filter Component - -```tsx -const TypeFilter: React.FC<{ - value: string - onChange: (value: string) => void - types: string[] -}> = ({ value, onChange, types }) => { - return ( -
-

Filter by Type

-
- - - {types.map((type) => ( - - ))} -
-
- ) -} -``` - -### Tag Filter Component - -```tsx -const TagFilter: React.FC<{ - selectedTags: string[] - onChange: (tags: string[]) => void - availableTags: string[] -}> = ({ selectedTags, onChange, availableTags }) => { - const toggleTag = (tag: string) => { - if (selectedTags.includes(tag)) { - onChange(selectedTags.filter((t) => t !== tag)) - } else { - onChange([...selectedTags, tag]) - } - } - - return ( -
-

Filter by Tags

-
- {availableTags.map((tag) => ( - - ))} -
-
- ) -} -``` - -## Performance Considerations - -The search and filter implementation includes several performance optimizations: - -### Large Dataset Handling - -For large datasets, the Package Manager implements: - -1. **Pagination**: - - - Limits the number of items displayed at once - - Implements virtual scrolling for smooth performance - - Loads additional items as needed - -2. **Progressive Loading**: - - - Shows initial results quickly - - Loads additional details asynchronously - - Provides visual feedback during loading - -3. **Background Processing**: - - Performs heavy operations in a web worker - - Keeps the UI responsive during filtering - - Updates results incrementally - -### Search Optimizations - -For efficient searching: - -1. **Debounced Input**: - - ```typescript - function useDebounce(value: T, delay: number): T { - const [debouncedValue, setDebouncedValue] = useState(value) - - useEffect(() => { - const timer = setTimeout(() => { - setDebouncedValue(value) - }, delay) - - return () => { - clearTimeout(timer) - } - }, [value, delay]) - - return debouncedValue - } - ``` - -2. **Incremental Matching**: - - - Matches characters in sequence - - Prioritizes prefix matches - - Supports fuzzy matching for better results - -3. **Result Highlighting**: - - Highlights matching text portions - - Provides visual feedback on match quality - - Improves user understanding of results - -### Filter Combinations - -For efficient filter combinations: - -1. **Filter Order Optimization**: - - - Applies most restrictive filters first - - Reduces dataset size early in the pipeline - - Improves performance for complex filter combinations - -2. **Filter Caching**: - - - Caches results for recent filter combinations - - Avoids recomputing the same filters - - Clears cache when underlying data changes - -3. **Progressive Filtering**: - - Shows initial results based on simple filters - - Applies complex filters incrementally - - Provides feedback during filtering process - -## Edge Cases and Error Handling - -The search and filter implementation handles several edge cases: - -### Empty Results - -When no items match the filters: - -```tsx -const NoResults: React.FC<{ - filters: Filters - clearFilters: () => void -}> = ({ filters, clearFilters }) => { - return ( -
- -

No matching packages found

-

- No packages match your current filters. - {isFilterActive(filters) && ( - <> -
- - - )} -

-
- ) -} -``` - -### Invalid Search Terms - -The system handles invalid search terms: - -- Empty searches show all items -- Special characters are escaped -- Very long search terms are truncated -- Malformed regex patterns are handled safely - -### Filter Conflicts - -When filters conflict: - -- Shows a warning when appropriate -- Provides suggestions to resolve conflicts -- Falls back to reasonable defaults -- Preserves user intent when possible + - Normalizes text once + - Uses native string methods + - Avoids regex for simple matches + +3. **State Management**: + - Optimistic updates + - Batched filter changes + - Efficient re-renders ## Testing Strategy -The search and filter functionality includes comprehensive tests: - -### Unit Tests - ```typescript -describe("Search Utils", () => { - describe("containsSearchTerm", () => { - it("should return true for exact matches", () => { - expect(containsSearchTerm("hello world", "hello")).toBe(true) +describe("Filter System", () => { + describe("Match Tracking", () => { + it("should track type matches", () => { + const result = filterItems([testItem], { type: "mode" }) + expect(result[0].matchInfo.matchReason.typeMatch).toBe(true) }) - it("should be case insensitive", () => { - expect(containsSearchTerm("Hello World", "hello")).toBe(true) - expect(containsSearchTerm("hello world", "WORLD")).toBe(true) - }) - - it("should handle undefined inputs", () => { - expect(containsSearchTerm(undefined, "test")).toBe(false) - expect(containsSearchTerm("test", "")).toBe(false) + it("should track subcomponent matches", () => { + const result = filterItems([testPackage], { search: "test" }) + const subItem = result[0].items![0] + expect(subItem.matchInfo.matched).toBe(true) }) }) - describe("filterItems", () => { - const items = [ - { - name: "Test Package", - description: "A test package", - type: "package", - tags: ["test", "example"], - }, - { - name: "Another Package", - description: "Another test package", - type: "mode", - tags: ["example"], - }, - ] - - it("should filter by type", () => { - const result = filterItems(items, { type: "package" }) - expect(result).toHaveLength(1) - expect(result[0].name).toBe("Test Package") - }) - - it("should filter by search term", () => { - const result = filterItems(items, { search: "another" }) - expect(result).toHaveLength(1) - expect(result[0].name).toBe("Another Package") - }) - - it("should filter by tags", () => { - const result = filterItems(items, { tags: ["test"] }) - expect(result).toHaveLength(1) - expect(result[0].name).toBe("Test Package") - }) - - it("should combine filters", () => { - const result = filterItems(items, { - type: "package", - tags: ["example"], - }) - expect(result).toHaveLength(1) - expect(result[0].name).toBe("Test Package") + describe("Sort System", () => { + it("should sort subcomponents", () => { + const result = sortItems([testPackage], "name", "asc", true) + expect(result[0].items).toBeSorted((a, b) => a.metadata.name.localeCompare(b.metadata.name)) }) }) }) ``` -### Integration Tests +## Error Handling -```typescript -describe("Package Manager Search Integration", () => { - let manager: PackageManagerManager - let metadataScanner: MetadataScanner - let templateItems: PackageManagerItem[] +The system includes robust error handling: - beforeAll(async () => { - // Load real data from template - metadataScanner = new MetadataScanner() - const templatePath = path.resolve(__dirname, "../../../../package-manager-template") - templateItems = await metadataScanner.scanDirectory(templatePath, "https://example.com") - }) +1. **Filter Errors**: - beforeEach(() => { - // Create a real context-like object - const context = { - extensionPath: path.resolve(__dirname, "../../../../"), - globalStorageUri: { fsPath: path.resolve(__dirname, "../../../../mock/settings/path") }, - } as vscode.ExtensionContext + - Invalid filter types + - Malformed search terms + - Missing metadata - // Create real instances - manager = new PackageManagerManager(context) +2. **Sort Errors**: - // Set up manager with template data - manager["currentItems"] = [...templateItems] - }) + - Invalid sort fields + - Missing sort values + - Type mismatches - it("should find items by name", () => { - const message = { - type: "search", - search: "data platform", - typeFilter: "", - tagFilters: [], - } +3. **State Errors**: + - Concurrent updates + - Invalid state transitions + - Cache inconsistencies - const result = handlePackageManagerMessages(message, manager) - expect(result.data).toHaveLength(1) - expect(result.data[0].name).toContain("Data Platform") - }) - - it("should find items with matching subcomponents", () => { - const message = { - type: "search", - search: "validator", - typeFilter: "", - tagFilters: [], - } - - const result = handlePackageManagerMessages(message, manager) - expect(result.data.length).toBeGreaterThan(0) - - // Check that subcomponents are marked as matches - const hasMatchingSubcomponent = result.data.some((item) => - item.items?.some((subItem) => subItem.matchInfo?.matched), - ) - expect(hasMatchingSubcomponent).toBe(true) - }) -}) -``` +--- **Previous**: [Data Structures](./03-data-structures.md) | **Next**: [UI Component Design](./05-ui-components.md)