2 commits

Author	SHA1	Message	Date
SannidhyaSah	a80795b78d	feat: Add OpenAI Compatible embedder for codebase indexing (#4066) ... * feat: Add OpenAI Compatible embedder for codebase indexing - Implement OpenAiCompatibleEmbedder with batching and retry logic - Add configuration support for base URL and API key - Update UI with provider selection and input fields - Add comprehensive test coverage - Support for all OpenAI-compatible endpoints (LiteLLM, LMStudio, Ollama, etc.) - Add internationalization for 17 languages * fix: Update CodeIndexSettings tests for OpenAI Compatible provider - Fix field count expectations (4 fields including Qdrant) - Use specific test IDs for button selection - Fix input handling with clear() before type() - Use toHaveBeenLastCalledWith for better assertions - Fix status text matching with regex pattern * fix: resolve UI test failures and ESLint errors - Remove unused waitFor import to fix ESLint error - Fix test expectations to match actual component behavior for input fields - Simplify provider selection test by removing complex mock interactions - All CodeIndexSettings tests now pass (20/20) * feat: add custom model infrastructure for OpenAI-compatible embedder - Add manual model ID and embedding dimension configuration - Enable custom model input via text field in settings UI - Add modelDimension parameter to OpenAiCompatibleEmbedder - Update configuration management to persist dimension setting - Prioritize manual dimension over hardcoded model profiles - Add comprehensive test coverage for new functionality This allows users to specify any custom embedding model and its dimension for OpenAI-compatible providers, removing dependency on hardcoded model profiles. * Add missing translations for OpenAI-compatible model dimension settings in all locales * refactor: remove unused modelDimension parameter from OpenAiCompatibleEmbedder - Remove modelDimension property and constructor parameter from OpenAiCompatibleEmbedder class - Update ServiceFactory to not pass dimension to embedder constructor - Update tests to match new constructor signature - The dimension is still used for QdrantVectorStore configuration * chore: bot suggestion Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * chore: bot suggestion Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * refactor: rename OpenAiCompatibleEmbedder to OpenAICompatibleEmbedder for consistency * feat: add model dimension validation for OpenAI-compatible settings * refactor: improve default model ID retrieval logic for embedding providers * feat: add default model ID retrieval for openai-compatible provider * refactor: update default model ID retrieval to use shared utility function * fix: Remove unnecessary type assertion in OpenAICompatibleEmbedder * feat: add model dimension input for openai-compatible provider --------- Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com> Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>	2025-06-04 17:00:52 -04:00
Daniel	61e122dc34	Codebase indexing (Clean history) (#3137) ... * feat: apply changes from local main * fix: add missing types * feat: deduplicate code blocks coming out of parser * feat: implement a cache manager to improve cache handling * refactor: move code index service initialization to extension and remove await from indexing process * fix: return undefined instead of throwing if no workspace is detected * feat: allow auto approve if it is active for read tools * refactor: improve UI of the results and allow opening the ranges directly in the editor * refactor: use dependency injection to improve performance * feat: implement result filtering by directory path * refactor: centralize path normalization logic * refactor: remove unnecessary barrel file * refactor: prevent restarting the service if no settings change * fix: the indexing process should never be awaited * refactor: cleanup unused method * refactor: remove batch limits for ollama * refactor(parser): simplify method signatures and improve chunking logic - Remove redundant min/max chars parameters - Add better handling for oversized lines - Improve chunking logic with segment handling - Clean up method signatures and parameter ordering * fix(settings): make select inputs full width in CodeIndexSettings * refactor: increase max list file limit * feat(ui): improve codebase search result display formatting * test: add tests for cache and config managers * test: create unit tests for parser and scanner * feat(parser): improve segment hash uniqueness - Added startCharIndex to segment hash calculation in _chunkTextByLines - Track character position when splitting oversized lines - Ensures unique identification of segments from same line * feat(file-watcher): add error logging and optional ignoreController injection * fix: allow getting the state if the service is disabled * fix: set the embedding models when cline provider is initialized * feat: use zod to validate form * feat(file-watcher): enhance file watcher for batched deletions and improved vector store interactions Improve file watcher to handle file deletions in batches and optimize vector store operations. * feat(CodeIndexSettings): move OpenAI key input to a conditional rendering block * feat(CodeIndexSettings): update button visibility based on indexing status * feat(file-watcher): refactor vscode mock and enhance file watcher tests * fix(CodeIndexManager): do not await startIndexing on configuration changes * feat(types): add codeIndexOpenAiKey and codeIndexQdrantApiKey to ProviderSettings and IpcMessage * feat(FileWatcher): enhance file processing with batch operations and new status handling * fix(webviewMessageHandler): handle errors during CodeIndexManager initialization * refactor(CodeIndexManager): streamline service creation by consolidating into a single method * feat(CodeIndex): implement minimum search score configuration and update search methods * refactor(CodeIndexSettings): replace ApiConfiguration with ProviderSettings and update related methods * refactor: move contants to centralized file * refactor(constants): rename CODEBASE_INDEX_SEARCH_MIN_SCORE to SEARCH_MIN_SCORE * feat(QdrantVectorStore): enhance search functionality with new query structure and indexing * feat(FileWatcher): implement batch processing and retry logic for upserting points * fix(CodeIndexSettings): rename setProviderSettingsField to setApiConfigurationField and move model label * fix(ChatRow): remove limit from search query messages * refactor(CodebaseSearchResult): remove unused props from component * feat: implement batch processing for file events in FileWatcher - Introduced a new mechanism to accumulate file events (create, change, delete) and process them in batches. - Added debounce functionality to optimize processing frequency. - Emitted events for batch processing start, progress updates, and completion with detailed summaries. - Refactored existing processing logic to handle batch deletions and upserts efficiently. - Enhanced error handling and logging for better traceability during batch operations. * feat(CodeIndex): implement batch processing and update progress reporting * fix: define a default url for qdrant * feat(CodeIndexManager): add initialization check and update startIndexing logic * feat(CodeIndexSettings): validate Qdrant URL and update settings commitment logic * feat: refactor progress calculation and update progress bar rendering * refactor: remove webview provider and related methods * fix: simplify indexing status update by directly using update values * feat: integrate .gitignore support into file processing and scanning logic * fix: update clearCacheFile method to write an empty object instead of deleting the cache file * Revert this * Run prettier * fix: add new dependencies for qdrant client and directory scanner * feat: add codebase search functionality to localization files * feat: add localization strings for codebase indexing settings * feat: integrate CodeIndexSettings into ExperimentalSettings and update settings localization * refactor: remove console logs from various components for cleaner output * feat: enhance capabilities section and codebase search tool description * feat: add code indexing localization for multiple languages * fix: correct indentation for CodeIndexSettings component in ExperimentalSettings * refactor: update unit tests to properly test current functionality * feat: add mock implementation for p-limit and update Jest config * feat: track file creation, change, and deletion events in accumulatedEvents * refactor: simplify file watcher tests by removing waitForFileProcessingToFinish and using direct event accumulation * refactor: mock ContextProxy's getValue method to return current config name in ClineProvider tests * refactor: mock missing properties required by codebase indexing manager --------- Co-authored-by: cte <cestreich@gmail.com>	2025-05-23 19:07:19 -06:00