* feat: Add safeWriteJson utility for atomic file operations
Implements a robust JSON file writing utility that:
- Prevents concurrent writes to the same file using in-memory locks
- Ensures atomic operations with temporary file and backup strategies
- Handles error cases with proper rollback mechanisms
- Cleans up temporary files even when operations fail
- Provides comprehensive test coverage for success and failure scenarios
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* fix: use safeWriteJson for all JSON file writes
This change refactors all direct JSON file writes to use the safeWriteJson
utility, which implements atomic file writes to prevent data corruption
during write operations.
- Modified safeWriteJson to accept optional replacer and space arguments
- Updated tests to verify correct behavior with the new implementation
Fixes: #722
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* feat: Implement inter-process file locking for safeWriteJson
Replaces the previous in-memory lock in `safeWriteJson` with
`proper-lockfile` to provide robust, cross-process advisory file
locking. This enhances safety when multiple processes might attempt
concurrent writes to the same JSON file.
Key changes:
- Added `proper-lockfile` and `@types/proper-lockfile` dependencies.
- `safeWriteJson` now uses `proper-lockfile.lock()` with configured
retries, staleness checks (31s), and lock update intervals (10s).
- An `onCompromised` handler is included to manage scenarios where
the lock state is unexpectedly altered.
- Logging and comments within `safeWriteJson` have been refined for
clarity, ensuring error logs include backtraces.
- The test suite `safeWriteJson.test.ts` has been significantly
updated to:
- Use real timers (`jest.useRealTimers()`).
- Employ a more comprehensive mock for `fs/promises`.
- Correctly manage file pre-existence for various scenarios.
- Simulate lock contention by mocking `proper-lockfile.lock()`
using `jest.doMock` and a dynamic require for the SUT.
- Verify lock release by checking for the absence of the `.lock`
file.
All tests are passing with these changes.
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* feat: implement streaming JSON write in safeWriteJson
Refactor safeWriteJson to use stream-json for memory-efficient JSON serialization:
- Replace in-memory string creation with streaming pipeline
- Add Disassembler and Stringer from stream-json library
- Extract streaming logic to a dedicated helper function
- Add proper-lockfile and stream-json dependencies
This implementation reduces memory usage when writing large JSON objects.
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* fix: improve safeWriteJson locking mechanism
- Use file path itself for locking instead of separate lock file
- Improve error handling and clarity of code
- Enhance cleanup of temporary files
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* test: fix safeWriteJson test failures
- Ensure test file exists before locking
- Add proper mocking for fs.createWriteStream
- Fix test assertions to match expected behavior
- Improve test comments to follow project guidelines
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* test: update tests to work with safeWriteJson
Updated tests to work with safeWriteJson instead of direct fs.writeFile calls:
- Updated importExport.test.ts to expect safeWriteJson calls instead of fs.writeFile
- Fixed McpHub.test.ts by properly mocking fs/promises module:
- Moved jest.mock() to the top of the file before any imports
- Added mock implementations for all fs functions used by safeWriteJson
- Updated the test setup to work with the mocked fs module
All tests now pass successfully.
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* refactor: replace JSON.stringify with safeWriteJson for file operations
Replace all non-test instances of JSON.stringify used for writing to JSON files with safeWriteJson to ensure safer file operations with proper locking, error handling, and atomic writes.
- Updated src/services/mcp/McpHub.ts
- Updated src/services/code-index/cache-manager.ts
- Updated src/api/providers/fetchers/modelEndpointCache.ts
- Updated src/api/providers/fetchers/modelCache.ts
- Updated tests to match the new implementation
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
* docs: add rules for using safeWriteJson
Add concise rules for using safeWriteJson instead of JSON.stringify with file operations to ensure atomic writes and prevent data corruption.
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
---------
Signed-off-by: Eric Wheeler <roo-code@z.ewheeler.org>
Co-authored-by: Eric Wheeler <roo-code@z.ewheeler.org>
Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com>
* fix(webview): resolve memory leak in ChatView by stabilizing callback props
- Stabilize handleSendMessage using clineAskRef to prevent frequent re-creation
- Stabilize toggleRowExpansion by extracting handleSetExpandedRow and managing dependencies
- Re-integrate scrolling logic into useEffect hook to avoid destabilizing callbacks
- Add everVisibleMessagesTsRef to reduce unnecessary ChatRow remounts by Virtuoso
- Update onToggleExpand signature to accept timestamp parameter for better stability
- Remove diagnostic console.log statements used for debugging callback changes
These changes address detached DOM elements memory leak caused by frequent
callback re-creation triggering unnecessary component re-renders and preventing
proper garbage collection of chat message DOM nodes.
* comment correct TTL
---------
Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com>
* feat: apply changes from local main
* fix: add missing types
* feat: deduplicate code blocks coming out of parser
* feat: implement a cache manager to improve cache handling
* refactor: move code index service initialization to extension and remove await from indexing process
* fix: return undefined instead of throwing if no workspace is detected
* feat: allow auto approve if it is active for read tools
* refactor: improve UI of the results and allow opening the ranges directly in the editor
* refactor: use dependency injection to improve performance
* feat: implement result filtering by directory path
* refactor: centralize path normalization logic
* refactor: remove unnecessary barrel file
* refactor: prevent restarting the service if no settings change
* fix: the indexing process should never be awaited
* refactor: cleanup unused method
* refactor: remove batch limits for ollama
* refactor(parser): simplify method signatures and improve chunking logic
- Remove redundant min/max chars parameters
- Add better handling for oversized lines
- Improve chunking logic with segment handling
- Clean up method signatures and parameter ordering
* fix(settings): make select inputs full width in CodeIndexSettings
* refactor: increase max list file limit
* feat(ui): improve codebase search result display formatting
* test: add tests for cache and config managers
* test: create unit tests for parser and scanner
* feat(parser): improve segment hash uniqueness
- Added startCharIndex to segment hash calculation in _chunkTextByLines
- Track character position when splitting oversized lines
- Ensures unique identification of segments from same line
* feat(file-watcher): add error logging and optional ignoreController injection
* fix: allow getting the state if the service is disabled
* fix: set the embedding models when cline provider is initialized
* feat: use zod to validate form
* feat(file-watcher): enhance file watcher for batched deletions and improved vector store interactions
Improve file watcher to handle file deletions in batches and optimize vector store operations.
* feat(CodeIndexSettings): move OpenAI key input to a conditional rendering block
* feat(CodeIndexSettings): update button visibility based on indexing status
* feat(file-watcher): refactor vscode mock and enhance file watcher tests
* fix(CodeIndexManager): do not await startIndexing on configuration changes
* feat(types): add codeIndexOpenAiKey and codeIndexQdrantApiKey to ProviderSettings and IpcMessage
* feat(FileWatcher): enhance file processing with batch operations and new status handling
* fix(webviewMessageHandler): handle errors during CodeIndexManager initialization
* refactor(CodeIndexManager): streamline service creation by consolidating into a single method
* feat(CodeIndex): implement minimum search score configuration and update search methods
* refactor(CodeIndexSettings): replace ApiConfiguration with ProviderSettings and update related methods
* refactor: move contants to centralized file
* refactor(constants): rename CODEBASE_INDEX_SEARCH_MIN_SCORE to SEARCH_MIN_SCORE
* feat(QdrantVectorStore): enhance search functionality with new query structure and indexing
* feat(FileWatcher): implement batch processing and retry logic for upserting points
* fix(CodeIndexSettings): rename setProviderSettingsField to setApiConfigurationField and move model label
* fix(ChatRow): remove limit from search query messages
* refactor(CodebaseSearchResult): remove unused props from component
* feat: implement batch processing for file events in FileWatcher
- Introduced a new mechanism to accumulate file events (create, change, delete) and process them in batches.
- Added debounce functionality to optimize processing frequency.
- Emitted events for batch processing start, progress updates, and completion with detailed summaries.
- Refactored existing processing logic to handle batch deletions and upserts efficiently.
- Enhanced error handling and logging for better traceability during batch operations.
* feat(CodeIndex): implement batch processing and update progress reporting
* fix: define a default url for qdrant
* feat(CodeIndexManager): add initialization check and update startIndexing logic
* feat(CodeIndexSettings): validate Qdrant URL and update settings commitment logic
* feat: refactor progress calculation and update progress bar rendering
* refactor: remove webview provider and related methods
* fix: simplify indexing status update by directly using update values
* feat: integrate .gitignore support into file processing and scanning logic
* fix: update clearCacheFile method to write an empty object instead of deleting the cache file
* Revert this
* Run prettier
* fix: add new dependencies for qdrant client and directory scanner
* feat: add codebase search functionality to localization files
* feat: add localization strings for codebase indexing settings
* feat: integrate CodeIndexSettings into ExperimentalSettings and update settings localization
* refactor: remove console logs from various components for cleaner output
* feat: enhance capabilities section and codebase search tool description
* feat: add code indexing localization for multiple languages
* fix: correct indentation for CodeIndexSettings component in ExperimentalSettings
* refactor: update unit tests to properly test current functionality
* feat: add mock implementation for p-limit and update Jest config
* feat: track file creation, change, and deletion events in accumulatedEvents
* refactor: simplify file watcher tests by removing waitForFileProcessingToFinish and using direct event accumulation
* refactor: mock ContextProxy's getValue method to return current config name in ClineProvider tests
* refactor: mock missing properties required by codebase indexing manager
---------
Co-authored-by: cte <cestreich@gmail.com>
