Commit graph

33 commits

Author SHA1 Message Date
cscvenkatmadurai
0892455db2
feat(bedrock): add Cohere Embed v4 model and improve credential handling (Fixes #11823) (#11824)
feat(bedrock): add Cohere Embed v4 model and improve credential handling

- Add cohere.embed-v4:0 (1536-dim) to Bedrock embedding model profiles
- Add v4-specific request format (embedding_types: ["float"]) and response
  parsing (embeddings.float[0]) in BedrockEmbedder
- Replace fromEnv() with fromNodeProviderChain() for default credential
  chain when no AWS profile is specified, supporting SSO, IMDS, ECS, and
  other credential sources with built-in memoization
- Add unit tests for Cohere v4 request/response handling, credential
  provider selection, and v3 regression coverage

Fixes #11823
2026-03-05 18:45:17 -07:00
Hannes Rudolph
b2b77809ff
Reapply Batch 1: 22 clean non-AI-SDK cherry-picks (#11473)
* fix: add image content support to MCP tool responses (#10874)

Co-authored-by: Roo Code <roomote@roocode.com>

* fix: transform tool blocks to text before condensing (EXT-624) (#10975)

* refactor(read_file): Codex-inspired read_file refactor EXT-617 (#10981)

* feat: allow import settings in initial welcome screen (#10994)

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>

* fix(code-index): remove deprecated text-embedding-004 and migrate to gemini-embedding-001 (#11038)

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Hannes Rudolph <hrudolph@gmail.com>

* chore: treat extension .env as optional (#11116)

* fix: sanitize tool_use_id in tool_result blocks to match API history (#11131)

Tool IDs from providers like Gemini/OpenRouter contain special characters
(e.g., 'functions.read_file:0') that are sanitized when saving tool_use
blocks to API history. However, tool_result blocks were using the original
unsanitized IDs, causing ToolResultIdMismatchError.

This fix ensures tool_result blocks use sanitizeToolUseId() to match the
sanitized tool_use IDs in conversation history.

Fixes EXT-711

* fix: queue messages during command execution instead of losing them (#11140)

* IPC fixes for task cancellation and queued messages (#11162)

* feat: add support for AGENTS.local.md personal override files (#11183)

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>

* fix(cli): resolve race condition causing provider switch during mode changes (#11205)

When using slash commands with `mode:` frontmatter (e.g., `/cli-release`
with `mode: code`), the CLI would fail with "Could not resolve
authentication method" from the Anthropic SDK, even when using a
non-Anthropic provider like `--provider roo`.

Root cause: In `markWebviewReady()`, the `webviewDidLaunch` message was
sent before `updateSettings`, creating a race condition. The
`webviewDidLaunch` handler's "first-time init" sync would read
`getState()` before CLI-provided settings were applied to the context
proxy. Since `getState()` defaults `apiProvider` to "anthropic" when
unset, this default was saved to the provider profile. When a slash
command triggered `handleModeSwitch()`, it found this corrupted profile
with `apiProvider: "anthropic"` (but no API key) and activated it,
overwriting the CLI's working roo provider configuration.

Fix:
1. Reorder `markWebviewReady()` to send `updateSettings` before
   `webviewDidLaunch`, ensuring the context proxy has CLI-provided
   values when the initialization handler runs.
2. Guard the first-time init sync with `checkExistKey(apiConfiguration)`
   to prevent saving a profile with only the default "anthropic"
   fallback and no actual API keys configured.

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* chore: remove dead toolFormat code from getEnvironmentDetails (#11207)

Remove the toolFormat constant and <tool_format> line from environment
details output. Native tool calling is now the only supported protocol,
making this code unnecessary.

Fixes #11206

Co-authored-by: Roo Code <roomote@roocode.com>

* feat: extract translation and merge resolver modes into reusable skills (#11215)

* feat: extract translation and merge resolver modes into reusable skills

- Add roo-translation skill with comprehensive i18n guidelines
- Add roo-conflict-resolution skill for intelligent merge conflict resolution
- Add /roo-translate slash command as shortcut for translation skill
- Add /roo-resolve-conflicts slash command as shortcut for conflict resolution skill

The existing translate and merge-resolver modes are preserved. These new skills
and commands provide reusable access to the same functionality.

Closes CLO-722

* feat: add guidances directory with translator guidance file

- Add .roo/guidances/roo-translator.md for brand voice, tone, and word choice guidance
- Update roo-translation skill to reference the guidance file

The guidance file serves as a placeholder for translation style guidelines
that will be interpolated at runtime.

* fix: rename guidances directory to guidance (singular)

* fix: remove language-specific section from translator guidance

The guidance file should focus on brand voice, tone, and word choice only.

* fix: remove language-specific guidelines section from skill file

* Update .roo/skills/roo-translation/SKILL.md

Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>

---------

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Bruno Bergher <bruno@roocode.com>
Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>

* feat: add Claude Opus 4.6 support across all providers (#11224)

* feat: add Claude Opus 4.6 support across all providers

Add Claude Opus 4.6 (claude-opus-4-6) model definitions and 1M context
support across Anthropic, Bedrock, Vertex AI, OpenRouter, and Vercel AI
Gateway providers.

- Anthropic: 128K max output, /5 pricing, 1M context tiers
- Bedrock: anthropic.claude-opus-4-6-v1:0 with 1M context + global inference
- Vertex: claude-opus-4-6 with 1M context tiers
- OpenRouter: prompt caching + reasoning budget sets
- Vercel AI Gateway: Opus 4.5 and 4.6 added to capability sets
- UI: 1M context checkbox for Opus 4.6 on all providers
- i18n: Updated 1M context descriptions across 18 locales

Also adds Opus 4.5 to Vercel AI Gateway (previously missing) and
OpenRouter maxTokens overrides for Opus 4.5/4.6.

Closes #11223

* fix: apply tier pricing when 1M context is enabled on Bedrock

When awsBedrock1MContext is enabled for tiered models like Opus 4.6,
also apply the 1M tier pricing (inputPrice, outputPrice, cache prices)
instead of only updating contextWindow. This ensures cost calculations
and UI display use the correct >200K rates.

* feat: add gpt-5.3-codex model to OpenAI Codex provider (#11225)

feat: add gpt-5.3-codex model and make it default for OpenAI Codex provider

Co-authored-by: Roo Code <roomote@roocode.com>

* fix: prevent parent task state loss during orchestrator delegation (#11281)

* fix: make removeClineFromStack() delegation-aware to prevent orphaned parent tasks (#11302)

* fix: make removeClineFromStack() delegation-aware to prevent orphaned parent tasks

When a delegated child task is removed via removeClineFromStack() (e.g., Clear
Task, navigate to history, start new task), the parent task was left orphaned
in "delegated" status with a stale awaitingChildId. This made the parent
unresumable without manual history repair.

This fix captures parentTaskId and childTaskId before abort/dispose, then
repairs the parent metadata (status -> active, clear awaitingChildId) when
the popped task is a delegated child and awaitingChildId matches.

Parent lookup + updateTaskHistory are wrapped in try/catch so failures are
non-fatal (logged but do not block the pop).

Closes #11301

* fix: add skipDelegationRepair opt-out to removeClineFromStack() for nested delegation

---------

Co-authored-by: Roo Code <roomote@roocode.com>

* fix(reliability): prevent webview postMessage crashes and make dispose idempotent (#11313)

* fix(reliability): prevent webview postMessage crashes and make dispose idempotent

Closes: #11311

1. postMessageToWebview() now catches rejections from
   webview.postMessage() so that messages sent after the webview is
   disposed do not surface as unhandled promise rejections.

2. dispose() is guarded by a _disposed flag so that repeated calls
   (e.g. during rapid extension deactivation) are no-ops.

3. CloudService mock in ClineProvider.spec.ts updated to include
   off() — a pre-existing gap exposed by the new dispose test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add early _disposed check in postMessageToWebview

Skip the postMessage call entirely when the provider is already disposed,
avoiding unnecessary try/catch execution. Added test coverage for this path.

* chore: trigger CI

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>

* fix: resolve race condition in new_task delegation that loses parent task history (#11331)

* fix: resolve race condition in new_task delegation that loses parent task history

When delegateParentAndOpenChild creates a child task via createTask(), the
Task constructor fires startTask() as a fire-and-forget async call. The child
immediately begins its task loop and eventually calls saveClineMessages() →
updateTaskHistory(), which reads globalState, modifies it, and writes back.

Meanwhile, delegateParentAndOpenChild persists the parent's delegation
metadata (status: 'delegated', delegatedToId, awaitingChildId, childIds) via
a separate updateTaskHistory() call AFTER createTask() returns.

These two concurrent read-modify-write operations on globalState race: the
last writer wins, overwriting the other's changes. When the child's write
lands last, the parent's delegation fields are lost, making the parent task
unresumable when the child finishes.

Fix: create the child task with startTask: false, persist the parent's
delegation metadata first, then manually call child.start(). This ensures
the parent metadata is safely in globalState before the child begins writing.

* docs: clarify Task.start() only handles new tasks, not history resume

* fix: serialize taskHistory writes and fix delegation status overwrite race (#11335)

Add a promise-chain mutex (withTaskHistoryLock) to serialize all
read-modify-write operations on taskHistory, preventing concurrent
interleaving from silently dropping entries.

Reorder reopenParentFromDelegation to close the child instance
before marking it completed, so the abort path's stale 'active'
status write no longer overwrites the 'completed' state.

Covered by new tests: RPD-04/05/06, UTH-02/04, and a full mutex
concurrency suite.

* Fix task resumption in the API module (#11369)

* chore: clean up repo-facing mode rules (#11410)

* fix: add maxReadFileLine to ExtensionState type for webview compatibility

---------

Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
Co-authored-by: Chris Estreich <cestreich@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Bruno Bergher <bruno@roocode.com>
Co-authored-by: 0xMink <dennis@dennismink.com>
Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>
2026-02-14 13:47:12 -07:00
Daniel
6cfa82f571
Revert to pre-AI-SDK state (January 29, 2026) (#11462)
Revert to pre-AI-SDK state (commit 67e568f6b)

This commit reverts the codebase to the state before AI SDK migration work began.

Target commit: 67e568f6b - refactor: replace fetch_instructions with skill tool and built-in skills (#10913)
Date: January 29, 2026

This removes approximately 152 commits of AI SDK migration work.
A follow-up PR will add back bug fixes and features that are unrelated to AI SDK.

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 16:45:18 -05:00
roomote[bot]
1e790b0d39
fix(code-index): remove deprecated text-embedding-004 and migrate to gemini-embedding-001 (#11038)
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Hannes Rudolph <hrudolph@gmail.com>
2026-02-02 22:30:07 -05:00
roomote[bot]
f899de1f53
fix: add userAgentAppId to Bedrock embedder for code indexing (#10166)
Adds userAgentAppId configuration to the BedrockRuntimeClient in the
code indexing embedder, matching the implementation pattern already
used in the main Bedrock API provider.

This enables proper user agent identification in CloudTrail AWS requests
when using Bedrock for code indexing embeddings.

Fixes #10165

Co-authored-by: Roo Code <roomote@roocode.com>
2025-12-17 20:20:36 -05:00
SannidhyaSah
873a763ea7
feat: Add provider routing selection for OpenRouter embeddings (#9144) (#9693)
Co-authored-by: Sannidhya <sann@Sannidhyas-MacBook-Pro.local>
2025-12-02 22:59:55 -05:00
George Goranov
56c630ca92
Feature/bedrock embeddings support (#9475)
* feat: add AWS Bedrock support for codebase indexing

- Add bedrock as a new EmbedderProvider type
- Add AWS Bedrock embedding model profiles (titan-embed-text models)
- Create BedrockEmbedder class with support for Titan and Cohere models
- Add Bedrock configuration support to config manager and interfaces
- Update service factory to create BedrockEmbedder instances
- Add comprehensive tests for BedrockEmbedder
- Add localization strings for Bedrock support

Closes #8658

* fix: add missing bedrockOptions to loadConfiguration return type

* Fix various issues that the original PR missed.

* Remove debug logs

* Rename AWS Bedrock -> Amazon Bedrock

* Remove some 'as any's

* Revert README changes

* Add translations

* More translations

* Remove leftover code from a debugging session.

* fix: add bedrock to codebaseIndexModelsSchema and update brace-expansion override

- Add bedrock provider to codebaseIndexModelsSchema type definition to fix empty model dropdown in UI
- Update pnpm override for brace-expansion from '>=2.0.2' to '^2.0.2' to resolve ESM/CommonJS compatibility issues

* Improvements to AWS Bedrock embeddings support

- Enhanced bedrock.ts embedder implementation
- Added comprehensive test coverage in bedrock.spec.ts
- Updated config-manager.ts for better Bedrock configuration handling
- Improved service-factory.ts integration
- Updated embeddingModels.ts with Bedrock models
- Enhanced CodeIndexPopover.tsx UI for Bedrock options
- Added auto-populate test for CodeIndexPopover
- Updated pnpm-lock.yaml dependencies

* Restore openrouter config

* Remove debug log

* Fix config-manager.spec.ts unit test.

* Add translations for "optional"

* Revert unnecessary change related to open ia embedder

---------

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
Co-authored-by: Smartsheet-JB-Brown <jb.brown@smartsheet.com>
2025-11-26 08:58:58 -05:00
Matt Rubens
3c989d3591
Revert "Add support for Roo Code Cloud as an embeddings provider" (#9602) 2025-11-26 00:08:39 -05:00
Matt Rubens
fb4f23537a
Add support for Roo Code Cloud as an embeddings provider (#9543) 2025-11-24 17:21:30 -05:00
roomote[bot]
413ab1a95c
feat: add Qwen3 embedding models (0.6B and 4B) to OpenRouter support (#9060)
Co-authored-by: Roo Code <roomote@roocode.com>
2025-11-05 15:35:25 -05:00
David Markey
34f45f1b28
feat: add OpenRouter embedding provider support (#8973)
* feat: add OpenRouter embedding provider support

Implement comprehensive OpenRouter embedding provider support for codebase indexing with the following features:

- New OpenRouterEmbedder class with full API compatibility
- Support for OpenRouter's OpenAI-compatible embedding endpoint
- Rate limiting and retry logic with exponential backoff
- Base64 embedding handling to bypass OpenAI package limitations
- Global rate limit state management across embedder instances
- Configuration updates for API key storage and provider selection
- UI integration for OpenRouter provider settings
- Comprehensive test suite with mocking
- Model dimension support for OpenRouter's embedding models

This adds OpenRouter as the 7th supported embedding provider alongside OpenAI, Ollama, OpenAI-compatible, Gemini, Mistral, and Vercel AI Gateway.

* Add translation key

* Fix mutex double release bug

* Add translations

* Add more translations

* Fix failing tests

* code-index(openrouter): fix HTTP-Referer header to RooCodeInc/Roo-Code; i18n: add and wire OpenRouter Code Index strings; test: assert default headers in embedder

---------

Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>
2025-11-02 19:12:28 -05:00
Daniel
759454b455
fix: handle ByteString conversion errors in OpenAI embedders (#8008) 2025-09-15 22:56:20 -04:00
Chris Estreich
cd9e92fa9b
Move @roo-code/cloud to the Roo-Code repo (#7503) 2025-08-28 11:18:45 -07:00
Matt Rubens
1ad2129964
Add support for Vercel embeddings (#7445)
Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>
2025-08-27 13:30:19 -04:00
Adam Brand
7a07088802
Add pattern to support Databricks /invocations endpoints (#6317)
For using other models in Azure (e.g., Claude); you have to use Databricks, and the other patterns didn't match that.
2025-07-30 18:31:26 -04:00
Nahim EL ATMANI
b1406342a8
Fix Ollama API URL normalization by removing trailing slashes (#6079)
Co-authored-by: Nahim El Atmani <2959826+Naam@users.noreply.github.com>
2025-07-25 00:41:35 -04:00
SannidhyaSah
7ddc4e64f3
feat: Add Mistral embedding provider (#5932) (#5946)
* feat: add Mistral embedding provider with OpenAI Compatible Wrapper

- Implement MistralEmbedder class using OpenAI-compatible API
- Add comprehensive unit tests with 100% coverage
- Update type definitions for Mistral provider support
- Integrate Mistral option in UI components and configuration
- Add internationalization support for Mistral provider
- Fix API key storage and retrieval for embedding providers
- Update service factory to support Mistral embeddings
- Add proper error handling and validation

This implementation allows users to use Mistral's embedding models
through the existing OpenAI-compatible wrapper approach, providing
a seamless integration experience.

* feat: add Mistral embedding provider support

- Implement MistralEmbedder class with API integration
- Add Mistral models to embedding model configurations
- Update UI to include Mistral provider option
- Add comprehensive unit tests for Mistral embedder
- Update type definitions and interfaces
- Add internationalization support for Mistral provider

* fix: add missing translations for Mistral embedding provider

* fix: address PR review feedback - improve translations and add clarifying comment
2025-07-19 13:24:24 -04:00
Daniel
020c2337dd
feat: add global rate limiting for OpenAI-compatible embeddings (#5854)
* feat: add global rate limiting for OpenAI-compatible embeddings

- Implement shared rate limit state across all embedder instances
- Add exponential backoff (5s base, up to 5 minutes max)
- Track consecutive rate limit errors with auto-reset after 60s
- Add thread-safe mutex for concurrent access
- Remove verbose logging to prevent log flooding during retries
- Add comprehensive test coverage for rate limiting behavior

This prevents multiple parallel batches from overwhelming APIs with
restrictive rate limits by coordinating delays globally.

* fix: restore original logging while keeping rate limit logging silent

- Restored all original console.warn and console.error calls that were inadvertently removed
- Only the rate limit retry warning remains silent (no console output)
- Fixed test expectation to use correct error variable
- All existing logging behavior is preserved
2025-07-19 09:37:01 -04:00
Daniel
8c8888a977
feat: increase Ollama API timeout values and extract as constants (#5778)
- Increase embedding request timeout from 10s to 60s
- Increase validation request timeouts from 5s to 30s
- Extract timeout values as module-level constants for better maintainability
- OLLAMA_EMBEDDING_TIMEOUT_MS = 60000 (60 seconds)
- OLLAMA_VALIDATION_TIMEOUT_MS = 30000 (30 seconds)
2025-07-16 09:30:33 -04:00
Daniel
d7787a2de3
feat: add gemini-embedding-001 model to code-index service (#5698)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-14 14:10:40 -04:00
Daniel
32308d79c4
feat: add comprehensive error telemetry to code-index service (#5595) 2025-07-11 23:01:50 -04:00
Daniel
50598b22b5
fix(i18n): Correct translation fallback logic for embedding errors (#5574)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-10 15:47:47 -04:00
Murilo Pires
fa60a31578
Indexing field validation (#5483)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com>
2025-07-08 23:15:37 -04:00
Hannes Rudolph
d4abe73875
fix: add embedder validation to prevent misleading status indicators (#4398) (#5404)
* fix: add embedder validation to prevent misleading status indicators (#4398)

* fix: address PR feedback and fix critical issues

- Fixed settings-save flow to save before validation
- Fixed Error constructor usage in scanner.ts
- Fixed segment identification in file-watcher.ts
- Added missing translation keys for embedder validation errors

* fix: add missing Ollama translation keys

- Added missing ollama.title, description, and settings keys
- Fixed translation check failure in CI/CD pipeline
- Synchronized all 17 non-English locale files

* feat: add proactive embedder validation on provider switch

- Validate embedder connection when switching providers
- Prevent misleading 'Indexed' status when embedder is unavailable
- Show immediate error feedback for invalid configurations
- Add comprehensive test coverage for validation flow

This ensures users get immediate feedback when configuring embedders,
preventing confusion when providers like Ollama are not accessible.

* fix: improve error handling and validation in code indexing process

* refactor: extract common embedder validation and error handling logic

- Created shared/validation-helpers.ts with centralized error handling utilities
- Refactored OpenAI, OpenAI-Compatible, and Ollama embedders to use shared helpers
- Eliminated duplicate error handling code across embedders
- Improved maintainability and consistency of error handling
- Fixed test compatibility in manager.spec.ts
- All 2721 tests passing

* refactor: simplify validation helpers by removing unnecessary wrapper functions

- Removed getErrorMessageForConnectionError and inlined logic into handleValidationError
- Removed isRateLimitError, logRateLimitRetry, and logEmbeddingError wrapper functions
- Updated openai.ts and openai-compatible.ts to inline rate limit checking and logging
- Reduced code complexity while maintaining all functionality
- All 311 tests continue to pass

* fix: add missing invalidResponse i18n key and fix French translation

- Added missing 'invalidResponse' key to all locale files
- Fixed French translation: changed 'and accessible' to 'et accessible'
- Ensures proper error messages are displayed when embedder returns invalid responses

* fix: restore removed score settings in webviewMessageHandler

- Restored codebaseIndexSearchMaxResults and codebaseIndexSearchMinScore settings that were unintentionally removed
- Keep embedder validation related changes

* fix: revert unintended changes to file-watcher and scanner

- Reverted point ID generation back to using line numbers instead of segmentHash
- Restored { cause: deleteError } parameter in scanner error handling
- These changes were unrelated to the embedder validation feature

---------

Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
2025-07-07 18:46:35 -04:00
SannidhyaSah
37619d7850
fix: support full endpoint URLs in OpenAI Compatible provider (#5212) (#5214)
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-07-02 23:58:35 -04:00
SannidhyaSah
87aa688f33
feat: add Gemini embedding provider for codebase indexing (#5228)
Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
2025-07-02 23:51:42 -04:00
Hannes Rudolph
a348a2ae9a
feat: add user-configurable search score threshold slider for semantic search (#5027) (#5041)
Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
2025-07-01 14:41:23 -04:00
Hannes Rudolph
9b18b145b4
fix: improve error handling for codebase search embeddings (#4432)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
2025-06-18 11:19:02 -04:00
Chris Estreich
62c3914034
Farewell jest (#4607)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-16 21:39:45 -07:00
Dixie Flatline
c481827cbb
Manually specify openai-compat format and parse it (#4463)
* Manually specify openai-compat format and parse it

* fixup! Manually specify openai-compat format and parse it

* Expect base64 in embedding test arguments

* fixup! Manually specify openai-compat format and parse it

Remove debug logs

* fixup! Manually specify openai-compat format and parse it

Improve comment

* fixup! Manually specify openai-compat format and parse it

* Add tests to exercise base64 decode of embeddings

* Add tests to verify openai base64 and brokenness behavior

* feat: improve typing

* refactor: switch from jest to vitest for mocking in tests

---------

Co-authored-by: Dixie Flatline <dflatline>
Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
2025-06-12 13:12:48 -04:00
Chris Estreich
395f55b31f
Convert jest tests to vitest and disable default watch mode for vitest (#4568)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-11 14:48:01 -07:00
SannidhyaSah
a80795b78d
feat: Add OpenAI Compatible embedder for codebase indexing (#4066)
* feat: Add OpenAI Compatible embedder for codebase indexing

- Implement OpenAiCompatibleEmbedder with batching and retry logic
- Add configuration support for base URL and API key
- Update UI with provider selection and input fields
- Add comprehensive test coverage
- Support for all OpenAI-compatible endpoints (LiteLLM, LMStudio, Ollama, etc.)
- Add internationalization for 17 languages

* fix: Update CodeIndexSettings tests for OpenAI Compatible provider

- Fix field count expectations (4 fields including Qdrant)
- Use specific test IDs for button selection
- Fix input handling with clear() before type()
- Use toHaveBeenLastCalledWith for better assertions
- Fix status text matching with regex pattern

* fix: resolve UI test failures and ESLint errors

- Remove unused waitFor import to fix ESLint error
- Fix test expectations to match actual component behavior for input fields
- Simplify provider selection test by removing complex mock interactions
- All CodeIndexSettings tests now pass (20/20)

* feat: add custom model infrastructure for OpenAI-compatible embedder

- Add manual model ID and embedding dimension configuration
- Enable custom model input via text field in settings UI
- Add modelDimension parameter to OpenAiCompatibleEmbedder
- Update configuration management to persist dimension setting
- Prioritize manual dimension over hardcoded model profiles
- Add comprehensive test coverage for new functionality

This allows users to specify any custom embedding model and its
dimension for OpenAI-compatible providers, removing dependency
on hardcoded model profiles.

* Add missing translations for OpenAI-compatible model dimension settings in all locales

* refactor: remove unused modelDimension parameter from OpenAiCompatibleEmbedder

- Remove modelDimension property and constructor parameter from OpenAiCompatibleEmbedder class
- Update ServiceFactory to not pass dimension to embedder constructor
- Update tests to match new constructor signature
- The dimension is still used for QdrantVectorStore configuration

* chore: bot suggestion

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* chore: bot suggestion

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* refactor: rename OpenAiCompatibleEmbedder to OpenAICompatibleEmbedder for consistency

* feat: add model dimension validation for OpenAI-compatible settings

* refactor: improve default model ID retrieval logic for embedding providers

* feat: add default model ID retrieval for openai-compatible provider

* refactor: update default model ID retrieval to use shared utility function

* fix: Remove unnecessary type assertion in OpenAICompatibleEmbedder

* feat: add model dimension input for openai-compatible provider

---------

Co-authored-by: Daniel Riccio <ricciodaniel98@gmail.com>
Co-authored-by: Daniel <57051444+daniel-lxs@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-04 17:00:52 -04:00
Daniel
61e122dc34
Codebase indexing (Clean history) (#3137)
* feat: apply changes from local main

* fix: add missing types

* feat: deduplicate code blocks coming out of parser

* feat: implement a cache manager to improve cache handling

* refactor: move code index service initialization to extension and remove await from indexing process

* fix: return undefined instead of throwing if no workspace is detected

* feat: allow auto approve if it is active for read tools

* refactor: improve UI of the results and allow opening the ranges directly in the editor

* refactor: use dependency injection to improve performance

* feat: implement result filtering by directory path

* refactor: centralize path normalization logic

* refactor: remove unnecessary barrel file

* refactor: prevent restarting the service if no settings change

* fix: the indexing process should never be awaited

* refactor: cleanup unused method

* refactor: remove batch limits for ollama

* refactor(parser): simplify method signatures and improve chunking logic

- Remove redundant min/max chars parameters
- Add better handling for oversized lines
- Improve chunking logic with segment handling
- Clean up method signatures and parameter ordering

* fix(settings): make select inputs full width in CodeIndexSettings

* refactor: increase max list file limit

* feat(ui): improve codebase search result display formatting

* test: add tests for cache and config managers

* test: create unit tests for parser and scanner

* feat(parser): improve segment hash uniqueness

- Added startCharIndex to segment hash calculation in _chunkTextByLines
- Track character position when splitting oversized lines
- Ensures unique identification of segments from same line

* feat(file-watcher): add error logging and optional ignoreController injection

* fix: allow getting the state if the service is disabled

* fix: set the embedding models when cline provider is initialized

* feat: use zod to validate form

* feat(file-watcher): enhance file watcher for batched deletions and improved vector store interactions

Improve file watcher to handle file deletions in batches and optimize vector store operations.

* feat(CodeIndexSettings): move OpenAI key input to a conditional rendering block

* feat(CodeIndexSettings): update button visibility based on indexing status

* feat(file-watcher): refactor vscode mock and enhance file watcher tests

* fix(CodeIndexManager): do not await startIndexing on configuration changes

* feat(types): add codeIndexOpenAiKey and codeIndexQdrantApiKey to ProviderSettings and IpcMessage

* feat(FileWatcher): enhance file processing with batch operations and new status handling

* fix(webviewMessageHandler): handle errors during CodeIndexManager initialization

* refactor(CodeIndexManager): streamline service creation by consolidating into a single method

* feat(CodeIndex): implement minimum search score configuration and update search methods

* refactor(CodeIndexSettings): replace ApiConfiguration with ProviderSettings and update related methods

* refactor: move contants to centralized file

* refactor(constants): rename CODEBASE_INDEX_SEARCH_MIN_SCORE to SEARCH_MIN_SCORE

* feat(QdrantVectorStore): enhance search functionality with new query structure and indexing

* feat(FileWatcher): implement batch processing and retry logic for upserting points

* fix(CodeIndexSettings): rename setProviderSettingsField to setApiConfigurationField and move model label

* fix(ChatRow): remove limit from search query messages

* refactor(CodebaseSearchResult): remove unused props from component

* feat: implement batch processing for file events in FileWatcher

- Introduced a new mechanism to accumulate file events (create, change, delete) and process them in batches.
- Added debounce functionality to optimize processing frequency.
- Emitted events for batch processing start, progress updates, and completion with detailed summaries.
- Refactored existing processing logic to handle batch deletions and upserts efficiently.
- Enhanced error handling and logging for better traceability during batch operations.

* feat(CodeIndex): implement batch processing and update progress reporting

* fix: define a default url for qdrant

* feat(CodeIndexManager): add initialization check and update startIndexing logic

* feat(CodeIndexSettings): validate Qdrant URL and update settings commitment logic

* feat: refactor progress calculation and update progress bar rendering

* refactor: remove webview provider and related methods

* fix: simplify indexing status update by directly using update values

* feat: integrate .gitignore support into file processing and scanning logic

* fix: update clearCacheFile method to write an empty object instead of deleting the cache file

* Revert this

* Run prettier

* fix: add new dependencies for qdrant client and directory scanner

* feat: add codebase search functionality to localization files

* feat: add localization strings for codebase indexing settings

* feat: integrate CodeIndexSettings into ExperimentalSettings and update settings localization

* refactor: remove console logs from various components for cleaner output

* feat: enhance capabilities section and codebase search tool description

* feat: add code indexing localization for multiple languages

* fix: correct indentation for CodeIndexSettings component in ExperimentalSettings

* refactor: update unit tests to properly test current functionality

* feat: add mock implementation for p-limit and update Jest config

* feat: track file creation, change, and deletion events in accumulatedEvents

* refactor: simplify file watcher tests by removing waitForFileProcessingToFinish and using direct event accumulation

* refactor: mock ContextProxy's getValue method to return current config name in ClineProvider tests

* refactor: mock missing properties required by codebase indexing manager

---------

Co-authored-by: cte <cestreich@gmail.com>
2025-05-23 19:07:19 -06:00