diff --git a/docs/mistral-devstral2-performance.md b/docs/mistral-devstral2-performance.md new file mode 100644 index 0000000000..60acf480d0 --- /dev/null +++ b/docs/mistral-devstral2-performance.md @@ -0,0 +1,144 @@ +# Mistral Devstral 2 Performance and Thinking Tokens + +## Overview + +This document explains how Roo Code handles Mistral Devstral 2 models, including thinking token support and performance considerations. + +## Thinking Token Support + +### What are Thinking Tokens? + +Mistral Devstral 2 models support a "thinking" mode where the model's reasoning process is streamed separately from the final response. This allows users to see the model's thought process in real-time, similar to how the Mistral Vibe CLI displays "thinking" indicators. + +### Implementation Status + +**✅ Thinking tokens are fully supported** in Roo Code's Mistral handler. + +The implementation is located in [`src/api/providers/mistral.ts`](../src/api/providers/mistral.ts): + +```typescript +// Lines 122-129 +if (chunk.type === "thinking" && chunk.thinking) { + // Handle thinking content as reasoning chunks + for (const thinkingPart of chunk.thinking) { + if (thinkingPart.type === "text" && thinkingPart.text) { + yield { type: "reasoning", text: thinkingPart.text } + } + } +} +``` + +### How It Works + +1. **Streaming**: When Devstral 2 generates a response, it streams two types of content: + + - `"thinking"` chunks: The model's reasoning process + - `"text"` chunks: The final response + +2. **Display**: Thinking chunks are yielded as `{ type: "reasoning", text: ... }` which the UI displays separately from the final response, allowing users to see the model's thought process. + +3. **SDK Support**: The Mistral SDK v1.9.18+ includes `ThinkChunk` support for handling thinking tokens. + +## Performance Considerations + +### Current Performance + +The user reported that "performance is suboptimal" when using Devstral 2 models. Here are the key factors: + +1. **Thinking Tokens Add Latency**: When the model generates thinking tokens, it takes additional time before the final response begins. This is expected behavior and shows the model is working through the problem. + +2. **Streaming is Enabled**: The handler properly streams both thinking and text tokens, so users see progress in real-time rather than waiting for the complete response. + +3. **Temperature Setting**: PR #9957 set the default temperature to 0.2 based on Mistral's recommendations for optimal performance. + +### Prompt Caching + +**Current Status**: Mistral models currently have `supportsPromptCache: false` in the model definitions. + +**Investigation Needed**: + +- The Mistral API documentation does not clearly indicate whether prompt caching is supported for Devstral 2 models +- Unlike Anthropic or OpenAI, Mistral's API documentation doesn't provide explicit caching endpoints or parameters +- Further investigation with Mistral's API team would be needed to determine if caching is available + +### Recommendations for Users + +1. **Expect Thinking Time**: When using Devstral 2, the "thinking" phase is normal and indicates the model is reasoning through the problem. This is a feature, not a bug. + +2. **Temperature**: The default temperature of 0.2 is optimized for code generation tasks. Users can adjust this in settings if needed. + +3. **Model Selection**: + + - Use `devstral-latest` or `devstral-2512` for complex reasoning tasks where thinking tokens are valuable + - Use `devstral-small-latest` or `labs-devstral-small-2512` for faster responses on simpler tasks + +4. **Monitor the UI**: The thinking tokens should be visible in the UI, showing the model's reasoning process. If they're not appearing, this may indicate a UI rendering issue rather than an API issue. + +## Technical Details + +### Mistral SDK Version + +Roo Code uses `@mistralai/mistralai` version `^1.9.18`, which includes support for thinking chunks. + +### Content Chunk Types + +The Mistral API returns content in different formats: + +```typescript +type ContentChunkWithThinking = { + type: string // "thinking" or "text" + text?: string // For text chunks + thinking?: Array<{ + // For thinking chunks + type: string + text?: string + }> +} +``` + +### Streaming Flow + +1. API request is made with streaming enabled +2. Server streams back chunks as they're generated +3. Thinking chunks are processed and yielded as `reasoning` type +4. Text chunks are processed and yielded as `text` type +5. UI displays both types appropriately + +## Future Improvements + +### Potential Optimizations + +1. **Prompt Caching**: If Mistral adds prompt caching support, we can: + + - Cache system prompts across requests + - Cache conversation history + - Reduce latency for follow-up requests + +2. **Batch Processing**: For multiple requests, investigate if Mistral supports batch APIs + +3. **Connection Pooling**: Ensure HTTP connections are properly pooled and reused + +### Monitoring + +To help users understand performance: + +1. **Token Metrics**: Display thinking token count vs. response token count +2. **Timing Metrics**: Show time spent in thinking phase vs. response phase +3. **Progress Indicators**: Enhance UI to better show when model is thinking + +## References + +- [Mistral Devstral 2 Documentation](https://docs.mistral.ai/models/devstral-2-25-12) +- [Mistral Vibe CLI](https://github.com/mistralai/mistral-vibe) +- [Mistral SDK](https://github.com/mistralai/client-ts) +- [Issue #9951](https://github.com/RooCodeInc/Roo-Code/issues/9951) +- [PR #9957](https://github.com/RooCodeInc/Roo-Code/pull/9957) + +## Questions for Mistral Team + +To further optimize performance, we need clarification from Mistral on: + +1. Does the Mistral API support prompt caching for Devstral 2 models? +2. Are there any API parameters to control thinking token generation? +3. What are the recommended best practices for minimizing latency? +4. Are there any batch or concurrent request optimizations available? diff --git a/src/api/providers/mistral.ts b/src/api/providers/mistral.ts index 96d2c33255..a146f7684f 100644 --- a/src/api/providers/mistral.ts +++ b/src/api/providers/mistral.ts @@ -12,8 +12,18 @@ import { ApiStream } from "../transform/stream" import { BaseProvider } from "./base-provider" import type { SingleCompletionHandler, ApiHandlerCreateMessageMetadata } from "../index" -// Type helper to handle thinking chunks from Mistral API -// The SDK includes ThinkChunk but TypeScript has trouble with the discriminated union +/** + * Type helper to handle thinking chunks from Mistral API. + * + * Mistral Devstral 2 models support "thinking" mode where the model's reasoning process + * is streamed separately from the final response. This allows users to see the model's + * thought process in real-time. + * + * The SDK includes ThinkChunk but TypeScript has trouble with the discriminated union, + * so we define our own type here. + * + * @see https://docs.mistral.ai/models/devstral-2-25-12 + */ type ContentChunkWithThinking = { type: string text?: string @@ -106,8 +116,23 @@ export class MistralHandler extends BaseProvider implements SingleCompletionHand // Handle string content as text yield { type: "text", text: delta.content } } else if (Array.isArray(delta.content)) { - // Handle array of content chunks - // The SDK v1.9.18 supports ThinkChunk with type "thinking" + /** + * Handle array of content chunks from Mistral API. + * + * Mistral Devstral 2 models support streaming "thinking" tokens that show + * the model's reasoning process. These are streamed as separate chunks with + * type "thinking" and are displayed to users as reasoning steps. + * + * The SDK v1.9.18+ supports ThinkChunk with type "thinking". + * + * Content chunk types: + * - "thinking": Model's reasoning process (yielded as "reasoning" type) + * - "text": Final response text (yielded as "text" type) + * + * This implementation ensures thinking tokens are properly streamed and + * displayed in the UI, addressing performance concerns about showing + * model progress during generation. + */ for (const chunk of delta.content as ContentChunkWithThinking[]) { if (chunk.type === "thinking" && chunk.thinking) { // Handle thinking content as reasoning chunks