diff --git a/docs/my-website/docs/proxy/model_compare_ui.md b/docs/my-website/docs/proxy/model_compare_ui.md new file mode 100644 index 00000000000..a3fb236393f --- /dev/null +++ b/docs/my-website/docs/proxy/model_compare_ui.md @@ -0,0 +1,193 @@ +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Model Compare Playground UI + +Compare multiple LLM models side-by-side in an interactive playground interface. Evaluate model responses, performance metrics, and costs to make informed decisions about which models work best for your use case. + +This feature is **available in v1.80.0-stable and above**. + +## Overview + +The Model Compare Playground UI enables side-by-side comparison of up to 3 different LLM models simultaneously. Configure models, parameters, and test prompts to evaluate and compare model responses with detailed metrics including latency, token usage, and cost. + + + +## Getting Started + +### Accessing the Model Compare UI + +#### 1. Navigate to the Playground + +Go to the Playground page in the Admin UI (`PROXY_BASE_URL/ui/?login=success&page=llm-playground`) + + + +#### 2. Switch to Compare Tab + +Click on the **Compare** tab in the Playground interface. + +## Configuration + +### Setting Up Models + +#### 1. Select Models to Compare + +You can compare up to 3 models simultaneously. For each comparison panel: + +- Click on the model dropdown to see available models +- Select a model from your configured endpoints +- Models are loaded from your LiteLLM proxy configuration + + + +#### 2. Configure Model Parameters + +Each model panel supports individual parameter configuration: + +**Basic Parameters:** + +- **Temperature**: Controls randomness (0.0 to 2.0) +- **Max Tokens**: Maximum tokens in the response + +**Advanced Parameters:** + +- Enable "Use Advanced Params" to configure additional model-specific parameters +- Supports all parameters available for the selected model/provider + + + +#### 3. Apply Parameters Across Models + +Use the "Sync Settings Across Models" toggle to synchronize parameters (tags, guardrails, temperature, max tokens, etc.) across all comparison panels for consistent testing. + + + +### Guardrails + +Configure and test guardrails directly in the playground: + +1. Click on the guardrails selector in a model panel +2. Select one or more guardrails from your configured list +3. Test how different models respond to guardrail filtering +4. Compare guardrail behavior across models + + + +### Tags + +Apply tags to organize and filter your comparisons: + +1. Select tags from the tag dropdown +2. Tags help categorize and track different test scenarios + + + +### Vector Stores + +Configure vector store retrieval for RAG (Retrieval Augmented Generation) comparisons: + +1. Select vector stores from the dropdown +2. Compare how different models utilize retrieved context +3. Evaluate RAG performance across models + + + +## Running Comparisons + +### 1. Enter Your Prompt + +Type your test prompt in the message input area. You can: + +- Enter a single message for all models +- Use suggested prompts for quick testing +- Build multi-turn conversations + + + +### 2. Send Request + +Click the send button (or press Enter) to start the comparison. All selected models will process the request simultaneously. + +### 3. View Responses + +Responses appear side-by-side in each model panel, making it easy to compare: + +- Response quality and content +- Response length and structure +- Model-specific formatting + + + +## Comparison Metrics + +Each comparison panel displays detailed metrics to help you evaluate model performance: + +### Time To First Token (TTFT) + +Measures the latency from request submission to the first token received. Lower values indicate faster initial response times. + +### Token Usage + +- **Input Tokens**: Number of tokens in the prompt/request +- **Output Tokens**: Number of tokens in the model's response +- **Reasoning Tokens**: Tokens used for reasoning (if applicable, e.g., o1 models) + +### Total Latency + +Complete time from request to final response, including streaming time. + +### Cost + +If cost tracking is enabled in your LiteLLM configuration, you'll see: + +- Cost per request +- Cost breakdown by input/output tokens +- Comparison of costs across models + + + +## Use Cases + +### Model Selection + +Compare multiple models on the same prompt to determine which performs best for your specific use case: + +- Response quality +- Response time +- Cost efficiency +- Token usage + +### Parameter Tuning + +Test different parameter configurations across models to find optimal settings: + +- Temperature variations +- Max token limits +- Advanced parameter combinations + +### Guardrail Testing + +Evaluate how different models respond to safety filters and guardrails: + +- Filter effectiveness +- False positive rates +- Model-specific guardrail behavior + +### A/B Testing + +Use tags and multiple comparisons to run structured A/B tests: + +- Compare model versions +- Test prompt variations +- Evaluate feature rollouts + +--- + +## Related Features + +- [Playground Chat UI](./playground.md) - Single model testing interface +- [Model Management](./model_management.md) - Configure and manage models +- [Guardrails](./guardrails.md) - Set up safety filters +- [AI Hub](./ai_hub.md) - Share models and agents with your organization diff --git a/docs/my-website/img/model_compare_overview.png b/docs/my-website/img/model_compare_overview.png new file mode 100644 index 00000000000..f4af0eaee3c Binary files /dev/null and b/docs/my-website/img/model_compare_overview.png differ diff --git a/docs/my-website/img/ui_model_compare_cost_metrics.png b/docs/my-website/img/ui_model_compare_cost_metrics.png new file mode 100644 index 00000000000..b4639348c88 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_cost_metrics.png differ diff --git a/docs/my-website/img/ui_model_compare_enter_prompt.png b/docs/my-website/img/ui_model_compare_enter_prompt.png new file mode 100644 index 00000000000..af643abf6b8 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_enter_prompt.png differ diff --git a/docs/my-website/img/ui_model_compare_guardrails_config.png b/docs/my-website/img/ui_model_compare_guardrails_config.png new file mode 100644 index 00000000000..a85f9901299 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_guardrails_config.png differ diff --git a/docs/my-website/img/ui_model_compare_model_parameters.png b/docs/my-website/img/ui_model_compare_model_parameters.png new file mode 100644 index 00000000000..1ad0dfc4095 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_model_parameters.png differ diff --git a/docs/my-website/img/ui_model_compare_overview.png b/docs/my-website/img/ui_model_compare_overview.png new file mode 100644 index 00000000000..f4af0eaee3c Binary files /dev/null and b/docs/my-website/img/ui_model_compare_overview.png differ diff --git a/docs/my-website/img/ui_model_compare_responses.png b/docs/my-website/img/ui_model_compare_responses.png new file mode 100644 index 00000000000..5d207cd0155 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_responses.png differ diff --git a/docs/my-website/img/ui_model_compare_select_model.png b/docs/my-website/img/ui_model_compare_select_model.png new file mode 100644 index 00000000000..ba7bf948fcc Binary files /dev/null and b/docs/my-website/img/ui_model_compare_select_model.png differ diff --git a/docs/my-website/img/ui_model_compare_sync_across_models.png b/docs/my-website/img/ui_model_compare_sync_across_models.png new file mode 100644 index 00000000000..d59696a4bd2 Binary files /dev/null and b/docs/my-website/img/ui_model_compare_sync_across_models.png differ diff --git a/docs/my-website/img/ui_model_compare_tags_config.png b/docs/my-website/img/ui_model_compare_tags_config.png new file mode 100644 index 00000000000..bf36d9a987e Binary files /dev/null and b/docs/my-website/img/ui_model_compare_tags_config.png differ diff --git a/docs/my-website/img/ui_model_compare_vector_stores_config.png b/docs/my-website/img/ui_model_compare_vector_stores_config.png new file mode 100644 index 00000000000..b3bae046abf Binary files /dev/null and b/docs/my-website/img/ui_model_compare_vector_stores_config.png differ diff --git a/docs/my-website/img/ui_playground_navigation.png b/docs/my-website/img/ui_playground_navigation.png new file mode 100644 index 00000000000..202224b4069 Binary files /dev/null and b/docs/my-website/img/ui_playground_navigation.png differ diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 376bcfd5357..08857ce07b4 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -151,6 +151,7 @@ const sidebars = { "proxy/custom_root_ui", "proxy/custom_sso", "proxy/ai_hub", + "proxy/model_compare_ui", "proxy/public_teams", "proxy/self_serve", "proxy/ui/bulk_edit_users",