diff --git a/docs/my-website/blog/vllm_embeddings_incident/index.md b/docs/my-website/blog/vllm_embeddings_incident/index.md new file mode 100644 index 00000000000..a1ce8152857 --- /dev/null +++ b/docs/my-website/blog/vllm_embeddings_incident/index.md @@ -0,0 +1,117 @@ +--- +slug: vllm-embeddings-incident +title: "Incident Report: vLLM Embeddings Broken by encoding_format Parameter" +date: 2026-02-18T10:00:00 +authors: + - name: Sameer Kankute + title: SWE @ LiteLLM (LLM Translation) + url: https://www.linkedin.com/in/sameer-kankute/ + image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg + - name: Krrish Dholakia + title: "CEO, LiteLLM" + url: https://www.linkedin.com/in/krish-d/ + image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg + - name: Ishaan Jaff + title: "CTO, LiteLLM" + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg +tags: [incident-report, embeddings, vllm] +hide_table_of_contents: false +--- + +**Date:** Feb 16, 2026 +**Duration:** ~3 hours +**Severity:** High (for vLLM embedding users) +**Status:** Resolved + +## Summary + +A commit ([`dbcae4a`](https://github.com/BerriAI/litellm/commit/dbcae4aca5836770d0e9cd43abab0333c3d61ab2)) intended to fix OpenAI SDK behavior broke vLLM embeddings by explicitly passing `encoding_format=None` in API requests. vLLM rejects this with error: `"unknown variant \`\`, expected float or base64"`. + +- **vLLM embedding calls:** Complete failure - all requests rejected +- **Other providers:** No impact - OpenAI and other providers functioned normally +- **Other vLLM functionality:** No impact - only embeddings were affected + +{/* truncate */} + +--- + +## Background + +The `encoding_format` parameter for embeddings specifies whether vectors should be returned as `float` arrays or `base64` encoded strings. Different providers have different expectations: + +- **OpenAI SDK:** If `encoding_format` is omitted, the SDK adds a default value of `"float"` +- **vLLM:** Strictly validates `encoding_format` - only accepts `"float"`, `"base64"`, or complete omission. Rejects `None` or empty string values. + +```mermaid +flowchart TD + A["1. User calls litellm.embedding() + litellm/main.py"] --> B["2. Transform request for provider + litellm/llms/openai_like/embedding/handler.py"] + B --> C["3. Send request to vLLM endpoint"] + C -->|"encoding_format omitted"| D["4a. ✅ vLLM processes request"] + C -->|"encoding_format='float' or 'base64'"| D + C -->|"encoding_format=None or ''"| E["4b. ❌ vLLM rejects with error: + 'unknown variant, expected float or base64'"] + + style D fill:#d4edda,stroke:#28a745 + style E fill:#f8d7da,stroke:#dc3545 + style B fill:#fff3cd,stroke:#ffc107 +``` + +--- + +## Root cause + +A well-intentioned fix for OpenAI SDK behavior inadvertently broke vLLM embeddings: + +**The Breaking Change ([`dbcae4a`](https://github.com/BerriAI/litellm/commit/dbcae4aca5836770d0e9cd43abab0333c3d61ab2)):** + +In `litellm/main.py`, the code was changed to explicitly set `encoding_format=None` instead of omitting it: + +```python +# Added in dbcae4a +if encoding_format is not None: + optional_params["encoding_format"] = encoding_format +else: + # Omitting causes openai sdk to add default value of "float" + optional_params["encoding_format"] = None +``` + +This fix worked correctly for OpenAI - explicitly passing `None` prevented the SDK from adding its default value. However, vLLM's strict parameter validation rejected `None` values, causing all embedding requests to fail. + +--- + +## The Fix + +Fix deployed ([`55348dd`](https://github.com/BerriAI/litellm/commit/55348dd9c51b5b028f676d25ad023b8f052fc071)). The solution filters out `None` and empty string values from `optional_params` before sending requests to OpenAI-like providers (including vLLM). + +**In `litellm/llms/openai_like/embedding/handler.py`:** + +```python +# Before (broken) +data = {"model": model, "input": input, **optional_params} + +# After (fixed) +filtered_optional_params = {k: v for k, v in optional_params.items() if v not in (None, '')} +data = {"model": model, "input": input, **filtered_optional_params} +``` + +This ensures: +- Valid values (`"float"`, `"base64"`) are preserved and sent +- `None` and empty string values are filtered out (parameter omitted entirely) +- OpenAI SDK no longer adds defaults because liteLLM handles the parameter upstream + +--- + +## Remediation + +| # | Action | Status | Code | +|---|---|---|---| +| 1 | Filter `None` and empty string values in OpenAI-like embedding handler | ✅ Done | [`handler.py#L108`](https://github.com/BerriAI/litellm/blob/main/litellm/llms/openai_like/embedding/handler.py#L108) | +| 2 | Unit tests for parameter filtering (None, empty string, valid values) | ✅ Done | [`test_openai_like_embedding.py`](https://github.com/BerriAI/litellm/blob/main/tests/test_litellm/llms/openai_like/embedding/test_openai_like_embedding.py) | +| 3 | Transformation tests for hosted_vllm embedding config | ✅ Done | [`test_hosted_vllm_embedding_transformation.py`](https://github.com/BerriAI/litellm/blob/main/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_transformation.py) | +| 4 | E2E tests with actual vLLM endpoint | ✅ Done | [`test_hosted_vllm_embedding_e2e.py`](https://github.com/BerriAI/litellm/blob/main/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_e2e.py) | +| 5 | Validate JSON payload structure matches vLLM expectations | ✅ Done | Tests verify exact JSON sent to endpoint | + +--- diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json index 04183e398eb..ae89a82e964 100644 --- a/litellm/model_prices_and_context_window_backup.json +++ b/litellm/model_prices_and_context_window_backup.json @@ -22465,6 +22465,20 @@ "supports_response_schema": true, "supports_tool_choice": true }, + "mistral/devstral-small-latest": { + "input_cost_per_token": 1e-07, + "litellm_provider": "mistral", + "max_input_tokens": 256000, + "max_output_tokens": 256000, + "max_tokens": 256000, + "mode": "chat", + "output_cost_per_token": 3e-07, + "source": "https://docs.mistral.ai/models/devstral-small-2-25-12", + "supports_assistant_prefill": true, + "supports_function_calling": true, + "supports_response_schema": true, + "supports_tool_choice": true + }, "mistral/labs-devstral-small-2512": { "input_cost_per_token": 1e-07, "litellm_provider": "mistral", @@ -22479,6 +22493,34 @@ "supports_response_schema": true, "supports_tool_choice": true }, + "mistral/devstral-latest": { + "input_cost_per_token": 4e-07, + "litellm_provider": "mistral", + "max_input_tokens": 256000, + "max_output_tokens": 256000, + "max_tokens": 256000, + "mode": "chat", + "output_cost_per_token": 2e-06, + "source": "https://mistral.ai/news/devstral-2-vibe-cli", + "supports_assistant_prefill": true, + "supports_function_calling": true, + "supports_response_schema": true, + "supports_tool_choice": true + }, + "mistral/devstral-medium-latest": { + "input_cost_per_token": 4e-07, + "litellm_provider": "mistral", + "max_input_tokens": 256000, + "max_output_tokens": 256000, + "max_tokens": 256000, + "mode": "chat", + "output_cost_per_token": 2e-06, + "source": "https://mistral.ai/news/devstral-2-vibe-cli", + "supports_assistant_prefill": true, + "supports_function_calling": true, + "supports_response_schema": true, + "supports_tool_choice": true + }, "mistral/devstral-2512": { "input_cost_per_token": 4e-07, "litellm_provider": "mistral", diff --git a/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_transformation.py b/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_transformation.py index f3842214e4b..4cb20154570 100644 --- a/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_transformation.py +++ b/tests/test_litellm/llms/hosted_vllm/embedding/test_hosted_vllm_embedding_transformation.py @@ -232,23 +232,6 @@ class TestHostedVLLMEmbeddingTransformation: assert result["Authorization"] == "Bearer test-api-key" assert result["Content-Type"] == "application/json" - def test_validate_environment_without_api_key(self): - """Test environment validation without API key (uses fake-api-key).""" - headers = {} - - result = self.config.validate_environment( - headers=headers, - model=self.model, - messages=[], - optional_params={}, - litellm_params={}, - api_key=None, - ) - - # Should not include Authorization header with fake-api-key - assert "Authorization" not in result - assert result["Content-Type"] == "application/json" - def test_encoding_format_not_sent_in_actual_request(self): """ E2E test that encoding_format is not sent when not provided. @@ -306,61 +289,5 @@ class TestHostedVLLMEmbeddingTransformation: assert sent_data["model"] == "BAAI/bge-small-en-v1.5" assert sent_data["input"] == ["Hello world"] - def test_encoding_format_float_sent_in_actual_request(self): - """ - Test that encoding_format='float' is sent when explicitly provided. - """ - from litellm.llms.custom_httpx.http_handler import HTTPHandler - - client = HTTPHandler() - - with patch.object(client, "post") as mock_post: - # Mock response - mock_response = Mock() - mock_response.status_code = 200 - mock_response.headers = {"content-type": "application/json"} - mock_response.json.return_value = { - "object": "list", - "data": [ - { - "object": "embedding", - "index": 0, - "embedding": [0.1, 0.2, 0.3, 0.4, 0.5], - } - ], - "model": "BAAI/bge-small-en-v1.5", - "usage": { - "prompt_tokens": 5, - "total_tokens": 5, - }, - } - mock_response.text = json.dumps(mock_response.json.return_value) - mock_post.return_value = mock_response - - try: - litellm.embedding( - model=self.model, - input=["Hello world"], - api_base="https://test-vllm.example.com/v1", - encoding_format="float", - client=client, - ) - except Exception: - pass - - # Verify the request was made - mock_post.assert_called_once() - - # Get the data that was sent - call_kwargs = mock_post.call_args[1] - sent_data = json.loads(call_kwargs["data"]) - - # Assert that encoding_format IS in the sent data - assert "encoding_format" in sent_data, ( - "encoding_format='float' should be in request when provided" - ) - assert sent_data["encoding_format"] == "float" - - if __name__ == "__main__": pytest.main([__file__, "-v", "-s"])