mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
Merge remote-tracking branch 'origin' into docs_yj_feb7
This commit is contained in:
yuneng-jiang
commit
ca24f56b39
6 changed files with 205 additions and 58 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
slug: litellm-observatory
|
||||
title: "LiteLLM Observatory: Raising the Bar for Release Reliability"
|
||||
title: "Improve release stability with 24 hour load tests"
|
||||
date: 2026-02-06T10:00:00
|
||||
authors:
|
||||
- name: Alexsander Hamir
|
||||
|
|
@ -22,7 +22,7 @@ hide_table_of_contents: false
|
|||
|
||||
|
||||
|
||||
# LiteLLM Observatory: Raising the Bar for Release Reliability
|
||||
# Improve release stability with 24 hour load tests
|
||||
|
||||
As LiteLLM adoption has grown, so have expectations around reliability, performance, and operational safety. Meeting those expectations requires more than correctness-focused tests, it requires validating how the system behaves over time, under real-world conditions.
|
||||
|
||||
|
|
@ -53,22 +53,23 @@ The issue that surfaced was not caused by a single incorrect line of logic, but
|
|||
Cannot send a request, as the client has been closed
|
||||
```
|
||||
|
||||
**Before (with bug):**
|
||||
|
||||
| Provider | Requests | Success | Failures | Fail % |
|
||||
|----------|----------|---------|----------|--------|
|
||||
| OpenAI | 720,000 | 432,000 | 288,000 | 40% |
|
||||
| Azure | 692,000 | 415,200 | 276,800 | 40% |
|
||||
|
||||
**After (fixed):**
|
||||
|
||||
| Provider | Requests | Success | Failures | Fail % |
|
||||
|----------|------------|-----------|----------|---------|
|
||||
| OpenAI | 1,200,000 | 1,199,988 | 12 | 0.001% |
|
||||
| Azure | 1,150,000 | 1,149,982 | 18 | 0.002% |
|
||||
|
||||
Our focus moving forward is on being the first to detect issues, even when they aren’t covered by unit tests. LiteLLM Observatory is designed to surface latency regressions, OOMs, and failure modes that only appear under real traffic patterns in **our own production deployments** during release validation.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Introducing LiteLLM Observatory
|
||||
|
||||
To systematically address this class of issues, we built **LiteLLM Observatory**.
|
||||
|
||||
The Observatory is a long-running testing orchestrator used during release validation to exercise LiteLLM under production-like conditions for extended periods of time.
|
||||
|
||||
Its core goals are:
|
||||
|
||||
- Validate behavior over hours, not minutes
|
||||
- Turn production learnings into permanent release safeguards
|
||||
|
||||
---
|
||||
|
||||
### How the Observatory Works
|
||||
|
|
@ -132,5 +133,4 @@ Reliability is an ongoing investment.
|
|||
LiteLLM Observatory is one of several systems we’re building to continuously raise the bar on release quality and operational safety. As LiteLLM evolves, so will our validation tooling, informed by real-world usage and lessons learned.
|
||||
|
||||
We’ll continue to share those improvements openly as we go.
|
||||
```
|
||||
|
||||
|
|
|
|||
BIN
docs/my-website/img/release_notes/mcp_internet.png
Normal file
BIN
docs/my-website/img/release_notes/mcp_internet.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 724 KiB |
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
title: "v1.81.9 - Claude Opus 4.6, A2A Agent Gateway, MCP Semantic Filtering & IP Controls"
|
||||
title: "v1.81.9 - Control which MCP Servers are exposed on the Internet"
|
||||
slug: "v1-81-9"
|
||||
date: 2026-02-07T00:00:00
|
||||
authors:
|
||||
|
|
@ -42,9 +42,9 @@ pip install litellm==1.81.9
|
|||
|
||||
## Key Highlights
|
||||
|
||||
- **Claude Opus 4.6** - Full support across Anthropic, AWS Bedrock, Azure AI, and Vertex AI with adaptive thinking and 1M context window
|
||||
- **A2A Agent Gateway** - Call A2A (Agent-to-Agent) registered agents through the standard `/chat/completions` API
|
||||
- **Expose MCP servers on the public internet** - Launch MCP servers with public/private visibility and IP-based access control for internet-facing deployments
|
||||
- **Claude Opus 4.6** - [Full support across Anthropic, AWS Bedrock, Azure AI, and Vertex AI with adaptive thinking and 1M context window](../../blog/claude_opus_4_6)
|
||||
- **A2A Agent Gateway** - [Call A2A (Agent-to-Agent) registered agents through the standard `/chat/completions` API](../../docs/a2a_invoking_agents)
|
||||
- **Expose MCP servers on the public internet** - [Launch MCP servers with public/private visibility and IP-based access control for internet-facing deployments](../../docs/mcp_public_internet)
|
||||
- **Performance Optimizations** - Multiple performance improvements including ~40% Prometheus CPU reduction, LRU caching, and optimized logging paths
|
||||
|
||||
---
|
||||
|
|
@ -55,8 +55,10 @@ This release makes it safe to expose MCP servers on the public internet by addin
|
|||
|
||||
[Get started](../../docs/mcp_public_internet)
|
||||
|
||||
{/* TODO: Add screenshot - save as mcp_public_internet.png */}
|
||||
{/* <Image img={require('../../img/mcp_public_internet.png')} style={{ maxWidth: '900px', width: '100%' }} /> */}
|
||||
<Image
|
||||
img={require('../img/release_notes/mcp_internet.png')}
|
||||
style={{ maxWidth: '900px', width: '100%' }}
|
||||
/>
|
||||
|
||||
Let's dive in.
|
||||
|
||||
|
|
@ -66,29 +68,27 @@ Let's dive in.
|
|||
|
||||
#### New Model Support (13 new models)
|
||||
|
||||
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features |
|
||||
| -------- | ----- | -------------- | ------------------- | -------------------- | -------- |
|
||||
| Anthropic | `claude-opus-4-6` | 1M | $5.00 | $25.00 | Chat, vision, reasoning, function calling, computer use, PDF, prompt caching |
|
||||
| AWS Bedrock | `anthropic.claude-opus-4-6-v1` | 1M | $5.00 | $25.00 | Chat, vision, reasoning, function calling, computer use, PDF, prompt caching |
|
||||
| Azure AI | `azure_ai/claude-opus-4-6` | 200K | $5.00 | $25.00 | Chat, vision, reasoning, function calling, computer use, PDF, prompt caching |
|
||||
| Vertex AI | `vertex_ai/claude-opus-4-6` | 1M | $5.00 | $25.00 | Chat, vision, reasoning, function calling, computer use, PDF, prompt caching |
|
||||
| Google Gemini | `gemini/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 | Deep research |
|
||||
| Vertex AI | `vertex_ai/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 | Deep research |
|
||||
| Moonshot | `moonshot/kimi-k2.5` | 262K | $0.60 | $3.00 | Chat, vision, function calling |
|
||||
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-2507` | 262K | $0.07 | $0.10 | Chat, function calling |
|
||||
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-thinking-2507` | 262K | $0.11 | $0.60 | Chat, function calling, reasoning |
|
||||
| Together AI | `together_ai/zai-org/GLM-4.7` | 200K | $0.45 | $2.00 | Chat, function calling, reasoning |
|
||||
| Together AI | `together_ai/moonshotai/Kimi-K2.5` | 256K | $0.50 | $2.80 | Chat, vision, function calling, reasoning |
|
||||
| ElevenLabs | `elevenlabs/eleven_v3` | - | $0.18/1K chars | - | Text-to-speech, 70+ languages |
|
||||
| ElevenLabs | `elevenlabs/eleven_multilingual_v2` | - | $0.18/1K chars | - | Text-to-speech, 29 languages |
|
||||
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) |
|
||||
| -------- | ----- | -------------- | ------------------- | -------------------- |
|
||||
| Anthropic | `claude-opus-4-6` | 1M | $5.00 | $25.00 |
|
||||
| AWS Bedrock | `anthropic.claude-opus-4-6-v1` | 1M | $5.00 | $25.00 |
|
||||
| Azure AI | `azure_ai/claude-opus-4-6` | 200K | $5.00 | $25.00 |
|
||||
| Vertex AI | `vertex_ai/claude-opus-4-6` | 1M | $5.00 | $25.00 |
|
||||
| Google Gemini | `gemini/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 |
|
||||
| Vertex AI | `vertex_ai/deep-research-pro-preview-12-2025` | 65K | $2.00 | $12.00 |
|
||||
| Moonshot | `moonshot/kimi-k2.5` | 262K | $0.60 | $3.00 |
|
||||
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-2507` | 262K | $0.07 | $0.10 |
|
||||
| OpenRouter | `openrouter/qwen/qwen3-235b-a22b-thinking-2507` | 262K | $0.11 | $0.60 |
|
||||
| Together AI | `together_ai/zai-org/GLM-4.7` | 200K | $0.45 | $2.00 |
|
||||
| Together AI | `together_ai/moonshotai/Kimi-K2.5` | 256K | $0.50 | $2.80 |
|
||||
| ElevenLabs | `elevenlabs/eleven_v3` | - | $0.18/1K chars | - |
|
||||
| ElevenLabs | `elevenlabs/eleven_multilingual_v2` | - | $0.18/1K chars | - |
|
||||
|
||||
#### Features
|
||||
|
||||
- **[Anthropic](../../docs/providers/anthropic)**
|
||||
- Full Claude Opus 4.6 support with adaptive thinking across all regions (us, eu, apac, au) - [PR #20506](https://github.com/BerriAI/litellm/pull/20506), [PR #20508](https://github.com/BerriAI/litellm/pull/20508), [PR #20514](https://github.com/BerriAI/litellm/pull/20514), [PR #20551](https://github.com/BerriAI/litellm/pull/20551)
|
||||
- Add unsupported claude code beta headers filtering - [PR #20578](https://github.com/BerriAI/litellm/pull/20578)
|
||||
- Map reasoning content to anthropic thinking block (streaming + non-streaming) - [PR #20254](https://github.com/BerriAI/litellm/pull/20254)
|
||||
- Fix inconsistent response format in `anthropic.messages.acreate()` when using non-Anthropic providers - [PR #20442](https://github.com/BerriAI/litellm/pull/20442)
|
||||
|
||||
- **[AWS Bedrock](../../docs/providers/bedrock)**
|
||||
- Add 1hr tiered caching costs for long-context models - [PR #20214](https://github.com/BerriAI/litellm/pull/20214)
|
||||
|
|
@ -150,14 +150,14 @@ Let's dive in.
|
|||
|
||||
---
|
||||
|
||||
## LLM API Endpoints (2 new endpoints)
|
||||
## LLM API Endpoints
|
||||
|
||||
#### Features
|
||||
|
||||
| Endpoint | Method | Description | Documentation |
|
||||
| -------- | ------ | ----------- | ------------- |
|
||||
| `/chat/completions` | POST | A2A agent invocation via standard completions API | [Docs](../../docs/a2a) |
|
||||
| `/files` | GET/DELETE | Gemini file retrieval and deletion by `file_id` | [Docs](../../docs/providers/gemini) |
|
||||
- **[Messages API](../../docs/providers/anthropic)**
|
||||
- Filter unsupported Claude Code beta headers for non-Anthropic providers - [PR #20578](https://github.com/BerriAI/litellm/pull/20578)
|
||||
- Fix inconsistent response format in `anthropic.messages.acreate()` when using non-Anthropic providers - [PR #20442](https://github.com/BerriAI/litellm/pull/20442)
|
||||
- Fix 404 on `/api/event_logging/batch` endpoint that caused Claude Code "route not found" errors - [PR #20504](https://github.com/BerriAI/litellm/pull/20504)
|
||||
|
||||
- **[A2A Agent Gateway](../../docs/a2a)**
|
||||
- Allow calling A2A agents through LiteLLM `/chat/completions` API - [PR #20358](https://github.com/BerriAI/litellm/pull/20358)
|
||||
|
|
@ -221,7 +221,6 @@ Let's dive in.
|
|||
- Fix `/key/list` `user_id` Empty String Edge Case - [PR #20623](https://github.com/BerriAI/litellm/pull/20623)
|
||||
- Add array type checks for model, agent, and MCP hub data to prevent UI crashes - [PR #20469](https://github.com/BerriAI/litellm/pull/20469)
|
||||
- Fix unique constraint on daily tables + logging when updates fail - [PR #20394](https://github.com/BerriAI/litellm/pull/20394)
|
||||
- Fix 404 Not Found on `/api/event_logging/batch` endpoint - [PR #20504](https://github.com/BerriAI/litellm/pull/20504)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -312,9 +311,8 @@ Let's dive in.
|
|||
|
||||
---
|
||||
|
||||
## Documentation Updates (15 updates)
|
||||
## Documentation Updates (14 updates)
|
||||
|
||||
- Update v1.81.6 release notes - Logs v2 with Tool Call Tracing - [PR #20225](https://github.com/BerriAI/litellm/pull/20225)
|
||||
- Add FAQ for setting up and verifying LITELLM_LICENSE - [PR #20284](https://github.com/BerriAI/litellm/pull/20284)
|
||||
- Model request tags documentation - [PR #20290](https://github.com/BerriAI/litellm/pull/20290)
|
||||
- Add Prisma migration troubleshooting guide - [PR #20300](https://github.com/BerriAI/litellm/pull/20300)
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from typing import Any, Dict, Optional, Tuple
|
|||
from litellm._logging import verbose_logger
|
||||
from litellm._uuid import uuid
|
||||
from litellm.integrations.custom_batch_logger import CustomBatchLogger
|
||||
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
|
||||
from litellm.integrations.posthog_mock_client import (
|
||||
should_use_posthog_mock,
|
||||
create_mock_posthog_client,
|
||||
|
|
@ -100,7 +101,7 @@ class PostHogLogger(CustomBatchLogger):
|
|||
|
||||
response = self.sync_client.post(
|
||||
url=capture_url,
|
||||
json=payload,
|
||||
content=safe_dumps(payload),
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
|
@ -356,7 +357,7 @@ class PostHogLogger(CustomBatchLogger):
|
|||
|
||||
response = await self.async_client.post(
|
||||
url=capture_url,
|
||||
json=payload,
|
||||
content=safe_dumps(payload),
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
|
@ -438,7 +439,7 @@ class PostHogLogger(CustomBatchLogger):
|
|||
|
||||
response = self.sync_client.post(
|
||||
url=capture_url,
|
||||
json=payload,
|
||||
content=safe_dumps(payload),
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
|
|
|||
|
|
@ -4942,13 +4942,17 @@ class ProxyStartupEvent:
|
|||
"""Initialize MCP semantic tool filter if configured"""
|
||||
from litellm.proxy.hooks.mcp_semantic_filter import SemanticToolFilterHook
|
||||
|
||||
verbose_proxy_logger.info(
|
||||
f"Initializing semantic tool filter: llm_router={llm_router is not None}, "
|
||||
f"litellm_settings keys={list(litellm_settings.keys())}"
|
||||
)
|
||||
|
||||
mcp_semantic_filter_config = litellm_settings.get("mcp_semantic_tool_filter", None)
|
||||
verbose_proxy_logger.debug(f"Semantic filter config: {mcp_semantic_filter_config}")
|
||||
|
||||
# Only proceed if the feature is configured and enabled
|
||||
if not mcp_semantic_filter_config or not mcp_semantic_filter_config.get("enabled", False):
|
||||
verbose_proxy_logger.debug("Semantic tool filter not configured or not enabled, skipping initialization")
|
||||
return
|
||||
|
||||
verbose_proxy_logger.debug(
|
||||
f"Initializing semantic tool filter: llm_router={llm_router is not None}, "
|
||||
f"config={mcp_semantic_filter_config}"
|
||||
)
|
||||
|
||||
hook = await SemanticToolFilterHook.initialize_from_config(
|
||||
config=mcp_semantic_filter_config,
|
||||
|
|
@ -4956,10 +4960,11 @@ class ProxyStartupEvent:
|
|||
)
|
||||
|
||||
if hook:
|
||||
verbose_proxy_logger.debug("✅ Semantic tool filter hook registered")
|
||||
verbose_proxy_logger.debug("Semantic tool filter hook registered")
|
||||
litellm.logging_callback_manager.add_litellm_callback(hook)
|
||||
else:
|
||||
verbose_proxy_logger.warning("❌ Semantic tool filter hook not initialized")
|
||||
# Only warn if the feature was configured but failed to initialize
|
||||
verbose_proxy_logger.warning("Semantic tool filter hook was configured but failed to initialize")
|
||||
|
||||
@classmethod
|
||||
def _initialize_jwt_auth(
|
||||
|
|
|
|||
|
|
@ -381,6 +381,149 @@ async def test_posthog_atexit_flushes_internal_queue():
|
|||
assert "/batch/" in call_args.kwargs['url'], "Should POST to /batch/ endpoint"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_safe_dumps_serialization_in_sync_log():
|
||||
"""
|
||||
Regression test: sync log_success_event should not raise when the payload
|
||||
contains objects that are not natively JSON-serializable (e.g. Pydantic
|
||||
models like UserAPIKeyAuth).
|
||||
|
||||
Before the fix httpx's json= kwarg called stdlib json.dumps which would
|
||||
raise ``TypeError: Object of type UserAPIKeyAuth is not JSON serializable``.
|
||||
After the fix the body is pre-serialized via safe_dumps() and sent with
|
||||
content= so non-primitive values are coerced to their str() representation.
|
||||
"""
|
||||
from unittest.mock import Mock, patch
|
||||
from pydantic import BaseModel
|
||||
|
||||
class FakeNonSerializable(BaseModel):
|
||||
"""Stand-in for UserAPIKeyAuth or any Pydantic object in metadata."""
|
||||
token: str = "sk-secret"
|
||||
|
||||
posthog_logger = PostHogLogger()
|
||||
standard_payload = create_standard_logging_payload()
|
||||
|
||||
kwargs = {
|
||||
"standard_logging_object": standard_payload,
|
||||
"litellm_params": {
|
||||
"metadata": {
|
||||
# This custom key would leak a non-serializable object into
|
||||
# the PostHog properties dict:
|
||||
"custom_auth_obj": FakeNonSerializable(),
|
||||
}
|
||||
},
|
||||
"standard_callback_dynamic_params": None,
|
||||
}
|
||||
|
||||
with patch.object(posthog_logger.sync_client, "post") as mock_post:
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.raise_for_status = Mock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
# Should NOT raise TypeError
|
||||
posthog_logger.log_success_event(kwargs, None, 0.0, 0.0)
|
||||
|
||||
assert mock_post.called, "sync_client.post should have been called"
|
||||
call_kwargs = mock_post.call_args.kwargs
|
||||
# Must use content= (pre-serialized), NOT json=
|
||||
assert "content" in call_kwargs, "Should send pre-serialized body via content="
|
||||
assert "json" not in call_kwargs, "Should NOT use json= kwarg"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_safe_dumps_serialization_in_async_send_batch():
|
||||
"""
|
||||
Regression test: async_send_batch should not raise when the event payload
|
||||
contains non-JSON-serializable objects.
|
||||
"""
|
||||
from unittest.mock import Mock, AsyncMock, patch
|
||||
from pydantic import BaseModel
|
||||
|
||||
class FakeNonSerializable(BaseModel):
|
||||
token: str = "sk-secret"
|
||||
|
||||
posthog_logger = PostHogLogger()
|
||||
standard_payload = create_standard_logging_payload()
|
||||
|
||||
kwargs = {
|
||||
"standard_logging_object": standard_payload,
|
||||
"litellm_params": {
|
||||
"metadata": {
|
||||
"custom_auth_obj": FakeNonSerializable(),
|
||||
}
|
||||
},
|
||||
}
|
||||
event_payload = posthog_logger.create_posthog_event_payload(kwargs)
|
||||
|
||||
posthog_logger.log_queue.append({
|
||||
"event": event_payload,
|
||||
"api_key": "test_key",
|
||||
"api_url": "https://app.posthog.com",
|
||||
})
|
||||
|
||||
with patch.object(posthog_logger.async_client, "post") as mock_post:
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.raise_for_status = Mock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
# Should NOT raise TypeError
|
||||
await posthog_logger.async_send_batch()
|
||||
|
||||
assert mock_post.called, "async_client.post should have been called"
|
||||
call_kwargs = mock_post.call_args.kwargs
|
||||
assert "content" in call_kwargs, "Should send pre-serialized body via content="
|
||||
assert "json" not in call_kwargs, "Should NOT use json= kwarg"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_safe_dumps_serialization_in_flush_on_exit():
|
||||
"""
|
||||
Regression test: _flush_on_exit (atexit path) should not raise when the
|
||||
event payload contains non-JSON-serializable objects.
|
||||
"""
|
||||
from unittest.mock import Mock, patch
|
||||
from pydantic import BaseModel
|
||||
|
||||
class FakeNonSerializable(BaseModel):
|
||||
token: str = "sk-secret"
|
||||
|
||||
posthog_logger = PostHogLogger()
|
||||
standard_payload = create_standard_logging_payload()
|
||||
|
||||
kwargs = {
|
||||
"standard_logging_object": standard_payload,
|
||||
"litellm_params": {
|
||||
"metadata": {
|
||||
"custom_auth_obj": FakeNonSerializable(),
|
||||
}
|
||||
},
|
||||
}
|
||||
event_payload = posthog_logger.create_posthog_event_payload(kwargs)
|
||||
|
||||
posthog_logger.log_queue.append({
|
||||
"event": event_payload,
|
||||
"api_key": "test_key",
|
||||
"api_url": "https://app.posthog.com",
|
||||
})
|
||||
|
||||
with patch.object(posthog_logger.sync_client, "post") as mock_post:
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.raise_for_status = Mock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
# Should NOT raise TypeError
|
||||
posthog_logger._flush_on_exit()
|
||||
|
||||
assert mock_post.called, "sync_client.post should have been called"
|
||||
call_kwargs = mock_post.call_args.kwargs
|
||||
assert "content" in call_kwargs, "Should send pre-serialized body via content="
|
||||
assert "json" not in call_kwargs, "Should NOT use json= kwarg"
|
||||
assert len(posthog_logger.log_queue) == 0, "Queue should be empty after flush"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_sync_callback_not_affected_by_atexit():
|
||||
"""
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue