mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
Merge pull request #26092 from BerriAI/litellm_claude_code_any_advisor_tutorial
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
Some checks failed
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
docs: add Claude Code any-advisor tutorial and fix streaming logging
This commit is contained in:
commit
0f16133adb
14 changed files with 1147 additions and 160 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
slug: advisor-tool-chat-completions
|
||||
title: "Advisor Tool (SDK + Proxy)"
|
||||
title: "[Beta] Advisor Tool (SDK + Proxy)"
|
||||
date: 2026-04-14T19:30:00
|
||||
authors:
|
||||
- sameer
|
||||
|
|
@ -14,7 +14,7 @@ hide_table_of_contents: false
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Advisor Tool
|
||||
# Advisor Tool (Beta)
|
||||
|
||||
LiteLLM now supports the Anthropic advisor tool across `chat/completions` and `messages` APIs (SDK + proxy).
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Advisor Tool
|
||||
# Advisor Tool (Beta)
|
||||
|
||||
LiteLLM supports the Anthropic advisor tool across `chat/completions` and `messages` APIs (SDK + proxy).
|
||||
|
||||
|
|
@ -728,6 +728,102 @@ tools=[
|
|||
|
||||
---
|
||||
|
||||
## Remapping the advisor model
|
||||
|
||||
Some clients (e.g. Claude Code) hardcode the advisor tool's `model` field — you cannot change what the client sends. If you still want the advisor sub-call to hit a different model (for cost, availability, or routing reasons), remap the advisor model using `model_group_alias` on the router.
|
||||
|
||||
When the advisor tool's `model` resolves through `model_group_alias` to a **non-native Anthropic advisor model**, LiteLLM automatically takes over the orchestration loop — even when the executor is direct Anthropic — and routes the advisor sub-call through the router. The client keeps seeing the original alias in every response surface (`iterations[].model`), so the remap stays opaque to the caller.
|
||||
|
||||
```yaml showLineNumbers title="config.yaml — remap claude-opus-4-7 advisor to o3"
|
||||
model_list:
|
||||
- model_name: o3
|
||||
litellm_params:
|
||||
model: openai/o3
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-6
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias:
|
||||
claude-opus-4-7: o3
|
||||
```
|
||||
|
||||
With the config above, a client request that includes:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-opus-4-7"
|
||||
}
|
||||
```
|
||||
|
||||
will:
|
||||
|
||||
1. Run the executor against Anthropic as usual.
|
||||
2. When the executor calls the advisor, route the sub-call through the router to `openai/o3` using the `o3` deployment's credentials.
|
||||
3. Emit `iterations[].model == "claude-opus-4-7"` in the response so the client never sees `o3`.
|
||||
|
||||
:::info When does the remap trigger?
|
||||
|
||||
Only when the resolved model is **not** a native Anthropic advisor (currently `claude-opus-4-6` and `claude-opus-4-7`). If you alias one native advisor model to another (e.g. `claude-opus-4-7 -> claude-opus-4-6`), Anthropic's server-side advisor still handles the request.
|
||||
|
||||
:::
|
||||
|
||||
### Claude Code quickstart: use any advisor model
|
||||
|
||||
If you are using Claude Code and want to run the advisor on a non-Claude model (for example `openai/o3`, Gemini, Bedrock, etc.), use this pattern:
|
||||
|
||||
1. Keep Claude Code's advisor tool unchanged:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-opus-4-7"
|
||||
}
|
||||
```
|
||||
|
||||
2. Map that model name to your actual advisor deployment in LiteLLM:
|
||||
|
||||
```yaml showLineNumbers title="config.yaml — Claude Code advisor alias"
|
||||
model_list:
|
||||
- model_name: my-real-advisor
|
||||
litellm_params:
|
||||
model: openai/o3
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-6
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias:
|
||||
claude-opus-4-7: my-real-advisor
|
||||
```
|
||||
|
||||
3. Send requests through `/v1/messages` as usual from Claude Code.
|
||||
|
||||
What happens at runtime:
|
||||
|
||||
- Claude Code sends `model: "claude-opus-4-7"` in the advisor tool.
|
||||
- LiteLLM resolves it to `my-real-advisor` and routes the sub-call to `openai/o3`.
|
||||
- Claude Code still sees `claude-opus-4-7` in response-visible fields (alias stays opaque).
|
||||
|
||||
:::tip Troubleshooting (Claude Code + non-native advisor)
|
||||
|
||||
- If you see `Invalid value: 'thinking'` from OpenAI, upgrade to a LiteLLM build that includes advisor sub-call message translation for non-Anthropic providers.
|
||||
- If advisor output appears blank in streamed UI, use a build with `advisor_tool_result` text included in `content_block_start` for fake-streamed advisor responses.
|
||||
- If spend logs are missing for streamed advisor calls, use a build with deferred logging support for non-`CustomStreamWrapper` anthropic streams.
|
||||
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Additional resources
|
||||
|
||||
- [Anthropic Advisor Tool Documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool)
|
||||
|
|
|
|||
166
docs/my-website/docs/tutorials/claude_code_any_advisor_model.md
Normal file
166
docs/my-website/docs/tutorials/claude_code_any_advisor_model.md
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
# Claude Code: Any Advisor Model via LiteLLM (Beta)
|
||||
|
||||
This tutorial shows how to use **Claude Code's advisor tool** with **any model/provider** (OpenAI, Gemini, Bedrock, etc.) by routing through LiteLLM.
|
||||
|
||||
Claude Code sends advisor tools with a fixed Anthropic model name (for example `claude-opus-4-7`). LiteLLM can remap that to your real advisor model using `model_group_alias`.
|
||||
|
||||
<br />
|
||||
|
||||
<iframe width="840" height="500" src="https://www.loom.com/embed/db5bc6156c584e0998c7624821bd5272" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
|
||||
- LiteLLM proxy installed (`uv tool install 'litellm[proxy]'`)
|
||||
- API keys for:
|
||||
- your **executor** model (for example Anthropic Sonnet)
|
||||
- your **advisor** model (for example OpenAI `o3`)
|
||||
|
||||
## Step 1: Create `config.yaml`
|
||||
|
||||
Create a LiteLLM config where:
|
||||
|
||||
1. your executor model is configured in `model_list`
|
||||
2. your real advisor model is configured in `model_list`
|
||||
3. `router_settings.model_group_alias` remaps Claude Code's advisor model name to your real advisor deployment
|
||||
|
||||
```yaml showLineNumbers title="config.yaml"
|
||||
model_list:
|
||||
# Executor model (the main model Claude Code runs on)
|
||||
- model_name: claude-sonnet
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-6
|
||||
api_key: os.environ/ANTHROPIC_API_KEY
|
||||
|
||||
# Real advisor model (can be any provider)
|
||||
- model_name: my-advisor
|
||||
litellm_params:
|
||||
model: openai/o3
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
|
||||
router_settings:
|
||||
model_group_alias:
|
||||
# Claude Code sends this in advisor_20260301 tool model
|
||||
claude-opus-4-7: my-advisor
|
||||
|
||||
litellm_settings:
|
||||
drop_params: true
|
||||
```
|
||||
|
||||
Set env vars:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="your-anthropic-key"
|
||||
export OPENAI_API_KEY="your-openai-key"
|
||||
export LITELLM_MASTER_KEY="sk-1234"
|
||||
```
|
||||
|
||||
## Step 2: Start LiteLLM Proxy
|
||||
|
||||
```bash showLineNumbers title="Run LiteLLM Proxy"
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
Expected startup endpoint:
|
||||
|
||||
```bash
|
||||
# RUNNING on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
## Step 3: Point Claude Code to LiteLLM
|
||||
|
||||
Configure Claude Code to call your LiteLLM proxy:
|
||||
|
||||
```bash showLineNumbers title="Claude Code environment variables"
|
||||
export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
|
||||
export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
|
||||
export ANTHROPIC_MODEL="claude-sonnet"
|
||||
```
|
||||
|
||||
Then launch Claude Code:
|
||||
|
||||
```bash
|
||||
claude
|
||||
```
|
||||
|
||||
## Step 4: Call advisor from Claude Code
|
||||
|
||||
In Claude Code, ask for an advisor run (for example):
|
||||
|
||||
```text
|
||||
Can you call advisor as integration test and confirm it works?
|
||||
```
|
||||
|
||||
Claude Code will send an advisor tool like:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-opus-4-7"
|
||||
}
|
||||
```
|
||||
|
||||
LiteLLM will remap `claude-opus-4-7` -> `my-advisor` -> `openai/o3` and run the advisor loop.
|
||||
|
||||
## Step 5: Verify it is using your advisor model
|
||||
|
||||
Check proxy logs for advisor sub-calls:
|
||||
|
||||
```bash
|
||||
rg "advisor_sub_call|litellm.acompletion\\(|openai/o3|my-advisor" proxy_server.log
|
||||
```
|
||||
|
||||
You should see:
|
||||
|
||||
- the outer `/v1/messages` call on your executor (`claude-sonnet`)
|
||||
- advisor sub-call routed to your mapped model (`openai/o3`)
|
||||
|
||||
## Internal flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant CC as Claude Code
|
||||
participant LL as LiteLLM Proxy
|
||||
participant EX as Executor Model
|
||||
participant AD as Real Advisor Model
|
||||
|
||||
CC->>LL: /v1/messages + advisor_20260301 model=claude-opus-4-7
|
||||
LL->>LL: model_group_alias claude-opus-4-7 -> my-advisor
|
||||
LL->>EX: Executor call
|
||||
EX-->>LL: advisor tool_use
|
||||
LL->>AD: Advisor sub-call (my-advisor)
|
||||
AD-->>LL: advisor result
|
||||
LL->>EX: inject advisor result + continue
|
||||
EX-->>LL: final answer
|
||||
LL-->>CC: final response (client-facing advisor alias unchanged)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Why is my advisor model not found?
|
||||
|
||||
- Ensure alias target (`my-advisor`) exists as a `model_name` in `model_list`
|
||||
- Confirm with:
|
||||
|
||||
```bash
|
||||
curl http://0.0.0.0:4000/v1/models -H "Authorization: Bearer $LITELLM_MASTER_KEY"
|
||||
```
|
||||
|
||||
### Why do I get `Invalid value: 'thinking'` with a non-Anthropic advisor?
|
||||
|
||||
- Upgrade LiteLLM to a version that includes advisor sub-call message translation for non-Anthropic providers
|
||||
|
||||
### Why is advisor output blank in streamed UI?
|
||||
|
||||
- Upgrade LiteLLM to a version where `advisor_tool_result` includes text in `content_block_start` for fake-stream iterator responses
|
||||
|
||||
### Why are spend/log rows missing for streamed advisor calls?
|
||||
|
||||
- Upgrade LiteLLM to a version that adds deferred logging support for non-`CustomStreamWrapper` anthropic streams
|
||||
|
||||
## Related docs
|
||||
|
||||
- [Advisor Tool Reference](/docs/completion/anthropic_advisor_tool)
|
||||
- [Use Claude Code with Non-Anthropic Models](/docs/tutorials/claude_non_anthropic_models)
|
||||
- [Forward Client Headers](/docs/proxy/forward_client_headers)
|
||||
|
|
@ -147,6 +147,7 @@ const sidebars = {
|
|||
"tutorials/claude_code_websearch",
|
||||
"tutorials/claude_mcp",
|
||||
"tutorials/claude_non_anthropic_models",
|
||||
"tutorials/claude_code_any_advisor_model",
|
||||
"tutorials/claude_code_plugin_marketplace",
|
||||
"tutorials/claude_code_beta_headers",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1478,6 +1478,21 @@ class Logging(LiteLLMLoggingBaseClass):
|
|||
if cache_hit is True:
|
||||
return 0.0
|
||||
|
||||
# If an orchestrator (e.g. advisor tool loop in /v1/messages) has
|
||||
# already aggregated cost into self.cost_breakdown before this path
|
||||
# runs again (typically on the streaming @client wrapper's
|
||||
# update_response_metadata pass with a FakeAnthropicMessagesStream
|
||||
# iterator), preserve the breakdown and return the aggregated total.
|
||||
# Falling through to litellm.response_cost_calculator would recompute
|
||||
# with zero usage (stream not yet consumed) and call
|
||||
# _store_cost_breakdown_in_logging_obj, wiping additional_costs.
|
||||
if (
|
||||
self.cost_breakdown is not None
|
||||
and self.cost_breakdown.get("total_cost") is not None
|
||||
and self.cost_breakdown["total_cost"] > 0
|
||||
):
|
||||
return self.cost_breakdown["total_cost"]
|
||||
|
||||
if isinstance(result, BaseModel) and hasattr(result, "_hidden_params"):
|
||||
hidden_params = getattr(result, "_hidden_params", {})
|
||||
if (
|
||||
|
|
|
|||
|
|
@ -146,29 +146,29 @@ class FakeAnthropicMessagesStreamIterator:
|
|||
)
|
||||
|
||||
elif block_type == "advisor_tool_result":
|
||||
advisor_content = block_dict.get("content") or {}
|
||||
advisor_text = ""
|
||||
if isinstance(advisor_content, dict):
|
||||
advisor_text = advisor_content.get("text", "") or ""
|
||||
elif isinstance(advisor_content, str):
|
||||
advisor_text = advisor_content
|
||||
|
||||
# Keep advisor result payload fully populated in content_block_start.
|
||||
# Anthropic tool_result-like blocks are treated as complete in start
|
||||
# events (no follow-up delta required), and Claude Code renders the
|
||||
# advisor panel from this payload.
|
||||
content_block_start = {
|
||||
"type": "content_block_start",
|
||||
"index": index,
|
||||
"content_block": {
|
||||
"type": "advisor_tool_result",
|
||||
"tool_use_id": block_dict.get("tool_use_id"),
|
||||
"content": {"type": "advisor_result", "text": ""},
|
||||
"content": {"type": "advisor_result", "text": advisor_text},
|
||||
},
|
||||
}
|
||||
chunks.append(
|
||||
f"event: content_block_start\ndata: {json.dumps(content_block_start)}\n\n".encode()
|
||||
)
|
||||
advisor_content = block_dict.get("content") or {}
|
||||
advisor_text = advisor_content.get("text", "") if isinstance(advisor_content, dict) else ""
|
||||
if advisor_text:
|
||||
content_block_delta = {
|
||||
"type": "content_block_delta",
|
||||
"index": index,
|
||||
"delta": {"type": "advisor_result_delta", "text": advisor_text},
|
||||
}
|
||||
chunks.append(
|
||||
f"event: content_block_delta\ndata: {json.dumps(content_block_delta)}\n\n".encode()
|
||||
)
|
||||
|
||||
content_block_stop = {"type": "content_block_stop", "index": index}
|
||||
chunks.append(
|
||||
|
|
|
|||
|
|
@ -16,18 +16,25 @@ How it works:
|
|||
|
||||
import asyncio
|
||||
import uuid
|
||||
from typing import Any, AsyncIterator, Dict, List, Optional, Union, cast
|
||||
from typing import Any, AsyncIterator, Dict, List, Optional, Union
|
||||
|
||||
import litellm
|
||||
import litellm.constants as _c
|
||||
from litellm._internal_context import is_internal_call
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.llms.anthropic.common_utils import strip_advisor_blocks_from_messages
|
||||
from litellm.llms.anthropic.experimental_pass_through.adapters.handler import (
|
||||
LiteLLMMessagesToCompletionTransformationHandler,
|
||||
)
|
||||
from litellm.types.llms.anthropic_messages.anthropic_response import (
|
||||
AnthropicMessagesResponse,
|
||||
AnthropicUsageIteration,
|
||||
)
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.llms.anthropic import ANTHROPIC_ADVISOR_TOOL_TYPE
|
||||
from litellm.utils import (
|
||||
resolve_proxy_model_alias_to_litellm_model,
|
||||
supports_native_advisor_tool,
|
||||
)
|
||||
|
||||
ADVISOR_MAX_USES: int = _c.ADVISOR_MAX_USES
|
||||
ADVISOR_TOOL_DESCRIPTION: str = _c.ADVISOR_TOOL_DESCRIPTION
|
||||
|
|
@ -49,16 +56,24 @@ class AdvisorOrchestrationHandler(MessagesInterceptor):
|
|||
) -> bool:
|
||||
if not tools:
|
||||
return False
|
||||
has_advisor = any(t.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE for t in tools)
|
||||
if not has_advisor:
|
||||
advisor_tools = [
|
||||
t for t in tools if t.get("type") == ANTHROPIC_ADVISOR_TOOL_TYPE
|
||||
]
|
||||
if not advisor_tools:
|
||||
return False
|
||||
# Direct Anthropic /messages: the API handles advisor_20260301 natively;
|
||||
# do not run the LiteLLM orchestration loop here.
|
||||
# Direct Anthropic /messages: the API handles advisor_20260301 natively
|
||||
# *only* when the tool's model resolves to a native Anthropic advisor
|
||||
# model. When an operator remaps the tool's model via model_group_alias
|
||||
# to a non-native model (e.g. claude-opus-4-7 -> o3) we must take over
|
||||
# the loop here so the sub-call is routed through litellm.
|
||||
if custom_llm_provider == "anthropic":
|
||||
for advisor_tool in advisor_tools:
|
||||
if not _advisor_tool_uses_native_anthropic_model(advisor_tool):
|
||||
return True
|
||||
return False
|
||||
return True
|
||||
|
||||
async def handle(
|
||||
async def handle( # noqa: PLR0915
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
|
|
@ -82,15 +97,24 @@ class AdvisorOrchestrationHandler(MessagesInterceptor):
|
|||
raise ValueError(
|
||||
f"handle() called but no {ANTHROPIC_ADVISOR_TOOL_TYPE} tool found in tools list"
|
||||
)
|
||||
advisor_model: str = advisor_tool.get("model") or ""
|
||||
if not advisor_model:
|
||||
advisor_model = _resolve_default_advisor_model()
|
||||
if not advisor_model:
|
||||
advisor_model_alias: str = advisor_tool.get("model") or ""
|
||||
if not advisor_model_alias:
|
||||
advisor_model_alias = _resolve_default_advisor_model()
|
||||
if not advisor_model_alias:
|
||||
raise ValueError(
|
||||
"No advisor model specified. Either:\n"
|
||||
" 1. Set 'default_advisor_model' in advisor_interception_params in your proxy config YAML, or\n"
|
||||
" 2. Include a 'model' field in the advisor tool definition."
|
||||
)
|
||||
# Resolve the tool's ``model`` (which may be a proxy model_group_alias
|
||||
# like ``claude-opus-4-7`` pointing at ``o3``) to the actual underlying
|
||||
# litellm model for the sub-call. Keep the alias separate so every
|
||||
# client-visible surface (iterations[].model) continues to show the
|
||||
# original name the caller sent — the remap is opaque to the caller.
|
||||
resolved_advisor_model: str = (
|
||||
resolve_proxy_model_alias_to_litellm_model(advisor_model_alias)
|
||||
or advisor_model_alias
|
||||
)
|
||||
_raw_max_uses = advisor_tool.get("max_uses")
|
||||
max_uses: int = (
|
||||
ADVISOR_MAX_USES if _raw_max_uses is None else int(_raw_max_uses)
|
||||
|
|
@ -129,10 +153,13 @@ class AdvisorOrchestrationHandler(MessagesInterceptor):
|
|||
# their inner @client wrappers (aresponses/acompletion) still populate
|
||||
# ``custom_llm_provider``, ``api_base``, ``model_id`` on this shared
|
||||
# model_call_details via ``update_environment_variables`` — fields the
|
||||
# outer anthropic_messages path never sets on its own. We mark the
|
||||
# sub-calls ``_is_litellm_internal_call=True`` so their @client skips
|
||||
# emitting a duplicate log row; only the outer call emits a single
|
||||
# aggregated entry.
|
||||
# outer anthropic_messages path never sets on its own.
|
||||
#
|
||||
# NOTE: ``_is_litellm_internal_call`` in kwargs is not sufficient to
|
||||
# suppress @client logging; wrapper_async checks the ContextVar
|
||||
# ``is_internal_call``. Keep the kwarg for compatibility, but also set
|
||||
# the ContextVar around the orchestration loop so nested sub-calls do
|
||||
# not emit separate proxy billing rows.
|
||||
litellm_logging_obj = kwargs.get("litellm_logging_obj", None)
|
||||
kwargs["_is_litellm_internal_call"] = True
|
||||
iteration = 0
|
||||
|
|
@ -141,110 +168,138 @@ class AdvisorOrchestrationHandler(MessagesInterceptor):
|
|||
advisor_first_call_cost: float = 0.0
|
||||
advisor_subcall_cost: float = 0.0
|
||||
|
||||
while True:
|
||||
# --- Executor call (always non-streaming) ---
|
||||
executor_response: AnthropicMessagesResponse = await _call_messages_handler(
|
||||
model=model,
|
||||
messages=current_messages,
|
||||
tools=executor_tools,
|
||||
stream=False,
|
||||
max_tokens=max_tokens,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
metadata={
|
||||
**metadata_base,
|
||||
"advisor_sub_call": False,
|
||||
"parent_request_id": parent_request_id,
|
||||
},
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
executor_cost = _get_response_cost(executor_response, model=model)
|
||||
iterations.append(
|
||||
_build_iteration_entry(
|
||||
response=executor_response, iteration_type="message"
|
||||
)
|
||||
)
|
||||
|
||||
advisor_use_block = _find_advisor_tool_use(executor_response)
|
||||
|
||||
if advisor_use_block is None:
|
||||
# No more advisor calls — this is the final response.
|
||||
# Inject advisor_tool_result blocks to match Anthropic native format.
|
||||
_inject_advisor_blocks_into_response(
|
||||
executor_response, advisor_interactions
|
||||
)
|
||||
total_cost = (
|
||||
advisor_first_call_cost + advisor_subcall_cost + executor_cost
|
||||
)
|
||||
_finalize_orchestrated_response(
|
||||
response=executor_response,
|
||||
iterations=iterations,
|
||||
total_cost=total_cost,
|
||||
final_executor_cost=executor_cost,
|
||||
advisor_first_call_cost=advisor_first_call_cost,
|
||||
advisor_subcall_cost=advisor_subcall_cost,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
)
|
||||
if stream:
|
||||
return FakeAnthropicMessagesStreamIterator(executor_response)
|
||||
return executor_response
|
||||
|
||||
# Executor response triggered another advisor call → count it as a
|
||||
# "first/intermediate" executor turn. Only the terminating turn is
|
||||
# treated as the base response.
|
||||
advisor_first_call_cost += executor_cost
|
||||
|
||||
iteration += 1
|
||||
if iteration > max_uses:
|
||||
raise AdvisorMaxIterationsError(
|
||||
f"Advisor orchestration loop exceeded max_uses={max_uses}. "
|
||||
"Increase max_uses in the advisor tool definition or cap the request."
|
||||
_prev_internal = is_internal_call.get()
|
||||
is_internal_call.set(True)
|
||||
try:
|
||||
while True:
|
||||
# --- Executor call (always non-streaming) ---
|
||||
executor_response: AnthropicMessagesResponse = await _call_messages_handler(
|
||||
model=model,
|
||||
messages=current_messages,
|
||||
tools=executor_tools,
|
||||
stream=False,
|
||||
max_tokens=max_tokens,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
metadata={
|
||||
**metadata_base,
|
||||
"advisor_sub_call": False,
|
||||
"parent_request_id": parent_request_id,
|
||||
},
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
# --- Build advisor context ---
|
||||
advisor_messages = _build_advisor_context(
|
||||
current_messages, executor_response, advisor_use_block
|
||||
)
|
||||
|
||||
# --- Advisor sub-call (always non-streaming, no tools) ---
|
||||
advisor_response: AnthropicMessagesResponse = await _call_advisor_with_router(
|
||||
model=advisor_model,
|
||||
messages=advisor_messages,
|
||||
max_tokens=max_tokens,
|
||||
metadata={
|
||||
**metadata_base,
|
||||
"advisor_sub_call": True,
|
||||
"parent_request_id": parent_request_id,
|
||||
},
|
||||
api_key=advisor_api_key,
|
||||
api_base=advisor_api_base,
|
||||
)
|
||||
|
||||
advisor_call_cost = _get_response_cost(advisor_response, model=advisor_model)
|
||||
advisor_subcall_cost += advisor_call_cost
|
||||
iterations.append(
|
||||
_build_iteration_entry(
|
||||
response=advisor_response,
|
||||
iteration_type="advisor_message",
|
||||
model=advisor_model,
|
||||
executor_cost = _get_response_cost(executor_response, model=model)
|
||||
iterations.append(
|
||||
_build_iteration_entry(
|
||||
response=executor_response, iteration_type="message"
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
advisor_text = _extract_response_text(advisor_response)
|
||||
advisor_use_block = _find_advisor_tool_use(executor_response)
|
||||
|
||||
# Record the interaction for later injection into the final response.
|
||||
advisor_interactions.append({
|
||||
"tool_use_id": advisor_use_block.get("id", f"srvtoolu_{uuid.uuid4().hex[:24]}"),
|
||||
"advisor_text": advisor_text,
|
||||
})
|
||||
if advisor_use_block is None:
|
||||
# No more advisor calls — this is the final response.
|
||||
# Inject advisor_tool_result blocks to match Anthropic native format.
|
||||
_inject_advisor_blocks_into_response(
|
||||
executor_response, advisor_interactions
|
||||
)
|
||||
total_cost = (
|
||||
advisor_first_call_cost + advisor_subcall_cost + executor_cost
|
||||
)
|
||||
_finalize_orchestrated_response(
|
||||
response=executor_response,
|
||||
iterations=iterations,
|
||||
total_cost=total_cost,
|
||||
final_executor_cost=executor_cost,
|
||||
advisor_first_call_cost=advisor_first_call_cost,
|
||||
advisor_subcall_cost=advisor_subcall_cost,
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
)
|
||||
if stream:
|
||||
# The outer ``@client`` async wrapper skips
|
||||
# ``_client_async_logging_helper`` for streaming
|
||||
# requests — it assumes a ``CustomStreamWrapper`` will
|
||||
# fire logging on iteration. ``FakeAnthropicMessagesStreamIterator``
|
||||
# is a plain iterator over pre-built SSE bytes and
|
||||
# does not know about the logging obj, so nothing fires
|
||||
# the proxy log row. We've already aggregated the full
|
||||
# response into ``executor_response`` (same dict shape
|
||||
# the non-streaming path uses for logging), so fire
|
||||
# the success handler ourselves with that dict before
|
||||
# wrapping — this mirrors the non-streaming flow and
|
||||
# avoids double-logging (the @client path is skipped).
|
||||
_fire_async_success_logging(
|
||||
litellm_logging_obj=litellm_logging_obj,
|
||||
result=executor_response,
|
||||
)
|
||||
return FakeAnthropicMessagesStreamIterator(executor_response)
|
||||
return executor_response
|
||||
|
||||
# --- Inject advisor result and continue loop ---
|
||||
current_messages = _inject_advisor_turn(
|
||||
current_messages,
|
||||
executor_response,
|
||||
advisor_use_block,
|
||||
advisor_text,
|
||||
)
|
||||
# Executor response triggered another advisor call → count it as a
|
||||
# "first/intermediate" executor turn. Only the terminating turn is
|
||||
# treated as the base response.
|
||||
advisor_first_call_cost += executor_cost
|
||||
|
||||
iteration += 1
|
||||
if iteration > max_uses:
|
||||
raise AdvisorMaxIterationsError(
|
||||
f"Advisor orchestration loop exceeded max_uses={max_uses}. "
|
||||
"Increase max_uses in the advisor tool definition or cap the request."
|
||||
)
|
||||
|
||||
# --- Build advisor context ---
|
||||
advisor_messages = _build_advisor_context(
|
||||
current_messages, executor_response, advisor_use_block
|
||||
)
|
||||
|
||||
# --- Advisor sub-call (always non-streaming, no tools) ---
|
||||
# Use the resolved model so router routing / cost lookup hit the
|
||||
# real underlying deployment; the alias is kept only for the
|
||||
# client-visible iteration entry below.
|
||||
advisor_response: AnthropicMessagesResponse = await _call_advisor_with_router(
|
||||
model=resolved_advisor_model,
|
||||
messages=advisor_messages,
|
||||
max_tokens=max_tokens,
|
||||
metadata={
|
||||
**metadata_base,
|
||||
"advisor_sub_call": True,
|
||||
"parent_request_id": parent_request_id,
|
||||
},
|
||||
api_key=advisor_api_key,
|
||||
api_base=advisor_api_base,
|
||||
)
|
||||
|
||||
advisor_call_cost = _get_response_cost(
|
||||
advisor_response, model=resolved_advisor_model
|
||||
)
|
||||
advisor_subcall_cost += advisor_call_cost
|
||||
iterations.append(
|
||||
_build_iteration_entry(
|
||||
response=advisor_response,
|
||||
iteration_type="advisor_message",
|
||||
model=advisor_model_alias,
|
||||
)
|
||||
)
|
||||
|
||||
advisor_text = _extract_response_text(advisor_response)
|
||||
|
||||
# Record the interaction for later injection into the final response.
|
||||
advisor_interactions.append({
|
||||
"tool_use_id": advisor_use_block.get(
|
||||
"id", f"srvtoolu_{uuid.uuid4().hex[:24]}"
|
||||
),
|
||||
"advisor_text": advisor_text,
|
||||
})
|
||||
|
||||
# --- Inject advisor result and continue loop ---
|
||||
current_messages = _inject_advisor_turn(
|
||||
current_messages,
|
||||
executor_response,
|
||||
advisor_use_block,
|
||||
advisor_text,
|
||||
)
|
||||
finally:
|
||||
is_internal_call.set(_prev_internal)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -260,6 +315,31 @@ def _resolve_default_advisor_model() -> str:
|
|||
return params.get("default_advisor_model", "") or ""
|
||||
|
||||
|
||||
def _advisor_tool_uses_native_anthropic_model(advisor_tool: Dict) -> bool:
|
||||
"""
|
||||
Return True iff the advisor tool's ``model`` (after proxy alias resolution)
|
||||
is a native Anthropic advisor model.
|
||||
|
||||
Used by :class:`AdvisorOrchestrationHandler.can_handle` to decide whether
|
||||
to let Anthropic's server-side advisor handle the tool or to intercept it
|
||||
and route the sub-call through LiteLLM.
|
||||
"""
|
||||
advisor_model = advisor_tool.get("model") or _resolve_default_advisor_model()
|
||||
if not advisor_model:
|
||||
# No model specified — let native Anthropic handle it (or fail there
|
||||
# with its own error). This path should not be hit in practice because
|
||||
# advisor_interception_params enforces a default upstream.
|
||||
return True
|
||||
resolved_model = (
|
||||
resolve_proxy_model_alias_to_litellm_model(advisor_model) or advisor_model
|
||||
)
|
||||
if resolved_model.startswith("anthropic/"):
|
||||
resolved_model = resolved_model.split("/", 1)[1]
|
||||
return supports_native_advisor_tool(
|
||||
model=resolved_model, custom_llm_provider="anthropic"
|
||||
)
|
||||
|
||||
|
||||
_SYNTHETIC_ADVISOR_TOOL_NAME = "consult_advisor"
|
||||
|
||||
|
||||
|
|
@ -445,6 +525,55 @@ def _finalize_orchestrated_response(
|
|||
)
|
||||
|
||||
|
||||
def _fire_async_success_logging(
|
||||
litellm_logging_obj: Any,
|
||||
result: Any,
|
||||
) -> None:
|
||||
"""
|
||||
Manually enqueue the ``async_success_handler`` for a streaming advisor
|
||||
response.
|
||||
|
||||
The outer ``@client`` async wrapper only calls
|
||||
``_client_async_logging_helper`` for non-streaming results; streaming
|
||||
results are expected to log from inside a ``CustomStreamWrapper``. Our
|
||||
synthetic :class:`FakeAnthropicMessagesStreamIterator` has no logging
|
||||
hook, so without this helper the proxy UI would never get a row for
|
||||
streaming advisor calls. We already built the aggregated response dict
|
||||
(same shape the non-streaming path logs from), so we can fire logging
|
||||
exactly once here with the same arguments the non-streaming path uses.
|
||||
"""
|
||||
if litellm_logging_obj is None:
|
||||
return
|
||||
try:
|
||||
import datetime as _dt
|
||||
|
||||
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
|
||||
|
||||
start_time = getattr(litellm_logging_obj, "start_time", None) or _dt.datetime.now()
|
||||
end_time = _dt.datetime.now()
|
||||
GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue(
|
||||
async_coroutine=litellm_logging_obj.async_success_handler(
|
||||
result=result, start_time=start_time, end_time=end_time
|
||||
)
|
||||
)
|
||||
try:
|
||||
litellm_logging_obj.handle_sync_success_callbacks_for_async_calls(
|
||||
result=result,
|
||||
start_time=start_time,
|
||||
end_time=end_time,
|
||||
)
|
||||
except Exception as sync_cb_error:
|
||||
verbose_logger.debug(
|
||||
"AdvisorOrchestration: sync success callbacks failed: %s",
|
||||
str(sync_cb_error),
|
||||
)
|
||||
except Exception as logging_error:
|
||||
verbose_logger.debug(
|
||||
"AdvisorOrchestration: failed to fire async success logging: %s",
|
||||
str(logging_error),
|
||||
)
|
||||
|
||||
|
||||
def _find_advisor_tool_use(response: Any) -> Optional[Dict]:
|
||||
"""Return the first tool_use block whose name matches our synthetic advisor."""
|
||||
content = response.get("content") if isinstance(response, dict) else []
|
||||
|
|
@ -609,6 +738,12 @@ def _build_advisor_context(
|
|||
|
||||
tool_use blocks are excluded because Anthropic requires tool_use to be
|
||||
immediately followed by tool_result — not the advisor question.
|
||||
|
||||
Messages stay in Anthropic ``/v1/messages`` shape here; ``_call_advisor_with_router``
|
||||
runs the same ``LiteLLMMessagesToCompletionTransformationHandler`` path used when
|
||||
a client calls the messages endpoint with a non-Anthropic model, so provider
|
||||
translation (including interleaved ``thinking`` blocks) matches the rest of
|
||||
the stack.
|
||||
"""
|
||||
question = (advisor_use_block.get("input") or {}).get("question") or (
|
||||
"Please provide guidance on the current task."
|
||||
|
|
@ -774,18 +909,26 @@ async def _call_advisor_with_router(
|
|||
if api_base is not None:
|
||||
kwargs["api_base"] = api_base
|
||||
|
||||
openai_messages: List[AllMessageValues] = cast(List[AllMessageValues], messages)
|
||||
# Same translation path as ``/v1/messages`` → non-Anthropic model: Anthropic
|
||||
# request shape → Chat Completions kwargs for the target provider.
|
||||
(
|
||||
completion_kwargs,
|
||||
_tool_name_mapping,
|
||||
) = LiteLLMMessagesToCompletionTransformationHandler._prepare_completion_kwargs(
|
||||
max_tokens=max_tokens,
|
||||
messages=messages,
|
||||
model=model,
|
||||
metadata=metadata,
|
||||
stream=False,
|
||||
extra_kwargs=kwargs,
|
||||
)
|
||||
# Inner advisor call is always a plain completion (no tools).
|
||||
completion_kwargs["tools"] = None
|
||||
|
||||
openai_response = None
|
||||
if llm_router is not None:
|
||||
try:
|
||||
openai_response = await llm_router.acompletion(
|
||||
model=model,
|
||||
messages=openai_messages,
|
||||
tools=None,
|
||||
max_tokens=max_tokens,
|
||||
**kwargs,
|
||||
)
|
||||
openai_response = await llm_router.acompletion(**completion_kwargs)
|
||||
except Exception:
|
||||
verbose_logger.debug(
|
||||
"AdvisorOrchestration: Router call for advisor model '%s' failed, "
|
||||
|
|
@ -794,12 +937,6 @@ async def _call_advisor_with_router(
|
|||
)
|
||||
|
||||
if openai_response is None:
|
||||
openai_response = await _litellm.acompletion(
|
||||
model=model,
|
||||
messages=openai_messages,
|
||||
tools=None,
|
||||
max_tokens=max_tokens,
|
||||
**kwargs,
|
||||
)
|
||||
openai_response = await _litellm.acompletion(**completion_kwargs)
|
||||
|
||||
return _openai_response_to_anthropic_dict(openai_response)
|
||||
|
|
|
|||
|
|
@ -17,7 +17,10 @@ from litellm.types.llms.anthropic_messages.anthropic_response import (
|
|||
)
|
||||
from litellm.types.llms.anthropic_tool_search import get_tool_search_beta_header
|
||||
from litellm.types.router import GenericLiteLLMParams
|
||||
from litellm.utils import resolve_proxy_model_alias_to_litellm_model
|
||||
from litellm.utils import (
|
||||
resolve_proxy_model_alias_to_litellm_model,
|
||||
supports_native_advisor_tool,
|
||||
)
|
||||
|
||||
from ...common_utils import (
|
||||
AnthropicError,
|
||||
|
|
@ -36,6 +39,14 @@ def _normalize_anthropic_advisor_tool_models(tools: List[Dict]) -> List[Dict]:
|
|||
Anthropic expects advisor tool model values like ``claude-opus-4-6``.
|
||||
Proxy alias names (e.g. ``claude_opus``) and provider-prefixed values
|
||||
(e.g. ``anthropic/claude-opus-4-6``) are converted.
|
||||
|
||||
Defensive guard: if the alias resolves to a model Anthropic's native
|
||||
advisor tool does not support (e.g. ``claude-opus-4-7`` -> ``o3`` via
|
||||
``model_group_alias``), leave the original alias string in place rather
|
||||
than forwarding the unsupported model to Anthropic. In that case
|
||||
``AdvisorOrchestrationHandler`` is responsible for intercepting the
|
||||
request and running the loop through litellm; this branch only runs if
|
||||
the interceptor was somehow bypassed.
|
||||
"""
|
||||
normalized_tools: List[Dict] = []
|
||||
for tool in tools:
|
||||
|
|
@ -49,11 +60,21 @@ def _normalize_anthropic_advisor_tool_models(tools: List[Dict]) -> List[Dict]:
|
|||
updated_tool = dict(tool)
|
||||
advisor_model = updated_tool.get("model")
|
||||
if isinstance(advisor_model, str) and advisor_model.strip():
|
||||
resolved = resolve_proxy_model_alias_to_litellm_model(advisor_model.strip())
|
||||
canonical_model = resolved or advisor_model.strip()
|
||||
original_model = advisor_model.strip()
|
||||
resolved = resolve_proxy_model_alias_to_litellm_model(original_model)
|
||||
canonical_model = resolved or original_model
|
||||
if canonical_model.startswith("anthropic/"):
|
||||
canonical_model = canonical_model.split("/", 1)[1]
|
||||
updated_tool["model"] = canonical_model
|
||||
# Only substitute the resolved/canonical value if Anthropic
|
||||
# natively supports it as an advisor model. Otherwise keep the
|
||||
# caller's original alias so we never leak a non-Anthropic model
|
||||
# name (e.g. ``o3``) into the Anthropic request body.
|
||||
if supports_native_advisor_tool(
|
||||
model=canonical_model, custom_llm_provider="anthropic"
|
||||
):
|
||||
updated_tool["model"] = canonical_model
|
||||
else:
|
||||
updated_tool["model"] = original_model
|
||||
normalized_tools.append(updated_tool)
|
||||
return normalized_tools
|
||||
|
||||
|
|
|
|||
|
|
@ -9094,6 +9094,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
@ -9126,6 +9127,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
@ -9158,6 +9160,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
@ -9190,6 +9193,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
|
|||
|
|
@ -1171,6 +1171,37 @@ class ProxyBaseLLMRequestProcessing:
|
|||
|
||||
logging_obj._on_deferred_stream_complete = _on_deferred_stream_complete # type: ignore[union-attr]
|
||||
|
||||
# Fallback for streaming responses that are NOT CustomStreamWrapper
|
||||
# (e.g. FakeAnthropicMessagesStreamIterator used by advisor
|
||||
# orchestration). These responses bypass CSW's internal deferred
|
||||
# callback wiring, so without this fallback the outer proxy log row
|
||||
# is never emitted.
|
||||
if (
|
||||
self._is_streaming_response(response)
|
||||
and not isinstance(response, CustomStreamWrapper)
|
||||
and getattr(logging_obj, "_on_deferred_stream_complete", None)
|
||||
is None
|
||||
):
|
||||
_captured_data = self.data
|
||||
_captured_user_api_key_dict = user_api_key_dict
|
||||
_captured_logging_obj = logging_obj
|
||||
|
||||
async def _on_deferred_stream_complete_non_csw(
|
||||
assembled_response, cache_hit
|
||||
):
|
||||
await ProxyBaseLLMRequestProcessing._run_deferred_stream_guardrails(
|
||||
captured_data=_captured_data,
|
||||
captured_user_api_key_dict=_captured_user_api_key_dict,
|
||||
captured_logging_obj=_captured_logging_obj,
|
||||
assembled_response=assembled_response,
|
||||
cache_hit=cache_hit,
|
||||
)
|
||||
|
||||
logging_obj._on_deferred_stream_complete = _on_deferred_stream_complete_non_csw # type: ignore[union-attr]
|
||||
# _fire_deferred_stream_logging only fires when args are present.
|
||||
# Non-CSW iterators never set these, so seed default args here.
|
||||
logging_obj._deferred_stream_complete_args = (None, None) # type: ignore[union-attr]
|
||||
|
||||
if route_type == "allm_passthrough_route":
|
||||
# Check if response is an async generator
|
||||
if self._is_streaming_response(response):
|
||||
|
|
|
|||
|
|
@ -9158,6 +9158,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
@ -9190,6 +9191,7 @@
|
|||
"supports_assistant_prefill": false,
|
||||
"supports_computer_use": true,
|
||||
"supports_function_calling": true,
|
||||
"supports_native_advisor_tool": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,413 @@
|
|||
"""
|
||||
Tests for advisor tool model aliasing on the /v1/messages path.
|
||||
|
||||
Scenario: an operator remaps the advisor tool's model via
|
||||
``model_group_alias`` (e.g. ``claude-opus-4-7 -> o3``) because the client
|
||||
(Claude Code) hardcodes the advisor ``model`` field to ``claude-opus-4-7``.
|
||||
|
||||
Required behaviour:
|
||||
* ``AdvisorOrchestrationHandler.can_handle`` intercepts the request when
|
||||
the executor is direct Anthropic but the advisor tool resolves to a
|
||||
non-native advisor model.
|
||||
* ``handle()`` dispatches the advisor sub-call with the *resolved* model,
|
||||
but every client-visible surface (``iterations[].model``) keeps the
|
||||
original alias so the remap is opaque to the caller.
|
||||
* ``_normalize_anthropic_advisor_tool_models`` never forwards a
|
||||
non-Anthropic model to the native API — it leaves the alias untouched
|
||||
if the resolved model is not natively supported.
|
||||
"""
|
||||
|
||||
from typing import Dict
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
ADVISOR_TOOL_ALIAS = {
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-opus-4-7",
|
||||
}
|
||||
|
||||
MESSAGES = [
|
||||
{"role": "user", "content": "Write a Python function that checks if a number is prime."}
|
||||
]
|
||||
|
||||
|
||||
def _make_text_response(text: str, model: str = "openai/o3") -> Dict:
|
||||
return {
|
||||
"id": "msg_test",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": model,
|
||||
"content": [{"type": "text", "text": text}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 20},
|
||||
}
|
||||
|
||||
|
||||
def _make_advisor_tool_use_response(
|
||||
question: str = "How should I approach this?",
|
||||
tool_id: str = "toolu_advisor_01",
|
||||
model: str = "claude-opus-4-7",
|
||||
) -> Dict:
|
||||
return {
|
||||
"id": "msg_test",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": model,
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": tool_id,
|
||||
"name": "consult_advisor",
|
||||
"input": {"question": question},
|
||||
}
|
||||
],
|
||||
"stop_reason": "tool_use",
|
||||
"usage": {"input_tokens": 10, "output_tokens": 15},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. can_handle: alias -> non-native forces interception even on anthropic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_can_handle_alias_to_non_native_intercepts_on_anthropic():
|
||||
"""
|
||||
When the advisor tool's model (``claude-opus-4-7``) aliases to a
|
||||
non-Anthropic model (``o3``), the handler must intercept even though the
|
||||
executor provider is direct Anthropic — Anthropic's native advisor tool
|
||||
can't run ``o3`` for us.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="openai/o3",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.supports_native_advisor_tool",
|
||||
return_value=False,
|
||||
):
|
||||
h = AdvisorOrchestrationHandler()
|
||||
assert h.can_handle([ADVISOR_TOOL_ALIAS], "anthropic") is True
|
||||
|
||||
|
||||
def test_can_handle_alias_to_native_still_defers_to_anthropic():
|
||||
"""
|
||||
When the advisor tool's model aliases to a still-native Anthropic model
|
||||
(e.g. someone maps ``claude-opus-4-7 -> claude-opus-4-6``), the native
|
||||
Anthropic server-side advisor can still handle it — we must not
|
||||
intercept.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="anthropic/claude-opus-4-6",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.supports_native_advisor_tool",
|
||||
return_value=True,
|
||||
):
|
||||
h = AdvisorOrchestrationHandler()
|
||||
assert h.can_handle([ADVISOR_TOOL_ALIAS], "anthropic") is False
|
||||
|
||||
|
||||
def test_can_handle_non_anthropic_executor_always_intercepts():
|
||||
"""
|
||||
Non-Anthropic executors always need orchestration regardless of the
|
||||
advisor tool's resolved model — no behaviour change from before.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
h = AdvisorOrchestrationHandler()
|
||||
assert h.can_handle([ADVISOR_TOOL_ALIAS], "openai") is True
|
||||
assert h.can_handle([ADVISOR_TOOL_ALIAS], "bedrock") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. handle(): alias vs resolved separation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_uses_resolved_model_for_subcall_and_alias_for_iterations():
|
||||
"""
|
||||
The advisor sub-call receives the *resolved* model (``openai/o3``) so
|
||||
routing and cost lookup hit the real deployment, while every
|
||||
client-visible ``iterations[].model`` entry of type ``advisor_message``
|
||||
keeps the original alias (``claude-opus-4-7``). The alias must never
|
||||
leak to the sub-call, and the resolved name must never leak to the
|
||||
response.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
advisor_tool_use_resp = _make_advisor_tool_use_response(
|
||||
question="What algorithm should I use?",
|
||||
tool_id="toolu_01",
|
||||
)
|
||||
advisor_advice_resp = _make_text_response(
|
||||
"Use a sieve for large n, trial division for small n.",
|
||||
model="openai/o3",
|
||||
)
|
||||
final_resp = _make_text_response(
|
||||
"def is_prime(n): ...",
|
||||
model="claude-opus-4-7",
|
||||
)
|
||||
|
||||
executor_call_count = 0
|
||||
|
||||
async def mock_messages(model, messages, tools, stream, max_tokens, **kwargs):
|
||||
nonlocal executor_call_count
|
||||
executor_call_count += 1
|
||||
if executor_call_count == 1:
|
||||
return advisor_tool_use_resp
|
||||
return final_resp
|
||||
|
||||
advisor_mock = AsyncMock(return_value=advisor_advice_resp)
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_messages_handler",
|
||||
side_effect=mock_messages,
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_advisor_with_router",
|
||||
advisor_mock,
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="openai/o3",
|
||||
):
|
||||
h = AdvisorOrchestrationHandler()
|
||||
result = await h.handle(
|
||||
model="claude-opus-4-6",
|
||||
messages=MESSAGES,
|
||||
tools=[ADVISOR_TOOL_ALIAS],
|
||||
stream=False,
|
||||
max_tokens=512,
|
||||
custom_llm_provider="anthropic",
|
||||
)
|
||||
|
||||
assert advisor_mock.await_count == 1
|
||||
advisor_call_kwargs = advisor_mock.await_args.kwargs
|
||||
assert advisor_call_kwargs["model"] == "openai/o3", (
|
||||
"Advisor sub-call must use the resolved router model, not the alias"
|
||||
)
|
||||
|
||||
usage = result.get("usage", {})
|
||||
iterations = usage.get("iterations", [])
|
||||
advisor_iterations = [
|
||||
it for it in iterations if it.get("type") == "advisor_message"
|
||||
]
|
||||
assert len(advisor_iterations) == 1
|
||||
assert advisor_iterations[0]["model"] == "claude-opus-4-7", (
|
||||
"iterations[].model must preserve the client-facing alias"
|
||||
)
|
||||
# Resolved model must never appear anywhere in the iterations surface.
|
||||
for it in iterations:
|
||||
assert it.get("model") != "openai/o3"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_without_alias_still_works():
|
||||
"""
|
||||
When ``resolve_proxy_model_alias_to_litellm_model`` returns ``""`` (no
|
||||
alias configured), ``handle()`` must fall back to using the tool's
|
||||
original model for both the sub-call and the iteration entry. Nothing
|
||||
regresses for users who don't configure an alias.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
advisor_tool = {
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "openai/gpt-4o-mini",
|
||||
}
|
||||
|
||||
advisor_tool_use_resp = _make_advisor_tool_use_response()
|
||||
advisor_advice_resp = _make_text_response("advice", model="openai/gpt-4o-mini")
|
||||
final_resp = _make_text_response("final")
|
||||
|
||||
executor_call_count = 0
|
||||
|
||||
async def mock_messages(model, messages, tools, stream, max_tokens, **kwargs):
|
||||
nonlocal executor_call_count
|
||||
executor_call_count += 1
|
||||
return advisor_tool_use_resp if executor_call_count == 1 else final_resp
|
||||
|
||||
advisor_mock = AsyncMock(return_value=advisor_advice_resp)
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_messages_handler",
|
||||
side_effect=mock_messages,
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_advisor_with_router",
|
||||
advisor_mock,
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="",
|
||||
):
|
||||
h = AdvisorOrchestrationHandler()
|
||||
result = await h.handle(
|
||||
model="openai/gpt-4o-mini",
|
||||
messages=MESSAGES,
|
||||
tools=[advisor_tool],
|
||||
stream=False,
|
||||
max_tokens=512,
|
||||
custom_llm_provider="openai",
|
||||
)
|
||||
|
||||
assert advisor_mock.await_args.kwargs["model"] == "openai/gpt-4o-mini"
|
||||
advisor_iterations = [
|
||||
it
|
||||
for it in result["usage"]["iterations"]
|
||||
if it.get("type") == "advisor_message"
|
||||
]
|
||||
assert advisor_iterations[0]["model"] == "openai/gpt-4o-mini"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. _normalize_anthropic_advisor_tool_models defensive guard
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_normalize_leaves_alias_when_resolved_model_is_non_native():
|
||||
"""
|
||||
Defensive guard: if the alias resolves to a non-Anthropic advisor model,
|
||||
the normalizer must leave the caller's original alias in place rather
|
||||
than substituting the unsupported model into the Anthropic request body.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
|
||||
_normalize_anthropic_advisor_tool_models,
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude-opus-4-7",
|
||||
}
|
||||
]
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.transformation.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="openai/o3",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.transformation.supports_native_advisor_tool",
|
||||
return_value=False,
|
||||
):
|
||||
normalized = _normalize_anthropic_advisor_tool_models(tools)
|
||||
|
||||
assert normalized[0]["model"] == "claude-opus-4-7", (
|
||||
"Normalizer must not push the non-native resolved model to Anthropic"
|
||||
)
|
||||
|
||||
|
||||
def test_normalize_strips_anthropic_prefix_when_resolved_model_is_native():
|
||||
"""
|
||||
Regression: for the classic path (alias resolves to a native Anthropic
|
||||
model), the normalizer still strips the ``anthropic/`` prefix so the
|
||||
Anthropic API receives a bare model name.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
|
||||
_normalize_anthropic_advisor_tool_models,
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "advisor_20260301",
|
||||
"name": "advisor",
|
||||
"model": "claude_opus",
|
||||
}
|
||||
]
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.transformation.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="anthropic/claude-opus-4-6",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.transformation.supports_native_advisor_tool",
|
||||
return_value=True,
|
||||
):
|
||||
normalized = _normalize_anthropic_advisor_tool_models(tools)
|
||||
|
||||
assert normalized[0]["model"] == "claude-opus-4-6"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. Advisor sub-call uses the same /v1/messages → completion translation path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prepare_completion_kwargs_moves_thinking_out_of_content():
|
||||
"""
|
||||
Advisor sub-calls must use ``LiteLLMMessagesToCompletionTransformationHandler``
|
||||
(same as non-Anthropic ``/v1/messages``), so interleaved ``thinking`` blocks
|
||||
become OpenAI-shaped messages — never raw ``content[].type == "thinking"``.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.adapters.handler import (
|
||||
LiteLLMMessagesToCompletionTransformationHandler,
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": [{"type": "text", "text": "hi"}]},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "thinking", "thinking": "secret reasoning", "signature": "s"},
|
||||
{"type": "redacted_thinking", "data": "redacted"},
|
||||
{"type": "text", "text": "hello"},
|
||||
],
|
||||
},
|
||||
]
|
||||
completion_kwargs, _ = (
|
||||
LiteLLMMessagesToCompletionTransformationHandler._prepare_completion_kwargs(
|
||||
max_tokens=100,
|
||||
messages=messages,
|
||||
model="openai/gpt-5-nano",
|
||||
stream=False,
|
||||
)
|
||||
)
|
||||
for msg in completion_kwargs["messages"]:
|
||||
content = msg.get("content")
|
||||
if isinstance(content, list):
|
||||
for part in content:
|
||||
if isinstance(part, dict):
|
||||
assert part.get("type") != "thinking"
|
||||
assert part.get("type") != "redacted_thinking"
|
||||
|
||||
|
||||
def test_build_advisor_context_preserves_string_content_and_plain_messages():
|
||||
"""
|
||||
Messages with plain string content or only supported block types must be
|
||||
passed through unchanged.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
_build_advisor_context,
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "plain string user"},
|
||||
{"role": "assistant", "content": [{"type": "text", "text": "plain reply"}]},
|
||||
]
|
||||
executor_response = {"content": [{"type": "text", "text": "draft"}]}
|
||||
advisor_use_block = {
|
||||
"type": "tool_use",
|
||||
"name": "advisor",
|
||||
"input": {"question": "advise"},
|
||||
}
|
||||
|
||||
result = _build_advisor_context(messages, executor_response, advisor_use_block)
|
||||
|
||||
assert result[0] == {"role": "user", "content": "plain string user"}
|
||||
assert result[1] == {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "plain reply"}],
|
||||
}
|
||||
|
|
@ -74,16 +74,24 @@ def test_can_handle_edge_cases():
|
|||
)
|
||||
|
||||
h = AdvisorOrchestrationHandler()
|
||||
|
||||
assert h.can_handle([ADVISOR_TOOL], "openai")
|
||||
assert h.can_handle([ADVISOR_TOOL], "bedrock")
|
||||
assert h.can_handle([ADVISOR_TOOL], "gemini")
|
||||
assert not h.can_handle([ADVISOR_TOOL], "anthropic")
|
||||
assert not h.can_handle([], "openai")
|
||||
assert not h.can_handle(None, "openai")
|
||||
assert not h.can_handle([{"type": "function", "name": "bash"}], "openai")
|
||||
# provider=None: unknown → should intercept (treat as non-native)
|
||||
assert h.can_handle([ADVISOR_TOOL], None)
|
||||
# Ensure this edge-case test is deterministic regardless of any proxy-level
|
||||
# model_group_alias configured by other tests.
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.supports_native_advisor_tool",
|
||||
return_value=True,
|
||||
):
|
||||
assert h.can_handle([ADVISOR_TOOL], "openai")
|
||||
assert h.can_handle([ADVISOR_TOOL], "bedrock")
|
||||
assert h.can_handle([ADVISOR_TOOL], "gemini")
|
||||
assert not h.can_handle([ADVISOR_TOOL], "anthropic")
|
||||
assert not h.can_handle([], "openai")
|
||||
assert not h.can_handle(None, "openai")
|
||||
assert not h.can_handle([{"type": "function", "name": "bash"}], "openai")
|
||||
# provider=None: unknown → should intercept (treat as non-native)
|
||||
assert h.can_handle([ADVISOR_TOOL], None)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -102,9 +110,16 @@ async def test_anthropic_native_interceptor_skipped():
|
|||
)
|
||||
|
||||
h = AdvisorOrchestrationHandler()
|
||||
assert not h.can_handle(
|
||||
[ADVISOR_TOOL], "anthropic"
|
||||
), "Interceptor must NOT trigger for anthropic provider"
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.resolve_proxy_model_alias_to_litellm_model",
|
||||
return_value="",
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor.supports_native_advisor_tool",
|
||||
return_value=True,
|
||||
):
|
||||
assert not h.can_handle(
|
||||
[ADVISOR_TOOL], "anthropic"
|
||||
), "Interceptor must NOT trigger for anthropic provider"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -300,6 +315,61 @@ async def test_loop_streaming_wraps_response():
|
|||
assert "message_start" in first
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_loop_streaming_advisor_block_start_contains_text():
|
||||
"""
|
||||
Regression: when advisor orchestration is streamed via FakeAnthropicMessagesStreamIterator,
|
||||
``advisor_tool_result`` must carry the full advisor text in content_block_start.
|
||||
Claude Code renders the advisor panel from that payload.
|
||||
"""
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor import (
|
||||
AdvisorOrchestrationHandler,
|
||||
)
|
||||
|
||||
executor_first = _make_advisor_tool_use_response(
|
||||
question="Please confirm integration status.", tool_id="toolu_advisor_123"
|
||||
)
|
||||
advisor_response = _make_text_response("Integration test: working correctly.")
|
||||
executor_final = _make_text_response("All set.")
|
||||
|
||||
with patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_messages_handler",
|
||||
new_callable=AsyncMock,
|
||||
side_effect=[executor_first, executor_final],
|
||||
), patch(
|
||||
"litellm.llms.anthropic.experimental_pass_through.messages.interceptors.advisor._call_advisor_with_router",
|
||||
new_callable=AsyncMock,
|
||||
return_value=advisor_response,
|
||||
):
|
||||
h = AdvisorOrchestrationHandler()
|
||||
stream_iter = await h.handle(
|
||||
model="openai/gpt-4o-mini",
|
||||
messages=MESSAGES,
|
||||
tools=[ADVISOR_TOOL],
|
||||
stream=True,
|
||||
max_tokens=512,
|
||||
custom_llm_provider="openai",
|
||||
)
|
||||
|
||||
chunks = []
|
||||
async for chunk in stream_iter:
|
||||
chunks.append(chunk.decode() if isinstance(chunk, bytes) else str(chunk))
|
||||
|
||||
advisor_start_events = [
|
||||
c
|
||||
for c in chunks
|
||||
if '"type": "content_block_start"' in c
|
||||
and '"type": "advisor_tool_result"' in c
|
||||
]
|
||||
assert advisor_start_events, "Expected advisor_tool_result content_block_start event"
|
||||
assert (
|
||||
'"text": "Integration test: working correctly."' in advisor_start_events[0]
|
||||
)
|
||||
|
||||
# advisor_tool_result should be complete in content_block_start (no extra delta needed)
|
||||
assert not any('"type": "advisor_result_delta"' in c for c in chunks)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. Multi-turn: prior advisor blocks replaced with text in history
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -0,0 +1,31 @@
|
|||
import litellm
|
||||
from litellm.router_utils.provider_account_fallback_errors import (
|
||||
is_provider_account_fallback_eligible_error,
|
||||
)
|
||||
|
||||
|
||||
def test_credit_balance_message_detected():
|
||||
err = litellm.BadRequestError(
|
||||
message="AnthropicException - Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.",
|
||||
model="claude-opus-4-6",
|
||||
llm_provider="anthropic",
|
||||
)
|
||||
assert is_provider_account_fallback_eligible_error(err) is True
|
||||
|
||||
|
||||
def test_openai_billing_hard_limit_detected():
|
||||
err = litellm.BadRequestError(
|
||||
message='{"error":{"code":"billing_hard_limit_reached","message":"Billing"}}',
|
||||
model="gpt-4",
|
||||
llm_provider="openai",
|
||||
)
|
||||
assert is_provider_account_fallback_eligible_error(err) is True
|
||||
|
||||
|
||||
def test_generic_validation_400_not_detected():
|
||||
err = litellm.BadRequestError(
|
||||
message="invalid maxOutputTokens",
|
||||
model="gemini-pro",
|
||||
llm_provider="vertex_ai",
|
||||
)
|
||||
assert is_provider_account_fallback_eligible_error(err) is False
|
||||
Loading…
Add table
Reference in a new issue