scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-09 03:18:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 109

feat(hosted_vllm): opt in to forwarding assistant reasoning content

Browse source
This commit is contained in:
jibanez-staticduo 2026-09-14 09:32:08 +02:00
parent 30f33a949b
commit 30bee8adc9
No known key found for this signature in database
9 changed files with 371 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -26,7 +26,7 @@ AWS_CREDENTIAL_KWARGS_KEYS: Final = frozenset(
# Keys `completion()` forwards from its own kwargs into `get_litellm_params`,
# which are otherwise invisible to it because that call site passes explicit
# named arguments rather than `**kwargs`.
FORWARDED_KWARGS_KEYS: Final = AWS_CREDENTIAL_KWARGS_KEYS
FORWARDED_KWARGS_KEYS: Final = AWS_CREDENTIAL_KWARGS_KEYS | frozenset({"forward_reasoning_content"})
# Pre-define optional kwargs keys as frozenset for O(1) lookups
# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
@ -59,7 +59,7 @@ OPTIONAL_KWARGS_KEYS: Final = (
"use_xai_oauth",
}
)
| AWS_CREDENTIAL_KWARGS_KEYS
| FORWARDED_KWARGS_KEYS
)
# Backward-compatible alias for existing imports/tests.

39
litellm/llms/hosted_vllm/chat/reasoning_content.md Normal file
View file

@ -0,0 +1,39 @@
# Forwarding assistant reasoning to hosted vLLM
Some vLLM backends and chat templates accept previous assistant reasoning alongside content and tool calls. Set `forward_reasoning_content: true` on an individual model entry to forward the `reasoning_content` field supplied by the client
```yaml
model_list:
- model_name: reasoning-history
litellm_params:
model: hosted_vllm/your-served-model
api_base: http://localhost:8000/v1
forward_reasoning_content: true
- model_name: default-history
litellm_params:
model: hosted_vllm/your-served-model
api_base: http://localhost:8000/v1
forward_reasoning_content: false
```
The default is false. Omitting the option or setting it to false retains the existing removal of assistant `reasoning_content`. The option is local to each request, applies only to `hosted_vllm`, and is not sent to the backend. It can also be passed to `litellm.completion` or `litellm.acompletion`
When enabled, LiteLLM forwards the supplied field without trimming it or inserting it into visible `content`. Content and tool calls keep their existing transformations. It does not reconstruct missing reasoning or convert Anthropic `thinking_blocks`, signatures, or redacted thinking. Their existing handling is unchanged
For Responses requests routed through Chat Completions, also set `use_chat_completions_api: true`. The option preserves the `reasoning_content` produced by that bridge from supported Responses reasoning input. It does not change native vLLM Responses requests or make opaque encrypted reasoning portable
## Backend compatibility
Enable this only after checking both your vLLM request parser and model chat template. A backend returning reasoning in its responses does not necessarily accept reasoning in previous assistant messages
The vLLM v0.12.0 and v0.13.0 chat parsers accept `reasoning` and `reasoning_content` and expose both names to the template. Newer vLLM versions may normalize the deprecated input name `reasoning_content` to `reasoning` before rendering. LiteLLM forwards its existing `reasoning_content` field, without adding a duplicate `reasoning` field or selecting behavior by model name. Older releases, vendor forks and custom templates need separate verification
Sources: [vLLM v0.12.0 chat parser](https://github.com/vllm-project/vllm/blob/v0.12.0/vllm/entrypoints/chat_utils.py#L1532), [vLLM v0.13.0 chat parser](https://github.com/vllm-project/vllm/blob/v0.13.0/vllm/entrypoints/chat_utils.py)
## Template policy and performance
`forward_reasoning_content` controls transport. A model-specific setting such as `chat_template_kwargs.preserve_thinking` controls which received history its template uses. In the Qwen3.8-Flash-Next template, `preserve_thinking: false` removes reasoning from earlier user turns but still retains reasoning within the current tool sequence. It does not mean that every assistant reasoning field should be deleted
See [Qwen3.8-Flash-Next preserved thinking](https://huggingface.co/Qwen/Qwen3.8-Flash-Next#disable-preserved-thinking)
Forwarding history can change the rendered prompt and token prefix. Boundary whitespace normalization may have no effect when the template trims that field. Validate rendered tokens with the actual tokenizer and template before drawing cache conclusions. This option provides no measured latency, cache-hit or reasoning-quality guarantee

31
litellm/llms/hosted_vllm/chat/transformation.py
View file

@ -4,6 +4,7 @@ Translate from OpenAI's `/v1/chat/completions` to VLLM's `/v1/chat/completions`
import json
from collections.abc import Coroutine
from copy import deepcopy
from typing import Any, Final, Literal, cast, overload
from litellm.litellm_core_utils.prompt_templates.common_utils import (
@ -145,6 +146,33 @@ class HostedVLLMChatConfig(OpenAIGPTConfig):
return ChatCompletionVideoObject(type="video_url", video_url=ChatCompletionVideoUrlObject(url=file_data))
raise ValueError("file_id or file_data is required")
def transform_request(
self,
model: str,
messages: list[AllMessageValues], # mutable-ok: provider request contract
optional_params: dict, # mutable-ok: provider request contract
litellm_params: dict, # mutable-ok: provider request contract
headers: dict, # mutable-ok: provider request contract
) -> dict: # mutable-ok: provider request contract
request_messages: Final = deepcopy(messages)
if litellm_params.get("forward_reasoning_content") is not True:
for message in request_messages:
if message["role"] == "assistant":
message.pop("reasoning_content", None)
return super().transform_request(model, request_messages, optional_params, litellm_params, headers)
async def async_transform_request(
self,
model: str,
messages: list[AllMessageValues], # mutable-ok: provider request contract
optional_params: dict, # mutable-ok: provider request contract
litellm_params: dict, # mutable-ok: provider request contract
headers: dict, # mutable-ok: provider request contract
) -> dict: # mutable-ok: provider request contract
return await super().async_transform_request(
model, deepcopy(messages), optional_params, litellm_params, headers
)
@overload
def _transform_messages(
self, messages: list[AllMessageValues], model: str, is_async: Literal[True]
@ -164,13 +192,12 @@ class HostedVLLMChatConfig(OpenAIGPTConfig):
"""
Support translating:
- video files from file_id or file_data to video_url
- thinking_blocks and reasoning_content on assistant messages are removed,
- thinking_blocks on assistant messages are removed,
and content lists are converted to strings for vLLM compatibility
"""
for message in messages:
if message["role"] == "assistant":
message.pop("thinking_blocks", None)
message.pop("reasoning_content", None)
existing_content = message.get("content")
if isinstance(existing_content, list):
text_parts = []

1
litellm/types/router.py
View file

@ -357,6 +357,7 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
)
model_config = ConfigDict(extra="allow", arbitrary_types_allowed=True)
merge_reasoning_content_in_choices: bool | None = False
forward_reasoning_content: bool | None = False
model_info: dict | None = None
mock_response: str | ModelResponse | Exception | Any | None = None

1
litellm/types/utils.py
View file

@ -3826,6 +3826,7 @@ all_litellm_params = (
"budget_duration",
"use_in_pass_through",
"merge_reasoning_content_in_choices",
"forward_reasoning_content",
"litellm_credential_name",
"allowed_openai_params",
"litellm_session_id",

119
tests/router_unit_tests/test_router_forward_reasoning_content.py Normal file
View file

@ -0,0 +1,119 @@
import json
from copy import deepcopy
from typing import Final
import httpx
import pytest
import respx
import litellm
from litellm import Router
URL: Final = "https://reasoning-test.invalid/v1/chat/completions"
MODEL: Final = "hosted_vllm/reasoning-test"
REASONING: Final = "Inspect both tool results before answering."
def _messages():
return [
{"role": "user", "content": "Compare both records"},
{
"role": "assistant",
"content": None,
"reasoning_content": REASONING,
"tool_calls": [
{"id": f"call_{index}", "type": "function", "function": {"name": "lookup", "arguments": "{}"}}
for index in (1, 2)
],
},
{"role": "tool", "tool_call_id": "call_1", "content": "first record"},
{"role": "tool", "tool_call_id": "call_2", "content": "second record"},
]
def _route(mock: respx.MockRouter):
return mock.post(URL).respond(
200,
json={
"id": "chatcmpl-reasoning-test",
"object": "chat.completion",
"created": 1,
"model": "reasoning-test",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "Compared"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12},
},
)
def _assert_wire(request: httpx.Request, enabled: bool):
body: Final = json.loads(request.content)
assert body["model"] == "reasoning-test"
assert "forward_reasoning_content" not in body
assert "forward_reasoning_content" not in request.content.decode()
messages: Final = body["messages"]
assert [message["role"] for message in messages] == ["user", "assistant", "tool", "tool"]
assert [tool["id"] for tool in messages[1]["tool_calls"]] == ["call_1", "call_2"]
assert [message["tool_call_id"] for message in messages[2:]] == ["call_1", "call_2"]
assert [message["content"] for message in messages[2:]] == ["first record", "second record"]
assert messages[1].get("reasoning_content") == (REASONING if enabled else None)
assert request.content.decode().count(REASONING) == int(enabled)
@pytest.mark.asyncio
@pytest.mark.parametrize("async_mode", [False, True], ids=["completion", "acompletion"])
@pytest.mark.parametrize("forward", [None, False, True], ids=["absent", "false", "true"])
async def test_direct_completion_reasoning_flag_reaches_final_wire(
async_mode: bool, forward: bool | None, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
messages: Final = _messages()
original: Final = deepcopy(messages)
kwargs: Final = {
"model": MODEL,
"api_base": URL.removesuffix("/chat/completions"),
"api_key": "test-key",
"messages": messages,
**({} if forward is None else {"forward_reasoning_content": forward}),
}
with respx.mock(assert_all_called=True) as mock:
route: Final = _route(mock)
response: Final = await litellm.acompletion(**kwargs) if async_mode else litellm.completion(**kwargs)
assert response.choices[0].message.content == "Compared"
assert route.call_count == 1
_assert_wire(route.calls[0].request, forward is True)
assert messages == original
@pytest.mark.asyncio
@pytest.mark.parametrize("async_mode", [False, True], ids=["completion", "acompletion"])
async def test_router_aliases_isolate_reasoning_flag_on_same_backend(async_mode: bool, monkeypatch: pytest.MonkeyPatch):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
model_list: Final = [
{
"model_name": alias,
"litellm_params": {
"model": MODEL,
"api_base": URL.removesuffix("/chat/completions"),
"api_key": "test-key",
**({} if forward is None else {"forward_reasoning_content": forward}),
},
}
for alias, forward in (("default", None), ("disabled", False), ("enabled", True))
]
original_models: Final = deepcopy(model_list)
router: Final = Router(model_list=model_list, num_retries=0)
messages: Final = _messages()
original_messages: Final = deepcopy(messages)
with respx.mock(assert_all_called=True) as mock:
route: Final = _route(mock)
for alias in ("enabled", "default", "disabled", "enabled"):
response = (
await router.acompletion(model=alias, messages=messages)
if async_mode
else router.completion(model=alias, messages=messages)
)
assert response.choices[0].message.content == "Compared"
_assert_wire(route.calls[-1].request, alias == "enabled")
assert messages == original_messages
assert route.call_count == 4
assert model_list == original_models

84
tests/test_litellm/llms/hosted_vllm/chat/test_hosted_vllm_chat_transformation.py
View file

@ -1,6 +1,8 @@
import json
from copy import deepcopy
from unittest.mock import MagicMock, patch
import pytest
from litellm.constants import (
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET,
@ -9,6 +11,88 @@ from litellm.constants import (
from litellm.llms.hosted_vllm.chat.transformation import HostedVLLMChatConfig
@pytest.mark.parametrize("params", [{}, {"forward_reasoning_content": False}, {"forward_reasoning_content": True}])
@pytest.mark.parametrize("is_async", [False, True])
@pytest.mark.asyncio
async def test_forward_reasoning_content_preserves_only_explicit_history(params, is_async):
config = HostedVLLMChatConfig()
messages = [
{"role": "user", "content": "Check the counter"},
{
"role": "assistant",
"content": None,
"reasoning_content": " synthetic history\n",
"thinking_blocks": [{"type": "thinking", "thinking": "Do not convert this", "signature": "sig"}],
"tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "read", "arguments": "{}"}}],
},
{"role": "tool", "tool_call_id": "call_1", "content": "7"},
{
"role": "assistant",
"content": None,
"thinking_blocks": [{"type": "thinking", "thinking": "Never synthesize history", "signature": "sig"}],
"tool_calls": [{"id": "call_2", "type": "function", "function": {"name": "verify", "arguments": "{}"}}],
},
{"role": "tool", "tool_call_id": "call_2", "content": "verified"},
]
original = deepcopy(messages)
arguments = dict(
model="qwen3.8-flash-next", messages=messages, optional_params={}, litellm_params=params, headers={}
)
result = await config.async_transform_request(**arguments) if is_async else config.transform_request(**arguments)
expected = deepcopy(original)
expected[1].pop("thinking_blocks")
expected[3].pop("thinking_blocks")
if params.get("forward_reasoning_content") is not True:
expected[1].pop("reasoning_content")
assert result["messages"] == expected
assert messages == original
assert "forward_reasoning_content" not in result
@pytest.mark.asyncio
async def test_forward_reasoning_content_reused_config_and_caller_are_isolated():
config = HostedVLLMChatConfig()
messages = [{"role": "assistant", "content": "answer", "reasoning_content": "synthetic history"}]
original = deepcopy(messages)
for enabled in (False, True, False, True):
for transform in (config.transform_request, config.async_transform_request):
result = transform(
model="qwen3.8-flash-next",
messages=messages,
optional_params={},
litellm_params={"forward_reasoning_content": enabled},
headers={},
)
if transform == config.async_transform_request:
result = await result
assert result["messages"] == (original if enabled else [{"role": "assistant", "content": "answer"}])
assert messages == original
@pytest.mark.asyncio
async def test_forward_reasoning_content_keeps_async_content_conversion():
class AsyncContentConfig(HostedVLLMChatConfig):
async def _async_transform_content_item(self, content_item):
return {"type": "image_url", "image_url": {"url": "data:image/png;base64,c3ludGhldGlj"}}
config = AsyncContentConfig()
messages = [
{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://example.invalid/image.png"}}]},
{"role": "assistant", "content": "answer", "reasoning_content": "synthetic history"},
]
original = deepcopy(messages)
result = await config.async_transform_request(
model="qwen3.8-flash-next",
messages=messages,
optional_params={},
litellm_params={"forward_reasoning_content": True},
headers={},
)
assert result["messages"][0]["content"][0]["image_url"]["url"] == "data:image/png;base64,c3ludGhldGlj"
assert result["messages"][1]["reasoning_content"] == "synthetic history"
assert messages == original
def test_hosted_vllm_chat_transformation_file_url():
config = HostedVLLMChatConfig()
video_url = "https://example.com/video.mp4"

86
tests/test_litellm/responses/litellm_completion_transformation/test_litellm_completion_responses.py
View file

@ -2,7 +2,9 @@ import json
from copy import deepcopy
from typing import Final, Literal
import httpx
import pytest
import respx
from openai.types.responses.response_function_web_search import (
ActionFind,
ActionOpenPage,
@ -30,6 +32,90 @@ from litellm.types.utils import (
)
@pytest.mark.asyncio
@pytest.mark.parametrize("async_mode", [False, True], ids=["responses", "aresponses"])
@pytest.mark.parametrize("forward", [None, False, True], ids=["absent", "false", "true"])
@pytest.mark.parametrize("sequential", [False, True], ids=["parallel-tools", "sequential-tools"])
async def test_hosted_vllm_responses_reasoning_and_parallel_tools_final_wire(
async_mode: bool, forward: bool | None, sequential: bool, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
reasoning: Final = "Inspect both tool results before answering."
next_reasoning: Final = "Use the first result to inspect the second."
input_items: Final = [
{"role": "user", "content": "Compare both records"},
{
"type": "reasoning",
"id": "rs_previous",
"summary": [],
"content": [{"type": "reasoning_text", "text": reasoning}],
},
{"type": "function_call", "call_id": "call_1", "name": "lookup", "arguments": "{}"},
*(
[
{"type": "function_call_output", "call_id": "call_1", "output": "first record"},
{
"type": "reasoning",
"id": "rs_next",
"summary": [],
"content": [{"type": "reasoning_text", "text": next_reasoning}],
},
]
if sequential
else []
),
{"type": "function_call", "call_id": "call_2", "name": "lookup", "arguments": "{}"},
*([] if sequential else [{"type": "function_call_output", "call_id": "call_1", "output": "first record"}]),
{"type": "function_call_output", "call_id": "call_2", "output": "second record"},
]
original: Final = deepcopy(input_items)
kwargs: Final = {
"model": "hosted_vllm/reasoning-test",
"input": input_items,
"api_base": "https://responses-reasoning-test.invalid/v1",
"api_key": "test-key",
"use_chat_completions_api": True,
**({} if forward is None else {"forward_reasoning_content": forward}),
}
with respx.mock(assert_all_called=True) as mock:
route: Final = mock.post("https://responses-reasoning-test.invalid/v1/chat/completions").mock(
return_value=httpx.Response(
200,
json={
"id": "chatcmpl-bridge-reasoning",
"object": "chat.completion",
"created": 1,
"model": "reasoning-test",
"choices": [
{"index": 0, "message": {"role": "assistant", "content": "Compared"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 2, "total_tokens": 12},
},
)
)
response: Final = await litellm.aresponses(**kwargs) if async_mode else litellm.responses(**kwargs)
assert response.output[0].content[0].text == "Compared"
assert route.call_count == 1
payload: Final = json.loads(route.calls[0].request.content)
assert payload["model"] == "reasoning-test"
assert "forward_reasoning_content" not in route.calls[0].request.content.decode()
assert "use_chat_completions_api" not in payload
messages: Final = payload["messages"]
assert [message["role"] for message in messages] == (
["user", "assistant", "tool", "assistant", "tool"] if sequential else ["user", "assistant", "tool", "tool"]
)
assert [tool["id"] for message in messages for tool in message.get("tool_calls", [])] == ["call_1", "call_2"]
results: Final = [message for message in messages if message["role"] == "tool"]
assert [message["tool_call_id"] for message in results] == ["call_1", "call_2"]
assert [message["content"] for message in results] == ["first record", "second record"]
assert messages[1].get("reasoning_content") == (reasoning if forward is True else None)
assert route.calls[0].request.content.decode().count(reasoning) == int(forward is True)
if sequential:
assert messages[3].get("reasoning_content") == (next_reasoning if forward is True else None)
assert route.calls[0].request.content.decode().count(next_reasoning) == int(forward is True)
assert input_items == original
class TestLiteLLMCompletionResponsesConfig:
def test_transform_input_file_item_to_file_item_with_file_id(self):
"""Test transformation of input_file item with file_id to Chat Completion file format"""

12
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -16781,7 +16781,6 @@ export interface paths {
* - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking.
* - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" }
* - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.
* - soft_budget: Optional[float] - Get alerts when user crosses given budget, doesn't block requests.
* - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)
* - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}.
* - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys)
@ -16887,7 +16886,6 @@ export interface paths {
* - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking.
* - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" }
* - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.
* - soft_budget: Optional[float] - Get alerts when user crosses given budget, doesn't block requests.
* - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)
* - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}.
* - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys)
@ -29713,6 +29711,11 @@ export interface components {
default_api_key_tpm_limit?: number | null;
/** Drop Params */
drop_params?: boolean | string | null;
/**
* Forward Reasoning Content
* @default false
*/
forward_reasoning_content: boolean | null;
/** Gcs Bucket Name */
gcs_bucket_name?: string | null;
/** Google Maps Grounding Cost Per Query */
@ -39927,6 +39930,11 @@ export interface components {
default_api_key_tpm_limit?: number | null;
/** Drop Params */
drop_params?: boolean | string | null;
/**
* Forward Reasoning Content
* @default false
*/
forward_reasoning_content: boolean | null;
/** Gcs Bucket Name */
gcs_bucket_name?: string | null;
/** Google Maps Grounding Cost Per Query */