scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-09 03:18:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 108

fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393)

Browse source 
* fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt

An order-based or configured fallback hop replayed the failed provider's
encrypted reasoning items to the next deployment, which answered 400
(Bedrock Mantle: invalid encrypted reasoning; OpenAI:
invalid_encrypted_content), so every multi-turn Responses fallback for
Codex-style clients failed. The hop now drops the encrypted reasoning
its target cannot decrypt and keeps each item's readable summary. With
encrypted_content_affinity on, the pin narrows to the hop's target
order instead of emptying it, so the hop reaches the next order
instead of failing with no deployments available.

* fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt

Unmarked encrypted reasoning on a hop is attributed to the deployment that just
failed, read from the retry breadcrumb, so a hop to a deployment on the same
api_base and api_key keeps it and a cross-provider hop still drops it. The hop
tests script the upstream at the httpx boundary instead of doubling the handler,
and the router coverage script lists the two hop helpers with their tests

* fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot

* test(router): use a real Router without the origin deployment in the hop strip test

* test(router): cover the hop strip when no failed deployment is known

* fix(proxy): drop the router's fallback hop state keys from the client body

* fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys

* test(integration): audit cells for the fallback hop encrypted reasoning strip

Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot] 2026-10-08 17:30:01 -07:00 • committed by GitHub
parent 51fea825cc
commit 8a176c0f0a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 1957 additions and 23 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
litellm/proxy/litellm_pre_call_utils.py
View file

@ -286,6 +286,9 @@ LITELLM_TRACE_CONTROL_METADATA_FIELDS: Final = frozenset(
_UNTRUSTED_ROOT_CONTROL_FIELDS: Final = (
"weights",
"_router_weights",
"fallback_depth",
"_target_order",
"attempted_targets",
"proxy_server_request",
"standard_logging_object",
"secret_fields",

38
litellm/router.py
View file

@ -500,19 +500,33 @@ _MODEL_INFO_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_SILENT_MODEL_ADAPTER: Final = TypeAdapter(str | list[str])
_RESOLVED_RETRY_POLICY_ADAPTER: Final = TypeAdapter(RetryPolicy | None)
_ROUTING_KWARGS_ADAPTER: Final[TypeAdapter[Mapping[str, object] | None]] = TypeAdapter(Mapping[str, object] | None)
_FALLBACK_HOP_ADAPTER: Final = TypeAdapter(Mapping[str, object])
_DEPLOYMENT_SELECTED_EVENT: Final = "litellm.request.deployment_selected"
def _is_fallback_hop(request_kwargs: Mapping[str, object]) -> bool:
fallback_depth: Final = request_kwargs.get("fallback_depth")
return isinstance(fallback_depth, int) and fallback_depth > 0
def _deployment_that_just_failed(request_metadata: object) -> str | None:
try:
model_info: Final = _FALLBACK_HOP_ADAPTER.validate_python(
_FALLBACK_HOP_ADAPTER.validate_python(request_metadata).get("model_info")
)
except ValidationError:
return None
model_id: Final = model_info.get("id")
return model_id if isinstance(model_id, str) else None
def _deployment_pick_attributes(model: str, request_kwargs: Mapping[str, object] | None) -> Mapping[str, str | int]:
"""Bounded attributes for one deployment pick; attempt is 1-based within the current model group."""
kwargs: Final = request_kwargs or {}
metadata: Final = kwargs.get("litellm_metadata", kwargs.get("metadata"))
attempted_retries: Final = metadata.get("attempted_retries") if isinstance(metadata, Mapping) else None
retries: Final = attempted_retries if isinstance(attempted_retries, int) else 0
fallback_depth: Final = kwargs.get("fallback_depth")
reason: Final = (
"retry" if retries > 0 else "fallback" if isinstance(fallback_depth, int) and fallback_depth > 0 else "initial"
)
reason: Final = "retry" if retries > 0 else "fallback" if _is_fallback_hop(kwargs) else "initial"
return MappingProxyType(
{
"litellm.deployment.attempt": retries + 1,
@ -4139,10 +4153,11 @@ class Router:
function_name: str | None = None,
) -> None:
"""
3 jobs:
4 jobs:
- Adds selected deployment, model_info and api_base to kwargs["metadata"] (used for logging)
- Adds default litellm params to kwargs, if set.
- Merges tools from deployment with request (proxy-configured tools + request tools).
- On a fallback hop, drops the encrypted reasoning this deployment cannot decrypt, keeping its summary.
"""
for key in self._forwarded_alias_marker_keys_the_deployment_sets(
deployment=deployment, forwarded_keys=kwargs.pop(_ALIAS_MARKER_FORWARDED_PARAMS_KWARG, ())
@ -4165,6 +4180,9 @@ class Router:
metadata_variable_name: Final = get_router_metadata_variable_name(
function_name=function_name,
)
deployment_that_just_failed: Final = _deployment_that_just_failed(
_FALLBACK_HOP_ADAPTER.validate_python(kwargs).get(metadata_variable_name)
)
kwargs.setdefault(metadata_variable_name, {}).update(
{
@ -4237,6 +4255,15 @@ class Router:
kwargs["timeout"] = self._get_timeout(kwargs=kwargs, data=deployment["litellm_params"])
self._update_kwargs_with_default_litellm_params(kwargs=kwargs, metadata_variable_name=metadata_variable_name)
hop_kwargs: Final = _FALLBACK_HOP_ADAPTER.validate_python(kwargs)
if _is_fallback_hop(hop_kwargs):
EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
self,
hop_kwargs.get("input"),
hop_kwargs.get("messages"),
(_FALLBACK_HOP_ADAPTER.validate_python(deployment),),
unmarked_origin=deployment_that_just_failed,
)
def _get_async_openai_model_client(self, deployment: dict, kwargs: dict):
"""
@ -5532,6 +5559,7 @@ class Router:
model=model, original_generic_function=original_generic_function, **kwargs
)
carry_over_pre_routing_selection(live_kwargs=kwargs, snapshot=hop_kwargs)
carry_over_routed_deployment(live_kwargs=kwargs, snapshot=hop_kwargs)
if kwargs.get("stream") and isinstance(response, BaseResponsesAPIStreamingIterator):
return await self._aresponses_streaming_iterator(response=response, initial_kwargs=hop_kwargs)
return response

68
litellm/router_utils/pre_call_checks/encrypted_content_affinity_check.py
View file

@ -40,6 +40,8 @@ from collections.abc import Iterator, Mapping, Sequence
from functools import cache
from typing import TYPE_CHECKING, Final, Optional, cast
from pydantic import TypeAdapter
from litellm._logging import verbose_router_logger
from litellm.integrations.custom_logger import CustomLogger, Span
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
@ -51,10 +53,13 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.types.llms.openai import AllMessageValues
from litellm.types.router import Deployment
from litellm.utils import get_order_filtered_deployments
if TYPE_CHECKING:
from litellm.router import Router
_REQUEST_KWARGS_ADAPTER: Final = TypeAdapter(Mapping[str, object])
class EncryptedContentAffinityCheck(CustomLogger):
"""
@ -253,12 +258,22 @@ class EncryptedContentAffinityCheck(CustomLogger):
]
return matches, originating
def _strip_reasoning_the_target_cannot_decrypt(
self,
@staticmethod
def strip_reasoning_the_targets_cannot_decrypt(
router: "Router | None",
request_input: object,
anthropic_messages: object,
target_deployments: Sequence[Mapping[str, object]],
*,
unmarked_origin: str | None,
) -> None:
"""
Drop the encrypted reasoning that none of ``target_deployments`` minted or shares an
encryption boundary with, keeping each item's readable summary. Encrypted reasoning that
carries no litellm origin marker is attributed to ``unmarked_origin``: the affinity pin
names the deployment its marker decoded to, a fallback hop names the deployment that just
failed, and ``None`` drops it, since no deployment is known to have minted it.
"""
target_ids: Final = frozenset(
str(model_info["id"])
for target in target_deployments
@ -267,30 +282,34 @@ class EncryptedContentAffinityCheck(CustomLogger):
target_boundaries: Final = frozenset(
boundary
for target in target_deployments
if (boundary := self._encryption_boundary_key(target.get("litellm_params"))) is not None
if (boundary := EncryptedContentAffinityCheck._encryption_boundary_key(target.get("litellm_params")))
is not None
)
@cache
def target_can_decrypt(origin_model_id: str) -> bool:
def target_can_decrypt(marked_origin: str | None) -> bool:
origin_model_id: Final = marked_origin if marked_origin is not None else unmarked_origin
if origin_model_id is None:
return False
if origin_model_id in target_ids:
return True
if self.router is None:
if router is None:
return False
origin: Final = self.router.get_deployment(model_id=origin_model_id)
origin: Final = router.get_deployment(model_id=origin_model_id)
origin_boundary: Final = (
self._encryption_boundary_key(origin.litellm_params.model_dump(exclude_none=True))
EncryptedContentAffinityCheck._encryption_boundary_key(
origin.litellm_params.model_dump(exclude_none=True)
)
if origin is not None
else None
)
return origin_boundary is not None and origin_boundary in target_boundaries
def should_strip_input_item(item: Mapping[str, object]) -> bool:
origin_model_id: Final = self._model_id_of_input_item(item)
return origin_model_id is not None and not target_can_decrypt(origin_model_id)
return not target_can_decrypt(EncryptedContentAffinityCheck._model_id_of_input_item(item))
def should_strip_anthropic_block(block: Mapping[str, object]) -> bool:
origin_model_id: Final = self._model_id_of_anthropic_block(block)
return origin_model_id is not None and not target_can_decrypt(origin_model_id)
return not target_can_decrypt(EncryptedContentAffinityCheck._model_id_of_anthropic_block(block))
ResponsesAPIRequestUtils.strip_encrypted_reasoning_from_input(
request_input, should_strip=should_strip_input_item
@ -317,12 +336,21 @@ class EncryptedContentAffinityCheck(CustomLogger):
unhealthy, the request was routed to a different group by an auto-router tier
change or model switch, or the marker is removed/unknown/forged), the
encrypted reasoning is stripped and the request dispatches to the healthy
pool with its readable history instead of failing.
pool with its readable history instead of failing. An order-based fallback hop
carries ``_target_order``, and the pin only considers deployments of that order,
so the hop reaches the next order with the origin's reasoning stripped instead
of replaying it to a deployment that cannot decrypt it.
"""
request_kwargs = request_kwargs or {}
typed_healthy_deployments: Final = cast(list[dict], healthy_deployments)
typed_healthy_deployments: Final = cast(list[dict[str, object]], healthy_deployments)
if not self._is_enabled_for_model_group(model):
return typed_healthy_deployments
target_order: Final = _REQUEST_KWARGS_ADAPTER.validate_python(request_kwargs).get("_target_order")
candidates: Final = (
get_order_filtered_deployments(typed_healthy_deployments, target_order=target_order)
if isinstance(target_order, int)
else typed_healthy_deployments
)
# Signal to the response post-processor that encrypted item IDs should be
# encoded in the output of this request. Only set the flag when
@ -348,7 +376,7 @@ class EncryptedContentAffinityCheck(CustomLogger):
)
deployment: Final = self._find_deployment_by_model_id(
healthy_deployments=typed_healthy_deployments,
healthy_deployments=candidates,
model_id=model_id,
)
if deployment is not None:
@ -357,12 +385,14 @@ class EncryptedContentAffinityCheck(CustomLogger):
model_id,
)
request_kwargs["_encrypted_content_affinity_pinned"] = True
self._strip_reasoning_the_target_cannot_decrypt(request_input, anthropic_messages, (deployment,))
self.strip_reasoning_the_targets_cannot_decrypt(
self.router, request_input, anthropic_messages, (deployment,), unmarked_origin=model_id
)
return [deployment]
# Follow-up switched model_name (LIT-2531): pin by Azure resource instead.
boundary_matches, _originating = self._find_deployments_on_same_encryption_boundary(
healthy_deployments=typed_healthy_deployments,
healthy_deployments=candidates,
model_id=model_id,
)
if boundary_matches:
@ -373,7 +403,9 @@ class EncryptedContentAffinityCheck(CustomLogger):
len(boundary_matches),
)
request_kwargs["_encrypted_content_affinity_pinned"] = True
self._strip_reasoning_the_target_cannot_decrypt(request_input, anthropic_messages, boundary_matches)
self.strip_reasoning_the_targets_cannot_decrypt(
self.router, request_input, anthropic_messages, boundary_matches, unmarked_origin=model_id
)
return boundary_matches
# The origin cannot serve this turn and no peer shares its encryption boundary, so its
@ -389,4 +421,4 @@ class EncryptedContentAffinityCheck(CustomLogger):
)
ResponsesAPIRequestUtils.strip_encrypted_reasoning_from_input(request_input)
strip_encrypted_reasoning_from_messages(anthropic_messages)
return typed_healthy_deployments
return candidates

2
tests/code_coverage_tests/router_code_coverage.py
View file

@ -110,6 +110,8 @@ ignored_function_names = [
"_aanthropic_messages_yield_recovered", # Tested through every mid-stream retry and fallback test in test_router.py
"_anthropic_messages_policy_retries", # Tested through the retry budget precedence test in test_router.py
"_get_wildcard_deployments", # Tested through the get_model_list_of_routed_group wildcard test in test_router.py
"_is_fallback_hop", # Tested through the order fallback hop tests in test_router_order_fallback.py
"_deployment_that_just_failed", # Tested through the same-boundary hop test in test_router_order_fallback.py
]

1299
tests/integration/routing/test_fallback_hop_foreign_encrypted_reasoning.py Normal file
View file

File diff suppressed because it is too large Load diff

60
tests/unit/proxy/test_litellm_pre_call_utils.py
View file

@ -1067,6 +1067,66 @@ async def test_add_litellm_data_to_request_strips_string_encoded_admin_injection
assert "_pipeline_managed_guardrails" not in other
@pytest.mark.asyncio
@pytest.mark.parametrize(
"forged_field,forged_value",
[
("fallback_depth", 1),
("fallback_depth", True),
("_target_order", 2),
("attempted_targets", ["forged"]),
],
)
async def test_add_litellm_data_to_request_strips_forged_fallback_hop_state(
forged_field: str, forged_value: object
) -> None:
request_mock = MagicMock(spec=Request)
request_mock.url = MagicMock()
request_mock.url.path = "/v1/responses"
request_mock.url.__str__.return_value = "http://localhost/v1/responses"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
updated = await add_litellm_data_to_request(
data={"model": "hop", "input": "hello", forged_field: forged_value},
request=request_mock,
user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert forged_field not in updated
assert forged_field not in updated["proxy_server_request"]["body"]
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_keeps_the_request_max_fallbacks_cap() -> None:
request_mock = MagicMock(spec=Request)
request_mock.url = MagicMock()
request_mock.url.path = "/v1/responses"
request_mock.url.__str__.return_value = "http://localhost/v1/responses"
request_mock.method = "POST"
request_mock.query_params = {}
request_mock.headers = {"Content-Type": "application/json"}
request_mock.client = MagicMock()
request_mock.client.host = "127.0.0.1"
updated = await add_litellm_data_to_request(
data={"model": "hop", "input": "hello", "max_fallbacks": 0},
request=request_mock,
user_api_key_dict=UserAPIKeyAuth(api_key="hashed-key"),
proxy_config=MagicMock(),
general_settings={},
version="test-version",
)
assert updated["max_fallbacks"] == 0
@pytest.mark.asyncio
async def test_add_litellm_data_to_request_strips_user_control_fields():
"""Strip untrusted proxy-control fields before guardrails, logging, and headers read metadata."""

115
tests/unit/router_utils/pre_call_checks/test_encrypted_content_affinity_check.py
View file

@ -2059,6 +2059,71 @@ async def test_real_router_selection_keeps_origin_reasoning_and_strips_foreign_o
router.discard()
@pytest.mark.asyncio
async def test_affinity_pin_yields_to_the_fallback_hops_target_order_and_strips_the_origins_reasoning():
router = litellm.Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra",
"api_base": "https://api.openai.com/v1",
"api_key": "key-openai",
"order": 1,
},
"model_info": {"id": "dep-openai"},
},
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "bedrock_mantle/openai.gpt-6-astra",
"api_base": "https://bedrock-mantle.us-east-1.api.aws",
"api_key": "key-mantle",
"order": 2,
},
"model_info": {"id": "dep-mantle"},
},
],
optional_pre_call_checks=["encrypted_content_affinity"],
num_retries=0,
)
openai_wrapped = ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id("blob-openai", "dep-openai")
def history() -> list:
return [
{"type": "message", "role": "user", "content": "first question"},
{
"type": "reasoning",
"encrypted_content": openai_wrapped,
"summary": [{"type": "summary_text", "text": "openai summary"}],
},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "first answer"}]},
{"type": "message", "role": "user", "content": "second question"},
]
try:
first_attempt = {"input": history(), "store": False}
pinned = await router.async_get_available_deployment(
model="gpt-6-astra", request_kwargs=first_attempt, input=first_attempt["input"]
)
assert pinned["model_info"]["id"] == "dep-openai"
assert first_attempt["input"] == history()
hop = {"input": history(), "store": False, "_target_order": 2, "fallback_depth": 1}
hop_deployment = await router.async_get_available_deployment(
model="gpt-6-astra", request_kwargs=hop, input=hop["input"]
)
assert hop_deployment["model_info"]["id"] == "dep-mantle"
assert hop["input"] == [
{"type": "message", "role": "user", "content": "first question"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "openai summary"}]},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "first answer"}]},
{"type": "message", "role": "user", "content": "second question"},
]
finally:
router.discard()
@pytest.mark.asyncio
async def test_affinity_keeps_mixed_origins_on_the_same_encryption_boundary():
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
@ -2271,6 +2336,56 @@ async def test_affinity_strips_unknown_origins_but_leaves_unmarked_encrypted_con
]
def _router_without_the_origin():
return litellm.Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra",
"api_base": "https://api.openai.com/v1",
"api_key": "openai-key",
},
"model_info": {"id": "target-order-2"},
}
],
num_retries=0,
)
@pytest.mark.parametrize("router", [None, _router_without_the_origin()], ids=["no router", "origin removed"])
@pytest.mark.parametrize(
"unmarked_origin", ["origin-removed", None], ids=["failed deployment named", "failed deployment unknown"]
)
def test_hop_strip_drops_unmarked_reasoning_whose_origin_cannot_be_resolved(router, unmarked_origin):
from litellm.router_utils.pre_call_checks.encrypted_content_affinity_check import (
EncryptedContentAffinityCheck,
)
request_input = [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{
"type": "reasoning",
"id": "rs_unmarked",
"encrypted_content": "gAAAAA-minted-by-a-removed-deployment",
"summary": [{"type": "summary_text", "text": "multiply 17 by 23"}],
},
]
target = {
"model_info": {"id": "target-order-2"},
"litellm_params": {"api_base": "https://api.openai.com/v1", "api_key": "openai-key"},
}
EncryptedContentAffinityCheck.strip_reasoning_the_targets_cannot_decrypt(
router, request_input, None, (target,), unmarked_origin=unmarked_origin
)
assert request_input == [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "multiply 17 by 23"}]},
]
def _cross_group_request_kwargs():
wrapped = ResponsesAPIRequestUtils.wrap_encrypted_content_with_model_id("gAAAAA-blob", "deployment-a")
return {

229
tests/unit/test_router/test_router.py
View file

@ -4723,6 +4723,235 @@ async def test_aresponses_streaming_iterator_fallback():
assert call_kwargs["disable_fallbacks"] is False
@pytest.mark.asyncio
async def test_aresponses_mid_stream_order_fallback_hop_drops_the_encrypted_reasoning_the_next_provider_cannot_decrypt(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
):
"""A Codex-style multi-turn history replays the order-1 provider's encrypted reasoning. When that
provider's stream breaks before its first output chunk, the order-2 hop must not replay reasoning
the next provider cannot decrypt; the readable summary stays."""
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
def history() -> list:
return [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{
"type": "reasoning",
"id": "rs_order1",
"encrypted_content": "gAAAAA-minted-by-order-1",
"summary": [{"type": "summary_text", "text": "multiply 17 by 23"}],
},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "391"}]},
{"type": "message", "role": "user", "content": "And 19*21?"},
]
def response_body(response_id: str, model: str, status: str, output: list) -> dict:
return {
"id": response_id,
"object": "response",
"created_at": 0,
"status": status,
"model": model,
"output": output,
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2} if status == "completed" else None,
}
def sse(events: list) -> httpx.Response:
body: Final = "".join(f"data: {json.dumps(event)}\n\n" for event in events)
return httpx.Response(200, content=body, headers={"content-type": "text/event-stream"})
openai_opened: Final = response_body("resp_openai", "gpt-6-astra", "in_progress", [])
openai_route: Final = respx_mock.post("https://api.openai.com/v1/responses").mock(
return_value=sse(
[
{"type": "response.created", "sequence_number": 0, "response": openai_opened},
{"type": "response.in_progress", "sequence_number": 1, "response": openai_opened},
{
"type": "error",
"sequence_number": 2,
"error": {
"type": "server_error",
"code": "server_error",
"message": "The server had an error while processing your request",
"param": None,
},
},
]
)
)
mantle_answer: Final = [
{
"id": "msg_mantle",
"type": "message",
"status": "completed",
"role": "assistant",
"content": [{"type": "output_text", "text": "399", "annotations": []}],
}
]
mantle_route: Final = respx_mock.post("https://bedrock-mantle.us-east-1.api.aws/openai/v1/responses").mock(
return_value=sse(
[
{
"type": "response.created",
"sequence_number": 0,
"response": response_body("resp_mantle", "openai.gpt-6-astra", "in_progress", []),
},
{
"type": "response.completed",
"sequence_number": 1,
"response": response_body("resp_mantle", "openai.gpt-6-astra", "completed", mantle_answer),
},
]
)
)
router = litellm.Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {"model": "openai/gpt-6-astra", "api_key": "openai-key", "order": 1},
"model_info": {"id": "openai-order-1"},
},
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "bedrock_mantle/openai.gpt-6-astra",
"api_key": "mantle-bearer-token",
"aws_region_name": "us-east-1",
"order": 2,
},
"model_info": {"id": "mantle-order-2"},
},
],
num_retries=0,
)
stream = await router.aresponses(model="gpt-6-astra", input=history(), store=False, stream=True)
collected = [event async for event in stream]
assert [event.type for event in collected] == ["response.created", "response.completed"]
assert json.loads(openai_route.calls.last.request.read())["input"] == history()
assert json.loads(mantle_route.calls.last.request.read())["input"] == [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "multiply 17 by 23"}]},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "391"}]},
{"type": "message", "role": "user", "content": "And 19*21?"},
]
@pytest.mark.asyncio
async def test_aresponses_mid_stream_order_fallback_hop_keeps_the_encrypted_reasoning_the_same_boundary_can_decrypt(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
):
"""The mid-stream hop re-enters the chain on a snapshot taken before routing, so the snapshot has to
carry the deployment that streamed and failed: a same-boundary order-2 deployment can decrypt that
deployment's unmarked reasoning and must receive it unchanged."""
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
def history() -> list:
return [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{
"type": "reasoning",
"id": "rs_order1",
"encrypted_content": "gAAAAA-minted-by-order-1",
"summary": [{"type": "summary_text", "text": "multiply 17 by 23"}],
},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "391"}]},
{"type": "message", "role": "user", "content": "And 19*21?"},
]
def response_body(response_id: str, model: str, status: str, output: list) -> dict:
return {
"id": response_id,
"object": "response",
"created_at": 0,
"status": status,
"model": model,
"output": output,
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2} if status == "completed" else None,
}
def sse(events: list) -> httpx.Response:
body: Final = "".join(f"data: {json.dumps(event)}\n\n" for event in events)
return httpx.Response(200, content=body, headers={"content-type": "text/event-stream"})
order_1_opened: Final = response_body("resp_order1", "gpt-6-astra", "in_progress", [])
order_2_answer: Final = [
{
"id": "msg_order2",
"type": "message",
"status": "completed",
"role": "assistant",
"content": [{"type": "output_text", "text": "399", "annotations": []}],
}
]
openai_route: Final = respx_mock.post("https://api.openai.com/v1/responses").mock(
side_effect=[
sse(
[
{"type": "response.created", "sequence_number": 0, "response": order_1_opened},
{"type": "response.in_progress", "sequence_number": 1, "response": order_1_opened},
{
"type": "error",
"sequence_number": 2,
"error": {
"type": "server_error",
"code": "server_error",
"message": "The server had an error while processing your request",
"param": None,
},
},
]
),
sse(
[
{
"type": "response.created",
"sequence_number": 0,
"response": response_body("resp_order2", "gpt-6-astra-mini", "in_progress", []),
},
{
"type": "response.completed",
"sequence_number": 1,
"response": response_body("resp_order2", "gpt-6-astra-mini", "completed", order_2_answer),
},
]
),
]
)
router = litellm.Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra",
"api_base": "https://api.openai.com/v1",
"api_key": "openai-key",
"order": 1,
},
"model_info": {"id": "openai-order-1"},
},
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra-mini",
"api_base": "https://api.openai.com/v1",
"api_key": "openai-key",
"order": 2,
},
"model_info": {"id": "openai-order-2"},
},
],
num_retries=0,
)
stream = await router.aresponses(model="gpt-6-astra", input=history(), store=False, stream=True)
collected = [event async for event in stream]
assert [event.type for event in collected] == ["response.created", "response.completed"]
assert [json.loads(call.request.read())["input"] for call in openai_route.calls] == [history(), history()]
@pytest.mark.asyncio
async def test_aresponses_streaming_content_policy_error_event_routes_to_content_policy_fallback():
"""Regression: a mid-stream content_policy_violation error event never reached

166
tests/unit/test_router_order_fallback.py
View file

@ -11,6 +11,7 @@ from typing import Final, Optional
import httpx
import pytest
import respx
from openai import AsyncOpenAI
import litellm
@ -645,6 +646,171 @@ async def test_text_completion_order_fallback_hop_does_not_send_target_order_ups
assert all("_target_order" not in body for body in upstream_bodies)
_OPENAI_RESPONSES_URL: Final = "https://api.openai.com/v1/responses"
_MANTLE_RESPONSES_URL: Final = "https://bedrock-mantle.us-east-1.api.aws/openai/v1/responses"
_OVERLOADED_UPSTREAM: Final = {"error": {"message": "overloaded", "type": "server_error", "code": "server_error"}}
def _completed_response_body(response_id: str, model: str, text: str) -> dict[str, object]:
return {
"id": response_id,
"object": "response",
"created_at": 0,
"status": "completed",
"model": model,
"output": [
{
"id": f"msg_{response_id}",
"type": "message",
"status": "completed",
"role": "assistant",
"content": [{"type": "output_text", "text": text, "annotations": []}],
}
],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2},
}
def _responses_history_with_order_1_reasoning() -> list[dict]:
return [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{
"type": "reasoning",
"id": "rs_order1",
"encrypted_content": "gAAAAA-minted-by-order-1",
"summary": [{"type": "summary_text", "text": "multiply 17 by 23"}],
},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "391"}]},
{"type": "message", "role": "user", "content": "And 19*21?"},
]
def _responses_history_without_order_1_encrypted_reasoning() -> list[dict]:
return [
{"type": "message", "role": "user", "content": "What is 17*23?"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "multiply 17 by 23"}]},
{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "391"}]},
{"type": "message", "role": "user", "content": "And 19*21?"},
]
def _openai_then_mantle_order_router() -> Router:
return Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {"model": "openai/gpt-6-astra", "api_key": "openai-key", "order": 1},
"model_info": {"id": "openai-order-1"},
},
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "bedrock_mantle/openai.gpt-6-astra",
"api_key": "mantle-bearer-token",
"aws_region_name": "us-east-1",
"order": 2,
},
"model_info": {"id": "mantle-order-2"},
},
],
num_retries=0,
)
def _two_openai_orders_on_one_encryption_boundary_router() -> Router:
return Router(
model_list=[
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra",
"api_base": "https://api.openai.com/v1",
"api_key": "openai-key",
"order": 1,
},
"model_info": {"id": "openai-order-1"},
},
{
"model_name": "gpt-6-astra",
"litellm_params": {
"model": "openai/gpt-6-astra-mini",
"api_base": "https://api.openai.com/v1",
"api_key": "openai-key",
"order": 2,
},
"model_info": {"id": "openai-order-2"},
},
],
num_retries=0,
)
@pytest.mark.asyncio
async def test_responses_order_fallback_hop_drops_the_encrypted_reasoning_the_next_provider_cannot_decrypt(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
openai_route: Final = respx_mock.post(_OPENAI_RESPONSES_URL).mock(
return_value=httpx.Response(500, json=_OVERLOADED_UPSTREAM)
)
mantle_route: Final = respx_mock.post(_MANTLE_RESPONSES_URL).mock(
return_value=httpx.Response(200, json=_completed_response_body("resp_mantle", "openai.gpt-6-astra", "399"))
)
response = await _openai_then_mantle_order_router().aresponses(
model="gpt-6-astra", input=_responses_history_with_order_1_reasoning(), store=False
)
assert response._hidden_params["model_id"] == "mantle-order-2"
assert json.loads(openai_route.calls.last.request.read())["input"] == _responses_history_with_order_1_reasoning()
assert (
json.loads(mantle_route.calls.last.request.read())["input"]
== _responses_history_without_order_1_encrypted_reasoning()
)
@pytest.mark.asyncio
async def test_responses_order_fallback_hop_keeps_the_encrypted_reasoning_the_same_boundary_can_decrypt(
respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
openai_route: Final = respx_mock.post(_OPENAI_RESPONSES_URL).mock(
side_effect=[
httpx.Response(500, json=_OVERLOADED_UPSTREAM),
httpx.Response(200, json=_completed_response_body("resp_order2", "gpt-6-astra-mini", "399")),
]
)
response = await _two_openai_orders_on_one_encryption_boundary_router().aresponses(
model="gpt-6-astra", input=_responses_history_with_order_1_reasoning(), store=False
)
assert response._hidden_params["model_id"] == "openai-order-2"
assert [json.loads(call.request.read())["input"] for call in openai_route.calls] == [
_responses_history_with_order_1_reasoning(),
_responses_history_with_order_1_reasoning(),
]
def test_fallback_hop_reads_the_deployment_that_just_failed_from_the_metadata_bucket_it_writes():
router: Final = _two_openai_orders_on_one_encryption_boundary_router()
order_2: Final = router.get_deployment(model_id="openai-order-2").model_dump(exclude_none=True)
hop_input: Final = _responses_history_with_order_1_reasoning()
hop_kwargs: Final = {
"model": "gpt-6-astra",
"input": hop_input,
"fallback_depth": 1,
"metadata": {"model_info": {"id": "openai-order-1"}},
"litellm_metadata": {"previous_models": [{"deployment_id": None}]},
}
router._update_kwargs_with_deployment(deployment=order_2, kwargs=hop_kwargs)
assert hop_input == _responses_history_with_order_1_reasoning()
assert hop_kwargs["metadata"]["model_info"]["id"] == "openai-order-2"
def test_check_non_standard_fallback_format():
from litellm.router_utils.fallback_event_handlers import (
check_non_standard_fallback_format,