scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-07 07:58:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 150

feat(router): add the Switchyard stage router as an auto router strategy (#50407)

Browse source 
* feat(router): add Switchyard Stage Router strategy

* feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router

Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes
ModelCall.respond and fail async. The classifier is now a tagged union of an LLM
judge and a decision model, and decision calls go through the router's adecisions
so any decisions provider works. Algorithm misbehavior and library errors route to
the capable model instead of failing the request, and the session id plus client
session headers are always passed so capable holds apply without a classifier

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): keep the Switchyard router inside the type-check gate

The gate's environment did not install the optional switchyard extra, so every
libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other
optional runtime packages, and imports go through switchyard_rust.libsy, which
ships py.typed. Step handling moves into helpers so no Final is rebound in the
loop, and the dashboard API types are regenerated for the new routing fields

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): route Switchyard on every endpoint and reject unsafe router saves

The router now passes each request's call type to routing strategies, and
Switchyard converts the request to chat messages with the existing converter for
that call type before translating it. /v1/messages turns with tool or thinking
blocks used to fall back to the capable model, and now route on their signals.
The Anthropic converter can keep tool_result.is_error on an analysis copy, so
Switchyard sees tool failures there, while requests sent to models never carry it

Saves of a strategy router that lists itself now fail with a 400 instead of a
500 with a dead row, the capable and efficient models must differ, judge and
decision calls are not retried before failing open, the Switchyard routing
fields are classified for redaction, and the edit dialog returns the saved
router instead of the stale one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): catch self-routing on patch and rename, and trim Switchyard tests

The self-routing check now reads the stored params through the decrypting
helper and runs when a patch carries no litellm_params, so a PATCH that points
a router at itself or renames it onto one of its targets returns 400 instead of
a 500 with a dead row. The Switchyard tests share one transcript builder and
router fixture, and reformatting noise in unrelated test files is reverted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(ui): make Switchyard a strategy in the existing auto router form

Switchyard is now a card in the existing strategy picker. Picking it swaps the
tier and classifier fields for the Switchyard fields, and the shared name, team
and save button create it through one submit branch, the same flow as the other
strategies. The separate add form, the extracted picker and the hidden-form
swap are gone, and the edit dialog keeps its own Switchyard form

The judge's session_affinity and message_hash_fallback settings are removed,
since Switchyard rejects both for the stage router

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ui): accept partial tool semantics and show server errors on model edit

The Switchyard tool semantics field rejected mappings that list only some
categories, although the server accepts them. Model PATCH errors were
reduced to a generic message, so it now goes through the shared API client
like model create and surfaces the server text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(ui): show the routing strategy picker up front in the auto router form

Every strategy was hidden behind an Advanced configuration link, so the
quick setup read as the only option and never said it creates a heuristic
router. The picker now always shows with Heuristics as the labelled
default, the quick setup sits under it, and picking another strategy opens
that strategy prefilled with the chosen models

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ui): show the server's reason when a model settings or API key update fails

Both screens replaced the rejection with a fixed message, so a 400 such as
a router routing to itself read only as a generic failure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(router): route Switchyard on the Rust router core

main replaced the Python router core with the Rust one, which left
Switchyard routers registered nowhere. Register them with the other
pre-routing strategies, hand each strategy the endpoint the request came
through, and evict Switchyard routers on reload and delete like the rest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form

Switchyard session state is keyed by a hash of the caller's key hash and session id, and client
session headers no longer reach Switchyard unscoped. The add form carries the quick setup models
into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the
schema reads API-saved configs with nulls and structured decision instructions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(router): make the Switchyard test folder a package

* fix(router): read the Switchyard session key hash only from the proxy's own metadata

* fix(proxy): keep nested nulls as null when masking deployment params

* fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies

* fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
joshua-berri 2026-11-06 12:07:57 -08:00 • committed by GitHub
parent 8013d5b8c1
commit de0b389205
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
61 changed files with 2675 additions and 200 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
.github/workflows/test-unit.yml vendored
View file

@ -1035,9 +1035,9 @@ jobs:
run: |
diff -u model_prices_and_context_window.json litellm/model_prices_and_context_window_backup.json
if [ -z "$RUST_BRIDGE_ARTIFACT" ]; then
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
else
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/*.whl
cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
uv run --no-sync python -c "import importlib.metadata; import litellm.rust_bridge._native; print(importlib.metadata.version('litellm'))"
@ -1294,7 +1294,7 @@ jobs:
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard
- name: Cache Prisma binaries
if: steps.changes.outputs.decision != 'skip'
@ -1497,9 +1497,9 @@ jobs:
RUST_BRIDGE_ARTIFACT: ${{ needs.rust-bridge.outputs.artifact }}
run: |
if [ -z "$RUST_BRIDGE_ARTIFACT" ]; then
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils
else
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
.github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra switchyard --extra saml --extra extra_proxy --extra proxy-runtime --extra utils --no-install-project --no-install-package litellm-proxy
uv pip install --no-deps --python .venv/bin/python rust-bridge-dist/*.whl
cp rust-bridge-dist/litellm/rust_bridge/_native.abi3.so litellm/rust_bridge/_native.abi3.so
fi

6
litellm/litellm_core_utils/sensitive_data_masker.py
View file

@ -117,7 +117,9 @@ class SensitiveDataMasker:
elif key_is_sensitive and isinstance(item, str):
masked_items.append(self.mask_value(item))
else:
masked_items.append(item if isinstance(item, (int, float, bool, str, list)) else str(item))
masked_items.append(
item if item is None or isinstance(item, (int, float, bool, str, list)) else str(item)
)
return masked_items
def mask_dict(
@ -144,7 +146,7 @@ class SensitiveDataMasker:
str_value = str(v) if v is not None else ""
masked_data[k] = self.mask_value(str_value)
else:
masked_data[k] = v if isinstance(v, (int, float, bool, str, list)) else str(v)
masked_data[k] = v if v is None or isinstance(v, (int, float, bool, str, list)) else str(v)
except Exception:
masked_data[k] = "<unable to serialize>"

9
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -795,18 +795,21 @@ class AnthropicMessagesHandler(BaseTranslation):
data: dict, # mutable-ok: the request body the adapter reads
*,
keep_cache_control: bool = False,
keep_tool_errors: bool = False,
) -> ChatCompletionRequest:
"""Translate Anthropic request to OpenAI chat completion format."""
(
chat_completion_compatible_request,
_tool_name_mapping,
) = LiteLLMAnthropicMessagesAdapter(keep_cache_control=keep_cache_control).translate_anthropic_to_openai(
) = LiteLLMAnthropicMessagesAdapter(
keep_cache_control=keep_cache_control, keep_tool_errors=keep_tool_errors
).translate_anthropic_to_openai(
anthropic_message_request=cast(AnthropicMessagesRequest, scannable_tool_result_request(data)),
preserve_midturn_system=True,
)
return chat_completion_compatible_request
def get_structured_messages(self, data: dict) -> list[AllMessageValues] | None:
def get_structured_messages(self, data: dict, *, keep_tool_errors: bool = False) -> list[AllMessageValues] | None:
"""
Convert Anthropic messages request data to OpenAI-spec structured messages.
@ -815,7 +818,7 @@ class AnthropicMessagesHandler(BaseTranslation):
messages: Final = data.get("messages")
if messages is None:
return None
chat_completion_compatible_request: Final = self._translate_to_openai(data)
chat_completion_compatible_request: Final = self._translate_to_openai(data, keep_tool_errors=keep_tool_errors)
result: Final = cast(
list[AllMessageValues],
chat_completion_compatible_request.get("messages", []),

20
litellm/llms/anthropic/messages/adapters/transformation.py
View file

@ -8,7 +8,7 @@ from typing import TYPE_CHECKING, Any, Final, Literal, TypeAlias, TypeVar, cast
from pydantic import JsonValue, TypeAdapter
import litellm
from litellm.litellm_core_utils.core_helpers import normalize_drop_params
from litellm.litellm_core_utils.core_helpers import as_str_mapping, normalize_drop_params
from litellm.litellm_core_utils.hidden_params import get_hidden_params
from litellm.llms.anthropic.messages.convertible_blocks import tool_result_text_part
from litellm.llms.anthropic.messages.utils import (
@ -434,8 +434,9 @@ _CLAUDE_OVER_REASONING_EFFORT_PROVIDERS: Final = frozenset({LlmProviders.KYMA.va
class LiteLLMAnthropicMessagesAdapter:
def __init__(self, *, keep_cache_control: bool = False) -> None:
def __init__(self, *, keep_cache_control: bool = False, keep_tool_errors: bool = False) -> None:
self.keep_cache_control: Final = keep_cache_control
self.keep_tool_errors: Final = keep_tool_errors
### FOR [BETA] `/v1/messages` endpoint support
@ -615,11 +616,7 @@ class LiteLLMAnthropicMessagesAdapter:
self._add_cache_control_if_applicable(content, doc_obj, model)
new_user_content_list.append(doc_obj)
elif content.get("type") == "tool_result":
tool_result = ChatCompletionToolMessage(
role="tool",
tool_call_id=content.get("tool_use_id", ""),
content=self._tool_result_content(content.get("content")),
)
tool_result = self._tool_message(content)
self._add_cache_control_if_applicable(content, tool_result, model)
tool_message_list.append(tool_result)
@ -1432,6 +1429,15 @@ class LiteLLMAnthropicMessagesAdapter:
return None
return anthropic_image_source_to_openai_url(image_source)
def _tool_message(self, block: object) -> ChatCompletionToolMessage:
fields: Final = as_str_mapping(block) or {}
is_error: Final = fields.get("is_error")
tool_call_id: Final = tool_use_id if isinstance(tool_use_id := fields.get("tool_use_id"), str) else ""
content: Final = self._tool_result_content(fields.get("content"))
if self.keep_tool_errors and isinstance(is_error, bool):
return ChatCompletionToolMessage(role="tool", tool_call_id=tool_call_id, content=content, is_error=is_error)
return ChatCompletionToolMessage(role="tool", tool_call_id=tool_call_id, content=content)
def _tool_result_content(self, raw_content: object) -> ToolResultContent:
if isinstance(raw_content, str):
return raw_content

5
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -257,11 +257,12 @@ class BaseTranslation(ABC):
"""
return None
def get_structured_messages(self, data: dict) -> list["AllMessageValues"] | None:
def get_structured_messages(self, data: dict, *, keep_tool_errors: bool = False) -> list["AllMessageValues"] | None:
"""
Convert request data to OpenAI-spec structured messages.
Override in subclasses for format-specific conversion.
Override in subclasses for format-specific conversion. ``keep_tool_errors`` copies a provider's
tool-failure marker onto tool messages as ``is_error``; only for copies that are never sent to a model.
Returns None if no convertible content is found.
"""

2
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -103,7 +103,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
delivers_ended_stream_rewrites = True
assembles_streamed_response = True
def get_structured_messages(self, data: dict) -> list[AllMessageValues] | None:
def get_structured_messages(self, data: dict, *, keep_tool_errors: bool = False) -> list[AllMessageValues] | None:
"""
Convert chat completions request data to OpenAI-spec structured messages.

2
litellm/llms/openai/responses/guardrail_translation/handler.py
View file

@ -777,7 +777,7 @@ class OpenAIResponsesHandler(BaseTranslation):
delivers_ended_stream_rewrites = True
assembles_streamed_response = True
def get_structured_messages(self, data: dict) -> list[AllMessageValues] | None:
def get_structured_messages(self, data: dict, *, keep_tool_errors: bool = False) -> list[AllMessageValues] | None:
"""
Convert Responses API request data to OpenAI-spec structured messages.

4
litellm/proxy/_lazy_openapi_snapshot.json
View file

@ -49950,6 +49950,10 @@
],
"title": "Content"
},
"is_error": {
"title": "Is Error",
"type": "boolean"
},
"role": {
"const": "tool",
"title": "Role",

1
litellm/proxy/management_endpoints/auto_router_endpoints.py
View file

@ -1112,6 +1112,7 @@ def _is_configured_pre_routing_strategy(llm_router: "Router", router_name: str)
router_registry.plugins("complexity"),
router_registry.plugins("adaptive"),
router_registry.plugins("quality"),
router_registry.plugins("switchyard"),
)
)

31
litellm/proxy/management_endpoints/model_management_endpoints.py
View file

@ -142,6 +142,7 @@ from litellm.router_strategy.auto_router.auto_router_model_naming import (
count_capability_routers,
gated_capabilities_of,
is_complexity_router_model,
strategy_router_dependencies,
validate_complexity_router_config_placement,
validate_complexity_router_config_write,
validate_strategy_router_model_write,
@ -355,6 +356,22 @@ async def get_db_model(model_id: str, prisma_client: PrismaClient) -> Deployment
return deployment_pydantic_obj
def _self_routing_violation(
model_name: str | None,
incoming_params: GenericLiteLLMParams | None,
existing_params: GenericLiteLLMParams | None,
) -> str | None:
if not model_name:
return None
effective_params: Final = {
**(_decrypted_litellm_params(existing_params) if existing_params is not None else {}),
**(_dump_without_none(incoming_params) if incoming_params is not None else {}),
}
if any(dependency.model_name == model_name for dependency in strategy_router_dependencies(effective_params)):
return f"A strategy router cannot route to itself: {model_name} is listed as one of its own models"
return None
def _strategy_router_write_violation(
incoming_params: GenericLiteLLMParams | None,
existing_params: GenericLiteLLMParams | None,
@ -374,6 +391,11 @@ def _strategy_router_write_violation(
"""
if incoming_params is None:
return None
from litellm.router_strategy.switchyard_router.config import validate_switchyard_config_write
switchyard_violation: Final = validate_switchyard_config_write(incoming_params.switchyard_router_config)
if switchyard_violation is not None:
return switchyard_violation
config_violation: Final = validate_complexity_router_config_write(
complexity_router_config=(
_effective_complexity_router_config(incoming_params, existing_params)
@ -406,8 +428,11 @@ def _strategy_router_write_violation(
def _raise_on_strategy_router_write_violation(
incoming_params: GenericLiteLLMParams | None,
existing_params: GenericLiteLLMParams | None,
model_name: str | None = None,
) -> None:
violation = _strategy_router_write_violation(incoming_params=incoming_params, existing_params=existing_params)
violation: Final = _self_routing_violation(model_name, incoming_params, existing_params) or (
_strategy_router_write_violation(incoming_params=incoming_params, existing_params=existing_params)
)
if violation is None:
return
raise ProxyException(
@ -1751,6 +1776,7 @@ async def patch_model(
_raise_on_strategy_router_write_violation(
incoming_params=incoming_params,
existing_params=remaining_params,
model_name=patch_data.model_name or db_model.model_name,
)
written_team_id: Final = (
@ -3120,6 +3146,7 @@ async def add_new_model(
_raise_on_strategy_router_write_violation(
incoming_params=model_params.litellm_params,
existing_params=None,
model_name=model_params.model_name,
)
_raise_if_rate_limits_required_but_missing(
@ -3325,6 +3352,7 @@ async def update_model(
_raise_on_strategy_router_write_violation(
incoming_params=model_params.litellm_params,
existing_params=deployment.litellm_params,
model_name=model_params.model_name or deployment.model_name,
)
renamed_to: Final = (
model_params.model_name
@ -4028,6 +4056,7 @@ async def clear_cache() -> ReconcileOutcome:
llm_router.registry().plugins("complexity").pop(model_name, None)
llm_router.registry().plugins("adaptive").pop(model_name, None)
llm_router.registry().plugins("quality").pop(model_name, None)
llm_router.registry().plugins("switchyard").pop(model_name, None)
# Reload only DB models. _add_deployment_locked, not add_deployment: this
# coroutine already holds MODEL_RECONCILE_LOCK and asyncio.Lock is not

10
litellm/router_api.py
View file

@ -65,6 +65,7 @@ if TYPE_CHECKING:
from litellm.router_strategy.budget_limiter import RouterBudgetLimiting
from litellm.router_strategy.complexity_router.complexity_router import ComplexityRouter
from litellm.router_strategy.quality_router.quality_router import QualityRouter
from litellm.router_strategy.switchyard_router.switchyard_router import SwitchyardRouter
from litellm.router_utils.cooldown_cache import CooldownCache
from litellm.router_utils.fallback_event_handlers import AttemptedFallbackTargets as _AttemptedFallbackTargets
from litellm.router_utils.health_state_cache import DeploymentHealthCache
@ -106,6 +107,7 @@ if TYPE_CHECKING:
| dict[str, list[TaggedPreRoutingStrategy[ComplexityRouter]]]
| dict[str, list[TaggedPreRoutingStrategy[QualityRouter]]]
| dict[str, list[TaggedPreRoutingStrategy[AdaptiveRouter]]]
| dict[str, list[TaggedPreRoutingStrategy[SwitchyardRouter]]]
| list[RoutingPlugin]
| RouterBudgetLimiting
| DualCache
@ -171,6 +173,7 @@ PluginKind: TypeAlias = Literal[
"complexity",
"quality",
"adaptive",
"switchyard",
"routing",
"budget_limiter",
"prompt_caching_pins",
@ -1106,6 +1109,11 @@ class RouterRegistry:
self, kind: Literal["adaptive"]
) -> "dict[str, list[TaggedPreRoutingStrategy[AdaptiveRouter]]]": ... # mutable-ok: the live dict
@overload
def plugins(
self, kind: Literal["switchyard"]
) -> "dict[str, list[TaggedPreRoutingStrategy[SwitchyardRouter]]]": ... # mutable-ok: the live dict
@overload
def plugins(self, kind: Literal["routing"]) -> "list[RoutingPlugin]": ... # mutable-ok: the live list
@ -1132,6 +1140,8 @@ class RouterRegistry:
return r._core_quality_routers
if kind == "adaptive":
return r._core_adaptive_routers
if kind == "switchyard":
return r._core_switchyard_routers
if kind == "routing":
return r._core_routing_plugins
if kind == "budget_limiter":

10
litellm/router_rust/core.py
View file

@ -1177,6 +1177,7 @@ _HOSTED_STATE: Final = frozenset(
"_core_auto_routers",
"_core_complexity_routers",
"_core_quality_routers",
"_core_switchyard_routers",
"_core_adaptive_routers",
"_core_routing_plugins",
"_core_deployment_latency_map",
@ -1722,6 +1723,7 @@ class RustCore(
run_plugins=partial(self._run_plugins, request_kwargs, messages)
if self._hosted._core_routing_plugins # pyright: ignore[reportPrivateUsage] # the live plugin list
else None,
call_type=None,
)
def _core_log_retry(
@ -2715,7 +2717,7 @@ class RustCore(
web_search: Final = not plan.sync and self._picks_async() and is_web_search_request(request)
# The generic handler `_route_capability_pin` names: a call only some providers serve
# drops the others, and a Decisions call carrying media skips the deployments refusing it.
handler: Final = plan.host_op.rsplit(".", 1)[-1]
handler: Final = plan.handler
decisions_media: Final = handler in DECISIONS_MEDIA_CALL_TYPES and bool(
request.get("audio") or request.get("videos")
)
@ -3113,6 +3115,7 @@ class RustCore(
specific_deployment: object = _NO_INPUT,
*,
pass_through: bool = False,
call_type: str | None = None,
) -> object:
"""The host's pre-routing of one pick (router_rust.steps.prerouting): the model group to resolve
instead of the step's ``model``, a custom strategy's deployment, or ``None``. ``pass_through``
@ -3166,6 +3169,7 @@ class RustCore(
fallback_depth=depth if isinstance(depth, int) else 0,
router_strategy=normalize_strategy(r.routing_strategy),
pass_through=pass_through,
call_type=call_type,
enable_tag_filtering=r.enable_tag_filtering,
match_any=r.tag_filtering_match_any,
public_router=public if public is not None else self,
@ -5266,7 +5270,9 @@ class _Driver:
async def pre_route(self, step: Mapping[str, object]) -> object:
request: Final = self._request
try:
return await self._core._pre_route(step, request, *self._plan.pick_input(request)) # pyright: ignore[reportPrivateUsage] # see above
return await self._core._pre_route( # pyright: ignore[reportPrivateUsage] # see above
step, request, *self._plan.pick_input(request), call_type=self._plan.handler
)
except Exception as error: # noqa: BLE001 # see select
return self._pick_raised(error, logged=True)

5
litellm/router_rust/invoke/endpoints.py
View file

@ -244,6 +244,11 @@ class Plan:
# the request (`{**data, name: value, **kwargs}`).
named: tuple[str, ...] = ()
@property
def handler(self) -> str:
"""The handler name the host op ends in, which route capabilities and pre-routing strategies key on."""
return self.host_op.rsplit(".", 1)[-1]
@property
def on_its_own_loop(self) -> bool:
"""``function_with_fallbacks`` runs these sync endpoints' whole walk on a private event loop."""

41
litellm/router_rust/steps/prerouting.py
View file

@ -197,9 +197,13 @@ class PreRoutingStrategies:
def quality(self) -> StrategyRegistry:
return self._hosted._core_quality_routers # pyright: ignore[reportPrivateUsage] # see above
@property
def switchyard(self) -> StrategyRegistry:
return self._hosted._core_switchyard_routers # pyright: ignore[reportPrivateUsage] # see above
def all(self) -> tuple[StrategyRegistry, ...]:
"""In the order ``_select_pre_routing_strategy`` searches them."""
return (self.auto, self.complexity, self.adaptive, self.quality)
return (self.switchyard, self.auto, self.complexity, self.adaptive, self.quality)
def any_registered(self) -> bool:
return any(self.all())
@ -211,6 +215,7 @@ class PreRoutingStrategies:
self._hosted._core_quality_routers = {} # pyright: ignore[reportPrivateUsage] # see above
self._hosted._core_complexity_routers = {} # pyright: ignore[reportPrivateUsage] # see above
self._hosted._core_auto_routers = {} # pyright: ignore[reportPrivateUsage] # see above
self._hosted._core_switchyard_routers = {} # pyright: ignore[reportPrivateUsage] # see above
def register(self, deployment: Deployment) -> None:
"""The end of ``_add_deployment``: an auto, complexity or quality marker registers its router
@ -222,6 +227,8 @@ class PreRoutingStrategies:
self._init_complexity_router(deployment)
case "quality":
self._init_quality_router(deployment)
case "switchyard":
self._init_switchyard_router(deployment)
case _:
pass
@ -231,7 +238,7 @@ class PreRoutingStrategies:
return
model_name: Final = deployment.model_name
tags: Final = _deployment_tags(deployment)
for registry in (self.auto, self.complexity, self.quality):
for registry in (self.auto, self.complexity, self.quality, self.switchyard):
_ = _unregister(registry, model_name, tags)
if _unregister(self.adaptive, model_name, tags):
self._sync_adaptive_router_hooks()
@ -463,6 +470,32 @@ class PreRoutingStrategies:
)
self._register(self.quality, deployment, router, "Quality-router")
def _init_switchyard_router(self, deployment: Deployment) -> None:
from litellm.router_strategy.switchyard_router.config import SwitchyardRouterConfig, build_algorithm
from litellm.router_strategy.switchyard_router.switchyard_router import SwitchyardRouter
from litellm.types.decisions import OpenAIDecisionResponse
from litellm.types.utils import ModelResponse
config: Final = SwitchyardRouterConfig.model_validate(deployment.litellm_params.switchyard_router_config)
if deployment.model_name in (config.models.capable, config.models.efficient, config.models.judge):
raise ValueError("Switchyard cannot route to itself")
ctx: Final = self._ctx
async def complete_judge(request: Mapping[str, object]) -> ModelResponse:
result: Final = await ctx.acompletion(**request) # pyright: ignore[reportAttributeAccessIssue] # the core's RoutingContext
if not isinstance(result, ModelResponse):
raise TypeError("Switchyard judge must return a non-streaming completion")
return result
async def decide(request: Mapping[str, object]) -> OpenAIDecisionResponse:
result: Final = await ctx.adecisions(**request) # pyright: ignore[reportAttributeAccessIssue] # see above
if not isinstance(result, OpenAIDecisionResponse):
raise TypeError("Switchyard decision judge must return an OpenAI-shaped decision response")
return result
router: Final = SwitchyardRouter(deployment.model_name, config, build_algorithm(config), complete_judge, decide)
self._register(self.switchyard, deployment, router, "Switchyard")
def _init_adaptive_router(self, deployment: Deployment) -> None:
from litellm.router_strategy.adaptive_router.adaptive_router import AdaptiveRouter
from litellm.router_strategy.adaptive_router.hooks import AdaptiveRouterPostCallHook
@ -741,6 +774,7 @@ class PreRouter:
public_router: object,
run_plugins: Callable[[str], Awaitable[None]] | None = None,
pass_through: bool = False,
call_type: str | None = None,
) -> str | None:
"""The start of ``_async_get_available_deployment``: run the hook, record its pick and pin the
tier's litellm_params. The model group to resolve, or ``None`` to resolve ``model`` itself.
@ -759,6 +793,7 @@ class PreRouter:
match_any=match_any,
public_router=public_router,
run_plugins=run_plugins,
call_type=call_type,
)
routed: Final = response.model if response is not None else model
if response is not None:
@ -782,6 +817,7 @@ class PreRouter:
match_any: bool,
public_router: object,
run_plugins: Callable[[str], Awaitable[None]] | None,
call_type: str | None,
) -> "PreRoutingHookResponse | None":
_restore_client_output_ceiling(request_kwargs)
_record_routing_decision(request_kwargs, None)
@ -861,6 +897,7 @@ class PreRouter:
messages=routing_messages if routing_messages is not None else messages, # pyright: ignore[reportArgumentType] # the request's messages, as given
input=input, # pyright: ignore[reportArgumentType] # the request's input, as given
specific_deployment=False,
call_type=call_type,
)
hooked: Final[PreRoutingHookResponse | None] = routed
traced: Final = (

1
litellm/router_rust/surface/registry.py
View file

@ -658,6 +658,7 @@ class HostRegistry(HostModelInfo):
self._core_complexity_routers: dict[str, list[TaggedPreRoutingStrategy[object]]] = {} # mutable-ok: same
self._core_adaptive_routers: dict[str, list[TaggedPreRoutingStrategy[object]]] = {} # mutable-ok: same
self._core_quality_routers: dict[str, list[TaggedPreRoutingStrategy[object]]] = {} # mutable-ok: same
self._core_switchyard_routers: dict[str, list[TaggedPreRoutingStrategy[object]]] = {} # mutable-ok: same
self.strategies: PreRoutingStrategies | None = None
# The router's optional pre-call checks (deployment budgets, prompt caching pins);
# RustCore installs its ``OptionalChecks`` (litellm/router_rust/steps/callbacks.py).

1
litellm/router_strategy/adaptive_router/adaptive_router.py
View file

@ -184,6 +184,7 @@ class AdaptiveRouter:
messages: list[dict[str, object]] | None = None,
input: str | list | None = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> PreRoutingHookResponse | None:
"""
Plugin entry point invoked by `Router.async_pre_routing_hook` when the

1
litellm/router_strategy/auto_router/auto_router.py
View file

@ -204,6 +204,7 @@ class AutoRouter(CustomLogger):
messages: list[dict[str, object]] | None = None,
input: str | list | None = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> Optional["PreRoutingHookResponse"]:
"""
This hook is called before the routing decision is made.

13
litellm/router_strategy/auto_router/auto_router_model_naming.py
View file

@ -26,7 +26,7 @@ from litellm.router_strategy.complexity_router.config import (
AUTO_ROUTER_MODEL_PREFIX: Final = "auto_router/"
StrategyRouterKind = Literal["semantic", "complexity", "adaptive", "quality"]
StrategyRouterKind: TypeAlias = Literal["semantic", "complexity", "adaptive", "quality", "switchyard"]
StrategyRouterDependencyRole: TypeAlias = Literal[
"tier", "default", "classifier", "embedding", "evaluation", "compactor"
@ -43,6 +43,7 @@ class StrategyRouterDependency:
STRATEGY_ROUTER_PARAM_FIELDS: Final[frozenset[str]] = frozenset(
{
"switchyard_router_config",
"auto_router_config",
"auto_router_config_path",
"auto_router_default_model",
@ -57,6 +58,7 @@ STRATEGY_ROUTER_PARAM_FIELDS: Final[frozenset[str]] = frozenset(
)
_REQUIRED_FIELD_GROUPS: Final[Mapping[StrategyRouterKind, tuple[tuple[str, ...], ...]]] = {
"switchyard": (("switchyard_router_config",),),
"semantic": (
("auto_router_config", "auto_router_config_path"),
("auto_router_default_model",),
@ -78,6 +80,8 @@ def classify_strategy_router_model(model: str) -> StrategyRouterKind | None:
if not model.startswith(AUTO_ROUTER_MODEL_PREFIX):
return None
remainder: Final = model[len(AUTO_ROUTER_MODEL_PREFIX) :]
if remainder.startswith("switchyard"):
return "switchyard"
if remainder.startswith("complexity_router"):
return "complexity"
if remainder.startswith("adaptive_router"):
@ -137,6 +141,13 @@ def strategy_router_dependencies(
kind: Final = classify_strategy_router_model(str(litellm_params.get("model", "")))
if kind is None:
return ()
if kind == "switchyard":
models: Final = _mapping(_mapping(litellm_params.get("switchyard_router_config")).get("models"))
return (
_named(models.get("capable"), "tier")
+ _named(models.get("efficient"), "tier")
+ _named(models.get("judge"), "classifier")
)
if kind == "semantic":
return _named(litellm_params.get("auto_router_default_model"), "default") + _named(
litellm_params.get("auto_router_embedding_model"), "embedding"

1
litellm/router_strategy/complexity_router/complexity_router.py
View file

@ -4490,6 +4490,7 @@ class ComplexityRouter(CustomLogger):
messages: list[dict[str, object]] | None = None,
input: str | list | None = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> PreRoutingHookResponse | None:
"""
Pre-routing hook called before the routing decision.

1
litellm/router_strategy/quality_router/quality_router.py
View file

@ -330,6 +330,7 @@ class QualityRouter(CustomLogger):
messages: list[dict[str, object]] | None = None,
input: str | list | None = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> Optional["PreRoutingHookResponse"]:
"""Try keyword override first; fall back to complexity-tier routing."""
from litellm.types.router import PreRoutingHookResponse

0
litellm/router_strategy/switchyard_router/__init__.py Normal file
View file

154
litellm/router_strategy/switchyard_router/config.py Normal file
View file

@ -0,0 +1,154 @@
from __future__ import annotations
from collections.abc import Mapping
from typing import TYPE_CHECKING, Annotated, Final, Literal, TypeAlias
from pydantic import BaseModel, ConfigDict, Discriminator, Field, Tag, TypeAdapter, model_validator
if TYPE_CHECKING:
from switchyard_rust.libsy import Algorithm, TaskClassifierConfig
class LlmClassifierConfig(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
kind: Literal["llm"] = "llm"
base_threshold: float = Field(default=0.5, ge=0, le=1)
threshold_step: float = Field(default=0.0, ge=0, le=1)
recent_turn_window: int | None = Field(default=None, gt=0)
max_output_tokens: int = Field(default=4096, gt=0)
response_format_type: Literal["json_schema", "json_object"] = "json_schema"
prompt: str | None = None
class DecisionClassifierConfig(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
kind: Literal["decision"]
cutoff: float = Field(default=0.6, ge=0, le=1)
instructions: str | Mapping[str, object] | None = None
evidence: Mapping[str, object] | None = None
_KIND: Final = TypeAdapter(Mapping[str, object])
def _classifier_kind(value: object) -> str:
match value:
case LlmClassifierConfig() | DecisionClassifierConfig():
return value.kind
case Mapping():
return str(_KIND.validate_python(value).get("kind", "llm"))
case _:
return "llm"
ClassifierConfig: TypeAlias = Annotated[
Annotated[LlmClassifierConfig, Tag("llm")] | Annotated[DecisionClassifierConfig, Tag("decision")],
Discriminator(_classifier_kind),
]
class StageOptions(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
picker: Literal["efficient_first", "capable_first"] = "efficient_first"
confidence_threshold: float | None = Field(default=None, ge=0, le=1)
recent_window: int | None = Field(default=None, gt=0)
tool_semantics: Mapping[str, tuple[str, ...]] | None = None
only_on_wrong_signal_escalation: bool = True
class StageModels(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid", str_strip_whitespace=True)
capable: str = Field(min_length=1)
efficient: str = Field(min_length=1)
judge: str | None = Field(default=None, min_length=1)
@model_validator(mode="after")
def reject_recursive_targets(self) -> StageModels:
if any(name.startswith("auto_router/") for name in (self.capable, self.efficient, self.judge or "")):
raise ValueError("Switchyard targets must be concrete model groups, not auto-router markers")
if self.capable == self.efficient:
raise ValueError("The capable and efficient models must be different model groups")
return self
class SwitchyardRouterConfig(BaseModel):
model_config = ConfigDict(frozen=True, extra="forbid")
algorithm: Literal["stage_router"] = "stage_router"
models: StageModels
options: StageOptions = Field(default_factory=StageOptions)
classifier: ClassifierConfig | None = None
@model_validator(mode="after")
def validate_judge(self) -> SwitchyardRouterConfig:
if (self.classifier is None) != (self.models.judge is None):
raise ValueError("A judge model is required exactly when the classifier is enabled")
return self
@property
def confidence_threshold(self) -> float:
if self.options.confidence_threshold is not None:
return self.options.confidence_threshold
return 0.7 if self.classifier is not None else 0.5
def runtime_models(self) -> Mapping[str, tuple[str, ...]]:
return {
"any": (self.models.capable, self.models.efficient),
"capable": (self.models.capable,),
"efficient": (self.models.efficient,),
**({"judge": (self.models.judge,)} if self.models.judge is not None else {}),
}
def _task_classifier(config: SwitchyardRouterConfig) -> TaskClassifierConfig | None:
from switchyard_rust.libsy import TaskClassifierConfig
match config.classifier:
case None:
return None
case LlmClassifierConfig() as llm:
return TaskClassifierConfig(
base_threshold=llm.base_threshold,
threshold_step=llm.threshold_step,
recent_turn_window=llm.recent_turn_window,
max_output_tokens=llm.max_output_tokens,
response_format_type=llm.response_format_type,
prompt=llm.prompt,
)
case DecisionClassifierConfig() as decision:
return TaskClassifierConfig.decision(
cutoff=decision.cutoff,
candidates={"capable": config.models.capable, "efficient": config.models.efficient},
evidence=dict(decision.evidence or {}),
instructions=decision.instructions,
)
def build_algorithm(config: SwitchyardRouterConfig) -> Algorithm:
from switchyard_rust.libsy import LlmFallback, stage_router
classifier: Final = _task_classifier(config)
return stage_router(
picker=config.options.picker,
confidence_threshold=config.confidence_threshold,
recent_window=config.options.recent_window,
tool_semantics=dict(config.options.tool_semantics) if config.options.tool_semantics is not None else None,
only_on_wrong_signal_escalation=config.options.only_on_wrong_signal_escalation,
classifier=LlmFallback(config=classifier) if classifier is not None else None,
)
def validate_switchyard_config_write(config: Mapping[str, object] | None) -> str | None:
if config is None:
return None
try:
build_algorithm(SwitchyardRouterConfig.model_validate(config))
except ImportError:
return "Switchyard requires the optional dependency. Install litellm[switchyard] (nemo-switchyard==0.4.0a1)."
except (ValueError, TypeError) as error:
return f"Invalid switchyard_router_config: {error}"
return None

238
litellm/router_strategy/switchyard_router/switchyard_router.py Normal file
View file

@ -0,0 +1,238 @@
from __future__ import annotations
import hashlib
from collections.abc import Awaitable, Callable, Mapping, Sequence
from typing import TYPE_CHECKING, Final, Literal, TypeAlias, TypeVar
from litellm.constants import INTERNAL_CALL_ORIGIN_METADATA_KEY
from litellm.litellm_core_utils.core_helpers import get_metadata_variable_name_from_kwargs
from litellm.litellm_core_utils.internal_call_metadata import (
effective_turn_off_message_logging,
forwarded_internal_call_metadata,
parent_session_kwargs,
)
from litellm.router_strategy.switchyard_router.config import (
DecisionClassifierConfig,
LlmClassifierConfig,
SwitchyardRouterConfig,
)
from litellm.router_strategy.switchyard_router.translation import (
chat_messages,
decision_request,
decision_response,
judge_request,
judge_response,
mapping,
request_to_switchyard,
)
from litellm.types.decisions import OpenAIDecisionResponse
from litellm.types.router import PreRoutingHookResponse
from litellm.types.utils import AUTOROUTER_CLASSIFIER_CALL_ORIGIN, ModelResponse, StandardLoggingRoutingDecision
if TYPE_CHECKING:
from switchyard_rust.libsy import Algorithm, DecisionCall, ModelCall, RoutingOutcome
JudgeCompletion: TypeAlias = Callable[[Mapping[str, object]], Awaitable[ModelResponse]]
DecisionCompletion: TypeAlias = Callable[[Mapping[str, object]], Awaitable[OpenAIDecisionResponse]]
FallbackReason: TypeAlias = Literal[
"unsupported_request_type",
"unsupported_content",
"libsy_error",
"unexpected_judge_call",
"unexpected_decision_call",
"target_outside_config",
"no_decision",
]
_T = TypeVar("_T")
_SESSION_HEADER: Final = "x-switchyard-session-id"
_FORWARDED_HEADER_PREFIXES: Final = ("x-switchyard-", "x-codex-", "x-nemo-relay-", "x-dynamo-")
_FORWARDED_HEADERS: Final = frozenset(
{"x-session-id", "session-id", "thread-id", "x-task-id", "x-request-id", "x-client-request-id"}
)
async def _resolved(value: _T) -> _T:
return value
_SESSION_ALIASES: Final = frozenset(
{_SESSION_HEADER, "x-claude-code-session-id", "x-nemo-relay-session-id", "x-session-id", "session-id"}
)
_CODEX_TURN_METADATA_HEADER: Final = "x-codex-turn-metadata"
def _forwarded_header(name: str, session_scoped: bool) -> bool:
lowered: Final = name.lower()
if lowered in _SESSION_ALIASES or (lowered == _CODEX_TURN_METADATA_HEADER and not session_scoped):
return False
return lowered in _FORWARDED_HEADERS or lowered.startswith(_FORWARDED_HEADER_PREFIXES)
def _caller_key_hash(request_kwargs: Mapping[str, object]) -> str:
internal: Final = mapping(request_kwargs.get(get_metadata_variable_name_from_kwargs(request_kwargs)) or {})
key: Final = internal.get("user_api_key_hash")
return str(key) if key is not None else ""
def _scoped_session(request_kwargs: Mapping[str, object]) -> str | None:
session: Final = parent_session_kwargs(request_kwargs).get("litellm_session_id")
if not session:
return None
return hashlib.sha256(f"{_caller_key_hash(request_kwargs)}\0{session}".encode()).hexdigest()
def switchyard_headers(request_kwargs: Mapping[str, object]) -> Mapping[str, str]:
proxy_request: Final = mapping(request_kwargs.get("proxy_server_request") or {})
client: Final = mapping(proxy_request.get("headers") or {})
session: Final = _scoped_session(request_kwargs)
forwarded: Final = {
name.lower(): value
for name, value in client.items()
if isinstance(value, str) and _forwarded_header(name, session is not None)
}
return {**forwarded, **({_SESSION_HEADER: session} if session else {})}
class SwitchyardRouter:
def __init__(
self,
model_name: str,
config: SwitchyardRouterConfig,
algorithm: Algorithm,
completion: JudgeCompletion,
decide: DecisionCompletion,
) -> None:
self.model_name = model_name
self.config = config
self.algorithm = algorithm
self.completion = completion
self.decide = decide
def _fallback(
self, reason: FallbackReason, messages: Sequence[Mapping[str, object]] | None
) -> PreRoutingHookResponse:
return PreRoutingHookResponse(
model=self.config.models.capable,
messages=[dict(message) for message in messages] if messages is not None else None,
routing_decision={
"router_type": "switchyard",
"router_model_name": self.model_name,
"routed_model": self.config.models.capable,
"switchyard_algorithm": self.config.algorithm,
"switchyard_fallback_reason": reason,
},
)
def _internal_call(
self,
request: Mapping[str, object],
caller: Mapping[str, object],
bucket: Literal["metadata", "litellm_metadata"],
) -> Mapping[str, object]:
caller_metadata: Final = mapping(caller.get(get_metadata_variable_name_from_kwargs(caller)) or {})
return {
**request,
bucket: {
**forwarded_internal_call_metadata(caller_metadata, AUTOROUTER_CLASSIFIER_CALL_ORIGIN),
INTERNAL_CALL_ORIGIN_METADATA_KEY: AUTOROUTER_CLASSIFIER_CALL_ORIGIN,
},
"proxy_server_request": {"body": request},
"num_retries": 0,
"turn_off_message_logging": effective_turn_off_message_logging(caller),
**parent_session_kwargs(caller),
}
def _decision(
self, pick: str, evidence: object, messages: Sequence[Mapping[str, object]] | None
) -> PreRoutingHookResponse:
decision: Final[StandardLoggingRoutingDecision] = {
"router_type": "switchyard",
"router_model_name": self.model_name,
"routed_model": pick,
"switchyard_algorithm": self.config.algorithm,
"switchyard_evidence": evidence,
}
return PreRoutingHookResponse(
model=pick,
messages=[dict(message) for message in messages] if messages is not None else None,
routing_decision=decision,
)
async def _answer_judge(self, call: ModelCall, caller: Mapping[str, object]) -> None:
try:
converted: Final = judge_response(
await self.completion(
self._internal_call(judge_request(call.request, call.models[0]), caller, "metadata")
)
)
except Exception as error: # noqa: BLE001 # judge/provider failures are inputs to Switchyard's fall-open decision
await call.fail(error)
else:
await call.respond(_resolved(converted))
async def _answer_decision(self, call: DecisionCall, caller: Mapping[str, object]) -> None:
try:
answer: Final = decision_response(
await self.decide(
self._internal_call(decision_request(call.request, call.model), caller, "litellm_metadata")
),
call.request,
)
except Exception as error: # noqa: BLE001 # decision/provider failures are inputs to Switchyard's fall-open decision
await call.fail(error)
else:
await call.respond(_resolved(answer))
def _outcome(
self, outcome: RoutingOutcome, messages: Sequence[Mapping[str, object]] | None
) -> PreRoutingHookResponse:
pick: Final = outcome.selected_model_ids[0]
if pick not in (self.config.models.capable, self.config.models.efficient):
return self._fallback("target_outside_config", messages)
metadata: Final = outcome.metadata
evidence: Final[object] = metadata.evidence if metadata is not None else None
return self._decision(pick, evidence, messages)
async def async_pre_routing_hook(
self,
model: str,
request_kwargs: Mapping[str, object],
messages: Sequence[Mapping[str, object]] | None = None,
input: object = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> PreRoutingHookResponse | None:
from switchyard_rust.libsy import LibsyError, Step
if model != self.model_name:
return None
try:
chat: Final = chat_messages(call_type, model, messages, input, request_kwargs)
request: Final = request_to_switchyard(chat) if chat is not None else None
except (ValueError, KeyError, TypeError):
return self._fallback("unsupported_content", messages)
if request is None:
return self._fallback("unsupported_request_type", messages)
judge: Final = self.config.models.judge
try:
async for step in self.algorithm.run_stream(
request, self.config.runtime_models(), headers=switchyard_headers(request_kwargs)
):
match step:
case Step.CallModel(call) if isinstance(
self.config.classifier, LlmClassifierConfig
) and call.models[:1] == [judge]:
await self._answer_judge(call, request_kwargs)
case Step.CallDecision(call) if (
isinstance(self.config.classifier, DecisionClassifierConfig) and call.model == judge
):
await self._answer_decision(call, request_kwargs)
case Step.CallModel():
return self._fallback("unexpected_judge_call", messages)
case Step.CallDecision():
return self._fallback("unexpected_decision_call", messages)
case Step.Done(outcome):
return self._outcome(outcome, messages)
except LibsyError:
return self._fallback("libsy_error", messages)
return self._fallback("no_decision", messages)

232
litellm/router_strategy/switchyard_router/translation.py Normal file
View file

@ -0,0 +1,232 @@
from __future__ import annotations
import json
from collections.abc import Callable, Mapping, Sequence
from typing import TYPE_CHECKING, Final, Protocol
from pydantic import TypeAdapter
from litellm.types.decisions import (
OpenAIChoiceAnswer,
OpenAIDecisionResponse,
systemone_choice_key,
)
from litellm.types.utils import CallTypes, ModelResponse
if TYPE_CHECKING:
from switchyard_rust.libsy import LlmResponse
_MAPPING: Final = TypeAdapter(Mapping[str, object])
_SEQUENCE: Final = TypeAdapter(tuple[object, ...])
_CALL_TYPES: Final = frozenset(call_type.value for call_type in CallTypes)
def mapping(value: object) -> Mapping[str, object]:
return _MAPPING.validate_python(value)
def sequence(value: object) -> tuple[object, ...]:
return _SEQUENCE.validate_python(value)
def text_content(value: object) -> tuple[Mapping[str, object], ...]:
if value is None:
return ()
if isinstance(value, str):
return ({"type": "text", "text": value},)
blocks: Final = tuple(mapping(block) for block in sequence(value))
if any(block.get("type") != "text" or not isinstance(block.get("text"), str) for block in blocks):
raise ValueError("Switchyard routing supports text content only; media and audio are unsupported")
return tuple({"type": "text", "text": block["text"]} for block in blocks)
def tool_call(value: object) -> Mapping[str, object]:
call: Final = mapping(value)
function: Final = mapping(call.get("function"))
arguments: Final = function.get("arguments")
if call.get("type") != "function" or not isinstance(arguments, str):
raise ValueError("Switchyard requires function calls with JSON arguments")
return {
"type": "tool_call",
"id": call["id"],
"name": function["name"],
"arguments": _MAPPING.validate_json(arguments),
}
def message_to_switchyard(value: object) -> Mapping[str, object]:
message: Final = mapping(value)
if message.get("audio") is not None:
raise ValueError("Switchyard routing does not support audio")
role: Final = message.get("role")
content: Final = text_content(message.get("content"))
if role == "tool":
return {
"role": "tool",
"content": (
{
"type": "tool_result",
"tool_call_id": message["tool_call_id"],
"content": content,
"is_error": message.get("is_error"),
},
),
}
if role not in ("user", "system", "developer", "assistant"):
raise ValueError("Switchyard received an unsupported message role")
calls: Final = tuple(tool_call(call) for call in sequence(message.get("tool_calls") or ()))
return {"role": role, "content": (*content, *calls)}
class ChatConverter(Protocol):
def get_structured_messages(
self,
data: dict[str, object], # mutable-ok: matches the request converters' existing dict parameter
*,
keep_tool_errors: bool = False,
) -> Sequence[object] | None: ...
def _instantiate(converter: Callable[[], ChatConverter]) -> ChatConverter:
return converter()
def chat_messages(
call_type: str | None,
model: str,
messages: Sequence[Mapping[str, object]] | None,
input: object,
request_kwargs: Mapping[str, object],
) -> tuple[Mapping[str, object], ...] | None:
from litellm.llms import load_guardrail_translation_mappings
converter: Final = (
load_guardrail_translation_mappings().get(CallTypes(call_type)) if call_type in _CALL_TYPES else None
)
if converter is None:
return None
request: Final = {
**request_kwargs,
"model": model,
**({"messages": [dict(message) for message in messages]} if messages is not None else {}),
**({"input": input} if input is not None else {}),
}
return tuple(map(mapping, _instantiate(converter).get_structured_messages(request, keep_tool_errors=True) or ()))
def request_to_switchyard(messages: Sequence[Mapping[str, object]]) -> Mapping[str, object]:
if not messages:
raise ValueError("Switchyard requires at least one message")
return {
"model": "auto",
"messages": tuple(message_to_switchyard(message) for message in messages),
"stream": False,
}
def judge_message(value: object) -> Mapping[str, object]:
message: Final = mapping(value)
blocks: Final = text_content(message.get("content"))
return {"role": message["role"], "content": "".join(str(block["text"]) for block in blocks)}
def judge_request(request: Mapping[str, object], model: str) -> Mapping[str, object]:
output: Final = mapping(request.get("output") or {})
sampling: Final = mapping(request.get("sampling") or {})
options: Final = {
"max_tokens": output.get("max_output_tokens"),
"response_format": output.get("response_format"),
"temperature": sampling.get("temperature"),
"top_p": sampling.get("top_p"),
}
return {
"model": model,
"stream": False,
"messages": [
judge_message(message)
for message in (
*sequence(request.get("instructions") or ()),
*sequence(request.get("messages") or ()),
)
],
**{key: value for key, value in options.items() if value is not None},
}
def judge_response(response: ModelResponse) -> LlmResponse.Agg:
from switchyard_rust.libsy import LlmResponse
body: Final = _MAPPING.validate_json(response.model_dump_json(exclude_unset=True))
choices: Final = tuple(mapping(choice) for choice in sequence(body["choices"]))
return LlmResponse.Agg(
{
"id": response.id,
"model": response.model,
"outputs": tuple(message_to_switchyard(choice["message"]) for choice in choices),
}
)
def _decision_text(value: object) -> str:
return value if isinstance(value, str) else json.dumps(value)
def _decision_question(name: str, question: Mapping[str, object]) -> Mapping[str, object]:
kind: Final = mapping(question["kind"])
if kind.get("type") != "choice":
raise ValueError("Switchyard decision routing asks choice questions only")
return {
"type": "choice",
"name": name,
"instructions": _decision_text(question.get("instructions")),
"choices": [
{
"value": option["id"],
**({"description": _decision_text(option["description"])} if option.get("description") else {}),
}
for option in map(mapping, sequence(mapping(kind["data"])["options"]))
],
}
def decision_request(request: Mapping[str, object], model: str) -> Mapping[str, object]:
questions: Final = mapping(request["questions"])
return {
"model": model,
"input": _decision_text(request["context"]),
"questions": [_decision_question(name, mapping(question)) for name, question in questions.items()],
}
def _choice_value(answer: object) -> Mapping[str, object]:
if not isinstance(answer, OpenAIChoiceAnswer):
raise TypeError("The decision model refused or returned a non-choice answer")
return {
"type": "choice",
"data": {
"selected": systemone_choice_key(answer.choice),
"probabilities": {systemone_choice_key(p.value): p.probability for p in answer.probabilities},
},
}
def decision_response(response: OpenAIDecisionResponse, request: Mapping[str, object]) -> Mapping[str, object]:
names: Final = tuple(mapping(request["questions"]))
if len(response.answers) != len(names):
raise ValueError("The decision model did not answer every question")
return {
"id": None,
"model": response.model,
"answers": {
answer.name or name: {
"value": _choice_value(answer),
"provider_confidence": getattr(answer, "confidence", None),
}
for name, answer in zip(names, response.answers, strict=True)
},
"usage": {
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"total_tokens": response.usage.total_tokens,
},
}

1
litellm/types/litellm_params.py
View file

@ -207,6 +207,7 @@ class DeploymentOptions:
@dataclass(frozen=True, slots=True, kw_only=True)
class SpecializedRouterOptions:
switchyard_router_config: Mapping[str, object] | None = None
auto_router_config_path: str | None = None
auto_router_config: str | None = None
auto_router_default_model: str | None = None

1
litellm/types/llms/openai.py
View file

@ -957,6 +957,7 @@ class ChatCompletionToolMessage(TypedDict):
role: Literal["tool"]
content: str | Iterable[ToolMessageContentPart] # writable-ok: Pydantic warns on ReadOnly TypedDict fields
tool_call_id: str
is_error: NotRequired[bool] # writable-ok: Pydantic warns on ReadOnly TypedDict fields
class ChatCompletionFunctionMessage(TypedDict):

2
litellm/types/router.py
View file

@ -659,6 +659,7 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
tag_regex: list[str] | None = None
# auto-router params
switchyard_router_config: Mapping[str, object] | None = None
auto_router_config_path: str | None = None
auto_router_config: str | None = None
auto_router_default_model: str | None = None
@ -1461,6 +1462,7 @@ class PreRoutingStrategy(Protocol):
messages: list[dict[str, Any]] | None = None,
input: "str | list[Any] | None" = None,
specific_deployment: bool | None = False,
call_type: str | None = None,
) -> "PreRoutingHookResponse | None": ...

9
litellm/types/utils.py
View file

@ -3382,7 +3382,10 @@ class StandardLoggingRoutingDecision(TypedDict, total=False):
"""Per-request provenance for a pre-routing strategy (auto-router) decision."""
router_model_name: str
router_type: ReadOnly[Literal["complexity", "adaptive", "quality", "semantic"]]
router_type: ReadOnly[Literal["complexity", "adaptive", "quality", "semantic", "switchyard"]]
switchyard_algorithm: ReadOnly[str]
switchyard_evidence: ReadOnly[object]
switchyard_fallback_reason: ReadOnly[str]
router_config_id: ReadOnly[str]
router_config_updated_at: ReadOnly[str]
router_config_fingerprint: ReadOnly[str]
@ -3443,10 +3446,12 @@ class StandardLoggingKubernetesPodRouting(TypedDict):
# so a redacted row stays explainable. `test_every_routing_decision_field_is_classified`
# fails if a field is added to the record without being placed in one set or the other.
PROMPT_QUOTING_ROUTING_DECISION_FIELDS: frozenset[str] = frozenset(
{"signals", "matched_keyword", "escalation_keyword", "classifier_crux"}
{"signals", "matched_keyword", "escalation_keyword", "classifier_crux", "switchyard_evidence"}
)
DERIVED_ROUTING_DECISION_FIELDS: Final[frozenset[str]] = frozenset(
{
"switchyard_algorithm",
"switchyard_fallback_reason",
"router_model_name",
"router_type",
"router_config_id",

2
pyproject.toml
View file

@ -54,6 +54,7 @@ Documentation = "https://docs.litellm.ai"
# sub-packages and the opentelemetry trio move in lockstep, and grpcio is
# supply-chain-pinned to a vetted, aged release.
[project.optional-dependencies]
switchyard = ["nemo-switchyard==0.4.0a1"]
proxy = [
"gunicorn>=23.0.0,<24.0",
"uvicorn>=0.33.0,<1.0",
@ -268,6 +269,7 @@ proxy-dev = [
"opentelemetry-instrumentation-fastapi==0.66b1",
"azure-identity==1.25.2",
"a2a-sdk==1.1.0",
"nemo-switchyard==0.4.0a1",
"backports-zstd==1.8.0; python_version < '3.14'",
"httptools>=0.7.1,<0.8",
"semver>=3.0.4,<4",

38
tests/unit/llms/anthropic/chat/guardrail_translation/test_anthropic_guardrail_handler.py
View file

@ -3458,3 +3458,41 @@ def test_reading_a_dropped_budget_for_a_guardrail_logs_no_drop_warning(caplog: p
assert messages == [{"role": "user", "content": "hi"}]
assert not [record for record in caplog.records if "Dropping" in record.getMessage()]
FAILED_TOOL_RESULT_REQUEST: Final = {
"model": "claude-sonnet",
"max_tokens": 16,
"messages": [
{"role": "user", "content": "Rename the helper"},
{
"role": "assistant",
"content": [{"type": "tool_use", "id": "edit0", "name": "Edit", "input": {"file_path": "a.py"}}],
},
{
"role": "user",
"content": [
{"type": "tool_result", "tool_use_id": "edit0", "content": "String not found", "is_error": True}
],
},
],
}
@pytest.mark.parametrize("keep_tool_errors", [True, False])
def test_structured_messages_carry_tool_failures_only_when_asked(keep_tool_errors: bool) -> None:
messages: Final = AnthropicMessagesHandler().get_structured_messages(
FAILED_TOOL_RESULT_REQUEST, keep_tool_errors=keep_tool_errors
)
assert messages is not None
(tool_message,) = [message for message in messages if message["role"] == "tool"]
assert tool_message.get("is_error") is (True if keep_tool_errors else None)
def test_request_sent_to_a_model_never_carries_tool_failure_marker() -> None:
from litellm.llms.anthropic.messages.adapters.transformation import LiteLLMAnthropicMessagesAdapter
request, _ = LiteLLMAnthropicMessagesAdapter().translate_anthropic_to_openai(
anthropic_message_request=FAILED_TOOL_RESULT_REQUEST # pyright: ignore[reportArgumentType] # a JSON request in the Anthropic shape
)
assert "is_error" not in json.dumps(request["messages"], default=str)

30
tests/unit/proxy/common_utils/test_openai_endpoint_utils.py
View file

@ -77,9 +77,7 @@ from litellm.proxy.common_utils.openai_endpoint_utils import (
),
],
)
def test_remove_sensitive_info_from_deployment(
model_config: dict, expected_config: dict
):
def test_remove_sensitive_info_from_deployment(model_config: dict, expected_config: dict):
sanitized_config = remove_sensitive_info_from_deployment(model_config)
assert sanitized_config == expected_config
@ -110,10 +108,7 @@ def test_remove_sensitive_info_from_deployment_with_excluded_keys():
sanitized_config = remove_sensitive_info_from_deployment(
copy.deepcopy(base_config), excluded_keys={"litellm_credentials_name"}
)
assert (
sanitized_config["litellm_params"]["litellm_credentials_name"]
== "my-credential-name"
)
assert sanitized_config["litellm_params"]["litellm_credentials_name"] == "my-credential-name"
# access_token should still be masked (not in excluded_keys)
assert sanitized_config["litellm_params"]["access_token"] != "token-12345"
@ -121,3 +116,24 @@ def test_remove_sensitive_info_from_deployment_with_excluded_keys():
# api_key should still be removed (popped) regardless of excluded_keys
assert "api_key" not in sanitized_config["litellm_params"]
def test_remove_sensitive_info_from_deployment_keeps_nested_nulls():
router_config = {
"models": {"capable": "strong", "efficient": "fast", "judge": None},
"options": {"confidence_threshold": None, "tool_semantics": None},
"fallbacks": [None, "fast"],
}
deployment = {
"model_name": "route",
"litellm_params": {
"model": "auto_router/switchyard",
"api_key": "sk-sensitive-key-123",
"switchyard_router_config": copy.deepcopy(router_config),
},
}
sanitized = remove_sensitive_info_from_deployment(deployment)
assert "api_key" not in sanitized["litellm_params"]
assert sanitized["litellm_params"]["switchyard_router_config"] == router_config

117
tests/unit/proxy/management_endpoints/test_model_management_endpoints.py
View file

@ -1085,7 +1085,7 @@ class TestClearCache:
Test that clear_cache successfully clears router model caches and reloads models.
"""
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = ["openai/gpt-4o", "openai/gpt-4o-mini"]
@ -1125,7 +1125,7 @@ class TestClearCache:
# entries are auto_router/* deployments (so their router-map entries should be
# cleared for reload); the config-defined router is preserved.
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = [
{
@ -1209,7 +1209,7 @@ class TestClearCache:
)
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = [
{
@ -1270,7 +1270,7 @@ class TestClearCachePreservesConfigRouters:
)
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = [
{
@ -1315,7 +1315,7 @@ class TestClearCachePreservesConfigRouters:
from litellm.proxy.management_endpoints.model_management_endpoints import clear_cache
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = [
{
@ -1346,15 +1346,15 @@ class TestClearCachePreservesConfigRouters:
assert "shared-name" in plugins["complexity"]
@pytest.mark.asyncio
async def test_db_quality_and_adaptive_routers_are_evicted(self):
"""The auto_router/ prefix also covers quality_router/ and adaptive_router/. Their
async def test_db_quality_adaptive_and_switchyard_routers_are_evicted(self):
"""The auto_router/ prefix also covers quality_router/, adaptive_router/ and switchyard. Their
registry entries must be popped too, or reload's init raises 'already exists'
(quality) or leaves a stale entry (adaptive).
"""
from litellm.proxy.management_endpoints.model_management_endpoints import clear_cache
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.registry.return_value.all.return_value = [
{
@ -1367,11 +1367,17 @@ class TestClearCachePreservesConfigRouters:
"model_info": {"id": "db-a", "db_model": True},
"litellm_params": {"model": "auto_router/adaptive_router/a1"},
},
{
"model_name": "s1",
"model_info": {"id": "db-s", "db_model": True},
"litellm_params": {"model": "auto_router/switchyard"},
},
]
plugins["auto"] = {}
plugins["complexity"] = {}
plugins["quality"] = {"q1": MagicMock()}
plugins["adaptive"] = {"a1": MagicMock()}
plugins["switchyard"] = {"s1": MagicMock()}
mock_config = MagicMock()
mock_config.add_deployment_locked = AsyncMock(
@ -1389,6 +1395,7 @@ class TestClearCachePreservesConfigRouters:
assert "q1" not in plugins["quality"]
assert "a1" not in plugins["adaptive"]
assert "s1" not in plugins["switchyard"]
class TestDeleteModelClearsRouterRegistry:
@ -1503,7 +1510,7 @@ class TestDeleteModelClearsRouterRegistry:
mock_prisma.db.litellm_proxymodeltable.delete = AsyncMock(return_value=db_row)
mock_router = MagicMock()
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}}
plugins: dict[str, dict] = {"auto": {}, "complexity": {}, "adaptive": {}, "quality": {}, "switchyard": {}}
mock_router.registry.return_value.plugins.side_effect = plugins.__getitem__
mock_router.apply.return_value = ApplyResult(
version=1,
@ -11431,6 +11438,98 @@ async def test_add_new_model_reports_a_database_outage_during_the_insert_as_503(
assert (exc_info.value.code, exc_info.value.type) == ("503", "no_db_connection")
@pytest.mark.parametrize("invalid", [
{"algorithm": "random"},
{"classifier": {}},
{"options": {"capable_system_prompt": "ignored"}},
])
def test_switchyard_write_rejects_unloadable_configuration(invalid: dict[str, object]) -> None:
from litellm.proxy.management_endpoints.model_management_endpoints import _strategy_router_write_violation
from litellm.types.router import updateLiteLLMParams
violation: Final = _strategy_router_write_violation(
incoming_params=updateLiteLLMParams(
model="auto_router/switchyard",
switchyard_router_config={"models": {"capable": "strong", "efficient": "fast"}, **invalid},
),
existing_params=None,
)
assert violation is not None
assert "Invalid switchyard_router_config" in violation
SWITCHYARD_PARAMS: Final = {
"model": "auto_router/switchyard",
"switchyard_router_config": {"models": {"capable": "strong", "efficient": "fast"}},
}
@pytest.mark.parametrize(
("model_name", "incoming", "existing"),
[
("strong", SWITCHYARD_PARAMS, None),
("fast", {"model": "auto_router/switchyard"}, SWITCHYARD_PARAMS),
("fast", None, SWITCHYARD_PARAMS),
(
"smart",
{
"model": "auto_router/complexity_router",
"complexity_router_config": {"tiers": {"SIMPLE": "smart", "COMPLEX": "strong"}},
},
None,
),
],
)
def test_strategy_router_that_lists_itself_is_rejected_before_saving(
model_name: str, incoming: dict[str, object] | None, existing: dict[str, object] | None
) -> None:
from litellm.proxy._types import ProxyException
from litellm.proxy.management_endpoints.model_management_endpoints import (
_raise_on_strategy_router_write_violation,
)
from litellm.types.router import updateLiteLLMParams
with pytest.raises(ProxyException) as exc_info:
_raise_on_strategy_router_write_violation(
incoming_params=updateLiteLLMParams(**incoming) if incoming is not None else None,
existing_params=updateLiteLLMParams(**existing) if existing is not None else None,
model_name=model_name,
)
assert exc_info.value.code == "400"
assert model_name in exc_info.value.message
def test_strategy_router_with_other_targets_saves() -> None:
from litellm.proxy.management_endpoints.model_management_endpoints import _self_routing_violation
from litellm.types.router import updateLiteLLMParams
assert _self_routing_violation("route", updateLiteLLMParams(**SWITCHYARD_PARAMS), None) is None
def test_self_routing_is_caught_when_the_stored_model_is_encrypted(monkeypatch: pytest.MonkeyPatch) -> None:
from litellm.proxy._types import ProxyException
from litellm.proxy.common_utils.encrypt_decrypt_utils import encrypt_value_helper
from litellm.proxy.management_endpoints.model_management_endpoints import (
_raise_on_strategy_router_write_violation,
)
from litellm.types.router import updateLiteLLMParams
monkeypatch.setenv("LITELLM_SALT_KEY", "sk-test-salt-key-for-encryption")
stored: Final = updateLiteLLMParams(
**{**SWITCHYARD_PARAMS, "model": encrypt_value_helper("auto_router/switchyard")}
)
assert stored.model != "auto_router/switchyard"
with pytest.raises(ProxyException) as exc_info:
_raise_on_strategy_router_write_violation(
incoming_params=updateLiteLLMParams(
switchyard_router_config={"models": {"capable": "route", "efficient": "fast"}}
),
existing_params=stored,
model_name="route",
)
assert exc_info.value.code == "400"
@pytest.mark.asyncio
async def test_rename_rewrites_only_in_a_transaction_holding_every_locked_teams_advisory_lock():
"""A team that starts listing the old name between the advisory snapshot and the row lock must not be

0
tests/unit/router_strategy/switchyard_router/__init__.py Normal file
View file

100
tests/unit/router_strategy/switchyard_router/test_config.py Normal file
View file

@ -0,0 +1,100 @@
from typing import Final
import pytest
from pydantic import ValidationError
from litellm.router_strategy.switchyard_router.config import SwitchyardRouterConfig, validate_switchyard_config_write
@pytest.mark.parametrize("classifier", [None, {}])
def test_default_confidence_depends_on_classifier(classifier: dict[str, object] | None) -> None:
config: Final = SwitchyardRouterConfig.model_validate(
{
"models": {
"capable": "strong",
"efficient": "fast",
**({"judge": "judge"} if classifier is not None else {}),
},
"classifier": classifier,
}
)
assert config.confidence_threshold == (0.7 if classifier is not None else 0.5)
@pytest.mark.parametrize("classifier", [None, {}])
def test_explicit_confidence_is_preserved(classifier: dict[str, object] | None) -> None:
config: Final = SwitchyardRouterConfig.model_validate(
{
"models": {
"capable": "strong",
"efficient": "fast",
**({"judge": "judge"} if classifier is not None else {}),
},
"classifier": classifier,
"options": {"confidence_threshold": 0.63, "picker": "capable_first"},
}
)
assert config.confidence_threshold == 0.63
@pytest.mark.parametrize(
"option", ["escalation_note", "deescalation_note", "capable_system_prompt", "efficient_system_prompt", "unknown"]
)
def test_request_rewrite_and_unknown_options_are_rejected(option: str) -> None:
with pytest.raises(ValidationError, match=option):
SwitchyardRouterConfig.model_validate(
{"models": {"capable": "strong", "efficient": "fast"}, "options": {option: "ignored"}}
)
@pytest.mark.parametrize(
"bad",
[
{"algorithm": "random"},
{"classifier": {}},
{"options": {"confidence_threshold": 2}},
{"models": {"capable": "", "efficient": "fast"}},
{"models": {"capable": "same", "efficient": "same"}},
{"models": {"capable": "strong", "efficient": "fast", "judge": "j"}, "classifier": {"session_affinity": True}},
{"models": {"capable": "strong", "efficient": "fast", "judge": "judge"}},
{"models": {"capable": "strong", "efficient": "fast", "judge": "j"}, "classifier": {"kind": "other"}},
{
"models": {"capable": "strong", "efficient": "fast", "judge": "j"},
"classifier": {"kind": "decision", "cutoff": 1.5},
},
{
"models": {"capable": "strong", "efficient": "fast", "judge": "j"},
"classifier": {"kind": "decision", "base_threshold": 0.5},
},
{
"models": {"capable": "strong", "efficient": "fast", "judge": "j"},
"classifier": {"kind": "llm", "cutoff": 0.5},
},
],
)
def test_invalid_config_cannot_be_saved(bad: dict[str, object]) -> None:
violation: Final = validate_switchyard_config_write({"models": {"capable": "strong", "efficient": "fast"}, **bad})
assert violation is not None
assert "Invalid switchyard_router_config" in violation
def test_missing_package_rejects_save_with_install_instructions() -> None:
import sys
from unittest.mock import patch
with patch.dict(sys.modules, {"switchyard_rust.libsy": None}):
violation: Final = validate_switchyard_config_write({"models": {"capable": "strong", "efficient": "fast"}})
assert violation is not None
assert "litellm[switchyard]" in violation
@pytest.mark.parametrize(
"classifier", [{"kind": "decision"}, {"kind": "decision", "cutoff": 0.4, "evidence": {"a": 1}}]
)
def test_decision_classifier_config_can_be_saved(classifier: dict[str, object]) -> None:
assert (
validate_switchyard_config_write(
{"models": {"capable": "strong", "efficient": "fast", "judge": "jev"}, "classifier": classifier}
)
is None
)

438
tests/unit/router_strategy/switchyard_router/test_switchyard_router.py Normal file
View file

@ -0,0 +1,438 @@
import asyncio
import json
from itertools import chain
from collections.abc import AsyncIterator, Mapping
from typing import Final
import pytest
import litellm
from litellm.litellm_core_utils.core_helpers import get_metadata_variable_name_from_kwargs
from litellm.integrations.custom_logger import CustomLogger
from litellm.router_api import ChangeSet
from litellm.router_strategy.switchyard_router.config import SwitchyardRouterConfig, build_algorithm
from litellm.router_strategy.switchyard_router.switchyard_router import SwitchyardRouter
from litellm.types.decisions import OpenAIDecisionResponse
from litellm.types.router import Deployment
from litellm.types.utils import ModelResponse
libsy = pytest.importorskip("switchyard_rust.libsy")
HELLO: Final = [{"role": "user", "content": "Hello"}]
async def unexpected_judge(request: Mapping[str, object]) -> ModelResponse:
raise AssertionError("This route must not call an LLM judge")
async def unexpected_decision(request: Mapping[str, object]) -> OpenAIDecisionResponse:
raise AssertionError("This route must not call a decision model")
def config(
*, classifier: Mapping[str, object] | None = None, picker: str = "efficient_first", threshold: float | None = None
) -> SwitchyardRouterConfig:
return SwitchyardRouterConfig.model_validate(
{
"models": {
"capable": "strong",
"efficient": "fast",
**({"judge": "judge"} if classifier is not None else {}),
},
"options": {"picker": picker, **({"confidence_threshold": threshold} if threshold is not None else {})},
**({"classifier": classifier} if classifier is not None else {}),
}
)
def router(
route_config: SwitchyardRouterConfig,
*,
completion=unexpected_judge,
decide=unexpected_decision,
algorithm: object = None,
) -> SwitchyardRouter:
return SwitchyardRouter("route", route_config, algorithm or build_algorithm(route_config), completion, decide)
async def route(
route_config: SwitchyardRouterConfig,
messages: object,
*,
call_type: str | None = "acompletion",
request_kwargs: Mapping[str, object] | None = None,
**router_kwargs: object,
) -> tuple[str, Mapping[str, object]]:
result: Final = await router(route_config, **router_kwargs).async_pre_routing_hook( # pyright: ignore[reportArgumentType] # test doubles
"route", request_kwargs or {}, messages, call_type=call_type
)
assert result is not None
assert result.routing_decision is not None
return result.model, result.routing_decision
def anthropic_turn(call_id: str, tool: str, result: str, marked: bool) -> tuple[dict[str, object], ...]:
error: Final = {"is_error": True} if marked else {}
return (
{
"role": "assistant",
"content": [
{"type": "thinking", "thinking": "Next step", "signature": "sig"},
{"type": "tool_use", "id": call_id, "name": tool, "input": {"file_path": "a.py"}},
],
},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": call_id, "content": result, **error}]},
)
def responses_turn(call_id: str, tool: str, result: str, marked: bool) -> tuple[dict[str, object], ...]:
return (
{"type": "function_call", "call_id": call_id, "name": tool, "arguments": '{"file_path":"a.py"}'},
{"type": "function_call_output", "call_id": call_id, "output": result},
)
def chat_turn(call_id: str, tool: str, result: str, marked: bool) -> tuple[dict[str, object], ...]:
call: Final = {"id": call_id, "type": "function", "function": {"name": tool, "arguments": '{"file_path":"a.py"}'}}
return (
{"role": "assistant", "content": None, "tool_calls": [call]},
{"role": "tool", "tool_call_id": call_id, "content": result},
)
TURN_FORMATS: Final = {"anthropic_messages": anthropic_turn, "aresponses": responses_turn, "acompletion": chat_turn}
def transcript(endpoint: str, tool: str, results: tuple[str, ...], *, marked: bool = False) -> list[dict[str, object]]:
turn: Final = TURN_FORMATS[endpoint]
turns: Final = chain.from_iterable(
turn(f"call{index}", tool, result, marked) for index, result in enumerate(results)
)
return [{"role": "user", "content": "Fix the helper"}, *turns]
def llm_router(
*, route_config: Mapping[str, object], judge_fails: bool = False, num_retries: int = 0
) -> litellm.Router:
def deployment(name: str, reply: object) -> Mapping[str, object]:
return {
"model_name": name,
"litellm_params": {"model": f"openai/{name}", "api_key": "k", "mock_response": reply},
}
return litellm.Router(
num_retries=num_retries,
model_list=[
deployment("strong", "strong"),
deployment("fast", "fast"),
*((deployment("judge", Exception("judge down")),) if judge_fails else ()),
switchyard_deployment(route_config),
],
)
def switchyard_deployment(route_config: Mapping[str, object]) -> dict[str, object]:
return {
"model_name": "route",
"litellm_params": {"model": "auto_router/switchyard", "switchyard_router_config": route_config},
"model_info": {"id": "route-id"},
}
@pytest.mark.asyncio
async def test_critical_tool_error_selects_capable_without_rewriting_messages() -> None:
messages: Final = transcript("acompletion", "bash", ("CUDA out of memory",))
result: Final = await router(config()).async_pre_routing_hook("route", {}, messages, call_type="acompletion")
assert result is not None
assert result.model == "strong"
assert result.messages == messages
async def routed_reply(proxy: litellm.Router, endpoint: str, edits: tuple[str, ...]) -> object:
match endpoint:
case "anthropic_messages":
message: Final = await proxy.aanthropic_messages(
model="route", messages=transcript(endpoint, "Edit", edits), max_tokens=16
)
return message["content"][0]["text"] # pyright: ignore[reportIndexIssue] # non-streaming response
case "aresponses":
response: Final = await proxy.aresponses(model="route", input=transcript(endpoint, "Edit", edits))
return response.output[0].content[0].text # pyright: ignore[reportAttributeAccessIssue] # non-streaming response
case _:
chat: Final = await proxy.acompletion(model="route", messages=transcript(endpoint, "Edit", edits))
return chat.choices[0].message.content # pyright: ignore[reportAttributeAccessIssue] # non-streaming response
@pytest.mark.asyncio
@pytest.mark.parametrize("endpoint", ["acompletion", "anthropic_messages", "aresponses"])
async def test_router_routes_agent_traffic_from_every_endpoint(endpoint: str) -> None:
proxy: Final = llm_router(route_config={"models": {"capable": "strong", "efficient": "fast"}})
assert await routed_reply(proxy, endpoint, ("Edit applied",) * 3) == "fast"
@pytest.mark.asyncio
async def test_router_follows_an_edited_switchyard_router_and_forgets_a_deleted_one() -> None:
proxy: Final = llm_router(route_config={"models": {"capable": "strong", "efficient": "fast"}})
assert await routed_reply(proxy, "acompletion", ("Edit applied",) * 3) == "fast"
swapped: Final = switchyard_deployment({"models": {"capable": "fast", "efficient": "strong"}})
_ = proxy.apply(ChangeSet(upsert=[Deployment(**swapped)]))
assert await routed_reply(proxy, "acompletion", ("Edit applied",) * 3) == "strong"
_ = proxy.apply(ChangeSet(delete=["route-id"]))
with pytest.raises(litellm.BadRequestError):
await proxy.acompletion(model="route", messages=HELLO)
@pytest.mark.asyncio
@pytest.mark.parametrize(("marked", "expected"), [(True, "strong"), (False, "fast")])
async def test_anthropic_tool_failure_marker_reaches_switchyard(marked: bool, expected: str) -> None:
failures: Final = transcript("anthropic_messages", "Edit", ("String to replace not found.",) * 4, marked=marked)
model, decision = await route(config(), failures, call_type="anthropic_messages")
assert model == expected
assert "switchyard_fallback_reason" not in decision
@pytest.mark.asyncio
@pytest.mark.parametrize(
("request_kwargs", "expected"),
[
({"litellm_session_id": "s1"}, ["strong", "strong", "strong", "fast"]),
(
{"proxy_server_request": {"headers": {"X-Session-Id": "s1", "session-id": "s1"}}},
["strong", "fast", "fast", "fast"],
),
({}, ["strong", "fast", "fast", "fast"]),
],
)
async def test_capable_hold_follows_the_session(request_kwargs: Mapping[str, object], expected: list[str]) -> None:
switchyard: Final = router(config())
turns: Final = (("CUDA out of memory",), *((("All tests passed",) * count) for count in (2, 3, 4)))
picks: Final = [await pick(switchyard, request_kwargs, results) for results in turns]
assert picks == expected
async def pick(switchyard: SwitchyardRouter, request_kwargs: Mapping[str, object], results: tuple[str, ...]) -> str:
result: Final = await switchyard.async_pre_routing_hook(
"route", request_kwargs, transcript("acompletion", "bash", results), call_type="acompletion"
)
assert result is not None
return result.model
@pytest.mark.asyncio
async def test_a_session_hold_never_crosses_api_keys() -> None:
switchyard: Final = router(config())
owner: Final = {"litellm_session_id": "shared", "metadata": {"user_api_key_hash": "owner"}}
other: Final = {"litellm_session_id": "shared", "metadata": {"user_api_key_hash": "other"}}
assert await pick(switchyard, owner, ("CUDA out of memory",)) == "strong"
assert await pick(switchyard, other, ("All tests passed",) * 2) == "fast"
assert await pick(switchyard, owner, ("All tests passed",) * 2) == "strong"
@pytest.mark.asyncio
async def test_a_client_metadata_key_hash_cannot_join_another_keys_session() -> None:
switchyard: Final = router(config())
owner: Final = {"litellm_session_id": "shared", "litellm_metadata": {"user_api_key_hash": "owner"}}
spoofer: Final = {
"litellm_session_id": "shared",
"metadata": {"user_api_key_hash": "owner"},
"litellm_metadata": {"user_api_key_hash": "spoofer"},
}
assert await pick(switchyard, owner, ("CUDA out of memory",)) == "strong"
assert await pick(switchyard, spoofer, ("All tests passed",) * 2) == "fast"
@pytest.mark.asyncio
@pytest.mark.parametrize(("p_solve", "expected"), [(0.0, "strong"), (1.0, "fast")])
async def test_llm_judge_verdict_selects_the_target(p_solve: float, expected: str) -> None:
async def judge(request: Mapping[str, object]) -> ModelResponse:
assert "autorouter_classifier" in str(request["metadata"])
verdict: Final = {"crux": "c", "primary_rule": "none", "capability_boundary": "unmatched", "p_solve": p_solve}
return ModelResponse(choices=[{"message": {"role": "assistant", "content": json.dumps(verdict)}}])
model, _ = await route(config(classifier={}), HELLO, completion=judge)
assert model == expected
def decision_answer(advantage: float) -> OpenAIDecisionResponse:
return OpenAIDecisionResponse.model_validate(
{
"model": "jev",
"answers": [
{
"type": "choice",
"name": "route",
"choice": "advantage",
"probabilities": [
{"value": "advantage", "probability": advantage},
{"value": "no_advantage", "probability": 1 - advantage},
],
}
],
"usage": {"input_tokens": 1, "output_tokens": 0, "total_tokens": 1},
}
)
@pytest.mark.asyncio
@pytest.mark.parametrize("parent_bucket", ["metadata", "litellm_metadata"])
@pytest.mark.parametrize("classifier", [{}, {"kind": "decision"}])
async def test_judge_calls_carry_the_callers_tags_and_origin_where_their_endpoint_reads_them(
parent_bucket: str, classifier: Mapping[str, object]
) -> None:
seen: Final[list[Mapping[str, object]]] = [] # mutable-ok: collects the judge request
async def judge(request: Mapping[str, object]) -> ModelResponse:
seen.append(request)
verdict: Final = {"crux": "c", "primary_rule": "none", "capability_boundary": "unmatched", "p_solve": 1.0}
return ModelResponse(choices=[{"message": {"role": "assistant", "content": json.dumps(verdict)}}])
async def decide(request: Mapping[str, object]) -> OpenAIDecisionResponse:
seen.append(request)
return decision_answer(0.0)
caller: Final = {parent_bucket: {"tags": ["eu-only"], "inherited_tags": ["team-eu"]}}
await route(
config(classifier=classifier, threshold=1.0), HELLO, completion=judge, decide=decide, request_kwargs=caller
)
(request,) = seen
read_by_endpoint: Final = "litellm_metadata" if classifier else "metadata"
assert set(request) & {"metadata", "litellm_metadata"} == {read_by_endpoint}
metadata: Final = request[read_by_endpoint]
assert metadata["tags"] == ["eu-only"] # pyright: ignore[reportIndexIssue] # the judge request's metadata dict
assert metadata["inherited_tags"] == ["team-eu"] # pyright: ignore[reportIndexIssue] # same
assert metadata["internal_call_origin"] == "autorouter_classifier" # pyright: ignore[reportIndexIssue] # same
@pytest.mark.asyncio
@pytest.mark.parametrize(("advantage", "expected"), [(0.61, "strong"), (0.6, "fast")])
async def test_decision_model_advantage_must_exceed_the_cutoff(advantage: float, expected: str) -> None:
async def decide(request: Mapping[str, object]) -> OpenAIDecisionResponse:
(question,) = request["questions"] # pyright: ignore[reportGeneralTypeIssues] # the decisions request shape
assert {choice["value"] for choice in question["choices"]} == {"advantage", "no_advantage"}
return decision_answer(advantage)
model, decision = await route(
config(classifier={"kind": "decision", "cutoff": 0.6}, threshold=1.0), HELLO, decide=decide
)
assert model == expected
assert decision["switchyard_evidence"]["score"] == advantage
async def judge_down(request: Mapping[str, object]) -> ModelResponse:
raise ConnectionError("judge down")
async def decision_refused(request: Mapping[str, object]) -> OpenAIDecisionResponse:
return OpenAIDecisionResponse.model_validate(
{
"model": "jev",
"answers": [{"type": "refusal", "name": "route"}],
"usage": {"input_tokens": 1, "output_tokens": 0, "total_tokens": 1},
}
)
@pytest.mark.asyncio
@pytest.mark.parametrize(
("classifier", "router_kwargs", "picker", "expected"),
[
({}, {"completion": judge_down}, "efficient_first", "fast"),
({}, {"completion": judge_down}, "capable_first", "strong"),
({"kind": "decision"}, {"decide": decision_refused}, "efficient_first", "fast"),
],
)
async def test_classifier_failure_falls_open_to_the_picker_default(
classifier: Mapping[str, object], router_kwargs: Mapping[str, object], picker: str, expected: str
) -> None:
model, decision = await route(config(classifier=classifier, picker=picker, threshold=1.0), HELLO, **router_kwargs)
assert model == expected
assert "switchyard_fallback_reason" not in decision
class EndsWithoutDecision:
async def _steps(self) -> AsyncIterator[object]:
for step in ():
yield step
def run_stream(self, *args: object, **kwargs: object) -> AsyncIterator[object]:
return self._steps()
class RaisesLibsyError:
async def _steps(self) -> AsyncIterator[object]:
raise libsy.LibsyError("algorithm failed")
yield
def run_stream(self, *args: object, **kwargs: object) -> AsyncIterator[object]:
return self._steps()
@pytest.mark.asyncio
@pytest.mark.parametrize(
("route_config", "messages", "call_type", "algorithm", "reason"),
[
(
config(),
[{"role": "user", "content": [{"type": "input_audio"}]}],
"acompletion",
None,
"unsupported_content",
),
(config(), HELLO, None, None, "unsupported_request_type"),
(config(), HELLO, "acompletion", EndsWithoutDecision(), "no_decision"),
(config(), HELLO, "acompletion", RaisesLibsyError(), "libsy_error"),
(
config(classifier={}, threshold=1.0),
HELLO,
"acompletion",
build_algorithm(config(classifier={"kind": "decision"}, threshold=1.0)),
"unexpected_decision_call",
),
(
config(classifier={"kind": "decision"}, threshold=1.0),
HELLO,
"acompletion",
build_algorithm(config(classifier={}, threshold=1.0)),
"unexpected_judge_call",
),
],
)
async def test_anything_switchyard_cannot_route_goes_to_capable(
route_config: SwitchyardRouterConfig, messages: object, call_type: str | None, algorithm: object, reason: str
) -> None:
model, decision = await route(route_config, messages, call_type=call_type, algorithm=algorithm)
assert model == "strong"
assert decision["switchyard_fallback_reason"] == reason
@pytest.mark.asyncio
async def test_failing_judge_is_called_once_even_when_the_router_retries(monkeypatch: pytest.MonkeyPatch) -> None:
attempts: Final[list[str]] = []
class CountAttempts(CustomLogger):
def log_pre_api_call(self, model: str, messages: object, kwargs: dict[str, object]) -> None:
attempts.append(model)
monkeypatch.setattr(litellm, "callbacks", [CountAttempts()])
proxy: Final = llm_router(
route_config={
"models": {"capable": "strong", "efficient": "fast", "judge": "judge"},
"classifier": {},
"options": {"confidence_threshold": 1.0},
},
judge_fails=True,
num_retries=3,
)
await proxy.acompletion(model="route", messages=HELLO)
assert attempts.count("judge") == 1
@pytest.mark.asyncio
async def test_judge_cancellation_propagates() -> None:
async def cancelled(request: Mapping[str, object]) -> ModelResponse:
raise asyncio.CancelledError()
with pytest.raises(asyncio.CancelledError):
await route(config(classifier={}), HELLO, completion=cancelled)

2
tests/unit/router_strategy/test_router_tag_routing.py
View file

@ -3023,7 +3023,7 @@ class _RewriteToTierStrategy:
self.rewrite_to = rewrite_to
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse

10
tests/unit/test_router/test_router.py
View file

@ -9006,7 +9006,7 @@ class TestConsumedRequestTagsStamp:
self.rewrite_to = rewrite_to
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse
@ -9097,7 +9097,7 @@ class TestClaudeCodeSubagentSessionRouterBinding:
self.routed_model = routed_model
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse
@ -10151,7 +10151,7 @@ class TestModelGroupAliasReachesPreRoutingStrategies:
class _RewriteStrategy:
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse
@ -10268,7 +10268,7 @@ class TestTeamPublicNameReachesPreRoutingStrategies:
self.rewrite_to = rewrite_to
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse
@ -10505,7 +10505,7 @@ class TestAutoRouterCompressionDecoupling:
self.received_messages: list[dict] | None = None
async def async_pre_routing_hook(
self, model, request_kwargs, messages=None, input=None, specific_deployment=False
self, model, request_kwargs, messages=None, input=None, specific_deployment=False, call_type=None
):
from litellm.types.router import PreRoutingHookResponse

1
tests/unit/types/test_litellm_params.py
View file

@ -168,6 +168,7 @@ OPTION_NAMES: Final = (
"tag_regex",
"max_file_size_mb",
"silent_model",
"switchyard_router_config",
"auto_router_config_path",
"auto_router_config",
"auto_router_default_model",

1
ui/litellm-dashboard/src/app/(dashboard)/hooks/models/useModels.ts
View file

@ -109,6 +109,7 @@ export interface AutoRouterDeployment extends AutoRouterCandidateDeployment {
model?: string | null;
base_model?: string | null;
custom_llm_provider?: string | null;
switchyard_router_config?: unknown;
complexity_router_config?: unknown;
complexity_router_default_model?: string | null;
auto_router_config?: unknown;

11
ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/AutoRouters/autoRouterRows.ts
View file

@ -87,6 +87,14 @@ const configManaged = (label: string, config: Record<string, unknown>): Presenta
/** How each strategy renders itself, given its own config object. */
const PRESENTERS: Record<AutoRouterKind, (config: Record<string, unknown>) => Presentation> = {
switchyard: (config) => ({
typeLabel: "Switchyard",
targets: dedupe(
[asRecord(config.models).capable, asRecord(config.models).efficient].filter(
(model): model is string => typeof model === "string",
),
),
}),
complexity: (config) => ({
typeLabel: complexityTypeLabel(config),
targets: dedupe(Object.values(asRecord(config.tiers)).flatMap(normalizeTierModels)),
@ -131,7 +139,8 @@ export const toAutoRouterRow = (
canDelete: canDelete && mayActOnRow,
editBlockedReason,
createdAt: info.created_at ?? undefined,
defaultModel: (params[strategy.defaultModelKey] as string | null | undefined) ?? null,
defaultModel:
strategy.kind === "switchyard" ? null : (params[strategy.defaultModelKey] as string | null | undefined) ?? null,
deployment,
...PRESENTERS[strategy.kind](asRecord(params[strategy.configKey])),
};

20
ui/litellm-dashboard/src/components/add_model/AutoRouterClassifierTabs.tsx
View file

@ -135,6 +135,7 @@ interface AutoRouterClassifierTabsProps {
children: React.ReactNode;
judgeModel?: React.ReactNode;
decisionModel?: React.ReactNode;
switchyard?: { selected: boolean; onSelect: () => void; fields: React.ReactNode };
}
const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
@ -143,6 +144,7 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
children,
judgeModel,
decisionModel,
switchyard,
}) => {
const { t } = useTranslation();
const id = useId();
@ -159,14 +161,16 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
jev: "jev",
custom: "custom",
};
const family = familyByType[classifierType];
const family = switchyard?.selected ? "switchyard" : familyByType[classifierType];
const hasCustomTiers = Boolean(value.custom_tier_set);
const changeType = (next: ClassifierType) => {
if (next !== classifierType) onChange(transitionClassifierType(value, next));
};
const changeFamily = (next: unknown) => {
if (next === "switchyard") return switchyard?.onSelect();
if (next === family) return;
if (next === familyByType[classifierType]) return onChange(value);
if (next === "heuristics") changeType("heuristic");
if (next === "llm") changeType("llm");
if (next === "jev") changeType("jev");
@ -207,7 +211,11 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
</div>
<AutoRouterLimits />
</legend>
<RadioGroup value={family} onValueChange={changeFamily} className="grid gap-3 sm:grid-cols-3">
<RadioGroup
value={family}
onValueChange={changeFamily}
className={switchyard ? "grid gap-3 sm:grid-cols-2 xl:grid-cols-4" : "grid gap-3 sm:grid-cols-3"}
>
{[
{
value: "heuristics",
@ -220,6 +228,9 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
label: t("addModel.autoRouterClassifierTabs.families.decisionModel"),
description: t("addModel.autoRouterClassifierTabs.families.decisionModelDescription"),
},
...(switchyard
? [{ value: "switchyard", label: "Switchyard", description: "Route by the conversation’s tool history." }]
: []),
].map((option) => (
<Label
key={option.value}
@ -244,6 +255,7 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
))}
</RadioGroup>
</fieldset>
{family === "switchyard" && switchyard?.fields}
{family === "jev" &&
(decisionModel ?? (
<div className="space-y-2">
@ -355,7 +367,7 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
</p>
</div>
)}
{hasCustomTiers && (
{family !== "switchyard" && hasCustomTiers && (
<p className="text-xs text-muted-foreground">{t("addModel.autoRouterClassifierTabs.restoreStandardTiers")}</p>
)}
{availability.data?.error && !availability.isChecking && (
@ -372,7 +384,7 @@ const AutoRouterClassifierTabs: React.FC<AutoRouterClassifierTabsProps> = ({
</button>
</p>
)}
{children}
{family !== "switchyard" && children}
</div>
);
};

17
ui/litellm-dashboard/src/components/add_model/BlockedReasonTooltip.tsx Normal file
View file

@ -0,0 +1,17 @@
import React from "react";
import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip";
const BlockedReasonTooltip: React.FC<{ reason: string | null; children: React.ReactElement }> = ({
reason,
children,
}) =>
reason === null ? (
children
) : (
<Tooltip>
<TooltipTrigger render={children} />
<TooltipContent>{reason}</TooltipContent>
</Tooltip>
);
export default BlockedReasonTooltip;

342
ui/litellm-dashboard/src/components/add_model/SwitchyardConfigFields.tsx Normal file
View file

@ -0,0 +1,342 @@
"use client";
import { useId, useState } from "react";
import { Input } from "@/components/ui/input";
import { Textarea } from "@/components/ui/textarea";
import { Slider } from "@/components/ui/slider";
import { Switch } from "@/components/ui/switch";
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
import ModelChoiceCombobox, { type ModelChoice } from "./ModelChoiceCombobox";
import {
stageConfidence,
toggleClassifier,
setClassifierKind,
parseToolSemantics,
type SwitchyardConfig,
} from "./switchyard_config";
interface Props {
value: SwitchyardConfig;
onChange: (value: SwitchyardConfig) => void;
models: ModelChoice[];
decisionModels: ModelChoice[];
onToolSemanticsError: (error: string | null) => void;
}
const isStructured = (instructions: unknown): instructions is Record<string, unknown> =>
typeof instructions === "object" && instructions !== null;
const instructionsText = (instructions: string | Record<string, unknown> | null | undefined): string =>
isStructured(instructions) ? JSON.stringify(instructions, null, 2) : instructions ?? "";
export default function SwitchyardConfigFields({
value,
onChange,
models,
decisionModels,
onToolSemanticsError,
}: Props) {
const id = useId();
const [toolSemantics, setToolSemantics] = useState(
value.options.tool_semantics ? JSON.stringify(value.options.tool_semantics, null, 2) : "",
);
const classifier = value.classifier;
const llm = classifier?.kind === "llm" ? classifier : undefined;
const decision = classifier?.kind === "decision" ? classifier : undefined;
const setOption = (key: "recent_window", text: string) =>
onChange({ ...value, options: { ...value.options, [key]: text === "" ? undefined : Number(text) } });
const setClassifierNumber = (
key: "base_threshold" | "threshold_step" | "max_output_tokens" | "recent_turn_window",
text: string,
) => {
if (llm) onChange({ ...value, classifier: { ...llm, [key]: text === "" ? undefined : Number(text) } });
};
const changeToolSemantics = (text: string) => {
setToolSemantics(text);
try {
const parsed = parseToolSemantics(text);
onChange({ ...value, options: { ...value.options, tool_semantics: parsed } });
onToolSemanticsError(null);
} catch {
onToolSemanticsError("Tool semantics must be JSON mapping observe, mutate, plan, or new to arrays of tool names");
}
};
return (
<div className="space-y-6">
<div className="grid gap-5 md:grid-cols-2">
<div className="min-w-0 space-y-2">
<span id={`${id}-heuristic`} className="text-sm font-medium">
Heuristic
</span>
<p
aria-labelledby={`${id}-heuristic`}
className="flex min-h-10 items-center rounded-md border bg-muted/30 px-3 py-2 text-sm"
>
Stage Router
</p>
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-bias`} className="text-sm font-medium">
Bias
</label>
<Select
value={value.options.picker}
items={[
{ value: "efficient_first", label: "Cost-first" },
{ value: "capable_first", label: "Quality-first (experimental)" },
]}
onValueChange={(picker) =>
onChange({
...value,
options: { ...value.options, picker: picker === "capable_first" ? "capable_first" : "efficient_first" },
})
}
>
<SelectTrigger
id={`${id}-bias`}
className="h-auto min-h-10 w-full py-2 *:data-[slot=select-value]:line-clamp-none"
>
<SelectValue className="whitespace-normal break-words" />
</SelectTrigger>
<SelectContent>
<SelectItem value="efficient_first">Cost-first</SelectItem>
<SelectItem value="capable_first">
<span className="whitespace-normal">Quality-first (experimental)</span>
</SelectItem>
</SelectContent>
</Select>
{value.options.picker === "capable_first" && (
<p className="text-sm text-muted-foreground">
Switchyard has not benchmarked this bias, so it has no calibrated confidence thresholds.
</p>
)}
</div>
{(["capable", "efficient"] as const).map((tier) => (
<div key={tier} className="min-w-0 space-y-2">
<label htmlFor={`${id}-${tier}`} className="text-sm font-medium">
{tier === "capable" ? "Capable model" : "Efficient model"}
</label>
<ModelChoiceCombobox
id={`${id}-${tier}`}
value={value.models[tier]}
onChange={(model) => onChange({ ...value, models: { ...value.models, [tier]: model ?? "" } })}
choices={models}
placeholder="Select a model"
ariaInvalid={undefined}
ariaDescribedBy={undefined}
/>
</div>
))}
</div>
<div className="rounded-lg border p-4 space-y-4">
<div className="flex items-center justify-between gap-4">
<div>
<label htmlFor={`${id}-classifier`} className="text-sm font-medium">
Classifier
</label>
<p className="text-sm text-muted-foreground">
Ask a judge model when the tool history does not give a confident routing decision.
</p>
</div>
<Switch
id={`${id}-classifier`}
checked={Boolean(classifier)}
onCheckedChange={(enabled) => onChange(toggleClassifier(value, enabled))}
/>
</div>
{classifier && (
<div className="grid gap-5 md:grid-cols-2">
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-classifier-kind`} className="text-sm font-medium">
Classifier type
</label>
<Select
value={classifier.kind}
items={[
{ value: "llm", label: "LLM judge" },
{ value: "decision", label: "Decision model" },
]}
onValueChange={(kind) => onChange(setClassifierKind(value, kind === "decision" ? "decision" : "llm"))}
>
<SelectTrigger id={`${id}-classifier-kind`} className="min-h-10 w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="llm">LLM judge</SelectItem>
<SelectItem value="decision">Decision model</SelectItem>
</SelectContent>
</Select>
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-judge`} className="text-sm font-medium">
{decision ? "Decision model" : "Judge model"}
</label>
<ModelChoiceCombobox
id={`${id}-judge`}
value={value.models.judge ?? ""}
onChange={(judge) => onChange({ ...value, models: { ...value.models, judge: judge ?? "" } })}
choices={decision ? decisionModels : models}
placeholder={decision ? "Select a decision model" : "Select a judge model"}
ariaInvalid={undefined}
ariaDescribedBy={undefined}
/>
</div>
</div>
)}
</div>
<div className="space-y-3">
<div className="flex justify-between">
<span id={`${id}-confidence`} className="text-sm font-medium">
Confidence threshold
</span>
<span className="text-sm tabular-nums">{stageConfidence(value).toFixed(2)}</span>
</div>
<Slider
aria-labelledby={`${id}-confidence`}
value={[stageConfidence(value)]}
min={0}
max={1}
step={0.01}
onValueChange={(next) =>
onChange({
...value,
options: { ...value.options, confidence_threshold: Array.isArray(next) ? next[0] : next },
})
}
/>
<p className="text-sm text-muted-foreground">
Below this confidence, use {classifier ? "the judge model" : "the bias’s default model"}. Critical errors can
still select the capable model.
</p>
</div>
<details className="rounded-lg border p-4">
<summary className="cursor-pointer text-sm font-medium">Advanced settings</summary>
<div className="mt-5 space-y-5">
<h3 className="text-sm font-semibold">Stage</h3>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-recent`}>Recent window</label>
<Input
id={`${id}-recent`}
type="number"
min={1}
value={value.options.recent_window ?? ""}
placeholder="Library default"
onChange={(event) => setOption("recent_window", event.target.value)}
/>
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-tools`}>Tool semantics</label>
<Textarea
id={`${id}-tools`}
value={toolSemantics}
placeholder={'{"observe": ["read_file"], "mutate": ["edit_file"]}'}
onChange={(event) => changeToolSemantics(event.target.value)}
/>
</div>
{decision && (
<>
<h3 className="text-sm font-semibold">Decision model</h3>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-cutoff`}>Advantage cutoff</label>
<Input
id={`${id}-cutoff`}
type="number"
min={0}
max={1}
step={0.01}
value={decision.cutoff}
onChange={(event) =>
onChange({ ...value, classifier: { ...decision, cutoff: Number(event.target.value) } })
}
/>
<p className="text-sm text-muted-foreground">
Route to the capable model when the decision model’s advantage probability is above this value.
</p>
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-instructions`}>Decision instructions</label>
<Textarea
id={`${id}-instructions`}
readOnly={isStructured(decision.instructions)}
value={instructionsText(decision.instructions)}
placeholder="Use Switchyard’s built-in instructions"
onChange={(event) =>
onChange({
...value,
classifier: { ...decision, instructions: event.target.value || undefined },
})
}
/>
</div>
</>
)}
{llm && (
<>
<h3 className="text-sm font-semibold">Classifier</h3>
<div className="grid gap-4 md:grid-cols-2">
{(
[
["base_threshold", "Capability threshold"],
["threshold_step", "Threshold step"],
["max_output_tokens", "Maximum output tokens"],
["recent_turn_window", "Recent turn window"],
] as const
).map(([key, label]) => (
<div key={key} className="min-w-0 space-y-2">
<label htmlFor={`${id}-${key}`}>{label}</label>
<Input
id={`${id}-${key}`}
type="number"
min={key.includes("threshold") ? 0 : 1}
max={key.includes("threshold") ? 1 : undefined}
step={key.includes("threshold") ? 0.01 : 1}
value={llm[key] ?? ""}
onChange={(event) => setClassifierNumber(key, event.target.value)}
/>
</div>
))}
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-format`}>Response format</label>
<Select
value={llm.response_format_type}
items={[
{ value: "json_schema", label: "JSON schema" },
{ value: "json_object", label: "JSON object" },
]}
onValueChange={(format) =>
onChange({
...value,
classifier: {
...llm,
response_format_type: format === "json_object" ? "json_object" : "json_schema",
},
})
}
>
<SelectTrigger id={`${id}-format`} className="min-h-10 w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="json_schema">JSON schema</SelectItem>
<SelectItem value="json_object">JSON object</SelectItem>
</SelectContent>
</Select>
</div>
<div className="min-w-0 space-y-2">
<label htmlFor={`${id}-prompt`}>Classifier prompt</label>
<Textarea
id={`${id}-prompt`}
value={llm.prompt ?? ""}
placeholder="Use Switchyard’s built-in prompt"
onChange={(event) =>
onChange({ ...value, classifier: { ...llm, prompt: event.target.value || undefined } })
}
/>
</div>
</>
)}
</div>
</details>
</div>
);
}

176
ui/litellm-dashboard/src/components/add_model/SwitchyardRouterForm.tsx Normal file
View file

@ -0,0 +1,176 @@
"use client";
import { useId, useState } from "react";
import { useQuery } from "@tanstack/react-query";
import { Input } from "@/components/ui/input";
import { Button } from "@/components/ui/button";
import { fetchAvailableModels } from "@/components/llm_calls/fetch_models";
import { modelPatchUpdateCall } from "@/components/networking";
import {
emptySwitchyardConfig,
isDecisionModelMode,
switchyardConfigSchema,
switchyardRouterPayload,
switchyardSaveError,
type SwitchyardConfig,
} from "./switchyard_config";
import SwitchyardConfigFields from "./SwitchyardConfigFields";
import { handleAddSwitchyardSubmit, type SwitchyardModelInfo } from "./handle_add_auto_router_submit";
interface FieldsProps {
accessToken: string;
routerName: string;
value: SwitchyardConfig;
onChange: (value: SwitchyardConfig) => void;
onToolSemanticsError: (error: string | null) => void;
}
export function SwitchyardModelFields({ accessToken, routerName, value, onChange, onToolSemanticsError }: FieldsProps) {
const { data: models = [], isError } = useQuery({
queryKey: ["availableModels", "switchyard", accessToken],
queryFn: () => fetchAvailableModels(accessToken),
enabled: Boolean(accessToken),
});
const routable = models.filter(
(model) => model.model_group !== routerName && !model.model_group.startsWith("auto_router/"),
);
const toChoice = (model: { model_group: string }) => ({ value: model.model_group, label: model.model_group });
return (
<div className="space-y-6">
<p className="text-sm text-muted-foreground">
Route between a capable and an efficient model using the conversation’s tool history.
</p>
{isError && (
<p role="alert" className="text-sm text-destructive">
Unable to load available models.
</p>
)}
<SwitchyardConfigFields
value={value}
onChange={onChange}
models={routable.filter((model) => !isDecisionModelMode(model.mode)).map(toChoice)}
decisionModels={routable.filter((model) => isDecisionModelMode(model.mode)).map(toChoice)}
onToolSemanticsError={onToolSemanticsError}
/>
</div>
);
}
interface StageSeed {
capable?: string;
efficient?: string;
}
export function useSwitchyardOption(accessToken: string, routerName: string, enabled: boolean) {
const [selected, setSelected] = useState(false);
const [config, setConfig] = useState(emptySwitchyardConfig);
const [toolSemanticsError, setToolSemanticsError] = useState<string | null>(null);
const select = (seed: StageSeed) => {
setSelected(true);
setToolSemanticsError(null);
setConfig((current) => ({
...current,
models: {
...current.models,
capable: current.models.capable || seed.capable || "",
efficient: current.models.efficient || seed.efficient || "",
},
}));
};
return {
selected,
deselect: () => setSelected(false),
option: (seed: StageSeed) =>
enabled
? {
selected,
onSelect: () => select(seed),
fields: (
<SwitchyardModelFields
accessToken={accessToken}
routerName={routerName}
value={config}
onChange={setConfig}
onToolSemanticsError={setToolSemanticsError}
/>
),
}
: undefined,
submit: (name: string, modelInfo: SwitchyardModelInfo, resetForm: () => void, onSuccess: () => void) => {
const draft = { name, config, toolSemanticsError, modelInfo };
return handleAddSwitchyardSubmit(draft, accessToken, resetForm, onSuccess);
},
};
}
interface Props {
accessToken: string;
onSuccess: (saved: { model_name: string; switchyard_router_config: SwitchyardConfig }) => void;
initial: { name: string; modelId: string; config: unknown };
}
export default function SwitchyardRouterForm({ accessToken, onSuccess, initial }: Props) {
const id = useId();
const parsed = switchyardConfigSchema.safeParse(initial.config);
const [config, setConfig] = useState<SwitchyardConfig>(parsed.success ? parsed.data : emptySwitchyardConfig());
const [name, setName] = useState(initial.name);
const [error, setError] = useState<string | null>(
parsed.success ? null : "The stored Switchyard configuration is invalid. Correct it before saving.",
);
const [toolError, setToolError] = useState<string | null>(null);
const [saving, setSaving] = useState(false);
const save = async () => {
const invalid = switchyardSaveError(name, config);
if (invalid) {
setError(invalid);
return;
}
setSaving(true);
setError(null);
try {
const payload = switchyardRouterPayload(name, config);
await modelPatchUpdateCall(accessToken, payload, initial.modelId);
onSuccess({
model_name: payload.model_name,
switchyard_router_config: payload.litellm_params.switchyard_router_config,
});
} catch (failure) {
setError(failure instanceof Error ? failure.message : "Unable to save Switchyard router");
} finally {
setSaving(false);
}
};
return (
<form
className="space-y-6"
onSubmit={(event) => {
event.preventDefault();
void save();
}}
>
<div className="space-y-2">
<label htmlFor={`${id}-name`} className="text-sm font-medium">
Router name
</label>
<Input id={`${id}-name`} value={name} onChange={(event) => setName(event.target.value)} />
</div>
<SwitchyardModelFields
accessToken={accessToken}
routerName={name}
value={config}
onChange={setConfig}
onToolSemanticsError={setToolError}
/>
{(error || toolError) && (
<p role="alert" className="text-sm text-destructive">
{toolError || error}
</p>
)}
<div className="flex justify-end">
<Button type="submit" disabled={saving || Boolean(toolError)}>
{saving ? "Saving…" : "Save changes"}
</Button>
</div>
</form>
);
}

88
ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.integration.test.tsx
View file

@ -19,7 +19,7 @@ import userEvent from "@testing-library/user-event";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import AddAutoRouterTab from "./add_auto_router_tab";
import { toast } from "@/lib/toast";
import { handleAddAutoRouterSubmit } from "./handle_add_auto_router_submit";
import { handleAddAutoRouterSubmit, handleAddSwitchyardSubmit } from "./handle_add_auto_router_submit";
import { getMissingTiersError } from "./build_complexity_router_config";
import { getSubmitBlockedReason } from "./add_auto_router_tab";
import { buildModelAvailability } from "@/lib/autorouter_presets";
@ -145,6 +145,7 @@ vi.mock("@/app/(dashboard)/hooks/models/useModels", async (importOriginal) => {
vi.mock("./handle_add_auto_router_submit", () => ({
handleAddAutoRouterSubmit: vi.fn(),
handleAddSwitchyardSubmit: vi.fn(),
}));
// Kept real by default so the "mandatory field" test still sees genuine tier validation; one
@ -179,7 +180,7 @@ vi.mock("../common_components/team_dropdown", () => ({
const renderAdvanced: typeof renderWithProviders = (...args) => {
const result = renderWithProviders(...args);
fireEvent.click(screen.getByRole("button", { name: "Advanced configuration", exact: true }));
fireEvent.click(screen.getByRole("button", { name: "Customize tiers and keywords", exact: true }));
return result;
};
@ -202,6 +203,64 @@ describe("AddAutoRouterTab", () => {
vi.mocked(modelCostMap).mockResolvedValue({});
});
it("swaps the tiers for the Switchyard fields and saves Switchyard through the same form", async () => {
renderAdvanced(<Harness />);
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
expect(screen.getByText("Stage Router", { exact: true })).toBeVisible();
expect(screen.queryByText("Define your tiers")).not.toBeInTheDocument();
expect(screen.getByTestId("auto-router-test-connect-btn")).not.toBeVisible();
fireEvent.click(screen.getByRole("radio", { name: "Heuristics", exact: true }));
expect(screen.getByText("Define your tiers")).toBeVisible();
expect(screen.queryByText("Stage Router", { exact: true })).not.toBeInTheDocument();
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
fireEvent.change(screen.getByLabelText(/Auto Router Name/), { target: { value: "stage-test" } });
fireEvent.click(screen.getByRole("button", { name: "Add Auto Router" }));
await waitFor(() => expect(handleAddSwitchyardSubmit).toHaveBeenCalledOnce());
expect(vi.mocked(handleAddSwitchyardSubmit).mock.calls[0][0].name).toBe("stage-test");
expect(handleAddAutoRouterSubmit).not.toHaveBeenCalled();
});
it("starts Switchyard from the quick setup models and creates it for the selected team", async () => {
mockFetchAvailableModels.mockResolvedValue(
["claude-haiku-5-5", "claude-opus-5-5"].map((model_group) => ({ model_group, mode: "chat" })),
);
const user = userEvent.setup();
renderWithProviders(
<AddAutoRouterTab handleOk={vi.fn()} accessToken="token" userRole="Admin" createScope="team-required" />,
);
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
fireEvent.change(screen.getByLabelText(/Auto Router Name/), { target: { value: "stage-test" } });
fireEvent.click(screen.getByRole("button", { name: "Add Auto Router" }));
await waitFor(() => expect(screen.getByLabelText(/Auto Router Name/)).toHaveValue("stage-test"));
expect(handleAddSwitchyardSubmit).not.toHaveBeenCalled();
fireEvent.click(screen.getByRole("radio", { name: "Heuristics", exact: true }));
await user.selectOptions(screen.getByTestId("team-dropdown"), "team-1");
expect(await screen.findByRole("combobox", { name: "Everyday model" })).toHaveTextContent("claude-haiku-5-5");
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
fireEvent.click(screen.getByRole("button", { name: "Add Auto Router" }));
await waitFor(() => expect(handleAddSwitchyardSubmit).toHaveBeenCalledOnce());
expect(vi.mocked(handleAddSwitchyardSubmit).mock.calls[0][0]).toMatchObject({
config: { models: { capable: "claude-opus-5-5", efficient: "claude-haiku-5-5" } },
modelInfo: { team_id: "team-1" },
});
});
it("forgets a tool semantics error once Switchyard is reopened from the saved settings", async () => {
renderAdvanced(<Harness />);
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
fireEvent.change(screen.getByLabelText("Tool semantics"), { target: { value: "{not json" } });
fireEvent.click(screen.getByRole("radio", { name: "Heuristics", exact: true }));
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
expect(screen.getByLabelText("Tool semantics")).toHaveValue("");
fireEvent.change(screen.getByLabelText(/Auto Router Name/), { target: { value: "stage-test" } });
fireEvent.click(screen.getByRole("button", { name: "Add Auto Router" }));
await waitFor(() => expect(handleAddSwitchyardSubmit).toHaveBeenCalledOnce());
expect(vi.mocked(handleAddSwitchyardSubmit).mock.calls[0][0].toolSemanticsError).toBeNull();
});
it("creates the goal-selected latest OpenAI pair using its Bedrock deployment aliases", async () => {
mockFetchAvailableModels.mockResolvedValue(
["small", "middle-old", "middle", "large"].map((model_group) => ({ model_group, mode: "chat" })),
@ -322,6 +381,25 @@ describe("AddAutoRouterTab", () => {
},
);
it("shows every strategy up front and carries the quick setup models into the one picked", async () => {
mockFetchAvailableModels.mockResolvedValue(
["claude-haiku-5-5", "claude-opus-5-5"].map((model_group) => ({ model_group, mode: "chat" })),
);
renderWithProviders(<Harness />);
expect(await screen.findByRole("combobox", { name: "Everyday model" })).toHaveTextContent("claude-haiku-5-5");
expect(screen.getByRole("radio", { name: "Heuristics", exact: true })).toBeChecked();
fireEvent.click(screen.getByRole("radio", { name: "Switchyard", exact: true }));
expect(screen.queryByRole("combobox", { name: "Everyday model" })).not.toBeInTheDocument();
fireEvent.click(screen.getByRole("radio", { name: "Heuristics", exact: true }));
expect(screen.getByRole("combobox", { name: "Everyday model" })).toHaveTextContent("claude-haiku-5-5");
fireEvent.click(screen.getByRole("radio", { name: "LLM", exact: true }));
expect(screen.getByRole("combobox", { name: "Judge model" })).toBeInTheDocument();
expectTierModel("Simple", "claude-haiku-5-5");
expectTierModel("Reasoning", "claude-opus-5-5");
});
it("carries Responses model choices from simple setup into Advanced tiers", async () => {
const models = ["gpt-6-luna", "gpt-6-astra"];
mockFetchAvailableModels.mockResolvedValue(models.map((model_group) => ({ model_group, mode: "responses" })));
@ -338,7 +416,7 @@ describe("AddAutoRouterTab", () => {
);
expect(screen.getByRole("combobox", { name: "Hard tasks model" })).toHaveTextContent("gpt-6-astra");
expect(screen.queryByText(/Two deployed models are needed/)).not.toBeInTheDocument();
fireEvent.click(screen.getByRole("button", { name: "Advanced configuration", exact: true }));
fireEvent.click(screen.getByRole("button", { name: "Customize tiers and keywords", exact: true }));
expectTierModel("Simple", "gpt-6-luna");
expectTierModel("Reasoning", "gpt-6-astra");
});
@ -357,7 +435,7 @@ describe("AddAutoRouterTab", () => {
fireEvent.click(screen.getByRole("button", { name: "Cheapest", exact: true }));
expect(screen.getByRole("combobox", { name: "Everyday model" })).toHaveTextContent("custom-fast");
expect(screen.getByRole("combobox", { name: "Hard tasks model" })).toHaveTextContent("claude-opus-5-5");
fireEvent.click(screen.getByRole("button", { name: "Advanced configuration", exact: true }));
fireEvent.click(screen.getByRole("button", { name: "Customize tiers and keywords", exact: true }));
expectTierModel("Simple", "custom-fast");
expectTierModel("Reasoning", "claude-opus-5-5");
});
@ -392,7 +470,7 @@ describe("AddAutoRouterTab", () => {
renderWithProviders(<Harness />);
expect(await screen.findByText(/Two deployed models are needed/)).toBeVisible();
expect(screen.getByRole("button", { name: "Add Auto Router" })).toBeDisabled();
fireEvent.click(screen.getByRole("button", { name: "Advanced configuration", exact: true }));
fireEvent.click(screen.getByRole("button", { name: "Customize tiers and keywords", exact: true }));
expect(screen.getByText("Define your tiers")).toBeVisible();
});

188
ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx
View file

@ -5,6 +5,8 @@ import ClassifierPrimarySettings from "./ClassifierPrimarySettings";
import AutoRouterClassifierTabs from "./AutoRouterClassifierTabs";
import { getForecastConfigError, isForecastClassifier } from "../add_model/forecast_classifier_config";
import React, { useEffect, useState } from "react";
import BlockedReasonTooltip from "./BlockedReasonTooltip";
import { useSwitchyardOption } from "./SwitchyardRouterForm";
import { useTranslation } from "react-i18next";
import { useQuery } from "@tanstack/react-query";
import { useWatch } from "react-hook-form";
@ -166,19 +168,6 @@ const EMPTY_FORM_VALUES: AddAutoRouterFormValues = {
const teamScopePayload = (requiresTeamScope: boolean, teamId: string | null): { team_id?: string } =>
requiresTeamScope && teamId ? { team_id: teamId } : {};
const BlockedReasonTooltip: React.FC<{ reason: string | null; children: React.ReactElement }> = ({
reason,
children,
}) =>
reason === null ? (
children
) : (
<Tooltip>
<TooltipTrigger render={children} />
<TooltipContent>{reason}</TooltipContent>
</Tooltip>
);
const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
handleOk,
accessToken,
@ -191,6 +180,7 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
const requiresTeamScope = createScope === "team-required";
const form = useZodForm(autoRouterSchema(requiresTeamScope), { defaultValues: EMPTY_FORM_VALUES });
const watchedName = useWatch({ control: form.control, name: "auto_router_name" });
const switchyard = useSwitchyardOption(accessToken, watchedName, isProxyAdminRole(userRole));
const watchedTeamId = useWatch({ control: form.control, name: "team_id" });
const actor = { userRole, userID: userId ?? null, isViewOnly: false };
const isMemberManaged =
@ -341,11 +331,8 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
setAdvanced(true);
};
// A preset's models can only be trusted against a successfully loaded list. Selection and the
// greyed-out state derive from this one function, so a preset that cannot be selected can never
// have been applied: while loading we withhold selection rather than let a caller pick a preset
// whose models we cannot yet verify, and a failed fetch leaves every preset unverifiable. This
// makes the load-race (pick during loading, then discover a missing model) unrepresentable.
// Withhold preset selection until its models are verified against a successfully loaded list.
// Reuse this check for application and disabled state to prevent load races.
const presetAvailability = React.useCallback(
(preset: AutoRouterPreset): PresetAvailability => {
if (modelsLoading) return { kind: "loading" };
@ -444,8 +431,15 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
? t("addModel.addAutoRouterTab.selectTeamToSeeModels")
: simpleBlockedReason ?? submitBlockedReason ?? routerAvailability.saveBlockedReason;
const handleClassifierChange = (config: ComplexityRouterConfigValue) => {
setSelectedPreset(undefined);
switchyard.deselect();
if (advanced || effectiveClassifierType(config) === effectiveClassifierType(complexityRouterConfig)) {
setSelectedPreset(undefined);
setComplexityRouterConfig(config);
return;
}
setComplexityRouterConfig(config);
setPrefillAttempted(simpleConfig !== null);
setAdvanced(true);
};
const jevRequestParams =
effectiveClassifierType(complexityRouterConfig) === "jev"
@ -523,6 +517,23 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
await handleAddAutoRouterSubmit(submitValues, accessToken, () => form.reset(EMPTY_FORM_VALUES), handleOk);
};
const submitSwitchyard = async (name: string) => {
if (requiresTeamScope && !(await form.trigger(["auto_router_name", "team_id"]))) {
toast.fromError(t("addModel.addAutoRouterTab.requiredFieldsMissing"));
return;
}
const accessGroups = isMemberManaged ? undefined : form.getValues("model_access_group");
await switchyard.submit(
name,
{
...teamScopePayload(requiresTeamScope, form.getValues("team_id")),
...(accessGroups?.length ? { access_groups: accessGroups } : {}),
},
() => form.reset(EMPTY_FORM_VALUES),
handleOk,
);
};
const handleAutoRouterSubmit = async () => {
if (isSubmitting) return;
const name = form.getValues("auto_router_name");
@ -535,7 +546,7 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
setIsSubmitting(true);
try {
await submitRecommendedRouter(name);
await (switchyard.selected ? submitSwitchyard(name) : submitRecommendedRouter(name));
} finally {
setIsSubmitting(false);
}
@ -672,63 +683,31 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
</FormField>
)}
{!advanced && (
<AutoRouterSimpleSetup
customized={Boolean(solverOverrides.everyday || solverOverrides.hard)}
hasSuggestions={Boolean(recommended || recommendSolverPair(solverModels, goal, "auto"))}
onReset={() => setSolverOverrides({})}
goal={goal}
family={family}
models={solverModels}
everyday={everyday}
hard={hard}
waitingForTeam={requiresTeamScope && !watchedTeamId}
loading={modelsLoading}
error={inventoryUnavailable}
notice={
Object.keys(solverOverrides).length > 0
? t("addModel.addAutoRouterTab.choicesKept")
: recommended?.notice ?? null
}
onGoalChange={setGoal}
onFamilyChange={(next) => {
setFamily(next);
setSolverOverrides({});
}}
onModelChange={(role, model) =>
setSolverOverrides({ everyday: everyday?.name, hard: hard?.name, [role]: model })
}
onAdvanced={openAdvanced}
onRetry={() => {
void refetchModels();
void refetchDeployments();
}}
/>
)}
{advanced && (
<AutoRouterClassifierTabs
value={complexityRouterConfig}
onChange={handleClassifierChange}
decisionModel={
<DecisionModelSelect
value={complexityRouterConfig}
onChange={setComplexityRouterConfig}
modelInfo={modelInfo}
deployments={deployments}
/>
}
judgeModel={
<ClassifierPrimarySettings
section="judge"
value={complexityRouterConfig}
onChange={setComplexityRouterConfig}
modelOptions={modelInfo
.filter((model) => model.mode !== "embedding")
.map((model) => ({ value: model.model_group, label: model.model_group }))}
showValidationErrors={showValidationErrors}
/>
}
>
<AutoRouterClassifierTabs
switchyard={switchyard.option({ capable: hard?.name, efficient: everyday?.name })}
value={complexityRouterConfig}
onChange={handleClassifierChange}
decisionModel={
<DecisionModelSelect
value={complexityRouterConfig}
onChange={setComplexityRouterConfig}
modelInfo={modelInfo}
deployments={deployments}
/>
}
judgeModel={
<ClassifierPrimarySettings
section="judge"
value={complexityRouterConfig}
onChange={setComplexityRouterConfig}
modelOptions={modelInfo
.filter((model) => model.mode !== "embedding")
.map((model) => ({ value: model.model_group, label: model.model_group }))}
showValidationErrors={showValidationErrors}
/>
}
>
{advanced ? (
<FieldGroup>
{modelsUnverifiable && (
<p className="text-sm text-destructive">
@ -740,21 +719,40 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
)}
{configurationForm}
</FieldGroup>
</AutoRouterClassifierTabs>
)}
{!advanced && routerAvailability.isError && (
<p className="mt-3 text-sm text-destructive" role="alert">
{t("addModel.addAutoRouterTab.availabilityCheckFailed")}{" "}
<Button type="button" variant="link" onClick={() => void routerAvailability.refetch()}>
{t("addModel.addAutoRouterTab.retry")}
</Button>
</p>
)}
{!advanced && routerAvailability.data?.error && (
<p className="mt-3 text-sm text-destructive" role="alert">
{routerAvailability.data.error}
</p>
)}
) : (
<AutoRouterSimpleSetup
customized={Boolean(solverOverrides.everyday || solverOverrides.hard)}
hasSuggestions={Boolean(recommended || recommendSolverPair(solverModels, goal, "auto"))}
onReset={() => setSolverOverrides({})}
goal={goal}
family={family}
models={solverModels}
everyday={everyday}
hard={hard}
waitingForTeam={requiresTeamScope && !watchedTeamId}
loading={modelsLoading}
error={inventoryUnavailable}
notice={
Object.keys(solverOverrides).length > 0
? t("addModel.addAutoRouterTab.choicesKept")
: recommended?.notice ?? null
}
onGoalChange={setGoal}
onFamilyChange={(next) => {
setFamily(next);
setSolverOverrides({});
}}
onModelChange={(role, model) =>
setSolverOverrides({ everyday: everyday?.name, hard: hard?.name, [role]: model })
}
onAdvanced={openAdvanced}
onRetry={() => {
void refetchModels();
void refetchDeployments();
}}
/>
)}
</AutoRouterClassifierTabs>
<div className="mt-5 flex flex-wrap items-center justify-between gap-3">
<Tooltip>
<TooltipTrigger
@ -775,6 +773,7 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
type="button"
variant="outline"
data-testid="auto-router-test-routing-btn"
hidden={switchyard.selected}
disabled={submitBlockedReason !== null || isSubmitting}
onClick={() => setIsRoutingTestVisible(true)}
>
@ -785,16 +784,17 @@ const AddAutoRouterTab: React.FC<AddAutoRouterTabProps> = ({
type="button"
variant="outline"
data-testid="auto-router-test-connect-btn"
hidden={switchyard.selected}
onClick={handleTestConnection}
disabled={isTestingConnection}
>
{isTestingConnection && <UiLoadingSpinner className="size-4" />}
{t("addModel.addAutoRouterTab.testConnection")}
</Button>
<BlockedReasonTooltip reason={saveBlockedReason}>
<BlockedReasonTooltip reason={switchyard.selected ? null : saveBlockedReason}>
<Button
type="button"
disabled={saveBlockedReason !== null || isSubmitting}
disabled={(!switchyard.selected && saveBlockedReason !== null) || isSubmitting}
onClick={() => {
void handleAutoRouterSubmit();
}}

11
ui/litellm-dashboard/src/components/add_model/auto_router_strategies.ts
View file

@ -13,10 +13,11 @@
* prefixes are matched first, and only a bare `auto_router/<name>` is the semantic router.
*/
export type AutoRouterKind = "complexity" | "adaptive" | "quality" | "semantic";
export type AutoRouterKind = "complexity" | "adaptive" | "quality" | "semantic" | "switchyard";
export interface AutoRouterParams {
model?: string | null;
switchyard_router_config?: unknown;
complexity_router_config?: unknown;
complexity_router_default_model?: string | null;
auto_router_config?: unknown;
@ -42,6 +43,14 @@ const startsWith = (params: AutoRouterParams, prefix: string): boolean => params
/** Ordered; the semantic entry matches anything left and must stay last. */
export const AUTO_ROUTER_STRATEGIES: readonly AutoRouterStrategy[] = [
{
kind: "switchyard",
label: "Switchyard",
configKey: "switchyard_router_config",
defaultModelKey: "switchyard_router_config",
hasEditor: true,
matches: (params) => startsWith(params, "auto_router/switchyard"),
},
{
kind: "complexity",
label: "Complexity",

27
ui/litellm-dashboard/src/components/add_model/handle_add_auto_router_submit.tsx
View file

@ -3,6 +3,7 @@ import { toast } from "@/lib/toast";
import i18n from "@/i18n/i18n";
import type { ComplexityRouterConfigPayload } from "./build_complexity_router_config";
import type { AutoRouterCompressionLitellmParams } from "./buildAutoRouterCompression";
import { switchyardRouterPayload, switchyardSaveError, type SwitchyardConfig } from "./switchyard_config";
export interface AddAutoRouterValues extends AutoRouterCompressionLitellmParams {
auto_router_name: string;
@ -57,3 +58,29 @@ export const handleAddAutoRouterSubmit = async (
toast.fromError(i18n.t("addModel.handleAddAutoRouterSubmit.failed", { error: String(error) }));
}
};
export interface SwitchyardModelInfo {
team_id?: string;
access_groups?: string[];
}
export const handleAddSwitchyardSubmit = async (
draft: { name: string; config: SwitchyardConfig; toolSemanticsError: string | null; modelInfo: SwitchyardModelInfo },
accessToken: string,
resetForm: () => void,
callback: () => void,
) => {
const invalid = draft.toolSemanticsError ?? switchyardSaveError(draft.name, draft.config);
if (invalid) return toast.fromError(invalid);
try {
await modelCreateCall(accessToken, {
...switchyardRouterPayload(draft.name, draft.config),
model_info: draft.modelInfo,
});
toast.success(i18n.t("addModel.handleAddAutoRouterSubmit.created", { name: draft.name }));
resetForm();
callback();
} catch (error) {
toast.fromError(i18n.t("addModel.handleAddAutoRouterSubmit.failed", { error: String(error) }));
}
};

132
ui/litellm-dashboard/src/components/add_model/switchyard_config.test.ts Normal file
View file

@ -0,0 +1,132 @@
import { describe, expect, it } from "vitest";
import {
emptySwitchyardConfig,
isDecisionModelMode,
parseToolSemantics,
setClassifierKind,
stageConfidence,
switchyardSaveError,
switchyardConfigSchema,
switchyardRouterPayload,
toggleClassifier,
} from "./switchyard_config";
describe("Switchyard configuration", () => {
it("changes the confidence default when the classifier is toggled", () => {
const config = emptySwitchyardConfig();
expect(stageConfidence(config)).toBe(0.5);
expect(stageConfidence(toggleClassifier(config, true))).toBe(0.7);
expect(stageConfidence(toggleClassifier(toggleClassifier(config, true), false))).toBe(0.5);
});
it("preserves an admin's confidence choice across toggles and save/reload", () => {
const config = {
...emptySwitchyardConfig(),
models: { capable: "strong", efficient: "fast" },
options: { picker: "efficient_first" as const, confidence_threshold: 0.64 },
};
const enabled = { ...toggleClassifier(config, true), models: { ...config.models, judge: "judge" } };
const saved = switchyardConfigSchema.parse(JSON.parse(JSON.stringify(enabled)));
expect(stageConfidence(saved)).toBe(0.64);
expect(stageConfidence(toggleClassifier(saved, false))).toBe(0.64);
});
it("requires a judge only with a classifier and removes it when disabled", () => {
const config = { ...emptySwitchyardConfig(), models: { capable: "strong", efficient: "fast" } };
expect(switchyardConfigSchema.safeParse(config).success).toBe(true);
expect(switchyardConfigSchema.safeParse(toggleClassifier(config, true)).success).toBe(false);
const enabled = { ...toggleClassifier(config, true), models: { ...config.models, judge: "judge" } };
expect(switchyardConfigSchema.safeParse(enabled).success).toBe(true);
expect(toggleClassifier(enabled, false).models).toEqual(config.models);
});
it.each(["escalation_note", "deescalation_note", "capable_system_prompt", "efficient_system_prompt"])(
"rejects the unsupported rewrite option %s",
(option) => {
expect(
switchyardConfigSchema.safeParse({
models: { capable: "strong", efficient: "fast" },
options: { [option]: "ignored" },
}).success,
).toBe(false);
},
);
it("rejects unsupported algorithms and settings the stage router judge ignores", () => {
const models = { capable: "strong", efficient: "fast", judge: "judge" };
expect(switchyardConfigSchema.safeParse({ ...emptySwitchyardConfig(), algorithm: "random" }).success).toBe(false);
expect(switchyardConfigSchema.safeParse({ models, classifier: { session_affinity: true } }).success).toBe(false);
});
it("reads a stored classifier without a kind as an LLM judge", () => {
const parsed = switchyardConfigSchema.parse({
models: { capable: "strong", efficient: "fast", judge: "judge" },
classifier: { base_threshold: 0.4 },
});
expect(parsed.classifier).toMatchObject({ kind: "llm", base_threshold: 0.4 });
});
it("opens routers saved through the API with nulls and structured decision instructions", () => {
const instructions = { system: "Prefer the capable model for schema migrations" };
const unsetOptions = {
picker: "efficient_first",
confidence_threshold: null,
recent_window: null,
tool_semantics: null,
};
const apiSaved = {
algorithm: "stage_router",
models: { capable: "strong", efficient: "fast", judge: "decider" },
options: unsetOptions,
classifier: { kind: "decision", cutoff: 0.6, instructions, evidence: null },
};
const decision = switchyardConfigSchema.parse(apiSaved);
expect(switchyardSaveError("route", decision)).toBeNull();
expect(switchyardRouterPayload("route", decision).litellm_params.switchyard_router_config.classifier).toMatchObject(
{
instructions,
},
);
const plain = switchyardConfigSchema.parse({
models: { capable: "strong", efficient: "fast", judge: null },
classifier: null,
});
expect(switchyardSaveError("route", plain)).toBeNull();
});
it("accepts tool semantics that list only some categories", () => {
expect(parseToolSemantics('{"observe": ["Grep"], "mutate": ["Deploy"]}')).toEqual({
observe: ["Grep"],
mutate: ["Deploy"],
});
expect(() => parseToolSemantics('{"unknown": ["Grep"]}')).toThrow();
});
it("switches classifier kind with fresh settings and clears a judge of the wrong kind", () => {
const llm = {
...toggleClassifier(emptySwitchyardConfig(), true),
models: { capable: "a", efficient: "b", judge: "gpt" },
};
const decision = setClassifierKind(llm, "decision");
expect(decision.classifier).toEqual({ kind: "decision", cutoff: 0.6 });
expect(decision.models.judge).toBe("");
expect(setClassifierKind(decision, "decision")).toBe(decision);
const saved = switchyardConfigSchema.parse({ ...decision, models: { ...decision.models, judge: "jev" } });
expect(saved.classifier).toEqual({ kind: "decision", cutoff: 0.6 });
});
it("rejects LLM-only settings on a decision classifier and out-of-range cutoffs", () => {
const models = { capable: "a", efficient: "b", judge: "jev" };
expect(
switchyardConfigSchema.safeParse({ models, classifier: { kind: "decision", base_threshold: 0.5 } }).success,
).toBe(false);
expect(switchyardConfigSchema.safeParse({ models, classifier: { kind: "decision", cutoff: 1.2 } }).success).toBe(
false,
);
});
it.each([
["evaluation", true],
["decisions", true],
["chat", false],
[undefined, false],
])("treats mode %s as a decision model: %s", (mode, expected) => {
expect(isDecisionModelMode(mode)).toBe(expected);
});
it("blocks saving without a name or with the router as one of its own targets", () => {
const config = { ...emptySwitchyardConfig(), models: { capable: "strong", efficient: "fast" } };
expect(switchyardSaveError("route", config)).toBeNull();
expect(switchyardSaveError(" ", config)).toBe("Enter a router name");
expect(switchyardSaveError("strong", config)).toBe("The router cannot select itself");
});
});

102
ui/litellm-dashboard/src/components/add_model/switchyard_config.ts Normal file
View file

@ -0,0 +1,102 @@
import { z } from "zod";
const threshold = z.number().min(0).max(1);
const positiveInteger = z.number().int().positive();
const modelName = z
.string()
.trim()
.min(1, "Select a model")
.refine((name) => !name.startsWith("auto_router/"), "Select a concrete model");
const optionsFields = {
picker: z.enum(["efficient_first", "capable_first"]).default("efficient_first"),
confidence_threshold: threshold.nullish(),
recent_window: positiveInteger.nullish(),
tool_semantics: z.partialRecord(z.enum(["observe", "mutate", "plan", "new"]), z.array(z.string())).nullish(),
only_on_wrong_signal_escalation: z.boolean().optional(),
};
const llmClassifierFields = {
kind: z.literal("llm").default("llm"),
base_threshold: threshold.default(0.5),
threshold_step: threshold.default(0),
max_output_tokens: positiveInteger.default(4096),
recent_turn_window: positiveInteger.nullish(),
response_format_type: z.enum(["json_schema", "json_object"]).default("json_schema"),
prompt: z.string().nullish(),
};
const decisionClassifierFields = {
kind: z.literal("decision"),
cutoff: threshold.default(0.6),
instructions: z.union([z.string(), z.record(z.string(), z.unknown())]).nullish(),
evidence: z.record(z.string(), z.unknown()).nullish(),
};
const configFields = {
algorithm: z.literal("stage_router").default("stage_router"),
models: z.object({ capable: modelName, efficient: modelName, judge: modelName.nullish() }).strict(),
options: z.object(optionsFields).strict().default({ picker: "efficient_first" }),
classifier: z.union([z.object(decisionClassifierFields).strict(), z.object(llmClassifierFields).strict()]).nullish(),
};
export const switchyardConfigSchema = z
.object(configFields)
.strict()
.superRefine((value, ctx) => {
if (Boolean(value.classifier) !== Boolean(value.models.judge)) {
ctx.addIssue({
code: "custom",
message: "Select a judge when the classifier is enabled",
path: ["models", "judge"],
});
}
});
export type SwitchyardConfig = z.infer<typeof switchyardConfigSchema>;
export type ClassifierKind = NonNullable<SwitchyardConfig["classifier"]>["kind"];
export const isDecisionModelMode = (mode: string | undefined): boolean => mode === "decisions" || mode === "evaluation";
export const defaultClassifier = (kind: ClassifierKind = "llm"): NonNullable<SwitchyardConfig["classifier"]> =>
kind === "decision"
? { kind: "decision", cutoff: 0.6 }
: {
kind: "llm",
base_threshold: 0.5,
threshold_step: 0,
max_output_tokens: 4096,
response_format_type: "json_schema",
};
export const emptySwitchyardConfig = (): SwitchyardConfig => ({
algorithm: "stage_router",
models: { capable: "", efficient: "" },
options: { picker: "efficient_first" },
});
export const stageConfidence = (config: SwitchyardConfig): number =>
config.options.confidence_threshold ?? (config.classifier ? 0.7 : 0.5);
export const toggleClassifier = (config: SwitchyardConfig, enabled: boolean): SwitchyardConfig => ({
...config,
models: {
capable: config.models.capable,
efficient: config.models.efficient,
...(enabled ? { judge: config.models.judge ?? "" } : {}),
},
classifier: enabled ? config.classifier ?? defaultClassifier() : undefined,
});
export const setClassifierKind = (config: SwitchyardConfig, kind: ClassifierKind): SwitchyardConfig =>
config.classifier?.kind === kind
? config
: { ...config, models: { ...config.models, judge: "" }, classifier: defaultClassifier(kind) };
export function parseToolSemantics(text: string): SwitchyardConfig["options"]["tool_semantics"] {
if (!text.trim()) return undefined;
return optionsFields.tool_semantics.parse(JSON.parse(text));
}
export function switchyardSaveError(name: string, config: SwitchyardConfig): string | null {
if (!name.trim()) return "Enter a router name";
const validated = switchyardConfigSchema.safeParse(config);
if (!validated.success) return validated.error.issues[0].message;
if (Object.values(validated.data.models).includes(name.trim())) return "The router cannot select itself";
return null;
}
export const switchyardRouterPayload = (name: string, config: SwitchyardConfig) => ({
model_name: name.trim(),
litellm_params: { model: "auto_router/switchyard", switchyard_router_config: switchyardConfigSchema.parse(config) },
});

31
ui/litellm-dashboard/src/components/edit_auto_router/edit_auto_router_modal.integration.test.tsx
View file

@ -1649,3 +1649,34 @@ describe("EditAutoRouterModal deployed decision classifier", () => {
},
);
});
describe("EditAutoRouterModal Switchyard", () => {
const SWITCHYARD_MODEL = {
model_name: "stage-router",
litellm_params: {
model: "auto_router/switchyard",
switchyard_router_config: {
algorithm: "stage_router",
models: { capable: "strong", efficient: "fast" },
options: { picker: "efficient_first" },
},
},
model_info: { id: "switchyard-1", access_groups: [] },
};
it("hands the saved router back to the caller and closes, so reopening shows the new values", async () => {
const onSuccess = vi.fn();
const onCancel = vi.fn();
renderModal({ modelData: SWITCHYARD_MODEL, onSuccess, onCancel });
const name = await screen.findByLabelText("Router name");
fireEvent.change(name, { target: { value: "renamed-router" } });
fireEvent.click(screen.getByRole("button", { name: "Save changes" }));
await waitFor(() => expect(onSuccess).toHaveBeenCalledTimes(1));
const saved = onSuccess.mock.calls[0][0];
expect(saved.model_name).toBe("renamed-router");
expect(saved.model_info.id).toBe("switchyard-1");
expect(saved.litellm_params.switchyard_router_config.models).toEqual({ capable: "strong", efficient: "fast" });
expect(onCancel).toHaveBeenCalled();
});
});

43
ui/litellm-dashboard/src/components/edit_auto_router/edit_auto_router_modal.tsx
View file

@ -8,6 +8,7 @@ import { usesClassifierContext } from "../add_model/classifier_types";
export type { StoredComplexityRouterConfig } from "../add_model/build_complexity_router_config";
import { getForecastConfigError, isForecastClassifier } from "../add_model/forecast_classifier_config";
import React, { useEffect, useMemo, useState } from "react";
import SwitchyardRouterForm from "../add_model/SwitchyardRouterForm";
import { useTranslation } from "react-i18next";
import {
complexityRouterSchema,
@ -277,7 +278,7 @@ const EditClassifierTabs = ({
);
};
const EditAutoRouterModal: React.FC<EditAutoRouterModalProps> = ({
const ExistingEditAutoRouterModal: React.FC<EditAutoRouterModalProps> = ({
isVisible,
onCancel,
onSuccess,
@ -824,4 +825,44 @@ const EditAutoRouterModal: React.FC<EditAutoRouterModalProps> = ({
);
};
const EditAutoRouterModal: React.FC<EditAutoRouterModalProps> = (props) => {
if (props.modelData?.litellm_params?.model?.startsWith("auto_router/switchyard")) {
return (
<Dialog
open={props.isVisible}
onOpenChange={(open) => {
if (!open) props.onCancel();
}}
>
<DialogContent className="max-h-[90vh] overflow-y-auto sm:max-w-2xl">
<DialogHeader>
<DialogTitle>Edit Switchyard router</DialogTitle>
</DialogHeader>
<SwitchyardRouterForm
key={props.modelData.model_info.id}
accessToken={props.accessToken}
onSuccess={(saved) => {
props.onSuccess({
...props.modelData,
model_name: saved.model_name,
litellm_params: {
...props.modelData.litellm_params,
switchyard_router_config: saved.switchyard_router_config,
},
});
props.onCancel();
}}
initial={{
name: props.modelData.model_name,
modelId: props.modelData.model_info.id,
config: props.modelData.litellm_params.switchyard_router_config,
}}
/>
</DialogContent>
</Dialog>
);
}
return <ExistingEditAutoRouterModal {...props} />;
};
export default EditAutoRouterModal;

12
ui/litellm-dashboard/src/components/model_info_view.integration.test.tsx
View file

@ -601,6 +601,18 @@ describe("ModelInfoView", () => {
expect(screen.queryByText(/\$0\.00\/1M tokens/)).not.toBeInTheDocument();
});
it("shows the server's reason when saving the model settings is rejected", async () => {
const rejection = new Error("A strategy router cannot route to itself");
mockModelPatchUpdateCall.mockRejectedValueOnce(rejection);
const user = userEvent.setup();
render(<ModelInfoView {...DEFAULT_ADMIN_PROPS} />, { wrapper });
await user.click(await screen.findByRole("button", { name: /edit settings/i }));
await user.click(screen.getByRole("button", { name: /save changes/i }));
await waitFor(() => expect(mockToast.fromError).toHaveBeenCalledWith(rejection));
});
it("should display edit settings button when user can edit model", async () => {
render(<ModelInfoView {...DEFAULT_ADMIN_PROPS} />, { wrapper });
await waitFor(() => {

2
ui/litellm-dashboard/src/components/model_info_view.tsx
View file

@ -402,7 +402,7 @@ export default function ModelInfoView({
setIsEditing(false);
} catch (error) {
console.error("Error updating model:", error);
toast.fromError(t("modelInfoView.toasts.updateFailed"));
toast.fromError(error);
} finally {
setIsSaving(false);
}

22
ui/litellm-dashboard/src/components/networking.test.ts
View file

@ -1245,3 +1245,25 @@ describe("uploadClaudeCodeSkill", () => {
expect(Object.fromEntries(init?.body as FormData)).toEqual({ file, metadata: JSON.stringify(metadata) });
});
});
describe("modelPatchUpdateCall", () => {
const originalFetch = global.fetch;
afterEach(() => {
global.fetch = originalFetch;
});
it("surfaces the server's error message", async () => {
const rejected = {
ok: false,
status: 400,
headers: new Headers(),
text: async () => JSON.stringify({ detail: { error: "tool Read already has built-in semantics" } }),
};
global.fetch = vi.fn().mockResolvedValue(rejected) as unknown as typeof global.fetch;
await expect(Networking.modelPatchUpdateCall("token", { model_name: "router" }, "model-1")).rejects.toThrow(
"tool Read already has built-in semantics",
);
});
});

31
ui/litellm-dashboard/src/components/networking.tsx
View file

@ -2587,36 +2587,7 @@ export const modelPatchUpdateCall = async (
accessToken: string,
formValues: Record<string, any>, // Assuming formValues is an object
modelId: string,
) => {
try {
// Intentionally not logging the payload: it can contain freshly-entered
// provider secrets (api_key, vertex_credentials, AWS creds).
const url = proxyBaseUrl ? `${proxyBaseUrl}/model/${modelId}/update` : `/model/${modelId}/update`;
const response = await fetch(url, {
method: "PATCH",
headers: {
[globalLitellmHeaderName]: `Bearer ${accessToken}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
...formValues, // Include formValues in the request body
}),
});
if (!response.ok) {
const errorData = await response.text();
handleError(errorData);
console.error("Error update from the server:", errorData);
throw new Error("Network response was not ok");
}
const data = await response.json();
return data;
// Handle success - you might want to update some state or UI based on the created key
} catch (error) {
console.error("Failed to update model:", error);
throw error;
}
};
) => apiClient.patch(`/model/${encodeURIComponent(modelId)}/update`, { accessToken, body: formValues });
export interface Member {
role: string;

12
ui/litellm-dashboard/src/components/update_model_credentials_modal.test.tsx
View file

@ -68,6 +68,18 @@ describe("UpdateModelCredentialsModal", () => {
expect(onCancel).toHaveBeenCalledTimes(1);
});
it("shows the server's reason when the update is rejected", async () => {
const rejection = new Error("A strategy router cannot route to itself");
mockModelPatchUpdateCall.mockRejectedValueOnce(rejection);
const user = userEvent.setup();
renderModal();
fireEvent.change(screen.getByLabelText(/new api key/i), { target: { value: "sk-rotated-1" } });
await user.click(screen.getByRole("button", { name: /update api key/i }));
await waitFor(() => expect(mockToast.fromError).toHaveBeenCalledWith(rejection));
});
it("does not call the update API when the field is left blank", async () => {
const user = userEvent.setup();
renderModal();

2
ui/litellm-dashboard/src/components/update_model_credentials_modal.tsx
View file

@ -65,7 +65,7 @@ export default function UpdateModelCredentialsModal({
onCancel();
} catch (error) {
console.error("Error updating API key:", error);
toast.fromError(t("updateModelCredentialsModal.updateFailed"));
toast.fromError(error);
} finally {
setIsSaving(false);
}

14
ui/litellm-dashboard/src/i18n/locales/en.json
View file

@ -17414,12 +17414,12 @@
"menu": {
"unlimited": "Unlimited"
},
"title": "Choose your classifier",
"titleTooltip": "Choose how requests are evaluated and routed to your tiers",
"aboutClassifiers": "About classifiers",
"title": "Choose your routing strategy",
"titleTooltip": "Choose how each request is evaluated and which model it is routed to",
"aboutClassifiers": "About routing strategies",
"families": {
"heuristics": "Heuristics",
"heuristicsDescription": "Classify locally, with no API call",
"heuristicsDescription": "Default. Classify locally, with no API call",
"llmDescription": "Use a judge model to choose a solver",
"decisionModel": "Decisions Model",
"decisionModelDescription": "Use a decision model to choose a tier"
@ -18443,9 +18443,9 @@
"hardDescription": "Your stronger model",
"roleModel": "{{label}} model",
"selectModel": "Select a deployed model",
"needTwoModels": "Two deployed models are needed. Add a model or use Advanced configuration.",
"needTwoModels": "Two deployed models are needed. Add a model or customize tiers and keywords.",
"chooseTwoModels": "Choose two models to get started.",
"advancedConfiguration": "Advanced configuration"
"advancedConfiguration": "Customize tiers and keywords"
},
"routerConfigBuilder": {
"removeUtterance": "Remove {{utterance}}",
@ -18751,7 +18751,6 @@
"updateModelCredentialsModal": {
"apiKeyRequired": "Enter a new API key",
"updated": "API key updated",
"updateFailed": "Failed to update API key",
"title": "Update API Key",
"description": "Update this model's API key. Only the new key is sent; the rest of the deployment configuration is left untouched.",
"apiKeyOnlyWarning": "Only the API key is rotated here. Models that authenticate with an Azure AD token, AWS credentials, or a Vertex service-account JSON aren't supported yet; update those from the model's LiteLLM Params for now.",
@ -19607,7 +19606,6 @@
"invalidLitellmParams": "LiteLLM Params must be a valid JSON object",
"invalidModelInfo": "Invalid JSON in Model Info",
"updateSuccess": "Model settings updated successfully",
"updateFailed": "Failed to update model settings",
"noComplexityTiers": "No complexity tiers are configured yet, so there is nothing to test.",
"testingConnection": "Testing connection...",
"connectionSuccess": "Connection test successful!",

18
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -29997,6 +29997,8 @@ export interface components {
ChatCompletionToolMessage: {
/** Content */
content: string | (components["schemas"]["ChatCompletionTextObject"] | components["schemas"]["ChatCompletionImageObject"] | components["schemas"]["ChatCompletionAudioObject"] | components["schemas"]["ChatCompletionFileObject"] | components["schemas"]["ChatCompletionToolReferenceObject"])[];
/** Is Error */
is_error?: boolean;
/**
* Role
* @constant
@ -36850,6 +36852,10 @@ export interface components {
stream_timeout_by_service_tier?: {
[key: string]: number;
} | null;
/** Switchyard Router Config */
switchyard_router_config?: {
[key: string]: unknown;
} | null;
/** Tag Regex */
tag_regex?: string[] | null;
/** Tags */
@ -47664,7 +47670,7 @@ export interface components {
* Router Type
* @enum {string}
*/
router_type?: "complexity" | "adaptive" | "quality" | "semantic";
router_type?: "complexity" | "adaptive" | "quality" | "semantic" | "switchyard";
/** Savings Baseline Deployment Id */
savings_baseline_deployment_id?: string;
/** Savings Baseline Model */
@ -47673,6 +47679,12 @@ export interface components {
score?: number;
/** Signals */
signals?: string[];
/** Switchyard Algorithm */
switchyard_algorithm?: string;
/** Switchyard Evidence */
switchyard_evidence?: unknown;
/** Switchyard Fallback Reason */
switchyard_fallback_reason?: string;
/** Tier */
tier?: string;
tier_boundaries?: components["schemas"]["StandardLoggingRoutingDecisionTierBoundaries"];
@ -52966,6 +52978,10 @@ export interface components {
stream_timeout_by_service_tier?: {
[key: string]: number;
} | null;
/** Switchyard Router Config */
switchyard_router_config?: {
[key: string]: unknown;
} | null;
/** Tag Regex */
tag_regex?: string[] | null;
/** Tags */

24
uv.lock generated
View file

@ -1,5 +1,5 @@
version = 1
revision = 5
revision = 3
requires-python = ">=3.10, <3.16"
resolution-markers = [
"python_full_version >= '3.15'",
@ -5168,6 +5168,9 @@ stt-nvidia-riva = [
stt-vertex-chirp = [
{ name = "google-cloud-speech" },
]
switchyard = [
{ name = "nemo-switchyard" },
]
utils = [
{ name = "numpydoc" },
]
@ -5277,6 +5280,7 @@ proxy-dev = [
{ name = "httptools" },
{ name = "hypercorn" },
{ name = "litellm-admin-mcp", marker = "python_full_version >= '3.12'" },
{ name = "nemo-switchyard" },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp" },
{ name = "opentelemetry-instrumentation-fastapi" },
@ -5351,6 +5355,7 @@ requires-dist = [
{ name = "mcp", marker = "extra == 'mcp'", specifier = ">=2.3.0,<3" },
{ name = "mcp", marker = "extra == 'proxy'", specifier = ">=2.3.0,<3" },
{ name = "mlflow", marker = "extra == 'mlflow'", specifier = ">=3.17.2,<4.0" },
{ name = "nemo-switchyard", marker = "extra == 'switchyard'", specifier = "==0.4.0a1" },
{ name = "numpy", marker = "extra == 'stt-nvidia-riva'", specifier = ">=1.26.0" },
{ name = "numpydoc", marker = "extra == 'utils'", specifier = ">=1.8.0,<2.0" },
{ name = "openai", specifier = ">=2.20.0,<4.0.0" },
@ -5407,7 +5412,7 @@ requires-dist = [
{ name = "uvloop", marker = "sys_platform != 'win32' and extra == 'proxy'", specifier = ">=0.22.1,<1.0" },
{ name = "websockets", marker = "extra == 'proxy'", specifier = ">=15.0.1,<16.0" },
]
provides-extras = ["proxy", "cli", "extra-proxy", "utils", "mcp", "saml", "semantic-router", "mlflow", "grpc", "stt-vertex-chirp", "stt-nvidia-riva", "google", "bedrock-realtime", "proxy-runtime"]
provides-extras = ["switchyard", "proxy", "cli", "extra-proxy", "utils", "mcp", "saml", "semantic-router", "mlflow", "grpc", "stt-vertex-chirp", "stt-nvidia-riva", "google", "bedrock-realtime", "proxy-runtime"]
[package.metadata.requires-dev]
admin-mcp = [{ name = "litellm-admin-mcp", marker = "python_full_version >= '3.12'", url = "https://github.com/BerriAI/liteadmin-mcp/archive/31fb096904ae05684c0779110a93b135b5601182.tar.gz" }]
@ -5511,6 +5516,7 @@ proxy-dev = [
{ name = "httptools", specifier = ">=0.7.1,<0.8" },
{ name = "hypercorn", specifier = "==0.17.3" },
{ name = "litellm-admin-mcp", marker = "python_full_version >= '3.12'", url = "https://github.com/BerriAI/liteadmin-mcp/archive/31fb096904ae05684c0779110a93b135b5601182.tar.gz" },
{ name = "nemo-switchyard", specifier = "==0.4.0a1" },
{ name = "opentelemetry-api", specifier = "==1.45.1" },
{ name = "opentelemetry-exporter-otlp", specifier = "==1.45.1" },
{ name = "opentelemetry-instrumentation-fastapi", specifier = "==0.66b1" },
@ -6579,6 +6585,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" },
]
[[package]]
name = "nemo-switchyard"
version = "0.4.0a1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/8d/f8/088042a9e7152402384107a900d252f9aae1a9ad92764b1f866387413929/nemo_switchyard-0.4.0a1.tar.gz", hash = "sha256:8399becc03cc6f77e4885a95f70c0565d9aba27f15bfea6758596fdf81043097", size = 768645, upload-time = "2026-10-29T22:20:19.364Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0d/38/e1b2473dac6c847d7513c17d75eca220d8a9088b991c5fb73a4934791027/nemo_switchyard-0.4.0a1-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:1f34de8c0a4626c34b566db8b55c9591acda5121e249b8fffc9010cbe4578d3f", size = 9101897, upload-time = "2026-10-29T22:20:08.381Z" },
{ url = "https://files.pythonhosted.org/packages/38/82/4e6cefc7db47967f903718d54e7b08b9c48ffbd71f74f3dc06fc37fbe0ee/nemo_switchyard-0.4.0a1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:2c9777180635fceb30ff6144ec03443337fc0ab0e008b9ad25ab6802433214a7", size = 8796148, upload-time = "2026-10-29T22:20:10.886Z" },
{ url = "https://files.pythonhosted.org/packages/7c/6b/abc7e73629e4d34ad2c923e63996bbdcbb3c96e515c44fdd2b5734688e45/nemo_switchyard-0.4.0a1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a943433bd0294e86942f9c29ffc6ff6ad2717c27619a59dd730635603f1dea12", size = 9841418, upload-time = "2026-10-29T22:20:12.703Z" },
{ url = "https://files.pythonhosted.org/packages/a2/92/c53d48d794fc11c59c1219f8839f7313fa9d499e75753d22a58407be2535/nemo_switchyard-0.4.0a1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a63b3d26b49802827dec3ba6a187765c0b6e26b732bf0a16de59f593ad32a9e", size = 9709820, upload-time = "2026-10-29T22:20:14.612Z" },
{ url = "https://files.pythonhosted.org/packages/ff/95/058f0fbb48670c904fe2c93b8d9959073402293cc6dfbb6ac074b70c2c3f/nemo_switchyard-0.4.0a1-cp310-abi3-win_amd64.whl", hash = "sha256:ae42923283f3b88ebf0a92d3161729654a9987a1208478498e764e06cb8bd7dc", size = 8273177, upload-time = "2026-10-29T22:20:16.372Z" },
{ url = "https://files.pythonhosted.org/packages/7c/52/7561cc1988733e87fce82df8cf4a52beaa109e6b0bf5eff52661db113f97/nemo_switchyard-0.4.0a1-cp310-abi3-win_arm64.whl", hash = "sha256:6e7517ff21b7c8a4f1617e84e45ed03686ed30c31709c0a6d3be8b8f4a50ae23", size = 7910792, upload-time = "2026-10-29T22:20:17.997Z" },
]
[[package]]
name = "nodeenv"
version = "1.10.0"