fix(router): address fusion review findings

cookbook/litellm_router/fusion_models.md

@@ -1,157 +0,0 @@
-# Fusion models (beta)
-
-Fusion models expose several LiteLLM model groups as one model. For each client call, every panel model receives the
-same canonical conversation and runs in parallel. An aggregator then synthesizes their work into the sole response
-returned to the client
-
-Fusion operates at the model layer. Coding agents, research loops, chat applications, and tool-using workflows keep
-their current control flow and use the Fusion model name anywhere they would name one model
-
-```text
-client or harness
-      |
-      | one model request
-      v
-panel A ----\
-panel B -----+--> aggregator --> one response or tool call
-panel C ----/
-```
-
-If the aggregator returns a tool call, the existing harness executes it, appends the result to its conversation, and
-calls the Fusion model again. That next call starts a new panel round. Fusion does not execute tools, retain private
-panel transcripts, create subagents, or replace the harness
-
-## Create a Fusion model
-
-In the dashboard, open **Models & Endpoints**, select **Fusion Models (Beta)**, and choose **Add Fusion Model**. Select
-two to six existing model groups for the panel and one existing model group as the aggregator. LiteLLM rejects nested
-Fusion models
-
-The equivalent `config.yaml` entry is:
-
-```yaml
-model_list:
-  - model_name: panel/one
-    litellm_params:
-      model: openai/gpt-5
-
-  - model_name: panel/two
-    litellm_params:
-      model: anthropic/claude-sonnet-4-5-20250929
-
-  - model_name: fusion-aggregator
-    litellm_params:
-      model: openai/gpt-5
-
-  - model_name: fusion/coding
-    litellm_params:
-      model: fusion_router
-      fusion_router_config:
-        panel_models:
-          - panel/one
-          - panel/two
-        aggregator_model: fusion-aggregator
-        min_successful_panelists: 2
-        panel_timeout_seconds: 120
-        max_candidate_chars: 12000
-        on_quorum_failure: fail
-```
-
-Clients call `fusion/coding` like a regular model. The beta supports Chat Completions, Responses, and Anthropic
-Messages, including async streaming. Panels finish before the aggregator starts, so the first streamed token arrives
-during aggregation
-
-## Presets and settings
-
-The dashboard offers two behavior presets:
-
-- **Quality First** sets `on_quorum_failure: fail`. The request fails when fewer than `min_successful_panelists` panel
-  calls succeed, preserving the configured quality floor
-- **High Availability** sets `on_quorum_failure: aggregator_only`. When the panel misses quorum, the aggregator receives
-  the original request without partial candidates and answers alone
-
-Advanced settings stay limited to the controls that affect one Fusion round:
-
-| Setting | Meaning | Bounds |
-| --- | --- | --- |
-| `min_successful_panelists` | Successful panel responses required before synthesis | 1 to panel size |
-| `panel_timeout_seconds` | Deadline applied to each panel call | More than 0, at most 600 seconds |
-| `max_candidate_chars` | Text copied from each candidate into the synthesis request | 1,000 to 50,000 characters |
-
-The default uses Quality First with a quorum of two, a 120-second panel deadline, and 12,000 characters per candidate.
-LiteLLM runs every configured panel member because this feature optimizes answer quality rather than call cost
-
-## Tools and active work
-
-LiteLLM sends client-defined function schemas to every panel member. A panel can reason about available actions and
-propose a function name and arguments. LiteLLM serializes those proposals as untrusted advice, discards their call IDs,
-and never returns or executes them
-
-The aggregator receives the original tools and has sole authority to emit a tool call. It creates the call and arguments
-after considering the panel. LiteLLM withholds provider-hosted tools such as hosted web search from panel members because
-those tools execute inside the provider. The aggregator retains them
-
-A coding or active-task harness follows this loop:
-
-1. The harness sends its transcript, context, and tool schemas to `fusion/coding`
-2. Panel members propose answers, edits, commands, or tool use in isolation
-3. The aggregator synthesizes one response or tool call
-4. The harness executes the call, records the result, and invokes `fusion/coding` again
-
-Research applications use the same loop. Fusion improves the model decision on each call while the application owns
-browsing, citation collection, retries, approvals, and its completion criteria
-
-## Conversation history and compaction
-
-Every panel member receives the complete message list supplied on that call. The aggregator receives that list plus one
-developer message containing bounded panel candidates. LiteLLM inserts the developer message after leading system and
-developer instructions so the insertion preserves assistant/tool adjacency in the transcript
-
-Only the aggregator output enters client-visible history. Panel outputs last for one Fusion round and are not replayed
-on later turns. The canonical conversation therefore matches the transcript a client would retain for one model, and
-every later panel round sees it. Replaying panel reasoning would multiply context use and create conflicting histories
-
-Fusion does not compact across turns. If the client or harness summarizes or truncates the canonical conversation,
-every panel member and the aggregator see that compacted transcript on the next call. Anthropic Messages context
-management runs before the same Fusion core
-
-## Failure, health, and observability
-
-Panel calls fail independently. A provider error, timeout, streaming response where Fusion expected a complete
-candidate, or empty response counts as one failed panelist. Quality First requires a healthy aggregator and enough
-healthy panel dependencies to meet quorum. High Availability requires a healthy aggregator and still reports each
-panel's health without taking the virtual model down
-
-Each child provider call keeps its LiteLLM logging and spend record. Panel calls include
-`internal_call_origin: fusion_panel`; the aggregator remains the authoritative call for the parent request. A successful
-Fusion round makes one billable call per panelist plus one aggregator call
-
-## Beta boundaries
-
-- Fusion supports `n=1`. LiteLLM rejects multiple returned choices because Fusion must produce one authoritative result
-- A Fusion model cannot serve as a panel member or aggregator for another Fusion model
-- Responses background jobs are unsupported
-- Synchronous Python streaming through `Router.completion` and `Router.responses` is unsupported. Use their async
-  counterparts. Proxy streaming uses the async paths
-- Embeddings, image generation, audio, batch jobs, and other non-conversational endpoints bypass Fusion. The feature
-  covers conversational model calls and tool loops rather than every LiteLLM API type
-
-These boundaries keep each model call deterministic: one parallel panel round runs before one aggregation, and the
-caller retains task lifecycle control
-
-## Design assumptions
-
-The implementation starts with five testable assumptions:
-
-1. The aggregator should synthesize the panel's work. Its prompt permits combining, correcting, rejecting, or replacing
-   candidates and preserving supported minority observations
-2. Aggregating on every model call gives operators one predictable policy. The first beta has no hidden cadence or
-   conductor-owned state
-3. The canonical client transcript is the sole durable history. Private panel histories would create divergent agents,
-   which belongs in a harness
-4. Function-tool awareness helps coding and active work, while one aggregator retains execution authority
-5. Quality is the primary optimization target. LiteLLM exposes cost and latency as consequences rather than routing
-   inputs
-
-Evaluations can vary panel composition, aggregator choice, quorum, and failure preset without changing the API contract
-or the harness under test

litellm/proxy/spend_tracking/budget_reservation.py

@@ -1012,7 +1012,7 @@ def estimate_request_max_cost(
     input_token_counts: Mapping[str, int] | None = None,
 ) -> float | None:
     estimates = [
-        _estimate_request_max_cost_for_model(
+        _estimate_request_model_max_cost(
             request_body=request_body,
             route=route,
             model=model_name,
@@ -1027,6 +1027,64 @@ def estimate_request_max_cost(
     return max(cast(list[float], estimates))
 
 
+def _estimate_request_model_max_cost(
+    request_body: dict,
+    route: str,
+    model: str,
+    llm_router: Router | None,
+    input_tokens: int | None = None,
+) -> float | None:
+    """Estimate one selectable model, expanding Fusion into every billable child call."""
+    registered_model_name: Final = (
+        llm_router._get_model_from_alias(model=model) or model  # pyright: ignore[reportPrivateUsage]  # admission must price the routed group
+        if llm_router is not None
+        else model
+    )
+    fusion_router: Final = (
+        llm_router.fusion_routers.get(registered_model_name) if llm_router is not None else None
+    )
+    if fusion_router is None:
+        return _estimate_request_max_cost_for_model(
+            request_body=request_body,
+            route=route,
+            model=model,
+            llm_router=llm_router,
+            input_tokens=input_tokens,
+        )
+
+    panel_estimates: Final = tuple(
+        _estimate_request_max_cost_for_model(
+            request_body=request_body,
+            route=route,
+            model=panel_model,
+            llm_router=llm_router,
+        )
+        for panel_model in fusion_router.config.panel_models
+    )
+    original_aggregator_tokens: Final = _count_input_tokens(
+        request_body=request_body,
+        model=fusion_router.config.aggregator_model,
+    )
+    # Four tokens per bounded character plus fixed protocol headroom safely covers
+    # candidate serialization without materializing a synthetic prompt at admission.
+    candidate_token_ceiling: Final = (
+        4 * fusion_router.config.max_candidate_chars * len(fusion_router.config.panel_models)
+    ) + 1024
+    aggregator_input_tokens: Final = (
+        original_aggregator_tokens + candidate_token_ceiling if original_aggregator_tokens is not None else None
+    )
+    aggregator_estimate: Final = _estimate_request_max_cost_for_model(
+        request_body=request_body,
+        route=route,
+        model=fusion_router.config.aggregator_model,
+        llm_router=llm_router,
+        input_tokens=aggregator_input_tokens,
+    )
+    child_estimates: Final = (*panel_estimates, aggregator_estimate)
+    known_estimates: Final = tuple(estimate for estimate in child_estimates if estimate is not None)
+    return sum(known_estimates) if known_estimates else None
+
+
 def estimate_request_input_cost(
     request_body: dict,
     route: str,

tests/test_litellm/proxy/test_budget_reservation.py

@@ -1048,6 +1048,60 @@ async def test_should_reserve_tiered_pricing_cost(spend_counter_state):
     await release_budget_reservation(reservation)
 
 
+def test_fusion_reservation_sums_panels_and_candidate_inflated_aggregator() -> None:
+    router = Router(
+        model_list=[
+            {
+                "model_name": "panel-a",
+                "litellm_params": {"model": "openai/panel-a", "api_key": "fake"},
+            },
+            {
+                "model_name": "panel-b",
+                "litellm_params": {"model": "openai/panel-b", "api_key": "fake"},
+            },
+            {
+                "model_name": "aggregator",
+                "litellm_params": {"model": "openai/aggregator", "api_key": "fake"},
+            },
+            {
+                "model_name": "fusion/test",
+                "litellm_params": {
+                    "model": "fusion_router",
+                    "fusion_router_config": {
+                        "panel_models": ["panel-a", "panel-b"],
+                        "aggregator_model": "aggregator",
+                        "max_candidate_chars": 1000,
+                    },
+                },
+            },
+        ]
+    )
+    request_body = {
+        "model": "fusion/test",
+        "messages": [{"role": "user", "content": "hello"}],
+        "max_tokens": 10,
+    }
+
+    def child_estimate(*, model: str, input_tokens: int | None = None, **_: object) -> float:
+        if model == "aggregator":
+            assert input_tokens is not None
+            assert input_tokens >= 9000
+            return 3.0
+        return {"panel-a": 1.0, "panel-b": 2.0}[model]
+
+    with patch(
+        "litellm.proxy.spend_tracking.budget_reservation._estimate_request_max_cost_for_model",
+        side_effect=child_estimate,
+    ):
+        estimated = estimate_request_max_cost(
+            request_body=request_body,
+            route="/chat/completions",
+            llm_router=router,
+        )
+
+    assert estimated == pytest.approx(6.0)
+
+
 def test_tiered_reservation_is_all_or_nothing_with_output_tier_from_input_length():
     """Dashscope tiered pricing is all-or-nothing: the tier is chosen by the total
     input tokens and every token (input and output) is billed at that tier's rate.
@@ -2264,6 +2318,7 @@ async def test_should_reserve_all_budgeted_counters(spend_counter_state):
             proxy_logging_obj=proxy_logging_obj,
         )
 
+    assert reservation is not None
     assert (
         counter_cache.in_memory_cache.get_cache(key="spend:key:key-budget-all") == 0.3
     )

tests/test_litellm/test_fusion_router.py

@@ -1,7 +1,7 @@
 import asyncio
 import json
 from collections.abc import Mapping
-from typing import Final
+from typing import Final, cast
 
 import pytest
 
@@ -157,6 +157,8 @@ async def test_panel_gets_only_function_schemas_and_aggregator_owns_tool_call()
     aggregator_call = completion.calls[-1]
     assert aggregator_call["tools"] == [function_tool, hosted_tool]
     assert aggregator_call["tool_choice"] == hosted_tool_choice
+    aggregator_metadata = cast(Mapping[str, object], aggregator_call["litellm_metadata"])
+    assert aggregator_metadata["user_api_key_budget_reservation"] == {"id": "must-not-propagate"}
     aggregator_messages = aggregator_call["messages"]
     assert isinstance(aggregator_messages, list)
     instruction = str(aggregator_messages[0]["content"])
@@ -306,6 +308,14 @@ async def test_router_registers_and_executes_fusion_deployment() -> None:
     assert response.choices[0].message.content == "Final"
     deployment = router.get_deployment(model_id=router.model_list[-1]["model_info"]["id"])
     assert deployment is not None
+
+    router._unregister_fusion_router_for_deployment(  # pyright: ignore[reportPrivateUsage]  # regression covers registry lifecycle
+        deployment
+    )
+    assert "fusion/test" not in router.fusion_routers
+    router.init_fusion_router_deployment(deployment)
+    assert "fusion/test" in router.fusion_routers
+
     router.delete_deployment(id=deployment.model_info.id)
     assert "fusion/test" not in router.fusion_routers
 
@@ -341,8 +351,12 @@ async def test_router_responses_api_bridges_through_the_same_fusion_model() -> N
     router = Router(model_list=_router_model_list())
 
     response = await router.aresponses(model="fusion/test", input="Answer")
+    direct_response = await router._fusion_aware_aresponses(  # pyright: ignore[reportPrivateUsage]  # regression covers the Fusion bridge
+        model="fusion/test", input="Answer"
+    )
 
     assert response.output[0].content[0].text == "Final"
+    assert direct_response.output[0].content[0].text == "Final"
     with pytest.raises(litellm.BadRequestError, match="Background Responses"):
         await router.aresponses(model="fusion/test", input="Answer", background=True)
 
@@ -361,17 +375,27 @@ async def test_router_anthropic_messages_bridges_through_the_same_fusion_model()
         messages=[{"role": "user", "content": "Answer"}],
         max_tokens=256,
     )
+    direct_response = await router._fusion_aware_aanthropic_messages(  # pyright: ignore[reportPrivateUsage]  # regression covers the Fusion bridge
+        model="fusion/test",
+        messages=[{"role": "user", "content": "Answer"}],
+        max_tokens=256,
+    )
 
     assert response["content"][0]["text"] == "Final"
     assert alias_response["content"][0]["text"] == "Final"
+    assert direct_response["content"][0]["text"] == "Final"
 
 
 def test_sync_responses_api_supports_nonstreaming_fusion() -> None:
     router = Router(model_list=_router_model_list())
 
     response = router.responses(model="fusion/test", input="Answer")
+    direct_response = router._fusion_aware_responses(  # pyright: ignore[reportPrivateUsage]  # regression covers the Fusion bridge
+        model="fusion/test", input="Answer"
+    )
 
     assert response.output[0].content[0].text == "Final"
+    assert direct_response.output[0].content[0].text == "Final"
     with pytest.raises(litellm.BadRequestError, match="Synchronous Responses streaming"):
         router.responses(model="fusion/test", input="Answer", stream=True)
 

ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/FusionModels/FusionModelsPanel.test.tsx

@@ -62,7 +62,14 @@ const existingDeployment = {
 
 const renderPanel = (createScope: "unscoped-ok" | "team-required" = "unscoped-ok") =>
   render(
-    <FusionModelsPanel accessToken="token" userRole="Admin" userID="user-1" teams={[]} createScope={createScope} />,
+    <FusionModelsPanel
+      accessToken="token"
+      userRole="Admin"
+      userID="user-1"
+      isViewOnly={false}
+      teams={[]}
+      createScope={createScope}
+    />,
   );
 
 describe("FusionModelsPanel", () => {

ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/FusionModels/FusionModelsPanel.tsx

@@ -34,6 +34,7 @@ interface FusionModelsPanelProps {
   accessToken: string;
   userRole: string;
   userID: string | null;
+  isViewOnly: boolean;
   teams: Team[] | null;
   createScope: ModelWriteScope;
 }
@@ -293,7 +294,14 @@ function FusionModelDialog({
   );
 }
 
-export function FusionModelsPanel({ accessToken, userRole, userID, teams, createScope }: FusionModelsPanelProps) {
+export function FusionModelsPanel({
+  accessToken,
+  userRole,
+  userID,
+  isViewOnly,
+  teams,
+  createScope,
+}: FusionModelsPanelProps) {
   const { data: deployments, isLoading } = useFusionRouters();
   const availableModels = usePlainModelGroups();
   const invalidateFusionRouters = useInvalidateFusionRouters();
@@ -305,7 +313,7 @@ export function FusionModelsPanel({ accessToken, userRole, userID, teams, create
   const modelOptions = useMemo(() => Array.from(availableModels).sort(), [availableModels]);
 
   const canModify = (deployment: AutoRouterDeployment) =>
-    canModifyModel({ userRole, userID }, teams, {
+    canModifyModel({ userRole, userID, isViewOnly }, teams, {
       teamId: deployment.model_info?.team_id,
       isDbModel: deployment.model_info?.db_model === true,
     });

ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/components/FusionModels/fusionModelConfig.test.ts

@@ -55,19 +55,20 @@ describe("Fusion model configuration", () => {
   });
 
   it("parses stored configs defensively and supplies stable defaults", () => {
-    expect(
-      parseFusionConfig({
-        panel_models: ["a", "a", "b", 4],
-        aggregator_model: "judge",
-        on_quorum_failure: "aggregator_only",
-      }),
-    ).toEqual({
+    const expectedConfig = {
       panel_models: ["a", "b"],
       aggregator_model: "judge",
       min_successful_panelists: 2,
       panel_timeout_seconds: 120,
       max_candidate_chars: 12000,
       on_quorum_failure: "aggregator_only",
-    });
+    };
+    expect(
+      parseFusionConfig({
+        panel_models: ["a", "a", "b", 4],
+        aggregator_model: "judge",
+        on_quorum_failure: "aggregator_only",
+      }),
+    ).toEqual(expectedConfig);
   });
 });

ui/litellm-dashboard/src/app/(dashboard)/models-and-endpoints/panels/FusionModelsTabPanel.tsx

@@ -9,12 +9,12 @@ import { modelCreationScope } from "@/utils/modelPermissions";
 import { FusionModelsPanel } from "../components/FusionModels/FusionModelsPanel";
 
 export default function FusionModelsTabPanel() {
-  const { accessToken, userRole, userId: userID } = useAuthorized();
+  const { accessToken, userRole, userId: userID, isViewOnly } = useAuthorized();
   const { data: teams } = useTeams();
   const { data: uiSettings } = useUISettings();
   const isInternalUser = userRole != null && internalUserRoles.includes(userRole);
   const scope = modelCreationScope(
-    { userRole, userID },
+    { userRole, userID, isViewOnly },
     {
       teams: teams ?? null,
       disabledForInternalUsers: isInternalUser && uiSettings?.values?.disable_model_add_for_internal_users === true,
@@ -26,6 +26,7 @@ export default function FusionModelsTabPanel() {
       accessToken={accessToken}
       userRole={userRole ?? ""}
       userID={userID ?? null}
+      isViewOnly={isViewOnly}
       teams={teams ?? null}
       createScope={scope}
     />