mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-13 08:52:02 +00:00
feat(proxy): publish opted-in pass-through endpoints on the Model Hub (#43905)
* feat(proxy): publish opted-in pass-through endpoints on the Model Hub Pass-through endpoints gain display_name and show_in_model_hub. Endpoints that opt in appear on /public/v1/model_hub and /public/model_hub as rows in mode passthrough, named by their display name or their path, with the route in pass_through_path. The admin UI exposes both fields on the create and edit forms and the public hub renders passthrough rows with a route-specific curl example Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): carry pass-through methods onto hub rows and key dashboard rows by route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep health, cost and shell quoting correct for pass-through hub rows Pass-through rows no longer inherit a router model's health check when they share its name, the public table shows n/a instead of Free for their token costs, and the generated curl example escapes shell characters in the route Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): quote the method in the pass-through hub curl example A configured method string is placed after curl -X inside escaped double quotes so it cannot alter the copied command Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): build pass-through hub fixtures through a helper Keeps the dashboard lint budget for inline object arguments under its limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): audit the pass-through Model Hub surface end to end Cover forwarding, pagination, validation, auth, updates, duplicate names, config-file precedence, health-name isolation, repeated reads and two chaos cells (create/delete churn, a killed proxy worker). Build the stored pass_through_endpoints list in one shot instead of appending to the list the config reader handed back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep concurrent pass-through creates from overwriting each other Creating an endpoint appends to the general_settings list the proxy already holds instead of persisting a private copy, so two admins creating at the same time both land, as on main before this branch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): serialize pass-through endpoint creates behind a process lock Each create copies the stored list under an asyncio lock before persisting, so simultaneous creates in one worker no longer overwrite each other and a rejected write (config-owned path) never lands in the in-memory settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): collect paginated pass-through rows without mutating a list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): let a pass-through with no stored timeout save from the edit form Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): type the pass-through edit fixture so a null timeout compiles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(proxy): read published pass-throughs from the in-memory settings store The public Model Hub routes read the pass-through list this worker already serves from proxy_config.settings instead of querying the config table on every anonymous hub read. Also drops a mutable-ok suppression that no longer guards a violation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): list a config file pass-through beside database rows on the hub Main merges database pass-throughs with config file ones on other paths since #43962, so a database write beside a config entry is accepted rather than rejected as config-owned Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type the monkeypatch fixture in the pass-through hub tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): validate pass-through config entries instead of splatting unknown dicts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): list model hub rows as a tagged union of model groups and pass-throughs Health enrichment and the no-router check now match on ModelGroupRow or PassThroughRow instead of testing pass_through_path in each route. Both hub routes share published_hub_rows, the v1 row helper is sync again, and the config list route goes back to main's inline merge since the hub no longer needs configured_pass_through_endpoints * test(ui): find the pass-through hub row by role to stay under the no-node-access budget * fix(proxy): keep database pass-through endpoints in the settings store across config reloads A config reload cleared the stored database row before rebuilding the pass-through list, so without store_model_in_db the list fell back to the config file entries alone. The Admin UI triggers that reload on page load through /get/config/callbacks, which dropped UI-created endpoints from the Model Hub until the next write * refactor(proxy): make the hub row matches exhaustive with assert_never * refactor(proxy): return the exhaustive match fallthrough and build hub test inputs in one shot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop the in-process create lock from pass-through endpoints The lock only covered one worker, and the read-modify-write on the general_settings blob is a separate storage problem tracked as tech debt. Create goes back to main's behavior. * fix: align pass-through model hub with current lint rules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: share worker state for pass-through hub restart coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: show pass-through curl examples in model hub inspector Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: skip invalid pass-through hub entries and await peer refresh Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot]
GitHub
parent
f10b42b1ce
commit
382588a365
23 changed files with 1256 additions and 148 deletions
|
|
@ -2348,6 +2348,14 @@ class PassThroughGenericEndpoint(LiteLLMPydanticObjectBase):
|
|||
default=None,
|
||||
description="List of HTTP methods this endpoint handles (e.g., ['GET', 'POST']). If None or empty, all methods (GET, POST, PUT, DELETE, PATCH) are supported for backward compatibility. This allows the same path to have different targets for different HTTP methods.",
|
||||
)
|
||||
display_name: str | None = Field(
|
||||
default=None,
|
||||
description="Human-readable name for this endpoint. The Model Hub lists the endpoint under this name, falling back to the path when unset.",
|
||||
)
|
||||
show_in_model_hub: bool = Field(
|
||||
default=False,
|
||||
description="If True, this endpoint is published on the public Model Hub as a row in mode 'passthrough'.",
|
||||
)
|
||||
|
||||
|
||||
class PassThroughEndpointResponse(LiteLLMPydanticObjectBase):
|
||||
|
|
|
|||
|
|
@ -4748,7 +4748,7 @@ async def update_pass_through_endpoints(
|
|||
# Find the index for updating the list
|
||||
endpoint_index = None
|
||||
for idx, endpoint in enumerate(pass_through_endpoint_data):
|
||||
_endpoint = PassThroughGenericEndpoint(**endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
_endpoint = PassThroughGenericEndpoint.model_validate(endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
if _endpoint.id == endpoint_id:
|
||||
endpoint_index = idx
|
||||
break
|
||||
|
|
@ -4981,7 +4981,7 @@ async def delete_pass_through_endpoints(
|
|||
# Find the index for deleting from the list
|
||||
endpoint_index = None
|
||||
for idx, endpoint in enumerate(pass_through_endpoint_data):
|
||||
_endpoint = PassThroughGenericEndpoint(**endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
_endpoint = PassThroughGenericEndpoint.model_validate(endpoint) if isinstance(endpoint, dict) else endpoint
|
||||
if _endpoint.id == endpoint_id:
|
||||
endpoint_index = idx
|
||||
break
|
||||
|
|
@ -5029,7 +5029,7 @@ def _find_endpoint_by_id(
|
|||
for endpoint in endpoints_data:
|
||||
_endpoint: PassThroughGenericEndpoint | None = None
|
||||
if isinstance(endpoint, dict):
|
||||
_endpoint = PassThroughGenericEndpoint(**endpoint)
|
||||
_endpoint = PassThroughGenericEndpoint.model_validate(endpoint)
|
||||
elif isinstance(endpoint, PassThroughGenericEndpoint):
|
||||
_endpoint = endpoint
|
||||
|
||||
|
|
|
|||
|
|
@ -5858,6 +5858,16 @@ def _reload_settings_store(section: Section, store: SettingsStore, section_confi
|
|||
store["pass_through_endpoints"] = serving_pass_throughs
|
||||
|
||||
|
||||
def _clear_db_settings_beside_pass_through_endpoints(section: Section, store: SettingsStore) -> None:
|
||||
stored_pass_throughs: Final = store.db_value("pass_through_endpoints")
|
||||
store.apply_db_row(
|
||||
section,
|
||||
MappingProxyType({"pass_through_endpoints": stored_pass_throughs})
|
||||
if isinstance(stored_pass_throughs, list)
|
||||
else _EMPTY_SETTINGS_MAPPING,
|
||||
)
|
||||
|
||||
|
||||
def _general_settings_row_is_readable(row: _ConfigParamValueRow | None) -> bool:
|
||||
return row is None or row.param_value is None or isinstance(row.param_value, Mapping)
|
||||
|
||||
|
|
@ -6708,7 +6718,7 @@ class ProxyConfig:
|
|||
)
|
||||
else:
|
||||
for section, store in self._settings_stores.items():
|
||||
store.apply_db_row(section, _EMPTY_SETTINGS_MAPPING)
|
||||
_clear_db_settings_beside_pass_through_endpoints(section, store)
|
||||
|
||||
config = self._config_with_resolved_settings(config)
|
||||
|
||||
|
|
|
|||
86
litellm/proxy/public_endpoints/model_hub_rows.py
Normal file
86
litellm/proxy/public_endpoints/model_hub_rows.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
"""The rows the public Model Hub lists: published model groups and opted-in pass-through endpoints."""
|
||||
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
from typing import Final, TypeAlias
|
||||
|
||||
from pydantic import TypeAdapter, ValidationError
|
||||
|
||||
import litellm
|
||||
from litellm.proxy._types import PassThroughGenericEndpoint
|
||||
from litellm.types.proxy.management_endpoints.model_management_endpoints import ModelGroupInfoProxy
|
||||
|
||||
PASS_THROUGH_MODE: Final = "passthrough"
|
||||
_SERVING_ENTRY: Final = TypeAdapter(dict[str, object])
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ModelGroupRow:
|
||||
info: ModelGroupInfoProxy
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PassThroughRow:
|
||||
info: ModelGroupInfoProxy
|
||||
|
||||
|
||||
HubRow: TypeAlias = ModelGroupRow | PassThroughRow
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class NoModelsConfigured:
|
||||
"""Neither a router nor a published pass-through, so the hub has nothing to list."""
|
||||
|
||||
|
||||
def pass_through_row(endpoint: PassThroughGenericEndpoint) -> PassThroughRow:
|
||||
return PassThroughRow(
|
||||
info=ModelGroupInfoProxy(
|
||||
model_group=endpoint.display_name or endpoint.path,
|
||||
providers=[],
|
||||
mode=PASS_THROUGH_MODE,
|
||||
is_public_model_group=True,
|
||||
pass_through_path=endpoint.path,
|
||||
pass_through_methods=tuple(endpoint.methods) if endpoint.methods is not None else None,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _serving_pass_through_endpoints() -> Iterator[PassThroughGenericEndpoint]:
|
||||
"""The endpoints this worker serves, from the in-memory general settings the config sync keeps current."""
|
||||
from litellm.proxy.proxy_server import proxy_config
|
||||
|
||||
configured: Final = proxy_config.settings.get("pass_through_endpoints")
|
||||
for entry in configured if isinstance(configured, list) else ():
|
||||
try:
|
||||
yield PassThroughGenericEndpoint.model_validate(_SERVING_ENTRY.validate_python(entry))
|
||||
except ValidationError:
|
||||
continue
|
||||
|
||||
|
||||
def _published_model_group_rows() -> tuple[ModelGroupRow, ...] | None:
|
||||
"""None when there is no router, which differs from a router that publishes nothing."""
|
||||
from litellm.proxy.proxy_server import (
|
||||
get_model_group_info,
|
||||
llm_router,
|
||||
)
|
||||
|
||||
if llm_router is None:
|
||||
return None
|
||||
if litellm.public_model_groups is None:
|
||||
return ()
|
||||
return tuple(
|
||||
ModelGroupRow(info=info)
|
||||
for info in get_model_group_info(
|
||||
llm_router=llm_router, all_models_str=litellm.public_model_groups, model_group=None
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def published_hub_rows() -> tuple[HubRow, ...] | NoModelsConfigured:
|
||||
pass_through_rows: Final = tuple(
|
||||
pass_through_row(endpoint) for endpoint in _serving_pass_through_endpoints() if endpoint.show_in_model_hub
|
||||
)
|
||||
model_group_rows: Final = _published_model_group_rows()
|
||||
if model_group_rows is None and not pass_through_rows:
|
||||
return NoModelsConfigured()
|
||||
return (*(model_group_rows or ()), *pass_through_rows)
|
||||
|
|
@ -5,6 +5,7 @@ import re
|
|||
from collections.abc import Awaitable, Callable, Mapping, Sequence
|
||||
from functools import cache
|
||||
from importlib.resources import files
|
||||
from types import MappingProxyType
|
||||
from typing import TYPE_CHECKING, Final, Protocol
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
|
|
@ -25,6 +26,13 @@ from litellm.proxy._types import (
|
|||
)
|
||||
from litellm.proxy.db.health_check_latest import latest_by_model_name
|
||||
from litellm.proxy.deps.repositories import PluginUploadRepositoryDep
|
||||
from litellm.proxy.public_endpoints.model_hub_rows import (
|
||||
HubRow,
|
||||
ModelGroupRow,
|
||||
NoModelsConfigured,
|
||||
PassThroughRow,
|
||||
published_hub_rows,
|
||||
)
|
||||
from litellm.proxy.utils import get_custom_url
|
||||
from litellm.repositories.plugin_publication import read_plugin_publisher, read_plugin_review
|
||||
from litellm.repositories.table_repositories import ClaudeCodePluginRepository
|
||||
|
|
@ -276,45 +284,52 @@ def _agent_create_fields() -> tuple[AgentCreateInfo, ...]:
|
|||
response_model=list[ModelGroupInfoProxy],
|
||||
)
|
||||
async def public_model_hub():
|
||||
import litellm
|
||||
from litellm.proxy.health_endpoints._health_endpoints import (
|
||||
convert_health_check_to_dict,
|
||||
)
|
||||
from litellm.proxy.proxy_server import (
|
||||
get_model_group_info,
|
||||
llm_router,
|
||||
prisma_client,
|
||||
)
|
||||
|
||||
if llm_router is None:
|
||||
rows: Final = published_hub_rows()
|
||||
if isinstance(rows, NoModelsConfigured):
|
||||
raise HTTPException(status_code=400, detail=CommonProxyErrors.no_llm_router.value)
|
||||
|
||||
model_groups: list[ModelGroupInfoProxy] = []
|
||||
if litellm.public_model_groups is not None:
|
||||
model_groups = get_model_group_info( # rebind-ok: pre-existing rebinding on a rename-only line
|
||||
llm_router=llm_router,
|
||||
all_models_str=litellm.public_model_groups,
|
||||
model_group=None,
|
||||
health_checks_map: Final = await _legacy_health_checks()
|
||||
return [_with_legacy_health(row, health_checks_map) for row in rows]
|
||||
|
||||
|
||||
async def _legacy_health_checks() -> Mapping[str, Mapping[str, object]]:
|
||||
from litellm.proxy.proxy_server import prisma_client
|
||||
|
||||
if prisma_client is None:
|
||||
return MappingProxyType({})
|
||||
try:
|
||||
latest_checks: Final = await prisma_client.get_all_latest_health_checks()
|
||||
return MappingProxyType(
|
||||
{
|
||||
model_name: MappingProxyType(
|
||||
{
|
||||
"status": check.status,
|
||||
"response_time_ms": check.response_time_ms,
|
||||
"checked_at": check.checked_at.isoformat() if check.checked_at else None,
|
||||
}
|
||||
)
|
||||
for model_name, check in latest_by_model_name(latest_checks).items()
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
return MappingProxyType({})
|
||||
|
||||
# Fetch health check information if available
|
||||
health_checks_map: Final = {}
|
||||
if prisma_client is not None:
|
||||
try:
|
||||
latest_checks: Final = await prisma_client.get_all_latest_health_checks()
|
||||
for model_name, check in latest_by_model_name(latest_checks).items():
|
||||
health_checks_map[model_name] = convert_health_check_to_dict(check)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for model_group in model_groups:
|
||||
health_info = health_checks_map.get(model_group.model_group)
|
||||
if health_info:
|
||||
model_group.health_status = health_info.get("status")
|
||||
model_group.health_response_time = health_info.get("response_time_ms")
|
||||
model_group.health_checked_at = health_info.get("checked_at")
|
||||
|
||||
return model_groups
|
||||
def _with_legacy_health(row: HubRow, health_checks_map: Mapping[str, Mapping[str, object]]) -> ModelGroupInfoProxy:
|
||||
match row:
|
||||
case PassThroughRow():
|
||||
return row.info
|
||||
case ModelGroupRow():
|
||||
health_info: Final = health_checks_map.get(row.info.model_group)
|
||||
if not health_info:
|
||||
return row.info
|
||||
return row.info.model_copy(
|
||||
update={
|
||||
"health_status": health_info.get("status"),
|
||||
"health_response_time": health_info.get("response_time_ms"),
|
||||
"health_checked_at": health_info.get("checked_at"),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
|
|
|
|||
|
|
@ -8,7 +8,6 @@ from typing import Annotated, Final, Literal, Protocol
|
|||
from fastapi import APIRouter, Depends, Request
|
||||
from typing_extensions import ReadOnly, TypedDict
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_proxy_logger
|
||||
from litellm.proxy._types import CommonProxyErrors, UserAPIKeyAuth
|
||||
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
|
||||
|
|
@ -24,6 +23,13 @@ from litellm.proxy.list_api.list_framework import (
|
|||
handle_facet,
|
||||
handle_list,
|
||||
)
|
||||
from litellm.proxy.public_endpoints.model_hub_rows import (
|
||||
HubRow,
|
||||
ModelGroupRow,
|
||||
NoModelsConfigured,
|
||||
PassThroughRow,
|
||||
published_hub_rows,
|
||||
)
|
||||
from litellm.proxy.utils import PrismaClient
|
||||
from litellm.types.proxy.management_endpoints.management_v1 import (
|
||||
FacetListResponse,
|
||||
|
|
@ -93,9 +99,19 @@ class HealthEnricher:
|
|||
|
||||
lookup: HealthSnapshotLookup
|
||||
|
||||
async def __call__(self, rows: Sequence[ModelGroupInfoProxy]) -> Sequence[ModelGroupInfoProxy]:
|
||||
health: Final = await self.lookup.latest_for(tuple(row.model_group for row in rows))
|
||||
return tuple(_with_health(row, health.get(row.model_group)) for row in rows)
|
||||
async def __call__(self, rows: Sequence[HubRow]) -> Sequence[HubRow]:
|
||||
health: Final = await self.lookup.latest_for(
|
||||
tuple(row.info.model_group for row in rows if isinstance(row, ModelGroupRow))
|
||||
)
|
||||
return tuple(_row_with_health(row, health) for row in rows)
|
||||
|
||||
|
||||
def _row_with_health(row: HubRow, health: Mapping[str, HealthSnapshot]) -> HubRow:
|
||||
match row:
|
||||
case PassThroughRow():
|
||||
return row
|
||||
case ModelGroupRow():
|
||||
return ModelGroupRow(info=_with_health(row.info, health.get(row.info.model_group)))
|
||||
|
||||
|
||||
FEATURE_PREFIX: Final = "supports_"
|
||||
|
|
@ -120,26 +136,26 @@ def _enabled_features(fields: Mapping[str, object]) -> tuple[str, ...]:
|
|||
)
|
||||
|
||||
|
||||
def _cells(row: ModelGroupInfoProxy) -> Cells:
|
||||
def _cells(row: HubRow) -> Cells:
|
||||
info: Final = row.info
|
||||
return MappingProxyType(
|
||||
{
|
||||
"model_group": row.model_group,
|
||||
"mode": row.mode,
|
||||
"providers": tuple(row.providers),
|
||||
"features": _features(row),
|
||||
"max_input_tokens": row.max_input_tokens,
|
||||
"max_output_tokens": row.max_output_tokens,
|
||||
"input_cost_per_token": row.input_cost_per_token,
|
||||
"output_cost_per_token": row.output_cost_per_token,
|
||||
"rpm": row.rpm,
|
||||
"tpm": row.tpm,
|
||||
"model_group": info.model_group,
|
||||
"mode": info.mode,
|
||||
"providers": tuple(info.providers),
|
||||
"features": _features(info),
|
||||
"max_input_tokens": info.max_input_tokens,
|
||||
"max_output_tokens": info.max_output_tokens,
|
||||
"input_cost_per_token": info.input_cost_per_token,
|
||||
"output_cost_per_token": info.output_cost_per_token,
|
||||
"rpm": info.rpm,
|
||||
"tpm": info.tpm,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _serialize(row: ModelGroupInfoProxy) -> ModelGroupInfoProxy:
|
||||
"""The row shape is the wire shape: the rows served are the router's own model group records."""
|
||||
return row
|
||||
def _serialize(row: HubRow) -> ModelGroupInfoProxy:
|
||||
return row.info
|
||||
|
||||
|
||||
def _scope(_caller: UserAPIKeyAuth) -> Scope:
|
||||
|
|
@ -163,7 +179,7 @@ MODEL_HUB_FACETS: Final[Mapping[str, str]] = MappingProxyType(
|
|||
{"providers": "providers", "modes": "mode", "features": "features"}
|
||||
)
|
||||
|
||||
MODEL_HUB_LIST_SPEC: Final[ListSpec[ModelGroupInfoProxy, ModelGroupInfoProxy]] = ListSpec(
|
||||
MODEL_HUB_LIST_SPEC: Final[ListSpec[HubRow, ModelGroupInfoProxy]] = ListSpec(
|
||||
resource="model groups",
|
||||
sortable=frozenset(
|
||||
(
|
||||
|
|
@ -189,13 +205,9 @@ MODEL_HUB_LIST_SPEC: Final[ListSpec[ModelGroupInfoProxy, ModelGroupInfoProxy]] =
|
|||
)
|
||||
|
||||
|
||||
def _published_rows() -> Sequence[ModelGroupInfoProxy]:
|
||||
from litellm.proxy.proxy_server import (
|
||||
get_model_group_info, # pyright: ignore[reportPrivateUsage] # /public/model_hub imports it the same way
|
||||
llm_router,
|
||||
)
|
||||
|
||||
if llm_router is None:
|
||||
def _published_rows() -> Sequence[HubRow]:
|
||||
rows: Final = published_hub_rows()
|
||||
if isinstance(rows, NoModelsConfigured):
|
||||
raise ManagementProblem(
|
||||
ProblemDetail(
|
||||
type=f"{PROBLEM_TYPE_BASE}no-llm-router",
|
||||
|
|
@ -204,21 +216,13 @@ def _published_rows() -> Sequence[ModelGroupInfoProxy]:
|
|||
detail=CommonProxyErrors.no_llm_router.value,
|
||||
)
|
||||
)
|
||||
if litellm.public_model_groups is None:
|
||||
return ()
|
||||
return tuple(
|
||||
get_model_group_info(
|
||||
llm_router=llm_router,
|
||||
all_models_str=litellm.public_model_groups,
|
||||
model_group=None,
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def _executor(
|
||||
rows: Sequence[ModelGroupInfoProxy],
|
||||
rows: Sequence[HubRow],
|
||||
prisma_client: PrismaClient | None,
|
||||
) -> InMemoryListExecutor[ModelGroupInfoProxy]:
|
||||
) -> InMemoryListExecutor[HubRow]:
|
||||
if prisma_client is None:
|
||||
return InMemoryListExecutor(rows=rows, cells=_cells)
|
||||
return InMemoryListExecutor(
|
||||
|
|
|
|||
|
|
@ -42,6 +42,8 @@ class ModelGroupInfoProxy(ModelGroupInfo):
|
|||
health_status: str | None = Field(default=None)
|
||||
health_response_time: float | None = Field(default=None)
|
||||
health_checked_at: str | None = Field(default=None)
|
||||
pass_through_path: str | None = Field(default=None)
|
||||
pass_through_methods: tuple[str, ...] | None = Field(default=None)
|
||||
custom_tags: tuple[str, ...] | None = None
|
||||
custom_metadata: Mapping[str, str | int | float | bool] | None = None
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,8 @@ export interface ModelGroupInfo {
|
|||
health_status?: string;
|
||||
health_response_time?: number;
|
||||
health_checked_at?: string;
|
||||
pass_through_path?: string | null;
|
||||
pass_through_methods?: string[] | null;
|
||||
description?: string;
|
||||
custom_tags?: string[] | null;
|
||||
custom_metadata?: Record<string, ModelCustomMetadataValue> | null;
|
||||
|
|
@ -90,6 +92,11 @@ const formatCapabilityName = (key: string) =>
|
|||
|
||||
const formatCost = (cost: number) => `$${(cost * 1_000_000).toFixed(4)}`;
|
||||
|
||||
const tokenCostLabel = (model: ModelGroupInfo, cost: number | null | undefined): string => {
|
||||
if (model.pass_through_path) return "n/a";
|
||||
return cost ? formatCost(cost) : i18n.t("aiHub.publicModelHubColumns.free");
|
||||
};
|
||||
|
||||
const formatTokens = (tokens: number | undefined) => {
|
||||
if (!tokens) return i18n.t("aiHub.publicModelHubColumns.notAvailable");
|
||||
if (tokens >= 1000) return `${(tokens / 1000).toFixed(0)}K`;
|
||||
|
|
@ -112,6 +119,8 @@ const getModeIcon = (mode: string) => {
|
|||
return "🔄";
|
||||
case "embedding":
|
||||
return "📄";
|
||||
case "passthrough":
|
||||
return "🔀";
|
||||
default:
|
||||
return "🤖";
|
||||
}
|
||||
|
|
@ -304,25 +313,17 @@ export const getPublicModelHubColumns = ({
|
|||
<div className="flex flex-col gap-0.5 text-xs tabular-nums">
|
||||
<span>
|
||||
{i18n.t("aiHub.publicModelHubColumns.inputCost", {
|
||||
cost: row.original.input_cost_per_token
|
||||
? formatCost(row.original.input_cost_per_token)
|
||||
: i18n.t("aiHub.publicModelHubColumns.free"),
|
||||
cost: tokenCostLabel(row.original, row.original.input_cost_per_token),
|
||||
})}
|
||||
</span>
|
||||
<span className="text-muted-foreground">
|
||||
{i18n.t("aiHub.publicModelHubColumns.outputCost", {
|
||||
cost: row.original.output_cost_per_token
|
||||
? formatCost(row.original.output_cost_per_token)
|
||||
: i18n.t("aiHub.publicModelHubColumns.free"),
|
||||
cost: tokenCostLabel(row.original, row.original.output_cost_per_token),
|
||||
})}
|
||||
</span>
|
||||
</div>
|
||||
) : (
|
||||
<span className="text-sm">
|
||||
{row.original.input_cost_per_token
|
||||
? formatCost(row.original.input_cost_per_token)
|
||||
: i18n.t("aiHub.publicModelHubColumns.free")}
|
||||
</span>
|
||||
<span className="text-sm">{tokenCostLabel(row.original, row.original.input_cost_per_token)}</span>
|
||||
),
|
||||
},
|
||||
{
|
||||
|
|
@ -334,11 +335,7 @@ export const getPublicModelHubColumns = ({
|
|||
),
|
||||
size: 110,
|
||||
cell: ({ row }) => (
|
||||
<span className="text-sm">
|
||||
{row.original.output_cost_per_token
|
||||
? formatCost(row.original.output_cost_per_token)
|
||||
: i18n.t("aiHub.publicModelHubColumns.free")}
|
||||
</span>
|
||||
<span className="text-sm">{tokenCostLabel(row.original, row.original.output_cost_per_token)}</span>
|
||||
),
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -80,6 +80,8 @@ const reopenedPayload = {
|
|||
auth: true,
|
||||
timeout: undefined,
|
||||
cost_per_request: undefined,
|
||||
display_name: undefined,
|
||||
show_in_model_hub: false,
|
||||
};
|
||||
|
||||
describe("add_pass_through submit payload", () => {
|
||||
|
|
@ -112,9 +114,26 @@ describe("add_pass_through submit payload", () => {
|
|||
auth: true,
|
||||
timeout: "900",
|
||||
cost_per_request: "1.5",
|
||||
display_name: undefined,
|
||||
show_in_model_hub: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("sends the display name and the Model Hub opt-in when they are set", async () => {
|
||||
const user = setup();
|
||||
renderForm();
|
||||
await openModal(user);
|
||||
await fillRequiredFields(user);
|
||||
fireEvent.change(screen.getByPlaceholderText("Clinical NER"), { target: { value: "Clinical NER" } });
|
||||
await user.click(screen.getByRole("switch", { name: "Show on Model Hub" }));
|
||||
|
||||
await submit(user);
|
||||
|
||||
await waitFor(() => expect(createPassThroughEndpoint).toHaveBeenCalled());
|
||||
expect(lastPayload().display_name).toBe("Clinical NER");
|
||||
expect(lastPayload().show_in_model_hub).toBe(true);
|
||||
});
|
||||
|
||||
it("submits numeric fields as strings, not numbers", async () => {
|
||||
const user = setup();
|
||||
renderForm();
|
||||
|
|
|
|||
|
|
@ -52,6 +52,8 @@ const passThroughFormSchema = z.object({
|
|||
auth: z.boolean(),
|
||||
timeout: z.string().optional(),
|
||||
cost_per_request: z.string().optional(),
|
||||
display_name: z.string().optional(),
|
||||
show_in_model_hub: z.boolean().optional(),
|
||||
});
|
||||
|
||||
type PassThroughFormValues = z.output<typeof passThroughFormSchema>;
|
||||
|
|
@ -66,6 +68,8 @@ const emptyFormValues = {
|
|||
auth: true,
|
||||
timeout: undefined,
|
||||
cost_per_request: undefined,
|
||||
display_name: undefined,
|
||||
show_in_model_hub: false,
|
||||
} as unknown as PassThroughFormValues;
|
||||
|
||||
const labelWithHint = (label: React.ReactNode, hint: string): React.ReactNode => (
|
||||
|
|
@ -129,6 +133,8 @@ const AddPassThroughEndpoint: React.FC<AddFallbacksProps> = ({
|
|||
auth: values.auth,
|
||||
timeout: values.timeout,
|
||||
cost_per_request: values.cost_per_request,
|
||||
display_name: optionalText(values.display_name ?? ""),
|
||||
show_in_model_hub: values.show_in_model_hub ?? false,
|
||||
...(Object.keys(guardrails).length > 0 ? { guardrails } : {}),
|
||||
};
|
||||
|
||||
|
|
@ -336,6 +342,45 @@ const AddPassThroughEndpoint: React.FC<AddFallbacksProps> = ({
|
|||
</FormField>
|
||||
</Card>
|
||||
|
||||
<Card className="block p-6">
|
||||
<h3 className="mb-2 text-lg font-semibold text-foreground">{t("passThroughModelHub.title")}</h3>
|
||||
<p className="mb-6 text-sm text-muted-foreground">{t("passThroughModelHub.description")}</p>
|
||||
|
||||
<div className="space-y-5">
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="display_name"
|
||||
label={t("passThroughModelHub.displayName")}
|
||||
description={t("passThroughModelHub.displayNameCreateHint")}
|
||||
>
|
||||
{({ value, ...field }) => (
|
||||
<Input
|
||||
{...field}
|
||||
placeholder={t("passThroughModelHub.displayNamePlaceholder")}
|
||||
value={value ?? ""}
|
||||
/>
|
||||
)}
|
||||
</FormField>
|
||||
|
||||
<div className="flex items-center justify-between py-3">
|
||||
<div>
|
||||
<div className="text-sm font-medium text-foreground">{t("passThroughModelHub.show")}</div>
|
||||
<div className="mt-0.5 text-xs text-muted-foreground">{t("passThroughModelHub.showHint")}</div>
|
||||
</div>
|
||||
<FormField control={form.control} name="show_in_model_hub">
|
||||
{({ value, onChange, ref: _ref, ...field }) => (
|
||||
<Switch
|
||||
{...field}
|
||||
aria-label={t("passThroughModelHub.show")}
|
||||
checked={value ?? false}
|
||||
onCheckedChange={onChange}
|
||||
/>
|
||||
)}
|
||||
</FormField>
|
||||
</div>
|
||||
</div>
|
||||
</Card>
|
||||
|
||||
<Card className="block p-6">
|
||||
<h3 className="mb-2 text-lg font-semibold text-foreground">{t("addPassThrough.billing.title")}</h3>
|
||||
<p className="mb-6 text-sm text-muted-foreground">{t("addPassThrough.billing.description")}</p>
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import type { ComponentProps } from "react";
|
||||
import userEvent, { PointerEventsCheckLevel } from "@testing-library/user-event";
|
||||
import { renderWithProviders, screen, waitFor } from "../../tests/test-utils";
|
||||
import { fireEvent, renderWithProviders, screen, waitFor } from "../../tests/test-utils";
|
||||
import PassThroughInfoView from "./pass_through_info";
|
||||
|
||||
const updatePassThroughEndpoint = vi.fn();
|
||||
|
|
@ -31,7 +32,9 @@ const endpoint = {
|
|||
methods: ["GET"],
|
||||
};
|
||||
|
||||
const renderView = (data = endpoint) =>
|
||||
type EndpointData = ComponentProps<typeof PassThroughInfoView>["endpointData"];
|
||||
|
||||
const renderView = (data: EndpointData = endpoint) =>
|
||||
renderWithProviders(<PassThroughInfoView endpointData={data} onClose={vi.fn()} accessToken="test-token" isAdmin />);
|
||||
|
||||
const openEditForm = async (user: User) => {
|
||||
|
|
@ -72,9 +75,34 @@ describe("pass_through_info update payload", () => {
|
|||
auth: false,
|
||||
methods: ["GET"],
|
||||
guardrails: undefined,
|
||||
display_name: "",
|
||||
show_in_model_hub: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("sends the edited display name and Model Hub opt-in", async () => {
|
||||
const user = setup();
|
||||
renderWithProviders(
|
||||
<PassThroughInfoView
|
||||
endpointData={{ ...endpoint, display_name: "Clinical NER", show_in_model_hub: true }}
|
||||
onClose={vi.fn()}
|
||||
accessToken="test-token"
|
||||
isAdmin
|
||||
/>,
|
||||
);
|
||||
await openEditForm(user);
|
||||
expect(screen.getByLabelText("Display Name")).toHaveValue("Clinical NER");
|
||||
expect(screen.getByRole("switch", { name: "Show on Model Hub" })).toBeChecked();
|
||||
fireEvent.change(screen.getByLabelText("Display Name"), { target: { value: " Clinical NER v2 " } });
|
||||
await user.click(screen.getByRole("switch", { name: "Show on Model Hub" }));
|
||||
|
||||
await save(user);
|
||||
|
||||
await waitFor(() => expect(updatePassThroughEndpoint).toHaveBeenCalled());
|
||||
expect(lastPayload().display_name).toBe("Clinical NER v2");
|
||||
expect(lastPayload().show_in_model_hub).toBe(false);
|
||||
});
|
||||
|
||||
it("submits numeric fields as numbers, not strings", async () => {
|
||||
const user = setup();
|
||||
renderView();
|
||||
|
|
@ -87,6 +115,18 @@ describe("pass_through_info update payload", () => {
|
|||
expect(lastPayload().timeout).toBe(600);
|
||||
});
|
||||
|
||||
it("saves an endpoint whose stored timeout is null without a validation error", async () => {
|
||||
const user = setup();
|
||||
renderView({ ...endpoint, timeout: null });
|
||||
await openEditForm(user);
|
||||
|
||||
await save(user);
|
||||
|
||||
await waitFor(() => expect(updatePassThroughEndpoint).toHaveBeenCalled());
|
||||
expect(screen.queryByText("Invalid input")).not.toBeInTheDocument();
|
||||
expect(lastPayload().timeout).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps the path from the loaded endpoint rather than the form", async () => {
|
||||
const user = setup();
|
||||
renderView();
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ const endpointSettingsSchema = z.object({
|
|||
cost_per_request: z.number().optional(),
|
||||
timeout: z.number().optional(),
|
||||
auth: z.boolean(),
|
||||
display_name: z.string().optional(),
|
||||
show_in_model_hub: z.boolean(),
|
||||
});
|
||||
|
||||
type EndpointSettingsValues = z.output<typeof endpointSettingsSchema>;
|
||||
|
|
@ -104,10 +106,12 @@ interface PassThroughEndpoint {
|
|||
headers: Record<string, any>;
|
||||
include_subpath?: boolean;
|
||||
cost_per_request?: number;
|
||||
timeout?: number;
|
||||
timeout?: number | null;
|
||||
auth?: boolean;
|
||||
methods?: string[];
|
||||
guardrails?: Record<string, { request_fields?: string[]; response_fields?: string[] } | null>;
|
||||
display_name?: string | null;
|
||||
show_in_model_hub?: boolean;
|
||||
}
|
||||
|
||||
// Password field component for headers
|
||||
|
|
@ -159,8 +163,10 @@ const PassThroughInfoView: React.FC<PassThroughInfoProps> = ({
|
|||
methods: initialEndpointData.methods || [],
|
||||
include_subpath: initialEndpointData.include_subpath || false,
|
||||
cost_per_request: initialEndpointData.cost_per_request,
|
||||
timeout: initialEndpointData.timeout,
|
||||
timeout: initialEndpointData.timeout ?? undefined,
|
||||
auth: initialEndpointData.auth ?? true,
|
||||
display_name: initialEndpointData.display_name ?? undefined,
|
||||
show_in_model_hub: initialEndpointData.show_in_model_hub ?? false,
|
||||
},
|
||||
});
|
||||
|
||||
|
|
@ -195,6 +201,8 @@ const PassThroughInfoView: React.FC<PassThroughInfoProps> = ({
|
|||
auth: values.auth,
|
||||
methods: values.methods.length > 0 ? values.methods : undefined,
|
||||
guardrails: guardrails && Object.keys(guardrails).length > 0 ? guardrails : undefined,
|
||||
display_name: values.display_name?.trim() ?? "",
|
||||
show_in_model_hub: values.show_in_model_hub,
|
||||
};
|
||||
|
||||
await updatePassThroughEndpoint(accessToken, endpointData.id, updateData);
|
||||
|
|
@ -323,6 +331,11 @@ const PassThroughInfoView: React.FC<PassThroughInfoProps> = ({
|
|||
</p>
|
||||
</div>
|
||||
)}
|
||||
{endpointData.show_in_model_hub && (
|
||||
<div>
|
||||
<Badge variant="secondary">{t("passThroughModelHub.published")}</Badge>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</Card>
|
||||
</div>
|
||||
|
|
@ -422,6 +435,27 @@ const PassThroughInfoView: React.FC<PassThroughInfoProps> = ({
|
|||
)}
|
||||
</FormField>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="display_name"
|
||||
label={t("passThroughModelHub.displayName")}
|
||||
description={t("passThroughModelHub.displayNameHint")}
|
||||
>
|
||||
{({ value, ...field }) => (
|
||||
<Input
|
||||
{...field}
|
||||
placeholder={t("passThroughModelHub.displayNamePlaceholder")}
|
||||
value={value ?? ""}
|
||||
/>
|
||||
)}
|
||||
</FormField>
|
||||
|
||||
<FormField control={form.control} name="show_in_model_hub" label={t("passThroughModelHub.show")}>
|
||||
{({ value, onChange, ref: _ref, ...field }) => (
|
||||
<Switch {...field} checked={value} onCheckedChange={onChange} />
|
||||
)}
|
||||
</FormField>
|
||||
|
||||
<FormField control={form.control} name="headers" label={t("passThroughInfo.form.headersJson")}>
|
||||
{({ value, ...field }) => (
|
||||
<Textarea
|
||||
|
|
@ -544,6 +578,20 @@ const PassThroughInfoView: React.FC<PassThroughInfoProps> = ({
|
|||
<p className="text-sm font-medium">{t("passThroughInfo.settings.targetUrl")}</p>
|
||||
<div>{endpointData.target}</div>
|
||||
</div>
|
||||
{endpointData.display_name && (
|
||||
<div>
|
||||
<p className="text-sm font-medium">{t("passThroughModelHub.displayName")}</p>
|
||||
<div>{endpointData.display_name}</div>
|
||||
</div>
|
||||
)}
|
||||
<div>
|
||||
<p className="text-sm font-medium">{t("passThroughModelHub.show")}</p>
|
||||
<Badge variant={endpointData.show_in_model_hub ? "secondary" : "outline"}>
|
||||
{endpointData.show_in_model_hub
|
||||
? t("passThroughInfo.settings.yes")
|
||||
: t("passThroughInfo.settings.no")}
|
||||
</Badge>
|
||||
</div>
|
||||
<div>
|
||||
<p className="text-sm font-medium">{t("passThroughInfo.settings.includeSubpath")}</p>
|
||||
<Badge variant={endpointData.include_subpath ? "secondary" : "outline"}>
|
||||
|
|
|
|||
|
|
@ -88,7 +88,7 @@ const modelFacts = (model: ModelGroupInfo): Fact[][] => {
|
|||
function CallThisModel({ model }: { model: ModelGroupInfo }) {
|
||||
const { t } = useTranslation();
|
||||
const curl = curlSnippet(model);
|
||||
if (!curl) {
|
||||
if (!curl || model.mode === "passthrough") {
|
||||
return (
|
||||
<PaneSection title={t("publicModelHub.modelInspector.callThisModel.title")}>
|
||||
<CodeBlock code={modelSnippet(model)} />
|
||||
|
|
|
|||
|
|
@ -191,6 +191,36 @@ beforeEach(() => {
|
|||
});
|
||||
|
||||
describe("PublicAiHub", () => {
|
||||
it.each([
|
||||
{ name: "Clinical NER", path: "/clinical-ner", methods: null, method: "POST", body: true },
|
||||
{ name: "LPR Catalog", path: "/lpr-catalog", methods: ["GET"], method: "GET", body: false },
|
||||
])("shows a route-specific curl example without the chat SDK for $name", async (endpoint) => {
|
||||
failModelsWith(() =>
|
||||
Promise.resolve(
|
||||
page([
|
||||
{
|
||||
model_group: endpoint.name,
|
||||
mode: "passthrough",
|
||||
providers: [],
|
||||
pass_through_path: endpoint.path,
|
||||
pass_through_methods: endpoint.methods,
|
||||
},
|
||||
]),
|
||||
),
|
||||
);
|
||||
renderWithProviders(<PublicAiHub accessToken={null} />);
|
||||
|
||||
fireEvent.click(await screen.findByRole("link", { name: new RegExp(endpoint.name) }));
|
||||
|
||||
const dialog = await screen.findByRole("dialog", { name: "Details" });
|
||||
expect(dialog).toHaveTextContent(`curl -X "${endpoint.method}" "https://gateway.example.com${endpoint.path}"`);
|
||||
expect(dialog).toHaveTextContent("Passthrough");
|
||||
expect(dialog.textContent?.includes(`-d '{"input": "Hello, how are you?"}'`)).toBe(endpoint.body);
|
||||
expect(dialog).not.toHaveTextContent("import openai");
|
||||
expect(dialog).not.toHaveTextContent("chat.completions");
|
||||
expect(dialog).not.toHaveTextContent("/v1/chat/completions");
|
||||
});
|
||||
|
||||
it("shows only Models and Skills without selecting an item or reading retired hubs", async () => {
|
||||
const user = userEvent.setup();
|
||||
renderWithProviders(<PublicAiHub accessToken={null} />);
|
||||
|
|
|
|||
|
|
@ -33,11 +33,50 @@ describe("model snippets", () => {
|
|||
expect(snippet).not.toContain("ui.example.com");
|
||||
});
|
||||
|
||||
it("offers cURL only for chat models", () => {
|
||||
it("offers cURL for chat models but not embeddings", () => {
|
||||
expect(curlSnippet(model({ mode: "chat" }))).toContain("curl https://gateway.example.com/v1/chat/completions");
|
||||
expect(curlSnippet(model({ mode: undefined }))).not.toBeNull();
|
||||
expect(curlSnippet(model({ mode: "embedding" }))).toBeNull();
|
||||
});
|
||||
|
||||
it.each([null, undefined, [], ["POST"], ["GET", "POST"]])(
|
||||
"uses the pass-through route with a JSON body when its methods are %j",
|
||||
(methods) => {
|
||||
const endpoint = model({
|
||||
mode: "passthrough",
|
||||
pass_through_path: "/clinical-ner",
|
||||
pass_through_methods: methods,
|
||||
});
|
||||
const expected = [
|
||||
'curl -X "POST" "https://gateway.example.com/clinical-ner" \\',
|
||||
' -H "Authorization: Bearer your_api_key" \\',
|
||||
' -H "Content-Type: application/json" \\',
|
||||
` -d '{"input": "Hello, how are you?"}'`,
|
||||
].join("\n");
|
||||
expect(curlSnippet(endpoint)).toBe(expected);
|
||||
expect(modelSnippet(endpoint)).toBe(expected);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([["GET"], ["PUT", "GET"]])("uses the first non-POST method with no body for %j", (...methods) => {
|
||||
const endpoint = model({
|
||||
mode: "passthrough",
|
||||
pass_through_path: "/lpr-catalog",
|
||||
pass_through_methods: methods,
|
||||
});
|
||||
const expected = [
|
||||
`curl -X "${methods[0]}" "https://gateway.example.com/lpr-catalog" \\`,
|
||||
' -H "Authorization: Bearer your_api_key"',
|
||||
].join("\n");
|
||||
expect(curlSnippet(endpoint)).toBe(expected);
|
||||
expect(modelSnippet(endpoint)).toBe(expected);
|
||||
});
|
||||
|
||||
it("does not offer a chat SDK example for a pass-through row without its route", () => {
|
||||
const endpoint = model({ mode: "passthrough" });
|
||||
expect(curlSnippet(endpoint)).toBeNull();
|
||||
expect(modelSnippet(endpoint)).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("MCP and skill snippets", () => {
|
||||
|
|
|
|||
|
|
@ -12,7 +12,28 @@ import { gatewayUrl, wildcardExample } from "./hubFormat";
|
|||
|
||||
const GREETING = "Hello, how are you?";
|
||||
|
||||
const passThroughExampleMethod = (methods: string[] | null | undefined): string =>
|
||||
!methods?.length || methods.includes("POST") ? "POST" : methods[0];
|
||||
|
||||
const shellDoubleQuoted = (value: string): string => value.replace(/[\\"$`]/g, "\\$&");
|
||||
|
||||
export const passThroughSnippet = (model: ModelGroupInfo): string | null => {
|
||||
if (!model.pass_through_path) return null;
|
||||
const method = passThroughExampleMethod(model.pass_through_methods);
|
||||
const quotedMethod = shellDoubleQuoted(method);
|
||||
const url = shellDoubleQuoted(`${gatewayUrl()}${model.pass_through_path}`);
|
||||
return method === "POST"
|
||||
? [
|
||||
`curl -X "${quotedMethod}" "${url}" \\`,
|
||||
` -H "Authorization: Bearer your_api_key" \\`,
|
||||
` -H "Content-Type: application/json" \\`,
|
||||
` -d '{"input": "Hello, how are you?"}'`,
|
||||
].join("\n")
|
||||
: [`curl -X "${quotedMethod}" "${url}" \\`, ` -H "Authorization: Bearer your_api_key"`].join("\n");
|
||||
};
|
||||
|
||||
export const modelSnippet = (model: ModelGroupInfo): string => {
|
||||
if (model.mode === "passthrough" || model.pass_through_path) return passThroughSnippet(model) ?? "";
|
||||
const name = wildcardExample(model.model_group);
|
||||
if (isDecisionMode(model.mode)) {
|
||||
return modelUsageExample(model.mode, gatewayUrl(), name, getGlobalLitellmHeaderName());
|
||||
|
|
@ -37,6 +58,7 @@ export const modelSnippet = (model: ModelGroupInfo): string => {
|
|||
};
|
||||
|
||||
export const curlSnippet = (model: ModelGroupInfo): string | null => {
|
||||
if (model.pass_through_path) return passThroughSnippet(model);
|
||||
if (model.mode && model.mode !== "chat") return null;
|
||||
return `curl ${gatewayUrl()}/v1/chat/completions \\
|
||||
-H "Authorization: Bearer $LITELLM_API_KEY" \\
|
||||
|
|
|
|||
|
|
@ -62,6 +62,17 @@ const model = (overrides: Partial<ModelGroupInfo> & { model_group: string }): Mo
|
|||
...overrides,
|
||||
});
|
||||
|
||||
const passThrough = (name: string, path: string, methods?: string[]): ModelGroupInfo => {
|
||||
const spec: Partial<ModelGroupInfo> & { model_group: string } = {
|
||||
model_group: name,
|
||||
providers: [],
|
||||
mode: "passthrough",
|
||||
pass_through_path: path,
|
||||
pass_through_methods: methods,
|
||||
};
|
||||
return model(spec);
|
||||
};
|
||||
|
||||
const DEFAULT_MODELS = [model({ model_group: "gpt-4" }), model({ model_group: "claude-3", providers: ["anthropic"] })];
|
||||
|
||||
const respondWith = (
|
||||
|
|
@ -438,6 +449,60 @@ describe("PublicModelHub", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("lists a published pass-through endpoint under its display name and opens its route", async () => {
|
||||
respondWith([passThrough("Clinical NER", "/clinical-ner")]);
|
||||
renderHub();
|
||||
|
||||
const row = await screen.findByRole("row", { name: /Clinical NER/ });
|
||||
expect(row).toHaveTextContent("passthrough");
|
||||
expect(row).toHaveTextContent("n/a");
|
||||
expect(row).not.toHaveTextContent("Free");
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Clinical NER" }));
|
||||
|
||||
const dialog = await screen.findByRole("dialog");
|
||||
expect(within(dialog).getByText("/clinical-ner")).toBeInTheDocument();
|
||||
expect(
|
||||
within(dialog).getByText(/curl -X "POST" "https:\/\/backend.example.com\/clinical-ner"/),
|
||||
).toBeInTheDocument();
|
||||
expect(within(dialog).queryByText("Token & Cost Information")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("quotes a hostile configured method in the usage example", async () => {
|
||||
respondWith([passThrough("Odd", "/odd", ['GET" ; touch /tmp/pwned ; "'])]);
|
||||
renderHub();
|
||||
|
||||
fireEvent.click(await screen.findByRole("button", { name: "Odd" }));
|
||||
|
||||
const dialog = await screen.findByRole("dialog");
|
||||
expect(
|
||||
within(dialog).getByText(/curl -X "GET\\" ; touch \/tmp\/pwned ; \\"" "https:\/\/backend.example.com\/odd"/),
|
||||
).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("escapes shell characters in the pass-through route of the usage example", async () => {
|
||||
respondWith([passThrough("Tenant", "/tenant/$acct")]);
|
||||
renderHub();
|
||||
|
||||
fireEvent.click(await screen.findByRole("button", { name: "Tenant" }));
|
||||
|
||||
const dialog = await screen.findByRole("dialog");
|
||||
expect(
|
||||
within(dialog).getByText(/curl -X "POST" "https:\/\/backend.example.com\/tenant\/\\\$acct"/),
|
||||
).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("writes the usage example with a method the pass-through route accepts", async () => {
|
||||
respondWith([passThrough("Catalog", "/catalog", ["GET"])]);
|
||||
renderHub();
|
||||
|
||||
fireEvent.click(await screen.findByRole("button", { name: "Catalog" }));
|
||||
|
||||
const dialog = await screen.findByRole("dialog");
|
||||
expect(within(dialog).getByText(/curl -X "GET" "https:\/\/backend.example.com\/catalog"/)).toBeInTheDocument();
|
||||
expect(within(dialog).queryByText(/-d '\{"input"/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("shows no models when the search has no matches (LIT-5230 regression)", async () => {
|
||||
renderHub();
|
||||
expect(await screen.findByText("gpt-4")).toBeInTheDocument();
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ import { MultiSelect } from "./shared/MultiSelect";
|
|||
import { featureLabel } from "./publicModelHub/publicModelHubFilters";
|
||||
import { usePublicModelHubFacets } from "./publicModelHub/usePublicModelHubFacets";
|
||||
import { usePublicModelHubList } from "./publicModelHub/usePublicModelHubList";
|
||||
import { passThroughSnippet } from "./publicModelHub/hubSnippets";
|
||||
import { DataTable } from "./shared/DataTable";
|
||||
import { toast } from "@/lib/toast";
|
||||
import Navbar from "./navbar";
|
||||
|
|
@ -299,6 +300,8 @@ const PublicModelHub: React.FC<PublicModelHubProps> = ({ accessToken, isEmbedded
|
|||
return `$${(cost * 1_000_000).toFixed(4)}`;
|
||||
};
|
||||
|
||||
const usageSnippet = (model: ModelGroupInfo): string => passThroughSnippet(model) ?? usageExample(model);
|
||||
|
||||
const models = usePublicModelHubList(proxyConfigured);
|
||||
const modelFacets = usePublicModelHubFacets(proxyConfigured);
|
||||
const modeOptions = useMemo(
|
||||
|
|
@ -611,7 +614,11 @@ const PublicModelHub: React.FC<PublicModelHubProps> = ({ accessToken, isEmbedded
|
|||
<DataTable
|
||||
data={models.rows}
|
||||
columns={modelColumns}
|
||||
getRowId={(model, index) => model.model_group || String(index)}
|
||||
getRowId={(model, index) =>
|
||||
model.pass_through_path
|
||||
? `passthrough:${model.pass_through_path}:${index}`
|
||||
: model.model_group || String(index)
|
||||
}
|
||||
sortingMode="server"
|
||||
sorting={models.sorting}
|
||||
onSortingChange={models.onSortingChange}
|
||||
|
|
@ -845,6 +852,12 @@ const PublicModelHub: React.FC<PublicModelHubProps> = ({ accessToken, isEmbedded
|
|||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.mode")}</p>
|
||||
<p>{displayMode(selectedModel.mode) ?? t("aiHub.publicModelHub.modelDialog.notSpecified")}</p>
|
||||
</div>
|
||||
{selectedModel.pass_through_path && (
|
||||
<div>
|
||||
<p className="font-medium">{t("passThroughModelHub.endpoint")}</p>
|
||||
<p className="font-mono text-sm">{selectedModel.pass_through_path}</p>
|
||||
</div>
|
||||
)}
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.providers")}</p>
|
||||
<div className="flex flex-wrap gap-1 mt-1">
|
||||
|
|
@ -920,49 +933,50 @@ const PublicModelHub: React.FC<PublicModelHubProps> = ({ accessToken, isEmbedded
|
|||
)}
|
||||
</div>
|
||||
|
||||
{/* Token and Cost Information */}
|
||||
<div>
|
||||
<p className="text-lg font-semibold mb-4">{t("aiHub.publicModelHub.modelDialog.tokenCost")}</p>
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.maxInputTokens")}</p>
|
||||
<p>
|
||||
{selectedModel.max_input_tokens?.toLocaleString() ||
|
||||
t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.maxOutputTokens")}</p>
|
||||
<p>
|
||||
{selectedModel.max_output_tokens?.toLocaleString() ||
|
||||
t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.inputCost")}</p>
|
||||
<p>
|
||||
{selectedModel.input_cost_per_token
|
||||
? formatCost(selectedModel.input_cost_per_token)
|
||||
: t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.outputCost")}</p>
|
||||
<p>
|
||||
{selectedModel.output_cost_per_token
|
||||
? formatCost(selectedModel.output_cost_per_token)
|
||||
: t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
{!selectedModel.pass_through_path && (
|
||||
<div>
|
||||
<p className="text-lg font-semibold mb-4">{t("aiHub.publicModelHub.modelDialog.tokenCost")}</p>
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.maxInputTokens")}</p>
|
||||
<p>
|
||||
{selectedModel.max_input_tokens?.toLocaleString() ||
|
||||
t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.maxOutputTokens")}</p>
|
||||
<p>
|
||||
{selectedModel.max_output_tokens?.toLocaleString() ||
|
||||
t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.inputCost")}</p>
|
||||
<p>
|
||||
{selectedModel.input_cost_per_token
|
||||
? formatCost(selectedModel.input_cost_per_token)
|
||||
: t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
<div>
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.outputCost")}</p>
|
||||
<p>
|
||||
{selectedModel.output_cost_per_token
|
||||
? formatCost(selectedModel.output_cost_per_token)
|
||||
: t("aiHub.publicModelHub.modelDialog.notSpecified")}
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
<DeploymentPricingBreakdown
|
||||
className="mt-4"
|
||||
deployments={selectedModel.deployment_pricing}
|
||||
heading={
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.pricingByDeployment")}</p>
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
<DeploymentPricingBreakdown
|
||||
className="mt-4"
|
||||
deployments={selectedModel.deployment_pricing}
|
||||
heading={
|
||||
<p className="font-medium">{t("aiHub.publicModelHub.modelDialog.pricingByDeployment")}</p>
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Capabilities */}
|
||||
<div>
|
||||
|
|
@ -1027,11 +1041,11 @@ const PublicModelHub: React.FC<PublicModelHubProps> = ({ accessToken, isEmbedded
|
|||
<div>
|
||||
<p className="text-lg font-semibold mb-4">{t("aiHub.publicModelHub.modelDialog.usageExample")}</p>
|
||||
<div className="bg-gray-900 text-gray-100 p-4 rounded-lg overflow-x-auto">
|
||||
<pre className="text-sm">{usageExample(selectedModel)}</pre>
|
||||
<pre className="text-sm">{usageSnippet(selectedModel)}</pre>
|
||||
</div>
|
||||
<div className="mt-2 text-right">
|
||||
<button
|
||||
onClick={() => copyToClipboard(usageExample(selectedModel))}
|
||||
onClick={() => copyToClipboard(usageSnippet(selectedModel))}
|
||||
className="text-sm text-info hover:text-info/80 cursor-pointer"
|
||||
>
|
||||
{t("aiHub.publicModelHub.modelDialog.copyToClipboard")}
|
||||
|
|
|
|||
|
|
@ -19278,6 +19278,18 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"passThroughModelHub": {
|
||||
"title": "Model Hub",
|
||||
"description": "Optionally list this endpoint on the public Model Hub page",
|
||||
"displayName": "Display Name",
|
||||
"displayNameCreateHint": "Shown on the Model Hub instead of the path. Example: Clinical NER",
|
||||
"displayNameHint": "Shown on the Model Hub instead of the path",
|
||||
"displayNamePlaceholder": "Clinical NER",
|
||||
"show": "Show on Model Hub",
|
||||
"showHint": "Publish this endpoint on the public Model Hub page",
|
||||
"published": "On Model Hub",
|
||||
"endpoint": "Endpoint:"
|
||||
},
|
||||
"addPassThrough": {
|
||||
"validation": {
|
||||
"pathRequired": "Path is required",
|
||||
|
|
|
|||
15
litellm/ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
15
litellm/ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
|
|
@ -40285,6 +40285,10 @@ export interface components {
|
|||
otpm?: number | null;
|
||||
/** Output Cost Per Token */
|
||||
output_cost_per_token?: number | null;
|
||||
/** Pass Through Methods */
|
||||
pass_through_methods?: string[] | null;
|
||||
/** Pass Through Path */
|
||||
pass_through_path?: string | null;
|
||||
/** Providers */
|
||||
providers: string[];
|
||||
/** Rpm */
|
||||
|
|
@ -42278,6 +42282,11 @@ export interface components {
|
|||
default_query_params: {
|
||||
[key: string]: unknown;
|
||||
};
|
||||
/**
|
||||
* Display Name
|
||||
* @description Human-readable name for this endpoint. The Model Hub lists the endpoint under this name, falling back to the path when unset.
|
||||
*/
|
||||
display_name?: string | null;
|
||||
/**
|
||||
* Guardrails
|
||||
* @description Guardrails configuration for this passthrough endpoint. Dict keys are guardrail names, values are optional settings for field targeting. When set, all org/team/key level guardrails will also execute. Defaults to None (no guardrails execute).
|
||||
|
|
@ -42320,6 +42329,12 @@ export interface components {
|
|||
* @description The route to be added to the LiteLLM Proxy Server.
|
||||
*/
|
||||
path: string;
|
||||
/**
|
||||
* Show In Model Hub
|
||||
* @description If True, this endpoint is published on the public Model Hub as a row in mode 'passthrough'.
|
||||
* @default false
|
||||
*/
|
||||
show_in_model_hub: boolean;
|
||||
/**
|
||||
* Target
|
||||
* @description The URL to which requests for this path should be forwarded.
|
||||
|
|
|
|||
454
tests/integration/management/test_pass_through_model_hub.py
Normal file
454
tests/integration/management/test_pass_through_model_hub.py
Normal file
|
|
@ -0,0 +1,454 @@
|
|||
"""A pass-through endpoint opted into the Model Hub is listed there under its display name.
|
||||
|
||||
`POST /config/pass_through_endpoint` accepts `display_name` and `show_in_model_hub`. The public hub
|
||||
routes then carry one row per opted-in endpoint, named by its display name (or its path when no name
|
||||
was given), in mode `passthrough`, naming the route it describes. An endpoint that did not opt in
|
||||
stays off the hub. Hub rows come from the shared database, so a peer proxy lists them as well.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import uuid
|
||||
from collections.abc import Generator
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
import httpx
|
||||
import yaml
|
||||
from pydantic import JsonValue, TypeAdapter
|
||||
|
||||
from tests.integration._support.client import Gateway, eventually, object_value, string_value
|
||||
from tests.integration._support.database import write_rows
|
||||
from tests.integration._support.process import OwnedState, owned_proxy, owned_proxy_process
|
||||
|
||||
PASS_THROUGH_MODE: Final = "passthrough"
|
||||
STARTED_WORKER: Final = re.compile(r"Started server process \[(\d+)\]")
|
||||
HUB_PARAMS: Final = {"filter[mode]": PASS_THROUGH_MODE, "page_size": "100"}
|
||||
BURST: Final = 40
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PassThrough:
|
||||
endpoint_id: str
|
||||
path: str
|
||||
display_name: str | None
|
||||
|
||||
|
||||
@contextmanager
|
||||
def pass_through(
|
||||
gateway: Gateway,
|
||||
*,
|
||||
show_in_model_hub: bool,
|
||||
display_name: str | None,
|
||||
methods: tuple[str, ...] | None = None,
|
||||
path: str | None = None,
|
||||
target_path: str = "",
|
||||
) -> Generator[PassThrough]:
|
||||
optional: Final[dict[str, JsonValue]] = {
|
||||
"display_name": display_name,
|
||||
"methods": list(methods) if methods is not None else None,
|
||||
}
|
||||
body: Final[dict[str, JsonValue]] = {
|
||||
"path": path or f"/integration-nlp-{uuid.uuid4().hex[:8]}",
|
||||
"target": gateway.upstream_url + target_path,
|
||||
"show_in_model_hub": show_in_model_hub,
|
||||
**{key: value for key, value in optional.items() if value is not None},
|
||||
}
|
||||
created: Final = gateway.request("POST", "/config/pass_through_endpoint", body)
|
||||
assert created.status_code == 200, created.text
|
||||
endpoints: Final = object_value(created.json())["endpoints"]
|
||||
assert isinstance(endpoints, list) and len(endpoints) == 1, created.text
|
||||
endpoint_id: Final = string_value(object_value(endpoints[0])["id"])
|
||||
try:
|
||||
yield PassThrough(endpoint_id, string_value(body["path"]), display_name)
|
||||
finally:
|
||||
deleted = gateway.request("DELETE", "/config/pass_through_endpoint", params={"endpoint_id": endpoint_id})
|
||||
assert deleted.status_code == 200, deleted.text
|
||||
|
||||
|
||||
def configured_endpoint(gateway: Gateway, endpoint_id: str) -> dict[str, JsonValue]:
|
||||
listed: Final = gateway.request("GET", "/config/pass_through_endpoint")
|
||||
assert listed.status_code == 200, listed.text
|
||||
endpoints: Final = object_value(listed.json())["endpoints"]
|
||||
assert isinstance(endpoints, list), listed.text
|
||||
matches: Final = tuple(object_value(row) for row in endpoints if object_value(row).get("id") == endpoint_id)
|
||||
assert len(matches) == 1, f"{endpoint_id} appears {len(matches)} times in {listed.text}"
|
||||
return matches[0]
|
||||
|
||||
|
||||
def hub_rows(gateway: Gateway) -> tuple[dict[str, JsonValue], ...]:
|
||||
listed: Final = gateway.request("GET", "/public/v1/model_hub", params=HUB_PARAMS)
|
||||
assert listed.status_code == 200, listed.text
|
||||
return rows_of(listed)
|
||||
|
||||
|
||||
def rows_of(listed: httpx.Response) -> tuple[dict[str, JsonValue], ...]:
|
||||
rows: Final = object_value(listed.json())["data"]
|
||||
assert isinstance(rows, list), listed.text
|
||||
return tuple(object_value(row) for row in rows)
|
||||
|
||||
|
||||
def names_of(rows: tuple[dict[str, JsonValue], ...]) -> tuple[str, ...]:
|
||||
return tuple(string_value(row["model_group"]) for row in rows)
|
||||
|
||||
|
||||
def update_pass_through(gateway: Gateway, endpoint_id: str, body: dict[str, JsonValue]) -> None:
|
||||
updated: Final = gateway.request("POST", f"/config/pass_through_endpoint/{endpoint_id}", body)
|
||||
assert updated.status_code == 200, updated.text
|
||||
|
||||
|
||||
def relayed_contents(gateway: Gateway) -> tuple[str, ...]:
|
||||
observed: Final = httpx.get(f"{gateway.upstream_url}/__observations", timeout=30, trust_env=False)
|
||||
assert observed.status_code == 200, observed.text
|
||||
requests: Final = object_value(observed.json())["requests"]
|
||||
assert isinstance(requests, list), observed.text
|
||||
return tuple(_first_message_content(object_value(request)["body"]) for request in requests)
|
||||
|
||||
|
||||
def _first_message_content(body: JsonValue) -> str:
|
||||
messages: Final = object_value(body).get("messages")
|
||||
if not isinstance(messages, list) or not messages:
|
||||
return ""
|
||||
return string_value(object_value(messages[0]).get("content", ""))
|
||||
|
||||
|
||||
def legacy_hub_rows(gateway: Gateway) -> tuple[dict[str, JsonValue], ...]:
|
||||
listed: Final = gateway.request("GET", "/public/model_hub")
|
||||
assert listed.status_code == 200, listed.text
|
||||
rows: Final = TypeAdapter(list[JsonValue]).validate_python(listed.json())
|
||||
return tuple(object_value(row) for row in rows)
|
||||
|
||||
|
||||
def hub_modes(gateway: Gateway) -> tuple[str, ...]:
|
||||
listed: Final = gateway.request("GET", "/public/v1/model_hub/modes")
|
||||
assert listed.status_code == 200, listed.text
|
||||
modes: Final = object_value(listed.json())["data"]
|
||||
assert isinstance(modes, list), listed.text
|
||||
return tuple(string_value(mode) for mode in modes)
|
||||
|
||||
|
||||
def hub_row(rows: tuple[dict[str, JsonValue], ...], model_group: str) -> dict[str, JsonValue]:
|
||||
matches: Final = tuple(row for row in rows if row["model_group"] == model_group)
|
||||
assert len(matches) == 1, f"{model_group} appears {len(matches)} times in {rows}"
|
||||
return matches[0]
|
||||
|
||||
|
||||
def test_opted_in_pass_through_is_listed_under_its_display_name(gateway: Gateway, peer: Gateway) -> None:
|
||||
display_name: Final = f"Clinical NER {uuid.uuid4().hex[:6]}"
|
||||
with (
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown,
|
||||
pass_through(gateway, show_in_model_hub=False, display_name="Kept off the hub") as hidden,
|
||||
):
|
||||
configured: Final = configured_endpoint(gateway, shown.endpoint_id)
|
||||
assert configured["display_name"] == display_name, configured
|
||||
assert configured["show_in_model_hub"] is True, configured
|
||||
|
||||
rows: Final = hub_rows(gateway)
|
||||
row: Final = hub_row(rows, display_name)
|
||||
assert row["mode"] == PASS_THROUGH_MODE, row
|
||||
assert row["pass_through_path"] == shown.path, row
|
||||
assert row["providers"] == [], row
|
||||
assert row["is_public_model_group"] is True, row
|
||||
assert not any(r["model_group"] in ("Kept off the hub", hidden.path) for r in rows), rows
|
||||
|
||||
assert PASS_THROUGH_MODE in hub_modes(gateway)
|
||||
|
||||
legacy: Final = hub_row(legacy_hub_rows(gateway), display_name)
|
||||
assert legacy["mode"] == PASS_THROUGH_MODE, legacy
|
||||
assert legacy["pass_through_path"] == shown.path, legacy
|
||||
|
||||
peer_rows: Final = eventually(
|
||||
lambda: hub_rows(peer),
|
||||
lambda rows: any(r["model_group"] == display_name for r in rows),
|
||||
seconds=70,
|
||||
)
|
||||
assert hub_row(peer_rows, display_name)["pass_through_path"] == shown.path
|
||||
|
||||
assert not any(r["model_group"] == display_name for r in hub_rows(gateway)), "deleted endpoint still on the hub"
|
||||
|
||||
|
||||
def test_opted_in_pass_through_without_a_display_name_is_listed_by_path(gateway: Gateway) -> None:
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=None, methods=("GET",)) as shown:
|
||||
row: Final = hub_row(hub_rows(gateway), shown.path)
|
||||
assert row["mode"] == PASS_THROUGH_MODE, row
|
||||
assert row["pass_through_path"] == shown.path, row
|
||||
assert row["pass_through_methods"] == ["GET"], row
|
||||
|
||||
|
||||
def test_a_published_pass_through_still_forwards_to_its_target(gateway: Gateway) -> None:
|
||||
marker: Final = f"relay {uuid.uuid4().hex}"
|
||||
with pass_through(
|
||||
gateway, show_in_model_hub=True, display_name="Chat relay", target_path="/v1/chat/completions"
|
||||
) as shown:
|
||||
assert hub_row(hub_rows(gateway), "Chat relay")["pass_through_path"] == shown.path
|
||||
relayed: Final = gateway.request(
|
||||
"POST", shown.path, {"model": "relay", "messages": [{"role": "user", "content": marker}]}
|
||||
)
|
||||
assert relayed.status_code == 200, relayed.text
|
||||
assert object_value(relayed.json())["object"] == "chat.completion", relayed.text
|
||||
assert marker in relayed_contents(gateway), "the upstream never saw the relayed request"
|
||||
|
||||
|
||||
def test_an_async_client_reads_the_same_pass_through_rows(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Async NER {uuid.uuid4().hex[:6]}"
|
||||
|
||||
async def read() -> httpx.Response:
|
||||
async with httpx.AsyncClient(base_url=str(gateway.client.base_url), timeout=30, trust_env=False) as client:
|
||||
return await client.get("/public/v1/model_hub", params=HUB_PARAMS)
|
||||
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
listed: Final = asyncio.run(read())
|
||||
assert listed.status_code == 200, listed.text
|
||||
rows: Final = rows_of(listed)
|
||||
assert hub_row(rows, display_name)["pass_through_path"] == shown.path
|
||||
assert rows == hub_rows(gateway)
|
||||
|
||||
|
||||
def single_row_page_names(gateway: Gateway, page: httpx.Response) -> Generator[str, None, None]:
|
||||
page_rows: Final = rows_of(page)
|
||||
assert len(page_rows) == 1, page.text
|
||||
yield from names_of(page_rows)
|
||||
following: Final = object_value(object_value(page.json())["links"])["next"]
|
||||
if following is None:
|
||||
return
|
||||
following_page: Final = gateway.request("GET", string_value(following))
|
||||
assert following_page.status_code == 200, following_page.text
|
||||
yield from single_row_page_names(gateway, following_page)
|
||||
|
||||
|
||||
def test_pass_through_rows_paginate_one_per_page(gateway: Gateway) -> None:
|
||||
suffix: Final = uuid.uuid4().hex[:6]
|
||||
with (
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=f"Pager A {suffix}"),
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=f"Pager B {suffix}"),
|
||||
):
|
||||
first: Final = gateway.request(
|
||||
"GET", "/public/v1/model_hub", params={"filter[mode]": PASS_THROUGH_MODE, "page_size": "1"}
|
||||
)
|
||||
assert first.status_code == 200, first.text
|
||||
meta: Final = object_value(object_value(first.json())["meta"])
|
||||
assert meta["page_size"] == 1, meta
|
||||
assert meta["total_count"] == meta["total_pages"], meta
|
||||
seen: Final = tuple(single_row_page_names(gateway, first))
|
||||
assert len(seen) == meta["total_count"], seen
|
||||
assert {f"Pager A {suffix}", f"Pager B {suffix}"} <= set(seen), seen
|
||||
|
||||
|
||||
def test_hub_fields_of_the_wrong_type_are_rejected(gateway: Gateway) -> None:
|
||||
path: Final = f"/integration-nlp-{uuid.uuid4().hex[:8]}"
|
||||
wrong: Final[tuple[tuple[str, JsonValue], ...]] = (
|
||||
("display_name", 5),
|
||||
("display_name", ["Clinical NER"]),
|
||||
("show_in_model_hub", "maybe"),
|
||||
)
|
||||
for field, value in wrong:
|
||||
created = gateway.request(
|
||||
"POST", "/config/pass_through_endpoint", {"path": path, "target": gateway.upstream_url, field: value}
|
||||
)
|
||||
assert created.status_code == 422, f"{field}={value!r}: {created.status_code} {created.text}"
|
||||
assert field in created.text, created.text
|
||||
listed: Final = gateway.request("GET", "/config/pass_through_endpoint")
|
||||
assert listed.status_code == 200, listed.text
|
||||
assert path not in listed.text, listed.text
|
||||
assert path not in names_of(hub_rows(gateway))
|
||||
|
||||
|
||||
def test_a_five_kilobyte_display_name_is_listed_verbatim(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Long {uuid.uuid4().hex[:6]} " + "n" * 5000
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
assert configured_endpoint(gateway, shown.endpoint_id)["display_name"] == display_name
|
||||
assert hub_row(hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
assert hub_row(legacy_hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
|
||||
|
||||
def test_unauthenticated_callers_read_the_hub_but_cannot_change_publication(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Guarded {uuid.uuid4().hex[:6]}"
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
anonymous: Final = gateway.client
|
||||
body: Final[dict[str, JsonValue]] = {
|
||||
"path": shown.path,
|
||||
"target": gateway.upstream_url,
|
||||
"show_in_model_hub": False,
|
||||
}
|
||||
assert anonymous.post("/config/pass_through_endpoint", json={**body, "path": "/intruder"}).status_code == 401
|
||||
assert anonymous.post(f"/config/pass_through_endpoint/{shown.endpoint_id}", json=body).status_code == 401
|
||||
assert (
|
||||
anonymous.delete("/config/pass_through_endpoint", params={"endpoint_id": shown.endpoint_id}).status_code
|
||||
== 401
|
||||
)
|
||||
public: Final = anonymous.get("/public/v1/model_hub", params=HUB_PARAMS)
|
||||
assert public.status_code == 200, public.text
|
||||
assert hub_row(rows_of(public), display_name)["pass_through_path"] == shown.path
|
||||
legacy: Final = anonymous.get("/public/model_hub")
|
||||
assert legacy.status_code == 200, legacy.text
|
||||
assert display_name in legacy.text
|
||||
assert configured_endpoint(gateway, shown.endpoint_id)["show_in_model_hub"] is True
|
||||
|
||||
|
||||
def test_an_empty_display_name_falls_back_to_the_path(gateway: Gateway) -> None:
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name="") as shown:
|
||||
assert configured_endpoint(gateway, shown.endpoint_id)["display_name"] == ""
|
||||
assert hub_row(hub_rows(gateway), shown.path)["pass_through_path"] == shown.path
|
||||
assert "" not in names_of(hub_rows(gateway))
|
||||
|
||||
|
||||
def test_an_update_from_an_older_client_keeps_the_hub_fields(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Sticky {uuid.uuid4().hex[:6]}"
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
update_pass_through(gateway, shown.endpoint_id, {"path": shown.path, "target": gateway.upstream_url + "/v2"})
|
||||
configured: Final = configured_endpoint(gateway, shown.endpoint_id)
|
||||
assert configured["target"] == gateway.upstream_url + "/v2", configured
|
||||
assert configured["display_name"] == display_name, configured
|
||||
assert configured["show_in_model_hub"] is True, configured
|
||||
assert hub_row(hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
|
||||
|
||||
def test_updates_clear_the_name_and_toggle_hub_visibility(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Toggled {uuid.uuid4().hex[:6]}"
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
base: Final[dict[str, JsonValue]] = {"path": shown.path, "target": gateway.upstream_url}
|
||||
update_pass_through(gateway, shown.endpoint_id, {**base, "display_name": ""})
|
||||
assert hub_row(hub_rows(gateway), shown.path)["pass_through_path"] == shown.path
|
||||
assert display_name not in names_of(hub_rows(gateway))
|
||||
|
||||
update_pass_through(gateway, shown.endpoint_id, {**base, "show_in_model_hub": False})
|
||||
assert shown.path not in names_of(hub_rows(gateway))
|
||||
assert shown.path not in names_of(legacy_hub_rows(gateway))
|
||||
|
||||
update_pass_through(
|
||||
gateway, shown.endpoint_id, {**base, "show_in_model_hub": True, "display_name": display_name}
|
||||
)
|
||||
assert hub_row(hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
assert hub_row(legacy_hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
|
||||
|
||||
def test_two_pass_throughs_sharing_a_display_name_are_two_rows(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Twin {uuid.uuid4().hex[:6]}"
|
||||
with (
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=display_name) as first,
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=display_name) as second,
|
||||
):
|
||||
twins: Final = tuple(row for row in hub_rows(gateway) if row["model_group"] == display_name)
|
||||
assert sorted(string_value(row["pass_through_path"]) for row in twins) == sorted((first.path, second.path))
|
||||
|
||||
|
||||
def test_a_config_file_pass_through_is_listed_once_beside_database_rows(gateway: Gateway, tmp_path: Path) -> None:
|
||||
display_name: Final = f"Config NLP {uuid.uuid4().hex[:6]}"
|
||||
path: Final = f"/integration-config-nlp-{uuid.uuid4().hex[:8]}"
|
||||
base_config: Final = object_value(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text()))
|
||||
config: Final = {
|
||||
**base_config,
|
||||
"general_settings": {
|
||||
**object_value(base_config["general_settings"]),
|
||||
"pass_through_endpoints": [
|
||||
{"path": path, "target": gateway.upstream_url, "display_name": display_name, "show_in_model_hub": True},
|
||||
{"path": f"{path}-hidden", "target": gateway.upstream_url, "display_name": "Hidden config NLP"},
|
||||
],
|
||||
},
|
||||
}
|
||||
config_path: Final = tmp_path / "pass_through_hub.yaml"
|
||||
config_path.write_text(yaml.safe_dump(config))
|
||||
with owned_proxy(gateway, tmp_path, {}, config=config_path) as owned:
|
||||
rows: Final = hub_rows(owned)
|
||||
assert names_of(rows).count(display_name) == 1, rows
|
||||
assert hub_row(rows, display_name)["pass_through_path"] == path
|
||||
assert "Hidden config NLP" not in names_of(rows), rows
|
||||
assert hub_row(legacy_hub_rows(owned), display_name)["pass_through_path"] == path
|
||||
with pass_through(owned, show_in_model_hub=True, display_name=f"{display_name} DB", path=f"{path}-db"):
|
||||
beside: Final = eventually(lambda: hub_rows(owned), lambda listed: f"{display_name} DB" in names_of(listed))
|
||||
assert names_of(beside).count(display_name) == 1, beside
|
||||
assert hub_row(beside, f"{display_name} DB")["pass_through_path"] == f"{path}-db"
|
||||
with pass_through(owned, show_in_model_hub=True, display_name=f"{display_name} DB", path=path):
|
||||
replaced: Final = eventually(lambda: hub_rows(owned), lambda listed: display_name not in names_of(listed))
|
||||
assert [string_value(row["pass_through_path"]) for row in replaced if row["pass_through_path"] == path] == [
|
||||
path
|
||||
], replaced
|
||||
assert hub_row(replaced, f"{display_name} DB")["pass_through_path"] == path
|
||||
restored: Final = eventually(lambda: hub_rows(owned), lambda listed: display_name in names_of(listed))
|
||||
assert hub_row(restored, display_name)["pass_through_path"] == path
|
||||
assert display_name not in names_of(hub_rows(gateway)), "the config row leaked into the shared proxy"
|
||||
|
||||
|
||||
def test_a_health_row_under_the_pass_through_name_is_not_borrowed(gateway: Gateway) -> None:
|
||||
display_name: Final = f"gpt-shadow-{uuid.uuid4().hex[:6]}"
|
||||
health_check_id: Final = str(uuid.uuid4())
|
||||
write_rows(
|
||||
'INSERT INTO "LiteLLM_HealthCheckTable" (health_check_id, model_name, status, updated_at) '
|
||||
"VALUES (%s, %s, %s, NOW())",
|
||||
(health_check_id, display_name, "unhealthy"),
|
||||
)
|
||||
try:
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown:
|
||||
row: Final = hub_row(hub_rows(gateway), display_name)
|
||||
assert (row["pass_through_path"], row["health_status"]) == (shown.path, None), row
|
||||
legacy: Final = hub_row(legacy_hub_rows(gateway), display_name)
|
||||
assert (legacy["pass_through_path"], legacy["health_status"]) == (shown.path, None), legacy
|
||||
finally:
|
||||
write_rows('DELETE FROM "LiteLLM_HealthCheckTable" WHERE health_check_id = %s', (health_check_id,))
|
||||
|
||||
|
||||
def test_repeated_hub_reads_return_the_same_rows(gateway: Gateway) -> None:
|
||||
display_name: Final = f"Stable {uuid.uuid4().hex[:6]}"
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name):
|
||||
reads: Final = tuple(hub_rows(gateway) for _ in range(3))
|
||||
assert reads[0] == reads[1] == reads[2], reads
|
||||
assert names_of(reads[0]).count(display_name) == 1, reads[0]
|
||||
assert len(legacy_hub_rows(gateway)) == len(reads[0]), "legacy and v1 disagree on the row count"
|
||||
|
||||
|
||||
def test_hub_reads_stay_200_while_an_endpoint_is_created_and_deleted_under_load(
|
||||
gateway: Gateway, peer: Gateway
|
||||
) -> None:
|
||||
display_name: Final = f"Churn {uuid.uuid4().hex[:6]}"
|
||||
|
||||
def read(index: int) -> tuple[int, int]:
|
||||
listed = (gateway if index % 2 else peer).request("GET", "/public/v1/model_hub", params=HUB_PARAMS)
|
||||
return listed.status_code, names_of(rows_of(listed)).count(display_name) if listed.status_code == 200 else -1
|
||||
|
||||
with ThreadPoolExecutor(8) as pool:
|
||||
before: Final = [pool.submit(read, index) for index in range(BURST // 2)]
|
||||
with pass_through(gateway, show_in_model_hub=True, display_name=display_name):
|
||||
during: Final = [pool.submit(read, index) for index in range(BURST // 2, BURST)]
|
||||
results: Final = [future.result() for future in (*before, *during)]
|
||||
assert all(status == 200 for status, _count in results), results
|
||||
assert all(count in (0, 1) for _status, count in results), results
|
||||
gateway_rows: Final = eventually(lambda: hub_rows(gateway), lambda rows: display_name not in names_of(rows))
|
||||
peer_rows: Final = eventually(lambda: hub_rows(peer), lambda rows: display_name not in names_of(rows))
|
||||
assert display_name not in names_of(gateway_rows), gateway_rows
|
||||
assert display_name not in names_of(peer_rows), peer_rows
|
||||
|
||||
|
||||
def test_hub_reads_survive_a_killed_proxy_worker(gateway: Gateway, tmp_path: Path) -> None:
|
||||
display_name: Final = f"Survivor {uuid.uuid4().hex[:6]}"
|
||||
|
||||
def read(side: Gateway) -> object:
|
||||
try:
|
||||
return side.request("GET", "/public/v1/model_hub", params=HUB_PARAMS).status_code
|
||||
except httpx.HTTPError as error:
|
||||
return error
|
||||
|
||||
with (
|
||||
pass_through(gateway, show_in_model_hub=True, display_name=display_name) as shown,
|
||||
owned_proxy_process(gateway, tmp_path, {}, workers=2, state=OwnedState.SHARED_WITH_WORKER) as owned,
|
||||
):
|
||||
workers: Final = eventually(
|
||||
lambda: tuple(int(pid) for pid in STARTED_WORKER.findall(owned.log.read_text())),
|
||||
lambda pids: len(pids) == 2,
|
||||
seconds=30,
|
||||
)
|
||||
assert hub_row(hub_rows(owned.gateway), display_name)["pass_through_path"] == shown.path
|
||||
with ThreadPoolExecutor(8) as pool:
|
||||
futures: Final = [pool.submit(read, owned.gateway if index % 2 else gateway) for index in range(BURST)]
|
||||
owned.kill_worker(workers[0])
|
||||
results: Final = [future.result() for future in futures]
|
||||
assert all(status == 200 for status in results[0::2]), f"shared proxy answers: {results[0::2]!r}"
|
||||
answered: Final = tuple(status for status in results[1::2] if isinstance(status, int))
|
||||
assert answered and all(status == 200 for status in answered), f"owned proxy answers: {results[1::2]!r}"
|
||||
eventually(lambda: read(owned.gateway), lambda status: status == 200, seconds=60)
|
||||
assert hub_row(hub_rows(owned.gateway), display_name)["pass_through_path"] == shown.path
|
||||
assert hub_row(hub_rows(gateway), display_name)["pass_through_path"] == shown.path
|
||||
|
|
@ -5248,6 +5248,30 @@ async def test_ProxyConfig__update_config_from_db_resolves_through_settings_stor
|
|||
assert pc.settings.source("max_parallel_requests") == "config"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ProxyConfig_get_config_reload_keeps_db_pass_through_endpoints_without_store_model_in_db(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
from litellm.proxy import proxy_server
|
||||
|
||||
config_file: Final = tmp_path / "config.yaml"
|
||||
config_file.write_text(
|
||||
"general_settings:\n pass_through_endpoints:\n - path: /config\n target: http://config.test\n"
|
||||
)
|
||||
monkeypatch.setattr(proxy_server, "prisma_client", None)
|
||||
monkeypatch.setattr(proxy_server, "store_model_in_db", False)
|
||||
monkeypatch.setattr(proxy_server, "general_settings", {})
|
||||
monkeypatch.setattr(proxy_server, "initialize_pass_through_endpoints", AsyncMock())
|
||||
monkeypatch.delenv("LITELLM_CONFIG_BUCKET_NAME", raising=False)
|
||||
pc: Final = ProxyConfig()
|
||||
await pc.get_config(config_file_path=str(config_file))
|
||||
await pc._update_general_settings({"pass_through_endpoints": [{"path": "/db", "target": "http://db.test"}]})
|
||||
|
||||
await pc.get_config(config_file_path=str(config_file))
|
||||
|
||||
assert [endpoint["path"] for endpoint in pc.settings["pass_through_endpoints"]] == ["/db", "/config"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ProxyConfig__update_config_from_db_keeps_keys_the_config_file_omits(monkeypatch):
|
||||
pc = ProxyConfig()
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from fastapi.testclient import TestClient
|
|||
|
||||
import litellm
|
||||
from litellm.proxy._types import LiteLLMRoutes
|
||||
from litellm.proxy.proxy_server import app
|
||||
from litellm.proxy.proxy_server import app, proxy_config
|
||||
from litellm.types.router import DeploymentTypedDict, ModelGroupInfo
|
||||
from tests._master_key import MASTER_KEY
|
||||
from tests.unit.proxy.router_entry_point_stubs import route_entry_points_to_stubs
|
||||
|
|
@ -428,7 +428,9 @@ def test_features_filter_matches_a_model_with_any_of_the_named_features(monkeypa
|
|||
|
||||
|
||||
def test_a_single_feature_filter_selects_only_models_with_it(monkeypatch):
|
||||
_publish(monkeypatch, (_info("sees", supports_vision=True), _info("plain"), _info("reasons", supports_reasoning=True)))
|
||||
_publish(
|
||||
monkeypatch, (_info("sees", supports_vision=True), _info("plain"), _info("reasons", supports_reasoning=True))
|
||||
)
|
||||
|
||||
assert _groups(_get("filter[features][in]=vision")) == ["sees"]
|
||||
assert _groups(_get("filter[features][in]=reasoning")) == ["reasons"]
|
||||
|
|
@ -463,11 +465,14 @@ def test_a_facet_serves_the_distinct_values_of_its_column(monkeypatch, facet):
|
|||
response = _facet(facet)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["data"] == {
|
||||
"providers": ["anthropic", "mistral", "openai"],
|
||||
"modes": ["chat", "embedding"],
|
||||
"features": ["vision"],
|
||||
}[facet]
|
||||
assert (
|
||||
response.json()["data"]
|
||||
== {
|
||||
"providers": ["anthropic", "mistral", "openai"],
|
||||
"modes": ["chat", "embedding"],
|
||||
"features": ["vision"],
|
||||
}[facet]
|
||||
)
|
||||
|
||||
|
||||
def test_a_facet_offers_only_values_the_table_can_show(monkeypatch):
|
||||
|
|
@ -509,3 +514,157 @@ def test_a_facet_rejects_a_sort_it_does_not_offer(monkeypatch):
|
|||
@pytest.mark.parametrize("facet", FACET_PATHS)
|
||||
def test_a_facet_is_reachable_without_a_key(facet):
|
||||
assert f"{MODEL_HUB_PATH}/{facet}" in LiteLLMRoutes.public_routes.value
|
||||
|
||||
|
||||
PASS_THROUGH_TARGET = "https://nlp.internal.example/v1"
|
||||
|
||||
|
||||
def _configure_pass_throughs(monkeypatch, *endpoints: Mapping[str, object]) -> None:
|
||||
"""Pass-through endpoints as the config sync publishes them into the in-memory settings store."""
|
||||
monkeypatch.setitem(
|
||||
proxy_config.settings,
|
||||
"pass_through_endpoints",
|
||||
[{"target": PASS_THROUGH_TARGET, **endpoint} for endpoint in endpoints],
|
||||
)
|
||||
|
||||
|
||||
def _pass_through_row(model_group: str, path: str, methods: list[str] | None = None) -> dict[str, object]:
|
||||
return {
|
||||
**ModelGroupInfo(model_group=model_group, providers=[], mode="passthrough").model_dump(mode="json"),
|
||||
"is_public_model_group": True,
|
||||
"custom_tags": None,
|
||||
"custom_metadata": None,
|
||||
"supports_background_chat": False,
|
||||
"health_status": None,
|
||||
"health_response_time": None,
|
||||
"health_checked_at": None,
|
||||
"pass_through_path": path,
|
||||
"pass_through_methods": methods,
|
||||
}
|
||||
|
||||
|
||||
def test_a_pass_through_endpoint_opted_into_the_hub_is_listed_under_its_display_name(monkeypatch: pytest.MonkeyPatch):
|
||||
_publish(monkeypatch, (_info("gpt-4o"),))
|
||||
_configure_pass_throughs(
|
||||
monkeypatch,
|
||||
{"path": "/clinical-ner", "display_name": "Clinical NER", "show_in_model_hub": True},
|
||||
{"path": "/kept-private", "display_name": "Kept private", "show_in_model_hub": False},
|
||||
{"path": "/unnamed-tagger", "show_in_model_hub": True},
|
||||
{"path": "/default-hidden"},
|
||||
)
|
||||
|
||||
response = _get()
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert _groups(response) == ["/unnamed-tagger", "Clinical NER", "gpt-4o"]
|
||||
assert response.json()["data"][:2] == [
|
||||
_pass_through_row("/unnamed-tagger", "/unnamed-tagger"),
|
||||
_pass_through_row("Clinical NER", "/clinical-ner"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("route", [MODEL_HUB_PATH, LEGACY_MODEL_HUB_PATH])
|
||||
@pytest.mark.parametrize("invalid", [None, "invalid", 17, [], {}, {"path": "/missing-target"}])
|
||||
def test_invalid_pass_through_entries_do_not_hide_valid_hub_rows(monkeypatch: pytest.MonkeyPatch, route, invalid):
|
||||
_publish(monkeypatch, ())
|
||||
monkeypatch.setitem(
|
||||
proxy_config.settings,
|
||||
"pass_through_endpoints",
|
||||
[
|
||||
{"path": "/first", "target": PASS_THROUGH_TARGET, "show_in_model_hub": True},
|
||||
invalid,
|
||||
{"path": "/last", "target": PASS_THROUGH_TARGET, "show_in_model_hub": True},
|
||||
],
|
||||
)
|
||||
|
||||
response = client.get(route)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
rows = response.json()["data"] if route == MODEL_HUB_PATH else response.json()
|
||||
assert rows == [_pass_through_row("/first", "/first"), _pass_through_row("/last", "/last")]
|
||||
|
||||
|
||||
def test_a_pass_through_row_carries_the_methods_its_route_accepts(monkeypatch: pytest.MonkeyPatch):
|
||||
_publish(monkeypatch, (_info("gpt-4o"),))
|
||||
_configure_pass_throughs(
|
||||
monkeypatch, {"path": "/catalog", "display_name": "Catalog", "show_in_model_hub": True, "methods": ["GET"]}
|
||||
)
|
||||
|
||||
response = _get()
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["data"][0] == _pass_through_row("Catalog", "/catalog", methods=["GET"])
|
||||
|
||||
|
||||
def test_a_pass_through_row_is_filterable_by_its_mode_and_offered_by_the_modes_facet(monkeypatch: pytest.MonkeyPatch):
|
||||
_publish(monkeypatch, (_info("gpt-4o"), _info("embedder", mode="embedding")))
|
||||
_configure_pass_throughs(
|
||||
monkeypatch, {"path": "/clinical-ner", "display_name": "Clinical NER", "show_in_model_hub": True}
|
||||
)
|
||||
|
||||
assert _groups(_get("filter[mode]=passthrough")) == ["Clinical NER"]
|
||||
assert _groups(_get("filter[mode]=chat")) == ["gpt-4o"]
|
||||
assert _facet("modes").json()["data"] == ["chat", "embedding", "passthrough"]
|
||||
|
||||
|
||||
def test_a_pass_through_endpoint_is_listed_even_when_no_model_group_is_published(monkeypatch: pytest.MonkeyPatch):
|
||||
_publish(monkeypatch, ())
|
||||
monkeypatch.setattr(litellm, "public_model_groups", None)
|
||||
_configure_pass_throughs(
|
||||
monkeypatch, {"path": "/clinical-ner", "display_name": "Clinical NER", "show_in_model_hub": True}
|
||||
)
|
||||
|
||||
response = _get()
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["data"] == [_pass_through_row("Clinical NER", "/clinical-ner")]
|
||||
|
||||
|
||||
def test_a_pass_through_named_like_a_model_does_not_borrow_its_health(monkeypatch: pytest.MonkeyPatch):
|
||||
prisma_client, read = _recording_prisma([_health_check("gpt-4o", status="unhealthy")])
|
||||
prisma_client.get_all_latest_health_checks = AsyncMock(return_value=[_health_check("gpt-4o", status="unhealthy")])
|
||||
_publish(monkeypatch, (_info("gpt-4o"),), prisma_client=prisma_client)
|
||||
_configure_pass_throughs(monkeypatch, {"path": "/gpt-4o", "display_name": "gpt-4o", "show_in_model_hub": True})
|
||||
|
||||
rows = _get().json()["data"]
|
||||
legacy_rows = client.get(LEGACY_MODEL_HUB_PATH).json()
|
||||
|
||||
assert [(row["pass_through_path"], row["health_status"]) for row in rows] == [
|
||||
(None, "unhealthy"),
|
||||
("/gpt-4o", None),
|
||||
]
|
||||
assert _asked_about(read) == ["gpt-4o"]
|
||||
assert [(row["pass_through_path"], row["health_status"]) for row in legacy_rows] == [
|
||||
(None, "unhealthy"),
|
||||
("/gpt-4o", None),
|
||||
]
|
||||
|
||||
|
||||
def test_a_pass_through_only_proxy_without_a_router_still_publishes_its_endpoints(monkeypatch: pytest.MonkeyPatch):
|
||||
monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", None)
|
||||
monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", None)
|
||||
_configure_pass_throughs(
|
||||
monkeypatch, {"path": "/clinical-ner", "display_name": "Clinical NER", "show_in_model_hub": True}
|
||||
)
|
||||
|
||||
response = _get()
|
||||
legacy = client.get(LEGACY_MODEL_HUB_PATH)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["data"] == [_pass_through_row("Clinical NER", "/clinical-ner")]
|
||||
assert legacy.status_code == 200, legacy.text
|
||||
assert legacy.json() == [_pass_through_row("Clinical NER", "/clinical-ner")]
|
||||
|
||||
|
||||
def test_the_endpoint_it_supersedes_lists_pass_through_rows_after_the_model_groups(monkeypatch: pytest.MonkeyPatch):
|
||||
_publish(monkeypatch, _named(2))
|
||||
_configure_pass_throughs(
|
||||
monkeypatch, {"path": "/clinical-ner", "display_name": "Clinical NER", "show_in_model_hub": True}
|
||||
)
|
||||
|
||||
response = client.get(LEGACY_MODEL_HUB_PATH)
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
body = response.json()
|
||||
assert [row["model_group"] for row in body] == ["model-000", "model-001", "Clinical NER"], body
|
||||
assert body[2] == _pass_through_row("Clinical NER", "/clinical-ner")
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue