scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236)

Browse source 
* feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): register typesafe as a provider so Jev deployments load

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): move provider endpoints under llms and validate proxy bodies

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(decisions): add Cloudflare Clef and Strands Decider backends

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): register decisions routes for managed agents and gateway

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(decisions): use raw regex for cloudflare missing account match

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): avoid cast in Cloudflare response unwrapping

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): default model, evaluation health probe, short Cloudflare names

The proxy validates only state and questions, so a request without a
model falls through to the configured default model like every other
route. Health checks probe evaluation-mode deployments through the
Decisions API instead of failing with an unsupported mode, and
cloudflare/clef and cloudflare/clef-flash get cost-map rows so the short
names resolve a mode and a price. The registry no longer claims typed
decisions for a provider with no backend.

* fix(decisions): let health_check_params override the evaluation probe

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): audit the decisions endpoint across providers, limits, health and chaos

Adds the /v1/decisions audit cells: one wire contract per provider (path, key, body and cost-map billing), the gateway-only fields and tags, the sad paths (invalid bodies, unknown model, key checks, api_base in the body, upstream 401/429/500, a 200 without answers, an unreachable upstream), the two evaluation-mode health probes, and three chaos cells (a mixed-failure burst over both routes, a worker SIGKILL mid-burst, an upstream outage and restart on the same port).

The PR's cost case read the upstream observations through the gateway, which answers 404 for that path; it now reads them from the upstream URL. The owned proxy harness takes extra CLI arguments, and its graceful stop waits as long as a worker boot may take, since a worker still starting honors SIGTERM only once it is up and the 30 second wait forced a cleanup under load.

* fix(decisions): send env API keys to a configured api_base

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(decisions): add zero-cost evaluation cost-map entry for Strands Decider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): register the routes through the lazy feature registry

The Decisions router was included at import, ahead of the config and DB
pass-through endpoints, so a pass-through configured at /v1/decisions
was skipped and answered 400 as an unknown Decisions provider. The
routes now register through LAZY_FEATURES, which splices them in after
every eager route, so a pass-through at /v1/decisions keeps its route
while /decisions still serves natively. The lazy OpenAPI snapshot carries
the two paths so the schema shows them before the first call.

The audit cells add the env-key egress to a configured api_base, the
client api_base opt-in shared with chat, the pass-through precedence on
an owned proxy, and the Strands evaluation health check resolved from
the cost map. The integration config exports the Perplexity env key the
first cell needs.

* fix(decisions): keep the Cloudflare api_base message in its transformation and read the audit upstream once per cell

* fix(proxy): let a config pass-through beat a lazily registered route in eager mode

With LITELLM_DISABLE_LAZY_ROUTES set the decisions routes are registered at
startup, so SafeRouteAdder treated a config pass-through at exactly
/v1/decisions as already registered and dropped it. In lazy mode a pass-through
created through the API after the first native call was skipped the same way.
Routes a lazy feature owns no longer count as registered, and a route added at
one of their paths is placed ahead of them, the precedence lazy mode gives a
config pass-through when the feature has not loaded yet.

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot] 2026-10-03 17:38:38 +00:00 • committed by GitHub
parent b024950353
commit 8b1990b4bc
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
50 changed files with 3011 additions and 40 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.github/workflows/test-unit.yml vendored
View file

@ -61,7 +61,7 @@ jobs:
- shard: core-utils
artifact-name: core-utils
test-path: ""
test-path: tests/unit/decisions
unit-flag: core-utils
workers: 2
reruns: 1
@ -141,6 +141,7 @@ jobs:
artifact-name: proxy-endpoints
test-path: >-
tests/unit/proxy/analytics_endpoints
tests/unit/proxy/decisions_endpoints
tests/unit/proxy/management_endpoints
tests/unit/proxy/list_api
tests/unit/proxy/memory

2
README.md
View file

@ -390,11 +390,13 @@ Set `LITELLM_PROXY_API_BASE` and `LITELLM_PROXY_API_KEY` and every model call th
| [Sail (`sail`)](https://docs.litellm.ai/docs/providers/sail) | ✅ | ✅ | ✅ | | | | | | | |
| [Sambanova (`sambanova`)](https://docs.litellm.ai/docs/providers/sambanova) | ✅ | ✅ | ✅ | | | | | | | |
| [Snowflake (`snowflake`)](https://docs.litellm.ai/docs/providers/snowflake) | ✅ | ✅ | ✅ | | | | | | | |
| [Strands Decider (`strands_decider`)](https://docs.litellm.ai/docs/providers) | | | | | | | | | | |
| [Text Completion Codestral (`text-completion-codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | |
| [Text Completion OpenAI (`text-completion-openai`)](https://docs.litellm.ai/docs/providers/text_completion_openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| [Together AI (`together_ai`)](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | ✅ | ✅ | | | | | | | |
| [Topaz (`topaz`)](https://docs.litellm.ai/docs/providers/topaz) | ✅ | ✅ | ✅ | | | | | | | |
| [Triton (`triton`)](https://docs.litellm.ai/docs/providers/triton-inference-server) | ✅ | ✅ | ✅ | | | | | | | |
| [Typesafe Decisions API (`typesafe`)](https://docs.litellm.ai/docs/providers) | | | | | | | | | | |
| [V0 (`v0`)](https://docs.litellm.ai/docs/providers/v0) | ✅ | ✅ | ✅ | | | | | | | |
| [Vercel AI Gateway (`vercel_ai_gateway`)](https://docs.litellm.ai/docs/providers/vercel_ai_gateway) | ✅ | ✅ | ✅ | | | | | | | |
| [VLLM (`vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | |

4
gateway/routes/allowlist.py
View file

@ -1,7 +1,7 @@
"""Path allowlist for the gateway component.
The gateway exposes the LLM data-plane surface: chat/completions, embeddings,
audio, batches, files, fine-tuning, rerank, ocr, rag, video, search, image,
audio, batches, files, fine-tuning, rerank, decisions, ocr, rag, video, search, image,
responses, vector stores, passthrough providers, realtime websockets, MCP
tool-call endpoints, and operational endpoints (/health, /metrics, and the
/debug/memory/summary read of the serving worker's RSS).
@ -60,6 +60,8 @@ GATEWAY_PATH_PREFIXES: tuple[str, ...] = (
"/v1/rerank",
"/v2/rerank",
"/rerank",
"/v1/decisions",
"/decisions",
"/v1/ocr",
"/ocr",
"/v1/rag/",

1
litellm/__init__.py
View file

@ -1473,6 +1473,7 @@ from .embeddings.dispatch import *
from .rust_bridge import rust
from .rag.main import *
from .sandbox.main import *
from .decisions.main import *
from .search.main import *
from .realtime_api.main import (
_arealtime,

14
litellm/cost_calculator.py
View file

@ -99,6 +99,7 @@ from litellm.llms.vertex_ai.cost_calculator import cost_router as google_cost_ro
from litellm.llms.xai.cost_calculator import cost_per_token as xai_cost_per_token
from litellm.responses.utils import ResponseAPILoggingUtils
from litellm.types.agents import LiteLLMSendMessageResponse
from litellm.types.decisions import DecisionsResponse, DecisionsUsage
from litellm.types.llms.base import CachedTokensDetails
from litellm.types.llms.openai import (
HttpxBinaryResponseContent,
@ -1058,6 +1059,7 @@ def _is_known_usage_objects(usage_obj):
return (
isinstance(usage_obj, litellm.Usage)
or isinstance(usage_obj, ResponseAPIUsage)
or isinstance(usage_obj, DecisionsUsage)
or TranscriptionUsageObjectTransformation.is_transcription_usage_object(usage_obj)
)
@ -1466,7 +1468,12 @@ def completion_cost(
"usage",
litellm.Usage(**_usage_for_dump.model_dump()),
)
if usage_obj is None:
if isinstance(usage_obj, DecisionsUsage):
_usage = {
"prompt_tokens": usage_obj.input_tokens,
"completion_tokens": usage_obj.output_tokens,
}
elif usage_obj is None:
_usage = {}
elif isinstance(usage_obj, BaseModel):
_usage = cast(BaseModel, usage_obj).model_dump()
@ -1957,7 +1964,8 @@ def response_cost_calculator(
| LiteLLMRealtimeStreamLoggingObject
| OpenAIModerationResponse
| Response
| SearchResponse,
| SearchResponse
| DecisionsResponse,
model: str,
custom_llm_provider: str | None,
call_type: Literal[
@ -1979,6 +1987,8 @@ def response_cost_calculator(
"arerank",
"search",
"asearch",
"decisions",
"adecisions",
],
optional_params: dict,
cache_hit: bool | None = None,

3
litellm/decisions/__init__.py Normal file
View file

@ -0,0 +1,3 @@
from litellm.decisions.main import adecisions, decisions
__all__ = ["adecisions", "decisions"]

299
litellm/decisions/main.py Normal file
View file

@ -0,0 +1,299 @@
from collections.abc import Mapping
from dataclasses import dataclass, field
from types import MappingProxyType
from typing import Final
import httpx
from pydantic import TypeAdapter, ValidationError
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.decisions.transformation import DecisionsProviderConfig
from litellm.llms.cloudflare.decisions.transformation import CLOUDFLARE_DECISIONS_ENDPOINT
from litellm.llms.custom_httpx.http_handler import _get_httpx_client, get_async_httpx_client
from litellm.llms.openrouter.decisions.transformation import OPENROUTER_DECISIONS_ENDPOINT
from litellm.llms.perplexity.decisions.transformation import PERPLEXITY_DECISIONS_ENDPOINT
from litellm.llms.strands_decider.decisions.transformation import STRANDS_DECIDER_DECISIONS_ENDPOINT
from litellm.llms.typesafe.decisions.transformation import TYPESAFE_DECISIONS_ENDPOINT
from litellm.secret_managers.main import get_secret_str
from litellm.types.decisions import (
DecisionQuestion,
DecisionsJSON,
DecisionsRequest,
DecisionsResponse,
)
from litellm.utils import client
DECISIONS_ENDPOINTS: Final[Mapping[str, DecisionsProviderConfig]] = MappingProxyType(
{
"perplexity": PERPLEXITY_DECISIONS_ENDPOINT,
"typesafe": TYPESAFE_DECISIONS_ENDPOINT,
"openrouter": OPENROUTER_DECISIONS_ENDPOINT,
"cloudflare": CLOUDFLARE_DECISIONS_ENDPOINT,
"strands_decider": STRANDS_DECIDER_DECISIONS_ENDPOINT,
}
)
_DECISIONS_REQUEST_ADAPTER: Final[TypeAdapter[DecisionsRequest]] = TypeAdapter(DecisionsRequest)
_DECISIONS_PAYLOAD_ADAPTER: Final[TypeAdapter[object]] = TypeAdapter(object)
_DECISIONS_RESPONSE_ADAPTER: Final[TypeAdapter[DecisionsResponse]] = TypeAdapter(DecisionsResponse)
@dataclass(frozen=True, slots=True, repr=False)
class _PreparedDecisionsRequest:
config: DecisionsProviderConfig
provider: str
upstream_model: str
url: str
api_key: str | None = field(repr=False)
headers: Mapping[str, str] = field(repr=False)
body: Mapping[str, object] = field(repr=False)
def _resolve_provider_model(model: str, custom_llm_provider: str | None) -> tuple[str, str]:
provider: Final = model.partition("/")[0] if custom_llm_provider is None else custom_llm_provider
if provider not in DECISIONS_ENDPOINTS:
supported: Final = ", ".join(DECISIONS_ENDPOINTS)
raise litellm.BadRequestError(
message=f"Unknown Decisions provider '{provider}'. Supported providers: {supported}",
model=model,
llm_provider=provider,
)
upstream_model: Final = model.removeprefix(f"{provider}/") if model.startswith(f"{provider}/") else model
if not upstream_model:
raise litellm.BadRequestError(
message="A model name is required for the Decisions API",
model=model,
llm_provider=provider,
)
return provider, upstream_model
def _resolve_api_key(
*,
provider: str,
model: str,
endpoint: DecisionsProviderConfig,
api_key: str | None,
) -> str | None:
if api_key is not None:
return api_key
server_api_key: Final = next(
(key for key in (get_secret_str(name) for name in endpoint.api_key_env) if key),
None,
)
if server_api_key is None:
if not endpoint.api_key_required:
return None
raise litellm.AuthenticationError(
message=f"Missing API key for Decisions provider '{provider}'",
model=model,
llm_provider=provider,
)
return server_api_key
def _prepare_request(
*,
model: str,
state: DecisionsJSON,
questions: Mapping[str, DecisionQuestion | Mapping[str, object]],
api_key: str | None,
api_base: str | None,
custom_llm_provider: str | None,
extra_headers: Mapping[str, str] | None,
) -> _PreparedDecisionsRequest:
provider, upstream_model = _resolve_provider_model(model, custom_llm_provider)
try:
validated_request: Final = _DECISIONS_REQUEST_ADAPTER.validate_python(
{"model": model, "state": state, "questions": questions}
)
except ValidationError as error:
raise litellm.BadRequestError(
message=f"Invalid Decisions request: {error}",
model=model,
llm_provider=provider,
) from error
endpoint: Final = DECISIONS_ENDPOINTS[provider]
env_api_base: Final = get_secret_str(endpoint.api_base_env)
default_api_base: Final = endpoint.default_api_base()
resolved_api_base: Final = api_base or env_api_base or default_api_base
if resolved_api_base is None:
raise litellm.BadRequestError(
message=endpoint.missing_api_base_message(provider),
model=model,
llm_provider=provider,
)
resolved_api_key: Final = _resolve_api_key(
provider=provider,
model=model,
endpoint=endpoint,
api_key=api_key,
)
canonical_model: Final = endpoint.canonical_model(upstream_model)
outbound_headers: Final = MappingProxyType(
{
**{
name: value
for name, value in (extra_headers or {}).items()
if name.lower() not in {"authorization", "content-type"}
},
**({"Authorization": f"Bearer {resolved_api_key}"} if resolved_api_key is not None else {}),
"Content-Type": "application/json",
}
)
body: Final = MappingProxyType(
{
"model": endpoint.request_model(canonical_model),
"state": validated_request.state,
"questions": {
name: question.model_dump(mode="json", exclude_none=True)
for name, question in validated_request.questions.items()
},
}
)
return _PreparedDecisionsRequest(
config=endpoint,
provider=provider,
upstream_model=canonical_model,
url=endpoint.endpoint_url(resolved_api_base, canonical_model),
api_key=resolved_api_key,
headers=outbound_headers,
body=body,
)
def _log_request(
prepared: _PreparedDecisionsRequest,
kwargs: Mapping[str, object],
) -> LiteLLMLoggingObj | None:
logging_obj: Final = kwargs.get("litellm_logging_obj")
if not isinstance(logging_obj, LiteLLMLoggingObj):
return None
logging_obj.update_from_kwargs(
kwargs=dict(kwargs),
model=prepared.upstream_model,
litellm_params={
"litellm_call_id": kwargs.get("litellm_call_id"),
"api_base": prepared.url,
},
custom_llm_provider=prepared.provider,
)
request_body: Final = dict(prepared.body)
request_headers: Final = dict(prepared.headers)
logging_obj.pre_call(
input=request_body,
api_key=prepared.api_key,
model=prepared.upstream_model,
additional_args={
"api_base": prepared.url,
"complete_input_dict": request_body,
"headers": request_headers,
},
)
return logging_obj
def _parse_response(
response: httpx.Response,
prepared: _PreparedDecisionsRequest,
) -> DecisionsResponse:
response.raise_for_status()
payload: Final[object] = _DECISIONS_PAYLOAD_ADAPTER.validate_json(response.content)
result: Final = _DECISIONS_RESPONSE_ADAPTER.validate_python(prepared.config.unwrap_response(payload))
result._hidden_params.update(
{
"model": f"{prepared.provider}/{prepared.upstream_model}",
"custom_llm_provider": prepared.provider,
"provider_response_model": f"{prepared.provider}/{prepared.upstream_model}",
}
)
return result
def _map_upstream_exception(error: Exception, prepared: _PreparedDecisionsRequest) -> Exception:
return litellm.exception_type(
model=f"{prepared.provider}/{prepared.upstream_model}",
custom_llm_provider=prepared.provider,
original_exception=error,
)
@client
async def adecisions(
model: str,
state: DecisionsJSON,
questions: Mapping[str, DecisionQuestion | Mapping[str, object]],
api_key: str | None = None,
api_base: str | None = None,
timeout: float | httpx.Timeout | None = None,
custom_llm_provider: str | None = None,
extra_headers: Mapping[str, str] | None = None,
**kwargs: object,
) -> DecisionsResponse:
prepared: Final = _prepare_request(
model=model,
state=state,
questions=questions,
api_key=api_key,
api_base=api_base,
custom_llm_provider=custom_llm_provider,
extra_headers=extra_headers,
)
logging_obj: Final = _log_request(prepared, kwargs)
try:
handler: Final = get_async_httpx_client(llm_provider=prepared.provider)
response: Final = await handler.post(
prepared.url,
json=dict(prepared.body),
headers=dict(prepared.headers),
timeout=timeout,
logging_obj=logging_obj,
)
return _parse_response(response=response, prepared=prepared)
except Exception as error:
raise _map_upstream_exception(error, prepared) from error
@client
def decisions(
model: str,
state: DecisionsJSON,
questions: Mapping[str, DecisionQuestion | Mapping[str, object]],
api_key: str | None = None,
api_base: str | None = None,
timeout: float | httpx.Timeout | None = None,
custom_llm_provider: str | None = None,
extra_headers: Mapping[str, str] | None = None,
**kwargs: object,
) -> DecisionsResponse:
prepared: Final = _prepare_request(
model=model,
state=state,
questions=questions,
api_key=api_key,
api_base=api_base,
custom_llm_provider=custom_llm_provider,
extra_headers=extra_headers,
)
logging_obj: Final = _log_request(prepared, kwargs)
try:
handler: Final = _get_httpx_client()
response: Final = handler.post(
prepared.url,
json=dict(prepared.body),
headers=dict(prepared.headers),
timeout=timeout,
logging_obj=logging_obj,
)
return _parse_response(response=response, prepared=prepared)
except Exception as error:
raise _map_upstream_exception(error, prepared) from error
__all__ = ["DECISIONS_ENDPOINTS", "adecisions", "decisions"]

12
litellm/litellm_core_utils/health_check_helpers.py
View file

@ -168,6 +168,7 @@ class HealthCheckHelpers:
"batch",
"responses",
"ocr",
"evaluation",
],
Callable,
]:
@ -190,7 +191,7 @@ class HealthCheckHelpers:
from litellm.litellm_core_utils.audio_utils.utils import (
get_audio_file_for_health_check,
)
from litellm.litellm_core_utils.health_check_utils import _filter_model_params
from litellm.litellm_core_utils.health_check_utils import DECISIONS_CALL_PARAMS, _filter_model_params
from litellm.realtime_api.main import _realtime_health_check
return {
@ -257,4 +258,13 @@ class HealthCheckHelpers:
**_filter_model_params(model_params=model_params),
document=_ocr_health_check_document(model=model, custom_llm_provider=custom_llm_provider),
),
"evaluation": lambda: litellm.adecisions(
**DECISIONS_CALL_PARAMS.validate_python(
{
"state": prompt or "health check",
"questions": {"reachable": {"type": "noul", "instructions": "Is the service reachable?"}},
**_filter_model_params(model_params=model_params),
}
)
),
}

6
litellm/litellm_core_utils/health_check_utils.py
View file

@ -4,6 +4,12 @@ Utils used for litellm.ahealth_check()
from typing import Final
from pydantic import TypeAdapter
from litellm.types.decisions import DecisionsCallParams
DECISIONS_CALL_PARAMS: Final[TypeAdapter[DecisionsCallParams]] = TypeAdapter(DecisionsCallParams)
def _filter_model_params(model_params: dict) -> dict:
"""Remove 'messages' param from model params."""

3
litellm/litellm_core_utils/litellm_logging.py
View file

@ -118,6 +118,7 @@ from litellm.llms.base_llm.search.transformation import SearchResponse
from litellm.responses.utils import ResponseAPILoggingUtils
from litellm.types.agents import LiteLLMSendMessageResponse
from litellm.types.containers.main import ContainerObject
from litellm.types.decisions import DecisionsResponse
from litellm.types.integrations.s3_v2 import S3PartitionGranularity
from litellm.types.interactions import (
InteractionsAPIResponse,
@ -1815,6 +1816,7 @@ class Logging(LiteLLMLoggingBaseClass):
LiteLLMRealtimeStreamLoggingObject,
OpenAIModerationResponse,
"SearchResponse",
DecisionsResponse,
dict,
list,
],
@ -2600,6 +2602,7 @@ class Logging(LiteLLMLoggingBaseClass):
or isinstance(logging_result, OpenAIModerationResponse)
or isinstance(logging_result, OCRResponse) # OCR
or isinstance(logging_result, SearchResponse) # Search API
or isinstance(logging_result, DecisionsResponse)
or (
isinstance(logging_result, InteractionsAPIResponse)
and logging_result.usage is not None

3
litellm/llms/base_llm/decisions/__init__.py Normal file
View file

@ -0,0 +1,3 @@
from .transformation import DecisionsProviderConfig, JevCompatibleDecisionsEndpoint
__all__ = ["DecisionsProviderConfig", "JevCompatibleDecisionsEndpoint"]

52
litellm/llms/base_llm/decisions/transformation.py Normal file
View file

@ -0,0 +1,52 @@
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True, slots=True)
class JevCompatibleDecisionsEndpoint:
default_api_base_value: str | None
path: str
api_key_env: tuple[str, ...]
api_base_env: str
api_key_required: bool = True
def default_api_base(self) -> str | None:
return self.default_api_base_value
def missing_api_base_message(self, provider: str) -> str:
return f"api_base is required for Decisions provider '{provider}'"
def canonical_model(self, model: str) -> str:
return model
def request_model(self, model: str) -> str:
return model
def endpoint_url(self, api_base: str, model: str) -> str:
return f"{api_base.rstrip('/').removesuffix('/v1')}{self.path}"
def unwrap_response(self, payload: object) -> object:
return payload
class DecisionsProviderConfig(Protocol):
@property
def api_key_env(self) -> tuple[str, ...]: ...
@property
def api_base_env(self) -> str: ...
@property
def api_key_required(self) -> bool: ...
def default_api_base(self) -> str | None: ...
def missing_api_base_message(self, provider: str) -> str: ...
def canonical_model(self, model: str) -> str: ...
def request_model(self, model: str) -> str: ...
def endpoint_url(self, api_base: str, model: str) -> str: ...
def unwrap_response(self, payload: object) -> object: ...

58
litellm/llms/cloudflare/decisions/transformation.py Normal file
View file

@ -0,0 +1,58 @@
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Final
from pydantic import TypeAdapter
from litellm.secret_managers.main import (
get_secret_str,
normalize_nonempty_secret_str,
)
_RESPONSE_MAPPING_ADAPTER: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, object])
@dataclass(frozen=True, slots=True)
class CloudflareDecisionsEndpoint:
api_key_env: tuple[str, ...] = ("CLOUDFLARE_API_KEY",)
api_base_env: str = "CLOUDFLARE_API_BASE"
api_key_required: bool = True
def default_api_base(self) -> str | None:
account_id: Final = normalize_nonempty_secret_str(get_secret_str("CLOUDFLARE_ACCOUNT_ID"))
if account_id is None:
return None
return f"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run"
def missing_api_base_message(self, provider: str) -> str:
return "Missing CLOUDFLARE_ACCOUNT_ID - set CLOUDFLARE_ACCOUNT_ID or pass api_base explicitly"
def canonical_model(self, model: str) -> str:
if model.startswith("@cf/"):
return model
return f"@cf/cloudflare/{model}"
def request_model(self, model: str) -> str:
return model.rsplit("/", maxsplit=1)[-1]
def endpoint_url(self, api_base: str, model: str) -> str:
normalized_api_base: Final = api_base.rstrip("/")
if normalized_api_base.endswith("/ai/v1"):
return f"{normalized_api_base.removesuffix('/ai/v1')}/ai/run/{model}"
if normalized_api_base.endswith("/ai/run"):
return f"{normalized_api_base}/{model}"
return f"{normalized_api_base}/ai/run/{model}"
def unwrap_response(self, payload: object) -> object:
if not isinstance(payload, Mapping):
return payload
response_mapping: Final = _RESPONSE_MAPPING_ADAPTER.validate_python(payload)
if "answers" in response_mapping:
return payload
result: Final = response_mapping.get("result")
if isinstance(result, Mapping):
return result
return payload
CLOUDFLARE_DECISIONS_ENDPOINT: Final[CloudflareDecisionsEndpoint] = CloudflareDecisionsEndpoint()

10
litellm/llms/openrouter/decisions/transformation.py Normal file
View file

@ -0,0 +1,10 @@
from typing import Final
from litellm.llms.base_llm.decisions.transformation import JevCompatibleDecisionsEndpoint
OPENROUTER_DECISIONS_ENDPOINT: Final[JevCompatibleDecisionsEndpoint] = JevCompatibleDecisionsEndpoint(
default_api_base_value="https://openrouter.ai/api",
path="/alpha/decisions",
api_key_env=("OPENROUTER_API_KEY",),
api_base_env="OPENROUTER_API_BASE",
)

10
litellm/llms/perplexity/decisions/transformation.py Normal file
View file

@ -0,0 +1,10 @@
from typing import Final
from litellm.llms.base_llm.decisions.transformation import JevCompatibleDecisionsEndpoint
PERPLEXITY_DECISIONS_ENDPOINT: Final[JevCompatibleDecisionsEndpoint] = JevCompatibleDecisionsEndpoint(
default_api_base_value="https://api.perplexity.ai",
path="/v1/decisions",
api_key_env=("PERPLEXITYAI_API_KEY", "PERPLEXITY_API_KEY"),
api_base_env="PERPLEXITY_API_BASE",
)

11
litellm/llms/strands_decider/decisions/transformation.py Normal file
View file

@ -0,0 +1,11 @@
from typing import Final
from litellm.llms.base_llm.decisions.transformation import JevCompatibleDecisionsEndpoint
STRANDS_DECIDER_DECISIONS_ENDPOINT: Final[JevCompatibleDecisionsEndpoint] = JevCompatibleDecisionsEndpoint(
default_api_base_value=None,
path="/v1/systemone",
api_key_env=("STRANDS_DECIDER_API_KEY",),
api_base_env="STRANDS_DECIDER_API_BASE",
api_key_required=False,
)

10
litellm/llms/typesafe/decisions/transformation.py Normal file
View file

@ -0,0 +1,10 @@
from typing import Final
from litellm.llms.base_llm.decisions.transformation import JevCompatibleDecisionsEndpoint
TYPESAFE_DECISIONS_ENDPOINT: Final[JevCompatibleDecisionsEndpoint] = JevCompatibleDecisionsEndpoint(
default_api_base_value="https://api.typesafe.ai",
path="/v1/systemone",
api_key_env=("TYPESAFE_API_KEY",),
api_base_env="TYPESAFE_API_BASE",
)

50
litellm/model_prices_and_context_window_backup.json
View file

@ -15612,6 +15612,38 @@
"prompt_cache_min_tokens": 1024,
"source": "https://platform.claude.com/docs/en/about-claude/pricing"
},
"cloudflare/clef": {
"input_cost_per_token": 2.4e-07,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/clef-flash": {
"input_cost_per_token": 9e-08,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/cloudflare/clef": {
"input_cost_per_token": 2.4e-07,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/cloudflare/clef-flash": {
"input_cost_per_token": 9e-08,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/meta/llama-2-7b-chat-fp16": {
"input_cost_per_token": 1.923e-06,
"litellm_provider": "cloudflare",
@ -44421,6 +44453,14 @@
"mode": "chat",
"output_cost_per_token": 2.8e-07
},
"perplexity/pplx-decider-v1-27b": {
"input_cost_per_token": 4e-08,
"litellm_provider": "perplexity",
"max_input_tokens": 262144,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://docs.perplexity.ai/api-reference/decisions-post"
},
"perplexity/sonar": {
"input_cost_per_token": 1e-06,
"litellm_provider": "perplexity",
@ -72800,6 +72840,16 @@
"notes": "Self-hosted decision model; infrastructure costs are paid separately"
}
},
"strands_decider/strands-decider-2B-hobson-v19": {
"input_cost_per_token": 0.0,
"litellm_provider": "strands_decider",
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://huggingface.co/StrandsAgents/strands-decider-2B-hobson-v19",
"metadata": {
"notes": "Self-hosted decision model; infrastructure costs are paid separately"
}
},
"typesafe/jev-1.13.0": {
"input_cost_per_token": 4.2e-08,
"litellm_provider": "typesafe",

27
litellm/proxy/_lazy_features.py
View file

@ -16,6 +16,7 @@ from collections.abc import Set as AbstractSet
from contextlib import asynccontextmanager
from dataclasses import dataclass, field
from functools import partial
from itertools import chain
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
@ -268,6 +269,11 @@ LAZY_FEATURES: Final[tuple[LazyFeature, ...]] = (
module_path="litellm.proxy.openai_evals_endpoints.endpoints",
path_prefixes=("/v1/evals", "/evals"),
),
LazyFeature(
name="decisions",
module_path="litellm.proxy.decisions_endpoints.endpoints",
path_prefixes=("/v1/decisions", "/decisions"),
),
LazyFeature(
name="claude_code_marketplace",
module_path="litellm.proxy.anthropic_endpoints.claude_code_endpoints",
@ -378,6 +384,10 @@ def _lazy_slots(app: "FastAPI") -> Mapping[str, BaseRoute | None]:
return app.state.lazy_slots if hasattr(app.state, "lazy_slots") else MappingProxyType({})
def _lazy_routes(app: "FastAPI") -> Mapping[str, tuple[BaseRoute, ...]]:
return app.state.lazy_routes if hasattr(app.state, "lazy_routes") else MappingProxyType({})
def reserve_lazy_slot(app: "FastAPI", name: str, features: tuple[LazyFeature, ...] = LAZY_FEATURES) -> None:
"""Record the route the feature's router used to be included after, so its routes
are spliced back in there once it loads and keep the same precedence. Anchoring on
@ -474,11 +484,8 @@ def _lazy_lock(app: "FastAPI", module_path: str) -> asyncio.Lock:
def _register_feature(app: "FastAPI", feat: LazyFeature, module: object, features: tuple[LazyFeature, ...]) -> None:
before: Final = len(app.router.routes)
feat.register_fn(app, module)
previous: Final[Mapping[str, tuple[BaseRoute, ...]]] = (
app.state.lazy_routes if hasattr(app.state, "lazy_routes") else MappingProxyType({})
)
lazy_routes: Final[Mapping[str, tuple[BaseRoute, ...]]] = MappingProxyType(
{**previous, feat.module_path: tuple(app.router.routes[before:])}
{**_lazy_routes(app), feat.module_path: tuple(app.router.routes[before:])}
)
app.state.lazy_routes = lazy_routes # rebind-ok: the app owns the record of which routes each feature added
app.router.routes[:] = hot_routes_first( # rebind-ok: the app owns its route table
@ -543,11 +550,8 @@ def _register_all_on_startup(inner: "Lifespan[FastAPI]", features: tuple[LazyFea
def _restore_registry_order(app: "FastAPI", features: tuple[LazyFeature, ...]) -> None:
present: Final = frozenset(id(route) for route in app.router.routes)
registered: Final[Mapping[str, tuple[BaseRoute, ...]]] = (
app.state.lazy_routes if hasattr(app.state, "lazy_routes") else MappingProxyType({})
)
still_routed: Final = MappingProxyType(
{module_path: tuple(r for r in routes if id(r) in present) for module_path, routes in registered.items()}
{module_path: tuple(r for r in routes if id(r) in present) for module_path, routes in _lazy_routes(app).items()}
)
app.router.routes[:] = hot_routes_first( # rebind-ok: the app owns its route table
_in_registry_order(app.router.routes, still_routed, features, _lazy_slots(app))
@ -599,6 +603,13 @@ def _make_warmup_router(app: "FastAPI", features: tuple[LazyFeature, ...] = LAZY
return router
def lazy_owned_routes(app: "FastAPI") -> frozenset[int]:
"""ids of the routes lazy features have registered on this app. A route added later at
one of their paths (a config pass-through at /v1/decisions) goes ahead of them, the
precedence lazy mode gives it when the feature has not loaded by the time the config is read."""
return frozenset(id(route) for route in chain.from_iterable(_lazy_routes(app).values()))
def loaded_lazy_modules(app: "FastAPI") -> frozenset[str]:
"""The set of lazy feature modules whose routers are actually registered
on this app (tracked by _install), empty until a feature loads or eager startup runs.

55
litellm/proxy/_lazy_openapi_snapshot.json
View file

@ -9492,6 +9492,61 @@
}
}
},
"decisions": {
"components": {
"schemas": {}
},
"paths": {
"/decisions": {
"post": {
"operationId": "decisions_decisions_post",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {}
}
},
"description": "Successful Response"
}
},
"security": [
{
"APIKeyHeader": []
}
],
"summary": "Decisions",
"tags": [
"decisions"
]
}
},
"/v1/decisions": {
"post": {
"operationId": "decisions_v1_decisions_post",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {}
}
},
"description": "Successful Response"
}
},
"security": [
{
"APIKeyHeader": []
}
],
"summary": "Decisions",
"tags": [
"decisions"
]
}
}
}
},
"evals": {
"components": {
"schemas": {

2
litellm/proxy/_types.py
View file

@ -478,6 +478,8 @@ class LiteLLMRoutes(enum.Enum):
"/v1/search",
"/search/{search_tool_name}",
"/v1/search/{search_tool_name}",
"/decisions",
"/v1/decisions",
# OCR
"/ocr",
"/v1/ocr",

2
litellm/proxy/agent_endpoints/auth/managed_authorization.py
View file

@ -29,6 +29,7 @@ _MANAGED_MODEL_ROUTES: Final = frozenset(
"audio/speech",
"moderations",
"rerank",
"decisions",
"ocr",
),
)
@ -72,6 +73,7 @@ _MODEL_ROUTE_KINDS: Final[
"/audio/transcriptions": "moderation",
"/audio/speech": "speech",
"/rerank": "body",
"/decisions": "body",
"/messages/count_tokens": "body",
":countTokens": "path",
}

2
litellm/proxy/common_request_processing.py
View file

@ -176,6 +176,7 @@ ProxyRouteType: TypeAlias = Literal[
"avector_store_file_delete",
"aocr",
"asearch",
"adecisions",
"avideo_generation",
"avideo_list",
"avideo_status",
@ -1961,6 +1962,7 @@ class ProxyBaseLLMRequestProcessing:
"avector_store_file_delete",
"aocr",
"asearch",
"adecisions",
"avideo_generation",
"avideo_list",
"avideo_status",

1
litellm/proxy/decisions_endpoints/__init__.py Normal file
View file

@ -0,0 +1 @@
__all__ = ()

102
litellm/proxy/decisions_endpoints/endpoints.py Normal file
View file

@ -0,0 +1,102 @@
from typing import Annotated, Final
from fastapi import APIRouter, Depends, Request, Response
from fastapi.responses import ORJSONResponse # pyright: ignore[reportDeprecated] # required endpoint contract
from pydantic import TypeAdapter, ValidationError
from litellm.exceptions import BadRequestError
from litellm.proxy.auth.user_api_key_auth import UserAPIKeyAuth, user_api_key_auth
from litellm.proxy.common_request_processing import ProxyBaseLLMRequestProcessing
from litellm.types.decisions import DecisionsRequestBody
router: Final = APIRouter()
_REQUEST_DATA_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
_DECISIONS_REQUEST_BODY_ADAPTER: Final[TypeAdapter[DecisionsRequestBody]] = TypeAdapter(DecisionsRequestBody)
_GENERAL_SETTINGS_ADAPTER: Final[TypeAdapter[dict[str, object]]] = TypeAdapter(dict[str, object])
_OPTIONAL_STRING_ADAPTER: Final[TypeAdapter[str | None]] = TypeAdapter(str | None)
_OPTIONAL_FLOAT_ADAPTER: Final[TypeAdapter[float | None]] = TypeAdapter(float | None)
@router.post(
"/v1/decisions",
dependencies=[Depends(user_api_key_auth)],
response_class=ORJSONResponse, # pyright: ignore[reportDeprecated] # required endpoint contract
tags=["decisions"],
)
@router.post(
"/decisions",
dependencies=[Depends(user_api_key_auth)],
response_class=ORJSONResponse, # pyright: ignore[reportDeprecated] # required endpoint contract
tags=["decisions"],
)
async def decisions(
request: Request,
fastapi_response: Response,
user_api_key_dict: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
):
from litellm.proxy.proxy_server import (
general_settings as proxy_general_settings,
)
from litellm.proxy.proxy_server import (
llm_router,
proxy_config,
proxy_logging_obj,
user_max_tokens,
user_request_timeout,
version,
)
from litellm.proxy.proxy_server import (
user_api_base as proxy_user_api_base,
)
from litellm.proxy.proxy_server import (
user_model as proxy_user_model,
)
from litellm.proxy.proxy_server import (
user_temperature as proxy_user_temperature,
)
data: Final = _REQUEST_DATA_ADAPTER.validate_json(await request.body())
general_settings: Final = _GENERAL_SETTINGS_ADAPTER.validate_python(proxy_general_settings)
user_api_base: Final = _OPTIONAL_STRING_ADAPTER.validate_python(proxy_user_api_base)
user_model: Final = _OPTIONAL_STRING_ADAPTER.validate_python(proxy_user_model)
user_temperature: Final = _OPTIONAL_FLOAT_ADAPTER.validate_python(proxy_user_temperature)
processor: Final = ProxyBaseLLMRequestProcessing(data=data)
try:
_DECISIONS_REQUEST_BODY_ADAPTER.validate_python(data)
return await processor.base_process_llm_request(
request=request,
fastapi_response=fastapi_response,
user_api_key_dict=user_api_key_dict,
route_type="adecisions",
proxy_logging_obj=proxy_logging_obj,
llm_router=llm_router,
general_settings=general_settings,
proxy_config=proxy_config,
select_data_generator=None,
model=None,
user_model=user_model,
user_temperature=user_temperature,
user_request_timeout=user_request_timeout,
user_max_tokens=user_max_tokens,
user_api_base=user_api_base,
version=version,
)
except ValidationError as error:
bad_request_error: Final = BadRequestError(
message=f"Invalid Decisions request: {error}",
model=str(data.get("model", "")),
llm_provider="",
)
raise await processor._handle_llm_api_exception(
e=bad_request_error,
user_api_key_dict=user_api_key_dict,
proxy_logging_obj=proxy_logging_obj,
version=version,
)
except Exception as error:
raise await processor._handle_llm_api_exception(
e=error,
user_api_key_dict=user_api_key_dict,
proxy_logging_obj=proxy_logging_obj,
version=version,
)

50
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py
View file

@ -28,6 +28,7 @@ from fastapi import (
from fastapi.responses import StreamingResponse
from pydantic import TypeAdapter
from starlette.datastructures import UploadFile as StarletteUploadFile
from starlette.routing import BaseRoute, Route
from starlette.websockets import WebSocketState
from websockets.asyncio.client import connect
from websockets.exceptions import (
@ -67,6 +68,7 @@ from litellm.llms.base_llm.managed_resources.utils import (
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.llms.oss_decision import validate_oss_request
from litellm.passthrough import BasePassthroughUtils
from litellm.proxy._lazy_features import lazy_owned_routes
from litellm.proxy._types import (
ConfigFieldInfo,
ConfigFieldUpdate,
@ -2942,35 +2944,34 @@ def _extract_model_from_vertex_ai_setup(setup_response: Mapping[str, object]) ->
return None
def _placed_ahead(routes: Sequence[BaseRoute], moving: BaseRoute, before: BaseRoute) -> tuple[BaseRoute, ...]:
kept: Final = tuple(route for route in routes if route is not moving)
at: Final = next(index for index, route in enumerate(kept) if route is before)
return (*kept[:at], moving, *kept[at:])
class SafeRouteAdder:
"""
Wrapper class for adding routes to FastAPI app.
Only adds routes if they don't already exist on the app.
Only adds routes if they don't already exist on the app. A route a lazy feature registered
does not count: a route added at its path goes ahead of it, the precedence a config
pass-through at /v1/decisions gets in lazy mode, where the feature has not loaded yet.
"""
@staticmethod
def _colliding_routes(app: FastAPI, path: str, methods: Sequence[str]) -> tuple[Route, ...]:
wanted: Final = frozenset(methods)
return tuple(
route
for route in app.routes
if isinstance(route, Route) and route.path == path and not wanted.isdisjoint(route.methods or ())
)
@staticmethod
def _is_path_registered(app: FastAPI, path: str, methods: list[str]) -> bool:
"""
Check if a path with any of the specified methods is already registered on the app.
Args:
app: The FastAPI application instance
path: The path to check (e.g., "/v1/chat/completions")
methods: List of HTTP methods to check (e.g., ["GET", "POST"])
Returns:
True if the path is already registered with any of the methods, False otherwise
"""
for route in app.routes:
# Use getattr to safely access route attributes
route_path = getattr(route, "path", None)
route_methods = getattr(route, "methods", None)
if route_path == path and route_methods is not None:
# Check if any of the methods overlap
if any(method in route_methods for method in methods):
return True
return False
"""True when a route the app itself defines already serves the path with one of the methods."""
lazy_owned: Final = lazy_owned_routes(app)
return any(id(route) not in lazy_owned for route in SafeRouteAdder._colliding_routes(app, path, methods))
@staticmethod
def add_api_route_if_not_exists(
@ -3001,12 +3002,17 @@ class SafeRouteAdder:
)
return False
shadowed: Final = SafeRouteAdder._colliding_routes(app, path, methods)
app.add_api_route(
path=path,
endpoint=endpoint,
methods=methods,
dependencies=dependencies,
)
if shadowed:
app.router.routes[:] = _placed_ahead( # rebind-ok: the app owns its route table
app.router.routes, app.router.routes[-1], shadowed[0]
)
verbose_proxy_logger.debug(
"Successfully added route: %s with methods %s",
path,

2
litellm/proxy/route_llm_request.py
View file

@ -93,6 +93,7 @@ ROUTE_ENDPOINT_MAPPING: Final = {
"acompact_responses": "/responses/compact",
"aocr": "/ocr",
"asearch": "/search",
"adecisions": "/decisions",
"avideo_generation": "/videos",
"avideo_list": "/videos",
"avideo_status": "/videos/{video_id}",
@ -487,6 +488,7 @@ RouteType = Literal[
"avector_store_file_delete",
"aocr",
"asearch",
"adecisions",
"avideo_generation",
"avideo_list",
"avideo_status",

9
litellm/router.py
View file

@ -2110,6 +2110,11 @@ class Router:
self.asearch = self.factory_function(asearch, call_type="asearch")
self.search = self.factory_function(search, call_type="search")
from litellm.decisions import adecisions, decisions
self.adecisions = self.factory_function(adecisions, call_type="adecisions")
self.decisions = self.factory_function(decisions, call_type="decisions")
def _initialize_video_endpoints(self):
"""Initialize video endpoints."""
from litellm.videos import (
@ -6663,6 +6668,8 @@ class Router:
"ocr",
"asearch",
"search",
"adecisions",
"decisions",
"aadapter_generate_content",
"avideo_generation",
"video_generation",
@ -6736,6 +6743,7 @@ class Router:
"generate_content_stream",
"ocr",
"search",
"decisions",
"video_generation",
"video_list",
"video_status",
@ -6903,6 +6911,7 @@ class Router:
"agenerate_content_stream",
"aocr",
"ocr",
"adecisions",
"avideo_generation",
"avideo_list",
"avideo_status",

123
litellm/types/decisions.py Normal file
View file

@ -0,0 +1,123 @@
from collections.abc import Mapping, Sequence
from typing import Annotated, Literal, TypeAlias
from pydantic import ConfigDict, Field, PrivateAttr, model_validator, with_config
from typing_extensions import ReadOnly, Required, TypedDict
from litellm.types.llms.base import LiteLLMPydanticObjectBase
DecisionsJSON: TypeAlias = str | Mapping[str, object] | Sequence[object]
NoulCriteria: TypeAlias = Mapping[Literal["true", "false"], DecisionsJSON | None]
class NoulQuestion(LiteLLMPydanticObjectBase):
type: Literal["noul"]
instructions: DecisionsJSON | None = None
criteria: NoulCriteria | None = None
model_config = ConfigDict(extra="allow", frozen=True)
@model_validator(mode="after")
def require_instructions_or_criteria(self) -> "NoulQuestion":
if self.instructions is None and self.criteria is None:
raise ValueError("A noul question requires instructions or criteria")
return self
class ChoiceQuestion(LiteLLMPydanticObjectBase):
type: Literal["choice"]
instructions: DecisionsJSON | None = None
criteria: Annotated[Mapping[str, DecisionsJSON | None], Field(min_length=1, max_length=255)]
model_config = ConfigDict(extra="allow", frozen=True)
class ScoreQuestion(LiteLLMPydanticObjectBase):
type: Literal["score"]
instructions: DecisionsJSON | None = None
criteria: Annotated[Sequence[DecisionsJSON], Field(min_length=1, max_length=10)]
model_config = ConfigDict(extra="allow", frozen=True)
DecisionQuestion: TypeAlias = Annotated[
NoulQuestion | ChoiceQuestion | ScoreQuestion,
Field(discriminator="type"),
]
DecisionQuestionMap: TypeAlias = Annotated[
Mapping[Annotated[str, Field(min_length=1)], DecisionQuestion],
Field(min_length=1, max_length=128),
]
class DecisionsRequestBody(LiteLLMPydanticObjectBase):
state: DecisionsJSON
questions: DecisionQuestionMap
model_config = ConfigDict(extra="allow", frozen=True)
class DecisionsRequest(DecisionsRequestBody):
model: str
@with_config(ConfigDict(extra="allow"))
class DecisionsCallParams(TypedDict, total=False):
model: Required[ReadOnly[str]]
state: Required[ReadOnly[DecisionsJSON]]
questions: Required[ReadOnly[DecisionQuestionMap]]
api_key: ReadOnly[str | None]
api_base: ReadOnly[str | None]
timeout: ReadOnly[float | None]
custom_llm_provider: ReadOnly[str | None]
extra_headers: ReadOnly[Mapping[str, str] | None]
class NoulAnswer(LiteLLMPydanticObjectBase):
type: Literal["noul"]
noul: float
model_config = ConfigDict(extra="allow", frozen=True)
class ChoiceAnswer(LiteLLMPydanticObjectBase):
type: Literal["choice"]
choice: str
confidence: float
probabilities: Mapping[str, float]
model_config = ConfigDict(extra="allow", frozen=True)
class ScoreAnswer(LiteLLMPydanticObjectBase):
type: Literal["score"]
score: float
confidence: float
legend: Mapping[str, DecisionsJSON]
probabilities: Mapping[str, float]
model_config = ConfigDict(extra="allow", frozen=True)
DecisionAnswer: TypeAlias = Annotated[
NoulAnswer | ChoiceAnswer | ScoreAnswer,
Field(discriminator="type"),
]
class DecisionsUsage(LiteLLMPydanticObjectBase):
input_tokens: int = 0
output_tokens: int = 0
model_config = ConfigDict(extra="allow", frozen=True)
class DecisionsResponse(LiteLLMPydanticObjectBase):
model: str | None = None
answers: Mapping[str, DecisionAnswer]
usage: DecisionsUsage | None = None
model_config = ConfigDict(extra="allow", frozen=True)
_hidden_params: dict[str, object] = PrivateAttr(default_factory=dict)

8
litellm/types/utils.py
View file

@ -468,6 +468,8 @@ class CallTypes(str, Enum):
arerank = "arerank"
search = "search"
asearch = "asearch"
decisions = "decisions"
adecisions = "adecisions"
arealtime = "_arealtime"
aresponses_websocket = "_aresponses_websocket"
create_batch = "create_batch"
@ -654,6 +656,8 @@ CallTypesLiteral = Literal[
"arerank",
"search",
"asearch",
"decisions",
"adecisions",
"_arealtime",
"_aresponses_websocket",
"create_batch",
@ -763,6 +767,8 @@ API_ROUTE_TO_CALL_TYPES: Final[Mapping[str, Sequence[CallTypes]]] = {
# Search
"/search": [CallTypes.asearch, CallTypes.search],
"/v1/search": [CallTypes.asearch, CallTypes.search],
"/decisions": [CallTypes.adecisions, CallTypes.decisions],
"/v1/decisions": [CallTypes.adecisions, CallTypes.decisions],
# Batches
"/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
"/v1/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
@ -4048,6 +4054,8 @@ class LlmProviders(str, Enum):
OLLAMA_CHAT = "ollama_chat"
DEEPINFRA = "deepinfra"
PERPLEXITY = "perplexity"
TYPESAFE = "typesafe"
STRANDS_DECIDER = "strands_decider"
MISTRAL = "mistral"
MILVUS = "milvus"
GROQ = "groq"

3
litellm/utils.py
View file

@ -1218,6 +1218,9 @@ def function_setup(
if isinstance(search_query, list)
else search_query
)
elif call_type in (CallTypes.decisions.value, CallTypes.adecisions.value):
decisions_state: Final = args[1] if len(args) > 1 else kwargs.get("state", "")
messages = decisions_state if isinstance(decisions_state, str) else json.dumps(decisions_state)
elif call_type in (CallTypes.image_edit.value, CallTypes.aimage_edit.value):
messages = args[1] if len(args) > 1 else kwargs.get("prompt")
elif call_type in (CallTypes.ocr.value, CallTypes.aocr.value):

50
model_prices_and_context_window.json
View file

@ -15612,6 +15612,38 @@
"prompt_cache_min_tokens": 1024,
"source": "https://platform.claude.com/docs/en/about-claude/pricing"
},
"cloudflare/clef": {
"input_cost_per_token": 2.4e-07,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/clef-flash": {
"input_cost_per_token": 9e-08,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/cloudflare/clef": {
"input_cost_per_token": 2.4e-07,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/cloudflare/clef-flash": {
"input_cost_per_token": 9e-08,
"litellm_provider": "cloudflare",
"max_input_tokens": 65536,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://developers.cloudflare.com/workers-ai/platform/pricing/"
},
"cloudflare/@cf/meta/llama-2-7b-chat-fp16": {
"input_cost_per_token": 1.923e-06,
"litellm_provider": "cloudflare",
@ -44421,6 +44453,14 @@
"mode": "chat",
"output_cost_per_token": 2.8e-07
},
"perplexity/pplx-decider-v1-27b": {
"input_cost_per_token": 4e-08,
"litellm_provider": "perplexity",
"max_input_tokens": 262144,
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://docs.perplexity.ai/api-reference/decisions-post"
},
"perplexity/sonar": {
"input_cost_per_token": 1e-06,
"litellm_provider": "perplexity",
@ -72800,6 +72840,16 @@
"notes": "Self-hosted decision model; infrastructure costs are paid separately"
}
},
"strands_decider/strands-decider-2B-hobson-v19": {
"input_cost_per_token": 0.0,
"litellm_provider": "strands_decider",
"mode": "evaluation",
"output_cost_per_token": 0.0,
"source": "https://huggingface.co/StrandsAgents/strands-decider-2B-hobson-v19",
"metadata": {
"notes": "Self-hosted decision model; infrastructure costs are paid separately"
}
},
"typesafe/jev-1.13.0": {
"input_cost_per_token": 4.2e-08,
"litellm_provider": "typesafe",

14
provider_endpoints_support.json
View file

@ -2566,6 +2566,20 @@
"image_variations": true
}
},
"typesafe": {
"display_name": "TypeSafe (`typesafe`)",
"url": "https://docs.typesafe.ai/models",
"endpoints": {
"systemone": true
}
},
"strands_decider": {
"display_name": "Strands Decider (`strands_decider`)",
"url": "https://docs.litellm.ai/docs/providers",
"endpoints": {
"systemone": true
}
},
"tavily": {
"display_name": "Tavily (`tavily`)",
"url": "https://docs.litellm.ai/docs/search/tavily",

8
tests/integration/_support/process.py
View file

@ -50,12 +50,16 @@ def signal_group(group: int, action: int) -> None:
pass
def graceful_stop_seconds() -> float:
return max(30.0, float(os.environ.get("INTEGRATION_PROXY_READY_SECONDS", "70")))
def stop_root_process(process: subprocess.Popen[bytes]) -> bool:
if process.poll() is not None:
return True
process.terminate()
try:
process.wait(timeout=30)
process.wait(timeout=graceful_stop_seconds())
except subprocess.TimeoutExpired:
return False
return True
@ -183,6 +187,7 @@ def owned_proxy_process(
remove_environment: tuple[str, ...] = (),
workers: int = 1,
database_setup: tuple[str, ...] = DB_PUSH,
extra_arguments: tuple[str, ...] = (),
) -> Iterator[OwnedProxy]:
root: Final = Path(os.environ.get("INTEGRATION_PROXY_ROOT") or Path(__file__).resolve().parents[3])
environment: Final = {
@ -209,6 +214,7 @@ def owned_proxy_process(
"--num_workers",
str(workers),
*database_setup,
*extra_arguments,
)
launch: Final = _launch_until_bound(command, root, environment, output, _PORT_ATTEMPTS)
process: Final = launch.process

1
tests/integration/cost_calculation/cost_tracking_case.py
View file

@ -246,6 +246,7 @@ class CostTrackingTestCase(BaseModel):
"/v1/audio/speech",
"/v1/images/generations",
"/v1/images/edits",
"/v1/decisions",
]
| Annotated[str, Field(pattern=r"^/(gemini|anthropic|bedrock)/")]
) = "/v1/chat/completions"

48
tests/integration/cost_calculation/cost_tracking_cases.json
View file

@ -57,6 +57,13 @@
"search_context_size_high": 0.012
}
},
"perplexity/pplx-decider-v1-27b": {
"input_cost_per_token": 4e-08,
"litellm_provider": "perplexity",
"max_input_tokens": 262144,
"mode": "evaluation",
"output_cost_per_token": 0.0
},
"deepseek/deepseek-v4-chat": {
"litellm_provider": "deepseek",
"mode": "chat",
@ -30925,6 +30932,47 @@
"completion_tokens": 412
}
},
{
"name": "perplexity/pplx-decider-v1-27b-decisions",
"covers": "quota_management.spend_tracking.decisions_costs",
"model": "perplexity/pplx-decider-v1-27b",
"endpoint": "/v1/decisions",
"request": {
"model": "$MODEL",
"state": {
"source": "cost-tracking"
},
"questions": {
"is_defect": {
"type": "noul",
"instructions": "Is this a defect?"
}
}
},
"response": {
"content_type": "application/json",
"body": {
"model": "pplx-decider-v1-27b",
"answers": {
"is_defect": {
"type": "noul",
"noul": 0.9
}
},
"usage": {
"input_tokens": 367,
"output_tokens": 3
}
}
},
"expected": {
"spend": 1.468e-05,
"input_cost": 1.468e-05,
"output_cost": 0.0,
"prompt_tokens": 367,
"completion_tokens": 3
}
},
{
"name": "gpt-5.6-client_disconnect_mid_stream",
"covers": "quota_management.spend_tracking.scripted_wire.client_disconnect",

25
tests/integration/cost_calculation/test_cost_tracking.py
View file

@ -268,6 +268,31 @@ def test_case_bills_expected_cost(gateway: Gateway, case: CostTrackingTestCase)
assert row.spend == 0, f"{case.name}: failure spend was {row.spend}"
return
assert response.is_success, f"{case.name}: proxy returned {response.status_code}: {response.text[:400]}"
if case.endpoint == "/v1/decisions":
observed: Final = JSON_OBJECT.validate_json(
httpx.get(f"{gateway.upstream_url}/__observations", timeout=5, trust_env=False).content
)
decision_observations: Final = tuple(
value
for value in observed["requests"]
if isinstance(value, dict) and value.get("path") == f"/{scenario_id}/v1/decisions"
)
assert decision_observations == (
{
"path": f"/{scenario_id}/v1/decisions",
"authorization": "Bearer sk-scripted-provider",
"body": {
"model": "pplx-decider-v1-27b",
"state": {"source": "cost-tracking"},
"questions": {
"is_defect": {
"type": "noul",
"instructions": "Is this a defect?",
}
},
},
},
)
if case.response.content_type == "text/event-stream":
_assert_stream_has_no_error(response.text)
rows: Final = poll_rows(key, len(responses) + (prior_response_id is not None))

117
tests/integration/management/test_model_health_check.py
View file

@ -1,8 +1,41 @@
import os
import uuid
from typing import Final
import httpx
from integration._support.client import Gateway, object_value
from integration._support.client import Gateway, object_value, string_value
from integration._support.upstream import ScenarioHandle, delete_scenario, register_scenario
from integration.cost_calculation.cost_tracking_case import JsonResponse
from pydantic import JsonValue
_DECISIONS_PROBE_REPLY: Final = JsonResponse(
content_type="application/json",
body={
"model": "pplx-decider-v1-27b",
"answers": {"reachable": {"type": "noul", "noul": 1.0}},
"usage": {"input_tokens": 10, "output_tokens": 1},
},
)
_CONFIGURED_PROBE_REPLY: Final = JsonResponse(
content_type="application/json",
body={
"model": "jev-custom",
"answers": {"alive": {"type": "choice", "choice": "yes", "confidence": 0.9, "probabilities": {"yes": 0.9}}},
"usage": {"input_tokens": 10, "output_tokens": 1},
},
)
_STRANDS_PROBE_REPLY: Final = JsonResponse(
content_type="application/json",
body={
"model": "strands-decider-2B-hobson-v19",
"answers": {"reachable": {"type": "noul", "noul": 1.0}},
"usage": {"input_tokens": 10, "output_tokens": 1},
},
)
_CONFIGURED_STATE: Final[dict[str, JsonValue]] = {"ticket": "health probe"}
_CONFIGURED_QUESTIONS: Final[dict[str, JsonValue]] = {
"alive": {"type": "choice", "criteria": {"yes": "the service answers", "no": "the service is down"}}
}
def test_health_check_of_a_model_added_through_the_api_calls_its_upstream_and_reports_it_healthy(
@ -32,3 +65,85 @@ def test_health_check_of_a_model_added_through_the_api_calls_its_upstream_and_re
assert [request["body"]["model"] for request in upstream.get("/__observations").json()["requests"]] == [
provider_model
]
def _health_report(gateway: Gateway, model: str) -> dict[str, JsonValue]:
health: Final = gateway.request("GET", "/health", params={"model": model})
assert health.status_code == 200, health.text
return health.json()
def _probes_sent_to(gateway: Gateway, handle: ScenarioHandle) -> list[tuple[str, JsonValue]]:
with httpx.Client(base_url=gateway.upstream_url, timeout=5, trust_env=False) as upstream:
requests: Final = upstream.get("/__observations").json()["requests"]
return [
(string_value(request["path"]), request["body"])
for request in map(object_value, requests)
if string_value(request["path"]).startswith(f"/{handle.scenario_id}/")
]
def test_evaluation_mode_health_check_resolves_the_mode_from_the_cost_map_and_sends_the_default_probe(
gateway: Gateway,
) -> None:
with gateway.scenario() as scenario:
handle: Final = register_scenario(f"health-decisions-{uuid.uuid4().hex[:12]}", _DECISIONS_PROBE_REPLY)
scenario.cleanups.callback(delete_scenario, handle)
model: Final = scenario.model(model="perplexity/pplx-decider-v1-27b", api_base=handle.api_base())
report: Final = _health_report(gateway, model)
assert (report["healthy_count"], report["unhealthy_count"]) == (1, 0), report
assert _probes_sent_to(gateway, handle) == [
(
f"/{handle.scenario_id}/v1/decisions",
{
"model": "pplx-decider-v1-27b",
"state": os.environ.get("DEFAULT_HEALTH_CHECK_PROMPT", "test from litellm"),
"questions": {"reachable": {"type": "noul", "instructions": "Is the service reachable?"}},
},
)
]
def test_evaluation_mode_health_check_sends_the_configured_state_and_questions(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = register_scenario(f"health-decisions-{uuid.uuid4().hex[:12]}", _CONFIGURED_PROBE_REPLY)
scenario.cleanups.callback(delete_scenario, handle)
model: Final = scenario.model(
model="typesafe/jev-custom",
api_base=handle.api_base(),
model_info={
"mode": "evaluation",
"health_check_params": {"state": _CONFIGURED_STATE, "questions": _CONFIGURED_QUESTIONS},
},
)
report: Final = _health_report(gateway, model)
assert (report["healthy_count"], report["unhealthy_count"]) == (1, 0), report
assert _probes_sent_to(gateway, handle) == [
(
f"/{handle.scenario_id}/v1/systemone",
{"model": "jev-custom", "state": _CONFIGURED_STATE, "questions": _CONFIGURED_QUESTIONS},
)
]
def test_evaluation_mode_health_check_of_the_self_hosted_strands_model_resolves_the_mode_from_the_cost_map(
gateway: Gateway,
) -> None:
with gateway.scenario() as scenario:
handle: Final = register_scenario(f"health-decisions-{uuid.uuid4().hex[:12]}", _STRANDS_PROBE_REPLY)
scenario.cleanups.callback(delete_scenario, handle)
model: Final = scenario.model(
model="strands_decider/strands-decider-2B-hobson-v19", api_base=handle.api_base(), api_key=None
)
report: Final = _health_report(gateway, model)
assert (report["healthy_count"], report["unhealthy_count"]) == (1, 0), report
assert _probes_sent_to(gateway, handle) == [
(
f"/{handle.scenario_id}/v1/systemone",
{
"model": "strands-decider-2B-hobson-v19",
"state": os.environ.get("DEFAULT_HEALTH_CHECK_PROMPT", "test from litellm"),
"questions": {"reachable": {"type": "noul", "instructions": "Is the service reachable?"}},
},
)
]

262
tests/integration/providers/test_decisions_chaos.py Normal file
View file

@ -0,0 +1,262 @@
import asyncio
import json
import re
import signal
import threading
import uuid
from dataclasses import dataclass
from pathlib import Path
from queue import SimpleQueue
from types import MappingProxyType
from typing import Final
from urllib.parse import urlsplit
import httpx
import psutil
import pytest
import yaml
from integration._support.client import Gateway, eventually
from integration._support.database import read_rows
from integration._support.process import owned_proxy_process
from integration._support.wire import Reply, Request, Wire, wire_server
from pydantic import JsonValue, TypeAdapter
_BACKEND: Final = "pplx-decider-v1-27b"
_CONFIG_MODEL: Final = "decisions-chaos"
_API_KEY: Final = "synthetic-decisions-key"
_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])
_STARTED_WORKER: Final = re.compile(r"Started server process \[(\d+)\]")
_QUESTIONS: Final[dict[str, JsonValue]] = {"fine": {"type": "noul", "instructions": "Is the state fine?"}}
_ROUTES: Final = ("/v1/decisions", "/decisions")
@dataclass(frozen=True, slots=True)
class _Call:
route: str
marker: str
fail: bool
@dataclass(frozen=True, slots=True)
class _Served:
call: _Call
status: int
call_id: str
model_group: str
text: str
def _calls(count: int, *, fail: bool) -> tuple[_Call, ...]:
return tuple(
_Call(
route=_ROUTES[index % len(_ROUTES)],
marker=f"{'fail' if fail and index % 2 else 'ok'}-{uuid.uuid4().hex}",
fail=fail and index % 2 == 1,
)
for index in range(count)
)
def _marker_of(request: Request) -> str:
state: Final = _JSON_OBJECT.validate_json(request.body)["state"]
assert isinstance(state, str), request.body
return state
def _reply(request: Request) -> Reply:
marker: Final = _marker_of(request)
if marker.startswith("fail-"):
return Reply(status=500, body=json.dumps({"error": {"message": f"scripted outage {marker}"}}).encode())
answer: Final = {
"model": f"model-{marker}",
"answers": {"fine": {"type": "noul", "noul": 0.5}},
"usage": {"input_tokens": 12, "output_tokens": 1},
}
return Reply(body=json.dumps(answer).encode())
async def _send(client: httpx.AsyncClient, key: str, model: str | None, call: _Call) -> _Served:
body: Final[dict[str, JsonValue]] = {
**({"model": model} if model is not None else {}),
"state": call.marker,
"questions": _QUESTIONS,
"num_retries": 0,
}
response: Final = await client.post(call.route, json=body, headers={"Authorization": f"Bearer {key}"})
return _Served(
call=call,
status=response.status_code,
call_id=response.headers.get("x-litellm-call-id", ""),
model_group=response.headers.get("x-litellm-model-group", ""),
text=response.text,
)
async def _burst(
base_url: str, key: str, model: str | None, calls: tuple[_Call, ...], *, tolerate_transport_errors: bool = False
) -> tuple[_Served, ...]:
async with httpx.AsyncClient(base_url=base_url, timeout=60, trust_env=False) as client:
results: Final = await asyncio.gather(
*(_send(client, key, model, call) for call in calls), return_exceptions=tolerate_transport_errors
)
for result in results:
assert not isinstance(result, BaseException) or isinstance(result, httpx.TransportError), repr(result)
return tuple(result for result in results if isinstance(result, _Served))
def _assert_served_its_own(served: _Served) -> None:
assert served.call_id, served.text
if served.call.fail:
assert served.status == 500, (served.status, served.text)
assert served.call.marker in served.text, served.text
return
assert served.status == 200, (served.status, served.text)
assert _JSON_OBJECT.validate_json(served.text)["model"] == f"model-{served.call.marker}", served.text
def _statuses_by_call_id(call_ids: tuple[str, ...]) -> dict[str, JsonValue]:
placeholders: Final = ", ".join("%s" for _ in call_ids)
rows: Final = eventually(
lambda: read_rows(
f'SELECT request_id, status FROM "LiteLLM_SpendLogs" WHERE request_id IN ({placeholders})', call_ids
),
lambda found: len(found) >= len(call_ids),
seconds=70,
)
assert len(rows) == len(call_ids), rows
return {str(row["request_id"]): row["status"] for row in rows}
def _expected_statuses(served: tuple[_Served, ...]) -> dict[str, JsonValue]:
return {item.call_id: "failure" if item.call.fail else "success" for item in served}
def _health(gateway: Gateway, model: str) -> tuple[int, int]:
health: Final = gateway.request("GET", "/health", params={"model": model})
assert health.status_code in (200, 503), health.text
report: Final = health.json()
return (report["healthy_count"], report["unhealthy_count"])
def _chaos_config(wire: Wire, tmp_path: Path) -> Path:
config: Final = {
**_JSON_OBJECT.validate_python(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text())),
"model_list": [
{
"model_name": _CONFIG_MODEL,
"litellm_params": {"model": f"perplexity/{_BACKEND}", "api_base": wire.url, "api_key": _API_KEY},
}
],
}
path: Final = tmp_path / "decisions-chaos.yaml"
path.write_text(yaml.safe_dump(config))
return path
def _worker_startups(log: Path) -> tuple[tuple[int, ...], int]:
text: Final = log.read_text()
return tuple(int(pid) for pid in _STARTED_WORKER.findall(text)), text.count("Application startup complete.")
def _open_upstream_connections(pid: int, upstream: str) -> int:
port: Final = urlsplit(upstream).port
return sum(
1
for connection in psutil.Process(pid).net_connections(kind="tcp")
if connection.status == psutil.CONN_ESTABLISHED and connection.raddr and connection.raddr.port == port
)
@pytest.mark.timeout(180)
async def test_burst_over_both_routes_bills_each_call_once_with_its_own_status(gateway: Gateway) -> None:
calls: Final = _calls(30, fail=True)
with wire_server(_reply) as wire, gateway.scenario() as scenario:
model: Final = scenario.model(model=f"perplexity/{_BACKEND}", api_base=wire.url, api_key=_API_KEY)
served: Final = await _burst(str(gateway.client.base_url), gateway.key, model, calls)
assert len(served) == 30
for item in served:
_assert_served_its_own(item)
assert len({item.call_id for item in served}) == 30
assert _statuses_by_call_id(tuple(item.call_id for item in served)) == _expected_statuses(served)
received: Final = wire.drain()
assert sorted(_marker_of(request) for request in received) == sorted(call.marker for call in calls)
assert {request.target for request in received} == {"/v1/decisions"}, received
@pytest.mark.timeout(420)
async def test_worker_sigkill_mid_burst_leaves_the_sibling_serving_the_default_model(
gateway: Gateway, tmp_path: Path
) -> None:
calls: Final = _calls(20, fail=False)
release: Final = threading.Event()
held_markers: Final[SimpleQueue[str]] = SimpleQueue()
def held(request: Request) -> Reply:
held_markers.put(_marker_of(request))
assert release.wait(timeout=60), "The burst was never released"
return _reply(request)
with wire_server(held) as wire:
path: Final = _chaos_config(wire, tmp_path)
with owned_proxy_process(
gateway, tmp_path, {}, config=path, workers=2, extra_arguments=("--model", _CONFIG_MODEL)
) as owned:
candidate: Final = owned.gateway
base_url: Final = str(candidate.client.base_url)
workers, _ = eventually(
lambda: _worker_startups(owned.log),
lambda found: len(found[0]) == 2 and found[1] == 2,
seconds=120,
)
burst: Final = asyncio.create_task(
_burst(base_url, candidate.key, None, calls, tolerate_transport_errors=True)
)
await asyncio.to_thread(eventually, held_markers.qsize, lambda size: size == 20, 60)
held_by: Final = MappingProxyType({pid: _open_upstream_connections(pid, wire.url) for pid in workers})
assert sum(held_by.values()) == 20, held_by
victim_pid, survivor_pid = sorted(workers, key=held_by.__getitem__)
victim: Final = psutil.Process(victim_pid)
victim.suspend()
victim.send_signal(signal.SIGKILL)
release.set()
served: Final = await burst
assert len(served) == held_by[survivor_pid], (held_by, len(served))
follow_up: Final = _Call(route="/decisions", marker=f"ok-{uuid.uuid4().hex}", fail=False)
(answered,) = await _burst(base_url, candidate.key, None, (follow_up,))
await asyncio.to_thread(
eventually,
lambda: _worker_startups(owned.log),
lambda found: len(found[0]) == 3 and found[1] == 3,
180,
)
for item in (*served, answered):
_assert_served_its_own(item)
assert item.model_group == _CONFIG_MODEL, item.model_group
call_ids: Final = tuple(item.call_id for item in (*served, answered))
assert set(_statuses_by_call_id(call_ids).values()) == {"success"}
assert len({_marker_of(request) for request in wire.drain()}) == 21
@pytest.mark.timeout(180)
async def test_upstream_outage_fails_its_calls_and_recovery_on_the_same_port_restores_them(gateway: Gateway) -> None:
base_url: Final = str(gateway.client.base_url)
with gateway.scenario() as scenario:
with wire_server(_reply) as wire:
port: Final = urlsplit(wire.url).port
assert port is not None
model: Final = scenario.model(model=f"perplexity/{_BACKEND}", api_base=wire.url, api_key=_API_KEY)
before: Final = await _burst(base_url, gateway.key, model, _calls(5, fail=False))
assert _health(gateway, model) == (1, 0)
during: Final = await _burst(base_url, gateway.key, model, _calls(5, fail=False))
assert [item.status for item in during] == [500] * 5, [item.text for item in during]
assert _health(gateway, model) == (0, 1)
with wire_server(_reply, port=port):
after: Final = await _burst(base_url, gateway.key, model, _calls(5, fail=False))
assert _health(gateway, model) == (1, 0)
for item in (*before, *after):
_assert_served_its_own(item)
statuses: Final = _statuses_by_call_id(tuple(item.call_id for item in (*before, *during, *after)))
assert statuses == {
**{item.call_id: "success" for item in (*before, *after)},
**{item.call_id: "failure" for item in during},
}

456
tests/integration/providers/test_decisions_wire.py Normal file
View file

@ -0,0 +1,456 @@
import json
import math
import socket
import uuid
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Final
import httpx
import pytest
import yaml
from integration._support.client import Gateway, Scenario, eventually, object_value, string_value
from integration._support.database import read_rows
from integration._support.process import owned_proxy_process
from integration._support.upstream import ScenarioHandle, delete_scenario, register_scenario
from integration.cost_calculation.cost_tracking_case import JsonResponse
from pydantic import JsonValue, TypeAdapter
import litellm
_API_KEY: Final = "synthetic-decisions-key"
_ENV_KEY: Final = "synthetic-decisions-env-key"
_PASS_THROUGH_MODEL: Final = "gpt-6-luna"
_PASS_THROUGH_AUTHORIZATION: Final = "Bearer customer-held-upstream-key"
_PASS_THROUGH_NEIGHBOUR: Final = "decisions-beside-a-pass-through"
_JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])
_USAGE: Final[dict[str, JsonValue]] = {"input_tokens": 367, "output_tokens": 3}
_STATE: Final[dict[str, JsonValue]] = {"ticket": "The export job hangs at 99%", "component": "billing"}
_QUESTIONS: Final[dict[str, JsonValue]] = {
"defect": {"type": "noul", "instructions": "Is this a defect?"},
"severity": {"type": "choice", "criteria": {"low": "cosmetic", "high": "blocks users"}, "weight": 2},
"confidence": {"type": "score", "instructions": "How sure are you?", "criteria": ["unsure", "sure"]},
}
_ANSWERS: Final[dict[str, JsonValue]] = {
"defect": {"type": "noul", "noul": 0.93},
"severity": {"type": "choice", "choice": "high", "confidence": 0.8, "probabilities": {"low": 0.2, "high": 0.8}},
"confidence": {
"type": "score",
"score": 1.0,
"confidence": 0.7,
"legend": {"0": "unsure", "1": "sure"},
"probabilities": {"0": 0.3, "1": 0.7},
},
}
_CHAT_BODY: Final[dict[str, JsonValue]] = {"messages": [{"role": "user", "content": "hi"}]}
_CHAT_REPLY: Final[dict[str, JsonValue]] = {
"id": "chatcmpl-decisions-parity",
"object": "chat.completion",
"created": 1700000000,
"model": "pplx-decider-v1-27b",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},
}
_SPEND_QUERY: Final = (
"SELECT spend, status, call_type, model_group, custom_llm_provider, api_base, prompt_tokens, completion_tokens, "
'request_tags FROM "LiteLLM_SpendLogs" WHERE request_id = %s'
)
@dataclass(frozen=True, slots=True)
class _Provider:
name: str
model: str
path: str
body_model: str
api_key: str | None
wraps_result: bool
cost_map_key: str | None
_PROVIDERS: Final = (
_Provider(
"perplexity",
"perplexity/pplx-decider-v1-27b",
"/v1/decisions",
"pplx-decider-v1-27b",
_API_KEY,
False,
"perplexity/pplx-decider-v1-27b",
),
_Provider("typesafe", "typesafe/jev-1.13.0", "/v1/systemone", "jev-1.13.0", _API_KEY, False, "typesafe/jev-1.13.0"),
_Provider(
"openrouter",
"openrouter/typesafe/jev-1.13",
"/alpha/decisions",
"typesafe/jev-1.13",
_API_KEY,
False,
"openrouter/typesafe/jev-1.13",
),
_Provider(
"strands_decider", "strands_decider/systemone-decider", "/v1/systemone", "systemone-decider", None, False, None
),
_Provider(
"cloudflare",
"cloudflare/clef",
"/ai/run/@cf/cloudflare/clef",
"clef",
_API_KEY,
True,
"cloudflare/@cf/cloudflare/clef",
),
)
_PERPLEXITY: Final = _PROVIDERS[0]
_INVALID_BODIES: Final[tuple[tuple[str, dict[str, JsonValue]], ...]] = (
("missing questions", {"state": _STATE}),
("missing state", {"questions": _QUESTIONS}),
("numeric state", {"state": 5, "questions": _QUESTIONS}),
("empty questions", {"state": _STATE, "questions": {}}),
("noul without instructions or criteria", {"state": _STATE, "questions": {"q": {"type": "noul"}}}),
("choice without criteria", {"state": _STATE, "questions": {"q": {"type": "choice", "criteria": {}}}}),
(
"score with eleven criteria",
{"state": _STATE, "questions": {"q": {"type": "score", "criteria": [f"level-{index}" for index in range(11)]}}},
),
("unknown question type", {"state": _STATE, "questions": {"q": {"type": "ranking", "criteria": ["a"]}}}),
)
def _number(value: JsonValue) -> float:
assert isinstance(value, (int, float)) and not isinstance(value, bool), value
return float(value)
def _expected_spend(cost_map_key: str | None) -> float:
if cost_map_key is None:
return 0.0
prices: Final = object_value(json.loads(Path("model_prices_and_context_window.json").read_text())[cost_map_key])
return _number(_USAGE["input_tokens"]) * _number(prices["input_cost_per_token"]) + _number(
_USAGE["output_tokens"]
) * _number(prices["output_cost_per_token"])
def _answer_body(provider: _Provider) -> dict[str, JsonValue]:
answer: Final[dict[str, JsonValue]] = {"model": provider.body_model, "answers": _ANSWERS, "usage": _USAGE}
return {"result": answer, "success": True} if provider.wraps_result else answer
def _register(scenario: Scenario, body: dict[str, JsonValue], *, status: int = 200) -> ScenarioHandle:
handle: Final = register_scenario(
f"decisions-{uuid.uuid4().hex[:12]}", JsonResponse(content_type="application/json", body=body, status=status)
)
scenario.cleanups.callback(delete_scenario, handle)
return handle
def _deployment(scenario: Scenario, handle: ScenarioHandle, provider: _Provider) -> str:
return scenario.model(model=provider.model, api_base=handle.api_base(), api_key=provider.api_key)
def _decide(gateway: Gateway, model: str, *, key: str | None = None, **extra: JsonValue) -> httpx.Response:
return gateway.request(
"POST", "/v1/decisions", {"model": model, "state": _STATE, "questions": _QUESTIONS, **extra}, key=key
)
def _chat(gateway: Gateway, model: str, *, key: str | None = None, **extra: JsonValue) -> httpx.Response:
return gateway.request("POST", "/v1/chat/completions", {"model": model, **_CHAT_BODY, **extra}, key=key)
def _observed_requests(gateway: Gateway) -> tuple[dict[str, JsonValue], ...]:
with httpx.Client(base_url=gateway.upstream_url, timeout=5, trust_env=False) as upstream:
return tuple(map(object_value, upstream.get("/__observations").json()["requests"]))
def _calls_to(requests: Sequence[dict[str, JsonValue]], handle: ScenarioHandle) -> list[dict[str, JsonValue]]:
return [request for request in requests if string_value(request["path"]).startswith(f"/{handle.scenario_id}/")]
def _upstream_calls(gateway: Gateway, handle: ScenarioHandle) -> list[dict[str, JsonValue]]:
return _calls_to(_observed_requests(gateway), handle)
def _spend_row(call_id: str) -> dict[str, JsonValue]:
rows: Final = eventually(lambda: read_rows(_SPEND_QUERY, (call_id,)), lambda found: len(found) == 1, seconds=70)
return rows[0]
def _free_closed_port() -> int:
with socket.socket() as probe:
probe.bind(("127.0.0.1", 0))
return probe.getsockname()[1]
def _pass_through_config(directory: Path, pass_through_target: str, native_api_base: str) -> Path:
base: Final = _JSON_OBJECT.validate_python(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text()))
config: Final = {
**base,
"general_settings": {
**object_value(base["general_settings"]),
"pass_through_endpoints": [
{
"path": "/v1/decisions",
"target": pass_through_target,
"headers": {"Authorization": _PASS_THROUGH_AUTHORIZATION},
}
],
},
"model_list": [
{
"model_name": _PASS_THROUGH_NEIGHBOUR,
"litellm_params": {"model": _PERPLEXITY.model, "api_base": native_api_base, "api_key": _API_KEY},
}
],
}
path: Final = directory / "decisions-pass-through.yaml"
path.write_text(yaml.safe_dump(config))
return path
@pytest.mark.parametrize("provider", _PROVIDERS, ids=lambda provider: provider.name)
def test_each_provider_gets_its_own_path_key_and_body_and_is_billed_from_the_cost_map(
gateway: Gateway, provider: _Provider
) -> None:
expected_spend: Final = _expected_spend(provider.cost_map_key)
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(provider))
model: Final = _deployment(scenario, handle, provider)
response: Final = _decide(gateway, model)
assert response.status_code == 200, response.text
assert response.json() == {"model": provider.body_model, "answers": _ANSWERS, "usage": _USAGE}
assert response.headers["x-litellm-model-group"] == model
assert math.isclose(float(response.headers.get("x-litellm-response-cost", "0")), expected_spend, rel_tol=1e-9)
(call,) = _upstream_calls(gateway, handle)
assert call["path"] == f"/{handle.scenario_id}{provider.path}"
assert call["authorization"] == (f"Bearer {provider.api_key}" if provider.api_key else "")
assert call["body"] == {"model": provider.body_model, "state": _STATE, "questions": _QUESTIONS}
row: Final = _spend_row(response.headers["x-litellm-call-id"])
assert (
row["status"],
row["call_type"],
row["custom_llm_provider"],
row["model_group"],
row["api_base"],
row["prompt_tokens"],
row["completion_tokens"],
) == ("success", "adecisions", provider.name, model, f"{handle.api_base()}{provider.path}", 367, 3)
assert math.isclose(_number(row["spend"]), expected_spend, rel_tol=1e-9), row
def test_repeated_identical_requests_each_reach_the_upstream_and_are_each_billed(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = _deployment(scenario, handle, _PERPLEXITY)
responses: Final = tuple(_decide(gateway, model) for _ in range(2))
assert [response.status_code for response in responses] == [200, 200], [r.text for r in responses]
call_ids: Final = tuple(response.headers["x-litellm-call-id"] for response in responses)
assert len(set(call_ids)) == 2, call_ids
assert len(_upstream_calls(gateway, handle)) == 2
for call_id in call_ids:
assert _spend_row(call_id)["status"] == "success"
async def test_sdk_sync_and_async_clients_send_the_same_request(gateway: Gateway) -> None:
provider: Final = _PROVIDERS[1]
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(provider))
synchronous: Final = litellm.decisions(
model=provider.model, state=_STATE, questions=_QUESTIONS, api_base=handle.api_base(), api_key=_API_KEY
)
asynchronous: Final = await litellm.adecisions(
model=provider.model, state=_STATE, questions=_QUESTIONS, api_base=handle.api_base(), api_key=_API_KEY
)
for response in (synchronous, asynchronous):
assert response.model_dump(mode="json") == {
"model": provider.body_model,
"answers": _ANSWERS,
"usage": _USAGE,
}
calls: Final = _upstream_calls(gateway, handle)
assert len(calls) == 2, calls
for call in calls:
assert call["path"] == f"/{handle.scenario_id}{provider.path}"
assert call["authorization"] == f"Bearer {_API_KEY}"
assert call["body"] == {"model": provider.body_model, "state": _STATE, "questions": _QUESTIONS}
def test_gateway_only_fields_stay_at_the_gateway_and_tags_reach_the_spend_log(gateway: Gateway) -> None:
tag: Final = f"decisions-audit-{uuid.uuid4().hex[:8]}"
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = _deployment(scenario, handle, _PERPLEXITY)
response: Final = _decide(
gateway, model, user="auditor", num_retries=0, temperature=0.2, metadata={"tags": [tag]}
)
assert response.status_code == 200, response.text
(call,) = _upstream_calls(gateway, handle)
assert call["body"] == {"model": _PERPLEXITY.body_model, "state": _STATE, "questions": _QUESTIONS}
row: Final = _spend_row(response.headers["x-litellm-call-id"])
tags: Final = row["request_tags"]
assert isinstance(tags, list) and tag in tags, row
def test_invalid_bodies_are_refused_at_the_gateway_without_an_upstream_call(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = _deployment(scenario, handle, _PERPLEXITY)
for label, body in _INVALID_BODIES:
response: Final = gateway.request("POST", "/v1/decisions", {"model": model, **body})
assert response.status_code == 400, (label, response.text)
assert "Invalid Decisions request" in response.text, (label, response.text)
assert _upstream_calls(gateway, handle) == []
def test_unknown_model_is_refused_like_chat(gateway: Gateway) -> None:
model: Final = f"missing-{uuid.uuid4().hex}"
decisions: Final = _decide(gateway, model)
chat: Final = _chat(gateway, model)
assert 400 <= decisions.status_code < 500, decisions.text
assert decisions.status_code == chat.status_code, (decisions.text, chat.text)
def test_key_checks_match_chat(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = _deployment(scenario, handle, _PERPLEXITY)
anonymous: Final = gateway.client.post(
"/v1/decisions", json={"model": model, "state": _STATE, "questions": _QUESTIONS}
)
assert anonymous.status_code == 401, anonymous.text
restricted: Final = scenario.key(models=[f"other-{uuid.uuid4().hex}"])
refused: Final = _decide(gateway, model, key=restricted)
assert 400 <= refused.status_code < 500, refused.text
assert refused.status_code == _chat(gateway, model, key=restricted).status_code, refused.text
assert _upstream_calls(gateway, handle) == []
spender: Final = scenario.key(max_budget=1e-06)
first: Final = _decide(gateway, model, key=spender)
assert first.status_code == 200, first.text
blocked: Final = eventually(
lambda: _decide(gateway, model, key=spender), lambda response: response.status_code != 200, seconds=70
)
assert 400 <= blocked.status_code < 500, blocked.text
assert blocked.status_code == _chat(gateway, model, key=spender).status_code, blocked.text
def test_request_body_api_base_is_refused_like_chat_without_an_upstream_call(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = _deployment(scenario, handle, _PERPLEXITY)
decisions: Final = _decide(gateway, model, api_base=f"http://127.0.0.1:{_free_closed_port()}")
chat: Final = _chat(gateway, model, api_base=f"http://127.0.0.1:{_free_closed_port()}")
assert 400 <= decisions.status_code < 500, decisions.text
assert decisions.status_code == chat.status_code, (decisions.text, chat.text)
assert _upstream_calls(gateway, handle) == []
def test_a_deployment_without_a_key_sends_the_provider_env_key_to_its_configured_api_base(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
model: Final = scenario.model(model=_PERPLEXITY.model, api_base=handle.api_base(), api_key=None)
response: Final = _decide(gateway, model)
assert response.status_code == 200, response.text
(call,) = _upstream_calls(gateway, handle)
assert (call["path"], call["authorization"]) == (f"/{handle.scenario_id}/v1/decisions", f"Bearer {_ENV_KEY}")
def test_a_deployment_opted_into_client_api_base_sends_decisions_and_chat_to_the_body_api_base(
gateway: Gateway,
) -> None:
with gateway.scenario() as scenario:
configured: Final = _register(scenario, _answer_body(_PERPLEXITY))
decisions_target: Final = _register(scenario, _answer_body(_PERPLEXITY))
chat_target: Final = _register(scenario, _CHAT_REPLY)
model: Final = scenario.model(
model=_PERPLEXITY.model,
api_base=configured.api_base(),
api_key=_API_KEY,
configurable_clientside_auth_params=["api_base"],
)
decisions: Final = _decide(gateway, model, api_base=decisions_target.api_base())
chat: Final = _chat(gateway, model, api_base=chat_target.api_base())
assert decisions.status_code == 200, decisions.text
assert chat.status_code == 200, chat.text
observed: Final = _observed_requests(gateway)
assert [call["path"] for call in _calls_to(observed, decisions_target)] == [
f"/{decisions_target.scenario_id}/v1/decisions"
]
assert [call["path"] for call in _calls_to(observed, chat_target)] == [
f"/{chat_target.scenario_id}/chat/completions"
]
assert _calls_to(observed, configured) == []
def test_a_config_pass_through_at_v1_decisions_keeps_answering_and_the_native_api_serves_decisions(
gateway: Gateway, tmp_path: Path
) -> None:
with gateway.scenario() as scenario:
pass_through_target: Final = _register(
scenario, {"model": _PASS_THROUGH_MODEL, "answers": _ANSWERS, "usage": _USAGE}
)
native_target: Final = _register(scenario, _answer_body(_PERPLEXITY))
config: Final = _pass_through_config(
tmp_path, f"{pass_through_target.api_base()}/v1/decisions", native_target.api_base()
)
with owned_proxy_process(gateway, tmp_path, {}, config=config) as owned:
through: Final = _decide(owned.gateway, _PASS_THROUGH_MODEL)
native: Final = owned.gateway.request(
"POST", "/decisions", {"model": _PASS_THROUGH_NEIGHBOUR, "state": _STATE, "questions": _QUESTIONS}
)
assert through.status_code == 200, through.text
assert through.json() == {"model": _PASS_THROUGH_MODEL, "answers": _ANSWERS, "usage": _USAGE}
observed: Final = _observed_requests(gateway)
(forwarded,) = _calls_to(observed, pass_through_target)
assert (forwarded["path"], forwarded["authorization"], object_value(forwarded["body"])["model"]) == (
f"/{pass_through_target.scenario_id}/v1/decisions",
_PASS_THROUGH_AUTHORIZATION,
_PASS_THROUGH_MODEL,
)
assert native.status_code == 200, native.text
assert [call["path"] for call in _calls_to(observed, native_target)] == [
f"/{native_target.scenario_id}/v1/decisions"
]
@pytest.mark.parametrize("status", (401, 429, 500))
def test_upstream_errors_keep_their_status_and_log_an_unbilled_failure(gateway: Gateway, status: int) -> None:
marker: Final = f"scripted-{status}-{uuid.uuid4().hex[:8]}"
with gateway.scenario() as scenario:
handle: Final = _register(scenario, {"error": {"message": marker}}, status=status)
model: Final = _deployment(scenario, handle, _PERPLEXITY)
response: Final = _decide(gateway, model, num_retries=0)
assert response.status_code == status, response.text
assert marker in response.text
row: Final = _spend_row(response.headers["x-litellm-call-id"])
assert (row["status"], row["call_type"], row["model_group"], _number(row["spend"])) == (
"failure",
"adecisions",
model,
0.0,
)
def test_upstream_success_without_answers_is_a_gateway_side_server_error(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, {"model": _PERPLEXITY.body_model, "usage": _USAGE})
model: Final = _deployment(scenario, handle, _PERPLEXITY)
response: Final = _decide(gateway, model, num_retries=0)
assert 500 <= response.status_code < 600, response.text
assert "answers" in response.text
assert _spend_row(response.headers["x-litellm-call-id"])["status"] == "failure"
def test_unreachable_upstream_fails_only_its_own_deployment(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
handle: Final = _register(scenario, _answer_body(_PERPLEXITY))
healthy: Final = _deployment(scenario, handle, _PERPLEXITY)
dead: Final = scenario.model(
model=_PERPLEXITY.model, api_base=f"http://127.0.0.1:{_free_closed_port()}", api_key=_API_KEY
)
failed: Final = _decide(gateway, dead, num_retries=0)
assert 500 <= failed.status_code < 600, failed.text
assert _spend_row(failed.headers["x-litellm-call-id"])["status"] == "failure"
served: Final = _decide(gateway, healthy)
assert served.status_code == 200, served.text
assert len(_upstream_calls(gateway, handle)) == 1

2
tests/integration/proxy_config.yaml
View file

@ -23,3 +23,5 @@ vector_store_registry:
api_base: os.environ/INTEGRATION_UPSTREAM_URL
api_key: integration-provider-key
vector_store_description: declared in tests/integration/proxy_config.yaml
environment_variables:
PERPLEXITYAI_API_KEY: synthetic-decisions-env-key

0
tests/unit/decisions/__init__.py Normal file
View file

592
tests/unit/decisions/test_main.py Normal file
View file

@ -0,0 +1,592 @@
from __future__ import annotations
import asyncio
import json
from collections.abc import Mapping
from types import MappingProxyType
from typing import Final
import pytest
import respx
import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER
from litellm.types.decisions import (
ChoiceAnswer,
DecisionsResponse,
DecisionsUsage,
NoulAnswer,
ScoreAnswer,
)
_QUESTIONS: Final[Mapping[str, object]] = MappingProxyType(
{
"is_defect": {"type": "noul", "instructions": "Is this a defect?", "provider_field": "kept"},
"sentiment": {"type": "choice", "criteria": {"positive": None, "negative": "unhappy"}},
"severity": {"type": "score", "criteria": ["none", "low", "high"]},
}
)
_INPUT_TOKENS: Final[int] = 367
_OUTPUT_TOKENS: Final[int] = 3
_RESPONSE: Final[Mapping[str, object]] = {
"model": "jev-1.13",
"answers": {
"is_defect": {"type": "noul", "noul": 0.9},
"sentiment": {
"type": "choice",
"choice": "positive",
"confidence": 0.8,
"probabilities": {"positive": 0.8, "negative": 0.2},
},
"severity": {
"type": "score",
"score": 1,
"confidence": 0.7,
"legend": {"0": "none", "1": "low", "2": "high"},
"probabilities": {"0": 0.1, "1": 0.8, "2": 0.1},
},
},
"usage": {"input_tokens": _INPUT_TOKENS, "output_tokens": _OUTPUT_TOKENS},
}
_STRANDS_RESPONSE: Final[Mapping[str, object]] = {
"model": "strands-decider-2B-hobson-v19",
"answers": {
"severity": {
"type": "score",
"score": 1,
"confidence": 0.7,
"legend": {"0": "none", "1": "low", "2": "high"},
"probabilities": {"0": 0.1, "1": 0.8, "2": 0.1},
}
},
"usage": {"input_tokens": 216, "output_tokens": 3},
"latency_ms": 3722.17,
}
_PROVIDERS: Final[tuple[tuple[str, str, str, str], ...]] = (
(
"perplexity",
"perplexity/pplx-decider-v1-27b",
"https://api.perplexity.ai/v1/decisions",
"pplx-decider-v1-27b",
),
("typesafe", "typesafe/jev-1.13", "https://api.typesafe.ai/v1/systemone", "jev-1.13"),
(
"openrouter",
"openrouter/typesafe/jev-1.13",
"https://openrouter.ai/api/alpha/decisions",
"typesafe/jev-1.13",
),
)
class _RecordingLogger(CustomLogger):
def __init__(self) -> None:
super().__init__()
self.standard_logging_object: Mapping[str, object] | None = None
async def async_log_success_event(
self,
kwargs: Mapping[str, object],
response_obj: object,
start_time: object,
end_time: object,
) -> None:
standard_logging_object: Final = kwargs.get("standard_logging_object")
if isinstance(standard_logging_object, dict):
self.standard_logging_object = standard_logging_object
async def _drain_logging_worker() -> None:
await asyncio.sleep(0)
GLOBAL_LOGGING_WORKER.start()
await asyncio.wait_for(GLOBAL_LOGGING_WORKER.flush(), timeout=10.0)
@pytest.fixture(autouse=True)
def _httpx_transport(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
@pytest.mark.asyncio
@pytest.mark.parametrize(("provider", "model", "url", "upstream_model"), _PROVIDERS)
async def test_adecisions_sends_the_provider_wire_contract(
provider: str,
model: str,
url: str,
upstream_model: str,
respx_mock: respx.MockRouter,
) -> None:
route: Final = respx_mock.post(url).respond(json=_RESPONSE)
response: Final = await litellm.adecisions(
model=model,
state={"source": "unit-test"},
questions=_QUESTIONS,
api_key="caller-key",
extra_headers={
"x-request-tag": "decisions-test",
"AUTHORIZATION": "attacker-key",
"Content-Type": "text/plain",
},
internal_kwarg="must-not-leak",
)
assert route.called
assert len(respx_mock.calls) == 1
request: Final = respx_mock.calls[0].request
assert request.headers["authorization"] == "Bearer caller-key"
assert request.headers["content-type"] == "application/json"
assert request.headers["x-request-tag"] == "decisions-test"
assert json.loads(request.content) == {
"model": upstream_model,
"state": {"source": "unit-test"},
"questions": {
"is_defect": {
"type": "noul",
"instructions": "Is this a defect?",
"provider_field": "kept",
},
"sentiment": {"type": "choice", "criteria": {"positive": None, "negative": "unhappy"}},
"severity": {"type": "score", "criteria": ["none", "low", "high"]},
},
}
assert isinstance(response.answers["is_defect"], NoulAnswer)
assert isinstance(response.answers["sentiment"], ChoiceAnswer)
assert isinstance(response.answers["severity"], ScoreAnswer)
assert response._hidden_params["custom_llm_provider"] == provider
@pytest.mark.asyncio
async def test_router_dispatches_typesafe_decisions_without_api_base(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("TYPESAFE_API_KEY", raising=False)
monkeypatch.delenv("TYPESAFE_API_BASE", raising=False)
provider_resolution: Final = litellm.get_llm_provider("typesafe/jev-latest")
assert provider_resolution[:2] == ("jev-latest", "typesafe")
router: Final = litellm.Router(
model_list=[
{
"model_name": "jev",
"litellm_params": {
"model": "typesafe/jev-latest",
"api_key": "k",
},
}
]
)
upstream: Final = respx_mock.post("https://api.typesafe.ai/v1/systemone").respond(json=_RESPONSE)
response: Final = await router.adecisions(
model="jev",
state="router-test",
questions={
"sentiment": {
"type": "choice",
"criteria": {"positive": None, "negative": "unhappy"},
}
},
)
assert upstream.called
assert len(respx_mock.calls) == 1
assert json.loads(respx_mock.calls[0].request.content) == {
"model": "jev-latest",
"state": "router-test",
"questions": {
"sentiment": {
"type": "choice",
"criteria": {"positive": None, "negative": "unhappy"},
}
},
}
assert respx_mock.calls[0].request.headers["authorization"] == "Bearer k"
assert isinstance(response.answers["sentiment"], ChoiceAnswer)
assert response.answers["sentiment"].choice == "positive"
def test_decisions_uses_the_same_wire_contract_for_sync_calls(respx_mock: respx.MockRouter) -> None:
route: Final = respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(json=_RESPONSE)
response: Final = litellm.decisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
)
assert route.called
assert response.model == "jev-1.13"
def test_openrouter_response_keeps_provider_fields(respx_mock: respx.MockRouter) -> None:
payload: Final = {
**_RESPONSE,
"id": "decision-1",
"provider": "typesafe",
"usage": {**_RESPONSE["usage"], "cost": 0.25},
}
respx_mock.post("https://openrouter.ai/api/alpha/decisions").respond(json=payload)
response: Final = litellm.decisions(
model="openrouter/typesafe/jev-1.13",
state="review",
questions=_QUESTIONS,
api_key="caller-key",
)
assert response.model_extra["id"] == "decision-1"
assert response.model_extra["provider"] == "typesafe"
assert response.usage is not None
assert response.usage.model_extra["cost"] == 0.25
def test_decisions_cost_uses_litellm_token_pricing() -> None:
response: Final = DecisionsResponse(
model="pplx-decider-v1-27b",
answers={},
usage=DecisionsUsage(input_tokens=_INPUT_TOKENS, output_tokens=_OUTPUT_TOKENS),
)
response._hidden_params = {
"model": "perplexity/pplx-decider-v1-27b",
"custom_llm_provider": "perplexity",
}
cost: Final = litellm.completion_cost(completion_response=response)
perplexity_cost: Final = litellm.model_cost["perplexity/pplx-decider-v1-27b"]
expected_cost: Final = _INPUT_TOKENS * float(perplexity_cost["input_cost_per_token"]) + _OUTPUT_TOKENS * float(
perplexity_cost["output_cost_per_token"]
)
assert expected_cost > 0
assert cost == pytest.approx(expected_cost)
@pytest.mark.asyncio
async def test_decisions_cost_is_in_standard_logging_object(respx_mock: respx.MockRouter) -> None:
respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(json=_RESPONSE)
recording_logger: Final = _RecordingLogger()
original_callbacks: Final = litellm.callbacks
litellm.callbacks = [recording_logger]
try:
await litellm.adecisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
)
await _drain_logging_worker()
finally:
litellm.callbacks = original_callbacks
assert recording_logger.standard_logging_object is not None
perplexity_cost: Final = litellm.model_cost["perplexity/pplx-decider-v1-27b"]
expected_cost: Final = _INPUT_TOKENS * float(perplexity_cost["input_cost_per_token"]) + _OUTPUT_TOKENS * float(
perplexity_cost["output_cost_per_token"]
)
assert expected_cost > 0
assert recording_logger.standard_logging_object["response_cost"] == pytest.approx(expected_cost)
assert recording_logger.standard_logging_object["prompt_tokens"] == _INPUT_TOKENS
assert recording_logger.standard_logging_object["completion_tokens"] == _OUTPUT_TOKENS
@pytest.mark.asyncio
async def test_unknown_provider_is_rejected_before_http(respx_mock: respx.MockRouter) -> None:
with pytest.raises(litellm.BadRequestError, match="Supported providers"):
await litellm.adecisions(
model="unknown/jev-1.13",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
)
assert len(respx_mock.calls) == 0
@pytest.mark.asyncio
async def test_empty_custom_provider_is_rejected_before_http(respx_mock: respx.MockRouter) -> None:
with pytest.raises(litellm.BadRequestError, match="Supported providers"):
await litellm.adecisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
custom_llm_provider="",
)
assert len(respx_mock.calls) == 0
@pytest.mark.asyncio
async def test_invalid_question_is_rejected_before_http(respx_mock: respx.MockRouter) -> None:
with pytest.raises(litellm.BadRequestError, match="Invalid Decisions request"):
await litellm.adecisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"sentiment": {"type": "choice"}},
api_key="caller-key",
)
assert len(respx_mock.calls) == 0
def test_upstream_bad_request_maps_to_litellm_error(respx_mock: respx.MockRouter) -> None:
respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(
status_code=400,
json={"error": {"message": "invalid decision"}},
)
with pytest.raises(litellm.BadRequestError):
litellm.decisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
)
def test_server_key_is_sent_to_an_explicit_api_base(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("PERPLEXITYAI_API_KEY", "server-key")
monkeypatch.delenv("PERPLEXITY_API_KEY", raising=False)
route: Final = respx_mock.post("https://egress.example/perplexity/v1/decisions").respond(json=_RESPONSE)
litellm.decisions(
model="perplexity/pplx-decider-v1-27b",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_base="https://egress.example/perplexity",
)
assert route.call_count == 1
assert route.calls[0].request.headers["authorization"] == "Bearer server-key"
@pytest.mark.asyncio
@pytest.mark.parametrize("model", ("cloudflare/clef", "cloudflare/@cf/cloudflare/clef"))
@pytest.mark.parametrize("wrapped", (False, True))
async def test_cloudflare_clef_resolves_model_and_response_envelope(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
model: str,
wrapped: bool,
) -> None:
monkeypatch.setenv("CLOUDFLARE_ACCOUNT_ID", "acct")
monkeypatch.setenv("CLOUDFLARE_API_KEY", "cloudflare-key")
monkeypatch.delenv("CLOUDFLARE_API_BASE", raising=False)
response_body: Final[Mapping[str, object]] = (
{"result": _RESPONSE, "success": True, "errors": [], "messages": []} if wrapped else _RESPONSE
)
route: Final = respx_mock.post(
"https://api.cloudflare.com/client/v4/accounts/acct/ai/run/@cf/cloudflare/clef"
).respond(json=response_body)
response: Final = await litellm.adecisions(
model=model,
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
)
assert route.called
request: Final = respx_mock.calls[0].request
assert request.headers["authorization"] == "Bearer cloudflare-key"
assert json.loads(request.content) == {
"model": "clef",
"state": "review",
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
}
assert response.answers == DecisionsResponse.model_validate(_RESPONSE).answers
assert response._hidden_params["model"] == "cloudflare/@cf/cloudflare/clef"
@pytest.mark.asyncio
async def test_cloudflare_clef_flash_uses_flash_endpoint_and_request_model(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("CLOUDFLARE_ACCOUNT_ID", "acct")
monkeypatch.setenv("CLOUDFLARE_API_KEY", "cloudflare-key")
monkeypatch.delenv("CLOUDFLARE_API_BASE", raising=False)
route: Final = respx_mock.post(
"https://api.cloudflare.com/client/v4/accounts/acct/ai/run/@cf/cloudflare/clef-flash"
).respond(json=_RESPONSE)
await litellm.adecisions(
model="cloudflare/clef-flash",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
)
assert route.called
assert json.loads(respx_mock.calls[0].request.content)["model"] == "clef-flash"
@pytest.mark.asyncio
async def test_cloudflare_api_base_from_env_uses_workers_ai_run_path(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("CLOUDFLARE_API_BASE", "https://api.cloudflare.com/client/v4/accounts/acct/ai/v1")
monkeypatch.setenv("CLOUDFLARE_API_KEY", "cloudflare-key")
monkeypatch.delenv("CLOUDFLARE_ACCOUNT_ID", raising=False)
route: Final = respx_mock.post(
"https://api.cloudflare.com/client/v4/accounts/acct/ai/run/@cf/cloudflare/clef"
).respond(json=_RESPONSE)
await litellm.adecisions(
model="cloudflare/clef",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
)
assert route.called
@pytest.mark.asyncio
async def test_cloudflare_requires_account_id_or_api_base_before_http(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("CLOUDFLARE_ACCOUNT_ID", raising=False)
monkeypatch.delenv("CLOUDFLARE_API_BASE", raising=False)
monkeypatch.setenv("CLOUDFLARE_API_KEY", "cloudflare-key")
with pytest.raises(litellm.BadRequestError, match="Missing CLOUDFLARE_ACCOUNT_ID - set CLOUDFLARE_ACCOUNT_ID"):
await litellm.adecisions(
model="cloudflare/clef",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
)
assert len(respx_mock.calls) == 0
@pytest.mark.asyncio
async def test_cloudflare_clef_cost_uses_the_model_cost_map(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("CLOUDFLARE_ACCOUNT_ID", "acct")
monkeypatch.setenv("CLOUDFLARE_API_KEY", "cloudflare-key")
monkeypatch.delenv("CLOUDFLARE_API_BASE", raising=False)
respx_mock.post("https://api.cloudflare.com/client/v4/accounts/acct/ai/run/@cf/cloudflare/clef").respond(
json=_RESPONSE
)
response: Final = await litellm.adecisions(
model="cloudflare/clef",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
)
cost: Final = litellm.completion_cost(completion_response=response)
clef_cost: Final = litellm.model_cost["cloudflare/@cf/cloudflare/clef"]
expected_cost: Final = _INPUT_TOKENS * float(clef_cost["input_cost_per_token"]) + _OUTPUT_TOKENS * float(
clef_cost["output_cost_per_token"]
)
assert expected_cost > 0
assert cost == pytest.approx(expected_cost)
@pytest.mark.asyncio
async def test_strands_decider_requires_api_base_before_http(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("STRANDS_DECIDER_API_BASE", raising=False)
monkeypatch.delenv("STRANDS_DECIDER_API_KEY", raising=False)
with pytest.raises(litellm.BadRequestError, match="api_base is required"):
await litellm.adecisions(
model="strands_decider/strands-decider-2B-hobson-v19",
state="review",
questions={"severity": {"type": "score", "criteria": ["none", "low", "high"]}},
)
assert len(respx_mock.calls) == 0
@pytest.mark.asyncio
async def test_strands_decider_without_key_preserves_response_extras(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("STRANDS_DECIDER_API_BASE", raising=False)
monkeypatch.delenv("STRANDS_DECIDER_API_KEY", raising=False)
route: Final = respx_mock.post("https://strands.example/v1/systemone").respond(json=_STRANDS_RESPONSE)
response: Final = await litellm.adecisions(
model="strands_decider/strands-decider-2B-hobson-v19",
state="review",
questions={"severity": {"type": "score", "criteria": ["none", "low", "high"]}},
api_base="https://strands.example",
)
assert route.called
assert "authorization" not in respx_mock.calls[0].request.headers
assert response.model_extra["latency_ms"] == _STRANDS_RESPONSE["latency_ms"]
severity: Final = response.answers["severity"]
assert isinstance(severity, ScoreAnswer)
assert severity.legend == {"0": "none", "1": "low", "2": "high"}
@pytest.mark.asyncio
async def test_strands_decider_uses_key_from_matching_environment_base(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("STRANDS_DECIDER_API_BASE", "https://strands.example")
monkeypatch.setenv("STRANDS_DECIDER_API_KEY", "strands-key")
route: Final = respx_mock.post("https://strands.example/v1/systemone").respond(json=_STRANDS_RESPONSE)
await litellm.adecisions(
model="strands_decider/strands-decider-2B-hobson-v19",
state="review",
questions={"severity": {"type": "score", "criteria": ["none", "low", "high"]}},
api_base="https://strands.example",
)
assert route.called
assert respx_mock.calls[0].request.headers["authorization"] == "Bearer strands-key"
@pytest.mark.asyncio
async def test_strands_decider_provider_resolution_and_router_dispatch(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("STRANDS_DECIDER_API_BASE", raising=False)
monkeypatch.delenv("STRANDS_DECIDER_API_KEY", raising=False)
provider_resolution: Final = litellm.get_llm_provider("strands_decider/strands-decider-2B-hobson-v19")
router: Final = litellm.Router(
model_list=[
{
"model_name": "strands",
"litellm_params": {
"model": "strands_decider/strands-decider-2B-hobson-v19",
"api_base": "https://strands.example",
},
}
]
)
route: Final = respx_mock.post("https://strands.example/v1/systemone").respond(json=_STRANDS_RESPONSE)
response: Final = await router.adecisions(
model="strands",
state="review",
questions={"severity": {"type": "score", "criteria": ["none", "low", "high"]}},
)
assert provider_resolution[:2] == ("strands-decider-2B-hobson-v19", "strands_decider")
assert route.called
assert response.model == _STRANDS_RESPONSE["model"]

99
tests/unit/litellm_core_utils/test_health_check_helpers.py
View file

@ -1,5 +1,6 @@
"""Test health check helper functions"""
import json
import socket
import struct
import zlib
@ -8,6 +9,7 @@ from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
import respx
import litellm
from litellm.constants import LITTELM_INTERNAL_HEALTH_SERVICE_ACCOUNT_NAME
@ -548,3 +550,100 @@ def test_ocr_health_check_document_raises_without_the_extension():
_ocr_health_check_document(model="mistral/mistral-ocr-latest", custom_llm_provider="mistral")
finally:
NATIVE_OCR_HEALTH_CHECK_DOCUMENT.reset()
@pytest.mark.parametrize(
("model", "upstream_url"),
(
("perplexity/pplx-decider-v1-27b", "https://api.perplexity.ai/v1/decisions"),
("cloudflare/clef", "https://api.cloudflare.com/client/v4/accounts/acct-1/ai/run/@cf/cloudflare/clef"),
),
)
async def test_ahealth_check_probes_evaluation_models_through_the_decisions_api(
model: str,
upstream_url: str,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("CLOUDFLARE_ACCOUNT_ID", "acct-1")
monkeypatch.delenv("CLOUDFLARE_API_BASE", raising=False)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
upstream: Final = respx_mock.post(upstream_url).respond(
json={
"model": model,
"answers": {"reachable": {"type": "noul", "noul": 1.0}},
"usage": {"input_tokens": 12, "output_tokens": 1},
}
)
result: Final = await ahealth_check({"model": model, "api_key": "sk-test"}, mode=None)
assert "error" not in result, result
assert upstream.called
sent: Final = json.loads(upstream.calls[0].request.content)
assert sent["state"] == "health check"
assert sent["questions"]["reachable"]["type"] == "noul"
@pytest.mark.asyncio
async def test_ahealth_check_evaluation_uses_configured_probe_state_and_questions(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
upstream: Final = respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(
json={
"model": "perplexity/pplx-decider-v1-27b",
"answers": {"ok": {"type": "noul", "noul": 1.0}},
"usage": {"input_tokens": 12, "output_tokens": 1},
}
)
result: Final = await ahealth_check(
{
"model": "perplexity/pplx-decider-v1-27b",
"api_key": "sk-test",
"state": "custom probe",
"questions": {"ok": {"type": "noul", "instructions": "Is it ok?"}},
},
mode=None,
)
assert "error" not in result, result
assert upstream.called
sent: Final = json.loads(upstream.calls[0].request.content)
assert sent["state"] == "custom probe"
assert set(sent["questions"]) == {"ok"}
@pytest.mark.asyncio
async def test_ahealth_check_probes_strands_through_decisions_without_mode(
local_model_cost_map: None,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("STRANDS_DECIDER_API_KEY", raising=False)
monkeypatch.delenv("STRANDS_DECIDER_API_BASE", raising=False)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
upstream: Final = respx_mock.post("http://strands.local:8080/v1/systemone").respond(
json={
"model": "strands-decider-2B-hobson-v19",
"answers": {"reachable": {"type": "noul", "noul": 1.0}},
"usage": {"input_tokens": 12, "output_tokens": 1},
}
)
result: Final = await ahealth_check(
{
"model": "strands_decider/strands-decider-2B-hobson-v19",
"api_base": "http://strands.local:8080",
},
mode=None,
)
assert "error" not in result, result
assert upstream.called
assert "authorization" not in upstream.calls[0].request.headers

1
tests/unit/llms/azure/test_azure_common_utils.py
View file

@ -470,6 +470,7 @@ def test_default_max_retries_env_var_reaches_azure_sdk_client():
"allm_passthrough_route",
"llm_passthrough_route",
"asearch",
"adecisions",
"avector_store_create",
"avector_store_search",
"acreate_skill",

0
tests/unit/proxy/decisions_endpoints/__init__.py Normal file
View file

326
tests/unit/proxy/decisions_endpoints/test_endpoints.py Normal file
View file

@ -0,0 +1,326 @@
from __future__ import annotations
import asyncio
import json
from collections.abc import AsyncGenerator, Iterator, Mapping
from contextlib import asynccontextmanager
from pathlib import Path
from typing import Final
import pytest
import respx
from fastapi import FastAPI
from fastapi.routing import APIRoute
from fastapi.testclient import TestClient
from starlette.routing import Match
import litellm
from litellm.proxy._lazy_features import LAZY_FEATURES, LazyFeature, attach_lazy_features
from litellm.proxy.decisions_endpoints.endpoints import decisions
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import SafeRouteAdder
from litellm.proxy.proxy_server import (
app,
cleanup_router_config_variables,
initialize,
)
_INPUT_TOKENS: Final[int] = 367
_OUTPUT_TOKENS: Final[int] = 3
_RESPONSE: Final[Mapping[str, object]] = {
"model": "pplx-decider-v1-27b",
"answers": {
"is_defect": {"type": "noul", "noul": 0.9},
},
"usage": {"input_tokens": _INPUT_TOKENS, "output_tokens": _OUTPUT_TOKENS},
}
_STRANDS_RESPONSE: Final[Mapping[str, object]] = {
"model": "strands-decider-2B-hobson-v19",
"answers": {
"is_defect": {"type": "noul", "noul": 0.9},
},
"usage": {"input_tokens": 216, "output_tokens": 3},
"latency_ms": 3722.17,
}
_REQUEST: Final[Mapping[str, object]] = {
"model": "decider",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
}
@pytest.fixture
def client(monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
monkeypatch.setenv("OPENAI_API_KEY", "fake-openai-key")
monkeypatch.setenv("OPENAI_API_BASE", "https://fake-openai.example")
monkeypatch.setenv("REDIS_HOST", "localhost")
cleanup_router_config_variables()
config_path: Final = Path(__file__).parents[1] / "test_configs" / "test_config_no_auth.yaml"
asyncio.run(initialize(config=str(config_path), debug=True))
monkeypatch.setattr(
litellm.proxy.proxy_server,
"llm_router",
litellm.Router(
model_list=[
{
"model_name": "decider",
"litellm_params": {
"model": "perplexity/pplx-decider-v1-27b",
"api_key": "test-key",
},
}
]
),
)
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
yield TestClient(app)
litellm.in_memory_llm_clients_cache.flush_cache()
@pytest.mark.parametrize("endpoint", ("/v1/decisions", "/decisions"))
def test_proxy_decisions_route_returns_answers_and_cost(
client: TestClient,
respx_mock: respx.MockRouter,
endpoint: str,
) -> None:
upstream: Final = respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(json=_RESPONSE)
response: Final = client.post(endpoint, json=_REQUEST)
assert response.status_code == 200, response.text
assert response.json()["answers"] == _RESPONSE["answers"]
assert "_hidden_params" not in response.json()
perplexity_cost: Final = litellm.model_cost["perplexity/pplx-decider-v1-27b"]
expected_cost: Final = _INPUT_TOKENS * float(perplexity_cost["input_cost_per_token"]) + _OUTPUT_TOKENS * float(
perplexity_cost["output_cost_per_token"]
)
assert expected_cost > 0
assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(expected_cost)
assert upstream.called
assert json.loads(upstream.calls[0].request.content) == {
"model": "pplx-decider-v1-27b",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
}
assert upstream.calls[0].request.headers["authorization"] == "Bearer test-key"
def test_proxy_decisions_dispatches_typesafe_deployment(
client: TestClient,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
router: Final = litellm.Router(
model_list=[
{
"model_name": "jev",
"litellm_params": {
"model": "typesafe/jev-latest",
"api_key": "k",
},
}
]
)
monkeypatch.setattr(litellm.proxy.proxy_server, "llm_router", router)
upstream: Final = respx_mock.post("https://api.typesafe.ai/v1/systemone").respond(json=_RESPONSE)
response: Final = client.post(
"/v1/decisions",
json={
"model": "jev",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
)
assert response.status_code == 200, response.text
assert response.json()["answers"] == _RESPONSE["answers"]
assert upstream.called
assert json.loads(upstream.calls[0].request.content) == {
"model": "jev-latest",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
}
def test_proxy_decisions_sends_the_env_key_to_the_deployment_api_base(
client: TestClient,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setenv("PERPLEXITYAI_API_KEY", "server-key")
monkeypatch.delenv("PERPLEXITY_API_KEY", raising=False)
router: Final = litellm.Router(
model_list=[
{
"model_name": "decider",
"litellm_params": {
"model": "perplexity/pplx-decider-v1-27b",
"api_base": "https://egress.example/perplexity",
},
}
]
)
monkeypatch.setattr(litellm.proxy.proxy_server, "llm_router", router)
upstream: Final = respx_mock.post("https://egress.example/perplexity/v1/decisions").respond(json=_RESPONSE)
response: Final = client.post("/v1/decisions", json=_REQUEST)
assert response.status_code == 200, response.text
assert upstream.call_count == 1
assert upstream.calls[0].request.headers["authorization"] == "Bearer server-key"
def test_proxy_decisions_unknown_model_is_a_client_error(
client: TestClient,
respx_mock: respx.MockRouter,
) -> None:
response: Final = client.post(
"/v1/decisions",
json={
"model": "missing-model",
"state": "review",
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
)
assert 400 <= response.status_code < 500, response.text
assert len(respx_mock.calls) == 0
@pytest.mark.parametrize(
"request_body",
(
{
"model": "decider",
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
{
"model": "decider",
"state": {"source": "proxy-test"},
},
),
ids=("missing_state", "missing_questions"),
)
def test_proxy_decisions_missing_required_field_is_a_client_error(
client: TestClient,
respx_mock: respx.MockRouter,
request_body: Mapping[str, object],
) -> None:
upstream: Final = respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(json=_RESPONSE)
response: Final = client.post("/v1/decisions", json=request_body)
assert response.status_code == 400, response.text
assert not upstream.called
def test_proxy_decisions_dispatches_strands_decider(
client: TestClient,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.delenv("STRANDS_DECIDER_API_BASE", raising=False)
monkeypatch.delenv("STRANDS_DECIDER_API_KEY", raising=False)
router: Final = litellm.Router(
model_list=[
{
"model_name": "strands",
"litellm_params": {
"model": "strands_decider/strands-decider-2B-hobson-v19",
"api_base": "https://strands.example",
},
}
]
)
monkeypatch.setattr(litellm.proxy.proxy_server, "llm_router", router)
upstream: Final = respx_mock.post("https://strands.example/v1/systemone").respond(json=_STRANDS_RESPONSE)
response: Final = client.post(
"/v1/decisions",
json={
"model": "strands",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
)
assert response.status_code == 200, response.text
assert response.json()["answers"] == _STRANDS_RESPONSE["answers"]
assert upstream.called
assert json.loads(upstream.calls[0].request.content) == {
"model": "strands-decider-2B-hobson-v19",
"state": {"source": "proxy-test"},
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
}
assert "authorization" not in upstream.calls[0].request.headers
def test_proxy_decisions_without_model_uses_the_proxy_default_model(
client: TestClient,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setattr(litellm.proxy.proxy_server, "user_model", "decider")
upstream: Final = respx_mock.post("https://api.perplexity.ai/v1/decisions").respond(json=_RESPONSE)
response: Final = client.post(
"/v1/decisions", json={key: value for key, value in _REQUEST.items() if key != "model"}
)
assert response.status_code == 200, response.text
assert response.json()["answers"] == _RESPONSE["answers"]
assert upstream.called
assert json.loads(upstream.calls[0].request.content)["model"] == "pplx-decider-v1-27b"
def _decisions_feature() -> LazyFeature:
return next(feature for feature in LAZY_FEATURES if feature.name == "decisions")
def _serving_endpoint(bare: FastAPI, path: str) -> object:
scope: Final = {"type": "http", "method": "POST", "path": path, "root_path": "", "query_string": b"", "headers": ()}
return next(
route.endpoint for route in bare.routes if isinstance(route, APIRoute) and route.matches(scope)[0] is Match.FULL
)
def test_a_config_pass_through_at_v1_decisions_keeps_its_route_and_the_native_api_serves_decisions(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv("LITELLM_DISABLE_LAZY_ROUTES", raising=False)
async def pass_through() -> dict[str, str]:
return {"served_by": "pass-through"}
bare: Final = FastAPI()
attach_lazy_features(bare, (_decisions_feature(),))
SafeRouteAdder.add_api_route_if_not_exists(bare, "/v1/decisions", pass_through, ["POST"])
with TestClient(bare) as client:
assert client.post("/v1/decisions", json={"model": "gpt-6-luna"}).json() == {"served_by": "pass-through"}
assert _serving_endpoint(bare, "/v1/decisions") is pass_through
assert _serving_endpoint(bare, "/decisions") is decisions
def test_with_lazy_routes_disabled_a_config_pass_through_at_v1_decisions_still_wins(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("LITELLM_DISABLE_LAZY_ROUTES", "true")
async def pass_through() -> dict[str, str]:
return {"served_by": "pass-through"}
@asynccontextmanager
async def loads_the_config(app_: FastAPI) -> AsyncGenerator[None]:
assert SafeRouteAdder.add_api_route_if_not_exists(app_, "/v1/decisions", pass_through, ["POST"]), (
"the native route registered at startup must not block the config pass-through"
)
yield
bare: Final = FastAPI(lifespan=loads_the_config)
attach_lazy_features(bare, (_decisions_feature(),))
with TestClient(bare) as client:
assert client.post("/v1/decisions", json={"model": "gpt-6-luna"}).json() == {"served_by": "pass-through"}
assert _serving_endpoint(bare, "/v1/decisions") is pass_through
assert _serving_endpoint(bare, "/decisions") is decisions

35
tests/unit/proxy/pass_through_endpoints/test_pass_through_endpoints.py
View file

@ -9,15 +9,16 @@ from collections.abc import Callable, Mapping
from contextlib import ExitStack, contextmanager
from dataclasses import dataclass
from io import BytesIO
from types import MappingProxyType, SimpleNamespace
from types import MappingProxyType, ModuleType, SimpleNamespace
from typing import Final
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
import respx
from fastapi import HTTPException, Request, Response, UploadFile
from fastapi import APIRouter, FastAPI, HTTPException, Request, Response, UploadFile
from fastapi.responses import StreamingResponse
from fastapi.testclient import TestClient
from pydantic import TypeAdapter, ValidationError
from starlette.datastructures import FormData, Headers, QueryParams
from starlette.datastructures import UploadFile as StarletteUploadFile
@ -27,12 +28,14 @@ from litellm._logging import verbose_proxy_logger
from litellm.constants import DEFAULT_REQUEST_TIMEOUT_SECONDS
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.proxy._lazy_features import LazyFeature, attach_lazy_features
from litellm.proxy._types import ProxyException, UserAPIKeyAuth
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (
DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS,
LITELLM_PASS_THROUGH_CUSTOM_BODY_STATE_KEY,
HttpPassThroughEndpointHelpers,
InitPassThroughEndpointHelpers,
SafeRouteAdder,
_registered_pass_through_routes,
_truncate_upstream_error_body,
_with_trace_context,
@ -8319,3 +8322,31 @@ async def test_pass_throughs_stay_open_while_a_db_sync_reads_the_database(tmp_pa
await sync
assert (config_during_sync.status_code, db_during_sync.status_code) == (200, 200)
def _lazy_feature(monkeypatch: pytest.MonkeyPatch, name: str, path: str) -> LazyFeature:
async def served() -> dict[str, str]:
return {"served_by": name}
router: Final = APIRouter()
router.add_api_route(path, served, methods=["POST"])
module: Final = ModuleType(f"tests.unit.proxy.pass_through_endpoints.lazy_fixture_{name}")
module.router = router # pyright: ignore[reportAttributeAccessIssue] # fixture module built at test time
monkeypatch.setitem(sys.modules, module.__name__, module)
return LazyFeature(name=name, module_path=module.__name__, path_prefixes=(path,))
def test_a_pass_through_added_after_a_lazy_feature_loaded_takes_over_its_path(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("LITELLM_DISABLE_LAZY_ROUTES", raising=False)
async def pass_through() -> dict[str, str]:
return {"served_by": "pass-through"}
app: Final = FastAPI()
attach_lazy_features(app, (_lazy_feature(monkeypatch, "decider", "/v1/decider"),))
with TestClient(app) as client:
assert client.post("/v1/decider").json() == {"served_by": "decider"}
assert SafeRouteAdder.add_api_route_if_not_exists(app, "/v1/decider", pass_through, ["POST"])
assert client.post("/v1/decider").json() == {"served_by": "pass-through"}
assert not SafeRouteAdder.add_api_route_if_not_exists(app, "/v1/decider", pass_through, ["POST"])
assert client.post("/v1/decider").json() == {"served_by": "pass-through"}

2
tests/unit/proxy/public_endpoints/test_public_endpoints.py
View file

@ -436,10 +436,12 @@ ADD_MODEL_UNLISTED_PROVIDERS: Final = frozenset(
"sagemaker_nova",
"scaleway",
"stability",
"strands_decider",
"synthetic",
"tensormesh",
"text-completion-inception",
"transcribe",
"typesafe",
"valkey",
"xiaomi_mimo",
"zai",

76
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -4586,6 +4586,23 @@ export interface paths {
patch?: never;
trace?: never;
};
"/decisions": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/** Decisions */
post: operations["decisions_decisions_post"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/delete/allowed_ip": {
parameters: {
query?: never;
@ -19790,6 +19807,23 @@ export interface paths {
patch?: never;
trace?: never;
};
"/v1/decisions": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/** Decisions */
post: operations["decisions_v1_decisions_post"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/v1/embeddings": {
parameters: {
query?: never;
@ -28092,7 +28126,7 @@ export interface components {
* CallTypes
* @enum {string}
*/
CallTypes: "embedding" | "aembedding" | "completion" | "acompletion" | "atext_completion" | "text_completion" | "image_generation" | "aimage_generation" | "image_edit" | "aimage_edit" | "moderation" | "amoderation" | "atranscription" | "transcription" | "aspeech" | "speech" | "rerank" | "arerank" | "search" | "asearch" | "_arealtime" | "_aresponses_websocket" | "create_batch" | "acreate_batch" | "aretrieve_batch" | "retrieve_batch" | "acancel_batch" | "cancel_batch" | "pass_through_endpoint" | "anthropic_messages" | "aanthropic_messages" | "get_assistants" | "aget_assistants" | "create_assistants" | "acreate_assistants" | "delete_assistant" | "adelete_assistant" | "acreate_thread" | "create_thread" | "aget_thread" | "get_thread" | "a_add_message" | "add_message" | "aget_messages" | "get_messages" | "arun_thread" | "run_thread" | "arun_thread_stream" | "run_thread_stream" | "afile_retrieve" | "file_retrieve" | "afile_delete" | "file_delete" | "afile_list" | "file_list" | "acreate_file" | "create_file" | "afile_content" | "file_content" | "create_fine_tuning_job" | "acreate_fine_tuning_job" | "create_video" | "acreate_video" | "video_generation" | "avideo_generation" | "avideo_retrieve" | "video_retrieve" | "avideo_content" | "video_content" | "video_remix" | "avideo_remix" | "video_list" | "avideo_list" | "video_retrieve_job" | "avideo_retrieve_job" | "video_delete" | "avideo_delete" | "video_create_character" | "avideo_create_character" | "video_get_character" | "avideo_get_character" | "video_edit" | "avideo_edit" | "video_extension" | "avideo_extension" | "vector_store_file_create" | "avector_store_file_create" | "vector_store_file_list" | "avector_store_file_list" | "vector_store_file_retrieve" | "avector_store_file_retrieve" | "vector_store_file_content" | "avector_store_file_content" | "vector_store_file_update" | "avector_store_file_update" | "vector_store_file_delete" | "avector_store_file_delete" | "vector_store_create" | "avector_store_create" | "vector_store_search" | "avector_store_search" | "ingest" | "aingest" | "query" | "aquery" | "create_interaction" | "acreate_interaction" | "create_container" | "acreate_container" | "list_containers" | "alist_containers" | "retrieve_container" | "aretrieve_container" | "delete_container" | "adelete_container" | "list_container_files" | "alist_container_files" | "upload_container_file" | "aupload_container_file" | "create_sandbox" | "acreate_sandbox" | "delete_sandbox" | "adelete_sandbox" | "run_code" | "arun_code" | "code_interpreter_tool" | "acode_interpreter_tool" | "acancel_fine_tuning_job" | "cancel_fine_tuning_job" | "alist_fine_tuning_jobs" | "list_fine_tuning_jobs" | "aretrieve_fine_tuning_job" | "retrieve_fine_tuning_job" | "responses" | "aresponses" | "alist_input_items" | "llm_passthrough_route" | "allm_passthrough_route" | "generate_content" | "agenerate_content" | "generate_content_stream" | "agenerate_content_stream" | "ocr" | "aocr" | "call_mcp_tool" | "list_mcp_tools" | "asend_message" | "send_message" | "acreate_skill";
CallTypes: "embedding" | "aembedding" | "completion" | "acompletion" | "atext_completion" | "text_completion" | "image_generation" | "aimage_generation" | "image_edit" | "aimage_edit" | "moderation" | "amoderation" | "atranscription" | "transcription" | "aspeech" | "speech" | "rerank" | "arerank" | "search" | "asearch" | "decisions" | "adecisions" | "_arealtime" | "_aresponses_websocket" | "create_batch" | "acreate_batch" | "aretrieve_batch" | "retrieve_batch" | "acancel_batch" | "cancel_batch" | "pass_through_endpoint" | "anthropic_messages" | "aanthropic_messages" | "get_assistants" | "aget_assistants" | "create_assistants" | "acreate_assistants" | "delete_assistant" | "adelete_assistant" | "acreate_thread" | "create_thread" | "aget_thread" | "get_thread" | "a_add_message" | "add_message" | "aget_messages" | "get_messages" | "arun_thread" | "run_thread" | "arun_thread_stream" | "run_thread_stream" | "afile_retrieve" | "file_retrieve" | "afile_delete" | "file_delete" | "afile_list" | "file_list" | "acreate_file" | "create_file" | "afile_content" | "file_content" | "create_fine_tuning_job" | "acreate_fine_tuning_job" | "create_video" | "acreate_video" | "video_generation" | "avideo_generation" | "avideo_retrieve" | "video_retrieve" | "avideo_content" | "video_content" | "video_remix" | "avideo_remix" | "video_list" | "avideo_list" | "video_retrieve_job" | "avideo_retrieve_job" | "video_delete" | "avideo_delete" | "video_create_character" | "avideo_create_character" | "video_get_character" | "avideo_get_character" | "video_edit" | "avideo_edit" | "video_extension" | "avideo_extension" | "vector_store_file_create" | "avector_store_file_create" | "vector_store_file_list" | "avector_store_file_list" | "vector_store_file_retrieve" | "avector_store_file_retrieve" | "vector_store_file_content" | "avector_store_file_content" | "vector_store_file_update" | "avector_store_file_update" | "vector_store_file_delete" | "avector_store_file_delete" | "vector_store_create" | "avector_store_create" | "vector_store_search" | "avector_store_search" | "ingest" | "aingest" | "query" | "aquery" | "create_interaction" | "acreate_interaction" | "create_container" | "acreate_container" | "list_containers" | "alist_containers" | "retrieve_container" | "aretrieve_container" | "delete_container" | "adelete_container" | "list_container_files" | "alist_container_files" | "upload_container_file" | "aupload_container_file" | "create_sandbox" | "acreate_sandbox" | "delete_sandbox" | "adelete_sandbox" | "run_code" | "arun_code" | "code_interpreter_tool" | "acode_interpreter_tool" | "acancel_fine_tuning_job" | "cancel_fine_tuning_job" | "alist_fine_tuning_jobs" | "list_fine_tuning_jobs" | "aretrieve_fine_tuning_job" | "retrieve_fine_tuning_job" | "responses" | "aresponses" | "alist_input_items" | "llm_passthrough_route" | "allm_passthrough_route" | "generate_content" | "agenerate_content" | "generate_content_stream" | "agenerate_content_stream" | "ocr" | "aocr" | "call_mcp_tool" | "list_mcp_tools" | "asend_message" | "send_message" | "acreate_skill";
/** CallbackDelete */
CallbackDelete: {
/** Callback Name */
@ -56843,6 +56877,26 @@ export interface operations {
};
};
};
decisions_decisions_post: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
};
};
delete_allowed_ip_delete_allowed_ip_post: {
parameters: {
query?: never;
@ -76518,6 +76572,26 @@ export interface operations {
};
};
};
decisions_v1_decisions_post: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
};
};
embeddings_v1_embeddings_post: {
parameters: {
query?: never;