scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-13 03:59:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

feat(decisions): bill decisions and systemone from _decisions cost map fields (#45904)

Browse source 
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(decisions): mark decisions pricing rows with mode decisions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): sync generated API types for decisions mode

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): update pricing expectations in proxy tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(decisions): price classifier calls from decision rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(health): drop redundant evaluation handler after main merge

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(decisions): inject pricing row for deployed classifier

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): ship the decisions pricing lookup without cost map rows

Restore both cost maps to main. Tests inject their own
<provider>/decisions/<model> rows. The Add Model dropdown hides those
pricing keys and offers mode decisions instead of evaluation.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): expect the Decisions mode option in the Add Model form

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(decisions): bill decisions and systemone from _decisions cost map fields

Replace the provider/decisions/model pricing rows with <field>_decisions
fields on the model's own row. cost_per_token overlays them for the
decisions, adecisions, systemone and asystemone call types and prices
through generic_cost_per_token; chat, responses and batch keep the plain
fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0,
cached and cache-write input 1e-07, 2x above 272k tokens).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): cover gpt-6-luna decisions and systemone above the 272k tier

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(decisions): keep the decisions pricing block clear of main's stale azure comment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(cost_calculator): drop stale azure models doc comment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
devin-ai-integration[bot] 2026-10-12 14:31:12 -07:00 • committed by GitHub
parent 932a0d6151
commit 7dc7f101d9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
20 changed files with 707 additions and 82 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
ci_cd/cost_map_guard.py
View file

@ -56,7 +56,7 @@ def _rendered_schema(cost_map: CostMap) -> str:
def _evaluation_output_price_failure(
key: str, entry: object, evaluation_output_price_sources: Mapping[str, str]
) -> str | None:
if not isinstance(entry, dict) or entry.get("mode") != "evaluation":
if not isinstance(entry, dict) or entry.get("mode") not in ("evaluation", "decisions"):
return None
output_cost: Final = entry.get("output_cost_per_token")
if (
@ -67,7 +67,7 @@ def _evaluation_output_price_failure(
):
return None
return (
f"{key}: mode evaluation rows bill input only; output_cost_per_token must be 0 unless the vendor "
f"{key}: mode evaluation and decisions rows bill input only; output_cost_per_token must be 0 unless the vendor "
"prices output tokens for this model (add the source to EVALUATION_OUTPUT_PRICE_SOURCES)"
)

20
litellm/cost_calculator.py
View file

@ -18,6 +18,7 @@ from litellm.constants import (
DEFAULT_MAX_LRU_CACHE_SIZE,
DEFAULT_REPLICATE_GPU_PRICE_PER_SECOND,
)
from litellm.litellm_core_utils.decisions_pricing import decisions_pricing_fields, is_decisions_call_type
from litellm.litellm_core_utils.hidden_params import HIDDEN_PARAMS_ATTR
from litellm.litellm_core_utils.llm_cost_calc.tool_call_cost_tracking import (
StandardBuiltInToolCostTracking,
@ -570,7 +571,22 @@ def cost_per_token(
): # Option 3. if user passed model="bedrock/anthropic.claude-3", use model="anthropic.claude-3"
model = model_without_prefix
# see this https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models
if is_decisions_call_type(call_type):
decisions_model_info: Final = cached_get_model_info_helper(model=model, custom_llm_provider=custom_llm_provider)
decisions_overrides: Final = decisions_pricing_fields(
litellm.model_cost.get(decisions_model_info["key"], {}) # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType] # model cost map is dynamically loaded
)
if decisions_overrides:
return generic_cost_per_token(
model=model,
usage=usage_block,
custom_llm_provider=custom_llm_provider,
service_tier=service_tier,
data_residency=data_residency,
model_info=cast( # cast-ok: _decisions fields override same-named ModelInfo rates
ModelInfo, {**decisions_model_info, **decisions_overrides}
),
)
if call_type == "speech" or call_type == "aspeech":
lyria_generation_cost: Final = (
get_vertex_ai_lyria_generation_cost(model=model_without_prefix)
@ -1124,6 +1140,8 @@ def infer_call_type(call_type: CallTypesLiteral | None, completion_response: obj
return "text_completion"
elif isinstance(completion_response, LiteLLMSendMessageResponse):
return "send_message"
elif isinstance(completion_response, (DecisionsResponse, OpenAIDecisionResponse)):
return "adecisions"
return call_type

33
litellm/litellm_core_utils/decisions_pricing.py Normal file
View file

@ -0,0 +1,33 @@
"""Decisions-specific pricing read from ``<field>_decisions`` cost map fields.
A model that serves both chat and Decisions keeps one cost map row. Rates that
differ for ``/v1/decisions`` and ``/v1/systemone`` live on that row under the
same field name with a ``_decisions`` suffix, e.g. ``output_cost_per_token_decisions``,
and override the plain field for those two routes only. A row without any such
field bills Decisions calls from its plain fields.
"""
from collections.abc import Mapping
from typing import Final
DECISIONS_PRICING_SUFFIX: Final = "_decisions"
DECISIONS_CALL_TYPES: Final = frozenset({"decisions", "adecisions", "systemone", "asystemone"})
def is_decisions_call_type(call_type: str) -> bool:
return call_type in DECISIONS_CALL_TYPES
def decisions_pricing_fields(row: Mapping[str, object]) -> Mapping[str, object]:
"""Return ``{plain_field: value}`` for every ``<plain_field>_decisions`` field on ``row``."""
return {
key.removesuffix(DECISIONS_PRICING_SUFFIX): value
for key, value in row.items()
if key.endswith(DECISIONS_PRICING_SUFFIX)
}
def apply_decisions_pricing(row: Mapping[str, object]) -> Mapping[str, object]:
"""Return ``row`` priced for a Decisions call: ``_decisions`` values replace their plain fields."""
plain: Final = {key: value for key, value in row.items() if not key.endswith(DECISIONS_PRICING_SUFFIX)}
return {**plain, **decisions_pricing_fields(row)}

5
litellm/llms/pass_through/typesafe_logging_handler.py
View file

@ -6,6 +6,7 @@ import httpx
from pydantic import TypeAdapter, ValidationError
import litellm
from litellm.litellm_core_utils.decisions_pricing import apply_decisions_pricing
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.litellm_logging import (
get_standard_logging_object_payload, # pyright: ignore[reportUnknownVariableType] # legacy helper has an untyped signature
@ -51,7 +52,9 @@ def _pricing_for(model_keys: tuple[str, ...]) -> _RegistryPricing:
continue
try:
return _REGISTRY_PRICING_ADAPTER.validate_python(
litellm.model_cost[model_key] # pyright: ignore[reportUnknownMemberType] # registry is dynamically typed
apply_decisions_pricing(
litellm.model_cost[model_key] # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType] # registry is dynamically typed
)
)
except ValidationError:
continue

8
litellm/model_prices_and_context_window_backup.json
View file

@ -35086,13 +35086,17 @@
},
"gpt-6-luna": {
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_decisions": 1e-07,
"cache_creation_input_token_cost_above_272k_tokens": 2.5e-07,
"cache_creation_input_token_cost_above_272k_tokens_decisions": 2e-07,
"cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-07,
"cache_creation_input_token_cost_above_272k_tokens_priority": 5e-07,
"cache_creation_input_token_cost_flex": 6.25e-08,
"cache_creation_input_token_cost_priority": 2.5e-07,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_decisions": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-08,
"cache_read_input_token_cost_above_272k_tokens_decisions": 2e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 1e-08,
"cache_read_input_token_cost_above_272k_tokens_priority": 4e-08,
"cache_read_input_token_cost_batches": 5e-09,
@ -35102,7 +35106,9 @@
"cache_read_input_token_cost_flex": 5e-09,
"cache_read_input_token_cost_priority": 2e-08,
"input_cost_per_token": 1e-07,
"input_cost_per_token_decisions": 1e-07,
"input_cost_per_token_above_272k_tokens": 2e-07,
"input_cost_per_token_above_272k_tokens_decisions": 2e-07,
"input_cost_per_token_above_272k_tokens_flex": 1e-07,
"input_cost_per_token_above_272k_tokens_priority": 4e-07,
"input_cost_per_token_batches": 5e-08,
@ -35115,7 +35121,9 @@
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_decisions": 0.0,
"output_cost_per_token_above_272k_tokens": 7.5e-07,
"output_cost_per_token_above_272k_tokens_decisions": 0.0,
"output_cost_per_token_above_272k_tokens_flex": 3.75e-07,
"output_cost_per_token_above_272k_tokens_priority": 1.5e-06,
"output_cost_per_token_batches": 2.5e-07,

8
model_prices_and_context_window.json
View file

@ -35086,13 +35086,17 @@
},
"gpt-6-luna": {
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_decisions": 1e-07,
"cache_creation_input_token_cost_above_272k_tokens": 2.5e-07,
"cache_creation_input_token_cost_above_272k_tokens_decisions": 2e-07,
"cache_creation_input_token_cost_above_272k_tokens_flex": 1.25e-07,
"cache_creation_input_token_cost_above_272k_tokens_priority": 5e-07,
"cache_creation_input_token_cost_flex": 6.25e-08,
"cache_creation_input_token_cost_priority": 2.5e-07,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_decisions": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-08,
"cache_read_input_token_cost_above_272k_tokens_decisions": 2e-07,
"cache_read_input_token_cost_above_272k_tokens_flex": 1e-08,
"cache_read_input_token_cost_above_272k_tokens_priority": 4e-08,
"cache_read_input_token_cost_batches": 5e-09,
@ -35102,7 +35106,9 @@
"cache_read_input_token_cost_flex": 5e-09,
"cache_read_input_token_cost_priority": 2e-08,
"input_cost_per_token": 1e-07,
"input_cost_per_token_decisions": 1e-07,
"input_cost_per_token_above_272k_tokens": 2e-07,
"input_cost_per_token_above_272k_tokens_decisions": 2e-07,
"input_cost_per_token_above_272k_tokens_flex": 1e-07,
"input_cost_per_token_above_272k_tokens_priority": 4e-07,
"input_cost_per_token_batches": 5e-08,
@ -35115,7 +35121,9 @@
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_decisions": 0.0,
"output_cost_per_token_above_272k_tokens": 7.5e-07,
"output_cost_per_token_above_272k_tokens_decisions": 0.0,
"output_cost_per_token_above_272k_tokens_flex": 3.75e-07,
"output_cost_per_token_above_272k_tokens_priority": 1.5e-06,
"output_cost_per_token_batches": 2.5e-07,

36
model_prices_and_context_window.schema.json
View file

@ -123,6 +123,11 @@
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"cache_creation_input_token_cost_above_272k_tokens_decisions": {
"type": "number",
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"cache_creation_input_token_cost_above_272k_tokens_flex": {
"type": "number",
"minimum": 0,
@ -142,6 +147,10 @@
"type": "number",
"minimum": 0
},
"cache_creation_input_token_cost_decisions": {
"type": "number",
"minimum": 0
},
"cache_creation_input_token_cost_flex": {
"type": "number",
"minimum": 0,
@ -204,6 +213,11 @@
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"cache_read_input_token_cost_above_272k_tokens_decisions": {
"type": "number",
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"cache_read_input_token_cost_above_272k_tokens_flex": {
"type": "number",
"minimum": 0,
@ -232,6 +246,10 @@
"type": "number",
"minimum": 0
},
"cache_read_input_token_cost_decisions": {
"type": "number",
"minimum": 0
},
"cache_read_input_token_cost_flex": {
"type": "number",
"minimum": 0,
@ -407,6 +425,11 @@
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"input_cost_per_token_above_272k_tokens_decisions": {
"type": "number",
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"input_cost_per_token_above_272k_tokens_flex": {
"type": "number",
"minimum": 0,
@ -440,6 +463,10 @@
"type": "number",
"minimum": 0
},
"input_cost_per_token_decisions": {
"type": "number",
"minimum": 0
},
"input_cost_per_token_flex": {
"type": "number",
"minimum": 0,
@ -786,6 +813,11 @@
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"output_cost_per_token_above_272k_tokens_decisions": {
"type": "number",
"minimum": 0,
"description": "Rate applied once the prompt exceeds the token threshold in the field name."
},
"output_cost_per_token_above_272k_tokens_flex": {
"type": "number",
"minimum": 0,
@ -815,6 +847,10 @@
"minimum": 0,
"description": "USD per generated token via the provider's batch API."
},
"output_cost_per_token_decisions": {
"type": "number",
"minimum": 0
},
"output_cost_per_token_flex": {
"type": "number",
"minimum": 0,

175
tests/integration/cost_calculation/cases/openai.py
View file

@ -2,9 +2,10 @@
PARITY pairs the streamed case with its plain twin where both bill the same row."""
from typing import Final
from typing import Final, cast
from integration.cost_calculation.cost_tracking_case import (
COST_MAP,
BinaryResponse,
CostTrackingTestCase,
Deployment,
@ -18,6 +19,7 @@ from integration.cost_calculation.cost_tracking_case import (
SseResponse,
WavUpload,
)
from pydantic import JsonValue
from integration.cost_calculation.stream_parity.case import StreamParityTestCase, sse_frames
@ -7848,6 +7850,173 @@ GPT_6_1_SOL_INPUT_TEXT: Final = CostTrackingTestCase(
)
_OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS: Final = 367
_OPENAI_GPT_6_LUNA_DECISIONS_OUTPUT_TOKENS: Final = 3
_OPENAI_GPT_6_LUNA_DECISIONS_INPUT_RATE: Final = cast(
float,
COST_MAP["gpt-6-luna"].input_cost_per_token_decisions,
)
_OPENAI_GPT_6_LUNA_DECISIONS_SPEND: Final = (
_OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS * _OPENAI_GPT_6_LUNA_DECISIONS_INPUT_RATE
)
_OPENAI_GPT_6_LUNA_DECISIONS_UPSTREAM_BODY: Final[dict[str, JsonValue]] = {
"model": "gpt-6-luna",
"input": "The package arrived broken.",
"questions": [{"type": "predicate", "name": "is_defect", "instructions": "Is this a defect?"}],
}
_OPENAI_GPT_6_LUNA_DECISIONS_RESPONSE: Final = JsonResponse(
content_type="application/json",
body={
"model": "gpt-6-luna",
"answers": [{"type": "predicate", "name": "is_defect", "probability": 0.9}],
"usage": {
"input_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS,
"input_tokens_details": {"cached_tokens": 256, "cache_write_tokens": 64},
"output_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_OUTPUT_TOKENS,
"output_tokens_details": {"reasoning_tokens": 0},
"total_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS + _OPENAI_GPT_6_LUNA_DECISIONS_OUTPUT_TOKENS,
},
},
)
OPENAI_GPT_6_LUNA_DECISIONS: Final = CostTrackingTestCase(
name="openai-gpt-6-luna-decisions-evaluation-pricing",
covers="quota_management.spend_tracking.decisions_costs",
model="gpt-6-luna",
endpoint="/v1/decisions",
deployment=Deployment(model="openai/gpt-6-luna"),
request={
"model": "$MODEL",
"input": "The package arrived broken.",
"questions": [{"type": "predicate", "name": "is_defect", "instructions": "Is this a defect?"}],
},
upstream_body=_OPENAI_GPT_6_LUNA_DECISIONS_UPSTREAM_BODY,
response=_OPENAI_GPT_6_LUNA_DECISIONS_RESPONSE,
expected=ExactExpected(
spend=_OPENAI_GPT_6_LUNA_DECISIONS_SPEND,
input_cost=_OPENAI_GPT_6_LUNA_DECISIONS_SPEND,
output_cost=0.0,
prompt_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS,
completion_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_OUTPUT_TOKENS,
),
)
OPENAI_GPT_6_LUNA_SYSTEMONE: Final = CostTrackingTestCase(
name="openai-gpt-6-luna-systemone-evaluation-pricing",
covers="quota_management.spend_tracking.decisions_costs",
model="gpt-6-luna",
endpoint="/v1/systemone",
deployment=Deployment(model="openai/gpt-6-luna"),
request={
"model": "$MODEL",
"state": "The package arrived broken.",
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
upstream_body=_OPENAI_GPT_6_LUNA_DECISIONS_UPSTREAM_BODY,
response=_OPENAI_GPT_6_LUNA_DECISIONS_RESPONSE,
expected=ExactExpected(
spend=_OPENAI_GPT_6_LUNA_DECISIONS_SPEND,
input_cost=_OPENAI_GPT_6_LUNA_DECISIONS_SPEND,
output_cost=0.0,
prompt_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_INPUT_TOKENS,
completion_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_OUTPUT_TOKENS,
),
)
# Above 272k input tokens OpenAI bills the whole request at the long-context column. For decisions
# that column is the `_above_272k_tokens_decisions` fields: 2e-07 for input, cached and cache-write
# tokens alike, output still 0. Mirrors the live 320,162-token call that billed 0.0640324.
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS: Final = 320162
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHED_TOKENS: Final = 100000
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_TOKENS: Final = 10000
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_OUTPUT_TOKENS: Final = 3
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_RATE: Final = cast(
float,
COST_MAP["gpt-6-luna"].input_cost_per_token_above_272k_tokens_decisions,
)
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_READ_RATE: Final = cast(
float,
COST_MAP["gpt-6-luna"].cache_read_input_token_cost_above_272k_tokens_decisions,
)
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_RATE: Final = cast(
float,
COST_MAP["gpt-6-luna"].cache_creation_input_token_cost_above_272k_tokens_decisions,
)
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_SPEND: Final = (
(
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS
- _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHED_TOKENS
- _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_TOKENS
)
* _OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_RATE
+ _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHED_TOKENS * _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_READ_RATE
+ _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_TOKENS * _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_RATE
)
_OPENAI_GPT_6_LUNA_DECISIONS_LONG_RESPONSE: Final = JsonResponse(
content_type="application/json",
body={
"model": "gpt-6-luna",
"answers": [{"type": "predicate", "name": "is_defect", "probability": 0.84}],
"usage": {
"input_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS,
"input_tokens_details": {
"cached_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHED_TOKENS,
"cache_write_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_LONG_CACHE_WRITE_TOKENS,
},
"output_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_LONG_OUTPUT_TOKENS,
"output_tokens_details": {"reasoning_tokens": 0},
"total_tokens": _OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS
+ _OPENAI_GPT_6_LUNA_DECISIONS_LONG_OUTPUT_TOKENS,
},
},
)
OPENAI_GPT_6_LUNA_DECISIONS_LONG_CONTEXT: Final = CostTrackingTestCase(
name="openai-gpt-6-luna-decisions-above-272k-pricing",
covers="quota_management.spend_tracking.decisions_costs",
model="gpt-6-luna",
endpoint="/v1/decisions",
deployment=Deployment(model="openai/gpt-6-luna"),
request={
"model": "$MODEL",
"input": "The package arrived broken.",
"questions": [{"type": "predicate", "name": "is_defect", "instructions": "Is this a defect?"}],
},
upstream_body=_OPENAI_GPT_6_LUNA_DECISIONS_UPSTREAM_BODY,
response=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_RESPONSE,
expected=ExactExpected(
spend=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_SPEND,
input_cost=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_SPEND,
output_cost=0.0,
prompt_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS,
completion_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_OUTPUT_TOKENS,
),
)
OPENAI_GPT_6_LUNA_SYSTEMONE_LONG_CONTEXT: Final = CostTrackingTestCase(
name="openai-gpt-6-luna-systemone-above-272k-pricing",
covers="quota_management.spend_tracking.decisions_costs",
model="gpt-6-luna",
endpoint="/v1/systemone",
deployment=Deployment(model="openai/gpt-6-luna"),
request={
"model": "$MODEL",
"state": "The package arrived broken.",
"questions": {"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
},
upstream_body=_OPENAI_GPT_6_LUNA_DECISIONS_UPSTREAM_BODY,
response=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_RESPONSE,
expected=ExactExpected(
spend=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_SPEND,
input_cost=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_SPEND,
output_cost=0.0,
prompt_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_INPUT_TOKENS,
completion_tokens=_OPENAI_GPT_6_LUNA_DECISIONS_LONG_OUTPUT_TOKENS,
),
)
CASES: Final[tuple[CostTrackingTestCase, ...]] = (
GPT_5_3_CODEX_INPUT_TEXT,
GPT_5_3_CODEX_CACHE_READ,
@ -7980,6 +8149,10 @@ CASES: Final[tuple[CostTrackingTestCase, ...]] = (
GPT_5_6_CHAT_REQUEST_TO_EMBEDDING_ENTRY,
GPT_5_6_CLIENT_DISCONNECT_MID_STREAM,
GPT_6_1_SOL_INPUT_TEXT,
OPENAI_GPT_6_LUNA_DECISIONS,
OPENAI_GPT_6_LUNA_SYSTEMONE,
OPENAI_GPT_6_LUNA_DECISIONS_LONG_CONTEXT,
OPENAI_GPT_6_LUNA_SYSTEMONE_LONG_CONTEXT,
)
PARITY: Final[tuple[StreamParityTestCase, ...]] = (

28
tests/integration/cost_calculation/cost_map.py
View file

@ -32,6 +32,27 @@ COST_MAP_ENTRIES: Final[dict[str, dict[str, JsonValue]]] = {
"input_cost_per_token": 3.2e-06,
"output_cost_per_token": 1.6e-05,
},
"gpt-6-luna": {
"cache_creation_input_token_cost": 1.25e-07,
"cache_creation_input_token_cost_decisions": 1e-07,
"cache_creation_input_token_cost_above_272k_tokens": 2.5e-07,
"cache_creation_input_token_cost_above_272k_tokens_decisions": 2e-07,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_decisions": 1e-07,
"cache_read_input_token_cost_above_272k_tokens": 2e-08,
"cache_read_input_token_cost_above_272k_tokens_decisions": 2e-07,
"input_cost_per_token": 1e-07,
"input_cost_per_token_decisions": 1e-07,
"input_cost_per_token_above_272k_tokens": 2e-07,
"input_cost_per_token_above_272k_tokens_decisions": 2e-07,
"litellm_provider": "openai",
"max_input_tokens": 922000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"output_cost_per_token_decisions": 0.0,
"output_cost_per_token_above_272k_tokens": 7.5e-07,
"output_cost_per_token_above_272k_tokens_decisions": 0.0,
},
"perplexity/sonar-next": {
"litellm_provider": "perplexity",
"mode": "chat",
@ -43,13 +64,6 @@ COST_MAP_ENTRIES: Final[dict[str, dict[str, JsonValue]]] = {
"search_context_size_high": 0.012,
},
},
"gpt-6-luna": {
"litellm_provider": "openai",
"mode": "chat",
"input_cost_per_token": 1e-07,
"output_cost_per_token": 5e-07,
"cache_read_input_token_cost": 1e-08,
},
"typesafe/jev-latest": {
"litellm_provider": "typesafe",
"mode": "evaluation",

12
tests/integration/cost_calculation/cost_tracking_case.py
View file

@ -44,8 +44,20 @@ class CostMapEntry(BaseModel):
max_output_tokens: int | None = None
supports_function_calling: bool | None = None
input_cost_per_token: float | None = None
input_cost_per_token_decisions: float | None = None
input_cost_per_query: float | None = None
output_cost_per_token: float | None = None
output_cost_per_token_decisions: float | None = None
cache_read_input_token_cost_decisions: float | None = None
cache_creation_input_token_cost_decisions: float | None = None
input_cost_per_token_above_272k_tokens: float | None = None
input_cost_per_token_above_272k_tokens_decisions: float | None = None
output_cost_per_token_above_272k_tokens: float | None = None
output_cost_per_token_above_272k_tokens_decisions: float | None = None
cache_read_input_token_cost_above_272k_tokens: float | None = None
cache_read_input_token_cost_above_272k_tokens_decisions: float | None = None
cache_creation_input_token_cost_above_272k_tokens: float | None = None
cache_creation_input_token_cost_above_272k_tokens_decisions: float | None = None
input_cost_per_token_batches: float | None = None
output_cost_per_token_batches: float | None = None
output_cost_per_image_token_batches: float | None = None

123
tests/unit/decisions/test_main.py
View file

@ -168,6 +168,25 @@ def _httpx_transport(monkeypatch: pytest.MonkeyPatch) -> None:
litellm.in_memory_llm_clients_cache.flush_cache()
_GPT_6_LUNA_DECISIONS_INPUT_RATE: Final = 1e-07
_MICROSOFT_DECISION_1_DECISIONS_INPUT_RATE: Final = 7e-08
_DECISIONS_PRICING_FIELDS: Final[dict[str, dict[str, object]]] = {
"gpt-6-luna": {
"input_cost_per_token_decisions": _GPT_6_LUNA_DECISIONS_INPUT_RATE,
"cache_read_input_token_cost_decisions": _GPT_6_LUNA_DECISIONS_INPUT_RATE,
"cache_creation_input_token_cost_decisions": _GPT_6_LUNA_DECISIONS_INPUT_RATE,
"output_cost_per_token_decisions": 0.0,
},
"azure_ai/Microsoft-Decision-1": {"input_cost_per_token_decisions": _MICROSOFT_DECISION_1_DECISIONS_INPUT_RATE},
}
@pytest.fixture
def decisions_pricing_fields(monkeypatch: pytest.MonkeyPatch) -> None:
for key, fields in _DECISIONS_PRICING_FIELDS.items():
monkeypatch.setitem(litellm.model_cost, key, {**litellm.model_cost[key], **fields})
@pytest.mark.asyncio
@pytest.mark.parametrize(("provider", "model", "url", "upstream_model"), _PROVIDERS)
async def test_adecisions_sends_the_provider_wire_contract(
@ -323,6 +342,79 @@ def test_decisions_cost_uses_litellm_token_pricing() -> None:
assert cost == pytest.approx(expected_cost)
@pytest.mark.asyncio
async def test_openai_decisions_uses_evaluation_pricing_and_explicit_base_model(
respx_mock: respx.MockRouter,
decisions_pricing_fields: None,
) -> None:
route: Final = respx_mock.post("https://api.openai.com/v1/decisions").respond(json=_OPENAI_RESPONSE)
response: Final = await litellm.adecisions(
model="openai/gpt-6-luna",
input="The package arrived broken.",
questions=[{"type": "predicate", "name": "is_defect", "instructions": "Is this a defect?"}],
api_key="caller-key",
api_base="https://api.openai.com/v1",
)
chat_cost: Mapping[str, object] = litellm.model_cost["gpt-6-luna"]
expected_chat_cost: Final = (
(_INPUT_TOKENS - _CACHED_TOKENS - _CACHE_WRITE_TOKENS) * float(chat_cost["input_cost_per_token"])
+ _CACHED_TOKENS * float(chat_cost["cache_read_input_token_cost"])
+ _CACHE_WRITE_TOKENS * float(chat_cost["cache_creation_input_token_cost"])
+ _OUTPUT_TOKENS * float(chat_cost["output_cost_per_token"])
)
assert route.called
assert response._hidden_params["model"] == "openai/gpt-6-luna"
assert response._hidden_params["provider_response_model"] == "openai/gpt-6-luna"
assert litellm.completion_cost(completion_response=response) == pytest.approx(
_INPUT_TOKENS * _GPT_6_LUNA_DECISIONS_INPUT_RATE
)
assert litellm.completion_cost(completion_response=response, call_type="acompletion") == pytest.approx(
expected_chat_cost
)
@pytest.mark.asyncio
async def test_systemone_uses_evaluation_pricing(respx_mock: respx.MockRouter, decisions_pricing_fields: None) -> None:
route: Final = respx_mock.post("https://api.openai.com/v1/decisions").respond(json=_OPENAI_RESPONSE)
response: Final = await litellm.asystemone(
model="openai/gpt-6-luna",
state="The package arrived broken.",
questions=_QUESTIONS,
api_key="caller-key",
api_base="https://api.openai.com/v1",
)
assert route.called
assert response._hidden_params["model"] == "openai/gpt-6-luna"
assert response._hidden_params["provider_response_model"] == "openai/gpt-6-luna"
assert litellm.completion_cost(completion_response=response) == pytest.approx(
_INPUT_TOKENS * _GPT_6_LUNA_DECISIONS_INPUT_RATE
)
@pytest.mark.asyncio
async def test_systemone_falls_back_to_plain_pricing_without_decisions_row(
respx_mock: respx.MockRouter,
) -> None:
route: Final = respx_mock.post("https://decisions.example/v1/decisions").respond(json=_RESPONSE)
response: Final = await litellm.asystemone(
model="perplexity/sonar",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_key="caller-key",
api_base="https://decisions.example/v1",
)
plain_cost: Mapping[str, object] = litellm.model_cost["perplexity/sonar"]
assert route.called
assert response._hidden_params["provider_response_model"] == "perplexity/sonar"
assert litellm.completion_cost(completion_response=response) == pytest.approx(
_INPUT_TOKENS * float(plain_cost["input_cost_per_token"])
+ _OUTPUT_TOKENS * float(plain_cost["output_cost_per_token"])
)
@pytest.mark.parametrize(
"response",
(
@ -1310,6 +1402,7 @@ async def test_openai_decisions_translate_systemone_to_the_openai_wire_contract_
url: str,
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
decisions_pricing_fields: None,
) -> None:
monkeypatch.setattr(litellm, "api_base", None)
monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
@ -1349,13 +1442,7 @@ async def test_openai_decisions_translate_systemone_to_the_openai_wire_contract_
}
assert response.answers == {"is_defect": NoulAnswer(type="noul", noul=0.9)}
assert response.hidden_params["custom_llm_provider"] == "openai"
luna_cost: Final = litellm.model_cost["gpt-6-luna"]
expected_cost: Final = (
(_INPUT_TOKENS - _CACHED_TOKENS - _CACHE_WRITE_TOKENS) * float(luna_cost["input_cost_per_token"])
+ _CACHED_TOKENS * float(luna_cost["cache_read_input_token_cost"])
+ _CACHE_WRITE_TOKENS * float(luna_cost["cache_creation_input_token_cost"])
+ _OUTPUT_TOKENS * float(luna_cost["output_cost_per_token"])
)
expected_cost: Final = _INPUT_TOKENS * _GPT_6_LUNA_DECISIONS_INPUT_RATE
assert expected_cost > 0
assert litellm.completion_cost(completion_response=response) == pytest.approx(expected_cost)
@ -1694,6 +1781,28 @@ async def test_azure_ai_decision_posts_deployment_to_foundry_systemone_route(
assert response._hidden_params["model"] == "azure_ai/decision-1"
@pytest.mark.asyncio
async def test_azure_ai_systemone_uses_its_decisions_pricing_fields(
respx_mock: respx.MockRouter, decisions_pricing_fields: None
) -> None:
route: Final = respx_mock.post("https://res.services.ai.azure.com/providers/microsoft/v1/systemone").respond(
json={**_RESPONSE, "model": "microsoft-decision-1"}
)
response: Final = await litellm.asystemone(
model="azure_ai/Microsoft-Decision-1",
state="review",
questions={"is_defect": {"type": "noul", "instructions": "Is this a defect?"}},
api_base="https://res.services.ai.azure.com",
api_key="foundry-key",
)
assert route.called
assert response._hidden_params["provider_response_model"] == "azure_ai/Microsoft-Decision-1"
assert litellm.completion_cost(completion_response=response) == pytest.approx(
_INPUT_TOKENS * _MICROSOFT_DECISION_1_DECISIONS_INPUT_RATE
)
@pytest.mark.asyncio
async def test_azure_ai_decision_reads_foundry_env_and_keeps_gateway_path_prefix(
monkeypatch: pytest.MonkeyPatch,

122
tests/unit/litellm_core_utils/test_decisions_pricing.py Normal file
View file

@ -0,0 +1,122 @@
from typing import Final, cast
import pytest
import litellm
from litellm import cost_calculator
from litellm.litellm_core_utils.decisions_pricing import (
apply_decisions_pricing,
decisions_pricing_fields,
is_decisions_call_type,
)
from litellm.types.utils import CallTypesLiteral, PromptTokensDetailsWrapper, Usage
_ROW: Final[dict[str, object]] = {
"input_cost_per_token": 1e-07,
"input_cost_per_token_above_272k_tokens": 2e-07,
"output_cost_per_token": 5e-07,
"output_cost_per_token_decisions": 0.0,
"cache_read_input_token_cost": 1e-08,
"cache_read_input_token_cost_decisions": 1e-07,
"litellm_provider": "openai",
"mode": "chat",
}
def test_is_decisions_call_type_covers_decisions_and_systemone_only() -> None:
assert all(
is_decisions_call_type(call_type) for call_type in ("decisions", "adecisions", "systemone", "asystemone")
)
assert not any(is_decisions_call_type(call_type) for call_type in ("completion", "acompletion", "aresponses"))
def test_decisions_pricing_fields_maps_suffixed_fields_onto_plain_names() -> None:
assert decisions_pricing_fields(_ROW) == {
"output_cost_per_token": 0.0,
"cache_read_input_token_cost": 1e-07,
}
assert decisions_pricing_fields({"input_cost_per_token": 1e-07, "mode": "evaluation"}) == {}
def test_apply_decisions_pricing_overrides_plain_fields_and_keeps_the_rest() -> None:
priced: Final = apply_decisions_pricing(_ROW)
assert priced == {
"input_cost_per_token": 1e-07,
"input_cost_per_token_above_272k_tokens": 2e-07,
"output_cost_per_token": 0.0,
"cache_read_input_token_cost": 1e-07,
"litellm_provider": "openai",
"mode": "chat",
}
assert _ROW["output_cost_per_token"] == 5e-07
def test_apply_decisions_pricing_leaves_rows_without_decisions_fields_unchanged() -> None:
plain_row: Final[dict[str, object]] = {
"input_cost_per_token": 4e-08,
"output_cost_per_token": 0.0,
"mode": "evaluation",
}
assert apply_decisions_pricing(plain_row) == plain_row
_LUNA_USAGE: Final = Usage(
prompt_tokens=100,
completion_tokens=10,
total_tokens=110,
prompt_tokens_details=PromptTokensDetailsWrapper(cached_tokens=50),
)
@pytest.fixture
def luna_with_decisions_fields(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setitem(litellm.model_cost, "gpt-6-luna", {**litellm.model_cost["gpt-6-luna"], **_ROW})
@pytest.mark.parametrize("call_type", ["decisions", "adecisions", "systemone", "asystemone"])
def test_cost_per_token_bills_decisions_call_types_from_decisions_fields(
luna_with_decisions_fields: None, call_type: str
) -> None:
prompt_cost, completion_cost = cost_calculator.cost_per_token(
model="gpt-6-luna",
custom_llm_provider="openai",
usage_object=_LUNA_USAGE,
call_type=cast(CallTypesLiteral, call_type),
)
assert prompt_cost == pytest.approx(50 * 1e-07 + 50 * 1e-07)
assert completion_cost == 0.0
def test_cost_per_token_keeps_chat_on_plain_fields(luna_with_decisions_fields: None) -> None:
prompt_cost, completion_cost = cost_calculator.cost_per_token(
model="gpt-6-luna",
custom_llm_provider="openai",
usage_object=_LUNA_USAGE,
call_type="completion",
)
assert prompt_cost == pytest.approx(50 * 1e-07 + 50 * 1e-08)
assert completion_cost == pytest.approx(10 * 5e-07)
def test_cost_per_token_falls_back_to_plain_fields_without_decisions_fields(monkeypatch: pytest.MonkeyPatch) -> None:
plain_row: Final[dict[str, object]] = {
"input_cost_per_token": 4e-08,
"output_cost_per_token": 0.0,
"litellm_provider": "perplexity",
"mode": "evaluation",
}
monkeypatch.setitem(litellm.model_cost, "perplexity/pplx-decider-v1-27b", plain_row)
prompt_cost, completion_cost = cost_calculator.cost_per_token(
model="pplx-decider-v1-27b",
custom_llm_provider="perplexity",
usage_object=_LUNA_USAGE,
call_type="adecisions",
)
assert prompt_cost == pytest.approx(100 * 4e-08)
assert completion_cost == 0.0

6
tests/unit/litellm_core_utils/test_health_check_helpers.py
View file

@ -5,7 +5,7 @@ import socket
import struct
import zlib
from types import MappingProxyType
from typing import Final
from typing import Final, Literal
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@ -726,9 +726,11 @@ async def test_ahealth_check_probes_hosted_vllm_with_a_choice_question(
@pytest.mark.asyncio
@pytest.mark.parametrize("mode", ("evaluation", "decisions"))
async def test_ahealth_check_keeps_the_noul_probe_for_other_decisions_providers(
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
mode: Literal["evaluation", "decisions"],
) -> None:
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()
@ -745,7 +747,7 @@ async def test_ahealth_check_keeps_the_noul_probe_for_other_decisions_providers(
"model": "typesafe/jev-1.13",
"api_key": "sk-test",
},
mode="evaluation",
mode=mode,
)
assert "error" not in result, result

19
tests/unit/proxy/decisions_endpoints/test_endpoints.py
View file

@ -16,6 +16,7 @@ from pydantic import TypeAdapter
from starlette.routing import Match
import litellm
from litellm.litellm_core_utils.decisions_pricing import apply_decisions_pricing
from litellm.proxy._lazy_features import LAZY_FEATURES, LazyFeature, attach_lazy_features
from litellm.proxy.decisions_endpoints.endpoints import decisions, router, systemone
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import SafeRouteAdder
@ -563,6 +564,16 @@ def test_openai_format_decisions_reach_an_openai_deployment_unchanged_including_
monkeypatch: pytest.MonkeyPatch,
respx_mock: respx.MockRouter,
) -> None:
monkeypatch.setitem(
litellm.model_cost,
"gpt-6-luna",
{
**litellm.model_cost["gpt-6-luna"],
"input_cost_per_token_decisions": 1e-07,
"cache_read_input_token_cost_decisions": 1e-07,
"output_cost_per_token_decisions": 0.0,
},
)
monkeypatch.setattr(litellm, "api_base", None)
monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
monkeypatch.delenv("OPENAI_API_BASE", raising=False)
@ -614,11 +625,9 @@ def test_openai_format_decisions_reach_an_openai_deployment_unchanged_including_
body: Final = response.json()
assert body["answers"] == _OPENAI_FORMAT_ANSWERS
assert body["usage"] == upstream_usage
luna_cost: Final = litellm.model_cost["gpt-6-luna"]
expected_cost: Final = (
(_INPUT_TOKENS - cached_tokens) * float(luna_cost["input_cost_per_token"])
+ cached_tokens * float(luna_cost["cache_read_input_token_cost"])
+ _OUTPUT_TOKENS * float(luna_cost["output_cost_per_token"])
luna_decisions_cost: Final = apply_decisions_pricing(litellm.model_cost["gpt-6-luna"])
expected_cost: Final = _INPUT_TOKENS * float(luna_decisions_cost["input_cost_per_token"]) + _OUTPUT_TOKENS * float(
luna_decisions_cost["output_cost_per_token"]
)
assert expected_cost > 0
assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(expected_cost)

6
tests/unit/proxy/health_endpoints/test_health_endpoints.py
View file

@ -7,7 +7,7 @@ from contextlib import contextmanager
from datetime import datetime, timedelta
from importlib import import_module
from types import MappingProxyType, SimpleNamespace
from typing import Final
from typing import Final, Literal
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
@ -5013,7 +5013,9 @@ def test_test_model_connection_accepts_image_edit_mode(monkeypatch):
@pytest.mark.parametrize("mode", ("decisions", "evaluation"), ids=("decisions", "legacy-evaluation"))
def test_test_model_connection_accepts_decision_modes(monkeypatch, mode: str):
def test_test_model_connection_accepts_decision_modes(
monkeypatch, mode: Literal["evaluation", "decisions"]
):
monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)
litellm.in_memory_llm_clients_cache.flush_cache()

128
tests/unit/proxy/pass_through_endpoints/llm_provider_handlers/test_typesafe_passthrough_logging_handler.py
View file

@ -1,5 +1,5 @@
from datetime import datetime
from typing import Final
from typing import Final, Literal
from unittest.mock import MagicMock
import httpx
@ -72,6 +72,33 @@ def test_uses_registry_pricing_and_standard_usage():
assert response["kwargs"]["combined_usage_object"].total_tokens == 360
def test_uses_decisions_pricing_fields_when_the_row_has_them(monkeypatch: pytest.MonkeyPatch) -> None:
model: Final = "nimble-latest"
pricing: Final = {
"input_cost_per_token": 1.0,
"input_cost_per_token_decisions": 2.0,
"output_cost_per_token": 1.0,
"output_cost_per_token_decisions": 3.0,
"litellm_provider": "bespoke",
"mode": "chat",
}
monkeypatch.setitem(litellm.model_cost, f"bespoke/{model}", pricing)
result: Final = TypeSafePassthroughLoggingHandler.typesafe_passthrough_handler(
httpx_response=_response(),
response_body={"model": model, "usage": {"input_tokens": 2, "output_tokens": 1}},
logging_obj=_logging_obj(),
url_route="https://api.bespoke.example/v1/systemone",
result='{"answers": {}}',
start_time=datetime(2000, 1, 1),
end_time=datetime(2000, 1, 1),
cache_hit=False,
request_body={"model": model},
custom_llm_provider="bespoke",
)
assert result["kwargs"]["response_cost"] == pytest.approx(7.0)
def test_falls_back_to_request_model_when_response_model_is_missing():
result = _handler_result(
{"usage": {"input_tokens": 10, "output_tokens": 2}},
@ -143,68 +170,111 @@ def test_success_handler_dispatches_to_typesafe_handler():
@pytest.mark.parametrize("mode", ("decisions", "evaluation"), ids=("decisions", "legacy-evaluation"))
@pytest.mark.parametrize("guardrail_cost", [0.0, 0.25])
@pytest.mark.parametrize("metadata_slot", ["metadata", "litellm_metadata"])
@pytest.mark.parametrize("provider,requested,routing_model", [
("laya", "english", "multilingual"), ("laya", "english", None),
("bespoke", "nimble-latest", None),
("bespoke", "bespokelabs/Bespoke-Nimble-9B", None),
])
@pytest.mark.parametrize(
"provider,requested,routing_model",
[
("laya", "english", "multilingual"),
("laya", "english", None),
("bespoke", "nimble-latest", None),
("bespoke", "bespokelabs/Bespoke-Nimble-9B", None),
],
)
async def test_oss_gateway_accounts_for_checkpoint_usage_and_registered_cost(
monkeypatch: pytest.MonkeyPatch, routing_model: str | None, metadata_slot: str, guardrail_cost: float,
provider: str, requested: str, mode: str
monkeypatch: pytest.MonkeyPatch,
routing_model: str | None,
metadata_slot: str,
guardrail_cost: float,
provider: str,
requested: str,
mode: Literal["decisions", "evaluation"],
) -> None:
checkpoint: Final = routing_model or requested
model: Final = f"{provider}/{checkpoint}"
input_rate: Final = 0.002
output_rate: Final = 0.005
monkeypatch.setitem(litellm.model_cost, model, {
"input_cost_per_token": input_rate, "output_cost_per_token": output_rate,
"litellm_provider": provider, "mode": mode,
})
pricing_row: Final = {
"input_cost_per_token": input_rate,
"output_cost_per_token": output_rate,
"litellm_provider": provider,
"mode": mode,
}
monkeypatch.setitem(litellm.model_cost, model, pricing_row)
start: Final = datetime.now()
logging_obj: Final = Logging(
model=requested, messages=[], stream=False, call_type="pass_through_endpoint",
start_time=start, litellm_call_id="oss-accounting", function_id="oss-accounting", kwargs={},
model=requested,
messages=[],
stream=False,
call_type="pass_through_endpoint",
start_time=start,
litellm_call_id="oss-accounting",
function_id="oss-accounting",
kwargs={},
)
from fastapi import Request
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.pass_through_endpoints.pass_through_endpoints import HttpPassThroughEndpointHelpers
request: Final = Request({
"type": "http", "method": "POST", "path": f"/{provider}/v1/systemone",
"headers": [], "query_string": b"",
})
request: Final = Request(
{
"type": "http",
"method": "POST",
"path": f"/{provider}/v1/systemone",
"headers": [],
"query_string": b"",
}
)
auth: Final = UserAPIKeyAuth(
api_key="oss-budget-key", token="oss-budget-key",
api_key="oss-budget-key",
token="oss-budget-key",
model_max_budget={f"{provider}/{requested}": {"budget_limit": 0.01, "time_period": "1d"}},
)
request_body: Final = {"model": requested, metadata_slot: {"model_group": "unbounded-client-choice"}}
logging_kwargs: Final = HttpPassThroughEndpointHelpers._init_kwargs_for_pass_through_endpoint(
request=request, user_api_key_dict=auth, logging_obj=logging_obj,
passthrough_logging_payload={"url": f"https://{provider}.test/v1/systemone"}, _parsed_body=request_body,
request=request,
user_api_key_dict=auth,
logging_obj=logging_obj,
passthrough_logging_payload={"url": f"https://{provider}.test/v1/systemone"},
_parsed_body=request_body,
)
logging_kwargs["litellm_params"]["metadata"]["standard_logging_guardrail_information"] = [
{"guardrail_name": "trusted-hook", "guardrail_cost": guardrail_cost},
]
logging_obj.update_environment_variables(
model=requested, user="unknown", optional_params={},
litellm_params=logging_kwargs["litellm_params"], call_type="pass_through_endpoint",
model=requested,
user="unknown",
optional_params={},
litellm_params=logging_kwargs["litellm_params"],
call_type="pass_through_endpoint",
)
body: Final = {
"model": "laya-rl-agent" if provider == "laya" else requested, "usage": {"input_tokens": 10, "output_tokens": 3},
"model": "laya-rl-agent" if provider == "laya" else requested,
"usage": {"input_tokens": 10, "output_tokens": 3},
**({"routing": {"model": routing_model}} if routing_model else {}),
}
normalized: Final = PassThroughEndpointLogging().normalize_llm_passthrough_logging_payload(
httpx_response=httpx.Response(200, request=httpx.Request("POST", f"https://{provider}.test/v1/systemone"), json=body),
response_body=body, request_body={"model": requested}, logging_obj=logging_obj,
url_route=f"https://{provider}.test/v1/systemone", result="{}", start_time=start,
end_time=datetime.now(), cache_hit=False, custom_llm_provider=provider, **logging_kwargs,
httpx_response=httpx.Response(
200, request=httpx.Request("POST", f"https://{provider}.test/v1/systemone"), json=body
),
response_body=body,
request_body={"model": requested},
logging_obj=logging_obj,
url_route=f"https://{provider}.test/v1/systemone",
result="{}",
start_time=start,
end_time=datetime.now(),
cache_hit=False,
custom_llm_provider=provider,
**logging_kwargs,
)
logged: Final = normalized["kwargs"]
expected_cost: Final = 10 * input_rate + 3 * output_rate
assert (logged["model"], logged["custom_llm_provider"]) == (model, provider)
assert logged["response_cost"] == pytest.approx(expected_cost)
assert logged["combined_usage_object"].model_dump(exclude_none=True) == {
"prompt_tokens": 10, "completion_tokens": 3, "total_tokens": 13,
"prompt_tokens": 10,
"completion_tokens": 3,
"total_tokens": 13,
}
assert logging_obj.model_call_details["model"] == model
assert logging_obj.model_call_details["response_cost"] == pytest.approx(expected_cost)

42
tests/unit/test_cost_map_guard.py
View file

@ -1,8 +1,8 @@
from collections.abc import Mapping
import importlib.util
import json
import subprocess
import sys
from collections.abc import Mapping
from pathlib import Path
from types import ModuleType
from typing import Final
@ -84,30 +84,29 @@ def test_in_sync_files_pass_for_humans_and_bots() -> None:
EVALUATION_MODEL: Final = "azure_ai/Microsoft-Decision-1"
EVALUATION_OUTPUT_PRICE_FAILURE: Final = (
f"{EVALUATION_MODEL}: mode evaluation rows bill input only; output_cost_per_token must be 0 unless the vendor "
"prices output tokens for this model (add the source to EVALUATION_OUTPUT_PRICE_SOURCES)"
DECISIONS_MODEL: Final = "azure_ai/decisions/Microsoft-Decision-1"
INPUT_ONLY_OUTPUT_PRICE_FAILURE: Final = (
"mode evaluation and decisions rows bill input only; output_cost_per_token must be 0 unless the vendor prices "
"output tokens for this model (add the source to EVALUATION_OUTPUT_PRICE_SOURCES)"
)
@pytest.mark.parametrize(("model", "mode"), ((EVALUATION_MODEL, "evaluation"), (DECISIONS_MODEL, "decisions")))
@pytest.mark.parametrize("bot", (True, False))
def test_evaluation_rows_with_output_price_fail_for_humans_and_bots(bot: bool) -> None:
head: Final = _snapshot(
{**BASE_MAP, EVALUATION_MODEL: _entry(mode="evaluation", output_cost_per_token=4.2e-08)}
)
assert _failures(head, bot=bot) == (EVALUATION_OUTPUT_PRICE_FAILURE,)
def test_evaluation_and_decisions_rows_with_output_price_fail_for_humans_and_bots(
model: str, mode: str, bot: bool
) -> None:
head: Final = _snapshot({**BASE_MAP, model: _entry(mode=mode, output_cost_per_token=4.2e-08)})
assert _failures(head, bot=bot) == (f"{model}: {INPUT_ONLY_OUTPUT_PRICE_FAILURE}",)
def test_evaluation_rows_with_zero_or_missing_output_price_pass() -> None:
zero_output: Final = _snapshot(
{**BASE_MAP, EVALUATION_MODEL: _entry(mode="evaluation", output_cost_per_token=0.0)}
)
@pytest.mark.parametrize(("model", "mode"), ((EVALUATION_MODEL, "evaluation"), (DECISIONS_MODEL, "decisions")))
def test_evaluation_and_decisions_rows_with_zero_or_missing_output_price_pass(model: str, mode: str) -> None:
zero_output: Final = _snapshot({**BASE_MAP, model: _entry(mode=mode, output_cost_per_token=0.0)})
missing_output_entry: Final = {
key: value
for key, value in _entry(mode="evaluation").items()
if key != "output_cost_per_token"
key: value for key, value in _entry(mode=mode).items() if key != "output_cost_per_token"
}
missing_output: Final = _snapshot({**BASE_MAP, EVALUATION_MODEL: missing_output_entry})
missing_output: Final = _snapshot({**BASE_MAP, model: missing_output_entry})
assert _failures(zero_output) == ()
assert _failures(missing_output) == ()
@ -120,13 +119,12 @@ def test_chat_rows_with_output_price_pass() -> None:
assert _failures(head) == ()
def test_evaluation_output_price_source_allowlist_passes() -> None:
head: Final = _snapshot(
{**BASE_MAP, EVALUATION_MODEL: _entry(mode="evaluation", output_cost_per_token=4.2e-08)}
)
@pytest.mark.parametrize(("model", "mode"), ((EVALUATION_MODEL, "evaluation"), (DECISIONS_MODEL, "decisions")))
def test_evaluation_and_decisions_output_price_source_allowlist_passes(model: str, mode: str) -> None:
head: Final = _snapshot({**BASE_MAP, model: _entry(mode=mode, output_cost_per_token=4.2e-08)})
source: Final = 'https://vendor.example/pricing "Output tokens are charged per token."'
assert _failures(head, evaluation_output_price_sources={EVALUATION_MODEL: source}) == ()
assert _failures(head, evaluation_output_price_sources={model: source}) == ()
def test_bot_may_add_and_reprice_models() -> None:

8
tests/unit/test_utils.py
View file

@ -938,6 +938,14 @@ def test_aaamodel_prices_and_context_window_json_is_valid():
"cache_creation_input_token_cost_above_200k_tokens_batches": {"type": "number"},
"cache_creation_input_token_cost_above_272k_tokens_batches": {"type": "number"},
"cache_creation_input_token_cost_batches": {"type": "number"},
"cache_creation_input_token_cost_decisions": {"type": "number"},
"cache_creation_input_token_cost_above_272k_tokens_decisions": {"type": "number"},
"cache_read_input_token_cost_decisions": {"type": "number"},
"cache_read_input_token_cost_above_272k_tokens_decisions": {"type": "number"},
"input_cost_per_token_decisions": {"type": "number"},
"input_cost_per_token_above_272k_tokens_decisions": {"type": "number"},
"output_cost_per_token_decisions": {"type": "number"},
"output_cost_per_token_above_272k_tokens_decisions": {"type": "number"},
"cache_creation_input_token_cost_flex": {"type": "number"},
"cache_creation_input_token_cost_priority": {"type": "number"},
"cache_creation_input_token_cost_ultrafast": {"type": "number"},

4
ui/litellm-dashboard/src/components/add_model/AddModelForm.integration.test.tsx
View file

@ -429,7 +429,7 @@ describe("AddModelForm", () => {
expect(await screen.findByRole("button", { name: "Add Model" })).toBeInTheDocument();
});
it("offers the Evaluation decisions mode", async () => {
it("offers the Decisions mode", async () => {
const mockUseAuthorized = vi.mocked(await import("@/app/(dashboard)/hooks/useAuthorized"));
mockUseAuthorized.default.mockReturnValue(mockAuthorizedUser("proxy_admin", "user-1", true));
@ -438,7 +438,7 @@ describe("AddModelForm", () => {
await screen.findByText("Provider");
await userEvent.click(screen.getByRole("combobox", { name: "Mode" }));
expect(await screen.findByRole("option", { name: "Evaluation - /v1/decisions", exact: true })).toBeInTheDocument();
expect(await screen.findByRole("option", { name: "Decisions - /v1/decisions", exact: true })).toBeInTheDocument();
});
it("shows only the Close button in the connection test dialog footer", async () => {

2
ui/litellm-dashboard/src/components/add_model/add_model_modes.tsx
View file

@ -13,7 +13,7 @@ export const TEST_MODES = [
{ value: "batch", label: "Batch - /batch" },
{ value: "anthropic_messages", label: "Anthropic Messages - /v1/messages" },
{ value: "ocr", label: "OCR - /ocr" },
{ value: "evaluation", label: "Evaluation - /v1/decisions" },
{ value: "decisions", label: "Decisions - /v1/decisions" },
];
// Define the available auto router routing strategies