scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-19 00:01:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 88

test(e2e): make cost-calculation cases MECE by rate-key ownership with realistic fixtures

Browse source 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
kerry 2026-09-18 13:15:29 +00:00
parent aac1456e07
commit dda7776346
8 changed files with 3694 additions and 954 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
tests/e2e/CLAUDE.md
View file

@ -21,7 +21,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
- `load/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163); do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend; marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see); needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_REDIS_CHAOS` is set and excluded from the per-PR selector like the rest of `load/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in), and markerless harness unit tests for the locust, process-usage, and session-anomaly aggregation logic
- `other/` - the holding-pen suite for the `other.*` registry cluster with no home of its own yet: the master-key auth gate, JWT auth (access tokens issued by a real Keycloak realm, `idp.py` plus `idp_realm.json`, whose JWKS the proxy's `JWT_PUBLIC_KEY_URL` points at; see CONTRIBUTING.md for the start command and config block), and the process-lifecycle health probes (liveness, public readiness, authenticated readiness diagnostics). Promote a cluster out once it is large/stable enough for its own suite
- `gateway/` - proxy configuration only (`litellm-config.yml`); no tests
- `cost_calculation/` - cost accounting against a dedicated proxy whose whole model cost map is the test-owned `tests/e2e/cost_map.json` (loaded via `LITELLM_MODEL_COST_MAP_URL`), with provider calls answered by the scripted-provider sidecar in `scripted_provider.py`; every cost-map entry is a deployment and the cases plus asserted goldens are data in `cases.json` (each exact-spend case carries a literal `expected` cell per map key; `cost_matrix.matrix_data_errors()` runs at collection time so a key absent from the cost map fails the suite's collection loudly), deselected unless `E2E_COST_MAP_STACK` is set, driven by the Buildkite `e2e-cost-calculation` step in project-releaser, which runs a proxy booted from `gateway/cost_calculation_ci_config.yml`, Postgres and the scripted provider co-located with pytest in one pod and sets the opt-in
- `cost_calculation/` - cost accounting against a dedicated proxy whose whole model cost map is the test-owned `tests/e2e/cost_map.json` (loaded via `LITELLM_MODEL_COST_MAP_URL`), with provider calls answered by the scripted-provider sidecar in `scripted_provider.py`; every cost-map entry is a deployment and the cases plus asserted goldens are data in `cases.json` (each `pricing` case owns (model, cost key) pairs via `owns`/`fallback_for` so every rate key present on each map entry has exactly one owning case, and each carries a literal `expected` cell per map key; `transport` cases list `models` and exercise token counting only; `cost_matrix.matrix_data_errors()` runs at collection time so a key absent from the cost map, an unowned or double-owned (model, rate key) pair, an `owns` key absent on all of the case's models, or a `fallback_for` key present on a case model fails the suite's collection loudly), deselected unless `E2E_COST_MAP_STACK` is set, driven by the Buildkite `e2e-cost-calculation` step in project-releaser, which runs a proxy booted from `gateway/cost_calculation_ci_config.yml` with `MODEL_COST_MAP_MIN_MODEL_COUNT=1` and `MODEL_COST_MAP_MAX_SHRINK_RATIO=0` (the 21-entry test map trips the fetched-cost-map integrity check at the defaults), Postgres and the scripted provider co-located with pytest in one pod and sets the opt-in
- `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture; ships its own driver/builder/publisher plus `_*_unit_tests/` trees. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`); the CLI-driving path stays bespoke
- `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream; see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport; the Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json`

3135
tests/e2e/cost_calculation/cases.json
View file

File diff suppressed because it is too large Load diff

5
tests/e2e/cost_calculation/conftest.py
View file

@ -7,6 +7,11 @@ deployment under test, and the request shapes plus asserted goldens live in
scripted-provider sidecar (``scripted_provider.py``), registered per scenario
over its control API.
The proxy must also run with ``MODEL_COST_MAP_MIN_MODEL_COUNT=1`` and
``MODEL_COST_MAP_MAX_SHRINK_RATIO=0``: the 21-entry test map trips the
fetched-cost-map integrity check (too few models, large shrink versus the
bundled map) at those env vars' defaults.
Deselected unless E2E_COST_MAP_STACK is set (marker `cost_map_stack`).
"""

227
tests/e2e/cost_calculation/cost_matrix.py
View file

@ -13,9 +13,12 @@ Two data files drive the suite; nothing in Python lists models or cases:
from __future__ import annotations
import base64
import io
import json
import math
import random
import struct
import wave
import zlib
from collections.abc import Mapping
from dataclasses import dataclass
@ -37,22 +40,41 @@ class SearchContextCostPerQuery(BaseModel):
search_context_size_high: float | None = None
class ProviderSpecificEntry(BaseModel):
"""Provider-specific key rates, keyed by the named suffix litellm looks up
(``fast`` for Anthropic fast mode, ``us`` for US inference geography)."""
model_config = ConfigDict(frozen=True)
fast: float | None = None
us: float | None = None
class CostMapEntry(BaseModel):
"""The pricing fields of a cost-map entry the matrix reads. Shaped like a
``model_prices_and_context_window.json`` entry; unmodelled keys are ignored."""
``model_prices_and_context_window.json`` entry; the file is test-owned so
undeclared keys are forbidden rather than ignored."""
model_config = ConfigDict(frozen=True, extra="ignore")
model_config = ConfigDict(frozen=True, extra="forbid")
litellm_provider: str
mode: str
max_tokens: int | None = None
max_input_tokens: int | None = None
max_output_tokens: int | None = None
supports_function_calling: bool | None = None
input_cost_per_token: float | None = None
output_cost_per_token: float | None = None
cache_read_input_token_cost: float | None = None
cache_creation_input_token_cost: float | None = None
cache_creation_input_token_cost_above_1hr: float | None = None
cache_read_input_token_cost_above_200k_tokens: float | None = None
cache_creation_input_token_cost_above_200k_tokens: float | None = None
output_cost_per_reasoning_token: float | None = None
input_cost_per_audio_token: float | None = None
output_cost_per_audio_token: float | None = None
input_cost_per_image_token: float | None = None
input_cost_per_video_token: float | None = None
input_cost_per_token_above_200k_tokens: float | None = None
output_cost_per_token_above_200k_tokens: float | None = None
input_cost_per_token_flex: float | None = None
@ -61,6 +83,70 @@ class CostMapEntry(BaseModel):
output_cost_per_token_priority: float | None = None
search_context_cost_per_query: SearchContextCostPerQuery | None = None
web_search_billing_unit: str | None = None
google_maps_grounding_cost_per_query: float | None = None
file_search_cost_per_1k_calls: float | None = None
provider_specific_entry: ProviderSpecificEntry | None = None
_METADATA_FIELDS: Final = frozenset(
{
"litellm_provider",
"mode",
"max_tokens",
"max_input_tokens",
"max_output_tokens",
"supports_function_calling",
}
)
_CONTAINER_FIELDS: Final = frozenset({"search_context_cost_per_query", "provider_specific_entry"})
def _submodel_rate_keys(
field: str, sub: SearchContextCostPerQuery | ProviderSpecificEntry | None
) -> tuple[str, ...]:
if sub is None:
return ()
return tuple(
f"{field}.{name}"
for name in type(sub).model_fields
if getattr(sub, name) is not None
)
def _entry_rate_keys(entry: CostMapEntry) -> frozenset[str]:
"""Every cost key an entry carries, with container subfields expanded to
dotted names (``search_context_cost_per_query.search_context_size_low``).
``web_search_billing_unit`` counts as a rate key whenever present,
for both ``per_query`` and ``per_prompt`` values."""
plain: Final = frozenset(
name
for name in CostMapEntry.model_fields
if name not in _METADATA_FIELDS
and name not in _CONTAINER_FIELDS
and getattr(entry, name) is not None
)
return (
plain
| frozenset(
_submodel_rate_keys("search_context_cost_per_query", entry.search_context_cost_per_query)
)
| frozenset(_submodel_rate_keys("provider_specific_entry", entry.provider_specific_entry))
)
def _entry_has_rate_key(entry: CostMapEntry, rate_key: str) -> bool:
outer, _, inner = rate_key.partition(".")
if outer == "search_context_cost_per_query":
return f"{outer}.{inner}" in _submodel_rate_keys(outer, entry.search_context_cost_per_query)
if outer == "provider_specific_entry":
return f"{outer}.{inner}" in _submodel_rate_keys(outer, entry.provider_specific_entry)
value: Final[object] = getattr(entry, outer, None)
return value is not None
SERVICE_TIER_REQUEST_WIRES: Final = frozenset(
{"openai_chat", "azure_chat", "openai_responses", "bedrock_converse"}
)
COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, CostMapEntry])
@ -94,22 +180,42 @@ class ExpectedCell(BaseModel):
class Case(BaseModel):
"""One request/response shape from cases.json. An exact-spend case names
its models implicitly by carrying one ``expected`` golden per map key; a
recount case (``exact_spend=False``) names them in ``models`` instead."""
"""One request/response shape from cases.json.
``family`` splits the matrix: ``pricing`` cases own cost keys (``owns``,
dotted subfield names allowed) or declare which keys they deliberately
leave absent (``fallback_for``) so every cost key in the map has exactly
one owning case; ``transport`` cases exercise counting/transport only and
run wherever they list membership. An exact-spend case names its models
implicitly by carrying one ``expected`` golden per map key; a recount
case (``exact_spend=False``) names them in ``models`` instead. The
feature flags drive request realism in ``_chat_body``."""
model_config = ConfigDict(frozen=True)
name: str
family: Literal["pricing", "transport"]
usage: ScriptedUsage
usage_by_model: Mapping[str, ScriptedUsage] = Field(default_factory=lambda: MappingProxyType({}))
stream: bool = False
stream_usage: Literal["final_chunk", "absent"] = "final_chunk"
service_tier: Literal["flex", "priority"] | None = None
speed: Literal["fast"] | None = None
inference_geo: Literal["us"] | None = None
response_model_override: bool = False
exact_spend: bool = True
tool_call: bool = False
image_input: bool = False
audio_input: bool = False
audio_output: bool = False
video_input: bool = False
reasoning: bool = False
web_search: Literal["low", "medium", "high"] | None = None
google_maps: bool = False
file_search: bool = False
terminal: Literal["completed", "incomplete", "unvalidated", "prompt_blocked"] = "completed"
owns: tuple[str, ...] = ()
fallback_for: tuple[str, ...] = ()
expected: Mapping[str, ExpectedCell] = Field(default_factory=lambda: MappingProxyType({}))
models: tuple[str, ...] = ()
@ -121,11 +227,14 @@ class Case(BaseModel):
def expected_for(self, model: FrontierModel) -> ExpectedCell:
return self.expected[model.map_key]
def usage_for(self, map_key: str) -> ScriptedUsage:
return self.usage_by_model.get(map_key, self.usage)
def scenario(self, scenario_id: str, model: FrontierModel, text: str) -> Scenario:
return Scenario(
scenario_id=scenario_id,
wire=model.wire,
usage=self.usage,
usage=self.usage_for(model.map_key),
model=model.provider_model,
output=ScriptedOutput(
text=text,
@ -137,6 +246,8 @@ class Case(BaseModel):
),
stream_usage=self.stream_usage,
service_tier=self.service_tier,
speed=self.speed,
inference_geo=self.inference_geo,
)
@ -183,7 +294,7 @@ _PROVIDER_WIRING: Final[Mapping[tuple[str, str], _ProviderWiring]] = MappingProx
{
("openai", "chat"): _ProviderWiring("openai_chat", "openai", MappingProxyType({})),
("openai", "responses"): _ProviderWiring(
"openai_responses", "openai", MappingProxyType({})
"openai_responses", "openai/responses", MappingProxyType({})
),
("anthropic", "chat"): _ProviderWiring(
"anthropic_messages", "anthropic", MappingProxyType({})
@ -226,7 +337,13 @@ class FrontierModel:
@property
def override_rates(self) -> CostMapEntry:
if self.base_model is not None or self.override_map_key is None:
# bedrock_converse responses carry no model field, so a reported-model
# override can never repoint pricing there, same as a base_model pin.
if (
self.base_model is not None
or self.wire == "bedrock_converse"
or self.override_map_key is None
):
return self.rates
return COST_MAP[self.override_map_key]
@ -338,6 +455,31 @@ def _png_chunk(tag: bytes, payload: bytes) -> bytes:
return struct.pack(">I", len(payload)) + tag + payload + struct.pack(">I", zlib.crc32(tag + payload))
def audio_input_data_url() -> str:
"""A deterministic 0.5 s 16-bit PCM WAV (8 kHz, 220 Hz sine) as a data
URL, small enough to stay a fixture but real audio to the provider."""
frames: Final = b"".join(
struct.pack("<h", int(12000 * math.sin(2 * math.pi * 220 * i / 8000)))
for i in range(4000)
)
buffer: Final = io.BytesIO()
with wave.open(buffer, "wb") as wav:
wav.setnchannels(1)
wav.setsampwidth(2)
wav.setframerate(8000)
wav.writeframes(frames)
return "data:audio/wav;base64," + base64.b64encode(buffer.getvalue()).decode()
def video_input_data_url() -> str:
"""A deterministic mp4-looking blob (ftyp box plus a fixed mdat payload)
as a data URL; only the media type and bytes matter to the wire."""
ftyp: Final = struct.pack(">I4s4sI4s4s", 24, b"ftyp", b"isom", 0x200, b"isom", b"iso6")
mdat_payload: Final = bytes((i * 7 + 13) % 256 for i in range(4096))
mdat: Final = struct.pack(">I4s", 8 + len(mdat_payload), b"mdat") + mdat_payload
return "data:video/mp4;base64," + base64.b64encode(ftyp + mdat).decode()
def image_input_data_url() -> str:
"""A deterministic 256x256 RGB noise PNG as a data URL; noise compresses
poorly on purpose so the base64 payload stays well above 100 KB and would
@ -357,6 +499,8 @@ def image_input_data_url() -> str:
IMAGE_INPUT_DATA_URL: Final = image_input_data_url()
AUDIO_INPUT_DATA_URL: Final = audio_input_data_url()
VIDEO_INPUT_DATA_URL: Final = video_input_data_url()
def matrix_data_errors() -> tuple[str, ...]:
@ -381,6 +525,48 @@ def matrix_data_errors() -> tuple[str, ...]:
for case in CASES
if case.exact_spend == bool(case.models) or case.exact_spend != bool(case.expected)
)
all_pairs: Final = frozenset(
(map_key, key)
for map_key, entry in COST_MAP.items()
for key in _entry_rate_keys(entry)
)
owned_pairs: Final = tuple(
(map_key, key)
for case in CASES
if case.family == "pricing"
for map_key in case.expected
for key in case.owns
if map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key)
)
unowned_pairs: Final = sorted(
f"{map_key}:{key}" for map_key, key in all_pairs - frozenset(owned_pairs)
)
duplicate_pairs: Final = sorted(
f"{map_key}:{key}"
for map_key, key in set(owned_pairs)
if owned_pairs.count((map_key, key)) > 1
)
owns_without_holder: Final = sorted(
f"{case.name}:{key}"
for case in CASES
for key in case.owns
if not any(
map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key)
for map_key in case.expected
)
)
fallback_violations: Final = sorted(
f"{case.name}:{map_key}:{key}"
for case in CASES
for key in case.fallback_for
for map_key in (*case.expected, *case.models)
if map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key)
)
family_violations: Final = sorted(
case.name
for case in CASES
if (case.family == "transport") != (not case.owns and not case.fallback_for)
)
input_rates: Final = tuple(entry.input_cost_per_token for entry in COST_MAP.values())
findings: Final = (
(
@ -404,5 +590,30 @@ def matrix_data_errors() -> tuple[str, ...]:
if len(input_rates) != len(set(input_rates))
else None
),
(
f"(model, rate key) pairs with no owning case: {unowned_pairs}"
if unowned_pairs
else None
),
(
f"(model, rate key) pairs owned by more than one case: {duplicate_pairs}"
if duplicate_pairs
else None
),
(
f"owns keys absent on all of the case's expected models: {owns_without_holder}"
if owns_without_holder
else None
),
(
f"fallback_for keys a case's models actually carry: {fallback_violations}"
if fallback_violations
else None
),
(
f"cases with owns/fallback_for inconsistent with family: {family_violations}"
if family_violations
else None
),
)
return tuple(finding for finding in findings if finding is not None)

268
tests/e2e/cost_calculation/scripted_provider.py
View file

@ -87,6 +87,56 @@ _TERMINAL_CAPS: Final[Mapping[str, frozenset[str]]] = MappingProxyType(
)
_BASE_USAGE_FIELDS: Final = frozenset({"fresh_input_tokens", "output_tokens"})
_OPENAI_FAMILY_USAGE: Final = frozenset(
{
"cache_read_tokens",
"reasoning_tokens",
"audio_input_tokens",
"audio_output_tokens",
"web_search_calls",
}
)
_CACHE_WRITE_USAGE: Final = frozenset({"cache_write_5m_tokens", "cache_write_1h_tokens"})
_GEMINI_USAGE: Final = frozenset(
{
"cache_read_tokens",
"reasoning_tokens",
"audio_input_tokens",
"audio_output_tokens",
"image_input_tokens",
"video_input_tokens",
"web_search_calls",
"google_maps_calls",
}
)
_USAGE_CAPS: Final[Mapping[str, frozenset[str]]] = MappingProxyType(
{
wire: usage
for wire, usage in (
("openai_chat", _OPENAI_FAMILY_USAGE),
("azure_chat", _OPENAI_FAMILY_USAGE),
("together_chat", _OPENAI_FAMILY_USAGE),
("fireworks_chat", _OPENAI_FAMILY_USAGE),
(
"openai_responses",
frozenset(
{"cache_read_tokens", "reasoning_tokens", "web_search_calls", "file_search_calls"}
),
),
(
"anthropic_messages",
frozenset({"cache_read_tokens", "web_search_calls"}) | _CACHE_WRITE_USAGE,
),
("bedrock_converse", frozenset({"cache_read_tokens"}) | _CACHE_WRITE_USAGE),
("gemini_generate", _GEMINI_USAGE),
("vertex_generate", _GEMINI_USAGE),
)
}
)
class ScriptedToolCall(BaseModel):
"""A single function call the scripted output emits instead of text.
``arguments`` is the wire's JSON string (~250 chars), sliced into deltas
@ -116,7 +166,11 @@ class ScriptedUsage(BaseModel):
reasoning_tokens: int = 0
audio_input_tokens: int = 0
audio_output_tokens: int = 0
image_input_tokens: int = 0
video_input_tokens: int = 0
web_search_calls: int = 0
google_maps_calls: int = 0
file_search_calls: int = 0
class ScriptedOutput(BaseModel):
@ -151,6 +205,10 @@ class Scenario(BaseModel):
model: str
stream_usage: StreamUsage = "final_chunk"
service_tier: ServiceTier | None = None
# Anthropic fast mode and US inference geography; emitted on the anthropic
# usage object only (litellm reads them there), so they are response-side.
speed: Literal["fast"] | None = None
inference_geo: Literal["us"] | None = None
@model_validator(mode="after")
def _check_terminal_supported(self) -> Scenario:
@ -161,6 +219,20 @@ class Scenario(BaseModel):
raise ValueError(
f"wire {self.wire} cannot emit terminal={self.output.terminal}"
)
unsupported: Final = frozenset(
field
for field in self.usage.model_fields_set
if getattr(self.usage, field)
and field not in (_USAGE_CAPS.get(self.wire, frozenset()) | _BASE_USAGE_FIELDS)
)
if unsupported:
raise ValueError(
f"wire {self.wire} cannot express usage fields {sorted(unsupported)}"
)
if (self.speed or self.inference_geo) and self.wire != "anthropic_messages":
raise ValueError(
f"wire {self.wire} cannot emit speed/inference_geo (anthropic usage fields)"
)
return self
@property
@ -215,32 +287,10 @@ def _sse(events: tuple[tuple[str | None, Mapping[str, object] | str], ...]) -> b
def _openai_usage(u: ScriptedUsage) -> Mapping[str, object]:
prompt_tokens: Final = (
u.fresh_input_tokens
+ u.cache_read_tokens
+ u.cache_write_5m_tokens
+ u.cache_write_1h_tokens
+ u.audio_input_tokens
)
prompt_tokens: Final = u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens
completion_tokens: Final = u.output_tokens + u.reasoning_tokens + u.audio_output_tokens
prompt_details: Final = _jobj_opt(
("cached_tokens", u.cache_read_tokens) if u.cache_read_tokens else None,
(
("cache_write_tokens", u.cache_write_5m_tokens + u.cache_write_1h_tokens)
if u.cache_write_5m_tokens or u.cache_write_1h_tokens
else None
),
(
(
"cache_creation_token_details",
_jobj(
("ephemeral_5m_input_tokens", u.cache_write_5m_tokens),
("ephemeral_1h_input_tokens", u.cache_write_1h_tokens),
),
)
if u.cache_write_5m_tokens or u.cache_write_1h_tokens
else None
),
("audio_tokens", u.audio_input_tokens) if u.audio_input_tokens else None,
)
completion_details: Final = _jobj_opt(
@ -256,12 +306,16 @@ def _openai_usage(u: ScriptedUsage) -> Mapping[str, object]:
)
def _anthropic_usage(u: ScriptedUsage) -> Mapping[str, object]:
def _anthropic_usage(scenario: Scenario) -> Mapping[str, object]:
# Anthropic reports uncached-only input_tokens; cache reads and writes ride
# top-level fields, with the 5m/1h write split under cache_creation.
u: Final = scenario.usage
return _jobj_opt(
("input_tokens", u.fresh_input_tokens),
("output_tokens", u.output_tokens),
("service_tier", scenario.service_tier) if scenario.service_tier else None,
("speed", scenario.speed) if scenario.speed else None,
("inference_geo", scenario.inference_geo) if scenario.inference_geo else None,
("cache_read_input_tokens", u.cache_read_tokens) if u.cache_read_tokens else None,
(
("cache_creation_input_tokens", u.cache_write_5m_tokens + u.cache_write_1h_tokens)
@ -287,18 +341,24 @@ def _anthropic_usage(u: ScriptedUsage) -> Mapping[str, object]:
)
def _gemini_usage(u: ScriptedUsage) -> Mapping[str, object]:
# promptTokenCount carries the cached count inside it; TEXT modality is the
# cached-inclusive text count so litellm's implicit-caching subtraction lands
# on the fresh figure. candidatesTokenCount includes reasoning + audio.
prompt_tokens: Final = u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens
candidates: Final = u.output_tokens + u.reasoning_tokens + u.audio_output_tokens
def _gemini_usage(scenario: Scenario) -> Mapping[str, object]:
# Real generateContent accounting: promptTokenCount carries the cached count
# inside it (TEXT modality is the cached-inclusive text count so litellm's
# implicit-caching subtraction lands on the fresh figure), candidatesTokenCount
# excludes thoughts, thoughtsTokenCount reports them separately, and
# totalTokenCount sums all three. Image/video input ride promptTokensDetails.
u: Final = scenario.usage
prompt_tokens: Final = (
u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens
+ u.image_input_tokens + u.video_input_tokens
)
candidates: Final = u.output_tokens + u.audio_output_tokens
return _jobj_opt(
("promptTokenCount", prompt_tokens),
("candidatesTokenCount", candidates),
("totalTokenCount", prompt_tokens + candidates),
("cachedContentTokenCount", u.cache_read_tokens) if u.cache_read_tokens else None,
("thoughtsTokenCount", u.reasoning_tokens) if u.reasoning_tokens else None,
("totalTokenCount", prompt_tokens + candidates + u.reasoning_tokens),
("cachedContentTokenCount", u.cache_read_tokens) if u.cache_read_tokens else None,
(
"promptTokensDetails",
(
@ -308,19 +368,66 @@ def _gemini_usage(u: ScriptedUsage) -> Mapping[str, object]:
if u.audio_input_tokens
else ()
),
*(
(_jobj(("modality", "IMAGE"), ("tokenCount", u.image_input_tokens)),)
if u.image_input_tokens
else ()
),
*(
(_jobj(("modality", "VIDEO"), ("tokenCount", u.video_input_tokens)),)
if u.video_input_tokens
else ()
),
),
),
(
(
"candidatesTokensDetails",
(
_jobj(("modality", "TEXT"), ("tokenCount", u.output_tokens + u.reasoning_tokens)),
_jobj(("modality", "TEXT"), ("tokenCount", u.output_tokens)),
_jobj(("modality", "AUDIO"), ("tokenCount", u.audio_output_tokens)),
),
)
if u.audio_output_tokens
else None
),
(
(
"trafficType",
{"flex": "ON_DEMAND_FLEX", "priority": "ON_DEMAND_PRIORITY"}[
scenario.service_tier
],
)
if scenario.service_tier
else None
),
)
def _gemini_grounding_metadata(scenario: Scenario) -> Mapping[str, object] | None:
"""groundingMetadata for the search/Maps flags. Maps items carry maps
chunks and googleMapsWidgetContextToken so litellm bills them as Maps
queries, not web search."""
u: Final = scenario.usage
if not u.web_search_calls and not u.google_maps_calls:
return None
if u.google_maps_calls:
return _jobj(
(
"webSearchQueries",
tuple(f"maps query {i}" for i in range(u.google_maps_calls)),
),
(
"groundingChunks",
tuple(
_jobj(("maps", _jobj(("uri", f"https://maps.google.com/?cid={i}"))))
for i in range(u.google_maps_calls)
),
),
("googleMapsWidgetContextToken", f"token_{scenario.scenario_id}"),
)
return _jobj(
("webSearchQueries", tuple(f"query {i}" for i in range(u.web_search_calls))),
)
@ -572,7 +679,7 @@ def _anthropic_body(scenario: Scenario, requested_model: str) -> Mapping[str, ob
("model", scenario.output.response_model or requested_model),
("content", _anthropic_content(scenario)),
("stop_reason", _anthropic_stop_reason(scenario)),
("usage", _anthropic_usage(scenario.usage)),
("usage", _anthropic_usage(scenario)),
)
@ -581,7 +688,7 @@ def _anthropic_sse(scenario: Scenario, requested_model: str) -> bytes:
input_usage: Final = _jobj(
*(
(key, value)
for key, value in _anthropic_usage(scenario.usage).items()
for key, value in _anthropic_usage(scenario).items()
if key != "output_tokens"
)
)
@ -685,7 +792,7 @@ def _gemini_prompt_blocked_body(scenario: Scenario, requested_model: str) -> Map
),
),
),
("usageMetadata", _gemini_usage(scenario.usage)),
("usageMetadata", _gemini_usage(scenario)),
("modelVersion", scenario.output.response_model or requested_model),
)
@ -728,22 +835,14 @@ def _gemini_body(scenario: Scenario, requested_model: str) -> Mapping[str, objec
),
("index", 0),
(
(
"groundingMetadata",
_jobj(
(
"webSearchQueries",
tuple(f"query {i}" for i in range(scenario.usage.web_search_calls)),
)
),
)
if scenario.usage.web_search_calls
("groundingMetadata", _gemini_grounding_metadata(scenario))
if _gemini_grounding_metadata(scenario) is not None
else None
),
),
),
),
("usageMetadata", _gemini_usage(scenario.usage)),
("usageMetadata", _gemini_usage(scenario)),
("modelVersion", scenario.output.response_model or requested_model),
)
@ -762,7 +861,7 @@ def _gemini_sse(scenario: Scenario, requested_model: str) -> bytes:
None,
_jobj(
("candidates", ()),
("usageMetadata", _gemini_usage(scenario.usage)),
("usageMetadata", _gemini_usage(scenario)),
("modelVersion", scenario.output.response_model or requested_model),
),
),
@ -788,6 +887,16 @@ def _responses_output(scenario: Scenario) -> tuple[Mapping[str, object], ...]:
_jobj(("type", "web_search_call"), ("id", f"ws_{i}"), ("status", "completed"))
for i in range(scenario.usage.web_search_calls)
),
*(
_jobj(
("type", "file_search_call"),
("id", f"fs_{i}"),
("status", "completed"),
("queries", (f"query {i}",)),
("results", ()),
)
for i in range(scenario.usage.file_search_calls)
),
_jobj(
("type", "function_call"),
("id", f"fc_{scenario.scenario_id}"),
@ -853,9 +962,50 @@ def _responses_sse(scenario: Scenario, requested_model: str) -> bytes:
"response.incomplete" if scenario.output.terminal == "incomplete" else "response.completed"
)
output_index: Final = (
scenario.usage.web_search_calls + (1 if scenario.output.terminal == "unvalidated" else 0)
scenario.usage.web_search_calls
+ scenario.usage.file_search_calls
+ (1 if scenario.output.terminal == "unvalidated" else 0)
)
middle_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = (
file_search_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = tuple(
event
for i in range(scenario.usage.file_search_calls)
for event in (
(
"response.output_item.added",
_jobj(
("type", "response.output_item.added"),
("output_index", i),
(
"item",
_jobj(
("type", "file_search_call"),
("id", f"fs_{i}"),
("status", "in_progress"),
("queries", ()),
),
),
),
),
(
"response.output_item.done",
_jobj(
("type", "response.output_item.done"),
("output_index", i),
(
"item",
_jobj(
("type", "file_search_call"),
("id", f"fs_{i}"),
("status", "completed"),
("queries", (f"query {i}",)),
("results", ()),
),
),
),
),
)
)
call_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = (
(
(
"response.output_item.added",
@ -911,6 +1061,10 @@ def _responses_sse(scenario: Scenario, requested_model: str) -> bytes:
),
)
)
middle_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = (
*file_search_events,
*call_events,
)
return _sse(
(
("response.created", _jobj(("type", "response.created"), ("response", created))),
@ -976,7 +1130,7 @@ def _bedrock_content(scenario: Scenario) -> tuple[Mapping[str, object], ...]:
def _bedrock_body(scenario: Scenario) -> Mapping[str, object]:
return _jobj(
return _jobj_opt(
(
"output",
_jobj(
@ -992,6 +1146,11 @@ def _bedrock_body(scenario: Scenario) -> Mapping[str, object]:
("stopReason", _bedrock_stop_reason(scenario)),
("usage", _bedrock_usage(scenario.usage)),
("metrics", _jobj(("latencyMs", 42))),
(
("serviceTier", _jobj(("type", scenario.service_tier)))
if scenario.service_tier
else None
),
)
@ -1083,9 +1242,14 @@ def _bedrock_eventstream(scenario: Scenario) -> bytes:
(
_aws_event_frame(
"metadata",
_jobj(
_jobj_opt(
("usage", _bedrock_usage(scenario.usage)),
("metrics", _jobj(("latencyMs", 42))),
(
("serviceTier", _jobj(("type", scenario.service_tier)))
if scenario.service_tier
else None
),
),
),
)

148
tests/e2e/cost_calculation/test_token_pricing_e2e.py
View file

@ -16,8 +16,11 @@ from typing import Final
from conftest import CostCalcClient, cost_rows, register_scenario_deployment
from cost_matrix import (
AUDIO_INPUT_DATA_URL,
FRONTIER_MODELS,
IMAGE_INPUT_DATA_URL,
SERVICE_TIER_REQUEST_WIRES,
VIDEO_INPUT_DATA_URL,
Case,
FrontierModel,
cases_for,
@ -27,15 +30,27 @@ from cost_matrix import (
from e2e_config import unique_marker
from lifecycle import ResourceManager
from models import (
CacheControl,
ChatAudio,
ChatBody,
ChatMessage,
ChatStreamOptions,
ChatTool,
ChatToolFunction,
FileContentPart,
FileObject,
FileSearchTool,
GoogleMapsTool,
GoogleSearchTool,
HostedWebSearchTool,
ImageContentPart,
ImageUrl,
InputAudio,
InputAudioContentPart,
TextContentPart,
WebSearchOptions,
)
from scripted_provider import ScriptedUsage, Wire
pytestmark: Final = [pytest.mark.e2e, pytest.mark.cost_map_stack] # mutable-ok: pytest only accepts a list for pytestmark
@ -52,40 +67,131 @@ def _case_id(param: tuple[FrontierModel, Case]) -> str:
return f"{model.map_key.replace('/', '-')}-{case.name}"
def _chat_body(model_name: str, marker: str, case: Case) -> ChatBody:
return ChatBody(
model=model_name,
messages=(
ChatMessage(
role="user",
content=(
[
TextContentPart(text=f"{marker} scripted pricing call"),
ImageContentPart(image_url=ImageUrl(url=IMAGE_INPUT_DATA_URL)),
]
if case.image_input
else f"{marker} scripted pricing call"
),
),
_CACHE_WIRES: Final = frozenset({"anthropic_messages", "bedrock_converse"})
_WEB_SEARCH_OPTION_WIRES: Final = frozenset({"openai_chat", "azure_chat", "openai_responses"})
def _cache_control(usage: ScriptedUsage, wire: Wire) -> CacheControl | None:
if wire not in _CACHE_WIRES:
return None
if not (usage.cache_read_tokens or usage.cache_write_5m_tokens or usage.cache_write_1h_tokens):
return None
return CacheControl(type="ephemeral", ttl="1h" if usage.cache_write_1h_tokens else None)
def _chat_body(model: FrontierModel, case: Case, model_name: str, marker: str) -> ChatBody:
usage: Final = case.usage_for(model.map_key)
user_parts: Final = (
TextContentPart(
text=f"{marker} summarize the attached material in one line and name the city weather",
),
stream=case.stream,
stream_options=ChatStreamOptions(include_usage=True) if case.stream else None,
service_tier=case.service_tier,
tools=(
*(
(ImageContentPart(image_url=ImageUrl(url=IMAGE_INPUT_DATA_URL, detail="high")),)
if case.image_input
else ()
),
*(
(
InputAudioContentPart(
input_audio=InputAudio(data=AUDIO_INPUT_DATA_URL.split(",", 1)[1], format="wav")
),
)
if case.audio_input
else ()
),
*(
(FileContentPart(file=FileObject(file_data=VIDEO_INPUT_DATA_URL, format="mp4")),)
if case.video_input
else ()
),
)
tools: Final = (
*(
(
ChatTool(
function=ChatToolFunction(
name="get_weather",
description="Get the current weather and a short forecast for a city.",
parameters={
"type": "object",
"properties": {"city": {"type": "string"}},
"properties": {
"city": {"type": "string", "description": "City name"},
"days": {"type": "integer", "description": "Forecast horizon in days"},
"units": {"type": "string", "enum": ["metric", "imperial"]},
},
"required": ["city"],
},
)
),
)
if case.tool_call
else ()
),
*(
(HostedWebSearchTool(type="web_search_20250305", name="web_search", max_uses=5),)
if case.web_search is not None and model.wire == "anthropic_messages"
else ()
),
*(
(GoogleSearchTool(),)
if case.web_search is not None and model.wire in ("gemini_generate", "vertex_generate")
else ()
),
*((GoogleMapsTool(),) if case.google_maps else ()),
*((FileSearchTool(vector_store_ids=["vs_cost_calc_fixture"]),) if case.file_search else ()),
)
return ChatBody(
model=model_name,
messages=(
ChatMessage(
role="system",
content=[
TextContentPart(
text=(
"You are a deterministic pricing-harness assistant. "
"Keep answers to a single short line."
),
cache_control=_cache_control(usage, model.wire),
)
],
),
ChatMessage(role="user", content=list(user_parts)),
),
stream=case.stream,
stream_options=ChatStreamOptions(include_usage=True) if case.stream else None,
service_tier=(
case.service_tier
if case.service_tier is not None and model.wire in SERVICE_TIER_REQUEST_WIRES
else None
),
reasoning_effort="medium" if case.reasoning else None,
modalities=(
["text"] if case.audio_input else (["text", "audio"] if case.audio_output else None)
),
audio=(
ChatAudio(voice="alloy", format="pcm16") if case.audio_output else None
),
web_search_options=(
WebSearchOptions(search_context_size=case.web_search)
if case.web_search is not None and model.wire in _WEB_SEARCH_OPTION_WIRES
else None
),
tools=tools or None,
tool_choice="auto" if case.tool_call and model.wire != "bedrock_converse" else None,
# The test-owned cost map carries no supports_* flags, so litellm's
# optional-params gate rejects the realistic request fields; allowlist
# exactly the ones this case sends.
allowed_openai_params=[
name
for name, sent in (
("tool_choice", case.tool_call and model.wire != "bedrock_converse"),
("modalities", case.audio_input or case.audio_output),
("audio", case.audio_output),
("web_search_options", case.web_search is not None),
("reasoning_effort", case.reasoning),
)
if sent
],
)
@ -105,7 +211,7 @@ class TestTokenPricing:
response: Final = client.proxy.transport.send(
"/chat/completions",
headers=client.proxy.transport.bearer(scoped_key),
json=_chat_body(model_name, marker, case),
json=_chat_body(model, case, model_name, marker),
stream=case.stream,
)
assert response.ok, (

800
tests/e2e/cost_map.json
View file

@ -1,525 +1,411 @@
{
"anthropic.claude-sonnet-5-v1:0": {
"cache_creation_input_token_cost": 0.00051,
"cache_creation_input_token_cost_above_1hr": 0.00068,
"cache_read_input_token_cost": 1.7e-05,
"input_cost_per_token": 0.00017,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"gpt-5.6": {
"cache_read_input_token_cost": 1.75e-07,
"input_cost_per_audio_token": 4e-05,
"input_cost_per_token": 1.75e-06,
"input_cost_per_token_flex": 8.75e-07,
"input_cost_per_token_priority": 3.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 0.00034,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true
},
"azure/gpt-5.4-mini": {
"cache_creation_input_token_cost": 0.00048,
"cache_creation_input_token_cost_above_1hr": 0.00064,
"cache_read_input_token_cost": 1.6e-05,
"input_cost_per_audio_token": 0.00096,
"input_cost_per_token": 0.00016,
"input_cost_per_token_above_200k_tokens": 0.00128,
"input_cost_per_token_flex": 0.00024,
"input_cost_per_token_priority": 0.000272,
"litellm_provider": "azure",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_audio_token": 0.00112,
"output_cost_per_reasoning_token": 0.0008,
"output_cost_per_token": 0.00032,
"output_cost_per_token_above_200k_tokens": 0.00144,
"output_cost_per_token_flex": 0.0004,
"output_cost_per_token_priority": 0.000432,
"output_cost_per_audio_token": 8e-05,
"output_cost_per_reasoning_token": 1.6e-05,
"output_cost_per_token": 1.4e-05,
"output_cost_per_token_flex": 7e-06,
"output_cost_per_token_priority": 2.8e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"supports_function_calling": true
},
"gpt-5.4-mini": {
"cache_read_input_token_cost": 3.5e-08,
"input_cost_per_audio_token": 1e-05,
"input_cost_per_token": 3.5e-07,
"input_cost_per_token_flex": 1.75e-07,
"input_cost_per_token_priority": 7e-07,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 2e-05,
"output_cost_per_reasoning_token": 3.2e-06,
"output_cost_per_token": 2.8e-06,
"output_cost_per_token_flex": 1.4e-06,
"output_cost_per_token_priority": 5.6e-06,
"search_context_cost_per_query": {
"search_context_size_low": 0.01,
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true
},
"azure/gpt-5.6": {
"cache_creation_input_token_cost": 0.00045,
"cache_creation_input_token_cost_above_1hr": 0.0006,
"cache_read_input_token_cost": 1.5e-05,
"input_cost_per_audio_token": 0.0009,
"input_cost_per_token": 0.00015,
"input_cost_per_token_above_200k_tokens": 0.0012,
"input_cost_per_token_flex": 0.000225,
"input_cost_per_token_priority": 0.000255,
"cache_read_input_token_cost": 1.8e-07,
"input_cost_per_audio_token": 4.1e-05,
"input_cost_per_token": 1.8e-06,
"input_cost_per_token_flex": 9e-07,
"input_cost_per_token_priority": 3.6e-06,
"litellm_provider": "azure",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00105,
"output_cost_per_reasoning_token": 0.00075,
"output_cost_per_token": 0.0003,
"output_cost_per_token_above_200k_tokens": 0.00135,
"output_cost_per_token_flex": 0.000375,
"output_cost_per_token_priority": 0.000405,
"output_cost_per_audio_token": 8.2e-05,
"output_cost_per_reasoning_token": 1.65e-05,
"output_cost_per_token": 1.44e-05,
"output_cost_per_token_flex": 7.2e-06,
"output_cost_per_token_priority": 2.88e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"supports_function_calling": true
},
"claude-haiku-4-5": {
"cache_creation_input_token_cost": 0.00021,
"cache_creation_input_token_cost_above_1hr": 0.00028000000000000003,
"cache_read_input_token_cost": 7e-06,
"input_cost_per_token": 7.000000000000001e-05,
"litellm_provider": "anthropic",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"azure/gpt-5.4-mini": {
"cache_read_input_token_cost": 3.6e-08,
"input_cost_per_audio_token": 1.05e-05,
"input_cost_per_token": 3.6e-07,
"input_cost_per_token_flex": 1.8e-07,
"input_cost_per_token_priority": 7.2e-07,
"litellm_provider": "azure",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 0.00014000000000000001,
"output_cost_per_audio_token": 2.1e-05,
"output_cost_per_reasoning_token": 3.3e-06,
"output_cost_per_token": 2.88e-06,
"output_cost_per_token_flex": 1.44e-06,
"output_cost_per_token_priority": 5.76e-06,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"supports_function_calling": true
},
"gpt-5.3-codex": {
"cache_read_input_token_cost": 1.5e-07,
"file_search_cost_per_1k_calls": 0.0025,
"input_cost_per_token": 1.5e-06,
"input_cost_per_token_flex": 7.5e-07,
"input_cost_per_token_priority": 3e-06,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "responses",
"output_cost_per_reasoning_token": 1.3e-05,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_flex": 6e-06,
"output_cost_per_token_priority": 2.4e-05,
"search_context_cost_per_query": {
"search_context_size_low": 0.01,
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true
},
"gpt-5.5-pro": {
"cache_read_input_token_cost": 1.5e-06,
"file_search_cost_per_1k_calls": 0.0025,
"input_cost_per_token": 1.5e-05,
"input_cost_per_token_flex": 7.5e-06,
"input_cost_per_token_priority": 3e-05,
"litellm_provider": "openai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "responses",
"output_cost_per_reasoning_token": 0.00013,
"output_cost_per_token": 0.00012,
"output_cost_per_token_flex": 6e-05,
"output_cost_per_token_priority": 0.00024,
"search_context_cost_per_query": {
"search_context_size_low": 0.01,
"search_context_size_medium": 0.0125,
"search_context_size_high": 0.015
},
"supports_function_calling": true
},
"claude-opus-5": {
"cache_creation_input_token_cost": 0.00015000000000000001,
"cache_creation_input_token_cost_above_1hr": 0.0002,
"cache_read_input_token_cost": 4.9999999999999996e-06,
"input_cost_per_token": 5e-05,
"cache_creation_input_token_cost": 6.25e-06,
"cache_creation_input_token_cost_above_1hr": 1e-05,
"cache_creation_input_token_cost_above_200k_tokens": 1.25e-05,
"cache_read_input_token_cost": 5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1e-06,
"input_cost_per_token": 5e-06,
"input_cost_per_token_above_200k_tokens": 1e-05,
"input_cost_per_token_priority": 6.25e-06,
"litellm_provider": "anthropic",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 0.0001,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"output_cost_per_token": 2.5e-05,
"output_cost_per_token_above_200k_tokens": 3.75e-05,
"output_cost_per_token_priority": 3.125e-05,
"provider_specific_entry": {
"fast": 6.0,
"us": 1.1
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"search_context_cost_per_query": {
"search_context_size_medium": 0.01
},
"supports_function_calling": true
},
"claude-sonnet-5": {
"cache_creation_input_token_cost": 0.00018,
"cache_creation_input_token_cost_above_1hr": 0.00024000000000000003,
"cache_read_input_token_cost": 6e-06,
"input_cost_per_token": 6.000000000000001e-05,
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_creation_input_token_cost_above_200k_tokens": 7.5e-06,
"cache_read_input_token_cost": 3e-07,
"cache_read_input_token_cost_above_200k_tokens": 6e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"input_cost_per_token_priority": 3.75e-06,
"litellm_provider": "anthropic",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 0.00012000000000000002,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"output_cost_per_token": 1.5e-05,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"output_cost_per_token_priority": 1.875e-05,
"provider_specific_entry": {
"us": 1.1
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"search_context_cost_per_query": {
"search_context_size_medium": 0.01
},
"supports_function_calling": true
},
"fireworks_ai/deepseek-v4p1-flash": {
"cache_creation_input_token_cost": 0.00033,
"cache_creation_input_token_cost_above_1hr": 0.00044,
"cache_read_input_token_cost": 1.4e-05,
"input_cost_per_audio_token": 0.00066,
"input_cost_per_token": 0.00014000000000000001,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"claude-haiku-4-5": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_creation_input_token_cost_above_1hr": 2e-06,
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"input_cost_per_token_priority": 1.25e-06,
"litellm_provider": "anthropic",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00077,
"output_cost_per_reasoning_token": 0.00055,
"output_cost_per_token": 0.00028000000000000003,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"output_cost_per_token": 5e-06,
"output_cost_per_token_priority": 6.25e-06,
"provider_specific_entry": {
"us": 1.1
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"search_context_cost_per_query": {
"search_context_size_medium": 0.01
},
"supports_function_calling": true
},
"fireworks_ai/kimi-k3": {
"cache_creation_input_token_cost": 0.00033,
"cache_creation_input_token_cost_above_1hr": 0.00044,
"cache_read_input_token_cost": 1.2e-05,
"input_cost_per_audio_token": 0.00066,
"input_cost_per_token": 0.00012000000000000002,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"us.anthropic.claude-opus-5-v1:0": {
"cache_creation_input_token_cost": 6.875e-06,
"cache_creation_input_token_cost_above_1hr": 1.1e-05,
"cache_creation_input_token_cost_above_200k_tokens": 1.375e-05,
"cache_read_input_token_cost": 5.5e-07,
"cache_read_input_token_cost_above_200k_tokens": 1.1e-06,
"input_cost_per_token": 5.5e-06,
"input_cost_per_token_above_200k_tokens": 1.1e-05,
"input_cost_per_token_flex": 2.75e-06,
"input_cost_per_token_priority": 6.875e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00077,
"output_cost_per_reasoning_token": 0.00055,
"output_cost_per_token": 0.00024000000000000003,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"output_cost_per_token": 2.75e-05,
"output_cost_per_token_above_200k_tokens": 4.125e-05,
"output_cost_per_token_flex": 1.375e-05,
"output_cost_per_token_priority": 3.4375e-05,
"supports_function_calling": true
},
"fireworks_ai/qwen3p8-max": {
"cache_creation_input_token_cost": 0.00033,
"cache_creation_input_token_cost_above_1hr": 0.00044,
"cache_read_input_token_cost": 1.3e-05,
"input_cost_per_audio_token": 0.00066,
"input_cost_per_token": 0.00013000000000000002,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"anthropic.claude-sonnet-5-v1:0": {
"cache_creation_input_token_cost": 4.125e-06,
"cache_creation_input_token_cost_above_1hr": 6.6e-06,
"cache_read_input_token_cost": 3.3e-07,
"input_cost_per_token": 3.3e-06,
"input_cost_per_token_flex": 1.65e-06,
"input_cost_per_token_priority": 4.125e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00077,
"output_cost_per_reasoning_token": 0.00055,
"output_cost_per_token": 0.00026000000000000003,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"output_cost_per_token": 1.65e-05,
"output_cost_per_token_flex": 8.25e-06,
"output_cost_per_token_priority": 2.0625e-05,
"supports_function_calling": true
},
"gemini-3.1-pro-preview": {
"cache_read_input_token_cost": 2.1e-05,
"input_cost_per_audio_token": 0.00126,
"input_cost_per_token": 0.00021,
"input_cost_per_token_above_200k_tokens": 0.00168,
"input_cost_per_token_flex": 0.000315,
"input_cost_per_token_priority": 0.000357,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"meta.llama4-maverick-17b-instruct-v1:0": {
"input_cost_per_token": 2.4e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00147,
"output_cost_per_reasoning_token": 0.00105,
"output_cost_per_token": 0.00042,
"output_cost_per_token_above_200k_tokens": 0.00189,
"output_cost_per_token_flex": 0.000525,
"output_cost_per_token_priority": 0.000567,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true,
"web_search_billing_unit": "per_query"
"output_cost_per_token": 9.7e-07,
"supports_function_calling": true
},
"gemini-3.8-flash": {
"cache_read_input_token_cost": 2e-05,
"input_cost_per_audio_token": 0.0012,
"input_cost_per_token": 0.0002,
"input_cost_per_token_above_200k_tokens": 0.0016,
"input_cost_per_token_flex": 0.0003,
"input_cost_per_token_priority": 0.00034,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_audio_token": 0.0014,
"output_cost_per_reasoning_token": 0.001,
"output_cost_per_token": 0.0004,
"output_cost_per_token_above_200k_tokens": 0.0018,
"output_cost_per_token_flex": 0.0005,
"output_cost_per_token_priority": 0.00054,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true,
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.1-pro-preview": {
"cache_read_input_token_cost": 9e-06,
"input_cost_per_audio_token": 0.00054,
"input_cost_per_token": 9e-05,
"input_cost_per_token_above_200k_tokens": 0.00072,
"input_cost_per_token_flex": 0.000135,
"input_cost_per_token_priority": 0.000153,
"gemini/gemini-3.1-pro": {
"cache_read_input_token_cost": 2e-07,
"cache_read_input_token_cost_above_200k_tokens": 4e-07,
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token": 2.6e-06,
"input_cost_per_image_token": 2.2e-06,
"input_cost_per_token": 2e-06,
"input_cost_per_token_above_200k_tokens": 4e-06,
"input_cost_per_token_flex": 1e-06,
"input_cost_per_token_priority": 2.5e-06,
"input_cost_per_video_token": 2.4e-06,
"litellm_provider": "gemini",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.0006299999999999999,
"output_cost_per_reasoning_token": 0.00045000000000000004,
"output_cost_per_token": 0.00018,
"output_cost_per_token_above_200k_tokens": 0.0008100000000000001,
"output_cost_per_token_flex": 0.00022500000000000002,
"output_cost_per_token_priority": 0.000243,
"output_cost_per_reasoning_token": 1.3e-05,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_above_200k_tokens": 1.8e-05,
"output_cost_per_token_flex": 6e-06,
"output_cost_per_token_priority": 1.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.035
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true,
"web_search_billing_unit": "per_query"
},
"gemini/gemini-3.8-flash": {
"cache_read_input_token_cost": 8e-06,
"input_cost_per_audio_token": 0.00048,
"input_cost_per_token": 8e-05,
"input_cost_per_token_above_200k_tokens": 0.00064,
"input_cost_per_token_flex": 0.00012,
"input_cost_per_token_priority": 0.000136,
"cache_read_input_token_cost": 5e-08,
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token": 1e-06,
"input_cost_per_image_token": 5.5e-07,
"input_cost_per_token": 5e-07,
"input_cost_per_token_flex": 2.5e-07,
"input_cost_per_token_priority": 6.25e-07,
"input_cost_per_video_token": 6e-07,
"litellm_provider": "gemini",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00056,
"output_cost_per_reasoning_token": 0.0004,
"output_cost_per_token": 0.00016,
"output_cost_per_token_above_200k_tokens": 0.00072,
"output_cost_per_token_flex": 0.0002,
"output_cost_per_token_priority": 0.000216,
"output_cost_per_audio_token": 6e-06,
"output_cost_per_reasoning_token": 3.5e-06,
"output_cost_per_token": 3e-06,
"output_cost_per_token_flex": 1.5e-06,
"output_cost_per_token_priority": 3.75e-06,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.035
},
"supports_function_calling": true,
"web_search_billing_unit": "per_prompt"
},
"gemini-3.1-pro": {
"cache_read_input_token_cost": 2.1e-07,
"cache_read_input_token_cost_above_200k_tokens": 4.2e-07,
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token": 2.7e-06,
"input_cost_per_image_token": 2.3e-06,
"input_cost_per_token": 2.1e-06,
"input_cost_per_token_above_200k_tokens": 4.2e-06,
"input_cost_per_token_flex": 1.05e-06,
"input_cost_per_token_priority": 2.625e-06,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_reasoning_token": 1.35e-05,
"output_cost_per_token": 1.26e-05,
"output_cost_per_token_above_200k_tokens": 1.89e-05,
"output_cost_per_token_flex": 6.3e-06,
"output_cost_per_token_priority": 1.575e-05,
"search_context_cost_per_query": {
"search_context_size_medium": 0.035
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true,
"web_search_billing_unit": "per_query"
},
"gpt-5.3-codex": {
"cache_read_input_token_cost": 3e-06,
"input_cost_per_token": 3.0000000000000004e-05,
"input_cost_per_token_above_200k_tokens": 0.00024000000000000003,
"input_cost_per_token_flex": 4.5e-05,
"input_cost_per_token_priority": 5.1e-05,
"litellm_provider": "openai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"output_cost_per_reasoning_token": 0.00015000000000000001,
"output_cost_per_token": 6.000000000000001e-05,
"output_cost_per_token_above_200k_tokens": 0.00027,
"output_cost_per_token_flex": 7.500000000000001e-05,
"output_cost_per_token_priority": 8.099999999999999e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
},
"gpt-5.4-mini": {
"cache_creation_input_token_cost": 0.00012,
"cache_creation_input_token_cost_above_1hr": 0.00016,
"cache_read_input_token_cost": 4e-06,
"input_cost_per_audio_token": 0.00024,
"input_cost_per_token": 4e-05,
"input_cost_per_token_above_200k_tokens": 0.00032,
"input_cost_per_token_flex": 6e-05,
"input_cost_per_token_priority": 6.8e-05,
"litellm_provider": "openai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"gemini-3.8-flash": {
"cache_read_input_token_cost": 5.2e-08,
"google_maps_grounding_cost_per_query": 0.025,
"input_cost_per_audio_token": 1.04e-06,
"input_cost_per_token": 5.2e-07,
"input_cost_per_token_flex": 2.6e-07,
"input_cost_per_token_priority": 6.5e-07,
"input_cost_per_video_token": 6.2e-07,
"litellm_provider": "vertex_ai-language-models",
"max_input_tokens": 1048576,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00028,
"output_cost_per_reasoning_token": 0.0002,
"output_cost_per_token": 8e-05,
"output_cost_per_token_above_200k_tokens": 0.00036,
"output_cost_per_token_flex": 0.0001,
"output_cost_per_token_priority": 0.000108,
"output_cost_per_audio_token": 6.24e-06,
"output_cost_per_token": 3.12e-06,
"output_cost_per_token_flex": 1.56e-06,
"output_cost_per_token_priority": 3.9e-06,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
"search_context_size_medium": 0.035
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
},
"gpt-5.5-pro": {
"cache_read_input_token_cost": 2e-06,
"input_cost_per_token": 2e-05,
"input_cost_per_token_above_200k_tokens": 0.00016,
"input_cost_per_token_flex": 3e-05,
"input_cost_per_token_priority": 3.4e-05,
"litellm_provider": "openai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "responses",
"output_cost_per_reasoning_token": 0.0001,
"output_cost_per_token": 4e-05,
"output_cost_per_token_above_200k_tokens": 0.00018,
"output_cost_per_token_flex": 5e-05,
"output_cost_per_token_priority": 5.4e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
},
"gpt-5.6": {
"cache_creation_input_token_cost": 3e-05,
"cache_creation_input_token_cost_above_1hr": 4e-05,
"cache_read_input_token_cost": 1e-06,
"input_cost_per_audio_token": 6e-05,
"input_cost_per_token": 1e-05,
"input_cost_per_token_above_200k_tokens": 8e-05,
"input_cost_per_token_flex": 1.5e-05,
"input_cost_per_token_priority": 1.7e-05,
"litellm_provider": "openai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_audio_token": 7e-05,
"output_cost_per_reasoning_token": 5e-05,
"output_cost_per_token": 2e-05,
"output_cost_per_token_above_200k_tokens": 9e-05,
"output_cost_per_token_flex": 2.5e-05,
"output_cost_per_token_priority": 2.7e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
},
"meta.llama4-maverick-17b-instruct-v1:0": {
"input_cost_per_token": 0.00019,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 0.00038,
"supports_function_calling": true
"web_search_billing_unit": "per_prompt"
},
"together_ai/moonshotai/Kimi-K3": {
"cache_creation_input_token_cost": 0.00030000000000000003,
"cache_creation_input_token_cost_above_1hr": 0.0004,
"cache_read_input_token_cost": 9.999999999999999e-06,
"input_cost_per_audio_token": 0.0006000000000000001,
"input_cost_per_token": 0.0001,
"input_cost_per_token_above_200k_tokens": 0.0008,
"input_cost_per_token_flex": 0.00015000000000000001,
"input_cost_per_token_priority": 0.00017,
"input_cost_per_token": 1.15e-06,
"litellm_provider": "together_ai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.0006999999999999999,
"output_cost_per_reasoning_token": 0.0005,
"output_cost_per_token": 0.0002,
"output_cost_per_token_above_200k_tokens": 0.0009000000000000001,
"output_cost_per_token_flex": 0.00025,
"output_cost_per_token_priority": 0.00027,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"output_cost_per_token": 3.45e-06,
"supports_function_calling": true
},
"together_ai/zai-org/GLM-5.3": {
"cache_creation_input_token_cost": 0.00033,
"cache_creation_input_token_cost_above_1hr": 0.00044,
"cache_read_input_token_cost": 1.1e-05,
"input_cost_per_audio_token": 0.00066,
"input_cost_per_token": 0.00011,
"input_cost_per_token_above_200k_tokens": 0.00088,
"input_cost_per_token_flex": 0.000165,
"input_cost_per_token_priority": 0.000187,
"input_cost_per_token": 5.5e-07,
"litellm_provider": "together_ai",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_audio_token": 0.00077,
"output_cost_per_reasoning_token": 0.00055,
"output_cost_per_token": 0.00022,
"output_cost_per_token_above_200k_tokens": 0.00099,
"output_cost_per_token_flex": 0.000275,
"output_cost_per_token_priority": 0.000297,
"search_context_cost_per_query": {
"search_context_size_high": 0.03,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.02
},
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_web_search": true
"output_cost_per_token": 2.2e-06,
"supports_function_calling": true
},
"us.anthropic.claude-opus-5-v1:0": {
"cache_creation_input_token_cost": 0.00054,
"cache_creation_input_token_cost_above_1hr": 0.00072,
"cache_read_input_token_cost": 1.8e-05,
"input_cost_per_token": 0.00018,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 2000000,
"max_output_tokens": 128000,
"max_tokens": 128000,
"fireworks_ai/accounts/fireworks/models/kimi-k3": {
"cache_read_input_token_cost": 6e-08,
"input_cost_per_token": 6e-07,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 0.00036,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true
"output_cost_per_token": 2.5e-06,
"supports_function_calling": true
},
"fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 6e-07,
"supports_function_calling": true
},
"fireworks_ai/accounts/fireworks/models/qwen3p8-max": {
"cache_read_input_token_cost": 9e-08,
"input_cost_per_token": 9e-07,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 400000,
"max_output_tokens": 65536,
"max_tokens": 65536,
"mode": "chat",
"output_cost_per_token": 3.6e-06,
"supports_function_calling": true
}
}

63
tests/e2e/models.py
View file

@ -186,6 +186,18 @@ class ChatMetadata(BaseModel):
class ImageUrl(BaseModel):
url: str
detail: str | None = None
class InputAudio(BaseModel):
data: str
format: str
class FileObject(BaseModel):
file_data: str | None = None
file_id: str | None = None
format: str | None = None
class TextContentPart(BaseModel):
@ -199,7 +211,17 @@ class ImageContentPart(BaseModel):
image_url: ImageUrl
ContentPart = TextContentPart | ImageContentPart
class InputAudioContentPart(BaseModel):
type: str = "input_audio"
input_audio: InputAudio
class FileContentPart(BaseModel):
type: str = "file"
file: FileObject
ContentPart = TextContentPart | ImageContentPart | InputAudioContentPart | FileContentPart
class ChatMessage(BaseModel):
@ -284,6 +306,37 @@ class ChatToolResultTurn(BaseModel):
type ChatTurn = ChatMessage | ChatAssistantTurn | ChatToolResultTurn
class HostedWebSearchTool(BaseModel):
"""A provider-hosted web-search tool sent inside an OpenAI tools list
(Anthropic's ``web_search_20250305`` shape)."""
type: str
name: str
max_uses: int | None = None
class GoogleSearchTool(BaseModel):
googleSearch: dict[str, object] = {}
class GoogleMapsTool(BaseModel):
googleMaps: dict[str, object] = {}
class FileSearchTool(BaseModel):
type: Literal["file_search"] = "file_search"
vector_store_ids: list[str]
class WebSearchOptions(BaseModel):
search_context_size: Literal["low", "medium", "high"] | None = None
class ChatAudio(BaseModel):
voice: str
format: str
class ChatStreamOptions(BaseModel):
include_usage: bool
@ -302,10 +355,16 @@ class ChatBody(BaseModel):
thinking: ThinkingParam | None = None
service_tier: str | None = None
prompt_cache_key: str | None = None
tools: Sequence[ChatTool | McpChatTool] | None = None
tools: Sequence[
ChatTool | McpChatTool | HostedWebSearchTool | GoogleSearchTool | GoogleMapsTool | FileSearchTool
] | None = None
tool_choice: str | None = None
modalities: list[str] | None = None
audio: ChatAudio | None = None
web_search_options: WebSearchOptions | None = None
guardrails: list[str] | None = None
response_format: dict[str, object] | None = None
allowed_openai_params: list[str] | None = None
chat_template_kwargs: dict[str, bool] | None = None
cache: dict[str, bool] | None = {"no-cache": True}