diff --git a/tests/e2e/CLAUDE.md b/tests/e2e/CLAUDE.md index a3e5696ef9d..54c143c11d9 100644 --- a/tests/e2e/CLAUDE.md +++ b/tests/e2e/CLAUDE.md @@ -21,7 +21,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family - `load/` - performance-category tests, kept OUT of the main suite: throughput/load SLO tests are a different testing category from functional e2e (variance-driven, historically flaky) and live outside this suite until re-implemented as their own pipeline (LIT-5163); do not add a live load test that runs in the default collection. What lives here: the weekly session-anomaly test (`test_weekly_session_anomaly_e2e.py`, Claude Code-shaped multi-turn sessions against real providers with ceilings on error rate, cache read/write, turn time, and spend; marked `weekly` and deselected unless `E2E_WEEKLY_ANOMALY` is set, driven by `.github/workflows/weekly_load_anomaly.yml`), the Redis chaos test (`test_redis_chaos_e2e.py`, locust load against mock deployments split round robin over `/chat/completions` and `/v1/messages`, one endpoint per simulated user, with `CLIENT PAUSE ALL` on the proxy's Redis mid-run to simulate it being down outright, asserting zero failed requests on every endpoint, budgeting RSS and CPU-per-request as ratios against the same run's healthy phase, and holding p50/p90/p99 latency and log-bytes-per-request to flat ceilings (a ratio cannot bound those two: an open breaker skips Redis instead of waiting on it, so the chaos phase can measure cheaper than baseline while still being far slower than a user should see); needs a proxy booted from `gateway/redis_chaos_ci_config.yml` on the same host with `E2E_PROXY_PID` and `E2E_PROXY_LOG` set, marked `redis_chaos`, deselected unless `E2E_REDIS_CHAOS` is set and excluded from the per-PR selector like the rest of `load/`, driven by `.github/workflows/test-e2e-redis-chaos.yml` and by the Buildkite `e2e-redis-chaos` step in project-releaser, which runs the proxy, Postgres and Valkey co-located with pytest in one pod and sets the opt-in), and markerless harness unit tests for the locust, process-usage, and session-anomaly aggregation logic - `other/` - the holding-pen suite for the `other.*` registry cluster with no home of its own yet: the master-key auth gate, JWT auth (access tokens issued by a real Keycloak realm, `idp.py` plus `idp_realm.json`, whose JWKS the proxy's `JWT_PUBLIC_KEY_URL` points at; see CONTRIBUTING.md for the start command and config block), and the process-lifecycle health probes (liveness, public readiness, authenticated readiness diagnostics). Promote a cluster out once it is large/stable enough for its own suite - `gateway/` - proxy configuration only (`litellm-config.yml`); no tests -- `cost_calculation/` - cost accounting against a dedicated proxy whose whole model cost map is the test-owned `tests/e2e/cost_map.json` (loaded via `LITELLM_MODEL_COST_MAP_URL`), with provider calls answered by the scripted-provider sidecar in `scripted_provider.py`; every cost-map entry is a deployment and the cases plus asserted goldens are data in `cases.json` (each exact-spend case carries a literal `expected` cell per map key; `cost_matrix.matrix_data_errors()` runs at collection time so a key absent from the cost map fails the suite's collection loudly), deselected unless `E2E_COST_MAP_STACK` is set, driven by the Buildkite `e2e-cost-calculation` step in project-releaser, which runs a proxy booted from `gateway/cost_calculation_ci_config.yml`, Postgres and the scripted provider co-located with pytest in one pod and sets the opt-in +- `cost_calculation/` - cost accounting against a dedicated proxy whose whole model cost map is the test-owned `tests/e2e/cost_map.json` (loaded via `LITELLM_MODEL_COST_MAP_URL`), with provider calls answered by the scripted-provider sidecar in `scripted_provider.py`; every cost-map entry is a deployment and the cases plus asserted goldens are data in `cases.json` (each `pricing` case owns (model, cost key) pairs via `owns`/`fallback_for` so every rate key present on each map entry has exactly one owning case, and each carries a literal `expected` cell per map key; `transport` cases list `models` and exercise token counting only; `cost_matrix.matrix_data_errors()` runs at collection time so a key absent from the cost map, an unowned or double-owned (model, rate key) pair, an `owns` key absent on all of the case's models, or a `fallback_for` key present on a case model fails the suite's collection loudly), deselected unless `E2E_COST_MAP_STACK` is set, driven by the Buildkite `e2e-cost-calculation` step in project-releaser, which runs a proxy booted from `gateway/cost_calculation_ci_config.yml` with `MODEL_COST_MAP_MIN_MODEL_COUNT=1` and `MODEL_COST_MAP_MAX_SHRINK_RATIO=0` (the 21-entry test map trips the fetched-cost-map integrity check at the defaults), Postgres and the scripted provider co-located with pytest in one pod and sets the opt-in - `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture; ships its own driver/builder/publisher plus `_*_unit_tests/` trees. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`); the CLI-driving path stays bespoke - `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream; see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport; the Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json` diff --git a/tests/e2e/cost_calculation/cases.json b/tests/e2e/cost_calculation/cases.json index cda7bc6e67a..d2cdd40aa94 100644 --- a/tests/e2e/cost_calculation/cases.json +++ b/tests/e2e/cost_calculation/cases.json @@ -1,468 +1,1837 @@ { "deployments": [ - {"map_key": "azure/gpt-5.4-mini", "litellm_model": "azure/cc-pinned-deployment", "base_model": "azure/gpt-5.4-mini"} + { + "map_key": "azure/gpt-5.4-mini", + "litellm_model": "azure/cc-pinned-deployment", + "base_model": "azure/gpt-5.4-mini" + } ], "cases": [ { - "name": "basic", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "name": "input_text", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "owns": [ + "input_cost_per_token", + "output_cost_per_token" + ], + "fallback_for": [], "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.034, "input_cost": 0.0204, "output_cost": 0.0136, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.4-mini": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.03, "input_cost": 0.018, "output_cost": 0.012, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-haiku-4-5": {"spend": 0.014, "input_cost": 0.0084, "output_cost": 0.0056, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-opus-5": {"spend": 0.01, "input_cost": 0.006, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-sonnet-5": {"spend": 0.012, "input_cost": 0.0072, "output_cost": 0.0048, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.028, "input_cost": 0.0168, "output_cost": 0.0112, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/kimi-k3": {"spend": 0.024, "input_cost": 0.0144, "output_cost": 0.0096, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/qwen3p8-max": {"spend": 0.026, "input_cost": 0.0156, "output_cost": 0.0104, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.042, "input_cost": 0.0252, "output_cost": 0.0168, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.04, "input_cost": 0.024, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.018, "input_cost": 0.0108, "output_cost": 0.0072, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.016, "input_cost": 0.0096, "output_cost": 0.0064, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.008, "input_cost": 0.0048, "output_cost": 0.0032, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.002, "input_cost": 0.0012, "output_cost": 0.0008, "prompt_tokens": 120, "completion_tokens": 40}, - "meta.llama4-maverick-17b-instruct-v1:0": {"spend": 0.038, "input_cost": 0.0228, "output_cost": 0.0152, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.02, "input_cost": 0.012, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.022, "input_cost": 0.0132, "output_cost": 0.0088, "prompt_tokens": 120, "completion_tokens": 40}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.036, "input_cost": 0.0216, "output_cost": 0.0144, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.6": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0092448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { "name": "cache_read", - "usage": {"fresh_input_tokens": 100, "cache_read_tokens": 50, "output_tokens": 30}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 640, + "cache_read_tokens": 12288, + "output_tokens": 380 + }, + "owns": [ + "cache_read_input_token_cost" + ], + "fallback_for": [], "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.02805, "input_cost": 0.01785, "output_cost": 0.0102, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.4-mini": {"spend": 0.0264, "input_cost": 0.0168, "output_cost": 0.0096, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.6": {"spend": 0.02475, "input_cost": 0.01575, "output_cost": 0.009, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-haiku-4-5": {"spend": 0.01155, "input_cost": 0.00735, "output_cost": 0.0042, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-opus-5": {"spend": 0.00825, "input_cost": 0.00525, "output_cost": 0.003, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-sonnet-5": {"spend": 0.0099, "input_cost": 0.0063, "output_cost": 0.0036, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0231, "input_cost": 0.0147, "output_cost": 0.0084, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/kimi-k3": {"spend": 0.0198, "input_cost": 0.0126, "output_cost": 0.0072, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/qwen3p8-max": {"spend": 0.02145, "input_cost": 0.01365, "output_cost": 0.0078, "prompt_tokens": 150, "completion_tokens": 30}, - "gemini-3.1-pro-preview": {"spend": 0.03465, "input_cost": 0.02205, "output_cost": 0.0126, "prompt_tokens": 150, "completion_tokens": 30}, - "gemini-3.8-flash": {"spend": 0.033, "input_cost": 0.021, "output_cost": 0.012, "prompt_tokens": 150, "completion_tokens": 30}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.01485, "input_cost": 0.00945, "output_cost": 0.0054, "prompt_tokens": 150, "completion_tokens": 30}, - "gemini/gemini-3.8-flash": {"spend": 0.0132, "input_cost": 0.0084, "output_cost": 0.0048, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.3-codex": {"spend": 0.00495, "input_cost": 0.00315, "output_cost": 0.0018, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.4-mini": {"spend": 0.0066, "input_cost": 0.0042, "output_cost": 0.0024, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.5-pro": {"spend": 0.0033, "input_cost": 0.0021, "output_cost": 0.0012, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.6": {"spend": 0.00165, "input_cost": 0.00105, "output_cost": 0.0006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.0165, "input_cost": 0.0105, "output_cost": 0.006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.01815, "input_cost": 0.01155, "output_cost": 0.0066, "prompt_tokens": 150, "completion_tokens": 30}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.0297, "input_cost": 0.0189, "output_cost": 0.0108, "prompt_tokens": 150, "completion_tokens": 30} + "gpt-5.6": { + "spend": 0.0085904, + "input_cost": 0.0032704, + "output_cost": 0.00532, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gpt-5.4-mini": { + "spend": 0.00171808, + "input_cost": 0.00065408, + "output_cost": 0.001064, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "azure/gpt-5.6": { + "spend": 0.00883584, + "input_cost": 0.00336384, + "output_cost": 0.005472, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "azure/gpt-5.4-mini": { + "spend": 0.001767168, + "input_cost": 0.000672768, + "output_cost": 0.0010944, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gpt-5.3-codex": { + "spend": 0.0073632, + "input_cost": 0.0028032, + "output_cost": 0.00456, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gpt-5.5-pro": { + "spend": 0.073632, + "input_cost": 0.028032, + "output_cost": 0.0456, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "claude-opus-5": { + "spend": 0.018844, + "input_cost": 0.009344, + "output_cost": 0.0095, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "claude-sonnet-5": { + "spend": 0.0113064, + "input_cost": 0.0056064, + "output_cost": 0.0057, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "claude-haiku-4-5": { + "spend": 0.0037688, + "input_cost": 0.0018688, + "output_cost": 0.0019, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.0207284, + "input_cost": 0.0102784, + "output_cost": 0.01045, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01243704, + "input_cost": 0.00616704, + "output_cost": 0.00627, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.0082976, + "input_cost": 0.0037376, + "output_cost": 0.00456, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.0020744, + "input_cost": 0.0009344, + "output_cost": 0.00114, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gemini-3.1-pro": { + "spend": 0.00871248, + "input_cost": 0.00392448, + "output_cost": 0.004788, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "gemini-3.8-flash": { + "spend": 0.002157376, + "input_cost": 0.000971776, + "output_cost": 0.0011856, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.00207128, + "input_cost": 0.00112128, + "output_cost": 0.00095, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.00304992, + "input_cost": 0.00168192, + "output_cost": 0.001368, + "prompt_tokens": 12928, + "completion_tokens": 380 + } } }, { "name": "cache_write_5m", - "usage": {"fresh_input_tokens": 90, "cache_write_5m_tokens": 60, "output_tokens": 30}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 512, + "cache_write_5m_tokens": 9216, + "output_tokens": 350 + }, + "owns": [ + "cache_creation_input_token_cost" + ], + "fallback_for": [], "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.0561, "input_cost": 0.0459, "output_cost": 0.0102, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.4-mini": {"spend": 0.0528, "input_cost": 0.0432, "output_cost": 0.0096, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.6": {"spend": 0.0495, "input_cost": 0.0405, "output_cost": 0.009, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-haiku-4-5": {"spend": 0.0231, "input_cost": 0.0189, "output_cost": 0.0042, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-opus-5": {"spend": 0.0165, "input_cost": 0.0135, "output_cost": 0.003, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-sonnet-5": {"spend": 0.0198, "input_cost": 0.0162, "output_cost": 0.0036, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0408, "input_cost": 0.0324, "output_cost": 0.0084, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/kimi-k3": {"spend": 0.0378, "input_cost": 0.0306, "output_cost": 0.0072, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/qwen3p8-max": {"spend": 0.0393, "input_cost": 0.0315, "output_cost": 0.0078, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.4-mini": {"spend": 0.0132, "input_cost": 0.0108, "output_cost": 0.0024, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.6": {"spend": 0.0033, "input_cost": 0.0027, "output_cost": 0.0006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.033, "input_cost": 0.027, "output_cost": 0.006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0363, "input_cost": 0.0297, "output_cost": 0.0066, "prompt_tokens": 150, "completion_tokens": 30}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.0594, "input_cost": 0.0486, "output_cost": 0.0108, "prompt_tokens": 150, "completion_tokens": 30} + "claude-opus-5": { + "spend": 0.06891, + "input_cost": 0.06016, + "output_cost": 0.00875, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "claude-sonnet-5": { + "spend": 0.041346, + "input_cost": 0.036096, + "output_cost": 0.00525, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "claude-haiku-4-5": { + "spend": 0.013782, + "input_cost": 0.012032, + "output_cost": 0.00175, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.075801, + "input_cost": 0.066176, + "output_cost": 0.009625, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.0454806, + "input_cost": 0.0397056, + "output_cost": 0.005775, + "prompt_tokens": 9728, + "completion_tokens": 350 + } } }, { "name": "cache_write_1h", - "usage": {"fresh_input_tokens": 90, "cache_write_5m_tokens": 20, "cache_write_1h_tokens": 40, "output_tokens": 30}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 512, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 7168, + "output_tokens": 350 + }, + "owns": [ + "cache_creation_input_token_cost_above_1hr" + ], + "fallback_for": [], "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.0629, "input_cost": 0.0527, "output_cost": 0.0102, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.4-mini": {"spend": 0.0592, "input_cost": 0.0496, "output_cost": 0.0096, "prompt_tokens": 150, "completion_tokens": 30}, - "azure/gpt-5.6": {"spend": 0.0555, "input_cost": 0.0465, "output_cost": 0.009, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-haiku-4-5": {"spend": 0.0259, "input_cost": 0.0217, "output_cost": 0.0042, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-opus-5": {"spend": 0.0185, "input_cost": 0.0155, "output_cost": 0.003, "prompt_tokens": 150, "completion_tokens": 30}, - "claude-sonnet-5": {"spend": 0.0222, "input_cost": 0.0186, "output_cost": 0.0036, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0452, "input_cost": 0.0368, "output_cost": 0.0084, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/kimi-k3": {"spend": 0.0422, "input_cost": 0.035, "output_cost": 0.0072, "prompt_tokens": 150, "completion_tokens": 30}, - "fireworks_ai/qwen3p8-max": {"spend": 0.0437, "input_cost": 0.0359, "output_cost": 0.0078, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.4-mini": {"spend": 0.0148, "input_cost": 0.0124, "output_cost": 0.0024, "prompt_tokens": 150, "completion_tokens": 30}, - "gpt-5.6": {"spend": 0.0037, "input_cost": 0.0031, "output_cost": 0.0006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.037, "input_cost": 0.031, "output_cost": 0.006, "prompt_tokens": 150, "completion_tokens": 30}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0407, "input_cost": 0.0341, "output_cost": 0.0066, "prompt_tokens": 150, "completion_tokens": 30}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.0666, "input_cost": 0.0558, "output_cost": 0.0108, "prompt_tokens": 150, "completion_tokens": 30} + "claude-opus-5": { + "spend": 0.09579, + "input_cost": 0.08704, + "output_cost": 0.00875, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "claude-sonnet-5": { + "spend": 0.057474, + "input_cost": 0.052224, + "output_cost": 0.00525, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "claude-haiku-4-5": { + "spend": 0.019158, + "input_cost": 0.017408, + "output_cost": 0.00175, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.105369, + "input_cost": 0.095744, + "output_cost": 0.009625, + "prompt_tokens": 9728, + "completion_tokens": 350 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.0632214, + "input_cost": 0.0574464, + "output_cost": 0.005775, + "prompt_tokens": 9728, + "completion_tokens": 350 + } + } + }, + { + "name": "audio_input", + "family": "pricing", + "usage": { + "fresh_input_tokens": 96, + "audio_input_tokens": 1450, + "output_tokens": 210 + }, + "owns": [ + "input_cost_per_audio_token" + ], + "fallback_for": [], + "audio_input": true, + "expected": { + "gpt-5.6": { + "spend": 0.061108, + "input_cost": 0.058168, + "output_cost": 0.00294, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "gpt-5.4-mini": { + "spend": 0.0151216, + "input_cost": 0.0145336, + "output_cost": 0.000588, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "azure/gpt-5.6": { + "spend": 0.0626468, + "input_cost": 0.0596228, + "output_cost": 0.003024, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "azure/gpt-5.4-mini": { + "spend": 0.01586436, + "input_cost": 0.01525956, + "output_cost": 0.0006048, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.006482, + "input_cost": 0.003962, + "output_cost": 0.00252, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002128, + "input_cost": 0.001498, + "output_cost": 0.00063, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "gemini-3.1-pro": { + "spend": 0.0067626, + "input_cost": 0.0041166, + "output_cost": 0.002646, + "prompt_tokens": 1546, + "completion_tokens": 210 + }, + "gemini-3.8-flash": { + "spend": 0.00221312, + "input_cost": 0.00155792, + "output_cost": 0.0006552, + "prompt_tokens": 1546, + "completion_tokens": 210 + } + } + }, + { + "name": "audio_output", + "family": "pricing", + "usage": { + "fresh_input_tokens": 220, + "output_tokens": 180, + "audio_output_tokens": 1120 + }, + "owns": [ + "output_cost_per_audio_token" + ], + "fallback_for": [], + "audio_output": true, + "expected": { + "gpt-5.6": { + "spend": 0.092505, + "input_cost": 0.000385, + "output_cost": 0.09212, + "prompt_tokens": 220, + "completion_tokens": 1300 + }, + "gpt-5.4-mini": { + "spend": 0.022981, + "input_cost": 7.7e-05, + "output_cost": 0.022904, + "prompt_tokens": 220, + "completion_tokens": 1300 + }, + "azure/gpt-5.6": { + "spend": 0.094828, + "input_cost": 0.000396, + "output_cost": 0.094432, + "prompt_tokens": 220, + "completion_tokens": 1300 + }, + "azure/gpt-5.4-mini": { + "spend": 0.0241176, + "input_cost": 7.92e-05, + "output_cost": 0.0240384, + "prompt_tokens": 220, + "completion_tokens": 1300 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.00737, + "input_cost": 0.00011, + "output_cost": 0.00726, + "prompt_tokens": 220, + "completion_tokens": 1300 + }, + "gemini-3.8-flash": { + "spend": 0.0076648, + "input_cost": 0.0001144, + "output_cost": 0.0075504, + "prompt_tokens": 220, + "completion_tokens": 1300 + } + } + }, + { + "name": "image_input", + "family": "pricing", + "usage": { + "fresh_input_tokens": 310, + "image_input_tokens": 1806, + "output_tokens": 240 + }, + "owns": [ + "input_cost_per_image_token" + ], + "fallback_for": [], + "image_input": true, + "expected": { + "gemini/gemini-3.1-pro": { + "spend": 0.0074732, + "input_cost": 0.0045932, + "output_cost": 0.00288, + "prompt_tokens": 2116, + "completion_tokens": 240 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.0018683, + "input_cost": 0.0011483, + "output_cost": 0.00072, + "prompt_tokens": 2116, + "completion_tokens": 240 + }, + "gemini-3.1-pro": { + "spend": 0.0078288, + "input_cost": 0.0048048, + "output_cost": 0.003024, + "prompt_tokens": 2116, + "completion_tokens": 240 + } + } + }, + { + "name": "video_input", + "family": "pricing", + "usage": { + "fresh_input_tokens": 140, + "video_input_tokens": 7920, + "output_tokens": 300 + }, + "owns": [ + "input_cost_per_video_token" + ], + "fallback_for": [], + "video_input": true, + "expected": { + "gemini/gemini-3.1-pro": { + "spend": 0.022888, + "input_cost": 0.019288, + "output_cost": 0.0036, + "prompt_tokens": 8060, + "completion_tokens": 300 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.005722, + "input_cost": 0.004822, + "output_cost": 0.0009, + "prompt_tokens": 8060, + "completion_tokens": 300 + }, + "gemini-3.8-flash": { + "spend": 0.0059192, + "input_cost": 0.0049832, + "output_cost": 0.000936, + "prompt_tokens": 8060, + "completion_tokens": 300 + } } }, { "name": "reasoning", - "usage": {"fresh_input_tokens": 100, "output_tokens": 30, "reasoning_tokens": 70}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 1240, + "output_tokens": 560, + "reasoning_tokens": 3480 + }, + "owns": [ + "output_cost_per_reasoning_token" + ], + "fallback_for": [], + "reasoning": true, "expected": { - "azure/gpt-5.4-mini": {"spend": 0.0816, "input_cost": 0.016, "output_cost": 0.0656, "prompt_tokens": 100, "completion_tokens": 100}, - "azure/gpt-5.6": {"spend": 0.0765, "input_cost": 0.015, "output_cost": 0.0615, "prompt_tokens": 100, "completion_tokens": 100}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0609, "input_cost": 0.014, "output_cost": 0.0469, "prompt_tokens": 100, "completion_tokens": 100}, - "fireworks_ai/kimi-k3": {"spend": 0.0577, "input_cost": 0.012, "output_cost": 0.0457, "prompt_tokens": 100, "completion_tokens": 100}, - "fireworks_ai/qwen3p8-max": {"spend": 0.0593, "input_cost": 0.013, "output_cost": 0.0463, "prompt_tokens": 100, "completion_tokens": 100}, - "gemini-3.1-pro-preview": {"spend": 0.1071, "input_cost": 0.021, "output_cost": 0.0861, "prompt_tokens": 100, "completion_tokens": 100}, - "gemini-3.8-flash": {"spend": 0.102, "input_cost": 0.02, "output_cost": 0.082, "prompt_tokens": 100, "completion_tokens": 100}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.0459, "input_cost": 0.009, "output_cost": 0.0369, "prompt_tokens": 100, "completion_tokens": 100}, - "gemini/gemini-3.8-flash": {"spend": 0.0408, "input_cost": 0.008, "output_cost": 0.0328, "prompt_tokens": 100, "completion_tokens": 100}, - "gpt-5.3-codex": {"spend": 0.0153, "input_cost": 0.003, "output_cost": 0.0123, "prompt_tokens": 100, "completion_tokens": 100}, - "gpt-5.4-mini": {"spend": 0.0204, "input_cost": 0.004, "output_cost": 0.0164, "prompt_tokens": 100, "completion_tokens": 100}, - "gpt-5.5-pro": {"spend": 0.0102, "input_cost": 0.002, "output_cost": 0.0082, "prompt_tokens": 100, "completion_tokens": 100}, - "gpt-5.6": {"spend": 0.0051, "input_cost": 0.001, "output_cost": 0.0041, "prompt_tokens": 100, "completion_tokens": 100}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.051, "input_cost": 0.01, "output_cost": 0.041, "prompt_tokens": 100, "completion_tokens": 100}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0561, "input_cost": 0.011, "output_cost": 0.0451, "prompt_tokens": 100, "completion_tokens": 100} + "gpt-5.6": { + "spend": 0.06569, + "input_cost": 0.00217, + "output_cost": 0.06352, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gpt-5.4-mini": { + "spend": 0.013138, + "input_cost": 0.000434, + "output_cost": 0.012704, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "azure/gpt-5.6": { + "spend": 0.067716, + "input_cost": 0.002232, + "output_cost": 0.065484, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "azure/gpt-5.4-mini": { + "spend": 0.0135432, + "input_cost": 0.0004464, + "output_cost": 0.0130968, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gpt-5.3-codex": { + "spend": 0.05382, + "input_cost": 0.00186, + "output_cost": 0.05196, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gpt-5.5-pro": { + "spend": 0.5382, + "input_cost": 0.0186, + "output_cost": 0.5196, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.05444, + "input_cost": 0.00248, + "output_cost": 0.05196, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.01448, + "input_cost": 0.00062, + "output_cost": 0.01386, + "prompt_tokens": 1240, + "completion_tokens": 4040 + }, + "gemini-3.1-pro": { + "spend": 0.05664, + "input_cost": 0.002604, + "output_cost": 0.054036, + "prompt_tokens": 1240, + "completion_tokens": 4040 + } } }, { - "name": "audio", - "usage": {"fresh_input_tokens": 100, "audio_input_tokens": 25, "output_tokens": 30, "audio_output_tokens": 15}, + "name": "tiered_input_above_200k", + "family": "pricing", + "usage": { + "fresh_input_tokens": 204800, + "output_tokens": 620 + }, + "owns": [ + "input_cost_per_token_above_200k_tokens", + "output_cost_per_token_above_200k_tokens" + ], + "fallback_for": [], "expected": { - "azure/gpt-5.4-mini": {"spend": 0.0664, "input_cost": 0.04, "output_cost": 0.0264, "prompt_tokens": 125, "completion_tokens": 45}, - "azure/gpt-5.6": {"spend": 0.06225, "input_cost": 0.0375, "output_cost": 0.02475, "prompt_tokens": 125, "completion_tokens": 45}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.05045, "input_cost": 0.0305, "output_cost": 0.01995, "prompt_tokens": 125, "completion_tokens": 45}, - "fireworks_ai/kimi-k3": {"spend": 0.04725, "input_cost": 0.0285, "output_cost": 0.01875, "prompt_tokens": 125, "completion_tokens": 45}, - "fireworks_ai/qwen3p8-max": {"spend": 0.04885, "input_cost": 0.0295, "output_cost": 0.01935, "prompt_tokens": 125, "completion_tokens": 45}, - "gemini-3.1-pro-preview": {"spend": 0.08715, "input_cost": 0.0525, "output_cost": 0.03465, "prompt_tokens": 125, "completion_tokens": 45}, - "gemini-3.8-flash": {"spend": 0.083, "input_cost": 0.05, "output_cost": 0.033, "prompt_tokens": 125, "completion_tokens": 45}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.03735, "input_cost": 0.0225, "output_cost": 0.01485, "prompt_tokens": 125, "completion_tokens": 45}, - "gemini/gemini-3.8-flash": {"spend": 0.0332, "input_cost": 0.02, "output_cost": 0.0132, "prompt_tokens": 125, "completion_tokens": 45}, - "gpt-5.4-mini": {"spend": 0.0166, "input_cost": 0.01, "output_cost": 0.0066, "prompt_tokens": 125, "completion_tokens": 45}, - "gpt-5.6": {"spend": 0.00415, "input_cost": 0.0025, "output_cost": 0.00165, "prompt_tokens": 125, "completion_tokens": 45}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.0415, "input_cost": 0.025, "output_cost": 0.0165, "prompt_tokens": 125, "completion_tokens": 45}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.04565, "input_cost": 0.0275, "output_cost": 0.01815, "prompt_tokens": 125, "completion_tokens": 45} + "claude-opus-5": { + "spend": 2.07125, + "input_cost": 2.048, + "output_cost": 0.02325, + "prompt_tokens": 204800, + "completion_tokens": 620 + }, + "claude-sonnet-5": { + "spend": 1.24275, + "input_cost": 1.2288, + "output_cost": 0.01395, + "prompt_tokens": 204800, + "completion_tokens": 620 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 2.278375, + "input_cost": 2.2528, + "output_cost": 0.025575, + "prompt_tokens": 204800, + "completion_tokens": 620 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.83036, + "input_cost": 0.8192, + "output_cost": 0.01116, + "prompt_tokens": 204800, + "completion_tokens": 620 + }, + "gemini-3.1-pro": { + "spend": 0.871878, + "input_cost": 0.86016, + "output_cost": 0.011718, + "prompt_tokens": 204800, + "completion_tokens": 620 + } } }, { - "name": "tiered", - "usage": {"fresh_input_tokens": 200001, "output_tokens": 30}, + "name": "tiered_cache_read_above_200k", + "family": "pricing", + "usage": { + "fresh_input_tokens": 4096, + "cache_read_tokens": 201728, + "output_tokens": 480 + }, + "owns": [ + "cache_read_input_token_cost_above_200k_tokens" + ], + "fallback_for": [], "expected": { - "azure/gpt-5.4-mini": {"spend": 256.04448, "input_cost": 256.00128, "output_cost": 0.0432, "prompt_tokens": 200001, "completion_tokens": 30}, - "azure/gpt-5.6": {"spend": 240.0417, "input_cost": 240.0012, "output_cost": 0.0405, "prompt_tokens": 200001, "completion_tokens": 30}, - "gemini-3.1-pro-preview": {"spend": 336.05838, "input_cost": 336.00168, "output_cost": 0.0567, "prompt_tokens": 200001, "completion_tokens": 30}, - "gemini-3.8-flash": {"spend": 320.0556, "input_cost": 320.0016, "output_cost": 0.054, "prompt_tokens": 200001, "completion_tokens": 30}, - "gemini/gemini-3.1-pro-preview": {"spend": 144.02502, "input_cost": 144.00072, "output_cost": 0.0243, "prompt_tokens": 200001, "completion_tokens": 30}, - "gemini/gemini-3.8-flash": {"spend": 128.02224, "input_cost": 128.00064, "output_cost": 0.0216, "prompt_tokens": 200001, "completion_tokens": 30}, - "gpt-5.3-codex": {"spend": 48.00834, "input_cost": 48.00024, "output_cost": 0.0081, "prompt_tokens": 200001, "completion_tokens": 30}, - "gpt-5.4-mini": {"spend": 64.01112, "input_cost": 64.00032, "output_cost": 0.0108, "prompt_tokens": 200001, "completion_tokens": 30}, - "gpt-5.5-pro": {"spend": 32.00556, "input_cost": 32.00016, "output_cost": 0.0054, "prompt_tokens": 200001, "completion_tokens": 30}, - "gpt-5.6": {"spend": 16.00278, "input_cost": 16.00008, "output_cost": 0.0027, "prompt_tokens": 200001, "completion_tokens": 30}, - "together_ai/moonshotai/Kimi-K3": {"spend": 160.0278, "input_cost": 160.0008, "output_cost": 0.027, "prompt_tokens": 200001, "completion_tokens": 30}, - "together_ai/zai-org/GLM-5.3": {"spend": 176.03058, "input_cost": 176.00088, "output_cost": 0.0297, "prompt_tokens": 200001, "completion_tokens": 30} + "claude-opus-5": { + "spend": 0.260688, + "input_cost": 0.242688, + "output_cost": 0.018, + "prompt_tokens": 205824, + "completion_tokens": 480 + }, + "claude-sonnet-5": { + "spend": 0.1564128, + "input_cost": 0.1456128, + "output_cost": 0.0108, + "prompt_tokens": 205824, + "completion_tokens": 480 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.2867568, + "input_cost": 0.2669568, + "output_cost": 0.0198, + "prompt_tokens": 205824, + "completion_tokens": 480 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.1057152, + "input_cost": 0.0970752, + "output_cost": 0.00864, + "prompt_tokens": 205824, + "completion_tokens": 480 + }, + "gemini-3.1-pro": { + "spend": 0.11100096, + "input_cost": 0.10192896, + "output_cost": 0.009072, + "prompt_tokens": 205824, + "completion_tokens": 480 + } + } + }, + { + "name": "tiered_cache_write_above_200k", + "family": "pricing", + "usage": { + "fresh_input_tokens": 4096, + "cache_write_5m_tokens": 200704, + "output_tokens": 480 + }, + "owns": [ + "cache_creation_input_token_cost_above_200k_tokens" + ], + "fallback_for": [], + "expected": { + "claude-opus-5": { + "spend": 2.56776, + "input_cost": 2.54976, + "output_cost": 0.018, + "prompt_tokens": 204800, + "completion_tokens": 480 + }, + "claude-sonnet-5": { + "spend": 1.540656, + "input_cost": 1.529856, + "output_cost": 0.0108, + "prompt_tokens": 204800, + "completion_tokens": 480 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 2.824536, + "input_cost": 2.804736, + "output_cost": 0.0198, + "prompt_tokens": 204800, + "completion_tokens": 480 + } } }, { "name": "service_tier_flex", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "owns": [ + "input_cost_per_token_flex", + "output_cost_per_token_flex" + ], + "fallback_for": [], "service_tier": "flex", "expected": { - "azure/gpt-5.4-mini": {"spend": 0.0448, "input_cost": 0.0288, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.042, "input_cost": 0.027, "output_cost": 0.015, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.0588, "input_cost": 0.0378, "output_cost": 0.021, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.056, "input_cost": 0.036, "output_cost": 0.02, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.0252, "input_cost": 0.0162, "output_cost": 0.009, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.0224, "input_cost": 0.0144, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.0084, "input_cost": 0.0054, "output_cost": 0.003, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.0112, "input_cost": 0.0072, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.0056, "input_cost": 0.0036, "output_cost": 0.002, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.0028, "input_cost": 0.0018, "output_cost": 0.001, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.028, "input_cost": 0.018, "output_cost": 0.01, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0308, "input_cost": 0.0198, "output_cost": 0.011, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.6": { + "spend": 0.004494, + "input_cost": 0.00161, + "output_cost": 0.002884, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0008988, + "input_cost": 0.000322, + "output_cost": 0.0005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0046224, + "input_cost": 0.001656, + "output_cost": 0.0029664, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00092448, + "input_cost": 0.0003312, + "output_cost": 0.00059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.003852, + "input_cost": 0.00138, + "output_cost": 0.002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.03852, + "input_cost": 0.0138, + "output_cost": 0.02472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.010725, + "input_cost": 0.00506, + "output_cost": 0.005665, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.006435, + "input_cost": 0.003036, + "output_cost": 0.003399, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.004312, + "input_cost": 0.00184, + "output_cost": 0.002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.001078, + "input_cost": 0.00046, + "output_cost": 0.000618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0045276, + "input_cost": 0.001932, + "output_cost": 0.0025956, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.00112112, + "input_cost": 0.0004784, + "output_cost": 0.00064272, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { "name": "service_tier_priority", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "owns": [ + "input_cost_per_token_priority", + "output_cost_per_token_priority" + ], + "fallback_for": [], "service_tier": "priority", "expected": { - "azure/gpt-5.4-mini": {"spend": 0.04992, "input_cost": 0.03264, "output_cost": 0.01728, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.0468, "input_cost": 0.0306, "output_cost": 0.0162, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.06552, "input_cost": 0.04284, "output_cost": 0.02268, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.0624, "input_cost": 0.0408, "output_cost": 0.0216, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.02808, "input_cost": 0.01836, "output_cost": 0.00972, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.02496, "input_cost": 0.01632, "output_cost": 0.00864, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.00936, "input_cost": 0.00612, "output_cost": 0.00324, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.01248, "input_cost": 0.00816, "output_cost": 0.00432, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.00624, "input_cost": 0.00408, "output_cost": 0.00216, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.00312, "input_cost": 0.00204, "output_cost": 0.00108, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.0312, "input_cost": 0.0204, "output_cost": 0.0108, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.03432, "input_cost": 0.02244, "output_cost": 0.01188, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.6": { + "spend": 0.017976, + "input_cost": 0.00644, + "output_cost": 0.011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0035952, + "input_cost": 0.001288, + "output_cost": 0.0023072, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0184896, + "input_cost": 0.006624, + "output_cost": 0.0118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00369792, + "input_cost": 0.0013248, + "output_cost": 0.00237312, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.015408, + "input_cost": 0.00552, + "output_cost": 0.009888, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.15408, + "input_cost": 0.0552, + "output_cost": 0.09888, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.024375, + "input_cost": 0.0115, + "output_cost": 0.012875, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.014625, + "input_cost": 0.0069, + "output_cost": 0.007725, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.004875, + "input_cost": 0.0023, + "output_cost": 0.002575, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.0268125, + "input_cost": 0.01265, + "output_cost": 0.0141625, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.0160875, + "input_cost": 0.00759, + "output_cost": 0.0084975, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.01078, + "input_cost": 0.0046, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002695, + "input_cost": 0.00115, + "output_cost": 0.001545, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.011319, + "input_cost": 0.00483, + "output_cost": 0.006489, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.0028028, + "input_cost": 0.001196, + "output_cost": 0.0016068, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { - "name": "web_search", - "usage": {"fresh_input_tokens": 100, "output_tokens": 30, "web_search_calls": 3}, + "name": "anthropic_fast_mode", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "owns": [ + "provider_specific_entry.fast" + ], + "fallback_for": [], + "speed": "fast", "expected": { - "claude-haiku-4-5": {"spend": 0.0712, "input_cost": 0.007, "output_cost": 0.0042, "prompt_tokens": 100, "completion_tokens": 30}, - "claude-opus-5": {"spend": 0.068, "input_cost": 0.005, "output_cost": 0.003, "prompt_tokens": 100, "completion_tokens": 30}, - "claude-sonnet-5": {"spend": 0.0696, "input_cost": 0.006, "output_cost": 0.0036, "prompt_tokens": 100, "completion_tokens": 30}, - "gemini-3.1-pro-preview": {"spend": 0.0936, "input_cost": 0.021, "output_cost": 0.0126, "prompt_tokens": 100, "completion_tokens": 30}, - "gemini-3.8-flash": {"spend": 0.092, "input_cost": 0.02, "output_cost": 0.012, "prompt_tokens": 100, "completion_tokens": 30}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.0744, "input_cost": 0.009, "output_cost": 0.0054, "prompt_tokens": 100, "completion_tokens": 30}, - "gemini/gemini-3.8-flash": {"spend": 0.0728, "input_cost": 0.008, "output_cost": 0.0048, "prompt_tokens": 100, "completion_tokens": 30}, - "gpt-5.3-codex": {"spend": 0.0648, "input_cost": 0.003, "output_cost": 0.0018, "prompt_tokens": 100, "completion_tokens": 30}, - "gpt-5.5-pro": {"spend": 0.0632, "input_cost": 0.002, "output_cost": 0.0012, "prompt_tokens": 100, "completion_tokens": 30} + "claude-opus-5": { + "spend": 0.117, + "input_cost": 0.0552, + "output_cost": 0.0618, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { - "name": "web_search_single", - "usage": {"fresh_input_tokens": 100, "output_tokens": 30, "web_search_calls": 1}, + "name": "anthropic_us_inference", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "owns": [ + "provider_specific_entry.us" + ], + "fallback_for": [], + "inference_geo": "us", "expected": { - "azure/gpt-5.4-mini": {"spend": 0.0456, "input_cost": 0.016, "output_cost": 0.0096, "prompt_tokens": 100, "completion_tokens": 30}, - "azure/gpt-5.6": {"spend": 0.044, "input_cost": 0.015, "output_cost": 0.009, "prompt_tokens": 100, "completion_tokens": 30}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0424, "input_cost": 0.014, "output_cost": 0.0084, "prompt_tokens": 100, "completion_tokens": 30}, - "fireworks_ai/kimi-k3": {"spend": 0.0392, "input_cost": 0.012, "output_cost": 0.0072, "prompt_tokens": 100, "completion_tokens": 30}, - "fireworks_ai/qwen3p8-max": {"spend": 0.0408, "input_cost": 0.013, "output_cost": 0.0078, "prompt_tokens": 100, "completion_tokens": 30}, - "gpt-5.4-mini": {"spend": 0.0264, "input_cost": 0.004, "output_cost": 0.0024, "prompt_tokens": 100, "completion_tokens": 30}, - "gpt-5.6": {"spend": 0.0216, "input_cost": 0.001, "output_cost": 0.0006, "prompt_tokens": 100, "completion_tokens": 30}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.036, "input_cost": 0.01, "output_cost": 0.006, "prompt_tokens": 100, "completion_tokens": 30}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0376, "input_cost": 0.011, "output_cost": 0.0066, "prompt_tokens": 100, "completion_tokens": 30} + "claude-opus-5": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.00429, + "input_cost": 0.002024, + "output_cost": 0.002266, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "web_search_medium", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "web_search_calls": 3 + }, + "owns": [ + "search_context_cost_per_query.search_context_size_medium", + "web_search_billing_unit" + ], + "fallback_for": [], + "web_search": "medium", + "expected": { + "gpt-5.6": { + "spend": 0.021488, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0142976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0217448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.01434896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.045204, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.11454, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0495, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0417, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0339, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.113624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.1140552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "web_search_low", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "web_search_calls": 1 + }, + "owns": [ + "search_context_cost_per_query.search_context_size_low" + ], + "fallback_for": [], + "web_search": "low", + "expected": { + "gpt-5.6": { + "spend": 0.018988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0117976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0192448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.01184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.017704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.08704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "web_search_high", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "web_search_calls": 1 + }, + "owns": [ + "search_context_cost_per_query.search_context_size_high" + ], + "fallback_for": [], + "web_search": "high", + "expected": { + "gpt-5.6": { + "spend": 0.023988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0167976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0242448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.01684896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.022704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.09204, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "web_search_per_prompt", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "web_search_calls": 3 + }, + "owns": [ + "search_context_cost_per_query.search_context_size_medium", + "web_search_billing_unit" + ], + "fallback_for": [], + "web_search": "medium", + "expected": { + "gemini/gemini-3.8-flash": { + "spend": 0.037156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.03724224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "google_maps_grounding", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "google_maps_calls": 1 + }, + "owns": [ + "google_maps_grounding_cost_per_query" + ], + "fallback_for": [], + "google_maps": true, + "expected": { + "gemini/gemini-3.1-pro": { + "spend": 0.033624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.027156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0340552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.02724224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "file_search", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "file_search_calls": 1 + }, + "owns": [ + "file_search_cost_per_1k_calls" + ], + "fallback_for": [], + "file_search": true, + "expected": { + "gpt-5.3-codex": { + "spend": 0.010204, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07954, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "fallback_cache_read_at_input_rate", + "family": "pricing", + "usage": { + "fresh_input_tokens": 640, + "cache_read_tokens": 12288, + "output_tokens": 380 + }, + "owns": [], + "fallback_for": [ + "cache_read_input_token_cost" + ], + "expected": { + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00347132, + "input_cost": 0.00310272, + "output_cost": 0.0003686, + "prompt_tokens": 12928, + "completion_tokens": 380 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0021672, + "input_cost": 0.0019392, + "output_cost": 0.000228, + "prompt_tokens": 12928, + "completion_tokens": 380 + } + } + }, + { + "name": "fallback_cache_write_at_input_rate", + "family": "pricing", + "usage": { + "fresh_input_tokens": 512, + "cache_write_5m_tokens": 9216, + "output_tokens": 350 + }, + "owns": [], + "fallback_for": [ + "cache_creation_input_token_cost" + ], + "expected": { + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00267422, + "input_cost": 0.00233472, + "output_cost": 0.0003395, + "prompt_tokens": 9728, + "completion_tokens": 350 + } + } + }, + { + "name": "fallback_reasoning_at_output_rate", + "family": "pricing", + "usage": { + "fresh_input_tokens": 1240, + "output_tokens": 560, + "reasoning_tokens": 3480 + }, + "owns": [], + "fallback_for": [ + "output_cost_per_reasoning_token" + ], + "reasoning": true, + "expected": { + "gemini-3.8-flash": { + "spend": 0.0132496, + "input_cost": 0.0006448, + "output_cost": 0.0126048, + "prompt_tokens": 1240, + "completion_tokens": 4040 + } + } + }, + { + "name": "fallback_image_tokens_at_input_rate", + "family": "pricing", + "usage": { + "fresh_input_tokens": 310, + "image_input_tokens": 1806, + "output_tokens": 240 + }, + "owns": [], + "fallback_for": [ + "input_cost_per_image_token" + ], + "image_input": true, + "expected": { + "gemini-3.8-flash": { + "spend": 0.00184912, + "input_cost": 0.00110032, + "output_cost": 0.0007488, + "prompt_tokens": 2116, + "completion_tokens": 240 + } + } + }, + { + "name": "fallback_video_tokens_at_input_rate", + "family": "pricing", + "usage": { + "fresh_input_tokens": 140, + "video_input_tokens": 7920, + "output_tokens": 300 + }, + "owns": [], + "fallback_for": [ + "input_cost_per_video_token" + ], + "video_input": true, + "expected": { + "gemini-3.1-pro": { + "spend": 0.020706, + "input_cost": 0.016926, + "output_cost": 0.00378, + "prompt_tokens": 8060, + "completion_tokens": 300 + } } }, { "name": "stream", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.034, "input_cost": 0.0204, "output_cost": 0.0136, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.4-mini": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.03, "input_cost": 0.018, "output_cost": 0.012, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-haiku-4-5": {"spend": 0.014, "input_cost": 0.0084, "output_cost": 0.0056, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-opus-5": {"spend": 0.01, "input_cost": 0.006, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-sonnet-5": {"spend": 0.012, "input_cost": 0.0072, "output_cost": 0.0048, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.028, "input_cost": 0.0168, "output_cost": 0.0112, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/kimi-k3": {"spend": 0.024, "input_cost": 0.0144, "output_cost": 0.0096, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/qwen3p8-max": {"spend": 0.026, "input_cost": 0.0156, "output_cost": 0.0104, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.042, "input_cost": 0.0252, "output_cost": 0.0168, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.04, "input_cost": 0.024, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.018, "input_cost": 0.0108, "output_cost": 0.0072, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.016, "input_cost": 0.0096, "output_cost": 0.0064, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.008, "input_cost": 0.0048, "output_cost": 0.0032, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.002, "input_cost": 0.0012, "output_cost": 0.0008, "prompt_tokens": 120, "completion_tokens": 40}, - "meta.llama4-maverick-17b-instruct-v1:0": {"spend": 0.038, "input_cost": 0.0228, "output_cost": 0.0152, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.02, "input_cost": 0.012, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.022, "input_cost": 0.0132, "output_cost": 0.0088, "prompt_tokens": 120, "completion_tokens": 40}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.036, "input_cost": 0.0216, "output_cost": 0.0144, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.6": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0092448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { "name": "stream_no_usage", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "stream_usage": "absent", "exact_spend": false, "models": [ - "anthropic.claude-sonnet-5-v1:0", - "azure/gpt-5.4-mini", + "gpt-5.6", + "gpt-5.4-mini", "azure/gpt-5.6", - "claude-haiku-4-5", + "azure/gpt-5.4-mini", + "gpt-5.3-codex", + "gpt-5.5-pro", "claude-opus-5", "claude-sonnet-5", - "fireworks_ai/deepseek-v4p1-flash", - "fireworks_ai/kimi-k3", - "fireworks_ai/qwen3p8-max", - "gemini-3.1-pro-preview", - "gemini-3.8-flash", - "gemini/gemini-3.1-pro-preview", - "gemini/gemini-3.8-flash", - "gpt-5.3-codex", - "gpt-5.4-mini", - "gpt-5.5-pro", - "gpt-5.6", + "claude-haiku-4-5", + "us.anthropic.claude-opus-5-v1:0", + "anthropic.claude-sonnet-5-v1:0", "meta.llama4-maverick-17b-instruct-v1:0", + "gemini/gemini-3.1-pro", + "gemini/gemini-3.8-flash", + "gemini-3.1-pro", + "gemini-3.8-flash", "together_ai/moonshotai/Kimi-K3", "together_ai/zai-org/GLM-5.3", - "us.anthropic.claude-opus-5-v1:0" + "fireworks_ai/accounts/fireworks/models/kimi-k3", + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash", + "fireworks_ai/accounts/fireworks/models/qwen3p8-max" ] }, - { - "name": "response_model_override", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, - "response_model_override": true, - "expected": { - "azure/gpt-5.4-mini": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-haiku-4-5": {"spend": 0.01, "input_cost": 0.006, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-opus-5": {"spend": 0.012, "input_cost": 0.0072, "output_cost": 0.0048, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-sonnet-5": {"spend": 0.014, "input_cost": 0.0084, "output_cost": 0.0056, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.024, "input_cost": 0.0144, "output_cost": 0.0096, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/kimi-k3": {"spend": 0.026, "input_cost": 0.0156, "output_cost": 0.0104, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/qwen3p8-max": {"spend": 0.028, "input_cost": 0.0168, "output_cost": 0.0112, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.04, "input_cost": 0.024, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.042, "input_cost": 0.0252, "output_cost": 0.0168, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.016, "input_cost": 0.0096, "output_cost": 0.0064, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.018, "input_cost": 0.0108, "output_cost": 0.0072, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.002, "input_cost": 0.0012, "output_cost": 0.0008, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.008, "input_cost": 0.0048, "output_cost": 0.0032, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.022, "input_cost": 0.0132, "output_cost": 0.0088, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.02, "input_cost": 0.012, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40} - } - }, - { - "name": "stream_response_model_override", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, - "stream": true, - "response_model_override": true, - "expected": { - "azure/gpt-5.4-mini": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-haiku-4-5": {"spend": 0.01, "input_cost": 0.006, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-opus-5": {"spend": 0.012, "input_cost": 0.0072, "output_cost": 0.0048, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-sonnet-5": {"spend": 0.014, "input_cost": 0.0084, "output_cost": 0.0056, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.024, "input_cost": 0.0144, "output_cost": 0.0096, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/kimi-k3": {"spend": 0.026, "input_cost": 0.0156, "output_cost": 0.0104, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/qwen3p8-max": {"spend": 0.028, "input_cost": 0.0168, "output_cost": 0.0112, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.04, "input_cost": 0.024, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.042, "input_cost": 0.0252, "output_cost": 0.0168, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.016, "input_cost": 0.0096, "output_cost": 0.0064, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.018, "input_cost": 0.0108, "output_cost": 0.0072, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.002, "input_cost": 0.0012, "output_cost": 0.0008, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.008, "input_cost": 0.0048, "output_cost": 0.0032, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.022, "input_cost": 0.0132, "output_cost": 0.0088, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.02, "input_cost": 0.012, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40} - } - }, - { - "name": "tool_call", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, - "tool_call": true, - "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.034, "input_cost": 0.0204, "output_cost": 0.0136, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.4-mini": {"spend": 0.032, "input_cost": 0.0192, "output_cost": 0.0128, "prompt_tokens": 120, "completion_tokens": 40}, - "azure/gpt-5.6": {"spend": 0.03, "input_cost": 0.018, "output_cost": 0.012, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-haiku-4-5": {"spend": 0.014, "input_cost": 0.0084, "output_cost": 0.0056, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-opus-5": {"spend": 0.01, "input_cost": 0.006, "output_cost": 0.004, "prompt_tokens": 120, "completion_tokens": 40}, - "claude-sonnet-5": {"spend": 0.012, "input_cost": 0.0072, "output_cost": 0.0048, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.028, "input_cost": 0.0168, "output_cost": 0.0112, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/kimi-k3": {"spend": 0.024, "input_cost": 0.0144, "output_cost": 0.0096, "prompt_tokens": 120, "completion_tokens": 40}, - "fireworks_ai/qwen3p8-max": {"spend": 0.026, "input_cost": 0.0156, "output_cost": 0.0104, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.1-pro-preview": {"spend": 0.042, "input_cost": 0.0252, "output_cost": 0.0168, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini-3.8-flash": {"spend": 0.04, "input_cost": 0.024, "output_cost": 0.016, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.018, "input_cost": 0.0108, "output_cost": 0.0072, "prompt_tokens": 120, "completion_tokens": 40}, - "gemini/gemini-3.8-flash": {"spend": 0.016, "input_cost": 0.0096, "output_cost": 0.0064, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.3-codex": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.4-mini": {"spend": 0.008, "input_cost": 0.0048, "output_cost": 0.0032, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.6": {"spend": 0.002, "input_cost": 0.0012, "output_cost": 0.0008, "prompt_tokens": 120, "completion_tokens": 40}, - "meta.llama4-maverick-17b-instruct-v1:0": {"spend": 0.038, "input_cost": 0.0228, "output_cost": 0.0152, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.02, "input_cost": 0.012, "output_cost": 0.008, "prompt_tokens": 120, "completion_tokens": 40}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.022, "input_cost": 0.0132, "output_cost": 0.0088, "prompt_tokens": 120, "completion_tokens": 40}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.036, "input_cost": 0.0216, "output_cost": 0.0144, "prompt_tokens": 120, "completion_tokens": 40} - } - }, - { - "name": "stream_tool_call", - "usage": {"fresh_input_tokens": 80, "output_tokens": 25}, - "stream": true, - "tool_call": true, - "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.0221, "input_cost": 0.0136, "output_cost": 0.0085, "prompt_tokens": 80, "completion_tokens": 25}, - "azure/gpt-5.4-mini": {"spend": 0.0208, "input_cost": 0.0128, "output_cost": 0.008, "prompt_tokens": 80, "completion_tokens": 25}, - "azure/gpt-5.6": {"spend": 0.0195, "input_cost": 0.012, "output_cost": 0.0075, "prompt_tokens": 80, "completion_tokens": 25}, - "claude-haiku-4-5": {"spend": 0.0091, "input_cost": 0.0056, "output_cost": 0.0035, "prompt_tokens": 80, "completion_tokens": 25}, - "claude-opus-5": {"spend": 0.0065, "input_cost": 0.004, "output_cost": 0.0025, "prompt_tokens": 80, "completion_tokens": 25}, - "claude-sonnet-5": {"spend": 0.0078, "input_cost": 0.0048, "output_cost": 0.003, "prompt_tokens": 80, "completion_tokens": 25}, - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.0182, "input_cost": 0.0112, "output_cost": 0.007, "prompt_tokens": 80, "completion_tokens": 25}, - "fireworks_ai/kimi-k3": {"spend": 0.0156, "input_cost": 0.0096, "output_cost": 0.006, "prompt_tokens": 80, "completion_tokens": 25}, - "fireworks_ai/qwen3p8-max": {"spend": 0.0169, "input_cost": 0.0104, "output_cost": 0.0065, "prompt_tokens": 80, "completion_tokens": 25}, - "gemini-3.1-pro-preview": {"spend": 0.0273, "input_cost": 0.0168, "output_cost": 0.0105, "prompt_tokens": 80, "completion_tokens": 25}, - "gemini-3.8-flash": {"spend": 0.026, "input_cost": 0.016, "output_cost": 0.01, "prompt_tokens": 80, "completion_tokens": 25}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.0117, "input_cost": 0.0072, "output_cost": 0.0045, "prompt_tokens": 80, "completion_tokens": 25}, - "gemini/gemini-3.8-flash": {"spend": 0.0104, "input_cost": 0.0064, "output_cost": 0.004, "prompt_tokens": 80, "completion_tokens": 25}, - "gpt-5.3-codex": {"spend": 0.0039, "input_cost": 0.0024, "output_cost": 0.0015, "prompt_tokens": 80, "completion_tokens": 25}, - "gpt-5.4-mini": {"spend": 0.0052, "input_cost": 0.0032, "output_cost": 0.002, "prompt_tokens": 80, "completion_tokens": 25}, - "gpt-5.5-pro": {"spend": 0.0026, "input_cost": 0.0016, "output_cost": 0.001, "prompt_tokens": 80, "completion_tokens": 25}, - "gpt-5.6": {"spend": 0.0013, "input_cost": 0.0008, "output_cost": 0.0005, "prompt_tokens": 80, "completion_tokens": 25}, - "meta.llama4-maverick-17b-instruct-v1:0": {"spend": 0.0247, "input_cost": 0.0152, "output_cost": 0.0095, "prompt_tokens": 80, "completion_tokens": 25}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.013, "input_cost": 0.008, "output_cost": 0.005, "prompt_tokens": 80, "completion_tokens": 25}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0143, "input_cost": 0.0088, "output_cost": 0.0055, "prompt_tokens": 80, "completion_tokens": 25}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.0234, "input_cost": 0.0144, "output_cost": 0.009, "prompt_tokens": 80, "completion_tokens": 25} - } - }, { "name": "stream_no_usage_tool_call", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "stream_usage": "absent", "tool_call": true, "exact_spend": false, "models": [ - "anthropic.claude-sonnet-5-v1:0", - "azure/gpt-5.4-mini", + "gpt-5.6", + "gpt-5.4-mini", "azure/gpt-5.6", - "claude-haiku-4-5", + "azure/gpt-5.4-mini", + "gpt-5.3-codex", + "gpt-5.5-pro", "claude-opus-5", "claude-sonnet-5", - "fireworks_ai/deepseek-v4p1-flash", - "fireworks_ai/kimi-k3", - "fireworks_ai/qwen3p8-max", - "gemini-3.1-pro-preview", - "gemini-3.8-flash", - "gemini/gemini-3.1-pro-preview", - "gemini/gemini-3.8-flash", - "gpt-5.3-codex", - "gpt-5.4-mini", - "gpt-5.5-pro", - "gpt-5.6", + "claude-haiku-4-5", + "us.anthropic.claude-opus-5-v1:0", + "anthropic.claude-sonnet-5-v1:0", "meta.llama4-maverick-17b-instruct-v1:0", + "gemini/gemini-3.1-pro", + "gemini/gemini-3.8-flash", + "gemini-3.1-pro", + "gemini-3.8-flash", "together_ai/moonshotai/Kimi-K3", "together_ai/zai-org/GLM-5.3", - "us.anthropic.claude-opus-5-v1:0" + "fireworks_ai/accounts/fireworks/models/kimi-k3", + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash", + "fireworks_ai/accounts/fireworks/models/qwen3p8-max" ] }, { "name": "stream_no_usage_image_input", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "stream_usage": "absent", "image_input": true, "exact_spend": false, "models": [ - "anthropic.claude-sonnet-5-v1:0", - "azure/gpt-5.4-mini", + "gpt-5.6", + "gpt-5.4-mini", "azure/gpt-5.6", - "claude-haiku-4-5", + "azure/gpt-5.4-mini", + "gpt-5.3-codex", + "gpt-5.5-pro", "claude-opus-5", "claude-sonnet-5", - "fireworks_ai/deepseek-v4p1-flash", - "fireworks_ai/kimi-k3", - "fireworks_ai/qwen3p8-max", - "gemini-3.1-pro-preview", - "gemini-3.8-flash", - "gemini/gemini-3.1-pro-preview", - "gemini/gemini-3.8-flash", - "gpt-5.3-codex", - "gpt-5.4-mini", - "gpt-5.5-pro", - "gpt-5.6", + "claude-haiku-4-5", + "us.anthropic.claude-opus-5-v1:0", + "anthropic.claude-sonnet-5-v1:0", "meta.llama4-maverick-17b-instruct-v1:0", + "gemini/gemini-3.1-pro", + "gemini/gemini-3.8-flash", + "gemini-3.1-pro", + "gemini-3.8-flash", "together_ai/moonshotai/Kimi-K3", "together_ai/zai-org/GLM-5.3", - "us.anthropic.claude-opus-5-v1:0" + "fireworks_ai/accounts/fireworks/models/kimi-k3", + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash", + "fireworks_ai/accounts/fireworks/models/qwen3p8-max" ] }, { "name": "stream_incomplete", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "terminal": "incomplete", "expected": { - "gpt-5.3-codex": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { "name": "stream_no_usage_incomplete", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "stream_usage": "absent", "terminal": "incomplete", @@ -474,17 +1843,37 @@ }, { "name": "stream_unvalidated", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "terminal": "unvalidated", "expected": { - "gpt-5.3-codex": {"spend": 0.006, "input_cost": 0.0036, "output_cost": 0.0024, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.004, "input_cost": 0.0024, "output_cost": 0.0016, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { "name": "stream_no_usage_unvalidated", - "usage": {"fresh_input_tokens": 120, "output_tokens": 40}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "stream": true, "stream_usage": "absent", "terminal": "unvalidated", @@ -496,88 +1885,1008 @@ }, { "name": "prompt_blocked", - "usage": {"fresh_input_tokens": 1000, "output_tokens": 0}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840 + }, "terminal": "prompt_blocked", - "response_model_override": true, "expected": { - "gemini-3.1-pro-preview": {"spend": 0.2, "input_cost": 0.2, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini-3.8-flash": {"spend": 0.21, "input_cost": 0.21, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.08, "input_cost": 0.08, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini/gemini-3.8-flash": {"spend": 0.09, "input_cost": 0.09, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0} + "gemini/gemini-3.1-pro": { + "spend": 0.00368, + "input_cost": 0.00368, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.00092, + "input_cost": 0.00092, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini-3.1-pro": { + "spend": 0.003864, + "input_cost": 0.003864, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini-3.8-flash": { + "spend": 0.0009568, + "input_cost": 0.0009568, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + } } }, { "name": "stream_prompt_blocked", - "usage": {"fresh_input_tokens": 1000, "output_tokens": 0}, + "family": "transport", + "usage": { + "fresh_input_tokens": 1840 + }, "stream": true, "terminal": "prompt_blocked", + "expected": { + "gemini/gemini-3.1-pro": { + "spend": 0.00368, + "input_cost": 0.00368, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.00092, + "input_cost": 0.00092, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini-3.1-pro": { + "spend": 0.003864, + "input_cost": 0.003864, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + }, + "gemini-3.8-flash": { + "spend": 0.0009568, + "input_cost": 0.0009568, + "output_cost": 0.0, + "prompt_tokens": 1840, + "completion_tokens": 0 + } + } + }, + { + "name": "response_model_override", + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, "response_model_override": true, "expected": { - "gemini-3.1-pro-preview": {"spend": 0.2, "input_cost": 0.2, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini-3.8-flash": {"spend": 0.21, "input_cost": 0.21, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.08, "input_cost": 0.08, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0}, - "gemini/gemini-3.8-flash": {"spend": 0.09, "input_cost": 0.09, "output_cost": 0.0, "prompt_tokens": 1000, "completion_tokens": 0} + "gpt-5.6": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { - "name": "all_components_chat", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "cache_write_5m_tokens": 20, "cache_write_1h_tokens": 10, "output_tokens": 25, "reasoning_tokens": 15, "audio_input_tokens": 5, "audio_output_tokens": 3}, - "expected": { - "azure/gpt-5.4-mini": {"spend": 0.0576, "input_cost": 0.03424, "output_cost": 0.02336, "prompt_tokens": 155, "completion_tokens": 43}, - "azure/gpt-5.6": {"spend": 0.054, "input_cost": 0.0321, "output_cost": 0.0219, "prompt_tokens": 155, "completion_tokens": 43}, - "gpt-5.4-mini": {"spend": 0.0144, "input_cost": 0.00856, "output_cost": 0.00584, "prompt_tokens": 155, "completion_tokens": 43}, - "gpt-5.6": {"spend": 0.0036, "input_cost": 0.00214, "output_cost": 0.00146, "prompt_tokens": 155, "completion_tokens": 43}, - "together_ai/moonshotai/Kimi-K3": {"spend": 0.036, "input_cost": 0.0214, "output_cost": 0.0146, "prompt_tokens": 155, "completion_tokens": 43}, - "together_ai/zai-org/GLM-5.3": {"spend": 0.0396, "input_cost": 0.02354, "output_cost": 0.01606, "prompt_tokens": 155, "completion_tokens": 43} - } - }, - { - "name": "all_components_fireworks", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "output_tokens": 25}, - "expected": { - "fireworks_ai/deepseek-v4p1-flash": {"spend": 0.01876, "input_cost": 0.01176, "output_cost": 0.007, "prompt_tokens": 120, "completion_tokens": 25}, - "fireworks_ai/kimi-k3": {"spend": 0.01608, "input_cost": 0.01008, "output_cost": 0.006, "prompt_tokens": 120, "completion_tokens": 25}, - "fireworks_ai/qwen3p8-max": {"spend": 0.01742, "input_cost": 0.01092, "output_cost": 0.0065, "prompt_tokens": 120, "completion_tokens": 25} - } - }, - { - "name": "all_components_anthropic", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "cache_write_5m_tokens": 20, "cache_write_1h_tokens": 10, "output_tokens": 25}, - "expected": { - "anthropic.claude-sonnet-5-v1:0": {"spend": 0.03978, "input_cost": 0.03128, "output_cost": 0.0085, "prompt_tokens": 150, "completion_tokens": 25}, - "claude-haiku-4-5": {"spend": 0.01638, "input_cost": 0.01288, "output_cost": 0.0035, "prompt_tokens": 150, "completion_tokens": 25}, - "claude-opus-5": {"spend": 0.0117, "input_cost": 0.0092, "output_cost": 0.0025, "prompt_tokens": 150, "completion_tokens": 25}, - "claude-sonnet-5": {"spend": 0.01404, "input_cost": 0.01104, "output_cost": 0.003, "prompt_tokens": 150, "completion_tokens": 25}, - "meta.llama4-maverick-17b-instruct-v1:0": {"spend": 0.038, "input_cost": 0.0285, "output_cost": 0.0095, "prompt_tokens": 150, "completion_tokens": 25}, - "us.anthropic.claude-opus-5-v1:0": {"spend": 0.04212, "input_cost": 0.03312, "output_cost": 0.009, "prompt_tokens": 150, "completion_tokens": 25} - } - }, - { - "name": "all_components_anthropic_stream", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "cache_write_5m_tokens": 20, "cache_write_1h_tokens": 10, "output_tokens": 25}, + "name": "stream_response_model_override", + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "response_model_override": true, "stream": true, "expected": { - "claude-haiku-4-5": {"spend": 0.01638, "input_cost": 0.01288, "output_cost": 0.0035, "prompt_tokens": 150, "completion_tokens": 25}, - "claude-opus-5": {"spend": 0.0117, "input_cost": 0.0092, "output_cost": 0.0025, "prompt_tokens": 150, "completion_tokens": 25}, - "claude-sonnet-5": {"spend": 0.01404, "input_cost": 0.01104, "output_cost": 0.003, "prompt_tokens": 150, "completion_tokens": 25} + "gpt-5.6": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { - "name": "all_components_gemini", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "output_tokens": 25, "reasoning_tokens": 15, "audio_input_tokens": 5, "audio_output_tokens": 3}, + "name": "tool_call", + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "tool_call": true, "expected": { - "gemini-3.1-pro-preview": {"spend": 0.0546, "input_cost": 0.02394, "output_cost": 0.03066, "prompt_tokens": 125, "completion_tokens": 43}, - "gemini-3.8-flash": {"spend": 0.052, "input_cost": 0.0228, "output_cost": 0.0292, "prompt_tokens": 125, "completion_tokens": 43}, - "gemini/gemini-3.1-pro-preview": {"spend": 0.0234, "input_cost": 0.01026, "output_cost": 0.01314, "prompt_tokens": 125, "completion_tokens": 43}, - "gemini/gemini-3.8-flash": {"spend": 0.0208, "input_cost": 0.00912, "output_cost": 0.01168, "prompt_tokens": 125, "completion_tokens": 43} + "gpt-5.6": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0092448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + } } }, { - "name": "all_components_responses", - "usage": {"fresh_input_tokens": 80, "cache_read_tokens": 40, "output_tokens": 25, "reasoning_tokens": 15}, + "name": "stream_tool_call", + "family": "transport", + "usage": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "stream": true, + "tool_call": true, "expected": { - "gpt-5.3-codex": {"spend": 0.00627, "input_cost": 0.00252, "output_cost": 0.00375, "prompt_tokens": 120, "completion_tokens": 40}, - "gpt-5.5-pro": {"spend": 0.00418, "input_cost": 0.00168, "output_cost": 0.0025, "prompt_tokens": 120, "completion_tokens": 40} + "gpt-5.6": { + "spend": 0.008988, + "input_cost": 0.00322, + "output_cost": 0.005768, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.4-mini": { + "spend": 0.0017976, + "input_cost": 0.000644, + "output_cost": 0.0011536, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.6": { + "spend": 0.0092448, + "input_cost": 0.003312, + "output_cost": 0.0059328, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "azure/gpt-5.4-mini": { + "spend": 0.00184896, + "input_cost": 0.0006624, + "output_cost": 0.00118656, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.3-codex": { + "spend": 0.007704, + "input_cost": 0.00276, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gpt-5.5-pro": { + "spend": 0.07704, + "input_cost": 0.0276, + "output_cost": 0.04944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-opus-5": { + "spend": 0.0195, + "input_cost": 0.0092, + "output_cost": 0.0103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0117, + "input_cost": 0.00552, + "output_cost": 0.00618, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0039, + "input_cost": 0.00184, + "output_cost": 0.00206, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.02145, + "input_cost": 0.01012, + "output_cost": 0.01133, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.01287, + "input_cost": 0.006072, + "output_cost": 0.006798, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00084124, + "input_cost": 0.0004416, + "output_cost": 0.00039964, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.008624, + "input_cost": 0.00368, + "output_cost": 0.004944, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.002156, + "input_cost": 0.00092, + "output_cost": 0.001236, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.1-pro": { + "spend": 0.0090552, + "input_cost": 0.003864, + "output_cost": 0.0051912, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "gemini-3.8-flash": { + "spend": 0.00224224, + "input_cost": 0.0009568, + "output_cost": 0.00128544, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.002134, + "input_cost": 0.001104, + "output_cost": 0.00103, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.0031392, + "input_cost": 0.001656, + "output_cost": 0.0014832, + "prompt_tokens": 1840, + "completion_tokens": 412 + } + } + }, + { + "name": "stream_full_usage", + "family": "transport", + "usage": {}, + "stream": true, + "usage_by_model": { + "gpt-5.6": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "gpt-5.4-mini": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "azure/gpt-5.6": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "azure/gpt-5.4-mini": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "gpt-5.3-codex": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900 + }, + "gpt-5.5-pro": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900 + }, + "claude-opus-5": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "claude-sonnet-5": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "claude-haiku-4-5": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "us.anthropic.claude-opus-5-v1:0": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "anthropic.claude-sonnet-5-v1:0": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "cache_write_5m_tokens": 2048, + "cache_write_1h_tokens": 1024 + }, + "gemini/gemini-3.1-pro": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330 + }, + "gemini/gemini-3.8-flash": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "gemini-3.1-pro": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330 + }, + "gemini-3.8-flash": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144, + "reasoning_tokens": 900, + "audio_input_tokens": 330, + "audio_output_tokens": 280 + }, + "together_ai/moonshotai/Kimi-K3": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "fresh_input_tokens": 1840, + "output_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "fresh_input_tokens": 1840, + "output_tokens": 412, + "cache_read_tokens": 6144 + } + }, + "expected": { + "gpt-5.6": { + "spend": 0.0600632, + "input_cost": 0.0174952, + "output_cost": 0.042568, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "gpt-5.4-mini": { + "spend": 0.01379264, + "input_cost": 0.00415904, + "output_cost": 0.0096336, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "azure/gpt-5.6": { + "spend": 0.06169072, + "input_cost": 0.01794792, + "output_cost": 0.0437428, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "azure/gpt-5.4-mini": { + "spend": 0.014385144, + "input_cost": 0.004348584, + "output_cost": 0.01003656, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "gpt-5.3-codex": { + "spend": 0.0203256, + "input_cost": 0.0036816, + "output_cost": 0.016644, + "prompt_tokens": 7984, + "completion_tokens": 1312 + }, + "gpt-5.5-pro": { + "spend": 0.203256, + "input_cost": 0.036816, + "output_cost": 0.16644, + "prompt_tokens": 7984, + "completion_tokens": 1312 + }, + "claude-opus-5": { + "spend": 0.045612, + "input_cost": 0.035312, + "output_cost": 0.0103, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "claude-sonnet-5": { + "spend": 0.0273672, + "input_cost": 0.0211872, + "output_cost": 0.00618, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "claude-haiku-4-5": { + "spend": 0.0091224, + "input_cost": 0.0070624, + "output_cost": 0.00206, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "us.anthropic.claude-opus-5-v1:0": { + "spend": 0.0501732, + "input_cost": 0.0388432, + "output_cost": 0.01133, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "anthropic.claude-sonnet-5-v1:0": { + "spend": 0.03010392, + "input_cost": 0.02330592, + "output_cost": 0.006798, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "meta.llama4-maverick-17b-instruct-v1:0": { + "spend": 0.00305308, + "input_cost": 0.00265344, + "output_cost": 0.00039964, + "prompt_tokens": 11056, + "completion_tokens": 412 + }, + "gemini/gemini-3.1-pro": { + "spend": 0.0224108, + "input_cost": 0.0057668, + "output_cost": 0.016644, + "prompt_tokens": 8314, + "completion_tokens": 1312 + }, + "gemini/gemini-3.8-flash": { + "spend": 0.0076232, + "input_cost": 0.0015572, + "output_cost": 0.006066, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "gemini-3.1-pro": { + "spend": 0.02338644, + "input_cost": 0.00604524, + "output_cost": 0.0173412, + "prompt_tokens": 8314, + "completion_tokens": 1312 + }, + "gemini-3.8-flash": { + "spend": 0.007460128, + "input_cost": 0.001619488, + "output_cost": 0.00584064, + "prompt_tokens": 8314, + "completion_tokens": 1592 + }, + "together_ai/moonshotai/Kimi-K3": { + "spend": 0.0035374, + "input_cost": 0.002116, + "output_cost": 0.0014214, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "together_ai/zai-org/GLM-5.3": { + "spend": 0.0019184, + "input_cost": 0.001012, + "output_cost": 0.0009064, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "spend": 0.00250264, + "input_cost": 0.00147264, + "output_cost": 0.00103, + "prompt_tokens": 7984, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "spend": 0.0005232, + "input_cost": 0.000276, + "output_cost": 0.0002472, + "prompt_tokens": 1840, + "completion_tokens": 412 + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "spend": 0.00369216, + "input_cost": 0.00220896, + "output_cost": 0.0014832, + "prompt_tokens": 7984, + "completion_tokens": 412 + } } } ] diff --git a/tests/e2e/cost_calculation/conftest.py b/tests/e2e/cost_calculation/conftest.py index 1473edb119b..e735de40027 100644 --- a/tests/e2e/cost_calculation/conftest.py +++ b/tests/e2e/cost_calculation/conftest.py @@ -7,6 +7,11 @@ deployment under test, and the request shapes plus asserted goldens live in scripted-provider sidecar (``scripted_provider.py``), registered per scenario over its control API. +The proxy must also run with ``MODEL_COST_MAP_MIN_MODEL_COUNT=1`` and +``MODEL_COST_MAP_MAX_SHRINK_RATIO=0``: the 21-entry test map trips the +fetched-cost-map integrity check (too few models, large shrink versus the +bundled map) at those env vars' defaults. + Deselected unless E2E_COST_MAP_STACK is set (marker `cost_map_stack`). """ diff --git a/tests/e2e/cost_calculation/cost_matrix.py b/tests/e2e/cost_calculation/cost_matrix.py index 7999d827060..5e652421182 100644 --- a/tests/e2e/cost_calculation/cost_matrix.py +++ b/tests/e2e/cost_calculation/cost_matrix.py @@ -13,9 +13,12 @@ Two data files drive the suite; nothing in Python lists models or cases: from __future__ import annotations import base64 +import io import json +import math import random import struct +import wave import zlib from collections.abc import Mapping from dataclasses import dataclass @@ -37,22 +40,41 @@ class SearchContextCostPerQuery(BaseModel): search_context_size_high: float | None = None +class ProviderSpecificEntry(BaseModel): + """Provider-specific key rates, keyed by the named suffix litellm looks up + (``fast`` for Anthropic fast mode, ``us`` for US inference geography).""" + + model_config = ConfigDict(frozen=True) + + fast: float | None = None + us: float | None = None + + class CostMapEntry(BaseModel): """The pricing fields of a cost-map entry the matrix reads. Shaped like a - ``model_prices_and_context_window.json`` entry; unmodelled keys are ignored.""" + ``model_prices_and_context_window.json`` entry; the file is test-owned so + undeclared keys are forbidden rather than ignored.""" - model_config = ConfigDict(frozen=True, extra="ignore") + model_config = ConfigDict(frozen=True, extra="forbid") litellm_provider: str mode: str + max_tokens: int | None = None + max_input_tokens: int | None = None + max_output_tokens: int | None = None + supports_function_calling: bool | None = None input_cost_per_token: float | None = None output_cost_per_token: float | None = None cache_read_input_token_cost: float | None = None cache_creation_input_token_cost: float | None = None cache_creation_input_token_cost_above_1hr: float | None = None + cache_read_input_token_cost_above_200k_tokens: float | None = None + cache_creation_input_token_cost_above_200k_tokens: float | None = None output_cost_per_reasoning_token: float | None = None input_cost_per_audio_token: float | None = None output_cost_per_audio_token: float | None = None + input_cost_per_image_token: float | None = None + input_cost_per_video_token: float | None = None input_cost_per_token_above_200k_tokens: float | None = None output_cost_per_token_above_200k_tokens: float | None = None input_cost_per_token_flex: float | None = None @@ -61,6 +83,70 @@ class CostMapEntry(BaseModel): output_cost_per_token_priority: float | None = None search_context_cost_per_query: SearchContextCostPerQuery | None = None web_search_billing_unit: str | None = None + google_maps_grounding_cost_per_query: float | None = None + file_search_cost_per_1k_calls: float | None = None + provider_specific_entry: ProviderSpecificEntry | None = None + + +_METADATA_FIELDS: Final = frozenset( + { + "litellm_provider", + "mode", + "max_tokens", + "max_input_tokens", + "max_output_tokens", + "supports_function_calling", + } +) +_CONTAINER_FIELDS: Final = frozenset({"search_context_cost_per_query", "provider_specific_entry"}) + + +def _submodel_rate_keys( + field: str, sub: SearchContextCostPerQuery | ProviderSpecificEntry | None +) -> tuple[str, ...]: + if sub is None: + return () + return tuple( + f"{field}.{name}" + for name in type(sub).model_fields + if getattr(sub, name) is not None + ) + + +def _entry_rate_keys(entry: CostMapEntry) -> frozenset[str]: + """Every cost key an entry carries, with container subfields expanded to + dotted names (``search_context_cost_per_query.search_context_size_low``). + ``web_search_billing_unit`` counts as a rate key whenever present, + for both ``per_query`` and ``per_prompt`` values.""" + plain: Final = frozenset( + name + for name in CostMapEntry.model_fields + if name not in _METADATA_FIELDS + and name not in _CONTAINER_FIELDS + and getattr(entry, name) is not None + ) + return ( + plain + | frozenset( + _submodel_rate_keys("search_context_cost_per_query", entry.search_context_cost_per_query) + ) + | frozenset(_submodel_rate_keys("provider_specific_entry", entry.provider_specific_entry)) + ) + + +def _entry_has_rate_key(entry: CostMapEntry, rate_key: str) -> bool: + outer, _, inner = rate_key.partition(".") + if outer == "search_context_cost_per_query": + return f"{outer}.{inner}" in _submodel_rate_keys(outer, entry.search_context_cost_per_query) + if outer == "provider_specific_entry": + return f"{outer}.{inner}" in _submodel_rate_keys(outer, entry.provider_specific_entry) + value: Final[object] = getattr(entry, outer, None) + return value is not None + + +SERVICE_TIER_REQUEST_WIRES: Final = frozenset( + {"openai_chat", "azure_chat", "openai_responses", "bedrock_converse"} +) COST_MAP_ADAPTER: Final = TypeAdapter(dict[str, CostMapEntry]) @@ -94,22 +180,42 @@ class ExpectedCell(BaseModel): class Case(BaseModel): - """One request/response shape from cases.json. An exact-spend case names - its models implicitly by carrying one ``expected`` golden per map key; a - recount case (``exact_spend=False``) names them in ``models`` instead.""" + """One request/response shape from cases.json. + + ``family`` splits the matrix: ``pricing`` cases own cost keys (``owns``, + dotted subfield names allowed) or declare which keys they deliberately + leave absent (``fallback_for``) so every cost key in the map has exactly + one owning case; ``transport`` cases exercise counting/transport only and + run wherever they list membership. An exact-spend case names its models + implicitly by carrying one ``expected`` golden per map key; a recount + case (``exact_spend=False``) names them in ``models`` instead. The + feature flags drive request realism in ``_chat_body``.""" model_config = ConfigDict(frozen=True) name: str + family: Literal["pricing", "transport"] usage: ScriptedUsage + usage_by_model: Mapping[str, ScriptedUsage] = Field(default_factory=lambda: MappingProxyType({})) stream: bool = False stream_usage: Literal["final_chunk", "absent"] = "final_chunk" service_tier: Literal["flex", "priority"] | None = None + speed: Literal["fast"] | None = None + inference_geo: Literal["us"] | None = None response_model_override: bool = False exact_spend: bool = True tool_call: bool = False image_input: bool = False + audio_input: bool = False + audio_output: bool = False + video_input: bool = False + reasoning: bool = False + web_search: Literal["low", "medium", "high"] | None = None + google_maps: bool = False + file_search: bool = False terminal: Literal["completed", "incomplete", "unvalidated", "prompt_blocked"] = "completed" + owns: tuple[str, ...] = () + fallback_for: tuple[str, ...] = () expected: Mapping[str, ExpectedCell] = Field(default_factory=lambda: MappingProxyType({})) models: tuple[str, ...] = () @@ -121,11 +227,14 @@ class Case(BaseModel): def expected_for(self, model: FrontierModel) -> ExpectedCell: return self.expected[model.map_key] + def usage_for(self, map_key: str) -> ScriptedUsage: + return self.usage_by_model.get(map_key, self.usage) + def scenario(self, scenario_id: str, model: FrontierModel, text: str) -> Scenario: return Scenario( scenario_id=scenario_id, wire=model.wire, - usage=self.usage, + usage=self.usage_for(model.map_key), model=model.provider_model, output=ScriptedOutput( text=text, @@ -137,6 +246,8 @@ class Case(BaseModel): ), stream_usage=self.stream_usage, service_tier=self.service_tier, + speed=self.speed, + inference_geo=self.inference_geo, ) @@ -183,7 +294,7 @@ _PROVIDER_WIRING: Final[Mapping[tuple[str, str], _ProviderWiring]] = MappingProx { ("openai", "chat"): _ProviderWiring("openai_chat", "openai", MappingProxyType({})), ("openai", "responses"): _ProviderWiring( - "openai_responses", "openai", MappingProxyType({}) + "openai_responses", "openai/responses", MappingProxyType({}) ), ("anthropic", "chat"): _ProviderWiring( "anthropic_messages", "anthropic", MappingProxyType({}) @@ -226,7 +337,13 @@ class FrontierModel: @property def override_rates(self) -> CostMapEntry: - if self.base_model is not None or self.override_map_key is None: + # bedrock_converse responses carry no model field, so a reported-model + # override can never repoint pricing there, same as a base_model pin. + if ( + self.base_model is not None + or self.wire == "bedrock_converse" + or self.override_map_key is None + ): return self.rates return COST_MAP[self.override_map_key] @@ -338,6 +455,31 @@ def _png_chunk(tag: bytes, payload: bytes) -> bytes: return struct.pack(">I", len(payload)) + tag + payload + struct.pack(">I", zlib.crc32(tag + payload)) +def audio_input_data_url() -> str: + """A deterministic 0.5 s 16-bit PCM WAV (8 kHz, 220 Hz sine) as a data + URL, small enough to stay a fixture but real audio to the provider.""" + frames: Final = b"".join( + struct.pack(" str: + """A deterministic mp4-looking blob (ftyp box plus a fixed mdat payload) + as a data URL; only the media type and bytes matter to the wire.""" + ftyp: Final = struct.pack(">I4s4sI4s4s", 24, b"ftyp", b"isom", 0x200, b"isom", b"iso6") + mdat_payload: Final = bytes((i * 7 + 13) % 256 for i in range(4096)) + mdat: Final = struct.pack(">I4s", 8 + len(mdat_payload), b"mdat") + mdat_payload + return "data:video/mp4;base64," + base64.b64encode(ftyp + mdat).decode() + + def image_input_data_url() -> str: """A deterministic 256x256 RGB noise PNG as a data URL; noise compresses poorly on purpose so the base64 payload stays well above 100 KB and would @@ -357,6 +499,8 @@ def image_input_data_url() -> str: IMAGE_INPUT_DATA_URL: Final = image_input_data_url() +AUDIO_INPUT_DATA_URL: Final = audio_input_data_url() +VIDEO_INPUT_DATA_URL: Final = video_input_data_url() def matrix_data_errors() -> tuple[str, ...]: @@ -381,6 +525,48 @@ def matrix_data_errors() -> tuple[str, ...]: for case in CASES if case.exact_spend == bool(case.models) or case.exact_spend != bool(case.expected) ) + all_pairs: Final = frozenset( + (map_key, key) + for map_key, entry in COST_MAP.items() + for key in _entry_rate_keys(entry) + ) + owned_pairs: Final = tuple( + (map_key, key) + for case in CASES + if case.family == "pricing" + for map_key in case.expected + for key in case.owns + if map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key) + ) + unowned_pairs: Final = sorted( + f"{map_key}:{key}" for map_key, key in all_pairs - frozenset(owned_pairs) + ) + duplicate_pairs: Final = sorted( + f"{map_key}:{key}" + for map_key, key in set(owned_pairs) + if owned_pairs.count((map_key, key)) > 1 + ) + owns_without_holder: Final = sorted( + f"{case.name}:{key}" + for case in CASES + for key in case.owns + if not any( + map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key) + for map_key in case.expected + ) + ) + fallback_violations: Final = sorted( + f"{case.name}:{map_key}:{key}" + for case in CASES + for key in case.fallback_for + for map_key in (*case.expected, *case.models) + if map_key in COST_MAP and _entry_has_rate_key(COST_MAP[map_key], key) + ) + family_violations: Final = sorted( + case.name + for case in CASES + if (case.family == "transport") != (not case.owns and not case.fallback_for) + ) input_rates: Final = tuple(entry.input_cost_per_token for entry in COST_MAP.values()) findings: Final = ( ( @@ -404,5 +590,30 @@ def matrix_data_errors() -> tuple[str, ...]: if len(input_rates) != len(set(input_rates)) else None ), + ( + f"(model, rate key) pairs with no owning case: {unowned_pairs}" + if unowned_pairs + else None + ), + ( + f"(model, rate key) pairs owned by more than one case: {duplicate_pairs}" + if duplicate_pairs + else None + ), + ( + f"owns keys absent on all of the case's expected models: {owns_without_holder}" + if owns_without_holder + else None + ), + ( + f"fallback_for keys a case's models actually carry: {fallback_violations}" + if fallback_violations + else None + ), + ( + f"cases with owns/fallback_for inconsistent with family: {family_violations}" + if family_violations + else None + ), ) return tuple(finding for finding in findings if finding is not None) diff --git a/tests/e2e/cost_calculation/scripted_provider.py b/tests/e2e/cost_calculation/scripted_provider.py index 90d95441e5c..c154dcdae62 100644 --- a/tests/e2e/cost_calculation/scripted_provider.py +++ b/tests/e2e/cost_calculation/scripted_provider.py @@ -87,6 +87,56 @@ _TERMINAL_CAPS: Final[Mapping[str, frozenset[str]]] = MappingProxyType( ) +_BASE_USAGE_FIELDS: Final = frozenset({"fresh_input_tokens", "output_tokens"}) +_OPENAI_FAMILY_USAGE: Final = frozenset( + { + "cache_read_tokens", + "reasoning_tokens", + "audio_input_tokens", + "audio_output_tokens", + "web_search_calls", + } +) +_CACHE_WRITE_USAGE: Final = frozenset({"cache_write_5m_tokens", "cache_write_1h_tokens"}) +_GEMINI_USAGE: Final = frozenset( + { + "cache_read_tokens", + "reasoning_tokens", + "audio_input_tokens", + "audio_output_tokens", + "image_input_tokens", + "video_input_tokens", + "web_search_calls", + "google_maps_calls", + } +) + +_USAGE_CAPS: Final[Mapping[str, frozenset[str]]] = MappingProxyType( + { + wire: usage + for wire, usage in ( + ("openai_chat", _OPENAI_FAMILY_USAGE), + ("azure_chat", _OPENAI_FAMILY_USAGE), + ("together_chat", _OPENAI_FAMILY_USAGE), + ("fireworks_chat", _OPENAI_FAMILY_USAGE), + ( + "openai_responses", + frozenset( + {"cache_read_tokens", "reasoning_tokens", "web_search_calls", "file_search_calls"} + ), + ), + ( + "anthropic_messages", + frozenset({"cache_read_tokens", "web_search_calls"}) | _CACHE_WRITE_USAGE, + ), + ("bedrock_converse", frozenset({"cache_read_tokens"}) | _CACHE_WRITE_USAGE), + ("gemini_generate", _GEMINI_USAGE), + ("vertex_generate", _GEMINI_USAGE), + ) + } +) + + class ScriptedToolCall(BaseModel): """A single function call the scripted output emits instead of text. ``arguments`` is the wire's JSON string (~250 chars), sliced into deltas @@ -116,7 +166,11 @@ class ScriptedUsage(BaseModel): reasoning_tokens: int = 0 audio_input_tokens: int = 0 audio_output_tokens: int = 0 + image_input_tokens: int = 0 + video_input_tokens: int = 0 web_search_calls: int = 0 + google_maps_calls: int = 0 + file_search_calls: int = 0 class ScriptedOutput(BaseModel): @@ -151,6 +205,10 @@ class Scenario(BaseModel): model: str stream_usage: StreamUsage = "final_chunk" service_tier: ServiceTier | None = None + # Anthropic fast mode and US inference geography; emitted on the anthropic + # usage object only (litellm reads them there), so they are response-side. + speed: Literal["fast"] | None = None + inference_geo: Literal["us"] | None = None @model_validator(mode="after") def _check_terminal_supported(self) -> Scenario: @@ -161,6 +219,20 @@ class Scenario(BaseModel): raise ValueError( f"wire {self.wire} cannot emit terminal={self.output.terminal}" ) + unsupported: Final = frozenset( + field + for field in self.usage.model_fields_set + if getattr(self.usage, field) + and field not in (_USAGE_CAPS.get(self.wire, frozenset()) | _BASE_USAGE_FIELDS) + ) + if unsupported: + raise ValueError( + f"wire {self.wire} cannot express usage fields {sorted(unsupported)}" + ) + if (self.speed or self.inference_geo) and self.wire != "anthropic_messages": + raise ValueError( + f"wire {self.wire} cannot emit speed/inference_geo (anthropic usage fields)" + ) return self @property @@ -215,32 +287,10 @@ def _sse(events: tuple[tuple[str | None, Mapping[str, object] | str], ...]) -> b def _openai_usage(u: ScriptedUsage) -> Mapping[str, object]: - prompt_tokens: Final = ( - u.fresh_input_tokens - + u.cache_read_tokens - + u.cache_write_5m_tokens - + u.cache_write_1h_tokens - + u.audio_input_tokens - ) + prompt_tokens: Final = u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens completion_tokens: Final = u.output_tokens + u.reasoning_tokens + u.audio_output_tokens prompt_details: Final = _jobj_opt( ("cached_tokens", u.cache_read_tokens) if u.cache_read_tokens else None, - ( - ("cache_write_tokens", u.cache_write_5m_tokens + u.cache_write_1h_tokens) - if u.cache_write_5m_tokens or u.cache_write_1h_tokens - else None - ), - ( - ( - "cache_creation_token_details", - _jobj( - ("ephemeral_5m_input_tokens", u.cache_write_5m_tokens), - ("ephemeral_1h_input_tokens", u.cache_write_1h_tokens), - ), - ) - if u.cache_write_5m_tokens or u.cache_write_1h_tokens - else None - ), ("audio_tokens", u.audio_input_tokens) if u.audio_input_tokens else None, ) completion_details: Final = _jobj_opt( @@ -256,12 +306,16 @@ def _openai_usage(u: ScriptedUsage) -> Mapping[str, object]: ) -def _anthropic_usage(u: ScriptedUsage) -> Mapping[str, object]: +def _anthropic_usage(scenario: Scenario) -> Mapping[str, object]: # Anthropic reports uncached-only input_tokens; cache reads and writes ride # top-level fields, with the 5m/1h write split under cache_creation. + u: Final = scenario.usage return _jobj_opt( ("input_tokens", u.fresh_input_tokens), ("output_tokens", u.output_tokens), + ("service_tier", scenario.service_tier) if scenario.service_tier else None, + ("speed", scenario.speed) if scenario.speed else None, + ("inference_geo", scenario.inference_geo) if scenario.inference_geo else None, ("cache_read_input_tokens", u.cache_read_tokens) if u.cache_read_tokens else None, ( ("cache_creation_input_tokens", u.cache_write_5m_tokens + u.cache_write_1h_tokens) @@ -287,18 +341,24 @@ def _anthropic_usage(u: ScriptedUsage) -> Mapping[str, object]: ) -def _gemini_usage(u: ScriptedUsage) -> Mapping[str, object]: - # promptTokenCount carries the cached count inside it; TEXT modality is the - # cached-inclusive text count so litellm's implicit-caching subtraction lands - # on the fresh figure. candidatesTokenCount includes reasoning + audio. - prompt_tokens: Final = u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens - candidates: Final = u.output_tokens + u.reasoning_tokens + u.audio_output_tokens +def _gemini_usage(scenario: Scenario) -> Mapping[str, object]: + # Real generateContent accounting: promptTokenCount carries the cached count + # inside it (TEXT modality is the cached-inclusive text count so litellm's + # implicit-caching subtraction lands on the fresh figure), candidatesTokenCount + # excludes thoughts, thoughtsTokenCount reports them separately, and + # totalTokenCount sums all three. Image/video input ride promptTokensDetails. + u: Final = scenario.usage + prompt_tokens: Final = ( + u.fresh_input_tokens + u.cache_read_tokens + u.audio_input_tokens + + u.image_input_tokens + u.video_input_tokens + ) + candidates: Final = u.output_tokens + u.audio_output_tokens return _jobj_opt( ("promptTokenCount", prompt_tokens), ("candidatesTokenCount", candidates), - ("totalTokenCount", prompt_tokens + candidates), - ("cachedContentTokenCount", u.cache_read_tokens) if u.cache_read_tokens else None, ("thoughtsTokenCount", u.reasoning_tokens) if u.reasoning_tokens else None, + ("totalTokenCount", prompt_tokens + candidates + u.reasoning_tokens), + ("cachedContentTokenCount", u.cache_read_tokens) if u.cache_read_tokens else None, ( "promptTokensDetails", ( @@ -308,19 +368,66 @@ def _gemini_usage(u: ScriptedUsage) -> Mapping[str, object]: if u.audio_input_tokens else () ), + *( + (_jobj(("modality", "IMAGE"), ("tokenCount", u.image_input_tokens)),) + if u.image_input_tokens + else () + ), + *( + (_jobj(("modality", "VIDEO"), ("tokenCount", u.video_input_tokens)),) + if u.video_input_tokens + else () + ), ), ), ( ( "candidatesTokensDetails", ( - _jobj(("modality", "TEXT"), ("tokenCount", u.output_tokens + u.reasoning_tokens)), + _jobj(("modality", "TEXT"), ("tokenCount", u.output_tokens)), _jobj(("modality", "AUDIO"), ("tokenCount", u.audio_output_tokens)), ), ) if u.audio_output_tokens else None ), + ( + ( + "trafficType", + {"flex": "ON_DEMAND_FLEX", "priority": "ON_DEMAND_PRIORITY"}[ + scenario.service_tier + ], + ) + if scenario.service_tier + else None + ), + ) + + +def _gemini_grounding_metadata(scenario: Scenario) -> Mapping[str, object] | None: + """groundingMetadata for the search/Maps flags. Maps items carry maps + chunks and googleMapsWidgetContextToken so litellm bills them as Maps + queries, not web search.""" + u: Final = scenario.usage + if not u.web_search_calls and not u.google_maps_calls: + return None + if u.google_maps_calls: + return _jobj( + ( + "webSearchQueries", + tuple(f"maps query {i}" for i in range(u.google_maps_calls)), + ), + ( + "groundingChunks", + tuple( + _jobj(("maps", _jobj(("uri", f"https://maps.google.com/?cid={i}")))) + for i in range(u.google_maps_calls) + ), + ), + ("googleMapsWidgetContextToken", f"token_{scenario.scenario_id}"), + ) + return _jobj( + ("webSearchQueries", tuple(f"query {i}" for i in range(u.web_search_calls))), ) @@ -572,7 +679,7 @@ def _anthropic_body(scenario: Scenario, requested_model: str) -> Mapping[str, ob ("model", scenario.output.response_model or requested_model), ("content", _anthropic_content(scenario)), ("stop_reason", _anthropic_stop_reason(scenario)), - ("usage", _anthropic_usage(scenario.usage)), + ("usage", _anthropic_usage(scenario)), ) @@ -581,7 +688,7 @@ def _anthropic_sse(scenario: Scenario, requested_model: str) -> bytes: input_usage: Final = _jobj( *( (key, value) - for key, value in _anthropic_usage(scenario.usage).items() + for key, value in _anthropic_usage(scenario).items() if key != "output_tokens" ) ) @@ -685,7 +792,7 @@ def _gemini_prompt_blocked_body(scenario: Scenario, requested_model: str) -> Map ), ), ), - ("usageMetadata", _gemini_usage(scenario.usage)), + ("usageMetadata", _gemini_usage(scenario)), ("modelVersion", scenario.output.response_model or requested_model), ) @@ -728,22 +835,14 @@ def _gemini_body(scenario: Scenario, requested_model: str) -> Mapping[str, objec ), ("index", 0), ( - ( - "groundingMetadata", - _jobj( - ( - "webSearchQueries", - tuple(f"query {i}" for i in range(scenario.usage.web_search_calls)), - ) - ), - ) - if scenario.usage.web_search_calls + ("groundingMetadata", _gemini_grounding_metadata(scenario)) + if _gemini_grounding_metadata(scenario) is not None else None ), ), ), ), - ("usageMetadata", _gemini_usage(scenario.usage)), + ("usageMetadata", _gemini_usage(scenario)), ("modelVersion", scenario.output.response_model or requested_model), ) @@ -762,7 +861,7 @@ def _gemini_sse(scenario: Scenario, requested_model: str) -> bytes: None, _jobj( ("candidates", ()), - ("usageMetadata", _gemini_usage(scenario.usage)), + ("usageMetadata", _gemini_usage(scenario)), ("modelVersion", scenario.output.response_model or requested_model), ), ), @@ -788,6 +887,16 @@ def _responses_output(scenario: Scenario) -> tuple[Mapping[str, object], ...]: _jobj(("type", "web_search_call"), ("id", f"ws_{i}"), ("status", "completed")) for i in range(scenario.usage.web_search_calls) ), + *( + _jobj( + ("type", "file_search_call"), + ("id", f"fs_{i}"), + ("status", "completed"), + ("queries", (f"query {i}",)), + ("results", ()), + ) + for i in range(scenario.usage.file_search_calls) + ), _jobj( ("type", "function_call"), ("id", f"fc_{scenario.scenario_id}"), @@ -853,9 +962,50 @@ def _responses_sse(scenario: Scenario, requested_model: str) -> bytes: "response.incomplete" if scenario.output.terminal == "incomplete" else "response.completed" ) output_index: Final = ( - scenario.usage.web_search_calls + (1 if scenario.output.terminal == "unvalidated" else 0) + scenario.usage.web_search_calls + + scenario.usage.file_search_calls + + (1 if scenario.output.terminal == "unvalidated" else 0) ) - middle_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = ( + file_search_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = tuple( + event + for i in range(scenario.usage.file_search_calls) + for event in ( + ( + "response.output_item.added", + _jobj( + ("type", "response.output_item.added"), + ("output_index", i), + ( + "item", + _jobj( + ("type", "file_search_call"), + ("id", f"fs_{i}"), + ("status", "in_progress"), + ("queries", ()), + ), + ), + ), + ), + ( + "response.output_item.done", + _jobj( + ("type", "response.output_item.done"), + ("output_index", i), + ( + "item", + _jobj( + ("type", "file_search_call"), + ("id", f"fs_{i}"), + ("status", "completed"), + ("queries", (f"query {i}",)), + ("results", ()), + ), + ), + ), + ), + ) + ) + call_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = ( ( ( "response.output_item.added", @@ -911,6 +1061,10 @@ def _responses_sse(scenario: Scenario, requested_model: str) -> bytes: ), ) ) + middle_events: Final[tuple[tuple[str, Mapping[str, object]], ...]] = ( + *file_search_events, + *call_events, + ) return _sse( ( ("response.created", _jobj(("type", "response.created"), ("response", created))), @@ -976,7 +1130,7 @@ def _bedrock_content(scenario: Scenario) -> tuple[Mapping[str, object], ...]: def _bedrock_body(scenario: Scenario) -> Mapping[str, object]: - return _jobj( + return _jobj_opt( ( "output", _jobj( @@ -992,6 +1146,11 @@ def _bedrock_body(scenario: Scenario) -> Mapping[str, object]: ("stopReason", _bedrock_stop_reason(scenario)), ("usage", _bedrock_usage(scenario.usage)), ("metrics", _jobj(("latencyMs", 42))), + ( + ("serviceTier", _jobj(("type", scenario.service_tier))) + if scenario.service_tier + else None + ), ) @@ -1083,9 +1242,14 @@ def _bedrock_eventstream(scenario: Scenario) -> bytes: ( _aws_event_frame( "metadata", - _jobj( + _jobj_opt( ("usage", _bedrock_usage(scenario.usage)), ("metrics", _jobj(("latencyMs", 42))), + ( + ("serviceTier", _jobj(("type", scenario.service_tier))) + if scenario.service_tier + else None + ), ), ), ) diff --git a/tests/e2e/cost_calculation/test_token_pricing_e2e.py b/tests/e2e/cost_calculation/test_token_pricing_e2e.py index 03dab6be5e7..004cb4d839e 100644 --- a/tests/e2e/cost_calculation/test_token_pricing_e2e.py +++ b/tests/e2e/cost_calculation/test_token_pricing_e2e.py @@ -16,8 +16,11 @@ from typing import Final from conftest import CostCalcClient, cost_rows, register_scenario_deployment from cost_matrix import ( + AUDIO_INPUT_DATA_URL, FRONTIER_MODELS, IMAGE_INPUT_DATA_URL, + SERVICE_TIER_REQUEST_WIRES, + VIDEO_INPUT_DATA_URL, Case, FrontierModel, cases_for, @@ -27,15 +30,27 @@ from cost_matrix import ( from e2e_config import unique_marker from lifecycle import ResourceManager from models import ( + CacheControl, + ChatAudio, ChatBody, ChatMessage, ChatStreamOptions, ChatTool, ChatToolFunction, + FileContentPart, + FileObject, + FileSearchTool, + GoogleMapsTool, + GoogleSearchTool, + HostedWebSearchTool, ImageContentPart, ImageUrl, + InputAudio, + InputAudioContentPart, TextContentPart, + WebSearchOptions, ) +from scripted_provider import ScriptedUsage, Wire pytestmark: Final = [pytest.mark.e2e, pytest.mark.cost_map_stack] # mutable-ok: pytest only accepts a list for pytestmark @@ -52,40 +67,131 @@ def _case_id(param: tuple[FrontierModel, Case]) -> str: return f"{model.map_key.replace('/', '-')}-{case.name}" -def _chat_body(model_name: str, marker: str, case: Case) -> ChatBody: - return ChatBody( - model=model_name, - messages=( - ChatMessage( - role="user", - content=( - [ - TextContentPart(text=f"{marker} scripted pricing call"), - ImageContentPart(image_url=ImageUrl(url=IMAGE_INPUT_DATA_URL)), - ] - if case.image_input - else f"{marker} scripted pricing call" - ), - ), +_CACHE_WIRES: Final = frozenset({"anthropic_messages", "bedrock_converse"}) +_WEB_SEARCH_OPTION_WIRES: Final = frozenset({"openai_chat", "azure_chat", "openai_responses"}) + + +def _cache_control(usage: ScriptedUsage, wire: Wire) -> CacheControl | None: + if wire not in _CACHE_WIRES: + return None + if not (usage.cache_read_tokens or usage.cache_write_5m_tokens or usage.cache_write_1h_tokens): + return None + return CacheControl(type="ephemeral", ttl="1h" if usage.cache_write_1h_tokens else None) + + +def _chat_body(model: FrontierModel, case: Case, model_name: str, marker: str) -> ChatBody: + usage: Final = case.usage_for(model.map_key) + user_parts: Final = ( + TextContentPart( + text=f"{marker} summarize the attached material in one line and name the city weather", ), - stream=case.stream, - stream_options=ChatStreamOptions(include_usage=True) if case.stream else None, - service_tier=case.service_tier, - tools=( + *( + (ImageContentPart(image_url=ImageUrl(url=IMAGE_INPUT_DATA_URL, detail="high")),) + if case.image_input + else () + ), + *( + ( + InputAudioContentPart( + input_audio=InputAudio(data=AUDIO_INPUT_DATA_URL.split(",", 1)[1], format="wav") + ), + ) + if case.audio_input + else () + ), + *( + (FileContentPart(file=FileObject(file_data=VIDEO_INPUT_DATA_URL, format="mp4")),) + if case.video_input + else () + ), + ) + tools: Final = ( + *( ( ChatTool( function=ChatToolFunction( name="get_weather", + description="Get the current weather and a short forecast for a city.", parameters={ "type": "object", - "properties": {"city": {"type": "string"}}, + "properties": { + "city": {"type": "string", "description": "City name"}, + "days": {"type": "integer", "description": "Forecast horizon in days"}, + "units": {"type": "string", "enum": ["metric", "imperial"]}, + }, + "required": ["city"], }, ) ), ) if case.tool_call + else () + ), + *( + (HostedWebSearchTool(type="web_search_20250305", name="web_search", max_uses=5),) + if case.web_search is not None and model.wire == "anthropic_messages" + else () + ), + *( + (GoogleSearchTool(),) + if case.web_search is not None and model.wire in ("gemini_generate", "vertex_generate") + else () + ), + *((GoogleMapsTool(),) if case.google_maps else ()), + *((FileSearchTool(vector_store_ids=["vs_cost_calc_fixture"]),) if case.file_search else ()), + ) + return ChatBody( + model=model_name, + messages=( + ChatMessage( + role="system", + content=[ + TextContentPart( + text=( + "You are a deterministic pricing-harness assistant. " + "Keep answers to a single short line." + ), + cache_control=_cache_control(usage, model.wire), + ) + ], + ), + ChatMessage(role="user", content=list(user_parts)), + ), + stream=case.stream, + stream_options=ChatStreamOptions(include_usage=True) if case.stream else None, + service_tier=( + case.service_tier + if case.service_tier is not None and model.wire in SERVICE_TIER_REQUEST_WIRES else None ), + reasoning_effort="medium" if case.reasoning else None, + modalities=( + ["text"] if case.audio_input else (["text", "audio"] if case.audio_output else None) + ), + audio=( + ChatAudio(voice="alloy", format="pcm16") if case.audio_output else None + ), + web_search_options=( + WebSearchOptions(search_context_size=case.web_search) + if case.web_search is not None and model.wire in _WEB_SEARCH_OPTION_WIRES + else None + ), + tools=tools or None, + tool_choice="auto" if case.tool_call and model.wire != "bedrock_converse" else None, + # The test-owned cost map carries no supports_* flags, so litellm's + # optional-params gate rejects the realistic request fields; allowlist + # exactly the ones this case sends. + allowed_openai_params=[ + name + for name, sent in ( + ("tool_choice", case.tool_call and model.wire != "bedrock_converse"), + ("modalities", case.audio_input or case.audio_output), + ("audio", case.audio_output), + ("web_search_options", case.web_search is not None), + ("reasoning_effort", case.reasoning), + ) + if sent + ], ) @@ -105,7 +211,7 @@ class TestTokenPricing: response: Final = client.proxy.transport.send( "/chat/completions", headers=client.proxy.transport.bearer(scoped_key), - json=_chat_body(model_name, marker, case), + json=_chat_body(model, case, model_name, marker), stream=case.stream, ) assert response.ok, ( diff --git a/tests/e2e/cost_map.json b/tests/e2e/cost_map.json index 85cd5ade3d5..117e9b33636 100644 --- a/tests/e2e/cost_map.json +++ b/tests/e2e/cost_map.json @@ -1,525 +1,411 @@ { - "anthropic.claude-sonnet-5-v1:0": { - "cache_creation_input_token_cost": 0.00051, - "cache_creation_input_token_cost_above_1hr": 0.00068, - "cache_read_input_token_cost": 1.7e-05, - "input_cost_per_token": 0.00017, - "litellm_provider": "bedrock_converse", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "gpt-5.6": { + "cache_read_input_token_cost": 1.75e-07, + "input_cost_per_audio_token": 4e-05, + "input_cost_per_token": 1.75e-06, + "input_cost_per_token_flex": 8.75e-07, + "input_cost_per_token_priority": 3.5e-06, + "litellm_provider": "openai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_token": 0.00034, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true - }, - "azure/gpt-5.4-mini": { - "cache_creation_input_token_cost": 0.00048, - "cache_creation_input_token_cost_above_1hr": 0.00064, - "cache_read_input_token_cost": 1.6e-05, - "input_cost_per_audio_token": 0.00096, - "input_cost_per_token": 0.00016, - "input_cost_per_token_above_200k_tokens": 0.00128, - "input_cost_per_token_flex": 0.00024, - "input_cost_per_token_priority": 0.000272, - "litellm_provider": "azure", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "chat", - "output_cost_per_audio_token": 0.00112, - "output_cost_per_reasoning_token": 0.0008, - "output_cost_per_token": 0.00032, - "output_cost_per_token_above_200k_tokens": 0.00144, - "output_cost_per_token_flex": 0.0004, - "output_cost_per_token_priority": 0.000432, + "output_cost_per_audio_token": 8e-05, + "output_cost_per_reasoning_token": 1.6e-05, + "output_cost_per_token": 1.4e-05, + "output_cost_per_token_flex": 7e-06, + "output_cost_per_token_priority": 2.8e-05, "search_context_cost_per_query": { - "search_context_size_high": 0.03, "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "supports_function_calling": true + }, + "gpt-5.4-mini": { + "cache_read_input_token_cost": 3.5e-08, + "input_cost_per_audio_token": 1e-05, + "input_cost_per_token": 3.5e-07, + "input_cost_per_token_flex": 1.75e-07, + "input_cost_per_token_priority": 7e-07, + "litellm_provider": "openai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "chat", + "output_cost_per_audio_token": 2e-05, + "output_cost_per_reasoning_token": 3.2e-06, + "output_cost_per_token": 2.8e-06, + "output_cost_per_token_flex": 1.4e-06, + "output_cost_per_token_priority": 5.6e-06, + "search_context_cost_per_query": { + "search_context_size_low": 0.01, + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 + }, + "supports_function_calling": true }, "azure/gpt-5.6": { - "cache_creation_input_token_cost": 0.00045, - "cache_creation_input_token_cost_above_1hr": 0.0006, - "cache_read_input_token_cost": 1.5e-05, - "input_cost_per_audio_token": 0.0009, - "input_cost_per_token": 0.00015, - "input_cost_per_token_above_200k_tokens": 0.0012, - "input_cost_per_token_flex": 0.000225, - "input_cost_per_token_priority": 0.000255, + "cache_read_input_token_cost": 1.8e-07, + "input_cost_per_audio_token": 4.1e-05, + "input_cost_per_token": 1.8e-06, + "input_cost_per_token_flex": 9e-07, + "input_cost_per_token_priority": 3.6e-06, "litellm_provider": "azure", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00105, - "output_cost_per_reasoning_token": 0.00075, - "output_cost_per_token": 0.0003, - "output_cost_per_token_above_200k_tokens": 0.00135, - "output_cost_per_token_flex": 0.000375, - "output_cost_per_token_priority": 0.000405, + "output_cost_per_audio_token": 8.2e-05, + "output_cost_per_reasoning_token": 1.65e-05, + "output_cost_per_token": 1.44e-05, + "output_cost_per_token_flex": 7.2e-06, + "output_cost_per_token_priority": 2.88e-05, "search_context_cost_per_query": { - "search_context_size_high": 0.03, "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "supports_function_calling": true }, - "claude-haiku-4-5": { - "cache_creation_input_token_cost": 0.00021, - "cache_creation_input_token_cost_above_1hr": 0.00028000000000000003, - "cache_read_input_token_cost": 7e-06, - "input_cost_per_token": 7.000000000000001e-05, - "litellm_provider": "anthropic", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "azure/gpt-5.4-mini": { + "cache_read_input_token_cost": 3.6e-08, + "input_cost_per_audio_token": 1.05e-05, + "input_cost_per_token": 3.6e-07, + "input_cost_per_token_flex": 1.8e-07, + "input_cost_per_token_priority": 7.2e-07, + "litellm_provider": "azure", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_token": 0.00014000000000000001, + "output_cost_per_audio_token": 2.1e-05, + "output_cost_per_reasoning_token": 3.3e-06, + "output_cost_per_token": 2.88e-06, + "output_cost_per_token_flex": 1.44e-06, + "output_cost_per_token_priority": 5.76e-06, "search_context_cost_per_query": { - "search_context_size_high": 0.03, "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "supports_function_calling": true + }, + "gpt-5.3-codex": { + "cache_read_input_token_cost": 1.5e-07, + "file_search_cost_per_1k_calls": 0.0025, + "input_cost_per_token": 1.5e-06, + "input_cost_per_token_flex": 7.5e-07, + "input_cost_per_token_priority": 3e-06, + "litellm_provider": "openai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "responses", + "output_cost_per_reasoning_token": 1.3e-05, + "output_cost_per_token": 1.2e-05, + "output_cost_per_token_flex": 6e-06, + "output_cost_per_token_priority": 2.4e-05, + "search_context_cost_per_query": { + "search_context_size_low": 0.01, + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 + }, + "supports_function_calling": true + }, + "gpt-5.5-pro": { + "cache_read_input_token_cost": 1.5e-06, + "file_search_cost_per_1k_calls": 0.0025, + "input_cost_per_token": 1.5e-05, + "input_cost_per_token_flex": 7.5e-06, + "input_cost_per_token_priority": 3e-05, + "litellm_provider": "openai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "responses", + "output_cost_per_reasoning_token": 0.00013, + "output_cost_per_token": 0.00012, + "output_cost_per_token_flex": 6e-05, + "output_cost_per_token_priority": 0.00024, + "search_context_cost_per_query": { + "search_context_size_low": 0.01, + "search_context_size_medium": 0.0125, + "search_context_size_high": 0.015 + }, + "supports_function_calling": true }, "claude-opus-5": { - "cache_creation_input_token_cost": 0.00015000000000000001, - "cache_creation_input_token_cost_above_1hr": 0.0002, - "cache_read_input_token_cost": 4.9999999999999996e-06, - "input_cost_per_token": 5e-05, + "cache_creation_input_token_cost": 6.25e-06, + "cache_creation_input_token_cost_above_1hr": 1e-05, + "cache_creation_input_token_cost_above_200k_tokens": 1.25e-05, + "cache_read_input_token_cost": 5e-07, + "cache_read_input_token_cost_above_200k_tokens": 1e-06, + "input_cost_per_token": 5e-06, + "input_cost_per_token_above_200k_tokens": 1e-05, + "input_cost_per_token_priority": 6.25e-06, "litellm_provider": "anthropic", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_token": 0.0001, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "output_cost_per_token": 2.5e-05, + "output_cost_per_token_above_200k_tokens": 3.75e-05, + "output_cost_per_token_priority": 3.125e-05, + "provider_specific_entry": { + "fast": 6.0, + "us": 1.1 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "search_context_cost_per_query": { + "search_context_size_medium": 0.01 + }, + "supports_function_calling": true }, "claude-sonnet-5": { - "cache_creation_input_token_cost": 0.00018, - "cache_creation_input_token_cost_above_1hr": 0.00024000000000000003, - "cache_read_input_token_cost": 6e-06, - "input_cost_per_token": 6.000000000000001e-05, + "cache_creation_input_token_cost": 3.75e-06, + "cache_creation_input_token_cost_above_1hr": 6e-06, + "cache_creation_input_token_cost_above_200k_tokens": 7.5e-06, + "cache_read_input_token_cost": 3e-07, + "cache_read_input_token_cost_above_200k_tokens": 6e-07, + "input_cost_per_token": 3e-06, + "input_cost_per_token_above_200k_tokens": 6e-06, + "input_cost_per_token_priority": 3.75e-06, "litellm_provider": "anthropic", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_token": 0.00012000000000000002, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "output_cost_per_token": 1.5e-05, + "output_cost_per_token_above_200k_tokens": 2.25e-05, + "output_cost_per_token_priority": 1.875e-05, + "provider_specific_entry": { + "us": 1.1 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "search_context_cost_per_query": { + "search_context_size_medium": 0.01 + }, + "supports_function_calling": true }, - "fireworks_ai/deepseek-v4p1-flash": { - "cache_creation_input_token_cost": 0.00033, - "cache_creation_input_token_cost_above_1hr": 0.00044, - "cache_read_input_token_cost": 1.4e-05, - "input_cost_per_audio_token": 0.00066, - "input_cost_per_token": 0.00014000000000000001, - "litellm_provider": "fireworks_ai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "claude-haiku-4-5": { + "cache_creation_input_token_cost": 1.25e-06, + "cache_creation_input_token_cost_above_1hr": 2e-06, + "cache_read_input_token_cost": 1e-07, + "input_cost_per_token": 1e-06, + "input_cost_per_token_priority": 1.25e-06, + "litellm_provider": "anthropic", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00077, - "output_cost_per_reasoning_token": 0.00055, - "output_cost_per_token": 0.00028000000000000003, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "output_cost_per_token": 5e-06, + "output_cost_per_token_priority": 6.25e-06, + "provider_specific_entry": { + "us": 1.1 }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "search_context_cost_per_query": { + "search_context_size_medium": 0.01 + }, + "supports_function_calling": true }, - "fireworks_ai/kimi-k3": { - "cache_creation_input_token_cost": 0.00033, - "cache_creation_input_token_cost_above_1hr": 0.00044, - "cache_read_input_token_cost": 1.2e-05, - "input_cost_per_audio_token": 0.00066, - "input_cost_per_token": 0.00012000000000000002, - "litellm_provider": "fireworks_ai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "us.anthropic.claude-opus-5-v1:0": { + "cache_creation_input_token_cost": 6.875e-06, + "cache_creation_input_token_cost_above_1hr": 1.1e-05, + "cache_creation_input_token_cost_above_200k_tokens": 1.375e-05, + "cache_read_input_token_cost": 5.5e-07, + "cache_read_input_token_cost_above_200k_tokens": 1.1e-06, + "input_cost_per_token": 5.5e-06, + "input_cost_per_token_above_200k_tokens": 1.1e-05, + "input_cost_per_token_flex": 2.75e-06, + "input_cost_per_token_priority": 6.875e-06, + "litellm_provider": "bedrock_converse", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00077, - "output_cost_per_reasoning_token": 0.00055, - "output_cost_per_token": 0.00024000000000000003, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "output_cost_per_token": 2.75e-05, + "output_cost_per_token_above_200k_tokens": 4.125e-05, + "output_cost_per_token_flex": 1.375e-05, + "output_cost_per_token_priority": 3.4375e-05, + "supports_function_calling": true }, - "fireworks_ai/qwen3p8-max": { - "cache_creation_input_token_cost": 0.00033, - "cache_creation_input_token_cost_above_1hr": 0.00044, - "cache_read_input_token_cost": 1.3e-05, - "input_cost_per_audio_token": 0.00066, - "input_cost_per_token": 0.00013000000000000002, - "litellm_provider": "fireworks_ai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "anthropic.claude-sonnet-5-v1:0": { + "cache_creation_input_token_cost": 4.125e-06, + "cache_creation_input_token_cost_above_1hr": 6.6e-06, + "cache_read_input_token_cost": 3.3e-07, + "input_cost_per_token": 3.3e-06, + "input_cost_per_token_flex": 1.65e-06, + "input_cost_per_token_priority": 4.125e-06, + "litellm_provider": "bedrock_converse", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00077, - "output_cost_per_reasoning_token": 0.00055, - "output_cost_per_token": 0.00026000000000000003, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "output_cost_per_token": 1.65e-05, + "output_cost_per_token_flex": 8.25e-06, + "output_cost_per_token_priority": 2.0625e-05, + "supports_function_calling": true }, - "gemini-3.1-pro-preview": { - "cache_read_input_token_cost": 2.1e-05, - "input_cost_per_audio_token": 0.00126, - "input_cost_per_token": 0.00021, - "input_cost_per_token_above_200k_tokens": 0.00168, - "input_cost_per_token_flex": 0.000315, - "input_cost_per_token_priority": 0.000357, - "litellm_provider": "vertex_ai-language-models", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "meta.llama4-maverick-17b-instruct-v1:0": { + "input_cost_per_token": 2.4e-07, + "litellm_provider": "bedrock_converse", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00147, - "output_cost_per_reasoning_token": 0.00105, - "output_cost_per_token": 0.00042, - "output_cost_per_token_above_200k_tokens": 0.00189, - "output_cost_per_token_flex": 0.000525, - "output_cost_per_token_priority": 0.000567, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true, - "web_search_billing_unit": "per_query" + "output_cost_per_token": 9.7e-07, + "supports_function_calling": true }, - "gemini-3.8-flash": { - "cache_read_input_token_cost": 2e-05, - "input_cost_per_audio_token": 0.0012, - "input_cost_per_token": 0.0002, - "input_cost_per_token_above_200k_tokens": 0.0016, - "input_cost_per_token_flex": 0.0003, - "input_cost_per_token_priority": 0.00034, - "litellm_provider": "vertex_ai-language-models", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "chat", - "output_cost_per_audio_token": 0.0014, - "output_cost_per_reasoning_token": 0.001, - "output_cost_per_token": 0.0004, - "output_cost_per_token_above_200k_tokens": 0.0018, - "output_cost_per_token_flex": 0.0005, - "output_cost_per_token_priority": 0.00054, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true, - "web_search_billing_unit": "per_query" - }, - "gemini/gemini-3.1-pro-preview": { - "cache_read_input_token_cost": 9e-06, - "input_cost_per_audio_token": 0.00054, - "input_cost_per_token": 9e-05, - "input_cost_per_token_above_200k_tokens": 0.00072, - "input_cost_per_token_flex": 0.000135, - "input_cost_per_token_priority": 0.000153, + "gemini/gemini-3.1-pro": { + "cache_read_input_token_cost": 2e-07, + "cache_read_input_token_cost_above_200k_tokens": 4e-07, + "google_maps_grounding_cost_per_query": 0.025, + "input_cost_per_audio_token": 2.6e-06, + "input_cost_per_image_token": 2.2e-06, + "input_cost_per_token": 2e-06, + "input_cost_per_token_above_200k_tokens": 4e-06, + "input_cost_per_token_flex": 1e-06, + "input_cost_per_token_priority": 2.5e-06, + "input_cost_per_video_token": 2.4e-06, "litellm_provider": "gemini", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 1048576, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.0006299999999999999, - "output_cost_per_reasoning_token": 0.00045000000000000004, - "output_cost_per_token": 0.00018, - "output_cost_per_token_above_200k_tokens": 0.0008100000000000001, - "output_cost_per_token_flex": 0.00022500000000000002, - "output_cost_per_token_priority": 0.000243, + "output_cost_per_reasoning_token": 1.3e-05, + "output_cost_per_token": 1.2e-05, + "output_cost_per_token_above_200k_tokens": 1.8e-05, + "output_cost_per_token_flex": 6e-06, + "output_cost_per_token_priority": 1.5e-05, "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.035 }, "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true, "web_search_billing_unit": "per_query" }, "gemini/gemini-3.8-flash": { - "cache_read_input_token_cost": 8e-06, - "input_cost_per_audio_token": 0.00048, - "input_cost_per_token": 8e-05, - "input_cost_per_token_above_200k_tokens": 0.00064, - "input_cost_per_token_flex": 0.00012, - "input_cost_per_token_priority": 0.000136, + "cache_read_input_token_cost": 5e-08, + "google_maps_grounding_cost_per_query": 0.025, + "input_cost_per_audio_token": 1e-06, + "input_cost_per_image_token": 5.5e-07, + "input_cost_per_token": 5e-07, + "input_cost_per_token_flex": 2.5e-07, + "input_cost_per_token_priority": 6.25e-07, + "input_cost_per_video_token": 6e-07, "litellm_provider": "gemini", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 1048576, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00056, - "output_cost_per_reasoning_token": 0.0004, - "output_cost_per_token": 0.00016, - "output_cost_per_token_above_200k_tokens": 0.00072, - "output_cost_per_token_flex": 0.0002, - "output_cost_per_token_priority": 0.000216, + "output_cost_per_audio_token": 6e-06, + "output_cost_per_reasoning_token": 3.5e-06, + "output_cost_per_token": 3e-06, + "output_cost_per_token_flex": 1.5e-06, + "output_cost_per_token_priority": 3.75e-06, "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.035 + }, + "supports_function_calling": true, + "web_search_billing_unit": "per_prompt" + }, + "gemini-3.1-pro": { + "cache_read_input_token_cost": 2.1e-07, + "cache_read_input_token_cost_above_200k_tokens": 4.2e-07, + "google_maps_grounding_cost_per_query": 0.025, + "input_cost_per_audio_token": 2.7e-06, + "input_cost_per_image_token": 2.3e-06, + "input_cost_per_token": 2.1e-06, + "input_cost_per_token_above_200k_tokens": 4.2e-06, + "input_cost_per_token_flex": 1.05e-06, + "input_cost_per_token_priority": 2.625e-06, + "litellm_provider": "vertex_ai-language-models", + "max_input_tokens": 1048576, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "chat", + "output_cost_per_reasoning_token": 1.35e-05, + "output_cost_per_token": 1.26e-05, + "output_cost_per_token_above_200k_tokens": 1.89e-05, + "output_cost_per_token_flex": 6.3e-06, + "output_cost_per_token_priority": 1.575e-05, + "search_context_cost_per_query": { + "search_context_size_medium": 0.035 }, "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true, "web_search_billing_unit": "per_query" }, - "gpt-5.3-codex": { - "cache_read_input_token_cost": 3e-06, - "input_cost_per_token": 3.0000000000000004e-05, - "input_cost_per_token_above_200k_tokens": 0.00024000000000000003, - "input_cost_per_token_flex": 4.5e-05, - "input_cost_per_token_priority": 5.1e-05, - "litellm_provider": "openai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "responses", - "output_cost_per_reasoning_token": 0.00015000000000000001, - "output_cost_per_token": 6.000000000000001e-05, - "output_cost_per_token_above_200k_tokens": 0.00027, - "output_cost_per_token_flex": 7.500000000000001e-05, - "output_cost_per_token_priority": 8.099999999999999e-05, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true - }, - "gpt-5.4-mini": { - "cache_creation_input_token_cost": 0.00012, - "cache_creation_input_token_cost_above_1hr": 0.00016, - "cache_read_input_token_cost": 4e-06, - "input_cost_per_audio_token": 0.00024, - "input_cost_per_token": 4e-05, - "input_cost_per_token_above_200k_tokens": 0.00032, - "input_cost_per_token_flex": 6e-05, - "input_cost_per_token_priority": 6.8e-05, - "litellm_provider": "openai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "gemini-3.8-flash": { + "cache_read_input_token_cost": 5.2e-08, + "google_maps_grounding_cost_per_query": 0.025, + "input_cost_per_audio_token": 1.04e-06, + "input_cost_per_token": 5.2e-07, + "input_cost_per_token_flex": 2.6e-07, + "input_cost_per_token_priority": 6.5e-07, + "input_cost_per_video_token": 6.2e-07, + "litellm_provider": "vertex_ai-language-models", + "max_input_tokens": 1048576, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00028, - "output_cost_per_reasoning_token": 0.0002, - "output_cost_per_token": 8e-05, - "output_cost_per_token_above_200k_tokens": 0.00036, - "output_cost_per_token_flex": 0.0001, - "output_cost_per_token_priority": 0.000108, + "output_cost_per_audio_token": 6.24e-06, + "output_cost_per_token": 3.12e-06, + "output_cost_per_token_flex": 1.56e-06, + "output_cost_per_token_priority": 3.9e-06, "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 + "search_context_size_medium": 0.035 }, "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true - }, - "gpt-5.5-pro": { - "cache_read_input_token_cost": 2e-06, - "input_cost_per_token": 2e-05, - "input_cost_per_token_above_200k_tokens": 0.00016, - "input_cost_per_token_flex": 3e-05, - "input_cost_per_token_priority": 3.4e-05, - "litellm_provider": "openai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "responses", - "output_cost_per_reasoning_token": 0.0001, - "output_cost_per_token": 4e-05, - "output_cost_per_token_above_200k_tokens": 0.00018, - "output_cost_per_token_flex": 5e-05, - "output_cost_per_token_priority": 5.4e-05, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true - }, - "gpt-5.6": { - "cache_creation_input_token_cost": 3e-05, - "cache_creation_input_token_cost_above_1hr": 4e-05, - "cache_read_input_token_cost": 1e-06, - "input_cost_per_audio_token": 6e-05, - "input_cost_per_token": 1e-05, - "input_cost_per_token_above_200k_tokens": 8e-05, - "input_cost_per_token_flex": 1.5e-05, - "input_cost_per_token_priority": 1.7e-05, - "litellm_provider": "openai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "chat", - "output_cost_per_audio_token": 7e-05, - "output_cost_per_reasoning_token": 5e-05, - "output_cost_per_token": 2e-05, - "output_cost_per_token_above_200k_tokens": 9e-05, - "output_cost_per_token_flex": 2.5e-05, - "output_cost_per_token_priority": 2.7e-05, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true - }, - "meta.llama4-maverick-17b-instruct-v1:0": { - "input_cost_per_token": 0.00019, - "litellm_provider": "bedrock_converse", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, - "mode": "chat", - "output_cost_per_token": 0.00038, - "supports_function_calling": true + "web_search_billing_unit": "per_prompt" }, "together_ai/moonshotai/Kimi-K3": { - "cache_creation_input_token_cost": 0.00030000000000000003, - "cache_creation_input_token_cost_above_1hr": 0.0004, - "cache_read_input_token_cost": 9.999999999999999e-06, - "input_cost_per_audio_token": 0.0006000000000000001, - "input_cost_per_token": 0.0001, - "input_cost_per_token_above_200k_tokens": 0.0008, - "input_cost_per_token_flex": 0.00015000000000000001, - "input_cost_per_token_priority": 0.00017, + "input_cost_per_token": 1.15e-06, "litellm_provider": "together_ai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.0006999999999999999, - "output_cost_per_reasoning_token": 0.0005, - "output_cost_per_token": 0.0002, - "output_cost_per_token_above_200k_tokens": 0.0009000000000000001, - "output_cost_per_token_flex": 0.00025, - "output_cost_per_token_priority": 0.00027, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "output_cost_per_token": 3.45e-06, + "supports_function_calling": true }, "together_ai/zai-org/GLM-5.3": { - "cache_creation_input_token_cost": 0.00033, - "cache_creation_input_token_cost_above_1hr": 0.00044, - "cache_read_input_token_cost": 1.1e-05, - "input_cost_per_audio_token": 0.00066, - "input_cost_per_token": 0.00011, - "input_cost_per_token_above_200k_tokens": 0.00088, - "input_cost_per_token_flex": 0.000165, - "input_cost_per_token_priority": 0.000187, + "input_cost_per_token": 5.5e-07, "litellm_provider": "together_ai", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_audio_token": 0.00077, - "output_cost_per_reasoning_token": 0.00055, - "output_cost_per_token": 0.00022, - "output_cost_per_token_above_200k_tokens": 0.00099, - "output_cost_per_token_flex": 0.000275, - "output_cost_per_token_priority": 0.000297, - "search_context_cost_per_query": { - "search_context_size_high": 0.03, - "search_context_size_low": 0.01, - "search_context_size_medium": 0.02 - }, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true, - "supports_web_search": true + "output_cost_per_token": 2.2e-06, + "supports_function_calling": true }, - "us.anthropic.claude-opus-5-v1:0": { - "cache_creation_input_token_cost": 0.00054, - "cache_creation_input_token_cost_above_1hr": 0.00072, - "cache_read_input_token_cost": 1.8e-05, - "input_cost_per_token": 0.00018, - "litellm_provider": "bedrock_converse", - "max_input_tokens": 2000000, - "max_output_tokens": 128000, - "max_tokens": 128000, + "fireworks_ai/accounts/fireworks/models/kimi-k3": { + "cache_read_input_token_cost": 6e-08, + "input_cost_per_token": 6e-07, + "litellm_provider": "fireworks_ai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, "mode": "chat", - "output_cost_per_token": 0.00036, - "supports_function_calling": true, - "supports_prompt_caching": true, - "supports_reasoning": true + "output_cost_per_token": 2.5e-06, + "supports_function_calling": true + }, + "fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash": { + "input_cost_per_token": 1.5e-07, + "litellm_provider": "fireworks_ai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "chat", + "output_cost_per_token": 6e-07, + "supports_function_calling": true + }, + "fireworks_ai/accounts/fireworks/models/qwen3p8-max": { + "cache_read_input_token_cost": 9e-08, + "input_cost_per_token": 9e-07, + "litellm_provider": "fireworks_ai", + "max_input_tokens": 400000, + "max_output_tokens": 65536, + "max_tokens": 65536, + "mode": "chat", + "output_cost_per_token": 3.6e-06, + "supports_function_calling": true } } diff --git a/tests/e2e/models.py b/tests/e2e/models.py index 98fcc1b1f04..b0ff6fdcd86 100644 --- a/tests/e2e/models.py +++ b/tests/e2e/models.py @@ -186,6 +186,18 @@ class ChatMetadata(BaseModel): class ImageUrl(BaseModel): url: str + detail: str | None = None + + +class InputAudio(BaseModel): + data: str + format: str + + +class FileObject(BaseModel): + file_data: str | None = None + file_id: str | None = None + format: str | None = None class TextContentPart(BaseModel): @@ -199,7 +211,17 @@ class ImageContentPart(BaseModel): image_url: ImageUrl -ContentPart = TextContentPart | ImageContentPart +class InputAudioContentPart(BaseModel): + type: str = "input_audio" + input_audio: InputAudio + + +class FileContentPart(BaseModel): + type: str = "file" + file: FileObject + + +ContentPart = TextContentPart | ImageContentPart | InputAudioContentPart | FileContentPart class ChatMessage(BaseModel): @@ -284,6 +306,37 @@ class ChatToolResultTurn(BaseModel): type ChatTurn = ChatMessage | ChatAssistantTurn | ChatToolResultTurn +class HostedWebSearchTool(BaseModel): + """A provider-hosted web-search tool sent inside an OpenAI tools list + (Anthropic's ``web_search_20250305`` shape).""" + + type: str + name: str + max_uses: int | None = None + + +class GoogleSearchTool(BaseModel): + googleSearch: dict[str, object] = {} + + +class GoogleMapsTool(BaseModel): + googleMaps: dict[str, object] = {} + + +class FileSearchTool(BaseModel): + type: Literal["file_search"] = "file_search" + vector_store_ids: list[str] + + +class WebSearchOptions(BaseModel): + search_context_size: Literal["low", "medium", "high"] | None = None + + +class ChatAudio(BaseModel): + voice: str + format: str + + class ChatStreamOptions(BaseModel): include_usage: bool @@ -302,10 +355,16 @@ class ChatBody(BaseModel): thinking: ThinkingParam | None = None service_tier: str | None = None prompt_cache_key: str | None = None - tools: Sequence[ChatTool | McpChatTool] | None = None + tools: Sequence[ + ChatTool | McpChatTool | HostedWebSearchTool | GoogleSearchTool | GoogleMapsTool | FileSearchTool + ] | None = None tool_choice: str | None = None + modalities: list[str] | None = None + audio: ChatAudio | None = None + web_search_options: WebSearchOptions | None = None guardrails: list[str] | None = None response_format: dict[str, object] | None = None + allowed_openai_params: list[str] | None = None chat_template_kwargs: dict[str, bool] | None = None cache: dict[str, bool] | None = {"no-cache": True}