Commit graph

53023 commits

Author SHA1 Message Date
Yujong Lee
63cc59930c test(cost): pin strip_unregistered_leading_segments region handling against Python
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 01:22:53 -07:00
Yujong Lee
d354c8e985 test(cost): kill the surviving mutants in model lookup and provider inference
Records Python results for the vertex AI21 alias, authenticating provider
prefixes, the replicate id length boundary, the cohere and anthropic prefix
rules, mantle segments that are not regions, fireworks resource names, bedrock
routing-prefix splits, and served-model regional selection. Pins the
case-collision tie-break, where Rust keeps the smallest key because the catalog
has no insertion order. Merges the two AI21 lists, which route identically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 00:19:40 -07:00
Yujong Lee
583dfa97c1 test(cost): pin every provider model list and rule edge case against Python
Adds Python-recorded cases for each model list get_llm_provider routes through,
the finetune and guardrail exclusions, legacy rules that inherit capabilities
through extends, rules with a malformed fill_missing_for_providers, and the
Stability 1 empty size default. Drops the bedrock pricing-only filter: every key
it excludes starts with bedrock/, which the provider-prefix branch resolves first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 23:16:44 -07:00
Yujong Lee
edbf88daf2 feat(cost): port get_model_info key resolution and get_llm_provider inference
ModelInfoCatalog now resolves models the way Python's _get_model_info_helper
does: the six candidate names from _get_potential_model_names, case-insensitive
keys, bedrock routing and base-model stripping, the mantle region prefix, the
fireworks cost key, vertex @latest and meta/ aliases, the azure alias table, the
huggingface zero-rated entry, and the fallback_generalizations capability rules.

Provider inference follows get_llm_provider instead of reading the cost-map
litellm_provider: bare ids route through the provider model lists, the JSON
provider registry, the static openai and cohere lists, and the routing rules.
cost_per_token infers the provider when the caller passes none and fails with
MissingProvider where Python raises BadRequestError.

Bedrock and fal image pricing resolve through get_model_info, and Stability 1
rejects a steps value Python cannot compare with 50.

A differential run over every cost-map key and its prefixed, cased, dated, and
bedrock variants (21,365 lookups) matches Python except for ollama and lemonade,
whose Python lookup queries a local server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 22:43:03 -07:00
Yujong Lee
eb627547c0 test(cost): cover the negative sides of the provider-family match rules
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:36:54 -07:00
Yujong Lee
d4bb9cb85b refactor(cost): match OpenAI, Together, and Vertex call types through CallTypes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:36:19 -07:00
Yujong Lee
bdf1cd17fc refactor(cost): dispatch the completion flow on CallTypes instead of strings
PassthroughCallTypes mirrors Python's enum for the passthrough image call, and
ResponseKind maps to CallTypes variants

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:28:58 -07:00
Yujong Lee
a57afedff3 fix(cost): read batch rates with Python's _batch_rate conversion
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:19:38 -07:00
Yujong Lee
a6644925f3 fix(cost): price Azure AI pixel images through default_image_cost_calculator
Python hands the pixel case to default_image_cost_calculator with the
entry's key, so a size-specific row wins over the base pixel rate, and a
zero width or height prices zero pixels instead of failing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:16:30 -07:00
Yujong Lee
c55f9d7a08 fix(cost): build the zero-cost warning from the same values Python prints
The warning falls back from an empty model group to the model and from an
empty provider to unknown, and prints the caller's provider and model
instead of inferring them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:12:07 -07:00
Yujong Lee
da277a6ebb fix(cost): apply regional uplifts one at a time like generic_cost_per_token
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:11:32 -07:00
Yujong Lee
a56310c2df fix(cost): read Anthropic thinking and cache-creation detail counts as pydantic ints
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:07:46 -07:00
Yujong Lee
c8716673db fix(cost): compare search result tiers as Python numbers
Float max_results and float tier bounds select a tier as they do in Python,
and wire::py_real is the one home for Python's isinstance(value, (int, float))

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:07:08 -07:00
Yujong Lee
091dc10d3c fix(cost): validate guardrail entries with pydantic's lax float and bool
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:06:40 -07:00
Yujong Lee
faf1e9be68 feat(python-compat): reproduce pydantic's lax float and bool validation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:05:21 -07:00
Yujong Lee
f5f4856491 fix(cost): let the requested service tier price every Responses WebSocket part
Python prices each tier part through completion_cost with the caller's
service_tier, so an explicit or optional_params tier overrides the tier each
event reports

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:04:33 -07:00
Yujong Lee
03cd44d829 fix(cost): read discount and margin config the way Python applies it
A discount or scalar margin applies only when it is a Python int or float,
dict margins convert through float(), an empty provider uses the global
margin, and the provider-reported cost header goes through float()

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 20:01:05 -07:00
Yujong Lee
d28f28e1f0 fix(cost): fall back to the response model when custom pricing has no model
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:59:53 -07:00
Yujong Lee
c31c4acf13 fix(cost): count search queries and resolve the search row like completion_cost
An empty query list bills zero queries instead of one, and a bare search model
reads the {provider}/search row, as completion_cost does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:59:33 -07:00
Yujong Lee
c7c1e4be91 fix(cost): price unlisted call types as tokens like Python's cost_per_token
cost_call matches CallTypes and falls through to token pricing for any other
call type, so anthropic_messages and pass-through calls bill instead of
failing. UnsupportedCallType had no remaining use and is removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:56:29 -07:00
Yujong Lee
b98af38a49 refactor(cost): convert provider rates through python-compat's float() and int()
Dashscope, Perplexity, xAI, search, tiered, video, A2A, and video usage rates
now use wire::py_float or py_int, and the Fireworks cache default and off-peak
rates share py_float_unless_bool, matching Python's float() on padded strings,
booleans, and digit separators

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:53:10 -07:00
Yujong Lee
04bffb26d7 refactor(cost): take Python conversions from python-compat
wire.rs keeps thin adapters over litellm-python-compat's truthy, int, float,
str, and pydantic lax int instead of its own reimplementations, and PTU team
ids render through python-compat's str()

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:48:25 -07:00
Yujong Lee
fcff8775c7 feat(python-compat): reproduce int(), float(), and pydantic's lax int
number::int and number::float follow CPython's conversions of plain data, and
pydantic::lax_int follows TypeAdapter(int) in lax mode, which LiteLLM's usage
models apply to provider numbers. The fixture generator records each
conversion's result or exception class for every corpus row, now including
numeric text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:44:05 -07:00
Yujong Lee
3e8b7d6775 fix(cost): read threshold keys the way Python splits them
A key without the _tokens suffix still names a threshold, and without a
service tier its own value is the input rate, as in _get_token_base_cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:41:21 -07:00
Yujong Lee
1437b87f38 fix(cost): coerce off-peak rates, uplift multipliers, and media durations like Python
Off-peak rate strings are trimmed like float(), uplift multipliers go through
wire::py_float so padded strings and booleans convert, and any nonzero audio
or video duration replaces the per-token charge the way Python's truthiness
check does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:39:54 -07:00
Yujong Lee
923f32e4ce fix(cost): parse chat and Responses usage the way Python's Usage does
Token counts accept pydantic's lax ints, a top-level cache_creation_input_tokens
of 0 overrides the details like Usage.__init__, and a cache_write_tokens of 0
wins over cache_creation_tokens. Responses usage copies only the input detail
fields Python copies and drops output video_tokens, and requires total_tokens
when the counts are not plain ints

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:38:44 -07:00
Yujong Lee
9e177da72b fix(cost): read Lyria speech prices from the vertex_ai key before model lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:32:16 -07:00
Yujong Lee
75170ed344 fix(cost): map Together models to their size category once, before cost_per_token
Python swaps an unpriced Together model for its size category in
completion_cost, with the call type, before cost_per_token runs. Rust did it
inside cost_per_token as chat, so embeddings were mapped twice and per-second
pricing was read from the unpriced model. together_pricing_model now runs in
the completion layer only. The billed-rate breakdown also falls back to the
private cache and reasoning counters with Python's _coerce_token_count rules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:31:49 -07:00
Yujong Lee
bf6ea4d781 fix(cost): match Python's Vertex character routing and Fireworks size parsing
Vertex routes and prices characters with the provider prefix stripped, so
prefixed gemini-pro keeps its fixed pricing, and Fireworks parses the
parameter count as a float so oversized counts land in the above-16b tier

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:16:12 -07:00
Yujong Lee
76decc8ae7 fix(cost): strip every provider prefix in the default image and video lookups
default_image_cost_calculator moves next to default_video_cost_calculator in
cost_calculator.rs, matching cost_calculator.py, and both remove the provider
prefix the way Python's str.replace does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:15:02 -07:00
Yujong Lee
29a8f4ce83 fix(cost): pass each provider only the pricing flags Python forwards
A new generic_cost_per_token mirrors Python's: it applies the provider cache
read default and inclusive thresholds itself. cost_per_token forwards the
service tier to anthropic, bedrock, azure, and gemini but not data residency,
drops both for deepseek and tencent, and never forwards the Vertex location
outside the Vertex route. Speech priced per token, transcription token usage,
and realtime candidates go through generic_cost_per_token like Python

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:13:23 -07:00
Yujong Lee
f74ee2b856 fix(cost): resolve built-in tool pricing and model keys the way Python does
get_cost_for_built_in_tools now takes the catalog and model like Python,
adopts the resolved entry's litellm_provider, returns zero for Anthropic usage
without a search count, keeps Perplexity's per-call fallback when no usage
reaches the router, and bills Gemini per query for any unit other than
per_prompt. Search counts are read with pydantic's lax int rules, and
select_model_key applies _check_provider_match. Python int(), float(), and
lax-int coercion now live once in wire.rs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 19:00:37 -07:00
Yujong Lee
a323ad73bc test(cost): pin PTU team id, count, rate, and start coercion against Python
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:33:48 -07:00
Yujong Lee
5bf6748040 test(cost): cover custom pricing guards and the Fireworks cache default coercion
The Fireworks default cache read rate now trims padded numeric strings the
way Python's float() does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:32:01 -07:00
Yujong Lee
822f5a6d61 test(cost): drop a wire contract line that only existed for coverage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:30:13 -07:00
Yujong Lee
e54ac548ec refactor(cost): compare providers through LlmProviders instead of string literals
LlmProviders::matches mirrors Python's custom_llm_provider == LlmProviders.X.value
so a misspelled provider no longer compiles

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:28:30 -07:00
Yujong Lee
ec55adffa9 fix(cost): count and size images in the router the way Python resolves them
route_image_generation_cost_calculator now bills zero images when the
response has no data, skips empty size and quality strings like Python's or
chain, and dispatches on LlmProviders and CallTypes instead of string
literals. The three copies of Python truthiness collapse into wire::is_truthy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:24:45 -07:00
Yujong Lee
d56892eea8 test(cost): run every test through rstest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:19:04 -07:00
Yujong Lee
e16cc34a76 test(cost): restore the provider tests dropped when cost_per_token left the catalog
Nine mirror files lost their bodies in a8189817fa. They come back unchanged
apart from calling the cost_calculator free functions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:14:52 -07:00
Yujong Lee
3a732585f3 refactor(cost): derive enum names with strum and errors with thiserror
CallTypes, LlmProviders, Metric, ServiceTier, and ZeroCostReason drop their
hand-rolled name tables for strum derives, and CostError and TokenizeError
derive Display through thiserror

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:13:19 -07:00
Yujong Lee
2ac3f2e405 refactor(cost): move catalog methods into the modules that mirror their Python functions
ModelInfoCatalog keeps only model selection. Speech, transcription, realtime,
completion, and video costs live in cost_calculator.rs, image wrappers in
image_cost_router.rs, and the rest next to their Python module. The video
lookup now follows default_video_cost_calculator's candidate order and the
fal passthrough lookup no longer strips a fal_ai/ prefix Python keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 18:09:27 -07:00
Yujong Lee
a8189817fa refactor(cost): form cost_calculator.rs as the public token entry surface
Some checks are pending
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
cost_per_token, cost_per_token_for_call, and their custom-pricing variants
move out of ModelInfoCatalog into cost_calculator.rs as free functions
mirroring cost_calculator.py's entry points, taking the catalog only for
entry lookup and model-key selection. Together with the provider-module
moves this starts the split; speech, realtime, image, and billed-rate entry
points still to migrate
2026-09-23 17:43:37 -07:00
Yujong Lee
14201a6c2e refactor(cost): move the provider calculators out of the catalog module
vertex_cost, databricks_cost_per_token, lemonade_cost_per_token,
azure_ai_cost_per_token, and azure_ai_router_fee become free functions in
their provider modules (mirroring the Python provider cost_calculator.py
layout), taking the catalog only for entry lookup and model-key selection
through the new entry and entries accessors; together_ai_cost_per_token
joins its thresholds in together_cost.rs. This is the first slice of the
catalog split, leaving calculators to migrate as cost_calculator.rs forms
2026-09-23 17:35:24 -07:00
Yujong Lee
a880fdeae0 refactor(cost): replace the nineteen nested error enums with one CostError
Every leaf and wrapper error type (UsageError, ResponseCostError, the
Catalog* wrappers, SpeechCostError, BatchError, OcrCostError,
CustomPricingError, A2ACostError, ImageCostRouteError, NonTokenError, and
CompletionResponseCostError) collapses into the single tagged CostError in
error.rs, whose Display impl is the one mapper the future bridge maps from.
The tuple-variant propagation shimmed by the wrappers disappears with them,
so ? passes errors through unchanged across module boundaries
2026-09-23 17:30:10 -07:00
Yujong Lee
26a5dcb96d refactor(cost): dispatch providers and call types through mirrored enums
LlmProviders and CallTypes are generated from Python's enums in
litellm/types/utils.py, pinned value-for-value against executed Python via
the fixture pinning surface, and the catalog's token-cost dispatch now
matches on them instead of strings: the pre-else provider set, the together
and fireworks fallbacks, and the embedding call routing all go through the
enum, with uses_inclusive_token_thresholds parsing XAI the same way. The
_python_-shaped variant names keep Python's member spelling so the
correspondence is mechanical
2026-09-23 17:20:00 -07:00
Yujong Lee
0f4ce30103 refactor(cost): delete the Pricing scaffold and run everything through the live path
The lib.rs scaffold (Pricing, compile, calculate, PricingPlan, CostCatalog,
Request, Cost, and their error type) is gone; non-token, OCR, batch, and
image rates move onto pricing::Rate, batch owns ThresholdPolicy, and
ChatUsage loses its scaffold bridge. The python_reference.tsv replay now
exercises calculate_generic_cost_from_model_info_with_region instead of the
scaffold, retargeting it at the live generic path, and the scaffold-based
test blocks in the anthropic, responses, and model-lookup files replay
through the generic path with identical expectations. Rate::checked plus a
seeded CostError carry the deliberate negative, NaN, and infinity strictness
with pins, replacing the compile-time rejection the scaffold provided
2026-09-23 17:14:55 -07:00
Yujong Lee
0bdf8abf1e test(cost): pin allocation-free pricing lookups and overflow strictness
A counting-allocator test asserts ModelPricing lookups on the request path
allocate nothing, keeping the Copy-able parsed-struct invariant as the
Value-based modules migrate over. The anthropic iteration sums error on
u64 overflow and compile rejects negative rates, both documented as the
deliberate strictness divergence class where Python yields huge or negative
numbers
2026-09-23 17:03:22 -07:00
Yujong Lee
9ca9795354 fix(cost): close seven Python divergences in usage folds, tier rates, and dispatch gates
tier_rate returns 0.0 for a present-but-garbage string instead of falling
back (tiered_pricing.py:83), string rates are trimmed, and tier ranges parse
as floats so the dashscope entries in the cost map select tiers again. The
usage sniff order now matches Python (responses before transcription), chat
usage folds deepseek prompt_cache_hit_tokens behind the anthropic mapping and
emulates the private cache attrs Python's Usage constructor sets, the
anthropic transform zeroes non-numeric counts instead of erroring and carries
the summed cache keys, and the interactions transform sets the
cache_read_input_tokens extra Python passes. Generic totals no longer apply
the provider-specific geo multiplier (only the anthropic wrapper does, per
executed Python), the missing _has_token_or_tiered_pricing gate returns
(0,0) for providers Python routes through its else branch, and the image
calculator tries the {provider}/{quality}/{size}/{model} candidate Python
builds second. Each fix carries a regression test that fails if either side
changes
2026-09-23 17:01:48 -07:00
Yujong Lee
cbd6e82a82 fix(cost): match Python service-tier suffixes and enforce mirrors markers
get_cost_per_unit now delegates to the wire cost_per_unit policy pinned
against executed Python, so _auto-suffixed keys fall back to their base key
and _standard no longer does, matching _SERVICE_TIER_SUFFIXES; threshold
detection excludes the same suffix set Python derives from its ServiceTier
enum plus _batches. The two private tier_key copies in generic_input and
generic_output collapse onto base_rate_selection's shared one.

scripts/check_cost_test_mirrors.py requires every tests/python_*.rs file to
carry a // mirrors: marker naming a real Python test under tests/ (or the
fixture generator for replay files), runs in the rust CI job, and all 57
files now carry verified markers
2026-09-23 16:39:31 -07:00
Yujong Lee
6908fe19e6 feat(cost): parse ModelPricing once with a grammar-checked tokenizer and one Rate policy
ModelPricing is parsed from a raw catalog entry into typed storage: base
rates, tier and batch maps, a threshold vec sorted by token count, off_peak,
tiered_pricing, search_context_cost_per_query, provider_specific_entry,
guardrail_cost_per_unit, and PTU fields, with an extra map no calculation
reads. Rate::{Missing, Null, Invalid, Value} mirrors _get_cost_per_unit
coercions (int, float, bool as 1 or 0, trimmed numeric string) and
wire::cost_per_unit reproduces its tier-suffix base-key fallback with
Python's exact suffix order, pinned against executed Python along with
_parse_above_token_threshold, _BATCH_TIER_KEY, and get_model_info's
projection allowlist via new fixture surfaces. A coverage test walks every
embedded catalog key and fails on any pattern the tokenizer or allowlist
does not recognize, so a new key shape cannot ship silently unpriced, and
clippy disallowed-types now confines serde_json::Value to wire.rs with
explicit allows marking the modules not yet migrated
2026-09-23 16:12:57 -07:00