diff --git a/litellm-rust/crates/cost/AGENTS.md b/litellm-rust/crates/cost/AGENTS.md new file mode 100644 index 00000000000..0f70dea8921 --- /dev/null +++ b/litellm-rust/crates/cost/AGENTS.md @@ -0,0 +1 @@ +The price vocabulary (meter, band, channel, `When`, `Billing`) is explained in [../model-catalog/AGENTS.md](../model-catalog/AGENTS.md#concepts), the only crate that maps it to JSON keys diff --git a/litellm-rust/crates/cost/src/price.rs b/litellm-rust/crates/cost/src/price.rs index a90bedacb0f..12cec683748 100644 --- a/litellm-rust/crates/cost/src/price.rs +++ b/litellm-rust/crates/cost/src/price.rs @@ -19,6 +19,7 @@ pub enum ServiceTier { Balanced, } +/// A prompt-size range. `OverPromptTokens(n)` is spelled `_above__tokens` or `_above_k_tokens` #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum Band { #[default] @@ -26,12 +27,15 @@ pub enum Band { OverPromptTokens(NonZeroU64), } +/// The request condition a price applies under, its service tier and prompt band #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] pub struct When { pub tier: ServiceTier, pub band: Band, } +/// The route whose price list a call reads, from the `_batches` and `_decisions` suffixes. Not part of `When`, +/// since a missing batch price is half the online rate and a missing decisions price is the full online rate #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum Channel { #[default] @@ -40,6 +44,7 @@ pub enum Channel { Decisions, } +/// The key of one rate, what is counted, through which channel and under which condition #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] pub struct Price { pub meter: Meter, @@ -47,6 +52,7 @@ pub struct Price { pub when: When, } +/// Every rate a row lists, its off-peak overlay and the row attributes that say how the rates apply #[derive(Clone, Debug, Default, PartialEq)] pub struct Prices { rates: BTreeMap, @@ -117,6 +123,7 @@ pub enum TierScope { WholeCall, } +/// Rates that replace the row's token rates while a UTC window in the schedule is open #[derive(Clone, Debug, PartialEq)] pub struct OffPeak { schedule: Schedule, @@ -133,6 +140,7 @@ pub struct OffPeakRates { pub reasoning: Option, } +/// Multipliers applied on top of a priced call, never prices themselves #[derive(Clone, Debug, Default, PartialEq)] pub struct Adjustments(BTreeMap); diff --git a/litellm-rust/crates/cost/src/price/meter.rs b/litellm-rust/crates/cost/src/price/meter.rs index 6e68981d011..09fd23e42ed 100644 --- a/litellm-rust/crates/cost/src/price/meter.rs +++ b/litellm-rust/crates/cost/src/price/meter.rs @@ -1,5 +1,6 @@ use std::num::NonZeroU64; +/// What a price counts. A variation only one meter has, such as cache TTL or image size, lives inside its variant #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum Meter { Input(InputUnit), @@ -51,12 +52,14 @@ pub enum OutputUnit { VideoSecond, } +/// `Explicit` is a cache the caller created ahead of time, spelled `explicit_cache_read_*`. `Implicit` is every other read #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum CacheReadMode { Implicit, Explicit, } +/// How long a cache write lives. `OneHour` and `TwelveHours` are spelled `_above_1hr` and `_above_12hr` #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] pub enum CacheTtl { Default, diff --git a/litellm-rust/crates/model-catalog/AGENTS.md b/litellm-rust/crates/model-catalog/AGENTS.md index ff24fb81143..abc5d198729 100644 --- a/litellm-rust/crates/model-catalog/AGENTS.md +++ b/litellm-rust/crates/model-catalog/AGENTS.md @@ -1,3 +1,41 @@ +## Concepts + +Every price in a row lowers to one `Price { meter, channel, when }` and its `UnitPrice`. These keys come from the `FlatKey::parse` tests + +- `input_cost_per_token` + - meter `Input(Token(Text))`, channel `Online`, tier `Standard`, band `Base` +- `output_cost_per_token_above_272k_tokens_priority` + - meter `Output(Token(Text))`, channel `Online` + - `_priority` sets tier `Priority` + - `_above_272k_tokens` sets band `OverPromptTokens(272000)` +- `cache_creation_input_token_cost_above_1hr_above_200k_tokens` + - meter `CacheWrite { Text, OneHour }`, where `_above_1hr` is the cache TTL, not a band + - `_above_200k_tokens` sets band `OverPromptTokens(200000)` + - channel `Online`, tier `Standard` +- `ocr_cost_per_page_batches` + - meter `Feature(OcrPage)`, tier `Standard`, band `Base` + - `_batches` sets channel `Batch` + +- Meter is what gets counted, such as text input tokens, reasoning tokens, an image of a given size or a web search + - A variation only one meter has stays inside it (cache TTL, image size, resolution, `_above_Ns_interval`, `max_results_range`) +- `UnitPrice` is USD for one unit of the meter, such as one token, image, page or second +- Band is a range of prompt sizes, starting where `_above__tokens` or `_above_k_tokens` says + - `Prices::band` picks one band from prompt tokens alone, the highest edge passed wins, and `Band::Base` is below every edge + - The whole call moves, so every token meter, output included, bills all its tokens at that band's rate + - `Billing::band_edge` says whether a prompt exactly at the edge is in the band (exclusive by default, inclusive for xAI) + - A band only another tier or the batch channel prices never moves the call +- `When` is the request's condition, its service tier and band + - A missing price falls back from `(tier, band)` to `(standard, band)`, `(tier, base)`, then `(standard, base)` +- `Channel` is the route whose price list the call reads, and each falls back differently + - `Online` falls back across `When` as above + - `Batch` (`_batches`) uses bands but no service tier, and a missing price is half the online rate + - `Decisions` (`_decisions`, OpenAI `/v1/decisions`) overrides single online fields, and a missing price is the full online rate +- Off-peak and `Adjustments` are never prices + - Off-peak swaps in other token rates while a UTC window is open + - `Adjustments` multiplies a priced call for region, residency, geo or `speed: fast` +- `Billing` on `Prices` holds row attributes that change how its prices apply (text unit, band edge, cache-read fallback, wall clock, web search unit, tier scope) + - `lower.rs` derives it from the row and its `litellm_provider`, so `litellm-cost` holds no provider list + ## Layout `wire/` reads one entry as written. `EntryWire::read` makes one pass over the entry's keys and puts every non-null key in exactly one place, which is an attribute, a flat price parsed by `FlatKey::parse`, a price block, a `supports_*` capability or `unknown`. A known key with a bad value becomes an `Error` that names the key, and the rest of the entry is still read @@ -6,8 +44,6 @@ `catalog.rs` reads the document, indexes aliases and runs the checks across entries. A bad entry keeps its raw fields and its errors are listed in `Loaded.errors`. Deciding what to do with them is the caller's job. `Catalog::written` gives back every top-level key and value as the document wrote them, in document order. `Catalog::served` gives back every model and alias as Python serves it, without `sample_spec` and `fallback_generalizations` -`Billing` on `Prices` holds the row attributes pricing reads (text unit, band edge, cache-read fallback, wall clock, web search unit, tier scope). `lower.rs` derives it from the row and its `litellm_provider`, so `litellm-cost` holds no provider list - `store/` is the catalog's in-memory store and the only place that indexes model names. Its rules are in [src/store/AGENTS.md](src/store/AGENTS.md) `layers.rs`, `resolve/`, `model_info.rs` and `register.rs` hold types only. The functions over them land with their first caller @@ -16,7 +52,7 @@ Catalog errors use one opaque `Error` with typed causes and JSON Pointer accesso ## Prices -The price vocabulary lives in `litellm-cost`, and only this crate knows JSON key names. Every price is keyed by `Meter`, `Channel` and `When`. A variation only one meter has stays in the meter (cache TTL, image size, resolution, `_above_Ns_interval`, `max_results_range`). Service tier and prompt band make up `When`. Batch and decisions are a `Channel` because Python bills them with their own rules. Off-peak is an overlay on `Prices`, and regional uplifts are `Adjustments`, never prices +The price vocabulary lives in `litellm-cost`, and only this crate knows JSON key names A new `N`, image size or resolution in an existing key shape parses with no Rust change. A new unit, token kind, feature or tier needs a variant in `litellm-cost` plus its spelling in `wire/flat_key.rs` and `FLAT_KEY_PATTERN` in `schema.rs`