* fix(spend): return 400 from /spend/calculate for a model with no pricing row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert error type and param for unpriced /spend/calculate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): move the repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: alias ModelNotMappedError re-export to satisfy F401
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): raise ModelNotMappedError only when the pricing row is missing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(logs): add span type filter to request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logs): look up span type sql conditions from a mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(proxy): release unclaimed budget reservations at request end
* fix(proxy): release unclaimed budget reservations of websocket sessions too
* test(proxy): drop the structural middleware inheritance check
* fix(proxy): claim the budget reservation on streaming pass-through before its cost callback
The SSE chunk processor hands its success handler to the logging worker
after the response, so the request-end release freed the reservation
first and left the key unguarded until the worker drained. Claim it at
both end-of-stream hand-offs, the immediate enqueue and the coroutine
parked for deferred dispatch.
Give the xai realtime test double the litellm_params attribute every
real Logging object carries, since the wrapper now reads it.
* test(pass-through): give the vertex streaming test doubles a litellm_params dict
The spec'd Logging mocks in test_vertex_ai_anthropic_streaming_cost_injection.py
lacked the instance attribute the chunk processor now reads to claim the budget
reservation. Also restores main's _lazy_openapi_snapshot.json: the branch's copy
had been regenerated under Python 3.14, which dedents one docstring description
that the CI regeneration on Python 3.12 keeps indented, and the PR adds no lazily
loaded route, so main's file is the correct one.
* fix(pass-through): claim the budget reservation only after its cost callback is enqueued
Every pass-through success hand-off stamped callback_bound before handing the
coroutine to the logging worker. When that enqueue raised, the reservation stayed
claimed with no callback left to reconcile it, so the request-end release skipped it
and the reserved cost stayed pinned on the key's counter. Enqueue first, then claim,
so a failed hand-off leaves the reservation for the request-end release.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
With store_prompts_in_spend_logs on, the persisted request body kept the client's model string even when the row's model, model_group, and error text had been replaced by the unknown-model placeholder. The body's model now takes the same placeholder on those rows. Also annotates the new test locals with Final and wraps the four test lines that ran past 120 characters.
A temporary-only member update no longer clones the team default budget into the private row. The row stores just the temp pair and auth, spend admission and reservation add the active increase to the current shared default, so a later lowering of the default reaches members with an active grant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds temp_budget_increase and temp_budget_expiry to the team member edit form with pair validation,
seeds stored values into edit mode, sends both through /team/member_update, and adds cached-key auth
and reservation regression tests for active and expired increases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
SpendLogsMetadata gains a typed azure_spillover key so a request Azure
served off pay-as-you-go capacity is visible in spend tracking, stamped
from the provider response headers or the processed llm_provider- headers
on the standard logging payload. The header parsing moves into a shared
azure_spillover() helper that is_spilled_over_ptu_request() now wraps.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A request for a configured model group that the router rejects before picking a deployment (all deployments in cooldown, no healthy deployment) never gets a custom_llm_provider in its logging kwargs. The spend log payload persisted an empty provider, the daily spend tables carried it through, and the Admin UI Usage page rendered those requests under unknown even though every model in the group has a provider
get_logging_payload now takes the proxy router and, when the logged provider is missing, infers it from the model group's deployments. It only attributes when every deployment in the group resolves to the same provider; mixed groups, unknown groups and a missing router leave the value empty as before. Explicitly logged providers keep precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Project-scoped keys never wrote spend to LiteLLM_ProjectTable, so
/project/info stayed at 0 and project budgets could not block. Wire the
PROJECT entity through the spend queue, redis buffer, and db writer,
reserve and increment a spend:project counter, reseed it from the
project row, reset project spend in the budget cascade, and read the
live counter in the project max budget check. Team member budgets keep
gating project-scoped keys alongside the project budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The reconcile now records the database clock of the scan behind the last complete
run and, on the next run, rewrites every closed day with per-key rows updated since
then, however old the day is. Replaying only the marker day and the one before it
missed a delayed flush or retry that landed on an older date, and reads through the
marker come from the global table alone, so that spend was never counted.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A team can now carry a per-model budget map that every key on the team
inherits. A key's own model_max_budget entry for the same model takes
precedence, so it is gated on and billed to the key alone.
Backend: NewTeamRequest/UpdateTeamRequest accept model_max_budget (validated
like the key-level field, enterprise gated); the value is hydrated onto
UserAPIKeyAuth via the token view, TeamGrants and the carried budget state;
_check_team_model_budget enforces it in the centralized common checks; the
limiter meters spend under team_model_spend:<team>:<model>:<duration> and
skips the team counter when the key overrides; /team/update lets only a
proxy admin raise, re-window or drop a cap; /team/info exposes usage.
The Anthropic context-management compaction summary subrequest runs the
same team gate. Both fallback token-view SQL definitions project the column.
UI: team create and edit forms reuse the key-level ModelMaxBudgetEditor,
premium gated, sending {} to clear and omitting unchanged fields.
A key entry overrides the team cap only when it spend-gates the model
(non-negative max_budget); a row that only carries tpm/rpm limits or a
negative cap leaves the team cap in force.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The write path no longer dual-writes the global table. The cron rolls up closed UTC days
only, so a pod still flushing the current day can never leave the global table short. The
key-free arm reads days through the marker from the global table and later days from
LiteLLM_DailyUserSpend in one UNION ALL, and the marker comes from the config cache
rather than a per-request database lookup.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds a daily spend table without api_key or user_id, written atomically alongside
LiteLLM_DailyUserSpend from the batched writer, reconciled from history by a
scheduled job that advances a marker in LiteLLM_Config, and read by the key-free
arm of the aggregated usage query once the marker covers the requested range.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
