* feat(rust): count tiktoken cl100k_base admission tokens in Rust
The Rust admission token counter only had the Anthropic tokenizer, so every
other model (OpenAI gpt-4 family, Azure, Gemini, Bedrock non-Claude, Mistral)
tokenized with tiktoken on the Python inference worker.
Add an exact cl100k_base counter to litellm-token-counter: the vendored rank
file (base64 token / rank lines, the bytes Python's tiktoken uses) is parsed
into a byte-level BPE model and the cl100k split pattern is a handwritten
scanner over the shared Unicode classes, so no regex engine runs per request.
Both tokenizers share the message, tool and reply-priming accounting.
The PyO3 TokenCounter gains a from_cl100k_ranks constructor; Python reads the
rank file and passes it in, the way claude_json_str already works. The bridge
selects the counter through the same predicates litellm.token_counter uses
(huggingface_tokenizer_kind, openai_tokenizer_encoding), declines o200k_base,
downloaded HuggingFace and custom tokenizers to Python, and budget reservation
counts once per distinct tokenizer a request names.
The legacy gpt-3.5-turbo-0301 message accounting (4 per message, -1 per name)
stays in Python: the selector declines it through the predicate token_counter
itself uses.
* feat(rust): count tiktoken o200k_base admission tokens in Rust (#40794)
Add a handwritten o200k_base split scanner and TokenCounter::from_o200k_ranks
next to the cl100k_base counter, sharing MergeRanks and the request
accounting. The Python bridge selects it when openai_tokenizer_encoding
names o200k_base, so gpt-4o, gpt-4.1, gpt-5, o1/o3/o4 and chatgpt-4o
requests stop tokenizing on the Python worker under LITELLM_RUST=true
Co-authored-by: yassin <yassin@berri.ai>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An out-of-range cursorless page returns nothing, and reading its total off the
offset reported more sessions than exist (page 4 of 100 sessions at page size 50
claimed 150). Only a page that holds rows, or the first page, ends the list;
anything past it falls back to the bounded count.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A cursorless page that comes back without its lookahead row is the end of the
list, so the total is offset + len(page) and the bounded grouped COUNT over the
whole spend-log table is skipped. First pages on small deployments and every
offset last page now cost one query less.
Moves the count into _count_grouped_sessions and reworks the query-optimization
test that asserted the count always runs second onto a full page, where it does.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
The streaming iterator hook timed the whole provider stream and logged that as the
guardrail duration, so PrometheusLogger added LLM generation time to
litellm_overhead_with_guardrails_latency_metric. The hook now accumulates the time
spent inside _filter_single_text per chunk and logs that sum, keeping start_time and
end_time as the wall-clock window.
Resolves LIT-7589
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A page size that does not divide SPEND_LOGS_PAGINATION_COUNT_CAP left the last
page starting inside the capped window and reading past it, so the rows
disagreed with the total reported next to them. The page limit now stops at the
end of that window, and has_more plus next_session_cursor still hand back a
cursor for walking further.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A page starting at or past SPEND_LOGS_PAGINATION_COUNT_CAP lies outside the
total the client is given, so it now returns no rows without running the page
query and the grouped top-N sort bound stays capped.
Rewrites the offset test to page a fake session store instead of asserting on
the generated SQL, and covers the last page inside the cap next to the first
page past it.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
* fix(mcp): write failure spend log for guardrail-blocked /mcp-rest/tools/call
call_tool_rest_api only translated exceptions to HTTP responses, so a pre_mcp_call
guardrail block never reached failure_handler / async_failure_handler /
post_call_failure_hook and no LiteLLM_SpendLogs failure row was written. Extract
the failure logging from call_mcp_tool into _fire_mcp_tool_call_failure_logging
and run it in the REST route for anything raised between
common_processing_pre_call_logic and execute_mcp_tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the original REST tool error when failure logging raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): log virtual mcp_tool_call failures and keep REST success latency scoped to tool execution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep call_type and request start time on failed-request spend logs
post_call_failure_hook pops litellm_logging_obj before the failure callbacks
run, so the spend row built from request_data had a blank call_type and used
datetime.now() as the start time. A guardrail-blocked MCP tool call therefore
showed up in the Logs page as an LLM row with no call type and a 0s duration.
Lift call_type and start_time off the logging object alongside the fields
already lifted, and have the DB failure hook prefer the lifted start time.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the spend writer into _ProxyDBLogger instead of patching a module global
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Starlette scans the route table in registration order, so a request pays one
regex match per route registered ahead of its own. The proxy registers several
hundred routes and left the liveness probe near position 280 and the lazy
loaded /v1/messages at the very end. Move /health/liveliness, /health/liveness,
/v1/chat/completions, /chat/completions and /v1/messages to the front of the
route table after startup registration and again after a lazy router loads.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): make the in-memory management cache capacity configurable
Add general_settings.user_api_key_cache_max_size (positive int, default 200) to resize the
in-memory tier of the shared user_api_key_cache at startup and on DB config reloads, expose it
in the Admin UI general settings, and cover it with behavioral tests. Prior art: #34726
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): resize the in-memory tier from DualCache so any cache instance honours the cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): wrap the cache capacity field description to the 120 col limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Key objects share the 200-entry UserApiKeyCache in-memory store with teams,
end users, tags and memberships, so churn in those objects evicts hot keys
and forces a LiteLLM_VerificationToken lookup on the next request. Route
bare hashed-token keys to a dedicated InMemoryCache inside UserApiKeyCache
while keeping Redis, TTL, serialization and invalidation shared
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A call admitted before the breaker opened could finish while the breaker was
HALF_OPEN and close it before the designated probe reported, so Redis traffic
resumed on a stale answer. The admission now records whether the call is the
probe and only the probe's success closes a half-open breaker.
The cron job lock manager also logged an error every cycle the open breaker
refused its Redis call, one line per job per pod. That refusal is now a debug
line like every other guarded call, while real Redis errors still log at error
The sync get path was unguarded, logged with a stray format argument, and never fed the
breaker. The sync batch read swallowed the breaker's refusal as an ERROR plus a service
failure event per call, so DualCache dropped its in-memory hits and left batch reservations
behind. record_success closed an OPEN breaker on stale in-flight successes, skipping the
recovery timeout and the half-open probe. The spend counter pipeline re-raised the refusal
into the cost callback, which logged an ERROR and fired the failed-tracking alert per request.
A team or key priority that is not Latin-1 encodable (for example CJK text) was
attached as a response header by the dynamic rate limiter v3 post-call hook, and
Starlette then raised UnicodeEncodeError while writing headers, turning a
successful /v1/messages call into HTTP 500. The header is now omitted for such
values while x-litellm-rate-limiter-version and the v3 rate limit headers are
still attached.
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
