* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline
Auth read each object with its own Redis GET and, on a miss, its own DB
query, then the admission spend counters with one GET each. The prefetch
warms every entry the checks read with one MGET, one raw query for the
Redis misses and one pipeline write, and a per-request batch serves the
spend counter reads from one MGET. The per-object getters stay the
readers and the fallback, so enforcement does not depend on the prefetch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): keep prefetch and spend batch collections immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): prefetch referenced auth objects only after the key's model access check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): give the prefetch-ordering test's patches their test-quality reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read NULL nested permission and budget lists as [] in the prefetch join
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): batch post-call spend counter reads and carry budget state through the request
Post-call warm checks, reservation reads and reconcile reads for one request now go through a task-local spend counter batch: one MGET answers every counter, successful increments write their result back into the batch so no second Redis read follows, and invalidation forgets the key. RedisCache.async_increment sends INCRBYFLOAT and its TTL command in one pipeline round trip.
Auth pins frozen team, user and org budget snapshots on UserAPIKeyAuth, the pre-call setup writes them into the request metadata, and Prometheus reads them back instead of calling get_key_object, get_team_object, get_user_object and get_org_object on the response path. The getters stay as the fallback for requests that carried nothing (custom auth, unauthenticated routes, skipped checks).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): reconcile the budget reservation and the post-call warm checks from one MGET and one pipeline
A scope opened inside an open spend counter batch binds into it instead of starting its own, so the reservation reconcile and the post-call warm checks share the request's single MGET. The reconcile reads every reserved counter concurrently, sends the consistent adjustments in one INCRBYFLOAT+EXPIRE pipeline and settles a flushed or reseeded counter on its own afterwards, keeping the pre-call resize fail-closed. PendingSpendIncrement moves to spend_counter_batch so budget_reservation can build a pipeline without importing a private name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): drop the dataclass import left behind by the PendingSpendIncrement move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): import Self from typing_extensions so the proxy imports on Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use a neutral organization alias in the carried budget state tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover recorded and forgotten spend counter values in the request batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): type the reservation entry carried through reconcile adjustments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline
Auth read each object with its own Redis GET and, on a miss, its own DB
query, then the admission spend counters with one GET each. The prefetch
warms every entry the checks read with one MGET, one raw query for the
Redis misses and one pipeline write, and a per-request batch serves the
spend counter reads from one MGET. The per-object getters stay the
readers and the fallback, so enforcement does not depend on the prefetch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): keep prefetch and spend batch collections immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): prefetch referenced auth objects only after the key's model access check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): give the prefetch-ordering test's patches their test-quality reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read NULL nested permission and budget lists as [] in the prefetch join
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A team's copies published under the name win, then deployments named that way, then a public name only another team's deployment carries (an admin reaches it, routing does too). The endpoint resolver and the live narrowing share one rule.
A public name a team publishes its own deployment copy under now targets that copy only for a caller from that team, so an admin or another team probing the shared name gets the global deployment alone
model_id wins when paired with model: a foreign id still gets the 403, and an id no deployment carries gets the 404 of the lone-id path, before any probe runs or a result is stored under it
cache_health_check_results accepts the Mapping sequences perform_health_check returns
* fix(db): carry DATABASE_SSLMODE/DATABASE_SSLROOTCERT into the assembled writer and reader URLs
The componentized gateway supervisor starts the in-container PgBouncer from the
DATABASE_URL assembled out of the discrete DATABASE_* vars before config.yaml is
read, so an IAM URL had no way to request verified TLS: PgBouncer dialed the
server with server_tls_sslmode = prefer (no SNI, no verification) and public
RDS endpoints rejected the handshake. Two new env vars, exposed by the chart as
database.writer.sslMode / sslRootCert, are appended as libpq sslmode/sslrootcert
to every writer and reader URL the settings assemble (never to a pinned URL),
then translated for Prisma as before. Token refresh now also carries Prisma's
sslmode/sslcert/sslaccept over into the re-minted URL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): keep TLS params on the CLI password URL and the initial IAM reader mint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db): treat DATABASE_SSLROOTCERT on its own as verify-full and cover collector and migrations TLS env
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(db): type the reader mint TLS test double and drop its mutable capture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): count tiktoken cl100k_base admission tokens in Rust
The Rust admission token counter only had the Anthropic tokenizer, so every
other model (OpenAI gpt-4 family, Azure, Gemini, Bedrock non-Claude, Mistral)
tokenized with tiktoken on the Python inference worker.
Add an exact cl100k_base counter to litellm-token-counter: the vendored rank
file (base64 token / rank lines, the bytes Python's tiktoken uses) is parsed
into a byte-level BPE model and the cl100k split pattern is a handwritten
scanner over the shared Unicode classes, so no regex engine runs per request.
Both tokenizers share the message, tool and reply-priming accounting.
The PyO3 TokenCounter gains a from_cl100k_ranks constructor; Python reads the
rank file and passes it in, the way claude_json_str already works. The bridge
selects the counter through the same predicates litellm.token_counter uses
(huggingface_tokenizer_kind, openai_tokenizer_encoding), declines o200k_base,
downloaded HuggingFace and custom tokenizers to Python, and budget reservation
counts once per distinct tokenizer a request names.
The legacy gpt-3.5-turbo-0301 message accounting (4 per message, -1 per name)
stays in Python: the selector declines it through the predicate token_counter
itself uses.
* feat(rust): count tiktoken o200k_base admission tokens in Rust (#40794)
Add a handwritten o200k_base split scanner and TokenCounter::from_o200k_ranks
next to the cl100k_base counter, sharing MergeRanks and the request
accounting. The Python bridge selects it when openai_tokenizer_encoding
names o200k_base, so gpt-4o, gpt-4.1, gpt-5, o1/o3/o4 and chatgpt-4o
requests stop tokenizing on the Python worker under LITELLM_RUST=true
Co-authored-by: yassin <yassin@berri.ai>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An out-of-range cursorless page returns nothing, and reading its total off the
offset reported more sessions than exist (page 4 of 100 sessions at page size 50
claimed 150). Only a page that holds rows, or the first page, ends the list;
anything past it falls back to the bounded count.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A cursorless page that comes back without its lookahead row is the end of the
list, so the total is offset + len(page) and the bounded grouped COUNT over the
whole spend-log table is skipped. First pages on small deployments and every
offset last page now cost one query less.
Moves the count into _count_grouped_sessions and reworks the query-optimization
test that asserted the count always runs second onto a full page, where it does.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
The streaming iterator hook timed the whole provider stream and logged that as the
guardrail duration, so PrometheusLogger added LLM generation time to
litellm_overhead_with_guardrails_latency_metric. The hook now accumulates the time
spent inside _filter_single_text per chunk and logs that sum, keeping start_time and
end_time as the wall-clock window.
Resolves LIT-7589
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A page size that does not divide SPEND_LOGS_PAGINATION_COUNT_CAP left the last
page starting inside the capped window and reading past it, so the rows
disagreed with the total reported next to them. The page limit now stops at the
end of that window, and has_more plus next_session_cursor still hand back a
cursor for walking further.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A page starting at or past SPEND_LOGS_PAGINATION_COUNT_CAP lies outside the
total the client is given, so it now returns no rows without running the page
query and the grouped top-N sort bound stays capped.
Rewrites the offset test to page a fake session store instead of asserting on
the generated SQL, and covers the last page inside the cap next to the first
page past it.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
* fix(mcp): write failure spend log for guardrail-blocked /mcp-rest/tools/call
call_tool_rest_api only translated exceptions to HTTP responses, so a pre_mcp_call
guardrail block never reached failure_handler / async_failure_handler /
post_call_failure_hook and no LiteLLM_SpendLogs failure row was written. Extract
the failure logging from call_mcp_tool into _fire_mcp_tool_call_failure_logging
and run it in the REST route for anything raised between
common_processing_pre_call_logic and execute_mcp_tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the original REST tool error when failure logging raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): log virtual mcp_tool_call failures and keep REST success latency scoped to tool execution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep call_type and request start time on failed-request spend logs
post_call_failure_hook pops litellm_logging_obj before the failure callbacks
run, so the spend row built from request_data had a blank call_type and used
datetime.now() as the start time. A guardrail-blocked MCP tool call therefore
showed up in the Logs page as an LLM row with no call type and a 0s duration.
Lift call_type and start_time off the logging object alongside the fields
already lifted, and have the DB failure hook prefer the lifted start time.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the spend writer into _ProxyDBLogger instead of patching a module global
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
