A request can name more than one model, through a comma-separated model or target_model_names on
the batch and fine-tuning routes, and the gate only looked at the string case, so an unpriced model
riding alongside a priced one went through and billed. Check every candidate and name the unpriced
ones in the 403
Aliases had the same problem on the other side: a group that prices itself through its model_info
block lands in the cost map under its deployment id, and the explicit-cost check walked the raw
model list by group name, so an alias pointing at that group read as unpriced. Resolve the group
through the router the way the pricing check already does
Also correct the 403 copy. Providers that return their own usage cost still bill for these models,
so the accurate claim is that litellm has no pricing of its own for them
The three helpers added for the incomplete-response tests took untyped
parameters, which the repo's typing rule does not allow. Annotate them
through a TYPE_CHECKING block so the runtime imports stay inside the
function bodies like the rest of this file.
Every pod and uvicorn worker schedules its own CheckBatchCost poller against the
shared managed-object table, so two of them can select the same completed batch in
one polling window and both write an aretrieve_batch spend log for it, counting
that batch's cost twice.
Claim the row with a compare-and-swap on batch_processed, and skip the batch when
another pod already holds it. The claim sits immediately before the spend log is
written rather than before the results fetch, because batch_processed is also what
blocks deletion of the files the fetch reads and what keeps an unbilled row
selectable by later poll cycles, so claiming up front would strand the spend of any
worker that died mid-fetch. A failed spend log write hands the row back.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Some IdPs, ADFS among them, return only `sub` from UserInfo and put the real
identity claims in the ID token or the access token. Those users land in the
Admin UI with no username, email, groups or teams.
Adds an opt-in `GENERIC_INCLUDE_TOKEN_CLAIMS` that merges token claims into the
UserInfo response before the existing `GENERIC_USER_*_ATTRIBUTE` mappings run.
Precedence is UserInfo, then id_token, then access token, and it applies to both
the PKCE and non-PKCE login flows. With the flag unset, behavior is unchanged.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
OpenAI documents computer_use as a supported tool for Daybreak Blue and its
default snapshot gpt-5.6-sol, but neither entry carried supports_computer_use.
Sibling gpt-5.6-cyber and daybreak-red-latest already set it, so /model/info
and the capability gates reported blue as unable to use computer tools.
The gap came in with the source PR rather than the consolidation: #37029 sets
the flag on cyber and red only. Pinned by a new metadata test covering the
daybreak family and the blue alias agreeing with its snapshot.
Adds a chat_completions route module to litellm-core, mirroring the messages
route, plus Anthropic Messages and Bedrock Converse provider configs. The
per-model `rust: true` opt-in now covers /chat/completions for both providers.
The core accepts an allowlisted subset (text conversations, non-streaming) and
returns CoreError::Unsupported for anything else, so tool calls, multimodal
content and streaming fall back to the Python path transparently.
Resolves LIT-5698
redis-py awaits a redis_connect_func that is a coroutine function, so
dropping every connect func the async paths cannot convert took away an
auth path that worked.
A semantic cache lookup embeds the prompt before the request reaches the LLM,
and that embedding call carried no deadline of its own. It inherited the 6000s
request timeout and the Router's num_retries, so an embedding endpoint that is
down or unroutable parked every proxied request for minutes and gave back
nothing but x-litellm-semantic-similarity 0.0 once it finally gave up.
The lookup now runs on its own short deadline, 5s by default, with retries off
so failures cannot stack. Redis, Valkey and qdrant all pick it up, and the
deadline is settable per cache with semantic_cache_embedding_timeout or
globally with SEMANTIC_CACHE_EMBEDDING_TIMEOUT_SECONDS.
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long
run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in
bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary.
Chunking alone only makes the stall shorter, so the async paths now count in a worker
thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other
requests while a count is in flight. The /utils/token_counter endpoint awaits the new
atoken_counter, and the router's async deployment selection counts off-loop and hands
the result to _pre_call_checks instead of making it count inline.
The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or
silently report zero tokens, and an arbitrarily large one restored the quadratic cost
this exists to remove. Out-of-range and unparseable values warn and fall back to 1024.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shared logging ThreadPoolExecutor uses an unbounded work queue, so
sync callbacks that fall behind request arrival pin every queued payload
in memory until the task restarts. Cap queued-plus-running work with a
semaphore, shed submissions past the cap, and warn at most once every 30
seconds naming the knob that raises it. No caller of the shared executor
reads the returned future, so shedding is safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(helm): compose DATABASE_URL_READ_REPLICA from a reader host secret key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(helm): cover reader host composition and readReplicaUrlKey precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): suppress unused reader host env when readReplicaUrlKey is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(helm): emit reader host only when readReplicaUrl composition is active
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The hook swallowed errors into the console, so an admin flipping the switch without
STORE_MODEL_IN_DB saw nothing happen and got no reason why. Adds the missing hook tests.
The AUTH exchange it runs is the blocking client API, so on an async
connection send_command and read_response hand back coroutines nobody
awaits and the connect fails outright.
REDIS_URL-based async clients and every async connection pool dropped the
managed-identity credential the caller configured, so they connected
unauthenticated against an auth-enforcing Redis. The conversion from
redis_connect_func to a CredentialProvider now happens once, before any
branch, and covers the url, sentinel, cluster, and pool paths alike.
Also adds credential_provider to the cluster kwargs allowlist, which
silently filtered it out.
A streaming chat completion that ends early (client disconnect, or the proxy
cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log
row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields
in usage_object. The proxy restamps chunk.model in place to the client-facing
alias, so the partial response rebuilt from those chunks priced the unmapped
alias and came out at 0. The failure path also rebuilt usage without the
request messages, so prompt tokens counted to 0, and a cut stream never sees
the final usage event that normally zero-fills the cache fields.
Restamp the rebuilt partial response with the wrapper's real model before cost
calculation on both the disconnect and the failure paths, pass the request
messages when rebuilding usage on the failure path, and zero-fill missing
cache usage fields the way completed streams already do.
Peer workers previously kept their startup value for block_requests_for_models_without_pricing
until a restart, so a toggle from the UI only took effect on the worker that served the request.
Route fal.ai's openai/gpt-image-2 endpoints through a dedicated transformation that maps OpenAI image params (n, size, quality, output_format) into fal's schema, and register the model in the cost map.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): surface the paginated fallback on Cost Optimization
The page streamed its fallback silently: useDailyActivityRange dropped
the hook's progress and cancel fields and CacheLeakageCard only showed
a loading state while empty. Extract the Usage page's fetch banner into
a shared PaginationStatusAlerts component, render it above the tabs,
and note on the cache leakage tables when pages are still arriving.
* fix(ui): gate the cache leakage streaming note on isFetchingMore only
loading also covers a fresh aggregated request over the previous
range's rows, where pagination copy mislabels stale data. Drop the
redundant component comment flagged against the repo comment policy.
