A Straiker verdict that blocks the prompt now comes back as a reply (HTTP 200,
finish_reason content_filter) on chat completions, text completions, messages and
responses, the way an answer-side block already does, instead of an HTTP 400 that
some clients show as a malformed request. Routes that cannot send a reply, such as
embeddings and /guardrails/apply_guardrail, keep the HTTP 400. A blocked answer on
those routes is now an HTTP 400 too, where it was a 500. A backend failure still
fails closed with an error.
HTTP 408, 429 and 500 from Straiker now follow unreachable_fallback once retries
are spent, and an attempt that times out is no longer retried.
The new stream_mode option picks how a streamed response reaches the client.
"buffered", the default, holds it until Straiker has checked the whole response.
"streaming" sends chunks as they arrive and checks the assembled response at the end.
Each v3 phase now writes its own guardrail log entry with the session, action,
turn id and controls, in place of the generic entry that read "allow" even when
the guardrail failed open.
A claude-cli/ User-Agent is named Claude only for Claude Code entrypoints, so
other clients that share the prefix are no longer misnamed.
* fix(otel): send the Langfuse environment, release and version on OTel v2
* fix(otel): keep the Langfuse environment on the Langfuse exporter only
* fix(otel): annotate the exporter resource attributes as a read-only mapping
* fix(otel): shed the Langfuse release and version first at the attribute limit
* fix(otel): keep base attributes when the Langfuse release and version do not fit
- the model span sheds release and version before the identity fields
- the request root takes release and version only when they fit under its attribute limit
- an exporter routed to a tenant's credentials drops the proxy's Langfuse environment
- a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT
* test(otel): annotate the new attribute-limit and routing tests
* feat(proxy): add per-model project batch enqueued token limits
Enqueued batch token limits only read key and team metadata, so a deployment
that gives each application its own project under a shared team cannot cap one
application's queued batch tokens for a given model. The team limit is shared
by every project in it, and JWT or custom-auth callers get a new key scope with
every token.
Read model_batch_enqueued_token_limit from project metadata, keyed by model like
model_tpm_limit. Each limited model group the batch runs is reserved against
only its own rows' tokens, rows that name no model count toward the routing
model, and the reservation records each scope's charge so the terminal-state
refund releases exactly that. Only proxy admins may set it, on project create
and update as well as on keys and teams, so a team admin cannot lift their own
project's cap.
* fix(proxy): charge the routing model and keep daily caps for project batch limits
Rows can name any model while the batch runs on the routing deployment, for
example LiteLLM-executed batches replace every row's model with the run model,
so charging only row models let a batch skip a capped deployment's per-model
limit. Charge the routing model the whole batch as well as each other limited
row model its own rows' tokens.
An enqueued-token reservation replaced every batch counter, so a project limit
also skipped the key, team and end-user daily token caps. Keep checking those
daily caps after reserving, and refund the reservation if one rejects the batch.
* fix(proxy): keep per-minute checks for batches a project allowance does not cover
A per-model project allowance replaced every per-minute batch check as soon as
any row's model had one. A batch routed to an unlimited model could add a
single tiny row naming a limited model and skip the routing model's RPM/TPM and
project ITPM/OTPM quotas, although every row runs on the routing model. Waive
those checks only when the reservation covers the whole batch, through a key or
team allowance or a project allowance on the routing model; otherwise reserve
the covered models and run the normal checks too.
* test(proxy): cover the project per-model batch cap through custom auth on a live proxy
* fix(proxy): keep key and team batch allowances authoritative and harden project limits
Key and team enqueued allowances again replace the other batch checks. A project
allowance on the routing model keeps only the daily caps. Team admins can only be
blocked from the project key, invalid project maps answer 400, the edit modal keeps
stored limits, aliases share their tightest limit, file-bound batches charge only
the routing model, deployment ids resolve to their model group, and reservation
records keep project scopes apart so older pods still refund key and team scopes.
* fix(proxy): charge row models for model-bound batch uploads
Only a managed-file upload rewrites rows to the deployment model. A model-bound
upload keeps each row's model, so its rows still count toward the project caps of
the models they name.
* fix(proxy): skip per-minute batch limits only when the project caps every model in the batch
A project cap on the routing model alone used to switch off the project's
per-minute input and output limits for the other models a batch file names.
The skip now needs a project cap on every one of those models.
-latest names are charged under the cap of the model they point to, the
limit map only takes JSON integers, and a /project/update that sends only
top-level fields keeps the stored metadata instead of replacing it.
* fix(proxy): lift per-minute limits off managed batches under a project cap and edit the cap in the Admin UI
* fix(ui): keep the project batch cap read-only for team admins and require a whole number above 0
* fix(ui): narrow project metadata directly when keeping the stored batch cap
* fix(proxy): charge dated snapshot rows and header or query routed batches to project caps
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore: move LiteLLM's configs, cookbooks and e2e tests off gpt-4o and gpt-4.1 before their 2027-01-14 shutdown
OpenAI shuts down gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini on 2027-01-14. The example proxy
configs, cookbook guides and e2e suites now call gpt-5.6, gpt-5.6-luna or gpt-5.4-mini, and the
predicted-outputs e2e test is removed because no current OpenAI model accepts prediction
* docs(cookbook): name gpt-5.6 instead of gpt-4o as a tool-capable model in the gollem troubleshooting tips
* chore(ci): delete the observatory proxy config, which nothing has loaded since its workflow was removed
* chore: send max_completion_tokens in batch lines and move README and deploy examples to gpt-5.6
gpt-5.6-luna rejects max_tokens on OpenAI's batch lines, which reach OpenAI unchanged, so the batch JSONL builders send max_completion_tokens. The README quick start, the adaptive router README and the Helm and Terraform example configs move off gpt-4o with the configs
* chore: pin output prices next to input prices in the adaptive router example
The adaptive router reads only input_cost_per_token from config, and a deployment priced by input alone bills output at $0, so the example and its README pin both at the cost map's current prices
* chore: move the configs and e2e tests to gpt-6-luna and gpt-6.1-sol
The cost map names gpt-6-luna as the successor of gpt-4o, gpt-4o-mini, gpt-4.1
and gpt-4.1-mini. Cheap roles move to gpt-6-luna and full-size and vision roles
to gpt-6.1-sol, the pair main's other configs use. The adaptive router example
pins the gpt-6 prices, and the installer quick start and two READMEs drop gpt-4o
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user
A group the IdP pushes before LiteLLM has a team for it made the roster
write fail, which answered 404 before the user row was written, so the
user stayed active and their keys kept serving. #37700 made that failure
fatal.
Groups with no team row are now logged and left out of the roster write,
the persisted teams, and the role resolution. Group ops after a replace
now apply in order, as RFC 7644 section 3.5.2 says.
* fix(scim): drop the redundant unchanged-groups return and cover the warning and PATCH admin role
* fix(scim): strip line breaks from the teamless-group warning
* fix(scim): read the PUT role from every requested group, as POST does
A PUT naming only a group with no team no longer demotes an admin whose admin group has no team. The new tests also get typed fixtures and Final locals
* fix(scim): look up the requested teams on the primary database, as the roster write does
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions
* fix(proxy): check the key's max_budget on /queue/chat/completions
* fix(proxy): run budget, tag, and request policy checks on queued chat calls
* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls
A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.
* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses
* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls
* fix(proxy): send the call ID on a queued call the provider rejects before streaming
* fix(proxy): release a dropped stream's budget hold only when nothing bills it
A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired
* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata
A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either
* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers
* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it
* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure): send no api-version on Foundry project /openai/v1 Responses bases
Azure answers 400 "api-version query parameter is not allowed when using /v1 path" for every api-version on a Foundry project's /api/projects/<p>/openai/v1/responses route, so drop it there and keep it everywhere else
* fix(azure): send no api-version on Foundry project v1 chat and WebSocket, bill routed models
Chat completions on a Foundry project /openai/v1 base now use the v1 OpenAI client, so they send no
api-version, which that path rejects. The Responses WebSocket URL builder strips a bare /openai/v1 base
before adding /openai/v1/responses, so it no longer dials a doubled path. An azure/model-router call that
routes to a model priced only under azure_ai/ is billed from that row plus the router fee once
* fix(azure): keep the Azure client off chat-only paths, honor api_version none, drop api-version from v1 base queries
Only chat completions switch to the v1 OpenAI client on a Foundry project /openai/v1 base, so transcription and
azure_text keep the Azure client and no longer fail inside LiteLLM. A deployment with api_version none sends no
api-version on Responses and chat, for gateways that hide the project path. The v1 client now moves an api_base
query into default_query without api-version instead of gluing the route after it
* fix(azure): pick the v1 client for o-series chat, keep api_version none off a doubled /openai/responses path
* fix(azure): send no api-version on embeddings under api_version none, test async and streaming chat on v1 bases
* fix(azure): join openai/v1 once onto a v1 client base that ends in /openai
Chat and embeddings under api_version none, and chat under api_version v1, built /openai/openai/v1 from an api_base ending in /openai, while Responses on the same base already built /openai/v1. The v1 client now joins the route the same overlap-aware way
* fix(azure): keep a percent-encoded gateway path as typed on the v1 client base
Join openai/v1 by string instead of add_path_to_api_base, which decodes %2F, %25, %3F and %23 in the base path. Type the chat test helper's mode as a Literal with assert_never.
* test(azure): import assert_never from typing_extensions so the Azure tests load on Python 3.10
typing.assert_never needs Python 3.11, and the package supports 3.10
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them
A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to
prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a
function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode
caching never wrote. The bridge now renames the marks before converting the messages when the
served deployment reads breakpoints, using the same gate the Responses config uses for native
/v1/responses input. Deployments that do not read breakpoints keep today's behavior.
The rename also leaves unmarked content parts untouched instead of validating them, so a part with
non-string keys no longer raises on either path.
* fix(responses): keep leading system messages in client order once one goes to input
A marked first system string became a list and moved to input while the
next unmarked string stayed in instructions, so the model read them in
reverse. The leading run that joins instructions now stops at the first
system message whose content is not a string
* fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge
Bedrock Mantle already caches the growing prefix on its own, so turning
cache_control into breakpoints there added write charges. The client-order
rule for leading system messages now applies only when a mark turns a system
string into a list, so unmarked chats send the same request as before.
* test(responses): cover a marked user turn in the unmarked system split test
A cache_control mark on the user message must not switch unmarked leading
system messages to the client-order rule. The split test now runs with the
user turn plain and marked, so dropping the role check from the system
string test fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): honor cancel_on_disconnect on the rerank endpoints
* test(proxy): cover a connected rerank succeeding with cancel_on_disconnect on
* test(proxy): cover a connected rerank answer through the cancel watcher in the integration rig
* fix(proxy): read cancel_on_disconnect on rerank the same way the chat path does
* fix(proxy): set up the rerank request logger before routing so a hang-up after dispatch logs the query's tokens like chat
* fix(proxy): keep flag-off rerank logging unchanged and price a cut-off rerank from its query and documents
The rerank endpoint now builds its request logger only under cancel_on_disconnect, so a flag-off rerank logs one unbilled 408 row as on main.
The failure estimate for rerank counts the query and every document sent, prices the deployment the router picked, and bills one query on a model priced per query.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(mistral): add native upstream token counting support
* test(mistral): standardize on mistral-large-4 and add tool calls tests
* fix(mistral): unify credential resolution and guard Azure endpoint boundary
* fix(mistral): resolve api base from provider env vars and route responses count natively
* fix(test): use groq for bridged proxy tests and increase mistral coverage
Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.
* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter
* fix(mistral): suppress basedpyright warnings on untyped httpx handler
* fix(mistral): keep chat credential resolution and count with the key chat resolves
The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.
* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route
* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them
Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.
The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.
Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.
* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does
Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.
* fix(mistral): count what chat sends and keep the bundled Mistral model list
A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape
A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills
Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400
Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate
MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*
The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run
* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral
* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout
Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy
* fix(mistral): join count text blocks the way Mistral chat does
* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat
* fix(mistral): count the thinking chunks Mistral chat replays in assistant history
The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): reuse one Google token across vector store searches
* fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst
* test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy
* test(vertex_ai): type the fixtures and token callback in the concurrent search test
* fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects
A shared RAG Engine config made one store's slow token fetch hold up every
other store, and kept a rotated key file's deleted key until a restart.
First loads and expired refreshes now take a lock per credentials and project,
refreshes take a lock per credentials object, and any RefreshError reloads the
credentials once.
* fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls
* test(vertex_ai): check async callers sharing an expired credential refresh it once
* fix(vertex_ai): reload Google credentials only when a refresh is rejected for good
A retryable refresh error such as a 503 now raises without reloading the key, so an outage
does not double google-auth's retries. Credentials that cannot be weak-referenced share the
config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase
and holds the token response until the other callers have started
* fix(vertex_ai): reload Google credentials only when Google answers invalid_grant
google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads
The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did
* fix(vertex_ai): pick up a swapped Google key file on the next call
The credential cache key now carries the key file's inode, mtime and size, for a
vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced
or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of
after Google rejects the old key or the proxy restarts
* fix(vertex_ai): keep the earlier key while a replaced key file does not load
A key file caught halfway through a non-atomic write, or replaced by one that does not load,
now keeps the credentials loaded from its earlier version in use and logs a warning, as the
cache did before key files were re-read. The next complete file is loaded on the next call.
The pub/sub test no longer writes a key file it only reaches through the seeded cache
* fix(vertex_ai): try a replaced key file again when its first token fetch fails
The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one
* fix(vertex): raise when an async reload keeps an earlier key Google rejected
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project
Adds the opt-in litellm.vertex_request_label_fields setting, the Vertex twin of bedrock_request_metadata_fields. When set, Gemini, embeddings, Imagen and Gemini image generation, rerank, and Claude on Vertex (chat and /v1/messages) send the listed proxy identity fields and spend_logs_metadata as request labels, identity first, with the user_api_key_ prefix reserved so a caller cannot forge or evict them. Identity text is rewritten to fit Google's label rules. Gemini image generation now forwards labels at all.
* feat(vertex_ai): send the opt-in identity labels on multimodal embeddings, the Google GenAI route, and batches
With litellm_settings.vertex_request_label_fields on, the multimodal embedding
predict body, the native generateContent and streamGenerateContent body, and the
BatchPredictionJob now carry the key's identity labels the same way chat does
Caller labels on GenAI are resolved after the extra_body merge, so neither a
top-level labels field nor extra_body.labels can write the reserved prefix. Batch
labels come only from the proxy's litellm_metadata, never the caller's OpenAI
batch metadata. With the setting off every route sends the same body as before
* fix(vertex_ai): keep identity labels when a Gemini caller sends extra_body labels
* fix(vertex_ai): label Interactions, image edits, Mistral, and the /vertex_ai pass-through, and gate Gemini image labels on the setting
* fix(vertex_ai): move body labels into the labels header on Vertex rawPredict pass-through calls
* fix(vertex_ai): send Mistral extra_body labels in the labels header instead of the body
Labels a caller nests in extra_body now merge over top-level labels before the
X-Vertex-AI-Labels header is built, the way the Gemini body merges them, and
leave the outbound body. The token-fetch stubs in the Mistral and multimodal
embedding tests are scoped to their instance so later tests see the real method
* fix(vertex_ai): limit pass-through labels to POST and to publishers that take the labels header
rawPredict to custom endpoints and publishers other than Anthropic and Mistral, and PATCH or PUT bodies, now reach Vertex as the caller sent them. The pass-through drops a caller's labels header where it writes body labels, a single field written as text turns the setting on for that field, and an undecodable Claude labels header falls back to the caller's labels
* fix(vertex): label Veo video, Gemini transcription, and pass-through predictLongRunning requests
* fix(vertex): read the pass-through publisher from the end of the upstream path
A project id named publishers shifted the first-match lookup, so a forged X-Vertex-AI-Labels header reached Claude or Mistral untouched
* fix(vertex): drop a caller labels header on body-labelled Vertex routes while identity labels are on
* test(vertex): script the v1beta1 GenAI target in the labels header integration test
* test(vertex): type the transcription test's access token override
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): answer 400, not 500, for a message without role or content
Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.
* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count
* refactor(bedrock): leave message validation to the Bedrock count handlers
Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy
* fix(count_tokens): name the chat message chat translation cannot send
A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back
* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content
Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it
* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse
The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate
* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse
A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again
* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given
* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks
* test(integration): expect the 400 for a count whose leading system message has no content
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route
/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.
DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.
* perf(proxy): delete count drop paths from one copy of the options
additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear
* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing
Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills
Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400
additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500
The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models
* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's
A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference
A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does
The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any
* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude
A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status
/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none
* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking
A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts
A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it
* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking
A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count
* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle
* fix(bedrock): send the thinking beta headers on Mantle counts
A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends
* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test
* fix(deepseek): count only the tools DeepSeek chat inference sends
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep carried end-user spend when a budget reset is retried
End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.
Fixes#50898
* fix(proxy): leave never-reset tiers unguarded and test each tier's own window
A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.
* test(proxy): give the end-user ledger its own fake prisma client
* fix(proxy): pin a reset time on tiers without one before end-user pages commit
* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced
* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset
* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration
* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window
* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time
A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users
* fix(proxy): name each end-user reset window by the tier's last write
Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due
* fix(proxy): name end-user reset windows by the tier and its advance count
Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.
* fix(proxy): keep carried spend for customer teams and members on a retried reset
Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.
The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.
* test(integration): list every budget column and relation in the budget update field check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.
Also restores behavior the Rust cost map move changed for SDK callers:
- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
utils._runtime_registered_model_cost, which downstream test code replaces
litellm_core_utils/AGENTS.md now says which removed names come back.
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry
The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction
delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.
* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too
The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.
* fix(proxy): delete the auth cache entry before the generation write inside the script
A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.
* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test
* fix(proxy): refuse a leased auth cache write for a key no generation key can guard
* test(proxy): undo the unrelated reformat of the spend ledger parity tests
* test(redis-store): mark the updated memory-only auth cache test with rstest
* chore(proxy): drop two pointer comments from the cache lease code
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop a customer override expiry sent without an override
* test(proxy): clear the override before deleting the row so the integration test leaves no budget behind
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): reset long unit spellings like 12months on the schedule they are charged for
* test(budgets): match the 3mo error message in the long month spelling test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias
* test(proxy): a pinned key naming an unknown org alias is refused with 403 even when registration would fail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): invalidate customer hierarchy caches through stored team IDs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: restore customer budget alerts after hierarchy schema changes (#51645)
* fix(proxy): upsert customer orgs and customer members safely (#51607)
* fix(proxy): upsert customer orgs and customer members safely
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): lock customer org snapshots during upsert
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): reject empty customer org upsert identifiers
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): reuse the customer org when its alias is already registered
A request for an alias that another worker registered a moment earlier got
alias_already_exists back from the create and failed. The registration now
treats that the same as an existing id and returns the registered customer org.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* test(spend): synchronize concurrent member resets before validation completes (#51655)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): invalidate customer hierarchy caches through stored team IDs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(budgets): supply async customer team lookups in caller fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishaan-j-berri <ishaan.j@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): reset customer org, team and member spend on their budget row's window
The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.
* feat(budgets): support concurrent customer budget windows
* fix(budgets): resolve compatibility and validation checks
* feat(proxy): name budget rows, give customer teams an override and members a parent org
* fix(budgets): preserve admitted windows during accounting
* chore(schema): allow dropping the unreleased customer budget_reset_at columns
* fix(budgets): apply standard migration lock timeout
* feat(proxy): customer members belong to many teams, charged to the one the request names
* test(migrations): allow the unreleased customer budget_reset_at drop
* style(ui): prettier the customer dashboard files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(reset-job): name the batch bound on the customer budget IN filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(budgets): preserve mixed limits and resolve CI regressions
* fix(budgets): reuse shared spend verification for customer windows
* refactor(ui): isolate customer budget group creation
Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.
* fix(proxy): restore Field import and bound the member team lookup after the merge
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): send parent_team_ids in the budget window customer tests
Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(proxy): cover customer budget increase requests and the cap_policy field
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* chore(ui): regenerate API types for cap_policy and customer budget increase requests
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): hang the increase request back-reference on the customer member model
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): declare increase request list filters with Annotated Query
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(providers): keep registration lookups in the model catalog
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(providers): format catalog registration lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences
Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call
fallback walk reads router settings Key > Team > Global instead of the key's
column alone, and skips a fallback the caller may not call.
A key or team with router_settings.fallback_on_saturation: true walks that same
fallback list as soon as the requested model reaches saturation_threshold in
dynamic_rate_limiter_v3, and goes back to the requested model when no fallback
admits the request. Responses carry x-litellm-fallback-reason:
rate_limit:model_saturation_fallback.
Removes the team_metadata.model_preferences steering and the
x-litellm-model-steered-from header from #51350, which did the same job through
a second knob.
* fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests
fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call.
* fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does
A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access
groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a
fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client
body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team
router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it
cannot pass before saturation.
* refactor(proxy): one helper decides which refused fallbacks end the pre-call walk
_pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller.
* fix(proxy): keep the caller's request object through the pre-call fallback walk
Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place.
* fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes
A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it).
* refactor(proxy): read fallback prices through the router registry entry point
router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info.
* fix(proxy): a fallback's group budget check skips the groups the request already holds
The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read.
* test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering
A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle.
* fix(proxy): the saturation handoff only fires under the fallback walk
_pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429.
* style(proxy): keep the cast-ok reason within the line length
* fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only load the end user for fallback checks when a database is connected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model
The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share
reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request
after admission, and _release_pre_call_hold calls it next to the registered limiter.
An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the
request on to a further alias. It now replays with the model argument the first attempt saw.
The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): enforce customer org, team and member budgets and limits on the request path
Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): customer hierarchy survives virtual key auth without a legacy end user
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): keys pinned to a customer organization may only act for that organization's members
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): keep the customer hierarchy spend maps within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): check the key's customer org scope after the key is resolved
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the hierarchy cache key family and keep type counts flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): resolve the customer hierarchy in the shared authorization gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop a member's cached hierarchy through the declared cache target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(auth): keep the hierarchy reset within the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): type the requested models as a sequence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend): add daily customer usage and budget resets (#51493)
* feat(spend): add daily customer usage and budget resets
* chore(spend): document customer reset query bounds
* fix(spend): chart customer queue metrics and update recovery test
* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit ccac47608f.
* fix(proxy): evict cached member hierarchies when team, org or template budgets change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the auth cache dependency alias as a TypeAlias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"
This reverts commit 7542471e17.
* fix(spend): reconcile the daily customer spend table with main
Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): tolerate failure metadata without customer hierarchy ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): key daily customer spend rows on main's hashed identity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(pagerduty): read customer ids from failure metadata through a typed lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
Deployment affinity and the complexity router's session and tier pins move
off the router's DualCache onto RouterSessionPinStore, backed by the native
redis-store. The read and claim Lua scripts run byte-identical to main, the
pin read rides the request's pre-call batch, and keys and values keep main's
format.
The --target builder image copies the repo root to /app, so after the
schema moved under litellm/ it lives at /app/litellm/schema.prisma there.
Only the runtime stage adds the /app/schema.prisma alias.
Claude-Session: https://claude.ai/code/session_01Bne5zS8tnYUDfKJYENWcYA
Co-authored-by: Claude <noreply@anthropic.com>
* feat(ui): set per-threshold budget alert recipients on virtual keys
Create Key and Edit Key get the threshold and recipients rows the team
member budget alerts already use, writing metadata.max_budget_alert_emails.
The helpers and the row editor move to key_team_helpers so the team form
and both key forms share them.
* test(ui): cover the empty metadata box path and drop a redundant payload case
* fix(ui): keep comma-separated string alert recipients stored through the API
The backend accepts a string of recipients per threshold, so Edit Key and the
team form now load them as rows instead of dropping them and deleting the
alert on the next save
* fix(ui): keep the stored key alert map on saves that do not edit its rows
Edit Key rebuilt max_budget_alert_emails from the rows on every save, so
API-written entries the rows cannot show (a null recipient list, a list with
a null, thresholds outside 1 to 100) were deleted by unrelated edits. The
map is now rebuilt only when the rows were edited and otherwise sent back
as stored
* fix(ui): load null and mixed alert recipient lists as rows and drop a typed map on Create Key
A null recipient list loads as an owner-only row and a mixed list keeps its
strings, matching what the backend sends. Create Key now ignores a
max_budget_alert_emails typed into the Metadata box, as Edit Key does, and
the hint says thresholds replace the default 80% alert
* fix(ui): keep Create Key alert rows when the Metadata box holds null
* fix(ui): move typed alert maps into rows, keep hidden alert entries, hide alert rows from team admins
A max_budget_alert_emails map typed into the Metadata box on Edit Key or Create Key now moves
into the threshold rows on blur with a toast, the same way a tags array moves to the Tags field.
Entries the rows cannot show stay in the JSON box.
When the rows are edited, the saved map keeps the stored entries the rows cannot show (thresholds
outside 1 to 100, non-numeric keys, unreadable values). Create Key keeps such typed entries too.
Team admins editing a member's key no longer see the alert rows, since that save only accepts
budget fields
* fix(ui): let the Metadata box own key alert entries the threshold rows cannot show
Edit Key now shows max_budget_alert_emails entries outside whole numbers 1 to 100 in the
Metadata JSON box and saves them from the box as the admin left it, so they can be seen,
edited and removed. Rows still own every entry they can show, and an untouched save sends
the stored map unchanged
* fix(ui): leave zero-padded alert thresholds to the Metadata box instead of duplicating a row
* fix(ui): keep loading zero-padded team member alert thresholds as rows
The canonical-key rule applies only to key alert maps, where the Metadata box
holds entries the rows cannot show. The team form has no such fallback, so it
keeps loading "080" as row 80 as it did before
* fix(ui): let the Metadata box own a non-object max_budget_alert_emails value
A string, list, number or null stored under max_budget_alert_emails stays in the
Edit Key Metadata box, is saved back as the box holds it, and is stored as typed
on Create Key. Threshold rows only own an object map, so rows the admin adds
replace a non-object value
* fix(ui): move typed alert thresholds into the rows when Edit Key saves without a blur
Submit now moves a max_budget_alert_emails map typed into the Metadata box
into the rows, as it already does for tags, and reads the edited fields at
submit time so the moved rows are saved
* refactor(ui): name the Edit Key submit handler
* fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates
The per-team share counters only count requests admitted while the model is
saturated, so what a team used while the model was idle is not charged
against its share once contention starts. A team below its share is still
admitted ahead of a borrower, and the model's own limits bound the total.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): idle fallback targets and WebSocket turns follow the same share accounting
A router fallback target attempted while idle no longer holds the team's
share, and a Responses WebSocket connection refreshes its share descriptor
on every turn, so idle turns are forgiven and saturated turns count.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): an idle fallback target still records its hold
Settlement finds the hold, gives the admitting model's reservation back and
charges nothing for the idle target, instead of charging the target's
tokens to the admitting model's share. The integration fixture builds its
configuration without mutating the loaded one.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
