The previous iterations hand-rolled a separate cache path for the global
proxy spend scalar, and each variant reintroduced a race the shared spend
counter system was built to avoid. This routes global proxy spend through
that machinery instead:
- Write: _init_and_increment_spend_counter seeds a cold counter from the
authoritative litellm-proxy-budget row (cross-pod SET NX) and then
increments atomically, keyed spend:user:litellm-proxy-budget.
- Read: get_current_spend reads Redis-first, reseeds from the DB on a clean
miss, and only floors to the DB row when the freshly loaded fallback
exceeds the counter, repaired with the monotonic async_set_max.
- Reset: the existing user-reset path already zeroes the counter for the
proxy-budget row; the job now also invalidates the cached row-spend hint
so no worker acts on a pre-reset fallback.
Removes the bespoke coordinator, per-worker DB-floor marker, and delete-on-
reset invalidation.
Add regression tests for:
- Redis async_set_max returning None (fallback to DB floor).
- Unparseable Redis value being treated as 0.0 and floored to DB.
Improves patch coverage for the auth-time floor repair path.
Responding to Greptile/veria review:
- The global proxy spend write path no longer mirrors the Redis value into
the per-worker in-memory tier. Auth reads the shared Redis scalar directly,
so the mirror could only create confusion/staleness; removing it makes the
cross-worker contract explicit.
- The DB-floor repair now uses the value returned by async_set_max as the
enforced total, so if a concurrent increment already pushed Redis higher,
we enforce that higher value instead of the lower DB floor.
- Updated the mirror test to assert that no in-memory mirror is written.
This should resolve the 'stale worker-local spend survives reset' and
'spend repair overwrites concurrent increments' review items.
Address post-commit review from Greptile and veria:
- ResetBudgetJob now invalidates the worker-local DB-floor marker when it
resets the proxy budget. Without this, a stale pre-reset marker could be
used to write previous-window spend back into Redis.
- The auth-time DB floor now uses a monotonic Redis write (async_set_max)
instead of SET, so a concurrent increment that pushed the shared counter
above the DB value is not overwritten by a repair carrying a slightly-stale
DB total.
- Added regression tests for marker invalidation, monotonic repair, and
marker-cache reuse.
No functional change to the happy path; this only tightens edge cases around
reset and concurrent increments.
This switches to the same pattern the spend counters already use:
- Write path: Redis INCRBYFLOAT creates the counter from zero when missing
(so post-reset increments are represented immediately) and mirrors the
authoritative Redis value into local memory.
- Read path: read the shared Redis scalar directly, then floor to the
authoritative DB spend. This prevents a stale-low Redis value (from a reset
race, Redis restart, or key expiry) from underestimating current-period spend
and letting requests leak past the global cap.
- Reset job: deletes the shared key. With atomic increments and the DB floor,
this neither resurrects stale values nor clobbers concurrent increments.
Removes the now-unnecessary async_increment_if_exists helper and its tests.
Adds regression tests for the Redis-direct read and DB-floor repair.
Addresses remaining review feedback on #36953:
- greptile (P1): a worker retaining a stale pre-reset in-memory value could
satisfy the auth-time spend lookup from local memory and falsely reject
requests after a reset. With Redis configured, read the global spend scalar
straight from the shared Redis cache instead of the per-worker DualCache
in-memory tier; the coordinator still loads the authoritative DB spend on a
Redis miss.
- veria (Low): a plain SET-to-zero on reset could clobber a concurrent
increment. ResetBudgetJob now invalidates the key instead, which is safe
because reads/writes are atomic (increment-if-present) and the auth-time
loader restores the authoritative DB spend on a miss.
Adds tests for the Redis-direct auth read, missing-key, and failure-swallowing
paths to close coverage gaps.
Addresses review feedback on #36953:
- veria (High): a plain INCRBYFLOAT recreates a missing global spend key
from zero, so an evicted/expired entry loses the current-period total and
continued traffic preserves the underestimate. Add an atomic
increment-if-present Lua operation to RedisCache and use it for the global
proxy spend update, leaving an absent key missing so the auth-time loader
restores the authoritative DB spend.
- greptile (P1): the increment refreshed a stale worker-local value because
DualCache.async_increment_cache accrues in-memory. Mirror the authoritative
Redis result into the worker's in-memory tier instead, matching the
spend-counter pattern, so budget checks never read a stale pre-reset value.
Adds unit tests for increment-if-present and update_cache Redis-path behavior.
Replace the non-atomic read-modify-write on the global proxy spend
cache key with async_increment_cache, which uses Redis INCRBYFLOAT.
This prevents a stale pre-reset value from being resurrected when
ResetBudgetJob deletes the key while concurrent spend updates are in
flight.
Also change ResetBudgetJob to set the key to 0.0 instead of deleting
it, removing the cache-miss window entirely.
Adds regression tests covering the reset-then-increment scenario and
concurrent updates after reset.
Request metadata carries the whole UserAPIKeyAuth object, whose team_metadata
holds the customer's own langfuse callback_vars. The only filter on the emitted
blob was a four key deny list written as a circular reference crash guard, so
those credentials reached the customer's own langfuse traces.
The emitted blob is now the StandardLoggingPayload allowlist plus the litellm
computed enrichments, and nothing is copied across from raw request metadata.
That makes the credential exclusion structural rather than a filter someone has
to keep correct. Steering keys keep reading raw metadata, matching literal_ai.
Proxy callers are unaffected: their request metadata already rides under the
allowlisted requester_metadata key, nesting intact.
debug_langfuse dumped raw request metadata into the trace as a second copy of
the same leak. It now emits caller scalars only.
When StandardLoggingPayload is absent the trace is still emitted with the
existing trace_id fallback, so failure traces survive.
Both are covered on GitHub Actions. test-litellm-ui-build.yml runs the
dashboard build on every PR, and test-litellm-ui-unit.yml runs the vitest
suite with ui-unit-tests already a required check, so neither CircleCI job
gates anything that GHA does not already gate.
ui_build additionally produced nothing anyone consumed. It persisted
litellm/proxy/_experimental/out to the workspace, and the only job
downstream of it was ui_unit_tests, which never attached the workspace and
reinstalled from source instead. The requires edge was pure sequencing, so
the build output was written and discarded on every client-touching PR.
One real narrowing comes with this, and it is deliberate. ui_unit_tests ran
the full vitest suite on PRs, while the GHA job scopes PR runs to tests
reachable from the diff and keeps the full suite on pushes to staging. That
split was a measured decision in #34175 and it still holds: the suite is
252s and 248s of that is CreateMCPServer.integration.test.tsx alone, so
running everything per PR buys about four minutes to re-run one file.
Note that assert-ci-coverage does not speak to this. It walks
tests/**/test_*.py only, so it is blind to vitest files by construction;
it stays green here because no Python test lost a runner, which is a
narrower claim than the UI side being unaffected.
auth_ui_unit_tests is a different job, a Python suite on a Postgres
sidecar, and is untouched
* fix(proxy): track spend for OpenAI passthrough /v1/embeddings
OpenAI passthrough embeddings returned 200 but wrote no spend because the
route was unsupported and Cohere's /v1/embed prefix stole the match.
* fix(proxy): clear embeddings lint and Greptile comment nits
Inline embeddings cost tracking to avoid new LIT001/002 hits, trim
redundant doc comments, and cover the Cohere /v1/embeddings collision.
* fix(proxy): drop unreachable embeddings TypeError guard
convert_to_model_response_object with response_type=embedding already
returns EmbeddingResponse; the isinstance check was dead patch coverage.
A deployment with PTU flat-cost attribution also billed every request per
token, so a team paid for reserved capacity and again for the traffic that
capacity serves. Nothing set the per-token price and an unset price falls
back to the public cost map, which made the double charge the default.
/model/new and /model/{id}/update now store zero for every pricing field the
cost map could otherwise fill, refuse a price the caller supplies alongside
PTU config with a 400 naming the field, zero a price already on the row
rather than rejecting later edits of unrelated fields, and drop the zeros
again when the PTU config goes.
A PTU deployment is no longer read as a free model by the budget checks,
which would have waived every budget for it.
* fix(mcp): expose client HTTP headers to logging callbacks and hooks
MCP protocol tool calls built a synthetic Request with only content-type, so metadata.headers reaching logging callbacks and guardrails was empty while /mcp-rest/tools/call exposed the full set. Rebuild the synthetic request from the connection's raw headers (shared with the sampling path), and pass sanitized headers to the pre-call hook, the MCP to LLM guardrail bridge and the Responses API MCP bridge. Credential headers stay masked and proxy key headers stripped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and upstream MCP credential headers from logging copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): make client side auth header name accessor public
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and client redaction opt-out from mcp headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop custom proxy key header in the synthetic request builder
Strips general_settings.litellm_key_header_name in build_synthetic_mcp_request so every caller, including sampling, is covered, and reverts passing general_settings into add_litellm_data_to_request on the tool call path since that also switches on enforced_params.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
The Admin UI's Authorize & Fetch Token flow stored its pending server in a
module-level dict, so /register, /authorize and /token only succeeded when
every leg happened to land on the process that served /session. On a proxy
with NUM_WORKERS greater than 1, or more than one replica, each click was an
independent draw and failed with a bare 404, which reads as intermittent.
Persist the pending server as a short-lived draft row instead, so any worker
resolves it. The in-memory cache is kept as the fallback for proxies with no
database configured, which keeps single-process deployments working as before.
A session runs under a caller-supplied id only when that id names a server
that really exists, which is the edit form re-authorizing a saved server.
Anything else gets a fresh id, so two concurrent sessions can never share one
draft and silently adopt each other's URL or client credentials. Drafts past
their lifetime are swept on each write so abandoned sessions do not
accumulate, and a lost create race adopts the winner rather than failing a
caller whose session is ready.
Drafts are excluded from listings and never enter the runtime registry. The
exclusion keeps rows whose approval status is NULL, which both short spellings
of the filter drop, silently hiding every server predating the approval
workflow.
Measured on a two-worker proxy against the live GitHub MCP server, 120
concurrent authorize calls per leg: staging 56/120 failures, this branch
0/120, staging again 65/120 as a positive control.
* fix(team): sweep dangling team references and cache on team delete
delete_team drove all of its cleanup off the team's members_with_roles roster, so any
user row referencing the team by another route kept a dangling team id forever and the
deleted team stayed visible on /user/info. Nothing swept LiteLLM_UserTable.teams or
LiteLLM_TeamMembership by team id, schema.prisma declares no relation between the
membership table and the team table so there is no cascade to fall back on, and the
cached team object was never invalidated on delete.
Adds a sweep that runs before the team rows are dropped: it strips the deleted ids from
every user row that still lists them and removes every membership row for those teams.
Adds _delete_cache_team_object in auth_checks and calls it per deleted team so the
team_id:{team_id} entry cannot outlive the team.
The sweep is targeted, not indiscriminate: only the deleted ids are removed and the
other teams on a user record are left intact.
* fix(team): fail member_add when the team is deleted under the row lock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(team): correct the post-delete sweep note for the member_add lock path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): stop a deselected MCP server keeping its grant on a virtual key
The key editor sent `mcp_tool_permissions` unfiltered, and the MCP resolver
counts a server named only under `mcp_tool_permissions` as entitled, unioning
`tool_perm_servers` into `all_servers` at four sites in
`user_api_key_auth_mcp.py`. Deselecting a server, or removing the access group
that supplied it, therefore left a stale entry that kept the key reaching that
server with its old tool allowlist attached.
Reuse `extractMcpEntitlement`, which already landed for the internal-user
surface, so the key surface drops an entry only once the server is known and no
longer granted, and keeps it whenever a retained access group or toolset could
still supply it. The helper moves to a shared module so the key template does
not import a users page component.
Setting the map unconditionally is part of the same fix: the old
`Object.keys(...).length > 0` guard let the previous map ride through the
`object_permission` spread, which filtering to an empty map would otherwise hit
in exactly the case the fix is for.
* fix(ui): resolve retained MCP groups and toolsets per server when pruning tool permissions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): mock the MCP toolsets hook in the key update suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/ui/chat never rendered a metrics bar. The Responses helper already parses
usage off the response.completed event, but the chat page passed positional
undefined where onTimingData, onUsageData and onTotalLatency sit, ChatMessage
had nowhere to hold them, and ChatMessages never rendered ResponseMetrics.
Thread the three callbacks through, persist the values on the assistant
message, and reuse the playground's ResponseMetrics to show latency, TTFT,
input/output/total tokens and cost. Also map the cost the proxy reports on the
streamed usage object, which only the chat-completions helper did before.
Two behaviour differences the port introduced, both found in browser QA
The antd Select matched a prebuilt pattern on its display_name and its
internal name; a Base UI Combobox only searches itemToStringLabel, so
queries like "amex" and "sg" stopped matching. Restore the second field
with a filter predicate on the Root, covered by a regression test that
searches on a token the visible label does not contain
Base UI portals a popup into a positioner whose "isolate z-50" is fixed
in the primitive, so inside an antd Modal at z-index 1000 the options
were visible but not clickable. antd hid this because its own dropdowns
and tooltips already sat above its Modal. The positioner is not reachable
from the call site, so this needs one app-wide rule keyed on an antd
modal being present, and it becomes deletable when the last one goes
* feat(azure_ai): add Fireworks FW model pricing on Azure AI Foundry
* fix(azure_ai): drop incorrect FW-Kimi-K2.6-Code alias
* test(azure-ai): assert FW max token metadata
* feat(azure_ai): add Inkling and Nemotron 3 Ultra pricing
* fix(cli): hide codex and opencode from the lite command listings
They stay registered and invokable, so existing `lite codex` users keep
working; they just no longer show up in `lite --help` or the interactive
shell's command list.
* feat(cli): make the hidden lite command list configurable
codex and opencode are supported, so hardcoding them as hidden was wrong. Let deployments curate their own listing with `lite config set hidden_commands codex,opencode` instead; nothing is hidden by default and hidden commands stay invokable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
os.exec* has no process-replacement semantics on Windows, so `lite claude`
printed its routing line and returned to the prompt while Claude Code was left
detached without a usable console. Windows now spawns the agent, waits for it,
and exits with the child's status. Batch shims such as the npm-installed
claude.cmd go through cmd.exe because CreateProcess cannot run them directly,
and that command line is emitted verbatim with every token quoted so a spaced
path or an argument holding a shell metacharacter cannot be re-parsed by the
command processor. POSIX keeps using os.execvpe unchanged.
/team/member_delete dropped the roster entry by matching user_email against
members_with_roles, then built its user-row lookup from that same raw email
instead of from the user_id the roster entry already carries. An email the user
row does not literally hold matched nothing, so the team id stayed in the user's
teams array and the team-membership row was left orphaned while the call still
returned 200.
/team/member_add resolves an email to a user case-insensitively but stores the
caller's casing on the roster, so inviting "Alice@Example.com" for a row holding
"alice@example.com" and removing by that same string is enough to reach it.
_cleanup_members_with_roles now returns the roster entries it removed, and both
the user-row update and the membership delete run against their user ids.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
The chat metrics bar reported In/Out/Reasoning/Total/cost only, so a
playground user had no signal that provider prompt caching worked. The
cached-token counts were already visible in the Logs drawer, which meant
the answer to "does caching work here" lived on a different page.
Adds cacheReadTokens and cacheCreationTokens to TokenUsage and renders
them as two chips, reusing the prompt-cache tooltip wording already
introduced for the Logs drawer so both surfaces say the same thing.
A single helper, extractPromptCacheTokens, normalizes the three usage
shapes the playground consumes: Anthropic Messages
(cache_read_input_tokens / cache_creation_input_tokens), chat
completions (prompt_tokens_details) and the Responses API
(input_tokens_details). All three producers call it instead of parsing
per surface. Counts that are absent, zero or non-finite are dropped, so
providers without prompt caching render exactly what they render today.
CLI session tokens minted by /sso/cli/poll set team_id and team_alias but
never team_models or team_model_aliases, so the token carried a team with
none of that team's grants. /v1/models bails out to "unrestricted" when both
key_models and team_models are empty and listed the whole proxy, and team
model aliases never resolved because both can_team_access_model and the
pre-call rewrite read team_model_aliases off the token.
The team data was not close at hand: _fetch_cli_sso_team_details projected
full team rows down to team_id and team_alias before they reached the mint.
Widen that projection to include the team's models and its joined alias
table, and populate both fields at mint time.
Also stop writing the user's personal allowlist into the key models slot
when a team is bound, matching virtual-key semantics where a team-bound
credential is governed by the team grant.
Because an empty team grant is itself a real value meaning unrestricted, a
team whose grants cannot be resolved must not be minted as empty: that is
the same "unrestricted" bail-out this fix exists to close. The poll now
refuses to mint when the selected team has no complete cached detail.
That refusal is only safe because a login can no longer be pinned to a team
whose grants will never resolve. Deleting an organization drops its team
rows but leaves the memberships behind, so the login now offers only teams
whose rows still exist, and a lookup that fails outright fails the login
rather than caching a session that silently drops every team.
The repository's comment policy keeps reasoning out of the source, so the
notes on the delete dialog, the tooltip wrapper, and the test query choices
move to the pull request instead.
