request_id is the primary key of LiteLLM_SpendLogs and the flush inserts with
skip_duplicates, so a spend log whose id already exists is dropped with no error
raised and a "processed 1 spend log" line still logged. Batch cost accounting
produced exactly such an id twice over, and on a proxy with message redaction
enabled no batch cost row could be written at all.
get_spend_logs_id derived the id by md5-hashing the response for two call types,
aretrieve_batch and acreate_file. Redaction makes that hash a constant:
perform_redaction returns the fixed {"text": "redacted-by-litellm"} placeholder
for any shape it cannot redact, which is what a batch object and a file body both
become, so every such row hashed to md5('{"text": "redacted-by-litellm"}') =
00fcbef15a3b0097e14b0ca016ed30a0 regardless of provider, user, or amount. The
first row to claim that id owned it and every later row was discarded. Verified
against a live proxy: four payloads spanning two providers and three distinct
spend values all computed that id, and the table held one acreate_file row dating
to 2025-05-25, the row that had claimed it.
Keying off the batch's own identity instead is necessary but not sufficient,
because creating a batch already writes an acreate_batch row under exactly that
id, so the cost row becomes a duplicate of the batch's own creation row. Also
verified live: after the hash was removed the poller computed and flushed a
batch's cost, and the only row carrying that id was the acreate_batch row from
when the batch was submitted.
The id now comes from the response's own id, then the standard logging payload's
id, then litellm_call_id, and a batch cost row is namespaced with a _batch_cost
suffix so it cannot collide with the creation row. The middle term is what keeps
this correct under redaction: that payload is built from the unredacted response,
so it still carries the batch id after redaction has flattened the body. Keying
the cost row to the batch rather than to the call also keeps accounting the same
batch twice collapsing to one row instead of billing it twice. Every other call
type still derives its key exactly as before.
Cost and usage themselves are unaffected by redaction: the token columns fall back
to the standard logging payload and spend comes from its response_cost, neither of
which redaction touches. generate_hash_from_response had no other caller and is
removed with it.
* fix(proxy): track spend for OpenAI passthrough /v1/embeddings
OpenAI passthrough embeddings returned 200 but wrote no spend because the
route was unsupported and Cohere's /v1/embed prefix stole the match.
* fix(proxy): clear embeddings lint and Greptile comment nits
Inline embeddings cost tracking to avoid new LIT001/002 hits, trim
redundant doc comments, and cover the Cohere /v1/embeddings collision.
* fix(proxy): drop unreachable embeddings TypeError guard
convert_to_model_response_object with response_type=embedding already
returns EmbeddingResponse; the isinstance check was dead patch coverage.
A managed batch whose request lines all failed can reach a terminal provider
status (completed) with output_file_id=None and only an error_file_id. Such a
row matched neither the completed-with-output billing branch nor the
failed/expired/cancelled branch, so batch_processed stayed False and the poller
re-selected it on every cycle for the lifetime of the deployment; output/error
file deletion is also gated on batch_processed, so those files could never be
deleted.
Broaden the terminal handling so a completed/complete/expired batch with an
output file is billed, and any terminal batch with nothing to bill
(failed/cancelled, or completed/expired with no output) is marked terminal
exactly once. Non-terminal statuses (validating/in_progress) are still left for
the next poll, and an expired batch that did produce output is now billed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A deployment with PTU flat-cost attribution also billed every request per
token, so a team paid for reserved capacity and again for the traffic that
capacity serves. Nothing set the per-token price and an unset price falls
back to the public cost map, which made the double charge the default.
/model/new and /model/{id}/update now store zero for every pricing field the
cost map could otherwise fill, refuse a price the caller supplies alongside
PTU config with a 400 naming the field, zero a price already on the row
rather than rejecting later edits of unrelated fields, and drop the zeros
again when the PTU config goes.
A PTU deployment is no longer read as a free model by the budget checks,
which would have waived every budget for it.
* fix(mcp): expose client HTTP headers to logging callbacks and hooks
MCP protocol tool calls built a synthetic Request with only content-type, so metadata.headers reaching logging callbacks and guardrails was empty while /mcp-rest/tools/call exposed the full set. Rebuild the synthetic request from the connection's raw headers (shared with the sampling path), and pass sanitized headers to the pre-call hook, the MCP to LLM guardrail bridge and the Responses API MCP bridge. Credential headers stay masked and proxy key headers stripped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and upstream MCP credential headers from logging copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): make client side auth header name accessor public
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and client redaction opt-out from mcp headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop custom proxy key header in the synthetic request builder
Strips general_settings.litellm_key_header_name in build_synthetic_mcp_request so every caller, including sampling, is covered, and reverts passing general_settings into add_litellm_data_to_request on the tool call path since that also switches on enforced_params.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Spend-update transactions increment non-idempotent counters
(spend = spend + x) inside prisma interactive transactions. Every retry
loop only caught DB_RETRY_SAFE_ERROR_TYPES (httpx.ConnectError); a
Postgres deadlock (SQLSTATE 40P01, surfaced by prisma as transaction
conflict code P2034) fell through to a bare except that re-raised
immediately, so on multi-pod / high-concurrency deployments any pod that
lost a deadlock silently dropped its increment.
A deadlock is replay-safe even though the increment is non-idempotent:
Postgres aborts and fully rolls back the victim transaction, so no
partial spend is committed. Add PrismaDBExceptionHandler.is_deadlock_error
and route every spend path (user, end-user/key, team, team_member, org,
tag/agent via _update_entity_spend_in_db, and the daily-spend upsert)
through a shared _handle_spend_update_failure that retries connection
errors and deadlocks with randomized jitter backoff and re-raises
everything else or on exhaustion.
The Admin UI's Authorize & Fetch Token flow stored its pending server in a
module-level dict, so /register, /authorize and /token only succeeded when
every leg happened to land on the process that served /session. On a proxy
with NUM_WORKERS greater than 1, or more than one replica, each click was an
independent draw and failed with a bare 404, which reads as intermittent.
Persist the pending server as a short-lived draft row instead, so any worker
resolves it. The in-memory cache is kept as the fallback for proxies with no
database configured, which keeps single-process deployments working as before.
A session runs under a caller-supplied id only when that id names a server
that really exists, which is the edit form re-authorizing a saved server.
Anything else gets a fresh id, so two concurrent sessions can never share one
draft and silently adopt each other's URL or client credentials. Drafts past
their lifetime are swept on each write so abandoned sessions do not
accumulate, and a lost create race adopts the winner rather than failing a
caller whose session is ready.
Drafts are excluded from listings and never enter the runtime registry. The
exclusion keeps rows whose approval status is NULL, which both short spellings
of the filter drop, silently hiding every server predating the approval
workflow.
Measured on a two-worker proxy against the live GitHub MCP server, 120
concurrent authorize calls per leg: staging 56/120 failures, this branch
0/120, staging again 65/120 as a positive control.
* fix(team): sweep dangling team references and cache on team delete
delete_team drove all of its cleanup off the team's members_with_roles roster, so any
user row referencing the team by another route kept a dangling team id forever and the
deleted team stayed visible on /user/info. Nothing swept LiteLLM_UserTable.teams or
LiteLLM_TeamMembership by team id, schema.prisma declares no relation between the
membership table and the team table so there is no cascade to fall back on, and the
cached team object was never invalidated on delete.
Adds a sweep that runs before the team rows are dropped: it strips the deleted ids from
every user row that still lists them and removes every membership row for those teams.
Adds _delete_cache_team_object in auth_checks and calls it per deleted team so the
team_id:{team_id} entry cannot outlive the team.
The sweep is targeted, not indiscriminate: only the deleted ids are removed and the
other teams on a user record are left intact.
* fix(team): fail member_add when the team is deleted under the row lock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(team): correct the post-delete sweep note for the member_add lock path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(azure_ai): add Fireworks FW model pricing on Azure AI Foundry
* fix(azure_ai): drop incorrect FW-Kimi-K2.6-Code alias
* test(azure-ai): assert FW max token metadata
* feat(azure_ai): add Inkling and Nemotron 3 Ultra pricing
* fix(cli): hide codex and opencode from the lite command listings
They stay registered and invokable, so existing `lite codex` users keep
working; they just no longer show up in `lite --help` or the interactive
shell's command list.
* feat(cli): make the hidden lite command list configurable
codex and opencode are supported, so hardcoding them as hidden was wrong. Let deployments curate their own listing with `lite config set hidden_commands codex,opencode` instead; nothing is hidden by default and hidden commands stay invokable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register Nano Banana 2 Lite on the unprefixed, gemini/, and vertex_ai/ keys
so completion_cost and pass-through spend tracking no longer treat the
model as unmapped
os.exec* has no process-replacement semantics on Windows, so `lite claude`
printed its routing line and returned to the prompt while Claude Code was left
detached without a usable console. Windows now spawns the agent, waits for it,
and exits with the child's status. Batch shims such as the npm-installed
claude.cmd go through cmd.exe because CreateProcess cannot run them directly,
and that command line is emitted verbatim with every token quoted so a spaced
path or an argument holding a shell metacharacter cannot be re-parsed by the
command processor. POSIX keeps using os.execvpe unchanged.
/team/member_delete dropped the roster entry by matching user_email against
members_with_roles, then built its user-row lookup from that same raw email
instead of from the user_id the roster entry already carries. An email the user
row does not literally hold matched nothing, so the team id stayed in the user's
teams array and the team-membership row was left orphaned while the call still
returned 200.
/team/member_add resolves an email to a user case-insensitively but stores the
caller's casing on the roster, so inviting "Alice@Example.com" for a row holding
"alice@example.com" and removing by that same string is enough to reach it.
_cleanup_members_with_roles now returns the roster entries it removed, and both
the user-row update and the membership delete run against their user ids.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
CLI session tokens minted by /sso/cli/poll set team_id and team_alias but
never team_models or team_model_aliases, so the token carried a team with
none of that team's grants. /v1/models bails out to "unrestricted" when both
key_models and team_models are empty and listed the whole proxy, and team
model aliases never resolved because both can_team_access_model and the
pre-call rewrite read team_model_aliases off the token.
The team data was not close at hand: _fetch_cli_sso_team_details projected
full team rows down to team_id and team_alias before they reached the mint.
Widen that projection to include the team's models and its joined alias
table, and populate both fields at mint time.
Also stop writing the user's personal allowlist into the key models slot
when a team is bound, matching virtual-key semantics where a team-bound
credential is governed by the team grant.
Because an empty team grant is itself a real value meaning unrestricted, a
team whose grants cannot be resolved must not be minted as empty: that is
the same "unrestricted" bail-out this fix exists to close. The poll now
refuses to mint when the selected team has no complete cached detail.
That refusal is only safe because a login can no longer be pinned to a team
whose grants will never resolve. Deleting an organization drops its team
rows but leaves the memberships behind, so the login now offers only teams
whose rows still exist, and a lookup that fails outright fails the login
rather than caching a session that silently drops every team.
Against a real Postgres the previous commit still died on MonthlyGlobalSpend:
only 2 of the 8 creation sites went through the tolerant helper, so the losing
replica re-raised on the first unguarded one and skipped the rest.
The regression test now makes every CREATE lose the race and asserts all 8 are
still attempted, which fails on the partial fix.
Every replica booting against the same fresh database sees each view as
absent and issues the CREATE. Postgres fails all but one with a
duplicate-object error, and that exception propagated out of
create_missing_views, so every view after the first was never created and
/global/spend* 500'd for the life of the deployment.
Losing that race reaches the desired end state, so treat it as success.
Genuine DDL errors still propagate.
Every case in TestAccessControl asserted that something was refused. A gateway
that denied the allow-listed model too would have passed all of them, so the
suite could not tell "denied correctly" from "broken outright".
Adds the positive half: a key allow-listed for gemini-2.5-flash can call it and
gets back a real completion rather than a 200-wrapped error.
Also tightens the unknown-model case. It accepted any valid JSON, so a bare
"{}" or even "null" satisfied it. It now requires the OpenAI-shaped error
envelope with a message a client can actually surface, parsed through a typed
model instead of json.loads.
Completed batches that contain only failed requests do not generate an
output file, leaving output_file_id unset while the failures are recorded
through error_file_id instead.
The completion handler attempted to read the output payload regardless of
whether an output file actually existed. During retrieve polling this caused
the logging pipeline to fail with "Output file id is None cannot retrieve
file content", preventing normal completion bookkeeping from running.
Skip output retrieval when no output file is available and return an empty
batch summary (zero usage, zero cost, no model entries). The lower-level
file retrieval helper still reports an error if it is called directly with
an invalid or missing file identifier.
Closes#33987
* fix(proxy/batches): stop forwarding custom_llm_provider twice in list and cancel
The model-routing branches of list_batches and cancel_batch passed
custom_llm_provider as an explicit kwarg while also leaving it inside the dict
they splat, so every such call raised "got multiple values for keyword argument
'custom_llm_provider'" and returned a 500.
list_batches SCENARIO 2 called data.update(credentials) but never removed
custom_llm_provider before litellm.alist_batches(custom_llm_provider=..., **data);
it now uses prepare_data_with_credentials, the same helper the create and
retrieve branches already use, which pops it out.
cancel_batch SCENARIO 3 resolved the provider with
`provider or data.pop("custom_llm_provider", None) or ...`, so when the path
param provider was set the pop short-circuited and a body custom_llm_provider
stayed in data and collided with the explicit kwarg. The body value is now
popped unconditionally before the fallback chain, so the path param wins cleanly
and data no longer carries a duplicate.
Both paths already had strict-xfail regression tests documented "remove when
fixed"; those markers are dropped so the tests now guard the fix.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
* fix(proxy/files): avoid duplicate custom_llm_provider in list
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
---------
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
