Commit graph

41837 commits

Author SHA1 Message Date
mateo-berri
bac17a9248 fix(ci): scope UI lint to the files a PR actually changed
`UI Lint / frontend-lint` collected its file list from
`"$BASE_SHA"...HEAD`, where `BASE_SHA` is the base branch tip captured when
the PR was opened and `HEAD` is the merge of the PR into the *current* base
tip that actions/checkout leaves behind. The three-dot merge base of those two
is `BASE_SHA` itself, so the diff spans every base-branch commit landed since
the PR was opened.

Any PR opened before an eslint violation landed on the base branch therefore
fails on files it never touched. PR #34192 changes two Python files and no UI
file at all, and the job still linted 283 dashboard files and failed on three
`no-restricted-imports` antd errors from unrelated commits.

Diffing the PR head against its own merge base gives exactly the files the PR
changed, whether the checkout leaves HEAD on a merge commit or on the head
commit.
2026-07-24 20:31:27 -07:00
tin-berri
b9b27c2beb
Merge pull request #34582 from BerriAI/litellm_toolspend_30d_bound
fix(proxy): cap /v1/tool/spend window at 30 days and bound every SpendLogs read
2026-07-24 20:05:42 -07:00
Tin Chi Lo
d478b9955e fix(anthropic-adapter): drop redundant casts to satisfy type-discipline budget
_apply_reasoning_summary_wrapping already returns Any, so wrapping its
dict-literal returns in cast(Any, ...) was a no-op that only inflated the
LIT006 cast-count budget the lint gate enforces.
2026-07-24 20:01:43 -07:00
mateo-berri
85ad6971e9 fix(batches): keep an empty after meaning "start from the beginning"
Validating the cursor whenever `after` was non-None turned `?after=` into a
400, which the listing has always read as "no cursor". Only a cursor the
client actually sent is looked up now, matching the sibling managed-resource
listing.
2026-07-24 19:49:13 -07:00
mateo-berri
c1ea54a1d0 fix(batches): reject unresolvable list cursors and derive has_more from row count
An `after` that does not resolve to a batch the caller can list now returns
400 instead of an empty page. An empty page is indistinguishable from the end
of the list, so a stale or malformed cursor silently truncated a client's batch
list. The lookup is scoped to the caller's own rows, so a Prisma cursor can no
longer be anchored to another user's batch.

`has_more` now comes from whether an extra row exists rather than from whether
the page came back full. Reporting fullness made every client fetch one extra
empty page when the batch count was an exact multiple of `limit`, and made a
page shortened by an unparseable row look like the end of the list, hiding the
older batches behind it.

Also drops the unreachable `target_model_names` oversampling branch; that
argument raises a few lines above it.
2026-07-24 19:33:33 -07:00
Tin Chi Lo
5072590c27 fix(anthropic-adapter): dedupe reasoning_effort wrapping to close sibling gap
translate_thinking_for_model duplicated the same summary/auto_summary
wrapping logic as _translate_thinking_to_openai without the
disabled-thinking guard, so it could still wrap "none" into an
{effort, summary} dict when reasoning_auto_summary is enabled (caught
by Cursor Bugbot). Extract the wrapping rule into one shared
_apply_reasoning_summary_wrapping helper used by both call sites so
this invariant can't drift apart again.
2026-07-24 19:31:29 -07:00
Mateo Wang
7b019cf152
Merge pull request #34446 from BerriAI/litellm_fix_mantle_missing_usage
fix(bedrock-mantle): backfill usage on non-streaming /v1/messages responses
2026-07-24 19:05:40 -07:00
Tin Chi Lo
fed03a41d1 test(anthropic-adapter): cover empty stop_sequences edge case
Codecov flagged the empty-list early-return in
_translate_stop_sequences_to_openai as an uncovered line in the diff —
add a regression test asserting stop_sequences=[] does not set
new_kwargs["stop"].
2026-07-24 19:00:20 -07:00
Yuneng Jiang
9c48ad41ac
fix(passthrough): honor the zero fallback and aggregate-only TPM usage
Two follow-ups from review on the upstream-reported usage contract.

An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.

The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
2026-07-24 18:49:41 -07:00
Tin Chi Lo
9da21f38a9 fix(anthropic-adapter): keep disabled-thinking reasoning_effort a plain string
Guard against reasoning_auto_summary wrapping "none" into a dict when
thinking is disabled — there's no reasoning trace to summarize, and
non-Claude providers (e.g. Fireworks) expect reasoning_effort as a
plain string.
2026-07-24 18:48:07 -07:00
Tin Chi Lo
b3e27a0bc3 fix(anthropic-adapter): translate stop_sequences and disabled thinking for non-Claude targets
Claude Code's auto-mode classifier sends stop_sequences and thinking:
{type: disabled} on /v1/messages. The Anthropic adapter passed
stop_sequences through unchanged instead of mapping it to OpenAI's stop,
which Fireworks' OpenAI-compatible endpoint rejects with HTTP 400. It also
dropped disabled thinking instead of mapping it to reasoning_effort: none,
so the model spent its output budget on reasoning it was told to skip.

Resolves LIT-4798
2026-07-24 18:48:07 -07:00
yucheng-berri
fc5ab31fba
refactor(proxy/batches): make managed-file resolution purely additive, fall back like base (#34584)
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.

The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
2026-07-24 18:42:48 -07:00
Yuneng Jiang
ab44c8a8ce
fix(passthrough): let an upstream-reported cost outrank cost_per_request
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
2026-07-24 18:26:12 -07:00
Yuneng Jiang
b03004a1eb
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/litellm-logs-ui-lag-0ca4b8 2026-07-24 18:24:14 -07:00
Yuneng Jiang
48e77738ca
refactor(logs): make the end-user filter scan cap a fixed constant
The cap was an env-tunable knob in constants.py. Nothing needs to tune it:
it exists so DISTINCT cannot run over an unbounded row set, and picking a
value is a correctness decision, not deployment configuration. An env var
also makes the bound unverifiable, since the same code can behave very
differently between two proxies.

It is now a plain constant next to its only caller, mirroring how
SPEND_LOGS_PAGINATION_COUNT_CAP sits beside ui_view_spend_logs, and it takes
that constant's value: both reads of LiteLLM_SpendLogs now stop at the same
depth. constants.py goes back to matching staging exactly.

The existing test only asserted the parameter equalled the constant, which
is tautological; raising the constant to a billion kept it green while
removing the bound. A second test pins the value against the logs page's
cap, so an arbitrary change to either one fails.
2026-07-24 18:24:13 -07:00
ryan-crabbe-berri
579f41d57f
fix(proxy): attribute org spend for team-linked credentials minted without org_id (#34577)
* fix(proxy): attribute spend to org for team-linked keys minted without org_id

Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.

Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.

* test(proxy): cover CLI session-token org backfill from team

CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
2026-07-24 18:20:42 -07:00
Yuneng Jiang
f2315b070f
Merge branch 'litellm_internal_staging' into litellm_/fil-china-passthrough-contract-fef51c 2026-07-24 18:10:20 -07:00
Yuneng Jiang
838c7a7ea7
feat(passthrough): record upstream-reported cost and token usage
A pass-through target that fans a single HTTP request out to several models
internally cannot be priced from its response body, so LiteLLM had nothing to
record and every such request landed in the spend logs with zero cost and zero
tokens. The target now reports the totals for the whole request in
x-litellm-response-cost and x-litellm-total-tokens response headers, and
LiteLLM records those values as-is rather than recomputing them.

The headers are read on every upstream response, so a request that burned
tokens before failing still books its spend on the failure row instead of
being dropped for having a 4xx/5xx status. Only what the upstream actually
reported is written, so a target that sends a cost but no token count keeps
the token count LiteLLM derived on its own; a target that sends neither header
is untouched, which is the normal case for Anthropic, Vertex and friends.

Two supporting fixes fall out of this. The rate limiter only pulled token
counts off response shapes it models, so pass-through usage never charged the
TPM window and a team could exceed its shared token limit through pass-through
traffic alone; it now falls back to combined_usage_object. And the streaming
success path reset response_cost unconditionally before the assembled response
recomputed it, which discarded any cost a pass-through handler had already
established (the pass-through branch right below it has always intended to
preserve exactly that).
2026-07-24 18:10:15 -07:00
yuneng-jiang
78348fd1c7
Merge pull request #34571 from BerriAI/litellm_/migrate-simple-table-status-74b530
refactor(ui): migrate routing groups table onto the shared DataTable
2026-07-24 18:06:58 -07:00
Tin Chi Lo
3ca9d551dc fix(proxy): anchor the tool spend clamp floor to UTC midnight
With end_date omitted the floor was end-anchored to now including its
time-of-day, so an explicit start_date exactly 30 days back parsed as
midnight, compared below the floor, and was invisibly clamped to a
mid-day instant: up to a day of spend disappeared while the response
start_date still printed the full calendar date. Anchoring the floor to
today's UTC midnight makes every comparison in the window derivation
date-pure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:00:16 -07:00
Tin Chi Lo
5fc4a058c1 test(proxy): cover the cleanup run-loop backstop branch
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 17:43:15 -07:00
Yuneng Jiang
fc278bbf9f
refactor(logs): match ui_view_spend_logs' team-scope clause shape
The scope predicate expanded permitted team ids into an IN list with one
placeholder each, copied from /key/aliases. ui_view_spend_logs, which owns
the same page and the same scoping rules, builds ("user" = $X OR team_id =
ANY($Y::text[])) instead: a single array parameter whatever the team count,
and no placeholder arithmetic to keep in step with the rest of the query.

Same semantics, but the two clauses now read identically, so a future change
to how spend logs are scoped is harder to apply to one and miss in the other.
2026-07-24 17:39:29 -07:00
tin-berri
2d6b57407d
Merge pull request #34578 from BerriAI/litellm_headroom_tokens_saved
fix(guardrails): derive tokens_saved when Headroom compression service omits it
2026-07-24 17:37:13 -07:00
Tin Chi Lo
30b7fd16f0 fix(proxy): label the tool spend clamp accurately (start capped at 30 days before end)
The clamp floor is end_date minus 30 days, serving up to 31 calendar
dates inclusive: deliberately the same width as the endpoint's default
window, so the dashboard's own default range never triggers the clamp
note. The docstring, card note, and test name now state that invariant
instead of the misleading 'most recent 30 days'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 17:31:00 -07:00
yucheng-berri
5677bc237c
fix(proxy/batches): resolve managed unified input_file_id to storage_url with ownership check before dispatch (#34474)
* fix: resolve unified_file_id to real storage_url before dispatching batch create

litellm.create_batch() against a Vertex AI-backed model crashes with an
opaque error when the input file was uploaded as a LiteLLM-managed
'unified file' (multi-model file upload). The base64-encoded
unified_file_id token is a LiteLLM-internal identifier, not a real
provider-side file reference, but the batches_endpoints create_batch
handler forwards it unchanged to llm_router.acreate_batch() /
litellm.acreate_batch() for the unified_file_id branch. Provider-specific
code that expects a real file location (e.g. Vertex AI's batch
transformation, which parses a 'publishers/' segment out of the GCS URI)
then fails on the opaque token.

Resolve the unified_file_id to its real backend location
(LiteLLM_ManagedFileTable.storage_url) before dispatch, mirroring the
same lookup already used by the files retrieve/download endpoints for
managed files. Falls back to the previous (unchanged) behavior if no
managed-file record exists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(proxy/batches): null-guard await on find_first for sync MagicMock test harnesses

* fix(proxy/batches): enforce ownership and correct lookup key when resolving managed input_file_id

The adopted resolution queried LiteLLM_ManagedFileTable with the decoded
litellm_proxy string, but the unified_file_id column stores the raw base64
file id (see schema.prisma and the enterprise managed-files hook), so the
lookup never matched in production and silently fell back to the opaque id.
Query with the raw id instead and lock the key with a regression test.

Move the resolution above the dispatch branches so the load-balanced router
path receives the resolved storage_url too, enforce managed-file ownership
with the same can_access_resource semantics the files retrieve and download
endpoints use (404 on denial), and downgrade database failures to a logged
fallback instead of aborting batch creation. Unresolved ids still dispatch
unchanged because the managed-files deployment hook can map them via
model_file_id_mapping

* fix(proxy/batches): fail closed when the managed file ownership lookup errors

A lookup exception previously fell back to dispatching the original
unified id with the ownership gate unexecuted; the managed-files
deployment hook maps unified ids from cache without re-checking
ownership, so a database outage let a caller dispatch another tenant's
file. Raise a clear 503 instead and lock the behavior with a regression
test. No-database and no-row cases still fall back unchanged

* test(proxy/batches): default harness prisma_client to None

The batch routing harness left proxy_server.prisma_client at its module
global, which a sibling test in the same shard can leave as a MagicMock.
The unified-file rows that do not opt into managed-file resolution then
entered the resolver and awaited a non-awaitable mock, surfacing as a
503. Patch prisma_client to None by default so those rows stay a no-op;
resolution tests still override it explicitly

* fix(proxy/batches): keep unified resolution in its own branch and fail closed on missing row

Cursor flagged that hoisting the storage_url substitution above the
load-balanced dispatch branch broke two things on that path: the
model_file_id_mapping deployment filter keys on the original unified id,
and the response returned the internal storage_url instead of the
unified id. Move the resolution back inside the unified branch and
exclude unified ids from the load-balanced branch so a managed file
always takes the resolving path (which restores input_file_id and the
unified_file_id hidden param on the response), and a load-balanced batch
keeps the original id for deployment filtering.

Also fail closed with a 404 when a unified id has no managed-file row
while a database is present: the id cannot be ownership-verified, and
dispatching it would both bypass the gate and hit the Vertex
publishers-segment IndexError. Owned rows without a storage_url (legacy)
still dispatch the original id

* fix(proxy/batches): do not divert unified files off the load-balanced branch

Excluding unified ids from the load-balanced branch (and not
unified_file_id) regressed a path that works on the base revision: a
multi-model managed file dispatched with an explicit router model under
load balancing was routed into the unified branch, which raises a 400
for anything other than exactly one target model. Verified live against
base (200, managed-files deployment hook remaps the unified id per
model) versus the guarded branch (400 Expected 1 model, got 2).

Restore the original three-condition load-balanced branch so that path
keeps working unchanged. Unified-file storage_url resolution and the
ownership 404 still apply on the non-load-balanced unified branch, which
is the common managed-batch flow; the load-balanced managed path retains
its existing behavior and its pre-existing enterprise-hook ownership gap,
unchanged from base

* refactor(proxy/batches): scope managed-file handling to resolution, drop ownership check

Narrow this PR to its one problem: resolving a managed unified input_file_id
to its backend storage_url so provider batch handlers (Vertex parses a
publishers/ segment) receive a real location instead of the opaque token,
and failing closed with a 404 when the token has no backing row so it is
never dispatched into the provider crash.

Remove the cross-tenant ownership check (can_access_resource) added earlier.
Batch-create had no ownership enforcement before this PR, and the gap spans
every managed-file call type, so it belongs in the enterprise managed-files
pre-call hook (its acreate_batch branch) where files, batches and
fine-tuning are covered uniformly, not partially in this one endpoint. Filed
as a follow-up. This also removes the load-balanced-path ownership
inconsistency the bots flagged, since there is no ownership branch to skip.

Drop the inline comments flagged against the no-comments rule; behavior is
documented in the helper docstring and the test docstrings

* fix(proxy/batches): fail closed with 503 when the managed-file lookup errors

A lookup exception previously fell back to dispatching the unresolved
unified token, which defeats the fail-closed guarantee: the token still
reaches the provider and can hit the same publishers-segment IndexError
the resolution prevents. Treat a lookup error like the missing-row case
and fail closed, but with a retryable 503 since the condition is
transient. No-database and no-storage_url rows still fall back unchanged

---------

Co-authored-by: htourinho-clgx <htourinho@cotality.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-24 17:29:38 -07:00
Yuneng Jiang
2a50b3a087
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/litellm-logs-ui-lag-0ca4b8 2026-07-24 17:28:22 -07:00
Yuneng Jiang
44f571a8aa
fix(logs): keep the End User filter window in step with the logs table
Two issues Greptile raised on the filter window and the capped scan.

A preset date range ends at "now", which the logs query re-reads on every
fetch, so live tail keeps moving the table's end bound. The filter window
was memoized on the date controls alone, so it pinned whichever "now" it
was first built with: an end user that started sending traffic afterwards
showed up in the table but stayed missing from the dropdown until
something remounted it.

formatLogsWindow now takes the preset end bound as an argument, and
getLogsWindowEndBound derives it from the logs query's last fetch, rounded
up to the next minute. Rounding up rather than down means the filter window
never trails the table; bucketing means the query key holds steady between
ticks instead of churning once per render. The panel reads it from
logsQuery.dataUpdatedAt so it advances exactly when the table refreshes,
falling back to the stored end time before the first fetch. Deriving it
from Date.now() during render is what the purity rule forbids.

The capped inner scan ordered by startTime alone, so rows sharing a
timestamp could be cut differently between two requests and successive
OFFSET pages would disagree about the set they were paging through.
request_id now breaks the tie, which the (startTime, request_id) index
already covers.

Drift from rows genuinely arriving inside the window between page fetches
is left alone. Removing it means keyset pagination over the distinct set,
which cannot keep the inner row cap, and that cap is what stops this
query from degrading into a full scan of LiteLLM_SpendLogs.
2026-07-24 17:28:17 -07:00
Tin Chi Lo
26f6ff24d8 fix(proxy): cap tool spend window at 30 days and bound every SpendLogs read
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to
LiteLLM_SpendLogs with a start_time-only predicate the composite
(tool_name, start_time) index cannot serve, and the dedup total query
left the outer SpendLogs scan unwindowed, so every dashboard load
walked both per-request tables end to end.

- clamp the window to the most recent 30 days ending at end_date; the
  response start_date reflects the effective window and the dashboard
  notes the clamp
- index SpendLogToolIndex on start_time (all schema copies + migration)
- window the SpendLogs side of both queries (1s margin: the two writers
  can disagree by ~1ms on the same request)
- expire SpendLogToolIndex rows on the spend-log retention cutoff via a
  parametrized batch-delete engine shared with the SpendLogs cleanup

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 17:24:24 -07:00
mateo
f68abdc861 chore: merge litellm_internal_staging into litellm_fix_responses_bridge_streaming_contract
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-25 00:20:31 +00:00
yucheng-berri
76b0b10908
fix(guardrails): add /v1/messages support for Straiker plugin (#34548)
* fix(guardrails): add /v1/messages support for Straiker plugin

- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs

* fix(guardrails): gate cross-surface message resolution and cover streaming request data

Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.

Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.

Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.

Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.

---------

Co-authored-by: cs-mehta <chandra@straiker.ai>
2026-07-24 17:13:11 -07:00
Mateo Wang
cfb7edb54e
Merge pull request #34549 from BerriAI/litellm_fix_stream_options_responses_api
fix(responses): strip include_usage from stream_options instead of dropping the param
2026-07-24 17:12:26 -07:00
Tin
3489e650da fix(mcp): use a toolset row's stored tool name as written
A toolset row is {server_id, tool_name}. The server is already identified by
server_id, so the stored name is the tool's own name and there is nothing for a
prefix to disambiguate. Resolution nevertheless reduced the stored name by the
server's wire prefix, which is a guess about a string that carries no such
marker.

The wire prefix is added on the way out and is not part of any tool's identity,
so when a native tool name happens to begin with it the guess renamed the tool:
a row for greyhound_internal_events on a server prefixed greyhound resolved to
internal_events. That is a different tool on the same server, so the selected
tool disappeared from /toolset/<name>/mcp and an unselected sibling was served,
and executed, under the selected tool's wire name. Toolsets are the tool-level
permission boundary, so the row granted access to something never selected.

Match the stored name as written and keep stripping the prefix off the live name
only. This is the only producer that rewrote allowlist values; every other one
stores what the admin typed, so the tools/list filter, the tools/call permission
check, the REST listing and the Responses API path are all corrected without
touching them.

A row that stores an already-prefixed name no longer resolves. Such a row names
a tool that does not exist on the server, and the dashboard has never written
one; it was only ever accepted because of the guess this removes.
2026-07-24 17:06:08 -07:00
tin-berri
842f32dbaa
Merge pull request #34458 from BerriAI/litellm_lit4759_guardrail_metadata_bucket
fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata
2026-07-24 17:02:48 -07:00
Shivam Rawat
a9e7decd09
Merge pull request #34547 from BerriAI/litellm_fix_chat_completions_missing_messages_400
fix(proxy): return 400 instead of 500 for chat completions without messages
2026-07-24 16:56:13 -07:00
tin-berri
d389f837ae
Merge pull request #34411 from BerriAI/litellm_lit4650_passthrough_guardrail_log
fix(guardrails): stop reporting a no-op guardrail as applied on passthrough
2026-07-24 16:52:46 -07:00
mateo-berri
6ff88ba5e9 fix(responses): strip include_usage from stream_options instead of dropping the param 2026-07-24 16:46:49 -07:00
Tin Chi Lo
9bd89290cb fix(guardrails): derive tokens_saved when Headroom compression service omits it
The savings readers (extract_compression_saved_tokens, feeding
compression_saved_tokens on the daily spend tables) key exclusively on
tokens_saved in the guardrail_response stats, but the Headroom guardrail
builds those stats as a filtered pass-through of the compression service
response and the live service omits tokens_saved. Every compressed request
recorded 0 saved tokens on the Cost Optimization dashboard.

Derive tokens_saved = tokens_before - tokens_after when the key is absent
and both operands are numeric; a service-sent value still wins. The two
sibling writers (compresr, native compression interception) already derive
it the same way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 16:44:00 -07:00
Tin Chi Lo
9777e9524a fix(guardrails): stop reporting a no-op guardrail as applied on passthrough
On passthrough requests the shared guardrail plumbing still dispatches
headroom's pre_call apply_guardrail, but the passthrough translation hands it
only `texts` and no `structured_messages`, so it early-returns a no-op. The
@log_guardrail_information decorator then synthesized an "allow"/"success"
StandardLoggingGuardrailInformation entry, and the unified hook added the
guardrail to applied_guardrails, so spend logs reported the compression
guardrail as succeeded even though nothing ran.

Add a records_own_guardrail_information flag for guardrails that log their own
execution (headroom). The decorator skips the synthetic success entry for them,
and the unified hook lists such a guardrail in applied_guardrails only when it
actually recorded a run. A guardrail that owns its logging must record every
outcome it runs, so headroom now records a guardrail_failed_to_respond entry on
the fail_open path (compression attempted, service unreachable, request
forwarded uncompressed) instead of leaving it unlogged; fail_closed is still
recorded by the decorator's error path, and a genuine no-op stays not_run.
2026-07-24 16:29:29 -07:00
yuneng-jiang
3eaf7b1c0a
Merge pull request #34573 from BerriAI/litellm_/key-activity-missing-c5383e
Some checks are pending
CodSpeed Benchmarks / benchmarks (push) Waiting to run
UI Unit Tests / ui-unit-tests (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
fix(ui): keep entity usage tabs aligned with their panels
2026-07-24 16:25:29 -07:00
yuneng-jiang
7047a37f2f
Merge pull request #34475 from BerriAI/litellm_/test-coverage-mutation-analysis-e42223
test: remove tests that mutation analysis proved assert nothing
2026-07-24 16:23:34 -07:00
Yuneng Jiang
12bd6b5e5b
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/litellm-logs-ui-lag-0ca4b8 2026-07-24 16:21:50 -07:00
Yuneng Jiang
44b95bbfcb
fix(logs): scope the End User filter to the caller's teams and bound its scan
The End User filter listed every row of LiteLLM_EndUserTable, which is both
unscoped and the wrong source. Team admins and internal users can open the
Logs page, and their log view is already restricted to their own requests
plus the teams they administer, but the filter dropdown offered them every
end user on the proxy.

Team attribution only exists on spend logs, so /customer/aliases now reads
LiteLLM_SpendLogs and applies the same scoping /spend/logs/ui does: a proxy
admin sees the whole window, everyone else sees ("user" = caller OR team_id
IN permitted_teams), reusing _get_permitted_team_ids_for_spend_logs so the
two paths cannot drift. A caller with neither matches FALSE rather than
falling through to unscoped, and a failed team lookup degrades to
own-rows-only.

Querying spend logs safely is the other half. start_date/end_date are now
required, so the query always has the indexed startTime bound, and the
inner scan is capped at MAX_SPENDLOG_ROWS_TO_SCAN_FOR_FILTERS rows ordered
by startTime DESC. DISTINCT therefore runs over a bounded row set instead
of the whole table the way /global/all_end_users does.

Also adds /customer/aliases to spend_tracking_routes. Without it RouteChecks
rejects INTERNAL_USER and INTERNAL_USER_VIEW_ONLY before the handler runs,
which would have made the scoping above dead code; a test pins the route to
the same access tier as /spend/logs/ui.

The dropdown now shows the end users present in the window the table is
showing, so the filter list matches what it filters. formatLogsWindow is
shared with the logs query so the two windows cannot diverge.
2026-07-24 16:21:46 -07:00
Tin Chi Lo
770f41b5fa fix(guardrails): keep guardrail information in spend logs when the caller sends its own metadata
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.

Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.

Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.

Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
2026-07-24 16:20:44 -07:00
mubashir1osmani
86a02f4f52
test(e2e): cover MCP access-group tool selection at key creation (#34480)
Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.

Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.

Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
2026-07-24 16:18:40 -07:00
tin-berri
166c443b4f
Merge pull request #34454 from BerriAI/litellm_dashboard_object_permission_generated_type
refactor(ui): derive the dashboard object_permission type from the generated schema
2026-07-24 16:18:07 -07:00
Yuneng Jiang
b9e922cee7
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/key-activity-missing-c5383e 2026-07-24 16:12:14 -07:00
ryan-crabbe-berri
79c5c169d8
feat(ui): migrate the Create Organization form to shadcn and react-hook-form (#34552)
* feat(ui): migrate the Create Organization form to shadcn and react-hook-form

* fix(ui): guard double submit, test escape-close reset, drop dead organizationCreateCall

* fix(ui): block org create dialog dismissal while a create is in flight

* fix(ui): render budget duration labels instead of raw values in the shadcn Select
2026-07-24 16:09:30 -07:00
Yuneng Jiang
3966fbf5ec
chore(ui): drop the unused antd table header sort dropdown
TableHeaderSortDropdown had no importers left; the shared DataTable's
DataTableSortHeader covers the same ascending/descending/reset menu on Base UI.
knip did not flag it because its own test file counted as a usage.
2026-07-24 16:08:35 -07:00
ryan-crabbe-berri
5f2c9a952d
fix(ui): bind key duration input to one Form.Item so pre-filled expiry submits (#34521)
* fix(ui): bind key duration input to one Form.Item so pre-filled expiry submits

The Create and Edit key forms kept the displayed expiry in a Tremor TextInput's
local state while the value actually submitted lived in a separate hidden antd
Form.Item. After the first create, form.resetFields() cleared the hidden field
but not the local state, so a second create showed a stale "1d" that was never
sent unless the user deleted and retyped it

Wrap the visible input in a real Form.Item (name="duration") inside
KeyLifecycleSettings and drop both hidden mirror fields plus the local
durationValue state, so what is displayed is always what is submitted. The
Regenerate key flow already used this pattern

* test(ui): restore custom rotation interval coverage in real-form harness

The KeyLifecycleSettings test rewrite dropped the custom interval branch:
selecting Custom interval, typing a value, propagation to the parent, and
hiding the input when switching back to a predefined interval. Cover it in
the real antd Form harness, asserting the parent-held rotationInterval state
instead of a mocked callback
2026-07-24 16:08:32 -07:00
Yuneng Jiang
9e56630347
refactor(ui): migrate routing groups table onto the shared DataTable
Rebuilds the Router Settings > Routing Groups table on the shared DataTable
and cell library, the last antd entity grid in the dashboard.

The table splits into a thin RoutingGroupsTable container plus
RoutingGroupsTableColumns, with the usage snippets moving to their own
RoutingGroupUsagePanel on ui/tabs and the shared CodeBlock instead of antd
Tabs and Paragraph copyable. Models render through the shared ModelsCell so
long lists collapse behind "+N more" rather than wrapping the row, and the
two inline icon buttons become a single overflow menu with Edit and Delete.

antd gave the snippet panel its own chevron column; under the shared pattern
a row has two click targets, the name cell and the overflow menu, so clicking
the group name now opens the panel. Column set, order, actions, and the
backend row order are otherwise unchanged.
2026-07-24 16:08:29 -07:00