Clicking a slice of the Cache Hits vs API Requests chart now lands on the
Logs page pre-filtered to exactly those requests: the failed slice maps to
status=failure, the cache-hit slice to cache_hit=true, and the api-requests
slice to status=success plus cache_hit=false, all scoped to the bar's
call_type and the chart's date range. To support that, /spend/logs/ui and
/spend/logs/v2 gain optional call_type and cache_hit query filters (additive,
so no breaking change to the public v2 surface), the Logs filter bar gets
matching Call Type and Cache Hit controls, and the Logs page seeds its
initial filters and time window from URL query params, which also makes
filtered logs views shareable by link. A seeded window disables live tail
so the drill-down target stays fixed on the clicked range
The /global/activity/cache_hits endpoint previously returned raw per
(key, call_type, model) spend-log aggregates typed as LiteLLM_SpendLogs
(wrong), and the dashboard reduced them in the browser: grouping by
call_type, relabeling empty call_type as Unknown, and computing the stat
card totals. All of that now happens server-side. The SQL groups per
call_type and splits cache hits vs successful vs failed requests, a new
cache_activity module validates rows into Pydantic models and computes
totals plus the key-alias/model filter options, and the endpoint declares
a real response_model so schema.d.ts types it correctly. The dashboard
consumes it through a typed $api react-query hook (filters ride the
query key and are applied in SQL instead of the browser), the hand-rolled
summarizeCacheActivity transform and the adminGlobalCacheActivity fetch
helper are deleted, and the refresh button now actually refetches.
The endpoint is UI-internal (hidden from the public swagger), so the
response reshape is not a public API break.
Spend logs for failed requests are stored with an empty call_type, so the
Cache Hits vs API Requests chart lumped them into an Unknown bar that read
as normal LLM API traffic. The activity query now also returns a per-group
failed_rows count (status = 'failure') and the dashboard charts it as a
third stacked series, so failures are visibly separate from successful
requests and cache hits. The chart data transform moves into a pure
summarizeCacheActivity helper with unit tests; header stats keep their
existing semantics (cache hit ratio still counts failures in the
denominator).
Same cause as the otel handler test: this file builds its request as a
SimpleNamespace carrying only `state`, and the validation handler now reads
`request.url.path` to pick an error contract, so the fake needs a url
While here, cover what the two existing tests do not. They only exercise the
proxy-wide 422, and the control plane's 400 problem document was reachable only
through the route test, which registers its own copy of the handler in a local
app rather than the real one. Two cases now pin the real handler directly: a
`/management/v1` path returns problem+json with a `detail` string, and paths that
merely resemble the prefix (`/management`, `/v1/management/foo`) keep the 422
shape their callers parse
The RFC 9457 `type` was `https://docs.litellm.ai/errors/<slug>`, copied from the
standard's own error example. That path is a 404 and there is no docs section
behind it, so every error body shipped a broken link
RFC 9457 only requires `type` to identify the problem type; it encourages, but
does not require, that dereferencing it yield documentation. An https URI makes a
promise we are not keeping, so use `urn:litellm:error:<slug>` instead, which
carries the same machine-readable identity with nothing to resolve. Switching to
an https base later is a contract change for anyone matching on `type`, so that
should wait for pages that actually exist
A test pins the identifier against regressing to an https docs URL, since the
existing assertion built the expected value from the same constant and would have
stayed green whatever it held
`/customer/aliases` shipped two days ago and has not been in a release, so its
wire contract is still free to change. This lands it on the control-plane
contract before that stops being true, since after a release the path, the param
names and the envelope would all need a permanent legacy adapter
The endpoint becomes `GET /management/v1/spend_logs/end_users`. It is a facet,
the distinct values one column takes over a filtered query on a resource, not an
entity collection; naming it after `customers` implied it listed the end-user
table when it actually reads spend logs, which is a different row set. Serving it
under the parent resource means its filters are the parent's filters, so the
dropdown offers exactly the values the logs table can show without two endpoints
having to keep agreeing on that
Contract changes: `size` becomes `page_size`, `search` becomes `q`, the window
moves from flat `start_date` / `end_date` to `filter[startTime][gte]` / `[lte]`,
and the body becomes `{data, meta, links}`. Unknown query params are now a 400
rather than being silently dropped, because an ignored filter over-returns data.
Errors are RFC 9457 problem documents on this prefix only; every other route
keeps the shape its callers already parse
`links` is what makes the rest deferrable. The dashboard hook follows the
server's `links.next` instead of computing `page + 1`, so moving this to cursor
pagination later changes the links and nothing the client does. That matters
because the inner scan is a sliding window, so offset paging can currently skip
or repeat an end user across pages; the fix is a follow-up, and the hypermedia
means it will not be a breaking one
Cursor mode, `sort`, `include`, ETag / `If-None-Match` and the generic `ListSpec`
framework are all deliberately out of scope here. They are additive or internal,
so none of them needs to beat the release
* fix(guardrails): resolve judge_model credentials via Router in llm_as_a_judge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): wire llm_router into DB-backed judge guardrail init paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert patch endpoint forwards llm_router to sync
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): resolve judge Router lazily and fix wildcard/alias dispatch
Resolve the proxy Router at judge-call time via an injected provider instead of
capturing it at construction, so a DB-backed judge guardrail created before the
Router exists no longer captures None permanently. Select the Router path with
router.get_model_list(model_name=judge_model) so wildcard routes and
model_group_alias keys resolve, not just literal deployment names. Isolate the
judge call from user-traffic routing with num_retries=0 and fallbacks=[].
Revert the llm_router threading through the DB sync/reinit/create/approve/patch
paths since the lazy provider makes it unnecessary. Replace mocked-Router tests
with real Router coverage for plain deployments, model_group_alias, and wildcard
routes, plus lazy per-call resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden judge verdict parsing and guard proxy import
Strip markdown fences and surrounding prose before json.loads so fencing-prone
judge models evaluate instead of failing open, guard the proxy_server import in
_default_router_provider so an unimportable proxy falls back to the SDK, and
snapshot/restore global callback lists in the DB-path judge registry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-object judge verdicts instead of failing open as success
* fix(guardrails): route hidden model_group_alias judge models through the Router
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Anthropic cache_control breakpoints are positional: each one caches the
prefix ending at the part that carries it. Compresr flattened every text
part of a message into one string and wrote the compressed result back
into the first text part only, which dropped every later breakpoint and,
when a non-text part sat between text parts, moved the trailing text to
the other side of it.
The positional invariant now has one owner. guardrail_hooks/content_text.py
holds content_to_text alongside is_all_text_parts and
merge_rewritten_text_parts, so a compressed string is only ever written
back over a contiguous run of text parts, and the merged part carries the
last declared breakpoint and its TTL.
Compresr consumes that owner at both ends: _select_targets no longer
selects a row holding a non-text part, and _replace_text_in_content
returns such a row unchanged rather than merging across it. Rows whose
content is a plain string are unaffected.
Mixed rows therefore stop being compressed, which is a deliberate trade;
no single-string write-back can preserve a breakpoint across a non-text
part, so the alternative is silently caching a different prefix than the
caller configured.
* fix(auth): route JWT default-team into memberships instead of the create payload
JWT auto-provisioning (get_user_object with user_id_upsert) merged
litellm.default_internal_user_params verbatim into the Prisma user create,
including a teams key. When a default team is configured through the Admin
UI it is stored as a list of NewUserRequestTeam objects, but the user
table's teams column is String[], so the create raised a Prisma type error
and every JWT-authenticated request 401'd with the user never created.
Mirror the /user/new path: strip teams (and available_teams) out of the
create payload, then route the configured default team through
check_if_default_team_set / add_new_user_to_default_team so provisioned
users get real membership rows. Reuse the synthetic PROXY_ADMIN
UserAPIKeyAuth pattern already used by the team-upsert path to satisfy the
membership permission gate, and import the helpers lazily to avoid the
auth_checks <-> internal_user_endpoints import cycle.
* fix(auth): propagate max_budget_in_team when adding users to default teams
* fix: use pipe union instead of Optional for UP045 budget
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
* fix(proxy): hash caller-supplied key in key update audit log object_id
* test: bound audit-log wait to the captured task instead of gathering the loop
* fix(proxy): merge model-level guardrails before pre_call_hook
DB/UI-assigned guardrails (litellm_params.guardrails) only fire on
post_call paths today: _check_and_merge_model_level_guardrails is called
in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but
never before pre_call_hook in common_request_processing.py:963. PR #23774
fixed the non-streaming post_call case; pre_call was left broken.
At the pre_call site, add_litellm_data_to_request strips client-supplied
metadata.model_info (pricing spoofing guard) and route_request hasn't run
yet, so model_info.id is unavailable. Extend the helper to fall back to
llm_router.get_deployment_by_model_group_name(model_alias) when model_id
is missing — that uses the O(1) model-name index already maintained by
the router.
Closes#29652
* fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload
Addresses veria-ai Medium finding + proxy-infra CI failure on this PR.
ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool
hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format
was omitting the field, so a key with mcp_rpm_limit could exceed it via the
MCP path.
Reads from kwargs.get("mcp_rate_limit_server_name") to match how
pre_call_tool_check resolves the alias-then-server-name fallback before
invoking hooks.
* fix(proxy): union guardrails across group deployments on alias fallback
Addresses second veria-ai Medium on #29654: the alias fallback called
get_deployment_by_model_group_name(), which returns ONE deployment.
A guardrail set on a non-first deployment would silently not run on
pre_call when the model_id is missing.
Switch to get_model_list(model_name=...) and take the UNION of
litellm_params.guardrails across all matching deployments (with dedup).
Trade-off documented in the comment: pre_call cannot know which
deployment route_request will select, so the conservative choice is to
apply any guardrail set on any eligible deployment.
Updated test stubs to use get_model_list. Added 3 new tests covering
union, dedup, and the all-empty case.
* test(model_level_guardrails): align integration test with get_model_list union API
* fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint
Addresses 3rd veria-ai Medium on #29654:
add_litellm_data_to_request preserves client-supplied metadata.model_info
when the caller's key/team has allow_client_pricing_override. The pre_call
merge previously trusted that id, so a caller could spoof an unguarded
model_info.id while requesting a guarded alias and bypass guardrails.
New `trust_client_model_info: bool` param on the helper. The pre_call
call site passes False; post_call paths (existing) keep True.
Also fixes the ruff failure on the union loop: pulled the .get() into a
local + isinstance(list) check before iterating, so mypy stops complaining
about `object` not being iterable.
2 new regression tests covering spoof-and-bypass + default-trust behavior.
* fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance
Addresses two more reviewer findings on #29654:
veria-ai Medium: route_request resolves team-scoped public model names
with metadata.user_api_key_team_id. The pre_call alias fallback called
get_model_list(model_name=...) without the team_id, so team-scoped
deployments were invisible and their pre_call guardrails silently
skipped. Now reads team_id from metadata or litellm_metadata and passes
it to get_model_list.
greptile P1: the isinstance(deployment_guardrails, list) guard added for
mypy narrowing silently dropped bare-string guardrail values that the
existing post_call path used to truthy-accept. Restored by wrapping a
scalar string into a one-element list on both paths.
4 new tests: team_id passthrough (metadata + litellm_metadata), scalar
on post_call, scalar on alias-union. 36/36 tests pass.
* style: black formatting on _check_and_merge_model_level_guardrails team_id assignment
* chore: ruff format
* fix(lint): remove unused noqa PLR0915 directive
RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic
because PLR0915 is not in this repo's enabled ruff rule set
(lint.extend-select in ruff.toml), so the directive suppresses nothing
and fails the lint job.
* refactor(proxy): hoist guardrail-merge import to module top
The pre_call guardrail-merge helper was imported inside
common_processing_pre_call_logic with a # noqa: PLC0415, which the
type-discipline gate counts as an unexplained suppression (LIT003). The
inline import's cyclic-import justification does not hold: this module
already imports from litellm.proxy.utils at top level, and utils.py does
not import common_request_processing at module load. Fold the helper
into the existing top-level import and drop the inline import, clearing
the suppression instead of budgeting for it.
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Two follow-ups from review on the upstream-reported usage contract.
An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.
The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.
The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
The cap was an env-tunable knob in constants.py. Nothing needs to tune it:
it exists so DISTINCT cannot run over an unbounded row set, and picking a
value is a correctness decision, not deployment configuration. An env var
also makes the bound unverifiable, since the same code can behave very
differently between two proxies.
It is now a plain constant next to its only caller, mirroring how
SPEND_LOGS_PAGINATION_COUNT_CAP sits beside ui_view_spend_logs, and it takes
that constant's value: both reads of LiteLLM_SpendLogs now stop at the same
depth. constants.py goes back to matching staging exactly.
The existing test only asserted the parameter equalled the constant, which
is tautological; raising the constant to a billion kept it green while
removing the bound. A second test pins the value against the logs page's
cap, so an arbitrary change to either one fails.
* fix(proxy): attribute spend to org for team-linked keys minted without org_id
Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.
Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.
* test(proxy): cover CLI session-token org backfill from team
CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
A pass-through target that fans a single HTTP request out to several models
internally cannot be priced from its response body, so LiteLLM had nothing to
record and every such request landed in the spend logs with zero cost and zero
tokens. The target now reports the totals for the whole request in
x-litellm-response-cost and x-litellm-total-tokens response headers, and
LiteLLM records those values as-is rather than recomputing them.
The headers are read on every upstream response, so a request that burned
tokens before failing still books its spend on the failure row instead of
being dropped for having a 4xx/5xx status. Only what the upstream actually
reported is written, so a target that sends a cost but no token count keeps
the token count LiteLLM derived on its own; a target that sends neither header
is untouched, which is the normal case for Anthropic, Vertex and friends.
Two supporting fixes fall out of this. The rate limiter only pulled token
counts off response shapes it models, so pass-through usage never charged the
TPM window and a team could exceed its shared token limit through pass-through
traffic alone; it now falls back to combined_usage_object. And the streaming
success path reset response_cost unconditionally before the assembled response
recomputed it, which discarded any cost a pass-through handler had already
established (the pass-through branch right below it has always intended to
preserve exactly that).
With end_date omitted the floor was end-anchored to now including its
time-of-day, so an explicit start_date exactly 30 days back parsed as
midnight, compared below the floor, and was invisibly clamped to a
mid-day instant: up to a day of spend disappeared while the response
start_date still printed the full calendar date. Anchoring the floor to
today's UTC midnight makes every comparison in the window derivation
date-pure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scope predicate expanded permitted team ids into an IN list with one
placeholder each, copied from /key/aliases. ui_view_spend_logs, which owns
the same page and the same scoping rules, builds ("user" = $X OR team_id =
ANY($Y::text[])) instead: a single array parameter whatever the team count,
and no placeholder arithmetic to keep in step with the rest of the query.
Same semantics, but the two clauses now read identically, so a future change
to how spend logs are scoped is harder to apply to one and miss in the other.
The clamp floor is end_date minus 30 days, serving up to 31 calendar
dates inclusive: deliberately the same width as the endpoint's default
window, so the dashboard's own default range never triggers the clamp
note. The docstring, card note, and test name now state that invariant
instead of the misleading 'most recent 30 days'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: resolve unified_file_id to real storage_url before dispatching batch create
litellm.create_batch() against a Vertex AI-backed model crashes with an
opaque error when the input file was uploaded as a LiteLLM-managed
'unified file' (multi-model file upload). The base64-encoded
unified_file_id token is a LiteLLM-internal identifier, not a real
provider-side file reference, but the batches_endpoints create_batch
handler forwards it unchanged to llm_router.acreate_batch() /
litellm.acreate_batch() for the unified_file_id branch. Provider-specific
code that expects a real file location (e.g. Vertex AI's batch
transformation, which parses a 'publishers/' segment out of the GCS URI)
then fails on the opaque token.
Resolve the unified_file_id to its real backend location
(LiteLLM_ManagedFileTable.storage_url) before dispatch, mirroring the
same lookup already used by the files retrieve/download endpoints for
managed files. Falls back to the previous (unchanged) behavior if no
managed-file record exists.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(proxy/batches): null-guard await on find_first for sync MagicMock test harnesses
* fix(proxy/batches): enforce ownership and correct lookup key when resolving managed input_file_id
The adopted resolution queried LiteLLM_ManagedFileTable with the decoded
litellm_proxy string, but the unified_file_id column stores the raw base64
file id (see schema.prisma and the enterprise managed-files hook), so the
lookup never matched in production and silently fell back to the opaque id.
Query with the raw id instead and lock the key with a regression test.
Move the resolution above the dispatch branches so the load-balanced router
path receives the resolved storage_url too, enforce managed-file ownership
with the same can_access_resource semantics the files retrieve and download
endpoints use (404 on denial), and downgrade database failures to a logged
fallback instead of aborting batch creation. Unresolved ids still dispatch
unchanged because the managed-files deployment hook can map them via
model_file_id_mapping
* fix(proxy/batches): fail closed when the managed file ownership lookup errors
A lookup exception previously fell back to dispatching the original
unified id with the ownership gate unexecuted; the managed-files
deployment hook maps unified ids from cache without re-checking
ownership, so a database outage let a caller dispatch another tenant's
file. Raise a clear 503 instead and lock the behavior with a regression
test. No-database and no-row cases still fall back unchanged
* test(proxy/batches): default harness prisma_client to None
The batch routing harness left proxy_server.prisma_client at its module
global, which a sibling test in the same shard can leave as a MagicMock.
The unified-file rows that do not opt into managed-file resolution then
entered the resolver and awaited a non-awaitable mock, surfacing as a
503. Patch prisma_client to None by default so those rows stay a no-op;
resolution tests still override it explicitly
* fix(proxy/batches): keep unified resolution in its own branch and fail closed on missing row
Cursor flagged that hoisting the storage_url substitution above the
load-balanced dispatch branch broke two things on that path: the
model_file_id_mapping deployment filter keys on the original unified id,
and the response returned the internal storage_url instead of the
unified id. Move the resolution back inside the unified branch and
exclude unified ids from the load-balanced branch so a managed file
always takes the resolving path (which restores input_file_id and the
unified_file_id hidden param on the response), and a load-balanced batch
keeps the original id for deployment filtering.
Also fail closed with a 404 when a unified id has no managed-file row
while a database is present: the id cannot be ownership-verified, and
dispatching it would both bypass the gate and hit the Vertex
publishers-segment IndexError. Owned rows without a storage_url (legacy)
still dispatch the original id
* fix(proxy/batches): do not divert unified files off the load-balanced branch
Excluding unified ids from the load-balanced branch (and not
unified_file_id) regressed a path that works on the base revision: a
multi-model managed file dispatched with an explicit router model under
load balancing was routed into the unified branch, which raises a 400
for anything other than exactly one target model. Verified live against
base (200, managed-files deployment hook remaps the unified id per
model) versus the guarded branch (400 Expected 1 model, got 2).
Restore the original three-condition load-balanced branch so that path
keeps working unchanged. Unified-file storage_url resolution and the
ownership 404 still apply on the non-load-balanced unified branch, which
is the common managed-batch flow; the load-balanced managed path retains
its existing behavior and its pre-existing enterprise-hook ownership gap,
unchanged from base
* refactor(proxy/batches): scope managed-file handling to resolution, drop ownership check
Narrow this PR to its one problem: resolving a managed unified input_file_id
to its backend storage_url so provider batch handlers (Vertex parses a
publishers/ segment) receive a real location instead of the opaque token,
and failing closed with a 404 when the token has no backing row so it is
never dispatched into the provider crash.
Remove the cross-tenant ownership check (can_access_resource) added earlier.
Batch-create had no ownership enforcement before this PR, and the gap spans
every managed-file call type, so it belongs in the enterprise managed-files
pre-call hook (its acreate_batch branch) where files, batches and
fine-tuning are covered uniformly, not partially in this one endpoint. Filed
as a follow-up. This also removes the load-balanced-path ownership
inconsistency the bots flagged, since there is no ownership branch to skip.
Drop the inline comments flagged against the no-comments rule; behavior is
documented in the helper docstring and the test docstrings
* fix(proxy/batches): fail closed with 503 when the managed-file lookup errors
A lookup exception previously fell back to dispatching the unresolved
unified token, which defeats the fail-closed guarantee: the token still
reaches the provider and can hit the same publishers-segment IndexError
the resolution prevents. Treat a lookup error like the missing-row case
and fail closed, but with a retryable 503 since the condition is
transient. No-database and no-storage_url rows still fall back unchanged
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two issues Greptile raised on the filter window and the capped scan.
A preset date range ends at "now", which the logs query re-reads on every
fetch, so live tail keeps moving the table's end bound. The filter window
was memoized on the date controls alone, so it pinned whichever "now" it
was first built with: an end user that started sending traffic afterwards
showed up in the table but stayed missing from the dropdown until
something remounted it.
formatLogsWindow now takes the preset end bound as an argument, and
getLogsWindowEndBound derives it from the logs query's last fetch, rounded
up to the next minute. Rounding up rather than down means the filter window
never trails the table; bucketing means the query key holds steady between
ticks instead of churning once per render. The panel reads it from
logsQuery.dataUpdatedAt so it advances exactly when the table refreshes,
falling back to the stored end time before the first fetch. Deriving it
from Date.now() during render is what the purity rule forbids.
The capped inner scan ordered by startTime alone, so rows sharing a
timestamp could be cut differently between two requests and successive
OFFSET pages would disagree about the set they were paging through.
request_id now breaks the tie, which the (startTime, request_id) index
already covers.
Drift from rows genuinely arriving inside the window between page fetches
is left alone. Removing it means keyset pagination over the distinct set,
which cannot keep the inner row cap, and that cap is what stops this
query from degrading into a full scan of LiteLLM_SpendLogs.
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to
LiteLLM_SpendLogs with a start_time-only predicate the composite
(tool_name, start_time) index cannot serve, and the dedup total query
left the outer SpendLogs scan unwindowed, so every dashboard load
walked both per-request tables end to end.
- clamp the window to the most recent 30 days ending at end_date; the
response start_date reflects the effective window and the dashboard
notes the clamp
- index SpendLogToolIndex on start_time (all schema copies + migration)
- window the SpendLogs side of both queries (1s margin: the two writers
can disagree by ~1ms on the same request)
- expire SpendLogToolIndex rows on the spend-log retention cutoff via a
parametrized batch-delete engine shared with the SpendLogs cleanup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
A toolset row is {server_id, tool_name}. The server is already identified by
server_id, so the stored name is the tool's own name and there is nothing for a
prefix to disambiguate. Resolution nevertheless reduced the stored name by the
server's wire prefix, which is a guess about a string that carries no such
marker.
The wire prefix is added on the way out and is not part of any tool's identity,
so when a native tool name happens to begin with it the guess renamed the tool:
a row for greyhound_internal_events on a server prefixed greyhound resolved to
internal_events. That is a different tool on the same server, so the selected
tool disappeared from /toolset/<name>/mcp and an unselected sibling was served,
and executed, under the selected tool's wire name. Toolsets are the tool-level
permission boundary, so the row granted access to something never selected.
Match the stored name as written and keep stripping the prefix off the live name
only. This is the only producer that rewrote allowlist values; every other one
stores what the admin typed, so the tools/list filter, the tools/call permission
check, the REST listing and the Responses API path are all corrected without
touching them.
A row that stores an already-prefixed name no longer resolves. Such a row names
a tool that does not exist on the server, and the dashboard has never written
one; it was only ever accepted because of the guess this removes.
The savings readers (extract_compression_saved_tokens, feeding
compression_saved_tokens on the daily spend tables) key exclusively on
tokens_saved in the guardrail_response stats, but the Headroom guardrail
builds those stats as a filtered pass-through of the compression service
response and the live service omits tokens_saved. Every compressed request
recorded 0 saved tokens on the Cost Optimization dashboard.
Derive tokens_saved = tokens_before - tokens_after when the key is absent
and both operands are numeric; a service-sent value still wins. The two
sibling writers (compresr, native compression interception) already derive
it the same way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On passthrough requests the shared guardrail plumbing still dispatches
headroom's pre_call apply_guardrail, but the passthrough translation hands it
only `texts` and no `structured_messages`, so it early-returns a no-op. The
@log_guardrail_information decorator then synthesized an "allow"/"success"
StandardLoggingGuardrailInformation entry, and the unified hook added the
guardrail to applied_guardrails, so spend logs reported the compression
guardrail as succeeded even though nothing ran.
Add a records_own_guardrail_information flag for guardrails that log their own
execution (headroom). The decorator skips the synthetic success entry for them,
and the unified hook lists such a guardrail in applied_guardrails only when it
actually recorded a run. A guardrail that owns its logging must record every
outcome it runs, so headroom now records a guardrail_failed_to_respond entry on
the fail_open path (compression attempted, service unreachable, request
forwarded uncompressed) instead of leaving it unlogged; fail_closed is still
recorded by the decorator's error path, and a genuine no-op stays not_run.
The End User filter listed every row of LiteLLM_EndUserTable, which is both
unscoped and the wrong source. Team admins and internal users can open the
Logs page, and their log view is already restricted to their own requests
plus the teams they administer, but the filter dropdown offered them every
end user on the proxy.
Team attribution only exists on spend logs, so /customer/aliases now reads
LiteLLM_SpendLogs and applies the same scoping /spend/logs/ui does: a proxy
admin sees the whole window, everyone else sees ("user" = caller OR team_id
IN permitted_teams), reusing _get_permitted_team_ids_for_spend_logs so the
two paths cannot drift. A caller with neither matches FALSE rather than
falling through to unscoped, and a failed team lookup degrades to
own-rows-only.
Querying spend logs safely is the other half. start_date/end_date are now
required, so the query always has the indexed startTime bound, and the
inner scan is capped at MAX_SPENDLOG_ROWS_TO_SCAN_FOR_FILTERS rows ordered
by startTime DESC. DISTINCT therefore runs over a bounded row set instead
of the whole table the way /global/all_end_users does.
Also adds /customer/aliases to spend_tracking_routes. Without it RouteChecks
rejects INTERNAL_USER and INTERNAL_USER_VIEW_ONLY before the handler runs,
which would have made the scoping above dead code; a test pins the route to
the same access tier as /spend/logs/ui.
The dropdown now shows the end users present in the window the table is
showing, so the filter list matches what it filters. formatLogsWindow is
shared with the logs query so the two windows cannot diverge.
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.
Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.
Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.
Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
Opening Logs > Filters fetched the entire customer table through
/customer/list, which is an unbounded find_many that eagerly loads the
budget and object-permission relations for every row. On a proxy with
61k customers that is a 20 MB, 7.6 s response; the dropdown then built an
option per row and rendered all of them, since the combobox does not
virtualize. The result was a multi-second freeze every time the drawer
opened.
Adds GET /customer/aliases, a projection of user_id alone with page/size/
search, mirroring /key/aliases. The End User field now uses
PaginatedSearchSelect behind an infinite query, the same shape the Key
Alias and Model filters already use, so it fetches 50 rows at a time and
pushes the typed query to the server.
The response reports has_more instead of a total count. A total needs
COUNT(*) over the whole match set on every keystroke, which is the cost
this endpoint exists to avoid; ordering by the user_id primary key and
fetching one row past the page lets Postgres stop early and still tells
the client whether to request more.
LIKE metacharacters in the search term are escaped, because end-user ids
routinely contain underscores and an unescaped one silently widens the
match.
Drops the now-unused accessToken prop threaded from RequestLogsPanel
through RequestLogsTable into the filters.
The gateway acts as an MCP client toward upstream MCP servers, and the MCP
authorization spec requires an MCP client to send the RFC 8707 resource
parameter on both the authorization request and every token request. The
gateway sent it on none of its upstream OAuth legs, so an authorization server
that requires resource indicators rejected the exchange with invalid_target
with no way to configure around it.
Authorization servers disagree irreconcilably and nothing advertises which
camp they are in, so this is a per-server opt-in rather than a default: most
providers ignore the parameter, some hard-reject it and carry audience in
scopes instead, and strict or MCP-native ones refuse to mint a correctly
scoped token without it. The new upstream_resource setting is unset by
default, which keeps today's requests byte-identical.
Both outbound OAuth stacks resolve the value from the server exactly once and
carry it structurally rather than attaching it per call site. In v1 every
plain-OAuth2 token leg builds its body through one helper that resolves the
resource in the same call as the mandatory client authentication; in v2 the
adapter, the single place an MCPServer becomes an outbound config, resolves it
onto the client_credentials config that the HTTP/SSE M2M path uses, and it
joins the config's mint identity so retargeting a live server refreshes the
token rather than serving the previous audience's. A leg cannot authenticate
without also naming the resource its sibling legs named, which is what an
attach-per-call-site approach kept getting wrong.
The setting is non-secret admin config sharing a blob with real secrets, and
the backend classifies which key is which rather than nulling the blob
wholesale or gating on its truthiness: redaction returns admin config to an
admin, session inheritance ignores it when deciding whether a real credential
was supplied and carries it onto the derived server, and the edit form renders
the same shared OAuth component as create so the field exists on both, an
emptied field submitting an explicit null that the credential merge drops.
Router.acompletion() takes messages positionally, so splatting a body that omits it raised a TypeError that the generic handler mapped to a 500. Validate the required body param at the routing boundary and raise the existing 400 contract instead.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
