* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
The /openai/{endpoint} passthrough forwards a raw OpenAI-format request to
api.openai.com (or OPENAI_API_BASE) with the proxy's OPENAI_API_KEY swapped in,
and still logs a costed pass_through_endpoint SpendLogs row. Nothing exercised
that path end to end.
Adds a live /openai/v1/chat/completions passthrough test that asserts a 2xx
completion and a costed row with custom_llm_provider=openai, mirroring the
gemini and anthropic passthrough cost tests, and registers
llm.chat_completions.openai.passthrough.nonstream.cost_logged.
* fix(proxy): hash caller-supplied key in key update audit log object_id
* test: bound audit-log wait to the captured task instead of gathering the loop
/images/edits is a distinct native route from /images/generations: a multipart
request with the source image sent as the 'image' part plus an edit prompt, not
a JSON body. Nothing exercised it end to end.
Adds a live test that registers an OpenAI image model, sends a small generated
PNG plus an edit prompt to /v1/images/edits, and asserts an image comes back
(b64 or url). Generalizes the multipart transport helper with a file_field
argument (default 'file') so the image part can be named 'image', adds an
image_edit client method, the images_edits endpoint to the coverage schema, and
the llm.images_edits.openai.basic.nonstream.works cell.
Both OpenAI tool-call tests failed with "Function tools with reasoning_effort
are not supported for gpt-5.6 in /v1/chat/completions. To use function tools,
use /v1/responses or set reasoning_effort to 'none'."
This is a provider constraint, not a litellm defect. gpt-5.6 applies a default
reasoning effort, so the raw OpenAI API rejects tools even when the request
sets no reasoning_effort at all; only an explicit "none" is accepted. litellm
does not force that value when tools are present, and gpt-5.6 carries
supports_none_reasoning_effort=True in the model map, so passing it through is
the supported path and keeps these tests on /chat/completions.
Verified against the live stage proxy: the old body still reproduces the 400,
while adding reasoning_effort="none" returns tool_calls=1 non-streaming and
streams tool_calls deltas.
A failed MCP tool call aimed its error.* attributes at request_root_span(),
a ContextVar written on the ASGI request task. A stateful streamable-HTTP
session runs every message on the single task the session's initialize POST
spawned, so inside the message handler that ContextVar still holds the
initialize request's SERVER span. That span ended long ago, so the SDK
dropped every write (five 'Setting attribute on ended span' warnings plus
set_status and _add_event per failed call) and the POST that actually
failed carried no error at all. The identity attributes seeded onto the
server span went the same way.
Publish the live transport span on the ASGI scope of the request being
handled and read it back in the message handler through req_ctx.request,
the Request the streamable-HTTP transport attaches to each message. That
replaces the session-scoped field with a per-message one: a JSON-RPC
response POST deliberately skips the per-session lock, since it can arrive
while the tool call awaiting it is still in flight, so a field on the
shared auth object could be overwritten mid-call and send the tool call's
telemetry to the response's request. A scope also dies with its request
rather than holding a finished span on idle session state.
Publishing re-anchors the request root for the message so guardrail spans
and identity seeding follow, and only a transport still open for writes is
anchored or stamped: a notification POST can answer before the session task
is done, and moving dropped writes from one finished span to another is no
fix. Live capture goes from seven ended-span warnings and an unmarked
transaction to zero warnings and ERROR on the POST that carried the call.
The bedrock guardrail e2e test could never pass on stage. Two reasons.
It sent a bomb-making prompt expecting "stock hate/violence filters" to block,
but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no
contentPolicy at all; it denies the topic and words "bread"/"cake" plus
profanity. ApplyGuardrail returns action=NONE for the old prompt, so the
request passes and the test reports "default-on guardrail did not block".
Send a prompt the configured policy actually denies instead.
It also registered the guardrail with aws_access_key_id /
aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those
env vars are deliberately absent from the gateway (static AWS keys hijack RDS
IAM auth), and guardrail litellm_params do not expand os.environ/ indirection,
so the literal string reached boto and failed with "Invalid AWS region format:
'os.environ/AWS_REGION'". Drop all three and let the gateway sign
ApplyGuardrail with its own pod-identity role, which is how the standard stack
is meant to reach Bedrock.
Verified against the live stage proxy: registering the guardrail with only
identifier/version and sending the new prompt returns 400 "Violated guardrail
policy", satisfying both assertions.
* test(e2e): point four suites at models the providers still serve
Four llm_translation tests failed against upstream because the model they name
no longer exists. Each replacement was verified against the live stage proxy.
deepseek/deepseek-reasoner is gone; the DeepSeek API now lists only
deepseek-v4-flash and deepseek-v4-pro. Use deepseek/deepseek-v4-pro, which
still returns message.reasoning_content by default and still drops it for both
reasoning_effort="none" and thinking={"type": "disabled"} (litellm maps the
former to the latter, so the provider rejecting a bare "none" does not matter).
amazon.titan-image-generator-v2:0 returns "This model version has reached the
end of its life"; amazon.nova-canvas-v1:0 is the text-to-image model Bedrock
still offers in us-east-1.
Bedrock's Rerank API requires a full model ARN and rejects a bare model id with
"The provided model ARN for reranking is invalid", regardless of model or
region. Pass the ARN for cohere.rerank-v3-5:0, which is available in the
stack's us-east-1.
vertex_ai/gemini-embedding-2 404s as an unknown publisher model on this
project; vertex_ai/text-embedding-005 returns a vector.
* test(e2e): skip the hosted_vllm chat test when its server is unset
test_hosted_vllm_chat_returns_content read os.environ["HOSTED_VLLM_API_BASE"]
directly, so a stack without that env var failed the test with a bare KeyError
instead of reporting an environment gap. The batches suite already skips on the
same variable, and the vertex passthrough tests use pytest.skip for the same
reason, so follow that idiom here.
Drop the HOSTED_VLLM_API_KEY plumbing: the stage vLLM stand-in serves
/v1/chat/completions unauthenticated, and api_key is optional on
LiteLLMParamsBody, so passing it added nothing.
Default the backend to the model that server actually serves,
Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M, rather than a Llama id it never had.
Verified against the live stage proxy: a deployment with just that model and
api_base returns "hello".
* fix(model_map): mark deepseek v4-pro and v4-flash as reasoning-capable
Review on #34567 flagged that deepseek/deepseek-v4-pro is not marked
reasoning-capable while the e2e control case requires reasoning_content back
from it. The behavior premise is inverted, but it surfaced a real data gap: the
model map never gained supports_reasoning for the v4 models when DeepSeek
retired deepseek-reasoner, which did carry the flag.
Both models do reason. Against the live API with no reasoning params, v4-pro
returns 106 chars of reasoning_content and v4-flash returns 54, and both drop
it for thinking={"type":"disabled"}.
The stale flag had a real consequence beyond metadata: DeepSeekChatConfig
._thinking_mode_active() gates on supports_reasoning(), so with the flag unset
it returned False even when a caller passed thinking={"type": "enabled"},
skipping the multi-turn check that reasoning_content be passed back on
assistant messages. Param support itself was never gated, which is why
reasoning_effort="none" still mapped to thinking disabled.
Verified with LITELLM_LOCAL_MODEL_COST_MAP=True: supports_reasoning now
reports True for deepseek/deepseek-v4-pro and deepseek/deepseek-v4-flash.
tencent/deepseek-v4-pro is left alone; that route was not exercised here.
/vllm/batches and /vllm/files are high-volume passthrough routes with no e2e
coverage. They ride litellm's generic /vllm/{endpoint} forwarder, so the test
uploads a JSONL through /vllm/v1/files and creates a batch through
/vllm/v1/batches (BatchClient with provider=vllm), asserting the forwarded file
and batch objects come back. Lives next to TestHostedVllmBatch and is skip-marked
for the same reason: no live vLLM server (HOSTED_VLLM_API_BASE) in the e2e env.
Adds the two llm-translation registry cells.
Clears five OSV findings the scanner flags on every PR: four gitpython
advisories fixed in 3.1.54, and one postcss advisory fixed in 8.5.18.
gitpython 3.1.55 and brace-expansion 5.0.8 are left for a follow-up; both
were published less than three days ago and are still inside the
dependency cooldown window.
* fix(router): don't cool down parent deployment on advisor sub-call failure
Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group.
Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): tag advisor orchestration failures via provider-neutral util
Address review on LIT-4565: move the cooldown-exemption marker into
litellm/router_utils/cooldown_handlers.py so the router imports it at
module top instead of an in-function anthropic import, and extend the
exemption to AdvisorMaxIterationsError so a max-iterations orchestration
failure no longer cools down the healthy executor deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): merge model-level guardrails before pre_call_hook
DB/UI-assigned guardrails (litellm_params.guardrails) only fire on
post_call paths today: _check_and_merge_model_level_guardrails is called
in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but
never before pre_call_hook in common_request_processing.py:963. PR #23774
fixed the non-streaming post_call case; pre_call was left broken.
At the pre_call site, add_litellm_data_to_request strips client-supplied
metadata.model_info (pricing spoofing guard) and route_request hasn't run
yet, so model_info.id is unavailable. Extend the helper to fall back to
llm_router.get_deployment_by_model_group_name(model_alias) when model_id
is missing — that uses the O(1) model-name index already maintained by
the router.
Closes#29652
* fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload
Addresses veria-ai Medium finding + proxy-infra CI failure on this PR.
ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool
hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format
was omitting the field, so a key with mcp_rpm_limit could exceed it via the
MCP path.
Reads from kwargs.get("mcp_rate_limit_server_name") to match how
pre_call_tool_check resolves the alias-then-server-name fallback before
invoking hooks.
* fix(proxy): union guardrails across group deployments on alias fallback
Addresses second veria-ai Medium on #29654: the alias fallback called
get_deployment_by_model_group_name(), which returns ONE deployment.
A guardrail set on a non-first deployment would silently not run on
pre_call when the model_id is missing.
Switch to get_model_list(model_name=...) and take the UNION of
litellm_params.guardrails across all matching deployments (with dedup).
Trade-off documented in the comment: pre_call cannot know which
deployment route_request will select, so the conservative choice is to
apply any guardrail set on any eligible deployment.
Updated test stubs to use get_model_list. Added 3 new tests covering
union, dedup, and the all-empty case.
* test(model_level_guardrails): align integration test with get_model_list union API
* fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint
Addresses 3rd veria-ai Medium on #29654:
add_litellm_data_to_request preserves client-supplied metadata.model_info
when the caller's key/team has allow_client_pricing_override. The pre_call
merge previously trusted that id, so a caller could spoof an unguarded
model_info.id while requesting a guarded alias and bypass guardrails.
New `trust_client_model_info: bool` param on the helper. The pre_call
call site passes False; post_call paths (existing) keep True.
Also fixes the ruff failure on the union loop: pulled the .get() into a
local + isinstance(list) check before iterating, so mypy stops complaining
about `object` not being iterable.
2 new regression tests covering spoof-and-bypass + default-trust behavior.
* fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance
Addresses two more reviewer findings on #29654:
veria-ai Medium: route_request resolves team-scoped public model names
with metadata.user_api_key_team_id. The pre_call alias fallback called
get_model_list(model_name=...) without the team_id, so team-scoped
deployments were invisible and their pre_call guardrails silently
skipped. Now reads team_id from metadata or litellm_metadata and passes
it to get_model_list.
greptile P1: the isinstance(deployment_guardrails, list) guard added for
mypy narrowing silently dropped bare-string guardrail values that the
existing post_call path used to truthy-accept. Restored by wrapping a
scalar string into a one-element list on both paths.
4 new tests: team_id passthrough (metadata + litellm_metadata), scalar
on post_call, scalar on alias-union. 36/36 tests pass.
* style: black formatting on _check_and_merge_model_level_guardrails team_id assignment
* chore: ruff format
* fix(lint): remove unused noqa PLR0915 directive
RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic
because PLR0915 is not in this repo's enabled ruff rule set
(lint.extend-select in ruff.toml), so the directive suppresses nothing
and fails the lint job.
* refactor(proxy): hoist guardrail-merge import to module top
The pre_call guardrail-merge helper was imported inside
common_processing_pre_call_logic with a # noqa: PLC0415, which the
type-discipline gate counts as an unexplained suppression (LIT003). The
inline import's cyclic-import justification does not hold: this module
already imports from litellm.proxy.utils at top level, and utils.py does
not import common_request_processing at module load. Fold the helper
into the existing top-level import and drop the inline import, clearing
the suppression instead of budgeting for it.
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Per-tab path routing for Models made each tab a separate route parsed out
of the pathname, which is fragile under a static export mounted at a
runtime-variable server-root prefix. Revert the tabs to in-memory state:
a single /models-and-endpoints route renders an antd Tabs whose active
tab is React state, and each tab body moves from its own page.tsx into a
non-routed panel component under panels/. Role-gating (which tabs show),
the refresh control and the header are unchanged.
The ?model= / ?team= query drill-in stays: it is query-param based (read
via useSearchParams, written via history.pushState), so it is unaffected
by the server-root prefix and remains shareable. The shared tab-routing
helpers (createTabRoutes / useTabRouting) are untouched; the other four
pages still use them.
Removes the per-tab route dirs, layout.tsx and tabRoutes.ts (+ their
path-routing tests) and replaces the layout's coverage with a page test
for in-memory tab switching, the drill-in overlays and role-gating.
* feat(ui): deep-link virtual key detail view via ?key= query param
Clicking a key on the Virtual Keys page now sets ?key=<token> with
history.pushState, mirroring the models page's ?model= routing, so the
detail view survives reloads and can be shared as a URL. The key is
resolved from the loaded page when present and fetched via /key/info
otherwise. Extracts the shared navigateWithParams helper out of the
models detailNavigation hook
* test(ui): use a realistic hashed token in the virtual keys fixture
Two follow-ups from review on the upstream-reported usage contract.
An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.
The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.
The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
The cap was an env-tunable knob in constants.py. Nothing needs to tune it:
it exists so DISTINCT cannot run over an unbounded row set, and picking a
value is a correctness decision, not deployment configuration. An env var
also makes the bound unverifiable, since the same code can behave very
differently between two proxies.
It is now a plain constant next to its only caller, mirroring how
SPEND_LOGS_PAGINATION_COUNT_CAP sits beside ui_view_spend_logs, and it takes
that constant's value: both reads of LiteLLM_SpendLogs now stop at the same
depth. constants.py goes back to matching staging exactly.
The existing test only asserted the parameter equalled the constant, which
is tautological; raising the constant to a billion kept it green while
removing the bound. A second test pins the value against the logs page's
cap, so an arbitrary change to either one fails.
* fix(proxy): attribute spend to org for team-linked keys minted without org_id
Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.
Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.
* test(proxy): cover CLI session-token org backfill from team
CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
A pass-through target that fans a single HTTP request out to several models
internally cannot be priced from its response body, so LiteLLM had nothing to
record and every such request landed in the spend logs with zero cost and zero
tokens. The target now reports the totals for the whole request in
x-litellm-response-cost and x-litellm-total-tokens response headers, and
LiteLLM records those values as-is rather than recomputing them.
The headers are read on every upstream response, so a request that burned
tokens before failing still books its spend on the failure row instead of
being dropped for having a 4xx/5xx status. Only what the upstream actually
reported is written, so a target that sends a cost but no token count keeps
the token count LiteLLM derived on its own; a target that sends neither header
is untouched, which is the normal case for Anthropic, Vertex and friends.
Two supporting fixes fall out of this. The rate limiter only pulled token
counts off response shapes it models, so pass-through usage never charged the
TPM window and a team could exceed its shared token limit through pass-through
traffic alone; it now falls back to combined_usage_object. And the streaming
success path reset response_cost unconditionally before the assembled response
recomputed it, which discarded any cost a pass-through handler had already
established (the pass-through branch right below it has always intended to
preserve exactly that).
With end_date omitted the floor was end-anchored to now including its
time-of-day, so an explicit start_date exactly 30 days back parsed as
midnight, compared below the floor, and was invisibly clamped to a
mid-day instant: up to a day of spend disappeared while the response
start_date still printed the full calendar date. Anchoring the floor to
today's UTC midnight makes every comparison in the window derivation
date-pure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scope predicate expanded permitted team ids into an IN list with one
placeholder each, copied from /key/aliases. ui_view_spend_logs, which owns
the same page and the same scoping rules, builds ("user" = $X OR team_id =
ANY($Y::text[])) instead: a single array parameter whatever the team count,
and no placeholder arithmetic to keep in step with the rest of the query.
Same semantics, but the two clauses now read identically, so a future change
to how spend logs are scoped is harder to apply to one and miss in the other.
The clamp floor is end_date minus 30 days, serving up to 31 calendar
dates inclusive: deliberately the same width as the endpoint's default
window, so the dashboard's own default range never triggers the clamp
note. The docstring, card note, and test name now state that invariant
instead of the misleading 'most recent 30 days'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: resolve unified_file_id to real storage_url before dispatching batch create
litellm.create_batch() against a Vertex AI-backed model crashes with an
opaque error when the input file was uploaded as a LiteLLM-managed
'unified file' (multi-model file upload). The base64-encoded
unified_file_id token is a LiteLLM-internal identifier, not a real
provider-side file reference, but the batches_endpoints create_batch
handler forwards it unchanged to llm_router.acreate_batch() /
litellm.acreate_batch() for the unified_file_id branch. Provider-specific
code that expects a real file location (e.g. Vertex AI's batch
transformation, which parses a 'publishers/' segment out of the GCS URI)
then fails on the opaque token.
Resolve the unified_file_id to its real backend location
(LiteLLM_ManagedFileTable.storage_url) before dispatch, mirroring the
same lookup already used by the files retrieve/download endpoints for
managed files. Falls back to the previous (unchanged) behavior if no
managed-file record exists.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(proxy/batches): null-guard await on find_first for sync MagicMock test harnesses
* fix(proxy/batches): enforce ownership and correct lookup key when resolving managed input_file_id
The adopted resolution queried LiteLLM_ManagedFileTable with the decoded
litellm_proxy string, but the unified_file_id column stores the raw base64
file id (see schema.prisma and the enterprise managed-files hook), so the
lookup never matched in production and silently fell back to the opaque id.
Query with the raw id instead and lock the key with a regression test.
Move the resolution above the dispatch branches so the load-balanced router
path receives the resolved storage_url too, enforce managed-file ownership
with the same can_access_resource semantics the files retrieve and download
endpoints use (404 on denial), and downgrade database failures to a logged
fallback instead of aborting batch creation. Unresolved ids still dispatch
unchanged because the managed-files deployment hook can map them via
model_file_id_mapping
* fix(proxy/batches): fail closed when the managed file ownership lookup errors
A lookup exception previously fell back to dispatching the original
unified id with the ownership gate unexecuted; the managed-files
deployment hook maps unified ids from cache without re-checking
ownership, so a database outage let a caller dispatch another tenant's
file. Raise a clear 503 instead and lock the behavior with a regression
test. No-database and no-row cases still fall back unchanged
* test(proxy/batches): default harness prisma_client to None
The batch routing harness left proxy_server.prisma_client at its module
global, which a sibling test in the same shard can leave as a MagicMock.
The unified-file rows that do not opt into managed-file resolution then
entered the resolver and awaited a non-awaitable mock, surfacing as a
503. Patch prisma_client to None by default so those rows stay a no-op;
resolution tests still override it explicitly
* fix(proxy/batches): keep unified resolution in its own branch and fail closed on missing row
Cursor flagged that hoisting the storage_url substitution above the
load-balanced dispatch branch broke two things on that path: the
model_file_id_mapping deployment filter keys on the original unified id,
and the response returned the internal storage_url instead of the
unified id. Move the resolution back inside the unified branch and
exclude unified ids from the load-balanced branch so a managed file
always takes the resolving path (which restores input_file_id and the
unified_file_id hidden param on the response), and a load-balanced batch
keeps the original id for deployment filtering.
Also fail closed with a 404 when a unified id has no managed-file row
while a database is present: the id cannot be ownership-verified, and
dispatching it would both bypass the gate and hit the Vertex
publishers-segment IndexError. Owned rows without a storage_url (legacy)
still dispatch the original id
* fix(proxy/batches): do not divert unified files off the load-balanced branch
Excluding unified ids from the load-balanced branch (and not
unified_file_id) regressed a path that works on the base revision: a
multi-model managed file dispatched with an explicit router model under
load balancing was routed into the unified branch, which raises a 400
for anything other than exactly one target model. Verified live against
base (200, managed-files deployment hook remaps the unified id per
model) versus the guarded branch (400 Expected 1 model, got 2).
Restore the original three-condition load-balanced branch so that path
keeps working unchanged. Unified-file storage_url resolution and the
ownership 404 still apply on the non-load-balanced unified branch, which
is the common managed-batch flow; the load-balanced managed path retains
its existing behavior and its pre-existing enterprise-hook ownership gap,
unchanged from base
* refactor(proxy/batches): scope managed-file handling to resolution, drop ownership check
Narrow this PR to its one problem: resolving a managed unified input_file_id
to its backend storage_url so provider batch handlers (Vertex parses a
publishers/ segment) receive a real location instead of the opaque token,
and failing closed with a 404 when the token has no backing row so it is
never dispatched into the provider crash.
Remove the cross-tenant ownership check (can_access_resource) added earlier.
Batch-create had no ownership enforcement before this PR, and the gap spans
every managed-file call type, so it belongs in the enterprise managed-files
pre-call hook (its acreate_batch branch) where files, batches and
fine-tuning are covered uniformly, not partially in this one endpoint. Filed
as a follow-up. This also removes the load-balanced-path ownership
inconsistency the bots flagged, since there is no ownership branch to skip.
Drop the inline comments flagged against the no-comments rule; behavior is
documented in the helper docstring and the test docstrings
* fix(proxy/batches): fail closed with 503 when the managed-file lookup errors
A lookup exception previously fell back to dispatching the unresolved
unified token, which defeats the fail-closed guarantee: the token still
reaches the provider and can hit the same publishers-segment IndexError
the resolution prevents. Treat a lookup error like the missing-row case
and fail closed, but with a retryable 503 since the condition is
transient. No-database and no-storage_url rows still fall back unchanged
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two issues Greptile raised on the filter window and the capped scan.
A preset date range ends at "now", which the logs query re-reads on every
fetch, so live tail keeps moving the table's end bound. The filter window
was memoized on the date controls alone, so it pinned whichever "now" it
was first built with: an end user that started sending traffic afterwards
showed up in the table but stayed missing from the dropdown until
something remounted it.
formatLogsWindow now takes the preset end bound as an argument, and
getLogsWindowEndBound derives it from the logs query's last fetch, rounded
up to the next minute. Rounding up rather than down means the filter window
never trails the table; bucketing means the query key holds steady between
ticks instead of churning once per render. The panel reads it from
logsQuery.dataUpdatedAt so it advances exactly when the table refreshes,
falling back to the stored end time before the first fetch. Deriving it
from Date.now() during render is what the purity rule forbids.
The capped inner scan ordered by startTime alone, so rows sharing a
timestamp could be cut differently between two requests and successive
OFFSET pages would disagree about the set they were paging through.
request_id now breaks the tie, which the (startTime, request_id) index
already covers.
Drift from rows genuinely arriving inside the window between page fetches
is left alone. Removing it means keyset pagination over the distinct set,
which cannot keep the inner row cap, and that cap is what stops this
query from degrading into a full scan of LiteLLM_SpendLogs.
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to
LiteLLM_SpendLogs with a start_time-only predicate the composite
(tool_name, start_time) index cannot serve, and the dedup total query
left the outer SpendLogs scan unwindowed, so every dashboard load
walked both per-request tables end to end.
- clamp the window to the most recent 30 days ending at end_date; the
response start_date reflects the effective window and the dashboard
notes the clamp
- index SpendLogToolIndex on start_time (all schema copies + migration)
- window the SpendLogs side of both queries (1s margin: the two writers
can disagree by ~1ms on the same request)
- expire SpendLogToolIndex rows on the spend-log retention cutoff via a
parametrized batch-delete engine shared with the SpendLogs cleanup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
The savings readers (extract_compression_saved_tokens, feeding
compression_saved_tokens on the daily spend tables) key exclusively on
tokens_saved in the guardrail_response stats, but the Headroom guardrail
builds those stats as a filtered pass-through of the compression service
response and the live service omits tokens_saved. Every compressed request
recorded 0 saved tokens on the Cost Optimization dashboard.
Derive tokens_saved = tokens_before - tokens_after when the key is absent
and both operands are numeric; a service-sent value still wins. The two
sibling writers (compresr, native compression interception) already derive
it the same way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On passthrough requests the shared guardrail plumbing still dispatches
headroom's pre_call apply_guardrail, but the passthrough translation hands it
only `texts` and no `structured_messages`, so it early-returns a no-op. The
@log_guardrail_information decorator then synthesized an "allow"/"success"
StandardLoggingGuardrailInformation entry, and the unified hook added the
guardrail to applied_guardrails, so spend logs reported the compression
guardrail as succeeded even though nothing ran.
Add a records_own_guardrail_information flag for guardrails that log their own
execution (headroom). The decorator skips the synthetic success entry for them,
and the unified hook lists such a guardrail in applied_guardrails only when it
actually recorded a run. A guardrail that owns its logging must record every
outcome it runs, so headroom now records a guardrail_failed_to_respond entry on
the fail_open path (compression attempted, service unreachable, request
forwarded uncompressed) instead of leaving it unlogged; fail_closed is still
recorded by the decorator's error path, and a genuine no-op stays not_run.
The End User filter listed every row of LiteLLM_EndUserTable, which is both
unscoped and the wrong source. Team admins and internal users can open the
Logs page, and their log view is already restricted to their own requests
plus the teams they administer, but the filter dropdown offered them every
end user on the proxy.
Team attribution only exists on spend logs, so /customer/aliases now reads
LiteLLM_SpendLogs and applies the same scoping /spend/logs/ui does: a proxy
admin sees the whole window, everyone else sees ("user" = caller OR team_id
IN permitted_teams), reusing _get_permitted_team_ids_for_spend_logs so the
two paths cannot drift. A caller with neither matches FALSE rather than
falling through to unscoped, and a failed team lookup degrades to
own-rows-only.
Querying spend logs safely is the other half. start_date/end_date are now
required, so the query always has the indexed startTime bound, and the
inner scan is capped at MAX_SPENDLOG_ROWS_TO_SCAN_FOR_FILTERS rows ordered
by startTime DESC. DISTINCT therefore runs over a bounded row set instead
of the whole table the way /global/all_end_users does.
Also adds /customer/aliases to spend_tracking_routes. Without it RouteChecks
rejects INTERNAL_USER and INTERNAL_USER_VIEW_ONLY before the handler runs,
which would have made the scoping above dead code; a test pins the route to
the same access tier as /spend/logs/ui.
The dropdown now shows the end users present in the window the table is
showing, so the filter list matches what it filters. formatLogsWindow is
shared with the logs query so the two windows cannot diverge.
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.
Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.
Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.
Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
