get_team_callbacks only read metadata.callback_settings, while add_team_callbacks writes to metadata.logging, so any callback registered through the API came back as an empty list even though the UI and request-time resolution both showed it. Resolve both slots, redact credential-bearing callback_vars, and have disable_team_logging clear the logging slot it left behind.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
delete_deployment resolved the outgoing deployment through get_deployment before
popping it, and ran the strategy release before repairing the index maps. Both
halves of that ordering could leave the router inconsistent. A resolution failure
meant the entry left the model_list with its registry slots still held, so the
alias stayed routable and the name could not be reused; a failure inside the
release meant the outer handler returned None with the entry already popped and
model_id_to_deployment_index_map never repaired, breaking every later lookup and
delete until a restart.
upsert_deployment already had this right: it pops, repairs the caches and indices,
and only then releases the slot. delete_deployment now follows the same sequence
and resolves the deployment from the item it just popped rather than through a
lookup that can fail. Releasing the slot is secondary to structural integrity, so
it runs last and a failure there is logged instead of abandoning a removal that has
already happened.
* fix(ui): validate default team values in Default User Settings
The Default User Settings form accepted any free-text team id, and the
proxy persisted it without checking the team exists. New users were then
silently never added to the default team because the consume-time 404
from team_member_add was swallowed at debug level.
Backend: PATCH /update/internal_user_settings now rejects unknown and
duplicate team ids with a 400 naming them, before any persistence or
team budget side effects. Team-add failures in _add_user_to_team now log
at ERROR with user and team ids.
UI: DefaultUserSettings rewritten as a shadcn + react-hook-form + zod
form following the org-settings pattern. The team id free-text input is
replaced with a searchable server-backed team picker, so only existing
teams can be selected; zod blocks empty and duplicate rows. The shared
deriveErrorMessage helper now unwraps the HTTPException detail.error
shape so backend validation errors surface readably in toasts.
* fix(ui): restore read-only view with Edit Settings toggle on default user settings
Parity with the pre-migration form: the tab renders a read-only summary
of the saved defaults, Edit Settings opens the RHF form, Cancel discards
pending edits and returns to the summary, and a successful save returns
to the summary showing the new values. Model sentinel labels in the
summary are derived from ModelSelect's now-exported special values
instead of duplicating the strings.
* refactor(ui): rename MODEL_SELECT_SPECIAL_VALUES_ARRAY to MODEL_SENTINEL_OPTIONS
* fix(ui): move Edit Settings into the card header action slot
The finalize re-run in upsert_deployment keyed off the auto_router/adaptive_router
prefix only, so editing a complexity router with adaptive enabled released its
adaptive_routers entry (and post-call hook) without rebuilding it: complexity
routing kept serving while bandit recording, DB persistence and
/adaptive_router/state went silently dark until the next full reload. Gate the
re-run on a participation predicate that mirrors both arms of the finalize pass,
drop the import that pass no longer uses, and pin the registry helpers with
direct contract tests
Auto-router-family deployments live in two structures: the model_list, and a
pre-routing strategy registry keyed by (model_name, tags). Removing a deployment
dropped it from the model_list without releasing its registry slot, so the re-add
that follows hit the "already exists" guard in _register_pre_routing_strategy and
ignore_invalid_deployments swallowed it. The deployment came out and never went
back, while the DB row and the endpoint response both looked fine. Only a restart
healed it, and under multiple replicas each pod diverged into holding a different
subset of routers.
Removal now releases the (model_name, tags) slot from every strategy registry, in
both upsert_deployment and delete_deployment, guarded on the auto_router/ prefix so
removing a regular deployment cannot evict a router that merely shares its
model_name. Releasing from every registry rather than the first match is what makes
this correct for hybrids: registration is one-to-many, since a complexity router
configured with adaptive is also registered in adaptive_routers under the same key
by the deferred finalize pass. Releasing only the first match left that adaptive
strategy live, so a deleted or replaced alias stayed routable through it.
Adaptive post-call hooks are rebuilt whenever the adaptive registry changes, not
only at the end of set_model_list. The hook set is defined as exactly one hook per
registered adaptive router, so a released router stops recording turns instead of
holding a hook bound to a strategy nothing points at any more.
The swallowed upsert failure is logged at warning instead of debug, which is below
the default log level and left this failure with no observable signal anywhere.
delete_deployment resolves the outgoing deployment before popping it, and a
resolution failure no longer aborts the removal; previously an entry that failed
validation would have been left in the model_list permanently.
delete_model drops its blanket pop across all four registries. That predates this
change and over-evicts: it removes every tag variant registered under the name
while only one is being deleted, and nothing reloads on that path to restore the
survivors. delete_deployment now handles it correctly and tag-scoped, so the
endpoint-level eviction and its helper are removed rather than left to mask it.
Exercises the streaming field-fill heuristics, model_construct fallbacks,
and the get/cancel/delete/list request and response transforms that had no
tests.
litellm_provider_cache_creation_input_tokens_metric only read the
Anthropic-style top-level usage.cache_creation_input_tokens and had no
prompt_tokens_details fallback, unlike its cache-read twin. OpenAI models
that bill prompt cache writes report them only in
prompt_tokens_details.cache_write_tokens, so the counter never fired for
them. Resolve provider cache read/write tokens through a shared helper
that falls back to prompt_tokens_details.cache_write_tokens (canonical)
then cache_creation_tokens when the explicit top-level field is absent,
and give litellm_input_cache_creation_tokens_metric the same fallback for
raw usage dicts that only carry cache_write_tokens
Resolves LIT-4823. An adversarial review reproduced against real Postgres
that a batched increment upsert stalling past the prisma engine timeout
leaves its transaction open on the pooled connection; the retry draws the
same connection, its statements stack into the still-open transaction, and
one commit applies both increment sets while the writer reports success.
httpx.ReadTimeout is exactly that post-send case and every spend writer
retried it.
DB_RETRY_SAFE_ERROR_TYPES (ConnectError only, the failure that proves the
statements never reached the database) is now the single owner of what a
non-idempotent writer may retry. All seven entity and daily spend writer
retry arms and the tool usage flush consume it. DB_CONNECTION_ERROR_TYPES
is unchanged for the idempotent spend-log writer, whose create_many with
skip_duplicates may safely retry the full tuple.
The corruption was reproduced on update_daily_user_spend (seeded 10|100|1,
expected 11|110|2, observed 12|120|3); the new policy tests pin that a
ReadTimeout drops the batch loudly on the first attempt and a ConnectError
still retries.
Anthropic-format requests translate to messages whose content is a list
of part dicts, which the headroom compression service's transforms
silently skip (they only rewrite string content), so compression never
applied to Anthropic client traffic while the guardrail still reported
itself as applied.
Flatten all-text part lists to plain strings for /v1/compress and
restore the original shapes from the response: untouched rows keep
their exact original parts, a rewritten row collapses to one part
carrying the last declared cache_control breakpoint (a breakpoint
caches the prefix ending at its part, so the last one and its TTL
still describe the merged row). Rows with any non-text part are never
flattened, since merging text across a non-text part would move a
later breakpoint to the other side of it; they pass through the
service untouched, matching its own behavior for non-string content.
Flattening and write-back use the shared content_text helpers that
compresr's breakpoint fix also uses.
Resolves LIT-4795
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same cause as the otel handler test: this file builds its request as a
SimpleNamespace carrying only `state`, and the validation handler now reads
`request.url.path` to pick an error contract, so the fake needs a url
While here, cover what the two existing tests do not. They only exercise the
proxy-wide 422, and the control plane's 400 problem document was reachable only
through the route test, which registers its own copy of the handler in a local
app rather than the real one. Two cases now pin the real handler directly: a
`/management/v1` path returns problem+json with a `detail` string, and paths that
merely resemble the prefix (`/management`, `/v1/management/foo`) keep the 422
shape their callers parse
Both failures are from this branch, not pre-existing
The component allowlist test asserts the gateway and backend route sets union to
the whole app, so any route on neither is a 404 on both pods. Allowlist the
`/management/v1/` prefix on the backend, next to the other control plane
entries, so every resource that moves under it later is covered without a
per-resource edit
The otel handler test builds its request as a SimpleNamespace carrying only
`state`. The validation handler now reads `request.url.path` to decide whether
the caller is on a surface with its own error contract, so the fake needs a url;
a real Request always has one, which is why the handler does not guard for it
The control plane branch returns early, and nothing covered that it still closes
the dangling SERVER span first, so those requests would have leaked a span
apiece. Added a case that pins it; removing the close call fails it
Three fixes from an adversarial review of this branch, each at the owning
seam rather than the report site.
The flush retried DB_CONNECTION_ERROR_TYPES, which includes ReadTimeout.
A ReadTimeout is the committed-but-unacked case: the review reproduced the
engine abandoning the transaction open on the pooled connection, the retry
stacking its statements into it, and one commit applying both increment
sets while the flush reports success. The retry now covers only
ConnectError, the one failure that proves the statements never reached the
database; post-send failures drop the batch with an error log. The
docstring no longer claims an idempotency the pattern does not have. The
same hazard exists in the untouched daily spend writer and is left for its
own change.
get_tool_calls_from_response read choices[0] only, so a tool invoked in a
later choice of an n>1 response earned spend but never reached the rollup,
the index, or the registry. Choice scope is now an explicit parameter:
accounting passes include_all_choices=True because every choice costs
money; guardrails keep the primary-choice default because they rebuild the
primary assistant message. First multi-choice fixtures in the suite pin
both scopes.
maxBarSize=64 had been added to the shared BarChart unconditionally,
resizing every existing consumer. It is now a prop; only the tool spend
charts opt in. The legend flex-wrap changes stay global because clipping
overflow was a defect, not a preference.
The RFC 9457 `type` was `https://docs.litellm.ai/errors/<slug>`, copied from the
standard's own error example. That path is a 404 and there is no docs section
behind it, so every error body shipped a broken link
RFC 9457 only requires `type` to identify the problem type; it encourages, but
does not require, that dereferencing it yield documentation. An https URI makes a
promise we are not keeping, so use `urn:litellm:error:<slug>` instead, which
carries the same machine-readable identity with nothing to resolve. Switching to
an https base later is a contract change for anyone matching on `type`, so that
should wait for pages that actually exist
A test pins the identifier against regressing to an https docs URL, since the
existing assertion built the expected value from the same constant and would have
stayed green whatever it held
`/customer/aliases` shipped two days ago and has not been in a release, so its
wire contract is still free to change. This lands it on the control-plane
contract before that stops being true, since after a release the path, the param
names and the envelope would all need a permanent legacy adapter
The endpoint becomes `GET /management/v1/spend_logs/end_users`. It is a facet,
the distinct values one column takes over a filtered query on a resource, not an
entity collection; naming it after `customers` implied it listed the end-user
table when it actually reads spend logs, which is a different row set. Serving it
under the parent resource means its filters are the parent's filters, so the
dropdown offers exactly the values the logs table can show without two endpoints
having to keep agreeing on that
Contract changes: `size` becomes `page_size`, `search` becomes `q`, the window
moves from flat `start_date` / `end_date` to `filter[startTime][gte]` / `[lte]`,
and the body becomes `{data, meta, links}`. Unknown query params are now a 400
rather than being silently dropped, because an ignored filter over-returns data.
Errors are RFC 9457 problem documents on this prefix only; every other route
keeps the shape its callers already parse
`links` is what makes the rest deferrable. The dashboard hook follows the
server's `links.next` instead of computing `page + 1`, so moving this to cursor
pagination later changes the links and nothing the client does. That matters
because the inner scan is a sliding window, so offset paging can currently skip
or repeat an end user across pages; the fix is a follow-up, and the hypermedia
means it will not be a breaking one
Cursor mode, `sort`, `include`, ETag / `If-None-Match` and the generic `ListSpec`
framework are all deliberately out of scope here. They are additive or internal,
so none of them needs to beat the release
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
* fix(guardrails): resolve judge_model credentials via Router in llm_as_a_judge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): wire llm_router into DB-backed judge guardrail init paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert patch endpoint forwards llm_router to sync
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): resolve judge Router lazily and fix wildcard/alias dispatch
Resolve the proxy Router at judge-call time via an injected provider instead of
capturing it at construction, so a DB-backed judge guardrail created before the
Router exists no longer captures None permanently. Select the Router path with
router.get_model_list(model_name=judge_model) so wildcard routes and
model_group_alias keys resolve, not just literal deployment names. Isolate the
judge call from user-traffic routing with num_retries=0 and fallbacks=[].
Revert the llm_router threading through the DB sync/reinit/create/approve/patch
paths since the lazy provider makes it unnecessary. Replace mocked-Router tests
with real Router coverage for plain deployments, model_group_alias, and wildcard
routes, plus lazy per-call resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden judge verdict parsing and guard proxy import
Strip markdown fences and surrounding prose before json.loads so fencing-prone
judge models evaluate instead of failing open, guard the proxy_server import in
_default_router_provider so an unimportable proxy falls back to the SDK, and
snapshot/restore global callback lists in the DB-path judge registry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-object judge verdicts instead of failing open as success
* fix(guardrails): route hidden model_group_alias judge models through the Router
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Cache web identity STS credentials in the shared IAM cache (restores the
pre-v1.85.0 behavior removed by #27125) and cap the Google OIDC token cache
TTL at the token's own exp claim minus a 60s margin, never caching an
already-expired token
Anthropic cache_control breakpoints are positional: each one caches the
prefix ending at the part that carries it. Compresr flattened every text
part of a message into one string and wrote the compressed result back
into the first text part only, which dropped every later breakpoint and,
when a non-text part sat between text parts, moved the trailing text to
the other side of it.
The positional invariant now has one owner. guardrail_hooks/content_text.py
holds content_to_text alongside is_all_text_parts and
merge_rewritten_text_parts, so a compressed string is only ever written
back over a contiguous run of text parts, and the merged part carries the
last declared breakpoint and its TTL.
Compresr consumes that owner at both ends: _select_targets no longer
selects a row holding a non-text part, and _replace_text_in_content
returns such a row unchanged rather than merging across it. Rows whose
content is a plain string are unaffected.
Mixed rows therefore stop being compressed, which is a deliberate trade;
no single-string write-back can preserve a breakpoint across a non-text
part, so the alternative is silently caching a different prefix than the
caller configured.
* fix(auth): route JWT default-team into memberships instead of the create payload
JWT auto-provisioning (get_user_object with user_id_upsert) merged
litellm.default_internal_user_params verbatim into the Prisma user create,
including a teams key. When a default team is configured through the Admin
UI it is stored as a list of NewUserRequestTeam objects, but the user
table's teams column is String[], so the create raised a Prisma type error
and every JWT-authenticated request 401'd with the user never created.
Mirror the /user/new path: strip teams (and available_teams) out of the
create payload, then route the configured default team through
check_if_default_team_set / add_new_user_to_default_team so provisioned
users get real membership rows. Reuse the synthetic PROXY_ADMIN
UserAPIKeyAuth pattern already used by the team-upsert path to satisfy the
membership permission gate, and import the helpers lazily to avoid the
auth_checks <-> internal_user_endpoints import cycle.
* fix(auth): propagate max_budget_in_team when adding users to default teams
* fix: use pipe union instead of Optional for UP045 budget
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
* fix(proxy): hash caller-supplied key in key update audit log object_id
* test: bound audit-log wait to the captured task instead of gathering the loop
A failed MCP tool call aimed its error.* attributes at request_root_span(),
a ContextVar written on the ASGI request task. A stateful streamable-HTTP
session runs every message on the single task the session's initialize POST
spawned, so inside the message handler that ContextVar still holds the
initialize request's SERVER span. That span ended long ago, so the SDK
dropped every write (five 'Setting attribute on ended span' warnings plus
set_status and _add_event per failed call) and the POST that actually
failed carried no error at all. The identity attributes seeded onto the
server span went the same way.
Publish the live transport span on the ASGI scope of the request being
handled and read it back in the message handler through req_ctx.request,
the Request the streamable-HTTP transport attaches to each message. That
replaces the session-scoped field with a per-message one: a JSON-RPC
response POST deliberately skips the per-session lock, since it can arrive
while the tool call awaiting it is still in flight, so a field on the
shared auth object could be overwritten mid-call and send the tool call's
telemetry to the response's request. A scope also dies with its request
rather than holding a finished span on idle session state.
Publishing re-anchors the request root for the message so guardrail spans
and identity seeding follow, and only a transport still open for writes is
anchored or stamped: a notification POST can answer before the session task
is done, and moving dropped writes from one finished span to another is no
fix. Live capture goes from seven ended-span warnings and an unmarked
transaction to zero warnings and ERROR on the POST that carried the call.
* fix(router): don't cool down parent deployment on advisor sub-call failure
Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group.
Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): tag advisor orchestration failures via provider-neutral util
Address review on LIT-4565: move the cooldown-exemption marker into
litellm/router_utils/cooldown_handlers.py so the router imports it at
module top instead of an in-function anthropic import, and extend the
exemption to AdvisorMaxIterationsError so a max-iterations orchestration
failure no longer cools down the healthy executor deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): merge model-level guardrails before pre_call_hook
DB/UI-assigned guardrails (litellm_params.guardrails) only fire on
post_call paths today: _check_and_merge_model_level_guardrails is called
in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but
never before pre_call_hook in common_request_processing.py:963. PR #23774
fixed the non-streaming post_call case; pre_call was left broken.
At the pre_call site, add_litellm_data_to_request strips client-supplied
metadata.model_info (pricing spoofing guard) and route_request hasn't run
yet, so model_info.id is unavailable. Extend the helper to fall back to
llm_router.get_deployment_by_model_group_name(model_alias) when model_id
is missing — that uses the O(1) model-name index already maintained by
the router.
Closes#29652
* fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload
Addresses veria-ai Medium finding + proxy-infra CI failure on this PR.
ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool
hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format
was omitting the field, so a key with mcp_rpm_limit could exceed it via the
MCP path.
Reads from kwargs.get("mcp_rate_limit_server_name") to match how
pre_call_tool_check resolves the alias-then-server-name fallback before
invoking hooks.
* fix(proxy): union guardrails across group deployments on alias fallback
Addresses second veria-ai Medium on #29654: the alias fallback called
get_deployment_by_model_group_name(), which returns ONE deployment.
A guardrail set on a non-first deployment would silently not run on
pre_call when the model_id is missing.
Switch to get_model_list(model_name=...) and take the UNION of
litellm_params.guardrails across all matching deployments (with dedup).
Trade-off documented in the comment: pre_call cannot know which
deployment route_request will select, so the conservative choice is to
apply any guardrail set on any eligible deployment.
Updated test stubs to use get_model_list. Added 3 new tests covering
union, dedup, and the all-empty case.
* test(model_level_guardrails): align integration test with get_model_list union API
* fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint
Addresses 3rd veria-ai Medium on #29654:
add_litellm_data_to_request preserves client-supplied metadata.model_info
when the caller's key/team has allow_client_pricing_override. The pre_call
merge previously trusted that id, so a caller could spoof an unguarded
model_info.id while requesting a guarded alias and bypass guardrails.
New `trust_client_model_info: bool` param on the helper. The pre_call
call site passes False; post_call paths (existing) keep True.
Also fixes the ruff failure on the union loop: pulled the .get() into a
local + isinstance(list) check before iterating, so mypy stops complaining
about `object` not being iterable.
2 new regression tests covering spoof-and-bypass + default-trust behavior.
* fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance
Addresses two more reviewer findings on #29654:
veria-ai Medium: route_request resolves team-scoped public model names
with metadata.user_api_key_team_id. The pre_call alias fallback called
get_model_list(model_name=...) without the team_id, so team-scoped
deployments were invisible and their pre_call guardrails silently
skipped. Now reads team_id from metadata or litellm_metadata and passes
it to get_model_list.
greptile P1: the isinstance(deployment_guardrails, list) guard added for
mypy narrowing silently dropped bare-string guardrail values that the
existing post_call path used to truthy-accept. Restored by wrapping a
scalar string into a one-element list on both paths.
4 new tests: team_id passthrough (metadata + litellm_metadata), scalar
on post_call, scalar on alias-union. 36/36 tests pass.
* style: black formatting on _check_and_merge_model_level_guardrails team_id assignment
* chore: ruff format
* fix(lint): remove unused noqa PLR0915 directive
RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic
because PLR0915 is not in this repo's enabled ruff rule set
(lint.extend-select in ruff.toml), so the directive suppresses nothing
and fails the lint job.
* refactor(proxy): hoist guardrail-merge import to module top
The pre_call guardrail-merge helper was imported inside
common_processing_pre_call_logic with a # noqa: PLC0415, which the
type-discipline gate counts as an unexplained suppression (LIT003). The
inline import's cyclic-import justification does not hold: this module
already imports from litellm.proxy.utils at top level, and utils.py does
not import common_request_processing at module load. Fold the helper
into the existing top-level import and drop the inline import, clearing
the suppression instead of budgeting for it.
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
get_sanitized_user_information_from_key copied UserAPIKeyAuth.metadata
verbatim into user_api_key_auth_metadata, so the key's callback
configuration - including the integration credentials inside callback_vars -
reached the StandardLoggingPayload every integration receives. The two other
sites that stamp key/team metadata into request metadata did the same.
Sanitize at those sources with strip_callback_config, which drops the
`logging` and `callback_settings` slots and leaves everything else (notably
`priority`, read back by the dynamic rate limiter) untouched. Those slots are
resolved from UserAPIKeyAuth during pre-call setup and never read off the
logged copies, so nothing downstream loses input.
This makes the scrub in scrub_sensitive_keys_in_metadata dead - it only
matched the string "logging" under one of the two field names and never
covered callback_settings - so it is removed.
Separately, LangSmith set the run's `inputs` to the raw StandardLoggingPayload
while redacting only `extra`, so redact_user_api_key_info left every
user_api_key_* field in inputs.metadata. Both now go through one
_redact_metadata helper, which also covers the nested requester_metadata copy.
Two follow-ups from review on the upstream-reported usage contract.
An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.
The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.
The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
