* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
A deployment that carries one provider's credentials while resolving to
another is silently broken: litellm ignores the credentials and sends the
request to the resolved provider. The common shape is a Bedrock model
group where one entry lost its route prefix, so `model: claude-sonnet-5`
with aws_region_name set resolves to the first-party Anthropic API and
returns "x-api-key header is required". Because the router load balances
across the group, only the fraction of requests routed to that entry
fails, which reads as an intermittent provider outage rather than a
config error, and nothing at startup says otherwise.
Warn at deployment registration when provider-scoped credential params
(aws_*, vertex_*) sit on a model that resolves elsewhere, naming the
params, the resolved provider, and the likely missing prefix. Warn only:
an operator may be overriding a route deliberately, so this must not
block startup. Deployments litellm cannot classify are left alone.
Resolves LIT-5391
The iterations branch in AnthropicConfig.calculate_usage summed
cache_creation_input_tokens but never aggregated the per-iteration
cache_creation 5m/1h breakdown, leaving cache_creation_token_details
as None. As a result all cache-creation tokens fell back to the flat
5m write rate, underbilling 1h cache writes by up to 2x.
Aggregate the ephemeral_5m/ephemeral_1h split across iterations so 1h
writes are priced at the 1h rate.
Fixes LIT-4868
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The mateo-berri PAT now arrives via systemd LoadCredential as a file
instead of the EnvironmentFile, so pytest, the proxy, and the
model-driven claude CLI never inherit it and a same-UID /proc read
cannot lift it. run_daily.sh reads the credential when present, still
accepts an exported GITHUB_TOKEN for manual runs, and dies up front
when publishing is enabled with neither. Full CLI sandboxing is
tracked in LIT-5420
The existing web_search cells drive Claude Code's client-side WebSearch
tool, which the CLI executes itself and feeds back as a tool_result. The
CLI never emits a web_search_20250305 definition, so those cells stayed
green while the Anthropic-managed server tool 400'd on Bedrock.
Add a cell that posts the server tool to a Bedrock deployment over
/v1/messages and asserts a web_search_tool_result block comes back, and
reword the compat row so it no longer reads as coverage of the server
tool. Model the server tool as a composed base shared with tool_search.
Resolves LIT-5391
The web-search interception hooks stamp
_websearch_interception_emit_native_blocks and
_websearch_interception_converted_stream onto kwargs to carry state across
the agentic loop, but neither was registered in all_litellm_params. The
param builder sweeps anything it does not recognize into the outbound
request, so a provider that validates its body rejects the whole call:
Bedrock Converse answers "_websearch_interception_emit_native_blocks:
Extra inputs are not permitted" with a 400, which breaks every request
interception touches on that route.
Register both alongside their code-interpreter counterparts, which were
already listed for exactly this reason.
Resolves LIT-5391
The daily spend flush emitted one INSERT ... ON CONFLICT per aggregated key,
so every replica put hundreds of statements on the database each interval,
all contending for the same handful of hot rows and each holding its row
locks for the rest of the enclosing batch transaction. LiteLLM_DailyTagSpend
felt it worst because a request writes one row per tag, and litellm adds two
user-agent tags of its own by default.
A batch now goes out as a single multi-row statement. Rows are folded by the
conflict tuple first, and every nullable member of that tuple is normalized
to '': a NULL can never match itself in a unique index, so such a row was
re-inserted on every flush rather than aggregating, and a NULL model made
prisma reject the whole batch.
* fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill
A deployment that failed partway through a fallback chain (any attempt after
the first) was silently exempt from cooldown, because the has_logged_async_failure
dedup flag blocks the normal failure callback for every attempt past the first.
_trigger_cooldown_for_failed_deployment now explicitly evaluates cooldown for
that deployment when the dedup flag is set, using the same deployment-config >
response-header > router-default precedence as the primary failure path, and
skips advisor-orchestration failures. Deployment-ID resolution prefers the
exception's stamped failed_deployment_id, now also set from the generic-API-call
fallback path (rerank, embeddings, /v1/messages, etc.), falling back to metadata
inspection for call paths that don't stamp it yet.
CooldownCache also recomputes the remaining TTL when DualCache promotes a Redis
entry into the in-memory layer: before this, a cooldown entry restored from Redis
kept the in-memory layer's default 600s TTL regardless of the deployment's real
cooldown_time, so a deployment could stay excluded from routing for up to 10
minutes after a much shorter cooldown had already expired.
* fix(router): address Greptile review on the fallback-cooldown trigger
Two P1 findings on PR #35104:
- _trigger_cooldown_for_failed_deployment never incremented the deployment's
per-minute failure counter before evaluating cooldown, so a fallback
deployment's repeated retryable failures never accumulated toward the
default percent-fail-rate threshold that _should_cooldown_deployment checks.
- The metadata-bucket fallback (checking "metadata" before "litellm_metadata"
for a deployment_model_name marker) could be fooled by a caller with
permission to set metadata, since neither bucket's authorship can be
determined without knowing the call's function_name. Removed it entirely;
cooldown now requires the server-stamped failed_deployment_id, matching
what the primary chat-completions path and the generic-API-call path
(rerank, embeddings, /v1/messages, etc.) already set unconditionally.
* fix(router): freeze the litellm_params fallback mapping to satisfy the type-discipline gate
* fix(router): defer f-string interpolation in fallback-cooldown debug logs
* fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget
* fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks
* fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one
* fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout
* fix(router): stamp dynamic client-side-credential id in completion fallback paths too
The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential)
deployment id on exceptions, but the regular _completion/_acompletion exception handlers still
stamped the static shared deployment's id. A tenant using invalid forwarded credentials could
generate repeated failures attributed to, and eventually cooling down, the shared deployment
other tenants rely on. Extracted the stamping logic into one shared helper used by all three
call sites (generic API, sync completion, async completion) so the fix and future changes to it
stay in one place.
* test(router): add direct-reference unit tests for the new stamping helper
router_code_coverage.py's coverage gate flags _stamp_failed_deployment_id_with_effective_model_info
as untested because it only sees the function invoked indirectly through _completion/_acompletion's
exception handlers. Added two tests that call it directly, covering both the dynamic-id-present and
static-fallback branches.
* test(router): cover the timeout stamping branch and async active-cooldown append
_acompletion's litellm.Timeout handler and async_get_active_cooldowns' happy
path both lacked direct coverage despite their sibling branches (the generic
Exception handler, the sync get_active_cooldowns) being tested.
* test(router): remove duplicate cooldown-trigger and fallback-helper tests
#34416 landed its own TestTriggerCooldownForFailedDeployment/
TestRunAsyncFallbackTriggersCooldown classes and
test_ageneric_api_call_with_fallbacks_helper_stamps_failed_deployment_id
covering the exact same scenarios as this branch's earlier flat-function
tests, once its version of fallback_event_handlers.py was taken as-is
during the last merge. Dropping the redundant copies.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
The daily loop no longer captures the startup Router or bails when the alert type is off at startup, so config reloads take effect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams
Adds _iter_with_keepalive, _keepalive_from_deployment_config, and
_resolve_keepalive_seconds helpers to proxy_server.py. When enabled
(keepalive_seconds > 0 in request body or deployment litellm_params),
async_data_generator emits ': ping\n\n' SSE comment frames every N
seconds during idle upstream intervals, preventing load-balancer
idle-timeout drops on long chain-of-thought reasoning streams.
The hot path (keepalive_seconds absent or 0) is a plain async-for with
no per-chunk Task wrapping — zero overhead. Includes 8 new unit tests
covering sentinel emission, hot-path pass-through, early-close cleanup,
priority resolution, deployment-config lookup, and end-to-end heartbeat
emission through async_data_generator.
Registers keepalive_seconds in all_litellm_params (types/utils.py) so
the parameter is not stripped from request bodies. Adds the field to
LiteLLMParamsTypedDict and GenericLiteLLMParams (types/router.py) so
deployment YAML config is parsed and validated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): narrow BaseException to CancelledError to fix BLE001 strict lint gate
* fix: use explicit None check instead of truthiness in keepalive_seconds extraction
`float(raw or 0)` would treat any falsy value (including the integer 0)
as absent and substitute 0.0 before float() saw it. Replace with
`float(raw) if raw is not None else 0.0` so a caller-supplied zero is
correctly passed through to the `value <= 0` guard that disables
keepalive, rather than being silently overwritten.
* fix(proxy): don't guess a deployment's keepalive_seconds when model_id is missing
When a streaming response lacks _hidden_params.model_id, the fallback that
looks up keepalive_seconds by model_name previously returned the first
configured deployment's value, which could apply the wrong interval (or
override an explicit disable) when multiple deployments share the same
model_name with different keepalive_seconds settings. Only resolve the
fallback when every deployment agrees; otherwise leave it unset.
* fix(proxy): also treat an unset keepalive_seconds as disagreement in the fallback
The model_name fallback for keepalive_seconds only compared configured
values, filtering out deployments that leave the field unset entirely.
That meant a deployment with no keepalive_seconds configured could still
inherit a sibling deployment's interval when model_id is unavailable.
Compare the raw per-deployment value (including None for unset) so an
unconfigured deployment never silently adopts another's heartbeat.
* fix(proxy): deployment-level keepalive_seconds: 0 is a hard disable clients can't override
Previously an authenticated client's request-level keepalive_seconds always
took precedence over the deployment default, including when a deployment
operator explicitly set keepalive_seconds: 0 to disable heartbeats. That let
any client re-enable heartbeats for a deployment the operator opted out of,
using them to keep an idle-looking stream alive past a load balancer's idle
timeout and hold a parallel-request slot open longer than intended.
Treat an explicit deployment-level 0 as authoritative: resolve the
deployment's configured value first, and short-circuit to disabled before
ever looking at the request body if the deployment hard-disabled it.
* fix(proxy): a stale (unresolvable) model_id must not fall through to model_name guessing
A populated _hidden_params.model_id names the specific deployment that
served a stream. If that ID no longer resolves (e.g. a deployment removed
by a config reload mid-stream), the resolver was falling through to the
model_name-based fallback, letting a currently-live sibling deployment's
keepalive_seconds silently apply to a stream it never served. Return None
once a populated model_id fails to resolve, rather than degrading to a
guess.
* fix(proxy): keepalive_seconds is operator-only by default; require deployment opt-in for client override
A security review flagged that a client's request-level keepalive_seconds
could unilaterally enable heartbeats for any deployment, even one that
never configured keepalive_seconds at all, letting an authenticated client
defeat load-balancer idle timeouts and hold a parallel-request slot open
for longer than the deployment operator ever intended, with no way for
the operator to prevent it short of explicitly setting keepalive_seconds: 0.
Add allow_client_keepalive_override (default False) to LiteLLMParamsTypedDict
and GenericLiteLLMParams. _resolve_keepalive_seconds now ignores the request
body's keepalive_seconds entirely unless the resolved deployment explicitly
grants override permission; only the deployment's own configured value (or
disabled, if unset) applies otherwise. An explicit deployment-level 0 still
takes priority over everything, including a grant of override permission.
* fix(proxy): register allow_client_keepalive_override in all_litellm_params
Caught during live proxy verification against the real Anthropic API:
allow_client_keepalive_override was added to LiteLLMParamsTypedDict and
GenericLiteLLMParams but never registered in all_litellm_params, so it
leaked straight through into the provider request body as an unrecognized
field. Anthropic rejected every call on a deployment that had this field
configured with a 400 ("Extra inputs are not permitted"), regardless of
its value. Register it alongside keepalive_seconds so it's stripped
before reaching the provider, matching what keepalive_seconds already
does.
* feat(proxy): support keepalive_seconds via x-litellm-keepalive-seconds header
Some clients (e.g. the Vercel AI SDK) can set custom headers more easily
than extra JSON body fields. Add x-litellm-keepalive-seconds, following
the existing x-litellm-timeout/x-litellm-stream-timeout/x-litellm-num-retries
convention in LiteLLMProxyRequestSetup: the header merges into the same
data["keepalive_seconds"] field the request body already populates, so it
goes through the exact same _resolve_keepalive_seconds precedence and the
allow_client_keepalive_override gate -- a header can't enable heartbeats
for a deployment that hasn't opted in any more than the body field can.
Verified live against the real Anthropic API: the header produces real
heartbeats on an opt-in deployment (88 pings over a genuine long-reasoning
stall) and is silently ignored on a deployment without override permission
(0 pings), matching the existing body-field behavior exactly.
* chore: rebase onto litellm_internal_staging, drop unrelated credential_migration.py reformat, fix budget-ratchet drift
Rebased onto the current litellm_internal_staging (merge-base was 5 days
stale). Dropped the now-redundant schema.d.ts-only regen commit entirely
(the new base's own schema.d.ts already supersedes it) and regenerated
schema.d.ts fresh against the new base.
Reverted litellm/proxy/management_endpoints/credential_migration.py to
exactly match litellm_internal_staging: it was a pure reformat with no
semantic change, unrelated to this PR, flagged by review as unnecessary
noise in an encryption-migration file.
Fixed two lint-budget-ratchet failures caused by the base's ceilings
tightening since this branch last synced (other merged work lowered
ANN401/LIT001 budgets; this code was previously under budget and didn't
change):
- _iter_with_keepalive's aiter param: Any -> AsyncIterator[Any], a real
narrowing (it's always the result of .__aiter__()).
- _keepalive_from_deployment_config/_resolve_keepalive_seconds's
request_data param: dict[str, Any] -> Mapping[str, Any], matching the
existing read-only-dict convention already used elsewhere in this file
(_apply_ssrf_general_settings, _build_redis_usage_cache, etc.) for
params that are only ever read, never mutated.
- response/raw params: dropped the explicit `Any` annotation to match
async_data_generator's own (deliberately unannotated) `response` param,
its actual caller.
- litellm_pre_call_utils.py's new headers param: dict -> Mapping[str, str],
same read-only-dict rationale.
* fix(proxy): freeze the transient collections in the keepalive helpers
_iter_with_keepalive and _keepalive_from_deployment_config built a set
literal for asyncio.wait, a set comprehension for the per-deployment
config-agreement check, and two dict-literal fallbacks, all flagged by
the LIT002 mutable-collection-construction gate. Switched to a tuple
for asyncio.wait, a frozenset-wrapped generator plus next(iter(...))
for the config check, and a shared MappingProxyType({}) empty mapping
for the fallbacks.
* fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback
Greptile P1: when a streaming request falls back from model group A to
group B and the response's _hidden_params carries no model_id,
_keepalive_from_deployment_config fell straight through to guessing
via request_data["model"], which still names the pre-fallback group A
since the fallback handler mutates its own local **kwargs copy, not
this dict. request_data[metadata|litellm_metadata]["model_info"]["id"],
by contrast, is mutated on this same dict by
Router._update_kwargs_with_deployment on every attempt including
fallbacks (the same source ProxyLogging._build_litellm_call_info uses
for logging), so check it before falling through to the model-name
guess.
Added two regression tests that fail on the prior code (assert
get_model_list is never called once metadata.model_info.id resolves)
and pass with the fix.
* Revert "fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback"
This reverts commit d779067864.
* fix(proxy): satisfy the new LIT010/ANN001 gates in the keepalive helpers
litellm_internal_staging picked up a LIT010 (every local/module variable
must be declared Final unless it's genuinely rebound) and tightened
ANN001 (missing parameter annotations) since this branch last synced.
Annotated every single-assignment local and module constant with
Final, suppressed pending's loop-carried reassignment with
# rebind-ok, and typed the previously-bare response/raw parameters as
object with isinstance narrowing at their use sites instead of cast
(LIT006 discourages cast; validate into a concrete type instead).
Also swapped the hand-rolled getattr(response, "_hidden_params", None)
+ isinstance(hidden, dict) check for the existing
get_hidden_params_dict() helper already used for this exact purpose
elsewhere in this file and in common_request_processing.py.
* fix(proxy): re-resolve keepalive_seconds per chunk to track mid-stream fallback
Greptile P1: the router can perform a mid-stream fallback to a
different deployment partway through a stream (MidStreamFallbackError
in router.py), and Router._apply_fallback_hidden_params_to_item merges
the fallback deployment's hidden params onto every subsequent chunk.
But _resolve_keepalive_seconds was only ever called once, before
iteration started, against the pre-fallback response wrapper, so a
stream that fell back to a deployment with a different (or disabled)
keepalive policy kept using the original deployment's interval for the
rest of the stream.
_iter_with_keepalive now takes a resolve_keepalive_seconds(item)
callback and re-resolves after every real chunk using that chunk's own
_hidden_params (which do carry the fallback deployment's identity),
rather than trusting the value picked before iteration began. Updated
the three existing timing tests to inject a constant-returning
resolver, since they pin the sentinel/cancellation mechanics rather
than re-resolution, and added two regression tests (interval lowered
and raised mid-stream) that fail against the prior static-resolve
signature and pass with the fix.
* fix(proxy): keep re-resolving keepalive even when a stream starts disabled
Greptile P1: a stream that starts on a deployment with keepalive off
(or unset) skipped _iter_with_keepalive entirely at the call site, so
a mid-stream fallback to a deployment that enables it never got a
chance to activate heartbeats for the rest of that stream, risking the
exact load-balancer idle-timeout this feature exists to prevent.
_iter_with_keepalive now has an internal fast path for
keepalive_seconds <= 0 that still re-resolves after every chunk (no
asyncio.create_task/wait overhead while inactive, same cost as a bare
async for), so activation from a disabled start works the same way
deactivation and interval changes already do. The caller now only
skips wrapping entirely when there's no router to ever fall back
through in the first place (llm_router is None), rather than whenever
the first chunk's deployment happens to start with keepalive off.
Added a regression test that starts keepalive_seconds=0, has the
resolver enable a short interval on a later chunk, and asserts
sentinels appear afterward; it fails against the prior
call-site-gated code and passes with the fix.
* perf(proxy): memoize keepalive resolution per chunk's model_id
_resolve_keepalive_seconds ran a full llm_router.get_deployment() Pydantic
rebuild after every streamed chunk, even when keepalive was unconfigured
anywhere in the deployment list, since async_data_generator wraps every
stream once a router exists. Caching the result by model_id keeps mid-stream
fallback re-resolution correct while paying the router lookup once per
deployment instead of once per token.
* fix(proxy): expire cached keepalive resolution after a bounded TTL
veria-ai flagged that caching by model_id alone lets an already-in-flight
stream keep evading a live config reload (deployment removed, keepalive
disabled, or client override revoked) for the rest of the stream. Expiring
the memo after _KEEPALIVE_CACHE_TTL_SECONDS bounds that window instead of
freezing the resolved value for the stream's full lifetime, while still
avoiding a full deployment rebuild on every chunk in the steady state.
Also fixes add_litellm_data_for_backend_llm_call's now-required request_data
kwarg in the header-merge test, picked up by rebasing onto
litellm_internal_staging.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(arize): stop MCP CallToolResult from aborting span attribute setting
`call_mcp_tool` logs the MCP SDK's `CallToolResult`, a Pydantic model with
no `.get`. `_coerce_response_obj_for_attrs` left it untouched and
`_set_request_attributes` then raised AttributeError, which aborted the rest
of the attribute block, so MCP tool spans lost their invocation params,
input messages, and outputs.
Dump Pydantic models that lack `.get` to a dict, and guard the response
id/model reads the same way `_set_response_attributes` already does so any
other uncoercible response object degrades instead of crashing.
* feat(arize): render MCP tool calls as OpenInference TOOL spans
`call_mcp_tool` spans carry neither `messages` nor `choices`, so every
generic extraction path left Input and Output blank and the span showed only
provider/model metadata.
Emit `tool.name` from `metadata.mcp_tool_call_metadata`, `input.value` from
the tool arguments, and `output.value` from the `CallToolResult` content
(text parts when present, JSON otherwise). Arguments and results are user
content, so the input/output emit is gated on the same
`should_redact_message_logging` check the passthrough normalizer uses.
Reuse `_to_plain_dict` for the Pydantic coercion instead of the local
BaseModel branch added in the previous commit.
* fix(arize): annotate the new MCP helper parameters
The strict-rule gate flagged three new ANN001 violations. Type the payload
as StandardLoggingPayload | None and the coerced response as object, which
the isinstance guards already narrow.
* fix(arize): annotate the MCP helper against the type-discipline gate
LIT001 bans mutable collections in annotations, so the kwargs parameter
becomes Mapping[str, object]. should_redact_message_logging still declares a
dict it only ever reads, and widening it would cascade into core_helpers, so
the call carries a scoped ignore instead. Narrow the payload by None rather
than isinstance now that it is typed, and annotate the values read out of the
untyped logging payload.
* fix(arize): record empty MCP arguments and results instead of dropping them
Zero-argument tools record arguments={} and successful calls can return
content=[]; both were skipped by truthiness, leaving the generic placeholder
on Input and nothing on Output. Read structuredContent when content yields
no text, and cover the list_mcp_tools response shape.
* fix(arize): keep media parts in mixed MCP results
A result mixing text and media returned the text alone, so Arize showed
text/plain and dropped the image or resource parts.
---------
Co-authored-by: Sean Lee <yihsean@gmail.com>
The prompt editor is a base-ui Dialog at z-index 50. The create form houses it in
the same base-ui Dialog, so it stacks on top, but the edit form was an antd Modal
whose portal computes to z-index 1000, so the editor opened underneath it and was
neither readable nor clickable.
Move the edit form onto the Dialog the create form already uses, which puts the
whole nesting chain in one overlay layer. A dialog opened from inside another
dialog now reads as a drill-down rather than a stack: base-ui stamps
data-nested-dialog-open on the parent while a child is open, so the parent steps
aside instead of showing its own edges around a differently sized child.
Hiding the tab behind all_admin_roles took it away from org admins, who are
entitled to it: /v2/team/list?status=deleted returns 200 for them, scoped to
their own organizations. An org admin is an organization membership rather
than a global role, so their session carries user_role "internal_user" and no
role-based gate can ever see them.
Lift the membership lookup the left nav already did into a shared
useIsOrgAdmin hook, and let a capability opt into allowing org admins.
viewDeletedTeams is the only one that opts in; the backend still refuses org
admins on /v1/tool/list, /policies/list, /prompts/list and /audit, so those
gates stay as they are. The hook also accepts a session role of org_admin, in
case a deployment maps one through SSO.
* fix(bedrock): reject Anthropic server-side web_search tool with actionable error
Bedrock's Anthropic Messages endpoints cannot execute Anthropic's
server-side web_search tool, so forwarding it returns an opaque
"The provided request is not valid" 400 from Bedrock. Fail fast in the
invoke transform with an error that names the unsupported tool, the
model, and links the web search interception docs as the fix.
* refactor(bedrock): address review nits on web_search guard typing
Drops Any-typed router plumbing, immutable bucketing, generated dashboard API types, and adds endpoint plus resolution-fallback tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The proxy documentation lives in BerriAI/litellm-docs and any new env
key flagged by os.getenv() must be added there before the
test_env_keys.py CI check passes. Rather than fork the docs repo for
two niche tunables, hard-code the defaults:
- DEFAULT_DEPRECATION_WARN_DAYS = 30 (already overridable per-request
via ?warn_within_days=N on /model/deprecations).
- DEFAULT_DEPRECATION_CHECK_INTERVAL_SECONDS = 24h.
Both can still be raised as env-var follow-ups together with their docs
update if operators ask for it.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Surfaces deprecation_date metadata that is already shipped in
model_prices_and_context_window.json so operators get lead time to
migrate before a provider sunsets a model.
- New helper litellm.proxy.common_utils.model_deprecation classifies the
router's configured models into deprecated / imminent / upcoming
buckets. Resolution order: explicit model_info.deprecation_date >
model_info.base_model > litellm_params.model.
- New GET /model/deprecations (and /v1/model/deprecations) endpoint
returns a ModelDeprecationResponse, gated by user_api_key_auth.
- New AlertType.model_deprecation_warnings (in DEFAULT_ALERT_TYPES) plus
SlackAlerting.send_model_deprecation_alert dispatches a Slack message
for deprecated/imminent models. Severity is High when any model is
already past its date, Medium when only imminent.
- ProxyLogging.startup_event schedules a daily background task
(_run_scheduled_deprecation_check) when the alert type is enabled. The
interval is configurable via LITELLM_MODEL_DEPRECATION_CHECK_INTERVAL
and the warn window via LITELLM_MODEL_DEPRECATION_WARN_DAYS.
- Tests: 16 unit tests for the helper plus 4 for the Slack hook in
tests/test_litellm/.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
knip derives its entry points from vitest's `test.include`, which does not
cover `test.typecheck.include`, so the new `*.test-d.tsx` file read as an
unused file and failed the lint job. Declare the glob as an entry point.
Also drops the doc comment on `DataTableResolvedProps`; the rationale for
the resolved/public split belongs in the commit that introduced it.
