mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
8121 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tin-berri
|
79d412efc2
|
fix: net prompt-caching savings against the cache-write premium (#36452)
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
|
||
|
Yassin Kortam
|
d8762bf4db
|
fix(router): warn when a deployment's credentials contradict its provider (#36486)
A deployment that carries one provider's credentials while resolving to another is silently broken: litellm ignores the credentials and sends the request to the resolved provider. The common shape is a Bedrock model group where one entry lost its route prefix, so `model: claude-sonnet-5` with aws_region_name set resolves to the first-party Anthropic API and returns "x-api-key header is required". Because the router load balances across the group, only the fraction of requests routed to that entry fails, which reads as an intermittent provider outage rather than a config error, and nothing at startup says otherwise. Warn at deployment registration when provider-scoped credential params (aws_*, vertex_*) sit on a model that resolves elsewhere, naming the params, the resolved provider, and the likely missing prefix. Warn only: an operator may be overriding a route deliberately, so this must not block startup. Deployments litellm cannot classify are left alone. Resolves LIT-5391 |
||
|
shivam
|
66d9752db5 |
fix(anthropic): aggregate 5m/1h cache-write split across iterations path
The iterations branch in AnthropicConfig.calculate_usage summed cache_creation_input_tokens but never aggregated the per-iteration cache_creation 5m/1h breakdown, leaving cache_creation_token_details as None. As a result all cache-creation tokens fell back to the flat 5m write rate, underbilling 1h cache writes by up to 2x. Aggregate the ephemeral_5m/ephemeral_1h split across iterations so 1h writes are priced at the 1h rate. Fixes LIT-4868 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
shivam
|
2351aaba74 |
test(anthropic cost): scope local cost-map env flag with monkeypatch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
shivam
|
dc58c35bba |
fix(anthropic cost): apply regional geo uplift to cached tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
d0c65f83f1
|
fix(websearch): stop leaking interception control fields to providers (#36480)
The web-search interception hooks stamp _websearch_interception_emit_native_blocks and _websearch_interception_converted_stream onto kwargs to carry state across the agentic loop, but neither was registered in all_litellm_params. The param builder sweeps anything it does not recognize into the outbound request, so a provider that validates its body rejects the whole call: Bedrock Converse answers "_websearch_interception_emit_native_blocks: Extra inputs are not permitted" with a 400, which breaks every request interception touches on that route. Register both alongside their code-interpreter counterparts, which were already listed for exactly this reason. Resolves LIT-5391 |
||
|
Irosh
|
b16e6111d3 |
fix(mcp): scope authorization server issuer
Generated with AI Co-Authored-By: Claude Code <noreply@anthropic.com> |
||
|
Yassin Kortam
|
be5e9000b2
|
perf(spend): write each daily spend batch in one upsert statement (#36448)
The daily spend flush emitted one INSERT ... ON CONFLICT per aggregated key, so every replica put hundreds of statements on the database each interval, all contending for the same handful of hot rows and each holding its row locks for the rest of the enclosing batch transaction. LiteLLM_DailyTagSpend felt it worst because a request writes one row per tag, and litellm adds two user-agent tags of its own by default. A batch now goes out as a single multi-row statement. Rows are folded by the conflict tuple first, and every nullable member of that tuple is normalized to '': a NULL can never match itself in a unique index, so such a row was re-inserted on every flush rather than aggregating, and a NULL model made prisma reject the whole batch. |
||
|
mateo
|
2fe152a1d2 |
fix(proxy): only schedule the deprecation loop when alerting is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Deepanshu Lulla
|
05943b47a3
|
fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill (#35104)
* fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill A deployment that failed partway through a fallback chain (any attempt after the first) was silently exempt from cooldown, because the has_logged_async_failure dedup flag blocks the normal failure callback for every attempt past the first. _trigger_cooldown_for_failed_deployment now explicitly evaluates cooldown for that deployment when the dedup flag is set, using the same deployment-config > response-header > router-default precedence as the primary failure path, and skips advisor-orchestration failures. Deployment-ID resolution prefers the exception's stamped failed_deployment_id, now also set from the generic-API-call fallback path (rerank, embeddings, /v1/messages, etc.), falling back to metadata inspection for call paths that don't stamp it yet. CooldownCache also recomputes the remaining TTL when DualCache promotes a Redis entry into the in-memory layer: before this, a cooldown entry restored from Redis kept the in-memory layer's default 600s TTL regardless of the deployment's real cooldown_time, so a deployment could stay excluded from routing for up to 10 minutes after a much shorter cooldown had already expired. * fix(router): address Greptile review on the fallback-cooldown trigger Two P1 findings on PR #35104: - _trigger_cooldown_for_failed_deployment never incremented the deployment's per-minute failure counter before evaluating cooldown, so a fallback deployment's repeated retryable failures never accumulated toward the default percent-fail-rate threshold that _should_cooldown_deployment checks. - The metadata-bucket fallback (checking "metadata" before "litellm_metadata" for a deployment_model_name marker) could be fooled by a caller with permission to set metadata, since neither bucket's authorship can be determined without knowing the call's function_name. Removed it entirely; cooldown now requires the server-stamped failed_deployment_id, matching what the primary chat-completions path and the generic-API-call path (rerank, embeddings, /v1/messages, etc.) already set unconditionally. * fix(router): freeze the litellm_params fallback mapping to satisfy the type-discipline gate * fix(router): defer f-string interpolation in fallback-cooldown debug logs * fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget * fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks * fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one * fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout * fix(router): stamp dynamic client-side-credential id in completion fallback paths too The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential) deployment id on exceptions, but the regular _completion/_acompletion exception handlers still stamped the static shared deployment's id. A tenant using invalid forwarded credentials could generate repeated failures attributed to, and eventually cooling down, the shared deployment other tenants rely on. Extracted the stamping logic into one shared helper used by all three call sites (generic API, sync completion, async completion) so the fix and future changes to it stay in one place. * test(router): add direct-reference unit tests for the new stamping helper router_code_coverage.py's coverage gate flags _stamp_failed_deployment_id_with_effective_model_info as untested because it only sees the function invoked indirectly through _completion/_acompletion's exception handlers. Added two tests that call it directly, covering both the dynamic-id-present and static-fallback branches. * test(router): cover the timeout stamping branch and async active-cooldown append _acompletion's litellm.Timeout handler and async_get_active_cooldowns' happy path both lacked direct coverage despite their sibling branches (the generic Exception handler, the sync get_active_cooldowns) being tested. * test(router): remove duplicate cooldown-trigger and fallback-helper tests #34416 landed its own TestTriggerCooldownForFailedDeployment/ TestRunAsyncFallbackTriggersCooldown classes and test_ageneric_api_call_with_fallbacks_helper_stamps_failed_deployment_id covering the exact same scenarios as this branch's earlier flat-function tests, once its version of fallback_event_handlers.py was taken as-is during the last merge. Dropping the redundant copies. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
mateo
|
25f343a547 |
fix(proxy): re-read router and alert types on each deprecation check
The daily loop no longer captures the startup Router or bails when the alert type is off at startup, so config reloads take effect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Deepanshu Lulla
|
363d56f917
|
feat(proxy): add per-deployment keepalive_seconds SSE heartbeat to prevent load-balancer timeout on long streams (#34423)
* feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams
Adds _iter_with_keepalive, _keepalive_from_deployment_config, and
_resolve_keepalive_seconds helpers to proxy_server.py. When enabled
(keepalive_seconds > 0 in request body or deployment litellm_params),
async_data_generator emits ': ping\n\n' SSE comment frames every N
seconds during idle upstream intervals, preventing load-balancer
idle-timeout drops on long chain-of-thought reasoning streams.
The hot path (keepalive_seconds absent or 0) is a plain async-for with
no per-chunk Task wrapping — zero overhead. Includes 8 new unit tests
covering sentinel emission, hot-path pass-through, early-close cleanup,
priority resolution, deployment-config lookup, and end-to-end heartbeat
emission through async_data_generator.
Registers keepalive_seconds in all_litellm_params (types/utils.py) so
the parameter is not stripped from request bodies. Adds the field to
LiteLLMParamsTypedDict and GenericLiteLLMParams (types/router.py) so
deployment YAML config is parsed and validated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): narrow BaseException to CancelledError to fix BLE001 strict lint gate
* fix: use explicit None check instead of truthiness in keepalive_seconds extraction
`float(raw or 0)` would treat any falsy value (including the integer 0)
as absent and substitute 0.0 before float() saw it. Replace with
`float(raw) if raw is not None else 0.0` so a caller-supplied zero is
correctly passed through to the `value <= 0` guard that disables
keepalive, rather than being silently overwritten.
* fix(proxy): don't guess a deployment's keepalive_seconds when model_id is missing
When a streaming response lacks _hidden_params.model_id, the fallback that
looks up keepalive_seconds by model_name previously returned the first
configured deployment's value, which could apply the wrong interval (or
override an explicit disable) when multiple deployments share the same
model_name with different keepalive_seconds settings. Only resolve the
fallback when every deployment agrees; otherwise leave it unset.
* fix(proxy): also treat an unset keepalive_seconds as disagreement in the fallback
The model_name fallback for keepalive_seconds only compared configured
values, filtering out deployments that leave the field unset entirely.
That meant a deployment with no keepalive_seconds configured could still
inherit a sibling deployment's interval when model_id is unavailable.
Compare the raw per-deployment value (including None for unset) so an
unconfigured deployment never silently adopts another's heartbeat.
* fix(proxy): deployment-level keepalive_seconds: 0 is a hard disable clients can't override
Previously an authenticated client's request-level keepalive_seconds always
took precedence over the deployment default, including when a deployment
operator explicitly set keepalive_seconds: 0 to disable heartbeats. That let
any client re-enable heartbeats for a deployment the operator opted out of,
using them to keep an idle-looking stream alive past a load balancer's idle
timeout and hold a parallel-request slot open longer than intended.
Treat an explicit deployment-level 0 as authoritative: resolve the
deployment's configured value first, and short-circuit to disabled before
ever looking at the request body if the deployment hard-disabled it.
* fix(proxy): a stale (unresolvable) model_id must not fall through to model_name guessing
A populated _hidden_params.model_id names the specific deployment that
served a stream. If that ID no longer resolves (e.g. a deployment removed
by a config reload mid-stream), the resolver was falling through to the
model_name-based fallback, letting a currently-live sibling deployment's
keepalive_seconds silently apply to a stream it never served. Return None
once a populated model_id fails to resolve, rather than degrading to a
guess.
* fix(proxy): keepalive_seconds is operator-only by default; require deployment opt-in for client override
A security review flagged that a client's request-level keepalive_seconds
could unilaterally enable heartbeats for any deployment, even one that
never configured keepalive_seconds at all, letting an authenticated client
defeat load-balancer idle timeouts and hold a parallel-request slot open
for longer than the deployment operator ever intended, with no way for
the operator to prevent it short of explicitly setting keepalive_seconds: 0.
Add allow_client_keepalive_override (default False) to LiteLLMParamsTypedDict
and GenericLiteLLMParams. _resolve_keepalive_seconds now ignores the request
body's keepalive_seconds entirely unless the resolved deployment explicitly
grants override permission; only the deployment's own configured value (or
disabled, if unset) applies otherwise. An explicit deployment-level 0 still
takes priority over everything, including a grant of override permission.
* fix(proxy): register allow_client_keepalive_override in all_litellm_params
Caught during live proxy verification against the real Anthropic API:
allow_client_keepalive_override was added to LiteLLMParamsTypedDict and
GenericLiteLLMParams but never registered in all_litellm_params, so it
leaked straight through into the provider request body as an unrecognized
field. Anthropic rejected every call on a deployment that had this field
configured with a 400 ("Extra inputs are not permitted"), regardless of
its value. Register it alongside keepalive_seconds so it's stripped
before reaching the provider, matching what keepalive_seconds already
does.
* feat(proxy): support keepalive_seconds via x-litellm-keepalive-seconds header
Some clients (e.g. the Vercel AI SDK) can set custom headers more easily
than extra JSON body fields. Add x-litellm-keepalive-seconds, following
the existing x-litellm-timeout/x-litellm-stream-timeout/x-litellm-num-retries
convention in LiteLLMProxyRequestSetup: the header merges into the same
data["keepalive_seconds"] field the request body already populates, so it
goes through the exact same _resolve_keepalive_seconds precedence and the
allow_client_keepalive_override gate -- a header can't enable heartbeats
for a deployment that hasn't opted in any more than the body field can.
Verified live against the real Anthropic API: the header produces real
heartbeats on an opt-in deployment (88 pings over a genuine long-reasoning
stall) and is silently ignored on a deployment without override permission
(0 pings), matching the existing body-field behavior exactly.
* chore: rebase onto litellm_internal_staging, drop unrelated credential_migration.py reformat, fix budget-ratchet drift
Rebased onto the current litellm_internal_staging (merge-base was 5 days
stale). Dropped the now-redundant schema.d.ts-only regen commit entirely
(the new base's own schema.d.ts already supersedes it) and regenerated
schema.d.ts fresh against the new base.
Reverted litellm/proxy/management_endpoints/credential_migration.py to
exactly match litellm_internal_staging: it was a pure reformat with no
semantic change, unrelated to this PR, flagged by review as unnecessary
noise in an encryption-migration file.
Fixed two lint-budget-ratchet failures caused by the base's ceilings
tightening since this branch last synced (other merged work lowered
ANN401/LIT001 budgets; this code was previously under budget and didn't
change):
- _iter_with_keepalive's aiter param: Any -> AsyncIterator[Any], a real
narrowing (it's always the result of .__aiter__()).
- _keepalive_from_deployment_config/_resolve_keepalive_seconds's
request_data param: dict[str, Any] -> Mapping[str, Any], matching the
existing read-only-dict convention already used elsewhere in this file
(_apply_ssrf_general_settings, _build_redis_usage_cache, etc.) for
params that are only ever read, never mutated.
- response/raw params: dropped the explicit `Any` annotation to match
async_data_generator's own (deliberately unannotated) `response` param,
its actual caller.
- litellm_pre_call_utils.py's new headers param: dict -> Mapping[str, str],
same read-only-dict rationale.
* fix(proxy): freeze the transient collections in the keepalive helpers
_iter_with_keepalive and _keepalive_from_deployment_config built a set
literal for asyncio.wait, a set comprehension for the per-deployment
config-agreement check, and two dict-literal fallbacks, all flagged by
the LIT002 mutable-collection-construction gate. Switched to a tuple
for asyncio.wait, a frozenset-wrapped generator plus next(iter(...))
for the config check, and a shared MappingProxyType({}) empty mapping
for the fallbacks.
* fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback
Greptile P1: when a streaming request falls back from model group A to
group B and the response's _hidden_params carries no model_id,
_keepalive_from_deployment_config fell straight through to guessing
via request_data["model"], which still names the pre-fallback group A
since the fallback handler mutates its own local **kwargs copy, not
this dict. request_data[metadata|litellm_metadata]["model_info"]["id"],
by contrast, is mutated on this same dict by
Router._update_kwargs_with_deployment on every attempt including
fallbacks (the same source ProxyLogging._build_litellm_call_info uses
for logging), so check it before falling through to the model-name
guess.
Added two regression tests that fail on the prior code (assert
get_model_list is never called once metadata.model_info.id resolves)
and pass with the fix.
* Revert "fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback"
This reverts commit
|
||
|
yucheng-berri
|
5c1623888e
|
fix(arize): trace MCP tool calls instead of crashing on CallToolResult (#36453)
* fix(arize): stop MCP CallToolResult from aborting span attribute setting
`call_mcp_tool` logs the MCP SDK's `CallToolResult`, a Pydantic model with
no `.get`. `_coerce_response_obj_for_attrs` left it untouched and
`_set_request_attributes` then raised AttributeError, which aborted the rest
of the attribute block, so MCP tool spans lost their invocation params,
input messages, and outputs.
Dump Pydantic models that lack `.get` to a dict, and guard the response
id/model reads the same way `_set_response_attributes` already does so any
other uncoercible response object degrades instead of crashing.
* feat(arize): render MCP tool calls as OpenInference TOOL spans
`call_mcp_tool` spans carry neither `messages` nor `choices`, so every
generic extraction path left Input and Output blank and the span showed only
provider/model metadata.
Emit `tool.name` from `metadata.mcp_tool_call_metadata`, `input.value` from
the tool arguments, and `output.value` from the `CallToolResult` content
(text parts when present, JSON otherwise). Arguments and results are user
content, so the input/output emit is gated on the same
`should_redact_message_logging` check the passthrough normalizer uses.
Reuse `_to_plain_dict` for the Pydantic coercion instead of the local
BaseModel branch added in the previous commit.
* fix(arize): annotate the new MCP helper parameters
The strict-rule gate flagged three new ANN001 violations. Type the payload
as StandardLoggingPayload | None and the coerced response as object, which
the isinstance guards already narrow.
* fix(arize): annotate the MCP helper against the type-discipline gate
LIT001 bans mutable collections in annotations, so the kwargs parameter
becomes Mapping[str, object]. should_redact_message_logging still declares a
dict it only ever reads, and widening it would cascade into core_helpers, so
the call carries a scoped ignore instead. Narrow the payload by None rather
than isinstance now that it is typed, and annotate the values read out of the
untyped logging payload.
* fix(arize): record empty MCP arguments and results instead of dropping them
Zero-argument tools record arguments={} and successful calls can return
content=[]; both were skipped by truthiness, leaving the generic placeholder
on Input and nothing on Output. Read structuredContent when content yields
no text, and cover the list_mcp_tools response shape.
* fix(arize): keep media parts in mixed MCP results
A result mixing text and media returned the text alone, so Arize showed
text/plain and dropped the image or resource parts.
---------
Co-authored-by: Sean Lee <yihsean@gmail.com>
|
||
|
ryan-crabbe-berri
|
20354bfcdc
|
fix(bedrock): reject Anthropic server-side web_search tool with actionable error (#36473)
* fix(bedrock): reject Anthropic server-side web_search tool with actionable error Bedrock's Anthropic Messages endpoints cannot execute Anthropic's server-side web_search tool, so forwarding it returns an opaque "The provided request is not valid" 400 from Bedrock. Fail fast in the invoke transform with an error that names the unsupported tool, the model, and links the web search interception docs as the fix. * refactor(bedrock): address review nits on web_search guard typing |
||
|
mateo
|
8f1aea5e0a |
refactor(proxy): tighten model deprecation typing and cover the endpoint
Drops Any-typed router plumbing, immutable bucketing, generated dashboard API types, and adds endpoint plus resolution-fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Cursor Agent
|
f249356e16 |
feat(proxy): proactive model deprecation alerts and /model/deprecations endpoint
Surfaces deprecation_date metadata that is already shipped in model_prices_and_context_window.json so operators get lead time to migrate before a provider sunsets a model. - New helper litellm.proxy.common_utils.model_deprecation classifies the router's configured models into deprecated / imminent / upcoming buckets. Resolution order: explicit model_info.deprecation_date > model_info.base_model > litellm_params.model. - New GET /model/deprecations (and /v1/model/deprecations) endpoint returns a ModelDeprecationResponse, gated by user_api_key_auth. - New AlertType.model_deprecation_warnings (in DEFAULT_ALERT_TYPES) plus SlackAlerting.send_model_deprecation_alert dispatches a Slack message for deprecated/imminent models. Severity is High when any model is already past its date, Medium when only imminent. - ProxyLogging.startup_event schedules a daily background task (_run_scheduled_deprecation_check) when the alert type is enabled. The interval is configurable via LITELLM_MODEL_DEPRECATION_CHECK_INTERVAL and the warn window via LITELLM_MODEL_DEPRECATION_WARN_DAYS. - Tests: 16 unit tests for the helper plus 4 for the Slack hook in tests/test_litellm/. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
fancybear-dev
|
f1ed4690bb
|
fix(proxy): treat SAML as configured in UI SSO detection (#36196)
* fix(proxy): treat SAML as configured in UI SSO detection _has_user_setup_sso only checked OAuth client IDs, so SAML-only setups left /.well-known/litellm-ui-config sso_configured=false and the login button gray even when SAML IdP metadata was set. Include SAML_IDP_METADATA_URL / SAML_IDP_METADATA_XML so UI discovery matches the login redirect path. * chore: adhere to comment policy Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: apply suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
76ad1c319d
|
feat(proxy): add GET /v1/indexes to list vector store indexes (#36289)
* fix(scripts): stop type-discipline checker reading Literal strings as forward refs The checker re-parsed every string constant inside an annotation as a forward reference, so Literal["list"] was counted as the mutable list type. Skip Literal subtrees and ratchet the LIT001 ceiling down to the corrected count. * fix(proxy): keep lazy openapi snapshot fragments for transitively imported features generate_snapshot skipped register_fn for any feature module already in sys.modules, so a module pulled in transitively by an earlier feature never mounted its routes and its fragment silently vanished on regen (vector_store_management). Route collection also matched path_prefixes only, dropping suffix-matched routes from fragments. Register every feature and collect routes with feat.matches, mirroring the runtime loader. * feat(proxy): add GET /v1/indexes to list vector store indexes /v1/indexes was POST-only, so indexes created through it could never be viewed again. Add an admin-only list endpoint returning the stored index rows newest first, fix the stale index_create docstring curl, and regenerate the lazy openapi snapshot and dashboard schema types. * chore(proxy): defer lazy openapi snapshot catch-up regen to a follow-up Reverts _lazy_openapi_snapshot.json and schema.d.ts to the staging versions. The snapshot was months stale, so regenerating it here buried the actual change under ten thousand generated lines. A follow-up will land the regen together with CI enforcement that keeps the snapshot current. Until then GET /v1/indexes is served but absent from the dashboard's generated types, which the UI step needs anyway. * fix(proxy): use Annotated dependency to avoid new B008 violation |
||
|
ryan-crabbe-berri
|
c40828509b
|
fix(reset_budget_job): atomic budget cascade with chunked reset scans (#36287)
* fix(reset_budget_job): advance budget_reset_at atomically with the spend cascade A postgres timeout mid-cascade previously left LiteLLM_BudgetTable rows stamped for the next window while team member, enduser, org and tag spend stayed at cap, so every later tick skipped them until the window rolled over. All cascade writes and the budget_reset_at advance now share one prisma batch transaction; a failed run persists nothing and the rows stay due for the next ~10 minute tick. Cache and counter invalidation runs only after commit, and the catch-all enduser log line now names the cascade. * fix(reset_budget_job): elect one runner per tick and chunk the reset scans Every pod and worker previously ran the reset job every ~10 minutes, each fetching every expired row with no limit and writing one giant transaction at the same calendar-aligned boundary; that concurrency is what piled up postgres lock contention and timeouts. The job now takes the shared PodLockManager redis lock (no redis keeps the old behavior), and each phase walks its due rows in 500-row chunks, one transaction per chunk, stopping when a chunk is short, makes no forward progress, or hits the per-run cap; leftovers wait for the next tick. * chore(lint): ratchet budget ceilings down for fixed violations * fix(reset_budget_job): harden chunk loop, fail open on redis errors, heartbeat the lock Review fixes on the two prior commits. Reset scans now skip rows with no budget_duration, so permanently due rows can neither starve a phase nor have a lifetime cap zeroed every tick. Chunk progress counts rows whose new budget_reset_at actually cleared the cutoff, so a zero-length duration cannot burn the per-run chunk cap. A failed lock acquire only skips the run when another pod verifiably holds the lock; a broken redis runs unguarded instead of silently disabling resets fleet-wide. Partial row failures report real progress and fire the failure hook without killing the phase. The leader re-asserts the lock between phases and stops if another pod took over, and the budget window advance uses update_many so a tier deleted mid-chunk cannot abort the transaction. Lint budget ceilings re-ratcheted for the net-fixed violations. * fix(reset_budget_job): renew the leader lease and reject non-positive budget durations Bot review follow-ups. PodLockManager now extends the lock TTL when the holding pod re-acquires, via an atomic compare-and-expire script with a plain SET fallback, so a run longer than the TTL keeps its lease instead of silently sharing the job with another pod. The positive-duration validation that team member endpoints already had is hoisted to management common_utils and applied to key, internal user, budget, customer and team intake, so a tenant can no longer create zero-duration budgets whose permanently due rows starve other tenants' resets. Such durations now return 400 at intake; existing rows are untouched. * refactor(reset_budget_job): defer leader election to a follow-up PR * fix(reset_budget_job): satisfy strict lint gates String defaults for the two getenv calls (PLW1508) and the chunk outcome returns moved to try/else (TRY300). |
||
|
yucheng-berri
|
ade5a425e8
|
fix(proxy): isolate guardrail load failures per row (#36432)
* fix(proxy): isolate guardrail load failures per row One DB guardrail row that fails to initialize aborted the whole _init_guardrails_in_db loop, so a single typo'd guardrail type or a missing required param left the proxy running with zero DB guardrails registered and requests that should have been blocked reaching the provider. Catch per row around sync_guardrail_from_db, log the guardrail name, id and error, and continue with the remaining rows. The failing row's id is still added to db_guardrail_ids before the attempt so reconcile_db_guardrails cannot mistake a live row for a deleted one. * test(proxy): drop inline note and record reconcile via a handler double Replaces the patched bound method with an InMemoryGuardrailHandler subclass that records what reconcile_db_guardrails received, so the test injects a double instead of swapping a method on a live object. |
||
|
yuneng-jiang
|
3726bceb53
|
Merge pull request #36336 from BerriAI/litellm_/standard-lists-api-d1dc4a
test(proxy): guard management_v1 against fastapi names removed in supported releases |
||
|
Devin AI
|
584e88073d | chore: merge litellm_internal_staging | ||
|
Yassin Kortam
|
ade805ef0c
|
feat(rate limiting): configurable estimated output tokens per key, team and model (#36143) | ||
|
Devin AI
|
f0c3d8dcda |
chore: merge litellm_internal_staging into litellm_anthropic_fast_mode_speed_usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
e014b341c8
|
feat(ptu): gate PTU flat-cost attribution behind an opt-in env var (#36138)
LITELLM_ENABLE_PTU_COST_ATTRIBUTION, read through get_secret_bool and defaulting to
false, makes the whole PTU flat-cost feature inert unless an operator opts in. The
daily rollup cron is not registered at all, so no sentinel row is ever written;
/model/new and /model/{id}/update reject a request that carries any PTU model_info
field with a 400 naming the fields and the env var rather than dropping them; the
daily activity read path reports zero flat cost; and the model add and edit forms
hide the four PTU inputs.
The read gate lives where flat cost enters SpendMetrics rather than in the aggregated
SQL select. /team/daily/activity, the endpoint the Usage page reads, is served by the
paginated find_many path and never runs that query, so forcing the select to a
constant zero would have left the reporting surface that matters still showing flat
cost.
Sentinel row filtering is deliberately not gated. An operator can enable the flag,
accrue rows under the __ptu_flat_cost__ api_key, then disable it, and those rows stay
in LiteLLM_DailyTeamSpend; gating the filter too would surface the sentinel as a bogus
api_key and mint a provider bucket for its empty provider. Response fields keep their
shape and report 0.0, so typed clients are unaffected, and the migration and the
ModelInfo field declarations are untouched.
The write gate reads the incoming request rather than the merged deployment, so a
model configured during an earlier opt-in stays editable, and the edit form drops the
PTU keys from the payload instead of sending nulls that would clear stored config.
The dashboard reads the flag from a read-only enable_ptu_cost_attribution key on
/get/ui_settings, computed from the environment on every read. It is deliberately not
an allowlisted persisted setting, and PATCH /update/ui_settings rejects it with a 400,
so an admin cannot flip an env-gated feature from the UI.
Two review findings on the gate itself. The PTU clear loop now runs only when the
feature is enabled: the write gate rejects a value but lets an explicit null through,
and a client round-tripping a model_info blob sends the PTU keys as nulls, so a
disabled proxy would have quietly erased a billing configuration set up during an
earlier opt-in. Disabling pauses PTU rather than discarding its setup. And the
dashboard flag is re-read every thirty seconds instead of the hour the other UI settings
use, since those are persisted records while this one tracks the proxy process; a
restart that flips the variable would otherwise leave the model form offering inputs
the backend now rejects. The flag is polled rather than only marked stale, since a form
that stays mounted and focused never refetches on its own.
The read gate checks the row before the flag. It runs once per metric accumulation and a
record fans out across roughly a dozen breakdowns, while the flag reads through the secret
manager uncached, so consulting it for every accumulation put thousands of lookups on a
shared endpoint that made none before. Only a row actually carrying flat cost reaches it.
|
||
|
Mateo Wang
|
9de3315dad
|
Merge pull request #36403 from BerriAI/litellm_model_registry_deprecation_audit
fix(model_prices): refresh deprecation dates, correct xAI pricing and add missing provider models |
||
|
Vairo Di Pasquale
|
ea6c18baa5
|
fix(cost): price dict-shaped image input token details at image rate (#33490)
calculate_image_response_cost_from_usage read input_tokens_details with getattr(), but OpenAI images.edit responses carry it as a plain dict, so both fields came back None and all input tokens were priced at input_cost_per_token instead of input_cost_per_image_token (e.g. $5/M instead of $8/M for gpt-image-2). Read it with the dict-tolerant _get_token_detail_value helper, as the output side of the same function already does. Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> |
||
|
Deepanshu Lulla
|
0580465384
|
feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416)
* feat(router): add per-deployment allowed_fails_policy and cooldown_time override support
Three bugs fixed in the router cooldown system: (1) deployment-level allowed_fails and
allowed_fails_policy in model_info now take precedence over router-level settings in
_should_cooldown_deployment; (2) failed fallback deployments now get evaluated for
cooldown via _trigger_cooldown_for_failed_deployment, bypassing the Logging dedup gate;
(3) DualCache promotes Redis cooldown entries using default 600s TTL instead of true
remaining cooldown time -- _corrected_active_cooldown now evicts expired entries and
corrects stale in-memory TTLs on backfill. Adds ServiceUnavailableError, BadGatewayError,
and NotFoundError fields to AllowedFailsPolicy and cooldown_time to LiteLLMParamsTypedDict.
* fix(router): gate fallback cooldown trigger on has_logged_async_failure; use only litellm_metadata for deployment ID
* fix(router): use X | Y union syntax to fix UP007 strict lint gate
* test(router_utils): add coverage for _trigger_cooldown_for_failed_deployment and has_logged_async_failure gate
* test(router_utils): cover deployment cooldown override and exception swallow paths
* fix(router): add InternalServerError/ServiceUnavailableError/BadGatewayError/NotFoundError to router-level get_allowed_fails_from_policy
* fix(router): format router.py and add router-level policy tests
* test(router): add CI-visible coverage for per-deployment cooldown policy
Tests for `_get_deployment_cooldown_policy`, `_resolve_allowed_fails_from_policy`,
and `_should_cooldown_based_on_deployment_policy` (cooldown_handlers.py), the
`_corrected_active_cooldown` branches in CooldownCache, and the four new
exception-type branches in `Router.get_allowed_fails_from_policy` (router.py) --
all in `tests/test_litellm/` which the enterprise-routing CI job runs.
* fix(router): use is not None guard for cooldown_time_override in should_cooldown_based_on_allowed_fails_policy
A cooldown_time_override of 0 was previously treated as falsy and silently
fell through to the router-level cooldown_time value. Switched to an explicit
is not None check so that zero is honored as a valid override.
Added a regression test covering the zero case.
* fix(router): honor has_logged_async_failure and metadata for fallback cooldown; support both model_info and litellm_params locations
Manual verification against a live proxy surfaced that the fallback-cooldown-gap
trigger never actually fired: the has_logged_async_failure check read a plain
attribute that Logging never sets (the real flag lives in model_call_details),
and the deployment_id lookup only trusted litellm_metadata, which regular chat
completions never populate (only batch/thread/file endpoints do). Router
overwrites model_info on whichever key is present before every attempt, so
metadata is equally authoritative there, not caller-controlled as previously
assumed. Also let allowed_fails/allowed_fails_policy/cooldown_time be set under
either model_info or litellm_params, each preferring its own canonical location.
* fix(router): fix ContentPolicyViolationError policy shadowing and partial-policy zero-threshold
Two bugs from Greptile review on PR #34416:
- ContentPolicyViolationError subclasses BadRequestError, so listing
BadRequestError first in _EXCEPTION_POLICY_FIELDS made the isinstance
check always match BadRequestError for content-policy errors, using the
wrong allowed_fails threshold. Reordered so the subclass is checked first.
- A deployment with a partial allowed_fails_policy and no deployment-wide
allowed_fails forced allowed_fails_override=0 for any exception type its
policy didn't cover, cooling the deployment down on the first unrelated
failure. Now defers to router-level behavior for uncovered exception
types instead of forcing an immediate cooldown.
* fix(router): only trust a metadata/litellm_metadata bucket the router itself wrote deployment info into
veria-ai flagged that preferring litellm_metadata whenever present could pick up a
caller-supplied litellm_metadata.model_info.id (preserved via allow_client_pricing_override)
instead of the metadata bucket the router actually populated for a regular completion's
fallback attempt, naming an arbitrary "victim" deployment for cooldown.
Router._update_kwargs_with_deployment() always writes model_info and
deployment_model_name into the same bucket together. Only trust a bucket that
carries deployment_model_name alongside model_info, since that marker is only
ever set by the router itself, not by request-body metadata.
* test(router): add regression coverage for ContentPolicyViolationError policy shadowing
The subclass-ordering fix in commit
|
||
|
yucheng-berri
|
457be8f00a
|
feat(ptu): surface PTU flat cost on the daily activity read path (#35391)
Aggregate the ptu_flat_cost written by the rollup into SpendMetrics.flat_cost and DailySpendMetadata.total_flat_cost, so /team/daily/activity returns flat cost alongside per-request spend. The aggregated SQL path selects ptu_flat_cost only for LiteLLM_DailyTeamSpend and a constant zero for the other daily tables, keeping the response shape uniform. Rows written under the PTU sentinel api_key add their flat cost to every parent bucket (per-model, per-day, per-team totals) but never appear as an api_key row in any breakdown, and are excluded from the per-request provider breakdown; the sentinel string is not a real key alias. Both flat_cost and total_flat_cost default to zero, so a read of any entity without PTU config is unchanged. The sentinel row now keys on the deployment id, so the per-model breakdown keys it on model_group instead. That breakdown key is rendered directly as a label by the Usage page and the daily_with_models export, and a deployment id there would read as a UUID. Two deployments sharing a public name merge under it, which is the collapse the write path used to do by summing them into one row. Request rows are untouched and still key on model, since their model_group is a routing concept rather than a display name. |
||
|
mateo-berri
|
726abc68a6 |
fix(cost_calc): apply xAI's inclusive 200k threshold to the token-type breakdown
Derive threshold inclusivity from the provider inside generic_cost_per_token and get_token_type_cost_breakdown so the spend-log breakdown can never disagree with the billed totals at exactly 200k prompt tokens |
||
|
Deepanshu Lulla
|
9ce96c2d34
|
feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418)
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both setters after assigning litellm_trace_id so every JSON log record emitted within the async request context carries trace_id and, when provided, session_id — enabling log correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without any changes to individual log call sites. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields * fix(logging): always reset session_id_var to empty string when no session_id provided * feat: gate request correlation IDs in logs behind request_correlation_in_logs flag * refactor: move correlation ID injection into CorrelationContextFilter * feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload Plaintext log lines (json_logs off) now get the same trace_id/session_id suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a visible effect regardless of log format. StandardLoggingPayload gets a new independent session_id field, populated from litellm_session_id. trace_id's existing session_id-first fallback is preserved when request_correlation_in_logs is off; with the flag on, an explicit litellm_trace_id now takes priority over litellm_session_id so the two fields carry genuinely independent values. * fix(logging): restore correlation context after nested calls; sanitize correlation ids Addresses two review findings on this PR. CorrelationContextFilter's trace_id/session_id contextvars were set on every Logging.__init__ but never reset, so a nested LiteLLM call sharing the same asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call, an MCP sampling call) would leave the outer request's subsequent log lines stamped with the nested call's ids instead of its own. set_trace_id/ set_session_id now return their contextvars.Token, and Logging stores them and resets both once its own success/failure handler actually completes, via a new idempotent _restore_correlation_context() called from all four terminal handlers. set_trace_id/set_session_id also now strip control characters and bound length before storing a caller-controlled trace_id/session_id, since these values can originate from request input (litellm_session_id, x-litellm- trace-id) and get interpolated into plain-text log lines - without this, a caller could embed \r/\n or escape sequences to forge fake log entries. * fix(logging): restore correlation context after nested calls, not before The previous commit called _restore_correlation_context() as the first line of each terminal handler, before that handler's own callback dispatch loop runs. That's backwards: a nested LiteLLM call triggered from within a callback (e.g. a guardrail's own LLM-as-judge call) would then capture the *already-reset* value as its own pre-call baseline, and its own reset would restore to that instead of the true outer value - verified live to still leak. success_handler/async_success_handler/failure_handler/async_failure_handler are now thin wrappers: the original bodies move to _success_handler_body/etc, called inside a try/finally that restores context only once the full body - including any nested calls its own callback dispatch triggers - has actually finished, mirroring proper stack-scoped nesting semantics. * test(logging): cover async_failure_handler's correlation-context restore Codecov flagged the new async_failure_handler wrapper (try/finally around _async_failure_handler_body) as uncovered - the method had no direct test at all before this PR's refactor split it into a wrapper. Adds a test that awaits it directly and asserts both that async_log_failure_event still fires and that _restore_correlation_context() puts the pre-call trace_id/session_id back. * fix(logging): restore correlation context by value, not by contextvars.Token veria-ai correctly flagged that contextvars.Token.reset() only works in the exact Context it was created in, and litellm's async success path (and streaming failure path) dispatch async_success_handler/async_failure_handler via asyncio.create_task and the global logging worker - a different Context than Logging.__init__ ran in. reset_trace_id/reset_session_id silently swallowed the resulting ValueError, so the restore was a no-op for exactly those paths. Verified independently: reproduced the raw contextvars behavior, then confirmed litellm's async success dispatch really does go through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py). Logging now captures the pre-call *value* (not a Token) and restores via a plain set_trace_id()/set_session_id() call, which works regardless of which Task/Context calls it. reset_trace_id/reset_session_id are removed as dead/unreliable code. Added a regression test that spawns __init__ and the restore in different asyncio Tasks - confirmed it fails against the prior Token-based commit and passes here. * fix(logging): restore correlation context in the originating task too Greptile's re-review correctly identified a remaining gap: for a successful acompletion(), async_success_handler is dispatched via asyncio.create_task + the global logging worker into a *different* Task than the one wrapper_async/Logging.__init__ ran in. The prior fix ( |
||
|
yucheng-berri
|
61218f5f9f
|
feat(ptu): daily rollup writes per-model PTU flat cost by active hour (#35343)
Add the daily rollup that reads PTU config off model deployments and writes flat cost to LiteLLM_DailyTeamSpend. For each UTC day a deployment carrying ptu_count and cost_per_ptu_per_hour accrues ptu_count * cost_per_ptu_per_hour * active_hours, where active_hours is the overlap between the day and the optional [ptu_effective_from, ptu_effective_to) window clamped to 24; a window opening at 23:00 charges one hour that day. Rows use a sentinel api_key so they stay distinguishable from per-request rows and share the existing unique constraint, and the write is idempotent so re-runs never double count. The cron is registered at proxy startup and runs at 00:15 UTC. Pricing one day per fire leaves two ways for a day to end up unpriced and stay that way: a window backdated at configuration time, which no fire ever revisits, and a fire that is missed or lands late, which the next one does not replay because the billed day comes from the wall clock rather than the scheduled time. Both are silent, since the failure alert only fires for a charge that was attempted. Each scheduled run therefore follows the day's reconcile with a catch-up pass that prices the (team, model, date) charges inside every declared window that carry no row yet, bounded at the earliest ptu_effective_from and floored at PTU_ROLLUP_MAX_BACKFILL_DAYS. It writes only what is missing: a day already priced keeps the amount it was billed whatever the config says now, and it runs no prune, so deciding a row is stale stays the single-day path's job. Zero-cost days write nothing, which leaves an out-of-window day reconsidered each run rather than recorded as done. A catch-up pass that fails cannot take the day's own result with it, and an explicit target_date still means reconcile exactly that day. The sentinel row keys on the deployment id, with the operator-facing name alongside it in model_group, which sits outside the table's unique key. The name is what a usage view displays, but a deployment can be renamed, and two runs holding config views from either side of a rename then wrote the same day under two different keys, so nothing collided and both charges survived. A multi-pod rig reproduced that as a permanent double charge that no later run repaired. Keyed on the id both writes land on one key and the upsert collapses them; when the rate changed too, last writer wins on the amount rather than adding a row. Deployments sharing a public name inside a team therefore no longer need collapsing into a single charge: each keys its own row, and the read path merges them back under the shared name. The read path that surfaces the amount lands in a follow-up PR. The prune is the one destructive step, so it only runs when the pod took the cross-pod lock. The upserts stay unguarded, since they are idempotent and no lock problem may cost a day, but the delete compares a cutoff and an updated_at stamped on different hosts, and a live rig showed a pod whose clock ran ten minutes ahead sweeping the charge a concurrent pod had just written, leaving the day at zero. Its cutoff also allows PTU_PRUNE_SKEW_GRACE_SECONDS of slack, which separates the two populations without requiring clocks to agree: a stale row is hours old and a concurrently written one is seconds old. The catch-up deletes nothing. Removing a deployment or narrowing its window stops it accruing new charges and leaves the days it was already billed for standing, since those days were incurred and a usage view has to keep reporting them. A deployment carrying no ptu_effective_from is skipped rather than treated as open ended. The endpoints require a start, and substituting the cap floor for a missing one meant a windowless deployment accrued the whole ninety day window on its first run, billing days it did not exist while the result still reported a single row written. |
||
|
Alex Shtof
|
280c95ccb0
|
fix(bedrock): enable native structured output for GLM 5 and DeepSeek V3.2 (#35669)
* fix(bedrock): enable native structured output for GLM 5 and DeepSeek V3.2 * ci: empty commit --------- Co-authored-by: Alexander Shtoff <alexander.shtoff@tii.ae> |
||
|
yucheng-berri
|
e5386c10a7
|
feat(ptu): configure provisioned-throughput flat cost on a model deployment (#35341)
Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to
ModelInfo so a model deployment can carry the inputs for provisioned-throughput
flat-cost attribution. ModelInfo validates per-field bounds (positive count,
non-negative rate, effective_to after effective_from); model/new and
model/{id}/update enforce the cross-field invariant (count and rate set together,
team_id required) on the effective model_info so partial updates validate the
merged result, and v1/model/info returns the fields.
LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a
sentinel api_key constant; the daily rollup that writes them lands in a follow-up
PR. Adding the optional model_info fields is backward compatible; models without
them are unaffected.
ptu_effective_from is required alongside the count and rate rather than optional. Flat
cost accrues from that instant, so an absent start has to be inferred, and inferring it
let a deployment configured today be billed for days it did not exist. Both PTU validators
also run over the merged view before any write on the update path, beside the premium check the create path
already runs there: the team ACL update below autocommits, so a validator raising further
down left the team mutated and the deployment row never written.
The update path validates the model_info a patch would store rather than the patch
alone. An invariant holds over the deployment as it will exist, not over whichever
subset of fields a caller sent, and validating the patch rejected raising the rate on
an already configured model because that patch carries no start of its own.
|
||
|
Mateo Wang
|
60459c60b4
|
Merge pull request #36181 from BerriAI/litellm_fix_batch_group_fallback
fix(router): keep batch fallbacks inside the model group that owns the file |
||
|
mateo
|
b5b663fd8b |
fix(xai): bill the above-200k tier at exactly 200k prompt tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Devin AI
|
9456564b3c |
fix(model_prices): refresh deprecation dates and xAI pricing from provider docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration
|
1b2430b8b6 | fix(bedrock): report uploaded size in the FileObject returned by managed batch uploads | ||
|
Mateo Wang
|
f6b9518ddb
|
Merge pull request #36092 from BerriAI/devin_ai_fix_openai_passthrough_files_route_36086
Some checks are pending
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Proxy Legacy Tests / auth-and-jwt (push) Waiting to run
Unit Tests: Proxy Legacy Tests / key-generation (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-config (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-response-and-misc (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server-extras (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-token-counter (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-user-auth-and-spend (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-utils (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
fix(proxy): stop /{provider}/v1/files from capturing /openai_passthrough
|
||
|
mateo-berri
|
bd719c21dc | test(passthrough): annotate route match scope as Final | ||
|
Yuneng Jiang
|
00600c1af7
|
test(proxy): guard management_v1 against fastapi names removed in supported releases | ||
|
yuneng-jiang
|
ecba48dd7c
|
Merge pull request #35773 from HuanQian571/litellm_fix_management_v1_get_flat_params
fix(proxy): restore management_v1 query-param validation under fastapi>=0.140.7 |
||
|
Devin AI
|
07f14617f6 |
feat(search): add Amazon Bedrock AgentCore web search provider
Adds 'agentcore' as a search provider backed by an AgentCore Gateway MCP web-search target, usable from litellm.search()/`/search` and as a websearch_interception backend. Supports SigV4 (AWS_IAM gateways) and bearer tokens (CUSTOM_JWT gateways) via a new BaseSearchConfig.sign_request hook. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
933c18b21c |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_batch_group_fallback
# Conflicts: # litellm/router_utils/fallback_event_handlers.py # tests/test_litellm/router_utils/test_fallback_event_handlers.py |
||
|
mateo-berri
|
85c1b5d04a | Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_fix_openai_passthrough_files_route_36086 | ||
|
Mateo Wang
|
6eaeab8eae
|
Merge pull request #36273 from BerriAI/litellm_dbless_hook_registration
fix(proxy): skip prisma-dependent hooks when no database is attached |
||
|
Yang Yang
|
749a8b0701 |
fix(xai): keep chat Usage through Responses completions bridge
xAI already converts Responses usage to chat Usage so web_search_calls survive cost tracking. The chat completions bridge then re-ran the Responses usage transform and crashed on missing input_tokens. Pass through already-chat Usage and chat-shaped dumps instead |
||
|
Yang Yang
|
a9277b4b6e | style: black format xAI cost calculator tests | ||
|
Yang Yang
|
74100989a2 |
revert: remove xAI-specific web search gate from shared cost tracking
Gate web search like OpenAI (output/annotations/web_search_requests). xAI uses server_side_tool_usage_details only for per-call cost math, with web_search_requests mirrored in llms/xai for existing gate compatibility. |
||
|
Yang Yang
|
8ad0a57387 | style: drop unused pytest import in xAI responses tests |