The model resync now mutates the router under MODEL_RECONCILE_LOCK, and the
agent resync shares the new AGENT_RECONCILE_LOCK with the periodic agent
reload, so a reconcile built from a pre-write DB snapshot can no longer evict
or duplicate what a read-through just registered. Every resync checks
should_load_db_object for its object type, keeping read-through consistent
with what the replica is configured to load, and the a2a raise sites tag
ProxyModelNotFoundError as non-retryable so an agent miss no longer burns the
model resync budget.
Probe the column before the scheduler registers CheckBatchCost, closing the window where a retrieve that decided the poller was inactive billed a batch the first poll cycle then billed again. Also drop narration docstrings and section banners from the new tests.
* fix(proxy): expand config-defined model access groups when resolving team models for /v2/model/info
Teams whose only model grant is a config-defined access group (a model_info.access_groups
name listed in team.models) got an empty /v2/model/info?include_team_models=true result.
_add_team_models_to_all_models passed each team.models entry straight to
llm_router.get_model_list(model_name=...), which never matches an access-group name, so
the group's member deployments were dropped. Runtime auth and /v1/models were unaffected
because they expand team.models through get_team_models first.
Resolve team.models through the same get_team_models resolver before iterating, reusing the
exact path runtime auth and /v1/models trust so the two can't drift again. The get_model_names
and get_model_access_groups accessors are hoisted above the team loop so they run once.
* fix(proxy): keep a literal model whose name collides with an access-group name in listings
A grant string that names both a deployed model and a config access group grants
BOTH at runtime (_check_model_access_helper unions them), but the listing resolver
dropped the literal and substituted the group members, hiding a callable model from
/v1/models and /v2/model/info. Keep the literal when it is also a deployed model so
listings match runtime access exactly. Pure-group names (no collision) are still
replaced by their members. Also rewrites _get_models_from_access_groups to build
its result without mutating the input list.
Addresses the Greptile P1 on this PR.
* fix(proxy): type proxy_model_list param as Sequence to satisfy LIT001 budget
* fix(alerting): dedupe scheduled Slack spend reports across pods
Every pod ran its own weekly/monthly spend report jobs, prometheus
fallback stats cron, and daily report loop, so deployments with
multiple replicas or uvicorn workers received one copy per pod.
Gate each scheduled send behind the shared PodLockManager redis lock.
The lock is never released: its TTL (the full reporting window for the
weekly interval job, whose per-pod anchors drift by boot time and
jitter) doubles as a sent-this-window marker. acquire_lock returning
None (no redis wired) proceeds, preserving single-pod behavior.
Also generalize the pod lock could-not-acquire log line, which claimed
to be about spend tracking for every consumer.
Fixes#14809
* fix(alerting): harden spend report locks after adversarial review
Weekly lock TTL gets an hour haircut: with ttl equal to the interval,
the winner re-fires just before its own key expires, reacquires without
a TTL refresh, and the key then lapses in time for a trailing pod to
re-send. Job/lock ids move to litellm/constants.py per convention, and
spend_report_frequency now rejects non-positive day counts, which
previously coerced to an every-second schedule and would now compute a
negative lock TTL that silently never sends.
Adds the missing test coverage the review flagged: startup_event's
pod_lock_manager wiring (identity-asserted), the prometheus closure's
positive path, and the ungated immediate prometheus send pinned to
exactly one await.
* test(alerting): consolidate spend_report_frequency validator coverage
Drops a duplicate non-positive-days test and parametrizes the survivor
over the suffix half of the validator too
* fix(alerting): route the startup prometheus fallback send through the pod lock
Greptile caught that the boot-time send still ran once per pod when
PROMETHEUS_URL is set, the same duplication class this PR removes
* fix(alerting): make report lock acquisition non-reentrant
Greptile caught that a pod booting within an hour of the fallback stats
cron sent twice: the startup send takes the lock, then the cron fire
hits acquire_lock's reacquire branch, which returns True for the
holder. Window-marker gates now pass allow_reentrant=False so a live
lock blocks everyone including its holder; leader-election consumers
keep the reentrant default
* test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer
_initialize_slack_alerting_jobs now reads it for the pod lock manager,
and spec=ProxyLogging blocks instance-only attributes
LITELLM_ENABLE_PTU_COST_ATTRIBUTION, read through get_secret_bool and defaulting to
false, makes the whole PTU flat-cost feature inert unless an operator opts in. The
daily rollup cron is not registered at all, so no sentinel row is ever written;
/model/new and /model/{id}/update reject a request that carries any PTU model_info
field with a 400 naming the fields and the env var rather than dropping them; the
daily activity read path reports zero flat cost; and the model add and edit forms
hide the four PTU inputs.
The read gate lives where flat cost enters SpendMetrics rather than in the aggregated
SQL select. /team/daily/activity, the endpoint the Usage page reads, is served by the
paginated find_many path and never runs that query, so forcing the select to a
constant zero would have left the reporting surface that matters still showing flat
cost.
Sentinel row filtering is deliberately not gated. An operator can enable the flag,
accrue rows under the __ptu_flat_cost__ api_key, then disable it, and those rows stay
in LiteLLM_DailyTeamSpend; gating the filter too would surface the sentinel as a bogus
api_key and mint a provider bucket for its empty provider. Response fields keep their
shape and report 0.0, so typed clients are unaffected, and the migration and the
ModelInfo field declarations are untouched.
The write gate reads the incoming request rather than the merged deployment, so a
model configured during an earlier opt-in stays editable, and the edit form drops the
PTU keys from the payload instead of sending nulls that would clear stored config.
The dashboard reads the flag from a read-only enable_ptu_cost_attribution key on
/get/ui_settings, computed from the environment on every read. It is deliberately not
an allowlisted persisted setting, and PATCH /update/ui_settings rejects it with a 400,
so an admin cannot flip an env-gated feature from the UI.
Two review findings on the gate itself. The PTU clear loop now runs only when the
feature is enabled: the write gate rejects a value but lets an explicit null through,
and a client round-tripping a model_info blob sends the PTU keys as nulls, so a
disabled proxy would have quietly erased a billing configuration set up during an
earlier opt-in. Disabling pauses PTU rather than discarding its setup. And the
dashboard flag is re-read every thirty seconds instead of the hour the other UI settings
use, since those are persisted records while this one tracks the proxy process; a
restart that flips the variable would otherwise leave the model form offering inputs
the backend now rejects. The flag is polled rather than only marked stale, since a form
that stays mounted and focused never refetches on its own.
The read gate checks the row before the flag. It runs once per metric accumulation and a
record fans out across roughly a dozen breakdowns, while the flag reads through the secret
manager uncached, so consulting it for every accumulation put thousands of lookups on a
shared endpoint that made none before. Only a row actually carrying flat cost reaches it.
Add the daily rollup that reads PTU config off model deployments and writes flat
cost to LiteLLM_DailyTeamSpend. For each UTC day a deployment carrying ptu_count
and cost_per_ptu_per_hour accrues ptu_count * cost_per_ptu_per_hour * active_hours,
where active_hours is the overlap between the day and the optional
[ptu_effective_from, ptu_effective_to) window clamped to 24; a window opening at
23:00 charges one hour that day. Rows use a sentinel api_key so they stay
distinguishable from per-request rows and share the existing unique constraint,
and the write is idempotent so re-runs never double count.
The cron is registered at proxy startup and runs at 00:15 UTC. Pricing one day per
fire leaves two ways for a day to end up unpriced and stay that way: a window
backdated at configuration time, which no fire ever revisits, and a fire that is
missed or lands late, which the next one does not replay because the billed day
comes from the wall clock rather than the scheduled time. Both are silent, since
the failure alert only fires for a charge that was attempted. Each scheduled run
therefore follows the day's reconcile with a catch-up pass that prices the
(team, model, date) charges inside every declared window that carry no row yet,
bounded at the earliest ptu_effective_from and floored at
PTU_ROLLUP_MAX_BACKFILL_DAYS. It writes only what is missing: a day already priced
keeps the amount it was billed whatever the config says now, and it runs no prune,
so deciding a row is stale stays the single-day path's job. Zero-cost days write
nothing, which leaves an out-of-window day reconsidered each run rather than
recorded as done. A catch-up pass that fails cannot take the day's own result with
it, and an explicit target_date still means reconcile exactly that day.
The sentinel row keys on the deployment id, with the operator-facing name alongside it
in model_group, which sits outside the table's unique key. The name is what a usage view
displays, but a deployment can be renamed, and two runs holding config views from either
side of a rename then wrote the same day under two different keys, so nothing collided
and both charges survived. A multi-pod rig reproduced that as a permanent double charge
that no later run repaired. Keyed on the id both writes land on one key and the upsert
collapses them; when the rate changed too, last writer wins on the amount rather than
adding a row. Deployments sharing a public name inside a team therefore no longer need
collapsing into a single charge: each keys its own row, and the read path merges them
back under the shared name.
The read path that surfaces the amount lands in a follow-up PR.
The prune is the one destructive step, so it only runs when the pod took the cross-pod
lock. The upserts stay unguarded, since they are idempotent and no lock problem may cost
a day, but the delete compares a cutoff and an updated_at stamped on different hosts, and
a live rig showed a pod whose clock ran ten minutes ahead sweeping the charge a concurrent
pod had just written, leaving the day at zero. Its cutoff also allows
PTU_PRUNE_SKEW_GRACE_SECONDS of slack, which separates the two populations without
requiring clocks to agree: a stale row is hours old and a concurrently written one is
seconds old.
The catch-up deletes nothing. Removing a deployment or narrowing its window stops it
accruing new charges and leaves the days it was already billed for standing, since those
days were incurred and a usage view has to keep reporting them.
A deployment carrying no ptu_effective_from is skipped rather than treated as open ended.
The endpoints require a start, and substituting the cap floor for a missing one meant a
windowless deployment accrued the whole ninety day window on its first run, billing days
it did not exist while the result still reported a single row written.
* feat(proxy): add apply_user_budget_to_team_keys opt-in
PR #32005 made a user's personal max_budget apply to their team-scoped keys
too, and PR #35271 reverted the whole thing (behavior plus the
skip_user_budget_on_team_key opt-out) because that flipped the default for
everyone. This brings the behavior back the other way round: default is
unchanged, and general_settings.apply_user_budget_to_team_keys opts a
deployment into charging the key owner's personal budget on team keys.
The flag reaches all three personal-budget gates so an opted-in deployment
enforces consistently: the read-time check in common_checks, the optimistic
reservation counter in _get_budget_counters, and the _PROXY_MaxBudgetLimiter
pre-call hook. It is also in the /config/list allowed args and, unlike the
reverted flag, in the _update_general_settings propagation allowlist, so the
Admin UI General Settings toggle actually takes effect at runtime; an explicit
YAML value still wins over the DB value on reload.
get_config_list's allowed_args moves to a module-level frozen mapping of
field name to type string, dropping 18 LIT002 violations and rebuilding one
less dict per request.
* style(proxy): drop explanatory comments from the budget flag paths
A price data reload replaced litellm.model_cost wholesale, discarding every
runtime registration: the deployment model_info the Router registers from
model_list, and pricing overrides passed to litellm.register_model. Custom
model groups lost max_input_tokens / max_output_tokens in /model_group/info,
and a deployment whose backend model is in the catalog silently reverted to
upstream values. Runtime registrations are now recorded and replayed on top of
the freshly fetched catalog.
Router._pre_call_checks resolved the per-deployment model name only after the
model-info lookup, so an unregistered model left it unset and the supported
params check ran against the bare model group name, raising "LLM Provider NOT
provided" out of deployment selection. The name is now resolved first, and an
unresolvable provider skips that check rather than failing the request.
Resolves LIT-4675
`_setup_prisma_client` ran `connect()`, then a `SELECT 1` health check, then
armed the DB health watchdog. Any failure fell into one handler that, with
`allow_requests_on_db_unavailable` set, swallowed the error and returned None,
which the caller assigns to the module-level `prisma_client`. A single
transient timeout on that health check therefore discarded a client that had
already connected, for the life of the process, and skipped the watchdog that
exists to reconnect it.
The watchdog now starts before the health check, and a swallowed post-connect
failure returns the connected client instead of None. A client whose
`connect()` failed is still discarded, and startup still hard-fails when
`allow_requests_on_db_unavailable` is not set.
The same check also misreported its own failure. `health_check()` labelled its
error `disconnect()`, a copy-paste from the real `disconnect()` below it, so
grepping the logs for the health check turned up nothing and read as "the check
never ran". Both it and the sibling `connect()` failure reported through
`print_verbose`, which reaches `verbose_proxy_logger.debug` and otherwise prints
only under the deprecated `litellm.set_verbose`, leaving a startup-blocking
database fault invisible at the verbosity operators actually run. Both now log
at warning under their own names. The proxy logger's handler carries the secret
redaction filter, so a connection string in the exception text is redacted
exactly as it was on the old print path.
The proxy serves POST /openai/v1/responses alongside /responses and
/v1/responses, but only the latter two were in API_ROUTE_TO_CALL_TYPES.
UnifiedLLMGuardrails.async_post_call_success_hook resolves the call type
from request_route, so on the alias it resolved to None and returned the
response unscanned; model output reached the client with post-call
guardrails never running. The key and team tool allowlist was unenforced
on the same alias for the same reason.
Register the alias family in API_ROUTE_TO_CALL_TYPES and in
LiteLLMRoutes.openai_routes, mirroring how the /openai/v1/realtime
aliases are registered, and log a warning at the two points where the
unified guardrail skips post-call scanning so a future unmapped route is
visible instead of silent.
The Responses block of API_ROUTE_TO_CALL_TYPES moves from list to tuple
literals because the LIT002 budget rejects net-new mutable-collection
construction; the map is read-only, so it is now typed as a Mapping of
Sequence and the budgets ratchet down accordingly.
* fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db
The model cost map and Anthropic beta headers reload schedules kept their
last-run time in a per-pod module global, so GET /schedule/*/status reported
last_run null after any restart and the Admin UI showed the reload as never
having run. The reload check also only ran from the add_deployment job, which
is registered only when store_model_in_db is true, so config-file deployments
stored a schedule that never fired.
Persist last_run_at and reload_requested_at as dedicated columns on
LiteLLM_Config, owned by the reload job and manual reload endpoints, while the
schedule endpoints own the param_value JSON (interval_hours); no writer can
clobber another's fields. Serve status entirely from the row. Register the
check as its own periodic_reload_job outside the store_model_in_db gate.
Replace the force_reload boolean with a reload_requested_at timestamp each pod
compares against its own in-memory last reload, so a manual reload reaches
every pod exactly once instead of being cleared by the first poller. Run the
blocking fetches via asyncio.to_thread, and stamp last_run_at with update_many
so a schedule cancelled mid-poll is not resurrected.
* fix(proxy): compare reload requests against pod data age seeded at boot
A pod that had never reloaded kept its in-memory clock at None, and with no
interval configured nothing ever set it, so every manual reload request was
ignored by every pod except the one serving the click (Greptile P1 on the
previous commit). Seed the per-pod timestamp at boot as the time its data was
loaded and reload whenever a request or the interval is older than that, which
also removes both None special cases from the due predicate. A schedule whose
row has no last_run_at fires on the next tick so the first run does not wait a
full interval.
* fix(proxy): scope reload persistence to the model cost map and seed the pod clock from the actual load time
Revert the Anthropic beta headers reload path to its previous JSON-flag
implementation so this PR only changes the price data reload; the beta headers
path keeps working exactly as before and can migrate to the shared module in a
follow-up. The unused columns on its config row are inert.
Seed model_cost_map_loaded_at from the timestamp get_model_cost_map records at
the actual import-time fetch instead of ProxyConfig construction time, closing
the startup window where a manual reload request stamped between the fetch and
the constructor compared as older than the pod's data and was skipped
(Greptile P1 on the previous commit).
* refactor(proxy): drop the legacy force_reload backfill from the reload tracking migration
The backfill only carried over a manual reload clicked in the seconds before an
upgrade, and every upgrade restarts the pods, which re-fetch the cost map at
import and so already deliver what that request asked for. Removing it makes
the migration schema-only, so prisma db push and prisma migrate deploy leave
the database in the same state instead of diverging on a data statement that
only one of them runs.
* fix(proxy): stamp reload timestamps at the precision they are stored at
Postgres stores these columns as TIMESTAMP(3) while Python stamps microseconds,
so a pod comparing its in-memory clock against the persisted copy of the same
instant read as newer and skipped the reload request it had just recorded.
Truncate every stamp to milliseconds at the source, and floor the boot seed the
same way, so the in-memory value and its persisted copy compare exactly.
* fix(proxy): identify manual reloads by revision instead of comparing timestamps
Comparing a request timestamp against each pod's data age made correctness depend
on clock resolution: Postgres stores TIMESTAMP(3) while Python stamps microseconds,
and two events inside the same millisecond are indistinguishable no matter how the
comparison is written.
Replace reload_requested_at with a reload_revision counter the manual reload
endpoint increments atomically in the database. Each pod records the revision it
last applied and reloads whenever the row's differs, so a request reaches every pod
exactly once regardless of clock skew or precision, and concurrent requests publish
distinct revisions instead of overwriting one another. A pod adopts the current
revision on its first poll, since data it loaded at boot already satisfies any
earlier request. Interval reloads still key off the pod's own data age, where hour
scale comparisons make precision irrelevant.
* fix(proxy): seed the applied reload revision at startup
A pod adopted whatever revision it found on its first poll, so a manual reload
published while the pod was starting was marked applied without ever being
served and the pod kept the prices it fetched at import. Read the row once at
startup instead, right after that fetch, and treat a missing row as revision 0
* style(tests): revert incidental reformatting of test_proxy_server.py
An earlier ruff format run reflowed the whole file from its 88-column
formatting, adding ~1150 lines of churn unrelated to this PR. Replay only
the real test changes onto the original formatting
* fix(proxy): serve an outstanding reload request on a booting pod
Seeding the applied revision at startup left a window: a manual reload
published after the import-time cost map fetch but before startup read the
row was marked applied without ever being fetched, stranding that pod on
stale prices when no interval was configured. A pod now starts unapplied and
serves any outstanding request on its first poll, which costs one redundant
fetch per boot and removes the window along with the seeding step
* fix(proxy): accept a reload interval still encoded as JSON text
param_value is written with safe_dumps, and a raw row read can return it
decoded or as a string depending on the driver. Strict validation rejected
the string, so the schedule read as disabled and an admin's configured
reloads silently stopped. Mirrors the guard ConfigRepository.get_param
already carries for the same column
* fix(proxy): cancel a reload schedule without resetting the revision
* fix(proxy): null the interval in JSON so cancelling keeps the revision
prisma rejects a null literal for a Json? column, so update_many writes an
interval-less object instead. The fake config table now rejects the same input
the database does, which is what the live run caught and the mock did not.
Also records the run before adopting the revision, so a failed status write
leaves the request unserved for the next poll rather than reporting a run that
never landed.
* fix(ui): match the CI-generated user_role union order in schema.d.ts
* fix(proxy): retry model cost map fetch with Retry-After-aware backoff and stop downgrading to the packaged backup on reload failure
A 429 or transient network error during a manual or scheduled model cost map
reload used to silently replace litellm.model_cost with the stale backup JSON
bundled in the installed wheel, stamp the reload as successful, and clear the
force_reload flag, so a fleet could serve months-old pricing until the next
interval. Runtime reloads now go through refetch_model_cost_map, which retries
429/5xx/transport errors up to 3 times honoring Retry-After (capped at 30s,
exponential backoff with jitter otherwise) and returns a failure value instead
of the backup when the fetch or integrity validation fails. On failure the pod
keeps its currently loaded map, the periodic job leaves last_run and
force_reload untouched so it retries on the next config poll, and the manual
endpoint returns 502 with the reason instead of reporting a fake success.
Startup behavior is unchanged: boot still falls back to the packaged backup
since there is no previously loaded map to keep.
* fix(proxy): use shared async httpx client for cost map reload and make retry tests CI-env-proof
The reload fetch now goes through get_async_httpx_client with a dedicated
httpxSpecialProvider.ModelCostMap pool instead of constructing a raw
httpx.AsyncClient, so it inherits deployment-level TLS and transport settings
and passes the ensure_async_clients gate. Tests inject a MockTransport-backed
client through the same seam. An autouse fixture clears
LITELLM_LOCAL_MODEL_COST_MAP, which CI exports and which short-circuited the
retry tests; the two TestPriceDataReloadAPI tests and the config sync pubsub
reload test that still patched get_model_cost_map now patch
refetch_model_cost_map instead.
* fix(proxy): stop model writes 500ing on another pod's delete
A model write judges the reload it triggers by diffing this pod's router before and
after, and reports anything that stopped serving as damage. On a pod that has not yet
polled a delete another pod made, the snapshot still lists that model; the reload then
evicts it because the db no longer has it, and the guard reads its own correct
reconcile as degradation. The row is written and served, but the caller gets a 500.
Since propagation between pods is a 30s db poll, any delete followed by a create
inside that window can land on a pod that has not caught up, so a delete-then-create
pair returns 500 whenever the two requests hit different pods.
_delete_deployment already computes exactly the set that settles it: the ids the db
and config still want. Thread it up through _update_llm_router, add_deployment and
clear_cache to the verdict, and intersect the drop set with it so an id the db no
longer has stops counting as collateral. Where no reconcile ran the set is None and
every drop is still reported, so a genuinely broken reload is caught as before.
_delete_deployment now returns that set instead of a delete count; the count had no
callers in the proxy, and the tests asserting it already assert the eviction calls.
* test(proxy): fold reload-verdict test commentary into docstrings and assertions
Greptile flagged the inline comments against the repo's no-new-comments rule. The
case-by-case context moves into the test docstring, and the two return-contract
assertions carry their reasoning as failure messages instead.
* test: fix clear_cache mock return type in model block/unblock tests
Handling of the client-supplied mock testing params was split across three
places with different behavior for each. Three were dropped from every proxy
request, two reached the router untouched, and a request that asked for a
synthetic failure came back as an ordinary success with nothing to indicate
that no failure had been injected
Put all six behind one opt-in, general_settings.
dangerously_allow_mock_testing_request_params, and reject rather than drop
when it is unset, so a fallback drill cannot report a pass for a test that
never ran. The rejection names the params it saw and the config key to set,
which is also the answer for anyone following the older docs
The flag is config-file only. It is deliberately absent from
ConfigGeneralSettings, and that absence is what makes /config/update drop it
on parse and /config/field/update reject it; the tests pin both so the field
cannot be added back for tidiness without the reason surfacing. Enabling it
logs a startup warning naming every param it unlocks
BREAKING CHANGE: mock_timeout and mock_testing_rate_limit_error now require
general_settings.dangerously_allow_mock_testing_request_params to be set in
config.yaml. Previously they were accepted unconditionally
Streamed chat completions that did not opt into stream_options.include_usage
were logged with tiktoken estimates over the visible text, so hidden
reasoning tokens (billed as output by OpenAI-compatible providers) were
never counted and SpendLogs could undercount output tokens by 90%+ on
reasoning models. The proxy now injects include_usage upstream for
/v1/chat/completions streams by default and strips the injection artifacts
(the final usage chunk and the empty prompt-filter chunk) from the
client-facing SSE stream, so accounting uses provider-billed usage while
the client-visible stream stays byte-identical to today.
always_include_stream_usage keeps its existing semantics: true forwards
the usage chunk to clients as before, and an explicit false now acts as a
kill switch that disables the upstream injection for OpenAI-compatible
backends that reject stream_options.
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
Convert pydantic table-model construction from Cls(**row.model_dump())
kwargs-unpacking to Cls.model_validate(...) across the management endpoint
hotspot files (team, key, internal user, scim, model management, spend
tracking, auth checks, proxy_server). Unpacking an untyped dict reports one
Any-typed argument per matched model field, so each converted site clears
10-35 diagnostics while running the exact same pydantic validation.
Conversions were limited to models verified to use pydantic's default
__init__; UserAPIKeyAuth and LiteLLM_VerificationTokenView keep their custom
kwargs-rewriting __init__ and are untouched. Two locally-verified helper
params move from Any to object.
Whole-tree basedpyright, measured against the branch point in the same
environment: reportAny 24,431 -> 22,741 (-1,690), reportArgumentType
2,189 -> 2,136 (-53), reportUnknownArgumentType 34,370 -> 34,067 (-303),
reportExplicitAny 7,285 -> 7,283 (-2); total 154,882 -> 152,834 (-2,048)
with no rule increasing anywhere and no per-file increases. No casts, no
suppressions, no behavior changes. Budgets ratcheted: basedpyright -2,048
across 4 rules, ruff ANN401 -2.
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
25 test functions across three files pass unchanged when every function
they execute is mutated; the owning file killed zero of their scored
mutants. Four zero-kill tests tied to the fix in #31288 are kept for
rewrite instead of removal.
Admits a keyless SSO user (no virtual key) at the aggregate /mcp endpoint from a gateway DCR
session bearer, resolving team/org/SCIM/budget authorization fresh on every call.
- Aggregate DCR front door: stateless /register (sealed llm_dcrc_ client ids), SSO-backed
/authorize + /authorize/complete, and /token minting identity-only session tokens with PKCE,
single-use codes/flows, and rotating refresh tokens.
- Admission: a session-shaped Authorization at the aggregate scope opens via _admit_gateway_session,
reloads the live user, and runs the centralized policy gate; failures return the RFC 9728
invalid_token challenge. Gated on the un-forgeable, server-only mcp_admitted_user_subject marker,
so virtual-key and JWT auth are unchanged.
- Authorization model: an admitted subject is resolved as one plain UserAPIKeyAuth per grant source
(its own grants, plus each team it is a live roster member of), each answered by the SAME resolver
virtual keys use, then unioned. That branch is the FIRST statement of BOTH public resolvers, so no
single-credential prelude runs for it and a fault in a lookup it never uses cannot deny its grants. A source team counts only while it is a live grantor: roster membership, not
blocked, and neither the team nor its owning org over budget (enforced through the SAME
_team_max_budget_check / _organization_max_budget_check owners common_checks uses for keys).
Each team source carries that team's own org, so the existing org
ceiling caps it; for a keyless source the org list only ever intersects (a ceiling must not become
a grant) and an unresolvable ceiling denies rather than silently uncapping, on both the server and
tool axes. _roster_team_object is the single owner of "which teams count": a team whose roster no
longer lists the user neither grants servers nor throttles, in one place.
- Rate limits: the subject is bounded by its user rpm/tpm AND by the per-server mcp_rpm_limit of
the team a call is ATTRIBUTED to — the same single source billing charges, from the same owner. A key charges its one pinned team's bucket; a keyless
subject has no team_id, so admission stamps each granting team's limit map onto the auth
(server-only field, stripped from validated input like the marker) and the limiter emits that
team's mcp_per_team descriptor. Charging every granting team instead would let one cross-team user
drain several teams' SHARED buckets on a single call and block their other members; and a server
the user's OWN grant reaches charges no team bucket at all, because no team provided it. Per-KEY
MCP limits do not apply because there is no key.
- Wrapper channels: the manager-level union treats the admitted subject by the same grant model.
The admin-role short-circuit and the absolute no_mcp_servers early-return are key-credential
rules and never apply to it (a session bearer is a third-party client credential, not the
dashboard, and the subject's opt-out silences only its own source). Operator-open channels
(allow_all_keys, the user's own BYOM submissions) are owned by one operator_open_server_ids
helper that BOTH the server union and the admitted tool resolution consult (suppress-BYOM-when-
explicitly-scoped is a key-credential rule and never applies to the subject, whose user row
carries the DB-default empty mcp_servers), so an open-channel
server is default-open for tools instead of listable but uninvokable.
- Redirect URIs: one owner, validate_redirect_uri_shape, decides redirect-URI hygiene (bad scheme,
fragment, missing host, userinfo, backslash host) and resolves allowlisted native callbacks, shared
by DCR registration and the OAuth endpoints. Registration keeps a deliberately wider trust policy
than validate_trusted_redirect_uri: public dynamic registration accepts any https client, and its
controls are mandatory S256 PKCE plus the consent screen.
- Egress leak-defense: a gateway admission credential (session bearer / bridge envelope) is scrubbed
from EVERY egress header context, anchored to the credential shape, so it can never be forwarded
upstream and replayed.
- Single-use guard: auth-code, refresh and connect-flow claims resolve the proxy's cross-worker redis
cache themselves rather than trusting the cache passed in, and fail CLOSED on a Redis fault instead
of falling back to a per-worker count that a captured id could replay through another worker.
- Sign-in return_to: one shared, never-raising helper persists a safe return_to for every sign-in
branch (SSO/Okta/generic and username/password), and every branch RESUMES through the same
_sso_return_to_redirect the SSO callback uses, so however a deployment signs in the stored value
is honored identically (same-origin path directly; control_plane_url via the one-time login-code
handoff). A stale cookie is ignored rather than failing a completed sign-in.
- Budgets, both halves: ENFORCEMENT (an already over-budget team or its owning org stops being a
grantor, in the source gate) and ACCOUNTING (a team-derived tool call is billed to the granting
team and ITS org, so that budget accumulates and the right organization is charged). A server the
user's own grant reaches bills the user; when several teams grant one server the pick is the
lowest team_id, stable and auditable. Billing rides a COPY, so authorization still sees the full
union, and it is inert when the target server cannot be resolved from the tool name.
Deferred (tracked): client-selected server scoping of the session token (LIT-4680).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The SSO and Email Server settings pages read only stored config, so a gateway
configured entirely through environment variables rendered every field blank
even though both features were live. Rather than add per-endpoint env fallback,
resolve each setting through one typed config object.
A FieldDescriptor names, for one setting, where it lives in the stored row
(db_key), which process env var carries it (env_var), whether it is a secret,
and its effective default. A pure resolve_fields reconciles a descriptor table
against the stored row and the process environment with a fixed precedence and
reports per-field provenance (db, env, default, or unset). The SSO descriptor
table single-sources the field-to-env mapping that the read and write paths
previously duplicated, so they can no longer drift.
get_sso_settings and the /get/config/callbacks alerting block read through the
resolver instead of their own inline fallbacks. get_sso_settings no longer
decrypts stored values into os.environ; decryption happens once inside the
resolver via the pure helper, so a GET stops mutating the process environment.
The SSO response carries provenance so the UI can distinguish an env-sourced
value from a stored one, and secrets are masked at the endpoint (the resolver
returns them unmasked so the login path could consume them). os.environ remains
the runtime carrier; the SSO login and mail-send paths are unchanged.
The settings pages also submit only fields an admin actually edited, so a
rendered mask or env-sourced value is never written back over a working
secret, and generic_scope is a real SSO form field. Omitting a field from
/update/sso_settings clears it, which provider switching relies on; the deeper
write-path concern that behaviour points at is tracked in LIT-4498.
Every dashboard login mints a 24h session key whose max_budget comes from
litellm.max_ui_session_budget, and all dashboard LLM traffic (playground,
auto router per-tier Test Connection probes) spends against and is gated
by that one key. The $0.25 default locked sessions out mid-testing with
"Budget has been exceeded ... Max budget: 0.25" and the setting appeared
in no docs, no UI, and no error text, so it read as a hardcoded cap.
Raise the default to $1. Give the setting an explicit typed arm in the
config loader (float coercion for env-var strings, null disables the
cap). Surface it on the Admin UI General settings tab through the
existing litellm_settings bridge as a new Dollar field type (positive
USD, unbounded above; the existing Float type is validated to (0, 1] for
fractions), with a spec-level default so clearing the field restores $1
instead of silently removing the cap, and enroll it in
LITELLM_SETTINGS_SAFE_DB_OVERRIDES so UI edits propagate to peer workers.
The UI theme and logging-callback read endpoints reported only stored
config while the features resolve their values from the process
environment, so a gateway configured purely through env vars showed
blank settings pages even though branding rendered and callbacks fired.
/get/ui_theme_settings read only litellm_settings.ui_theme_config;
logo_url and favicon_url now fall back to UI_LOGO_PATH and
LITELLM_FAVICON_URL when the stored config leaves them blank.
process_callback (the logging-callbacks block of /get/config/callbacks)
reported every callback env var as unset unless it lived in the config
environment_variables overlay; it now falls back to os.getenv, matching
the slack block. Secret values stay redacted for non-admins via the
existing callback role gate.
Stored values keep winning over the environment, so the UI-driven flow
is unchanged.
Resolves LIT-4667
The add_deployment and get_credentials background jobs that keep a multi-pod
deployment in sync with config-in-DB objects (models, credentials, guardrails,
general settings, etc.) polled the database on a hardcoded 30s interval, with
no way to trade convergence latency against DB load.
Expose it as the general_setting proxy_config_reload_interval_seconds (env
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS parsed via get_env_int, default 30),
threaded like the existing proxy_batch_polling_interval knob, and surface it on
the admin general-settings page so it is reachable from the dashboard and
persists to the DB for all pods. Non-positive values are rejected at the UI
(gt=0) and fall back to 30s with a warning on the env/config/DB paths.
enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are set as live
litellm attributes on the worker that handles the UI save, exactly like
budget_exceeded_throttle_percentage, but they were missing from
LITELLM_SETTINGS_SAFE_DB_OVERRIDES, so a peer worker's config reload merged the DB
value without applying it to the live attribute and stayed stale.
Add both to the allowlist so they behave like the sibling field, and add
test_general_settings_ui_fields_are_db_overridable so the UI registry and the
override allowlist cannot drift again (the exact omission that caused this), plus
a regression test that the flag flips on a simulated peer-worker reload.
Rather than mixing the flag and its ttl into the generic General settings table
(which also surfaced the confusing Not Set / In Config / In DB provenance badges),
give prompt caching a dedicated tab with a purpose-built toggle and ttl dropdown.
Each registry field gains an optional tab, surfaced as ConfigList.field_tab, so
the General tab renders the ungrouped fields and the caching fields render on
their own tab. The update, persist and reset endpoints are unchanged.
Register enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl on the
General Settings table so caching can be turned on without hand-writing config.
The registry could not express either field: validation was hardcoded to a float in
(0, 1], reset set every field to None (not a bool for a boolean flag), and the listing
reported any non-None value as 'In Config', which a False default would always trip.
Validation now dispatches on the declared type and reset restores each field's own
default. ConfigList carries field_options so the table can render a Select for enums
instead of no editor at all.
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
* fix(proxy): coerce default_internal_user_params.max_budget to float on config load
* fix(proxy): log coerced default_internal_user_params and cover absent max_budget in tests
* fix: enforce user budget on team keys
User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.
Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: update test to expect user budget enforcement on team keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI
Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert budget_exceeded ProxyException in personal budget test
Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style
* fix: revert to bool | None to stay under UP045 strict budget
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
* feat(proxy): configure the coordination redis independently of the response cache
Adds general_settings.coordination_redis, an explicit block for the Redis
the proxy uses for cross-pod rate limits, parallel-request limits, spend
tracking, the pod lock manager, and shared health checks. Resolution order
is the explicit block, then a plain-Redis response-cache backend, then the
REDIS_* environment. Cluster and sentinel targets are supported, and a
cluster target now builds a RedisClusterCache so cluster-aware consumers
take the cluster path.
Admins can configure it from the Caching page of the dashboard via
/coordination_redis/settings, which reports which source is in effect,
redacts credentials on read, and offers a connection test. Settings saved
there are read back at startup so they take effect on restart.
Also fixes redis client construction so an explicitly configured host
outranks REDIS_URL in the environment. Previously the url branch stripped
the caller's host and port, so an explicit block, or a connection test
typed into the dashboard, silently targeted whatever REDIS_URL named
* fix(ui): move coordination_redis_settings into renamed _components directory
---------
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
Selecting a semantic (or any non-Redis-KV) response cache left
redis_usage_cache unset, silently downgrading cross-pod rate limits,
parallel-request limits, spend coordination, and the pod lock manager
to per-pod in-memory state. Fall back to a standalone RedisCache built
from REDIS_* environment variables, mirroring the existing
use_redis_transaction_buffer escape hatch, which now shares the same
helper.
Resolves LIT-3861
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a
globally configured percentage of its TPM/RPM instead of being blocked entirely.
A new litellm_settings global, budget_exceeded_throttle_percentage, sets the
fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in
key metadata via the existing management-endpoint metadata routing) opts the key
in. When both are set and the key is over budget, the budget check records the
percentage on a request-scoped budget_throttle_pct instead of raising, and the
rate limiter scales the key's configured TPM/RPM by it. Keys without the flag
keep hard-blocking; team/user/org budgets are unaffected.
The throttle is recomputed from the key's original limits on every request and
the decision is cleared before the auth object is cached, so it never compounds
across requests. Both the budget read-time check and the budget reservation path
honor the opt-in, and both the v3 and legacy rate limiters apply the scaling.
Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an
admin-imposed hard budget block into a soft throttle that keeps spending past
max_budget, so a non-admin must not be able to self-opt-in and bypass their own
spend cap. Both /key/generate and /key/update reject a non-admin setting it to
true (update only gates the transition to enabled, so a non-admin can still edit
other fields and turn the flag off). This matches the feature being wholly
proxy-admin operated: the global percentage is admin-only too.
A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays
hard-blocked rather than serving unlimited requests past its budget (fail-safe).
The global budget_exceeded_throttle_percentage is configurable from the admin UI
(Settings -> General Settings), persisted through litellm_settings so it survives
a restart, not only from config.yaml.
Resolves LIT-3894. Scope for LIT-3893.
* fix(proxy): wire general_settings SSRF allowlist to litellm globals
general_settings.user_url_allowed_hosts was documented in SSRF errors but
never applied at startup, so internal MCP/OpenAPI URLs stayed blocked.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): regenerate dashboard types and satisfy ruff UP006 budget
Use list[str] in ConfigGeneralSettings and run gen:api so schema.d.ts
matches the new SSRF general_settings fields.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: normalize ssrf general settings
* fix: clear ssrf allowlists from null settings
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The handler returned decrypted callback environment values and alerting
routing values verbatim to callers who were not full PROXY_ADMIN. Gate
those on full-admin role, matching the posture used on the sibling
config-inspection endpoints. Non-sensitive routing fields (host / base
URL / port style values) stay visible so the UI can still label which
integration is wired up. Full PROXY_ADMIN sees everything unchanged so
the edit form round-trips on save.
Resolves LIT-4115.
MCP servers created through the UI are persisted to the database independent of
store_model_in_db, but the in-memory registry that GET /v1/mcp/server reads was
hydrated from the database only through add_deployment, which runs solely when
store_model_in_db is True. On a DB-backed single-instance proxy with
store_model_in_db unset the registry started empty after a restart, so the MCP
Servers page showed nothing until an add or edit triggered a reload.
Hydrate the registry from the database on startup regardless of store_model_in_db
via a new ProxyConfig.init_mcp_servers_from_db, honoring supported_db_objects.
* feat(proxy): audit remaining system-wide settings updates
Extends the audit logging framework introduced in the parent PR to the
rest of the LiteLLM_Config writers and the two adjacent settings tables:
/config/update (general, environment_variables, litellm_settings,
router_settings sections), /config/field/update, /config/field/delete,
/config/callback/delete, /update/default_team_settings,
/update/mcp_semantic_filter_settings, /add/allowed_ip,
/delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings,
/update/ui_settings.
Each writer records the actor, action, the affected config section, and
a redacted before/after snapshot. SSO and UI settings rows use their own
table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback
and /update/sso_settings audits fire BEFORE the proxy reload and the env
cleanup step respectively, so a failure in either leaves the audit row
intact.
The audit-actor parameter on _update_litellm_setting is now required
rather than optional; the chokepoint covers default_team and
mcp_semantic_filter for free, and a future caller that forgets the
actor fails loudly instead of silently skipping the audit. The two
direct-calling tests pass a dummy actor.
The environment_variables section redacts every value rather than
relying on key-name matching, because it carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
* fix(proxy): capture redacted SSO before-snapshot in audit log
Greptile review of #31754 flagged update_sso_settings as the one endpoint
where before_value is permanently None, so the LiteLLM_SSOConfig audit
trail has no pre-change state. An auditor reviewing a secret-rotation
event could see what the SSO settings were changed to but not what they
were before.
Read the existing SSO row before the upsert, decrypt it via
proxy_config._decrypt_db_variables, and pass it as before_value.
create_config_audit_log's secret-name redaction then masks the
*_client_secret fields, so neither the old nor the new plaintext secret
lands in the audit row.
Add a regression test asserting the before-snapshot reflects the
pre-change values for non-secret fields (google_client_id) and is
redacted for secret fields (google_client_secret). Mutation-checked
against reverting to before_value=None.
The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique
since the endpoint reads it; the read returns None for tests that do not
care about the before-state.
* fix: remove committed zero init migration
* refactor(proxy): audit config writes via asyncio.create_task everywhere
PR A's chokepoint audit call was refactored from a blocking await to
asyncio.create_task so that a post-save audit-log failure could not
surface as a 500 to the caller. The 12 other audit call sites added in
this PR were still using await, reintroducing the exact 500-after-commit
exposure at every sibling endpoint. Wrap them all in asyncio.create_task
to match the model_management_endpoints / key_management_endpoints /
hooks / config_override_endpoints / team_callback_endpoints /
cache_settings_endpoints house pattern, so the codebase tells one story.
The two direct-invocation tests (test_update_config_general_settings and
test_delete_config_general_settings, which call the handler in-process
rather than via TestClient) yield with `await asyncio.sleep(0)` after the
handler returns so the scheduled audit task runs before the assertion.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(proxy): audit default user settings updates
Adds audit logging for the customer-impacting path: PATCH
/update/internal_user_settings, which is what the admin dashboard hits
when an admin changes Default User Settings and which today leaves no
record of who changed what.
Introduces the small framework that future system-wide settings audits
will share: a CONFIG_TABLE_NAME enum value, a create_config_audit_log
helper that reuses the existing create_object_audit_log path (so the
enterprise gate and store_audit_logs flag still apply), and a
_dump_redacted_config helper that strips secret leaves before the row is
written using the same matcher /config/field/info applies for non-admins.
The helper handles environment_variables as a special case where every
value is redacted, since that section carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
Only update_internal_user_settings is wired up in this change. Coverage
for the other LiteLLM_Config writers (/config/update sections,
/config/field/update, /config/field/delete, /config/callback/delete,
default_team_settings, mcp_semantic_filter, allowed_ip, sso_settings,
ui_theme, ui_settings) is intentionally a follow-up so each can be
verified live against the credential-bearing fields it actually carries.
The audit-actor parameter on _update_litellm_setting is optional today so
non-audited callers keep working unchanged; the follow-up will make it
required once every caller is wired up.
* fix(proxy): make audit-log call non-blocking and serializer defensive
Greptile review of #31753 surfaced three robustness issues with the
audit-log call path. The settings change always commits; these fixes
prevent post-commit audit failures from surfacing as 500 responses.
Switch the audit-log call in _update_litellm_setting from a blocking
await to asyncio.create_task, matching the create_object_audit_log
pattern every other call site uses (model_management_endpoints etc.).
A transient prisma blip or a JSON serialization error in the audit row
no longer turns a successful save_config into a 500 the caller sees.
Add default=str to both json.dumps calls in _dump_redacted_config so a
YAML-loaded value with a non-JSON-native leaf (datetime, custom object)
serializes cleanly. The sibling audit-log serializers in
team_endpoints.py already pass default=str for the same reason.
Tighten the redact_all_values branch to redact wholesale for non-dict
inputs rather than silently falling through to the key-name matcher;
defensive against a future change that stores a section as a list or
scalar.
Each fix has a regression test mutation-checked against reverting the
fix.
* refactor(proxy): drop unreachable non-dict redact_all_values branch
The defensive non-dict fallback in _dump_redacted_config emitted
json.dumps("REDACTED") which, if ever hit, would crash LiteLLM_AuditLogs
construction (mask_api_keys validator calls json.loads on the already-
parsed bare string). Reachability is zero: redact_all_values is True
only for param_name=="environment_variables", which is always a dict.
Delete the dead branch and its test rather than ship provably-wrong
defensive code with a test that green-lights it.
* chore: remove _experimental/out
* fix(ci): recreate _experimental/out before copying UI build output
The build scripts cp the Next.js output into litellm/proxy/_experimental/out,
which was removed from git. cp failed because the target directory no longer
existed; mkdir -p recreates it before the copy.
* fix(proxy): make UI serving resilient to a missing _experimental/out
Removing the committed UI export means the source/test tree no longer
ships litellm/proxy/_experimental/out. Three things assumed it was always
present and broke once it was gone:
- get_favicon hard-coded the built favicon path and 404'd without it; it
now falls back to the bundled swagger/favicon.ico
- the /_next and /ui static mounts raised at construction when the export
was absent, so the whole UI-setup block was swallowed and no mounts
registered; they now use check_dir=False
- _restructure_ui_html_files was a nested function only exposed as a
module attribute when that block happened to succeed; it is now a real
module-level function
test_admin_ui_export_serves_nested_extensionless_routes validated the
committed artifact, whose premise this PR removes; it now drives the same
MCP OAuth callback restructure guarantee through a synthetic export.
* chore(greptile): ignore generated _experimental/out so review fits the file limit
* Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit"
ignorePatterns is applied after Greptile counts the files changed, so it
does not bring the diff under the file limit; the config had no effect.
