* fix(proxy): invalidate cached project object on /project/update and /project/delete
The auth path reads projects cache-first via get_project_object with a 60s
TTL and no freshness check, but no project write endpoint ever evicted the
project_id:{id} cache entry. A project cached before /project/update added a
model allowlist kept an empty models list in cache, so _run_project_checks
skipped can_project_access_model and project-bound keys could call team
models outside the project allowlist until the TTL expired. The same
staleness applied to blocked status and budget fields, and /project/delete
left the deleted project enforceable from cache.
Evict the cache entry after the DB write in update_project and
delete_project via a shared delete_cached_project_object helper, with the
cache key derivation shared with get_project_object.
* fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort
Single-worker eviction leaves every other worker serving its in-memory copy
of the mutated project until the 60s TTL expires, so a project allowlist
change was still bypassable on multi-worker deployments. Add a coordination
Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project
eviction publishes the cache key and a per-worker subscriber deletes the
local in-memory entry, with the next auth read refetching from the DB.
Subscriber starts on any deployment with a coordination Redis and falls back
to the TTL when none is configured.
Also wrap the eviction in a best-effort catch: the DB write has already
committed when eviction runs, so a cache backend error must not turn a
successful update into a 500 or abort the remaining ids in /project/delete.
* fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch
The strict-budget gate flagged the new import block as un-sorted (I001) and
the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the
catch is intentional since a failing stop must not break proxy shutdown, so
it carries a named suppression instead of counting against the budget.
* fix(proxy): persist periodic reload schedule state so status survives restarts and fires without store_model_in_db
The model cost map and Anthropic beta headers reload schedules kept their
last-run time in a per-pod module global, so GET /schedule/*/status reported
last_run null after any restart and the Admin UI showed the reload as never
having run. The reload check also only ran from the add_deployment job, which
is registered only when store_model_in_db is true, so config-file deployments
stored a schedule that never fired.
Persist last_run_at and reload_requested_at as dedicated columns on
LiteLLM_Config, owned by the reload job and manual reload endpoints, while the
schedule endpoints own the param_value JSON (interval_hours); no writer can
clobber another's fields. Serve status entirely from the row. Register the
check as its own periodic_reload_job outside the store_model_in_db gate.
Replace the force_reload boolean with a reload_requested_at timestamp each pod
compares against its own in-memory last reload, so a manual reload reaches
every pod exactly once instead of being cleared by the first poller. Run the
blocking fetches via asyncio.to_thread, and stamp last_run_at with update_many
so a schedule cancelled mid-poll is not resurrected.
* fix(proxy): compare reload requests against pod data age seeded at boot
A pod that had never reloaded kept its in-memory clock at None, and with no
interval configured nothing ever set it, so every manual reload request was
ignored by every pod except the one serving the click (Greptile P1 on the
previous commit). Seed the per-pod timestamp at boot as the time its data was
loaded and reload whenever a request or the interval is older than that, which
also removes both None special cases from the due predicate. A schedule whose
row has no last_run_at fires on the next tick so the first run does not wait a
full interval.
* fix(proxy): scope reload persistence to the model cost map and seed the pod clock from the actual load time
Revert the Anthropic beta headers reload path to its previous JSON-flag
implementation so this PR only changes the price data reload; the beta headers
path keeps working exactly as before and can migrate to the shared module in a
follow-up. The unused columns on its config row are inert.
Seed model_cost_map_loaded_at from the timestamp get_model_cost_map records at
the actual import-time fetch instead of ProxyConfig construction time, closing
the startup window where a manual reload request stamped between the fetch and
the constructor compared as older than the pod's data and was skipped
(Greptile P1 on the previous commit).
* refactor(proxy): drop the legacy force_reload backfill from the reload tracking migration
The backfill only carried over a manual reload clicked in the seconds before an
upgrade, and every upgrade restarts the pods, which re-fetch the cost map at
import and so already deliver what that request asked for. Removing it makes
the migration schema-only, so prisma db push and prisma migrate deploy leave
the database in the same state instead of diverging on a data statement that
only one of them runs.
* fix(proxy): stamp reload timestamps at the precision they are stored at
Postgres stores these columns as TIMESTAMP(3) while Python stamps microseconds,
so a pod comparing its in-memory clock against the persisted copy of the same
instant read as newer and skipped the reload request it had just recorded.
Truncate every stamp to milliseconds at the source, and floor the boot seed the
same way, so the in-memory value and its persisted copy compare exactly.
* fix(proxy): identify manual reloads by revision instead of comparing timestamps
Comparing a request timestamp against each pod's data age made correctness depend
on clock resolution: Postgres stores TIMESTAMP(3) while Python stamps microseconds,
and two events inside the same millisecond are indistinguishable no matter how the
comparison is written.
Replace reload_requested_at with a reload_revision counter the manual reload
endpoint increments atomically in the database. Each pod records the revision it
last applied and reloads whenever the row's differs, so a request reaches every pod
exactly once regardless of clock skew or precision, and concurrent requests publish
distinct revisions instead of overwriting one another. A pod adopts the current
revision on its first poll, since data it loaded at boot already satisfies any
earlier request. Interval reloads still key off the pod's own data age, where hour
scale comparisons make precision irrelevant.
* fix(proxy): seed the applied reload revision at startup
A pod adopted whatever revision it found on its first poll, so a manual reload
published while the pod was starting was marked applied without ever being
served and the pod kept the prices it fetched at import. Read the row once at
startup instead, right after that fetch, and treat a missing row as revision 0
* style(tests): revert incidental reformatting of test_proxy_server.py
An earlier ruff format run reflowed the whole file from its 88-column
formatting, adding ~1150 lines of churn unrelated to this PR. Replay only
the real test changes onto the original formatting
* fix(proxy): serve an outstanding reload request on a booting pod
Seeding the applied revision at startup left a window: a manual reload
published after the import-time cost map fetch but before startup read the
row was marked applied without ever being fetched, stranding that pod on
stale prices when no interval was configured. A pod now starts unapplied and
serves any outstanding request on its first poll, which costs one redundant
fetch per boot and removes the window along with the seeding step
* fix(proxy): accept a reload interval still encoded as JSON text
param_value is written with safe_dumps, and a raw row read can return it
decoded or as a string depending on the driver. Strict validation rejected
the string, so the schedule read as disabled and an admin's configured
reloads silently stopped. Mirrors the guard ConfigRepository.get_param
already carries for the same column
* fix(proxy): cancel a reload schedule without resetting the revision
* fix(proxy): null the interval in JSON so cancelling keeps the revision
prisma rejects a null literal for a Json? column, so update_many writes an
interval-less object instead. The fake config table now rejects the same input
the database does, which is what the live run caught and the mock did not.
Also records the run before adopting the revision, so a failed status write
leaves the request unserved for the next poll rather than reporting a run that
never landed.
* fix(ui): match the CI-generated user_role union order in schema.d.ts
Moves reset_budget_job's hand-rolled private Prisma protocols into
litellm/repositories as shared seams, and replaces its three ad-hoc
db.batch_() write helpers with a composed unit of work that binds typed
per-table write repositories to a single batch, committing on clean exit
and writing nothing when the block raises.
* fix(proxy): retry model cost map fetch with Retry-After-aware backoff and stop downgrading to the packaged backup on reload failure
A 429 or transient network error during a manual or scheduled model cost map
reload used to silently replace litellm.model_cost with the stale backup JSON
bundled in the installed wheel, stamp the reload as successful, and clear the
force_reload flag, so a fleet could serve months-old pricing until the next
interval. Runtime reloads now go through refetch_model_cost_map, which retries
429/5xx/transport errors up to 3 times honoring Retry-After (capped at 30s,
exponential backoff with jitter otherwise) and returns a failure value instead
of the backup when the fetch or integrity validation fails. On failure the pod
keeps its currently loaded map, the periodic job leaves last_run and
force_reload untouched so it retries on the next config poll, and the manual
endpoint returns 502 with the reason instead of reporting a fake success.
Startup behavior is unchanged: boot still falls back to the packaged backup
since there is no previously loaded map to keep.
* fix(proxy): use shared async httpx client for cost map reload and make retry tests CI-env-proof
The reload fetch now goes through get_async_httpx_client with a dedicated
httpxSpecialProvider.ModelCostMap pool instead of constructing a raw
httpx.AsyncClient, so it inherits deployment-level TLS and transport settings
and passes the ensure_async_clients gate. Tests inject a MockTransport-backed
client through the same seam. An autouse fixture clears
LITELLM_LOCAL_MODEL_COST_MAP, which CI exports and which short-circuited the
retry tests; the two TestPriceDataReloadAPI tests and the config sync pubsub
reload test that still patched get_model_cost_map now patch
refetch_model_cost_map instead.
add_deployment already reapplies DB router settings through _update_llm_router,
so gating router_settings out of the pub/sub publish set left the push path
covering less than the resync actually applies
Caps fleet-wide reload rate at one resync per 10s per pod so a burst of
authenticated writes cannot amplify into continuous cross-pod reloads, and
skips publishing config params (environment_variables, router_settings) that
no resync callback applies outside proxy startup
After any management write to a DB-backed config table, publish an
invalidation event on the coordination Redis; every pod runs a
subscriber that debounces, jitters, and triggers an immediate
add_deployment plus get_credentials resync. The interval polls stay
as slow reconciliation fallback and behavior without Redis is
unchanged since publish and subscribe both no-op.
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
get_sanitized_user_information_from_key copied UserAPIKeyAuth.metadata
verbatim into user_api_key_auth_metadata, so the key's callback
configuration - including the integration credentials inside callback_vars -
reached the StandardLoggingPayload every integration receives. The two other
sites that stamp key/team metadata into request metadata did the same.
Sanitize at those sources with strip_callback_config, which drops the
`logging` and `callback_settings` slots and leaves everything else (notably
`priority`, read back by the dynamic rate limiter) untouched. Those slots are
resolved from UserAPIKeyAuth during pre-call setup and never read off the
logged copies, so nothing downstream loses input.
This makes the scrub in scrub_sensitive_keys_in_metadata dead - it only
matched the string "logging" under one of the two field names and never
covered callback_settings - so it is removed.
Separately, LangSmith set the run's `inputs` to the raw StandardLoggingPayload
while redacting only `extra`, so redact_user_api_key_info left every
user_api_key_* field in inputs.metadata. Both now go through one
_redact_metadata helper, which also covers the nested requester_metadata copy.
The UI theme and logging-callback read endpoints reported only stored
config while the features resolve their values from the process
environment, so a gateway configured purely through env vars showed
blank settings pages even though branding rendered and callbacks fired.
/get/ui_theme_settings read only litellm_settings.ui_theme_config;
logo_url and favicon_url now fall back to UI_LOGO_PATH and
LITELLM_FAVICON_URL when the stored config leaves them blank.
process_callback (the logging-callbacks block of /get/config/callbacks)
reported every callback env var as unset unless it lived in the config
environment_variables overlay; it now falls back to os.getenv, matching
the slack block. Secret values stay redacted for non-admins via the
existing callback role gate.
Stored values keep winning over the environment, so the UI-driven flow
is unchanged.
Resolves LIT-4667
Budgets reset at midnight in the configured timezone with no way to control
the time of day, so a drained daily budget surfaces as an overnight incident.
Add a litellm_settings.budget_reset_time option (e.g. "12:00") that shifts
day/week/month resets to a configurable wall-clock time in the existing
timezone, so the end of the budget window lands during business hours.
The reset time is parsed once into an immutable BudgetResetSettings and
injected into the reset job (constructor) and computation, rather than read
from a module-level global at call time. A malformed value fails fast at
startup. Sub-day durations ignore the offset. Unset preserves midnight resets.
Internal users seeded from default_internal_user_params (SSO/JWT first-login
upsert, or /user/new without an explicit budget_reset_at) get budget_duration
set but budget_reset_at = NULL. The ResetBudgetJob user/team queries filter on
{"budget_reset_at": {"lt": now}}, which never matches NULL, so these rows are
never reset: their spend accumulates for the lifetime of the row and silently
exceeds max_budget with no periodic reset.
The budget-table query already handles this by OR-ing in a
{budget_reset_at IS NULL AND budget_duration IS NOT NULL} branch. Apply the
same pattern to the user and team reset queries in PrismaClient.get_data.
Adds a regression test asserting both the user and team reset queries select
NULL-budget_reset_at rows that have a budget_duration.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers
Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update
The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
CacheCodec.serialize dumped cached Pydantic models with model_dump(exclude_none=True), which drops any None-valued key, while deserialize does a strict model_validate. For a model with a required-but-nullable field (Optional[X] with no default), a None value is dropped on write and then fails model_validate on read with "Field required", so the entry can never be read back; that is a permanent cache miss, and in readers that rebuild the model from the raw cached dict an uncaught ValidationError that surfaces to the client as a 401
Removing exclude_none makes serialize and deserialize a lossless pair, so None fields are written as null and survive the round trip. LiteLLM_ManagedVectorStoresTable, the one cached model still carrying required-nullable fields and mis-caching on every read today, also gets the None defaults its peers already have
decrypt_value_helper logged `Unable to decrypt value={value}` at DEBUG, which
printed the raw secret whenever decryption failed (for example after a salt or
master key change). This is the same environment_variables config path the
db-config redaction covers, so a DATABASE_URL connection string could still
leak here when the module regex scrubber is bypassed. Drop the value; the key
already identifies the failing pair.
Regression forces a decrypt failure with the redaction filter disabled and
asserts the raw value never reaches a log record while the key stays visible.
general_settings.user_api_key_cache_ttl was ignored for every management-object
write into user_api_key_cache. The configured value is propagated to the cache's
default_in_memory_ttl at startup, but DualCache only applies that default when no
explicit ttl kwarg is passed, and every management-object writer passed
ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL (60s), which always won. So keys,
teams, users, budgets, object permissions, vector stores, JWT user syncs and MCP
caches all expired after 60s regardless of the setting.
Adds get_management_object_ttl(cache) in user_api_key_cache.py, which returns the
configured default_in_memory_ttl and falls back to the 60s constant only when no
default is set, and routes every management-object writer through it. The helper
takes a DualCache so it works at the many call sites that are typed UserApiKeyCache
but exercised with a bare DualCache.
Also covers the spend-update writeback in update_cache (async_set_cache_pipeline),
which hardcoded ttl=60 on the same key/user/team objects and reset an active key's
cache entry back to 60s on every priced request, so the configured TTL was never
observed for keys receiving traffic.
Resolves LIT-3338
Skip token counting in Router._pre_call_checks when no deployment in the
group declares max_input_tokens, and skip the full-body surrogate-repair
regex in _read_request_body above a configurable size, raising the existing
400 immediately.
Resolves LIT-3541
* feat(proxy): add configurable response headers middleware
Adds a small ASGI middleware that sets standard response headers
(X-Frame-Options, Content-Security-Policy frame-ancestors, X-Content-Type-Options)
on proxy and UI responses. Strict-Transport-Security is optional and gated
behind LITELLM_ENABLE_HSTS for HTTPS deployments. Values use setdefault so a
route that sets its own header is preserved.
* feat(proxy/ui): make login page credentials hint configurable
build_ui_login_form accepts a hide_default_credentials_hint parameter and
google_login reads LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (or general_settings)
so the legacy login page behaves consistently with the new UI. Also collapses
a duplicated branch and removes an unused variable and module-level constant.
* fix(proxy/ui): apply credentials hint flag on /fallback/login
The /fallback/login handler still rendered the default-credentials hint
regardless of LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT. Collapse its duplicate
branch and forward the flag, matching google_login, so all login surfaces
behave consistently. Adds regression tests for /fallback/login and makes the
ui_sso test helper restore os.environ so env vars do not leak across tests.
* fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys (#29590)
* fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys
* test(proxy): regression test that load_config forwards callback_specific_params
* fix(proxy): guard lakera_prompt_injection callback_specific_params against non-dict
Addresses review feedback: forwarding callback_settings as callback_specific_params
(so DatadogCostManagementLogger receives cost_tag_keys) exposed the
lakera_prompt_injection branch, which did lakeraAI_Moderation(**callback_specific_params
["lakera_prompt_injection"]) with no type guard. A config like
`callback_settings: {lakera_prompt_injection: "any-string"}` then hit `**"any-string"`
-> TypeError: argument after ** must be a mapping, not str.
Guard the lakera branch with isinstance(dict), matching the existing presidio and
datadog_cost_management branches (non-dict values fall back to {}). Add a regression
test asserting initialize_callbacks_on_proxy ignores a non-dict value instead of crashing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: inject fake lakera_ai module to avoid importing the real one
CI fix for the lakera regression test: it stubbed litellm.proxy.proxy_server with
a SimpleNamespace and then monkeypatch.setattr'd the real lakera_ai module, which
forces importing it — and lakera_ai does `from litellm.proxy.proxy_server import
LiteLLM_TeamTable`, absent on the stub -> ImportError under proxy-infra tests.
Inject a fake lakera_ai module into sys.modules instead, so the callbacks branch's
`from ...lakera_ai import lakeraAI_Moderation` resolves to the stub without loading
the real module. The guard under test (isinstance(dict) in the lakera branch) is
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(callbacks): guard compression/websearch interceptors against non-dict callback_settings (#30153)
#29590 forwards the full callback_settings dict into initialize_callbacks_on_proxy, which activates the compression_interception and websearch_interception consumers. Their initialize_from_proxy_config read the callback_settings subkey without an isinstance(dict) guard, so a non-dict value such as `compression_interception: true` reached from_config_yaml(...).get(...) and aborted proxy startup with AttributeError. #29590 added that guard for lakera_prompt_injection but not for these two
Mirror the isinstance(dict) guard already used by the lakera, presidio, and datadog branches so a non-dict value is ignored and the callback initializes with defaults. A parametrized test feeds every callback_settings consumer a non-dict value through initialize_callbacks_on_proxy to catch a future consumer that forgets the guard
* fix(callbacks): normalize non-dict callback_specific_params to empty dict
A blank callback_settings: key in YAML loads as None, and
config.get('callback_settings', {}) returns None because dict.get only
falls back to the default when the key is absent. Forwarding that value
verbatim to initialize_callbacks_on_proxy made the first
'<name>' in callback_specific_params membership test raise
TypeError: argument of type 'NoneType' is not iterable, aborting proxy
startup. Same failure for any non-dict root such as callback_settings: true.
Normalize the value at the function boundary so both callsites (and any
future ones) initialize callbacks with their defaults instead of crashing.
---------
Co-authored-by: Hedi Daoud <150018939+hdaoud23@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ui): add budget duration to edit team member form
Editing a team member created a member budget with no duration, so the
budget never reset. This threads a budget reset period through the edit
flow end to end and reuses the shared duration dropdown so the options
stay in sync with the rest of the UI.
Resolves LIT-2651
* fix(proxy): validate member budget_duration and persist clears
Reject budget_duration values that can't be parsed, are non-positive, or overflow date math before any write, so a bad value can't be persisted and later crash the budget reset job.
Clearing the budget duration in the edit-member form now sends null and clears the column end to end, so the dropdown's clear control reflects a real change instead of being a no-op
* chore(ui): regenerate schema.d.ts for member budget_duration
Adds budget_duration to TeamMemberUpdateRequest/Response in the generated dashboard types so the Check UI API Types Sync gate passes
* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter
ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.
_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.
Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.
Fixes#27730.
* fix(reset_budget): address Greptile review on #29358
- Strengthen the bypass-half regression test: replace the for-loop over
call_args_list (vacuously true when empty) with assert_not_called(),
so the test would actually flag a re-introduction of counter-zeroing
via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
_write_team_reset_updates that _write_key_reset_updates already has,
so all three helpers point future maintainers at #27730.
* test(reset_budget): update test_proxy_budget_reset for new batch-write path
Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:
- _wire_batcher_for_test helper that returns a list which accumulates per-row
batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
alongside the dict item-access the fake_reset_* mocks rely on. The new
narrow-write helpers use getattr to pull out the row's id, and would
silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
(rows by id, payload contains only {spend, budget_reset_at}) instead of
update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
budget + enduser still flow through update_data; key/user/team go through
the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
is actually awaitable and the success hook fires.
These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
* fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata
Batch create was failing with `Invalid type for 'metadata.applied_policies':
expected a string, but got an array instead` whenever a policy attachment
matched the request. The policy engine helpers wrote `applied_policies`,
`applied_guardrails`, and `policy_sources` into `data["metadata"]`
unconditionally, and `/v1/batches` forwarded that dict straight to OpenAI,
which only accepts string values.
- Route proxy-internal tracking into `litellm_metadata` for batch/file
routes via a shared `_get_or_create_proxy_metadata_bucket` helper.
- Sanitize `data["metadata"]` in `create_batch` to drop known internal
keys and non-string values before building the OpenAI request.
- Cover both behaviors with unit + endpoint tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): merge metadata buckets for batch policy response headers
Ensure get_logging_caching_headers reads both metadata and litellm_metadata so policy/guardrail headers are emitted on batch routes with user metadata, and log dropped non-string OpenAI metadata at debug level.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(proxy): strict media-type match for form bodies (#27939)
* chore(proxy): strict media-type match for form bodies
``_read_request_body`` and ``get_request_body`` routed on
``"form" in content_type`` / ``"multipart/form-data" in content_type``,
which match any header containing the literal — ``application/form-json``,
``multiform/anything``, ``application/json; xform=1``. Starlette's
``request.form()`` returns an empty ``FormData`` for any non-canonical
type without consuming the body, so the auth-time pre-read saw ``{}``
and skipped the banned-param check while the handler's later
``request.body()`` saw the original JSON payload.
Parse the media type per RFC 7231 (substring before ``;``, trimmed,
lowercased) and accept only ``application/x-www-form-urlencoded`` and
``multipart/form-data``. Replace both substring sites with the shared
``_is_form_content_type`` helper.
Tests pin: case/whitespace/charset variants of the two real types
match; ``application/form-json`` and similar substring-match traps
fall through to the JSON parse path; real form POSTs continue to
route through ``request.form()``.
* chore(proxy): extract _is_json_content_type symmetric helper
Mirror ``_is_form_content_type`` for the JSON branch of
``get_request_body`` so both classifications share the same media-type
normalisation (strip params, trim, lowercase) and any future change
to the parsing rules has one place to update.
Adds tests for ``_is_json_content_type`` and for ``get_request_body``
covering the canonical JSON / form / unsupported / non-POST paths.
* chore(proxy): surface form-parse failures instead of caching empty body
Starlette's ``request.form()`` raises ``MultiPartException`` /
``ValueError`` / ``AssertionError`` on malformed multipart input
(missing boundary, malformed chunk encoding, etc.). The outer
``except Exception: return {}`` swallowed every form-parse failure
and cached an empty parsed body — auth-time pre-reads saw ``{}`` and
skipped every banned-param check while a later raw-body re-read in
the handler still saw the original payload. Same TOCTOU shape as the
substring-match bypass: the auth gate and the handler don't agree on
what the body is.
Wrap ``request.form()`` in a narrow ``try`` that converts any parse
failure to a 400 ``ProxyException``. The outer broad ``except`` is
retained for unrelated unexpected errors but no longer covers
form-parse-side bypass shapes.
Adds a regression test parametrised over the exception classes
Starlette can raise from ``request.form()``.
* chore(proxy): drop redundant _is_json_content_type test class
``_is_json_content_type`` is a 3-line wrapper around the shared
``_normalize_media_type`` helper. Positive coverage lives in
``TestGetRequestBody.test_json_with_charset_param_parses_as_json``;
negative coverage is covered transitively by
``TestIsFormContentType``'s non-form parametrize matrix (anything that
isn't a form type falls through to the JSON branch).
* chore(proxy): carry ASGI path into WebSocket auth synthetic Request (#27940)
``user_api_key_auth_websocket`` built a synthetic ``Request`` with a
two-key scope (``type`` + ``headers``) and set ``request._url =
websocket.url``. ``get_request_route`` reads ``scope.get("path", ...)``
and falls back to ``request.url.path`` only when ``path`` is absent.
For the WebSocket flow that fallback fires and resolves to the
Host-header-derived value (Starlette reconstructs ``websocket.url``
from the Host header), so a malformed Host collapses the resolved
route and lets the auth gate compare against the wrong value.
Carry the ASGI scope's ``path``, ``root_path``, and ``app_root_path``
into the synthetic scope so the lookup never reaches the fallback on
the legitimate path.
Regression test pins that the request handed to ``user_api_key_auth``
has ``scope["path"]`` equal to the ASGI scope's path.
---------
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
* fix(proxy): resolve cache handling issues in _lookup_deprecated_key
- Updated the in-memory cache for deprecated key lookups to store a 3-tuple (active_token_id, cache_expires_at_ts, revoke_at_ts) instead of a 2-tuple, ensuring proper unpacking and backward compatibility.
- Removed duplicate cache reads and added logic to handle legacy cache entries gracefully.
- Enhanced unit tests to cover scenarios for cache hits, DB misses, and respect for revoke_at timestamps, ensuring robust handling of the grace-period key-rotation feature.
* refactor(proxy): streamline cache handling in _lookup_deprecated_key
- Simplified the cache retrieval logic by directly unpacking the 3-tuple cache entries, removing the need for backward compatibility checks for 2-tuple entries.
- Updated unit tests to ensure that pre-warmed 3-tuple cache entries are served correctly without unnecessary database lookups.
* chore(ci): add new unit test for deprecated key grace period
- Included `test_deprecated_key_grace_period.py` in the CI workflow to enhance coverage for deprecated key handling scenarios.
* fix(proxy): remove unnecessary check for revoke_at in _lookup_deprecated_key
- Eliminated the redundant check for None on revoke_at, streamlining the logic for handling deprecated keys in the cache. This change enhances the efficiency of the key lookup process.
* test(proxy): add end-to-end tests for deprecated key lookup behavior
- Introduced a new test class `TestDeprecatedKeyLookupDbE2E` to validate the behavior of deprecated key lookups against a real Prisma-backed database.
- The test ensures that old key hashes resolve correctly and that repeated lookups utilize the in-memory cache without errors.
- Cleaned up the `_lookup_deprecated_key` function by removing an unnecessary check for `revoke_at`, enhancing the efficiency of the key lookup process.
* fix: invalidate cached tag object on tag budget reset (#27481) (#27572)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* chore(mcp): tighten stdio server registration paths (#27570)
Squash-merged by litellm-agent from stuxf's PR.
* fix(proxy): clear MCP OpenAPI mappings on server eviction; widen budget cache invalidation
Evict OpenAPI tools from global_mcp_tool_registry and strip tool_name_to_mcp_server_name_mapping entries when a server leaves the runtime registry (remove_server and approval-status eviction). Invalidate user_api_key_cache for keys, orgs, and team members on budget-tier spend resets alongside tags.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): align update_server eviction with remove_server name fallback
Document budget-reset test assertion flip (cross-pod cache staleness).
Greptile: eviction now pops by server_id then server_name like remove_server;
test docstring explains assert_not_awaited -> assert_any_await change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix org budget cache invalidation
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Two cleanups from the /simplify review pass:
* ``Response`` was imported inside the ``except OSError`` branch in
``/get_image`` and at the top of ``/get_favicon``. Per the project's
no-inline-imports rule (CLAUDE.md), hoisted to the existing
``from fastapi.responses import (...)`` block at the top of
``proxy_server.py``.
* The test class's ``_patches()`` helper returned a 2-element list of
patch context managers and tests indexed into them via
``self._patches(...)[0], self._patches()[1]`` — two distinct calls
with confusing aliasing semantics. Restructured to:
- module-level ``_patch_async_safe_get(...)`` that returns a single
patch context manager
- autouse fixture that patches ``get_async_httpx_client`` for every
test in the file (it's the same patch in every case)
- small ``_image_response(...)`` factory to deduplicate Mock setup
Tests now read as ``with _patch_async_safe_get(return_value=...):``
with no list-indexing or duplicate Mock construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three review items addressed:
* **Veria (Medium): SSRF via redirect.** ``fetch_validated_image_bytes``
was calling ``validate_url(url)`` once and then fetching with the
default httpx client, so a 3xx to an internal IP would have been
followed unvalidated. Switched to ``async_safe_get`` (the existing
SSRF primitive used elsewhere in the codebase) which walks each
redirect hop, re-validates, and rejects redirects to blocked
networks. Default ``litellm.user_url_validation`` is True so
protection is on out of the box.
* **Greptile (P2): SVG can embed JS.** Removed ``image/svg+xml`` from
the allowed-Content-Type set. The hardcoded response media type
(``image/jpeg`` / ``image/x-icon``) means a real SVG body wouldn't
render as SVG anyway in modern browsers — the allowlist entry was
giving up XSS surface for no actual SVG-rendering benefit. If real
SVG support is wanted later, that's a deliberate feature PR with CSP
/ nosniff bundled.
* **Greptile (P2): cache-write OSError drops validated bytes.** When
the upstream fetch succeeded but ``open(cache_path, "wb")`` raised
(read-only assets dir), the bytes were discarded and the default
logo was served — a silent regression for that deployment. Now
serve the validated bytes inline via ``Response(...)`` as a fallback
before falling back to default.
Tests:
- Replaced low-level mocks of ``validate_url`` with mocks of
``async_safe_get`` directly, exercising the helper's contract
rather than the SSRF primitive's internals.
- New ``test_rejects_svg_content_type`` confirms SVG is blocked.
- ``test_get_image_cache_logic`` fixture now sets
``mock_response.is_redirect = False`` so ``async_safe_get`` doesn't
treat the Mock's truthy attribute as a redirect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The unauthenticated ``/get_image`` and ``/get_favicon`` endpoints accept
the admin-set env vars ``UI_LOGO_PATH`` and ``LITELLM_FAVICON_URL`` and
return whatever bytes they resolve to, with a hard-coded ``image/jpeg``
or ``image/x-icon`` content-type. Two attack shapes:
* ``UI_LOGO_PATH=/etc/passwd`` (or any other readable file path) — any
unauthenticated caller exfiltrates the file via ``GET /get_image``.
The previous gate was ``os.path.exists(logo_path)`` which fires on
every readable file. Same shape for the favicon endpoint.
* ``UI_LOGO_PATH=http://169.254.169.254/iam`` (or any internal HTTP
service the admin pointed at) — the proxy fetches it server-side
and streams the response body to the unauthenticated caller. No
URL validation, no Content-Type validation; ``application/json``
AWS metadata gets tunneled out under the ``image/jpeg`` wrapper.
New helper module ``litellm/proxy/common_utils/static_asset_utils.py``:
* ``resolve_local_asset_path(candidate, allowed_roots)`` — returns the
resolved absolute path only if it lives within one of the allowed
asset roots. Uses ``realpath`` so symlinks pointing outside the roots
are caught.
* ``fetch_validated_image_bytes(url)`` — runs the URL through
``validate_url`` (rejecting private / cloud-metadata / loopback
targets) and only returns the response body if the upstream
Content-Type is in a small allowlist of image MIME types.
Both ``/get_image`` and ``/get_favicon`` are wired through the helpers.
The SSRF gate is enforced unconditionally — these endpoints are
unauthenticated, so the admin-facing ``litellm.user_url_validation``
toggle does not apply (an admin who opted out of URL validation for
LLM provider paths shouldn't also expose ``/get_image`` to SSRF).
Tests:
- ``TestResolveLocalAssetPath``: 10 cases covering legitimate paths,
``/etc/passwd``, ``/proc/self/environ``, symlink-out, ``..``
traversal, directories, missing files, and root list edge cases.
- ``TestFetchValidatedImageBytes``: 7 cases covering SSRF block, non-
image content-type rejection, valid image passthrough, non-200
response, fetch exception, empty URL, and parametrized coverage of
every allowed image MIME type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three issues surfaced in review of the previous commit:
1. **Veria — Medium**: ``litellm_params`` carries a nested
``litellm_embedding_config`` dict (auto-resolved from the model
registry on create / update) which itself holds ``api_key`` /
``aws_*`` / ``vertex_credentials``. The previous redactor only
inspected top-level keys, so the nested values passed through
unredacted. Recurse into nested dicts.
2. **Greptile — P2**: when ``litellm_params`` is a JSON-serialized
string (the in-memory registry occasionally stores it that way), the
previous redactor silently no-op'd via the ``isinstance(..., dict)``
guard and echoed the raw payload back. Now: parse, redact, re-serialize.
If the string is not valid JSON, replace it with the redaction
sentinel rather than echo it.
3. **mypy** flagged ``_redact_sensitive_litellm_params``'s
``Optional[Dict[str, Any]]`` signature as incompatible with the
``object``-typed call site. Widened to ``Any -> Any`` to reflect the
actual contract (the function now handles dict / str / None / other).
Also fixes a related test regression in
``test_remove_sensitive_info_from_deployment_with_excluded_keys``: the
``"credentials"`` plural addition to ``SensitiveDataMasker`` defaults
caused the first call (without ``excluded_keys``) to mutate the input
dict's ``litellm_credentials_name`` to a masked value. The second call
(with ``excluded_keys``) then saw the already-masked value rather than
the original. Construct fresh input for each call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Updated instances of DualCache to UserApiKeyCache across multiple files to enhance cache handling for user API keys.
- Adjusted cache retrieval and storage methods to ensure proper serialization and deserialization of cached objects.
- Introduced a new UserApiKeyCache class to streamline caching logic and improve type safety.
- Updated relevant tests to reflect changes in caching behavior and ensure compatibility with the new cache implementation.
