fix: keep a failed lock release from masking cancellation (#29979)

When Redis is unreachable at shutdown, the socket cleanup tasks do not stop. release_lock is a bare eval called from the finally of both periodic_session_pool_cleanup and periodic_usage_pool_cleanup, so it raises there and replaces the CancelledError already in flight. The usage task's except Exception then catches the Redis error and carries on reaping after shutdown cancelled it, and the session task ends with a ConnectionError in place of its cancellation.

release_lock now logs the failure and returns. aquire_lock sets the key with ex=self.timeout_secs and renew_lock re-expires it with the same value, so a release that never lands costs at most one lock timeout before another node can take over.

The except names both RedisClusterException and RedisError because the cluster-only types subclass Exception directly, and redis_cluster is a supported configuration, so RedisError alone would miss the outage on a cluster. Genuine bugs still propagate.

Verified against a real Redis: acquire, refusal while held, renew, compare-and-delete release and non-owner release are unchanged, a populated pool reaps identically with identical lock TTL lifecycles, and both tasks now cancel cleanly where before one kept running and the other died with the wrong exception.
This commit is contained in:
Classic298 2026-09-14 03:37:51 +02:00 committed by GitHub
parent d372bec704
commit 55b7343be8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -3,12 +3,16 @@
from __future__ import annotations
import hashlib
import logging
import uuid
import pycrdt as Y
from open_webui.env import REDIS_KEY_PREFIX
from open_webui.utils.json_codec import JSONCodec
from open_webui.utils.redis import get_redis_connection
from redis.exceptions import RedisClusterException, RedisError
log = logging.getLogger(__name__)
YDOC_KEY_PREFIX = f'{REDIS_KEY_PREFIX}:ydoc:documents'
SCAN_BATCH_SIZE = 200
@ -58,7 +62,10 @@ class RedisLock:
return bool(self.redis.eval(self._RENEW_SCRIPT, 1, self.lock_name, self.lock_id, self.timeout_secs))
def release_lock(self):
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
try:
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
except (RedisClusterException, RedisError) as e:
log.warning('Failed to release lock %s; it expires on its own: %s', self.lock_name, e)
class RedisDict: