mirror of
https://github.com/open-webui/open-webui.git
synced 2026-10-01 02:01:35 +00:00
19222eff68
Issue #25216 named two root causes for unbounded SESSION_POOL/USAGE_POOL growth. The first — `periodic_usage_pool_cleanup` exiting permanently on Redis lock acquire/renew failure — is being addressed by PR #21798's `run_with_lock` helper. This commit closes the second: the `asyncio.create_task(...)` calls at the cleanup-daemon spawn sites discard their return value, so if either coroutine ever does exit for a reason that escapes the inner loop (notably `asyncio.CancelledError`, which is `BaseException` and is not caught by the `except Exception` guards in `run_with_lock`), there is no real-time operator signal — the asyncio default exception handler emits only a delayed "Task exception was never retrieved" WARNING at GC time. By that point both pools have already been growing unbounded for an unknown interval. Three changes against `backend/open_webui/main.py`, mirroring the existing `app.state.redis_task_command_listener` pattern at lines 676 and 754: 1. Store each cleanup-task handle on `app.state` and give it a `name=` so log lines identify which daemon died. 2. Attach a `_daemon_done_cb` that logs the exception at `ERROR` with `exc_info` at the moment of task death, not at GC time. The callback filters out `task.cancelled()` so it produces no noise during normal shutdown. 3. Cancel both tasks cooperatively in the lifespan shutdown block. Without this, `CancelledError` is never delivered, the tasks die while pending, and asyncio emits "Task was destroyed but it is pending!" warnings on every clean restart — which trains operators to ignore that warning and masks real failures. The change is orthogonal to PR #21798 (zero file overlap; that PR modifies `socket/main.py` and `socket/utils.py`, this one touches only `main.py`). The two PRs can be reviewed, merged, and reverted independently. With both in place, the cleanup loop is internally resilient (it will not die from Redis events) AND the spawn site is observable (if it dies anyway, the operator knows immediately). Note: line 687 spawns `scheduler_worker_loop` with the same fire-and-forget pattern. It is out of scope for this PR — issue #25216 specifically tracks the pool cleanup daemons — but the same treatment would apply. Refs: #25216, #21798 |
||
|---|---|---|
| .. | ||
| data | ||
| internal | ||
| migrations | ||
| models | ||
| retrieval | ||
| routers | ||
| socket | ||
| static | ||
| storage | ||
| tools | ||
| utils | ||
| __init__.py | ||
| alembic.ini | ||
| config.py | ||
| constants.py | ||
| env.py | ||
| functions.py | ||
| main.py | ||
| tasks.py | ||