Commit graph

29 commits

Author SHA1 Message Date
Classic298
68a74da70b
Fix: mps inference evaluations (#29735)
* fix(retrieval): serialize local embedding and reranking on MPS

On Apple Silicon the server process is killed outright (SIGSEGV or SIGTRAP, no traceback) partway through answering any question that retrieves from a knowledge base with hybrid search and a local reranking model. The client sees a dropped connection and the answer is lost.

Hybrid search fans its queries out concurrently and every task calls the same shared local model on a worker thread. Torch's Metal shader cache is a process-wide singleton whose lookup tables have no lock, so two of those threads racing inside it corrupt the cache and take the process down with it.

Guard the local SentenceTransformer and CrossEncoder calls with a shared lock that is only a real lock when the selected device is MPS. CPU and CUDA installs keep the concurrency they have today, and external reranking endpoints are untouched. Reranking several queries on a Mac now runs one at a time, which is the cost of the process staying alive.

Verified by driving the real hybrid-search fan-out with 16 concurrent queries: peak simultaneous entries into the local model drops from 16 to 1 on MPS, stays at 16 on CPU, and the returned documents, scores and ordering are byte-identical in every case.

Fixes #29722

* fix(evaluations): serialize the leaderboard embedder against retrieval on MPS

The leaderboard's tag-similarity search builds its own SentenceTransformer, and on Apple Silicon sentence-transformers places it on the MPS device. It runs on a worker thread, so an admin running a leaderboard search while anyone queries a knowledge base puts two threads into torch's Metal backend at the same time, which kills the server process outright with no traceback.

Move the lock added for the retrieval path into env.py, beside the device selection that decides whether MPS is used at all, and take it around the leaderboard's embedding calls as well. Sharing one lock between the two modules is the whole point, since two separate locks would still let a leaderboard search collide with a retrieval query.

Only inference is guarded, matching the retrieval path. Model construction stays as it is here and in the retrieval routers.

Verified by driving the leaderboard similarity path and retrieval reranking from six threads against one instrumented model: peak simultaneous entries drops from six to one on MPS, and the similarity scores are unchanged.

Related to #29722.
2026-09-06 15:20:41 -04:00
Timothy Jaeryang Baek
300302d432 refac 2026-07-24 02:36:10 -04:00
Timothy Jaeryang Baek
517cd8d102 refac 2026-06-29 13:03:14 -05:00
Timothy Jaeryang Baek
b5c43968db refac 2026-06-25 03:31:45 +01:00
Timothy Jaeryang Baek
5cdcdbaeec refac 2026-06-17 02:52:35 +02:00
Timothy Jaeryang Baek
160a6694e4 refac 2026-06-01 13:20:33 -07:00
Classic298
92e727b1fc
refactor: remove unused GET /evaluations/feedbacks/all endpoint (#24778)
GET /evaluations/feedbacks/all returned the entire feedback table in a
single response (flagged as a Medium OOM risk for admins in
open-webui#22206). Its only frontend wrapper, getAllFeedbacks in
src/lib/apis/evaluations/index.ts, is dead: nothing imports or calls it
anywhere in the codebase. The endpoint is a redundant view-only twin of
GET /evaluations/feedbacks/all/export, which is what the admin Feedbacks
UI actually uses.

Removes the endpoint, the now-unused FeedbackResponse import in the
evaluations router, and the dead getAllFeedbacks frontend wrapper. The
shared Feedbacks.get_all_feedbacks data-layer method is kept, since the
live /feedbacks/all/export endpoint still uses it.

Ref: open-webui#22206

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 20:49:22 +04:00
Timothy Jaeryang Baek
6d0295588e refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
Timothy Jaeryang Baek
25898116ea chore: format 2026-04-12 18:12:59 -05:00
Timothy Jaeryang Baek
27169124f2 refac: async db 2026-04-12 14:22:11 -05:00
Timothy Jaeryang Baek
a71d927a0c chore: format 2026-04-02 08:11:06 -05:00
Timothy Jaeryang Baek
60e4d75174 refac 2026-04-02 02:50:05 -05:00
Timothy Jaeryang Baek
ade617efa8 refac 2026-03-24 04:49:48 -05:00
Timothy Jaeryang Baek
de3317e26b refac 2026-03-17 17:58:01 -05:00
Timothy Jaeryang Baek
7a7a0c423b chore: format 2026-01-09 20:44:31 +04:00
Timothy Jaeryang Baek
74c4af6e11 refac 2026-01-09 20:25:51 +04:00
Timothy Jaeryang Baek
9496e8f7b5 feat: model evaluation activity chart 2026-01-09 20:19:51 +04:00
Timothy Jaeryang Baek
10838b3654 refac/fix: feedback leaderboard 2026-01-09 18:24:09 +04:00
Timothy Jaeryang Baek
b2a1f71d92 refac: get feedback ids 2026-01-09 03:06:24 +04:00
Timothy Jaeryang Baek
b1d0f00d8c refac/enh: db session sharing 2025-12-29 00:21:18 +04:00
Timothy Jaeryang Baek
53c1ca64b7 enh: feedback ids endpoint 2025-12-21 15:33:06 +04:00
Timothy Jaeryang Baek
6083960655 refac: feedback list optimisation 2025-11-20 17:10:12 -05:00
expruc
1ff2ad0c08
ci fix 2025-07-21 22:17:08 +03:00
expruc
14c6f8bf80 added a check for admins to avoid 404 errors on feedbacks get and post 2025-07-21 21:18:37 +03:00
Timothy Jaeryang Baek
6e8481e157 fix: evaluation feedback endpoints 2025-05-14 20:08:49 +04:00
Timothy Jaeryang Baek
e6ca7a414b refac 2025-05-05 16:41:51 +04:00
omahs
863f227be9
fix: typos 2025-05-05 14:14:59 +02:00
Timothy Jaeryang Baek
d55735dc1e refac: rm profile image from feedback user object 2025-03-30 21:23:42 -07:00
Timothy Jaeryang Baek
d3d161f723 wip 2024-12-10 00:54:13 -08:00
Renamed from backend/open_webui/apps/webui/routers/evaluations.py (Browse further)