Any authenticated user could fetch the profile image of a model they have no access to, and could tell an existing model id from an unknown one by whether the response carried the image or the default logo.
The endpoint now serves an image only to callers who can see the model itself: the owner, an admin under the admin bypass, or the holder of a read grant, with the same rule applied to arena models defined in config. Everyone else gets the default logo, byte for byte the response an unknown id already returned, so ids can no longer be probed. BYPASS_MODEL_ACCESS_CONTROL is honoured here because it is what decides which models reach a user's model list to begin with.
Avatars now fall back to the default logo wherever a viewer meets a model id without holding a grant on it: a model reply in a channel shown to the other members, and the admin analytics and evaluation pages when the admin bypass is switched off.
* fix(retrieval): serialize local embedding and reranking on MPS
On Apple Silicon the server process is killed outright (SIGSEGV or SIGTRAP, no traceback) partway through answering any question that retrieves from a knowledge base with hybrid search and a local reranking model. The client sees a dropped connection and the answer is lost.
Hybrid search fans its queries out concurrently and every task calls the same shared local model on a worker thread. Torch's Metal shader cache is a process-wide singleton whose lookup tables have no lock, so two of those threads racing inside it corrupt the cache and take the process down with it.
Guard the local SentenceTransformer and CrossEncoder calls with a shared lock that is only a real lock when the selected device is MPS. CPU and CUDA installs keep the concurrency they have today, and external reranking endpoints are untouched. Reranking several queries on a Mac now runs one at a time, which is the cost of the process staying alive.
Verified by driving the real hybrid-search fan-out with 16 concurrent queries: peak simultaneous entries into the local model drops from 16 to 1 on MPS, stays at 16 on CPU, and the returned documents, scores and ordering are byte-identical in every case.
Fixes#29722
* fix(evaluations): serialize the leaderboard embedder against retrieval on MPS
The leaderboard's tag-similarity search builds its own SentenceTransformer, and on Apple Silicon sentence-transformers places it on the MPS device. It runs on a worker thread, so an admin running a leaderboard search while anyone queries a knowledge base puts two threads into torch's Metal backend at the same time, which kills the server process outright with no traceback.
Move the lock added for the retrieval path into env.py, beside the device selection that decides whether MPS is used at all, and take it around the leaderboard's embedding calls as well. Sharing one lock between the two modules is the whole point, since two separate locks would still let a leaderboard search collide with a retrieval query.
Only inference is guarded, matching the retrieval path. Model construction stays as it is here and in the retrieval routers.
Verified by driving the leaderboard similarity path and retrieval reranking from six threads against one instrumented model: peak simultaneous entries drops from six to one on MPS, and the similarity scores are unchanged.
Related to #29722.
Custom metadata attached to a file upload (e.g. via the API's
metadata field) was stored on the file row but silently dropped
when the file was chunked and embedded for RAG, so it never reached
the LLM through retrieved sources.
process_file() builds chunk metadata in several branches; three of
them already merge the file's meta dict in, but the branch used for
a fresh upload processed through a document loader did not. Bring
it in line with the others so custom metadata flows through
consistently regardless of upload path.
Reported in open-webui/open-webui#29486.
* perf: stop the Socket.IO session pool blocking the websocket event loop
With WEBSOCKET_MANAGER=redis the session pool is a synchronous Redis client, so every call into it blocks the whole worker's event loop, not just the caller. Two paths did it constantly: the orphan reaper walked the pool one round trip per session with no await anywhere, freezing the loop for the entire sweep every cycle, and nearly every socket event re-read the sender's session back out of Redis. Other users' events and every in-flight generation on that pod wait behind both.
The reaper now walks the pool in HSCAN batches and deletes in bulk, yielding between batches, and no longer sleeps past half the lock TTL, which previously guaranteed a failed renew every cycle. The per-event reads are gone: Socket.IO events only reach the worker holding the connection, and that worker already saved the same session dict locally when the user authenticated, so it was asking Redis for its own data. The writes stay, since those are what other pods read.
Measured at 4000 users / 16 containers, Redis 1.1 ms away:
| | before | after |
|---|---|---|
| reaper sweep, 5k sessions | 5.6 s, loop frozen throughout | 62 ms, 4.2 ms worst block |
| same, crash recovery with every session expired | 11.5 s | 96 ms |
| heartbeat / usage ping / disconnect | 2 / 3 / 2 round trips | 1 / 2 / 1 |
| 50-member channel post | 50 round trips, 57.2 ms block | 0 round trips, 0.02 ms |
| loop time per wall second at rest | 103 ms (10.3%) | 67 ms (6.7%) |
The alternative, converting RedisDict to the async client, fixes the same paths with a far larger blast radius (every call site gains await, and `in`/`[]`/`del` cannot be awaited so the dict interface goes) and still round-trips for data already in memory. Two deliberate behaviour changes: a heartbeat re-adds a session the reaper already removed, so a tab that survives a stall recovers instead of staying out of the pool until it reconnects; and disconnect no longer skips Yjs document cleanup when the pool entry is already gone, which previously leaked that document's update log forever.
Closes#28172
* perf: cut disconnect and user session lookup pool round trips, harden the session reaper
Follow-up on top of the session pool reaper branch. With WEBSOCKET_MANAGER=redis two paths still blocked the worker's event loop on synchronous Redis calls. Every disconnect listed all models in use cluster-wide and fetched each one individually, one blocking round trip per model. Disconnecting all sessions of a user (admin role change or deletion) pulled the entire session pool in one HGETALL and decoded every entry in a single uninterrupted block.
Disconnect now fetches the usage pool once with items(), going from 2+N+M round trips to 2+M (N models in use cluster-wide, M models the session used), and its delete of an emptied model entry is KeyError-guarded because another node can remove the same key between snapshot and delete; unguarded, that race aborted the handler and skipped its Yjs document cleanup. The user session lookup reuses the reaper's HSCAN batches and yields to the loop between pages. The reaper previously died permanently on the first Redis connection error, on every node at once during an outage; it now logs, releases the lock and returns to retrying acquisition.
* refac: keep the socket pool perf work to the round trips
A review pass on this branch turned up four changes riding along with the round-trip work without belonging to it, so they are backed out here. The `disconnect` handler keeps its `if sid in SESSION_POOL:` guard, so USAGE_POOL and ydoc cleanup stay off the path for sockets that never authenticated. `RedisDict.set()` keeps its own inline HDEL. `get_session_ids_by_user_id` stays synchronous over one HGETALL, since it runs on user delete and role change rather than per message. The crash-resilience wrapper around the reaper loop is dropped; if that guard is worth having, it belongs in its own change.
What stays is the perf part. The reaper now sweeps the pool in bounded HSCAN batches and deletes expired sids with one HDEL per batch, down from HKEYS plus an HGET and a per-sid HDEL across the whole pool. The `disconnect` handler reads USAGE_POOL with a single HGETALL, down from HKEYS plus one HGET per model in use. Session lookups in the socket handlers come from the local Socket.IO store, which removes one Redis GET from every heartbeat, usage, channel and ydoc event.
Naming and annotations follow the file: `get_session_pool_batches` for the module's `get_` prefix, `RedisDict.pop_many` so both reaper branches use one word for removing keys, a named `SCAN_BATCH_SIZE`, and types on the new helpers.
* fix: invalidate the RedisDict write signature on batch delete
RedisDict.set() skips the write when the payload fingerprint matches the last one this process wrote, so a mutation that goes around set() has to clear that fingerprint. The new batch delete did not, leaving a stale fingerprint behind: the next refresh with identical content is treated as already written and silently skipped, so the hash stays empty.
Renamed pop_many to delete_many. In a dict emulation pop removes and returns; this returns nothing and cannot without an extra HMGET, so the name promised something it does not do. delete_many matches __delitem__ and the HDEL underneath. Its only call site is the session pool reaper, whose behaviour is unchanged: same fields deleted, same batching, same return.
process_pipeline_inlet_filter() and its outlet counterpart construct and tear
down an aiohttp ClientSession, with its own connector and cookie jar, on
every chat completion and every task generation request just to iterate an
empty filter list. On deployments without pipelines, which is the default,
that is wasted setup on every message.
Both functions now return the payload untouched before the session is
created when there is nothing to call. The per-call saving is small, a few
microseconds of object construction per request on the pinned aiohttp; the
point is that requests stop paying setup for a feature that is not
configured.
Some providers send one very large piece of a streamed answer in a single go: a long reasoning trace, a code execution result, a turn with many tool calls, or a response echo carrying a big tool list. Anything past 128 KB in one line killed the chat mid-answer with a misleading `400, message: Got more than 131072 bytes when reading`. Nothing was rejected upstream, that is our own reader giving up on an oversized line.
Open WebUI already had code that assembles lines itself with no such limit, but it only ran when CHAT_STREAM_RESPONSE_CHUNK_MAX_BUFFER_SIZE was set. Unset is the default, and in that case the raw capped reader was used instead, so a default install always broke. That path now always assembles lines, and the setting goes back to being what its name says: an optional cap, off by default. It applies to the Ollama stream as well, since both now share the same reader.
The assembly loop only splits once a line actually completes, because the old one re-concatenated and re-split the whole buffer on every network chunk. Without that, allowing long lines would have traded an error for multi-second event loop stalls.
| | 20 MB in one line | 200k small lines |
| --- | --- | --- |
| before | 4249 ms | 27.3 ms |
| after | 37 ms | 25.2 ms |
Deleting an external knowledge base now clears its connection only when an admin removes the last knowledge base referencing it, matching the connection delete route.
Moving a folder under one of its own subfolders was accepted. A folder in a parent loop is never a root, so it and everything under it silently disappeared from the sidebar, and there was no way to get it back from the UI.
The move is now rejected with a 400, folders whose parent chain loops are put back at the root on the next folder list, and the folder tree traversals skip ids they have already visited so existing data in that state stays workable.
* fix: use the pooled client timeout for the Anthropic Messages passthrough
The native `/api/v1/messages` passthrough still referenced `openai.AIOHTTP_CLIENT_TIMEOUT`, which stopped existing when `routers/openai.py` moved onto `session_pool.get_client_timeout()`. Every passthrough request therefore raised `AttributeError: module 'open_webui.routers.openai' has no attribute 'AIOHTTP_CLIENT_TIMEOUT'` before it was sent, and the surrounding handler turned that into a 502 "Open WebUI: Server Connection Error", so Anthropic-format clients such as Cline could not reach any model at all.
Use `get_client_timeout(stream=...)` like the OpenAI and Ollama proxies do, so the configured `AIOHTTP_CLIENT_TIMEOUT` applies and streaming requests additionally get the idle-read timeout.
Fixes#27595
* fix: authenticate native Anthropic requests with x-api-key
The Anthropic Messages passthrough and the token-count forwarding both build their upstream request through `get_anthropic_request_target`, which sends the connection key as `Authorization: Bearer <key>`. Anthropic's OpenAI-compatible `/chat/completions` endpoint accepts that, which is why the model works in the chat UI, but the native `/v1/messages` and `/v1/messages/count_tokens` endpoints do not: they require the key in `x-api-key` and reject a bearer token with 401 `Invalid bearer token` (and `jwt auth is not yet supported on count_tokens`). They also require an `anthropic-version` header, which was never sent.
For `api.anthropic.com` connections, send `anthropic-version` and move the key into `x-api-key`, dropping the bearer header. Connections using session, OAuth or Entra ID auth keep their token untouched, LiteLLM passthrough connections are unaffected, and admin-configured custom headers still win over both defaults.
Fixes#27695
`x in d` and `x in d.keys()` are identical for a plain dict, so the `.keys()` call builds a throwaway view and reads as if it were doing something. Both sites operate on a plain dict: `combined` in `merge_and_sort_query_results` is a local `dict()`, and `ui_settings` comes from `UserSettings.model_dump()` where `ui` is annotated `dict | None` and is already guarded against None on the preceding line.
No behaviour change, and no measurable speedup either, so this is a readability cleanup rather than a performance one.
Sites where `.keys()` is load-bearing are left alone: the `list(d.keys())` snapshots taken before mutating during iteration, and the places where `.keys()` is the iteration or comprehension source rather than a membership test.
The web search error message was a lambda with a passthrough branch that returned whatever it was handed. Since #28942 both call sites pass no arguments, so that branch is unreachable, and it is the trap that let a caller drop a raw exception object into an HTTP response body and turn an intended 400 into an unserialisable 500.
A plain string constant removes the trap and lines the message up with every other fixed message in that file. Behaviour is unchanged: the response detail comes out byte for byte identical, because the enum already overrides __str__ to render members as their value. Verified on Python 3.11 and 3.12, both producing the same string and the same JSON body.
Any failure during a web search comes back to the client as a bare HTTP 500 with nothing in it. The handler tries to build a 400 whose detail is the caught exception object itself, FastAPI cannot serialise that into a response body, so rendering the error response fails and the request falls through to the generic 500 handler. In chat this surfaces as a web search that fails with no explanation at all, and the most common trigger is simply selecting a search engine without configuring its API key.
This routes the failure through the standard error formatter, which is what the sibling handler for content loading failures in the same function already does. Web search failures now return 400 with a readable message, and the exception itself keeps going to the server log exactly as before.
Passing str(e) into the response was the other option and was rejected: the rest of the backend deliberately keeps provider exception text out of client responses and in the log, and provider exceptions here can carry request details that should not be echoed back.
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Exporting workspace models loaded every model row, built a full response object with its owner for each, and only then dropped the ones the caller may not see. On a large model table that made the export endpoint slow in proportion to models the user cannot even access.
The owner-or-grant check now happens in the query itself, reusing the permission filter this file already applies to the paginated list endpoint, so only visible rows are ever hydrated. The by-user wrapper had one caller left and is gone with it.
Measured with 500 workspace models of which 3 are visible to the caller: 5 queries and ~12.7 ms before, 4 queries and ~2.8 ms after. The resulting set is unchanged for owner, public, direct-user, group and multi-grant entries, and base model entries stay excluded as before.
Listing a user's folders re-checks which entries they may still see, and it resolved their group membership again for every folder, then again inside the collection and note branches for every entry. A comment in that helper claims one membership fetch for the whole listing, but the caller invokes it once per folder, so the claim never held.
The listing now resolves membership once, and only when some folder actually carries entries, then threads it through the file, collection and note checks. Callers that do not supply it are unchanged and still resolve for themselves.
Measured with twenty folders holding six files, two knowledge bases and two notes each: 245 queries and ~145 ms before, 186 and ~117 ms after. The folders returned, and the entries the integrity pass writes back, are unchanged. That was checked against entries the caller owns, entries shared through a group, entries shared with nobody, another user's files, and an unrecognised entry type.
Listing skills ran one database query per skill in the instance. A non-admin opening the list on a workspace with 500 skills issued over 500 queries, the paginated list re-resolved the caller's group membership once per row, and every chat message carrying a skill loaded every skill the user can read, full body and owner included, to use the two or three it actually referenced.
Skills now arrive already filtered: the owner-or-grant check runs in the query as an EXISTS subquery, the same way prompts and the search endpoints already do it, the per-item write flag uses the existing batch grant lookup, and the chat path asks only for the skill ids the request names.
Measured with 500 skills of which 3 are visible to the caller: 504 queries and ~300 ms before, 4 queries and ~2.6 ms after. The resulting set is unchanged for owner, public, direct-user, group and multi-grant entries, for both read and write.
Opening the shared folder list fetched every shared folder in its own query, fetched a chunk of them a second time to walk their children, and looked up each distinct owner separately. With forty folders shared with a user that is over a hundred queries before any subtree work starts.
The folders and their owners now come back in one query each, and the inheritance pass reuses the rows already in hand. Both folder listings also gained an explicit order: the sidebar merges shared subfolders in response order without sorting them, and neither query had an ORDER BY, so on Postgres a folder rename could reshuffle its siblings.
Measured with forty shared folders and no subtrees: 181 queries and ~105 ms before, 92 and ~66 ms after. With subtrees attached, 203 folders in total, it is 341 queries before against 252 after; the remainder is the recursive child walk, which this change deliberately leaves alone. The returned set, permissions and owner names are unchanged, including for a grant pointing at a deleted folder row, a folder the caller owns that is also shared with them, a folder whose owner record is gone, and a child folder that is itself directly shared.
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Changing a password left every other logged-in device working until the JWT expired on its own, up to four weeks with the default settings. The hardening docs already promise the opposite: with Redis configured a password change is supposed to put the user's tokens on the revocation list, but only sign-out and OIDC back-channel logout ever wrote to it.
Both password-change paths, self-service and an admin resetting someone's password, now stamp the per-user revocation marker that token validation already checks, so every session issued before the change stops working. The acting device is signed out as well and asked to sign in again, which is the safer default when the password is being changed precisely because the old one may be compromised. Without Redis nothing can be revoked, as before, and the backend now logs a warning saying so.
The marker is written through one shared helper, so its lifetime follows the configured JWT lifetime instead of a fixed 30 days and never expires at all when JWT_EXPIRES_IN disables expiry. Back-channel logout picks that up too, where a long or disabled JWT lifetime previously let the marker expire while the tokens it revoked were still valid. API keys keep working, they are separate credentials with their own lifecycle.
Discussed in #28647.
Seven route handlers declare a request-scoped database session as a FastAPI dependency and then never touch it. Three of them are `GET /api/v1/users/user/settings`, `/user/status` and `/user/info`, which the frontend hits on every page load, and all three carry a comment saying the user object is already available, so the parameter is leftover from the refactor that removed the refetch. The other four are admin-only external-knowledge connection endpoints that read their data from the config store.
Measured on a route with and without the dependency, 20k requests, best of 5:
| | µs per request |
| --- | --- |
| no dependency | 16.18 |
| unused session dependency | 62.85 |
The dependency costs about three times as much as everything else the request does put together. It is worth being precise about why, because the obvious guess is wrong: this is not database I/O and not connection pool pressure. SQLAlchemy connects lazily, so a session that is never used checks out zero connections, verified by watching the pool's counter stay at zero across the request. The cost is FastAPI resolving an extra async-generator dependency onto the request's exit stack, plus constructing and closing the session object.
Deleting the seven parameters is the whole change. An AST scan over the backend finds exactly these seven handlers before and none after.
With database session sharing enabled, which the docs recommend for PostgreSQL and for multi-replica deployments, the knowledge pending-files and file process-status endpoints each pinned one pooled connection for as long as their SSE stream stayed open, up to one and two hours respectively. A file wedged in processing keeps a stream open for the full duration, so a handful of users sitting on that page can consume every connection in the pool, and the held transactions sit idle and block autovacuum on those tables.
Both handlers took a request-scoped session for their access checks, and FastAPI only releases a yield dependency once the response body has finished streaming, so the session outlived the handler by the whole life of the stream. Neither generator ever used it. They no longer take that dependency, and the queries they run already open their own short-lived sessions when none is passed. This is the approach the chat completion endpoints already use for the same long-response problem.
Measured against a pool with capacity 11: before, at most 11 concurrent streams could ever be open and every further attempt failed, deterministically across repeat runs. After, 25 of 25 opened. Non-stream latency is unchanged, within run-to-run noise, and behaviour is identical whether session sharing is on or off.
