A slow Redis freezes the whole worker during sign-in, not just the user signing in. RateLimiter held a synchronous redis-py client and signin called is_limited inline from a coroutine, so every attempt did blocking round trips on the event-loop thread, with REDIS_SOCKET_TIMEOUT defaulting to None so nothing bounded the wait. Its Redis methods are now async and take the handle as their first argument, and both handlers pass request.app.state.redis, the async client the lifespan already creates. Building one in the limiter instead would pin its pooled connection to the first event loop that used it.
Without Redis, which is the default single-instance setup, the fallback store leaked. It was keyed by the rate-limit key and pruned a key's expired buckets only when that same key was checked again, so a login email never seen again was never reclaimed, and that email comes straight from an unauthenticated request body. It is now keyed by bucket, so one prune drops every key an expired bucket held, and it lives on the instance: pruning uses the per-instance num_buckets, so a shared store would let a limiter with a short window delete buckets a longer-windowed one still needs.
With a Redis costing a second per call, the widest event-loop tick gap drops from 2.010s to 0.010s and a concurrent request is answered at 0.05s instead of 2.05s, at no cost to the caller's own latency. Across 20,000 distinct keys the store goes from 40,000 entries and 6.4 MB, growing linearly, to a flat 1,004 entries and 100 KB. Rate-limiting decisions are unchanged across 700,000 randomised calls over 14 window, bucket and limit combinations, against a real Redis and the in-memory fallback alike, and sign-in still returns its first 429 on attempt 16.
Two behaviour changes worth naming. Pruning is now global rather than per key, so a wall clock that jumps forward past a full window and back forgets a hit it previously kept. The two limiters also stop sharing a store, which previously let a sign-in attempt with an IP-shaped email touch the token-exchange limiter's counters.
Deleting a tool or a function leaves its full source text in memory for the life of the process. Each delete handler pops the module cache and leaves the matching content cache untouched, so the code of every plugin ever deleted stays resident.
Both handlers now pop the content cache next to the module cache. Measured over 200 deletes of an 8.8KB plugin: 1.78 MB of source retained per kind before, nothing after.
This is memory only, never a stale module. A cache hit needs the id present in both caches and delete already popped the module cache, so the leftover source could not have produced one.
The two routers are one change because it is the same pair of lines at the same point in sibling handlers, with no per-site reasoning and no reason to revert one without the other.
Setting ENABLE_PROFILE_IMAGE_URL_FORWARDING=false stops the user and model profile image endpoints from redirecting browsers to external avatar URLs, but channel webhook avatars kept redirecting regardless. An operator who turned the setting off precisely to stop clients leaking their IP, User-Agent and Referer to outside origins still leaked all three whenever anyone viewed a channel message posted by a webhook with an external profile image URL.
The webhook profile image endpoint now reads the same setting the user and model endpoints already read, and serves the bundled default image instead of the redirect when forwarding is off. Stored URLs are untouched, so turning the setting back on restores the previous behaviour.
Verified against the real handler with seeded webhook rows: with the setting unset or true the endpoint still returns the 302 with the original Location, with it false it returns the default favicon as image/png with no Location and no header carrying the external host, and the data URI, no image and unknown webhook responses are byte identical in both states.
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Streamed chat completions and the other proxied OpenAI and Ollama responses
went out with two Server and two Date headers: the upstream's copies, forwarded
verbatim, plus uvicorn's own. nginx in front of Open WebUI logs "upstream sent
duplicate header line" for both on every streamed request.
Server and Date belong to whoever terminates the connection, so both proxies now
drop the upstream's copies next to the encoding headers they already stripped.
The filter also compares header names case-insensitively. Before, it matched
title-case names only, so uvicorn-based upstreams such as vLLM and LiteLLM,
which send lowercase header names, had none of their headers stripped at all,
including the Content-Encoding entry the filter exists for.
Same fix as #29824 for the terminal proxy, applied to the other two proxy paths.
Proxied terminal responses (GET /api/v1/terminals/{id}/ports and every other
proxied route) went out with two Server and two Date headers: the terminal
server's copies, forwarded verbatim, plus uvicorn's own. nginx in front of Open
WebUI logs "upstream sent duplicate header line" for both on every request, and
the ports route is polled often enough to fill gigabytes of error log per day.
Server and Date belong to whoever terminates the connection, so the proxy now
drops the upstream's copies next to the framing headers it already stripped.
uvicorn's own values still go out, once. Everything else, including custom
upstream headers and TERMINAL_PROXY_HEADERS, passes through unchanged.
The OpenAI and Ollama proxies forward upstream Server and Date the same way and
are left for a separate change.
Fixes#29824
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
The bulk tool export returned every tool the caller could read, while the per-tool export path returns only what the caller can write. This aligns the two, matching how model export already scopes its query.
Callers still export their own tools and any tool shared with them for writing; admins running with BYPASS_ADMIN_ACCESS_CONTROL are unaffected.
Any authenticated user could fetch a channel webhook's avatar, or be redirected to its external profile image URL, without belonging to the channel or holding any read access to it. This was the only webhook route with neither a channel check nor the channels feature gate.
The route now applies the same read gate every other route in this router uses: active membership for group and direct message channels, admin or a channel read grant otherwise, answering with 403 on denial and 404 when the webhook's channel row no longer exists. It also runs the channels feature and permission gate, so with channels disabled, or the permission withdrawn from regular users, the endpoint now refuses where it previously served the image.
Avatars keep rendering for channel members, and a denied request shows the default logo rather than a broken image, because the avatar component already falls back on an image error.
Any authenticated user could fetch the profile image of a model they have no access to, and could tell an existing model id from an unknown one by whether the response carried the image or the default logo.
The endpoint now serves an image only to callers who can see the model itself: the owner, an admin under the admin bypass, or the holder of a read grant, with the same rule applied to arena models defined in config. Everyone else gets the default logo, byte for byte the response an unknown id already returned, so ids can no longer be probed. BYPASS_MODEL_ACCESS_CONTROL is honoured here because it is what decides which models reach a user's model list to begin with.
Avatars now fall back to the default logo wherever a viewer meets a model id without holding a grant on it: a model reply in a channel shown to the other members, and the admin analytics and evaluation pages when the admin bypass is switched off.
* fix(retrieval): serialize local embedding and reranking on MPS
On Apple Silicon the server process is killed outright (SIGSEGV or SIGTRAP, no traceback) partway through answering any question that retrieves from a knowledge base with hybrid search and a local reranking model. The client sees a dropped connection and the answer is lost.
Hybrid search fans its queries out concurrently and every task calls the same shared local model on a worker thread. Torch's Metal shader cache is a process-wide singleton whose lookup tables have no lock, so two of those threads racing inside it corrupt the cache and take the process down with it.
Guard the local SentenceTransformer and CrossEncoder calls with a shared lock that is only a real lock when the selected device is MPS. CPU and CUDA installs keep the concurrency they have today, and external reranking endpoints are untouched. Reranking several queries on a Mac now runs one at a time, which is the cost of the process staying alive.
Verified by driving the real hybrid-search fan-out with 16 concurrent queries: peak simultaneous entries into the local model drops from 16 to 1 on MPS, stays at 16 on CPU, and the returned documents, scores and ordering are byte-identical in every case.
Fixes#29722
* fix(evaluations): serialize the leaderboard embedder against retrieval on MPS
The leaderboard's tag-similarity search builds its own SentenceTransformer, and on Apple Silicon sentence-transformers places it on the MPS device. It runs on a worker thread, so an admin running a leaderboard search while anyone queries a knowledge base puts two threads into torch's Metal backend at the same time, which kills the server process outright with no traceback.
Move the lock added for the retrieval path into env.py, beside the device selection that decides whether MPS is used at all, and take it around the leaderboard's embedding calls as well. Sharing one lock between the two modules is the whole point, since two separate locks would still let a leaderboard search collide with a retrieval query.
Only inference is guarded, matching the retrieval path. Model construction stays as it is here and in the retrieval routers.
Verified by driving the leaderboard similarity path and retrieval reranking from six threads against one instrumented model: peak simultaneous entries drops from six to one on MPS, and the similarity scores are unchanged.
Related to #29722.
Custom metadata attached to a file upload (e.g. via the API's
metadata field) was stored on the file row but silently dropped
when the file was chunked and embedded for RAG, so it never reached
the LLM through retrieved sources.
process_file() builds chunk metadata in several branches; three of
them already merge the file's meta dict in, but the branch used for
a fresh upload processed through a document loader did not. Bring
it in line with the others so custom metadata flows through
consistently regardless of upload path.
Reported in open-webui/open-webui#29486.
* perf: stop the Socket.IO session pool blocking the websocket event loop
With WEBSOCKET_MANAGER=redis the session pool is a synchronous Redis client, so every call into it blocks the whole worker's event loop, not just the caller. Two paths did it constantly: the orphan reaper walked the pool one round trip per session with no await anywhere, freezing the loop for the entire sweep every cycle, and nearly every socket event re-read the sender's session back out of Redis. Other users' events and every in-flight generation on that pod wait behind both.
The reaper now walks the pool in HSCAN batches and deletes in bulk, yielding between batches, and no longer sleeps past half the lock TTL, which previously guaranteed a failed renew every cycle. The per-event reads are gone: Socket.IO events only reach the worker holding the connection, and that worker already saved the same session dict locally when the user authenticated, so it was asking Redis for its own data. The writes stay, since those are what other pods read.
Measured at 4000 users / 16 containers, Redis 1.1 ms away:
| | before | after |
|---|---|---|
| reaper sweep, 5k sessions | 5.6 s, loop frozen throughout | 62 ms, 4.2 ms worst block |
| same, crash recovery with every session expired | 11.5 s | 96 ms |
| heartbeat / usage ping / disconnect | 2 / 3 / 2 round trips | 1 / 2 / 1 |
| 50-member channel post | 50 round trips, 57.2 ms block | 0 round trips, 0.02 ms |
| loop time per wall second at rest | 103 ms (10.3%) | 67 ms (6.7%) |
The alternative, converting RedisDict to the async client, fixes the same paths with a far larger blast radius (every call site gains await, and `in`/`[]`/`del` cannot be awaited so the dict interface goes) and still round-trips for data already in memory. Two deliberate behaviour changes: a heartbeat re-adds a session the reaper already removed, so a tab that survives a stall recovers instead of staying out of the pool until it reconnects; and disconnect no longer skips Yjs document cleanup when the pool entry is already gone, which previously leaked that document's update log forever.
Closes#28172
* perf: cut disconnect and user session lookup pool round trips, harden the session reaper
Follow-up on top of the session pool reaper branch. With WEBSOCKET_MANAGER=redis two paths still blocked the worker's event loop on synchronous Redis calls. Every disconnect listed all models in use cluster-wide and fetched each one individually, one blocking round trip per model. Disconnecting all sessions of a user (admin role change or deletion) pulled the entire session pool in one HGETALL and decoded every entry in a single uninterrupted block.
Disconnect now fetches the usage pool once with items(), going from 2+N+M round trips to 2+M (N models in use cluster-wide, M models the session used), and its delete of an emptied model entry is KeyError-guarded because another node can remove the same key between snapshot and delete; unguarded, that race aborted the handler and skipped its Yjs document cleanup. The user session lookup reuses the reaper's HSCAN batches and yields to the loop between pages. The reaper previously died permanently on the first Redis connection error, on every node at once during an outage; it now logs, releases the lock and returns to retrying acquisition.
* refac: keep the socket pool perf work to the round trips
A review pass on this branch turned up four changes riding along with the round-trip work without belonging to it, so they are backed out here. The `disconnect` handler keeps its `if sid in SESSION_POOL:` guard, so USAGE_POOL and ydoc cleanup stay off the path for sockets that never authenticated. `RedisDict.set()` keeps its own inline HDEL. `get_session_ids_by_user_id` stays synchronous over one HGETALL, since it runs on user delete and role change rather than per message. The crash-resilience wrapper around the reaper loop is dropped; if that guard is worth having, it belongs in its own change.
What stays is the perf part. The reaper now sweeps the pool in bounded HSCAN batches and deletes expired sids with one HDEL per batch, down from HKEYS plus an HGET and a per-sid HDEL across the whole pool. The `disconnect` handler reads USAGE_POOL with a single HGETALL, down from HKEYS plus one HGET per model in use. Session lookups in the socket handlers come from the local Socket.IO store, which removes one Redis GET from every heartbeat, usage, channel and ydoc event.
Naming and annotations follow the file: `get_session_pool_batches` for the module's `get_` prefix, `RedisDict.pop_many` so both reaper branches use one word for removing keys, a named `SCAN_BATCH_SIZE`, and types on the new helpers.
* fix: invalidate the RedisDict write signature on batch delete
RedisDict.set() skips the write when the payload fingerprint matches the last one this process wrote, so a mutation that goes around set() has to clear that fingerprint. The new batch delete did not, leaving a stale fingerprint behind: the next refresh with identical content is treated as already written and silently skipped, so the hash stays empty.
Renamed pop_many to delete_many. In a dict emulation pop removes and returns; this returns nothing and cannot without an extra HMGET, so the name promised something it does not do. delete_many matches __delitem__ and the HDEL underneath. Its only call site is the session pool reaper, whose behaviour is unchanged: same fields deleted, same batching, same return.
process_pipeline_inlet_filter() and its outlet counterpart construct and tear
down an aiohttp ClientSession, with its own connector and cookie jar, on
every chat completion and every task generation request just to iterate an
empty filter list. On deployments without pipelines, which is the default,
that is wasted setup on every message.
Both functions now return the payload untouched before the session is
created when there is nothing to call. The per-call saving is small, a few
microseconds of object construction per request on the pinned aiohttp; the
point is that requests stop paying setup for a feature that is not
configured.
Some providers send one very large piece of a streamed answer in a single go: a long reasoning trace, a code execution result, a turn with many tool calls, or a response echo carrying a big tool list. Anything past 128 KB in one line killed the chat mid-answer with a misleading `400, message: Got more than 131072 bytes when reading`. Nothing was rejected upstream, that is our own reader giving up on an oversized line.
Open WebUI already had code that assembles lines itself with no such limit, but it only ran when CHAT_STREAM_RESPONSE_CHUNK_MAX_BUFFER_SIZE was set. Unset is the default, and in that case the raw capped reader was used instead, so a default install always broke. That path now always assembles lines, and the setting goes back to being what its name says: an optional cap, off by default. It applies to the Ollama stream as well, since both now share the same reader.
The assembly loop only splits once a line actually completes, because the old one re-concatenated and re-split the whole buffer on every network chunk. Without that, allowing long lines would have traded an error for multi-second event loop stalls.
| | 20 MB in one line | 200k small lines |
| --- | --- | --- |
| before | 4249 ms | 27.3 ms |
| after | 37 ms | 25.2 ms |
Deleting an external knowledge base now clears its connection only when an admin removes the last knowledge base referencing it, matching the connection delete route.
