Critical: the seq counter was emitter-local and reset to 0 whenever a
new emitter was created for the same message_id (crash-retry,
continuation, etc.). The client's dedupe guard drops any frame with
seq <= lastSeq, so a second emitter starting at 1 after the first hit
120 meant every live frame from the retry was silently dropped AND
the replay filter (seq > after_seq) excluded them too. Exactly the
failure scenarios this feature is for.
Fix: seed the counter from the log's existing max seq on emitter
construction via a new _stream_log_max_seq (XREVRANGE + COUNT 1 —
cheap tail read). Retries now continue the sequence instead of
colliding with it.
Warning: _stream_log_truncate became dead code after the switch to
TTL-shortening at done. Removed — if destructive reset is ever
needed again it can come back with a specific call site.
Suggestion: _stream_log_append / _stream_log_read / done-TTL-shorten
upgraded from log.debug to log.warning. Silent Redis failures in
debug-level were effectively invisible in prod; warnings surface the
outage at the right severity without being noisy when Redis is
healthy (these paths only log on exceptions).
Two review findings:
1. clearAllResumeFences was firing unawaited async flushes from
lifecycle transitions (disconnect, initNewChat, loadChat). That
undermined the ordering guarantees the fence exists to provide,
because the flushed events could land on a component that had
already moved to a different chat or connection state.
Split into two explicit verbs:
- dropResumeFence / dropAllResumeFences — synchronous, no flush.
Used in lifecycle transitions where the buffered events refer
to state that is about to become stale.
- clearResumeFence — async, flushes before dropping. Used by the
replay-ack handler (happy path) and the fence timeout (safety).
2. Unconditional _stream_log_truncate at emitter creation could wipe
an actively-streaming log if two emitters happened to overlap for
the same (user_id, message_id). Removed the truncate entirely and
switched XADD from explicit `0-{seq}` IDs to Redis-generated IDs,
so overlapping emitters cannot collide on stream IDs regardless.
seq now lives as a field on each entry and _stream_log_read filters
by it in Python (full scan bounded by MAXLEN=2000, a few ms worst
case, cost irrelevant for a user-driven event).
Suggestion on replay payload size deferred: in practice resumes are
tiny (handful of frames during a brief disconnect) and MAXLEN already
caps the worst case at ~1MB. Chunking would add protocol complexity
for a ceiling that isn't being hit. Easy to add later if telemetry
shows real reconnect-storm spikes.
Three correctness fixes from review:
Critical: in non-Redis deployments the old resume_stream handler
returned early without clearing the client fence, so any call to
requestResumeForMessage would buffer live frames forever and freeze
the UI. Server now always emits exactly one reply (with an empty
envelope list when Redis is disabled or the log is empty), so the
client's fence always clears regardless of backend configuration.
Warning: a lost ack — whether from a transient disconnect between
emit and reply, a server-side exception, a not-yet-registered client
handler, or anything else — would deadlock a message's UI updates
indefinitely. Added:
- 10s per-message fence timeout that clears the fence and flushes
buffered frames if no reply arrives
- disconnect listener that drops all fences on socket loss so the
reconnect-triggered fresh resume doesn't inherit a stale timer
- try/finally around replay application so a malformed envelope
can't skip the fence clear
Suggestion: replay is now a single batch emit instead of N sequential
emits. The server bundles all envelopes (possibly zero) into one
`resume-stream:replay` message; the client applies them in order and
immediately flushes the live-frame fence. Collapses "replay stream
+ completion ack" into a single round-trip, removes the per-envelope
emit loop latency, and simplifies the state machine.
Critical: replay and live frames can interleave on the client.
Previously the seq dedupe guard would drop replay frames whose seq was
already covered by a racing live frame that arrived first, permanently
losing the content between last_seq and the racing live seq.
Fix with a replay fence scoped per message:
- Server tags replay frames with `_replayed: true` and emits a
`resume-stream:complete` event after the last one.
- Client sets a fence (empty queue) BEFORE emitting resume-stream,
so any live frame arriving while the server reads XRANGE is
buffered — not applied, not advancing lastSeq.
- Replay frames (`_replayed: true`) bypass the fence, apply in order,
advance lastSeq.
- On `resume-stream:complete`, client flushes the buffered live
frames in seq order. The dedupe guard naturally drops any that were
already covered by replay.
Warning: removed the `taskIds && taskIds.length > 0` gate from the
loadChat resume trigger. getTaskIdsByChatId is not an authoritative
source of "should we resume" — it can fail, race, or return stale
data, which would leave in-progress assistant messages unrecovered.
requestResumeForAllInProgress already filters to assistants with
done !== true, and the server replay is a no-op when no log exists,
so the gate was only adding fragility.
Comment cleanup: cut the essay-length explanations on the resume-stream
feature down to single-line navigation aids. Logic is unchanged by the
tone-down itself.
Finding 1 (resume depended on DB presence of in-flight message):
scoped the Redis stream key by user_id — `stream:{user_id}:{message_id}`
instead of `stream:{message_id}`. An authenticated session can now only
ever read its own user's logs, so the separate chat-ownership +
message-in-chat DB lookups are no longer needed for auth. This also
fixes the case the review flagged: if the assistant stub has not yet
been persisted to the DB at refresh time (the exact scenario
ENABLE_REALTIME_CHAT_SAVE=False exposes), the prior message-in-chat
check would reject resume; now it doesn't apply. Frontend drops the
now-unused chat_id from the resume payload.
Finding 2 (resumeSeqByMessageId accumulated on non-done terminal paths):
prune the map in every terminal transition, not just the happy-path
chatCompletionEventHandler done branch:
- chat:tasks:cancel (both same-message and sibling arms)
- the HTTP error-finalization path (responseMessage.done + error)
- the generic error handler that marks all siblings done
Long-lived sessions with cancelled/errored generations no longer leak
stale seq entries.
The background asyncio task scheduled in the previous fix had a race:
if a second emitter for the same message_id started within the 30s
grace window, the pending DELETE would fire mid-stream and wipe the
new run's log.
Switch to shortening the key's TTL instead. EXPIRE is race-free —
a second emitter's eager truncate at creation resets the key and its
subsequent EXPIRE calls in _stream_log_append extend the TTL normally.
Same 30s post-done retention window, same clean-up behavior, no
background task to reason about.
Fixes a silent-correctness gap flagged in review: explicit stream IDs
of the form `0-{seq}` combined with a per-emitter seq counter that
resets to 0 mean a second emitter for the same message_id (continuation,
regeneration-into-same-id, or a retried producer after a crashed
worker) would try to XADD `0-1` against a stream whose top item is
`0-{N>1}`. Redis rejects the append, our try/except swallows it, and
resume logging silently degrades exactly in the flows where resume
matters most.
Fix: when get_event_emitter is constructed, await a _stream_log_truncate
for the message_id before any XADD. This guarantees our first XADD
(`0-1`) is accepted and that the log reflects only the current run,
not a mix of a crashed prior attempt and the retry.
A background _delayed_truncate task from a previously-completed run is
harmless here — it fires 30s after done:True on the OLD emitter, by
which time either (a) no new emitter has started, in which case the
delete is a legitimate cleanup, or (b) this new emitter has already
truncated + started appending, in which case the delayed delete racing
with the new run could wipe live data. To rule that out, the eager
truncate at emitter start supersedes any pending delayed truncate for
the same key; the next XADD then resets the stream, and when the new
run's delayed truncate eventually fires, it just repeats the cleanup.
Three review findings, all valid:
1. Race: log-then-emit instead of emit-then-log
The previous ordering emitted the live WS frame first and appended
to Redis afterward. A client disconnecting in that window and
reconnecting before the append completed would issue resume-stream,
see nothing newer, and never ask again — permanently losing that
frame (most painfully the terminal done:True frame). Inverted the
order: the log is now the source of truth for what was "sent", and
the live emit follows. A reconnect that races the append now may
see a replayed frame before the live frame arrives in the new
session, but the seq idempotency guard in chatEventHandler drops
the duplicate harmlessly. Duplicates are safe; losses are not.
2. Resume reads no longer full-scan
XADD now uses an explicit stream ID of `0-{seq}` (the per-emitter
seq counter guarantees strict monotonicity), so _stream_log_read
can start XRANGE at `0-{after_seq+1}` and let Redis skip entries
already delivered. Near-tail resumes (the common case) go from
O(MAXLEN) to O(missed frames). Dropped the now-redundant seq field
from the stream entries and the Python-side seq filter — the
envelope still carries seq internally for client-side idempotency.
3. Client map is now pruned
resumeSeqByMessageId was growing unboundedly across the session.
Now:
- cleared on loadChat (chat switch)
- cleared on initNewChat (new chat)
- individual entries deleted in chatCompletionEventHandler's
done:true branch (most messages evict quickly this way)
Long-lived sessions no longer accumulate dead keys for every
completed message they've ever seen.
Two review findings:
1. Backend: the `done:True` detection in get_event_emitter called
`.get('done')` on whatever `event_data['data']` happened to be. For
most event types that's a dict, but some custom/pipeline events can
legitimately emit non-dict inner payloads (list/string/None), which
would raise AttributeError and break emission for that event. Now
narrowed properly:
inner = event_data.get('data') if isinstance(event_data, dict) else None
if isinstance(inner, dict) and inner.get('done') is True:
...
2. Frontend: previously stored `message.lastSeq = incomingSeq` directly
on the message object, which is part of `history` and gets serialized
by saveChatHandler. That leaked transport-level resume metadata into
persisted chat state. Moved bookkeeping into a component-local
`resumeSeqByMessageId: Map<string, number>` so it stays in memory
only and never touches the persisted schema. All reads/writes of
lastSeq now go through the map.
- Critical (auth): the resume-stream handler verified chat ownership but
not that the requested message_id actually lives in that chat. Because
the Redis log is keyed by message_id alone, an attacker who learned a
victim's message_id could pass one of their own chat_ids to satisfy
the ownership check and replay the victim's stream. Added an explicit
message-to-chat binding check via Chats.get_message_by_id_and_message_id
after the ownership check.
- Safety: reject non-dict payloads (`if not isinstance(data, dict)`) so
stray client input — string/list/null — doesn't raise AttributeError
on `data.get(...)`.
- Perf: the per-token log append was doing two Redis round-trips (XADD +
EXPIRE) on the streaming hot path. Collapse them into a single pipeline
execute (one RTT), and refresh the TTL only every 64 appends instead
of every append. With a 1h TTL that still leaves comfortable headroom
for even pathologically long responses without EXPIRE ever risking
mid-stream expiry.
- Protocol cleanup: removed the resume-stream:ack emission. The frontend
doesn't consume it and YAGNI — the seq idempotency guard in
chatEventHandler already delivers the observability (you can see
last_seq advance as replays arrive). Can be added back when a concrete
client-side use case appears.
Adds a bounded Redis stream log for every in-flight assistant message so
clients can reconnect mid-stream (page refresh, network drop, device
switch) and catch up on frames they missed without re-fetching the full
chat from the database.
Problem this solves
-------------------
With ENABLE_REALTIME_CHAT_SAVE=False (default), the backend does not
write the assistant message to the DB until the stream finishes. If the
client refreshes the page mid-stream, chat load from the DB returns
nothing for the in-progress message and the response appears to vanish
until the stream eventually completes. Users staring at an empty chat
while the backend quietly keeps emitting tokens into the void.
Design
------
* Every outbound WS envelope gets stamped with a monotonic per-message
`seq` inside `get_event_emitter` and appended to a bounded Redis
stream keyed `{REDIS_KEY_PREFIX}:stream:{message_id}`.
MAXLEN ~ 2000 entries, TTL 1h as a safety net.
* Clients track `message.lastSeq` in Chat.svelte as events arrive. On
chat load (mid-stream refresh) and on socket reconnect they emit
`resume-stream {chat_id, message_id, last_seq}`.
* The server authenticates the session, verifies the user owns the
chat, XRANGEs the log, filters by `seq > last_seq`, and emits the
missed envelopes to THAT session only (via `to=sid`) so live listeners
in the user room keep receiving their normal live stream unchanged.
* The existing chat event handler drops any envelope with
`seq <= message.lastSeq`, making replay idempotent against live
frames that race the replay after a reconnect.
* When an event with `done: True` fires, a background task truncates
the log after a 30s grace window so late reconnects still catch the
finalization; anything beyond that resumes from the now-up-to-date DB.
Orthogonality
-------------
Zero touches to middleware.py or the streaming hot path. The log stores
whatever gets emitted; any future change to emit shape
(chat:message:delta, per-block ops, JSON Patch, ...) is logged and
replayed verbatim with no coupling.
Graceful degradation
--------------------
No-op when Redis is not configured (WEBSOCKET_MANAGER != 'redis'). In
that deployment mode, refresh during streaming retains the current
behavior of waiting for the stream to complete.
Auth model
----------
The log is keyed by message_id only. The resume handler must do a chat
ownership check (Chats.get_chat_by_id_and_user_id) before replaying, so
a malicious client cannot read another user's stream by guessing a
message_id.
SESSION_POOL caches user.role at connection time and never refreshes it. When an admin demotes or deletes a user, their socket sessions retain the old cached role until voluntary disconnect, allowing continued use of admin-gated socket features (ydoc editing, channel access).
Adds disconnect_user_sessions() helper that disconnects all sockets for a user ID. Called from update_user_by_id (on role change) and delete_user_by_id. The client auto-reconnects and re-authenticates with fresh DB data.
Replace bare except clauses with except Exception to follow Python best practices and avoid catching unexpected system exceptions like KeyboardInterrupt and SystemExit.
Three improvements to the socket event emitter hot path (when realtime chat save is enabled):
1. Wrap all synchronous Chats.* DB calls in asyncio.to_thread() to avoid blocking the event loop during streaming. With N concurrent users, sync DB calls serialize all writes and block socket event delivery.
2. Only persist final (done=True) status events to DB. Intermediate statuses (tool calling progress, web search progress, etc.) are ephemeral UI-only data already delivered via socket — writing every one to DB is unnecessary I/O.
3. Convert if/if/if chain to if/elif since event types are mutually exclusive, avoiding unnecessary string comparisons after a match.
Add null check in list comprehension before accessing session['id']. When a session_id exists in the room but has been removed from SESSION_POOL, the function now skips it instead of crashing with TypeError.
* This PR optimizes socket delta event broadcasting by leveraging rooms. Instead of iterating through a user's sessions and emitting events individually, this change sends a single event to a user-specific room. This approach is more efficient, reducing overhead and improving performance, particularly for users with multiple concurrent sessions.
In testing this dramatically reduces emits and server load.
* Update main.py
Added userroom join
---------
Co-authored-by: Tim Baek <tim@openwebui.com>
