scriptoriumadmin/open-webui
1
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-08-28 05:27:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

perf: stop recopying the streamed message text on every delta

Browse source 
Every streamed token appended to the response text through a dict slot, which allocates a new string and copies everything received so far. A long answer pays that copy thousands of times on the event loop, so every other request on the worker waits behind it.

The appends now clear the slot before growing the string, which leaves a single reference and lets CPython resize it in place. This covers the six accumulation points in the Chat Completions streaming path. The Responses API transport copies its items before appending, so it holds two references and is unchanged.

Measured on CPython 3.12 with 10-character deltas:

| accumulated text | before | after |
|---|---|---|
| 100 KB | 5.97 ms | 0.98 ms |
| 250 KB | 29.53 ms | 2.83 ms |
| 1 MB | 659 ms | 15.5 ms |

Results match the replaced expressions for every input where the key exists and holds a string, key order and exception type included. Buffering into a list and joining at the end was the alternative, but each delta is read back immediately by the stream save, and joining per read measured 666 ms at 250 KB.
This commit is contained in:
Classic298 2026-08-20 22:11:09 +02:00
parent 29e8d7db67
commit b2a9be27c5
1 changed files with 15 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
backend/open_webui/utils/middleware.py
View file

@ -303,6 +303,15 @@ def tool_result_content(tool_result: Any) -> str:
return str(tool_result)
def append_text(container: dict, key: str, value: str) -> None:
text = container.get(key, '')
container[key] = '' # dropping the dict's reference lets CPython extend the str in place
try:
text += value
finally: # a non-str value raises above and would leave the slot cleared
container[key] = text
def merge_streamed_reasoning_details(target: list, details) -> None:
items = details if isinstance(details, list) else [details]
for item in items:
@ -321,7 +330,7 @@ def merge_streamed_reasoning_details(target: list, details) -> None:
for key, value in item.items():
if key in ('text', 'summary') and isinstance(value, str) and isinstance(existing.get(key), str):
existing[key] += value
append_text(existing, key, value)
else:
existing[key] = value
@ -5136,7 +5145,7 @@ async def streaming_chat_response_handler(response, ctx):
# Append to reasoning content
parts = reasoning_item.get('content', [])
if parts and parts[-1].get('type') == 'output_text':
parts[-1]['text'] += reasoning_content
append_text(parts[-1], 'text', reasoning_content)
else:
reasoning_item['content'] = [
{
@ -5234,11 +5243,11 @@ async def streaming_chat_response_handler(response, ctx):
if inside_tag_block:
# Append to the existing tag-based item
if last_item_type == 'open_webui:code_interpreter':
last_item['code'] = last_item.get('code', '') + value
append_text(last_item, 'code', value)
elif last_item_type == 'reasoning':
parts = last_item.get('content', [])
if parts and parts[-1].get('type') == 'output_text':
parts[-1]['text'] += value
append_text(parts[-1], 'text', value)
else:
last_item['content'] = [
{
@ -5250,7 +5259,7 @@ async def streaming_chat_response_handler(response, ctx):
# solution or other _tag_type message
msg_parts = last_item.get('content', [])
if msg_parts and msg_parts[-1].get('type') == 'output_text':
msg_parts[-1]['text'] += value
append_text(msg_parts[-1], 'text', value)
else:
last_item['content'] = [
{
@ -5278,7 +5287,7 @@ async def streaming_chat_response_handler(response, ctx):
# Append value to last message item's text
msg_parts = output[-1].get('content', [])
if msg_parts and msg_parts[-1].get('type') == 'output_text':
msg_parts[-1]['text'] += value
append_text(msg_parts[-1], 'text', value)
else:
output[-1]['content'] = [
{