open-webui/backend/open_webui
Classic298 afda094544
fix: stop base64 images in tool results from reaching the model as text (#29665)
When a tool returns a base64 image, Open WebUI only moves it out of the model's context if the entire result is that image, or if the tool is MCP. An image sitting inside a returned object or a list was serialised into the tool message instead, so a single screenshot could cost hundreds of thousands of tokens, push out the rest of the conversation and leave the model answering from garbage.

Any string in a tool result that is exactly one image data URI is now moved into the result's files wherever it sits in the structure, and replaced with a short marker. The model gets the image as an attachment rather than as text, and it renders for the user instead of being dropped.

Detection is deliberately limited to values that are entirely a data URI. Scanning inside longer strings was tried and abandoned: a payload wrapped over several lines, or one followed by text, gets cut short, and shipping a truncated image is worse than the bloat because the provider rejects the whole request.

This also fixes the OpenAPI branch removing entries from the list it was iterating over, which skipped every second data URI and left it in the model's context.

Fixes #29208
2026-09-12 17:15:05 -05:00
..
data refac: mv backend files to /open_webui dir 2024-09-04 16:54:48 +02:00
internal refac 2026-09-06 17:13:32 -04:00
migrations chore: format 2026-08-25 16:53:53 -04:00
models refac 2026-09-09 17:09:53 -04:00
retrieval fix: let validate_url accept dotless hosts when local web fetch is enabled (#29945) 2026-09-12 17:56:16 -04:00
routers fix: honor ENABLE_PROFILE_IMAGE_URL_FORWARDING for channel webhook profile images (#29889) 2026-09-12 16:06:34 -05:00
socket perf: stop scanning every socket.io payload for binary data (#28180) 2026-08-31 01:22:06 -04:00
static refac 2026-09-06 17:27:30 -04:00
storage refac 2026-09-06 17:13:32 -04:00
tools feat: sort flags for kb_exec file listings (#29840) 2026-09-09 12:30:47 -04:00
utils fix: stop base64 images in tool results from reaching the model as text (#29665) 2026-09-12 17:15:05 -05:00
__init__.py perf: allow disabling websocket per-message-deflate (#28613) 2026-08-24 18:46:07 -04:00
alembic.ini fix: Alembic CLI commands from failing 2025-08-15 04:17:47 -04:00
config.py refac 2026-09-08 23:18:48 -04:00
constants.py refac 2026-08-29 16:14:32 -04:00
env.py refac 2026-09-09 12:51:59 -04:00
events.py refac 2026-08-16 23:21:00 -07:00
functions.py fix: honor bypass_system_prompt on the pipe route (#28739) 2026-08-19 11:08:02 -07:00
main.py refac 2026-09-06 19:36:17 -04:00
tasks.py fix: stop every task of a chat when stopping a response without Redis (#29844) 2026-09-12 13:55:13 -05:00