open-webui/backend
Classic298 afda094544
fix: stop base64 images in tool results from reaching the model as text (#29665)
When a tool returns a base64 image, Open WebUI only moves it out of the model's context if the entire result is that image, or if the tool is MCP. An image sitting inside a returned object or a list was serialised into the tool message instead, so a single screenshot could cost hundreds of thousands of tokens, push out the rest of the conversation and leave the model answering from garbage.

Any string in a tool result that is exactly one image data URI is now moved into the result's files wherever it sits in the structure, and replaced with a short marker. The model gets the image as an attachment rather than as text, and it renders for the user instead of being dropped.

Detection is deliberately limited to values that are entirely a data URI. Scanning inside longer strings was tried and abandoned: a payload wrapped over several lines, or one followed by text, gets cut short, and shipping a truncated image is worse than the bloat because the provider rejects the whole request.

This also fixes the OpenAPI branch removing entries from the list it was iterating over, which skipped every second data URI and left it in the model's context.

Fixes #29208
2026-09-12 17:15:05 -05:00
..
data refac: mv backend files to /open_webui dir 2024-09-04 16:54:48 +02:00
open_webui fix: stop base64 images in tool results from reaching the model as text (#29665) 2026-09-12 17:15:05 -05:00
.dockerignore fix: litellm config issue 2024-02-24 22:35:11 -08:00
.gitignore refac 2024-09-06 04:59:20 +02:00
dev.sh perf: allow disabling websocket per-message-deflate (#28613) 2026-08-24 18:46:07 -04:00
requirements-slim.txt refac 2026-09-06 17:35:49 -04:00
requirements.txt refac 2026-09-06 17:35:49 -04:00
start.sh refac 2026-09-06 16:48:30 -04:00
start_windows.bat chore: drop nltk, unused at the pinned versions (#29725) 2026-09-06 16:39:05 -04:00