open-webui/backend/open_webui/utils
Classic298 afda094544
fix: stop base64 images in tool results from reaching the model as text (#29665)
When a tool returns a base64 image, Open WebUI only moves it out of the model's context if the entire result is that image, or if the tool is MCP. An image sitting inside a returned object or a list was serialised into the tool message instead, so a single screenshot could cost hundreds of thousands of tokens, push out the rest of the conversation and leave the model answering from garbage.

Any string in a tool result that is exactly one image data URI is now moved into the result's files wherever it sits in the structure, and replaced with a short marker. The model gets the image as an attachment rather than as text, and it renders for the user instead of being dropped.

Detection is deliberately limited to values that are entirely a data URI. Scanning inside longer strings was tried and abandoned: a payload wrapped over several lines, or one followed by text, gets cut short, and shipping a truncated image is worse than the bloat because the provider rejects the whole request.

This also fixes the OpenAPI branch removing entries from the list it was iterating over, which skipped every second data URI and left it in the model's context.

Fixes #29208
2026-09-12 17:15:05 -05:00
..
access_control refac 2026-09-07 12:40:44 -04:00
images perf: build debug log messages lazily so disabled debug logs cost nothing (#27834) 2026-07-31 19:09:01 -05:00
mcp refac 2026-09-12 14:57:22 -04:00
telemetry refac(telemetry): drop deprecated semconv SpanAttributes subclass (#25784) 2026-06-29 02:05:34 -05:00
actions.py fix: enforce action availability and model access on the chat action route (#27243) 2026-07-23 12:23:05 -04:00
anthropic.py fix: drop thinking blocks when converting Anthropic Messages requests to Chat Completions (#29849) 2026-09-12 15:56:59 -05:00
asgi_middleware.py refac 2026-08-24 18:29:36 -04:00
ask_user.py fix: a rejected ask_user call ending the turn with no reply (#29252) 2026-08-31 00:17:21 -04:00
audit.py perf: build debug log messages lazily so disabled debug logs cost nothing (#27834) 2026-07-31 19:09:01 -05:00
auth.py refac 2026-09-09 12:51:59 -04:00
automations.py refac 2026-09-12 16:56:26 -04:00
calendar.py refac 2026-08-29 16:14:32 -04:00
channels.py refac 2026-03-17 17:58:01 -05:00
chat.py fix: surface upstream errors on arena models instead of crashing (#29662) 2026-09-04 17:17:02 -04:00
chat_fork.py refac 2026-07-23 02:54:56 -04:00
chat_id.py refac 2026-07-26 21:12:14 -04:00
chat_variables.py perf: stabilize the model registry signature across workers (#29264) 2026-08-30 16:12:31 -04:00
code_interpreter.py refac 2026-07-31 17:41:14 -04:00
context_compaction.py refac 2026-08-23 13:36:53 -04:00
embeddings.py refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
files.py refac 2026-08-31 00:11:13 -04:00
filter.py refac 2026-08-31 01:29:36 -04:00
groups.py refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
headers.py refac 2026-09-12 15:41:57 -04:00
json_codec.py perf: write task payloads to Redis as bytes (#28833) 2026-08-20 12:58:52 -07:00
json_response.py perf: optional orjson JSON codec behind ENABLE_ORJSON (#27583) 2026-07-27 03:45:37 -04:00
logger.py perf: stop formatting every exported log record twice under OTEL log export (#27840) 2026-08-10 23:13:52 -06:00
memory.py refac 2026-08-10 19:28:21 -06:00
middleware.py fix: stop base64 images in tool results from reaching the model as text (#29665) 2026-09-12 17:15:05 -05:00
misc.py refac 2026-08-31 00:39:16 -04:00
model_ids.py refac 2026-07-26 23:09:22 -04:00
models.py refac 2026-09-07 12:40:44 -04:00
notifications.py fix: sanitize user-typed notification target ids the same way generated ones are (#29947) 2026-09-12 13:50:19 -05:00
oauth.py refac 2026-09-08 12:38:07 -04:00
payload.py refac 2026-08-16 22:56:19 -07:00
plugin.py perf: build info log messages lazily so raising the log level actually saves work (#27837) 2026-08-02 15:39:10 -05:00
rate_limit.py refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
redis.py fix: Set default Redis socket timeout to None (#27104) 2026-07-27 00:30:00 -04:00
response.py refac 2026-08-17 00:57:57 -07:00
sanitize.py refac 2026-03-17 17:58:01 -05:00
security_headers.py refac 2026-08-24 18:29:36 -04:00
session_pool.py fix: long streamed lines no longer abort the response (#28114) 2026-08-25 12:16:37 -04:00
subagents.py chore: format 2026-08-25 16:53:53 -04:00
task.py refac 2026-06-19 00:16:06 +02:00
terminals.py refac 2026-08-19 22:48:32 -07:00
timers.py fix: index the chat queries that make large SQLite instances unusable (#27663) 2026-08-23 16:11:54 -05:00
tool_approval.py chore: format 2026-08-25 16:53:53 -04:00
tools.py refac 2026-09-07 12:16:09 -04:00
validate.py refac 2026-07-27 19:39:36 -04:00
valves.py refac 2026-07-31 17:41:14 -04:00
webhook.py perf: build debug log messages lazily so disabled debug logs cost nothing (#27834) 2026-07-31 19:09:01 -05:00