diff --git a/docs/en/auto_resource.md b/docs/en/auto_resource.md index 2f8feef9..18a5a3e2 100644 --- a/docs/en/auto_resource.md +++ b/docs/en/auto_resource.md @@ -36,7 +36,8 @@ In short, it turns "a file was archived" into "the resource is usable." Auto Resource uses `resource/` as the entry point for source material. Date directories are recommended, and their date determines which daily memory layer receives the interpreted card. A file directly under `resource/` is also supported -and uses today in the application timezone. +and uses today in the application timezone when it is first processed. On later days, an exact `source_resource` match +keeps updates and deletion tied to that original daily card instead of creating a new card or leaving an orphan. Example directory: @@ -65,10 +66,14 @@ The card body starts with an `![[resource/...]]` embed link and the frontmatter so text search reaches image content through the caption. The vision model is the `vision` instance of `as_llm` when configured, and otherwise falls back to the `default` -instance — a multimodal default model needs no extra configuration. Images larger than the request budget or in -provider-unfriendly formats are downscaled or re-encoded in memory for the request only; the original file under -`resource/` is never modified. When an image changes, its card is rewritten in place; when the image is deleted, the -card is removed with it. +instance — a multimodal default model needs no extra configuration. Images wider or taller than 2048px are downscaled, +and provider-unfriendly formats are re-encoded, in memory for the request only; the original file under +`resource/` is never modified. Before a full decode, image dimensions are checked against a default limit of 40,000,000 +pixels; images over the limit and Pillow decompression-bomb warnings fail only that resource. EXIF orientation is +applied to the in-memory request copy before resizing or conversion. Oversized JPEGs first use decoder-level +downsampling, followed by a final thumbnail pass when needed. The VLM request MIME and the card's frontmatter +`media_type` use the format Pillow detects from the image bytes, rather than trusting the filename extension. When an +image changes, its card is rewritten in place; when the image is deleted, the card is removed with it. Image preprocessing uses Pillow from the `core` extra. HEIC resources additionally require the optional `image-heif` extra: `pip install "reme-ai[image-heif]"`. Other supported image formats do not load or require the HEIF diff --git a/docs/en/quick_start.md b/docs/en/quick_start.md index a15edef2..01b7714d 100644 --- a/docs/en/quick_start.md +++ b/docs/en/quick_start.md @@ -185,8 +185,8 @@ reme auto_memory \ ``` After placing external material under `resource/YYYY-MM-DD/` or directly under `resource/`, the default background task -watches -`md/txt/json/jsonl/csv/yaml/html`. You can also trigger processing manually: +watches text resources (`md/txt/json/jsonl/csv/yaml/html`) and image resources +(`png/jpg/jpeg/webp/gif/bmp/tiff/heic`). You can also trigger processing manually: ```bash reme auto_resource changes='[{"path":"resource/2026-06-20/report.md","change":"added"}]' diff --git a/docs/zh/auto_resource.md b/docs/zh/auto_resource.md index 5cb583b9..23fa0d24 100644 --- a/docs/zh/auto_resource.md +++ b/docs/zh/auto_resource.md @@ -33,7 +33,8 @@ resource/[YYYY-MM-DD/] ## 原始资料入口 Auto Resource 以 `resource/` 作为原始资料入口。推荐按日期放置，目录日期会决定它进入哪一天的 daily 记忆层；也支持直接放在 -`resource/` 根目录，此时使用应用时区中的今天。 +`resource/` 根目录，首次处理时使用应用时区中的今天。之后即使跨天更新或删除，也会通过精确匹配 +`source_resource` 继续操作原 daily 卡片，不会重复新建卡片或留下孤立链接。 示例目录： @@ -56,7 +57,13 @@ workspace/ 图像文件的解读方式相同：视觉模型写入一张 caption 卡片并链接原图。卡片正文以 `![[resource/...]]` 嵌入链接开头，frontmatter 携带 `kind: image` 与 `media_type`，文本检索因此可以通过 caption 命中图像内容。 -视觉模型优先使用配置中的 `as_llm` `vision` 实例，未配置时回退到 `default` 实例——默认模型具备视觉能力时无需额外配置。超过请求预算或格式不被模型接受的图像，仅在请求前于内存中降采样或转码；`resource/` 下的原图文件不会被修改。图像变更时卡片原地重写；图像删除时卡片随之删除。 +视觉模型优先使用配置中的 `as_llm` `vision` 实例，未配置时回退到 `default` 实例——默认模型具备视觉能力时无需额外配置。宽或高超过 2048px 的图像会降采样，格式不被模型接受的图像会转码；这些处理只发生在请求前的内存副本中，`resource/` 下的原图文件不会被修改。图像变更时卡片原地重写；图像删除时卡片随之删除。 + +在完整解码前，系统会检查图像尺寸，默认上限为 40,000,000 像素；超限图像或 Pillow +decompression-bomb 警告只会导致当前资源失败。缩放或转码前，会按 EXIF orientation 校正仅用于请求的内存副本。 +尺寸过大的 JPEG 会先使用 decoder-level downsampling，并在需要时再完成最终缩放。 +VLM 请求的 MIME 和卡片 frontmatter 中的 `media_type` 都使用 Pillow 根据实际图像字节识别的格式， +而不是直接信任文件扩展名。 图像预处理使用 `core` extra 中的 Pillow。HEIC 资源还需要可选的 `image-heif` extra： `pip install "reme-ai[image-heif]"`。其他受支持图像格式不会加载或依赖 HEIF 插件。 diff --git a/docs/zh/quick_start.md b/docs/zh/quick_start.md index ee41626a..f8f14340 100644 --- a/docs/zh/quick_start.md +++ b/docs/zh/quick_start.md @@ -178,7 +178,8 @@ reme auto_memory \ memory_hint="记录用户偏好" ``` -外部资料放入 `resource/YYYY-MM-DD/` 或直接放在 `resource/` 下后，默认后台会监听 `md/txt/json/jsonl/csv/yaml/html`。 +外部资料放入 `resource/YYYY-MM-DD/` 或直接放在 `resource/` 下后，默认后台会监听文本资源 +(`md/txt/json/jsonl/csv/yaml/html`) 和图像资源 (`png/jpg/jpeg/webp/gif/bmp/tiff/heic`)。 也可以手动触发： ```bash diff --git a/reme/steps/evolve/auto_image_resource.py b/reme/steps/evolve/auto_image_resource.py index 8e4593a4..6180e71a 100644 --- a/reme/steps/evolve/auto_image_resource.py +++ b/reme/steps/evolve/auto_image_resource.py @@ -4,6 +4,7 @@ import base64 import io import json import re +import warnings from pathlib import Path, PurePosixPath import aiofiles @@ -11,17 +12,22 @@ from agentscope.message import Base64Source, DataBlock, TextBlock, UserMsg from agentscope.model import ChatModelBase from pydantic import BaseModel, Field -from ..file_io._path import IMAGE_MIME_BY_EXT, IMAGE_SUFFIXES +from ..file_io._path import IMAGE_SUFFIXES from .base_auto_resource import _SOURCE_RESOURCE_KEY, _sanitize_note_name, BaseAutoResourceStep from ...components import R from ...enumeration import ComponentEnum DEFAULT_MAX_IMAGE_INPUT_BYTES = 50 * 1024 * 1024 +DEFAULT_MAX_IMAGE_PIXELS = 40_000_000 MAX_IMAGE_REQUEST_DIMENSION = 2048 _JPEG_QUALITY = 85 -# Suffixes re-encoded to provider-friendly PNG/JPEG for VLM requests; -# the stored resource file is never modified. -_CONVERT_SUFFIXES = {".bmp", ".tiff", ".heic"} +# Decoded formats outside this set are re-encoded to provider-friendly +# PNG/JPEG for VLM requests; the stored resource file is never modified. +_PASSTHROUGH_IMAGE_MIMES = frozenset({"image/png", "image/jpeg", "image/webp", "image/gif"}) +_HEIF_BRANDS = frozenset( + {b"heic", b"heif", b"heix", b"heim", b"heis", b"hevc", b"hevx", b"hevm", b"hevs", b"mif1", b"msf1"}, +) +_MAX_FTYP_SCAN_BYTES = 4096 _JSON_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*(.*?)\s*```\s*$", re.DOTALL) @@ -37,84 +43,170 @@ class _CaptionOutput(BaseModel): ) -def _load_pillow(suffix: str): - """Load Pillow and optional HEIC support only when image processing runs.""" +def _load_pillow(): + """Load the core image dependency only when image processing runs.""" try: - from PIL import Image # pylint: disable=import-outside-toplevel + from PIL import Image, ImageOps # pylint: disable=import-outside-toplevel except ImportError as exc: raise RuntimeError("Image captioning requires Pillow; install reme-ai[core]") from exc - - if suffix == ".heic": - try: - from pillow_heif import register_heif_opener # pylint: disable=import-outside-toplevel - except ImportError as exc: - raise RuntimeError("HEIC image captioning requires pillow-heif; install reme-ai[image-heif]") from exc - try: - register_heif_opener() - except Exception as exc: # pylint: disable=broad-except - raise RuntimeError(f"Failed to initialize HEIC image support: {exc}") from exc - return Image + return Image, ImageOps -def _normalize_image_bytes(data: bytes, suffix: str) -> tuple[bytes, str] | None: - """Downscale or re-encode image bytes in memory for a VLM request. +def _looks_like_heif(data: bytes) -> bool: + """Return whether an ISO-BMFF header declares a HEIC/HEIF brand.""" + if len(data) < 12 or data[4:8] != b"ftyp": + return False + box_size = int.from_bytes(data[:4], "big") + if box_size < 12: + return False + end = min(box_size, len(data), _MAX_FTYP_SCAN_BYTES) + if data[8:12] in _HEIF_BRANDS: + return True + return any(data[index : index + 4] in _HEIF_BRANDS for index in range(16, end - 3, 4)) - Returns ``None`` only when valid image bytes already have a suitable size - and format. Missing dependencies and decode/convert failures are explicit. - The stored resource file is never modified. - """ - image_module = _load_pillow(suffix) + +def _register_heif_opener() -> None: + """Load and register HEIC support only for bytes that declare HEIF.""" try: - image = image_module.open(io.BytesIO(data)) + from pillow_heif import register_heif_opener # pylint: disable=import-outside-toplevel + except ImportError as exc: + raise RuntimeError("HEIC image captioning requires pillow-heif; install reme-ai[image-heif]") from exc + try: + register_heif_opener() + except Exception as exc: # pylint: disable=broad-except + raise RuntimeError(f"Failed to initialize HEIC image support: {exc}") from exc + + +def _normalize_image_bytes( + data: bytes, + suffix: str, + *, + max_image_pixels: int = DEFAULT_MAX_IMAGE_PIXELS, +) -> tuple[bytes | None, str, str]: + """Validate and optionally normalize image bytes for a VLM request. + + The returned tuple is ``(normalized_bytes, request_mime, source_mime)``. + ``normalized_bytes`` is ``None`` only when the original bytes can be sent + unchanged. MIME values come from the decoded image rather than its suffix. + Missing dependencies, unsafe pixel counts, and decode/convert failures are + explicit. The stored resource file is never modified. + """ + try: + pixel_limit = int(max_image_pixels) + except (TypeError, ValueError) as exc: + raise ValueError(f"max_image_pixels must be a positive integer: {max_image_pixels!r}") from exc + if pixel_limit <= 0: + raise ValueError(f"max_image_pixels must be a positive integer: {max_image_pixels!r}") + + image_module, image_ops = _load_pillow() + if _looks_like_heif(data): + _register_heif_opener() + try: + with warnings.catch_warnings(): + warnings.simplefilter("error", image_module.DecompressionBombWarning) + image = image_module.open(io.BytesIO(data)) + except (image_module.DecompressionBombWarning, image_module.DecompressionBombError) as exc: + raise RuntimeError( + f"Image rejected by Pillow decompression-bomb protection ({suffix or 'unknown suffix'})", + ) from exc except Exception as exc: # pylint: disable=broad-except raise RuntimeError(f"Failed to decode image ({suffix or 'unknown suffix'}): {exc}") from exc with image: + width, height = image.size + pixel_count = width * height + if pixel_count > pixel_limit: + raise RuntimeError( + f"Image exceeds max_image_pixels before decode: " f"{width}x{height}={pixel_count} > {pixel_limit}", + ) + + source_mime = str(image.get_format_mimetype() or "").strip().lower() + if not source_mime.startswith("image/"): + raise RuntimeError( + f"Cannot determine decoded image MIME type ({suffix or 'unknown suffix'}, format={image.format!r})", + ) + + needs_resize = width > MAX_IMAGE_REQUEST_DIMENSION or height > MAX_IMAGE_REQUEST_DIMENSION + if source_mime == "image/jpeg" and needs_resize: + max_dimension = max(width, height) + decoder_size = ( + max(1, (width * MAX_IMAGE_REQUEST_DIMENSION + max_dimension - 1) // max_dimension), + max(1, (height * MAX_IMAGE_REQUEST_DIMENSION + max_dimension - 1) // max_dimension), + ) + try: + # JPEG supports power-of-two decoder scaling. ``draft`` picks + # the smallest decoded frame that still covers decoder_size, + # reducing peak memory before the final LANCZOS thumbnail. + image.draft(None, decoder_size) + except Exception as exc: # pylint: disable=broad-except + raise RuntimeError(f"Failed to prepare JPEG decoder downsampling ({width}x{height}): {exc}") from exc + try: - image.load() + with warnings.catch_warnings(): + warnings.simplefilter("error", image_module.DecompressionBombWarning) + image.load() + orientation = int(image.getexif().get(274, 1) or 1) + image_ops.exif_transpose(image, in_place=True) + except (image_module.DecompressionBombWarning, image_module.DecompressionBombError) as exc: + raise RuntimeError( + f"Image rejected by Pillow decompression-bomb protection ({suffix or 'unknown suffix'})", + ) from exc except Exception as exc: # pylint: disable=broad-except raise RuntimeError(f"Failed to decode image ({suffix or 'unknown suffix'}): {exc}") from exc - needs_resize = image.width > MAX_IMAGE_REQUEST_DIMENSION or image.height > MAX_IMAGE_REQUEST_DIMENSION - needs_convert = suffix in _CONVERT_SUFFIXES - if not needs_resize and not needs_convert: - return None + needs_convert = source_mime not in _PASSTHROUGH_IMAGE_MIMES + needs_orientation = orientation in range(2, 9) + if not needs_resize and not needs_convert and not needs_orientation: + return None, source_mime, source_mime try: + if needs_resize: + # Resize the decoded source before color conversion so large + # non-JPEG images do not require a second full-size frame. + # Pillow 10 cannot apply LANCZOS directly to 16-bit integer + # modes; NEAREST keeps that path bounded without a full-size + # RGB conversion first. + resize_filter = image_module.Resampling.LANCZOS + if image.mode.startswith("I;16"): + resize_filter = image_module.Resampling.NEAREST + image.thumbnail((MAX_IMAGE_REQUEST_DIMENSION, MAX_IMAGE_REQUEST_DIMENSION), resize_filter) has_alpha = image.mode in ("RGBA", "LA", "P") frame = image.convert("RGBA" if has_alpha else "RGB") - if needs_resize: - frame.thumbnail((MAX_IMAGE_REQUEST_DIMENSION, MAX_IMAGE_REQUEST_DIMENSION), image_module.LANCZOS) - buffer = io.BytesIO() - if frame.mode == "RGBA": - frame.save(buffer, format="PNG") - return buffer.getvalue(), "image/png" - frame.save(buffer, format="JPEG", quality=_JPEG_QUALITY) - return buffer.getvalue(), "image/jpeg" + try: + buffer = io.BytesIO() + if frame.mode == "RGBA": + frame.save(buffer, format="PNG") + return buffer.getvalue(), "image/png", source_mime + frame.save(buffer, format="JPEG", quality=_JPEG_QUALITY) + return buffer.getvalue(), "image/jpeg", source_mime + finally: + frame.close() except Exception as exc: # pylint: disable=broad-except raise RuntimeError(f"Failed to convert/resize image ({suffix or 'unknown suffix'}): {exc}") from exc -def _build_image_request_payload(data: bytes, suffix: str) -> dict: +def _build_image_request_payload( + data: bytes, + suffix: str, + *, + max_image_pixels: int = DEFAULT_MAX_IMAGE_PIXELS, +) -> dict: """Return ``{"data_b64", "mime", "source_mime", "converted"}`` for a VLM request. ``mime`` is the format actually sent (after in-memory downscale/re-encode); - ``source_mime`` describes the stored resource file and is what notes record. + ``source_mime`` is the decoded format of the stored resource file and is + what notes record. Both are based on actual bytes, not the filename suffix. """ - source_mime = IMAGE_MIME_BY_EXT.get(suffix, "image/png") - normalized = _normalize_image_bytes(data, suffix) - if normalized is None: - return { - "data_b64": base64.b64encode(data).decode("ascii"), - "mime": source_mime, - "source_mime": source_mime, - "converted": False, - } - normalized_bytes, mime = normalized + normalized_bytes, mime, source_mime = _normalize_image_bytes( + data, + suffix, + max_image_pixels=max_image_pixels, + ) + request_bytes = data if normalized_bytes is None else normalized_bytes return { - "data_b64": base64.b64encode(normalized_bytes).decode("ascii"), + "data_b64": base64.b64encode(request_bytes).decode("ascii"), "mime": mime, "source_mime": source_mime, - "converted": normalized_bytes != data, + "converted": normalized_bytes is not None, } @@ -193,7 +285,9 @@ class AutoImageResourceStep(BaseAutoResourceStep): """ resource_suffixes = IMAGE_SUFFIXES - router_inherit_keys = BaseAutoResourceStep.router_inherit_keys | frozenset({"as_llm", "max_image_bytes"}) + router_inherit_keys = BaseAutoResourceStep.router_inherit_keys | frozenset( + {"as_llm", "max_image_bytes", "max_image_pixels"}, + ) def _max_image_bytes(self) -> int: """Return the image read limit from Step or Job context.""" @@ -202,6 +296,17 @@ class AutoImageResourceStep(BaseAutoResourceStep): value = self.context.get("max_image_bytes") return int(value) if value is not None else DEFAULT_MAX_IMAGE_INPUT_BYTES + def _max_image_pixels(self) -> int: + """Return the deployment-controlled pre-decode pixel limit.""" + value = self.kwargs.get("max_image_pixels", DEFAULT_MAX_IMAGE_PIXELS) + try: + limit = int(value) + except (TypeError, ValueError) as exc: + raise ValueError(f"max_image_pixels must be a positive integer: {value!r}") from exc + if limit <= 0: + raise ValueError(f"max_image_pixels must be a positive integer: {value!r}") + return limit + def _vision_model(self) -> ChatModelBase | None: """Resolve explicit ``as_llm`` through Ref, otherwise prefer vision/default.""" context_model = self.context.get("as_llm") if self.context is not None else None @@ -266,37 +371,49 @@ class AutoImageResourceStep(BaseAutoResourceStep): self.logger.warning(f"[{self.name}] resource stat failed file_path={file_path} error={exc}") return None if size_bytes > max_image_bytes: - self.context.response.success = True - self.context.response.answer = ( - f"Skipped oversized image resource file: {file_path} ({size_bytes} > {max_image_bytes} bytes)" - ) - self.context.response.metadata.update( - { - "path": file_path, - "action": "skipped", - "reason": "file_too_large", - "oversized": True, - "size_bytes": size_bytes, - "max_image_bytes": max_image_bytes, - "modified": False, - }, - ) - self.logger.warning( - f"[{self.name}] skip oversized image resource file_path={file_path} " - f"size_bytes={size_bytes} max_image_bytes={max_image_bytes}", - ) + self._record_oversized_image(file_path, size_bytes, max_image_bytes) return None self.logger.info(f"[{self.name}] read image start file_path={file_path}") async with aiofiles.open(source_path, "rb") as f: - data = await f.read() - payload = _build_image_request_payload(data, Path(file_path).suffix.lower()) + data = await f.read(max_image_bytes + 1) + if len(data) > max_image_bytes: + self._record_oversized_image(file_path, len(data), max_image_bytes) + return None + payload = _build_image_request_payload( + data, + Path(file_path).suffix.lower(), + max_image_pixels=self._max_image_pixels(), + ) self.logger.info( f"[{self.name}] read image done file_path={file_path} size_bytes={size_bytes} " - f"mime={payload['mime']} converted={payload['converted']}", + f"mime={payload['mime']} source_mime={payload['source_mime']} converted={payload['converted']}", ) return payload + def _record_oversized_image(self, file_path: str, size_bytes: int, max_image_bytes: int) -> None: + """Record a stable skip response for an image over the compressed-byte limit.""" + assert self.context is not None + self.context.response.success = True + self.context.response.answer = ( + f"Skipped oversized image resource file: {file_path} ({size_bytes} > {max_image_bytes} bytes)" + ) + self.context.response.metadata.update( + { + "path": file_path, + "action": "skipped", + "reason": "file_too_large", + "oversized": True, + "size_bytes": size_bytes, + "max_image_bytes": max_image_bytes, + "modified": False, + }, + ) + self.logger.warning( + f"[{self.name}] skip oversized image resource file_path={file_path} " + f"size_bytes={size_bytes} max_image_bytes={max_image_bytes}", + ) + async def _handle_upsert( self, file_path: str, diff --git a/reme/steps/evolve/base_auto_resource.py b/reme/steps/evolve/base_auto_resource.py index 3ab8c66b..1020a183 100644 --- a/reme/steps/evolve/base_auto_resource.py +++ b/reme/steps/evolve/base_auto_resource.py @@ -236,6 +236,33 @@ class BaseAutoResourceStep(BaseStep): notes = list_response.metadata.get("notes") or [] return self._find_resource_note(notes, file_path) + def _daily_note_days(self) -> list[str]: + """Return safe, deterministic daily subdirectories available for lookup.""" + workspace = self.workspace_path.resolve() + daily_dir = str(self.config_value("daily_dir")) + daily_root, error = resolve_path(workspace, daily_dir) + if error or daily_root is None: + raise ValueError(f"invalid daily_dir {daily_dir!r}: {error or 'cannot resolve path'}") + if not daily_root.is_dir(): + return [] + return sorted( + entry.name + for entry in daily_root.iterdir() + if _DATE_RE.fullmatch(entry.name) and entry.is_dir() and not entry.is_symlink() + ) + + async def _find_loose_resource_day(self, file_path: str) -> str | None: + """Find the single daily-card owner for a root-level resource.""" + matches: list[tuple[str, str]] = [] + for day in self._daily_note_days(): + note = await self._list_resource_note(day, file_path) + if note is not None: + matches.append((day, str(note["path"]))) + if len(matches) > 1: + paths = ", ".join(path for _, path in matches) + raise RuntimeError(f"Multiple daily resource notes claim {file_path}: {paths}") + return matches[0][0] if matches else None + async def _prepare_resource_note(self, day: str, file_path: str, note_stem: str) -> _ResourceNoteState: """Find the owned note or allocate a safe path before the first write.""" note = await self._list_resource_note(day, file_path) @@ -492,8 +519,11 @@ class BaseAutoResourceStep(BaseStep): loose_filename = _loose_resource_filename(file_path, resource_dir) if loose_filename: - date_str, filename = self._today(), loose_filename - self.logger.info(f"[{self.name}] loose resource file_path={file_path} date={date_str}") + existing_day = await self._find_loose_resource_day(file_path) + date_str, filename = existing_day or self._today(), loose_filename + self.logger.info( + f"[{self.name}] loose resource file_path={file_path} date={date_str} existing={bool(existing_day)}", + ) else: date_str, filename = _parse_resource_path(file_path, resource_dir) diff --git a/tests/unit/auto_resource_test_support.py b/tests/unit/auto_resource_test_support.py index 7db8c4cd..3c11b8d8 100644 --- a/tests/unit/auto_resource_test_support.py +++ b/tests/unit/auto_resource_test_support.py @@ -210,13 +210,14 @@ def caption_json(name: str, description: str, caption: str) -> str: return json.dumps({"name": name, "description": description, "caption": caption}) -def image_processor(app_context, file_store, model, *, routed: bool): +def image_processor(app_context, file_store, model, *, routed: bool, **kwargs): """Build either the image processor or the public unified-router path.""" if not routed: - return AutoImageResourceStep(app_context=app_context, file_store=file_store, as_llm=model) + return AutoImageResourceStep(app_context=app_context, file_store=file_store, as_llm=model, **kwargs) app_context.registry = R return AutoResourceStep( app_context=app_context, + **kwargs, dispatch_steps=[ {"backend": "auto_image_resource_step", "file_store": file_store, "as_llm": model}, { @@ -244,9 +245,9 @@ class AutoResourceTestEnv: """Write a source-owned note relative to this workspace.""" return write_note(self.workspace / relative_path, source_resource, body) - def processor(self, model, *, routed: bool = False): + def processor(self, model, *, routed: bool = False, **kwargs): """Build the direct processor or unified router for this workspace.""" - return image_processor(self.app_context, self.file_store, model, routed=routed) + return image_processor(self.app_context, self.file_store, model, routed=routed, **kwargs) async def run(self, step, changes, **context_kwargs): """Run one processor invocation with a fresh runtime context.""" diff --git a/tests/unit/test_auto_image_steps.py b/tests/unit/test_auto_image_steps.py index 18fdd6ae..e7af5a96 100644 --- a/tests/unit/test_auto_image_steps.py +++ b/tests/unit/test_auto_image_steps.py @@ -9,9 +9,11 @@ with PIL inside a temporary workspace. import base64 import hashlib import io +import struct import subprocess import sys import tomllib +import zlib from pathlib import Path from types import SimpleNamespace from unittest.mock import patch @@ -19,7 +21,7 @@ from unittest.mock import patch import frontmatter import pytest import yaml -from PIL import Image +from PIL import Image, JpegImagePlugin from reme.components import R from reme.components.component_registry import ComponentRegistry @@ -28,6 +30,7 @@ from reme.components.runtime_context import RuntimeContext from reme.enumeration import ComponentEnum from reme.steps.evolve.auto_image_resource import ( AutoImageResourceStep, + DEFAULT_MAX_IMAGE_PIXELS, _build_image_request_payload, _normalize_image_bytes, _parse_caption_json, @@ -52,6 +55,14 @@ from .auto_resource_test_support import ( pytest_plugins = ("unit.auto_resource_test_plugin",) +def _png_bytes_with_header_size(width: int, height: int) -> bytes: + """Change only a tiny PNG's IHDR dimensions without allocating its pixels.""" + data = bytearray(_png_bytes()) + data[16:24] = struct.pack(">II", width, height) + data[29:33] = struct.pack(">I", zlib.crc32(data[12:29]) & 0xFFFFFFFF) + return bytes(data) + + @pytest.mark.parametrize( ("image_format", "suffix", "source_mime", "request_mime"), [ @@ -178,6 +189,197 @@ async def test_auto_image_downscales_oversized_image_for_request_only(auto_resou assert (env.workspace / "daily/2026-01-01/huge-image.md").is_file() +@pytest.mark.asyncio +async def test_auto_image_uses_jpeg_decoder_downsampling_before_load(auto_resource_env): + """Large JPEGs lower their decoder allocation before full pixel decode.""" + env = auto_resource_env + source = env.write_binary("resource/2026-01-01/large.jpg", _img_bytes("JPEG", (4096, 2048))) + stored_bytes = source.read_bytes() + model = _FakeVisionModel(_caption_json("large-jpeg", "Large", "A large JPEG.")) + decoded_sizes = [] + original_load = JpegImagePlugin.JpegImageFile.load + + def record_decoder_size(image, *args, **kwargs): + decoded_sizes.append(image.size) + return original_load(image, *args, **kwargs) + + with patch.object(JpegImagePlugin.JpegImageFile, "load", new=record_decoder_size): + response = await env.run(env.processor(model), [{"change": "added", "path": str(source)}]) + + assert response.success is True + assert decoded_sizes + assert decoded_sizes[0] == (2048, 1024) + data_block = model.calls[0][0].content[1] + with Image.open(io.BytesIO(base64.b64decode(data_block.source.data))) as sent: + assert max(sent.size) <= 2048 + assert source.read_bytes() == stored_bytes + + +def test_image_preprocessing_resizes_16_bit_tiff_before_rgb_conversion(): + """Pillow 10-compatible scaling keeps large I;16 TIFF images memory-bounded.""" + image = Image.new("I;16", (2049, 2), 1000) + buffer = io.BytesIO() + image.save(buffer, format="TIFF") + calls = [] + original_thumbnail = Image.Image.thumbnail + + def record_thumbnail(frame, size, resample, *args, **kwargs): + calls.append((frame.mode, resample)) + return original_thumbnail(frame, size, resample, *args, **kwargs) + + with patch.object(Image.Image, "thumbnail", new=record_thumbnail): + payload = _build_image_request_payload(buffer.getvalue(), ".tiff") + + assert calls == [("I;16", Image.Resampling.NEAREST)] + assert payload["mime"] == "image/jpeg" + assert payload["source_mime"] == "image/tiff" + with Image.open(io.BytesIO(base64.b64decode(payload["data_b64"]))) as sent: + assert max(sent.size) <= 2048 + + +@pytest.mark.asyncio +async def test_auto_image_applies_exif_orientation_before_resizing(auto_resource_env): + """A rotated phone JPEG is normalized upright for the VLM without touching its source.""" + env = auto_resource_env + image = Image.new("RGB", (3000, 1000), (40, 80, 120)) + exif = Image.Exif() + exif[274] = 6 + buffer = io.BytesIO() + image.save(buffer, format="JPEG", exif=exif) + source = env.write_binary("resource/2026-01-01/phone.jpg", buffer.getvalue()) + stored_bytes = source.read_bytes() + model = _FakeVisionModel(_caption_json("upright-phone-photo", "Upright", "An upright phone photo.")) + + response = await env.run(env.processor(model), [{"change": "added", "path": str(source)}]) + + assert response.success is True + data_block = model.calls[0][0].content[1] + assert data_block.source.media_type == "image/jpeg" + with Image.open(io.BytesIO(base64.b64decode(data_block.source.data))) as sent: + assert sent.size == (683, 2048) + assert sent.getexif().get(274) is None + assert source.read_bytes() == stored_bytes + assert (env.workspace / "daily/2026-01-01/upright-phone-photo.md").is_file() + + +@pytest.mark.parametrize( + ("image_format", "suffix", "source_mime", "request_mime"), + [ + ("JPEG", ".png", "image/jpeg", "image/jpeg"), + ("PNG", ".jpg", "image/png", "image/png"), + ("BMP", ".png", "image/bmp", "image/jpeg"), + ], +) +@pytest.mark.asyncio +async def test_auto_image_uses_decoded_format_when_suffix_is_misleading( + image_format, + suffix, + source_mime, + request_mime, + auto_resource_env, +): + """Request and note MIME values come from decoded bytes, with conversion when needed.""" + env = auto_resource_env + source = env.write_binary(f"resource/2026-01-01/mislabeled{suffix}", _img_bytes(image_format)) + stored_bytes = source.read_bytes() + model = _StructuredVisionModel( + content={"name": "actual-format", "description": "Decoded", "caption": "Decoded image content."}, + ) + + response = await env.run(env.processor(model), [{"change": "added", "path": str(source)}]) + + assert response.success is True + data_block = model.structured_calls[0][0].content[1] + assert data_block.source.media_type == request_mime + with Image.open(io.BytesIO(base64.b64decode(data_block.source.data))) as sent: + assert sent.get_format_mimetype() == request_mime + note = frontmatter.load(env.workspace / "daily/2026-01-01/actual-format.md") + assert note.metadata["media_type"] == source_mime + assert source.read_bytes() == stored_bytes + + +@pytest.mark.parametrize("routed", [False, True], ids=["image", "unified-router"]) +@pytest.mark.asyncio +async def test_auto_image_rejects_pixel_bomb_before_decode_and_isolates_batch(routed, auto_resource_env): + """A small compressed file with unsafe dimensions fails without stopping the next image.""" + env = auto_resource_env + unsafe = env.write_binary( + "resource/2026-01-01/pixel-bomb.png", + _png_bytes_with_header_size(8000, 6000), + ) + safe = env.write_binary("resource/2026-01-01/safe.png", _png_bytes()) + model = _FakeVisionModel(_caption_json("safe-image", "Safe", "A safe image.")) + + response = await env.run( + env.processor(model, routed=routed), + [ + {"change": "added", "path": str(unsafe)}, + {"change": "added", "path": str(safe)}, + ], + ) + + results = response.metadata["results"] + assert response.success is False + assert [item["success"] for item in results] == [False, True] + assert results[0]["metadata"]["action"] == "failed" + assert results[0]["metadata"]["modified"] is False + assert f"48000000 > {DEFAULT_MAX_IMAGE_PIXELS}" in results[0]["metadata"]["error"] + assert len(model.calls) == 1 + assert not (env.workspace / "daily/2026-01-01/pixel-bomb.md").exists() + assert (env.workspace / "daily/2026-01-01/safe-image.md").is_file() + + +def test_image_pixel_limit_is_checked_before_full_decode(): + """The explicit pixel budget is enforced from image headers before ``load``.""" + with patch("PIL.PngImagePlugin.PngImageFile.load", side_effect=AssertionError("must not decode")) as image_load: + with pytest.raises(RuntimeError, match=r"8x8=64 > 63"): + _normalize_image_bytes(_png_bytes(), ".png", max_image_pixels=63) + image_load.assert_not_called() + + +def test_image_decompression_bomb_warning_becomes_an_error(monkeypatch): + """Pillow's warning-only bomb threshold becomes a reportable processor error.""" + monkeypatch.setattr(Image, "MAX_IMAGE_PIXELS", 32) + + with pytest.raises(RuntimeError, match="decompression-bomb protection"): + _build_image_request_payload(_png_bytes(), ".png") + + +@pytest.mark.parametrize("routed", [False, True], ids=["image", "unified-router"]) +@pytest.mark.asyncio +async def test_decompression_bomb_warning_is_isolated_per_change(routed, auto_resource_env, monkeypatch): + """A Pillow bomb warning fails one resource while a safe batch peer still completes.""" + env = auto_resource_env + warned = env.write_binary("resource/2026-01-01/warned.png", _png_bytes()) + safe = env.write_binary("resource/2026-01-01/safe.png", _png_bytes(width=4, height=4)) + model = _FakeVisionModel(_caption_json("safe-image", "Safe", "A safe image.")) + monkeypatch.setattr(Image, "MAX_IMAGE_PIXELS", 32) + + response = await env.run( + env.processor(model, routed=routed), + [ + {"change": "added", "path": str(warned)}, + {"change": "added", "path": str(safe)}, + ], + ) + + results = response.metadata["results"] + assert response.success is False + assert [item["success"] for item in results] == [False, True] + assert "decompression-bomb protection" in results[0]["metadata"]["error"] + assert len(model.calls) == 1 + assert not (env.workspace / "daily/2026-01-01/warned.md").exists() + assert (env.workspace / "daily/2026-01-01/safe-image.md").is_file() + + +def test_auto_image_pixel_limit_cannot_be_raised_by_runtime_context(): + """Request-scoped kwargs cannot relax the processor's deployment safety cap.""" + step = AutoImageResourceStep(max_image_pixels=63) + step.context = RuntimeContext(max_image_pixels=DEFAULT_MAX_IMAGE_PIXELS) + + assert step._max_image_pixels() == 63 + + @pytest.mark.asyncio async def test_auto_image_skips_oversized_file(auto_resource_env): """Files beyond max_image_bytes are skipped without a VLM call.""" @@ -334,8 +536,9 @@ def test_auto_resource_router_inherits_declared_options_with_child_override(): prompt_dict=prompt_dict, max_file_bytes=4, max_image_bytes=8, + max_image_pixels=64, dispatch_steps=[ - {"backend": "auto_image_resource_step", "max_image_bytes": 32}, + {"backend": "auto_image_resource_step", "max_image_bytes": 32, "max_image_pixels": 128}, {"backend": "auto_text_resource_step", "max_file_bytes": 16}, ], ) @@ -355,8 +558,16 @@ def test_auto_resource_router_inherits_declared_options_with_child_override(): "as_llm": vision_model, "language": "zh", "max_image_bytes": 32, + "max_image_pixels": 128, } + inherited = AutoResourceStep( + max_image_pixels=64, + dispatch_steps=["auto_image_resource_step", "auto_text_resource_step"], + ) + inherited_specs = {spec["backend"]: spec for spec, _, _ in inherited._processor_routes()} + assert inherited_specs["auto_image_resource_step"]["max_image_pixels"] == 64 + @pytest.mark.asyncio async def test_auto_resource_router_accepts_a_registered_third_modality_without_code_changes(auto_resource_env): @@ -511,13 +722,18 @@ def test_auto_image_named_model_uses_standard_ref_resolution(): @pytest.mark.parametrize( - ("blocked_module", "suffix", "error_pattern"), + ("blocked_module", "payload", "suffix", "error_pattern"), [ - ("PIL", ".png", r"Pillow.*reme-ai\[core\]"), - ("pillow_heif", ".heic", r"pillow-heif.*reme-ai\[image-heif\]"), + ("PIL", _png_bytes(), ".png", r"Pillow.*reme-ai\[core\]"), + ( + "pillow_heif", + b"\x00\x00\x00\x18ftypheic\x00\x00\x00\x00mif1heic", + ".heic", + r"pillow-heif.*reme-ai\[image-heif\]", + ), ], ) -def test_image_preprocessing_reports_dependency_errors(blocked_module, suffix, error_pattern): +def test_image_preprocessing_reports_dependency_errors(blocked_module, payload, suffix, error_pattern): """Lazy image dependencies produce actionable installation errors.""" real_import = __import__ @@ -528,7 +744,24 @@ def test_image_preprocessing_reports_dependency_errors(blocked_module, suffix, e with patch("builtins.__import__", side_effect=import_without_dependency): with pytest.raises(RuntimeError, match=error_pattern): - _normalize_image_bytes(_png_bytes(), suffix) + _normalize_image_bytes(payload, suffix) + + +def test_misleading_heic_suffix_does_not_load_optional_dependency(): + """A core image named ``.heic`` is decoded by content without loading pillow-heif.""" + real_import = __import__ + + def import_without_heif(name, *args, **kwargs): + if name == "pillow_heif": + raise AssertionError("pillow-heif must not be loaded for PNG bytes") + return real_import(name, *args, **kwargs) + + with patch("builtins.__import__", side_effect=import_without_heif): + payload = _build_image_request_payload(_png_bytes(), ".heic") + + assert payload["mime"] == "image/png" + assert payload["source_mime"] == "image/png" + assert payload["converted"] is False @pytest.mark.parametrize( diff --git a/tests/unit/test_auto_resource_review_regressions.py b/tests/unit/test_auto_resource_review_regressions.py index 5b8a1d4b..9fa866c7 100644 --- a/tests/unit/test_auto_resource_review_regressions.py +++ b/tests/unit/test_auto_resource_review_regressions.py @@ -1,8 +1,12 @@ """Regression tests for the safety findings from the Auto Resource PR review.""" +from unittest.mock import patch + import frontmatter import pytest +from reme.steps.evolve.base_auto_resource import BaseAutoResourceStep + from .auto_resource_test_support import ( FakeVisionModel, StructuredVisionModel, @@ -167,3 +171,120 @@ async def test_blank_plain_caption_does_not_create_or_overwrite_note(routed, pla assert not (env.workspace / "daily/2026-01-01/blank-new.md").exists() assert old_note.read_bytes() == before assert len(model.structured_calls) == len(model.plain_calls) == 2 + + +@pytest.mark.parametrize("routed", [False, True], ids=["image", "unified-router"]) +async def test_loose_root_image_keeps_original_daily_card_across_days(routed, auto_resource_env): + """Later updates and deletion keep a loose resource's first daily-card ownership.""" + env = auto_resource_env + source = env.write_binary("resource/photo.png", image_bytes(color=(200, 30, 30))) + initial_model = FakeVisionModel(caption_json("original-card", "Original", "first-day caption")) + with patch.object(BaseAutoResourceStep, "_today", return_value="2026-01-01"): + added = await env.run( + env.processor(initial_model, routed=routed), + [{"change": "added", "path": str(source)}], + ) + + owned_note = env.workspace / "daily/2026-01-01/original-card.md" + add_result = added.metadata["results"][0]["metadata"] + assert added.success is True + assert add_result["path"] == "daily/2026-01-01/original-card.md" + assert add_result["action"] == "added" + assert add_result["index"]["date"] == "2026-01-01" + assert "first-day caption" in owned_note.read_text(encoding="utf-8") + + unrelated_note = env.write_note( + "daily/2026-01-02/photo.md", + "[[resource/other.png]]", + body="unrelated note that must survive", + ) + unrelated_before = unrelated_note.read_bytes() + + first_model = FakeVisionModel(caption_json("renamed-on-day-two", "Updated", "second-day caption")) + with patch.object(BaseAutoResourceStep, "_today", return_value="2026-01-02"): + first_update = await env.run( + env.processor(first_model, routed=routed), + [{"change": "modified", "path": str(source)}], + ) + + first_result = first_update.metadata["results"][0]["metadata"] + assert first_update.success is True + assert first_result["path"] == "daily/2026-01-01/original-card.md" + assert first_result["action"] == "modified" + assert first_result["created"] is False + assert first_result["index"]["date"] == "2026-01-01" + assert "second-day caption" in owned_note.read_text(encoding="utf-8") + assert not (env.workspace / "daily/2026-01-02/renamed-on-day-two.md").exists() + assert unrelated_note.read_bytes() == unrelated_before + assert not (env.workspace / "daily/2026-01-02.md").exists() + + source.write_bytes(image_bytes(color=(20, 90, 200))) + second_model = FakeVisionModel(caption_json("renamed-on-day-three", "Updated again", "third-day caption")) + with patch.object(BaseAutoResourceStep, "_today", return_value="2026-01-03"): + second_update = await env.run( + env.processor(second_model, routed=routed), + [{"change": "modified", "path": str(source)}], + ) + + second_result = second_update.metadata["results"][0]["metadata"] + assert second_update.success is True + assert second_result["path"] == "daily/2026-01-01/original-card.md" + assert second_result["action"] == "modified" + assert second_result["created"] is False + assert second_result["index"]["date"] == "2026-01-01" + assert "third-day caption" in owned_note.read_text(encoding="utf-8") + assert not (env.workspace / "daily/2026-01-03/renamed-on-day-three.md").exists() + assert unrelated_note.read_bytes() == unrelated_before + assert not (env.workspace / "daily/2026-01-03.md").exists() + + source.unlink() + delete_model = FakeVisionModel(caption_json("unused", "Unused", "Must not be requested.")) + with patch.object(BaseAutoResourceStep, "_today", return_value="2026-01-04"): + deleted = await env.run( + env.processor(delete_model, routed=routed), + [{"change": "deleted", "path": str(source)}], + ) + + delete_result = deleted.metadata["results"][0]["metadata"] + assert deleted.success is True + assert delete_result["path"] == "daily/2026-01-01/original-card.md" + assert delete_result["action"] == "deleted" + assert delete_result["modified"] is True + assert delete_result["index"]["date"] == "2026-01-01" + assert delete_result["index"]["notes"] == [] + assert not owned_note.exists() + assert unrelated_note.read_bytes() == unrelated_before + assert "(none)" in (env.workspace / "daily/2026-01-01.md").read_text(encoding="utf-8") + assert not (env.workspace / "daily/2026-01-04.md").exists() + assert len(initial_model.calls) == len(first_model.calls) == len(second_model.calls) == 1 + assert not delete_model.calls + + +@pytest.mark.parametrize("routed", [False, True], ids=["image", "unified-router"]) +async def test_loose_root_image_duplicate_daily_owners_fail_closed(routed, auto_resource_env): + """Ambiguous exact ownership is reported without reading the image model or changing notes.""" + env = auto_resource_env + source = env.write_binary("resource/duplicate.png", image_bytes()) + first_note = env.write_note("daily/2026-01-01/first.md", "[[resource/duplicate.png]]", body="first owner") + second_note = env.write_note("daily/2026-01-02/second.md", "[[resource/duplicate.png]]", body="second owner") + before = {first_note: first_note.read_bytes(), second_note: second_note.read_bytes()} + model = FakeVisionModel(caption_json("replacement", "Replacement", "Must not be generated.")) + + with patch.object(BaseAutoResourceStep, "_today", return_value="2026-01-03"): + response = await env.run( + env.processor(model, routed=routed), + [{"change": "modified", "path": str(source)}], + ) + + result = response.metadata["results"][0]["metadata"] + assert response.success is False + assert result["action"] == "failed" + assert result["modified"] is False + assert "Multiple daily resource notes claim resource/duplicate.png" in result["error"] + assert "daily/2026-01-01/first.md" in result["error"] + assert "daily/2026-01-02/second.md" in result["error"] + assert not model.calls + assert all(path.read_bytes() == contents for path, contents in before.items()) + assert not (env.workspace / "daily/2026-01-01.md").exists() + assert not (env.workspace / "daily/2026-01-02.md").exists() + assert not (env.workspace / "daily/2026-01-03.md").exists()