diff --git a/docs/en/auto_memory.md b/docs/en/auto_memory.md index 7b6c241e..41912282 100644 --- a/docs/en/auto_memory.md +++ b/docs/en/auto_memory.md @@ -78,33 +78,26 @@ session/ Each daily note points to its corresponding conversation record. Saved messages omit tool-result blocks and base64 data blocks, preventing recalled memory and binary payloads from being mistaken for user-provided evidence later. -## Optional Session Images +## Images in Conversations -Images are disabled by default. Set `include_images=true` to let the memory Agent use images alongside the conversation: +Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a +call with `include_images=true`: ```bash reme auto_memory session_id=session-a include_images=true messages='[...]' ``` -For a one-shot CLI invocation, use `reme start job=auto_memory` with the same arguments. -No YAML edit or rebuild is needed. When images are disabled, or the input contains no image blocks, Auto Memory keeps its -existing text-only behavior on every wrapper. Only enabled, image-bearing calls require the AgentScope wrapper -(`AsAgentWrapper`); other backends raise `NotImplementedError`. Enabling images means the caller has selected a compatible -model and formatter. Auto Memory uses the model already bound through the wrapper's `as_llm`, without capability probing -or model switching. +Image input requires an AgentScope wrapper (`AsAgentWrapper`) with a vision-capable `as_llm` model and compatible formatter. +Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no +image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers. -Pass AgentScope messages in `messages`, with top-level image `DataBlock` values (`source.media_type` starting with -`image/`). Auto Memory builds a `UserMsg` with text and original image blocks interleaved in conversation order, retaining -speaker, timestamp and task-prompt boundaries. The Agent extracts memory from text and images together, without separate -caption calls. This is one Agent workflow, not necessarily one API request: tool use can trigger further model turns. +Pass images as top-level AgentScope `DataBlock` values in `messages`, with an `image/` media type. Text and images stay in +their original order, with speaker and timestamp boundaries preserved. Base64 sources and HTTP(S) URLs pass unchanged to +the formatter; Auto Memory does not download or preprocess the images. URLs must be accessible to the model provider. For local +files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported. -Base64 sources and HTTP(S) image URLs are passed unchanged to the AgentScope formatter. Auto Memory does not download, -decode, resize or transcode them. The model provider must be able to access supplied URLs. `file://` and other URL schemes -are rejected; convert local images to Base64 before submitting them. Caller messages and original image files are unchanged. - -Auto Memory checks the wrapper's effective `context_config.max_image_num` and rejects inputs that exceed it; it never -raises the limit automatically. The current AgentScope default is 5 images. To allow more, configure the wrapper explicitly, -for example: +The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather +than increasing the limit. The AgentScope default is 5. To use a higher limit in a one-shot CLI call: ```bash reme start job=auto_memory \ @@ -112,18 +105,13 @@ reme start job=auto_memory \ session_id=session-a include_images=true messages='[...]' ``` -Model and formatter limits still apply. Invalid image options, unsupported backends, prohibited URL schemes and image-count -overflow are checked before saving the source conversation. Formatter or provider failures follow the existing error path, -without a text-only retry. As with text-only calls, failures after saving do not roll back the source conversation. +Model and formatter limits still apply. Options, wrapper type, URL schemes and image count are checked before saving the +conversation. Later formatter or provider errors are returned without retrying as text-only. As with text-only calls, +those errors do not roll back an already saved conversation. -**Source JSONL saving is unchanged whether images are enabled or disabled.** The filtering above still applies; no captions -or image metadata are added. Replaying a saved JSONL cannot recover omitted Base64 images: submit the original image-bearing -messages again. Auto Memory creates no separate image resources or caption cards, but the existing AgentScope wrapper's -internal Agent state under `mem_session/agentscope` can include image inputs. - -For a persistent default, set `jobs.auto_memory.include_images` in the application configuration or its `reme start` override. -Call-time values override Job defaults, which override Step configuration; the default is `false`. This adapter targets -AgentScope messages, not Claude Code transcript parsing. +Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images +again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created, +though the wrapper's internal Agent state under `mem_session/agentscope` can contain image inputs. ## Message Timestamps diff --git a/docs/zh/auto_memory.md b/docs/zh/auto_memory.md index fbac1093..202052c1 100644 --- a/docs/zh/auto_memory.md +++ b/docs/zh/auto_memory.md @@ -71,31 +71,24 @@ session/ daily note 会指向对应的对话记录。持久化时会排除 tool-result block 和 base64 data block,避免召回记忆或二进制负载在后续流程中被误当成 用户提供的证据。 -## 可选的 Session 图像 +## 对话中的图像 -默认不开启图像。设置 `include_images=true` 后,记忆 Agent 可以同时利用对话文本和图像: +Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像: ```bash reme auto_memory session_id=session-a include_images=true messages='[...]' ``` -单次 CLI 调用可使用 `reme start job=auto_memory`,其余参数相同。 -不需要修改 YAML 或重新编译。关闭图像,或输入没有图像块时,各种 wrapper 都保持原有纯文本行为。 -只有开启且实际包含图像的调用才要求 AgentScope wrapper(`AsAgentWrapper`),其他后端会抛出 `NotImplementedError`。 -开启图像表示调用方已经选用兼容的模型与 formatter。Auto Memory 沿用 wrapper 通过 `as_llm` 绑定的模型, -不会自动探测能力或切换模型。 +图像输入需要 AgentScope wrapper(`AsAgentWrapper`),其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。 +Auto Memory 直接用这个模型理解图文,不先生成 caption。关闭图像或消息中没有图像块时,仍按原有方式处理文本,也不限制 +wrapper 类型。 -`messages` 使用 AgentScope 格式,处理顶层图像 `DataBlock`(`source.media_type` 以 `image/` 开头)。 -Auto Memory 构建 `UserMsg`,按原会话顺序交错放置文本与原始图像块,保留说话人、时间和任务提示的文本边界。 -记忆 Agent 同时从图文中提取信息,不单独调用 caption 模型。这是一次 Agent 工作流,不保证只有一次 API 请求: -工具调用可能触发后续模型轮次。 +在 `messages` 中用 AgentScope 顶层 `DataBlock` 传入图像,媒体类型以 `image/` 开头。文本和图像按原顺序交错排列, +保留说话人和时间信息。Base64 source 与 HTTP(S) URL 原样交给 formatter,Auto Memory 不下载或预处理图像。URL 需要能被模型 +供应商访问;本地文件请先转为 Base64,不使用 `file://` URL,其他 URL scheme 也不支持。 -Base64 source 与 HTTP(S) 图像 URL 原样传给 AgentScope formatter,Auto Memory 不下载、解码、缩放或转码图像。 -所提供的 URL 必须能被模型供应商访问。`file://` 和其他 URL scheme 会被拒绝,请先将本地图像转换为 Base64 再提交。 -调用方原始消息与图像文件保持不变。 - -Auto Memory 检查 wrapper 实际生效的 `context_config.max_image_num`,超限时明确报错,不自动提高上限。 -当前 AgentScope 默认允许 5 张图像。如果需要更多,请显式配置 wrapper,例如: +每次调用的图像数量受 wrapper 的 `context_config.max_image_num` 限制,超限会报错,不会自动提高上限。 +AgentScope 默认允许 5 张图像。需要更多时,可以在单次 CLI 调用中指定: ```bash reme start job=auto_memory \ @@ -103,18 +96,11 @@ reme start job=auto_memory \ session_id=session-a include_images=true messages='[...]' ``` -模型与 formatter 自身的限制仍然适用。无效的图像选项、不支持的后端、禁止的 URL scheme 和图像数量超限, -都在保存源会话之前检查。formatter 或 provider 的错误沿现有错误流程返回,不以纯文本重试。 -与纯文本调用相同,保存后发生的错误不会回滚源会话记录。 +模型与 formatter 自身的限制仍然适用。图像选项、wrapper 类型、URL scheme 和图像数量会在保存对话前检查。 +之后的 formatter 或 provider 错误直接返回,不转为纯文本重试;与纯文本调用相同,已保存的对话不会因此回滚。 -**开启与关闭图像时的源 JSONL 保存行为完全不变。** 仍执行上文的过滤规则,不补充 caption 或图像 metadata。 -重放已保存的 JSONL 无法恢复被过滤掉的 Base64 图像;再次处理需要提交原始带图消息。 -Auto Memory 不创建独立图像资源或 caption 卡片,但 AgentScope wrapper 按原有行为保存在 `mem_session/agentscope` -的内部 Agent 状态可能包含图像输入。 - -若需持久默认值,在应用配置或 `reme start` 对应覆盖项中设置 `jobs.auto_memory.include_images`。 -调用时参数高于 Job 默认值,Job 默认值高于 Step 配置;默认值为 `false`。 -本适配针对 AgentScope 消息,不扩展 Claude Code 原始 transcript 的解析。 +源 JSONL 仍按上文规则保存,包括过滤 Base64 block。因此,再次处理这些图像需要提交原始消息,而不是读取已保存的 JSONL。 +不会另外生成图像文件或 caption 卡片,但 wrapper 保存在 `mem_session/agentscope` 中的内部 Agent 状态可能包含图像输入。 ## 消息时间 diff --git a/tests/unit/test_auto_memory_direct_images.py b/tests/unit/test_auto_memory_direct_images.py index 81782146..d8cabbc7 100644 --- a/tests/unit/test_auto_memory_direct_images.py +++ b/tests/unit/test_auto_memory_direct_images.py @@ -10,7 +10,6 @@ from unittest.mock import AsyncMock, Mock from agentscope.formatter import DashScopeChatFormatter, OpenAIChatFormatter from agentscope.message import Base64Source, DataBlock, Msg, TextBlock, URLSource -from agentscope.model import ChatModelBase import httpx import pytest import yaml @@ -18,11 +17,9 @@ import yaml from reme.components import R from reme.components.agent_wrapper import BaseAgentWrapper from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper -from reme.components.as_llm import BaseAsLLM from reme.components.file_store import LocalFileStore from reme.components.job import BaseJob from reme.components.tag_index import LocalTagIndex -from reme.enumeration import ComponentEnum from reme.schema import ApplicationConfig from reme.steps.evolve.auto_memory import AutoMemoryStep @@ -64,25 +61,6 @@ def _saved_line(message): return (message.model_copy(update={"content": content}).model_dump_json() + "\n").encode("utf-8") -class _Model(ChatModelBase): - """Native model binding without a provider client.""" - - def __init__(self): - self.model = "bound-memory-model" - self.context_size = 200000 - self.formatter = OpenAIChatFormatter() - - -class _Wrapper(AsAgentWrapper): - """Observe the real AgentScope wrapper boundary without invoking a provider.""" - - def __init__(self): - super().__init__(backend="agentscope") - self.as_llm = BaseAsLLM() - self.as_llm.model = _Model() - self.reply = AsyncMock(return_value={"result": "ok"}) - - class _OtherWrapper(BaseAgentWrapper): """Text-only backend; a matching label must not bypass the type check.""" @@ -93,14 +71,15 @@ class _OtherWrapper(BaseAgentWrapper): @pytest.fixture(name="setup") def memory_setup(tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) - wrapper = _Wrapper() + wrapper = AsAgentWrapper(backend="agentscope", as_llm="") + wrapper.reply = AsyncMock(return_value={"result": "ok"}) store = LocalFileStore(embedding_store="") app = SimpleNamespace( registry=R, metadata={}, app_config=ApplicationConfig(workspace_dir=str(tmp_path)), jobs={}, - components={ComponentEnum.AS_LLM: {"default": wrapper.as_llm}}, + components={}, ) step = AutoMemoryStep(app_context=app, file_store=store, agent_wrapper=wrapper) monkeypatch.setattr(step, "_list_session_note", AsyncMock(return_value=None)) @@ -343,50 +322,22 @@ async def test_provider_error_is_not_retried_and_keeps_main_saved_source(setup, assert "auto_memory_images" not in step.context.response.metadata -@pytest.mark.asyncio -async def test_uses_existing_bound_model_without_an_override(setup): - step, wrapper, _ = setup - model = wrapper.as_llm.model - await _run(step, [_message()], include_images=True) - inputs, options = wrapper.reply.call_args.args[0], wrapper.reply.call_args.kwargs - assert "_model" not in options - agent, _ = await wrapper._build_agent(inputs, **{**options, "job_tools": []}) - assert agent.model is model - - @pytest.mark.asyncio @pytest.mark.parametrize( - "step_switch,job_switch,call_switch,enabled", - [(True, False, None, False), (False, False, True, True)], + "configured,options,enabled", + [ + (True, {}, True), + (False, {}, False), + (True, {"include_images": False}, False), + (False, {"include_images": True}, True), + ], ) -async def test_runtime_switch_uses_existing_job_precedence( - setup, - monkeypatch, - step_switch, - job_switch, - call_switch, - enabled, -): +async def test_runtime_switch_overrides_step_config(setup, configured, options, enabled): step, wrapper, _ = setup - monkeypatch.setattr(AutoMemoryStep, "_list_session_note", AsyncMock(return_value=None)) - job = BaseJob( - app_context=step.app_context, - include_images=job_switch, - steps=[ - { - "backend": "auto_memory_step", - "include_images": step_switch, - "file_store": step.file_store, - "agent_wrapper": wrapper, - }, - ], - ) - await job.start() - try: - options = {} if call_switch is None else {"include_images": call_switch} - response = await job(session_id=_SESSION, date=_DAY, messages=[_message()], **options) - finally: - await job.close() + step.kwargs["include_images"] = configured + + response = await _run(step, [_message()], **options) + assert response.success is True assert isinstance(wrapper.reply.call_args.args[0], Msg if enabled else str) @@ -394,19 +345,9 @@ async def test_runtime_switch_uses_existing_job_precedence( @pytest.mark.asyncio @pytest.mark.parametrize("enabled", [False, True]) @pytest.mark.parametrize("history", ["append", "backfill", "same-id", "replace-and-backfill"]) -async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enabled, history): +async def test_history_merge_keeps_main_source_contract(setup, enabled, history): step, _, path = setup original = _message() - raw = original.model_dump() - raw["role"] = "assistant" - raw["content"].extend( - [ - {"type": "tool_call", "id": "recall", "name": "read", "input": "{}"}, - {"type": "tool_result", "id": "recall", "name": "read", "output": "RECALLED_NOT_SOURCE"}, - {"type": "data", "source": {"type": "base64", "media_type": "application/pdf", "data": "YWJj"}}, - ], - ) - original = Msg.model_validate(raw) await _run(step, [original], include_images=enabled) initial = path.read_bytes() replacement = original.model_copy(deep=True) @@ -426,7 +367,6 @@ async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enab await _run(step, messages, include_images=enabled) assert path.read_bytes() == expected - assert b"RECALLED_NOT_SOURCE" not in expected and b"YWJj" not in expected assert [message.model_dump() for message in messages] == before