diff --git a/docs/en/auto_memory.md b/docs/en/auto_memory.md index 41912282..1a1e3570 100644 --- a/docs/en/auto_memory.md +++ b/docs/en/auto_memory.md @@ -81,13 +81,9 @@ blocks, preventing recalled memory and binary payloads from being mistaken for u ## Images in Conversations Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a -call with `include_images=true`: +call with `include_images=true`. -```bash -reme auto_memory session_id=session-a include_images=true messages='[...]' -``` - -Image input requires an AgentScope wrapper (`AsAgentWrapper`) with a vision-capable `as_llm` model and compatible formatter. +Image input requires an `agentscope` wrapper with a vision-capable `as_llm` model and compatible formatter. Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers. @@ -97,17 +93,21 @@ the formatter; Auto Memory does not download or preprocess the images. URLs must files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported. The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather -than increasing the limit. The AgentScope default is 5. To use a higher limit in a one-shot CLI call: +than increasing the limit. The AgentScope default is 5. To use a higher limit, set it when starting the service: ```bash -reme start job=auto_memory \ - components.agent_wrapper.default.context_config.max_image_num=20 \ - session_id=session-a include_images=true messages='[...]' +reme start components.agent_wrapper.default.context_config.max_image_num=20 ``` -Model and formatter limits still apply. Options, wrapper type, URL schemes and image count are checked before saving the -conversation. Later formatter or provider errors are returned without retrying as text-only. As with text-only calls, -those errors do not roll back an already saved conversation. +Then call the running service from another terminal, using the same workspace: + +```bash +reme auto_memory session_id=session-a include_images=true messages='[...]' +``` + +Model and formatter limits still apply. When image input is enabled and images are present, Auto Memory checks the option, +wrapper backend, URL schemes and image count before saving the conversation. Later formatter or provider errors are returned +without retrying as text-only. As with text-only calls, those errors do not roll back an already saved conversation. Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created, diff --git a/docs/zh/auto_memory.md b/docs/zh/auto_memory.md index 202052c1..7e729b3f 100644 --- a/docs/zh/auto_memory.md +++ b/docs/zh/auto_memory.md @@ -73,13 +73,9 @@ daily note 会指向对应的对话记录。持久化时会排除 tool-result bl ## 对话中的图像 -Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像: +Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像。 -```bash -reme auto_memory session_id=session-a include_images=true messages='[...]' -``` - -图像输入需要 AgentScope wrapper(`AsAgentWrapper`),其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。 +图像输入需要 `agentscope` wrapper,其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。 Auto Memory 直接用这个模型理解图文,不先生成 caption。关闭图像或消息中没有图像块时,仍按原有方式处理文本,也不限制 wrapper 类型。 @@ -88,15 +84,20 @@ wrapper 类型。 供应商访问;本地文件请先转为 Base64,不使用 `file://` URL,其他 URL scheme 也不支持。 每次调用的图像数量受 wrapper 的 `context_config.max_image_num` 限制,超限会报错,不会自动提高上限。 -AgentScope 默认允许 5 张图像。需要更多时,可以在单次 CLI 调用中指定: +AgentScope 默认允许 5 张图像。需要更多时,在启动服务时设置: ```bash -reme start job=auto_memory \ - components.agent_wrapper.default.context_config.max_image_num=20 \ - session_id=session-a include_images=true messages='[...]' +reme start components.agent_wrapper.default.context_config.max_image_num=20 ``` -模型与 formatter 自身的限制仍然适用。图像选项、wrapper 类型、URL scheme 和图像数量会在保存对话前检查。 +然后在另一个终端中,使用同一 workspace 调用已启动的服务: + +```bash +reme auto_memory session_id=session-a include_images=true messages='[...]' +``` + +模型与 formatter 自身的限制仍然适用。开启图像且消息中包含图像时,才会在保存对话前检查图像选项、wrapper backend、 +URL scheme 和图像数量。 之后的 formatter 或 provider 错误直接返回,不转为纯文本重试;与纯文本调用相同,已保存的对话不会因此回滚。 源 JSONL 仍按上文规则保存,包括过滤 Base64 block。因此,再次处理这些图像需要提交原始消息,而不是读取已保存的 JSONL。 diff --git a/reme/config/default.yaml b/reme/config/default.yaml index 64034825..4b772442 100644 --- a/reme/config/default.yaml +++ b/reme/config/default.yaml @@ -194,7 +194,6 @@ jobs: - messages steps: - backend: auto_memory_step - include_images: false - backend: auto_tag_step auto_memory_cc: diff --git a/reme/steps/evolve/auto_memory.py b/reme/steps/evolve/auto_memory.py index 1ff08170..20b71530 100644 --- a/reme/steps/evolve/auto_memory.py +++ b/reme/steps/evolve/auto_memory.py @@ -18,7 +18,6 @@ from ..file_io import extract_daily_date, parse_daily_date, refresh_day_index from ..file_io import validate_filename_component, validate_session_id from ..index import normalize_posix_path from ...components import R -from ...components.agent_wrapper.as_agent_wrapper import AsAgentWrapper _SESSION_ID_KEY = "session_id" _SOURCE_CONVERSATION_KEY = "source_conversation" @@ -284,10 +283,8 @@ class AutoMemoryStep(BaseStep): day: str, ) -> tuple[list[Msg], dict[str, DataBlock], dict | None]: """Validate image inputs before saving, without reading or changing their sources.""" - include_images = self.context.get("include_images", self.kwargs.get("include_images", False)) - if not isinstance(include_images, bool): - raise ValueError("include_images must be a boolean") - if not include_images: + include_images = self.context.get("include_images", False) + if include_images is False: return messages, {}, None images = [ (message_index, block_index, block) @@ -297,8 +294,10 @@ class AutoMemoryStep(BaseStep): ] if not images: return messages, {}, None + if not isinstance(include_images, bool): + raise ValueError("include_images must be a boolean") wrapper = self.agent_wrapper - if not isinstance(wrapper, AsAgentWrapper): + if wrapper is None or wrapper.backend != "agentscope": raise NotImplementedError("Auto Memory image inputs require the AgentScope wrapper") for _, _, block in images: if block.source.type == "url" and urlsplit(str(block.source.url)).scheme not in {"http", "https"}: diff --git a/tests/unit/test_auto_memory_direct_images.py b/tests/unit/test_auto_memory_direct_images.py index d8cabbc7..6aecf379 100644 --- a/tests/unit/test_auto_memory_direct_images.py +++ b/tests/unit/test_auto_memory_direct_images.py @@ -14,9 +14,10 @@ import httpx import pytest import yaml +from reme.application import Application from reme.components import R -from reme.components.agent_wrapper import BaseAgentWrapper from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper +from reme.components.agent_wrapper.cc_agent_wrapper import CcAgentWrapper from reme.components.file_store import LocalFileStore from reme.components.job import BaseJob from reme.components.tag_index import LocalTagIndex @@ -61,13 +62,6 @@ def _saved_line(message): return (message.model_copy(update={"content": content}).model_dump_json() + "\n").encode("utf-8") -class _OtherWrapper(BaseAgentWrapper): - """Text-only backend; a matching label must not bypass the type check.""" - - async def reply(self, _inputs, **_kwargs): - return {"result": "ok"} - - @pytest.fixture(name="setup") def memory_setup(tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) @@ -96,17 +90,25 @@ def test_only_include_images_is_exposed_and_disabled_by_default(): job = yaml.safe_load(path.read_text(encoding="utf-8"))["jobs"]["auto_memory"] assert job["parameters"]["properties"]["include_images"]["default"] is False assert {"supports_vision", "image_mode"}.isdisjoint(job["parameters"]["properties"]) - assert job["steps"] == [{"backend": "auto_memory_step", "include_images": False}, {"backend": "auto_tag_step"}] + assert job["steps"] == [{"backend": "auto_memory_step"}, {"backend": "auto_tag_step"}] @pytest.mark.asyncio @pytest.mark.parametrize( "options,images", - [({}, True), ({"include_images": False}, True), ({"include_images": True}, False)], + [ + ({}, True), + ({"include_images": False}, True), + ({"include_images": False}, False), + ({"include_images": True}, False), + ({"include_images": "false"}, False), + ({"include_images": 0}, False), + ({"include_images": None}, False), + ], ) async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monkeypatch, options, images): step, _, path = setup - wrapper = _OtherWrapper(backend="not-agentscope") + wrapper = CcAgentWrapper(backend="claude_code") wrapper.reply = AsyncMock(return_value={"result": "ok"}) wrapper.kwargs["context_config"] = {"max_image_num": 0} step.kwargs["agent_wrapper"] = wrapper @@ -114,7 +116,6 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke message.content.append(DataBlock(source=URLSource(media_type="application/pdf", url="file:///not-read.pdf"))) before = message.model_dump() extra = {"model_config": {"max_retries": 2}} - monkeypatch.setattr(step, "_reply_extra_kwargs", Mock(return_value=extra)) expected = step.prompt_format( "user_message_create", today=_DAY, @@ -124,6 +125,24 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke session_file=f"session/dialog/{_SESSION}.jsonl", history=step._format_history([message]), ) + events = [] + save, history = step._save_session_messages, step._format_history + + async def save_source(*args): + await save(*args) + events.append("save") + + def format_history(messages): + events.append("history") + return history(messages) + + def reply_kwargs(_day): + events.append("reply_kwargs") + return extra + + monkeypatch.setattr(step, "_save_session_messages", save_source) + monkeypatch.setattr(step, "_format_history", format_history) + monkeypatch.setattr(step, "_reply_extra_kwargs", reply_kwargs) response = await _run(step, [message], **options) @@ -139,6 +158,7 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke assert path.read_bytes() == _saved_line(message) assert message.model_dump() == before assert wrapper.kwargs["context_config"] == {"max_image_num": 0} + assert events == ["save", "history", "reply_kwargs"] @pytest.mark.asyncio @@ -186,7 +206,7 @@ async def test_sources_interleave_unchanged_through_native_formatters(setup, mon @pytest.mark.asyncio @pytest.mark.parametrize("existing", [False, True]) -@pytest.mark.parametrize("failure", ["bool", "backend", "file", "ftp", "default-limit"]) +@pytest.mark.parametrize("failure", ["bool", "zero", "null", "backend", "file", "ftp", "default-limit"]) async def test_static_errors_precede_any_source_session_write(setup, existing, failure): step, wrapper, path = setup old = _message("old", images=False, timestamp=f"{_DAY}T09:00:00") @@ -196,10 +216,10 @@ async def test_static_errors_precede_any_source_session_write(setup, existing, f path.write_bytes(before) message, options = _message(), {"include_images": True} error, match = ValueError, "include_images" - if failure == "bool": - options["include_images"] = "true" + if failure in ("bool", "zero", "null"): + options["include_images"] = {"bool": "true", "zero": 0, "null": None}[failure] elif failure == "backend": - step.kwargs["agent_wrapper"] = _OtherWrapper(backend="agentscope") + step.kwargs["agent_wrapper"] = CcAgentWrapper(backend="claude_code") error, match = NotImplementedError, "AgentScope" elif failure in ("file", "ftp"): message.content[1].source = URLSource(media_type="image/png", url=f"{failure}:///private/image.png") @@ -217,10 +237,11 @@ async def test_static_errors_precede_any_source_session_write(setup, existing, f @pytest.mark.asyncio @pytest.mark.parametrize("value", [None, 1, [], "false"]) -async def test_include_images_requires_a_boolean_even_without_messages(setup, value): +async def test_empty_messages_ignore_image_option_and_keep_main_skip_behavior(setup, value): step, wrapper, path = setup - with pytest.raises(ValueError, match="include_images"): - await _run(step, [], include_images=value) + response = await _run(step, [], include_images=value) + assert response.success is True and response.answer == "Skipped: no messages" + assert response.metadata == {"date": _DAY, "modified": False, "n_messages": 0} assert not path.exists() wrapper.reply.assert_not_called() @@ -324,22 +345,53 @@ async def test_provider_error_is_not_retried_and_keeps_main_saved_source(setup, @pytest.mark.asyncio @pytest.mark.parametrize( - "configured,options,enabled", + "backend,configured,options,images,expected", [ - (True, {}, True), - (False, {}, False), - (True, {"include_images": False}, False), - (False, {"include_images": True}, True), + ("agentscope", True, {}, True, Msg), + ("agentscope", False, {}, True, str), + ("agentscope", True, {"include_images": False}, True, str), + ("agentscope", False, {"include_images": True}, True, Msg), + ("claude_code", True, {}, True, "AgentScope"), + ("claude_code", False, {}, True, str), + ("claude_code", True, {}, False, str), + ("agentscope", False, {"include_images": "true"}, True, "boolean"), + ("claude_code", False, {"include_images": "true"}, False, str), ], ) -async def test_runtime_switch_overrides_step_config(setup, configured, options, enabled): - step, wrapper, _ = setup - step.kwargs["include_images"] = configured - - response = await _run(step, [_message()], **options) - - assert response.success is True - assert isinstance(wrapper.reply.call_args.args[0], Msg if enabled else str) +async def test_configured_backend_and_job_switch(tmp_path, monkeypatch, backend, configured, options, images, expected): + config = Path(__file__).resolve().parents[2] / "reme/config/default.yaml" + job_config = yaml.safe_load(config.read_text(encoding="utf-8"))["jobs"]["auto_memory"] + job_config.update(include_images=configured, steps=[{"backend": "auto_memory_step"}]) + app = Application( + workspace_dir=str(tmp_path), + enable_logo=False, + log_to_console=False, + log_to_file=False, + service={"backend": "cli"}, + components={ + "agent_wrapper": {"default": {"backend": backend, "as_llm": ""}}, + "file_store": {"default": {"backend": "local", "embedding_store": ""}}, + }, + jobs={"auto_memory": job_config}, + ) + wrapper = app.context.components["agent_wrapper"]["default"] + assert isinstance(wrapper, AsAgentWrapper if backend == "agentscope" else CcAgentWrapper) + assert wrapper.backend == backend + wrapper.reply = AsyncMock(return_value={"result": "ok"}) + monkeypatch.setattr(AutoMemoryStep, "_list_session_note", AsyncMock(return_value=None)) + job = app.context.jobs["auto_memory"] + await job.start() + try: + response = await job(session_id=_SESSION, date=_DAY, messages=[_message(images=images)], **options) + finally: + await job.close() + if isinstance(expected, str): + assert response.success is False and expected in response.answer + assert not (tmp_path / "session/dialog" / f"{_SESSION}.jsonl").exists() + wrapper.reply.assert_not_called() + else: + assert response.success is True + assert isinstance(wrapper.reply.call_args.args[0], expected) @pytest.mark.asyncio diff --git a/tests/unit/test_injected_job_kwargs.py b/tests/unit/test_injected_job_kwargs.py index d91a927c..5da892e8 100644 --- a/tests/unit/test_injected_job_kwargs.py +++ b/tests/unit/test_injected_job_kwargs.py @@ -260,7 +260,7 @@ def test_configs_define_original_jobs_without_daily_variants(): default = resolve_app_config(config="default", log_config=False) assert default["jobs"]["auto_memory"]["steps"] == [ - {"backend": "auto_memory_step", "include_images": False}, + {"backend": "auto_memory_step"}, {"backend": "auto_tag_step"}, ] assert default["jobs"]["auto_memory_cc"]["steps"] == [ diff --git a/tests/unit/test_tag_index.py b/tests/unit/test_tag_index.py index fc1b54de..845eb279 100644 --- a/tests/unit/test_tag_index.py +++ b/tests/unit/test_tag_index.py @@ -591,7 +591,7 @@ def test_default_config_enables_tag_index_with_explicit_key() -> None: assert config["components"]["file_store"]["default"]["tag_index"] == "default" assert config["jobs"]["search"]["parameters"]["properties"]["tags"]["default"] == [] assert config["jobs"]["auto_memory"]["steps"] == [ - {"backend": "auto_memory_step", "include_images": False}, + {"backend": "auto_memory_step"}, {"backend": "auto_tag_step"}, ] assert config["jobs"]["auto_memory_cc"]["steps"] == [