refactor(auto-memory): simplify image docs and tests

This commit is contained in:
wang-qisen 2026-09-15 23:25:20 +08:00
parent 2f4cb0578f
commit 39c2f5a68f
3 changed files with 48 additions and 134 deletions

View file

@ -78,33 +78,26 @@ session/
Each daily note points to its corresponding conversation record. Saved messages omit tool-result blocks and base64 data
blocks, preventing recalled memory and binary payloads from being mistaken for user-provided evidence later.
## Optional Session Images
## Images in Conversations
Images are disabled by default. Set `include_images=true` to let the memory Agent use images alongside the conversation:
Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a
call with `include_images=true`:
```bash
reme auto_memory session_id=session-a include_images=true messages='[...]'
```
For a one-shot CLI invocation, use `reme start job=auto_memory` with the same arguments.
No YAML edit or rebuild is needed. When images are disabled, or the input contains no image blocks, Auto Memory keeps its
existing text-only behavior on every wrapper. Only enabled, image-bearing calls require the AgentScope wrapper
(`AsAgentWrapper`); other backends raise `NotImplementedError`. Enabling images means the caller has selected a compatible
model and formatter. Auto Memory uses the model already bound through the wrapper's `as_llm`, without capability probing
or model switching.
Image input requires an AgentScope wrapper (`AsAgentWrapper`) with a vision-capable `as_llm` model and compatible formatter.
Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no
image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers.
Pass AgentScope messages in `messages`, with top-level image `DataBlock` values (`source.media_type` starting with
`image/`). Auto Memory builds a `UserMsg` with text and original image blocks interleaved in conversation order, retaining
speaker, timestamp and task-prompt boundaries. The Agent extracts memory from text and images together, without separate
caption calls. This is one Agent workflow, not necessarily one API request: tool use can trigger further model turns.
Pass images as top-level AgentScope `DataBlock` values in `messages`, with an `image/` media type. Text and images stay in
their original order, with speaker and timestamp boundaries preserved. Base64 sources and HTTP(S) URLs pass unchanged to
the formatter; Auto Memory does not download or preprocess the images. URLs must be accessible to the model provider. For local
files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported.
Base64 sources and HTTP(S) image URLs are passed unchanged to the AgentScope formatter. Auto Memory does not download,
decode, resize or transcode them. The model provider must be able to access supplied URLs. `file://` and other URL schemes
are rejected; convert local images to Base64 before submitting them. Caller messages and original image files are unchanged.
Auto Memory checks the wrapper's effective `context_config.max_image_num` and rejects inputs that exceed it; it never
raises the limit automatically. The current AgentScope default is 5 images. To allow more, configure the wrapper explicitly,
for example:
The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather
than increasing the limit. The AgentScope default is 5. To use a higher limit in a one-shot CLI call:
```bash
reme start job=auto_memory \
@ -112,18 +105,13 @@ reme start job=auto_memory \
session_id=session-a include_images=true messages='[...]'
```
Model and formatter limits still apply. Invalid image options, unsupported backends, prohibited URL schemes and image-count
overflow are checked before saving the source conversation. Formatter or provider failures follow the existing error path,
without a text-only retry. As with text-only calls, failures after saving do not roll back the source conversation.
Model and formatter limits still apply. Options, wrapper type, URL schemes and image count are checked before saving the
conversation. Later formatter or provider errors are returned without retrying as text-only. As with text-only calls,
those errors do not roll back an already saved conversation.
**Source JSONL saving is unchanged whether images are enabled or disabled.** The filtering above still applies; no captions
or image metadata are added. Replaying a saved JSONL cannot recover omitted Base64 images: submit the original image-bearing
messages again. Auto Memory creates no separate image resources or caption cards, but the existing AgentScope wrapper's
internal Agent state under `mem_session/agentscope` can include image inputs.
For a persistent default, set `jobs.auto_memory.include_images` in the application configuration or its `reme start` override.
Call-time values override Job defaults, which override Step configuration; the default is `false`. This adapter targets
AgentScope messages, not Claude Code transcript parsing.
Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images
again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created,
though the wrapper's internal Agent state under `mem_session/agentscope` can contain image inputs.
## Message Timestamps

View file

@ -71,31 +71,24 @@ session/
daily note 会指向对应的对话记录。持久化时会排除 tool-result block 和 base64 data block,避免召回记忆或二进制负载在后续流程中被误当成
用户提供的证据。
## 可选的 Session 图像
## 对话中的图像
默认不开启图像。设置 `include_images=true` 后,记忆 Agent 可以同时利用对话文本和图像:
Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像:
```bash
reme auto_memory session_id=session-a include_images=true messages='[...]'
```
单次 CLI 调用可使用 `reme start job=auto_memory`,其余参数相同。
不需要修改 YAML 或重新编译。关闭图像,或输入没有图像块时,各种 wrapper 都保持原有纯文本行为。
只有开启且实际包含图像的调用才要求 AgentScope wrapper(`AsAgentWrapper`),其他后端会抛出 `NotImplementedError`。
开启图像表示调用方已经选用兼容的模型与 formatter。Auto Memory 沿用 wrapper 通过 `as_llm` 绑定的模型,
不会自动探测能力或切换模型。
图像输入需要 AgentScope wrapper(`AsAgentWrapper`),其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。
Auto Memory 直接用这个模型理解图文,不先生成 caption。关闭图像或消息中没有图像块时,仍按原有方式处理文本,也不限制
wrapper 类型。
`messages` 使用 AgentScope 格式,处理顶层图像 `DataBlock`(`source.media_type` 以 `image/` 开头)。
Auto Memory 构建 `UserMsg`,按原会话顺序交错放置文本与原始图像块,保留说话人、时间和任务提示的文本边界。
记忆 Agent 同时从图文中提取信息,不单独调用 caption 模型。这是一次 Agent 工作流,不保证只有一次 API 请求:
工具调用可能触发后续模型轮次。
在 `messages` 中用 AgentScope 顶层 `DataBlock` 传入图像,媒体类型以 `image/` 开头。文本和图像按原顺序交错排列,
保留说话人和时间信息。Base64 source 与 HTTP(S) URL 原样交给 formatter,Auto Memory 不下载或预处理图像。URL 需要能被模型
供应商访问;本地文件请先转为 Base64,不使用 `file://` URL,其他 URL scheme 也不支持。
Base64 source 与 HTTP(S) 图像 URL 原样传给 AgentScope formatter,Auto Memory 不下载、解码、缩放或转码图像。
所提供的 URL 必须能被模型供应商访问。`file://` 和其他 URL scheme 会被拒绝,请先将本地图像转换为 Base64 再提交。
调用方原始消息与图像文件保持不变。
Auto Memory 检查 wrapper 实际生效的 `context_config.max_image_num`,超限时明确报错,不自动提高上限。
当前 AgentScope 默认允许 5 张图像。如果需要更多,请显式配置 wrapper,例如:
每次调用的图像数量受 wrapper 的 `context_config.max_image_num` 限制,超限会报错,不会自动提高上限。
AgentScope 默认允许 5 张图像。需要更多时,可以在单次 CLI 调用中指定:
```bash
reme start job=auto_memory \
@ -103,18 +96,11 @@ reme start job=auto_memory \
session_id=session-a include_images=true messages='[...]'
```
模型与 formatter 自身的限制仍然适用。无效的图像选项、不支持的后端、禁止的 URL scheme 和图像数量超限,
都在保存源会话之前检查。formatter 或 provider 的错误沿现有错误流程返回,不以纯文本重试。
与纯文本调用相同,保存后发生的错误不会回滚源会话记录。
模型与 formatter 自身的限制仍然适用。图像选项、wrapper 类型、URL scheme 和图像数量会在保存对话前检查。
之后的 formatter 或 provider 错误直接返回,不转为纯文本重试;与纯文本调用相同,已保存的对话不会因此回滚。
**开启与关闭图像时的源 JSONL 保存行为完全不变。** 仍执行上文的过滤规则,不补充 caption 或图像 metadata。
重放已保存的 JSONL 无法恢复被过滤掉的 Base64 图像;再次处理需要提交原始带图消息。
Auto Memory 不创建独立图像资源或 caption 卡片,但 AgentScope wrapper 按原有行为保存在 `mem_session/agentscope`
的内部 Agent 状态可能包含图像输入。
若需持久默认值,在应用配置或 `reme start` 对应覆盖项中设置 `jobs.auto_memory.include_images`。
调用时参数高于 Job 默认值,Job 默认值高于 Step 配置;默认值为 `false`。
本适配针对 AgentScope 消息,不扩展 Claude Code 原始 transcript 的解析。
源 JSONL 仍按上文规则保存,包括过滤 Base64 block。因此,再次处理这些图像需要提交原始消息,而不是读取已保存的 JSONL。
不会另外生成图像文件或 caption 卡片,但 wrapper 保存在 `mem_session/agentscope` 中的内部 Agent 状态可能包含图像输入。
## 消息时间

View file

@ -10,7 +10,6 @@ from unittest.mock import AsyncMock, Mock
from agentscope.formatter import DashScopeChatFormatter, OpenAIChatFormatter
from agentscope.message import Base64Source, DataBlock, Msg, TextBlock, URLSource
from agentscope.model import ChatModelBase
import httpx
import pytest
import yaml
@ -18,11 +17,9 @@ import yaml
from reme.components import R
from reme.components.agent_wrapper import BaseAgentWrapper
from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
from reme.components.as_llm import BaseAsLLM
from reme.components.file_store import LocalFileStore
from reme.components.job import BaseJob
from reme.components.tag_index import LocalTagIndex
from reme.enumeration import ComponentEnum
from reme.schema import ApplicationConfig
from reme.steps.evolve.auto_memory import AutoMemoryStep
@ -64,25 +61,6 @@ def _saved_line(message):
return (message.model_copy(update={"content": content}).model_dump_json() + "\n").encode("utf-8")
class _Model(ChatModelBase):
"""Native model binding without a provider client."""
def __init__(self):
self.model = "bound-memory-model"
self.context_size = 200000
self.formatter = OpenAIChatFormatter()
class _Wrapper(AsAgentWrapper):
"""Observe the real AgentScope wrapper boundary without invoking a provider."""
def __init__(self):
super().__init__(backend="agentscope")
self.as_llm = BaseAsLLM()
self.as_llm.model = _Model()
self.reply = AsyncMock(return_value={"result": "ok"})
class _OtherWrapper(BaseAgentWrapper):
"""Text-only backend; a matching label must not bypass the type check."""
@ -93,14 +71,15 @@ class _OtherWrapper(BaseAgentWrapper):
@pytest.fixture(name="setup")
def memory_setup(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
wrapper = _Wrapper()
wrapper = AsAgentWrapper(backend="agentscope", as_llm="")
wrapper.reply = AsyncMock(return_value={"result": "ok"})
store = LocalFileStore(embedding_store="")
app = SimpleNamespace(
registry=R,
metadata={},
app_config=ApplicationConfig(workspace_dir=str(tmp_path)),
jobs={},
components={ComponentEnum.AS_LLM: {"default": wrapper.as_llm}},
components={},
)
step = AutoMemoryStep(app_context=app, file_store=store, agent_wrapper=wrapper)
monkeypatch.setattr(step, "_list_session_note", AsyncMock(return_value=None))
@ -343,50 +322,22 @@ async def test_provider_error_is_not_retried_and_keeps_main_saved_source(setup,
assert "auto_memory_images" not in step.context.response.metadata
@pytest.mark.asyncio
async def test_uses_existing_bound_model_without_an_override(setup):
step, wrapper, _ = setup
model = wrapper.as_llm.model
await _run(step, [_message()], include_images=True)
inputs, options = wrapper.reply.call_args.args[0], wrapper.reply.call_args.kwargs
assert "_model" not in options
agent, _ = await wrapper._build_agent(inputs, **{**options, "job_tools": []})
assert agent.model is model
@pytest.mark.asyncio
@pytest.mark.parametrize(
"step_switch,job_switch,call_switch,enabled",
[(True, False, None, False), (False, False, True, True)],
"configured,options,enabled",
[
(True, {}, True),
(False, {}, False),
(True, {"include_images": False}, False),
(False, {"include_images": True}, True),
],
)
async def test_runtime_switch_uses_existing_job_precedence(
setup,
monkeypatch,
step_switch,
job_switch,
call_switch,
enabled,
):
async def test_runtime_switch_overrides_step_config(setup, configured, options, enabled):
step, wrapper, _ = setup
monkeypatch.setattr(AutoMemoryStep, "_list_session_note", AsyncMock(return_value=None))
job = BaseJob(
app_context=step.app_context,
include_images=job_switch,
steps=[
{
"backend": "auto_memory_step",
"include_images": step_switch,
"file_store": step.file_store,
"agent_wrapper": wrapper,
},
],
)
await job.start()
try:
options = {} if call_switch is None else {"include_images": call_switch}
response = await job(session_id=_SESSION, date=_DAY, messages=[_message()], **options)
finally:
await job.close()
step.kwargs["include_images"] = configured
response = await _run(step, [_message()], **options)
assert response.success is True
assert isinstance(wrapper.reply.call_args.args[0], Msg if enabled else str)
@ -394,19 +345,9 @@ async def test_runtime_switch_uses_existing_job_precedence(
@pytest.mark.asyncio
@pytest.mark.parametrize("enabled", [False, True])
@pytest.mark.parametrize("history", ["append", "backfill", "same-id", "replace-and-backfill"])
async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enabled, history):
async def test_history_merge_keeps_main_source_contract(setup, enabled, history):
step, _, path = setup
original = _message()
raw = original.model_dump()
raw["role"] = "assistant"
raw["content"].extend(
[
{"type": "tool_call", "id": "recall", "name": "read", "input": "{}"},
{"type": "tool_result", "id": "recall", "name": "read", "output": "RECALLED_NOT_SOURCE"},
{"type": "data", "source": {"type": "base64", "media_type": "application/pdf", "data": "YWJj"}},
],
)
original = Msg.model_validate(raw)
await _run(step, [original], include_images=enabled)
initial = path.read_bytes()
replacement = original.model_copy(deep=True)
@ -426,7 +367,6 @@ async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enab
await _run(step, messages, include_images=enabled)
assert path.read_bytes() == expected
assert b"RECALLED_NOT_SOURCE" not in expected and b"YWJj" not in expected
assert [message.model_dump() for message in messages] == before