mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-17 09:32:01 +00:00
refactor(auto-memory): simplify image docs and tests
This commit is contained in:
parent
2f4cb0578f
commit
39c2f5a68f
3 changed files with 48 additions and 134 deletions
|
|
@ -78,33 +78,26 @@ session/
|
|||
Each daily note points to its corresponding conversation record. Saved messages omit tool-result blocks and base64 data
|
||||
blocks, preventing recalled memory and binary payloads from being mistaken for user-provided evidence later.
|
||||
|
||||
## Optional Session Images
|
||||
## Images in Conversations
|
||||
|
||||
Images are disabled by default. Set `include_images=true` to let the memory Agent use images alongside the conversation:
|
||||
Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a
|
||||
call with `include_images=true`:
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
For a one-shot CLI invocation, use `reme start job=auto_memory` with the same arguments.
|
||||
No YAML edit or rebuild is needed. When images are disabled, or the input contains no image blocks, Auto Memory keeps its
|
||||
existing text-only behavior on every wrapper. Only enabled, image-bearing calls require the AgentScope wrapper
|
||||
(`AsAgentWrapper`); other backends raise `NotImplementedError`. Enabling images means the caller has selected a compatible
|
||||
model and formatter. Auto Memory uses the model already bound through the wrapper's `as_llm`, without capability probing
|
||||
or model switching.
|
||||
Image input requires an AgentScope wrapper (`AsAgentWrapper`) with a vision-capable `as_llm` model and compatible formatter.
|
||||
Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no
|
||||
image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers.
|
||||
|
||||
Pass AgentScope messages in `messages`, with top-level image `DataBlock` values (`source.media_type` starting with
|
||||
`image/`). Auto Memory builds a `UserMsg` with text and original image blocks interleaved in conversation order, retaining
|
||||
speaker, timestamp and task-prompt boundaries. The Agent extracts memory from text and images together, without separate
|
||||
caption calls. This is one Agent workflow, not necessarily one API request: tool use can trigger further model turns.
|
||||
Pass images as top-level AgentScope `DataBlock` values in `messages`, with an `image/` media type. Text and images stay in
|
||||
their original order, with speaker and timestamp boundaries preserved. Base64 sources and HTTP(S) URLs pass unchanged to
|
||||
the formatter; Auto Memory does not download or preprocess the images. URLs must be accessible to the model provider. For local
|
||||
files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported.
|
||||
|
||||
Base64 sources and HTTP(S) image URLs are passed unchanged to the AgentScope formatter. Auto Memory does not download,
|
||||
decode, resize or transcode them. The model provider must be able to access supplied URLs. `file://` and other URL schemes
|
||||
are rejected; convert local images to Base64 before submitting them. Caller messages and original image files are unchanged.
|
||||
|
||||
Auto Memory checks the wrapper's effective `context_config.max_image_num` and rejects inputs that exceed it; it never
|
||||
raises the limit automatically. The current AgentScope default is 5 images. To allow more, configure the wrapper explicitly,
|
||||
for example:
|
||||
The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather
|
||||
than increasing the limit. The AgentScope default is 5. To use a higher limit in a one-shot CLI call:
|
||||
|
||||
```bash
|
||||
reme start job=auto_memory \
|
||||
|
|
@ -112,18 +105,13 @@ reme start job=auto_memory \
|
|||
session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
Model and formatter limits still apply. Invalid image options, unsupported backends, prohibited URL schemes and image-count
|
||||
overflow are checked before saving the source conversation. Formatter or provider failures follow the existing error path,
|
||||
without a text-only retry. As with text-only calls, failures after saving do not roll back the source conversation.
|
||||
Model and formatter limits still apply. Options, wrapper type, URL schemes and image count are checked before saving the
|
||||
conversation. Later formatter or provider errors are returned without retrying as text-only. As with text-only calls,
|
||||
those errors do not roll back an already saved conversation.
|
||||
|
||||
**Source JSONL saving is unchanged whether images are enabled or disabled.** The filtering above still applies; no captions
|
||||
or image metadata are added. Replaying a saved JSONL cannot recover omitted Base64 images: submit the original image-bearing
|
||||
messages again. Auto Memory creates no separate image resources or caption cards, but the existing AgentScope wrapper's
|
||||
internal Agent state under `mem_session/agentscope` can include image inputs.
|
||||
|
||||
For a persistent default, set `jobs.auto_memory.include_images` in the application configuration or its `reme start` override.
|
||||
Call-time values override Job defaults, which override Step configuration; the default is `false`. This adapter targets
|
||||
AgentScope messages, not Claude Code transcript parsing.
|
||||
Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images
|
||||
again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created,
|
||||
though the wrapper's internal Agent state under `mem_session/agentscope` can contain image inputs.
|
||||
|
||||
## Message Timestamps
|
||||
|
||||
|
|
|
|||
|
|
@ -71,31 +71,24 @@ session/
|
|||
daily note 会指向对应的对话记录。持久化时会排除 tool-result block 和 base64 data block,避免召回记忆或二进制负载在后续流程中被误当成
|
||||
用户提供的证据。
|
||||
|
||||
## 可选的 Session 图像
|
||||
## 对话中的图像
|
||||
|
||||
默认不开启图像。设置 `include_images=true` 后,记忆 Agent 可以同时利用对话文本和图像:
|
||||
Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像:
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
单次 CLI 调用可使用 `reme start job=auto_memory`,其余参数相同。
|
||||
不需要修改 YAML 或重新编译。关闭图像,或输入没有图像块时,各种 wrapper 都保持原有纯文本行为。
|
||||
只有开启且实际包含图像的调用才要求 AgentScope wrapper(`AsAgentWrapper`),其他后端会抛出 `NotImplementedError`。
|
||||
开启图像表示调用方已经选用兼容的模型与 formatter。Auto Memory 沿用 wrapper 通过 `as_llm` 绑定的模型,
|
||||
不会自动探测能力或切换模型。
|
||||
图像输入需要 AgentScope wrapper(`AsAgentWrapper`),其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。
|
||||
Auto Memory 直接用这个模型理解图文,不先生成 caption。关闭图像或消息中没有图像块时,仍按原有方式处理文本,也不限制
|
||||
wrapper 类型。
|
||||
|
||||
`messages` 使用 AgentScope 格式,处理顶层图像 `DataBlock`(`source.media_type` 以 `image/` 开头)。
|
||||
Auto Memory 构建 `UserMsg`,按原会话顺序交错放置文本与原始图像块,保留说话人、时间和任务提示的文本边界。
|
||||
记忆 Agent 同时从图文中提取信息,不单独调用 caption 模型。这是一次 Agent 工作流,不保证只有一次 API 请求:
|
||||
工具调用可能触发后续模型轮次。
|
||||
在 `messages` 中用 AgentScope 顶层 `DataBlock` 传入图像,媒体类型以 `image/` 开头。文本和图像按原顺序交错排列,
|
||||
保留说话人和时间信息。Base64 source 与 HTTP(S) URL 原样交给 formatter,Auto Memory 不下载或预处理图像。URL 需要能被模型
|
||||
供应商访问;本地文件请先转为 Base64,不使用 `file://` URL,其他 URL scheme 也不支持。
|
||||
|
||||
Base64 source 与 HTTP(S) 图像 URL 原样传给 AgentScope formatter,Auto Memory 不下载、解码、缩放或转码图像。
|
||||
所提供的 URL 必须能被模型供应商访问。`file://` 和其他 URL scheme 会被拒绝,请先将本地图像转换为 Base64 再提交。
|
||||
调用方原始消息与图像文件保持不变。
|
||||
|
||||
Auto Memory 检查 wrapper 实际生效的 `context_config.max_image_num`,超限时明确报错,不自动提高上限。
|
||||
当前 AgentScope 默认允许 5 张图像。如果需要更多,请显式配置 wrapper,例如:
|
||||
每次调用的图像数量受 wrapper 的 `context_config.max_image_num` 限制,超限会报错,不会自动提高上限。
|
||||
AgentScope 默认允许 5 张图像。需要更多时,可以在单次 CLI 调用中指定:
|
||||
|
||||
```bash
|
||||
reme start job=auto_memory \
|
||||
|
|
@ -103,18 +96,11 @@ reme start job=auto_memory \
|
|||
session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
模型与 formatter 自身的限制仍然适用。无效的图像选项、不支持的后端、禁止的 URL scheme 和图像数量超限,
|
||||
都在保存源会话之前检查。formatter 或 provider 的错误沿现有错误流程返回,不以纯文本重试。
|
||||
与纯文本调用相同,保存后发生的错误不会回滚源会话记录。
|
||||
模型与 formatter 自身的限制仍然适用。图像选项、wrapper 类型、URL scheme 和图像数量会在保存对话前检查。
|
||||
之后的 formatter 或 provider 错误直接返回,不转为纯文本重试;与纯文本调用相同,已保存的对话不会因此回滚。
|
||||
|
||||
**开启与关闭图像时的源 JSONL 保存行为完全不变。** 仍执行上文的过滤规则,不补充 caption 或图像 metadata。
|
||||
重放已保存的 JSONL 无法恢复被过滤掉的 Base64 图像;再次处理需要提交原始带图消息。
|
||||
Auto Memory 不创建独立图像资源或 caption 卡片,但 AgentScope wrapper 按原有行为保存在 `mem_session/agentscope`
|
||||
的内部 Agent 状态可能包含图像输入。
|
||||
|
||||
若需持久默认值,在应用配置或 `reme start` 对应覆盖项中设置 `jobs.auto_memory.include_images`。
|
||||
调用时参数高于 Job 默认值,Job 默认值高于 Step 配置;默认值为 `false`。
|
||||
本适配针对 AgentScope 消息,不扩展 Claude Code 原始 transcript 的解析。
|
||||
源 JSONL 仍按上文规则保存,包括过滤 Base64 block。因此,再次处理这些图像需要提交原始消息,而不是读取已保存的 JSONL。
|
||||
不会另外生成图像文件或 caption 卡片,但 wrapper 保存在 `mem_session/agentscope` 中的内部 Agent 状态可能包含图像输入。
|
||||
|
||||
## 消息时间
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,6 @@ from unittest.mock import AsyncMock, Mock
|
|||
|
||||
from agentscope.formatter import DashScopeChatFormatter, OpenAIChatFormatter
|
||||
from agentscope.message import Base64Source, DataBlock, Msg, TextBlock, URLSource
|
||||
from agentscope.model import ChatModelBase
|
||||
import httpx
|
||||
import pytest
|
||||
import yaml
|
||||
|
|
@ -18,11 +17,9 @@ import yaml
|
|||
from reme.components import R
|
||||
from reme.components.agent_wrapper import BaseAgentWrapper
|
||||
from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
|
||||
from reme.components.as_llm import BaseAsLLM
|
||||
from reme.components.file_store import LocalFileStore
|
||||
from reme.components.job import BaseJob
|
||||
from reme.components.tag_index import LocalTagIndex
|
||||
from reme.enumeration import ComponentEnum
|
||||
from reme.schema import ApplicationConfig
|
||||
from reme.steps.evolve.auto_memory import AutoMemoryStep
|
||||
|
||||
|
|
@ -64,25 +61,6 @@ def _saved_line(message):
|
|||
return (message.model_copy(update={"content": content}).model_dump_json() + "\n").encode("utf-8")
|
||||
|
||||
|
||||
class _Model(ChatModelBase):
|
||||
"""Native model binding without a provider client."""
|
||||
|
||||
def __init__(self):
|
||||
self.model = "bound-memory-model"
|
||||
self.context_size = 200000
|
||||
self.formatter = OpenAIChatFormatter()
|
||||
|
||||
|
||||
class _Wrapper(AsAgentWrapper):
|
||||
"""Observe the real AgentScope wrapper boundary without invoking a provider."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(backend="agentscope")
|
||||
self.as_llm = BaseAsLLM()
|
||||
self.as_llm.model = _Model()
|
||||
self.reply = AsyncMock(return_value={"result": "ok"})
|
||||
|
||||
|
||||
class _OtherWrapper(BaseAgentWrapper):
|
||||
"""Text-only backend; a matching label must not bypass the type check."""
|
||||
|
||||
|
|
@ -93,14 +71,15 @@ class _OtherWrapper(BaseAgentWrapper):
|
|||
@pytest.fixture(name="setup")
|
||||
def memory_setup(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
wrapper = _Wrapper()
|
||||
wrapper = AsAgentWrapper(backend="agentscope", as_llm="")
|
||||
wrapper.reply = AsyncMock(return_value={"result": "ok"})
|
||||
store = LocalFileStore(embedding_store="")
|
||||
app = SimpleNamespace(
|
||||
registry=R,
|
||||
metadata={},
|
||||
app_config=ApplicationConfig(workspace_dir=str(tmp_path)),
|
||||
jobs={},
|
||||
components={ComponentEnum.AS_LLM: {"default": wrapper.as_llm}},
|
||||
components={},
|
||||
)
|
||||
step = AutoMemoryStep(app_context=app, file_store=store, agent_wrapper=wrapper)
|
||||
monkeypatch.setattr(step, "_list_session_note", AsyncMock(return_value=None))
|
||||
|
|
@ -343,50 +322,22 @@ async def test_provider_error_is_not_retried_and_keeps_main_saved_source(setup,
|
|||
assert "auto_memory_images" not in step.context.response.metadata
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_uses_existing_bound_model_without_an_override(setup):
|
||||
step, wrapper, _ = setup
|
||||
model = wrapper.as_llm.model
|
||||
await _run(step, [_message()], include_images=True)
|
||||
inputs, options = wrapper.reply.call_args.args[0], wrapper.reply.call_args.kwargs
|
||||
assert "_model" not in options
|
||||
agent, _ = await wrapper._build_agent(inputs, **{**options, "job_tools": []})
|
||||
assert agent.model is model
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"step_switch,job_switch,call_switch,enabled",
|
||||
[(True, False, None, False), (False, False, True, True)],
|
||||
"configured,options,enabled",
|
||||
[
|
||||
(True, {}, True),
|
||||
(False, {}, False),
|
||||
(True, {"include_images": False}, False),
|
||||
(False, {"include_images": True}, True),
|
||||
],
|
||||
)
|
||||
async def test_runtime_switch_uses_existing_job_precedence(
|
||||
setup,
|
||||
monkeypatch,
|
||||
step_switch,
|
||||
job_switch,
|
||||
call_switch,
|
||||
enabled,
|
||||
):
|
||||
async def test_runtime_switch_overrides_step_config(setup, configured, options, enabled):
|
||||
step, wrapper, _ = setup
|
||||
monkeypatch.setattr(AutoMemoryStep, "_list_session_note", AsyncMock(return_value=None))
|
||||
job = BaseJob(
|
||||
app_context=step.app_context,
|
||||
include_images=job_switch,
|
||||
steps=[
|
||||
{
|
||||
"backend": "auto_memory_step",
|
||||
"include_images": step_switch,
|
||||
"file_store": step.file_store,
|
||||
"agent_wrapper": wrapper,
|
||||
},
|
||||
],
|
||||
)
|
||||
await job.start()
|
||||
try:
|
||||
options = {} if call_switch is None else {"include_images": call_switch}
|
||||
response = await job(session_id=_SESSION, date=_DAY, messages=[_message()], **options)
|
||||
finally:
|
||||
await job.close()
|
||||
step.kwargs["include_images"] = configured
|
||||
|
||||
response = await _run(step, [_message()], **options)
|
||||
|
||||
assert response.success is True
|
||||
assert isinstance(wrapper.reply.call_args.args[0], Msg if enabled else str)
|
||||
|
||||
|
|
@ -394,19 +345,9 @@ async def test_runtime_switch_uses_existing_job_precedence(
|
|||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("enabled", [False, True])
|
||||
@pytest.mark.parametrize("history", ["append", "backfill", "same-id", "replace-and-backfill"])
|
||||
async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enabled, history):
|
||||
async def test_history_merge_keeps_main_source_contract(setup, enabled, history):
|
||||
step, _, path = setup
|
||||
original = _message()
|
||||
raw = original.model_dump()
|
||||
raw["role"] = "assistant"
|
||||
raw["content"].extend(
|
||||
[
|
||||
{"type": "tool_call", "id": "recall", "name": "read", "input": "{}"},
|
||||
{"type": "tool_result", "id": "recall", "name": "read", "output": "RECALLED_NOT_SOURCE"},
|
||||
{"type": "data", "source": {"type": "base64", "media_type": "application/pdf", "data": "YWJj"}},
|
||||
],
|
||||
)
|
||||
original = Msg.model_validate(raw)
|
||||
await _run(step, [original], include_images=enabled)
|
||||
initial = path.read_bytes()
|
||||
replacement = original.model_copy(deep=True)
|
||||
|
|
@ -426,7 +367,6 @@ async def test_history_merge_and_sanitizer_keep_main_source_contract(setup, enab
|
|||
await _run(step, messages, include_images=enabled)
|
||||
|
||||
assert path.read_bytes() == expected
|
||||
assert b"RECALLED_NOT_SOURCE" not in expected and b"YWJj" not in expected
|
||||
assert [message.model_dump() for message in messages] == before
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue