mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-17 09:32:01 +00:00
refactor(auto-memory): confine image adaptation to image requests
This commit is contained in:
parent
f21e8a2acd
commit
c8be319c20
7 changed files with 116 additions and 65 deletions
|
|
@ -81,13 +81,9 @@ blocks, preventing recalled memory and binary payloads from being mistaken for u
|
|||
## Images in Conversations
|
||||
|
||||
Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a
|
||||
call with `include_images=true`:
|
||||
call with `include_images=true`.
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
Image input requires an AgentScope wrapper (`AsAgentWrapper`) with a vision-capable `as_llm` model and compatible formatter.
|
||||
Image input requires an `agentscope` wrapper with a vision-capable `as_llm` model and compatible formatter.
|
||||
Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no
|
||||
image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers.
|
||||
|
||||
|
|
@ -97,17 +93,21 @@ the formatter; Auto Memory does not download or preprocess the images. URLs must
|
|||
files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported.
|
||||
|
||||
The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather
|
||||
than increasing the limit. The AgentScope default is 5. To use a higher limit in a one-shot CLI call:
|
||||
than increasing the limit. The AgentScope default is 5. To use a higher limit, set it when starting the service:
|
||||
|
||||
```bash
|
||||
reme start job=auto_memory \
|
||||
components.agent_wrapper.default.context_config.max_image_num=20 \
|
||||
session_id=session-a include_images=true messages='[...]'
|
||||
reme start components.agent_wrapper.default.context_config.max_image_num=20
|
||||
```
|
||||
|
||||
Model and formatter limits still apply. Options, wrapper type, URL schemes and image count are checked before saving the
|
||||
conversation. Later formatter or provider errors are returned without retrying as text-only. As with text-only calls,
|
||||
those errors do not roll back an already saved conversation.
|
||||
Then call the running service from another terminal, using the same workspace:
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
Model and formatter limits still apply. When image input is enabled and images are present, Auto Memory checks the option,
|
||||
wrapper backend, URL schemes and image count before saving the conversation. Later formatter or provider errors are returned
|
||||
without retrying as text-only. As with text-only calls, those errors do not roll back an already saved conversation.
|
||||
|
||||
Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images
|
||||
again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created,
|
||||
|
|
|
|||
|
|
@ -73,13 +73,9 @@ daily note 会指向对应的对话记录。持久化时会排除 tool-result bl
|
|||
|
||||
## 对话中的图像
|
||||
|
||||
Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像:
|
||||
Auto Memory 可以结合上下文理解对话中的图像。默认只处理文本,调用时加上 `include_images=true` 即可开启图像。
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
图像输入需要 AgentScope wrapper(`AsAgentWrapper`),其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。
|
||||
图像输入需要 `agentscope` wrapper,其 `as_llm` 应绑定支持视觉的模型,并使用兼容的 formatter。
|
||||
Auto Memory 直接用这个模型理解图文,不先生成 caption。关闭图像或消息中没有图像块时,仍按原有方式处理文本,也不限制
|
||||
wrapper 类型。
|
||||
|
||||
|
|
@ -88,15 +84,20 @@ wrapper 类型。
|
|||
供应商访问;本地文件请先转为 Base64,不使用 `file://` URL,其他 URL scheme 也不支持。
|
||||
|
||||
每次调用的图像数量受 wrapper 的 `context_config.max_image_num` 限制,超限会报错,不会自动提高上限。
|
||||
AgentScope 默认允许 5 张图像。需要更多时,可以在单次 CLI 调用中指定:
|
||||
AgentScope 默认允许 5 张图像。需要更多时,在启动服务时设置:
|
||||
|
||||
```bash
|
||||
reme start job=auto_memory \
|
||||
components.agent_wrapper.default.context_config.max_image_num=20 \
|
||||
session_id=session-a include_images=true messages='[...]'
|
||||
reme start components.agent_wrapper.default.context_config.max_image_num=20
|
||||
```
|
||||
|
||||
模型与 formatter 自身的限制仍然适用。图像选项、wrapper 类型、URL scheme 和图像数量会在保存对话前检查。
|
||||
然后在另一个终端中,使用同一 workspace 调用已启动的服务:
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=session-a include_images=true messages='[...]'
|
||||
```
|
||||
|
||||
模型与 formatter 自身的限制仍然适用。开启图像且消息中包含图像时,才会在保存对话前检查图像选项、wrapper backend、
|
||||
URL scheme 和图像数量。
|
||||
之后的 formatter 或 provider 错误直接返回,不转为纯文本重试;与纯文本调用相同,已保存的对话不会因此回滚。
|
||||
|
||||
源 JSONL 仍按上文规则保存,包括过滤 Base64 block。因此,再次处理这些图像需要提交原始消息,而不是读取已保存的 JSONL。
|
||||
|
|
|
|||
|
|
@ -194,7 +194,6 @@ jobs:
|
|||
- messages
|
||||
steps:
|
||||
- backend: auto_memory_step
|
||||
include_images: false
|
||||
- backend: auto_tag_step
|
||||
|
||||
auto_memory_cc:
|
||||
|
|
|
|||
|
|
@ -18,7 +18,6 @@ from ..file_io import extract_daily_date, parse_daily_date, refresh_day_index
|
|||
from ..file_io import validate_filename_component, validate_session_id
|
||||
from ..index import normalize_posix_path
|
||||
from ...components import R
|
||||
from ...components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
|
||||
|
||||
_SESSION_ID_KEY = "session_id"
|
||||
_SOURCE_CONVERSATION_KEY = "source_conversation"
|
||||
|
|
@ -284,10 +283,8 @@ class AutoMemoryStep(BaseStep):
|
|||
day: str,
|
||||
) -> tuple[list[Msg], dict[str, DataBlock], dict | None]:
|
||||
"""Validate image inputs before saving, without reading or changing their sources."""
|
||||
include_images = self.context.get("include_images", self.kwargs.get("include_images", False))
|
||||
if not isinstance(include_images, bool):
|
||||
raise ValueError("include_images must be a boolean")
|
||||
if not include_images:
|
||||
include_images = self.context.get("include_images", False)
|
||||
if include_images is False:
|
||||
return messages, {}, None
|
||||
images = [
|
||||
(message_index, block_index, block)
|
||||
|
|
@ -297,8 +294,10 @@ class AutoMemoryStep(BaseStep):
|
|||
]
|
||||
if not images:
|
||||
return messages, {}, None
|
||||
if not isinstance(include_images, bool):
|
||||
raise ValueError("include_images must be a boolean")
|
||||
wrapper = self.agent_wrapper
|
||||
if not isinstance(wrapper, AsAgentWrapper):
|
||||
if wrapper is None or wrapper.backend != "agentscope":
|
||||
raise NotImplementedError("Auto Memory image inputs require the AgentScope wrapper")
|
||||
for _, _, block in images:
|
||||
if block.source.type == "url" and urlsplit(str(block.source.url)).scheme not in {"http", "https"}:
|
||||
|
|
|
|||
|
|
@ -14,9 +14,10 @@ import httpx
|
|||
import pytest
|
||||
import yaml
|
||||
|
||||
from reme.application import Application
|
||||
from reme.components import R
|
||||
from reme.components.agent_wrapper import BaseAgentWrapper
|
||||
from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
|
||||
from reme.components.agent_wrapper.cc_agent_wrapper import CcAgentWrapper
|
||||
from reme.components.file_store import LocalFileStore
|
||||
from reme.components.job import BaseJob
|
||||
from reme.components.tag_index import LocalTagIndex
|
||||
|
|
@ -61,13 +62,6 @@ def _saved_line(message):
|
|||
return (message.model_copy(update={"content": content}).model_dump_json() + "\n").encode("utf-8")
|
||||
|
||||
|
||||
class _OtherWrapper(BaseAgentWrapper):
|
||||
"""Text-only backend; a matching label must not bypass the type check."""
|
||||
|
||||
async def reply(self, _inputs, **_kwargs):
|
||||
return {"result": "ok"}
|
||||
|
||||
|
||||
@pytest.fixture(name="setup")
|
||||
def memory_setup(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
|
|
@ -96,17 +90,25 @@ def test_only_include_images_is_exposed_and_disabled_by_default():
|
|||
job = yaml.safe_load(path.read_text(encoding="utf-8"))["jobs"]["auto_memory"]
|
||||
assert job["parameters"]["properties"]["include_images"]["default"] is False
|
||||
assert {"supports_vision", "image_mode"}.isdisjoint(job["parameters"]["properties"])
|
||||
assert job["steps"] == [{"backend": "auto_memory_step", "include_images": False}, {"backend": "auto_tag_step"}]
|
||||
assert job["steps"] == [{"backend": "auto_memory_step"}, {"backend": "auto_tag_step"}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"options,images",
|
||||
[({}, True), ({"include_images": False}, True), ({"include_images": True}, False)],
|
||||
[
|
||||
({}, True),
|
||||
({"include_images": False}, True),
|
||||
({"include_images": False}, False),
|
||||
({"include_images": True}, False),
|
||||
({"include_images": "false"}, False),
|
||||
({"include_images": 0}, False),
|
||||
({"include_images": None}, False),
|
||||
],
|
||||
)
|
||||
async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monkeypatch, options, images):
|
||||
step, _, path = setup
|
||||
wrapper = _OtherWrapper(backend="not-agentscope")
|
||||
wrapper = CcAgentWrapper(backend="claude_code")
|
||||
wrapper.reply = AsyncMock(return_value={"result": "ok"})
|
||||
wrapper.kwargs["context_config"] = {"max_image_num": 0}
|
||||
step.kwargs["agent_wrapper"] = wrapper
|
||||
|
|
@ -114,7 +116,6 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke
|
|||
message.content.append(DataBlock(source=URLSource(media_type="application/pdf", url="file:///not-read.pdf")))
|
||||
before = message.model_dump()
|
||||
extra = {"model_config": {"max_retries": 2}}
|
||||
monkeypatch.setattr(step, "_reply_extra_kwargs", Mock(return_value=extra))
|
||||
expected = step.prompt_format(
|
||||
"user_message_create",
|
||||
today=_DAY,
|
||||
|
|
@ -124,6 +125,24 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke
|
|||
session_file=f"session/dialog/{_SESSION}.jsonl",
|
||||
history=step._format_history([message]),
|
||||
)
|
||||
events = []
|
||||
save, history = step._save_session_messages, step._format_history
|
||||
|
||||
async def save_source(*args):
|
||||
await save(*args)
|
||||
events.append("save")
|
||||
|
||||
def format_history(messages):
|
||||
events.append("history")
|
||||
return history(messages)
|
||||
|
||||
def reply_kwargs(_day):
|
||||
events.append("reply_kwargs")
|
||||
return extra
|
||||
|
||||
monkeypatch.setattr(step, "_save_session_messages", save_source)
|
||||
monkeypatch.setattr(step, "_format_history", format_history)
|
||||
monkeypatch.setattr(step, "_reply_extra_kwargs", reply_kwargs)
|
||||
|
||||
response = await _run(step, [message], **options)
|
||||
|
||||
|
|
@ -139,6 +158,7 @@ async def test_text_path_keeps_main_input_kwargs_metadata_and_jsonl(setup, monke
|
|||
assert path.read_bytes() == _saved_line(message)
|
||||
assert message.model_dump() == before
|
||||
assert wrapper.kwargs["context_config"] == {"max_image_num": 0}
|
||||
assert events == ["save", "history", "reply_kwargs"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
@ -186,7 +206,7 @@ async def test_sources_interleave_unchanged_through_native_formatters(setup, mon
|
|||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("existing", [False, True])
|
||||
@pytest.mark.parametrize("failure", ["bool", "backend", "file", "ftp", "default-limit"])
|
||||
@pytest.mark.parametrize("failure", ["bool", "zero", "null", "backend", "file", "ftp", "default-limit"])
|
||||
async def test_static_errors_precede_any_source_session_write(setup, existing, failure):
|
||||
step, wrapper, path = setup
|
||||
old = _message("old", images=False, timestamp=f"{_DAY}T09:00:00")
|
||||
|
|
@ -196,10 +216,10 @@ async def test_static_errors_precede_any_source_session_write(setup, existing, f
|
|||
path.write_bytes(before)
|
||||
message, options = _message(), {"include_images": True}
|
||||
error, match = ValueError, "include_images"
|
||||
if failure == "bool":
|
||||
options["include_images"] = "true"
|
||||
if failure in ("bool", "zero", "null"):
|
||||
options["include_images"] = {"bool": "true", "zero": 0, "null": None}[failure]
|
||||
elif failure == "backend":
|
||||
step.kwargs["agent_wrapper"] = _OtherWrapper(backend="agentscope")
|
||||
step.kwargs["agent_wrapper"] = CcAgentWrapper(backend="claude_code")
|
||||
error, match = NotImplementedError, "AgentScope"
|
||||
elif failure in ("file", "ftp"):
|
||||
message.content[1].source = URLSource(media_type="image/png", url=f"{failure}:///private/image.png")
|
||||
|
|
@ -217,10 +237,11 @@ async def test_static_errors_precede_any_source_session_write(setup, existing, f
|
|||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("value", [None, 1, [], "false"])
|
||||
async def test_include_images_requires_a_boolean_even_without_messages(setup, value):
|
||||
async def test_empty_messages_ignore_image_option_and_keep_main_skip_behavior(setup, value):
|
||||
step, wrapper, path = setup
|
||||
with pytest.raises(ValueError, match="include_images"):
|
||||
await _run(step, [], include_images=value)
|
||||
response = await _run(step, [], include_images=value)
|
||||
assert response.success is True and response.answer == "Skipped: no messages"
|
||||
assert response.metadata == {"date": _DAY, "modified": False, "n_messages": 0}
|
||||
assert not path.exists()
|
||||
wrapper.reply.assert_not_called()
|
||||
|
||||
|
|
@ -324,22 +345,53 @@ async def test_provider_error_is_not_retried_and_keeps_main_saved_source(setup,
|
|||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"configured,options,enabled",
|
||||
"backend,configured,options,images,expected",
|
||||
[
|
||||
(True, {}, True),
|
||||
(False, {}, False),
|
||||
(True, {"include_images": False}, False),
|
||||
(False, {"include_images": True}, True),
|
||||
("agentscope", True, {}, True, Msg),
|
||||
("agentscope", False, {}, True, str),
|
||||
("agentscope", True, {"include_images": False}, True, str),
|
||||
("agentscope", False, {"include_images": True}, True, Msg),
|
||||
("claude_code", True, {}, True, "AgentScope"),
|
||||
("claude_code", False, {}, True, str),
|
||||
("claude_code", True, {}, False, str),
|
||||
("agentscope", False, {"include_images": "true"}, True, "boolean"),
|
||||
("claude_code", False, {"include_images": "true"}, False, str),
|
||||
],
|
||||
)
|
||||
async def test_runtime_switch_overrides_step_config(setup, configured, options, enabled):
|
||||
step, wrapper, _ = setup
|
||||
step.kwargs["include_images"] = configured
|
||||
|
||||
response = await _run(step, [_message()], **options)
|
||||
|
||||
assert response.success is True
|
||||
assert isinstance(wrapper.reply.call_args.args[0], Msg if enabled else str)
|
||||
async def test_configured_backend_and_job_switch(tmp_path, monkeypatch, backend, configured, options, images, expected):
|
||||
config = Path(__file__).resolve().parents[2] / "reme/config/default.yaml"
|
||||
job_config = yaml.safe_load(config.read_text(encoding="utf-8"))["jobs"]["auto_memory"]
|
||||
job_config.update(include_images=configured, steps=[{"backend": "auto_memory_step"}])
|
||||
app = Application(
|
||||
workspace_dir=str(tmp_path),
|
||||
enable_logo=False,
|
||||
log_to_console=False,
|
||||
log_to_file=False,
|
||||
service={"backend": "cli"},
|
||||
components={
|
||||
"agent_wrapper": {"default": {"backend": backend, "as_llm": ""}},
|
||||
"file_store": {"default": {"backend": "local", "embedding_store": ""}},
|
||||
},
|
||||
jobs={"auto_memory": job_config},
|
||||
)
|
||||
wrapper = app.context.components["agent_wrapper"]["default"]
|
||||
assert isinstance(wrapper, AsAgentWrapper if backend == "agentscope" else CcAgentWrapper)
|
||||
assert wrapper.backend == backend
|
||||
wrapper.reply = AsyncMock(return_value={"result": "ok"})
|
||||
monkeypatch.setattr(AutoMemoryStep, "_list_session_note", AsyncMock(return_value=None))
|
||||
job = app.context.jobs["auto_memory"]
|
||||
await job.start()
|
||||
try:
|
||||
response = await job(session_id=_SESSION, date=_DAY, messages=[_message(images=images)], **options)
|
||||
finally:
|
||||
await job.close()
|
||||
if isinstance(expected, str):
|
||||
assert response.success is False and expected in response.answer
|
||||
assert not (tmp_path / "session/dialog" / f"{_SESSION}.jsonl").exists()
|
||||
wrapper.reply.assert_not_called()
|
||||
else:
|
||||
assert response.success is True
|
||||
assert isinstance(wrapper.reply.call_args.args[0], expected)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
|
|
|||
|
|
@ -260,7 +260,7 @@ def test_configs_define_original_jobs_without_daily_variants():
|
|||
|
||||
default = resolve_app_config(config="default", log_config=False)
|
||||
assert default["jobs"]["auto_memory"]["steps"] == [
|
||||
{"backend": "auto_memory_step", "include_images": False},
|
||||
{"backend": "auto_memory_step"},
|
||||
{"backend": "auto_tag_step"},
|
||||
]
|
||||
assert default["jobs"]["auto_memory_cc"]["steps"] == [
|
||||
|
|
|
|||
|
|
@ -591,7 +591,7 @@ def test_default_config_enables_tag_index_with_explicit_key() -> None:
|
|||
assert config["components"]["file_store"]["default"]["tag_index"] == "default"
|
||||
assert config["jobs"]["search"]["parameters"]["properties"]["tags"]["default"] == []
|
||||
assert config["jobs"]["auto_memory"]["steps"] == [
|
||||
{"backend": "auto_memory_step", "include_images": False},
|
||||
{"backend": "auto_memory_step"},
|
||||
{"backend": "auto_tag_step"},
|
||||
]
|
||||
assert config["jobs"]["auto_memory_cc"]["steps"] == [
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue