mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-13 04:01:35 +00:00
docs: add bilingual multimodal memory blog and latest updates (#601)
* docs: introduce multimodal memory update * docs: fix multimodal memory intro formatting * docs: simplify multimodal memory startup examples * docs: simplify blog sidebars and verify multimodal pages --------- Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
This commit is contained in:
WQS
GitHub
parent
e2da221fc4
commit
64dcc5cdfa
6 changed files with 270 additions and 1 deletions
|
|
@ -49,6 +49,9 @@ users retain control of the durable files.
|
|||
|
||||
## 📰 Latest Updates
|
||||
|
||||
- [2026.10] - **[ReMe Multimodal Memory](https://reme.agentscope.io/en/blog_20261009) update**: Auto Memory reads images
|
||||
with their conversation context and preserves image sources; Auto Resource turns image files into searchable daily
|
||||
cards linked to the originals.
|
||||
- [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=en) is live**: explore example memory
|
||||
files, edit Markdown, and browse linked memory graphs right in your browser—no installation or backend required.
|
||||
Everyone is welcome to [try it out](https://reme.agentscope.io/studio/?lang=en)!
|
||||
|
|
|
|||
|
|
@ -47,6 +47,8 @@
|
|||
|
||||
## 📰 最新动态
|
||||
|
||||
- [2026.10] - **[ReMe 多模态记忆更新](https://reme.agentscope.io/zh/blog_20261009)**：Auto Memory 将会话图片和上下文一起
|
||||
写入记忆并保留图片来源；Auto Resource 将图片资料整理为可检索、可回溯原图的 daily 卡片。
|
||||
- [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=zh) 上线**：无需安装或启动后端，
|
||||
即可在浏览器中浏览示例记忆文件、编辑 Markdown、探索记忆关联图谱。欢迎大家[来体验](https://reme.agentscope.io/studio/?lang=zh)！
|
||||
|
||||
|
|
|
|||
|
|
@ -226,6 +226,7 @@ function singlePageSidebar(language: "zh" | "en", page: "blog" | "faq"): Default
|
|||
items: [
|
||||
{ text: zh ? "ReMe介绍" : "About ReMe", link: `/${language}/reme-blog` },
|
||||
{ text: zh ? "记忆标签" : "Memory Tags", link: `/${language}/blog_20260920` },
|
||||
{ text: zh ? "图片记忆" : "Image Memory", link: `/${language}/blog_20261009` },
|
||||
],
|
||||
}];
|
||||
}
|
||||
|
|
@ -248,7 +249,7 @@ function sidebars(language: "zh" | "en"): DefaultTheme.SidebarMulti {
|
|||
[`/${language}/plugin_development`]: pluginsSidebar(language),
|
||||
[`/${language}/benchmarks/`]: benchmarksSidebar(language),
|
||||
[`/${language}/reme-blog`]: singlePageSidebar(language, "blog"),
|
||||
[`/${language}/blog_20260920`]: singlePageSidebar(language, "blog"),
|
||||
[`/${language}/blog_`]: singlePageSidebar(language, "blog"),
|
||||
[`/${language}/faq`]: singlePageSidebar(language, "faq"),
|
||||
[`/${language}/`]: docsSidebar(language),
|
||||
};
|
||||
|
|
|
|||
131
docs/en/blog_20261009.md
Normal file
131
docs/en/blog_20261009.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# Bringing Images into Memory
|
||||
|
||||
When discussing a system design with an agent, we might send an architecture diagram and use it to decide which modules to change and which designs to keep.
|
||||
|
||||
When we later ask, "Why did we keep the task queue last time?", the agent needs to find the earlier discussion and revisit the relationships shown in the diagram.
|
||||
|
||||
**This multimodal memory update lets Auto Memory and Auto Resource extract information from images and preserve their sources.**
|
||||
|
||||
Images in a conversation can become memories together with the surrounding discussion, while images in the resource directory can become standalone caption cards. Both paths write to the existing `daily/` memory layer.
|
||||
|
||||
## Auto Memory: Remember the Image and the Discussion
|
||||
|
||||
Suppose you send an agent an architecture diagram and add:
|
||||
|
||||
> "Only change the Gateway this time. Leave the Worker and PostgreSQL as they are. Keep the task queue so that traffic spikes do not hit the database directly."
|
||||
|
||||
With image input enabled, Auto Memory sends the diagram and the discussion to a vision model together. Text and images stay in their original conversation order, with the speaker and timestamp included. The agent can use the relationships shown in the diagram to record the scope of the change, the reason for keeping the queue, and the constraints on later work.
|
||||
|
||||
Auto Memory reads images directly from the conversation. The host application supplies them as message content blocks and passes `include_images=true` when calling `auto_memory` to include them in that round of memory generation.
|
||||
|
||||
### Visual Facts Have Sources Too
|
||||
|
||||
Conversation images can be uploaded as Base64 data or supplied as HTTP(S) URLs. ReMe preserves their sources as follows:
|
||||
|
||||
- The original bytes of Base64 images are saved under `session/images/<session_id>/` by default for later viewing; the directory follows the `session_dir` configuration.
|
||||
- URL images keep their original addresses. The model service reads them; ReMe does not download them locally.
|
||||
|
||||
Auto Memory passes each image's exact source to the agent alongside the image. Its prompt asks the agent to cite the source next to relevant visual facts, and the daily memory's `source_images` frontmatter records these links as well. Later updates merge new sources while preserving existing links. A reader can follow a fact back to the image it came from.
|
||||
|
||||
Here is an illustrative memory after uploading a Base64 architecture diagram:
|
||||
|
||||
```markdown
|
||||
Only the Gateway will change this time. The Worker and PostgreSQL remain unchanged.
|
||||
The architecture diagram shows the Gateway connected to the Worker through a task queue.
|
||||
See [[session/images/design-review/msg-61-image-1.png]].
|
||||
|
||||
The task queue is retained to prevent traffic spikes from hitting the database directly.
|
||||
```
|
||||
|
||||
This records both a relationship visible in the diagram and a decision made explicitly in the discussion. `source_conversation` continues to point to the conversation record under `session/dialog/`, while image sources are preserved separately.
|
||||
|
||||
These conversation attachments stay under `session/`. They do not enter resource watching or produce a separate image caption card.
|
||||
|
||||
## Auto Resource: Turn Image Resources into Searchable Cards
|
||||
|
||||
Some images have no accompanying conversation: a whiteboard photo, a saved report screenshot, or a flowchart you want to consult later. You can place them directly in the workspace's `resource/` directory.
|
||||
|
||||
Auto Resource can now use a vision model to read these images and write their visible content into daily cards. Image inputs use the same agent memory-writing tools as text resources, and the original images stay in the resource directory.
|
||||
|
||||
For example, place a flowchart at:
|
||||
|
||||
```text
|
||||
resource/2026-10-09/system-overview.png
|
||||
```
|
||||
|
||||
Here is an illustrative image resource card:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: System request processing flow
|
||||
description: The Gateway connects to the Worker through a task queue, and the Worker writes results to PostgreSQL.
|
||||
kind: image
|
||||
media_type: image/png
|
||||
source_resource: "[[resource/2026-10-09/system-overview.png]]"
|
||||
---
|
||||
|
||||
![[resource/2026-10-09/system-overview.png]]
|
||||
|
||||
## Caption
|
||||
|
||||
The diagram contains a Gateway, a task queue, a Worker, and PostgreSQL.
|
||||
Requests enter the task queue through the Gateway. The Worker consumes them
|
||||
and writes the results to PostgreSQL.
|
||||
```
|
||||
|
||||
The card brings the embedded image, its description, and its source link together. Text in screenshots, numbers in charts, and relationships between modules can all become part of the caption. The content depends on the image and the vision model used.
|
||||
|
||||
Image resources then participate in the existing retrieval workflow. A later search for "Which modules does the task queue connect?" can find the relevant image through the card's text and follow `source_resource` back to the original file. When a resource image changes, its card is updated. With image processing enabled, deleting the original image removes its resource card.
|
||||
|
||||
Auto Resource supports PNG, JPG/JPEG, WebP, GIF, BMP, TIFF, and HEIC. Larger images may be resized or converted for the model request; the original resource files remain unchanged. HEIC requires the optional `reme-ai[image-heif]` package.
|
||||
|
||||
## Getting Started
|
||||
|
||||
Both paths require a vision model that supports image input and tool calling, along with a compatible AgentScope wrapper and formatter. The selected wrapper's `as_llm` binding determines which model is used. See [Configuration](./configuration.md) for credential setup.
|
||||
|
||||
Using ReMe's default configuration with an OpenAI-compatible service, replace `YOUR_VISION_MODEL` with the model name provided by the service and start ReMe:
|
||||
|
||||
```bash
|
||||
reme start \
|
||||
components.as_llm.default.backend=openai \
|
||||
components.as_llm.default.model=YOUR_VISION_MODEL \
|
||||
components.as_llm.default.parameters.max_tokens=4096
|
||||
```
|
||||
|
||||
Each path has its own image setting:
|
||||
|
||||
| Path | Default | How to enable images |
|
||||
|---|---|---|
|
||||
| Auto Memory | `include_images=false` | Pass `include_images=true` when calling `auto_memory` |
|
||||
| Auto Resource | `include_images=true` | Resource watching processes images by default; manual calls can also pass the parameter explicitly |
|
||||
|
||||
In another terminal, call the service for the same workspace. The following example shows the message format for a conversation image. Replace the URL with a PNG image address that the model service can access:
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=design-review include_images=true messages='[
|
||||
{
|
||||
"id": "a",
|
||||
"name": "user",
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Only change the Gateway this time. Keep the task queue so that traffic spikes do not hit the database directly."},
|
||||
{"type": "data", "source": {
|
||||
"type": "url",
|
||||
"media_type": "image/png",
|
||||
"url": "https://example.com/system-overview.png"
|
||||
}}
|
||||
]
|
||||
}
|
||||
]'
|
||||
```
|
||||
|
||||
Submit local images as Base64 message content blocks. When resubmitting Base64 images, keep the session ID, message IDs, and image positions within each message stable. Existing attachments are reused without being overwritten. Use a new message ID when replacing an image. See [Auto Memory](./auto_memory.md) for the full input and attachment rules.
|
||||
|
||||
For images already in the resource directory, wait for resource watching to process them or submit the change manually:
|
||||
|
||||
```bash
|
||||
reme auto_resource include_images=true \
|
||||
changes='[{"path":"resource/2026-10-09/system-overview.png","change":"added"}]'
|
||||
```
|
||||
|
||||
Image-count limits imposed by the model and wrapper still apply. See [Auto Resource](./auto_resource.md) for detailed configuration, format requirements, and processing rules.
|
||||
130
docs/zh/blog_20261009.md
Normal file
130
docs/zh/blog_20261009.md
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
# 让图片进入记忆
|
||||
|
||||
和 Agent 讨论系统设计时，我们会发一张架构图，再对照着图确认：哪些模块要改，哪些设计先保留。
|
||||
|
||||
以后再问“上次为什么保留任务队列”，Agent 需要找回当时的讨论，也需要回看图中的模块关系。
|
||||
|
||||
**ReMe 的这次多模态记忆更新，让 Auto Memory 和 Auto Resource 都能从图片中提取信息，并保留图片来源。**
|
||||
|
||||
会话中的图片可以和上下文一起形成记忆，资源目录里的图片可以生成独立的描述卡片。两条路径都写入现有的 `daily/` 记忆层。
|
||||
|
||||
## Auto Memory：把图片和当时的讨论一起记住
|
||||
|
||||
假设你给 Agent 发了一张系统架构图，并补充：
|
||||
|
||||
> “这次只调整 Gateway，Worker 和 PostgreSQL 先不动。任务队列仍然保留，避免流量高峰时直接压到数据库。”
|
||||
|
||||
开启图片输入后，Auto Memory 会把图和这段讨论一起交给视觉模型。文字与图片保留在原来的对话顺序中，说话人和时间也会随之传入。这样，Agent 可以结合图中的模块关系，记录本次调整的范围、保留队列的原因和后续约束。
|
||||
|
||||
它直接读取会话里的图片。宿主只需把图片作为消息内容块传入，并在调用 `auto_memory` 时设置 `include_images=true`，就能让图片参与这一轮记忆生成。
|
||||
|
||||
### 视觉事实也有来源
|
||||
|
||||
对话图片既可能是上传的 Base64 数据，也可能是 HTTP(S) URL。ReMe 对它们分别保留来源：
|
||||
|
||||
- Base64 图片的原始字节默认保存在 `session/images/<session_id>/`，目录随 `session_dir` 配置，供以后查看；
|
||||
- URL 图片保留原始地址，由模型服务读取，不下载到本地。
|
||||
|
||||
Auto Memory 会把每张图片的准确来源随图片一起传给 Agent，并在提示词中要求它在相关视觉事实旁引用来源。每日记忆的 `source_images` frontmatter 也会记录这些链接，后续更新会保留并合并已有来源。读到一条来自图片的事实时，可以顺着链接回到原图。
|
||||
|
||||
上传 Base64 架构图后的记忆内容示意：
|
||||
|
||||
```markdown
|
||||
这次只调整 Gateway，Worker 和 PostgreSQL 保持不变。
|
||||
架构图中的 Gateway 通过任务队列连接 Worker。
|
||||
见 [[session/images/design-review/msg-61-image-1.png]]。
|
||||
|
||||
保留任务队列的原因是避免高峰流量直接压到数据库。
|
||||
```
|
||||
|
||||
这里既有从图中读出的关系，也有用户在讨论中明确作出的决定。会话来源继续由 `source_conversation` 指向 `session/dialog/` 下的记录，图片来源则单独保留。
|
||||
|
||||
这些会话附件留在 `session/` 中，不会进入资源监听流程，也不会再生成一张独立的图片描述卡。
|
||||
|
||||
## Auto Resource：把图片资料整理成可检索的卡片
|
||||
|
||||
有些图片没有对应的对话：一张白板照片、一份保存下来的报表截图，或者一张以后还要查阅的流程图。它们可以直接放进工作区的 `resource/`。
|
||||
|
||||
Auto Resource 现在可以使用视觉模型解读这些图片，把可见内容整理成 daily 卡片。图片输入与文本资源共用 Agent 的记忆写入工具，原图保留在资源目录里。
|
||||
|
||||
例如，把一张流程图放在：
|
||||
|
||||
```text
|
||||
resource/2026-10-09/system-overview.png
|
||||
```
|
||||
|
||||
图片资源卡片示意：
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: 系统请求处理流程
|
||||
description: Gateway 通过任务队列连接 Worker，Worker 将结果写入 PostgreSQL。
|
||||
kind: image
|
||||
media_type: image/png
|
||||
source_resource: "[[resource/2026-10-09/system-overview.png]]"
|
||||
---
|
||||
|
||||
![[resource/2026-10-09/system-overview.png]]
|
||||
|
||||
## Caption
|
||||
|
||||
图中包含 Gateway、任务队列、Worker 和 PostgreSQL。
|
||||
请求从 Gateway 进入任务队列，由 Worker 消费并将结果写入 PostgreSQL。
|
||||
```
|
||||
|
||||
卡片把原图嵌入、内容描述和来源链接放在一起。截图中的文字、图表中的数字和模块之间的关系，都可以成为描述的一部分；内容取决于图片本身和所用视觉模型。
|
||||
|
||||
图片资料由此进入现有检索流程。以后搜索“任务队列连接哪些模块”，ReMe 可以通过卡片中的文字找到相关图片，再沿 `source_resource` 回看原文件。资源图片更新时，对应卡片也会更新；开启图片处理时，删除原图会移除对应的资源卡片。
|
||||
|
||||
Auto Resource 支持 PNG、JPG/JPEG、WebP、GIF、BMP、TIFF 和 HEIC。较大的图片会在模型请求中按需缩小或转换格式，原始资源文件保持不变。HEIC 需要额外安装 `reme-ai[image-heif]`。
|
||||
|
||||
## 如何开始使用
|
||||
|
||||
两个入口都需要支持图片和工具调用的视觉模型，以及兼容的 AgentScope wrapper 和 formatter。模型由所选 wrapper 的 `as_llm` 绑定决定。凭证配置见[基础配置](./configuration.md)。
|
||||
|
||||
以 ReMe 默认配置和 OpenAI 兼容服务为例，把 `YOUR_VISION_MODEL` 换成服务提供的模型名称后启动 ReMe：
|
||||
|
||||
```bash
|
||||
reme start \
|
||||
components.as_llm.default.backend=openai \
|
||||
components.as_llm.default.model=YOUR_VISION_MODEL \
|
||||
components.as_llm.default.parameters.max_tokens=4096
|
||||
```
|
||||
|
||||
图片开关分别作用于两个入口：
|
||||
|
||||
| 入口 | 默认值 | 启用方式 |
|
||||
|---|---|---|
|
||||
| Auto Memory | `include_images=false` | 调用 `auto_memory` 时传入 `include_images=true` |
|
||||
| Auto Resource | `include_images=true` | 资源监听默认处理图片；手动调用也可显式传入该参数 |
|
||||
|
||||
在另一个终端中调用同一工作区的服务。下面展示会话图片的消息格式；将 URL 换成模型服务可访问的 PNG 图片地址：
|
||||
|
||||
```bash
|
||||
reme auto_memory session_id=design-review include_images=true messages='[
|
||||
{
|
||||
"id": "a",
|
||||
"name": "user",
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "这次只调整 Gateway。保留任务队列，避免高峰流量直接压到数据库。"},
|
||||
{"type": "data", "source": {
|
||||
"type": "url",
|
||||
"media_type": "image/png",
|
||||
"url": "https://example.com/system-overview.png"
|
||||
}}
|
||||
]
|
||||
}
|
||||
]'
|
||||
```
|
||||
|
||||
本地图片应使用 Base64 消息内容块提交。重复提交 Base64 图片时，保持会话 ID、消息 ID 和图片在消息中的位置稳定；已有附件会直接复用，不会覆盖。替换图片应使用新的消息 ID。完整输入与附件规则见 [Auto Memory](./auto_memory.md)。
|
||||
|
||||
对于已放入资源目录的图片，可以等待资源监听处理，也可以手动提交：
|
||||
|
||||
```bash
|
||||
reme auto_resource include_images=true \
|
||||
changes='[{"path":"resource/2026-10-09/system-overview.png","change":"added"}]'
|
||||
```
|
||||
|
||||
模型和 wrapper 的图片数量限制仍然生效。资源图片的详细配置、格式要求与处理规则见 [Auto Resource](./auto_resource.md)。
|
||||
|
|
@ -58,6 +58,8 @@ const requiredFiles = [
|
|||
"en/reme-blog.html",
|
||||
"zh/blog_20260920.html",
|
||||
"en/blog_20260920.html",
|
||||
"zh/blog_20261009.html",
|
||||
"en/blog_20261009.html",
|
||||
"zh/services.html",
|
||||
"en/services.html",
|
||||
"studio/index.html",
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue