mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
Add JSONL support, reorganize vault structure, and update design docs (#274)
* feat(config): add jsonl support and update LLM integration tests - Added jsonl extension to supported extensions in chunked backend - Refactored LLM integration tests to use async functions instead of nested runs - Created helper functions _run_basic_chat, _run_with_tool, _run_structured_output, _run_structured_output_enum - Implemented _run_all function to execute all test scenarios sequentially - Updated main execution block to use asyncio.run with consolidated test runner - Maintained all original test functionality while improving code structure * feat(config): add dialog directory configuration and reorganize vault structure - Add new dialog_dir field for dialog memory storage - Reorder directory initialization sequence in application setup - Simplify vault_dir description in configuration schema - Update thread_pool_max_workers description to be more concise - Move resource_dir definition earlier in the configuration schema - Remove redundant text from digest_dir description * docs(reme): update design documentation with layered memory architecture - Replace quick test section with comprehensive layered memory structure - Add detailed explanation of three-tier memory organization (resource, daily, digest) - Document Obsidian-compatible Markdown format with YAML front matter - Describe four types of wikilink syntax and semantic linking features - Explain AST-aware semantic chunking for document parsing - Detail self-evolving system with auto-resource, auto-memory, and auto-dream - Document directory structure and lifecycle characteristics - Add comprehensive table showing content nature, triggers, and examples - Include semantic link extraction and knowledge graph formation processes - Describe automated indexing and relationship building workflows * docs(reme): update design documentation with simplified structure and clearer explanations - Simplified directory structure overview with cleaner formatting - Updated memory layering explanation with more concise descriptions - Improved table layouts for better readability - Clarified auto-resource, auto-memory, and auto-dream processes - Streamlined indexing and search mechanism descriptions - Enhanced component system documentation with clearer backend options - Refined job list with more precise functional descriptions - Modernized layout diagrams and process flows - Consolidated repetitive content while maintaining comprehensive coverage * docs(reme): add application scenario documentation for financial industry use case - Document comprehensive example of ReMe usage in新能源 industry research - Detail the week-long process of automatic knowledge graph construction - Explain the auto-memory and auto-dream pipeline with concrete examples - Describe the extract and integrate phases for creating wiki nodes - Illustrate cross-file linking through relates_to and derived_from predicates - Show progressive graph growth from daily sessions to complete ecosystem - Demonstrate hybrid retrieval with vector and keyword search capabilities - Provide detailed directory structure and file organization patterns - Explain the three-phase workflow: ingestion, processing, and retrieval - Document the financial analyst persona and their information management needs * style(config): fix spacing in thread_pool_max_workers field definition - Corrected spacing around description parameter in Field definition - Simplified multi-line assertion to single line in LLM integration test
This commit is contained in:
jinliyl
GitHub
parent
36a5512fc8
commit
cee2c3e338
17 changed files with 911 additions and 281 deletions
55
docs4/old/reme4_resource.md
Normal file
55
docs4/old/reme4_resource.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
- 增加agent 的component
|
||||
- 增加全局 时区time_zone，全局作用
|
||||
|
||||
|
||||
数据结构
|
||||
- resource
|
||||
- YYYYMMDD
|
||||
- xxxx.html
|
||||
- xxxx.txt
|
||||
- xxxx.md
|
||||
- dialog
|
||||
- YYYYMMDD
|
||||
- session_{session_id}.jsonl msg->dict格式
|
||||
- daily
|
||||
- YYYYMMDD.md 索引
|
||||
- YYYYMMDD
|
||||
- session_{session_id}.md 日志本
|
||||
- resource_{resource_id}.md
|
||||
- digest
|
||||
- personal 个性化信息
|
||||
- xxx.md
|
||||
- procedural 程序化记忆
|
||||
- xxx.md
|
||||
- wiki 知识化记忆
|
||||
- xxx.md
|
||||
|
||||
|
||||
监控任务【后台】：
|
||||
- 构建bm25+emb的index【5s】
|
||||
- 目录：daily+digest 的 所有md，使用markdown的ast解析
|
||||
- 可选：jsonl，要求保存的时候对工具结果截断，使用rag方案解析
|
||||
- daily md监控：【1min】
|
||||
- 暂无
|
||||
- digest md监控：【1min】
|
||||
- 暂无
|
||||
- resource 监控：【间隔5min】
|
||||
- 针对每一个文件，生成一个hashid，作为session_id
|
||||
- 生成一个agent，解读文件（读前1MB内容）防止上下文炸了
|
||||
- 把抽取的内容写到daily/YYYYMMDD/resource_{resource_id}.md
|
||||
- 调用推送工具：
|
||||
- qwenpaw推送收件箱/agent
|
||||
- cc可以直接推送agent @sen
|
||||
|
||||
|
||||
hook任务：
|
||||
- auto-memory：在上下文满/每隔多少轮/session_end
|
||||
- qwenpaw：直接传message session_id date，直接append session_{session_id}.jsonl @jinli
|
||||
- cc：给出新的path，对比我们保存的session_{session_id}.jsonl，看到增量msg，接着解析path的内容，保存到session_{session_id}.jsonl @sen
|
||||
- 注意保存的时候截断工具调用结构，防止太长
|
||||
|
||||
|
||||
定时任务：
|
||||
- auto-dream：每天夜晚触发
|
||||
- 记忆整理：按照session/resource_id去做for循环整理：把YYYYMMDD.md + YYYYMMDD/* 消化更新到 digest下的personal/procedural/wiki
|
||||
- 记忆link：markdown之间link，digest内部链接，可以链接到外面。
|
||||
192
docs4/old/reme_design.md
Normal file
192
docs4/old/reme_design.md
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
# 快速测试
|
||||
|
||||
```bash
|
||||
# 终端 A：启动服务
|
||||
reme4 start
|
||||
|
||||
# 终端 B：调用 version 验证服务可用
|
||||
reme4 version
|
||||
# 预期输出：✅ ReMe v{__version__}
|
||||
```
|
||||
|
||||
# 基础Job
|
||||
|
||||
@jinli
|
||||
|
||||
入口：`reme4/reme.py::main()` → `parse_args(*sys.argv[1:])` 解析首个位置参数为 `action`，后续 `key=value` 解析为 kwargs（支持
|
||||
`service.port=8080` 的 dot notation；自动剥离 `--` / `-` 前缀；值会做 bool / int / float / JSON 转换）。
|
||||
|
||||
调用模式：
|
||||
|
||||
- `start`：本地启动 `ReMe(Application)` 服务（不经过 client）
|
||||
- `find_reme`：本地探测正在运行的 reme，不调用服务
|
||||
- `list`：在 client 端拦截，不转发到服务端，直接返回 action 目录
|
||||
- 其他 action：通过 `call_server(action, **kwargs)` → `R.get(ComponentEnum.CLIENT, backend)` 实例化客户端并流式打印（任意未列出的
|
||||
step register name 都按本规则透传）
|
||||
|
||||
通用可选参数 `backend:str=http`（取值 `http` / `mcp`，对应 `reme4/components/client/{http_client,mcp_client}.py` 中
|
||||
`@R.register` 注册名）；服务端默认 host/port 见 `reme4/constants.py`，可由 `start` 端通过 `service.host=` / `service.port=`
|
||||
覆盖。
|
||||
|
||||
说明：📥 输入参数 ｜ 📤 输出 ｜ ⭐ 必填 ｜ 🎚️ 默认值 ｜ 🛠️ 内部行为 ｜ 📊 metadata
|
||||
|
||||
| 分类 | 指令 (register name) | 入口 | 参数 & 行为 |
|
||||
|------------|--------------------------------------------------|-------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| 🚀 本地 | 🟢 `start` | `reme.py:30` → `ReMe(**kwargs).run_app()` | 📥 可选 `config=<name\|path>`（默认加载 `reme4/config/default.yaml`，`.yaml/.yml/.json` 都支持，含 `${ENV:-default}` 占位符）｜ 可选 `service.host=` / `service.port=` 等任意 dot-notation 覆盖 ｜ 🛠️ 流程：`load_env()` → `resolve_app_config(**kwargs)` deep merge → `precheck_start(svc)`（`utils/service_utils.py:72`：目标 host:port 已有 reme → 打印 `reme already running ...` 直接返回；端口被其他进程占用 → stderr 提示 `port {port} occupied. Start on another port: reme4 start service.port=<other_port>` 并 `sys.exit(1)`）→ 启动服务 |
|
||||
| 🚀 本地 | 🧭 `find_reme` | `reme.py:36` → `utils/service_utils.py:89` | 📥 无 ｜ 📤 发现服务则 stdout 打印 `HOST={host} PORT={port} PID={pid or 'unknown'}`；未发现则 stderr 提示 `reme not started. Try: reme start` 并 `sys.exit(1)` ｜ 🛠️ 流程：先探 `REME_DEFAULT_HOST:REME_DEFAULT_PORT`（`health_check` 命中算 `reme`），再 `pgrep -af "reme.* start"` 扫描其他端口 |
|
||||
| 🛰️ 客户端 | 📜 `list` | `components/client/base_client.py:36` | 📥 无 ｜ 📤 服务端可用 action 目录（JSON，`indent=2 ensure_ascii=False`）｜ 🛠️ 在 `BaseClient.__call__` 中拦截，不进入 `_execute`，直接调用 `list_actions()`（HTTP/MCP backend 各自实现） |
|
||||
| 🌐 通用 step | 🆘 `help` (`help_step`) | `call_server("help")` | 📥 无 ｜ 📤 `answer` 一行一个 job：`🛠️ \`{name}\` — {description} 📥 {params}`，参数渲染为 `name:type*`(必填) / `name:type={default}` / `name:type` ｜ 📊 `metadata.job_count` ｜ 🛠️ 自动跳过名为 `help` 的 job |
|
||||
| 🌐 通用 step | 🩺 `health_check` (`health_check_step`) | `call_server("health_check")` | 📥 无 ｜ 📤 `answer = "✅/❌ ReMe v{version} - healthy/unhealthy"` ｜ 📊 `metadata.health = {version, healthy, components}` ｜ 🧩 覆盖组件：`embedding_model`(🟢 is_started/is_healthy/model_name/dimensions/cache_size/memory) · `file_graph`(🕸️ n_nodes/n_edges/n_virtual\|n_pending/memory) · `file_store`(📦 n_chunks/n_chunks_with_embedding/memory) · `file_watcher`(👀 background_running/watch_paths) · `keyword_index`(🔤 n_docs/vocab_size/memory) ｜ 🛠️ deep sizeof（含 numpy.nbytes），未启动 / 后台未跑 / embedding 不健康 → ❌ |
|
||||
| 🌐 通用 step | 🏷️ `version` (`version_step`) | `call_server("version")` | 📥 无 ｜ 📤 `answer = reme4.__version__` ｜ 📊 `metadata.version` |
|
||||
| 🌐 通用 step | 🔄 `reindex` (`reindex_step`) | `call_server("reindex")` | 📥 无 ｜ 📤 `answer = "🔄 Reindexed {added} file(s)"` ｜ 📊 `metadata.counts = {added, ...}` ｜ 🛠️ 流程：`file_watcher.close()` → `file_store.clear()` → `file_watcher.update_store()` → `file_watcher.start()`（finally 保证重启） |
|
||||
| 🔎 search | 🔍 `search` (`search_step`) | `call_server("search", query=…, …)` | 📥 `query:str` ⭐ ｜ 🎚️ `limit:int=5`(>0) ｜ 🎚️ `min_score:float=0.0` ｜ ⚖️ `vector_weight:float=0.7` ∈[0,1]（keyword 权 = 1-vw）｜ 🔀 `candidate_multiplier:float=3.0`（candidates = min(200, limit×mult)）｜ 🔗 `expand_links:bool=True` ｜ 🔢 `max_links_per_direction:int=10` ｜ 🎚️ `search_filter:dict={}` ｜ 📤 `answer` 每命中一行 `path:start-end [score=… vector=… keyword=…] text` + 缩进的 `→ outlinks (n)` / `← inlinks (n)` + `via predicate=… anchor=#…` ｜ 📊 `metadata.results` / `metadata.link_expansion` / `metadata.counts={vector,keyword,returned,hybrid}` ｜ 🛠️ 并行 `vector_search` + `keyword_search` → RRF 融合（K=60，按 chunk.id 合并）→ `min_score` 过滤 → `limit` 截断 → 邻居 meta 注入 |
|
||||
| 🧪 demo | 🪄 `demo_echo` (`demo_echo_step1` + `step2`) | `call_server("demo_echo", query=…, min_score=…)` | 📥 `query:str=""` ｜ 🎚️ `min_score:float=0.5` ｜ 🛠️ step1：`processed_query = query.strip().lower()`，`adjusted_min_score = min_score * 0.9`，写回 context ｜ 📤 step2：`answer = "echo: {processed_query} (min_score={adjusted_min_score})"` ｜ 📊 `metadata = {step, query, min_score, processed_query, adjusted_min_score}` |
|
||||
| 🌊 demo | 🌊 `stream_demo` (`stream_demo_step1` + `step2`) | `call_server("stream_demo", query=…, repeat=…, interval=…)` | 📥 `query:str=""` ｜ 🎚️ `repeat:int=10` ｜ 🎚️ `interval:float=0.1`（秒/字符）｜ 🛠️ step1：`stream_text = query * repeat` 写回 context ｜ 📤 step2：按字符 `add_stream_string(ch, ChunkEnum.CONTENT)` 流式输出，`asyncio.sleep(interval)` 节流 |
|
||||
| 📂 crud | 📖 `read` (`read_step`) | `call_server("read", path=…, …)` | 📥 `path:str` ⭐（**完整相对路径**，相对于 vault；绝对路径会被拒绝；非 `.md` 后缀拒绝）｜ 🎚️ `start_line:int=null`（1-based, 含端点）｜ 🎚️ `end_line:int=null`（1-based, 含端点）｜ 🎚️ `max_bytes:int=51200`（截断阈值）｜ 📤 `answer = 选中的行内容`，超过 `max_bytes` 时附加 `--- TRUNCATED ---` 续读指引（`start_line=…`）｜ 📊 `metadata.path` / `metadata.total_lines`（出错路径才会附带）｜ 🛠️ 流程：`BaseStep.resolve_path(raw, require_md=True)` → `aiofiles.os.stat` → `read_file_safe`（utf-8-sig BOM 容忍、UnicodeDecodeError fallback `errors=ignore`）→ `split("\n")` 切片 `[s-1:e]` → `truncate_text_output` 按字节截断保行 |
|
||||
|
||||
使用示例：
|
||||
|
||||
```bash
|
||||
# 启动（默认 default.yaml）
|
||||
reme4 start
|
||||
|
||||
# 指定 config 与服务端口
|
||||
reme4 start config=paw.yaml service.port=8181
|
||||
|
||||
# 查找在跑的 reme
|
||||
reme4 find_reme
|
||||
# HOST=127.0.0.1 PORT=8000 PID=12345
|
||||
|
||||
# 列出所有可用 action（client 端处理，不转服务端）
|
||||
reme4 list
|
||||
|
||||
# 转发到服务端的 step：所有 key=value 透传为 step kwargs
|
||||
reme4 help
|
||||
reme4 health_check
|
||||
reme4 version
|
||||
reme4 reindex
|
||||
reme4 search query="latency 问题" limit=10 min_score=0.2 vector_weight=0.6
|
||||
|
||||
# 读取 vault 下的 markdown（完整相对路径；无后缀自动补 .md；可按行切片或限制字节）
|
||||
reme4 read path=Templates/Recipe.md
|
||||
reme4 read path=Notes start_line=1 end_line=20
|
||||
reme4 read path=Big.md max_bytes=4096
|
||||
|
||||
# 通过 MCP backend 调用
|
||||
reme4 search query="..." backend=mcp
|
||||
```
|
||||
|
||||
@sen
|
||||
| file | upload/download/move/delete/stat/list | 文件操作CRUD |
|
||||
| property | read/update/delete | frontmatter CRUD | |
|
||||
| graph | traverse/retarget | path="My Note" directtion=forward/backward depth=1 predicat=xxx |
|
||||
|
||||
@wangce
|
||||
| crud | write | path="New Note" name="xxx" description="xxx" metadata={}, content="# Hello" (4 字段都必填，frontmatter 只写 name/description) |
|
||||
| crud | read | path="Templates/Recipe.md" |
|
||||
| crud | edit | path="Templates/Recipe.md" old="xxx" new="xxx" |
|
||||
| crud | append | path="My Note" content="New line" |
|
||||
|
||||
| crud | delete | path="My Note
|
||||
| daily_crud | daily_xxx | 与 crud 参数保持一致 |
|
||||
|
||||
- daily_resolve name=xxxx (符合一定规范 win下要求)
|
||||
- daily_list date=xxxx 返回path
|
||||
- daily_index
|
||||
|
||||
frontmatter read path
|
||||
frontmatter update path metadata={}
|
||||
frontmatter delete path keys=[]
|
||||
|
||||
delete path
|
||||
download path=xxx（内部相对路径）download_path=(外部绝对路径，可选)
|
||||
upload path=xxx（外部绝对路径）description="xxx" metadata=xxx 返回内部相对路径 加metadata
|
||||
stat path
|
||||
list path
|
||||
mv path=xxx new_path=xxx
|
||||
|
||||
traverse path=xxx direction=xxx depth=xxx
|
||||
|
||||
# 日记类型
|
||||
|
||||
| 类型 | 路径 | 说明 |
|
||||
|-----------|-----------------------------------------------|-----------------------------|
|
||||
| daily | {daily}/xxxx-mm-dd.md + xxxx-mm-dd/{event}.md | 按日期归档的原始信息记录 |
|
||||
| topic | topic/{topic:-personal(agent)}/{xxxx}.md | 按主题聚类的二次加工内容 |
|
||||
| proactive | todo | 基于 daily / topic 思考后主动推送的消息 |
|
||||
|
||||
# 生成Job
|
||||
|
||||
| 任务 | 输入 | 输出 | 触发时机 | 说明 |
|
||||
|-------------------------|---------------|-----------------------------------------------|-----------------------------|------------------------------------------------------|
|
||||
| 日记summary @sen @wangce | msg | {daily}/xxxx-mm-dd.md + xxxx-mm-dd/{event}.md | freq (every_n_turn、compact) | 把 msg 的信息写入 daily 目录 |
|
||||
| 主题dream + 生成链接 @sen | daily/xxx | knowledge/xxx | /dream | 把 daily 目录的内容按主题聚类合并到 topic 目录, 主动在文档中建立 [[link]] 关联 |
|
||||
| 主动proactive @wangce | daily / topic | proactive_query | pre_query | 思考 daily / topic 信息，主动决定推送给用户的消息 |
|
||||
|
||||
2. file_parser
|
||||
a. 抽象基类 parse: @jinli
|
||||
ⅰ. 输入是path：相对路径
|
||||
ⅱ. 输出是FileMetadata & list[FileChunks] & list[FileEdge]
|
||||
b. default parser 兼容老方案 @jinli
|
||||
ⅰ. 带overlap的chunking策略 ，不输出FileEdge
|
||||
c. markdown parser @sen
|
||||
ⅰ. 根据markdown ast做chunk，不需要overlap
|
||||
ⅱ. 增加一个索引的chunk chunk_type @锦鲤 file_chunk_type content/index
|
||||
ⅲ. 增加link的正则解析：predicate:: [[path#anchor]]
|
||||
3. file_store @sen
|
||||
a. 抽象存储：
|
||||
ⅰ. filenode = file + path + st_mtime + metadata + list[FileEdge]
|
||||
ⅱ. graph=dict[str, filenode] 内存+json
|
||||
ⅲ. list[FileChunk] 存db
|
||||
b. 抽象基类
|
||||
ⅰ. graph：fellow dict的操作 update/get/set
|
||||
ⅱ. chunks dict[str, list[chunk]]
|
||||
1. delete_chunks_by_path
|
||||
2. update_chunks_by_path
|
||||
3. list_chunks_by_path
|
||||
4. vector_search/keyword_search
|
||||
ⅲ. 手写一个bm25检索
|
||||
ⅳ. 【核心】检索机制 vector bm25 graph 如何进行融合
|
||||
4. file_watcher @jinli
|
||||
a. 抽象基类
|
||||
ⅰ. on_start:
|
||||
1. file_store 的start 在前，加载graph，file_watcher在后，递归扫描目录
|
||||
a. 通过ms_time对比graph，on_change 进行改动
|
||||
ⅱ. on_change:
|
||||
1. 更新/增加:
|
||||
a. delete_chunks_by_path 更新数据库
|
||||
b. upate_chunks_by_path 更新数据库
|
||||
c. 更新graph
|
||||
2. 删除
|
||||
a. delete_chunks_by_path 更新数据库
|
||||
|
||||
MemorySchema
|
||||
|
||||
1. markdown文件结构 @sen
|
||||
a. formatter：
|
||||
ⅰ. name
|
||||
ⅱ. desc
|
||||
2. memory文件结构目录
|
||||
a. MEMORY.md
|
||||
b. msg/files -> daily/YYYYMMDD/YYYYMMDD.md + xxxx.md
|
||||
ⅰ. YYYYMMDD.md
|
||||
1. xxx -> xxxx.md
|
||||
2. xxx -> xxxd.md
|
||||
ⅱ.
|
||||
c. daily -> topic/topic_l1/topic_l1.md + xxx.md + topic_l2
|
||||
d. proactive
|
||||
|
||||
steps:
|
||||
|
||||
1. 治理（算法+LLM）：
|
||||
a. 节点关联P0：现有的链接做补充，挖掘新的LLM的link
|
||||
ⅰ. /Users/yuli/workspace/ReMe/reme2/component/edge_extractor/llm_edge_extractor.py
|
||||
ⅱ. 移动到steps
|
||||
b. 节点整合/节点拆分/节点归档
|
||||
c. 健康度检查
|
||||
2. retrieve 调用store的检索
|
||||
3. 原子steps：reme edit
|
||||
4. 组合steps：总结：
|
||||
a. - freq (every_n_turn、compact) -> daily_summarizer
|
||||
b. topic (/dream ) -> topic_summarizer(daily_xx -> topic_xx)
|
||||
c. proactive -> proactive_summarizer(personal_xxx -> proactive_query - pre_query
|
||||
|
|
@ -1,192 +1,385 @@
|
|||
# 快速测试
|
||||
# ReMe 设计文档
|
||||
|
||||
```bash
|
||||
# 终端 A：启动服务
|
||||
reme4 start
|
||||
## 整体定位
|
||||
|
||||
# 终端 B：调用 version 验证服务可用
|
||||
reme4 version
|
||||
# 预期输出：✅ ReMe v{__version__}
|
||||
> 一句话总结：**自进化的个人知识库**——你只管往里扔东西和对话，它自己长成一张知识图谱。
|
||||
|
||||
## 特性1：记忆分层
|
||||
|
||||
记忆按"原始 → 浅加工 → 深加工"三层组织：
|
||||
|
||||
### 1.1 目录结构
|
||||
|
||||
```
|
||||
- resource/【原始素材】 # 外部渠道摄入 / 手动放入
|
||||
- YYYY-MM-DD/ # 按日期归档
|
||||
- session_{id}.jsonl # 对话原始记录
|
||||
- {channel}_{xxxx}.html # 网页抓取、邮件等
|
||||
- {channel}_{xxxx}.md # Markdown 资料
|
||||
- daily/【日记，浅加工】 # auto-memory 自动写入
|
||||
- YYYY-MM-DD.md # 当天索引页，汇总所有事件
|
||||
- YYYY-MM-DD/
|
||||
- session_{id}.md # 按 session 拆分的日志
|
||||
- resource_{id}.md # 对素材的加工笔记
|
||||
- digest/【深加工】 # auto-dream 持续打磨
|
||||
- personal/ # 用户偏好、习惯、身份
|
||||
- procedure/ # 方法论、步骤、工作流
|
||||
- wiki/ # 通用知识、决策先例
|
||||
```
|
||||
|
||||
# 基础Job
|
||||
### 1.2 分层详解
|
||||
|
||||
@jinli
|
||||
| 目录 | 存什么 | 谁写入 | 举例 |
|
||||
|---------------------|----------------|-------------|-----------------------------------|
|
||||
| `resource/` | 原始文件（研报、网页、邮件） | upload / 手动 | PDF 研报、对话 JSONL |
|
||||
| `daily/` | 每天的事件记录 | auto-memory | "调试登录 CSS"、"与 Alice 聚餐" |
|
||||
| `digest/procedure/` | 方法论、步骤 | auto-dream | "webpack 编译卡死排查路径" |
|
||||
| `digest/personal/` | 用户画像、偏好 | auto-dream | "用户不爱写注释"、"用户喜欢 pnpm" |
|
||||
| `digest/wiki/` | 通用知识、决策先例 | auto-dream | "光伏产业链"、"React Server Components" |
|
||||
|
||||
入口：`reme4/reme.py::main()` → `parse_args(*sys.argv[1:])` 解析首个位置参数为 `action`，后续 `key=value` 解析为 kwargs（支持
|
||||
`service.port=8080` 的 dot notation；自动剥离 `--` / `-` 前缀；值会做 bool / int / float / JSON 转换）。
|
||||
`resource/` 和 `daily/` 是只增不删的流水账；`digest/` 下三个桶是反复消费的精华层，各桶有独立的整合 prompt。
|
||||
|
||||
调用模式：
|
||||
## 特性2：Obsidian 兼容的 Markdown 格式
|
||||
|
||||
- `start`：本地启动 `ReMe(Application)` 服务（不经过 client）
|
||||
- `find_reme`：本地探测正在运行的 reme，不调用服务
|
||||
- `list`：在 client 端拦截，不转发到服务端，直接返回 action 目录
|
||||
- 其他 action：通过 `call_server(action, **kwargs)` → `R.get(ComponentEnum.CLIENT, backend)` 实例化客户端并流式打印（任意未列出的
|
||||
step register name 都按本规则透传）
|
||||
所有笔记都是标准 Markdown + Obsidian 语法，可以直接用 Obsidian 打开浏览：
|
||||
|
||||
通用可选参数 `backend:str=http`（取值 `http` / `mcp`，对应 `reme4/components/client/{http_client,mcp_client}.py` 中
|
||||
`@R.register` 注册名）；服务端默认 host/port 见 `reme4/constants.py`，可由 `start` 端通过 `service.host=` / `service.port=`
|
||||
覆盖。
|
||||
|
||||
说明：📥 输入参数 ｜ 📤 输出 ｜ ⭐ 必填 ｜ 🎚️ 默认值 ｜ 🛠️ 内部行为 ｜ 📊 metadata
|
||||
|
||||
| 分类 | 指令 (register name) | 入口 | 参数 & 行为 |
|
||||
|------------|--------------------------------------------------|-------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| 🚀 本地 | 🟢 `start` | `reme.py:30` → `ReMe(**kwargs).run_app()` | 📥 可选 `config=<name\|path>`（默认加载 `reme4/config/default.yaml`，`.yaml/.yml/.json` 都支持，含 `${ENV:-default}` 占位符）｜ 可选 `service.host=` / `service.port=` 等任意 dot-notation 覆盖 ｜ 🛠️ 流程：`load_env()` → `resolve_app_config(**kwargs)` deep merge → `precheck_start(svc)`（`utils/service_utils.py:72`：目标 host:port 已有 reme → 打印 `reme already running ...` 直接返回；端口被其他进程占用 → stderr 提示 `port {port} occupied. Start on another port: reme4 start service.port=<other_port>` 并 `sys.exit(1)`）→ 启动服务 |
|
||||
| 🚀 本地 | 🧭 `find_reme` | `reme.py:36` → `utils/service_utils.py:89` | 📥 无 ｜ 📤 发现服务则 stdout 打印 `HOST={host} PORT={port} PID={pid or 'unknown'}`；未发现则 stderr 提示 `reme not started. Try: reme start` 并 `sys.exit(1)` ｜ 🛠️ 流程：先探 `REME_DEFAULT_HOST:REME_DEFAULT_PORT`（`health_check` 命中算 `reme`），再 `pgrep -af "reme.* start"` 扫描其他端口 |
|
||||
| 🛰️ 客户端 | 📜 `list` | `components/client/base_client.py:36` | 📥 无 ｜ 📤 服务端可用 action 目录（JSON，`indent=2 ensure_ascii=False`）｜ 🛠️ 在 `BaseClient.__call__` 中拦截，不进入 `_execute`，直接调用 `list_actions()`（HTTP/MCP backend 各自实现） |
|
||||
| 🌐 通用 step | 🆘 `help` (`help_step`) | `call_server("help")` | 📥 无 ｜ 📤 `answer` 一行一个 job：`🛠️ \`{name}\` — {description} 📥 {params}`，参数渲染为 `name:type*`(必填) / `name:type={default}` / `name:type` ｜ 📊 `metadata.job_count` ｜ 🛠️ 自动跳过名为 `help` 的 job |
|
||||
| 🌐 通用 step | 🩺 `health_check` (`health_check_step`) | `call_server("health_check")` | 📥 无 ｜ 📤 `answer = "✅/❌ ReMe v{version} - healthy/unhealthy"` ｜ 📊 `metadata.health = {version, healthy, components}` ｜ 🧩 覆盖组件：`embedding_model`(🟢 is_started/is_healthy/model_name/dimensions/cache_size/memory) · `file_graph`(🕸️ n_nodes/n_edges/n_virtual\|n_pending/memory) · `file_store`(📦 n_chunks/n_chunks_with_embedding/memory) · `file_watcher`(👀 background_running/watch_paths) · `keyword_index`(🔤 n_docs/vocab_size/memory) ｜ 🛠️ deep sizeof（含 numpy.nbytes），未启动 / 后台未跑 / embedding 不健康 → ❌ |
|
||||
| 🌐 通用 step | 🏷️ `version` (`version_step`) | `call_server("version")` | 📥 无 ｜ 📤 `answer = reme4.__version__` ｜ 📊 `metadata.version` |
|
||||
| 🌐 通用 step | 🔄 `reindex` (`reindex_step`) | `call_server("reindex")` | 📥 无 ｜ 📤 `answer = "🔄 Reindexed {added} file(s)"` ｜ 📊 `metadata.counts = {added, ...}` ｜ 🛠️ 流程：`file_watcher.close()` → `file_store.clear()` → `file_watcher.update_store()` → `file_watcher.start()`（finally 保证重启） |
|
||||
| 🔎 search | 🔍 `search` (`search_step`) | `call_server("search", query=…, …)` | 📥 `query:str` ⭐ ｜ 🎚️ `limit:int=5`(>0) ｜ 🎚️ `min_score:float=0.0` ｜ ⚖️ `vector_weight:float=0.7` ∈[0,1]（keyword 权 = 1-vw）｜ 🔀 `candidate_multiplier:float=3.0`（candidates = min(200, limit×mult)）｜ 🔗 `expand_links:bool=True` ｜ 🔢 `max_links_per_direction:int=10` ｜ 🎚️ `search_filter:dict={}` ｜ 📤 `answer` 每命中一行 `path:start-end [score=… vector=… keyword=…] text` + 缩进的 `→ outlinks (n)` / `← inlinks (n)` + `via predicate=… anchor=#…` ｜ 📊 `metadata.results` / `metadata.link_expansion` / `metadata.counts={vector,keyword,returned,hybrid}` ｜ 🛠️ 并行 `vector_search` + `keyword_search` → RRF 融合（K=60，按 chunk.id 合并）→ `min_score` 过滤 → `limit` 截断 → 邻居 meta 注入 |
|
||||
| 🧪 demo | 🪄 `demo_echo` (`demo_echo_step1` + `step2`) | `call_server("demo_echo", query=…, min_score=…)` | 📥 `query:str=""` ｜ 🎚️ `min_score:float=0.5` ｜ 🛠️ step1：`processed_query = query.strip().lower()`，`adjusted_min_score = min_score * 0.9`，写回 context ｜ 📤 step2：`answer = "echo: {processed_query} (min_score={adjusted_min_score})"` ｜ 📊 `metadata = {step, query, min_score, processed_query, adjusted_min_score}` |
|
||||
| 🌊 demo | 🌊 `stream_demo` (`stream_demo_step1` + `step2`) | `call_server("stream_demo", query=…, repeat=…, interval=…)` | 📥 `query:str=""` ｜ 🎚️ `repeat:int=10` ｜ 🎚️ `interval:float=0.1`（秒/字符）｜ 🛠️ step1：`stream_text = query * repeat` 写回 context ｜ 📤 step2：按字符 `add_stream_string(ch, ChunkEnum.CONTENT)` 流式输出，`asyncio.sleep(interval)` 节流 |
|
||||
| 📂 crud | 📖 `read` (`read_step`) | `call_server("read", path=…, …)` | 📥 `path:str` ⭐（**完整相对路径**，相对于 vault；绝对路径会被拒绝；非 `.md` 后缀拒绝）｜ 🎚️ `start_line:int=null`（1-based, 含端点）｜ 🎚️ `end_line:int=null`（1-based, 含端点）｜ 🎚️ `max_bytes:int=51200`（截断阈值）｜ 📤 `answer = 选中的行内容`，超过 `max_bytes` 时附加 `--- TRUNCATED ---` 续读指引（`start_line=…`）｜ 📊 `metadata.path` / `metadata.total_lines`（出错路径才会附带）｜ 🛠️ 流程：`BaseStep.resolve_path(raw, require_md=True)` → `aiofiles.os.stat` → `read_file_safe`（utf-8-sig BOM 容忍、UnicodeDecodeError fallback `errors=ignore`）→ `split("\n")` 切片 `[s-1:e]` → `truncate_text_output` 按字节截断保行 |
|
||||
|
||||
使用示例：
|
||||
|
||||
```bash
|
||||
# 启动（默认 default.yaml）
|
||||
reme4 start
|
||||
|
||||
# 指定 config 与服务端口
|
||||
reme4 start config=paw.yaml service.port=8181
|
||||
|
||||
# 查找在跑的 reme
|
||||
reme4 find_reme
|
||||
# HOST=127.0.0.1 PORT=8000 PID=12345
|
||||
|
||||
# 列出所有可用 action（client 端处理，不转服务端）
|
||||
reme4 list
|
||||
|
||||
# 转发到服务端的 step：所有 key=value 透传为 step kwargs
|
||||
reme4 help
|
||||
reme4 health_check
|
||||
reme4 version
|
||||
reme4 reindex
|
||||
reme4 search query="latency 问题" limit=10 min_score=0.2 vector_weight=0.6
|
||||
|
||||
# 读取 vault 下的 markdown（完整相对路径；无后缀自动补 .md；可按行切片或限制字节）
|
||||
reme4 read path=Templates/Recipe.md
|
||||
reme4 read path=Notes start_line=1 end_line=20
|
||||
reme4 read path=Big.md max_bytes=4096
|
||||
|
||||
# 通过 MCP backend 调用
|
||||
reme4 search query="..." backend=mcp
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 一个 .md 文件的完整结构 │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ --- │
|
||||
│ name: 宁德时代 ← YAML front matter │
|
||||
│ description: 全球动力电池龙头 │
|
||||
│ tags: [新能源, 电池] │
|
||||
│ --- │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 所属行业:: [[新能源]] ← 语义化链接（Dataview） │
|
||||
│ 竞争对手:: [[比亚迪]] │
|
||||
│ │
|
||||
│ # 基本面 ← Markdown 正文 │
|
||||
│ 全球动力电池出货量第一，核心技术为 │
|
||||
│ [[CTP]] 和 [[钠离子电池]]…… ← 标准 wikilink │
|
||||
│ │
|
||||
│ 参考 ![[2026Q1调研纪要]] ← 嵌入引用 │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ ↓ AST 语义分块 ↓ │
|
||||
│ chunk 1: [标题骨架] + 正文片段 │
|
||||
│ chunk 2: [标题骨架] + 正文片段 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@sen
|
||||
| file | upload/download/move/delete/stat/list | 文件操作CRUD |
|
||||
| property | read/update/delete | frontmatter CRUD | |
|
||||
| graph | traverse/retarget | path="My Note" directtion=forward/backward depth=1 predicat=xxx |
|
||||
### 2.1 YAML front matter
|
||||
|
||||
@wangce
|
||||
| crud | write | path="New Note" name="xxx" description="xxx" metadata={}, content="# Hello" (4 字段都必填，frontmatter 只写 name/description) |
|
||||
| crud | read | path="Templates/Recipe.md" |
|
||||
| crud | edit | path="Templates/Recipe.md" old="xxx" new="xxx" |
|
||||
| crud | append | path="My Note" content="New line" |
|
||||
每个笔记头部的元数据：
|
||||
|
||||
| crud | delete | path="My Note
|
||||
| daily_crud | daily_xxx | 与 crud 参数保持一致 |
|
||||
```markdown
|
||||
---
|
||||
name: 光伏产业链研究
|
||||
description: 从硅料到组件的全链条梳理
|
||||
tags: [新能源, 光伏, 产业链]
|
||||
---
|
||||
```
|
||||
|
||||
- daily_resolve name=xxxx (符合一定规范 win下要求)
|
||||
- daily_list date=xxxx 返回path
|
||||
- daily_index
|
||||
`name` / `description` 是约定字段，其余键值对全部保留，不会丢弃任何自定义字段。
|
||||
|
||||
frontmatter read path
|
||||
frontmatter update path metadata={}
|
||||
frontmatter delete path keys=[]
|
||||
### 2.2 四种 wikilink 写法
|
||||
|
||||
delete path
|
||||
download path=xxx（内部相对路径）download_path=(外部绝对路径，可选)
|
||||
upload path=xxx（外部绝对路径）description="xxx" metadata=xxx 返回内部相对路径 加metadata
|
||||
stat path
|
||||
list path
|
||||
mv path=xxx new_path=xxx
|
||||
| 写法 | 示例 | 语义 |
|
||||
|------|----------------|----------|
|
||||
| 标准链接 | `[[光伏产业链]]` | 指向目标文件 |
|
||||
| 锚点链接 | `[[钴#应用]]` | 指向特定章节 |
|
||||
| 别名链接 | `[[宁德时代\|宁德]]` | 自定义显示文本 |
|
||||
| 嵌入引用 | `![[钴]]` | 内联嵌入目标内容 |
|
||||
|
||||
traverse path=xxx direction=xxx depth=xxx
|
||||
### 2.3 语义化链接（Dataview 风格）
|
||||
|
||||
# 日记类型
|
||||
普通 wikilink 只说"A 提到了 B"，语义化链接还能表达"A 和 B 是什么关系"：
|
||||
|
||||
| 类型 | 路径 | 说明 |
|
||||
|-----------|-----------------------------------------------|-----------------------------|
|
||||
| daily | {daily}/xxxx-mm-dd.md + xxxx-mm-dd/{event}.md | 按日期归档的原始信息记录 |
|
||||
| topic | topic/{topic:-personal(agent)}/{xxxx}.md | 按主题聚类的二次加工内容 |
|
||||
| proactive | todo | 基于 daily / topic 思考后主动推送的消息 |
|
||||
```markdown
|
||||
所属行业:: [[新能源]] ← 行级属性（独占一行）
|
||||
总部:: [[宁德]]
|
||||
[竞争对手:: [[比亚迪]]] ← 内联属性（嵌入正文中）
|
||||
```
|
||||
|
||||
# 生成Job
|
||||
`WikilinkHandler` 是全系统唯一的 wikilink 解析入口，确保 parser、graph、search 各层规则一致。
|
||||
|
||||
| 任务 | 输入 | 输出 | 触发时机 | 说明 |
|
||||
|-------------------------|---------------|-----------------------------------------------|-----------------------------|------------------------------------------------------|
|
||||
| 日记summary @sen @wangce | msg | {daily}/xxxx-mm-dd.md + xxxx-mm-dd/{event}.md | freq (every_n_turn、compact) | 把 msg 的信息写入 daily 目录 |
|
||||
| 主题dream + 生成链接 @sen | daily/xxx | knowledge/xxx | /dream | 把 daily 目录的内容按主题聚类合并到 topic 目录, 主动在文档中建立 [[link]] 关联 |
|
||||
| 主动proactive @wangce | daily / topic | proactive_query | pre_query | 思考 daily / topic 信息，主动决定推送给用户的消息 |
|
||||
### 2.4 AST 感知的语义分块
|
||||
|
||||
2. file_parser
|
||||
a. 抽象基类 parse: @jinli
|
||||
ⅰ. 输入是path：相对路径
|
||||
ⅱ. 输出是FileMetadata & list[FileChunks] & list[FileEdge]
|
||||
b. default parser 兼容老方案 @jinli
|
||||
ⅰ. 带overlap的chunking策略 ，不输出FileEdge
|
||||
c. markdown parser @sen
|
||||
ⅰ. 根据markdown ast做chunk，不需要overlap
|
||||
ⅱ. 增加一个索引的chunk chunk_type @锦鲤 file_chunk_type content/index
|
||||
ⅲ. 增加link的正则解析：predicate:: [[path#anchor]]
|
||||
3. file_store @sen
|
||||
a. 抽象存储：
|
||||
ⅰ. filenode = file + path + st_mtime + metadata + list[FileEdge]
|
||||
ⅱ. graph=dict[str, filenode] 内存+json
|
||||
ⅲ. list[FileChunk] 存db
|
||||
b. 抽象基类
|
||||
ⅰ. graph：fellow dict的操作 update/get/set
|
||||
ⅱ. chunks dict[str, list[chunk]]
|
||||
1. delete_chunks_by_path
|
||||
2. update_chunks_by_path
|
||||
3. list_chunks_by_path
|
||||
4. vector_search/keyword_search
|
||||
ⅲ. 手写一个bm25检索
|
||||
ⅳ. 【核心】检索机制 vector bm25 graph 如何进行融合
|
||||
4. file_watcher @jinli
|
||||
a. 抽象基类
|
||||
ⅰ. on_start:
|
||||
1. file_store 的start 在前，加载graph，file_watcher在后，递归扫描目录
|
||||
a. 通过ms_time对比graph，on_change 进行改动
|
||||
ⅱ. on_change:
|
||||
1. 更新/增加:
|
||||
a. delete_chunks_by_path 更新数据库
|
||||
b. upate_chunks_by_path 更新数据库
|
||||
c. 更新graph
|
||||
2. 删除
|
||||
a. delete_chunks_by_path 更新数据库
|
||||
传统 RAG 按固定 token 长度切片，经常切坏文档结构。ReMe 基于 Markdown AST 做语义分块：
|
||||
|
||||
MemorySchema
|
||||
- 按 H1/H2/H3 章节嵌套建树，递归分块
|
||||
- **每个 chunk 保留完整标题骨架**——检索到片段后一眼看出它在哪个章节下
|
||||
- 表格自动重复表头、代码块保留 fence、列表按项打包
|
||||
|
||||
1. markdown文件结构 @sen
|
||||
a. formatter：
|
||||
ⅰ. name
|
||||
ⅱ. desc
|
||||
2. memory文件结构目录
|
||||
a. MEMORY.md
|
||||
b. msg/files -> daily/YYYYMMDD/YYYYMMDD.md + xxxx.md
|
||||
ⅰ. YYYYMMDD.md
|
||||
1. xxx -> xxxx.md
|
||||
2. xxx -> xxxd.md
|
||||
ⅱ.
|
||||
c. daily -> topic/topic_l1/topic_l1.md + xxx.md + topic_l2
|
||||
d. proactive
|
||||
```
|
||||
示例 chunk：
|
||||
─────────────────────
|
||||
# 光伏产业链
|
||||
## 上游：硅料
|
||||
### 多晶硅工艺
|
||||
[chunk 正文] ← 实际内容
|
||||
## 中游：硅片 ← 骨架（只有标题）
|
||||
## 下游：组件
|
||||
─────────────────────
|
||||
```
|
||||
|
||||
steps:
|
||||
## 特性3：自进化
|
||||
|
||||
1. 治理（算法+LLM）：
|
||||
a. 节点关联P0：现有的链接做补充，挖掘新的LLM的link
|
||||
ⅰ. /Users/yuli/workspace/ReMe/reme2/component/edge_extractor/llm_edge_extractor.py
|
||||
ⅱ. 移动到steps
|
||||
b. 节点整合/节点拆分/节点归档
|
||||
c. 健康度检查
|
||||
2. retrieve 调用store的检索
|
||||
3. 原子steps：reme edit
|
||||
4. 组合steps：总结：
|
||||
a. - freq (every_n_turn、compact) -> daily_summarizer
|
||||
b. topic (/dream ) -> topic_summarizer(daily_xx -> topic_xx)
|
||||
c. proactive -> proactive_summarizer(personal_xxx -> proactive_query - pre_query
|
||||
> **ReMe 的记忆不是被动存的，是主动长成知识图谱的。**
|
||||
|
||||
```
|
||||
用户对话 / 外部素材
|
||||
│
|
||||
├───────────────────────────────────┐
|
||||
▼ ▼
|
||||
┌────────────┐ ┌────────────┐
|
||||
│ auto-memory│ │auto-resource│
|
||||
│ 对话→日记 │ │ 素材→解析 │
|
||||
└─────┬──────┘ └──────┬─────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ daily/ │
|
||||
│ (事件日记 + resource 加工笔记) │
|
||||
└─────────────────────┬───────────────────────────┘
|
||||
│
|
||||
▼ 定时触发
|
||||
┌─────────────┐
|
||||
│ auto-dream │
|
||||
│ 提炼 + 建图谱 │
|
||||
└──────┬──────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ digest/ │
|
||||
│ (知识卡片 + wikilink 互联 = 知识图谱) │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
用户什么都不用做，Agent 在后台让笔记自己长出结构。
|
||||
|
||||
### 3.1 auto-resource
|
||||
|
||||
监控 `resource/` 目录，新文件进来后自动解析内容、整理为结构化笔记写入 `daily/` 下。
|
||||
|
||||
### 3.2 auto-memory
|
||||
|
||||
对话进行时，ReMe 在后台把上下文自动写入当天日记。不是简单的对话摘要——而是一个拥有完整读写能力的 LLM
|
||||
Agent，自己决定记什么、怎么组织、合并还是新增。
|
||||
|
||||
### 3.3 auto-dream + auto-link：睡眠式记忆整理
|
||||
|
||||
借鉴人在睡眠中巩固记忆的机制——把日记和素材提炼成知识卡片，并自动织出图谱关系：
|
||||
|
||||
```
|
||||
┌───────────────────────────┐
|
||||
│ daily/2026-05-28/xxx.md │ ← 一篇日记或素材
|
||||
└─────────────┬─────────────┘
|
||||
│
|
||||
╔═════════════════════════════════════╗
|
||||
║ Phase 1 — Extract（一个 Agent） ║
|
||||
║ "这份材料教了什么道理？" ║
|
||||
║ ║
|
||||
║ 输出 N 个抽象单元，各带 bucket 标签 ║
|
||||
║ (空 → 结束，没东西值得记) ║
|
||||
╚══════════╤══════════╤═══════════════╝
|
||||
│ │
|
||||
┌─────────────┘ └──────────────┐
|
||||
▼ ▼
|
||||
╔══════════════════════════════╗ ╔══════════════════════════════╗
|
||||
║ Phase 2 — Integrate ║ ║ Phase 2 — Integrate ║
|
||||
║ (每个 unit 独立一个 Agent) ║ ║ (每个 unit 独立一个 Agent) ║
|
||||
║ ║ ║ ║
|
||||
║ 1. search + traverse 召回 ║ ║ 1. search + traverse 召回 ║
|
||||
║ 2. 决策: CREATE / UPDATE ║ ║ 2. 决策: CREATE / UPDATE ║
|
||||
║ 3. 写入 + 自动织链接 ║ ║ 3. 写入 + 自动织链接 ║
|
||||
╚══════════════╤═══════════════╝ ╚══════════════╤═══════════════╝
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ digest/ │
|
||||
│ procedure/key-rotation.md ←─ derived_from:: [[daily/..]] │
|
||||
│ wiki/credential-compliance.md ─ relates_to:: [[...]] │
|
||||
│ personal/user-pr-pref.md │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
知识图谱自动生长
|
||||
```
|
||||
|
||||
**Phase 1 筛选**——多个事实说明同一个道理就合并为一个 unit，分到三个桶：`procedure`（怎么做）/ `personal`（用户偏好）/ `wiki`
|
||||
（通用知识）。没东西值得记则流程结束。
|
||||
|
||||
**Phase 2 先搜后写**——先搜已有 digest，再决策：新建（CREATE）、追加佐证（CORROBORATE）、补充精度（REFINE）、修正矛盾（CORRECT）。
|
||||
|
||||
**auto-link 是写入的副产品**——写 digest 时自动加 `derived_from:: [[素材]]` 溯源 + `relates_to::` 概念互联，图谱随每次
|
||||
dream 自动变密。
|
||||
|
||||
**CronDreamer 定时批跑**——每天扫描当天所有 daily + resource 文件，逐个执行上述管线。
|
||||
|
||||
## 特性4：混合索引 + 渐进式展开
|
||||
|
||||
```
|
||||
用户提问: "宁德时代的电池技术？"
|
||||
│
|
||||
├──────────────────────┬──────────────────────────┐
|
||||
▼ ▼ │
|
||||
┌─────────────────┐ ┌──────────────────┐ │
|
||||
│ 全文倒排索引 │ │ 向量索引 │ │
|
||||
│ (numpy + jieba) │ │ (faiss) │ │
|
||||
│ │ │ │ │
|
||||
│ "宁德时代" 精确 │ │ "动力电池龙头" │ │
|
||||
│ 命中 │ │ 语义近似命中 │ │
|
||||
└────────┬────────┘ └────────┬─────────┘ │
|
||||
│ text_weight=0.3 │ vector_weight=0.7 │
|
||||
└──────────┬──────────┘ │
|
||||
▼ │
|
||||
┌───────────────┐ │
|
||||
│ RRF 融合排序 │ │
|
||||
│ score = Σ(w/(k+rank)) │
|
||||
└───────┬───────┘ │
|
||||
▼ │
|
||||
┌──────────────────────────────────────┐ │
|
||||
│ 第一跳：Top-K chunk 全文 + 评分 │ │
|
||||
└───────────────────┬──────────────────┘ │
|
||||
▼ │
|
||||
┌──────────────────────────────────────┐ │
|
||||
│ 第二跳：邻居目录（只有标题，不展开正文）│ ← wikilink 图谱 │
|
||||
└───────────────────┬──────────────────┘ │
|
||||
▼ │
|
||||
┌──────────────────────────────────────┐ │
|
||||
│ 第 N 跳：Agent 按需追问，展开正文 │ │
|
||||
└──────────────────────────────────────┘ │
|
||||
```
|
||||
|
||||
### 4.1 混合索引构建
|
||||
|
||||
两套索引并行维护，各擅其长：
|
||||
|
||||
- **全文倒排索引**（numpy + jieba BM25）——精确匹配专有名词，搜"宁德时代"必须命中。纯 Python 实现，增量更新，无原生扩展依赖。
|
||||
- **向量索引**（faiss）——语义相似度，搜"锂电正极原料"能命中"钴"。基于 embedding 模型生成稠密向量。
|
||||
|
||||
### 4.2 基于 RRF 的混合检索
|
||||
|
||||
两条通路并行跑（`asyncio.gather`），用 RRF（Reciprocal Rank Fusion）融合排序：
|
||||
|
||||
```
|
||||
融合分 = Σ( weight_i / (k + rank_i) ) k=60, vector_weight=0.7, text_weight=0.3
|
||||
```
|
||||
|
||||
为什么要两路？纯向量容易错配名词（"苹果公司"≈"水果"），纯关键词抓不到同义改写——融合互补盲区。
|
||||
|
||||
### 4.3 渐进式链接展开
|
||||
|
||||
传统 RAG 一次性把 Top-K 全塞进上下文，token 浪费且噪音多。ReMe 分跳展开，按需深入：
|
||||
|
||||
**第一跳** — 返回命中 chunk 全文 + 分数明细
|
||||
|
||||
**第二跳** — 展开 wikilink 邻居的"目录"（只有标题，不展开正文）：
|
||||
|
||||
```
|
||||
========== digest/wiki/宁德时代.md:5-22 [score=0.0247 vector=0.0156 keyword=0.0091] ==========
|
||||
# 宁德时代
|
||||
全球动力电池出货量第一，核心技术为 CTP（Cell to Pack）和钠离子电池……
|
||||
|
||||
outlinks (2):
|
||||
→ digest/wiki/磷酸铁锂.md name="磷酸铁锂正极路线" description="磷酸铁锂与三元路线对比" via predicate=相关技术
|
||||
→ digest/wiki/固态电池.md name="固态电池技术路线" description="全固态与半固态进展" via predicate=技术演进
|
||||
inlinks (2):
|
||||
← daily/2026-03-18/宁德调研.md name="宁德时代调研纪要" description="2026Q1产能与订单跟踪" via plain
|
||||
← digest/wiki/新能源产业链.md name="新能源产业链全景" description="从锂矿到整车的全链条" via predicate=下游应用
|
||||
```
|
||||
|
||||
**第 N 跳** — Agent 看过"目录"后，自己决定哪些邻居值得深入，再发起 read 拿正文。
|
||||
|
||||
二跳目录每条只占一行（最多 10 outlink + 10 inlink），Agent 拥有全局视野却不撑爆上下文。
|
||||
|
||||
## 特性5：多 Agent 框架集成
|
||||
|
||||
ReMe 不做独立 Agent 产品，而是作为**能力层**被任意框架调用：
|
||||
|
||||
| 集成路径 | 适用对象 | 方式 |
|
||||
|---------------------|----------------------|---------------------------------------------|
|
||||
| SDK 深度集成 | AgentScope / Qwenpaw | middleware 注册 tools + prompt，hook 注册 auto-* |
|
||||
| MCP Tool + skill.md | Claude Code | MCP 注册 Tool，配 skill.md 开箱即用，hook 注册 auto-* |
|
||||
| HTTP API + CLI | 通用方案 | skill.md + CLI 调用 |
|
||||
|
||||
---
|
||||
|
||||
# 二、工程架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Service 层（HTTP / MCP 双协议） │
|
||||
│ FastAPI + FastMCP，同一套 Job 同时暴露为 REST 和 MCP Tool │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Application 层 │
|
||||
│ 配置加载 → 组件初始化 → Job 注册 → start() / close() 生命周期 │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Job 层（编排） │
|
||||
│ 每个 Job = 一组 Step 的有序管线，YAML 声明式配置 │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Step 层（业务逻辑） │
|
||||
│ 原子操作单元，按功能域分组：file_io / index / evolve / common │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Component 层（可插拔基础设施） │
|
||||
│ 统一注册表 R，一行配置切换实现 │
|
||||
│ file_store / embedding / keyword_index / llm / file_graph │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2.1 服务层
|
||||
|
||||
每个 Job 同时暴露为两种协议，写一次逻辑、两种方式调用：
|
||||
|
||||
| 协议 | 传输方式 | 适用场景 |
|
||||
|---------------|-------------------------------|------------------------------|
|
||||
| HTTP（FastAPI） | JSON POST / SSE | REST 调用、Web 前端 |
|
||||
| MCP（FastMCP） | stdio / SSE / streamable-http | Claude Code、Cursor 等 MCP 客户端 |
|
||||
|
||||
- **按需拉起**：Agent 检测到服务未运行时自动后台启动，用户无感知
|
||||
- **服务发现**：通过 `REME_SERVICE_INFO` 环境变量广播地址，`find_reme` 一键探活
|
||||
|
||||
## 2.2 组件系统（Component）
|
||||
|
||||
统一注册表 `R`，所有基础设施都是可插拔的——改一行配置就能切换后端：
|
||||
|
||||
| 组件 | 干什么 | 可选后端 |
|
||||
|-----------------|---------------|-----------------------|
|
||||
| file_store | 文件存储 + 索引协调 | local |
|
||||
| file_graph | wikilink 双向图谱 | local / nx / neo4j |
|
||||
| keyword_index | 全文倒排索引 | bm25（numpy + jieba） |
|
||||
| embedding_store | 向量存储与检索 | local（faiss） |
|
||||
| embedding | 文本转向量 | openai 兼容接口 |
|
||||
| llm | 大模型调用 | anthropic / openai 兼容 |
|
||||
| tokenizer | 分词 | regex / jieba |
|
||||
|
||||
## 2.3 Job 列表
|
||||
|
||||
**Job** 是 ReMe 暴露给外部的操作单元——同一个 Job 可以作为 Python 函数直接调用、作为 MCP Tool 被 Agent 使用、也可以作为 CLI
|
||||
命令执行。
|
||||
|
||||
| 类别 | Job | 功能 |
|
||||
|------|---------------------------|----------------------------------|
|
||||
| 检索 | `search` | 混合检索（向量 + BM25 + RRF）+ 渐进式图展开 |
|
||||
| 检索 | `traverse` | 从指定路径遍历 wikilink 图谱 |
|
||||
| 文件读写 | `read` | 读取 markdown 文件内容 |
|
||||
| 文件读写 | `read_image` | 读取图片文件（base64） |
|
||||
| 文件读写 | `write` | 新建或覆写 markdown 文件（含 frontmatter） |
|
||||
| 文件读写 | `edit` | 文件内查找替换 |
|
||||
| 文件读写 | `delete` | 删除文件，返回残留入边 |
|
||||
| 文件读写 | `move` | 移动 / 重命名，自动重写 wikilink |
|
||||
| 文件读写 | `list` | 列出目录下文件 |
|
||||
| 文件读写 | `stat` | 文件元信息（大小、修改时间） |
|
||||
| 文件读写 | `frontmatter_read` | 读取 frontmatter |
|
||||
| 文件读写 | `frontmatter_update` | 合并更新 frontmatter |
|
||||
| 文件读写 | `frontmatter_delete` | 删除 frontmatter 字段 |
|
||||
| 日记管理 | `daily_create` | 幂等创建当天日记文件 |
|
||||
| 日记管理 | `daily_list` | 列出某天的所有日记 |
|
||||
| 日记管理 | `daily_reindex` | 重建当天索引页 |
|
||||
| 索引维护 | `reindex` | 清空并全量重建索引 |
|
||||
| 索引维护 | `update_store_index_loop` | 后台监听文件变更，增量更新 |
|
||||
| 自进化 | `auto_memory` | 对话记录写入日记（LLM Agent） |
|
||||
| 自进化 | `dream` | 单文件记忆提炼到 digest（LLM Agent） |
|
||||
| 自进化 | `auto-dream` | 批量扫描当天文件，逐个 dream |
|
||||
| 系统 | `health_check` | 组件健康检查 |
|
||||
| 系统 | `version` | 返回版本号 |
|
||||
| 系统 | `help` | 列出所有已注册 Job |
|
||||
|
|
|
|||
231
docs4/reme_scene.md
Normal file
231
docs4/reme_scene.md
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
# ReMe 应用场景
|
||||
|
||||
## 金融场景：产业链知识库
|
||||
|
||||
**主角**：王分析师，新能源行业研究员，每天处理 10+ 篇研报、数十条产业新闻、若干场公司调研。
|
||||
|
||||
**痛点**：信息散落在飞书文档、PDF 研报、微信群消息、调研纪要里，"上次调研宁德时代时聊到的钴价话题"再也找不回来。
|
||||
|
||||
### 一周内 ReMe 自动织出的产业链图谱
|
||||
|
||||
---
|
||||
|
||||
#### Day 1（周一）盘后：素材摄入 + 对话
|
||||
|
||||
王分析师把今天看到的 3 篇研报扔进 `resource/`，又和 Agent 口述了对刚果(金)矿权变更的看法：
|
||||
|
||||
```
|
||||
对话片段：
|
||||
> 今天嘉能可发了三季报，钴产量同比下滑 18%……
|
||||
> 刚果(金)那边的政策变化，对洛阳钼业 KFM 矿的影响要重点跟……
|
||||
> 下游三元正极厂商已经开始转向高镍低钴方案……
|
||||
```
|
||||
|
||||
**auto-memory** 实时把对话写入当天日记；**auto-resource** 自动解析研报写入加工笔记：
|
||||
|
||||
```
|
||||
daily/
|
||||
├── 2026-05-18.md ← 当天索引页，汇总所有事件
|
||||
└── 2026-05-18/
|
||||
├── session_001.md ← auto-memory 写入的对话日志
|
||||
│ （含嘉能可三季报、刚果金矿权、高镍化趋势等事件）
|
||||
├── resource_001.md ← auto-resource 对研报 1 的加工笔记
|
||||
├── resource_002.md ← 研报 2 加工笔记
|
||||
└── resource_003.md ← 研报 3 加工笔记
|
||||
```
|
||||
|
||||
**Day 1 夜间 auto-dream**——CronDreamer 扫描当天 4 个文件，逐个执行 Extract → Integrate 管线：
|
||||
|
||||
处理 `session_001.md`：
|
||||
- **Phase 1 Extract**：从对话日志中提取 3 个抽象单元——「嘉能可钴产量下滑」(wiki)、「刚果金矿权政策风险」(wiki)、「三元正极高镍化趋势」(wiki)
|
||||
- **Phase 2 Integrate**（每个 unit 独立一个 Agent）：
|
||||
- 搜索已有 digest，均无匹配 → 决策 **CREATE**
|
||||
- 新建 `digest/wiki/嘉能可.md`、`digest/wiki/钴.md`、`digest/wiki/三元正极.md`
|
||||
- 写入时自动织链接：`derived_from:: [[daily/2026-05-18/session_001]]`，以及概念互联 `relates_to:: [[三元正极]]`
|
||||
|
||||
处理 `resource_001.md`（嘉能可三季报）：
|
||||
- **Phase 1**：提取「嘉能可钴业务财务数据」(wiki)
|
||||
- **Phase 2**：搜索到刚刚新建的 `digest/wiki/嘉能可.md` → 决策 **CORROBORATE**，追加财务佐证段落
|
||||
|
||||
Day 1 结束时 `digest/wiki/` 下新增：
|
||||
|
||||
```
|
||||
digest/wiki/
|
||||
├── 嘉能可.md ← CREATE + CORROBORATE（研报佐证）
|
||||
├── 钴.md ← CREATE
|
||||
└── 三元正极.md ← CREATE
|
||||
```
|
||||
|
||||
`钴.md` 长这样：
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: 钴
|
||||
description: 锂电正极材料关键原料，主产区刚果(金)
|
||||
tags: [新能源, 原料, 钴]
|
||||
---
|
||||
|
||||
所属领域:: [[新能源]]
|
||||
下游产品:: [[三元正极]]
|
||||
|
||||
# 钴
|
||||
|
||||
## 供给端
|
||||
主要生产商 [[嘉能可]]，产能集中于刚果(金)。
|
||||
嘉能可三季度钴产量同比下滑 18%。
|
||||
derived_from:: [[daily/2026-05-18/session_001]]
|
||||
|
||||
## 政策风险
|
||||
刚果(金)矿权政策变化，可能影响 KFM 矿运营。
|
||||
derived_from:: [[daily/2026-05-18/session_001]]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Day 2（周二）：宁德时代调研
|
||||
|
||||
王分析师参加宁德时代调研会后，和 Agent 聊调研要点：
|
||||
|
||||
```
|
||||
> 宁德今年全面切换 9 系高镍三元，钴用量还会继续降……
|
||||
> 产能利用率 85%，比上季度高 5 个点……
|
||||
```
|
||||
|
||||
auto-memory 写入 `daily/2026-05-19/session_001.md`。
|
||||
|
||||
**Day 2 夜间 auto-dream** 处理这份 session：
|
||||
- **Phase 1**：提取「宁德时代高镍切换」(wiki)、「宁德时代产能利用率」(wiki)
|
||||
- **Phase 2**：
|
||||
- 「宁德时代高镍切换」→ 搜索到 `digest/wiki/三元正极.md` 已存在高镍化内容 → 决策 **REFINE**，补充"宁德 9 系切换"作为具体案例，并新建 `digest/wiki/宁德时代.md`
|
||||
- 「产能利用率」→ 无匹配 → 写入 `digest/wiki/宁德时代.md`（已存在，追加章节）
|
||||
|
||||
Day 2 结束时图谱新增节点和边：
|
||||
|
||||
```
|
||||
digest/wiki/
|
||||
├── 嘉能可.md
|
||||
├── 钴.md
|
||||
├── 三元正极.md ← REFINE：新增宁德 9 系切换案例
|
||||
└── 宁德时代.md ← CREATE：含高镍切换 + 产能数据
|
||||
relates_to:: [[三元正极]]
|
||||
relates_to:: [[钴]]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Day 3（周三）：亿纬电话会 + 洛阳钼业跟踪
|
||||
|
||||
两场对话产生两份 session。夜间 auto-dream 逐个处理：
|
||||
|
||||
- `session_001.md`（亿纬电话会）→ Phase 2 搜到 `宁德时代.md`、`三元正极.md` → CREATE `digest/wiki/亿纬锂能.md`，并在 `三元正极.md` 上 CORROBORATE 高镍趋势
|
||||
- `session_002.md`（洛阳钼业跟踪）→ Phase 2 搜到 `钴.md` → REFINE `钴.md`，补充洛阳钼业 KFM 矿最新动态；CREATE `digest/wiki/洛阳钼业.md`
|
||||
|
||||
Day 3 结束时图谱：
|
||||
|
||||
```
|
||||
digest/wiki/
|
||||
├── 嘉能可.md
|
||||
├── 洛阳钼业.md ← CREATE
|
||||
├── 钴.md ← REFINE：补充洛阳钼业信息
|
||||
│ relates_to:: [[嘉能可]], [[洛阳钼业]], [[三元正极]]
|
||||
├── 三元正极.md ← CORROBORATE：亿纬佐证
|
||||
│ relates_to:: [[钴]], [[宁德时代]], [[亿纬锂能]]
|
||||
├── 宁德时代.md
|
||||
└── 亿纬锂能.md ← CREATE
|
||||
```
|
||||
|
||||
每个节点都是**当天 dream 从一份 daily 文件中提取并整合的结果**，不存在跨天"聚合"——跨文件的关联通过 Phase 2 的 search 自然发现已有 digest，从而把新信息写入正确的位置。
|
||||
|
||||
---
|
||||
|
||||
#### Day 5（周五）：用户主动检索
|
||||
|
||||
王分析师准备组会要讲新能源板块，主动问 Agent：
|
||||
|
||||
> **"帮我分析一下锂电相关上下游"**
|
||||
|
||||
Agent 调用 ReMe 的 `search` Job，走向量 + BM25 + RRF 融合检索，命中已有的 digest 节点，并通过渐进式展开获取上下游全貌：
|
||||
|
||||
**第一跳——直接命中 chunk 全文 + 评分**：
|
||||
|
||||
```
|
||||
digest/wiki/钴.md:5-22 [score=0.0234 vector=0.0156 keyword=0.0078]
|
||||
# 钴 / ## 供给端
|
||||
主要生产商嘉能可、洛阳钼业，产能集中于刚果(金)……
|
||||
|
||||
digest/wiki/三元正极.md:10-30 [score=0.0211 vector=0.0148 keyword=0.0063]
|
||||
# 三元正极 / ## 高镍低钴路线
|
||||
2026 年起主流厂商加速 9 系产品，宁德已全面切换……
|
||||
|
||||
digest/wiki/宁德时代.md:1-20 [score=0.0193 vector=0.0135 keyword=0.0058]
|
||||
digest/wiki/嘉能可.md:5-30 [score=0.0167 vector=0.0117 keyword=0.0050]
|
||||
digest/wiki/洛阳钼业.md:1-22 [score=0.0152 vector=0.0106 keyword=0.0046]
|
||||
```
|
||||
|
||||
**第二跳——展开 wikilink 邻居目录（只有标题，不展开正文）**：
|
||||
|
||||
```
|
||||
digest/wiki/钴.md 的邻居：
|
||||
outlinks (3):
|
||||
→ digest/wiki/三元正极.md name="三元正极" description="高镍低钴技术路线" via predicate=下游产品
|
||||
→ digest/wiki/嘉能可.md name="嘉能可" description="全球钴业巨头" via predicate=relates_to
|
||||
→ digest/wiki/洛阳钼业.md name="洛阳钼业" description="KFM 矿运营商" via predicate=relates_to
|
||||
inlinks (2):
|
||||
← daily/2026-05-18/session_001.md name="盘后对话" via plain
|
||||
← daily/2026-05-19/session_001.md name="宁德调研" via plain
|
||||
|
||||
digest/wiki/三元正极.md 的邻居：
|
||||
outlinks (2):
|
||||
→ digest/wiki/宁德时代.md name="宁德时代" description="全球动力电池龙头" via predicate=relates_to
|
||||
→ digest/wiki/钴.md name="钴" description="锂电正极关键原料" via predicate=relates_to
|
||||
inlinks (1):
|
||||
← digest/wiki/亿纬锂能.md name="亿纬锂能" description="动力电池厂商" via predicate=relates_to
|
||||
```
|
||||
|
||||
**第 N 跳——Agent 按需深入**：Agent 看过目录后，决定展开 `亿纬锂能.md` 的正文获取补充信息，调用 `read` Job 拉取。
|
||||
|
||||
Agent 基于检索结果**直接回复**王分析师：
|
||||
|
||||
> "锂电产业链分三段：上游钴矿（嘉能可、洛阳钼业，刚果金集中）、中游三元正极（高镍化加速）、下游电池厂（宁德/亿纬）。这周你提到的事件分别落在：嘉能可三季报 → 上游产能收缩；高镍化趋势 → 中游路线切换；宁德 9 系切换 → 下游需求验证。"
|
||||
|
||||
---
|
||||
|
||||
#### Day 7：图谱已经长出层次
|
||||
|
||||
经过一周每天的 auto-dream 逐文件处理，图谱自然生长出来：
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
┌────────►│ 锂电 │◄────────┐
|
||||
│ └──────┬──────┘ │
|
||||
│ upstream │ │ upstream
|
||||
│ │ 下游产品 │
|
||||
┌───────┴──────┐ ▼ ┌──────┴──────┐
|
||||
│ 钴 │ ┌─────────┐ │ 锂 │
|
||||
│ (刚果金产区) │◄──┤ 原料 ├────►│ (盐湖产区) │
|
||||
└───────┬──────┘ └────┬────┘ └─────────────┘
|
||||
│ relates_to │ relates_to
|
||||
▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐
|
||||
│ 嘉能可 │ │ 三元正极 │◄── 高镍化趋势
|
||||
│ 洛阳钼业 │ └──────┬───────┘
|
||||
└──────────────┘ │ relates_to
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ 宁德时代 │ ← Day 2 调研
|
||||
│ 亿纬锂能 │ ← Day 3 电话会
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
**没有一个节点是凭空编造的**——每条边对应笔记里的一句 `relates_to:: [[X]]` 或 `derived_from:: [[daily/...]]`，每个节点点开就是 Markdown，每段内容都能追溯到原始 daily 事件。图谱不是一次性生成的，而是每天 dream 一点、链接一点，渐进生长出来的。
|
||||
|
||||
---
|
||||
|
||||
### ReMe 在这个场景下的核心价值
|
||||
|
||||
分析师只负责"看 + 说"，知识图谱自己长出来：
|
||||
|
||||
- **auto-memory** 把每次对话实时写入当天日记
|
||||
- **auto-dream** 每天逐文件执行 Extract → Integrate 管线，先搜已有 digest 再决策（CREATE / CORROBORATE / REFINE / CORRECT），知识卡片渐进生长
|
||||
- **auto-link** 是 dream 写入的副产品——`derived_from::` 溯源 + `relates_to::` 概念互联，图谱随每次 dream 自动变密
|
||||
- **混合检索 + 渐进展开** 让 Agent 先看骨架再决定深入哪个节点，不把 Top-K 全文塞进上下文
|
||||
|
|
@ -48,7 +48,7 @@ class Application(BaseComponent):
|
|||
cfg = self.config
|
||||
vault_path = Path(cfg.vault_dir).absolute()
|
||||
vault_path.mkdir(parents=True, exist_ok=True)
|
||||
for subdir in [cfg.metadata_dir, cfg.daily_dir, cfg.digest_dir, cfg.resource_dir]:
|
||||
for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.dialog_dir, cfg.daily_dir, cfg.digest_dir]:
|
||||
if subdir:
|
||||
(vault_path / subdir).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
|
|
|||
|
|
@ -475,7 +475,7 @@ components:
|
|||
supported_extensions: [ "md" ]
|
||||
chunked:
|
||||
backend: chunked
|
||||
supported_extensions: [ "txt", "html", "json", "yaml", "py" ]
|
||||
supported_extensions: [ "txt", "html", "json", "jsonl", "yaml", "py" ]
|
||||
default:
|
||||
backend: default
|
||||
|
||||
|
|
|
|||
|
|
@ -28,28 +28,20 @@ class ApplicationConfig(BaseModel):
|
|||
"""Root config for the ReMe application."""
|
||||
|
||||
app_name: str = Field(default=os.getenv("APP_NAME", "ReMe"), description="Application display name")
|
||||
vault_dir: str = Field(default=".reme", description="Vault root directory (knowledge base) for runtime files")
|
||||
vault_dir: str = Field(default=".reme", description="Vault root directory for runtime files")
|
||||
metadata_dir: str = Field(default="reme_metadata", description="Subdirectory for ReMe persistent state")
|
||||
resource_dir: str = Field(default="resource", description="Subdirectory for external assets")
|
||||
dialog_dir: str = Field(default="dialog", description="Subdirectory for dialog memory")
|
||||
daily_dir: str = Field(default="daily", description="Subdirectory for daily memory")
|
||||
digest_dir: str = Field(default="digest", description="Subdirectory for digest")
|
||||
resource_dir: str = Field(
|
||||
default="resource",
|
||||
description="Subdirectory for passively-received external assets (upload bucket)",
|
||||
)
|
||||
digest_dir: str = Field(default="digest", description="Subdirectory for digest memory")
|
||||
enable_logo: bool = Field(default=True, description="Show ASCII logo on startup")
|
||||
language: str = Field(default="", description="Default language for LLM interactions")
|
||||
log_to_console: bool = Field(default=True, description="Log to console")
|
||||
log_to_file: bool = Field(default=True, description="Log to file")
|
||||
mcp_servers: dict[str, dict] = Field(default_factory=dict, description="MCP server configs by name")
|
||||
service: ComponentConfig = Field(default_factory=ComponentConfig, description="Service endpoint config")
|
||||
jobs: dict[str, JobConfig] = Field(
|
||||
default_factory=dict,
|
||||
description="Job definitions keyed by job name",
|
||||
)
|
||||
thread_pool_max_workers: int = Field(
|
||||
default=0,
|
||||
description="Max worker threads in the shared thread pool; 0 to disable",
|
||||
)
|
||||
jobs: dict[str, JobConfig] = Field(default_factory=dict, description="Job definitions keyed by job name")
|
||||
thread_pool_max_workers: int = Field(default=0, description="Max worker threads; 0 to disable")
|
||||
components: dict[ComponentEnum, dict[str, ComponentConfig]] = Field(
|
||||
default_factory=dict,
|
||||
description="Component registry keyed by type then name",
|
||||
|
|
|
|||
|
|
@ -43,51 +43,6 @@ async def _make_app() -> Application:
|
|||
return app
|
||||
|
||||
|
||||
def test_llm_demo_step_basic_chat():
|
||||
"""LLMDemoStep drives Agent through self.llm."""
|
||||
|
||||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
|
||||
app = await _make_app()
|
||||
try:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="What is 1 + 1? Reply with just the number.",
|
||||
)
|
||||
text = (response.answer or "").strip()
|
||||
print(f"\n[basic_chat] response: {text!r}")
|
||||
assert text, "Empty assistant response"
|
||||
assert "2" in text, f"Expected '2' in response, got: {text!r}"
|
||||
print("✓ test_llm_demo_step_basic_chat passed")
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_llm_demo_step_with_tool():
|
||||
"""LLMDemoStep registers the add tool and the agent invokes it."""
|
||||
|
||||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
|
||||
app = await _make_app()
|
||||
try:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="Use the add tool to compute 21 + 21 and report the result.",
|
||||
sys_prompt="Use the `add` tool whenever the user asks to add numbers.",
|
||||
use_add_tool=True,
|
||||
)
|
||||
text = (response.answer or "").strip()
|
||||
print(f"\n[with_tool] response: {text!r}")
|
||||
assert "42" in text, f"Expected '42' in response, got: {text!r}"
|
||||
print("✓ test_llm_demo_step_with_tool passed")
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
class MathResult(BaseModel):
|
||||
"""Structured output for a math computation."""
|
||||
|
||||
|
|
@ -110,66 +65,78 @@ class SentimentAnalysis(BaseModel):
|
|||
)
|
||||
|
||||
|
||||
def test_llm_demo_step_structured_output():
|
||||
"""LLMDemoStep generates structured output via generate_structured_output."""
|
||||
|
||||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
|
||||
app = await _make_app()
|
||||
try:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="What is 15 multiplied by 7? Show your work.",
|
||||
sys_prompt="You are a math tutor. Solve the problem step by step.",
|
||||
structured_model=MathResult,
|
||||
)
|
||||
structured = response.metadata.get("structured_output")
|
||||
print(f"\n[structured_output] result: {structured}")
|
||||
assert structured is not None, "structured_output should not be None"
|
||||
assert "result" in structured, "structured_output should have 'result' field"
|
||||
assert structured["result"] == 105, f"Expected result=105, got: {structured['result']}"
|
||||
assert "expression" in structured, "structured_output should have 'expression' field"
|
||||
assert "explanation" in structured, "structured_output should have 'explanation' field"
|
||||
print("✓ test_llm_demo_step_structured_output passed")
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
asyncio.run(run())
|
||||
async def _run_basic_chat(app: Application) -> None:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="What is 1 + 1? Reply with just the number.",
|
||||
)
|
||||
text = (response.answer or "").strip()
|
||||
print(f"\n[basic_chat] response: {text!r}")
|
||||
assert text, "Empty assistant response"
|
||||
assert "2" in text, f"Expected '2' in response, got: {text!r}"
|
||||
print("✓ test_llm_demo_step_basic_chat passed")
|
||||
|
||||
|
||||
def test_llm_demo_step_structured_output_enum():
|
||||
"""LLMDemoStep structured output with Literal/enum fields."""
|
||||
async def _run_with_tool(app: Application) -> None:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="Use the add tool to compute 21 + 21 and report the result.",
|
||||
sys_prompt="Use the `add` tool whenever the user asks to add numbers.",
|
||||
use_add_tool=True,
|
||||
)
|
||||
text = (response.answer or "").strip()
|
||||
print(f"\n[with_tool] response: {text!r}")
|
||||
assert "42" in text, f"Expected '42' in response, got: {text!r}"
|
||||
print("✓ test_llm_demo_step_with_tool passed")
|
||||
|
||||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
|
||||
app = await _make_app()
|
||||
try:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="Analyze the sentiment: 'I absolutely love this product! It exceeded all my expectations.'",
|
||||
sys_prompt="You are a sentiment analysis expert. Analyze the given text.",
|
||||
structured_model=SentimentAnalysis,
|
||||
)
|
||||
structured = response.metadata.get("structured_output")
|
||||
print(f"\n[structured_enum] result: {structured}")
|
||||
assert structured is not None, "structured_output should not be None"
|
||||
assert (
|
||||
structured["sentiment"] == "positive"
|
||||
), f"Expected sentiment='positive', got: {structured['sentiment']}"
|
||||
assert 0 <= structured["confidence"] <= 1, f"Confidence should be 0-1, got: {structured['confidence']}"
|
||||
assert isinstance(structured["key_phrases"], list), "key_phrases should be a list"
|
||||
assert len(structured["key_phrases"]) > 0, "key_phrases should not be empty"
|
||||
print("✓ test_llm_demo_step_structured_output_enum passed")
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
asyncio.run(run())
|
||||
async def _run_structured_output(app: Application) -> None:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="What is 15 multiplied by 7? Show your work.",
|
||||
sys_prompt="You are a math tutor. Solve the problem step by step.",
|
||||
structured_model=MathResult,
|
||||
)
|
||||
structured = response.metadata.get("structured_output")
|
||||
print(f"\n[structured_output] result: {structured}")
|
||||
assert structured is not None, "structured_output should not be None"
|
||||
assert "result" in structured, "structured_output should have 'result' field"
|
||||
assert structured["result"] == 105, f"Expected result=105, got: {structured['result']}"
|
||||
assert "expression" in structured, "structured_output should have 'expression' field"
|
||||
assert "explanation" in structured, "structured_output should have 'explanation' field"
|
||||
print("✓ test_llm_demo_step_structured_output passed")
|
||||
|
||||
|
||||
async def _run_structured_output_enum(app: Application) -> None:
|
||||
step = LLMDemoStep(app_context=app.context)
|
||||
response = await step(
|
||||
query="Analyze the sentiment: 'I absolutely love this product! It exceeded all my expectations.'",
|
||||
sys_prompt="You are a sentiment analysis expert. Analyze the given text.",
|
||||
structured_model=SentimentAnalysis,
|
||||
)
|
||||
structured = response.metadata.get("structured_output")
|
||||
print(f"\n[structured_enum] result: {structured}")
|
||||
assert structured is not None, "structured_output should not be None"
|
||||
assert structured["sentiment"] == "positive", f"Expected sentiment='positive', got: {structured['sentiment']}"
|
||||
assert 0 <= structured["confidence"] <= 1, f"Confidence should be 0-1, got: {structured['confidence']}"
|
||||
assert isinstance(structured["key_phrases"], list), "key_phrases should be a list"
|
||||
assert len(structured["key_phrases"]) > 0, "key_phrases should not be empty"
|
||||
print("✓ test_llm_demo_step_structured_output_enum passed")
|
||||
|
||||
|
||||
async def _run_all() -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
|
||||
app = await _make_app()
|
||||
try:
|
||||
await _run_basic_chat(app)
|
||||
await _run_with_tool(app)
|
||||
await _run_structured_output(app)
|
||||
await _run_structured_output_enum(app)
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("=== LLMDemoStep + Agent integration tests ===")
|
||||
test_llm_demo_step_basic_chat()
|
||||
test_llm_demo_step_with_tool()
|
||||
test_llm_demo_step_structured_output()
|
||||
test_llm_demo_step_structured_output_enum()
|
||||
asyncio.run(_run_all())
|
||||
print("\nAll integration tests passed!")
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue