mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-29 01:41:38 +00:00
docs(auto-cognition): add comprehensive design document for auto-cognition system
This commit is contained in:
huangsen
parent
3a355a657d
commit
87eca2a6de
26 changed files with 2569 additions and 1066 deletions
330
docs4/auto_cognition_design.md
Normal file
330
docs4/auto_cognition_design.md
Normal file
|
|
@ -0,0 +1,330 @@
|
|||
# auto-cognition 设计(顶层:心智循环)
|
||||
|
||||
> 本文档:reme4 中**长期记忆系统**的顶层认知模型 —— 把 agent 的记忆生命周期类比人类睡眠/觉醒回路,推导出**三阶段分工**与**15 维能力清单**。
|
||||
>
|
||||
> **三阶段实现各有专属文档**:
|
||||
> - Stage 1 写入(REM 重放抽象) → `auto_dream_design.md`
|
||||
> - Stage 2 巩固(NREM 深度整合) → `auto_consolidate_design.md`
|
||||
> - Stage 3 检索(觉醒态提取) → `auto_recall_design.md`
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_memory_design.md`:入流端(daily 写入),与 cognition 平行 —— cognition 负责"已落地后的认知循环",memory 负责"经历落地"
|
||||
> - `structure.md` §4(retrieve 三种问法)
|
||||
>
|
||||
> **核心立场**:
|
||||
> - 长期记忆不是"存 + 取"两个动作,是**写入 → 巩固 → 提取**的循环 —— 三段时间尺度不同(同步 / 周期 / 同步),设计形态不同
|
||||
> - vault 是**事实层**,只承载经过 LLM 写入认证的关系;`meta/` 是**派生层**,承载概率推断的统计信号
|
||||
> - 任一阶段独立演化,任一信号缺失系统降级而不崩
|
||||
|
||||
---
|
||||
|
||||
## 0. 心智循环:reme 的认知模型
|
||||
|
||||
agent 的长期记忆系统在概念上对应人脑的**海马—皮层回路 + 睡眠—觉醒周期**:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ 外部经验(daily / resource) │
|
||||
└─────────────┬───────────────────┘
|
||||
│ (auto-memory 写 daily)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ │
|
||||
│ ┌────────────────┐ 抽象 / 关系编织 │
|
||||
│ │ Stage 1 │ ◄─ 类比 REM 睡眠 │
|
||||
│ │ auto-dream │ "重放 + 写进 schema" │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 写 vault(digest body + wikilink) │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ vault(事实) │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 只读 │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ 长期组织 / 派生指标 │
|
||||
│ │ Stage 2 │ ◄─ 类比 NREM 慢波睡眠 │
|
||||
│ │ auto-consol- │ "巩固 + 修剪 + 集群" │
|
||||
│ │ idate │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 写 meta/ + audit/(派生层) │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ meta(派生) │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 只读 │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ query → 答案合成 │
|
||||
│ │ Stage 3 │ ◄─ 类比觉醒态 cue retrieval│
|
||||
│ │ auto-recall │ "融合 + pattern complete"│
|
||||
│ └───────┬────────┘ │
|
||||
│ │ │
|
||||
└───────────┼─────────────────────────────────────┘
|
||||
│ 召回结果给 agent
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ agent query │
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
**心智循环回答四个根本问题**:
|
||||
|
||||
| 问题 | 谁回答 |
|
||||
|---|---|
|
||||
| 我经历过什么? | auto-memory(daily 入流) |
|
||||
| 我从中学到什么? | Stage 1 — auto-dream |
|
||||
| 这些知识如何长期组织? | Stage 2 — auto-consolidate |
|
||||
| 我需要时如何调用? | Stage 3 — auto-recall |
|
||||
|
||||
memory 负责"经历落地",cognition 三阶段负责"已落地经历的认知循环"。
|
||||
|
||||
---
|
||||
|
||||
## 1. 三阶段全景
|
||||
|
||||
| 阶段 | 神经科学类比 | 时间尺度 | 改 vault | 实现归属 |
|
||||
|---|---|---|---|---|
|
||||
| **Stage 1 dream** | REM 重放抽象 | 同步(随入流即跑) | 是(写 digest body) | `auto_dream_design.md` |
|
||||
| **Stage 2 consolidate** | NREM 深度巩固 | 周期 / idle(daily / weekly)| **否**(写 `meta/` + `audit/`)| `auto_consolidate_design.md` |
|
||||
| **Stage 3 recall** | 觉醒态 cue retrieval | 同步(query 触发) | 否(只读;唯一对外写是 `meta/access_log.json`)| `auto_recall_design.md` |
|
||||
|
||||
**关键的不对称**:
|
||||
- 写入与检索是**同步**的(用户 / agent 等待),巩固是**离线**的(idle / 周期)
|
||||
- 改 vault 的资格被严格限制在 **dream + consolidate 中的 split** —— 其它阶段全只读
|
||||
- 三阶段时间尺度差三个数量级,这是设计形态(同步 vs 异步 vs idle)的根本来源
|
||||
|
||||
---
|
||||
|
||||
## 2. 系统级能力(贯穿三阶段)
|
||||
|
||||
不属任何单阶段,但任一阶段不能违反:
|
||||
|
||||
| 能力 | 含义 |
|
||||
|---|---|
|
||||
| **事实层 vs 派生层分离** | vault 只承载经 LLM 写入认证的关系(显式 wikilink);`meta/` 承载概率推断的派生指标(community / recency / archived);两者绝不混同 |
|
||||
| **不变量守恒** | F-invariants(0 文件移动 / 改正文限定 subject / wikilink 是 body 一部分)+ E-invariants(边守恒 E-1/E-2/E-3)横跨三阶段;详 `auto_dream_design.md` §4.3-§4.4 |
|
||||
| **阶段独立演化** | 任一阶段算法升级不破坏其它阶段(community 算法换 → dream 不变;打分公式调 → consolidate 不变) |
|
||||
| **缺失即降级** | 任一派生信号缺失,系统降级而不崩;冷启动可用 |
|
||||
| **全程可审计** | 每阶段产 audit / report / log,人 / agent 可检视追溯 |
|
||||
|
||||
---
|
||||
|
||||
## 3. Stage 1 — auto-dream:经验 → 抽象
|
||||
|
||||
**类比**:REM 睡眠的记忆重放与抽象提炼。脑在做梦时把白天事件拆解、重组,提取出可泛化的模式,登记进皮层 schema。
|
||||
|
||||
**根本目的**:把"原始经历"转化为"长期值得调取的教训",同时把它编织进已有知识图谱。
|
||||
|
||||
### 3.1 五个能力维度
|
||||
|
||||
逻辑递进 —— 输入 → 抽象 → 整合 → 编织 → 写入:
|
||||
|
||||
| # | 能力 | 它在问什么 | 失效后果 |
|
||||
|---|---|---|---|
|
||||
| 1 | **抽象判断**(gate) | 这段材料里有"值得长期记住"的东西吗? | 噪声进 vault / 只蒸馏不抽象 |
|
||||
| 2 | **经验重放**(召回) | 这个抽象在已有记忆里**已经存在**吗?以什么形式? | 重复节点 / 错过整合机会 |
|
||||
| 3 | **整合决策** | 创建新节点,还是丰富已有节点?若已有 —— 是再次印证 / 精化范围 / 修正错误? | 已有信息丢失 / 错误没纠正 |
|
||||
| 4 | **关系编织** | 这个抽象与谁有关系?谁是它的来源? | wikilink 缺失,后续 retrieve 漏召 |
|
||||
| 5 | **写入安全** | 写入会不会破坏 vault 既有事实?并发冲突如何处理? | 边丢失 / race condition |
|
||||
|
||||
### 3.2 关键定性
|
||||
|
||||
- dream 是 vault 的**唯一写者**(在 cognition 三阶段里;memory 写 daily 不算)
|
||||
- **写入瞬间是关系建立的唯一可信时机** —— 错过的关系不靠后台扫回(那不是 consolidate 的工作)
|
||||
- 一次写入,所有未来检索受益(持久化优于实时计算)
|
||||
|
||||
详细机制见 `auto_dream_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 4. Stage 2 — auto-consolidate:抽象 → 网络
|
||||
|
||||
**类比**:NREM 慢波睡眠的系统巩固 + 突触代谢稳态。脑在深睡时把分散事件融入 schema、修剪弱连接、把长期不用的记忆淡出意识可达范围。
|
||||
|
||||
**根本目的**:跨时间累积地把 vault 从"一堆节点"组织成"有结构、有权重、有时效的网络",但**只产派生信号,不污染事实层**。
|
||||
|
||||
### 4.1 五个能力维度
|
||||
|
||||
按作用尺度从微观到宏观:
|
||||
|
||||
| # | 能力 | 作用尺度 | 类比 | 输出形态 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **结构维护** | 节点级 | 海马表征过密 → 分化新单元 | 改 vault(split,唯一例外)|
|
||||
| 2 | **跨节点关系发现** | 节点对级 | 多次睡眠中识别"同一件事" → schema | `audit/` 报告 |
|
||||
| 3 | **主题集群形成** | 子图级 | 皮层网络的功能性分区 | `meta/communities.json` |
|
||||
| 4 | **时效性管理** | 节点级 / 时间维度 | 突触代谢稳态 + 遗忘 | `meta/access_log.json` + `meta/archived.json` |
|
||||
| 5 | **健康监控** | 系统级 | 神经环路诊断 | 告警 / 严重告警 |
|
||||
|
||||
### 4.2 关键定性
|
||||
|
||||
- consolidate 是**纯只读 + 派生写**(读 vault,写 `meta/` + `audit/`)
|
||||
- **唯一例外是 split** —— 改 vault 的维护任务,但触发严格(D3 inline 写后)且只改自身负责的 parent + children
|
||||
- **关系判断有错率 → 报告优先,人/agent 介入,不主动合并**(夸大置信度的代价是污染事实层)
|
||||
- 离线 / 周期 / idle —— 与前台不抢资源;失败不影响主流程,下次重跑
|
||||
|
||||
详细机制见 `auto_consolidate_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 5. Stage 3 — auto-recall:网络 → 答案
|
||||
|
||||
**类比**:觉醒态的 cue-driven retrieval + pattern completion。脑接到 query,激活相关皮层模式,补全成完整答案;同时召回过程本身强化被用到的记忆痕迹。
|
||||
|
||||
**根本目的**:接到当前 query 时,从 vault + 派生信号合成最相关的过去经验 —— 既要**覆盖率**(不漏)也要**信噪比**(不冗余)。
|
||||
|
||||
### 5.1 五个能力维度
|
||||
|
||||
按召回流程从输入到输出:
|
||||
|
||||
| # | 能力 | 它在解决什么 |
|
||||
|---|---|---|
|
||||
| 1 | **多路召回** | 不同问法走不同算子(state / semantic / topological 三分立);agent 自选,不强加聚合 verb |
|
||||
| 2 | **多信号融合** | 单一文本相似度不够 —— 还要节点权威性 / 主题集群 / 时效性;乘法融合 |
|
||||
| 3 | **信噪比管理** | 节点级去重 + 节点级 surface(frontmatter 一同呈现)+ multi-hop 可控展开 + 冷藏过滤 |
|
||||
| 4 | **召回反馈** | 被命中的节点 → 写访问日志 → 影响下次 recency / archived 判定 |
|
||||
| 5 | **鲁棒降级** | 派生信号缺失 → 退到基础召回;version 不兼容 → warning + 跳过该因子 |
|
||||
|
||||
### 5.2 关键定性
|
||||
|
||||
- recall 是**只读** —— 唯一对外写入是 `meta/access_log.json`(经 ring buffer + consolidate 聚合)
|
||||
- recall **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法分别由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖
|
||||
- 默认路径 **0 LLM 调用**(信号都是离线维护好的);LLM rerank / query rewrite 是 SDK 上层选项
|
||||
|
||||
详细机制见 `auto_recall_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 6. 能力地图(横切视角)
|
||||
|
||||
15 维按"作用对象"重排,可以看到三阶段如何分工:
|
||||
|
||||
| 作用对象 | dream(写入) | consolidate(巩固)| recall(检索)|
|
||||
|---|---|---|---|
|
||||
| **节点(单个)** | 1 抽象判断 / 3 整合决策 / 5 写入安全 | 1 结构维护(split) | 3 信噪比(节点级合并/surface) |
|
||||
| **节点对 / 关系** | 4 关系编织(wikilink) | 2 跨节点关系发现(dups 报告) | (消费已有边,不产新关系) |
|
||||
| **子图 / 集群** | 2 经验重放(召回邻居) | 3 主题集群形成(community)| 2 多信号融合(community boost) |
|
||||
| **时间维度** | (写入瞬间) | 4 时效性管理(decay / archived)| 4 召回反馈(access log)|
|
||||
| **系统健康** | 5 守恒校验 | 5 健康监控(D1 / D10) | 5 鲁棒降级 |
|
||||
| **入口形态** | 异步 fan-out per sub-unit | 周期 batch / idle | 同步 query response |
|
||||
|
||||
**几个观察**:
|
||||
- "节点对 / 关系"列在 recall 是空 —— recall 不产新关系,只用已有边(避免 query-time 高成本推断)
|
||||
- "时间维度"行 dream 缺位 —— 写入瞬间无"时间维度"概念(那是 consolidate 后续才能提取的统计)
|
||||
- 每行至少有一个阶段负责 —— 没有能力被全阶段忽略
|
||||
|
||||
---
|
||||
|
||||
## 7. 跨阶段不变量
|
||||
|
||||
所有阶段共同遵守的硬约束。任何阶段越界 = 设计错误。
|
||||
|
||||
### 7.1 F-invariants(继承 `auto_dream_design.md` §4.3)
|
||||
|
||||
| # | 约束 | 跨阶段含义 |
|
||||
|---|---|---|
|
||||
| F-1 | 0 文件移动 | 没有任何阶段可以 move 文件;rename 走 `wikilink_handler.retarget_links` 显式路径 |
|
||||
| F-2 | 改正文限定 subject | dream 改 subject body / consolidate split 改 parent + children body;**recall 绝不改任何 body** |
|
||||
| F-3 | maintainer 只做 split | consolidate 内的结构维护只做 split;无 merge / dissolve / re-edge |
|
||||
| F-10 | inbound 不动 | split 后外部 wikilink 仍指 parent,不强制重定向 |
|
||||
| F-11 | wikilink 是 body 一部分 | 没有"独立的边";所有关系变化是 body 编辑副作用 |
|
||||
|
||||
### 7.2 E-invariants(边守恒)
|
||||
|
||||
- E-1:dream update 出边 ⊇ 原出边
|
||||
- E-2:split 后 `(parent_new ∪ ∪children_outbound) ⊇ parent_old`
|
||||
- E-3:inbound wikilink split 时不动
|
||||
|
||||
**recall 不写 body** → E-* 与之无关;但 recall 看到的 wikilink 图永远是 dream / split 守恒后的状态。
|
||||
|
||||
### 7.3 派生信号边界
|
||||
|
||||
- **consolidate / recall 不写 vault** —— 关系判断、活跃度统计、社区划分都是概率推断,不污染事实层
|
||||
- **`meta/*.json` 不被 retrieve 召回** —— 只作权重信号,不进入"召回结果"集合
|
||||
- **audit/ 不被自动消费** —— 报告永远等待人 / agent 介入,不闭环回写
|
||||
|
||||
---
|
||||
|
||||
## 8. 跨阶段数据流(契约总览)
|
||||
|
||||
```
|
||||
┌──────────────┐ wikilink ┌──────────────┐
|
||||
│ auto-dream │─落 body──►│ vault/ │
|
||||
│ (Stage 1) │ │ (事实层) │
|
||||
└──────────────┘ └──────┬──────┘
|
||||
│ 只读
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ auto-consolidate │
|
||||
│ (Stage 2) │
|
||||
└─┬────────┬───────┘
|
||||
│ │
|
||||
meta/ 元数据───┘ └─── audit/ 报告
|
||||
(派生层) (人工介入)
|
||||
│
|
||||
│ 只读
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ auto-recall │ ◄─ user query
|
||||
│ (Stage 3) │
|
||||
└──────┬───────┘
|
||||
│ 命中钩子(异步)
|
||||
▼
|
||||
meta/access_log.json
|
||||
(recall 唯一对外写入,经 consolidate 聚合)
|
||||
```
|
||||
|
||||
| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 |
|
||||
|---|---|---|---|---|
|
||||
| **vault wikilink** | `digest/**.md` body | dream / split | recall(图遍历) | — |
|
||||
| **dups 报告** | `audit/<date>/auto_link_dups.md` | consolidate | 人 / agent | — |
|
||||
| **communities** | `meta/communities.json` | consolidate | recall | 不做同社区 boost |
|
||||
| **access log** | `meta/access_log.json` | recall(写命中) + consolidate(聚合) | recall(读 recency)| recency_factor = 1.0 |
|
||||
| **archived list** | `meta/archived.json` | consolidate | recall(默认过滤)| 不过滤 |
|
||||
| **centrality** | `file_graph` 反向索引(实时,不存)| 自动 | recall(O(1) 查) | — |
|
||||
|
||||
**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。
|
||||
|
||||
**冷启动**:`meta/` 为空 → recall 仍能跑(base + centrality + 图)→ 排序略弱不崩。
|
||||
|
||||
---
|
||||
|
||||
## 9. 系统级断言(把"要什么"提炼到 5 条)
|
||||
|
||||
1. **抽象与事实分层** —— vault 是经 LLM 写过的事实;`meta/` 是统计 / 算法的派生;两者绝不混同
|
||||
|
||||
2. **关系建立的时机集中在写入瞬间** —— dream 写入是关系唯一可信来源;consolidate 不补 vault 关系,recall 不预存关系矩阵
|
||||
|
||||
3. **维护是离线的派生劳动,不是补救** —— consolidate 不修 dream 的疏漏(那叫返工),它做的是 dream 不擅长的事(全局视角 / 统计视角 / 时间视角)
|
||||
|
||||
4. **检索是融合,不是检索** —— recall 的价值不在"找文本相似",而在"把文本 / 图 / 时效 / 权威多个独立信号合成一个答案"
|
||||
|
||||
5. **整个心智循环可降级** —— 任一阶段失效或失准,整个系统降级而不崩;冷启动有意义;dogfooding 可演进
|
||||
|
||||
---
|
||||
|
||||
## 10. 与 auto-memory 的边界
|
||||
|
||||
auto-memory 写入的 daily event 节点也是图的一部分(承载 daily → digest 的 `derived_from::` 边)。但 daily 节点**不参与 cognition 三阶段的全部改造**:
|
||||
|
||||
| cognition 阶段 | 是否触及 daily |
|
||||
|---|---|
|
||||
| **dream** | 只读(作为入流之一) |
|
||||
| **consolidate** | 不参与 dups / community / decay(daily 是时间索引,本质不去重 / 不冷藏) |
|
||||
| **recall** | 三层并行召回时 daily 也参与命中(`structure.md` R-2 默认 `digest > daily > resource`) |
|
||||
|
||||
**关键约束**:cognition 三阶段任何子阶段都**不改写 daily**(无写回路径);daily 由 auto-memory 写完即只读。
|
||||
|
||||
---
|
||||
|
||||
## 11. 演进 / 待补
|
||||
|
||||
**当前实现状态**:
|
||||
- ✅ Stage 1 dream 已实现并跑通(`reme4/steps/evolve/dream/`)
|
||||
- ⏳ Stage 2 consolidate split 部分将实现;dups / community / decay / archived 待实现
|
||||
- ⏳ Stage 3 recall 增强未实现(当前 search.py 已有 vector + keyword + RRF + 一跳 expand)
|
||||
|
||||
**顶层级演进议题**(不属任何单阶段):
|
||||
- ⏳ **能力成熟度路标** —— 把 15 个能力维度按 M0(必须)/ M1(期望)/ M2(演进)分级
|
||||
- ⏳ **跨阶段集成测试** —— vault 从空到充实的端到端 dogfooding,验证三阶段配合是否符合"心智循环"预期
|
||||
- ⏳ **可观测性聚合** —— 三阶段各自的 audit / log 现在分散;是否需要统一的 cognition 健康面板
|
||||
|
||||
各阶段实现进度详见各自文档的"下一步"章节。
|
||||
726
docs4/auto_consolidate_design.md
Normal file
726
docs4/auto_consolidate_design.md
Normal file
|
|
@ -0,0 +1,726 @@
|
|||
# auto-consolidate 设计(Stage 2 巩固:主动解决 vault 长期演化的实际问题)
|
||||
|
||||
> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 2 — 巩固阶段** 实现。覆盖 vault 长期演化中累积的实际问题(冗余 / 过载 / 稀疏 / 腐败 / 抽象缺位),通过周期 batch + 写后 inline 的方式**主动改 vault**,让记忆系统保持健康。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_cognition_design.md`:三阶段顶层心智循环
|
||||
> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型 / F-invariants 原始定义 / 边守恒
|
||||
> - `auto_recall_design.md`:Stage 3 检索 —— 消费本文档产出的信号
|
||||
> - `auto_memory_design.md`:auto-memory 写 daily,daily 节点不参与本文档的巩固改造
|
||||
> - `structure.md` §3.6(maintain 动作语义)
|
||||
>
|
||||
> **核心立场**:
|
||||
> - consolidate **不是产报告等人介入**,是**主动解决问题** —— 类比 NREM 慢波睡眠的 systems consolidation:跨多事件抽 schema、修剪弱连接、稳态突触强度。这些都是真实发生的改造
|
||||
> - vault **会被 consolidate 改**,但每个动作有严格的**置信度门槛 + 守恒规则 + 审计 trail + 渐进 rollout**
|
||||
> - 灰色地带(置信度不够)才产报告等人介入;高置信度自己解决
|
||||
> - **community detection 是巩固的中枢** —— P0 基础设施,P1-P3 三个动作(abstract / merge / reinforce)都依赖它
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述与五大动作全景
|
||||
|
||||
dream 写入是单点视角,有三类视野局限:**写入瞬间没有跨节点视角 / 跨时间视角 / 全局拓扑视角**。这些局限会让 vault 长期演化中累积五类实际问题:
|
||||
|
||||
| # | 问题 | 类比 | 表现 | 解决 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **冗余** | 同事件留下重复记忆痕迹 | dream 漏判去重 / 术语演化 / 跨桶建成两份 | merge |
|
||||
| 2 | **过载** | 单一突触表征过密 | 节点 body 累积过长 / 单节点杂糅多主题 | split |
|
||||
| 3 | **稀疏** | 应有连接未建立 | dream 写入瞬间漏召回的相关节点 / 反复共现但无 wikilink | reinforce |
|
||||
| 4 | **腐败** | 长期不激活的痕迹 | 旧节点过时 / 半年没人读 / 内容已被矛盾 | archive |
|
||||
| 5 | **抽象缺位** | 跨多 instance 缺 schema | vault 只有原子节点,没有"主题层"视角承接全局问 | abstract |
|
||||
|
||||
### 0.1 五大动作 + 优先级
|
||||
|
||||
| 优先级 | 动作 | 解决问题 | 触发节奏 | 改 vault | 风险 | 收益 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **P0** | **community detection** | (基础设施) | weekly batch | 否 | 0(只产 meta) | 基础(其它三个都靠它)|
|
||||
| **P1** | **abstract** | 抽象缺位 | weekly batch(基于 P0) | 是(新建 summary) | 低(additive) | **最高**(GraphRAG 核心) |
|
||||
| **P2** | **merge** | 冗余 | weekly batch(基于 P0) | 是(合并 + retarget) | 高(lossy) | 中(消除可见冗余) |
|
||||
| **P3** | **reinforce** | 稀疏 | weekly batch(基于 P0) | 是(additive 加 wikilink) | 低 | 低(retrieve multi-hop 已部分弥补)|
|
||||
| **(独立)** | **split** | 过载 | inline 写后(D3) | 是(拆 parent + children) | 低 | 中 |
|
||||
| **(独立)** | **archive** | 腐败 | daily batch | 软(meta 标记) | 0 | 中 |
|
||||
|
||||
**关键论断**:**P1 比 P2 优先** —— abstract additive 失败可逆且回报最大;merge lossy 失败要回滚 inbound,价值是消除冗余(必要但不增能力)。
|
||||
|
||||
### 0.2 实施路径
|
||||
|
||||
```
|
||||
M0: P0 community detection (基础设施)
|
||||
+ split (已实现)
|
||||
+ archive (软标记,完全可逆)
|
||||
|
||||
M1.1: P1 abstract (additive,最低风险开始改 vault)
|
||||
M1.2: P2 merge (lossy,高门槛 + 多数票)
|
||||
M1.3: P3 reinforce (additive,价值最低,可缓做)
|
||||
|
||||
M2+: 多层 abstract (L2 super-community) / delete / typed predicate reinforce
|
||||
```
|
||||
|
||||
### 0.3 显式排除
|
||||
|
||||
- ❌ 重做"抽象判断" —— gate 决策只在 dream(consolidate 不重新判定"该不该记")
|
||||
- ❌ 重做"语义内容" —— UPDATE 三种 flavor(CORROBORATE / REFINE / CORRECT)只在 dream;consolidate 做结构层,不做语义层
|
||||
- ❌ 改 daily / resource —— consolidate 只动 digest 节点(I-2 / I-3 仍守)
|
||||
|
||||
---
|
||||
|
||||
# Part A — community 工作群(本文档核心)
|
||||
|
||||
P0-P3 四件套围绕 community detection 协同工作:**community 提供"哪些节点同主题"的判据,abstract / merge / reinforce 各自利用这个判据做不同的解决动作**。
|
||||
|
||||
## 1. community detection(P0,基础设施)
|
||||
|
||||
**目的**:在 vault wikilink 图上做 community detection,产出"节点 → community_id"映射。这是 P1-P3 三个动作的**唯一前置**。
|
||||
|
||||
### 1.1 算法选择:Leiden
|
||||
|
||||
| 选项 | 评估 |
|
||||
|---|---|
|
||||
| Louvain | 经典,但有 resolution limit + disconnected community 风险 |
|
||||
| **Leiden** ✅ | Louvain 改进版(2019),稳定性显著好;GraphRAG 采用;Python `igraph.community_leiden` 现成 |
|
||||
| label propagation | 实现最简,但结果不稳定(随机种子敏感) |
|
||||
|
||||
**首版决策:Leiden**,直接对齐 GraphRAG 路线,后续接它的多层抽象更顺。
|
||||
|
||||
### 1.2 图的形态
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **节点范围** | **只 digest 节点**;daily / resource 不参与 |
|
||||
| **边权重** | **首版 unweighted undirected**(所有 wikilink 等权)—— 加权方案(predicate 类型加权)留 M2+ 视效果 |
|
||||
| **跨桶 community** | **必须允许** —— bucket 是物理归档,community 是语义聚合,二者本就正交。"错桶节点"会被自然纳入 community,可作 audit 信号但不强制 move(F-1 守住)|
|
||||
| **resolution** | **1.0 起步**(Leiden 默认 / GraphRAG 默认)—— dogfooding 后视 community 平均规模(理想 5-15 节点)调 |
|
||||
| **更新模式** | **全量重算**;vault 千节点级 Leiden < 1 秒,M0/M1 不引入增量复杂度 |
|
||||
|
||||
### 1.3 多层级:M1 只 L1
|
||||
|
||||
| 层数 | 适用 | reme 决策 |
|
||||
|---|---|---|
|
||||
| 单层 L1(原子 → community)| vault < 500 节点足够 | **M1 起步** |
|
||||
| 双层 L1 + L2(community → super-community) | vault > 500 节点 / 跨主题大类涌现 | M2+ 视规模 |
|
||||
| GraphRAG 4 层 | 大规模文档库 | M3+ 不优先 |
|
||||
|
||||
理由:GraphRAG 论文证明 L1 拿走 60-80% 效果。先把 L1 跑稳,L2 看实际是否需要。
|
||||
|
||||
### 1.4 输出
|
||||
|
||||
**`meta/communities.json`**:
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"computed_at": "2026-06-08T03:00:00Z",
|
||||
"algorithm": "leiden",
|
||||
"resolution": 1.0,
|
||||
"communities": {
|
||||
"digest/auth/jwt-rotation.md": "c_07",
|
||||
"digest/auth/oauth-flow.md": "c_07",
|
||||
"digest/api/rate-limit.md": "c_12"
|
||||
},
|
||||
"stats": {
|
||||
"n_communities": 14,
|
||||
"median_size": 7,
|
||||
"max_size": 23
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`meta/community_changes.json`**(供 abstract 稳定度判据):
|
||||
```json
|
||||
{
|
||||
"computed_at": "...",
|
||||
"previous": "...",
|
||||
"stability_per_community": {
|
||||
"c_07": 0.92, // 1 - (Jaccard 距离与上周该 community 节点集)
|
||||
"c_12": 0.45 // 不稳定,abstract 跳过
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 1.5 community_id 不需要稳定
|
||||
|
||||
下游(abstract / merge / reinforce)只关心"两节点是否同 community";id 本身可重排。每周重算后 id 不需要保持与上周对齐。stability 信号通过节点集 Jaccard 距离计算,不依赖 id。
|
||||
|
||||
### 1.6 用途总览
|
||||
|
||||
| 下游 | 用法 |
|
||||
|---|---|
|
||||
| **abstract**(§2)| 判据"该 community 节点数 ≥ N + 稳定度满足 + 无 hub" → 创建 summary |
|
||||
| **merge**(§3)| 候选 pair 必须在同 community(降错率;不同 community 的相似 description 多是同名异义)|
|
||||
| **reinforce**(§4)| 候选 wikilink 必须在同 community(避免假关联)|
|
||||
| **recall**(`auto_recall_design.md` §3) | 同 community 节点 boost |
|
||||
|
||||
---
|
||||
|
||||
## 2. abstract(P1,抽象提升)
|
||||
|
||||
**类比**:NREM systems consolidation —— 跨多次睡眠把分散事件抽出共同 schema,从 episodic 升到 semantic。
|
||||
|
||||
**目的**:vault 演化到一定规模后,某些 community 形成稳定主题群,需要一个 hub 节点统领,让 retrieve 能召回到"主题概览"而非散点。
|
||||
|
||||
### 2.1 等价处理立场(关键)
|
||||
|
||||
**summary 节点完全等同普通节点**:
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **路径** | LLM 选桶,正常 slug 命名(如 `digest/auth/authentication-mechanisms.md`);**无 `__community__` / `__hub__` 等结构性标识** |
|
||||
| **frontmatter** | 仅 `name + description`(reme 核心保留);**无 `kind: community_summary`、无 `auto_generated`** |
|
||||
| **summary 性质** | 完全体现在 **body 形态** —— 主题概述 + 列出 source 节点 wikilink + 跨节点 pattern;但这是内容自然形态,不是结构性宣告 |
|
||||
| **后续维护** | **无** —— 跟其它节点等价,被 dream / split / merge / archive 自然演化(参见 §2.6) |
|
||||
|
||||
这跟 dream 的核心立场对齐:"节点角色由 body 内容决定,不由 frontmatter 类型标记"。abstract 是"用一种新方式创造节点",不是"创造一种新节点类型"。
|
||||
|
||||
### 2.2 触发判据(组合门槛)
|
||||
|
||||
```
|
||||
weekly batch:
|
||||
for community in communities.json:
|
||||
if community_has_hub(community): # §2.5 结构化判据
|
||||
continue
|
||||
if len(community) < MIN_NODES (5): # 节点数门槛
|
||||
continue
|
||||
if stability(community) < 0.7: # 稳定度门槛
|
||||
continue
|
||||
if active_node_count(community, 30d) < 3: # 活跃度门槛
|
||||
continue
|
||||
if name_diversity(community) < 0.5: # 多样性门槛
|
||||
continue
|
||||
→ enqueue abstract job
|
||||
```
|
||||
|
||||
| 门槛 | 默认 | 含义 | 防的是 |
|
||||
|---|---|---|---|
|
||||
| **节点数** | ≥ 5 | community 大小 | 给 2-3 节点造 hub 不划算 |
|
||||
| **稳定度** | ≥ 0.7 | 与上周边界 Jaccard 距离 | 给短命 community 造 hub 浪费 |
|
||||
| **活跃度** | ≥ 3 节点近 30 天 hit | community 仍在用 | 给死社区造 hub(下次没人看)|
|
||||
| **多样性** | name 差异度 ≥ 0.5 | frontmatter `name` 互不相同 | 给"一组重复节点"造 summary —— 那是 merge 的事 |
|
||||
|
||||
### 2.3 创建动作 + grounding 守恒
|
||||
|
||||
```
|
||||
LLM 看 community 内所有节点 (frontmatter + body)
|
||||
↓
|
||||
产 planned summary body (三段):
|
||||
1. 主题概述 (1-2 段,跨多节点共同主题)
|
||||
2. 关键支柱 (列表,3-5 节点 + 一句话 + wikilink)
|
||||
3. 不在概览的细节 (明说哪些细节留原节点)
|
||||
↓
|
||||
长度限制: summary body < 1500 token
|
||||
(防 abstract 创建后立刻被 split 触发,§5)
|
||||
↓
|
||||
LLM 决定 path: digest/<bucket>/<slug>.md
|
||||
↓
|
||||
CAS 写入 (§9) + 双重守恒校验:
|
||||
- 机械: 出边集合 ⊇ "关键支柱"声称引用的节点 (防套话)
|
||||
- 机械: 出边集合 ⊇ source_nodes 的至少 60% (allow LLM 漏列少数)
|
||||
↓
|
||||
audit 记录: audit/<date>/consolidate_actions.md
|
||||
```
|
||||
|
||||
**grounding 守恒**:summary body 中**声称引用某节点必须真写 wikilink**。LLM 不能仅口头提及"我们在 X 中看到..."而不带 `[[X.md]]`。这是机械可校验的,LLM 跑不掉。
|
||||
|
||||
### 2.4 长度限制为什么重要
|
||||
|
||||
summary body < 1500 token 是**与 split 互锁的机制**:
|
||||
|
||||
- 不限长 → LLM 会写"完整覆盖" → 最终 body 累积接近 split 阈值(2000 token)→ 下次 D3 触发拆 → 拆出来的 children 又被 community 视为同主题 → 下次 abstract 又造一个 hub → 循环
|
||||
- 限长 1500 → summary 留出 split 阈值的 25% buffer,稳定不触发拆
|
||||
|
||||
### 2.5 "community 已有 hub"的结构化判据
|
||||
|
||||
不靠 frontmatter / 路径标识,靠**结构**:
|
||||
|
||||
```
|
||||
def community_has_hub(community):
|
||||
for node in community:
|
||||
out_targets = outbound(node) ∩ community
|
||||
if len(out_targets) / len(community) >= 0.6:
|
||||
return True # 该节点出边覆盖 community 60% 以上 → 它已是 hub
|
||||
return False
|
||||
```
|
||||
|
||||
**好处**:
|
||||
- split parent overview 自然被识别为 hub(split parent 出边覆盖大部分 children)→ abstract **复用** split 的工作,不重复创建
|
||||
- 已有 abstract 创建过的节点,只要它出边没退化,下次 batch 自然识别为 hub,不重复创建
|
||||
- 节点被 dream update 后形态变化,出边变了 → 自动重新评估
|
||||
|
||||
**M1 实施关键验证点**:跑实测验证这个涌现 —— split parent 是否真被识别为 hub。如有 corner case,调阈值 0.6 → 0.5 / 0.7。
|
||||
|
||||
### 2.6 后续维护:无 —— 完全靠 5 大动作演化
|
||||
|
||||
abstract 创建即放归 vault,**consolidate 不再"管"它**。后续命运:
|
||||
|
||||
| 演化路径 | 结果 |
|
||||
|---|---|
|
||||
| 新材料触及该主题 | dream update 自然修正 body(走 CORROBORATE / REFINE / CORRECT)|
|
||||
| 老 summary 长期不被引用 | archive 自动归档(§6)|
|
||||
| community 边界变了 → 下次 batch 创建新 summary | 新老 summary 描述同主题 → merge 自动合并(§3)|
|
||||
| summary body 累积过长 | split 自动拆(§5)|
|
||||
|
||||
这是真正的"vault 自我代谢"。**没有特殊维护通道**。
|
||||
|
||||
---
|
||||
|
||||
## 3. merge(P2,同概念合并)
|
||||
|
||||
**类比**:NREM 跨多次睡眠识别"同一件事" → 合一个记忆痕迹。
|
||||
|
||||
**目的**:消除 vault 内的冗余 —— 同概念多节点。
|
||||
|
||||
### 3.1 候选挖掘(community 内三层过滤)
|
||||
|
||||
```
|
||||
weekly batch (依赖 community detection):
|
||||
for community in communities:
|
||||
pairs = all_pairs(community)
|
||||
for (A, B) in pairs:
|
||||
if description_sim(A, B) < 0.6: # 第一层: frontmatter 相似
|
||||
continue
|
||||
if body_topic_overlap(A, B) < 0.5: # 第二层: body 主题词重合
|
||||
continue
|
||||
if cooldown_active(A) or cooldown_active(B): # 第三层: cooldown 检查
|
||||
continue
|
||||
candidates.append((A, B))
|
||||
```
|
||||
|
||||
**关键约束**:候选必须在**同 community**(降错率)。
|
||||
|
||||
### 3.2 多数票决策
|
||||
|
||||
merge 是高风险动作(lossy + 改 inbound),用多数票降错:
|
||||
|
||||
```
|
||||
for (A, B) in candidates:
|
||||
votes = parallel_run(N=3, prompt="A 和 B 是否同一概念? 返回 {is_same, confidence}")
|
||||
agree = sum(v.is_same and v.confidence >= 0.8 for v in votes)
|
||||
if agree >= 2:
|
||||
→ enqueue merge job
|
||||
elif agree == 1:
|
||||
→ 写 audit/<date>/dups_uncertain.md (灰色地带,人介入)
|
||||
else:
|
||||
→ 丢弃
|
||||
```
|
||||
|
||||
### 3.3 merge 动作:body 重写归 consolidate(方案 B)
|
||||
|
||||
**关键决策**:merge 后的 body 由 **consolidate 自跑合并 prompt**,不走 dream update 路径。
|
||||
|
||||
| 方案 | 评估 | 决策 |
|
||||
|---|---|---|
|
||||
| A. 走 dream update 路径(把 loser body 作"新材料")| 优雅但跨阶段;dream 不应知道 caller 是 consolidate 还是新材料 | ❌ |
|
||||
| **B. consolidate 自跑合并 prompt** | 简单自包含;通过严格 prompt 约束化解"做语义工作"张力 | ✅ |
|
||||
| C. 不重写 body(留 redirect stub) | 完全不做语义,但 vault 留无用节点 | ❌ |
|
||||
|
||||
**B 方案的边界守住**(避免 consolidate 真在做语义判断):
|
||||
|
||||
| 边界 | 含义 |
|
||||
|---|---|
|
||||
| **prompt 严格约束** | "只合并不精化" —— 不重写措辞、不加新内容、不做精化决策 |
|
||||
| **机械守恒** | 出边 ⊇ A.outbound ∪ B.outbound + provenance 全保留(LLM 跑不掉) |
|
||||
| **信息守恒抽样** | LLM 自检 "merged.body ⊇ A.body ∪ B.body 全部信息";audit 抽样人审 |
|
||||
| **失败拒写** | 守恒校验失败 → LLM 重试一次 → 二次失败拒写 + audit |
|
||||
|
||||
### 3.4 完整动作流
|
||||
|
||||
```
|
||||
A, B → 选择 winner (path):
|
||||
- inbound 数大者赢 (保护既有 inbound,降 retarget 量)
|
||||
- 平局取路径短者
|
||||
↓
|
||||
LLM 跑 merge prompt → planned merged_body (B 方案)
|
||||
↓
|
||||
机械 retarget 准备:
|
||||
- 扫所有 inbound(loser): [[loser.md]] → [[winner.md]]
|
||||
- alias 保留;predicate 保留
|
||||
- 这是机械算子,非 LLM
|
||||
↓
|
||||
事务式 CAS 写入:
|
||||
1. winner body 改写
|
||||
2. 所有 inbound 节点 body 改写 (retarget)
|
||||
3. 删除 loser 文件
|
||||
任一步失败 → 全部回滚
|
||||
↓
|
||||
audit 记录 + cooldown 设置 (winner 进 cooldown 2 weeks)
|
||||
```
|
||||
|
||||
### 3.5 灰色地带:报告
|
||||
|
||||
- 多数票通过(agree ≥ 2)→ 自动 merge
|
||||
- 仅 1 票通过 → 写报告 `audit/<date>/dups_uncertain.md`,人 / agent 介入
|
||||
- 0 票 → 丢弃
|
||||
|
||||
报告格式:
|
||||
```markdown
|
||||
# dups uncertain 2026-06-08
|
||||
|
||||
## pair 1 (1/3 votes)
|
||||
- A: digest/auth/jwt-rotation.md ("JWT 密钥轮换")
|
||||
- B: digest/security/key-rotation.md ("密钥轮换原则")
|
||||
- vote 1 (yes, 0.85): "同一概念,A 偏 JWT 场景"
|
||||
- vote 2 (no, 0.72): "B 是通用原则,A 是具体应用"
|
||||
- vote 3 (no, 0.68): "粒度不同,不应合并"
|
||||
|
||||
建议:走 dream update 通道把 A 内容作为 B 的实例并入。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. reinforce(P3,关系强化:补 dream 漏的 wikilink)
|
||||
|
||||
**类比**:NREM 突触强化 LTP —— 反复共激活的连接被强化。
|
||||
|
||||
**目的**:vault 演化中,某些节点对应该有 wikilink 但 dream 写入时漏召。reinforce 周期检测并 additive 补。
|
||||
|
||||
### 4.1 候选挖掘(三层过滤)
|
||||
|
||||
```
|
||||
weekly batch (依赖 community detection):
|
||||
for community in communities:
|
||||
for (A, B) in all_pairs(community):
|
||||
if has_wikilink(A, B):
|
||||
continue
|
||||
# 第一层: 字符串 mention 锚点
|
||||
if not has_mention(A.body, B.frontmatter.name):
|
||||
continue
|
||||
# 第二层: embedding 相似度验证
|
||||
if embedding_sim(A.context_around_mention, B.body) < 0.7:
|
||||
continue
|
||||
# 第三层: 同 community (已经是,但显式说明)
|
||||
candidates.append((A, mention_pos, B))
|
||||
```
|
||||
|
||||
**三层过滤的角色**:
|
||||
|
||||
| 层 | 防的是 |
|
||||
|---|---|
|
||||
| 字符串 mention | 大幅降候选数(从 O(N²) 降到 O(实际共现)) |
|
||||
| embedding 相似度 | 防同名异义("Apple" 公司 vs 水果)|
|
||||
| 同 community | 防表面术语共现但语义无关 |
|
||||
|
||||
### 4.2 决策(单票即可,门槛较高)
|
||||
|
||||
reinforce 是 additive 低风险动作,不需要多数票:
|
||||
|
||||
```
|
||||
for (A, mention_pos, B) in candidates:
|
||||
vote = LLM("A.body 在该位置提到 B 的概念。是否合理加 [[B.md]] 链接?")
|
||||
if vote.confidence >= 0.85:
|
||||
additive_wikilink(A, mention_pos, target=B.path)
|
||||
→ CAS 写入 (E-1 自动满足:additive 只增不删)
|
||||
→ audit 记录
|
||||
else:
|
||||
丢弃
|
||||
```
|
||||
|
||||
### 4.3 边界
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **只 additive 加 wikilink** | 不改 body 文字,不升级 typed predicate(predicate 升级是语义判断,留 dream)|
|
||||
| **alias 保留原文** | `[[B.md\|<原文 mention>]]`;原文一字不改 |
|
||||
| **写入位置** | mention 第一次出现处加;后续保持原文(防 wikilink 满文) |
|
||||
| **不动 anchor** | 与 dream 一致 |
|
||||
| **守恒** | E-1 天然满足(纯增) |
|
||||
| **rollback** | 误链发生时,人 / agent 直接编辑 body 删除 wikilink 即可;reinforce 不维护"我加过哪些"audit log(每次动作进 `audit/<date>/consolidate_actions.md`)|
|
||||
|
||||
### 4.4 reinforce 与 dream 的边界
|
||||
|
||||
dream 写入时 LLM 应已尽力召回相关节点 + 加 wikilink。reinforce 是**周期性兜底** —— 写入瞬间漏的、术语后才一致的、被 split 拆出来后才相关的,在 reinforce batch 里被检出。
|
||||
|
||||
这不违反"consolidate 不修 dream 漏的"立场 —— **dream 漏的 wikilink 在巩固阶段补,是合法工作**(它的依据是 dream 单点视角永远做不到的"周期统计 + 全局视角");**dream 漏的语义抽象在巩固阶段不补**(那是 dream 的语义判断,consolidate 不重做)。
|
||||
|
||||
---
|
||||
|
||||
# Part B — 独立工作
|
||||
|
||||
P0-P3 围绕 community,这两个动作独立运行。
|
||||
|
||||
## 5. split(过载分化:inline 写后)
|
||||
|
||||
**类比**:海马表征过密 → 分化新单元。
|
||||
|
||||
**目的**:节点 body 累积过长 / 主题离散后,拆成 parent overview + N children,保持单节点"一个原子语义单元"的粒度。
|
||||
|
||||
### 5.1 触发模型(写后立即,inline)
|
||||
|
||||
split 是 5 大动作中**唯一 inline** 的 —— 跟 dream 写入流强耦合,不走 weekly batch:
|
||||
|
||||
```
|
||||
dream / split 写 body 成功 (CAS 通过)
|
||||
└─ if len(body) > T_token (default 2000):
|
||||
└─ LLM 判离散度
|
||||
└─ if is_overloaded:
|
||||
└─ enqueue split job (FIFO, CAS-protected)
|
||||
└─ return (不阻塞 dream)
|
||||
```
|
||||
|
||||
理由:节点过载是**写入瞬间的本地信号**(token + 离散度),延后无价值;反应即时。
|
||||
|
||||
### 5.2 split 动作
|
||||
|
||||
```
|
||||
LLM 看 parent body:
|
||||
- 拆成 1 个 parent overview body + N 个 children body
|
||||
- 每个 child 自带 [[parent]] 反向链接
|
||||
- inbound 不动 (F-10)
|
||||
↓
|
||||
机械 outbound 守恒校验 (E-2):
|
||||
(parent_new ∪ ∪children_outbound) ⊇ parent_old
|
||||
失败 → LLM 重试 → 二次失败拒写 + audit
|
||||
↓
|
||||
事务式 CAS 写入: parent body 改写 + N 个新 children 文件创建
|
||||
↓
|
||||
audit + cooldown 设置 (parent + children 进 cooldown,与 merge 互锁)
|
||||
```
|
||||
|
||||
### 5.3 split 与 abstract 的协同(关键)
|
||||
|
||||
| | 起源 | 方向 | 触发 |
|
||||
|---|---|---|---|
|
||||
| split overview | 单节点过载分化 | 自上而下(一拆多)| inline 写后 D3 |
|
||||
| abstract summary | 多节点抽象凝聚 | 自下而上(多归一)| weekly batch + 稳定度阈值 |
|
||||
|
||||
**协同**:split 产出的 overview 节点会被 §2.5 的"已有 hub"判据识别,abstract 不重复创建。两者互补,不冲突。
|
||||
|
||||
---
|
||||
|
||||
## 6. archive(时效衰减:让长期不激活的节点淡出)
|
||||
|
||||
**类比**:突触代谢稳态 —— 长期不用的连接被减弱,但不删除。
|
||||
|
||||
**目的**:让 retrieve 默认排除"已不活跃"的节点,提升信噪比;不删 vault 文件,保持可逆。
|
||||
|
||||
### 6.1 recency_score:连续衰减信号
|
||||
|
||||
```
|
||||
recency_score(node) =
|
||||
exp(-(now - last_update) / τ_update) # 时间衰减
|
||||
× (1 + log(1 + last_hit_count_30d)) # 活跃度增强
|
||||
× (1 + log(1 + inbound_count) / SCALE) # 中心性 cushion(避免 hub 被冷藏)
|
||||
```
|
||||
|
||||
| 参数 | 默认 | 含义 |
|
||||
|---|---|---|
|
||||
| τ_update | 60 days | 时间衰减常数 |
|
||||
| SCALE | 10 | 中心性 cushion 缩放 |
|
||||
|
||||
输出:`meta/recency.json`,每节点 0.0~1.0 连续值。
|
||||
|
||||
### 6.2 archived 派生快照
|
||||
|
||||
archived 是 recency_score 的二元化派生:
|
||||
|
||||
```
|
||||
archived = {node | recency_score(node) < 0.15}
|
||||
```
|
||||
|
||||
输出:`meta/archived.json`,recall 默认过滤这个列表。
|
||||
|
||||
### 6.3 解冻
|
||||
|
||||
任何动作触及节点 → 自动从 archived 移除:
|
||||
- retrieve 命中(写 access_log)
|
||||
- dream update 触及
|
||||
- merge / reinforce 触及
|
||||
|
||||
下次 batch 时 recency_score 重算自然超过阈值。
|
||||
|
||||
### 6.4 daily 节奏
|
||||
|
||||
archive 是唯一不需要 community detection 的动作 → 节奏可以更快(daily batch),让冷启动后第二天就能影响 recall。
|
||||
|
||||
```
|
||||
daily batch:
|
||||
1. 读 access_log (retrieve / dream / consolidate 钩子记录的命中事件)
|
||||
2. 重算 recency_score for all digest nodes
|
||||
3. 输出 meta/recency.json
|
||||
4. 阈值过滤 → meta/archived.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Part C — 共享基础设施
|
||||
|
||||
## 7. F-invariants 松绑与守恒规则
|
||||
|
||||
旧 F-invariants(`auto_dream_design.md` §4.3)在"vault 只读"立场下定义,新立场要松绑。但松绑不是"自由改",是用**动作级守恒规则**换"一刀切禁令"。
|
||||
|
||||
### 7.1 F-invariants 修订
|
||||
|
||||
| # | 旧约束 | 新立场 |
|
||||
|---|---|---|
|
||||
| **F-1** | 0 文件移动 | **改为**:"非 consolidate 动作不移动文件";merge 删除 loser 文件是**合法移动**(逻辑上等价 retarget) |
|
||||
| **F-2** | 改正文限定 subject | **改为**:"dream / split / reinforce 改 subject body;merge 在受控算子内可改 inbound 节点 body";其它阶段(recall)绝不改 |
|
||||
| **F-3** | maintainer 只做 split | **作废** —— consolidate 5 大动作合法 |
|
||||
| **F-10** | inbound 不动 | **改为**:"split 时 inbound 不动";merge 必须 retarget inbound(机械算子) |
|
||||
| **F-11** | wikilink 是 body 一部分 | **保留** —— 没有"独立的边"基础设施 |
|
||||
|
||||
### 7.2 动作级守恒规则矩阵
|
||||
|
||||
| 动作 | 置信度门槛 | 守恒规则 |
|
||||
|---|---|---|
|
||||
| **abstract** | community 节点 ≥ 5 + 稳定度 ≥ 0.7 + 活跃度 ≥ 3 + 多样性 ≥ 0.5 + 无 hub | 出边 ⊇ "关键支柱"列表 + 出边 ⊇ source 节点 60%(机械)|
|
||||
| **merge** | LLM 多数票 ≥ 2/3 + similarity ≥ 0.6 + body overlap ≥ 0.5 | 信息守恒(merged.body ⊇ A ∪ B)+ 出边 ⊇ A.out ∪ B.out + inbound 全 retarget(机械)|
|
||||
| **reinforce** | LLM 单票 ≥ 0.85 + 同 community + mention 锚点存在 + embedding ≥ 0.7 | E-1 天然(additive)|
|
||||
| **archive** | recency_score < 0.15 | 软标记,无破坏性 |
|
||||
| **split** | token > T + LLM 判离散 | E-2(parent ∪ children ⊇ parent_old)+ inbound 不动 |
|
||||
|
||||
---
|
||||
|
||||
## 8. cooldown 与防循环
|
||||
|
||||
5 大动作之间的潜在循环:
|
||||
|
||||
```
|
||||
A merge B → AB body 长 → split AB 回 A' + B' → 又 merge → ...
|
||||
```
|
||||
|
||||
防御:
|
||||
|
||||
| 互锁对 | 窗口 | 实现 |
|
||||
|---|---|---|
|
||||
| **split → merge** | 2 weeks | 刚 split 出的兄弟节点不参与 merge 候选 |
|
||||
| **merge → split** | 2 weeks | 刚 merge 的节点不参与 split 评估(D3 检测时跳过)|
|
||||
| **merge → merge**(同对反复) | 12 weeks | 同一 path 12 周内被 merge 又被识别为新 merge 候选 → audit 警报,人介入 |
|
||||
| **abstract → merge**(同主题反复 abstract) | 4 weeks | 刚 abstract 出的 hub 节点 4 周内不参与 merge 候选 |
|
||||
|
||||
cooldown 状态外置 `meta/cooldowns.json`,不污染 vault。
|
||||
|
||||
---
|
||||
|
||||
## 9. CAS 写入协议(共享基础设施)
|
||||
|
||||
CAS 是 dream(`auto_dream_design.md` §4.2)、split / merge / reinforce / abstract(本文档)**多方共用**的 vault 写入协议。归本文档因 consolidate 是写入主战场。
|
||||
|
||||
archive 不写 vault → 不走 CAS;它写 `meta/`,各任务的 atomic write(write-temp + rename)即可。
|
||||
|
||||
### 9.1 协议
|
||||
|
||||
```
|
||||
1. 读 + 记戳: read body → version_stamp = sha256(body) | mtime
|
||||
2. 决策: LLM / 算法 → 产 planned new_body
|
||||
3. CAS 写入: 重读 body 比 version_stamp
|
||||
- 未变: 跑动作级守恒校验 → 通过 → atomic write (write-temp + rename) → done
|
||||
- 已变: 丢弃 planned new_body, 带最新 body 重走 step 1
|
||||
4. 守恒校验失败: LLM 重试一次, 二次失败拒写 + audit
|
||||
5. 重做次数上限: 3 次 → 跳过候选 + audit log
|
||||
```
|
||||
|
||||
### 9.2 事务式 merge / split 写入
|
||||
|
||||
merge 涉及多文件写入(winner body + N 个 inbound retarget + loser 删除);split 涉及多文件创建(parent body + N children)。需要事务语义:
|
||||
|
||||
- 准备阶段:全部 planned new_body 写到 temp 区(带 version_stamp)
|
||||
- 提交阶段:逐个 CAS 检查 + atomic write(write-temp + rename)
|
||||
- 任一 CAS 失败 → 全部回滚(temp 区清理,已 rename 的恢复)
|
||||
|
||||
实现细节:可借 fs-level 事务库(如 `pyrsistent` 模式)或自实现 journal。M0 起步用最简的"先全部检查 → 再全部写入"两阶段,接受窗口期(检查到写入间)的极小并发风险。
|
||||
|
||||
### 9.3 create 路径 race
|
||||
|
||||
merge / abstract 都可能并发 create 同一 path → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者 EEXIST → 重走 step 1(此时大概率改判 update 或丢弃)。
|
||||
|
||||
### 9.4 不解决
|
||||
|
||||
- 跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+)
|
||||
- 高冲突 workload(同候选反复触发)→ 重做上限触发后 audit
|
||||
|
||||
---
|
||||
|
||||
## 10. D 健康检查(D1 / D10)
|
||||
|
||||
不属"巩固"主语义,但跟 consolidate 同节奏(周期 batch 顺手跑),归本文档:
|
||||
|
||||
| # | 信号 | 节奏 | 修复策略 |
|
||||
|---|---|---|---|
|
||||
| **D1** | 断链(wikilink → 不存在 path) | 写时 inline + weekly batch 巡检(双重保险)| 就地删 wikilink 或保留 alias 文本 → audit |
|
||||
| **D10** | provenance 断裂(digest 反指的 daily/resource 不可达)| 同上 | I-不变量违反 → 严重告警 + 人介入 |
|
||||
|
||||
D1 / D10 不算 5 大动作之一(它们不解决"vault 演化问题",只检测异常)。但它们的修复(就地删 wikilink)需要走 CAS,所以协议共享。
|
||||
|
||||
---
|
||||
|
||||
# Part D — 契约与实施
|
||||
|
||||
## 11. 维护 → 检索契约
|
||||
|
||||
5 大动作产物给 retrieve 消费(详细 retrieve 逻辑见 `auto_recall_design.md`):
|
||||
|
||||
| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 |
|
||||
|---|---|---|---|---|
|
||||
| **vault 节点变化** | `digest/**.md` | merge / split / reinforce / abstract | recall(图遍历 / 命中) | — |
|
||||
| **communities** | `meta/communities.json` | community detection | recall + abstract / merge / reinforce | 不做同社区 boost / 三个动作跳过 |
|
||||
| **community changes** | `meta/community_changes.json` | community detection | abstract 决策 | abstract 跳过(无稳定度判据)|
|
||||
| **recency** | `meta/recency.json` | archive daily batch | recall | recency_factor = 1.0 |
|
||||
| **archived** | `meta/archived.json` | archive daily batch | recall(默认过滤)| 不过滤 |
|
||||
| **cooldowns** | `meta/cooldowns.json` | split / merge | consolidate 内部 | 无防御循环 |
|
||||
| **access_log** | `meta/access_log.json` | recall(写命中) + archive(聚合) | archive(读 recency) | recency 不衰减 |
|
||||
| **dups uncertain** | `audit/<date>/dups_uncertain.md` | merge | 人 / agent | — |
|
||||
| **consolidate actions** | `audit/<date>/consolidate_actions.md` | 全部 5 动作 | 审计 | — |
|
||||
| **D1 / D10 健康** | `audit/<date>/health_*.md` | inline check + weekly | 人 / agent | — |
|
||||
|
||||
**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。
|
||||
|
||||
---
|
||||
|
||||
## 12. 与 dream 模型的引用关系
|
||||
|
||||
本文档松绑了部分 F-invariants(§7),但仍在 dream 定义的底层模型上工作:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法 | `auto_dream_design.md` §3 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 |
|
||||
| F-invariants 原始定义 | `auto_dream_design.md` §4.3(本文档 §7 修订)|
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §2 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §3 |
|
||||
| provenance 载体形态 | `auto_dream_design.md` §4.2 |
|
||||
| dream 写入路径 | `auto_dream_design.md` §4.2 |
|
||||
|
||||
---
|
||||
|
||||
## 13. 下一步(M0 → M1.1 → M1.2 → M1.3 → M2)
|
||||
|
||||
实现进入 `reme4/steps/consolidate/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md`(顶层)/ `auto_recall_design.md` 共同作为契约依据。
|
||||
|
||||
### M0:基础设施 + 完全可逆动作
|
||||
|
||||
- ✅ split inline 触发 + LLM 离散度判 + E-2 守恒(基础部分)
|
||||
- ⏳ **community detection weekly batch**(Leiden via `igraph`)+ `meta/communities.json` + `meta/community_changes.json`
|
||||
- ⏳ **archive daily batch** + recency_score + access_log 收集
|
||||
- ⏳ CAS 写入框架 + version_stamp + EEXIST race + 重做上限 + audit
|
||||
- ⏳ D1 / D10 写时 inline 检测 + weekly 巡检
|
||||
|
||||
### M1.1:abstract(P1,additive 最低风险)
|
||||
|
||||
- ⏳ abstract 候选挖掘(community 大小 + 稳定度 + 活跃度 + 多样性 + 无 hub 五重判据)
|
||||
- ⏳ abstract LLM prompt(三段输出 + 长度限制 1500 token)
|
||||
- ⏳ grounding 守恒校验(出边 ⊇ 关键支柱 + 出边 ⊇ source 60%)
|
||||
- ⏳ "已有 hub" 结构化判据(outbound 覆盖度 ≥ 60%)
|
||||
- ⏳ **关键验证点**:实测 split parent 是否被识别为 hub
|
||||
|
||||
### M1.2:merge(P2,lossy 高门槛)
|
||||
|
||||
- ⏳ 候选挖掘(community 内 description 相似 + body 重合 + cooldown 检查)
|
||||
- ⏳ 多数票框架(N=3 LLM,2/3 通过)
|
||||
- ⏳ merge prompt(B 方案:"只合并不精化")
|
||||
- ⏳ inbound retarget 机械算子(扫所有 `[[loser.md]]` → `[[winner.md]]`,alias / predicate 保留)
|
||||
- ⏳ 事务式多文件 CAS 写入
|
||||
- ⏳ 灰色地带报告(`audit/<date>/dups_uncertain.md`)
|
||||
- ⏳ cooldown 框架(`meta/cooldowns.json` + 各动作互锁)
|
||||
|
||||
### M1.3:reinforce(P3,价值最低,可缓做)
|
||||
|
||||
- ⏳ 候选挖掘(三层过滤:mention + embedding + 同 community)
|
||||
- ⏳ 单票决策(门槛 0.85)
|
||||
- ⏳ additive wikilink 写入(alias 保留原文)
|
||||
|
||||
### M2+:演进
|
||||
|
||||
- ⏳ 多层级 community(L2 super-community)+ L2 abstract
|
||||
- ⏳ delete(永久删除 vault 文件)—— 视 dogfooding 效果决定是否开启
|
||||
- ⏳ predicate upgrade(typed link reinforce —— 当前 reinforce 只 additive 加无谓词)
|
||||
- ⏳ PageRank 替代 simple inbound count(若 retrieve 质量瓶颈在中心性)
|
||||
- ⏳ 跨进程并发(fs lock 支持多 reme 实例同 vault)
|
||||
- ⏳ Leiden 边权重(按 predicate 类型加权)
|
||||
|
|
@ -5,8 +5,8 @@
|
|||
> 配套阅读:
|
||||
> - `structure.md` §1.2(数据视角)/ §2(三层存储)/ §3.5(digest 动作)
|
||||
> - `auto_memory_design.md`:daily 实时事件 = dream 的入流之一
|
||||
> - `auto_maintain_design.md`:M split / D 检测 / CAS 写入协议(dream 模型的运行时实现)
|
||||
> - `auto_link_design.md`:dream 写完后的后置增强(背景实体识别 + wikilink 写回)
|
||||
> - `auto_consolidate_design.md`:M split / D 检测 / CAS 写入协议(dream 模型的运行时实现)
|
||||
> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想 —— dream 是其 Stage 1(写入阶段)的实现
|
||||
>
|
||||
> **核心**:digest = **浅桶(shallow bucket)+ flat .md** + **一张图(节点 + 边)**;dream 定义模型与主流程(create_or_update),maintain 负责 split / 写入运行时。
|
||||
>
|
||||
|
|
@ -117,8 +117,8 @@ dream 设计回答四个问题:**桶**怎么布局 / **节点**长什么样 / **
|
|||
|
||||
| op | 谁 | 何时 | 改什么 |
|
||||
|---|---|---|---|
|
||||
| **dream**(create_or_update) | dreamer(本文档 §4.2) | 入流(新材料进入) | 创建新节点 / update 已有节点 body(语义守恒重写) |
|
||||
| **M split** | maintainer(`auto_maintain_design.md` §1) | 节点过载(token / 主题离散度超阈值) | 把 parent body 拆成 parent overview + N children;parent 文件原地 |
|
||||
| **dream**(create_or_update) | dreamer(本文档 §4.2) | 入流(新材料进入) | 创建新节点 / update 已有节点 body(语义守恒重写;UPDATE 内分 **CORROBORATE / REFINE / CORRECT** 三种 flavor,详 §4.2.3) |
|
||||
| **M split** | maintainer(`auto_consolidate_design.md` §1) | 节点过载(token / 主题离散度超阈值) | 把 parent body 拆成 parent overview + N children;parent 文件原地 |
|
||||
|
||||
> **关键观察**:"主题概览节点"不是一种 kind,也不是 maintainer 主动涌现的产物 —— 它是 split 的副产品(parent 节点天然成为该 cluster 的 overview,中心性自然高)。
|
||||
|
||||
|
|
@ -131,51 +131,77 @@ dream 设计回答四个问题:**桶**怎么布局 / **节点**长什么样 / **
|
|||
|
||||
**dream = dreamer 入流唯一改 body 的操作,且只改 subject node。**
|
||||
|
||||
#### 4.2.0 digest 是抽象记忆层
|
||||
|
||||
Digest 是 agent 长期记忆的**抽象层** —— 类比前额叶对认知的聚合。原始细节(数字、流程文本、谁说了什么)留在材料(daily / resource),digest 只承载细节淡忘后仍想调取的那一层:原则、模式、可作为先例的决策、认知要点。这一立场决定了 dream 流程的形态:**Phase 1 识别抽象,Phase 2 把抽象登记到 digest 节点**。
|
||||
|
||||
#### 4.2.1 两阶段流程
|
||||
|
||||
```
|
||||
material 进入(daily / resource 选定 scope)
|
||||
│
|
||||
▼
|
||||
LLM 抽取原子单元 → N 个候选
|
||||
Phase 1 — extract (轻量)
|
||||
LLM 读材料 → 识别其中教导的"抽象"(原则 / 模式 / 先例)
|
||||
→ 发出 ExtractedUnits 结构化输出 = K 个 sub-unit
|
||||
(每个: {name, summary},summary 标注证据在材料的哪段)
|
||||
说明:多个支撑事实说明同一抽象 → 合并为同一 sub-unit
|
||||
(倾向少而精);Phase 1 是 gate ——
|
||||
无新抽象时发空列表,Phase 2 跳过整轮
|
||||
│
|
||||
▼ 对每个候选:
|
||||
SearchStep 召回相似候选节点
|
||||
(reme4/steps/index/search.py;vector + keyword 并发 → RRF 融合
|
||||
→ expand_links 邻接展开;scope `digest/`;默认 limit 5~10)
|
||||
▼ (Python 外循环,K 次)
|
||||
Phase 2 — integrate (per sub-unit,每次独立 ReAct 会话)
|
||||
│ sub-unit ↔ digest 节点 1:1;Phase 2 必写,无 SKIP 出口
|
||||
│
|
||||
├─ RECALL: search(关键词 + 向量 + RRF) + traverse(对 top hit
|
||||
│ 做图扩展,跨 bucket) → 候选路径集
|
||||
│
|
||||
├─ HIT: frontmatter_read 廉价 triage → read 完整 body
|
||||
│ 确认候选是否承载同一抽象 → hit 集合
|
||||
│
|
||||
├─ 决策:
|
||||
│ ├─ hit 空 → CREATE 路径 (挑 bucket,写新节点)
|
||||
│ └─ hit 非空 → UPDATE 路径 (CORROBORATE / REFINE / CORRECT)
|
||||
│
|
||||
▼
|
||||
LLM 终判:候选池里有"同概念节点"吗?
|
||||
├─ 有 → update 路径
|
||||
│ (a) 把新内容融入已有 body(语义守恒重写)
|
||||
│ (b) 加 provenance 反指
|
||||
│ (c) 必要时加 / 改 wikilink
|
||||
│
|
||||
└─ 无 → create 路径
|
||||
(a) 挑 bucket(固定集合;无合适专属桶 → `unknown`)
|
||||
(b) 写文件名(同 bucket 唯一,fs 层断言)
|
||||
(c) 写 body + provenance + 横向 link
|
||||
写入(digest_write 创建 / digest_edit 改正文,E-1 强守恒,§4.4)
|
||||
│
|
||||
▼
|
||||
写入前 outbound diff 守恒校验(update 走 E-1;create 无 old outbound)
|
||||
│
|
||||
▼
|
||||
CAS 写入(`auto_maintain_design.md` §5)
|
||||
│
|
||||
▼
|
||||
写完 inline 触发 D3 检测(`auto_maintain_design.md` §4)
|
||||
agent 上报 IntegrateOutcome {action, target_path}
|
||||
```
|
||||
|
||||
**关键边界**:
|
||||
- **dream update 必须语义守恒** —— LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁);**写入前机械校验出边强守恒**(E-1,详 §4.4)
|
||||
**两阶段 trade-off**:Phase 2 把完整材料发 LLM K 次(一次一 sub-unit),不做 summary loss;代价是 K 倍 prompt token。换来的是 Phase 1 只做"识别抽象"这一件事(粒度集中在一个 prompt),Phase 2 每次会话上下文干净、聚焦单一抽象的写决策。
|
||||
|
||||
#### 4.2.2 召回二段
|
||||
|
||||
**RECALL = search + traverse**:search 给关键词 + 向量 RRF 命中;只要 search 在 `digest/` 下返回任何 hit,就对 top hit 跑 `traverse depth=2 direction=both`。理由是 search 关键词导向,会漏掉用不同术语归档的语义相邻抽象,那些常常一跳之外。search 在 `digest/` 下完全无命中 → 无 traverse 起点 → 候选集为空 → 直接 CREATE。
|
||||
|
||||
**HIT = frontmatter_read + read**:渐进披露 —— 先 `frontmatter_read` 读 `name + description` 廉价 triage 淘汰明显无关候选,剩下的再 `read` 整 body。**不可仅凭 chunk 片段或 frontmatter 决定 UPDATE**,body 才是判定依据。
|
||||
|
||||
#### 4.2.3 UPDATE 三种 flavor
|
||||
|
||||
| flavor | 何时 | body 怎么动 |
|
||||
|---|---|---|
|
||||
| **CORROBORATE**(最常见)| 已有节点已覆盖此抽象,材料是又一个实例 | body 实质不变 —— 追加 `derived_from::` 溯源,可选强化措辞("似乎"→"确实") |
|
||||
| **REFINE**(常见)| 已有节点覆盖了核心,但材料揭示新的范围 / 边界 / 维度 | 改相关片段使更精确,加新维度,加 `derived_from::`。正文在**精度**上长,不在**细节**上膨胀 |
|
||||
| **CORRECT**(少见)| 材料与已有抽象矛盾 / 表明它被夸大 | 收紧到新旧证据都支持的窄形式,或内联标注 `> note: contradicted by [[...]]` 不仲裁。仍加溯源 |
|
||||
|
||||
三种都受 §4.4 E-1 强守恒约束(出边集合不能缩)。
|
||||
|
||||
#### 4.2.4 关键边界
|
||||
|
||||
- **Phase 1 是 gate** —— "不值得记忆"在 Phase 1 过滤(空列表);Phase 2 必然写,sub-unit 与 digest 节点 1:1
|
||||
- **dream update 必须语义守恒** —— LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁);写入前机械校验出边强守恒(E-1,详 §4.4)
|
||||
- **dream 不改其它节点正文**(F-2) —— 只动 subject
|
||||
- **dreamer 不做事件级伞节点** —— 材料本身(daily / resource 文件)就是 fan-out 点,每个 sub-unit 的 `derived_from::` 让材料天然聚合到所有派生节点
|
||||
- **0 出边节点合法**(没识别到合适邻居),后续 dream 进入时其它节点可以反向链回来 —— 不强求 LLM 一次性给全
|
||||
- **dream 漏判去重**(同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显重复,由 auto-link L4 离线 audit 工具产报告(`auto_link_design.md` §1.3)
|
||||
- **dream 漏判去重**(同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显重复,由 auto-consolidate 的 dups 检测周期 batch 产报告(`auto_consolidate_design.md` §3)
|
||||
- **召回不做 bucket 粗筛** —— LLM 拥有完整跨桶视野,可识别"概念错分到 unknown"或"跨桶同概念"
|
||||
|
||||
**provenance 写出**:
|
||||
- 行文中自然带:"... 该模式最早出现在 [[daily/2026/05/15.md]] 的实践中"
|
||||
- 可选 predicate:`derived_from:: [[daily/2026/05/15.md]]`,不强制
|
||||
- prompt 必须要求"出处用 `[[...]]` 形式表达"(纯散文会被守恒校验视为丢边)
|
||||
- **首版可先用 append 起步**(出边集合天然 ⊇,守恒校验自动通过);成熟后切到重写
|
||||
- **强制 typed predicate `derived_from::`** —— body 必须织入至少一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`,纯散文形式不被守恒校验视作边,下次 update 时会消失
|
||||
- 不走"首版 append 起步"的过渡路径 —— digest_edit 自一开始就跑 E-1 强守恒,LLM 直接做语义守恒重写
|
||||
|
||||
### 4.3 F-invariants(演化的硬约束)
|
||||
|
||||
|
|
@ -228,21 +254,18 @@ write_subject_body(subject, new_body):
|
|||
|---|---|
|
||||
| ← **auto-memory**(daily) | dream 读 daily 作为入流;daily 写完即对 dream 可见 |
|
||||
| ← **resource** | dream 读 resource 作为入流(只读,不写) |
|
||||
| → **auto-maintain** | dream 写完触发 D3 inline 检测;D3 过载 → enqueue split job(maintain 异步消费);写入并发由 CAS 协议(`auto_maintain_design.md` §5)保护 |
|
||||
| → **auto-link** | dream 写完 enqueue auto-link L1(背景实体识别 + wikilink 写回);走同一 CAS 队列(`auto_link_design.md` §1.2) |
|
||||
|
||||
**关键边界**:dream 不写 daily / resource(I-2 / I-3);只写 digest 节点 body(自身 subject)。
|
||||
**关键边界**:dream 不写 daily / resource(I-2 / I-3);只写 digest 节点 body(自身 subject)。dream 不感知下游 —— split / 链接增强 / 索引刷新 / rename 等由 `auto_consolidate_design.md` / `auto_cognition_design.md` / `update_store_index_loop` 各自负责。
|
||||
|
||||
---
|
||||
|
||||
## 6. 下一步
|
||||
|
||||
本文档覆盖 dream 模型(桶 / 节点 / 边 / 演化)。组织端实现清单(M split / D 检测 / CAS 框架)见 `auto_maintain_design.md` §10。
|
||||
本文档覆盖 dream 模型(桶 / 节点 / 边 / 演化)。组织端实现清单(M split / D 检测 / CAS 框架)见 `auto_consolidate_design.md` §10。
|
||||
|
||||
1. **dream step 实现** —— scope → 抽取 → SearchStep 召回 → LLM 终判 → CAS 写入 + E-1 守恒校验
|
||||
2. **rename 路径封装** —— `wikilink_handler.retarget_links(old, new)` 已就绪;封装为单步 step,无 alias 表,无透明展开
|
||||
3. **bucket 集合配置** —— `vault.yaml` schema / 默认桶模板 / `unknown` 兜底 / `_buckets.md` 视图生成
|
||||
4. **边守恒校验工具** —— `extract_links` 已就绪;新增 outbound diff 比较器 + LLM 重试编排 + ConservationViolation audit 事件
|
||||
5. **provenance prompt 规范** —— dream 引导 LLM 写 `[[daily/...]]` / `[[resource/...]]`
|
||||
- ✅ **dream step 实现** —— Phase 1 extract + Phase 2 integrate(per sub-unit)+ E-1 守恒校验(`reme4/steps/evolve/dream/`)
|
||||
- ✅ **边守恒校验工具** —— `digest_edit` 的 outbound diff 比较器 + REJECT_CONSERVATION 重试 + 违规上报
|
||||
- ✅ **provenance prompt 规范** —— `derived_from:: [[daily/...]]` / `[[resource/...]]` 强制
|
||||
- ⏳ **bucket 集合配置外置** —— 当前 hardcoded 在 `digest_write.py` 的 `DEFAULT_BUCKETS`;目标 `vault.yaml` schema + `_buckets.md` 视图生成
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_memory_design.md` / `auto_maintain_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
实现进入 `reme4/steps/evolve/` 时,本文档与 `auto_memory_design.md` / `auto_consolidate_design.md` / `auto_cognition_design.md` 共同作为契约依据。
|
||||
|
|
|
|||
|
|
@ -1,209 +0,0 @@
|
|||
# auto-link 设计(背景实体识别 + wikilink 写回)
|
||||
|
||||
> 本文档记录 reme4 中 **auto-link** 的设计讨论 —— 在已写入节点之间发现隐含关系,把这些关系作为 `[[...]]` wikilink **写回 body**,形成可见、可编辑的图结构增强。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §1.2(三层数据视角)/ §4(retrieve 三种问法)
|
||||
> - `auto_memory_design.md`:auto-link 可反向扫 daily event,补实体 wikilink(daily → digest)
|
||||
> - `auto_dream_design.md`:wikilink 模型(§3 边语法 / §4 演化 / §4.4 边守恒 E-1 / E-2 / E-3);auto-link 借这套基础设施
|
||||
> - `auto_maintain_design.md`:CAS 写入协议(§5);auto-link L1 写回与 dream / maintain split 三方共用同一套 CAS
|
||||
>
|
||||
> **三层对应**:reme 服务整体三层 —— auto-memory / auto-dream / **auto-link(本文档)**。auto-link 是图关系的**后置增强** —— 在已落地的 vault 上做实体识别 + wikilink 写回,补足 content link(写记忆时由 LLM 直接产生的 `[[...]]`)在长 tail 隐含关系上的盲区。
|
||||
>
|
||||
> **核心立场**:auto-link **写回 body**,不只是产报告。生成的 wikilink 是**可见、可编辑**的(写在 Markdown 文件里),agent / 人可后续 curate。auto-link 不引入新材料,纯 additive 插入 wikilink,天然满足 E-1 守恒;复用 dream 的 CAS 写入协议,不引入新基础设施。
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
content link(`auto_dream_design.md` dream / split 写入时由 LLM inline 产生的 `[[...]]`)解决了"写记忆时显式的关系"。但有一类关系不会在 inline 写入时自然涌现,需要后台扫描已写入的 vault 才能识别:
|
||||
|
||||
1. **历史 body 的实体未链接** —— dream update 时 LLM 关注新材料融入,可能忽略已有 body 中某个未链接的实体(例如 body 提到 "JWT" 但没写 `[[digest/auth/jwt-overview.md]]`)
|
||||
2. **跨节点 / 跨桶的隐含关联** —— 节点 A 提到 "rate limit",但 `digest/api/rate-limit.md` 是后来才被 split 创建 → A 写入时没机会建立这条边
|
||||
3. **同主题未连 / 同概念重复** —— dream 漏判去重把同概念建成两个节点;或两个主题相关但 0 链接的节点彼此不知晓
|
||||
|
||||
auto-link 承担这部分:**后台扫描已写入节点 → 实体识别 / 候选挖掘 → wikilink 写回 body**。
|
||||
|
||||
---
|
||||
|
||||
## 1. 已对齐决策
|
||||
|
||||
### 1.1 与 content link 的边界
|
||||
|
||||
| 维度 | content link(在 dream) | auto-link(本文档) |
|
||||
|---|---|---|
|
||||
| 何时产生 | 写记忆 inline:dream update / M split prompt | 后台扫描:离线 / 周期 / 触发后异步 |
|
||||
| 由谁产生 | LLM 在 dream 写入流中顺手写出 | LLM 在 auto-link 扫描流中识别后写出 |
|
||||
| 输入 | 新材料 + 召回候选节点 | 已写入 body + 全 vault 索引 |
|
||||
| 改 body | 是(重写整段 body) | 是(纯 additive 插入 wikilink,不改文字) |
|
||||
| 守恒 | E-1 强守恒(out ⊇ old) | E-1 天然满足(纯增) |
|
||||
| 用途 | 写入即关系明示 | 弥补 inline 漏判,挖掘长 tail 关系 |
|
||||
|
||||
### 1.2 写回模型:纯 additive,复用 dream CAS
|
||||
|
||||
auto-link 写回是**纯 additive** 操作 —— 在已有 body 文字中找到实体 mention,替换为 wikilink 形态:
|
||||
|
||||
```
|
||||
Before: "JWT 轮换的核心是密钥派生 ..."
|
||||
After: "[[digest/auth/jwt-rotation.md|JWT 轮换]]的核心是[[digest/auth/jwt-key-derivation.md|密钥派生]] ..."
|
||||
```
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **alias 必须保留原文** | `[[path.md\|<原文>]]` 形态;原文一字不改 —— 守住"不改写其它节点正文" (`auto_dream_design.md` §4.3 F-2) 的精神 |
|
||||
| **predicate 默认为空** | auto-link 默认产生无谓词 wikilink;升 typed link 走 L3(详 §1.3) |
|
||||
| **不引入 anchor** | 与 dream 一致(`auto_dream_design.md` §3);target 永远是节点路径 |
|
||||
| **CAS 写入** | 完全复用 `auto_maintain_design.md` §5 的 read-stamp + CAS-write 协议(冲突重做 ≤ 3 次) |
|
||||
| **E-1 守恒** | 纯 additive:`new outbound = old outbound ∪ {new wikilinks}`;`new ⊇ old` 天然满足,守恒校验默认通过 |
|
||||
| **rollback** | 若 auto-link 误插入(例如 entity mention 是同名歧义),走标准 edit 或 retarget 撤销;auto-link 不维护"我插过哪些"audit log(留给 SDK 决定) |
|
||||
|
||||
**为什么是 additive 而不是重写**:
|
||||
- additive = 0 文字风险(原文不变,只在原 mention 周围加 `[[ | ]]` 包装)
|
||||
- 重写 = 触发完整 E-1 守恒校验 + LLM 重写整段语义守恒 prompt + 多次 LLM 调用 = 跟 dream update 重复
|
||||
- additive 失败可见:产生坏 wikilink 时,人/agent 直接编辑 body 修就行
|
||||
|
||||
### 1.3 候选挖掘类型(L1-L4)
|
||||
|
||||
| # | 类型 | 描述 | 写回形态 |
|
||||
|---|---|---|---|
|
||||
| **L1** | **实体识别**(主路径) | 扫 body,识别已是 digest 节点的实体名(模糊匹配 + 语义召回);未被 wikilink 化的 mention → 加 `[[path.md\|<mention>]]` | additive wikilink 插入 |
|
||||
| **L2** | **同主题未连**(旧 D7) | 两个 digest 节点谈相关主题但 0 wikilink → 候选 add link;LLM 判后在 body 末尾追加一句引用 | additive(在合适位置 / 节末追加 `参见 [[other.md\|other]]`)|
|
||||
| **L3** | **隐含 predicate 推导** | 已有 `[[A]]` 但 LLM 可推断关系类型(`is_a` / `causes` / `extends` / ...)→ 升级为 typed link | 改 `[[A]]` → `is_a:: [[A]]`(predicate 升降级走显式 audit,详 §2.1)|
|
||||
| **L4** | **重复语义检测**(旧 D8) | 两个节点描述同一概念但被独立 create(dream 漏判去重)→ 候选 merge | **不写回**;产报告 + 提示人/agent 触发 dream update 路径手工合并 |
|
||||
|
||||
**L1 是主路径** —— 它是 auto-link 最核心、最频繁、最高 ROI 的操作:每个 digest 节点写完后,后台扫一遍 body,找未链接的已知实体,additive 加 wikilink。
|
||||
|
||||
**L2-L3 是辅助** —— 周期扫,产候选,LLM 终判,写回部分(L2 节末追加 / L3 升 predicate)。
|
||||
|
||||
**L4 不写回** —— 节点合并是结构改动,影响 E-1 守恒边界 + inbound 链路 + provenance 链路,不适合自动写;auto-link 只产报告,人/agent 决定走 dream update 路径解决。
|
||||
|
||||
### 1.4 触发节奏
|
||||
|
||||
| 模式 | 何时 | 适用 |
|
||||
|---|---|---|
|
||||
| **inline post-write**(默认) | 每次 dream update / M split 写完 body → enqueue auto-link L1 job(异步,FIFO,CAS 保护)| L1 实体识别;反应即时,与 D3 写后检测同节奏 |
|
||||
| **周期 batch**(可选)| cron(daily / weekly)扫全 vault | L2 / L3 候选挖掘;成本可控 |
|
||||
| **手动触发** | SDK / 人显式调用 | 全量重扫 / 修复 |
|
||||
|
||||
**L1 inline 的必要性**:新 split 出的 child 节点立即被既有 body 引用(用 wikilink 而非纯 mention)的关键 = 写入即扫描;不 inline 会让"刚创建的 child 节点"在很长时间内只有 split parent 一个 inbound,中心性失真。
|
||||
|
||||
**已排除**:
|
||||
- inline 时同步 auto-link(阻塞 dream return)—— 时延不可接受;auto-link 始终异步
|
||||
- 所有 L\* 都 inline —— L2-L3 候选挖掘 RTL 跨节点,成本高,只适合 batch
|
||||
- 全 cron 唯一触发 —— L1 滞后过久,新节点孤岛
|
||||
|
||||
### 1.5 中心性算法(retrieve 加权依赖)
|
||||
|
||||
retrieve 时节点权重 = base × intent 调节 × **中心性增益**(详 `auto_dream_design.md` §5)。中心性需要 auto-link 这一层提供 —— content link 给底子,auto-link 补 long tail,二者合起来才是完整的图。
|
||||
|
||||
| 选项 | 优点 | 缺点 |
|
||||
|---|---|---|
|
||||
| **简单入度** | 实现最简;split parent 入度天然高;auto-link L1 加边后入度即时反映 | 不区分"权威节点"vs"被随手提的节点";高入度 ≠ 高权威 |
|
||||
| **PageRank** | 经典;权威性传递 | 实现复杂 + 增量更新成本(每次写边重算成本高,需 incremental algorithm)|
|
||||
| **eigenvector centrality** | 与 PageRank 相近 | 同上 |
|
||||
|
||||
**首版决策**:**简单入度**(file_graph 已有 inbound 链表,O(1) 查);auto-link L1 加边后入度立刻更新,split parent 自然涌现高入度。dogfooding 后视 retrieve 质量演进。
|
||||
|
||||
中心性是 retrieve 时**查询时计算**,不预存:
|
||||
- file_graph 已建反向索引(inbound),计算 `len(inbound(node))` 是 O(1)
|
||||
- 不预存避免"加边后中心性陈旧"问题
|
||||
- PageRank 演进时可加增量计算 + 周期 refresh
|
||||
|
||||
---
|
||||
|
||||
## 2. 待对齐边界点
|
||||
|
||||
### 2.1 L3 predicate 升降级的 audit
|
||||
|
||||
L3 把 `[[A]]` 升级为 `is_a:: [[A]]` 时,**改了 edge identity** —— `(target, None)` 变成 `(target, "is_a")`,在 E-1 守恒视角下 = 删一条边 + 加一条边:
|
||||
|
||||
```
|
||||
old outbound: {(A, None)}
|
||||
new outbound: {(A, "is_a")}
|
||||
diff: missing = {(A, None)}; added = {(A, "is_a")}
|
||||
```
|
||||
|
||||
不打 audit 走默认会被守恒校验拦下(`missing != ∅` → 重试 / 拒写)。
|
||||
|
||||
**决策方向**:
|
||||
- L3 写入必须打 audit flag(消费层意图:升级 predicate,允许 drop + add 同时发生)
|
||||
- audit flag 由 reme4 step 暴露(`maintainer_step(action="predicate_upgrade", from=..., to=...)`),不放在普通 write 路径
|
||||
- 普通 dream / auto-link L1 写入永远不带 audit flag,守恒校验照常严格
|
||||
|
||||
详细 audit flag 接口形态留到 SDK 阶段。
|
||||
|
||||
### 2.2 多歧义实体识别
|
||||
|
||||
L1 扫 body 找 "JWT" 这个 mention,vault 中有 `digest/auth/jwt-overview.md` 和 `digest/payment/jwt-payment-flow.md` 两个 candidate:
|
||||
|
||||
候选方案:
|
||||
- LLM 上下文判 —— 把 body 周围段落给 LLM,选最相关 target
|
||||
- 跳过模糊 case —— L1 只处理 unambiguous mention,歧义 case 留人/agent
|
||||
- 全部链 —— `[[overview]][[payment-flow]]`,后续人 curate
|
||||
|
||||
**首版**:LLM 上下文判(每个候选 candidate 提供 description / 周围若干节点 summary,LLM 选择 top-1 或 drop);成本可接受(扫描已是离线 batch)。
|
||||
|
||||
### 2.3 auto-link 写回与 dream / split 的并发
|
||||
|
||||
auto-link 写 body 走 §1.2 CAS,但有特殊情况:
|
||||
- 同节点同时被 dream update 与 auto-link L1 写入 → CAS 协议自动序列化 (`auto_maintain_design.md` §5):后到者重做
|
||||
- auto-link L1 写完后立刻被 dream update 覆盖(dream 重写 body) → 看 dream prompt 是否守住 auto-link 加的 wikilink(E-1 强守恒 → 守住)
|
||||
- auto-link L1 与 D3 派发的 split job 同节点并发 → split 先到 / 后到都不影响最终拓扑(split 把 body 拆成 parent + children,auto-link 加的 wikilink 跟着对应内容段自然分配到 parent / child)
|
||||
|
||||
**结论**:CAS + E-1 + E-2 守恒已覆盖所有并发场景,auto-link 不需要新协调机制。
|
||||
|
||||
### 2.4 跨 vault / 跨进程
|
||||
|
||||
M0 单 reme 实例 + 单 vault,auto-link 走内进程 enqueue;多实例 / 跨进程留 M1+(同 `auto_maintain_design.md` §5)。
|
||||
|
||||
### 2.5 实体识别 vs 现成 NER 库
|
||||
|
||||
L1 实体识别可选:
|
||||
- LLM 直接扫(贵但灵活,与 digest 节点同构)
|
||||
- 现成 NER 库(spaCy 等)预筛 + LLM 终判(快但 entity 类型与 digest 节点形态可能不匹配)
|
||||
- 纯字符串匹配(已知节点名字 + 简单变体)+ LLM 终判 ambiguity
|
||||
|
||||
**倾向**:从纯字符串匹配 + LLM 终判 ambiguity 起步(实现最简,效果可能已经够好);视 dogfooding 决定是否引入 NER 库。
|
||||
|
||||
---
|
||||
|
||||
## 3. 与其它层的协作
|
||||
|
||||
| 上下游 | 关系 |
|
||||
|---|---|
|
||||
| ← **auto-dream** | dream 写完一个节点 → 通过 inline post-write enqueue auto-link L1(§1.4);auto-link 用 dream 的 CAS 协议 |
|
||||
| ← **auto-memory** | auto-link 可反向扫 daily event,把实体识别成 `[[digest/...]]`(daily → digest);auto-memory 写入端不主动调 auto-link,触发同 dream 路径 |
|
||||
| → **digest body** | 主要写入对象 —— L1 additive 加 wikilink / L2 节末追加引用 / L3 升 predicate(走 audit) |
|
||||
| → **daily body** | auto-link 扫 daily event 时同样可加 `[[digest/...]]`(I-2 daily 单作者需协调:auto-link 应在 event 关闭后才动该 event,不与 active event 并发改;实现细节留 step 层处理) |
|
||||
| → **resource body** | I-3 immutable;auto-link **不写 resource**(reading-only) |
|
||||
| → **L4 候选 report** | L4 重复语义检测产报告,落 `audit/<date>/auto_link_l4.md`(具体路径 / 形态留 step 层) |
|
||||
|
||||
---
|
||||
|
||||
## 4. 与 auto-dream 模型的引用关系
|
||||
|
||||
本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §3 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 |
|
||||
| F-invariants(F-1..F-11)| `auto_dream_design.md` §4.3 |
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §2 |
|
||||
| CAS 写入协议 | `auto_maintain_design.md` §5 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §3 |
|
||||
| SearchStep 召回 | `auto_dream_design.md` §4.2 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 下一步
|
||||
|
||||
1. **L1 实体识别 step 实现** —— 字符串匹配 + 语义召回 + LLM ambiguity 终判 + additive wikilink 写回(§1.2 / §1.3)
|
||||
2. **inline post-write trigger 接入** —— dream update / M split CAS 写入成功后 enqueue auto-link L1 job(§1.4)
|
||||
3. **L2 / L3 周期 batch 框架** —— cron(daily / weekly)+ 候选挖掘 prompt + 写回路径(§1.3)
|
||||
4. **L3 audit flag 接口** —— `maintainer_step` 提供 `predicate_upgrade` 操作,带 audit context 走特殊守恒规则(§2.1)
|
||||
5. **中心性 retrieve 增益** —— file_graph inbound count → retrieve 加权乘子(§1.5)
|
||||
6. **L4 报告框架** —— 重复语义检测产报告,提供 SDK / 人介入入口(§1.3 / §3)
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` 共同作为契约依据。
|
||||
|
|
@ -1,190 +0,0 @@
|
|||
# auto-maintain 设计(digest 组织端:M split / 检测 / 写入并发)
|
||||
|
||||
> 本文档记录 reme4 中 **auto-maintain** 的设计讨论 —— digest 层的组织 / 重组 / 写入运行时,含 M split、D 检测信号、写后触发模型、CAS 写入协议。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §3.6(maintain 动作语义)/ §7.3(maintainer 模块)
|
||||
> - `auto_dream_design.md`:节点 + 边模型(§1-§4)/ F-invariants(§4.3)/ 边守恒 E-1/E-2/E-3(§4.4)/ dream 操作(§4.2)—— maintain 复用这套底层模型
|
||||
> - `auto_link_design.md`:auto-link 写回也走本文档的 CAS 协议(§5)
|
||||
> - `auto_memory_design.md`:auto-memory 不直接复用 maintain,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
>
|
||||
> **三层框架的位置**:报告 §5 三层为 auto-memory / auto-dream / auto-link。maintain 严格按 `structure.md` §3.5-3.6 的 L4 action 分类是独立 action(`maintain: digest → digest`),不属 `digest` action(`digest: resource + daily → digest`)。本文档作为四方分工的**第四份**,专门覆盖 dream 写完之后 digest 的组织 / 重组 / 写入运行时。
|
||||
>
|
||||
> **核心立场**:
|
||||
> - **maintain 与 dream 同 pace**(idle background)、同模型(节点 + 边 / 守恒规则),但**语义边界不同**:dream 是 compose(资料 → digest),maintain 是 reorganize(digest → digest)
|
||||
> - **maintain 只做 split**,不做 merge / dissolve / re-edge / unify;过载就拆,其它跨节点重组留给消费层 / 人工
|
||||
> - **CAS 写入协议是基础设施**,被 dream / maintain split / auto-link L1 共用,统一编排在本文档(§5)
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
dream 模型(`auto_dream_design.md` §4)规定 digest 的演化只做两件事:dream create_or_update(入流型,新材料融入)+ M split(后台,过载就拆)。dream 文档负责 dream 与节点 / 边模型;**本文档负责 M split 与运行时机制**(D 检测 / 触发模型 / 写入并发协议)。
|
||||
|
||||
| 输入 | 输出 |
|
||||
|---|---|
|
||||
| dream 写入后的 digest 状态 + 触发信号(D3 过载,inline) | parent overview 重写 + N 个新 children 文件;边守恒 E-2 通过 |
|
||||
|
||||
**设计目标**:
|
||||
1. **形状匹配** —— 让节点粒度持续与实际语义结构对齐(过载节点拆;不过载不动)
|
||||
2. **最小变更面** —— split 改 parent + 创建 N children,不动其它节点(F-2)
|
||||
3. **不引入新基础设施** —— 复用 dream 的节点 + 边模型 / 守恒规则;CAS 写协议自洽
|
||||
|
||||
**显式排除**:
|
||||
- ❌ merge / dissolve / re-edge / unify —— 跨节点重组不做(简化模型;同概念二次进入靠 dream update)
|
||||
- ❌ 改其它节点正文 —— split 只改 parent body(重写为 overview)+ 创建 children body
|
||||
- ❌ 重建 inbound —— split 时 inbound 一律不动(F-10)
|
||||
|
||||
---
|
||||
|
||||
## 1. M split(maintainer 唯一 op)
|
||||
|
||||
| # | 能力 | 服务 | 触发 | graph | file | body |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **M split** | 节点过载 → LLM 拆成 parent overview + N 个 children;parent 文件原地保留,children 是新文件;children 加 `[[parent]]` 反向链接;inbound 边不动 | 形状匹配(粒度对齐)+ 任意尺度(涌现层级) | D3 过载 | parent 0 拓扑改;新 children 节点 + 各自加 `[[parent]]` 出边 | 创建 N 个 children 文件;parent 文件原地 | parent body 重写为 overview;children 各自有新 body |
|
||||
|
||||
**关键边界**:
|
||||
- **M split 改两类 body**:parent body(重写为 overview)+ N 个新 children body;不改任何**其它**节点(`auto_dream_design.md` §4.3 F-2)
|
||||
- **inbound 不重定向** —— 外部对 parent 的 wikilink 全部保留指 parent;后续 dream 进入时若 LLM 觉得 child 粒度更合适,直接加新边到 child 即可(F-10)
|
||||
- **没有 dissolve 操作** —— children 长期空也不主动删;消费层 / 人工显式介入
|
||||
- **没有 merge / re-edge / unify** —— 跨节点重组不做;同概念二次进入靠 dream update;错桶节点不主动 move(若严重,人工介入)
|
||||
- **边守恒** —— split 写新 parent body + N children body 前,机械对比 outbound:`(parent_new ∪ ∪children_outbound) ⊇ parent_old`;失败 → LLM 重试或拒写(F-11 / E-2,详 `auto_dream_design.md` §4.4)
|
||||
|
||||
---
|
||||
|
||||
## 2. 检测信号 D1 / D3 / D10
|
||||
|
||||
| # | 信号 | 服务 | 服务能力 |
|
||||
|---|---|---|---|
|
||||
| **D1** | 断链(wikilink → 不存在的 path) | 任意尺度(可达性) | 告警 / 简单修复(就地删 wikilink 或保留 alias 文本) |
|
||||
| **D3** | 过载节点(token 阈值 → LLM 判离散度) | 形状匹配(粒度) | maintainer(M split) |
|
||||
| **D10** | provenance 断裂(digest 节点反指的 daily/resource 不可达) | 任意尺度(跨层不变量) | 严重告警(I 不变量违反) |
|
||||
|
||||
**触发模型**:**写后立即** —— dream / split 写完 body inline 检测;无后台 watcher / 无周期 tick / 无 dirty 队列(详 §4)。D1 / D10 是 wikilink 断链的子集,跟 file_graph 链路一起在写时检测。
|
||||
|
||||
> **简化模型砍掉的信号**:
|
||||
> - **D2 隔离 / D4 过疏 / D5 高入度 / D5b 低入度摘要 / D6 slug 冲突 / D7 相似未链 / D8 重复语义 / D9 邻居异质** —— 全部 DROPPED
|
||||
> - 旧 D5 高入度涌现 → 由 split 副产品(parent + children)等价覆盖;触发源换成节点过载(D3)
|
||||
> - 旧 D6 slug 冲突 → 路径即 ID 后,同 bucket 内文件名冲突由文件系统层断言(写入即拒),不需要独立信号(详 `auto_dream_design.md` §2)
|
||||
> - 旧 D7 / D8 → 简化模型不做 link / merge 提议;若 vault 累积明显的同概念重复,由 `auto_link_design.md` §1.3 L4 离线 audit 工具产报告
|
||||
> - 旧 D9 邻居异质 → 简化模型不做跨桶 move;桶选择只在 dream 桶决策一次性决定,后续不重排
|
||||
>
|
||||
> **D3 过载的判据**:token 阈值机械检查 + LLM 判离散度;**写后立即 inline**。阈值见 §3,触发模型见 §4。
|
||||
|
||||
---
|
||||
|
||||
## 3. 检测阈值校准
|
||||
|
||||
简化模型只剩 D3(过载)是核心阈值,其它都是 invariant 触发(无可调阈值)或 informational(无 maintenance 联动)。
|
||||
|
||||
| 信号 | 阈值类型 | 默认 | 备注 |
|
||||
|---|---|---|---|
|
||||
| **D3 过载** | token + 主题离散度 | token 2000 / 离散度由 LLM 写后 inline 判 | **唯一驱动 split 的阈值**(详 §4) |
|
||||
| **D1 断链** | 0 容忍 | 任意 1 条断链 → 告警 | 修复策略简单(就地删 wikilink) |
|
||||
| **D10 provenance 断裂** | 0 容忍 | 任意 1 条断裂 → 严重告警 | I 不变量 |
|
||||
|
||||
D3 阈值作为 `vault.yaml` 配置项(opinionated default,reme 核心提供机制不写死阈值),消费层可改;dogfooding 后调优。token 阈值起点 2000(对应"约 5 个独立子主题"的常见过载点),首版可调。
|
||||
|
||||
---
|
||||
|
||||
## 4. D3 触发模型(已收敛)
|
||||
|
||||
**决策**:**写后立即检测,无 watcher 抽象,无 batch 窗口** —— 每次 dream update / split 写 body 成功后,**inline** 在同一 job 内跑 D3:token 阈值 + LLM 离散度判定 → 必要时 enqueue split job(异步,走 §5 CAS 队列)。
|
||||
|
||||
```
|
||||
dream / split 写 body 成功(CAS 通过)
|
||||
└─ if len(body) > T:
|
||||
└─ LLM 判离散度
|
||||
└─ if is_overloaded:
|
||||
└─ enqueue split job (FIFO, CAS-protected)
|
||||
└─ return
|
||||
```
|
||||
|
||||
**协议**:
|
||||
- token 阈值默认 `2000`(§3 已定,`vault.yaml` 可配)
|
||||
- 离散度 prompt 输出 `{is_overloaded: bool, suggested_clusters: [...]}`(若 overloaded 直接供 split job 吃,不重判)
|
||||
- 启动无全扫(避免长启动);新写入立即检测覆盖增长路径;历史遗留过载随下次 update 自然检出
|
||||
- 无 dirty 标 / 无 dirty 集合 / 无后台 worker —— D3 是写路径的合成函数
|
||||
|
||||
**为什么 inline**:反应即时(不等下一次 ingest);实现最简(无批处理窗口 / dirty 状态 / 独立 worker);LLM 判定成本可接受(大多写入 < T 不触发,触发后 split 切小后续不再越界);不引入 watcher = 少一层部署/监控。
|
||||
|
||||
**已排除**:定时 cron tick(静止 vault 浪费扫描)/ ingest-after batch(引入 dirty 集合)/ 独立 L2 watcher worker(多余部署层)。
|
||||
|
||||
**演进路径(M1+)**:若 inline LLM 阻塞 dream 时延成问题 → D3 改为 fire-and-forget enqueue;若同节点重复触发 LLM 成本高 → 加节点级 body hash 缓存。
|
||||
|
||||
---
|
||||
|
||||
## 5. CAS 写入协议(共享基础设施)
|
||||
|
||||
**位置说明**:CAS 是 dream update(`auto_dream_design.md` §4.2)、M split(本文档 §1)、auto-link L1 写回(`auto_link_design.md` §1.2)**三方共用**的写入协议。归在本文档是因为 maintain 是 digest 的"组织 / 运行时"端,运行时机制(检测 / 触发 / 写入)集中在一处方便对照。
|
||||
|
||||
**决策**:**并行决策 + 乐观冲突重做(CAS)** —— 所有 dream / split / auto-link L1 决策并发跑,写入前用 body 版本戳(hash / mtime)做 CAS 比对;变了就丢弃 planned body 重做。无锁,无 ingest 级互斥。冲突率低 + E-1 / E-2 守恒校验顺手承担 race 兜底,无需新基础设施。
|
||||
|
||||
**协议(单个写入调用)**:
|
||||
1. **读 + 记戳**:读 subject body → `version_stamp = sha256(body) | mtime`
|
||||
2. **决策**:LLM 看候选池 → 决定 create / update / drop / split / additive-link;产 planned new_body
|
||||
3. **CAS 写入**:重读 body 比 version_stamp
|
||||
- **未变**:跑 E-1 / E-2 守恒校验 → 通过则 atomic write(write-temp + rename)→ done
|
||||
- **已变**:丢弃 planned new_body,带最新 body 重走 step 1
|
||||
4. **守恒校验失败**:走 `auto_dream_design.md` §4.4 既有重试路径(LLM 重试一次,二次失败拒写 + audit)
|
||||
5. **重做次数上限**:CAS-冲突重做最多 3 次;超出 → 跳过候选 + audit log(避免活锁)
|
||||
|
||||
**create 路径 race**:两个 dream 都决定 `create digest/auth/jwt-rotation.md` → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者拿 EEXIST → 重走 step 1(此时大概率改判 update)。
|
||||
|
||||
**适用范围**(全部走同一套 CAS):同 ingest 内 N 个候选并发 / 跨 ingest job 并发 / 后台 split 与前台 dream 命中同节点(split 同样走 CAS)/ auto-link 写回(`auto_link_design.md` §1.2)。
|
||||
|
||||
**不解决的**:高冲突 workload(同概念被反复 ingest)→ 重做上限触发后 audit;跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+)。
|
||||
|
||||
---
|
||||
|
||||
## 6. split 时的 provenance 处理(已收敛)
|
||||
|
||||
**坍缩到 E-2 合计守恒** —— provenance 是 body 内联 wikilink(`auto_dream_design.md` §4.2),split 时跟其它 body 边完全同形:LLM 把 parent body 拆成 parent overview + N children,provenance wikilink 跟着对应内容段自然分配;机械层 outbound 合计守恒校验保证 `(parent_new ∪ ∪children_outbound) ⊇ parent_old`,旧 provenance 不可能丢。无需专门的 provenance 分配逻辑或"全部复制到 child / parent 保留全量"等特殊策略 —— LLM 按"哪个 child 谈到了哪段上游就带走哪条 provenance"自然处理。
|
||||
|
||||
---
|
||||
|
||||
## 7. dream / split / auto-link L1 时序(已收敛)
|
||||
|
||||
时序由 §4 / §5 与 `auto_dream_design.md` §4.2 共同规定,这里给最小汇总:
|
||||
|
||||
- **dream 调用本身同步** —— material 进来就走 dream 决策(召回 + LLM 终判)+ CAS 写入(§5)
|
||||
- **D3 检测 inline** —— dream / split 写完 body 顺手跑 token 阈值 + LLM 判离散度(§4),无 tick / batch / watcher
|
||||
- **split 异步** —— D3 触发后 enqueue split job 进 §5 CAS 队列,跟其它 ingest / split job FIFO 共享,异步消费;**不阻塞 dream return**
|
||||
- **auto-link L1 异步** —— 写入成功后 enqueue auto-link L1 job(`auto_link_design.md` §1.4),与 split job 同 CAS 队列、FIFO 共享;不阻塞 dream return
|
||||
|
||||
检测延迟 ≈ 0(inline);split 执行延迟 ≈ 队列等待时间(typically 数秒~数十秒);新建 / update 节点不必等 split 完成,体验连续。
|
||||
|
||||
---
|
||||
|
||||
## 8. maintainer 的人 / agent 后门(暂缓 — 非底层)
|
||||
|
||||
消费层 / SDK 接口问题,不影响底层机制。底层只需保证 split / rename / delete 等 op 走同一套 §5 CAS + 守恒校验链路:F-3 仍成立(maintainer 自动路径只做 split);merge / dissolve / re-edge 在底层**不存在**(无对应机械算子)。后门接口形态推迟到 SDK 阶段再定。
|
||||
|
||||
---
|
||||
|
||||
## 9. 与 dream 模型的引用关系
|
||||
|
||||
本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §3 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 |
|
||||
| F-invariants(F-1..F-11) | `auto_dream_design.md` §4.3 |
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §2 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §3 |
|
||||
| provenance 载体形态 | `auto_dream_design.md` §4.2 |
|
||||
| dream 行为 | `auto_dream_design.md` §4.2 |
|
||||
|
||||
---
|
||||
|
||||
## 10. 下一步
|
||||
|
||||
1. **M split step 实现** —— D3 触发 → 候选 → split prompt → 写入 + E-2 守恒(§1 / §4 / §5)
|
||||
2. **D 检测信号实现清单**(D1 断链 / D3 写后 inline / D10 provenance 哪些已就绪 / 缺哪些)—— §2
|
||||
3. **D3 阈值配置**(`vault.yaml` 中 D3 token / 离散度阈值)—— §3
|
||||
4. **CAS 写入框架** —— per-path body version_stamp + CAS 写入 + EEXIST create race + 重做上限 + audit;对外暴露给 dream / auto-link L1 复用 —— §5
|
||||
5. **后门 SDK 接口形态**(暂缓 M1+)—— §8
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
|
|
@ -5,10 +5,10 @@
|
|||
> 配套阅读:
|
||||
> - `structure.md` §2.1-2.2(daily 层定位)/ §3.4(sync 动作语义)/ §7.1(synchronizer 模块)
|
||||
> - `auto_dream_design.md`:auto-memory 产物如何被 dream 消化(dream 读 daily 作为入流之一)
|
||||
> - `auto_maintain_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
> - `auto_link_design.md`:auto-link 可反向扫 daily 事件,补充实体 wikilink(daily → digest)
|
||||
> - `auto_consolidate_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想(写入 / 巩固 / 检索);daily 节点是 cognition 图视图的一部分(承载 `derived_from::` 反指),但不参与 Stage 2 巩固改造
|
||||
>
|
||||
> **四份分工**:reme 服务整体四份设计 —— **auto-memory(本文档)** / auto-dream / auto-maintain / auto-link。auto-memory 是入流端,把 agent 实时事件流切成 daily 事件原子;它的产物是 dream 消化的两路输入之一(另一路是 resource)。
|
||||
> **服务全景**:reme 服务两条主线 —— **auto-memory**(本文档,入流端 / daily 写入)与 **auto-cognition**(顶层思想:写入 = auto-dream,巩固 = auto-consolidate,检索 = auto-recall)。auto-memory 把 agent 实时事件流切成 daily 事件原子;它的产物是 dream(cognition Stage 1)消化的两路输入之一(另一路是 resource)。
|
||||
>
|
||||
> **核心立场**:auto-memory 是 `structure.md` §3.4 `sync` 动作的实现侧 —— 强调 **inline 实时**与**事件边界检测**。是不是改名 sync → auto-memory 留给上层文档对齐,本文档聚焦机制。
|
||||
|
||||
|
|
@ -29,7 +29,7 @@ agent 的对话与任务过程是连续事件流(用户回合、工具调用、
|
|||
|
||||
**显式排除**(不属于 auto-memory 职责):
|
||||
- ❌ 蒸馏 / 沉淀:那是 auto-dream(`auto_dream_design.md`)的事
|
||||
- ❌ 实体识别 / wikilink 自动补全:那是 auto-link(`auto_link_design.md`)的事
|
||||
- ❌ 实体识别 / wikilink 自动补全:cognition 三阶段不在写入后做"事后补 wikilink"(详 `auto_cognition_design.md` §1.1);所有 wikilink 由 dream 在写入瞬间产出
|
||||
- ❌ 改写 resource / digest:auto-memory 只写 daily(I-1 / I-3)
|
||||
|
||||
---
|
||||
|
|
@ -180,9 +180,9 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective
|
|||
| ← **resource** | 只读(通过 wikilink 引);不写 |
|
||||
| → **daily** | **唯一写者**(I-2);写 event folder + 主索引 |
|
||||
| → **auto-dream** | dream 读 daily 作为入流(`auto_dream_design.md` §4.2 dream scope);auto-memory 写完即对 dream 可见(走 L2 索引,有 eventual 窗口) |
|
||||
| → **auto-link** | auto-link 可反向扫 daily event,做实体识别 + wikilink 写回(`auto_link_design.md` §1.3)—— 与 auto-memory 写入不冲突(双方写不同字段段落 / CAS 协议保护)|
|
||||
| → **auto-cognition (三阶段)** | daily 节点是 cognition 图视图的一部分;dream(Stage 1)读 daily 作为入流;consolidate(Stage 2)只对 digest 节点跑 dups / community / decay,**不改 daily**;recall(Stage 3)三层并行召回时 daily 也参与命中 |
|
||||
|
||||
**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;dream / link 不写 daily 主路径,只通过 auto-link 走 §1.3 写回(read-only audit-then-write,CAS 保护)。
|
||||
**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;cognition 三阶段没有任何子阶段会**事后改写 daily**(无写回路径)。daily 一旦由 auto-memory 写完,就只被读不被改(I-2 / I-3 仍守);后续 dream / consolidate / recall 都是只读消费。
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -194,4 +194,4 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective
|
|||
4. **多 agent 隔离 schema**(M1+):若实际有并发 agent,确定 daily 子目录 / slug 命名约定
|
||||
5. **粗 / 细粒度 prompt 调参**:dogfooding 后看实际 event 数 / dream 消化效率,调 boundary prompt
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md` 共同作为契约依据。
|
||||
|
|
|
|||
323
docs4/auto_recall_design.md
Normal file
323
docs4/auto_recall_design.md
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
# auto-recall 设计(Stage 3 检索:信号融合 + 召回增强)
|
||||
|
||||
> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 3 — 检索阶段** 实现。覆盖 query 到来时如何把 vault 一等公民信号(wikilink 图 / frontmatter)与维护阶段产出信号(centrality / community / recency / archived)融合,生成最终召回。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_cognition_design.md`:三阶段顶层思想(本文档是 Stage 3)
|
||||
> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型
|
||||
> - `auto_consolidate_design.md`:Stage 2 维护 —— **本文档消费它产出的所有 `meta/*.json`**
|
||||
> - `structure.md` §4(retrieve 三种问法)/ §7.4(为什么没有 retriever 模块)
|
||||
> - `reme4/steps/index/search.py` / `traverse.py`:现有原子实现
|
||||
>
|
||||
> **核心立场**:
|
||||
> - retrieve **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法各自由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖
|
||||
> - 本文档增强**集中在 `search_step` 内部**:把维护信号融入打分 / 排序 / 过滤;`traverse_step` 仅做小幅参数扩展
|
||||
> - retrieve **只读 vault,不写 body / 不写 frontmatter**;唯一写入是 `meta/access_log.json`(命中计数,供下次 recency 计算)
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
`structure.md` §4 已规定 retrieve 三种问法(state / semantic / topological)正交分立(R-1)。本文档**只增强 semantic 问法**;state 问法已被 `list_step` 覆盖,topological 问法已被 `traverse_step` 覆盖。
|
||||
|
||||
semantic 问法当前在 `reme4/steps/index/search.py` 实现:
|
||||
|
||||
| 已就绪 | 缺口 |
|
||||
|---|---|
|
||||
| ✅ vector + keyword 并行召回 | ❌ 节点中心性加权(高权威节点不被 boost) |
|
||||
| ✅ RRF fusion(vector_weight=0.7) | ❌ 同社区 boost(`meta/communities.json` 未消费) |
|
||||
| ✅ 一跳 expand_links(向前向后,max=10) | ❌ 时效衰减 / 冷藏过滤(`meta/access_log.json`、`meta/archived.json` 未消费) |
|
||||
| ✅ min_score 过滤 + limit 截断 | ❌ 同 file 多 chunk 冗余(top-K 可全来自同节点) |
|
||||
| ✅ chunk-level 命中(start_line / end_line) | ❌ 节点级 surface(frontmatter `name + description` 未与 chunk 命中合并展示) |
|
||||
| ✅ 二跳 traverse 作为独立工具 | ❌ search 内 multi-hop expand(只一跳,跨术语关系到不了) |
|
||||
| | ❌ query rewrite / multi-query(单一表达式漏召) |
|
||||
|
||||
**本文档的工作 = 设计这些缺口怎么填**,在 `search_step` / `traverse_step` 现有形态上增量。
|
||||
|
||||
---
|
||||
|
||||
## 1. 三种问法分立(继承 R-1)
|
||||
|
||||
```
|
||||
┌─────────────┐ state 问 ──────► list_step + frontmatter filter
|
||||
│ agent │ semantic 问 ──► search_step (本文档主要增强)
|
||||
└─────────────┘ topological 问 ► traverse_step (小幅参数扩展)
|
||||
```
|
||||
|
||||
| 问法 | 原子工具 | 本文档涉及 | 备注 |
|
||||
|---|---|---|---|
|
||||
| **state** | `list_step` / `daily_list_step` / `frontmatter_read_step` | 不涉及 | frontmatter 过滤无需维护信号 |
|
||||
| **semantic** | `search_step` | **主战场**(§3-§7) | RRF fusion + 信号加权 + multi-hop + query rewrite |
|
||||
| **topological** | `traverse_step` | 小幅(§8) | 起点选择可借助维护信号 |
|
||||
|
||||
**关键约束**(继承 `structure.md` ✗-8):**绝不合并三种问法成单一 read verb**。本文档增强 search_step,但不把 list / traverse 揉进 search;agent 按需各自调用。
|
||||
|
||||
---
|
||||
|
||||
## 2. 维护信号契约消费总览
|
||||
|
||||
`auto_consolidate_design.md` §11 列出维护产出。retrieve 端按以下方式读:
|
||||
|
||||
| 信号 | 来源 | 加载时机 | 缺失行为(降级) |
|
||||
|---|---|---|---|
|
||||
| **centrality** | `file_graph` 反向索引(实时) | search_step init 时引用 file_store | 总在线(file_graph 是核心组件) |
|
||||
| **community** | `meta/communities.json` | search_step 启动 lazy load(LRU 缓存,文件 mtime 失效) | 缺失 → 不做同社区 boost |
|
||||
| **recency** | `meta/access_log.json` | 同上 | 缺失 → recency_factor = 1.0 |
|
||||
| **archived** | `meta/archived.json` | 同上 | 缺失 → 不过滤,所有节点参与 |
|
||||
| **wikilink 图** | vault 自身(file_graph) | 实时 | 总在线 |
|
||||
| **frontmatter** | vault 自身(`name` / `description`) | chunk 已带 metadata | 总在线 |
|
||||
|
||||
**version 校验**:`meta/*.json` 加载时检查 `version` 字段,与本文档约定的 schema 版本不匹配 → 走"该信号缺失"降级,日志告警(不崩)。
|
||||
|
||||
**新鲜度**:每个信号文件的 `computed_at` 暴露给调用者(metadata 中带 `signals_freshness`),调用方知道当前权重基于多久前的快照。超过阈值(默认 14 days)→ logger.warning + 仍使用(避免维护偶尔失效就拒绝服务)。
|
||||
|
||||
---
|
||||
|
||||
## 3. semantic 问法增强:打分公式
|
||||
|
||||
**目标**:把维护信号融入 fused chunk 的最终 score,让排序兼顾"文本相关 + 节点权威 + 同社区 + 时效"。
|
||||
|
||||
### 3.1 当前打分(基线)
|
||||
|
||||
```
|
||||
score = RRF_fused(vector_rank, keyword_rank, vector_weight=0.7)
|
||||
```
|
||||
|
||||
仅文本相似度。
|
||||
|
||||
### 3.2 新打分公式
|
||||
|
||||
```
|
||||
final_score = base_score
|
||||
× centrality_factor(path)
|
||||
× community_factor(path, query_seed_paths)
|
||||
× recency_factor(path)
|
||||
```
|
||||
|
||||
| 因子 | 公式 | 默认参数 | 来源 |
|
||||
|---|---|---|---|
|
||||
| **base_score** | RRF 融合分(现状) | vector_weight=0.7 | search.py |
|
||||
| **centrality_factor** | `1 + α · log(1 + inbound_count)` | α = 0.15 | file_graph 实时 |
|
||||
| **community_factor** | 同 community 命中节点 → ×β,否则 1.0 | β = 1.20 | `meta/communities.json` |
|
||||
| **recency_factor** | `exp(-Δt / τ)`,Δt = 距 last_hit_or_update | τ = 60 days | `meta/access_log.json` |
|
||||
|
||||
**为什么乘法而非加法**:
|
||||
- 各因子量级不同(base_score ≤ 0.02,centrality 与 query 无关),加法需大量 normalization;乘法天然处理量级差
|
||||
- 任一因子接近 0(极冷藏 / 极孤立)→ 整体压低,符合"弱信号一票否决"直觉
|
||||
- 默认 α/β/τ 让 factor 落在 [0.5, 2.0] 区间,不会让 base_score 完全失声
|
||||
|
||||
**已排除**:LLM rerank。它是 query-time 多调一次 LLM,成本高,M0 不引入;留 M1+ 视 dogfooding 决定。
|
||||
|
||||
### 3.3 query_seed_paths 的角色
|
||||
|
||||
community_factor 需要"query 主关注的节点是哪些"才能判断同/异社区。做法:
|
||||
1. RRF 融合后取 top-N(N=3)的 fused chunk 的 path 作 seed
|
||||
2. 后续每个候选 chunk 的 path → 查它和任一 seed 是否同社区 → boost
|
||||
3. 不需要 query 自身被映射到 community(query 是字符串,不在图里)
|
||||
|
||||
**边界**:N=3 是经验起点;N 太大会让"同社区"几乎等于"全召回"失去区分度。dogfooding 后调。
|
||||
|
||||
---
|
||||
|
||||
## 4. semantic 增强:节点级合并(unique_paths)
|
||||
|
||||
**问题(gap 5)**:fused 列表里 top-5 可能是同 file 的 5 个 chunk,信噪比退化。
|
||||
|
||||
**当前**:`expand_links` 已用 `unique_paths = list(dict.fromkeys(c.path for c in fused))`,但 fused 本身没去重,limit=5 仍可全是同节点。
|
||||
|
||||
**新方案**(节点级 dedupe + 节点级 surface):
|
||||
|
||||
```
|
||||
fused (chunk-level) → group by path → 每组保留 top_chunks_per_path 个
|
||||
→ 每组追加节点 frontmatter (name + description) 作"节点级 surface"
|
||||
→ 再按节点 best_score 排序 → limit
|
||||
```
|
||||
|
||||
| 参数 | 默认 | 含义 |
|
||||
|---|---|---|
|
||||
| `top_chunks_per_path` | 2 | 同节点最多保留多少 chunk |
|
||||
| `surface_node` | true | 是否在每组前追加 frontmatter `name + description` |
|
||||
|
||||
**为什么**:
|
||||
- 节点是 retrieve 的语义单位(`auto_dream_design.md` §2 路径即 ID),chunk 只是"展示窗口"
|
||||
- frontmatter 是节点级摘要(name + description)—— 已是 dream 写入时认证过的信号,不召它浪费
|
||||
- 同节点多 chunk 时,frontmatter + top-2 chunk 比 5 个 chunk 信息密度高
|
||||
|
||||
### 4.1 答案展示形态
|
||||
|
||||
```
|
||||
========== digest/auth/jwt-rotation.md ==========
|
||||
[node] JWT Key Rotation
|
||||
Process for rotating JWT signing keys without downtime.
|
||||
[score=0.0241 centrality=2.1 community=1.2 recency=0.91]
|
||||
|
||||
---------- chunk @5-23 ----------
|
||||
<chunk text>
|
||||
|
||||
---------- chunk @45-60 ----------
|
||||
<chunk text>
|
||||
|
||||
[expansion] 1 inbound, 2 outbound (...)
|
||||
```
|
||||
|
||||
**对照旧形态**:每个 chunk 独立成块,无节点级 surface,scores 散在 chunk 头。新形态以**节点为视觉单位**,人 / agent 看到的第一眼是"哪个节点中了",而非"哪段文字中了"。
|
||||
|
||||
---
|
||||
|
||||
## 5. semantic 增强:multi-hop expand
|
||||
|
||||
**问题(gap 4)**:当前 expand_links 只展一跳,跨术语关系("分布式锁" → 一跳到"租约机制",再一跳才到"心跳协议")到不了。
|
||||
|
||||
**新方案**:expand_links 支持 `depth` 参数;默认仍 1(保守),agent / 配置可调到 2。
|
||||
|
||||
| 参数 | 默认 | 限制 |
|
||||
|---|---|---|
|
||||
| `expand_depth` | 1 | 最大 3(避免组合爆炸) |
|
||||
| `max_links_per_direction` | 10(现状)| 每跳每方向上限,深度不展开时限到当跳总数 |
|
||||
| `expand_path_budget` | 30 | 总扩展节点数硬上限,优先深度优先(深度浅但条数少) |
|
||||
|
||||
**为什么默认仍 1**:
|
||||
- 二跳延迟不可忽略(N × 10 × 10 = 100 候选 IO)
|
||||
- agent 需要"再深一层"时显式调 `traverse_step(depth=2)` —— 三种问法分立(R-1)
|
||||
- 默认深拉会让"语义召回"变成"图召回",违背 R-1
|
||||
|
||||
**何时调 2**:dogfooding 发现 vault 节点平均出度低 / 跨术语关系频繁 → 调到 2(改 search_step 配置,不改协议)。
|
||||
|
||||
---
|
||||
|
||||
## 6. semantic 增强:query rewrite / multi-query
|
||||
|
||||
**问题(gap 6)**:用户 query "JWT 怎么轮换" 可能错过 body 写"密钥定期更换"的节点(术语不同)。
|
||||
|
||||
**方案矩阵**:
|
||||
|
||||
| 方案 | 成本 | 效果 |
|
||||
|---|---|---|
|
||||
| **(a) 不做** | 0 | 漏召部分跨术语 |
|
||||
| **(b) embedding 多 query**(用同 LLM 生成 N 个表述) | LLM 调用 1 次(query → N 表述)+ N 次 vector_search | 中等 |
|
||||
| **(c) BM25 同义词扩展**(用静态词表 / 嵌入式词表) | 0(若有词表) | 弱(中文场景词表缺) |
|
||||
| **(d) HyDE**(LLM 生成假设答案 → 嵌入这个答案而非 query) | LLM 1 次 | 高,文献证实 |
|
||||
|
||||
**首版决策**:**(a) 不做**。理由:
|
||||
- vault 本身规模 M0 不大,推断增加召回但增 LLM cost 不划算
|
||||
- 维护阶段的 community 聚类已部分弥补"跨术语关系"(同社区 boost)
|
||||
- 真要做,优先 (d) HyDE,延 M1+ 再启,实施只需加一层 query 预处理
|
||||
|
||||
**契约预留**:search_step kwargs 加 `query_rewrite: str | None`(默认 None;非 None 则用此重写代替原 query 做 vector_search,keyword_search 仍用原 query)。SDK 层可调用 LLM 生成重写后传入,reme 核心不强加 LLM 依赖。
|
||||
|
||||
---
|
||||
|
||||
## 7. semantic 增强:archived 过滤
|
||||
|
||||
**问题**:长期未访问的旧节点应该默认排除。
|
||||
|
||||
**方案**:search_step kwargs 加 `include_archived: bool`,默认 false。
|
||||
|
||||
```
|
||||
fused → drop where path in archived_set → 后续打分 / unique_paths
|
||||
```
|
||||
|
||||
**何时绕过**:
|
||||
- agent 显式 `include_archived=true`(找历史 / debug)
|
||||
- query 命中节点本身在 archived → boost 推回(冷节点突然被命中,说明不是真冷)
|
||||
- **首版不做**,过滤即过滤;如有需要,M1+ 加"intent override"机制
|
||||
|
||||
**冷启动**(`meta/archived.json` 缺失)→ 不过滤,等同 `include_archived=true`。
|
||||
|
||||
---
|
||||
|
||||
## 8. topological 问法的小增强
|
||||
|
||||
`traverse_step` 当前完整:BFS / 多 seed / direction / depth / per-edge 输出。本文档不重构,仅:
|
||||
|
||||
### 8.1 起点选择借助维护信号(可选 hint)
|
||||
|
||||
agent 调用 traverse 时往往不知道"哪个节点是该主题的中心";维护阶段产出的 centrality 可作 hint:
|
||||
|
||||
| 用例 | 做法 |
|
||||
|---|---|
|
||||
| traverse 给定 seed | 不变,直接 BFS |
|
||||
| traverse 给定主题字符串(SDK 上层语法糖) | 先 search_step 找 top-1 → 用其作 seed → traverse depth=2 |
|
||||
|
||||
**位置**:这个组合在 SDK 上层做,不进 traverse_step;reme 核心保留 traverse 原子形态。
|
||||
|
||||
### 8.2 traverse 输出消费 archived
|
||||
|
||||
traverse_step 当前不知道 archived 信号。改造:加 `exclude_archived: bool` kwarg 默认 false(traverse 默认不过滤,因为它是图问法,过滤会破坏图视角)。SDK / agent 可显式开启。
|
||||
|
||||
---
|
||||
|
||||
## 9. retrieve 写访问日志(唯一对外写入)
|
||||
|
||||
**问题**:`meta/access_log.json` 的 `last_read` / `last_hit_count_30d` 谁写?
|
||||
|
||||
**约定**:retrieve 命中节点 → 异步 append 到访问日志缓冲区;由 maintain daily batch 聚合写入 `meta/access_log.json`。
|
||||
|
||||
| 路径 | 实现 |
|
||||
|---|---|
|
||||
| **同步写**(每 query) | retrieve 把命中 path 写入内存 ring buffer(进程级)|
|
||||
| **异步落盘** | 进程退出 / 维护 daily batch / 周期 flush(默认 10 min)|
|
||||
| **聚合** | maintain 在 daily access_log 重算时:读 ring buffer + 上一份 access_log → 合并写新版 |
|
||||
|
||||
**幂等**:同 query 多次重读同节点不应放大 last_hit_count;ring buffer 按 (path, day) 去重,每天每节点最多记一次"被读"。
|
||||
|
||||
**降级**:ring buffer 写失败 / flush 失败 → 不影响 retrieve 返回,只是日志少一条;recency 信号略迟。
|
||||
|
||||
---
|
||||
|
||||
## 10. 不变量 / 边界
|
||||
|
||||
| # | 约束 | 含义 |
|
||||
|---|---|---|
|
||||
| **R-1**(继承)| 三种问法分立 | 不合并 list / search / traverse 成单一 verb |
|
||||
| **R-2**(继承)| 默认 `digest > daily > resource`,可覆盖 | search_step 通过 `search_filter` 支持限层 |
|
||||
| **R-3**(继承)| 拓扑问与层无关 | traverse 跨三层(I-4) |
|
||||
| **R-4**(继承)| Provenance 默认 lazy | retrieve 不自动 traverse(R-4);expand_links 是性能优化非语义展开 |
|
||||
| **Re-1**(本文档)| retrieve 不引入 L4 模块 | 增强限定在原子 step 内部 |
|
||||
| **Re-2**(本文档)| retrieve 只读 vault | 不改 body / frontmatter / 文件位置 |
|
||||
| **Re-3**(本文档)| retrieve 唯一对外写入是 `meta/access_log.json` | 通过 ring buffer + maintain 聚合,不直接写 |
|
||||
| **Re-4**(本文档)| 任一维护信号缺失 → 降级不崩 | `meta/*.json` 缺 → 跳过对应因子,系统始终可用 |
|
||||
| **Re-5**(本文档)| version 不兼容 → 降级 + warning | 不阻断 retrieve |
|
||||
|
||||
---
|
||||
|
||||
## 11. 与其它文档的引用关系
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| 三种问法 / R-1..R-5 | `structure.md` §4 |
|
||||
| 没有 retriever 模块 | `structure.md` §7.4 |
|
||||
| 节点 / 边 / wikilink 模型 | `auto_dream_design.md` §2 / §3 |
|
||||
| 维护信号契约 | `auto_consolidate_design.md` §11 |
|
||||
| centrality / community / recency / archived 输出 | `auto_consolidate_design.md` §3-§5 |
|
||||
| 路径即 ID | `auto_dream_design.md` §2 |
|
||||
|
||||
---
|
||||
|
||||
## 12. 下一步
|
||||
|
||||
实现进入 `reme4/steps/index/` 时,本文档与 `auto_cognition_design.md`(顶层)/ `auto_dream_design.md` / `auto_consolidate_design.md` 共同作为契约依据。
|
||||
|
||||
**search_step 增强(§3-§7)**:
|
||||
- ⏳ **打分公式**:加 centrality_factor / community_factor / recency_factor;config 化 α / β / τ(§3)
|
||||
- ⏳ **节点级合并 + surface**:group-by-path + frontmatter surface + top_chunks_per_path(§4)
|
||||
- ⏳ **multi-hop expand**:`expand_links` 支持 depth 参数,加 `expand_path_budget` 硬上限(§5)
|
||||
- ⏳ **query_rewrite kwarg**:契约预留,reme 核心不强加 LLM(§6)
|
||||
- ⏳ **archived 过滤**:`include_archived` kwarg,默认 false(§7)
|
||||
|
||||
**traverse_step 增强(§8)**:
|
||||
- ⏳ **`exclude_archived` kwarg**(默认 false)
|
||||
|
||||
**信号加载基础设施(§2)**:
|
||||
- ⏳ **`meta/*.json` lazy loader + LRU 缓存 + mtime 失效**
|
||||
- ⏳ **version 校验 + 降级路径 + warning logger**
|
||||
- ⏳ **signals_freshness metadata 暴露**
|
||||
|
||||
**access log 写入路径(§9)**:
|
||||
- ⏳ **进程级 ring buffer**(命中 path 异步 append)
|
||||
- ⏳ **周期 flush + (path, day) 幂等**
|
||||
- ⏳ **maintain daily 聚合接口**(读 ring → 合并旧 access_log → 写新版)
|
||||
|
||||
**性能与回归**:
|
||||
- ⏳ **基准测试**:打分公式启用前后的 召回 P@5 / MRR(用合成 vault + ground-truth query)
|
||||
- ⏳ **延迟监控**:维护信号读取 + multi-hop expand 的 p50 / p95
|
||||
|
|
@ -2,5 +2,7 @@
|
|||
LLM_API_KEY=sk-xxxx
|
||||
LLM_BASE_URL=https://xxxx/v1
|
||||
LLM_MODEL_NAME=xxxx
|
||||
LLM_FORMATTER_BACKEND=XXX
|
||||
LLM_BACKEND=XXX
|
||||
#EMBEDDING_API_KEY=sk-xxxx
|
||||
#EMBEDDING_BASE_URL=https://xxxx/v1
|
||||
147
reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md
Normal file
147
reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
---
|
||||
name: reme-service
|
||||
description: Use this skill whenever the user references their personal vault (markdown notes managed by the `reme` MCP, service-tier surface, backed by reme4), or when there's a meaningful session outcome to record / a question that prior work might answer. Triggers include "what do I know about X", "did I work on Y before", "save this", "记下", "落盘", "提炼", "vault", any mention of resource/ / daily/ / digest/ files, or recognizing that a non-trivial session outcome should be recorded. Skill follows a 3-phase paradigm (Recall / Log / Distill) over a 4-tier lifecycle (external channel → resource → daily → digest). Log + Distill route to two SERVICE LAYER MCP tools — `synchronizer` and `digester` — whose internal ReActAgents run the LLM loop INSIDE reme4. Inbound assets from external channels land via `upload` into `resource/<date>/` (service-only). All other tools (search / traverse / file_list / file_read / file_write / file_append / file_stat / frontmatter_*) are shared atomic primitives — whole-file CRUD covers all body changes; `frontmatter` is the one sliced RUD surface (YAML is structured data — surgical key edits cannot be safely emulated with string-substitution on the body).
|
||||
---
|
||||
|
||||
# vault — service tier (reme4)
|
||||
|
||||
The vault is a personal markdown knowledge base managed by the `reme` MCP server. **Service tier**: two service-layer LLM-driven tools (`synchronizer`, `digester`) + the service-only resource-ingest primitive (`upload`) + the full shared atomic primitive surface. Log + Distill phases hand off to the service layer; the R-M-W loops run **inside reme4** in those tools' internal ReActAgents.
|
||||
|
||||
## Business objects
|
||||
|
||||
- **Resource bucket** — passive ingest from external channels. `resource/<YYYY-MM-DD>/` is a flat folder keyed by the day the asset was received, containing the assets themselves (any file type), a `meta.json` array of provenance rows (channel / source / received_at / description), and a derived `<date>.md` view assembled from meta.json. **One ingest path only**: the `upload` tool. Read-only for everything else (synchronizer / digester / hand-edits never write here).
|
||||
- **Daily note** — hot, streaming fact log of one thread. Single file `daily/<YYYY-MM-DD>/<slug>.md`; everything worth keeping (verbatim user prompt, key tool output, intermediate data) inlined inside the body. One upstream writer per note; every other consumer treats it as read-only. Inbound channel assets do NOT live here — those go to the resource bucket and are referenced via `[[resource/<date>/<name>]]` wikilinks in the note's `## References` section when the task consumes them.
|
||||
- **digest node** — cold, curated long-lived cognition. `digest/<slug>/<slug>.md` (or nested at any depth: `digest/<scope>/<slug>/<slug>.md`). Each scope folder must contain `<folder>/<folder>.md` as its canonical entry. **Slugs are globally unique under `digest/`** — a folder name appears at most once anywhere in the tree.
|
||||
|
||||
Lifecycle: **external channel → `upload` → resource/<date>/ → session work + daily folder → distill → digest node**. Each tier is one-way downstream. Inbound assets are passive (someone sends you a file); daily materials are active (you fetched / produced them during a task); digest entries are distilled cognition. The distill marker is the daily's `status` frontmatter — a **daily-tier convention owned by the Digester** (reme core reserves only `name` / `description`; `status` is an extra used exclusively by Sync/Digester). Convention: absent ≡ `pending`; the Digester flips it to `completed` (or `skipped`) once it has processed the daily. Find unprocessed work by listing `daily/` and `frontmatter_read`-ing each summary — the ones with no `status` are pending.
|
||||
|
||||
## Tool surface (service tier)
|
||||
|
||||
| Group | Tools | Where the work runs |
|
||||
|---|---|---|
|
||||
| **Service layer (LLM-driven)** | `synchronizer`, `digester` | **Inside reme4** — internal ReActAgent |
|
||||
| **Service-only ingest** | `upload` (external channel → `resource/<date>/`) | reme4 thin primitive (no LLM) |
|
||||
| Shared retrieve | `search`, `traverse` | reme4 thin primitive (no LLM) |
|
||||
| Shared read | `file_list`, `file_read`, `file_stat`, `frontmatter_read` | reme4 thin primitive (no LLM) |
|
||||
| Shared write | `file_write`, `file_append`, `file_edit`, `frontmatter_update`, `frontmatter_delete` | reme4 thin primitive (no LLM) |
|
||||
| Shared file ops | `file_move`, `file_delete`, `file_download` | reme4 thin primitive (no LLM) |
|
||||
| Shared daily | `daily_read`, `daily_write`, `daily_list`, `daily_reindex` | reme4 thin primitive (no LLM) |
|
||||
|
||||
The shared block is identical to expert tier; what makes this **service** tier is the two service-layer tools at the top plus the service-only `upload` ingest primitive.
|
||||
|
||||
## 3-Phase paradigm
|
||||
|
||||
### Phase 1: Recall
|
||||
|
||||
**What** — retrieve relevant context (chunks ranked by RRF-fused vector + BM25 score, with optional wikilink expansion via the file graph).
|
||||
**Triggers** — intent-driven only: "what do I know about X" / "did I work on Y" / "what's connected to [[Z]]" / task needs prior methodology.
|
||||
**How** —
|
||||
- `search(query, limit?, expand_links?, ...)` for hybrid chunk retrieval.
|
||||
- `traverse(path, depth?, direction?)` to chase a seed file's wikilink neighborhood.
|
||||
- `file_list` / `file_read` / `file_stat` / `frontmatter_read` for primary-key reads.
|
||||
|
||||
```
|
||||
search query="auth refactor decisions" limit=5
|
||||
search query="see [[张三.md]]"
|
||||
traverse path="digest/zhang-san/zhang-san.md" depth=1
|
||||
```
|
||||
|
||||
### Phase 2: Log (service layer)
|
||||
|
||||
**What** — digest the recent conversation slice into a daily note. The Synchronizer's internal ReActAgent picks a slug, writes the note (everything inlined into a single file), and handles continuation (same slug → same file).
|
||||
**Triggers** — (a) intent-driven: meaningful fact / output / decision just landed; (b) **PreCompact hook**: prompt fires to dump volatile state.
|
||||
**How** — `synchronizer(messages, note?)`.
|
||||
|
||||
```
|
||||
synchronizer
|
||||
messages:
|
||||
- {role: user, content: "let's design the auth refactor"}
|
||||
- {role: assistant, content: "two options: JWT vs session..."}
|
||||
- {role: user, content: "go with JWT + refresh token rotation"}
|
||||
note: "auth refactor" # optional hint to bias the slug
|
||||
```
|
||||
|
||||
The Synchronizer reads the conversation, picks a stable slug (or reuses an existing one when `note` matches), writes `daily/<today>/<slug>.md`, and returns a `SynchronizerResult` audit (`note` path, `summary` of the just-written note, `actions`). Surface the summary verbatim if the user wants to see what landed.
|
||||
|
||||
**Surgical edits** (without going through Synchronizer's LLM loop):
|
||||
- `daily_read(slug, date?)` to probe / merge — returns body in `answer` and the parsed frontmatter dict in metadata. `exists: false` = fresh, `exists: true` = upsert.
|
||||
- `daily_write(slug, body, frontmatter?, date?, overwrite?)` for a full-note write — `overwrite=false` (default, idempotent skip-if-exists; mirrors the old `daily_resolve` probe) for fresh threads, `overwrite=true` for UPDATE after a `daily_read`. Auto-mkdirs the day folder and refreshes the day index.
|
||||
- `file_append(path, content)` for cheap end-of-file extensions to trailing sections (`## Progress`, `## Findings`, `## Decisions`) — saves the read-modify-write round-trip.
|
||||
- `frontmatter_update(path, metadata={key: value, ...})` to merge one or more frontmatter keys (call `daily_reindex` afterward if you touched `name` / `description`).
|
||||
- `frontmatter_delete(path, keys=[...])` to drop frontmatter keys.
|
||||
- For mid-body edits on a daily note, `daily_read` then `daily_write overwrite=true`. For non-daily paths, `file_read` then `file_edit` (string substitution) or `file_write` (full body) — there's no body/section slice tool; YAML is the only structured surface that earns its own RUD package.
|
||||
|
||||
Use these when you know exactly what to write; use `synchronizer` when you want the service layer to decide what's worth keeping from the conversation.
|
||||
|
||||
### Phase 3: Distill (service layer)
|
||||
|
||||
**What** — promote daily notes into the digest knowledge graph. The Digester's internal ReActAgent reads each daily note (single-file inline content), looks up existing digest nodes (globally unique slugs — same slug at any nesting depth is the same node), applies the R-M-W decision rules (CREATE / UPDATE / MOVE; mere mentions with no own-node substance are left as-is). Relations are recorded as typed wikilinks in the source node's body (`predicate:: [[X.md]]`); target bodies are never edited (backlinks come from `traverse direction=backward` at query time). After each daily note is processed, the Digester flips its `status` frontmatter to `completed` (or `skipped` if nothing was lifted) — **that flip IS the distill marker** (a daily-tier convention the Digester owns; absent ≡ pending). The Digester scans `daily/` and `frontmatter_read`s each note, picking the ones whose `status` is absent.
|
||||
**Triggers** — (a) intent-driven: task wraps and the working set is ready; (b) **SessionEnd hook**: prompt fires to call `digester` once.
|
||||
**How** — `digester(daily_paths, hint?)`.
|
||||
|
||||
```
|
||||
digester
|
||||
daily_paths:
|
||||
- daily/2026-05-17/auth-refactor
|
||||
- daily/2026-05-17/perf-bench
|
||||
hint: "End-of-task distillation — focus on the auth decisions; perf-bench is a methodology dump."
|
||||
```
|
||||
|
||||
Returns a `DistillResult` (`used_llm`, `skipped`, `daily_read`, `summary`, `error`). Surface the `summary` verbatim.
|
||||
|
||||
**Cold-path rule**: handoff once at task wrap, not per turn.
|
||||
|
||||
## Inbound channel ingest (outside the 3-phase loop)
|
||||
|
||||
When the user hands you an externally-received asset (file from wechat / email / browser / api / ...), land it in the resource bucket before doing anything else:
|
||||
|
||||
```
|
||||
upload
|
||||
path: /tmp/report-q1.pdf
|
||||
channel: wechat
|
||||
source: design-group
|
||||
description: Q1 sales report
|
||||
```
|
||||
|
||||
The tool copies the file into `resource/<today>/<basename>`, appends a `ResourceEntry` to that day's `meta.json`, and regenerates `resource/<today>/<today>.md`. Returns `{date, name, path}` — surface `path` so the user knows where the asset landed. If a downstream task consumes the asset, reference it from the daily note's References section with `[[resource/<date>/<name>]]` (the canonical resource path) rather than inlining the file.
|
||||
|
||||
Triggers — user phrases like "save this file", "上传这个", "存一下刚收到的", or a channel hook that hands you an inbound payload.
|
||||
|
||||
## Trigger → Phase quick reference
|
||||
|
||||
| Trigger | Phase | What you do |
|
||||
|---|---|---|
|
||||
| User hands you an inbound asset from an external channel | Ingest | `upload(path=..., channel=..., source=?, description=?)` |
|
||||
| User asks about prior work / [[X]] | Recall | `search` / `traverse` |
|
||||
| Fact lands during task | Log | `synchronizer(messages=[...])` |
|
||||
| Surgical edit needed | Log | `daily_read` / `daily_write` / `file_append` / `frontmatter_update` / `frontmatter_delete` / `daily_reindex` |
|
||||
| **PreCompact hook** fires | Log (urgent dump) | `synchronizer(messages=[...], note=...)` |
|
||||
| Task wraps | Distill | `digester(daily_paths=[...])` |
|
||||
| **SessionEnd hook** fires | Log + Distill | `synchronizer` then `digester` |
|
||||
|
||||
## Protocol (the rules every write must respect)
|
||||
|
||||
@../../../../protocol.md
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- ❌ Picking a fresh `note` slug on every `synchronizer` call within the same logical thread → fragments the thread. **Reuse the slug.**
|
||||
- ❌ Calling `digester` per turn → it's a handoff tool, not a per-turn tool. Once at end-of-task is the rule.
|
||||
- ❌ Calling `digester` on a daily whose `status` is already `completed` or `skipped` → it'll be a no-op; don't keep re-pushing. (The Digester's own pending scan — `file_list` + per-item `frontmatter_read` — filters those out for you.)
|
||||
- ❌ Creating a new `digest/X/x.md` when `X` already exists somewhere else under `digest/` (e.g. `digest/people/X/x.md`) — slugs are globally unique; reuse the existing node and fold the new facts in.
|
||||
- ❌ Manually `file_write`-ing under `digest/` instead of going through `digester` — digest nodes are the Digester's domain. (You can still do it for one-off corrections; just don't bypass the service layer for routine distillation.)
|
||||
- ❌ Writing `status` from outside the Digester — `status` is a Digester-owned daily-tier convention (enum `pending` / `completed` / `skipped`; absent ≡ pending). Flipping it from a hand-written tool call makes the note look already-processed (the Digester's pending scan skips it) and the next `digester` invocation never picks it up.
|
||||
- ❌ Using `file_write` (full-file replacement) to flip one frontmatter key — use `frontmatter_update`.
|
||||
- ❌ Using `file_write` to extend trailing sections like `## Progress` — use `file_append`; saves the R-M-W round-trip and the prompt tokens of echoing the whole body back.
|
||||
- ❌ Writing under `resource/` from anything other than `upload` — that bucket is the passive ingest contract. Hand-edits / `synchronizer` / `digester` must never touch it.
|
||||
- ❌ Inlining an inbound asset into the daily note body — leave it in `resource/<date>/<name>` and reference it via `[[resource/<date>/<name>]]` in the note's `## References` section. Daily notes are single-file; inbound assets stay in `resource/`.
|
||||
- ❌ Writing short-form (`[[Alice]]`) or no-extension (`[[k/x]]`) wikilinks — they don't resolve. Always full path relative to the vault with `.md`: `[[digest/alice/alice.md]]`.
|
||||
|
||||
## What you DON'T have to think about
|
||||
|
||||
- Slug uniqueness within a thread — `synchronizer` / `digester` pick paths and reuse existing notes; wikilinks are literal full paths, so two different paths never silently merge.
|
||||
- Status flips — the Digester writes `status=completed` (or `skipped`) per processed daily note; that frontmatter flag IS the distill marker, so it's never optional but you never write it yourself.
|
||||
- Frontmatter schema — only `name` / `description` / `status` are reserved (all optional); the protocol defines opinionated default axes (`lifecycle` / `scope` / `source` / `role`) but enforcement is caller-side, not protocol-side.
|
||||
- Pending-vs-digest bookkeeping — the digester scans `daily/` and `frontmatter_read`s each note to find ones whose `status` is absent; it maintains the queue.
|
||||
|
||||
If you need fine-grained control over every R-M-W decision visible in the main session's tool log, switch to [reme-expert](../reme-expert) — same shared tools, no service layer, plus a subagent that owns the Distill LLM loop in its own context window.
|
||||
|
|
@ -26,11 +26,6 @@ class OpenAIAsLLM(BaseAsLLM):
|
|||
|
||||
async def _start(self) -> None:
|
||||
kwargs = dict(self.kwargs)
|
||||
base_url = kwargs.pop("base_url", None)
|
||||
if base_url:
|
||||
client_kwargs = dict(kwargs.pop("client_kwargs", None) or {})
|
||||
client_kwargs.setdefault("base_url", base_url)
|
||||
kwargs["client_kwargs"] = client_kwargs
|
||||
self.model = OpenAIChatModel(**kwargs)
|
||||
|
||||
async def _close(self) -> None:
|
||||
|
|
|
|||
|
|
@ -53,6 +53,9 @@ class BaseJob(BaseComponent):
|
|||
raise ValueError(f"Unregistered backend '{config.backend}' of type '{ComponentEnum.STEP}'")
|
||||
params = config.model_dump()
|
||||
params["app_context"] = self.app_context
|
||||
# Inherit app-level language unless the step's own config overrides it.
|
||||
if not params.get("language") and self.app_context is not None:
|
||||
params["language"] = getattr(self.app_context.app_config, "language", "") or ""
|
||||
return step_cls, params
|
||||
|
||||
def _build_steps(self) -> list["BaseStep"]:
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ vault_dir: .reme
|
|||
daily_dir: daily
|
||||
digest_dir: digest
|
||||
resource_dir: ""
|
||||
# language: zh
|
||||
|
||||
jobs:
|
||||
update_store_index_loop:
|
||||
|
|
@ -482,15 +483,3 @@ components:
|
|||
embedding_model: ""
|
||||
keyword_index: default
|
||||
file_graph: default
|
||||
|
||||
as_llm:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
model_name: ${LLM_MODEL_NAME:-}
|
||||
stream: false
|
||||
|
||||
as_llm_formatter:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
|
|
|
|||
|
|
@ -8,6 +8,10 @@ from .common.llm_demo import LLMDemoStep
|
|||
from .common.stream_demo import StreamDemoStep1, StreamDemoStep2
|
||||
from .common.version import VersionStep
|
||||
from .evolve.auto_memory import AutoMemoryStep
|
||||
from .evolve.dream.cron_dreamer import CronDreamer
|
||||
from .evolve.dream.digest_edit import DigestEditStep
|
||||
from .evolve.dream.digest_write import DigestWriteStep
|
||||
from .evolve.dream.dreamer import Dreamer
|
||||
from .file_io.daily_create import DailyCreateStep
|
||||
from .file_io.daily_list import DailyListStep
|
||||
from .file_io.daily_reindex import DailyReindexStep
|
||||
|
|
@ -29,11 +33,6 @@ from .index.traverse import TraverseStep
|
|||
from .index.update_catalog import UpdateCatalogStep
|
||||
from .index.update_index import UpdateIndexStep
|
||||
from .index.watch_changes import WatchChangesStep
|
||||
from .dream.cron_dreamer import CronDreamer
|
||||
from .dream.digest_edit import DigestEditStep
|
||||
from .dream.digest_write import DigestWriteStep
|
||||
from .dream.dreamer import Dreamer
|
||||
from .jobs.synchronizer import Synchronizer
|
||||
from .transfer.download import DownloadStep
|
||||
from .transfer.ingest import IngestStep
|
||||
from .transfer.upload import UploadStep
|
||||
|
|
@ -76,13 +75,11 @@ __all__ = [
|
|||
"UpdateCatalogStep",
|
||||
"UpdateIndexStep",
|
||||
"WatchChangesStep",
|
||||
# dream
|
||||
# evolve.dream
|
||||
"CronDreamer",
|
||||
"DigestEditStep",
|
||||
"DigestWriteStep",
|
||||
"Dreamer",
|
||||
# jobs
|
||||
"Synchronizer",
|
||||
# transfer
|
||||
"DownloadStep",
|
||||
"IngestStep",
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Demo steps for smoke-testing the application stack."""
|
||||
"""Demo steps for integration-testing the application stack."""
|
||||
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
|
|
|
|||
|
|
@ -1,408 +0,0 @@
|
|||
extract_system_prompt: |
|
||||
You are the **dreamer** — auto-dream's create_or_update step,
|
||||
in its EXTRACT phase. Your ONLY job here is to read the material
|
||||
and identify the ABSTRACTIONS it teaches — the principles,
|
||||
patterns, decisions-as-precedent, cognitive takeaways — that
|
||||
belong in long-term memory. You commit them by calling
|
||||
`declare_units` exactly once. You do NOT do recall, integrate,
|
||||
or write. A separate downstream invocation processes each unit
|
||||
with the full material in context.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## What digest memory is for
|
||||
|
||||
Digest is the **abstract memory layer** — analogous to the
|
||||
prefrontal cortex aggregating cognition. The raw details of
|
||||
what happened (numbers, narratives, who said what, full
|
||||
procedure text) STAY IN THE MATERIAL. Digest holds the
|
||||
generalized lesson the reader should recall next time —
|
||||
the part that survives once the specific event fades.
|
||||
|
||||
When you cluster, you are NOT cataloguing the material's
|
||||
contents — you are answering: *"What abstractions does
|
||||
this material teach that I'd want a future agent / human
|
||||
to have at-hand when facing a similar situation?"*
|
||||
|
||||
## What is a memory sub-unit?
|
||||
|
||||
One sub-unit = one abstraction the material teaches. **One
|
||||
sub-unit maps to AT MOST one digest node** — Phase 2 will
|
||||
make exactly one write decision per sub-unit (CREATE /
|
||||
UPDATE / SKIP).
|
||||
|
||||
Multiple raw facts in the material that all illustrate the
|
||||
same abstraction collapse to ONE sub-unit. The Redis-kid
|
||||
versioning mechanism, the SOC2 CC6.1 rationale, and the new
|
||||
24h cadence are three FACTS, but they teach one abstraction:
|
||||
"JWT rotation cadence is driven by short-credential
|
||||
compliance, not by procedural convenience". That's one
|
||||
sub-unit. The mechanism / numbers / RFC citation are
|
||||
details — they stay in the daily note, the digest reaches
|
||||
them through `derived_from::` provenance edges.
|
||||
|
||||
Sub-units are NOT bucket names, NOT kinds, NOT the eventual
|
||||
digest slug — they are an agent-internal handle for the
|
||||
abstraction you've identified. Phase 2 picks the bucket /
|
||||
slug / write decision per sub-unit.
|
||||
|
||||
Typical abstractions, by material shape:
|
||||
|
||||
* Analysis / decision notes: the underlying principle the
|
||||
decision rests on; a pattern the analysis surfaces;
|
||||
a constraint that will recur in similar problems.
|
||||
* Discussion notes: a preference / convention that should
|
||||
shape future work; a stable concept the discussion
|
||||
crystallizes; an open question worth carrying forward.
|
||||
* Resource content: a foundational concept; a procedure
|
||||
that generalizes beyond this resource.
|
||||
|
||||
### Bias: fewer, richer sub-units over many narrow ones
|
||||
|
||||
This is the abstract layer — heavy lifting toward few
|
||||
high-leverage sub-units, not toward exhaustive coverage.
|
||||
Heuristic for splitting two pieces into two sub-units vs
|
||||
one:
|
||||
|
||||
* Same abstraction shown by different facts? → ONE sub-unit.
|
||||
* Genuinely different abstractions that a future reader
|
||||
would invoke in DIFFERENT situations? → TWO sub-units.
|
||||
* Will they evolve independently as more materials arrive?
|
||||
→ TWO sub-units.
|
||||
|
||||
When in doubt, KEEP TOGETHER (or SKIP one of them entirely).
|
||||
|
||||
Examples:
|
||||
|
||||
* "JWT rotation cadence changed to 24h" + "Redis kid
|
||||
versioning mechanism" + "SOC2 CC6.1 cited" → ONE
|
||||
sub-unit (the abstraction: *short-credential compliance
|
||||
drives auth infra cadence*). Mechanism + numbers are
|
||||
details — they stay in the daily.
|
||||
* "preference: small PRs" + "preference: no trailing
|
||||
summary in replies" → TWO sub-units. Different
|
||||
situations of invocation (code review vs response
|
||||
style), independent evolution.
|
||||
|
||||
### What NOT to declare
|
||||
|
||||
- A passing mention with no new abstraction (e.g. an OAuth
|
||||
recap that just restates a known concept) → don't declare.
|
||||
The material remains searchable via daily-note indexing;
|
||||
detail-level recall doesn't need a digest entry.
|
||||
- A fact whose only audience is the material itself
|
||||
(a one-off timestamp, a single meeting attendance) →
|
||||
don't declare. Not an abstraction.
|
||||
|
||||
### No event-level umbrella needed
|
||||
|
||||
The material itself (the daily note or resource file) IS the
|
||||
event-level aggregator. Every sub-unit you declare here will
|
||||
carry a `derived_from:: [[<material-path>]]` provenance
|
||||
wikilink, so the material becomes the fan-out point linking
|
||||
to all its derived digest nodes. Do NOT manufacture an
|
||||
extra "X-event-summary" sub-unit just to aggregate the
|
||||
others — the provenance graph already provides that view.
|
||||
|
||||
## What to do
|
||||
|
||||
1. **Read the material** — its body is packed in the user
|
||||
message below. If it references `[[resource/<date>/<name>]]`
|
||||
and that asset is critical to understanding what
|
||||
abstractions are present, you MAY open it via `read`;
|
||||
otherwise skip external reads (this is the light phase).
|
||||
|
||||
2. **Identify the abstractions** the material teaches.
|
||||
For each candidate, ask: *if I forgot all the details
|
||||
of this material in 6 months, what one-line lesson
|
||||
would I still want to recall?* That lesson is a
|
||||
sub-unit candidate.
|
||||
|
||||
3. **Call `declare_units` ONCE** with the surviving list:
|
||||
- `name` — short kebab-case handle for the abstraction
|
||||
(e.g. `auth-cadence-compliance-driven`,
|
||||
`small-pr-pref`). Agent-internal only; Phase 2
|
||||
picks the actual digest slug + bucket.
|
||||
- `summary` — 1-2 sentences describing the abstraction
|
||||
AND pointing at where in the material it's illustrated
|
||||
(e.g. "abstraction: short-credential compliance
|
||||
drives auth infra rotation cadence; illustrated by
|
||||
the 30→24h decision in 决定 backed by the SOC2 CC6.1
|
||||
criticism in 观察"). Be concrete about WHERE the
|
||||
supporting evidence lives, so Phase 2 can cite it as
|
||||
provenance without re-reading.
|
||||
|
||||
After `declare_units` returns OK, reply with one short line
|
||||
listing the sub-unit names.
|
||||
|
||||
If the material teaches no new abstraction worth long-term
|
||||
memory (e.g. routine status updates, pure logs), do NOT call
|
||||
`declare_units`; reply starting with `SKIP`.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- You CANNOT write to digest in this phase (no
|
||||
digest_write / digest_edit tools here).
|
||||
- You CANNOT do recall in this phase (no search/traverse here).
|
||||
- You declare ABSTRACTIONS (sub-units), not detail copies.
|
||||
Phase 2 handles recall + the single write decision per
|
||||
sub-unit.
|
||||
- The list you declare is the final scope for this dream call.
|
||||
|
||||
extract_user_message: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# Material to cluster
|
||||
|
||||
{material_blob}
|
||||
|
||||
Identify the ABSTRACTIONS this material teaches (lessons /
|
||||
principles / patterns worth recalling after the details fade).
|
||||
Collapse multiple supporting facts into one sub-unit when they
|
||||
illustrate the same abstraction. Call `declare_units([...])`
|
||||
exactly once with the surviving list. Reply with one short
|
||||
line listing the sub-unit names (or `SKIP` if the material
|
||||
teaches no new abstraction).
|
||||
|
||||
|
||||
integrate_system_prompt: |
|
||||
You are the **dreamer** — auto-dream's create_or_update step,
|
||||
in its INTEGRATE phase. This invocation processes ONE MEMORY
|
||||
SUB-UNIT against the full material. You see the entire material
|
||||
in the user message; Phase 1 told you which abstraction to
|
||||
focus on and pointed you at the supporting evidence. Your job:
|
||||
recall existing digest nodes (cross-bucket), decide CREATE /
|
||||
UPDATE / SKIP for this sub-unit, and write.
|
||||
|
||||
**Sub-unit maps 1:1 to a digest node.** Exactly ONE write
|
||||
decision per session.
|
||||
|
||||
## Digest is the abstract memory layer
|
||||
|
||||
Digest is **not** a faithful copy of the material — it is the
|
||||
cognitive aggregation (think prefrontal cortex). The details
|
||||
stay in the daily / resource file; digest holds the principle,
|
||||
pattern, or precedent the agent should recall later. So:
|
||||
|
||||
- **Body should be SHORT and abstract** (≈ 50-200 words for
|
||||
most nodes; longer only when the concept genuinely needs it).
|
||||
If your draft starts copying paragraphs from the material,
|
||||
you're filing detail in the wrong layer.
|
||||
- **Provenance edges carry the details.** Whenever this
|
||||
abstraction is illustrated by a specific material, add a
|
||||
`derived_from:: [[daily/...]]` or `[[resource/...]]`
|
||||
wikilink — readers drill down through the edge, not through
|
||||
re-stated facts in the body.
|
||||
- **Wikilinks between digest nodes** carry the conceptual
|
||||
graph: `relates_to::`, `depends_on::`, `is_a::`, etc.
|
||||
|
||||
## What to do
|
||||
|
||||
### a. Recall (search + read + optional traverse)
|
||||
|
||||
- **Search** — call `search` with the sub-unit's likely slug
|
||||
+ its summary. The step returns top-K matched chunks PLUS a
|
||||
one-hop link expansion (immediate wikilink neighbors of each
|
||||
hit). Hits come from any path under the vault; you care
|
||||
primarily about ones under `{digest_dir}/`.
|
||||
|
||||
- **Read full bodies** — do NOT decide UPDATE on chunk snippets
|
||||
alone. A snippet shows ~a paragraph of context, not the full
|
||||
node. For any hit (or expanded neighbor) that looks like the
|
||||
same abstraction, follow up with `read path=<hit-path>` to
|
||||
read the complete body before deciding.
|
||||
|
||||
- **Walk further if needed** — for 2+ hop exploration, use
|
||||
`traverse path=<hit-path> depth=2 direction=both`, then
|
||||
`read` the interesting paths.
|
||||
|
||||
Recall is intentionally cross-bucket — the same abstraction
|
||||
may already be filed under any bucket; surface it regardless
|
||||
of where it lives. UPDATE may target a node in any bucket.
|
||||
|
||||
### b. Decide bucket + write — exactly one of:
|
||||
|
||||
- **`digest_write(path, name, description, content)`** — for CREATE.
|
||||
Same shape as the canonical `write` job; the digest variant
|
||||
only adds path-shape validation. Use ONLY when no existing
|
||||
digest node captures this abstraction.
|
||||
- `path` must be `{digest_dir}/<bucket>/<slug>.md` where `bucket`
|
||||
is one of the FIXED bucket vocabulary below (pick the
|
||||
one a human would browse for this abstraction; use
|
||||
`unknown` only as a last resort).
|
||||
- `name` is the frontmatter name (usually the slug).
|
||||
- `description` is the one-line summary of the abstraction
|
||||
(lands in YAML frontmatter; downstream search relies on it).
|
||||
- `content` is the body — short (≈ 50-200 words), abstract,
|
||||
principle-oriented — NOT a transcript of the material.
|
||||
Do NOT prepend `---` frontmatter into `content`; the step
|
||||
composes the frontmatter from `name` + `description`
|
||||
automatically. Include at least one
|
||||
`derived_from:: [[<material-path>]]` provenance wikilink
|
||||
in the body so the abstraction can be traced back to its
|
||||
source.
|
||||
Fails if path exists; if so, this is actually an UPDATE —
|
||||
re-do recall and switch to `digest_edit`.
|
||||
|
||||
- **`digest_edit(path, old, new)`** — for UPDATE.
|
||||
This is the cognitive engagement step. The existing digest
|
||||
captures an earlier version of the abstraction; the new
|
||||
material **corroborates, corrects, or refines** it. Three
|
||||
typical shapes:
|
||||
|
||||
1. **Corroborate** (most common). The material is one
|
||||
more instance of an abstraction already captured.
|
||||
Body usually unchanged in substance — append a new
|
||||
`derived_from:: [[<this-material>]]` provenance
|
||||
wikilink so the supporting evidence accumulates.
|
||||
Optionally strengthen wording ("consistently
|
||||
observed across N sources" / replace "appears to" with
|
||||
"does"). One small `digest_edit` call is enough.
|
||||
2. **Refine** (frequent). The material reveals nuance,
|
||||
scope, or edge cases the existing abstraction
|
||||
under-specified. Edit the relevant span to be more
|
||||
precise; add the new dimension; still add the new
|
||||
`derived_from::` link. The body grows in precision,
|
||||
not in detail.
|
||||
3. **Correct** (rarer). The material contradicts the
|
||||
existing abstraction or shows it was overstated.
|
||||
Either tighten the abstraction to the narrower form
|
||||
that both old and new evidence support, or annotate
|
||||
inline (`> note: contradicted by [[new-material]] —
|
||||
<one-line>`) without arbitrating; future passes can
|
||||
reconcile. Still add the provenance link.
|
||||
|
||||
Body-only find-and-replace (frontmatter is untouched).
|
||||
Pick a `old` span big enough to be unique in the body.
|
||||
Prefer narrow spans over rewriting the whole body.
|
||||
Composition rule for `new`: only-add, not-delete — never
|
||||
drop facts the old span contained. You MAY issue more
|
||||
than one `digest_edit` against the SAME target if
|
||||
multiple sections need updating; never write to a
|
||||
different target as a side-effect.
|
||||
|
||||
`digest_edit` ENFORCES edge conservation (E-1): every
|
||||
outbound wikilink present BEFORE the replacement must still
|
||||
be present AFTER. On `REJECT_CONSERVATION` the missing
|
||||
links are listed — adjust `new` to keep them (or narrow
|
||||
`old` so the link stays outside the replaced span), then
|
||||
retry.
|
||||
|
||||
- **SKIP** — use when:
|
||||
* Phase 1 declared this sub-unit but on closer reading
|
||||
the material teaches nothing new (the existing
|
||||
abstraction's body already covers this instance AND
|
||||
already has provenance to a comparable source), OR
|
||||
* the sub-unit is too thin to lift as an abstraction —
|
||||
a one-off datapoint that doesn't generalize.
|
||||
|
||||
SKIP should be uncommon. If the abstraction exists and the
|
||||
material adds even ONE new datapoint, prefer a Corroborate-
|
||||
style UPDATE (provenance append) over SKIP — that's how
|
||||
the abstraction's confidence accumulates.
|
||||
|
||||
Write only the target you committed to for this sub-unit.
|
||||
Never edit other nodes' bodies sideways — inbound relations are
|
||||
queried later at search time, never written into target bodies.
|
||||
|
||||
## Bucket vocabulary
|
||||
|
||||
Pick the bucket per sub-unit when you write. The vocabulary
|
||||
is fixed and injected here (each line is one allowed bucket
|
||||
with its picking heuristic — `{digest_dir}/<bucket>/` is what
|
||||
a human will browse):
|
||||
|
||||
{buckets}
|
||||
|
||||
If the sub-unit straddles two buckets, pick the one matching
|
||||
its CENTER OF GRAVITY — what a reader is most likely to search
|
||||
for. Don't split into two writes.
|
||||
|
||||
User-memory ground rule (applies when both `preference` and
|
||||
`entity` are in the vocabulary above): anything about how the
|
||||
user / team likes to work, what they explicitly said NOT to
|
||||
do, what conventions they follow → `preference`. The user
|
||||
themselves, when named as an individual, is `entity`; their
|
||||
preferences live separately in `preference`.
|
||||
|
||||
## Wikilink form
|
||||
|
||||
Always full vault-relative path with `.md`:
|
||||
|
||||
- `[[{digest_dir}/<bucket>/<slug>.md]]`
|
||||
- `[[daily/<date>/<event-slug>/<note>.md]]`
|
||||
- `[[resource/<date>/<name>]]`
|
||||
|
||||
Short or extension-less forms do not resolve.
|
||||
|
||||
Optional Dataview-style typed predicates (the predicate sits
|
||||
outside the brackets):
|
||||
|
||||
- line-level: `is_a:: [[{digest_dir}/concept/jwt.md]]`
|
||||
- inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]`
|
||||
- typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]`
|
||||
|
||||
Predicate vocabulary is open (any `[A-Za-z][A-Za-z0-9_]*`);
|
||||
reuse existing predicates when reasonable. Most wikilinks are
|
||||
bare (no predicate) — use a predicate only when the relation
|
||||
has clear semantic weight.
|
||||
|
||||
## Provenance
|
||||
|
||||
The body must weave at least one provenance wikilink —
|
||||
`[[daily/...]]` or `[[resource/...]]` — so the graph stays
|
||||
connected upstream. Do NOT write provenance as bare prose
|
||||
("from yesterday's notes"); the conservation check only sees
|
||||
wikilinks, so prose provenance effectively vanishes on the
|
||||
next update.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
Reserved fields (both optional):
|
||||
|
||||
- `name` — basename without extension
|
||||
- `description` — one-line summary
|
||||
|
||||
Optional `kind` (downstream filtering hint; e.g. `concept` /
|
||||
`procedure` / `entity` / `observation` / `preference` / ...)
|
||||
— reme core does not read it for any structural decision. Do
|
||||
NOT write a `status` field — there is no distill-pass marker
|
||||
in this design.
|
||||
|
||||
## Reply
|
||||
|
||||
Reply with ONE LINE summarizing your decision for this sub-unit,
|
||||
including the UPDATE shape when applicable:
|
||||
|
||||
- `CREATE {digest_dir}/<bucket>/<slug>.md` — for create
|
||||
- `UPDATE {digest_dir}/<bucket>/<slug>.md (corroborate)` — provenance append + maybe wording strengthening
|
||||
- `UPDATE {digest_dir}/<bucket>/<slug>.md (refine)` — abstraction made more precise / extended in scope
|
||||
- `UPDATE {digest_dir}/<bucket>/<slug>.md (correct)` — abstraction tightened or contradiction annotated
|
||||
- `SKIP: <one-line reason>` — for skip
|
||||
|
||||
If `digest_edit` returned REJECT_CONSERVATION and you
|
||||
recovered, append `(recovered from REJECT_CONSERVATION)` to
|
||||
the UPDATE line.
|
||||
|
||||
integrate_user_message: |
|
||||
hint: {hint}
|
||||
|
||||
# Your assigned memory sub-unit for this call
|
||||
|
||||
name: {unit_name}
|
||||
summary: {unit_summary}
|
||||
|
||||
# Full material
|
||||
|
||||
{material_blob}
|
||||
|
||||
Process sub-unit `{unit_name}` (the summary above tells you
|
||||
what abstraction this is and where its evidence lives in the
|
||||
material). Do recall (search → read → optional traverse),
|
||||
then make EXACTLY ONE write decision: CREATE one new node,
|
||||
UPDATE one existing node (corroborate / refine / correct), or
|
||||
SKIP. Pick the bucket. Keep the body short and abstract —
|
||||
details stay in the material, reachable via `derived_from::`
|
||||
provenance links. Reply with the one-line decision per the
|
||||
format in the system prompt.
|
||||
|
|
@ -31,7 +31,7 @@ from pathlib import Path
|
|||
from pydantic import BaseModel, Field
|
||||
|
||||
from .dreamer import Dreamer, DreamResult
|
||||
from ...components import R
|
||||
from ....components import R
|
||||
|
||||
|
||||
class CronDreamResult(BaseModel):
|
||||
|
|
@ -21,10 +21,10 @@ from pathlib import Path
|
|||
import frontmatter
|
||||
|
||||
from .digest_write import _validate_digest_path, bucket_names, normalize_buckets
|
||||
from ..file_io._file_io import read_file_safe
|
||||
from ..file_io.edit import EditStep
|
||||
from ...components import R
|
||||
from ...utils.wikilink_handler import WikilinkHandler
|
||||
from ...file_io._file_io import read_file_safe
|
||||
from ...file_io.edit import EditStep
|
||||
from ....components import R
|
||||
from ....utils.wikilink_handler import WikilinkHandler
|
||||
|
||||
|
||||
@R.register("digest_edit_step")
|
||||
|
|
@ -20,8 +20,8 @@ accepts an override.
|
|||
|
||||
from pathlib import Path
|
||||
|
||||
from ..file_io.write import WriteStep
|
||||
from ...components import R
|
||||
from ...file_io.write import WriteStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
# Each bucket carries a name (the filesystem folder under ``digest/``) and a
|
||||
|
|
@ -2,8 +2,9 @@
|
|||
|
||||
Reads one daily-event note or resource file at the given vault-relative
|
||||
``path``, identifies the ABSTRACTIONS the material teaches in Phase 1,
|
||||
then in Phase 2 makes ONE cognitive write decision (CREATE / UPDATE /
|
||||
SKIP) per abstraction. See ``docs4/auto_dream_design.md`` for the model
|
||||
then in Phase 2 makes ONE cognitive write decision (CREATE or one of
|
||||
the three UPDATE flavors: CORROBORATE / REFINE / CORRECT) per
|
||||
abstraction. See ``docs4/auto_dream_design.md`` for the model
|
||||
contract (buckets / nodes / edges / evolution) and ``§4.2`` for the
|
||||
pipeline.
|
||||
|
||||
|
|
@ -19,28 +20,33 @@ Pipeline (external loop in Python, two distinct ReAct agent invocations,
|
|||
|
||||
execute():
|
||||
_extract(material_blob) # 1× ReAct: identify abstractions
|
||||
# agent calls declare_units([{name, summary}, ...])
|
||||
# agent emits ExtractedUnits structured output
|
||||
# ({units: [{name, summary}, ...]})
|
||||
for unit in self._units: # Python loop, K iterations (K = num abstractions)
|
||||
_integrate_unit(unit) # 1× ReAct per abstraction: agent sees full material +
|
||||
# the sub-unit's name/summary, recalls, decides
|
||||
# bucket, makes ONE write decision (CREATE /
|
||||
# UPDATE / SKIP). Sub-unit ↔ digest node is 1:1.
|
||||
# bucket, makes ONE write decision (CREATE or
|
||||
# one of the UPDATE flavors). Sub-unit ↔ digest
|
||||
# node is 1:1.
|
||||
|
||||
* **Phase 1 (extract / abstract)** uses a minimal toolkit
|
||||
(``declare_units`` + ``read``). The agent identifies the
|
||||
abstractions the material teaches — principles, patterns,
|
||||
precedents worth carrying forward once specifics fade. Multiple
|
||||
raw facts that illustrate the same abstraction collapse into
|
||||
ONE sub-unit. Prompt biases toward fewer / coarser sub-units;
|
||||
filing detail under a digest sub-unit is the wrong layer.
|
||||
No event-level umbrella node is manufactured — the material
|
||||
itself plays that role via ``derived_from`` provenance edges.
|
||||
* **Phase 1 (extract / abstract)** uses a read-only toolkit
|
||||
and emits an :class:`ExtractedUnits` Pydantic model as its
|
||||
final structured answer (no tool call needed for the unit
|
||||
list — agentscope's ``structured_model`` enforces the shape).
|
||||
The agent identifies the abstractions the material teaches —
|
||||
principles, patterns, precedents worth carrying forward once
|
||||
specifics fade. Multiple raw facts that illustrate the same
|
||||
abstraction collapse into ONE sub-unit. Prompt biases toward
|
||||
fewer / coarser sub-units; filing detail under a digest
|
||||
sub-unit is the wrong layer. No event-level umbrella node is
|
||||
manufactured — the material itself plays that role via
|
||||
``derived_from`` provenance edges.
|
||||
|
||||
* **Phase 2 (integrate per abstraction)** runs once per declared
|
||||
sub-unit with a fresh ReAct session (clean context) and the full
|
||||
read + write toolkit (``search``, ``traverse``, ``read``,
|
||||
``list``, ``stat``, ``frontmatter:read``, ``digest_write``,
|
||||
``digest_edit``). Three UPDATE shapes are surfaced explicitly
|
||||
``frontmatter_read``, ``digest_write``, ``digest_edit``). Three
|
||||
UPDATE shapes are surfaced explicitly
|
||||
in the prompt:
|
||||
|
||||
- **corroborate** (most common): the abstraction already
|
||||
|
|
@ -55,9 +61,10 @@ Pipeline (external loop in Python, two distinct ReAct agent invocations,
|
|||
or annotate the contradiction inline + add provenance.
|
||||
|
||||
CREATE is reserved for genuinely new abstractions not yet in
|
||||
the vault. SKIP should be uncommon — even an additional
|
||||
instance of an existing abstraction usually warrants a
|
||||
corroborate-style UPDATE.
|
||||
the vault — even thin first-encounter seeds, which grow via
|
||||
CORROBORATE / REFINE on later passes. There is no SKIP outcome:
|
||||
Phase 1 is the gate for "not worth memorizing"; anything that
|
||||
reaches Phase 2 warrants a write.
|
||||
|
||||
The trade-off vs heavy Phase 1: full material is sent to LLM K
|
||||
times in Phase 2 (one per abstraction). The advantages: no
|
||||
|
|
@ -85,16 +92,17 @@ Invocation form (CLI / MCP):
|
|||
import datetime
|
||||
import zoneinfo
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from agentscope.agent import ReActAgent
|
||||
from agentscope.message import Msg, TextBlock
|
||||
from agentscope.tool import Toolkit, ToolResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .digest_edit import DigestEditStep
|
||||
from .digest_write import DigestWriteStep, bucket_names, normalize_buckets
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
from .._evolve import FlexReActAgent
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
_EXTRACT_READ_TOOLS: tuple[str, ...] = ("read",)
|
||||
|
|
@ -103,9 +111,7 @@ _INTEGRATE_READ_TOOLS: tuple[str, ...] = (
|
|||
"search",
|
||||
"traverse",
|
||||
"read",
|
||||
"list",
|
||||
"stat",
|
||||
"frontmatter:read",
|
||||
"frontmatter_read",
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -125,6 +131,101 @@ def _pack_material(file_store, path: str) -> str:
|
|||
return f"### {path}\n(error reading: {type(e).__name__}: {e})\n"
|
||||
|
||||
|
||||
class MemoryUnit(BaseModel):
|
||||
"""One memory sub-unit identified by Phase 1's structured output."""
|
||||
|
||||
name: str = Field(
|
||||
description=(
|
||||
"Short kebab-case identifier for the abstraction "
|
||||
"(e.g. 'jwt-rotation-decision', 'pr-size-pref'). "
|
||||
"Agent-internal handle — NOT the eventual digest slug; "
|
||||
"Phase 2 picks the actual filing path + bucket."
|
||||
),
|
||||
)
|
||||
summary: str = Field(
|
||||
description=(
|
||||
"1-2 sentences naming the abstraction AND pointing at where "
|
||||
"in the material the supporting evidence lives "
|
||||
"(e.g. 'short-credential compliance drives auth cadence; "
|
||||
"illustrated by the 30→24h decision in the 'Decision' section "
|
||||
"+ the SOC2 CC6.1 criticism in the 'Observation' section')."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ExtractedUnits(BaseModel):
|
||||
"""Structured output emitted by Phase 1's extract agent."""
|
||||
|
||||
units: list[MemoryUnit] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"Memory sub-units identified in the material — orthogonal "
|
||||
"abstractions (principles / patterns / precedents) worth "
|
||||
"lifting into long-term memory. Empty list = nothing worth "
|
||||
"lifting (Phase 2 is skipped)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _render_outcome_line(unit_name: str, o: "IntegrateOutcome") -> str:
|
||||
"""Format one IntegrateOutcome as a one-line summary entry."""
|
||||
if o.action == "CREATE":
|
||||
body = f"CREATE {o.target_path}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
else: # CORROBORATE / REFINE / CORRECT (all UPDATE-flavored)
|
||||
recovered = " (recovered from REJECT_CONSERVATION)" if o.recovered_from_conservation else ""
|
||||
body = f"{o.action} {o.target_path}{recovered}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
return f"[{unit_name}] {body}"
|
||||
|
||||
|
||||
class IntegrateOutcome(BaseModel):
|
||||
"""Structured outcome reported by Phase 2 for one sub-unit."""
|
||||
|
||||
action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field(
|
||||
description=(
|
||||
"Outcome of the write decision for this sub-unit. Phase 1 already "
|
||||
"filtered out non-abstractions, so every sub-unit reaching you "
|
||||
"warrants a write — pick the matching fine-grained action: "
|
||||
"`CREATE` — brand-new digest node (recall returned no node "
|
||||
"covering this abstraction); even thin first-encounter seeds go "
|
||||
"here, they grow via CORROBORATE / REFINE on later passes. "
|
||||
"`CORROBORATE` (most common when a covering node exists) — "
|
||||
"provenance append + optional wording strengthening; the "
|
||||
"abstraction already covers this material. `REFINE` — covering "
|
||||
"node exists but the material reveals nuance, scope, or edge "
|
||||
"cases the abstraction under-specified. `CORRECT` — covering "
|
||||
"node exists but the material contradicts it; tighten the "
|
||||
"abstraction or annotate the contradiction inline."
|
||||
),
|
||||
)
|
||||
target_path: str = Field(
|
||||
description=(
|
||||
"The digest path you wrote to — must match what your " "`digest_write` / `digest_edit` call(s) targeted."
|
||||
),
|
||||
)
|
||||
note: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Optional ONE short line, ≤ 200 chars, no newlines, summarizing "
|
||||
"what landed (e.g. 'extended scope to also cover X'). Do NOT "
|
||||
"dump recall summaries, search results, internal reasoning, or "
|
||||
"transcripts here — those belong in the ReAct trace, not the "
|
||||
"outcome note."
|
||||
),
|
||||
)
|
||||
recovered_from_conservation: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Set to true if `digest_edit` initially returned "
|
||||
"REJECT_CONSERVATION and you re-composed `new` to preserve the "
|
||||
"missing links. Only meaningful for CORROBORATE / REFINE / CORRECT."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class DreamResult(BaseModel):
|
||||
"""Outcome of one dreamer invocation.
|
||||
|
||||
|
|
@ -204,66 +305,6 @@ class Dreamer(BaseStep):
|
|||
except Exception:
|
||||
return False
|
||||
|
||||
def _make_declare_units_tool(self):
|
||||
"""Tool closure: agent commits the memory sub-units present in the material.
|
||||
|
||||
Each unit is one orthogonal chunk of memory-worth information in this
|
||||
material — e.g. for an analysis note: the subject, the method, the
|
||||
decision, the finding, the open question. Free-form; not bound to the
|
||||
digest bucket vocabulary (Phase 2 picks the bucket per atom at write
|
||||
time).
|
||||
"""
|
||||
|
||||
async def declare_units(units: list[dict]) -> ToolResponse:
|
||||
if not isinstance(units, list):
|
||||
return ToolResponse(
|
||||
content=[
|
||||
TextBlock(
|
||||
type="text",
|
||||
text=f"REJECT: units must be a list, got {type(units).__name__}",
|
||||
),
|
||||
],
|
||||
)
|
||||
cleaned: list[dict] = []
|
||||
for i, u in enumerate(units):
|
||||
if not isinstance(u, dict):
|
||||
return ToolResponse(
|
||||
content=[
|
||||
TextBlock(
|
||||
type="text",
|
||||
text=f"REJECT: units[{i}] must be an object",
|
||||
),
|
||||
],
|
||||
)
|
||||
name = str(u.get("name", "")).strip()
|
||||
summary = str(u.get("summary", "")).strip()
|
||||
if not name or not summary:
|
||||
return ToolResponse(
|
||||
content=[
|
||||
TextBlock(
|
||||
type="text",
|
||||
text=f"REJECT: units[{i}] missing required 'name' or 'summary'",
|
||||
),
|
||||
],
|
||||
)
|
||||
cleaned.append({"name": name, "summary": summary})
|
||||
# Last call wins; replaces any previous declaration in this session.
|
||||
self._units = cleaned
|
||||
return ToolResponse(
|
||||
content=[
|
||||
TextBlock(
|
||||
type="text",
|
||||
text=(
|
||||
f"OK: declared {len(cleaned)} memory sub-unit(s) "
|
||||
f"({', '.join(u['name'] for u in cleaned)}). "
|
||||
"Phase 1 closed. Downstream will process each sub-unit in a separate session."
|
||||
),
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
return declare_units
|
||||
|
||||
def _make_digest_write_tool(self):
|
||||
"""Tool closure: wraps :class:`DigestWriteStep` and tracks creates."""
|
||||
|
||||
|
|
@ -306,67 +347,11 @@ class Dreamer(BaseStep):
|
|||
return digest_edit
|
||||
|
||||
def _build_extract_toolkit(self) -> Toolkit:
|
||||
"""Minimal toolkit for the extract agent: declare_units + read-only."""
|
||||
"""Read-only toolkit for the extract agent. Sub-units come back via
|
||||
:class:`ExtractedUnits` structured output, not via a tool call."""
|
||||
toolkit = Toolkit()
|
||||
for job_name in _EXTRACT_READ_TOOLS:
|
||||
self.add_as_tool(toolkit, job_name)
|
||||
declare_units_desc = (
|
||||
"Commit the list of MEMORY SUB-UNITS present in this material — the orthogonal "
|
||||
"information chunks worth lifting into long-term memory. Each entry is one focused "
|
||||
"sub-unit (e.g. for an analysis note: the subject, the method, a decision, a finding). "
|
||||
"Free-form — sub-units are NOT bucket names, just an agent-internal clustering of the "
|
||||
"material's key information. Call EXACTLY ONCE after reading. Downstream processes each "
|
||||
"sub-unit in its own session and picks the bucket per atom at write time."
|
||||
)
|
||||
toolkit.register_tool_function(
|
||||
tool_func=self._make_declare_units_tool(),
|
||||
func_name="declare_units",
|
||||
func_description=declare_units_desc,
|
||||
json_schema={
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "declare_units",
|
||||
"description": declare_units_desc,
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"units": {
|
||||
"type": "array",
|
||||
"description": (
|
||||
"Memory sub-units identified in the material. Each is one "
|
||||
"orthogonal information chunk; the same topic does not get "
|
||||
"duplicated, but multiple distinct topics each get their own entry."
|
||||
),
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Short kebab-case identifier for the sub-unit "
|
||||
"(e.g. 'jwt-rotation-decision', 'pr-size-pref'). "
|
||||
"Agent-internal only — not the eventual digest slug."
|
||||
),
|
||||
},
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"1-2 sentences pointing the downstream agent at "
|
||||
"the SPECIFIC part of the material this sub-unit "
|
||||
"covers (e.g. 'the JWT rotation cadence decision "
|
||||
"in the 决定 section, driven by SOC2')."
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["name", "summary"],
|
||||
},
|
||||
},
|
||||
},
|
||||
"required": ["units"],
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
return toolkit
|
||||
|
||||
def _build_integrate_toolkit(self) -> Toolkit:
|
||||
|
|
@ -467,9 +452,9 @@ class Dreamer(BaseStep):
|
|||
return toolkit
|
||||
|
||||
async def _extract(self, material_blob: str, hint: str, vault_dir: Path) -> str:
|
||||
"""Phase 1: one ReAct invocation — read material + declare_units. Returns LLM summary."""
|
||||
"""Phase 1: one ReAct invocation — read material + emit ExtractedUnits. Returns LLM summary."""
|
||||
toolkit = self._build_extract_toolkit()
|
||||
agent = ReActAgent(
|
||||
agent = FlexReActAgent(
|
||||
name="reme_dreamer_extract",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
|
|
@ -487,18 +472,43 @@ class Dreamer(BaseStep):
|
|||
hint=hint or "(none)",
|
||||
material_blob=material_blob,
|
||||
)
|
||||
msg = await agent.reply(Msg(name="reme", role="user", content=user_message))
|
||||
msg = await agent.reply(
|
||||
Msg(name="reme", role="user", content=user_message),
|
||||
structured_model=ExtractedUnits,
|
||||
)
|
||||
|
||||
# Structured output lands in msg.metadata as a dict matching ExtractedUnits.
|
||||
# Empty / missing → no sub-units (Phase 2 will skip).
|
||||
meta = msg.metadata if isinstance(msg.metadata, dict) else {}
|
||||
cleaned: list[dict] = []
|
||||
for raw in meta.get("units") or []:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
name = str(raw.get("name") or "").strip()
|
||||
summary = str(raw.get("summary") or "").strip()
|
||||
if name and summary:
|
||||
cleaned.append({"name": name, "summary": summary})
|
||||
self._units = cleaned
|
||||
return (msg.get_text_content() or "").strip()
|
||||
|
||||
async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> str:
|
||||
"""One ReAct invocation per memory sub-unit. Returns LLM summary of writes."""
|
||||
async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome:
|
||||
"""One ReAct invocation per memory sub-unit. Returns the parsed
|
||||
:class:`IntegrateOutcome` reported by the agent.
|
||||
|
||||
File writes happen as side effects via the ``digest_write`` /
|
||||
``digest_edit`` tool calls during the ReAct loop (which populate
|
||||
``self._created`` / ``self._updated`` / ``self._violations``); the
|
||||
structured outcome here is the agent's own summary of what it
|
||||
decided — useful for rendering and for catching hallucinations
|
||||
(action=CREATE without the matching write call landing in trackers).
|
||||
"""
|
||||
toolkit = self._build_integrate_toolkit()
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
buckets_block = "\n".join(
|
||||
f" - `{digest_dir}/{b['name']}/`" + (f" — {b['description']}" if b.get("description") else "")
|
||||
for b in self.buckets
|
||||
)
|
||||
agent = ReActAgent(
|
||||
agent = FlexReActAgent(
|
||||
name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
|
|
@ -518,8 +528,29 @@ class Dreamer(BaseStep):
|
|||
unit_summary=unit.get("summary", ""),
|
||||
material_blob=material_blob,
|
||||
)
|
||||
msg = await agent.reply(Msg(name="reme", role="user", content=user_message))
|
||||
return (msg.get_text_content() or "").strip()
|
||||
# Snapshot trackers so we can reconstruct the outcome from the
|
||||
# filesystem side effects if the agent's structured emission slips.
|
||||
created_before = len(self._created)
|
||||
updated_before = len(self._updated)
|
||||
msg = await agent.reply(
|
||||
Msg(name="reme", role="user", content=user_message),
|
||||
structured_model=IntegrateOutcome,
|
||||
)
|
||||
meta = msg.metadata if isinstance(msg.metadata, dict) else {}
|
||||
try:
|
||||
return IntegrateOutcome.model_validate(meta)
|
||||
except Exception:
|
||||
# The LLM occasionally drops the final structured emission even
|
||||
# after a successful tool call. The trackers are the source of
|
||||
# truth — reconstruct the outcome from the new entries this
|
||||
# session added.
|
||||
new_created = self._created[created_before:]
|
||||
new_updated = self._updated[updated_before:]
|
||||
if new_created:
|
||||
return IntegrateOutcome(action="CREATE", target_path=new_created[-1])
|
||||
if new_updated:
|
||||
return IntegrateOutcome(action="CORROBORATE", target_path=new_updated[-1])
|
||||
raise
|
||||
|
||||
async def dream_one(self, path: str, hint: str = "") -> DreamResult:
|
||||
"""Run the full extract + integrate pipeline on one vault-relative
|
||||
|
|
@ -553,7 +584,7 @@ class Dreamer(BaseStep):
|
|||
|
||||
vault_dir = self._vault_dir()
|
||||
|
||||
# Phase 1 — extract (light). Agent calls declare_units to commit the
|
||||
# Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the
|
||||
# memory sub-units worth lifting.
|
||||
self.logger.info(f"[{self.name}] extract phase: path={path!r}")
|
||||
extract_summary = await self._extract(material_blob, hint, vault_dir)
|
||||
|
|
@ -562,7 +593,7 @@ class Dreamer(BaseStep):
|
|||
return DreamResult(
|
||||
used_llm=True,
|
||||
path=path,
|
||||
summary=extract_summary or "SKIP: no memory sub-units declared",
|
||||
summary=extract_summary or "no memory sub-units declared",
|
||||
skipped=True,
|
||||
)
|
||||
|
||||
|
|
@ -572,25 +603,27 @@ class Dreamer(BaseStep):
|
|||
)
|
||||
|
||||
# Phase 2 — integrate, one fresh ReAct per sub-unit. Python-level
|
||||
# loop, not agent loop. Each session decides bucket per atom written.
|
||||
per_unit_replies: list[str] = []
|
||||
# loop, not agent loop. Each session emits a structured
|
||||
# IntegrateOutcome; file writes happen as side effects via
|
||||
# digest_write / digest_edit tool calls.
|
||||
per_unit_lines: list[str] = []
|
||||
for i, unit in enumerate(self._units, start=1):
|
||||
name = unit.get("name", "?")
|
||||
try:
|
||||
reply = await self._integrate_unit(unit, material_blob, hint, vault_dir)
|
||||
outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir)
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"[{self.name}] integrate {i}/{len(self._units)} (unit={name}) " f"failed: {type(e).__name__}: {e}",
|
||||
)
|
||||
per_unit_replies.append(f"[{name}] FAILED: {type(e).__name__}: {e}")
|
||||
per_unit_lines.append(f"[{name}] FAILED: {type(e).__name__}: {e}")
|
||||
continue
|
||||
per_unit_replies.append(f"[{name}]\n{reply}")
|
||||
per_unit_lines.append(_render_outcome_line(name, outcome))
|
||||
|
||||
summary = (
|
||||
f"Declared {len(self._units)} sub-unit(s) "
|
||||
f"({', '.join(u['name'] for u in self._units)}); "
|
||||
f"created {len(self._created)}, updated {len(self._updated)}, "
|
||||
f"conservation violations {len(self._violations)}.\n" + "\n\n".join(per_unit_replies)
|
||||
f"conservation violations {len(self._violations)}.\n" + "\n".join(per_unit_lines)
|
||||
)
|
||||
|
||||
return DreamResult(
|
||||
|
|
@ -619,7 +652,7 @@ class Dreamer(BaseStep):
|
|||
self.context.response.answer = f"Error: {result.error}"
|
||||
elif result.skipped:
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = result.summary or "SKIP"
|
||||
self.context.response.answer = result.summary or "Skipped: no memory sub-units declared"
|
||||
else:
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = result.summary
|
||||
741
reme4/steps/evolve/dream/dreamer.yaml
Normal file
741
reme4/steps/evolve/dream/dreamer.yaml
Normal file
|
|
@ -0,0 +1,741 @@
|
|||
extract_system_prompt: |
|
||||
You are the **dreamer** — EXTRACT phase. Your ONLY job here
|
||||
is to read the material
|
||||
and identify the ABSTRACTIONS it teaches — the principles,
|
||||
patterns, decisions-as-precedent, cognitive takeaways — that
|
||||
belong in long-term memory. You commit them via the structured
|
||||
output schema attached to this call (an `ExtractedUnits`
|
||||
object). You do NOT do recall, integrate, or write. A separate
|
||||
downstream invocation processes each unit with the full material
|
||||
in context.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## What digest memory is for
|
||||
|
||||
Digest is the **abstract memory layer** — analogous to the
|
||||
prefrontal cortex aggregating cognition. The raw details of
|
||||
what happened (numbers, narratives, who said what, full
|
||||
procedure text) STAY IN THE MATERIAL. Digest holds the
|
||||
generalized lesson the reader should recall next time —
|
||||
the part that survives once the specific event fades.
|
||||
|
||||
When you cluster, you are NOT cataloguing the material's
|
||||
contents — you are answering: *"What abstractions does
|
||||
this material teach that I'd want a future agent / human
|
||||
to have at-hand when facing a similar situation?"*
|
||||
|
||||
## What is a memory sub-unit?
|
||||
|
||||
One sub-unit = one abstraction the material teaches. **One
|
||||
sub-unit maps to exactly one digest node** — Phase 2 will
|
||||
make one write decision per sub-unit (CREATE or one of the
|
||||
three UPDATE flavors). Phase 1 is the gate for "not worth
|
||||
memorizing"; once a sub-unit reaches Phase 2 it WILL be
|
||||
written.
|
||||
|
||||
Multiple raw facts in the material that all illustrate the
|
||||
same abstraction collapse to ONE sub-unit. The Redis-kid
|
||||
versioning mechanism, the SOC2 CC6.1 rationale, and the new
|
||||
24h cadence are three FACTS, but they teach one abstraction:
|
||||
"JWT rotation cadence is driven by short-credential
|
||||
compliance, not by procedural convenience". That's one
|
||||
sub-unit. The mechanism / numbers / RFC citation are
|
||||
details — they stay in the daily note, the digest reaches
|
||||
them through `derived_from::` provenance edges.
|
||||
|
||||
Sub-units are NOT bucket names, NOT kinds, NOT the eventual
|
||||
digest slug — they are an agent-internal handle for the
|
||||
abstraction you've identified. Phase 2 picks the bucket /
|
||||
slug / write decision per sub-unit.
|
||||
|
||||
Typical abstractions, by material shape:
|
||||
|
||||
* Analysis / decision notes: the underlying principle the
|
||||
decision rests on; a pattern the analysis surfaces;
|
||||
a constraint that will recur in similar problems.
|
||||
* Discussion notes: a preference / convention that should
|
||||
shape future work; a stable concept the discussion
|
||||
crystallizes; an open question worth carrying forward.
|
||||
* Resource content: a foundational concept; a procedure
|
||||
that generalizes beyond this resource.
|
||||
|
||||
### Bias: fewer, richer sub-units over many narrow ones
|
||||
|
||||
This is the abstract layer — heavy lifting toward few
|
||||
high-leverage sub-units, not toward exhaustive coverage.
|
||||
Heuristic for splitting two pieces into two sub-units vs
|
||||
one:
|
||||
|
||||
* Same abstraction shown by different facts? → ONE sub-unit.
|
||||
* Genuinely different abstractions that a future reader
|
||||
would invoke in DIFFERENT situations? → TWO sub-units.
|
||||
* Will they evolve independently as more materials arrive?
|
||||
→ TWO sub-units.
|
||||
|
||||
When in doubt, KEEP TOGETHER (or DROP one of them entirely).
|
||||
|
||||
Counter-example for splitting: "preference: small PRs" +
|
||||
"preference: no trailing summary in replies" → TWO sub-units.
|
||||
Different situations of invocation (code review vs response
|
||||
style), independent evolution.
|
||||
|
||||
### What NOT to declare
|
||||
|
||||
- Passing mentions with no new abstraction (e.g. an OAuth
|
||||
recap that restates a known concept) — daily-note indexing
|
||||
already covers detail-level recall.
|
||||
- Facts whose only audience is the material itself (one-off
|
||||
timestamps, single meeting attendance) — not an abstraction.
|
||||
- Event-level umbrella sub-units (e.g. "X-event-summary") —
|
||||
every sub-unit already carries a `derived_from:: [[<material-path>]]`
|
||||
wikilink, so the material itself is the fan-out point linking
|
||||
to all its derived digest nodes; the provenance graph
|
||||
already provides that view.
|
||||
|
||||
## What to do
|
||||
|
||||
1. **Read the material** — its body is packed in the user
|
||||
message below. If it references `[[resource/<date>/<name>]]`
|
||||
and that asset is critical to understanding what
|
||||
abstractions are present, you MAY open it via `read`;
|
||||
otherwise skip external reads (this is the light phase).
|
||||
|
||||
2. **Identify the abstractions** the material teaches.
|
||||
For each candidate, ask: *if I forgot all the details
|
||||
of this material in 6 months, what one-line lesson
|
||||
would I still want to recall?* That lesson is a
|
||||
sub-unit candidate.
|
||||
|
||||
3. **Emit the surviving list** as your structured output. Each
|
||||
entry's `summary` should be concrete about WHERE in the
|
||||
material the supporting evidence lives (e.g. "the 30→24h
|
||||
decision in the 'Decision' section backed by the SOC2 CC6.1
|
||||
criticism in the 'Observation' section"), so Phase 2 can
|
||||
cite it as provenance without re-reading. Field shapes are
|
||||
enforced by the schema attached to this call.
|
||||
|
||||
If the material teaches no new abstraction worth long-term
|
||||
memory (e.g. routine status updates, pure logs), emit an empty
|
||||
unit list.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- You CANNOT write to digest in this phase (no
|
||||
digest_write / digest_edit tools here).
|
||||
- You CANNOT do recall in this phase (no search/traverse here).
|
||||
- You declare ABSTRACTIONS (sub-units), not detail copies.
|
||||
Phase 2 handles recall + the single write decision per
|
||||
sub-unit.
|
||||
- The structured output you emit is the final scope for this
|
||||
dream call.
|
||||
|
||||
extract_user_message: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# Material to cluster
|
||||
|
||||
{material_blob}
|
||||
|
||||
Identify the ABSTRACTIONS this material teaches (lessons /
|
||||
principles / patterns worth recalling after the details fade).
|
||||
Collapse multiple supporting facts into one sub-unit when they
|
||||
illustrate the same abstraction. Emit the result via the
|
||||
structured output schema attached to this call. Use an empty
|
||||
unit list when the material teaches no new abstraction.
|
||||
|
||||
|
||||
integrate_system_prompt: |
|
||||
You are the **dreamer** — INTEGRATE phase. This invocation
|
||||
processes ONE MEMORY
|
||||
SUB-UNIT against the full material. You see the entire material
|
||||
in the user message; Phase 1 told you which abstraction to
|
||||
focus on and pointed you at the supporting evidence. Your job:
|
||||
recall existing digest nodes (cross-bucket), decide between
|
||||
CREATE and the three UPDATE flavors (CORROBORATE / REFINE /
|
||||
CORRECT), and write.
|
||||
|
||||
**Sub-unit maps 1:1 to a digest node.** Exactly ONE write
|
||||
per session — there is no "no-write" outcome; Phase 1 is the
|
||||
gate for "not worth memorizing".
|
||||
|
||||
## Digest is the abstract memory layer
|
||||
|
||||
Digest is **not** a faithful copy of the material — it is the
|
||||
cognitive aggregation (think prefrontal cortex). The details
|
||||
stay in the daily / resource file; digest holds the principle,
|
||||
pattern, or precedent the agent should recall later. So:
|
||||
|
||||
- **Body should be SHORT and abstract** (≈ 50-200 words for
|
||||
most nodes; longer only when the concept genuinely needs it).
|
||||
If your draft starts copying paragraphs from the material,
|
||||
you're filing detail in the wrong layer.
|
||||
- **Provenance edges carry the details.** Whenever this
|
||||
abstraction is illustrated by a specific material, add a
|
||||
`derived_from:: [[daily/...]]` or `[[resource/...]]`
|
||||
wikilink — readers drill down through the edge, not through
|
||||
re-stated facts in the body.
|
||||
- **Wikilinks between digest nodes** carry the conceptual
|
||||
graph: `relates_to::`, `depends_on::`, `is_a::`, etc.
|
||||
|
||||
## What to do
|
||||
|
||||
Two-stage flow: **RECALL** (assemble candidate paths) → **HIT**
|
||||
(confirm whether any candidate carries this sub-unit's
|
||||
abstraction). The decision falls out of stage 2:
|
||||
|
||||
hit set empty ⇒ CREATE
|
||||
hit set non-empty ⇒ UPDATE the best match
|
||||
(CORROBORATE / REFINE / CORRECT)
|
||||
|
||||
### Stage 1 — RECALL (search + traverse)
|
||||
|
||||
Goal: surface candidate paths under `{digest_dir}/`. Recall is
|
||||
intentionally cross-bucket; UPDATE may target any bucket.
|
||||
|
||||
- **`search`** — keyword + vector hits. Call with the sub-unit's
|
||||
likely slug + its summary. Returns top-K matched chunks plus
|
||||
a one-hop wikilink expansion.
|
||||
|
||||
- **`traverse path=<top-hit> depth=2 direction=both`** — graph
|
||||
expansion. Run this whenever `search` returned ANY hit under
|
||||
`{digest_dir}/`, even if the top hit looks unrelated by
|
||||
snippet alone. Search is keyword-based and routinely misses
|
||||
semantically close abstractions filed under different
|
||||
terminology — those live one wikilink away from a noisy hit.
|
||||
Skipping traverse is the main failure mode that produces
|
||||
duplicate nodes under different bucket / slug.
|
||||
|
||||
If `search` returns nothing under `{digest_dir}/`, there is no
|
||||
anchor to traverse from. Recall ends with an empty candidate
|
||||
set; proceed to CREATE.
|
||||
|
||||
### Stage 2 — HIT (frontmatter_read + read)
|
||||
|
||||
Goal: for each candidate path, decide whether it carries the
|
||||
same abstraction as your sub-unit. Progressive disclosure —
|
||||
cheap triage first.
|
||||
|
||||
- **`frontmatter_read path=<candidate>`** — peek `name` +
|
||||
`description`. If they clearly refer to a DIFFERENT
|
||||
abstraction, drop the candidate without paying for the body.
|
||||
|
||||
- **`read path=<candidate>`** — full body for every survivor.
|
||||
Do NOT decide UPDATE on chunk snippets or frontmatter alone.
|
||||
The body is what you compare your sub-unit against.
|
||||
|
||||
Hit set = candidates whose body confirms the same abstraction.
|
||||
|
||||
### Decision
|
||||
|
||||
- **Hit set empty** ⇒ CREATE a new digest node.
|
||||
- **Hit set non-empty** ⇒ UPDATE the best-matching hit:
|
||||
same instance restated → CORROBORATE; nuance/scope added →
|
||||
REFINE; contradiction or overstatement → CORRECT.
|
||||
|
||||
### b. Decide bucket + write — exactly one of:
|
||||
|
||||
- **`digest_write(path, name, description, content)`** — for CREATE.
|
||||
Same shape as the canonical `write` job; the digest variant
|
||||
only adds path-shape validation. Use ONLY when no existing
|
||||
digest node captures this abstraction.
|
||||
- `path` must be `{digest_dir}/<bucket>/<slug>.md` where `bucket`
|
||||
is one of the FIXED bucket vocabulary below (pick the
|
||||
one a human would browse for this abstraction; use
|
||||
`unknown` only as a last resort).
|
||||
- `name` is the frontmatter name (usually the slug).
|
||||
- `description` is the one-line summary of the abstraction
|
||||
(lands in YAML frontmatter; downstream search relies on it).
|
||||
- `content` is the body — short (≈ 50-200 words), abstract,
|
||||
principle-oriented — NOT a transcript of the material.
|
||||
Do NOT prepend `---` frontmatter into `content`; the step
|
||||
composes the frontmatter from `name` + `description`
|
||||
automatically. Include at least one
|
||||
`derived_from:: [[<material-path>]]` provenance wikilink
|
||||
in the body so the abstraction can be traced back to its
|
||||
source.
|
||||
Fails if path exists; if so, this is actually an UPDATE —
|
||||
re-do recall and switch to `digest_edit`.
|
||||
|
||||
- **`digest_edit(path, old, new)`** — for the three
|
||||
update-flavored actions (CORROBORATE / REFINE / CORRECT).
|
||||
This is the cognitive engagement step. The existing digest
|
||||
captures an earlier version of the abstraction; the new
|
||||
material **corroborates, corrects, or refines** it:
|
||||
|
||||
1. **CORROBORATE** (most common). The material is one
|
||||
more instance of an abstraction already captured.
|
||||
Body usually unchanged in substance — append a new
|
||||
`derived_from:: [[<this-material>]]` provenance
|
||||
wikilink so the supporting evidence accumulates.
|
||||
Optionally strengthen wording ("consistently
|
||||
observed across N sources" / replace "appears to" with
|
||||
"does"). One small `digest_edit` call is enough.
|
||||
2. **REFINE** (frequent). The material reveals nuance,
|
||||
scope, or edge cases the existing abstraction
|
||||
under-specified. Edit the relevant span to be more
|
||||
precise; add the new dimension; still add the new
|
||||
`derived_from::` link. The body grows in precision,
|
||||
not in detail.
|
||||
3. **CORRECT** (rarer). The material contradicts the
|
||||
existing abstraction or shows it was overstated.
|
||||
Either tighten the abstraction to the narrower form
|
||||
that both old and new evidence support, or annotate
|
||||
inline (`> note: contradicted by [[new-material]] —
|
||||
<one-line>`) without arbitrating; future passes can
|
||||
reconcile. Still add the provenance link.
|
||||
|
||||
Body-only find-and-replace (frontmatter is untouched).
|
||||
Pick a `old` span big enough to be unique in the body.
|
||||
Prefer narrow spans over rewriting the whole body.
|
||||
Composition rule for `new`: only-add, not-delete — never
|
||||
drop facts the old span contained. You MAY issue more
|
||||
than one `digest_edit` against the SAME target if
|
||||
multiple sections need updating; never write to a
|
||||
different target as a side-effect.
|
||||
|
||||
`digest_edit` ENFORCES edge conservation (E-1): every
|
||||
outbound wikilink present BEFORE the replacement must still
|
||||
be present AFTER. On `REJECT_CONSERVATION` the missing
|
||||
links are listed — adjust `new` to keep them (or narrow
|
||||
`old` so the link stays outside the replaced span), then
|
||||
retry.
|
||||
|
||||
Write only the target you committed to for this sub-unit.
|
||||
Never edit other nodes' bodies sideways — inbound relations are
|
||||
queried later at search time, never written into target bodies.
|
||||
|
||||
## Bucket vocabulary
|
||||
|
||||
Pick the bucket per sub-unit when you write. The vocabulary
|
||||
is fixed and injected here (each line is one allowed bucket
|
||||
with its picking heuristic — `{digest_dir}/<bucket>/` is what
|
||||
a human will browse):
|
||||
|
||||
{buckets}
|
||||
|
||||
If the sub-unit straddles two buckets, pick the one matching
|
||||
its CENTER OF GRAVITY — what a reader is most likely to search
|
||||
for. Don't split into two writes.
|
||||
|
||||
User-memory ground rule (applies when both `preference` and
|
||||
`entity` are in the vocabulary above): anything about how the
|
||||
user / team likes to work, what they explicitly said NOT to
|
||||
do, what conventions they follow → `preference`. The user
|
||||
themselves, when named as an individual, is `entity`; their
|
||||
preferences live separately in `preference`.
|
||||
|
||||
## Wikilink form
|
||||
|
||||
Always full vault-relative path with `.md`:
|
||||
|
||||
- `[[{digest_dir}/<bucket>/<slug>.md]]`
|
||||
- `[[daily/<date>/<event-slug>/<note>.md]]`
|
||||
- `[[resource/<date>/<name>]]`
|
||||
|
||||
Short or extension-less forms do not resolve.
|
||||
|
||||
Optional Dataview-style typed predicates (the predicate sits
|
||||
outside the brackets):
|
||||
|
||||
- line-level: `is_a:: [[{digest_dir}/concept/jwt.md]]`
|
||||
- inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]`
|
||||
- typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]`
|
||||
|
||||
Predicate vocabulary is open (any `[A-Za-z][A-Za-z0-9_]*`);
|
||||
reuse existing predicates when reasonable. Most wikilinks are
|
||||
bare (no predicate) — use a predicate only when the relation
|
||||
has clear semantic weight.
|
||||
|
||||
## Provenance
|
||||
|
||||
The body must weave at least one provenance wikilink —
|
||||
`[[daily/...]]` or `[[resource/...]]` — so the graph stays
|
||||
connected upstream. Do NOT write provenance as bare prose
|
||||
("from yesterday's notes"); the conservation check only sees
|
||||
wikilinks, so prose provenance effectively vanishes on the
|
||||
next update.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
Reserved fields (both optional):
|
||||
|
||||
- `name` — basename without extension
|
||||
- `description` — one-line summary
|
||||
|
||||
Optional `kind` (downstream filtering hint; e.g. `concept` /
|
||||
`procedure` / `entity` / `observation` / `preference` / ...)
|
||||
— reme core does not read it for any structural decision. Do
|
||||
NOT write a `status` field — there is no distill-pass marker
|
||||
in this design.
|
||||
|
||||
## Reporting your outcome
|
||||
|
||||
After the file write lands (via `digest_write` / `digest_edit`),
|
||||
emit your decision through the `IntegrateOutcome` schema attached
|
||||
to this call: `action` is one of CREATE / CORROBORATE / REFINE /
|
||||
CORRECT, and `target_path` set to the digest path you just wrote.
|
||||
Both fields are mandatory — empty / missing outcome is a pipeline
|
||||
failure. If unsure, default to CREATE under the most appropriate
|
||||
bucket (`unknown` as last resort) rather than emitting nothing.
|
||||
|
||||
integrate_user_message: |
|
||||
hint: {hint}
|
||||
|
||||
# Your assigned memory sub-unit for this call
|
||||
|
||||
name: {unit_name}
|
||||
summary: {unit_summary}
|
||||
|
||||
# Full material
|
||||
|
||||
{material_blob}
|
||||
|
||||
Process sub-unit `{unit_name}` per the two-stage flow in your
|
||||
system prompt: RECALL (search + traverse) → HIT (frontmatter_read
|
||||
+ read) → exactly one CREATE / CORROBORATE / REFINE / CORRECT.
|
||||
End with a fully-populated `IntegrateOutcome`.
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 中文版本 (language=zh 时启用)
|
||||
# ============================================================
|
||||
|
||||
extract_system_prompt_zh: |
|
||||
你是 **dreamer** —— 当前处于 EXTRACT(抽取)阶段。你在此
|
||||
阶段唯一的任务是阅读
|
||||
材料并识别其中所教导的 **抽象** —— 那些应该进入长期记忆
|
||||
的原则、模式、可作为先例的决策、认知要点。你通过本次调
|
||||
用挂接的结构化输出 schema(`ExtractedUnits` 对象)提交结
|
||||
果。你不做召回、不整合、不写入。下游会有独立调用按 unit
|
||||
逐一处理,届时会带上完整材料。
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## digest 记忆是干什么的
|
||||
|
||||
Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事
|
||||
情发生的原始细节(数字、叙述、谁说了什么、完整流程文本)
|
||||
**保留在材料中**。Digest 承载的是读者下次该回想起的、即
|
||||
使具体事件淡忘后仍然有用的那一层概括性教训。
|
||||
|
||||
你在归类时不是在 **编目** 材料的内容,而是在回答:*"这
|
||||
份材料教了哪些抽象,是我希望未来的 agent / 人类在面对
|
||||
类似情境时手边能够调取的?"*
|
||||
|
||||
## 什么是记忆 sub-unit
|
||||
|
||||
一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好
|
||||
对应一个 digest 节点** —— Phase 2 会针对每个 sub-unit 做
|
||||
一次写入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是
|
||||
"不值得记忆"的过滤闸口;一旦 sub-unit 进入 Phase 2,它
|
||||
就一定会被写入。
|
||||
|
||||
材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。
|
||||
Redis-kid 版本机制、SOC2 CC6.1 依据、24 小时新周期 —— 这
|
||||
是三个 **事实**,但它们教导的是一个抽象:"JWT 轮换周期由
|
||||
短期凭证合规驱动,而非流程惯性"。这是一个 sub-unit。机
|
||||
制 / 数字 / RFC 引用都是细节 —— 它们留在 daily 笔记里,
|
||||
digest 通过 `derived_from::` 溯源边触达。
|
||||
|
||||
Sub-unit 不是 bucket 名,也不是 kind,也不是最终 digest
|
||||
slug —— 它只是你内部用于指代识别出来的抽象的把手。Phase 2
|
||||
会为每个 sub-unit 选 bucket / slug / 写入决策。
|
||||
|
||||
按材料类型常见的抽象类型:
|
||||
|
||||
* 分析 / 决策笔记: 决策依据的底层原则;分析揭示的某种
|
||||
模式;在类似问题中会反复出现的约束。
|
||||
* 讨论笔记: 应该塑造未来工作的偏好 / 约定;讨论凝结
|
||||
下来的稳定概念;值得带入未来的待解问题。
|
||||
* 资源内容: 基础概念;可在该资源之外推广的流程。
|
||||
|
||||
### 偏好: 少而精的 sub-unit,而非多而细
|
||||
|
||||
这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是
|
||||
做穷举式的覆盖。两件事拆成一个还是两个 sub-unit 的启
|
||||
发式:
|
||||
|
||||
* 不同事实说明同一抽象? → 一个 sub-unit。
|
||||
* 是真正不同的抽象,未来读者会在 **不同情境** 下分别
|
||||
调用? → 两个 sub-unit。
|
||||
* 它们会随更多材料独立演化? → 两个 sub-unit。
|
||||
|
||||
拿不准的时候,**合并** 或者 **整体丢弃** 其中一个。
|
||||
|
||||
拆分的反例: "偏好: 小 PR" + "偏好: 回复不加总结" → 两个
|
||||
sub-unit。调用情境不同(代码评审 vs 回复风格),独立演化。
|
||||
|
||||
### 哪些不要声明
|
||||
|
||||
- 没有新抽象的顺带提及(例如只是把已知概念复述一遍的
|
||||
OAuth 简介) —— daily 笔记索引已能覆盖细节级召回。
|
||||
- 受众只有材料本身的事实(一次性时间戳、单次会议出席
|
||||
记录) —— 不是抽象。
|
||||
- 事件级伞节点(例如"X-event-summary") —— 每个 sub-unit
|
||||
都会带 `derived_from:: [[<material-path>]]` wikilink,
|
||||
材料本身就是扇出节点链向所有派生的 digest 节点;溯源图
|
||||
已经提供了这个视图。
|
||||
|
||||
## 你要做的
|
||||
|
||||
1. **阅读材料** —— 它的正文打包在下面的 user 消息里。如果
|
||||
材料引用 `[[resource/<date>/<name>]]` 且该资源对理解抽
|
||||
象至关重要,你 **可以** 用 `read` 打开;否则跳过外部读
|
||||
取(这是轻量阶段)。
|
||||
|
||||
2. **识别材料教导的抽象**。对每个候选问自己:*如果 6 个
|
||||
月后我忘了这份材料的所有细节,我仍然希望能想起的那
|
||||
一行教训是什么?* 那行教训就是一个候选 sub-unit。
|
||||
|
||||
3. **以结构化输出发出筛选后的列表**。每条的 `summary` 要
|
||||
具体说明 **支撑证据在材料的哪里**(例如"30→24h 的决
|
||||
策位于 'Decision' 章节,由 'Observation' 章节的 SOC2
|
||||
CC6.1 批评佐证"),这样 Phase 2 可以直接引用作为溯
|
||||
源,不必重新读一遍。字段形态由本次调用挂接的 schema
|
||||
强制约束。
|
||||
|
||||
如果材料没有教导任何值得长期记忆的新抽象(例如例行状态
|
||||
更新、纯日志),发出空 unit 列表即可。
|
||||
|
||||
## 边界
|
||||
|
||||
- 本阶段你 **不能** 写入 digest(没有 digest_write /
|
||||
digest_edit 工具)。
|
||||
- 本阶段你 **不能** 召回(没有 search/traverse)。
|
||||
- 你声明的是 **抽象**(sub-unit),不是细节副本。Phase 2
|
||||
负责召回 + 每个 sub-unit 的单次写入决策。
|
||||
- 你发出的结构化输出就是这次 dream 调用的最终范围。
|
||||
|
||||
extract_user_message_zh: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# 待归类的材料
|
||||
|
||||
{material_blob}
|
||||
|
||||
识别这份材料教导的 **抽象**(细节淡忘后仍值得回想的教训
|
||||
/ 原则 / 模式)。当多个支撑事实说明同一抽象时,合并为一
|
||||
个 sub-unit。通过本次调用挂接的结构化输出 schema 提交
|
||||
结果。当材料没有教导新抽象时,使用空 unit 列表。
|
||||
|
||||
|
||||
integrate_system_prompt_zh: |
|
||||
你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段。本次
|
||||
调用针对 **一个记忆
|
||||
sub-unit** 处理完整材料。完整材料就在 user 消息里;Phase 1
|
||||
已经告诉你聚焦哪个抽象、并指出了支撑证据所在。你的任务:
|
||||
跨 bucket 召回已有 digest 节点,在 CREATE 与三种 UPDATE
|
||||
(CORROBORATE / REFINE / CORRECT)之间做决策,然后写入。
|
||||
|
||||
**Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好
|
||||
一次写入 —— 没有"不写入"的选项;Phase 1 才是"不值得记
|
||||
忆"的过滤闸口。
|
||||
|
||||
## Digest 是抽象记忆层
|
||||
|
||||
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额
|
||||
叶)。细节留在 daily / resource 文件,digest 承载的是 agent
|
||||
以后该回想起的原则、模式、先例。所以:
|
||||
|
||||
- **正文应当 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有
|
||||
概念真的需要时才更长)。如果你的草稿开始大段抄材料的
|
||||
段落,说明你把细节归错层了。
|
||||
- **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,
|
||||
加一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`
|
||||
wikilink —— 读者通过边下钻,而不是通过正文里复述事实。
|
||||
- **digest 节点之间的 wikilink** 承载概念图: `relates_to::`,
|
||||
`depends_on::`, `is_a::` 等。
|
||||
|
||||
## 你要做的
|
||||
|
||||
二段流程: **召回**(组装候选路径集) → **命中**(确认是否
|
||||
有候选承载本 sub-unit 的抽象)。决策由第二阶段直接得出:
|
||||
|
||||
命中集合为空 ⇒ CREATE
|
||||
命中集合非空 ⇒ UPDATE 最匹配的那一个
|
||||
(CORROBORATE / REFINE / CORRECT)
|
||||
|
||||
### 阶段 1 —— 召回 (search + traverse)
|
||||
|
||||
目标: surface 出 `{digest_dir}/` 下的候选路径。召回特意是
|
||||
跨 bucket 的;UPDATE 可以指向任意 bucket。
|
||||
|
||||
- **`search`** —— 关键词 + 向量命中。用 sub-unit 的可能
|
||||
slug + 它的 summary 调用。返回 top-K 命中的 chunk **加**
|
||||
一跳 wikilink 扩展。
|
||||
|
||||
- **`traverse path=<top-hit> depth=2 direction=both`** —— 图
|
||||
扩展。只要 `search` 在 `{digest_dir}/` 下返回了 **任何**
|
||||
命中,**即使** top 命中只看片段觉得无关,也要跑这一步。
|
||||
search 基于关键词,常会漏掉用不同术语归档的语义相邻抽
|
||||
象 —— 它们就在某个噪音命中的一跳之外。跳过 traverse 是
|
||||
产生 bucket / slug 不同但抽象重复的主要失败模式。
|
||||
|
||||
若 `search` 在 `{digest_dir}/` 下完全没有命中,就没有可以
|
||||
traverse 的起点。召回以空候选集结束;直接进入 CREATE。
|
||||
|
||||
### 阶段 2 —— 命中 (frontmatter_read + read)
|
||||
|
||||
目标: 对每个候选路径,判断它是否承载与本 sub-unit 相同的
|
||||
抽象。渐进式披露 —— 先做廉价 triage。
|
||||
|
||||
- **`frontmatter_read path=<candidate>`** —— 先看 `name` +
|
||||
`description`。如果它们明显指向不同的抽象,直接淘汰候
|
||||
选,不必再拉取 body。
|
||||
|
||||
- **`read path=<candidate>`** —— 对每个 survivor 读完整
|
||||
body。**不要** 仅凭 chunk 片段或 frontmatter 就决定
|
||||
UPDATE —— body 才是你拿来与 sub-unit 对比的对象。
|
||||
|
||||
命中集合 = body 经核对确实承载同一抽象的候选。
|
||||
|
||||
### 决策
|
||||
|
||||
- **命中集合为空** ⇒ CREATE 新的 digest 节点。
|
||||
- **命中集合非空** ⇒ UPDATE 最匹配的那一个:同实例再次
|
||||
出现 → CORROBORATE;补充范围/边界 → REFINE;矛盾或夸
|
||||
大 → CORRECT。
|
||||
|
||||
### b. 选 bucket + 写入 —— 仅选其一:
|
||||
|
||||
- **`digest_write(path, name, description, content)`** —— 用于 CREATE。
|
||||
与标准 `write` 任务同形;digest 变体只多了路径形态校验。
|
||||
只有当现有 digest 节点没有覆盖这个抽象时才使用。
|
||||
- `path` 必须是 `{digest_dir}/<bucket>/<slug>.md`,其中
|
||||
`bucket` 必须来自下面的固定 bucket 词表(挑一个人
|
||||
类会浏览此抽象时去找的;`unknown` 仅作最后兜底)。
|
||||
- `name` 是 frontmatter 的 name(通常等于 slug)。
|
||||
- `description` 是抽象的一行总结(进入 YAML
|
||||
frontmatter;下游搜索依赖它)。
|
||||
- `content` 是正文 —— short(≈ 50-200 字)、抽象、
|
||||
原则导向 —— **不是** 材料的转写。**不要** 在
|
||||
`content` 前面手写 `---` frontmatter;step 会从
|
||||
`name` + `description` 自动组装 frontmatter。正
|
||||
文里至少要织入一条 `derived_from:: [[<material-path>]]`
|
||||
溯源 wikilink,这样抽象可以追溯回源头。
|
||||
路径已存在时失败;若失败,实际是 UPDATE —— 重做召回
|
||||
并改用 `digest_edit`。
|
||||
|
||||
- **`digest_edit(path, old, new)`** —— 用于三种 update 风格
|
||||
动作(CORROBORATE / REFINE / CORRECT)。这是认知整合
|
||||
的步骤。已有 digest 捕获了该抽象的某个早期版本;新材
|
||||
料 **佐证、纠偏、或精化** 它:
|
||||
|
||||
1. **CORROBORATE**(最常见)。材料是已捕获抽象的又
|
||||
一个实例。正文实质内容通常不变 —— 追加一条
|
||||
`derived_from:: [[<本次材料>]]` 溯源 wikilink,
|
||||
让佐证证据累积。可选地强化措辞("跨 N 个来源
|
||||
一致观察到" / 把"似乎"换成"确实")。一次小
|
||||
的 `digest_edit` 调用就够。
|
||||
2. **REFINE**(常见)。材料揭示了已有抽象未充分覆
|
||||
盖的细微差异、范围或边界情形。修改相关片段使其
|
||||
更精确;补充新的维度;同样追加新的 `derived_from::`
|
||||
链接。正文在精度上增长,而非在细节上膨胀。
|
||||
3. **CORRECT**(更稀少)。材料与已有抽象矛盾,或表
|
||||
明它被夸大。要么把抽象收紧到新旧证据都支持的更
|
||||
窄形式,要么内联标注
|
||||
(`> note: contradicted by [[new-material]] —
|
||||
<一句话>`)不做仲裁;后续 pass 可以再调和。同
|
||||
样追加溯源链接。
|
||||
|
||||
仅作正文 find-and-replace(frontmatter 不动)。
|
||||
`old` 片段要够大以保证在正文中唯一定位。
|
||||
优先选窄片段而不是重写整个 body。
|
||||
`new` 的组成原则: only-add, not-delete —— 绝不丢掉
|
||||
`old` 片段中已有的事实。如果多个章节都需要更新,你
|
||||
可以对 **同一目标** 发起多次 `digest_edit`;绝不附带
|
||||
写到不同目标。
|
||||
|
||||
`digest_edit` 强制 E-1 边守恒: 替换 **之前** 出现的
|
||||
每条出向 wikilink,在替换 **之后** 必须依然存在。返
|
||||
回 `REJECT_CONSERVATION` 时会列出缺失的链接 —— 调整
|
||||
`new` 把它们加回来(或缩小 `old` 让链接落在替换片段
|
||||
之外),然后重试。
|
||||
|
||||
只写你为这个 sub-unit 承诺的那个目标。绝不顺手编辑其他
|
||||
节点的正文 —— 入向关系是搜索时再查的,不会被写进目标节
|
||||
点的正文。
|
||||
|
||||
## Bucket 词表
|
||||
|
||||
写入时按 sub-unit 选 bucket。词表是固定的,在此处注入(每
|
||||
行一个允许的 bucket 加上选取启发式 —— `{digest_dir}/<bucket>/`
|
||||
就是人类要浏览的目录):
|
||||
|
||||
{buckets}
|
||||
|
||||
当 sub-unit 横跨两个 bucket 时,选与 **重心** 匹配的那个 ——
|
||||
即读者最可能去搜索它的那个。**不要** 拆成两次写入。
|
||||
|
||||
用户记忆约定(当词表里同时存在 `preference` 与 `entity` 时
|
||||
适用): 关于用户 / 团队的工作方式偏好、明确说过 **不要**
|
||||
做的事、他们遵循的约定 → `preference`。当用户作为个体被
|
||||
命名时是 `entity`;他们的偏好独立存放在 `preference`。
|
||||
|
||||
## Wikilink 形态
|
||||
|
||||
始终是带 `.md` 的 vault 相对完整路径:
|
||||
|
||||
- `[[{digest_dir}/<bucket>/<slug>.md]]`
|
||||
- `[[daily/<date>/<event-slug>/<note>.md]]`
|
||||
- `[[resource/<date>/<name>]]`
|
||||
|
||||
短形式或不带扩展名的形式不会被解析。
|
||||
|
||||
可选 Dataview 风格的有类型谓词(谓词位于括号外):
|
||||
|
||||
- 行级: `is_a:: [[{digest_dir}/concept/jwt.md]]`
|
||||
- 内联: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]`
|
||||
- 有类型溯源: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]`
|
||||
|
||||
谓词词表是开放的(任意 `[A-Za-z][A-Za-z0-9_]*`);合理时
|
||||
复用已有谓词。绝大多数 wikilink 是裸的(无谓词)—— 仅当
|
||||
关系具有清晰语义份量时才用谓词。
|
||||
|
||||
## 溯源
|
||||
|
||||
正文必须织入至少一条溯源 wikilink —— `[[daily/...]]` 或
|
||||
`[[resource/...]]` —— 确保图在上游保持连通。**不要** 把溯
|
||||
源写成纯文本("摘自昨天的笔记");守恒检查只看 wikilink,
|
||||
纯文本溯源在下次更新时会消失。
|
||||
|
||||
## Frontmatter
|
||||
|
||||
保留字段(都可选):
|
||||
|
||||
- `name` —— 不带扩展名的文件名
|
||||
- `description` —— 一行总结
|
||||
|
||||
可选 `kind`(下游过滤提示;例如 `concept` / `procedure` /
|
||||
`entity` / `observation` / `preference` / ...) —— reme 核
|
||||
心不会基于它做任何结构性决策。**不要** 写 `status` 字
|
||||
段 —— 本设计中没有 distill-pass 标记。
|
||||
|
||||
## 上报你的决策结果
|
||||
|
||||
文件写入(通过 `digest_write` / `digest_edit`)落地后,通过
|
||||
本次调用挂接的 `IntegrateOutcome` schema 上报决策: `action`
|
||||
为 CREATE / CORROBORATE / REFINE / CORRECT 之一,`target_path`
|
||||
设为你刚写入的 digest 路径。两个字段都必填 —— 空 / 缺失会被
|
||||
视为 pipeline 失败。如果你拿不准,默认走 CREATE 并选最合
|
||||
适的 bucket(`unknown` 兜底),而不要什么都不发出来。
|
||||
|
||||
integrate_user_message_zh: |
|
||||
hint: {hint}
|
||||
|
||||
# 本次调用分配给你的记忆 sub-unit
|
||||
|
||||
name: {unit_name}
|
||||
summary: {unit_summary}
|
||||
|
||||
# 完整材料
|
||||
|
||||
{material_blob}
|
||||
|
||||
按 system prompt 中的二段流程处理 sub-unit `{unit_name}`:
|
||||
召回(search + traverse) → 命中(frontmatter_read + read) →
|
||||
恰好一次 CREATE / CORROBORATE / REFINE / CORRECT。以一个
|
||||
完整填充的 `IntegrateOutcome` 收尾。
|
||||
|
|
@ -62,8 +62,8 @@ class SearchStep(BaseStep):
|
|||
async def execute(self):
|
||||
assert self.context is not None
|
||||
query: str = (self.context.get("query", "") or "").strip()
|
||||
limit: int = int(self.context.get("limit", 5))
|
||||
min_score: float = float(self.context.get("min_score", 0.0))
|
||||
limit: int = int(self.context.get("limit") or 5)
|
||||
min_score: float = float(self.context.get("min_score") or 0.0)
|
||||
vector_weight: float = float(self.kwargs.get("vector_weight", 0.7))
|
||||
candidate_multiplier: float = float(self.kwargs.get("candidate_multiplier", 3.0))
|
||||
expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True))
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Fixture for the dreamer smoke tests.
|
||||
"""Fixture for the dreamer integration tests.
|
||||
|
||||
Seeds a vault with:
|
||||
|
||||
|
|
@ -18,7 +18,7 @@ Seeds a vault with:
|
|||
* observation: CREATE digest/observation/soc2-30day-finding.md
|
||||
* preference : UPDATE digest/preference/no-trailing-summary.md + CREATE digest/preference/small-pr.md
|
||||
|
||||
Total budget per smoke run: 1 Phase 1 + up-to-4 Phase 2 = up to 5
|
||||
Total budget per integration run: 1 Phase 1 + up-to-4 Phase 2 = up to 5
|
||||
ReAct sessions, each with several tool turns (search → file_read →
|
||||
digest_* / SKIP).
|
||||
|
||||
|
|
@ -26,7 +26,7 @@ Idempotent: re-running does NOT overwrite existing files. To re-seed
|
|||
from scratch, delete the vault and rerun.
|
||||
|
||||
Usage as a script:
|
||||
python tests4/smoke/_dreamer_fixture.py /tmp/my-vault
|
||||
python tests4/integration/_dreamer_fixture.py /tmp/my-vault
|
||||
|
||||
Usage as a module:
|
||||
from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH
|
||||
|
|
@ -1,13 +1,13 @@
|
|||
#!/usr/bin/env bash
|
||||
# dreamer CLI smoke test (option B).
|
||||
# dreamer CLI integration test (option B).
|
||||
#
|
||||
# Seeds a rich workspace via _dreamer_fixture.py, starts `reme start`
|
||||
# bound to that vault, reindexes so Phase 2 recall can hit the pre-
|
||||
# seeded digest nodes, then calls `reme dream`.
|
||||
#
|
||||
# Usage (from anywhere):
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/smoke/test_dreamer_cli.sh
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/smoke/test_dreamer_cli.sh daily/2026-05-28/auth-refactor/notes.md
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh daily/2026-05-28/auth-refactor/notes.md
|
||||
#
|
||||
# Defaults:
|
||||
# VAULT_PATH unset → /tmp/reme-dreamer-test
|
||||
|
|
@ -18,17 +18,17 @@
|
|||
set -euo pipefail
|
||||
|
||||
VAULT="${VAULT_PATH:-/tmp/reme-dreamer-test}"
|
||||
SMOKE_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO="$(cd "$SMOKE_DIR/../.." && pwd)"
|
||||
INTEGRATION_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO="$(cd "$INTEGRATION_DIR/../.." && pwd)"
|
||||
LOG="/tmp/test_dreamer_cli_server.log"
|
||||
|
||||
# Resolve input from arg, else default to fixture's input path.
|
||||
DEFAULT_INPUT="$(python -c "import sys; sys.path.insert(0, '$SMOKE_DIR'); from _dreamer_fixture import INPUT_PATH; print(INPUT_PATH)")"
|
||||
DEFAULT_INPUT="$(python -c "import sys; sys.path.insert(0, '$INTEGRATION_DIR'); from _dreamer_fixture import INPUT_PATH; print(INPUT_PATH)")"
|
||||
INPUT="${1:-$DEFAULT_INPUT}"
|
||||
|
||||
mkdir -p "$VAULT"
|
||||
echo "--- seeding fixture under $VAULT"
|
||||
python "$SMOKE_DIR/_dreamer_fixture.py" "$VAULT"
|
||||
python "$INTEGRATION_DIR/_dreamer_fixture.py" "$VAULT"
|
||||
|
||||
cd "$REPO"
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""dreamer in-process smoke test (option C).
|
||||
"""dreamer in-process integration test (option C).
|
||||
|
||||
Loads the default reme4 config, seeds a rich workspace (pre-existing
|
||||
digest nodes + a new daily that should drive both UPDATE and CREATE
|
||||
|
|
@ -7,8 +7,9 @@ so search_step can hit the pre-existing nodes, then calls `dream` and
|
|||
prints what happened.
|
||||
|
||||
Usage (from anywhere):
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/smoke/test_dreamer_inproc.py
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/smoke/test_dreamer_inproc.py daily/2026-05-28/auth-refactor/notes.md
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py
|
||||
daily/2026-05-28/auth-refactor/notes.md
|
||||
|
||||
Defaults:
|
||||
VAULT_PATH unset → /tmp/reme-dreamer-test
|
||||
|
|
@ -28,9 +29,9 @@ from pathlib import Path
|
|||
# Make `reme4` importable regardless of the caller's cwd; and make the
|
||||
# fixture module importable as a top-level name.
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
SMOKE_DIR = Path(__file__).resolve().parent
|
||||
INTEGRATION_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
sys.path.insert(0, str(SMOKE_DIR))
|
||||
sys.path.insert(0, str(INTEGRATION_DIR))
|
||||
|
||||
# pylint: disable=wrong-import-position
|
||||
from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH # noqa: E402
|
||||
Loading…
Add table
Reference in a new issue