diff --git a/docs4/auto_cognition_design.md b/docs4/auto_cognition_design.md new file mode 100644 index 00000000..445982c1 --- /dev/null +++ b/docs4/auto_cognition_design.md @@ -0,0 +1,330 @@ +# auto-cognition 设计(顶层:心智循环) + +> 本文档:reme4 中**长期记忆系统**的顶层认知模型 —— 把 agent 的记忆生命周期类比人类睡眠/觉醒回路,推导出**三阶段分工**与**15 维能力清单**。 +> +> **三阶段实现各有专属文档**: +> - Stage 1 写入(REM 重放抽象) → `auto_dream_design.md` +> - Stage 2 巩固(NREM 深度整合) → `auto_consolidate_design.md` +> - Stage 3 检索(觉醒态提取) → `auto_recall_design.md` +> +> 配套阅读: +> - `auto_memory_design.md`:入流端(daily 写入),与 cognition 平行 —— cognition 负责"已落地后的认知循环",memory 负责"经历落地" +> - `structure.md` §4(retrieve 三种问法) +> +> **核心立场**: +> - 长期记忆不是"存 + 取"两个动作,是**写入 → 巩固 → 提取**的循环 —— 三段时间尺度不同(同步 / 周期 / 同步),设计形态不同 +> - vault 是**事实层**,只承载经过 LLM 写入认证的关系;`meta/` 是**派生层**,承载概率推断的统计信号 +> - 任一阶段独立演化,任一信号缺失系统降级而不崩 + +--- + +## 0. 心智循环:reme 的认知模型 + +agent 的长期记忆系统在概念上对应人脑的**海马—皮层回路 + 睡眠—觉醒周期**: + +``` + ┌─────────────────────────────────┐ + │ 外部经验(daily / resource) │ + └─────────────┬───────────────────┘ + │ (auto-memory 写 daily) + ▼ + ┌─────────────────────────────────────────────────┐ + │ │ + │ ┌────────────────┐ 抽象 / 关系编织 │ + │ │ Stage 1 │ ◄─ 类比 REM 睡眠 │ + │ │ auto-dream │ "重放 + 写进 schema" │ + │ └───────┬────────┘ │ + │ │ 写 vault(digest body + wikilink) │ + │ ▼ │ + │ ┌────────────────┐ │ + │ │ vault(事实) │ │ + │ └───────┬────────┘ │ + │ │ 只读 │ + │ ▼ │ + │ ┌────────────────┐ 长期组织 / 派生指标 │ + │ │ Stage 2 │ ◄─ 类比 NREM 慢波睡眠 │ + │ │ auto-consol- │ "巩固 + 修剪 + 集群" │ + │ │ idate │ │ + │ └───────┬────────┘ │ + │ │ 写 meta/ + audit/(派生层) │ + │ ▼ │ + │ ┌────────────────┐ │ + │ │ meta(派生) │ │ + │ └───────┬────────┘ │ + │ │ 只读 │ + │ ▼ │ + │ ┌────────────────┐ query → 答案合成 │ + │ │ Stage 3 │ ◄─ 类比觉醒态 cue retrieval│ + │ │ auto-recall │ "融合 + pattern complete"│ + │ └───────┬────────┘ │ + │ │ │ + └───────────┼─────────────────────────────────────┘ + │ 召回结果给 agent + ▼ + ┌─────────────────────────────────┐ + │ agent query │ + └─────────────────────────────────┘ +``` + +**心智循环回答四个根本问题**: + +| 问题 | 谁回答 | +|---|---| +| 我经历过什么? | auto-memory(daily 入流) | +| 我从中学到什么? | Stage 1 — auto-dream | +| 这些知识如何长期组织? | Stage 2 — auto-consolidate | +| 我需要时如何调用? | Stage 3 — auto-recall | + +memory 负责"经历落地",cognition 三阶段负责"已落地经历的认知循环"。 + +--- + +## 1. 三阶段全景 + +| 阶段 | 神经科学类比 | 时间尺度 | 改 vault | 实现归属 | +|---|---|---|---|---| +| **Stage 1 dream** | REM 重放抽象 | 同步(随入流即跑) | 是(写 digest body) | `auto_dream_design.md` | +| **Stage 2 consolidate** | NREM 深度巩固 | 周期 / idle(daily / weekly)| **否**(写 `meta/` + `audit/`)| `auto_consolidate_design.md` | +| **Stage 3 recall** | 觉醒态 cue retrieval | 同步(query 触发) | 否(只读;唯一对外写是 `meta/access_log.json`)| `auto_recall_design.md` | + +**关键的不对称**: +- 写入与检索是**同步**的(用户 / agent 等待),巩固是**离线**的(idle / 周期) +- 改 vault 的资格被严格限制在 **dream + consolidate 中的 split** —— 其它阶段全只读 +- 三阶段时间尺度差三个数量级,这是设计形态(同步 vs 异步 vs idle)的根本来源 + +--- + +## 2. 系统级能力(贯穿三阶段) + +不属任何单阶段,但任一阶段不能违反: + +| 能力 | 含义 | +|---|---| +| **事实层 vs 派生层分离** | vault 只承载经 LLM 写入认证的关系(显式 wikilink);`meta/` 承载概率推断的派生指标(community / recency / archived);两者绝不混同 | +| **不变量守恒** | F-invariants(0 文件移动 / 改正文限定 subject / wikilink 是 body 一部分)+ E-invariants(边守恒 E-1/E-2/E-3)横跨三阶段;详 `auto_dream_design.md` §4.3-§4.4 | +| **阶段独立演化** | 任一阶段算法升级不破坏其它阶段(community 算法换 → dream 不变;打分公式调 → consolidate 不变) | +| **缺失即降级** | 任一派生信号缺失,系统降级而不崩;冷启动可用 | +| **全程可审计** | 每阶段产 audit / report / log,人 / agent 可检视追溯 | + +--- + +## 3. Stage 1 — auto-dream:经验 → 抽象 + +**类比**:REM 睡眠的记忆重放与抽象提炼。脑在做梦时把白天事件拆解、重组,提取出可泛化的模式,登记进皮层 schema。 + +**根本目的**:把"原始经历"转化为"长期值得调取的教训",同时把它编织进已有知识图谱。 + +### 3.1 五个能力维度 + +逻辑递进 —— 输入 → 抽象 → 整合 → 编织 → 写入: + +| # | 能力 | 它在问什么 | 失效后果 | +|---|---|---|---| +| 1 | **抽象判断**(gate) | 这段材料里有"值得长期记住"的东西吗? | 噪声进 vault / 只蒸馏不抽象 | +| 2 | **经验重放**(召回) | 这个抽象在已有记忆里**已经存在**吗?以什么形式? | 重复节点 / 错过整合机会 | +| 3 | **整合决策** | 创建新节点,还是丰富已有节点?若已有 —— 是再次印证 / 精化范围 / 修正错误? | 已有信息丢失 / 错误没纠正 | +| 4 | **关系编织** | 这个抽象与谁有关系?谁是它的来源? | wikilink 缺失,后续 retrieve 漏召 | +| 5 | **写入安全** | 写入会不会破坏 vault 既有事实?并发冲突如何处理? | 边丢失 / race condition | + +### 3.2 关键定性 + +- dream 是 vault 的**唯一写者**(在 cognition 三阶段里;memory 写 daily 不算) +- **写入瞬间是关系建立的唯一可信时机** —— 错过的关系不靠后台扫回(那不是 consolidate 的工作) +- 一次写入,所有未来检索受益(持久化优于实时计算) + +详细机制见 `auto_dream_design.md`。 + +--- + +## 4. Stage 2 — auto-consolidate:抽象 → 网络 + +**类比**:NREM 慢波睡眠的系统巩固 + 突触代谢稳态。脑在深睡时把分散事件融入 schema、修剪弱连接、把长期不用的记忆淡出意识可达范围。 + +**根本目的**:跨时间累积地把 vault 从"一堆节点"组织成"有结构、有权重、有时效的网络",但**只产派生信号,不污染事实层**。 + +### 4.1 五个能力维度 + +按作用尺度从微观到宏观: + +| # | 能力 | 作用尺度 | 类比 | 输出形态 | +|---|---|---|---|---| +| 1 | **结构维护** | 节点级 | 海马表征过密 → 分化新单元 | 改 vault(split,唯一例外)| +| 2 | **跨节点关系发现** | 节点对级 | 多次睡眠中识别"同一件事" → schema | `audit/` 报告 | +| 3 | **主题集群形成** | 子图级 | 皮层网络的功能性分区 | `meta/communities.json` | +| 4 | **时效性管理** | 节点级 / 时间维度 | 突触代谢稳态 + 遗忘 | `meta/access_log.json` + `meta/archived.json` | +| 5 | **健康监控** | 系统级 | 神经环路诊断 | 告警 / 严重告警 | + +### 4.2 关键定性 + +- consolidate 是**纯只读 + 派生写**(读 vault,写 `meta/` + `audit/`) +- **唯一例外是 split** —— 改 vault 的维护任务,但触发严格(D3 inline 写后)且只改自身负责的 parent + children +- **关系判断有错率 → 报告优先,人/agent 介入,不主动合并**(夸大置信度的代价是污染事实层) +- 离线 / 周期 / idle —— 与前台不抢资源;失败不影响主流程,下次重跑 + +详细机制见 `auto_consolidate_design.md`。 + +--- + +## 5. Stage 3 — auto-recall:网络 → 答案 + +**类比**:觉醒态的 cue-driven retrieval + pattern completion。脑接到 query,激活相关皮层模式,补全成完整答案;同时召回过程本身强化被用到的记忆痕迹。 + +**根本目的**:接到当前 query 时,从 vault + 派生信号合成最相关的过去经验 —— 既要**覆盖率**(不漏)也要**信噪比**(不冗余)。 + +### 5.1 五个能力维度 + +按召回流程从输入到输出: + +| # | 能力 | 它在解决什么 | +|---|---|---| +| 1 | **多路召回** | 不同问法走不同算子(state / semantic / topological 三分立);agent 自选,不强加聚合 verb | +| 2 | **多信号融合** | 单一文本相似度不够 —— 还要节点权威性 / 主题集群 / 时效性;乘法融合 | +| 3 | **信噪比管理** | 节点级去重 + 节点级 surface(frontmatter 一同呈现)+ multi-hop 可控展开 + 冷藏过滤 | +| 4 | **召回反馈** | 被命中的节点 → 写访问日志 → 影响下次 recency / archived 判定 | +| 5 | **鲁棒降级** | 派生信号缺失 → 退到基础召回;version 不兼容 → warning + 跳过该因子 | + +### 5.2 关键定性 + +- recall 是**只读** —— 唯一对外写入是 `meta/access_log.json`(经 ring buffer + consolidate 聚合) +- recall **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法分别由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖 +- 默认路径 **0 LLM 调用**(信号都是离线维护好的);LLM rerank / query rewrite 是 SDK 上层选项 + +详细机制见 `auto_recall_design.md`。 + +--- + +## 6. 能力地图(横切视角) + +15 维按"作用对象"重排,可以看到三阶段如何分工: + +| 作用对象 | dream(写入) | consolidate(巩固)| recall(检索)| +|---|---|---|---| +| **节点(单个)** | 1 抽象判断 / 3 整合决策 / 5 写入安全 | 1 结构维护(split) | 3 信噪比(节点级合并/surface) | +| **节点对 / 关系** | 4 关系编织(wikilink) | 2 跨节点关系发现(dups 报告) | (消费已有边,不产新关系) | +| **子图 / 集群** | 2 经验重放(召回邻居) | 3 主题集群形成(community)| 2 多信号融合(community boost) | +| **时间维度** | (写入瞬间) | 4 时效性管理(decay / archived)| 4 召回反馈(access log)| +| **系统健康** | 5 守恒校验 | 5 健康监控(D1 / D10) | 5 鲁棒降级 | +| **入口形态** | 异步 fan-out per sub-unit | 周期 batch / idle | 同步 query response | + +**几个观察**: +- "节点对 / 关系"列在 recall 是空 —— recall 不产新关系,只用已有边(避免 query-time 高成本推断) +- "时间维度"行 dream 缺位 —— 写入瞬间无"时间维度"概念(那是 consolidate 后续才能提取的统计) +- 每行至少有一个阶段负责 —— 没有能力被全阶段忽略 + +--- + +## 7. 跨阶段不变量 + +所有阶段共同遵守的硬约束。任何阶段越界 = 设计错误。 + +### 7.1 F-invariants(继承 `auto_dream_design.md` §4.3) + +| # | 约束 | 跨阶段含义 | +|---|---|---| +| F-1 | 0 文件移动 | 没有任何阶段可以 move 文件;rename 走 `wikilink_handler.retarget_links` 显式路径 | +| F-2 | 改正文限定 subject | dream 改 subject body / consolidate split 改 parent + children body;**recall 绝不改任何 body** | +| F-3 | maintainer 只做 split | consolidate 内的结构维护只做 split;无 merge / dissolve / re-edge | +| F-10 | inbound 不动 | split 后外部 wikilink 仍指 parent,不强制重定向 | +| F-11 | wikilink 是 body 一部分 | 没有"独立的边";所有关系变化是 body 编辑副作用 | + +### 7.2 E-invariants(边守恒) + +- E-1:dream update 出边 ⊇ 原出边 +- E-2:split 后 `(parent_new ∪ ∪children_outbound) ⊇ parent_old` +- E-3:inbound wikilink split 时不动 + +**recall 不写 body** → E-* 与之无关;但 recall 看到的 wikilink 图永远是 dream / split 守恒后的状态。 + +### 7.3 派生信号边界 + +- **consolidate / recall 不写 vault** —— 关系判断、活跃度统计、社区划分都是概率推断,不污染事实层 +- **`meta/*.json` 不被 retrieve 召回** —— 只作权重信号,不进入"召回结果"集合 +- **audit/ 不被自动消费** —— 报告永远等待人 / agent 介入,不闭环回写 + +--- + +## 8. 跨阶段数据流(契约总览) + +``` +┌──────────────┐ wikilink ┌──────────────┐ +│ auto-dream │─落 body──►│ vault/ │ +│ (Stage 1) │ │ (事实层) │ +└──────────────┘ └──────┬──────┘ + │ 只读 + ▼ + ┌──────────────────┐ + │ auto-consolidate │ + │ (Stage 2) │ + └─┬────────┬───────┘ + │ │ + meta/ 元数据───┘ └─── audit/ 报告 + (派生层) (人工介入) + │ + │ 只读 + ▼ + ┌──────────────┐ + │ auto-recall │ ◄─ user query + │ (Stage 3) │ + └──────┬───────┘ + │ 命中钩子(异步) + ▼ + meta/access_log.json + (recall 唯一对外写入,经 consolidate 聚合) +``` + +| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 | +|---|---|---|---|---| +| **vault wikilink** | `digest/**.md` body | dream / split | recall(图遍历) | — | +| **dups 报告** | `audit//auto_link_dups.md` | consolidate | 人 / agent | — | +| **communities** | `meta/communities.json` | consolidate | recall | 不做同社区 boost | +| **access log** | `meta/access_log.json` | recall(写命中) + consolidate(聚合) | recall(读 recency)| recency_factor = 1.0 | +| **archived list** | `meta/archived.json` | consolidate | recall(默认过滤)| 不过滤 | +| **centrality** | `file_graph` 反向索引(实时,不存)| 自动 | recall(O(1) 查) | — | + +**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。 + +**冷启动**:`meta/` 为空 → recall 仍能跑(base + centrality + 图)→ 排序略弱不崩。 + +--- + +## 9. 系统级断言(把"要什么"提炼到 5 条) + +1. **抽象与事实分层** —— vault 是经 LLM 写过的事实;`meta/` 是统计 / 算法的派生;两者绝不混同 + +2. **关系建立的时机集中在写入瞬间** —— dream 写入是关系唯一可信来源;consolidate 不补 vault 关系,recall 不预存关系矩阵 + +3. **维护是离线的派生劳动,不是补救** —— consolidate 不修 dream 的疏漏(那叫返工),它做的是 dream 不擅长的事(全局视角 / 统计视角 / 时间视角) + +4. **检索是融合,不是检索** —— recall 的价值不在"找文本相似",而在"把文本 / 图 / 时效 / 权威多个独立信号合成一个答案" + +5. **整个心智循环可降级** —— 任一阶段失效或失准,整个系统降级而不崩;冷启动有意义;dogfooding 可演进 + +--- + +## 10. 与 auto-memory 的边界 + +auto-memory 写入的 daily event 节点也是图的一部分(承载 daily → digest 的 `derived_from::` 边)。但 daily 节点**不参与 cognition 三阶段的全部改造**: + +| cognition 阶段 | 是否触及 daily | +|---|---| +| **dream** | 只读(作为入流之一) | +| **consolidate** | 不参与 dups / community / decay(daily 是时间索引,本质不去重 / 不冷藏) | +| **recall** | 三层并行召回时 daily 也参与命中(`structure.md` R-2 默认 `digest > daily > resource`) | + +**关键约束**:cognition 三阶段任何子阶段都**不改写 daily**(无写回路径);daily 由 auto-memory 写完即只读。 + +--- + +## 11. 演进 / 待补 + +**当前实现状态**: +- ✅ Stage 1 dream 已实现并跑通(`reme4/steps/evolve/dream/`) +- ⏳ Stage 2 consolidate split 部分将实现;dups / community / decay / archived 待实现 +- ⏳ Stage 3 recall 增强未实现(当前 search.py 已有 vector + keyword + RRF + 一跳 expand) + +**顶层级演进议题**(不属任何单阶段): +- ⏳ **能力成熟度路标** —— 把 15 个能力维度按 M0(必须)/ M1(期望)/ M2(演进)分级 +- ⏳ **跨阶段集成测试** —— vault 从空到充实的端到端 dogfooding,验证三阶段配合是否符合"心智循环"预期 +- ⏳ **可观测性聚合** —— 三阶段各自的 audit / log 现在分散;是否需要统一的 cognition 健康面板 + +各阶段实现进度详见各自文档的"下一步"章节。 diff --git a/docs4/auto_consolidate_design.md b/docs4/auto_consolidate_design.md new file mode 100644 index 00000000..e9d0d170 --- /dev/null +++ b/docs4/auto_consolidate_design.md @@ -0,0 +1,726 @@ +# auto-consolidate 设计(Stage 2 巩固:主动解决 vault 长期演化的实际问题) + +> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 2 — 巩固阶段** 实现。覆盖 vault 长期演化中累积的实际问题(冗余 / 过载 / 稀疏 / 腐败 / 抽象缺位),通过周期 batch + 写后 inline 的方式**主动改 vault**,让记忆系统保持健康。 +> +> 配套阅读: +> - `auto_cognition_design.md`:三阶段顶层心智循环 +> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型 / F-invariants 原始定义 / 边守恒 +> - `auto_recall_design.md`:Stage 3 检索 —— 消费本文档产出的信号 +> - `auto_memory_design.md`:auto-memory 写 daily,daily 节点不参与本文档的巩固改造 +> - `structure.md` §3.6(maintain 动作语义) +> +> **核心立场**: +> - consolidate **不是产报告等人介入**,是**主动解决问题** —— 类比 NREM 慢波睡眠的 systems consolidation:跨多事件抽 schema、修剪弱连接、稳态突触强度。这些都是真实发生的改造 +> - vault **会被 consolidate 改**,但每个动作有严格的**置信度门槛 + 守恒规则 + 审计 trail + 渐进 rollout** +> - 灰色地带(置信度不够)才产报告等人介入;高置信度自己解决 +> - **community detection 是巩固的中枢** —— P0 基础设施,P1-P3 三个动作(abstract / merge / reinforce)都依赖它 + +--- + +## 0. 问题陈述与五大动作全景 + +dream 写入是单点视角,有三类视野局限:**写入瞬间没有跨节点视角 / 跨时间视角 / 全局拓扑视角**。这些局限会让 vault 长期演化中累积五类实际问题: + +| # | 问题 | 类比 | 表现 | 解决 | +|---|---|---|---|---| +| 1 | **冗余** | 同事件留下重复记忆痕迹 | dream 漏判去重 / 术语演化 / 跨桶建成两份 | merge | +| 2 | **过载** | 单一突触表征过密 | 节点 body 累积过长 / 单节点杂糅多主题 | split | +| 3 | **稀疏** | 应有连接未建立 | dream 写入瞬间漏召回的相关节点 / 反复共现但无 wikilink | reinforce | +| 4 | **腐败** | 长期不激活的痕迹 | 旧节点过时 / 半年没人读 / 内容已被矛盾 | archive | +| 5 | **抽象缺位** | 跨多 instance 缺 schema | vault 只有原子节点,没有"主题层"视角承接全局问 | abstract | + +### 0.1 五大动作 + 优先级 + +| 优先级 | 动作 | 解决问题 | 触发节奏 | 改 vault | 风险 | 收益 | +|---|---|---|---|---|---|---| +| **P0** | **community detection** | (基础设施) | weekly batch | 否 | 0(只产 meta) | 基础(其它三个都靠它)| +| **P1** | **abstract** | 抽象缺位 | weekly batch(基于 P0) | 是(新建 summary) | 低(additive) | **最高**(GraphRAG 核心) | +| **P2** | **merge** | 冗余 | weekly batch(基于 P0) | 是(合并 + retarget) | 高(lossy) | 中(消除可见冗余) | +| **P3** | **reinforce** | 稀疏 | weekly batch(基于 P0) | 是(additive 加 wikilink) | 低 | 低(retrieve multi-hop 已部分弥补)| +| **(独立)** | **split** | 过载 | inline 写后(D3) | 是(拆 parent + children) | 低 | 中 | +| **(独立)** | **archive** | 腐败 | daily batch | 软(meta 标记) | 0 | 中 | + +**关键论断**:**P1 比 P2 优先** —— abstract additive 失败可逆且回报最大;merge lossy 失败要回滚 inbound,价值是消除冗余(必要但不增能力)。 + +### 0.2 实施路径 + +``` +M0: P0 community detection (基础设施) + + split (已实现) + + archive (软标记,完全可逆) + +M1.1: P1 abstract (additive,最低风险开始改 vault) +M1.2: P2 merge (lossy,高门槛 + 多数票) +M1.3: P3 reinforce (additive,价值最低,可缓做) + +M2+: 多层 abstract (L2 super-community) / delete / typed predicate reinforce +``` + +### 0.3 显式排除 + +- ❌ 重做"抽象判断" —— gate 决策只在 dream(consolidate 不重新判定"该不该记") +- ❌ 重做"语义内容" —— UPDATE 三种 flavor(CORROBORATE / REFINE / CORRECT)只在 dream;consolidate 做结构层,不做语义层 +- ❌ 改 daily / resource —— consolidate 只动 digest 节点(I-2 / I-3 仍守) + +--- + +# Part A — community 工作群(本文档核心) + +P0-P3 四件套围绕 community detection 协同工作:**community 提供"哪些节点同主题"的判据,abstract / merge / reinforce 各自利用这个判据做不同的解决动作**。 + +## 1. community detection(P0,基础设施) + +**目的**:在 vault wikilink 图上做 community detection,产出"节点 → community_id"映射。这是 P1-P3 三个动作的**唯一前置**。 + +### 1.1 算法选择:Leiden + +| 选项 | 评估 | +|---|---| +| Louvain | 经典,但有 resolution limit + disconnected community 风险 | +| **Leiden** ✅ | Louvain 改进版(2019),稳定性显著好;GraphRAG 采用;Python `igraph.community_leiden` 现成 | +| label propagation | 实现最简,但结果不稳定(随机种子敏感) | + +**首版决策:Leiden**,直接对齐 GraphRAG 路线,后续接它的多层抽象更顺。 + +### 1.2 图的形态 + +| 维度 | 决策 | +|---|---| +| **节点范围** | **只 digest 节点**;daily / resource 不参与 | +| **边权重** | **首版 unweighted undirected**(所有 wikilink 等权)—— 加权方案(predicate 类型加权)留 M2+ 视效果 | +| **跨桶 community** | **必须允许** —— bucket 是物理归档,community 是语义聚合,二者本就正交。"错桶节点"会被自然纳入 community,可作 audit 信号但不强制 move(F-1 守住)| +| **resolution** | **1.0 起步**(Leiden 默认 / GraphRAG 默认)—— dogfooding 后视 community 平均规模(理想 5-15 节点)调 | +| **更新模式** | **全量重算**;vault 千节点级 Leiden < 1 秒,M0/M1 不引入增量复杂度 | + +### 1.3 多层级:M1 只 L1 + +| 层数 | 适用 | reme 决策 | +|---|---|---| +| 单层 L1(原子 → community)| vault < 500 节点足够 | **M1 起步** | +| 双层 L1 + L2(community → super-community) | vault > 500 节点 / 跨主题大类涌现 | M2+ 视规模 | +| GraphRAG 4 层 | 大规模文档库 | M3+ 不优先 | + +理由:GraphRAG 论文证明 L1 拿走 60-80% 效果。先把 L1 跑稳,L2 看实际是否需要。 + +### 1.4 输出 + +**`meta/communities.json`**: +```json +{ + "version": 1, + "computed_at": "2026-06-08T03:00:00Z", + "algorithm": "leiden", + "resolution": 1.0, + "communities": { + "digest/auth/jwt-rotation.md": "c_07", + "digest/auth/oauth-flow.md": "c_07", + "digest/api/rate-limit.md": "c_12" + }, + "stats": { + "n_communities": 14, + "median_size": 7, + "max_size": 23 + } +} +``` + +**`meta/community_changes.json`**(供 abstract 稳定度判据): +```json +{ + "computed_at": "...", + "previous": "...", + "stability_per_community": { + "c_07": 0.92, // 1 - (Jaccard 距离与上周该 community 节点集) + "c_12": 0.45 // 不稳定,abstract 跳过 + } +} +``` + +### 1.5 community_id 不需要稳定 + +下游(abstract / merge / reinforce)只关心"两节点是否同 community";id 本身可重排。每周重算后 id 不需要保持与上周对齐。stability 信号通过节点集 Jaccard 距离计算,不依赖 id。 + +### 1.6 用途总览 + +| 下游 | 用法 | +|---|---| +| **abstract**(§2)| 判据"该 community 节点数 ≥ N + 稳定度满足 + 无 hub" → 创建 summary | +| **merge**(§3)| 候选 pair 必须在同 community(降错率;不同 community 的相似 description 多是同名异义)| +| **reinforce**(§4)| 候选 wikilink 必须在同 community(避免假关联)| +| **recall**(`auto_recall_design.md` §3) | 同 community 节点 boost | + +--- + +## 2. abstract(P1,抽象提升) + +**类比**:NREM systems consolidation —— 跨多次睡眠把分散事件抽出共同 schema,从 episodic 升到 semantic。 + +**目的**:vault 演化到一定规模后,某些 community 形成稳定主题群,需要一个 hub 节点统领,让 retrieve 能召回到"主题概览"而非散点。 + +### 2.1 等价处理立场(关键) + +**summary 节点完全等同普通节点**: + +| 维度 | 决策 | +|---|---| +| **路径** | LLM 选桶,正常 slug 命名(如 `digest/auth/authentication-mechanisms.md`);**无 `__community__` / `__hub__` 等结构性标识** | +| **frontmatter** | 仅 `name + description`(reme 核心保留);**无 `kind: community_summary`、无 `auto_generated`** | +| **summary 性质** | 完全体现在 **body 形态** —— 主题概述 + 列出 source 节点 wikilink + 跨节点 pattern;但这是内容自然形态,不是结构性宣告 | +| **后续维护** | **无** —— 跟其它节点等价,被 dream / split / merge / archive 自然演化(参见 §2.6) | + +这跟 dream 的核心立场对齐:"节点角色由 body 内容决定,不由 frontmatter 类型标记"。abstract 是"用一种新方式创造节点",不是"创造一种新节点类型"。 + +### 2.2 触发判据(组合门槛) + +``` +weekly batch: + for community in communities.json: + if community_has_hub(community): # §2.5 结构化判据 + continue + if len(community) < MIN_NODES (5): # 节点数门槛 + continue + if stability(community) < 0.7: # 稳定度门槛 + continue + if active_node_count(community, 30d) < 3: # 活跃度门槛 + continue + if name_diversity(community) < 0.5: # 多样性门槛 + continue + → enqueue abstract job +``` + +| 门槛 | 默认 | 含义 | 防的是 | +|---|---|---|---| +| **节点数** | ≥ 5 | community 大小 | 给 2-3 节点造 hub 不划算 | +| **稳定度** | ≥ 0.7 | 与上周边界 Jaccard 距离 | 给短命 community 造 hub 浪费 | +| **活跃度** | ≥ 3 节点近 30 天 hit | community 仍在用 | 给死社区造 hub(下次没人看)| +| **多样性** | name 差异度 ≥ 0.5 | frontmatter `name` 互不相同 | 给"一组重复节点"造 summary —— 那是 merge 的事 | + +### 2.3 创建动作 + grounding 守恒 + +``` +LLM 看 community 内所有节点 (frontmatter + body) + ↓ +产 planned summary body (三段): + 1. 主题概述 (1-2 段,跨多节点共同主题) + 2. 关键支柱 (列表,3-5 节点 + 一句话 + wikilink) + 3. 不在概览的细节 (明说哪些细节留原节点) + ↓ +长度限制: summary body < 1500 token + (防 abstract 创建后立刻被 split 触发,§5) + ↓ +LLM 决定 path: digest//.md + ↓ +CAS 写入 (§9) + 双重守恒校验: + - 机械: 出边集合 ⊇ "关键支柱"声称引用的节点 (防套话) + - 机械: 出边集合 ⊇ source_nodes 的至少 60% (allow LLM 漏列少数) + ↓ +audit 记录: audit//consolidate_actions.md +``` + +**grounding 守恒**:summary body 中**声称引用某节点必须真写 wikilink**。LLM 不能仅口头提及"我们在 X 中看到..."而不带 `[[X.md]]`。这是机械可校验的,LLM 跑不掉。 + +### 2.4 长度限制为什么重要 + +summary body < 1500 token 是**与 split 互锁的机制**: + +- 不限长 → LLM 会写"完整覆盖" → 最终 body 累积接近 split 阈值(2000 token)→ 下次 D3 触发拆 → 拆出来的 children 又被 community 视为同主题 → 下次 abstract 又造一个 hub → 循环 +- 限长 1500 → summary 留出 split 阈值的 25% buffer,稳定不触发拆 + +### 2.5 "community 已有 hub"的结构化判据 + +不靠 frontmatter / 路径标识,靠**结构**: + +``` +def community_has_hub(community): + for node in community: + out_targets = outbound(node) ∩ community + if len(out_targets) / len(community) >= 0.6: + return True # 该节点出边覆盖 community 60% 以上 → 它已是 hub + return False +``` + +**好处**: +- split parent overview 自然被识别为 hub(split parent 出边覆盖大部分 children)→ abstract **复用** split 的工作,不重复创建 +- 已有 abstract 创建过的节点,只要它出边没退化,下次 batch 自然识别为 hub,不重复创建 +- 节点被 dream update 后形态变化,出边变了 → 自动重新评估 + +**M1 实施关键验证点**:跑实测验证这个涌现 —— split parent 是否真被识别为 hub。如有 corner case,调阈值 0.6 → 0.5 / 0.7。 + +### 2.6 后续维护:无 —— 完全靠 5 大动作演化 + +abstract 创建即放归 vault,**consolidate 不再"管"它**。后续命运: + +| 演化路径 | 结果 | +|---|---| +| 新材料触及该主题 | dream update 自然修正 body(走 CORROBORATE / REFINE / CORRECT)| +| 老 summary 长期不被引用 | archive 自动归档(§6)| +| community 边界变了 → 下次 batch 创建新 summary | 新老 summary 描述同主题 → merge 自动合并(§3)| +| summary body 累积过长 | split 自动拆(§5)| + +这是真正的"vault 自我代谢"。**没有特殊维护通道**。 + +--- + +## 3. merge(P2,同概念合并) + +**类比**:NREM 跨多次睡眠识别"同一件事" → 合一个记忆痕迹。 + +**目的**:消除 vault 内的冗余 —— 同概念多节点。 + +### 3.1 候选挖掘(community 内三层过滤) + +``` +weekly batch (依赖 community detection): + for community in communities: + pairs = all_pairs(community) + for (A, B) in pairs: + if description_sim(A, B) < 0.6: # 第一层: frontmatter 相似 + continue + if body_topic_overlap(A, B) < 0.5: # 第二层: body 主题词重合 + continue + if cooldown_active(A) or cooldown_active(B): # 第三层: cooldown 检查 + continue + candidates.append((A, B)) +``` + +**关键约束**:候选必须在**同 community**(降错率)。 + +### 3.2 多数票决策 + +merge 是高风险动作(lossy + 改 inbound),用多数票降错: + +``` +for (A, B) in candidates: + votes = parallel_run(N=3, prompt="A 和 B 是否同一概念? 返回 {is_same, confidence}") + agree = sum(v.is_same and v.confidence >= 0.8 for v in votes) + if agree >= 2: + → enqueue merge job + elif agree == 1: + → 写 audit//dups_uncertain.md (灰色地带,人介入) + else: + → 丢弃 +``` + +### 3.3 merge 动作:body 重写归 consolidate(方案 B) + +**关键决策**:merge 后的 body 由 **consolidate 自跑合并 prompt**,不走 dream update 路径。 + +| 方案 | 评估 | 决策 | +|---|---|---| +| A. 走 dream update 路径(把 loser body 作"新材料")| 优雅但跨阶段;dream 不应知道 caller 是 consolidate 还是新材料 | ❌ | +| **B. consolidate 自跑合并 prompt** | 简单自包含;通过严格 prompt 约束化解"做语义工作"张力 | ✅ | +| C. 不重写 body(留 redirect stub) | 完全不做语义,但 vault 留无用节点 | ❌ | + +**B 方案的边界守住**(避免 consolidate 真在做语义判断): + +| 边界 | 含义 | +|---|---| +| **prompt 严格约束** | "只合并不精化" —— 不重写措辞、不加新内容、不做精化决策 | +| **机械守恒** | 出边 ⊇ A.outbound ∪ B.outbound + provenance 全保留(LLM 跑不掉) | +| **信息守恒抽样** | LLM 自检 "merged.body ⊇ A.body ∪ B.body 全部信息";audit 抽样人审 | +| **失败拒写** | 守恒校验失败 → LLM 重试一次 → 二次失败拒写 + audit | + +### 3.4 完整动作流 + +``` +A, B → 选择 winner (path): + - inbound 数大者赢 (保护既有 inbound,降 retarget 量) + - 平局取路径短者 + ↓ +LLM 跑 merge prompt → planned merged_body (B 方案) + ↓ +机械 retarget 准备: + - 扫所有 inbound(loser): [[loser.md]] → [[winner.md]] + - alias 保留;predicate 保留 + - 这是机械算子,非 LLM + ↓ +事务式 CAS 写入: + 1. winner body 改写 + 2. 所有 inbound 节点 body 改写 (retarget) + 3. 删除 loser 文件 + 任一步失败 → 全部回滚 + ↓ +audit 记录 + cooldown 设置 (winner 进 cooldown 2 weeks) +``` + +### 3.5 灰色地带:报告 + +- 多数票通过(agree ≥ 2)→ 自动 merge +- 仅 1 票通过 → 写报告 `audit//dups_uncertain.md`,人 / agent 介入 +- 0 票 → 丢弃 + +报告格式: +```markdown +# dups uncertain 2026-06-08 + +## pair 1 (1/3 votes) +- A: digest/auth/jwt-rotation.md ("JWT 密钥轮换") +- B: digest/security/key-rotation.md ("密钥轮换原则") +- vote 1 (yes, 0.85): "同一概念,A 偏 JWT 场景" +- vote 2 (no, 0.72): "B 是通用原则,A 是具体应用" +- vote 3 (no, 0.68): "粒度不同,不应合并" + +建议:走 dream update 通道把 A 内容作为 B 的实例并入。 +``` + +--- + +## 4. reinforce(P3,关系强化:补 dream 漏的 wikilink) + +**类比**:NREM 突触强化 LTP —— 反复共激活的连接被强化。 + +**目的**:vault 演化中,某些节点对应该有 wikilink 但 dream 写入时漏召。reinforce 周期检测并 additive 补。 + +### 4.1 候选挖掘(三层过滤) + +``` +weekly batch (依赖 community detection): + for community in communities: + for (A, B) in all_pairs(community): + if has_wikilink(A, B): + continue + # 第一层: 字符串 mention 锚点 + if not has_mention(A.body, B.frontmatter.name): + continue + # 第二层: embedding 相似度验证 + if embedding_sim(A.context_around_mention, B.body) < 0.7: + continue + # 第三层: 同 community (已经是,但显式说明) + candidates.append((A, mention_pos, B)) +``` + +**三层过滤的角色**: + +| 层 | 防的是 | +|---|---| +| 字符串 mention | 大幅降候选数(从 O(N²) 降到 O(实际共现)) | +| embedding 相似度 | 防同名异义("Apple" 公司 vs 水果)| +| 同 community | 防表面术语共现但语义无关 | + +### 4.2 决策(单票即可,门槛较高) + +reinforce 是 additive 低风险动作,不需要多数票: + +``` +for (A, mention_pos, B) in candidates: + vote = LLM("A.body 在该位置提到 B 的概念。是否合理加 [[B.md]] 链接?") + if vote.confidence >= 0.85: + additive_wikilink(A, mention_pos, target=B.path) + → CAS 写入 (E-1 自动满足:additive 只增不删) + → audit 记录 + else: + 丢弃 +``` + +### 4.3 边界 + +| 维度 | 决策 | +|---|---| +| **只 additive 加 wikilink** | 不改 body 文字,不升级 typed predicate(predicate 升级是语义判断,留 dream)| +| **alias 保留原文** | `[[B.md\|<原文 mention>]]`;原文一字不改 | +| **写入位置** | mention 第一次出现处加;后续保持原文(防 wikilink 满文) | +| **不动 anchor** | 与 dream 一致 | +| **守恒** | E-1 天然满足(纯增) | +| **rollback** | 误链发生时,人 / agent 直接编辑 body 删除 wikilink 即可;reinforce 不维护"我加过哪些"audit log(每次动作进 `audit//consolidate_actions.md`)| + +### 4.4 reinforce 与 dream 的边界 + +dream 写入时 LLM 应已尽力召回相关节点 + 加 wikilink。reinforce 是**周期性兜底** —— 写入瞬间漏的、术语后才一致的、被 split 拆出来后才相关的,在 reinforce batch 里被检出。 + +这不违反"consolidate 不修 dream 漏的"立场 —— **dream 漏的 wikilink 在巩固阶段补,是合法工作**(它的依据是 dream 单点视角永远做不到的"周期统计 + 全局视角");**dream 漏的语义抽象在巩固阶段不补**(那是 dream 的语义判断,consolidate 不重做)。 + +--- + +# Part B — 独立工作 + +P0-P3 围绕 community,这两个动作独立运行。 + +## 5. split(过载分化:inline 写后) + +**类比**:海马表征过密 → 分化新单元。 + +**目的**:节点 body 累积过长 / 主题离散后,拆成 parent overview + N children,保持单节点"一个原子语义单元"的粒度。 + +### 5.1 触发模型(写后立即,inline) + +split 是 5 大动作中**唯一 inline** 的 —— 跟 dream 写入流强耦合,不走 weekly batch: + +``` +dream / split 写 body 成功 (CAS 通过) + └─ if len(body) > T_token (default 2000): + └─ LLM 判离散度 + └─ if is_overloaded: + └─ enqueue split job (FIFO, CAS-protected) + └─ return (不阻塞 dream) +``` + +理由:节点过载是**写入瞬间的本地信号**(token + 离散度),延后无价值;反应即时。 + +### 5.2 split 动作 + +``` +LLM 看 parent body: + - 拆成 1 个 parent overview body + N 个 children body + - 每个 child 自带 [[parent]] 反向链接 + - inbound 不动 (F-10) + ↓ +机械 outbound 守恒校验 (E-2): + (parent_new ∪ ∪children_outbound) ⊇ parent_old + 失败 → LLM 重试 → 二次失败拒写 + audit + ↓ +事务式 CAS 写入: parent body 改写 + N 个新 children 文件创建 + ↓ +audit + cooldown 设置 (parent + children 进 cooldown,与 merge 互锁) +``` + +### 5.3 split 与 abstract 的协同(关键) + +| | 起源 | 方向 | 触发 | +|---|---|---|---| +| split overview | 单节点过载分化 | 自上而下(一拆多)| inline 写后 D3 | +| abstract summary | 多节点抽象凝聚 | 自下而上(多归一)| weekly batch + 稳定度阈值 | + +**协同**:split 产出的 overview 节点会被 §2.5 的"已有 hub"判据识别,abstract 不重复创建。两者互补,不冲突。 + +--- + +## 6. archive(时效衰减:让长期不激活的节点淡出) + +**类比**:突触代谢稳态 —— 长期不用的连接被减弱,但不删除。 + +**目的**:让 retrieve 默认排除"已不活跃"的节点,提升信噪比;不删 vault 文件,保持可逆。 + +### 6.1 recency_score:连续衰减信号 + +``` +recency_score(node) = + exp(-(now - last_update) / τ_update) # 时间衰减 + × (1 + log(1 + last_hit_count_30d)) # 活跃度增强 + × (1 + log(1 + inbound_count) / SCALE) # 中心性 cushion(避免 hub 被冷藏) +``` + +| 参数 | 默认 | 含义 | +|---|---|---| +| τ_update | 60 days | 时间衰减常数 | +| SCALE | 10 | 中心性 cushion 缩放 | + +输出:`meta/recency.json`,每节点 0.0~1.0 连续值。 + +### 6.2 archived 派生快照 + +archived 是 recency_score 的二元化派生: + +``` +archived = {node | recency_score(node) < 0.15} +``` + +输出:`meta/archived.json`,recall 默认过滤这个列表。 + +### 6.3 解冻 + +任何动作触及节点 → 自动从 archived 移除: +- retrieve 命中(写 access_log) +- dream update 触及 +- merge / reinforce 触及 + +下次 batch 时 recency_score 重算自然超过阈值。 + +### 6.4 daily 节奏 + +archive 是唯一不需要 community detection 的动作 → 节奏可以更快(daily batch),让冷启动后第二天就能影响 recall。 + +``` +daily batch: + 1. 读 access_log (retrieve / dream / consolidate 钩子记录的命中事件) + 2. 重算 recency_score for all digest nodes + 3. 输出 meta/recency.json + 4. 阈值过滤 → meta/archived.json +``` + +--- + +# Part C — 共享基础设施 + +## 7. F-invariants 松绑与守恒规则 + +旧 F-invariants(`auto_dream_design.md` §4.3)在"vault 只读"立场下定义,新立场要松绑。但松绑不是"自由改",是用**动作级守恒规则**换"一刀切禁令"。 + +### 7.1 F-invariants 修订 + +| # | 旧约束 | 新立场 | +|---|---|---| +| **F-1** | 0 文件移动 | **改为**:"非 consolidate 动作不移动文件";merge 删除 loser 文件是**合法移动**(逻辑上等价 retarget) | +| **F-2** | 改正文限定 subject | **改为**:"dream / split / reinforce 改 subject body;merge 在受控算子内可改 inbound 节点 body";其它阶段(recall)绝不改 | +| **F-3** | maintainer 只做 split | **作废** —— consolidate 5 大动作合法 | +| **F-10** | inbound 不动 | **改为**:"split 时 inbound 不动";merge 必须 retarget inbound(机械算子) | +| **F-11** | wikilink 是 body 一部分 | **保留** —— 没有"独立的边"基础设施 | + +### 7.2 动作级守恒规则矩阵 + +| 动作 | 置信度门槛 | 守恒规则 | +|---|---|---| +| **abstract** | community 节点 ≥ 5 + 稳定度 ≥ 0.7 + 活跃度 ≥ 3 + 多样性 ≥ 0.5 + 无 hub | 出边 ⊇ "关键支柱"列表 + 出边 ⊇ source 节点 60%(机械)| +| **merge** | LLM 多数票 ≥ 2/3 + similarity ≥ 0.6 + body overlap ≥ 0.5 | 信息守恒(merged.body ⊇ A ∪ B)+ 出边 ⊇ A.out ∪ B.out + inbound 全 retarget(机械)| +| **reinforce** | LLM 单票 ≥ 0.85 + 同 community + mention 锚点存在 + embedding ≥ 0.7 | E-1 天然(additive)| +| **archive** | recency_score < 0.15 | 软标记,无破坏性 | +| **split** | token > T + LLM 判离散 | E-2(parent ∪ children ⊇ parent_old)+ inbound 不动 | + +--- + +## 8. cooldown 与防循环 + +5 大动作之间的潜在循环: + +``` +A merge B → AB body 长 → split AB 回 A' + B' → 又 merge → ... +``` + +防御: + +| 互锁对 | 窗口 | 实现 | +|---|---|---| +| **split → merge** | 2 weeks | 刚 split 出的兄弟节点不参与 merge 候选 | +| **merge → split** | 2 weeks | 刚 merge 的节点不参与 split 评估(D3 检测时跳过)| +| **merge → merge**(同对反复) | 12 weeks | 同一 path 12 周内被 merge 又被识别为新 merge 候选 → audit 警报,人介入 | +| **abstract → merge**(同主题反复 abstract) | 4 weeks | 刚 abstract 出的 hub 节点 4 周内不参与 merge 候选 | + +cooldown 状态外置 `meta/cooldowns.json`,不污染 vault。 + +--- + +## 9. CAS 写入协议(共享基础设施) + +CAS 是 dream(`auto_dream_design.md` §4.2)、split / merge / reinforce / abstract(本文档)**多方共用**的 vault 写入协议。归本文档因 consolidate 是写入主战场。 + +archive 不写 vault → 不走 CAS;它写 `meta/`,各任务的 atomic write(write-temp + rename)即可。 + +### 9.1 协议 + +``` +1. 读 + 记戳: read body → version_stamp = sha256(body) | mtime +2. 决策: LLM / 算法 → 产 planned new_body +3. CAS 写入: 重读 body 比 version_stamp + - 未变: 跑动作级守恒校验 → 通过 → atomic write (write-temp + rename) → done + - 已变: 丢弃 planned new_body, 带最新 body 重走 step 1 +4. 守恒校验失败: LLM 重试一次, 二次失败拒写 + audit +5. 重做次数上限: 3 次 → 跳过候选 + audit log +``` + +### 9.2 事务式 merge / split 写入 + +merge 涉及多文件写入(winner body + N 个 inbound retarget + loser 删除);split 涉及多文件创建(parent body + N children)。需要事务语义: + +- 准备阶段:全部 planned new_body 写到 temp 区(带 version_stamp) +- 提交阶段:逐个 CAS 检查 + atomic write(write-temp + rename) +- 任一 CAS 失败 → 全部回滚(temp 区清理,已 rename 的恢复) + +实现细节:可借 fs-level 事务库(如 `pyrsistent` 模式)或自实现 journal。M0 起步用最简的"先全部检查 → 再全部写入"两阶段,接受窗口期(检查到写入间)的极小并发风险。 + +### 9.3 create 路径 race + +merge / abstract 都可能并发 create 同一 path → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者 EEXIST → 重走 step 1(此时大概率改判 update 或丢弃)。 + +### 9.4 不解决 + +- 跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+) +- 高冲突 workload(同候选反复触发)→ 重做上限触发后 audit + +--- + +## 10. D 健康检查(D1 / D10) + +不属"巩固"主语义,但跟 consolidate 同节奏(周期 batch 顺手跑),归本文档: + +| # | 信号 | 节奏 | 修复策略 | +|---|---|---|---| +| **D1** | 断链(wikilink → 不存在 path) | 写时 inline + weekly batch 巡检(双重保险)| 就地删 wikilink 或保留 alias 文本 → audit | +| **D10** | provenance 断裂(digest 反指的 daily/resource 不可达)| 同上 | I-不变量违反 → 严重告警 + 人介入 | + +D1 / D10 不算 5 大动作之一(它们不解决"vault 演化问题",只检测异常)。但它们的修复(就地删 wikilink)需要走 CAS,所以协议共享。 + +--- + +# Part D — 契约与实施 + +## 11. 维护 → 检索契约 + +5 大动作产物给 retrieve 消费(详细 retrieve 逻辑见 `auto_recall_design.md`): + +| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 | +|---|---|---|---|---| +| **vault 节点变化** | `digest/**.md` | merge / split / reinforce / abstract | recall(图遍历 / 命中) | — | +| **communities** | `meta/communities.json` | community detection | recall + abstract / merge / reinforce | 不做同社区 boost / 三个动作跳过 | +| **community changes** | `meta/community_changes.json` | community detection | abstract 决策 | abstract 跳过(无稳定度判据)| +| **recency** | `meta/recency.json` | archive daily batch | recall | recency_factor = 1.0 | +| **archived** | `meta/archived.json` | archive daily batch | recall(默认过滤)| 不过滤 | +| **cooldowns** | `meta/cooldowns.json` | split / merge | consolidate 内部 | 无防御循环 | +| **access_log** | `meta/access_log.json` | recall(写命中) + archive(聚合) | archive(读 recency) | recency 不衰减 | +| **dups uncertain** | `audit//dups_uncertain.md` | merge | 人 / agent | — | +| **consolidate actions** | `audit//consolidate_actions.md` | 全部 5 动作 | 审计 | — | +| **D1 / D10 健康** | `audit//health_*.md` | inline check + weekly | 人 / agent | — | + +**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。 + +--- + +## 12. 与 dream 模型的引用关系 + +本文档松绑了部分 F-invariants(§7),但仍在 dream 定义的底层模型上工作: + +| 引用 | 来源 | +|---|---| +| wikilink 基础语法 | `auto_dream_design.md` §3 | +| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 | +| F-invariants 原始定义 | `auto_dream_design.md` §4.3(本文档 §7 修订)| +| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 | +| 路径即 ID / rename | `auto_dream_design.md` §2 | +| anchor 不引入 | `auto_dream_design.md` §3 | +| provenance 载体形态 | `auto_dream_design.md` §4.2 | +| dream 写入路径 | `auto_dream_design.md` §4.2 | + +--- + +## 13. 下一步(M0 → M1.1 → M1.2 → M1.3 → M2) + +实现进入 `reme4/steps/consolidate/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md`(顶层)/ `auto_recall_design.md` 共同作为契约依据。 + +### M0:基础设施 + 完全可逆动作 + +- ✅ split inline 触发 + LLM 离散度判 + E-2 守恒(基础部分) +- ⏳ **community detection weekly batch**(Leiden via `igraph`)+ `meta/communities.json` + `meta/community_changes.json` +- ⏳ **archive daily batch** + recency_score + access_log 收集 +- ⏳ CAS 写入框架 + version_stamp + EEXIST race + 重做上限 + audit +- ⏳ D1 / D10 写时 inline 检测 + weekly 巡检 + +### M1.1:abstract(P1,additive 最低风险) + +- ⏳ abstract 候选挖掘(community 大小 + 稳定度 + 活跃度 + 多样性 + 无 hub 五重判据) +- ⏳ abstract LLM prompt(三段输出 + 长度限制 1500 token) +- ⏳ grounding 守恒校验(出边 ⊇ 关键支柱 + 出边 ⊇ source 60%) +- ⏳ "已有 hub" 结构化判据(outbound 覆盖度 ≥ 60%) +- ⏳ **关键验证点**:实测 split parent 是否被识别为 hub + +### M1.2:merge(P2,lossy 高门槛) + +- ⏳ 候选挖掘(community 内 description 相似 + body 重合 + cooldown 检查) +- ⏳ 多数票框架(N=3 LLM,2/3 通过) +- ⏳ merge prompt(B 方案:"只合并不精化") +- ⏳ inbound retarget 机械算子(扫所有 `[[loser.md]]` → `[[winner.md]]`,alias / predicate 保留) +- ⏳ 事务式多文件 CAS 写入 +- ⏳ 灰色地带报告(`audit//dups_uncertain.md`) +- ⏳ cooldown 框架(`meta/cooldowns.json` + 各动作互锁) + +### M1.3:reinforce(P3,价值最低,可缓做) + +- ⏳ 候选挖掘(三层过滤:mention + embedding + 同 community) +- ⏳ 单票决策(门槛 0.85) +- ⏳ additive wikilink 写入(alias 保留原文) + +### M2+:演进 + +- ⏳ 多层级 community(L2 super-community)+ L2 abstract +- ⏳ delete(永久删除 vault 文件)—— 视 dogfooding 效果决定是否开启 +- ⏳ predicate upgrade(typed link reinforce —— 当前 reinforce 只 additive 加无谓词) +- ⏳ PageRank 替代 simple inbound count(若 retrieve 质量瓶颈在中心性) +- ⏳ 跨进程并发(fs lock 支持多 reme 实例同 vault) +- ⏳ Leiden 边权重(按 predicate 类型加权) diff --git a/docs4/auto_dream_design.md b/docs4/auto_dream_design.md index c2480e2b..94dfef5e 100644 --- a/docs4/auto_dream_design.md +++ b/docs4/auto_dream_design.md @@ -5,8 +5,8 @@ > 配套阅读: > - `structure.md` §1.2(数据视角)/ §2(三层存储)/ §3.5(digest 动作) > - `auto_memory_design.md`:daily 实时事件 = dream 的入流之一 -> - `auto_maintain_design.md`:M split / D 检测 / CAS 写入协议(dream 模型的运行时实现) -> - `auto_link_design.md`:dream 写完后的后置增强(背景实体识别 + wikilink 写回) +> - `auto_consolidate_design.md`:M split / D 检测 / CAS 写入协议(dream 模型的运行时实现) +> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想 —— dream 是其 Stage 1(写入阶段)的实现 > > **核心**:digest = **浅桶(shallow bucket)+ flat .md** + **一张图(节点 + 边)**;dream 定义模型与主流程(create_or_update),maintain 负责 split / 写入运行时。 > @@ -117,8 +117,8 @@ dream 设计回答四个问题:**桶**怎么布局 / **节点**长什么样 / ** | op | 谁 | 何时 | 改什么 | |---|---|---|---| -| **dream**(create_or_update) | dreamer(本文档 §4.2) | 入流(新材料进入) | 创建新节点 / update 已有节点 body(语义守恒重写) | -| **M split** | maintainer(`auto_maintain_design.md` §1) | 节点过载(token / 主题离散度超阈值) | 把 parent body 拆成 parent overview + N children;parent 文件原地 | +| **dream**(create_or_update) | dreamer(本文档 §4.2) | 入流(新材料进入) | 创建新节点 / update 已有节点 body(语义守恒重写;UPDATE 内分 **CORROBORATE / REFINE / CORRECT** 三种 flavor,详 §4.2.3) | +| **M split** | maintainer(`auto_consolidate_design.md` §1) | 节点过载(token / 主题离散度超阈值) | 把 parent body 拆成 parent overview + N children;parent 文件原地 | > **关键观察**:"主题概览节点"不是一种 kind,也不是 maintainer 主动涌现的产物 —— 它是 split 的副产品(parent 节点天然成为该 cluster 的 overview,中心性自然高)。 @@ -131,51 +131,77 @@ dream 设计回答四个问题:**桶**怎么布局 / **节点**长什么样 / ** **dream = dreamer 入流唯一改 body 的操作,且只改 subject node。** +#### 4.2.0 digest 是抽象记忆层 + +Digest 是 agent 长期记忆的**抽象层** —— 类比前额叶对认知的聚合。原始细节(数字、流程文本、谁说了什么)留在材料(daily / resource),digest 只承载细节淡忘后仍想调取的那一层:原则、模式、可作为先例的决策、认知要点。这一立场决定了 dream 流程的形态:**Phase 1 识别抽象,Phase 2 把抽象登记到 digest 节点**。 + +#### 4.2.1 两阶段流程 + ``` material 进入(daily / resource 选定 scope) │ ▼ -LLM 抽取原子单元 → N 个候选 +Phase 1 — extract (轻量) + LLM 读材料 → 识别其中教导的"抽象"(原则 / 模式 / 先例) + → 发出 ExtractedUnits 结构化输出 = K 个 sub-unit + (每个: {name, summary},summary 标注证据在材料的哪段) + 说明:多个支撑事实说明同一抽象 → 合并为同一 sub-unit + (倾向少而精);Phase 1 是 gate —— + 无新抽象时发空列表,Phase 2 跳过整轮 │ - ▼ 对每个候选: -SearchStep 召回相似候选节点 - (reme4/steps/index/search.py;vector + keyword 并发 → RRF 融合 - → expand_links 邻接展开;scope `digest/`;默认 limit 5~10) + ▼ (Python 外循环,K 次) +Phase 2 — integrate (per sub-unit,每次独立 ReAct 会话) + │ sub-unit ↔ digest 节点 1:1;Phase 2 必写,无 SKIP 出口 + │ + ├─ RECALL: search(关键词 + 向量 + RRF) + traverse(对 top hit + │ 做图扩展,跨 bucket) → 候选路径集 + │ + ├─ HIT: frontmatter_read 廉价 triage → read 完整 body + │ 确认候选是否承载同一抽象 → hit 集合 + │ + ├─ 决策: + │ ├─ hit 空 → CREATE 路径 (挑 bucket,写新节点) + │ └─ hit 非空 → UPDATE 路径 (CORROBORATE / REFINE / CORRECT) │ ▼ -LLM 终判:候选池里有"同概念节点"吗? - ├─ 有 → update 路径 - │ (a) 把新内容融入已有 body(语义守恒重写) - │ (b) 加 provenance 反指 - │ (c) 必要时加 / 改 wikilink - │ - └─ 无 → create 路径 - (a) 挑 bucket(固定集合;无合适专属桶 → `unknown`) - (b) 写文件名(同 bucket 唯一,fs 层断言) - (c) 写 body + provenance + 横向 link +写入(digest_write 创建 / digest_edit 改正文,E-1 强守恒,§4.4) │ ▼ -写入前 outbound diff 守恒校验(update 走 E-1;create 无 old outbound) - │ - ▼ -CAS 写入(`auto_maintain_design.md` §5) - │ - ▼ -写完 inline 触发 D3 检测(`auto_maintain_design.md` §4) +agent 上报 IntegrateOutcome {action, target_path} ``` -**关键边界**: -- **dream update 必须语义守恒** —— LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁);**写入前机械校验出边强守恒**(E-1,详 §4.4) +**两阶段 trade-off**:Phase 2 把完整材料发 LLM K 次(一次一 sub-unit),不做 summary loss;代价是 K 倍 prompt token。换来的是 Phase 1 只做"识别抽象"这一件事(粒度集中在一个 prompt),Phase 2 每次会话上下文干净、聚焦单一抽象的写决策。 + +#### 4.2.2 召回二段 + +**RECALL = search + traverse**:search 给关键词 + 向量 RRF 命中;只要 search 在 `digest/` 下返回任何 hit,就对 top hit 跑 `traverse depth=2 direction=both`。理由是 search 关键词导向,会漏掉用不同术语归档的语义相邻抽象,那些常常一跳之外。search 在 `digest/` 下完全无命中 → 无 traverse 起点 → 候选集为空 → 直接 CREATE。 + +**HIT = frontmatter_read + read**:渐进披露 —— 先 `frontmatter_read` 读 `name + description` 廉价 triage 淘汰明显无关候选,剩下的再 `read` 整 body。**不可仅凭 chunk 片段或 frontmatter 决定 UPDATE**,body 才是判定依据。 + +#### 4.2.3 UPDATE 三种 flavor + +| flavor | 何时 | body 怎么动 | +|---|---|---| +| **CORROBORATE**(最常见)| 已有节点已覆盖此抽象,材料是又一个实例 | body 实质不变 —— 追加 `derived_from::` 溯源,可选强化措辞("似乎"→"确实") | +| **REFINE**(常见)| 已有节点覆盖了核心,但材料揭示新的范围 / 边界 / 维度 | 改相关片段使更精确,加新维度,加 `derived_from::`。正文在**精度**上长,不在**细节**上膨胀 | +| **CORRECT**(少见)| 材料与已有抽象矛盾 / 表明它被夸大 | 收紧到新旧证据都支持的窄形式,或内联标注 `> note: contradicted by [[...]]` 不仲裁。仍加溯源 | + +三种都受 §4.4 E-1 强守恒约束(出边集合不能缩)。 + +#### 4.2.4 关键边界 + +- **Phase 1 是 gate** —— "不值得记忆"在 Phase 1 过滤(空列表);Phase 2 必然写,sub-unit 与 digest 节点 1:1 +- **dream update 必须语义守恒** —— LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁);写入前机械校验出边强守恒(E-1,详 §4.4) - **dream 不改其它节点正文**(F-2) —— 只动 subject +- **dreamer 不做事件级伞节点** —— 材料本身(daily / resource 文件)就是 fan-out 点,每个 sub-unit 的 `derived_from::` 让材料天然聚合到所有派生节点 - **0 出边节点合法**(没识别到合适邻居),后续 dream 进入时其它节点可以反向链回来 —— 不强求 LLM 一次性给全 -- **dream 漏判去重**(同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显重复,由 auto-link L4 离线 audit 工具产报告(`auto_link_design.md` §1.3) +- **dream 漏判去重**(同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显重复,由 auto-consolidate 的 dups 检测周期 batch 产报告(`auto_consolidate_design.md` §3) - **召回不做 bucket 粗筛** —— LLM 拥有完整跨桶视野,可识别"概念错分到 unknown"或"跨桶同概念" **provenance 写出**: - 行文中自然带:"... 该模式最早出现在 [[daily/2026/05/15.md]] 的实践中" -- 可选 predicate:`derived_from:: [[daily/2026/05/15.md]]`,不强制 -- prompt 必须要求"出处用 `[[...]]` 形式表达"(纯散文会被守恒校验视为丢边) -- **首版可先用 append 起步**(出边集合天然 ⊇,守恒校验自动通过);成熟后切到重写 +- **强制 typed predicate `derived_from::`** —— body 必须织入至少一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`,纯散文形式不被守恒校验视作边,下次 update 时会消失 +- 不走"首版 append 起步"的过渡路径 —— digest_edit 自一开始就跑 E-1 强守恒,LLM 直接做语义守恒重写 ### 4.3 F-invariants(演化的硬约束) @@ -228,21 +254,18 @@ write_subject_body(subject, new_body): |---|---| | ← **auto-memory**(daily) | dream 读 daily 作为入流;daily 写完即对 dream 可见 | | ← **resource** | dream 读 resource 作为入流(只读,不写) | -| → **auto-maintain** | dream 写完触发 D3 inline 检测;D3 过载 → enqueue split job(maintain 异步消费);写入并发由 CAS 协议(`auto_maintain_design.md` §5)保护 | -| → **auto-link** | dream 写完 enqueue auto-link L1(背景实体识别 + wikilink 写回);走同一 CAS 队列(`auto_link_design.md` §1.2) | -**关键边界**:dream 不写 daily / resource(I-2 / I-3);只写 digest 节点 body(自身 subject)。 +**关键边界**:dream 不写 daily / resource(I-2 / I-3);只写 digest 节点 body(自身 subject)。dream 不感知下游 —— split / 链接增强 / 索引刷新 / rename 等由 `auto_consolidate_design.md` / `auto_cognition_design.md` / `update_store_index_loop` 各自负责。 --- ## 6. 下一步 -本文档覆盖 dream 模型(桶 / 节点 / 边 / 演化)。组织端实现清单(M split / D 检测 / CAS 框架)见 `auto_maintain_design.md` §10。 +本文档覆盖 dream 模型(桶 / 节点 / 边 / 演化)。组织端实现清单(M split / D 检测 / CAS 框架)见 `auto_consolidate_design.md` §10。 -1. **dream step 实现** —— scope → 抽取 → SearchStep 召回 → LLM 终判 → CAS 写入 + E-1 守恒校验 -2. **rename 路径封装** —— `wikilink_handler.retarget_links(old, new)` 已就绪;封装为单步 step,无 alias 表,无透明展开 -3. **bucket 集合配置** —— `vault.yaml` schema / 默认桶模板 / `unknown` 兜底 / `_buckets.md` 视图生成 -4. **边守恒校验工具** —— `extract_links` 已就绪;新增 outbound diff 比较器 + LLM 重试编排 + ConservationViolation audit 事件 -5. **provenance prompt 规范** —— dream 引导 LLM 写 `[[daily/...]]` / `[[resource/...]]` +- ✅ **dream step 实现** —— Phase 1 extract + Phase 2 integrate(per sub-unit)+ E-1 守恒校验(`reme4/steps/evolve/dream/`) +- ✅ **边守恒校验工具** —— `digest_edit` 的 outbound diff 比较器 + REJECT_CONSERVATION 重试 + 违规上报 +- ✅ **provenance prompt 规范** —— `derived_from:: [[daily/...]]` / `[[resource/...]]` 强制 +- ⏳ **bucket 集合配置外置** —— 当前 hardcoded 在 `digest_write.py` 的 `DEFAULT_BUCKETS`;目标 `vault.yaml` schema + `_buckets.md` 视图生成 -实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_memory_design.md` / `auto_maintain_design.md` / `auto_link_design.md` 共同作为契约依据。 +实现进入 `reme4/steps/evolve/` 时,本文档与 `auto_memory_design.md` / `auto_consolidate_design.md` / `auto_cognition_design.md` 共同作为契约依据。 diff --git a/docs4/auto_link_design.md b/docs4/auto_link_design.md deleted file mode 100644 index 4ff28bbd..00000000 --- a/docs4/auto_link_design.md +++ /dev/null @@ -1,209 +0,0 @@ -# auto-link 设计(背景实体识别 + wikilink 写回) - -> 本文档记录 reme4 中 **auto-link** 的设计讨论 —— 在已写入节点之间发现隐含关系,把这些关系作为 `[[...]]` wikilink **写回 body**,形成可见、可编辑的图结构增强。 -> -> 配套阅读: -> - `structure.md` §1.2(三层数据视角)/ §4(retrieve 三种问法) -> - `auto_memory_design.md`:auto-link 可反向扫 daily event,补实体 wikilink(daily → digest) -> - `auto_dream_design.md`:wikilink 模型(§3 边语法 / §4 演化 / §4.4 边守恒 E-1 / E-2 / E-3);auto-link 借这套基础设施 -> - `auto_maintain_design.md`:CAS 写入协议(§5);auto-link L1 写回与 dream / maintain split 三方共用同一套 CAS -> -> **三层对应**:reme 服务整体三层 —— auto-memory / auto-dream / **auto-link(本文档)**。auto-link 是图关系的**后置增强** —— 在已落地的 vault 上做实体识别 + wikilink 写回,补足 content link(写记忆时由 LLM 直接产生的 `[[...]]`)在长 tail 隐含关系上的盲区。 -> -> **核心立场**:auto-link **写回 body**,不只是产报告。生成的 wikilink 是**可见、可编辑**的(写在 Markdown 文件里),agent / 人可后续 curate。auto-link 不引入新材料,纯 additive 插入 wikilink,天然满足 E-1 守恒;复用 dream 的 CAS 写入协议,不引入新基础设施。 - ---- - -## 0. 问题陈述 - -content link(`auto_dream_design.md` dream / split 写入时由 LLM inline 产生的 `[[...]]`)解决了"写记忆时显式的关系"。但有一类关系不会在 inline 写入时自然涌现,需要后台扫描已写入的 vault 才能识别: - -1. **历史 body 的实体未链接** —— dream update 时 LLM 关注新材料融入,可能忽略已有 body 中某个未链接的实体(例如 body 提到 "JWT" 但没写 `[[digest/auth/jwt-overview.md]]`) -2. **跨节点 / 跨桶的隐含关联** —— 节点 A 提到 "rate limit",但 `digest/api/rate-limit.md` 是后来才被 split 创建 → A 写入时没机会建立这条边 -3. **同主题未连 / 同概念重复** —— dream 漏判去重把同概念建成两个节点;或两个主题相关但 0 链接的节点彼此不知晓 - -auto-link 承担这部分:**后台扫描已写入节点 → 实体识别 / 候选挖掘 → wikilink 写回 body**。 - ---- - -## 1. 已对齐决策 - -### 1.1 与 content link 的边界 - -| 维度 | content link(在 dream) | auto-link(本文档) | -|---|---|---| -| 何时产生 | 写记忆 inline:dream update / M split prompt | 后台扫描:离线 / 周期 / 触发后异步 | -| 由谁产生 | LLM 在 dream 写入流中顺手写出 | LLM 在 auto-link 扫描流中识别后写出 | -| 输入 | 新材料 + 召回候选节点 | 已写入 body + 全 vault 索引 | -| 改 body | 是(重写整段 body) | 是(纯 additive 插入 wikilink,不改文字) | -| 守恒 | E-1 强守恒(out ⊇ old) | E-1 天然满足(纯增) | -| 用途 | 写入即关系明示 | 弥补 inline 漏判,挖掘长 tail 关系 | - -### 1.2 写回模型:纯 additive,复用 dream CAS - -auto-link 写回是**纯 additive** 操作 —— 在已有 body 文字中找到实体 mention,替换为 wikilink 形态: - -``` -Before: "JWT 轮换的核心是密钥派生 ..." -After: "[[digest/auth/jwt-rotation.md|JWT 轮换]]的核心是[[digest/auth/jwt-key-derivation.md|密钥派生]] ..." -``` - -| 维度 | 决策 | -|---|---| -| **alias 必须保留原文** | `[[path.md\|<原文>]]` 形态;原文一字不改 —— 守住"不改写其它节点正文" (`auto_dream_design.md` §4.3 F-2) 的精神 | -| **predicate 默认为空** | auto-link 默认产生无谓词 wikilink;升 typed link 走 L3(详 §1.3) | -| **不引入 anchor** | 与 dream 一致(`auto_dream_design.md` §3);target 永远是节点路径 | -| **CAS 写入** | 完全复用 `auto_maintain_design.md` §5 的 read-stamp + CAS-write 协议(冲突重做 ≤ 3 次) | -| **E-1 守恒** | 纯 additive:`new outbound = old outbound ∪ {new wikilinks}`;`new ⊇ old` 天然满足,守恒校验默认通过 | -| **rollback** | 若 auto-link 误插入(例如 entity mention 是同名歧义),走标准 edit 或 retarget 撤销;auto-link 不维护"我插过哪些"audit log(留给 SDK 决定) | - -**为什么是 additive 而不是重写**: -- additive = 0 文字风险(原文不变,只在原 mention 周围加 `[[ | ]]` 包装) -- 重写 = 触发完整 E-1 守恒校验 + LLM 重写整段语义守恒 prompt + 多次 LLM 调用 = 跟 dream update 重复 -- additive 失败可见:产生坏 wikilink 时,人/agent 直接编辑 body 修就行 - -### 1.3 候选挖掘类型(L1-L4) - -| # | 类型 | 描述 | 写回形态 | -|---|---|---|---| -| **L1** | **实体识别**(主路径) | 扫 body,识别已是 digest 节点的实体名(模糊匹配 + 语义召回);未被 wikilink 化的 mention → 加 `[[path.md\|]]` | additive wikilink 插入 | -| **L2** | **同主题未连**(旧 D7) | 两个 digest 节点谈相关主题但 0 wikilink → 候选 add link;LLM 判后在 body 末尾追加一句引用 | additive(在合适位置 / 节末追加 `参见 [[other.md\|other]]`)| -| **L3** | **隐含 predicate 推导** | 已有 `[[A]]` 但 LLM 可推断关系类型(`is_a` / `causes` / `extends` / ...)→ 升级为 typed link | 改 `[[A]]` → `is_a:: [[A]]`(predicate 升降级走显式 audit,详 §2.1)| -| **L4** | **重复语义检测**(旧 D8) | 两个节点描述同一概念但被独立 create(dream 漏判去重)→ 候选 merge | **不写回**;产报告 + 提示人/agent 触发 dream update 路径手工合并 | - -**L1 是主路径** —— 它是 auto-link 最核心、最频繁、最高 ROI 的操作:每个 digest 节点写完后,后台扫一遍 body,找未链接的已知实体,additive 加 wikilink。 - -**L2-L3 是辅助** —— 周期扫,产候选,LLM 终判,写回部分(L2 节末追加 / L3 升 predicate)。 - -**L4 不写回** —— 节点合并是结构改动,影响 E-1 守恒边界 + inbound 链路 + provenance 链路,不适合自动写;auto-link 只产报告,人/agent 决定走 dream update 路径解决。 - -### 1.4 触发节奏 - -| 模式 | 何时 | 适用 | -|---|---|---| -| **inline post-write**(默认) | 每次 dream update / M split 写完 body → enqueue auto-link L1 job(异步,FIFO,CAS 保护)| L1 实体识别;反应即时,与 D3 写后检测同节奏 | -| **周期 batch**(可选)| cron(daily / weekly)扫全 vault | L2 / L3 候选挖掘;成本可控 | -| **手动触发** | SDK / 人显式调用 | 全量重扫 / 修复 | - -**L1 inline 的必要性**:新 split 出的 child 节点立即被既有 body 引用(用 wikilink 而非纯 mention)的关键 = 写入即扫描;不 inline 会让"刚创建的 child 节点"在很长时间内只有 split parent 一个 inbound,中心性失真。 - -**已排除**: -- inline 时同步 auto-link(阻塞 dream return)—— 时延不可接受;auto-link 始终异步 -- 所有 L\* 都 inline —— L2-L3 候选挖掘 RTL 跨节点,成本高,只适合 batch -- 全 cron 唯一触发 —— L1 滞后过久,新节点孤岛 - -### 1.5 中心性算法(retrieve 加权依赖) - -retrieve 时节点权重 = base × intent 调节 × **中心性增益**(详 `auto_dream_design.md` §5)。中心性需要 auto-link 这一层提供 —— content link 给底子,auto-link 补 long tail,二者合起来才是完整的图。 - -| 选项 | 优点 | 缺点 | -|---|---|---| -| **简单入度** | 实现最简;split parent 入度天然高;auto-link L1 加边后入度即时反映 | 不区分"权威节点"vs"被随手提的节点";高入度 ≠ 高权威 | -| **PageRank** | 经典;权威性传递 | 实现复杂 + 增量更新成本(每次写边重算成本高,需 incremental algorithm)| -| **eigenvector centrality** | 与 PageRank 相近 | 同上 | - -**首版决策**:**简单入度**(file_graph 已有 inbound 链表,O(1) 查);auto-link L1 加边后入度立刻更新,split parent 自然涌现高入度。dogfooding 后视 retrieve 质量演进。 - -中心性是 retrieve 时**查询时计算**,不预存: -- file_graph 已建反向索引(inbound),计算 `len(inbound(node))` 是 O(1) -- 不预存避免"加边后中心性陈旧"问题 -- PageRank 演进时可加增量计算 + 周期 refresh - ---- - -## 2. 待对齐边界点 - -### 2.1 L3 predicate 升降级的 audit - -L3 把 `[[A]]` 升级为 `is_a:: [[A]]` 时,**改了 edge identity** —— `(target, None)` 变成 `(target, "is_a")`,在 E-1 守恒视角下 = 删一条边 + 加一条边: - -``` -old outbound: {(A, None)} -new outbound: {(A, "is_a")} -diff: missing = {(A, None)}; added = {(A, "is_a")} -``` - -不打 audit 走默认会被守恒校验拦下(`missing != ∅` → 重试 / 拒写)。 - -**决策方向**: -- L3 写入必须打 audit flag(消费层意图:升级 predicate,允许 drop + add 同时发生) -- audit flag 由 reme4 step 暴露(`maintainer_step(action="predicate_upgrade", from=..., to=...)`),不放在普通 write 路径 -- 普通 dream / auto-link L1 写入永远不带 audit flag,守恒校验照常严格 - -详细 audit flag 接口形态留到 SDK 阶段。 - -### 2.2 多歧义实体识别 - -L1 扫 body 找 "JWT" 这个 mention,vault 中有 `digest/auth/jwt-overview.md` 和 `digest/payment/jwt-payment-flow.md` 两个 candidate: - -候选方案: -- LLM 上下文判 —— 把 body 周围段落给 LLM,选最相关 target -- 跳过模糊 case —— L1 只处理 unambiguous mention,歧义 case 留人/agent -- 全部链 —— `[[overview]][[payment-flow]]`,后续人 curate - -**首版**:LLM 上下文判(每个候选 candidate 提供 description / 周围若干节点 summary,LLM 选择 top-1 或 drop);成本可接受(扫描已是离线 batch)。 - -### 2.3 auto-link 写回与 dream / split 的并发 - -auto-link 写 body 走 §1.2 CAS,但有特殊情况: -- 同节点同时被 dream update 与 auto-link L1 写入 → CAS 协议自动序列化 (`auto_maintain_design.md` §5):后到者重做 -- auto-link L1 写完后立刻被 dream update 覆盖(dream 重写 body) → 看 dream prompt 是否守住 auto-link 加的 wikilink(E-1 强守恒 → 守住) -- auto-link L1 与 D3 派发的 split job 同节点并发 → split 先到 / 后到都不影响最终拓扑(split 把 body 拆成 parent + children,auto-link 加的 wikilink 跟着对应内容段自然分配到 parent / child) - -**结论**:CAS + E-1 + E-2 守恒已覆盖所有并发场景,auto-link 不需要新协调机制。 - -### 2.4 跨 vault / 跨进程 - -M0 单 reme 实例 + 单 vault,auto-link 走内进程 enqueue;多实例 / 跨进程留 M1+(同 `auto_maintain_design.md` §5)。 - -### 2.5 实体识别 vs 现成 NER 库 - -L1 实体识别可选: -- LLM 直接扫(贵但灵活,与 digest 节点同构) -- 现成 NER 库(spaCy 等)预筛 + LLM 终判(快但 entity 类型与 digest 节点形态可能不匹配) -- 纯字符串匹配(已知节点名字 + 简单变体)+ LLM 终判 ambiguity - -**倾向**:从纯字符串匹配 + LLM 终判 ambiguity 起步(实现最简,效果可能已经够好);视 dogfooding 决定是否引入 NER 库。 - ---- - -## 3. 与其它层的协作 - -| 上下游 | 关系 | -|---|---| -| ← **auto-dream** | dream 写完一个节点 → 通过 inline post-write enqueue auto-link L1(§1.4);auto-link 用 dream 的 CAS 协议 | -| ← **auto-memory** | auto-link 可反向扫 daily event,把实体识别成 `[[digest/...]]`(daily → digest);auto-memory 写入端不主动调 auto-link,触发同 dream 路径 | -| → **digest body** | 主要写入对象 —— L1 additive 加 wikilink / L2 节末追加引用 / L3 升 predicate(走 audit) | -| → **daily body** | auto-link 扫 daily event 时同样可加 `[[digest/...]]`(I-2 daily 单作者需协调:auto-link 应在 event 关闭后才动该 event,不与 active event 并发改;实现细节留 step 层处理) | -| → **resource body** | I-3 immutable;auto-link **不写 resource**(reading-only) | -| → **L4 候选 report** | L4 重复语义检测产报告,落 `audit//auto_link_l4.md`(具体路径 / 形态留 step 层) | - ---- - -## 4. 与 auto-dream 模型的引用关系 - -本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中: - -| 引用 | 来源 | -|---|---| -| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §3 | -| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 | -| F-invariants(F-1..F-11)| `auto_dream_design.md` §4.3 | -| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 | -| 路径即 ID / rename | `auto_dream_design.md` §2 | -| CAS 写入协议 | `auto_maintain_design.md` §5 | -| anchor 不引入 | `auto_dream_design.md` §3 | -| SearchStep 召回 | `auto_dream_design.md` §4.2 | - ---- - -## 5. 下一步 - -1. **L1 实体识别 step 实现** —— 字符串匹配 + 语义召回 + LLM ambiguity 终判 + additive wikilink 写回(§1.2 / §1.3) -2. **inline post-write trigger 接入** —— dream update / M split CAS 写入成功后 enqueue auto-link L1 job(§1.4) -3. **L2 / L3 周期 batch 框架** —— cron(daily / weekly)+ 候选挖掘 prompt + 写回路径(§1.3) -4. **L3 audit flag 接口** —— `maintainer_step` 提供 `predicate_upgrade` 操作,带 audit context 走特殊守恒规则(§2.1) -5. **中心性 retrieve 增益** —— file_graph inbound count → retrieve 加权乘子(§1.5) -6. **L4 报告框架** —— 重复语义检测产报告,提供 SDK / 人介入入口(§1.3 / §3) - -实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` 共同作为契约依据。 diff --git a/docs4/auto_maintain_design.md b/docs4/auto_maintain_design.md deleted file mode 100644 index 5f6f923a..00000000 --- a/docs4/auto_maintain_design.md +++ /dev/null @@ -1,190 +0,0 @@ -# auto-maintain 设计(digest 组织端:M split / 检测 / 写入并发) - -> 本文档记录 reme4 中 **auto-maintain** 的设计讨论 —— digest 层的组织 / 重组 / 写入运行时,含 M split、D 检测信号、写后触发模型、CAS 写入协议。 -> -> 配套阅读: -> - `structure.md` §3.6(maintain 动作语义)/ §7.3(maintainer 模块) -> - `auto_dream_design.md`:节点 + 边模型(§1-§4)/ F-invariants(§4.3)/ 边守恒 E-1/E-2/E-3(§4.4)/ dream 操作(§4.2)—— maintain 复用这套底层模型 -> - `auto_link_design.md`:auto-link 写回也走本文档的 CAS 协议(§5) -> - `auto_memory_design.md`:auto-memory 不直接复用 maintain,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小) -> -> **三层框架的位置**:报告 §5 三层为 auto-memory / auto-dream / auto-link。maintain 严格按 `structure.md` §3.5-3.6 的 L4 action 分类是独立 action(`maintain: digest → digest`),不属 `digest` action(`digest: resource + daily → digest`)。本文档作为四方分工的**第四份**,专门覆盖 dream 写完之后 digest 的组织 / 重组 / 写入运行时。 -> -> **核心立场**: -> - **maintain 与 dream 同 pace**(idle background)、同模型(节点 + 边 / 守恒规则),但**语义边界不同**:dream 是 compose(资料 → digest),maintain 是 reorganize(digest → digest) -> - **maintain 只做 split**,不做 merge / dissolve / re-edge / unify;过载就拆,其它跨节点重组留给消费层 / 人工 -> - **CAS 写入协议是基础设施**,被 dream / maintain split / auto-link L1 共用,统一编排在本文档(§5) - ---- - -## 0. 问题陈述 - -dream 模型(`auto_dream_design.md` §4)规定 digest 的演化只做两件事:dream create_or_update(入流型,新材料融入)+ M split(后台,过载就拆)。dream 文档负责 dream 与节点 / 边模型;**本文档负责 M split 与运行时机制**(D 检测 / 触发模型 / 写入并发协议)。 - -| 输入 | 输出 | -|---|---| -| dream 写入后的 digest 状态 + 触发信号(D3 过载,inline) | parent overview 重写 + N 个新 children 文件;边守恒 E-2 通过 | - -**设计目标**: -1. **形状匹配** —— 让节点粒度持续与实际语义结构对齐(过载节点拆;不过载不动) -2. **最小变更面** —— split 改 parent + 创建 N children,不动其它节点(F-2) -3. **不引入新基础设施** —— 复用 dream 的节点 + 边模型 / 守恒规则;CAS 写协议自洽 - -**显式排除**: -- ❌ merge / dissolve / re-edge / unify —— 跨节点重组不做(简化模型;同概念二次进入靠 dream update) -- ❌ 改其它节点正文 —— split 只改 parent body(重写为 overview)+ 创建 children body -- ❌ 重建 inbound —— split 时 inbound 一律不动(F-10) - ---- - -## 1. M split(maintainer 唯一 op) - -| # | 能力 | 服务 | 触发 | graph | file | body | -|---|---|---|---|---|---|---| -| **M split** | 节点过载 → LLM 拆成 parent overview + N 个 children;parent 文件原地保留,children 是新文件;children 加 `[[parent]]` 反向链接;inbound 边不动 | 形状匹配(粒度对齐)+ 任意尺度(涌现层级) | D3 过载 | parent 0 拓扑改;新 children 节点 + 各自加 `[[parent]]` 出边 | 创建 N 个 children 文件;parent 文件原地 | parent body 重写为 overview;children 各自有新 body | - -**关键边界**: -- **M split 改两类 body**:parent body(重写为 overview)+ N 个新 children body;不改任何**其它**节点(`auto_dream_design.md` §4.3 F-2) -- **inbound 不重定向** —— 外部对 parent 的 wikilink 全部保留指 parent;后续 dream 进入时若 LLM 觉得 child 粒度更合适,直接加新边到 child 即可(F-10) -- **没有 dissolve 操作** —— children 长期空也不主动删;消费层 / 人工显式介入 -- **没有 merge / re-edge / unify** —— 跨节点重组不做;同概念二次进入靠 dream update;错桶节点不主动 move(若严重,人工介入) -- **边守恒** —— split 写新 parent body + N children body 前,机械对比 outbound:`(parent_new ∪ ∪children_outbound) ⊇ parent_old`;失败 → LLM 重试或拒写(F-11 / E-2,详 `auto_dream_design.md` §4.4) - ---- - -## 2. 检测信号 D1 / D3 / D10 - -| # | 信号 | 服务 | 服务能力 | -|---|---|---|---| -| **D1** | 断链(wikilink → 不存在的 path) | 任意尺度(可达性) | 告警 / 简单修复(就地删 wikilink 或保留 alias 文本) | -| **D3** | 过载节点(token 阈值 → LLM 判离散度) | 形状匹配(粒度) | maintainer(M split) | -| **D10** | provenance 断裂(digest 节点反指的 daily/resource 不可达) | 任意尺度(跨层不变量) | 严重告警(I 不变量违反) | - -**触发模型**:**写后立即** —— dream / split 写完 body inline 检测;无后台 watcher / 无周期 tick / 无 dirty 队列(详 §4)。D1 / D10 是 wikilink 断链的子集,跟 file_graph 链路一起在写时检测。 - -> **简化模型砍掉的信号**: -> - **D2 隔离 / D4 过疏 / D5 高入度 / D5b 低入度摘要 / D6 slug 冲突 / D7 相似未链 / D8 重复语义 / D9 邻居异质** —— 全部 DROPPED -> - 旧 D5 高入度涌现 → 由 split 副产品(parent + children)等价覆盖;触发源换成节点过载(D3) -> - 旧 D6 slug 冲突 → 路径即 ID 后,同 bucket 内文件名冲突由文件系统层断言(写入即拒),不需要独立信号(详 `auto_dream_design.md` §2) -> - 旧 D7 / D8 → 简化模型不做 link / merge 提议;若 vault 累积明显的同概念重复,由 `auto_link_design.md` §1.3 L4 离线 audit 工具产报告 -> - 旧 D9 邻居异质 → 简化模型不做跨桶 move;桶选择只在 dream 桶决策一次性决定,后续不重排 -> -> **D3 过载的判据**:token 阈值机械检查 + LLM 判离散度;**写后立即 inline**。阈值见 §3,触发模型见 §4。 - ---- - -## 3. 检测阈值校准 - -简化模型只剩 D3(过载)是核心阈值,其它都是 invariant 触发(无可调阈值)或 informational(无 maintenance 联动)。 - -| 信号 | 阈值类型 | 默认 | 备注 | -|---|---|---|---| -| **D3 过载** | token + 主题离散度 | token 2000 / 离散度由 LLM 写后 inline 判 | **唯一驱动 split 的阈值**(详 §4) | -| **D1 断链** | 0 容忍 | 任意 1 条断链 → 告警 | 修复策略简单(就地删 wikilink) | -| **D10 provenance 断裂** | 0 容忍 | 任意 1 条断裂 → 严重告警 | I 不变量 | - -D3 阈值作为 `vault.yaml` 配置项(opinionated default,reme 核心提供机制不写死阈值),消费层可改;dogfooding 后调优。token 阈值起点 2000(对应"约 5 个独立子主题"的常见过载点),首版可调。 - ---- - -## 4. D3 触发模型(已收敛) - -**决策**:**写后立即检测,无 watcher 抽象,无 batch 窗口** —— 每次 dream update / split 写 body 成功后,**inline** 在同一 job 内跑 D3:token 阈值 + LLM 离散度判定 → 必要时 enqueue split job(异步,走 §5 CAS 队列)。 - -``` -dream / split 写 body 成功(CAS 通过) - └─ if len(body) > T: - └─ LLM 判离散度 - └─ if is_overloaded: - └─ enqueue split job (FIFO, CAS-protected) - └─ return -``` - -**协议**: -- token 阈值默认 `2000`(§3 已定,`vault.yaml` 可配) -- 离散度 prompt 输出 `{is_overloaded: bool, suggested_clusters: [...]}`(若 overloaded 直接供 split job 吃,不重判) -- 启动无全扫(避免长启动);新写入立即检测覆盖增长路径;历史遗留过载随下次 update 自然检出 -- 无 dirty 标 / 无 dirty 集合 / 无后台 worker —— D3 是写路径的合成函数 - -**为什么 inline**:反应即时(不等下一次 ingest);实现最简(无批处理窗口 / dirty 状态 / 独立 worker);LLM 判定成本可接受(大多写入 < T 不触发,触发后 split 切小后续不再越界);不引入 watcher = 少一层部署/监控。 - -**已排除**:定时 cron tick(静止 vault 浪费扫描)/ ingest-after batch(引入 dirty 集合)/ 独立 L2 watcher worker(多余部署层)。 - -**演进路径(M1+)**:若 inline LLM 阻塞 dream 时延成问题 → D3 改为 fire-and-forget enqueue;若同节点重复触发 LLM 成本高 → 加节点级 body hash 缓存。 - ---- - -## 5. CAS 写入协议(共享基础设施) - -**位置说明**:CAS 是 dream update(`auto_dream_design.md` §4.2)、M split(本文档 §1)、auto-link L1 写回(`auto_link_design.md` §1.2)**三方共用**的写入协议。归在本文档是因为 maintain 是 digest 的"组织 / 运行时"端,运行时机制(检测 / 触发 / 写入)集中在一处方便对照。 - -**决策**:**并行决策 + 乐观冲突重做(CAS)** —— 所有 dream / split / auto-link L1 决策并发跑,写入前用 body 版本戳(hash / mtime)做 CAS 比对;变了就丢弃 planned body 重做。无锁,无 ingest 级互斥。冲突率低 + E-1 / E-2 守恒校验顺手承担 race 兜底,无需新基础设施。 - -**协议(单个写入调用)**: -1. **读 + 记戳**:读 subject body → `version_stamp = sha256(body) | mtime` -2. **决策**:LLM 看候选池 → 决定 create / update / drop / split / additive-link;产 planned new_body -3. **CAS 写入**:重读 body 比 version_stamp - - **未变**:跑 E-1 / E-2 守恒校验 → 通过则 atomic write(write-temp + rename)→ done - - **已变**:丢弃 planned new_body,带最新 body 重走 step 1 -4. **守恒校验失败**:走 `auto_dream_design.md` §4.4 既有重试路径(LLM 重试一次,二次失败拒写 + audit) -5. **重做次数上限**:CAS-冲突重做最多 3 次;超出 → 跳过候选 + audit log(避免活锁) - -**create 路径 race**:两个 dream 都决定 `create digest/auth/jwt-rotation.md` → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者拿 EEXIST → 重走 step 1(此时大概率改判 update)。 - -**适用范围**(全部走同一套 CAS):同 ingest 内 N 个候选并发 / 跨 ingest job 并发 / 后台 split 与前台 dream 命中同节点(split 同样走 CAS)/ auto-link 写回(`auto_link_design.md` §1.2)。 - -**不解决的**:高冲突 workload(同概念被反复 ingest)→ 重做上限触发后 audit;跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+)。 - ---- - -## 6. split 时的 provenance 处理(已收敛) - -**坍缩到 E-2 合计守恒** —— provenance 是 body 内联 wikilink(`auto_dream_design.md` §4.2),split 时跟其它 body 边完全同形:LLM 把 parent body 拆成 parent overview + N children,provenance wikilink 跟着对应内容段自然分配;机械层 outbound 合计守恒校验保证 `(parent_new ∪ ∪children_outbound) ⊇ parent_old`,旧 provenance 不可能丢。无需专门的 provenance 分配逻辑或"全部复制到 child / parent 保留全量"等特殊策略 —— LLM 按"哪个 child 谈到了哪段上游就带走哪条 provenance"自然处理。 - ---- - -## 7. dream / split / auto-link L1 时序(已收敛) - -时序由 §4 / §5 与 `auto_dream_design.md` §4.2 共同规定,这里给最小汇总: - -- **dream 调用本身同步** —— material 进来就走 dream 决策(召回 + LLM 终判)+ CAS 写入(§5) -- **D3 检测 inline** —— dream / split 写完 body 顺手跑 token 阈值 + LLM 判离散度(§4),无 tick / batch / watcher -- **split 异步** —— D3 触发后 enqueue split job 进 §5 CAS 队列,跟其它 ingest / split job FIFO 共享,异步消费;**不阻塞 dream return** -- **auto-link L1 异步** —— 写入成功后 enqueue auto-link L1 job(`auto_link_design.md` §1.4),与 split job 同 CAS 队列、FIFO 共享;不阻塞 dream return - -检测延迟 ≈ 0(inline);split 执行延迟 ≈ 队列等待时间(typically 数秒~数十秒);新建 / update 节点不必等 split 完成,体验连续。 - ---- - -## 8. maintainer 的人 / agent 后门(暂缓 — 非底层) - -消费层 / SDK 接口问题,不影响底层机制。底层只需保证 split / rename / delete 等 op 走同一套 §5 CAS + 守恒校验链路:F-3 仍成立(maintainer 自动路径只做 split);merge / dissolve / re-edge 在底层**不存在**(无对应机械算子)。后门接口形态推迟到 SDK 阶段再定。 - ---- - -## 9. 与 dream 模型的引用关系 - -本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中: - -| 引用 | 来源 | -|---|---| -| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §3 | -| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 | -| F-invariants(F-1..F-11) | `auto_dream_design.md` §4.3 | -| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 | -| 路径即 ID / rename | `auto_dream_design.md` §2 | -| anchor 不引入 | `auto_dream_design.md` §3 | -| provenance 载体形态 | `auto_dream_design.md` §4.2 | -| dream 行为 | `auto_dream_design.md` §4.2 | - ---- - -## 10. 下一步 - -1. **M split step 实现** —— D3 触发 → 候选 → split prompt → 写入 + E-2 守恒(§1 / §4 / §5) -2. **D 检测信号实现清单**(D1 断链 / D3 写后 inline / D10 provenance 哪些已就绪 / 缺哪些)—— §2 -3. **D3 阈值配置**(`vault.yaml` 中 D3 token / 离散度阈值)—— §3 -4. **CAS 写入框架** —— per-path body version_stamp + CAS 写入 + EEXIST create race + 重做上限 + audit;对外暴露给 dream / auto-link L1 复用 —— §5 -5. **后门 SDK 接口形态**(暂缓 M1+)—— §8 - -实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。 diff --git a/docs4/auto_memory_design.md b/docs4/auto_memory_design.md index 7747cb89..76881eb3 100644 --- a/docs4/auto_memory_design.md +++ b/docs4/auto_memory_design.md @@ -5,10 +5,10 @@ > 配套阅读: > - `structure.md` §2.1-2.2(daily 层定位)/ §3.4(sync 动作语义)/ §7.1(synchronizer 模块) > - `auto_dream_design.md`:auto-memory 产物如何被 dream 消化(dream 读 daily 作为入流之一) -> - `auto_maintain_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小) -> - `auto_link_design.md`:auto-link 可反向扫 daily 事件,补充实体 wikilink(daily → digest) +> - `auto_consolidate_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小) +> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想(写入 / 巩固 / 检索);daily 节点是 cognition 图视图的一部分(承载 `derived_from::` 反指),但不参与 Stage 2 巩固改造 > -> **四份分工**:reme 服务整体四份设计 —— **auto-memory(本文档)** / auto-dream / auto-maintain / auto-link。auto-memory 是入流端,把 agent 实时事件流切成 daily 事件原子;它的产物是 dream 消化的两路输入之一(另一路是 resource)。 +> **服务全景**:reme 服务两条主线 —— **auto-memory**(本文档,入流端 / daily 写入)与 **auto-cognition**(顶层思想:写入 = auto-dream,巩固 = auto-consolidate,检索 = auto-recall)。auto-memory 把 agent 实时事件流切成 daily 事件原子;它的产物是 dream(cognition Stage 1)消化的两路输入之一(另一路是 resource)。 > > **核心立场**:auto-memory 是 `structure.md` §3.4 `sync` 动作的实现侧 —— 强调 **inline 实时**与**事件边界检测**。是不是改名 sync → auto-memory 留给上层文档对齐,本文档聚焦机制。 @@ -29,7 +29,7 @@ agent 的对话与任务过程是连续事件流(用户回合、工具调用、 **显式排除**(不属于 auto-memory 职责): - ❌ 蒸馏 / 沉淀:那是 auto-dream(`auto_dream_design.md`)的事 -- ❌ 实体识别 / wikilink 自动补全:那是 auto-link(`auto_link_design.md`)的事 +- ❌ 实体识别 / wikilink 自动补全:cognition 三阶段不在写入后做"事后补 wikilink"(详 `auto_cognition_design.md` §1.1);所有 wikilink 由 dream 在写入瞬间产出 - ❌ 改写 resource / digest:auto-memory 只写 daily(I-1 / I-3) --- @@ -180,9 +180,9 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective | ← **resource** | 只读(通过 wikilink 引);不写 | | → **daily** | **唯一写者**(I-2);写 event folder + 主索引 | | → **auto-dream** | dream 读 daily 作为入流(`auto_dream_design.md` §4.2 dream scope);auto-memory 写完即对 dream 可见(走 L2 索引,有 eventual 窗口) | -| → **auto-link** | auto-link 可反向扫 daily event,做实体识别 + wikilink 写回(`auto_link_design.md` §1.3)—— 与 auto-memory 写入不冲突(双方写不同字段段落 / CAS 协议保护)| +| → **auto-cognition (三阶段)** | daily 节点是 cognition 图视图的一部分;dream(Stage 1)读 daily 作为入流;consolidate(Stage 2)只对 digest 节点跑 dups / community / decay,**不改 daily**;recall(Stage 3)三层并行召回时 daily 也参与命中 | -**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;dream / link 不写 daily 主路径,只通过 auto-link 走 §1.3 写回(read-only audit-then-write,CAS 保护)。 +**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;cognition 三阶段没有任何子阶段会**事后改写 daily**(无写回路径)。daily 一旦由 auto-memory 写完,就只被读不被改(I-2 / I-3 仍守);后续 dream / consolidate / recall 都是只读消费。 --- @@ -194,4 +194,4 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective 4. **多 agent 隔离 schema**(M1+):若实际有并发 agent,确定 daily 子目录 / slug 命名约定 5. **粗 / 细粒度 prompt 调参**:dogfooding 后看实际 event 数 / dream 消化效率,调 boundary prompt -实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。 +实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md` 共同作为契约依据。 diff --git a/docs4/auto_recall_design.md b/docs4/auto_recall_design.md new file mode 100644 index 00000000..bf638b20 --- /dev/null +++ b/docs4/auto_recall_design.md @@ -0,0 +1,323 @@ +# auto-recall 设计(Stage 3 检索:信号融合 + 召回增强) + +> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 3 — 检索阶段** 实现。覆盖 query 到来时如何把 vault 一等公民信号(wikilink 图 / frontmatter)与维护阶段产出信号(centrality / community / recency / archived)融合,生成最终召回。 +> +> 配套阅读: +> - `auto_cognition_design.md`:三阶段顶层思想(本文档是 Stage 3) +> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型 +> - `auto_consolidate_design.md`:Stage 2 维护 —— **本文档消费它产出的所有 `meta/*.json`** +> - `structure.md` §4(retrieve 三种问法)/ §7.4(为什么没有 retriever 模块) +> - `reme4/steps/index/search.py` / `traverse.py`:现有原子实现 +> +> **核心立场**: +> - retrieve **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法各自由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖 +> - 本文档增强**集中在 `search_step` 内部**:把维护信号融入打分 / 排序 / 过滤;`traverse_step` 仅做小幅参数扩展 +> - retrieve **只读 vault,不写 body / 不写 frontmatter**;唯一写入是 `meta/access_log.json`(命中计数,供下次 recency 计算) + +--- + +## 0. 问题陈述 + +`structure.md` §4 已规定 retrieve 三种问法(state / semantic / topological)正交分立(R-1)。本文档**只增强 semantic 问法**;state 问法已被 `list_step` 覆盖,topological 问法已被 `traverse_step` 覆盖。 + +semantic 问法当前在 `reme4/steps/index/search.py` 实现: + +| 已就绪 | 缺口 | +|---|---| +| ✅ vector + keyword 并行召回 | ❌ 节点中心性加权(高权威节点不被 boost) | +| ✅ RRF fusion(vector_weight=0.7) | ❌ 同社区 boost(`meta/communities.json` 未消费) | +| ✅ 一跳 expand_links(向前向后,max=10) | ❌ 时效衰减 / 冷藏过滤(`meta/access_log.json`、`meta/archived.json` 未消费) | +| ✅ min_score 过滤 + limit 截断 | ❌ 同 file 多 chunk 冗余(top-K 可全来自同节点) | +| ✅ chunk-level 命中(start_line / end_line) | ❌ 节点级 surface(frontmatter `name + description` 未与 chunk 命中合并展示) | +| ✅ 二跳 traverse 作为独立工具 | ❌ search 内 multi-hop expand(只一跳,跨术语关系到不了) | +| | ❌ query rewrite / multi-query(单一表达式漏召) | + +**本文档的工作 = 设计这些缺口怎么填**,在 `search_step` / `traverse_step` 现有形态上增量。 + +--- + +## 1. 三种问法分立(继承 R-1) + +``` +┌─────────────┐ state 问 ──────► list_step + frontmatter filter +│ agent │ semantic 问 ──► search_step (本文档主要增强) +└─────────────┘ topological 问 ► traverse_step (小幅参数扩展) +``` + +| 问法 | 原子工具 | 本文档涉及 | 备注 | +|---|---|---|---| +| **state** | `list_step` / `daily_list_step` / `frontmatter_read_step` | 不涉及 | frontmatter 过滤无需维护信号 | +| **semantic** | `search_step` | **主战场**(§3-§7) | RRF fusion + 信号加权 + multi-hop + query rewrite | +| **topological** | `traverse_step` | 小幅(§8) | 起点选择可借助维护信号 | + +**关键约束**(继承 `structure.md` ✗-8):**绝不合并三种问法成单一 read verb**。本文档增强 search_step,但不把 list / traverse 揉进 search;agent 按需各自调用。 + +--- + +## 2. 维护信号契约消费总览 + +`auto_consolidate_design.md` §11 列出维护产出。retrieve 端按以下方式读: + +| 信号 | 来源 | 加载时机 | 缺失行为(降级) | +|---|---|---|---| +| **centrality** | `file_graph` 反向索引(实时) | search_step init 时引用 file_store | 总在线(file_graph 是核心组件) | +| **community** | `meta/communities.json` | search_step 启动 lazy load(LRU 缓存,文件 mtime 失效) | 缺失 → 不做同社区 boost | +| **recency** | `meta/access_log.json` | 同上 | 缺失 → recency_factor = 1.0 | +| **archived** | `meta/archived.json` | 同上 | 缺失 → 不过滤,所有节点参与 | +| **wikilink 图** | vault 自身(file_graph) | 实时 | 总在线 | +| **frontmatter** | vault 自身(`name` / `description`) | chunk 已带 metadata | 总在线 | + +**version 校验**:`meta/*.json` 加载时检查 `version` 字段,与本文档约定的 schema 版本不匹配 → 走"该信号缺失"降级,日志告警(不崩)。 + +**新鲜度**:每个信号文件的 `computed_at` 暴露给调用者(metadata 中带 `signals_freshness`),调用方知道当前权重基于多久前的快照。超过阈值(默认 14 days)→ logger.warning + 仍使用(避免维护偶尔失效就拒绝服务)。 + +--- + +## 3. semantic 问法增强:打分公式 + +**目标**:把维护信号融入 fused chunk 的最终 score,让排序兼顾"文本相关 + 节点权威 + 同社区 + 时效"。 + +### 3.1 当前打分(基线) + +``` +score = RRF_fused(vector_rank, keyword_rank, vector_weight=0.7) +``` + +仅文本相似度。 + +### 3.2 新打分公式 + +``` +final_score = base_score + × centrality_factor(path) + × community_factor(path, query_seed_paths) + × recency_factor(path) +``` + +| 因子 | 公式 | 默认参数 | 来源 | +|---|---|---|---| +| **base_score** | RRF 融合分(现状) | vector_weight=0.7 | search.py | +| **centrality_factor** | `1 + α · log(1 + inbound_count)` | α = 0.15 | file_graph 实时 | +| **community_factor** | 同 community 命中节点 → ×β,否则 1.0 | β = 1.20 | `meta/communities.json` | +| **recency_factor** | `exp(-Δt / τ)`,Δt = 距 last_hit_or_update | τ = 60 days | `meta/access_log.json` | + +**为什么乘法而非加法**: +- 各因子量级不同(base_score ≤ 0.02,centrality 与 query 无关),加法需大量 normalization;乘法天然处理量级差 +- 任一因子接近 0(极冷藏 / 极孤立)→ 整体压低,符合"弱信号一票否决"直觉 +- 默认 α/β/τ 让 factor 落在 [0.5, 2.0] 区间,不会让 base_score 完全失声 + +**已排除**:LLM rerank。它是 query-time 多调一次 LLM,成本高,M0 不引入;留 M1+ 视 dogfooding 决定。 + +### 3.3 query_seed_paths 的角色 + +community_factor 需要"query 主关注的节点是哪些"才能判断同/异社区。做法: +1. RRF 融合后取 top-N(N=3)的 fused chunk 的 path 作 seed +2. 后续每个候选 chunk 的 path → 查它和任一 seed 是否同社区 → boost +3. 不需要 query 自身被映射到 community(query 是字符串,不在图里) + +**边界**:N=3 是经验起点;N 太大会让"同社区"几乎等于"全召回"失去区分度。dogfooding 后调。 + +--- + +## 4. semantic 增强:节点级合并(unique_paths) + +**问题(gap 5)**:fused 列表里 top-5 可能是同 file 的 5 个 chunk,信噪比退化。 + +**当前**:`expand_links` 已用 `unique_paths = list(dict.fromkeys(c.path for c in fused))`,但 fused 本身没去重,limit=5 仍可全是同节点。 + +**新方案**(节点级 dedupe + 节点级 surface): + +``` +fused (chunk-level) → group by path → 每组保留 top_chunks_per_path 个 + → 每组追加节点 frontmatter (name + description) 作"节点级 surface" + → 再按节点 best_score 排序 → limit +``` + +| 参数 | 默认 | 含义 | +|---|---|---| +| `top_chunks_per_path` | 2 | 同节点最多保留多少 chunk | +| `surface_node` | true | 是否在每组前追加 frontmatter `name + description` | + +**为什么**: +- 节点是 retrieve 的语义单位(`auto_dream_design.md` §2 路径即 ID),chunk 只是"展示窗口" +- frontmatter 是节点级摘要(name + description)—— 已是 dream 写入时认证过的信号,不召它浪费 +- 同节点多 chunk 时,frontmatter + top-2 chunk 比 5 个 chunk 信息密度高 + +### 4.1 答案展示形态 + +``` +========== digest/auth/jwt-rotation.md ========== +[node] JWT Key Rotation + Process for rotating JWT signing keys without downtime. +[score=0.0241 centrality=2.1 community=1.2 recency=0.91] + +---------- chunk @5-23 ---------- + + +---------- chunk @45-60 ---------- + + +[expansion] 1 inbound, 2 outbound (...) +``` + +**对照旧形态**:每个 chunk 独立成块,无节点级 surface,scores 散在 chunk 头。新形态以**节点为视觉单位**,人 / agent 看到的第一眼是"哪个节点中了",而非"哪段文字中了"。 + +--- + +## 5. semantic 增强:multi-hop expand + +**问题(gap 4)**:当前 expand_links 只展一跳,跨术语关系("分布式锁" → 一跳到"租约机制",再一跳才到"心跳协议")到不了。 + +**新方案**:expand_links 支持 `depth` 参数;默认仍 1(保守),agent / 配置可调到 2。 + +| 参数 | 默认 | 限制 | +|---|---|---| +| `expand_depth` | 1 | 最大 3(避免组合爆炸) | +| `max_links_per_direction` | 10(现状)| 每跳每方向上限,深度不展开时限到当跳总数 | +| `expand_path_budget` | 30 | 总扩展节点数硬上限,优先深度优先(深度浅但条数少) | + +**为什么默认仍 1**: +- 二跳延迟不可忽略(N × 10 × 10 = 100 候选 IO) +- agent 需要"再深一层"时显式调 `traverse_step(depth=2)` —— 三种问法分立(R-1) +- 默认深拉会让"语义召回"变成"图召回",违背 R-1 + +**何时调 2**:dogfooding 发现 vault 节点平均出度低 / 跨术语关系频繁 → 调到 2(改 search_step 配置,不改协议)。 + +--- + +## 6. semantic 增强:query rewrite / multi-query + +**问题(gap 6)**:用户 query "JWT 怎么轮换" 可能错过 body 写"密钥定期更换"的节点(术语不同)。 + +**方案矩阵**: + +| 方案 | 成本 | 效果 | +|---|---|---| +| **(a) 不做** | 0 | 漏召部分跨术语 | +| **(b) embedding 多 query**(用同 LLM 生成 N 个表述) | LLM 调用 1 次(query → N 表述)+ N 次 vector_search | 中等 | +| **(c) BM25 同义词扩展**(用静态词表 / 嵌入式词表) | 0(若有词表) | 弱(中文场景词表缺) | +| **(d) HyDE**(LLM 生成假设答案 → 嵌入这个答案而非 query) | LLM 1 次 | 高,文献证实 | + +**首版决策**:**(a) 不做**。理由: +- vault 本身规模 M0 不大,推断增加召回但增 LLM cost 不划算 +- 维护阶段的 community 聚类已部分弥补"跨术语关系"(同社区 boost) +- 真要做,优先 (d) HyDE,延 M1+ 再启,实施只需加一层 query 预处理 + +**契约预留**:search_step kwargs 加 `query_rewrite: str | None`(默认 None;非 None 则用此重写代替原 query 做 vector_search,keyword_search 仍用原 query)。SDK 层可调用 LLM 生成重写后传入,reme 核心不强加 LLM 依赖。 + +--- + +## 7. semantic 增强:archived 过滤 + +**问题**:长期未访问的旧节点应该默认排除。 + +**方案**:search_step kwargs 加 `include_archived: bool`,默认 false。 + +``` +fused → drop where path in archived_set → 后续打分 / unique_paths +``` + +**何时绕过**: +- agent 显式 `include_archived=true`(找历史 / debug) +- query 命中节点本身在 archived → boost 推回(冷节点突然被命中,说明不是真冷) + - **首版不做**,过滤即过滤;如有需要,M1+ 加"intent override"机制 + +**冷启动**(`meta/archived.json` 缺失)→ 不过滤,等同 `include_archived=true`。 + +--- + +## 8. topological 问法的小增强 + +`traverse_step` 当前完整:BFS / 多 seed / direction / depth / per-edge 输出。本文档不重构,仅: + +### 8.1 起点选择借助维护信号(可选 hint) + +agent 调用 traverse 时往往不知道"哪个节点是该主题的中心";维护阶段产出的 centrality 可作 hint: + +| 用例 | 做法 | +|---|---| +| traverse 给定 seed | 不变,直接 BFS | +| traverse 给定主题字符串(SDK 上层语法糖) | 先 search_step 找 top-1 → 用其作 seed → traverse depth=2 | + +**位置**:这个组合在 SDK 上层做,不进 traverse_step;reme 核心保留 traverse 原子形态。 + +### 8.2 traverse 输出消费 archived + +traverse_step 当前不知道 archived 信号。改造:加 `exclude_archived: bool` kwarg 默认 false(traverse 默认不过滤,因为它是图问法,过滤会破坏图视角)。SDK / agent 可显式开启。 + +--- + +## 9. retrieve 写访问日志(唯一对外写入) + +**问题**:`meta/access_log.json` 的 `last_read` / `last_hit_count_30d` 谁写? + +**约定**:retrieve 命中节点 → 异步 append 到访问日志缓冲区;由 maintain daily batch 聚合写入 `meta/access_log.json`。 + +| 路径 | 实现 | +|---|---| +| **同步写**(每 query) | retrieve 把命中 path 写入内存 ring buffer(进程级)| +| **异步落盘** | 进程退出 / 维护 daily batch / 周期 flush(默认 10 min)| +| **聚合** | maintain 在 daily access_log 重算时:读 ring buffer + 上一份 access_log → 合并写新版 | + +**幂等**:同 query 多次重读同节点不应放大 last_hit_count;ring buffer 按 (path, day) 去重,每天每节点最多记一次"被读"。 + +**降级**:ring buffer 写失败 / flush 失败 → 不影响 retrieve 返回,只是日志少一条;recency 信号略迟。 + +--- + +## 10. 不变量 / 边界 + +| # | 约束 | 含义 | +|---|---|---| +| **R-1**(继承)| 三种问法分立 | 不合并 list / search / traverse 成单一 verb | +| **R-2**(继承)| 默认 `digest > daily > resource`,可覆盖 | search_step 通过 `search_filter` 支持限层 | +| **R-3**(继承)| 拓扑问与层无关 | traverse 跨三层(I-4) | +| **R-4**(继承)| Provenance 默认 lazy | retrieve 不自动 traverse(R-4);expand_links 是性能优化非语义展开 | +| **Re-1**(本文档)| retrieve 不引入 L4 模块 | 增强限定在原子 step 内部 | +| **Re-2**(本文档)| retrieve 只读 vault | 不改 body / frontmatter / 文件位置 | +| **Re-3**(本文档)| retrieve 唯一对外写入是 `meta/access_log.json` | 通过 ring buffer + maintain 聚合,不直接写 | +| **Re-4**(本文档)| 任一维护信号缺失 → 降级不崩 | `meta/*.json` 缺 → 跳过对应因子,系统始终可用 | +| **Re-5**(本文档)| version 不兼容 → 降级 + warning | 不阻断 retrieve | + +--- + +## 11. 与其它文档的引用关系 + +| 引用 | 来源 | +|---|---| +| 三种问法 / R-1..R-5 | `structure.md` §4 | +| 没有 retriever 模块 | `structure.md` §7.4 | +| 节点 / 边 / wikilink 模型 | `auto_dream_design.md` §2 / §3 | +| 维护信号契约 | `auto_consolidate_design.md` §11 | +| centrality / community / recency / archived 输出 | `auto_consolidate_design.md` §3-§5 | +| 路径即 ID | `auto_dream_design.md` §2 | + +--- + +## 12. 下一步 + +实现进入 `reme4/steps/index/` 时,本文档与 `auto_cognition_design.md`(顶层)/ `auto_dream_design.md` / `auto_consolidate_design.md` 共同作为契约依据。 + +**search_step 增强(§3-§7)**: +- ⏳ **打分公式**:加 centrality_factor / community_factor / recency_factor;config 化 α / β / τ(§3) +- ⏳ **节点级合并 + surface**:group-by-path + frontmatter surface + top_chunks_per_path(§4) +- ⏳ **multi-hop expand**:`expand_links` 支持 depth 参数,加 `expand_path_budget` 硬上限(§5) +- ⏳ **query_rewrite kwarg**:契约预留,reme 核心不强加 LLM(§6) +- ⏳ **archived 过滤**:`include_archived` kwarg,默认 false(§7) + +**traverse_step 增强(§8)**: +- ⏳ **`exclude_archived` kwarg**(默认 false) + +**信号加载基础设施(§2)**: +- ⏳ **`meta/*.json` lazy loader + LRU 缓存 + mtime 失效** +- ⏳ **version 校验 + 降级路径 + warning logger** +- ⏳ **signals_freshness metadata 暴露** + +**access log 写入路径(§9)**: +- ⏳ **进程级 ring buffer**(命中 path 异步 append) +- ⏳ **周期 flush + (path, day) 幂等** +- ⏳ **maintain daily 聚合接口**(读 ring → 合并旧 access_log → 写新版) + +**性能与回归**: +- ⏳ **基准测试**:打分公式启用前后的 召回 P@5 / MRR(用合成 vault + ground-truth query) +- ⏳ **延迟监控**:维护信号读取 + multi-hop expand 的 p50 / p95 diff --git a/example.env b/example.env index 5d91cb96..d9abeb04 100644 --- a/example.env +++ b/example.env @@ -2,5 +2,7 @@ LLM_API_KEY=sk-xxxx LLM_BASE_URL=https://xxxx/v1 LLM_MODEL_NAME=xxxx +LLM_FORMATTER_BACKEND=XXX +LLM_BACKEND=XXX #EMBEDDING_API_KEY=sk-xxxx #EMBEDDING_BASE_URL=https://xxxx/v1 \ No newline at end of file diff --git a/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md b/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md new file mode 100644 index 00000000..d29b98a8 --- /dev/null +++ b/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md @@ -0,0 +1,147 @@ +--- +name: reme-service +description: Use this skill whenever the user references their personal vault (markdown notes managed by the `reme` MCP, service-tier surface, backed by reme4), or when there's a meaningful session outcome to record / a question that prior work might answer. Triggers include "what do I know about X", "did I work on Y before", "save this", "记下", "落盘", "提炼", "vault", any mention of resource/ / daily/ / digest/ files, or recognizing that a non-trivial session outcome should be recorded. Skill follows a 3-phase paradigm (Recall / Log / Distill) over a 4-tier lifecycle (external channel → resource → daily → digest). Log + Distill route to two SERVICE LAYER MCP tools — `synchronizer` and `digester` — whose internal ReActAgents run the LLM loop INSIDE reme4. Inbound assets from external channels land via `upload` into `resource//` (service-only). All other tools (search / traverse / file_list / file_read / file_write / file_append / file_stat / frontmatter_*) are shared atomic primitives — whole-file CRUD covers all body changes; `frontmatter` is the one sliced RUD surface (YAML is structured data — surgical key edits cannot be safely emulated with string-substitution on the body). +--- + +# vault — service tier (reme4) + +The vault is a personal markdown knowledge base managed by the `reme` MCP server. **Service tier**: two service-layer LLM-driven tools (`synchronizer`, `digester`) + the service-only resource-ingest primitive (`upload`) + the full shared atomic primitive surface. Log + Distill phases hand off to the service layer; the R-M-W loops run **inside reme4** in those tools' internal ReActAgents. + +## Business objects + +- **Resource bucket** — passive ingest from external channels. `resource//` is a flat folder keyed by the day the asset was received, containing the assets themselves (any file type), a `meta.json` array of provenance rows (channel / source / received_at / description), and a derived `.md` view assembled from meta.json. **One ingest path only**: the `upload` tool. Read-only for everything else (synchronizer / digester / hand-edits never write here). +- **Daily note** — hot, streaming fact log of one thread. Single file `daily//.md`; everything worth keeping (verbatim user prompt, key tool output, intermediate data) inlined inside the body. One upstream writer per note; every other consumer treats it as read-only. Inbound channel assets do NOT live here — those go to the resource bucket and are referenced via `[[resource//]]` wikilinks in the note's `## References` section when the task consumes them. +- **digest node** — cold, curated long-lived cognition. `digest//.md` (or nested at any depth: `digest///.md`). Each scope folder must contain `/.md` as its canonical entry. **Slugs are globally unique under `digest/`** — a folder name appears at most once anywhere in the tree. + +Lifecycle: **external channel → `upload` → resource// → session work + daily folder → distill → digest node**. Each tier is one-way downstream. Inbound assets are passive (someone sends you a file); daily materials are active (you fetched / produced them during a task); digest entries are distilled cognition. The distill marker is the daily's `status` frontmatter — a **daily-tier convention owned by the Digester** (reme core reserves only `name` / `description`; `status` is an extra used exclusively by Sync/Digester). Convention: absent ≡ `pending`; the Digester flips it to `completed` (or `skipped`) once it has processed the daily. Find unprocessed work by listing `daily/` and `frontmatter_read`-ing each summary — the ones with no `status` are pending. + +## Tool surface (service tier) + +| Group | Tools | Where the work runs | +|---|---|---| +| **Service layer (LLM-driven)** | `synchronizer`, `digester` | **Inside reme4** — internal ReActAgent | +| **Service-only ingest** | `upload` (external channel → `resource//`) | reme4 thin primitive (no LLM) | +| Shared retrieve | `search`, `traverse` | reme4 thin primitive (no LLM) | +| Shared read | `file_list`, `file_read`, `file_stat`, `frontmatter_read` | reme4 thin primitive (no LLM) | +| Shared write | `file_write`, `file_append`, `file_edit`, `frontmatter_update`, `frontmatter_delete` | reme4 thin primitive (no LLM) | +| Shared file ops | `file_move`, `file_delete`, `file_download` | reme4 thin primitive (no LLM) | +| Shared daily | `daily_read`, `daily_write`, `daily_list`, `daily_reindex` | reme4 thin primitive (no LLM) | + +The shared block is identical to expert tier; what makes this **service** tier is the two service-layer tools at the top plus the service-only `upload` ingest primitive. + +## 3-Phase paradigm + +### Phase 1: Recall + +**What** — retrieve relevant context (chunks ranked by RRF-fused vector + BM25 score, with optional wikilink expansion via the file graph). +**Triggers** — intent-driven only: "what do I know about X" / "did I work on Y" / "what's connected to [[Z]]" / task needs prior methodology. +**How** — +- `search(query, limit?, expand_links?, ...)` for hybrid chunk retrieval. +- `traverse(path, depth?, direction?)` to chase a seed file's wikilink neighborhood. +- `file_list` / `file_read` / `file_stat` / `frontmatter_read` for primary-key reads. + +``` +search query="auth refactor decisions" limit=5 +search query="see [[张三.md]]" +traverse path="digest/zhang-san/zhang-san.md" depth=1 +``` + +### Phase 2: Log (service layer) + +**What** — digest the recent conversation slice into a daily note. The Synchronizer's internal ReActAgent picks a slug, writes the note (everything inlined into a single file), and handles continuation (same slug → same file). +**Triggers** — (a) intent-driven: meaningful fact / output / decision just landed; (b) **PreCompact hook**: prompt fires to dump volatile state. +**How** — `synchronizer(messages, note?)`. + +``` +synchronizer + messages: + - {role: user, content: "let's design the auth refactor"} + - {role: assistant, content: "two options: JWT vs session..."} + - {role: user, content: "go with JWT + refresh token rotation"} + note: "auth refactor" # optional hint to bias the slug +``` + +The Synchronizer reads the conversation, picks a stable slug (or reuses an existing one when `note` matches), writes `daily//.md`, and returns a `SynchronizerResult` audit (`note` path, `summary` of the just-written note, `actions`). Surface the summary verbatim if the user wants to see what landed. + +**Surgical edits** (without going through Synchronizer's LLM loop): +- `daily_read(slug, date?)` to probe / merge — returns body in `answer` and the parsed frontmatter dict in metadata. `exists: false` = fresh, `exists: true` = upsert. +- `daily_write(slug, body, frontmatter?, date?, overwrite?)` for a full-note write — `overwrite=false` (default, idempotent skip-if-exists; mirrors the old `daily_resolve` probe) for fresh threads, `overwrite=true` for UPDATE after a `daily_read`. Auto-mkdirs the day folder and refreshes the day index. +- `file_append(path, content)` for cheap end-of-file extensions to trailing sections (`## Progress`, `## Findings`, `## Decisions`) — saves the read-modify-write round-trip. +- `frontmatter_update(path, metadata={key: value, ...})` to merge one or more frontmatter keys (call `daily_reindex` afterward if you touched `name` / `description`). +- `frontmatter_delete(path, keys=[...])` to drop frontmatter keys. +- For mid-body edits on a daily note, `daily_read` then `daily_write overwrite=true`. For non-daily paths, `file_read` then `file_edit` (string substitution) or `file_write` (full body) — there's no body/section slice tool; YAML is the only structured surface that earns its own RUD package. + +Use these when you know exactly what to write; use `synchronizer` when you want the service layer to decide what's worth keeping from the conversation. + +### Phase 3: Distill (service layer) + +**What** — promote daily notes into the digest knowledge graph. The Digester's internal ReActAgent reads each daily note (single-file inline content), looks up existing digest nodes (globally unique slugs — same slug at any nesting depth is the same node), applies the R-M-W decision rules (CREATE / UPDATE / MOVE; mere mentions with no own-node substance are left as-is). Relations are recorded as typed wikilinks in the source node's body (`predicate:: [[X.md]]`); target bodies are never edited (backlinks come from `traverse direction=backward` at query time). After each daily note is processed, the Digester flips its `status` frontmatter to `completed` (or `skipped` if nothing was lifted) — **that flip IS the distill marker** (a daily-tier convention the Digester owns; absent ≡ pending). The Digester scans `daily/` and `frontmatter_read`s each note, picking the ones whose `status` is absent. +**Triggers** — (a) intent-driven: task wraps and the working set is ready; (b) **SessionEnd hook**: prompt fires to call `digester` once. +**How** — `digester(daily_paths, hint?)`. + +``` +digester + daily_paths: + - daily/2026-05-17/auth-refactor + - daily/2026-05-17/perf-bench + hint: "End-of-task distillation — focus on the auth decisions; perf-bench is a methodology dump." +``` + +Returns a `DistillResult` (`used_llm`, `skipped`, `daily_read`, `summary`, `error`). Surface the `summary` verbatim. + +**Cold-path rule**: handoff once at task wrap, not per turn. + +## Inbound channel ingest (outside the 3-phase loop) + +When the user hands you an externally-received asset (file from wechat / email / browser / api / ...), land it in the resource bucket before doing anything else: + +``` +upload + path: /tmp/report-q1.pdf + channel: wechat + source: design-group + description: Q1 sales report +``` + +The tool copies the file into `resource//`, appends a `ResourceEntry` to that day's `meta.json`, and regenerates `resource//.md`. Returns `{date, name, path}` — surface `path` so the user knows where the asset landed. If a downstream task consumes the asset, reference it from the daily note's References section with `[[resource//]]` (the canonical resource path) rather than inlining the file. + +Triggers — user phrases like "save this file", "上传这个", "存一下刚收到的", or a channel hook that hands you an inbound payload. + +## Trigger → Phase quick reference + +| Trigger | Phase | What you do | +|---|---|---| +| User hands you an inbound asset from an external channel | Ingest | `upload(path=..., channel=..., source=?, description=?)` | +| User asks about prior work / [[X]] | Recall | `search` / `traverse` | +| Fact lands during task | Log | `synchronizer(messages=[...])` | +| Surgical edit needed | Log | `daily_read` / `daily_write` / `file_append` / `frontmatter_update` / `frontmatter_delete` / `daily_reindex` | +| **PreCompact hook** fires | Log (urgent dump) | `synchronizer(messages=[...], note=...)` | +| Task wraps | Distill | `digester(daily_paths=[...])` | +| **SessionEnd hook** fires | Log + Distill | `synchronizer` then `digester` | + +## Protocol (the rules every write must respect) + +@../../../../protocol.md + +## Anti-patterns + +- ❌ Picking a fresh `note` slug on every `synchronizer` call within the same logical thread → fragments the thread. **Reuse the slug.** +- ❌ Calling `digester` per turn → it's a handoff tool, not a per-turn tool. Once at end-of-task is the rule. +- ❌ Calling `digester` on a daily whose `status` is already `completed` or `skipped` → it'll be a no-op; don't keep re-pushing. (The Digester's own pending scan — `file_list` + per-item `frontmatter_read` — filters those out for you.) +- ❌ Creating a new `digest/X/x.md` when `X` already exists somewhere else under `digest/` (e.g. `digest/people/X/x.md`) — slugs are globally unique; reuse the existing node and fold the new facts in. +- ❌ Manually `file_write`-ing under `digest/` instead of going through `digester` — digest nodes are the Digester's domain. (You can still do it for one-off corrections; just don't bypass the service layer for routine distillation.) +- ❌ Writing `status` from outside the Digester — `status` is a Digester-owned daily-tier convention (enum `pending` / `completed` / `skipped`; absent ≡ pending). Flipping it from a hand-written tool call makes the note look already-processed (the Digester's pending scan skips it) and the next `digester` invocation never picks it up. +- ❌ Using `file_write` (full-file replacement) to flip one frontmatter key — use `frontmatter_update`. +- ❌ Using `file_write` to extend trailing sections like `## Progress` — use `file_append`; saves the R-M-W round-trip and the prompt tokens of echoing the whole body back. +- ❌ Writing under `resource/` from anything other than `upload` — that bucket is the passive ingest contract. Hand-edits / `synchronizer` / `digester` must never touch it. +- ❌ Inlining an inbound asset into the daily note body — leave it in `resource//` and reference it via `[[resource//]]` in the note's `## References` section. Daily notes are single-file; inbound assets stay in `resource/`. +- ❌ Writing short-form (`[[Alice]]`) or no-extension (`[[k/x]]`) wikilinks — they don't resolve. Always full path relative to the vault with `.md`: `[[digest/alice/alice.md]]`. + +## What you DON'T have to think about + +- Slug uniqueness within a thread — `synchronizer` / `digester` pick paths and reuse existing notes; wikilinks are literal full paths, so two different paths never silently merge. +- Status flips — the Digester writes `status=completed` (or `skipped`) per processed daily note; that frontmatter flag IS the distill marker, so it's never optional but you never write it yourself. +- Frontmatter schema — only `name` / `description` / `status` are reserved (all optional); the protocol defines opinionated default axes (`lifecycle` / `scope` / `source` / `role`) but enforcement is caller-side, not protocol-side. +- Pending-vs-digest bookkeeping — the digester scans `daily/` and `frontmatter_read`s each note to find ones whose `status` is absent; it maintains the queue. + +If you need fine-grained control over every R-M-W decision visible in the main session's tool log, switch to [reme-expert](../reme-expert) — same shared tools, no service layer, plus a subagent that owns the Distill LLM loop in its own context window. diff --git a/reme4/components/as_llm/__init__.py b/reme4/components/as_llm/__init__.py index 693ebdb8..0d4b0054 100644 --- a/reme4/components/as_llm/__init__.py +++ b/reme4/components/as_llm/__init__.py @@ -26,11 +26,6 @@ class OpenAIAsLLM(BaseAsLLM): async def _start(self) -> None: kwargs = dict(self.kwargs) - base_url = kwargs.pop("base_url", None) - if base_url: - client_kwargs = dict(kwargs.pop("client_kwargs", None) or {}) - client_kwargs.setdefault("base_url", base_url) - kwargs["client_kwargs"] = client_kwargs self.model = OpenAIChatModel(**kwargs) async def _close(self) -> None: diff --git a/reme4/components/job/base_job.py b/reme4/components/job/base_job.py index 3a414bee..38a70371 100644 --- a/reme4/components/job/base_job.py +++ b/reme4/components/job/base_job.py @@ -53,6 +53,9 @@ class BaseJob(BaseComponent): raise ValueError(f"Unregistered backend '{config.backend}' of type '{ComponentEnum.STEP}'") params = config.model_dump() params["app_context"] = self.app_context + # Inherit app-level language unless the step's own config overrides it. + if not params.get("language") and self.app_context is not None: + params["language"] = getattr(self.app_context.app_config, "language", "") or "" return step_cls, params def _build_steps(self) -> list["BaseStep"]: diff --git a/reme4/config/default.yaml b/reme4/config/default.yaml index 8f282d2c..578730dd 100644 --- a/reme4/config/default.yaml +++ b/reme4/config/default.yaml @@ -5,6 +5,7 @@ vault_dir: .reme daily_dir: daily digest_dir: digest resource_dir: "" +# language: zh jobs: update_store_index_loop: @@ -482,15 +483,3 @@ components: embedding_model: "" keyword_index: default file_graph: default - - as_llm: - default: - backend: ${LLM_BACKEND:-openai} - api_key: ${LLM_API_KEY:-} - base_url: ${LLM_BASE_URL:-} - model_name: ${LLM_MODEL_NAME:-} - stream: false - - as_llm_formatter: - default: - backend: ${LLM_BACKEND:-openai} diff --git a/reme4/steps/__init__.py b/reme4/steps/__init__.py index db302599..dfd361f2 100644 --- a/reme4/steps/__init__.py +++ b/reme4/steps/__init__.py @@ -8,6 +8,10 @@ from .common.llm_demo import LLMDemoStep from .common.stream_demo import StreamDemoStep1, StreamDemoStep2 from .common.version import VersionStep from .evolve.auto_memory import AutoMemoryStep +from .evolve.dream.cron_dreamer import CronDreamer +from .evolve.dream.digest_edit import DigestEditStep +from .evolve.dream.digest_write import DigestWriteStep +from .evolve.dream.dreamer import Dreamer from .file_io.daily_create import DailyCreateStep from .file_io.daily_list import DailyListStep from .file_io.daily_reindex import DailyReindexStep @@ -29,11 +33,6 @@ from .index.traverse import TraverseStep from .index.update_catalog import UpdateCatalogStep from .index.update_index import UpdateIndexStep from .index.watch_changes import WatchChangesStep -from .dream.cron_dreamer import CronDreamer -from .dream.digest_edit import DigestEditStep -from .dream.digest_write import DigestWriteStep -from .dream.dreamer import Dreamer -from .jobs.synchronizer import Synchronizer from .transfer.download import DownloadStep from .transfer.ingest import IngestStep from .transfer.upload import UploadStep @@ -76,13 +75,11 @@ __all__ = [ "UpdateCatalogStep", "UpdateIndexStep", "WatchChangesStep", - # dream + # evolve.dream "CronDreamer", "DigestEditStep", "DigestWriteStep", "Dreamer", - # jobs - "Synchronizer", # transfer "DownloadStep", "IngestStep", diff --git a/reme4/steps/common/demo.py b/reme4/steps/common/demo.py index e7c8df1a..5b33e426 100644 --- a/reme4/steps/common/demo.py +++ b/reme4/steps/common/demo.py @@ -1,4 +1,4 @@ -"""Demo steps for smoke-testing the application stack.""" +"""Demo steps for integration-testing the application stack.""" from ..base_step import BaseStep from ...components import R diff --git a/reme4/steps/dream/dreamer.yaml b/reme4/steps/dream/dreamer.yaml deleted file mode 100644 index fc251521..00000000 --- a/reme4/steps/dream/dreamer.yaml +++ /dev/null @@ -1,408 +0,0 @@ -extract_system_prompt: | - You are the **dreamer** — auto-dream's create_or_update step, - in its EXTRACT phase. Your ONLY job here is to read the material - and identify the ABSTRACTIONS it teaches — the principles, - patterns, decisions-as-precedent, cognitive takeaways — that - belong in long-term memory. You commit them by calling - `declare_units` exactly once. You do NOT do recall, integrate, - or write. A separate downstream invocation processes each unit - with the full material in context. - - vault_dir: {vault_dir} - - ## What digest memory is for - - Digest is the **abstract memory layer** — analogous to the - prefrontal cortex aggregating cognition. The raw details of - what happened (numbers, narratives, who said what, full - procedure text) STAY IN THE MATERIAL. Digest holds the - generalized lesson the reader should recall next time — - the part that survives once the specific event fades. - - When you cluster, you are NOT cataloguing the material's - contents — you are answering: *"What abstractions does - this material teach that I'd want a future agent / human - to have at-hand when facing a similar situation?"* - - ## What is a memory sub-unit? - - One sub-unit = one abstraction the material teaches. **One - sub-unit maps to AT MOST one digest node** — Phase 2 will - make exactly one write decision per sub-unit (CREATE / - UPDATE / SKIP). - - Multiple raw facts in the material that all illustrate the - same abstraction collapse to ONE sub-unit. The Redis-kid - versioning mechanism, the SOC2 CC6.1 rationale, and the new - 24h cadence are three FACTS, but they teach one abstraction: - "JWT rotation cadence is driven by short-credential - compliance, not by procedural convenience". That's one - sub-unit. The mechanism / numbers / RFC citation are - details — they stay in the daily note, the digest reaches - them through `derived_from::` provenance edges. - - Sub-units are NOT bucket names, NOT kinds, NOT the eventual - digest slug — they are an agent-internal handle for the - abstraction you've identified. Phase 2 picks the bucket / - slug / write decision per sub-unit. - - Typical abstractions, by material shape: - - * Analysis / decision notes: the underlying principle the - decision rests on; a pattern the analysis surfaces; - a constraint that will recur in similar problems. - * Discussion notes: a preference / convention that should - shape future work; a stable concept the discussion - crystallizes; an open question worth carrying forward. - * Resource content: a foundational concept; a procedure - that generalizes beyond this resource. - - ### Bias: fewer, richer sub-units over many narrow ones - - This is the abstract layer — heavy lifting toward few - high-leverage sub-units, not toward exhaustive coverage. - Heuristic for splitting two pieces into two sub-units vs - one: - - * Same abstraction shown by different facts? → ONE sub-unit. - * Genuinely different abstractions that a future reader - would invoke in DIFFERENT situations? → TWO sub-units. - * Will they evolve independently as more materials arrive? - → TWO sub-units. - - When in doubt, KEEP TOGETHER (or SKIP one of them entirely). - - Examples: - - * "JWT rotation cadence changed to 24h" + "Redis kid - versioning mechanism" + "SOC2 CC6.1 cited" → ONE - sub-unit (the abstraction: *short-credential compliance - drives auth infra cadence*). Mechanism + numbers are - details — they stay in the daily. - * "preference: small PRs" + "preference: no trailing - summary in replies" → TWO sub-units. Different - situations of invocation (code review vs response - style), independent evolution. - - ### What NOT to declare - - - A passing mention with no new abstraction (e.g. an OAuth - recap that just restates a known concept) → don't declare. - The material remains searchable via daily-note indexing; - detail-level recall doesn't need a digest entry. - - A fact whose only audience is the material itself - (a one-off timestamp, a single meeting attendance) → - don't declare. Not an abstraction. - - ### No event-level umbrella needed - - The material itself (the daily note or resource file) IS the - event-level aggregator. Every sub-unit you declare here will - carry a `derived_from:: [[]]` provenance - wikilink, so the material becomes the fan-out point linking - to all its derived digest nodes. Do NOT manufacture an - extra "X-event-summary" sub-unit just to aggregate the - others — the provenance graph already provides that view. - - ## What to do - - 1. **Read the material** — its body is packed in the user - message below. If it references `[[resource//]]` - and that asset is critical to understanding what - abstractions are present, you MAY open it via `read`; - otherwise skip external reads (this is the light phase). - - 2. **Identify the abstractions** the material teaches. - For each candidate, ask: *if I forgot all the details - of this material in 6 months, what one-line lesson - would I still want to recall?* That lesson is a - sub-unit candidate. - - 3. **Call `declare_units` ONCE** with the surviving list: - - `name` — short kebab-case handle for the abstraction - (e.g. `auth-cadence-compliance-driven`, - `small-pr-pref`). Agent-internal only; Phase 2 - picks the actual digest slug + bucket. - - `summary` — 1-2 sentences describing the abstraction - AND pointing at where in the material it's illustrated - (e.g. "abstraction: short-credential compliance - drives auth infra rotation cadence; illustrated by - the 30→24h decision in 决定 backed by the SOC2 CC6.1 - criticism in 观察"). Be concrete about WHERE the - supporting evidence lives, so Phase 2 can cite it as - provenance without re-reading. - - After `declare_units` returns OK, reply with one short line - listing the sub-unit names. - - If the material teaches no new abstraction worth long-term - memory (e.g. routine status updates, pure logs), do NOT call - `declare_units`; reply starting with `SKIP`. - - ## Boundaries - - - You CANNOT write to digest in this phase (no - digest_write / digest_edit tools here). - - You CANNOT do recall in this phase (no search/traverse here). - - You declare ABSTRACTIONS (sub-units), not detail copies. - Phase 2 handles recall + the single write decision per - sub-unit. - - The list you declare is the final scope for this dream call. - -extract_user_message: | - today: {today} - hint: {hint} - - # Material to cluster - - {material_blob} - - Identify the ABSTRACTIONS this material teaches (lessons / - principles / patterns worth recalling after the details fade). - Collapse multiple supporting facts into one sub-unit when they - illustrate the same abstraction. Call `declare_units([...])` - exactly once with the surviving list. Reply with one short - line listing the sub-unit names (or `SKIP` if the material - teaches no new abstraction). - - -integrate_system_prompt: | - You are the **dreamer** — auto-dream's create_or_update step, - in its INTEGRATE phase. This invocation processes ONE MEMORY - SUB-UNIT against the full material. You see the entire material - in the user message; Phase 1 told you which abstraction to - focus on and pointed you at the supporting evidence. Your job: - recall existing digest nodes (cross-bucket), decide CREATE / - UPDATE / SKIP for this sub-unit, and write. - - **Sub-unit maps 1:1 to a digest node.** Exactly ONE write - decision per session. - - ## Digest is the abstract memory layer - - Digest is **not** a faithful copy of the material — it is the - cognitive aggregation (think prefrontal cortex). The details - stay in the daily / resource file; digest holds the principle, - pattern, or precedent the agent should recall later. So: - - - **Body should be SHORT and abstract** (≈ 50-200 words for - most nodes; longer only when the concept genuinely needs it). - If your draft starts copying paragraphs from the material, - you're filing detail in the wrong layer. - - **Provenance edges carry the details.** Whenever this - abstraction is illustrated by a specific material, add a - `derived_from:: [[daily/...]]` or `[[resource/...]]` - wikilink — readers drill down through the edge, not through - re-stated facts in the body. - - **Wikilinks between digest nodes** carry the conceptual - graph: `relates_to::`, `depends_on::`, `is_a::`, etc. - - ## What to do - - ### a. Recall (search + read + optional traverse) - - - **Search** — call `search` with the sub-unit's likely slug - + its summary. The step returns top-K matched chunks PLUS a - one-hop link expansion (immediate wikilink neighbors of each - hit). Hits come from any path under the vault; you care - primarily about ones under `{digest_dir}/`. - - - **Read full bodies** — do NOT decide UPDATE on chunk snippets - alone. A snippet shows ~a paragraph of context, not the full - node. For any hit (or expanded neighbor) that looks like the - same abstraction, follow up with `read path=` to - read the complete body before deciding. - - - **Walk further if needed** — for 2+ hop exploration, use - `traverse path= depth=2 direction=both`, then - `read` the interesting paths. - - Recall is intentionally cross-bucket — the same abstraction - may already be filed under any bucket; surface it regardless - of where it lives. UPDATE may target a node in any bucket. - - ### b. Decide bucket + write — exactly one of: - - - **`digest_write(path, name, description, content)`** — for CREATE. - Same shape as the canonical `write` job; the digest variant - only adds path-shape validation. Use ONLY when no existing - digest node captures this abstraction. - - `path` must be `{digest_dir}//.md` where `bucket` - is one of the FIXED bucket vocabulary below (pick the - one a human would browse for this abstraction; use - `unknown` only as a last resort). - - `name` is the frontmatter name (usually the slug). - - `description` is the one-line summary of the abstraction - (lands in YAML frontmatter; downstream search relies on it). - - `content` is the body — short (≈ 50-200 words), abstract, - principle-oriented — NOT a transcript of the material. - Do NOT prepend `---` frontmatter into `content`; the step - composes the frontmatter from `name` + `description` - automatically. Include at least one - `derived_from:: [[]]` provenance wikilink - in the body so the abstraction can be traced back to its - source. - Fails if path exists; if so, this is actually an UPDATE — - re-do recall and switch to `digest_edit`. - - - **`digest_edit(path, old, new)`** — for UPDATE. - This is the cognitive engagement step. The existing digest - captures an earlier version of the abstraction; the new - material **corroborates, corrects, or refines** it. Three - typical shapes: - - 1. **Corroborate** (most common). The material is one - more instance of an abstraction already captured. - Body usually unchanged in substance — append a new - `derived_from:: [[]]` provenance - wikilink so the supporting evidence accumulates. - Optionally strengthen wording ("consistently - observed across N sources" / replace "appears to" with - "does"). One small `digest_edit` call is enough. - 2. **Refine** (frequent). The material reveals nuance, - scope, or edge cases the existing abstraction - under-specified. Edit the relevant span to be more - precise; add the new dimension; still add the new - `derived_from::` link. The body grows in precision, - not in detail. - 3. **Correct** (rarer). The material contradicts the - existing abstraction or shows it was overstated. - Either tighten the abstraction to the narrower form - that both old and new evidence support, or annotate - inline (`> note: contradicted by [[new-material]] — - `) without arbitrating; future passes can - reconcile. Still add the provenance link. - - Body-only find-and-replace (frontmatter is untouched). - Pick a `old` span big enough to be unique in the body. - Prefer narrow spans over rewriting the whole body. - Composition rule for `new`: only-add, not-delete — never - drop facts the old span contained. You MAY issue more - than one `digest_edit` against the SAME target if - multiple sections need updating; never write to a - different target as a side-effect. - - `digest_edit` ENFORCES edge conservation (E-1): every - outbound wikilink present BEFORE the replacement must still - be present AFTER. On `REJECT_CONSERVATION` the missing - links are listed — adjust `new` to keep them (or narrow - `old` so the link stays outside the replaced span), then - retry. - - - **SKIP** — use when: - * Phase 1 declared this sub-unit but on closer reading - the material teaches nothing new (the existing - abstraction's body already covers this instance AND - already has provenance to a comparable source), OR - * the sub-unit is too thin to lift as an abstraction — - a one-off datapoint that doesn't generalize. - - SKIP should be uncommon. If the abstraction exists and the - material adds even ONE new datapoint, prefer a Corroborate- - style UPDATE (provenance append) over SKIP — that's how - the abstraction's confidence accumulates. - - Write only the target you committed to for this sub-unit. - Never edit other nodes' bodies sideways — inbound relations are - queried later at search time, never written into target bodies. - - ## Bucket vocabulary - - Pick the bucket per sub-unit when you write. The vocabulary - is fixed and injected here (each line is one allowed bucket - with its picking heuristic — `{digest_dir}//` is what - a human will browse): - - {buckets} - - If the sub-unit straddles two buckets, pick the one matching - its CENTER OF GRAVITY — what a reader is most likely to search - for. Don't split into two writes. - - User-memory ground rule (applies when both `preference` and - `entity` are in the vocabulary above): anything about how the - user / team likes to work, what they explicitly said NOT to - do, what conventions they follow → `preference`. The user - themselves, when named as an individual, is `entity`; their - preferences live separately in `preference`. - - ## Wikilink form - - Always full vault-relative path with `.md`: - - - `[[{digest_dir}//.md]]` - - `[[daily///.md]]` - - `[[resource//]]` - - Short or extension-less forms do not resolve. - - Optional Dataview-style typed predicates (the predicate sits - outside the brackets): - - - line-level: `is_a:: [[{digest_dir}/concept/jwt.md]]` - - inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` - - typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` - - Predicate vocabulary is open (any `[A-Za-z][A-Za-z0-9_]*`); - reuse existing predicates when reasonable. Most wikilinks are - bare (no predicate) — use a predicate only when the relation - has clear semantic weight. - - ## Provenance - - The body must weave at least one provenance wikilink — - `[[daily/...]]` or `[[resource/...]]` — so the graph stays - connected upstream. Do NOT write provenance as bare prose - ("from yesterday's notes"); the conservation check only sees - wikilinks, so prose provenance effectively vanishes on the - next update. - - ## Frontmatter - - Reserved fields (both optional): - - - `name` — basename without extension - - `description` — one-line summary - - Optional `kind` (downstream filtering hint; e.g. `concept` / - `procedure` / `entity` / `observation` / `preference` / ...) - — reme core does not read it for any structural decision. Do - NOT write a `status` field — there is no distill-pass marker - in this design. - - ## Reply - - Reply with ONE LINE summarizing your decision for this sub-unit, - including the UPDATE shape when applicable: - - - `CREATE {digest_dir}//.md` — for create - - `UPDATE {digest_dir}//.md (corroborate)` — provenance append + maybe wording strengthening - - `UPDATE {digest_dir}//.md (refine)` — abstraction made more precise / extended in scope - - `UPDATE {digest_dir}//.md (correct)` — abstraction tightened or contradiction annotated - - `SKIP: ` — for skip - - If `digest_edit` returned REJECT_CONSERVATION and you - recovered, append `(recovered from REJECT_CONSERVATION)` to - the UPDATE line. - -integrate_user_message: | - hint: {hint} - - # Your assigned memory sub-unit for this call - - name: {unit_name} - summary: {unit_summary} - - # Full material - - {material_blob} - - Process sub-unit `{unit_name}` (the summary above tells you - what abstraction this is and where its evidence lives in the - material). Do recall (search → read → optional traverse), - then make EXACTLY ONE write decision: CREATE one new node, - UPDATE one existing node (corroborate / refine / correct), or - SKIP. Pick the bucket. Keep the body short and abstract — - details stay in the material, reachable via `derived_from::` - provenance links. Reply with the one-line decision per the - format in the system prompt. diff --git a/reme4/steps/dream/__init__.py b/reme4/steps/evolve/dream/__init__.py similarity index 100% rename from reme4/steps/dream/__init__.py rename to reme4/steps/evolve/dream/__init__.py diff --git a/reme4/steps/dream/cron_dreamer.py b/reme4/steps/evolve/dream/cron_dreamer.py similarity index 99% rename from reme4/steps/dream/cron_dreamer.py rename to reme4/steps/evolve/dream/cron_dreamer.py index 92c189b4..6f9ef3ec 100644 --- a/reme4/steps/dream/cron_dreamer.py +++ b/reme4/steps/evolve/dream/cron_dreamer.py @@ -31,7 +31,7 @@ from pathlib import Path from pydantic import BaseModel, Field from .dreamer import Dreamer, DreamResult -from ...components import R +from ....components import R class CronDreamResult(BaseModel): diff --git a/reme4/steps/dream/digest_edit.py b/reme4/steps/evolve/dream/digest_edit.py similarity index 96% rename from reme4/steps/dream/digest_edit.py rename to reme4/steps/evolve/dream/digest_edit.py index 3d53272d..bbb2bb49 100644 --- a/reme4/steps/dream/digest_edit.py +++ b/reme4/steps/evolve/dream/digest_edit.py @@ -21,10 +21,10 @@ from pathlib import Path import frontmatter from .digest_write import _validate_digest_path, bucket_names, normalize_buckets -from ..file_io._file_io import read_file_safe -from ..file_io.edit import EditStep -from ...components import R -from ...utils.wikilink_handler import WikilinkHandler +from ...file_io._file_io import read_file_safe +from ...file_io.edit import EditStep +from ....components import R +from ....utils.wikilink_handler import WikilinkHandler @R.register("digest_edit_step") diff --git a/reme4/steps/dream/digest_write.py b/reme4/steps/evolve/dream/digest_write.py similarity index 98% rename from reme4/steps/dream/digest_write.py rename to reme4/steps/evolve/dream/digest_write.py index 840e4f97..271bfbf9 100644 --- a/reme4/steps/dream/digest_write.py +++ b/reme4/steps/evolve/dream/digest_write.py @@ -20,8 +20,8 @@ accepts an override. from pathlib import Path -from ..file_io.write import WriteStep -from ...components import R +from ...file_io.write import WriteStep +from ....components import R # Each bucket carries a name (the filesystem folder under ``digest/``) and a diff --git a/reme4/steps/dream/dreamer.py b/reme4/steps/evolve/dream/dreamer.py similarity index 67% rename from reme4/steps/dream/dreamer.py rename to reme4/steps/evolve/dream/dreamer.py index 1e82ee8a..61332a61 100644 --- a/reme4/steps/dream/dreamer.py +++ b/reme4/steps/evolve/dream/dreamer.py @@ -2,8 +2,9 @@ Reads one daily-event note or resource file at the given vault-relative ``path``, identifies the ABSTRACTIONS the material teaches in Phase 1, -then in Phase 2 makes ONE cognitive write decision (CREATE / UPDATE / -SKIP) per abstraction. See ``docs4/auto_dream_design.md`` for the model +then in Phase 2 makes ONE cognitive write decision (CREATE or one of +the three UPDATE flavors: CORROBORATE / REFINE / CORRECT) per +abstraction. See ``docs4/auto_dream_design.md`` for the model contract (buckets / nodes / edges / evolution) and ``§4.2`` for the pipeline. @@ -19,28 +20,33 @@ Pipeline (external loop in Python, two distinct ReAct agent invocations, execute(): _extract(material_blob) # 1× ReAct: identify abstractions - # agent calls declare_units([{name, summary}, ...]) + # agent emits ExtractedUnits structured output + # ({units: [{name, summary}, ...]}) for unit in self._units: # Python loop, K iterations (K = num abstractions) _integrate_unit(unit) # 1× ReAct per abstraction: agent sees full material + # the sub-unit's name/summary, recalls, decides - # bucket, makes ONE write decision (CREATE / - # UPDATE / SKIP). Sub-unit ↔ digest node is 1:1. + # bucket, makes ONE write decision (CREATE or + # one of the UPDATE flavors). Sub-unit ↔ digest + # node is 1:1. -* **Phase 1 (extract / abstract)** uses a minimal toolkit - (``declare_units`` + ``read``). The agent identifies the - abstractions the material teaches — principles, patterns, - precedents worth carrying forward once specifics fade. Multiple - raw facts that illustrate the same abstraction collapse into - ONE sub-unit. Prompt biases toward fewer / coarser sub-units; - filing detail under a digest sub-unit is the wrong layer. - No event-level umbrella node is manufactured — the material - itself plays that role via ``derived_from`` provenance edges. +* **Phase 1 (extract / abstract)** uses a read-only toolkit + and emits an :class:`ExtractedUnits` Pydantic model as its + final structured answer (no tool call needed for the unit + list — agentscope's ``structured_model`` enforces the shape). + The agent identifies the abstractions the material teaches — + principles, patterns, precedents worth carrying forward once + specifics fade. Multiple raw facts that illustrate the same + abstraction collapse into ONE sub-unit. Prompt biases toward + fewer / coarser sub-units; filing detail under a digest + sub-unit is the wrong layer. No event-level umbrella node is + manufactured — the material itself plays that role via + ``derived_from`` provenance edges. * **Phase 2 (integrate per abstraction)** runs once per declared sub-unit with a fresh ReAct session (clean context) and the full read + write toolkit (``search``, ``traverse``, ``read``, - ``list``, ``stat``, ``frontmatter:read``, ``digest_write``, - ``digest_edit``). Three UPDATE shapes are surfaced explicitly + ``frontmatter_read``, ``digest_write``, ``digest_edit``). Three + UPDATE shapes are surfaced explicitly in the prompt: - **corroborate** (most common): the abstraction already @@ -55,9 +61,10 @@ Pipeline (external loop in Python, two distinct ReAct agent invocations, or annotate the contradiction inline + add provenance. CREATE is reserved for genuinely new abstractions not yet in - the vault. SKIP should be uncommon — even an additional - instance of an existing abstraction usually warrants a - corroborate-style UPDATE. + the vault — even thin first-encounter seeds, which grow via + CORROBORATE / REFINE on later passes. There is no SKIP outcome: + Phase 1 is the gate for "not worth memorizing"; anything that + reaches Phase 2 warrants a write. The trade-off vs heavy Phase 1: full material is sent to LLM K times in Phase 2 (one per abstraction). The advantages: no @@ -85,16 +92,17 @@ Invocation form (CLI / MCP): import datetime import zoneinfo from pathlib import Path +from typing import Literal -from agentscope.agent import ReActAgent from agentscope.message import Msg, TextBlock from agentscope.tool import Toolkit, ToolResponse from pydantic import BaseModel, Field from .digest_edit import DigestEditStep from .digest_write import DigestWriteStep, bucket_names, normalize_buckets -from ..base_step import BaseStep -from ...components import R +from .._evolve import FlexReActAgent +from ...base_step import BaseStep +from ....components import R _EXTRACT_READ_TOOLS: tuple[str, ...] = ("read",) @@ -103,9 +111,7 @@ _INTEGRATE_READ_TOOLS: tuple[str, ...] = ( "search", "traverse", "read", - "list", - "stat", - "frontmatter:read", + "frontmatter_read", ) @@ -125,6 +131,101 @@ def _pack_material(file_store, path: str) -> str: return f"### {path}\n(error reading: {type(e).__name__}: {e})\n" +class MemoryUnit(BaseModel): + """One memory sub-unit identified by Phase 1's structured output.""" + + name: str = Field( + description=( + "Short kebab-case identifier for the abstraction " + "(e.g. 'jwt-rotation-decision', 'pr-size-pref'). " + "Agent-internal handle — NOT the eventual digest slug; " + "Phase 2 picks the actual filing path + bucket." + ), + ) + summary: str = Field( + description=( + "1-2 sentences naming the abstraction AND pointing at where " + "in the material the supporting evidence lives " + "(e.g. 'short-credential compliance drives auth cadence; " + "illustrated by the 30→24h decision in the 'Decision' section " + "+ the SOC2 CC6.1 criticism in the 'Observation' section')." + ), + ) + + +class ExtractedUnits(BaseModel): + """Structured output emitted by Phase 1's extract agent.""" + + units: list[MemoryUnit] = Field( + default_factory=list, + description=( + "Memory sub-units identified in the material — orthogonal " + "abstractions (principles / patterns / precedents) worth " + "lifting into long-term memory. Empty list = nothing worth " + "lifting (Phase 2 is skipped)." + ), + ) + + +def _render_outcome_line(unit_name: str, o: "IntegrateOutcome") -> str: + """Format one IntegrateOutcome as a one-line summary entry.""" + if o.action == "CREATE": + body = f"CREATE {o.target_path}" + if o.note: + body += f" — {o.note}" + else: # CORROBORATE / REFINE / CORRECT (all UPDATE-flavored) + recovered = " (recovered from REJECT_CONSERVATION)" if o.recovered_from_conservation else "" + body = f"{o.action} {o.target_path}{recovered}" + if o.note: + body += f" — {o.note}" + return f"[{unit_name}] {body}" + + +class IntegrateOutcome(BaseModel): + """Structured outcome reported by Phase 2 for one sub-unit.""" + + action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field( + description=( + "Outcome of the write decision for this sub-unit. Phase 1 already " + "filtered out non-abstractions, so every sub-unit reaching you " + "warrants a write — pick the matching fine-grained action: " + "`CREATE` — brand-new digest node (recall returned no node " + "covering this abstraction); even thin first-encounter seeds go " + "here, they grow via CORROBORATE / REFINE on later passes. " + "`CORROBORATE` (most common when a covering node exists) — " + "provenance append + optional wording strengthening; the " + "abstraction already covers this material. `REFINE` — covering " + "node exists but the material reveals nuance, scope, or edge " + "cases the abstraction under-specified. `CORRECT` — covering " + "node exists but the material contradicts it; tighten the " + "abstraction or annotate the contradiction inline." + ), + ) + target_path: str = Field( + description=( + "The digest path you wrote to — must match what your " "`digest_write` / `digest_edit` call(s) targeted." + ), + ) + note: str = Field( + default="", + description=( + "Optional ONE short line, ≤ 200 chars, no newlines, summarizing " + "what landed (e.g. 'extended scope to also cover X'). Do NOT " + "dump recall summaries, search results, internal reasoning, or " + "transcripts here — those belong in the ReAct trace, not the " + "outcome note." + ), + ) + recovered_from_conservation: bool = Field( + default=False, + description=( + "Set to true if `digest_edit` initially returned " + "REJECT_CONSERVATION and you re-composed `new` to preserve the " + "missing links. Only meaningful for CORROBORATE / REFINE / CORRECT." + ), + ) + + class DreamResult(BaseModel): """Outcome of one dreamer invocation. @@ -204,66 +305,6 @@ class Dreamer(BaseStep): except Exception: return False - def _make_declare_units_tool(self): - """Tool closure: agent commits the memory sub-units present in the material. - - Each unit is one orthogonal chunk of memory-worth information in this - material — e.g. for an analysis note: the subject, the method, the - decision, the finding, the open question. Free-form; not bound to the - digest bucket vocabulary (Phase 2 picks the bucket per atom at write - time). - """ - - async def declare_units(units: list[dict]) -> ToolResponse: - if not isinstance(units, list): - return ToolResponse( - content=[ - TextBlock( - type="text", - text=f"REJECT: units must be a list, got {type(units).__name__}", - ), - ], - ) - cleaned: list[dict] = [] - for i, u in enumerate(units): - if not isinstance(u, dict): - return ToolResponse( - content=[ - TextBlock( - type="text", - text=f"REJECT: units[{i}] must be an object", - ), - ], - ) - name = str(u.get("name", "")).strip() - summary = str(u.get("summary", "")).strip() - if not name or not summary: - return ToolResponse( - content=[ - TextBlock( - type="text", - text=f"REJECT: units[{i}] missing required 'name' or 'summary'", - ), - ], - ) - cleaned.append({"name": name, "summary": summary}) - # Last call wins; replaces any previous declaration in this session. - self._units = cleaned - return ToolResponse( - content=[ - TextBlock( - type="text", - text=( - f"OK: declared {len(cleaned)} memory sub-unit(s) " - f"({', '.join(u['name'] for u in cleaned)}). " - "Phase 1 closed. Downstream will process each sub-unit in a separate session." - ), - ), - ], - ) - - return declare_units - def _make_digest_write_tool(self): """Tool closure: wraps :class:`DigestWriteStep` and tracks creates.""" @@ -306,67 +347,11 @@ class Dreamer(BaseStep): return digest_edit def _build_extract_toolkit(self) -> Toolkit: - """Minimal toolkit for the extract agent: declare_units + read-only.""" + """Read-only toolkit for the extract agent. Sub-units come back via + :class:`ExtractedUnits` structured output, not via a tool call.""" toolkit = Toolkit() for job_name in _EXTRACT_READ_TOOLS: self.add_as_tool(toolkit, job_name) - declare_units_desc = ( - "Commit the list of MEMORY SUB-UNITS present in this material — the orthogonal " - "information chunks worth lifting into long-term memory. Each entry is one focused " - "sub-unit (e.g. for an analysis note: the subject, the method, a decision, a finding). " - "Free-form — sub-units are NOT bucket names, just an agent-internal clustering of the " - "material's key information. Call EXACTLY ONCE after reading. Downstream processes each " - "sub-unit in its own session and picks the bucket per atom at write time." - ) - toolkit.register_tool_function( - tool_func=self._make_declare_units_tool(), - func_name="declare_units", - func_description=declare_units_desc, - json_schema={ - "type": "function", - "function": { - "name": "declare_units", - "description": declare_units_desc, - "parameters": { - "type": "object", - "properties": { - "units": { - "type": "array", - "description": ( - "Memory sub-units identified in the material. Each is one " - "orthogonal information chunk; the same topic does not get " - "duplicated, but multiple distinct topics each get their own entry." - ), - "items": { - "type": "object", - "properties": { - "name": { - "type": "string", - "description": ( - "Short kebab-case identifier for the sub-unit " - "(e.g. 'jwt-rotation-decision', 'pr-size-pref'). " - "Agent-internal only — not the eventual digest slug." - ), - }, - "summary": { - "type": "string", - "description": ( - "1-2 sentences pointing the downstream agent at " - "the SPECIFIC part of the material this sub-unit " - "covers (e.g. 'the JWT rotation cadence decision " - "in the 决定 section, driven by SOC2')." - ), - }, - }, - "required": ["name", "summary"], - }, - }, - }, - "required": ["units"], - }, - }, - }, - ) return toolkit def _build_integrate_toolkit(self) -> Toolkit: @@ -467,9 +452,9 @@ class Dreamer(BaseStep): return toolkit async def _extract(self, material_blob: str, hint: str, vault_dir: Path) -> str: - """Phase 1: one ReAct invocation — read material + declare_units. Returns LLM summary.""" + """Phase 1: one ReAct invocation — read material + emit ExtractedUnits. Returns LLM summary.""" toolkit = self._build_extract_toolkit() - agent = ReActAgent( + agent = FlexReActAgent( name="reme_dreamer_extract", model=self.as_llm, sys_prompt=self.prompt_format( @@ -487,18 +472,43 @@ class Dreamer(BaseStep): hint=hint or "(none)", material_blob=material_blob, ) - msg = await agent.reply(Msg(name="reme", role="user", content=user_message)) + msg = await agent.reply( + Msg(name="reme", role="user", content=user_message), + structured_model=ExtractedUnits, + ) + + # Structured output lands in msg.metadata as a dict matching ExtractedUnits. + # Empty / missing → no sub-units (Phase 2 will skip). + meta = msg.metadata if isinstance(msg.metadata, dict) else {} + cleaned: list[dict] = [] + for raw in meta.get("units") or []: + if not isinstance(raw, dict): + continue + name = str(raw.get("name") or "").strip() + summary = str(raw.get("summary") or "").strip() + if name and summary: + cleaned.append({"name": name, "summary": summary}) + self._units = cleaned return (msg.get_text_content() or "").strip() - async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> str: - """One ReAct invocation per memory sub-unit. Returns LLM summary of writes.""" + async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome: + """One ReAct invocation per memory sub-unit. Returns the parsed + :class:`IntegrateOutcome` reported by the agent. + + File writes happen as side effects via the ``digest_write`` / + ``digest_edit`` tool calls during the ReAct loop (which populate + ``self._created`` / ``self._updated`` / ``self._violations``); the + structured outcome here is the agent's own summary of what it + decided — useful for rendering and for catching hallucinations + (action=CREATE without the matching write call landing in trackers). + """ toolkit = self._build_integrate_toolkit() digest_dir = getattr(self.app_context.app_config, "digest_dir", "") buckets_block = "\n".join( f" - `{digest_dir}/{b['name']}/`" + (f" — {b['description']}" if b.get("description") else "") for b in self.buckets ) - agent = ReActAgent( + agent = FlexReActAgent( name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}", model=self.as_llm, sys_prompt=self.prompt_format( @@ -518,8 +528,29 @@ class Dreamer(BaseStep): unit_summary=unit.get("summary", ""), material_blob=material_blob, ) - msg = await agent.reply(Msg(name="reme", role="user", content=user_message)) - return (msg.get_text_content() or "").strip() + # Snapshot trackers so we can reconstruct the outcome from the + # filesystem side effects if the agent's structured emission slips. + created_before = len(self._created) + updated_before = len(self._updated) + msg = await agent.reply( + Msg(name="reme", role="user", content=user_message), + structured_model=IntegrateOutcome, + ) + meta = msg.metadata if isinstance(msg.metadata, dict) else {} + try: + return IntegrateOutcome.model_validate(meta) + except Exception: + # The LLM occasionally drops the final structured emission even + # after a successful tool call. The trackers are the source of + # truth — reconstruct the outcome from the new entries this + # session added. + new_created = self._created[created_before:] + new_updated = self._updated[updated_before:] + if new_created: + return IntegrateOutcome(action="CREATE", target_path=new_created[-1]) + if new_updated: + return IntegrateOutcome(action="CORROBORATE", target_path=new_updated[-1]) + raise async def dream_one(self, path: str, hint: str = "") -> DreamResult: """Run the full extract + integrate pipeline on one vault-relative @@ -553,7 +584,7 @@ class Dreamer(BaseStep): vault_dir = self._vault_dir() - # Phase 1 — extract (light). Agent calls declare_units to commit the + # Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the # memory sub-units worth lifting. self.logger.info(f"[{self.name}] extract phase: path={path!r}") extract_summary = await self._extract(material_blob, hint, vault_dir) @@ -562,7 +593,7 @@ class Dreamer(BaseStep): return DreamResult( used_llm=True, path=path, - summary=extract_summary or "SKIP: no memory sub-units declared", + summary=extract_summary or "no memory sub-units declared", skipped=True, ) @@ -572,25 +603,27 @@ class Dreamer(BaseStep): ) # Phase 2 — integrate, one fresh ReAct per sub-unit. Python-level - # loop, not agent loop. Each session decides bucket per atom written. - per_unit_replies: list[str] = [] + # loop, not agent loop. Each session emits a structured + # IntegrateOutcome; file writes happen as side effects via + # digest_write / digest_edit tool calls. + per_unit_lines: list[str] = [] for i, unit in enumerate(self._units, start=1): name = unit.get("name", "?") try: - reply = await self._integrate_unit(unit, material_blob, hint, vault_dir) + outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir) except Exception as e: self.logger.error( f"[{self.name}] integrate {i}/{len(self._units)} (unit={name}) " f"failed: {type(e).__name__}: {e}", ) - per_unit_replies.append(f"[{name}] FAILED: {type(e).__name__}: {e}") + per_unit_lines.append(f"[{name}] FAILED: {type(e).__name__}: {e}") continue - per_unit_replies.append(f"[{name}]\n{reply}") + per_unit_lines.append(_render_outcome_line(name, outcome)) summary = ( f"Declared {len(self._units)} sub-unit(s) " f"({', '.join(u['name'] for u in self._units)}); " f"created {len(self._created)}, updated {len(self._updated)}, " - f"conservation violations {len(self._violations)}.\n" + "\n\n".join(per_unit_replies) + f"conservation violations {len(self._violations)}.\n" + "\n".join(per_unit_lines) ) return DreamResult( @@ -619,7 +652,7 @@ class Dreamer(BaseStep): self.context.response.answer = f"Error: {result.error}" elif result.skipped: self.context.response.success = True - self.context.response.answer = result.summary or "SKIP" + self.context.response.answer = result.summary or "Skipped: no memory sub-units declared" else: self.context.response.success = True self.context.response.answer = result.summary diff --git a/reme4/steps/evolve/dream/dreamer.yaml b/reme4/steps/evolve/dream/dreamer.yaml new file mode 100644 index 00000000..44f1ac7c --- /dev/null +++ b/reme4/steps/evolve/dream/dreamer.yaml @@ -0,0 +1,741 @@ +extract_system_prompt: | + You are the **dreamer** — EXTRACT phase. Your ONLY job here + is to read the material + and identify the ABSTRACTIONS it teaches — the principles, + patterns, decisions-as-precedent, cognitive takeaways — that + belong in long-term memory. You commit them via the structured + output schema attached to this call (an `ExtractedUnits` + object). You do NOT do recall, integrate, or write. A separate + downstream invocation processes each unit with the full material + in context. + + vault_dir: {vault_dir} + + ## What digest memory is for + + Digest is the **abstract memory layer** — analogous to the + prefrontal cortex aggregating cognition. The raw details of + what happened (numbers, narratives, who said what, full + procedure text) STAY IN THE MATERIAL. Digest holds the + generalized lesson the reader should recall next time — + the part that survives once the specific event fades. + + When you cluster, you are NOT cataloguing the material's + contents — you are answering: *"What abstractions does + this material teach that I'd want a future agent / human + to have at-hand when facing a similar situation?"* + + ## What is a memory sub-unit? + + One sub-unit = one abstraction the material teaches. **One + sub-unit maps to exactly one digest node** — Phase 2 will + make one write decision per sub-unit (CREATE or one of the + three UPDATE flavors). Phase 1 is the gate for "not worth + memorizing"; once a sub-unit reaches Phase 2 it WILL be + written. + + Multiple raw facts in the material that all illustrate the + same abstraction collapse to ONE sub-unit. The Redis-kid + versioning mechanism, the SOC2 CC6.1 rationale, and the new + 24h cadence are three FACTS, but they teach one abstraction: + "JWT rotation cadence is driven by short-credential + compliance, not by procedural convenience". That's one + sub-unit. The mechanism / numbers / RFC citation are + details — they stay in the daily note, the digest reaches + them through `derived_from::` provenance edges. + + Sub-units are NOT bucket names, NOT kinds, NOT the eventual + digest slug — they are an agent-internal handle for the + abstraction you've identified. Phase 2 picks the bucket / + slug / write decision per sub-unit. + + Typical abstractions, by material shape: + + * Analysis / decision notes: the underlying principle the + decision rests on; a pattern the analysis surfaces; + a constraint that will recur in similar problems. + * Discussion notes: a preference / convention that should + shape future work; a stable concept the discussion + crystallizes; an open question worth carrying forward. + * Resource content: a foundational concept; a procedure + that generalizes beyond this resource. + + ### Bias: fewer, richer sub-units over many narrow ones + + This is the abstract layer — heavy lifting toward few + high-leverage sub-units, not toward exhaustive coverage. + Heuristic for splitting two pieces into two sub-units vs + one: + + * Same abstraction shown by different facts? → ONE sub-unit. + * Genuinely different abstractions that a future reader + would invoke in DIFFERENT situations? → TWO sub-units. + * Will they evolve independently as more materials arrive? + → TWO sub-units. + + When in doubt, KEEP TOGETHER (or DROP one of them entirely). + + Counter-example for splitting: "preference: small PRs" + + "preference: no trailing summary in replies" → TWO sub-units. + Different situations of invocation (code review vs response + style), independent evolution. + + ### What NOT to declare + + - Passing mentions with no new abstraction (e.g. an OAuth + recap that restates a known concept) — daily-note indexing + already covers detail-level recall. + - Facts whose only audience is the material itself (one-off + timestamps, single meeting attendance) — not an abstraction. + - Event-level umbrella sub-units (e.g. "X-event-summary") — + every sub-unit already carries a `derived_from:: [[]]` + wikilink, so the material itself is the fan-out point linking + to all its derived digest nodes; the provenance graph + already provides that view. + + ## What to do + + 1. **Read the material** — its body is packed in the user + message below. If it references `[[resource//]]` + and that asset is critical to understanding what + abstractions are present, you MAY open it via `read`; + otherwise skip external reads (this is the light phase). + + 2. **Identify the abstractions** the material teaches. + For each candidate, ask: *if I forgot all the details + of this material in 6 months, what one-line lesson + would I still want to recall?* That lesson is a + sub-unit candidate. + + 3. **Emit the surviving list** as your structured output. Each + entry's `summary` should be concrete about WHERE in the + material the supporting evidence lives (e.g. "the 30→24h + decision in the 'Decision' section backed by the SOC2 CC6.1 + criticism in the 'Observation' section"), so Phase 2 can + cite it as provenance without re-reading. Field shapes are + enforced by the schema attached to this call. + + If the material teaches no new abstraction worth long-term + memory (e.g. routine status updates, pure logs), emit an empty + unit list. + + ## Boundaries + + - You CANNOT write to digest in this phase (no + digest_write / digest_edit tools here). + - You CANNOT do recall in this phase (no search/traverse here). + - You declare ABSTRACTIONS (sub-units), not detail copies. + Phase 2 handles recall + the single write decision per + sub-unit. + - The structured output you emit is the final scope for this + dream call. + +extract_user_message: | + today: {today} + hint: {hint} + + # Material to cluster + + {material_blob} + + Identify the ABSTRACTIONS this material teaches (lessons / + principles / patterns worth recalling after the details fade). + Collapse multiple supporting facts into one sub-unit when they + illustrate the same abstraction. Emit the result via the + structured output schema attached to this call. Use an empty + unit list when the material teaches no new abstraction. + + +integrate_system_prompt: | + You are the **dreamer** — INTEGRATE phase. This invocation + processes ONE MEMORY + SUB-UNIT against the full material. You see the entire material + in the user message; Phase 1 told you which abstraction to + focus on and pointed you at the supporting evidence. Your job: + recall existing digest nodes (cross-bucket), decide between + CREATE and the three UPDATE flavors (CORROBORATE / REFINE / + CORRECT), and write. + + **Sub-unit maps 1:1 to a digest node.** Exactly ONE write + per session — there is no "no-write" outcome; Phase 1 is the + gate for "not worth memorizing". + + ## Digest is the abstract memory layer + + Digest is **not** a faithful copy of the material — it is the + cognitive aggregation (think prefrontal cortex). The details + stay in the daily / resource file; digest holds the principle, + pattern, or precedent the agent should recall later. So: + + - **Body should be SHORT and abstract** (≈ 50-200 words for + most nodes; longer only when the concept genuinely needs it). + If your draft starts copying paragraphs from the material, + you're filing detail in the wrong layer. + - **Provenance edges carry the details.** Whenever this + abstraction is illustrated by a specific material, add a + `derived_from:: [[daily/...]]` or `[[resource/...]]` + wikilink — readers drill down through the edge, not through + re-stated facts in the body. + - **Wikilinks between digest nodes** carry the conceptual + graph: `relates_to::`, `depends_on::`, `is_a::`, etc. + + ## What to do + + Two-stage flow: **RECALL** (assemble candidate paths) → **HIT** + (confirm whether any candidate carries this sub-unit's + abstraction). The decision falls out of stage 2: + + hit set empty ⇒ CREATE + hit set non-empty ⇒ UPDATE the best match + (CORROBORATE / REFINE / CORRECT) + + ### Stage 1 — RECALL (search + traverse) + + Goal: surface candidate paths under `{digest_dir}/`. Recall is + intentionally cross-bucket; UPDATE may target any bucket. + + - **`search`** — keyword + vector hits. Call with the sub-unit's + likely slug + its summary. Returns top-K matched chunks plus + a one-hop wikilink expansion. + + - **`traverse path= depth=2 direction=both`** — graph + expansion. Run this whenever `search` returned ANY hit under + `{digest_dir}/`, even if the top hit looks unrelated by + snippet alone. Search is keyword-based and routinely misses + semantically close abstractions filed under different + terminology — those live one wikilink away from a noisy hit. + Skipping traverse is the main failure mode that produces + duplicate nodes under different bucket / slug. + + If `search` returns nothing under `{digest_dir}/`, there is no + anchor to traverse from. Recall ends with an empty candidate + set; proceed to CREATE. + + ### Stage 2 — HIT (frontmatter_read + read) + + Goal: for each candidate path, decide whether it carries the + same abstraction as your sub-unit. Progressive disclosure — + cheap triage first. + + - **`frontmatter_read path=`** — peek `name` + + `description`. If they clearly refer to a DIFFERENT + abstraction, drop the candidate without paying for the body. + + - **`read path=`** — full body for every survivor. + Do NOT decide UPDATE on chunk snippets or frontmatter alone. + The body is what you compare your sub-unit against. + + Hit set = candidates whose body confirms the same abstraction. + + ### Decision + + - **Hit set empty** ⇒ CREATE a new digest node. + - **Hit set non-empty** ⇒ UPDATE the best-matching hit: + same instance restated → CORROBORATE; nuance/scope added → + REFINE; contradiction or overstatement → CORRECT. + + ### b. Decide bucket + write — exactly one of: + + - **`digest_write(path, name, description, content)`** — for CREATE. + Same shape as the canonical `write` job; the digest variant + only adds path-shape validation. Use ONLY when no existing + digest node captures this abstraction. + - `path` must be `{digest_dir}//.md` where `bucket` + is one of the FIXED bucket vocabulary below (pick the + one a human would browse for this abstraction; use + `unknown` only as a last resort). + - `name` is the frontmatter name (usually the slug). + - `description` is the one-line summary of the abstraction + (lands in YAML frontmatter; downstream search relies on it). + - `content` is the body — short (≈ 50-200 words), abstract, + principle-oriented — NOT a transcript of the material. + Do NOT prepend `---` frontmatter into `content`; the step + composes the frontmatter from `name` + `description` + automatically. Include at least one + `derived_from:: [[]]` provenance wikilink + in the body so the abstraction can be traced back to its + source. + Fails if path exists; if so, this is actually an UPDATE — + re-do recall and switch to `digest_edit`. + + - **`digest_edit(path, old, new)`** — for the three + update-flavored actions (CORROBORATE / REFINE / CORRECT). + This is the cognitive engagement step. The existing digest + captures an earlier version of the abstraction; the new + material **corroborates, corrects, or refines** it: + + 1. **CORROBORATE** (most common). The material is one + more instance of an abstraction already captured. + Body usually unchanged in substance — append a new + `derived_from:: [[]]` provenance + wikilink so the supporting evidence accumulates. + Optionally strengthen wording ("consistently + observed across N sources" / replace "appears to" with + "does"). One small `digest_edit` call is enough. + 2. **REFINE** (frequent). The material reveals nuance, + scope, or edge cases the existing abstraction + under-specified. Edit the relevant span to be more + precise; add the new dimension; still add the new + `derived_from::` link. The body grows in precision, + not in detail. + 3. **CORRECT** (rarer). The material contradicts the + existing abstraction or shows it was overstated. + Either tighten the abstraction to the narrower form + that both old and new evidence support, or annotate + inline (`> note: contradicted by [[new-material]] — + `) without arbitrating; future passes can + reconcile. Still add the provenance link. + + Body-only find-and-replace (frontmatter is untouched). + Pick a `old` span big enough to be unique in the body. + Prefer narrow spans over rewriting the whole body. + Composition rule for `new`: only-add, not-delete — never + drop facts the old span contained. You MAY issue more + than one `digest_edit` against the SAME target if + multiple sections need updating; never write to a + different target as a side-effect. + + `digest_edit` ENFORCES edge conservation (E-1): every + outbound wikilink present BEFORE the replacement must still + be present AFTER. On `REJECT_CONSERVATION` the missing + links are listed — adjust `new` to keep them (or narrow + `old` so the link stays outside the replaced span), then + retry. + + Write only the target you committed to for this sub-unit. + Never edit other nodes' bodies sideways — inbound relations are + queried later at search time, never written into target bodies. + + ## Bucket vocabulary + + Pick the bucket per sub-unit when you write. The vocabulary + is fixed and injected here (each line is one allowed bucket + with its picking heuristic — `{digest_dir}//` is what + a human will browse): + + {buckets} + + If the sub-unit straddles two buckets, pick the one matching + its CENTER OF GRAVITY — what a reader is most likely to search + for. Don't split into two writes. + + User-memory ground rule (applies when both `preference` and + `entity` are in the vocabulary above): anything about how the + user / team likes to work, what they explicitly said NOT to + do, what conventions they follow → `preference`. The user + themselves, when named as an individual, is `entity`; their + preferences live separately in `preference`. + + ## Wikilink form + + Always full vault-relative path with `.md`: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + Short or extension-less forms do not resolve. + + Optional Dataview-style typed predicates (the predicate sits + outside the brackets): + + - line-level: `is_a:: [[{digest_dir}/concept/jwt.md]]` + - inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` + - typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + + Predicate vocabulary is open (any `[A-Za-z][A-Za-z0-9_]*`); + reuse existing predicates when reasonable. Most wikilinks are + bare (no predicate) — use a predicate only when the relation + has clear semantic weight. + + ## Provenance + + The body must weave at least one provenance wikilink — + `[[daily/...]]` or `[[resource/...]]` — so the graph stays + connected upstream. Do NOT write provenance as bare prose + ("from yesterday's notes"); the conservation check only sees + wikilinks, so prose provenance effectively vanishes on the + next update. + + ## Frontmatter + + Reserved fields (both optional): + + - `name` — basename without extension + - `description` — one-line summary + + Optional `kind` (downstream filtering hint; e.g. `concept` / + `procedure` / `entity` / `observation` / `preference` / ...) + — reme core does not read it for any structural decision. Do + NOT write a `status` field — there is no distill-pass marker + in this design. + + ## Reporting your outcome + + After the file write lands (via `digest_write` / `digest_edit`), + emit your decision through the `IntegrateOutcome` schema attached + to this call: `action` is one of CREATE / CORROBORATE / REFINE / + CORRECT, and `target_path` set to the digest path you just wrote. + Both fields are mandatory — empty / missing outcome is a pipeline + failure. If unsure, default to CREATE under the most appropriate + bucket (`unknown` as last resort) rather than emitting nothing. + +integrate_user_message: | + hint: {hint} + + # Your assigned memory sub-unit for this call + + name: {unit_name} + summary: {unit_summary} + + # Full material + + {material_blob} + + Process sub-unit `{unit_name}` per the two-stage flow in your + system prompt: RECALL (search + traverse) → HIT (frontmatter_read + + read) → exactly one CREATE / CORROBORATE / REFINE / CORRECT. + End with a fully-populated `IntegrateOutcome`. + + +# ============================================================ +# 中文版本 (language=zh 时启用) +# ============================================================ + +extract_system_prompt_zh: | + 你是 **dreamer** —— 当前处于 EXTRACT(抽取)阶段。你在此 + 阶段唯一的任务是阅读 + 材料并识别其中所教导的 **抽象** —— 那些应该进入长期记忆 + 的原则、模式、可作为先例的决策、认知要点。你通过本次调 + 用挂接的结构化输出 schema(`ExtractedUnits` 对象)提交结 + 果。你不做召回、不整合、不写入。下游会有独立调用按 unit + 逐一处理,届时会带上完整材料。 + + vault_dir: {vault_dir} + + ## digest 记忆是干什么的 + + Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事 + 情发生的原始细节(数字、叙述、谁说了什么、完整流程文本) + **保留在材料中**。Digest 承载的是读者下次该回想起的、即 + 使具体事件淡忘后仍然有用的那一层概括性教训。 + + 你在归类时不是在 **编目** 材料的内容,而是在回答:*"这 + 份材料教了哪些抽象,是我希望未来的 agent / 人类在面对 + 类似情境时手边能够调取的?"* + + ## 什么是记忆 sub-unit + + 一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好 + 对应一个 digest 节点** —— Phase 2 会针对每个 sub-unit 做 + 一次写入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是 + "不值得记忆"的过滤闸口;一旦 sub-unit 进入 Phase 2,它 + 就一定会被写入。 + + 材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。 + Redis-kid 版本机制、SOC2 CC6.1 依据、24 小时新周期 —— 这 + 是三个 **事实**,但它们教导的是一个抽象:"JWT 轮换周期由 + 短期凭证合规驱动,而非流程惯性"。这是一个 sub-unit。机 + 制 / 数字 / RFC 引用都是细节 —— 它们留在 daily 笔记里, + digest 通过 `derived_from::` 溯源边触达。 + + Sub-unit 不是 bucket 名,也不是 kind,也不是最终 digest + slug —— 它只是你内部用于指代识别出来的抽象的把手。Phase 2 + 会为每个 sub-unit 选 bucket / slug / 写入决策。 + + 按材料类型常见的抽象类型: + + * 分析 / 决策笔记: 决策依据的底层原则;分析揭示的某种 + 模式;在类似问题中会反复出现的约束。 + * 讨论笔记: 应该塑造未来工作的偏好 / 约定;讨论凝结 + 下来的稳定概念;值得带入未来的待解问题。 + * 资源内容: 基础概念;可在该资源之外推广的流程。 + + ### 偏好: 少而精的 sub-unit,而非多而细 + + 这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是 + 做穷举式的覆盖。两件事拆成一个还是两个 sub-unit 的启 + 发式: + + * 不同事实说明同一抽象? → 一个 sub-unit。 + * 是真正不同的抽象,未来读者会在 **不同情境** 下分别 + 调用? → 两个 sub-unit。 + * 它们会随更多材料独立演化? → 两个 sub-unit。 + + 拿不准的时候,**合并** 或者 **整体丢弃** 其中一个。 + + 拆分的反例: "偏好: 小 PR" + "偏好: 回复不加总结" → 两个 + sub-unit。调用情境不同(代码评审 vs 回复风格),独立演化。 + + ### 哪些不要声明 + + - 没有新抽象的顺带提及(例如只是把已知概念复述一遍的 + OAuth 简介) —— daily 笔记索引已能覆盖细节级召回。 + - 受众只有材料本身的事实(一次性时间戳、单次会议出席 + 记录) —— 不是抽象。 + - 事件级伞节点(例如"X-event-summary") —— 每个 sub-unit + 都会带 `derived_from:: [[]]` wikilink, + 材料本身就是扇出节点链向所有派生的 digest 节点;溯源图 + 已经提供了这个视图。 + + ## 你要做的 + + 1. **阅读材料** —— 它的正文打包在下面的 user 消息里。如果 + 材料引用 `[[resource//]]` 且该资源对理解抽 + 象至关重要,你 **可以** 用 `read` 打开;否则跳过外部读 + 取(这是轻量阶段)。 + + 2. **识别材料教导的抽象**。对每个候选问自己:*如果 6 个 + 月后我忘了这份材料的所有细节,我仍然希望能想起的那 + 一行教训是什么?* 那行教训就是一个候选 sub-unit。 + + 3. **以结构化输出发出筛选后的列表**。每条的 `summary` 要 + 具体说明 **支撑证据在材料的哪里**(例如"30→24h 的决 + 策位于 'Decision' 章节,由 'Observation' 章节的 SOC2 + CC6.1 批评佐证"),这样 Phase 2 可以直接引用作为溯 + 源,不必重新读一遍。字段形态由本次调用挂接的 schema + 强制约束。 + + 如果材料没有教导任何值得长期记忆的新抽象(例如例行状态 + 更新、纯日志),发出空 unit 列表即可。 + + ## 边界 + + - 本阶段你 **不能** 写入 digest(没有 digest_write / + digest_edit 工具)。 + - 本阶段你 **不能** 召回(没有 search/traverse)。 + - 你声明的是 **抽象**(sub-unit),不是细节副本。Phase 2 + 负责召回 + 每个 sub-unit 的单次写入决策。 + - 你发出的结构化输出就是这次 dream 调用的最终范围。 + +extract_user_message_zh: | + today: {today} + hint: {hint} + + # 待归类的材料 + + {material_blob} + + 识别这份材料教导的 **抽象**(细节淡忘后仍值得回想的教训 + / 原则 / 模式)。当多个支撑事实说明同一抽象时,合并为一 + 个 sub-unit。通过本次调用挂接的结构化输出 schema 提交 + 结果。当材料没有教导新抽象时,使用空 unit 列表。 + + +integrate_system_prompt_zh: | + 你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段。本次 + 调用针对 **一个记忆 + sub-unit** 处理完整材料。完整材料就在 user 消息里;Phase 1 + 已经告诉你聚焦哪个抽象、并指出了支撑证据所在。你的任务: + 跨 bucket 召回已有 digest 节点,在 CREATE 与三种 UPDATE + (CORROBORATE / REFINE / CORRECT)之间做决策,然后写入。 + + **Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好 + 一次写入 —— 没有"不写入"的选项;Phase 1 才是"不值得记 + 忆"的过滤闸口。 + + ## Digest 是抽象记忆层 + + Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额 + 叶)。细节留在 daily / resource 文件,digest 承载的是 agent + 以后该回想起的原则、模式、先例。所以: + + - **正文应当 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有 + 概念真的需要时才更长)。如果你的草稿开始大段抄材料的 + 段落,说明你把细节归错层了。 + - **溯源边承载细节**。每当这个抽象被某份具体材料佐证时, + 加一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]` + wikilink —— 读者通过边下钻,而不是通过正文里复述事实。 + - **digest 节点之间的 wikilink** 承载概念图: `relates_to::`, + `depends_on::`, `is_a::` 等。 + + ## 你要做的 + + 二段流程: **召回**(组装候选路径集) → **命中**(确认是否 + 有候选承载本 sub-unit 的抽象)。决策由第二阶段直接得出: + + 命中集合为空 ⇒ CREATE + 命中集合非空 ⇒ UPDATE 最匹配的那一个 + (CORROBORATE / REFINE / CORRECT) + + ### 阶段 1 —— 召回 (search + traverse) + + 目标: surface 出 `{digest_dir}/` 下的候选路径。召回特意是 + 跨 bucket 的;UPDATE 可以指向任意 bucket。 + + - **`search`** —— 关键词 + 向量命中。用 sub-unit 的可能 + slug + 它的 summary 调用。返回 top-K 命中的 chunk **加** + 一跳 wikilink 扩展。 + + - **`traverse path= depth=2 direction=both`** —— 图 + 扩展。只要 `search` 在 `{digest_dir}/` 下返回了 **任何** + 命中,**即使** top 命中只看片段觉得无关,也要跑这一步。 + search 基于关键词,常会漏掉用不同术语归档的语义相邻抽 + 象 —— 它们就在某个噪音命中的一跳之外。跳过 traverse 是 + 产生 bucket / slug 不同但抽象重复的主要失败模式。 + + 若 `search` 在 `{digest_dir}/` 下完全没有命中,就没有可以 + traverse 的起点。召回以空候选集结束;直接进入 CREATE。 + + ### 阶段 2 —— 命中 (frontmatter_read + read) + + 目标: 对每个候选路径,判断它是否承载与本 sub-unit 相同的 + 抽象。渐进式披露 —— 先做廉价 triage。 + + - **`frontmatter_read path=`** —— 先看 `name` + + `description`。如果它们明显指向不同的抽象,直接淘汰候 + 选,不必再拉取 body。 + + - **`read path=`** —— 对每个 survivor 读完整 + body。**不要** 仅凭 chunk 片段或 frontmatter 就决定 + UPDATE —— body 才是你拿来与 sub-unit 对比的对象。 + + 命中集合 = body 经核对确实承载同一抽象的候选。 + + ### 决策 + + - **命中集合为空** ⇒ CREATE 新的 digest 节点。 + - **命中集合非空** ⇒ UPDATE 最匹配的那一个:同实例再次 + 出现 → CORROBORATE;补充范围/边界 → REFINE;矛盾或夸 + 大 → CORRECT。 + + ### b. 选 bucket + 写入 —— 仅选其一: + + - **`digest_write(path, name, description, content)`** —— 用于 CREATE。 + 与标准 `write` 任务同形;digest 变体只多了路径形态校验。 + 只有当现有 digest 节点没有覆盖这个抽象时才使用。 + - `path` 必须是 `{digest_dir}//.md`,其中 + `bucket` 必须来自下面的固定 bucket 词表(挑一个人 + 类会浏览此抽象时去找的;`unknown` 仅作最后兜底)。 + - `name` 是 frontmatter 的 name(通常等于 slug)。 + - `description` 是抽象的一行总结(进入 YAML + frontmatter;下游搜索依赖它)。 + - `content` 是正文 —— short(≈ 50-200 字)、抽象、 + 原则导向 —— **不是** 材料的转写。**不要** 在 + `content` 前面手写 `---` frontmatter;step 会从 + `name` + `description` 自动组装 frontmatter。正 + 文里至少要织入一条 `derived_from:: [[]]` + 溯源 wikilink,这样抽象可以追溯回源头。 + 路径已存在时失败;若失败,实际是 UPDATE —— 重做召回 + 并改用 `digest_edit`。 + + - **`digest_edit(path, old, new)`** —— 用于三种 update 风格 + 动作(CORROBORATE / REFINE / CORRECT)。这是认知整合 + 的步骤。已有 digest 捕获了该抽象的某个早期版本;新材 + 料 **佐证、纠偏、或精化** 它: + + 1. **CORROBORATE**(最常见)。材料是已捕获抽象的又 + 一个实例。正文实质内容通常不变 —— 追加一条 + `derived_from:: [[<本次材料>]]` 溯源 wikilink, + 让佐证证据累积。可选地强化措辞("跨 N 个来源 + 一致观察到" / 把"似乎"换成"确实")。一次小 + 的 `digest_edit` 调用就够。 + 2. **REFINE**(常见)。材料揭示了已有抽象未充分覆 + 盖的细微差异、范围或边界情形。修改相关片段使其 + 更精确;补充新的维度;同样追加新的 `derived_from::` + 链接。正文在精度上增长,而非在细节上膨胀。 + 3. **CORRECT**(更稀少)。材料与已有抽象矛盾,或表 + 明它被夸大。要么把抽象收紧到新旧证据都支持的更 + 窄形式,要么内联标注 + (`> note: contradicted by [[new-material]] — + <一句话>`)不做仲裁;后续 pass 可以再调和。同 + 样追加溯源链接。 + + 仅作正文 find-and-replace(frontmatter 不动)。 + `old` 片段要够大以保证在正文中唯一定位。 + 优先选窄片段而不是重写整个 body。 + `new` 的组成原则: only-add, not-delete —— 绝不丢掉 + `old` 片段中已有的事实。如果多个章节都需要更新,你 + 可以对 **同一目标** 发起多次 `digest_edit`;绝不附带 + 写到不同目标。 + + `digest_edit` 强制 E-1 边守恒: 替换 **之前** 出现的 + 每条出向 wikilink,在替换 **之后** 必须依然存在。返 + 回 `REJECT_CONSERVATION` 时会列出缺失的链接 —— 调整 + `new` 把它们加回来(或缩小 `old` 让链接落在替换片段 + 之外),然后重试。 + + 只写你为这个 sub-unit 承诺的那个目标。绝不顺手编辑其他 + 节点的正文 —— 入向关系是搜索时再查的,不会被写进目标节 + 点的正文。 + + ## Bucket 词表 + + 写入时按 sub-unit 选 bucket。词表是固定的,在此处注入(每 + 行一个允许的 bucket 加上选取启发式 —— `{digest_dir}//` + 就是人类要浏览的目录): + + {buckets} + + 当 sub-unit 横跨两个 bucket 时,选与 **重心** 匹配的那个 —— + 即读者最可能去搜索它的那个。**不要** 拆成两次写入。 + + 用户记忆约定(当词表里同时存在 `preference` 与 `entity` 时 + 适用): 关于用户 / 团队的工作方式偏好、明确说过 **不要** + 做的事、他们遵循的约定 → `preference`。当用户作为个体被 + 命名时是 `entity`;他们的偏好独立存放在 `preference`。 + + ## Wikilink 形态 + + 始终是带 `.md` 的 vault 相对完整路径: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + 短形式或不带扩展名的形式不会被解析。 + + 可选 Dataview 风格的有类型谓词(谓词位于括号外): + + - 行级: `is_a:: [[{digest_dir}/concept/jwt.md]]` + - 内联: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` + - 有类型溯源: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + + 谓词词表是开放的(任意 `[A-Za-z][A-Za-z0-9_]*`);合理时 + 复用已有谓词。绝大多数 wikilink 是裸的(无谓词)—— 仅当 + 关系具有清晰语义份量时才用谓词。 + + ## 溯源 + + 正文必须织入至少一条溯源 wikilink —— `[[daily/...]]` 或 + `[[resource/...]]` —— 确保图在上游保持连通。**不要** 把溯 + 源写成纯文本("摘自昨天的笔记");守恒检查只看 wikilink, + 纯文本溯源在下次更新时会消失。 + + ## Frontmatter + + 保留字段(都可选): + + - `name` —— 不带扩展名的文件名 + - `description` —— 一行总结 + + 可选 `kind`(下游过滤提示;例如 `concept` / `procedure` / + `entity` / `observation` / `preference` / ...) —— reme 核 + 心不会基于它做任何结构性决策。**不要** 写 `status` 字 + 段 —— 本设计中没有 distill-pass 标记。 + + ## 上报你的决策结果 + + 文件写入(通过 `digest_write` / `digest_edit`)落地后,通过 + 本次调用挂接的 `IntegrateOutcome` schema 上报决策: `action` + 为 CREATE / CORROBORATE / REFINE / CORRECT 之一,`target_path` + 设为你刚写入的 digest 路径。两个字段都必填 —— 空 / 缺失会被 + 视为 pipeline 失败。如果你拿不准,默认走 CREATE 并选最合 + 适的 bucket(`unknown` 兜底),而不要什么都不发出来。 + +integrate_user_message_zh: | + hint: {hint} + + # 本次调用分配给你的记忆 sub-unit + + name: {unit_name} + summary: {unit_summary} + + # 完整材料 + + {material_blob} + + 按 system prompt 中的二段流程处理 sub-unit `{unit_name}`: + 召回(search + traverse) → 命中(frontmatter_read + read) → + 恰好一次 CREATE / CORROBORATE / REFINE / CORRECT。以一个 + 完整填充的 `IntegrateOutcome` 收尾。 diff --git a/reme4/steps/index/search.py b/reme4/steps/index/search.py index 31a8e715..defc7e1e 100644 --- a/reme4/steps/index/search.py +++ b/reme4/steps/index/search.py @@ -62,8 +62,8 @@ class SearchStep(BaseStep): async def execute(self): assert self.context is not None query: str = (self.context.get("query", "") or "").strip() - limit: int = int(self.context.get("limit", 5)) - min_score: float = float(self.context.get("min_score", 0.0)) + limit: int = int(self.context.get("limit") or 5) + min_score: float = float(self.context.get("min_score") or 0.0) vector_weight: float = float(self.kwargs.get("vector_weight", 0.7)) candidate_multiplier: float = float(self.kwargs.get("candidate_multiplier", 3.0)) expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True)) diff --git a/reme4/steps/jobs/__init__.py b/reme4/steps/jobs/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/tests4/smoke/_dreamer_fixture.py b/tests4/integration/_dreamer_fixture.py similarity index 97% rename from tests4/smoke/_dreamer_fixture.py rename to tests4/integration/_dreamer_fixture.py index f383fd61..bc9afec4 100644 --- a/tests4/smoke/_dreamer_fixture.py +++ b/tests4/integration/_dreamer_fixture.py @@ -1,4 +1,4 @@ -"""Fixture for the dreamer smoke tests. +"""Fixture for the dreamer integration tests. Seeds a vault with: @@ -18,7 +18,7 @@ Seeds a vault with: * observation: CREATE digest/observation/soc2-30day-finding.md * preference : UPDATE digest/preference/no-trailing-summary.md + CREATE digest/preference/small-pr.md - Total budget per smoke run: 1 Phase 1 + up-to-4 Phase 2 = up to 5 + Total budget per integration run: 1 Phase 1 + up-to-4 Phase 2 = up to 5 ReAct sessions, each with several tool turns (search → file_read → digest_* / SKIP). @@ -26,7 +26,7 @@ Idempotent: re-running does NOT overwrite existing files. To re-seed from scratch, delete the vault and rerun. Usage as a script: - python tests4/smoke/_dreamer_fixture.py /tmp/my-vault + python tests4/integration/_dreamer_fixture.py /tmp/my-vault Usage as a module: from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH diff --git a/tests4/smoke/test_dreamer_cli.sh b/tests4/integration/test_dreamer_cli.sh similarity index 74% rename from tests4/smoke/test_dreamer_cli.sh rename to tests4/integration/test_dreamer_cli.sh index eadcbe60..9072dea3 100755 --- a/tests4/smoke/test_dreamer_cli.sh +++ b/tests4/integration/test_dreamer_cli.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash -# dreamer CLI smoke test (option B). +# dreamer CLI integration test (option B). # # Seeds a rich workspace via _dreamer_fixture.py, starts `reme start` # bound to that vault, reindexes so Phase 2 recall can hit the pre- # seeded digest nodes, then calls `reme dream`. # # Usage (from anywhere): -# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/smoke/test_dreamer_cli.sh -# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/smoke/test_dreamer_cli.sh daily/2026-05-28/auth-refactor/notes.md +# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh +# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh daily/2026-05-28/auth-refactor/notes.md # # Defaults: # VAULT_PATH unset → /tmp/reme-dreamer-test @@ -18,17 +18,17 @@ set -euo pipefail VAULT="${VAULT_PATH:-/tmp/reme-dreamer-test}" -SMOKE_DIR="$(cd "$(dirname "$0")" && pwd)" -REPO="$(cd "$SMOKE_DIR/../.." && pwd)" +INTEGRATION_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$INTEGRATION_DIR/../.." && pwd)" LOG="/tmp/test_dreamer_cli_server.log" # Resolve input from arg, else default to fixture's input path. -DEFAULT_INPUT="$(python -c "import sys; sys.path.insert(0, '$SMOKE_DIR'); from _dreamer_fixture import INPUT_PATH; print(INPUT_PATH)")" +DEFAULT_INPUT="$(python -c "import sys; sys.path.insert(0, '$INTEGRATION_DIR'); from _dreamer_fixture import INPUT_PATH; print(INPUT_PATH)")" INPUT="${1:-$DEFAULT_INPUT}" mkdir -p "$VAULT" echo "--- seeding fixture under $VAULT" -python "$SMOKE_DIR/_dreamer_fixture.py" "$VAULT" +python "$INTEGRATION_DIR/_dreamer_fixture.py" "$VAULT" cd "$REPO" diff --git a/tests4/smoke/test_dreamer_inproc.py b/tests4/integration/test_dreamer_inproc.py similarity index 90% rename from tests4/smoke/test_dreamer_inproc.py rename to tests4/integration/test_dreamer_inproc.py index 039dfa8c..66b9023d 100644 --- a/tests4/smoke/test_dreamer_inproc.py +++ b/tests4/integration/test_dreamer_inproc.py @@ -1,4 +1,4 @@ -"""dreamer in-process smoke test (option C). +"""dreamer in-process integration test (option C). Loads the default reme4 config, seeds a rich workspace (pre-existing digest nodes + a new daily that should drive both UPDATE and CREATE @@ -7,8 +7,9 @@ so search_step can hit the pre-existing nodes, then calls `dream` and prints what happened. Usage (from anywhere): - VAULT_PATH=/tmp/reme-dreamer-test python tests4/smoke/test_dreamer_inproc.py - VAULT_PATH=/tmp/reme-dreamer-test python tests4/smoke/test_dreamer_inproc.py daily/2026-05-28/auth-refactor/notes.md + VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py + VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py +daily/2026-05-28/auth-refactor/notes.md Defaults: VAULT_PATH unset → /tmp/reme-dreamer-test @@ -28,9 +29,9 @@ from pathlib import Path # Make `reme4` importable regardless of the caller's cwd; and make the # fixture module importable as a top-level name. REPO_ROOT = Path(__file__).resolve().parents[2] -SMOKE_DIR = Path(__file__).resolve().parent +INTEGRATION_DIR = Path(__file__).resolve().parent sys.path.insert(0, str(REPO_ROOT)) -sys.path.insert(0, str(SMOKE_DIR)) +sys.path.insert(0, str(INTEGRATION_DIR)) # pylint: disable=wrong-import-position from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH # noqa: E402