mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
feat(dream): replace digester with abstraction-layer dreamer pipeline (#264)
* feat(dream): replace digester with abstraction-layer dreamer pipeline
Reframe digest as the abstract memory layer (details stay in the daily/
resource material; digest holds principles, patterns, precedents reachable
via derived_from provenance edges). Replaces the old digester with a
2-phase ReAct workflow + a daily-tick wrapper:
- Phase 1 (Dreamer extract): clusters material into orthogonal memory
sub-units; each sub-unit maps 1:1 to a digest node (no inner atom
enumeration). Biases toward fewer / richer sub-units.
- Phase 2 (Dreamer integrate per sub-unit): cross-bucket recall +
exactly one write decision (CREATE / UPDATE / SKIP); UPDATE shapes
surfaced explicitly (corroborate / refine / correct).
- CronDreamer: scans <daily_dir>/<today>.md + <daily_dir>/<today>/**
+ <resource_dir>/<today>/** and runs dream_one per file.
Write tools are proper subclasses of the canonical file_io WriteStep /
EditStep with only path-shape + bucket + E-1 edge-conservation rules
layered on top:
- DigestWriteStep(WriteStep): path = <digest_dir>/<bucket>/<slug>.md,
must-not-exist, schema mirrors `write` (path / name / description /
content) so frontmatter lands automatically.
- DigestEditStep(EditStep): body-only find-and-replace + must-exist +
E-1 conservation preflight (refuses if any outbound wikilink would
be dropped).
Configuration:
- Bucket vocabulary structured in code (tuple[{name, description}]);
prompt renders the heuristic block at runtime via {buckets}.
- digest_dir / daily_dir / resource_dir come from app config (not tool
params); prompts use {digest_dir} placeholder.
- BaseStep walks class MRO when loading prompts, so subclasses inherit
parent yaml without duplication.
Tooling: agentscope register_tool_function schemas now wrap in the
proper {"type":"function","function":{...}} envelope. OpenAIAsLLM
routes base_url through client_kwargs so non-default endpoints work.
Smoke: tests4/smoke/{_dreamer_fixture.py,test_dreamer_inproc.py,
test_dreamer_cli.sh} drive the end-to-end pipeline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(dreamer): split long description string across multiple lines
* refactor(dream): remove hardcoded DEFAULT_DIGEST_DIR and use app_config
* docs(auto-cognition): add comprehensive design document for auto-cognition system
* refactor(steps): remove deprecated digest edit/write steps
* refactor(config): remove redundant LLM formatter backend configuration
* refactor(dreamer): improve code formatting and line breaks
* feat(auto-dream): implement three-bucket classification system for knowledge organization
* feat: rename dream_today step to auto-dream and refactor extraction logic
This commit is contained in:
Sen Huang
GitHub
parent
aa87f4fdea
commit
16d2d84431
24 changed files with 3567 additions and 1487 deletions
330
docs4/auto_cognition_design.md
Normal file
330
docs4/auto_cognition_design.md
Normal file
|
|
@ -0,0 +1,330 @@
|
|||
# auto-cognition 设计(顶层:心智循环)
|
||||
|
||||
> 本文档:reme4 中**长期记忆系统**的顶层认知模型 —— 把 agent 的记忆生命周期类比人类睡眠/觉醒回路,推导出**三阶段分工**与**15 维能力清单**。
|
||||
>
|
||||
> **三阶段实现各有专属文档**:
|
||||
> - Stage 1 写入(REM 重放抽象) → `auto_dream_design.md`
|
||||
> - Stage 2 巩固(NREM 深度整合) → `auto_consolidate_design.md`
|
||||
> - Stage 3 检索(觉醒态提取) → `auto_recall_design.md`
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_memory_design.md`:入流端(daily 写入),与 cognition 平行 —— cognition 负责"已落地后的认知循环",memory 负责"经历落地"
|
||||
> - `structure.md` §4(retrieve 三种问法)
|
||||
>
|
||||
> **核心立场**:
|
||||
> - 长期记忆不是"存 + 取"两个动作,是**写入 → 巩固 → 提取**的循环 —— 三段时间尺度不同(同步 / 周期 / 同步),设计形态不同
|
||||
> - vault 是**事实层**,只承载经过 LLM 写入认证的关系;`meta/` 是**派生层**,承载概率推断的统计信号
|
||||
> - 任一阶段独立演化,任一信号缺失系统降级而不崩
|
||||
|
||||
---
|
||||
|
||||
## 0. 心智循环:reme 的认知模型
|
||||
|
||||
agent 的长期记忆系统在概念上对应人脑的**海马—皮层回路 + 睡眠—觉醒周期**:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ 外部经验(daily / resource) │
|
||||
└─────────────┬───────────────────┘
|
||||
│ (auto-memory 写 daily)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ │
|
||||
│ ┌────────────────┐ 抽象 / 关系编织 │
|
||||
│ │ Stage 1 │ ◄─ 类比 REM 睡眠 │
|
||||
│ │ auto-dream │ "重放 + 写进 schema" │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 写 vault(digest body + wikilink) │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ vault(事实) │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 只读 │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ 长期组织 / 派生指标 │
|
||||
│ │ Stage 2 │ ◄─ 类比 NREM 慢波睡眠 │
|
||||
│ │ auto-consol- │ "巩固 + 修剪 + 集群" │
|
||||
│ │ idate │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 写 meta/ + audit/(派生层) │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ │
|
||||
│ │ meta(派生) │ │
|
||||
│ └───────┬────────┘ │
|
||||
│ │ 只读 │
|
||||
│ ▼ │
|
||||
│ ┌────────────────┐ query → 答案合成 │
|
||||
│ │ Stage 3 │ ◄─ 类比觉醒态 cue retrieval│
|
||||
│ │ auto-recall │ "融合 + pattern complete"│
|
||||
│ └───────┬────────┘ │
|
||||
│ │ │
|
||||
└───────────┼─────────────────────────────────────┘
|
||||
│ 召回结果给 agent
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ agent query │
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
**心智循环回答四个根本问题**:
|
||||
|
||||
| 问题 | 谁回答 |
|
||||
|---|---|
|
||||
| 我经历过什么? | auto-memory(daily 入流) |
|
||||
| 我从中学到什么? | Stage 1 — auto-dream |
|
||||
| 这些知识如何长期组织? | Stage 2 — auto-consolidate |
|
||||
| 我需要时如何调用? | Stage 3 — auto-recall |
|
||||
|
||||
memory 负责"经历落地",cognition 三阶段负责"已落地经历的认知循环"。
|
||||
|
||||
---
|
||||
|
||||
## 1. 三阶段全景
|
||||
|
||||
| 阶段 | 神经科学类比 | 时间尺度 | 改 vault | 实现归属 |
|
||||
|---|---|---|---|---|
|
||||
| **Stage 1 dream** | REM 重放抽象 | 同步(随入流即跑) | 是(写 digest body) | `auto_dream_design.md` |
|
||||
| **Stage 2 consolidate** | NREM 深度巩固 | 周期 / idle(daily / weekly)| **否**(写 `meta/` + `audit/`)| `auto_consolidate_design.md` |
|
||||
| **Stage 3 recall** | 觉醒态 cue retrieval | 同步(query 触发) | 否(只读;唯一对外写是 `meta/access_log.json`)| `auto_recall_design.md` |
|
||||
|
||||
**关键的不对称**:
|
||||
- 写入与检索是**同步**的(用户 / agent 等待),巩固是**离线**的(idle / 周期)
|
||||
- 改 vault 的资格被严格限制在 **dream + consolidate 中的 split** —— 其它阶段全只读
|
||||
- 三阶段时间尺度差三个数量级,这是设计形态(同步 vs 异步 vs idle)的根本来源
|
||||
|
||||
---
|
||||
|
||||
## 2. 系统级能力(贯穿三阶段)
|
||||
|
||||
不属任何单阶段,但任一阶段不能违反:
|
||||
|
||||
| 能力 | 含义 |
|
||||
|---|---|
|
||||
| **事实层 vs 派生层分离** | vault 只承载经 LLM 写入认证的关系(显式 wikilink);`meta/` 承载概率推断的派生指标(community / recency / archived);两者绝不混同 |
|
||||
| **不变量守恒** | F-invariants(0 文件移动 / 改正文限定 subject / wikilink 是 body 一部分)+ E-invariants(边守恒 E-1/E-2/E-3)横跨三阶段;详 `auto_dream_design.md` §4.3-§4.4 |
|
||||
| **阶段独立演化** | 任一阶段算法升级不破坏其它阶段(community 算法换 → dream 不变;打分公式调 → consolidate 不变) |
|
||||
| **缺失即降级** | 任一派生信号缺失,系统降级而不崩;冷启动可用 |
|
||||
| **全程可审计** | 每阶段产 audit / report / log,人 / agent 可检视追溯 |
|
||||
|
||||
---
|
||||
|
||||
## 3. Stage 1 — auto-dream:经验 → 抽象
|
||||
|
||||
**类比**:REM 睡眠的记忆重放与抽象提炼。脑在做梦时把白天事件拆解、重组,提取出可泛化的模式,登记进皮层 schema。
|
||||
|
||||
**根本目的**:把"原始经历"转化为"长期值得调取的教训",同时把它编织进已有知识图谱。
|
||||
|
||||
### 3.1 五个能力维度
|
||||
|
||||
逻辑递进 —— 输入 → 抽象 → 整合 → 编织 → 写入:
|
||||
|
||||
| # | 能力 | 它在问什么 | 失效后果 |
|
||||
|---|---|---|---|
|
||||
| 1 | **抽象判断**(gate) | 这段材料里有"值得长期记住"的东西吗? | 噪声进 vault / 只蒸馏不抽象 |
|
||||
| 2 | **经验重放**(召回) | 这个抽象在已有记忆里**已经存在**吗?以什么形式? | 重复节点 / 错过整合机会 |
|
||||
| 3 | **整合决策** | 创建新节点,还是丰富已有节点?若已有 —— 是再次印证 / 精化范围 / 修正错误? | 已有信息丢失 / 错误没纠正 |
|
||||
| 4 | **关系编织** | 这个抽象与谁有关系?谁是它的来源? | wikilink 缺失,后续 retrieve 漏召 |
|
||||
| 5 | **写入安全** | 写入会不会破坏 vault 既有事实?并发冲突如何处理? | 边丢失 / race condition |
|
||||
|
||||
### 3.2 关键定性
|
||||
|
||||
- dream 是 vault 的**唯一写者**(在 cognition 三阶段里;memory 写 daily 不算)
|
||||
- **写入瞬间是关系建立的唯一可信时机** —— 错过的关系不靠后台扫回(那不是 consolidate 的工作)
|
||||
- 一次写入,所有未来检索受益(持久化优于实时计算)
|
||||
|
||||
详细机制见 `auto_dream_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 4. Stage 2 — auto-consolidate:抽象 → 网络
|
||||
|
||||
**类比**:NREM 慢波睡眠的系统巩固 + 突触代谢稳态。脑在深睡时把分散事件融入 schema、修剪弱连接、把长期不用的记忆淡出意识可达范围。
|
||||
|
||||
**根本目的**:跨时间累积地把 vault 从"一堆节点"组织成"有结构、有权重、有时效的网络",但**只产派生信号,不污染事实层**。
|
||||
|
||||
### 4.1 五个能力维度
|
||||
|
||||
按作用尺度从微观到宏观:
|
||||
|
||||
| # | 能力 | 作用尺度 | 类比 | 输出形态 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **结构维护** | 节点级 | 海马表征过密 → 分化新单元 | 改 vault(split,唯一例外)|
|
||||
| 2 | **跨节点关系发现** | 节点对级 | 多次睡眠中识别"同一件事" → schema | `audit/` 报告 |
|
||||
| 3 | **主题集群形成** | 子图级 | 皮层网络的功能性分区 | `meta/communities.json` |
|
||||
| 4 | **时效性管理** | 节点级 / 时间维度 | 突触代谢稳态 + 遗忘 | `meta/access_log.json` + `meta/archived.json` |
|
||||
| 5 | **健康监控** | 系统级 | 神经环路诊断 | 告警 / 严重告警 |
|
||||
|
||||
### 4.2 关键定性
|
||||
|
||||
- consolidate 是**纯只读 + 派生写**(读 vault,写 `meta/` + `audit/`)
|
||||
- **唯一例外是 split** —— 改 vault 的维护任务,但触发严格(D3 inline 写后)且只改自身负责的 parent + children
|
||||
- **关系判断有错率 → 报告优先,人/agent 介入,不主动合并**(夸大置信度的代价是污染事实层)
|
||||
- 离线 / 周期 / idle —— 与前台不抢资源;失败不影响主流程,下次重跑
|
||||
|
||||
详细机制见 `auto_consolidate_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 5. Stage 3 — auto-recall:网络 → 答案
|
||||
|
||||
**类比**:觉醒态的 cue-driven retrieval + pattern completion。脑接到 query,激活相关皮层模式,补全成完整答案;同时召回过程本身强化被用到的记忆痕迹。
|
||||
|
||||
**根本目的**:接到当前 query 时,从 vault + 派生信号合成最相关的过去经验 —— 既要**覆盖率**(不漏)也要**信噪比**(不冗余)。
|
||||
|
||||
### 5.1 五个能力维度
|
||||
|
||||
按召回流程从输入到输出:
|
||||
|
||||
| # | 能力 | 它在解决什么 |
|
||||
|---|---|---|
|
||||
| 1 | **多路召回** | 不同问法走不同算子(state / semantic / topological 三分立);agent 自选,不强加聚合 verb |
|
||||
| 2 | **多信号融合** | 单一文本相似度不够 —— 还要节点权威性 / 主题集群 / 时效性;乘法融合 |
|
||||
| 3 | **信噪比管理** | 节点级去重 + 节点级 surface(frontmatter 一同呈现)+ multi-hop 可控展开 + 冷藏过滤 |
|
||||
| 4 | **召回反馈** | 被命中的节点 → 写访问日志 → 影响下次 recency / archived 判定 |
|
||||
| 5 | **鲁棒降级** | 派生信号缺失 → 退到基础召回;version 不兼容 → warning + 跳过该因子 |
|
||||
|
||||
### 5.2 关键定性
|
||||
|
||||
- recall 是**只读** —— 唯一对外写入是 `meta/access_log.json`(经 ring buffer + consolidate 聚合)
|
||||
- recall **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法分别由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖
|
||||
- 默认路径 **0 LLM 调用**(信号都是离线维护好的);LLM rerank / query rewrite 是 SDK 上层选项
|
||||
|
||||
详细机制见 `auto_recall_design.md`。
|
||||
|
||||
---
|
||||
|
||||
## 6. 能力地图(横切视角)
|
||||
|
||||
15 维按"作用对象"重排,可以看到三阶段如何分工:
|
||||
|
||||
| 作用对象 | dream(写入) | consolidate(巩固)| recall(检索)|
|
||||
|---|---|---|---|
|
||||
| **节点(单个)** | 1 抽象判断 / 3 整合决策 / 5 写入安全 | 1 结构维护(split) | 3 信噪比(节点级合并/surface) |
|
||||
| **节点对 / 关系** | 4 关系编织(wikilink) | 2 跨节点关系发现(dups 报告) | (消费已有边,不产新关系) |
|
||||
| **子图 / 集群** | 2 经验重放(召回邻居) | 3 主题集群形成(community)| 2 多信号融合(community boost) |
|
||||
| **时间维度** | (写入瞬间) | 4 时效性管理(decay / archived)| 4 召回反馈(access log)|
|
||||
| **系统健康** | 5 守恒校验 | 5 健康监控(D1 / D10) | 5 鲁棒降级 |
|
||||
| **入口形态** | 异步 fan-out per sub-unit | 周期 batch / idle | 同步 query response |
|
||||
|
||||
**几个观察**:
|
||||
- "节点对 / 关系"列在 recall 是空 —— recall 不产新关系,只用已有边(避免 query-time 高成本推断)
|
||||
- "时间维度"行 dream 缺位 —— 写入瞬间无"时间维度"概念(那是 consolidate 后续才能提取的统计)
|
||||
- 每行至少有一个阶段负责 —— 没有能力被全阶段忽略
|
||||
|
||||
---
|
||||
|
||||
## 7. 跨阶段不变量
|
||||
|
||||
所有阶段共同遵守的硬约束。任何阶段越界 = 设计错误。
|
||||
|
||||
### 7.1 F-invariants(继承 `auto_dream_design.md` §4.3)
|
||||
|
||||
| # | 约束 | 跨阶段含义 |
|
||||
|---|---|---|
|
||||
| F-1 | 0 文件移动 | 没有任何阶段可以 move 文件;rename 走 `wikilink_handler.retarget_links` 显式路径 |
|
||||
| F-2 | 改正文限定 subject | dream 改 subject body / consolidate split 改 parent + children body;**recall 绝不改任何 body** |
|
||||
| F-3 | maintainer 只做 split | consolidate 内的结构维护只做 split;无 merge / dissolve / re-edge |
|
||||
| F-10 | inbound 不动 | split 后外部 wikilink 仍指 parent,不强制重定向 |
|
||||
| F-11 | wikilink 是 body 一部分 | 没有"独立的边";所有关系变化是 body 编辑副作用 |
|
||||
|
||||
### 7.2 E-invariants(边守恒)
|
||||
|
||||
- E-1:dream update 出边 ⊇ 原出边
|
||||
- E-2:split 后 `(parent_new ∪ ∪children_outbound) ⊇ parent_old`
|
||||
- E-3:inbound wikilink split 时不动
|
||||
|
||||
**recall 不写 body** → E-* 与之无关;但 recall 看到的 wikilink 图永远是 dream / split 守恒后的状态。
|
||||
|
||||
### 7.3 派生信号边界
|
||||
|
||||
- **consolidate / recall 不写 vault** —— 关系判断、活跃度统计、社区划分都是概率推断,不污染事实层
|
||||
- **`meta/*.json` 不被 retrieve 召回** —— 只作权重信号,不进入"召回结果"集合
|
||||
- **audit/ 不被自动消费** —— 报告永远等待人 / agent 介入,不闭环回写
|
||||
|
||||
---
|
||||
|
||||
## 8. 跨阶段数据流(契约总览)
|
||||
|
||||
```
|
||||
┌──────────────┐ wikilink ┌──────────────┐
|
||||
│ auto-dream │─落 body──►│ vault/ │
|
||||
│ (Stage 1) │ │ (事实层) │
|
||||
└──────────────┘ └──────┬──────┘
|
||||
│ 只读
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ auto-consolidate │
|
||||
│ (Stage 2) │
|
||||
└─┬────────┬───────┘
|
||||
│ │
|
||||
meta/ 元数据───┘ └─── audit/ 报告
|
||||
(派生层) (人工介入)
|
||||
│
|
||||
│ 只读
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ auto-recall │ ◄─ user query
|
||||
│ (Stage 3) │
|
||||
└──────┬───────┘
|
||||
│ 命中钩子(异步)
|
||||
▼
|
||||
meta/access_log.json
|
||||
(recall 唯一对外写入,经 consolidate 聚合)
|
||||
```
|
||||
|
||||
| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 |
|
||||
|---|---|---|---|---|
|
||||
| **vault wikilink** | `digest/**.md` body | dream / split | recall(图遍历) | — |
|
||||
| **dups 报告** | `audit/<date>/auto_link_dups.md` | consolidate | 人 / agent | — |
|
||||
| **communities** | `meta/communities.json` | consolidate | recall | 不做同社区 boost |
|
||||
| **access log** | `meta/access_log.json` | recall(写命中) + consolidate(聚合) | recall(读 recency)| recency_factor = 1.0 |
|
||||
| **archived list** | `meta/archived.json` | consolidate | recall(默认过滤)| 不过滤 |
|
||||
| **centrality** | `file_graph` 反向索引(实时,不存)| 自动 | recall(O(1) 查) | — |
|
||||
|
||||
**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。
|
||||
|
||||
**冷启动**:`meta/` 为空 → recall 仍能跑(base + centrality + 图)→ 排序略弱不崩。
|
||||
|
||||
---
|
||||
|
||||
## 9. 系统级断言(把"要什么"提炼到 5 条)
|
||||
|
||||
1. **抽象与事实分层** —— vault 是经 LLM 写过的事实;`meta/` 是统计 / 算法的派生;两者绝不混同
|
||||
|
||||
2. **关系建立的时机集中在写入瞬间** —— dream 写入是关系唯一可信来源;consolidate 不补 vault 关系,recall 不预存关系矩阵
|
||||
|
||||
3. **维护是离线的派生劳动,不是补救** —— consolidate 不修 dream 的疏漏(那叫返工),它做的是 dream 不擅长的事(全局视角 / 统计视角 / 时间视角)
|
||||
|
||||
4. **检索是融合,不是检索** —— recall 的价值不在"找文本相似",而在"把文本 / 图 / 时效 / 权威多个独立信号合成一个答案"
|
||||
|
||||
5. **整个心智循环可降级** —— 任一阶段失效或失准,整个系统降级而不崩;冷启动有意义;dogfooding 可演进
|
||||
|
||||
---
|
||||
|
||||
## 10. 与 auto-memory 的边界
|
||||
|
||||
auto-memory 写入的 daily event 节点也是图的一部分(承载 daily → digest 的 `derived_from::` 边)。但 daily 节点**不参与 cognition 三阶段的全部改造**:
|
||||
|
||||
| cognition 阶段 | 是否触及 daily |
|
||||
|---|---|
|
||||
| **dream** | 只读(作为入流之一) |
|
||||
| **consolidate** | 不参与 dups / community / decay(daily 是时间索引,本质不去重 / 不冷藏) |
|
||||
| **recall** | 三层并行召回时 daily 也参与命中(`structure.md` R-2 默认 `digest > daily > resource`) |
|
||||
|
||||
**关键约束**:cognition 三阶段任何子阶段都**不改写 daily**(无写回路径);daily 由 auto-memory 写完即只读。
|
||||
|
||||
---
|
||||
|
||||
## 11. 演进 / 待补
|
||||
|
||||
**当前实现状态**:
|
||||
- ✅ Stage 1 dream 已实现并跑通(`reme4/steps/evolve/auto_dream.py` + `auto_dream.yaml`)
|
||||
- ⏳ Stage 2 consolidate split 部分将实现;dups / community / decay / archived 待实现
|
||||
- ⏳ Stage 3 recall 增强未实现(当前 search.py 已有 vector + keyword + RRF + 一跳 expand)
|
||||
|
||||
**顶层级演进议题**(不属任何单阶段):
|
||||
- ⏳ **能力成熟度路标** —— 把 15 个能力维度按 M0(必须)/ M1(期望)/ M2(演进)分级
|
||||
- ⏳ **跨阶段集成测试** —— vault 从空到充实的端到端 dogfooding,验证三阶段配合是否符合"心智循环"预期
|
||||
- ⏳ **可观测性聚合** —— 三阶段各自的 audit / log 现在分散;是否需要统一的 cognition 健康面板
|
||||
|
||||
各阶段实现进度详见各自文档的"下一步"章节。
|
||||
726
docs4/auto_consolidate_design.md
Normal file
726
docs4/auto_consolidate_design.md
Normal file
|
|
@ -0,0 +1,726 @@
|
|||
# auto-consolidate 设计(Stage 2 巩固:主动解决 vault 长期演化的实际问题)
|
||||
|
||||
> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 2 — 巩固阶段** 实现。覆盖 vault 长期演化中累积的实际问题(冗余 / 过载 / 稀疏 / 腐败 / 抽象缺位),通过周期 batch + 写后 inline 的方式**主动改 vault**,让记忆系统保持健康。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_cognition_design.md`:三阶段顶层心智循环
|
||||
> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型 / F-invariants 原始定义 / 边守恒
|
||||
> - `auto_recall_design.md`:Stage 3 检索 —— 消费本文档产出的信号
|
||||
> - `auto_memory_design.md`:auto-memory 写 daily,daily 节点不参与本文档的巩固改造
|
||||
> - `structure.md` §3.6(maintain 动作语义)
|
||||
>
|
||||
> **核心立场**:
|
||||
> - consolidate **不是产报告等人介入**,是**主动解决问题** —— 类比 NREM 慢波睡眠的 systems consolidation:跨多事件抽 schema、修剪弱连接、稳态突触强度。这些都是真实发生的改造
|
||||
> - vault **会被 consolidate 改**,但每个动作有严格的**置信度门槛 + 守恒规则 + 审计 trail + 渐进 rollout**
|
||||
> - 灰色地带(置信度不够)才产报告等人介入;高置信度自己解决
|
||||
> - **community detection 是巩固的中枢** —— P0 基础设施,P1-P3 三个动作(abstract / merge / reinforce)都依赖它
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述与五大动作全景
|
||||
|
||||
dream 写入是单点视角,有三类视野局限:**写入瞬间没有跨节点视角 / 跨时间视角 / 全局拓扑视角**。这些局限会让 vault 长期演化中累积五类实际问题:
|
||||
|
||||
| # | 问题 | 类比 | 表现 | 解决 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **冗余** | 同事件留下重复记忆痕迹 | dream 漏判去重 / 术语演化 / 跨桶建成两份 | merge |
|
||||
| 2 | **过载** | 单一突触表征过密 | 节点 body 累积过长 / 单节点杂糅多主题 | split |
|
||||
| 3 | **稀疏** | 应有连接未建立 | dream 写入瞬间漏召回的相关节点 / 反复共现但无 wikilink | reinforce |
|
||||
| 4 | **腐败** | 长期不激活的痕迹 | 旧节点过时 / 半年没人读 / 内容已被矛盾 | archive |
|
||||
| 5 | **抽象缺位** | 跨多 instance 缺 schema | vault 只有原子节点,没有"主题层"视角承接全局问 | abstract |
|
||||
|
||||
### 0.1 五大动作 + 优先级
|
||||
|
||||
| 优先级 | 动作 | 解决问题 | 触发节奏 | 改 vault | 风险 | 收益 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **P0** | **community detection** | (基础设施) | weekly batch | 否 | 0(只产 meta) | 基础(其它三个都靠它)|
|
||||
| **P1** | **abstract** | 抽象缺位 | weekly batch(基于 P0) | 是(新建 summary) | 低(additive) | **最高**(GraphRAG 核心) |
|
||||
| **P2** | **merge** | 冗余 | weekly batch(基于 P0) | 是(合并 + retarget) | 高(lossy) | 中(消除可见冗余) |
|
||||
| **P3** | **reinforce** | 稀疏 | weekly batch(基于 P0) | 是(additive 加 wikilink) | 低 | 低(retrieve multi-hop 已部分弥补)|
|
||||
| **(独立)** | **split** | 过载 | inline 写后(D3) | 是(拆 parent + children) | 低 | 中 |
|
||||
| **(独立)** | **archive** | 腐败 | daily batch | 软(meta 标记) | 0 | 中 |
|
||||
|
||||
**关键论断**:**P1 比 P2 优先** —— abstract additive 失败可逆且回报最大;merge lossy 失败要回滚 inbound,价值是消除冗余(必要但不增能力)。
|
||||
|
||||
### 0.2 实施路径
|
||||
|
||||
```
|
||||
M0: P0 community detection (基础设施)
|
||||
+ split (已实现)
|
||||
+ archive (软标记,完全可逆)
|
||||
|
||||
M1.1: P1 abstract (additive,最低风险开始改 vault)
|
||||
M1.2: P2 merge (lossy,高门槛 + 多数票)
|
||||
M1.3: P3 reinforce (additive,价值最低,可缓做)
|
||||
|
||||
M2+: 多层 abstract (L2 super-community) / delete / typed predicate reinforce
|
||||
```
|
||||
|
||||
### 0.3 显式排除
|
||||
|
||||
- ❌ 重做"抽象判断" —— gate 决策只在 dream(consolidate 不重新判定"该不该记")
|
||||
- ❌ 重做"语义内容" —— UPDATE 三种 flavor(CORROBORATE / REFINE / CORRECT)只在 dream;consolidate 做结构层,不做语义层
|
||||
- ❌ 改 daily / resource —— consolidate 只动 digest 节点(I-2 / I-3 仍守)
|
||||
|
||||
---
|
||||
|
||||
# Part A — community 工作群(本文档核心)
|
||||
|
||||
P0-P3 四件套围绕 community detection 协同工作:**community 提供"哪些节点同主题"的判据,abstract / merge / reinforce 各自利用这个判据做不同的解决动作**。
|
||||
|
||||
## 1. community detection(P0,基础设施)
|
||||
|
||||
**目的**:在 vault wikilink 图上做 community detection,产出"节点 → community_id"映射。这是 P1-P3 三个动作的**唯一前置**。
|
||||
|
||||
### 1.1 算法选择:Leiden
|
||||
|
||||
| 选项 | 评估 |
|
||||
|---|---|
|
||||
| Louvain | 经典,但有 resolution limit + disconnected community 风险 |
|
||||
| **Leiden** ✅ | Louvain 改进版(2019),稳定性显著好;GraphRAG 采用;Python `igraph.community_leiden` 现成 |
|
||||
| label propagation | 实现最简,但结果不稳定(随机种子敏感) |
|
||||
|
||||
**首版决策:Leiden**,直接对齐 GraphRAG 路线,后续接它的多层抽象更顺。
|
||||
|
||||
### 1.2 图的形态
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **节点范围** | **只 digest 节点**;daily / resource 不参与 |
|
||||
| **边权重** | **首版 unweighted undirected**(所有 wikilink 等权)—— 加权方案(predicate 类型加权)留 M2+ 视效果 |
|
||||
| **跨桶 community** | **必须允许** —— bucket 是物理归档,community 是语义聚合,二者本就正交。"错桶节点"会被自然纳入 community,可作 audit 信号但不强制 move(F-1 守住)|
|
||||
| **resolution** | **1.0 起步**(Leiden 默认 / GraphRAG 默认)—— dogfooding 后视 community 平均规模(理想 5-15 节点)调 |
|
||||
| **更新模式** | **全量重算**;vault 千节点级 Leiden < 1 秒,M0/M1 不引入增量复杂度 |
|
||||
|
||||
### 1.3 多层级:M1 只 L1
|
||||
|
||||
| 层数 | 适用 | reme 决策 |
|
||||
|---|---|---|
|
||||
| 单层 L1(原子 → community)| vault < 500 节点足够 | **M1 起步** |
|
||||
| 双层 L1 + L2(community → super-community) | vault > 500 节点 / 跨主题大类涌现 | M2+ 视规模 |
|
||||
| GraphRAG 4 层 | 大规模文档库 | M3+ 不优先 |
|
||||
|
||||
理由:GraphRAG 论文证明 L1 拿走 60-80% 效果。先把 L1 跑稳,L2 看实际是否需要。
|
||||
|
||||
### 1.4 输出
|
||||
|
||||
**`meta/communities.json`**:
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"computed_at": "2026-06-08T03:00:00Z",
|
||||
"algorithm": "leiden",
|
||||
"resolution": 1.0,
|
||||
"communities": {
|
||||
"digest/auth/jwt-rotation.md": "c_07",
|
||||
"digest/auth/oauth-flow.md": "c_07",
|
||||
"digest/api/rate-limit.md": "c_12"
|
||||
},
|
||||
"stats": {
|
||||
"n_communities": 14,
|
||||
"median_size": 7,
|
||||
"max_size": 23
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`meta/community_changes.json`**(供 abstract 稳定度判据):
|
||||
```json
|
||||
{
|
||||
"computed_at": "...",
|
||||
"previous": "...",
|
||||
"stability_per_community": {
|
||||
"c_07": 0.92, // 1 - (Jaccard 距离与上周该 community 节点集)
|
||||
"c_12": 0.45 // 不稳定,abstract 跳过
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 1.5 community_id 不需要稳定
|
||||
|
||||
下游(abstract / merge / reinforce)只关心"两节点是否同 community";id 本身可重排。每周重算后 id 不需要保持与上周对齐。stability 信号通过节点集 Jaccard 距离计算,不依赖 id。
|
||||
|
||||
### 1.6 用途总览
|
||||
|
||||
| 下游 | 用法 |
|
||||
|---|---|
|
||||
| **abstract**(§2)| 判据"该 community 节点数 ≥ N + 稳定度满足 + 无 hub" → 创建 summary |
|
||||
| **merge**(§3)| 候选 pair 必须在同 community(降错率;不同 community 的相似 description 多是同名异义)|
|
||||
| **reinforce**(§4)| 候选 wikilink 必须在同 community(避免假关联)|
|
||||
| **recall**(`auto_recall_design.md` §3) | 同 community 节点 boost |
|
||||
|
||||
---
|
||||
|
||||
## 2. abstract(P1,抽象提升)
|
||||
|
||||
**类比**:NREM systems consolidation —— 跨多次睡眠把分散事件抽出共同 schema,从 episodic 升到 semantic。
|
||||
|
||||
**目的**:vault 演化到一定规模后,某些 community 形成稳定主题群,需要一个 hub 节点统领,让 retrieve 能召回到"主题概览"而非散点。
|
||||
|
||||
### 2.1 等价处理立场(关键)
|
||||
|
||||
**summary 节点完全等同普通节点**:
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **路径** | LLM 选桶,正常 slug 命名(如 `digest/auth/authentication-mechanisms.md`);**无 `__community__` / `__hub__` 等结构性标识** |
|
||||
| **frontmatter** | 仅 `name + description`(reme 核心保留);**无 `kind: community_summary`、无 `auto_generated`** |
|
||||
| **summary 性质** | 完全体现在 **body 形态** —— 主题概述 + 列出 source 节点 wikilink + 跨节点 pattern;但这是内容自然形态,不是结构性宣告 |
|
||||
| **后续维护** | **无** —— 跟其它节点等价,被 dream / split / merge / archive 自然演化(参见 §2.6) |
|
||||
|
||||
这跟 dream 的核心立场对齐:"节点角色由 body 内容决定,不由 frontmatter 类型标记"。abstract 是"用一种新方式创造节点",不是"创造一种新节点类型"。
|
||||
|
||||
### 2.2 触发判据(组合门槛)
|
||||
|
||||
```
|
||||
weekly batch:
|
||||
for community in communities.json:
|
||||
if community_has_hub(community): # §2.5 结构化判据
|
||||
continue
|
||||
if len(community) < MIN_NODES (5): # 节点数门槛
|
||||
continue
|
||||
if stability(community) < 0.7: # 稳定度门槛
|
||||
continue
|
||||
if active_node_count(community, 30d) < 3: # 活跃度门槛
|
||||
continue
|
||||
if name_diversity(community) < 0.5: # 多样性门槛
|
||||
continue
|
||||
→ enqueue abstract job
|
||||
```
|
||||
|
||||
| 门槛 | 默认 | 含义 | 防的是 |
|
||||
|---|---|---|---|
|
||||
| **节点数** | ≥ 5 | community 大小 | 给 2-3 节点造 hub 不划算 |
|
||||
| **稳定度** | ≥ 0.7 | 与上周边界 Jaccard 距离 | 给短命 community 造 hub 浪费 |
|
||||
| **活跃度** | ≥ 3 节点近 30 天 hit | community 仍在用 | 给死社区造 hub(下次没人看)|
|
||||
| **多样性** | name 差异度 ≥ 0.5 | frontmatter `name` 互不相同 | 给"一组重复节点"造 summary —— 那是 merge 的事 |
|
||||
|
||||
### 2.3 创建动作 + grounding 守恒
|
||||
|
||||
```
|
||||
LLM 看 community 内所有节点 (frontmatter + body)
|
||||
↓
|
||||
产 planned summary body (三段):
|
||||
1. 主题概述 (1-2 段,跨多节点共同主题)
|
||||
2. 关键支柱 (列表,3-5 节点 + 一句话 + wikilink)
|
||||
3. 不在概览的细节 (明说哪些细节留原节点)
|
||||
↓
|
||||
长度限制: summary body < 1500 token
|
||||
(防 abstract 创建后立刻被 split 触发,§5)
|
||||
↓
|
||||
LLM 决定 path: digest/<bucket>/<slug>.md
|
||||
↓
|
||||
CAS 写入 (§9) + 双重守恒校验:
|
||||
- 机械: 出边集合 ⊇ "关键支柱"声称引用的节点 (防套话)
|
||||
- 机械: 出边集合 ⊇ source_nodes 的至少 60% (allow LLM 漏列少数)
|
||||
↓
|
||||
audit 记录: audit/<date>/consolidate_actions.md
|
||||
```
|
||||
|
||||
**grounding 守恒**:summary body 中**声称引用某节点必须真写 wikilink**。LLM 不能仅口头提及"我们在 X 中看到..."而不带 `[[X.md]]`。这是机械可校验的,LLM 跑不掉。
|
||||
|
||||
### 2.4 长度限制为什么重要
|
||||
|
||||
summary body < 1500 token 是**与 split 互锁的机制**:
|
||||
|
||||
- 不限长 → LLM 会写"完整覆盖" → 最终 body 累积接近 split 阈值(2000 token)→ 下次 D3 触发拆 → 拆出来的 children 又被 community 视为同主题 → 下次 abstract 又造一个 hub → 循环
|
||||
- 限长 1500 → summary 留出 split 阈值的 25% buffer,稳定不触发拆
|
||||
|
||||
### 2.5 "community 已有 hub"的结构化判据
|
||||
|
||||
不靠 frontmatter / 路径标识,靠**结构**:
|
||||
|
||||
```
|
||||
def community_has_hub(community):
|
||||
for node in community:
|
||||
out_targets = outbound(node) ∩ community
|
||||
if len(out_targets) / len(community) >= 0.6:
|
||||
return True # 该节点出边覆盖 community 60% 以上 → 它已是 hub
|
||||
return False
|
||||
```
|
||||
|
||||
**好处**:
|
||||
- split parent overview 自然被识别为 hub(split parent 出边覆盖大部分 children)→ abstract **复用** split 的工作,不重复创建
|
||||
- 已有 abstract 创建过的节点,只要它出边没退化,下次 batch 自然识别为 hub,不重复创建
|
||||
- 节点被 dream update 后形态变化,出边变了 → 自动重新评估
|
||||
|
||||
**M1 实施关键验证点**:跑实测验证这个涌现 —— split parent 是否真被识别为 hub。如有 corner case,调阈值 0.6 → 0.5 / 0.7。
|
||||
|
||||
### 2.6 后续维护:无 —— 完全靠 5 大动作演化
|
||||
|
||||
abstract 创建即放归 vault,**consolidate 不再"管"它**。后续命运:
|
||||
|
||||
| 演化路径 | 结果 |
|
||||
|---|---|
|
||||
| 新材料触及该主题 | dream update 自然修正 body(走 CORROBORATE / REFINE / CORRECT)|
|
||||
| 老 summary 长期不被引用 | archive 自动归档(§6)|
|
||||
| community 边界变了 → 下次 batch 创建新 summary | 新老 summary 描述同主题 → merge 自动合并(§3)|
|
||||
| summary body 累积过长 | split 自动拆(§5)|
|
||||
|
||||
这是真正的"vault 自我代谢"。**没有特殊维护通道**。
|
||||
|
||||
---
|
||||
|
||||
## 3. merge(P2,同概念合并)
|
||||
|
||||
**类比**:NREM 跨多次睡眠识别"同一件事" → 合一个记忆痕迹。
|
||||
|
||||
**目的**:消除 vault 内的冗余 —— 同概念多节点。
|
||||
|
||||
### 3.1 候选挖掘(community 内三层过滤)
|
||||
|
||||
```
|
||||
weekly batch (依赖 community detection):
|
||||
for community in communities:
|
||||
pairs = all_pairs(community)
|
||||
for (A, B) in pairs:
|
||||
if description_sim(A, B) < 0.6: # 第一层: frontmatter 相似
|
||||
continue
|
||||
if body_topic_overlap(A, B) < 0.5: # 第二层: body 主题词重合
|
||||
continue
|
||||
if cooldown_active(A) or cooldown_active(B): # 第三层: cooldown 检查
|
||||
continue
|
||||
candidates.append((A, B))
|
||||
```
|
||||
|
||||
**关键约束**:候选必须在**同 community**(降错率)。
|
||||
|
||||
### 3.2 多数票决策
|
||||
|
||||
merge 是高风险动作(lossy + 改 inbound),用多数票降错:
|
||||
|
||||
```
|
||||
for (A, B) in candidates:
|
||||
votes = parallel_run(N=3, prompt="A 和 B 是否同一概念? 返回 {is_same, confidence}")
|
||||
agree = sum(v.is_same and v.confidence >= 0.8 for v in votes)
|
||||
if agree >= 2:
|
||||
→ enqueue merge job
|
||||
elif agree == 1:
|
||||
→ 写 audit/<date>/dups_uncertain.md (灰色地带,人介入)
|
||||
else:
|
||||
→ 丢弃
|
||||
```
|
||||
|
||||
### 3.3 merge 动作:body 重写归 consolidate(方案 B)
|
||||
|
||||
**关键决策**:merge 后的 body 由 **consolidate 自跑合并 prompt**,不走 dream update 路径。
|
||||
|
||||
| 方案 | 评估 | 决策 |
|
||||
|---|---|---|
|
||||
| A. 走 dream update 路径(把 loser body 作"新材料")| 优雅但跨阶段;dream 不应知道 caller 是 consolidate 还是新材料 | ❌ |
|
||||
| **B. consolidate 自跑合并 prompt** | 简单自包含;通过严格 prompt 约束化解"做语义工作"张力 | ✅ |
|
||||
| C. 不重写 body(留 redirect stub) | 完全不做语义,但 vault 留无用节点 | ❌ |
|
||||
|
||||
**B 方案的边界守住**(避免 consolidate 真在做语义判断):
|
||||
|
||||
| 边界 | 含义 |
|
||||
|---|---|
|
||||
| **prompt 严格约束** | "只合并不精化" —— 不重写措辞、不加新内容、不做精化决策 |
|
||||
| **机械守恒** | 出边 ⊇ A.outbound ∪ B.outbound + provenance 全保留(LLM 跑不掉) |
|
||||
| **信息守恒抽样** | LLM 自检 "merged.body ⊇ A.body ∪ B.body 全部信息";audit 抽样人审 |
|
||||
| **失败拒写** | 守恒校验失败 → LLM 重试一次 → 二次失败拒写 + audit |
|
||||
|
||||
### 3.4 完整动作流
|
||||
|
||||
```
|
||||
A, B → 选择 winner (path):
|
||||
- inbound 数大者赢 (保护既有 inbound,降 retarget 量)
|
||||
- 平局取路径短者
|
||||
↓
|
||||
LLM 跑 merge prompt → planned merged_body (B 方案)
|
||||
↓
|
||||
机械 retarget 准备:
|
||||
- 扫所有 inbound(loser): [[loser.md]] → [[winner.md]]
|
||||
- alias 保留;predicate 保留
|
||||
- 这是机械算子,非 LLM
|
||||
↓
|
||||
事务式 CAS 写入:
|
||||
1. winner body 改写
|
||||
2. 所有 inbound 节点 body 改写 (retarget)
|
||||
3. 删除 loser 文件
|
||||
任一步失败 → 全部回滚
|
||||
↓
|
||||
audit 记录 + cooldown 设置 (winner 进 cooldown 2 weeks)
|
||||
```
|
||||
|
||||
### 3.5 灰色地带:报告
|
||||
|
||||
- 多数票通过(agree ≥ 2)→ 自动 merge
|
||||
- 仅 1 票通过 → 写报告 `audit/<date>/dups_uncertain.md`,人 / agent 介入
|
||||
- 0 票 → 丢弃
|
||||
|
||||
报告格式:
|
||||
```markdown
|
||||
# dups uncertain 2026-06-08
|
||||
|
||||
## pair 1 (1/3 votes)
|
||||
- A: digest/auth/jwt-rotation.md ("JWT 密钥轮换")
|
||||
- B: digest/security/key-rotation.md ("密钥轮换原则")
|
||||
- vote 1 (yes, 0.85): "同一概念,A 偏 JWT 场景"
|
||||
- vote 2 (no, 0.72): "B 是通用原则,A 是具体应用"
|
||||
- vote 3 (no, 0.68): "粒度不同,不应合并"
|
||||
|
||||
建议:走 dream update 通道把 A 内容作为 B 的实例并入。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. reinforce(P3,关系强化:补 dream 漏的 wikilink)
|
||||
|
||||
**类比**:NREM 突触强化 LTP —— 反复共激活的连接被强化。
|
||||
|
||||
**目的**:vault 演化中,某些节点对应该有 wikilink 但 dream 写入时漏召。reinforce 周期检测并 additive 补。
|
||||
|
||||
### 4.1 候选挖掘(三层过滤)
|
||||
|
||||
```
|
||||
weekly batch (依赖 community detection):
|
||||
for community in communities:
|
||||
for (A, B) in all_pairs(community):
|
||||
if has_wikilink(A, B):
|
||||
continue
|
||||
# 第一层: 字符串 mention 锚点
|
||||
if not has_mention(A.body, B.frontmatter.name):
|
||||
continue
|
||||
# 第二层: embedding 相似度验证
|
||||
if embedding_sim(A.context_around_mention, B.body) < 0.7:
|
||||
continue
|
||||
# 第三层: 同 community (已经是,但显式说明)
|
||||
candidates.append((A, mention_pos, B))
|
||||
```
|
||||
|
||||
**三层过滤的角色**:
|
||||
|
||||
| 层 | 防的是 |
|
||||
|---|---|
|
||||
| 字符串 mention | 大幅降候选数(从 O(N²) 降到 O(实际共现)) |
|
||||
| embedding 相似度 | 防同名异义("Apple" 公司 vs 水果)|
|
||||
| 同 community | 防表面术语共现但语义无关 |
|
||||
|
||||
### 4.2 决策(单票即可,门槛较高)
|
||||
|
||||
reinforce 是 additive 低风险动作,不需要多数票:
|
||||
|
||||
```
|
||||
for (A, mention_pos, B) in candidates:
|
||||
vote = LLM("A.body 在该位置提到 B 的概念。是否合理加 [[B.md]] 链接?")
|
||||
if vote.confidence >= 0.85:
|
||||
additive_wikilink(A, mention_pos, target=B.path)
|
||||
→ CAS 写入 (E-1 自动满足:additive 只增不删)
|
||||
→ audit 记录
|
||||
else:
|
||||
丢弃
|
||||
```
|
||||
|
||||
### 4.3 边界
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **只 additive 加 wikilink** | 不改 body 文字,不升级 typed predicate(predicate 升级是语义判断,留 dream)|
|
||||
| **alias 保留原文** | `[[B.md\|<原文 mention>]]`;原文一字不改 |
|
||||
| **写入位置** | mention 第一次出现处加;后续保持原文(防 wikilink 满文) |
|
||||
| **不动 anchor** | 与 dream 一致 |
|
||||
| **守恒** | E-1 天然满足(纯增) |
|
||||
| **rollback** | 误链发生时,人 / agent 直接编辑 body 删除 wikilink 即可;reinforce 不维护"我加过哪些"audit log(每次动作进 `audit/<date>/consolidate_actions.md`)|
|
||||
|
||||
### 4.4 reinforce 与 dream 的边界
|
||||
|
||||
dream 写入时 LLM 应已尽力召回相关节点 + 加 wikilink。reinforce 是**周期性兜底** —— 写入瞬间漏的、术语后才一致的、被 split 拆出来后才相关的,在 reinforce batch 里被检出。
|
||||
|
||||
这不违反"consolidate 不修 dream 漏的"立场 —— **dream 漏的 wikilink 在巩固阶段补,是合法工作**(它的依据是 dream 单点视角永远做不到的"周期统计 + 全局视角");**dream 漏的语义抽象在巩固阶段不补**(那是 dream 的语义判断,consolidate 不重做)。
|
||||
|
||||
---
|
||||
|
||||
# Part B — 独立工作
|
||||
|
||||
P0-P3 围绕 community,这两个动作独立运行。
|
||||
|
||||
## 5. split(过载分化:inline 写后)
|
||||
|
||||
**类比**:海马表征过密 → 分化新单元。
|
||||
|
||||
**目的**:节点 body 累积过长 / 主题离散后,拆成 parent overview + N children,保持单节点"一个原子语义单元"的粒度。
|
||||
|
||||
### 5.1 触发模型(写后立即,inline)
|
||||
|
||||
split 是 5 大动作中**唯一 inline** 的 —— 跟 dream 写入流强耦合,不走 weekly batch:
|
||||
|
||||
```
|
||||
dream / split 写 body 成功 (CAS 通过)
|
||||
└─ if len(body) > T_token (default 2000):
|
||||
└─ LLM 判离散度
|
||||
└─ if is_overloaded:
|
||||
└─ enqueue split job (FIFO, CAS-protected)
|
||||
└─ return (不阻塞 dream)
|
||||
```
|
||||
|
||||
理由:节点过载是**写入瞬间的本地信号**(token + 离散度),延后无价值;反应即时。
|
||||
|
||||
### 5.2 split 动作
|
||||
|
||||
```
|
||||
LLM 看 parent body:
|
||||
- 拆成 1 个 parent overview body + N 个 children body
|
||||
- 每个 child 自带 [[parent]] 反向链接
|
||||
- inbound 不动 (F-10)
|
||||
↓
|
||||
机械 outbound 守恒校验 (E-2):
|
||||
(parent_new ∪ ∪children_outbound) ⊇ parent_old
|
||||
失败 → LLM 重试 → 二次失败拒写 + audit
|
||||
↓
|
||||
事务式 CAS 写入: parent body 改写 + N 个新 children 文件创建
|
||||
↓
|
||||
audit + cooldown 设置 (parent + children 进 cooldown,与 merge 互锁)
|
||||
```
|
||||
|
||||
### 5.3 split 与 abstract 的协同(关键)
|
||||
|
||||
| | 起源 | 方向 | 触发 |
|
||||
|---|---|---|---|
|
||||
| split overview | 单节点过载分化 | 自上而下(一拆多)| inline 写后 D3 |
|
||||
| abstract summary | 多节点抽象凝聚 | 自下而上(多归一)| weekly batch + 稳定度阈值 |
|
||||
|
||||
**协同**:split 产出的 overview 节点会被 §2.5 的"已有 hub"判据识别,abstract 不重复创建。两者互补,不冲突。
|
||||
|
||||
---
|
||||
|
||||
## 6. archive(时效衰减:让长期不激活的节点淡出)
|
||||
|
||||
**类比**:突触代谢稳态 —— 长期不用的连接被减弱,但不删除。
|
||||
|
||||
**目的**:让 retrieve 默认排除"已不活跃"的节点,提升信噪比;不删 vault 文件,保持可逆。
|
||||
|
||||
### 6.1 recency_score:连续衰减信号
|
||||
|
||||
```
|
||||
recency_score(node) =
|
||||
exp(-(now - last_update) / τ_update) # 时间衰减
|
||||
× (1 + log(1 + last_hit_count_30d)) # 活跃度增强
|
||||
× (1 + log(1 + inbound_count) / SCALE) # 中心性 cushion(避免 hub 被冷藏)
|
||||
```
|
||||
|
||||
| 参数 | 默认 | 含义 |
|
||||
|---|---|---|
|
||||
| τ_update | 60 days | 时间衰减常数 |
|
||||
| SCALE | 10 | 中心性 cushion 缩放 |
|
||||
|
||||
输出:`meta/recency.json`,每节点 0.0~1.0 连续值。
|
||||
|
||||
### 6.2 archived 派生快照
|
||||
|
||||
archived 是 recency_score 的二元化派生:
|
||||
|
||||
```
|
||||
archived = {node | recency_score(node) < 0.15}
|
||||
```
|
||||
|
||||
输出:`meta/archived.json`,recall 默认过滤这个列表。
|
||||
|
||||
### 6.3 解冻
|
||||
|
||||
任何动作触及节点 → 自动从 archived 移除:
|
||||
- retrieve 命中(写 access_log)
|
||||
- dream update 触及
|
||||
- merge / reinforce 触及
|
||||
|
||||
下次 batch 时 recency_score 重算自然超过阈值。
|
||||
|
||||
### 6.4 daily 节奏
|
||||
|
||||
archive 是唯一不需要 community detection 的动作 → 节奏可以更快(daily batch),让冷启动后第二天就能影响 recall。
|
||||
|
||||
```
|
||||
daily batch:
|
||||
1. 读 access_log (retrieve / dream / consolidate 钩子记录的命中事件)
|
||||
2. 重算 recency_score for all digest nodes
|
||||
3. 输出 meta/recency.json
|
||||
4. 阈值过滤 → meta/archived.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Part C — 共享基础设施
|
||||
|
||||
## 7. F-invariants 松绑与守恒规则
|
||||
|
||||
旧 F-invariants(`auto_dream_design.md` §4.3)在"vault 只读"立场下定义,新立场要松绑。但松绑不是"自由改",是用**动作级守恒规则**换"一刀切禁令"。
|
||||
|
||||
### 7.1 F-invariants 修订
|
||||
|
||||
| # | 旧约束 | 新立场 |
|
||||
|---|---|---|
|
||||
| **F-1** | 0 文件移动 | **改为**:"非 consolidate 动作不移动文件";merge 删除 loser 文件是**合法移动**(逻辑上等价 retarget) |
|
||||
| **F-2** | 改正文限定 subject | **改为**:"dream / split / reinforce 改 subject body;merge 在受控算子内可改 inbound 节点 body";其它阶段(recall)绝不改 |
|
||||
| **F-3** | maintainer 只做 split | **作废** —— consolidate 5 大动作合法 |
|
||||
| **F-10** | inbound 不动 | **改为**:"split 时 inbound 不动";merge 必须 retarget inbound(机械算子) |
|
||||
| **F-11** | wikilink 是 body 一部分 | **保留** —— 没有"独立的边"基础设施 |
|
||||
|
||||
### 7.2 动作级守恒规则矩阵
|
||||
|
||||
| 动作 | 置信度门槛 | 守恒规则 |
|
||||
|---|---|---|
|
||||
| **abstract** | community 节点 ≥ 5 + 稳定度 ≥ 0.7 + 活跃度 ≥ 3 + 多样性 ≥ 0.5 + 无 hub | 出边 ⊇ "关键支柱"列表 + 出边 ⊇ source 节点 60%(机械)|
|
||||
| **merge** | LLM 多数票 ≥ 2/3 + similarity ≥ 0.6 + body overlap ≥ 0.5 | 信息守恒(merged.body ⊇ A ∪ B)+ 出边 ⊇ A.out ∪ B.out + inbound 全 retarget(机械)|
|
||||
| **reinforce** | LLM 单票 ≥ 0.85 + 同 community + mention 锚点存在 + embedding ≥ 0.7 | E-1 天然(additive)|
|
||||
| **archive** | recency_score < 0.15 | 软标记,无破坏性 |
|
||||
| **split** | token > T + LLM 判离散 | E-2(parent ∪ children ⊇ parent_old)+ inbound 不动 |
|
||||
|
||||
---
|
||||
|
||||
## 8. cooldown 与防循环
|
||||
|
||||
5 大动作之间的潜在循环:
|
||||
|
||||
```
|
||||
A merge B → AB body 长 → split AB 回 A' + B' → 又 merge → ...
|
||||
```
|
||||
|
||||
防御:
|
||||
|
||||
| 互锁对 | 窗口 | 实现 |
|
||||
|---|---|---|
|
||||
| **split → merge** | 2 weeks | 刚 split 出的兄弟节点不参与 merge 候选 |
|
||||
| **merge → split** | 2 weeks | 刚 merge 的节点不参与 split 评估(D3 检测时跳过)|
|
||||
| **merge → merge**(同对反复) | 12 weeks | 同一 path 12 周内被 merge 又被识别为新 merge 候选 → audit 警报,人介入 |
|
||||
| **abstract → merge**(同主题反复 abstract) | 4 weeks | 刚 abstract 出的 hub 节点 4 周内不参与 merge 候选 |
|
||||
|
||||
cooldown 状态外置 `meta/cooldowns.json`,不污染 vault。
|
||||
|
||||
---
|
||||
|
||||
## 9. CAS 写入协议(共享基础设施)
|
||||
|
||||
CAS 是 dream(`auto_dream_design.md` §4.2)、split / merge / reinforce / abstract(本文档)**多方共用**的 vault 写入协议。归本文档因 consolidate 是写入主战场。
|
||||
|
||||
archive 不写 vault → 不走 CAS;它写 `meta/`,各任务的 atomic write(write-temp + rename)即可。
|
||||
|
||||
### 9.1 协议
|
||||
|
||||
```
|
||||
1. 读 + 记戳: read body → version_stamp = sha256(body) | mtime
|
||||
2. 决策: LLM / 算法 → 产 planned new_body
|
||||
3. CAS 写入: 重读 body 比 version_stamp
|
||||
- 未变: 跑动作级守恒校验 → 通过 → atomic write (write-temp + rename) → done
|
||||
- 已变: 丢弃 planned new_body, 带最新 body 重走 step 1
|
||||
4. 守恒校验失败: LLM 重试一次, 二次失败拒写 + audit
|
||||
5. 重做次数上限: 3 次 → 跳过候选 + audit log
|
||||
```
|
||||
|
||||
### 9.2 事务式 merge / split 写入
|
||||
|
||||
merge 涉及多文件写入(winner body + N 个 inbound retarget + loser 删除);split 涉及多文件创建(parent body + N children)。需要事务语义:
|
||||
|
||||
- 准备阶段:全部 planned new_body 写到 temp 区(带 version_stamp)
|
||||
- 提交阶段:逐个 CAS 检查 + atomic write(write-temp + rename)
|
||||
- 任一 CAS 失败 → 全部回滚(temp 区清理,已 rename 的恢复)
|
||||
|
||||
实现细节:可借 fs-level 事务库(如 `pyrsistent` 模式)或自实现 journal。M0 起步用最简的"先全部检查 → 再全部写入"两阶段,接受窗口期(检查到写入间)的极小并发风险。
|
||||
|
||||
### 9.3 create 路径 race
|
||||
|
||||
merge / abstract 都可能并发 create 同一 path → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者 EEXIST → 重走 step 1(此时大概率改判 update 或丢弃)。
|
||||
|
||||
### 9.4 不解决
|
||||
|
||||
- 跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+)
|
||||
- 高冲突 workload(同候选反复触发)→ 重做上限触发后 audit
|
||||
|
||||
---
|
||||
|
||||
## 10. D 健康检查(D1 / D10)
|
||||
|
||||
不属"巩固"主语义,但跟 consolidate 同节奏(周期 batch 顺手跑),归本文档:
|
||||
|
||||
| # | 信号 | 节奏 | 修复策略 |
|
||||
|---|---|---|---|
|
||||
| **D1** | 断链(wikilink → 不存在 path) | 写时 inline + weekly batch 巡检(双重保险)| 就地删 wikilink 或保留 alias 文本 → audit |
|
||||
| **D10** | provenance 断裂(digest 反指的 daily/resource 不可达)| 同上 | I-不变量违反 → 严重告警 + 人介入 |
|
||||
|
||||
D1 / D10 不算 5 大动作之一(它们不解决"vault 演化问题",只检测异常)。但它们的修复(就地删 wikilink)需要走 CAS,所以协议共享。
|
||||
|
||||
---
|
||||
|
||||
# Part D — 契约与实施
|
||||
|
||||
## 11. 维护 → 检索契约
|
||||
|
||||
5 大动作产物给 retrieve 消费(详细 retrieve 逻辑见 `auto_recall_design.md`):
|
||||
|
||||
| 产物 | 路径 | 写入者 | 读取者 | 缺失行为 |
|
||||
|---|---|---|---|---|
|
||||
| **vault 节点变化** | `digest/**.md` | merge / split / reinforce / abstract | recall(图遍历 / 命中) | — |
|
||||
| **communities** | `meta/communities.json` | community detection | recall + abstract / merge / reinforce | 不做同社区 boost / 三个动作跳过 |
|
||||
| **community changes** | `meta/community_changes.json` | community detection | abstract 决策 | abstract 跳过(无稳定度判据)|
|
||||
| **recency** | `meta/recency.json` | archive daily batch | recall | recency_factor = 1.0 |
|
||||
| **archived** | `meta/archived.json` | archive daily batch | recall(默认过滤)| 不过滤 |
|
||||
| **cooldowns** | `meta/cooldowns.json` | split / merge | consolidate 内部 | 无防御循环 |
|
||||
| **access_log** | `meta/access_log.json` | recall(写命中) + archive(聚合) | archive(读 recency) | recency 不衰减 |
|
||||
| **dups uncertain** | `audit/<date>/dups_uncertain.md` | merge | 人 / agent | — |
|
||||
| **consolidate actions** | `audit/<date>/consolidate_actions.md` | 全部 5 动作 | 审计 | — |
|
||||
| **D1 / D10 健康** | `audit/<date>/health_*.md` | inline check + weekly | 人 / agent | — |
|
||||
|
||||
**契约稳定性**:`meta/*.json` 都带 `version` + `computed_at`;recall 启动时校验 version,不兼容则降级。
|
||||
|
||||
---
|
||||
|
||||
## 12. 与 dream 模型的引用关系
|
||||
|
||||
本文档松绑了部分 F-invariants(§7),但仍在 dream 定义的底层模型上工作:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法 | `auto_dream_design.md` §3 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §4 / §2 / §3 |
|
||||
| F-invariants 原始定义 | `auto_dream_design.md` §4.3(本文档 §7 修订)|
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §4.4 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §2 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §3 |
|
||||
| provenance 载体形态 | `auto_dream_design.md` §4.2 |
|
||||
| dream 写入路径 | `auto_dream_design.md` §4.2 |
|
||||
|
||||
---
|
||||
|
||||
## 13. 下一步(M0 → M1.1 → M1.2 → M1.3 → M2)
|
||||
|
||||
实现进入 `reme4/steps/consolidate/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md`(顶层)/ `auto_recall_design.md` 共同作为契约依据。
|
||||
|
||||
### M0:基础设施 + 完全可逆动作
|
||||
|
||||
- ✅ split inline 触发 + LLM 离散度判 + E-2 守恒(基础部分)
|
||||
- ⏳ **community detection weekly batch**(Leiden via `igraph`)+ `meta/communities.json` + `meta/community_changes.json`
|
||||
- ⏳ **archive daily batch** + recency_score + access_log 收集
|
||||
- ⏳ CAS 写入框架 + version_stamp + EEXIST race + 重做上限 + audit
|
||||
- ⏳ D1 / D10 写时 inline 检测 + weekly 巡检
|
||||
|
||||
### M1.1:abstract(P1,additive 最低风险)
|
||||
|
||||
- ⏳ abstract 候选挖掘(community 大小 + 稳定度 + 活跃度 + 多样性 + 无 hub 五重判据)
|
||||
- ⏳ abstract LLM prompt(三段输出 + 长度限制 1500 token)
|
||||
- ⏳ grounding 守恒校验(出边 ⊇ 关键支柱 + 出边 ⊇ source 60%)
|
||||
- ⏳ "已有 hub" 结构化判据(outbound 覆盖度 ≥ 60%)
|
||||
- ⏳ **关键验证点**:实测 split parent 是否被识别为 hub
|
||||
|
||||
### M1.2:merge(P2,lossy 高门槛)
|
||||
|
||||
- ⏳ 候选挖掘(community 内 description 相似 + body 重合 + cooldown 检查)
|
||||
- ⏳ 多数票框架(N=3 LLM,2/3 通过)
|
||||
- ⏳ merge prompt(B 方案:"只合并不精化")
|
||||
- ⏳ inbound retarget 机械算子(扫所有 `[[loser.md]]` → `[[winner.md]]`,alias / predicate 保留)
|
||||
- ⏳ 事务式多文件 CAS 写入
|
||||
- ⏳ 灰色地带报告(`audit/<date>/dups_uncertain.md`)
|
||||
- ⏳ cooldown 框架(`meta/cooldowns.json` + 各动作互锁)
|
||||
|
||||
### M1.3:reinforce(P3,价值最低,可缓做)
|
||||
|
||||
- ⏳ 候选挖掘(三层过滤:mention + embedding + 同 community)
|
||||
- ⏳ 单票决策(门槛 0.85)
|
||||
- ⏳ additive wikilink 写入(alias 保留原文)
|
||||
|
||||
### M2+:演进
|
||||
|
||||
- ⏳ 多层级 community(L2 super-community)+ L2 abstract
|
||||
- ⏳ delete(永久删除 vault 文件)—— 视 dogfooding 效果决定是否开启
|
||||
- ⏳ predicate upgrade(typed link reinforce —— 当前 reinforce 只 additive 加无谓词)
|
||||
- ⏳ PageRank 替代 simple inbound count(若 retrieve 质量瓶颈在中心性)
|
||||
- ⏳ 跨进程并发(fs lock 支持多 reme 实例同 vault)
|
||||
- ⏳ Leiden 边权重(按 predicate 类型加权)
|
||||
|
|
@ -1,18 +1,16 @@
|
|||
# auto-dream 设计(digest 沉淀:物理 + 图)
|
||||
# auto-dream 设计(桶 / 节点 / 边 / 演化)
|
||||
|
||||
> 本文档记录 reme4 中 **auto-dream**(digest 沉淀知识层)的设计讨论 —— 含物理布局、图模型(节点 + 边)、入流端 digester(G\*)。
|
||||
> 本文档:digest 沉淀层的**桶**(物理布局)/ **节点**(原子单元)/ **边**(wikilink)/ **演化**(dream create_or_update;split 归 maintain)。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §1.2(数据视角)/ §2(三层存储)/ §3.5(digest 动作)/ §7(L4 模块)
|
||||
> - `auto_memory_design.md`:auto-memory(daily 实时事件)是 dream 的入流之一(G1 scope 拉取);auto-memory 写完即对 dream 可见
|
||||
> - `auto_maintain_design.md`:digest 的组织 / 重组 / 写入运行时(M split / D 检测 / CAS 写入协议) —— 本文档定义节点 + 边模型与 G\*,maintain 定义运行时
|
||||
> - `auto_link_design.md`:auto-link 是 dream 写完之后的后置增强(实体识别 + wikilink 写回);复用 maintain 的 CAS 协议
|
||||
> - `structure.md` §1.2(数据视角)/ §2(三层存储)/ §3.5(digest 动作)
|
||||
> - `auto_memory_design.md`:daily 实时事件 = dream 的入流之一
|
||||
> - `auto_consolidate_design.md`:M split / D 检测 / CAS 写入协议(dream 模型的运行时实现)
|
||||
> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想 —— dream 是其 Stage 1(写入阶段)的实现
|
||||
>
|
||||
> **四层对应**:reme 服务整体四份设计 —— auto-memory(daily 入流)/ **auto-dream(本文档:digest 沉淀 + content link + G\*)** / auto-maintain(digest 组织端:split + 检测 + CAS)/ auto-link(背景图关系增强)。`structure.md` §3.5-3.6 的 L4 action 视角:dream 对应 `digest` action(`resource + daily → digest`),maintain 对应 `maintain` action(`digest → digest`)。auto-dream 承担"空闲整理"(报告 §5.2):把 daily / resource 的散点抽出共性、合并重复、生成总结,落到 `digest/<bucket>/<slug>.md`。
|
||||
> **核心**:digest = **浅桶(shallow bucket)+ flat .md** + **一张图(节点 + 边)**;dream 定义模型与主流程(create_or_update),maintain 负责 split / 写入运行时。
|
||||
>
|
||||
> **核心立场**:dream 定义**模型层**(节点 + 边 / 守恒规则)+ **生成侧**(G\* create_or_update)+ **召回**(SearchStep);组织端(M split / D 检测 / CAS / 时序)归 `auto_maintain_design.md`;后置增强(实体识别 / wikilink 写回)归 `auto_link_design.md`。三方共享 §1.5 节点 + 边模型 + §1.5.5 边守恒 + §1.5.4 F-invariants。
|
||||
>
|
||||
> **关键收敛**:digest 不分"逻辑层"。整个 digest = **物理布局(浅桶 + flat .md)** + **一张图(节点 + 边)**。没有 hub / topic / leaf 之分 —— 所有 .md 文件都是同一种节点,内容决定它扮演什么角色(主题概览 / 概念定义 / 方法描述 / 实体记录 ...)。"主题"是从图中涌现的,不是结构性宣告的。
|
||||
> **关键收敛**:digest 不分"逻辑层"。所有 .md 文件都是同一种节点,内容决定它扮演什么角色(主题概览 / 概念定义 / 方法描述 / 实体记录 ...)。"主题"从图中涌现,不是结构性宣告。
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -23,556 +21,255 @@ digest 是 agent 长期记忆的"组织化沉淀"层,与三层架构的另两层
|
|||
| 层 | 组织主轴 | 形态 |
|
||||
|---|---|---|
|
||||
| resource/ | 时间(`<date>/<name>`) | 外部原始资料,不可变 |
|
||||
| daily/ | 时间 + 任务(`<date>/<slug>/`) | agent 任务过程,半可变 |
|
||||
| daily/ | 时间 + 任务(`<date>/<event-slug>/`) | agent 任务过程,半可变 |
|
||||
| **digest/** | **语义** | **跨任务知识,可重组** |
|
||||
|
||||
digest 的核心问题:
|
||||
- **物理布局**:文件系统怎么组织?(目录 / 文件 / 路径)
|
||||
- **节点与边**:概念粒度 / 链接形态 / 主题如何涌现?
|
||||
- **演化机制**:从碎片到网络的过程谁负责?(digester 创建或更新节点 / maintainer 拆分过载节点 / D 信号写后 inline 检测)
|
||||
dream 设计回答四个问题:**桶**怎么布局 / **节点**长什么样 / **边**怎么连 / **演化**谁负责怎么做。
|
||||
|
||||
---
|
||||
|
||||
## 1. 已对齐决策
|
||||
## 1. 桶(物理布局)
|
||||
|
||||
### 1.1 节点粒度:Atomic 节点优先
|
||||
|
||||
| 项 | 决策 |
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **粒度** | 一个文件 = 一个原子单元(概念 / 方法 / 实体 / 案例 / 原则 / 主题概览) |
|
||||
| **节点角色** | 由内容决定,不由 frontmatter 类型标记;一个节点扮演"主题概览"还是"具体方法",看它的 body 写了什么 |
|
||||
| **风格参考** | Zettelkasten:atomic note + 高密度 wikilink 网络 |
|
||||
|
||||
**理由**:
|
||||
1. **节点粒度 = retrieve 精度上限** —— semantic 检索召回 "一个原子单元" 远比召回 "一个 5000 字的主题文档" 信噪比高;agent 上下文窗口经不起粗粒度文档塞满
|
||||
2. **wikilink 在 atomic 粒度才真有意义** —— `[[jwt-rotation]]` 指向"一个具体方法"比指向"auth 主题文档"精确一个数量级,这也是 I-4(wikilink 唯一跨层载体)能撑起来的前提
|
||||
3. **物理目录纯做 navigate,图承担关系** —— 两套机制各司其职,不互相绑架
|
||||
|
||||
**Tradeoff**:
|
||||
- 文件数量爆炸(一个领域几百节点)→ 浅桶物理归档 + 路径作 ID,wikilink 走完整路径写法
|
||||
- 节点会频繁演化(新材料 update 已有节点 / 节点过载触发 split)→ G\* 去重质量、SearchStep 召回、写后 inline 检测都得到位
|
||||
|
||||
### 1.2 物理几何:浅桶(shallow bucket,**固定集合**)
|
||||
|
||||
| 项 | 决策 |
|
||||
|---|---|
|
||||
| **物理布局** | `digest/<bucket>/<slug>.md`,bucket 一层(顶多两层),桶内 flat |
|
||||
| **bucket 角色** | **仅承担物理归档与 OS-level 浏览锚点**;不承担语义本体角色 —— 主题由图中的节点表达 |
|
||||
| **bucket 内** | 不再分子目录,所有节点平铺 |
|
||||
| **bucket 集合** | **固定预定义,不由 digester / maintainer 动态生成** |
|
||||
| **bucket 主页节点** | **不强制存在**;若 split 在该 bucket 内累积出层级(parent 节点天然中心性高),parent 节点天然成为浏览主页(纯约定,非架构必需) |
|
||||
| **新节点归属** | digester (G4) 只能从已有桶集合里**挑选**;LLM 不能造新桶 |
|
||||
| **集合来源** | vault 配置(opinionated default + 消费层可改),与 schema/prompt 同属服务消费层 |
|
||||
|
||||
**理由**:
|
||||
- 物理浏览有"主题轮廓"(打开 `digest/auth/` 能看到这一族节点),不像纯 flat 那样毫无锚点
|
||||
- 节点不被深路径绑死("属 auth/jwt 还是 auth/session"这种归属焦虑被消解 —— 一个节点可以同时被多个主题通过 wikilink 引用)
|
||||
- maintain 操作面坍缩到只剩 split:节点过载就拆,不做跨节点重组(详 §1.5 / §2)
|
||||
- **固定集合的关键意义**:
|
||||
- LLM 在 G4 桶决定时只做"分类",不做"造类" —— 决策面坍缩,错率大幅下降
|
||||
- bucket 集合作为**预先约定的物理归档规则**,跨任务跨时间稳定;不会出现 "auth-stuff" / "auth" / "authentication" 三个语义重叠的桶共存
|
||||
- 与 reme 核心立场一致:bucket 集合是消费层契约,reme 不自主造桶
|
||||
- **bucket 主页节点不强制的关键意义**:
|
||||
- 旧设计中 root hub 是架构必需(检测稳定锚点 / vault 总览结构性入口);新设计中这些角色都由图中心性自然承担,不需要为每个 bucket 强制创建一个空架子节点
|
||||
- 主页节点的"主页"地位是涌现的:某节点入度高 / 中心性高 → 它就是浏览入口
|
||||
- 若一个 bucket 完全没节点,它就只是个空目录;不需要先造一个 placeholder
|
||||
|
||||
**未归类节点**:digester 抽到一个原子单元但找不到合适的专属桶时,**不允许造新桶**;统一落入兜底桶 `digest/general/`。general 桶在固定集合内是一等公民,详见 §3.7。
|
||||
|
||||
### 1.3 节点身份:路径即 ID
|
||||
|
||||
| 项 | 决策 |
|
||||
|---|---|
|
||||
| **ID 载体** | **vault-relative 路径**(含 `.md`)即节点身份 —— `digest/auth/jwt-rotation.md` |
|
||||
| **wikilink 写法** | `[[digest/auth/jwt-rotation.md]]`(literal,与 `wikilink_handler.py` 默认形态对齐;不隐含 `.md`,无 short-form 补全) |
|
||||
| **`name` frontmatter** | 文件名 basename(不含扩展名),与文件名同步 —— 检索 hint / 人读标签,**不当 ID 用** |
|
||||
| **同名冲突** | 同 bucket 内文件名冲突 → 文件系统层断言;**不需要独立 D6 信号** |
|
||||
| **rename 成本** | 一次 `wikilink_handler.retarget_links(old_path, new_path)`,机制现成 |
|
||||
| **跨桶移动** | F-1 已禁止;若必须做(人工介入修错桶),走一次 retarget |
|
||||
|
||||
**理由**:
|
||||
- F-1(0 文件移动)+ 平铺 + 下层 immutable 后,slug abstraction 的核心价值(移动鲁棒性)蒸发;只剩下"wikilink 短形式"这一项收益,但代价是 file_graph slug 索引 + D6 冲突检测 + alias 表 + retrieve 透明展开,**净亏**
|
||||
- `wikilink_handler.py` docstring 自己写的就是 *Recommended form: full path relative to the vault with extension* —— literal 匹配,无 short-link 补全;路径作 ID 与核心库默认完全对齐
|
||||
- 完整路径前缀 `digest/auth/` 给 LLM 读写时提供语义 context(知道节点在哪个桶),不全是负担
|
||||
- provenance wikilink 反指 daily/resource 本来就用路径,统一后整个 vault 一种 wikilink 形态,不必区分"slug 形态 vs 路径形态"
|
||||
|
||||
### 1.4 链接语义:基础 wikilink + 可选 Dataview 谓词
|
||||
|
||||
参考实现:`reme4/utils/wikilink_handler.py` + `reme4/schema/file_link.py`。
|
||||
|
||||
| 项 | 决策 |
|
||||
|---|---|
|
||||
| **link 基础形态** | `[[<vault-relative-path>.md]]` —— target 字面取(literal,不隐含 `.md`,不自动短链补全);路径即 ID(详 §1.3) |
|
||||
| **alias / image** | `[[path.md\|alias]]`(显示文本)/ `![[image.png]]`(图片资源)—— rewrite 时 alias 保持 |
|
||||
| **anchor 不引入** | digest 设计层**不使用** `[[path.md#section]]` —— atomic 节点 + child 边界已充当精度替代品(详 §1.3 / §3.13);`wikilink_handler` 仍可解析 anchor 字面(供其它消费层),但 digest 不生成、不依赖、不在 split 时迁移 anchor |
|
||||
| **可选谓词(Dataview 风格)** | 行级: `predicate:: [[path.md]]` / 内联: `[predicate:: [[path.md]]]`;**谓词写在 `[[]]` 外**,不是 `[[predicate::path]]` |
|
||||
| **谓词标识符** | `[A-Za-z][A-Za-z0-9_]*`(如 `is_a` / `extends` / `causes` / `references`);词表**开放**,任意标识符 |
|
||||
| **未类型化合法** | 绝大多数 wikilink **不加** predicate;`predicate=None` 是默认 / 常态 |
|
||||
| **edge 唯一性键** | `(target_path, predicate)`(二元组);同源同标但不同 predicate = 不同边。`FileLink.target_anchor` 字段在 schema 中保留(供其它消费层),digest 层永远写 `None` |
|
||||
| **类型信息载体** | 节点 frontmatter `kind` + 边 `predicate`(双轨可选);二者都是**消费层 schema 提示**,reme 核心解析 / 存储 / 索引,但**不读它们做结构决策** |
|
||||
| **plugin 层扩展** | transclusion / 引用图谱视图等留给消费层加,reme 核心不固化语义 |
|
||||
|
||||
**理由**:
|
||||
- 与 I-4 "wikilink 是唯一跨层载体" 对齐 —— reme 核心永远只看机械拓扑
|
||||
- 与 [[reme4_schema_layering]] 一致 —— 类型语义(无论是节点 `kind` 还是边 `predicate`)都是消费层契约,reme 核心不固化
|
||||
- predicate 走 Dataview 而非内嵌:`[[]]` 内容保持"纯目标"(rewrite / retarget 不必感知 predicate);predicate 是文本上的**装饰位**,与 wikilink 解耦
|
||||
- `kind` 与 predicate 严格只是内容标签:reme 核心**只有节点这一种结构类型 + 边这一种结构关系**,kind / predicate 永远不参与"hub / topic / leaf"这类结构角色判断
|
||||
|
||||
**reme 核心对 predicate 的"透明"边界**(关键):
|
||||
- G7(横向 link)、retrieve 中心性 —— 都**聚合所有 predicate** 算,不分桶
|
||||
- 只有 edge 唯一性 / 反向索引会用到 predicate(否则 `[[A]]` 和 `is_a:: [[A]]` 会被当作同一条边互相覆盖)
|
||||
- 消费层若要按 predicate 做更精细的推理(如"taxonomic 路径只走 `is_a` 边"),自己读 `FileLink.predicate` 即可
|
||||
|
||||
### 1.5 图模型与节点演化
|
||||
|
||||
**核心模型**:digest = **物理布局(浅桶 + flat .md)** + **一张图(节点 + 边)**。
|
||||
|
||||
| 维度 | 形态 |
|
||||
|---|---|
|
||||
| **节点** | 每个 .md 文件 = 一个节点;无结构性 kind,角色由 body 内容决定(主题概览 / 概念定义 / 方法描述 / 实体记录 / 案例 ...) |
|
||||
| **边** | 基础 `[[<vault-path>.md]]` wikilink(路径即 target,详 §1.3 / §1.4);可选 Dataview 谓词 `predicate:: [[path.md]]` / `[predicate:: [[path.md]]]` 写在 `[[]]` 外;边唯一性键 = `(target, predicate)`;**digest 层不引入 anchor**(详 §1.4 / §3.13);**reme 核心结构决策不读 predicate**(详 §1.4) |
|
||||
| **多归属** | 一个节点可被多个其它节点引用,也可指向多个其它节点;**不存在"单父"约束** |
|
||||
|
||||
**演化只做两件事**:
|
||||
1. **G\* create_or_update**:新材料进入,LLM 提取原子单元 → 命中已有节点就 update 该节点 body(语义守恒地融合新旧),否则新建节点
|
||||
2. **M split**:节点累积过载(token / 主题离散度超阈值)→ LLM 把它拆成 parent overview + N 个 children,parent 文件原地保留作 overview,children 是新文件
|
||||
|
||||
"主题概览节点 / 摘要节点"不是一种 kind,也不是 maintainer 主动涌现的产物 —— 它是 split 的副产品(parent 节点天然成为该 cluster 的 overview)。
|
||||
|
||||
#### 1.5.1 单一节点 / 单一边
|
||||
|
||||
**节点 frontmatter** —— 只有保留字段:
|
||||
|
||||
| 字段 | 内容 | 用途 |
|
||||
|---|---|---|
|
||||
| `name` | 文件名 basename(不含扩展名),与文件名同步 | 检索 hint / 人读标签(I-4 不再用它做身份;路径才是 ID,详 §1.3) |
|
||||
| `description` | 一句话 | 标题 / 检索 hint |
|
||||
| (可选)`kind` | concept / method / case / entity / topic / ... | **消费层 schema 提示**,reme 核心透明,不读它做结构决策 |
|
||||
| body | 任意内容 | 一句定义 / 一段方法 / 一篇主题概览 / 一份案例,皆可 |
|
||||
|
||||
`hub__` / `topic__` 前缀**不存在**;文件名自然命名(`auth-fundamentals.md` / `jwt-rotation.md` / `jwt-overview.md`)。"overview 节点"靠内容形态识别,不靠前缀。
|
||||
|
||||
**边的形态** —— 与 §1.4 一致,这里给最小汇总:
|
||||
|
||||
| 维度 | 形态 |
|
||||
|---|---|
|
||||
| 基础 | `[[<vault-path>.md]]`(无谓词;常态;`predicate=None`) |
|
||||
| 可选谓词 | `predicate:: [[path.md]]`(行级)/ `[predicate:: [[path.md]]]`(内联);谓词在 `[[]]` 外 |
|
||||
| alias | `[[path.md|display-text]]`(rewrite 时 alias 保持) |
|
||||
| image | `![[image.png]]`(资源引用,不是知识边) |
|
||||
| **不引入 anchor** | digest 层不使用 `#section`;详 §1.4 / §3.13 |
|
||||
| **边唯一性键** | `(target_path, predicate)` —— 同源同标不同 predicate = 不同边 |
|
||||
| **角色识别** | 默认无谓词时由端点内容形态推断;有谓词时谓词即角色标签(消费层语义,核心不读) |
|
||||
|
||||
> **关键收敛**:reme 核心**只有节点 + 边两种结构类型**;kind / predicate 都是内容标签,绝不参与 hub / topic / leaf 这类结构角色。
|
||||
|
||||
#### 1.5.2 节点演化:create_or_update + split
|
||||
|
||||
```
|
||||
[入流] material 进入(daily / resource)
|
||||
│
|
||||
▼
|
||||
G1 scope:选哪些 daily/resource 进入本轮
|
||||
│
|
||||
▼
|
||||
G2 提取原子单元(LLM,可能产 N 个候选)
|
||||
│
|
||||
▼
|
||||
对每个候选:
|
||||
│
|
||||
▼
|
||||
G* create_or_update(LLM 决策点)
|
||||
├─ 语义相似查 → 拉相似候选节点(top-k)
|
||||
├─ LLM 判:候选中有"同概念节点"吗?
|
||||
│ ├─ 有 → update 路径
|
||||
│ │ (a) 把新内容融入已有 body(语义守恒重写)
|
||||
│ │ (b) 加 provenance 反指
|
||||
│ │ (c) 必要时加 / 改 wikilink
|
||||
│ └─ 无 → create 路径
|
||||
│ G3 路径 / G4 bucket / G6 provenance / G7 横向 link / 写 body
|
||||
│
|
||||
▼
|
||||
写入(机械)
|
||||
|
||||
[D3 检测] 写后立即:G\* / split 写完 body 顺手 inline 检测(token 阈值 → 超阈值则 LLM 判离散度;详 `auto_maintain_design.md` §4)
|
||||
│
|
||||
▼
|
||||
D3 派发候选节点(F-4 一次一个)
|
||||
│
|
||||
▼
|
||||
M split(LLM + 机械)
|
||||
├─ LLM 把节点 body 拆成 N 个 cluster(每个是个原子单元)
|
||||
├─ parent 文件原地保留 → body 重写为 overview + 列出 children wikilinks
|
||||
├─ 每个 child 创建新文件(文件名 / bucket / body 由 LLM 给)
|
||||
├─ children 各自加 [[<parent-path>.md]] 反向链接
|
||||
├─ 边守恒机械校验:`(parent_new ∪ ∪children) ⊇ parent_old` 出边集合(F-11 / E-2)
|
||||
└─ inbound 链 `[[<parent-path>.md]]` 不动(F-10 / E-3) —— digest 层无 anchor 链,无需 retarget
|
||||
```
|
||||
|
||||
**关键**:
|
||||
- **G\* update 改 subject body(语义守恒重写)** —— 新材料融入已有节点正文,要求 LLM 守住"只增不删 / 不改原意",老内容不能丢;**写入前机械校验出边强守恒**(`new outbound ⊇ old outbound`,详 §1.5.5 E-1)
|
||||
- **G\* 不改其它节点正文** —— 只动 subject;不像旧 M-E 会到邻居 body append wikilink
|
||||
- **M split 不改其它节点正文** —— 只动 parent(重写为 overview)+ 新建 children
|
||||
- **inbound 在 split 时一律不动** —— digest 设计不引入 anchor,inbound 全是裸链 `[[<parent-path>.md]]`,parent 路径未变即天然有效;后续 G\* 进入若 LLM 觉得 child 粒度更合适,直接加新边到 child(F-10)
|
||||
|
||||
#### 1.5.3 走一个具体例子
|
||||
|
||||
**场景**:`digest/auth/` 桶,初始只有几个零散 auth 节点,没有 jwt-rotation。
|
||||
|
||||
**第 1 轮 G\***:某 daily 提到 "JWT rotation:每 24 小时换密钥,旧密钥保留 1 小时窗口给未过期 token"。
|
||||
- 语义查 → 没找到 jwt-rotation 节点
|
||||
- 走 create 路径 → 新建 `jwt-rotation.md`,body = 一段 200 字的 rotation 描述
|
||||
|
||||
**第 2 轮 G\***:另一个 daily 提到 "JWT rotation 的 grace period 通常是 1-2 小时"。
|
||||
- 语义查 → 命中 `jwt-rotation`(高相似)
|
||||
- LLM 判:这是同概念,走 update 路径
|
||||
- 把 grace period 信息**融入** `jwt-rotation.md` body(不只是 append):
|
||||
```
|
||||
Before: "...旧密钥保留 1 小时窗口..."
|
||||
After: "...旧密钥保留 1-2 小时 grace period(典型值,具体看 token 寿命)..."
|
||||
```
|
||||
- body 略增长,加一条 provenance 反指
|
||||
|
||||
**第 N 轮 G\***:经过几个月,各种 daily 持续 update `jwt-rotation` —— 加了密钥派生算法、加了 RS256/HS256 区别、加了 key rotation 失败处理、加了 with-leeway 实践、加了 monitoring 建议 ...
|
||||
|
||||
`jwt-rotation.md` body 现在 ~3500 token,涵盖:轮换策略 / 密钥派生 / 算法选择 / 失败处理 / 监控。
|
||||
|
||||
**触发**:D3 检测 token > 2000 阈值 → 派发候选。
|
||||
|
||||
**M split**:LLM 拉 `jwt-rotation.md` body + frontmatter,判断主题离散度(5 个相对独立的子主题),决定拆:
|
||||
- parent: `jwt-rotation`(留下,body 重写为 overview)
|
||||
- children: `jwt-key-derivation` / `jwt-algorithm-selection` / `jwt-rotation-failure-handling` / `jwt-rotation-monitoring`(4 个新文件)
|
||||
- "with-leeway 实践"内容并入 parent overview(粒度太细不单独拆)
|
||||
|
||||
**执行后**:
|
||||
|
||||
```
|
||||
digest/auth/
|
||||
├── jwt-rotation.md ← 文件原地;body 重写为 overview
|
||||
│ description: JWT 轮换策略总览
|
||||
│ body: JWT 轮换的核心是 X,主要环节包括:
|
||||
│ 密钥派生 [[digest/auth/jwt-key-derivation.md]]
|
||||
│ 算法选择 [[digest/auth/jwt-algorithm-selection.md]]
|
||||
│ 失败处理 [[digest/auth/jwt-rotation-failure-handling.md]]
|
||||
│ 监控告警 [[digest/auth/jwt-rotation-monitoring.md]]
|
||||
│ (参考:with-leeway 实践 ...)
|
||||
├── jwt-key-derivation.md ← 新建,body 来自原 jwt-rotation 拆出片段
|
||||
│ → [[digest/auth/jwt-rotation.md]] ← child 反指 parent
|
||||
├── jwt-algorithm-selection.md ← 同上
|
||||
│ → [[digest/auth/jwt-rotation.md]]
|
||||
├── jwt-rotation-failure-handling.md ← 同上
|
||||
│ → [[digest/auth/jwt-rotation.md]]
|
||||
├── jwt-rotation-monitoring.md ← 同上
|
||||
│ → [[digest/auth/jwt-rotation.md]]
|
||||
└── ...(其它 auth 节点不变)
|
||||
```
|
||||
|
||||
**inbound 不动**:之前指 `jwt-rotation.md` 的所有外部 wikilink(无论裸链还是 typed)都仍然指 `[[digest/auth/jwt-rotation.md]]`。如果后续某外部节点写新材料时 LLM 觉得 child 粒度更合适,直接 G\* 时新加 `[[digest/auth/jwt-key-derivation.md]]` 这种边即可 —— 不强求 split 时即时重定向。
|
||||
|
||||
**继续演化**:
|
||||
- 若某个 child(如 `jwt-key-derivation`)被持续 update,某天也长到过载 → D3 又触发 → 它再次 split,自然涌现第三层
|
||||
- 若某 child 长期空 / 0 入度 / 0 update —— 不主动删(没有 dissolve 操作);除非人工介入
|
||||
|
||||
#### 1.5.4 操作的核心约束
|
||||
|
||||
| # | 约束 | 含义 |
|
||||
|---|---|---|
|
||||
| **F-1** | **0 文件移动** | G\* / split 都不移动现有文件;split 创建的是**新文件**,parent 文件原地 |
|
||||
| **F-2** | **改正文限定 subject** | G\* update 改 subject node body(语义守恒重写,不改其它节点);M split 改 parent body(重写为 overview)+ 创建 children body;**没有任何操作改"其它节点正文"** |
|
||||
| **F-3** | **maintainer 只做 split** | 没有 summarize / merge / re-edge / link / unify / dissolve;过载 → 拆 |
|
||||
| **F-4** | **一次一个候选** | M split 一次拆一个节点;G\* 一次处理一个原子单元(N 个候选 = N 次 G\*) |
|
||||
| **F-5** | **不确定时不动** | G\* 拿不准是 create 还是 update → 倾向 create(不污染已有节点);split 拿不准 cluster 边界 → 不拆 |
|
||||
| **F-7** | **多归属合法** | 一个节点可被多个其它节点引用,也可指向多个其它节点;**没有"单父"约束** |
|
||||
| **F-10** | **inbound 目标节点不动** | 所有 inbound 都是裸链 `[[<parent-path>.md]]`(digest 不引入 anchor,详 §1.4 / §3.13);split 时全部保持不动,parent 路径未变即天然有效;后续 G\* 进入时 LLM 可自由选择更精细 target(直接加新边到 child) |
|
||||
| **F-11** | **wikilink 是 body 的一部分** | 不存在"独立的边" —— 边的所有迁移都是 body 文本变化的副作用;reme 核心机械算子只感知字符层,语义责任在 LLM(G\* / split prompt)+ 守恒校验(outbound diff 机械验证;详 §1.5.5) |
|
||||
|
||||
#### 1.5.5 边的迁移规则
|
||||
|
||||
**前提**:wikilink 是 body 的一部分(F-11)。"边"不是独立抽象 —— body 一变,边就跟着变。reme 核心**没有"修边"算子**,边的所有变化都是 body 文本编辑的副作用。
|
||||
|
||||
但语义守恒不能放任 LLM:守恒责任在 prompt + 机械校验,不在算子。
|
||||
|
||||
**3 类边按"在哪类操作中变化"区分**:
|
||||
|
||||
| # | 类别 | 规则 | 谁负责 |
|
||||
|---|---|---|---|
|
||||
| **E-1** | **G\* update 节点出边**(subject 自身) | **强守恒**:新 body 出边集合 ⊇ 原 body 出边集合(`(target, predicate)` 二元组比对,**predicate 一并守住**);不满足 → LLM 重试或拒写 | LLM(prompt 强约束)+ 机械校验(outbound diff) |
|
||||
| **E-2** | **split parent 出边**(parent body 拆解) | parent overview + N 个 children 各持一段,原 parent 出边按内容自然分配到 parent overview + children;**机械校验合计守恒**:`(parent_new ∪ ∪children_outbound) ⊇ parent_old` | LLM(split prompt)+ 机械校验 |
|
||||
| **E-3** | **inbound wikilink** `[[<parent-path>.md]]` | split 时**不动** —— 仍指 parent;后续 G\* 进入若 LLM 觉得 child 粒度更合适,直接加新边到 child(F-10) | 不动 |
|
||||
|
||||
**provenance 不单列一类**:节点反指上游 daily/resource 的 wikilink 是 body 正文的一部分(§3.9),由 LLM 在 G\* / split prompt 中自然写出 —— 跟其它 body wikilink 走同一套规则:G\* update 走 E-1 强守恒(老 provenance 链不能丢,新材料追加新 provenance),split 走 E-2 合计守恒(parent 全量 provenance ⊆ parent_overview ∪ ∪children_outbound)。reme 核心**没有** provenance 专用算子。
|
||||
|
||||
**inbound anchor 这一类不存在**:digest 设计层不引入 anchor(详 §1.4 / §3.13),所有 inbound 都是裸链,走 E-3 即可,无需机械 retarget 子流程。
|
||||
|
||||
> **关键拆分**:
|
||||
> - **守恒**(E-1 / E-2):LLM 写正文时不能丢边;靠 prompt + 写后 outbound diff 校验
|
||||
> - **保守**(E-3):没有信号说一定要变;不变的代价 = 后续 G\* 自然纠正,变的代价 = 错信号大量假阳;选不变
|
||||
|
||||
**机械 outbound diff 校验**(E-1 / E-2)伪码:
|
||||
|
||||
```
|
||||
write_subject_body(subject, new_body):
|
||||
old_outbound = extract_links(old_body) # set of (target, predicate)
|
||||
new_outbound = extract_links(new_body)
|
||||
missing = old_outbound - new_outbound
|
||||
if missing:
|
||||
# LLM 漏了原边 —— 重试一次
|
||||
new_body = llm_retry_with_missing(missing)
|
||||
new_outbound = extract_links(new_body)
|
||||
if old_outbound - new_outbound:
|
||||
raise ConservationViolation(...) # 拒写,记 audit,等人介入
|
||||
write(subject, new_body)
|
||||
```
|
||||
|
||||
机械层只做集合比对,**不判断"为什么丢了"** —— 那是 LLM 的事。
|
||||
|
||||
**强守恒(集合包含)而非等价**:`new ⊇ old` 是"新材料融入,老知识保留"的最小契约 —— 允许加新边(新关联),不允许减边(老内容不能丢);等价(`new == old`)会拒绝任何新出边,update 失去意义。
|
||||
|
||||
**predicate 守住** —— `[[A]]` ↔ `is_a:: [[A]]` 视为不同 key,升降级走显式 audit 路径,不走默认。重排 / 改 alias / 加新边都不被拦下(集合相同或只增)。
|
||||
|
||||
#### 1.5.6 图模型 vs 建子目录
|
||||
|
||||
| 维度 | 建子目录(深树) | 图模型 + split 演化 |
|
||||
|---|---|---|
|
||||
| 物理变化 | 移动文件,改路径 | 0 文件移动(F-1);split 只创建新文件 |
|
||||
| wikilink 影响 | 路径 ID 模型下要全图 retarget(代价大) | 0 影响(parent 路径未动);split 不触发任何 retarget |
|
||||
| 主题归属 | 一个节点只能属一棵子树 | 一个节点可同时属多个主题(被多源 wikilink) |
|
||||
| 撤销成本 | 移回文件 + 重建上下文 | 删 children + parent body 还原(手工) |
|
||||
| 演化路径 | 子树重组痛苦 | parent 只增不减,children 是 parent 拆出的快照 |
|
||||
| navigate | 浏览目录树 | 任意节点入手沿出边漫游;parent 节点是天然中心 |
|
||||
| retrieve 精度 | 路径反映主题但与 link 无关 | 节点中心性 + 内容形态共同决定权重 |
|
||||
|
||||
#### 1.5.7 多级结构自然涌现
|
||||
|
||||
split 是节点的**局部操作**(只看一个过载节点),多级深度自然涌现:
|
||||
|
||||
```
|
||||
digest/auth/
|
||||
├── auth-fundamentals.md ← 早期写下,~1500 token,稳定
|
||||
├── jwt-rotation.md ← 第一次 split:body 从 3500 token 重写为 overview
|
||||
├── jwt-key-derivation.md ← 第一次 split 的 child
|
||||
├── jwt-key-derivation-hkdf.md ← 二次 split:jwt-key-derivation 累积 update 后过载,再拆
|
||||
├── jwt-key-derivation-pbkdf2.md ← 二次 split 的 child
|
||||
├── ...
|
||||
```
|
||||
|
||||
**物理仍是浅桶(1 层),"层级"由 split 链 + 节点中心性自然承载**。每一级的过载条件、决策机制、执行步骤完全相同 —— 没有"二级 split"特殊逻辑,只有"过载节点的 body 可以被 split 进一步拆"。
|
||||
|
||||
#### 1.5.8 retrieve 时节点怎么参与
|
||||
|
||||
| query | 期望返回 |
|
||||
|---|---|
|
||||
| "JWT 怎么轮换" | 优先 `jwt-rotation`(具体 overview)+ children(如 `jwt-key-derivation`) |
|
||||
| "auth 体系" | 优先中心性高的节点(`auth-fundamentals` / `jwt-rotation` 等被多次 update / 是 split parent 的节点) |
|
||||
| "auth 有什么子主题" | 沿高中心性节点的入/出邻居遍历;parent 节点优先返回 |
|
||||
| "vault 里都有什么" | 各 bucket 中心性最高的节点(自然形成 vault 总览) |
|
||||
|
||||
**加权策略**(opinionated default,消费层可改):
|
||||
- 节点权重 = base(=1.0) × intent 调节 × 中心性增益
|
||||
- query 含"概览 / 主题 / 入门 / 全景"等**元意图**时,中心性高的节点加权(intent 调节 > 1)
|
||||
- 中心性低 / body 短的具体节点权重稳定(默认 1.0,不被压低)
|
||||
|
||||
**topological traverse**:
|
||||
- 沿 wikilink 自由走(不区分边类型 / predicate)
|
||||
- 经过中心性高的节点默认**不强行展开**(否则一次 traverse 把整族 children 拉进来);agent 可显式深入
|
||||
|
||||
**中心性的天然来源 = split parent**:被拆过的节点是 parent,自然有 children 反向链接它,中心性自然高 —— 不需要单独维护 `kind: hub` 标记。
|
||||
|
||||
---
|
||||
|
||||
## 2. 完整能力集
|
||||
|
||||
### 2.0 设计目标
|
||||
|
||||
**让图的形状持续匹配实际知识的语义结构,在最小变更面 + 渐进演化的前提下,使任意尺度的知识访问都能命中合适粒度的节点。**
|
||||
|
||||
这个目标直接来自结构本身的设计意图 —— 浅桶 + 单一节点类型 + 单一边类型 + create_or_update + split 的组合,每一项都是为它服务。能力集的入选标准:**对至少一个验证维度有贡献**。
|
||||
|
||||
| 维度 | 含义 | 失败示例 |
|
||||
|---|---|---|
|
||||
| **形状匹配** | 节点中心性 / 边连接 / 节点邻域反映知识间的实际语义关系 | 一个节点 token 5000+ 长期不拆;同主题节点彼此 0 链接;同概念被建成多个独立节点 |
|
||||
| **最小变更面** | 不重写其它节点正文,不大规模移文件,不破坏现有 wikilink | 任何"全图重组"或"批量改其它节点正文"的方案 |
|
||||
| **任意尺度访问** | 具体方法节点 / overview 节点 / 节点邻居遍历都能命中 | 全 flat,主题级 query 命中不到东西 |
|
||||
|
||||
**显式排除**(不在目标内,避免能力集内卷):
|
||||
- ❌ "完美归簇" —— F-5 留白,不确定就不动
|
||||
- ❌ "实时一致" —— 异步 / eventual,节点写完不必立刻 split
|
||||
- ❌ "零冲突 / 零违反" —— invariants 检测 + 事后修复,不追求永不发生
|
||||
- ❌ 替消费层做检索 / 决策 —— digest 自治边界止于"维持图的形状"
|
||||
- ❌ 跨节点重组(merge / re-edge / unify / dissolve)—— 简化模型不做这些;同概念二次进入由 G\* update 路径处理
|
||||
|
||||
**演化只有两件事**:G\* create_or_update(入流型,新材料融入)+ M split(后台,过载就拆)。detection 派生信号驱动这套循环。
|
||||
|
||||
### 2.1 生成侧:digester(入流型)
|
||||
|
||||
| # | 能力 | 服务 | 性质 | 何时发生 |
|
||||
|---|---|---|---|---|
|
||||
| **G1** | **scope 决定**:选哪组 daily/resource 进入本轮蒸馏 | 形状匹配(决定形状从哪生长) | LLM | digester 启动 |
|
||||
| **G2** | **原子单元抽取**:从 scope 中识别值得沉淀的原子单元(N 个候选) | 形状匹配 + 任意尺度 | LLM | 核心环节 |
|
||||
| **G\*** | **create_or_update**:对每个候选,**多路召回(SearchStep:vector + keyword + 邻接展开,RRF 融合,scope 限 `digest/`)** → LLM 看完整候选池 → 终判 create / update / drop;create 路径走 G3/G4/G6/G7 + 写 body;update 路径融入已有 body(语义守恒重写)+ 自然追加 provenance(详 §3.10) | 形状匹配(去重内置)+ 最小变更面 | LLM(决策)+ 机械(召回 + 守恒写入) | 每个候选 |
|
||||
| **G3** | **路径命名**(create 路径):在 G4 选定 bucket 内,文件名同 bucket 唯一(fs 层断言);风格与同主题节点一致 | 任意尺度(可寻址) | LLM(命名)+ 机械(同 bucket 文件名冲突 → 拒写) | create 时 |
|
||||
| **G4** | **bucket 落地**(create 路径):从固定集合中挑选;找不到合适专属桶 → 落 `general/`(§3.7) | 形状匹配(物理归档) | LLM(读 bucket 列表) | create 时 |
|
||||
| **G6** | **provenance 写入**(create 与 update):新节点 body 内联反指上游 daily/resource 的 wikilink;update 时 LLM 在融入新材料时自然追加新 provenance 链,旧 provenance 链由 E-1 守恒校验保住(§3.9) | 任意尺度(跨层访问) | LLM(prompt 引导写出 `[[daily/...]]` / `[[resource/...]]`)+ 机械(outbound diff 校验) | 写入时 |
|
||||
| **G7** | **横向 link**(create 时):新节点链到相关的已有节点(出边);update 时也可加新 link | 形状匹配 + 任意尺度 | LLM | 写入时 |
|
||||
|
||||
**关键边界**:
|
||||
- **G\* 是入流唯一改 body 的操作**,且**只改 subject node** —— update 改的是同概念那个节点自己,不改其它节点
|
||||
- **G\* update 必须语义守恒**:LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意)
|
||||
- **0 出边节点合法**(G7 没识别到合适邻居),后续 G\* 进入时其它节点可以反向链回来 —— 不强求 LLM 一次性给全
|
||||
- **G\* 漏判去重**(把同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显的同概念重复,可由 auto-link 离线 audit 工具产报告(详 `auto_link_design.md` §1.3 L4)
|
||||
- **digester 不做 split** —— split 是后台 M 操作
|
||||
|
||||
### 2.2 组织侧 / 检测 / 写入并发 → `auto_maintain_design.md`
|
||||
|
||||
M split / D 检测信号(D1 / D3 / D10)/ 阈值校准 / D3 写后触发模型 / G\* / split / auto-link L1 三方共用的 CAS 写入协议 / split provenance / 时序 / 后门 —— 全部归 `auto_maintain_design.md`。
|
||||
|
||||
dream 保留**模型层**(§1 节点 + 边 + 守恒规则)+ **生成侧**(§2.1 G\*)+ **召回**(§3.10 SearchStep);maintain 负责**组织 / 运行时**(split + D + CAS + 时序)。两份文档共享 §1.5 节点 + 边模型、§1.5.5 边守恒、§1.5.4 F-invariants。
|
||||
|
||||
| 在 maintain 文档中 | 内容 |
|
||||
|---|---|
|
||||
| §1 | M split 能力卡 + 关键边界 |
|
||||
| §2 | 检测信号 D1 / D3 / D10 |
|
||||
| §3 | 阈值校准 |
|
||||
| §4 | D3 写后触发模型 |
|
||||
| §5 | CAS 写入协议(三方共用) |
|
||||
| §6 | split 时 provenance |
|
||||
| §7 | G\* / split / auto-link L1 时序 |
|
||||
| §8 | 后门(暂缓) |
|
||||
|
||||
### 2.4 边界协议(谁不能做什么)
|
||||
|
||||
| 边界 | 内容 | 来源 |
|
||||
|---|---|---|
|
||||
| digester ∩ maintainer | digester 不做 split;maintainer 不做原子单元抽取 / 新具体节点 create | 入流 vs 自维护职责分离 |
|
||||
| digester → 其它节点 | G\* update 改 subject node body,**不改其它任何节点正文** | F-2 |
|
||||
| digester → "摘要 / overview" | digester 不为做 overview 而创建节点;它产的节点都是具体原子单元;overview 是后续 split 的副产品 | F-3 |
|
||||
| maintainer → 其它节点 | M split 改 parent body(重写为 overview)+ 创建 N 个 children body;**不改任何其它节点** | F-2 |
|
||||
| maintainer → inbound 链 | split 时**全部不动** —— digest 不引入 anchor,inbound 一律是裸链 `[[<parent-path>.md]]`,parent 路径未变 | F-10 / E-3 |
|
||||
| digester → 边守恒 | G\* update 写新 body 前,机械对比 old/new outbound:`new ⊇ old`((target, predicate) 二元组);失败 → LLM 重试一次,再失败拒写 | F-11 / E-1 |
|
||||
| maintainer → 边守恒 | split 写新 parent body + N children body 前,机械对比:`(parent_new ∪ ∪children_outbound) ⊇ parent_old`;失败 → LLM 重试或拒写 | F-11 / E-2 |
|
||||
| 全员 → typed link predicate | wikilink 的 predicate 是 edge identity 的一部分;G\* update / split 不能丢 predicate(`is_a:: [[A]]` 必须保持;否则被守恒校验当作 drop edge + add edge 拦下);predicate 升 / 降级走显式 audit 路径 | F-11 / §1.4 |
|
||||
| 全员 → resource/daily | 都不能改 | I-2 / I-3 |
|
||||
| 全员 → 节点 rename | rename = 一次 `wikilink_handler.retarget_links(old_path, new_path)`;无 alias 表,无透明展开 | §1.3 |
|
||||
| 全员 → provenance link | 永远必须可达(I 不变量 + D10 检测) | I-1 / I-4 |
|
||||
| 全员 → kind 字段 | reme 核心**透明**:不读取 frontmatter `kind` 做结构决策;`kind` 是消费层 schema 提示 | [[reme4_schema_layering]] |
|
||||
| 全员 → predicate 谓词 | reme 核心**结构决策不读**:G7 / 中心性都聚合所有 predicate 算;edge 唯一性 / 反向索引会用到 predicate(防同源同标不同 predicate 互相覆盖);未类型化 link 是默认形态 | [[reme4_schema_layering]] / §1.4 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 待对齐边界点(后续讨论清单)
|
||||
|
||||
### 3.1 G\* update 的语义守恒边界(已收敛)
|
||||
|
||||
**决策**:**LLM 重写整段**(prompt 强约束"语义守恒,只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁")+ **机械守恒校验**(详 §1.5.5 E-1)。校验失败 LLM 重试一次,再失败拒写 + audit。
|
||||
|
||||
首版可先用 append 起步(出边集合天然 ⊇,守恒校验自动通过),prompt 工程量小;成熟后切到重写。
|
||||
|
||||
### 3.2 maintainer 的人 / agent 后门 → `auto_maintain_design.md` §8
|
||||
|
||||
### 3.3 G\* 与 split 的时序 → `auto_maintain_design.md` §7
|
||||
|
||||
### 3.4 节点 kind / 边 predicate(已收敛)
|
||||
|
||||
reme 核心**只有节点 + 边两种结构类型**:
|
||||
- frontmatter `kind` 字段(若存在)= 消费层的**节点内容标签**(concept / method / case / entity / topic / ...),reme 不读它做结构决策
|
||||
- 边 `predicate`(Dataview 风格,若存在)= 消费层的**边关系标签**(`is_a` / `extends` / `causes` / ...),reme 解析 / 存储 / 参与 edge 唯一性,但**结构决策不读**(G7 不区分 predicate;中心性不区分)
|
||||
- "overview 节点"角色靠图位置(高中心性 / 是 split parent)+ body 内容形态识别,不靠 frontmatter 或 predicate 标记
|
||||
- 未类型化 wikilink 是默认 / 常态形态
|
||||
|
||||
详见 §1.4 / §1.5.1 / §2.4。
|
||||
|
||||
### 3.5 retrieve 时的权重策略(部分收敛 → §1.5.8)
|
||||
|
||||
- 加权策略:节点权重 = base(=1.0) × intent 调节 × 中心性增益;query 含元意图("概览 / 主题 / 入门 / 全景"等)时,中心性高的节点加权
|
||||
- traverse 默认不强行展开高中心性节点(防止整族 children 拉进来);agent 可显式深入
|
||||
- 不按 frontmatter `kind` 加权;中心性天然来源 = split parent(详 §1.5.8)
|
||||
|
||||
剩余待定:**中心性算法选型**(eigenvector / PageRank / 简单入度,初期可用入度,后续校准)。
|
||||
|
||||
### 3.6 M split 时的 provenance 处理 → `auto_maintain_design.md` §6
|
||||
|
||||
### 3.7 bucket 集合管理
|
||||
|
||||
§1.2 已定:bucket 集合**固定预定义**,不由 digester / maintainer 动态生成。补足细节:
|
||||
|
||||
- **定义位置**:`vault.yaml` 顶层 `digest.buckets:` 是源 + 自动生成 `digest/_buckets.md` 作为人/LLM 可读视图;digester G4 时读后者作为 prompt context
|
||||
- **初始化**:opinionated default(通用桶 `concept` / `method` / `pattern` / `tool` / `domain` 等 + 必带 `general`);消费层可改桶名,但 **`general` 不可删**(否则 G4 失去兜底)
|
||||
- **扩展路径**:reme 不主动提议扩 bucket(对比旧设计的 maintainer 周期建议已 DROPPED);用户编辑 `vault.yaml` 后下次 G4 即生效
|
||||
|
||||
**未归类节点处理**(G4 找不到合适专属桶时):**统一落入 `digest/general/`**。
|
||||
| **物理几何** | `digest/<bucket>/<slug>.md`;**浅桶一层**(顶多两层),桶内 flat |
|
||||
| **bucket 角色** | **仅承担物理归档 + OS-level 浏览锚点**;不承担语义本体角色 —— 主题由图中节点表达 |
|
||||
| **bucket 集合** | **代码内 hard-coded**(`reme4/steps/evolve/auto_dream.py` 的 `BUCKETS` 常量),不通过配置外置,不由 dreamer / maintainer 动态生成 —— 三桶设定是 dream 模型本身的一部分(Phase 2 prompt 按 bucket 专化),不是可调参数 |
|
||||
| **集合视图** | 桶名内嵌在 prompt 中(extract 阶段三桶判别启发 + 三份独立 integrate prompt);不再生成独立 `_buckets.md` 视图 |
|
||||
| **初始化** | opinionated **三桶**,按"答什么问 + 谁在问"划分:`procedure`(答"怎么做 X" —— 步骤 / 方法 / runbook)/ `personal`(答"X 是谁 / 喜欢什么 / 不要做什么" —— 用户 / 团队 specific 身份 + 偏好)/ `wiki`(答"X 是什么 / 发生了什么 / 决策依据是什么" —— 通用知识 / 定义 / 原则 / 观察 / 决策先例;**也是默认兜底**) |
|
||||
| **bucket 主页** | 不强制存在;split 累积出层级时 parent 节点天然成为浏览主页(中心性涌现,非架构必需) |
|
||||
| **新节点归属** | bucket 由 **Phase 1** 在 unit 级别分配(写进 `MemoryUnit.bucket`),Phase 2 据此分发到对应 bucket 的专用 prompt;LLM 不能造新桶 |
|
||||
| **未归类节点** | Phase 1 找不到更明确归属时强制归入 `wiki` —— 它就是默认兜底,不是失败状态 |
|
||||
| **跨桶 move** | F-1 已禁止;若必须做(人工介入修错桶),走一次 `wikilink_handler.retarget_links(old, new)` |
|
||||
|
||||
**`wiki` 兜底桶**:
|
||||
|
||||
| 维度 | 内容 |
|
||||
|---|---|
|
||||
| **bucket 名** | `general`(固定集合一等公民,默认包含) |
|
||||
| **语义** | "通用主题 / 暂无专属归属" —— 合法常态,非故障状态 |
|
||||
| **路径** | `digest/general/<slug>.md`,与其它 bucket 完全等同 |
|
||||
| **节点演化** | 与其它 bucket 一致 |
|
||||
| **错桶后续** | 不主动跨桶 move(无 D9 / M-D);若严重,人工 mv + `retarget_links(old, new)` |
|
||||
| **语义** | "通用知识 / 默认归属" —— `wiki` 在三桶中 scope 最广(定义 / 原则 / 观察 / 决策先例),Phase 1 没有更明确归属(不属于 `procedure` 的可执行流程,也不属于 `personal` 的用户 specific 偏好)时归入此桶;**是合法常态,不是故障状态** |
|
||||
| **路径** | `digest/wiki/<slug>.md`,与其它 bucket 完全等同;节点演化与其它桶一致 |
|
||||
| **错桶后续** | 不主动跨桶 move;若严重,人工 mv + `retarget_links(old, new)` |
|
||||
|
||||
**为什么是 `general` 而不是 `_unclassified`**:`_unclassified` 暗示待处理状态,LLM/人都想清理掉;`general` 是合法常态,G4 选桶时是显式合法选项而非 fallback 故障路径。
|
||||
**为什么 `wiki` 兜底,而不是另设 `unknown`**:三桶设计中 `procedure` / `personal` 都有明确语义边界,剩下的"X 是什么 / 决策依据 / 一般原则"自然落在通用知识那一边 —— 这恰好就是 `wiki` 的本职。再设独立 `unknown` 会出现两类语义重叠的兜底(`wiki` 的"通用知识" vs `unknown` 的"分类未定"),反倒让 LLM 在 Phase 1 多一道无意义的犹豫。`wiki` 节点本身就是合法常态,不需要后续清理。
|
||||
|
||||
**已排除**:拒绝写入(候选丢失)/ 强行选最近似专属桶(本体污染,general 反而更安全)。
|
||||
**为什么是浅桶而不是深树**:
|
||||
- 物理浏览有"主题轮廓"(打开 `digest/wiki/` 能看到这一族节点),不像纯 flat 那样毫无锚点
|
||||
- 节点不被深路径绑死("属 wiki/auth 还是 wiki/session"这种归属焦虑被消解 —— 一个节点可以同时被多个主题通过 wikilink 引用)
|
||||
- F-1(0 文件移动)+ 平铺后,深树的核心收益(子树重组)消失,只剩深路径维护负担
|
||||
- **固定三桶的关键意义**:LLM 在 dream 桶决定时只做"分类"(三选一),不做"造类" —— 决策面坍缩,跨任务跨时间稳定;不会出现 "knowledge" / "wiki" / "concepts" 三个语义重叠的桶共存。三桶覆盖 personal-knowledge 的核心切片(做什么 / 谁喜欢什么 / 知识本身),进一步细分由桶内 wikilink 图自然涌现
|
||||
|
||||
### 3.8 检测阈值校准 → `auto_maintain_design.md` §3
|
||||
|
||||
### 3.9 provenance 载体形态(已收敛)
|
||||
|
||||
**决策**:**provenance wikilink 嵌在节点 body 正文中**(inline body prose),由 LLM 在 G\* / split prompt 里自然写出,跟其它 body wikilink 完全同形,**靠语义维护**。reme 核心没有 provenance 专用算子。
|
||||
|
||||
**写出形态**:
|
||||
- 行文中自然带出处:"... 该模式最早出现在 [[daily/2026/05/15.md]] 的实践中"
|
||||
- 或专门一段总结式段落,内含若干 wikilink 指向上游
|
||||
- 可选 predicate(`derived_from:: [[daily/2026/05/15.md]]`),不强制
|
||||
|
||||
**机械保护**:
|
||||
- E-1 守恒(G\* update):旧 provenance 不在新 outbound 集合 → 重试或拒写,机械兜底
|
||||
- E-2 守恒(split):provenance 跟着对应内容段自然分配到 parent overview / children,合计守恒
|
||||
- D10 检测:provenance 断裂 = D1 断链子集(target 命中 `daily/` / `resource/` 前缀);D10 严重程度高于普通 D1(I 不变量)
|
||||
|
||||
**E-5 / G6 等"provenance 专用机制"全部坍缩** —— 不再单列。Prompt 必须要求"出处用 `[[...]]` 形式表达"(纯散文会被守恒校验视为丢边)。
|
||||
|
||||
### 3.10 G\* 语义查后端(已收敛)
|
||||
|
||||
**决策**:**直接复用 `SearchStep`(`reme4/steps/index/search.py`)** —— 多路召回并发(vector + keyword)+ RRF 融合 + file_graph 邻接展开,把**完整候选池交给 LLM 终判**;G\* 入口不做 bucket 粗筛(LLM 拥有完整跨桶视野,可识别"概念错分到 general"或"跨桶同概念";三路信号 RRF 融合后噪声可控)。
|
||||
|
||||
**召回链路**:
|
||||
1. 候选原子单元(摘要 / 关键词)→ `SearchStep`(`search_filter={"path_prefix": "digest/"}`,I-2/I-3 daily/resource 不入池)
|
||||
2. `SearchStep` 内部:`vector_search` + `keyword_search` 并发 → RRF 融合 → `expand_links` 邻接展开 → 返回 top-`limit` FileChunks(含 path / 行号 / 邻接节点)
|
||||
3. 候选池整体喂 LLM,按 path 自然聚合(同节点多 chunk 命中 = 强信号);终判输出节点路径
|
||||
4. LLM 终判 create / update / drop;update 选定 subject node → 走 E-1 守恒重写
|
||||
|
||||
**provenance 不依赖召回** —— G\* / split prompt 让 LLM 直接写 `[[daily/...]]` / `[[resource/...]]`(§3.9)。
|
||||
|
||||
**索引维护**:沿用 `update_index` step,G\* / split 写 body 后调一次刷该节点索引;启动一次全建(`clear_and_scan` 已就绪),损坏走全建兜底。
|
||||
|
||||
**默认参数**(可按 dogfooding 调):`limit` 5~10 / `vector_weight` 0.7 / `expand_links` on / `min_score` 0(初版不过滤,LLM 兜底)。
|
||||
|
||||
### 3.11 G\* / split 写入并发 / 原子性 → `auto_maintain_design.md` §5
|
||||
|
||||
### 3.12 D3 触发模型 → `auto_maintain_design.md` §4
|
||||
|
||||
### 3.13 anchor 不引入 wikilink 设计(已收敛)
|
||||
|
||||
**决策**:**digest 设计层不使用 `[[path.md#section]]` 形态** —— wikilink 只有 `[[path.md]]`(可选 alias / 谓词),anchor 不进入 digest。当 LLM 想"指向某个具体子主题"时,正确做法是让那个子主题升级为独立节点(必要时通过 split),而不是在过载 parent 内部用 anchor 凑合。
|
||||
|
||||
**连锁简化**:
|
||||
- E-4(inbound anchor 机械 retarget)整类**消失**;split 流程末尾不再扫 inbound anchor 子流程;`{anchor → child}` 映射输出从 split prompt 中移除
|
||||
- 边唯一性键从三元组 `(target, predicate, anchor)` 简化为二元组 `(target, predicate)`
|
||||
- §1.5.5 边迁移类别从 4 类(E-1..E-4)简化为 3 类(E-1..E-3)
|
||||
- `FileLink.target_anchor` 字段在 schema 中保留(供其它消费层),digest 层永远写 `None`
|
||||
|
||||
**Prompt 约束**:G\* / split 的 prompt 必须明确告知 LLM 写 wikilink 时不带 `#section`。若 LLM 仍写出 `[[path.md#section]]`,wikilink_handler 仍能解析,守恒校验只看 `(target, predicate)`,不会形成"丢边"风险 —— 但 anchor 在 digest 层无语义。若引用方依赖某 anchor 锚定具体段落,表明该内容应升级为 child 节点。
|
||||
**已排除**:动态扩桶 / 拒绝写入(候选丢失)/ 强行选最近似专属桶(本体污染) / 把 bucket 数推回 6+(决策面失控)。
|
||||
|
||||
---
|
||||
|
||||
## 4. 下一步
|
||||
## 2. 节点
|
||||
|
||||
本文档覆盖 dream 模型 + 生成侧 + 召回(G\* / 节点+边模型 / SearchStep)。组织端实现清单(M split / D 检测 / CAS)见 `auto_maintain_design.md` §10。
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **粒度** | atomic;一个 .md 文件 = 一个原子单元(概念 / 方法 / 实体 / 案例 / 原则 / 主题概览)|
|
||||
| **节点角色** | **由 body 内容决定,不由 frontmatter 类型标记**;同一节点扮演"主题概览"还是"具体方法",看它的 body 写了什么 |
|
||||
| **身份(ID)** | **vault-relative 路径(含 `.md`)即节点身份** —— `digest/auth/jwt-rotation.md` |
|
||||
| **`name` frontmatter** | 文件名 basename(不含扩展名),与文件名同步 —— 检索 hint / 人读标签,**不当 ID 用** |
|
||||
| **frontmatter 保留字段** | 只有 `name` + `description`(reme 核心保留)|
|
||||
| **可选 `kind` 字段** | 例:concept / procedure / preference / observation / ...;**消费层 schema 提示**,reme 核心透明,不读它做结构决策。与 bucket 是不同概念 —— bucket 决定物理归档(三桶)+ Phase 2 prompt 走哪份;`kind` 是更细粒度的 frontmatter 标签,留给消费层自由使用 |
|
||||
| **文件名冲突** | 同 bucket 内文件名冲突 → 文件系统层断言(写入即拒);不需要独立检测信号 |
|
||||
| **rename** | 一次 `wikilink_handler.retarget_links(old_path, new_path)`(机制现成);无 alias 表,无透明展开 |
|
||||
|
||||
1. **digester 流程图**(G1 / G2 / G\* 的实际编排;G\* 内 create / update 路径分流;**召回直接复用 `SearchStep`**(vector + keyword + 邻接展开,RRF 融合,scope `digest/`)—— 详 §3.10;**G\* update 写入前 outbound diff 守恒校验** — E-1)
|
||||
2. **rename 路径设计**(`wikilink_handler.retarget_links(old_path, new_path)` 已就绪;封装为单步 step 入口,无 alias 表 / 无透明展开)
|
||||
3. **bucket 集合配置**(`vault.yaml` schema / 默认桶模板 / `general` 兜底机制 / `_buckets.md` 视图生成)
|
||||
4. **边守恒校验工具**(`extract_links` 已就绪;新增 outbound diff 比较器 + LLM 重试编排 + ConservationViolation audit 事件)
|
||||
5. **provenance prompt 规范**(G\* / split 引导 LLM 写 `[[daily/...]]` / `[[resource/...]]` —— §3.9)
|
||||
**为什么 atomic + 路径即 ID**:
|
||||
- **节点粒度 = retrieve 精度上限** —— semantic 检索召回 "一个原子单元" 远比召回 "一个 5000 字的主题文档" 信噪比高
|
||||
- **wikilink 在 atomic 粒度才真有意义** —— `[[digest/auth/jwt-rotation.md]]` 指向"一个具体方法"比指向"auth 主题文档"精确一个数量级
|
||||
- F-1 + 平铺 + 下层 immutable 后,slug abstraction 的核心价值(移动鲁棒性)蒸发;路径作 ID 与 `wikilink_handler.py` 默认形态完全对齐(*Recommended form: full path relative to the vault with extension*)
|
||||
- provenance wikilink 反指 daily/resource 本来就用路径,统一后整个 vault 一种 wikilink 形态
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_memory_design.md` / `auto_maintain_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
**"主题概览节点"靠内容识别,不靠前缀 / kind**:`hub__` / `topic__` 前缀**不存在**;文件名自然命名(`auth-fundamentals.md` / `jwt-rotation.md`)。主题概览身份是图位置(中心性 / split parent)+ body 形态共同涌现。
|
||||
|
||||
---
|
||||
|
||||
## 3. 边
|
||||
|
||||
参考实现:`reme4/utils/wikilink_handler.py` + `reme4/schema/file_link.py`。
|
||||
|
||||
| 形态 | 写法 | 说明 |
|
||||
|---|---|---|
|
||||
| **基础** | `[[<vault-path>.md]]` | literal,不隐含 `.md`,不自动短链补全 |
|
||||
| **alias** | `[[path.md\|display-text]]` | rewrite 时 alias 保持 |
|
||||
| **image** | `![[image.png]]` | 资源引用,不是知识边 |
|
||||
| **可选谓词** | `predicate:: [[path.md]]`(行级)/ `[predicate:: [[path.md]]]`(内联) | Dataview 风格;谓词在 `[[]]` 外,`[[]]` 内只保留纯目标 |
|
||||
| **谓词标识符** | `[A-Za-z][A-Za-z0-9_]*`(`is_a` / `extends` / `causes` / `references` ...) | 词表**开放**,任意标识符 |
|
||||
| **未类型化合法** | 绝大多数 wikilink 不加 predicate;`predicate=None` 是默认 / 常态 | |
|
||||
| **边唯一性键** | `(target_path, predicate)` 二元组 | 同源同标不同 predicate = 不同边 |
|
||||
| **不引入 anchor** | digest 设计层不使用 `[[path.md#section]]` | `FileLink.target_anchor` schema 保留(供其它消费层),digest 层永远写 `None` |
|
||||
|
||||
**reme 核心对 predicate 的"透明"边界**(关键):
|
||||
- 横向 link/ retrieve 中心性 —— 都**聚合所有 predicate** 算,不分桶
|
||||
- 只有 edge 唯一性 / 反向索引会用到 predicate(否则 `[[A]]` 和 `is_a:: [[A]]` 会被当作同一条边互相覆盖)
|
||||
- 消费层若要按 predicate 做更精细的推理(如"taxonomic 路径只走 `is_a` 边"),自己读 `FileLink.predicate` 即可
|
||||
|
||||
**与 `kind` 一致的立场**(与 [[reme4_schema_layering]] 对齐):reme 核心**只有节点 + 边两种结构类型**;`kind` / `predicate` 都是内容标签,绝不参与"hub / topic / leaf"这类结构角色判断。
|
||||
|
||||
**为什么不引入 anchor**:LLM 想"指向具体子主题"时,**正确做法是让那个子主题升级为独立节点**(必要时通过 split),不在过载 parent 内部用 anchor 凑合。anchor 在 digest 层无语义;prompt 必须明确告知 LLM 写 wikilink 时不带 `#section`。
|
||||
|
||||
---
|
||||
|
||||
## 4. 演化
|
||||
|
||||
### 4.1 演化只做两件事
|
||||
|
||||
| op | 谁 | 何时 | 改什么 |
|
||||
|---|---|---|---|
|
||||
| **dream**(create_or_update) | dreamer(本文档 §4.2) | 入流(新材料进入) | 创建新节点 / update 已有节点 body(语义守恒重写;UPDATE 内分 **CORROBORATE / REFINE / CORRECT** 三种 flavor,详 §4.2.3) |
|
||||
| **M split** | maintainer(`auto_consolidate_design.md` §1) | 节点过载(token / 主题离散度超阈值) | 把 parent body 拆成 parent overview + N children;parent 文件原地 |
|
||||
|
||||
> **关键观察**:"主题概览节点"不是一种 kind,也不是 maintainer 主动涌现的产物 —— 它是 split 的副产品(parent 节点天然成为该 cluster 的 overview,中心性自然高)。
|
||||
|
||||
显式排除:
|
||||
- ❌ merge / dissolve / re-edge / unify —— 跨节点重组不做(同概念二次进入靠 dream update;错桶节点不主动 move)
|
||||
- ❌ 完美归簇 —— F-5 留白,不确定就不动
|
||||
- ❌ 实时一致 —— 异步 / eventual
|
||||
|
||||
### 4.2 dream(create_or_update)流程
|
||||
|
||||
**dream = dreamer 入流唯一改 body 的操作,且只改 subject node。**
|
||||
|
||||
#### 4.2.0 digest 是抽象记忆层
|
||||
|
||||
Digest 是 agent 长期记忆的**抽象层** —— 类比前额叶对认知的聚合。原始细节(数字、流程文本、谁说了什么)留在材料(daily / resource),digest 只承载细节淡忘后仍想调取的那一层:原则、模式、可作为先例的决策、认知要点。这一立场决定了 dream 流程的形态:**Phase 1 识别抽象,Phase 2 把抽象登记到 digest 节点**。
|
||||
|
||||
#### 4.2.1 两阶段流程
|
||||
|
||||
```
|
||||
material 进入(daily / resource 选定 scope)
|
||||
│
|
||||
▼
|
||||
Phase 1 — extract (轻量)
|
||||
LLM 读材料 → 识别其中教导的"抽象"(原则 / 模式 / 先例)
|
||||
→ 为每个 unit **分配 bucket**(procedure / personal / wiki)
|
||||
→ 发出 ExtractedUnits 结构化输出 = K 个 sub-unit
|
||||
(每个: {name, bucket, summary})
|
||||
说明:多个支撑事实说明同一抽象 → 合并为同一 sub-unit
|
||||
(倾向少而精);Phase 1 是 gate ——
|
||||
无新抽象时发空列表,Phase 2 跳过整轮;
|
||||
bucket 由 Phase 1 一次性决定,Phase 2 不再回选
|
||||
│
|
||||
▼ (Python 外循环,K 次)
|
||||
Phase 2 — integrate (per sub-unit,**按 bucket 分发到独立 prompt**)
|
||||
│ system prompt = integrate_system_prompt_<unit.bucket>
|
||||
│ procedure / personal / wiki 三份独立 prompt,**不共用一套**
|
||||
│ sub-unit ↔ digest 节点 1:1;Phase 2 必写,无 SKIP 出口
|
||||
│
|
||||
├─ RECALL: search(关键词 + 向量 + RRF) + traverse(对 top hit
|
||||
│ 做图扩展,**跨 bucket**) → 候选路径集
|
||||
│
|
||||
├─ HIT: frontmatter_read 廉价 triage → read 完整 body
|
||||
│ 确认候选是否承载同一抽象 → hit 集合
|
||||
│
|
||||
├─ 决策:
|
||||
│ ├─ hit 空 → CREATE 在 digest/<unit.bucket>/<slug>.md
|
||||
│ └─ hit 非空 → UPDATE 路径 (CORROBORATE / REFINE / CORRECT;
|
||||
│ 目标可在任意桶 —— 召回是跨桶的)
|
||||
│
|
||||
▼
|
||||
写入(canonical write 创建 / canonical edit 改正文)
|
||||
│
|
||||
▼
|
||||
agent 上报 IntegrateOutcome {action, target_path}
|
||||
```
|
||||
|
||||
**两阶段 trade-off**:Phase 2 把完整材料发 LLM K 次(一次一 sub-unit),不做 summary loss;代价是 K 倍 prompt token。换来的是 Phase 1 只做"识别抽象 + 分类 bucket"两件事(粒度集中在一个 prompt),Phase 2 每次会话上下文干净、bucket-specific prompt 让推理聚焦于"这一桶要怎么写 / 怎么改"。
|
||||
|
||||
**Phase 2 的 bucket 专化**:三桶各有独立 system prompt,因为各桶的 body 形态、决策偏置不同 —— `procedure` 节点是 runbook 风(触发 / 步骤 / 前置 / 失败模式),`personal` 节点是规则风(rule + Why + How to apply),`wiki` 节点是百科风(定义 + 性质 + 关系)。共用一份通用 prompt 会让"应该写成什么样"的指导被稀释,bucket 信号靠一段 if-this-then-that 散文承载,效果劣于让每桶自带专属 prompt。
|
||||
|
||||
#### 4.2.2 召回二段
|
||||
|
||||
**RECALL = search + traverse**:search 给关键词 + 向量 RRF 命中;只要 search 在 `digest/` 下返回任何 hit,就对 top hit 跑 `traverse depth=2 direction=both`。理由是 search 关键词导向,会漏掉用不同术语归档的语义相邻抽象,那些常常一跳之外。search 在 `digest/` 下完全无命中 → 无 traverse 起点 → 候选集为空 → 直接 CREATE。
|
||||
|
||||
**HIT = frontmatter_read + read**:渐进披露 —— 先 `frontmatter_read` 读 `name + description` 廉价 triage 淘汰明显无关候选,剩下的再 `read` 整 body。**不可仅凭 chunk 片段或 frontmatter 决定 UPDATE**,body 才是判定依据。
|
||||
|
||||
#### 4.2.3 UPDATE 三种 flavor
|
||||
|
||||
| flavor | 何时 | body 怎么动 |
|
||||
|---|---|---|
|
||||
| **CORROBORATE**(最常见)| 已有节点已覆盖此抽象,材料是又一个实例 | body 实质不变 —— 追加 `derived_from::` 溯源,可选强化措辞("似乎"→"确实") |
|
||||
| **REFINE**(常见)| 已有节点覆盖了核心,但材料揭示新的范围 / 边界 / 维度 | 改相关片段使更精确,加新维度,加 `derived_from::`。正文在**精度**上长,不在**细节**上膨胀 |
|
||||
| **CORRECT**(少见)| 材料与已有抽象矛盾 / 表明它被夸大 | 收紧到新旧证据都支持的窄形式,或内联标注 `> note: contradicted by [[...]]` 不仲裁。仍加溯源 |
|
||||
|
||||
三种都受 §4.4 E-1 强守恒约束(出边集合不能缩)。
|
||||
|
||||
#### 4.2.4 关键边界
|
||||
|
||||
- **Phase 1 是 gate + 分类器** —— "不值得记忆"在 Phase 1 过滤(空列表);此外 Phase 1 还为每个进入 Phase 2 的 unit 分配 bucket(procedure / personal / wiki),决定 Phase 2 走哪份专用 prompt;Phase 2 必然写,sub-unit 与 digest 节点 1:1
|
||||
- **Phase 2 prompt 按 bucket 分发** —— `integrate_system_prompt_procedure` / `_personal` / `_wiki` 三份独立 system prompt,各自承载该桶的 body 形态指南与决策偏置,**不共用一份通用 prompt**
|
||||
- **CREATE 写入桶 = Phase 1 分配的桶**;**UPDATE 目标可在任意桶**(召回跨桶,UPDATE 命中谁就写谁)
|
||||
- **dream update 必须语义守恒** —— LLM 重写 body 时只能"融入"新内容,不能删除已有信息(只增不删 / 不改原意;冲突标注 `> 注:不同来源记载...`,不擅自仲裁);**当前实现下 E-1 强守恒是 prompt-only 自律**(canonical edit 不做机械 outbound diff;早期 `digest_edit` 子类的机械校验已在切到 canonical 工具时移除,详 §4.4)
|
||||
- **Phase 2 用 canonical write / edit** —— 不再有 `digest_write_step` / `digest_edit_step` 子类;桶归位与边守恒都是 prompt-level 纪律
|
||||
- **dream 不改其它节点正文**(F-2) —— 只动 subject
|
||||
- **dreamer 不做事件级伞节点** —— 材料本身(daily / resource 文件)就是 fan-out 点,每个 sub-unit 的 `derived_from::` 让材料天然聚合到所有派生节点
|
||||
- **0 出边节点合法**(没识别到合适邻居),后续 dream 进入时其它节点可以反向链回来 —— 不强求 LLM 一次性给全
|
||||
- **dream 漏判去重**(同概念建成新节点)→ 不主动兜底,接受重复;若 vault 累积明显重复,由 auto-consolidate 的 dups 检测周期 batch 产报告(`auto_consolidate_design.md` §3)
|
||||
- **召回不做 bucket 粗筛** —— LLM 拥有完整跨桶视野,可识别"概念跨桶同抽象"(例如同一原则在 wiki 已有节点而 Phase 1 把新材料归入 personal,此时 UPDATE wiki 节点而非新建 personal 节点)
|
||||
|
||||
**provenance 写出**:
|
||||
- 行文中自然带:"... 该模式最早出现在 [[daily/2026/05/15.md]] 的实践中"
|
||||
- **强制 typed predicate `derived_from::`** —— body 必须织入至少一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`,纯散文形式不会被未来的 update / 守恒比对识别为边,下次 update 时会消失
|
||||
- LLM 直接做语义守恒重写(只增不删) —— 不走"首版 append 起步"的过渡路径
|
||||
|
||||
### 4.3 F-invariants(演化的硬约束)
|
||||
|
||||
| # | 约束 | 含义 |
|
||||
|---|---|---|
|
||||
| **F-1** | **0 文件移动** | dream / split 都不移动现有文件;split 创建的是**新文件**,parent 原地 |
|
||||
| **F-2** | **改正文限定 subject** | dream update 改 subject body;M split 改 parent body + 创建 children body;**没有任何操作改"其它节点正文"** |
|
||||
| **F-3** | **maintainer 只做 split** | 没有 summarize / merge / re-edge / link / unify / dissolve |
|
||||
| **F-4** | **一次一个候选** | M split 一次拆一个;dream 一次处理一个原子单元(N 候选 = N 次 dream) |
|
||||
| **F-5** | **不确定时不动** | dream 拿不准 create 还是 update → 倾向 create;split 拿不准 cluster → 不拆 |
|
||||
| **F-7** | **多归属合法** | 一个节点可被多个引用,也可指向多个;**没有"单父"约束** |
|
||||
| **F-10** | **inbound 目标节点不动** | 所有 inbound 是裸链 `[[<parent-path>.md]]`(digest 不引入 anchor);split 时全部保持,parent 路径未变即天然有效 |
|
||||
| **F-11** | **wikilink 是 body 的一部分** | 不存在"独立的边";reme 核心机械算子只感知字符层,语义责任在 LLM(prompt 自律);split 写入路径仍带机械 outbound 校验,dream update 当前是 prompt-only(详 §4.4) |
|
||||
|
||||
### 4.4 边守恒(E-1 / E-2 / E-3)
|
||||
|
||||
**前提**:wikilink 是 body 的一部分(F-11)。"边"不是独立抽象 —— body 一变,边就跟着变。reme 核心**没有"修边"算子**;边的所有变化都是 body 文本编辑的副作用。语义层守恒由两条腿承担:**prompt 自律**(LLM 在 update 时被反复要求 only-add, not-delete)+ **必要时的机械校验**(下文区分了哪些保留、哪些已移除)。
|
||||
|
||||
| # | 类别 | 规则 | 谁负责 |
|
||||
|---|---|---|---|
|
||||
| **E-1** | dream update 节点出边(subject 自身) | **强守恒**:新 body 出边 ⊇ 原 body 出边(`(target, predicate)` 二元组,predicate 一并守住) | **当前实现:LLM(prompt)自律** —— canonical `edit` 不做机械 outbound diff,prompt 反复强调"never drop wikilinks the old span contained" |
|
||||
| **E-2** | split parent 出边(parent 拆解) | `(parent_new ∪ ∪children_outbound) ⊇ parent_old` | LLM(split prompt)+ 机械(由 maintainer 在 split 写入路径上实施,见 `auto_consolidate_design.md`) |
|
||||
| **E-3** | inbound wikilink `[[<parent-path>.md]]` | split 时**不动** —— 仍指 parent;后续 dream 进入若 LLM 觉得 child 粒度更合适,直接加新边到 child(F-10) | 不动 |
|
||||
|
||||
**E-1 实现取舍**:早期版本有专用 `digest_edit_step` 子类,在写入前对 body 做 outbound diff 比较,违反守恒时返回 `REJECT_CONSERVATION` 让 LLM 重试。在切到 canonical `edit` 工具(放弃 digest 子类)后,这道机械校验被移除 —— 守恒退化为 prompt-only 自律。trade-off:
|
||||
- **失**:LLM 偶尔会在 REFINE / CORRECT 时无意丢弃 `derived_from::` 链;系统不再自动拒写
|
||||
- **得**:Phase 2 工具与系统其它写入路径完全一致(write / edit 是 canonical job),没有 dream-private 写入语义;prompt 复杂度下降,工具表面更小
|
||||
- **后续**:若 prompt-only 守恒在生产中被证伪(掉链率高),可在 canonical `edit` 上挂一个可选的 conservation 校验 hook(不再走子类化路径),由 dreamer 在调用前后各 read 一次做 diff;但当前不做
|
||||
|
||||
**强守恒(集合包含)而非等价**:`new ⊇ old` = 允许加新边(新关联),不允许减边(老内容不能丢);`new == old` 会拒绝任何新出边 → update 失去意义。
|
||||
|
||||
**predicate 守住** —— `[[A]]` ↔ `is_a:: [[A]]` 视为不同 key,升降级走显式 audit 路径,不走默认。重排 / 改 alias / 加新边都不被拦下(集合相同或只增)。
|
||||
|
||||
**provenance 不单列** —— 节点反指上游 daily/resource 的 wikilink 是 body 正文的一部分,跟其它 wikilink 走同一套 E-1 / E-2;reme 核心没有 provenance 专用算子。
|
||||
|
||||
**inbound anchor 这一类不存在** —— digest 不引入 anchor,所有 inbound 都是裸链,走 E-3 即可,无需机械 retarget 子流程。
|
||||
|
||||
---
|
||||
|
||||
## 5. 与其它层
|
||||
|
||||
| 上下游 | 关系 |
|
||||
|---|---|
|
||||
| ← **auto-memory**(daily) | dream 读 daily 作为入流;daily 写完即对 dream 可见 |
|
||||
| ← **resource** | dream 读 resource 作为入流(只读,不写) |
|
||||
|
||||
**关键边界**:dream 不写 daily / resource(I-2 / I-3);只写 digest 节点 body(自身 subject)。dream 不感知下游 —— split / 链接增强 / 索引刷新 / rename 等由 `auto_consolidate_design.md` / `auto_cognition_design.md` / `update_store_index_loop` 各自负责。
|
||||
|
||||
---
|
||||
|
||||
## 6. 下一步
|
||||
|
||||
本文档覆盖 dream 模型(桶 / 节点 / 边 / 演化)。组织端实现清单(M split / D 检测 / CAS 框架)见 `auto_consolidate_design.md` §10。
|
||||
|
||||
- ✅ **dream step 实现** —— Phase 1 extract(识别抽象 + 分配 bucket)+ Phase 2 integrate(per sub-unit,**bucket-specific prompt 分发**;`reme4/steps/evolve/auto_dream.py` + `auto_dream.yaml`,与 `auto_memory` 同级同形)
|
||||
- ✅ **三桶 hard-coded** —— `procedure / personal / wiki`,`BUCKETS` 常量在 `dreamer.py` 顶部,Phase 1 通过 `MemoryUnit.bucket: Literal[...]` 由 Pydantic 强制约束
|
||||
- ✅ **provenance prompt 规范** —— `derived_from:: [[daily/...]]` / `[[resource/...]]` 强制(三桶 prompt 各自重申)
|
||||
- ❌ ~~**边守恒校验工具**~~ —— 早期 `digest_edit` 子类的 outbound diff 校验已随子类一并移除(切到 canonical `edit`);E-1 现由 prompt 自律,详 §4.4
|
||||
- ❌ ~~**bucket 集合配置外置**~~ —— 撤销:三桶是 dream 模型本身的一部分,不做配置参数(`vault.yaml` 不再承载 `digest.buckets`,`_buckets.md` 视图也不再生成)
|
||||
|
||||
实现进入 `reme4/steps/evolve/` 时,本文档与 `auto_memory_design.md` / `auto_consolidate_design.md` / `auto_cognition_design.md` 共同作为契约依据。
|
||||
|
|
|
|||
|
|
@ -1,209 +0,0 @@
|
|||
# auto-link 设计(背景实体识别 + wikilink 写回)
|
||||
|
||||
> 本文档记录 reme4 中 **auto-link** 的设计讨论 —— 在已写入节点之间发现隐含关系,把这些关系作为 `[[...]]` wikilink **写回 body**,形成可见、可编辑的图结构增强。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §1.2(三层数据视角)/ §4(retrieve 三种问法)
|
||||
> - `auto_memory_design.md`:auto-link 可反向扫 daily event,补实体 wikilink(daily → digest)
|
||||
> - `auto_dream_design.md`:wikilink 模型(§1.4 边语法 / §1.5 演化 / §1.5.5 边守恒 E-1 / E-2 / E-3);auto-link 借这套基础设施
|
||||
> - `auto_maintain_design.md`:CAS 写入协议(§5);auto-link L1 写回与 dream G\* / maintain split 三方共用同一套 CAS
|
||||
>
|
||||
> **三层对应**:reme 服务整体三层 —— auto-memory / auto-dream / **auto-link(本文档)**。auto-link 是图关系的**后置增强** —— 在已落地的 vault 上做实体识别 + wikilink 写回,补足 content link(写记忆时由 LLM 直接产生的 `[[...]]`)在长 tail 隐含关系上的盲区。
|
||||
>
|
||||
> **核心立场**:auto-link **写回 body**,不只是产报告。生成的 wikilink 是**可见、可编辑**的(写在 Markdown 文件里),agent / 人可后续 curate。auto-link 不引入新材料,纯 additive 插入 wikilink,天然满足 E-1 守恒;复用 dream 的 CAS 写入协议,不引入新基础设施。
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
content link(`auto_dream_design.md` G\* / split 写入时由 LLM inline 产生的 `[[...]]`)解决了"写记忆时显式的关系"。但有一类关系不会在 inline 写入时自然涌现,需要后台扫描已写入的 vault 才能识别:
|
||||
|
||||
1. **历史 body 的实体未链接** —— G\* update 时 LLM 关注新材料融入,可能忽略已有 body 中某个未链接的实体(例如 body 提到 "JWT" 但没写 `[[digest/auth/jwt-overview.md]]`)
|
||||
2. **跨节点 / 跨桶的隐含关联** —— 节点 A 提到 "rate limit",但 `digest/api/rate-limit.md` 是后来才被 split 创建 → A 写入时没机会建立这条边
|
||||
3. **同主题未连 / 同概念重复** —— G\* 漏判去重把同概念建成两个节点;或两个主题相关但 0 链接的节点彼此不知晓
|
||||
|
||||
auto-link 承担这部分:**后台扫描已写入节点 → 实体识别 / 候选挖掘 → wikilink 写回 body**。
|
||||
|
||||
---
|
||||
|
||||
## 1. 已对齐决策
|
||||
|
||||
### 1.1 与 content link 的边界
|
||||
|
||||
| 维度 | content link(在 dream) | auto-link(本文档) |
|
||||
|---|---|---|
|
||||
| 何时产生 | 写记忆 inline:G\* update / M split prompt | 后台扫描:离线 / 周期 / 触发后异步 |
|
||||
| 由谁产生 | LLM 在 dream 写入流中顺手写出 | LLM 在 auto-link 扫描流中识别后写出 |
|
||||
| 输入 | 新材料 + 召回候选节点 | 已写入 body + 全 vault 索引 |
|
||||
| 改 body | 是(重写整段 body) | 是(纯 additive 插入 wikilink,不改文字) |
|
||||
| 守恒 | E-1 强守恒(out ⊇ old) | E-1 天然满足(纯增) |
|
||||
| 用途 | 写入即关系明示 | 弥补 inline 漏判,挖掘长 tail 关系 |
|
||||
|
||||
### 1.2 写回模型:纯 additive,复用 dream CAS
|
||||
|
||||
auto-link 写回是**纯 additive** 操作 —— 在已有 body 文字中找到实体 mention,替换为 wikilink 形态:
|
||||
|
||||
```
|
||||
Before: "JWT 轮换的核心是密钥派生 ..."
|
||||
After: "[[digest/auth/jwt-rotation.md|JWT 轮换]]的核心是[[digest/auth/jwt-key-derivation.md|密钥派生]] ..."
|
||||
```
|
||||
|
||||
| 维度 | 决策 |
|
||||
|---|---|
|
||||
| **alias 必须保留原文** | `[[path.md\|<原文>]]` 形态;原文一字不改 —— 守住"不改写其它节点正文" (`auto_dream_design.md` §1.5.4 F-2) 的精神 |
|
||||
| **predicate 默认为空** | auto-link 默认产生无谓词 wikilink;升 typed link 走 L3(详 §1.3) |
|
||||
| **不引入 anchor** | 与 dream 一致(`auto_dream_design.md` §1.4 / §3.13);target 永远是节点路径 |
|
||||
| **CAS 写入** | 完全复用 `auto_maintain_design.md` §5 的 read-stamp + CAS-write 协议(冲突重做 ≤ 3 次) |
|
||||
| **E-1 守恒** | 纯 additive:`new outbound = old outbound ∪ {new wikilinks}`;`new ⊇ old` 天然满足,守恒校验默认通过 |
|
||||
| **rollback** | 若 auto-link 误插入(例如 entity mention 是同名歧义),走标准 edit 或 retarget 撤销;auto-link 不维护"我插过哪些"audit log(留给 SDK 决定) |
|
||||
|
||||
**为什么是 additive 而不是重写**:
|
||||
- additive = 0 文字风险(原文不变,只在原 mention 周围加 `[[ | ]]` 包装)
|
||||
- 重写 = 触发完整 E-1 守恒校验 + LLM 重写整段语义守恒 prompt + 多次 LLM 调用 = 跟 G\* update 重复
|
||||
- additive 失败可见:产生坏 wikilink 时,人/agent 直接编辑 body 修就行
|
||||
|
||||
### 1.3 候选挖掘类型(L1-L4)
|
||||
|
||||
| # | 类型 | 描述 | 写回形态 |
|
||||
|---|---|---|---|
|
||||
| **L1** | **实体识别**(主路径) | 扫 body,识别已是 digest 节点的实体名(模糊匹配 + 语义召回);未被 wikilink 化的 mention → 加 `[[path.md\|<mention>]]` | additive wikilink 插入 |
|
||||
| **L2** | **同主题未连**(旧 D7) | 两个 digest 节点谈相关主题但 0 wikilink → 候选 add link;LLM 判后在 body 末尾追加一句引用 | additive(在合适位置 / 节末追加 `参见 [[other.md\|other]]`)|
|
||||
| **L3** | **隐含 predicate 推导** | 已有 `[[A]]` 但 LLM 可推断关系类型(`is_a` / `causes` / `extends` / ...)→ 升级为 typed link | 改 `[[A]]` → `is_a:: [[A]]`(predicate 升降级走显式 audit,详 §2.1)|
|
||||
| **L4** | **重复语义检测**(旧 D8) | 两个节点描述同一概念但被独立 create(G\* 漏判去重)→ 候选 merge | **不写回**;产报告 + 提示人/agent 触发 G\* update 路径手工合并 |
|
||||
|
||||
**L1 是主路径** —— 它是 auto-link 最核心、最频繁、最高 ROI 的操作:每个 digest 节点写完后,后台扫一遍 body,找未链接的已知实体,additive 加 wikilink。
|
||||
|
||||
**L2-L3 是辅助** —— 周期扫,产候选,LLM 终判,写回部分(L2 节末追加 / L3 升 predicate)。
|
||||
|
||||
**L4 不写回** —— 节点合并是结构改动,影响 E-1 守恒边界 + inbound 链路 + provenance 链路,不适合自动写;auto-link 只产报告,人/agent 决定走 G\* update 路径解决。
|
||||
|
||||
### 1.4 触发节奏
|
||||
|
||||
| 模式 | 何时 | 适用 |
|
||||
|---|---|---|
|
||||
| **inline post-write**(默认) | 每次 G\* update / M split 写完 body → enqueue auto-link L1 job(异步,FIFO,CAS 保护)| L1 实体识别;反应即时,与 D3 写后检测同节奏 |
|
||||
| **周期 batch**(可选)| cron(daily / weekly)扫全 vault | L2 / L3 候选挖掘;成本可控 |
|
||||
| **手动触发** | SDK / 人显式调用 | 全量重扫 / 修复 |
|
||||
|
||||
**L1 inline 的必要性**:新 split 出的 child 节点立即被既有 body 引用(用 wikilink 而非纯 mention)的关键 = 写入即扫描;不 inline 会让"刚创建的 child 节点"在很长时间内只有 split parent 一个 inbound,中心性失真。
|
||||
|
||||
**已排除**:
|
||||
- inline 时同步 auto-link(阻塞 G\* return)—— 时延不可接受;auto-link 始终异步
|
||||
- 所有 L\* 都 inline —— L2-L3 候选挖掘 RTL 跨节点,成本高,只适合 batch
|
||||
- 全 cron 唯一触发 —— L1 滞后过久,新节点孤岛
|
||||
|
||||
### 1.5 中心性算法(retrieve 加权依赖)
|
||||
|
||||
retrieve 时节点权重 = base × intent 调节 × **中心性增益**(详 `auto_dream_design.md` §1.5.8)。中心性需要 auto-link 这一层提供 —— content link 给底子,auto-link 补 long tail,二者合起来才是完整的图。
|
||||
|
||||
| 选项 | 优点 | 缺点 |
|
||||
|---|---|---|
|
||||
| **简单入度** | 实现最简;split parent 入度天然高;auto-link L1 加边后入度即时反映 | 不区分"权威节点"vs"被随手提的节点";高入度 ≠ 高权威 |
|
||||
| **PageRank** | 经典;权威性传递 | 实现复杂 + 增量更新成本(每次写边重算成本高,需 incremental algorithm)|
|
||||
| **eigenvector centrality** | 与 PageRank 相近 | 同上 |
|
||||
|
||||
**首版决策**:**简单入度**(file_graph 已有 inbound 链表,O(1) 查);auto-link L1 加边后入度立刻更新,split parent 自然涌现高入度。dogfooding 后视 retrieve 质量演进。
|
||||
|
||||
中心性是 retrieve 时**查询时计算**,不预存:
|
||||
- file_graph 已建反向索引(inbound),计算 `len(inbound(node))` 是 O(1)
|
||||
- 不预存避免"加边后中心性陈旧"问题
|
||||
- PageRank 演进时可加增量计算 + 周期 refresh
|
||||
|
||||
---
|
||||
|
||||
## 2. 待对齐边界点
|
||||
|
||||
### 2.1 L3 predicate 升降级的 audit
|
||||
|
||||
L3 把 `[[A]]` 升级为 `is_a:: [[A]]` 时,**改了 edge identity** —— `(target, None)` 变成 `(target, "is_a")`,在 E-1 守恒视角下 = 删一条边 + 加一条边:
|
||||
|
||||
```
|
||||
old outbound: {(A, None)}
|
||||
new outbound: {(A, "is_a")}
|
||||
diff: missing = {(A, None)}; added = {(A, "is_a")}
|
||||
```
|
||||
|
||||
不打 audit 走默认会被守恒校验拦下(`missing != ∅` → 重试 / 拒写)。
|
||||
|
||||
**决策方向**:
|
||||
- L3 写入必须打 audit flag(消费层意图:升级 predicate,允许 drop + add 同时发生)
|
||||
- audit flag 由 reme4 step 暴露(`maintainer_step(action="predicate_upgrade", from=..., to=...)`),不放在普通 write 路径
|
||||
- 普通 G\* / auto-link L1 写入永远不带 audit flag,守恒校验照常严格
|
||||
|
||||
详细 audit flag 接口形态留到 SDK 阶段。
|
||||
|
||||
### 2.2 多歧义实体识别
|
||||
|
||||
L1 扫 body 找 "JWT" 这个 mention,vault 中有 `digest/auth/jwt-overview.md` 和 `digest/payment/jwt-payment-flow.md` 两个 candidate:
|
||||
|
||||
候选方案:
|
||||
- LLM 上下文判 —— 把 body 周围段落给 LLM,选最相关 target
|
||||
- 跳过模糊 case —— L1 只处理 unambiguous mention,歧义 case 留人/agent
|
||||
- 全部链 —— `[[overview]][[payment-flow]]`,后续人 curate
|
||||
|
||||
**首版**:LLM 上下文判(每个候选 candidate 提供 description / 周围若干节点 summary,LLM 选择 top-1 或 drop);成本可接受(扫描已是离线 batch)。
|
||||
|
||||
### 2.3 auto-link 写回与 G\* / split 的并发
|
||||
|
||||
auto-link 写 body 走 §1.2 CAS,但有特殊情况:
|
||||
- 同节点同时被 G\* update 与 auto-link L1 写入 → CAS 协议自动序列化 (`auto_maintain_design.md` §5):后到者重做
|
||||
- auto-link L1 写完后立刻被 G\* update 覆盖(G\* 重写 body) → 看 G\* prompt 是否守住 auto-link 加的 wikilink(E-1 强守恒 → 守住)
|
||||
- auto-link L1 与 D3 派发的 split job 同节点并发 → split 先到 / 后到都不影响最终拓扑(split 把 body 拆成 parent + children,auto-link 加的 wikilink 跟着对应内容段自然分配到 parent / child)
|
||||
|
||||
**结论**:CAS + E-1 + E-2 守恒已覆盖所有并发场景,auto-link 不需要新协调机制。
|
||||
|
||||
### 2.4 跨 vault / 跨进程
|
||||
|
||||
M0 单 reme 实例 + 单 vault,auto-link 走内进程 enqueue;多实例 / 跨进程留 M1+(同 `auto_maintain_design.md` §5)。
|
||||
|
||||
### 2.5 实体识别 vs 现成 NER 库
|
||||
|
||||
L1 实体识别可选:
|
||||
- LLM 直接扫(贵但灵活,与 digest 节点同构)
|
||||
- 现成 NER 库(spaCy 等)预筛 + LLM 终判(快但 entity 类型与 digest 节点形态可能不匹配)
|
||||
- 纯字符串匹配(已知节点名字 + 简单变体)+ LLM 终判 ambiguity
|
||||
|
||||
**倾向**:从纯字符串匹配 + LLM 终判 ambiguity 起步(实现最简,效果可能已经够好);视 dogfooding 决定是否引入 NER 库。
|
||||
|
||||
---
|
||||
|
||||
## 3. 与其它层的协作
|
||||
|
||||
| 上下游 | 关系 |
|
||||
|---|---|
|
||||
| ← **auto-dream** | dream 写完一个节点 → 通过 inline post-write enqueue auto-link L1(§1.4);auto-link 用 dream 的 CAS 协议 |
|
||||
| ← **auto-memory** | auto-link 可反向扫 daily event,把实体识别成 `[[digest/...]]`(daily → digest);auto-memory 写入端不主动调 auto-link,触发同 dream 路径 |
|
||||
| → **digest body** | 主要写入对象 —— L1 additive 加 wikilink / L2 节末追加引用 / L3 升 predicate(走 audit) |
|
||||
| → **daily body** | auto-link 扫 daily event 时同样可加 `[[digest/...]]`(I-2 daily 单作者需协调:auto-link 应在 event 关闭后才动该 event,不与 active event 并发改;实现细节留 step 层处理) |
|
||||
| → **resource body** | I-3 immutable;auto-link **不写 resource**(reading-only) |
|
||||
| → **L4 候选 report** | L4 重复语义检测产报告,落 `audit/<date>/auto_link_l4.md`(具体路径 / 形态留 step 层) |
|
||||
|
||||
---
|
||||
|
||||
## 4. 与 auto-dream 模型的引用关系
|
||||
|
||||
本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §1.4 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §1.5 / §1.5.1 |
|
||||
| F-invariants(F-1..F-11)| `auto_dream_design.md` §1.5.4 |
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §1.5.5 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §1.3 |
|
||||
| CAS 写入协议 | `auto_maintain_design.md` §5 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §1.4 / §3.13 |
|
||||
| SearchStep 召回 | `auto_dream_design.md` §3.10 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 下一步
|
||||
|
||||
1. **L1 实体识别 step 实现** —— 字符串匹配 + 语义召回 + LLM ambiguity 终判 + additive wikilink 写回(§1.2 / §1.3)
|
||||
2. **inline post-write trigger 接入** —— G\* update / M split CAS 写入成功后 enqueue auto-link L1 job(§1.4)
|
||||
3. **L2 / L3 周期 batch 框架** —— cron(daily / weekly)+ 候选挖掘 prompt + 写回路径(§1.3)
|
||||
4. **L3 audit flag 接口** —— `maintainer_step` 提供 `predicate_upgrade` 操作,带 audit context 走特殊守恒规则(§2.1)
|
||||
5. **中心性 retrieve 增益** —— file_graph inbound count → retrieve 加权乘子(§1.5)
|
||||
6. **L4 报告框架** —— 重复语义检测产报告,提供 SDK / 人介入入口(§1.3 / §3)
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` 共同作为契约依据。
|
||||
|
|
@ -1,190 +0,0 @@
|
|||
# auto-maintain 设计(digest 组织端:M split / 检测 / 写入并发)
|
||||
|
||||
> 本文档记录 reme4 中 **auto-maintain** 的设计讨论 —— digest 层的组织 / 重组 / 写入运行时,含 M split、D 检测信号、写后触发模型、CAS 写入协议。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §3.6(maintain 动作语义)/ §7.3(maintainer 模块)
|
||||
> - `auto_dream_design.md`:节点 + 边模型(§1.1-1.5)/ F-invariants(§1.5.4)/ 边守恒 E-1/E-2/E-3(§1.5.5)/ G\* 操作(§2.1)—— maintain 复用这套底层模型
|
||||
> - `auto_link_design.md`:auto-link 写回也走本文档的 CAS 协议(§5)
|
||||
> - `auto_memory_design.md`:auto-memory 不直接复用 maintain,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
>
|
||||
> **三层框架的位置**:报告 §5 三层为 auto-memory / auto-dream / auto-link。maintain 严格按 `structure.md` §3.5-3.6 的 L4 action 分类是独立 action(`maintain: digest → digest`),不属 `digest` action(`digest: resource + daily → digest`)。本文档作为四方分工的**第四份**,专门覆盖 dream 写完之后 digest 的组织 / 重组 / 写入运行时。
|
||||
>
|
||||
> **核心立场**:
|
||||
> - **maintain 与 dream 同 pace**(idle background)、同模型(节点 + 边 / 守恒规则),但**语义边界不同**:dream 是 compose(资料 → digest),maintain 是 reorganize(digest → digest)
|
||||
> - **maintain 只做 split**,不做 merge / dissolve / re-edge / unify;过载就拆,其它跨节点重组留给消费层 / 人工
|
||||
> - **CAS 写入协议是基础设施**,被 dream G\* / maintain split / auto-link L1 共用,统一编排在本文档(§5)
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
dream 模型(`auto_dream_design.md` §1.5)规定 digest 的演化只做两件事:G\* create_or_update(入流型,新材料融入)+ M split(后台,过载就拆)。dream 文档负责 G\* 与节点 / 边模型;**本文档负责 M split 与运行时机制**(D 检测 / 触发模型 / 写入并发协议)。
|
||||
|
||||
| 输入 | 输出 |
|
||||
|---|---|
|
||||
| dream 写入后的 digest 状态 + 触发信号(D3 过载,inline) | parent overview 重写 + N 个新 children 文件;边守恒 E-2 通过 |
|
||||
|
||||
**设计目标**:
|
||||
1. **形状匹配** —— 让节点粒度持续与实际语义结构对齐(过载节点拆;不过载不动)
|
||||
2. **最小变更面** —— split 改 parent + 创建 N children,不动其它节点(F-2)
|
||||
3. **不引入新基础设施** —— 复用 dream 的节点 + 边模型 / 守恒规则;CAS 写协议自洽
|
||||
|
||||
**显式排除**:
|
||||
- ❌ merge / dissolve / re-edge / unify —— 跨节点重组不做(简化模型;同概念二次进入靠 G\* update)
|
||||
- ❌ 改其它节点正文 —— split 只改 parent body(重写为 overview)+ 创建 children body
|
||||
- ❌ 重建 inbound —— split 时 inbound 一律不动(F-10)
|
||||
|
||||
---
|
||||
|
||||
## 1. M split(maintainer 唯一 op)
|
||||
|
||||
| # | 能力 | 服务 | 触发 | graph | file | body |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **M split** | 节点过载 → LLM 拆成 parent overview + N 个 children;parent 文件原地保留,children 是新文件;children 加 `[[parent]]` 反向链接;inbound 边不动 | 形状匹配(粒度对齐)+ 任意尺度(涌现层级) | D3 过载 | parent 0 拓扑改;新 children 节点 + 各自加 `[[parent]]` 出边 | 创建 N 个 children 文件;parent 文件原地 | parent body 重写为 overview;children 各自有新 body |
|
||||
|
||||
**关键边界**:
|
||||
- **M split 改两类 body**:parent body(重写为 overview)+ N 个新 children body;不改任何**其它**节点(`auto_dream_design.md` §1.5.4 F-2)
|
||||
- **inbound 不重定向** —— 外部对 parent 的 wikilink 全部保留指 parent;后续 G\* 进入时若 LLM 觉得 child 粒度更合适,直接加新边到 child 即可(F-10)
|
||||
- **没有 dissolve 操作** —— children 长期空也不主动删;消费层 / 人工显式介入
|
||||
- **没有 merge / re-edge / unify** —— 跨节点重组不做;同概念二次进入靠 G\* update;错桶节点不主动 move(若严重,人工介入)
|
||||
- **边守恒** —— split 写新 parent body + N children body 前,机械对比 outbound:`(parent_new ∪ ∪children_outbound) ⊇ parent_old`;失败 → LLM 重试或拒写(F-11 / E-2,详 `auto_dream_design.md` §1.5.5)
|
||||
|
||||
---
|
||||
|
||||
## 2. 检测信号 D1 / D3 / D10
|
||||
|
||||
| # | 信号 | 服务 | 服务能力 |
|
||||
|---|---|---|---|
|
||||
| **D1** | 断链(wikilink → 不存在的 path) | 任意尺度(可达性) | 告警 / 简单修复(就地删 wikilink 或保留 alias 文本) |
|
||||
| **D3** | 过载节点(token 阈值 → LLM 判离散度) | 形状匹配(粒度) | maintainer(M split) |
|
||||
| **D10** | provenance 断裂(digest 节点反指的 daily/resource 不可达) | 任意尺度(跨层不变量) | 严重告警(I 不变量违反) |
|
||||
|
||||
**触发模型**:**写后立即** —— G\* / split 写完 body inline 检测;无后台 watcher / 无周期 tick / 无 dirty 队列(详 §4)。D1 / D10 是 wikilink 断链的子集,跟 file_graph 链路一起在写时检测。
|
||||
|
||||
> **简化模型砍掉的信号**:
|
||||
> - **D2 隔离 / D4 过疏 / D5 高入度 / D5b 低入度摘要 / D6 slug 冲突 / D7 相似未链 / D8 重复语义 / D9 邻居异质** —— 全部 DROPPED
|
||||
> - 旧 D5 高入度涌现 → 由 split 副产品(parent + children)等价覆盖;触发源换成节点过载(D3)
|
||||
> - 旧 D6 slug 冲突 → 路径即 ID 后,同 bucket 内文件名冲突由文件系统层断言(写入即拒),不需要独立信号(详 `auto_dream_design.md` §1.3)
|
||||
> - 旧 D7 / D8 → 简化模型不做 link / merge 提议;若 vault 累积明显的同概念重复,由 `auto_link_design.md` §1.3 L4 离线 audit 工具产报告
|
||||
> - 旧 D9 邻居异质 → 简化模型不做跨桶 move;桶选择只在 G4 一次性决定,后续不重排
|
||||
>
|
||||
> **D3 过载的判据**:token 阈值机械检查 + LLM 判离散度;**写后立即 inline**。阈值见 §3,触发模型见 §4。
|
||||
|
||||
---
|
||||
|
||||
## 3. 检测阈值校准
|
||||
|
||||
简化模型只剩 D3(过载)是核心阈值,其它都是 invariant 触发(无可调阈值)或 informational(无 maintenance 联动)。
|
||||
|
||||
| 信号 | 阈值类型 | 默认 | 备注 |
|
||||
|---|---|---|---|
|
||||
| **D3 过载** | token + 主题离散度 | token 2000 / 离散度由 LLM 写后 inline 判 | **唯一驱动 split 的阈值**(详 §4) |
|
||||
| **D1 断链** | 0 容忍 | 任意 1 条断链 → 告警 | 修复策略简单(就地删 wikilink) |
|
||||
| **D10 provenance 断裂** | 0 容忍 | 任意 1 条断裂 → 严重告警 | I 不变量 |
|
||||
|
||||
D3 阈值作为 `vault.yaml` 配置项(opinionated default,reme 核心提供机制不写死阈值),消费层可改;dogfooding 后调优。token 阈值起点 2000(对应"约 5 个独立子主题"的常见过载点),首版可调。
|
||||
|
||||
---
|
||||
|
||||
## 4. D3 触发模型(已收敛)
|
||||
|
||||
**决策**:**写后立即检测,无 watcher 抽象,无 batch 窗口** —— 每次 G\* update / split 写 body 成功后,**inline** 在同一 job 内跑 D3:token 阈值 + LLM 离散度判定 → 必要时 enqueue split job(异步,走 §5 CAS 队列)。
|
||||
|
||||
```
|
||||
G* / split 写 body 成功(CAS 通过)
|
||||
└─ if len(body) > T:
|
||||
└─ LLM 判离散度
|
||||
└─ if is_overloaded:
|
||||
└─ enqueue split job (FIFO, CAS-protected)
|
||||
└─ return
|
||||
```
|
||||
|
||||
**协议**:
|
||||
- token 阈值默认 `2000`(§3 已定,`vault.yaml` 可配)
|
||||
- 离散度 prompt 输出 `{is_overloaded: bool, suggested_clusters: [...]}`(若 overloaded 直接供 split job 吃,不重判)
|
||||
- 启动无全扫(避免长启动);新写入立即检测覆盖增长路径;历史遗留过载随下次 update 自然检出
|
||||
- 无 dirty 标 / 无 dirty 集合 / 无后台 worker —— D3 是写路径的合成函数
|
||||
|
||||
**为什么 inline**:反应即时(不等下一次 ingest);实现最简(无批处理窗口 / dirty 状态 / 独立 worker);LLM 判定成本可接受(大多写入 < T 不触发,触发后 split 切小后续不再越界);不引入 watcher = 少一层部署/监控。
|
||||
|
||||
**已排除**:定时 cron tick(静止 vault 浪费扫描)/ ingest-after batch(引入 dirty 集合)/ 独立 L2 watcher worker(多余部署层)。
|
||||
|
||||
**演进路径(M1+)**:若 inline LLM 阻塞 G\* 时延成问题 → D3 改为 fire-and-forget enqueue;若同节点重复触发 LLM 成本高 → 加节点级 body hash 缓存。
|
||||
|
||||
---
|
||||
|
||||
## 5. CAS 写入协议(共享基础设施)
|
||||
|
||||
**位置说明**:CAS 是 G\* update(`auto_dream_design.md` §2.1)、M split(本文档 §1)、auto-link L1 写回(`auto_link_design.md` §1.2)**三方共用**的写入协议。归在本文档是因为 maintain 是 digest 的"组织 / 运行时"端,运行时机制(检测 / 触发 / 写入)集中在一处方便对照。
|
||||
|
||||
**决策**:**并行决策 + 乐观冲突重做(CAS)** —— 所有 G\* / split / auto-link L1 决策并发跑,写入前用 body 版本戳(hash / mtime)做 CAS 比对;变了就丢弃 planned body 重做。无锁,无 ingest 级互斥。冲突率低 + E-1 / E-2 守恒校验顺手承担 race 兜底,无需新基础设施。
|
||||
|
||||
**协议(单个写入调用)**:
|
||||
1. **读 + 记戳**:读 subject body → `version_stamp = sha256(body) | mtime`
|
||||
2. **决策**:LLM 看候选池 → 决定 create / update / drop / split / additive-link;产 planned new_body
|
||||
3. **CAS 写入**:重读 body 比 version_stamp
|
||||
- **未变**:跑 E-1 / E-2 守恒校验 → 通过则 atomic write(write-temp + rename)→ done
|
||||
- **已变**:丢弃 planned new_body,带最新 body 重走 step 1
|
||||
4. **守恒校验失败**:走 `auto_dream_design.md` §1.5.5 既有重试路径(LLM 重试一次,二次失败拒写 + audit)
|
||||
5. **重做次数上限**:CAS-冲突重做最多 3 次;超出 → 跳过候选 + audit log(避免活锁)
|
||||
|
||||
**create 路径 race**:两个 G\* 都决定 `create digest/auth/jwt-rotation.md` → atomic create(`O_CREAT | O_EXCL`)只让一个赢;输者拿 EEXIST → 重走 step 1(此时大概率改判 update)。
|
||||
|
||||
**适用范围**(全部走同一套 CAS):同 ingest 内 N 个候选并发 / 跨 ingest job 并发 / 后台 split 与前台 G\* 命中同节点(split 同样走 CAS)/ auto-link 写回(`auto_link_design.md` §1.2)。
|
||||
|
||||
**不解决的**:高冲突 workload(同概念被反复 ingest)→ 重做上限触发后 audit;跨进程并发(多 reme 实例同 vault)→ 不在 M0,需 fs lock(M1+)。
|
||||
|
||||
---
|
||||
|
||||
## 6. split 时的 provenance 处理(已收敛)
|
||||
|
||||
**坍缩到 E-2 合计守恒** —— provenance 是 body 内联 wikilink(`auto_dream_design.md` §3.9),split 时跟其它 body 边完全同形:LLM 把 parent body 拆成 parent overview + N children,provenance wikilink 跟着对应内容段自然分配;机械层 outbound 合计守恒校验保证 `(parent_new ∪ ∪children_outbound) ⊇ parent_old`,旧 provenance 不可能丢。无需专门的 provenance 分配逻辑或"全部复制到 child / parent 保留全量"等特殊策略 —— LLM 按"哪个 child 谈到了哪段上游就带走哪条 provenance"自然处理。
|
||||
|
||||
---
|
||||
|
||||
## 7. G\* / split / auto-link L1 时序(已收敛)
|
||||
|
||||
时序由 §4 / §5 与 `auto_dream_design.md` §3.10 共同规定,这里给最小汇总:
|
||||
|
||||
- **G\* 调用本身同步** —— material 进来就走 G\* 决策(召回 + LLM 终判)+ CAS 写入(§5)
|
||||
- **D3 检测 inline** —— G\* / split 写完 body 顺手跑 token 阈值 + LLM 判离散度(§4),无 tick / batch / watcher
|
||||
- **split 异步** —— D3 触发后 enqueue split job 进 §5 CAS 队列,跟其它 ingest / split job FIFO 共享,异步消费;**不阻塞 G\* return**
|
||||
- **auto-link L1 异步** —— 写入成功后 enqueue auto-link L1 job(`auto_link_design.md` §1.4),与 split job 同 CAS 队列、FIFO 共享;不阻塞 G\* return
|
||||
|
||||
检测延迟 ≈ 0(inline);split 执行延迟 ≈ 队列等待时间(typically 数秒~数十秒);新建 / update 节点不必等 split 完成,体验连续。
|
||||
|
||||
---
|
||||
|
||||
## 8. maintainer 的人 / agent 后门(暂缓 — 非底层)
|
||||
|
||||
消费层 / SDK 接口问题,不影响底层机制。底层只需保证 split / rename / delete 等 op 走同一套 §5 CAS + 守恒校验链路:F-3 仍成立(maintainer 自动路径只做 split);merge / dissolve / re-edge 在底层**不存在**(无对应机械算子)。后门接口形态推迟到 SDK 阶段再定。
|
||||
|
||||
---
|
||||
|
||||
## 9. 与 dream 模型的引用关系
|
||||
|
||||
本文档复用 dream 定义的底层模型,所有具体规则在 `auto_dream_design.md` 中:
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| wikilink 基础语法(`[[path.md\|alias]]` / predicate) | `auto_dream_design.md` §1.4 |
|
||||
| 节点 / 边模型 | `auto_dream_design.md` §1.5 / §1.5.1 |
|
||||
| F-invariants(F-1..F-11) | `auto_dream_design.md` §1.5.4 |
|
||||
| 边守恒 E-1 / E-2 / E-3 | `auto_dream_design.md` §1.5.5 |
|
||||
| 路径即 ID / rename | `auto_dream_design.md` §1.3 |
|
||||
| anchor 不引入 | `auto_dream_design.md` §1.4 / §3.13 |
|
||||
| provenance 载体形态 | `auto_dream_design.md` §3.9 |
|
||||
| G\* 行为 | `auto_dream_design.md` §2.1 |
|
||||
|
||||
---
|
||||
|
||||
## 10. 下一步
|
||||
|
||||
1. **M split step 实现** —— D3 触发 → 候选 → split prompt → 写入 + E-2 守恒(§1 / §4 / §5)
|
||||
2. **D 检测信号实现清单**(D1 断链 / D3 写后 inline / D10 provenance 哪些已就绪 / 缺哪些)—— §2
|
||||
3. **D3 阈值配置**(`vault.yaml` 中 D3 token / 离散度阈值)—— §3
|
||||
4. **CAS 写入框架** —— per-path body version_stamp + CAS 写入 + EEXIST create race + 重做上限 + audit;对外暴露给 dream G\* / auto-link L1 复用 —— §5
|
||||
5. **后门 SDK 接口形态**(暂缓 M1+)—— §8
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
|
|
@ -4,11 +4,11 @@
|
|||
>
|
||||
> 配套阅读:
|
||||
> - `structure.md` §2.1-2.2(daily 层定位)/ §3.4(sync 动作语义)/ §7.1(synchronizer 模块)
|
||||
> - `auto_dream_design.md`:auto-memory 产物如何被 dream 消化(G\* 读 daily 作为入流之一)
|
||||
> - `auto_maintain_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
> - `auto_link_design.md`:auto-link 可反向扫 daily 事件,补充实体 wikilink(daily → digest)
|
||||
> - `auto_dream_design.md`:auto-memory 产物如何被 dream 消化(dream 读 daily 作为入流之一)
|
||||
> - `auto_consolidate_design.md`:digest 的组织端 / CAS 写入协议;auto-memory 不直接复用,但事件级"拆"与节点级 split 在概念上同构(都把过载粒度切小)
|
||||
> - `auto_cognition_design.md`:auto-cognition 三阶段顶层思想(写入 / 巩固 / 检索);daily 节点是 cognition 图视图的一部分(承载 `derived_from::` 反指),但不参与 Stage 2 巩固改造
|
||||
>
|
||||
> **四份分工**:reme 服务整体四份设计 —— **auto-memory(本文档)** / auto-dream / auto-maintain / auto-link。auto-memory 是入流端,把 agent 实时事件流切成 daily 事件原子;它的产物是 dream 消化的两路输入之一(另一路是 resource)。
|
||||
> **服务全景**:reme 服务两条主线 —— **auto-memory**(本文档,入流端 / daily 写入)与 **auto-cognition**(顶层思想:写入 = auto-dream,巩固 = auto-consolidate,检索 = auto-recall)。auto-memory 把 agent 实时事件流切成 daily 事件原子;它的产物是 dream(cognition Stage 1)消化的两路输入之一(另一路是 resource)。
|
||||
>
|
||||
> **核心立场**:auto-memory 是 `structure.md` §3.4 `sync` 动作的实现侧 —— 强调 **inline 实时**与**事件边界检测**。是不是改名 sync → auto-memory 留给上层文档对齐,本文档聚焦机制。
|
||||
|
||||
|
|
@ -29,7 +29,7 @@ agent 的对话与任务过程是连续事件流(用户回合、工具调用、
|
|||
|
||||
**显式排除**(不属于 auto-memory 职责):
|
||||
- ❌ 蒸馏 / 沉淀:那是 auto-dream(`auto_dream_design.md`)的事
|
||||
- ❌ 实体识别 / wikilink 自动补全:那是 auto-link(`auto_link_design.md`)的事
|
||||
- ❌ 实体识别 / wikilink 自动补全:cognition 三阶段不在写入后做"事后补 wikilink"(详 `auto_cognition_design.md` §1.1);所有 wikilink 由 dream 在写入瞬间产出
|
||||
- ❌ 改写 resource / digest:auto-memory 只写 daily(I-1 / I-3)
|
||||
|
||||
---
|
||||
|
|
@ -179,10 +179,10 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective
|
|||
| ← **notify** | 接收 notify payload 作为新 event cue;不强制响应,不强制 wikilink 引 |
|
||||
| ← **resource** | 只读(通过 wikilink 引);不写 |
|
||||
| → **daily** | **唯一写者**(I-2);写 event folder + 主索引 |
|
||||
| → **auto-dream** | dream 的 G\* 读 daily 作为入流(`auto_dream_design.md` §2.1 G1 scope);auto-memory 写完即对 dream 可见(走 L2 索引,有 eventual 窗口) |
|
||||
| → **auto-link** | auto-link 可反向扫 daily event,做实体识别 + wikilink 写回(`auto_link_design.md` §1.3)—— 与 auto-memory 写入不冲突(双方写不同字段段落 / CAS 协议保护)|
|
||||
| → **auto-dream** | dream 读 daily 作为入流(`auto_dream_design.md` §4.2 dream scope);auto-memory 写完即对 dream 可见(走 L2 索引,有 eventual 窗口) |
|
||||
| → **auto-cognition (三阶段)** | daily 节点是 cognition 图视图的一部分;dream(Stage 1)读 daily 作为入流;consolidate(Stage 2)只对 digest 节点跑 dups / community / decay,**不改 daily**;recall(Stage 3)三层并行召回时 daily 也参与命中 |
|
||||
|
||||
**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;dream / link 不写 daily 主路径,只通过 auto-link 走 §1.3 写回(read-only audit-then-write,CAS 保护)。
|
||||
**关键边界**:auto-memory 是 daily 写入端的**唯一**入口;cognition 三阶段没有任何子阶段会**事后改写 daily**(无写回路径)。daily 一旦由 auto-memory 写完,就只被读不被改(I-2 / I-3 仍守);后续 dream / consolidate / recall 都是只读消费。
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -194,4 +194,4 @@ INHERIT 行为细节(扫描窗口、predecessor 是否关闭、Plan/Objective
|
|||
4. **多 agent 隔离 schema**(M1+):若实际有并发 agent,确定 daily 子目录 / slug 命名约定
|
||||
5. **粗 / 细粒度 prompt 调参**:dogfooding 后看实际 event 数 / dream 消化效率,调 boundary prompt
|
||||
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_link_design.md` 共同作为契约依据。
|
||||
实现进入 `reme4/steps/jobs/` 与 `reme4/file_graph/` 时,本文档与 `auto_dream_design.md` / `auto_cognition_design.md` 共同作为契约依据。
|
||||
|
|
|
|||
323
docs4/auto_recall_design.md
Normal file
323
docs4/auto_recall_design.md
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
# auto-recall 设计(Stage 3 检索:信号融合 + 召回增强)
|
||||
|
||||
> 本文档:reme4 中 **auto-cognition 三阶段** 的 **Stage 3 — 检索阶段** 实现。覆盖 query 到来时如何把 vault 一等公民信号(wikilink 图 / frontmatter)与维护阶段产出信号(centrality / community / recency / archived)融合,生成最终召回。
|
||||
>
|
||||
> 配套阅读:
|
||||
> - `auto_cognition_design.md`:三阶段顶层思想(本文档是 Stage 3)
|
||||
> - `auto_dream_design.md`:Stage 1 写入 / 节点 + 边模型
|
||||
> - `auto_consolidate_design.md`:Stage 2 维护 —— **本文档消费它产出的所有 `meta/*.json`**
|
||||
> - `structure.md` §4(retrieve 三种问法)/ §7.4(为什么没有 retriever 模块)
|
||||
> - `reme4/steps/index/search.py` / `traverse.py`:现有原子实现
|
||||
>
|
||||
> **核心立场**:
|
||||
> - retrieve **不引入新 L4 模块**(`structure.md` ✗-15)—— 三种问法各自由 L3 原子工具(`list_step` / `search_step` / `traverse_step`)直接覆盖
|
||||
> - 本文档增强**集中在 `search_step` 内部**:把维护信号融入打分 / 排序 / 过滤;`traverse_step` 仅做小幅参数扩展
|
||||
> - retrieve **只读 vault,不写 body / 不写 frontmatter**;唯一写入是 `meta/access_log.json`(命中计数,供下次 recency 计算)
|
||||
|
||||
---
|
||||
|
||||
## 0. 问题陈述
|
||||
|
||||
`structure.md` §4 已规定 retrieve 三种问法(state / semantic / topological)正交分立(R-1)。本文档**只增强 semantic 问法**;state 问法已被 `list_step` 覆盖,topological 问法已被 `traverse_step` 覆盖。
|
||||
|
||||
semantic 问法当前在 `reme4/steps/index/search.py` 实现:
|
||||
|
||||
| 已就绪 | 缺口 |
|
||||
|---|---|
|
||||
| ✅ vector + keyword 并行召回 | ❌ 节点中心性加权(高权威节点不被 boost) |
|
||||
| ✅ RRF fusion(vector_weight=0.7) | ❌ 同社区 boost(`meta/communities.json` 未消费) |
|
||||
| ✅ 一跳 expand_links(向前向后,max=10) | ❌ 时效衰减 / 冷藏过滤(`meta/access_log.json`、`meta/archived.json` 未消费) |
|
||||
| ✅ min_score 过滤 + limit 截断 | ❌ 同 file 多 chunk 冗余(top-K 可全来自同节点) |
|
||||
| ✅ chunk-level 命中(start_line / end_line) | ❌ 节点级 surface(frontmatter `name + description` 未与 chunk 命中合并展示) |
|
||||
| ✅ 二跳 traverse 作为独立工具 | ❌ search 内 multi-hop expand(只一跳,跨术语关系到不了) |
|
||||
| | ❌ query rewrite / multi-query(单一表达式漏召) |
|
||||
|
||||
**本文档的工作 = 设计这些缺口怎么填**,在 `search_step` / `traverse_step` 现有形态上增量。
|
||||
|
||||
---
|
||||
|
||||
## 1. 三种问法分立(继承 R-1)
|
||||
|
||||
```
|
||||
┌─────────────┐ state 问 ──────► list_step + frontmatter filter
|
||||
│ agent │ semantic 问 ──► search_step (本文档主要增强)
|
||||
└─────────────┘ topological 问 ► traverse_step (小幅参数扩展)
|
||||
```
|
||||
|
||||
| 问法 | 原子工具 | 本文档涉及 | 备注 |
|
||||
|---|---|---|---|
|
||||
| **state** | `list_step` / `daily_list_step` / `frontmatter_read_step` | 不涉及 | frontmatter 过滤无需维护信号 |
|
||||
| **semantic** | `search_step` | **主战场**(§3-§7) | RRF fusion + 信号加权 + multi-hop + query rewrite |
|
||||
| **topological** | `traverse_step` | 小幅(§8) | 起点选择可借助维护信号 |
|
||||
|
||||
**关键约束**(继承 `structure.md` ✗-8):**绝不合并三种问法成单一 read verb**。本文档增强 search_step,但不把 list / traverse 揉进 search;agent 按需各自调用。
|
||||
|
||||
---
|
||||
|
||||
## 2. 维护信号契约消费总览
|
||||
|
||||
`auto_consolidate_design.md` §11 列出维护产出。retrieve 端按以下方式读:
|
||||
|
||||
| 信号 | 来源 | 加载时机 | 缺失行为(降级) |
|
||||
|---|---|---|---|
|
||||
| **centrality** | `file_graph` 反向索引(实时) | search_step init 时引用 file_store | 总在线(file_graph 是核心组件) |
|
||||
| **community** | `meta/communities.json` | search_step 启动 lazy load(LRU 缓存,文件 mtime 失效) | 缺失 → 不做同社区 boost |
|
||||
| **recency** | `meta/access_log.json` | 同上 | 缺失 → recency_factor = 1.0 |
|
||||
| **archived** | `meta/archived.json` | 同上 | 缺失 → 不过滤,所有节点参与 |
|
||||
| **wikilink 图** | vault 自身(file_graph) | 实时 | 总在线 |
|
||||
| **frontmatter** | vault 自身(`name` / `description`) | chunk 已带 metadata | 总在线 |
|
||||
|
||||
**version 校验**:`meta/*.json` 加载时检查 `version` 字段,与本文档约定的 schema 版本不匹配 → 走"该信号缺失"降级,日志告警(不崩)。
|
||||
|
||||
**新鲜度**:每个信号文件的 `computed_at` 暴露给调用者(metadata 中带 `signals_freshness`),调用方知道当前权重基于多久前的快照。超过阈值(默认 14 days)→ logger.warning + 仍使用(避免维护偶尔失效就拒绝服务)。
|
||||
|
||||
---
|
||||
|
||||
## 3. semantic 问法增强:打分公式
|
||||
|
||||
**目标**:把维护信号融入 fused chunk 的最终 score,让排序兼顾"文本相关 + 节点权威 + 同社区 + 时效"。
|
||||
|
||||
### 3.1 当前打分(基线)
|
||||
|
||||
```
|
||||
score = RRF_fused(vector_rank, keyword_rank, vector_weight=0.7)
|
||||
```
|
||||
|
||||
仅文本相似度。
|
||||
|
||||
### 3.2 新打分公式
|
||||
|
||||
```
|
||||
final_score = base_score
|
||||
× centrality_factor(path)
|
||||
× community_factor(path, query_seed_paths)
|
||||
× recency_factor(path)
|
||||
```
|
||||
|
||||
| 因子 | 公式 | 默认参数 | 来源 |
|
||||
|---|---|---|---|
|
||||
| **base_score** | RRF 融合分(现状) | vector_weight=0.7 | search.py |
|
||||
| **centrality_factor** | `1 + α · log(1 + inbound_count)` | α = 0.15 | file_graph 实时 |
|
||||
| **community_factor** | 同 community 命中节点 → ×β,否则 1.0 | β = 1.20 | `meta/communities.json` |
|
||||
| **recency_factor** | `exp(-Δt / τ)`,Δt = 距 last_hit_or_update | τ = 60 days | `meta/access_log.json` |
|
||||
|
||||
**为什么乘法而非加法**:
|
||||
- 各因子量级不同(base_score ≤ 0.02,centrality 与 query 无关),加法需大量 normalization;乘法天然处理量级差
|
||||
- 任一因子接近 0(极冷藏 / 极孤立)→ 整体压低,符合"弱信号一票否决"直觉
|
||||
- 默认 α/β/τ 让 factor 落在 [0.5, 2.0] 区间,不会让 base_score 完全失声
|
||||
|
||||
**已排除**:LLM rerank。它是 query-time 多调一次 LLM,成本高,M0 不引入;留 M1+ 视 dogfooding 决定。
|
||||
|
||||
### 3.3 query_seed_paths 的角色
|
||||
|
||||
community_factor 需要"query 主关注的节点是哪些"才能判断同/异社区。做法:
|
||||
1. RRF 融合后取 top-N(N=3)的 fused chunk 的 path 作 seed
|
||||
2. 后续每个候选 chunk 的 path → 查它和任一 seed 是否同社区 → boost
|
||||
3. 不需要 query 自身被映射到 community(query 是字符串,不在图里)
|
||||
|
||||
**边界**:N=3 是经验起点;N 太大会让"同社区"几乎等于"全召回"失去区分度。dogfooding 后调。
|
||||
|
||||
---
|
||||
|
||||
## 4. semantic 增强:节点级合并(unique_paths)
|
||||
|
||||
**问题(gap 5)**:fused 列表里 top-5 可能是同 file 的 5 个 chunk,信噪比退化。
|
||||
|
||||
**当前**:`expand_links` 已用 `unique_paths = list(dict.fromkeys(c.path for c in fused))`,但 fused 本身没去重,limit=5 仍可全是同节点。
|
||||
|
||||
**新方案**(节点级 dedupe + 节点级 surface):
|
||||
|
||||
```
|
||||
fused (chunk-level) → group by path → 每组保留 top_chunks_per_path 个
|
||||
→ 每组追加节点 frontmatter (name + description) 作"节点级 surface"
|
||||
→ 再按节点 best_score 排序 → limit
|
||||
```
|
||||
|
||||
| 参数 | 默认 | 含义 |
|
||||
|---|---|---|
|
||||
| `top_chunks_per_path` | 2 | 同节点最多保留多少 chunk |
|
||||
| `surface_node` | true | 是否在每组前追加 frontmatter `name + description` |
|
||||
|
||||
**为什么**:
|
||||
- 节点是 retrieve 的语义单位(`auto_dream_design.md` §2 路径即 ID),chunk 只是"展示窗口"
|
||||
- frontmatter 是节点级摘要(name + description)—— 已是 dream 写入时认证过的信号,不召它浪费
|
||||
- 同节点多 chunk 时,frontmatter + top-2 chunk 比 5 个 chunk 信息密度高
|
||||
|
||||
### 4.1 答案展示形态
|
||||
|
||||
```
|
||||
========== digest/auth/jwt-rotation.md ==========
|
||||
[node] JWT Key Rotation
|
||||
Process for rotating JWT signing keys without downtime.
|
||||
[score=0.0241 centrality=2.1 community=1.2 recency=0.91]
|
||||
|
||||
---------- chunk @5-23 ----------
|
||||
<chunk text>
|
||||
|
||||
---------- chunk @45-60 ----------
|
||||
<chunk text>
|
||||
|
||||
[expansion] 1 inbound, 2 outbound (...)
|
||||
```
|
||||
|
||||
**对照旧形态**:每个 chunk 独立成块,无节点级 surface,scores 散在 chunk 头。新形态以**节点为视觉单位**,人 / agent 看到的第一眼是"哪个节点中了",而非"哪段文字中了"。
|
||||
|
||||
---
|
||||
|
||||
## 5. semantic 增强:multi-hop expand
|
||||
|
||||
**问题(gap 4)**:当前 expand_links 只展一跳,跨术语关系("分布式锁" → 一跳到"租约机制",再一跳才到"心跳协议")到不了。
|
||||
|
||||
**新方案**:expand_links 支持 `depth` 参数;默认仍 1(保守),agent / 配置可调到 2。
|
||||
|
||||
| 参数 | 默认 | 限制 |
|
||||
|---|---|---|
|
||||
| `expand_depth` | 1 | 最大 3(避免组合爆炸) |
|
||||
| `max_links_per_direction` | 10(现状)| 每跳每方向上限,深度不展开时限到当跳总数 |
|
||||
| `expand_path_budget` | 30 | 总扩展节点数硬上限,优先深度优先(深度浅但条数少) |
|
||||
|
||||
**为什么默认仍 1**:
|
||||
- 二跳延迟不可忽略(N × 10 × 10 = 100 候选 IO)
|
||||
- agent 需要"再深一层"时显式调 `traverse_step(depth=2)` —— 三种问法分立(R-1)
|
||||
- 默认深拉会让"语义召回"变成"图召回",违背 R-1
|
||||
|
||||
**何时调 2**:dogfooding 发现 vault 节点平均出度低 / 跨术语关系频繁 → 调到 2(改 search_step 配置,不改协议)。
|
||||
|
||||
---
|
||||
|
||||
## 6. semantic 增强:query rewrite / multi-query
|
||||
|
||||
**问题(gap 6)**:用户 query "JWT 怎么轮换" 可能错过 body 写"密钥定期更换"的节点(术语不同)。
|
||||
|
||||
**方案矩阵**:
|
||||
|
||||
| 方案 | 成本 | 效果 |
|
||||
|---|---|---|
|
||||
| **(a) 不做** | 0 | 漏召部分跨术语 |
|
||||
| **(b) embedding 多 query**(用同 LLM 生成 N 个表述) | LLM 调用 1 次(query → N 表述)+ N 次 vector_search | 中等 |
|
||||
| **(c) BM25 同义词扩展**(用静态词表 / 嵌入式词表) | 0(若有词表) | 弱(中文场景词表缺) |
|
||||
| **(d) HyDE**(LLM 生成假设答案 → 嵌入这个答案而非 query) | LLM 1 次 | 高,文献证实 |
|
||||
|
||||
**首版决策**:**(a) 不做**。理由:
|
||||
- vault 本身规模 M0 不大,推断增加召回但增 LLM cost 不划算
|
||||
- 维护阶段的 community 聚类已部分弥补"跨术语关系"(同社区 boost)
|
||||
- 真要做,优先 (d) HyDE,延 M1+ 再启,实施只需加一层 query 预处理
|
||||
|
||||
**契约预留**:search_step kwargs 加 `query_rewrite: str | None`(默认 None;非 None 则用此重写代替原 query 做 vector_search,keyword_search 仍用原 query)。SDK 层可调用 LLM 生成重写后传入,reme 核心不强加 LLM 依赖。
|
||||
|
||||
---
|
||||
|
||||
## 7. semantic 增强:archived 过滤
|
||||
|
||||
**问题**:长期未访问的旧节点应该默认排除。
|
||||
|
||||
**方案**:search_step kwargs 加 `include_archived: bool`,默认 false。
|
||||
|
||||
```
|
||||
fused → drop where path in archived_set → 后续打分 / unique_paths
|
||||
```
|
||||
|
||||
**何时绕过**:
|
||||
- agent 显式 `include_archived=true`(找历史 / debug)
|
||||
- query 命中节点本身在 archived → boost 推回(冷节点突然被命中,说明不是真冷)
|
||||
- **首版不做**,过滤即过滤;如有需要,M1+ 加"intent override"机制
|
||||
|
||||
**冷启动**(`meta/archived.json` 缺失)→ 不过滤,等同 `include_archived=true`。
|
||||
|
||||
---
|
||||
|
||||
## 8. topological 问法的小增强
|
||||
|
||||
`traverse_step` 当前完整:BFS / 多 seed / direction / depth / per-edge 输出。本文档不重构,仅:
|
||||
|
||||
### 8.1 起点选择借助维护信号(可选 hint)
|
||||
|
||||
agent 调用 traverse 时往往不知道"哪个节点是该主题的中心";维护阶段产出的 centrality 可作 hint:
|
||||
|
||||
| 用例 | 做法 |
|
||||
|---|---|
|
||||
| traverse 给定 seed | 不变,直接 BFS |
|
||||
| traverse 给定主题字符串(SDK 上层语法糖) | 先 search_step 找 top-1 → 用其作 seed → traverse depth=2 |
|
||||
|
||||
**位置**:这个组合在 SDK 上层做,不进 traverse_step;reme 核心保留 traverse 原子形态。
|
||||
|
||||
### 8.2 traverse 输出消费 archived
|
||||
|
||||
traverse_step 当前不知道 archived 信号。改造:加 `exclude_archived: bool` kwarg 默认 false(traverse 默认不过滤,因为它是图问法,过滤会破坏图视角)。SDK / agent 可显式开启。
|
||||
|
||||
---
|
||||
|
||||
## 9. retrieve 写访问日志(唯一对外写入)
|
||||
|
||||
**问题**:`meta/access_log.json` 的 `last_read` / `last_hit_count_30d` 谁写?
|
||||
|
||||
**约定**:retrieve 命中节点 → 异步 append 到访问日志缓冲区;由 maintain daily batch 聚合写入 `meta/access_log.json`。
|
||||
|
||||
| 路径 | 实现 |
|
||||
|---|---|
|
||||
| **同步写**(每 query) | retrieve 把命中 path 写入内存 ring buffer(进程级)|
|
||||
| **异步落盘** | 进程退出 / 维护 daily batch / 周期 flush(默认 10 min)|
|
||||
| **聚合** | maintain 在 daily access_log 重算时:读 ring buffer + 上一份 access_log → 合并写新版 |
|
||||
|
||||
**幂等**:同 query 多次重读同节点不应放大 last_hit_count;ring buffer 按 (path, day) 去重,每天每节点最多记一次"被读"。
|
||||
|
||||
**降级**:ring buffer 写失败 / flush 失败 → 不影响 retrieve 返回,只是日志少一条;recency 信号略迟。
|
||||
|
||||
---
|
||||
|
||||
## 10. 不变量 / 边界
|
||||
|
||||
| # | 约束 | 含义 |
|
||||
|---|---|---|
|
||||
| **R-1**(继承)| 三种问法分立 | 不合并 list / search / traverse 成单一 verb |
|
||||
| **R-2**(继承)| 默认 `digest > daily > resource`,可覆盖 | search_step 通过 `search_filter` 支持限层 |
|
||||
| **R-3**(继承)| 拓扑问与层无关 | traverse 跨三层(I-4) |
|
||||
| **R-4**(继承)| Provenance 默认 lazy | retrieve 不自动 traverse(R-4);expand_links 是性能优化非语义展开 |
|
||||
| **Re-1**(本文档)| retrieve 不引入 L4 模块 | 增强限定在原子 step 内部 |
|
||||
| **Re-2**(本文档)| retrieve 只读 vault | 不改 body / frontmatter / 文件位置 |
|
||||
| **Re-3**(本文档)| retrieve 唯一对外写入是 `meta/access_log.json` | 通过 ring buffer + maintain 聚合,不直接写 |
|
||||
| **Re-4**(本文档)| 任一维护信号缺失 → 降级不崩 | `meta/*.json` 缺 → 跳过对应因子,系统始终可用 |
|
||||
| **Re-5**(本文档)| version 不兼容 → 降级 + warning | 不阻断 retrieve |
|
||||
|
||||
---
|
||||
|
||||
## 11. 与其它文档的引用关系
|
||||
|
||||
| 引用 | 来源 |
|
||||
|---|---|
|
||||
| 三种问法 / R-1..R-5 | `structure.md` §4 |
|
||||
| 没有 retriever 模块 | `structure.md` §7.4 |
|
||||
| 节点 / 边 / wikilink 模型 | `auto_dream_design.md` §2 / §3 |
|
||||
| 维护信号契约 | `auto_consolidate_design.md` §11 |
|
||||
| centrality / community / recency / archived 输出 | `auto_consolidate_design.md` §3-§5 |
|
||||
| 路径即 ID | `auto_dream_design.md` §2 |
|
||||
|
||||
---
|
||||
|
||||
## 12. 下一步
|
||||
|
||||
实现进入 `reme4/steps/index/` 时,本文档与 `auto_cognition_design.md`(顶层)/ `auto_dream_design.md` / `auto_consolidate_design.md` 共同作为契约依据。
|
||||
|
||||
**search_step 增强(§3-§7)**:
|
||||
- ⏳ **打分公式**:加 centrality_factor / community_factor / recency_factor;config 化 α / β / τ(§3)
|
||||
- ⏳ **节点级合并 + surface**:group-by-path + frontmatter surface + top_chunks_per_path(§4)
|
||||
- ⏳ **multi-hop expand**:`expand_links` 支持 depth 参数,加 `expand_path_budget` 硬上限(§5)
|
||||
- ⏳ **query_rewrite kwarg**:契约预留,reme 核心不强加 LLM(§6)
|
||||
- ⏳ **archived 过滤**:`include_archived` kwarg,默认 false(§7)
|
||||
|
||||
**traverse_step 增强(§8)**:
|
||||
- ⏳ **`exclude_archived` kwarg**(默认 false)
|
||||
|
||||
**信号加载基础设施(§2)**:
|
||||
- ⏳ **`meta/*.json` lazy loader + LRU 缓存 + mtime 失效**
|
||||
- ⏳ **version 校验 + 降级路径 + warning logger**
|
||||
- ⏳ **signals_freshness metadata 暴露**
|
||||
|
||||
**access log 写入路径(§9)**:
|
||||
- ⏳ **进程级 ring buffer**(命中 path 异步 append)
|
||||
- ⏳ **周期 flush + (path, day) 幂等**
|
||||
- ⏳ **maintain daily 聚合接口**(读 ring → 合并旧 access_log → 写新版)
|
||||
|
||||
**性能与回归**:
|
||||
- ⏳ **基准测试**:打分公式启用前后的 召回 P@5 / MRR(用合成 vault + ground-truth query)
|
||||
- ⏳ **延迟监控**:维护信号读取 + multi-hop expand 的 p50 / p95
|
||||
|
|
@ -2,5 +2,6 @@
|
|||
LLM_API_KEY=sk-xxxx
|
||||
LLM_BASE_URL=https://xxxx/v1
|
||||
LLM_MODEL_NAME=xxxx
|
||||
LLM_BACKEND=XXX
|
||||
#EMBEDDING_API_KEY=sk-xxxx
|
||||
#EMBEDDING_BASE_URL=https://xxxx/v1
|
||||
|
|
@ -5,6 +5,7 @@ vault_dir: .reme
|
|||
daily_dir: daily
|
||||
digest_dir: digest
|
||||
resource_dir: ""
|
||||
# language: zh
|
||||
|
||||
jobs:
|
||||
update_store_index_loop:
|
||||
|
|
@ -368,6 +369,41 @@ jobs:
|
|||
steps:
|
||||
- backend: edit_step
|
||||
|
||||
dream:
|
||||
backend: base
|
||||
description: "Dream: lift atomic units from one daily/resource file into digest/ (LLM)."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
description: "vault-relative path of one daily-event note or resource file"
|
||||
hint:
|
||||
type: string
|
||||
description: "caller guidance to the dreamer LLM"
|
||||
default: ""
|
||||
required:
|
||||
- path
|
||||
steps:
|
||||
- backend: dreamer_step
|
||||
|
||||
auto-dream:
|
||||
backend: base
|
||||
description: "Auto-dream: scan today's day-index <daily_dir>/<today>.md and session notes under <daily_dir>/<today>/*.md — run dream on each (Phase 1 extract+classify, Phase 2 per-bucket integrate)."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
|
||||
default: ""
|
||||
hint:
|
||||
type: string
|
||||
description: "caller guidance passed through to each per-file dream"
|
||||
default: ""
|
||||
steps:
|
||||
- backend: cron_dreamer_step
|
||||
|
||||
auto_memory:
|
||||
backend: base
|
||||
description: "Auto-memory: record conversation facts into a daily note"
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from .common.llm_demo import LLMDemoStep
|
|||
from .common.stream_demo import StreamDemoStep1, StreamDemoStep2
|
||||
from .common.version import VersionStep
|
||||
from .evolve.auto_memory import AutoMemoryStep
|
||||
from .evolve.auto_dream import CronDreamer, Dreamer
|
||||
from .file_io.daily_create import DailyCreateStep
|
||||
from .file_io.daily_list import DailyListStep
|
||||
from .file_io.daily_reindex import DailyReindexStep
|
||||
|
|
@ -71,6 +72,9 @@ __all__ = [
|
|||
"UpdateCatalogStep",
|
||||
"UpdateIndexStep",
|
||||
"WatchChangesStep",
|
||||
# evolve (dream)
|
||||
"CronDreamer",
|
||||
"Dreamer",
|
||||
# transfer
|
||||
"DownloadStep",
|
||||
"IngestStep",
|
||||
|
|
|
|||
|
|
@ -132,8 +132,13 @@ class BaseStep(ComponentMixin, ABC):
|
|||
self.context: RuntimeContext | None = None
|
||||
|
||||
# Load class-level prompts first, then overlay caller-provided overrides.
|
||||
# Walk MRO in reverse so most-derived class wins; subclasses without their
|
||||
# own YAML inherit prompts from their parent (e.g. CronDreamer inherits
|
||||
# dreamer.yaml from Dreamer).
|
||||
self.prompt = PromptHandler(language=self.language)
|
||||
self.prompt.load_prompt_by_class(self.__class__).load_prompt_dict(prompt_dict)
|
||||
for cls in reversed(self.__class__.__mro__):
|
||||
self.prompt.load_prompt_by_class(cls)
|
||||
self.prompt.load_prompt_dict(prompt_dict)
|
||||
|
||||
# ----- Component references (resolved lazily on first access) ----------
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Demo steps for smoke-testing the application stack."""
|
||||
"""Demo steps for integration-testing the application stack."""
|
||||
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
|
|
|
|||
614
reme4/steps/evolve/auto_dream.py
Normal file
614
reme4/steps/evolve/auto_dream.py
Normal file
|
|
@ -0,0 +1,614 @@
|
|||
"""Dreamer — auto-dream's create_or_update step.
|
||||
|
||||
Reads one daily-event note or resource file at the given vault-relative
|
||||
``path``, identifies the ABSTRACTIONS the material teaches in Phase 1
|
||||
(each tagged with one of the three buckets), then in Phase 2 makes
|
||||
ONE cognitive write decision (CREATE or one of the three UPDATE
|
||||
flavors: CORROBORATE / REFINE / CORRECT) per abstraction using a
|
||||
**bucket-specific** integrate prompt.
|
||||
|
||||
**Digest is the abstract memory layer** — raw details stay in the
|
||||
material; digest holds the principle, pattern, or precedent worth
|
||||
recalling once the specifics fade. Provenance wikilinks
|
||||
(``derived_from::``) let readers drill back down to the source.
|
||||
|
||||
Pipeline (external loop in Python, two distinct ReAct agent invocations,
|
||||
**light Phase 1 / heavy Phase 2**):
|
||||
|
||||
execute():
|
||||
units, _ = _extract(material_blob) # 1× ReAct: identify abstractions
|
||||
# agent emits ExtractedUnits
|
||||
# ({units: [{name, bucket, summary}, ...]})
|
||||
for unit in units: # Python loop, K iterations
|
||||
_integrate_unit(unit) # 1× ReAct per abstraction, dispatched
|
||||
# to integrate_system_prompt_<bucket>;
|
||||
# recalls cross-bucket, decides write,
|
||||
# uses canonical write/edit/frontmatter_update tools.
|
||||
|
||||
The bucket vocabulary is hard-coded (:data:`BUCKETS`) — three buckets,
|
||||
each with a dedicated Phase 2 prompt:
|
||||
|
||||
* ``procedure`` — how-to-do-X: steps, methods, recipes, workflows.
|
||||
* ``personal`` — user/team specific: identity, preferences,
|
||||
conventions, things they avoid.
|
||||
* ``wiki`` — general knowledge: definitions, principles,
|
||||
observations, decisions-as-precedent. Default catch-all.
|
||||
|
||||
There is no SKIP outcome in Phase 2: Phase 1 is the gate for "not
|
||||
worth memorizing"; anything reaching Phase 2 warrants a write.
|
||||
|
||||
Phase 2 uses the **canonical** ``write`` / ``edit`` jobs (no
|
||||
constrained variants). Bucket placement and edge conservation are
|
||||
prompt-level discipline; the tools themselves perform no path-shape
|
||||
or conservation validation.
|
||||
|
||||
Invocation form (CLI / MCP):
|
||||
reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md
|
||||
reme dream path=resource/2026-05-28/spec.pdf hint="focus on auth"
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import zoneinfo
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from agentscope.message import Msg
|
||||
from agentscope.tool import Toolkit
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ._evolve import FlexReActAgent
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
|
||||
|
||||
# Hard-coded bucket vocabulary. Phase 1 classifies each sub-unit into
|
||||
# one of these; Phase 2 dispatches to the bucket-specific prompt.
|
||||
# Order matters for prompt rendering — keep procedure/personal/wiki.
|
||||
BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki")
|
||||
|
||||
# Bucket = Literal of BUCKETS. Pydantic Literal must be a static type;
|
||||
# update both BUCKETS and Bucket together if the vocabulary changes.
|
||||
Bucket = Literal["procedure", "personal", "wiki"]
|
||||
|
||||
|
||||
_EXTRACT_TOOLS: tuple[str, ...] = ("read",)
|
||||
|
||||
_INTEGRATE_TOOLS: tuple[str, ...] = (
|
||||
# read
|
||||
"search",
|
||||
"traverse",
|
||||
"read",
|
||||
"frontmatter_read",
|
||||
# write
|
||||
"write",
|
||||
"edit",
|
||||
"frontmatter_update",
|
||||
)
|
||||
|
||||
|
||||
def _pack_material(file_store, path: str) -> str:
|
||||
"""Render one daily-event note or resource file into a prompt block."""
|
||||
try:
|
||||
absolute = (Path(file_store.vault_path or ".") / path).resolve()
|
||||
except Exception as e:
|
||||
return f"### {path}\n(error resolving path: {type(e).__name__}: {e})\n"
|
||||
|
||||
if not absolute.is_file():
|
||||
return f"### {path}\n(file not found)\n"
|
||||
|
||||
try:
|
||||
return f"### {path}\n{absolute.read_text(encoding='utf-8')}\n"
|
||||
except Exception as e:
|
||||
return f"### {path}\n(error reading: {type(e).__name__}: {e})\n"
|
||||
|
||||
|
||||
class MemoryUnit(BaseModel):
|
||||
"""One memory sub-unit identified by Phase 1's structured output."""
|
||||
|
||||
name: str = Field(
|
||||
description=(
|
||||
"Short kebab-case identifier for the abstraction "
|
||||
"(e.g. 'jwt-rotation-decision', 'pr-size-pref'). "
|
||||
"Agent-internal handle — NOT the eventual digest slug; "
|
||||
"Phase 2 picks the actual filing path."
|
||||
),
|
||||
)
|
||||
bucket: Bucket = Field(
|
||||
description=(
|
||||
"Which bucket this abstraction belongs in — Phase 2 dispatches "
|
||||
"to a bucket-specific prompt based on this. Pick exactly one: "
|
||||
"`procedure` (how-to-do-X — steps, methods, recipes, workflows), "
|
||||
"`personal` (user/team-specific — identity, preferences, "
|
||||
"conventions, things they avoid), `wiki` (general knowledge — "
|
||||
"definitions, principles, observations, decisions-as-precedent; "
|
||||
"default catch-all when nothing else fits)."
|
||||
),
|
||||
)
|
||||
summary: str = Field(
|
||||
description=(
|
||||
"1-2 sentences naming the abstraction AND pointing at where "
|
||||
"in the material the supporting evidence lives "
|
||||
"(e.g. 'short-credential compliance drives auth cadence; "
|
||||
"illustrated by the 30→24h decision in the 'Decision' section "
|
||||
"+ the SOC2 CC6.1 criticism in the 'Observation' section')."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class ExtractedUnits(BaseModel):
|
||||
"""Structured output emitted by Phase 1's extract agent."""
|
||||
|
||||
units: list[MemoryUnit] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"Memory sub-units identified in the material — orthogonal "
|
||||
"abstractions (principles / patterns / precedents) worth "
|
||||
"lifting into long-term memory. Each is tagged with its "
|
||||
"bucket. Empty list = nothing worth lifting (Phase 2 is skipped)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _render_outcome_line(unit_name: str, bucket: str, o: "IntegrateOutcome") -> str:
|
||||
"""Format one IntegrateOutcome as a one-line summary entry."""
|
||||
body = f"{o.action} {o.target_path}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
return f"[{unit_name}/{bucket}] {body}"
|
||||
|
||||
|
||||
class IntegrateOutcome(BaseModel):
|
||||
"""Structured outcome reported by Phase 2 for one sub-unit."""
|
||||
|
||||
action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field(
|
||||
description=(
|
||||
"Outcome of the write decision for this sub-unit. Phase 1 already "
|
||||
"filtered out non-abstractions, so every sub-unit reaching you "
|
||||
"warrants a write — pick the matching fine-grained action: "
|
||||
"`CREATE` — brand-new digest node (recall returned no node "
|
||||
"covering this abstraction); even thin first-encounter seeds go "
|
||||
"here, they grow via CORROBORATE / REFINE on later passes. "
|
||||
"`CORROBORATE` (most common when a covering node exists) — "
|
||||
"provenance append + optional wording strengthening; the "
|
||||
"abstraction already covers this material. `REFINE` — covering "
|
||||
"node exists but the material reveals nuance, scope, or edge "
|
||||
"cases the abstraction under-specified. `CORRECT` — covering "
|
||||
"node exists but the material contradicts it; tighten the "
|
||||
"abstraction or annotate the contradiction inline."
|
||||
),
|
||||
)
|
||||
target_path: str = Field(
|
||||
description=("The digest path you wrote to — must match what your `write` / " "`edit` call(s) targeted."),
|
||||
)
|
||||
note: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Optional ONE short line, ≤ 200 chars, no newlines, summarizing "
|
||||
"what landed (e.g. 'extended scope to also cover X'). Do NOT "
|
||||
"dump recall summaries, search results, internal reasoning, or "
|
||||
"transcripts here — those belong in the ReAct trace, not the "
|
||||
"outcome note."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class DreamResult(BaseModel):
|
||||
"""Outcome of one dreamer invocation.
|
||||
|
||||
Per-tool audit lives in the toolkit layer (not exposed back to the
|
||||
orchestrator). Structured outcome here is the input path the call
|
||||
processed, the memory sub-units the agent declared in Phase 1, and
|
||||
what got created / updated in Phase 2.
|
||||
"""
|
||||
|
||||
used_llm: bool = False
|
||||
skipped: bool = False
|
||||
path: str = ""
|
||||
units: list[dict] = Field(default_factory=list)
|
||||
nodes_created: list[str] = Field(default_factory=list)
|
||||
nodes_updated: list[str] = Field(default_factory=list)
|
||||
summary: str = ""
|
||||
error: str = ""
|
||||
|
||||
|
||||
@R.register("dreamer_step")
|
||||
class Dreamer(BaseStep):
|
||||
"""auto-dream create_or_update step.
|
||||
|
||||
Inputs (from RuntimeContext):
|
||||
path (str, required): vault-relative path of one
|
||||
daily-event note or resource file to dream over. Pass
|
||||
empty string to no-op.
|
||||
hint (str, optional): caller guidance to the LLM
|
||||
(e.g. "focus on the auth-related decisions").
|
||||
|
||||
Output (written to context.response.answer):
|
||||
``DreamResult`` JSON in ``metadata``; LLM summary in ``answer``.
|
||||
|
||||
CLI / MCP form:
|
||||
reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
toolkit: Toolkit | None = None,
|
||||
console_enabled: bool = False,
|
||||
timezone: str | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
self.toolkit = toolkit
|
||||
self.console_enabled = console_enabled
|
||||
self.timezone = timezone
|
||||
|
||||
def _now(self) -> datetime.datetime:
|
||||
if self.timezone:
|
||||
try:
|
||||
return datetime.datetime.now(zoneinfo.ZoneInfo(self.timezone))
|
||||
except Exception as e:
|
||||
self.logger.error(f"Invalid timezone: {self.timezone}, error={e}")
|
||||
return datetime.datetime.now()
|
||||
|
||||
def _vault_dir(self) -> Path:
|
||||
vr = getattr(self.file_store, "vault_path", None)
|
||||
return Path(vr).resolve() if vr else Path.cwd().resolve()
|
||||
|
||||
def _llm_available(self) -> bool:
|
||||
try:
|
||||
return self.as_llm is not None
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _build_extract_toolkit(self) -> Toolkit:
|
||||
"""Read-only toolkit for the extract agent. Sub-units come back via
|
||||
:class:`ExtractedUnits` structured output, not via a tool call."""
|
||||
toolkit = Toolkit()
|
||||
for job_name in _EXTRACT_TOOLS:
|
||||
self.add_as_tool(toolkit, job_name)
|
||||
return toolkit
|
||||
|
||||
def _build_integrate_toolkit(self) -> Toolkit:
|
||||
"""Full read + canonical write/edit/frontmatter_update toolkit for
|
||||
the integrate agent. All tools are registered via :meth:`add_as_tool`
|
||||
— same as every other step in this codebase. Outcome tracking is
|
||||
driven by the agent's :class:`IntegrateOutcome` structured emission,
|
||||
not by per-tool callbacks."""
|
||||
toolkit = self.toolkit or Toolkit()
|
||||
for job_name in _INTEGRATE_TOOLS:
|
||||
self.add_as_tool(toolkit, job_name)
|
||||
return toolkit
|
||||
|
||||
async def _extract(self, material_blob: str, hint: str, vault_dir: Path) -> tuple[list[dict], str]:
|
||||
"""Phase 1: one ReAct invocation — read material + emit ExtractedUnits.
|
||||
|
||||
Returns ``(units, llm_summary)`` where ``units`` is the cleaned
|
||||
sub-unit list (each entry has ``name`` / ``bucket`` / ``summary``)
|
||||
and ``llm_summary`` is whatever free-form text the agent produced
|
||||
alongside its structured emission.
|
||||
"""
|
||||
toolkit = self._build_extract_toolkit()
|
||||
agent = FlexReActAgent(
|
||||
name="reme_dreamer_extract",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
"extract_system_prompt",
|
||||
vault_dir=str(vault_dir),
|
||||
buckets=", ".join(BUCKETS),
|
||||
),
|
||||
formatter=self.as_llm_formatter,
|
||||
toolkit=toolkit,
|
||||
)
|
||||
agent.set_console_output_enabled(self.console_enabled)
|
||||
user_message = self.prompt_format(
|
||||
"extract_user_message",
|
||||
today=self._now().strftime("%Y-%m-%d"),
|
||||
hint=hint or "(none)",
|
||||
material_blob=material_blob,
|
||||
)
|
||||
msg = await agent.reply(
|
||||
Msg(name="reme", role="user", content=user_message),
|
||||
structured_model=ExtractedUnits,
|
||||
)
|
||||
|
||||
# Structured output lands in msg.metadata as a dict matching ExtractedUnits.
|
||||
# Empty / missing → no sub-units (Phase 2 will skip).
|
||||
meta = msg.metadata if isinstance(msg.metadata, dict) else {}
|
||||
cleaned: list[dict] = []
|
||||
for raw in meta.get("units") or []:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
name = str(raw.get("name") or "").strip()
|
||||
summary = str(raw.get("summary") or "").strip()
|
||||
bucket = str(raw.get("bucket") or "").strip()
|
||||
if not name or not summary:
|
||||
continue
|
||||
if bucket not in BUCKETS:
|
||||
# Defensive: structured_model should already reject this,
|
||||
# but if it slips through we route to wiki (the catch-all).
|
||||
self.logger.warning(
|
||||
f"[{self.name}] unit {name!r} emitted bucket {bucket!r} "
|
||||
f"not in {list(BUCKETS)}; routing to 'wiki'",
|
||||
)
|
||||
bucket = "wiki"
|
||||
cleaned.append({"name": name, "summary": summary, "bucket": bucket})
|
||||
return cleaned, (msg.get_text_content() or "").strip()
|
||||
|
||||
async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome:
|
||||
"""One ReAct invocation per memory sub-unit, dispatched to the
|
||||
bucket-specific system prompt. Returns the parsed
|
||||
:class:`IntegrateOutcome` reported by the agent — that's the
|
||||
single source of truth for what got written (action +
|
||||
target_path)."""
|
||||
bucket = unit.get("bucket") or "wiki"
|
||||
toolkit = self._build_integrate_toolkit()
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
agent = FlexReActAgent(
|
||||
name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
f"integrate_system_prompt_{bucket}",
|
||||
vault_dir=str(vault_dir),
|
||||
digest_dir=digest_dir,
|
||||
bucket=bucket,
|
||||
),
|
||||
formatter=self.as_llm_formatter,
|
||||
toolkit=toolkit,
|
||||
)
|
||||
agent.set_console_output_enabled(self.console_enabled)
|
||||
user_message = self.prompt_format(
|
||||
"integrate_user_message",
|
||||
hint=hint or "(none)",
|
||||
unit_name=unit.get("name", ""),
|
||||
unit_bucket=bucket,
|
||||
unit_summary=unit.get("summary", ""),
|
||||
material_blob=material_blob,
|
||||
)
|
||||
msg = await agent.reply(
|
||||
Msg(name="reme", role="user", content=user_message),
|
||||
structured_model=IntegrateOutcome,
|
||||
)
|
||||
meta = msg.metadata if isinstance(msg.metadata, dict) else {}
|
||||
return IntegrateOutcome.model_validate(meta)
|
||||
|
||||
async def dream_one(self, path: str, hint: str = "") -> DreamResult:
|
||||
"""Run the full extract + integrate pipeline on one vault-relative
|
||||
material path. Returns a structured :class:`DreamResult`. Safe to
|
||||
call repeatedly on the same instance — per-invocation trackers are
|
||||
reset at the start of each call. Used both by :meth:`execute`
|
||||
(single file from context) and by :class:`CronDreamer` (loop over
|
||||
today's materials).
|
||||
"""
|
||||
path = (path or "").strip()
|
||||
hint = (hint or "").strip()
|
||||
|
||||
if not path:
|
||||
return DreamResult(used_llm=False, skipped=True)
|
||||
|
||||
if not self._llm_available():
|
||||
return DreamResult(
|
||||
used_llm=False,
|
||||
skipped=True,
|
||||
path=path,
|
||||
error="no as_llm configured; dreaming requires an LLM",
|
||||
)
|
||||
|
||||
material_blob = _pack_material(self.file_store, path)
|
||||
|
||||
vault_dir = self._vault_dir()
|
||||
|
||||
# Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the
|
||||
# memory sub-units worth lifting. Each unit carries its own bucket.
|
||||
self.logger.info(f"[{self.name}] extract phase: path={path!r}")
|
||||
units, extract_summary = await self._extract(material_blob, hint, vault_dir)
|
||||
|
||||
if not units:
|
||||
return DreamResult(
|
||||
used_llm=True,
|
||||
path=path,
|
||||
summary=extract_summary or "no memory sub-units declared",
|
||||
skipped=True,
|
||||
)
|
||||
|
||||
unit_handles = ", ".join(f"{u['name']}/{u['bucket']}" for u in units)
|
||||
self.logger.info(f"[{self.name}] integrate phase: {len(units)} sub-unit(s): {unit_handles}")
|
||||
|
||||
# Phase 2 — integrate, one fresh ReAct per sub-unit, dispatched to
|
||||
# the bucket-specific system prompt. Python-level loop, not agent
|
||||
# loop. Each session emits a structured IntegrateOutcome whose
|
||||
# action + target_path are the source of truth for what landed.
|
||||
nodes_created: list[str] = []
|
||||
nodes_updated: list[str] = []
|
||||
per_unit_lines: list[str] = []
|
||||
for i, unit in enumerate(units, start=1):
|
||||
name = unit.get("name", "?")
|
||||
bucket = unit.get("bucket", "?")
|
||||
try:
|
||||
outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir)
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"[{self.name}] integrate {i}/{len(units)} "
|
||||
f"(unit={name}, bucket={bucket}) failed: {type(e).__name__}: {e}",
|
||||
)
|
||||
per_unit_lines.append(f"[{name}/{bucket}] FAILED: {type(e).__name__}: {e}")
|
||||
continue
|
||||
if outcome.action == "CREATE":
|
||||
nodes_created.append(outcome.target_path)
|
||||
else:
|
||||
nodes_updated.append(outcome.target_path)
|
||||
per_unit_lines.append(_render_outcome_line(name, bucket, outcome))
|
||||
|
||||
per_unit_block = "\n".join(per_unit_lines)
|
||||
summary = (
|
||||
f"Declared {len(units)} sub-unit(s) ({unit_handles}); "
|
||||
f"created {len(nodes_created)}, updated {len(nodes_updated)}.\n"
|
||||
f"{per_unit_block}"
|
||||
)
|
||||
|
||||
return DreamResult(
|
||||
used_llm=True,
|
||||
path=path,
|
||||
units=units,
|
||||
nodes_created=nodes_created,
|
||||
nodes_updated=nodes_updated,
|
||||
summary=summary,
|
||||
skipped=False,
|
||||
)
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
path: str = (self.context.get("path", "") or "").strip()
|
||||
hint: str = (self.context.get("hint", "") or "").strip()
|
||||
|
||||
result = await self.dream_one(path, hint)
|
||||
|
||||
if not path:
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = "Skipped: no path supplied"
|
||||
elif result.error:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"Error: {result.error}"
|
||||
elif result.skipped:
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = result.summary or "Skipped: no memory sub-units declared"
|
||||
else:
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = result.summary
|
||||
self.context.response.metadata.update(result.model_dump())
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CronDreamer — daily-tick wrapper around Dreamer.
|
||||
#
|
||||
# Inherits the per-file pipeline from Dreamer.dream_one and adds the
|
||||
# outer loop over today's daily/ + resource/ files. Cron scheduling
|
||||
# itself is out of scope; this step is just the unit of work.
|
||||
#
|
||||
# Inputs (RuntimeContext):
|
||||
# date (str, optional): YYYY-MM-DD to scan. Defaults to today
|
||||
# in the dreamer's timezone.
|
||||
# hint (str, optional): passed through to each per-file dream.
|
||||
# ============================================================
|
||||
|
||||
|
||||
class CronDreamResult(BaseModel):
|
||||
"""Aggregated outcome of one cron tick."""
|
||||
|
||||
date: str = ""
|
||||
files_scanned: int = 0
|
||||
files_dreamed: int = 0
|
||||
files_skipped: int = 0
|
||||
files_failed: int = 0
|
||||
per_file: list[DreamResult] = Field(default_factory=list)
|
||||
summary: str = ""
|
||||
|
||||
|
||||
@R.register("cron_dreamer_step")
|
||||
class CronDreamer(Dreamer):
|
||||
"""Loop ``daily/<today>/`` + ``resource/<today>/`` and dream each file.
|
||||
|
||||
Inherits :data:`auto_dream.yaml` from :class:`Dreamer` (no separate
|
||||
yaml — there's no extra prompt for the outer loop).
|
||||
"""
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
date_input: str = (self.context.get("date", "") or "").strip()
|
||||
hint: str = (self.context.get("hint", "") or "").strip()
|
||||
|
||||
# daily_dir / resource_dir come from app config — NOT tool params.
|
||||
# Same convention as daily_create / daily_list / daily_reindex.
|
||||
# resource_dir may be empty (default) — that just skips the resource scan.
|
||||
cfg = self.app_context.app_config if self.app_context is not None else None
|
||||
daily_dir = (cfg.daily_dir if cfg else "") or "daily"
|
||||
resource_dir = cfg.resource_dir if cfg else ""
|
||||
|
||||
today = date_input or self._now().strftime("%Y-%m-%d")
|
||||
vault = self._vault_dir()
|
||||
files = _scan_today_files(vault, today, daily_dir, resource_dir)
|
||||
|
||||
result = CronDreamResult(date=today, files_scanned=len(files))
|
||||
self.logger.info(
|
||||
f"[{self.name}] cron tick date={today} scanned={len(files)} file(s) under "
|
||||
f"{daily_dir}/{today}/ + {resource_dir}/{today}/",
|
||||
)
|
||||
|
||||
for rel_path in files:
|
||||
try:
|
||||
dr = await self.dream_one(rel_path, hint)
|
||||
except Exception as e: # pylint: disable=broad-except
|
||||
self.logger.error(
|
||||
f"[{self.name}] dream_one failed on {rel_path}: {type(e).__name__}: {e}",
|
||||
)
|
||||
dr = DreamResult(
|
||||
path=rel_path,
|
||||
error=f"{type(e).__name__}: {e}",
|
||||
)
|
||||
result.per_file.append(dr)
|
||||
if dr.error:
|
||||
result.files_failed += 1
|
||||
elif dr.skipped:
|
||||
result.files_skipped += 1
|
||||
else:
|
||||
result.files_dreamed += 1
|
||||
|
||||
result.summary = _render_cron_summary(result)
|
||||
self.context.response.success = result.files_failed == 0
|
||||
self.context.response.answer = result.summary
|
||||
self.context.response.metadata.update(result.model_dump())
|
||||
|
||||
|
||||
def _scan_today_files(
|
||||
vault: Path,
|
||||
today: str,
|
||||
daily_dir: str,
|
||||
resource_dir: str,
|
||||
) -> list[str]:
|
||||
"""Return vault-relative paths of today's daily notes + resource files.
|
||||
|
||||
* ``<daily_dir>/<today>.md`` — the day-index file (auto-rebuilt
|
||||
rollup of all of today's notes). Included first so its day-level
|
||||
abstractions land before the per-event details.
|
||||
* ``<daily_dir>/<today>/**/*.md`` — event notes for the day,
|
||||
sorted by path.
|
||||
* ``<resource_dir>/<today>/**/*`` — any file type ingested under
|
||||
today's resource folder. Skipped when ``resource_dir`` is empty.
|
||||
|
||||
Results are sorted for deterministic processing order within each
|
||||
group; the day-index file leads.
|
||||
"""
|
||||
out: list[str] = []
|
||||
|
||||
if daily_dir:
|
||||
day_index = vault / daily_dir / f"{today}.md"
|
||||
if day_index.is_file():
|
||||
out.append(str(day_index.relative_to(vault)))
|
||||
daily_root = vault / daily_dir / today
|
||||
if daily_root.is_dir():
|
||||
for md in sorted(daily_root.rglob("*.md")):
|
||||
if md.is_file():
|
||||
out.append(str(md.relative_to(vault)))
|
||||
|
||||
if resource_dir:
|
||||
resource_root = vault / resource_dir / today
|
||||
if resource_root.is_dir():
|
||||
for f in sorted(p for p in resource_root.rglob("*") if p.is_file()):
|
||||
out.append(str(f.relative_to(vault)))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def _render_cron_summary(r: CronDreamResult) -> str:
|
||||
"""One-line header + one line per file with its outcome."""
|
||||
lines = [
|
||||
f"[CronDreamer] date={r.date} scanned={r.files_scanned} "
|
||||
f"dreamed={r.files_dreamed} skipped={r.files_skipped} failed={r.files_failed}",
|
||||
]
|
||||
for dr in r.per_file:
|
||||
if dr.error:
|
||||
status = f"ERROR ({dr.error})"
|
||||
elif dr.skipped:
|
||||
status = "SKIP"
|
||||
else:
|
||||
status = f"OK (+{len(dr.nodes_created)} created, ~{len(dr.nodes_updated)} updated)"
|
||||
lines.append(f" - {dr.path}: {status}")
|
||||
return "\n".join(lines)
|
||||
807
reme4/steps/evolve/auto_dream.yaml
Normal file
807
reme4/steps/evolve/auto_dream.yaml
Normal file
|
|
@ -0,0 +1,807 @@
|
|||
extract_system_prompt: |
|
||||
You are Phase 1 of dream — read the material, identify the
|
||||
ABSTRACTIONS it teaches, and tag each with a bucket. Phase 2
|
||||
picks the slug and writes; you only declare what's worth lifting.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## What digest memory is for
|
||||
|
||||
Digest is the **abstract memory layer** — analogous to the
|
||||
prefrontal cortex aggregating cognition. Raw details (numbers,
|
||||
narratives, who said what, full procedure text) STAY IN THE
|
||||
MATERIAL. Digest holds the generalized lesson the reader should
|
||||
recall next time — the part that survives once the specific
|
||||
event fades.
|
||||
|
||||
When you cluster, you are NOT cataloguing the material's contents
|
||||
— you are answering: *"what abstractions does this material teach
|
||||
that I'd want a future agent / human to have at-hand when facing
|
||||
a similar situation?"*
|
||||
|
||||
## What is a memory sub-unit?
|
||||
|
||||
One sub-unit = one abstraction the material teaches. **One sub-unit
|
||||
maps to exactly one digest node** — Phase 2 makes one write
|
||||
decision per sub-unit (CREATE or one of the three UPDATE flavors).
|
||||
Phase 1 is the gate for "not worth memorizing"; once a sub-unit
|
||||
reaches Phase 2 it WILL be written.
|
||||
|
||||
Multiple raw facts in the material that all illustrate the same
|
||||
abstraction collapse to ONE sub-unit. Example: the kid-versioning
|
||||
mechanism, the SOC2 CC6.1 rationale, and the new 24h cadence are
|
||||
three FACTS, but they teach one abstraction — "short-credential
|
||||
compliance drives auth cadence, not procedural convenience".
|
||||
That's one sub-unit. The mechanism / numbers / RFC citation are
|
||||
details — they stay in the daily note; the digest reaches them
|
||||
through `derived_from::` provenance edges.
|
||||
|
||||
Sub-units are NOT bucket names, NOT kinds, NOT the eventual digest
|
||||
slug — they're an agent-internal handle for the abstraction you've
|
||||
identified. Phase 2 picks the slug + write decision; YOU pick the
|
||||
bucket here.
|
||||
|
||||
### Bias: fewer, richer sub-units over many narrow ones
|
||||
|
||||
This is the abstract layer — heavy lifting toward few high-leverage
|
||||
units, not toward exhaustive coverage. Heuristic for splitting two
|
||||
pieces into two units vs one:
|
||||
|
||||
* Same abstraction shown by different facts? → ONE unit.
|
||||
* Genuinely different abstractions a future reader would invoke
|
||||
in DIFFERENT situations? → TWO units.
|
||||
* Will they evolve independently as more materials arrive?
|
||||
→ TWO units.
|
||||
|
||||
When in doubt, MERGE (or drop one of them entirely).
|
||||
|
||||
### What NOT to declare
|
||||
|
||||
- Passing mentions with no new abstraction (e.g. an OAuth recap
|
||||
that restates a known concept) — daily-note indexing already
|
||||
covers detail-level recall.
|
||||
- Facts whose only audience is the material itself (one-off
|
||||
timestamps, single meeting attendance) — not an abstraction.
|
||||
- Event-level umbrella sub-units (e.g. `X-event-summary`) —
|
||||
every sub-unit already carries
|
||||
`derived_from:: [[<material-path>]]`, so the material itself
|
||||
is the fan-out point linking to all its derived nodes; the
|
||||
umbrella adds nothing.
|
||||
|
||||
## Bucket — pick exactly one per unit
|
||||
|
||||
The bucket determines which specialized Phase 2 prompt processes
|
||||
this sub-unit. Pick by *kind of abstraction*, not by surface
|
||||
topic.
|
||||
|
||||
- **`procedure`** — *how to do X*. Steps, methods, recipes,
|
||||
workflows, runbooks, executable patterns. Reader's question:
|
||||
"how do I accomplish Y?" Pick when the abstraction is an
|
||||
actionable sequence or technique.
|
||||
Examples: "key-rotation procedure", "incident triage flow",
|
||||
"how to wire up a new MCP tool".
|
||||
|
||||
- **`personal`** — *user/team-specific facts about how WE work*.
|
||||
Identity ("who is X"), preferences ("user prefers terse
|
||||
replies"), conventions ("we use kebab-case for slug names"),
|
||||
things to avoid ("don't run schema migrations on Friday"),
|
||||
collaboration style. Reader's question: "what does THIS user /
|
||||
team want / do / dislike?" Pick when the abstraction is only
|
||||
valid in the context of this user / team / project.
|
||||
Examples: "huangsen prefers short PRs", "team avoids
|
||||
mocking the DB in integration tests", "we don't write
|
||||
`status` frontmatter".
|
||||
|
||||
- **`wiki`** — *general knowledge*. Definitions, principles,
|
||||
observations, decisions-as-precedent, factual claims, mental
|
||||
models. Reader's question: "what IS X / what happened / what
|
||||
was decided?" Pick when the abstraction is true independent
|
||||
of who's reading. Also the **default catch-all** when nothing
|
||||
else fits cleanly.
|
||||
Examples: "JWT is a signed token format", "short-credential
|
||||
compliance drives auth cadence", "moving to 24h refresh
|
||||
reduced p99 latency by 12%".
|
||||
|
||||
Straddling two buckets → pick by **center of gravity** (which
|
||||
bucket the future reader will search from):
|
||||
- "User prefers small PRs" → personal (rule for THIS user).
|
||||
- "Small PRs are easier to review" → wiki (general claim).
|
||||
- "Steps to split a large PR" → procedure.
|
||||
|
||||
Available: {buckets}
|
||||
|
||||
## Output
|
||||
|
||||
Each unit's `summary` should name the abstraction AND point at
|
||||
where in the material the supporting evidence lives — Phase 2
|
||||
cites it as provenance without re-reading. Field shapes are
|
||||
enforced by the structured-output schema.
|
||||
|
||||
You have read-only access (`read` for inline `[[resource/...]]`
|
||||
references that genuinely matter); no recall, no writing.
|
||||
|
||||
extract_user_message: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# Material to cluster
|
||||
|
||||
{material_blob}
|
||||
|
||||
Identify the abstractions this material teaches, classify each
|
||||
into one of {{procedure, personal, wiki}}, and emit via the
|
||||
structured output schema. Empty list if nothing new is taught.
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Phase 2 — bucket-specific INTEGRATE prompts.
|
||||
# Dispatcher picks integrate_system_prompt_<bucket> from the
|
||||
# bucket Phase 1 assigned to the unit.
|
||||
# ============================================================
|
||||
|
||||
integrate_system_prompt_procedure: |
|
||||
You are Phase 2 of dream, **procedure** bucket. The unit is a
|
||||
how-to-do-X (steps, methods, recipes, runbooks, executable
|
||||
patterns). Recall cross-bucket, decide CREATE / CORROBORATE /
|
||||
REFINE / CORRECT, write exactly once. Sub-unit ↔ digest node
|
||||
is 1:1; no SKIP outcome — Phase 1 already gated.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest is the abstract memory layer
|
||||
|
||||
Digest is NOT a faithful copy of the material — it's the cognitive
|
||||
aggregation (think prefrontal cortex). Details stay in the daily
|
||||
/ resource file; digest holds the principle, pattern, or
|
||||
precedent the agent should recall later.
|
||||
|
||||
- **Body is SHORT and abstract** (≈ 50-200 words for most nodes;
|
||||
longer only when the concept genuinely needs it). If your draft
|
||||
starts copying paragraphs from the material, you're filing
|
||||
detail in the wrong layer.
|
||||
- **Provenance edges carry the details.** Whenever this
|
||||
abstraction is illustrated by a specific material, add a
|
||||
`derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink
|
||||
— readers drill down through the edge, not through re-stated
|
||||
facts in the body.
|
||||
- **Wikilinks between digest nodes** carry the conceptual graph
|
||||
(`relates_to::`, `depends_on::`, `is_a::`, …).
|
||||
|
||||
## Procedure-bucket body shape
|
||||
|
||||
A runbook, not a recap:
|
||||
|
||||
- **Trigger / when to use** (1 line) — under what conditions
|
||||
does the reader reach for this procedure?
|
||||
- **Steps** — numbered or terse bullets; each is one verb-led
|
||||
imperative. Optional inline justification ("because X locks
|
||||
the row before Y commits") is fine.
|
||||
- **Pre-conditions / inputs** — short list, not prose.
|
||||
- **Failure modes / caveats** — brief ("if step 3 returns
|
||||
ROLLBACK, restart from step 1"); NOT a transcript of every
|
||||
observed failure.
|
||||
- **`derived_from:: [[<material-path>]]`** — at least one.
|
||||
Plain-prose provenance does NOT count (only wikilinks survive
|
||||
future updates).
|
||||
|
||||
## Recall → decide → write
|
||||
|
||||
1. **Recall** — `search` (include verb stems: rotate, migrate,
|
||||
deploy…) + `traverse depth=2 direction=both` on any hit under
|
||||
`{digest_dir}/`. Cross-bucket on purpose: an existing match
|
||||
filed elsewhere beats a duplicate.
|
||||
2. **Hit** — `frontmatter_read` triage, then `read` body for
|
||||
survivors. Same procedure = same trigger + substantially
|
||||
overlapping steps. New step or one-line nuance is REFINE,
|
||||
not "different procedure".
|
||||
3. **Decide** (exactly one):
|
||||
- empty hit set → **CREATE** at
|
||||
`{digest_dir}/procedure/<slug>.md`.
|
||||
- hit non-empty → **UPDATE** the best match:
|
||||
- **CORROBORATE** — same procedure observed again; append
|
||||
`derived_from::`, optionally strengthen wording
|
||||
("consistently used across N runs").
|
||||
- **REFINE** — new pre-condition / edge case / failure
|
||||
mode; expand the relevant span, slot new steps into
|
||||
the right position.
|
||||
- **CORRECT** — wrong order, missing critical step, bad
|
||||
outcome; tighten or annotate inline (`> note:
|
||||
contradicted by [[new-material]] — <one-line>`).
|
||||
|
||||
## Discipline
|
||||
|
||||
- CREATE writes inside `{digest_dir}/procedure/`. Phase 1 chose
|
||||
your bucket — don't pivot.
|
||||
- UPDATE may target any bucket if RECALL legitimately matched.
|
||||
- `edit` is body-only and **only-add, not-delete**: never drop
|
||||
wikilinks the `old` span contained (provenance must accumulate,
|
||||
not evaporate).
|
||||
- `frontmatter_update` is the only way to change frontmatter
|
||||
(e.g. tighten `description`, set `kind: procedure`).
|
||||
- One target per session. Never edit other nodes sideways.
|
||||
|
||||
Wikilinks are full vault-relative paths with `.md`
|
||||
(`[[{digest_dir}/<bucket>/<slug>.md]]`, `[[daily/...]]`,
|
||||
`[[resource/...]]`). Predicates are open
|
||||
(`[A-Za-z][A-Za-z0-9_]*`) and live outside the brackets.
|
||||
|
||||
|
||||
integrate_system_prompt_personal: |
|
||||
You are Phase 2 of dream, **personal** bucket. The unit is
|
||||
user/team-specific (identity, preference, convention, avoid-rule,
|
||||
collaboration style). Recall cross-bucket, decide CREATE /
|
||||
CORROBORATE / REFINE / CORRECT, write exactly once. Sub-unit ↔
|
||||
digest node is 1:1; no SKIP — Phase 1 already gated.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest is the abstract memory layer
|
||||
|
||||
Digest is NOT a faithful copy of the material — it's the cognitive
|
||||
aggregation (think prefrontal cortex). Details stay in the daily
|
||||
/ resource file; digest holds the rule, identity, or convention
|
||||
the agent should recall later.
|
||||
|
||||
- **Body is SHORT and abstract** (≈ 50-200 words). If your draft
|
||||
starts narrating *what the user said in detail*, you're filing
|
||||
detail in the wrong layer.
|
||||
- **Provenance edges carry the details.** Whenever this rule is
|
||||
set, restated, or revised by a specific material, add a
|
||||
`derived_from:: [[daily/...]]` wikilink — readers drill down
|
||||
through the edge, not through re-stated context.
|
||||
- **Wikilinks between digest nodes** carry the conceptual graph
|
||||
(`applies_to::`, `relates_to::`, …).
|
||||
|
||||
## Personal-bucket body shape
|
||||
|
||||
A short rule of engagement, not a biography:
|
||||
|
||||
- **Rule / fact** — one sentence stating the preference,
|
||||
convention, or identity claim.
|
||||
- **`Why:`** — the reason (a past incident, a constraint, a
|
||||
strong preference). Knowing *why* lets future readers judge
|
||||
edge cases instead of blindly applying.
|
||||
- **`How to apply:`** — when this rule kicks in: which contexts,
|
||||
tasks, boundaries.
|
||||
- **`derived_from:: [[<material-path>]]`** — at least one.
|
||||
Plain-prose provenance does NOT count.
|
||||
|
||||
Two common sub-shapes filed in this bucket:
|
||||
- *Identity* — biographical / role facts ("X is a backend
|
||||
engineer focused on observability"). Reader's question:
|
||||
"who is X?".
|
||||
- *Preference / convention / avoid-rule* — how someone likes
|
||||
to work / what to skip. Reader's question: "how does X like
|
||||
to work / what should I not do?".
|
||||
|
||||
When the same person has many preferences, prefer **one node per
|
||||
preference** (not one big node per person) — that's the
|
||||
granularity downstream search will hit.
|
||||
|
||||
## Recall → decide → write
|
||||
|
||||
1. **Recall** — `search` (user/team name + rule keywords:
|
||||
`user-X-pr-size-pref`, `team-no-friday-deploys`) +
|
||||
`traverse depth=2 direction=both` on any hit under
|
||||
`{digest_dir}/`. Personal nodes often link to each other and
|
||||
to the user's identity node; don't skip traverse.
|
||||
2. **Hit** — `frontmatter_read` triage; `read` body for
|
||||
survivors. Same rule = same actor scope + same governing
|
||||
principle. A new context where the rule applies is REFINE,
|
||||
not "different rule".
|
||||
3. **Decide** (exactly one):
|
||||
- empty hit set → **CREATE** at
|
||||
`{digest_dir}/personal/<slug>.md`.
|
||||
- hit non-empty → **UPDATE** the best match:
|
||||
- **CORROBORATE** — rule reaffirmed; append
|
||||
`derived_from::`, possibly strengthen certainty
|
||||
("observed across N independent contexts").
|
||||
- **REFINE** — scope clarified ("only in CI runs",
|
||||
"except when X holds"); expand `How to apply:`.
|
||||
- **CORRECT** — user changed their mind / contradicted by
|
||||
new behavior; tighten to the form both old and new
|
||||
evidence support, OR annotate (`> note: contradicted by
|
||||
[[new-material]] — user now prefers Y`) without
|
||||
arbitrating.
|
||||
|
||||
## Discipline
|
||||
|
||||
- CREATE writes inside `{digest_dir}/personal/`. Phase 1 chose
|
||||
your bucket — don't pivot.
|
||||
- UPDATE may target any bucket if RECALL legitimately matched.
|
||||
- `edit` is body-only and **only-add, not-delete**: never drop
|
||||
wikilinks the `old` span contained.
|
||||
- `frontmatter_update` is the only way to change frontmatter
|
||||
(e.g. tighten `description`, set `kind: preference`).
|
||||
- One target per session. Never edit other nodes sideways.
|
||||
|
||||
Useful predicates: `derived_from::`, `applies_to::` (whose rule),
|
||||
`relates_to::` (cross-link related preferences). Wikilinks are
|
||||
full vault-relative paths with `.md`.
|
||||
|
||||
|
||||
integrate_system_prompt_wiki: |
|
||||
You are Phase 2 of dream, **wiki** bucket. The unit is general
|
||||
knowledge (definition, principle, observation, decision-as-
|
||||
precedent, factual claim, mental model). `wiki` is also the
|
||||
catch-all when nothing more specific fits. Recall cross-bucket,
|
||||
decide CREATE / CORROBORATE / REFINE / CORRECT, write exactly
|
||||
once. Sub-unit ↔ digest node is 1:1; no SKIP — Phase 1 already
|
||||
gated.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest is the abstract memory layer
|
||||
|
||||
Digest is NOT a faithful copy of the material — it's the cognitive
|
||||
aggregation (think prefrontal cortex). Details stay in the daily
|
||||
/ resource file; digest holds the definition, principle, or
|
||||
precedent the agent should recall later.
|
||||
|
||||
- **Body is SHORT and abstract** (≈ 50-200 words; longer only
|
||||
when the concept genuinely needs it). If your draft starts
|
||||
copying paragraphs from the material, you're filing detail in
|
||||
the wrong layer.
|
||||
- **Provenance edges carry the details.** Whenever this
|
||||
abstraction is illustrated by a specific material, add a
|
||||
`derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink.
|
||||
- **Wikilinks between digest nodes** carry the conceptual graph
|
||||
(`is_a::`, `extends::`, `depends_on::`, `contradicts::`, …).
|
||||
|
||||
## Wiki-bucket body shape
|
||||
|
||||
Encyclopedia-flavored — definition + properties + relations,
|
||||
not narrative:
|
||||
|
||||
- **First line** — one-sentence definition / claim. The reader's
|
||||
eye lands here first; make it self-contained.
|
||||
- **Body** — short paragraphs OR tight bullets: properties,
|
||||
sub-claims, distinctions, illustrative one-line examples. Each
|
||||
non-obvious claim cites its source via `derived_from::`.
|
||||
- **Relations** — typed wikilinks where the relation has
|
||||
semantic weight. Most cross-node links can stay bare.
|
||||
- **`derived_from:: [[<material-path>]]`** — at least one.
|
||||
Plain-prose provenance does NOT count.
|
||||
|
||||
## Recall → decide → write
|
||||
|
||||
1. **Recall** — `search` (noun phrases + common synonyms) +
|
||||
`traverse depth=2 direction=both` on any hit under
|
||||
`{digest_dir}/`. Skipping `traverse` is the main failure mode
|
||||
producing duplicate concept nodes filed under different
|
||||
terminology — semantically close abstractions often live one
|
||||
wikilink away from a noisy hit.
|
||||
2. **Hit** — `frontmatter_read` triage; `read` body for
|
||||
survivors. Same abstraction = same definition / principle in
|
||||
the body, even if wording differs. Slightly different framing
|
||||
of the same idea is REFINE; outright different concepts are
|
||||
different nodes.
|
||||
3. **Decide** (exactly one):
|
||||
- empty hit set → **CREATE** at
|
||||
`{digest_dir}/wiki/<slug>.md`.
|
||||
- hit non-empty → **UPDATE** the best match:
|
||||
- **CORROBORATE** — principle reaffirmed by new instance;
|
||||
append `derived_from::`, optionally strengthen wording
|
||||
("consistently observed across N sources" / replace
|
||||
"appears to" with "does"); body unchanged in substance.
|
||||
- **REFINE** — definition's nuance / scope / edge cases
|
||||
sharpened by the new material; tighten the relevant span,
|
||||
add the new dimension. Body grows in precision, not in
|
||||
detail volume.
|
||||
- **CORRECT** — factual contradiction or overstatement;
|
||||
either tighten to the narrower form both old and new
|
||||
evidence support, or annotate inline (`> note:
|
||||
contradicted by [[new-material]] — <one-line>`) without
|
||||
arbitrating.
|
||||
|
||||
## Discipline
|
||||
|
||||
- CREATE writes inside `{digest_dir}/wiki/`. Phase 1 chose your
|
||||
bucket — don't pivot.
|
||||
- UPDATE may target any bucket if RECALL legitimately matched.
|
||||
- `edit` is body-only and **only-add, not-delete**: never drop
|
||||
wikilinks the `old` span contained.
|
||||
- `frontmatter_update` is the only way to change frontmatter
|
||||
(e.g. tighten `description`, set `kind: concept` /
|
||||
`kind: observation`).
|
||||
- One target per session. Never edit other nodes sideways.
|
||||
|
||||
Wikilinks are full vault-relative paths with `.md`. Predicates
|
||||
are open (`[A-Za-z][A-Za-z0-9_]*`); reuse existing predicates
|
||||
when reasonable. Most wikilinks are bare — use a predicate only
|
||||
when the relation has clear semantic weight.
|
||||
|
||||
|
||||
integrate_user_message: |
|
||||
hint: {hint}
|
||||
|
||||
# Sub-unit
|
||||
|
||||
name: {unit_name}
|
||||
bucket: {unit_bucket}
|
||||
summary: {unit_summary}
|
||||
|
||||
# Full material
|
||||
|
||||
{material_blob}
|
||||
|
||||
Process per your system prompt: recall (cross-bucket) → hit →
|
||||
exactly one CREATE / CORROBORATE / REFINE / CORRECT. End with a
|
||||
fully-populated `IntegrateOutcome`.
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 中文版本 (language=zh 时启用)
|
||||
# ============================================================
|
||||
|
||||
extract_system_prompt_zh: |
|
||||
你是 dream 的 Phase 1 —— 阅读材料,识别它教导的 **抽象**,为
|
||||
每个抽象标 bucket。Phase 2 选 slug、写入;你只声明值得提取的
|
||||
内容。
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## digest 记忆是干什么的
|
||||
|
||||
Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事情发
|
||||
生的原始细节(数字、叙述、谁说了什么、完整流程文本)**保留
|
||||
在材料中**。Digest 承载的是读者下次该回想起的、即使具体事件
|
||||
淡忘后仍然有用的概括性教训。
|
||||
|
||||
你不是在 **编目** 材料的内容,而是在回答:*"这份材料教了哪
|
||||
些抽象,是我希望未来的 agent / 人类在面对类似情境时手边能够
|
||||
调取的?"*
|
||||
|
||||
## 什么是记忆 sub-unit
|
||||
|
||||
一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好对
|
||||
应一个 digest 节点** —— Phase 2 针对每个 sub-unit 做一次写
|
||||
入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是"不值得记忆"
|
||||
的过滤闸口;一旦 sub-unit 进入 Phase 2,它就 **一定** 会被
|
||||
写入。
|
||||
|
||||
材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。例:
|
||||
kid 版本机制 + SOC2 CC6.1 依据 + 24h 新周期 是三个 **事实**,
|
||||
但教的是同一个抽象 —— "JWT 轮换周期由短期凭证合规驱动,而
|
||||
非流程惯性"。这是一个 sub-unit。机制 / 数字 / RFC 引用都是
|
||||
细节 —— 它们留在 daily 笔记里,digest 通过 `derived_from::`
|
||||
溯源边触达。
|
||||
|
||||
Sub-unit **不是** bucket 名,**不是** kind,**不是** 最终
|
||||
digest slug —— 它只是你内部用于指代识别出来的抽象的把手。
|
||||
Phase 2 选 slug + 写入决策;**bucket 由你在 Phase 1 决定**。
|
||||
|
||||
### 偏好:少而精的 sub-unit,而非多而细
|
||||
|
||||
这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是穷举
|
||||
覆盖。两件事拆成一个还是两个 sub-unit 的启发式:
|
||||
|
||||
* 不同事实说明同一抽象? → 一个 sub-unit。
|
||||
* 是真正不同的抽象,未来读者会在 **不同情境** 下分别调用?
|
||||
→ 两个 sub-unit。
|
||||
* 它们会随更多材料独立演化? → 两个 sub-unit。
|
||||
|
||||
拿不准时,**合并**(或整体丢弃其中一个)。
|
||||
|
||||
### 哪些不要声明
|
||||
|
||||
- 没有新抽象的顺带提及(例如只是把已知概念复述一遍的 OAuth
|
||||
简介) —— daily 笔记索引已能覆盖细节级召回。
|
||||
- 受众只有材料本身的事实(一次性时间戳、单次会议出席记
|
||||
录) —— 不是抽象。
|
||||
- 事件级伞节点(例如 `X-event-summary`) —— 每个 sub-unit
|
||||
都会带 `derived_from:: [[<material-path>]]`,材料本身就是
|
||||
扇出节点链向所有派生 digest;伞节点零增益。
|
||||
|
||||
## Bucket —— 每个 unit 必选其一
|
||||
|
||||
Bucket 决定哪份 Phase 2 prompt 处理这个 sub-unit。按 *抽象
|
||||
的种类* 选,**不是** 按材料表面话题选。
|
||||
|
||||
- **`procedure`** —— *怎么做 X*。步骤、方法、配方、工作流、
|
||||
runbook、可执行模式。读者问:"怎么完成 Y?"。当抽象是可
|
||||
执行的动作序列或技巧时选这个。
|
||||
例:"key-rotation 流程"、"事故 triage 流"、"如何接入新
|
||||
MCP 工具"。
|
||||
|
||||
- **`personal`** —— *用户 / 团队 specific 的 "我们怎么干"
|
||||
类事实*。身份("X 是谁")、偏好("用户偏好简短回复")、
|
||||
约定("我们用 kebab-case 命名 slug")、规避("周五不跑
|
||||
schema 迁移")、协作风格。读者问:"这个用户 / 团队 想要
|
||||
/ 不喜欢什么?"。当抽象只在这位用户 / 团队 / 项目上下文
|
||||
里成立时选这个。
|
||||
例:"huangsen 偏好小 PR"、"团队不在集成测试里 mock DB"、
|
||||
"我们不写 `status` frontmatter"。
|
||||
|
||||
- **`wiki`** —— *通用知识*。定义、原则、观察、决策先例、事
|
||||
实主张、心智模型。读者问:"X 是什么 / 决策依据是什么?" ——
|
||||
与谁在问无关。也是 **兜底** —— 没有更明确归属时落到这里。
|
||||
例:"JWT 是签名 token 格式"、"短期凭证合规驱动鉴权周
|
||||
期"、"切到 24h 刷新后 p99 降低 12%"。
|
||||
|
||||
跨桶时按 **重心** 选(未来读者最可能从哪个桶搜):
|
||||
- "用户偏好小 PR" → personal(这个用户的规则)。
|
||||
- "小 PR 更易评审" → wiki(通用主张)。
|
||||
- "如何拆分大 PR 的步骤" → procedure。
|
||||
|
||||
可用 buckets: {buckets}
|
||||
|
||||
## 输出
|
||||
|
||||
每个 unit 的 `summary` 要 **同时** 命名抽象 **并** 指出材料
|
||||
里支撑证据所在 —— Phase 2 直接引用做溯源,不必重读。字段形
|
||||
态由结构化输出 schema 强制约束。
|
||||
|
||||
你只有只读访问(`read` 用于打开内联 `[[resource/...]]` 引
|
||||
用,确实需要时);没有召回,没有写入。
|
||||
|
||||
extract_user_message_zh: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# 待归类的材料
|
||||
|
||||
{material_blob}
|
||||
|
||||
识别这份材料教导的 **抽象**,为每个 unit 分类到
|
||||
{{procedure, personal, wiki}} 之一,通过结构化输出 schema
|
||||
提交。没有新抽象时使用空 unit 列表。
|
||||
|
||||
|
||||
integrate_system_prompt_procedure_zh: |
|
||||
你是 dream 的 Phase 2,**procedure** 桶。本次处理的 unit 是
|
||||
一个"怎么做 X"(步骤、方法、配方、runbook、可执行模式)。
|
||||
跨 bucket 召回,在 CREATE / CORROBORATE / REFINE / CORRECT
|
||||
之间决策,**恰好一次** 写入。Sub-unit 与 digest 节点是 1:1;
|
||||
无 SKIP —— Phase 1 已过滤。
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest 是抽象记忆层
|
||||
|
||||
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
|
||||
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
|
||||
回想起的原则、模式、先例。
|
||||
|
||||
- **正文 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有概念真
|
||||
的需要时才更长)。如果你的草稿开始大段抄材料的段落,说明
|
||||
你把细节归错层了。
|
||||
- **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加
|
||||
一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`
|
||||
wikilink —— 读者通过边下钻,而不是通过正文里复述事实。
|
||||
- **digest 节点之间的 wikilink** 承载概念图(`relates_to::`、
|
||||
`depends_on::`、`is_a::`…)。
|
||||
|
||||
## procedure 桶的 body 形态
|
||||
|
||||
Runbook,不是叙述:
|
||||
|
||||
- **触发 / 何时使用**(1 行)—— 读者在什么条件下会调取这个
|
||||
流程?
|
||||
- **步骤** —— 编号或紧凑的子弹点列表;每一步是一个动词领头
|
||||
的祈使句。可选内联说明("因为 X 在 Y 提交前已锁住该行")。
|
||||
- **前置条件 / 输入** —— 简短列表,不要散文。
|
||||
- **失败模式 / 注意事项** —— 简短("若步骤 3 返回 ROLLBACK,
|
||||
从步骤 1 重启");**不是** 每次观察到的失败的转写。
|
||||
- **`derived_from:: [[<material-path>]]`** —— 至少一条。纯
|
||||
散文形式 **不算**(下次 update 时会消失)。
|
||||
|
||||
## 召回 → 决策 → 写入
|
||||
|
||||
1. **召回** —— `search`(带动词词根:rotate / migrate /
|
||||
deploy…)+ 对 `{digest_dir}/` 下 **任何** 命中跑
|
||||
`traverse depth=2 direction=both`。**跨 bucket** 是有意 ——
|
||||
就地更新优于复制创建。
|
||||
2. **命中** —— `frontmatter_read` 廉价 triage,幸存者用
|
||||
`read` 读完整 body。"同一流程" = 同触发 + 步骤大幅重叠。
|
||||
新增一步 / 细微差异是 REFINE,**不是** 另一个流程。
|
||||
3. **决策**(恰好一种):
|
||||
- 命中空 → **CREATE** 在
|
||||
`{digest_dir}/procedure/<slug>.md`。
|
||||
- 命中非空 → **UPDATE** 最匹配的:
|
||||
- **CORROBORATE** —— 同流程再次出现;加 `derived_from::`,
|
||||
可选强化措辞("跨 N 次运行一致使用");步骤不动。
|
||||
- **REFINE** —— 新前置 / 边界 / 失败模式;扩展相关片段,
|
||||
新步骤插入正确位置。
|
||||
- **CORRECT** —— 顺序错 / 缺关键步 / 结果不对;收紧或
|
||||
内联标注(`> note: contradicted by [[new-material]] —
|
||||
<一句话>`)。
|
||||
|
||||
## 纪律
|
||||
|
||||
- CREATE 必须写在 `{digest_dir}/procedure/`。Phase 1 已选定桶 ——
|
||||
不要换桶。
|
||||
- UPDATE 可指向任意 bucket(若召回合理命中)。
|
||||
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
|
||||
任何 wikilink(溯源必须累积,不可蒸发)。
|
||||
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
|
||||
(例如 REFINE 后收紧 `description`,加 `kind: procedure`)。
|
||||
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
|
||||
|
||||
Wikilink 是带 `.md` 的 vault 相对完整路径
|
||||
(`[[{digest_dir}/<bucket>/<slug>.md]]`、`[[daily/...]]`、
|
||||
`[[resource/...]]`)。谓词词表开放
|
||||
(`[A-Za-z][A-Za-z0-9_]*`),写在括号外。
|
||||
|
||||
|
||||
integrate_system_prompt_personal_zh: |
|
||||
你是 dream 的 Phase 2,**personal** 桶。本次处理的 unit 是
|
||||
用户 / 团队 specific(身份 / 偏好 / 约定 / 规避规则 / 协作
|
||||
风格)。跨 bucket 召回,在 CREATE / CORROBORATE / REFINE /
|
||||
CORRECT 之间决策,**恰好一次** 写入。Sub-unit 与 digest 节
|
||||
点是 1:1;无 SKIP —— Phase 1 已过滤。
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest 是抽象记忆层
|
||||
|
||||
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
|
||||
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
|
||||
回想起的规则、身份、约定。
|
||||
|
||||
- **正文 SHORT 且抽象**(≈ 50-200 字)。如果你的草稿开始详
|
||||
细叙述用户说了什么,说明归错层了。
|
||||
- **溯源边承载细节**。每当这条规则被某份具体材料设定 / 重申
|
||||
/ 修正时,加一条 `derived_from:: [[daily/...]]` wikilink ——
|
||||
读者通过边下钻,而不是通过正文里复述上下文。
|
||||
- **digest 节点之间的 wikilink** 承载概念图(`applies_to::`、
|
||||
`relates_to::`、…)。
|
||||
|
||||
## personal 桶的 body 形态
|
||||
|
||||
简短的协作规则,不是传记:
|
||||
|
||||
- **规则 / 事实** —— 一句话陈述偏好、约定或身份。
|
||||
- **`Why:`** —— 用户给出的原因(过往事故、所关心的约束、强
|
||||
烈偏好)。知道 *why* 让未来读者能判断边界,而非盲目套用。
|
||||
- **`How to apply:`** —— 这条规则什么时候启用:哪些情境、
|
||||
任务、边界。
|
||||
- **`derived_from:: [[<material-path>]]`** —— 至少一条;纯
|
||||
散文形式不算。
|
||||
|
||||
同桶常见两类子形态:
|
||||
- *身份* —— 用户 / 团队的传记 / 角色事实("X 是聚焦在
|
||||
observability 的 backend 工程师")。读者问:"X 是谁?"。
|
||||
- *偏好 / 约定 / 规避* —— 喜欢怎么干 / 该规避什么。读者问:
|
||||
"X 喜欢怎么干 / 我不该做什么?"。
|
||||
|
||||
同一人有多条偏好时,**一条偏好一个节点**(不是一个人一大
|
||||
节点) —— 这才是下游搜索的粒度。
|
||||
|
||||
## 召回 → 决策 → 写入
|
||||
|
||||
1. **召回** —— `search`(user / team 名 + 规则关键词:
|
||||
`user-X-pr-size-pref`、`team-no-friday-deploys`)+ 对
|
||||
`{digest_dir}/` 下 **任何** 命中跑
|
||||
`traverse depth=2 direction=both`。personal 节点常彼此互
|
||||
链并指向用户身份节点;**别跳过 traverse**。
|
||||
2. **命中** —— `frontmatter_read` triage,幸存者 `read` body。
|
||||
"同一规则" = 同 actor 范围 + 同支配原则。新增"规则适用情
|
||||
境"是 REFINE,**不是** 另一条规则。
|
||||
3. **决策**(恰好一种):
|
||||
- 命中空 → **CREATE** 在
|
||||
`{digest_dir}/personal/<slug>.md`。
|
||||
- 命中非空 → **UPDATE** 最匹配的:
|
||||
- **CORROBORATE** —— 规则在新场景再次坐实;加
|
||||
`derived_from::`,可选强化确定性("跨 N 个独立情境
|
||||
观察")。
|
||||
- **REFINE** —— 范围被澄清("仅在 CI 运行中"、"X 成立
|
||||
时除外");把新边界扩到 `How to apply:`。
|
||||
- **CORRECT** —— 用户改主意 / 规则被新行为否定;收紧到
|
||||
新旧证据都支持的形式,或内联标注
|
||||
(`> note: contradicted by [[new-material]] — 用户现在
|
||||
偏好 Y`)不仲裁。
|
||||
|
||||
## 纪律
|
||||
|
||||
- CREATE 必须写在 `{digest_dir}/personal/`。Phase 1 已选定桶 ——
|
||||
不要换桶。
|
||||
- UPDATE 可指向任意 bucket(若召回合理命中)。
|
||||
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
|
||||
任何 wikilink。
|
||||
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
|
||||
(例如 REFINE 后收紧 `description`,加 `kind: preference`)。
|
||||
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
|
||||
|
||||
常用谓词:`derived_from::`、`applies_to::`(规则归属哪个用
|
||||
户)、`relates_to::`(交叉链接相关偏好)。Wikilink 是带
|
||||
`.md` 的 vault 相对完整路径。
|
||||
|
||||
|
||||
integrate_system_prompt_wiki_zh: |
|
||||
你是 dream 的 Phase 2,**wiki** 桶。本次处理的 unit 是通用知
|
||||
识(定义 / 原则 / 观察 / 决策先例 / 事实主张 / 心智模型)。
|
||||
`wiki` 也是没有更明确归属时的 **兜底**。跨 bucket 召回,在
|
||||
CREATE / CORROBORATE / REFINE / CORRECT 之间决策,**恰好
|
||||
一次** 写入。Sub-unit 与 digest 节点是 1:1;无 SKIP —— Phase 1
|
||||
已过滤。
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
digest_dir: {digest_dir}
|
||||
|
||||
## Digest 是抽象记忆层
|
||||
|
||||
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
|
||||
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
|
||||
回想起的定义、原则、先例。
|
||||
|
||||
- **正文 SHORT 且抽象**(≈ 50-200 字;只有概念真的需要时才
|
||||
更长)。如果你的草稿开始大段抄材料,说明归错层了。
|
||||
- **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加
|
||||
一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`
|
||||
wikilink。
|
||||
- **digest 节点之间的 wikilink** 承载概念图(`is_a::`、
|
||||
`extends::`、`depends_on::`、`contradicts::`…)。
|
||||
|
||||
## wiki 桶的 body 形态
|
||||
|
||||
百科风 —— 定义 + 性质 + 关系,不是叙述:
|
||||
|
||||
- **首行** —— 一句话定义 / 主张。读者目光首先落在这里,要让
|
||||
它自包含。
|
||||
- **正文** —— 短段落或紧凑子弹点:性质、子主张、区分、举
|
||||
例片段。每条非显然主张靠 `derived_from::` 指回它的源材料。
|
||||
- **关系** —— 有语义份量时用谓词。绝大多数跨节点链接保持裸链。
|
||||
- **`derived_from:: [[<material-path>]]`** —— 至少一条;纯
|
||||
散文形式不算。
|
||||
|
||||
## 召回 → 决策 → 写入
|
||||
|
||||
1. **召回** —— `search`(名词短语 + 常见同义词)+ 对
|
||||
`{digest_dir}/` 下 **任何** 命中跑
|
||||
`traverse depth=2 direction=both`。跳过 `traverse` 是产生
|
||||
重复概念节点(不同 slug 同语义)的主要失败模式 —— 语义相
|
||||
邻的抽象常常就在某个噪音命中的一跳之外。
|
||||
2. **命中** —— `frontmatter_read` triage,幸存者 `read` body。
|
||||
"同一抽象" = body 中的定义 / 原则相同(措辞可不同)。同
|
||||
思想的略不同表述是 REFINE;真正不同的概念是不同节点。
|
||||
3. **决策**(恰好一种):
|
||||
- 命中空 → **CREATE** 在 `{digest_dir}/wiki/<slug>.md`。
|
||||
- 命中非空 → **UPDATE** 最匹配的:
|
||||
- **CORROBORATE** —— 原则被新实例再坐实;加
|
||||
`derived_from::`,可选强化措辞("跨 N 个来源一致观
|
||||
察"、把"似乎"换成"确实");正文实质不变。
|
||||
- **REFINE** —— 细微差异 / 范围被新材料补足;收紧片段、
|
||||
加新维度。正文 **精度** 上长,不在 **细节量** 上膨胀。
|
||||
- **CORRECT** —— 事实矛盾或夸大;收紧到新旧证据都支持
|
||||
的窄形式,或内联标注(`> note: contradicted by
|
||||
[[new-material]] — <一句话>`)不仲裁。
|
||||
|
||||
## 纪律
|
||||
|
||||
- CREATE 必须写在 `{digest_dir}/wiki/`。Phase 1 已选定桶 ——
|
||||
不要换桶。
|
||||
- UPDATE 可指向任意 bucket(若召回合理命中)。
|
||||
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
|
||||
任何 wikilink。
|
||||
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
|
||||
(例如 REFINE 后收紧 `description`,加 `kind: concept` /
|
||||
`kind: observation`)。
|
||||
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
|
||||
|
||||
Wikilink 是带 `.md` 的 vault 相对完整路径。谓词词表开放
|
||||
(`[A-Za-z][A-Za-z0-9_]*`),合理时复用。绝大多数 wikilink
|
||||
保持裸链 —— 仅当关系具有清晰语义份量时用谓词。
|
||||
|
||||
|
||||
integrate_user_message_zh: |
|
||||
hint: {hint}
|
||||
|
||||
# Sub-unit
|
||||
|
||||
name: {unit_name}
|
||||
bucket: {unit_bucket}
|
||||
summary: {unit_summary}
|
||||
|
||||
# 完整材料
|
||||
|
||||
{material_blob}
|
||||
|
||||
按 system prompt 处理:召回(跨 bucket)→ 命中 → 恰好一次
|
||||
CREATE / CORROBORATE / REFINE / CORRECT。以一个完整填充的
|
||||
`IntegrateOutcome` 收尾。
|
||||
|
|
@ -62,8 +62,8 @@ class SearchStep(BaseStep):
|
|||
async def execute(self):
|
||||
assert self.context is not None
|
||||
query: str = (self.context.get("query", "") or "").strip()
|
||||
limit: int = int(self.context.get("limit", 5))
|
||||
min_score: float = float(self.context.get("min_score", 0.0))
|
||||
limit: int = int(self.context.get("limit") or 5)
|
||||
min_score: float = float(self.context.get("min_score") or 0.0)
|
||||
vector_weight: float = float(self.kwargs.get("vector_weight", 0.7))
|
||||
candidate_multiplier: float = float(self.kwargs.get("candidate_multiplier", 3.0))
|
||||
expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True))
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
"""Jobs steps — composite ReAct-agent-driven workflows."""
|
||||
|
||||
from . import digester # noqa: F401 -- @R.register("digester")
|
||||
|
|
@ -1,300 +0,0 @@
|
|||
"""Smart Digester — knowledge distillation from daily notes to digest/.
|
||||
|
||||
The Digester is the **cold-write** counterpart to AutoMemory (hot-write).
|
||||
It reads completed work in ``daily/<date>/<slug>.md`` note files,
|
||||
identifies entities / concepts / claims / methods worth preserving
|
||||
long-term, and sinks them into ``digest/`` as canonical-entry nodes so
|
||||
the main agent can retrieve them later via search and graph traversal.
|
||||
|
||||
``digest/`` is the cold-tier root. Per ``protocol.md``, scope folders
|
||||
under it may nest arbitrarily; each folder's canonical entry is
|
||||
``<folder>/<folder>.md``, and slug (folder name) is globally unique across
|
||||
the whole tree. Pending detection and graph machinery treat nodes at any
|
||||
depth uniformly.
|
||||
|
||||
Drives a ReAct agent with a read/lookup/graph/write toolkit; the
|
||||
agent follows the protocol in ``protocol.md`` (the opinionated
|
||||
default schema + R-M-W decision tree). The schema is convention-driven
|
||||
— reme core only reserves ``name`` / ``description``, so the agent
|
||||
owns its own discipline rather than relying on a post-write linter.
|
||||
|
||||
Distillation state lives in the daily note's ``status`` frontmatter
|
||||
— a **daily-tier convention owned by this digester** (reme core
|
||||
reserves only ``name`` / ``description``; ``status`` is just an
|
||||
extra). After processing each daily, the agent must call
|
||||
``frontmatter_update`` with ``metadata={"status": "completed"}``
|
||||
(or ``metadata={"status": "skipped"}`` when intentionally bypassed). Convention: absent
|
||||
≡ ``pending``, so the next pass finds residual work via
|
||||
``file_list path=daily recursive=true`` + per-item ``frontmatter_read`` to filter for absent ``status``.
|
||||
Only the digester writes ``status``; AutoMemory / hand-edits must
|
||||
leave it alone.
|
||||
|
||||
No degraded path — distillation strictly requires an LLM. When ``as_llm``
|
||||
is unavailable, the step short-circuits with ``skipped=True`` and an
|
||||
error message.
|
||||
|
||||
Override interface — schema is a service-consumption concern, not a
|
||||
core invariant, so this step ships an **opinionated default** that any
|
||||
caller can fully replace without touching reme4:
|
||||
|
||||
* ``protocol`` / ``protocol_path`` constructor args replace the
|
||||
``protocol.md`` injected as ``{protocol}`` in the system prompt
|
||||
(use when keeping the default prompt template but swapping schema).
|
||||
* ``prompt_dict`` (inherited from ``BaseStep``) replaces the
|
||||
``system_prompt`` / ``user_message`` templates wholesale (use when
|
||||
the prompt structure itself needs to change).
|
||||
* ``toolkit`` replaces the tool surface ``_DIGESTER_TOOLS`` builds.
|
||||
|
||||
Service layers (e.g. plugin-side configs) wire these in via component
|
||||
config; ``digester.py`` / ``protocol.md`` shipped here are just a
|
||||
reference implementation of one viable convention.
|
||||
|
||||
Toolkit. Each entry in ``_DIGESTER_TOOLS`` is a job name registered
|
||||
in the active config; ``add_as_tool`` wraps ``job(**kwargs)`` into a
|
||||
``ToolResponse``. The job indirection means the agent sees the same
|
||||
tool surface (rich descriptions + JSON schema) as the L2 MCP layer.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import zoneinfo
|
||||
from pathlib import Path
|
||||
|
||||
from agentscope.agent import ReActAgent
|
||||
from agentscope.message import Msg
|
||||
from agentscope.tool import Toolkit
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from ..base_step import BaseStep
|
||||
|
||||
from ...components import R
|
||||
|
||||
|
||||
_DIGESTER_TOOLS: tuple[str, ...] = (
|
||||
"file_list",
|
||||
"file_read",
|
||||
"file_stat",
|
||||
"frontmatter_read",
|
||||
"traverse",
|
||||
"file_write",
|
||||
"file_append",
|
||||
"file_move",
|
||||
"frontmatter_update",
|
||||
"frontmatter_delete",
|
||||
)
|
||||
|
||||
|
||||
def _pack_daily(file_store, daily_path: str) -> str:
|
||||
"""Render one daily note file's body into a prompt-friendly block.
|
||||
|
||||
A daily note is a single self-contained markdown file at
|
||||
``daily/<date>/<slug>.md`` — everything the originating task wanted
|
||||
the digester to see is inline (no sibling materials). External
|
||||
assets land in ``resource/<date>/`` and are linked from the note's
|
||||
``## References`` section; the LLM opens those on demand via
|
||||
``file_read``.
|
||||
"""
|
||||
try:
|
||||
absolute = (Path(file_store.vault_path or ".") / daily_path).resolve()
|
||||
except Exception as e:
|
||||
return f"### {daily_path}\n(error resolving path: {type(e).__name__}: {e})\n"
|
||||
|
||||
if not absolute.is_file():
|
||||
return f"### {daily_path}\n(note file not found)\n"
|
||||
|
||||
parts: list[str] = [f"### {daily_path}"]
|
||||
try:
|
||||
parts.append(absolute.read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
parts.append(f"(error reading note: {type(e).__name__}: {e})")
|
||||
return "\n".join(parts) + "\n"
|
||||
|
||||
|
||||
class DistillResult(BaseModel):
|
||||
"""Outcome of a single distillation call.
|
||||
|
||||
Without per-tool audit (the agent's toolkit is the job surface,
|
||||
which doesn't expose per-call records back to the orchestrator),
|
||||
the structured outcome is just the inputs the call was asked to
|
||||
process plus the LLM's free-form summary. Per-file write outcomes
|
||||
can be verified by re-reading vault_dir afterwards if needed.
|
||||
|
||||
Field semantics:
|
||||
* ``daily_read`` — daily paths actually processed (input
|
||||
order, deduped)
|
||||
* ``summary`` — LLM's free-form one-paragraph summary
|
||||
* ``skipped`` — True when no LLM available, no daily
|
||||
paths provided, or the LLM reported ``SKIP``
|
||||
* ``error`` — short error string when skipped due
|
||||
to misconfiguration (e.g. no LLM)
|
||||
"""
|
||||
|
||||
used_llm: bool = False
|
||||
skipped: bool = False
|
||||
daily_read: list[str] = Field(default_factory=list)
|
||||
summary: str = ""
|
||||
error: str = ""
|
||||
|
||||
|
||||
@R.register("digester")
|
||||
class Digester(BaseStep):
|
||||
"""Knowledge digester: daily/ → digest/ via a ReAct agent.
|
||||
|
||||
Inputs (from RuntimeContext):
|
||||
daily_paths (list[str], required): vault-relative paths to
|
||||
daily note files (``daily/<date>/<slug>.md``) to distill.
|
||||
Pass ``[]`` to no-op.
|
||||
hint (str, optional): caller guidance to the LLM
|
||||
(e.g. "focus on the auth-related decisions").
|
||||
|
||||
Output (written to context.response.answer):
|
||||
DistillResult JSON — see model docstring.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
toolkit: Toolkit | None = None,
|
||||
console_enabled: bool = False,
|
||||
timezone: str | None = None,
|
||||
protocol: str | None = None,
|
||||
protocol_path: str | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
"""Constructor overrides (service layer customization points):
|
||||
|
||||
* ``toolkit`` — replace the agent's tool surface; default builds
|
||||
one from ``_DIGESTER_TOOLS``.
|
||||
* ``protocol`` — inline protocol document (highest precedence);
|
||||
overrides whatever the agent sees under ``{protocol}`` in the
|
||||
system prompt.
|
||||
* ``protocol_path`` — path (relative to the vault or absolute) to a protocol
|
||||
markdown file; used when ``protocol`` is not given.
|
||||
* ``prompt_dict`` (inherited via ``BaseStep``) — override the
|
||||
``system_prompt`` / ``user_message`` templates wholesale, e.g.
|
||||
to swap in a service-layer prompt that hardcodes a different
|
||||
schema entirely.
|
||||
|
||||
With none of the above, falls back to the opinionated default
|
||||
(the ``protocol.md`` and ``digester.yaml`` shipped alongside
|
||||
this module).
|
||||
"""
|
||||
super().__init__(**kwargs)
|
||||
self.toolkit = toolkit
|
||||
self.console_enabled = console_enabled
|
||||
self.timezone = timezone
|
||||
self._protocol = self._load_protocol(protocol, protocol_path)
|
||||
|
||||
@staticmethod
|
||||
def _load_protocol(protocol: str | None, protocol_path: str | None) -> str:
|
||||
"""Resolve the protocol document; explicit string > path > default."""
|
||||
if protocol is not None:
|
||||
return protocol
|
||||
if protocol_path:
|
||||
path = Path(protocol_path)
|
||||
if path.exists():
|
||||
return path.read_text(encoding="utf-8")
|
||||
default_path = Path(__file__).parent / "protocol.md"
|
||||
return default_path.read_text(encoding="utf-8") if default_path.exists() else ""
|
||||
|
||||
def _now(self) -> datetime.datetime:
|
||||
if self.timezone:
|
||||
try:
|
||||
return datetime.datetime.now(zoneinfo.ZoneInfo(self.timezone))
|
||||
except Exception as e:
|
||||
self.logger.error(f"Invalid timezone: {self.timezone}, error={e}")
|
||||
return datetime.datetime.now()
|
||||
|
||||
def _vault_dir(self) -> Path:
|
||||
vr = getattr(self.file_store, "vault_path", None)
|
||||
return Path(vr).resolve() if vr else Path.cwd().resolve()
|
||||
|
||||
def _llm_available(self) -> bool:
|
||||
"""Pre-flight check: can ``self.as_llm`` resolve without raising?
|
||||
|
||||
``BaseStep.as_llm`` asserts when no model is registered, so we
|
||||
wrap the access here to avoid hard-failing at the call site."""
|
||||
try:
|
||||
return self.as_llm is not None
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _build_toolkit(self) -> Toolkit:
|
||||
"""Bind every digester-relevant job as a tool function."""
|
||||
toolkit = self.toolkit or Toolkit()
|
||||
for job_name in _DIGESTER_TOOLS:
|
||||
self.add_as_tool(toolkit, job_name)
|
||||
return toolkit
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
daily_paths: list[str] = list(self.context.get("daily_paths") or [])
|
||||
hint: str = (self.context.get("hint", "") or "").strip()
|
||||
|
||||
# No work to do: no daily paths supplied.
|
||||
if not daily_paths:
|
||||
result = DistillResult(used_llm=False, skipped=True)
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = "Skipped: no daily paths supplied"
|
||||
self.context.response.metadata.update(result.model_dump())
|
||||
return
|
||||
|
||||
# No LLM available: distillation strictly requires one.
|
||||
if not self._llm_available():
|
||||
result = DistillResult(
|
||||
used_llm=False,
|
||||
skipped=True,
|
||||
error="no as_llm configured; distillation requires an LLM",
|
||||
)
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"Error: {result.error}"
|
||||
self.context.response.metadata.update(result.model_dump())
|
||||
return
|
||||
|
||||
# Dedupe daily_paths while preserving order.
|
||||
seen: set[str] = set()
|
||||
deduped: list[str] = []
|
||||
for p in daily_paths:
|
||||
if p and p not in seen:
|
||||
seen.add(p)
|
||||
deduped.append(p)
|
||||
daily_paths = deduped
|
||||
|
||||
# Build the per-daily blob the agent will see.
|
||||
daily_blob = "\n\n".join(_pack_daily(self.file_store, p) for p in daily_paths)
|
||||
|
||||
vault_dir = self._vault_dir()
|
||||
toolkit = self._build_toolkit()
|
||||
|
||||
agent = ReActAgent(
|
||||
name="reme_digester",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
"system_prompt",
|
||||
vault_dir=str(vault_dir),
|
||||
protocol=self._protocol,
|
||||
),
|
||||
formatter=self.as_llm_formatter,
|
||||
toolkit=toolkit,
|
||||
)
|
||||
agent.set_console_output_enabled(self.console_enabled)
|
||||
|
||||
user_message: str = self.prompt_format(
|
||||
"user_message",
|
||||
today=self._now().strftime("%Y-%m-%d"),
|
||||
hint=hint or "(none)",
|
||||
daily_blob=daily_blob or "(none)",
|
||||
)
|
||||
|
||||
final_msg: Msg = await agent.reply(
|
||||
Msg(name="reme", role="user", content=user_message),
|
||||
)
|
||||
summary = (final_msg.get_text_content() or "").strip()
|
||||
|
||||
result = DistillResult(
|
||||
used_llm=True,
|
||||
daily_read=list(daily_paths),
|
||||
summary=summary,
|
||||
skipped=summary.upper().startswith("SKIP"),
|
||||
)
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = summary or "Distillation completed"
|
||||
self.context.response.metadata.update(result.model_dump())
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
system_prompt: |
|
||||
You are the digester. You read completed work in `daily/<date>/<slug>.md`
|
||||
note files and lift entities, concepts, claims, and methods into
|
||||
canonical entries under `digest/<…>/<slug>/<slug>.md`. The main agent
|
||||
retrieves them later via search and graph traversal.
|
||||
|
||||
vault_dir: {vault_dir}
|
||||
|
||||
## Five steps per call
|
||||
|
||||
### Step 1 — Read each daily
|
||||
|
||||
For every daily note passed in, the full body is already packed
|
||||
in the user message below. Each note is a single self-contained
|
||||
markdown file — anything the originating task wanted you to see is
|
||||
inline. If a note's `## References` section points at
|
||||
`[[resource/<date>/<name>]]` items and you need them, open them via
|
||||
`file_read` on demand.
|
||||
|
||||
### Step 2 — Lookup candidates
|
||||
|
||||
Identify each entity / concept / claim / method named in the daily.
|
||||
Before deciding to CREATE anything, look it up in `digest/`:
|
||||
|
||||
- `file_list path=digest/ recursive=true` to scan the tree, or
|
||||
- `graph_traverse path=<candidate>.md depth=1` for neighborhood.
|
||||
|
||||
**Slugs are globally unique under `digest/`** — a prior occurrence
|
||||
at any nesting depth means the node already exists. Find and reuse
|
||||
it; do not CREATE a duplicate under a new path. This is the worst
|
||||
failure mode of the digester.
|
||||
|
||||
### Step 3 — R-M-W decision
|
||||
|
||||
Per candidate, pick exactly one branch. Every branch writes only
|
||||
the node being authored in this step — never sideways into other
|
||||
nodes' bodies. A relation worth recording is captured by a typed
|
||||
wikilink in the source body; the inbound view is queried later via
|
||||
`graph_traverse direction=in`.
|
||||
|
||||
- **CREATE** — no hit anywhere under `digest/`:
|
||||
`file_write digest/<…>/<slug>/<slug>.md`. Place it under the
|
||||
closest existing semantic parent scope; top-level if none applies.
|
||||
|
||||
- **UPDATE** — exact match exists:
|
||||
Merge new facts into the right section. Prefer
|
||||
`frontmatter_update` for metadata; `file_append` for purely
|
||||
additive trailing sections; `file_read` + `file_write` for
|
||||
mid-body edits. Override stale claims rather than stacking
|
||||
contradictions. Don't restructure unrelated parts.
|
||||
|
||||
- **MOVE / promote** — rename or relocate an existing node:
|
||||
`file_move` (the retarget pass rewrites inbound wikilinks
|
||||
atomically; never `file_write` to new + `file_delete` old).
|
||||
|
||||
When CREATE / UPDATE writes a relation into the body, prefer a
|
||||
typed wikilink (`predicate:: [[X]]` or `[predicate:: [[X]]]`) when
|
||||
the relation has clear semantic weight; default to bare `[[X]]` for
|
||||
plain mention. See protocol.md §4.1 for the recommended predicate
|
||||
vocabulary.
|
||||
|
||||
If a candidate is only a passing mention with no new fact to write,
|
||||
do nothing — the mention stays in the daily, search will still
|
||||
find it, and a future digester pass can lift it when it accumulates
|
||||
substance worth CREATE/UPDATE.
|
||||
|
||||
### Step 4 — Flip status per daily (mandatory)
|
||||
|
||||
After processing each daily, call:
|
||||
|
||||
frontmatter_update path=daily/<date>/<slug>.md metadata={status: completed}
|
||||
|
||||
Use `status=skipped` if the daily had nothing worth lifting (chitchat,
|
||||
dead end). This flip is the daily-tier convention this digester owns
|
||||
— absent ≡ pending, so the next pass uses
|
||||
`file_list path=daily recursive=true` + per-item `frontmatter_read`
|
||||
to find leftover work. Forgetting the flip leaves the daily in the
|
||||
queue forever.
|
||||
|
||||
### Step 5 — Reply
|
||||
|
||||
One short paragraph: which dailies you read, what you CREATEd /
|
||||
UPDATEd / MOVEd (with paths), and which dailies you flipped to
|
||||
`completed` vs `skipped`. Don't replay every tool call — those are
|
||||
on the audit trail.
|
||||
|
||||
## Wikilinks
|
||||
|
||||
Always full path relative to the vault with `.md`:
|
||||
`[[digest/<…>/<slug>/<slug>.md]]` for digest entries,
|
||||
`[[daily/<date>/<slug>.md]]` for daily notes,
|
||||
`[[resource/<date>/<name>]]` for ingested resources. Short forms or
|
||||
extension-less forms don't resolve.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- **Never write under `daily/`** except for the Step 4 `status` flip.
|
||||
- **Only this digester writes `status`** — sync / hand-edits leave
|
||||
it alone (absent ≡ pending is exactly how this digester finds its
|
||||
workload).
|
||||
|
||||
## Memory protocol
|
||||
|
||||
{protocol}
|
||||
|
||||
user_message: |
|
||||
today: {today}
|
||||
hint: {hint}
|
||||
|
||||
# Daily notes to distill
|
||||
|
||||
{daily_blob}
|
||||
|
||||
Run the five steps from the system prompt. Reply = one paragraph audit.
|
||||
|
|
@ -1,112 +0,0 @@
|
|||
# Memory Protocol
|
||||
|
||||
Opinionated default contract for writing memory into a reme vault.
|
||||
reme core reserves only `name` / `description` (both optional);
|
||||
everything below is convention that consumers may replace.
|
||||
|
||||
## 1. Directory architecture
|
||||
|
||||
```
|
||||
<vault>/
|
||||
├── daily/
|
||||
│ └── <YYYY-MM-DD>/
|
||||
│ └── <slug>/
|
||||
│ ├── <slug>.md # hot summary note (markdown)
|
||||
│ └── <material>.* # sibling materials (any file type)
|
||||
└── digest/
|
||||
└── <slug>/
|
||||
├── <slug>.md # cold canonical entry
|
||||
├── <material>.md # supporting docs
|
||||
└── <subslug>/ # nested narrower scope
|
||||
└── <subslug>.md
|
||||
```
|
||||
|
||||
- **Hot tier** (`daily/…`) — streaming. One upstream writer per
|
||||
folder; every other consumer treats it as read-only. The
|
||||
summary note `<slug>.md` is markdown; siblings may be any
|
||||
file type the writer chooses.
|
||||
- **Cold tier** (`digest/…`) — curated. Each folder is a
|
||||
scope and must contain `<folder>/<folder>.md` as its canonical
|
||||
entry. A scope's other children are sibling material files or
|
||||
narrower scope subfolders; nesting depth is unconstrained.
|
||||
Slugs are globally unique — a folder name appears at most once
|
||||
anywhere under `digest/`.
|
||||
|
||||
Facts flow one-way, `daily/` → `digest/`. References use the
|
||||
full path relative to the vault: `[[digest/<slug>/<subslug>/<subslug>.md]]`.
|
||||
|
||||
## 2. Frontmatter
|
||||
|
||||
Reserved (typed; all optional):
|
||||
|
||||
| key | type |
|
||||
|---|---|
|
||||
| `name` | string |
|
||||
| `description` | string |
|
||||
|
||||
Opinionated default axes (closed enums):
|
||||
|
||||
| key | values |
|
||||
|---|---|
|
||||
| `lifecycle` | `streaming` / `evolving` / `frozen` |
|
||||
| `scope` | `instance` / `class` |
|
||||
| `source` | `auto` / `curated` / `derived` |
|
||||
| `role` | `profile` / `concept` / `claim` / `method` / `reference` / `observation` / `question` / `fundamentals` |
|
||||
|
||||
Any other keys consumers want (e.g. a workflow `status` flag) live
|
||||
as extras — write them, read them with the `where` filter on `list`
|
||||
tools (`null` matches absent-or-null); the protocol does not name
|
||||
or enumerate them.
|
||||
|
||||
## 3. Body
|
||||
|
||||
Section structure is **advisory** — `## Summary`, `## Key Facts`,
|
||||
`## Decisions`, `## Related` are convenient defaults but the
|
||||
protocol mandates no specific section.
|
||||
|
||||
## 4. Wikilinks
|
||||
|
||||
Three recognized forms:
|
||||
|
||||
| Form | Example | Meaning |
|
||||
|---|---|---|
|
||||
| Bare | `See [[张三.md]]` | weakest layer — "mention" |
|
||||
| Line-level Dataview | `colleague:: [[李四.md]]` | typed relation, queryable by predicate |
|
||||
| Inline-bracketed Dataview | `主导 [负责:: [[项目X.md]]] 的重构` | typed relation, embedded inline |
|
||||
|
||||
Targets are stored **verbatim** as full paths relative to the vault.
|
||||
`[[digest/zhang-san/zhang-san.md]]` resolves; short or
|
||||
extension-less forms do not — no implicit `.md` completion, no
|
||||
basename search, no folder-note expansion.
|
||||
|
||||
Renaming a node requires atomically rewriting every inbound
|
||||
wikilink.
|
||||
|
||||
### 4.1 Typed predicates (half-open)
|
||||
|
||||
Recommended core vocabulary — writers may extend beyond this set,
|
||||
but new predicates should be reused consistently:
|
||||
|
||||
| predicate | meaning |
|
||||
|---|---|
|
||||
| `is_a` | hierarchical (X is a kind of Y) |
|
||||
| `part_of` | containment (X is part of Y) |
|
||||
| `depends_on` | dependency (X requires Y) |
|
||||
| `manages` | authority / responsibility |
|
||||
| `alias_of` | equivalence (X and Y are the same thing) |
|
||||
| `references` | citation / external pointer |
|
||||
|
||||
Use typed `predicate:: [[X]]` only when the relation has clear
|
||||
semantic weight; default to bare `[[X]]` for plain mentions. Typed
|
||||
edges become queryable via `graph_traverse predicate=<name>`.
|
||||
|
||||
### 4.2 One-way write rule
|
||||
|
||||
A wikilink lives in the **source** node's body only — the node whose
|
||||
prose introduces the relation. The **target** is never modified to
|
||||
record the inbound relation. Backlinks are discovered at query time
|
||||
via `graph_traverse direction=in`, never written into target bodies.
|
||||
|
||||
This keeps every write authoritative: a node's body reflects only
|
||||
what its own author/writer chose to say, never sideways annotations
|
||||
from other nodes' writers.
|
||||
|
|
@ -63,7 +63,7 @@ Parameters:
|
|||
only.
|
||||
* ``description`` (required) — analysis hint for downstream agents:
|
||||
where the asset came from, what kind of content it carries, and how
|
||||
it should be interpreted. The digester / auto_memory reads this
|
||||
it should be interpreted. The dreamer / auto_memory reads this
|
||||
verbatim from ``meta.json`` to decide how to read the asset (skim
|
||||
vs. deep parse, structured extraction vs. summarization, etc.), so
|
||||
callers should write enough detail to drive that decision — not
|
||||
|
|
|
|||
287
tests4/integration/_dreamer_fixture.py
Normal file
287
tests4/integration/_dreamer_fixture.py
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
"""Fixture for the dreamer integration tests.
|
||||
|
||||
Seeds a vault with:
|
||||
|
||||
- 4 pre-existing digest/ nodes spread across the three buckets
|
||||
(procedure / personal / wiki) — these are the **recall targets**;
|
||||
the new material partially overlaps them so Phase 2 must find
|
||||
them via search + read and decide UPDATE.
|
||||
- 4 small daily/ stubs the digest nodes already link to — they exist
|
||||
only so the seeded digest bodies don't dangle.
|
||||
- 1 NEW daily note (the file the dreamer will be invoked on).
|
||||
It exercises CREATE and UPDATE across the three buckets:
|
||||
|
||||
* wiki : UPDATE digest/wiki/jwt.md (24h rotation cadence
|
||||
refines short-credential-compliance framing) +
|
||||
CREATE digest/wiki/kid-versioning.md +
|
||||
CREATE digest/wiki/soc2-30day-finding.md
|
||||
(the OAuth2 restatement section is intentionally
|
||||
a non-abstraction — Phase 1 should NOT emit a
|
||||
sub-unit for it; tests Phase 1's gate-keeping)
|
||||
* procedure : UPDATE digest/procedure/key-rotation.md (24h
|
||||
cadence + kid-versioning supersede the 30-day
|
||||
JWKS-cache flow)
|
||||
* personal : UPDATE digest/personal/no-trailing-summary.md
|
||||
(extend "no trailing summary" to also forbid
|
||||
"next steps" lists) + CREATE
|
||||
digest/personal/small-pr.md
|
||||
|
||||
Total budget per integration run: 1 Phase 1 + up-to-6 Phase 2 = up
|
||||
to 7 ReAct sessions, each with several tool turns (search +
|
||||
traverse → frontmatter_read + read → write / edit).
|
||||
|
||||
Idempotent: re-running does NOT overwrite existing files. To re-seed
|
||||
from scratch, delete the vault and rerun.
|
||||
|
||||
Usage as a script:
|
||||
python tests4/integration/_dreamer_fixture.py /tmp/my-vault
|
||||
|
||||
Usage as a module:
|
||||
from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH
|
||||
clean_vault(Path("/tmp/my-vault"))
|
||||
seed_vault(Path("/tmp/my-vault"))
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
INPUT_PATH = "daily/2026-05-28/auth-refactor/notes.md"
|
||||
|
||||
|
||||
_FILES: dict[str, str] = {
|
||||
# ----- pre-existing digest nodes (recall targets) -----
|
||||
"digest/wiki/jwt.md": """\
|
||||
---
|
||||
name: jwt
|
||||
description: JSON Web Token — signed authentication token format
|
||||
---
|
||||
|
||||
# JWT
|
||||
|
||||
JSON Web Token (RFC 7519). A compact, signed (JWS) or encrypted (JWE)
|
||||
token used to assert identity and claims between parties.
|
||||
|
||||
## Structure
|
||||
- Header — `alg`, `typ`, `kid`
|
||||
- Payload — claims: `iss`, `sub`, `aud`, `exp`, `iat`
|
||||
- Signature
|
||||
|
||||
## Related
|
||||
Often issued by [[digest/wiki/oauth2.md]] flows.
|
||||
|
||||
derived_from:: [[daily/2026-05-15/auth-design/notes.md]]
|
||||
""",
|
||||
"digest/wiki/oauth2.md": """\
|
||||
---
|
||||
name: oauth2
|
||||
description: OAuth 2.0 — delegated authorization framework
|
||||
---
|
||||
|
||||
# OAuth 2.0
|
||||
|
||||
RFC 6749. A delegated authorization framework: a resource owner grants
|
||||
a client limited access to a protected resource via an access token
|
||||
issued by an authorization server.
|
||||
|
||||
## Grant types
|
||||
- Authorization code (with PKCE for public clients)
|
||||
- Client credentials
|
||||
- Refresh token
|
||||
|
||||
derived_from:: [[daily/2026-05-10/oauth-intro/notes.md]]
|
||||
""",
|
||||
"digest/procedure/key-rotation.md": """\
|
||||
---
|
||||
name: key-rotation
|
||||
description: Rotating signing keys for JWT issuance
|
||||
---
|
||||
|
||||
# Key rotation (current — pre 2026-05-28 refactor)
|
||||
|
||||
Procedure for rotating the signing key used by [[digest/wiki/jwt.md]]
|
||||
issuance.
|
||||
|
||||
## Steps
|
||||
1. Generate new keypair offline.
|
||||
2. Publish the public key to the JWKS endpoint with a fresh `kid`.
|
||||
3. Wait 24h for clients to refresh their JWKS cache.
|
||||
4. Cut over the signer to the new private key.
|
||||
5. Mark the old `kid` as deprecated; remove after 30 days.
|
||||
|
||||
## Cadence
|
||||
Default rotation cadence is **30 days**. Driven by historical practice;
|
||||
no formal compliance requirement has tightened this so far.
|
||||
|
||||
derived_from:: [[daily/2026-05-20/rotation-plan/notes.md]]
|
||||
""",
|
||||
"digest/personal/no-trailing-summary.md": """\
|
||||
---
|
||||
name: no-trailing-summary
|
||||
description: 不要在回复末尾加总结段落
|
||||
---
|
||||
|
||||
# 不要在回复末尾加总结段落
|
||||
|
||||
用户能看 diff,不需要在回复末尾重述刚做的事。
|
||||
|
||||
**Why**: diff 已经把"改了什么"摆在用户面前;再口述一遍是噪音。
|
||||
|
||||
**How to apply**: 任意编码 / 编辑任务回复结束时,直接停在最后一条
|
||||
有信息量的话上,不要再补一段"以上就是本次的修改..."。
|
||||
|
||||
derived_from:: [[daily/2026-05-01/style-feedback/notes.md]]
|
||||
""",
|
||||
# ----- daily provenance stubs (so the digest links don't dangle) -----
|
||||
"daily/2026-05-01/style-feedback/notes.md": """\
|
||||
---
|
||||
name: notes
|
||||
description: style feedback to Claude on 2026-05-01
|
||||
---
|
||||
|
||||
# Style feedback (2026-05-01)
|
||||
|
||||
每次任务结束都重述了一遍刚做的事——不需要,我能看 diff。以后直接停。
|
||||
""",
|
||||
"daily/2026-05-10/oauth-intro/notes.md": """\
|
||||
---
|
||||
name: notes
|
||||
description: OAuth 2.0 intro session
|
||||
---
|
||||
|
||||
# OAuth 2.0 intro
|
||||
|
||||
简介 grant types: authorization code (with PKCE), client credentials,
|
||||
refresh token。重点放在 PKCE 是给 public clients 用的。
|
||||
""",
|
||||
"daily/2026-05-15/auth-design/notes.md": """\
|
||||
---
|
||||
name: notes
|
||||
description: initial auth design discussion
|
||||
---
|
||||
|
||||
# Auth design
|
||||
|
||||
讨论 JWT 的结构 (header / payload / signature) 和我们项目里的 claim
|
||||
约定 (iss, sub, aud, exp, iat)。
|
||||
""",
|
||||
"daily/2026-05-20/rotation-plan/notes.md": """\
|
||||
---
|
||||
name: notes
|
||||
description: key rotation plan v1
|
||||
---
|
||||
|
||||
# Key rotation plan v1
|
||||
|
||||
定下当前的 5 步轮换流程:offline 生成 keypair → 发布到 JWKS (新 kid)
|
||||
→ 等 24h cache → 切签发 → 30 天后清旧 kid。周期定 30 天。
|
||||
""",
|
||||
# ----- the NEW daily note dreamer will be invoked on -----
|
||||
INPUT_PATH: """\
|
||||
---
|
||||
name: notes
|
||||
description: auth refactor working notes — 2026-05-28
|
||||
---
|
||||
|
||||
# Auth refactor — 2026-05-28
|
||||
|
||||
## 决定:JWT 轮换周期改为 24 小时
|
||||
|
||||
今天确定把 JWT 签名密钥的轮换周期从 30 天压到 **24 小时**。原因是
|
||||
SOC2 合规审计批评:30 天的会话 token 太长,不满足"短期凭证"原则。
|
||||
|
||||
新流程不再依赖 JWKS cache 的 24h 等待,改成走 Redis 里的 `kid`
|
||||
版本号实时下发。客户端在 token 验证失败时主动拉新 JWKS,而不是定
|
||||
时轮询。
|
||||
|
||||
(这条同时更新 JWT 概念笔记和 key-rotation 流程笔记。)
|
||||
|
||||
## 新概念:kid 版本号机制
|
||||
|
||||
`kid` (key ID) 是 JWT header 里的字段。我们把它当成版本号来用:
|
||||
Redis key `auth:jwks:current_kid` 保存当前活跃 kid;Auth Service
|
||||
在签发 token 时读这个 key,客户端验证失败时也读这个 key 再拉对应
|
||||
的 public key。这样无须等 cache TTL。
|
||||
|
||||
## 顺带复习:OAuth 2.0 是什么
|
||||
|
||||
(为了帮新同学接住上下文,这里把 OAuth 2.0 简单重述一下,不引入
|
||||
新事实。)OAuth 2.0 (RFC 6749) 是一个委托授权框架:资源所有者
|
||||
允许 client 通过 authorization server 颁发的 access token 来有
|
||||
限度地访问受保护资源。常见 grant types: authorization code
|
||||
(public client 用 PKCE)、client credentials、refresh token。
|
||||
——这一段没有任何新内容,纯粹是给后面 JWT 24h 轮换决定铺垫读者
|
||||
的背景知识。
|
||||
|
||||
## 观察:SOC2 审计在 30 天周期上的具体批评
|
||||
|
||||
审计员引用 SOC2 CC6.1 控制点:"会话凭证应有合理的短期有效期"。
|
||||
30 天对应于人类工作周期,但对自动化客户端 token 来说过长。审计
|
||||
要求 24h 或更短,且必须能在事件响应时立即吊销 (kid 切换可满足)。
|
||||
|
||||
## 偏好:小 PR 优先
|
||||
|
||||
后续这个 refactor 拆 PR 时,每个 PR 控制在 < 300 行。原因是 review
|
||||
负担太大时容易被拍脑袋通过,这违背了 SOC2 审计中变更管理的精神。
|
||||
|
||||
## 偏好:回复结尾再补充
|
||||
|
||||
之前说过不要总结段落 (我能看 diff),今天再补充一点:也不要"接下来
|
||||
的步骤"列表,除非我明确问 next steps。直接回答问题然后停。
|
||||
""",
|
||||
}
|
||||
|
||||
|
||||
_CLEAN_DIRS = ("daily", "digest", "reme_metadata")
|
||||
|
||||
|
||||
def clean_vault(vault: Path) -> list[str]:
|
||||
"""Remove fixture-managed subdirs (`daily/`, `digest/`, `reme_metadata/`)
|
||||
under `vault` so the next `seed_vault` starts from a clean slate.
|
||||
|
||||
Returns the relative paths that were actually removed."""
|
||||
removed: list[str] = []
|
||||
for rel in _CLEAN_DIRS:
|
||||
target = vault / rel
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
removed.append(rel)
|
||||
return removed
|
||||
|
||||
|
||||
def seed_vault(vault: Path) -> list[str]:
|
||||
"""Write any missing fixture files under `vault`. Return relative paths
|
||||
that were actually written (skipped existing ones)."""
|
||||
seeded: list[str] = []
|
||||
for rel, body in _FILES.items():
|
||||
target = vault / rel
|
||||
if target.exists():
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
seeded.append(rel)
|
||||
return seeded
|
||||
|
||||
|
||||
# pylint: disable=missing-function-docstring
|
||||
def main() -> None:
|
||||
if len(sys.argv) < 2:
|
||||
print(f"usage: {sys.argv[0]} <vault_dir>", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
vault = Path(sys.argv[1]).resolve()
|
||||
vault.mkdir(parents=True, exist_ok=True)
|
||||
removed = clean_vault(vault)
|
||||
if removed:
|
||||
print(f"cleaned {len(removed)} dir(s) under {vault}: {', '.join(removed)}")
|
||||
seeded = seed_vault(vault)
|
||||
if seeded:
|
||||
print(f"seeded {len(seeded)} file(s) under {vault}:")
|
||||
for f in seeded:
|
||||
print(f" + {f}")
|
||||
else:
|
||||
print(f"vault {vault} already seeded — no changes")
|
||||
print(f"\nDream this file:\n {vault}/{INPUT_PATH}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
68
tests4/integration/test_dreamer_cli.sh
Executable file
68
tests4/integration/test_dreamer_cli.sh
Executable file
|
|
@ -0,0 +1,68 @@
|
|||
#!/usr/bin/env bash
|
||||
# dreamer CLI integration test (option B).
|
||||
#
|
||||
# Seeds a rich workspace via _dreamer_fixture.py, starts `reme start`
|
||||
# bound to that vault, reindexes so Phase 2 recall can hit the pre-
|
||||
# seeded digest nodes, then calls `reme dream`.
|
||||
#
|
||||
# Usage (from anywhere):
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh
|
||||
# VAULT_PATH=/tmp/reme-dreamer-test bash tests4/integration/test_dreamer_cli.sh daily/2026-05-28/auth-refactor/notes.md
|
||||
#
|
||||
# Defaults:
|
||||
# VAULT_PATH unset → /tmp/reme-dreamer-test
|
||||
# Workspace seeded on first run (idempotent).
|
||||
#
|
||||
# Required env (from .env or shell):
|
||||
# LLM_API_KEY, LLM_BASE_URL, LLM_MODEL_NAME
|
||||
set -euo pipefail
|
||||
|
||||
VAULT="${VAULT_PATH:-/tmp/reme-dreamer-test}"
|
||||
INTEGRATION_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO="$(cd "$INTEGRATION_DIR/../.." && pwd)"
|
||||
LOG="/tmp/test_dreamer_cli_server.log"
|
||||
|
||||
# Resolve input from arg, else default to fixture's input path.
|
||||
DEFAULT_INPUT="$(python -c "import sys; sys.path.insert(0, '$INTEGRATION_DIR'); from _dreamer_fixture import INPUT_PATH; print(INPUT_PATH)")"
|
||||
INPUT="${1:-$DEFAULT_INPUT}"
|
||||
|
||||
mkdir -p "$VAULT"
|
||||
echo "--- seeding fixture under $VAULT"
|
||||
python "$INTEGRATION_DIR/_dreamer_fixture.py" "$VAULT"
|
||||
|
||||
cd "$REPO"
|
||||
|
||||
echo ""
|
||||
echo "--- starting reme server (log: $LOG)"
|
||||
reme start "vault_dir=$VAULT" >"$LOG" 2>&1 &
|
||||
SERVER_PID=$!
|
||||
trap 'echo "--- stopping reme server (pid $SERVER_PID)"; kill "$SERVER_PID" 2>/dev/null || true; wait "$SERVER_PID" 2>/dev/null || true' EXIT
|
||||
|
||||
echo "--- waiting for server"
|
||||
for _ in $(seq 1 30); do
|
||||
if curl -s -o /dev/null http://localhost:8000/docs 2>/dev/null; then
|
||||
echo "--- server up"
|
||||
break
|
||||
fi
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- reindexing vault so Phase 2 recall has something to hit"
|
||||
reme reindex
|
||||
|
||||
echo ""
|
||||
echo "=== reme dream path=$INPUT ==="
|
||||
reme dream "path=$INPUT"
|
||||
echo ""
|
||||
|
||||
echo "=== digest/ tree after dream ==="
|
||||
if [ -d "$VAULT/digest" ]; then
|
||||
find "$VAULT/digest" -name "*.md" | sort | while read -r f; do
|
||||
echo ""
|
||||
echo "--- ${f#$VAULT/} ---"
|
||||
cat "$f"
|
||||
done
|
||||
else
|
||||
echo " (no digest/ created)"
|
||||
fi
|
||||
110
tests4/integration/test_dreamer_inproc.py
Normal file
110
tests4/integration/test_dreamer_inproc.py
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
"""dreamer in-process integration test.
|
||||
|
||||
Loads the default reme4 config, seeds a rich workspace (pre-existing
|
||||
digest nodes spread across the three buckets + a new daily that
|
||||
exercises CREATE and UPDATE in each bucket), reindexes so search can
|
||||
hit the pre-existing nodes, then calls `dream` and prints what
|
||||
happened.
|
||||
|
||||
Phase 1 classifies each sub-unit into one of {procedure, personal,
|
||||
wiki}; Phase 2 dispatches to the bucket-specific integrate prompt
|
||||
and writes via the canonical `write` / `edit` tools.
|
||||
|
||||
Usage (from anywhere):
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py \\
|
||||
daily/2026-05-28/auth-refactor/notes.md
|
||||
|
||||
Defaults:
|
||||
VAULT_PATH unset → /tmp/reme-dreamer-test
|
||||
Each run wipes `daily/`, `digest/`, and `reme_metadata/` under the
|
||||
vault before reseeding, so the dreamer always starts from the same
|
||||
fixture state. See _dreamer_fixture.py for what gets created and
|
||||
the expected CREATE / UPDATE landings per bucket.
|
||||
|
||||
Required env (from .env or shell):
|
||||
LLM_API_KEY, LLM_BASE_URL, LLM_MODEL_NAME — for the Phase 1/2 agents
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Make `reme4` importable regardless of the caller's cwd; and make the
|
||||
# fixture module importable as a top-level name.
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
INTEGRATION_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
sys.path.insert(0, str(INTEGRATION_DIR))
|
||||
|
||||
# pylint: disable=wrong-import-position
|
||||
from _dreamer_fixture import clean_vault, seed_vault, INPUT_PATH # noqa: E402
|
||||
|
||||
VAULT = os.environ.get("VAULT_PATH", "/tmp/reme-dreamer-test")
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
"""Main function for testing the ReMeFs CLI."""
|
||||
from reme4 import ReMe # noqa: E402
|
||||
from reme4.config import resolve_app_config # noqa: E402
|
||||
from reme4.utils import load_env # noqa: E402
|
||||
|
||||
os.chdir(REPO_ROOT) # so load_env() picks up the repo's .env
|
||||
load_env()
|
||||
|
||||
vault = Path(VAULT).resolve()
|
||||
vault.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
removed = clean_vault(vault)
|
||||
if removed:
|
||||
print(f"--- cleaned {len(removed)} dir(s) under {vault}: {', '.join(removed)}")
|
||||
seeded = seed_vault(vault)
|
||||
print(f"--- seeded {len(seeded)} fixture file(s) under {vault}")
|
||||
|
||||
rel_input = sys.argv[1] if len(sys.argv) > 1 else INPUT_PATH
|
||||
|
||||
cfg = resolve_app_config(vault_dir=str(vault))
|
||||
print(f"--- vault_dir: {cfg.get('vault_dir')}")
|
||||
print(f"--- input: {rel_input}")
|
||||
|
||||
app = ReMe(**cfg)
|
||||
await app.start()
|
||||
try:
|
||||
# Reindex first so search can actually find the pre-seeded
|
||||
# digest/ nodes — otherwise Phase 2 recall returns empty and
|
||||
# every sub-unit ends up as CREATE (UPDATE path not exercised).
|
||||
print("\n--- reindexing vault so Phase 2 recall has something to hit")
|
||||
await app.run_job("reindex")
|
||||
|
||||
print(f"\n--- running dream path={rel_input}")
|
||||
resp = await app.run_job("dream", path=rel_input)
|
||||
|
||||
print("\n=== Response.success ===")
|
||||
print(resp.success)
|
||||
print("\n=== Response.answer ===")
|
||||
print(resp.answer)
|
||||
print("\n=== Response.metadata (DreamResult fields) ===")
|
||||
for k, v in (resp.metadata or {}).items():
|
||||
if isinstance(v, list) and len(v) > 8:
|
||||
print(f" {k}: list({len(v)} items) head={v[:3]!r}")
|
||||
else:
|
||||
print(f" {k}: {v!r}")
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
print("\n=== digest/ tree after dream ===")
|
||||
digest_root = vault / "digest"
|
||||
if not digest_root.exists():
|
||||
print(" (no digest/ created)")
|
||||
return
|
||||
files = sorted(digest_root.rglob("*.md"))
|
||||
if not files:
|
||||
print(" (digest/ is empty)")
|
||||
for p in files:
|
||||
print(f"\n--- {p.relative_to(vault)} ---")
|
||||
# print(p.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
|
@ -542,7 +542,7 @@ def test_upload_preserves_description_verbatim_in_meta():
|
|||
payload = _metadata(step)
|
||||
assert "error" not in payload, payload
|
||||
|
||||
# meta.json preserves the original — downstream digester sees full hint.
|
||||
# meta.json preserves the original — downstream dreamer sees full hint.
|
||||
meta = _meta(tmp, payload["date"])
|
||||
assert meta[0]["front_matter"]["description"] == multi
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue