diff --git a/docs4/auto_dream_logic_and_step_refactor.md b/docs4/auto_dream_logic_and_step_refactor.md new file mode 100644 index 00000000..46aa4942 --- /dev/null +++ b/docs4/auto_dream_logic_and_step_refactor.md @@ -0,0 +1,943 @@ +# auto_dream 逻辑解读与 Step 拆分方案 + +## 1. 配置入口 + +`/Users/yuli/workspace/ReMe/reme4/config/default.yaml` 里和 auto dream 相关的是四个 job: + +| job | 用途 | 当前 steps | +|---|---|---| +| `dream` | 对单个文件做完整 dream,并顺手写一次 daily topics | `dream_step` -> `daily_topics_step` | +| `dream_extract` | auto_dream 内部使用的单文件 dream,只抽取和整合 digest,不写 daily topics | `dream_step` | +| `auto_dream` | 扫描某一天的 daily index 和 session notes,只处理新增/修改文件,最后聚合写当天兴趣主题 | `auto_dream_step` | +| `daily_topics` | 从 dream 产生的 topic candidates 中选最终兴趣主题,写 `daily//interests.md` | `daily_topics_step` | + +关键点: + +- `dream` 和 `auto_dream` 不是同一条链路。 +- `dream` 是单文件命令,执行 `dream_step` 后立刻执行 `daily_topics_step`。 +- `auto_dream` 默认 `dispatch_job: dream_extract`,所以它对每个文件只跑 `dream_step`,把所有文件产生的 `topic_candidates` 收集起来,最后只调用一次 `daily_topics`。 +- `auto_dream` 默认写 3 个 topic,回看 7 天去重,输出 session id 是 `interests`。 + +配置片段的实际语义: + +```yaml +auto_dream: + steps: + - backend: auto_dream_step + dispatch_job: dream_extract + emit_topics: true + topic_dispatch_job: daily_topics + topic_count: 3 + topic_diversity_days: 7 + topic_session_id: interests +``` + +也就是说,`auto_dream` 本身是一个调度器和增量扫描器,真正的 LLM dream 逻辑在 `DreamStep`,topic 写入在 `DailyTopicsStep`。 + +## 2. auto_dream_step 执行链路 + +实现位置: + +- `reme4/steps/evolve/auto_dream.py` +- `reme4/steps/evolve/dream.py` +- `reme4/steps/evolve/daily_topics.py` +- `reme4/steps/file_io/_daily_index.py` + +### 2.1 读取输入和默认日期 + +`AutoDreamStep.execute()` 从 runtime context 读取: + +| 参数 | 语义 | +|---|---| +| `date` | 要扫描的日期,空字符串时用配置 timezone 下的今天 | +| `hint` | 透传给每个单文件 dream 的提示 | + +日期默认逻辑: + +```text +date_input 非空 -> 使用 date_input +date_input 为空 -> now(app_config.timezone).strftime("%Y-%m-%d") +``` + +`daily_dir` 不来自用户参数,而是来自 app config,默认是 `daily`。 + +### 2.2 先刷新 day-index + +正式扫描前,auto_dream 会先调用: + +```python +await refresh_day_index(self.file_store, today, daily_dir) +``` + +它会重建: + +```text +daily/.md +``` + +这个 day-index 文件包含 `daily//` 下每个 session note 的链接和 frontmatter 摘要。这样 auto_dream 后续处理的第一个文件就是当天总览。 + +隐含结果: + +- 如果 session notes 有新增/删除/frontmatter 变化,day-index 的内容可能变化。 +- day-index 被放在扫描列表第一位,所以当天总览先于具体 session note 被 dream。 + +### 2.3 扫描当天文件范围 + +扫描范围由 `_scan_today_files(vault, today, daily_dir)` 决定: + +```text +1. daily/.md +2. daily//**/*.md +``` + +处理顺序: + +```text +daily/.md first +daily//**/*.md sorted by path +``` + +但 auto_dream 会排除: + +```text +daily//interests.md +``` + +也就是 `topic_session_id` 对应的 daily topics 文件。原因是 `interests.md` 是 auto_dream 自己产出的兴趣主题,不能再作为 dream 输入,否则容易自我循环。 + +### 2.4 用 file_catalog 做增量判断 + +auto_dream 构造两张表: + +| 名称 | 来源 | 内容 | +|---|---|---| +| `existing` | 当前磁盘 | `{vault_relative_path: st_mtime}` | +| `indexed` | `file_catalog.get_nodes()` | `{vault_relative_path: st_mtime}` | + +`indexed` 只保留当天范围: + +```text +daily/.md +daily//* +``` + +并且同样排除: + +```text +daily//interests.md +``` + +然后做 diff: + +| 条件 | 分类 | 行为 | +|---|---|---| +| `rel in existing`,但 catalog 没有 | added | 需要 dream | +| `rel in existing`,但 mtime 不同 | modified | 需要 dream | +| `rel in existing`,且 mtime 相同 | unchanged | 跳过 | +| catalog 有,但磁盘没有 | deleted | 从 catalog 删除 | + +注意这里的 `file_catalog` 更像是 auto_dream 的“已处理 mtime 水位线”,不是语义索引本身。默认配置没有给 `auto_dream_step` 显式传 `file_catalog`,所以按 `BaseStep.Ref` 规则解析到 `file_catalog.default`。 + +### 2.5 先删除 catalog 中的缺失文件 + +如果某些当天文件已经不存在: + +```python +await self.file_catalog.delete(to_delete) +``` + +这一步不需要 LLM,也不会阻塞后续 dream。删除失败只记录日志,当前实现不会把它计入 `result.files_failed`。 + +### 2.6 对新增/修改文件逐个 dispatch dream_extract + +对每个 `to_dream` 文件,auto_dream 调: + +```python +resp = await self.run_job("dream_extract", path=rel_path, hint=hint) +``` + +`dream_extract` 只有一个 step: + +```yaml +steps: + - backend: dream_step +``` + +`AutoDreamStep._dispatch_dream()` 会把 `resp.metadata` 重新校验成 `DreamResult`。如果 job 抛异常、metadata 不是 `DreamResult`、或 response `success=False`,都会转成带 `error` 的 `DreamResult`。 + +### 2.7 单文件 DreamStep 内部逻辑 + +`DreamStep.dream_one(path, hint)` 是真正的 per-file create_or_update。 + +它的主流程: + +```text +1. path 为空 -> skipped +2. 没有 LLM -> error +3. _pack_material() 读取 vault-relative 文件内容 +4. Phase 1: _extract() +5. 如果 Phase 1 没有 units -> skipped,但保留 topic_candidates +6. Phase 2: 对每个 unit 调 _integrate_unit() +7. 返回 DreamResult +``` + +#### Phase 1: extract + +工具: + +```python +_EXTRACT_TOOLS = ("read",) +``` + +输出 schema 是 `ExtractedUnits`: + +```text +units: list[MemoryUnit] +topic_candidates: list[TopicCandidate] +``` + +每个 memory unit 包含: + +| 字段 | 语义 | +|---|---| +| `name` | agent 内部短名 | +| `bucket` | `procedure` / `personal` / `wiki` 三选一 | +| `summary` | 这个抽象是什么,证据在哪 | + +每个 topic candidate 包含: + +| 字段 | 语义 | +|---|---| +| `title` | 兴趣主题标题 | +| `reason` | 为什么用户可能关心 | +| `evidence` | 证据指针 | +| `keywords` | 去重关键词 | + +如果 LLM 输出了未知 bucket,当前代码会警告并改成 `wiki`。 + +#### Phase 2: integrate + +每个 unit 单独发起一次 ReAct: + +```text +system prompt = integrate_system_prompt_ +``` + +工具: + +```python +_INTEGRATE_TOOLS = ( + "node_search", + "read", + "frontmatter_read", + "write", + "edit", + "frontmatter_update", +) +``` + +输出 schema 是 `IntegrateOutcome`: + +| 字段 | 语义 | +|---|---| +| `action` | `CREATE` / `CORROBORATE` / `REFINE` / `CORRECT` | +| `target_path` | 实际写入或更新的 digest path | +| `note` | 简短说明 | + +`DreamStep` 根据 action 统计: + +```text +CREATE -> nodes_created +其他 action -> nodes_updated +``` + +当前实现里,某个 unit 的 integrate 失败不会让整个 DreamResult 变成 error,只会在 summary 里记录 `FAILED`。这意味着文件级别仍会被 auto_dream 当作成功并写入 catalog mtime。 + +### 2.8 汇总 per-file 结果并更新 catalog + +auto_dream 对每个文件的 `DreamResult` 做三件事: + +| 情况 | 行为 | +|---|---| +| `dr.error` 非空 | `files_failed += 1`,不更新这个文件的 catalog mtime,下次会重试 | +| `dr.skipped` 为 true | `files_skipped += 1`,仍然 upsert mtime,避免每次重复跑 Phase 1 | +| 正常 dream | `files_dreamed += 1`,upsert mtime | + +同时收集: + +```python +topic_candidates.extend(dr.topic_candidates or []) +``` + +最后批量: + +```python +await self.file_catalog.upsert(upsert_nodes) +``` + +### 2.9 聚合写 daily topics + +如果: + +```text +emit_topics == true +topic_candidates 非空 +``` + +auto_dream 会调用: + +```python +await self.run_job( + "daily_topics", + date=today, + candidates=topic_candidates, + topic_count=3, + diversity_days=7, + session_id="interests", +) +``` + +`DailyTopicsStep` 做: + +```text +1. 清洗 candidates +2. 读取过去 diversity_days 天的 interests.md +3. 有 LLM 时用 select prompt 选最终 topics +4. 没有 LLM 时 fallback: 简单标题去重 +5. 写 daily//interests.md +6. refresh_day_index() +``` + +写出的文件形态: + +```text +daily//interests.md +``` + +frontmatter 包含: + +```yaml +name: interests +description: " interest topic(s) inferred for ." +date: +topic_count: 3 +diversity_days: 7 +``` + +body 是 `# Interested Topics` 加编号列表。 + +auto_dream 收到 daily_topics 成功响应后,还会把这些文件的最新 mtime 写入 catalog: + +```text +daily//interests.md +daily/.md +``` + +这里有一个隐含行为:day-index 在 per-file dream 之后又因为 `interests.md` 被写入而刷新,auto_dream 会把刷新后的 `daily/.md` mtime 标记为已处理。也就是说,仅由 interests 写入引发的 day-index 变化不会在下一轮再次触发 dream。 + +### 2.10 持久化与响应 + +如果: + +```text +persist == true +并且有 upsert 或 delete +``` + +则: + +```python +await self.file_catalog.dump() +``` + +最终 response: + +```text +success = files_failed == 0 and not topics_error +answer = AutoDreamResult.summary +metadata = AutoDreamResult.model_dump() +``` + +summary 格式大致是: + +```text +[AutoDreamStep] date=2026-06-18 scanned=... unchanged=... dreamed=... skipped=... failed=... deleted=... + - daily/2026-06-18.md: OK (+1 created, ~2 updated) + - daily/2026-06-18/session.md: SKIP + - topics: OK (3 written to daily/2026-06-18/interests.md) +``` + +## 3. 当前逻辑的边界和风险 + +### 3.1 AutoDreamStep 职责过重 + +`AutoDreamStep` 同时负责: + +- 日期解析 +- day-index 刷新 +- 文件扫描 +- catalog diff +- 删除 catalog +- per-file job dispatch +- DreamResult 校验 +- topic candidates 汇总 +- daily_topics job dispatch +- topic 输出后的 catalog upsert +- catalog dump +- summary 渲染 + +这些职责可以拆成明确 step,提高可测试性和可替换性。 + +### 3.2 file_catalog 的语义不够显式 + +这里的 catalog 不是“今天有哪些文件”的普通目录索引,而是“哪些文件已经被 auto_dream 处理到某个 mtime”。建议在拆分后把它显式命名为 dream catalog / processed catalog,至少在 step 名和文档中说清楚。 + +### 3.3 integrate unit 失败不会触发文件重试 + +`DreamStep` 当前捕获单个 unit integrate 异常,写进 summary 后继续,但不设置 `DreamResult.error`。auto_dream 因此会把这个文件 mtime upsert,下次不会自动重试失败 unit。 + +这可能是有意的“尽量前进”,但如果要做严格一致性,应改成: + +```text +任一 unit integrate 失败 -> DreamResult.error 非空 -> auto_dream 不更新 mtime +``` + +### 3.4 topic 写入导致的 day-index 变化被标记为已处理 + +auto_dream 写 `interests.md` 后刷新 day-index,并把 day-index 最新 mtime upsert 到 catalog。这样可以避免自生成内容触发循环,但也意味着 `daily/.md` 中新增的 `interests.md` 链接不会被 dream。 + +这通常是合理的,因为 `interests.md` 本身被排除在 dream 输入之外。 + +### 3.5 删除 catalog 失败不影响 success + +删除 catalog entry 失败只打日志,不会让 response failure。拆分后可以明确这个策略: + +- catalog delete 是 best-effort,不影响 dream 主流程 +- 或者 catalog delete 失败应导致整个 job failure + +## 4. 拆分目标 + +重新拆分时,不按“每个小动作一个 step”拆,而按执行边界拆: + +```text +非 LLM 准备/扫描/diff + -> LLM: per-file dream + -> LLM: daily topics + -> 非 LLM response 汇总 +``` + +核心原则: + +- 使用 LLM 的阶段单独成 step,便于限流、重试、观测和替换模型。 +- 不使用 LLM 的准备、扫描、diff 可以合并,避免 step 过碎。 +- catalog 只是 auto_dream 的内部进度水位线,不提升为独立阶段。 +- prompt 重新写,但可以复用旧 prompt 的核心内容和约束。 +- 不做旧接口/旧格式兼容;按新 4-step pipeline 重新定义最干净的输入输出。 + +## 5. 新方案:拆成 4 个 Step + +新方案不再是“逐文件 extract + 逐文件 integrate”。核心变化是: + +```text +本轮 changed files + -> 1 个 agent 一次性阅读所有 changed paths + -> 输出全局去重/合并后的 unit list + -> Python for 循环逐 unit integrate +``` + +这样一个抽象可能来自多个文件,Phase 1 就能合并为同一个 unit,避免同一天多个 session note 反复提出同一概念。 + +目标代码位置: + +```text +reme4/steps/evolve/dream/ + __init__.py + models.py + plan.py + extract.py + integrate.py + topics.py + finish.py + prompts.yaml 或 dream.yaml +``` + +重构完成后删除旧文件,不保留兼容 alias: + +```text +reme4/steps/evolve/dream.py +reme4/steps/evolve/auto_dream.py +reme4/steps/evolve/daily_topics.py +``` + +### 5.0 Step 输入输出总表 + +| Step | 是否 LLM | 核心能力 | 输入 | 输出 / 写入 context | 副作用 | +|---|---:|---|---|---|---| +| `dream_extract_step` | 是 | 根据 dream catalog 找出本轮新增/修改/删除文件,把所有 changed paths 交给一个 agent,一次性输出跨文件合并后的 `unit_list` 和 `topic_list` | `context.date`; `context.hint`; `app_config.daily_dir`; `app_config.timezone`; `file_store.vault_path`; `file_catalog.dream`; step 参数 `topic_session_id=interests` | `dream.date`; `dream.hint`; `dream.daily_dir`; `dream.vault`; `dream.existing`; `dream.indexed`; `dream.changed_paths`; `dream.deleted_paths`; `dream.units`; `dream.topics`; `dream.extract_summary`; `dream.result.files_scanned/files_changed/files_deleted`; `dream.errors` | 刷新 `daily/.md`; 删除 catalog 中缺失文件 entry; 读取所有 changed files; 本 step 不写 digest | +| `dream_integrate_step` | 是 | `for unit in units` 逐个执行原 Phase 2 integrate 逻辑,保持 node_search/read/write/edit/frontmatter_update 工具和 bucket prompt 不变 | `dream.units`; `dream.hint`; `dream.vault`; `app_config.digest_dir`; agent tools: `node_search/read/frontmatter_read/write/edit/frontmatter_update` | `dream.integrate_results`; `dream.nodes_created`; `dream.nodes_updated`; `dream.result.units_integrated/units_failed`; `dream.errors` | 写/更新 `digest//*.md`; 不更新 dream catalog | +| `dream_topics_step` | 是 | 根据 `topic_list` 更新 `daily//interests.yaml`; 读取当天已有 topics 和最近 N 天 topics 做去重 | `dream.date`; `dream.daily_dir`; `dream.topics`; `file_store.vault_path`; step 参数 `topic_count`; `topic_diversity_days`; `topic_session_id=interests` | `dream.topics_path=daily//interests.yaml`; `dream.topics_written`; `dream.topics_merged`; `dream.topics_skipped_duplicates`; `dream.errors` | 新建或更新 `daily//interests.yaml`; 刷新 `daily/.md` | +| `dream_finish_step` | 否 | 统一收口:按 path checkpoint 成功处理的文件,持久化 catalog,渲染 summary 和 response metadata | `dream.changed_paths`; `dream.deleted_paths`; `dream.failed_paths`; `dream.integrate_results`; `dream.topics_path`; `dream.errors`; `dream.result`; `file_catalog.dream`; step 参数 `persist=true` | `context.response.success`; `context.response.answer`; `context.response.metadata` | upsert successful paths 的 mtime 到 `file_catalog.dream`; upsert `interests.yaml` 和 day-index mtime; `file_catalog.dump()` | + +### Step 1: `dream_extract_step` + +这是新的全局 Phase 1。它合并了当前 `auto_dream_step` 的扫描/diff 和当前 `DreamStep._extract()` 的抽取能力。 + +职责: + +- 解析 `date` / `hint`。 +- 刷新 day-index: `daily/.md`。 +- 扫描输入文件并统一交给 extract agent: + - `daily/.md` + - `daily//.md` + - `daily//.md` + - 以及 `daily//**/*.md` 下其它当天 note +- 排除自生成文件: + - `daily//interests.yaml` +- 读取 `file_catalog.dream`,按 mtime diff 出: + - `changed_paths` + - `unchanged_paths` + - `deleted_paths` +- 删除 catalog 中 `deleted_paths`。 +- 打包所有 `changed_paths` 的文件内容。 +- 调用一次 extract agent,让它看见所有 changed paths。 +- 输出跨文件合并后的 `unit_list` 和 `topic_list`。 + +新的 unit schema: + +```python +class DreamUnit(BaseModel): + name: str + bucket: Literal["procedure", "personal", "wiki"] + summary: str + paths: list[str] +``` + +`paths` 是这个 unit 的证据来源列表。多个文件讲的是同一抽象时,Phase 1 必须合并成一个 unit: + +```json +{ + "name": "jwt-session-expiry-policy", + "bucket": "procedure", + "summary": "How the project decides session expiry from compliance and product constraints.", + "paths": [ + "daily/2026-06-18/auth.md", + "daily/2026-06-18/api-review.md" + ] +} +``` + +topic schema 可以沿用当前 `TopicCandidate`,但建议把来源改成 `paths`: + +```python +class DreamTopicCandidate(BaseModel): + title: str + reason: str + evidence: str + keywords: list[str] = [] + paths: list[str] = [] +``` + +输出到 context: + +```python +{ + "dream": { + "date": "YYYY-MM-DD", + "changed_paths": [ + {"path": "daily/YYYY-MM-DD/a.md", "mtime": 1710000000.0} + ], + "deleted_paths": [], + "units": [ + { + "name": "jwt-session-expiry-policy", + "bucket": "procedure", + "summary": "...", + "paths": ["daily/YYYY-MM-DD/a.md", "daily/YYYY-MM-DD/b.md"] + } + ], + "topics": [ + { + "title": "...", + "reason": "...", + "evidence": "...", + "keywords": ["..."], + "paths": ["daily/YYYY-MM-DD/a.md"] + } + ] + } +} +``` + +LLM 调用数量: + +```text +1 个 agent 任务 +``` + +注意: + +- 这个 step 不再为每个文件分别调用 `dream_extract`。 +- 如果 `changed_paths` 为空,它不调用 LLM,直接输出空 `units/topics`。 +- 只有 `dream_finish_step` 才把 changed file mtime 标为已处理。这样 integrate/topics 失败时不会误跳过。 + +### Step 2: `dream_integrate_step` + +这是新的全局 Phase 2。它对 Step 1 输出的 `units` 做 Python for 循环,每个 unit 的 integrate 逻辑保持当前 `DreamStep._integrate_unit()` 不变。 + +职责: + +- 遍历 `dream.units`。 +- 每个 unit 根据 `unit.bucket` 选择: + - `integrate_system_prompt_procedure` + - `integrate_system_prompt_personal` + - `integrate_system_prompt_wiki` +- material 不再是单文件 blob,而是这个 unit 对应 `paths` 的证据包。 +- 调用当前相同工具: + - `node_search` + - `read` + - `frontmatter_read` + - `write` + - `edit` + - `frontmatter_update` +- 输出 `IntegrateOutcome`。 + +`integrate_user_message` 需要从单 `material_blob` 改成多路径 evidence blob: + +```text +unit_name: ... +unit_bucket: ... +unit_summary: ... +source_paths: + - daily/... + - daily/... + +# Evidence materials +### daily/.../a.md +... + +### daily/.../b.md +... +``` + +输出: + +```python +{ + "dream": { + "integrate_results": [ + { + "unit_name": "jwt-session-expiry-policy", + "bucket": "procedure", + "action": "CREATE", + "target_path": "digest/procedure/jwt-session-expiry-policy.md", + "source_paths": ["daily/YYYY-MM-DD/a.md", "daily/YYYY-MM-DD/b.md"], + "note": "..." + } + ], + "nodes_created": ["digest/procedure/jwt-session-expiry-policy.md"], + "nodes_updated": [] + } +} +``` + +LLM 调用数量: + +```text +N 个 agent 任务 +N = len(dream.units) +``` + +失败策略建议: + +- 任一 unit integrate 失败,记录到 `dream.errors`。 +- 因为每个 unit 都有明确的 `paths`,失败 unit 对应的 paths 进入 `dream.failed_paths`。 +- `dream_finish_step` 不 checkpoint `failed_paths`。 +- 不在任何失败 unit `paths` 里的 changed paths 可以 checkpoint。 +- 如果同一个 path 同时出现在成功 unit 和失败 unit 中,以失败为准,该 path 不 checkpoint。 + +### Step 3: `dream_topics_step` + +这个 step 取代当前 `daily_topics_step`。目标文件固定为: + +```text +daily//interests.yaml +``` + +职责: + +- 读取 `dream.topics`。 +- 如果 `daily//interests.yaml` 已存在,读取旧 topics。 +- 读取最近 `topic_diversity_days` 天的 `daily//interests.yaml` 作为历史去重上下文。 +- 合并当天旧 topics + 新 topics。 +- 去重: + - 标题 normalize 后相同视为重复。 + - keywords 高重叠视为可能重复。 + - evidence/paths 完全相同视为重复。 + - 与最近 N 天历史 topics 重复时跳过。 +- 可选使用 LLM 对候选 topic 做最终选择和改写。 +- 写回 YAML。 +- 刷新 day-index。 + +建议 YAML 格式: + +```yaml +date: "2026-06-18" +updated_at: "2026-06-18T22:00:00+08:00" +topic_count: 3 +diversity_days: 7 +topics: + - title: "JWT session expiry policy" + reason: "The user repeatedly worked through compliance-driven auth expiry tradeoffs." + evidence: "Mentioned in auth review and API notes." + keywords: ["auth", "jwt", "session", "compliance"] + paths: + - "daily/2026-06-18/auth.md" + - "daily/2026-06-18/api-review.md" +``` + +输入: + +```text +dream.date +dream.daily_dir +dream.topics +topic_count +topic_diversity_days +topic_session_id +``` + +输出: + +```python +{ + "dream": { + "topics_path": "daily/YYYY-MM-DD/interests.yaml", + "topics_written": 3, + "topics_merged": 5, + "topics_skipped_duplicates": 2 + } +} +``` + +LLM 调用数量: + +```text +0 或 1 个 agent 任务 +``` + +建议: + +- 如果只是 append/去重,不必 LLM。 +- 如果需要从很多 candidates 中挑 `topic_count` 个,才调用 LLM。 +- 不读取也不写 `interests.md`;全新格式只认 `interests.yaml`。 + +### Step 4: `dream_finish_step` + +这是非 LLM 收尾 step。 + +职责: + +- 根据前面步骤结果决定 success。 +- 计算 `failed_paths`: + - 每个失败 unit 的 `unit.paths` 都进入 failed set。 + - 如果某个 path 同时属于成功 unit 和失败 unit,以失败为准。 +- 计算 `checkpoint_paths`: + - `changed_paths - failed_paths` + - extract 成功但没有任何 unit/topics 的 changed paths 也可以 checkpoint,避免重复空跑。 +- 把 `checkpoint_paths` 的当前 mtime upsert 到 `file_catalog.dream`。 +- 把 `daily//interests.yaml` 的 mtime upsert 到 `file_catalog.dream`。 +- 把刷新后的 `daily/.md` 的 mtime upsert 到 `file_catalog.dream`。 +- `deleted_paths` 的 catalog 删除在 extract step 已完成,finish 只负责 dump。 +- `file_catalog.dump()`。 +- 渲染 summary。 +- 写 `context.response.metadata`。 + +输出 metadata 建议: + +```python +{ + "date": "YYYY-MM-DD", + "files_scanned": 10, + "files_changed": 3, + "files_deleted": 1, + "paths_checkpointed": ["daily/2026-06-18/a.md"], + "paths_failed": ["daily/2026-06-18/b.md"], + "units_extracted": 4, + "units_integrated": 4, + "units_failed": 0, + "topics_written": 3, + "nodes_created": [...], + "nodes_updated": [...], + "errors": [] +} +``` + +## 6. 拆分后的 YAML 形态 + +建议把 `auto_dream` 改成新的 dream pipeline: + +```yaml +auto_dream: + backend: base + description: "Auto-dream: scan daily changes, extract cross-file units, integrate digest nodes, update daily interests." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone" + default: "" + hint: + type: string + description: "caller guidance passed through to the dreamer LLM" + default: "" + steps: + - backend: dream_extract_step + file_catalog: dream + topic_session_id: interests + - backend: dream_integrate_step + - backend: dream_topics_step + topic_count: 3 + topic_diversity_days: 7 + topic_session_id: interests + - backend: dream_finish_step + file_catalog: dream + persist: true +``` + +旧 job 删除,不做兼容 wrapper: + +```yaml +dream: + # 删除 + +dream_extract: + # 删除 + +daily_topics: + # 删除 +``` + +## 7. 数据结构建议 + +建议所有跨 step 状态都放在 `context["dream"]`。 + +核心模型: + +```python +class DreamUnit(BaseModel): + name: str + bucket: Literal["procedure", "personal", "wiki"] + summary: str + paths: list[str] = Field(default_factory=list) + + +class DreamTopic(BaseModel): + title: str + reason: str + evidence: str = "" + keywords: list[str] = Field(default_factory=list) + paths: list[str] = Field(default_factory=list) + + +class DreamState(BaseModel): + date: str = "" + hint: str = "" + daily_dir: str = "daily" + vault: str = "" + changed_paths: list[dict] = Field(default_factory=list) + unchanged_paths: list[str] = Field(default_factory=list) + deleted_paths: list[str] = Field(default_factory=list) + failed_paths: list[str] = Field(default_factory=list) + checkpoint_paths: list[str] = Field(default_factory=list) + units: list[DreamUnit] = Field(default_factory=list) + topics: list[DreamTopic] = Field(default_factory=list) + integrate_results: list[dict] = Field(default_factory=list) + nodes_created: list[str] = Field(default_factory=list) + nodes_updated: list[str] = Field(default_factory=list) + topics_path: str = "" + topics_written: int = 0 + errors: list[str] = Field(default_factory=list) + result: dict = Field(default_factory=dict) +``` + +## 8. 实现任务 + +这是一次 breaking rewrite,不存在旧接口/旧格式迁移任务。剩余工作就是按新设计实现 4 个 step。 + +必须实现: + +1. 在 `reme4/steps/evolve/dream/` 下新增 `models.py`、helper 和 4 个 step 文件。 +2. 重写 prompts: + - `extract_system_prompt` + - `extract_user_message` + - `integrate_system_prompt_procedure` + - `integrate_system_prompt_personal` + - `integrate_system_prompt_wiki` + - `integrate_user_message` + - 可参考旧 prompt 的内容,但不保持旧 prompt 接口。 +3. 实现 `dream_extract_step`: + - 扫描 `daily/.md` 与 `daily//**/*.md`。 + - 排除 `daily//interests.yaml`。 + - 根据 `file_catalog.dream` 计算 changed/unchanged/deleted。 + - 一个 agent 统一读取所有 changed paths,输出全局 units/topics。 + - 清洗 units: unknown bucket fallback 到 `wiki`; paths 去重; paths 必须来自 changed paths。 +4. 实现 `dream_integrate_step`: + - 按 `unit.paths` 打包 evidence。 + - for 循环逐 unit integrate。 + - 保留工具集合和 action 语义。 + - unit 失败时记录 `failed_paths += unit.paths`。 +5. 实现 `dream_topics_step`: + - 只读写 `daily//interests.yaml`。 + - 读取当天已有 YAML topics。 + - 读取最近 `topic_diversity_days` 天的 `interests.yaml` 做历史去重。 + - 写回去重后的 YAML。 + - 刷新 day-index。 +6. 实现 `dream_finish_step`: + - `checkpoint_paths = changed_paths - failed_paths`。 + - checkpoint 成功 paths 的 mtime。 + - checkpoint `interests.yaml` 和 `daily/.md`。 + - dump `file_catalog.dream`。 + - 输出全新 response metadata。 +7. 更新 `default.yaml`: + - `auto_dream` 改为 4-step pipeline。 + - 删除 `dream`、`dream_extract`、`daily_topics` job。 + - 保留 `file_catalog.dream`。 +8. 删除旧代码: + - `reme4/steps/evolve/auto_dream.py` + - `reme4/steps/evolve/dream.py` + - `reme4/steps/evolve/daily_topics.py` + - 更新 `reme4/steps/evolve/__init__.py`。 + +现在没有保留的迁移项: + +- 不兼容 `interests.md`。 +- 不保留单文件 `dream path=...`。 +- 不保留 `dream_extract` job。 +- 不保留 `daily_topics` job。 +- 不要求 response metadata 兼容 `AutoDreamResult`。 +- 不要求 prompt 入参兼容旧 `dream.yaml`。 + +## 9. 推荐测试用例 + +最低测试集: + +| 场景 | 期望 | +|---|---| +| 当天没有任何文件 | scanned=0,success=true,no topics | +| 只有 day-index 新增 | `dream_extract_step` 调用一次全局 extract,finish checkpoint day-index | +| session note 新增 | extract evidence 中 day-index first,session notes sorted | +| session note mtime 未变 | unchanged+1,不进入 changed evidence | +| session note 删除 | catalog delete | +| `interests.yaml` 存在 | 不进入 scan/diff/changed evidence | +| 多个文件产出同一抽象 | extract 输出 1 个 unit,`paths` 包含多个 source path | +| extract 输出空 units/topics | finish 仍 checkpoint changed files,避免重复空跑 | +| 某个 unit integrate 失败 | 该 unit 的 `paths` 不 checkpoint,response failure | +| 同一 path 同时属于成功和失败 unit | 失败优先,该 path 不 checkpoint | +| 其它 path 的 units 都成功 | 这些 path 可以 checkpoint | +| 有 topic candidates | 写/更新 `daily//interests.yaml`,记录 topics_path/topics_written | +| 已存在 `interests.yaml` | 合并新旧 topics,不重复 | +| 最近 N 天已有相同 topic | 当前日 topics 去重跳过 | +| extract 输出 unknown bucket | 清洗后 bucket=`wiki` | +| prompt 输出 path 不在 changed paths | 该 unit 被丢弃或修正,不能 checkpoint 不明来源 | diff --git a/docs4/reme_design.md b/docs4/reme_design.md index ad469952..b1fb1c85 100644 --- a/docs4/reme_design.md +++ b/docs4/reme_design.md @@ -11,20 +11,24 @@ ### 1.1 目录结构 ``` -- resource/【原始素材】 # 外部渠道摄入 / 手动放入 - - YYYY-MM-DD/ # 按日期归档 - - session_{id}.jsonl # 对话原始记录 - - {channel}_{xxxx}.html # 网页抓取、邮件等 - - {channel}_{xxxx}.md # Markdown 资料 -- daily/【日记，浅加工】 # auto-memory 自动写入 - - YYYY-MM-DD.md # 当天索引页，汇总所有事件 +- reme_session/ + - agentscope|claude_code / # 使用内置的agent wrapper，session会保存在这里 + {session_id}.jsonl UUID格式要求 # /Users/yuli/workspace/ReMe/reme4/components/agent_wrapper + - dialog/ + {session_id}.jsonl # auto memory保存 可以监控可以被检索【可选】 +- resource/ - YYYY-MM-DD/ - - session_{id}.md # 按 session 拆分的日志 - - resource_{id}.md # 对素材的加工笔记 -- digest/【深加工】 # auto-dream 持续打磨 - - personal/ # 用户偏好、习惯、身份 - - procedure/ # 方法论、步骤、工作流 - - wiki/ # 通用知识、决策先例 + - {channel}_{xxxx}.html + - {channel}_{xxxx}.md +- daily/【日记，浅加工】 + - YYYY-MM-DD.md + - YYYY-MM-DD/ + - session_{session_id}.md + - {resource_stem}.md +- digest/ + - personal/ + - procedure/ + - wiki/ ``` ### 1.2 分层详解 diff --git a/docs4/todo.md b/docs4/todo.md new file mode 100644 index 00000000..d4c79b97 --- /dev/null +++ b/docs4/todo.md @@ -0,0 +1,38 @@ +- reme_session/ + - agentscope|claude_code / # 使用内置的agent wrapper，session会保存在这里 + {session_id}.jsonl UUID格式要求 # /Users/yuli/workspace/ReMe/reme4/components/agent_wrapper + - dialog/ + {session_id}.jsonl # auto memory保存 可以监控可以被检索【可选】 +- resource/ + - YYYY-MM-DD/ + - {channel}_{xxxx}.html + - {channel}_{xxxx}.md +- daily/【日记，浅加工】 + - YYYY-MM-DD.md + - YYYY-MM-DD/ + - {session_id}.md + - {和resource同名}.md +- digest/ + - personal/ + - procedure/ + - wiki/ + +函数接口： +- auto_memory + - message 应该会 会保存到 reme_session/dialog/{session_id}.jsonl + - 通过 message 更新 daily/YYYY-MM-DD/{session_id}.md +- auto-resource + - 会保存到 daily/YYYY-MM-DD/{resource_stem}.md +- auto-dream + - 读取所有的md + - 会生成link auto-link + - 会生成topic ？ +- proactive + - 会读取topic ？ +- search + + +后台任务： +- index_update_loop 索引监控 +- resource_watch_loop 资源监控 +- digest_watch_loop 应该是闲置？ diff --git a/docs4/watch_loop_step_refactor_plan.md b/docs4/watch_loop_step_refactor_plan.md new file mode 100644 index 00000000..12ff1883 --- /dev/null +++ b/docs4/watch_loop_step_refactor_plan.md @@ -0,0 +1,275 @@ +# Watch Loop Step 重构计划 + +## 背景 + +当前 `index_update_loop`、`resource_watch_loop`、`digest_watch_loop` 都是同一种范式： + +1. 启动时扫描已有文件变化。 +2. 用一组 step 处理扫描出来的变化。 +3. 进入持续监听。 +4. 持续监听到变化后，再用同一组 step 处理变化。 + +也就是说，初始化扫描和持续监听只是变化来源不同，后续更新逻辑应该共享。 + +现在的问题是这个范式没有被显式建模： + +- `index_update_loop` 初始化和监听都走 `update_index_step`，基本一致。 +- `resource_watch_loop` 初始化和监听都走 `update_catalog_step + foreach_dispatch_step`，基本一致。 +- `digest_watch_loop` 初始化走 `update_catalog_step`，但监听阶段只走 `log_changes_step`，导致 live changes 不更新 `file_catalog`。 +- `watch_changes_step` 同时负责监听和 dispatch 下游逻辑，职责偏重。 +- `update_index_step` 和 `update_catalog_step` 内部有较多重复的变化分桶、结果收集、删除、持久化逻辑。 + +## 目标 + +重构后希望形成统一模型： + +```text +change producer: + init_changes_step # 初始化，一次性产生 changes 并 dispatch + watch_changes_step # 持续监听，持续产生 changes + +change handlers: + update_index_step + update_catalog_step + foreach_dispatch_step + log_changes_step + channel_notify_step # 后续可选 +``` + +核心原则： + +- producer 只负责产生 `context["changes"]`。 +- handler 只负责消费 `context["changes"]`。 +- 初始化和持续监听都通过 `BaseStep.dispatch_steps(...)` 调用 handler。 +- `init_changes_step` 和 `watch_changes_step` 显式配置同一组 `dispatch_steps`，让范式直接可见。 + +## 配置设计 + +不新增 job 级 `change_steps`。每个 producer step 自己声明 `dispatch_steps`，初始化 producer 和持续监听 producer 配同一组 handler。 + +配置精简约定： + +- `init_changes_step.recursive` 和 `watch_changes_step.recursive` 默认就是 `true`，配置中不再显式写。 +- `update_index_step` / `update_catalog_step` 的 `persist` 语义统一为默认 `true`，配置中不再显式写。 +- 只有当某个 loop 需要关闭递归或关闭持久化时，才显式写 `recursive: false` / `persist: false`。 + +### index_update_loop + +```yaml +index_update_loop: + backend: background + watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_suffixes: [md, jsonl] + steps: + - backend: init_changes_step + store: file_store + dispatch_steps: [update_index_step] + - backend: watch_changes_step + dispatch_steps: [update_index_step] +``` + +### resource_watch_loop + +```yaml +resource_watch_loop: + backend: background + watch_dirs: [resource_dir] + watch_suffixes: [md, txt, json, jsonl, csv, yaml, html] + dispatch_job: auto_resource + steps: + - backend: init_changes_step + store: file_catalog + dispatch_steps: [update_catalog_step, foreach_dispatch_step] + - backend: watch_changes_step + dispatch_steps: [update_catalog_step, foreach_dispatch_step] +``` + +### digest_watch_loop + +```yaml +digest_watch_loop: + backend: background + watch_dirs: [daily_dir, digest_dir] + watch_suffixes: [md] + steps: + - backend: init_changes_step + store: file_catalog + dispatch_steps: [update_catalog_step, log_changes_step] + - backend: watch_changes_step + dispatch_steps: [update_catalog_step, log_changes_step] +``` + +这样三个 loop 都统一成： + +```text +startup: + scan changes + dispatch handlers + +runtime: + watch changes + dispatch same handlers +``` + +## Step 拆分 + +### 1. BaseStep dispatch 能力 + +把 dispatch 能力沉到 `BaseStep`，所有 producer step 共享： + +- `normalize_dispatch_steps(dispatch_step, dispatch_steps)` +- `dispatch_steps(dispatch_steps, **kwargs)` + +这样 `init_changes_step` 和 `watch_changes_step` 都不需要各自实现 registry 查询、step 实例化和 context 透传。 + +`dispatch_steps` 支持两种形式： + +```yaml +dispatch_steps: [update_catalog_step, log_changes_step] +``` + +也支持给单个 handler 传参数： + +```yaml +dispatch_steps: + - backend: update_catalog_step + - backend: some_step + option: value +``` + +### 2. init_changes_step + +新增 `init_changes_step`，替代现有两个初始化扫描 step： + +- `scan_store_changes_step` +- `scan_catalog_changes_step` + +参数： + +```yaml +store: file_catalog | file_store +dispatch_steps: [...] +``` + +职责： + +- 根据 `watch_dirs` / `watch_suffixes` 收集磁盘文件。 +- 根据 `store` 和目标状态源比较。 +- 生成统一格式的 `context["changes"]`。 +- 如果有变化，调用 `BaseStep.dispatch_steps(...)` 执行 handler。 + +输出格式： + +```python +[ + {"change": "added", "path": "/abs/path/to/file.md"}, + {"change": "modified", "path": "/abs/path/to/file.md"}, + {"change": "deleted", "path": "/abs/path/to/file.md"}, +] +``` + +### 3. watch_changes_step + +保留监听职责，弱化业务 dispatch 职责。 + +职责： + +- 根据 `watch_dirs` / `watch_suffixes` 建立文件监听。 +- 对每个 debounced batch 生成同样格式的 `changes`。 +- 调用 `BaseStep.dispatch_steps(...)` 执行 handler。 + +### 4. update_index_step / update_catalog_step + +第二阶段再精简。 + +它们现在重复逻辑包括： + +- 解析 `added` / `modified` / `deleted`。 +- 判断文件是否存在。 +- 收集 per-path result。 +- 删除旧记录。 +- upsert 新记录。 +- persist。 +- 写 response。 + +建议抽内部基类，例如： + +```python +class ChangeApplyStep(BaseStep): + async def parse_added_or_modified(self, path): ... + async def upsert_items(self, items): ... + async def delete_paths(self, rel_paths): ... + async def dump_target(self): ... +``` + +然后： + +- `UpdateCatalogStep` 只实现 `stat -> FileNode`，写 `file_catalog`。 +- `UpdateIndexStep` 只实现 `chunk_file -> FileNode + chunks`，写 `file_store`。 + +这一步可以在 `init_changes_step` 落地后做，降低一次性改动风险。 + +## 实施顺序 + +当前落地状态： + +- Phase 1 已完成：`BaseStep.dispatch_steps(...)`、`init_changes_step`、默认持久化、默认配置精简已落地。 +- Phase 2 已完成：`digest_watch_loop` 的初始化和监听都执行 `update_catalog_step + log_changes_step`。 +- 兼容旧 backend 不保留：`scan_store_changes_step` / `scan_catalog_changes_step` 已删除。 +- `reindex` 已改为 `clear_store_step + init_changes_step(store=file_store, dispatch_steps=[update_index_step])`。 +- Phase 3 已完成一层：`update_catalog_step` 和 `update_index_step` 已合并到 `update_changes.py`， + 并抽出 `ChangeApplyStep` 复用 added/modified/deleted、upsert/delete/persist 模板逻辑。 + +### Phase 1：统一范式 + +1. 把 dispatch 能力沉到 `BaseStep`。 +2. 新增 `init_changes_step`。 +3. 将 `update_index_step` 和 `update_catalog_step` 的 `persist` 默认值统一为 `true`。 +4. 修改 `default.yaml` 里的三个 loop： + - 初始化阶段统一使用 `init_changes_step`。 + - `init_changes_step` 和 `watch_changes_step` 配置相同的 `dispatch_steps`。 +5. `watch_changes_step` 保留 `dispatch_step` 到 `dispatch_steps` 的轻量兼容。 +6. 验证三个 loop 的启动扫描和 live watch 都会执行同一条 handler pipeline。 + +### Phase 2：修正 digest_watch_loop 语义 + +`digest_watch_loop` 的 live changes 应该更新 `file_catalog`，因此初始化和监听阶段都应配置同一组 `dispatch_steps`： + +```yaml +dispatch_steps: [update_catalog_step, log_changes_step] +``` + +如果后续要通知 channel，可以追加： + +```yaml + - backend: channel_notify_step +``` + +### Phase 3：精简 update 类 step + +1. 抽 `ChangeApplyStep` 基类或 helper 函数。 +2. 让 `update_index_step` 和 `update_catalog_step` 只保留各自差异逻辑。 +3. 保持外部行为不变： + - 输入仍然是 `context["changes"]`。 + - 输出仍然写 `response.answer` 和 `response.success`。 + - `persist` 语义不变。 + +## 验证点 + +最少需要覆盖这些场景： + +- `index_update_loop` 启动扫描新增文件，会更新 `file_store`。 +- `index_update_loop` live 新增/修改/删除文件，会更新 `file_store`。 +- `resource_watch_loop` 启动扫描新增文件，会更新 `file_catalog` 并触发 `auto_resource`。 +- `resource_watch_loop` live 新增文件，会更新 `file_catalog` 并触发 `auto_resource`。 +- `digest_watch_loop` 启动扫描新增/修改/删除文件，会更新 `file_catalog`。 +- `digest_watch_loop` live 新增/修改/删除文件，也会更新 `file_catalog`。 +- `changes` 为空时，`init_changes_step` 不应执行 handler，也不应报错。 + +## 预期收益 + +- 三个 background loop 的结构统一。 +- 初始化扫描和持续监听的处理逻辑完全复用。 +- `digest_watch_loop` 不再出现启动和 live 语义不一致。 +- `watch_changes_step` 和 `init_changes_step` 共享 `BaseStep` dispatch 能力。 +- 后续新增日志、channel notification、auto dream 等 handler 时，初始化和监听两处使用同一组 `dispatch_steps`。 diff --git a/reme/core/application.py b/reme/core/application.py index 042b2e80..f776ed98 100644 --- a/reme/core/application.py +++ b/reme/core/application.py @@ -514,17 +514,17 @@ class Application: logger.warning("Application is not started") return True - for name, vector_store in self.service_context.vector_stores.items(): - logger.info(f"Closing vector store: {name}") - await vector_store.close() + for name, file_watcher in self.service_context.file_watchers.items(): + logger.info(f"Closing file watcher: {name}") + await file_watcher.close() for name, file_store in self.service_context.file_stores.items(): logger.info(f"Closing file store: {name}") await file_store.close() - for name, file_watcher in self.service_context.file_watchers.items(): - logger.info(f"Closing file watcher: {name}") - await file_watcher.close() + for name, vector_store in self.service_context.vector_stores.items(): + logger.info(f"Closing vector store: {name}") + await vector_store.close() for name, llm in self.service_context.llms.items(): logger.info(f"Closing LLM: {name}") diff --git a/reme/core/file_store/local_file_store.py b/reme/core/file_store/local_file_store.py index 38757d7a..09b73d8f 100644 --- a/reme/core/file_store/local_file_store.py +++ b/reme/core/file_store/local_file_store.py @@ -107,6 +107,20 @@ class LocalFileStore(BaseFileStore): except Exception as e: logger.error(f"Failed to save file metadata to {self._metadata_file}: {e}") + async def _persist(self) -> None: + """Persist all in-memory indexes after mutations.""" + await self._save_metadata() + await self._save_chunks() + + def _delete_file_in_memory(self, path: str, source: MemorySource) -> None: + """Delete file data from memory without flushing to disk.""" + to_delete = [cid for cid, chunk in self._chunks.items() if chunk.path == path and chunk.source == source] + for cid in to_delete: + del self._chunks[cid] + + if source.value in self._files: + self._files[source.value].pop(path, None) + # ------------------------------------------------------------------ # Lifecycle # ------------------------------------------------------------------ @@ -146,7 +160,7 @@ class LocalFileStore(BaseFileStore): return # Remove existing chunks for this file/source first - await self.delete_file(file_meta.path, source) + self._delete_file_in_memory(file_meta.path, source) # Batch generate embeddings (base class returns mock embeddings when vector_enabled=False) chunks = await self.get_chunk_embeddings(chunks) @@ -163,15 +177,12 @@ class LocalFileStore(BaseFileStore): path=file_meta.path, chunk_count=len(chunks), ) + await self._persist() async def delete_file(self, path: str, source: MemorySource) -> None: """Delete file and all its chunks.""" - to_delete = [cid for cid, chunk in self._chunks.items() if chunk.path == path and chunk.source == source] - for cid in to_delete: - del self._chunks[cid] - - if source.value in self._files: - self._files[source.value].pop(path, None) + self._delete_file_in_memory(path, source) + await self._persist() async def delete_file_chunks(self, path: str, chunk_ids: list[str]) -> None: """Delete specific chunks for a file.""" @@ -187,6 +198,7 @@ class LocalFileStore(BaseFileStore): source_meta[path].chunk_count = sum( 1 for chunk in self._chunks.values() if chunk.path == path and chunk.source.value == source_key ) + await self._persist() async def upsert_chunks( self, @@ -201,6 +213,7 @@ class LocalFileStore(BaseFileStore): for chunk in chunks: self._chunks[chunk.id] = chunk + await self._persist() # ------------------------------------------------------------------ # Read operations @@ -230,6 +243,7 @@ class LocalFileStore(BaseFileStore): path=file_meta.path, chunk_count=file_meta.chunk_count, ) + await self._persist() async def get_file_chunks( self, @@ -456,6 +470,5 @@ class LocalFileStore(BaseFileStore): """Clear all indexed data from memory and disk.""" self._chunks.clear() self._files.clear() - await self._save_chunks() - await self._save_metadata() + await self._persist() logger.info(f"Cleared all data from LocalFileStore '{self.store_name}'") diff --git a/reme4/application.py b/reme4/application.py index a26b56ba..ceaf579c 100644 --- a/reme4/application.py +++ b/reme4/application.py @@ -7,7 +7,7 @@ from pathlib import Path from typing import AsyncGenerator, TypeVar from .components import BaseComponent, ApplicationContext -from .components.job import BaseJob +from .components.job import BackgroundJob, BaseJob, CronJob, StreamJob from .components.service import BaseService from .enumeration import ComponentEnum from .schema import ComponentConfig, Response, StreamChunk @@ -48,7 +48,7 @@ class Application(BaseComponent): cfg = self.config vault_path = Path(cfg.vault_dir).absolute() vault_path.mkdir(parents=True, exist_ok=True) - for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]: + for subdir in [cfg.metadata_dir, cfg.session_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]: if subdir: (vault_path / subdir).mkdir(parents=True, exist_ok=True) @@ -168,28 +168,34 @@ class Application(BaseComponent): # ----- Lifecycle ----------------------------------------------------- async def _start(self) -> None: - """Start components in dependency order, then jobs (background last).""" + """Start components, then jobs as base > stream > background > cron.""" pool_size = self.config.thread_pool_max_workers if pool_size > 0: self.context.thread_pool = ThreadPoolExecutor(max_workers=pool_size) self.logger.info(f"Thread pool created with max_workers={pool_size}") - components = self._topological_order() - jobs = list(self.context.jobs.values()) - # Background jobs come last so they observe a fully wired system. - foreground = [j for j in jobs if j.backend != "background"] - background = [j for j in jobs if j.backend == "background"] - for c in components + foreground + background: - await self._start_one(c) + try: + components = self._topological_order() + jobs = list(self.context.jobs.values()) + base_jobs = [j for j in jobs if not isinstance(j, (StreamJob, BackgroundJob))] + stream_jobs = [j for j in jobs if isinstance(j, StreamJob)] + background_jobs = [j for j in jobs if isinstance(j, BackgroundJob) and not isinstance(j, CronJob)] + cron_jobs = [j for j in jobs if isinstance(j, CronJob)] + for c in components + base_jobs + stream_jobs + background_jobs + cron_jobs: + await self._start_one(c) + except Exception: + await self._close() + raise async def _start_one(self, c: BaseComponent) -> None: - """Start one component and record it for ordered shutdown; log and swallow failures.""" + """Start one component and record it for ordered shutdown.""" try: - if c.backend == "background": + if isinstance(c, BackgroundJob): self.logger.info(f"Starting background job: {c.name}") await c.start() self._started_components.append(c) except Exception as e: self.logger.exception(f"Failed to start {c.component_type.value}:{c.name}: {e}") + raise async def _close(self) -> None: """Close in reverse start order so every peer outlives its dependents.""" @@ -211,6 +217,20 @@ class Application(BaseComponent): raise KeyError(f"Job '{name}' not found") return await self.context.jobs[name](**kwargs) + async def update_component(self, component_enum: ComponentEnum | str, name: str, /, **kwargs) -> BaseComponent: + """Update an existing component by type/name; never creates missing components.""" + component_enum = ComponentEnum(component_enum) + group = self.context.components.get(component_enum) + if not group or name not in group: + raise KeyError(f"Component '{name}' not found in {component_enum.value}") + + component = group[name] + for key, value in kwargs.items(): + if not hasattr(component, key): + raise AttributeError(f"Component {component_enum.value}:{name} has no attribute '{key}'") + setattr(component, key, value) + return component + async def run_stream_job(self, name: str, /, **kwargs) -> AsyncGenerator[StreamChunk, None]: """Execute a streaming job, yielding chunks as they are produced.""" if name not in self.context.jobs: diff --git a/reme4/components/agent_wrapper/as_agent_wrapper.py b/reme4/components/agent_wrapper/as_agent_wrapper.py index fa85db04..e66c25cb 100644 --- a/reme4/components/agent_wrapper/as_agent_wrapper.py +++ b/reme4/components/agent_wrapper/as_agent_wrapper.py @@ -1,34 +1,100 @@ """AgentScope backend for the unified agent wrapper.""" -import datetime -import uuid -import zoneinfo +import json +import re +import time from collections.abc import AsyncGenerator from pathlib import Path from typing import Any, TYPE_CHECKING +from uuid import uuid4 -from agentscope.agent import Agent, ContextConfig, ModelConfig, ReActConfig +from agentscope.agent import Agent, ContextConfig, ReActConfig +from agentscope.agent._config import ModelConfig +from agentscope.event import ( + DataBlockDeltaEvent, + DataBlockEndEvent, + DataBlockStartEvent, + ExceedMaxItersEvent, + ModelCallEndEvent, + ModelCallStartEvent, + ReplyEndEvent, + ReplyStartEvent, + ThinkingBlockDeltaEvent, + ThinkingBlockEndEvent, + ThinkingBlockStartEvent, + TextBlockDeltaEvent, + TextBlockEndEvent, + TextBlockStartEvent, + ToolCallDeltaEvent, + ToolCallEndEvent, + ToolCallStartEvent, + ToolResultDataDeltaEvent, + ToolResultEndEvent, + ToolResultStartEvent, + ToolResultTextDeltaEvent, +) from agentscope.message import TextBlock, ToolResultState, UserMsg -from agentscope.permission import PermissionContext, PermissionMode +from agentscope.permission import PermissionBehavior, PermissionContext, PermissionDecision, PermissionMode from agentscope.state import AgentState -from agentscope.tool import FunctionTool, ToolChunk, Toolkit +from agentscope.tool import ( + Bash, + Edit, + FunctionTool, + Glob, + Grep, + Read, + ToolBase, + ToolChunk, + Toolkit, + Write, +) from .base_agent_wrapper import BaseAgentWrapper from ..as_llm import BaseAsLLM from ..component_registry import R +from ...enumeration import ChunkEnum +from ...schema import StreamChunk from ...utils import AsStateHandler +from ...utils.env_utils import load_env if TYPE_CHECKING: from ..job.base_job import BaseJob +_UUID_RE = re.compile( + r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", + re.IGNORECASE, +) + + +class BypassAnalysisBash(Bash): + """Bash variant that delegates permission decisions to PermissionEngine. + + AgentScope's built-in Bash performs bypass-immune static analysis before + the engine can apply `permission_mode: bypass`. For this app we want the + configured permission mode to be authoritative. + """ + + async def check_permissions( + self, + _tool_input: dict[str, Any], + _context: PermissionContext, + ) -> PermissionDecision: + """Bypass Bash static analysis and let the permission engine decide.""" + return PermissionDecision( + behavior=PermissionBehavior.PASSTHROUGH, + message="Bash static analysis skipped; delegating to permission engine.", + ) + @R.register("agentscope") class AsAgentWrapper(BaseAgentWrapper): """Agent wrapper backed by AgentScope framework.""" - def __init__(self, as_llm: str = "default", **kwargs): + def __init__(self, as_llm: str = "default", session_retention_days: int = 10, **kwargs): super().__init__(**kwargs) self.as_llm = self.bind(as_llm, BaseAsLLM, optional=False) + self.session_retention_days = int(session_retention_days) + self._session_cleanup_done = False @staticmethod def _make_tool(job: "BaseJob") -> FunctionTool: @@ -42,23 +108,121 @@ class AsAgentWrapper(BaseAgentWrapper): tool.input_schema = job.parameters return tool - def _build_agent(self, inputs: Any, **kwargs) -> tuple[Agent, Any]: + @classmethod + def _builtin_tools(cls) -> list[ToolBase]: + """Return built-in tools expected by local skills.""" + return [BypassAnalysisBash(), Edit(), Glob(), Grep(), Read(), Write()] + + @property + def session_path(self) -> Path: + """Directory used for persisted AgentScope sessions.""" + if self.app_context is None: + return self.vault_path / "session" / "agentscope" + return self.vault_path / self.app_context.app_config.session_dir / "agentscope" + + @staticmethod + def _validate_session_id(session_id: str, field: str = "session_id") -> str: + if not _UUID_RE.match(session_id): + raise ValueError(f"{field} must be a valid UUID: {session_id!r}") + return session_id.lower() + + def _cleanup_expired_sessions(self) -> None: + """Delete persisted session files older than ``session_retention_days``.""" + if self._session_cleanup_done or self.session_retention_days <= 0: + self._session_cleanup_done = True + return + + session_path = self.session_path + if not session_path.is_dir(): + self._session_cleanup_done = True + return + + cutoff = time.time() - self.session_retention_days * 24 * 60 * 60 + removed = 0 + for path in session_path.glob("*.jsonl"): + try: + if path.is_file() and path.stat().st_mtime < cutoff: + path.unlink() + removed += 1 + except OSError as exc: + self.logger.warning(f"Failed to clean expired AgentScope session {path}: {exc}") + + if removed: + self.logger.info( + f"Cleaned {removed} AgentScope session(s) older than {self.session_retention_days} day(s)", + ) + self._session_cleanup_done = True + + async def _load_state(self, kwargs: dict[str, Any], perm_mode: PermissionMode) -> AgentState: + resume = kwargs.get("resume") or "" + session_id = kwargs.get("session_id") or "" + fork_session = bool(kwargs.get("fork_session", False)) + if resume: + resume = self._validate_session_id(resume, "resume") + if session_id: + session_id = self._validate_session_id(session_id) + + if session_id and resume and not fork_session: + raise ValueError("session_id cannot be used with resume unless fork_session=True") + + if resume: + handler = AsStateHandler.for_session(self.session_path, resume) + state = await handler.load_or_none() + if state is None: + raise FileNotFoundError(f"AgentScope session not found: {resume}") + state.permission_context = PermissionContext(mode=perm_mode) + state.session_id = resume + if fork_session: + forked = AgentState( + session_id=session_id or str(uuid4()), + summary=state.summary, + context=list(state.context), + permission_context=PermissionContext(mode=perm_mode), + ) + return forked + return state + + return AgentState(session_id=session_id or str(uuid4()), permission_context=PermissionContext(mode=perm_mode)) + + async def _dump_state(self, state: AgentState) -> None: + await AsStateHandler.for_session(self.session_path, state.session_id).dump(state) + + def _resolve_skills(self, skills: list[str] | str | None) -> list[str]: + """Resolve configured skill names to AgentScope local skill directories.""" + if skills is None: + return [] + if skills == "all": + return [str(self.project_skills_root)] + if isinstance(skills, str): + skills = [skills] + return [str(self.project_skills_root / skill) for skill in skills] + + def _load_tool_env(self) -> dict[str, str]: + """Load project environment variables for tools spawned by AgentScope.""" + project_env = self.project_path / ".env" + return load_env(project_env) if project_env.exists() else load_env() + + async def _build_agent(self, inputs: Any, **kwargs) -> tuple[Agent, Any]: """Build an Agent instance from kwargs. Returns (agent, processed_inputs).""" model = self.as_llm.model if self.as_llm else None if model is None: raise ValueError("AsAgentWrapper requires a bound as_llm component with a valid model.") - for k, v in self.kwargs.items(): - kwargs.setdefault(k, v) + kwargs = self._merged_kwargs(kwargs) + self._cleanup_expired_sessions() + self._load_tool_env() system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.") - tools: list["BaseJob"] = kwargs.get("tools", []) - toolkit = kwargs.get("toolkit") or ( - Toolkit(tools=[self._make_tool(job) for job in tools]) if tools else Toolkit() + job_tools: list[str] = kwargs.get("job_tools", []) + resolved_jobs = self._resolve_job_tools(job_tools) + skills = self._resolve_skills(kwargs.get("skills")) + toolkit = kwargs.get("toolkit") or Toolkit( + tools=[*self._builtin_tools(), *(self._make_tool(job) for job in resolved_jobs)], + skills_or_loaders=skills, ) perm_mode = PermissionMode(kwargs.get("permission_mode", "bypass")) - state = AgentState(permission_context=PermissionContext(mode=perm_mode)) + state = await self._load_state(kwargs, perm_mode) agent = Agent( name=self.name, @@ -76,48 +240,20 @@ class AsAgentWrapper(BaseAgentWrapper): return agent, inputs - def _session_path(self, session_id: str) -> Path: - tz_name = self.app_context.app_config.timezone if self.app_context else None - if tz_name: - try: - dt = datetime.datetime.now(zoneinfo.ZoneInfo(tz_name)) - except Exception: - dt = datetime.datetime.now() - else: - dt = datetime.datetime.now() - date_str = dt.strftime("%Y-%m-%d") - resource = self.app_context.app_config.resource_dir if self.app_context else "resource" - return self.vault_path / resource / date_str / f"session_reme_{session_id}.jsonl" - - async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]: - session_id: str = kwargs.pop("session_id", "") - fork_session: bool = kwargs.pop("fork_session", False) - - agent, inputs = self._build_agent(inputs, **kwargs) - - if session_id: - path = self._session_path(session_id) - if path.exists(): - loaded = await AsStateHandler(path).load() - agent.state.session_id = loaded.session_id - agent.state.summary = loaded.summary - agent.state.context = loaded.context - agent.state.reply_id = loaded.reply_id - agent.state.cur_iter = loaded.cur_iter + async def reply(self, inputs: Any, **kwargs) -> dict: + kwargs = self._merged_kwargs(kwargs) + agent, inputs = await self._build_agent(inputs, **kwargs) await agent.observe(inputs) await agent.reply() + await self._dump_state(agent.state) last_msg = agent.state.context[-1] - if session_id: - if fork_session: - new_sid = uuid.uuid4().hex - agent.state.session_id = new_sid - save_path = self._session_path(new_sid) - else: - save_path = self._session_path(session_id) - save_path.parent.mkdir(parents=True, exist_ok=True) - await AsStateHandler(save_path).dump(agent.state) + result = { + "session_id": agent.state.session_id, + "last_message": last_msg.model_dump(), + "result": last_msg.get_text_content(), + } output_schema: dict | None = kwargs.get("output_schema") if output_schema is not None: @@ -128,13 +264,107 @@ class AsAgentWrapper(BaseAgentWrapper): messages=agent.state.context, structured_model=output_schema, ) - return agent.state.session_id, {"message": last_msg, "structured_output": res.content} + result["structured_output"] = res.content - return agent.state.session_id, last_msg + return result - async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[Any, None]: - """Stream agent events via AgentScope's reply_stream API.""" - agent, inputs = self._build_agent(inputs, **kwargs) + # ----- StreamChunk conversion ------------------------------------------- + + @classmethod + # pylint: disable=too-many-return-statements + def _event_to_chunk(cls, event: Any) -> StreamChunk | None: + """Convert an AgentScope event to a unified StreamChunk. + + Returns ``None`` for events that should be silently skipped + (e.g. ``RequireUserConfirmEvent``). + """ + if isinstance(event, ReplyStartEvent): + meta = {"reply_id": event.reply_id, "name": event.name, "role": event.role} + return cls._chunk(ChunkEnum.REPLY_START, session_id=event.session_id, chunk="", metadata=meta) + if isinstance(event, ReplyEndEvent): + return cls._chunk( + ChunkEnum.REPLY_END, + session_id=event.session_id, + chunk="", + metadata={"reply_id": event.reply_id}, + ) + + for event_cls, chunk_type, attr in ( + (TextBlockStartEvent, ChunkEnum.CONTENT, None), + (TextBlockDeltaEvent, ChunkEnum.CONTENT, "delta"), + (TextBlockEndEvent, ChunkEnum.CONTENT, None), + (ThinkingBlockStartEvent, ChunkEnum.THINK, None), + (ThinkingBlockDeltaEvent, ChunkEnum.THINK, "delta"), + (ThinkingBlockEndEvent, ChunkEnum.THINK, None), + (DataBlockStartEvent, ChunkEnum.DATA, None), + (DataBlockDeltaEvent, ChunkEnum.DATA, "data"), + (DataBlockEndEvent, ChunkEnum.DATA, None), + ): + if isinstance(event, event_cls): + kwargs = {"block_id": event.block_id, "chunk": getattr(event, attr) if attr else ""} + if isinstance(event, (DataBlockStartEvent, DataBlockDeltaEvent)): + kwargs["media_type"] = event.media_type + return cls._chunk(chunk_type, **kwargs) + + if isinstance(event, ToolCallStartEvent): + payload = {"name": event.tool_call_name, "id": event.tool_call_id} + return cls._chunk( + ChunkEnum.TOOL_CALL, + tool_call_id=event.tool_call_id, + tool_call_name=event.tool_call_name, + chunk=json.dumps(payload), + ) + if isinstance(event, ToolCallDeltaEvent): + return cls._chunk(ChunkEnum.TOOL_CALL, tool_call_id=event.tool_call_id, chunk=event.delta) + if isinstance(event, ToolCallEndEvent): + return cls._chunk(ChunkEnum.TOOL_CALL, tool_call_id=event.tool_call_id, chunk="") + if isinstance(event, ToolResultStartEvent): + return cls._chunk( + ChunkEnum.TOOL_RESULT, + tool_call_id=event.tool_call_id, + tool_call_name=event.tool_call_name, + chunk="", + ) + if isinstance(event, ToolResultTextDeltaEvent): + return cls._chunk(ChunkEnum.TOOL_RESULT, tool_call_id=event.tool_call_id, chunk=event.delta) + if isinstance(event, ToolResultDataDeltaEvent): + return cls._chunk( + ChunkEnum.TOOL_RESULT, + tool_call_id=event.tool_call_id, + chunk=event.data, + media_type=event.media_type, + metadata={"url": event.url} if event.url else {}, + ) + if isinstance(event, ToolResultEndEvent): + return cls._chunk( + ChunkEnum.TOOL_RESULT, + tool_call_id=event.tool_call_id, + chunk="", + metadata={"state": str(event.state)}, + ) + if isinstance(event, ModelCallStartEvent): + return cls._chunk(ChunkEnum.USAGE, chunk="", metadata={"model_name": getattr(event, "model_name", None)}) + if isinstance(event, ModelCallEndEvent): + usage = {"input_tokens": event.input_tokens, "output_tokens": event.output_tokens} + return cls._chunk( + ChunkEnum.USAGE, + chunk=json.dumps(usage), + input_tokens=event.input_tokens, + output_tokens=event.output_tokens, + metadata={"model_name": getattr(event, "model_name", None)}, + ) + if isinstance(event, ExceedMaxItersEvent): + return cls._chunk(ChunkEnum.ERROR, chunk="Exceeded max iterations") + return None + + async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]: + """Stream agent events as unified StreamChunk objects.""" + agent, inputs = await self._build_agent(inputs, **kwargs) async for event in agent.reply_stream(inputs): - yield event + chunk = self._event_to_chunk(event) + if chunk is not None: + chunk.session_id = chunk.session_id or agent.state.session_id + yield chunk + + await self._dump_state(agent.state) diff --git a/reme4/components/agent_wrapper/base_agent_wrapper.py b/reme4/components/agent_wrapper/base_agent_wrapper.py index 8c8f281b..a76853b0 100644 --- a/reme4/components/agent_wrapper/base_agent_wrapper.py +++ b/reme4/components/agent_wrapper/base_agent_wrapper.py @@ -2,12 +2,14 @@ from abc import abstractmethod from collections.abc import AsyncGenerator +from pathlib import Path from typing import Any, TYPE_CHECKING from pydantic import BaseModel from ..base_component import BaseComponent -from ...enumeration import ComponentEnum +from ...enumeration import ChunkEnum, ComponentEnum +from ...schema import StreamChunk if TYPE_CHECKING: from ..job.base_job import BaseJob @@ -23,11 +25,26 @@ class BaseAgentWrapper(BaseComponent): self.kwargs["system_prompt"] = prompt return self - def add_tools(self, tools: list["BaseJob"]) -> "BaseAgentWrapper": - """Append callable tools to the agent. Returns self for chaining.""" - self.kwargs.setdefault("tools", []).extend(tools) + def add_job_tools(self, job_tools: list[str]) -> "BaseAgentWrapper": + """Append job names as tools to the agent. Returns self for chaining.""" + self.kwargs.setdefault("job_tools", []).extend(job_tools) return self + def add_skills(self, skills: list[str] | str) -> "BaseAgentWrapper": + """Set agent skill names. Returns self for chaining.""" + self.kwargs["skills"] = skills + return self + + @property + def project_path(self) -> Path: + """Project root that contains shared assets such as skills.""" + return self.vault_path + + @property + def project_skills_root(self) -> Path: + """Project-level skills directory shared by agent backends.""" + return self.project_path / "skills" + def set_output_schema(self, schema: dict | type[BaseModel]) -> "BaseAgentWrapper": """Set a JSON schema for structured output. Accepts dict or BaseModel class. Returns self for chaining.""" if isinstance(schema, type) and issubclass(schema, BaseModel): @@ -35,17 +52,31 @@ class BaseAgentWrapper(BaseComponent): self.kwargs["output_schema"] = schema return self - # TODO add skills + def _resolve_job_tools(self, job_tools: list[str]) -> list["BaseJob"]: + """Resolve job name strings to BaseJob instances via app_context.""" + if not job_tools: + return [] + if self.app_context is None: + raise RuntimeError("Cannot resolve job_tools without an app_context") + resolved: list["BaseJob"] = [] + for name in job_tools: + if (job := self.app_context.jobs.get(name)) is None: + raise KeyError(f"Job '{name}' not found in app_context.jobs") + resolved.append(job) + return resolved + + def _merged_kwargs(self, kwargs: dict[str, Any]) -> dict[str, Any]: + """Merge component defaults with call-time kwargs; call-time values win.""" + return {**self.kwargs, **kwargs} + + @staticmethod + def _chunk(chunk_type: ChunkEnum = ChunkEnum.CONTENT, **kwargs: Any) -> StreamChunk: + """Create a StreamChunk with a short backend-friendly call site.""" + return StreamChunk(chunk_type=chunk_type, **kwargs) @abstractmethod - async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]: - """Send inputs to the agent and return (session_id, last_message).""" + async def reply(self, inputs: Any, **kwargs) -> dict: + """Send inputs to the agent and return a dict with session_id and last_message.""" - async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[Any, None]: - """Stream agent events. Yields backend-specific event objects. - - Subclasses may override to provide streaming support. - Default implementation falls back to non-streaming reply and yields the final message. - """ - _, msg = await self.reply(inputs, **kwargs) - yield msg + async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]: + """Stream agent events as unified StreamChunk objects.""" diff --git a/reme4/components/agent_wrapper/cc_agent_wrapper.py b/reme4/components/agent_wrapper/cc_agent_wrapper.py index c2731847..37650b1f 100644 --- a/reme4/components/agent_wrapper/cc_agent_wrapper.py +++ b/reme4/components/agent_wrapper/cc_agent_wrapper.py @@ -1,18 +1,216 @@ """Claude Code SDK backend for the unified agent wrapper.""" +import json +import os +import shutil +from collections.abc import AsyncGenerator +from dataclasses import asdict +from pathlib import Path from typing import Any, TYPE_CHECKING from .base_agent_wrapper import BaseAgentWrapper from ..component_registry import R +from ...enumeration import ChunkEnum +from ...schema import StreamChunk +from ...utils.env_utils import load_env if TYPE_CHECKING: from ..job.base_job import BaseJob + from claude_agent_sdk.types import SessionKey, SessionStoreEntry, SessionStoreListEntry + + +class _CcFileSessionStore: + """File-backed Claude Code SessionStore rooted under the ReMe vault.""" + + def __init__(self, root: Path) -> None: + self.root = root + + @staticmethod + def _safe_parts(value: str) -> list[str]: + parts = [part for part in value.split("/") if part] + if not parts or any(part in {".", ".."} for part in parts): + raise ValueError(f"Invalid session store path component: {value!r}") + return parts + + def _path_for_key(self, key: "SessionKey") -> Path: + session_id = key["session_id"] + subpath = key.get("subpath") + + path = self.root.joinpath(*self._safe_parts(session_id)) + if subpath: + path = path.joinpath(*self._safe_parts(subpath)) + else: + path = path.with_suffix(".jsonl") + if subpath: + path = path.with_suffix(".jsonl") + + resolved_root = self.root.resolve() + resolved_path = path.resolve() + if resolved_root != resolved_path and resolved_root not in resolved_path.parents: + raise ValueError(f"Session store path escapes root: {resolved_path}") + return path + + @staticmethod + def _read_entries(path: Path) -> list["SessionStoreEntry"]: + """Read JSONL session-store entries from disk.""" + if not path.exists(): + return [] + entries = [] + for line in path.read_text(encoding="utf-8").splitlines(): + if line.strip(): + entries.append(json.loads(line)) + return entries + + async def append(self, key: "SessionKey", entries: list["SessionStoreEntry"]) -> None: + """Append new session-store entries, deduplicating by UUID.""" + path = self._path_for_key(key) + path.parent.mkdir(parents=True, exist_ok=True) + + existing_uuids = { + entry.get("uuid") for entry in self._read_entries(path) if isinstance(entry, dict) and entry.get("uuid") + } + new_entries = [ + entry for entry in entries if not (isinstance(entry, dict) and entry.get("uuid") in existing_uuids) + ] + if not new_entries: + return + + with path.open("a", encoding="utf-8") as f: + for entry in new_entries: + f.write(json.dumps(entry, ensure_ascii=False, separators=(",", ":")) + "\n") + + async def load(self, key: "SessionKey") -> list["SessionStoreEntry"] | None: + """Load session-store entries for a key.""" + path = self._path_for_key(key) + if not path.exists(): + return None + return self._read_entries(path) + + async def list_sessions(self, _project_key: str) -> list["SessionStoreListEntry"]: + """List root-level Claude Code sessions.""" + if not self.root.exists(): + return [] + return [ + {"session_id": path.stem, "mtime": int(path.stat().st_mtime * 1000)} + for path in self.root.glob("*.jsonl") + if path.is_file() + ] + + async def delete(self, key: "SessionKey") -> None: + """Delete a session-store entry and any subkey directory.""" + path = self._path_for_key(key) + if path.exists(): + path.unlink() + + if not key.get("subpath"): + session_dir = self.root.joinpath(*self._safe_parts(key["session_id"])) + if session_dir.exists(): + for child in sorted(session_dir.rglob("*"), reverse=True): + if child.is_file(): + child.unlink() + elif child.is_dir(): + child.rmdir() + session_dir.rmdir() + + async def list_subkeys(self, key: dict[str, str]) -> list[str]: + """List subkeys below a root session key.""" + session_dir = self.root.joinpath(*self._safe_parts(key["session_id"])) + if not session_dir.exists(): + return [] + subkeys = [] + for path in session_dir.rglob("*.jsonl"): + if path.is_file(): + subkeys.append(str(path.relative_to(session_dir).with_suffix(""))) + return subkeys @R.register("claude_code") class CcAgentWrapper(BaseAgentWrapper): """Agent wrapper backed by Claude Code SDK.""" + DEFAULT_DISALLOWED_TOOLS = ["WebSearch"] + + @staticmethod + def _first_non_empty(*values: Any) -> str: + for value in values: + if isinstance(value, str) and value: + return value + return "" + + def _default_llm_credential(self) -> dict[str, Any]: + """Return the default as_llm credential config, if available.""" + if self.app_context is None: + return {} + components = self.app_context.app_config.components + llm_configs = components.get("as_llm") or components.get("AS_LLM") or components.get("as_llm".upper()) + if llm_configs is None: + from ...enumeration import ComponentEnum + + llm_configs = components.get(ComponentEnum.AS_LLM) + if not isinstance(llm_configs, dict): + return {} + + default_llm = llm_configs.get("default") + credential = getattr(default_llm, "credential", None) + return credential if isinstance(credential, dict) else {} + + def _claude_code_api_env(self, kwargs: dict[str, Any]) -> dict[str, str]: + """Resolve Anthropic-compatible API environment for Claude Code.""" + credential = kwargs.get("credential") if isinstance(kwargs.get("credential"), dict) else {} + default_credential = self._default_llm_credential() + + base_url = self._first_non_empty( + kwargs.get("base_url"), + credential.get("base_url"), + os.getenv("ANTHROPIC_BASE_URL"), + os.getenv("CLAUDE_CODE_BASE_URL"), + os.getenv("LLM_BASE_URL"), + default_credential.get("base_url"), + ) + api_key = self._first_non_empty( + kwargs.get("api_key"), + credential.get("api_key"), + os.getenv("ANTHROPIC_AUTH_TOKEN"), + os.getenv("CLAUDE_CODE_API_KEY"), + os.getenv("LLM_API_KEY"), + default_credential.get("api_key"), + ) + + env: dict[str, str] = {} + if base_url: + env["ANTHROPIC_BASE_URL"] = base_url + if api_key: + env["ANTHROPIC_AUTH_TOKEN"] = api_key + return env + + @property + def session_path(self) -> Path: + """Directory used for persisted Claude Code sessions.""" + if self.app_context is None: + return self.vault_path / "session" + return self.vault_path / self.app_context.app_config.session_dir + + def _ensure_claude_skill_dir(self, config_dir: Path) -> None: + """Expose project skills through Claude Code skill discovery locations.""" + project_skills = self.project_skills_root + if not project_skills.exists(): + return + + for target in (self.project_path / ".claude" / "skills", config_dir / "skills"): + target.parent.mkdir(parents=True, exist_ok=True) + try: + if target.exists() or target.is_symlink(): + if target.resolve() == project_skills.resolve(): + continue + if target.is_dir() and not target.is_symlink(): + shutil.rmtree(target) + else: + target.unlink() + + target.symlink_to(project_skills, target_is_directory=True) + except OSError as exc: + self.logger.warning(f"Failed to link Claude Code skills directory {target}: {exc}") + @staticmethod def _make_tool(job: "BaseJob"): from claude_agent_sdk import SdkMcpTool @@ -23,46 +221,69 @@ class CcAgentWrapper(BaseAgentWrapper): return SdkMcpTool(name=job.name, description=job.description, input_schema=job.parameters, handler=run_job) - async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]: - from claude_agent_sdk import query, ResultMessage, create_sdk_mcp_server + def _build_options(self, inputs: Any, stream: bool = False, **kwargs) -> Any: + """Build ClaudeAgentOptions from kwargs. + + ``stream=True`` enables ``include_partial_messages`` so that + ``StreamEvent`` messages are emitted alongside the final + ``ResultMessage``. + """ + from claude_agent_sdk import create_sdk_mcp_server from claude_agent_sdk.types import ClaudeAgentOptions - for k, v in self.kwargs.items(): - kwargs.setdefault(k, v) + kwargs = self._merged_kwargs(kwargs) - session_id: str = kwargs.pop("session_id", "") - fork_session: bool = kwargs.pop("fork_session", False) + skills = kwargs.get("skills") + if isinstance(skills, str) and skills != "all": + kwargs["skills"] = [skills] - sp = kwargs.get("system_prompt") - if isinstance(sp, str): - kwargs["system_prompt"] = { - "type": "preset", - "preset": "claude_code", - "append": sp, - "exclude_dynamic_sections": True, - } - kwargs.setdefault("setting_sources", []) + if "setting_sources" not in kwargs and kwargs.get("skills") is None: + kwargs["setting_sources"] = [] + disallowed_tools = list(kwargs.get("disallowed_tools") or []) + for tool_name in self.DEFAULT_DISALLOWED_TOOLS: + if tool_name not in disallowed_tools: + disallowed_tools.append(tool_name) + kwargs["disallowed_tools"] = disallowed_tools opts = ClaudeAgentOptions() - skip_keys = {"tools", "output_schema"} + if stream: + opts.include_partial_messages = True + + skip_keys = {"job_tools", "output_schema", "api_key", "base_url", "credential"} for k, v in kwargs.items(): if k not in skip_keys and hasattr(opts, k): setattr(opts, k, v) - if session_id: - opts.resume = session_id - if fork_session: - opts.fork_session = True - elif fork_session: - # fork_session with no session_id to fork from is meaningless. - raise ValueError("fork_session=True requires a non-empty session_id") + model = getattr(opts, "model", None) or kwargs.get("model") + project_env = self.project_path / ".env" + opts.env.update(load_env(project_env) if project_env.exists() else load_env()) + extra_env_dict: dict = self._claude_code_api_env(kwargs) + if model: + extra_env_dict.update( + { + "ANTHROPIC_MODEL": model, + "ANTHROPIC_DEFAULT_HAIKU_MODEL": model, + "ANTHROPIC_DEFAULT_SONNET_MODEL": model, + "ANTHROPIC_DEFAULT_OPUS_MODEL": model, + }, + ) + opts.env.update(extra_env_dict) + self.session_path.mkdir(parents=True, exist_ok=True) + opts.cwd = opts.cwd or self.project_path + claude_config_dir = self.session_path / "claude_config" + opts.env.setdefault("CLAUDE_CONFIG_DIR", str(claude_config_dir)) + if opts.skills is not None: + self._ensure_claude_skill_dir(claude_config_dir) + opts.session_store = opts.session_store or _CcFileSessionStore(self.session_path / "claude_code") - tools: list["BaseJob"] = kwargs.get("tools", []) - if tools: - sdk_tools = [self._make_tool(job) for job in tools] - server = create_sdk_mcp_server(name="reme_tools", tools=sdk_tools) - opts.mcp_servers = (opts.mcp_servers if isinstance(opts.mcp_servers, dict) else {}) | {"reme": server} - opts.allowed_tools.extend(job.name for job in tools) + job_tools: list[str] = kwargs.get("job_tools", []) + resolved_jobs = self._resolve_job_tools(job_tools) + if resolved_jobs: + sdk_tools = [self._make_tool(job) for job in resolved_jobs] + server = create_sdk_mcp_server(name="mcp_server", tools=sdk_tools) + opts.mcp_servers = opts.mcp_servers if isinstance(opts.mcp_servers, dict) else {} + opts.mcp_servers["mcp_server"] = server + opts.allowed_tools.extend(job.name for job in resolved_jobs) if output_schema := kwargs.get("output_schema"): opts.output_format = {"type": "json_schema", "schema": output_schema} @@ -70,6 +291,201 @@ class CcAgentWrapper(BaseAgentWrapper): if not isinstance(inputs, str): raise NotImplementedError("Only string input is supported for Claude Code.") + return opts + + # ----- StreamChunk conversion ------------------------------------------- + + @classmethod + # pylint: disable=too-many-return-statements + def _raw_event_to_chunk( + cls, + raw: dict, + session_id: str | None = None, + block_ids: dict[int, str] | None = None, + block_types: dict[int, str] | None = None, + tool_call_names: dict[int, str] | None = None, + ) -> StreamChunk | None: + """Convert a raw Anthropic streaming event dict to a StreamChunk. + + ``block_ids`` / ``block_types`` / ``tool_call_names`` map + content-block ``index`` to metadata tracked from the + ``content_block_start`` event, so that later delta / stop + events can reference the correct ``block_id`` and + ``chunk_type``. + + Returns ``None`` for events that should be silently skipped. + """ + event_type = raw.get("type") + + # --- Message-level lifecycle ---------------------------------------- + + if event_type == "message_start": + message = raw.get("message", {}) + meta = {"message_id": message.get("id"), "model": message.get("model"), "role": message.get("role")} + return cls._chunk(ChunkEnum.REPLY_START, session_id=session_id, chunk="", metadata=meta) + + if event_type == "message_delta": + delta = raw.get("delta", {}) + usage = raw.get("usage", {}) + return cls._chunk( + ChunkEnum.REPLY_END, + session_id=session_id, + chunk="", + output_tokens=usage.get("output_tokens"), + metadata={"stop_reason": delta.get("stop_reason")}, + ) + + if event_type == "message_stop": + return cls._chunk(ChunkEnum.REPLY_END, session_id=session_id, chunk="") + + # --- Content-block lifecycle ---------------------------------------- + + if event_type == "content_block_start": + idx, content_block = raw.get("index", 0), raw.get("content_block", {}) + block_type, bid = content_block.get("type", ""), content_block.get("id", "") + + # Track for later delta / stop correlation + if block_ids is not None and bid: + block_ids[idx] = bid + if block_types is not None and block_type: + block_types[idx] = block_type + if tool_call_names is not None and content_block.get("name"): + tool_call_names[idx] = content_block["name"] + + if block_type == "text": + return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk=content_block.get("text", "")) + if block_type == "thinking": + return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk=content_block.get("thinking", "")) + if block_type == "tool_use": + payload = {"name": content_block.get("name"), "id": content_block.get("id")} + return cls._chunk( + ChunkEnum.TOOL_CALL, + block_id=bid, + tool_call_id=content_block.get("id"), + tool_call_name=content_block.get("name"), + chunk=json.dumps(payload), + ) + return None + + if event_type == "content_block_delta": + delta = raw.get("delta", {}) + delta_type = delta.get("type", "") + idx = raw.get("index", 0) + bid = block_ids.get(idx) if block_ids else None + tc_name = tool_call_names.get(idx) if tool_call_names else None + + if delta_type == "text_delta": + return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk=delta.get("text", "")) + if delta_type == "thinking_delta": + return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk=delta.get("thinking", "")) + if delta_type == "input_json_delta": + return cls._chunk( + ChunkEnum.TOOL_CALL, + block_id=bid, + tool_call_id=bid, + tool_call_name=tc_name, + chunk=delta.get("partial_json", ""), + ) + return None + + if event_type == "content_block_stop": + idx = raw.get("index", 0) + bid = block_ids.get(idx) if block_ids else None + btype = block_types.get(idx) if block_types else None + tc_name = tool_call_names.get(idx) if tool_call_names else None + + if btype == "tool_use": + return cls._chunk(ChunkEnum.TOOL_CALL, block_id=bid, tool_call_id=bid, tool_call_name=tc_name, chunk="") + if btype == "thinking": + return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk="") + # text or unknown -> CONTENT + return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk="") + + # Ping / other unknown types -> skip + return None + + @classmethod + def _message_content_to_chunks( + cls, + msg: Any, + session_id: str | None = None, + visible_tool_call_ids: set[str] | None = None, + include_text: bool = False, + ) -> list[StreamChunk]: + """Convert non-partial SDK message content blocks into stream chunks. + + Claude Code streams assistant text/tool-use deltas as ``StreamEvent`` + objects, but tool results can arrive later as regular message content + blocks. Surface those blocks so the UI can show what each tool + returned. Some SDK/CLI combinations also put assistant text only in + regular message blocks, so callers can opt into text conversion. + """ + chunks: list[StreamChunk] = [] + content = getattr(msg, "content", None) + if not isinstance(content, list): + return chunks + + for block in content: + block_name = block.__class__.__name__ + if include_text and block_name == "TextBlock": + text = getattr(block, "text", "") + if text: + chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=text)) + elif include_text and isinstance(block, dict) and block.get("type") == "text": + text = block.get("text", "") + if text: + chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=text)) + elif include_text and isinstance(block, str): + chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=block)) + elif block_name in {"ToolResultBlock", "ServerToolResultBlock"}: + tool_use_id = getattr(block, "tool_use_id", None) + if visible_tool_call_ids is not None and tool_use_id not in visible_tool_call_ids: + continue + payload: dict[str, Any] = { + "tool_use_id": tool_use_id, + "content": getattr(block, "content", None), + } + if hasattr(block, "is_error"): + payload["is_error"] = getattr(block, "is_error") + chunks.append( + cls._chunk( + ChunkEnum.TOOL_RESULT, + session_id=session_id, + block_id=tool_use_id, + tool_call_id=tool_use_id, + chunk=payload, + ), + ) + + return chunks + + @staticmethod + def _result_message_is_error(msg: Any) -> bool: + """Return whether an SDK ResultMessage represents a failed result.""" + subtype = getattr(msg, "subtype", None) + if isinstance(subtype, str) and subtype.lower() == "success": + return False + + is_error = getattr(msg, "is_error", False) + if isinstance(is_error, bool): + return is_error + if isinstance(is_error, str): + return is_error.lower() in {"true", "error", "errored", "failed", "failure"} + + return isinstance(subtype, str) and subtype.lower() in {"error", "failed", "failure"} + + @staticmethod + def _is_trailing_success_error(exc: Exception) -> bool: + """Return whether an SDK iterator error is the known success-exit artifact.""" + return "Claude Code returned an error result: success" in str(exc) + + # ----- reply / reply_stream -------------------------------------------- + + async def reply(self, inputs: Any, **kwargs) -> dict: + from claude_agent_sdk import query, ResultMessage + + opts = self._build_options(inputs, stream=False, **kwargs) + last_msg = None async for msg in query(prompt=inputs, options=opts): if isinstance(msg, ResultMessage): @@ -78,7 +494,108 @@ class CcAgentWrapper(BaseAgentWrapper): if last_msg is None: raise ValueError("No message received from Claude Code.") - if output_schema: - structured = last_msg.structured_output or {} - return last_msg.session_id or "", {"message": last_msg, "structured_output": structured} - return last_msg.session_id or "", last_msg + result = { + "session_id": last_msg.session_id or "", + "last_message": asdict(last_msg), + "result": last_msg.result, + } + output_schema = kwargs.get("output_schema") or self.kwargs.get("output_schema") + if output_schema and last_msg.structured_output: + result["structured_output"] = last_msg.structured_output + return result + + async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]: + """Stream Claude Code events as unified StreamChunk objects.""" + from claude_agent_sdk import query, ResultMessage, AssistantMessage, StreamEvent, UserMessage + from claude_agent_sdk.types import RateLimitEvent + + opts = self._build_options(inputs, stream=True, **kwargs) + + block_ids: dict[int, str] = {} + block_types: dict[int, str] = {} + tool_call_names: dict[int, str] = {} + visible_tool_call_ids: set[str] = set() + current_session_id: str | None = None + emitted_content = False + received_result_message = False + + stream = query(prompt=inputs, options=opts) + try: + async for msg in stream: + if isinstance(msg, StreamEvent): + current_session_id = msg.session_id or current_session_id + chunk = self._raw_event_to_chunk( + msg.event, + session_id=msg.session_id, + block_ids=block_ids, + block_types=block_types, + tool_call_names=tool_call_names, + ) + if chunk is not None: + chunk.session_id = chunk.session_id or msg.session_id + if chunk.chunk_type == ChunkEnum.TOOL_CALL and chunk.tool_call_id: + visible_tool_call_ids.add(chunk.tool_call_id) + if chunk.chunk_type == ChunkEnum.CONTENT and chunk.chunk: + emitted_content = True + yield chunk + + elif isinstance(msg, UserMessage): + for chunk in self._message_content_to_chunks(msg, current_session_id, visible_tool_call_ids): + yield chunk + + elif isinstance(msg, ResultMessage): + received_result_message = True + current_session_id = msg.session_id or current_session_id + if not emitted_content and getattr(msg, "result", None): + emitted_content = True + yield self._chunk(ChunkEnum.CONTENT, session_id=msg.session_id or "", chunk=msg.result) + # Final result: emit USAGE + REPLY_END + meta = { + "duration_ms": msg.duration_ms, + "duration_api_ms": msg.duration_api_ms, + "stop_reason": msg.stop_reason, + "num_turns": msg.num_turns, + } + yield self._chunk( + ChunkEnum.USAGE, + session_id=msg.session_id or "", + chunk=json.dumps(msg.usage or {}), + metadata=meta, + ) + if self._result_message_is_error(msg): + yield self._chunk( + ChunkEnum.ERROR, + session_id=msg.session_id or "", + chunk=str(msg.errors) if msg.errors else "Unknown error", + ) + yield self._chunk(ChunkEnum.REPLY_END, session_id=msg.session_id or "", chunk="") + + elif isinstance(msg, AssistantMessage): + current_session_id = msg.session_id or current_session_id + # Intermediate assistant text/tool-use is already streamed + # via StreamEvents. Still surface tool-result blocks if the + # SDK includes any in a regular assistant message. + for chunk in self._message_content_to_chunks( + msg, + current_session_id, + visible_tool_call_ids, + include_text=not emitted_content, + ): + if chunk.chunk_type == ChunkEnum.CONTENT and chunk.chunk: + emitted_content = True + yield chunk + + elif isinstance(msg, RateLimitEvent): + yield self._chunk(ChunkEnum.ERROR, session_id=msg.session_id, chunk="Rate limit exceeded") + except Exception as exc: + if received_result_message and self._is_trailing_success_error(exc): + self.logger.debug(f"Ignoring Claude Code trailing success error after final result: {exc}") + else: + raise + finally: + try: + await stream.aclose() + except Exception as exc: + if not (received_result_message and self._is_trailing_success_error(exc)): + raise + self.logger.debug(f"Ignoring Claude Code stream close error after final result: {exc}") diff --git a/reme4/components/as_embedding/__init__.py b/reme4/components/as_embedding/__init__.py index 995a3dfe..1305437f 100644 --- a/reme4/components/as_embedding/__init__.py +++ b/reme4/components/as_embedding/__init__.py @@ -1,12 +1,16 @@ """AgentScope embedding model wrappers.""" +from typing import Any + +from agentscope.credential import ( + CredentialBase, + DashScopeCredential, + GeminiCredential, + OllamaCredential, + OpenAICredential, +) from agentscope.embedding import ( - DashScopeMultiModalEmbedding, - DashScopeTextEmbedding, EmbeddingModelBase, - GeminiTextEmbedding, - OllamaTextEmbedding, - OpenAITextEmbedding, ) from ..base_component import BaseComponent @@ -15,13 +19,14 @@ from ...enumeration import ComponentEnum class BaseAsEmbedding(BaseComponent): - """Base wrapper for AgentScope embedding models. Builds ``self.model`` in ``_start``.""" + """Base wrapper for AgentScope embedding models.""" component_type = ComponentEnum.AS_EMBEDDING + credential_cls: type[CredentialBase] def __init__(self, **kwargs) -> None: super().__init__(**kwargs) - self.model: EmbeddingModelBase | None = None + self.model: EmbeddingModelBase[Any] | None = None @property def dimensions(self) -> int: @@ -29,58 +34,60 @@ class BaseAsEmbedding(BaseComponent): assert self.model is not None return self.model.dimensions - async def __call__(self, text: list[str], **kwargs) -> list[list[float]]: + async def __call__(self, inputs: list[Any], **kwargs) -> list[list[float]]: assert self.model is not None - response = await self.model(text, **kwargs) # pylint: disable=not-callable + response = await self.model(inputs, **kwargs) # pylint: disable=not-callable return response.embeddings - async def _close(self) -> None: - self.model = None + async def _start(self) -> None: + if self.model is not None: + return + + kwargs = dict(self.kwargs) + credential = self.credential_cls(**kwargs.pop("credential", {})) + + model_cls = self.credential_cls.get_embedding_model_class() + if model_cls is None: + raise ValueError(f"{self.credential_cls.__name__} does not support embeddings.") + + params_dict = kwargs.pop("parameters", None) + parameters = model_cls.Parameters(**params_dict) if params_dict else None + self.model = model_cls(credential=credential, parameters=parameters, **kwargs) @R.register("openai") class OpenAIAsEmbedding(BaseAsEmbedding): """OpenAI embedding model wrapper.""" - async def _start(self) -> None: - self.model = OpenAITextEmbedding(**self.kwargs) - - async def _close(self) -> None: - if self.model is not None: - assert isinstance(self.model, OpenAITextEmbedding) - await self.model.client.close() + credential_cls = OpenAICredential @R.register("dashscope") class DashScopeAsEmbedding(BaseAsEmbedding): - """DashScope text embedding model wrapper.""" + """DashScope embedding model wrapper.""" - async def _start(self) -> None: - self.model = DashScopeTextEmbedding(**self.kwargs) + credential_cls = DashScopeCredential @R.register("dashscope_multimodal") class DashScopeMultiModalAsEmbedding(BaseAsEmbedding): """DashScope multimodal embedding model wrapper.""" - async def _start(self) -> None: - self.model = DashScopeMultiModalEmbedding(**self.kwargs) + credential_cls = DashScopeCredential @R.register("gemini") class GeminiAsEmbedding(BaseAsEmbedding): """Gemini embedding model wrapper.""" - async def _start(self) -> None: - self.model = GeminiTextEmbedding(**self.kwargs) + credential_cls = GeminiCredential @R.register("ollama") class OllamaAsEmbedding(BaseAsEmbedding): """Ollama embedding model wrapper.""" - async def _start(self) -> None: - self.model = OllamaTextEmbedding(**self.kwargs) + credential_cls = OllamaCredential __all__ = [ diff --git a/reme4/components/as_llm/__init__.py b/reme4/components/as_llm/__init__.py index 7608986b..d4c5007d 100644 --- a/reme4/components/as_llm/__init__.py +++ b/reme4/components/as_llm/__init__.py @@ -32,6 +32,8 @@ class BaseAsLLM(BaseComponent): self.model: ChatModelBase | None = None async def _start(self) -> None: + if self.model is not None: + return kwargs = dict(self.kwargs) credential = self.credential_cls(**kwargs.pop("credential", {})) model_cls = credential.get_chat_model_class() @@ -39,9 +41,6 @@ class BaseAsLLM(BaseComponent): parameters = model_cls.Parameters(**params_dict) if params_dict else None self.model = model_cls(credential=credential, parameters=parameters, **kwargs) - async def _close(self) -> None: - self.model = None - @R.register("openai") class OpenAIAsLLM(BaseAsLLM): diff --git a/reme4/components/base_component.py b/reme4/components/base_component.py index 85b5a5ec..7ac273bd 100644 --- a/reme4/components/base_component.py +++ b/reme4/components/base_component.py @@ -220,10 +220,23 @@ class BaseComponent(ComponentMixin, ABC): async with self._lock: if not self._is_started: return - await self._close() - for owned in reversed(self._owned): - await owned.close() - self._is_started = False + first_error: BaseException | None = None + try: + await self._close() + except BaseException as exc: + first_error = exc + finally: + for owned in reversed(self._owned): + try: + await owned.close() + except BaseException as exc: + if first_error is None: + first_error = exc + else: + self.logger.exception(f"Failed to close owned component {owned.name}: {exc}") + self._is_started = False + if first_error is not None: + raise first_error async def restart(self) -> None: """Close then start the component.""" diff --git a/reme4/components/embedding_store/local_embedding_store.py b/reme4/components/embedding_store/local_embedding_store.py index 1e10413e..6f70eb20 100644 --- a/reme4/components/embedding_store/local_embedding_store.py +++ b/reme4/components/embedding_store/local_embedding_store.py @@ -172,17 +172,17 @@ class LocalEmbeddingStore(BaseEmbeddingStore): def _load_sync(self) -> None: try: - data = np.load(self.cache_path) + with np.load(self.cache_path) as data: + for key, emb in zip(data["keys"], data["embeddings"]): + if len(emb) != self.dimensions: + continue + if len(self._cache) >= self.max_cache_size: + break + self._cache[str(key)] = emb.astype(np.float16) except Exception: self.logger.exception("Failed to load embedding cache, removing") self.cache_path.unlink(missing_ok=True) return - for key, emb in zip(data["keys"], data["embeddings"]): - if len(emb) != self.dimensions: - continue - if len(self._cache) >= self.max_cache_size: - break - self._cache[str(key)] = emb.astype(np.float16) self.logger.info(f"Loaded {len(self._cache)} embeddings from {self.cache_path}") async def dump(self) -> None: diff --git a/reme4/components/file_catalog/local_file_catalog.py b/reme4/components/file_catalog/local_file_catalog.py index 150be1bb..1e6c3d4a 100644 --- a/reme4/components/file_catalog/local_file_catalog.py +++ b/reme4/components/file_catalog/local_file_catalog.py @@ -1,10 +1,11 @@ -"""Local file catalog backend: in-memory dict persisted as JSONL.""" +"""Local file catalog backend: in-memory dict persisted as compressed JSONL.""" -import aiofiles +import asyncio from .base_file_catalog import BaseFileCatalog from ..component_registry import R from ...schema import FileNode +from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst @R.register("local") @@ -15,48 +16,44 @@ class LocalFileCatalog(BaseFileCatalog): super().__init__(**kwargs) self.encoding = encoding self._nodes: dict[str, FileNode] = {} + self._io_lock = asyncio.Lock() self.component_metadata_path.mkdir(parents=True, exist_ok=True) - self._catalog_file = self.component_metadata_path / f"{self.name}.jsonl" + self._catalog_file = self.component_metadata_path / f"{self.name}.jsonl.zst" async def load(self) -> None: - if not self._catalog_file.exists(): - return - try: + async with self._io_lock: + if not self._catalog_file.exists(): + return await self._read_jsonl() self.logger.info(f"Loaded {len(self._nodes)} nodes from {self._catalog_file}") - except Exception as e: - self.logger.exception(f"Failed to load {self._catalog_file}: {e}") async def dump(self) -> None: - try: + async with self._io_lock: await self._write_jsonl() self.logger.info(f"Saved {len(self._nodes)} nodes to {self._catalog_file}") - except Exception as e: - self.logger.exception(f"Failed to write {self._catalog_file}: {e}") async def upsert(self, nodes: list[FileNode]) -> None: - for node in nodes: - self._nodes[node.path] = node + async with self._io_lock: + for node in nodes: + self._nodes[node.path] = node async def delete(self, path: str | list[str]) -> None: paths = [path] if isinstance(path, str) else path - for p in paths: - self._nodes.pop(p, None) + async with self._io_lock: + for p in paths: + self._nodes.pop(p, None) async def get_nodes(self, paths: list[str] | None = None) -> list[FileNode]: - if paths is None: - return list(self._nodes.values()) - return [self._nodes[p] for p in paths if p in self._nodes] + async with self._io_lock: + if paths is None: + return list(self._nodes.values()) + return [self._nodes[p] for p in paths if p in self._nodes] async def _read_jsonl(self) -> None: - async with aiofiles.open(self._catalog_file, encoding=self.encoding) as f: - async for line in f: - if stripped := line.strip(): - node = FileNode.model_validate_json(stripped) - self._nodes[node.path] = node + for line in read_jsonl_zst(self._catalog_file, self.encoding): + if stripped := line.strip(): + node = FileNode.model_validate_json(stripped) + self._nodes[node.path] = node async def _write_jsonl(self) -> None: - tmp = self._catalog_file.with_suffix(".tmp") - async with aiofiles.open(tmp, "w", encoding=self.encoding) as f: - await f.write("\n".join(n.model_dump_json() for n in self._nodes.values())) - tmp.replace(self._catalog_file) + write_jsonl_zst(self._catalog_file, (n.model_dump_json() for n in self._nodes.values()), self.encoding) diff --git a/reme4/components/file_chunker/markdown_file_chunker.py b/reme4/components/file_chunker/markdown_file_chunker.py index fc996edc..a77fb1eb 100644 --- a/reme4/components/file_chunker/markdown_file_chunker.py +++ b/reme4/components/file_chunker/markdown_file_chunker.py @@ -18,7 +18,8 @@ from dataclasses import dataclass, field from pathlib import Path from typing import Any -import frontmatter +import yaml +from pydantic import ValidationError from .base_file_chunker import BaseFileChunker @@ -128,26 +129,51 @@ class MarkdownFileChunker(BaseFileChunker): file_path = Path(path) rel_path = self.to_vault_relative(path) - post = frontmatter.loads(file_path.read_text(encoding=self.encoding)) + front_matter, content, line_offset = self._parse_front_matter(file_path.read_text(encoding=self.encoding)) chunks: list[FileChunk] = [] - if post.content and post.content.strip(): + if content and content.strip(): with MarkdownRenderer() as renderer: - tree = self._build_tree(Document(post.content), renderer) + tree = self._build_tree(Document(content), renderer, line_offset=line_offset) chunks = self._chunk_node(tree, "", "", rel_path, renderer) - links = WikilinkHandler.extract_links(post.content, rel_path) if post.content else [] + links = WikilinkHandler.extract_links(content, rel_path) if content else [] node = FileNode( path=rel_path, st_mtime=file_path.stat().st_mtime, chunk_ids=[chunk.id for chunk in chunks], links=links, - front_matter=FileFrontMatter(**dict(post.metadata)), + front_matter=front_matter, ) return node, chunks - def _build_tree(self, doc: Any, renderer) -> MdNode: + @staticmethod + def _parse_front_matter(text: str) -> tuple[FileFrontMatter, str, int]: + """Parse YAML frontmatter, returning 1-based line offset for body AST lines. + + Invalid YAML is ignored so a single bad frontmatter block does not + prevent indexing the markdown body. + """ + lines = text.splitlines(keepends=True) + if not lines or lines[0].strip() != "---": + return FileFrontMatter(), text, 0 + + close_idx = next((i for i, line in enumerate(lines[1:], 1) if line.strip() == "---"), None) + if close_idx is None: + return FileFrontMatter(), text, 0 + + front_matter = FileFrontMatter() + try: + data = yaml.safe_load("".join(lines[1:close_idx]).strip()) or {} + if isinstance(data, dict): + front_matter = FileFrontMatter(**data) + except (yaml.YAMLError, TypeError, ValidationError): + front_matter = FileFrontMatter() + + return front_matter, "".join(lines[close_idx + 1 :]), close_idx + 1 + + def _build_tree(self, doc: Any, renderer, line_offset: int = 0) -> MdNode: """Heading-level stack folds mistletoe's flat children into nested sections; non-headings attach as ``body`` to the current section (or root before the first heading).""" @@ -157,12 +183,13 @@ class MarkdownFileChunker(BaseFileChunker): SetextHeading, ) - root = MdNode(kind="root", start_line=1, end_line=1) + root = MdNode(kind="root", start_line=line_offset + 1, end_line=line_offset + 1) stack: list[MdNode] = [root] for child in doc.children or []: if isinstance(child, BlankLine): continue - line = getattr(child, "line_number", None) or stack[-1].start_line + raw_line = getattr(child, "line_number", None) + line = raw_line + line_offset if raw_line is not None else stack[-1].start_line if isinstance(child, (Heading, SetextHeading)): level = max(1, getattr(child, "level", 1)) while len(stack) > 1 and stack[-1].level >= level: @@ -369,10 +396,11 @@ class MarkdownFileChunker(BaseFileChunker): lines = body.text.split("\n") header, data = "\n".join(lines[:2]), lines[2:] rows = [r for r in (body.block.children or []) if isinstance(r, TableRow)] + line_offset = body.start_line - (getattr(body.block, "line_number", None) or body.start_line) base = body.start_line + 2 def line_of(i: int) -> int: - return rows[i].line_number if i < len(rows) and rows[i].line_number else base + i + return rows[i].line_number + line_offset if i < len(rows) and rows[i].line_number else base + i units = [(text, line_of(i), line_of(i)) for i, text in enumerate(data)] return self._emit_packed( @@ -426,11 +454,12 @@ class MarkdownFileChunker(BaseFileChunker): if not items: return self._split_lines(body, before, after, path) units: list[tuple[str, int, int]] = [] + line_offset = body.start_line - (getattr(body.block, "line_number", None) or body.start_line) for it in items: text = renderer.render(it).rstrip("\n") if not text: continue - line = it.line_number or body.start_line + line = it.line_number + line_offset if it.line_number else body.start_line units.append((text, line, line + text.count("\n"))) return self._emit_packed( units, diff --git a/reme4/components/file_graph/local_file_graph.py b/reme4/components/file_graph/local_file_graph.py index 6726de1e..a02602be 100644 --- a/reme4/components/file_graph/local_file_graph.py +++ b/reme4/components/file_graph/local_file_graph.py @@ -6,6 +6,7 @@ from .base_file_graph import BaseFileGraph from ..component_registry import R from ...enumeration import LinkScopeEnum from ...schema import FileLink, FileNode +from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst @R.register("local") @@ -17,7 +18,7 @@ class LocalFileGraph(BaseFileGraph): self._nodes: dict[str, FileNode] = {} self._inverse: dict[str, set[str]] = {} # real target → sources self._pending: dict[str, set[str]] = {} # virtual target → sources - self._graph_file: Path = self.component_metadata_path / f"{self.name}.jsonl" + self._graph_file: Path = self.component_metadata_path / f"{self.name}.jsonl.zst" # -- Lifecycle --------------------------------------------------------- @@ -30,21 +31,17 @@ class LocalFileGraph(BaseFileGraph): if not self._graph_file.exists(): return try: - with open(self._graph_file, "r", encoding="utf-8") as f: - for line in f: - if line.strip(): - node = FileNode.model_validate_json(line) - self._nodes[node.path] = node + for line in read_jsonl_zst(self._graph_file): + if line.strip(): + node = FileNode.model_validate_json(line) + self._nodes[node.path] = node self.logger.info(f"Loaded {len(self._nodes)} nodes from {self._graph_file}") except Exception as e: self.logger.exception(f"Failed to load {self._graph_file}: {e}") async def dump(self) -> None: try: - tmp = self._graph_file.with_suffix(".tmp") - with open(tmp, "w", encoding="utf-8") as f: - f.writelines(f"{n.model_dump_json()}\n" for n in self._nodes.values()) - tmp.replace(self._graph_file) + write_jsonl_zst(self._graph_file, (n.model_dump_json() for n in self._nodes.values())) self.logger.info(f"Saved {len(self._nodes)} nodes to {self._graph_file}") except Exception as e: self.logger.exception(f"Failed to write {self._graph_file}: {e}") @@ -67,7 +64,12 @@ class LocalFileGraph(BaseFileGraph): if not srcs: del bucket[target] - def _scope_match(self, target: str, scope: LinkScopeEnum) -> bool: + @staticmethod + def _normalize_scope(scope: LinkScopeEnum | str) -> LinkScopeEnum: + return scope if isinstance(scope, LinkScopeEnum) else LinkScopeEnum(scope) + + def _scope_match(self, target: str, scope: LinkScopeEnum | str) -> bool: + scope = self._normalize_scope(scope) if scope is LinkScopeEnum.ALL: return True is_real = target in self._nodes @@ -120,18 +122,24 @@ class LocalFileGraph(BaseFileGraph): # -- Link access ------------------------------------------------------- - async def get_outlinks(self, path: str, scope: LinkScopeEnum = LinkScopeEnum.REAL) -> list[FileLink]: + async def get_outlinks(self, path: str, scope: LinkScopeEnum | str = LinkScopeEnum.REAL) -> list[FileLink]: + scope = self._normalize_scope(scope) node = self._nodes.get(path) if node is None: return [] return [lnk for lnk in node.links if lnk.target_path and self._scope_match(lnk.target_path, scope)] - async def get_inlinks(self, path: str, scope: LinkScopeEnum = LinkScopeEnum.REAL) -> list[FileLink]: + async def get_inlinks(self, path: str, scope: LinkScopeEnum | str = LinkScopeEnum.REAL) -> list[FileLink]: + scope = self._normalize_scope(scope) sources: set[str] = set() if scope in (LinkScopeEnum.REAL, LinkScopeEnum.ALL): sources |= self._inverse.get(path, set()) if scope in (LinkScopeEnum.VIRTUAL, LinkScopeEnum.ALL): sources |= self._pending.get(path, set()) return [ - link for src in sources if src in self._nodes for link in self._nodes[src].links if link.target_path == path + link + for src in sorted(sources) + if src in self._nodes + for link in self._nodes[src].links + if link.target_path == path ] diff --git a/reme4/components/file_graph/neo4j_file_graph.py b/reme4/components/file_graph/neo4j_file_graph.py index c2fbd13e..1bc68439 100644 --- a/reme4/components/file_graph/neo4j_file_graph.py +++ b/reme4/components/file_graph/neo4j_file_graph.py @@ -88,6 +88,9 @@ class Neo4jFileGraph(BaseFileGraph): ) self._database: str = database self._driver = None + self._n_nodes = 0 + self._n_virtual = 0 + self._n_edges = 0 # -- Lifecycle --------------------------------------------------------- @@ -107,11 +110,11 @@ class Neo4jFileGraph(BaseFileGraph): await session.run( "CREATE CONSTRAINT file_path_unique IF NOT EXISTS FOR (f:File) REQUIRE f.path IS UNIQUE", ) - real, virtual, edges = await self._counts(session) + await self._refresh_counts(session) self.logger.info( f"Neo4jFileGraph '{self.name}' connected at " f"{self._uri}/{self._database}: " - f"{real} nodes, {edges} edges, {virtual} virtual", + f"{self._n_nodes} nodes, {self._n_edges} edges, {self._n_virtual} virtual", ) async def _close(self) -> None: @@ -140,6 +143,17 @@ class Neo4jFileGraph(BaseFileGraph): return 0, 0, 0 return int(row["real"] or 0), int(row["virtual"] or 0), int(row["edges"] or 0) + async def _refresh_counts(self, session=None) -> None: + """Refresh cached counts for synchronous health reporting.""" + if session is not None: + real, virtual, edges = await self._counts(session) + else: + async with self._session() as new_session: + real, virtual, edges = await self._counts(new_session) + self._n_nodes = real + self._n_virtual = virtual + self._n_edges = edges + # -- Node CRUD --------------------------------------------------------- async def upsert_nodes(self, nodes: list[FileNode]) -> None: @@ -167,6 +181,7 @@ class Neo4jFileGraph(BaseFileGraph): ] async with self._session() as session: await session.execute_write(self._upsert_nodes_tx, payload) + await self._refresh_counts(session) @staticmethod async def _upsert_nodes_tx(tx, payload): @@ -209,6 +224,7 @@ class Neo4jFileGraph(BaseFileGraph): return async with self._session() as session: await session.execute_write(self._delete_nodes_tx, list(paths)) + await self._refresh_counts(session) @staticmethod async def _delete_nodes_tx(tx, paths): @@ -308,6 +324,7 @@ class Neo4jFileGraph(BaseFileGraph): async with self._session() as session: await session.execute_write(self._rebuild_links_tx, payload) + await self._refresh_counts(session) @staticmethod async def _rebuild_links_tx(tx, payload): @@ -333,6 +350,7 @@ class Neo4jFileGraph(BaseFileGraph): """Remove every node and edge in the configured database.""" async with self._session() as session: await session.run("MATCH (f:File) DETACH DELETE f") + await self._refresh_counts(session) # -- Link access ------------------------------------------------------- diff --git a/reme4/components/file_store/faiss_local_file_store.py b/reme4/components/file_store/faiss_local_file_store.py index 32ace19a..32a2821e 100644 --- a/reme4/components/file_store/faiss_local_file_store.py +++ b/reme4/components/file_store/faiss_local_file_store.py @@ -132,9 +132,18 @@ class FaissLocalFileStore(LocalFileStore): id_map = list(data.get("id_map", [])) if len(id_map) != index.ntotal: raise ValueError(f"id_map size {len(id_map)} != index ntotal {index.ntotal}") + tombstones = {int(row) for row in data.get("tombstones", [])} + if any(row < 0 or row >= len(id_map) for row in tombstones): + raise ValueError("FAISS tombstones contain out-of-range rows") + live_ids = [cid for i, cid in enumerate(id_map) if i not in tombstones] + if len(live_ids) != len(set(live_ids)): + raise ValueError("FAISS id_map contains duplicate live chunk ids") + expected_ids = {cid for cid, chunk in self.file_chunks.items() if chunk.embedding is not None} + if set(live_ids) != expected_ids: + raise ValueError("FAISS sidecar live ids do not match persisted chunks") self._faiss_index = index self._id_map = id_map - self._tombstones = set(data.get("tombstones", [])) + self._tombstones = tombstones self._id_to_row = {cid: i for i, cid in enumerate(self._id_map) if i not in self._tombstones} self.logger.info(f"Loaded FAISS index: {index.ntotal} vectors from {self.faiss_path}") return True @@ -175,19 +184,25 @@ class FaissLocalFileStore(LocalFileStore): assert self.file_graph is not None # Snapshot pre-upsert chunk_ids so we can diff against the post-upsert state. - old_ids_by_path = { - n.path: set(n.chunk_ids) for n in await self.file_graph.get_nodes([node.path for node, _ in files]) + old_nodes = await self.file_graph.get_nodes([node.path for node, _ in files]) + old_ids_by_path = {n.path: set(n.chunk_ids) for n in old_nodes} + old_text_by_id = { + cid: chunk.text + for n in old_nodes + for cid in n.chunk_ids + if (chunk := self.file_chunks.get(cid)) is not None } await super().upsert(files) if self._faiss_index is None or self.embedding_store is None: return - self._sync_index_after_upsert(files, old_ids_by_path) + self._sync_index_after_upsert(files, old_ids_by_path, old_text_by_id) def _sync_index_after_upsert( self, files: list[tuple[FileNode, list[FileChunk]]], old_ids_by_path: dict[str, set[str]], + old_text_by_id: dict[str, str], ) -> None: """Apply add/tombstone deltas to FAISS based on chunk_id set differences.""" existing = set(self._id_to_row) @@ -196,9 +211,15 @@ class FaissLocalFileStore(LocalFileStore): new_ids = set(node.chunk_ids) for cid in old_ids_by_path.get(node.path, set()) - new_ids: self._tombstone(cid) - for cid in new_ids - existing: + for cid in new_ids: chunk = self.file_chunks.get(cid) - if chunk is not None and chunk.embedding is not None: + if chunk is None or chunk.embedding is None: + continue + if cid in existing and old_text_by_id.get(cid) == chunk.text: + continue + if cid in existing: + self._tombstone(cid) + if cid not in existing or old_text_by_id.get(cid) != chunk.text: to_add.append(chunk) if to_add: @@ -245,11 +266,20 @@ class FaissLocalFileStore(LocalFileStore): # Over-fetch by len(tombstones) so dropped rows can't starve the result set. q = self._prepare(query_embedding) - k = min(self._faiss_index.ntotal, limit + len(self._tombstones)) + if search_filter: + k = self._faiss_index.ntotal + else: + k = min(self._faiss_index.ntotal, limit + len(self._tombstones)) scores, rows = self._faiss_index.search(q, k) - return self._collect_hits(rows[0].tolist(), scores[0].tolist(), limit) + return self._collect_hits(rows[0].tolist(), scores[0].tolist(), limit, search_filter) - def _collect_hits(self, rows: list[int], scores: list[float], limit: int) -> list[FileChunk]: + def _collect_hits( + self, + rows: list[int], + scores: list[float], + limit: int, + search_filter: dict | None = None, + ) -> list[FileChunk]: """Map raw FAISS rows back to chunks, skipping tombstones and stale ids.""" results: list[FileChunk] = [] for raw_row, score in zip(rows, scores): @@ -257,7 +287,7 @@ class FaissLocalFileStore(LocalFileStore): if row < 0 or row in self._tombstones or row >= len(self._id_map): continue chunk = self.file_chunks.get(self._id_map[row]) - if chunk is None: + if chunk is None or not self._matches_search_filter(chunk, search_filter): continue results.append(chunk.model_copy(update={"scores": {"vector": float(score), "score": float(score)}})) if len(results) >= limit: diff --git a/reme4/components/file_store/local_file_store.py b/reme4/components/file_store/local_file_store.py index 3d653954..f8fa12f3 100644 --- a/reme4/components/file_store/local_file_store.py +++ b/reme4/components/file_store/local_file_store.py @@ -1,6 +1,5 @@ -"""In-memory file store with JSONL persistence on close.""" +"""In-memory file store with compressed JSONL persistence on close.""" -import aiofiles import numpy as np from .base_file_store import BaseFileStore @@ -11,6 +10,9 @@ from ..keyword_index import BaseKeywordIndex from ...enumeration import LinkScopeEnum from ...schema import FileChunk, FileLink, FileNode from ...utils import batch_cosine_similarity +from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst + +CachedEmbedding = tuple[str, np.ndarray] @R.register("local") @@ -49,7 +51,7 @@ class LocalFileStore(BaseFileStore): self.encoding = encoding self.store_version = store_version self.file_chunks: dict[str, FileChunk] = {} - self.chunks_path = self.component_metadata_path / f"file_chunks_{self.name}_{self.store_version}.jsonl" + self.chunks_path = self.component_metadata_path / f"file_chunks_{self.name}_{self.store_version}.jsonl.zst" # -- lifecycle ------------------------------------------------------------ @@ -80,12 +82,11 @@ class LocalFileStore(BaseFileStore): if not self.chunks_path.exists(): return try: - async with aiofiles.open(self.chunks_path, encoding=self.encoding) as f: - async for line in f: - line = line.strip() - if line: - chunk = FileChunk.model_validate_json(line) - self.file_chunks[chunk.id] = chunk + for line in read_jsonl_zst(self.chunks_path, self.encoding): + line = line.strip() + if line: + chunk = FileChunk.model_validate_json(line) + self.file_chunks[chunk.id] = chunk self.logger.info(f"Loaded {len(self.file_chunks)} chunks from {self.chunks_path}") except Exception as e: self.logger.exception(f"Failed to load {self.chunks_path}: {e}") @@ -94,10 +95,7 @@ class LocalFileStore(BaseFileStore): """Atomically rewrite the JSONL, then cascade dump into keyword_index and file_graph.""" assert self.file_graph is not None try: - tmp = self.chunks_path.with_suffix(".tmp") - async with aiofiles.open(tmp, "w", encoding=self.encoding) as f: - await f.write("\n".join(c.model_dump_json() for c in self.file_chunks.values())) - tmp.replace(self.chunks_path) + write_jsonl_zst(self.chunks_path, (c.model_dump_json() for c in self.file_chunks.values()), self.encoding) self.logger.info(f"Saved {len(self.file_chunks)} chunks to {self.chunks_path}") except Exception as e: self.logger.exception(f"Failed to write {self.chunks_path}: {e}") @@ -113,10 +111,13 @@ class LocalFileStore(BaseFileStore): assert self.file_graph is not None old_map = {n.path: n for n in await self.file_graph.get_nodes([node.path for node, _ in files])} + old_chunk_ids = {cid for n in old_map.values() for cid in n.chunk_ids} new_nodes, needs_embed, keyword_docs = self._stage_upsert(files, old_map) await self.file_graph.upsert_nodes(new_nodes) await self._embed_pending(needs_embed) + if self.keyword_index and old_chunk_ids: + await self.keyword_index.delete_docs(list(old_chunk_ids)) if self.keyword_index and keyword_docs: await self.keyword_index.add_docs(keyword_docs) @@ -143,29 +144,29 @@ class LocalFileStore(BaseFileStore): new_nodes.append(node) return new_nodes, needs_embed, keyword_docs - def _evict_prior_chunks(self, old_node: FileNode | None) -> dict[str, np.ndarray]: + def _evict_prior_chunks(self, old_node: FileNode | None) -> dict[str, CachedEmbedding]: """Drop chunks for the path being re-upserted; keep their embeddings around so - a new chunk reusing the same id avoids a redundant embedding call. + a new chunk reusing the same id and text avoids a redundant embedding call. """ - cached: dict[str, np.ndarray] = {} - if not (old_node and self.embedding_store): + cached: dict[str, CachedEmbedding] = {} + if old_node is None: return cached for cid in old_node.chunk_ids: old = self.file_chunks.pop(cid, None) - if old and old.embedding is not None: - cached[cid] = old.embedding + if self.embedding_store and old and old.embedding is not None: + cached[cid] = (old.text, old.embedding) return cached def _reuse_or_queue_embedding( self, chunk: FileChunk, - cached: dict[str, np.ndarray], + cached: dict[str, CachedEmbedding], needs_embed: list[FileChunk], ) -> None: if not self.embedding_store or chunk.embedding is not None: return - if chunk.id in cached: - chunk.embedding = cached[chunk.id] + if chunk.id in cached and cached[chunk.id][0] == chunk.text: + chunk.embedding = cached[chunk.id][1] elif chunk.text: needs_embed.append(chunk) @@ -232,7 +233,11 @@ class LocalFileStore(BaseFileStore): if query_embedding is None: return [] - candidates = [c for c in self.file_chunks.values() if c.embedding is not None] + candidates = [ + c + for c in self.file_chunks.values() + if c.embedding is not None and self._matches_search_filter(c, search_filter) + ] if not candidates: return [] @@ -254,17 +259,70 @@ class LocalFileStore(BaseFileStore): if not query: return [] - doc_id_score_dict = await self.keyword_index.retrieve(query, limit=limit) + retrieve_limit = limit + if search_filter: + retrieve_limit = max(limit, getattr(self.keyword_index, "n_docs", limit)) + doc_id_score_dict = await self.keyword_index.retrieve(query, limit=retrieve_limit) results = [] for doc_id, score in doc_id_score_dict.items(): chunk = self.file_chunks.get(doc_id) - if chunk: + if chunk and self._matches_search_filter(chunk, search_filter): results.append(chunk.model_copy(update={"scores": {"keyword": score, "score": score}})) + if len(results) >= limit: + break return results # -- extensions ----------------------------------------------------------- + @staticmethod + def _as_filter_values(value) -> set: + if isinstance(value, (list, tuple, set, frozenset)): + return set(value) + return {value} + + @classmethod + def _value_matches(cls, actual, expected) -> bool: + if isinstance(expected, (list, tuple, set, frozenset)): + return actual in set(expected) + return actual == expected + + @classmethod + def _matches_search_filter(cls, chunk: FileChunk, search_filter: dict | None) -> bool: + """Conservative post-filter shared by vector and keyword search.""" + if not search_filter: + return True + + exact_paths = set() + for key in ("path", "paths"): + if key in search_filter: + exact_paths.update(cls._as_filter_values(search_filter[key])) + if exact_paths and chunk.path not in exact_paths: + return False + + prefixes = [] + for key in ("path_prefix", "path_prefixes", "prefix", "prefixes"): + if key in search_filter: + prefixes.extend(str(v) for v in cls._as_filter_values(search_filter[key])) + if prefixes and not any(chunk.path.startswith(prefix) for prefix in prefixes): + return False + + metadata_filter = dict(search_filter.get("metadata") or {}) + reserved = { + "path", + "paths", + "path_prefix", + "path_prefixes", + "prefix", + "prefixes", + "metadata", + } + for key, value in search_filter.items(): + if key not in reserved: + metadata_filter[key] = value + + return all(cls._value_matches(chunk.metadata.get(key), value) for key, value in metadata_filter.items()) + async def rebuild_links(self) -> None: """Rebuild graph links via the underlying file graph.""" assert self.file_graph is not None diff --git a/reme4/components/job/base_job.py b/reme4/components/job/base_job.py index b2c935ae..834e0125 100644 --- a/reme4/components/job/base_job.py +++ b/reme4/components/job/base_job.py @@ -59,7 +59,8 @@ class BaseJob(BaseComponent): async def __call__(self, **kwargs) -> Response: """Run all steps in order, capturing any failure into the response.""" - context = RuntimeContext(**kwargs) + merged = {**self.kwargs, **kwargs} + context = RuntimeContext(**merged) try: for step in self._build_steps(): await step(context) diff --git a/reme4/components/job/cron_job.py b/reme4/components/job/cron_job.py index d0306a4c..595653e3 100644 --- a/reme4/components/job/cron_job.py +++ b/reme4/components/job/cron_job.py @@ -1,212 +1,55 @@ -"""``cron`` — a supervised background job that periodically dispatches -downstream job(s) and/or step(s) on a schedule. - -A ``CronJob`` is a :class:`BackgroundJob` whose body is a scheduler loop -rather than a fixed list of steps: it sleeps until the next fire time, -dispatches its configured downstream job(s) and/or step(s), and loops. -It exits when ``self._stop_event`` is set, so the surrounding -``Application`` shutdown / supervisor can stop it cleanly. - -Declared directly under ``jobs:`` in YAML (no step wrapper):: - - auto_dream_cron: - backend: cron - dispatch_job: auto_dream - cron: "0 3 * * *" # daily at 03:00; "0 */6 * * *" = every 6h - run_on_start: false - -Two dispatch modes — pick one (or both, executed in order: jobs first, -then steps): - -* ``dispatch_job`` / ``dispatch_jobs`` — invoke a **registered job** by - name through ``self.app_context.jobs``. Goes through the job's own - parameter validation and context construction, identical to a CLI / - MCP invocation. Prefer this when the periodic task already has a job - wrapping it (e.g. ``auto_dream``). -* ``dispatch_step`` / ``dispatch_steps`` — instantiate a **registered - step** by name and invoke it directly. Use for steps that don't have a - job wrapper. - -Three schedule modes (mutually exclusive — exactly one must be set): - -* ``cron: "M H DoM Mon DoW"`` — standard 5-field cron expression in - ``app_config.timezone``. Most flexible; use for non-daily cadence - (``"0 */6 * * *"`` = every 6 hours, ``"0 3 * * 1-5"`` = 3am on - weekdays, ``"*/15 * * * *"`` = every 15 minutes). -* ``daily_at: "HH:MM"`` — fire once per day at this wall-clock time, in - ``app_config.timezone``. Convenience shorthand for ``"M H * * *"``. -* ``interval_seconds: int`` — fire every N seconds since launch. Useful - for tests and for time-zone-independent sub-minute cadence. - -Per-dispatch exceptions are caught and logged — a failed downstream job -or step never kills the cron loop (so the ``BackgroundJob`` supervisor is -reserved for genuine scheduler-loop crashes). ``run_on_start: true`` -triggers one immediate dispatch on launch (default ``false``, to avoid a -burst when ``Application.start`` brings several cron jobs up at once). - -Cron expressions are evaluated via ``croniter`` and validated eagerly at -construction time, so a typo fails at app start rather than at 3 a.m. -""" +"""Cron-scheduled background job that runs its configured steps.""" import datetime -import zoneinfo - -from croniter import croniter +from zoneinfo import ZoneInfo from .background_job import BackgroundJob from ..component_registry import R -from ...enumeration import ComponentEnum +from ..runtime_context import RuntimeContext from ...schema import Response @R.register("cron") class CronJob(BackgroundJob): - """Dispatch downstream step(s) and/or job(s) on a schedule until ``stop_event`` fires.""" + """Run this job's own steps on a cron expression.""" - def __init__( - self, - dispatch_step: str = "", - dispatch_steps: list[str] | None = None, - dispatch_job: str = "", - dispatch_jobs: list[str] | None = None, - cron: str = "", - daily_at: str = "", - interval_seconds: int = 0, - run_on_start: bool = False, - **kwargs, - ): + def __init__(self, cron: str, **kwargs): super().__init__(**kwargs) - self.dispatch_steps: list[str] = dispatch_steps or ([dispatch_step] if dispatch_step else []) - self.dispatch_jobs: list[str] = dispatch_jobs or ([dispatch_job] if dispatch_job else []) - self.cron: str = cron - self.daily_at: str = daily_at - self.interval_seconds: int = interval_seconds - self.run_on_start: bool = run_on_start + self.cron_expr = cron - if not self.dispatch_steps and not self.dispatch_jobs: - raise ValueError( - "cron job requires at least one of 'dispatch_step'/'dispatch_steps' " - "or 'dispatch_job'/'dispatch_jobs'", - ) + async def _start(self) -> None: + from croniter import croniter - schedules_set = sum(bool(x) for x in (self.cron, self.daily_at, self.interval_seconds)) - if schedules_set != 1: - raise ValueError( - "cron job requires exactly one of " - "'cron' (5-field expression), 'daily_at' (HH:MM), " - "or 'interval_seconds' (int)", - ) - - if self.daily_at: - # Validate HH:MM format eagerly so misconfig fails at start, not at 3am. - h, m = self._parse_hh_mm(self.daily_at) - self._fire_hour, self._fire_minute = h, m - elif self.cron: - # Fail at start, not at the next scheduled tick. - if not croniter.is_valid(self.cron): - raise ValueError(f"cron expression invalid, got {self.cron!r}") - - @staticmethod - def _parse_hh_mm(value: str) -> tuple[int, int]: - try: - h_str, m_str = value.split(":", 1) - h, m = int(h_str), int(m_str) - except (ValueError, AttributeError) as exc: - raise ValueError(f"daily_at must be 'HH:MM', got {value!r}") from exc - if not (0 <= h < 24 and 0 <= m < 60): - raise ValueError(f"daily_at out of range, got {value!r}") - return h, m - - def _tz(self) -> datetime.tzinfo | None: - if not self.app_context: - return None - tz_name = self.app_context.app_config.timezone - if not tz_name: - return None - try: - return zoneinfo.ZoneInfo(tz_name) - except zoneinfo.ZoneInfoNotFoundError: - self.logger.warning(f"[{self.name}] unknown timezone {tz_name!r}; using local time") - return None + if not croniter.is_valid(self.cron_expr): + raise ValueError(f"Invalid cron expression: {self.cron_expr}") + await super()._start() def _next_fire_delay(self) -> float: - """Seconds from now until the next fire — never negative, never zero.""" - if self.interval_seconds: - return float(self.interval_seconds) - tz = self._tz() - now = datetime.datetime.now(tz) - if self.cron: - # croniter requires an explicit base time; tz comes through on `now`. - nxt = croniter(self.cron, now).get_next(datetime.datetime) - return (nxt - now).total_seconds() - # daily_at - target = now.replace(hour=self._fire_hour, minute=self._fire_minute, second=0, microsecond=0) - if target <= now: - target = target + datetime.timedelta(days=1) - return (target - now).total_seconds() + tz_name = None + if self.app_context is not None: + tz_name = self.app_context.app_config.timezone + now = datetime.datetime.now(ZoneInfo(tz_name)) if tz_name else datetime.datetime.now() + from croniter import croniter - async def _fire(self, dispatch_classes: list[type]) -> None: - """Dispatch each downstream job (via the registry) and step (class-level), in that - order; swallow exceptions per-dispatch so a single failure doesn't break the loop.""" - # Jobs first — they're the higher-level invocation path (matches CLI / MCP). - for name in self.dispatch_jobs: - try: - job = self.app_context.jobs.get(name) if self.app_context else None - if job is None: - raise RuntimeError(f"Job {name!r} not found") - await job() - self.logger.info(f"[{self.name}] dispatched job {name!r}") - except Exception as exc: - self.logger.exception(f"[{self.name}] dispatch job {name!r} raised: {exc}") + nxt = croniter(self.cron_expr, now).get_next(datetime.datetime) + return max(0.0, (nxt - now).total_seconds()) - for cls in dispatch_classes: - try: - s = cls(app_context=self.app_context) - await s() - self.logger.info(f"[{self.name}] dispatched step {cls.__name__}") - except Exception as exc: - self.logger.exception(f"[{self.name}] dispatch step {cls.__name__} raised: {exc}") + async def _execute_steps(self) -> Response: + context = RuntimeContext(**self.kwargs) + for step in self._build_steps(): + await step(context) + return context.response async def __call__(self, **kwargs) -> Response: - """Scheduler body: loop dispatching downstream jobs/steps until ``stop_event`` fires. - - Per-dispatch exceptions are swallowed in ``_fire``, so this body only propagates - genuine scheduler-loop failures to the ``BackgroundJob`` supervisor for restart. - """ assert self._stop_event is not None - stop_event = self._stop_event - - dispatch_classes: list[type] = [] - for name in self.dispatch_steps: - cls = R.get(ComponentEnum.STEP, name) - if cls is None: - raise RuntimeError(f"Unregistered step '{name}'") - dispatch_classes.append(cls) - - if self.cron: - mode = f"cron={self.cron!r}" - elif self.daily_at: - mode = f"daily_at={self.daily_at}" - else: - mode = f"interval_seconds={self.interval_seconds}" - self.logger.info( - f"[{self.name}] cron loop start " - f"dispatch_jobs={self.dispatch_jobs} dispatch_steps={self.dispatch_steps} " - f"{mode} run_on_start={self.run_on_start}", - ) - - if self.run_on_start and not stop_event.is_set(): - await self._fire(dispatch_classes) - - while not stop_event.is_set(): - delay = self._next_fire_delay() - self.logger.info(f"[{self.name}] next fire in {delay:.0f}s") - await self._wait_or_stop(delay) - if stop_event.is_set(): + while not self._stop_event.is_set(): + await self._wait_or_stop(self._next_fire_delay()) + if self._stop_event.is_set(): break - await self._fire(dispatch_classes) - + try: + await self._execute_steps() + except Exception as exc: + self.logger.exception(f"Cron job '{self.name}' failed: {exc}") response = Response() response.success = True - response.answer = f"cron job jobs={self.dispatch_jobs!r} steps={self.dispatch_steps!r} stopped" return response diff --git a/reme4/components/job/stream_job.py b/reme4/components/job/stream_job.py index 651d5488..8eb93f92 100644 --- a/reme4/components/job/stream_job.py +++ b/reme4/components/job/stream_job.py @@ -12,7 +12,8 @@ class StreamJob(BaseJob): async def __call__(self, **kwargs) -> None: """Run steps; emit failures as ERROR chunks, then a terminal DONE marker.""" - context = RuntimeContext(**kwargs) + merged = {**self.kwargs, **kwargs} + context = RuntimeContext(**merged) try: for step in self._build_steps(): await step(context) diff --git a/reme4/components/keyword_index/bm25_index.py b/reme4/components/keyword_index/bm25_index.py index 81c5de7a..027ec2ab 100644 --- a/reme4/components/keyword_index/bm25_index.py +++ b/reme4/components/keyword_index/bm25_index.py @@ -15,8 +15,11 @@ lists keep the stale entries until `optimize_index` rewrites them. Updating an existing doc_id retires the old slot first, then allocates a fresh idx. """ +import hashlib +import json import math import pickle +import re from collections import Counter from pathlib import Path @@ -53,11 +56,43 @@ class BM25Index(BaseKeywordIndex): @property def index_file(self) -> Path: - """Path of the persisted index, namespaced by tokenizer and version.""" + """Path of the persisted index, namespaced by component/tokenizer config.""" if self.tokenizer is None: raise RuntimeError("Tokenizer not initialized. Call start() first.") name = type(self.tokenizer).__name__.replace("Tokenizer", "").lower() - return self.component_metadata_path / f"bm25_{name}_{self.index_version}.pkl" + component_name = self._safe_filename_part(self.name) + fingerprint = self._tokenizer_fingerprint() + return self.component_metadata_path / f"bm25_{component_name}_{name}_{fingerprint}_{self.index_version}.pkl" + + @staticmethod + def _safe_filename_part(value: str) -> str: + """Make a short, stable filename segment from a component/config value.""" + safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._") + return safe or "default" + + def _tokenizer_config(self) -> dict: + """Return the tokenizer settings that affect token output.""" + if self.tokenizer is None: + raise RuntimeError("Tokenizer not initialized. Call start() first.") + + config = { + "class": type(self.tokenizer).__qualname__, + "filter_stopwords": getattr(self.tokenizer, "filter_stopwords", None), + } + stopwords_path = getattr(self.tokenizer, "stopwords_path", None) + if stopwords_path is not None: + path = Path(stopwords_path) + config["stopwords_path"] = str(path) + if path.exists() and path.is_file(): + config["stopwords_sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + else: + config["stopwords_sha256"] = None + return config + + def _tokenizer_fingerprint(self) -> str: + """Compact digest for tokenizer settings used in the index filename.""" + payload = json.dumps(self._tokenizer_config(), sort_keys=True, default=str) + return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12] @property def n_docs(self) -> int: @@ -276,6 +311,8 @@ class BM25Index(BaseKeywordIndex): def _snapshot(self) -> dict: """Bundle every persistent field; mirrors `_restore`.""" return { + "tokenizer_config": self._tokenizer_config(), + "tokenizer_fingerprint": self._tokenizer_fingerprint(), "vocab": self.vocab, "doc_ids": self._doc_ids, "doc_id_to_idx": self._doc_id_to_idx, @@ -290,6 +327,10 @@ class BM25Index(BaseKeywordIndex): def _restore(self, data: dict) -> None: """Restore index state from a `_snapshot` dict.""" + expected = self._tokenizer_fingerprint() + actual = data.get("tokenizer_fingerprint") + if actual is not None and actual != expected: + raise ValueError(f"Tokenizer fingerprint mismatch: expected {expected}, got {actual}") self.vocab = data["vocab"] self._doc_ids = data["doc_ids"] self._doc_id_to_idx = data["doc_id_to_idx"] @@ -304,7 +345,11 @@ class BM25Index(BaseKeywordIndex): async def dump(self) -> None: """Persist the index via temp file + atomic rename to avoid torn writes.""" + if self.n_docs == 0 and not self.vocab: + self.index_file.unlink(missing_ok=True) + return try: + self.index_file.parent.mkdir(parents=True, exist_ok=True) tmp = self.index_file.with_suffix(".tmp") with open(tmp, "wb") as f: pickle.dump(self._snapshot(), f) @@ -312,6 +357,7 @@ class BM25Index(BaseKeywordIndex): self.logger.info(f"Saved {self.n_docs} docs to {self.index_file}") except Exception as e: self.logger.exception(f"Failed to write {self.index_file}: {e}") + raise async def load(self) -> None: """Load from disk; missing file is a no-op, corrupt file resets state.""" diff --git a/reme4/components/prompt_handler.py b/reme4/components/prompt_handler.py index bbc4a046..5c1a3a94 100644 --- a/reme4/components/prompt_handler.py +++ b/reme4/components/prompt_handler.py @@ -4,7 +4,6 @@ import inspect import json import re from pathlib import Path -from string import Formatter import yaml @@ -108,11 +107,10 @@ class PromptHandler: # ----- Formatting ---------------------------------------------------- - def prompt_format(self, prompt_name: str, validate: bool = True, **kwargs) -> str: + def prompt_format(self, prompt_name: str, **kwargs) -> str: """Render a prompt: strip inactive ``[flag]`` lines, then ``str.format`` it. Boolean kwargs are treated as flag toggles; the rest are format variables. - With ``validate=True``, any missing ``{var}`` placeholder raises ``ValueError``. """ prompt = self.get_prompt(prompt_name) flags = {k: v for k, v in kwargs.items() if isinstance(v, bool)} @@ -120,8 +118,6 @@ class PromptHandler: if flags: prompt = self._apply_flag_filter(prompt, flags) - if validate: - self._check_required_vars(prompt, formats, prompt_name) return prompt.format(**formats).strip() if formats else prompt @@ -136,14 +132,5 @@ class PromptHandler: lines.append(cleaned) return "\n".join(lines) - @staticmethod - def _check_required_vars(prompt: str, formats: dict, prompt_name: str) -> None: - """Raise when any ``{var}`` placeholder lacks a corresponding kwarg.""" - required = {f for _, f, _, _ in Formatter().parse(prompt) if f is not None} - if missing := required - set(formats.keys()): - raise ValueError( - f"Missing format variables for '{prompt_name}': {sorted(missing)}", - ) - def __repr__(self) -> str: return f"PromptHandler(language='{self.language}', num_prompts={len(self.data)})" diff --git a/reme4/components/service/base_service.py b/reme4/components/service/base_service.py index 2384fe71..d34745b8 100644 --- a/reme4/components/service/base_service.py +++ b/reme4/components/service/base_service.py @@ -32,8 +32,12 @@ class BaseService(BaseComponent): """Instantiate and configure the underlying server framework.""" @abstractmethod - def add_job(self, job: BaseJob) -> None: - """Register a single job as a callable endpoint or tool.""" + def add_job(self, job: BaseJob) -> bool: + """Register a single job as a callable endpoint or tool. + + Returns True when the job is exposed, False when the service intentionally + skips it (for example, unsupported job types). + """ @abstractmethod def start_service(self, app: "Application") -> None: @@ -65,8 +69,10 @@ class BaseService(BaseComponent): if not job.enable_serve: continue try: - self.add_job(job) - self.logger.info(f"Added job: {name}") + if self.add_job(job): + self.logger.info(f"Added job: {name}") + else: + self.logger.warning(f"Skipped job: {name}") except Exception as e: self.logger.error(f"Failed to add job {name}: {e}") diff --git a/reme4/components/service/http_service.py b/reme4/components/service/http_service.py index 11c7e711..d8e010c6 100644 --- a/reme4/components/service/http_service.py +++ b/reme4/components/service/http_service.py @@ -55,12 +55,13 @@ class HttpService(BaseService): allow_headers=["*"], ) - def add_job(self, job: BaseJob) -> None: + def add_job(self, job: BaseJob) -> bool: """Dispatch to streaming or non-streaming registration based on job type.""" if isinstance(job, StreamJob): self._add_stream_job(job) else: self._add_json_job(job) + return True def start_service(self, app: "Application") -> None: """Run uvicorn, suppressing unrelated websocket deprecation noise.""" diff --git a/reme4/components/service/mcp_service.py b/reme4/components/service/mcp_service.py index 99287ccd..7135eb4c 100644 --- a/reme4/components/service/mcp_service.py +++ b/reme4/components/service/mcp_service.py @@ -136,10 +136,10 @@ class MCPService(BaseService): lifespan=self._lifespan(app, self.host, self.port), ) - def add_job(self, job: BaseJob) -> None: + def add_job(self, job: BaseJob) -> bool: """Register a non-stream job as an MCP tool; StreamJobs are unsupported.""" if isinstance(job, StreamJob): - return + return False async def execute_tool(**kwargs): response = await job(**kwargs) @@ -150,9 +150,10 @@ class MCPService(BaseService): name=job.name, description=job.description, fn=execute_tool, - parameters=job.parameters or None, + parameters=job.parameters or {}, ), ) + return True def start_service(self, app: "Application") -> None: """Run the MCP server; bind host/port only for network transports.""" diff --git a/reme4/components/tokenizer/jieba_tokenizer.py b/reme4/components/tokenizer/jieba_tokenizer.py index 5e35fd23..b0008f9f 100644 --- a/reme4/components/tokenizer/jieba_tokenizer.py +++ b/reme4/components/tokenizer/jieba_tokenizer.py @@ -40,4 +40,6 @@ class JiebaTokenizer(BaseTokenizer): self.logger.info(f"JiebaTokenizer using backend: {self.backend}") def _tokenize_one(self, text: str, **kwargs) -> list[str]: + if self._cut is None: + raise RuntimeError("Tokenizer not initialized. Call start() first.") return list(self._cut(text)) diff --git a/reme4/config/config_parser.py b/reme4/config/config_parser.py index 92fce784..76b6c8b5 100644 --- a/reme4/config/config_parser.py +++ b/reme4/config/config_parser.py @@ -32,7 +32,8 @@ def _repl(m: re.Match) -> str: def _expand_env_vars(value: Any) -> Any: """Recursively expand `${VAR}` / `${VAR:-default}` placeholders in strings.""" if isinstance(value, str): - return _ENV_VAR_RE.sub(_repl, value) + expanded = _ENV_VAR_RE.sub(_repl, value) + return _convert_value(expanded) if expanded != value else value if isinstance(value, dict): return {k: _expand_env_vars(v) for k, v in value.items()} if isinstance(value, list): @@ -65,6 +66,8 @@ def parse_dot_notation(dot_list: list[str]) -> dict: raise ValueError(f"Invalid dot notation format (missing '='): {item}") key_path, value_str = item.split("=", 1) keys = key_path.split(".") + if not key_path or any(not key for key in keys): + raise ValueError(f"Invalid dot notation key: {key_path!r}") current = result for key in keys[:-1]: if key in current and not isinstance(current[key], dict): @@ -127,9 +130,13 @@ def _load_config(name_or_path: str, encoding: str = "utf-8") -> dict: # 2. Treat as file path p = Path(name_or_path) if p.suffix in _SUPPORTED_EXTS: - if not p.exists(): - raise FileNotFoundError(f"Config file not found: {p}") - return _read_config_file(p, encoding) + candidates = [p] + if not p.is_absolute(): + candidates.append(_CONFIG_DIR / p) + for candidate in candidates: + if candidate.exists(): + return _read_config_file(candidate, encoding) + raise FileNotFoundError(f"Config file not found: {p}") known = ", ".join(sorted(_CONFIG_REGISTRY)) if _CONFIG_REGISTRY else "none" raise FileNotFoundError(f"Config file not found: {name_or_path}. Available: {known}") @@ -144,6 +151,8 @@ def _read_config_file(path: Path, encoding: str = "utf-8") -> dict: result = yaml.safe_load(f) if result is None: return {} + if not isinstance(result, dict): + raise ValueError(f"Config root must be a mapping/object: {path}") return _expand_env_vars(result) @@ -185,6 +194,8 @@ def parse_args(*args) -> tuple[str, dict]: arg = _strip_arg_dashes(raw) if "=" in arg: kvs.append(arg) + else: + raise ValueError(f"Invalid argument format (expected key=value): {raw}") parsed = parse_dot_notation(kvs) if kvs else {} return first, parsed diff --git a/reme4/config/default.yaml b/reme4/config/default.yaml index 5bf8c019..158acc99 100644 --- a/reme4/config/default.yaml +++ b/reme4/config/default.yaml @@ -1,58 +1,56 @@ -vault_dir: .reme -daily_dir: daily -digest_dir: digest -resource_dir: resource -# language: zh - service: backend: http jobs: index_update_loop: backend: background - watch_dirs: [daily_dir, digest_dir, resource_dir] - watch_suffixes: [md, jsonl] + # watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_dirs: [daily_dir, digest_dir] + # watch_suffixes: [md, jsonl] + watch_suffixes: [md] steps: - - backend: scan_store_changes_step - recursive: true - - backend: update_index_step - persist: true + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] - backend: watch_changes_step - dispatch_step: update_index_step + dispatch_steps: [update_index_step] resource_watch_loop: backend: background watch_dirs: [resource_dir] watch_suffixes: [md, txt, json, jsonl, csv, yaml, html] - dispatch_job: auto_resource - persist: true steps: - - backend: scan_catalog_changes_step - recursive: true - - backend: update_catalog_step - persist: true - - backend: foreach_dispatch_step + - backend: init_changes_step + monitor_type: file_catalog + monitor_name: resource + dispatch_steps: + - backend: update_catalog_step + file_catalog: resource + - backend: auto_resource_step - backend: watch_changes_step - dispatch_steps: [update_catalog_step, foreach_dispatch_step] + dispatch_steps: + - backend: update_catalog_step + file_catalog: resource + - backend: auto_resource_step digest_watch_loop: backend: background watch_dirs: [daily_dir, digest_dir] watch_suffixes: [md] - persist: true steps: - - backend: scan_catalog_changes_step - recursive: true - - backend: update_catalog_step - persist: true + - backend: init_changes_step + monitor_type: file_catalog + monitor_name: digest + dispatch_steps: + - backend: update_catalog_step + file_catalog: digest + - backend: log_changes_step - backend: watch_changes_step - dispatch_step: log_changes_step - - # auto_dream_cron: - # backend: cron - # dispatch_job: auto_dream - # cron: "0 3 * * *" # daily at 03:00; "0 */6 * * *" = every 6h - # run_on_start: false + dispatch_steps: + - backend: update_catalog_step + file_catalog: digest + - backend: log_changes_step version: backend: base @@ -109,13 +107,17 @@ jobs: reindex: backend: base description: "wipe the file store and rebuild it from the existing files" + watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_suffixes: [md, jsonl] parameters: type: object properties: { } steps: - - backend: clear_and_scan_step - - backend: update_index_step - persist: true + - backend: clear_store_step + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] search: backend: base @@ -424,27 +426,9 @@ jobs: steps: - backend: edit_step - dream: - backend: base - description: "Dream: lift atomic units from one daily/resource file into digest/ (LLM)." - parameters: - type: object - properties: - path: - type: string - description: "vault-relative path of one daily-event note or resource file" - hint: - type: string - description: "caller guidance to the dreamer LLM" - default: "" - required: - - path - steps: - - backend: dream_step - auto_dream: backend: base - description: "Auto-dream: scan today's day-index /.md and session notes under //*.md — run dream on each (Phase 1 extract+classify, Phase 2 per-bucket integrate)." + description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog." parameters: type: object properties: @@ -454,10 +438,43 @@ jobs: default: "" hint: type: string - description: "caller guidance passed through to each per-file dream" + description: "caller guidance passed through to dream extract/integrate" default: "" + topic_count: + type: integer + description: "maximum number of final daily interest topics" + default: 3 + topic_diversity_days: + type: integer + description: "number of previous interests.yaml days to avoid repeating" + default: 7 steps: - - backend: auto_dream_step + - backend: dream_extract_step + file_catalog: dream + topic_session_id: interests + - backend: dream_integrate_step + - backend: dream_topics_step + topic_count: 3 + topic_diversity_days: 7 + - backend: dream_finish_step + file_catalog: dream + + proactive: + backend: base + description: "Proactive: read daily//interests.yaml and expose the latest user-interest topics." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD to read; defaults to today in the dreamer's timezone" + default: "" + include_content: + type: boolean + description: "whether to include the raw YAML content in response metadata" + default: true + steps: + - backend: proactive_step auto_memory: backend: base @@ -488,15 +505,21 @@ jobs: parameters: type: object properties: - file_path: - type: string - description: "vault-relative resource file path, e.g. resource/2026-06-06/report.pdf" - change: - type: string - description: "added/modified/deleted" + changes: + type: array + description: "resource change batch, each item has path/file_path and change" + items: + type: object + properties: + path: + type: string + file_path: + type: string + change: + type: string + description: "added/modified/deleted" required: - - file_path - - change + - changes steps: - backend: auto_resource_step @@ -508,10 +531,12 @@ components: as_embedding: default: backend: ${EMBEDDING_BACKEND:-openai} - api_key: ${EMBEDDING_API_KEY:-} - base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} - model_name: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} - dimensions: 1024 + model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} + credential: + api_key: ${EMBEDDING_API_KEY:-} + base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} + parameters: + dimensions: 1024 embedding_store: default: @@ -520,17 +545,17 @@ components: as_llm: default: - backend: ${LLM_BACKEND:-anthropic} - model: ${LLM_MODEL_NAME:-glm-5.1} + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.7-plus} stream: true context_size: 200000 max_retries: 3 credential: api_key: ${LLM_API_KEY:-} - base_url: ${LLM_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic} + base_url: ${LLM_BASE_URL:-} parameters: max_tokens: 65536 - thinking_enable: true + thinking_enable: false agent_wrapper: default: @@ -542,15 +567,15 @@ components: context_config: trigger_ratio: 0.8 reserve_ratio: 0.1 - tool_result_limit: 3000 + tool_result_limit: 50000 model_config: max_retries: 1 claude_code: backend: claude_code - model: ${LLM_MODEL_NAME:-claude-opus-4-6} - system_prompt: "You are a helpful assistant." + model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.1} + api_key: ${CLAUDE_CODE_API_KEY:-} + base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic} permission_mode: bypassPermissions - max_turns: 50 file_graph: default: @@ -559,6 +584,12 @@ components: file_catalog: default: backend: local + resource: + backend: local + digest: + backend: local + dream: + backend: local file_chunker: markdown: diff --git a/reme4/config/demo.yaml b/reme4/config/demo.yaml index f765a194..1f6b2ae9 100644 --- a/reme4/config/demo.yaml +++ b/reme4/config/demo.yaml @@ -21,6 +21,24 @@ jobs: - backend: demo_echo_step1 - backend: demo_echo_step2 + add: + backend: base + description: "add two numbers" + parameters: + type: object + properties: + a: + type: number + description: "first addend" + b: + type: number + description: "second addend" + required: + - a + - b + steps: + - backend: add_step + stream_demo: backend: stream description: "stream demo job: repeat query 10x and stream char-by-char" diff --git a/reme4/config/qwenpaw.yaml b/reme4/config/qwenpaw.yaml index e69de29b..c6a9a685 100644 --- a/reme4/config/qwenpaw.yaml +++ b/reme4/config/qwenpaw.yaml @@ -0,0 +1,553 @@ +service: + backend: http + +jobs: + index_update_loop: + backend: background + # watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_dirs: [daily_dir, digest_dir] + # watch_suffixes: [md, jsonl] + watch_suffixes: [md] + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + - backend: watch_changes_step + dispatch_steps: [update_index_step] + + resource_watch_loop: + backend: background + watch_dirs: [resource_dir] + watch_suffixes: [md, txt, json, jsonl, csv, yaml, html] + steps: + - backend: init_changes_step + monitor_type: file_catalog + monitor_name: resource + dispatch_steps: + - backend: update_catalog_step + file_catalog: resource + - backend: auto_resource_step + - backend: watch_changes_step + dispatch_steps: + - backend: update_catalog_step + file_catalog: resource + - backend: auto_resource_step + + version: + backend: base + description: "return reme package version" + parameters: + type: object + properties: { } + steps: + - backend: version_step + + reindex: + backend: base + description: "wipe the file store and rebuild it from the existing files" + watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_suffixes: [md, jsonl] + parameters: + type: object + properties: { } + steps: + - backend: clear_store_step + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + + search: + backend: base + description: "Hybrid vault search (vector + BM25, RRF-fused)." + parameters: + type: object + properties: + query: + type: string + description: "search query" + limit: + type: integer + description: "max results" + default: 5 + min_score: + type: number + description: "min fused score" + default: 0.0 + required: + - query + steps: + - backend: search_step + vector_weight: 0.7 + candidate_multiplier: 3.0 + expand_links: true + max_links_per_direction: 10 + + node_search: + backend: base + description: "Digest node recall — given a candidate abstraction's name+description, surface existing digest nodes similar enough to either dedup against or link to as related." + parameters: + type: object + properties: + query: + type: string + description: "search query" + limit: + type: integer + description: "max digest nodes to return" + default: 20 + required: + - query + steps: + - backend: node_search_step + vector_weight: 0.7 + candidate_multiplier: 5.0 + + daily_create: + backend: base + description: "Provision a session note under a daily folder: daily//.md or daily/.md" + parameters: + type: object + properties: + session_id: + type: string + description: "the session identifier (also the file stem); empty = day-level file" + default: "" + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_create_step + + daily_list: + backend: base + description: "List notes under a single day." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_list_step + + daily_reindex: + backend: base + description: "Rebuild the day-index page daily/.md." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_reindex_step + + frontmatter_delete: + backend: base + description: "Drop keys from a file's frontmatter." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + keys: + type: array + description: "keys to remove" + items: + type: string + required: + - path + - keys + steps: + - backend: frontmatter_delete_step + + frontmatter_read: + backend: base + description: "Read a file's frontmatter as a dict." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + required: + - path + steps: + - backend: frontmatter_read_step + + frontmatter_update: + backend: base + description: "Merge key-values into a file's frontmatter." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + metadata: + type: object + description: "key-values to merge" + required: + - path + - metadata + steps: + - backend: frontmatter_update_step + + stat: + backend: base + description: "Stat path (size, mtime, exists, is_dir, is_file)." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + required: + - path + steps: + - backend: stat_step + + list: + backend: base + description: "List files under a vault path." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative dir; empty = root" + default: "" + recursive: + type: boolean + description: "recurse" + default: false + limit: + type: integer + description: "max results" + default: 100 + steps: + - backend: list_step + + move: + backend: base + description: "Move / rename a vault file; rewrites inbound wikilinks by default." + parameters: + type: object + properties: + src_path: + type: string + description: "vault-relative source" + dst_path: + type: string + description: "vault-relative destination" + overwrite: + type: boolean + description: "overwrite if dst exists" + default: false + retarget: + type: boolean + description: "rewrite [[src]] → [[dst]] across the vault" + default: true + required: + - src_path + - dst_path + steps: + - backend: move_step + + delete: + backend: base + description: "Delete a vault file or folder; returns surviving inbound wikilinks." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + required: + - path + steps: + - backend: delete_step + + read: + backend: base + description: "Read a markdown file under the vault." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path; markdown only" + start_line: + type: integer + description: "first line (1-based, inclusive)" + end_line: + type: integer + description: "last line (1-based, inclusive)" + required: + - path + steps: + - backend: read_step + with_neighbors: false + max_neighbors_per_direction: 10 + + read_image: + backend: base + description: "Read an image file as base64 (vault-relative path)." + parameters: + type: object + properties: + path: + type: string + description: >- + vault-relative path; common image formats supported + (png/jpg/jpeg/webp/gif/bmp/tiff/heic) + required: + - path + steps: + - backend: read_image_step + max_bytes: 5242880 + + write: + backend: base + description: "Write a markdown file (create or overwrite) with name/description frontmatter." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path; markdown only" + name: + type: string + description: "frontmatter name" + description: + type: string + description: "frontmatter description" + content: + type: string + description: "body" + metadata: + type: object + description: "Optional extra frontmatter fields (md only)." + required: + - path + - name + - description + - content + steps: + - backend: write_step + + edit: + backend: base + description: "Find-and-replace in a markdown file (all occurrences)." + parameters: + type: object + properties: + path: + type: string + description: "vault-relative path" + old: + type: string + description: "text to find" + new: + type: string + description: "replacement" + default: "" + required: + - path + - old + - new + steps: + - backend: edit_step + + auto_dream: + backend: base + description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone" + default: "" + hint: + type: string + description: "caller guidance passed through to dream extract/integrate" + default: "" + topic_count: + type: integer + description: "maximum number of final daily interest topics" + default: 3 + topic_diversity_days: + type: integer + description: "number of previous interests.yaml days to avoid repeating" + default: 7 + steps: + - backend: dream_extract_step + file_catalog: dream + topic_session_id: interests + - backend: dream_integrate_step + - backend: dream_topics_step + topic_count: 3 + topic_diversity_days: 7 + - backend: dream_finish_step + file_catalog: dream + + proactive: + backend: base + description: "Proactive: read daily//interests.yaml and expose the latest user-interest topics." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD to read; defaults to today in the dreamer's timezone" + default: "" + include_content: + type: boolean + description: "whether to include the raw YAML content in response metadata" + default: true + steps: + - backend: proactive_step + + auto_memory: + backend: base + description: "Auto-memory: record conversation facts into a daily note" + parameters: + type: object + properties: + messages: + type: array + description: "messages" + items: + type: object + session_id: + type: string + description: "session identifier passed to daily_create" + default: "" + memory_hint: + type: string + description: "optional hint" + required: + - messages + steps: + - backend: auto_memory_step + + auto_resource: + backend: base + description: "Auto-resource: interpret resource files into daily notes" + parameters: + type: object + properties: + changes: + type: array + description: "resource change batch, each item has path/file_path and change" + items: + type: object + properties: + path: + type: string + file_path: + type: string + change: + type: string + description: "added/modified/deleted" + required: + - changes + steps: + - backend: auto_resource_step + +components: + tokenizer: + default: + backend: regex + + as_embedding: + default: + backend: ${EMBEDDING_BACKEND:-openai} + model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} + credential: + api_key: ${EMBEDDING_API_KEY:-} + base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} + parameters: + dimensions: 1024 + + embedding_store: + default: + backend: local + as_embedding: default + + as_llm: + default: + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.7-plus} + stream: true + context_size: 200000 + max_retries: 3 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false + + agent_wrapper: + default: + backend: agentscope + as_llm: default + permission_mode: bypass + react_config: + max_iters: 30 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + claude_code: + backend: claude_code + model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.1} + api_key: ${CLAUDE_CODE_API_KEY:-} + base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic} + permission_mode: bypassPermissions + + file_graph: + default: + backend: local + + file_catalog: + default: + backend: local + resource: + backend: local + digest: + backend: local + dream: + backend: local + + file_chunker: + markdown: + backend: markdown + supported_extensions: [ "md" ] + default: + backend: default + supported_extensions: [ "jsonl" ] + + keyword_index: + default: + backend: bm25 + tokenizer: default + + file_store: + default: + backend: local + store_name: local + # embedding_store: default + embedding_store: "" + keyword_index: default + file_graph: default diff --git a/reme4/enumeration/chunk_enum.py b/reme4/enumeration/chunk_enum.py index d397be9e..32678bd5 100644 --- a/reme4/enumeration/chunk_enum.py +++ b/reme4/enumeration/chunk_enum.py @@ -4,18 +4,46 @@ from enum import Enum class ChunkEnum(str, Enum): - """Enumeration of possible chunk categories for stream processing.""" + """Enumeration of possible chunk categories for stream processing. + Covers both AgentScope and Claude Code SDK streaming protocols: + + AgentScope events -> ChunkEnum mapping: + ReplyStartEvent -> REPLY_START + ReplyEndEvent -> REPLY_END + TextBlockStart/Delta/End -> CONTENT + ThinkingBlockStart/Delta/End -> THINK + DataBlockStart/Delta/End -> DATA + ToolCallStart/Delta/End -> TOOL_CALL + ToolResultStart/TextDelta/DataDelta/End -> TOOL_RESULT + ModelCallEndEvent -> USAGE + ExceedMaxItersEvent -> ERROR + + Claude Code SDK events -> ChunkEnum mapping: + message_start -> REPLY_START + message_stop -> REPLY_END + content_block_start/delta/stop (text) -> CONTENT + content_block_start/delta/stop (thinking) -> THINK + content_block_start/delta/stop (tool_use) -> TOOL_CALL + ToolResultBlock -> TOOL_RESULT + ResultMessage -> USAGE + DONE + ResultMessage.is_error -> ERROR + """ + + # Lifecycle markers + REPLY_START = "reply_start" + REPLY_END = "reply_end" + + # Content types THINK = "think" - CONTENT = "content" + DATA = "data" + # Tool interaction TOOL_CALL = "tool_call" - TOOL_RESULT = "tool_result" + # Metadata & terminal USAGE = "usage" - ERROR = "error" - DONE = "done" diff --git a/reme4/pyproject.toml b/reme4/pyproject.toml index ce0e8d09..cbc803b3 100644 --- a/reme4/pyproject.toml +++ b/reme4/pyproject.toml @@ -35,11 +35,12 @@ dependencies = [ "rich>=14.3.3", "uvicorn>=0.41.0", "watchfiles>=1.1.1", + "zstandard>=0.23.0", ] [project.optional-dependencies] core = [ - "agentscope>=2.0.0", + "agentscope>=2.0.2", "faiss-cpu>=1.13.2", "jieba>=0.42.1", "rjieba>=0.2.1", @@ -70,6 +71,7 @@ include-package-data = true [tool.setuptools.package-data] "*" = ["py.typed", "**/*.yaml", "**/*.json"] +"reme4.components.tokenizer" = ["stopwords"] [tool.setuptools.dynamic] version = { attr = "reme4.__version__" } diff --git a/reme4/reme.py b/reme4/reme.py index 19598a90..1a27d9f3 100644 --- a/reme4/reme.py +++ b/reme4/reme.py @@ -9,6 +9,8 @@ from .config import parse_args, resolve_app_config from .enumeration import ComponentEnum from .utils import cli_find_reme, load_env, precheck_start +_CLIENT_KWARGS = {"host", "port", "timeout", "transport", "command", "args"} + class ReMe(Application): """ReMe memory management application.""" @@ -17,10 +19,11 @@ class ReMe(Application): async def call_server(action: str, **kwargs): """Call the appropriate server component.""" backend: str = kwargs.pop("backend", "http") + client_kwargs = {key: kwargs.pop(key) for key in list(kwargs) if key in _CLIENT_KWARGS} client_cls = R.get(ComponentEnum.CLIENT, backend) if client_cls is None: raise ValueError(f"Unknown client backend: {backend!r}") - async with client_cls() as client: + async with client_cls(**client_kwargs) as client: async for chunk in client(action=action, **kwargs): print(chunk, end="", flush=True) print() diff --git a/reme4/schema/application_config.py b/reme4/schema/application_config.py index 5875dcb6..42766812 100644 --- a/reme4/schema/application_config.py +++ b/reme4/schema/application_config.py @@ -30,6 +30,7 @@ class ApplicationConfig(BaseModel): app_name: str = Field(default=os.getenv("APP_NAME", "ReMe"), description="Application display name") vault_dir: str = Field(default=".reme", description="Vault root directory for runtime files") metadata_dir: str = Field(default="reme_metadata", description="Subdirectory for ReMe persistent state") + session_dir: str = Field(default="reme_session", description="Subdirectory for persisted agent sessions") resource_dir: str = Field(default="resource", description="Subdirectory for external assets") daily_dir: str = Field(default="daily", description="Subdirectory for daily memory") digest_dir: str = Field(default="digest", description="Subdirectory for digest memory") diff --git a/reme4/schema/file_chunk.py b/reme4/schema/file_chunk.py index c02cc98f..bd34f96b 100644 --- a/reme4/schema/file_chunk.py +++ b/reme4/schema/file_chunk.py @@ -9,8 +9,8 @@ class FileChunk(EmbNode): """A chunk of a file with positional info and per-stage retrieval scores.""" path: str = Field(default="", description="Path relative to the vault") - start_line: int = Field(default=0, description="Inclusive start line (0-based)") - end_line: int = Field(default=0, description="Exclusive end line") + start_line: int = Field(default=0, description="Inclusive start line (1-based)") + end_line: int = Field(default=0, description="Inclusive end line (1-based)") scores: dict[str, float] = Field(default_factory=dict, description="Retrieval scores keyed by stage") @property diff --git a/reme4/schema/stream_chunk.py b/reme4/schema/stream_chunk.py index b2fb0aef..3d006ecb 100644 --- a/reme4/schema/stream_chunk.py +++ b/reme4/schema/stream_chunk.py @@ -1,14 +1,49 @@ """Stream chunk schema for incremental responses (e.g. LLM streaming).""" +from typing import Any + from pydantic import BaseModel, Field from ..enumeration import ChunkEnum class StreamChunk(BaseModel): - """A single chunk in a streaming response sequence.""" + """A single chunk in a unified streaming response sequence. + + Carries full information from both AgentScope and Claude Code SDK + backends. Optional fields are ``None`` by default so that simple + text-only streams (e.g. plain CONTENT deltas) stay lightweight. + + Fields: + chunk_type: Category of this chunk (see ChunkEnum). + chunk: Payload, typically a text delta, but can be + dict/list for structured data (tool-call JSON, + usage stats, etc.). + done: Terminal marker. True only for the final DONE chunk. + session_id: Session identifier (AgentScope: agent.state.session_id; + Claude Code: ResultMessage.session_id). + block_id: Content-block identifier for matching start/delta/end + sequences (both backends assign block IDs). + tool_call_id: Tool-call identifier for correlating call deltas + with their start event and result. + tool_call_name: Name of the tool being invoked. + media_type: MIME type for DATA blocks (e.g. ``"image/png"``). + input_tokens: Prompt tokens consumed (populated on USAGE chunks). + output_tokens: Completion tokens generated (populated on USAGE chunks). + metadata: Backend-specific extras that don't warrant a dedicated + field. + """ chunk_type: ChunkEnum = Field(default=ChunkEnum.CONTENT, description="Type of chunk content") chunk: str | dict | list = Field(default="", description="Chunk payload") done: bool = Field(default=False, description="Whether this is the final chunk") - metadata: dict = Field(default_factory=dict, description="Chunk metadata") + + session_id: str | None = Field(default=None, description="Session identifier") + block_id: str | None = Field(default=None, description="Content block identifier") + tool_call_id: str | None = Field(default=None, description="Tool call identifier") + tool_call_name: str | None = Field(default=None, description="Tool call name") + media_type: str | None = Field(default=None, description="MIME type for data blocks") + input_tokens: int | None = Field(default=None, description="Prompt tokens consumed") + output_tokens: int | None = Field(default=None, description="Completion tokens generated") + + metadata: dict[str, Any] = Field(default_factory=dict, description="Chunk metadata") diff --git a/reme4/steps/__init__.py b/reme4/steps/__init__.py index ec586895..a4575ea4 100644 --- a/reme4/steps/__init__.py +++ b/reme4/steps/__init__.py @@ -1,90 +1,14 @@ """steps""" +from . import channel, common, evolve, file_io, index, transfer from .base_step import BaseStep -from .common.demo import DemoEchoStep1, DemoEchoStep2 -from .common.health_check import HealthCheckStep -from .common.help import HelpStep -from .common.llm_demo import LLMDemoStep -from .common.stream_demo import StreamDemoStep1, StreamDemoStep2 -from .common.version import VersionStep -from .evolve.auto_dream import AutoDreamStep -from .evolve.auto_memory import AutoMemoryStep -from .evolve.auto_resource import AutoResourceStep -from .evolve.dream import DreamStep -from .file_io.daily_create import DailyCreateStep -from .file_io.daily_list import DailyListStep -from .file_io.daily_reindex import DailyReindexStep -from .file_io.delete import DeleteStep -from .file_io.edit import EditStep -from .file_io.frontmatter_delete import FrontmatterDeleteStep -from .file_io.frontmatter_read import FrontmatterReadStep -from .file_io.frontmatter_update import FrontmatterUpdateStep -from .file_io.list import ListStep -from .file_io.move import MoveStep -from .file_io.read import ReadStep -from .file_io.read_image import ReadImageStep -from .file_io.stat import StatStep -from .file_io.write import WriteStep -from .channel.channel_notify import ChannelNotifyStep -from .channel.claim_channel import ClaimChannelStep -from .index.clear_and_scan import ClearAndScanStep -from .index.node_search import NodeSearchStep -from .index.scan_changes import ScanCatalogChangesStep, ScanStoreChangesStep -from .index.search import SearchStep -from .index.traverse import TraverseStep -from .index.update_catalog import UpdateCatalogStep -from .index.update_index import UpdateIndexStep -from .index.foreach_dispatch import ForeachDispatchStep -from .index.log_changes import LogChangesStep -from .index.watch_changes import WatchChangesStep __all__ = [ "BaseStep", - # common - "DemoEchoStep1", - "DemoEchoStep2", - "HealthCheckStep", - "HelpStep", - "LLMDemoStep", - "StreamDemoStep1", - "StreamDemoStep2", - "VersionStep", - # evolve - "AutoMemoryStep", - "AutoResourceStep", - # file_io - "DeleteStep", - "EditStep", - "ListStep", - "MoveStep", - "ReadStep", - "ReadImageStep", - "StatStep", - "WriteStep", - # file_io (daily) - "DailyCreateStep", - "DailyListStep", - "DailyReindexStep", - # file_io.frontmatter - "FrontmatterDeleteStep", - "FrontmatterReadStep", - "FrontmatterUpdateStep", - # channel - "ChannelNotifyStep", - "ClaimChannelStep", - # index - "ClearAndScanStep", - "ForeachDispatchStep", - "LogChangesStep", - "NodeSearchStep", - "ScanCatalogChangesStep", - "ScanStoreChangesStep", - "SearchStep", - "TraverseStep", - "UpdateCatalogStep", - "UpdateIndexStep", - "WatchChangesStep", - # evolve (dream) - "AutoDreamStep", - "DreamStep", + "channel", + "common", + "evolve", + "file_io", + "index", + "transfer", ] diff --git a/reme4/steps/base_step.py b/reme4/steps/base_step.py index 4eb7854d..d5fdc345 100644 --- a/reme4/steps/base_step.py +++ b/reme4/steps/base_step.py @@ -2,26 +2,26 @@ import copy from abc import abstractmethod, ABC -from typing import TypeVar, TYPE_CHECKING +from typing import Any, TYPE_CHECKING from agentscope.model import ChatModelBase from ..components.agent_wrapper.base_agent_wrapper import BaseAgentWrapper from ..components.base_component import ComponentMixin +from ..components.component_registry import R from ..components.file_catalog import BaseFileCatalog from ..components.file_store import BaseFileStore from ..components.prompt_handler import PromptHandler from ..components.runtime_context import RuntimeContext from ..enumeration import ComponentEnum -from ..schema import Response +from ..schema import ApplicationConfig, Response if TYPE_CHECKING: from ..components import ApplicationContext from ..components.job import BaseJob -T = TypeVar("T") - _UNSET = object() +_DispatchStep = str | dict[str, Any] class Ref: @@ -41,14 +41,7 @@ class Ref: __slots__ = ("base_cls", "comp_enum", "attr", "optional", "key", "_cache_attr") - def __init__( - self, - base_cls: type, - comp_enum: ComponentEnum, - attr: str | None = None, - *, - optional: bool = False, - ) -> None: + def __init__(self, base_cls: type, comp_enum: ComponentEnum, attr: str | None = None, *, optional: bool = False): self.base_cls = base_cls self.comp_enum = comp_enum self.attr = attr @@ -108,8 +101,7 @@ class BaseStep(ComponentMixin, ABC): def __new__(cls, *args, **kwargs): # Snapshot init args so copy() can rebuild an equivalent instance later. instance = object.__new__(cls) - instance._init_args = copy.copy(args) - instance._init_kwargs = copy.copy(kwargs) + instance._init_args, instance._init_kwargs = copy.copy(args), copy.copy(kwargs) return instance def __init__( @@ -121,14 +113,14 @@ class BaseStep(ComponentMixin, ABC): prompt_dict: dict[str, str] | None = None, input_mapping: dict[str, str] | None = None, output_mapping: dict[str, str] | None = None, + dispatch_steps: list[_DispatchStep] | None = None, **kwargs, ): super().__init__(name=name, backend=backend, app_context=app_context, **kwargs) - self.language: str = language - if not self.language and self.app_context is not None: - self.language = self.app_context.app_config.language + self.language = language or (self.app_context.app_config.language if self.app_context is not None else "") self.input_mapping = input_mapping self.output_mapping = output_mapping + self.dispatch_step_specs = list(dispatch_steps or []) self.context: RuntimeContext | None = None # Load class-level prompts first, then overlay caller-provided overrides. @@ -165,6 +157,13 @@ class BaseStep(ComponentMixin, ABC): """Return a named prompt template as-is.""" return self.prompt.get_prompt(prompt_name=prompt_name) + def config_value(self, key: str): + """Return an app config value, falling back to ApplicationConfig defaults.""" + defaults = ApplicationConfig() + cfg = self.app_context.app_config if self.app_context is not None else defaults + value = getattr(cfg, key) + return getattr(defaults, key) if value in (None, "") else value + def copy(self, **kwargs) -> "BaseStep": """Construct a new instance from the original init args, applying overrides.""" return self.__class__(*self._init_args, **{**self._init_kwargs, **kwargs}) @@ -181,3 +180,36 @@ class BaseStep(ComponentMixin, ABC): if job is None: raise RuntimeError(f"Job {name} not found") return await job(**kwargs) + + def _resolve_dispatch_step(self, raw: _DispatchStep): + """Resolve a dispatch step spec to (step class, init params).""" + if isinstance(raw, str): + params: dict[str, Any] = {"backend": raw} + elif isinstance(raw, dict): + params = dict(raw) + else: + raise TypeError(f"Invalid dispatch step spec: {raw!r}") + + backend = params.get("backend", "") + if not backend: + raise ValueError("Dispatch step is missing the required 'backend' field") + step_cls = R.get(ComponentEnum.STEP, backend) + if step_cls is None: + raise RuntimeError(f"Unregistered step '{backend}'") + params["app_context"] = self.app_context + return step_cls, params + + async def dispatch_steps(self, dispatch_steps: list[_DispatchStep], **kwargs) -> list[Response]: + """Run dispatch steps against the current context. + + Callers pass producer-specific values, usually ``changes=...``. Existing + context data is preserved for downstream handlers. + """ + if self.context is None: + raise RuntimeError("Cannot dispatch steps without a runtime context") + + responses: list[Response] = [] + for raw in dispatch_steps: + step_cls, params = self._resolve_dispatch_step(raw) + responses.append(await step_cls(**params)(self.context, **kwargs)) + return responses diff --git a/reme4/steps/channel/__init__.py b/reme4/steps/channel/__init__.py index e69de29b..51062e80 100644 --- a/reme4/steps/channel/__init__.py +++ b/reme4/steps/channel/__init__.py @@ -0,0 +1,9 @@ +"""Channel steps.""" + +from .channel_notify import ChannelNotifyStep +from .claim_channel import ClaimChannelStep + +__all__ = [ + "ChannelNotifyStep", + "ClaimChannelStep", +] diff --git a/reme4/steps/channel/channel_notify.py b/reme4/steps/channel/channel_notify.py index 099c8d56..fe2c1413 100644 --- a/reme4/steps/channel/channel_notify.py +++ b/reme4/steps/channel/channel_notify.py @@ -1,6 +1,6 @@ """``channel_notify_step`` — push a debounced batch of vault changes as one channel event. -Designed to be slotted into ``watch_changes_step.dispatch_step`` next to +Designed to be slotted into ``watch_changes_step.dispatch_steps`` next to ``update_index_step``: when the watcher emits a batch of changes, this step forwards a single human-readable summary to ``ApplicationContext.metadata["channel_sink"]``. The Claude Code main @@ -34,11 +34,11 @@ class ChannelNotifyStep(BaseStep): async def execute(self): sink = self.app_context.metadata.get("channel_sink") if self.app_context is not None else None if sink is None: - return + return self.context.response if self.context is not None else None changes = (self.context.get("changes", []) if self.context is not None else []) or [] if not changes: - return + return self.context.response if self.context is not None else None # Render paths vault-relative so the agent can pass them directly to # slash commands like /dream . Absolute paths that fall outside @@ -58,10 +58,11 @@ class ChannelNotifyStep(BaseStep): lines.append(f"{change.get('change', '?')}: {shown}") if not lines: - return + return self.context.response if self.context is not None else None self.logger.info(f"[channel_notify] emit batch count={len(lines)}") await sink.emit( content="Vault 变更:\n" + "\n".join(lines), meta={"kind": "vault_change", "count": str(len(lines))}, ) + return self.context.response if self.context is not None else None diff --git a/reme4/steps/channel/claim_channel.py b/reme4/steps/channel/claim_channel.py index 756b0afa..65edbb5d 100644 --- a/reme4/steps/channel/claim_channel.py +++ b/reme4/steps/channel/claim_channel.py @@ -25,24 +25,28 @@ class ClaimChannelStep(BaseStep): """Bind the current MCP session as the ```` recipient.""" async def execute(self): - assert self.context is not None + if self.context is None: + raise RuntimeError("claim_channel_step requires 'context'") try: from fastmcp.server.dependencies import get_context ctx = get_context() session = ctx.session - assert session is not None, "FastMCP context has no ServerSession" - assert self.app_context is not None, "claim_channel requires an application context" + if session is None: + raise RuntimeError("FastMCP context has no ServerSession") + if self.app_context is None: + raise RuntimeError("claim_channel requires an application context") sink = self.app_context.metadata.get("channel_sink") - assert sink is not None, "channel_sink not configured on application context metadata" + if sink is None: + raise RuntimeError("channel_sink not configured on application context metadata") + session_id = ctx.session_id or "" + sink.bind(session) except Exception as e: self.context.response.answer = {"claimed": False, "reason": f"{type(e).__name__}: {e}"} self.context.response.metadata["claimed"] = False return self.context.response - sink.bind(session) - session_id = ctx.session_id or "" self.logger.info(f"[claim_channel] channel bound to session={session_id}") self.context.response.answer = { "claimed": True, diff --git a/reme4/steps/common/__init__.py b/reme4/steps/common/__init__.py index e69de29b..5d938939 100644 --- a/reme4/steps/common/__init__.py +++ b/reme4/steps/common/__init__.py @@ -0,0 +1,23 @@ +"""common steps""" + +from .add import AddStep +from .demo import DemoEchoStep1, DemoEchoStep2 +from .health_check import HealthCheckStep +from .help import HelpStep +from .llm_demo import LLMDemoStep +from .stream_demo import StreamDemoStep1, StreamDemoStep2 +from .stream_llm_demo import StreamLLMDemoStep +from .version import VersionStep + +__all__ = [ + "AddStep", + "DemoEchoStep1", + "DemoEchoStep2", + "HealthCheckStep", + "HelpStep", + "LLMDemoStep", + "StreamDemoStep1", + "StreamDemoStep2", + "StreamLLMDemoStep", + "VersionStep", +] diff --git a/reme4/steps/common/add.py b/reme4/steps/common/add.py new file mode 100644 index 00000000..7f34cd5a --- /dev/null +++ b/reme4/steps/common/add.py @@ -0,0 +1,35 @@ +"""Simple step that adds two numbers — used as a demo tool for agent wrapper.""" + +from ..base_step import BaseStep +from ...components import R + + +@R.register("add_step") +class AddStep(BaseStep): + """Add two numbers and return the result as the response answer. + + Inputs (from RuntimeContext): + a (float, required): first addend. + b (float, required): second addend. + + Output (written to context.response.answer): + The sum of ``a`` and ``b`` as a string. + """ + + async def execute(self): + assert self.context is not None + try: + a = float(self.context.get("a", 0.0)) + b = float(self.context.get("b", 0.0)) + except (TypeError, ValueError) as exc: + self.context.response.success = False + self.context.response.answer = f"Invalid add arguments: {exc}" + return self.context.response + + result = a + b + self.logger.info(f"[{self.name}] add({a}, {b}) = {result}") + + self.context.response.success = True + self.context.response.answer = str(result) + self.context.response.metadata.update({"a": a, "b": b, "result": result}) + return self.context.response diff --git a/reme4/steps/common/health_check.py b/reme4/steps/common/health_check.py index cb83e234..6ef1f1ad 100644 --- a/reme4/steps/common/health_check.py +++ b/reme4/steps/common/health_check.py @@ -57,6 +57,7 @@ def _mb_str(*objs) -> str: def _embedding_status(comp) -> dict: cache = getattr(comp, "_embedding_cache", {}) or {} + model = getattr(comp, "model", None) try: dims = comp.dimensions except Exception: @@ -64,7 +65,7 @@ def _embedding_status(comp) -> dict: return { "is_started": comp.is_started, "is_healthy": getattr(comp, "is_healthy", None), - "model_name": getattr(comp, "model_name", None), + "model_name": getattr(model, "model", None), "dimensions": dims, "cache_size": len(cache), "memory": _mb_str(cache), @@ -97,9 +98,30 @@ def _file_graph_local_status(comp) -> dict: } +def _file_graph_neo4j_status(comp) -> dict: + """Neo4j backend: counts are cached on the async component for sync health checks.""" + return { + "is_started": comp.is_started, + "n_nodes": getattr(comp, "_n_nodes", 0), + "n_edges": getattr(comp, "_n_edges", 0), + "n_virtual": getattr(comp, "_n_virtual", 0), + "memory": _mb_str( + getattr(comp, "_uri", ""), + getattr(comp, "_database", ""), + getattr(comp, "_n_nodes", 0), + getattr(comp, "_n_edges", 0), + getattr(comp, "_n_virtual", 0), + ), + } + + def _file_graph_status(comp) -> dict: graph = getattr(comp, "_graph", None) - return _file_graph_nx_status(comp, graph) if graph is not None else _file_graph_local_status(comp) + if graph is not None: + return _file_graph_nx_status(comp, graph) + if hasattr(comp, "_driver"): + return _file_graph_neo4j_status(comp) + return _file_graph_local_status(comp) def _file_store_status(comp) -> dict: diff --git a/reme4/steps/common/llm_demo.py b/reme4/steps/common/llm_demo.py index 4dd0ac9a..a5f2e6bf 100644 --- a/reme4/steps/common/llm_demo.py +++ b/reme4/steps/common/llm_demo.py @@ -2,23 +2,12 @@ from typing import Type -from agentscope.tool import FunctionTool, Toolkit from pydantic import BaseModel from ..base_step import BaseStep from ...components import R -def add(a: float, b: float) -> str: - """Add two numbers and return the sum. - - Args: - a: first addend - b: second addend - """ - return str(a + b) - - @R.register("llm_demo_step") class LLMDemoStep(BaseStep): """Drive an Agent powered by the ``agent_wrapper`` component. @@ -26,7 +15,6 @@ class LLMDemoStep(BaseStep): Inputs (from RuntimeContext): query (str, required): user message content. sys_prompt (str, optional): system prompt for the agent. - use_add_tool (bool, optional): register the ``add`` tool when True. Output (written to context.response.answer): The agent's final reply text. @@ -38,7 +26,6 @@ class LLMDemoStep(BaseStep): assert self.context is not None query: str = self.context.get("query", "") sys_prompt: str = self.context.get("sys_prompt") or self.DEFAULT_SYS_PROMPT - use_add_tool: bool = bool(self.context.get("use_add_tool", False)) structured_model: Type[BaseModel] | None = self.context.get("structured_model") if not query: @@ -46,25 +33,17 @@ class LLMDemoStep(BaseStep): self.context.response.answer = "Skipped: empty query" return self.context.response - toolkit = Toolkit(tools=[FunctionTool(add)]) if use_add_tool else Toolkit() - wrapper_kwargs = { "system_prompt": sys_prompt, - "toolkit": toolkit, + "job_tools": ["add"], } if structured_model is not None: wrapper_kwargs["output_schema"] = structured_model - _, result = await self.agent_wrapper.reply(query, **wrapper_kwargs) + result = await self.agent_wrapper.reply(query, **wrapper_kwargs) - structured_content: dict | None = None - if isinstance(result, dict) and "message" in result: - msg = result["message"] - structured_content = result["structured_output"] - else: - msg = result - - text = (msg.get_text_content() or "").strip() + structured_content = result.get("structured_output") + text = (result.get("result") or "").strip() self.logger.info(f"[{self.name}] response: {text!r}") self.context.response.success = True @@ -73,7 +52,6 @@ class LLMDemoStep(BaseStep): { "query": query, "sys_prompt": sys_prompt, - "use_add_tool": use_add_tool, "response": text, "structured_output": structured_content, }, diff --git a/reme4/steps/common/stream_llm_demo.py b/reme4/steps/common/stream_llm_demo.py index 4eb8ca53..4fca8e71 100644 --- a/reme4/steps/common/stream_llm_demo.py +++ b/reme4/steps/common/stream_llm_demo.py @@ -1,34 +1,10 @@ """Demo step that drives an Agent via the agent_wrapper component with streaming output.""" -import json - -from agentscope.event import ( - TextBlockDeltaEvent, - ThinkingBlockDeltaEvent, - ToolCallStartEvent, - ToolCallDeltaEvent, - ToolResultTextDeltaEvent, - ModelCallEndEvent, - ReplyStartEvent, -) -from agentscope.message import Msg -from agentscope.tool import FunctionTool, Toolkit - from ..base_step import BaseStep from ...components import R from ...enumeration import ChunkEnum -def add(a: float, b: float) -> str: - """Add two numbers and return the sum. - - Args: - a: first addend - b: second addend - """ - return str(a + b) - - @R.register("stream_llm_demo_step") class StreamLLMDemoStep(BaseStep): """Drive an Agent powered by the ``agent_wrapper`` component with streaming output. @@ -40,7 +16,6 @@ class StreamLLMDemoStep(BaseStep): Inputs (from RuntimeContext): query (str, required): user message content. sys_prompt (str, optional): system prompt for the agent. - use_add_tool (bool, optional): register the ``add`` tool when True. Output (written to context.response.answer): The agent's final reply text. @@ -52,25 +27,22 @@ class StreamLLMDemoStep(BaseStep): assert self.context is not None query: str = self.context.get("query", "") sys_prompt: str = self.context.get("sys_prompt") or self.DEFAULT_SYS_PROMPT - use_add_tool: bool = bool(self.context.get("use_add_tool", False)) if not query: self.context.response.success = False self.context.response.answer = "Skipped: empty query" return self.context.response - toolkit = Toolkit(tools=[FunctionTool(add)]) if use_add_tool else Toolkit() - wrapper_kwargs = { "system_prompt": sys_prompt, - "toolkit": toolkit, + "job_tools": ["add"], } if self.context.stream: text = await self._stream_reply(query, **wrapper_kwargs) else: - _, msg = await self.agent_wrapper.reply(query, **wrapper_kwargs) - text = (msg.get_text_content() or "").strip() + result = await self.agent_wrapper.reply(query, **wrapper_kwargs) + text = (result.get("result") or "").strip() self.logger.debug(f"[{self.name}] response: {text!r}") @@ -80,45 +52,23 @@ class StreamLLMDemoStep(BaseStep): { "query": query, "sys_prompt": sys_prompt, - "use_add_tool": use_add_tool, "response": text, }, ) return self.context.response async def _stream_reply(self, query: str, **wrapper_kwargs) -> str: - """Stream agent reply events to the context stream queue.""" + """Stream unified chunks to the context stream queue.""" assert self.context is not None - reply_msg: Msg | None = None + text_parts: list[str] = [] - async for event in self.agent_wrapper.reply_stream(query, **wrapper_kwargs): - if isinstance(event, ReplyStartEvent): - reply_msg = Msg( - id=event.reply_id, - name=event.name, - role=event.role, - content=[], - ) - elif isinstance(event, TextBlockDeltaEvent): - await self.context.add_stream_string(event.delta, ChunkEnum.CONTENT) - elif isinstance(event, ThinkingBlockDeltaEvent): - await self.context.add_stream_string(event.delta, ChunkEnum.THINK) - elif isinstance(event, ToolCallStartEvent): - payload = json.dumps({"name": event.tool_call_name, "id": event.tool_call_id}) - await self.context.add_stream_string(payload, ChunkEnum.TOOL_CALL) - elif isinstance(event, ToolCallDeltaEvent): - await self.context.add_stream_string(event.delta, ChunkEnum.TOOL_CALL) - elif isinstance(event, ToolResultTextDeltaEvent): - await self.context.add_stream_string(event.delta, ChunkEnum.TOOL_RESULT) - elif isinstance(event, ModelCallEndEvent): - usage = json.dumps( - {"input_tokens": event.input_tokens, "output_tokens": event.output_tokens}, - ) - await self.context.add_stream_string(usage, ChunkEnum.USAGE) + async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs): + await self.context.add_stream_string(chunk.chunk, chunk.chunk_type) - if reply_msg is not None: - reply_msg.append_event(event) + if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str): + text_parts.append(chunk.chunk) - if reply_msg is not None: - return (reply_msg.get_text_content() or "").strip() - return "" + if chunk.session_id: + self.context.response.metadata["session_id"] = chunk.session_id + + return "".join(text_parts).strip() diff --git a/reme4/steps/evolve/__init__.py b/reme4/steps/evolve/__init__.py index 9895edab..c5f658a3 100644 --- a/reme4/steps/evolve/__init__.py +++ b/reme4/steps/evolve/__init__.py @@ -1,5 +1,17 @@ """Evolve steps.""" from ._evolve import now +from .auto_memory import AutoMemoryStep +from .auto_resource import AutoResourceStep +from .dream import DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep -__all__ = ["now"] +__all__ = [ + "now", + "AutoMemoryStep", + "AutoResourceStep", + "DreamExtractStep", + "DreamFinishStep", + "DreamIntegrateStep", + "DreamTopicsStep", + "ProactiveStep", +] diff --git a/reme4/steps/evolve/auto_dream.py b/reme4/steps/evolve/auto_dream.py deleted file mode 100644 index e9505667..00000000 --- a/reme4/steps/evolve/auto_dream.py +++ /dev/null @@ -1,286 +0,0 @@ -"""AutoDreamStep — daily-tick wrapper that dispatches per-file to the ``dream`` job. - -Each tick scans today's two surfaces under ``/``: - -* ``/.md`` — the day-index file (auto-rebuilt rollup - of today's notes); included first so day-level abstractions land - before per-event details. -* ``//**/*.md`` — event notes for the day. - -The diff vs ``file_catalog`` follows the same shape as -:class:`ScanCatalogChangesStep`: build ``existing`` (on-disk ``rel → mtime``) -and ``indexed`` (catalog ``rel → mtime``, restricted to today's -prefix so we never disturb entries from other days), then: - -* ``existing`` keys not in ``indexed`` → **added**, dream -* mtime mismatch → **modified**, dream -* ``indexed`` keys not in ``existing`` → **deleted**, drop from catalog -* mtime match → **unchanged**, skip - -For every to-dream file, the step calls the configured ``dispatch_job`` -(default ``"dream"``) via :meth:`BaseStep.run_job`. The dispatch job's -``Response`` carries a ``DreamResult`` payload in ``metadata``; -AutoDream re-hydrates that for its per-file aggregate. - -After dreaming, successful (and Phase 1 vacuously-skipped) files -upsert their current ``st_mtime`` so the next tick re-dreams only -what actually changed. Failures leave the catalog untouched and will -be retried on the next tick. - -The two writers (``WatchDreamStep`` event-driven, ``AutoDreamStep`` -catch-up scan) share the protocol: upsert ``(path, st_mtime)`` on -success. Last-writer-wins is fine — same path + same content yields -the same mtime, so they cannot disagree on what's "done". - -Cron scheduling itself is out of scope; this step is the unit of -work — invoke it from a system cron, ``reme auto-dream date=...``, -or any other catch-up trigger when ``auto_dream_loop`` missed a file -(e.g. process crashed before the watcher fired). - -**Backend-agnostic.** Because dispatch goes through the configured -``dream`` job, the per-file dream implementation is decided by the -YAML config — whichever step the ``dream`` job's ``backend`` resolves -to. AutoDream itself doesn't care which backend runs underneath. - -Inputs (RuntimeContext): - date (str, optional): YYYY-MM-DD to scan. Defaults to today - in the dreamer's timezone. - hint (str, optional): passed through to each per-file dream. - -Step kwargs (from yaml ``backend: auto_dream_step``): - dispatch_job (str, default "dream"): name of the job to call - per file. Override only if the deployment renamed the dream - job. The dispatched job must accept ``path`` and ``hint`` - kwargs and return a ``Response`` whose ``metadata`` matches - :class:`DreamResult`. - persist (bool, default True): when True, ``file_catalog.dump()`` - is called after the batch so progress survives a restart. -""" - -from pathlib import Path - -from pydantic import BaseModel, Field - -from ._evolve import now -from .dream import DreamResult -from ..base_step import BaseStep -from ...components import R -from ...schema import FileNode - - -class AutoDreamResult(BaseModel): - """Aggregated outcome of one auto-dream tick.""" - - date: str = "" - files_scanned: int = 0 - files_unchanged: int = 0 - files_dreamed: int = 0 - files_skipped: int = 0 - files_failed: int = 0 - files_deleted: int = 0 - per_file: list[DreamResult] = Field(default_factory=list) - summary: str = "" - - -@R.register("auto_dream_step") -class AutoDreamStep(BaseStep): - """Scan ``daily/.md`` + ``daily//`` and dispatch to the - configured ``dream`` job for each file whose ``st_mtime`` doesn't - already match its ``file_catalog`` entry.""" - - def __init__(self, dispatch_job: str = "dream", persist: bool = True, **kwargs): - super().__init__(**kwargs) - self.dispatch_job: str = dispatch_job - self.persist: bool = persist - - def _vault_dir(self) -> Path: - """Vault root as an absolute path (mirrors :meth:`DreamStep._vault_dir`).""" - vr = getattr(self.file_store, "vault_path", None) - return Path(vr).resolve() if vr else Path.cwd().resolve() - - async def _dispatch_dream(self, rel_path: str, hint: str) -> DreamResult: - """Call the configured ``dispatch_job`` once and re-hydrate its - ``Response.metadata`` into a :class:`DreamResult`. - - On dispatch failure (job raises) returns a ``DreamResult`` with - ``error`` populated so the caller's accounting stays uniform. - """ - try: - resp = await self.run_job(self.dispatch_job, path=rel_path, hint=hint) - except Exception as e: # pylint: disable=broad-except - self.logger.error( - f"[{self.name}] dispatch {self.dispatch_job!r} failed on {rel_path}: " f"{type(e).__name__}: {e}", - ) - return DreamResult(path=rel_path, error=f"{type(e).__name__}: {e}") - - # The dream job's execute() does context.response.metadata.update(result.model_dump()), - # so metadata carries every DreamResult field. extra keys are - # ignored by pydantic v2 default (extra='ignore'). - md = dict(resp.metadata or {}) - try: - dr = DreamResult.model_validate(md) - except Exception as e: # noqa: BLE001 - self.logger.error( - f"[{self.name}] dispatch {self.dispatch_job!r} returned non-DreamResult metadata " - f"for {rel_path}: {type(e).__name__}: {e}", - ) - return DreamResult(path=rel_path, error=f"bad dispatch metadata: {type(e).__name__}: {e}") - - # If the underlying step set success=False, treat as failure even - # if metadata didn't carry an error string (defensive). - if not resp.success and not dr.error: - dr.error = resp.answer or "dispatch returned success=False" - if not dr.path: - dr.path = rel_path - return dr - - async def execute(self): - assert self.context is not None - date_input: str = (self.context.get("date", "") or "").strip() - hint: str = (self.context.get("hint", "") or "").strip() - - # daily_dir comes from app config — NOT a tool param. Same convention - # as daily_create / daily_list / daily_reindex. - cfg = self.app_context.app_config if self.app_context is not None else None - daily_dir = (cfg.daily_dir if cfg else "") or "daily" - - tz = self.app_context.app_config.timezone if self.app_context is not None else None - today = date_input or now(tz).strftime("%Y-%m-%d") - vault = self._vault_dir() - files = _scan_today_files(vault, today, daily_dir) - - # existing: today's on-disk paths → st_mtime. Insertion order = scan - # order (date.md first, then sorted event notes); preserved through - # the diff so the day-index file is dreamed before per-event notes. - existing: dict[str, float] = {} - for rel in files: - try: - existing[rel] = (vault / rel).stat().st_mtime - except OSError as e: - self.logger.error(f"[{self.name}] stat failed on {rel}: {e}") - - # indexed: catalog entries restricted to today's prefix. Restriction - # is critical — get_nodes() returns all days, but we must only - # consider deletions within today's scan scope. - today_md = f"{daily_dir}/{today}.md" - today_dir = f"{daily_dir}/{today}/" - all_nodes = await self.file_catalog.get_nodes() - indexed: dict[str, float] = { - n.path: n.st_mtime for n in all_nodes if n.path == today_md or n.path.startswith(today_dir) - } - - # Diff — same vocabulary as scan_*_changes_step (added/modified/deleted). - to_dream: list[tuple[str, float]] = [(rel, mt) for rel, mt in existing.items() if indexed.get(rel) != mt] - unchanged: list[str] = [rel for rel, mt in existing.items() if indexed.get(rel) == mt] - to_delete: list[str] = sorted(indexed.keys() - existing.keys()) - - result = AutoDreamResult( - date=today, - files_scanned=len(existing), - files_unchanged=len(unchanged), - files_deleted=len(to_delete), - ) - self.logger.info( - f"[{self.name}] auto-dream tick date={today} scanned={len(existing)} " - f"unchanged={len(unchanged)} todo={len(to_dream)} deleted={len(to_delete)} " - f"under {daily_dir}/{today}{{.md,/}} dispatch={self.dispatch_job!r}", - ) - - # Drop catalog entries for files no longer on disk first. Cheap, no - # LLM, and keeps the catalog consistent even if the dream pass below - # errors. - if to_delete: - try: - await self.file_catalog.delete(to_delete) - except Exception as e: # noqa: BLE001 - self.logger.exception( - f"[{self.name}] file_catalog.delete failed: {type(e).__name__}: {e}", - ) - - # Dispatch + upsert per-file. Single-file granularity means a job - # failure on file N doesn't block files N+1..K from advancing their - # catalog mtime. The dispatch decouples backend choice (AS / CC) - # from this loop — the configured ``dream`` job picks the runner. - upsert_nodes: list[FileNode] = [] - for rel_path, mtime in to_dream: - dr = await self._dispatch_dream(rel_path, hint) - result.per_file.append(dr) - if dr.error: - # Failures leave the catalog untouched — next tick retries. - result.files_failed += 1 - continue - if dr.skipped: - # Phase 1 said "nothing to extract" — still mark as seen so - # Phase 1 doesn't re-run on every tick. - result.files_skipped += 1 - else: - result.files_dreamed += 1 - upsert_nodes.append(FileNode(path=rel_path, st_mtime=mtime)) - - if upsert_nodes: - try: - await self.file_catalog.upsert(upsert_nodes) - except Exception as e: # noqa: BLE001 - self.logger.exception( - f"[{self.name}] file_catalog.upsert failed: {type(e).__name__}: {e}", - ) - - if self.persist and (upsert_nodes or to_delete): - try: - await self.file_catalog.dump() - except Exception as e: # noqa: BLE001 - self.logger.exception( - f"[{self.name}] file_catalog.dump failed: {type(e).__name__}: {e}", - ) - - result.summary = _render_auto_dream_summary(result) - self.context.response.success = result.files_failed == 0 - self.context.response.answer = result.summary - self.context.response.metadata.update(result.model_dump()) - return self.context.response - - -def _scan_today_files(vault: Path, today: str, daily_dir: str) -> list[str]: - """Return vault-relative paths of today's day-index + event notes. - - * ``/.md`` — the day-index file (auto-rebuilt - rollup of all of today's notes). Included first so its day-level - abstractions land before the per-event details. - * ``//**/*.md`` — event notes for the day, - sorted by path for deterministic processing order. - """ - out: list[str] = [] - if not daily_dir: - return out - - day_index = vault / daily_dir / f"{today}.md" - if day_index.is_file(): - out.append(str(day_index.relative_to(vault))) - - daily_root = vault / daily_dir / today - if daily_root.is_dir(): - for md in sorted(daily_root.rglob("*.md")): - if md.is_file(): - out.append(str(md.relative_to(vault))) - - return out - - -def _render_auto_dream_summary(r: AutoDreamResult) -> str: - """One-line header + one line per dreamed file with its outcome. - Unchanged + deleted files are counted in the header only.""" - lines = [ - f"[AutoDreamStep] date={r.date} scanned={r.files_scanned} " - f"unchanged={r.files_unchanged} dreamed={r.files_dreamed} " - f"skipped={r.files_skipped} failed={r.files_failed} " - f"deleted={r.files_deleted}", - ] - for dr in r.per_file: - if dr.error: - status = f"ERROR ({dr.error})" - elif dr.skipped: - status = "SKIP" - else: - status = f"OK (+{len(dr.nodes_created)} created, ~{len(dr.nodes_updated)} updated)" - lines.append(f" - {dr.path}: {status}") - return "\n".join(lines) diff --git a/reme4/steps/evolve/auto_memory.py b/reme4/steps/evolve/auto_memory.py index f3d4b5e6..43e46aa2 100644 --- a/reme4/steps/evolve/auto_memory.py +++ b/reme4/steps/evolve/auto_memory.py @@ -7,10 +7,12 @@ from agentscope.message import Msg from ._evolve import format_history, now from ..base_step import BaseStep +from ..file_io import refresh_day_index, validate_session_id from ...components import R _TOOL_OUTPUT_MAX = 2048 _TOOL_OUTPUT_HALF = 1024 +_SOURCE_CONVERSATION_KEY = "source_conversation" def _truncate_text(text: str) -> str: @@ -73,17 +75,20 @@ class AutoMemoryStep(BaseStep): super().__init__(**kwargs) self.agent_tools: list[str] = ["read", "edit", "frontmatter_update", "write"] - def _session_path(self, session_id: str, tz: str | None) -> Path: - current = now(tz) - date_str = current.strftime("%Y-%m-%d") - resource = self.app_context.app_config.resource_dir if self.app_context else "resource" - return self.file_store.vault_path / resource / date_str / f"session_agent_{session_id}.jsonl" + def _session_dir(self) -> str: + return str(self.config_value("session_dir")).strip("/") - async def _save_session_messages(self, session_id: str, messages: list[Msg], tz: str | None) -> None: + def _session_path(self, session_id: str) -> Path: + return self.file_store.vault_path / self._session_dir() / "dialog" / f"{session_id}.jsonl" + + def _session_link(self, session_id: str) -> str: + return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]" + + async def _save_session_messages(self, session_id: str, messages: list[Msg]) -> None: if not session_id or not messages: return - path = self._session_path(session_id, tz) + path = self._session_path(session_id) existing: list[Msg] = [] if path.exists(): @@ -139,7 +144,12 @@ class AutoMemoryStep(BaseStep): messages: list[Msg] = [self._to_msg(item) for item in raw_messages] - await self._save_session_messages(session_id, messages, tz) + if session_id and (err := validate_session_id(session_id)): + self.context.response.success = False + self.context.response.answer = f"Error: {err}" + return + + await self._save_session_messages(session_id, messages) if not messages: self.context.response.success = True @@ -169,16 +179,44 @@ class AutoMemoryStep(BaseStep): history=format_history(messages), ) - tools = [self.get_job(name) for name in self.agent_tools] - _, msg = await self.agent_wrapper.reply( + result = await self.agent_wrapper.reply( user_message, system_prompt=self.prompt_format("system_prompt"), - tools=tools, + job_tools=self.agent_tools, ) + source_conversation = "" + if session_id: + source_conversation = self._session_link(session_id) + link_response = await self.run_job( + "frontmatter_update", + path=note_path, + metadata={_SOURCE_CONVERSATION_KEY: source_conversation}, + ) + if not link_response.success: + self.context.response.success = False + self.context.response.answer = f"frontmatter_update failed: {link_response.answer}" + self.context.response.metadata.update( + {"path": note_path, "created": created, "n_messages": len(messages), "index": None}, + ) + self.logger.info( + f"[{self.name}] source conversation link failed " + f"path={note_path} session_id={session_id!r} answer={link_response.answer!r}", + ) + return + + daily_dir = self.config_value("daily_dir") + index_payload = await refresh_day_index(self.file_store, create_response.metadata["date"], daily_dir) + self.context.response.success = True - self.context.response.answer = (msg.get_text_content() or "").strip() + self.context.response.answer = (result.get("result") or "").strip() self.context.response.metadata.update( - {"path": note_path, "created": created, "n_messages": len(messages)}, + { + "path": note_path, + "created": created, + "n_messages": len(messages), + "source_conversation": source_conversation, + "index": index_payload, + }, ) self.logger.info(f"[{self.name}] done {note_path}") diff --git a/reme4/steps/evolve/auto_resource.py b/reme4/steps/evolve/auto_resource.py index 9f5936f1..87f63a4c 100644 --- a/reme4/steps/evolve/auto_resource.py +++ b/reme4/steps/evolve/auto_resource.py @@ -1,19 +1,24 @@ -"""auto_resource — interpret resource files into daily notes via an agent.""" +"""auto_resource — interpret resource files into same-name daily notes via an agent.""" -import hashlib -from pathlib import PurePosixPath +import uuid +from pathlib import Path, PurePosixPath import aiofiles from watchfiles import Change from ..base_step import BaseStep +from ..file_io import refresh_day_index from ...components import R -def _compute_session_id(filename: str) -> str: - """Return 'resource_' + first 8 hex chars of MD5(filename).""" - digest = hashlib.md5(filename.encode()).hexdigest()[:8] - return f"resource_{digest}" +def _compute_agent_session_id(path: str) -> str: + """Return a stable UUID session id for agent backends.""" + return str(uuid.uuid5(uuid.NAMESPACE_URL, path)) + + +def _compute_note_stem(filename: str) -> str: + """Return the daily note stem for a resource filename.""" + return PurePosixPath(filename).stem def _parse_resource_path(file_path: str, resource_dir: str) -> tuple[str, str]: @@ -47,9 +52,9 @@ class AutoResourceStep(BaseStep): return Change.__members__.get(raw) return None - async def _handle_delete(self, date_str: str, session_id: str) -> None: - daily_dir = self.app_context.app_config.daily_dir if self.app_context else "daily" - note_rel = f"{daily_dir}/{date_str}/session_agent_{session_id}.md" + async def _handle_delete(self, date_str: str, note_stem: str) -> None: + daily_dir = self.config_value("daily_dir") + note_rel = f"{daily_dir}/{date_str}/{note_stem}.md" note_abs = self.vault_path / note_rel if note_abs.is_file(): @@ -57,13 +62,16 @@ class AutoResourceStep(BaseStep): self.logger.info(f"[{self.name}] Deleted file: {note_rel}") await self.file_store.delete([note_rel]) + index_payload = await refresh_day_index(self.file_store, date_str, daily_dir) self.context.response.success = True self.context.response.answer = f"Deleted resource note: {note_rel}" - self.context.response.metadata.update({"path": note_rel, "session_id": session_id, "action": "deleted"}) + self.context.response.metadata.update( + {"path": note_rel, "session_id": note_stem, "action": "deleted", "index": index_payload}, + ) - async def _handle_upsert(self, file_path: str, date_str: str, session_id: str, created: bool) -> None: - create_response = await self.run_job("daily_create", session_id=session_id, date=date_str) + async def _handle_upsert(self, file_path: str, date_str: str, note_stem: str, created: bool) -> None: + create_response = await self.run_job("daily_create", session_id=note_stem, date=date_str) if not create_response.success: self.context.response.success = False self.context.response.answer = f"daily_create failed: {create_response.answer}" @@ -92,59 +100,88 @@ class AutoResourceStep(BaseStep): date=date_str, ) - tools = [self.get_job(name) for name in self.agent_tools] - _, msg = await self.agent_wrapper.reply( + agent_session_id = _compute_agent_session_id(file_path) + result = await self.agent_wrapper.reply( user_message, system_prompt=self.prompt_format("system_prompt"), - tools=tools, - session_id=session_id, + job_tools=self.agent_tools, + session_id=agent_session_id, ) + daily_dir = self.config_value("daily_dir") + index_payload = await refresh_day_index(self.file_store, date_str, daily_dir) self.context.response.success = True - self.context.response.answer = (msg.get_text_content() or "").strip() + self.context.response.answer = (result.get("result") or "").strip() self.context.response.metadata.update( { "path": note_path, "created": note_created, - "session_id": session_id, + "session_id": note_stem, + "agent_session_id": agent_session_id, "action": "added" if created else "modified", + "index": index_payload, }, ) self.logger.info(f"[{self.name}] done {note_path}") - async def execute(self): + async def _handle_change(self, file_path: str, raw_change) -> dict: assert self.context is not None - file_path: str = self.context.get("file_path", "") - raw_change = self.context.get("change", "") - + file_path = self.to_vault_relative(file_path) if file_path and Path(file_path).is_absolute() else file_path if not file_path: self.context.response.success = False self.context.response.answer = "Missing file_path" - return + return {"success": False, "path": file_path, "change": raw_change, "answer": self.context.response.answer} change = self._normalize_change(raw_change) if change is None: self.context.response.success = False self.context.response.answer = f"Invalid change type: {raw_change}" - return + return {"success": False, "path": file_path, "change": raw_change, "answer": self.context.response.answer} - resource_dir = self.app_context.app_config.resource_dir if self.app_context else "resource" + resource_dir = self.config_value("resource_dir") date_str, filename = _parse_resource_path(file_path, resource_dir) if not date_str or not filename: self.context.response.success = False self.context.response.answer = f"Cannot parse date/filename from: {file_path}" - return + return {"success": False, "path": file_path, "change": change.name, "answer": self.context.response.answer} - session_id = _compute_session_id(filename) - self.logger.info(f"[{self.name}] {change.name} file_path={file_path} session_id={session_id}") + note_stem = _compute_note_stem(filename) + self.logger.info(f"[{self.name}] {change.name} file_path={file_path} note_stem={note_stem}") if change == Change.deleted: - await self._handle_delete(date_str, session_id) + await self._handle_delete(date_str, note_stem) else: await self._handle_upsert( file_path, date_str, - session_id, + note_stem, created=change == Change.added, ) + return { + "success": self.context.response.success, + "path": file_path, + "change": change.name, + "answer": self.context.response.answer, + "metadata": dict(self.context.response.metadata), + } + + async def execute(self): + assert self.context is not None + changes = self.context.get("changes") + if not isinstance(changes, list): + self.context.response.success = False + self.context.response.answer = "AutoResourceStep requires changes: list[dict]" + return self.context.response + + results = [ + await self._handle_change(item.get("path") or item.get("file_path", ""), item.get("change", "")) + for item in changes + if isinstance(item, dict) + ] + success_count = sum(1 for item in results if item.get("success")) + self.context.response.success = success_count == len(changes) + self.context.response.answer = f"Processed {success_count}/{len(changes)} resource change(s)" + self.context.response.metadata["processed"] = len(results) + self.context.response.metadata["results"] = results + return self.context.response diff --git a/reme4/steps/evolve/dream.py b/reme4/steps/evolve/dream.py deleted file mode 100644 index b302234a..00000000 --- a/reme4/steps/evolve/dream.py +++ /dev/null @@ -1,432 +0,0 @@ -"""DreamStep — single-file digest step (auto-dream's create_or_update primitive). - -Reads one daily-event note or resource file at the given vault-relative -``path``, identifies the ABSTRACTIONS the material teaches in Phase 1 -(each tagged with one of the three buckets), then in Phase 2 makes -ONE cognitive write decision (CREATE or one of the three UPDATE -flavors: CORROBORATE / REFINE / CORRECT) per abstraction using a -**bucket-specific** integrate prompt. - -**Digest is the abstract memory layer** — raw details stay in the -material; digest holds the principle, pattern, or precedent worth -recalling once the specifics fade. Provenance wikilinks -(``derived_from::``) let readers drill back down to the source. - -Pipeline (external loop in Python, two distinct ReAct agent invocations, -**light Phase 1 / heavy Phase 2**): - - execute(): - units, _ = _extract(material_blob) # 1× ReAct: identify abstractions - # agent emits ExtractedUnits - # ({units: [{name, bucket, summary}, ...]}) - for unit in units: # Python loop, K iterations - _integrate_unit(unit) # 1× ReAct per abstraction, dispatched - # to integrate_system_prompt_; - # recalls cross-bucket, decides write, - # uses canonical write/edit/frontmatter_update tools. - -The bucket vocabulary is hard-coded (:data:`BUCKETS`) — three buckets, -each with a dedicated Phase 2 prompt: - -* ``procedure`` — how-to-do-X: steps, methods, recipes, workflows. -* ``personal`` — user/team specific: identity, preferences, - conventions, things they avoid. -* ``wiki`` — general knowledge: definitions, principles, - observations, decisions-as-precedent. Default catch-all. - -There is no SKIP outcome in Phase 2: Phase 1 is the gate for "not -worth memorizing"; anything reaching Phase 2 warrants a write. - -Phase 2 uses the **canonical** ``write`` / ``edit`` jobs (no -constrained variants). Bucket placement and edge conservation are -prompt-level discipline; the tools themselves perform no path-shape -or conservation validation. - -Invocation form (CLI / MCP): - reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md - reme dream path=resource/2026-05-28/spec.pdf hint="focus on auth" -""" - -from pathlib import Path -from typing import Literal - -from pydantic import BaseModel, Field - -from ._evolve import now -from ..base_step import BaseStep -from ...components import R - - -# Hard-coded bucket vocabulary. Phase 1 classifies each sub-unit into -# one of these; Phase 2 dispatches to the bucket-specific prompt. -# Order matters for prompt rendering — keep procedure/personal/wiki. -BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki") - -# Bucket = Literal of BUCKETS. Pydantic Literal must be a static type; -# update both BUCKETS and Bucket together if the vocabulary changes. -Bucket = Literal["procedure", "personal", "wiki"] - - -_EXTRACT_TOOLS: tuple[str, ...] = ("read",) - -_INTEGRATE_TOOLS: tuple[str, ...] = ( - # read — dream uses its own node-level digest search (NOT the - # general chunk-level `search`), specialized for dedup + synapse - # recall. See reme4/steps/index/node_search.py for the rationale. - # NO traverse here: traverse is a retrieve-time subgraph mining - # tool (used by external retrieval agents); dream is a write-time - # candidate recall operation, structurally a different problem. - "node_search", - "read", - "frontmatter_read", - # write - "write", - "edit", - "frontmatter_update", -) - - -# ============================================================ -# Schema (Pydantic models for structured ReAct output) + helpers -# ============================================================ - - -def _pack_material(file_store, path: str) -> str: - """Render one daily-event note or resource file into a prompt block.""" - try: - absolute = (Path(file_store.vault_path or ".") / path).resolve() - except Exception as e: - return f"### {path}\n(error resolving path: {type(e).__name__}: {e})\n" - - if not absolute.is_file(): - return f"### {path}\n(file not found)\n" - - try: - return f"### {path}\n{absolute.read_text(encoding='utf-8')}\n" - except Exception as e: - return f"### {path}\n(error reading: {type(e).__name__}: {e})\n" - - -class MemoryUnit(BaseModel): - """One memory sub-unit identified by Phase 1's structured output.""" - - name: str = Field( - description=( - "Short kebab-case identifier for the abstraction " - "(e.g. 'jwt-rotation-decision', 'pr-size-pref'). " - "Agent-internal handle — NOT the eventual digest slug; " - "Phase 2 picks the actual filing path." - ), - ) - bucket: Bucket = Field( - description=( - "Which bucket this abstraction belongs in — Phase 2 dispatches " - "to a bucket-specific prompt based on this. Pick exactly one: " - "`procedure` (how-to-do-X — steps, methods, recipes, workflows), " - "`personal` (user/team-specific — identity, preferences, " - "conventions, things they avoid), `wiki` (general knowledge — " - "definitions, principles, observations, decisions-as-precedent; " - "default catch-all when nothing else fits)." - ), - ) - summary: str = Field( - description=( - "1-2 sentences naming the abstraction AND pointing at where " - "in the material the supporting evidence lives " - "(e.g. 'short-credential compliance drives auth cadence; " - "illustrated by the 30→24h decision in the 'Decision' section " - "+ the SOC2 CC6.1 criticism in the 'Observation' section')." - ), - ) - - -class ExtractedUnits(BaseModel): - """Structured output emitted by Phase 1's extract agent.""" - - units: list[MemoryUnit] = Field( - default_factory=list, - description=( - "Memory sub-units identified in the material — orthogonal " - "abstractions (principles / patterns / precedents) worth " - "lifting into long-term memory. Each is tagged with its " - "bucket. Empty list = nothing worth lifting (Phase 2 is skipped)." - ), - ) - - -def _render_outcome_line(unit_name: str, bucket: str, o: "IntegrateOutcome") -> str: - """Format one IntegrateOutcome as a one-line summary entry.""" - body = f"{o.action} {o.target_path}" - if o.note: - body += f" — {o.note}" - return f"[{unit_name}/{bucket}] {body}" - - -class IntegrateOutcome(BaseModel): - """Structured outcome reported by Phase 2 for one sub-unit.""" - - action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field( - description=( - "Outcome of the write decision for this sub-unit. Phase 1 already " - "filtered out non-abstractions, so every sub-unit reaching you " - "warrants a write — pick the matching fine-grained action: " - "`CREATE` — brand-new digest node (recall returned no node " - "covering this abstraction); even thin first-encounter seeds go " - "here, they grow via CORROBORATE / REFINE on later passes. " - "`CORROBORATE` (most common when a covering node exists) — " - "provenance append + optional wording strengthening; the " - "abstraction already covers this material. `REFINE` — covering " - "node exists but the material reveals nuance, scope, or edge " - "cases the abstraction under-specified. `CORRECT` — covering " - "node exists but the material contradicts it; tighten the " - "abstraction or annotate the contradiction inline." - ), - ) - target_path: str = Field( - description=("The digest path you wrote to — must match what your `write` / " "`edit` call(s) targeted."), - ) - note: str = Field( - default="", - description=( - "Optional ONE short line, ≤ 200 chars, no newlines, summarizing " - "what landed (e.g. 'extended scope to also cover X'). Do NOT " - "dump recall summaries, search results, internal reasoning, or " - "transcripts here — those belong in the ReAct trace, not the " - "outcome note." - ), - ) - - -class DreamResult(BaseModel): - """Outcome of one DreamStep invocation. - - Per-tool audit lives in the toolkit layer (not exposed back to the - orchestrator). Structured outcome here is the input path the call - processed, the memory sub-units the agent declared in Phase 1, and - what got created / updated in Phase 2. - """ - - used_llm: bool = False - skipped: bool = False - path: str = "" - units: list[dict] = Field(default_factory=list) - nodes_created: list[str] = Field(default_factory=list) - nodes_updated: list[str] = Field(default_factory=list) - summary: str = "" - error: str = "" - - -# ============================================================ -# DreamStep — the per-file create_or_update step. -# ============================================================ - - -@R.register("dream_step") -class DreamStep(BaseStep): - """auto-dream create_or_update step — one file per call. - - Inputs (from RuntimeContext): - path (str, required): vault-relative path of one - daily-event note or resource file to dream over. Pass - empty string to no-op. - hint (str, optional): caller guidance to the LLM - (e.g. "focus on the auth-related decisions"). - - Output (written to context.response.answer): - ``DreamResult`` JSON in ``metadata``; LLM summary in ``answer``. - - CLI / MCP form: - reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md - """ - - def _vault_dir(self) -> Path: - vr = getattr(self.file_store, "vault_path", None) - return Path(vr).resolve() if vr else Path.cwd().resolve() - - def _llm_available(self) -> bool: - try: - return self.as_llm is not None - except Exception: - return False - - async def _extract(self, material_blob: str, hint: str, vault_dir: Path) -> tuple[list[dict], str]: - """Phase 1: one ReAct invocation — read material + emit ExtractedUnits. - - Returns ``(units, llm_summary)`` where ``units`` is the cleaned - sub-unit list (each entry has ``name`` / ``bucket`` / ``summary``) - and ``llm_summary`` is whatever free-form text the agent produced - alongside its structured emission. - """ - tools = [self.get_job(name) for name in _EXTRACT_TOOLS] - tz = self.app_context.app_config.timezone if self.app_context is not None else None - user_message = self.prompt_format( - "extract_user_message", - today=now(tz).strftime("%Y-%m-%d"), - hint=hint or "(none)", - material_blob=material_blob, - ) - _, result = await self.agent_wrapper.reply( - user_message, - system_prompt=self.prompt_format( - "extract_system_prompt", - vault_dir=str(vault_dir), - buckets=", ".join(BUCKETS), - ), - tools=tools, - output_schema=ExtractedUnits, - ) - - msg = result["message"] - meta = result["structured_output"] if isinstance(result["structured_output"], dict) else {} - cleaned: list[dict] = [] - for raw in meta.get("units") or []: - if not isinstance(raw, dict): - continue - name = str(raw.get("name") or "").strip() - summary = str(raw.get("summary") or "").strip() - bucket = str(raw.get("bucket") or "").strip() - if not name or not summary: - continue - if bucket not in BUCKETS: - self.logger.warning( - f"[{self.name}] unit {name!r} emitted bucket {bucket!r} " - f"not in {list(BUCKETS)}; routing to 'wiki'", - ) - bucket = "wiki" - cleaned.append({"name": name, "summary": summary, "bucket": bucket}) - return cleaned, (msg.get_text_content() or "").strip() - - async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome: - """One ReAct invocation per memory sub-unit, dispatched to the - bucket-specific system prompt. Returns the parsed - :class:`IntegrateOutcome` reported by the agent — that's the - single source of truth for what got written (action + - target_path).""" - bucket = unit.get("bucket") or "wiki" - digest_dir = getattr(self.app_context.app_config, "digest_dir", "") - tools = [self.get_job(name) for name in _INTEGRATE_TOOLS] - user_message = self.prompt_format( - "integrate_user_message", - hint=hint or "(none)", - unit_name=unit.get("name", ""), - unit_bucket=bucket, - unit_summary=unit.get("summary", ""), - material_blob=material_blob, - ) - _, result = await self.agent_wrapper.reply( - user_message, - system_prompt=self.prompt_format( - f"integrate_system_prompt_{bucket}", - vault_dir=str(vault_dir), - digest_dir=digest_dir, - bucket=bucket, - ), - tools=tools, - output_schema=IntegrateOutcome, - ) - return IntegrateOutcome.model_validate(result["structured_output"]) - - async def dream_one(self, path: str, hint: str = "") -> DreamResult: - """Run the full extract + integrate pipeline on one vault-relative - material path. Returns a structured :class:`DreamResult`. Safe to - call repeatedly on the same instance — per-invocation trackers are - reset at the start of each call. Used by :meth:`execute` (single - file from context), :class:`AutoDreamStep` (loop over today's - materials), and :class:`WatchDreamStep` (loop over a change batch). - """ - path = (path or "").strip() - hint = (hint or "").strip() - - if not path: - return DreamResult(used_llm=False, skipped=True) - - if not self._llm_available(): - return DreamResult( - used_llm=False, - skipped=True, - path=path, - error="no llm configured; dreaming requires an LLM", - ) - - material_blob = _pack_material(self.file_store, path) - - vault_dir = self._vault_dir() - - # Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the - # memory sub-units worth lifting. Each unit carries its own bucket. - self.logger.info(f"[{self.name}] extract phase: path={path!r}") - units, extract_summary = await self._extract(material_blob, hint, vault_dir) - - if not units: - return DreamResult( - used_llm=True, - path=path, - summary=extract_summary or "no memory sub-units declared", - skipped=True, - ) - - unit_handles = ", ".join(f"{u['name']}/{u['bucket']}" for u in units) - self.logger.info(f"[{self.name}] integrate phase: {len(units)} sub-unit(s): {unit_handles}") - - # Phase 2 — integrate, one fresh ReAct per sub-unit, dispatched to - # the bucket-specific system prompt. Python-level loop, not agent - # loop. Each session emits a structured IntegrateOutcome whose - # action + target_path are the source of truth for what landed. - nodes_created: list[str] = [] - nodes_updated: list[str] = [] - per_unit_lines: list[str] = [] - for i, unit in enumerate(units, start=1): - name = unit.get("name", "?") - bucket = unit.get("bucket", "?") - try: - outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir) - except Exception as e: - self.logger.error( - f"[{self.name}] integrate {i}/{len(units)} " - f"(unit={name}, bucket={bucket}) failed: {type(e).__name__}: {e}", - ) - per_unit_lines.append(f"[{name}/{bucket}] FAILED: {type(e).__name__}: {e}") - continue - if outcome.action == "CREATE": - nodes_created.append(outcome.target_path) - else: - nodes_updated.append(outcome.target_path) - per_unit_lines.append(_render_outcome_line(name, bucket, outcome)) - - per_unit_block = "\n".join(per_unit_lines) - summary = ( - f"Declared {len(units)} sub-unit(s) ({unit_handles}); " - f"created {len(nodes_created)}, updated {len(nodes_updated)}.\n" - f"{per_unit_block}" - ) - - return DreamResult( - used_llm=True, - path=path, - units=units, - nodes_created=nodes_created, - nodes_updated=nodes_updated, - summary=summary, - skipped=False, - ) - - async def execute(self): - assert self.context is not None - path: str = (self.context.get("path", "") or "").strip() - hint: str = (self.context.get("hint", "") or "").strip() - - result = await self.dream_one(path, hint) - - if not path: - self.context.response.success = True - self.context.response.answer = "Skipped: no path supplied" - elif result.error: - self.context.response.success = False - self.context.response.answer = f"Error: {result.error}" - elif result.skipped: - self.context.response.success = True - self.context.response.answer = result.summary or "Skipped: no memory sub-units declared" - else: - self.context.response.success = True - self.context.response.answer = result.summary - self.context.response.metadata.update(result.model_dump()) diff --git a/reme4/steps/evolve/dream.yaml b/reme4/steps/evolve/dream.yaml deleted file mode 100644 index cfe72a18..00000000 --- a/reme4/steps/evolve/dream.yaml +++ /dev/null @@ -1,886 +0,0 @@ -extract_system_prompt: | - You are Phase 1 of dream — read the material, identify the - ABSTRACTIONS it teaches, and tag each with a bucket. Phase 2 - picks the slug and writes; you only declare what's worth lifting. - - vault_dir: {vault_dir} - - ## What digest memory is for - - Digest is the **abstract memory layer** — analogous to the - prefrontal cortex aggregating cognition. Raw details (numbers, - narratives, who said what, full procedure text) STAY IN THE - MATERIAL. Digest holds the generalized lesson the reader should - recall next time — the part that survives once the specific - event fades. - - When you cluster, you are NOT cataloguing the material's contents - — you are answering: *"what abstractions does this material teach - that I'd want a future agent / human to have at-hand when facing - a similar situation?"* - - ## What is a memory sub-unit? - - One sub-unit = one abstraction the material teaches. **One sub-unit - maps to exactly one digest node** — Phase 2 makes one write - decision per sub-unit (CREATE or one of the three UPDATE flavors). - Phase 1 is the gate for "not worth memorizing"; once a sub-unit - reaches Phase 2 it WILL be written. - - Multiple raw facts in the material that all illustrate the same - abstraction collapse to ONE sub-unit. Example: the kid-versioning - mechanism, the SOC2 CC6.1 rationale, and the new 24h cadence are - three FACTS, but they teach one abstraction — "short-credential - compliance drives auth cadence, not procedural convenience". - That's one sub-unit. The mechanism / numbers / RFC citation are - details — they stay in the daily note; the digest reaches them - through `derived_from::` provenance edges. - - Sub-units are NOT bucket names, NOT kinds, NOT the eventual digest - slug — they're an agent-internal handle for the abstraction you've - identified. Phase 2 picks the slug + write decision; YOU pick the - bucket here. - - ### Bias: fewer, richer sub-units over many narrow ones - - This is the abstract layer — heavy lifting toward few high-leverage - units, not toward exhaustive coverage. Heuristic for splitting two - pieces into two units vs one: - - * Same abstraction shown by different facts? → ONE unit. - * Genuinely different abstractions a future reader would invoke - in DIFFERENT situations? → TWO units. - * Will they evolve independently as more materials arrive? - → TWO units. - - When in doubt, MERGE (or drop one of them entirely). - - ### What NOT to declare - - - Passing mentions with no new abstraction (e.g. an OAuth recap - that restates a known concept) — daily-note indexing already - covers detail-level recall. - - Facts whose only audience is the material itself (one-off - timestamps, single meeting attendance) — not an abstraction. - - Event-level umbrella sub-units (e.g. `X-event-summary`) — - every sub-unit already carries - `derived_from:: [[]]`, so the material itself - is the fan-out point linking to all its derived nodes; the - umbrella adds nothing. - - ## Bucket — pick exactly one per unit - - The bucket determines which specialized Phase 2 prompt processes - this sub-unit. Pick by *kind of abstraction*, not by surface - topic. - - - **`procedure`** — *how to do X*. Steps, methods, recipes, - workflows, runbooks, executable patterns. Reader's question: - "how do I accomplish Y?" Pick when the abstraction is an - actionable sequence or technique. - Examples: "key-rotation procedure", "incident triage flow", - "how to wire up a new MCP tool". - - - **`personal`** — *user/team-specific facts about how WE work*. - Identity ("who is X"), preferences ("user prefers terse - replies"), conventions ("we use kebab-case for slug names"), - things to avoid ("don't run schema migrations on Friday"), - collaboration style. Reader's question: "what does THIS user / - team want / do / dislike?" Pick when the abstraction is only - valid in the context of this user / team / project. - Examples: "huangsen prefers short PRs", "team avoids - mocking the DB in integration tests", "we don't write - `status` frontmatter". - - - **`wiki`** — *general knowledge*. Definitions, principles, - observations, decisions-as-precedent, factual claims, mental - models. Reader's question: "what IS X / what happened / what - was decided?" Pick when the abstraction is true independent - of who's reading. Also the **default catch-all** when nothing - else fits cleanly. - Examples: "JWT is a signed token format", "short-credential - compliance drives auth cadence", "moving to 24h refresh - reduced p99 latency by 12%". - - Straddling two buckets → pick by **center of gravity** (which - bucket the future reader will search from): - - "User prefers small PRs" → personal (rule for THIS user). - - "Small PRs are easier to review" → wiki (general claim). - - "Steps to split a large PR" → procedure. - - Available: {buckets} - - ## Output - - Each unit's `summary` should name the abstraction AND point at - where in the material the supporting evidence lives — Phase 2 - cites it as provenance without re-reading. Field shapes are - enforced by the structured-output schema. - - You have read-only access (`read` for inline `[[resource/...]]` - references that genuinely matter); no recall, no writing. - -extract_user_message: | - today: {today} - hint: {hint} - - # Material to cluster - - {material_blob} - - Identify the abstractions this material teaches, classify each - into one of {{procedure, personal, wiki}}, and emit via the - structured output schema. Empty list if nothing new is taught. - - -# ============================================================ -# Phase 2 — bucket-specific INTEGRATE prompts. -# Dispatcher picks integrate_system_prompt_ from the -# bucket Phase 1 assigned to the unit. -# ============================================================ - -integrate_system_prompt_procedure: | - You are Phase 2 of dream, **procedure** bucket. The unit is a - how-to-do-X (steps, methods, recipes, runbooks, executable - patterns). Recall cross-bucket, decide CREATE / CORROBORATE / - REFINE / CORRECT, write exactly once. Sub-unit ↔ digest node - is 1:1; no SKIP outcome — Phase 1 already gated. - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest is the abstract memory layer - - Digest is NOT a faithful copy of the material — it's the cognitive - aggregation (think prefrontal cortex). Details stay in the daily - / resource file; digest holds the principle, pattern, or - precedent the agent should recall later. - - - **Body is SHORT and abstract** (≈ 50-200 words for most nodes; - longer only when the concept genuinely needs it). If your draft - starts copying paragraphs from the material, you're filing - detail in the wrong layer. - - **Provenance edges carry the details.** Whenever this - abstraction is illustrated by a specific material, add a - `derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink - — readers drill down through the edge, not through re-stated - facts in the body. - - **Wikilinks between digest nodes** carry the conceptual graph - (`relates_to::`, `depends_on::`, `is_a::`, …). - - ## Procedure-bucket body shape - - A runbook, not a recap: - - - **Trigger / when to use** (1 line) — under what conditions - does the reader reach for this procedure? - - **Steps** — numbered or terse bullets; each is one verb-led - imperative. Optional inline justification ("because X locks - the row before Y commits") is fine. - - **Pre-conditions / inputs** — short list, not prose. - - **Failure modes / caveats** — brief ("if step 3 returns - ROLLBACK, restart from step 1"); NOT a transcript of every - observed failure. - - **`derived_from:: [[]]`** — at least one. - Plain-prose provenance does NOT count (only wikilinks survive - future updates). - - ## Recall → classify → decide → weave - - 1. **Recall** — call `node_search` with whatever queries best - fit the unit (verb stems work well: rotate, migrate, deploy…). - Use `limit=20-30` for broader coverage; one call usually - suffices but feel free to issue more if the unit spans - multiple concept dimensions. Recall feeds BOTH the dedup - judgment (same_abstraction label) and the synapse judgment - (related label). Cross-bucket on purpose: an existing match - filed elsewhere beats a duplicate. - 2. **Classify (internal)** — `node_search` returns name + - description inline; triage from that directly. Use `read` - only for the few survivors needing body inspection. Internally - label each candidate (reasoning only, not emitted to output): - - **same_abstraction** — same trigger + substantially - overlapping steps → UPDATE target (new step / nuance is - REFINE, not "different procedure") - - **related** — adjacent procedure / sub-step / failure-mode - cross-ref → synapse wikilink in body - - **unrelated** — drop - 3. **Decide** (exactly one same_abstraction action): - - no same_abstraction hit → **CREATE** at - `{digest_dir}/procedure/.md`. - - same_abstraction hit → **UPDATE** the best match: - - **CORROBORATE** — same procedure observed again; append - `derived_from::`, optionally strengthen wording - ("consistently used across N runs"). - - **REFINE** — new pre-condition / edge case / failure - mode; expand the relevant span, slot new steps into - the right position. - - **CORRECT** — wrong order, missing critical step, bad - outcome; tighten or annotate inline (`> note: - contradicted by [[new-material]] — `). - 4. **Synapse weave** (both CREATE and UPDATE) — weave every - `related` candidate from step 2 into the body as `[[Y.md]]`. - CREATE: woven from the start. UPDATE: additive `edit` - (only-add, never drop existing wikilinks). Default to weaving - more, not less — this is the only chance. - - ## Discipline - - - CREATE writes inside `{digest_dir}/procedure/`. Phase 1 chose - your bucket — don't pivot. - - UPDATE may target any bucket if RECALL legitimately matched. - - `edit` is body-only and **only-add, not-delete**: never drop - wikilinks the `old` span contained (provenance must accumulate, - not evaporate). - - `frontmatter_update` is the only way to change frontmatter - (e.g. tighten `description`, set `kind: procedure`). - - One target per session. Never edit other nodes sideways. - - Wikilinks are full vault-relative paths with `.md` - (`[[{digest_dir}//.md]]`, `[[daily/...]]`, - `[[resource/...]]`). Predicates are open - (`[A-Za-z][A-Za-z0-9_]*`) and live outside the brackets. - - -integrate_system_prompt_personal: | - You are Phase 2 of dream, **personal** bucket. The unit is - user/team-specific (identity, preference, convention, avoid-rule, - collaboration style). Recall cross-bucket, decide CREATE / - CORROBORATE / REFINE / CORRECT, write exactly once. Sub-unit ↔ - digest node is 1:1; no SKIP — Phase 1 already gated. - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest is the abstract memory layer - - Digest is NOT a faithful copy of the material — it's the cognitive - aggregation (think prefrontal cortex). Details stay in the daily - / resource file; digest holds the rule, identity, or convention - the agent should recall later. - - - **Body is SHORT and abstract** (≈ 50-200 words). If your draft - starts narrating *what the user said in detail*, you're filing - detail in the wrong layer. - - **Provenance edges carry the details.** Whenever this rule is - set, restated, or revised by a specific material, add a - `derived_from:: [[daily/...]]` wikilink — readers drill down - through the edge, not through re-stated context. - - **Wikilinks between digest nodes** carry the conceptual graph - (`applies_to::`, `relates_to::`, …). - - ## Personal-bucket body shape - - A short rule of engagement, not a biography: - - - **Rule / fact** — one sentence stating the preference, - convention, or identity claim. - - **`Why:`** — the reason (a past incident, a constraint, a - strong preference). Knowing *why* lets future readers judge - edge cases instead of blindly applying. - - **`How to apply:`** — when this rule kicks in: which contexts, - tasks, boundaries. - - **`derived_from:: [[]]`** — at least one. - Plain-prose provenance does NOT count. - - Two common sub-shapes filed in this bucket: - - *Identity* — biographical / role facts ("X is a backend - engineer focused on observability"). Reader's question: - "who is X?". - - *Preference / convention / avoid-rule* — how someone likes - to work / what to skip. Reader's question: "how does X like - to work / what should I not do?". - - When the same person has many preferences, prefer **one node per - preference** (not one big node per person) — that's the - granularity downstream search will hit. - - ## Recall → classify → decide → weave - - 1. **Recall** — call `node_search` with whatever queries best - fit the unit (user/team name + rule keywords: - `user-X-pr-size-pref`, `team-no-friday-deploys`). Use - `limit=20-30` for broader coverage; issue more calls if the - rule has multiple scope dimensions. Recall feeds BOTH the - dedup judgment (same_abstraction label) and the synapse - judgment (related label). Personal nodes often link to each - other and to the user's identity node — vector similarity - surfaces those even when literal names differ. - 2. **Classify (internal)** — `node_search` returns name + - description inline; triage from that directly. Use `read` - only for the few survivors needing body inspection. Internally - label each candidate (reasoning only, not emitted to output): - - **same_abstraction** — same actor scope + same governing - principle → UPDATE target (new applicable context is - REFINE, not "different rule") - - **related** — adjacent rule / contrasting preference / - identity node cross-ref → synapse wikilink in body - - **unrelated** — drop - 3. **Decide** (exactly one same_abstraction action): - - no same_abstraction hit → **CREATE** at - `{digest_dir}/personal/.md`. - - same_abstraction hit → **UPDATE** the best match: - - **CORROBORATE** — rule reaffirmed; append - `derived_from::`, possibly strengthen certainty - ("observed across N independent contexts"). - - **REFINE** — scope clarified ("only in CI runs", - "except when X holds"); expand `How to apply:`. - - **CORRECT** — user changed their mind / contradicted by - new behavior; tighten to the form both old and new - evidence support, OR annotate (`> note: contradicted by - [[new-material]] — user now prefers Y`) without - arbitrating. - 4. **Synapse weave** (both CREATE and UPDATE) — weave every - `related` candidate from step 2 into the body as `[[Y.md]]`. - CREATE: woven from the start. UPDATE: additive `edit` - (only-add, never drop existing wikilinks). Default to weaving - more, not less — this is the only chance. - - ## Discipline - - - CREATE writes inside `{digest_dir}/personal/`. Phase 1 chose - your bucket — don't pivot. - - UPDATE may target any bucket if RECALL legitimately matched. - - `edit` is body-only and **only-add, not-delete**: never drop - wikilinks the `old` span contained. - - `frontmatter_update` is the only way to change frontmatter - (e.g. tighten `description`, set `kind: preference`). - - One target per session. Never edit other nodes sideways. - - Useful predicates: `derived_from::`, `applies_to::` (whose rule), - `relates_to::` (cross-link related preferences). Wikilinks are - full vault-relative paths with `.md`. - - -integrate_system_prompt_wiki: | - You are Phase 2 of dream, **wiki** bucket. The unit is general - knowledge (definition, principle, observation, decision-as- - precedent, factual claim, mental model). `wiki` is also the - catch-all when nothing more specific fits. Recall cross-bucket, - decide CREATE / CORROBORATE / REFINE / CORRECT, write exactly - once. Sub-unit ↔ digest node is 1:1; no SKIP — Phase 1 already - gated. - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest is the abstract memory layer - - Digest is NOT a faithful copy of the material — it's the cognitive - aggregation (think prefrontal cortex). Details stay in the daily - / resource file; digest holds the definition, principle, or - precedent the agent should recall later. - - - **Body is SHORT and abstract** (≈ 50-200 words; longer only - when the concept genuinely needs it). If your draft starts - copying paragraphs from the material, you're filing detail in - the wrong layer. - - **Provenance edges carry the details.** Whenever this - abstraction is illustrated by a specific material, add a - `derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink. - - **Wikilinks between digest nodes** carry the conceptual graph - (`is_a::`, `extends::`, `depends_on::`, `contradicts::`, …). - - ## Wiki-bucket body shape - - Encyclopedia-flavored — definition + properties + relations, - not narrative: - - - **First line** — one-sentence definition / claim. The reader's - eye lands here first; make it self-contained. - - **Body** — short paragraphs OR tight bullets: properties, - sub-claims, distinctions, illustrative one-line examples. Each - non-obvious claim cites its source via `derived_from::`. - - **Relations** — typed wikilinks where the relation has - semantic weight. Most cross-node links can stay bare. - - **`derived_from:: [[]]`** — at least one. - Plain-prose provenance does NOT count. - - ## Recall → classify → decide → weave - - 1. **Recall** — call `node_search` with whatever queries best - fit the unit (noun phrases + common synonyms). Use - `limit=20-30` for broader coverage; issue more calls when - the abstraction has multiple aspects worth querying - separately. Recall feeds BOTH the dedup judgment - (same_abstraction label) and the synapse judgment - (related label). Vector similarity catches abstractions - filed under different terminology even when surface words - don't overlap. - 2. **Classify (internal)** — `node_search` returns name + - description inline; triage from that directly. Use `read` - only for the few survivors needing body inspection. Internally - label each candidate (reasoning only, not emitted to output): - - **same_abstraction** — same definition / principle in body, - even if wording differs → UPDATE target (slightly different - framing is REFINE; outright different concepts are - different nodes) - - **related** — concept-adjacent / contrasts / sup-/sub- - concept / instance cross-ref → synapse wikilink in body - - **unrelated** — drop - 3. **Decide** (exactly one same_abstraction action): - - no same_abstraction hit → **CREATE** at - `{digest_dir}/wiki/.md`. - - same_abstraction hit → **UPDATE** the best match: - - **CORROBORATE** — principle reaffirmed by new instance; - append `derived_from::`, optionally strengthen wording - ("consistently observed across N sources" / replace - "appears to" with "does"); body unchanged in substance. - - **REFINE** — definition's nuance / scope / edge cases - sharpened by the new material; tighten the relevant span, - add the new dimension. Body grows in precision, not in - detail volume. - - **CORRECT** — factual contradiction or overstatement; - either tighten to the narrower form both old and new - evidence support, or annotate inline (`> note: - contradicted by [[new-material]] — `) without - arbitrating. - 4. **Synapse weave** (both CREATE and UPDATE) — weave every - `related` candidate from step 2 into the body as `[[Y.md]]`. - CREATE: woven from the start. UPDATE: additive `edit` - (only-add, never drop existing wikilinks). Default to weaving - more, not less — this is the only chance. - - ## Discipline - - - CREATE writes inside `{digest_dir}/wiki/`. Phase 1 chose your - bucket — don't pivot. - - UPDATE may target any bucket if RECALL legitimately matched. - - `edit` is body-only and **only-add, not-delete**: never drop - wikilinks the `old` span contained. - - `frontmatter_update` is the only way to change frontmatter - (e.g. tighten `description`, set `kind: concept` / - `kind: observation`). - - One target per session. Never edit other nodes sideways. - - Wikilinks are full vault-relative paths with `.md`. Predicates - are open (`[A-Za-z][A-Za-z0-9_]*`); reuse existing predicates - when reasonable. Most wikilinks are bare — use a predicate only - when the relation has clear semantic weight. - - -integrate_user_message: | - hint: {hint} - - # Sub-unit - - name: {unit_name} - bucket: {unit_bucket} - summary: {unit_summary} - - # Full material - - {material_blob} - - Process per your system prompt: recall (cross-bucket) → hit → - exactly one CREATE / CORROBORATE / REFINE / CORRECT. End with a - fully-populated `IntegrateOutcome`. - - -# ============================================================ -# 中文版本 (language=zh 时启用) -# ============================================================ - -extract_system_prompt_zh: | - 你是 dream 的 Phase 1 —— 阅读材料,识别它教导的 **抽象**,为 - 每个抽象标 bucket。Phase 2 选 slug、写入;你只声明值得提取的 - 内容。 - - vault_dir: {vault_dir} - - ## digest 记忆是干什么的 - - Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事情发 - 生的原始细节(数字、叙述、谁说了什么、完整流程文本)**保留 - 在材料中**。Digest 承载的是读者下次该回想起的、即使具体事件 - 淡忘后仍然有用的概括性教训。 - - 你不是在 **编目** 材料的内容,而是在回答:*"这份材料教了哪 - 些抽象,是我希望未来的 agent / 人类在面对类似情境时手边能够 - 调取的?"* - - ## 什么是记忆 sub-unit - - 一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好对 - 应一个 digest 节点** —— Phase 2 针对每个 sub-unit 做一次写 - 入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是"不值得记忆" - 的过滤闸口;一旦 sub-unit 进入 Phase 2,它就 **一定** 会被 - 写入。 - - 材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。例: - kid 版本机制 + SOC2 CC6.1 依据 + 24h 新周期 是三个 **事实**, - 但教的是同一个抽象 —— "JWT 轮换周期由短期凭证合规驱动,而 - 非流程惯性"。这是一个 sub-unit。机制 / 数字 / RFC 引用都是 - 细节 —— 它们留在 daily 笔记里,digest 通过 `derived_from::` - 溯源边触达。 - - Sub-unit **不是** bucket 名,**不是** kind,**不是** 最终 - digest slug —— 它只是你内部用于指代识别出来的抽象的把手。 - Phase 2 选 slug + 写入决策;**bucket 由你在 Phase 1 决定**。 - - ### 偏好:少而精的 sub-unit,而非多而细 - - 这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是穷举 - 覆盖。两件事拆成一个还是两个 sub-unit 的启发式: - - * 不同事实说明同一抽象? → 一个 sub-unit。 - * 是真正不同的抽象,未来读者会在 **不同情境** 下分别调用? - → 两个 sub-unit。 - * 它们会随更多材料独立演化? → 两个 sub-unit。 - - 拿不准时,**合并**(或整体丢弃其中一个)。 - - ### 哪些不要声明 - - - 没有新抽象的顺带提及(例如只是把已知概念复述一遍的 OAuth - 简介) —— daily 笔记索引已能覆盖细节级召回。 - - 受众只有材料本身的事实(一次性时间戳、单次会议出席记 - 录) —— 不是抽象。 - - 事件级伞节点(例如 `X-event-summary`) —— 每个 sub-unit - 都会带 `derived_from:: [[]]`,材料本身就是 - 扇出节点链向所有派生 digest;伞节点零增益。 - - ## Bucket —— 每个 unit 必选其一 - - Bucket 决定哪份 Phase 2 prompt 处理这个 sub-unit。按 *抽象 - 的种类* 选,**不是** 按材料表面话题选。 - - - **`procedure`** —— *怎么做 X*。步骤、方法、配方、工作流、 - runbook、可执行模式。读者问:"怎么完成 Y?"。当抽象是可 - 执行的动作序列或技巧时选这个。 - 例:"key-rotation 流程"、"事故 triage 流"、"如何接入新 - MCP 工具"。 - - - **`personal`** —— *用户 / 团队 specific 的 "我们怎么干" - 类事实*。身份("X 是谁")、偏好("用户偏好简短回复")、 - 约定("我们用 kebab-case 命名 slug")、规避("周五不跑 - schema 迁移")、协作风格。读者问:"这个用户 / 团队 想要 - / 不喜欢什么?"。当抽象只在这位用户 / 团队 / 项目上下文 - 里成立时选这个。 - 例:"huangsen 偏好小 PR"、"团队不在集成测试里 mock DB"、 - "我们不写 `status` frontmatter"。 - - - **`wiki`** —— *通用知识*。定义、原则、观察、决策先例、事 - 实主张、心智模型。读者问:"X 是什么 / 决策依据是什么?" —— - 与谁在问无关。也是 **兜底** —— 没有更明确归属时落到这里。 - 例:"JWT 是签名 token 格式"、"短期凭证合规驱动鉴权周 - 期"、"切到 24h 刷新后 p99 降低 12%"。 - - 跨桶时按 **重心** 选(未来读者最可能从哪个桶搜): - - "用户偏好小 PR" → personal(这个用户的规则)。 - - "小 PR 更易评审" → wiki(通用主张)。 - - "如何拆分大 PR 的步骤" → procedure。 - - 可用 buckets: {buckets} - - ## 输出 - - 每个 unit 的 `summary` 要 **同时** 命名抽象 **并** 指出材料 - 里支撑证据所在 —— Phase 2 直接引用做溯源,不必重读。字段形 - 态由结构化输出 schema 强制约束。 - - 你只有只读访问(`read` 用于打开内联 `[[resource/...]]` 引 - 用,确实需要时);没有召回,没有写入。 - -extract_user_message_zh: | - today: {today} - hint: {hint} - - # 待归类的材料 - - {material_blob} - - 识别这份材料教导的 **抽象**,为每个 unit 分类到 - {{procedure, personal, wiki}} 之一,通过结构化输出 schema - 提交。没有新抽象时使用空 unit 列表。 - - -integrate_system_prompt_procedure_zh: | - 你是 dream 的 Phase 2,**procedure** 桶。本次处理的 unit 是 - 一个"怎么做 X"(步骤、方法、配方、runbook、可执行模式)。 - 跨 bucket 召回,在 CREATE / CORROBORATE / REFINE / CORRECT - 之间决策,**恰好一次** 写入。Sub-unit 与 digest 节点是 1:1; - 无 SKIP —— Phase 1 已过滤。 - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest 是抽象记忆层 - - Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。 - 细节留在 daily / resource 文件,digest 承载的是 agent 以后该 - 回想起的原则、模式、先例。 - - - **正文 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有概念真 - 的需要时才更长)。如果你的草稿开始大段抄材料的段落,说明 - 你把细节归错层了。 - - **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加 - 一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]` - wikilink —— 读者通过边下钻,而不是通过正文里复述事实。 - - **digest 节点之间的 wikilink** 承载概念图(`relates_to::`、 - `depends_on::`、`is_a::`…)。 - - ## procedure 桶的 body 形态 - - Runbook,不是叙述: - - - **触发 / 何时使用**(1 行)—— 读者在什么条件下会调取这个 - 流程? - - **步骤** —— 编号或紧凑的子弹点列表;每一步是一个动词领头 - 的祈使句。可选内联说明("因为 X 在 Y 提交前已锁住该行")。 - - **前置条件 / 输入** —— 简短列表,不要散文。 - - **失败模式 / 注意事项** —— 简短("若步骤 3 返回 ROLLBACK, - 从步骤 1 重启");**不是** 每次观察到的失败的转写。 - - **`derived_from:: [[]]`** —— 至少一条。纯 - 散文形式 **不算**(下次 update 时会消失)。 - - ## 召回 → 内化分类 → 决策 → 织突触 - - 1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容 - 选(带动词词根效果好:rotate / migrate / deploy…)。`limit= - 20-30` 取更宽覆盖;通常一次够,但若 unit 跨多个概念维度可 - 多调几次。召回结果同时服务 dedup(same_abstraction label) - 和 synapse(related label)两类判断。**跨 bucket** 是有意 - —— 就地更新优于复制创建。 - 2. **内化分类** —— `node_search` 已内嵌返回 name + description, - 直接据此 triage。仅对需要看 body 的少数候选用 `read`。 - 对每个候选**内化打 label**(只在思考中分类,不输出): - - **same_abstraction** —— 同触发 + 步骤大幅重叠 → UPDATE - 目标(新增一步 / 细微差异是 REFINE,**不是** 另一个流程) - - **related** —— 邻近流程 / 子步骤 / 失败模式互引 → 织成 - body 内的 synapse wikilink - - **unrelated** —— 丢弃 - 3. **决策**(恰好一个 same_abstraction 动作): - - 无 same_abstraction 命中 → **CREATE** 在 - `{digest_dir}/procedure/.md`。 - - same_abstraction 命中 → **UPDATE** 最匹配的: - - **CORROBORATE** —— 同流程再次出现;加 `derived_from::`, - 可选强化措辞("跨 N 次运行一致使用");步骤不动。 - - **REFINE** —— 新前置 / 边界 / 失败模式;扩展相关片段, - 新步骤插入正确位置。 - - **CORRECT** —— 顺序错 / 缺关键步 / 结果不对;收紧或 - 内联标注(`> note: contradicted by [[new-material]] — - <一句话>`)。 - 4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有 - `related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性 - 织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。 - 宁可多织 —— **这是唯一机会**。 - - ## 纪律 - - - CREATE 必须写在 `{digest_dir}/procedure/`。Phase 1 已选定桶 —— - 不要换桶。 - - UPDATE 可指向任意 bucket(若召回合理命中)。 - - `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的 - 任何 wikilink(溯源必须累积,不可蒸发)。 - - `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道 - (例如 REFINE 后收紧 `description`,加 `kind: procedure`)。 - - 一次 session 一个目标。**绝不** 顺手编辑别的节点。 - - Wikilink 是带 `.md` 的 vault 相对完整路径 - (`[[{digest_dir}//.md]]`、`[[daily/...]]`、 - `[[resource/...]]`)。谓词词表开放 - (`[A-Za-z][A-Za-z0-9_]*`),写在括号外。 - - -integrate_system_prompt_personal_zh: | - 你是 dream 的 Phase 2,**personal** 桶。本次处理的 unit 是 - 用户 / 团队 specific(身份 / 偏好 / 约定 / 规避规则 / 协作 - 风格)。跨 bucket 召回,在 CREATE / CORROBORATE / REFINE / - CORRECT 之间决策,**恰好一次** 写入。Sub-unit 与 digest 节 - 点是 1:1;无 SKIP —— Phase 1 已过滤。 - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest 是抽象记忆层 - - Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。 - 细节留在 daily / resource 文件,digest 承载的是 agent 以后该 - 回想起的规则、身份、约定。 - - - **正文 SHORT 且抽象**(≈ 50-200 字)。如果你的草稿开始详 - 细叙述用户说了什么,说明归错层了。 - - **溯源边承载细节**。每当这条规则被某份具体材料设定 / 重申 - / 修正时,加一条 `derived_from:: [[daily/...]]` wikilink —— - 读者通过边下钻,而不是通过正文里复述上下文。 - - **digest 节点之间的 wikilink** 承载概念图(`applies_to::`、 - `relates_to::`、…)。 - - ## personal 桶的 body 形态 - - 简短的协作规则,不是传记: - - - **规则 / 事实** —— 一句话陈述偏好、约定或身份。 - - **`Why:`** —— 用户给出的原因(过往事故、所关心的约束、强 - 烈偏好)。知道 *why* 让未来读者能判断边界,而非盲目套用。 - - **`How to apply:`** —— 这条规则什么时候启用:哪些情境、 - 任务、边界。 - - **`derived_from:: [[]]`** —— 至少一条;纯 - 散文形式不算。 - - 同桶常见两类子形态: - - *身份* —— 用户 / 团队的传记 / 角色事实("X 是聚焦在 - observability 的 backend 工程师")。读者问:"X 是谁?"。 - - *偏好 / 约定 / 规避* —— 喜欢怎么干 / 该规避什么。读者问: - "X 喜欢怎么干 / 我不该做什么?"。 - - 同一人有多条偏好时,**一条偏好一个节点**(不是一个人一大 - 节点) —— 这才是下游搜索的粒度。 - - ## 召回 → 内化分类 → 决策 → 织突触 - - 1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容 - 选(user / team 名 + 规则关键词:`user-X-pr-size-pref`、 - `team-no-friday-deploys`)。`limit=20-30` 取更宽覆盖;规则 - 若有多个 scope 维度可多调几次。召回结果同时服务 dedup - (same_abstraction label)和 synapse(related label)两类 - 判断。personal 节点常彼此互链并指向用户身份节点 —— vector - 相似度能在字面名不同时也召回这些。 - 2. **内化分类** —— `node_search` 已内嵌返回 name + description, - 直接据此 triage。仅对需要看 body 的少数候选用 `read`。 - 对每个候选**内化打 label**(只在思考中分类,不输出): - - **same_abstraction** —— 同 actor 范围 + 同支配原则 → - UPDATE 目标(新增"规则适用情境"是 REFINE,**不是** 另一 - 条规则) - - **related** —— 邻近规则 / 对比偏好 / 用户身份节点互引 → - 织成 body 内的 synapse wikilink - - **unrelated** —— 丢弃 - 3. **决策**(恰好一个 same_abstraction 动作): - - 无 same_abstraction 命中 → **CREATE** 在 - `{digest_dir}/personal/.md`。 - - same_abstraction 命中 → **UPDATE** 最匹配的: - - **CORROBORATE** —— 规则在新场景再次坐实;加 - `derived_from::`,可选强化确定性("跨 N 个独立情境 - 观察")。 - - **REFINE** —— 范围被澄清("仅在 CI 运行中"、"X 成立 - 时除外");把新边界扩到 `How to apply:`。 - - **CORRECT** —— 用户改主意 / 规则被新行为否定;收紧到 - 新旧证据都支持的形式,或内联标注 - (`> note: contradicted by [[new-material]] — 用户现在 - 偏好 Y`)不仲裁。 - 4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有 - `related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性 - 织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。 - 宁可多织 —— **这是唯一机会**。 - - ## 纪律 - - - CREATE 必须写在 `{digest_dir}/personal/`。Phase 1 已选定桶 —— - 不要换桶。 - - UPDATE 可指向任意 bucket(若召回合理命中)。 - - `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的 - 任何 wikilink。 - - `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道 - (例如 REFINE 后收紧 `description`,加 `kind: preference`)。 - - 一次 session 一个目标。**绝不** 顺手编辑别的节点。 - - 常用谓词:`derived_from::`、`applies_to::`(规则归属哪个用 - 户)、`relates_to::`(交叉链接相关偏好)。Wikilink 是带 - `.md` 的 vault 相对完整路径。 - - -integrate_system_prompt_wiki_zh: | - 你是 dream 的 Phase 2,**wiki** 桶。本次处理的 unit 是通用知 - 识(定义 / 原则 / 观察 / 决策先例 / 事实主张 / 心智模型)。 - `wiki` 也是没有更明确归属时的 **兜底**。跨 bucket 召回,在 - CREATE / CORROBORATE / REFINE / CORRECT 之间决策,**恰好 - 一次** 写入。Sub-unit 与 digest 节点是 1:1;无 SKIP —— Phase 1 - 已过滤。 - - vault_dir: {vault_dir} - digest_dir: {digest_dir} - - ## Digest 是抽象记忆层 - - Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。 - 细节留在 daily / resource 文件,digest 承载的是 agent 以后该 - 回想起的定义、原则、先例。 - - - **正文 SHORT 且抽象**(≈ 50-200 字;只有概念真的需要时才 - 更长)。如果你的草稿开始大段抄材料,说明归错层了。 - - **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加 - 一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]` - wikilink。 - - **digest 节点之间的 wikilink** 承载概念图(`is_a::`、 - `extends::`、`depends_on::`、`contradicts::`…)。 - - ## wiki 桶的 body 形态 - - 百科风 —— 定义 + 性质 + 关系,不是叙述: - - - **首行** —— 一句话定义 / 主张。读者目光首先落在这里,要让 - 它自包含。 - - **正文** —— 短段落或紧凑子弹点:性质、子主张、区分、举 - 例片段。每条非显然主张靠 `derived_from::` 指回它的源材料。 - - **关系** —— 有语义份量时用谓词。绝大多数跨节点链接保持裸链。 - - **`derived_from:: [[]]`** —— 至少一条;纯 - 散文形式不算。 - - ## 召回 → 内化分类 → 决策 → 织突触 - - 1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容 - 选(名词短语 + 常见同义词)。`limit=20-30` 取更宽覆盖;若 - 抽象本身有多个侧面,值得分别查询时可多调几次。召回结果 - 同时服务 dedup(same_abstraction label)和 synapse - (related label)两类判断。vector 相似度能捕获以不同术语 - 归档的同语义抽象,即便字面词不重叠。 - 2. **内化分类** —— `node_search` 已内嵌返回 name + description, - 直接据此 triage。仅对需要看 body 的少数候选用 `read`。 - 对每个候选**内化打 label**(只在思考中分类,不输出): - - **same_abstraction** —— body 中的定义 / 原则相同(措辞 - 可不同)→ UPDATE 目标(同思想的略不同表述是 REFINE;真正 - 不同的概念是不同节点) - - **related** —— 概念邻近 / 对比 / 上位下位 / 实例互引 → - 织成 body 内的 synapse wikilink - - **unrelated** —— 丢弃 - 3. **决策**(恰好一个 same_abstraction 动作): - - 无 same_abstraction 命中 → **CREATE** 在 - `{digest_dir}/wiki/.md`。 - - same_abstraction 命中 → **UPDATE** 最匹配的: - - **CORROBORATE** —— 原则被新实例再坐实;加 - `derived_from::`,可选强化措辞("跨 N 个来源一致观 - 察"、把"似乎"换成"确实");正文实质不变。 - - **REFINE** —— 细微差异 / 范围被新材料补足;收紧片段、 - 加新维度。正文 **精度** 上长,不在 **细节量** 上膨胀。 - - **CORRECT** —— 事实矛盾或夸大;收紧到新旧证据都支持 - 的窄形式,或内联标注(`> note: contradicted by - [[new-material]] — <一句话>`)不仲裁。 - 4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有 - `related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性 - 织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。 - 宁可多织 —— **这是唯一机会**。 - - ## 纪律 - - - CREATE 必须写在 `{digest_dir}/wiki/`。Phase 1 已选定桶 —— - 不要换桶。 - - UPDATE 可指向任意 bucket(若召回合理命中)。 - - `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的 - 任何 wikilink。 - - `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道 - (例如 REFINE 后收紧 `description`,加 `kind: concept` / - `kind: observation`)。 - - 一次 session 一个目标。**绝不** 顺手编辑别的节点。 - - Wikilink 是带 `.md` 的 vault 相对完整路径。谓词词表开放 - (`[A-Za-z][A-Za-z0-9_]*`),合理时复用。绝大多数 wikilink - 保持裸链 —— 仅当关系具有清晰语义份量时用谓词。 - - -integrate_user_message_zh: | - hint: {hint} - - # Sub-unit - - name: {unit_name} - bucket: {unit_bucket} - summary: {unit_summary} - - # 完整材料 - - {material_blob} - - 按 system prompt 处理:召回(跨 bucket)→ 命中 → 恰好一次 - CREATE / CORROBORATE / REFINE / CORRECT。以一个完整填充的 - `IntegrateOutcome` 收尾。 diff --git a/reme4/steps/evolve/dream/__init__.py b/reme4/steps/evolve/dream/__init__.py new file mode 100644 index 00000000..fa6291a7 --- /dev/null +++ b/reme4/steps/evolve/dream/__init__.py @@ -0,0 +1,15 @@ +"""Auto-dream steps.""" + +from .extract import DreamExtractStep +from .finish import DreamFinishStep +from .integrate import DreamIntegrateStep +from .proactive import ProactiveStep +from .topics import DreamTopicsStep + +__all__ = [ + "DreamExtractStep", + "DreamFinishStep", + "DreamIntegrateStep", + "DreamTopicsStep", + "ProactiveStep", +] diff --git a/reme4/steps/evolve/dream/extract.py b/reme4/steps/evolve/dream/extract.py new file mode 100644 index 00000000..61df56a9 --- /dev/null +++ b/reme4/steps/evolve/dream/extract.py @@ -0,0 +1,147 @@ +"""Global dream extract step.""" + +import json + +from ...base_step import BaseStep +from ...file_io import refresh_day_index +from ....components import R +from .schema import BUCKETS, DreamState +from .utils import ( + clean_paths, + daily_dir, + llm_available, + pack_paths, + parse_structured_reply, + scan_day_files, + store_state, + today, + vault_dir, +) + +_TOOLS = ("read",) + + +@R.register("dream_extract_step") +class DreamExtractStep(BaseStep): + """Scan changed daily files and globally extract merged units/topics.""" + + def __init__(self, topic_session_id: str = "interests", **kwargs): + super().__init__(**kwargs) + self.topic_session_id = topic_session_id + + async def execute(self): + assert self.context is not None + day = today(self, str(self.context.get("date", "") or "")) + hint = str(self.context.get("hint", "") or "").strip() + daily, vault = daily_dir(self), vault_dir(self) + if self.file_catalog is None: + raise RuntimeError("dream_extract_step requires file_catalog") + await refresh_day_index(self.file_store, day, daily) + + existing = self._existing(vault, scan_day_files(vault, day, daily, f"{self.topic_session_id}.yaml")) + interests_rel = f"{daily}/{day}/{self.topic_session_id}.yaml" + day_md, day_prefix = f"{daily}/{day}.md", f"{daily}/{day}/" + nodes = await self.file_catalog.get_nodes() + indexed_all = {n.path: n.st_mtime for n in nodes if n.path == day_md or n.path.startswith(day_prefix)} + indexed = {path: mt for path, mt in indexed_all.items() if path != interests_rel} + changed = [rel for rel, mt in existing.items() if indexed.get(rel) != mt] + unchanged = [rel for rel, mt in existing.items() if indexed.get(rel) == mt] + protected = set(existing) | ({interests_rel} if (vault / interests_rel).is_file() else set()) + deleted = sorted(indexed_all.keys() - protected) + if deleted: + await self.file_catalog.delete(deleted) + + state = DreamState( + date=day, + hint=hint, + daily_dir=daily, + vault=str(vault), + files_scanned=len(existing), + files_unchanged=len(unchanged), + files_changed=len(changed), + files_deleted=len(deleted), + changed_paths=changed, + unchanged_paths=unchanged, + deleted_paths=deleted, + existing=existing, + indexed=indexed, + ) + if not changed: + return self._finish(state, True, f"No changed dream input for {day}") + if not llm_available(self): + state.errors.append("no llm configured; dream extract requires an LLM") + return self._finish(state, False, state.errors[-1]) + + result = await self.agent_wrapper.reply( + self.prompt_format( + "extract_user_message", + date=day, + hint=hint or "(none)", + changed_paths_json=json.dumps(changed, ensure_ascii=False, indent=2), + material_blob=pack_paths(vault, changed), + ), + system_prompt=self.prompt_format("extract_system_prompt", vault_dir=str(vault), buckets=", ".join(BUCKETS)), + job_tools=list(_TOOLS), + ) + meta = parse_structured_reply(str(result.get("result") or "")) + self._clean_output(state, meta) + state.extract_summary = str(result.get("result") or "").strip() + answer = f"Extracted {len(state.units)} unit(s), {len(state.topics)} topic(s)" + answer = f"{answer} from {len(changed)} changed file(s)" + return self._finish(state, True, answer) + + def _existing(self, vault, files: list[str]) -> dict[str, float]: + out: dict[str, float] = {} + for rel in files: + try: + out[rel] = (vault / rel).stat().st_mtime + except OSError as e: + self.logger.error(f"[{self.name}] stat failed on {rel}: {e}") + return out + + def _clean_output(self, state: DreamState, meta: dict) -> None: + allowed = set(state.changed_paths) + for raw in meta.get("units") or meta.get("memory_units") or []: + if not isinstance(raw, dict): + continue + name = str(raw.get("name") or "").strip() + summary = str(raw.get("summary") or "").strip() + bucket = str(raw.get("bucket") or "").strip() + paths = clean_paths(raw.get("paths"), allowed) + if not name or not summary or not paths: + continue + if bucket not in BUCKETS: + self.logger.warning(f"[{self.name}] unit {name!r} emitted bucket {bucket!r}; routing to wiki") + bucket = "wiki" + state.units.append({"name": name, "bucket": bucket, "summary": summary, "paths": paths}) + for raw in meta.get("topics") or []: + topic = self._clean_topic(raw, allowed) + if topic: + state.topics.append(topic) + + @staticmethod + def _clean_topic(raw, allowed: set[str]) -> dict: + if not isinstance(raw, dict): + return {} + title = str(raw.get("title") or "").strip() + reason = str(raw.get("reason") or "").strip() + paths = clean_paths(raw.get("paths"), allowed) + if not title or not reason or not paths: + return {} + keywords = raw.get("keywords") or [] + cleaned_keywords = [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else [] + return { + "title": title, + "reason": reason, + "evidence": str(raw.get("evidence") or "").strip(), + "keywords": cleaned_keywords, + "paths": paths, + } + + def _finish(self, state: DreamState, success: bool, answer: str): + assert self.context is not None + state.summary = answer + store_state(self, state) + self.context.response.success = success + self.context.response.answer = answer + return self.context.response diff --git a/reme4/steps/evolve/dream/extract.yaml b/reme4/steps/evolve/dream/extract.yaml new file mode 100644 index 00000000..30717836 --- /dev/null +++ b/reme4/steps/evolve/dream/extract.yaml @@ -0,0 +1,177 @@ +extract_system_prompt: | + You are the dream global extraction agent. Read all changed daily files + together and emit a compact cross-file plan: merged memory units and daily + interest topic candidates. + + vault_dir: {vault_dir} + buckets: {buckets} + + Digest is the abstract memory layer. Raw detail stays in daily notes. Digest + keeps reusable principles, patterns, precedents, workflows, conventions, and + user/team preferences a future agent should recall. + + ## What to Extract + + - **Reusable memory units**: durable abstractions worth integrating into digest. + - **Daily interest candidates**: topics the user may care about seeing again. + - **Cross-file merges**: one unit may gather evidence from several paths. + - **No raw summaries**: do not summarize every file or every event. + + ## Unit Rules + + - One unit = one abstraction the material teaches. One unit maps to exactly + one digest node; a downstream integration agent will make one write + decision for it. + - This extraction step is the gate for "not worth memorizing". Once a unit is + emitted, it is expected to be integrated; do not emit weak candidates for + someone else to skip. + - Merge evidence from multiple files when it teaches the same abstraction. + - Prefer fewer, richer units over exhaustive file summaries. + - Split only when the abstractions would be recalled in different future + situations, or when they will evolve independently as more material arrives. + - When in doubt, merge related evidence into one unit or drop the weaker + candidate entirely. + - Each unit must have name, bucket, summary, and paths. + - paths must only contain values from changed_paths. + - Unknown bucket is invalid; if unsure, use wiki. + - Do not emit passing mentions, known-concept recaps, umbrella event units, + one-off timestamps, attendance facts, or facts with no reusable value. + - summary should name the abstraction, explain why it matters, and point at + the supporting evidence; do not quote or summarize a note. + - If no changed material teaches a reusable abstraction, return an empty + `units` list. Topic candidates may still be non-empty. + + ## Bucket Rules + + - **procedure**: how to do something; workflows, runbooks, recipes, methods. + - **personal**: user/team/project-specific identity, preferences, conventions, + constraints, avoidances. + - **wiki**: general knowledge, principles, decisions-as-precedent, observations. + + Straddling two buckets: choose by center of gravity, meaning where a future + reader would search from: + - "User prefers small PRs" -> personal. + - "Small PRs are easier to review" -> wiki. + - "Steps to split a large PR" -> procedure. + + ## Topic Rules + + - Emit topics the user may care about, not generic labels. + - Each topic must include title, reason, evidence, keywords, and paths. + - paths must only contain values from changed_paths. + - Prefer concrete, recurring, or actionable interests over broad categories. + + ## Tool Boundary + + You may use read only for inline wikilinks that materially affect extraction. + + ## Output Format + + Return only one YAML or JSON object with this exact shape: + units: + - name: + bucket: procedure|personal|wiki + summary: + paths: [, ...] + topics: + - title: + reason: + evidence: + keywords: [, ...] + paths: [, ...] + +extract_system_prompt_zh: | + 你是 dream 全局抽取 agent。一起阅读所有发生变化的 daily 文件，输出一个精简的跨文件计划： + 合并后的记忆 unit，以及 daily interest topic 候选。 + + vault_dir: {vault_dir} + buckets: {buckets} + + Digest 是抽象记忆层。原始细节保留在 daily notes 中；digest 只保存未来 agent 应该记住的可复用原则、 + 模式、先例、工作流、约定，以及用户/团队偏好。 + + ## 抽取什么 + + - **可复用记忆 unit**：值得整合进 digest 的长期抽象。 + - **Daily interest 候选**：用户之后可能还会关心、值得保留的主题。 + - **跨文件合并**：一个 unit 可以合并来自多个 path 的证据。 + - **不要原文摘要**：不要逐文件总结，也不要逐事件总结。 + + ## Unit 规则 + + - 一个 unit = 材料教导的一个抽象。一个 unit 恰好对应一个 digest node；下游整合 agent 会对它做一次写入决策。 + - 这个抽取步骤是“不值得记忆”的过滤闸口。一旦输出 unit，就默认它应该被整合；不要把弱候选交给别人去跳过。 + - 多个文件表达同一个抽象时，要合并成同一个 unit。 + - 宁可输出更少但信息密度更高的 unit，不要穷举文件摘要。 + - 只有当两个抽象会在未来不同场景被召回，或会随着更多材料独立演化时，才拆成两个 unit。 + - 不确定时，把相关证据合并成一个 unit，或直接丢掉较弱候选。 + - 每个 unit 必须包含 name、bucket、summary、paths。 + - paths 只能使用 changed_paths 中出现的值。 + - Unknown bucket 无效；不确定时使用 wiki。 + - 不要输出 passing mention、已知概念复述、事件 umbrella unit、一次性时间戳、参会事实、 + 或没有复用价值的事实。 + - summary 应命名抽象、解释它为什么重要，并指向支持证据；不要只是摘抄或总结笔记。 + - 如果 changed material 没有教导任何可复用抽象，返回空 `units` list；topic 候选仍然可以非空。 + + ## Bucket 规则 + + - **procedure**：如何做某事；工作流、runbook、配方、方法。 + - **personal**：用户/团队/项目特定的身份、偏好、约定、约束、避让点。 + - **wiki**：通用知识、原则、作为先例的决策、观察。 + + 横跨两个 bucket 时，按 center of gravity 选择，也就是未来读者会从哪里搜索： + - “用户偏好小 PR” -> personal。 + - “小 PR 更容易 review” -> wiki。 + - “如何拆分大 PR 的步骤” -> procedure。 + + ## Topic 规则 + + - 输出用户可能真正关心的 topic，而不是泛泛的标签。 + - 每个 topic 必须包含 title、reason、evidence、keywords、paths。 + - paths 只能使用 changed_paths 中出现的值。 + - 优先选择具体、反复出现、可行动的兴趣，而不是宽泛类别。 + + ## 工具边界 + + 只有当 inline wikilink 会实质影响抽取时，才可以使用 read。 + + ## 输出格式 + + 只返回一个 YAML 或 JSON object，结构必须严格如下： + units: + - name: <短名称> + bucket: procedure|personal|wiki + summary: <基于证据的可复用抽象> + paths: [, ...] + topics: + - title: <具体的用户兴趣 topic> + reason: <为什么重要> + evidence: <简短证据指针> + keywords: [<关键词>, ...] + paths: [, ...] + +extract_user_message: | + date: {date} + hint: {hint} + + changed_paths: + {changed_paths_json} + + # Changed material + + {material_blob} + + Extract merged memory units and daily interest topic candidates. + +extract_user_message_zh: | + 日期：{date} + 提示：{hint} + + changed_paths: + {changed_paths_json} + + # 变化内容 + + {material_blob} + + 抽取合并后的记忆 units 和 daily interest topic 候选。 diff --git a/reme4/steps/evolve/dream/finish.py b/reme4/steps/evolve/dream/finish.py new file mode 100644 index 00000000..c0a30847 --- /dev/null +++ b/reme4/steps/evolve/dream/finish.py @@ -0,0 +1,67 @@ +"""Dream catalog persistence step.""" + +from pathlib import Path + +from ...base_step import BaseStep +from ....components import R +from ....schema import FileNode +from .schema import DreamState +from .utils import state_from_context, store_state, vault_dir + + +@R.register("dream_finish_step") +class DreamFinishStep(BaseStep): + """Persist dream catalog and render final auto-dream response.""" + + def __init__(self, persist: bool = True, **kwargs): + super().__init__(**kwargs) + self.persist = persist + + async def execute(self): + assert self.context is not None + state = state_from_context(self) + vault = Path(state.vault).resolve() if state.vault else vault_dir(self) + if self.file_catalog is None: + raise RuntimeError("dream_finish_step requires file_catalog") + + checkpoint = [p for p in state.changed_paths if p not in set(state.failed_paths)] + upsert_paths = checkpoint + [p for p in [state.interests_path, f"{state.daily_dir}/{state.date}.md"] if p] + upserts = self._nodes(vault, upsert_paths) + if upserts: + await self.file_catalog.upsert(upserts) + if self.persist and (upserts or state.deleted_paths): + await self.file_catalog.dump() + + state.checkpoint_paths = [n.path for n in upserts if n.path in checkpoint] + state.summary = render_summary(state) + store_state(self, state) + self.context.response.success = not state.failed_units and not state.errors + self.context.response.answer = state.summary + return self.context.response + + @staticmethod + def _nodes(vault: Path, paths: list[str]) -> list[FileNode]: + out: list[FileNode] = [] + for rel in paths: + try: + out.append(FileNode(path=rel, st_mtime=(vault / rel).stat().st_mtime)) + except OSError: + continue + return out + + +def render_summary(state: DreamState) -> str: + """Render summary.""" + lines = [ + f"[AutoDream] date={state.date} scanned={state.files_scanned} changed={state.files_changed} " + f"unchanged={state.files_unchanged} deleted={state.files_deleted}", + f" - extract: {len(state.units)} unit(s), {len(state.topics)} topic(s)", + f" - integrate: {len(state.integrate_results)} ok, {len(state.failed_units)} failed", + f" - topics: {state.topics_written} written" + (f" to {state.interests_path}" if state.interests_path else ""), + f" - catalog: checkpointed {len(state.checkpoint_paths)} changed path(s)", + ] + if state.failed_paths: + lines.append(f" - failed paths: {', '.join(state.failed_paths)}") + if state.errors: + lines.append(f" - errors: {'; '.join(state.errors)}") + return "\n".join(lines) diff --git a/reme4/steps/evolve/dream/integrate.py b/reme4/steps/evolve/dream/integrate.py new file mode 100644 index 00000000..b79c37e1 --- /dev/null +++ b/reme4/steps/evolve/dream/integrate.py @@ -0,0 +1,86 @@ +"""Dream unit integration step.""" + +import json +from pathlib import Path + +from ...base_step import BaseStep +from ....components import R +from .schema import BUCKETS, IntegrateOutcome +from .utils import llm_available, pack_paths, parse_structured_reply, state_from_context, store_state, vault_dir + +_TOOLS = ("node_search", "read", "frontmatter_read", "write", "edit", "frontmatter_update") + + +@R.register("dream_integrate_step") +class DreamIntegrateStep(BaseStep): + """Integrate each extracted unit into digest memory.""" + + async def execute(self): + assert self.context is not None + state = state_from_context(self) + if not state.units: + return self._finish(state, True, "No dream units to integrate") + if not llm_available(self): + err = "no llm configured; dream integrate requires an LLM" + state.errors.append(err) + state.failed_units = state.units + state.failed_paths = sorted({p for u in state.units for p in u.get("paths", [])}) + return self._finish(state, False, err) + + vault = Path(state.vault).resolve() if state.vault else vault_dir(self) + digest_dir = self.config_value("digest_dir") + for i, unit in enumerate(state.units, start=1): + await self._integrate_one(state, unit, i, vault, digest_dir) + state.failed_paths = sorted(set(state.failed_paths)) + answer = f"Integrated {len(state.integrate_results)} unit(s); failed {len(state.failed_units)} unit(s)" + return self._finish(state, not state.failed_units, answer) + + async def _integrate_one(self, state, unit: dict, index: int, vault: Path, digest_dir: str) -> None: + bucket = unit.get("bucket") if unit.get("bucket") in BUCKETS else "wiki" + paths = [str(p) for p in unit.get("paths", [])] + try: + result = await self.agent_wrapper.reply( + self.prompt_format( + "integrate_user_message", + hint=state.hint or "(none)", + unit_name=unit.get("name", ""), + unit_bucket=bucket, + unit_summary=unit.get("summary", ""), + unit_paths_json=json.dumps(paths, ensure_ascii=False, indent=2), + material_blob=pack_paths(vault, paths), + ), + system_prompt=self.prompt_format( + f"integrate_system_prompt_{bucket}", + vault_dir=str(vault), + digest_dir=digest_dir, + bucket=bucket, + ), + job_tools=list(_TOOLS), + ) + outcome = IntegrateOutcome.model_validate(parse_structured_reply(str(result.get("result") or ""))) + except Exception as e: # noqa: BLE001 + error = f"{type(e).__name__}: {e}" + self.logger.error(f"[{self.name}] unit {index}/{len(state.units)} failed: {error}") + state.failed_units.append({**unit, "error": error}) + state.failed_paths.extend(path for path in paths if path not in state.failed_paths) + return + + state.integrate_results.append( + { + "unit": unit.get("name", ""), + "bucket": bucket, + "paths": paths, + "action": outcome.action, + "target_path": outcome.target_path, + "note": outcome.note, + }, + ) + (state.nodes_created if outcome.action == "CREATE" else state.nodes_updated).append(outcome.target_path) + + def _finish(self, state, success: bool, answer: str): + assert self.context is not None + state.summary = answer + store_state(self, state) + self.context.response.success = success + self.context.response.answer = answer + return self.context.response diff --git a/reme4/steps/evolve/dream/integrate.yaml b/reme4/steps/evolve/dream/integrate.yaml new file mode 100644 index 00000000..b72816e6 --- /dev/null +++ b/reme4/steps/evolve/dream/integrate.yaml @@ -0,0 +1,510 @@ +integrate_system_prompt_procedure: | + You are the dream unit integration agent for the procedure bucket. The unit + is a how-to-do-X abstraction: steps, methods, recipes, runbooks, executable + patterns. Integrate exactly one unit into digest memory. Unit -> digest node + is 1:1; no SKIP outcome because the incoming unit has already been selected + as worth remembering. + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest is the abstract memory layer + + Digest is not a faithful copy of the material. Details stay in the daily / + resource files; digest holds the reusable procedure a future agent should + recall later. + + - Body is short and abstract, usually 50-200 words unless the procedure + genuinely needs more. + - If the draft starts copying paragraphs or narrating the event, it belongs + in the source material, not digest. + - Provenance edges carry details: cite every relevant unit_paths entry with + `derived_from:: [[]]`. + - Digest-to-digest wikilinks carry the conceptual graph. + + ## Procedure Body Shape + + A runbook, not a recap: + + - Trigger / when to use: one line. + - Steps: numbered or terse bullets; each step is verb-led. + - Pre-conditions / inputs: short list, not prose. + - Failure modes / caveats: brief. + - `derived_from:: [[]]`: at least one, and normally every + relevant path in unit_paths. Plain-prose provenance does NOT count; only + wikilinks survive future updates. + + ## Workflow + + 1. Recall first with node_search across digest buckets. Use queries that + fit the unit's trigger, verbs, nouns, synonyms, and likely failure modes. + Use `limit=20-30` for broad coverage; one call usually suffices, but issue + more if the unit spans multiple concept dimensions. Recall feeds both + dedup and synapse decisions. + 2. Use read/frontmatter_read only for likely matches. Internally classify + each recalled node: + - same_abstraction: same trigger and substantially overlapping steps; + this is the UPDATE target. + - related: adjacent procedure, sub-step, prerequisite, failure mode, + concept, or preference worth linking in the body. + - unrelated: ignore. + 3. Choose exactly one action: + - CREATE: no same_abstraction hit; write `{digest_dir}/procedure/.md`. + - CORROBORATE: same procedure observed again; append derived_from and + optionally strengthen wording. + - REFINE: new pre-condition, edge case, failure mode, scope, or step; + expand the relevant span or slot the step into the right position. + - CORRECT: wrong order, missing critical step, bad outcome, or conflict; + tighten or annotate inline with `> note: contradicted by [[]] - `. + 4. Weave related digest nodes into the body as wikilinks on both CREATE and + UPDATE. UPDATE must be additive: never remove existing wikilinks or + derived_from entries. Default to weaving more, not less; this is the only + chance to attach recalled related nodes. + + ## Wikilink Graph + + - Source provenance links point from digest back to material: + `derived_from:: [[daily//.md]]` or + `derived_from:: [[resource/]]`. + - Procedure nodes may link to any digest bucket: + `[[{digest_dir}/procedure/.md]]`, + `[[{digest_dir}/personal/.md]]`, or + `[[{digest_dir}/wiki/.md]]`. + - Useful predicates include `derived_from::`, `relates_to::`, + `depends_on::`, and `blocks_on::`; predicates are open and live outside + the brackets. + - Wikilinks must be full vault-relative paths with `.md`. + - CREATE writes inside `{digest_dir}/procedure/`. UPDATE may target any + bucket if recall legitimately found the same abstraction. + - Use `edit` for body-only changes, and make edits only-add whenever possible: + never remove wikilinks the old span contained. + - Use `frontmatter_update` for frontmatter changes such as description or + `kind: procedure`. + - One target per session. Never edit other nodes sideways. + + Return only one YAML or JSON object: + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: + note: + +integrate_system_prompt_procedure_zh: | + 你是 dream 的 procedure bucket unit 整合 agent。这个 unit 是 how-to-do-X 抽象：步骤、方法、 + recipe、runbook、可执行模式。把且只把一个 unit 整合进 digest memory。Unit -> digest node 是 1:1； + 没有 SKIP，因为传入的 unit 已经被判定为值得记忆。 + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest 是抽象记忆层 + + Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的可复用流程。 + + - 正文短且抽象，通常 50-200 words；只有流程本身确实需要时才更长。 + - 如果草稿开始复制段落或叙述事件，说明细节放错层了。 + - Provenance edge 承载细节：用 `derived_from:: [[]]` 引用 unit_paths 中每个相关来源。 + - Digest 之间的 wikilink 承载概念图。 + + ## Procedure 正文形态 + + 写 runbook，不写 recap： + + - Trigger / when to use：一行。 + - Steps：编号或短 bullet；每步以动词开头。 + - Pre-conditions / inputs：短列表，不写长 prose。 + - Failure modes / caveats：简短。 + - `derived_from:: [[]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。 + 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + + ## 工作流 + + 1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖 unit 的触发条件、动词、名词、同义词和可能的 + failure mode；使用 `limit=20-30` 做较宽召回。通常一次足够；如果 unit 跨多个概念维度，可以多次召回。 + Recall 同时服务 dedup 和 synapse。 + 2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类： + - same_abstraction：触发条件相同，步骤实质重叠；这是 UPDATE 目标。 + - related：相邻流程、子步骤、前置条件、失败模式、概念或偏好，值得写入正文 wikilink。 + - unrelated：忽略。 + 3. 选择且只选择一个 action： + - CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/procedure/.md`。 + - CORROBORATE：同一流程再次出现；追加 derived_from，可选强化措辞。 + - REFINE：新增前置条件、边界情况、失败模式、适用范围或步骤；扩展相关段落或把步骤插到正确位置。 + - CORRECT：顺序错误、缺关键步骤、结果不好或有冲突；收紧表述，或用 + `> note: contradicted by [[]] - ` 内联标注。 + 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 + wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + + ## Wikilink 图 + + - 来源 provenance 从 digest 指回材料： + `derived_from:: [[daily//.md]]` 或 `derived_from:: [[resource/]]`。 + - Procedure 节点可以链接任意 digest bucket： + `[[{digest_dir}/procedure/.md]]`、`[[{digest_dir}/personal/.md]]`、 + `[[{digest_dir}/wiki/.md]]`。 + - 常用 predicate：`derived_from::`、`relates_to::`、`depends_on::`、`blocks_on::`； + predicate 词表开放，写在方括号外。 + - Wikilink 必须是带 `.md` 的 vault-relative path。 + - CREATE 写入 `{digest_dir}/procedure/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。 + - `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。 + - `frontmatter_update` 才能改 frontmatter，例如 description 或 `kind: procedure`。 + - 每次 session 只处理一个 target。不要顺手编辑其他节点。 + + 只返回一个 YAML 或 JSON object： + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: <写入或编辑的 digest path> + note: <简短落地总结> + +integrate_system_prompt_personal: | + You are the dream unit integration agent for the personal bucket. The unit is + user/team/project-specific: identity, preference, convention, avoid-rule, + collaboration style, or constraint. Integrate exactly one unit into digest + memory. Unit -> digest node is 1:1; no SKIP outcome because the incoming + unit has already been selected as worth remembering. + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest is the abstract memory layer + + Digest is not a faithful copy of the material. Details stay in the daily / + resource files; digest holds the durable rule, identity, convention, or + preference a future agent should recall later. + + - Body is short and operational, usually 50-200 words. + - Do not narrate what the user said in detail; cite source material instead. + - Provenance edges carry details: cite every relevant unit_paths entry with + `derived_from:: [[]]`. + - Digest-to-digest wikilinks carry the conceptual graph. + + ## Personal Body Shape + + A short rule of engagement, not a biography: + + - Rule / fact: one sentence stating the preference, convention, identity + claim, constraint, or avoid-rule. + - `Why:` reason or context that helps judge edge cases. + - `How to apply:` contexts, tasks, boundaries, or exceptions. + - Do not invent exceptions or soften a hard preference unless the source + material explicitly supports that exception. + - `derived_from:: [[]]`: at least one, and normally every + relevant path in unit_paths. Plain-prose provenance does NOT count; only + wikilinks survive future updates. + + For preferences, prefer one node per preference rather than one large person + node; that is the granularity downstream search will hit. + + ## Workflow + + 1. Recall first with node_search across digest buckets. Use user/team/project + names plus rule, preference, convention, scope, and avoid-rule keywords. + Use `limit=20-30` for broad coverage; issue more calls if the rule has + multiple actor or scope dimensions. Recall feeds both dedup and synapse + decisions. + 2. Use read/frontmatter_read only for likely matches. Internally classify + each recalled node: + - same_abstraction: same actor scope and same governing rule; this is the + UPDATE target. + - related: adjacent rule, contrasting preference, identity node, workflow, + or concept worth linking in the body. + - unrelated: ignore. + 3. Choose exactly one action: + - CREATE: no same_abstraction hit; write `{digest_dir}/personal/.md`. + - CORROBORATE: the rule is reaffirmed; append derived_from and optionally + strengthen confidence. + - REFINE: scope, condition, exception, or example changed; expand `How to apply:`. + - CORRECT: the user/team changed their mind or evidence conflicts; tighten + to what both old and new evidence support, or annotate inline with + `> note: contradicted by [[]] - `. + 4. Weave related digest nodes into the body as wikilinks on both CREATE and + UPDATE. UPDATE must be additive: never remove existing wikilinks or + derived_from entries. Default to weaving more, not less; this is the only + chance to attach recalled related nodes. + + ## Wikilink Graph + + - Source provenance links point from digest back to material: + `derived_from:: [[daily//.md]]` or + `derived_from:: [[resource/]]`. + - Personal nodes may link to any digest bucket: + `[[{digest_dir}/personal/.md]]`, + `[[{digest_dir}/procedure/.md]]`, or + `[[{digest_dir}/wiki/.md]]`. + - Useful predicates include `derived_from::`, `applies_to::`, + `relates_to::`, `depends_on::`, and `contradicts::`; predicates are open + and live outside the brackets. + - Wikilinks must be full vault-relative paths with `.md`. + - CREATE writes inside `{digest_dir}/personal/`. UPDATE may target any bucket + if recall legitimately found the same abstraction. + - Use `edit` for body-only changes, and make edits only-add whenever possible: + never remove wikilinks the old span contained. + - Use `frontmatter_update` for frontmatter changes such as description or + `kind: preference`. + - One target per session. Never edit other nodes sideways. + + Return only one YAML or JSON object: + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: + note: + +integrate_system_prompt_personal_zh: | + 你是 dream 的 personal bucket unit 整合 agent。这个 unit 是用户/团队/项目特定内容：身份、 + 偏好、约定、avoid-rule、协作风格或约束。把且只把一个 unit 整合进 digest memory。Unit -> digest node + 是 1:1；没有 SKIP，因为传入的 unit 已经被判定为值得记忆。 + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest 是抽象记忆层 + + Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的长期规则、身份、 + 约定或偏好。 + + - 正文短且可操作，通常 50-200 words。 + - 不要详细复述用户说了什么；用来源材料承载细节。 + - Provenance edge 承载细节：用 `derived_from:: [[]]` 引用 unit_paths 中每个相关来源。 + - Digest 之间的 wikilink 承载概念图。 + + ## Personal 正文形态 + + 写短规则，不写 biography： + + - Rule / fact：一句话说明偏好、约定、身份事实、约束或 avoid-rule。 + - `Why:` 原因或上下文，帮助未来判断边界情况。 + - `How to apply:` 适用上下文、任务、边界或例外。 + - 不要凭空添加例外，也不要软化明确偏好；只有来源材料明确支持时才写例外。 + - `derived_from:: [[]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。 + 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + + 偏好类内容优先一条偏好一个 node，而不是一个人一个大 node；这是下游搜索更容易命中的粒度。 + + ## 工作流 + + 1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖用户/团队/项目名称、规则、偏好、约定、适用范围、 + avoid-rule 关键词；使用 `limit=20-30` 做较宽召回。如果规则有多个 actor 或 scope 维度，可以多次召回。 + Recall 同时服务 dedup 和 synapse。 + 2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类： + - same_abstraction：同一 actor scope + 同一 governing rule；这是 UPDATE 目标。 + - related：相邻规则、相反偏好、身份节点、工作流或概念，值得写入正文 wikilink。 + - unrelated：忽略。 + 3. 选择且只选择一个 action： + - CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/personal/.md`。 + - CORROBORATE：规则被再次确认；追加 derived_from，可选强化置信度。 + - REFINE：scope、条件、例外或例子变化；扩展 `How to apply:`。 + - CORRECT：用户/团队改变主意或证据冲突；收紧到新旧证据都支持的表述，或用 + `> note: contradicted by [[]] - ` 内联标注。 + 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 + wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + + ## Wikilink 图 + + - 来源 provenance 从 digest 指回材料： + `derived_from:: [[daily//.md]]` 或 `derived_from:: [[resource/]]`。 + - Personal 节点可以链接任意 digest bucket： + `[[{digest_dir}/personal/.md]]`、`[[{digest_dir}/procedure/.md]]`、 + `[[{digest_dir}/wiki/.md]]`。 + - 常用 predicate：`derived_from::`、`applies_to::`、`relates_to::`、`depends_on::`、 + `contradicts::`；predicate 词表开放，写在方括号外。 + - Wikilink 必须是带 `.md` 的 vault-relative path。 + - CREATE 写入 `{digest_dir}/personal/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。 + - `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。 + - `frontmatter_update` 才能改 frontmatter，例如 description 或 `kind: preference`。 + - 每次 session 只处理一个 target。不要顺手编辑其他节点。 + + 只返回一个 YAML 或 JSON object： + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: <写入或编辑的 digest path> + note: <简短落地总结> + +integrate_system_prompt_wiki: | + You are the dream unit integration agent for the wiki bucket. The unit is + general knowledge: definition, principle, observation, decision-as-precedent, + factual claim, or mental model. Wiki is also the catch-all when nothing more + specific fits. Integrate exactly one unit into digest memory. Unit -> digest + node is 1:1; no SKIP outcome because the incoming unit has already been + selected as worth remembering. + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest is the abstract memory layer + + Digest is not a faithful copy of the material. Details stay in the daily / + resource files; digest holds the definition, principle, observation, or + precedent a future agent should recall later. + + - Body is short and abstract, usually 50-200 words unless the concept + genuinely needs more. + - If the draft starts copying paragraphs or narrating the event, it belongs + in the source material, not digest. + - Provenance edges carry details: cite every relevant unit_paths entry with + `derived_from:: [[]]`. + - Digest-to-digest wikilinks carry the conceptual graph. + + ## Wiki Body Shape + + Encyclopedia-flavored, not narrative: + + - First line: one-sentence definition or claim. + - Body: short paragraphs or tight bullets with properties, sub-claims, + distinctions, and one-line examples. + - Relations: typed wikilinks where the relation has semantic weight; most + cross-node links can stay bare. + - `derived_from:: [[]]`: at least one, and normally every + relevant path in unit_paths. Plain-prose provenance does NOT count; only + wikilinks survive future updates. + + ## Workflow + + 1. Recall first with node_search across digest buckets. Use noun phrases, + synonyms, abbreviations, super/sub-concepts, and contrasting terms. Use + `limit=20-30` for broad coverage; issue more calls when the abstraction + has multiple aspects worth querying separately. Recall feeds both dedup + and synapse decisions. + 2. Use read/frontmatter_read only for likely matches. Internally classify + each recalled node: + - same_abstraction: same definition, principle, observation, or precedent; + this is the UPDATE target. + - related: adjacent concept, contrast, super/sub-concept, instance, + procedure, or preference worth linking in the body. + - unrelated: ignore. + 3. Choose exactly one action: + - CREATE: no same_abstraction hit; write `{digest_dir}/wiki/.md`. + - CORROBORATE: the principle is reaffirmed by a new instance; append + derived_from and optionally strengthen wording. + - REFINE: nuance, scope, edge case, or framing changes; tighten the + relevant span. Body grows in precision, not detail volume. + - CORRECT: factual contradiction or overstatement; tighten to the narrower + supported form, or annotate inline with + `> note: contradicted by [[]] - `. + 4. Weave related digest nodes into the body as wikilinks on both CREATE and + UPDATE. UPDATE must be additive: never remove existing wikilinks or + derived_from entries. Default to weaving more, not less; this is the only + chance to attach recalled related nodes. + + ## Wikilink Graph + + - Source provenance links point from digest back to material: + `derived_from:: [[daily//.md]]` or + `derived_from:: [[resource/]]`. + - Wiki nodes may link to any digest bucket: + `[[{digest_dir}/wiki/.md]]`, + `[[{digest_dir}/procedure/.md]]`, or + `[[{digest_dir}/personal/.md]]`. + - Useful predicates include `derived_from::`, `is_a::`, `extends::`, + `depends_on::`, `relates_to::`, and `contradicts::`; predicates are open + and live outside the brackets. Most cross-node links can stay bare unless + the relation has clear semantic weight. + - Wikilinks must be full vault-relative paths with `.md`. + - CREATE writes inside `{digest_dir}/wiki/`. UPDATE may target any bucket if + recall legitimately found the same abstraction. + - Use `edit` for body-only changes, and make edits only-add whenever possible: + never remove wikilinks the old span contained. + - Use `frontmatter_update` for frontmatter changes such as description, + `kind: concept`, or `kind: observation`. + - One target per session. Never edit other nodes sideways. + + Return only one YAML or JSON object: + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: + note: + +integrate_system_prompt_wiki_zh: | + 你是 dream 的 wiki bucket unit 整合 agent。这个 unit 是通用知识：定义、原则、观察、作为先例的决策、 + 事实主张或 mental model。Wiki 也是无法更具体分类时的兜底。把且只把一个 unit 整合进 digest memory。 + Unit -> digest node 是 1:1；没有 SKIP，因为传入的 unit 已经被判定为值得记忆。 + + vault_dir: {vault_dir} + digest_dir: {digest_dir} + + ## Digest 是抽象记忆层 + + Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的定义、原则、 + 观察或先例。 + + - 正文短且抽象，通常 50-200 words；只有概念本身确实需要时才更长。 + - 如果草稿开始复制段落或叙述事件，说明细节放错层了。 + - Provenance edge 承载细节：用 `derived_from:: [[]]` 引用 unit_paths 中每个相关来源。 + - Digest 之间的 wikilink 承载概念图。 + + ## Wiki 正文形态 + + 写 encyclopedia 风格，不写 narrative： + + - First line：一句话定义或主张。 + - Body：短段落或紧凑 bullets，写属性、子主张、区分和一行例子。 + - Relations：有明确语义重量时使用 typed wikilink；大多数 cross-node link 可以裸写。 + - `derived_from:: [[]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。 + 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + + ## 工作流 + + 1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖名词短语、同义词、缩写、上位/下位概念和对比词； + 使用 `limit=20-30` 做较宽召回。如果抽象有多个值得分别查询的 aspect，可以多次召回。Recall 同时服务 + dedup 和 synapse。 + 2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类： + - same_abstraction：同一定义、原则、观察或先例；这是 UPDATE 目标。 + - related：相邻概念、对比概念、上位/下位概念、实例、流程或偏好，值得写入正文 wikilink。 + - unrelated：忽略。 + 3. 选择且只选择一个 action： + - CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/wiki/.md`。 + - CORROBORATE：原则被新实例再次确认；追加 derived_from，可选强化措辞。 + - REFINE：nuance、scope、edge case 或 framing 改变；收紧相关段落。正文增长的是精度，不是细节量。 + - CORRECT：事实冲突或过度概括；收紧到更窄且有支持的表述，或用 + `> note: contradicted by [[]] - ` 内联标注。 + 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 + wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + + ## Wikilink 图 + + - 来源 provenance 从 digest 指回材料： + `derived_from:: [[daily//.md]]` 或 `derived_from:: [[resource/]]`。 + - Wiki 节点可以链接任意 digest bucket： + `[[{digest_dir}/wiki/.md]]`、`[[{digest_dir}/procedure/.md]]`、 + `[[{digest_dir}/personal/.md]]`。 + - 常用 predicate：`derived_from::`、`is_a::`、`extends::`、`depends_on::`、 + `relates_to::`、`contradicts::`；predicate 词表开放，写在方括号外。有明确语义重量时才加 + predicate，大多数 cross-node link 可以裸写。 + - Wikilink 必须是带 `.md` 的 vault-relative path。 + - CREATE 写入 `{digest_dir}/wiki/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。 + - `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。 + - `frontmatter_update` 才能改 frontmatter，例如 description、`kind: concept` 或 `kind: observation`。 + - 每次 session 只处理一个 target。不要顺手编辑其他节点。 + + 只返回一个 YAML 或 JSON object： + action: CREATE|CORROBORATE|REFINE|CORRECT + target_path: <写入或编辑的 digest path> + note: <简短落地总结> + +integrate_user_message: | + hint: {hint} + + unit_name: {unit_name} + unit_bucket: {unit_bucket} + unit_summary: {unit_summary} + + unit_paths: + {unit_paths_json} + + # Evidence + + {material_blob} + + Integrate this single unit into digest memory. Cite every relevant unit_paths + entry with `derived_from:: [[]]`, recall related digest nodes, and weave + useful digest wikilinks into the target node. + +integrate_user_message_zh: | + 提示：{hint} + + unit_name: {unit_name} + unit_bucket: {unit_bucket} + unit_summary: {unit_summary} + + unit_paths: + {unit_paths_json} + + # 证据 + + {material_blob} + + 将这个单独的 unit 整合进 digest memory。用 `derived_from:: [[]]` 引用 unit_paths 中每个相关来源， + 召回相关 digest 节点，并把有用的 digest wikilink 织入目标节点。 diff --git a/reme4/steps/evolve/dream/proactive.py b/reme4/steps/evolve/dream/proactive.py new file mode 100644 index 00000000..7cd2541f --- /dev/null +++ b/reme4/steps/evolve/dream/proactive.py @@ -0,0 +1,43 @@ +"""Read daily interests.yaml for proactive use.""" + +from ...base_step import BaseStep +from ....components import R +from .schema import ProactiveResult +from .utils import load_yaml_topics, today, vault_dir + + +@R.register("proactive_step") +class ProactiveStep(BaseStep): + """Read ``daily//interests.yaml``.""" + + def __init__(self, include_content: bool = True, **kwargs): + super().__init__(**kwargs) + self.include_content = include_content + + async def execute(self): + assert self.context is not None + day = today(self, str(self.context.get("date", "") or "")) + include_content = bool(self.context.get("include_content", self.include_content)) + daily = self.config_value("daily_dir") + rel_path, abs_path = f"{daily}/{day}/interests.yaml", vault_dir(self) / daily / day / "interests.yaml" + result = ProactiveResult(date=day, path=rel_path) + + if not abs_path.is_file(): + result.skipped, result.summary = True, f"Skipped: interests file not found at {rel_path}" + return self._finish(True, result) + try: + result.content = abs_path.read_text(encoding="utf-8") if include_content else "" + result.topics = load_yaml_topics(abs_path) + except Exception as e: # noqa: BLE001 + result.error, result.summary = f"{type(e).__name__}: {e}", "" + return self._finish(False, result) + + result.summary = f"Read {len(result.topics)} proactive topic(s) from {rel_path}" + return self._finish(True, result) + + def _finish(self, success: bool, result: ProactiveResult): + assert self.context is not None + self.context.response.success = success + self.context.response.answer = result.summary if success else f"Error: {result.error}" + self.context.response.metadata.update(result.model_dump()) + return self.context.response diff --git a/reme4/steps/evolve/dream/schema.py b/reme4/steps/evolve/dream/schema.py new file mode 100644 index 00000000..324452e7 --- /dev/null +++ b/reme4/steps/evolve/dream/schema.py @@ -0,0 +1,92 @@ +"""Shared auto-dream schemas.""" + +from typing import Literal + +from pydantic import BaseModel, Field + +BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki") +Bucket = Literal["procedure", "personal", "wiki"] + + +class DreamUnit(BaseModel): + """One cross-file memory unit emitted by global extract.""" + + name: str = Field(description="Short kebab-case handle for the abstraction.") + bucket: str = Field(description="procedure, personal, or wiki; unknown values route to wiki.") + summary: str = Field(description="Grounded abstraction summary with evidence pointers.") + paths: list[str] = Field(default_factory=list, description="Vault-relative source paths.") + + +class DreamTopic(BaseModel): + """One topic candidate emitted by global extract.""" + + title: str = Field(description="Specific user-interest topic title.") + reason: str = Field(description="Why this topic may interest the user.") + evidence: str = Field(description="Grounded evidence pointer.") + keywords: list[str] = Field(default_factory=list, description="Keywords for de-duplication.") + paths: list[str] = Field(default_factory=list, description="Vault-relative source paths.") + + +class DreamExtractOutput(BaseModel): + """Structured output for ``dream_extract_step``.""" + + units: list[DreamUnit] = Field(default_factory=list) + topics: list[DreamTopic] = Field(default_factory=list) + + +class IntegrateOutcome(BaseModel): + """Structured output for one unit integration.""" + + action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field(description="Write decision.") + target_path: str = Field(description="Digest path written or edited.") + note: str = Field(default="", description="Short summary of what landed.") + + +class TopicSelectionOutput(BaseModel): + """Structured output for daily topic selection.""" + + topics: list[DreamTopic] = Field(default_factory=list) + + +class ProactiveResult(BaseModel): + """Result of reading daily interest topics.""" + + date: str = "" + path: str = "" + topics: list[dict] = Field(default_factory=list) + content: str = "" + skipped: bool = False + error: str = "" + summary: str = "" + + +class DreamState(BaseModel): + """Shared state passed across the four dream steps.""" + + date: str = "" + hint: str = "" + daily_dir: str = "" + vault: str = "" + files_scanned: int = 0 + files_unchanged: int = 0 + files_changed: int = 0 + files_deleted: int = 0 + changed_paths: list[str] = Field(default_factory=list) + unchanged_paths: list[str] = Field(default_factory=list) + deleted_paths: list[str] = Field(default_factory=list) + existing: dict[str, float] = Field(default_factory=dict) + indexed: dict[str, float] = Field(default_factory=dict) + units: list[dict] = Field(default_factory=list) + topics: list[dict] = Field(default_factory=list) + extract_summary: str = "" + integrate_results: list[dict] = Field(default_factory=list) + nodes_created: list[str] = Field(default_factory=list) + nodes_updated: list[str] = Field(default_factory=list) + failed_units: list[dict] = Field(default_factory=list) + failed_paths: list[str] = Field(default_factory=list) + interests_path: str = "" + topics_written: int = 0 + topic_error: str = "" + checkpoint_paths: list[str] = Field(default_factory=list) + errors: list[str] = Field(default_factory=list) + summary: str = "" diff --git a/reme4/steps/evolve/dream/topics.py b/reme4/steps/evolve/dream/topics.py new file mode 100644 index 00000000..6f1f9e2a --- /dev/null +++ b/reme4/steps/evolve/dream/topics.py @@ -0,0 +1,132 @@ +"""Daily interests.yaml step.""" + +import json +from pathlib import Path + +from ...base_step import BaseStep +from ...file_io import refresh_day_index +from ....components import R +from .utils import ( + load_yaml_topics, + llm_available, + normalize_topic, + parse_structured_reply, + previous_dates, + state_from_context, + store_state, + vault_dir, + write_yaml, +) + + +@R.register("dream_topics_step") +class DreamTopicsStep(BaseStep): + """Write ``daily//interests.yaml`` with same-day and recent de-dup.""" + + def __init__(self, topic_count: int = 3, topic_diversity_days: int = 7, **kwargs): + super().__init__(**kwargs) + self.topic_count = topic_count + self.topic_diversity_days = topic_diversity_days + + async def execute(self): + assert self.context is not None + state = state_from_context(self) + topic_count = int(self.context.get("topic_count", self.topic_count) or self.topic_count) + raw_days = self.context.get("topic_diversity_days", self.topic_diversity_days) + diversity_days = int(raw_days or self.topic_diversity_days) + vault = Path(state.vault).resolve() if state.vault else vault_dir(self) + rel_path = f"{state.daily_dir}/{state.date}/interests.yaml" + abs_path = vault / state.daily_dir / state.date / "interests.yaml" + same_day = load_yaml_topics(abs_path) + + if not state.topics: + state.interests_path = rel_path if abs_path.is_file() else "" + state.topics_written = len(same_day) + answer = f"Kept {len(same_day)} existing interest topic(s) at {rel_path}" + answer = answer if abs_path.is_file() else "Skipped interests.yaml write: no new topic candidates" + return self._finish(state, True, answer) + + recent = [ + topic + for day in previous_dates(state.date, diversity_days) + for topic in load_yaml_topics(vault / state.daily_dir / day / "interests.yaml") + ] + try: + topics, _used_llm = await self._select_topics(state, same_day, recent, topic_count, diversity_days) + payload = { + "date": state.date, + "topic_count": topic_count, + "diversity_days": diversity_days, + "topics": topics, + } + write_yaml(abs_path, payload) + await refresh_day_index(self.file_store, state.date, state.daily_dir) + state.interests_path, state.topics_written = rel_path, len(topics) + return self._finish(state, True, f"Wrote {len(topics)} interest topic(s) to {rel_path}") + except Exception as e: # noqa: BLE001 + state.topic_error = f"{type(e).__name__}: {e}" + state.errors.append(state.topic_error) + return self._finish(state, False, f"Error: {state.topic_error}") + + async def _select_topics(self, state, same_day: list[dict], recent: list[dict], count: int, days: int): + if not state.topics: + return self._dedupe([], same_day, recent, count), False + if not llm_available(self): + return self._dedupe(state.topics, same_day, recent, count), False + result = await self.agent_wrapper.reply( + self.prompt_format( + "topics_user_message", + date=state.date, + topic_count=count, + diversity_days=days, + candidates_json=json.dumps(state.topics, ensure_ascii=False, indent=2), + same_day_json=json.dumps(same_day, ensure_ascii=False, indent=2), + recent_topics_json=json.dumps(recent, ensure_ascii=False, indent=2), + ), + system_prompt=self.prompt_format("topics_system_prompt"), + ) + meta = parse_structured_reply(str(result.get("result") or "")) + selected = [self._clean_topic(t) for t in meta.get("topics") or []] + if not any(selected): + selected = state.topics + return self._dedupe(selected, same_day, recent, count), True + + @staticmethod + def _clean_topic(raw) -> dict: + if not isinstance(raw, dict): + return {} + title, reason = str(raw.get("title") or "").strip(), str(raw.get("reason") or "").strip() + if not title or not reason: + return {} + keywords, paths = raw.get("keywords") or [], raw.get("paths") or [] + cleaned_keywords = [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else [] + cleaned_paths = [str(p).strip() for p in paths if str(p).strip()] if isinstance(paths, list) else [] + return { + "title": title, + "reason": reason, + "evidence": str(raw.get("evidence") or "").strip(), + "keywords": cleaned_keywords, + "paths": cleaned_paths, + } + + @staticmethod + def _dedupe(topics: list[dict], same_day: list[dict], recent: list[dict], count: int) -> list[dict]: + recent_norm = {normalize_topic(t.get("title", "")) for t in recent} + seen = {normalize_topic(t.get("title", "")) for t in same_day} + out = list(same_day) + for topic in [t for t in topics if t]: + title_norm = normalize_topic(topic.get("title", "")) + if title_norm and title_norm not in seen and title_norm not in recent_norm: + seen.add(title_norm) + out.append(topic) + if len(out) >= count: + break + return out[:count] + + def _finish(self, state, success: bool, answer: str): + assert self.context is not None + state.summary = answer + store_state(self, state) + self.context.response.success = success + self.context.response.answer = answer + return self.context.response diff --git a/reme4/steps/evolve/dream/topics.yaml b/reme4/steps/evolve/dream/topics.yaml new file mode 100644 index 00000000..8e3115db --- /dev/null +++ b/reme4/steps/evolve/dream/topics.yaml @@ -0,0 +1,107 @@ +topics_system_prompt: | + You select final daily user-interest topics from dream candidates for + daily//interests.yaml. + + ## Goals + + - Preserve existing same-day topics unless they are clear duplicates. + - Avoid duplicates within today. + - Avoid topics already covered in recent interests.yaml files. + - Prefer concrete, recurring, or actionable interests over file summaries. + - Return no more than topic_count topics. + + ## Topic Quality + + A good topic is something the user may want surfaced again: a live area of + attention, a recurring concern, a research direction, a project thread, or a + practical follow-up. Avoid generic labels, daily log summaries, and topics + whose only value is restating a file name. + + ## Output Rules + + - Keep titles concise and specific. + - Keep reason grounded in candidate evidence. + - Keep keywords short and retrieval-friendly. + - Keep paths limited to the paths supplied by the candidates. + - Quote scalar strings that contain punctuation such as `:` or use block + scalars (`>`). Write paths as block lists, not flow lists, so + `daily//...` stays parseable. + + Return only one YAML or JSON object: + topics: + - title: + reason: + evidence: + keywords: [, ...] + paths: [, ...] + +topics_system_prompt_zh: | + 你负责从 dream 候选中选择最终 daily user-interest topics，用于写入 daily//interests.yaml。 + + ## 目标 + + - 保留同一天已有的 topics，除非它们明显重复。 + - 避免当天内部重复。 + - 避免重复最近 interests.yaml 已经覆盖过的 topics。 + - 优先选择具体、反复出现、可行动的兴趣，而不是文件摘要。 + - 返回数量不能超过 topic_count。 + + ## Topic 质量 + + 好的 topic 应该是用户未来可能希望再次看到的内容：持续关注的方向、反复出现的问题、研究方向、 + 项目线索或实际 follow-up。避免泛泛标签、daily log 摘要，以及只是在复述文件名的 topic。 + + ## 输出规则 + + - title 要简短、具体。 + - reason 要基于候选证据。 + - keywords 要短，方便检索。 + - paths 只能使用候选中提供的 paths。 + - 如果字符串包含 `:` 等标点，要加引号或使用 block scalar (`>`)。 + paths 使用 block list，不要用 flow list，避免 `daily//...` 解析失败。 + + 只返回一个 YAML 或 JSON object： + topics: + - title: <具体 topic> + reason: <为什么重要> + evidence: <简短证据指针> + keywords: [<关键词>, ...] + paths: [, ...] + +topics_user_message: | + date: {date} + topic_count: {topic_count} + diversity_days: {diversity_days} + + # Candidate topics + + {candidates_json} + + # Existing same-day interests.yaml topics + + {same_day_json} + + # Recent interests.yaml topics to avoid repeating + + {recent_topics_json} + + Select the final topics for daily interests.yaml. + +topics_user_message_zh: | + 日期：{date} + topic_count: {topic_count} + diversity_days: {diversity_days} + + # 候选 topics + + {candidates_json} + + # 同一天已有的 interests.yaml topics + + {same_day_json} + + # 需要避免重复的最近 interests.yaml topics + + {recent_topics_json} + + 选择最终要写入 daily interests.yaml 的 topics。 diff --git a/reme4/steps/evolve/dream/utils.py b/reme4/steps/evolve/dream/utils.py new file mode 100644 index 00000000..976cb28d --- /dev/null +++ b/reme4/steps/evolve/dream/utils.py @@ -0,0 +1,174 @@ +"""Shared auto-dream helpers.""" + +import datetime as dt +import re +from pathlib import Path + +import yaml + +from .._evolve import now +from ...base_step import BaseStep +from .schema import DreamState + + +def state_from_context(step: BaseStep) -> DreamState: + """Get dream state from context.""" + assert step.context is not None + raw = step.context.get("dream") or step.context.response.metadata.get("dream") or {} + state = DreamState.model_validate(raw) + if not state.daily_dir: + state.daily_dir = step.config_value("daily_dir") + return state + + +def store_state(step: BaseStep, state: DreamState) -> None: + """Store dream state in context.""" + assert step.context is not None + data = state.model_dump() + step.context["dream"] = data + step.context.response.metadata["dream"] = data + + +def vault_dir(step: BaseStep) -> Path: + """Get vault directory.""" + vr = getattr(step.file_store, "vault_path", None) + return Path(vr).resolve() if vr else Path.cwd().resolve() + + +def daily_dir(step: BaseStep) -> str: + """Get daily directory.""" + return step.config_value("daily_dir") + + +def today(step: BaseStep, explicit: str = "") -> str: + """Get today's date.""" + if explicit.strip(): + return explicit.strip() + tz = step.app_context.app_config.timezone if step.app_context is not None else None + return now(tz).strftime("%Y-%m-%d") + + +def llm_available(step: BaseStep) -> bool: + """Check if LLM is available.""" + try: + return step.as_llm is not None and step.agent_wrapper is not None + except Exception: + return False + + +def scan_day_files(vault: Path, day: str, daily: str, interests_name: str = "interests.yaml") -> list[str]: + """Scan day files.""" + out: list[str] = [] + day_index = vault / daily / f"{day}.md" + if day_index.is_file(): + out.append(str(day_index.relative_to(vault))) + daily_root = vault / daily / day + if daily_root.is_dir(): + out.extend(str(p.relative_to(vault)) for p in sorted(daily_root.rglob("*.md")) if p.is_file()) + return [p for p in out if p != f"{daily}/{day}/{interests_name}"] + + +def pack_paths(vault: Path, paths: list[str], *, limit_per_file: int = 60000) -> str: + """Pack paths into a single string.""" + blocks: list[str] = [] + for rel in paths: + target = vault / rel + if not target.is_file(): + blocks.append(f"### {rel}\n(file not found)\n") + continue + try: + text = target.read_text(encoding="utf-8") + except Exception as e: # noqa: BLE001 + blocks.append(f"### {rel}\n(error reading: {type(e).__name__}: {e})\n") + continue + suffix = "\n\n[truncated]\n" if len(text) > limit_per_file else "" + blocks.append(f"### {rel}\n{text[:limit_per_file]}{suffix}\n") + return "\n".join(blocks) + + +def clean_paths(raw_paths, allowed: set[str]) -> list[str]: + """Clean paths.""" + if not isinstance(raw_paths, list): + return [] + out: list[str] = [] + for item in raw_paths: + path = str(item or "").strip() + if path in allowed and path not in out: + out.append(path) + return out + + +def normalize_topic(text: str) -> str: + """Normalize topic.""" + return re.sub(r"[^a-z0-9\u4e00-\u9fff]+", " ", text.lower()).strip() + + +def previous_dates(day: str, n_days: int) -> list[str]: + """Get previous dates.""" + try: + base = dt.date.fromisoformat(day) + except ValueError: + return [] + return [(base - dt.timedelta(days=i)).isoformat() for i in range(1, max(n_days, 0) + 1)] + + +def load_yaml_topics(path: Path) -> list[dict]: + """Load YAML topics.""" + if not path.is_file(): + return [] + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except Exception: + return [] + topics = data.get("topics") if isinstance(data, dict) else None + if not isinstance(topics, list): + return [] + return [cleaned for t in topics if isinstance(t, dict) and (cleaned := clean_topic(t))] + + +def clean_topic(raw: dict) -> dict: + """Clean topic.""" + title, reason = str(raw.get("title") or "").strip(), str(raw.get("reason") or "").strip() + if not title or not reason: + return {} + keywords = raw.get("keywords") or [] + paths = raw.get("paths") or [] + return { + "title": title, + "reason": reason, + "evidence": str(raw.get("evidence") or "").strip(), + "keywords": [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else [], + "paths": [str(p).strip() for p in paths if str(p).strip()] if isinstance(paths, list) else [], + } + + +def write_yaml(path: Path, payload: dict) -> None: + """Write YAML.""" + path.parent.mkdir(parents=True, exist_ok=True) + rendered = yaml.safe_dump(payload, allow_unicode=True, sort_keys=False) + path.write_text(rendered if rendered.endswith("\n") else f"{rendered}\n", encoding="utf-8") + + +def parse_structured_reply(text: str) -> dict: + """Parse a JSON/YAML object from an agent reply, including fenced blocks.""" + candidates = [text.strip()] + candidates.extend(m.group(1).strip() for m in re.finditer(r"```(?:json|ya?ml)?\s*(.*?)```", text, re.S | re.I)) + for raw in candidates: + if not raw: + continue + try: + data = yaml.safe_load(raw) + except yaml.YAMLError: + data = _parse_scalar_mapping(raw) + if isinstance(data, dict) and data: + return data + return {} + + +def _parse_scalar_mapping(raw: str) -> dict: + """Parse a scalar mapping.""" + out: dict[str, str] = {} + for line in raw.splitlines(): + if match := re.match(r"^\s*(action|target_path|note)\s*:\s*(.+?)\s*$", line): + out[match.group(1)] = match.group(2).strip().strip("\"'") + return out diff --git a/reme4/steps/file_io/__init__.py b/reme4/steps/file_io/__init__.py index e69de29b..0ec24fe5 100644 --- a/reme4/steps/file_io/__init__.py +++ b/reme4/steps/file_io/__init__.py @@ -0,0 +1,38 @@ +"""File I/O step helpers.""" + +from ._daily_index import refresh_day_index, validate_session_id +from ._file_io import write_file_safe +from .daily_create import DailyCreateStep +from .daily_list import DailyListStep +from .daily_reindex import DailyReindexStep +from .delete import DeleteStep +from .edit import EditStep +from .frontmatter_delete import FrontmatterDeleteStep +from .frontmatter_read import FrontmatterReadStep +from .frontmatter_update import FrontmatterUpdateStep +from .list import ListStep +from .move import MoveStep +from .read import ReadStep +from .read_image import ReadImageStep +from .stat import StatStep +from .write import WriteStep + +__all__ = [ + "refresh_day_index", + "validate_session_id", + "write_file_safe", + "DailyCreateStep", + "DailyListStep", + "DailyReindexStep", + "DeleteStep", + "EditStep", + "FrontmatterDeleteStep", + "FrontmatterReadStep", + "FrontmatterUpdateStep", + "ListStep", + "MoveStep", + "ReadStep", + "ReadImageStep", + "StatStep", + "WriteStep", +] diff --git a/reme4/steps/file_io/_daily_index.py b/reme4/steps/file_io/_daily_index.py index f31677ee..fad57f54 100644 --- a/reme4/steps/file_io/_daily_index.py +++ b/reme4/steps/file_io/_daily_index.py @@ -4,7 +4,7 @@ from pathlib import Path import frontmatter -from ._path import validate_filename_component +from ._path import resolve_path, validate_filename_component from ...utils import get_logger logger = get_logger() @@ -49,15 +49,15 @@ def scan_notes(vault_dir: Path, date: str, daily_dir: str) -> list[dict]: {"session_id": str, "path": str, "metadata": dict} """ - date_dir = vault_dir / daily_dir / date + date_dir, err = resolve_path(vault_dir, f"{daily_dir}/{date}") + if err or date_dir is None: + logger.info(f"scan_notes skipped invalid daily path date={date!r} daily_dir={daily_dir!r} error={err!r}") + return [] if not date_dir.is_dir(): return [] out: list[dict] = [] - prefix = "session_agent_" - for md_path in sorted( - p for p in date_dir.iterdir() if p.is_file() and p.suffix == ".md" and p.stem.startswith(prefix) - ): - session_id = md_path.stem[len(prefix) :] + for md_path in sorted(p for p in date_dir.iterdir() if p.is_file() and p.suffix == ".md"): + session_id = md_path.stem try: post = frontmatter.loads(md_path.read_text(encoding="utf-8")) except Exception: @@ -79,7 +79,16 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict: """ vault_dir = Path(file_store.vault_path or ".").resolve() index_rel = f"{daily_dir}/{date}.md" - index_abs = vault_dir / index_rel + index_abs, err = resolve_path(vault_dir, index_rel) + if err or index_abs is None: + return { + "date": date, + "path": index_rel, + "notes": [], + "created": False, + "changed": False, + "error": err or "invalid path", + } notes = scan_notes(vault_dir, date, daily_dir) notes_payload = [{"path": n["path"], "session_id": n["session_id"], "metadata": n["metadata"]} for n in notes] @@ -90,6 +99,7 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict: "path": index_rel, "notes": notes_payload, "created": False, + "changed": False, } notes_block = _render_notes_block(notes) @@ -97,13 +107,15 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict: n = len(notes) fm = {"name": date, "description": "No notes today." if n == 0 else f"{n} note(s) today."} + existing_text = "" if index_abs.is_file(): - post = frontmatter.loads(index_abs.read_text(encoding="utf-8")) + existing_text = index_abs.read_text(encoding="utf-8") + post = frontmatter.loads(existing_text) new_body = _rebuild_body(post.content, notes_block) merged = dict(post.metadata or {}) - for key, value in fm.items(): - if not merged.get(key): - merged[key] = value + if not merged.get("name"): + merged["name"] = fm["name"] + merged["description"] = fm["description"] fm = merged was_created = False else: @@ -111,11 +123,15 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict: new_body = f"{_NOTES_OPEN}\n{notes_block}\n{_NOTES_CLOSE}\n" was_created = True out = frontmatter.Post(new_body, **fm) - index_abs.write_text(frontmatter.dumps(out), encoding="utf-8") + rendered = frontmatter.dumps(out) + changed = not index_abs.is_file() or existing_text != rendered + if changed: + index_abs.write_text(rendered, encoding="utf-8") return { "date": date, "path": index_rel, "notes": notes_payload, "created": was_created, + "changed": changed, } diff --git a/reme4/steps/file_io/_file_io.py b/reme4/steps/file_io/_file_io.py index fb53773c..5effccdd 100644 --- a/reme4/steps/file_io/_file_io.py +++ b/reme4/steps/file_io/_file_io.py @@ -113,6 +113,32 @@ async def read_file_safe(file_path, max_bytes: int = MAX_FILE_READ_BYTES) -> tup return _decode_known_file(data, Path(file_path).suffix) +async def read_file_lines_safe( + file_path, + start_line: int, + end_line: int | None, + *, + max_collect_bytes: int = DEFAULT_MAX_BYTES * 2, +) -> tuple[str, int, str]: + """Read a 1-based inclusive line range without loading the full file. + + Returns ``(text, total_lines, encoding)``. + """ + encoding = await detect_file_encoding(file_path) + lines: list[str] = [] + collected_bytes = 0 + total = 0 + async with aiofiles.open(str(file_path), "r", encoding=encoding, errors="replace") as f: + async for line in f: + total += 1 + if total >= start_line and (end_line is None or total <= end_line): + if collected_bytes < max_collect_bytes: + cleaned = line.rstrip("\n") + lines.append(cleaned) + collected_bytes += len(cleaned.encode(encoding, errors="replace")) + 1 + return "\n".join(lines), total, encoding + + async def detect_file_encoding(file_path, sniff_bytes: int = 8192) -> str: """Detect the encoding of an existing file so writes can preserve it.""" try: diff --git a/reme4/steps/file_io/_path.py b/reme4/steps/file_io/_path.py index 259e63a4..f2e9f671 100644 --- a/reme4/steps/file_io/_path.py +++ b/reme4/steps/file_io/_path.py @@ -39,10 +39,13 @@ _RESERVED_NAMES = { } +# pylint: disable=too-many-return-statements def validate_filename_component(name: str, *, kind: str = "filename") -> str | None: """Return an error message, or ``None`` when ``name`` is a safe filename component.""" if not name: return f"{kind} is required" + if name in (".", ".."): + return f"{kind} cannot be '.' or '..': {name!r}" if name != name.strip(): return f"{kind} cannot have leading or trailing whitespace: {name!r}" if _INVALID_CHARS.search(name): @@ -54,25 +57,44 @@ def validate_filename_component(name: str, *, kind: str = "filename") -> str | N return None -def resolve_path(vault_path: Path, raw: str) -> tuple[Path | None, str | None]: +def is_relative_to(path: Path, parent: Path) -> bool: + """Return True when ``path`` is equal to or nested under ``parent``.""" + try: + path.relative_to(parent) + return True + except ValueError: + return False + + +# pylint: disable=too-many-return-statements +def resolve_path( + vault_path: Path, + raw: str, + *, + allow_empty: bool = False, +) -> tuple[Path | None, str | None]: """Resolve a `path=` argument against ``vault_path``. Returns ``(abs_path, None)`` on success, or ``(None, error_message)`` on failure. """ if not raw or not str(raw).strip(): + if allow_empty: + return vault_path.resolve(), None return None, "`path` is required" s = str(raw).strip() p = Path(s) + if p.is_absolute(): + logger.info("absolute path detected, recommending relative paths") + return p.resolve(), None for part in p.parts: - if part == p.anchor: - continue err = validate_filename_component(part, kind="path component") if err: return None, err - if p.is_absolute(): - logger.info("absolute path detected, recommending relative paths") - return p, None - return vault_path / p, None + vault = vault_path.resolve() + target = (vault / p).resolve() + if not is_relative_to(target, vault): + return None, "`path` must stay inside the vault" + return target, None def gate_md(target: Path) -> tuple[Path, bool]: diff --git a/reme4/steps/file_io/daily_create.py b/reme4/steps/file_io/daily_create.py index cac75e8b..2315dd39 100644 --- a/reme4/steps/file_io/daily_create.py +++ b/reme4/steps/file_io/daily_create.py @@ -15,7 +15,7 @@ The caller fills the body via ``file_write`` / ``file_edit`` / does not accept a body. Inputs: - session_id (optional) — the note's session identifier (also the file stem); + session_id (optional) — the note's session identifier; empty string → day-level file date (optional, ``YYYY-MM-DD``; empty = today) @@ -30,6 +30,7 @@ import frontmatter from ._daily_index import refresh_day_index, validate_session_id from ._file_io import write_file_safe +from ._path import resolve_path from ..base_step import BaseStep from ...components import R from ...steps.evolve import now @@ -53,7 +54,7 @@ class DailyCreateStep(BaseStep): session_id = self.context.get("session_id", "") tz = self.app_context.app_config.timezone if self.app_context is not None else None day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") - daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" + daily_dir = self.config_value("daily_dir") return session_id, day, daily_dir @staticmethod @@ -86,13 +87,16 @@ class DailyCreateStep(BaseStep): if err: self._fail(err) return None - path_rel = f"{daily_dir}/{day}/session_agent_{session_id}.md" name = session_id + path_rel = f"{daily_dir}/{day}/{session_id}.md" else: path_rel = f"{daily_dir}/{day}.md" name = day - path_abs = (self.vault_path / path_rel).resolve() + path_abs, err = resolve_path(self.vault_path, path_rel) + if err or path_abs is None: + self._fail(err or "invalid path", date=day, session_id=session_id, path=path_rel) + return None try: created = await self._create_if_missing(path_abs, name) except Exception as e: # pylint: disable=broad-except diff --git a/reme4/steps/file_io/daily_list.py b/reme4/steps/file_io/daily_list.py index 02408a67..25516521 100644 --- a/reme4/steps/file_io/daily_list.py +++ b/reme4/steps/file_io/daily_list.py @@ -15,6 +15,7 @@ to today. from pathlib import Path from ._daily_index import scan_notes +from ._path import resolve_path from ..base_step import BaseStep from ...components import R from ...steps.evolve import now @@ -29,7 +30,7 @@ class DailyListStep(BaseStep): assert self.context is not None tz = self.app_context.app_config.timezone if self.app_context is not None else None day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") - daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" + daily_dir = self.config_value("daily_dir") vault_dir = Path(self.file_store.vault_path or ".").resolve() return day, daily_dir, vault_dir @@ -57,6 +58,13 @@ class DailyListStep(BaseStep): """Scan ``//`` and emit one projected record per note.""" assert self.context is not None day, daily_dir, vault_dir = self._collect_params() + _target_dir, err = resolve_path(vault_dir, f"{daily_dir}/{day}") + if err: + self.context.response.success = False + self.context.response.answer = f"Error: {err}" + self.context.response.metadata.update({"date": day, "error": err}) + self.logger.info(f"[{self.name}] date={day} error={err!r}") + return notes = [self._project(n) for n in scan_notes(vault_dir, day, daily_dir)] self.context.response.success = True lines = [self._format_note_line(n) for n in notes] diff --git a/reme4/steps/file_io/daily_reindex.py b/reme4/steps/file_io/daily_reindex.py index 6264fb13..7c3fa200 100644 --- a/reme4/steps/file_io/daily_reindex.py +++ b/reme4/steps/file_io/daily_reindex.py @@ -34,7 +34,7 @@ class DailyReindexStep(BaseStep): assert self.context is not None tz = self.app_context.app_config.timezone if self.app_context is not None else None day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") - daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" + daily_dir = self.config_value("daily_dir") return day, daily_dir def _apply_result(self, refreshed: dict) -> None: diff --git a/reme4/steps/file_io/delete.py b/reme4/steps/file_io/delete.py index abac1fea..a0d26ba6 100644 --- a/reme4/steps/file_io/delete.py +++ b/reme4/steps/file_io/delete.py @@ -24,6 +24,7 @@ citing prose, or accept dangling. import shutil from pathlib import Path +from ._path import resolve_path from ..base_step import BaseStep from ...components import R from ...utils.wikilink_handler import WikilinkHandler @@ -70,7 +71,9 @@ class DeleteStep(BaseStep): if not path: return {"path": path, "error": "not found"} vault_dir = Path(self.file_store.vault_path or ".").resolve() - target = (vault_dir / path).resolve() + target, err = resolve_path(vault_dir, path) + if err or target is None: + return {"path": path, "error": err or "invalid path"} if target.is_file(): inbound = await WikilinkHandler.find_inbound(self.file_store, target=path) target.unlink() diff --git a/reme4/steps/file_io/frontmatter_delete.py b/reme4/steps/file_io/frontmatter_delete.py index d1f8bb3a..d6b64e73 100644 --- a/reme4/steps/file_io/frontmatter_delete.py +++ b/reme4/steps/file_io/frontmatter_delete.py @@ -13,6 +13,8 @@ from pathlib import Path import frontmatter +from ._file_io import get_path_lock +from ._path import resolve_path from ..base_step import BaseStep from ...components import R @@ -30,31 +32,37 @@ class FrontmatterDeleteStep(BaseStep): keys = [keys] keys = list(keys) - target = (Path(self.file_store.vault_path or ".") / path).resolve() - if not target.is_file(): - payload: dict = {"path": path, "error": "not found"} - elif target.suffix != ".md": - payload = {"path": path, "error": "not markdown"} - elif not keys: - payload = {"path": path, "error": "keys is empty"} + vault_dir = Path(self.file_store.vault_path or ".").resolve() + target, err = resolve_path(vault_dir, path) + if err or target is None: + payload: dict = {"path": path, "error": err or "invalid path"} else: - post = frontmatter.loads(target.read_text(encoding="utf-8")) - deleted: list[str] = [] - missing: list[str] = [] - for k in keys: - if k in post.metadata: - del post.metadata[k] - deleted.append(k) + lock = await get_path_lock(target) + async with lock: + if not target.is_file(): + payload = {"path": path, "error": "not found"} + elif target.suffix != ".md": + payload = {"path": path, "error": "not markdown"} + elif not keys: + payload = {"path": path, "error": "keys is empty"} else: - missing.append(k) - if deleted: - target.write_text(frontmatter.dumps(post), encoding="utf-8") - payload = { - "path": path, - "deleted": deleted, - "missing": missing, - "frontmatter": dict(post.metadata), - } + post = frontmatter.loads(target.read_text(encoding="utf-8")) + deleted: list[str] = [] + missing: list[str] = [] + for k in keys: + if k in post.metadata: + del post.metadata[k] + deleted.append(k) + else: + missing.append(k) + if deleted: + target.write_text(frontmatter.dumps(post), encoding="utf-8") + payload = { + "path": path, + "deleted": deleted, + "missing": missing, + "frontmatter": dict(post.metadata), + } if "error" in payload: self.context.response.success = False diff --git a/reme4/steps/file_io/frontmatter_read.py b/reme4/steps/file_io/frontmatter_read.py index 26dbcc7a..f3319f95 100644 --- a/reme4/steps/file_io/frontmatter_read.py +++ b/reme4/steps/file_io/frontmatter_read.py @@ -13,6 +13,7 @@ from pathlib import Path import frontmatter import yaml +from ._path import resolve_path from ..base_step import BaseStep from ...components import R @@ -26,7 +27,14 @@ class FrontmatterReadStep(BaseStep): path: str = self.context.get("path") or "" assert path, "path is required" - target = (Path(self.file_store.vault_path or ".") / path).resolve() + vault_dir = Path(self.file_store.vault_path or ".").resolve() + target, err = resolve_path(vault_dir, path) + if err or target is None: + self.context.response.success = False + self.context.response.answer = f"Error: {err or 'invalid path'}" + self.context.response.metadata.update({"path": path, "exists": False, "error": err or "invalid path"}) + self.logger.info(f"[{self.name}] path={path} error={err!r}") + return if not target.is_file(): self.context.response.success = False self.context.response.answer = f"Error: {path} not found" diff --git a/reme4/steps/file_io/frontmatter_update.py b/reme4/steps/file_io/frontmatter_update.py index fce9a1d0..bb1e48ed 100644 --- a/reme4/steps/file_io/frontmatter_update.py +++ b/reme4/steps/file_io/frontmatter_update.py @@ -16,6 +16,8 @@ from pathlib import Path import frontmatter +from ._file_io import get_path_lock +from ._path import resolve_path from ..base_step import BaseStep from ...components import R @@ -31,18 +33,24 @@ class FrontmatterUpdateStep(BaseStep): metadata = self.context.get("metadata") or {} assert isinstance(metadata, dict), "metadata must be a dict" - target = (Path(self.file_store.vault_path or ".") / path).resolve() - if not target.is_file(): - payload: dict = {"path": path, "error": "not found"} - elif target.suffix != ".md": - payload = {"path": path, "error": "not markdown"} - elif not metadata: - payload = {"path": path, "error": "no fields to update"} + vault_dir = Path(self.file_store.vault_path or ".").resolve() + target, err = resolve_path(vault_dir, path) + if err or target is None: + payload: dict = {"path": path, "error": err or "invalid path"} else: - post = frontmatter.loads(target.read_text(encoding="utf-8")) - post.metadata.update(metadata) - target.write_text(frontmatter.dumps(post), encoding="utf-8") - payload = {"path": path, "updated": metadata} + lock = await get_path_lock(target) + async with lock: + if not target.is_file(): + payload = {"path": path, "error": "not found"} + elif target.suffix != ".md": + payload = {"path": path, "error": "not markdown"} + elif not metadata: + payload = {"path": path, "error": "no fields to update"} + else: + post = frontmatter.loads(target.read_text(encoding="utf-8")) + post.metadata.update(metadata) + target.write_text(frontmatter.dumps(post), encoding="utf-8") + payload = {"path": path, "updated": metadata} if "error" in payload: self.context.response.success = False diff --git a/reme4/steps/file_io/list.py b/reme4/steps/file_io/list.py index 4506407d..b7cbdafa 100644 --- a/reme4/steps/file_io/list.py +++ b/reme4/steps/file_io/list.py @@ -16,6 +16,7 @@ should iterate the result and call ``frontmatter_read`` per candidate. from pathlib import Path from typing import Iterable +from ._path import resolve_path from ..base_step import BaseStep from ...components import R @@ -48,14 +49,6 @@ class ListStep(BaseStep): limit = DEFAULT_LIMIT return path, recursive, limit if limit > 0 else DEFAULT_LIMIT - @staticmethod - def _resolve_target_dir(vault_dir: Path, path: str) -> Path: - """Empty → vault root; absolute → as-is; relative → joined under vault_dir.""" - if not path: - return vault_dir - candidate = Path(path) - return candidate.resolve() if candidate.is_absolute() else (vault_dir / candidate).resolve() - @staticmethod def _walk_files(target_dir: Path, recursive: bool, limit: int) -> list[Path]: """Return up to ``limit`` regular files under ``target_dir``; short-circuits at the cap.""" @@ -84,7 +77,10 @@ class ListStep(BaseStep): assert self.context is not None path, recursive, limit = self._collect_params() vault_dir = Path(self.file_store.vault_path or ".").resolve() - target_dir = self._resolve_target_dir(vault_dir, path) + target_dir, err = resolve_path(vault_dir, path, allow_empty=True) + if err or target_dir is None: + self._fail(err or "invalid path", path=path) + return None if not target_dir.exists(): self._fail(f"directory {target_dir} does not exist", path=str(target_dir)) diff --git a/reme4/steps/file_io/move.py b/reme4/steps/file_io/move.py index de64d050..c8a1ed4c 100644 --- a/reme4/steps/file_io/move.py +++ b/reme4/steps/file_io/move.py @@ -32,6 +32,7 @@ original is still removed in that case (move semantics, not copy). import shutil from pathlib import Path +from ._path import resolve_path from ..base_step import BaseStep from ...components import R from ...utils.wikilink_handler import WikilinkHandler @@ -65,9 +66,13 @@ class MoveStep(BaseStep): self.context.response.metadata.update(payload) async def _move(self, src_path: str, dst_path: str, overwrite: bool, retarget: bool) -> dict: - vault_dir = Path(self.file_store.vault_path or ".") - src_abs = (vault_dir / src_path).resolve() if src_path else None - dst_abs = (vault_dir / dst_path).resolve() if not Path(dst_path).is_absolute() else None + vault_dir = Path(self.file_store.vault_path or ".").resolve() + src_abs, src_err = resolve_path(vault_dir, src_path) if src_path else (None, "src_path is required") + dst_abs, dst_err = resolve_path(vault_dir, dst_path) if dst_path else (None, "dst_path is required") + if src_err: + return {"src_path": src_path, "error": src_err} + if dst_err: + return {"dst_path": dst_path, "error": dst_err} precheck_error = _precheck_move(src_path, dst_path, src_abs, dst_abs, overwrite) if precheck_error: return precheck_error diff --git a/reme4/steps/file_io/read.py b/reme4/steps/file_io/read.py index d153ad87..d3ecd960 100644 --- a/reme4/steps/file_io/read.py +++ b/reme4/steps/file_io/read.py @@ -2,10 +2,11 @@ from pathlib import Path -from ._file_io import read_file_safe, truncate_text_output +from ._file_io import read_file_lines_safe, read_file_safe, truncate_text_output from ._path import NON_MD_WARNING, gate_md, resolve_path from ..base_step import BaseStep from ...components import R +from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES from ...utils import expand_links, render_expansion_lines @@ -88,6 +89,7 @@ class ReadStep(BaseStep): self._fail(f"read failed: {e}", path=str(target)) return None + # pylint: disable=too-many-return-statements async def execute(self): assert self.context is not None raw = str(self.context.get("path") or "") @@ -104,23 +106,40 @@ class ReadStep(BaseStep): if not self._check_file(target): return None - content = await self._load_content(target) - if content is None: - return None + if target.stat().st_size <= MAX_FILE_READ_BYTES: + content = await self._load_content(target) + if content is None: + return None - all_lines = content.split("\n") - total = len(all_lines) - bounds = self._resolve_range(total, start_line, end_line, target) - if bounds is None: - return None - s, e = bounds + all_lines = content.split("\n") + total = len(all_lines) + bounds = self._resolve_range(total, start_line, end_line, target) + if bounds is None: + return None + s, e = bounds + excerpt = "\n".join(all_lines[s - 1 : e]) + else: + s = max(1, int(start_line) if start_line is not None else 1) + requested_end = int(end_line) if end_line is not None else None + if requested_end is not None and s > requested_end: + self._fail(f"start_line ({s}) > end_line ({requested_end})", path=str(target)) + return None + try: + excerpt, total, _encoding = await read_file_lines_safe( + target, + s, + requested_end, + max_collect_bytes=DEFAULT_MAX_BYTES * 2, + ) + except Exception as e: # pylint: disable=broad-except + self._fail(f"read failed: {e}", path=str(target)) + return None + if s > total: + self._fail(f"start_line {s} exceeds file length ({total} lines)", path=str(target), total_lines=total) + return None + e = min(total, requested_end if requested_end is not None else total) - text = truncate_text_output( - "\n".join(all_lines[s - 1 : e]), - start_line=s, - total_lines=total, - file_path=str(target), - ) + text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target)) self.context.response.success = True self.context.response.answer = text diff --git a/reme4/steps/file_io/stat.py b/reme4/steps/file_io/stat.py index 41a82415..68093a95 100644 --- a/reme4/steps/file_io/stat.py +++ b/reme4/steps/file_io/stat.py @@ -24,6 +24,7 @@ from pathlib import Path import frontmatter +from ._path import resolve_path from ..base_step import BaseStep from ...components import R @@ -37,7 +38,14 @@ class StatStep(BaseStep): path: str = self.context.get("path", "") or "" assert path, "path is required" - target = (Path(self.file_store.vault_path or ".") / path).resolve() + vault_dir = Path(self.file_store.vault_path or ".").resolve() + target, err = resolve_path(vault_dir, path) + if err or target is None: + self.context.response.success = False + self.context.response.answer = f"Error: {err or 'invalid path'}" + self.context.response.metadata.update({"path": path, "exists": False, "error": err or "invalid path"}) + self.logger.info(f"[{self.name}] path={path} error={err!r}") + return if not target.exists(): self.context.response.success = False self.context.response.answer = f"stat: {path} not found" diff --git a/reme4/steps/index/__init__.py b/reme4/steps/index/__init__.py index e69de29b..7b903e3a 100644 --- a/reme4/steps/index/__init__.py +++ b/reme4/steps/index/__init__.py @@ -0,0 +1,31 @@ +"""Index steps.""" + +from .clear_store import ClearStoreStep +from .log_changes import LogChangesStep +from .node_search import NodeSearchStep +from .init_changes import InitChangesStep +from .search import SearchStep +from .traverse import TraverseStep +from .update_changes import ChangeApplyStep, UpdateCatalogStep, UpdateIndexStep +from .watch_changes import ( + DEFAULT_LOW_POWER_POLL_MS, + DEFAULT_WATCH_DEBOUNCE_MS, + DEFAULT_WATCH_STEP_MS, + WatchChangesStep, +) + +__all__ = [ + "ChangeApplyStep", + "ClearStoreStep", + "DEFAULT_LOW_POWER_POLL_MS", + "DEFAULT_WATCH_DEBOUNCE_MS", + "DEFAULT_WATCH_STEP_MS", + "InitChangesStep", + "LogChangesStep", + "NodeSearchStep", + "SearchStep", + "TraverseStep", + "UpdateCatalogStep", + "UpdateIndexStep", + "WatchChangesStep", +] diff --git a/reme4/steps/index/_change_batch.py b/reme4/steps/index/_change_batch.py new file mode 100644 index 00000000..5000ff27 --- /dev/null +++ b/reme4/steps/index/_change_batch.py @@ -0,0 +1,50 @@ +"""Helpers for the common file change batch shape.""" + +from collections import OrderedDict +from collections.abc import Callable +from pathlib import Path + +from watchfiles import Change + + +def _normalize_change(raw) -> Change | None: + if isinstance(raw, Change): + return raw + if isinstance(raw, str): + return Change.__members__.get(raw) + return None + + +def coalesce_changes(changes: list[dict], path_exists: Callable[[str], bool] | None = None) -> list[dict]: + """Collapse duplicate path events to the final on-disk state.""" + by_path: OrderedDict[str, set[Change]] = OrderedDict() + for item in changes: + if not isinstance(item, dict) or "path" not in item: + continue + change = _normalize_change(item.get("change")) + if change not in (Change.added, Change.modified, Change.deleted): + continue + by_path.setdefault(item["path"], set()).add(change) + + def exists(path: str) -> bool: + return path_exists(path) if path_exists is not None else Path(path).is_file() + + result: list[dict] = [] + for path, seen in by_path.items(): + if not exists(path): + result.append({"change": Change.deleted.name, "path": path}) + elif seen == {Change.added}: + result.append({"change": Change.added.name, "path": path}) + else: + result.append({"change": Change.modified.name, "path": path}) + return result + + +def bucket_changes(changes: list[dict], path_exists: Callable[[str], bool] | None = None) -> dict[Change, list[str]]: + """Group changes by watchfiles.Change.""" + buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []} + for item in coalesce_changes(changes, path_exists=path_exists): + change = _normalize_change(item["change"]) + if isinstance(change, Change) and change in buckets: + buckets[change].append(item["path"]) + return buckets diff --git a/reme4/steps/index/_watch_rules.py b/reme4/steps/index/_watch_rules.py index 631b4ada..c68440e7 100644 --- a/reme4/steps/index/_watch_rules.py +++ b/reme4/steps/index/_watch_rules.py @@ -1,4 +1,4 @@ -"""Shared watch-rule logic for scan_changes and watch_changes steps.""" +"""Shared watch-rule logic for init_changes and watch_changes steps.""" from dataclasses import dataclass, field from pathlib import Path @@ -6,6 +6,7 @@ from typing import TYPE_CHECKING if TYPE_CHECKING: from ...schema import ApplicationConfig + from ...components.runtime_context import RuntimeContext @dataclass @@ -31,6 +32,21 @@ def build_watch_rules( return rules +def build_context_watch_rules( + app_config: "ApplicationConfig | None", + vault_path: Path, + context: "RuntimeContext", +) -> list[WatchRule]: + """Build watch rules from context-level watch_dirs/watch_suffixes.""" + if app_config is None: + return [] + watch_dirs: list[str] = context.get("watch_dirs", []) + watch_suffixes: list[str] = context.get("watch_suffixes", []) + if not watch_dirs: + return [] + return build_watch_rules(app_config, vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes) + + def collect_existing(rules: list[WatchRule], recursive: bool) -> dict[str, float]: """Walk rule paths and return {abs_path: st_mtime} for matching files.""" existing: dict[str, float] = {} diff --git a/reme4/steps/index/clear_and_scan.py b/reme4/steps/index/clear_and_scan.py deleted file mode 100644 index 503ededc..00000000 --- a/reme4/steps/index/clear_and_scan.py +++ /dev/null @@ -1,30 +0,0 @@ -"""Wipe the file store and emit every vault file as an ``added`` change. - -Designed to be chained before ``update_index_step`` so that the second step -performs the actual re-indexing and persistence. -""" - -from ..base_step import BaseStep -from ...components import R - - -@R.register("clear_and_scan_step") -class ClearAndScanStep(BaseStep): - """Clear the file store, walk the vault, and write changes into the context.""" - - async def execute(self): - assert self.context is not None - suffixes = tuple("." + s.strip(".") for s in self.context.get("suffix_filters", ["md"])) - - await self.file_store.clear() - paths = [ - str(p.absolute()) - for p in self.vault_path.rglob("*") - if p.is_file() and (not suffixes or str(p).endswith(suffixes)) - ] - - self.context["changes"] = [{"change": "added", "path": p} for p in paths] - counts = {"added": len(paths), "modified": 0, "deleted": 0} - self.context.response.metadata["counts"] = counts - self.logger.info(f"[{self.name}] cleared store and scanned {len(paths)} file(s)") - return self.context.response diff --git a/reme4/steps/index/clear_store.py b/reme4/steps/index/clear_store.py new file mode 100644 index 00000000..14eece03 --- /dev/null +++ b/reme4/steps/index/clear_store.py @@ -0,0 +1,16 @@ +"""Clear the file store before reusing the standard init change producer.""" + +from ..base_step import BaseStep +from ...components import R + + +@R.register("clear_store_step") +class ClearStoreStep(BaseStep): + """Wipe ``file_store`` so ``init_changes_step(store=file_store)`` sees all files as added.""" + + async def execute(self): + assert self.context is not None + await self.file_store.clear() + self.context.response.metadata["cleared_store"] = True + self.logger.info(f"[{self.name}] cleared file_store") + return self.context.response diff --git a/reme4/steps/index/foreach_dispatch.py b/reme4/steps/index/foreach_dispatch.py deleted file mode 100644 index c2f6aa2f..00000000 --- a/reme4/steps/index/foreach_dispatch.py +++ /dev/null @@ -1,27 +0,0 @@ -"""Foreach dispatch: iterate changes and call a configured job per file.""" - -from ..base_step import BaseStep -from ...components import R - - -@R.register("foreach_dispatch_step") -class ForeachDispatchStep(BaseStep): - """For each change item, call ``dispatch_job`` with the vault-relative path.""" - - async def execute(self): - assert self.context is not None - changes: list[dict] = self.context.get("changes") or [] - dispatch_job: str = self.context.get("dispatch_job", "") - if not dispatch_job: - self.logger.warning(f"[{self.name}] no dispatch_job configured, skip") - self.context.response.success = True - return self.context.response - for item in changes: - rel_path = self.to_vault_relative(item["path"]) - try: - await self.run_job(dispatch_job, file_path=rel_path, change=item["change"]) - except Exception: - self.logger.exception(f"[{self.name}] dispatch {dispatch_job} failed: {rel_path}") - self.context.response.success = True - self.context.response.metadata["dispatched"] = len(changes) - return self.context.response diff --git a/reme4/steps/index/init_changes.py b/reme4/steps/index/init_changes.py new file mode 100644 index 00000000..d3275ab7 --- /dev/null +++ b/reme4/steps/index/init_changes.py @@ -0,0 +1,80 @@ +"""One-shot change producer: diff watched files against file_store/file_catalog.""" + +from pathlib import Path +from typing import Iterable + +from ._change_batch import coalesce_changes +from ._watch_rules import WatchRule, build_context_watch_rules, collect_existing +from ..base_step import BaseStep +from ...components import R +from ...schema import FileNode + + +@R.register("init_changes_step") +class InitChangesStep(BaseStep): + """Scan once, write ``context["changes"]``, then dispatch change handlers.""" + + def __init__( + self, + monitor_type: str | None = None, + monitor_name: str = "default", + store: str | None = None, + recursive: bool = True, + **kwargs, + ): + super().__init__(**kwargs) + self.monitor_type = monitor_type or store + self.monitor_name = monitor_name + self.recursive = recursive + if self.monitor_type in {"file_store", "file_catalog"}: + self.kwargs.setdefault(self.monitor_type, monitor_name) + + def _get_watch_rules(self) -> list[WatchRule]: + assert self.context is not None + app_config = self.app_context.app_config if self.app_context else None + return build_context_watch_rules(app_config, self.vault_path, self.context) + + async def _load_indexed_nodes(self) -> Iterable[FileNode]: + if self.monitor_type == "file_store": + return await self.file_store.get_nodes() + if self.monitor_type == "file_catalog": + if self.file_catalog is None: + raise RuntimeError("file_catalog is not initialized!") + return await self.file_catalog.get_nodes() + raise ValueError("init_changes_step.monitor_type must be 'file_store' or 'file_catalog'") + + @staticmethod + def diff(existing: dict[str, float], nodes: Iterable[FileNode], vault_path: Path) -> tuple[ + list[dict], + dict[str, int], + ]: + """Compute added/modified/deleted vs ``nodes`` and return (changes, counts).""" + indexed: dict[str, float] = { + str(Path(n.path) if Path(n.path).is_absolute() else vault_path / n.path): n.st_mtime for n in nodes + } + to_delete = list(indexed.keys() - existing.keys()) + to_add = list(existing.keys() - indexed.keys()) + to_modify = [p for p in existing.keys() & indexed.keys() if existing[p] != indexed[p]] + changes: list[dict] = ( + [{"change": "added", "path": p} for p in to_add] + + [{"change": "modified", "path": p} for p in to_modify] + + [{"change": "deleted", "path": p} for p in to_delete] + ) + counts = {"added": len(to_add), "modified": len(to_modify), "deleted": len(to_delete)} + return changes, counts + + async def execute(self): + assert self.context is not None + rules = self._get_watch_rules() + existing = collect_existing(rules, recursive=self.recursive) + nodes = await self._load_indexed_nodes() + changes, counts = self.diff(existing, nodes, self.vault_path) + changes = coalesce_changes(changes) + self.context["changes"] = changes + if changes: + self.logger.info(f"[{self.name}] scan {self.monitor_type}:{self.monitor_name}: {counts}") + await self.dispatch_steps(self.dispatch_step_specs, changes=changes) + else: + self.logger.info(f"[{self.name}] {self.monitor_type}:{self.monitor_name} is up to date") + self.context.response.metadata["counts"] = counts + return self.context.response diff --git a/reme4/steps/index/node_search.py b/reme4/steps/index/node_search.py index cef24c82..ae35e609 100644 --- a/reme4/steps/index/node_search.py +++ b/reme4/steps/index/node_search.py @@ -72,7 +72,7 @@ class NodeSearchStep(BaseStep): return self.context.response assert limit > 0, f"limit must be positive, got {limit}" - digest_dir = getattr(self.app_context.app_config, "digest_dir", "digest") or "digest" + digest_dir = self.config_value("digest_dir") digest_prefix = digest_dir.rstrip("/") + "/" # Over-fetch — digest filter drops a lot of raw hits. diff --git a/reme4/steps/index/scan_changes.py b/reme4/steps/index/scan_changes.py deleted file mode 100644 index c173a07a..00000000 --- a/reme4/steps/index/scan_changes.py +++ /dev/null @@ -1,97 +0,0 @@ -"""One-shot scan: diff watch_paths vs an indexed-state source, write changes into context. - -Two symmetric variants — pick the one whose state source matches the loop's -write target so the index loop and the dream loop never contend on the -same component: - -* :class:`ScanStoreChangesStep` (``scan_store_changes_step``) — diffs - against ``file_store``; used by ``index_update_loop`` (sole writer of - ``file_store``). -* :class:`ScanCatalogChangesStep` (``scan_catalog_changes_step``) — diffs - against ``file_catalog``; used by ``resource_watch_loop`` and - ``digest_watch_loop`` (sole writers of ``file_catalog``). - -Both share the same diff vocabulary (``added`` / ``modified`` / ``deleted``) -and write into ``context["changes"]`` in the same shape, so downstream -steps don't care which variant produced the batch. -""" - -from pathlib import Path -from typing import Iterable - -from ._watch_rules import WatchRule, build_watch_rules, collect_existing -from ..base_step import BaseStep -from ...components import R -from ...schema import FileNode - - -def _diff(existing: dict[str, float], nodes: Iterable[FileNode], vault_path: Path) -> tuple[list[dict], dict[str, int]]: - """Compute added/modified/deleted vs ``nodes`` and return (changes, counts).""" - indexed: dict[str, float] = { - str(Path(n.path) if Path(n.path).is_absolute() else vault_path / n.path): n.st_mtime for n in nodes - } - to_delete = list(indexed.keys() - existing.keys()) - to_add = list(existing.keys() - indexed.keys()) - to_modify = [p for p in existing.keys() & indexed.keys() if existing[p] != indexed[p]] - changes: list[dict] = ( - [{"change": "added", "path": p} for p in to_add] - + [{"change": "modified", "path": p} for p in to_modify] - + [{"change": "deleted", "path": p} for p in to_delete] - ) - counts = {"added": len(to_add), "modified": len(to_modify), "deleted": len(to_delete)} - return changes, counts - - -class _ScanChangesBase(BaseStep): - """Shared scaffolding: collect on-disk state, defer node-loading to the subclass.""" - - def __init__(self, recursive: bool = True, **kwargs): - super().__init__(**kwargs) - self.recursive: bool = recursive - - def _get_watch_rules(self) -> list[WatchRule]: - """Build watch rules from context-level watch_dirs/watch_suffixes.""" - assert self.context is not None - app_config = self.app_context.app_config if self.app_context else None - if app_config is None: - return [] - watch_dirs: list[str] = self.context.get("watch_dirs", []) - watch_suffixes: list[str] = self.context.get("watch_suffixes", []) - if not watch_dirs: - return [] - return build_watch_rules(app_config, self.vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes) - - async def _load_indexed_nodes(self) -> Iterable[FileNode]: - raise NotImplementedError - - async def execute(self): - assert self.context is not None - rules = self._get_watch_rules() - existing = collect_existing(rules, recursive=self.recursive) - nodes = await self._load_indexed_nodes() - changes, counts = _diff(existing, nodes, self.vault_path) - self.context["changes"] = changes - if changes: - self.logger.info(f"[{self.name}] scan: {counts}") - else: - self.logger.info(f"[{self.name}] store is up to date") - self.context.response.metadata["counts"] = counts - return self.context.response - - -@R.register("scan_store_changes_step") -class ScanStoreChangesStep(_ScanChangesBase): - """Diff vault against ``file_store``; used by the index loop.""" - - async def _load_indexed_nodes(self) -> Iterable[FileNode]: - if self.file_store is None: - raise RuntimeError("file_store is not initialized!") - return await self.file_store.get_nodes() - - -@R.register("scan_catalog_changes_step") -class ScanCatalogChangesStep(_ScanChangesBase): - """Diff vault against ``file_catalog``; used by resource/digest loops.""" - - async def _load_indexed_nodes(self) -> Iterable[FileNode]: - return await self.file_catalog.get_nodes() diff --git a/reme4/steps/index/update_catalog.py b/reme4/steps/index/update_catalog.py deleted file mode 100644 index 627c6886..00000000 --- a/reme4/steps/index/update_catalog.py +++ /dev/null @@ -1,83 +0,0 @@ -"""Update file catalog with a batch of file changes.""" - -from pathlib import Path - -from watchfiles import Change - -from ..base_step import BaseStep -from ...components import R -from ...schema import FileNode - - -@R.register("update_catalog_step") -class UpdateCatalogStep(BaseStep): - """Classify raw watcher changes and update the file_catalog.""" - - def __init__(self, persist: bool = False, **kwargs): - super().__init__(**kwargs) - self._persist: bool = persist - - async def execute(self): - assert self.context is not None - changes: list[dict] = self.context.get("changes") or [] - persist: bool = self._persist or bool(self.context.get("persist", False)) - - buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []} - for item in changes: - c = item["change"] - if isinstance(c, str): - c = Change.__members__.get(c) - if isinstance(c, Change) and c in buckets: - buckets[c].append(item["path"]) - - results: list[dict] = [] - - for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")): - paths = buckets[change] - if not paths: - continue - self.logger.info(f"Detected {len(paths)} {change.name} file(s)") - nodes: list[FileNode] = [] - ok_paths: list[str] = [] - for path in paths: - abs_path = Path(path) - if not abs_path.is_file(): - results.append({"change": change.name, "path": path, "success": False, "error": "not a file"}) - continue - self.logger.info(f"{action} file: {path}") - try: - stat = abs_path.stat() - nodes.append(FileNode(path=self.to_vault_relative(abs_path), st_mtime=stat.st_mtime)) - ok_paths.append(path) - except Exception as e: - self.logger.exception(f"Failed to stat {path}") - results.append({"change": change.name, "path": path, "success": False, "error": str(e)}) - if nodes: - try: - await self.file_catalog.delete([n.path for n in nodes]) - await self.file_catalog.upsert(nodes) - results.extend({"change": change.name, "path": p, "success": True} for p in ok_paths) - except Exception as e: - self.logger.exception(f"Failed to upsert {len(nodes)} {change.name} file(s)") - results.extend( - {"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths - ) - - if deleted := buckets[Change.deleted]: - if self.file_catalog is None: - raise RuntimeError("file_catalog is not initialized!") - self.logger.info(f"Detected {len(deleted)} deleted file(s)") - rel_deleted = [self.to_vault_relative(p) for p in deleted] - try: - await self.file_catalog.delete(rel_deleted) - results.extend({"change": "deleted", "path": p, "success": True} for p in deleted) - except Exception as e: - self.logger.exception(f"Failed to delete {len(deleted)} file(s)") - results.extend({"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted) - - if persist and results: - await self.file_catalog.dump() - - self.context.response.answer = results - self.context.response.success = all(r["success"] for r in results) if results else True - return self.context.response diff --git a/reme4/steps/index/update_changes.py b/reme4/steps/index/update_changes.py new file mode 100644 index 00000000..6e735752 --- /dev/null +++ b/reme4/steps/index/update_changes.py @@ -0,0 +1,179 @@ +"""Apply file change batches to file_catalog or file_store.""" + +from abc import abstractmethod +from pathlib import Path +from typing import Any + +from watchfiles import Change + +from ._change_batch import bucket_changes +from ..base_step import BaseStep +from ...components import R +from ...components.file_chunker import BaseFileChunker +from ...enumeration import ComponentEnum +from ...schema import FileChunk, FileNode + + +class ChangeApplyStep(BaseStep): + """Shared added/modified/deleted handling for index update targets.""" + + target_name = "target" + + def __init__(self, persist: bool | None = None, **kwargs): + super().__init__(**kwargs) + self.persist = persist + + @abstractmethod + async def build_item(self, path: Path) -> Any: + """Parse one existing file into the target item shape.""" + + @abstractmethod + def item_path(self, item: Any) -> str: + """Return the target-relative path for an upsert item.""" + + @abstractmethod + async def upsert_items(self, items: list[Any]) -> None: + """Upsert parsed items into the target.""" + + @abstractmethod + async def delete_paths(self, paths: list[str]) -> None: + """Delete target-relative paths from the target.""" + + @abstractmethod + async def dump_target(self) -> None: + """Persist the target.""" + + async def execute(self): + assert self.context is not None + changes: list[dict] = self.context.get("changes") or [] + persist = bool(self.context.get("persist", True)) if self.persist is None else self.persist + buckets = bucket_changes(changes, path_exists=lambda p: self._to_abs_path(p).is_file()) + results = await self._apply_existing(buckets) + results.extend(await self._apply_deleted(buckets[Change.deleted])) + if persist and results: + await self.dump_target() + self.context.response.answer = results + self.context.response.success = all(r["success"] for r in results) if results else True + return self.context.response + + async def _apply_existing(self, buckets: dict[Change, list[str]]) -> list[dict]: + results: list[dict] = [] + for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")): + paths = buckets[change] + if not paths: + continue + self.logger.info(f"Detected {len(paths)} {change.name} file(s)") + items, ok_paths = [], [] + for path in paths: + item = await self._try_build_item(change, action, path, results) + if item is not None: + items.append(item) + ok_paths.append(path) + if items: + results.extend(await self._try_upsert(change, items, ok_paths)) + return results + + async def _try_build_item(self, change: Change, action: str, path: str, results: list[dict]): + abs_path = self._to_abs_path(path) + if not abs_path.is_file(): + results.append({"change": change.name, "path": path, "success": False, "error": "not a file"}) + return None + self.logger.info(f"{action} file: {path}") + try: + return await self.build_item(abs_path) + except Exception as e: + self.logger.exception(f"Failed to parse {path}") + results.append({"change": change.name, "path": path, "success": False, "error": str(e)}) + return None + + async def _try_upsert(self, change: Change, items: list[Any], ok_paths: list[str]) -> list[dict]: + try: + await self.delete_paths([self.item_path(item) for item in items]) + await self.upsert_items(items) + return [{"change": change.name, "path": p, "success": True} for p in ok_paths] + except Exception as e: + self.logger.exception(f"Failed to upsert {len(items)} {change.name} file(s) into {self.target_name}") + return [{"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths] + + async def _apply_deleted(self, deleted: list[str]) -> list[dict]: + if not deleted: + return [] + self.logger.info(f"Detected {len(deleted)} deleted file(s)") + try: + await self.delete_paths([self.to_vault_relative(p) for p in deleted]) + return [{"change": "deleted", "path": p, "success": True} for p in deleted] + except Exception as e: + self.logger.exception(f"Failed to delete {len(deleted)} file(s) from {self.target_name}") + return [{"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted] + + def _to_abs_path(self, path: str | Path) -> Path: + p = Path(path) + return p if p.is_absolute() else self.vault_path / p + + +@R.register("update_catalog_step") +class UpdateCatalogStep(ChangeApplyStep): + """Update file_catalog with a batch of file changes.""" + + target_name = "file_catalog" + + async def build_item(self, path: Path) -> FileNode: + stat = path.stat() + return FileNode(path=self.to_vault_relative(path), st_mtime=stat.st_mtime) + + def item_path(self, item: FileNode) -> str: + return item.path + + async def upsert_items(self, items: list[FileNode]) -> None: + if self.file_catalog is None: + raise RuntimeError("file_catalog is not initialized!") + await self.file_catalog.upsert(items) + + async def delete_paths(self, paths: list[str]) -> None: + if self.file_catalog is None: + raise RuntimeError("file_catalog is not initialized!") + await self.file_catalog.delete(paths) + + async def dump_target(self) -> None: + if self.file_catalog is not None: + await self.file_catalog.dump() + + +@R.register("update_index_step") +class UpdateIndexStep(ChangeApplyStep): + """Update file_store with a batch of file changes.""" + + target_name = "file_store" + + async def build_item(self, path: Path) -> tuple[FileNode, list[FileChunk]]: + return await self.chunk_file(path) + + def item_path(self, item: tuple[FileNode, list[FileChunk]]) -> str: + return item[0].path + + async def upsert_items(self, items: list[tuple[FileNode, list[FileChunk]]]) -> None: + await self.file_store.upsert(items) + + async def delete_paths(self, paths: list[str]) -> None: + await self.file_store.delete(paths) + + async def dump_target(self) -> None: + await self.file_store.dump() + + async def chunk_file(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]: + """Chunk a file into (node, chunks).""" + if self.app_context is None: + raise RuntimeError("app_context is not set when resolving file chunker") + chunker = self._resolve_chunker(Path(path)) + return await chunker.chunk(path) + + def _resolve_chunker(self, path: Path) -> BaseFileChunker: + """Resolve a file chunker for a given path.""" + chunkers: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER] + suffix = path.suffix.lstrip(".").lower() + for candidate in chunkers.values(): + if suffix and suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}: + return candidate + if default := chunkers.get("default"): + return default + raise RuntimeError(f"No file chunker supports {path} (suffix={suffix!r}) and no default chunker is configured") diff --git a/reme4/steps/index/update_index.py b/reme4/steps/index/update_index.py deleted file mode 100644 index b8d1a258..00000000 --- a/reme4/steps/index/update_index.py +++ /dev/null @@ -1,119 +0,0 @@ -"""Update index with a batch of file changes.""" - -from pathlib import Path - -from watchfiles import Change - -from ..base_step import BaseStep -from ...components import R -from ...components.file_chunker import BaseFileChunker -from ...enumeration import ComponentEnum -from ...schema import FileChunk, FileNode - - -@R.register("update_index_step") -class UpdateIndexStep(BaseStep): - """Classify raw watcher changes and update the file_store index.""" - - def __init__(self, persist: bool = False, **kwargs): - super().__init__(**kwargs) - self.persist: bool = persist - - async def chunk_file(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]: - """Chunk ``path`` using the file chunker whose ``supported_extensions`` claims its suffix. - - First registered match wins (config insertion order). Falls back to the - ``default`` chunker when no chunker claims the suffix. - """ - if self.app_context is None: - raise RuntimeError("app_context is not set when resolving file chunker") - chunker_dict: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER] - - suffix = Path(path).suffix.lstrip(".").lower() - - chunker: BaseFileChunker | None = None - if suffix: - for candidate in chunker_dict.values(): - if suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}: - chunker = candidate - break - - if chunker is None: - chunker = chunker_dict.get("default") - - if chunker is None: - raise RuntimeError( - f"No file chunker supports {path} (suffix={suffix!r}) and no 'default' chunker is configured", - ) - - return await chunker.chunk(path) - - async def execute(self): - assert self.context is not None - # Each item: {"change": Change | "added"|"modified"|"deleted", "path": absolute path} - changes: list[dict] = self.context.get("changes") or [] - - buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []} - for item in changes: - c = item["change"] - if isinstance(c, str): - c = Change.__members__.get(c) - if isinstance(c, Change) and c in buckets: - buckets[c].append(item["path"]) - - results: list[dict] = [] - - for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")): - paths = buckets[change] - if not paths: - continue - self.logger.info(f"Detected {len(paths)} {change.name} file(s)") - parsed: list[tuple[FileNode, list[FileChunk]]] = [] - ok_paths: list[str] = [] - for path in paths: - abs_path = Path(path) - if not abs_path.is_file(): - results.append({"change": change.name, "path": path, "success": False, "error": "not a file"}) - continue - self.logger.info(f"{action} file: {path}") - try: - parsed.append(await self.chunk_file(abs_path)) - ok_paths.append(path) - except Exception as e: - self.logger.exception(f"Failed to parse {path}") - results.append({"change": change.name, "path": path, "success": False, "error": str(e)}) - if parsed: - try: - await self.file_store.delete([n.path for n, _ in parsed]) - await self.file_store.upsert(parsed) - results.extend({"change": change.name, "path": p, "success": True} for p in ok_paths) - except Exception as e: - self.logger.exception(f"Failed to persist {len(parsed)} {change.name} file(s)") - results.extend( - {"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths - ) - - if deleted := buckets[Change.deleted]: - if self.file_store is None: - raise RuntimeError("file_store is not initialized!") - self.logger.info(f"Detected {len(deleted)} deleted file(s)") - rel_deleted: list[str] = [] - for path in deleted: - p = Path(path).absolute() - try: - rel_deleted.append(str(p.relative_to(self.vault_path))) - except ValueError: - rel_deleted.append(str(p)) - try: - await self.file_store.delete(rel_deleted) - results.extend({"change": "deleted", "path": p, "success": True} for p in deleted) - except Exception as e: - self.logger.exception(f"Failed to delete {len(deleted)} file(s)") - results.extend({"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted) - - if self.persist and results: - await self.file_store.dump() - - self.context.response.answer = results - self.context.response.success = all(r["success"] for r in results) if results else True - return self.context.response diff --git a/reme4/steps/index/watch_changes.py b/reme4/steps/index/watch_changes.py index af0ce3f9..d36bd46f 100644 --- a/reme4/steps/index/watch_changes.py +++ b/reme4/steps/index/watch_changes.py @@ -1,20 +1,24 @@ -"""Long-running awatch loop: convert raw changes into dispatch_step calls. +"""Long-running awatch loop: convert raw changes into dispatch step calls. -Two relevant awatch parameters are exposed verbatim: +Relevant awatch parameters are exposed verbatim: -* ``step`` (default ``50ms``) — awatch yields when the entire watcher +* ``step`` — awatch yields when the entire watcher has gone this long without new changes (and at least one change is pending). Raise to ``5 minutes``-ish for ``auto_dream_loop`` so half-written sync output isn't dreamed mid-write; keep at default for ``index_update_loop`` where every fs change should hit the index promptly. -* ``debounce`` (default ``2000ms``) — per-batch ceiling, regardless +* ``debounce`` — per-batch ceiling, regardless of whether activity is still arriving. Set ``debounce > step`` so ``step`` is the operative limit; otherwise the watcher pre-empts long-quiet-window setups under bursty writes. -Both are global to the watcher (not per-path). The reme watchers +* ``poll_delay_ms`` — delay between polling scans when + ``force_polling=True``. This has the most direct effect on idle CPU + use for the default forced-polling watcher. + +These settings are global to the watcher (not per-path). The reme watchers have disjoint ``watch_dirs`` (configured per job), so global quiet windows are good enough — no per-path bookkeeping needed. @@ -27,10 +31,15 @@ import asyncio from watchfiles import Change, awatch -from ._watch_rules import WatchRule, build_watch_rules, match_file +from ._change_batch import coalesce_changes +from ._watch_rules import WatchRule, build_context_watch_rules, match_file from ..base_step import BaseStep -from ...components import R, BaseComponent -from ...enumeration import ComponentEnum +from ...components import R + + +DEFAULT_WATCH_DEBOUNCE_MS = 5_000 +DEFAULT_WATCH_STEP_MS = 1_000 +DEFAULT_LOW_POWER_POLL_MS = 5_000 @R.register("watch_changes_step") @@ -41,11 +50,9 @@ class WatchChangesStep(BaseStep): self, recursive: bool = True, force_polling: bool = True, - debounce: int = 2000, - step: int = 50, - poll_delay_ms: int = 2000, - dispatch_step: str = "", - dispatch_steps: list[str] | None = None, + debounce: int = DEFAULT_WATCH_DEBOUNCE_MS, + step: int = DEFAULT_WATCH_STEP_MS, + poll_delay_ms: int = DEFAULT_LOW_POWER_POLL_MS, **kwargs, ): super().__init__(**kwargs) @@ -54,20 +61,12 @@ class WatchChangesStep(BaseStep): self.debounce: int = debounce self.step: int = step self.poll_delay_ms: int = poll_delay_ms - self.dispatch_steps: list[str] = dispatch_steps or ([dispatch_step] if dispatch_step else []) self._rules: list[WatchRule] = [] def _get_watch_rules(self) -> list[WatchRule]: - """Build watch rules from context-level watch_dirs/watch_suffixes.""" assert self.context is not None app_config = self.app_context.app_config if self.app_context else None - if app_config is None: - return [] - watch_dirs: list[str] = self.context.get("watch_dirs", []) - watch_suffixes: list[str] = self.context.get("watch_suffixes", []) - if not watch_dirs: - return [] - return build_watch_rules(app_config, self.vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes) + return build_context_watch_rules(app_config, self.vault_path, self.context) def _filter(self, _change: Change, path: str) -> bool: return match_file(path, self._rules) @@ -87,14 +86,10 @@ class WatchChangesStep(BaseStep): if not valid_paths: raise RuntimeError(f"No valid watch paths exist: {[str(r.path) for r in self._rules]}") - dispatch_classes: list[type[BaseComponent]] = [] - for name in self.dispatch_steps: - cls = R.get(ComponentEnum.STEP, name) - if cls is None: - raise RuntimeError(f"Unregistered step '{name}'") - dispatch_classes.append(cls) - - self.logger.info(f"Watching: {[str(p) for p in valid_paths]} step={self.step}ms debounce={self.debounce}ms") + self.logger.info( + f"Watching: {[str(p) for p in valid_paths]} " + f"step={self.step}ms debounce={self.debounce}ms poll_delay={self.poll_delay_ms}ms", + ) async for raw_changes in awatch( *valid_paths, @@ -113,12 +108,9 @@ class WatchChangesStep(BaseStep): for c, p in raw_changes if c in (Change.added, Change.modified, Change.deleted) ] + changes = coalesce_changes(changes) if changes: self.logger.info(f"Detected {len(changes)} change(s)") - # TODO @jinli - extra = {k: v for k, v in self.context.data.items() if k not in ("stop_event", "changes")} - for cls in dispatch_classes: - s = cls(app_context=self.app_context) - await s(changes=changes, **extra) + await self.dispatch_steps(self.dispatch_step_specs, changes=changes) return self.context.response diff --git a/reme4/steps/transfer/__init__.py b/reme4/steps/transfer/__init__.py index e69de29b..fcaa0b02 100644 --- a/reme4/steps/transfer/__init__.py +++ b/reme4/steps/transfer/__init__.py @@ -0,0 +1,11 @@ +"""Transfer steps.""" + +from .download import DownloadStep +from .ingest import IngestStep +from .upload import UploadStep + +__all__ = [ + "DownloadStep", + "IngestStep", + "UploadStep", +] diff --git a/reme4/steps/transfer/download.py b/reme4/steps/transfer/download.py index 99db6420..b43e456a 100644 --- a/reme4/steps/transfer/download.py +++ b/reme4/steps/transfer/download.py @@ -45,7 +45,6 @@ class DownloadStep(BaseStep): src_path: str = self.context.get("src_path", "") or "" dst_path: str = self.context.get("dst_path", "") or "" overwrite: bool = bool(self.context.get("overwrite", False)) - assert src_path, "src_path is required" payload = await self._download(src_path, dst_path, overwrite) if "error" in payload: self.context.response.success = False @@ -60,15 +59,34 @@ class DownloadStep(BaseStep): ) self.context.response.metadata.update(payload) - async def _download(self, src_path: str, dst_path: str, overwrite: bool) -> dict: + async def _download( + self, + src_path: str, + dst_path: str, + overwrite: bool, + ) -> dict: # pylint: disable=too-many-return-statements + # pylint: disable=too-many-return-statements if not src_path: - return {"src_path": src_path, "error": "not found"} - src_abs = (Path(self.file_store.vault_path or ".") / src_path).resolve() + return {"src_path": src_path, "error": "src_path is required"} + vault_dir = Path(self.file_store.vault_path or ".").resolve() + src_abs = (vault_dir / src_path).resolve() + try: + src_abs.relative_to(vault_dir) + except ValueError: + return {"src_path": src_path, "error": "src_path must stay inside the vault"} if not src_abs.is_file(): return {"src_path": src_path, "error": "not found"} if dst_path: - dst_abs = Path(dst_path) + dst_abs = Path(dst_path).expanduser() + if not dst_abs.is_absolute(): + return { + "src_path": src_path, + "dst_path": dst_path, + "error": "dst_path must be an absolute filesystem path", + } + if dst_abs.is_dir(): + return {"src_path": src_path, "dst_path": dst_path, "error": "destination is a directory"} if dst_abs.exists() and not overwrite: return { "src_path": src_path, diff --git a/reme4/steps/transfer/ingest.py b/reme4/steps/transfer/ingest.py index db638fdf..0a31980e 100644 --- a/reme4/steps/transfer/ingest.py +++ b/reme4/steps/transfer/ingest.py @@ -155,10 +155,8 @@ class IngestStep(BaseStep): self.context.response.metadata.update(payload) def _resource_dir_name(self) -> str: - """Configured ``resource_dir`` subdir name; defaults to ``"resource"`` outside an app context.""" - if self.app_context is None: - return "resource" - return self.app_context.app_config.resource_dir + """Configured ``resource_dir`` subdir name.""" + return self.config_value("resource_dir") def _vault_dir(self) -> Path: vr = getattr(self.file_store, "vault_path", None) @@ -181,7 +179,8 @@ class IngestStep(BaseStep): # both surface the conflict rather than getting silently clobbered. on_disk = {p.name for p in bucket.iterdir() if p.is_file()} existing_names = {Path(e.path).name for e in existing_entries} | on_disk - if final_name in existing_names: + existing_names_folded = {name.casefold() for name in existing_names} + if final_name.casefold() in existing_names_folded: raise _DuplicateIngest( f"duplicate: {final_name!r} already exists in {resource_dir}/{date}/", ) @@ -413,7 +412,7 @@ def _assemble_day_md(entries: list[FileNode], date: str) -> str: lines: list[str] = [ "---", f"name: {date}", - f"assets: [{', '.join(names)}]", + f"assets: {json.dumps(names, ensure_ascii=False)}", "---", "", f"# {date} resources", diff --git a/reme4/steps/transfer/upload.py b/reme4/steps/transfer/upload.py index a60fd367..a3b24967 100644 --- a/reme4/steps/transfer/upload.py +++ b/reme4/steps/transfer/upload.py @@ -35,7 +35,6 @@ class UploadStep(BaseStep): src_path: str = self.context.get("src_path", "") or "" dst_path: str = self.context.get("dst_path", "") or "" overwrite: bool = bool(self.context.get("overwrite", False)) - assert src_path and dst_path, "src_path and dst_path are required" payload = await self._upload(src_path, dst_path, overwrite) if "error" in payload: self.context.response.success = False @@ -51,8 +50,19 @@ class UploadStep(BaseStep): ) self.context.response.metadata.update(payload) - async def _upload(self, src_path: str, dst_path: str, overwrite: bool) -> dict: - src_abs = Path(src_path) + async def _upload( + self, + src_path: str, + dst_path: str, + overwrite: bool, + ) -> dict: # pylint: disable=too-many-return-statements + # pylint: disable=too-many-return-statements + if not src_path: + return {"src_path": src_path, "error": "src_path is required"} + if not dst_path: + return {"dst_path": dst_path, "error": "dst_path is required"} + + src_abs = Path(src_path).resolve() if not src_abs.is_file(): return {"src_path": src_path, "error": "not found"} if "/" not in dst_path: @@ -60,10 +70,18 @@ class UploadStep(BaseStep): "dst_path": dst_path, "error": "dst_path must be relative to the vault with a directory component", } - vault_dir = Path(self.file_store.vault_path or ".") - dst_abs = (vault_dir / dst_path).resolve() if not Path(dst_path).is_absolute() else None - if dst_abs is None: + if Path(dst_path).is_absolute(): return {"dst_path": dst_path, "error": "dst_path must be relative to the vault"} + vault_dir = Path(self.file_store.vault_path or ".").resolve() + dst_abs = (vault_dir / dst_path).resolve() + try: + dst_abs.relative_to(vault_dir) + except ValueError: + return {"dst_path": dst_path, "error": "dst_path must stay inside the vault"} + if dst_abs == src_abs: + return {"src_path": src_path, "dst_path": dst_path, "error": "src_path and dst_path are the same"} + if dst_abs.is_dir(): + return {"dst_path": dst_path, "error": "destination is a directory"} if dst_abs.exists() and not overwrite: return { "src_path": src_path, diff --git a/reme4/utils/__init__.py b/reme4/utils/__init__.py index 47fa5ab1..4ad4d2c4 100644 --- a/reme4/utils/__init__.py +++ b/reme4/utils/__init__.py @@ -7,7 +7,7 @@ from .common_utils import ( call_action, call_and_check, ) -from .env_utils import load_env +from .env_utils import load_env, parse_env_file from .link_expansion import expand_links, render_expansion_lines from .logger_utils import get_logger from .logo_utils import print_logo @@ -23,6 +23,7 @@ __all__ = [ "call_action", "call_and_check", "load_env", + "parse_env_file", "expand_links", "render_expansion_lines", "get_logger", diff --git a/reme4/utils/agent_state_io.py b/reme4/utils/agent_state_io.py index 9720723e..adb62d67 100644 --- a/reme4/utils/agent_state_io.py +++ b/reme4/utils/agent_state_io.py @@ -5,6 +5,8 @@ Format: Lines 2+ — AgentState.context, one Msg per line. """ +import os +from uuid import uuid4 from pathlib import Path import aiofiles @@ -20,17 +22,44 @@ class AsStateHandler: def __init__(self, path: str | Path): self.path = Path(path) + @classmethod + def for_session(cls, directory: str | Path, session_id: str) -> "AsStateHandler": + """Create a handler for ``/.jsonl``.""" + if not session_id or Path(session_id).name != session_id: + raise ValueError(f"Invalid session_id: {session_id!r}") + return cls(Path(directory) / f"{session_id}.jsonl") + + def exists(self) -> bool: + """Return whether the state file exists.""" + return self.path.is_file() + + async def load_or_none(self) -> AgentState | None: + """Load state if the file exists, otherwise return ``None``.""" + if not self.exists(): + return None + return await self.load() + + async def delete(self) -> bool: + """Delete the state file if present. Returns whether a file was removed.""" + if not self.exists(): + return False + self.path.unlink() + return True + async def dump(self, state: AgentState) -> Path: """Write *state* to ``self.path`` in JSONL format.""" + self.path.parent.mkdir(parents=True, exist_ok=True) header = UserMsg( name="__state__", content=state.summary or "", metadata={k: getattr(state, k) for k in _META_KEYS}, ) - async with aiofiles.open(self.path, "w", encoding="utf-8") as f: + tmp_path = self.path.with_name(f".{self.path.name}.{uuid4().hex}.tmp") + async with aiofiles.open(tmp_path, "w", encoding="utf-8") as f: await f.write(header.model_dump_json() + "\n") for msg in state.context: await f.write(msg.model_dump_json() + "\n") + os.replace(tmp_path, self.path) return self.path async def load(self) -> AgentState: @@ -47,8 +76,9 @@ class AsStateHandler: else header.get_text_content() or "" ) + metadata = header.metadata or {} return AgentState( - **{k: header.metadata.get(k, d) for k, d in [("session_id", ""), ("reply_id", ""), ("cur_iter", 0)]}, + **{k: metadata.get(k, d) for k, d in [("session_id", ""), ("reply_id", ""), ("cur_iter", 0)]}, summary=summary, context=[Msg.model_validate_json(line) for line in lines[1:] if line.strip()], ) diff --git a/reme4/utils/common_utils.py b/reme4/utils/common_utils.py index fd03eb65..f874180f 100644 --- a/reme4/utils/common_utils.py +++ b/reme4/utils/common_utils.py @@ -154,7 +154,7 @@ async def mock_reme_server( cmd: list[str] = [ sys.executable, "-m", - "reme.reme", + "reme4.reme", "start", f"service.host={host}", f"service.port={port}", diff --git a/reme4/utils/env_utils.py b/reme4/utils/env_utils.py index 7c0aecbc..9ab78f31 100644 --- a/reme4/utils/env_utils.py +++ b/reme4/utils/env_utils.py @@ -4,33 +4,56 @@ import os from pathlib import Path _LOADED = False +_LOADED_VALUES: dict[str, str] = {} -def _parse(path: Path) -> None: +def parse_env_file(path: str | Path) -> dict[str, str]: + """Parse a simple KEY=VALUE env file and return a key/value dict.""" + path = Path(path) + values: dict[str, str] = {} for line in path.read_text(encoding="utf-8").splitlines(): line = line.strip() if not line or line.startswith("#") or "=" not in line: continue key, value = line.split("=", 1) - os.environ[key.strip()] = value.strip().strip("'\"") + key = key.strip() + if not key: + continue + values[key] = value.strip().strip("'\"") + return values -def load_env(path: str | Path | None = None) -> None: - """Load .env from given path, or search cwd and up to 5 parents.""" +def _load_values(values: dict[str, str], *, override: bool) -> dict[str, str]: + loaded: dict[str, str] = {} + for key, value in values.items(): + if override or key not in os.environ: + os.environ[key] = value + loaded[key] = value + return loaded + + +def load_env(path: str | Path | None = None, *, override: bool = True) -> dict[str, str]: + """Load .env from given path, or search cwd and up to 5 parents. + + Returns the key/value pairs loaded into ``os.environ``. Repeated calls without + an explicit path are idempotent and return the values loaded by the first + successful call. + """ global _LOADED - if _LOADED: - return + global _LOADED_VALUES + if path is None and _LOADED: + return dict(_LOADED_VALUES) if path: path = Path(path) if path.exists(): - _parse(path) - _LOADED = True - return + return _load_values(parse_env_file(path), override=override) + return {} for directory in [Path.cwd(), *Path.cwd().parents[:5]]: env_path = directory / ".env" if env_path.exists(): - _parse(env_path) + _LOADED_VALUES = _load_values(parse_env_file(env_path), override=override) _LOADED = True - return + return dict(_LOADED_VALUES) + return {} diff --git a/reme4/utils/jsonl_zst.py b/reme4/utils/jsonl_zst.py new file mode 100644 index 00000000..9f7ddb1c --- /dev/null +++ b/reme4/utils/jsonl_zst.py @@ -0,0 +1,38 @@ +"""Tiny JSONL-over-zstd helpers.""" + +import io +import os +from collections.abc import Iterable, Iterator +from pathlib import Path +from uuid import uuid4 + +import zstandard as zstd + + +def read_jsonl_zst(path: str | Path, encoding: str = "utf-8") -> Iterator[str]: + """Read JSONL-over-zstd lines from a file.""" + path = Path(path) + if not path.exists(): + return + with path.open("rb") as raw: + with zstd.ZstdDecompressor().stream_reader(raw) as reader: + text = io.TextIOWrapper(reader, encoding=encoding) + yield from text + + +def write_jsonl_zst(path: str | Path, lines: Iterable[str], encoding: str = "utf-8") -> Path: + """Write JSONL-over-zstd lines to a file.""" + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(f".{path.name}.{uuid4().hex}.tmp") + with tmp.open("wb") as raw: + with zstd.ZstdCompressor(level=3).stream_writer(raw) as writer: + text = io.TextIOWrapper(writer, encoding=encoding) + for line in lines: + text.write(line) + if not line.endswith("\n"): + text.write("\n") + text.flush() + text.detach() + os.replace(tmp, path) + return path diff --git a/reme4/utils/similarity_utils.py b/reme4/utils/similarity_utils.py index ec0f7100..ab5d7bba 100644 --- a/reme4/utils/similarity_utils.py +++ b/reme4/utils/similarity_utils.py @@ -20,6 +20,10 @@ def cosine_similarity(vec1: list[float], vec2: list[float]) -> float: def batch_cosine_similarity(nd_array1: np.ndarray, nd_array2: np.ndarray) -> np.ndarray: """Pairwise cosine similarity matrix between two batches; output shape (N1, N2).""" + if nd_array1.ndim != 2 or nd_array2.ndim != 2: + raise ValueError( + f"Expected 2D arrays, got shapes {nd_array1.shape} and {nd_array2.shape}", + ) if nd_array1.shape[1] != nd_array2.shape[1]: raise ValueError( f"Embedding dimensions must match: {nd_array1.shape[1]} != {nd_array2.shape[1]}", diff --git a/tests/light/test_reme4_file_io_helpers.py b/tests/light/test_reme4_file_io_helpers.py new file mode 100644 index 00000000..783c5f0d --- /dev/null +++ b/tests/light/test_reme4_file_io_helpers.py @@ -0,0 +1,43 @@ +"""Light tests for reme4 file I/O helpers.""" + +import asyncio +from pathlib import Path + +from reme4.steps.file_io._file_io import read_file_lines_safe +from reme4.steps.file_io._path import resolve_path + + +def test_resolve_path_accepts_absolute_and_rejects_parent_escape(tmp_path: Path): + """Absolute paths remain compatible; relative parent escapes are rejected.""" + vault = tmp_path / "vault" + vault.mkdir() + + absolute = str(tmp_path / "outside.md") + target, err = resolve_path(vault, absolute) + assert err is None + assert target == Path(absolute).resolve() + assert resolve_path(vault, "../outside.md")[1] == "path component cannot be '.' or '..': '..'" + + +def test_resolve_path_allows_empty_when_requested(tmp_path: Path): + """Empty paths can resolve to the vault root for list-like operations.""" + vault = tmp_path / "vault" + vault.mkdir() + + target, err = resolve_path(vault, "", allow_empty=True) + + assert err is None + assert target == vault.resolve() + + +def test_read_file_lines_safe_counts_full_file_but_limits_collected_text(tmp_path: Path): + """Large-file reader counts all lines while bounding returned text.""" + path = tmp_path / "large.md" + path.write_text("\n".join(f"line {i}" for i in range(1, 101)), encoding="utf-8") + + text, total, encoding = asyncio.run(read_file_lines_safe(path, 10, None, max_collect_bytes=25)) + + assert total == 100 + assert encoding == "utf-8" + assert text.startswith("line 10\nline 11") + assert "line 100" not in text diff --git a/tests/test_local_file_graph.py b/tests/test_local_file_graph.py new file mode 100644 index 00000000..55ec81f9 --- /dev/null +++ b/tests/test_local_file_graph.py @@ -0,0 +1,53 @@ +"""Tests for local file graph link scope filtering.""" + +import pytest + +from reme4.components.file_graph.local_file_graph import LocalFileGraph +from reme4.enumeration import LinkScopeEnum +from reme4.schema import FileLink, FileNode + + +def _node(path: str, *targets: str) -> FileNode: + return FileNode( + path=path, + st_mtime=1.0, + links=[FileLink(source_path=path, target_path=target) for target in targets], + ) + + +@pytest.mark.asyncio +async def test_local_file_graph_accepts_string_scope_for_outlinks(): + """String link scopes should filter outlinks.""" + graph = LocalFileGraph(name="test_scope_outlinks") + await graph.upsert_nodes([_node("A.md", "B.md", "Missing.md"), _node("B.md")]) + + assert [link.target_path for link in await graph.get_outlinks("A.md", "real")] == ["B.md"] + assert [link.target_path for link in await graph.get_outlinks("A.md", "virtual")] == ["Missing.md"] + assert [link.target_path for link in await graph.get_outlinks("A.md", "all")] == ["B.md", "Missing.md"] + + +@pytest.mark.asyncio +async def test_local_file_graph_accepts_string_scope_and_orders_inlinks(): + """String link scopes should filter and order inlinks.""" + graph = LocalFileGraph(name="test_scope_inlinks") + await graph.upsert_nodes( + [ + _node("B.md", "Target.md"), + _node("A.md", "Target.md"), + _node("C.md", "Missing.md"), + _node("Target.md"), + ], + ) + + assert [link.source_path for link in await graph.get_inlinks("Target.md", "real")] == ["A.md", "B.md"] + assert [link.source_path for link in await graph.get_inlinks("Missing.md", "virtual")] == ["C.md"] + + +@pytest.mark.asyncio +async def test_local_file_graph_scope_enum_still_filters_virtual_inlinks(): + """Enum link scopes should continue to filter virtual inlinks.""" + graph = LocalFileGraph(name="test_scope_enum_inlinks") + await graph.upsert_nodes([_node("A.md", "Missing.md")]) + + assert await graph.get_inlinks("Missing.md", LinkScopeEnum.REAL) == [] + assert [link.source_path for link in await graph.get_inlinks("Missing.md", LinkScopeEnum.VIRTUAL)] == ["A.md"] diff --git a/tests/test_local_file_store_persistence.py b/tests/test_local_file_store_persistence.py new file mode 100644 index 00000000..7beecfe5 --- /dev/null +++ b/tests/test_local_file_store_persistence.py @@ -0,0 +1,97 @@ +"""Persistence tests for LocalFileStore.""" + +import hashlib +import time + +import pytest + +from reme.core.enumeration.memory_source import MemorySource +from reme.core.file_store.local_file_store import LocalFileStore +from reme.core.schema.file_metadata import FileMetadata +from reme.core.schema.memory_chunk import MemoryChunk + + +def _file_meta(path: str, chunk_count: int) -> FileMetadata: + content = f"Sample content for {path}" + return FileMetadata( + path=path, + hash=hashlib.md5(content.encode()).hexdigest(), + mtime_ms=time.time() * 1000, + size=len(content), + chunk_count=chunk_count, + ) + + +def _chunk(path: str) -> MemoryChunk: + return MemoryChunk( + id="chunk_persist_1", + path=path, + source=MemorySource.MEMORY, + start_line=1, + end_line=2, + text="Persistent memory search content", + hash=hashlib.md5(b"chunk_persist_1").hexdigest(), + embedding=[0.0] * 8, + ) + + +@pytest.mark.asyncio +async def test_local_file_store_persists_upsert_without_close(tmp_path): + """A fresh instance can load data immediately after upsert.""" + path = "memory/persist.md" + chunks = [_chunk(path)] + store = LocalFileStore( + store_name="memory", + db_path=tmp_path, + vector_enabled=False, + fts_enabled=True, + ) + await store.start() + + await store.upsert_file(_file_meta(path, len(chunks)), MemorySource.MEMORY, chunks) + + reloaded = LocalFileStore( + store_name="memory", + db_path=tmp_path, + vector_enabled=False, + fts_enabled=True, + ) + await reloaded.start() + try: + assert await reloaded.list_files(MemorySource.MEMORY) == [path] + loaded_chunks = await reloaded.get_file_chunks(path, MemorySource.MEMORY) + assert [chunk.text for chunk in loaded_chunks] == [chunks[0].text] + finally: + await reloaded.close() + await store.close() + + +@pytest.mark.asyncio +async def test_local_file_store_persists_delete_without_close(tmp_path): + """Deletes are flushed immediately so stale chunks do not reappear.""" + path = "memory/delete.md" + chunks = [_chunk(path)] + store = LocalFileStore( + store_name="memory", + db_path=tmp_path, + vector_enabled=False, + fts_enabled=True, + ) + await store.start() + + await store.upsert_file(_file_meta(path, len(chunks)), MemorySource.MEMORY, chunks) + await store.delete_file(path, MemorySource.MEMORY) + + reloaded = LocalFileStore( + store_name="memory", + db_path=tmp_path, + vector_enabled=False, + fts_enabled=True, + ) + await reloaded.start() + try: + assert await reloaded.list_files(MemorySource.MEMORY) == [] + assert await reloaded.get_file_chunks(path, MemorySource.MEMORY) == [] + finally: + await reloaded.close() + await store.close() diff --git a/tests/test_transfer_steps.py b/tests/test_transfer_steps.py new file mode 100644 index 00000000..58640fb2 --- /dev/null +++ b/tests/test_transfer_steps.py @@ -0,0 +1,101 @@ +"""Tests for transfer step filesystem safety.""" + +# pylint: disable=protected-access + +import datetime +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from reme4.schema import FileFrontMatter, FileNode +from reme4.steps.transfer.download import DownloadStep +from reme4.steps.transfer.ingest import IngestStep, _DuplicateIngest, _assemble_day_md +from reme4.steps.transfer.upload import UploadStep + + +def _step(step_cls, vault_path: Path): + """Build a transfer step with a minimal file_store stub.""" + step = step_cls() + step.file_store = SimpleNamespace(vault_path=vault_path) + return step + + +@pytest.mark.asyncio +async def test_upload_rejects_dst_path_outside_vault(tmp_path): + """Upload must reject destinations that escape the vault.""" + vault = tmp_path / "vault" + vault.mkdir() + source = tmp_path / "source.txt" + source.write_text("payload", encoding="utf-8") + + payload = await _step(UploadStep, vault)._upload(str(source), "dir/../../outside.txt", False) + + assert payload["error"] == "dst_path must stay inside the vault" + assert not (tmp_path / "outside.txt").exists() + + +@pytest.mark.asyncio +async def test_download_rejects_src_path_outside_vault(tmp_path): + """Download must reject sources that escape the vault.""" + vault = tmp_path / "vault" + vault.mkdir() + outside = tmp_path / "outside.txt" + outside.write_text("secret", encoding="utf-8") + + payload = await _step(DownloadStep, vault)._download("../outside.txt", str(tmp_path / "copy.txt"), False) + + assert payload["error"] == "src_path must stay inside the vault" + assert not (tmp_path / "copy.txt").exists() + + +@pytest.mark.asyncio +async def test_download_requires_absolute_explicit_destination(tmp_path): + """Explicit download destinations must be absolute filesystem paths.""" + vault = tmp_path / "vault" + vault.mkdir() + (vault / "in.txt").write_text("content", encoding="utf-8") + + payload = await _step(DownloadStep, vault)._download("in.txt", "relative.txt", False) + + assert payload["error"] == "dst_path must be an absolute filesystem path" + + +def test_ingest_rejects_case_insensitive_duplicate(tmp_path): + """Ingest duplicate detection should be case-insensitive.""" + vault = tmp_path / "vault" + step = _step(IngestStep, vault) + source = tmp_path / "Report.pdf" + source.write_text("content", encoding="utf-8") + date = "2026-06-18" + final_name = "wechat__120000__Report.pdf" + bucket = vault / "resource" / date + bucket.mkdir(parents=True) + (bucket / "wechat__120000__report.pdf").write_text("old", encoding="utf-8") + + with pytest.raises(_DuplicateIngest): + step._land( + source, + date, + final_name, + {"channel": "wechat", "received_at": "2026-06-18T12:00:00", "description": "desc"}, + ) + + +def test_assemble_day_md_quotes_asset_names_for_yaml_frontmatter(): + """Asset names in generated markdown frontmatter should be JSON-quoted.""" + entry = FileNode( + path="resource/2026-06-18/wechat__120000__a, [b].pdf", + st_mtime=1.0, + front_matter=FileFrontMatter( + description="desc", + channel="wechat", + received_at=datetime.datetime(2026, 6, 18, 12, 0, 0).isoformat(), + ), + ) + + rendered = _assemble_day_md([entry], "2026-06-18") + assets_line = next(line for line in rendered.splitlines() if line.startswith("assets: ")) + + assert json.loads(assets_line.removeprefix("assets: ")) == ["wechat__120000__a, [b].pdf"] diff --git a/tests4/integration/_vault_fixture.py b/tests4/integration/_vault_fixture.py index 987115a5..33ff6466 100644 --- a/tests4/integration/_vault_fixture.py +++ b/tests4/integration/_vault_fixture.py @@ -242,6 +242,14 @@ Redis key `auth:jwks:current_kid` 保存当前活跃 kid;Auth Service 之前说过不要总结段落 (我能看 diff),今天再补充一点:也不要"接下来 的步骤"列表,除非我明确问 next steps。直接回答问题然后停。 + +## 关联:已有 digest + +这次 refactor 直接更新这几篇已有 digest 笔记(这里用 wikilink +引用,方便检索关联): + +- JWT 概念:[[digest/wiki/jwt.md]] +- 签名密钥轮换流程:[[digest/procedure/key-rotation.md]] """, } diff --git a/tests4/integration/test_auto_dream.py b/tests4/integration/test_auto_dream.py new file mode 100644 index 00000000..81a23a70 --- /dev/null +++ b/tests4/integration/test_auto_dream.py @@ -0,0 +1,229 @@ +"""Integration test for the 4-step auto_dream job and proactive reader. + +Runs against a real LLM. The test seeds a dream vault, runs ``auto_dream`` for +2026-05-28, verifies digest/interests/catalog effects, then runs ``proactive``. +Agent messages and generated markdown/yaml/jsonl artifacts are copied to +``tests4/integration/logs/auto_dream_latest/`` for manual inspection. +""" + +import asyncio +import json +import shutil +import sys +from pathlib import Path + +import yaml + +INTEGRATION_DIR = Path(__file__).resolve().parent +ARTIFACT_DIR = INTEGRATION_DIR / "logs" / "auto_dream_latest" +sys.path.insert(0, str(INTEGRATION_DIR)) + +# pylint: disable=wrong-import-position +from _vault_fixture import DREAM_INPUT_PATH, vault_env # noqa: E402 + +DREAM_DATE = "2026-05-28" + + +def _read_text(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _print_text_file(label: str, path: Path) -> str: + text = _read_text(path) + print("\n" + "=" * 70) + print(f"[{label}] {path} ({len(text)} bytes)") + print(text) + print("=" * 70) + return text + + +def _reset_artifacts() -> None: + if ARTIFACT_DIR.exists(): + shutil.rmtree(ARTIFACT_DIR) + ARTIFACT_DIR.mkdir(parents=True, exist_ok=True) + + +def _copy_artifact(src: Path, label: str) -> Path | None: + if not src.exists(): + return None + rel = Path(label) / src.name if src.is_file() else Path(label) + dst = ARTIFACT_DIR / rel + dst.parent.mkdir(parents=True, exist_ok=True) + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + else: + shutil.copy2(src, dst) + return dst + + +def _copy_outputs(env, message_files: list[Path]) -> list[Path]: + copied: list[Path] = [] + for path in message_files: + if dst := _copy_artifact(path, "messages"): + copied.append(dst) + for root in ("daily", "digest", "reme_metadata", "reme_session", "agent_logs"): + if dst := _copy_artifact(env.vault_dir / root, root): + copied.append(dst) + return copied + + +def _print_message_files(paths: list[Path]) -> None: + for idx, path in enumerate(paths, 1): + text = _read_text(path) + print("\n" + "=" * 70) + print(f"[messages] {idx}: {path} ({len(text)} bytes)") + print(text[:6000]) + if len(text) > 6000: + print("\n[truncated]") + print("=" * 70) + + +def _all_digest_text(env) -> str: + return "\n\n".join(_read_text(p) for p in env.digest_files()) + + +def _file_graph_links(env) -> dict[str, list[dict]]: + """Map ``path -> links`` from every ``reme_metadata/file_graph/*.jsonl``.""" + out: dict[str, list[dict]] = {} + graph_dir = env.vault_dir / "reme_metadata" / "file_graph" + if not graph_dir.is_dir(): + return out + for graph_path in sorted(graph_dir.glob("*.jsonl")): + for line in graph_path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + node = json.loads(line) + out[node.get("path", "")] = node.get("links") or [] + return out + + +def test_auto_dream_and_proactive(): + """Run auto_dream end to end, save transcripts/results, then read interests via proactive.""" + + async def run(): + _reset_artifacts() + with vault_env() as env: + seeded = env.seed_dream_vault() + app = await env.make_app() + message_files: list[Path] = [] + try: + print("\n" + "=" * 70) + print("[setup] vault_root =", env.vault_dir) + print("[setup] date =", DREAM_DATE) + print("[setup] seeded =", json.dumps(seeded, ensure_ascii=False, indent=2)) + print("=" * 70) + + before_digest = _all_digest_text(env) + with env.record_agents(prefix="agent_dream") as recorder: + response = await app.run_job( + "auto_dream", + date=DREAM_DATE, + hint="Integration test: preserve SOC2, JWT kid, Redis current_kid, and small-PR facts.", + topic_count=3, + topic_diversity_days=7, + ) + dumped = await recorder.dump() + session_jsonl = sorted((env.vault_dir / "reme_session" / "agentscope").glob("*.jsonl")) + message_files = [*dumped, *session_jsonl] + _print_message_files(message_files) + + assert response.success is True, f"auto_dream failed: {response.answer!r}\n{response.metadata!r}" + dream = (response.metadata or {}).get("dream") or {} + assert dream.get("date") == DREAM_DATE + assert dream.get("files_changed", 0) >= 1, dream + assert dream.get("units"), f"extract produced no units: {dream!r}" + assert dream.get("integrate_results"), f"integrate produced no results: {dream!r}" + assert dream.get("checkpoint_paths"), f"finish did not checkpoint changed paths: {dream!r}" + + day_index = env.vault_dir / "daily" / f"{DREAM_DATE}.md" + changed_note = env.vault_dir / DREAM_INPUT_PATH + interests = env.vault_dir / "daily" / DREAM_DATE / "interests.yaml" + catalog = env.vault_dir / "reme_metadata" / "file_catalog" / "dream.jsonl" + assert changed_note.is_file(), f"changed note missing: {changed_note}" + assert interests.is_file(), f"interests.yaml missing: {interests}" + assert catalog.is_file(), f"dream catalog missing: {catalog}" + + after_digest = _all_digest_text(env) + new_signal = [ + needle + for needle in ("SOC2", "24", "current_kid", "Redis", "300", "next steps", "kid") + if needle in after_digest and needle not in before_digest + ] + print(f"[dream] new digest signals: {new_signal}") + assert len(new_signal) >= 2, f"digest missed expected new signal\n--- digest ---\n{after_digest}" + + # wikilink: the seeded daily note cites existing digest nodes, so it + # should have outbound wikilink edges in the file_graph, and the dream + # integrate agents should emit provenance and digest↔digest wikilinks + # in the markdown they just created or updated. + graph_links = _file_graph_links(env) + note_links = [ + link.get("target_path") for link in graph_links.get(DREAM_INPUT_PATH, []) if isinstance(link, dict) + ] + print(f"[wikilink] {DREAM_INPUT_PATH} -> {note_links}") + assert note_links, ( + f"seeded daily note produced no wikilink out-edges in file_graph\n" + f"file_graph links: {graph_links}" + ) + assert any( + str(target).lstrip("!").startswith("digest/") for target in note_links + ), f"daily note did not link out to any digest node: {note_links}" + + target_paths = [ + str(result.get("target_path") or "") + for result in dream.get("integrate_results", []) + if result.get("target_path") + ] + target_texts = { + rel: _read_text(env.vault_dir / rel) for rel in target_paths if (env.vault_dir / rel).is_file() + } + digest_wikilinks = [rel for rel, text in target_texts.items() if "[[digest/" in text] + provenance_links = [ + rel for rel, text in target_texts.items() if f"derived_from:: [[{DREAM_INPUT_PATH}]]" in text + ] + print(f"[wikilink] integrated targets: {target_paths}") + print(f"[wikilink] integrated targets with [[digest/...]] links: {digest_wikilinks}") + print(f"[wikilink] integrated targets with derived_from source links: {provenance_links}") + assert target_texts, f"no integrated target files found: {target_paths}" + assert provenance_links, ( + "no derived_from wikilink back to the changed daily note in integrated targets\n" + f"targets: {target_paths}" + ) + assert digest_wikilinks, ( + "no digest↔digest wikilink found in integrated target markdown\n" f"targets: {target_paths}" + ) + + interests_text = _print_text_file("interests.yaml", interests) + interests_data = yaml.safe_load(interests_text) or {} + topics = interests_data.get("topics") or [] + assert isinstance(topics, list) and topics, f"no topics in interests.yaml\n{interests_text}" + + proactive = await app.run_job("proactive", date=DREAM_DATE, include_content=True) + assert proactive.success is True, f"proactive failed: {proactive.answer!r}" + assert proactive.metadata.get("path") == f"daily/{DREAM_DATE}/interests.yaml" + assert proactive.metadata.get("topics"), f"proactive returned no topics: {proactive.metadata!r}" + + if day_index.is_file(): + _print_text_file("day_index.md", day_index) + else: + print(f"[day_index] skipped: no direct daily notes, so {day_index} was not created") + _print_text_file("changed input.md", changed_note) + for path in env.digest_files(): + _print_text_file(f"digest {path.relative_to(env.vault_dir)}", path) + _print_text_file("dream catalog.jsonl", catalog) + finally: + copied = _copy_outputs(env, message_files) + print("\n" + "=" * 70) + print(f"[artifacts] copied {len(copied)} item(s) to {ARTIFACT_DIR}") + for path in copied: + print(f"[artifacts] {path}") + print("=" * 70) + await env.close_all() + + asyncio.run(run()) + + +if __name__ == "__main__": + print("=== auto_dream integration test ===") + test_auto_dream_and_proactive() + print("\nIntegration test passed!") diff --git a/tests4/integration/test_auto_memory.py b/tests4/integration/test_auto_memory.py index a1f0d4c5..4592658c 100644 --- a/tests4/integration/test_auto_memory.py +++ b/tests4/integration/test_auto_memory.py @@ -141,9 +141,7 @@ def test_auto_memory_create(): print("=" * 70) pytorch_session_id = "pytorch-distributed-training" - # daily_create stamps the file as ``session_.md``, - # so the path metadata daily_create returns carries the prefix. - expected_stem = f"session_{pytorch_session_id}" + expected_stem = pytorch_session_id with env.record_agents(prefix="agent_create") as recorder: response = await app.run_job( "auto_memory", @@ -200,10 +198,7 @@ def test_auto_memory_update(): app = await env.make_app() try: today = env.today - # daily_create resolves session_id=SEED_STEM to file stem - # ``session_`` — seed at that exact stem so the - # UPDATE branch finds the existing note rather than CREATE. - expected_stem = f"session_{SEED_STEM}" + expected_stem = SEED_STEM seed_path = env.seed_daily_note(expected_stem, SEED_BODY) seed_before = _read_text(seed_path) assert "legal/compliance" in seed_before diff --git a/tests4/integration/test_auto_resource.py b/tests4/integration/test_auto_resource.py index 67a81373..724e3534 100644 --- a/tests4/integration/test_auto_resource.py +++ b/tests4/integration/test_auto_resource.py @@ -3,19 +3,12 @@ Drives the ``auto_resource`` step against a real LLM. Three scenarios: 1. **CREATE (added)** / **UPDATE (modified)**: places a resource file in - ``resource/{date}/``, calls ``auto_resource`` with the matching change. - The current AS-backed step only captures the agent's read+reason - transcript to ``resource/{date}/session_state_{sid}.jsonl`` — daily-note - writing is handled by whichever ``auto_memory_*`` step comes next in - the chain (the CC variant fork-writes from session_id; the AS variant - needs explicit messages and is wired separately). So this test asserts - on the session_state landing + agent fact coverage, not on a daily - note file. + ``resource/{date}/``, calls ``auto_resource`` with a ``changes`` batch, + and expects the agent to write/update the same-name daily note. 2. **DELETE (deleted)**: seeds a resource note under - ``daily/{date}/session_{sid}.md``, calls ``auto_resource`` with - change="deleted". Expects the note file to be removed (the step - stamps ``path`` on its metadata only in this branch). + ``daily/{date}/{resource_stem}.md``, calls ``auto_resource`` with a + deleted change, and expects the note file to be removed. Requires LLM_API_KEY (and optionally LLM_BASE_URL / LLM_MODEL_NAME) in the environment or a .env file at the repo root. Hits the real LLM API. @@ -31,7 +24,7 @@ sys.path.insert(0, str(INTEGRATION_DIR)) # pylint: disable=wrong-import-position from _vault_fixture import vault_env # noqa: E402 -from reme4.steps.evolve.auto_resource import _compute_session_id # noqa: E402 +from reme4.steps.evolve.auto_resource import _compute_agent_session_id, _compute_note_stem # noqa: E402 RESOURCE_FILENAME = "project-roadmap.md" RESOURCE_CONTENT_V1 = """\ @@ -83,13 +76,29 @@ def _read_text(p: Path) -> str: return p.read_text(encoding="utf-8") -def test_auto_resource_create(): - """CREATE branch: agent reads the resource file and saves session_state. +def _print_text_file(label: str, path: Path) -> str: + text = _read_text(path) + print("\n" + "=" * 70) + print(f"[{label}] {path} ({len(text)} bytes)") + print(f"[{label}] body:\n{text}") + print("=" * 70) + return text - Daily-note writing belongs to the auto_memory_* step that follows; this - test only asserts that auto_resource_step ran the agent and persisted - its transcript under ``resource/{date}/session_state_{sid}.jsonl``. - """ + +def _print_message_files(label: str, paths: list[Path]) -> None: + for idx, path in enumerate(paths, 1): + if not path.is_file(): + print(f"[{label}] message file missing: {path}") + continue + text = _read_text(path) + print("\n" + "=" * 70) + print(f"[{label}] message {idx}: {path} ({len(text)} bytes)") + print(text) + print("=" * 70) + + +def test_auto_resource_create(): + """CREATE branch: agent writes the same-name daily note and saves its AgentScope session.""" async def run(): with vault_env() as env: @@ -103,18 +112,19 @@ def test_auto_resource_create(): print("=" * 70) file_path = env.place_resource(RESOURCE_FILENAME, RESOURCE_CONTENT_V1) - session_id = _compute_session_id(RESOURCE_FILENAME) - expected_session_jsonl = env.vault_dir / "resource" / today / f"session_reme_{session_id}.jsonl" + note_stem = _compute_note_stem(RESOURCE_FILENAME) + agent_session_id = _compute_agent_session_id(file_path) + expected_session_jsonl = env.vault_dir / "reme_session" / "agentscope" / f"{agent_session_id}.jsonl" print(f"[CREATE] file_path = {file_path}") - print(f"[CREATE] session_id = {session_id}") + print(f"[CREATE] note_stem = {note_stem}") + print(f"[CREATE] agent_session_id = {agent_session_id}") print(f"[CREATE] expected transcript = {expected_session_jsonl.relative_to(env.vault_dir)}") with env.record_agents(prefix="agent_resource_create") as recorder: response = await app.run_job( "auto_resource", - file_path=file_path, - change="added", + changes=[{"path": file_path, "change": "added"}], ) dumped = await recorder.dump() for p in dumped: @@ -122,21 +132,33 @@ def test_auto_resource_create(): assert response.success is True, f"CREATE job failed: {response.answer!r}" meta = response.metadata or {} - assert meta.get("action") == "added", f"Unexpected action: {meta!r}" - assert meta.get("session_id") == session_id, f"Unexpected session_id: {meta!r}" + result_meta = (meta.get("results") or [{}])[0].get("metadata") or {} + assert result_meta.get("action") == "added", f"Unexpected action: {meta!r}" + assert result_meta.get("session_id") == note_stem, f"Unexpected session_id: {meta!r}" + assert result_meta.get("path") == f"daily/{today}/{note_stem}.md", f"Unexpected note path: {meta!r}" + note_path = env.vault_dir / "daily" / today / f"{note_stem}.md" + assert note_path.is_file() assert expected_session_jsonl.is_file(), ( - f"agent session_state not persisted at {expected_session_jsonl}; " - f"session_state files under resource/: " - f"{[p.name for p in env.session_state_files(prefix='session_reme_')]}" + f"agent session not persisted at {expected_session_jsonl}; " + f"AgentScope files: " + f"{[p.name for p in (env.vault_dir / 'reme_session' / 'agentscope').glob('*.jsonl')]}" ) - # Read the agent transcript and check it actually opened - # the resource file (the file_path should show up in a - # tool-call argument) so we know the step did its job. + note_text = _print_text_file("CREATE result.md", note_path) + _print_message_files("CREATE intermediate messages", [*dumped, expected_session_jsonl]) + + note_hits = [ + needle + for needle in ("v2.0", "July 15", "Alice", "Bob", "p99", "200ms", "Redis") + if needle in note_text + ] + print(f"[CREATE] landed note facts: {note_hits}") + assert ( + len(note_hits) >= 3 + ), f"CREATE note missed expected facts {note_hits!r}\n--- NOTE ---\n{note_text}" + transcript = _read_text(expected_session_jsonl) - print("\n" + "=" * 70) - print(f"[CREATE] {expected_session_jsonl.name} ({len(transcript)} bytes)") topic_hits = [ needle for needle in ("v2.0", "July 15", "Alice", "Bob", "p99", "200ms", "Redis", file_path) @@ -158,7 +180,7 @@ def test_auto_resource_create(): def test_auto_resource_update(): - """UPDATE branch: same contract as CREATE — only session_state changes.""" + """UPDATE branch: agent updates the same-name daily note and appends to its AgentScope session.""" async def run(): with vault_env() as env: @@ -174,12 +196,13 @@ def test_auto_resource_update(): # First run as "added" so the resource file exists and the # initial transcript lands. file_path = env.place_resource(RESOURCE_FILENAME, RESOURCE_CONTENT_V1) - session_id = _compute_session_id(RESOURCE_FILENAME) - session_jsonl = env.vault_dir / "resource" / today / f"session_reme_{session_id}.jsonl" + note_stem = _compute_note_stem(RESOURCE_FILENAME) + agent_session_id = _compute_agent_session_id(file_path) + session_jsonl = env.vault_dir / "reme_session" / "agentscope" / f"{agent_session_id}.jsonl" - response = await app.run_job("auto_resource", file_path=file_path, change="added") + response = await app.run_job("auto_resource", changes=[{"path": file_path, "change": "added"}]) assert response.success is True, f"Initial create failed: {response.answer!r}" - assert session_jsonl.is_file(), "initial added run did not save session_state" + assert session_jsonl.is_file(), "initial added run did not save the AgentScope session" size_before = session_jsonl.stat().st_size print(f"[UPDATE] transcript before modify ({size_before} bytes)") @@ -189,8 +212,7 @@ def test_auto_resource_update(): with env.record_agents(prefix="agent_resource_update") as recorder: response = await app.run_job( "auto_resource", - file_path=file_path, - change="modified", + changes=[{"path": file_path, "change": "modified"}], ) dumped = await recorder.dump() for p in dumped: @@ -198,8 +220,12 @@ def test_auto_resource_update(): assert response.success is True, f"UPDATE job failed: {response.answer!r}" meta = response.metadata or {} - assert meta.get("action") == "modified", f"Unexpected action: {meta!r}" - assert meta.get("session_id") == session_id, f"Unexpected session_id: {meta!r}" + result_meta = (meta.get("results") or [{}])[0].get("metadata") or {} + assert result_meta.get("action") == "modified", f"Unexpected action: {meta!r}" + assert result_meta.get("session_id") == note_stem, f"Unexpected session_id: {meta!r}" + assert result_meta.get("path") == f"daily/{today}/{note_stem}.md", f"Unexpected note path: {meta!r}" + note_path = env.vault_dir / "daily" / today / f"{note_stem}.md" + assert note_path.is_file() size_after = session_jsonl.stat().st_size print(f"[UPDATE] transcript after modify ({size_after} bytes)") @@ -207,6 +233,19 @@ def test_auto_resource_update(): f"transcript did not grow after modified run " f"({size_before} -> {size_after})" ) + note_text = _print_text_file("UPDATE result.md", note_path) + _print_message_files("UPDATE intermediate messages", [*dumped, session_jsonl]) + + note_hits = [ + needle + for needle in ("July 20", "150ms", "Dave", "rate limiting", "resolved") + if needle in note_text + ] + print(f"[UPDATE] landed note facts: {note_hits}") + assert ( + len(note_hits) >= 2 + ), f"UPDATE note missed expected facts {note_hits!r}\n--- NOTE ---\n{note_text}" + transcript = _read_text(session_jsonl) new_hits = [ needle @@ -239,25 +278,23 @@ def test_auto_resource_delete(): print("[setup] today =", today) print("=" * 70) - session_id = _compute_session_id(RESOURCE_FILENAME) + note_stem = _compute_note_stem(RESOURCE_FILENAME) file_path = f"resource/{today}/{RESOURCE_FILENAME}" - # Seed the note file (daily_create prepends "session_agent_") - note_filename = f"session_agent_{session_id}" seed_body = "---\nname: test\ndescription: test note\n---\n\nSome content.\n" - note_path = env.seed_daily_note(note_filename, seed_body) + note_path = env.seed_daily_note(note_stem, seed_body) assert note_path.is_file() print(f"[DELETE] seeded note: {note_path}") response = await app.run_job( "auto_resource", - file_path=file_path, - change="deleted", + changes=[{"path": file_path, "change": "deleted"}], ) assert response.success is True, f"DELETE job failed: {response.answer!r}" meta = response.metadata or {} - assert meta.get("action") == "deleted" + result_meta = (meta.get("results") or [{}])[0].get("metadata") or {} + assert result_meta.get("action") == "deleted" assert not note_path.is_file(), f"Note file still exists after delete: {note_path}" print(f"[DELETE] note removed: {note_path}") diff --git a/tests4/integration/test_dreamer_inproc.py b/tests4/integration/test_dreamer_inproc.py deleted file mode 100644 index d0514f78..00000000 --- a/tests4/integration/test_dreamer_inproc.py +++ /dev/null @@ -1,92 +0,0 @@ -"""dreamer in-process integration test. - -Loads the default reme4 config, seeds a rich workspace via the shared -``vault_env`` fixture (pre-existing digest nodes spread across the three -buckets + a new daily that exercises CREATE and UPDATE in each bucket), -reindexes so search can hit the pre-existing nodes, then calls ``dream`` -and prints what happened. - -Phase 1 classifies each sub-unit into one of {procedure, personal, -wiki}; Phase 2 dispatches to the bucket-specific integrate prompt -and writes via the canonical ``write`` / ``edit`` tools. - -Usage (from anywhere): - python tests4/integration/test_dreamer_inproc.py - python tests4/integration/test_dreamer_inproc.py \\ - daily/2026-05-28/auth-refactor/notes.md - -Each run wipes ``daily/``, ``digest/``, ``resource/``, and -``reme_metadata/`` under a freshly-built throwaway vault before -reseeding, so the dreamer always starts from the same fixture state. -See ``_vault_fixture.py`` (``seed_dream_vault`` / ``DREAM_INPUT_PATH``) -for what gets created and the expected CREATE / UPDATE landings per -bucket. - -Required env (from .env or shell): - LLM_API_KEY, LLM_BASE_URL, LLM_MODEL_NAME — for the Phase 1/2 agents -""" - -import asyncio -import sys -from pathlib import Path - -# Make ``_vault_fixture`` importable as a top-level module regardless of -# the caller's cwd. -INTEGRATION_DIR = Path(__file__).resolve().parent -sys.path.insert(0, str(INTEGRATION_DIR)) - -# pylint: disable=wrong-import-position -from _vault_fixture import DREAM_INPUT_PATH, vault_env # noqa: E402 - - -async def main() -> None: - """Seed a vault, reindex it, run ``dream`` on the seeded daily note.""" - rel_input = sys.argv[1] if len(sys.argv) > 1 else DREAM_INPUT_PATH - - with vault_env() as env: - seeded = env.seed_dream_vault() - print(f"--- seeded {len(seeded)} fixture file(s) under {env.vault_dir}") - - app = await env.make_reme() - print(f"--- vault_dir: {env.vault_dir}") - print(f"--- input: {rel_input}") - - try: - # Reindex first so search can actually find the pre-seeded - # digest/ nodes — otherwise Phase 2 recall returns empty and - # every sub-unit ends up as CREATE (UPDATE path not exercised). - print("\n--- reindexing vault so Phase 2 recall has something to hit") - await app.run_job("reindex") - - print(f"\n--- running dream path={rel_input}") - with env.record_agents(prefix="dream") as recorder: - resp = await app.run_job("dream", path=rel_input) - dumped = await recorder.dump() - print(f"\n--- dumped {len(dumped)} agent memory file(s) to {recorder.dump_dir}") - for p in dumped: - print(f" {p.relative_to(recorder.dump_dir)}") - - print("\n=== Response.success ===") - print(resp.success) - print("\n=== Response.answer ===") - print(resp.answer) - print("\n=== Response.metadata (DreamResult fields) ===") - for k, v in (resp.metadata or {}).items(): - if isinstance(v, list) and len(v) > 8: - print(f" {k}: list({len(v)} items) head={v[:3]!r}") - else: - print(f" {k}: {v!r}") - finally: - await env.close_all() - - print("\n=== digest/ tree after dream ===") - digest_files = env.digest_files() - if not digest_files: - print(" (no digest files)") - for p in digest_files: - print(f"\n--- {p.relative_to(env.vault_dir)} ---") - # print(p.read_text(encoding="utf-8")) - - -if __name__ == "__main__": - asyncio.run(main()) diff --git a/tests4/integration/test_llm.py b/tests4/integration/test_llm.py index ff105a16..3607633a 100644 --- a/tests4/integration/test_llm.py +++ b/tests4/integration/test_llm.py @@ -59,7 +59,6 @@ async def _run_with_tool(app) -> None: response = await step( query="Use the add tool to compute 21 + 21 and report the result.", sys_prompt="Use the `add` tool whenever the user asks to add numbers.", - use_add_tool=True, ) text = (response.answer or "").strip() print(f"\n[with_tool] response: {text!r}") diff --git a/tests4/integration/test_stream_llm.py b/tests4/integration/test_stream_llm.py index 3605ca41..c01e94dc 100644 --- a/tests4/integration/test_stream_llm.py +++ b/tests4/integration/test_stream_llm.py @@ -78,7 +78,6 @@ async def _test_stream_llm_with_tool(): stream_queue=queue, query="Use the add tool to compute 21 + 21 and report the result.", sys_prompt="Use the `add` tool whenever the user asks to add numbers.", - use_add_tool=True, ), ) diff --git a/tests4/unit/test_auto_dream.py b/tests4/unit/test_auto_dream.py index c783f0d7..189461a3 100644 --- a/tests4/unit/test_auto_dream.py +++ b/tests4/unit/test_auto_dream.py @@ -1,373 +1,95 @@ -"""Tests for AutoDreamStep — daily-tick + file_catalog dedup. - -AutoDreamStep walks ``daily/.md`` + ``daily//**`` and -diffs the result against ``file_catalog`` (same shape as -``scan_catalog_changes_step``): - -* on-disk path missing from catalog → dream -* on-disk mtime != catalog mtime → dream (modified) -* on-disk mtime == catalog mtime → skip (unchanged) -* catalog entry under today's prefix missing → drop from catalog (deleted) - -Successful dreams (and Phase 1 vacuous skips) upsert the current -``st_mtime`` so the next tick re-dreams only what actually changed. -Failures leave the catalog untouched. - -We mock ``run_job`` (the dispatch hop that calls the configured -``dream`` job — needs an LLM) and inject a fake ``file_catalog`` -recording every get / upsert / delete / dump. -""" - -# pylint: disable=protected-access +"""Unit tests for the refactored dream package.""" import asyncio -import os import tempfile -import warnings from pathlib import Path -from unittest.mock import AsyncMock, MagicMock, patch from reme4.components.file_catalog import BaseFileCatalog from reme4.components.runtime_context import RuntimeContext -from reme4.schema import FileNode, Response -from reme4.steps import AutoDreamStep -from reme4.steps.evolve.dream import DreamResult - -warnings.filterwarnings("ignore", category=DeprecationWarning, module="jieba") -warnings.filterwarnings("ignore", category=DeprecationWarning, module="pkg_resources") +from reme4.steps.evolve.dream.finish import DreamFinishStep +from reme4.steps.evolve.dream.schema import DreamState +from reme4.steps.evolve.dream.utils import parse_structured_reply, scan_day_files -def _dream_response(path: str, **dream_fields) -> Response: - """Wrap a DreamResult as the dispatched job would return it. - - Mirrors what DreamStep.execute does: - self.context.response.metadata.update(result.model_dump()) - """ - dr = DreamResult(path=path, **dream_fields) - success = not dr.error - return Response(success=success, answer=dr.summary or "ok", metadata=dr.model_dump()) - - -class temp_chdir: - """Context manager that temporarily ``chdir``s into a directory and restores cwd on exit.""" - - def __init__(self, path): - self.path = path - self.old = None - - def __enter__(self): - self.old = os.getcwd() - os.chdir(self.path) - return self - - def __exit__(self, *exc): - os.chdir(self.old) - - -def _touch(path: Path, content: str = "x") -> Path: +def _touch(path: Path, text: str = "x") -> Path: path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(content, encoding="utf-8") + path.write_text(text, encoding="utf-8") return path -def _make_step(vault: Path, today: str, existing_nodes: list[FileNode] | None = None) -> AutoDreamStep: - """AutoDreamStep with vault forced, daily_dir='daily', and a fake catalog.""" - fake_catalog = MagicMock(spec=BaseFileCatalog) - fake_catalog.get_nodes = AsyncMock(return_value=list(existing_nodes or [])) - fake_catalog.upsert = AsyncMock() - fake_catalog.delete = AsyncMock() - fake_catalog.dump = AsyncMock() +class _Catalog(BaseFileCatalog): + def __init__(self): + super().__init__() + self.upserts = [] + self.dumps = 0 - class _Fixed(AutoDreamStep): - @property - def vault_path(self): - return vault + async def upsert(self, nodes): + self.upserts.extend(nodes) - def _vault_dir(self): - return vault + async def delete(self, path): + return None - def _now(self): - import datetime + async def get_nodes(self, paths=None): + return [] - return datetime.datetime.fromisoformat(f"{today}T00:00:00") - - step = _Fixed(file_catalog=fake_catalog, persist=True) - cfg = MagicMock() - cfg.daily_dir = "daily" - cfg.resource_dir = "" - step.app_context = MagicMock() - step.app_context.app_config = cfg - return step + async def dump(self): + self.dumps += 1 -def test_scans_date_md_and_date_folder(): - """Both ``daily/.md`` and files under ``daily//`` are picked up; - date.md is dreamed first so the day-index leads.""" +def test_scan_day_files_includes_nested_md_and_excludes_interests(): + """Scan day files.""" + with tempfile.TemporaryDirectory() as tmp: + vault = Path(tmp) + _touch(vault / "daily" / "2026-05-28.md") + _touch(vault / "daily" / "2026-05-28" / "session.md") + _touch(vault / "daily" / "2026-05-28" / "auth-refactor" / "notes.md") + _touch(vault / "daily" / "2026-05-28" / "interests.yaml") + + assert scan_day_files(vault, "2026-05-28", "daily") == [ + "daily/2026-05-28.md", + "daily/2026-05-28/auth-refactor/notes.md", + "daily/2026-05-28/session.md", + ] + + +def test_parse_structured_reply_handles_fenced_yaml_and_scalar_fallback(): + """Parse a JSON/YAML object from an agent reply, including fenced blocks.""" + data = parse_structured_reply( + "```yaml\n" + "action: REFINE\n" + "target_path: digest/personal/no-trailing-summary.md\n" + "note: Extended node. Core rule unchanged: answer directly and stop.\n" + "```", + ) + assert data["action"] == "REFINE" + assert data["target_path"] == "digest/personal/no-trailing-summary.md" + assert data["note"].startswith("Extended node") + + +def test_finish_does_not_checkpoint_failed_changed_paths(): + """Finish does not checkpoint failed changed paths.""" async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - _touch(vault / "daily" / f"{today}.md") - _touch(vault / "daily" / today / "session-a.md") - _touch(vault / "daily" / today / "session-b.md") - step = _make_step(vault, today) - ctx = RuntimeContext(date=today) + with tempfile.TemporaryDirectory() as tmp: + vault = Path(tmp) + ok = _touch(vault / "daily" / "2026-05-28" / "ok.md") + failed = _touch(vault / "daily" / "2026-05-28" / "failed.md") + interests = _touch(vault / "daily" / "2026-05-28" / "interests.yaml") + state = DreamState( + date="2026-05-28", + vault=str(vault), + changed_paths=[str(ok.relative_to(vault)), str(failed.relative_to(vault))], + failed_paths=[str(failed.relative_to(vault))], + interests_path=str(interests.relative_to(vault)), + ) + step, catalog = DreamFinishStep(), _Catalog() + resp = await step(RuntimeContext(dream=state.model_dump(), file_catalog=catalog)) - seen: list[str] = [] - - async def _fake_run_job(name, **kwargs): - assert name == "dream", f"expected dispatch to 'dream' job, got {name!r}" - seen.append(kwargs["path"]) - return _dream_response(kwargs["path"], used_llm=True, summary="ok") - - with patch.object(step, "run_job", side_effect=_fake_run_job): - resp = await step(ctx) - - assert resp.success - assert resp.metadata["files_scanned"] == 3 - assert resp.metadata["files_dreamed"] == 3 - assert seen[0] == f"daily/{today}.md" - assert seen[1:] == [f"daily/{today}/session-a.md", f"daily/{today}/session-b.md"] - print("✓ test_scans_date_md_and_date_folder passed") + upserted = [n.path for n in catalog.upserts] + assert resp.success is True + assert str(ok.relative_to(vault)) in upserted + assert str(failed.relative_to(vault)) not in upserted + assert str(interests.relative_to(vault)) in upserted + assert catalog.dumps == 1 asyncio.run(run()) - - -def test_resource_dir_is_not_scanned(): - """resource// files are NOT picked up by AutoDreamStep anymore.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - _touch(vault / "daily" / f"{today}.md") - _touch(vault / "resource" / today / "spec.pdf") - step = _make_step(vault, today) - step.app_context.app_config.resource_dir = "resource" - ctx = RuntimeContext(date=today) - - async def _fake_run_job(_name, **kwargs): - return _dream_response(kwargs["path"], used_llm=True, summary="ok") - - with patch.object(step, "run_job", side_effect=_fake_run_job) as run_job_mock: - await step(ctx) - - paths = [c.kwargs["path"] for c in run_job_mock.call_args_list] - assert paths == [f"daily/{today}.md"] - print("✓ test_resource_dir_is_not_scanned passed") - - asyncio.run(run()) - - -def test_unchanged_files_skipped_via_catalog_mtime(): - """A file whose catalog mtime matches on-disk mtime is NOT dreamed.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - note = _touch(vault / "daily" / today / "note.md") - mtime = note.stat().st_mtime - existing = [FileNode(path=f"daily/{today}/note.md", st_mtime=mtime)] - step = _make_step(vault, today, existing_nodes=existing) - ctx = RuntimeContext(date=today) - - with patch.object(step, "run_job") as run_job_mock: - resp = await step(ctx) - run_job_mock.assert_not_called() - - assert resp.success - assert resp.metadata["files_unchanged"] == 1 - assert resp.metadata["files_dreamed"] == 0 - step.file_catalog.upsert.assert_not_awaited() - step.file_catalog.dump.assert_not_awaited() - print("✓ test_unchanged_files_skipped_via_catalog_mtime passed") - - asyncio.run(run()) - - -def test_changed_file_dreamed_and_catalog_updated(): - """Stale catalog mtime → file is dreamed + catalog upserts new mtime.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - note = _touch(vault / "daily" / today / "note.md") - mtime = note.stat().st_mtime - stale = mtime - 999.0 - existing = [FileNode(path=f"daily/{today}/note.md", st_mtime=stale)] - step = _make_step(vault, today, existing_nodes=existing) - ctx = RuntimeContext(date=today) - - async def _fake_run_job(_name, **kwargs): - return _dream_response(kwargs["path"], used_llm=True, summary="ok") - - with patch.object(step, "run_job", side_effect=_fake_run_job): - resp = await step(ctx) - - assert resp.success - assert resp.metadata["files_dreamed"] == 1 - assert resp.metadata["files_unchanged"] == 0 - step.file_catalog.upsert.assert_awaited_once() - (nodes,), _ = step.file_catalog.upsert.call_args - assert len(nodes) == 1 - assert nodes[0].path == f"daily/{today}/note.md" - assert nodes[0].st_mtime == mtime # post-dream mtime, not stale - step.file_catalog.dump.assert_awaited_once() - print("✓ test_changed_file_dreamed_and_catalog_updated passed") - - asyncio.run(run()) - - -def test_deleted_file_dropped_from_catalog(): - """Catalog entry under today's prefix with no on-disk file → catalog.delete.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - # No on-disk files for today; catalog has a stale entry for today. - existing = [FileNode(path=f"daily/{today}/gone.md", st_mtime=123.0)] - step = _make_step(vault, today, existing_nodes=existing) - ctx = RuntimeContext(date=today) - - with patch.object(step, "run_job") as run_job_mock: - resp = await step(ctx) - run_job_mock.assert_not_called() - - assert resp.success - assert resp.metadata["files_deleted"] == 1 - step.file_catalog.delete.assert_awaited_once_with([f"daily/{today}/gone.md"]) - step.file_catalog.upsert.assert_not_awaited() - step.file_catalog.dump.assert_awaited_once() - print("✓ test_deleted_file_dropped_from_catalog passed") - - asyncio.run(run()) - - -def test_other_days_catalog_entries_untouched(): - """Catalog entries OUTSIDE today's prefix must not be dropped, even if - they don't appear in today's on-disk scan.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - yesterday = "2026-06-03" - # Today: nothing on disk. Catalog has yesterday's entry. - existing = [FileNode(path=f"daily/{yesterday}/note.md", st_mtime=99.0)] - step = _make_step(vault, today, existing_nodes=existing) - ctx = RuntimeContext(date=today) - - resp = await step(ctx) - - assert resp.success - assert resp.metadata["files_scanned"] == 0 - assert resp.metadata["files_deleted"] == 0 - step.file_catalog.delete.assert_not_awaited() - step.file_catalog.upsert.assert_not_awaited() - print("✓ test_other_days_catalog_entries_untouched passed") - - asyncio.run(run()) - - -def test_failure_does_not_upsert(): - """A dream error must leave the catalog untouched so the next tick retries.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - _touch(vault / "daily" / today / "note.md") - step = _make_step(vault, today) - ctx = RuntimeContext(date=today) - - async def _fake_run_job(_name, **kwargs): - return _dream_response(kwargs["path"], used_llm=False, error="boom") - - with patch.object(step, "run_job", side_effect=_fake_run_job): - resp = await step(ctx) - - assert not resp.success - assert resp.metadata["files_failed"] == 1 - step.file_catalog.upsert.assert_not_awaited() - step.file_catalog.dump.assert_not_awaited() - print("✓ test_failure_does_not_upsert passed") - - asyncio.run(run()) - - -def test_phase1_empty_still_upserts(): - """Phase 1 skipped (no abstractions) is still success — still catalogued so - the next tick doesn't redo Phase 1 unnecessarily.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - note = _touch(vault / "daily" / today / "note.md") - mtime = note.stat().st_mtime - step = _make_step(vault, today) - ctx = RuntimeContext(date=today) - - async def _fake_run_job(_name, **kwargs): - return _dream_response(kwargs["path"], used_llm=True, skipped=True, summary="empty") - - with patch.object(step, "run_job", side_effect=_fake_run_job): - resp = await step(ctx) - - assert resp.success - assert resp.metadata["files_skipped"] == 1 - step.file_catalog.upsert.assert_awaited_once() - (nodes,), _ = step.file_catalog.upsert.call_args - assert nodes[0].st_mtime == mtime - print("✓ test_phase1_empty_still_upserts passed") - - asyncio.run(run()) - - -def test_partial_failure_does_not_block_other_files(): - """One file's failure must not stop other files from being dreamed + catalogued.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - vault = Path(tmpdir).resolve() - today = "2026-06-04" - _touch(vault / "daily" / today / "a.md") - _touch(vault / "daily" / today / "b.md") - step = _make_step(vault, today) - ctx = RuntimeContext(date=today) - - async def _fake_run_job(_name, **kwargs): - if kwargs["path"].endswith("a.md"): - return _dream_response(kwargs["path"], used_llm=False, error="boom") - return _dream_response(kwargs["path"], used_llm=True, summary="ok") - - with patch.object(step, "run_job", side_effect=_fake_run_job): - resp = await step(ctx) - - assert not resp.success - assert resp.metadata["files_dreamed"] == 1 - assert resp.metadata["files_failed"] == 1 - step.file_catalog.upsert.assert_awaited_once() - (nodes,), _ = step.file_catalog.upsert.call_args - assert [n.path for n in nodes] == [f"daily/{today}/b.md"] - print("✓ test_partial_failure_does_not_block_other_files passed") - - asyncio.run(run()) - - -if __name__ == "__main__": - print("\n=== AutoDreamStep Tests ===") - test_scans_date_md_and_date_folder() - test_resource_dir_is_not_scanned() - test_unchanged_files_skipped_via_catalog_mtime() - test_changed_file_dreamed_and_catalog_updated() - test_deleted_file_dropped_from_catalog() - test_other_days_catalog_entries_untouched() - test_failure_does_not_upsert() - test_phase1_empty_still_upserts() - test_partial_failure_does_not_block_other_files() - print("\n所有测试通过!") diff --git a/tests4/unit/test_background_steps.py b/tests4/unit/test_background_steps.py index 515945a0..141c46fc 100644 --- a/tests4/unit/test_background_steps.py +++ b/tests4/unit/test_background_steps.py @@ -4,7 +4,7 @@ Both scan steps are subclasses of BaseStep. To exercise them without spinning up the full ApplicationContext, we pass real (started) file_store/file_chunker via the step's kwargs (so the BaseStep _resolve() machinery returns them). -ScanStoreChangesStep writes its result into ``context["changes"]`` for a +InitChangesStep writes its result into ``context["changes"]`` for a downstream ``update_index_step`` to consume; tests assert against that key. """ @@ -15,14 +15,27 @@ import os import tempfile import warnings from pathlib import Path -from unittest.mock import AsyncMock, MagicMock +from unittest.mock import MagicMock from watchfiles import Change from reme4.components.file_chunker import DefaultFileChunker +from reme4.components.file_catalog import LocalFileCatalog from reme4.components.file_store import LocalFileStore from reme4.components.runtime_context import RuntimeContext -from reme4.steps import ForeachDispatchStep, LogChangesStep, ScanStoreChangesStep, WatchChangesStep +from reme4.enumeration import ComponentEnum +from reme4.steps.evolve.auto_resource import AutoResourceStep, _compute_note_stem +from reme4.steps.index import ( + DEFAULT_LOW_POWER_POLL_MS, + DEFAULT_WATCH_DEBOUNCE_MS, + DEFAULT_WATCH_STEP_MS, + ClearStoreStep, + InitChangesStep, + LogChangesStep, + UpdateCatalogStep, + WatchChangesStep, +) +from reme4.steps.index._change_batch import bucket_changes from reme4.steps.index._watch_rules import WatchRule, build_watch_rules, collect_existing, match_file warnings.filterwarnings("ignore", category=DeprecationWarning, module="jieba") @@ -155,17 +168,51 @@ def test_collect_existing_filters(): # --------------------------------------------------------------------------- -# ScanStoreChangesStep +# InitChangesStep # --------------------------------------------------------------------------- +def test_clear_and_scan_defaults_include_jsonl(): + """Full reindex should include jsonl files when no explicit suffix filter is passed.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): + cwd = Path.cwd() + write_file(cwd / "daily" / "note.md", "alpha") + write_file(cwd / "resource" / "events.jsonl", '{"a": 1}\n') + write_file(cwd / "resource" / "ignore.txt", "skip") + + fs = LocalFileStore(name="test_store", embedding_store="") + await fs.start() + try: + clear_step = ClearStoreStep(file_store=fs, app_context=_make_app_context(cwd)) + scan_step = InitChangesStep(store="file_store", file_store=fs, app_context=_make_app_context(cwd)) + ctx = RuntimeContext(watch_dirs=["daily_dir", "resource_dir"], watch_suffixes=["md", "jsonl"]) + await clear_step(ctx) + resp = await scan_step(ctx) + paths = {Path(item["path"]).name for item in ctx["changes"]} + assert resp.metadata["counts"] == {"added": 2, "modified": 0, "deleted": 0} + assert paths == {"note.md", "events.jsonl"} + finally: + await fs.close() + print("✓ test_clear_and_scan_defaults_include_jsonl passed") + + asyncio.run(run()) + + async def _make_scan_step(vault_path: Path, watch_dirs=None, watch_suffixes=None, recursive=True): fs = LocalFileStore(name="test_store", embedding_store="") chunker = DefaultFileChunker() await fs.start() await chunker.start() app_ctx = _make_app_context(vault_path) - step = ScanStoreChangesStep(recursive=recursive, file_store=fs, file_chunker=chunker, app_context=app_ctx) + step = InitChangesStep( + store="file_store", + recursive=recursive, + file_store=fs, + file_chunker=chunker, + app_context=app_ctx, + ) context = RuntimeContext( watch_dirs=watch_dirs or ["daily_dir", "digest_dir"], watch_suffixes=watch_suffixes or ["md"], @@ -299,11 +346,220 @@ def test_scan_changes_resource_dir(): asyncio.run(run()) +def test_init_changes_named_file_catalog_monitor(): + """monitor_type/monitor_name selects the requested file_catalog component.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): + cwd = Path.cwd() + write_file(cwd / "resource" / "2026-01-01" / "a.md", "alpha") + + catalog = LocalFileCatalog(name="resource") + await catalog.start() + try: + app_ctx = _make_app_context(cwd) + app_ctx.components = {ComponentEnum.FILE_CATALOG: {"resource": catalog}} + step = InitChangesStep(monitor_type="file_catalog", monitor_name="resource", app_context=app_ctx) + ctx = RuntimeContext(watch_dirs=["resource_dir"], watch_suffixes=["md"]) + resp = await step(ctx) + + assert resp.metadata["counts"] == {"added": 1, "modified": 0, "deleted": 0} + assert ctx["changes"][0]["change"] == "added" + finally: + await catalog.close() + print("✓ test_init_changes_named_file_catalog_monitor passed") + + asyncio.run(run()) + + +def test_bucket_changes_coalesces_by_final_file_state(): + """A delete+add replacement batch for an existing file becomes one modified event.""" + with tempfile.TemporaryDirectory() as tmpdir: + p = write_file(Path(tmpdir) / "daily" / "a.md", "alpha") + buckets = bucket_changes( + [ + {"change": "deleted", "path": str(p)}, + {"change": "added", "path": str(p)}, + ], + ) + + assert buckets[Change.modified] == [str(p)] + assert buckets[Change.added] == [] + assert buckets[Change.deleted] == [] + print("✓ test_bucket_changes_coalesces_by_final_file_state passed") + + +def test_update_catalog_relative_path_uses_vault(): + """update_catalog_step resolves vault-relative change paths against vault_path.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): + cwd = Path.cwd() + write_file(cwd / "daily" / "a.md", "alpha") + + catalog = LocalFileCatalog(name="test_catalog") + await catalog.start() + try: + step = UpdateCatalogStep(file_catalog=catalog, app_context=_make_app_context(cwd)) + ctx = RuntimeContext(changes=[{"change": "added", "path": "daily/a.md"}]) + resp = await step(ctx) + + assert resp.success is True + nodes = await catalog.get_nodes() + assert [n.path for n in nodes] == ["daily/a.md"] + finally: + await catalog.close() + print("✓ test_update_catalog_relative_path_uses_vault passed") + + asyncio.run(run()) + + +def test_index_update_loop_init_dispatch_updates_store_across_batches(): + """index_update_loop init scan dispatches to update_index_step and preserves final store state.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): + cwd = Path.cwd() + daily_a = write_file(cwd / "daily" / "a.md", "alpha\n[[digest/report.md]]\n") + write_file(cwd / "digest" / "report.md", "# Report\nbeta\n") + write_file(cwd / "daily" / "ignore.txt", "skip") + + fs = LocalFileStore(name="default", embedding_store="") + chunker = DefaultFileChunker() + await fs.start() + await chunker.start() + try: + app_ctx = _make_app_context(cwd) + app_ctx.components = { + ComponentEnum.FILE_STORE: {"default": fs}, + ComponentEnum.FILE_CHUNKER: {"default": chunker}, + } + ctx = RuntimeContext(watch_dirs=["daily_dir", "digest_dir"], watch_suffixes=["md"]) + init_step = InitChangesStep( + monitor_type="file_store", + monitor_name="default", + dispatch_steps=["update_index_step"], + app_context=app_ctx, + ) + + first = await init_step(ctx) + assert first.metadata["counts"] == {"added": 2, "modified": 0, "deleted": 0} + nodes = {n.path: n for n in await fs.get_nodes()} + assert set(nodes) == {"daily/a.md", "digest/report.md"} + assert all(nodes[p].chunk_ids for p in nodes) + + daily_a.write_text("alpha v2\n[[digest/report.md]]\n", encoding="utf-8") + os.utime(daily_a, (9_999_999_999, 9_999_999_999)) + (cwd / "digest" / "report.md").unlink() + write_file(cwd / "daily" / "c.md", "gamma\n") + + second = await init_step(ctx) + assert second.metadata["counts"] == {"added": 1, "modified": 1, "deleted": 1} + assert {(c["change"], Path(c["path"]).name) for c in ctx["changes"]} == { + ("modified", "a.md"), + ("deleted", "report.md"), + ("added", "c.md"), + } + + nodes = {n.path: n for n in await fs.get_nodes()} + assert set(nodes) == {"daily/a.md", "daily/c.md"} + assert all(nodes[p].chunk_ids for p in nodes) + assert all(chunk.path in nodes for chunk in fs.file_chunks.values()) + finally: + await chunker.close() + await fs.close() + print("✓ test_index_update_loop_init_dispatch_updates_store_across_batches passed") + + asyncio.run(run()) + + +def test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs(): + """digest_watch_loop style config updates the digest catalog without touching resource/default catalogs.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): + cwd = Path.cwd() + daily = write_file(cwd / "daily" / "2026-01-01.md", "day one") + digest = write_file(cwd / "digest" / "week.md", "weekly") + write_file(cwd / "resource" / "asset.md", "not watched by digest loop") + + digest_catalog = LocalFileCatalog(name="digest") + resource_catalog = LocalFileCatalog(name="resource") + await digest_catalog.start() + await resource_catalog.start() + try: + app_ctx = _make_app_context(cwd) + app_ctx.components = { + ComponentEnum.FILE_CATALOG: { + "digest": digest_catalog, + "resource": resource_catalog, + }, + } + ctx = RuntimeContext(watch_dirs=["daily_dir", "digest_dir"], watch_suffixes=["md"]) + init_step = InitChangesStep( + monitor_type="file_catalog", + monitor_name="digest", + dispatch_steps=[ + {"backend": "update_catalog_step", "file_catalog": "digest"}, + {"backend": "log_changes_step"}, + ], + app_context=app_ctx, + ) + + first = await init_step(ctx) + assert first.metadata["counts"] == {"added": 2, "modified": 0, "deleted": 0} + assert {n.path for n in await digest_catalog.get_nodes()} == { + "daily/2026-01-01.md", + "digest/week.md", + } + assert await resource_catalog.get_nodes() == [] + + daily.write_text("day two", encoding="utf-8") + os.utime(daily, (9_999_999_999, 9_999_999_999)) + digest.unlink() + write_file(cwd / "daily" / "2026-01-02.md", "next day") + + second = await init_step(ctx) + assert second.metadata["counts"] == {"added": 1, "modified": 1, "deleted": 1} + assert {(c["change"], Path(c["path"]).name) for c in ctx["changes"]} == { + ("modified", "2026-01-01.md"), + ("deleted", "week.md"), + ("added", "2026-01-02.md"), + } + assert {n.path for n in await digest_catalog.get_nodes()} == { + "daily/2026-01-01.md", + "daily/2026-01-02.md", + } + assert await resource_catalog.get_nodes() == [] + finally: + await resource_catalog.close() + await digest_catalog.close() + print("✓ test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs passed") + + asyncio.run(run()) + + # --------------------------------------------------------------------------- # WatchChangesStep # --------------------------------------------------------------------------- +def test_watch_changes_default_low_power_timing(): + """Default watcher timing favors lower resource use.""" + step = WatchChangesStep() + + assert step.debounce == DEFAULT_WATCH_DEBOUNCE_MS + assert step.step == DEFAULT_WATCH_STEP_MS + assert step.poll_delay_ms == DEFAULT_LOW_POWER_POLL_MS + + custom = WatchChangesStep(debounce=1000, step=250, poll_delay_ms=3000) + assert custom.debounce == 1000 + assert custom.step == 250 + assert custom.poll_delay_ms == 3000 + + print("✓ test_watch_changes_default_low_power_timing passed") + + def test_watch_changes_requires_stop_event(): """Missing stop_event in context raises a clear error.""" @@ -368,97 +624,48 @@ def test_watch_changes_filter_matches_rules(): def test_watch_changes_dispatch_steps_list(): - """dispatch_steps config properly merges dispatch_step and dispatch_steps.""" - step1 = WatchChangesStep(dispatch_step="update_index_step") - assert step1.dispatch_steps == ["update_index_step"] - - step2 = WatchChangesStep(dispatch_steps=["update_catalog_step", "foreach_dispatch_step"]) - assert step2.dispatch_steps == ["update_catalog_step", "foreach_dispatch_step"] - - step3 = WatchChangesStep(dispatch_step="a", dispatch_steps=["b", "c"]) - assert step3.dispatch_steps == ["b", "c"] # dispatch_steps takes priority + """dispatch_steps config is stored by BaseStep.""" + step = WatchChangesStep(dispatch_steps=["update_catalog_step", "auto_resource_step"]) + assert step.dispatch_step_specs == ["update_catalog_step", "auto_resource_step"] print("✓ test_watch_changes_dispatch_steps_list passed") -# --------------------------------------------------------------------------- -# ForeachDispatchStep -# --------------------------------------------------------------------------- - - -def test_foreach_dispatch_no_job(): - """Without dispatch_job, step logs warning and returns success.""" +def test_auto_resource_batch_deleted_changes(): + """AutoResourceStep accepts a batch of change dicts from dispatch_steps.""" async def run(): with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): cwd = Path.cwd() app_ctx = _make_app_context(cwd) - step = ForeachDispatchStep(app_context=app_ctx) - ctx = RuntimeContext(changes=[{"change": "added", "path": "/x/y.md"}]) - resp = await step(ctx) - assert resp.success is True - assert resp.metadata.get("dispatched") is None # skipped early - print("✓ test_foreach_dispatch_no_job passed") + fs = LocalFileStore(name="test_store", embedding_store="") + await fs.start() + try: + filename = "file.md" + note_stem = _compute_note_stem(filename) + note_path = cwd / "daily" / "2026-01-01" / f"{note_stem}.md" + write_file(note_path, "---\nname: test\n---\nbody\n") + + step = AutoResourceStep(app_context=app_ctx, file_store=fs) + ctx = RuntimeContext( + changes=[ + {"change": "deleted", "path": str(cwd / "resource" / "2026-01-01" / filename)}, + ], + ) + resp = await step(ctx) + + assert resp.success is True + assert resp.answer == "Processed 1/1 resource change(s)" + assert resp.metadata["processed"] == 1 + assert resp.metadata["results"][0]["path"] == "resource/2026-01-01/file.md" + assert not note_path.exists() + finally: + await fs.close() + print("✓ test_auto_resource_batch_deleted_changes passed") asyncio.run(run()) -def test_foreach_dispatch_calls_job(): - """ForeachDispatchStep calls run_job for each change.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - cwd = Path.cwd() - app_ctx = _make_app_context(cwd) - step = ForeachDispatchStep(app_context=app_ctx) - changes = [ - {"change": "added", "path": str(cwd / "resource/2026-01-01/file.md")}, - {"change": "modified", "path": str(cwd / "resource/2026-01-01/data.json")}, - ] - ctx = RuntimeContext(changes=changes, dispatch_job="auto_resource") - - mock_job = AsyncMock() - app_ctx.jobs = {"auto_resource": mock_job} - resp = await step(ctx) - assert resp.success is True - assert resp.metadata["dispatched"] == 2 - assert mock_job.call_count == 2 - # Verify vault-relative paths were passed - calls = mock_job.call_args_list - assert calls[0].kwargs["file_path"] == "resource/2026-01-01/file.md" - assert calls[0].kwargs["change"] == "added" - assert calls[1].kwargs["file_path"] == "resource/2026-01-01/data.json" - assert calls[1].kwargs["change"] == "modified" - print("✓ test_foreach_dispatch_calls_job passed") - - asyncio.run(run()) - - -def test_foreach_dispatch_handles_error(): - """ForeachDispatchStep continues on job failure.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmpdir, temp_chdir(tmpdir): - cwd = Path.cwd() - app_ctx = _make_app_context(cwd) - step = ForeachDispatchStep(app_context=app_ctx) - changes = [ - {"change": "added", "path": str(cwd / "resource/a.md")}, - {"change": "added", "path": str(cwd / "resource/b.md")}, - ] - ctx = RuntimeContext(changes=changes, dispatch_job="failing_job") - - mock_job = AsyncMock(side_effect=RuntimeError("boom")) - app_ctx.jobs = {"failing_job": mock_job} - resp = await step(ctx) - assert resp.success is True # still succeeds - assert mock_job.call_count == 2 # tried both - print("✓ test_foreach_dispatch_handles_error passed") - - asyncio.run(run()) - - -# --------------------------------------------------------------------------- # LogChangesStep # --------------------------------------------------------------------------- @@ -490,21 +697,22 @@ if __name__ == "__main__": test_match_file_suffix() test_match_file_no_suffix_filter() test_collect_existing_filters() - # ScanStoreChangesStep + # InitChangesStep + test_clear_and_scan_defaults_include_jsonl() test_scan_changes_initial_all_added() test_scan_changes_no_changes() test_scan_changes_detect_modify_delete() test_scan_changes_missing_dir_skipped() test_scan_changes_resource_dir() + test_index_update_loop_init_dispatch_updates_store_across_batches() + test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs() # WatchChangesStep + test_watch_changes_default_low_power_timing() test_watch_changes_requires_stop_event() test_watch_changes_raises_no_valid_paths() test_watch_changes_filter_matches_rules() test_watch_changes_dispatch_steps_list() - # ForeachDispatchStep - test_foreach_dispatch_no_job() - test_foreach_dispatch_calls_job() - test_foreach_dispatch_handles_error() + test_auto_resource_batch_deleted_changes() # LogChangesStep test_log_changes_step() print("\n所有测试通过!") diff --git a/tests4/unit/test_base_component.py b/tests4/unit/test_base_component.py index ba64dea7..8a56baaf 100644 --- a/tests4/unit/test_base_component.py +++ b/tests4/unit/test_base_component.py @@ -38,6 +38,12 @@ class RequiredDepTarget(BaseComponent): component_type = ComponentEnum.FILE_GRAPH +class FailCloseComponent(StubComponent): + async def _close(self): + await super()._close() + raise RuntimeError("close failed") + + # -- Dependency --------------------------------------------------------------- @@ -161,6 +167,27 @@ def test_async_context_manager(): asyncio.run(run()) +def test_close_closes_owned_when_parent_close_fails(): + async def run(): + owned = StubComponent(name="owned") + parent = FailCloseComponent(name="parent") + parent.dep = BaseComponent.bind( + "sub", + StubComponent, + default_factory=lambda: owned, + ) + await parent.start() + + with pytest.raises(RuntimeError, match="close failed"): + await parent.close() + + assert owned.is_started is False + assert owned.close_count == 1 + assert parent.is_started is False + + asyncio.run(run()) + + # -- standalone resolution ---------------------------------------------------- @@ -330,6 +357,7 @@ if __name__ == "__main__": test_start_close_idempotent() test_restart() test_async_context_manager() + test_close_closes_owned_when_parent_close_fails() test_resolve_standalone_optional_becomes_none() test_resolve_standalone_with_default_factory() test_resolve_standalone_required_no_factory_keeps_placeholder() diff --git a/tests4/unit/test_channel_notify.py b/tests4/unit/test_channel_notify.py index 69a62354..3d5b01d0 100644 --- a/tests4/unit/test_channel_notify.py +++ b/tests4/unit/test_channel_notify.py @@ -54,7 +54,7 @@ def test_emits_one_event_per_batch_with_relative_paths(tmp_path): app_ctx, _ = _app_ctx_with_sink(vault, stub) step = ChannelNotifyStep(app_context=app_ctx) - _run( + response = _run( step( context=_ctx( [ @@ -65,6 +65,7 @@ def test_emits_one_event_per_batch_with_relative_paths(tmp_path): ), ) + assert response.success is True assert len(stub.sent) == 1 params = stub.sent[0].message.root.params assert params["meta"] == {"kind": "vault_change", "count": "2"} @@ -77,7 +78,8 @@ def test_noop_when_no_changes(tmp_path): stub = _StubSession() app_ctx, _ = _app_ctx_with_sink(tmp_path, stub) step = ChannelNotifyStep(app_context=app_ctx) - _run(step(context=_ctx([]))) + response = _run(step(context=_ctx([]))) + assert response.success is True assert not stub.sent @@ -86,7 +88,8 @@ def test_noop_when_sink_not_bound(tmp_path): app_ctx, _ = _app_ctx_with_sink(tmp_path, None) step = ChannelNotifyStep(app_context=app_ctx) # Should run without raising even though channel_sink is absent from metadata - _run(step(context=_ctx([{"change": "added", "path": "/tmp/x.md"}]))) + response = _run(step(context=_ctx([{"change": "added", "path": "/tmp/x.md"}]))) + assert response.success is True def test_path_outside_vault_passes_through_as_is(tmp_path): diff --git a/tests4/unit/test_claim_channel.py b/tests4/unit/test_claim_channel.py new file mode 100644 index 00000000..c31441e2 --- /dev/null +++ b/tests4/unit/test_claim_channel.py @@ -0,0 +1,100 @@ +"""Tests for ``ClaimChannelStep`` — current MCP session binding.""" + +import asyncio +import subprocess +import sys +from types import SimpleNamespace + +from fastmcp.server.context import _current_context + +from reme4.components.application_context import ApplicationContext +from reme4.components.service.mcp_service import ChannelSink +from reme4.steps.channel.claim_channel import ClaimChannelStep + + +class _StubSession: + """Capture outbound channel messages after being bound.""" + + def __init__(self) -> None: + self.sent: list = [] + + async def send_message(self, message) -> None: + """Record a message sent by the channel sink.""" + self.sent.append(message) + + +def _run(coro): + """Drive a coroutine on a fresh event loop.""" + return asyncio.new_event_loop().run_until_complete(coro) + + +def test_claim_channel_binds_current_session(tmp_path): + """The active FastMCP session becomes the sink recipient.""" + app_ctx = ApplicationContext(vault_dir=str(tmp_path), app_name="reme-test") + sink = ChannelSink() + app_ctx.metadata["channel_sink"] = sink + session = _StubSession() + ctx = SimpleNamespace(session=session, session_id="sid-1") + token = _current_context.set(ctx) + try: + step = ClaimChannelStep(app_context=app_ctx) + response = _run(step()) + finally: + _current_context.reset(token) + + assert response.success is True + assert response.answer["claimed"] is True + assert response.answer["session_id"] == "sid-1" + assert response.metadata["claimed"] is True + + _run(sink.emit("hello", {"kind": "test"})) + assert len(session.sent) == 1 + + +def test_claim_channel_reports_missing_fastmcp_context(tmp_path): + """Calling outside a FastMCP request reports a clean unclaimed result.""" + app_ctx = ApplicationContext(vault_dir=str(tmp_path), app_name="reme-test") + app_ctx.metadata["channel_sink"] = ChannelSink() + step = ClaimChannelStep(app_context=app_ctx) + + response = _run(step()) + + assert response.success is True + assert response.answer["claimed"] is False + assert response.metadata["claimed"] is False + assert "No active context" in response.answer["reason"] + + +def test_claim_channel_missing_sink_is_still_controlled_under_optimized_python(tmp_path): + """Runtime validation must not rely on assert, which Python -O removes.""" + code = f""" +import asyncio +from types import SimpleNamespace +from fastmcp.server.context import _current_context +from reme4.components.application_context import ApplicationContext +from reme4.steps.channel.claim_channel import ClaimChannelStep + +class Session: + async def send_message(self, message): + pass + +async def main(): + app_ctx = ApplicationContext(vault_dir={str(tmp_path)!r}, app_name="reme-test") + ctx = SimpleNamespace(session=Session(), session_id="sid-optimized") + token = _current_context.set(ctx) + try: + response = await ClaimChannelStep(app_context=app_ctx)() + print(response.answer) + finally: + _current_context.reset(token) + +asyncio.run(main()) +""" + result = subprocess.run( + [sys.executable, "-O", "-c", code], + check=True, + capture_output=True, + text=True, + ) + assert "'claimed': False" in result.stdout + assert "channel_sink not configured" in result.stdout diff --git a/tests4/unit/test_common_steps.py b/tests4/unit/test_common_steps.py index b6bfe52c..ff39b303 100644 --- a/tests4/unit/test_common_steps.py +++ b/tests4/unit/test_common_steps.py @@ -1,16 +1,4 @@ -"""Tests for reme4 common steps. - -Two surfaces share this file: - -* **In-process job tests** (top half) build an ``Application`` from the - default config and call ``run_job`` directly — no subprocess, no HTTP. - Each test uses an isolated cwd so the vault (``.reme`` by default) - does not collide. -* **Direct unit tests** (bottom half) exercise ``TraverseStep`` - (registered as ``traverse_step``) — BFS over wikilink edges from a - seed file, forward / backward / both — against a freshly built - ``LocalFileStore`` (embedding disabled). -""" +"""Tests for reme4 common steps with only local dependencies.""" # pylint: disable=protected-access @@ -19,12 +7,13 @@ import os import tempfile import warnings -from reme4 import Application, __version__ as REME_VERSION +from reme4.components.agent_wrapper import BaseAgentWrapper from reme4.components.file_store import LocalFileStore -from reme4.config import resolve_app_config from reme4.schema import FileLink, FileNode +from reme4.steps.common.add import AddStep +from reme4.steps.common.health_check import _file_graph_status +from reme4.steps.common.llm_demo import LLMDemoStep from reme4.steps.index import traverse as traverse_mod -from reme4.utils import load_env warnings.filterwarnings("ignore", category=DeprecationWarning, module="jieba") warnings.filterwarnings("ignore", category=DeprecationWarning, module="pkg_resources") @@ -51,15 +40,6 @@ def _run(coro): asyncio.run(coro) -async def _make_app() -> Application: - """Build and start an Application with the default config, logging silenced.""" - load_env() - cfg = resolve_app_config(log_to_console=False, log_to_file=False, enable_logo=False) - app = Application(**cfg) - await app.start() - return app - - def _node(path: str, links: list[tuple[str, str | None, str | None]] | None = None) -> FileNode: """Build a FileNode with (target_path, target_anchor, predicate) outgoing edges.""" return FileNode( @@ -82,112 +62,79 @@ def _edges(step) -> list[dict]: return step.context.response.metadata.get("edges", []) -# =========================================================================== -# In-process job tests: version / help / health_check / search / reindex -# =========================================================================== - - -def test_version_job(): - """version job should return the package version string.""" +def test_add_step_coerces_numeric_inputs(): + """add accepts numeric strings as numbers, not string concatenation.""" async def run(): - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): - app = await _make_app() - try: - resp = await app.run_job("version") - assert resp.success is True - assert resp.answer == REME_VERSION - assert resp.metadata.get("version") == REME_VERSION - finally: - await app.close() - print("✓ test_version_job passed") + step = AddStep() + resp = await step(a="1", b="2.5") + assert resp.success is True + assert resp.answer == "3.5" + assert resp.metadata["result"] == 3.5 + print("✓ test_add_step_coerces_numeric_inputs passed") _run(run()) -def test_help_job(): - """help job should list jobs except itself.""" +def test_add_step_rejects_invalid_inputs(): + """invalid add arguments should return a failed response instead of throwing or concatenating.""" async def run(): - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): - app = await _make_app() - try: - resp = await app.run_job("help") - assert resp.success is True - assert isinstance(resp.answer, str) - assert resp.metadata.get("job_count", 0) > 0 - assert "`help`" not in resp.answer - for expected_job in ("version", "health_check", "search"): - assert expected_job in resp.answer, f"help missing {expected_job!r}: {resp.answer!r}" - finally: - await app.close() - print("✓ test_help_job passed") + step = AddStep() + resp = await step(a="one", b=2) + assert resp.success is False + assert "Invalid add arguments" in resp.answer + print("✓ test_add_step_rejects_invalid_inputs passed") _run(run()) -def test_search_job_empty_store(): - """search on an empty store should return successfully with zero results.""" +class _FakeAgentWrapper(BaseAgentWrapper): + """Capture reply kwargs without calling a real model.""" + + def __init__(self): + super().__init__() + self.last_kwargs = None + + async def reply(self, inputs, **kwargs) -> dict: + self.last_kwargs = kwargs + return {"result": "ok"} + + +def test_llm_demo_always_registers_add_tool(): + """LLM demo always passes the add job as a tool.""" async def run(): - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): - app = await _make_app() - try: - resp = await app.run_job("search", query="hello world", limit=5) - assert resp.success is True - counts = resp.metadata.get("counts", {}) - assert isinstance(counts, dict) - assert counts.get("returned", -1) == 0 - finally: - await app.close() - print("✓ test_search_job_empty_store passed") + wrapper = _FakeAgentWrapper() + step = LLMDemoStep() + resp = await step(query="hello", agent_wrapper=wrapper) + assert resp.success is True + assert wrapper.last_kwargs["job_tools"] == ["add"] + assert "job_tools" not in resp.metadata + print("✓ test_llm_demo_always_registers_add_tool passed") _run(run()) -def test_search_job_missing_query(): - """search with empty query returns success=False and a query-related error in answer.""" +def test_file_graph_health_reports_neo4j_cached_counts(): + """Neo4j file graph health should not be reported as an empty local graph.""" - async def run(): - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): - app = await _make_app() - try: - resp = await app.run_job("search", query="") - assert resp.success is False - assert "query" in str(resp.answer).lower() - finally: - await app.close() - print("✓ test_search_job_missing_query passed") + class FakeNeo4jGraph: + """Minimal Neo4j graph stub with cached health counters.""" - _run(run()) + is_started = True + _driver = object() + _uri = "bolt://example" + _database = "neo4j" + _n_nodes = 3 + _n_edges = 4 + _n_virtual = 1 - -def test_all_jobs_single_app(): - """Run every common job against one shared in-process Application for efficiency.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): - app = await _make_app() - try: - resp = await app.run_job("version") - assert resp.answer == REME_VERSION - - resp = await app.run_job("help") - assert resp.metadata.get("job_count", 0) > 0 - - resp = await app.run_job("health_check") - assert isinstance(resp.metadata.get("health"), dict) - - resp = await app.run_job("search", query="anything") - assert resp.success is True - - resp = await app.run_job("reindex") - assert isinstance(resp.metadata.get("counts"), dict) - finally: - await app.close() - print("✓ test_all_jobs_single_app passed") - - _run(run()) + status = _file_graph_status(FakeNeo4jGraph()) + assert status["n_nodes"] == 3 + assert status["n_edges"] == 4 + assert status["n_virtual"] == 1 + print("✓ test_file_graph_health_reports_neo4j_cached_counts passed") # =========================================================================== @@ -333,12 +280,6 @@ def test_traverse_both_directions(): if __name__ == "__main__": - print("\n=== reme4 common steps in-process tests ===") - test_version_job() - test_help_job() - test_search_job_empty_store() - test_search_job_missing_query() - test_all_jobs_single_app() print("\n=== traverse step tests ===") test_traverse_forward_depth_1() test_traverse_backward_returns_inlinks() diff --git a/tests4/unit/test_config_parser.py b/tests4/unit/test_config_parser.py new file mode 100644 index 00000000..cb6aadd8 --- /dev/null +++ b/tests4/unit/test_config_parser.py @@ -0,0 +1,66 @@ +"""Tests for configuration parsing helpers.""" + +from pathlib import Path + +import pytest + +from reme4.config.config_parser import ( + _expand_env_vars, + _load_config, + _read_config_file, + parse_args, + parse_dot_notation, +) + + +def test_load_builtin_config_by_filename_with_suffix(): + """Built-in config names may include the YAML suffix.""" + cfg = _load_config("default.yaml") + + assert cfg["service"]["backend"] == "http" + + +def test_parse_args_rejects_non_key_value_extra_argument(): + """Extra CLI arguments must use key=value syntax.""" + with pytest.raises(ValueError, match="expected key=value"): + parse_args("search", "hello") + + +@pytest.mark.parametrize("item", ["=1", ".a=1", "a.=1", "a..b=1"]) +def test_parse_dot_notation_rejects_empty_key_segments(item): + """Dot notation keys cannot contain empty path segments.""" + with pytest.raises(ValueError, match="Invalid dot notation key"): + parse_dot_notation([item]) + + +def test_read_config_file_rejects_non_mapping_root(tmp_path: Path): + """Config files must contain a mapping at the root.""" + config_path = tmp_path / "bad.yaml" + config_path.write_text("- item\n", encoding="utf-8") + + with pytest.raises(ValueError, match="Config root must be a mapping"): + _read_config_file(config_path) + + +def test_expand_env_vars_converts_expanded_scalar_types(monkeypatch): + """Expanded environment values keep YAML scalar typing.""" + monkeypatch.setenv("PORT", "18080") + monkeypatch.setenv("ENABLED", "false") + + expanded = _expand_env_vars( + { + "port": "${PORT}", + "enabled": "${ENABLED}", + "zip": "${ZIP:-007}", + "url": "http://${HOST:-localhost}:${PORT}", + "string_bool": '${STRING_BOOL:-"false"}', + }, + ) + + assert expanded == { + "port": 18080, + "enabled": False, + "zip": "007", + "url": "http://localhost:18080", + "string_bool": "false", + } diff --git a/tests4/unit/test_cron_job.py b/tests4/unit/test_cron_job.py index d100fb83..b12d06eb 100644 --- a/tests4/unit/test_cron_job.py +++ b/tests4/unit/test_cron_job.py @@ -1,313 +1,75 @@ -"""Unit tests for the ``cron`` job — schedule math + dispatch wiring. - -Strategy: - -* ``_parse_hh_mm`` / ``_next_fire_delay`` are exercised directly without - the BackgroundJob supervisor. Avoids wall-clock waits. -* ``__call__`` is tested by driving the job's own ``_stop_event``: the - loop runs at most one iteration with a tiny ``interval_seconds`` and - ``run_on_start=True`` — verifies the fire path actually dispatches the - downstream step exactly once. -* The dispatch target is a tiny in-test counter step registered into - the same step registry the production code uses, so we exercise the - real ``R.get(ComponentEnum.STEP, name) → instantiate → __call__`` - path rather than mocking it. -* Tests run under a tempdir so the implicit Application context is - isolated from the real vault. -""" +"""Unit tests for the ``cron`` job.""" # pylint: disable=protected-access import asyncio -import datetime -import os -import tempfile -import zoneinfo -from contextlib import contextmanager -from pathlib import Path +from types import SimpleNamespace -from reme4 import Application from reme4.components import R +from reme4.components.job.base_job import BaseJob from reme4.components.job.cron_job import CronJob -from reme4.config import resolve_app_config from reme4.steps.base_step import BaseStep @R.register("test_cron_counter_step") class _CounterStep(BaseStep): - """In-test counter — increments a class-level fire count on each invocation.""" - fires: int = 0 async def execute(self): type(self).fires += 1 - if self.context is not None: - self.context.response.success = True + self.context.response.success = True return self.context.response -@contextmanager -def _temp_chdir(path: Path): - old = os.getcwd() - os.chdir(path) +def _test_cron_parameter_protocol() -> None: + assert CronJob("* * * * *").cron_expr == "* * * * *" + assert CronJob(cron="0 3 * * *").cron_expr == "0 3 * * *" + print("OK cron_parameter_protocol") + + +async def _invalid_cron_raises() -> None: try: - yield - finally: - os.chdir(old) - - -def _make_job(**kwargs) -> CronJob: - # Default to a known-registered step so most tests can ignore dispatch wiring. - kwargs.setdefault("dispatch_step", "version_step") - return CronJob(**kwargs) - - -def _test_parse_hh_mm_valid() -> None: - job = _make_job(daily_at="03:00") - assert job._fire_hour == 3 and job._fire_minute == 0 - job = _make_job(daily_at="23:59") - assert job._fire_hour == 23 and job._fire_minute == 59 - print("OK parse_hh_mm_valid") - - -def _test_parse_hh_mm_invalid() -> None: - for bad in ["24:00", "03:60", "abc", "3", "03:", ":00"]: - try: - _make_job(daily_at=bad) - except ValueError: - continue - raise AssertionError(f"expected ValueError for daily_at={bad!r}") - print("OK parse_hh_mm_invalid") - - -def _test_requires_dispatch_step() -> None: - # neither dispatch_step nor dispatch_steps → ValueError - try: - CronJob(daily_at="03:00") + await CronJob(cron="not a cron")._start() except ValueError: - print("OK requires_dispatch_step") return - raise AssertionError("expected ValueError when no dispatch step is configured") + raise AssertionError("expected ValueError for invalid cron expression") -def _test_dispatch_steps_list() -> None: - # Mirrors watch_changes_step: dispatch_steps takes priority over dispatch_step, - # both forms accepted, defaults coalesce. - job1 = CronJob(dispatch_step="version_step", interval_seconds=60) - assert job1.dispatch_steps == ["version_step"] - - job2 = CronJob(dispatch_steps=["a", "b"], interval_seconds=60) - assert job2.dispatch_steps == ["a", "b"] - - job3 = CronJob(dispatch_step="x", dispatch_steps=["y", "z"], interval_seconds=60) - assert job3.dispatch_steps == ["y", "z"] - print("OK dispatch_steps_list") +def _test_invalid_cron_raises_on_start() -> None: + asyncio.run(_invalid_cron_raises()) + print("OK invalid_cron_raises_on_start") -def _test_dispatch_jobs_list() -> None: - # dispatch_job / dispatch_jobs coalesce the same way and satisfy the - # "at least one dispatch target" requirement on their own. - job1 = CronJob(dispatch_job="auto_dream", interval_seconds=60) - assert job1.dispatch_jobs == ["auto_dream"] and job1.dispatch_steps == [] - - job2 = CronJob(dispatch_jobs=["a", "b"], interval_seconds=60) - assert job2.dispatch_jobs == ["a", "b"] - print("OK dispatch_jobs_list") - - -def _test_requires_exactly_one_schedule() -> None: - # none of the three set - try: - _make_job() - except ValueError: - pass - else: - raise AssertionError("expected ValueError when no schedule is set") - # any two together - pairs = [ - {"daily_at": "03:00", "interval_seconds": 60}, - {"daily_at": "03:00", "cron": "0 3 * * *"}, - {"interval_seconds": 60, "cron": "0 3 * * *"}, - ] - for kw in pairs: - try: - _make_job(**kw) - except ValueError: - continue - raise AssertionError(f"expected ValueError when two schedules set: {kw}") - # all three - try: - _make_job(daily_at="03:00", interval_seconds=60, cron="0 3 * * *") - except ValueError: - pass - else: - raise AssertionError("expected ValueError when all three schedules set") - print("OK requires_exactly_one_schedule") - - -def _test_cron_expression_valid() -> None: - for expr in ["0 3 * * *", "*/15 * * * *", "0 */6 * * *", "0 3 * * 1-5", "30 2 1 * *"]: - job = _make_job(cron=expr) - assert job.cron == expr - print("OK cron_expression_valid") - - -def _test_cron_expression_invalid() -> None: - # Eager validation — bad expressions must raise at construction time, - # so a typo fails at app start rather than at 3am. - for bad in ["not a cron", "0 25 * * *", "60 * * * *", "* * * 13 *", ""]: - try: - _make_job(cron=bad) - except ValueError: - continue - raise AssertionError(f"expected ValueError for cron={bad!r}") - print("OK cron_expression_invalid") - - -class _FrozenJob(CronJob): - """CronJob subclass with deterministic 'now' for daily_at / cron delay math.""" - - def __init__(self, frozen_now: datetime.datetime, **kwargs): - kwargs.setdefault("dispatch_step", "version_step") - super().__init__(**kwargs) - self._frozen_now = frozen_now - - def _next_fire_delay(self) -> float: - if self.interval_seconds: - return float(self.interval_seconds) - if self.cron: - from croniter import croniter - - nxt = croniter(self.cron, self._frozen_now).get_next(datetime.datetime) - return (nxt - self._frozen_now).total_seconds() - target = self._frozen_now.replace( - hour=self._fire_hour, - minute=self._fire_minute, - second=0, - microsecond=0, - ) - if target <= self._frozen_now: - target = target + datetime.timedelta(days=1) - return (target - self._frozen_now).total_seconds() - - -def _test_next_fire_delay_before_target() -> None: - tz = zoneinfo.ZoneInfo("Asia/Shanghai") - job = _FrozenJob( - frozen_now=datetime.datetime(2026, 6, 7, 2, 0, 0, tzinfo=tz), - daily_at="03:00", - ) - assert job._next_fire_delay() == 3600 - print("OK next_fire_delay_before_target") - - -def _test_next_fire_delay_after_target() -> None: - tz = zoneinfo.ZoneInfo("Asia/Shanghai") - job = _FrozenJob( - frozen_now=datetime.datetime(2026, 6, 7, 4, 0, 0, tzinfo=tz), - daily_at="03:00", - ) - assert job._next_fire_delay() == 23 * 3600 - print("OK next_fire_delay_after_target") - - -def _test_next_fire_delay_interval() -> None: - job = _make_job(interval_seconds=30) - assert job._next_fire_delay() == 30.0 - print("OK next_fire_delay_interval") - - -def _test_next_fire_delay_cron_daily() -> None: - # cron "0 3 * * *" is exactly equivalent to daily_at "03:00" — same math, - # but exercised via the croniter path. - tz = zoneinfo.ZoneInfo("Asia/Shanghai") - job = _FrozenJob( - frozen_now=datetime.datetime(2026, 6, 7, 2, 0, 0, tzinfo=tz), - cron="0 3 * * *", - ) - assert job._next_fire_delay() == 3600 - print("OK next_fire_delay_cron_daily") - - -def _test_next_fire_delay_cron_every_6h() -> None: - tz = zoneinfo.ZoneInfo("Asia/Shanghai") - # "0 */6 * * *" fires at 00:00 / 06:00 / 12:00 / 18:00. At 02:00, - # next fire is 06:00 → 4 hours out. - job = _FrozenJob( - frozen_now=datetime.datetime(2026, 6, 7, 2, 0, 0, tzinfo=tz), - cron="0 */6 * * *", - ) - assert job._next_fire_delay() == 4 * 3600 - print("OK next_fire_delay_cron_every_6h") - - -def _test_next_fire_delay_cron_weekday_only() -> None: - tz = zoneinfo.ZoneInfo("Asia/Shanghai") - # 2026-06-07 is a Sunday. "0 3 * * 1-5" → next fire is Mon 2026-06-08 03:00. - # From Sun 02:00, that's 25 hours. - job = _FrozenJob( - frozen_now=datetime.datetime(2026, 6, 7, 2, 0, 0, tzinfo=tz), - cron="0 3 * * 1-5", - ) - assert job._next_fire_delay() == 25 * 3600 - print("OK next_fire_delay_cron_weekday_only") - - -async def _drive_one_fire(_tmp: Path) -> int: - """Stand up an Application + dispatch the counter step on a cron tick.""" - cfg = resolve_app_config(log_to_console=False, log_to_file=False, enable_logo=False) - cfg["enable_logo"] = False - app = Application(**cfg) - await app.start() - +async def _drive_steps_once() -> int: _CounterStep.fires = 0 - try: - job = CronJob( - dispatch_step="test_cron_counter_step", - interval_seconds=1, - run_on_start=True, - ) - job.app_context = app.context - # The BackgroundJob supervisor normally creates this in _start(); here we - # drive __call__ directly, so wire up the stop_event by hand. - job._stop_event = asyncio.Event() + job = CronJob( + cron="* * * * *", + steps=[{"backend": "test_cron_counter_step"}], + ) + job.app_context = SimpleNamespace(app_config=SimpleNamespace(language="")) + await BaseJob._start(job) + job._stop_event = asyncio.Event() + job._next_fire_delay = lambda: 0.01 - # Fire once on start, then signal stop so the loop exits before the - # next interval elapses. - task = asyncio.create_task(job()) - await asyncio.sleep(0.3) # let run_on_start fire propagate - job._stop_event.set() - await asyncio.wait_for(task, timeout=5.0) - return _CounterStep.fires - finally: - await app.close() + task = asyncio.create_task(job()) + await asyncio.sleep(0.05) + job._stop_event.set() + await asyncio.wait_for(task, timeout=1) + return _CounterStep.fires -def _test_run_on_start_dispatches_once() -> None: - with tempfile.TemporaryDirectory() as tmp, _temp_chdir(Path(tmp)): - count = asyncio.run(_drive_one_fire(Path(tmp))) - assert count >= 1, f"expected at least one dispatch, got {count}" - print(f"OK run_on_start_dispatches_once (count={count})") +def _test_executes_own_steps() -> None: + count = asyncio.run(_drive_steps_once()) + assert count >= 1, f"expected cron to execute own steps, got {count}" + print(f"OK executes_own_steps count={count}") def main() -> None: - """Entry point for the cron job unit tests.""" + """Run all tests.""" print("=== cron job unit tests ===") - _test_parse_hh_mm_valid() - _test_parse_hh_mm_invalid() - _test_requires_dispatch_step() - _test_dispatch_steps_list() - _test_dispatch_jobs_list() - _test_requires_exactly_one_schedule() - _test_cron_expression_valid() - _test_cron_expression_invalid() - _test_next_fire_delay_before_target() - _test_next_fire_delay_after_target() - _test_next_fire_delay_interval() - _test_next_fire_delay_cron_daily() - _test_next_fire_delay_cron_every_6h() - _test_next_fire_delay_cron_weekday_only() - _test_run_on_start_dispatches_once() + _test_cron_parameter_protocol() + _test_invalid_cron_raises_on_start() + _test_executes_own_steps() print("=== passed ===") diff --git a/tests4/unit/test_daily_steps.py b/tests4/unit/test_daily_steps.py index c950b077..385e7cd6 100644 --- a/tests4/unit/test_daily_steps.py +++ b/tests4/unit/test_daily_steps.py @@ -79,7 +79,7 @@ async def _make_store_with_dailies(entries: list[tuple[str, str, str]]) -> Local day_dir = Path.cwd() / "daily" / day day_dir.mkdir(parents=True, exist_ok=True) text = f"---\nname: {session_id}\n---\n{body}\n" - (day_dir / f"session_agent_{session_id}.md").write_text(text, encoding="utf-8") + (day_dir / f"{session_id}.md").write_text(text, encoding="utf-8") return store @@ -95,7 +95,7 @@ async def _seed_note(date: str, session_id: str, name: str = "", description: st if description: fm_lines.append(f"description: {description}") text = "---\n" + "\n".join(fm_lines) + "\n---\nbody\n" - (day_dir / f"session_agent_{session_id}.md").write_text(text, encoding="utf-8") + (day_dir / f"{session_id}.md").write_text(text, encoding="utf-8") # -- daily_list_step ---------------------------------------------------------- @@ -119,8 +119,8 @@ def test_daily_list_default_date_is_today(): assert payload["date"] == _today() assert payload["count"] == 2 answer = step.context.response.answer - assert f"daily/{_today()}/session_agent_today-a.md" in answer - assert f"daily/{_today()}/session_agent_today-b.md" in answer + assert f"daily/{_today()}/today-a.md" in answer + assert f"daily/{_today()}/today-b.md" in answer await store.close() print("✓ test_daily_list_default_date_is_today passed") @@ -144,7 +144,7 @@ def test_daily_list_filters_by_date(): assert payload["date"] == "2026-05-18" assert payload["count"] == 1 answer = step.context.response.answer - assert "daily/2026-05-18/session_agent_a.md" in answer + assert "daily/2026-05-18/a.md" in answer await store.close() print("✓ test_daily_list_filters_by_date passed") @@ -169,7 +169,7 @@ def test_daily_list_returns_path_session_id_metadata(): payload = _metadata(step) assert payload["count"] == 1 answer = step.context.response.answer - assert "daily/2026-05-18/session_agent_alpha.md" in answer + assert "daily/2026-05-18/alpha.md" in answer assert "Alpha Project" in answer assert "JWT auth migration" in answer await store.close() @@ -200,7 +200,7 @@ def test_daily_list_ignores_subdirectories(): payload = _metadata(step) assert payload["count"] == 1 answer = step.context.response.answer - assert "daily/2026-05-18/session_agent_main.md" in answer + assert "daily/2026-05-18/main.md" in answer await store.close() print("✓ test_daily_list_ignores_subdirectories passed") @@ -285,9 +285,9 @@ def test_daily_create_provisions_note_and_refreshes_index(): assert payload["created"] is True assert payload["date"] == "2026-05-18" assert payload["session_id"] == "kickoff" - assert payload["path"] == "daily/2026-05-18/session_agent_kickoff.md" + assert payload["path"] == "daily/2026-05-18/kickoff.md" - note = Path(tmp) / "daily" / "2026-05-18" / "session_agent_kickoff.md" + note = Path(tmp) / "daily" / "2026-05-18" / "kickoff.md" text = note.read_text(encoding="utf-8") assert "name: kickoff" in text # Body is empty — file is frontmatter + trailing newline. @@ -295,7 +295,7 @@ def test_daily_create_provisions_note_and_refreshes_index(): index = Path(tmp) / "daily" / "2026-05-18.md" assert index.is_file() - assert "[[daily/2026-05-18/session_agent_kickoff.md]]" in index.read_text(encoding="utf-8") + assert "[[daily/2026-05-18/kickoff.md]]" in index.read_text(encoding="utf-8") await store.close() print("✓ test_daily_create_provisions_note_and_refreshes_index passed") @@ -310,7 +310,7 @@ def test_daily_create_is_idempotent_on_existing(): store = await _make_store_with_dailies( [("2026-05-18", "ongoing", "old body")], ) - file_path = Path(tmp) / "daily" / "2026-05-18" / "session_agent_ongoing.md" + file_path = Path(tmp) / "daily" / "2026-05-18" / "ongoing.md" before = file_path.read_text(encoding="utf-8") step = daily_create_step.DailyCreateStep(file_store=store) @@ -319,7 +319,7 @@ def test_daily_create_is_idempotent_on_existing(): assert step.context.response.success is True assert payload["created"] is False - assert payload["path"] == "daily/2026-05-18/session_agent_ongoing.md" + assert payload["path"] == "daily/2026-05-18/ongoing.md" assert file_path.read_text(encoding="utf-8") == before assert payload["index"]["path"] == "daily/2026-05-18.md" await store.close() @@ -338,7 +338,7 @@ def test_daily_create_default_date_is_today(): await step(session_id="today-task") payload = _metadata(step) assert payload["date"] == _today() - assert payload["path"] == f"daily/{_today()}/session_agent_today-task.md" + assert payload["path"] == f"daily/{_today()}/today-task.md" assert payload["created"] is True await store.close() print("✓ test_daily_create_default_date_is_today passed") @@ -355,7 +355,7 @@ def test_daily_create_default_frontmatter_uses_session_id_as_name(): step = daily_create_step.DailyCreateStep(file_store=store) await step(session_id="auth-refactor", date="2026-05-18") - note = Path(tmp) / "daily" / "2026-05-18" / "session_agent_auth-refactor.md" + note = Path(tmp) / "daily" / "2026-05-18" / "auth-refactor.md" text = note.read_text(encoding="utf-8") assert "name: auth-refactor" in text assert "description:" in text @@ -414,7 +414,7 @@ def test_daily_create_then_skip_round_trip(): first = _metadata(step) assert first["created"] is True - note = Path(tmp) / "daily" / "2026-05-18" / "session_agent_probe.md" + note = Path(tmp) / "daily" / "2026-05-18" / "probe.md" before = note.read_text(encoding="utf-8") await step(session_id="probe", date="2026-05-18") @@ -446,8 +446,8 @@ def test_day_index_lists_each_note(): await daily_reindex_step.DailyReindexStep(file_store=store)(date="2026-05-18") text = _day_index_text(tmp, "2026-05-18") - assert "[[daily/2026-05-18/session_agent_alpha.md]]" in text - assert "[[daily/2026-05-18/session_agent_beta.md]]" in text + assert "[[daily/2026-05-18/alpha.md]]" in text + assert "[[daily/2026-05-18/beta.md]]" in text assert "Alpha Project" in text assert "Beta Project" in text await store.close() @@ -474,19 +474,13 @@ def test_day_index_includes_note_descriptions(): await daily_reindex_step.DailyReindexStep(file_store=store)(date="2026-05-18") text = _day_index_text(tmp, "2026-05-18") # name + description inline on the same line as the wikilink - assert ( - "[[daily/2026-05-18/session_agent_alpha.md]] name: Alpha Project description: 实现 JWT auth 中间件" - in text - ) - assert ( - "[[daily/2026-05-18/session_agent_beta.md]] name: beta description: 调研增值税新政对 SaaS 的影响" - in text - ) + assert "[[daily/2026-05-18/alpha.md]] name: Alpha Project description: 实现 JWT auth 中间件" in text + assert "[[daily/2026-05-18/beta.md]] name: beta description: 调研增值税新政对 SaaS 的影响" in text # gamma has no description → only name is emitted, no trailing `description:` cruft - assert "[[daily/2026-05-18/session_agent_gamma.md]] name: Gamma\n" in text or text.rstrip().endswith( - "[[daily/2026-05-18/session_agent_gamma.md]] name: Gamma", + assert "[[daily/2026-05-18/gamma.md]] name: Gamma\n" in text or text.rstrip().endswith( + "[[daily/2026-05-18/gamma.md]] name: Gamma", ) - assert "description:" not in text.split("[[daily/2026-05-18/session_agent_gamma.md]]")[1].split("\n")[0] + assert "description:" not in text.split("[[daily/2026-05-18/gamma.md]]")[1].split("\n")[0] await store.close() print("✓ test_day_index_includes_note_descriptions passed") @@ -516,6 +510,29 @@ def test_day_index_description_is_note_count(): asyncio.run(run()) +def test_day_index_description_updates_when_note_count_changes(): + """Reindexing an existing day index refreshes the note-count description.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + store = LocalFileStore(name="t", embedding_store="") + await store.start() + await _seed_note("2026-05-18", "alpha") + reindex = daily_reindex_step.DailyReindexStep(file_store=store) + await reindex(date="2026-05-18") + + await _seed_note("2026-05-18", "beta") + await reindex(date="2026-05-18") + + text = _day_index_text(tmp, "2026-05-18") + assert "2 note(s) today." in text + assert "1 note(s) today." not in text + await store.close() + print("✓ test_day_index_description_updates_when_note_count_changes passed") + + asyncio.run(run()) + + def test_day_index_preserves_user_content_outside_marker(): """Any user-authored content sitting outside the auto markers is preserved verbatim across refreshes.""" @@ -540,7 +557,7 @@ def test_day_index_preserves_user_content_outside_marker(): assert "MY HAND-WRITTEN NOTE" in after assert "这是我手写的备忘" in after assert "## 我的笔记" in after - assert "[[daily/2026-05-18/session_agent_beta.md]]" in after + assert "[[daily/2026-05-18/beta.md]]" in after await store.close() print("✓ test_day_index_preserves_user_content_outside_marker passed") @@ -574,8 +591,8 @@ def test_daily_reindex_returns_write_view(): assert payload["notes_count"] == 2 text = _day_index_text(tmp, "2026-05-18") - assert "[[daily/2026-05-18/session_agent_alpha.md]]" in text - assert "[[daily/2026-05-18/session_agent_beta.md]]" in text + assert "[[daily/2026-05-18/alpha.md]]" in text + assert "[[daily/2026-05-18/beta.md]]" in text await store.close() print("✓ test_daily_reindex_returns_write_view passed") @@ -625,6 +642,7 @@ if __name__ == "__main__": test_day_index_lists_each_note() test_day_index_includes_note_descriptions() test_day_index_description_is_note_count() + test_day_index_description_updates_when_note_count_changes() test_day_index_preserves_user_content_outside_marker() test_daily_reindex_returns_write_view() test_daily_reindex_created_flag_flips_on_rerun() diff --git a/tests4/unit/test_file_store_consistency.py b/tests4/unit/test_file_store_consistency.py new file mode 100644 index 00000000..a8a16315 --- /dev/null +++ b/tests4/unit/test_file_store_consistency.py @@ -0,0 +1,166 @@ +"""Regression tests for LocalFileStore / FaissLocalFileStore consistency.""" + +# pylint: disable=protected-access + +import asyncio +import os +import tempfile + +import numpy as np +import pytest + +from reme4.components.file_store import FaissLocalFileStore, LocalFileStore +from reme4.schema import FileChunk, FileNode + + +class temp_chdir: + """Temporarily chdir into a test vault.""" + + def __init__(self, path): + self.path = path + self.old = None + + def __enter__(self): + self.old = os.getcwd() + os.chdir(self.path) + return self + + def __exit__(self, *exc): + os.chdir(self.old) + + +class FakeEmbeddingStore: + """Small deterministic embedding provider used by file-store tests.""" + + dimensions = 2 + + def _embed(self, text: str) -> np.ndarray: + if "beta" in text or "fresh" in text: + return np.array([0.0, 1.0], dtype=np.float16) + return np.array([1.0, 0.0], dtype=np.float16) + + async def health_check(self, _timeout: float = 2.0) -> bool: + """Report the fake embedding service as healthy.""" + return True + + async def get_embedding(self, input_text: str, **_kwargs) -> np.ndarray: + """Return a deterministic embedding for a single text.""" + return self._embed(input_text) + + async def get_node_embeddings(self, nodes: list[FileChunk], **_kwargs) -> list[FileChunk]: + """Attach deterministic embeddings to file chunks.""" + for chunk_node in nodes: + chunk_node.embedding = self._embed(chunk_node.text) + return nodes + + +def run(coro): + """Run an async test body.""" + return asyncio.run(coro) + + +def node(path: str) -> FileNode: + """Build a minimal file node.""" + return FileNode(path=path, st_mtime=1.0) + + +def chunk(chunk_id: str, path: str, text: str, **metadata) -> FileChunk: + """Build a minimal file chunk.""" + return FileChunk(id=chunk_id, path=path, text=text, start_line=1, end_line=1, metadata=metadata) + + +def test_keyword_only_upsert_removes_old_chunks_and_docs(): + """Keyword-only upsert removes stale chunks and keyword documents.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + store = LocalFileStore(name="t_keyword_only", embedding_store="") + await store.start() + + await store.upsert([(node("note.md"), [chunk("old", "note.md", "obsoleteword only")])]) + assert [c.id for c in await store.keyword_search("obsoleteword", 5, {})] == ["old"] + + await store.upsert([(node("note.md"), [chunk("new", "note.md", "freshword only")])]) + + assert "old" not in store.file_chunks + assert await store.keyword_search("obsoleteword", 5, {}) == [] + assert [c.id for c in await store.keyword_search("freshword", 5, {})] == ["new"] + await store.close() + + run(go()) + + +def test_same_chunk_id_with_changed_text_gets_new_embedding(): + """Changing a chunk text refreshes its embedding.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + store = LocalFileStore(name="t_embedding_reuse", embedding_store="") + await store.start() + store.embedding_store = FakeEmbeddingStore() + + await store.upsert([(node("note.md"), [chunk("same", "note.md", "alpha text")])]) + assert store.file_chunks["same"].embedding.tolist() == [1.0, 0.0] + + await store.upsert([(node("note.md"), [chunk("same", "note.md", "beta text")])]) + + assert store.file_chunks["same"].embedding.tolist() == [0.0, 1.0] + await store.close() + + run(go()) + + +def test_search_filter_applies_to_vector_and_keyword_results(): + """Search filters apply consistently to vector and keyword results.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + store = LocalFileStore(name="t_filter", embedding_store="") + await store.start() + store.embedding_store = FakeEmbeddingStore() + + await store.upsert( + [ + (node("daily/a.md"), [chunk("a", "daily/a.md", "fresh topic", kind="daily")]), + (node("resource/b.md"), [chunk("b", "resource/b.md", "fresh topic", kind="resource")]), + ], + ) + + filt = {"path_prefix": "daily/", "metadata": {"kind": "daily"}} + assert [c.path for c in await store.vector_search("fresh", 5, filt)] == ["daily/a.md"] + assert [c.path for c in await store.keyword_search("fresh", 5, filt)] == ["daily/a.md"] + await store.close() + + run(go()) + + +def test_faiss_rebuilds_stale_sidecar_and_updates_same_id_text(): + """FAISS sidecar rebuilds when persisted rows no longer match chunks.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + try: + store = FaissLocalFileStore(name="t_faiss", embedding_store="") + except ImportError: + pytest.skip("faiss is not installed") + await store.start() + store.embedding_store = FakeEmbeddingStore() + store._faiss_index = store._new_index() + + await store.upsert([(node("note.md"), [chunk("same", "note.md", "alpha text")])]) + assert [c.id for c in await store.vector_search("alpha", 5, {})] == ["same"] + + await store.upsert([(node("note.md"), [chunk("same", "note.md", "beta text")])]) + assert [c.id for c in await store.vector_search("beta", 5, {})] == ["same"] + assert store._id_to_row["same"] == 1 + + await store.dump() + store.file_chunks = {"other": chunk("other", "other.md", "alpha text")} + store.file_chunks["other"].embedding = np.array([1.0, 0.0], dtype=np.float16) + + assert await store._try_load_sidecar() is False + store._rebuild_index() + assert set(store._id_to_row) == {"other"} + await store.close() + + run(go()) diff --git a/tests4/unit/test_job.py b/tests4/unit/test_job.py index b2e222fd..e544cb6d 100644 --- a/tests4/unit/test_job.py +++ b/tests4/unit/test_job.py @@ -3,13 +3,20 @@ # pylint: disable=protected-access,missing-function-docstring,missing-class-docstring,no-self-argument,unused-argument import asyncio +from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock import pytest +from reme4.enumeration import ComponentEnum +from reme4.application import Application +from reme4.components.base_component import BaseComponent from reme4.components.component_registry import ComponentRegistry from reme4.components.job.background_job import BackgroundJob from reme4.components.job.base_job import BaseJob +from reme4.components.job.cron_job import CronJob +from reme4.components.job.stream_job import StreamJob +from reme4.components.job import cron_job as cron_job_module from reme4.schema import ComponentConfig @@ -86,6 +93,46 @@ def test_call_runs_steps_in_order(): asyncio.run(run()) +def test_base_job_merges_config_kwargs_into_context(): + async def run(): + seen = {} + + async def step(ctx): + seen.update(ctx.data) + + job = BaseJob(name="j", default_value="from-config") + job.app_context = MagicMock() + job.step_specs = [] + job._build_steps = lambda: [step] + + response = await job(default_value="from-call", call_only=True) + assert response.success is True + assert seen == {"default_value": "from-call", "call_only": True} + + asyncio.run(run()) + + +def test_stream_job_merges_config_kwargs_into_context(): + async def run(): + seen = {} + + async def step(ctx): + seen.update(ctx.data) + + queue = asyncio.Queue() + job = StreamJob(name="j", default_value="from-config") + job.app_context = MagicMock() + job.step_specs = [] + job._build_steps = lambda: [step] + + await job(stream_queue=queue) + done = await queue.get() + assert done.done is True + assert seen["default_value"] == "from-config" + + asyncio.run(run()) + + # -- BaseJob._start requires app_context ------------------------------------ @@ -228,6 +275,91 @@ def test_shutdown_task_cancels_on_timeout(): asyncio.run(run()) +def test_cron_uses_configured_timezone(monkeypatch): + from zoneinfo import ZoneInfo + + seen = {} + + def zone_info(name): + seen["timezone"] = name + return ZoneInfo(name) + + monkeypatch.setattr(cron_job_module, "ZoneInfo", zone_info) + + job = CronJob(cron="0 0 * * *") + job.app_context = SimpleNamespace(app_config=SimpleNamespace(timezone="America/New_York")) + delay = job._next_fire_delay() + + assert delay > 0 + assert seen["timezone"] == "America/New_York" + + +def test_application_starts_jobs_base_stream_background_cron(): + async def run(): + order = [] + app = object.__new__(Application) + app.context = SimpleNamespace( + app_config=SimpleNamespace(thread_pool_max_workers=0), + jobs={ + "cron": CronJob(cron="* * * * *", name="cron"), + "background": BackgroundJob(name="background"), + "stream": StreamJob(name="stream"), + "base": BaseJob(name="base"), + }, + thread_pool=None, + ) + app._topological_order = lambda: [] + + async def start_one(component): + order.append(component.name) + + app._start_one = start_one + + await Application._start(app) + assert order == ["base", "stream", "background", "cron"] + + asyncio.run(run()) + + +def test_application_start_failure_propagates_and_closes_started_components(): + async def run(): + class GoodComponent(BaseComponent): + component_type = ComponentEnum.TOKENIZER + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.closed = False + + async def _close(self): + self.closed = True + + class BrokenComponent(BaseComponent): + component_type = ComponentEnum.FILE_STORE + + async def _start(self): + raise RuntimeError("boom") + + good = GoodComponent(name="good") + bad = BrokenComponent(name="bad") + app = object.__new__(Application) + app.context = SimpleNamespace( + app_config=SimpleNamespace(thread_pool_max_workers=0), + jobs={}, + thread_pool=None, + ) + app._started_components = [] + app._topological_order = lambda: [good, bad] + app.logger = MagicMock() + + with pytest.raises(RuntimeError, match="boom"): + await Application._start(app) + + assert good.closed is True + assert not app._started_components + + asyncio.run(run()) + + if __name__ == "__main__": print("\n=== Job Tests ===") test_resolve_step_missing_backend() diff --git a/tests4/unit/test_keyword_index.py b/tests4/unit/test_keyword_index.py index 311d550b..b49c7a2c 100644 --- a/tests4/unit/test_keyword_index.py +++ b/tests4/unit/test_keyword_index.py @@ -137,7 +137,8 @@ def test_index_file_path_includes_tokenizer_and_version(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): bm25 = await create_bm25() path = str(bm25.index_file) - assert "bm25_regex_v1.pkl" in path + assert "bm25_BM25Index_regex_" in path + assert path.endswith("_v1.pkl") await bm25.close() run(go()) @@ -731,6 +732,82 @@ def test_load_corrupt_file_resets_index(): run(go()) +def test_index_file_isolated_by_component_name(): + """Different BM25Index names must not share one persisted pickle.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + alpha = BM25Index(name="alpha") + alpha_tokenizer = RegexTokenizer(filter_stopwords=False) + alpha.tokenizer = alpha_tokenizer + alpha._owned.append(alpha_tokenizer) + await alpha.start() + + beta = BM25Index(name="beta") + beta_tokenizer = RegexTokenizer(filter_stopwords=False) + beta.tokenizer = beta_tokenizer + beta._owned.append(beta_tokenizer) + await beta.start() + + assert alpha.index_file != beta.index_file + + await alpha.add_docs({"d1": "alpha only"}) + await alpha.dump() + await alpha.close() + + assert beta.n_docs == 0 + assert await beta.retrieve("alpha", limit=1) == {} + await beta.close() + + run(go()) + + +def test_index_file_isolated_by_tokenizer_config(): + """Tokenizer settings that affect tokens must map to different index files.""" + + async def go(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + unfiltered = BM25Index() + unfiltered_tokenizer = RegexTokenizer(filter_stopwords=False) + unfiltered.tokenizer = unfiltered_tokenizer + unfiltered._owned.append(unfiltered_tokenizer) + await unfiltered.start() + + filtered = BM25Index() + filtered_tokenizer = RegexTokenizer(filter_stopwords=True) + filtered.tokenizer = filtered_tokenizer + filtered._owned.append(filtered_tokenizer) + await filtered.start() + + assert unfiltered.index_file != filtered.index_file + + await unfiltered.close() + await filtered.close() + + run(go()) + + +def test_dump_failure_is_not_silent(): + """A failed write must be observable by callers.""" + + async def go(): + from unittest.mock import patch + + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + bm25 = await create_bm25() + await bm25.add_docs({"d1": "hello"}) + with patch("builtins.open", side_effect=OSError("disk full")): + try: + await bm25.dump() + except OSError: + pass + else: + raise AssertionError("expected dump() to raise OSError") + await bm25.close() + + run(go()) + + # --------------------------------------------------------------------------- # # clear / optimize / reset_index # # --------------------------------------------------------------------------- # @@ -755,6 +832,7 @@ def test_clear_wipes_everything(): assert bm25._idf_cache == {} assert not bm25.index_file.exists() await bm25.close() + assert not bm25.index_file.exists() run(go()) diff --git a/tests4/unit/test_markdown_file_chunker.py b/tests4/unit/test_markdown_file_chunker.py index 91ebc432..86f86ad5 100644 --- a/tests4/unit/test_markdown_file_chunker.py +++ b/tests4/unit/test_markdown_file_chunker.py @@ -218,6 +218,59 @@ def test_parse_embed_toc_prefixes_chunk_text(): asyncio.run(run()) +def test_parse_frontmatter_preserves_original_line_numbers(): + """Chunk line ranges are 1-based and refer to the original file, including frontmatter.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + body = "---\nname: t\n---\n# H\nline 1\nline 2\n" + path = _write_md(tmp, "front-lines.md", body) + chunker = MarkdownFileChunker(chunk_chars=500) + _, chunks = await chunker.chunk(path) + assert len(chunks) == 1 + assert chunks[0].start_line == 4 + assert chunks[0].end_line == 6 + print("✓ test_parse_frontmatter_preserves_original_line_numbers passed") + + asyncio.run(run()) + + +def test_parse_frontmatter_offsets_split_table_rows(): + """Split table row ranges include the YAML frontmatter line offset.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + rows = "".join(f"| {i} | {i} |\n" for i in range(12)) + body = "---\nname: t\n---\n| A | B |\n|---|---|\n" + rows + path = _write_md(tmp, "front-table.md", body) + chunker = MarkdownFileChunker(chunk_chars=100) + _, chunks = await chunker.chunk(path) + assert len(chunks) > 1 + assert chunks[0].start_line == 6 + assert chunks[0].end_line >= chunks[0].start_line + print("✓ test_parse_frontmatter_offsets_split_table_rows passed") + + asyncio.run(run()) + + +def test_parse_bad_frontmatter_does_not_abort_chunking(): + """Invalid YAML frontmatter is ignored while the markdown body still chunks.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + body = "---\nname: [\n---\n# H\nbody\n" + path = _write_md(tmp, "bad-frontmatter.md", body) + chunker = MarkdownFileChunker(chunk_chars=500) + node, chunks = await chunker.chunk(path) + assert node.front_matter.name == "" + assert len(chunks) == 1 + assert chunks[0].start_line == 4 + assert "body" in chunks[0].text + print("✓ test_parse_bad_frontmatter_does_not_abort_chunking passed") + + asyncio.run(run()) + + if __name__ == "__main__": print("\n=== MarkdownFileChunker tests ===") test_parse_empty_file() @@ -231,4 +284,7 @@ if __name__ == "__main__": test_parse_links_deduped() test_parse_min_chunk_chars_clamped() test_parse_embed_toc_prefixes_chunk_text() + test_parse_frontmatter_preserves_original_line_numbers() + test_parse_frontmatter_offsets_split_table_rows() + test_parse_bad_frontmatter_does_not_abort_chunking() print("\n所有测试通过!") diff --git a/tests4/unit/test_neo4j_file_graph.py b/tests4/unit/test_neo4j_file_graph.py deleted file mode 100644 index dc212126..00000000 --- a/tests4/unit/test_neo4j_file_graph.py +++ /dev/null @@ -1,336 +0,0 @@ -"""Tests for Neo4jFileGraph. - -Skipped automatically if (a) the ``neo4j`` driver isn't installed, -or (b) a Neo4j instance isn't reachable at the configured URI. - -Override the URI / auth via env vars: - NEO4J_URI (default ``bolt://localhost:7687``) - NEO4J_USER (default ``neo4j``) - NEO4J_PASSWORD (default ``neo4j``) - NEO4J_DATABASE (default ``neo4j``) -""" - -# pylint: disable=protected-access - -import asyncio -import os -import tempfile - -import pytest - -from reme4.schema import FileLink, FileNode - - -URI = os.environ.get("NEO4J_URI", "bolt://localhost:7687") -USER = os.environ.get("NEO4J_USER", "neo4j") -PASSWORD = os.environ.get("NEO4J_PASSWORD", "neo4j") -DATABASE = os.environ.get("NEO4J_DATABASE", "neo4j") - - -try: - from reme4.components.file_graph import Neo4jFileGraph - - _NEO4J_IMPORT_ERROR: Exception | None = None -except Exception as e: # pragma: no cover - _NEO4J_IMPORT_ERROR = e - - -async def _probe_neo4j() -> str | None: - """Try to connect; return None if OK, error string if unreachable.""" - if _NEO4J_IMPORT_ERROR is not None: - return f"import failed: {_NEO4J_IMPORT_ERROR}" - try: - from neo4j import AsyncGraphDatabase - except ImportError as e: - return f"neo4j driver not installed: {e}" - try: - driver = AsyncGraphDatabase.driver(URI, auth=(USER, PASSWORD)) - async with driver.session(database=DATABASE) as session: - await session.run("RETURN 1") - await driver.close() - return None - except Exception as e: # pragma: no cover - return f"connect failed: {e}" - - -_PROBE_NOT_RUN = object() -_PROBE_REASON: str | None | object = _PROBE_NOT_RUN # sentinel: "not probed" - - -def _probe_reason() -> str | None: - """Probe Neo4j once per process; cache the outcome.""" - global _PROBE_REASON - if _PROBE_REASON is _PROBE_NOT_RUN: - _PROBE_REASON = asyncio.run(_probe_neo4j()) - return _PROBE_REASON # type: ignore[return-value] - - -pytestmark = pytest.mark.skipif( - _probe_reason() is not None, - reason=f"Neo4j unavailable: {_probe_reason()}", -) - - -class temp_chdir: - """Context manager to temporarily chdir into a path and restore on exit.""" - - def __init__(self, path): - self.path = path - self.old = None - - def __enter__(self): - self.old = os.getcwd() - os.chdir(self.path) - return self - - def __exit__(self, *exc): - os.chdir(self.old) - - -def make_node(path: str, links: list[tuple[str, str | None]] | None = None) -> FileNode: - """Build a FileNode with outgoing (target_path, target_anchor) pairs.""" - return FileNode( - path=path, - st_mtime=1.0, - links=[FileLink(source_path=path, target_path=t, target_anchor=a) for t, a in (links or [])], - ) - - -async def _fresh_graph() -> "Neo4jFileGraph": # type: ignore[name-defined] - """Build a started Neo4jFileGraph wiped clean.""" - graph = Neo4jFileGraph(uri=URI, user=USER, password=PASSWORD, database=DATABASE) - await graph.start() - await graph.clear() - return graph - - -def test_upsert_and_get_nodes(): - """upsert_nodes stores; get_nodes returns by paths or all.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [make_node("a.md", [("b.md", None)]), make_node("b.md")], - ) - got_all = await graph.get_nodes() - assert {n.path for n in got_all} == {"a.md", "b.md"} - got_one = await graph.get_nodes(["a.md"]) - assert len(got_one) == 1 and got_one[0].path == "a.md" - assert await graph.get_nodes(["nope.md"]) == [] - assert await graph.get_nodes([]) == [] - finally: - await graph.clear() - await graph.close() - print("✓ test_upsert_and_get_nodes passed") - - asyncio.run(run()) - - -def test_outlinks_skip_virtual_targets(): - """get_outlinks excludes edges into virtual placeholder nodes.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [ - make_node("a.md", [("b.md", None), ("ghost.md", None)]), - make_node("b.md"), - ], - ) - outs = await graph.get_outlinks("a.md") - assert {link.target_path for link in outs} == {"b.md"} - for link in outs: - assert link.source_path == "a.md" - finally: - await graph.clear() - await graph.close() - print("✓ test_outlinks_skip_virtual_targets passed") - - asyncio.run(run()) - - -def test_inlinks_carry_source_path(): - """get_inlinks returns FileLinks whose source_path is the linking node.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [ - make_node("a.md", [("b.md", "anchor1")]), - make_node("c.md", [("b.md", None)]), - make_node("b.md"), - ], - ) - ins = await graph.get_inlinks("b.md") - sources = {link.source_path for link in ins} - assert sources == {"a.md", "c.md"} - # Each link's target should be the queried path. - for link in ins: - assert link.target_path == "b.md" - finally: - await graph.clear() - await graph.close() - print("✓ test_inlinks_carry_source_path passed") - - asyncio.run(run()) - - -def test_delete_demotes_then_repromotes(): - """delete_nodes makes a node virtual; re-upsert promotes pending edges back.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [make_node("a.md", [("b.md", None)]), make_node("b.md")], - ) - assert {link.source_path for link in await graph.get_inlinks("b.md")} == {"a.md"} - - await graph.delete_nodes(["b.md"]) - assert await graph.get_nodes(["b.md"]) == [] - # a's outlink is hidden because b is now virtual. - assert await graph.get_outlinks("a.md") == [] - - await graph.upsert_nodes([make_node("b.md")]) - assert {link.source_path for link in await graph.get_inlinks("b.md")} == {"a.md"} - finally: - await graph.clear() - await graph.close() - print("✓ test_delete_demotes_then_repromotes passed") - - asyncio.run(run()) - - -def test_rebuild_links_idempotent(): - """rebuild_links reconstructs identical out/in views from per-node payloads.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [ - make_node("a.md", [("b.md", None), ("c.md", "h")]), - make_node("b.md"), - make_node("c.md"), - ], - ) - before_out = sorted((link.target_path, link.target_anchor) for link in await graph.get_outlinks("a.md")) - before_in_b = sorted(link.source_path for link in await graph.get_inlinks("b.md")) - - await graph.rebuild_links() - - after_out = sorted((link.target_path, link.target_anchor) for link in await graph.get_outlinks("a.md")) - after_in_b = sorted(link.source_path for link in await graph.get_inlinks("b.md")) - assert before_out == after_out - assert before_in_b == after_in_b - finally: - await graph.clear() - await graph.close() - print("✓ test_rebuild_links_idempotent passed") - - asyncio.run(run()) - - -def test_clear_wipes_everything(): - """clear() drops every node and edge in the database.""" - - async def run(): - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - await graph.upsert_nodes( - [make_node("a.md", [("b.md", None)]), make_node("b.md")], - ) - await graph.clear() - assert await graph.get_nodes() == [] - finally: - await graph.close() - print("✓ test_clear_wipes_everything passed") - - asyncio.run(run()) - - -def test_node_roundtrip_preserves_frontmatter_and_links(): - """Upsert → get_nodes round-trip preserves frontmatter + links + chunk_ids.""" - - async def run(): - from reme4.schema.file_node import FileFrontMatter - - with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): - graph = await _fresh_graph() - try: - node = FileNode( - path="topics/Alice.md", - st_mtime=1234.5, - links=[ - FileLink( - source_path="topics/Alice.md", - target_path="topics/Bob.md", - target_anchor="intro", - predicate="knows", - ), - ], - chunk_ids=["chunk-a1", "chunk-a2", "chunk-a3"], - front_matter=FileFrontMatter( - name="Alice", - description="a person", - ), - ) - await graph.upsert_nodes([node, make_node("topics/Bob.md")]) - got = await graph.get_nodes(["topics/Alice.md"]) - assert len(got) == 1 - back = got[0] - assert back.path == "topics/Alice.md" - assert back.st_mtime == 1234.5 - assert back.front_matter.name == "Alice" - assert back.front_matter.description == "a person" - assert back.chunk_ids == ["chunk-a1", "chunk-a2", "chunk-a3"] - assert len(back.links) == 1 - link = back.links[0] - assert link.source_path == "topics/Alice.md" - assert link.target_path == "topics/Bob.md" - assert link.target_anchor == "intro" - assert link.predicate == "knows" - - # An upsert with empty chunk_ids should also round-trip cleanly - # (and overwrite the previous list). - await graph.upsert_nodes( - [ - FileNode( - path="topics/Alice.md", - st_mtime=1234.5, - chunk_ids=[], - ), - ], - ) - got2 = await graph.get_nodes(["topics/Alice.md"]) - assert got2 and got2[0].chunk_ids == [] - finally: - await graph.clear() - await graph.close() - print("✓ test_node_roundtrip_preserves_frontmatter_and_links passed") - - asyncio.run(run()) - - -if __name__ == "__main__": - if _probe_reason() is not None: - print(f"Skipping Neo4j tests: {_probe_reason()}") - else: - print("\n=== Neo4jFileGraph tests ===") - test_upsert_and_get_nodes() - test_outlinks_skip_virtual_targets() - test_inlinks_carry_source_path() - test_delete_demotes_then_repromotes() - test_rebuild_links_idempotent() - test_clear_wipes_everything() - test_node_roundtrip_preserves_frontmatter_and_links() - print("\n所有测试通过!") diff --git a/tests4/unit/test_packaging.py b/tests4/unit/test_packaging.py new file mode 100644 index 00000000..b85de529 --- /dev/null +++ b/tests4/unit/test_packaging.py @@ -0,0 +1,13 @@ +"""Tests for package metadata that affects runtime assets.""" + +import tomllib +from pathlib import Path + + +def test_reme4_packages_tokenizer_stopwords(): + """The default tokenizer stopwords file must be included in built packages.""" + pyproject = Path(__file__).parents[2] / "reme4" / "pyproject.toml" + data = tomllib.loads(pyproject.read_text(encoding="utf-8")) + package_data = data["tool"]["setuptools"]["package-data"] + + assert "stopwords" in package_data["reme4.components.tokenizer"] diff --git a/tests4/unit/test_prompt_handler.py b/tests4/unit/test_prompt_handler.py index bd0fa90b..9837026d 100644 --- a/tests4/unit/test_prompt_handler.py +++ b/tests4/unit/test_prompt_handler.py @@ -211,16 +211,9 @@ def test_format_variables(): def test_format_missing_variable_raises(): ph = PromptHandler() - ph.load_prompt_dict({"p": "Hello {name}"}) - with pytest.raises(ValueError, match="Missing format variables"): - ph.prompt_format("p") - - -def test_format_missing_variable_no_validate(): - ph = PromptHandler() - ph.load_prompt_dict({"p": "Hello {name}"}) - result = ph.prompt_format("p", validate=False) - assert "{name}" in result + ph.load_prompt_dict({"p": "Hello {name}, welcome to {place}"}) + with pytest.raises(KeyError, match="place"): + ph.prompt_format("p", name="Alice") def test_format_no_variables_no_error(): @@ -296,7 +289,6 @@ if __name__ == "__main__": test_flag_filter_unflagged_lines_always_kept() test_format_variables() test_format_missing_variable_raises() - test_format_missing_variable_no_validate() test_format_no_variables_no_error() test_format_flags_and_variables_combined() test_format_flags_false_variable_not_needed() diff --git a/tests4/unit/test_reme_cli.py b/tests4/unit/test_reme_cli.py new file mode 100644 index 00000000..62575f21 --- /dev/null +++ b/tests4/unit/test_reme_cli.py @@ -0,0 +1,46 @@ +"""Tests for the ReMe CLI entry helpers.""" + +from reme4 import reme as reme_module + + +def test_call_server_passes_client_kwargs_to_client(monkeypatch, capsys): + """CLI helper forwards connection options to the selected client.""" + seen = {} + + class FakeClient: + """Async client stub that records call arguments.""" + + def __init__(self, **kwargs): + seen["client_kwargs"] = kwargs + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc_val, exc_tb): + return None + + async def __call__(self, action: str, **kwargs): + seen["action"] = action + seen["payload"] = kwargs + yield "ok" + + monkeypatch.setattr(reme_module.R, "get", lambda component_type, backend: FakeClient) + + async def run(): + await reme_module.call_server( + "search", + backend="http", + host="127.0.0.2", + port=2444, + timeout=1.5, + query="hello", + ) + + import asyncio + + asyncio.run(run()) + + assert seen["client_kwargs"] == {"host": "127.0.0.2", "port": 2444, "timeout": 1.5} + assert seen["action"] == "search" + assert seen["payload"] == {"query": "hello"} + assert capsys.readouterr().out == "ok\n" diff --git a/tests4/unit/test_search_step.py b/tests4/unit/test_search_step.py new file mode 100644 index 00000000..908dea58 --- /dev/null +++ b/tests4/unit/test_search_step.py @@ -0,0 +1,145 @@ +"""Unit tests for SearchStep without embedding or LLM dependencies.""" + +import asyncio + +from reme4.components.file_store import BaseFileStore +from reme4.components.runtime_context import RuntimeContext +from reme4.enumeration import LinkScopeEnum +from reme4.schema import FileChunk, FileLink, FileNode +from reme4.steps.index import SearchStep + + +class FakeSearchStore(BaseFileStore): + """Minimal file_store for SearchStep: static search results and empty graph links.""" + + def __init__( + self, + vector_results: list[FileChunk] | None = None, + keyword_results: list[FileChunk] | None = None, + ): + super().__init__(name="fake_search_store") + self.vector_results = vector_results or [] + self.keyword_results = keyword_results or [] + self.calls: list[tuple[str, str, int, dict]] = [] + + async def upsert(self, files: list[tuple[FileNode, list[FileChunk]]]) -> None: + raise NotImplementedError + + async def delete(self, path: str | list[str]) -> None: + raise NotImplementedError + + async def clear(self) -> None: + raise NotImplementedError + + async def get_nodes(self, paths: list[str] | None = None) -> list[FileNode]: + return [] + + async def get_outlinks( + self, + path: str, + scope: LinkScopeEnum = LinkScopeEnum.REAL, + ) -> list[FileLink]: + return [] + + async def get_inlinks( + self, + path: str, + scope: LinkScopeEnum = LinkScopeEnum.REAL, + ) -> list[FileLink]: + return [] + + async def vector_search(self, query: str, limit: int, search_filter: dict) -> list[FileChunk]: + self.calls.append(("vector", query, limit, search_filter)) + return self.vector_results[:limit] + + async def keyword_search(self, query: str, limit: int, search_filter: dict) -> list[FileChunk]: + self.calls.append(("keyword", query, limit, search_filter)) + return self.keyword_results[:limit] + + +def _chunk( + chunk_id: str, + path: str, + text: str, + score_key: str, + score: float, + line: int = 1, +) -> FileChunk: + return FileChunk( + id=chunk_id, + path=path, + text=text, + start_line=line, + end_line=line, + scores={score_key: score, "score": score}, + ) + + +def test_search_step_rrf_merges_vector_and_keyword_by_chunk_id(): + """Hybrid search fuses same-id hits once and keeps per-branch scores in metadata.""" + + async def run(): + shared_v = _chunk("shared", "daily/a.md", "shared vector text", "vector", 0.92, line=3) + vector_only = _chunk("vector-only", "daily/b.md", "vector text", "vector", 0.71) + keyword_only = _chunk("keyword-only", "digest/c.md", "keyword text", "keyword", 8.0) + shared_k = _chunk("shared", "daily/a.md", "shared keyword text", "keyword", 7.0, line=3) + store = FakeSearchStore( + vector_results=[shared_v, vector_only], + keyword_results=[keyword_only, shared_k], + ) + step = SearchStep(file_store=store, vector_weight=0.5, candidate_multiplier=2, expand_links=False) + ctx = RuntimeContext(query="alpha", limit=3, search_filter={"path_prefix": "daily/"}) + + resp = await step(ctx) + + assert resp.success is True + assert resp.metadata["counts"] == {"vector": 2, "keyword": 2, "returned": 3, "hybrid": True} + assert [r["id"] for r in resp.metadata["results"]] == ["shared", "keyword-only", "vector-only"] + shared = resp.metadata["results"][0] + assert shared["scores"]["vector"] == 0.92 + assert shared["scores"]["keyword"] == 7.0 + assert shared["scores"]["score"] > resp.metadata["results"][1]["scores"]["score"] + assert "daily/a.md:3-3" in resp.answer + assert "vector=0.9200" in resp.answer + assert "keyword=7.0000" in resp.answer + assert {call[0] for call in store.calls} == {"vector", "keyword"} + assert all(call[2] == 6 for call in store.calls) + assert all(call[3] == {"path_prefix": "daily/"} for call in store.calls) + + asyncio.run(run()) + + +def test_search_step_keyword_only_uses_keyword_scores_and_min_score(): + """When vector has no hits, SearchStep returns keyword results directly and applies min_score.""" + + async def run(): + high = _chunk("high", "daily/high.md", "strong keyword hit", "keyword", 4.0) + low = _chunk("low", "daily/low.md", "weak keyword hit", "keyword", 0.2) + store = FakeSearchStore(keyword_results=[high, low]) + step = SearchStep(file_store=store, expand_links=False) + ctx = RuntimeContext(query="keyword", limit=5, min_score=1.0) + + resp = await step(ctx) + + assert resp.metadata["counts"] == {"vector": 0, "keyword": 2, "returned": 1, "hybrid": False} + assert [r["id"] for r in resp.metadata["results"]] == ["high"] + assert "keyword=4.0000" not in resp.answer + assert "score=4.0000" in resp.answer + assert "daily/low.md" not in resp.answer + + asyncio.run(run()) + + +def test_search_step_empty_query_fails_before_store_calls(): + """Empty queries fail fast and do not call file_store search methods.""" + + async def run(): + store = FakeSearchStore() + step = SearchStep(file_store=store) + resp = await step(RuntimeContext(query=" ", limit=5)) + + assert resp.success is False + assert resp.answer == "Error: query cannot be empty" + assert not store.calls + + asyncio.run(run()) diff --git a/tests4/unit/test_service.py b/tests4/unit/test_service.py new file mode 100644 index 00000000..02046f79 --- /dev/null +++ b/tests4/unit/test_service.py @@ -0,0 +1,43 @@ +"""Tests for service job registration behavior.""" + +from types import SimpleNamespace + +from reme4.components.job import BaseJob, StreamJob +from reme4.components.service import MCPService + + +def _dummy_app(): + """Minimal object needed by MCPService.build_service.""" + + async def start(): + return None + + async def close(): + return None + + return SimpleNamespace( + config=SimpleNamespace(app_name="test"), + context=SimpleNamespace(metadata={}), + start=start, + close=close, + ) + + +def test_mcp_service_registers_job_with_empty_parameters(): + """Empty job parameters must remain a dict for FastMCP FunctionTool validation.""" + service = MCPService() + service.build_service(_dummy_app()) + + job = BaseJob(name="empty_params", parameters={}) + + assert service.add_job(job) is True + + +def test_mcp_service_reports_stream_job_skipped(): + """MCPService intentionally does not expose StreamJob tools.""" + service = MCPService() + service.build_service(_dummy_app()) + + job = StreamJob(name="stream") + + assert service.add_job(job) is False diff --git a/tests4/unit/test_tokenizer.py b/tests4/unit/test_tokenizer.py index 5dc99308..c3fcdb3d 100644 --- a/tests4/unit/test_tokenizer.py +++ b/tests4/unit/test_tokenizer.py @@ -157,6 +157,17 @@ def test_tokenizer_lifecycle(): asyncio.run(run()) +def test_jieba_tokenizer_requires_start(): + """JiebaTokenizer should fail clearly when used before startup.""" + tokenizer = JiebaTokenizer(filter_stopwords=False) + try: + tokenizer.tokenize(["hello 世界"]) + except RuntimeError as exc: + assert "Call start() first" in str(exc) + else: + raise AssertionError("expected RuntimeError when JiebaTokenizer is used before start()") + + if __name__ == "__main__": print("\n=== Tokenizer Tests ===") test_basic_chinese() @@ -166,4 +177,5 @@ if __name__ == "__main__": test_with_stopwords() test_multiple_texts() test_tokenizer_lifecycle() + test_jieba_tokenizer_requires_start() print("\n所有测试通过!") diff --git a/tests4/unit/test_utils.py b/tests4/unit/test_utils.py new file mode 100644 index 00000000..2bdcc028 --- /dev/null +++ b/tests4/unit/test_utils.py @@ -0,0 +1,67 @@ +"""Tests for small utilities in ``reme4.utils``.""" + +import asyncio +import sys + +import numpy as np +import pytest + +from reme4.utils import common_utils +from reme4.utils.similarity_utils import batch_cosine_similarity, cosine_similarity + + +def test_batch_cosine_similarity_rejects_1d_inputs(): + """1D vectors should fail with a clear validation error, not IndexError.""" + with pytest.raises(ValueError, match="Expected 2D arrays"): + batch_cosine_similarity(np.array([1.0, 0.0]), np.array([[1.0, 0.0]])) + + +def test_batch_cosine_similarity_pairwise_matrix(): + """Batch cosine returns the full pairwise matrix for valid 2D inputs.""" + result = batch_cosine_similarity( + np.array([[1.0, 0.0], [0.0, 1.0]]), + np.array([[1.0, 0.0], [1.0, 1.0]]), + ) + + assert result.shape == (2, 2) + np.testing.assert_allclose(result[0], [1.0, 2**-0.5]) + np.testing.assert_allclose(result[1], [0.0, 2**-0.5]) + + +def test_cosine_similarity_rejects_mismatched_lengths(): + """Single-vector cosine validates dimensions before computing.""" + with pytest.raises(ValueError, match="Vectors must have same length"): + cosine_similarity([1.0], [1.0, 2.0]) + + +def test_mock_reme_server_uses_reme4_entrypoint(monkeypatch): + """The test server helper should spawn the reme4 CLI module, not legacy reme.""" + captured: dict[str, list[str]] = {} + + class DummyProcess: + """Process stub returned by the patched Popen.""" + + stdout = None + + def poll(self): + """Return a successful process status.""" + return 0 + + def fake_popen(cmd, **_kwargs): + """Capture the spawned command.""" + captured["cmd"] = cmd + return DummyProcess() + + async def fake_wait_ready(_host, _port, _timeout): + return None + + monkeypatch.setattr(common_utils.subprocess, "Popen", fake_popen) + monkeypatch.setattr(common_utils, "_wait_reme_ready", fake_wait_ready) + + async def run(): + async with common_utils.mock_reme_server(port=45678, log_to_file=False, enable_logo=False): + pass + + asyncio.run(run()) + + assert captured["cmd"][:4] == [sys.executable, "-m", "reme4.reme", "start"]