scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-08-28 05:25:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

feat(core): enhance reme4 (#281)

Browse source 
### 1. Agent Wrapper（统一 Agent 后端抽象）
- **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。
- **`as_agent_wrapper.py`（AgentScope 后端）**：
  - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。
  - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。
  - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。
- **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**：
  - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。
  - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。
  - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。

### 2. 统一流式协议（StreamChunk / ChunkEnum）
- **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。
- **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。

### 3. Index 模块重构（变化批次化 + dispatch）
- 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。
- 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。
- 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。
- **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。
- 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。

### 4. Evolve / Dream 模块（拆分为多步 pipeline）
- 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织：
  - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。
  - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。
  - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。
  - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。
  - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。
- 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。
- `evolve/__init__.py` 导出全部新 step。

### 5. auto_memory / auto_resource（适配新 Agent API）
- **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。
- **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。

### 6. BaseStep 基类增强
- 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。
- 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。
- 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。

### 7. Components 改动
- **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。
- **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。
- **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。
- **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。
- 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。

### 8. Application 生命周期
- `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。
- 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。

### 9. File IO / 路径安全
- **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。
- **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。
- **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。
- **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。

### 10. Config
- `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。
- `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。
- `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。

### 11. 其它
- 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。
- 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。
**
This commit is contained in:
jinliyl 2026-06-19 01:35:31 +08:00 • committed by GitHub
parent f458566e2c
commit 83831ec90c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
136 changed files with 8168 additions and 4307 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

943
docs4/auto_dream_logic_and_step_refactor.md Normal file
View file

@ -0,0 +1,943 @@
# auto_dream 逻辑解读与 Step 拆分方案
## 1. 配置入口
`/Users/yuli/workspace/ReMe/reme4/config/default.yaml` 里和 auto dream 相关的是四个 job:
| job | 用途 | 当前 steps |
|---|---|---|
| `dream` | 对单个文件做完整 dream,并顺手写一次 daily topics | `dream_step` -> `daily_topics_step` |
| `dream_extract` | auto_dream 内部使用的单文件 dream,只抽取和整合 digest,不写 daily topics | `dream_step` |
| `auto_dream` | 扫描某一天的 daily index 和 session notes,只处理新增/修改文件,最后聚合写当天兴趣主题 | `auto_dream_step` |
| `daily_topics` | 从 dream 产生的 topic candidates 中选最终兴趣主题,写 `daily/<date>/interests.md` | `daily_topics_step` |
关键点:
- `dream` 和 `auto_dream` 不是同一条链路。
- `dream` 是单文件命令,执行 `dream_step` 后立刻执行 `daily_topics_step`。
- `auto_dream` 默认 `dispatch_job: dream_extract`,所以它对每个文件只跑 `dream_step`,把所有文件产生的 `topic_candidates` 收集起来,最后只调用一次 `daily_topics`。
- `auto_dream` 默认写 3 个 topic,回看 7 天去重,输出 session id 是 `interests`。
配置片段的实际语义:
```yaml
auto_dream:
steps:
- backend: auto_dream_step
dispatch_job: dream_extract
emit_topics: true
topic_dispatch_job: daily_topics
topic_count: 3
topic_diversity_days: 7
topic_session_id: interests
```
也就是说,`auto_dream` 本身是一个调度器和增量扫描器,真正的 LLM dream 逻辑在 `DreamStep`,topic 写入在 `DailyTopicsStep`。
## 2. auto_dream_step 执行链路
实现位置:
- `reme4/steps/evolve/auto_dream.py`
- `reme4/steps/evolve/dream.py`
- `reme4/steps/evolve/daily_topics.py`
- `reme4/steps/file_io/_daily_index.py`
### 2.1 读取输入和默认日期
`AutoDreamStep.execute()` 从 runtime context 读取:
| 参数 | 语义 |
|---|---|
| `date` | 要扫描的日期,空字符串时用配置 timezone 下的今天 |
| `hint` | 透传给每个单文件 dream 的提示 |
日期默认逻辑:
```text
date_input 非空 -> 使用 date_input
date_input 为空 -> now(app_config.timezone).strftime("%Y-%m-%d")
```
`daily_dir` 不来自用户参数,而是来自 app config,默认是 `daily`。
### 2.2 先刷新 day-index
正式扫描前,auto_dream 会先调用:
```python
await refresh_day_index(self.file_store, today, daily_dir)
```
它会重建:
```text
daily/<date>.md
```
这个 day-index 文件包含 `daily/<date>/` 下每个 session note 的链接和 frontmatter 摘要。这样 auto_dream 后续处理的第一个文件就是当天总览。
隐含结果:
- 如果 session notes 有新增/删除/frontmatter 变化,day-index 的内容可能变化。
- day-index 被放在扫描列表第一位,所以当天总览先于具体 session note 被 dream。
### 2.3 扫描当天文件范围
扫描范围由 `_scan_today_files(vault, today, daily_dir)` 决定:
```text
1. daily/<date>.md
2. daily/<date>/**/*.md
```
处理顺序:
```text
daily/<date>.md first
daily/<date>/**/*.md sorted by path
```
但 auto_dream 会排除:
```text
daily/<date>/interests.md
```
也就是 `topic_session_id` 对应的 daily topics 文件。原因是 `interests.md` 是 auto_dream 自己产出的兴趣主题,不能再作为 dream 输入,否则容易自我循环。
### 2.4 用 file_catalog 做增量判断
auto_dream 构造两张表:
| 名称 | 来源 | 内容 |
|---|---|---|
| `existing` | 当前磁盘 | `{vault_relative_path: st_mtime}` |
| `indexed` | `file_catalog.get_nodes()` | `{vault_relative_path: st_mtime}` |
`indexed` 只保留当天范围:
```text
daily/<date>.md
daily/<date>/*
```
并且同样排除:
```text
daily/<date>/interests.md
```
然后做 diff:
| 条件 | 分类 | 行为 |
|---|---|---|
| `rel in existing`,但 catalog 没有 | added | 需要 dream |
| `rel in existing`,但 mtime 不同 | modified | 需要 dream |
| `rel in existing`,且 mtime 相同 | unchanged | 跳过 |
| catalog 有,但磁盘没有 | deleted | 从 catalog 删除 |
注意这里的 `file_catalog` 更像是 auto_dream 的“已处理 mtime 水位线”,不是语义索引本身。默认配置没有给 `auto_dream_step` 显式传 `file_catalog`,所以按 `BaseStep.Ref` 规则解析到 `file_catalog.default`。
### 2.5 先删除 catalog 中的缺失文件
如果某些当天文件已经不存在:
```python
await self.file_catalog.delete(to_delete)
```
这一步不需要 LLM,也不会阻塞后续 dream。删除失败只记录日志,当前实现不会把它计入 `result.files_failed`。
### 2.6 对新增/修改文件逐个 dispatch dream_extract
对每个 `to_dream` 文件,auto_dream 调:
```python
resp = await self.run_job("dream_extract", path=rel_path, hint=hint)
```
`dream_extract` 只有一个 step:
```yaml
steps:
- backend: dream_step
```
`AutoDreamStep._dispatch_dream()` 会把 `resp.metadata` 重新校验成 `DreamResult`。如果 job 抛异常、metadata 不是 `DreamResult`、或 response `success=False`,都会转成带 `error` 的 `DreamResult`。
### 2.7 单文件 DreamStep 内部逻辑
`DreamStep.dream_one(path, hint)` 是真正的 per-file create_or_update。
它的主流程:
```text
1. path 为空 -> skipped
2. 没有 LLM -> error
3. _pack_material() 读取 vault-relative 文件内容
4. Phase 1: _extract()
5. 如果 Phase 1 没有 units -> skipped,但保留 topic_candidates
6. Phase 2: 对每个 unit 调 _integrate_unit()
7. 返回 DreamResult
```
#### Phase 1: extract
工具:
```python
_EXTRACT_TOOLS = ("read",)
```
输出 schema 是 `ExtractedUnits`:
```text
units: list[MemoryUnit]
topic_candidates: list[TopicCandidate]
```
每个 memory unit 包含:
| 字段 | 语义 |
|---|---|
| `name` | agent 内部短名 |
| `bucket` | `procedure` / `personal` / `wiki` 三选一 |
| `summary` | 这个抽象是什么,证据在哪 |
每个 topic candidate 包含:
| 字段 | 语义 |
|---|---|
| `title` | 兴趣主题标题 |
| `reason` | 为什么用户可能关心 |
| `evidence` | 证据指针 |
| `keywords` | 去重关键词 |
如果 LLM 输出了未知 bucket,当前代码会警告并改成 `wiki`。
#### Phase 2: integrate
每个 unit 单独发起一次 ReAct:
```text
system prompt = integrate_system_prompt_<bucket>
```
工具:
```python
_INTEGRATE_TOOLS = (
"node_search",
"read",
"frontmatter_read",
"write",
"edit",
"frontmatter_update",
)
```
输出 schema 是 `IntegrateOutcome`:
| 字段 | 语义 |
|---|---|
| `action` | `CREATE` / `CORROBORATE` / `REFINE` / `CORRECT` |
| `target_path` | 实际写入或更新的 digest path |
| `note` | 简短说明 |
`DreamStep` 根据 action 统计:
```text
CREATE -> nodes_created
其他 action -> nodes_updated
```
当前实现里,某个 unit 的 integrate 失败不会让整个 DreamResult 变成 error,只会在 summary 里记录 `FAILED`。这意味着文件级别仍会被 auto_dream 当作成功并写入 catalog mtime。
### 2.8 汇总 per-file 结果并更新 catalog
auto_dream 对每个文件的 `DreamResult` 做三件事:
| 情况 | 行为 |
|---|---|
| `dr.error` 非空 | `files_failed += 1`,不更新这个文件的 catalog mtime,下次会重试 |
| `dr.skipped` 为 true | `files_skipped += 1`,仍然 upsert mtime,避免每次重复跑 Phase 1 |
| 正常 dream | `files_dreamed += 1`,upsert mtime |
同时收集:
```python
topic_candidates.extend(dr.topic_candidates or [])
```
最后批量:
```python
await self.file_catalog.upsert(upsert_nodes)
```
### 2.9 聚合写 daily topics
如果:
```text
emit_topics == true
topic_candidates 非空
```
auto_dream 会调用:
```python
await self.run_job(
"daily_topics",
date=today,
candidates=topic_candidates,
topic_count=3,
diversity_days=7,
session_id="interests",
)
```
`DailyTopicsStep` 做:
```text
1. 清洗 candidates
2. 读取过去 diversity_days 天的 interests.md
3. 有 LLM 时用 select prompt 选最终 topics
4. 没有 LLM 时 fallback: 简单标题去重
5. 写 daily/<date>/interests.md
6. refresh_day_index()
```
写出的文件形态:
```text
daily/<date>/interests.md
```
frontmatter 包含:
```yaml
name: interests
description: "<n> interest topic(s) inferred for <date>."
date: <date>
topic_count: 3
diversity_days: 7
```
body 是 `# Interested Topics` 加编号列表。
auto_dream 收到 daily_topics 成功响应后,还会把这些文件的最新 mtime 写入 catalog:
```text
daily/<date>/interests.md
daily/<date>.md
```
这里有一个隐含行为:day-index 在 per-file dream 之后又因为 `interests.md` 被写入而刷新,auto_dream 会把刷新后的 `daily/<date>.md` mtime 标记为已处理。也就是说,仅由 interests 写入引发的 day-index 变化不会在下一轮再次触发 dream。
### 2.10 持久化与响应
如果:
```text
persist == true
并且有 upsert 或 delete
```
则:
```python
await self.file_catalog.dump()
```
最终 response:
```text
success = files_failed == 0 and not topics_error
answer = AutoDreamResult.summary
metadata = AutoDreamResult.model_dump()
```
summary 格式大致是:
```text
[AutoDreamStep] date=2026-06-18 scanned=... unchanged=... dreamed=... skipped=... failed=... deleted=...
- daily/2026-06-18.md: OK (+1 created, ~2 updated)
- daily/2026-06-18/session.md: SKIP
- topics: OK (3 written to daily/2026-06-18/interests.md)
```
## 3. 当前逻辑的边界和风险
### 3.1 AutoDreamStep 职责过重
`AutoDreamStep` 同时负责:
- 日期解析
- day-index 刷新
- 文件扫描
- catalog diff
- 删除 catalog
- per-file job dispatch
- DreamResult 校验
- topic candidates 汇总
- daily_topics job dispatch
- topic 输出后的 catalog upsert
- catalog dump
- summary 渲染
这些职责可以拆成明确 step,提高可测试性和可替换性。
### 3.2 file_catalog 的语义不够显式
这里的 catalog 不是“今天有哪些文件”的普通目录索引,而是“哪些文件已经被 auto_dream 处理到某个 mtime”。建议在拆分后把它显式命名为 dream catalog / processed catalog,至少在 step 名和文档中说清楚。
### 3.3 integrate unit 失败不会触发文件重试
`DreamStep` 当前捕获单个 unit integrate 异常,写进 summary 后继续,但不设置 `DreamResult.error`。auto_dream 因此会把这个文件 mtime upsert,下次不会自动重试失败 unit。
这可能是有意的“尽量前进”,但如果要做严格一致性,应改成:
```text
任一 unit integrate 失败 -> DreamResult.error 非空 -> auto_dream 不更新 mtime
```
### 3.4 topic 写入导致的 day-index 变化被标记为已处理
auto_dream 写 `interests.md` 后刷新 day-index,并把 day-index 最新 mtime upsert 到 catalog。这样可以避免自生成内容触发循环,但也意味着 `daily/<date>.md` 中新增的 `interests.md` 链接不会被 dream。
这通常是合理的,因为 `interests.md` 本身被排除在 dream 输入之外。
### 3.5 删除 catalog 失败不影响 success
删除 catalog entry 失败只打日志,不会让 response failure。拆分后可以明确这个策略:
- catalog delete 是 best-effort,不影响 dream 主流程
- 或者 catalog delete 失败应导致整个 job failure
## 4. 拆分目标
重新拆分时,不按“每个小动作一个 step”拆,而按执行边界拆:
```text
非 LLM 准备/扫描/diff
-> LLM: per-file dream
-> LLM: daily topics
-> 非 LLM response 汇总
```
核心原则:
- 使用 LLM 的阶段单独成 step,便于限流、重试、观测和替换模型。
- 不使用 LLM 的准备、扫描、diff 可以合并,避免 step 过碎。
- catalog 只是 auto_dream 的内部进度水位线,不提升为独立阶段。
- prompt 重新写,但可以复用旧 prompt 的核心内容和约束。
- 不做旧接口/旧格式兼容;按新 4-step pipeline 重新定义最干净的输入输出。
## 5. 新方案:拆成 4 个 Step
新方案不再是“逐文件 extract + 逐文件 integrate”。核心变化是:
```text
本轮 changed files
-> 1 个 agent 一次性阅读所有 changed paths
-> 输出全局去重/合并后的 unit list
-> Python for 循环逐 unit integrate
```
这样一个抽象可能来自多个文件,Phase 1 就能合并为同一个 unit,避免同一天多个 session note 反复提出同一概念。
目标代码位置:
```text
reme4/steps/evolve/dream/
__init__.py
models.py
plan.py
extract.py
integrate.py
topics.py
finish.py
prompts.yaml 或 dream.yaml
```
重构完成后删除旧文件,不保留兼容 alias:
```text
reme4/steps/evolve/dream.py
reme4/steps/evolve/auto_dream.py
reme4/steps/evolve/daily_topics.py
```
### 5.0 Step 输入输出总表
| Step | 是否 LLM | 核心能力 | 输入 | 输出 / 写入 context | 副作用 |
|---|---:|---|---|---|---|
| `dream_extract_step` | 是 | 根据 dream catalog 找出本轮新增/修改/删除文件,把所有 changed paths 交给一个 agent,一次性输出跨文件合并后的 `unit_list` 和 `topic_list` | `context.date`; `context.hint`; `app_config.daily_dir`; `app_config.timezone`; `file_store.vault_path`; `file_catalog.dream`; step 参数 `topic_session_id=interests` | `dream.date`; `dream.hint`; `dream.daily_dir`; `dream.vault`; `dream.existing`; `dream.indexed`; `dream.changed_paths`; `dream.deleted_paths`; `dream.units`; `dream.topics`; `dream.extract_summary`; `dream.result.files_scanned/files_changed/files_deleted`; `dream.errors` | 刷新 `daily/<date>.md`; 删除 catalog 中缺失文件 entry; 读取所有 changed files; 本 step 不写 digest |
| `dream_integrate_step` | 是 | `for unit in units` 逐个执行原 Phase 2 integrate 逻辑,保持 node_search/read/write/edit/frontmatter_update 工具和 bucket prompt 不变 | `dream.units`; `dream.hint`; `dream.vault`; `app_config.digest_dir`; agent tools: `node_search/read/frontmatter_read/write/edit/frontmatter_update` | `dream.integrate_results`; `dream.nodes_created`; `dream.nodes_updated`; `dream.result.units_integrated/units_failed`; `dream.errors` | 写/更新 `digest/<bucket>/*.md`; 不更新 dream catalog |
| `dream_topics_step` | 是 | 根据 `topic_list` 更新 `daily/<date>/interests.yaml`; 读取当天已有 topics 和最近 N 天 topics 做去重 | `dream.date`; `dream.daily_dir`; `dream.topics`; `file_store.vault_path`; step 参数 `topic_count`; `topic_diversity_days`; `topic_session_id=interests` | `dream.topics_path=daily/<date>/interests.yaml`; `dream.topics_written`; `dream.topics_merged`; `dream.topics_skipped_duplicates`; `dream.errors` | 新建或更新 `daily/<date>/interests.yaml`; 刷新 `daily/<date>.md` |
| `dream_finish_step` | 否 | 统一收口:按 path checkpoint 成功处理的文件,持久化 catalog,渲染 summary 和 response metadata | `dream.changed_paths`; `dream.deleted_paths`; `dream.failed_paths`; `dream.integrate_results`; `dream.topics_path`; `dream.errors`; `dream.result`; `file_catalog.dream`; step 参数 `persist=true` | `context.response.success`; `context.response.answer`; `context.response.metadata` | upsert successful paths 的 mtime 到 `file_catalog.dream`; upsert `interests.yaml` 和 day-index mtime; `file_catalog.dump()` |
### Step 1: `dream_extract_step`
这是新的全局 Phase 1。它合并了当前 `auto_dream_step` 的扫描/diff 和当前 `DreamStep._extract()` 的抽取能力。
职责:
- 解析 `date` / `hint`。
- 刷新 day-index: `daily/<date>.md`。
- 扫描输入文件并统一交给 extract agent:
- `daily/<date>.md`
- `daily/<date>/<session_id>.md`
- `daily/<date>/<resource_stem>.md`
- 以及 `daily/<date>/**/*.md` 下其它当天 note
- 排除自生成文件:
- `daily/<date>/interests.yaml`
- 读取 `file_catalog.dream`,按 mtime diff 出:
- `changed_paths`
- `unchanged_paths`
- `deleted_paths`
- 删除 catalog 中 `deleted_paths`。
- 打包所有 `changed_paths` 的文件内容。
- 调用一次 extract agent,让它看见所有 changed paths。
- 输出跨文件合并后的 `unit_list` 和 `topic_list`。
新的 unit schema:
```python
class DreamUnit(BaseModel):
name: str
bucket: Literal["procedure", "personal", "wiki"]
summary: str
paths: list[str]
```
`paths` 是这个 unit 的证据来源列表。多个文件讲的是同一抽象时,Phase 1 必须合并成一个 unit:
```json
{
"name": "jwt-session-expiry-policy",
"bucket": "procedure",
"summary": "How the project decides session expiry from compliance and product constraints.",
"paths": [
"daily/2026-06-18/auth.md",
"daily/2026-06-18/api-review.md"
]
}
```
topic schema 可以沿用当前 `TopicCandidate`,但建议把来源改成 `paths`:
```python
class DreamTopicCandidate(BaseModel):
title: str
reason: str
evidence: str
keywords: list[str] = []
paths: list[str] = []
```
输出到 context:
```python
{
"dream": {
"date": "YYYY-MM-DD",
"changed_paths": [
{"path": "daily/YYYY-MM-DD/a.md", "mtime": 1710000000.0}
],
"deleted_paths": [],
"units": [
{
"name": "jwt-session-expiry-policy",
"bucket": "procedure",
"summary": "...",
"paths": ["daily/YYYY-MM-DD/a.md", "daily/YYYY-MM-DD/b.md"]
}
],
"topics": [
{
"title": "...",
"reason": "...",
"evidence": "...",
"keywords": ["..."],
"paths": ["daily/YYYY-MM-DD/a.md"]
}
]
}
}
```
LLM 调用数量:
```text
1 个 agent 任务
```
注意:
- 这个 step 不再为每个文件分别调用 `dream_extract`。
- 如果 `changed_paths` 为空,它不调用 LLM,直接输出空 `units/topics`。
- 只有 `dream_finish_step` 才把 changed file mtime 标为已处理。这样 integrate/topics 失败时不会误跳过。
### Step 2: `dream_integrate_step`
这是新的全局 Phase 2。它对 Step 1 输出的 `units` 做 Python for 循环,每个 unit 的 integrate 逻辑保持当前 `DreamStep._integrate_unit()` 不变。
职责:
- 遍历 `dream.units`。
- 每个 unit 根据 `unit.bucket` 选择:
- `integrate_system_prompt_procedure`
- `integrate_system_prompt_personal`
- `integrate_system_prompt_wiki`
- material 不再是单文件 blob,而是这个 unit 对应 `paths` 的证据包。
- 调用当前相同工具:
- `node_search`
- `read`
- `frontmatter_read`
- `write`
- `edit`
- `frontmatter_update`
- 输出 `IntegrateOutcome`。
`integrate_user_message` 需要从单 `material_blob` 改成多路径 evidence blob:
```text
unit_name: ...
unit_bucket: ...
unit_summary: ...
source_paths:
- daily/...
- daily/...
# Evidence materials
### daily/.../a.md
...
### daily/.../b.md
...
```
输出:
```python
{
"dream": {
"integrate_results": [
{
"unit_name": "jwt-session-expiry-policy",
"bucket": "procedure",
"action": "CREATE",
"target_path": "digest/procedure/jwt-session-expiry-policy.md",
"source_paths": ["daily/YYYY-MM-DD/a.md", "daily/YYYY-MM-DD/b.md"],
"note": "..."
}
],
"nodes_created": ["digest/procedure/jwt-session-expiry-policy.md"],
"nodes_updated": []
}
}
```
LLM 调用数量:
```text
N 个 agent 任务
N = len(dream.units)
```
失败策略建议:
- 任一 unit integrate 失败,记录到 `dream.errors`。
- 因为每个 unit 都有明确的 `paths`,失败 unit 对应的 paths 进入 `dream.failed_paths`。
- `dream_finish_step` 不 checkpoint `failed_paths`。
- 不在任何失败 unit `paths` 里的 changed paths 可以 checkpoint。
- 如果同一个 path 同时出现在成功 unit 和失败 unit 中,以失败为准,该 path 不 checkpoint。
### Step 3: `dream_topics_step`
这个 step 取代当前 `daily_topics_step`。目标文件固定为:
```text
daily/<date>/interests.yaml
```
职责:
- 读取 `dream.topics`。
- 如果 `daily/<date>/interests.yaml` 已存在,读取旧 topics。
- 读取最近 `topic_diversity_days` 天的 `daily/<previous-date>/interests.yaml` 作为历史去重上下文。
- 合并当天旧 topics + 新 topics。
- 去重:
- 标题 normalize 后相同视为重复。
- keywords 高重叠视为可能重复。
- evidence/paths 完全相同视为重复。
- 与最近 N 天历史 topics 重复时跳过。
- 可选使用 LLM 对候选 topic 做最终选择和改写。
- 写回 YAML。
- 刷新 day-index。
建议 YAML 格式:
```yaml
date: "2026-06-18"
updated_at: "2026-06-18T22:00:00+08:00"
topic_count: 3
diversity_days: 7
topics:
- title: "JWT session expiry policy"
reason: "The user repeatedly worked through compliance-driven auth expiry tradeoffs."
evidence: "Mentioned in auth review and API notes."
keywords: ["auth", "jwt", "session", "compliance"]
paths:
- "daily/2026-06-18/auth.md"
- "daily/2026-06-18/api-review.md"
```
输入:
```text
dream.date
dream.daily_dir
dream.topics
topic_count
topic_diversity_days
topic_session_id
```
输出:
```python
{
"dream": {
"topics_path": "daily/YYYY-MM-DD/interests.yaml",
"topics_written": 3,
"topics_merged": 5,
"topics_skipped_duplicates": 2
}
}
```
LLM 调用数量:
```text
0 或 1 个 agent 任务
```
建议:
- 如果只是 append/去重,不必 LLM。
- 如果需要从很多 candidates 中挑 `topic_count` 个,才调用 LLM。
- 不读取也不写 `interests.md`;全新格式只认 `interests.yaml`。
### Step 4: `dream_finish_step`
这是非 LLM 收尾 step。
职责:
- 根据前面步骤结果决定 success。
- 计算 `failed_paths`:
- 每个失败 unit 的 `unit.paths` 都进入 failed set。
- 如果某个 path 同时属于成功 unit 和失败 unit,以失败为准。
- 计算 `checkpoint_paths`:
- `changed_paths - failed_paths`
- extract 成功但没有任何 unit/topics 的 changed paths 也可以 checkpoint,避免重复空跑。
- 把 `checkpoint_paths` 的当前 mtime upsert 到 `file_catalog.dream`。
- 把 `daily/<date>/interests.yaml` 的 mtime upsert 到 `file_catalog.dream`。
- 把刷新后的 `daily/<date>.md` 的 mtime upsert 到 `file_catalog.dream`。
- `deleted_paths` 的 catalog 删除在 extract step 已完成,finish 只负责 dump。
- `file_catalog.dump()`。
- 渲染 summary。
- 写 `context.response.metadata`。
输出 metadata 建议:
```python
{
"date": "YYYY-MM-DD",
"files_scanned": 10,
"files_changed": 3,
"files_deleted": 1,
"paths_checkpointed": ["daily/2026-06-18/a.md"],
"paths_failed": ["daily/2026-06-18/b.md"],
"units_extracted": 4,
"units_integrated": 4,
"units_failed": 0,
"topics_written": 3,
"nodes_created": [...],
"nodes_updated": [...],
"errors": []
}
```
## 6. 拆分后的 YAML 形态
建议把 `auto_dream` 改成新的 dream pipeline:
```yaml
auto_dream:
backend: base
description: "Auto-dream: scan daily changes, extract cross-file units, integrate digest nodes, update daily interests."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
default: ""
hint:
type: string
description: "caller guidance passed through to the dreamer LLM"
default: ""
steps:
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
topic_diversity_days: 7
topic_session_id: interests
- backend: dream_finish_step
file_catalog: dream
persist: true
```
旧 job 删除,不做兼容 wrapper:
```yaml
dream:
# 删除
dream_extract:
# 删除
daily_topics:
# 删除
```
## 7. 数据结构建议
建议所有跨 step 状态都放在 `context["dream"]`。
核心模型:
```python
class DreamUnit(BaseModel):
name: str
bucket: Literal["procedure", "personal", "wiki"]
summary: str
paths: list[str] = Field(default_factory=list)
class DreamTopic(BaseModel):
title: str
reason: str
evidence: str = ""
keywords: list[str] = Field(default_factory=list)
paths: list[str] = Field(default_factory=list)
class DreamState(BaseModel):
date: str = ""
hint: str = ""
daily_dir: str = "daily"
vault: str = ""
changed_paths: list[dict] = Field(default_factory=list)
unchanged_paths: list[str] = Field(default_factory=list)
deleted_paths: list[str] = Field(default_factory=list)
failed_paths: list[str] = Field(default_factory=list)
checkpoint_paths: list[str] = Field(default_factory=list)
units: list[DreamUnit] = Field(default_factory=list)
topics: list[DreamTopic] = Field(default_factory=list)
integrate_results: list[dict] = Field(default_factory=list)
nodes_created: list[str] = Field(default_factory=list)
nodes_updated: list[str] = Field(default_factory=list)
topics_path: str = ""
topics_written: int = 0
errors: list[str] = Field(default_factory=list)
result: dict = Field(default_factory=dict)
```
## 8. 实现任务
这是一次 breaking rewrite,不存在旧接口/旧格式迁移任务。剩余工作就是按新设计实现 4 个 step。
必须实现:
1. 在 `reme4/steps/evolve/dream/` 下新增 `models.py`、helper 和 4 个 step 文件。
2. 重写 prompts:
- `extract_system_prompt`
- `extract_user_message`
- `integrate_system_prompt_procedure`
- `integrate_system_prompt_personal`
- `integrate_system_prompt_wiki`
- `integrate_user_message`
- 可参考旧 prompt 的内容,但不保持旧 prompt 接口。
3. 实现 `dream_extract_step`:
- 扫描 `daily/<date>.md` 与 `daily/<date>/**/*.md`。
- 排除 `daily/<date>/interests.yaml`。
- 根据 `file_catalog.dream` 计算 changed/unchanged/deleted。
- 一个 agent 统一读取所有 changed paths,输出全局 units/topics。
- 清洗 units: unknown bucket fallback 到 `wiki`; paths 去重; paths 必须来自 changed paths。
4. 实现 `dream_integrate_step`:
- 按 `unit.paths` 打包 evidence。
- for 循环逐 unit integrate。
- 保留工具集合和 action 语义。
- unit 失败时记录 `failed_paths += unit.paths`。
5. 实现 `dream_topics_step`:
- 只读写 `daily/<date>/interests.yaml`。
- 读取当天已有 YAML topics。
- 读取最近 `topic_diversity_days` 天的 `interests.yaml` 做历史去重。
- 写回去重后的 YAML。
- 刷新 day-index。
6. 实现 `dream_finish_step`:
- `checkpoint_paths = changed_paths - failed_paths`。
- checkpoint 成功 paths 的 mtime。
- checkpoint `interests.yaml` 和 `daily/<date>.md`。
- dump `file_catalog.dream`。
- 输出全新 response metadata。
7. 更新 `default.yaml`:
- `auto_dream` 改为 4-step pipeline。
- 删除 `dream`、`dream_extract`、`daily_topics` job。
- 保留 `file_catalog.dream`。
8. 删除旧代码:
- `reme4/steps/evolve/auto_dream.py`
- `reme4/steps/evolve/dream.py`
- `reme4/steps/evolve/daily_topics.py`
- 更新 `reme4/steps/evolve/__init__.py`。
现在没有保留的迁移项:
- 不兼容 `interests.md`。
- 不保留单文件 `dream path=...`。
- 不保留 `dream_extract` job。
- 不保留 `daily_topics` job。
- 不要求 response metadata 兼容 `AutoDreamResult`。
- 不要求 prompt 入参兼容旧 `dream.yaml`。
## 9. 推荐测试用例
最低测试集:
| 场景 | 期望 |
|---|---|
| 当天没有任何文件 | scanned=0,success=true,no topics |
| 只有 day-index 新增 | `dream_extract_step` 调用一次全局 extract,finish checkpoint day-index |
| session note 新增 | extract evidence 中 day-index first,session notes sorted |
| session note mtime 未变 | unchanged+1,不进入 changed evidence |
| session note 删除 | catalog delete |
| `interests.yaml` 存在 | 不进入 scan/diff/changed evidence |
| 多个文件产出同一抽象 | extract 输出 1 个 unit,`paths` 包含多个 source path |
| extract 输出空 units/topics | finish 仍 checkpoint changed files,避免重复空跑 |
| 某个 unit integrate 失败 | 该 unit 的 `paths` 不 checkpoint,response failure |
| 同一 path 同时属于成功和失败 unit | 失败优先,该 path 不 checkpoint |
| 其它 path 的 units 都成功 | 这些 path 可以 checkpoint |
| 有 topic candidates | 写/更新 `daily/<date>/interests.yaml`,记录 topics_path/topics_written |
| 已存在 `interests.yaml` | 合并新旧 topics,不重复 |
| 最近 N 天已有相同 topic | 当前日 topics 去重跳过 |
| extract 输出 unknown bucket | 清洗后 bucket=`wiki` |
| prompt 输出 path 不在 changed paths | 该 unit 被丢弃或修正,不能 checkpoint 不明来源 |

30
docs4/reme_design.md
View file

@ -11,20 +11,24 @@
### 1.1 目录结构
```
- resource/【原始素材】 # 外部渠道摄入 / 手动放入
- YYYY-MM-DD/ # 按日期归档
- session_{id}.jsonl # 对话原始记录
- {channel}_{xxxx}.html # 网页抓取、邮件等
- {channel}_{xxxx}.md # Markdown 资料
- daily/【日记，浅加工】 # auto-memory 自动写入
- YYYY-MM-DD.md # 当天索引页，汇总所有事件
- reme_session/
- agentscope|claude_code / # 使用内置的agent wrapper，session会保存在这里
{session_id}.jsonl UUID格式要求 # /Users/yuli/workspace/ReMe/reme4/components/agent_wrapper
- dialog/
{session_id}.jsonl # auto memory保存 可以监控可以被检索【可选】
- resource/
- YYYY-MM-DD/
- session_{id}.md # 按 session 拆分的日志
- resource_{id}.md # 对素材的加工笔记
- digest/【深加工】 # auto-dream 持续打磨
- personal/ # 用户偏好、习惯、身份
- procedure/ # 方法论、步骤、工作流
- wiki/ # 通用知识、决策先例
- {channel}_{xxxx}.html
- {channel}_{xxxx}.md
- daily/【日记，浅加工】
- YYYY-MM-DD.md
- YYYY-MM-DD/
- session_{session_id}.md
- {resource_stem}.md
- digest/
- personal/
- procedure/
- wiki/
```
### 1.2 分层详解

38
docs4/todo.md Normal file
View file

@ -0,0 +1,38 @@
- reme_session/
- agentscope|claude_code / # 使用内置的agent wrapper，session会保存在这里
{session_id}.jsonl UUID格式要求 # /Users/yuli/workspace/ReMe/reme4/components/agent_wrapper
- dialog/
{session_id}.jsonl # auto memory保存 可以监控可以被检索【可选】
- resource/
- YYYY-MM-DD/
- {channel}_{xxxx}.html
- {channel}_{xxxx}.md
- daily/【日记，浅加工】
- YYYY-MM-DD.md
- YYYY-MM-DD/
- {session_id}.md
- {和resource同名}.md
- digest/
- personal/
- procedure/
- wiki/
函数接口：
- auto_memory
- message 应该会 会保存到 reme_session/dialog/{session_id}.jsonl
- 通过 message 更新 daily/YYYY-MM-DD/{session_id}.md
- auto-resource
- 会保存到 daily/YYYY-MM-DD/{resource_stem}.md
- auto-dream
- 读取所有的md
- 会生成link auto-link
- 会生成topic ？
- proactive
- 会读取topic ？
- search
后台任务：
- index_update_loop 索引监控
- resource_watch_loop 资源监控
- digest_watch_loop 应该是闲置？

275
docs4/watch_loop_step_refactor_plan.md Normal file
View file

@ -0,0 +1,275 @@
# Watch Loop Step 重构计划
## 背景
当前 `index_update_loop`、`resource_watch_loop`、`digest_watch_loop` 都是同一种范式：
1. 启动时扫描已有文件变化。
2. 用一组 step 处理扫描出来的变化。
3. 进入持续监听。
4. 持续监听到变化后，再用同一组 step 处理变化。
也就是说，初始化扫描和持续监听只是变化来源不同，后续更新逻辑应该共享。
现在的问题是这个范式没有被显式建模：
- `index_update_loop` 初始化和监听都走 `update_index_step`，基本一致。
- `resource_watch_loop` 初始化和监听都走 `update_catalog_step + foreach_dispatch_step`，基本一致。
- `digest_watch_loop` 初始化走 `update_catalog_step`，但监听阶段只走 `log_changes_step`，导致 live changes 不更新 `file_catalog`。
- `watch_changes_step` 同时负责监听和 dispatch 下游逻辑，职责偏重。
- `update_index_step` 和 `update_catalog_step` 内部有较多重复的变化分桶、结果收集、删除、持久化逻辑。
## 目标
重构后希望形成统一模型：
```text
change producer:
init_changes_step # 初始化，一次性产生 changes 并 dispatch
watch_changes_step # 持续监听，持续产生 changes
change handlers:
update_index_step
update_catalog_step
foreach_dispatch_step
log_changes_step
channel_notify_step # 后续可选
```
核心原则：
- producer 只负责产生 `context["changes"]`。
- handler 只负责消费 `context["changes"]`。
- 初始化和持续监听都通过 `BaseStep.dispatch_steps(...)` 调用 handler。
- `init_changes_step` 和 `watch_changes_step` 显式配置同一组 `dispatch_steps`，让范式直接可见。
## 配置设计
不新增 job 级 `change_steps`。每个 producer step 自己声明 `dispatch_steps`，初始化 producer 和持续监听 producer 配同一组 handler。
配置精简约定：
- `init_changes_step.recursive` 和 `watch_changes_step.recursive` 默认就是 `true`，配置中不再显式写。
- `update_index_step` / `update_catalog_step` 的 `persist` 语义统一为默认 `true`，配置中不再显式写。
- 只有当某个 loop 需要关闭递归或关闭持久化时，才显式写 `recursive: false` / `persist: false`。
### index_update_loop
```yaml
index_update_loop:
backend: background
watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_suffixes: [md, jsonl]
steps:
- backend: init_changes_step
store: file_store
dispatch_steps: [update_index_step]
- backend: watch_changes_step
dispatch_steps: [update_index_step]
```
### resource_watch_loop
```yaml
resource_watch_loop:
backend: background
watch_dirs: [resource_dir]
watch_suffixes: [md, txt, json, jsonl, csv, yaml, html]
dispatch_job: auto_resource
steps:
- backend: init_changes_step
store: file_catalog
dispatch_steps: [update_catalog_step, foreach_dispatch_step]
- backend: watch_changes_step
dispatch_steps: [update_catalog_step, foreach_dispatch_step]
```
### digest_watch_loop
```yaml
digest_watch_loop:
backend: background
watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [md]
steps:
- backend: init_changes_step
store: file_catalog
dispatch_steps: [update_catalog_step, log_changes_step]
- backend: watch_changes_step
dispatch_steps: [update_catalog_step, log_changes_step]
```
这样三个 loop 都统一成：
```text
startup:
scan changes
dispatch handlers
runtime:
watch changes
dispatch same handlers
```
## Step 拆分
### 1. BaseStep dispatch 能力
把 dispatch 能力沉到 `BaseStep`，所有 producer step 共享：
- `normalize_dispatch_steps(dispatch_step, dispatch_steps)`
- `dispatch_steps(dispatch_steps, **kwargs)`
这样 `init_changes_step` 和 `watch_changes_step` 都不需要各自实现 registry 查询、step 实例化和 context 透传。
`dispatch_steps` 支持两种形式：
```yaml
dispatch_steps: [update_catalog_step, log_changes_step]
```
也支持给单个 handler 传参数：
```yaml
dispatch_steps:
- backend: update_catalog_step
- backend: some_step
option: value
```
### 2. init_changes_step
新增 `init_changes_step`，替代现有两个初始化扫描 step：
- `scan_store_changes_step`
- `scan_catalog_changes_step`
参数：
```yaml
store: file_catalog | file_store
dispatch_steps: [...]
```
职责：
- 根据 `watch_dirs` / `watch_suffixes` 收集磁盘文件。
- 根据 `store` 和目标状态源比较。
- 生成统一格式的 `context["changes"]`。
- 如果有变化，调用 `BaseStep.dispatch_steps(...)` 执行 handler。
输出格式：
```python
[
{"change": "added", "path": "/abs/path/to/file.md"},
{"change": "modified", "path": "/abs/path/to/file.md"},
{"change": "deleted", "path": "/abs/path/to/file.md"},
]
```
### 3. watch_changes_step
保留监听职责，弱化业务 dispatch 职责。
职责：
- 根据 `watch_dirs` / `watch_suffixes` 建立文件监听。
- 对每个 debounced batch 生成同样格式的 `changes`。
- 调用 `BaseStep.dispatch_steps(...)` 执行 handler。
### 4. update_index_step / update_catalog_step
第二阶段再精简。
它们现在重复逻辑包括：
- 解析 `added` / `modified` / `deleted`。
- 判断文件是否存在。
- 收集 per-path result。
- 删除旧记录。
- upsert 新记录。
- persist。
- 写 response。
建议抽内部基类，例如：
```python
class ChangeApplyStep(BaseStep):
async def parse_added_or_modified(self, path): ...
async def upsert_items(self, items): ...
async def delete_paths(self, rel_paths): ...
async def dump_target(self): ...
```
然后：
- `UpdateCatalogStep` 只实现 `stat -> FileNode`，写 `file_catalog`。
- `UpdateIndexStep` 只实现 `chunk_file -> FileNode + chunks`，写 `file_store`。
这一步可以在 `init_changes_step` 落地后做，降低一次性改动风险。
## 实施顺序
当前落地状态：
- Phase 1 已完成：`BaseStep.dispatch_steps(...)`、`init_changes_step`、默认持久化、默认配置精简已落地。
- Phase 2 已完成：`digest_watch_loop` 的初始化和监听都执行 `update_catalog_step + log_changes_step`。
- 兼容旧 backend 不保留：`scan_store_changes_step` / `scan_catalog_changes_step` 已删除。
- `reindex` 已改为 `clear_store_step + init_changes_step(store=file_store, dispatch_steps=[update_index_step])`。
- Phase 3 已完成一层：`update_catalog_step` 和 `update_index_step` 已合并到 `update_changes.py`，
并抽出 `ChangeApplyStep` 复用 added/modified/deleted、upsert/delete/persist 模板逻辑。
### Phase 1：统一范式
1. 把 dispatch 能力沉到 `BaseStep`。
2. 新增 `init_changes_step`。
3. 将 `update_index_step` 和 `update_catalog_step` 的 `persist` 默认值统一为 `true`。
4. 修改 `default.yaml` 里的三个 loop：
- 初始化阶段统一使用 `init_changes_step`。
- `init_changes_step` 和 `watch_changes_step` 配置相同的 `dispatch_steps`。
5. `watch_changes_step` 保留 `dispatch_step` 到 `dispatch_steps` 的轻量兼容。
6. 验证三个 loop 的启动扫描和 live watch 都会执行同一条 handler pipeline。
### Phase 2：修正 digest_watch_loop 语义
`digest_watch_loop` 的 live changes 应该更新 `file_catalog`，因此初始化和监听阶段都应配置同一组 `dispatch_steps`：
```yaml
dispatch_steps: [update_catalog_step, log_changes_step]
```
如果后续要通知 channel，可以追加：
```yaml
- backend: channel_notify_step
```
### Phase 3：精简 update 类 step
1. 抽 `ChangeApplyStep` 基类或 helper 函数。
2. 让 `update_index_step` 和 `update_catalog_step` 只保留各自差异逻辑。
3. 保持外部行为不变：
- 输入仍然是 `context["changes"]`。
- 输出仍然写 `response.answer` 和 `response.success`。
- `persist` 语义不变。
## 验证点
最少需要覆盖这些场景：
- `index_update_loop` 启动扫描新增文件，会更新 `file_store`。
- `index_update_loop` live 新增/修改/删除文件，会更新 `file_store`。
- `resource_watch_loop` 启动扫描新增文件，会更新 `file_catalog` 并触发 `auto_resource`。
- `resource_watch_loop` live 新增文件，会更新 `file_catalog` 并触发 `auto_resource`。
- `digest_watch_loop` 启动扫描新增/修改/删除文件，会更新 `file_catalog`。
- `digest_watch_loop` live 新增/修改/删除文件，也会更新 `file_catalog`。
- `changes` 为空时，`init_changes_step` 不应执行 handler，也不应报错。
## 预期收益
- 三个 background loop 的结构统一。
- 初始化扫描和持续监听的处理逻辑完全复用。
- `digest_watch_loop` 不再出现启动和 live 语义不一致。
- `watch_changes_step` 和 `init_changes_step` 共享 `BaseStep` dispatch 能力。
- 后续新增日志、channel notification、auto dream 等 handler 时，初始化和监听两处使用同一组 `dispatch_steps`。

12
reme/core/application.py
View file

@ -514,17 +514,17 @@ class Application:
logger.warning("Application is not started")
return True
for name, vector_store in self.service_context.vector_stores.items():
logger.info(f"Closing vector store: {name}")
await vector_store.close()
for name, file_watcher in self.service_context.file_watchers.items():
logger.info(f"Closing file watcher: {name}")
await file_watcher.close()
for name, file_store in self.service_context.file_stores.items():
logger.info(f"Closing file store: {name}")
await file_store.close()
for name, file_watcher in self.service_context.file_watchers.items():
logger.info(f"Closing file watcher: {name}")
await file_watcher.close()
for name, vector_store in self.service_context.vector_stores.items():
logger.info(f"Closing vector store: {name}")
await vector_store.close()
for name, llm in self.service_context.llms.items():
logger.info(f"Closing LLM: {name}")

31
reme/core/file_store/local_file_store.py
View file

@ -107,6 +107,20 @@ class LocalFileStore(BaseFileStore):
except Exception as e:
logger.error(f"Failed to save file metadata to {self._metadata_file}: {e}")
async def _persist(self) -> None:
"""Persist all in-memory indexes after mutations."""
await self._save_metadata()
await self._save_chunks()
def _delete_file_in_memory(self, path: str, source: MemorySource) -> None:
"""Delete file data from memory without flushing to disk."""
to_delete = [cid for cid, chunk in self._chunks.items() if chunk.path == path and chunk.source == source]
for cid in to_delete:
del self._chunks[cid]
if source.value in self._files:
self._files[source.value].pop(path, None)
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
@ -146,7 +160,7 @@ class LocalFileStore(BaseFileStore):
return
# Remove existing chunks for this file/source first
await self.delete_file(file_meta.path, source)
self._delete_file_in_memory(file_meta.path, source)
# Batch generate embeddings (base class returns mock embeddings when vector_enabled=False)
chunks = await self.get_chunk_embeddings(chunks)
@ -163,15 +177,12 @@ class LocalFileStore(BaseFileStore):
path=file_meta.path,
chunk_count=len(chunks),
)
await self._persist()
async def delete_file(self, path: str, source: MemorySource) -> None:
"""Delete file and all its chunks."""
to_delete = [cid for cid, chunk in self._chunks.items() if chunk.path == path and chunk.source == source]
for cid in to_delete:
del self._chunks[cid]
if source.value in self._files:
self._files[source.value].pop(path, None)
self._delete_file_in_memory(path, source)
await self._persist()
async def delete_file_chunks(self, path: str, chunk_ids: list[str]) -> None:
"""Delete specific chunks for a file."""
@ -187,6 +198,7 @@ class LocalFileStore(BaseFileStore):
source_meta[path].chunk_count = sum(
1 for chunk in self._chunks.values() if chunk.path == path and chunk.source.value == source_key
)
await self._persist()
async def upsert_chunks(
self,
@ -201,6 +213,7 @@ class LocalFileStore(BaseFileStore):
for chunk in chunks:
self._chunks[chunk.id] = chunk
await self._persist()
# ------------------------------------------------------------------
# Read operations
@ -230,6 +243,7 @@ class LocalFileStore(BaseFileStore):
path=file_meta.path,
chunk_count=file_meta.chunk_count,
)
await self._persist()
async def get_file_chunks(
self,
@ -456,6 +470,5 @@ class LocalFileStore(BaseFileStore):
"""Clear all indexed data from memory and disk."""
self._chunks.clear()
self._files.clear()
await self._save_chunks()
await self._save_metadata()
await self._persist()
logger.info(f"Cleared all data from LocalFileStore '{self.store_name}'")

44
reme4/application.py
View file

@ -7,7 +7,7 @@ from pathlib import Path
from typing import AsyncGenerator, TypeVar
from .components import BaseComponent, ApplicationContext
from .components.job import BaseJob
from .components.job import BackgroundJob, BaseJob, CronJob, StreamJob
from .components.service import BaseService
from .enumeration import ComponentEnum
from .schema import ComponentConfig, Response, StreamChunk
@ -48,7 +48,7 @@ class Application(BaseComponent):
cfg = self.config
vault_path = Path(cfg.vault_dir).absolute()
vault_path.mkdir(parents=True, exist_ok=True)
for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]:
for subdir in [cfg.metadata_dir, cfg.session_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]:
if subdir:
(vault_path / subdir).mkdir(parents=True, exist_ok=True)
@ -168,28 +168,34 @@ class Application(BaseComponent):
# ----- Lifecycle -----------------------------------------------------
async def _start(self) -> None:
"""Start components in dependency order, then jobs (background last)."""
"""Start components, then jobs as base > stream > background > cron."""
pool_size = self.config.thread_pool_max_workers
if pool_size > 0:
self.context.thread_pool = ThreadPoolExecutor(max_workers=pool_size)
self.logger.info(f"Thread pool created with max_workers={pool_size}")
components = self._topological_order()
jobs = list(self.context.jobs.values())
# Background jobs come last so they observe a fully wired system.
foreground = [j for j in jobs if j.backend != "background"]
background = [j for j in jobs if j.backend == "background"]
for c in components + foreground + background:
await self._start_one(c)
try:
components = self._topological_order()
jobs = list(self.context.jobs.values())
base_jobs = [j for j in jobs if not isinstance(j, (StreamJob, BackgroundJob))]
stream_jobs = [j for j in jobs if isinstance(j, StreamJob)]
background_jobs = [j for j in jobs if isinstance(j, BackgroundJob) and not isinstance(j, CronJob)]
cron_jobs = [j for j in jobs if isinstance(j, CronJob)]
for c in components + base_jobs + stream_jobs + background_jobs + cron_jobs:
await self._start_one(c)
except Exception:
await self._close()
raise
async def _start_one(self, c: BaseComponent) -> None:
"""Start one component and record it for ordered shutdown; log and swallow failures."""
"""Start one component and record it for ordered shutdown."""
try:
if c.backend == "background":
if isinstance(c, BackgroundJob):
self.logger.info(f"Starting background job: {c.name}")
await c.start()
self._started_components.append(c)
except Exception as e:
self.logger.exception(f"Failed to start {c.component_type.value}:{c.name}: {e}")
raise
async def _close(self) -> None:
"""Close in reverse start order so every peer outlives its dependents."""
@ -211,6 +217,20 @@ class Application(BaseComponent):
raise KeyError(f"Job '{name}' not found")
return await self.context.jobs[name](**kwargs)
async def update_component(self, component_enum: ComponentEnum | str, name: str, /, **kwargs) -> BaseComponent:
"""Update an existing component by type/name; never creates missing components."""
component_enum = ComponentEnum(component_enum)
group = self.context.components.get(component_enum)
if not group or name not in group:
raise KeyError(f"Component '{name}' not found in {component_enum.value}")
component = group[name]
for key, value in kwargs.items():
if not hasattr(component, key):
raise AttributeError(f"Component {component_enum.value}:{name} has no attribute '{key}'")
setattr(component, key, value)
return component
async def run_stream_job(self, name: str, /, **kwargs) -> AsyncGenerator[StreamChunk, None]:
"""Execute a streaming job, yielding chunks as they are produced."""
if name not in self.context.jobs:

344
reme4/components/agent_wrapper/as_agent_wrapper.py
View file

@ -1,34 +1,100 @@
"""AgentScope backend for the unified agent wrapper."""
import datetime
import uuid
import zoneinfo
import json
import re
import time
from collections.abc import AsyncGenerator
from pathlib import Path
from typing import Any, TYPE_CHECKING
from uuid import uuid4
from agentscope.agent import Agent, ContextConfig, ModelConfig, ReActConfig
from agentscope.agent import Agent, ContextConfig, ReActConfig
from agentscope.agent._config import ModelConfig
from agentscope.event import (
DataBlockDeltaEvent,
DataBlockEndEvent,
DataBlockStartEvent,
ExceedMaxItersEvent,
ModelCallEndEvent,
ModelCallStartEvent,
ReplyEndEvent,
ReplyStartEvent,
ThinkingBlockDeltaEvent,
ThinkingBlockEndEvent,
ThinkingBlockStartEvent,
TextBlockDeltaEvent,
TextBlockEndEvent,
TextBlockStartEvent,
ToolCallDeltaEvent,
ToolCallEndEvent,
ToolCallStartEvent,
ToolResultDataDeltaEvent,
ToolResultEndEvent,
ToolResultStartEvent,
ToolResultTextDeltaEvent,
)
from agentscope.message import TextBlock, ToolResultState, UserMsg
from agentscope.permission import PermissionContext, PermissionMode
from agentscope.permission import PermissionBehavior, PermissionContext, PermissionDecision, PermissionMode
from agentscope.state import AgentState
from agentscope.tool import FunctionTool, ToolChunk, Toolkit
from agentscope.tool import (
Bash,
Edit,
FunctionTool,
Glob,
Grep,
Read,
ToolBase,
ToolChunk,
Toolkit,
Write,
)
from .base_agent_wrapper import BaseAgentWrapper
from ..as_llm import BaseAsLLM
from ..component_registry import R
from ...enumeration import ChunkEnum
from ...schema import StreamChunk
from ...utils import AsStateHandler
from ...utils.env_utils import load_env
if TYPE_CHECKING:
from ..job.base_job import BaseJob
_UUID_RE = re.compile(
r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$",
re.IGNORECASE,
)
class BypassAnalysisBash(Bash):
"""Bash variant that delegates permission decisions to PermissionEngine.
AgentScope's built-in Bash performs bypass-immune static analysis before
the engine can apply `permission_mode: bypass`. For this app we want the
configured permission mode to be authoritative.
"""
async def check_permissions(
self,
_tool_input: dict[str, Any],
_context: PermissionContext,
) -> PermissionDecision:
"""Bypass Bash static analysis and let the permission engine decide."""
return PermissionDecision(
behavior=PermissionBehavior.PASSTHROUGH,
message="Bash static analysis skipped; delegating to permission engine.",
)
@R.register("agentscope")
class AsAgentWrapper(BaseAgentWrapper):
"""Agent wrapper backed by AgentScope framework."""
def __init__(self, as_llm: str = "default", **kwargs):
def __init__(self, as_llm: str = "default", session_retention_days: int = 10, **kwargs):
super().__init__(**kwargs)
self.as_llm = self.bind(as_llm, BaseAsLLM, optional=False)
self.session_retention_days = int(session_retention_days)
self._session_cleanup_done = False
@staticmethod
def _make_tool(job: "BaseJob") -> FunctionTool:
@ -42,23 +108,121 @@ class AsAgentWrapper(BaseAgentWrapper):
tool.input_schema = job.parameters
return tool
def _build_agent(self, inputs: Any, **kwargs) -> tuple[Agent, Any]:
@classmethod
def _builtin_tools(cls) -> list[ToolBase]:
"""Return built-in tools expected by local skills."""
return [BypassAnalysisBash(), Edit(), Glob(), Grep(), Read(), Write()]
@property
def session_path(self) -> Path:
"""Directory used for persisted AgentScope sessions."""
if self.app_context is None:
return self.vault_path / "session" / "agentscope"
return self.vault_path / self.app_context.app_config.session_dir / "agentscope"
@staticmethod
def _validate_session_id(session_id: str, field: str = "session_id") -> str:
if not _UUID_RE.match(session_id):
raise ValueError(f"{field} must be a valid UUID: {session_id!r}")
return session_id.lower()
def _cleanup_expired_sessions(self) -> None:
"""Delete persisted session files older than ``session_retention_days``."""
if self._session_cleanup_done or self.session_retention_days <= 0:
self._session_cleanup_done = True
return
session_path = self.session_path
if not session_path.is_dir():
self._session_cleanup_done = True
return
cutoff = time.time() - self.session_retention_days * 24 * 60 * 60
removed = 0
for path in session_path.glob("*.jsonl"):
try:
if path.is_file() and path.stat().st_mtime < cutoff:
path.unlink()
removed += 1
except OSError as exc:
self.logger.warning(f"Failed to clean expired AgentScope session {path}: {exc}")
if removed:
self.logger.info(
f"Cleaned {removed} AgentScope session(s) older than {self.session_retention_days} day(s)",
)
self._session_cleanup_done = True
async def _load_state(self, kwargs: dict[str, Any], perm_mode: PermissionMode) -> AgentState:
resume = kwargs.get("resume") or ""
session_id = kwargs.get("session_id") or ""
fork_session = bool(kwargs.get("fork_session", False))
if resume:
resume = self._validate_session_id(resume, "resume")
if session_id:
session_id = self._validate_session_id(session_id)
if session_id and resume and not fork_session:
raise ValueError("session_id cannot be used with resume unless fork_session=True")
if resume:
handler = AsStateHandler.for_session(self.session_path, resume)
state = await handler.load_or_none()
if state is None:
raise FileNotFoundError(f"AgentScope session not found: {resume}")
state.permission_context = PermissionContext(mode=perm_mode)
state.session_id = resume
if fork_session:
forked = AgentState(
session_id=session_id or str(uuid4()),
summary=state.summary,
context=list(state.context),
permission_context=PermissionContext(mode=perm_mode),
)
return forked
return state
return AgentState(session_id=session_id or str(uuid4()), permission_context=PermissionContext(mode=perm_mode))
async def _dump_state(self, state: AgentState) -> None:
await AsStateHandler.for_session(self.session_path, state.session_id).dump(state)
def _resolve_skills(self, skills: list[str] | str | None) -> list[str]:
"""Resolve configured skill names to AgentScope local skill directories."""
if skills is None:
return []
if skills == "all":
return [str(self.project_skills_root)]
if isinstance(skills, str):
skills = [skills]
return [str(self.project_skills_root / skill) for skill in skills]
def _load_tool_env(self) -> dict[str, str]:
"""Load project environment variables for tools spawned by AgentScope."""
project_env = self.project_path / ".env"
return load_env(project_env) if project_env.exists() else load_env()
async def _build_agent(self, inputs: Any, **kwargs) -> tuple[Agent, Any]:
"""Build an Agent instance from kwargs. Returns (agent, processed_inputs)."""
model = self.as_llm.model if self.as_llm else None
if model is None:
raise ValueError("AsAgentWrapper requires a bound as_llm component with a valid model.")
for k, v in self.kwargs.items():
kwargs.setdefault(k, v)
kwargs = self._merged_kwargs(kwargs)
self._cleanup_expired_sessions()
self._load_tool_env()
system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.")
tools: list["BaseJob"] = kwargs.get("tools", [])
toolkit = kwargs.get("toolkit") or (
Toolkit(tools=[self._make_tool(job) for job in tools]) if tools else Toolkit()
job_tools: list[str] = kwargs.get("job_tools", [])
resolved_jobs = self._resolve_job_tools(job_tools)
skills = self._resolve_skills(kwargs.get("skills"))
toolkit = kwargs.get("toolkit") or Toolkit(
tools=[*self._builtin_tools(), *(self._make_tool(job) for job in resolved_jobs)],
skills_or_loaders=skills,
)
perm_mode = PermissionMode(kwargs.get("permission_mode", "bypass"))
state = AgentState(permission_context=PermissionContext(mode=perm_mode))
state = await self._load_state(kwargs, perm_mode)
agent = Agent(
name=self.name,
@ -76,48 +240,20 @@ class AsAgentWrapper(BaseAgentWrapper):
return agent, inputs
def _session_path(self, session_id: str) -> Path:
tz_name = self.app_context.app_config.timezone if self.app_context else None
if tz_name:
try:
dt = datetime.datetime.now(zoneinfo.ZoneInfo(tz_name))
except Exception:
dt = datetime.datetime.now()
else:
dt = datetime.datetime.now()
date_str = dt.strftime("%Y-%m-%d")
resource = self.app_context.app_config.resource_dir if self.app_context else "resource"
return self.vault_path / resource / date_str / f"session_reme_{session_id}.jsonl"
async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]:
session_id: str = kwargs.pop("session_id", "")
fork_session: bool = kwargs.pop("fork_session", False)
agent, inputs = self._build_agent(inputs, **kwargs)
if session_id:
path = self._session_path(session_id)
if path.exists():
loaded = await AsStateHandler(path).load()
agent.state.session_id = loaded.session_id
agent.state.summary = loaded.summary
agent.state.context = loaded.context
agent.state.reply_id = loaded.reply_id
agent.state.cur_iter = loaded.cur_iter
async def reply(self, inputs: Any, **kwargs) -> dict:
kwargs = self._merged_kwargs(kwargs)
agent, inputs = await self._build_agent(inputs, **kwargs)
await agent.observe(inputs)
await agent.reply()
await self._dump_state(agent.state)
last_msg = agent.state.context[-1]
if session_id:
if fork_session:
new_sid = uuid.uuid4().hex
agent.state.session_id = new_sid
save_path = self._session_path(new_sid)
else:
save_path = self._session_path(session_id)
save_path.parent.mkdir(parents=True, exist_ok=True)
await AsStateHandler(save_path).dump(agent.state)
result = {
"session_id": agent.state.session_id,
"last_message": last_msg.model_dump(),
"result": last_msg.get_text_content(),
}
output_schema: dict | None = kwargs.get("output_schema")
if output_schema is not None:
@ -128,13 +264,107 @@ class AsAgentWrapper(BaseAgentWrapper):
messages=agent.state.context,
structured_model=output_schema,
)
return agent.state.session_id, {"message": last_msg, "structured_output": res.content}
result["structured_output"] = res.content
return agent.state.session_id, last_msg
return result
async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[Any, None]:
"""Stream agent events via AgentScope's reply_stream API."""
agent, inputs = self._build_agent(inputs, **kwargs)
# ----- StreamChunk conversion -------------------------------------------
@classmethod
# pylint: disable=too-many-return-statements
def _event_to_chunk(cls, event: Any) -> StreamChunk | None:
"""Convert an AgentScope event to a unified StreamChunk.
Returns ``None`` for events that should be silently skipped
(e.g. ``RequireUserConfirmEvent``).
"""
if isinstance(event, ReplyStartEvent):
meta = {"reply_id": event.reply_id, "name": event.name, "role": event.role}
return cls._chunk(ChunkEnum.REPLY_START, session_id=event.session_id, chunk="", metadata=meta)
if isinstance(event, ReplyEndEvent):
return cls._chunk(
ChunkEnum.REPLY_END,
session_id=event.session_id,
chunk="",
metadata={"reply_id": event.reply_id},
)
for event_cls, chunk_type, attr in (
(TextBlockStartEvent, ChunkEnum.CONTENT, None),
(TextBlockDeltaEvent, ChunkEnum.CONTENT, "delta"),
(TextBlockEndEvent, ChunkEnum.CONTENT, None),
(ThinkingBlockStartEvent, ChunkEnum.THINK, None),
(ThinkingBlockDeltaEvent, ChunkEnum.THINK, "delta"),
(ThinkingBlockEndEvent, ChunkEnum.THINK, None),
(DataBlockStartEvent, ChunkEnum.DATA, None),
(DataBlockDeltaEvent, ChunkEnum.DATA, "data"),
(DataBlockEndEvent, ChunkEnum.DATA, None),
):
if isinstance(event, event_cls):
kwargs = {"block_id": event.block_id, "chunk": getattr(event, attr) if attr else ""}
if isinstance(event, (DataBlockStartEvent, DataBlockDeltaEvent)):
kwargs["media_type"] = event.media_type
return cls._chunk(chunk_type, **kwargs)
if isinstance(event, ToolCallStartEvent):
payload = {"name": event.tool_call_name, "id": event.tool_call_id}
return cls._chunk(
ChunkEnum.TOOL_CALL,
tool_call_id=event.tool_call_id,
tool_call_name=event.tool_call_name,
chunk=json.dumps(payload),
)
if isinstance(event, ToolCallDeltaEvent):
return cls._chunk(ChunkEnum.TOOL_CALL, tool_call_id=event.tool_call_id, chunk=event.delta)
if isinstance(event, ToolCallEndEvent):
return cls._chunk(ChunkEnum.TOOL_CALL, tool_call_id=event.tool_call_id, chunk="")
if isinstance(event, ToolResultStartEvent):
return cls._chunk(
ChunkEnum.TOOL_RESULT,
tool_call_id=event.tool_call_id,
tool_call_name=event.tool_call_name,
chunk="",
)
if isinstance(event, ToolResultTextDeltaEvent):
return cls._chunk(ChunkEnum.TOOL_RESULT, tool_call_id=event.tool_call_id, chunk=event.delta)
if isinstance(event, ToolResultDataDeltaEvent):
return cls._chunk(
ChunkEnum.TOOL_RESULT,
tool_call_id=event.tool_call_id,
chunk=event.data,
media_type=event.media_type,
metadata={"url": event.url} if event.url else {},
)
if isinstance(event, ToolResultEndEvent):
return cls._chunk(
ChunkEnum.TOOL_RESULT,
tool_call_id=event.tool_call_id,
chunk="",
metadata={"state": str(event.state)},
)
if isinstance(event, ModelCallStartEvent):
return cls._chunk(ChunkEnum.USAGE, chunk="", metadata={"model_name": getattr(event, "model_name", None)})
if isinstance(event, ModelCallEndEvent):
usage = {"input_tokens": event.input_tokens, "output_tokens": event.output_tokens}
return cls._chunk(
ChunkEnum.USAGE,
chunk=json.dumps(usage),
input_tokens=event.input_tokens,
output_tokens=event.output_tokens,
metadata={"model_name": getattr(event, "model_name", None)},
)
if isinstance(event, ExceedMaxItersEvent):
return cls._chunk(ChunkEnum.ERROR, chunk="Exceeded max iterations")
return None
async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]:
"""Stream agent events as unified StreamChunk objects."""
agent, inputs = await self._build_agent(inputs, **kwargs)
async for event in agent.reply_stream(inputs):
yield event
chunk = self._event_to_chunk(event)
if chunk is not None:
chunk.session_id = chunk.session_id or agent.state.session_id
yield chunk
await self._dump_state(agent.state)

61
reme4/components/agent_wrapper/base_agent_wrapper.py
View file

@ -2,12 +2,14 @@
from abc import abstractmethod
from collections.abc import AsyncGenerator
from pathlib import Path
from typing import Any, TYPE_CHECKING
from pydantic import BaseModel
from ..base_component import BaseComponent
from ...enumeration import ComponentEnum
from ...enumeration import ChunkEnum, ComponentEnum
from ...schema import StreamChunk
if TYPE_CHECKING:
from ..job.base_job import BaseJob
@ -23,11 +25,26 @@ class BaseAgentWrapper(BaseComponent):
self.kwargs["system_prompt"] = prompt
return self
def add_tools(self, tools: list["BaseJob"]) -> "BaseAgentWrapper":
"""Append callable tools to the agent. Returns self for chaining."""
self.kwargs.setdefault("tools", []).extend(tools)
def add_job_tools(self, job_tools: list[str]) -> "BaseAgentWrapper":
"""Append job names as tools to the agent. Returns self for chaining."""
self.kwargs.setdefault("job_tools", []).extend(job_tools)
return self
def add_skills(self, skills: list[str] | str) -> "BaseAgentWrapper":
"""Set agent skill names. Returns self for chaining."""
self.kwargs["skills"] = skills
return self
@property
def project_path(self) -> Path:
"""Project root that contains shared assets such as skills."""
return self.vault_path
@property
def project_skills_root(self) -> Path:
"""Project-level skills directory shared by agent backends."""
return self.project_path / "skills"
def set_output_schema(self, schema: dict | type[BaseModel]) -> "BaseAgentWrapper":
"""Set a JSON schema for structured output. Accepts dict or BaseModel class. Returns self for chaining."""
if isinstance(schema, type) and issubclass(schema, BaseModel):
@ -35,17 +52,31 @@ class BaseAgentWrapper(BaseComponent):
self.kwargs["output_schema"] = schema
return self
# TODO add skills
def _resolve_job_tools(self, job_tools: list[str]) -> list["BaseJob"]:
"""Resolve job name strings to BaseJob instances via app_context."""
if not job_tools:
return []
if self.app_context is None:
raise RuntimeError("Cannot resolve job_tools without an app_context")
resolved: list["BaseJob"] = []
for name in job_tools:
if (job := self.app_context.jobs.get(name)) is None:
raise KeyError(f"Job '{name}' not found in app_context.jobs")
resolved.append(job)
return resolved
def _merged_kwargs(self, kwargs: dict[str, Any]) -> dict[str, Any]:
"""Merge component defaults with call-time kwargs; call-time values win."""
return {**self.kwargs, **kwargs}
@staticmethod
def _chunk(chunk_type: ChunkEnum = ChunkEnum.CONTENT, **kwargs: Any) -> StreamChunk:
"""Create a StreamChunk with a short backend-friendly call site."""
return StreamChunk(chunk_type=chunk_type, **kwargs)
@abstractmethod
async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]:
"""Send inputs to the agent and return (session_id, last_message)."""
async def reply(self, inputs: Any, **kwargs) -> dict:
"""Send inputs to the agent and return a dict with session_id and last_message."""
async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[Any, None]:
"""Stream agent events. Yields backend-specific event objects.
Subclasses may override to provide streaming support.
Default implementation falls back to non-streaming reply and yields the final message.
"""
_, msg = await self.reply(inputs, **kwargs)
yield msg
async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]:
"""Stream agent events as unified StreamChunk objects."""

583
reme4/components/agent_wrapper/cc_agent_wrapper.py
View file

@ -1,18 +1,216 @@
"""Claude Code SDK backend for the unified agent wrapper."""
import json
import os
import shutil
from collections.abc import AsyncGenerator
from dataclasses import asdict
from pathlib import Path
from typing import Any, TYPE_CHECKING
from .base_agent_wrapper import BaseAgentWrapper
from ..component_registry import R
from ...enumeration import ChunkEnum
from ...schema import StreamChunk
from ...utils.env_utils import load_env
if TYPE_CHECKING:
from ..job.base_job import BaseJob
from claude_agent_sdk.types import SessionKey, SessionStoreEntry, SessionStoreListEntry
class _CcFileSessionStore:
"""File-backed Claude Code SessionStore rooted under the ReMe vault."""
def __init__(self, root: Path) -> None:
self.root = root
@staticmethod
def _safe_parts(value: str) -> list[str]:
parts = [part for part in value.split("/") if part]
if not parts or any(part in {".", ".."} for part in parts):
raise ValueError(f"Invalid session store path component: {value!r}")
return parts
def _path_for_key(self, key: "SessionKey") -> Path:
session_id = key["session_id"]
subpath = key.get("subpath")
path = self.root.joinpath(*self._safe_parts(session_id))
if subpath:
path = path.joinpath(*self._safe_parts(subpath))
else:
path = path.with_suffix(".jsonl")
if subpath:
path = path.with_suffix(".jsonl")
resolved_root = self.root.resolve()
resolved_path = path.resolve()
if resolved_root != resolved_path and resolved_root not in resolved_path.parents:
raise ValueError(f"Session store path escapes root: {resolved_path}")
return path
@staticmethod
def _read_entries(path: Path) -> list["SessionStoreEntry"]:
"""Read JSONL session-store entries from disk."""
if not path.exists():
return []
entries = []
for line in path.read_text(encoding="utf-8").splitlines():
if line.strip():
entries.append(json.loads(line))
return entries
async def append(self, key: "SessionKey", entries: list["SessionStoreEntry"]) -> None:
"""Append new session-store entries, deduplicating by UUID."""
path = self._path_for_key(key)
path.parent.mkdir(parents=True, exist_ok=True)
existing_uuids = {
entry.get("uuid") for entry in self._read_entries(path) if isinstance(entry, dict) and entry.get("uuid")
}
new_entries = [
entry for entry in entries if not (isinstance(entry, dict) and entry.get("uuid") in existing_uuids)
]
if not new_entries:
return
with path.open("a", encoding="utf-8") as f:
for entry in new_entries:
f.write(json.dumps(entry, ensure_ascii=False, separators=(",", ":")) + "\n")
async def load(self, key: "SessionKey") -> list["SessionStoreEntry"] | None:
"""Load session-store entries for a key."""
path = self._path_for_key(key)
if not path.exists():
return None
return self._read_entries(path)
async def list_sessions(self, _project_key: str) -> list["SessionStoreListEntry"]:
"""List root-level Claude Code sessions."""
if not self.root.exists():
return []
return [
{"session_id": path.stem, "mtime": int(path.stat().st_mtime * 1000)}
for path in self.root.glob("*.jsonl")
if path.is_file()
]
async def delete(self, key: "SessionKey") -> None:
"""Delete a session-store entry and any subkey directory."""
path = self._path_for_key(key)
if path.exists():
path.unlink()
if not key.get("subpath"):
session_dir = self.root.joinpath(*self._safe_parts(key["session_id"]))
if session_dir.exists():
for child in sorted(session_dir.rglob("*"), reverse=True):
if child.is_file():
child.unlink()
elif child.is_dir():
child.rmdir()
session_dir.rmdir()
async def list_subkeys(self, key: dict[str, str]) -> list[str]:
"""List subkeys below a root session key."""
session_dir = self.root.joinpath(*self._safe_parts(key["session_id"]))
if not session_dir.exists():
return []
subkeys = []
for path in session_dir.rglob("*.jsonl"):
if path.is_file():
subkeys.append(str(path.relative_to(session_dir).with_suffix("")))
return subkeys
@R.register("claude_code")
class CcAgentWrapper(BaseAgentWrapper):
"""Agent wrapper backed by Claude Code SDK."""
DEFAULT_DISALLOWED_TOOLS = ["WebSearch"]
@staticmethod
def _first_non_empty(*values: Any) -> str:
for value in values:
if isinstance(value, str) and value:
return value
return ""
def _default_llm_credential(self) -> dict[str, Any]:
"""Return the default as_llm credential config, if available."""
if self.app_context is None:
return {}
components = self.app_context.app_config.components
llm_configs = components.get("as_llm") or components.get("AS_LLM") or components.get("as_llm".upper())
if llm_configs is None:
from ...enumeration import ComponentEnum
llm_configs = components.get(ComponentEnum.AS_LLM)
if not isinstance(llm_configs, dict):
return {}
default_llm = llm_configs.get("default")
credential = getattr(default_llm, "credential", None)
return credential if isinstance(credential, dict) else {}
def _claude_code_api_env(self, kwargs: dict[str, Any]) -> dict[str, str]:
"""Resolve Anthropic-compatible API environment for Claude Code."""
credential = kwargs.get("credential") if isinstance(kwargs.get("credential"), dict) else {}
default_credential = self._default_llm_credential()
base_url = self._first_non_empty(
kwargs.get("base_url"),
credential.get("base_url"),
os.getenv("ANTHROPIC_BASE_URL"),
os.getenv("CLAUDE_CODE_BASE_URL"),
os.getenv("LLM_BASE_URL"),
default_credential.get("base_url"),
)
api_key = self._first_non_empty(
kwargs.get("api_key"),
credential.get("api_key"),
os.getenv("ANTHROPIC_AUTH_TOKEN"),
os.getenv("CLAUDE_CODE_API_KEY"),
os.getenv("LLM_API_KEY"),
default_credential.get("api_key"),
)
env: dict[str, str] = {}
if base_url:
env["ANTHROPIC_BASE_URL"] = base_url
if api_key:
env["ANTHROPIC_AUTH_TOKEN"] = api_key
return env
@property
def session_path(self) -> Path:
"""Directory used for persisted Claude Code sessions."""
if self.app_context is None:
return self.vault_path / "session"
return self.vault_path / self.app_context.app_config.session_dir
def _ensure_claude_skill_dir(self, config_dir: Path) -> None:
"""Expose project skills through Claude Code skill discovery locations."""
project_skills = self.project_skills_root
if not project_skills.exists():
return
for target in (self.project_path / ".claude" / "skills", config_dir / "skills"):
target.parent.mkdir(parents=True, exist_ok=True)
try:
if target.exists() or target.is_symlink():
if target.resolve() == project_skills.resolve():
continue
if target.is_dir() and not target.is_symlink():
shutil.rmtree(target)
else:
target.unlink()
target.symlink_to(project_skills, target_is_directory=True)
except OSError as exc:
self.logger.warning(f"Failed to link Claude Code skills directory {target}: {exc}")
@staticmethod
def _make_tool(job: "BaseJob"):
from claude_agent_sdk import SdkMcpTool
@ -23,46 +221,69 @@ class CcAgentWrapper(BaseAgentWrapper):
return SdkMcpTool(name=job.name, description=job.description, input_schema=job.parameters, handler=run_job)
async def reply(self, inputs: Any, **kwargs) -> tuple[str, Any]:
from claude_agent_sdk import query, ResultMessage, create_sdk_mcp_server
def _build_options(self, inputs: Any, stream: bool = False, **kwargs) -> Any:
"""Build ClaudeAgentOptions from kwargs.
``stream=True`` enables ``include_partial_messages`` so that
``StreamEvent`` messages are emitted alongside the final
``ResultMessage``.
"""
from claude_agent_sdk import create_sdk_mcp_server
from claude_agent_sdk.types import ClaudeAgentOptions
for k, v in self.kwargs.items():
kwargs.setdefault(k, v)
kwargs = self._merged_kwargs(kwargs)
session_id: str = kwargs.pop("session_id", "")
fork_session: bool = kwargs.pop("fork_session", False)
skills = kwargs.get("skills")
if isinstance(skills, str) and skills != "all":
kwargs["skills"] = [skills]
sp = kwargs.get("system_prompt")
if isinstance(sp, str):
kwargs["system_prompt"] = {
"type": "preset",
"preset": "claude_code",
"append": sp,
"exclude_dynamic_sections": True,
}
kwargs.setdefault("setting_sources", [])
if "setting_sources" not in kwargs and kwargs.get("skills") is None:
kwargs["setting_sources"] = []
disallowed_tools = list(kwargs.get("disallowed_tools") or [])
for tool_name in self.DEFAULT_DISALLOWED_TOOLS:
if tool_name not in disallowed_tools:
disallowed_tools.append(tool_name)
kwargs["disallowed_tools"] = disallowed_tools
opts = ClaudeAgentOptions()
skip_keys = {"tools", "output_schema"}
if stream:
opts.include_partial_messages = True
skip_keys = {"job_tools", "output_schema", "api_key", "base_url", "credential"}
for k, v in kwargs.items():
if k not in skip_keys and hasattr(opts, k):
setattr(opts, k, v)
if session_id:
opts.resume = session_id
if fork_session:
opts.fork_session = True
elif fork_session:
# fork_session with no session_id to fork from is meaningless.
raise ValueError("fork_session=True requires a non-empty session_id")
model = getattr(opts, "model", None) or kwargs.get("model")
project_env = self.project_path / ".env"
opts.env.update(load_env(project_env) if project_env.exists() else load_env())
extra_env_dict: dict = self._claude_code_api_env(kwargs)
if model:
extra_env_dict.update(
{
"ANTHROPIC_MODEL": model,
"ANTHROPIC_DEFAULT_HAIKU_MODEL": model,
"ANTHROPIC_DEFAULT_SONNET_MODEL": model,
"ANTHROPIC_DEFAULT_OPUS_MODEL": model,
},
)
opts.env.update(extra_env_dict)
self.session_path.mkdir(parents=True, exist_ok=True)
opts.cwd = opts.cwd or self.project_path
claude_config_dir = self.session_path / "claude_config"
opts.env.setdefault("CLAUDE_CONFIG_DIR", str(claude_config_dir))
if opts.skills is not None:
self._ensure_claude_skill_dir(claude_config_dir)
opts.session_store = opts.session_store or _CcFileSessionStore(self.session_path / "claude_code")
tools: list["BaseJob"] = kwargs.get("tools", [])
if tools:
sdk_tools = [self._make_tool(job) for job in tools]
server = create_sdk_mcp_server(name="reme_tools", tools=sdk_tools)
opts.mcp_servers = (opts.mcp_servers if isinstance(opts.mcp_servers, dict) else {}) | {"reme": server}
opts.allowed_tools.extend(job.name for job in tools)
job_tools: list[str] = kwargs.get("job_tools", [])
resolved_jobs = self._resolve_job_tools(job_tools)
if resolved_jobs:
sdk_tools = [self._make_tool(job) for job in resolved_jobs]
server = create_sdk_mcp_server(name="mcp_server", tools=sdk_tools)
opts.mcp_servers = opts.mcp_servers if isinstance(opts.mcp_servers, dict) else {}
opts.mcp_servers["mcp_server"] = server
opts.allowed_tools.extend(job.name for job in resolved_jobs)
if output_schema := kwargs.get("output_schema"):
opts.output_format = {"type": "json_schema", "schema": output_schema}
@ -70,6 +291,201 @@ class CcAgentWrapper(BaseAgentWrapper):
if not isinstance(inputs, str):
raise NotImplementedError("Only string input is supported for Claude Code.")
return opts
# ----- StreamChunk conversion -------------------------------------------
@classmethod
# pylint: disable=too-many-return-statements
def _raw_event_to_chunk(
cls,
raw: dict,
session_id: str | None = None,
block_ids: dict[int, str] | None = None,
block_types: dict[int, str] | None = None,
tool_call_names: dict[int, str] | None = None,
) -> StreamChunk | None:
"""Convert a raw Anthropic streaming event dict to a StreamChunk.
``block_ids`` / ``block_types`` / ``tool_call_names`` map
content-block ``index`` to metadata tracked from the
``content_block_start`` event, so that later delta / stop
events can reference the correct ``block_id`` and
``chunk_type``.
Returns ``None`` for events that should be silently skipped.
"""
event_type = raw.get("type")
# --- Message-level lifecycle ----------------------------------------
if event_type == "message_start":
message = raw.get("message", {})
meta = {"message_id": message.get("id"), "model": message.get("model"), "role": message.get("role")}
return cls._chunk(ChunkEnum.REPLY_START, session_id=session_id, chunk="", metadata=meta)
if event_type == "message_delta":
delta = raw.get("delta", {})
usage = raw.get("usage", {})
return cls._chunk(
ChunkEnum.REPLY_END,
session_id=session_id,
chunk="",
output_tokens=usage.get("output_tokens"),
metadata={"stop_reason": delta.get("stop_reason")},
)
if event_type == "message_stop":
return cls._chunk(ChunkEnum.REPLY_END, session_id=session_id, chunk="")
# --- Content-block lifecycle ----------------------------------------
if event_type == "content_block_start":
idx, content_block = raw.get("index", 0), raw.get("content_block", {})
block_type, bid = content_block.get("type", ""), content_block.get("id", "")
# Track for later delta / stop correlation
if block_ids is not None and bid:
block_ids[idx] = bid
if block_types is not None and block_type:
block_types[idx] = block_type
if tool_call_names is not None and content_block.get("name"):
tool_call_names[idx] = content_block["name"]
if block_type == "text":
return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk=content_block.get("text", ""))
if block_type == "thinking":
return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk=content_block.get("thinking", ""))
if block_type == "tool_use":
payload = {"name": content_block.get("name"), "id": content_block.get("id")}
return cls._chunk(
ChunkEnum.TOOL_CALL,
block_id=bid,
tool_call_id=content_block.get("id"),
tool_call_name=content_block.get("name"),
chunk=json.dumps(payload),
)
return None
if event_type == "content_block_delta":
delta = raw.get("delta", {})
delta_type = delta.get("type", "")
idx = raw.get("index", 0)
bid = block_ids.get(idx) if block_ids else None
tc_name = tool_call_names.get(idx) if tool_call_names else None
if delta_type == "text_delta":
return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk=delta.get("text", ""))
if delta_type == "thinking_delta":
return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk=delta.get("thinking", ""))
if delta_type == "input_json_delta":
return cls._chunk(
ChunkEnum.TOOL_CALL,
block_id=bid,
tool_call_id=bid,
tool_call_name=tc_name,
chunk=delta.get("partial_json", ""),
)
return None
if event_type == "content_block_stop":
idx = raw.get("index", 0)
bid = block_ids.get(idx) if block_ids else None
btype = block_types.get(idx) if block_types else None
tc_name = tool_call_names.get(idx) if tool_call_names else None
if btype == "tool_use":
return cls._chunk(ChunkEnum.TOOL_CALL, block_id=bid, tool_call_id=bid, tool_call_name=tc_name, chunk="")
if btype == "thinking":
return cls._chunk(ChunkEnum.THINK, block_id=bid, chunk="")
# text or unknown -> CONTENT
return cls._chunk(ChunkEnum.CONTENT, block_id=bid, chunk="")
# Ping / other unknown types -> skip
return None
@classmethod
def _message_content_to_chunks(
cls,
msg: Any,
session_id: str | None = None,
visible_tool_call_ids: set[str] | None = None,
include_text: bool = False,
) -> list[StreamChunk]:
"""Convert non-partial SDK message content blocks into stream chunks.
Claude Code streams assistant text/tool-use deltas as ``StreamEvent``
objects, but tool results can arrive later as regular message content
blocks. Surface those blocks so the UI can show what each tool
returned. Some SDK/CLI combinations also put assistant text only in
regular message blocks, so callers can opt into text conversion.
"""
chunks: list[StreamChunk] = []
content = getattr(msg, "content", None)
if not isinstance(content, list):
return chunks
for block in content:
block_name = block.__class__.__name__
if include_text and block_name == "TextBlock":
text = getattr(block, "text", "")
if text:
chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=text))
elif include_text and isinstance(block, dict) and block.get("type") == "text":
text = block.get("text", "")
if text:
chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=text))
elif include_text and isinstance(block, str):
chunks.append(cls._chunk(ChunkEnum.CONTENT, session_id=session_id, chunk=block))
elif block_name in {"ToolResultBlock", "ServerToolResultBlock"}:
tool_use_id = getattr(block, "tool_use_id", None)
if visible_tool_call_ids is not None and tool_use_id not in visible_tool_call_ids:
continue
payload: dict[str, Any] = {
"tool_use_id": tool_use_id,
"content": getattr(block, "content", None),
}
if hasattr(block, "is_error"):
payload["is_error"] = getattr(block, "is_error")
chunks.append(
cls._chunk(
ChunkEnum.TOOL_RESULT,
session_id=session_id,
block_id=tool_use_id,
tool_call_id=tool_use_id,
chunk=payload,
),
)
return chunks
@staticmethod
def _result_message_is_error(msg: Any) -> bool:
"""Return whether an SDK ResultMessage represents a failed result."""
subtype = getattr(msg, "subtype", None)
if isinstance(subtype, str) and subtype.lower() == "success":
return False
is_error = getattr(msg, "is_error", False)
if isinstance(is_error, bool):
return is_error
if isinstance(is_error, str):
return is_error.lower() in {"true", "error", "errored", "failed", "failure"}
return isinstance(subtype, str) and subtype.lower() in {"error", "failed", "failure"}
@staticmethod
def _is_trailing_success_error(exc: Exception) -> bool:
"""Return whether an SDK iterator error is the known success-exit artifact."""
return "Claude Code returned an error result: success" in str(exc)
# ----- reply / reply_stream --------------------------------------------
async def reply(self, inputs: Any, **kwargs) -> dict:
from claude_agent_sdk import query, ResultMessage
opts = self._build_options(inputs, stream=False, **kwargs)
last_msg = None
async for msg in query(prompt=inputs, options=opts):
if isinstance(msg, ResultMessage):
@ -78,7 +494,108 @@ class CcAgentWrapper(BaseAgentWrapper):
if last_msg is None:
raise ValueError("No message received from Claude Code.")
if output_schema:
structured = last_msg.structured_output or {}
return last_msg.session_id or "", {"message": last_msg, "structured_output": structured}
return last_msg.session_id or "", last_msg
result = {
"session_id": last_msg.session_id or "",
"last_message": asdict(last_msg),
"result": last_msg.result,
}
output_schema = kwargs.get("output_schema") or self.kwargs.get("output_schema")
if output_schema and last_msg.structured_output:
result["structured_output"] = last_msg.structured_output
return result
async def reply_stream(self, inputs: Any, **kwargs) -> AsyncGenerator[StreamChunk, None]:
"""Stream Claude Code events as unified StreamChunk objects."""
from claude_agent_sdk import query, ResultMessage, AssistantMessage, StreamEvent, UserMessage
from claude_agent_sdk.types import RateLimitEvent
opts = self._build_options(inputs, stream=True, **kwargs)
block_ids: dict[int, str] = {}
block_types: dict[int, str] = {}
tool_call_names: dict[int, str] = {}
visible_tool_call_ids: set[str] = set()
current_session_id: str | None = None
emitted_content = False
received_result_message = False
stream = query(prompt=inputs, options=opts)
try:
async for msg in stream:
if isinstance(msg, StreamEvent):
current_session_id = msg.session_id or current_session_id
chunk = self._raw_event_to_chunk(
msg.event,
session_id=msg.session_id,
block_ids=block_ids,
block_types=block_types,
tool_call_names=tool_call_names,
)
if chunk is not None:
chunk.session_id = chunk.session_id or msg.session_id
if chunk.chunk_type == ChunkEnum.TOOL_CALL and chunk.tool_call_id:
visible_tool_call_ids.add(chunk.tool_call_id)
if chunk.chunk_type == ChunkEnum.CONTENT and chunk.chunk:
emitted_content = True
yield chunk
elif isinstance(msg, UserMessage):
for chunk in self._message_content_to_chunks(msg, current_session_id, visible_tool_call_ids):
yield chunk
elif isinstance(msg, ResultMessage):
received_result_message = True
current_session_id = msg.session_id or current_session_id
if not emitted_content and getattr(msg, "result", None):
emitted_content = True
yield self._chunk(ChunkEnum.CONTENT, session_id=msg.session_id or "", chunk=msg.result)
# Final result: emit USAGE + REPLY_END
meta = {
"duration_ms": msg.duration_ms,
"duration_api_ms": msg.duration_api_ms,
"stop_reason": msg.stop_reason,
"num_turns": msg.num_turns,
}
yield self._chunk(
ChunkEnum.USAGE,
session_id=msg.session_id or "",
chunk=json.dumps(msg.usage or {}),
metadata=meta,
)
if self._result_message_is_error(msg):
yield self._chunk(
ChunkEnum.ERROR,
session_id=msg.session_id or "",
chunk=str(msg.errors) if msg.errors else "Unknown error",
)
yield self._chunk(ChunkEnum.REPLY_END, session_id=msg.session_id or "", chunk="")
elif isinstance(msg, AssistantMessage):
current_session_id = msg.session_id or current_session_id
# Intermediate assistant text/tool-use is already streamed
# via StreamEvents. Still surface tool-result blocks if the
# SDK includes any in a regular assistant message.
for chunk in self._message_content_to_chunks(
msg,
current_session_id,
visible_tool_call_ids,
include_text=not emitted_content,
):
if chunk.chunk_type == ChunkEnum.CONTENT and chunk.chunk:
emitted_content = True
yield chunk
elif isinstance(msg, RateLimitEvent):
yield self._chunk(ChunkEnum.ERROR, session_id=msg.session_id, chunk="Rate limit exceeded")
except Exception as exc:
if received_result_message and self._is_trailing_success_error(exc):
self.logger.debug(f"Ignoring Claude Code trailing success error after final result: {exc}")
else:
raise
finally:
try:
await stream.aclose()
except Exception as exc:
if not (received_result_message and self._is_trailing_success_error(exc)):
raise
self.logger.debug(f"Ignoring Claude Code stream close error after final result: {exc}")

61
reme4/components/as_embedding/__init__.py
View file

@ -1,12 +1,16 @@
"""AgentScope embedding model wrappers."""
from typing import Any
from agentscope.credential import (
CredentialBase,
DashScopeCredential,
GeminiCredential,
OllamaCredential,
OpenAICredential,
)
from agentscope.embedding import (
DashScopeMultiModalEmbedding,
DashScopeTextEmbedding,
EmbeddingModelBase,
GeminiTextEmbedding,
OllamaTextEmbedding,
OpenAITextEmbedding,
)
from ..base_component import BaseComponent
@ -15,13 +19,14 @@ from ...enumeration import ComponentEnum
class BaseAsEmbedding(BaseComponent):
"""Base wrapper for AgentScope embedding models. Builds ``self.model`` in ``_start``."""
"""Base wrapper for AgentScope embedding models."""
component_type = ComponentEnum.AS_EMBEDDING
credential_cls: type[CredentialBase]
def __init__(self, **kwargs) -> None:
super().__init__(**kwargs)
self.model: EmbeddingModelBase | None = None
self.model: EmbeddingModelBase[Any] | None = None
@property
def dimensions(self) -> int:
@ -29,58 +34,60 @@ class BaseAsEmbedding(BaseComponent):
assert self.model is not None
return self.model.dimensions
async def __call__(self, text: list[str], **kwargs) -> list[list[float]]:
async def __call__(self, inputs: list[Any], **kwargs) -> list[list[float]]:
assert self.model is not None
response = await self.model(text, **kwargs) # pylint: disable=not-callable
response = await self.model(inputs, **kwargs) # pylint: disable=not-callable
return response.embeddings
async def _close(self) -> None:
self.model = None
async def _start(self) -> None:
if self.model is not None:
return
kwargs = dict(self.kwargs)
credential = self.credential_cls(**kwargs.pop("credential", {}))
model_cls = self.credential_cls.get_embedding_model_class()
if model_cls is None:
raise ValueError(f"{self.credential_cls.__name__} does not support embeddings.")
params_dict = kwargs.pop("parameters", None)
parameters = model_cls.Parameters(**params_dict) if params_dict else None
self.model = model_cls(credential=credential, parameters=parameters, **kwargs)
@R.register("openai")
class OpenAIAsEmbedding(BaseAsEmbedding):
"""OpenAI embedding model wrapper."""
async def _start(self) -> None:
self.model = OpenAITextEmbedding(**self.kwargs)
async def _close(self) -> None:
if self.model is not None:
assert isinstance(self.model, OpenAITextEmbedding)
await self.model.client.close()
credential_cls = OpenAICredential
@R.register("dashscope")
class DashScopeAsEmbedding(BaseAsEmbedding):
"""DashScope text embedding model wrapper."""
"""DashScope embedding model wrapper."""
async def _start(self) -> None:
self.model = DashScopeTextEmbedding(**self.kwargs)
credential_cls = DashScopeCredential
@R.register("dashscope_multimodal")
class DashScopeMultiModalAsEmbedding(BaseAsEmbedding):
"""DashScope multimodal embedding model wrapper."""
async def _start(self) -> None:
self.model = DashScopeMultiModalEmbedding(**self.kwargs)
credential_cls = DashScopeCredential
@R.register("gemini")
class GeminiAsEmbedding(BaseAsEmbedding):
"""Gemini embedding model wrapper."""
async def _start(self) -> None:
self.model = GeminiTextEmbedding(**self.kwargs)
credential_cls = GeminiCredential
@R.register("ollama")
class OllamaAsEmbedding(BaseAsEmbedding):
"""Ollama embedding model wrapper."""
async def _start(self) -> None:
self.model = OllamaTextEmbedding(**self.kwargs)
credential_cls = OllamaCredential
__all__ = [

5
reme4/components/as_llm/__init__.py
View file

@ -32,6 +32,8 @@ class BaseAsLLM(BaseComponent):
self.model: ChatModelBase | None = None
async def _start(self) -> None:
if self.model is not None:
return
kwargs = dict(self.kwargs)
credential = self.credential_cls(**kwargs.pop("credential", {}))
model_cls = credential.get_chat_model_class()
@ -39,9 +41,6 @@ class BaseAsLLM(BaseComponent):
parameters = model_cls.Parameters(**params_dict) if params_dict else None
self.model = model_cls(credential=credential, parameters=parameters, **kwargs)
async def _close(self) -> None:
self.model = None
@R.register("openai")
class OpenAIAsLLM(BaseAsLLM):

21
reme4/components/base_component.py
View file

@ -220,10 +220,23 @@ class BaseComponent(ComponentMixin, ABC):
async with self._lock:
if not self._is_started:
return
await self._close()
for owned in reversed(self._owned):
await owned.close()
self._is_started = False
first_error: BaseException | None = None
try:
await self._close()
except BaseException as exc:
first_error = exc
finally:
for owned in reversed(self._owned):
try:
await owned.close()
except BaseException as exc:
if first_error is None:
first_error = exc
else:
self.logger.exception(f"Failed to close owned component {owned.name}: {exc}")
self._is_started = False
if first_error is not None:
raise first_error
async def restart(self) -> None:
"""Close then start the component."""

14
reme4/components/embedding_store/local_embedding_store.py
View file

@ -172,17 +172,17 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
def _load_sync(self) -> None:
try:
data = np.load(self.cache_path)
with np.load(self.cache_path) as data:
for key, emb in zip(data["keys"], data["embeddings"]):
if len(emb) != self.dimensions:
continue
if len(self._cache) >= self.max_cache_size:
break
self._cache[str(key)] = emb.astype(np.float16)
except Exception:
self.logger.exception("Failed to load embedding cache, removing")
self.cache_path.unlink(missing_ok=True)
return
for key, emb in zip(data["keys"], data["embeddings"]):
if len(emb) != self.dimensions:
continue
if len(self._cache) >= self.max_cache_size:
break
self._cache[str(key)] = emb.astype(np.float16)
self.logger.info(f"Loaded {len(self._cache)} embeddings from {self.cache_path}")
async def dump(self) -> None:

51
reme4/components/file_catalog/local_file_catalog.py
View file

@ -1,10 +1,11 @@
"""Local file catalog backend: in-memory dict persisted as JSONL."""
"""Local file catalog backend: in-memory dict persisted as compressed JSONL."""
import aiofiles
import asyncio
from .base_file_catalog import BaseFileCatalog
from ..component_registry import R
from ...schema import FileNode
from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst
@R.register("local")
@ -15,48 +16,44 @@ class LocalFileCatalog(BaseFileCatalog):
super().__init__(**kwargs)
self.encoding = encoding
self._nodes: dict[str, FileNode] = {}
self._io_lock = asyncio.Lock()
self.component_metadata_path.mkdir(parents=True, exist_ok=True)
self._catalog_file = self.component_metadata_path / f"{self.name}.jsonl"
self._catalog_file = self.component_metadata_path / f"{self.name}.jsonl.zst"
async def load(self) -> None:
if not self._catalog_file.exists():
return
try:
async with self._io_lock:
if not self._catalog_file.exists():
return
await self._read_jsonl()
self.logger.info(f"Loaded {len(self._nodes)} nodes from {self._catalog_file}")
except Exception as e:
self.logger.exception(f"Failed to load {self._catalog_file}: {e}")
async def dump(self) -> None:
try:
async with self._io_lock:
await self._write_jsonl()
self.logger.info(f"Saved {len(self._nodes)} nodes to {self._catalog_file}")
except Exception as e:
self.logger.exception(f"Failed to write {self._catalog_file}: {e}")
async def upsert(self, nodes: list[FileNode]) -> None:
for node in nodes:
self._nodes[node.path] = node
async with self._io_lock:
for node in nodes:
self._nodes[node.path] = node
async def delete(self, path: str | list[str]) -> None:
paths = [path] if isinstance(path, str) else path
for p in paths:
self._nodes.pop(p, None)
async with self._io_lock:
for p in paths:
self._nodes.pop(p, None)
async def get_nodes(self, paths: list[str] | None = None) -> list[FileNode]:
if paths is None:
return list(self._nodes.values())
return [self._nodes[p] for p in paths if p in self._nodes]
async with self._io_lock:
if paths is None:
return list(self._nodes.values())
return [self._nodes[p] for p in paths if p in self._nodes]
async def _read_jsonl(self) -> None:
async with aiofiles.open(self._catalog_file, encoding=self.encoding) as f:
async for line in f:
if stripped := line.strip():
node = FileNode.model_validate_json(stripped)
self._nodes[node.path] = node
for line in read_jsonl_zst(self._catalog_file, self.encoding):
if stripped := line.strip():
node = FileNode.model_validate_json(stripped)
self._nodes[node.path] = node
async def _write_jsonl(self) -> None:
tmp = self._catalog_file.with_suffix(".tmp")
async with aiofiles.open(tmp, "w", encoding=self.encoding) as f:
await f.write("\n".join(n.model_dump_json() for n in self._nodes.values()))
tmp.replace(self._catalog_file)
write_jsonl_zst(self._catalog_file, (n.model_dump_json() for n in self._nodes.values()), self.encoding)

51
reme4/components/file_chunker/markdown_file_chunker.py
View file

@ -18,7 +18,8 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
import frontmatter
import yaml
from pydantic import ValidationError
from .base_file_chunker import BaseFileChunker
@ -128,26 +129,51 @@ class MarkdownFileChunker(BaseFileChunker):
file_path = Path(path)
rel_path = self.to_vault_relative(path)
post = frontmatter.loads(file_path.read_text(encoding=self.encoding))
front_matter, content, line_offset = self._parse_front_matter(file_path.read_text(encoding=self.encoding))
chunks: list[FileChunk] = []
if post.content and post.content.strip():
if content and content.strip():
with MarkdownRenderer() as renderer:
tree = self._build_tree(Document(post.content), renderer)
tree = self._build_tree(Document(content), renderer, line_offset=line_offset)
chunks = self._chunk_node(tree, "", "", rel_path, renderer)
links = WikilinkHandler.extract_links(post.content, rel_path) if post.content else []
links = WikilinkHandler.extract_links(content, rel_path) if content else []
node = FileNode(
path=rel_path,
st_mtime=file_path.stat().st_mtime,
chunk_ids=[chunk.id for chunk in chunks],
links=links,
front_matter=FileFrontMatter(**dict(post.metadata)),
front_matter=front_matter,
)
return node, chunks
def _build_tree(self, doc: Any, renderer) -> MdNode:
@staticmethod
def _parse_front_matter(text: str) -> tuple[FileFrontMatter, str, int]:
"""Parse YAML frontmatter, returning 1-based line offset for body AST lines.
Invalid YAML is ignored so a single bad frontmatter block does not
prevent indexing the markdown body.
"""
lines = text.splitlines(keepends=True)
if not lines or lines[0].strip() != "---":
return FileFrontMatter(), text, 0
close_idx = next((i for i, line in enumerate(lines[1:], 1) if line.strip() == "---"), None)
if close_idx is None:
return FileFrontMatter(), text, 0
front_matter = FileFrontMatter()
try:
data = yaml.safe_load("".join(lines[1:close_idx]).strip()) or {}
if isinstance(data, dict):
front_matter = FileFrontMatter(**data)
except (yaml.YAMLError, TypeError, ValidationError):
front_matter = FileFrontMatter()
return front_matter, "".join(lines[close_idx + 1 :]), close_idx + 1
def _build_tree(self, doc: Any, renderer, line_offset: int = 0) -> MdNode:
"""Heading-level stack folds mistletoe's flat children into nested
sections; non-headings attach as ``body`` to the current section
(or root before the first heading)."""
@ -157,12 +183,13 @@ class MarkdownFileChunker(BaseFileChunker):
SetextHeading,
)
root = MdNode(kind="root", start_line=1, end_line=1)
root = MdNode(kind="root", start_line=line_offset + 1, end_line=line_offset + 1)
stack: list[MdNode] = [root]
for child in doc.children or []:
if isinstance(child, BlankLine):
continue
line = getattr(child, "line_number", None) or stack[-1].start_line
raw_line = getattr(child, "line_number", None)
line = raw_line + line_offset if raw_line is not None else stack[-1].start_line
if isinstance(child, (Heading, SetextHeading)):
level = max(1, getattr(child, "level", 1))
while len(stack) > 1 and stack[-1].level >= level:
@ -369,10 +396,11 @@ class MarkdownFileChunker(BaseFileChunker):
lines = body.text.split("\n")
header, data = "\n".join(lines[:2]), lines[2:]
rows = [r for r in (body.block.children or []) if isinstance(r, TableRow)]
line_offset = body.start_line - (getattr(body.block, "line_number", None) or body.start_line)
base = body.start_line + 2
def line_of(i: int) -> int:
return rows[i].line_number if i < len(rows) and rows[i].line_number else base + i
return rows[i].line_number + line_offset if i < len(rows) and rows[i].line_number else base + i
units = [(text, line_of(i), line_of(i)) for i, text in enumerate(data)]
return self._emit_packed(
@ -426,11 +454,12 @@ class MarkdownFileChunker(BaseFileChunker):
if not items:
return self._split_lines(body, before, after, path)
units: list[tuple[str, int, int]] = []
line_offset = body.start_line - (getattr(body.block, "line_number", None) or body.start_line)
for it in items:
text = renderer.render(it).rstrip("\n")
if not text:
continue
line = it.line_number or body.start_line
line = it.line_number + line_offset if it.line_number else body.start_line
units.append((text, line, line + text.count("\n")))
return self._emit_packed(
units,

36
reme4/components/file_graph/local_file_graph.py
View file

@ -6,6 +6,7 @@ from .base_file_graph import BaseFileGraph
from ..component_registry import R
from ...enumeration import LinkScopeEnum
from ...schema import FileLink, FileNode
from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst
@R.register("local")
@ -17,7 +18,7 @@ class LocalFileGraph(BaseFileGraph):
self._nodes: dict[str, FileNode] = {}
self._inverse: dict[str, set[str]] = {} # real target → sources
self._pending: dict[str, set[str]] = {} # virtual target → sources
self._graph_file: Path = self.component_metadata_path / f"{self.name}.jsonl"
self._graph_file: Path = self.component_metadata_path / f"{self.name}.jsonl.zst"
# -- Lifecycle ---------------------------------------------------------
@ -30,21 +31,17 @@ class LocalFileGraph(BaseFileGraph):
if not self._graph_file.exists():
return
try:
with open(self._graph_file, "r", encoding="utf-8") as f:
for line in f:
if line.strip():
node = FileNode.model_validate_json(line)
self._nodes[node.path] = node
for line in read_jsonl_zst(self._graph_file):
if line.strip():
node = FileNode.model_validate_json(line)
self._nodes[node.path] = node
self.logger.info(f"Loaded {len(self._nodes)} nodes from {self._graph_file}")
except Exception as e:
self.logger.exception(f"Failed to load {self._graph_file}: {e}")
async def dump(self) -> None:
try:
tmp = self._graph_file.with_suffix(".tmp")
with open(tmp, "w", encoding="utf-8") as f:
f.writelines(f"{n.model_dump_json()}\n" for n in self._nodes.values())
tmp.replace(self._graph_file)
write_jsonl_zst(self._graph_file, (n.model_dump_json() for n in self._nodes.values()))
self.logger.info(f"Saved {len(self._nodes)} nodes to {self._graph_file}")
except Exception as e:
self.logger.exception(f"Failed to write {self._graph_file}: {e}")
@ -67,7 +64,12 @@ class LocalFileGraph(BaseFileGraph):
if not srcs:
del bucket[target]
def _scope_match(self, target: str, scope: LinkScopeEnum) -> bool:
@staticmethod
def _normalize_scope(scope: LinkScopeEnum | str) -> LinkScopeEnum:
return scope if isinstance(scope, LinkScopeEnum) else LinkScopeEnum(scope)
def _scope_match(self, target: str, scope: LinkScopeEnum | str) -> bool:
scope = self._normalize_scope(scope)
if scope is LinkScopeEnum.ALL:
return True
is_real = target in self._nodes
@ -120,18 +122,24 @@ class LocalFileGraph(BaseFileGraph):
# -- Link access -------------------------------------------------------
async def get_outlinks(self, path: str, scope: LinkScopeEnum = LinkScopeEnum.REAL) -> list[FileLink]:
async def get_outlinks(self, path: str, scope: LinkScopeEnum | str = LinkScopeEnum.REAL) -> list[FileLink]:
scope = self._normalize_scope(scope)
node = self._nodes.get(path)
if node is None:
return []
return [lnk for lnk in node.links if lnk.target_path and self._scope_match(lnk.target_path, scope)]
async def get_inlinks(self, path: str, scope: LinkScopeEnum = LinkScopeEnum.REAL) -> list[FileLink]:
async def get_inlinks(self, path: str, scope: LinkScopeEnum | str = LinkScopeEnum.REAL) -> list[FileLink]:
scope = self._normalize_scope(scope)
sources: set[str] = set()
if scope in (LinkScopeEnum.REAL, LinkScopeEnum.ALL):
sources |= self._inverse.get(path, set())
if scope in (LinkScopeEnum.VIRTUAL, LinkScopeEnum.ALL):
sources |= self._pending.get(path, set())
return [
link for src in sources if src in self._nodes for link in self._nodes[src].links if link.target_path == path
link
for src in sorted(sources)
if src in self._nodes
for link in self._nodes[src].links
if link.target_path == path
]

22
reme4/components/file_graph/neo4j_file_graph.py
View file

@ -88,6 +88,9 @@ class Neo4jFileGraph(BaseFileGraph):
)
self._database: str = database
self._driver = None
self._n_nodes = 0
self._n_virtual = 0
self._n_edges = 0
# -- Lifecycle ---------------------------------------------------------
@ -107,11 +110,11 @@ class Neo4jFileGraph(BaseFileGraph):
await session.run(
"CREATE CONSTRAINT file_path_unique IF NOT EXISTS FOR (f:File) REQUIRE f.path IS UNIQUE",
)
real, virtual, edges = await self._counts(session)
await self._refresh_counts(session)
self.logger.info(
f"Neo4jFileGraph '{self.name}' connected at "
f"{self._uri}/{self._database}: "
f"{real} nodes, {edges} edges, {virtual} virtual",
f"{self._n_nodes} nodes, {self._n_edges} edges, {self._n_virtual} virtual",
)
async def _close(self) -> None:
@ -140,6 +143,17 @@ class Neo4jFileGraph(BaseFileGraph):
return 0, 0, 0
return int(row["real"] or 0), int(row["virtual"] or 0), int(row["edges"] or 0)
async def _refresh_counts(self, session=None) -> None:
"""Refresh cached counts for synchronous health reporting."""
if session is not None:
real, virtual, edges = await self._counts(session)
else:
async with self._session() as new_session:
real, virtual, edges = await self._counts(new_session)
self._n_nodes = real
self._n_virtual = virtual
self._n_edges = edges
# -- Node CRUD ---------------------------------------------------------
async def upsert_nodes(self, nodes: list[FileNode]) -> None:
@ -167,6 +181,7 @@ class Neo4jFileGraph(BaseFileGraph):
]
async with self._session() as session:
await session.execute_write(self._upsert_nodes_tx, payload)
await self._refresh_counts(session)
@staticmethod
async def _upsert_nodes_tx(tx, payload):
@ -209,6 +224,7 @@ class Neo4jFileGraph(BaseFileGraph):
return
async with self._session() as session:
await session.execute_write(self._delete_nodes_tx, list(paths))
await self._refresh_counts(session)
@staticmethod
async def _delete_nodes_tx(tx, paths):
@ -308,6 +324,7 @@ class Neo4jFileGraph(BaseFileGraph):
async with self._session() as session:
await session.execute_write(self._rebuild_links_tx, payload)
await self._refresh_counts(session)
@staticmethod
async def _rebuild_links_tx(tx, payload):
@ -333,6 +350,7 @@ class Neo4jFileGraph(BaseFileGraph):
"""Remove every node and edge in the configured database."""
async with self._session() as session:
await session.run("MATCH (f:File) DETACH DELETE f")
await self._refresh_counts(session)
# -- Link access -------------------------------------------------------

50
reme4/components/file_store/faiss_local_file_store.py
View file

@ -132,9 +132,18 @@ class FaissLocalFileStore(LocalFileStore):
id_map = list(data.get("id_map", []))
if len(id_map) != index.ntotal:
raise ValueError(f"id_map size {len(id_map)} != index ntotal {index.ntotal}")
tombstones = {int(row) for row in data.get("tombstones", [])}
if any(row < 0 or row >= len(id_map) for row in tombstones):
raise ValueError("FAISS tombstones contain out-of-range rows")
live_ids = [cid for i, cid in enumerate(id_map) if i not in tombstones]
if len(live_ids) != len(set(live_ids)):
raise ValueError("FAISS id_map contains duplicate live chunk ids")
expected_ids = {cid for cid, chunk in self.file_chunks.items() if chunk.embedding is not None}
if set(live_ids) != expected_ids:
raise ValueError("FAISS sidecar live ids do not match persisted chunks")
self._faiss_index = index
self._id_map = id_map
self._tombstones = set(data.get("tombstones", []))
self._tombstones = tombstones
self._id_to_row = {cid: i for i, cid in enumerate(self._id_map) if i not in self._tombstones}
self.logger.info(f"Loaded FAISS index: {index.ntotal} vectors from {self.faiss_path}")
return True
@ -175,19 +184,25 @@ class FaissLocalFileStore(LocalFileStore):
assert self.file_graph is not None
# Snapshot pre-upsert chunk_ids so we can diff against the post-upsert state.
old_ids_by_path = {
n.path: set(n.chunk_ids) for n in await self.file_graph.get_nodes([node.path for node, _ in files])
old_nodes = await self.file_graph.get_nodes([node.path for node, _ in files])
old_ids_by_path = {n.path: set(n.chunk_ids) for n in old_nodes}
old_text_by_id = {
cid: chunk.text
for n in old_nodes
for cid in n.chunk_ids
if (chunk := self.file_chunks.get(cid)) is not None
}
await super().upsert(files)
if self._faiss_index is None or self.embedding_store is None:
return
self._sync_index_after_upsert(files, old_ids_by_path)
self._sync_index_after_upsert(files, old_ids_by_path, old_text_by_id)
def _sync_index_after_upsert(
self,
files: list[tuple[FileNode, list[FileChunk]]],
old_ids_by_path: dict[str, set[str]],
old_text_by_id: dict[str, str],
) -> None:
"""Apply add/tombstone deltas to FAISS based on chunk_id set differences."""
existing = set(self._id_to_row)
@ -196,9 +211,15 @@ class FaissLocalFileStore(LocalFileStore):
new_ids = set(node.chunk_ids)
for cid in old_ids_by_path.get(node.path, set()) - new_ids:
self._tombstone(cid)
for cid in new_ids - existing:
for cid in new_ids:
chunk = self.file_chunks.get(cid)
if chunk is not None and chunk.embedding is not None:
if chunk is None or chunk.embedding is None:
continue
if cid in existing and old_text_by_id.get(cid) == chunk.text:
continue
if cid in existing:
self._tombstone(cid)
if cid not in existing or old_text_by_id.get(cid) != chunk.text:
to_add.append(chunk)
if to_add:
@ -245,11 +266,20 @@ class FaissLocalFileStore(LocalFileStore):
# Over-fetch by len(tombstones) so dropped rows can't starve the result set.
q = self._prepare(query_embedding)
k = min(self._faiss_index.ntotal, limit + len(self._tombstones))
if search_filter:
k = self._faiss_index.ntotal
else:
k = min(self._faiss_index.ntotal, limit + len(self._tombstones))
scores, rows = self._faiss_index.search(q, k)
return self._collect_hits(rows[0].tolist(), scores[0].tolist(), limit)
return self._collect_hits(rows[0].tolist(), scores[0].tolist(), limit, search_filter)
def _collect_hits(self, rows: list[int], scores: list[float], limit: int) -> list[FileChunk]:
def _collect_hits(
self,
rows: list[int],
scores: list[float],
limit: int,
search_filter: dict | None = None,
) -> list[FileChunk]:
"""Map raw FAISS rows back to chunks, skipping tombstones and stale ids."""
results: list[FileChunk] = []
for raw_row, score in zip(rows, scores):
@ -257,7 +287,7 @@ class FaissLocalFileStore(LocalFileStore):
if row < 0 or row in self._tombstones or row >= len(self._id_map):
continue
chunk = self.file_chunks.get(self._id_map[row])
if chunk is None:
if chunk is None or not self._matches_search_filter(chunk, search_filter):
continue
results.append(chunk.model_copy(update={"scores": {"vector": float(score), "score": float(score)}}))
if len(results) >= limit:

108
reme4/components/file_store/local_file_store.py
View file

@ -1,6 +1,5 @@
"""In-memory file store with JSONL persistence on close."""
"""In-memory file store with compressed JSONL persistence on close."""
import aiofiles
import numpy as np
from .base_file_store import BaseFileStore
@ -11,6 +10,9 @@ from ..keyword_index import BaseKeywordIndex
from ...enumeration import LinkScopeEnum
from ...schema import FileChunk, FileLink, FileNode
from ...utils import batch_cosine_similarity
from ...utils.jsonl_zst import read_jsonl_zst, write_jsonl_zst
CachedEmbedding = tuple[str, np.ndarray]
@R.register("local")
@ -49,7 +51,7 @@ class LocalFileStore(BaseFileStore):
self.encoding = encoding
self.store_version = store_version
self.file_chunks: dict[str, FileChunk] = {}
self.chunks_path = self.component_metadata_path / f"file_chunks_{self.name}_{self.store_version}.jsonl"
self.chunks_path = self.component_metadata_path / f"file_chunks_{self.name}_{self.store_version}.jsonl.zst"
# -- lifecycle ------------------------------------------------------------
@ -80,12 +82,11 @@ class LocalFileStore(BaseFileStore):
if not self.chunks_path.exists():
return
try:
async with aiofiles.open(self.chunks_path, encoding=self.encoding) as f:
async for line in f:
line = line.strip()
if line:
chunk = FileChunk.model_validate_json(line)
self.file_chunks[chunk.id] = chunk
for line in read_jsonl_zst(self.chunks_path, self.encoding):
line = line.strip()
if line:
chunk = FileChunk.model_validate_json(line)
self.file_chunks[chunk.id] = chunk
self.logger.info(f"Loaded {len(self.file_chunks)} chunks from {self.chunks_path}")
except Exception as e:
self.logger.exception(f"Failed to load {self.chunks_path}: {e}")
@ -94,10 +95,7 @@ class LocalFileStore(BaseFileStore):
"""Atomically rewrite the JSONL, then cascade dump into keyword_index and file_graph."""
assert self.file_graph is not None
try:
tmp = self.chunks_path.with_suffix(".tmp")
async with aiofiles.open(tmp, "w", encoding=self.encoding) as f:
await f.write("\n".join(c.model_dump_json() for c in self.file_chunks.values()))
tmp.replace(self.chunks_path)
write_jsonl_zst(self.chunks_path, (c.model_dump_json() for c in self.file_chunks.values()), self.encoding)
self.logger.info(f"Saved {len(self.file_chunks)} chunks to {self.chunks_path}")
except Exception as e:
self.logger.exception(f"Failed to write {self.chunks_path}: {e}")
@ -113,10 +111,13 @@ class LocalFileStore(BaseFileStore):
assert self.file_graph is not None
old_map = {n.path: n for n in await self.file_graph.get_nodes([node.path for node, _ in files])}
old_chunk_ids = {cid for n in old_map.values() for cid in n.chunk_ids}
new_nodes, needs_embed, keyword_docs = self._stage_upsert(files, old_map)
await self.file_graph.upsert_nodes(new_nodes)
await self._embed_pending(needs_embed)
if self.keyword_index and old_chunk_ids:
await self.keyword_index.delete_docs(list(old_chunk_ids))
if self.keyword_index and keyword_docs:
await self.keyword_index.add_docs(keyword_docs)
@ -143,29 +144,29 @@ class LocalFileStore(BaseFileStore):
new_nodes.append(node)
return new_nodes, needs_embed, keyword_docs
def _evict_prior_chunks(self, old_node: FileNode | None) -> dict[str, np.ndarray]:
def _evict_prior_chunks(self, old_node: FileNode | None) -> dict[str, CachedEmbedding]:
"""Drop chunks for the path being re-upserted; keep their embeddings around so
a new chunk reusing the same id avoids a redundant embedding call.
a new chunk reusing the same id and text avoids a redundant embedding call.
"""
cached: dict[str, np.ndarray] = {}
if not (old_node and self.embedding_store):
cached: dict[str, CachedEmbedding] = {}
if old_node is None:
return cached
for cid in old_node.chunk_ids:
old = self.file_chunks.pop(cid, None)
if old and old.embedding is not None:
cached[cid] = old.embedding
if self.embedding_store and old and old.embedding is not None:
cached[cid] = (old.text, old.embedding)
return cached
def _reuse_or_queue_embedding(
self,
chunk: FileChunk,
cached: dict[str, np.ndarray],
cached: dict[str, CachedEmbedding],
needs_embed: list[FileChunk],
) -> None:
if not self.embedding_store or chunk.embedding is not None:
return
if chunk.id in cached:
chunk.embedding = cached[chunk.id]
if chunk.id in cached and cached[chunk.id][0] == chunk.text:
chunk.embedding = cached[chunk.id][1]
elif chunk.text:
needs_embed.append(chunk)
@ -232,7 +233,11 @@ class LocalFileStore(BaseFileStore):
if query_embedding is None:
return []
candidates = [c for c in self.file_chunks.values() if c.embedding is not None]
candidates = [
c
for c in self.file_chunks.values()
if c.embedding is not None and self._matches_search_filter(c, search_filter)
]
if not candidates:
return []
@ -254,17 +259,70 @@ class LocalFileStore(BaseFileStore):
if not query:
return []
doc_id_score_dict = await self.keyword_index.retrieve(query, limit=limit)
retrieve_limit = limit
if search_filter:
retrieve_limit = max(limit, getattr(self.keyword_index, "n_docs", limit))
doc_id_score_dict = await self.keyword_index.retrieve(query, limit=retrieve_limit)
results = []
for doc_id, score in doc_id_score_dict.items():
chunk = self.file_chunks.get(doc_id)
if chunk:
if chunk and self._matches_search_filter(chunk, search_filter):
results.append(chunk.model_copy(update={"scores": {"keyword": score, "score": score}}))
if len(results) >= limit:
break
return results
# -- extensions -----------------------------------------------------------
@staticmethod
def _as_filter_values(value) -> set:
if isinstance(value, (list, tuple, set, frozenset)):
return set(value)
return {value}
@classmethod
def _value_matches(cls, actual, expected) -> bool:
if isinstance(expected, (list, tuple, set, frozenset)):
return actual in set(expected)
return actual == expected
@classmethod
def _matches_search_filter(cls, chunk: FileChunk, search_filter: dict | None) -> bool:
"""Conservative post-filter shared by vector and keyword search."""
if not search_filter:
return True
exact_paths = set()
for key in ("path", "paths"):
if key in search_filter:
exact_paths.update(cls._as_filter_values(search_filter[key]))
if exact_paths and chunk.path not in exact_paths:
return False
prefixes = []
for key in ("path_prefix", "path_prefixes", "prefix", "prefixes"):
if key in search_filter:
prefixes.extend(str(v) for v in cls._as_filter_values(search_filter[key]))
if prefixes and not any(chunk.path.startswith(prefix) for prefix in prefixes):
return False
metadata_filter = dict(search_filter.get("metadata") or {})
reserved = {
"path",
"paths",
"path_prefix",
"path_prefixes",
"prefix",
"prefixes",
"metadata",
}
for key, value in search_filter.items():
if key not in reserved:
metadata_filter[key] = value
return all(cls._value_matches(chunk.metadata.get(key), value) for key, value in metadata_filter.items())
async def rebuild_links(self) -> None:
"""Rebuild graph links via the underlying file graph."""
assert self.file_graph is not None

3
reme4/components/job/base_job.py
View file

@ -59,7 +59,8 @@ class BaseJob(BaseComponent):
async def __call__(self, **kwargs) -> Response:
"""Run all steps in order, capturing any failure into the response."""
context = RuntimeContext(**kwargs)
merged = {**self.kwargs, **kwargs}
context = RuntimeContext(**merged)
try:
for step in self._build_steps():
await step(context)

217
reme4/components/job/cron_job.py
View file

@ -1,212 +1,55 @@
"""``cron`` — a supervised background job that periodically dispatches
downstream job(s) and/or step(s) on a schedule.
A ``CronJob`` is a :class:`BackgroundJob` whose body is a scheduler loop
rather than a fixed list of steps: it sleeps until the next fire time,
dispatches its configured downstream job(s) and/or step(s), and loops.
It exits when ``self._stop_event`` is set, so the surrounding
``Application`` shutdown / supervisor can stop it cleanly.
Declared directly under ``jobs:`` in YAML (no step wrapper)::
auto_dream_cron:
backend: cron
dispatch_job: auto_dream
cron: "0 3 * * *" # daily at 03:00; "0 */6 * * *" = every 6h
run_on_start: false
Two dispatch modes — pick one (or both, executed in order: jobs first,
then steps):
* ``dispatch_job`` / ``dispatch_jobs`` — invoke a **registered job** by
name through ``self.app_context.jobs``. Goes through the job's own
parameter validation and context construction, identical to a CLI /
MCP invocation. Prefer this when the periodic task already has a job
wrapping it (e.g. ``auto_dream``).
* ``dispatch_step`` / ``dispatch_steps`` — instantiate a **registered
step** by name and invoke it directly. Use for steps that don't have a
job wrapper.
Three schedule modes (mutually exclusive — exactly one must be set):
* ``cron: "M H DoM Mon DoW"`` — standard 5-field cron expression in
``app_config.timezone``. Most flexible; use for non-daily cadence
(``"0 */6 * * *"`` = every 6 hours, ``"0 3 * * 1-5"`` = 3am on
weekdays, ``"*/15 * * * *"`` = every 15 minutes).
* ``daily_at: "HH:MM"`` — fire once per day at this wall-clock time, in
``app_config.timezone``. Convenience shorthand for ``"M H * * *"``.
* ``interval_seconds: int`` — fire every N seconds since launch. Useful
for tests and for time-zone-independent sub-minute cadence.
Per-dispatch exceptions are caught and logged — a failed downstream job
or step never kills the cron loop (so the ``BackgroundJob`` supervisor is
reserved for genuine scheduler-loop crashes). ``run_on_start: true``
triggers one immediate dispatch on launch (default ``false``, to avoid a
burst when ``Application.start`` brings several cron jobs up at once).
Cron expressions are evaluated via ``croniter`` and validated eagerly at
construction time, so a typo fails at app start rather than at 3 a.m.
"""
"""Cron-scheduled background job that runs its configured steps."""
import datetime
import zoneinfo
from croniter import croniter
from zoneinfo import ZoneInfo
from .background_job import BackgroundJob
from ..component_registry import R
from ...enumeration import ComponentEnum
from ..runtime_context import RuntimeContext
from ...schema import Response
@R.register("cron")
class CronJob(BackgroundJob):
"""Dispatch downstream step(s) and/or job(s) on a schedule until ``stop_event`` fires."""
"""Run this job's own steps on a cron expression."""
def __init__(
self,
dispatch_step: str = "",
dispatch_steps: list[str] | None = None,
dispatch_job: str = "",
dispatch_jobs: list[str] | None = None,
cron: str = "",
daily_at: str = "",
interval_seconds: int = 0,
run_on_start: bool = False,
**kwargs,
):
def __init__(self, cron: str, **kwargs):
super().__init__(**kwargs)
self.dispatch_steps: list[str] = dispatch_steps or ([dispatch_step] if dispatch_step else [])
self.dispatch_jobs: list[str] = dispatch_jobs or ([dispatch_job] if dispatch_job else [])
self.cron: str = cron
self.daily_at: str = daily_at
self.interval_seconds: int = interval_seconds
self.run_on_start: bool = run_on_start
self.cron_expr = cron
if not self.dispatch_steps and not self.dispatch_jobs:
raise ValueError(
"cron job requires at least one of 'dispatch_step'/'dispatch_steps' "
"or 'dispatch_job'/'dispatch_jobs'",
)
async def _start(self) -> None:
from croniter import croniter
schedules_set = sum(bool(x) for x in (self.cron, self.daily_at, self.interval_seconds))
if schedules_set != 1:
raise ValueError(
"cron job requires exactly one of "
"'cron' (5-field expression), 'daily_at' (HH:MM), "
"or 'interval_seconds' (int)",
)
if self.daily_at:
# Validate HH:MM format eagerly so misconfig fails at start, not at 3am.
h, m = self._parse_hh_mm(self.daily_at)
self._fire_hour, self._fire_minute = h, m
elif self.cron:
# Fail at start, not at the next scheduled tick.
if not croniter.is_valid(self.cron):
raise ValueError(f"cron expression invalid, got {self.cron!r}")
@staticmethod
def _parse_hh_mm(value: str) -> tuple[int, int]:
try:
h_str, m_str = value.split(":", 1)
h, m = int(h_str), int(m_str)
except (ValueError, AttributeError) as exc:
raise ValueError(f"daily_at must be 'HH:MM', got {value!r}") from exc
if not (0 <= h < 24 and 0 <= m < 60):
raise ValueError(f"daily_at out of range, got {value!r}")
return h, m
def _tz(self) -> datetime.tzinfo | None:
if not self.app_context:
return None
tz_name = self.app_context.app_config.timezone
if not tz_name:
return None
try:
return zoneinfo.ZoneInfo(tz_name)
except zoneinfo.ZoneInfoNotFoundError:
self.logger.warning(f"[{self.name}] unknown timezone {tz_name!r}; using local time")
return None
if not croniter.is_valid(self.cron_expr):
raise ValueError(f"Invalid cron expression: {self.cron_expr}")
await super()._start()
def _next_fire_delay(self) -> float:
"""Seconds from now until the next fire — never negative, never zero."""
if self.interval_seconds:
return float(self.interval_seconds)
tz = self._tz()
now = datetime.datetime.now(tz)
if self.cron:
# croniter requires an explicit base time; tz comes through on `now`.
nxt = croniter(self.cron, now).get_next(datetime.datetime)
return (nxt - now).total_seconds()
# daily_at
target = now.replace(hour=self._fire_hour, minute=self._fire_minute, second=0, microsecond=0)
if target <= now:
target = target + datetime.timedelta(days=1)
return (target - now).total_seconds()
tz_name = None
if self.app_context is not None:
tz_name = self.app_context.app_config.timezone
now = datetime.datetime.now(ZoneInfo(tz_name)) if tz_name else datetime.datetime.now()
from croniter import croniter
async def _fire(self, dispatch_classes: list[type]) -> None:
"""Dispatch each downstream job (via the registry) and step (class-level), in that
order; swallow exceptions per-dispatch so a single failure doesn't break the loop."""
# Jobs first — they're the higher-level invocation path (matches CLI / MCP).
for name in self.dispatch_jobs:
try:
job = self.app_context.jobs.get(name) if self.app_context else None
if job is None:
raise RuntimeError(f"Job {name!r} not found")
await job()
self.logger.info(f"[{self.name}] dispatched job {name!r}")
except Exception as exc:
self.logger.exception(f"[{self.name}] dispatch job {name!r} raised: {exc}")
nxt = croniter(self.cron_expr, now).get_next(datetime.datetime)
return max(0.0, (nxt - now).total_seconds())
for cls in dispatch_classes:
try:
s = cls(app_context=self.app_context)
await s()
self.logger.info(f"[{self.name}] dispatched step {cls.__name__}")
except Exception as exc:
self.logger.exception(f"[{self.name}] dispatch step {cls.__name__} raised: {exc}")
async def _execute_steps(self) -> Response:
context = RuntimeContext(**self.kwargs)
for step in self._build_steps():
await step(context)
return context.response
async def __call__(self, **kwargs) -> Response:
"""Scheduler body: loop dispatching downstream jobs/steps until ``stop_event`` fires.
Per-dispatch exceptions are swallowed in ``_fire``, so this body only propagates
genuine scheduler-loop failures to the ``BackgroundJob`` supervisor for restart.
"""
assert self._stop_event is not None
stop_event = self._stop_event
dispatch_classes: list[type] = []
for name in self.dispatch_steps:
cls = R.get(ComponentEnum.STEP, name)
if cls is None:
raise RuntimeError(f"Unregistered step '{name}'")
dispatch_classes.append(cls)
if self.cron:
mode = f"cron={self.cron!r}"
elif self.daily_at:
mode = f"daily_at={self.daily_at}"
else:
mode = f"interval_seconds={self.interval_seconds}"
self.logger.info(
f"[{self.name}] cron loop start "
f"dispatch_jobs={self.dispatch_jobs} dispatch_steps={self.dispatch_steps} "
f"{mode} run_on_start={self.run_on_start}",
)
if self.run_on_start and not stop_event.is_set():
await self._fire(dispatch_classes)
while not stop_event.is_set():
delay = self._next_fire_delay()
self.logger.info(f"[{self.name}] next fire in {delay:.0f}s")
await self._wait_or_stop(delay)
if stop_event.is_set():
while not self._stop_event.is_set():
await self._wait_or_stop(self._next_fire_delay())
if self._stop_event.is_set():
break
await self._fire(dispatch_classes)
try:
await self._execute_steps()
except Exception as exc:
self.logger.exception(f"Cron job '{self.name}' failed: {exc}")
response = Response()
response.success = True
response.answer = f"cron job jobs={self.dispatch_jobs!r} steps={self.dispatch_steps!r} stopped"
return response

3
reme4/components/job/stream_job.py
View file

@ -12,7 +12,8 @@ class StreamJob(BaseJob):
async def __call__(self, **kwargs) -> None:
"""Run steps; emit failures as ERROR chunks, then a terminal DONE marker."""
context = RuntimeContext(**kwargs)
merged = {**self.kwargs, **kwargs}
context = RuntimeContext(**merged)
try:
for step in self._build_steps():
await step(context)

50
reme4/components/keyword_index/bm25_index.py
View file

@ -15,8 +15,11 @@ lists keep the stale entries until `optimize_index` rewrites them. Updating an
existing doc_id retires the old slot first, then allocates a fresh idx.
"""
import hashlib
import json
import math
import pickle
import re
from collections import Counter
from pathlib import Path
@ -53,11 +56,43 @@ class BM25Index(BaseKeywordIndex):
@property
def index_file(self) -> Path:
"""Path of the persisted index, namespaced by tokenizer and version."""
"""Path of the persisted index, namespaced by component/tokenizer config."""
if self.tokenizer is None:
raise RuntimeError("Tokenizer not initialized. Call start() first.")
name = type(self.tokenizer).__name__.replace("Tokenizer", "").lower()
return self.component_metadata_path / f"bm25_{name}_{self.index_version}.pkl"
component_name = self._safe_filename_part(self.name)
fingerprint = self._tokenizer_fingerprint()
return self.component_metadata_path / f"bm25_{component_name}_{name}_{fingerprint}_{self.index_version}.pkl"
@staticmethod
def _safe_filename_part(value: str) -> str:
"""Make a short, stable filename segment from a component/config value."""
safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._")
return safe or "default"
def _tokenizer_config(self) -> dict:
"""Return the tokenizer settings that affect token output."""
if self.tokenizer is None:
raise RuntimeError("Tokenizer not initialized. Call start() first.")
config = {
"class": type(self.tokenizer).__qualname__,
"filter_stopwords": getattr(self.tokenizer, "filter_stopwords", None),
}
stopwords_path = getattr(self.tokenizer, "stopwords_path", None)
if stopwords_path is not None:
path = Path(stopwords_path)
config["stopwords_path"] = str(path)
if path.exists() and path.is_file():
config["stopwords_sha256"] = hashlib.sha256(path.read_bytes()).hexdigest()
else:
config["stopwords_sha256"] = None
return config
def _tokenizer_fingerprint(self) -> str:
"""Compact digest for tokenizer settings used in the index filename."""
payload = json.dumps(self._tokenizer_config(), sort_keys=True, default=str)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
@property
def n_docs(self) -> int:
@ -276,6 +311,8 @@ class BM25Index(BaseKeywordIndex):
def _snapshot(self) -> dict:
"""Bundle every persistent field; mirrors `_restore`."""
return {
"tokenizer_config": self._tokenizer_config(),
"tokenizer_fingerprint": self._tokenizer_fingerprint(),
"vocab": self.vocab,
"doc_ids": self._doc_ids,
"doc_id_to_idx": self._doc_id_to_idx,
@ -290,6 +327,10 @@ class BM25Index(BaseKeywordIndex):
def _restore(self, data: dict) -> None:
"""Restore index state from a `_snapshot` dict."""
expected = self._tokenizer_fingerprint()
actual = data.get("tokenizer_fingerprint")
if actual is not None and actual != expected:
raise ValueError(f"Tokenizer fingerprint mismatch: expected {expected}, got {actual}")
self.vocab = data["vocab"]
self._doc_ids = data["doc_ids"]
self._doc_id_to_idx = data["doc_id_to_idx"]
@ -304,7 +345,11 @@ class BM25Index(BaseKeywordIndex):
async def dump(self) -> None:
"""Persist the index via temp file + atomic rename to avoid torn writes."""
if self.n_docs == 0 and not self.vocab:
self.index_file.unlink(missing_ok=True)
return
try:
self.index_file.parent.mkdir(parents=True, exist_ok=True)
tmp = self.index_file.with_suffix(".tmp")
with open(tmp, "wb") as f:
pickle.dump(self._snapshot(), f)
@ -312,6 +357,7 @@ class BM25Index(BaseKeywordIndex):
self.logger.info(f"Saved {self.n_docs} docs to {self.index_file}")
except Exception as e:
self.logger.exception(f"Failed to write {self.index_file}: {e}")
raise
async def load(self) -> None:
"""Load from disk; missing file is a no-op, corrupt file resets state."""

15
reme4/components/prompt_handler.py
View file

@ -4,7 +4,6 @@ import inspect
import json
import re
from pathlib import Path
from string import Formatter
import yaml
@ -108,11 +107,10 @@ class PromptHandler:
# ----- Formatting ----------------------------------------------------
def prompt_format(self, prompt_name: str, validate: bool = True, **kwargs) -> str:
def prompt_format(self, prompt_name: str, **kwargs) -> str:
"""Render a prompt: strip inactive ``[flag]`` lines, then ``str.format`` it.
Boolean kwargs are treated as flag toggles; the rest are format variables.
With ``validate=True``, any missing ``{var}`` placeholder raises ``ValueError``.
"""
prompt = self.get_prompt(prompt_name)
flags = {k: v for k, v in kwargs.items() if isinstance(v, bool)}
@ -120,8 +118,6 @@ class PromptHandler:
if flags:
prompt = self._apply_flag_filter(prompt, flags)
if validate:
self._check_required_vars(prompt, formats, prompt_name)
return prompt.format(**formats).strip() if formats else prompt
@ -136,14 +132,5 @@ class PromptHandler:
lines.append(cleaned)
return "\n".join(lines)
@staticmethod
def _check_required_vars(prompt: str, formats: dict, prompt_name: str) -> None:
"""Raise when any ``{var}`` placeholder lacks a corresponding kwarg."""
required = {f for _, f, _, _ in Formatter().parse(prompt) if f is not None}
if missing := required - set(formats.keys()):
raise ValueError(
f"Missing format variables for '{prompt_name}': {sorted(missing)}",
)
def __repr__(self) -> str:
return f"PromptHandler(language='{self.language}', num_prompts={len(self.data)})"

14
reme4/components/service/base_service.py
View file

@ -32,8 +32,12 @@ class BaseService(BaseComponent):
"""Instantiate and configure the underlying server framework."""
@abstractmethod
def add_job(self, job: BaseJob) -> None:
"""Register a single job as a callable endpoint or tool."""
def add_job(self, job: BaseJob) -> bool:
"""Register a single job as a callable endpoint or tool.
Returns True when the job is exposed, False when the service intentionally
skips it (for example, unsupported job types).
"""
@abstractmethod
def start_service(self, app: "Application") -> None:
@ -65,8 +69,10 @@ class BaseService(BaseComponent):
if not job.enable_serve:
continue
try:
self.add_job(job)
self.logger.info(f"Added job: {name}")
if self.add_job(job):
self.logger.info(f"Added job: {name}")
else:
self.logger.warning(f"Skipped job: {name}")
except Exception as e:
self.logger.error(f"Failed to add job {name}: {e}")

3
reme4/components/service/http_service.py
View file

@ -55,12 +55,13 @@ class HttpService(BaseService):
allow_headers=["*"],
)
def add_job(self, job: BaseJob) -> None:
def add_job(self, job: BaseJob) -> bool:
"""Dispatch to streaming or non-streaming registration based on job type."""
if isinstance(job, StreamJob):
self._add_stream_job(job)
else:
self._add_json_job(job)
return True
def start_service(self, app: "Application") -> None:
"""Run uvicorn, suppressing unrelated websocket deprecation noise."""

7
reme4/components/service/mcp_service.py
View file

@ -136,10 +136,10 @@ class MCPService(BaseService):
lifespan=self._lifespan(app, self.host, self.port),
)
def add_job(self, job: BaseJob) -> None:
def add_job(self, job: BaseJob) -> bool:
"""Register a non-stream job as an MCP tool; StreamJobs are unsupported."""
if isinstance(job, StreamJob):
return
return False
async def execute_tool(**kwargs):
response = await job(**kwargs)
@ -150,9 +150,10 @@ class MCPService(BaseService):
name=job.name,
description=job.description,
fn=execute_tool,
parameters=job.parameters or None,
parameters=job.parameters or {},
),
)
return True
def start_service(self, app: "Application") -> None:
"""Run the MCP server; bind host/port only for network transports."""

2
reme4/components/tokenizer/jieba_tokenizer.py
View file

@ -40,4 +40,6 @@ class JiebaTokenizer(BaseTokenizer):
self.logger.info(f"JiebaTokenizer using backend: {self.backend}")
def _tokenize_one(self, text: str, **kwargs) -> list[str]:
if self._cut is None:
raise RuntimeError("Tokenizer not initialized. Call start() first.")
return list(self._cut(text))

19
reme4/config/config_parser.py
View file

@ -32,7 +32,8 @@ def _repl(m: re.Match) -> str:
def _expand_env_vars(value: Any) -> Any:
"""Recursively expand `${VAR}` / `${VAR:-default}` placeholders in strings."""
if isinstance(value, str):
return _ENV_VAR_RE.sub(_repl, value)
expanded = _ENV_VAR_RE.sub(_repl, value)
return _convert_value(expanded) if expanded != value else value
if isinstance(value, dict):
return {k: _expand_env_vars(v) for k, v in value.items()}
if isinstance(value, list):
@ -65,6 +66,8 @@ def parse_dot_notation(dot_list: list[str]) -> dict:
raise ValueError(f"Invalid dot notation format (missing '='): {item}")
key_path, value_str = item.split("=", 1)
keys = key_path.split(".")
if not key_path or any(not key for key in keys):
raise ValueError(f"Invalid dot notation key: {key_path!r}")
current = result
for key in keys[:-1]:
if key in current and not isinstance(current[key], dict):
@ -127,9 +130,13 @@ def _load_config(name_or_path: str, encoding: str = "utf-8") -> dict:
# 2. Treat as file path
p = Path(name_or_path)
if p.suffix in _SUPPORTED_EXTS:
if not p.exists():
raise FileNotFoundError(f"Config file not found: {p}")
return _read_config_file(p, encoding)
candidates = [p]
if not p.is_absolute():
candidates.append(_CONFIG_DIR / p)
for candidate in candidates:
if candidate.exists():
return _read_config_file(candidate, encoding)
raise FileNotFoundError(f"Config file not found: {p}")
known = ", ".join(sorted(_CONFIG_REGISTRY)) if _CONFIG_REGISTRY else "none"
raise FileNotFoundError(f"Config file not found: {name_or_path}. Available: {known}")
@ -144,6 +151,8 @@ def _read_config_file(path: Path, encoding: str = "utf-8") -> dict:
result = yaml.safe_load(f)
if result is None:
return {}
if not isinstance(result, dict):
raise ValueError(f"Config root must be a mapping/object: {path}")
return _expand_env_vars(result)
@ -185,6 +194,8 @@ def parse_args(*args) -> tuple[str, dict]:
arg = _strip_arg_dashes(raw)
if "=" in arg:
kvs.append(arg)
else:
raise ValueError(f"Invalid argument format (expected key=value): {raw}")
parsed = parse_dot_notation(kvs) if kvs else {}
return first, parsed

185
reme4/config/default.yaml
View file

@ -1,58 +1,56 @@
vault_dir: .reme
daily_dir: daily
digest_dir: digest
resource_dir: resource
# language: zh
service:
backend: http
jobs:
index_update_loop:
backend: background
watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_suffixes: [md, jsonl]
# watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_dirs: [daily_dir, digest_dir]
# watch_suffixes: [md, jsonl]
watch_suffixes: [md]
steps:
- backend: scan_store_changes_step
recursive: true
- backend: update_index_step
persist: true
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
- backend: watch_changes_step
dispatch_step: update_index_step
dispatch_steps: [update_index_step]
resource_watch_loop:
backend: background
watch_dirs: [resource_dir]
watch_suffixes: [md, txt, json, jsonl, csv, yaml, html]
dispatch_job: auto_resource
persist: true
steps:
- backend: scan_catalog_changes_step
recursive: true
- backend: update_catalog_step
persist: true
- backend: foreach_dispatch_step
- backend: init_changes_step
monitor_type: file_catalog
monitor_name: resource
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
- backend: watch_changes_step
dispatch_steps: [update_catalog_step, foreach_dispatch_step]
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
digest_watch_loop:
backend: background
watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [md]
persist: true
steps:
- backend: scan_catalog_changes_step
recursive: true
- backend: update_catalog_step
persist: true
- backend: init_changes_step
monitor_type: file_catalog
monitor_name: digest
dispatch_steps:
- backend: update_catalog_step
file_catalog: digest
- backend: log_changes_step
- backend: watch_changes_step
dispatch_step: log_changes_step
# auto_dream_cron:
# backend: cron
# dispatch_job: auto_dream
# cron: "0 3 * * *" # daily at 03:00; "0 */6 * * *" = every 6h
# run_on_start: false
dispatch_steps:
- backend: update_catalog_step
file_catalog: digest
- backend: log_changes_step
version:
backend: base
@ -109,13 +107,17 @@ jobs:
reindex:
backend: base
description: "wipe the file store and rebuild it from the existing files"
watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_suffixes: [md, jsonl]
parameters:
type: object
properties: { }
steps:
- backend: clear_and_scan_step
- backend: update_index_step
persist: true
- backend: clear_store_step
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
search:
backend: base
@ -424,27 +426,9 @@ jobs:
steps:
- backend: edit_step
dream:
backend: base
description: "Dream: lift atomic units from one daily/resource file into digest/ (LLM)."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path of one daily-event note or resource file"
hint:
type: string
description: "caller guidance to the dreamer LLM"
default: ""
required:
- path
steps:
- backend: dream_step
auto_dream:
backend: base
description: "Auto-dream: scan today's day-index <daily_dir>/<today>.md and session notes under <daily_dir>/<today>/*.md — run dream on each (Phase 1 extract+classify, Phase 2 per-bucket integrate)."
description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog."
parameters:
type: object
properties:
@ -454,10 +438,43 @@ jobs:
default: ""
hint:
type: string
description: "caller guidance passed through to each per-file dream"
description: "caller guidance passed through to dream extract/integrate"
default: ""
topic_count:
type: integer
description: "maximum number of final daily interest topics"
default: 3
topic_diversity_days:
type: integer
description: "number of previous interests.yaml days to avoid repeating"
default: 7
steps:
- backend: auto_dream_step
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
topic_diversity_days: 7
- backend: dream_finish_step
file_catalog: dream
proactive:
backend: base
description: "Proactive: read daily/<date>/interests.yaml and expose the latest user-interest topics."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to read; defaults to today in the dreamer's timezone"
default: ""
include_content:
type: boolean
description: "whether to include the raw YAML content in response metadata"
default: true
steps:
- backend: proactive_step
auto_memory:
backend: base
@ -488,15 +505,21 @@ jobs:
parameters:
type: object
properties:
file_path:
type: string
description: "vault-relative resource file path, e.g. resource/2026-06-06/report.pdf"
change:
type: string
description: "added/modified/deleted"
changes:
type: array
description: "resource change batch, each item has path/file_path and change"
items:
type: object
properties:
path:
type: string
file_path:
type: string
change:
type: string
description: "added/modified/deleted"
required:
- file_path
- change
- changes
steps:
- backend: auto_resource_step
@ -508,10 +531,12 @@ components:
as_embedding:
default:
backend: ${EMBEDDING_BACKEND:-openai}
api_key: ${EMBEDDING_API_KEY:-}
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
model_name: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
dimensions: 1024
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
credential:
api_key: ${EMBEDDING_API_KEY:-}
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
parameters:
dimensions: 1024
embedding_store:
default:
@ -520,17 +545,17 @@ components:
as_llm:
default:
backend: ${LLM_BACKEND:-anthropic}
model: ${LLM_MODEL_NAME:-glm-5.1}
backend: ${LLM_BACKEND:-openai}
model: ${LLM_MODEL_NAME:-qwen3.7-plus}
stream: true
context_size: 200000
max_retries: 3
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: true
thinking_enable: false
agent_wrapper:
default:
@ -542,15 +567,15 @@ components:
context_config:
trigger_ratio: 0.8
reserve_ratio: 0.1
tool_result_limit: 3000
tool_result_limit: 50000
model_config:
max_retries: 1
claude_code:
backend: claude_code
model: ${LLM_MODEL_NAME:-claude-opus-4-6}
system_prompt: "You are a helpful assistant."
model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.1}
api_key: ${CLAUDE_CODE_API_KEY:-}
base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic}
permission_mode: bypassPermissions
max_turns: 50
file_graph:
default:
@ -559,6 +584,12 @@ components:
file_catalog:
default:
backend: local
resource:
backend: local
digest:
backend: local
dream:
backend: local
file_chunker:
markdown:

18
reme4/config/demo.yaml
View file

@ -21,6 +21,24 @@ jobs:
- backend: demo_echo_step1
- backend: demo_echo_step2
add:
backend: base
description: "add two numbers"
parameters:
type: object
properties:
a:
type: number
description: "first addend"
b:
type: number
description: "second addend"
required:
- a
- b
steps:
- backend: add_step
stream_demo:
backend: stream
description: "stream demo job: repeat query 10x and stream char-by-char"

553
reme4/config/qwenpaw.yaml
View file

@ -0,0 +1,553 @@
service:
backend: http
jobs:
index_update_loop:
backend: background
# watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_dirs: [daily_dir, digest_dir]
# watch_suffixes: [md, jsonl]
watch_suffixes: [md]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
- backend: watch_changes_step
dispatch_steps: [update_index_step]
resource_watch_loop:
backend: background
watch_dirs: [resource_dir]
watch_suffixes: [md, txt, json, jsonl, csv, yaml, html]
steps:
- backend: init_changes_step
monitor_type: file_catalog
monitor_name: resource
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
- backend: watch_changes_step
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
version:
backend: base
description: "return reme package version"
parameters:
type: object
properties: { }
steps:
- backend: version_step
reindex:
backend: base
description: "wipe the file store and rebuild it from the existing files"
watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_suffixes: [md, jsonl]
parameters:
type: object
properties: { }
steps:
- backend: clear_store_step
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
search:
backend: base
description: "Hybrid vault search (vector + BM25, RRF-fused)."
parameters:
type: object
properties:
query:
type: string
description: "search query"
limit:
type: integer
description: "max results"
default: 5
min_score:
type: number
description: "min fused score"
default: 0.0
required:
- query
steps:
- backend: search_step
vector_weight: 0.7
candidate_multiplier: 3.0
expand_links: true
max_links_per_direction: 10
node_search:
backend: base
description: "Digest node recall — given a candidate abstraction's name+description, surface existing digest nodes similar enough to either dedup against or link to as related."
parameters:
type: object
properties:
query:
type: string
description: "search query"
limit:
type: integer
description: "max digest nodes to return"
default: 20
required:
- query
steps:
- backend: node_search_step
vector_weight: 0.7
candidate_multiplier: 5.0
daily_create:
backend: base
description: "Provision a session note under a daily folder: daily/<date>/<session_id>.md or daily/<date>.md"
parameters:
type: object
properties:
session_id:
type: string
description: "the session identifier (also the file stem); empty = day-level file"
default: ""
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_create_step
daily_list:
backend: base
description: "List notes under a single day."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_list_step
daily_reindex:
backend: base
description: "Rebuild the day-index page daily/<date>.md."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_reindex_step
frontmatter_delete:
backend: base
description: "Drop keys from a file's frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
keys:
type: array
description: "keys to remove"
items:
type: string
required:
- path
- keys
steps:
- backend: frontmatter_delete_step
frontmatter_read:
backend: base
description: "Read a file's frontmatter as a dict."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: frontmatter_read_step
frontmatter_update:
backend: base
description: "Merge key-values into a file's frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
metadata:
type: object
description: "key-values to merge"
required:
- path
- metadata
steps:
- backend: frontmatter_update_step
stat:
backend: base
description: "Stat path (size, mtime, exists, is_dir, is_file)."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: stat_step
list:
backend: base
description: "List files under a vault path."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative dir; empty = root"
default: ""
recursive:
type: boolean
description: "recurse"
default: false
limit:
type: integer
description: "max results"
default: 100
steps:
- backend: list_step
move:
backend: base
description: "Move / rename a vault file; rewrites inbound wikilinks by default."
parameters:
type: object
properties:
src_path:
type: string
description: "vault-relative source"
dst_path:
type: string
description: "vault-relative destination"
overwrite:
type: boolean
description: "overwrite if dst exists"
default: false
retarget:
type: boolean
description: "rewrite [[src]] → [[dst]] across the vault"
default: true
required:
- src_path
- dst_path
steps:
- backend: move_step
delete:
backend: base
description: "Delete a vault file or folder; returns surviving inbound wikilinks."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: delete_step
read:
backend: base
description: "Read a markdown file under the vault."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path; markdown only"
start_line:
type: integer
description: "first line (1-based, inclusive)"
end_line:
type: integer
description: "last line (1-based, inclusive)"
required:
- path
steps:
- backend: read_step
with_neighbors: false
max_neighbors_per_direction: 10
read_image:
backend: base
description: "Read an image file as base64 (vault-relative path)."
parameters:
type: object
properties:
path:
type: string
description: >-
vault-relative path; common image formats supported
(png/jpg/jpeg/webp/gif/bmp/tiff/heic)
required:
- path
steps:
- backend: read_image_step
max_bytes: 5242880
write:
backend: base
description: "Write a markdown file (create or overwrite) with name/description frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path; markdown only"
name:
type: string
description: "frontmatter name"
description:
type: string
description: "frontmatter description"
content:
type: string
description: "body"
metadata:
type: object
description: "Optional extra frontmatter fields (md only)."
required:
- path
- name
- description
- content
steps:
- backend: write_step
edit:
backend: base
description: "Find-and-replace in a markdown file (all occurrences)."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
old:
type: string
description: "text to find"
new:
type: string
description: "replacement"
default: ""
required:
- path
- old
- new
steps:
- backend: edit_step
auto_dream:
backend: base
description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
default: ""
hint:
type: string
description: "caller guidance passed through to dream extract/integrate"
default: ""
topic_count:
type: integer
description: "maximum number of final daily interest topics"
default: 3
topic_diversity_days:
type: integer
description: "number of previous interests.yaml days to avoid repeating"
default: 7
steps:
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
topic_diversity_days: 7
- backend: dream_finish_step
file_catalog: dream
proactive:
backend: base
description: "Proactive: read daily/<date>/interests.yaml and expose the latest user-interest topics."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to read; defaults to today in the dreamer's timezone"
default: ""
include_content:
type: boolean
description: "whether to include the raw YAML content in response metadata"
default: true
steps:
- backend: proactive_step
auto_memory:
backend: base
description: "Auto-memory: record conversation facts into a daily note"
parameters:
type: object
properties:
messages:
type: array
description: "messages"
items:
type: object
session_id:
type: string
description: "session identifier passed to daily_create"
default: ""
memory_hint:
type: string
description: "optional hint"
required:
- messages
steps:
- backend: auto_memory_step
auto_resource:
backend: base
description: "Auto-resource: interpret resource files into daily notes"
parameters:
type: object
properties:
changes:
type: array
description: "resource change batch, each item has path/file_path and change"
items:
type: object
properties:
path:
type: string
file_path:
type: string
change:
type: string
description: "added/modified/deleted"
required:
- changes
steps:
- backend: auto_resource_step
components:
tokenizer:
default:
backend: regex
as_embedding:
default:
backend: ${EMBEDDING_BACKEND:-openai}
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
credential:
api_key: ${EMBEDDING_API_KEY:-}
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
parameters:
dimensions: 1024
embedding_store:
default:
backend: local
as_embedding: default
as_llm:
default:
backend: ${LLM_BACKEND:-openai}
model: ${LLM_MODEL_NAME:-qwen3.7-plus}
stream: true
context_size: 200000
max_retries: 3
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: false
agent_wrapper:
default:
backend: agentscope
as_llm: default
permission_mode: bypass
react_config:
max_iters: 30
context_config:
trigger_ratio: 0.8
reserve_ratio: 0.1
tool_result_limit: 50000
model_config:
max_retries: 1
claude_code:
backend: claude_code
model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.1}
api_key: ${CLAUDE_CODE_API_KEY:-}
base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic}
permission_mode: bypassPermissions
file_graph:
default:
backend: local
file_catalog:
default:
backend: local
resource:
backend: local
digest:
backend: local
dream:
backend: local
file_chunker:
markdown:
backend: markdown
supported_extensions: [ "md" ]
default:
backend: default
supported_extensions: [ "jsonl" ]
keyword_index:
default:
backend: bm25
tokenizer: default
file_store:
default:
backend: local
store_name: local
# embedding_store: default
embedding_store: ""
keyword_index: default
file_graph: default

38
reme4/enumeration/chunk_enum.py
View file

@ -4,18 +4,46 @@ from enum import Enum
class ChunkEnum(str, Enum):
"""Enumeration of possible chunk categories for stream processing."""
"""Enumeration of possible chunk categories for stream processing.
Covers both AgentScope and Claude Code SDK streaming protocols:
AgentScope events -> ChunkEnum mapping:
ReplyStartEvent -> REPLY_START
ReplyEndEvent -> REPLY_END
TextBlockStart/Delta/End -> CONTENT
ThinkingBlockStart/Delta/End -> THINK
DataBlockStart/Delta/End -> DATA
ToolCallStart/Delta/End -> TOOL_CALL
ToolResultStart/TextDelta/DataDelta/End -> TOOL_RESULT
ModelCallEndEvent -> USAGE
ExceedMaxItersEvent -> ERROR
Claude Code SDK events -> ChunkEnum mapping:
message_start -> REPLY_START
message_stop -> REPLY_END
content_block_start/delta/stop (text) -> CONTENT
content_block_start/delta/stop (thinking) -> THINK
content_block_start/delta/stop (tool_use) -> TOOL_CALL
ToolResultBlock -> TOOL_RESULT
ResultMessage -> USAGE + DONE
ResultMessage.is_error -> ERROR
"""
# Lifecycle markers
REPLY_START = "reply_start"
REPLY_END = "reply_end"
# Content types
THINK = "think"
CONTENT = "content"
DATA = "data"
# Tool interaction
TOOL_CALL = "tool_call"
TOOL_RESULT = "tool_result"
# Metadata & terminal
USAGE = "usage"
ERROR = "error"
DONE = "done"

4
reme4/pyproject.toml
View file

@ -35,11 +35,12 @@ dependencies = [
"rich>=14.3.3",
"uvicorn>=0.41.0",
"watchfiles>=1.1.1",
"zstandard>=0.23.0",
]
[project.optional-dependencies]
core = [
"agentscope>=2.0.0",
"agentscope>=2.0.2",
"faiss-cpu>=1.13.2",
"jieba>=0.42.1",
"rjieba>=0.2.1",
@ -70,6 +71,7 @@ include-package-data = true
[tool.setuptools.package-data]
"*" = ["py.typed", "**/*.yaml", "**/*.json"]
"reme4.components.tokenizer" = ["stopwords"]
[tool.setuptools.dynamic]
version = { attr = "reme4.__version__" }

5
reme4/reme.py
View file

@ -9,6 +9,8 @@ from .config import parse_args, resolve_app_config
from .enumeration import ComponentEnum
from .utils import cli_find_reme, load_env, precheck_start
_CLIENT_KWARGS = {"host", "port", "timeout", "transport", "command", "args"}
class ReMe(Application):
"""ReMe memory management application."""
@ -17,10 +19,11 @@ class ReMe(Application):
async def call_server(action: str, **kwargs):
"""Call the appropriate server component."""
backend: str = kwargs.pop("backend", "http")
client_kwargs = {key: kwargs.pop(key) for key in list(kwargs) if key in _CLIENT_KWARGS}
client_cls = R.get(ComponentEnum.CLIENT, backend)
if client_cls is None:
raise ValueError(f"Unknown client backend: {backend!r}")
async with client_cls() as client:
async with client_cls(**client_kwargs) as client:
async for chunk in client(action=action, **kwargs):
print(chunk, end="", flush=True)
print()

1
reme4/schema/application_config.py
View file

@ -30,6 +30,7 @@ class ApplicationConfig(BaseModel):
app_name: str = Field(default=os.getenv("APP_NAME", "ReMe"), description="Application display name")
vault_dir: str = Field(default=".reme", description="Vault root directory for runtime files")
metadata_dir: str = Field(default="reme_metadata", description="Subdirectory for ReMe persistent state")
session_dir: str = Field(default="reme_session", description="Subdirectory for persisted agent sessions")
resource_dir: str = Field(default="resource", description="Subdirectory for external assets")
daily_dir: str = Field(default="daily", description="Subdirectory for daily memory")
digest_dir: str = Field(default="digest", description="Subdirectory for digest memory")

4
reme4/schema/file_chunk.py
View file

@ -9,8 +9,8 @@ class FileChunk(EmbNode):
"""A chunk of a file with positional info and per-stage retrieval scores."""
path: str = Field(default="", description="Path relative to the vault")
start_line: int = Field(default=0, description="Inclusive start line (0-based)")
end_line: int = Field(default=0, description="Exclusive end line")
start_line: int = Field(default=0, description="Inclusive start line (1-based)")
end_line: int = Field(default=0, description="Inclusive end line (1-based)")
scores: dict[str, float] = Field(default_factory=dict, description="Retrieval scores keyed by stage")
@property

39
reme4/schema/stream_chunk.py
View file

@ -1,14 +1,49 @@
"""Stream chunk schema for incremental responses (e.g. LLM streaming)."""
from typing import Any
from pydantic import BaseModel, Field
from ..enumeration import ChunkEnum
class StreamChunk(BaseModel):
"""A single chunk in a streaming response sequence."""
"""A single chunk in a unified streaming response sequence.
Carries full information from both AgentScope and Claude Code SDK
backends. Optional fields are ``None`` by default so that simple
text-only streams (e.g. plain CONTENT deltas) stay lightweight.
Fields:
chunk_type: Category of this chunk (see ChunkEnum).
chunk: Payload, typically a text delta, but can be
dict/list for structured data (tool-call JSON,
usage stats, etc.).
done: Terminal marker. True only for the final DONE chunk.
session_id: Session identifier (AgentScope: agent.state.session_id;
Claude Code: ResultMessage.session_id).
block_id: Content-block identifier for matching start/delta/end
sequences (both backends assign block IDs).
tool_call_id: Tool-call identifier for correlating call deltas
with their start event and result.
tool_call_name: Name of the tool being invoked.
media_type: MIME type for DATA blocks (e.g. ``"image/png"``).
input_tokens: Prompt tokens consumed (populated on USAGE chunks).
output_tokens: Completion tokens generated (populated on USAGE chunks).
metadata: Backend-specific extras that don't warrant a dedicated
field.
"""
chunk_type: ChunkEnum = Field(default=ChunkEnum.CONTENT, description="Type of chunk content")
chunk: str | dict | list = Field(default="", description="Chunk payload")
done: bool = Field(default=False, description="Whether this is the final chunk")
metadata: dict = Field(default_factory=dict, description="Chunk metadata")
session_id: str | None = Field(default=None, description="Session identifier")
block_id: str | None = Field(default=None, description="Content block identifier")
tool_call_id: str | None = Field(default=None, description="Tool call identifier")
tool_call_name: str | None = Field(default=None, description="Tool call name")
media_type: str | None = Field(default=None, description="MIME type for data blocks")
input_tokens: int | None = Field(default=None, description="Prompt tokens consumed")
output_tokens: int | None = Field(default=None, description="Completion tokens generated")
metadata: dict[str, Any] = Field(default_factory=dict, description="Chunk metadata")

90
reme4/steps/__init__.py
View file

@ -1,90 +1,14 @@
"""steps"""
from . import channel, common, evolve, file_io, index, transfer
from .base_step import BaseStep
from .common.demo import DemoEchoStep1, DemoEchoStep2
from .common.health_check import HealthCheckStep
from .common.help import HelpStep
from .common.llm_demo import LLMDemoStep
from .common.stream_demo import StreamDemoStep1, StreamDemoStep2
from .common.version import VersionStep
from .evolve.auto_dream import AutoDreamStep
from .evolve.auto_memory import AutoMemoryStep
from .evolve.auto_resource import AutoResourceStep
from .evolve.dream import DreamStep
from .file_io.daily_create import DailyCreateStep
from .file_io.daily_list import DailyListStep
from .file_io.daily_reindex import DailyReindexStep
from .file_io.delete import DeleteStep
from .file_io.edit import EditStep
from .file_io.frontmatter_delete import FrontmatterDeleteStep
from .file_io.frontmatter_read import FrontmatterReadStep
from .file_io.frontmatter_update import FrontmatterUpdateStep
from .file_io.list import ListStep
from .file_io.move import MoveStep
from .file_io.read import ReadStep
from .file_io.read_image import ReadImageStep
from .file_io.stat import StatStep
from .file_io.write import WriteStep
from .channel.channel_notify import ChannelNotifyStep
from .channel.claim_channel import ClaimChannelStep
from .index.clear_and_scan import ClearAndScanStep
from .index.node_search import NodeSearchStep
from .index.scan_changes import ScanCatalogChangesStep, ScanStoreChangesStep
from .index.search import SearchStep
from .index.traverse import TraverseStep
from .index.update_catalog import UpdateCatalogStep
from .index.update_index import UpdateIndexStep
from .index.foreach_dispatch import ForeachDispatchStep
from .index.log_changes import LogChangesStep
from .index.watch_changes import WatchChangesStep
__all__ = [
"BaseStep",
# common
"DemoEchoStep1",
"DemoEchoStep2",
"HealthCheckStep",
"HelpStep",
"LLMDemoStep",
"StreamDemoStep1",
"StreamDemoStep2",
"VersionStep",
# evolve
"AutoMemoryStep",
"AutoResourceStep",
# file_io
"DeleteStep",
"EditStep",
"ListStep",
"MoveStep",
"ReadStep",
"ReadImageStep",
"StatStep",
"WriteStep",
# file_io (daily)
"DailyCreateStep",
"DailyListStep",
"DailyReindexStep",
# file_io.frontmatter
"FrontmatterDeleteStep",
"FrontmatterReadStep",
"FrontmatterUpdateStep",
# channel
"ChannelNotifyStep",
"ClaimChannelStep",
# index
"ClearAndScanStep",
"ForeachDispatchStep",
"LogChangesStep",
"NodeSearchStep",
"ScanCatalogChangesStep",
"ScanStoreChangesStep",
"SearchStep",
"TraverseStep",
"UpdateCatalogStep",
"UpdateIndexStep",
"WatchChangesStep",
# evolve (dream)
"AutoDreamStep",
"DreamStep",
"channel",
"common",
"evolve",
"file_io",
"index",
"transfer",
]

66
reme4/steps/base_step.py
View file

@ -2,26 +2,26 @@
import copy
from abc import abstractmethod, ABC
from typing import TypeVar, TYPE_CHECKING
from typing import Any, TYPE_CHECKING
from agentscope.model import ChatModelBase
from ..components.agent_wrapper.base_agent_wrapper import BaseAgentWrapper
from ..components.base_component import ComponentMixin
from ..components.component_registry import R
from ..components.file_catalog import BaseFileCatalog
from ..components.file_store import BaseFileStore
from ..components.prompt_handler import PromptHandler
from ..components.runtime_context import RuntimeContext
from ..enumeration import ComponentEnum
from ..schema import Response
from ..schema import ApplicationConfig, Response
if TYPE_CHECKING:
from ..components import ApplicationContext
from ..components.job import BaseJob
T = TypeVar("T")
_UNSET = object()
_DispatchStep = str | dict[str, Any]
class Ref:
@ -41,14 +41,7 @@ class Ref:
__slots__ = ("base_cls", "comp_enum", "attr", "optional", "key", "_cache_attr")
def __init__(
self,
base_cls: type,
comp_enum: ComponentEnum,
attr: str | None = None,
*,
optional: bool = False,
) -> None:
def __init__(self, base_cls: type, comp_enum: ComponentEnum, attr: str | None = None, *, optional: bool = False):
self.base_cls = base_cls
self.comp_enum = comp_enum
self.attr = attr
@ -108,8 +101,7 @@ class BaseStep(ComponentMixin, ABC):
def __new__(cls, *args, **kwargs):
# Snapshot init args so copy() can rebuild an equivalent instance later.
instance = object.__new__(cls)
instance._init_args = copy.copy(args)
instance._init_kwargs = copy.copy(kwargs)
instance._init_args, instance._init_kwargs = copy.copy(args), copy.copy(kwargs)
return instance
def __init__(
@ -121,14 +113,14 @@ class BaseStep(ComponentMixin, ABC):
prompt_dict: dict[str, str] | None = None,
input_mapping: dict[str, str] | None = None,
output_mapping: dict[str, str] | None = None,
dispatch_steps: list[_DispatchStep] | None = None,
**kwargs,
):
super().__init__(name=name, backend=backend, app_context=app_context, **kwargs)
self.language: str = language
if not self.language and self.app_context is not None:
self.language = self.app_context.app_config.language
self.language = language or (self.app_context.app_config.language if self.app_context is not None else "")
self.input_mapping = input_mapping
self.output_mapping = output_mapping
self.dispatch_step_specs = list(dispatch_steps or [])
self.context: RuntimeContext | None = None
# Load class-level prompts first, then overlay caller-provided overrides.
@ -165,6 +157,13 @@ class BaseStep(ComponentMixin, ABC):
"""Return a named prompt template as-is."""
return self.prompt.get_prompt(prompt_name=prompt_name)
def config_value(self, key: str):
"""Return an app config value, falling back to ApplicationConfig defaults."""
defaults = ApplicationConfig()
cfg = self.app_context.app_config if self.app_context is not None else defaults
value = getattr(cfg, key)
return getattr(defaults, key) if value in (None, "") else value
def copy(self, **kwargs) -> "BaseStep":
"""Construct a new instance from the original init args, applying overrides."""
return self.__class__(*self._init_args, **{**self._init_kwargs, **kwargs})
@ -181,3 +180,36 @@ class BaseStep(ComponentMixin, ABC):
if job is None:
raise RuntimeError(f"Job {name} not found")
return await job(**kwargs)
def _resolve_dispatch_step(self, raw: _DispatchStep):
"""Resolve a dispatch step spec to (step class, init params)."""
if isinstance(raw, str):
params: dict[str, Any] = {"backend": raw}
elif isinstance(raw, dict):
params = dict(raw)
else:
raise TypeError(f"Invalid dispatch step spec: {raw!r}")
backend = params.get("backend", "")
if not backend:
raise ValueError("Dispatch step is missing the required 'backend' field")
step_cls = R.get(ComponentEnum.STEP, backend)
if step_cls is None:
raise RuntimeError(f"Unregistered step '{backend}'")
params["app_context"] = self.app_context
return step_cls, params
async def dispatch_steps(self, dispatch_steps: list[_DispatchStep], **kwargs) -> list[Response]:
"""Run dispatch steps against the current context.
Callers pass producer-specific values, usually ``changes=...``. Existing
context data is preserved for downstream handlers.
"""
if self.context is None:
raise RuntimeError("Cannot dispatch steps without a runtime context")
responses: list[Response] = []
for raw in dispatch_steps:
step_cls, params = self._resolve_dispatch_step(raw)
responses.append(await step_cls(**params)(self.context, **kwargs))
return responses

9
reme4/steps/channel/__init__.py
View file

@ -0,0 +1,9 @@
"""Channel steps."""
from .channel_notify import ChannelNotifyStep
from .claim_channel import ClaimChannelStep
__all__ = [
"ChannelNotifyStep",
"ClaimChannelStep",
]

9
reme4/steps/channel/channel_notify.py
View file

@ -1,6 +1,6 @@
"""``channel_notify_step`` — push a debounced batch of vault changes as one channel event.
Designed to be slotted into ``watch_changes_step.dispatch_step`` next to
Designed to be slotted into ``watch_changes_step.dispatch_steps`` next to
``update_index_step``: when the watcher emits a batch of changes, this
step forwards a single human-readable summary to
``ApplicationContext.metadata["channel_sink"]``. The Claude Code main
@ -34,11 +34,11 @@ class ChannelNotifyStep(BaseStep):
async def execute(self):
sink = self.app_context.metadata.get("channel_sink") if self.app_context is not None else None
if sink is None:
return
return self.context.response if self.context is not None else None
changes = (self.context.get("changes", []) if self.context is not None else []) or []
if not changes:
return
return self.context.response if self.context is not None else None
# Render paths vault-relative so the agent can pass them directly to
# slash commands like /dream <path>. Absolute paths that fall outside
@ -58,10 +58,11 @@ class ChannelNotifyStep(BaseStep):
lines.append(f"{change.get('change', '?')}: {shown}")
if not lines:
return
return self.context.response if self.context is not None else None
self.logger.info(f"[channel_notify] emit batch count={len(lines)}")
await sink.emit(
content="Vault 变更:\n" + "\n".join(lines),
meta={"kind": "vault_change", "count": str(len(lines))},
)
return self.context.response if self.context is not None else None

16
reme4/steps/channel/claim_channel.py
View file

@ -25,24 +25,28 @@ class ClaimChannelStep(BaseStep):
"""Bind the current MCP session as the ``<channel>`` recipient."""
async def execute(self):
assert self.context is not None
if self.context is None:
raise RuntimeError("claim_channel_step requires 'context'")
try:
from fastmcp.server.dependencies import get_context
ctx = get_context()
session = ctx.session
assert session is not None, "FastMCP context has no ServerSession"
assert self.app_context is not None, "claim_channel requires an application context"
if session is None:
raise RuntimeError("FastMCP context has no ServerSession")
if self.app_context is None:
raise RuntimeError("claim_channel requires an application context")
sink = self.app_context.metadata.get("channel_sink")
assert sink is not None, "channel_sink not configured on application context metadata"
if sink is None:
raise RuntimeError("channel_sink not configured on application context metadata")
session_id = ctx.session_id or "<unknown>"
sink.bind(session)
except Exception as e:
self.context.response.answer = {"claimed": False, "reason": f"{type(e).__name__}: {e}"}
self.context.response.metadata["claimed"] = False
return self.context.response
sink.bind(session)
session_id = ctx.session_id or "<unknown>"
self.logger.info(f"[claim_channel] channel bound to session={session_id}")
self.context.response.answer = {
"claimed": True,

23
reme4/steps/common/__init__.py
View file

@ -0,0 +1,23 @@
"""common steps"""
from .add import AddStep
from .demo import DemoEchoStep1, DemoEchoStep2
from .health_check import HealthCheckStep
from .help import HelpStep
from .llm_demo import LLMDemoStep
from .stream_demo import StreamDemoStep1, StreamDemoStep2
from .stream_llm_demo import StreamLLMDemoStep
from .version import VersionStep
__all__ = [
"AddStep",
"DemoEchoStep1",
"DemoEchoStep2",
"HealthCheckStep",
"HelpStep",
"LLMDemoStep",
"StreamDemoStep1",
"StreamDemoStep2",
"StreamLLMDemoStep",
"VersionStep",
]

35
reme4/steps/common/add.py Normal file
View file

@ -0,0 +1,35 @@
"""Simple step that adds two numbers — used as a demo tool for agent wrapper."""
from ..base_step import BaseStep
from ...components import R
@R.register("add_step")
class AddStep(BaseStep):
"""Add two numbers and return the result as the response answer.
Inputs (from RuntimeContext):
a (float, required): first addend.
b (float, required): second addend.
Output (written to context.response.answer):
The sum of ``a`` and ``b`` as a string.
"""
async def execute(self):
assert self.context is not None
try:
a = float(self.context.get("a", 0.0))
b = float(self.context.get("b", 0.0))
except (TypeError, ValueError) as exc:
self.context.response.success = False
self.context.response.answer = f"Invalid add arguments: {exc}"
return self.context.response
result = a + b
self.logger.info(f"[{self.name}] add({a}, {b}) = {result}")
self.context.response.success = True
self.context.response.answer = str(result)
self.context.response.metadata.update({"a": a, "b": b, "result": result})
return self.context.response

26
reme4/steps/common/health_check.py
View file

@ -57,6 +57,7 @@ def _mb_str(*objs) -> str:
def _embedding_status(comp) -> dict:
cache = getattr(comp, "_embedding_cache", {}) or {}
model = getattr(comp, "model", None)
try:
dims = comp.dimensions
except Exception:
@ -64,7 +65,7 @@ def _embedding_status(comp) -> dict:
return {
"is_started": comp.is_started,
"is_healthy": getattr(comp, "is_healthy", None),
"model_name": getattr(comp, "model_name", None),
"model_name": getattr(model, "model", None),
"dimensions": dims,
"cache_size": len(cache),
"memory": _mb_str(cache),
@ -97,9 +98,30 @@ def _file_graph_local_status(comp) -> dict:
}
def _file_graph_neo4j_status(comp) -> dict:
"""Neo4j backend: counts are cached on the async component for sync health checks."""
return {
"is_started": comp.is_started,
"n_nodes": getattr(comp, "_n_nodes", 0),
"n_edges": getattr(comp, "_n_edges", 0),
"n_virtual": getattr(comp, "_n_virtual", 0),
"memory": _mb_str(
getattr(comp, "_uri", ""),
getattr(comp, "_database", ""),
getattr(comp, "_n_nodes", 0),
getattr(comp, "_n_edges", 0),
getattr(comp, "_n_virtual", 0),
),
}
def _file_graph_status(comp) -> dict:
graph = getattr(comp, "_graph", None)
return _file_graph_nx_status(comp, graph) if graph is not None else _file_graph_local_status(comp)
if graph is not None:
return _file_graph_nx_status(comp, graph)
if hasattr(comp, "_driver"):
return _file_graph_neo4j_status(comp)
return _file_graph_local_status(comp)
def _file_store_status(comp) -> dict:

30
reme4/steps/common/llm_demo.py
View file

@ -2,23 +2,12 @@
from typing import Type
from agentscope.tool import FunctionTool, Toolkit
from pydantic import BaseModel
from ..base_step import BaseStep
from ...components import R
def add(a: float, b: float) -> str:
"""Add two numbers and return the sum.
Args:
a: first addend
b: second addend
"""
return str(a + b)
@R.register("llm_demo_step")
class LLMDemoStep(BaseStep):
"""Drive an Agent powered by the ``agent_wrapper`` component.
@ -26,7 +15,6 @@ class LLMDemoStep(BaseStep):
Inputs (from RuntimeContext):
query (str, required): user message content.
sys_prompt (str, optional): system prompt for the agent.
use_add_tool (bool, optional): register the ``add`` tool when True.
Output (written to context.response.answer):
The agent's final reply text.
@ -38,7 +26,6 @@ class LLMDemoStep(BaseStep):
assert self.context is not None
query: str = self.context.get("query", "")
sys_prompt: str = self.context.get("sys_prompt") or self.DEFAULT_SYS_PROMPT
use_add_tool: bool = bool(self.context.get("use_add_tool", False))
structured_model: Type[BaseModel] | None = self.context.get("structured_model")
if not query:
@ -46,25 +33,17 @@ class LLMDemoStep(BaseStep):
self.context.response.answer = "Skipped: empty query"
return self.context.response
toolkit = Toolkit(tools=[FunctionTool(add)]) if use_add_tool else Toolkit()
wrapper_kwargs = {
"system_prompt": sys_prompt,
"toolkit": toolkit,
"job_tools": ["add"],
}
if structured_model is not None:
wrapper_kwargs["output_schema"] = structured_model
_, result = await self.agent_wrapper.reply(query, **wrapper_kwargs)
result = await self.agent_wrapper.reply(query, **wrapper_kwargs)
structured_content: dict | None = None
if isinstance(result, dict) and "message" in result:
msg = result["message"]
structured_content = result["structured_output"]
else:
msg = result
text = (msg.get_text_content() or "").strip()
structured_content = result.get("structured_output")
text = (result.get("result") or "").strip()
self.logger.info(f"[{self.name}] response: {text!r}")
self.context.response.success = True
@ -73,7 +52,6 @@ class LLMDemoStep(BaseStep):
{
"query": query,
"sys_prompt": sys_prompt,
"use_add_tool": use_add_tool,
"response": text,
"structured_output": structured_content,
},

76
reme4/steps/common/stream_llm_demo.py
View file

@ -1,34 +1,10 @@
"""Demo step that drives an Agent via the agent_wrapper component with streaming output."""
import json
from agentscope.event import (
TextBlockDeltaEvent,
ThinkingBlockDeltaEvent,
ToolCallStartEvent,
ToolCallDeltaEvent,
ToolResultTextDeltaEvent,
ModelCallEndEvent,
ReplyStartEvent,
)
from agentscope.message import Msg
from agentscope.tool import FunctionTool, Toolkit
from ..base_step import BaseStep
from ...components import R
from ...enumeration import ChunkEnum
def add(a: float, b: float) -> str:
"""Add two numbers and return the sum.
Args:
a: first addend
b: second addend
"""
return str(a + b)
@R.register("stream_llm_demo_step")
class StreamLLMDemoStep(BaseStep):
"""Drive an Agent powered by the ``agent_wrapper`` component with streaming output.
@ -40,7 +16,6 @@ class StreamLLMDemoStep(BaseStep):
Inputs (from RuntimeContext):
query (str, required): user message content.
sys_prompt (str, optional): system prompt for the agent.
use_add_tool (bool, optional): register the ``add`` tool when True.
Output (written to context.response.answer):
The agent's final reply text.
@ -52,25 +27,22 @@ class StreamLLMDemoStep(BaseStep):
assert self.context is not None
query: str = self.context.get("query", "")
sys_prompt: str = self.context.get("sys_prompt") or self.DEFAULT_SYS_PROMPT
use_add_tool: bool = bool(self.context.get("use_add_tool", False))
if not query:
self.context.response.success = False
self.context.response.answer = "Skipped: empty query"
return self.context.response
toolkit = Toolkit(tools=[FunctionTool(add)]) if use_add_tool else Toolkit()
wrapper_kwargs = {
"system_prompt": sys_prompt,
"toolkit": toolkit,
"job_tools": ["add"],
}
if self.context.stream:
text = await self._stream_reply(query, **wrapper_kwargs)
else:
_, msg = await self.agent_wrapper.reply(query, **wrapper_kwargs)
text = (msg.get_text_content() or "").strip()
result = await self.agent_wrapper.reply(query, **wrapper_kwargs)
text = (result.get("result") or "").strip()
self.logger.debug(f"[{self.name}] response: {text!r}")
@ -80,45 +52,23 @@ class StreamLLMDemoStep(BaseStep):
{
"query": query,
"sys_prompt": sys_prompt,
"use_add_tool": use_add_tool,
"response": text,
},
)
return self.context.response
async def _stream_reply(self, query: str, **wrapper_kwargs) -> str:
"""Stream agent reply events to the context stream queue."""
"""Stream unified chunks to the context stream queue."""
assert self.context is not None
reply_msg: Msg | None = None
text_parts: list[str] = []
async for event in self.agent_wrapper.reply_stream(query, **wrapper_kwargs):
if isinstance(event, ReplyStartEvent):
reply_msg = Msg(
id=event.reply_id,
name=event.name,
role=event.role,
content=[],
)
elif isinstance(event, TextBlockDeltaEvent):
await self.context.add_stream_string(event.delta, ChunkEnum.CONTENT)
elif isinstance(event, ThinkingBlockDeltaEvent):
await self.context.add_stream_string(event.delta, ChunkEnum.THINK)
elif isinstance(event, ToolCallStartEvent):
payload = json.dumps({"name": event.tool_call_name, "id": event.tool_call_id})
await self.context.add_stream_string(payload, ChunkEnum.TOOL_CALL)
elif isinstance(event, ToolCallDeltaEvent):
await self.context.add_stream_string(event.delta, ChunkEnum.TOOL_CALL)
elif isinstance(event, ToolResultTextDeltaEvent):
await self.context.add_stream_string(event.delta, ChunkEnum.TOOL_RESULT)
elif isinstance(event, ModelCallEndEvent):
usage = json.dumps(
{"input_tokens": event.input_tokens, "output_tokens": event.output_tokens},
)
await self.context.add_stream_string(usage, ChunkEnum.USAGE)
async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs):
await self.context.add_stream_string(chunk.chunk, chunk.chunk_type)
if reply_msg is not None:
reply_msg.append_event(event)
if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str):
text_parts.append(chunk.chunk)
if reply_msg is not None:
return (reply_msg.get_text_content() or "").strip()
return ""
if chunk.session_id:
self.context.response.metadata["session_id"] = chunk.session_id
return "".join(text_parts).strip()

14
reme4/steps/evolve/__init__.py
View file

@ -1,5 +1,17 @@
"""Evolve steps."""
from ._evolve import now
from .auto_memory import AutoMemoryStep
from .auto_resource import AutoResourceStep
from .dream import DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep
__all__ = ["now"]
__all__ = [
"now",
"AutoMemoryStep",
"AutoResourceStep",
"DreamExtractStep",
"DreamFinishStep",
"DreamIntegrateStep",
"DreamTopicsStep",
"ProactiveStep",
]

286
reme4/steps/evolve/auto_dream.py
View file

@ -1,286 +0,0 @@
"""AutoDreamStep — daily-tick wrapper that dispatches per-file to the ``dream`` job.
Each tick scans today's two surfaces under ``<daily_dir>/``:
* ``<daily_dir>/<today>.md`` — the day-index file (auto-rebuilt rollup
of today's notes); included first so day-level abstractions land
before per-event details.
* ``<daily_dir>/<today>/**/*.md`` — event notes for the day.
The diff vs ``file_catalog`` follows the same shape as
:class:`ScanCatalogChangesStep`: build ``existing`` (on-disk ``rel → mtime``)
and ``indexed`` (catalog ``rel → mtime``, restricted to today's
prefix so we never disturb entries from other days), then:
* ``existing`` keys not in ``indexed`` → **added**, dream
* mtime mismatch → **modified**, dream
* ``indexed`` keys not in ``existing`` → **deleted**, drop from catalog
* mtime match → **unchanged**, skip
For every to-dream file, the step calls the configured ``dispatch_job``
(default ``"dream"``) via :meth:`BaseStep.run_job`. The dispatch job's
``Response`` carries a ``DreamResult`` payload in ``metadata``;
AutoDream re-hydrates that for its per-file aggregate.
After dreaming, successful (and Phase 1 vacuously-skipped) files
upsert their current ``st_mtime`` so the next tick re-dreams only
what actually changed. Failures leave the catalog untouched and will
be retried on the next tick.
The two writers (``WatchDreamStep`` event-driven, ``AutoDreamStep``
catch-up scan) share the protocol: upsert ``(path, st_mtime)`` on
success. Last-writer-wins is fine — same path + same content yields
the same mtime, so they cannot disagree on what's "done".
Cron scheduling itself is out of scope; this step is the unit of
work — invoke it from a system cron, ``reme auto-dream date=...``,
or any other catch-up trigger when ``auto_dream_loop`` missed a file
(e.g. process crashed before the watcher fired).
**Backend-agnostic.** Because dispatch goes through the configured
``dream`` job, the per-file dream implementation is decided by the
YAML config — whichever step the ``dream`` job's ``backend`` resolves
to. AutoDream itself doesn't care which backend runs underneath.
Inputs (RuntimeContext):
date (str, optional): YYYY-MM-DD to scan. Defaults to today
in the dreamer's timezone.
hint (str, optional): passed through to each per-file dream.
Step kwargs (from yaml ``backend: auto_dream_step``):
dispatch_job (str, default "dream"): name of the job to call
per file. Override only if the deployment renamed the dream
job. The dispatched job must accept ``path`` and ``hint``
kwargs and return a ``Response`` whose ``metadata`` matches
:class:`DreamResult`.
persist (bool, default True): when True, ``file_catalog.dump()``
is called after the batch so progress survives a restart.
"""
from pathlib import Path
from pydantic import BaseModel, Field
from ._evolve import now
from .dream import DreamResult
from ..base_step import BaseStep
from ...components import R
from ...schema import FileNode
class AutoDreamResult(BaseModel):
"""Aggregated outcome of one auto-dream tick."""
date: str = ""
files_scanned: int = 0
files_unchanged: int = 0
files_dreamed: int = 0
files_skipped: int = 0
files_failed: int = 0
files_deleted: int = 0
per_file: list[DreamResult] = Field(default_factory=list)
summary: str = ""
@R.register("auto_dream_step")
class AutoDreamStep(BaseStep):
"""Scan ``daily/<today>.md`` + ``daily/<today>/`` and dispatch to the
configured ``dream`` job for each file whose ``st_mtime`` doesn't
already match its ``file_catalog`` entry."""
def __init__(self, dispatch_job: str = "dream", persist: bool = True, **kwargs):
super().__init__(**kwargs)
self.dispatch_job: str = dispatch_job
self.persist: bool = persist
def _vault_dir(self) -> Path:
"""Vault root as an absolute path (mirrors :meth:`DreamStep._vault_dir`)."""
vr = getattr(self.file_store, "vault_path", None)
return Path(vr).resolve() if vr else Path.cwd().resolve()
async def _dispatch_dream(self, rel_path: str, hint: str) -> DreamResult:
"""Call the configured ``dispatch_job`` once and re-hydrate its
``Response.metadata`` into a :class:`DreamResult`.
On dispatch failure (job raises) returns a ``DreamResult`` with
``error`` populated so the caller's accounting stays uniform.
"""
try:
resp = await self.run_job(self.dispatch_job, path=rel_path, hint=hint)
except Exception as e: # pylint: disable=broad-except
self.logger.error(
f"[{self.name}] dispatch {self.dispatch_job!r} failed on {rel_path}: " f"{type(e).__name__}: {e}",
)
return DreamResult(path=rel_path, error=f"{type(e).__name__}: {e}")
# The dream job's execute() does context.response.metadata.update(result.model_dump()),
# so metadata carries every DreamResult field. extra keys are
# ignored by pydantic v2 default (extra='ignore').
md = dict(resp.metadata or {})
try:
dr = DreamResult.model_validate(md)
except Exception as e: # noqa: BLE001
self.logger.error(
f"[{self.name}] dispatch {self.dispatch_job!r} returned non-DreamResult metadata "
f"for {rel_path}: {type(e).__name__}: {e}",
)
return DreamResult(path=rel_path, error=f"bad dispatch metadata: {type(e).__name__}: {e}")
# If the underlying step set success=False, treat as failure even
# if metadata didn't carry an error string (defensive).
if not resp.success and not dr.error:
dr.error = resp.answer or "dispatch returned success=False"
if not dr.path:
dr.path = rel_path
return dr
async def execute(self):
assert self.context is not None
date_input: str = (self.context.get("date", "") or "").strip()
hint: str = (self.context.get("hint", "") or "").strip()
# daily_dir comes from app config — NOT a tool param. Same convention
# as daily_create / daily_list / daily_reindex.
cfg = self.app_context.app_config if self.app_context is not None else None
daily_dir = (cfg.daily_dir if cfg else "") or "daily"
tz = self.app_context.app_config.timezone if self.app_context is not None else None
today = date_input or now(tz).strftime("%Y-%m-%d")
vault = self._vault_dir()
files = _scan_today_files(vault, today, daily_dir)
# existing: today's on-disk paths → st_mtime. Insertion order = scan
# order (date.md first, then sorted event notes); preserved through
# the diff so the day-index file is dreamed before per-event notes.
existing: dict[str, float] = {}
for rel in files:
try:
existing[rel] = (vault / rel).stat().st_mtime
except OSError as e:
self.logger.error(f"[{self.name}] stat failed on {rel}: {e}")
# indexed: catalog entries restricted to today's prefix. Restriction
# is critical — get_nodes() returns all days, but we must only
# consider deletions within today's scan scope.
today_md = f"{daily_dir}/{today}.md"
today_dir = f"{daily_dir}/{today}/"
all_nodes = await self.file_catalog.get_nodes()
indexed: dict[str, float] = {
n.path: n.st_mtime for n in all_nodes if n.path == today_md or n.path.startswith(today_dir)
}
# Diff — same vocabulary as scan_*_changes_step (added/modified/deleted).
to_dream: list[tuple[str, float]] = [(rel, mt) for rel, mt in existing.items() if indexed.get(rel) != mt]
unchanged: list[str] = [rel for rel, mt in existing.items() if indexed.get(rel) == mt]
to_delete: list[str] = sorted(indexed.keys() - existing.keys())
result = AutoDreamResult(
date=today,
files_scanned=len(existing),
files_unchanged=len(unchanged),
files_deleted=len(to_delete),
)
self.logger.info(
f"[{self.name}] auto-dream tick date={today} scanned={len(existing)} "
f"unchanged={len(unchanged)} todo={len(to_dream)} deleted={len(to_delete)} "
f"under {daily_dir}/{today}{{.md,/}} dispatch={self.dispatch_job!r}",
)
# Drop catalog entries for files no longer on disk first. Cheap, no
# LLM, and keeps the catalog consistent even if the dream pass below
# errors.
if to_delete:
try:
await self.file_catalog.delete(to_delete)
except Exception as e: # noqa: BLE001
self.logger.exception(
f"[{self.name}] file_catalog.delete failed: {type(e).__name__}: {e}",
)
# Dispatch + upsert per-file. Single-file granularity means a job
# failure on file N doesn't block files N+1..K from advancing their
# catalog mtime. The dispatch decouples backend choice (AS / CC)
# from this loop — the configured ``dream`` job picks the runner.
upsert_nodes: list[FileNode] = []
for rel_path, mtime in to_dream:
dr = await self._dispatch_dream(rel_path, hint)
result.per_file.append(dr)
if dr.error:
# Failures leave the catalog untouched — next tick retries.
result.files_failed += 1
continue
if dr.skipped:
# Phase 1 said "nothing to extract" — still mark as seen so
# Phase 1 doesn't re-run on every tick.
result.files_skipped += 1
else:
result.files_dreamed += 1
upsert_nodes.append(FileNode(path=rel_path, st_mtime=mtime))
if upsert_nodes:
try:
await self.file_catalog.upsert(upsert_nodes)
except Exception as e: # noqa: BLE001
self.logger.exception(
f"[{self.name}] file_catalog.upsert failed: {type(e).__name__}: {e}",
)
if self.persist and (upsert_nodes or to_delete):
try:
await self.file_catalog.dump()
except Exception as e: # noqa: BLE001
self.logger.exception(
f"[{self.name}] file_catalog.dump failed: {type(e).__name__}: {e}",
)
result.summary = _render_auto_dream_summary(result)
self.context.response.success = result.files_failed == 0
self.context.response.answer = result.summary
self.context.response.metadata.update(result.model_dump())
return self.context.response
def _scan_today_files(vault: Path, today: str, daily_dir: str) -> list[str]:
"""Return vault-relative paths of today's day-index + event notes.
* ``<daily_dir>/<today>.md`` — the day-index file (auto-rebuilt
rollup of all of today's notes). Included first so its day-level
abstractions land before the per-event details.
* ``<daily_dir>/<today>/**/*.md`` — event notes for the day,
sorted by path for deterministic processing order.
"""
out: list[str] = []
if not daily_dir:
return out
day_index = vault / daily_dir / f"{today}.md"
if day_index.is_file():
out.append(str(day_index.relative_to(vault)))
daily_root = vault / daily_dir / today
if daily_root.is_dir():
for md in sorted(daily_root.rglob("*.md")):
if md.is_file():
out.append(str(md.relative_to(vault)))
return out
def _render_auto_dream_summary(r: AutoDreamResult) -> str:
"""One-line header + one line per dreamed file with its outcome.
Unchanged + deleted files are counted in the header only."""
lines = [
f"[AutoDreamStep] date={r.date} scanned={r.files_scanned} "
f"unchanged={r.files_unchanged} dreamed={r.files_dreamed} "
f"skipped={r.files_skipped} failed={r.files_failed} "
f"deleted={r.files_deleted}",
]
for dr in r.per_file:
if dr.error:
status = f"ERROR ({dr.error})"
elif dr.skipped:
status = "SKIP"
else:
status = f"OK (+{len(dr.nodes_created)} created, ~{len(dr.nodes_updated)} updated)"
lines.append(f" - {dr.path}: {status}")
return "\n".join(lines)

64
reme4/steps/evolve/auto_memory.py
View file

@ -7,10 +7,12 @@ from agentscope.message import Msg
from ._evolve import format_history, now
from ..base_step import BaseStep
from ..file_io import refresh_day_index, validate_session_id
from ...components import R
_TOOL_OUTPUT_MAX = 2048
_TOOL_OUTPUT_HALF = 1024
_SOURCE_CONVERSATION_KEY = "source_conversation"
def _truncate_text(text: str) -> str:
@ -73,17 +75,20 @@ class AutoMemoryStep(BaseStep):
super().__init__(**kwargs)
self.agent_tools: list[str] = ["read", "edit", "frontmatter_update", "write"]
def _session_path(self, session_id: str, tz: str | None) -> Path:
current = now(tz)
date_str = current.strftime("%Y-%m-%d")
resource = self.app_context.app_config.resource_dir if self.app_context else "resource"
return self.file_store.vault_path / resource / date_str / f"session_agent_{session_id}.jsonl"
def _session_dir(self) -> str:
return str(self.config_value("session_dir")).strip("/")
async def _save_session_messages(self, session_id: str, messages: list[Msg], tz: str | None) -> None:
def _session_path(self, session_id: str) -> Path:
return self.file_store.vault_path / self._session_dir() / "dialog" / f"{session_id}.jsonl"
def _session_link(self, session_id: str) -> str:
return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]"
async def _save_session_messages(self, session_id: str, messages: list[Msg]) -> None:
if not session_id or not messages:
return
path = self._session_path(session_id, tz)
path = self._session_path(session_id)
existing: list[Msg] = []
if path.exists():
@ -139,7 +144,12 @@ class AutoMemoryStep(BaseStep):
messages: list[Msg] = [self._to_msg(item) for item in raw_messages]
await self._save_session_messages(session_id, messages, tz)
if session_id and (err := validate_session_id(session_id)):
self.context.response.success = False
self.context.response.answer = f"Error: {err}"
return
await self._save_session_messages(session_id, messages)
if not messages:
self.context.response.success = True
@ -169,16 +179,44 @@ class AutoMemoryStep(BaseStep):
history=format_history(messages),
)
tools = [self.get_job(name) for name in self.agent_tools]
_, msg = await self.agent_wrapper.reply(
result = await self.agent_wrapper.reply(
user_message,
system_prompt=self.prompt_format("system_prompt"),
tools=tools,
job_tools=self.agent_tools,
)
source_conversation = ""
if session_id:
source_conversation = self._session_link(session_id)
link_response = await self.run_job(
"frontmatter_update",
path=note_path,
metadata={_SOURCE_CONVERSATION_KEY: source_conversation},
)
if not link_response.success:
self.context.response.success = False
self.context.response.answer = f"frontmatter_update failed: {link_response.answer}"
self.context.response.metadata.update(
{"path": note_path, "created": created, "n_messages": len(messages), "index": None},
)
self.logger.info(
f"[{self.name}] source conversation link failed "
f"path={note_path} session_id={session_id!r} answer={link_response.answer!r}",
)
return
daily_dir = self.config_value("daily_dir")
index_payload = await refresh_day_index(self.file_store, create_response.metadata["date"], daily_dir)
self.context.response.success = True
self.context.response.answer = (msg.get_text_content() or "").strip()
self.context.response.answer = (result.get("result") or "").strip()
self.context.response.metadata.update(
{"path": note_path, "created": created, "n_messages": len(messages)},
{
"path": note_path,
"created": created,
"n_messages": len(messages),
"source_conversation": source_conversation,
"index": index_payload,
},
)
self.logger.info(f"[{self.name}] done {note_path}")

99
reme4/steps/evolve/auto_resource.py
View file

@ -1,19 +1,24 @@
"""auto_resource — interpret resource files into daily notes via an agent."""
"""auto_resource — interpret resource files into same-name daily notes via an agent."""
import hashlib
from pathlib import PurePosixPath
import uuid
from pathlib import Path, PurePosixPath
import aiofiles
from watchfiles import Change
from ..base_step import BaseStep
from ..file_io import refresh_day_index
from ...components import R
def _compute_session_id(filename: str) -> str:
"""Return 'resource_' + first 8 hex chars of MD5(filename)."""
digest = hashlib.md5(filename.encode()).hexdigest()[:8]
return f"resource_{digest}"
def _compute_agent_session_id(path: str) -> str:
"""Return a stable UUID session id for agent backends."""
return str(uuid.uuid5(uuid.NAMESPACE_URL, path))
def _compute_note_stem(filename: str) -> str:
"""Return the daily note stem for a resource filename."""
return PurePosixPath(filename).stem
def _parse_resource_path(file_path: str, resource_dir: str) -> tuple[str, str]:
@ -47,9 +52,9 @@ class AutoResourceStep(BaseStep):
return Change.__members__.get(raw)
return None
async def _handle_delete(self, date_str: str, session_id: str) -> None:
daily_dir = self.app_context.app_config.daily_dir if self.app_context else "daily"
note_rel = f"{daily_dir}/{date_str}/session_agent_{session_id}.md"
async def _handle_delete(self, date_str: str, note_stem: str) -> None:
daily_dir = self.config_value("daily_dir")
note_rel = f"{daily_dir}/{date_str}/{note_stem}.md"
note_abs = self.vault_path / note_rel
if note_abs.is_file():
@ -57,13 +62,16 @@ class AutoResourceStep(BaseStep):
self.logger.info(f"[{self.name}] Deleted file: {note_rel}")
await self.file_store.delete([note_rel])
index_payload = await refresh_day_index(self.file_store, date_str, daily_dir)
self.context.response.success = True
self.context.response.answer = f"Deleted resource note: {note_rel}"
self.context.response.metadata.update({"path": note_rel, "session_id": session_id, "action": "deleted"})
self.context.response.metadata.update(
{"path": note_rel, "session_id": note_stem, "action": "deleted", "index": index_payload},
)
async def _handle_upsert(self, file_path: str, date_str: str, session_id: str, created: bool) -> None:
create_response = await self.run_job("daily_create", session_id=session_id, date=date_str)
async def _handle_upsert(self, file_path: str, date_str: str, note_stem: str, created: bool) -> None:
create_response = await self.run_job("daily_create", session_id=note_stem, date=date_str)
if not create_response.success:
self.context.response.success = False
self.context.response.answer = f"daily_create failed: {create_response.answer}"
@ -92,59 +100,88 @@ class AutoResourceStep(BaseStep):
date=date_str,
)
tools = [self.get_job(name) for name in self.agent_tools]
_, msg = await self.agent_wrapper.reply(
agent_session_id = _compute_agent_session_id(file_path)
result = await self.agent_wrapper.reply(
user_message,
system_prompt=self.prompt_format("system_prompt"),
tools=tools,
session_id=session_id,
job_tools=self.agent_tools,
session_id=agent_session_id,
)
daily_dir = self.config_value("daily_dir")
index_payload = await refresh_day_index(self.file_store, date_str, daily_dir)
self.context.response.success = True
self.context.response.answer = (msg.get_text_content() or "").strip()
self.context.response.answer = (result.get("result") or "").strip()
self.context.response.metadata.update(
{
"path": note_path,
"created": note_created,
"session_id": session_id,
"session_id": note_stem,
"agent_session_id": agent_session_id,
"action": "added" if created else "modified",
"index": index_payload,
},
)
self.logger.info(f"[{self.name}] done {note_path}")
async def execute(self):
async def _handle_change(self, file_path: str, raw_change) -> dict:
assert self.context is not None
file_path: str = self.context.get("file_path", "")
raw_change = self.context.get("change", "")
file_path = self.to_vault_relative(file_path) if file_path and Path(file_path).is_absolute() else file_path
if not file_path:
self.context.response.success = False
self.context.response.answer = "Missing file_path"
return
return {"success": False, "path": file_path, "change": raw_change, "answer": self.context.response.answer}
change = self._normalize_change(raw_change)
if change is None:
self.context.response.success = False
self.context.response.answer = f"Invalid change type: {raw_change}"
return
return {"success": False, "path": file_path, "change": raw_change, "answer": self.context.response.answer}
resource_dir = self.app_context.app_config.resource_dir if self.app_context else "resource"
resource_dir = self.config_value("resource_dir")
date_str, filename = _parse_resource_path(file_path, resource_dir)
if not date_str or not filename:
self.context.response.success = False
self.context.response.answer = f"Cannot parse date/filename from: {file_path}"
return
return {"success": False, "path": file_path, "change": change.name, "answer": self.context.response.answer}
session_id = _compute_session_id(filename)
self.logger.info(f"[{self.name}] {change.name} file_path={file_path} session_id={session_id}")
note_stem = _compute_note_stem(filename)
self.logger.info(f"[{self.name}] {change.name} file_path={file_path} note_stem={note_stem}")
if change == Change.deleted:
await self._handle_delete(date_str, session_id)
await self._handle_delete(date_str, note_stem)
else:
await self._handle_upsert(
file_path,
date_str,
session_id,
note_stem,
created=change == Change.added,
)
return {
"success": self.context.response.success,
"path": file_path,
"change": change.name,
"answer": self.context.response.answer,
"metadata": dict(self.context.response.metadata),
}
async def execute(self):
assert self.context is not None
changes = self.context.get("changes")
if not isinstance(changes, list):
self.context.response.success = False
self.context.response.answer = "AutoResourceStep requires changes: list[dict]"
return self.context.response
results = [
await self._handle_change(item.get("path") or item.get("file_path", ""), item.get("change", ""))
for item in changes
if isinstance(item, dict)
]
success_count = sum(1 for item in results if item.get("success"))
self.context.response.success = success_count == len(changes)
self.context.response.answer = f"Processed {success_count}/{len(changes)} resource change(s)"
self.context.response.metadata["processed"] = len(results)
self.context.response.metadata["results"] = results
return self.context.response

432
reme4/steps/evolve/dream.py
View file

@ -1,432 +0,0 @@
"""DreamStep — single-file digest step (auto-dream's create_or_update primitive).
Reads one daily-event note or resource file at the given vault-relative
``path``, identifies the ABSTRACTIONS the material teaches in Phase 1
(each tagged with one of the three buckets), then in Phase 2 makes
ONE cognitive write decision (CREATE or one of the three UPDATE
flavors: CORROBORATE / REFINE / CORRECT) per abstraction using a
**bucket-specific** integrate prompt.
**Digest is the abstract memory layer** — raw details stay in the
material; digest holds the principle, pattern, or precedent worth
recalling once the specifics fade. Provenance wikilinks
(``derived_from::``) let readers drill back down to the source.
Pipeline (external loop in Python, two distinct ReAct agent invocations,
**light Phase 1 / heavy Phase 2**):
execute():
units, _ = _extract(material_blob) # 1× ReAct: identify abstractions
# agent emits ExtractedUnits
# ({units: [{name, bucket, summary}, ...]})
for unit in units: # Python loop, K iterations
_integrate_unit(unit) # 1× ReAct per abstraction, dispatched
# to integrate_system_prompt_<bucket>;
# recalls cross-bucket, decides write,
# uses canonical write/edit/frontmatter_update tools.
The bucket vocabulary is hard-coded (:data:`BUCKETS`) — three buckets,
each with a dedicated Phase 2 prompt:
* ``procedure`` — how-to-do-X: steps, methods, recipes, workflows.
* ``personal`` — user/team specific: identity, preferences,
conventions, things they avoid.
* ``wiki`` — general knowledge: definitions, principles,
observations, decisions-as-precedent. Default catch-all.
There is no SKIP outcome in Phase 2: Phase 1 is the gate for "not
worth memorizing"; anything reaching Phase 2 warrants a write.
Phase 2 uses the **canonical** ``write`` / ``edit`` jobs (no
constrained variants). Bucket placement and edge conservation are
prompt-level discipline; the tools themselves perform no path-shape
or conservation validation.
Invocation form (CLI / MCP):
reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md
reme dream path=resource/2026-05-28/spec.pdf hint="focus on auth"
"""
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, Field
from ._evolve import now
from ..base_step import BaseStep
from ...components import R
# Hard-coded bucket vocabulary. Phase 1 classifies each sub-unit into
# one of these; Phase 2 dispatches to the bucket-specific prompt.
# Order matters for prompt rendering — keep procedure/personal/wiki.
BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki")
# Bucket = Literal of BUCKETS. Pydantic Literal must be a static type;
# update both BUCKETS and Bucket together if the vocabulary changes.
Bucket = Literal["procedure", "personal", "wiki"]
_EXTRACT_TOOLS: tuple[str, ...] = ("read",)
_INTEGRATE_TOOLS: tuple[str, ...] = (
# read — dream uses its own node-level digest search (NOT the
# general chunk-level `search`), specialized for dedup + synapse
# recall. See reme4/steps/index/node_search.py for the rationale.
# NO traverse here: traverse is a retrieve-time subgraph mining
# tool (used by external retrieval agents); dream is a write-time
# candidate recall operation, structurally a different problem.
"node_search",
"read",
"frontmatter_read",
# write
"write",
"edit",
"frontmatter_update",
)
# ============================================================
# Schema (Pydantic models for structured ReAct output) + helpers
# ============================================================
def _pack_material(file_store, path: str) -> str:
"""Render one daily-event note or resource file into a prompt block."""
try:
absolute = (Path(file_store.vault_path or ".") / path).resolve()
except Exception as e:
return f"### {path}\n(error resolving path: {type(e).__name__}: {e})\n"
if not absolute.is_file():
return f"### {path}\n(file not found)\n"
try:
return f"### {path}\n{absolute.read_text(encoding='utf-8')}\n"
except Exception as e:
return f"### {path}\n(error reading: {type(e).__name__}: {e})\n"
class MemoryUnit(BaseModel):
"""One memory sub-unit identified by Phase 1's structured output."""
name: str = Field(
description=(
"Short kebab-case identifier for the abstraction "
"(e.g. 'jwt-rotation-decision', 'pr-size-pref'). "
"Agent-internal handle — NOT the eventual digest slug; "
"Phase 2 picks the actual filing path."
),
)
bucket: Bucket = Field(
description=(
"Which bucket this abstraction belongs in — Phase 2 dispatches "
"to a bucket-specific prompt based on this. Pick exactly one: "
"`procedure` (how-to-do-X — steps, methods, recipes, workflows), "
"`personal` (user/team-specific — identity, preferences, "
"conventions, things they avoid), `wiki` (general knowledge — "
"definitions, principles, observations, decisions-as-precedent; "
"default catch-all when nothing else fits)."
),
)
summary: str = Field(
description=(
"1-2 sentences naming the abstraction AND pointing at where "
"in the material the supporting evidence lives "
"(e.g. 'short-credential compliance drives auth cadence; "
"illustrated by the 30→24h decision in the 'Decision' section "
"+ the SOC2 CC6.1 criticism in the 'Observation' section')."
),
)
class ExtractedUnits(BaseModel):
"""Structured output emitted by Phase 1's extract agent."""
units: list[MemoryUnit] = Field(
default_factory=list,
description=(
"Memory sub-units identified in the material — orthogonal "
"abstractions (principles / patterns / precedents) worth "
"lifting into long-term memory. Each is tagged with its "
"bucket. Empty list = nothing worth lifting (Phase 2 is skipped)."
),
)
def _render_outcome_line(unit_name: str, bucket: str, o: "IntegrateOutcome") -> str:
"""Format one IntegrateOutcome as a one-line summary entry."""
body = f"{o.action} {o.target_path}"
if o.note:
body += f" — {o.note}"
return f"[{unit_name}/{bucket}] {body}"
class IntegrateOutcome(BaseModel):
"""Structured outcome reported by Phase 2 for one sub-unit."""
action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field(
description=(
"Outcome of the write decision for this sub-unit. Phase 1 already "
"filtered out non-abstractions, so every sub-unit reaching you "
"warrants a write — pick the matching fine-grained action: "
"`CREATE` — brand-new digest node (recall returned no node "
"covering this abstraction); even thin first-encounter seeds go "
"here, they grow via CORROBORATE / REFINE on later passes. "
"`CORROBORATE` (most common when a covering node exists) — "
"provenance append + optional wording strengthening; the "
"abstraction already covers this material. `REFINE` — covering "
"node exists but the material reveals nuance, scope, or edge "
"cases the abstraction under-specified. `CORRECT` — covering "
"node exists but the material contradicts it; tighten the "
"abstraction or annotate the contradiction inline."
),
)
target_path: str = Field(
description=("The digest path you wrote to — must match what your `write` / " "`edit` call(s) targeted."),
)
note: str = Field(
default="",
description=(
"Optional ONE short line, ≤ 200 chars, no newlines, summarizing "
"what landed (e.g. 'extended scope to also cover X'). Do NOT "
"dump recall summaries, search results, internal reasoning, or "
"transcripts here — those belong in the ReAct trace, not the "
"outcome note."
),
)
class DreamResult(BaseModel):
"""Outcome of one DreamStep invocation.
Per-tool audit lives in the toolkit layer (not exposed back to the
orchestrator). Structured outcome here is the input path the call
processed, the memory sub-units the agent declared in Phase 1, and
what got created / updated in Phase 2.
"""
used_llm: bool = False
skipped: bool = False
path: str = ""
units: list[dict] = Field(default_factory=list)
nodes_created: list[str] = Field(default_factory=list)
nodes_updated: list[str] = Field(default_factory=list)
summary: str = ""
error: str = ""
# ============================================================
# DreamStep — the per-file create_or_update step.
# ============================================================
@R.register("dream_step")
class DreamStep(BaseStep):
"""auto-dream create_or_update step — one file per call.
Inputs (from RuntimeContext):
path (str, required): vault-relative path of one
daily-event note or resource file to dream over. Pass
empty string to no-op.
hint (str, optional): caller guidance to the LLM
(e.g. "focus on the auth-related decisions").
Output (written to context.response.answer):
``DreamResult`` JSON in ``metadata``; LLM summary in ``answer``.
CLI / MCP form:
reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md
"""
def _vault_dir(self) -> Path:
vr = getattr(self.file_store, "vault_path", None)
return Path(vr).resolve() if vr else Path.cwd().resolve()
def _llm_available(self) -> bool:
try:
return self.as_llm is not None
except Exception:
return False
async def _extract(self, material_blob: str, hint: str, vault_dir: Path) -> tuple[list[dict], str]:
"""Phase 1: one ReAct invocation — read material + emit ExtractedUnits.
Returns ``(units, llm_summary)`` where ``units`` is the cleaned
sub-unit list (each entry has ``name`` / ``bucket`` / ``summary``)
and ``llm_summary`` is whatever free-form text the agent produced
alongside its structured emission.
"""
tools = [self.get_job(name) for name in _EXTRACT_TOOLS]
tz = self.app_context.app_config.timezone if self.app_context is not None else None
user_message = self.prompt_format(
"extract_user_message",
today=now(tz).strftime("%Y-%m-%d"),
hint=hint or "(none)",
material_blob=material_blob,
)
_, result = await self.agent_wrapper.reply(
user_message,
system_prompt=self.prompt_format(
"extract_system_prompt",
vault_dir=str(vault_dir),
buckets=", ".join(BUCKETS),
),
tools=tools,
output_schema=ExtractedUnits,
)
msg = result["message"]
meta = result["structured_output"] if isinstance(result["structured_output"], dict) else {}
cleaned: list[dict] = []
for raw in meta.get("units") or []:
if not isinstance(raw, dict):
continue
name = str(raw.get("name") or "").strip()
summary = str(raw.get("summary") or "").strip()
bucket = str(raw.get("bucket") or "").strip()
if not name or not summary:
continue
if bucket not in BUCKETS:
self.logger.warning(
f"[{self.name}] unit {name!r} emitted bucket {bucket!r} "
f"not in {list(BUCKETS)}; routing to 'wiki'",
)
bucket = "wiki"
cleaned.append({"name": name, "summary": summary, "bucket": bucket})
return cleaned, (msg.get_text_content() or "").strip()
async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome:
"""One ReAct invocation per memory sub-unit, dispatched to the
bucket-specific system prompt. Returns the parsed
:class:`IntegrateOutcome` reported by the agent — that's the
single source of truth for what got written (action +
target_path)."""
bucket = unit.get("bucket") or "wiki"
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
tools = [self.get_job(name) for name in _INTEGRATE_TOOLS]
user_message = self.prompt_format(
"integrate_user_message",
hint=hint or "(none)",
unit_name=unit.get("name", ""),
unit_bucket=bucket,
unit_summary=unit.get("summary", ""),
material_blob=material_blob,
)
_, result = await self.agent_wrapper.reply(
user_message,
system_prompt=self.prompt_format(
f"integrate_system_prompt_{bucket}",
vault_dir=str(vault_dir),
digest_dir=digest_dir,
bucket=bucket,
),
tools=tools,
output_schema=IntegrateOutcome,
)
return IntegrateOutcome.model_validate(result["structured_output"])
async def dream_one(self, path: str, hint: str = "") -> DreamResult:
"""Run the full extract + integrate pipeline on one vault-relative
material path. Returns a structured :class:`DreamResult`. Safe to
call repeatedly on the same instance — per-invocation trackers are
reset at the start of each call. Used by :meth:`execute` (single
file from context), :class:`AutoDreamStep` (loop over today's
materials), and :class:`WatchDreamStep` (loop over a change batch).
"""
path = (path or "").strip()
hint = (hint or "").strip()
if not path:
return DreamResult(used_llm=False, skipped=True)
if not self._llm_available():
return DreamResult(
used_llm=False,
skipped=True,
path=path,
error="no llm configured; dreaming requires an LLM",
)
material_blob = _pack_material(self.file_store, path)
vault_dir = self._vault_dir()
# Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the
# memory sub-units worth lifting. Each unit carries its own bucket.
self.logger.info(f"[{self.name}] extract phase: path={path!r}")
units, extract_summary = await self._extract(material_blob, hint, vault_dir)
if not units:
return DreamResult(
used_llm=True,
path=path,
summary=extract_summary or "no memory sub-units declared",
skipped=True,
)
unit_handles = ", ".join(f"{u['name']}/{u['bucket']}" for u in units)
self.logger.info(f"[{self.name}] integrate phase: {len(units)} sub-unit(s): {unit_handles}")
# Phase 2 — integrate, one fresh ReAct per sub-unit, dispatched to
# the bucket-specific system prompt. Python-level loop, not agent
# loop. Each session emits a structured IntegrateOutcome whose
# action + target_path are the source of truth for what landed.
nodes_created: list[str] = []
nodes_updated: list[str] = []
per_unit_lines: list[str] = []
for i, unit in enumerate(units, start=1):
name = unit.get("name", "?")
bucket = unit.get("bucket", "?")
try:
outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir)
except Exception as e:
self.logger.error(
f"[{self.name}] integrate {i}/{len(units)} "
f"(unit={name}, bucket={bucket}) failed: {type(e).__name__}: {e}",
)
per_unit_lines.append(f"[{name}/{bucket}] FAILED: {type(e).__name__}: {e}")
continue
if outcome.action == "CREATE":
nodes_created.append(outcome.target_path)
else:
nodes_updated.append(outcome.target_path)
per_unit_lines.append(_render_outcome_line(name, bucket, outcome))
per_unit_block = "\n".join(per_unit_lines)
summary = (
f"Declared {len(units)} sub-unit(s) ({unit_handles}); "
f"created {len(nodes_created)}, updated {len(nodes_updated)}.\n"
f"{per_unit_block}"
)
return DreamResult(
used_llm=True,
path=path,
units=units,
nodes_created=nodes_created,
nodes_updated=nodes_updated,
summary=summary,
skipped=False,
)
async def execute(self):
assert self.context is not None
path: str = (self.context.get("path", "") or "").strip()
hint: str = (self.context.get("hint", "") or "").strip()
result = await self.dream_one(path, hint)
if not path:
self.context.response.success = True
self.context.response.answer = "Skipped: no path supplied"
elif result.error:
self.context.response.success = False
self.context.response.answer = f"Error: {result.error}"
elif result.skipped:
self.context.response.success = True
self.context.response.answer = result.summary or "Skipped: no memory sub-units declared"
else:
self.context.response.success = True
self.context.response.answer = result.summary
self.context.response.metadata.update(result.model_dump())

886
reme4/steps/evolve/dream.yaml
View file

@ -1,886 +0,0 @@
extract_system_prompt: |
You are Phase 1 of dream — read the material, identify the
ABSTRACTIONS it teaches, and tag each with a bucket. Phase 2
picks the slug and writes; you only declare what's worth lifting.
vault_dir: {vault_dir}
## What digest memory is for
Digest is the **abstract memory layer** — analogous to the
prefrontal cortex aggregating cognition. Raw details (numbers,
narratives, who said what, full procedure text) STAY IN THE
MATERIAL. Digest holds the generalized lesson the reader should
recall next time — the part that survives once the specific
event fades.
When you cluster, you are NOT cataloguing the material's contents
— you are answering: *"what abstractions does this material teach
that I'd want a future agent / human to have at-hand when facing
a similar situation?"*
## What is a memory sub-unit?
One sub-unit = one abstraction the material teaches. **One sub-unit
maps to exactly one digest node** — Phase 2 makes one write
decision per sub-unit (CREATE or one of the three UPDATE flavors).
Phase 1 is the gate for "not worth memorizing"; once a sub-unit
reaches Phase 2 it WILL be written.
Multiple raw facts in the material that all illustrate the same
abstraction collapse to ONE sub-unit. Example: the kid-versioning
mechanism, the SOC2 CC6.1 rationale, and the new 24h cadence are
three FACTS, but they teach one abstraction — "short-credential
compliance drives auth cadence, not procedural convenience".
That's one sub-unit. The mechanism / numbers / RFC citation are
details — they stay in the daily note; the digest reaches them
through `derived_from::` provenance edges.
Sub-units are NOT bucket names, NOT kinds, NOT the eventual digest
slug — they're an agent-internal handle for the abstraction you've
identified. Phase 2 picks the slug + write decision; YOU pick the
bucket here.
### Bias: fewer, richer sub-units over many narrow ones
This is the abstract layer — heavy lifting toward few high-leverage
units, not toward exhaustive coverage. Heuristic for splitting two
pieces into two units vs one:
* Same abstraction shown by different facts? → ONE unit.
* Genuinely different abstractions a future reader would invoke
in DIFFERENT situations? → TWO units.
* Will they evolve independently as more materials arrive?
→ TWO units.
When in doubt, MERGE (or drop one of them entirely).
### What NOT to declare
- Passing mentions with no new abstraction (e.g. an OAuth recap
that restates a known concept) — daily-note indexing already
covers detail-level recall.
- Facts whose only audience is the material itself (one-off
timestamps, single meeting attendance) — not an abstraction.
- Event-level umbrella sub-units (e.g. `X-event-summary`) —
every sub-unit already carries
`derived_from:: [[<material-path>]]`, so the material itself
is the fan-out point linking to all its derived nodes; the
umbrella adds nothing.
## Bucket — pick exactly one per unit
The bucket determines which specialized Phase 2 prompt processes
this sub-unit. Pick by *kind of abstraction*, not by surface
topic.
- **`procedure`** — *how to do X*. Steps, methods, recipes,
workflows, runbooks, executable patterns. Reader's question:
"how do I accomplish Y?" Pick when the abstraction is an
actionable sequence or technique.
Examples: "key-rotation procedure", "incident triage flow",
"how to wire up a new MCP tool".
- **`personal`** — *user/team-specific facts about how WE work*.
Identity ("who is X"), preferences ("user prefers terse
replies"), conventions ("we use kebab-case for slug names"),
things to avoid ("don't run schema migrations on Friday"),
collaboration style. Reader's question: "what does THIS user /
team want / do / dislike?" Pick when the abstraction is only
valid in the context of this user / team / project.
Examples: "huangsen prefers short PRs", "team avoids
mocking the DB in integration tests", "we don't write
`status` frontmatter".
- **`wiki`** — *general knowledge*. Definitions, principles,
observations, decisions-as-precedent, factual claims, mental
models. Reader's question: "what IS X / what happened / what
was decided?" Pick when the abstraction is true independent
of who's reading. Also the **default catch-all** when nothing
else fits cleanly.
Examples: "JWT is a signed token format", "short-credential
compliance drives auth cadence", "moving to 24h refresh
reduced p99 latency by 12%".
Straddling two buckets → pick by **center of gravity** (which
bucket the future reader will search from):
- "User prefers small PRs" → personal (rule for THIS user).
- "Small PRs are easier to review" → wiki (general claim).
- "Steps to split a large PR" → procedure.
Available: {buckets}
## Output
Each unit's `summary` should name the abstraction AND point at
where in the material the supporting evidence lives — Phase 2
cites it as provenance without re-reading. Field shapes are
enforced by the structured-output schema.
You have read-only access (`read` for inline `[[resource/...]]`
references that genuinely matter); no recall, no writing.
extract_user_message: |
today: {today}
hint: {hint}
# Material to cluster
{material_blob}
Identify the abstractions this material teaches, classify each
into one of {{procedure, personal, wiki}}, and emit via the
structured output schema. Empty list if nothing new is taught.
# ============================================================
# Phase 2 — bucket-specific INTEGRATE prompts.
# Dispatcher picks integrate_system_prompt_<bucket> from the
# bucket Phase 1 assigned to the unit.
# ============================================================
integrate_system_prompt_procedure: |
You are Phase 2 of dream, **procedure** bucket. The unit is a
how-to-do-X (steps, methods, recipes, runbooks, executable
patterns). Recall cross-bucket, decide CREATE / CORROBORATE /
REFINE / CORRECT, write exactly once. Sub-unit ↔ digest node
is 1:1; no SKIP outcome — Phase 1 already gated.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is NOT a faithful copy of the material — it's the cognitive
aggregation (think prefrontal cortex). Details stay in the daily
/ resource file; digest holds the principle, pattern, or
precedent the agent should recall later.
- **Body is SHORT and abstract** (≈ 50-200 words for most nodes;
longer only when the concept genuinely needs it). If your draft
starts copying paragraphs from the material, you're filing
detail in the wrong layer.
- **Provenance edges carry the details.** Whenever this
abstraction is illustrated by a specific material, add a
`derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink
— readers drill down through the edge, not through re-stated
facts in the body.
- **Wikilinks between digest nodes** carry the conceptual graph
(`relates_to::`, `depends_on::`, `is_a::`, …).
## Procedure-bucket body shape
A runbook, not a recap:
- **Trigger / when to use** (1 line) — under what conditions
does the reader reach for this procedure?
- **Steps** — numbered or terse bullets; each is one verb-led
imperative. Optional inline justification ("because X locks
the row before Y commits") is fine.
- **Pre-conditions / inputs** — short list, not prose.
- **Failure modes / caveats** — brief ("if step 3 returns
ROLLBACK, restart from step 1"); NOT a transcript of every
observed failure.
- **`derived_from:: [[<material-path>]]`** — at least one.
Plain-prose provenance does NOT count (only wikilinks survive
future updates).
## Recall → classify → decide → weave
1. **Recall** — call `node_search` with whatever queries best
fit the unit (verb stems work well: rotate, migrate, deploy…).
Use `limit=20-30` for broader coverage; one call usually
suffices but feel free to issue more if the unit spans
multiple concept dimensions. Recall feeds BOTH the dedup
judgment (same_abstraction label) and the synapse judgment
(related label). Cross-bucket on purpose: an existing match
filed elsewhere beats a duplicate.
2. **Classify (internal)** — `node_search` returns name +
description inline; triage from that directly. Use `read`
only for the few survivors needing body inspection. Internally
label each candidate (reasoning only, not emitted to output):
- **same_abstraction** — same trigger + substantially
overlapping steps → UPDATE target (new step / nuance is
REFINE, not "different procedure")
- **related** — adjacent procedure / sub-step / failure-mode
cross-ref → synapse wikilink in body
- **unrelated** — drop
3. **Decide** (exactly one same_abstraction action):
- no same_abstraction hit → **CREATE** at
`{digest_dir}/procedure/<slug>.md`.
- same_abstraction hit → **UPDATE** the best match:
- **CORROBORATE** — same procedure observed again; append
`derived_from::`, optionally strengthen wording
("consistently used across N runs").
- **REFINE** — new pre-condition / edge case / failure
mode; expand the relevant span, slot new steps into
the right position.
- **CORRECT** — wrong order, missing critical step, bad
outcome; tighten or annotate inline (`> note:
contradicted by [[new-material]] — <one-line>`).
4. **Synapse weave** (both CREATE and UPDATE) — weave every
`related` candidate from step 2 into the body as `[[Y.md]]`.
CREATE: woven from the start. UPDATE: additive `edit`
(only-add, never drop existing wikilinks). Default to weaving
more, not less — this is the only chance.
## Discipline
- CREATE writes inside `{digest_dir}/procedure/`. Phase 1 chose
your bucket — don't pivot.
- UPDATE may target any bucket if RECALL legitimately matched.
- `edit` is body-only and **only-add, not-delete**: never drop
wikilinks the `old` span contained (provenance must accumulate,
not evaporate).
- `frontmatter_update` is the only way to change frontmatter
(e.g. tighten `description`, set `kind: procedure`).
- One target per session. Never edit other nodes sideways.
Wikilinks are full vault-relative paths with `.md`
(`[[{digest_dir}/<bucket>/<slug>.md]]`, `[[daily/...]]`,
`[[resource/...]]`). Predicates are open
(`[A-Za-z][A-Za-z0-9_]*`) and live outside the brackets.
integrate_system_prompt_personal: |
You are Phase 2 of dream, **personal** bucket. The unit is
user/team-specific (identity, preference, convention, avoid-rule,
collaboration style). Recall cross-bucket, decide CREATE /
CORROBORATE / REFINE / CORRECT, write exactly once. Sub-unit ↔
digest node is 1:1; no SKIP — Phase 1 already gated.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is NOT a faithful copy of the material — it's the cognitive
aggregation (think prefrontal cortex). Details stay in the daily
/ resource file; digest holds the rule, identity, or convention
the agent should recall later.
- **Body is SHORT and abstract** (≈ 50-200 words). If your draft
starts narrating *what the user said in detail*, you're filing
detail in the wrong layer.
- **Provenance edges carry the details.** Whenever this rule is
set, restated, or revised by a specific material, add a
`derived_from:: [[daily/...]]` wikilink — readers drill down
through the edge, not through re-stated context.
- **Wikilinks between digest nodes** carry the conceptual graph
(`applies_to::`, `relates_to::`, …).
## Personal-bucket body shape
A short rule of engagement, not a biography:
- **Rule / fact** — one sentence stating the preference,
convention, or identity claim.
- **`Why:`** — the reason (a past incident, a constraint, a
strong preference). Knowing *why* lets future readers judge
edge cases instead of blindly applying.
- **`How to apply:`** — when this rule kicks in: which contexts,
tasks, boundaries.
- **`derived_from:: [[<material-path>]]`** — at least one.
Plain-prose provenance does NOT count.
Two common sub-shapes filed in this bucket:
- *Identity* — biographical / role facts ("X is a backend
engineer focused on observability"). Reader's question:
"who is X?".
- *Preference / convention / avoid-rule* — how someone likes
to work / what to skip. Reader's question: "how does X like
to work / what should I not do?".
When the same person has many preferences, prefer **one node per
preference** (not one big node per person) — that's the
granularity downstream search will hit.
## Recall → classify → decide → weave
1. **Recall** — call `node_search` with whatever queries best
fit the unit (user/team name + rule keywords:
`user-X-pr-size-pref`, `team-no-friday-deploys`). Use
`limit=20-30` for broader coverage; issue more calls if the
rule has multiple scope dimensions. Recall feeds BOTH the
dedup judgment (same_abstraction label) and the synapse
judgment (related label). Personal nodes often link to each
other and to the user's identity node — vector similarity
surfaces those even when literal names differ.
2. **Classify (internal)** — `node_search` returns name +
description inline; triage from that directly. Use `read`
only for the few survivors needing body inspection. Internally
label each candidate (reasoning only, not emitted to output):
- **same_abstraction** — same actor scope + same governing
principle → UPDATE target (new applicable context is
REFINE, not "different rule")
- **related** — adjacent rule / contrasting preference /
identity node cross-ref → synapse wikilink in body
- **unrelated** — drop
3. **Decide** (exactly one same_abstraction action):
- no same_abstraction hit → **CREATE** at
`{digest_dir}/personal/<slug>.md`.
- same_abstraction hit → **UPDATE** the best match:
- **CORROBORATE** — rule reaffirmed; append
`derived_from::`, possibly strengthen certainty
("observed across N independent contexts").
- **REFINE** — scope clarified ("only in CI runs",
"except when X holds"); expand `How to apply:`.
- **CORRECT** — user changed their mind / contradicted by
new behavior; tighten to the form both old and new
evidence support, OR annotate (`> note: contradicted by
[[new-material]] — user now prefers Y`) without
arbitrating.
4. **Synapse weave** (both CREATE and UPDATE) — weave every
`related` candidate from step 2 into the body as `[[Y.md]]`.
CREATE: woven from the start. UPDATE: additive `edit`
(only-add, never drop existing wikilinks). Default to weaving
more, not less — this is the only chance.
## Discipline
- CREATE writes inside `{digest_dir}/personal/`. Phase 1 chose
your bucket — don't pivot.
- UPDATE may target any bucket if RECALL legitimately matched.
- `edit` is body-only and **only-add, not-delete**: never drop
wikilinks the `old` span contained.
- `frontmatter_update` is the only way to change frontmatter
(e.g. tighten `description`, set `kind: preference`).
- One target per session. Never edit other nodes sideways.
Useful predicates: `derived_from::`, `applies_to::` (whose rule),
`relates_to::` (cross-link related preferences). Wikilinks are
full vault-relative paths with `.md`.
integrate_system_prompt_wiki: |
You are Phase 2 of dream, **wiki** bucket. The unit is general
knowledge (definition, principle, observation, decision-as-
precedent, factual claim, mental model). `wiki` is also the
catch-all when nothing more specific fits. Recall cross-bucket,
decide CREATE / CORROBORATE / REFINE / CORRECT, write exactly
once. Sub-unit ↔ digest node is 1:1; no SKIP — Phase 1 already
gated.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is NOT a faithful copy of the material — it's the cognitive
aggregation (think prefrontal cortex). Details stay in the daily
/ resource file; digest holds the definition, principle, or
precedent the agent should recall later.
- **Body is SHORT and abstract** (≈ 50-200 words; longer only
when the concept genuinely needs it). If your draft starts
copying paragraphs from the material, you're filing detail in
the wrong layer.
- **Provenance edges carry the details.** Whenever this
abstraction is illustrated by a specific material, add a
`derived_from:: [[daily/...]]` or `[[resource/...]]` wikilink.
- **Wikilinks between digest nodes** carry the conceptual graph
(`is_a::`, `extends::`, `depends_on::`, `contradicts::`, …).
## Wiki-bucket body shape
Encyclopedia-flavored — definition + properties + relations,
not narrative:
- **First line** — one-sentence definition / claim. The reader's
eye lands here first; make it self-contained.
- **Body** — short paragraphs OR tight bullets: properties,
sub-claims, distinctions, illustrative one-line examples. Each
non-obvious claim cites its source via `derived_from::`.
- **Relations** — typed wikilinks where the relation has
semantic weight. Most cross-node links can stay bare.
- **`derived_from:: [[<material-path>]]`** — at least one.
Plain-prose provenance does NOT count.
## Recall → classify → decide → weave
1. **Recall** — call `node_search` with whatever queries best
fit the unit (noun phrases + common synonyms). Use
`limit=20-30` for broader coverage; issue more calls when
the abstraction has multiple aspects worth querying
separately. Recall feeds BOTH the dedup judgment
(same_abstraction label) and the synapse judgment
(related label). Vector similarity catches abstractions
filed under different terminology even when surface words
don't overlap.
2. **Classify (internal)** — `node_search` returns name +
description inline; triage from that directly. Use `read`
only for the few survivors needing body inspection. Internally
label each candidate (reasoning only, not emitted to output):
- **same_abstraction** — same definition / principle in body,
even if wording differs → UPDATE target (slightly different
framing is REFINE; outright different concepts are
different nodes)
- **related** — concept-adjacent / contrasts / sup-/sub-
concept / instance cross-ref → synapse wikilink in body
- **unrelated** — drop
3. **Decide** (exactly one same_abstraction action):
- no same_abstraction hit → **CREATE** at
`{digest_dir}/wiki/<slug>.md`.
- same_abstraction hit → **UPDATE** the best match:
- **CORROBORATE** — principle reaffirmed by new instance;
append `derived_from::`, optionally strengthen wording
("consistently observed across N sources" / replace
"appears to" with "does"); body unchanged in substance.
- **REFINE** — definition's nuance / scope / edge cases
sharpened by the new material; tighten the relevant span,
add the new dimension. Body grows in precision, not in
detail volume.
- **CORRECT** — factual contradiction or overstatement;
either tighten to the narrower form both old and new
evidence support, or annotate inline (`> note:
contradicted by [[new-material]] — <one-line>`) without
arbitrating.
4. **Synapse weave** (both CREATE and UPDATE) — weave every
`related` candidate from step 2 into the body as `[[Y.md]]`.
CREATE: woven from the start. UPDATE: additive `edit`
(only-add, never drop existing wikilinks). Default to weaving
more, not less — this is the only chance.
## Discipline
- CREATE writes inside `{digest_dir}/wiki/`. Phase 1 chose your
bucket — don't pivot.
- UPDATE may target any bucket if RECALL legitimately matched.
- `edit` is body-only and **only-add, not-delete**: never drop
wikilinks the `old` span contained.
- `frontmatter_update` is the only way to change frontmatter
(e.g. tighten `description`, set `kind: concept` /
`kind: observation`).
- One target per session. Never edit other nodes sideways.
Wikilinks are full vault-relative paths with `.md`. Predicates
are open (`[A-Za-z][A-Za-z0-9_]*`); reuse existing predicates
when reasonable. Most wikilinks are bare — use a predicate only
when the relation has clear semantic weight.
integrate_user_message: |
hint: {hint}
# Sub-unit
name: {unit_name}
bucket: {unit_bucket}
summary: {unit_summary}
# Full material
{material_blob}
Process per your system prompt: recall (cross-bucket) → hit →
exactly one CREATE / CORROBORATE / REFINE / CORRECT. End with a
fully-populated `IntegrateOutcome`.
# ============================================================
# 中文版本 (language=zh 时启用)
# ============================================================
extract_system_prompt_zh: |
你是 dream 的 Phase 1 —— 阅读材料,识别它教导的 **抽象**,为
每个抽象标 bucket。Phase 2 选 slug、写入;你只声明值得提取的
内容。
vault_dir: {vault_dir}
## digest 记忆是干什么的
Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事情发
生的原始细节(数字、叙述、谁说了什么、完整流程文本)**保留
在材料中**。Digest 承载的是读者下次该回想起的、即使具体事件
淡忘后仍然有用的概括性教训。
你不是在 **编目** 材料的内容,而是在回答:*"这份材料教了哪
些抽象,是我希望未来的 agent / 人类在面对类似情境时手边能够
调取的?"*
## 什么是记忆 sub-unit
一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好对
应一个 digest 节点** —— Phase 2 针对每个 sub-unit 做一次写
入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是"不值得记忆"
的过滤闸口;一旦 sub-unit 进入 Phase 2,它就 **一定** 会被
写入。
材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。例:
kid 版本机制 + SOC2 CC6.1 依据 + 24h 新周期 是三个 **事实**,
但教的是同一个抽象 —— "JWT 轮换周期由短期凭证合规驱动,而
非流程惯性"。这是一个 sub-unit。机制 / 数字 / RFC 引用都是
细节 —— 它们留在 daily 笔记里,digest 通过 `derived_from::`
溯源边触达。
Sub-unit **不是** bucket 名,**不是** kind,**不是** 最终
digest slug —— 它只是你内部用于指代识别出来的抽象的把手。
Phase 2 选 slug + 写入决策;**bucket 由你在 Phase 1 决定**。
### 偏好:少而精的 sub-unit,而非多而细
这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是穷举
覆盖。两件事拆成一个还是两个 sub-unit 的启发式:
* 不同事实说明同一抽象? → 一个 sub-unit。
* 是真正不同的抽象,未来读者会在 **不同情境** 下分别调用?
→ 两个 sub-unit。
* 它们会随更多材料独立演化? → 两个 sub-unit。
拿不准时,**合并**(或整体丢弃其中一个)。
### 哪些不要声明
- 没有新抽象的顺带提及(例如只是把已知概念复述一遍的 OAuth
简介) —— daily 笔记索引已能覆盖细节级召回。
- 受众只有材料本身的事实(一次性时间戳、单次会议出席记
录) —— 不是抽象。
- 事件级伞节点(例如 `X-event-summary`) —— 每个 sub-unit
都会带 `derived_from:: [[<material-path>]]`,材料本身就是
扇出节点链向所有派生 digest;伞节点零增益。
## Bucket —— 每个 unit 必选其一
Bucket 决定哪份 Phase 2 prompt 处理这个 sub-unit。按 *抽象
的种类* 选,**不是** 按材料表面话题选。
- **`procedure`** —— *怎么做 X*。步骤、方法、配方、工作流、
runbook、可执行模式。读者问:"怎么完成 Y?"。当抽象是可
执行的动作序列或技巧时选这个。
例:"key-rotation 流程"、"事故 triage 流"、"如何接入新
MCP 工具"。
- **`personal`** —— *用户 / 团队 specific 的 "我们怎么干"
类事实*。身份("X 是谁")、偏好("用户偏好简短回复")、
约定("我们用 kebab-case 命名 slug")、规避("周五不跑
schema 迁移")、协作风格。读者问:"这个用户 / 团队 想要
/ 不喜欢什么?"。当抽象只在这位用户 / 团队 / 项目上下文
里成立时选这个。
例:"huangsen 偏好小 PR"、"团队不在集成测试里 mock DB"、
"我们不写 `status` frontmatter"。
- **`wiki`** —— *通用知识*。定义、原则、观察、决策先例、事
实主张、心智模型。读者问:"X 是什么 / 决策依据是什么?" ——
与谁在问无关。也是 **兜底** —— 没有更明确归属时落到这里。
例:"JWT 是签名 token 格式"、"短期凭证合规驱动鉴权周
期"、"切到 24h 刷新后 p99 降低 12%"。
跨桶时按 **重心** 选(未来读者最可能从哪个桶搜):
- "用户偏好小 PR" → personal(这个用户的规则)。
- "小 PR 更易评审" → wiki(通用主张)。
- "如何拆分大 PR 的步骤" → procedure。
可用 buckets: {buckets}
## 输出
每个 unit 的 `summary` 要 **同时** 命名抽象 **并** 指出材料
里支撑证据所在 —— Phase 2 直接引用做溯源,不必重读。字段形
态由结构化输出 schema 强制约束。
你只有只读访问(`read` 用于打开内联 `[[resource/...]]` 引
用,确实需要时);没有召回,没有写入。
extract_user_message_zh: |
today: {today}
hint: {hint}
# 待归类的材料
{material_blob}
识别这份材料教导的 **抽象**,为每个 unit 分类到
{{procedure, personal, wiki}} 之一,通过结构化输出 schema
提交。没有新抽象时使用空 unit 列表。
integrate_system_prompt_procedure_zh: |
你是 dream 的 Phase 2,**procedure** 桶。本次处理的 unit 是
一个"怎么做 X"(步骤、方法、配方、runbook、可执行模式)。
跨 bucket 召回,在 CREATE / CORROBORATE / REFINE / CORRECT
之间决策,**恰好一次** 写入。Sub-unit 与 digest 节点是 1:1;
无 SKIP —— Phase 1 已过滤。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
回想起的原则、模式、先例。
- **正文 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有概念真
的需要时才更长)。如果你的草稿开始大段抄材料的段落,说明
你把细节归错层了。
- **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加
一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`
wikilink —— 读者通过边下钻,而不是通过正文里复述事实。
- **digest 节点之间的 wikilink** 承载概念图(`relates_to::`、
`depends_on::`、`is_a::`…)。
## procedure 桶的 body 形态
Runbook,不是叙述:
- **触发 / 何时使用**(1 行)—— 读者在什么条件下会调取这个
流程?
- **步骤** —— 编号或紧凑的子弹点列表;每一步是一个动词领头
的祈使句。可选内联说明("因为 X 在 Y 提交前已锁住该行")。
- **前置条件 / 输入** —— 简短列表,不要散文。
- **失败模式 / 注意事项** —— 简短("若步骤 3 返回 ROLLBACK,
从步骤 1 重启");**不是** 每次观察到的失败的转写。
- **`derived_from:: [[<material-path>]]`** —— 至少一条。纯
散文形式 **不算**(下次 update 时会消失)。
## 召回 → 内化分类 → 决策 → 织突触
1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容
选(带动词词根效果好:rotate / migrate / deploy…)。`limit=
20-30` 取更宽覆盖;通常一次够,但若 unit 跨多个概念维度可
多调几次。召回结果同时服务 dedup(same_abstraction label)
和 synapse(related label)两类判断。**跨 bucket** 是有意
—— 就地更新优于复制创建。
2. **内化分类** —— `node_search` 已内嵌返回 name + description,
直接据此 triage。仅对需要看 body 的少数候选用 `read`。
对每个候选**内化打 label**(只在思考中分类,不输出):
- **same_abstraction** —— 同触发 + 步骤大幅重叠 → UPDATE
目标(新增一步 / 细微差异是 REFINE,**不是** 另一个流程)
- **related** —— 邻近流程 / 子步骤 / 失败模式互引 → 织成
body 内的 synapse wikilink
- **unrelated** —— 丢弃
3. **决策**(恰好一个 same_abstraction 动作):
- 无 same_abstraction 命中 → **CREATE** 在
`{digest_dir}/procedure/<slug>.md`。
- same_abstraction 命中 → **UPDATE** 最匹配的:
- **CORROBORATE** —— 同流程再次出现;加 `derived_from::`,
可选强化措辞("跨 N 次运行一致使用");步骤不动。
- **REFINE** —— 新前置 / 边界 / 失败模式;扩展相关片段,
新步骤插入正确位置。
- **CORRECT** —— 顺序错 / 缺关键步 / 结果不对;收紧或
内联标注(`> note: contradicted by [[new-material]] —
<一句话>`)。
4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有
`related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性
织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。
宁可多织 —— **这是唯一机会**。
## 纪律
- CREATE 必须写在 `{digest_dir}/procedure/`。Phase 1 已选定桶 ——
不要换桶。
- UPDATE 可指向任意 bucket(若召回合理命中)。
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
任何 wikilink(溯源必须累积,不可蒸发)。
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
(例如 REFINE 后收紧 `description`,加 `kind: procedure`)。
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
Wikilink 是带 `.md` 的 vault 相对完整路径
(`[[{digest_dir}/<bucket>/<slug>.md]]`、`[[daily/...]]`、
`[[resource/...]]`)。谓词词表开放
(`[A-Za-z][A-Za-z0-9_]*`),写在括号外。
integrate_system_prompt_personal_zh: |
你是 dream 的 Phase 2,**personal** 桶。本次处理的 unit 是
用户 / 团队 specific(身份 / 偏好 / 约定 / 规避规则 / 协作
风格)。跨 bucket 召回,在 CREATE / CORROBORATE / REFINE /
CORRECT 之间决策,**恰好一次** 写入。Sub-unit 与 digest 节
点是 1:1;无 SKIP —— Phase 1 已过滤。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
回想起的规则、身份、约定。
- **正文 SHORT 且抽象**(≈ 50-200 字)。如果你的草稿开始详
细叙述用户说了什么,说明归错层了。
- **溯源边承载细节**。每当这条规则被某份具体材料设定 / 重申
/ 修正时,加一条 `derived_from:: [[daily/...]]` wikilink ——
读者通过边下钻,而不是通过正文里复述上下文。
- **digest 节点之间的 wikilink** 承载概念图(`applies_to::`、
`relates_to::`、…)。
## personal 桶的 body 形态
简短的协作规则,不是传记:
- **规则 / 事实** —— 一句话陈述偏好、约定或身份。
- **`Why:`** —— 用户给出的原因(过往事故、所关心的约束、强
烈偏好)。知道 *why* 让未来读者能判断边界,而非盲目套用。
- **`How to apply:`** —— 这条规则什么时候启用:哪些情境、
任务、边界。
- **`derived_from:: [[<material-path>]]`** —— 至少一条;纯
散文形式不算。
同桶常见两类子形态:
- *身份* —— 用户 / 团队的传记 / 角色事实("X 是聚焦在
observability 的 backend 工程师")。读者问:"X 是谁?"。
- *偏好 / 约定 / 规避* —— 喜欢怎么干 / 该规避什么。读者问:
"X 喜欢怎么干 / 我不该做什么?"。
同一人有多条偏好时,**一条偏好一个节点**(不是一个人一大
节点) —— 这才是下游搜索的粒度。
## 召回 → 内化分类 → 决策 → 织突触
1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容
选(user / team 名 + 规则关键词:`user-X-pr-size-pref`、
`team-no-friday-deploys`)。`limit=20-30` 取更宽覆盖;规则
若有多个 scope 维度可多调几次。召回结果同时服务 dedup
(same_abstraction label)和 synapse(related label)两类
判断。personal 节点常彼此互链并指向用户身份节点 —— vector
相似度能在字面名不同时也召回这些。
2. **内化分类** —— `node_search` 已内嵌返回 name + description,
直接据此 triage。仅对需要看 body 的少数候选用 `read`。
对每个候选**内化打 label**(只在思考中分类,不输出):
- **same_abstraction** —— 同 actor 范围 + 同支配原则 →
UPDATE 目标(新增"规则适用情境"是 REFINE,**不是** 另一
条规则)
- **related** —— 邻近规则 / 对比偏好 / 用户身份节点互引 →
织成 body 内的 synapse wikilink
- **unrelated** —— 丢弃
3. **决策**(恰好一个 same_abstraction 动作):
- 无 same_abstraction 命中 → **CREATE** 在
`{digest_dir}/personal/<slug>.md`。
- same_abstraction 命中 → **UPDATE** 最匹配的:
- **CORROBORATE** —— 规则在新场景再次坐实;加
`derived_from::`,可选强化确定性("跨 N 个独立情境
观察")。
- **REFINE** —— 范围被澄清("仅在 CI 运行中"、"X 成立
时除外");把新边界扩到 `How to apply:`。
- **CORRECT** —— 用户改主意 / 规则被新行为否定;收紧到
新旧证据都支持的形式,或内联标注
(`> note: contradicted by [[new-material]] — 用户现在
偏好 Y`)不仲裁。
4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有
`related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性
织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。
宁可多织 —— **这是唯一机会**。
## 纪律
- CREATE 必须写在 `{digest_dir}/personal/`。Phase 1 已选定桶 ——
不要换桶。
- UPDATE 可指向任意 bucket(若召回合理命中)。
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
任何 wikilink。
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
(例如 REFINE 后收紧 `description`,加 `kind: preference`)。
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
常用谓词:`derived_from::`、`applies_to::`(规则归属哪个用
户)、`relates_to::`(交叉链接相关偏好)。Wikilink 是带
`.md` 的 vault 相对完整路径。
integrate_system_prompt_wiki_zh: |
你是 dream 的 Phase 2,**wiki** 桶。本次处理的 unit 是通用知
识(定义 / 原则 / 观察 / 决策先例 / 事实主张 / 心智模型)。
`wiki` 也是没有更明确归属时的 **兜底**。跨 bucket 召回,在
CREATE / CORROBORATE / REFINE / CORRECT 之间决策,**恰好
一次** 写入。Sub-unit 与 digest 节点是 1:1;无 SKIP —— Phase 1
已过滤。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额叶)。
细节留在 daily / resource 文件,digest 承载的是 agent 以后该
回想起的定义、原则、先例。
- **正文 SHORT 且抽象**(≈ 50-200 字;只有概念真的需要时才
更长)。如果你的草稿开始大段抄材料,说明归错层了。
- **溯源边承载细节**。每当这个抽象被某份具体材料佐证时,加
一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]`
wikilink。
- **digest 节点之间的 wikilink** 承载概念图(`is_a::`、
`extends::`、`depends_on::`、`contradicts::`…)。
## wiki 桶的 body 形态
百科风 —— 定义 + 性质 + 关系,不是叙述:
- **首行** —— 一句话定义 / 主张。读者目光首先落在这里,要让
它自包含。
- **正文** —— 短段落或紧凑子弹点:性质、子主张、区分、举
例片段。每条非显然主张靠 `derived_from::` 指回它的源材料。
- **关系** —— 有语义份量时用谓词。绝大多数跨节点链接保持裸链。
- **`derived_from:: [[<material-path>]]`** —— 至少一条;纯
散文形式不算。
## 召回 → 内化分类 → 决策 → 织突触
1. **召回** —— 调 `node_search`,query 由你自己根据 unit 内容
选(名词短语 + 常见同义词)。`limit=20-30` 取更宽覆盖;若
抽象本身有多个侧面,值得分别查询时可多调几次。召回结果
同时服务 dedup(same_abstraction label)和 synapse
(related label)两类判断。vector 相似度能捕获以不同术语
归档的同语义抽象,即便字面词不重叠。
2. **内化分类** —— `node_search` 已内嵌返回 name + description,
直接据此 triage。仅对需要看 body 的少数候选用 `read`。
对每个候选**内化打 label**(只在思考中分类,不输出):
- **same_abstraction** —— body 中的定义 / 原则相同(措辞
可不同)→ UPDATE 目标(同思想的略不同表述是 REFINE;真正
不同的概念是不同节点)
- **related** —— 概念邻近 / 对比 / 上位下位 / 实例互引 →
织成 body 内的 synapse wikilink
- **unrelated** —— 丢弃
3. **决策**(恰好一个 same_abstraction 动作):
- 无 same_abstraction 命中 → **CREATE** 在
`{digest_dir}/wiki/<slug>.md`。
- same_abstraction 命中 → **UPDATE** 最匹配的:
- **CORROBORATE** —— 原则被新实例再坐实;加
`derived_from::`,可选强化措辞("跨 N 个来源一致观
察"、把"似乎"换成"确实");正文实质不变。
- **REFINE** —— 细微差异 / 范围被新材料补足;收紧片段、
加新维度。正文 **精度** 上长,不在 **细节量** 上膨胀。
- **CORRECT** —— 事实矛盾或夸大;收紧到新旧证据都支持
的窄形式,或内联标注(`> note: contradicted by
[[new-material]] — <一句话>`)不仲裁。
4. **织突触**(CREATE 与 UPDATE 都要做)—— 把第 2 步所有
`related` 候选织入 body 作 `[[Y.md]]`。CREATE:写入时一次性
织全。UPDATE:additive `edit`(只增不删,绝不丢已有 wikilink)。
宁可多织 —— **这是唯一机会**。
## 纪律
- CREATE 必须写在 `{digest_dir}/wiki/`。Phase 1 已选定桶 ——
不要换桶。
- UPDATE 可指向任意 bucket(若召回合理命中)。
- `edit` 是 body-only,**只增不删**:绝不丢掉 `old` 片段中的
任何 wikilink。
- `frontmatter_update` 是修改 frontmatter 的 **唯一** 通道
(例如 REFINE 后收紧 `description`,加 `kind: concept` /
`kind: observation`)。
- 一次 session 一个目标。**绝不** 顺手编辑别的节点。
Wikilink 是带 `.md` 的 vault 相对完整路径。谓词词表开放
(`[A-Za-z][A-Za-z0-9_]*`),合理时复用。绝大多数 wikilink
保持裸链 —— 仅当关系具有清晰语义份量时用谓词。
integrate_user_message_zh: |
hint: {hint}
# Sub-unit
name: {unit_name}
bucket: {unit_bucket}
summary: {unit_summary}
# 完整材料
{material_blob}
按 system prompt 处理:召回(跨 bucket)→ 命中 → 恰好一次
CREATE / CORROBORATE / REFINE / CORRECT。以一个完整填充的
`IntegrateOutcome` 收尾。

15
reme4/steps/evolve/dream/__init__.py Normal file
View file

@ -0,0 +1,15 @@
"""Auto-dream steps."""
from .extract import DreamExtractStep
from .finish import DreamFinishStep
from .integrate import DreamIntegrateStep
from .proactive import ProactiveStep
from .topics import DreamTopicsStep
__all__ = [
"DreamExtractStep",
"DreamFinishStep",
"DreamIntegrateStep",
"DreamTopicsStep",
"ProactiveStep",
]

147
reme4/steps/evolve/dream/extract.py Normal file
View file

@ -0,0 +1,147 @@
"""Global dream extract step."""
import json
from ...base_step import BaseStep
from ...file_io import refresh_day_index
from ....components import R
from .schema import BUCKETS, DreamState
from .utils import (
clean_paths,
daily_dir,
llm_available,
pack_paths,
parse_structured_reply,
scan_day_files,
store_state,
today,
vault_dir,
)
_TOOLS = ("read",)
@R.register("dream_extract_step")
class DreamExtractStep(BaseStep):
"""Scan changed daily files and globally extract merged units/topics."""
def __init__(self, topic_session_id: str = "interests", **kwargs):
super().__init__(**kwargs)
self.topic_session_id = topic_session_id
async def execute(self):
assert self.context is not None
day = today(self, str(self.context.get("date", "") or ""))
hint = str(self.context.get("hint", "") or "").strip()
daily, vault = daily_dir(self), vault_dir(self)
if self.file_catalog is None:
raise RuntimeError("dream_extract_step requires file_catalog")
await refresh_day_index(self.file_store, day, daily)
existing = self._existing(vault, scan_day_files(vault, day, daily, f"{self.topic_session_id}.yaml"))
interests_rel = f"{daily}/{day}/{self.topic_session_id}.yaml"
day_md, day_prefix = f"{daily}/{day}.md", f"{daily}/{day}/"
nodes = await self.file_catalog.get_nodes()
indexed_all = {n.path: n.st_mtime for n in nodes if n.path == day_md or n.path.startswith(day_prefix)}
indexed = {path: mt for path, mt in indexed_all.items() if path != interests_rel}
changed = [rel for rel, mt in existing.items() if indexed.get(rel) != mt]
unchanged = [rel for rel, mt in existing.items() if indexed.get(rel) == mt]
protected = set(existing) | ({interests_rel} if (vault / interests_rel).is_file() else set())
deleted = sorted(indexed_all.keys() - protected)
if deleted:
await self.file_catalog.delete(deleted)
state = DreamState(
date=day,
hint=hint,
daily_dir=daily,
vault=str(vault),
files_scanned=len(existing),
files_unchanged=len(unchanged),
files_changed=len(changed),
files_deleted=len(deleted),
changed_paths=changed,
unchanged_paths=unchanged,
deleted_paths=deleted,
existing=existing,
indexed=indexed,
)
if not changed:
return self._finish(state, True, f"No changed dream input for {day}")
if not llm_available(self):
state.errors.append("no llm configured; dream extract requires an LLM")
return self._finish(state, False, state.errors[-1])
result = await self.agent_wrapper.reply(
self.prompt_format(
"extract_user_message",
date=day,
hint=hint or "(none)",
changed_paths_json=json.dumps(changed, ensure_ascii=False, indent=2),
material_blob=pack_paths(vault, changed),
),
system_prompt=self.prompt_format("extract_system_prompt", vault_dir=str(vault), buckets=", ".join(BUCKETS)),
job_tools=list(_TOOLS),
)
meta = parse_structured_reply(str(result.get("result") or ""))
self._clean_output(state, meta)
state.extract_summary = str(result.get("result") or "").strip()
answer = f"Extracted {len(state.units)} unit(s), {len(state.topics)} topic(s)"
answer = f"{answer} from {len(changed)} changed file(s)"
return self._finish(state, True, answer)
def _existing(self, vault, files: list[str]) -> dict[str, float]:
out: dict[str, float] = {}
for rel in files:
try:
out[rel] = (vault / rel).stat().st_mtime
except OSError as e:
self.logger.error(f"[{self.name}] stat failed on {rel}: {e}")
return out
def _clean_output(self, state: DreamState, meta: dict) -> None:
allowed = set(state.changed_paths)
for raw in meta.get("units") or meta.get("memory_units") or []:
if not isinstance(raw, dict):
continue
name = str(raw.get("name") or "").strip()
summary = str(raw.get("summary") or "").strip()
bucket = str(raw.get("bucket") or "").strip()
paths = clean_paths(raw.get("paths"), allowed)
if not name or not summary or not paths:
continue
if bucket not in BUCKETS:
self.logger.warning(f"[{self.name}] unit {name!r} emitted bucket {bucket!r}; routing to wiki")
bucket = "wiki"
state.units.append({"name": name, "bucket": bucket, "summary": summary, "paths": paths})
for raw in meta.get("topics") or []:
topic = self._clean_topic(raw, allowed)
if topic:
state.topics.append(topic)
@staticmethod
def _clean_topic(raw, allowed: set[str]) -> dict:
if not isinstance(raw, dict):
return {}
title = str(raw.get("title") or "").strip()
reason = str(raw.get("reason") or "").strip()
paths = clean_paths(raw.get("paths"), allowed)
if not title or not reason or not paths:
return {}
keywords = raw.get("keywords") or []
cleaned_keywords = [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else []
return {
"title": title,
"reason": reason,
"evidence": str(raw.get("evidence") or "").strip(),
"keywords": cleaned_keywords,
"paths": paths,
}
def _finish(self, state: DreamState, success: bool, answer: str):
assert self.context is not None
state.summary = answer
store_state(self, state)
self.context.response.success = success
self.context.response.answer = answer
return self.context.response

177
reme4/steps/evolve/dream/extract.yaml Normal file
View file

@ -0,0 +1,177 @@
extract_system_prompt: |
You are the dream global extraction agent. Read all changed daily files
together and emit a compact cross-file plan: merged memory units and daily
interest topic candidates.
vault_dir: {vault_dir}
buckets: {buckets}
Digest is the abstract memory layer. Raw detail stays in daily notes. Digest
keeps reusable principles, patterns, precedents, workflows, conventions, and
user/team preferences a future agent should recall.
## What to Extract
- **Reusable memory units**: durable abstractions worth integrating into digest.
- **Daily interest candidates**: topics the user may care about seeing again.
- **Cross-file merges**: one unit may gather evidence from several paths.
- **No raw summaries**: do not summarize every file or every event.
## Unit Rules
- One unit = one abstraction the material teaches. One unit maps to exactly
one digest node; a downstream integration agent will make one write
decision for it.
- This extraction step is the gate for "not worth memorizing". Once a unit is
emitted, it is expected to be integrated; do not emit weak candidates for
someone else to skip.
- Merge evidence from multiple files when it teaches the same abstraction.
- Prefer fewer, richer units over exhaustive file summaries.
- Split only when the abstractions would be recalled in different future
situations, or when they will evolve independently as more material arrives.
- When in doubt, merge related evidence into one unit or drop the weaker
candidate entirely.
- Each unit must have name, bucket, summary, and paths.
- paths must only contain values from changed_paths.
- Unknown bucket is invalid; if unsure, use wiki.
- Do not emit passing mentions, known-concept recaps, umbrella event units,
one-off timestamps, attendance facts, or facts with no reusable value.
- summary should name the abstraction, explain why it matters, and point at
the supporting evidence; do not quote or summarize a note.
- If no changed material teaches a reusable abstraction, return an empty
`units` list. Topic candidates may still be non-empty.
## Bucket Rules
- **procedure**: how to do something; workflows, runbooks, recipes, methods.
- **personal**: user/team/project-specific identity, preferences, conventions,
constraints, avoidances.
- **wiki**: general knowledge, principles, decisions-as-precedent, observations.
Straddling two buckets: choose by center of gravity, meaning where a future
reader would search from:
- "User prefers small PRs" -> personal.
- "Small PRs are easier to review" -> wiki.
- "Steps to split a large PR" -> procedure.
## Topic Rules
- Emit topics the user may care about, not generic labels.
- Each topic must include title, reason, evidence, keywords, and paths.
- paths must only contain values from changed_paths.
- Prefer concrete, recurring, or actionable interests over broad categories.
## Tool Boundary
You may use read only for inline wikilinks that materially affect extraction.
## Output Format
Return only one YAML or JSON object with this exact shape:
units:
- name: <short-name>
bucket: procedure|personal|wiki
summary: <grounded reusable abstraction>
paths: [<changed path>, ...]
topics:
- title: <specific user-interest topic>
reason: <why it matters>
evidence: <short evidence pointer>
keywords: [<keyword>, ...]
paths: [<changed path>, ...]
extract_system_prompt_zh: |
你是 dream 全局抽取 agent。一起阅读所有发生变化的 daily 文件，输出一个精简的跨文件计划：
合并后的记忆 unit，以及 daily interest topic 候选。
vault_dir: {vault_dir}
buckets: {buckets}
Digest 是抽象记忆层。原始细节保留在 daily notes 中；digest 只保存未来 agent 应该记住的可复用原则、
模式、先例、工作流、约定，以及用户/团队偏好。
## 抽取什么
- **可复用记忆 unit**：值得整合进 digest 的长期抽象。
- **Daily interest 候选**：用户之后可能还会关心、值得保留的主题。
- **跨文件合并**：一个 unit 可以合并来自多个 path 的证据。
- **不要原文摘要**：不要逐文件总结，也不要逐事件总结。
## Unit 规则
- 一个 unit = 材料教导的一个抽象。一个 unit 恰好对应一个 digest node；下游整合 agent 会对它做一次写入决策。
- 这个抽取步骤是“不值得记忆”的过滤闸口。一旦输出 unit，就默认它应该被整合；不要把弱候选交给别人去跳过。
- 多个文件表达同一个抽象时，要合并成同一个 unit。
- 宁可输出更少但信息密度更高的 unit，不要穷举文件摘要。
- 只有当两个抽象会在未来不同场景被召回，或会随着更多材料独立演化时，才拆成两个 unit。
- 不确定时，把相关证据合并成一个 unit，或直接丢掉较弱候选。
- 每个 unit 必须包含 name、bucket、summary、paths。
- paths 只能使用 changed_paths 中出现的值。
- Unknown bucket 无效；不确定时使用 wiki。
- 不要输出 passing mention、已知概念复述、事件 umbrella unit、一次性时间戳、参会事实、
或没有复用价值的事实。
- summary 应命名抽象、解释它为什么重要，并指向支持证据；不要只是摘抄或总结笔记。
- 如果 changed material 没有教导任何可复用抽象，返回空 `units` list；topic 候选仍然可以非空。
## Bucket 规则
- **procedure**：如何做某事；工作流、runbook、配方、方法。
- **personal**：用户/团队/项目特定的身份、偏好、约定、约束、避让点。
- **wiki**：通用知识、原则、作为先例的决策、观察。
横跨两个 bucket 时，按 center of gravity 选择，也就是未来读者会从哪里搜索：
- “用户偏好小 PR” -> personal。
- “小 PR 更容易 review” -> wiki。
- “如何拆分大 PR 的步骤” -> procedure。
## Topic 规则
- 输出用户可能真正关心的 topic，而不是泛泛的标签。
- 每个 topic 必须包含 title、reason、evidence、keywords、paths。
- paths 只能使用 changed_paths 中出现的值。
- 优先选择具体、反复出现、可行动的兴趣，而不是宽泛类别。
## 工具边界
只有当 inline wikilink 会实质影响抽取时，才可以使用 read。
## 输出格式
只返回一个 YAML 或 JSON object，结构必须严格如下：
units:
- name: <短名称>
bucket: procedure|personal|wiki
summary: <基于证据的可复用抽象>
paths: [<changed path>, ...]
topics:
- title: <具体的用户兴趣 topic>
reason: <为什么重要>
evidence: <简短证据指针>
keywords: [<关键词>, ...]
paths: [<changed path>, ...]
extract_user_message: |
date: {date}
hint: {hint}
changed_paths:
{changed_paths_json}
# Changed material
{material_blob}
Extract merged memory units and daily interest topic candidates.
extract_user_message_zh: |
日期：{date}
提示：{hint}
changed_paths:
{changed_paths_json}
# 变化内容
{material_blob}
抽取合并后的记忆 units 和 daily interest topic 候选。

67
reme4/steps/evolve/dream/finish.py Normal file
View file

@ -0,0 +1,67 @@
"""Dream catalog persistence step."""
from pathlib import Path
from ...base_step import BaseStep
from ....components import R
from ....schema import FileNode
from .schema import DreamState
from .utils import state_from_context, store_state, vault_dir
@R.register("dream_finish_step")
class DreamFinishStep(BaseStep):
"""Persist dream catalog and render final auto-dream response."""
def __init__(self, persist: bool = True, **kwargs):
super().__init__(**kwargs)
self.persist = persist
async def execute(self):
assert self.context is not None
state = state_from_context(self)
vault = Path(state.vault).resolve() if state.vault else vault_dir(self)
if self.file_catalog is None:
raise RuntimeError("dream_finish_step requires file_catalog")
checkpoint = [p for p in state.changed_paths if p not in set(state.failed_paths)]
upsert_paths = checkpoint + [p for p in [state.interests_path, f"{state.daily_dir}/{state.date}.md"] if p]
upserts = self._nodes(vault, upsert_paths)
if upserts:
await self.file_catalog.upsert(upserts)
if self.persist and (upserts or state.deleted_paths):
await self.file_catalog.dump()
state.checkpoint_paths = [n.path for n in upserts if n.path in checkpoint]
state.summary = render_summary(state)
store_state(self, state)
self.context.response.success = not state.failed_units and not state.errors
self.context.response.answer = state.summary
return self.context.response
@staticmethod
def _nodes(vault: Path, paths: list[str]) -> list[FileNode]:
out: list[FileNode] = []
for rel in paths:
try:
out.append(FileNode(path=rel, st_mtime=(vault / rel).stat().st_mtime))
except OSError:
continue
return out
def render_summary(state: DreamState) -> str:
"""Render summary."""
lines = [
f"[AutoDream] date={state.date} scanned={state.files_scanned} changed={state.files_changed} "
f"unchanged={state.files_unchanged} deleted={state.files_deleted}",
f" - extract: {len(state.units)} unit(s), {len(state.topics)} topic(s)",
f" - integrate: {len(state.integrate_results)} ok, {len(state.failed_units)} failed",
f" - topics: {state.topics_written} written" + (f" to {state.interests_path}" if state.interests_path else ""),
f" - catalog: checkpointed {len(state.checkpoint_paths)} changed path(s)",
]
if state.failed_paths:
lines.append(f" - failed paths: {', '.join(state.failed_paths)}")
if state.errors:
lines.append(f" - errors: {'; '.join(state.errors)}")
return "\n".join(lines)

86
reme4/steps/evolve/dream/integrate.py Normal file
View file

@ -0,0 +1,86 @@
"""Dream unit integration step."""
import json
from pathlib import Path
from ...base_step import BaseStep
from ....components import R
from .schema import BUCKETS, IntegrateOutcome
from .utils import llm_available, pack_paths, parse_structured_reply, state_from_context, store_state, vault_dir
_TOOLS = ("node_search", "read", "frontmatter_read", "write", "edit", "frontmatter_update")
@R.register("dream_integrate_step")
class DreamIntegrateStep(BaseStep):
"""Integrate each extracted unit into digest memory."""
async def execute(self):
assert self.context is not None
state = state_from_context(self)
if not state.units:
return self._finish(state, True, "No dream units to integrate")
if not llm_available(self):
err = "no llm configured; dream integrate requires an LLM"
state.errors.append(err)
state.failed_units = state.units
state.failed_paths = sorted({p for u in state.units for p in u.get("paths", [])})
return self._finish(state, False, err)
vault = Path(state.vault).resolve() if state.vault else vault_dir(self)
digest_dir = self.config_value("digest_dir")
for i, unit in enumerate(state.units, start=1):
await self._integrate_one(state, unit, i, vault, digest_dir)
state.failed_paths = sorted(set(state.failed_paths))
answer = f"Integrated {len(state.integrate_results)} unit(s); failed {len(state.failed_units)} unit(s)"
return self._finish(state, not state.failed_units, answer)
async def _integrate_one(self, state, unit: dict, index: int, vault: Path, digest_dir: str) -> None:
bucket = unit.get("bucket") if unit.get("bucket") in BUCKETS else "wiki"
paths = [str(p) for p in unit.get("paths", [])]
try:
result = await self.agent_wrapper.reply(
self.prompt_format(
"integrate_user_message",
hint=state.hint or "(none)",
unit_name=unit.get("name", ""),
unit_bucket=bucket,
unit_summary=unit.get("summary", ""),
unit_paths_json=json.dumps(paths, ensure_ascii=False, indent=2),
material_blob=pack_paths(vault, paths),
),
system_prompt=self.prompt_format(
f"integrate_system_prompt_{bucket}",
vault_dir=str(vault),
digest_dir=digest_dir,
bucket=bucket,
),
job_tools=list(_TOOLS),
)
outcome = IntegrateOutcome.model_validate(parse_structured_reply(str(result.get("result") or "")))
except Exception as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
self.logger.error(f"[{self.name}] unit {index}/{len(state.units)} failed: {error}")
state.failed_units.append({**unit, "error": error})
state.failed_paths.extend(path for path in paths if path not in state.failed_paths)
return
state.integrate_results.append(
{
"unit": unit.get("name", ""),
"bucket": bucket,
"paths": paths,
"action": outcome.action,
"target_path": outcome.target_path,
"note": outcome.note,
},
)
(state.nodes_created if outcome.action == "CREATE" else state.nodes_updated).append(outcome.target_path)
def _finish(self, state, success: bool, answer: str):
assert self.context is not None
state.summary = answer
store_state(self, state)
self.context.response.success = success
self.context.response.answer = answer
return self.context.response

510
reme4/steps/evolve/dream/integrate.yaml Normal file
View file

@ -0,0 +1,510 @@
integrate_system_prompt_procedure: |
You are the dream unit integration agent for the procedure bucket. The unit
is a how-to-do-X abstraction: steps, methods, recipes, runbooks, executable
patterns. Integrate exactly one unit into digest memory. Unit -> digest node
is 1:1; no SKIP outcome because the incoming unit has already been selected
as worth remembering.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is not a faithful copy of the material. Details stay in the daily /
resource files; digest holds the reusable procedure a future agent should
recall later.
- Body is short and abstract, usually 50-200 words unless the procedure
genuinely needs more.
- If the draft starts copying paragraphs or narrating the event, it belongs
in the source material, not digest.
- Provenance edges carry details: cite every relevant unit_paths entry with
`derived_from:: [[<path>]]`.
- Digest-to-digest wikilinks carry the conceptual graph.
## Procedure Body Shape
A runbook, not a recap:
- Trigger / when to use: one line.
- Steps: numbered or terse bullets; each step is verb-led.
- Pre-conditions / inputs: short list, not prose.
- Failure modes / caveats: brief.
- `derived_from:: [[<material-path>]]`: at least one, and normally every
relevant path in unit_paths. Plain-prose provenance does NOT count; only
wikilinks survive future updates.
## Workflow
1. Recall first with node_search across digest buckets. Use queries that
fit the unit's trigger, verbs, nouns, synonyms, and likely failure modes.
Use `limit=20-30` for broad coverage; one call usually suffices, but issue
more if the unit spans multiple concept dimensions. Recall feeds both
dedup and synapse decisions.
2. Use read/frontmatter_read only for likely matches. Internally classify
each recalled node:
- same_abstraction: same trigger and substantially overlapping steps;
this is the UPDATE target.
- related: adjacent procedure, sub-step, prerequisite, failure mode,
concept, or preference worth linking in the body.
- unrelated: ignore.
3. Choose exactly one action:
- CREATE: no same_abstraction hit; write `{digest_dir}/procedure/<slug>.md`.
- CORROBORATE: same procedure observed again; append derived_from and
optionally strengthen wording.
- REFINE: new pre-condition, edge case, failure mode, scope, or step;
expand the relevant span or slot the step into the right position.
- CORRECT: wrong order, missing critical step, bad outcome, or conflict;
tighten or annotate inline with `> note: contradicted by [[<path>]] - <one-line>`.
4. Weave related digest nodes into the body as wikilinks on both CREATE and
UPDATE. UPDATE must be additive: never remove existing wikilinks or
derived_from entries. Default to weaving more, not less; this is the only
chance to attach recalled related nodes.
## Wikilink Graph
- Source provenance links point from digest back to material:
`derived_from:: [[daily/<date>/<session>.md]]` or
`derived_from:: [[resource/<path>]]`.
- Procedure nodes may link to any digest bucket:
`[[{digest_dir}/procedure/<slug>.md]]`,
`[[{digest_dir}/personal/<slug>.md]]`, or
`[[{digest_dir}/wiki/<slug>.md]]`.
- Useful predicates include `derived_from::`, `relates_to::`,
`depends_on::`, and `blocks_on::`; predicates are open and live outside
the brackets.
- Wikilinks must be full vault-relative paths with `.md`.
- CREATE writes inside `{digest_dir}/procedure/`. UPDATE may target any
bucket if recall legitimately found the same abstraction.
- Use `edit` for body-only changes, and make edits only-add whenever possible:
never remove wikilinks the old span contained.
- Use `frontmatter_update` for frontmatter changes such as description or
`kind: procedure`.
- One target per session. Never edit other nodes sideways.
Return only one YAML or JSON object:
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <digest path written or edited>
note: <short landing summary>
integrate_system_prompt_procedure_zh: |
你是 dream 的 procedure bucket unit 整合 agent。这个 unit 是 how-to-do-X 抽象：步骤、方法、
recipe、runbook、可执行模式。把且只把一个 unit 整合进 digest memory。Unit -> digest node 是 1:1；
没有 SKIP，因为传入的 unit 已经被判定为值得记忆。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的可复用流程。
- 正文短且抽象，通常 50-200 words；只有流程本身确实需要时才更长。
- 如果草稿开始复制段落或叙述事件，说明细节放错层了。
- Provenance edge 承载细节：用 `derived_from:: [[<path>]]` 引用 unit_paths 中每个相关来源。
- Digest 之间的 wikilink 承载概念图。
## Procedure 正文形态
写 runbook，不写 recap：
- Trigger / when to use：一行。
- Steps：编号或短 bullet；每步以动词开头。
- Pre-conditions / inputs：短列表，不写长 prose。
- Failure modes / caveats：简短。
- `derived_from:: [[<material-path>]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
## 工作流
1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖 unit 的触发条件、动词、名词、同义词和可能的
failure mode；使用 `limit=20-30` 做较宽召回。通常一次足够；如果 unit 跨多个概念维度，可以多次召回。
Recall 同时服务 dedup 和 synapse。
2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类：
- same_abstraction：触发条件相同，步骤实质重叠；这是 UPDATE 目标。
- related：相邻流程、子步骤、前置条件、失败模式、概念或偏好，值得写入正文 wikilink。
- unrelated：忽略。
3. 选择且只选择一个 action：
- CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/procedure/<slug>.md`。
- CORROBORATE：同一流程再次出现；追加 derived_from，可选强化措辞。
- REFINE：新增前置条件、边界情况、失败模式、适用范围或步骤；扩展相关段落或把步骤插到正确位置。
- CORRECT：顺序错误、缺关键步骤、结果不好或有冲突；收紧表述，或用
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
## Wikilink 图
- 来源 provenance 从 digest 指回材料：
`derived_from:: [[daily/<date>/<session>.md]]` 或 `derived_from:: [[resource/<path>]]`。
- Procedure 节点可以链接任意 digest bucket：
`[[{digest_dir}/procedure/<slug>.md]]`、`[[{digest_dir}/personal/<slug>.md]]`、
`[[{digest_dir}/wiki/<slug>.md]]`。
- 常用 predicate：`derived_from::`、`relates_to::`、`depends_on::`、`blocks_on::`；
predicate 词表开放，写在方括号外。
- Wikilink 必须是带 `.md` 的 vault-relative path。
- CREATE 写入 `{digest_dir}/procedure/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。
- `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。
- `frontmatter_update` 才能改 frontmatter，例如 description 或 `kind: procedure`。
- 每次 session 只处理一个 target。不要顺手编辑其他节点。
只返回一个 YAML 或 JSON object：
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <写入或编辑的 digest path>
note: <简短落地总结>
integrate_system_prompt_personal: |
You are the dream unit integration agent for the personal bucket. The unit is
user/team/project-specific: identity, preference, convention, avoid-rule,
collaboration style, or constraint. Integrate exactly one unit into digest
memory. Unit -> digest node is 1:1; no SKIP outcome because the incoming
unit has already been selected as worth remembering.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is not a faithful copy of the material. Details stay in the daily /
resource files; digest holds the durable rule, identity, convention, or
preference a future agent should recall later.
- Body is short and operational, usually 50-200 words.
- Do not narrate what the user said in detail; cite source material instead.
- Provenance edges carry details: cite every relevant unit_paths entry with
`derived_from:: [[<path>]]`.
- Digest-to-digest wikilinks carry the conceptual graph.
## Personal Body Shape
A short rule of engagement, not a biography:
- Rule / fact: one sentence stating the preference, convention, identity
claim, constraint, or avoid-rule.
- `Why:` reason or context that helps judge edge cases.
- `How to apply:` contexts, tasks, boundaries, or exceptions.
- Do not invent exceptions or soften a hard preference unless the source
material explicitly supports that exception.
- `derived_from:: [[<material-path>]]`: at least one, and normally every
relevant path in unit_paths. Plain-prose provenance does NOT count; only
wikilinks survive future updates.
For preferences, prefer one node per preference rather than one large person
node; that is the granularity downstream search will hit.
## Workflow
1. Recall first with node_search across digest buckets. Use user/team/project
names plus rule, preference, convention, scope, and avoid-rule keywords.
Use `limit=20-30` for broad coverage; issue more calls if the rule has
multiple actor or scope dimensions. Recall feeds both dedup and synapse
decisions.
2. Use read/frontmatter_read only for likely matches. Internally classify
each recalled node:
- same_abstraction: same actor scope and same governing rule; this is the
UPDATE target.
- related: adjacent rule, contrasting preference, identity node, workflow,
or concept worth linking in the body.
- unrelated: ignore.
3. Choose exactly one action:
- CREATE: no same_abstraction hit; write `{digest_dir}/personal/<slug>.md`.
- CORROBORATE: the rule is reaffirmed; append derived_from and optionally
strengthen confidence.
- REFINE: scope, condition, exception, or example changed; expand `How to apply:`.
- CORRECT: the user/team changed their mind or evidence conflicts; tighten
to what both old and new evidence support, or annotate inline with
`> note: contradicted by [[<path>]] - <one-line>`.
4. Weave related digest nodes into the body as wikilinks on both CREATE and
UPDATE. UPDATE must be additive: never remove existing wikilinks or
derived_from entries. Default to weaving more, not less; this is the only
chance to attach recalled related nodes.
## Wikilink Graph
- Source provenance links point from digest back to material:
`derived_from:: [[daily/<date>/<session>.md]]` or
`derived_from:: [[resource/<path>]]`.
- Personal nodes may link to any digest bucket:
`[[{digest_dir}/personal/<slug>.md]]`,
`[[{digest_dir}/procedure/<slug>.md]]`, or
`[[{digest_dir}/wiki/<slug>.md]]`.
- Useful predicates include `derived_from::`, `applies_to::`,
`relates_to::`, `depends_on::`, and `contradicts::`; predicates are open
and live outside the brackets.
- Wikilinks must be full vault-relative paths with `.md`.
- CREATE writes inside `{digest_dir}/personal/`. UPDATE may target any bucket
if recall legitimately found the same abstraction.
- Use `edit` for body-only changes, and make edits only-add whenever possible:
never remove wikilinks the old span contained.
- Use `frontmatter_update` for frontmatter changes such as description or
`kind: preference`.
- One target per session. Never edit other nodes sideways.
Return only one YAML or JSON object:
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <digest path written or edited>
note: <short landing summary>
integrate_system_prompt_personal_zh: |
你是 dream 的 personal bucket unit 整合 agent。这个 unit 是用户/团队/项目特定内容：身份、
偏好、约定、avoid-rule、协作风格或约束。把且只把一个 unit 整合进 digest memory。Unit -> digest node
是 1:1；没有 SKIP，因为传入的 unit 已经被判定为值得记忆。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的长期规则、身份、
约定或偏好。
- 正文短且可操作，通常 50-200 words。
- 不要详细复述用户说了什么；用来源材料承载细节。
- Provenance edge 承载细节：用 `derived_from:: [[<path>]]` 引用 unit_paths 中每个相关来源。
- Digest 之间的 wikilink 承载概念图。
## Personal 正文形态
写短规则，不写 biography：
- Rule / fact：一句话说明偏好、约定、身份事实、约束或 avoid-rule。
- `Why:` 原因或上下文，帮助未来判断边界情况。
- `How to apply:` 适用上下文、任务、边界或例外。
- 不要凭空添加例外，也不要软化明确偏好；只有来源材料明确支持时才写例外。
- `derived_from:: [[<material-path>]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
偏好类内容优先一条偏好一个 node，而不是一个人一个大 node；这是下游搜索更容易命中的粒度。
## 工作流
1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖用户/团队/项目名称、规则、偏好、约定、适用范围、
avoid-rule 关键词；使用 `limit=20-30` 做较宽召回。如果规则有多个 actor 或 scope 维度，可以多次召回。
Recall 同时服务 dedup 和 synapse。
2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类：
- same_abstraction：同一 actor scope + 同一 governing rule；这是 UPDATE 目标。
- related：相邻规则、相反偏好、身份节点、工作流或概念，值得写入正文 wikilink。
- unrelated：忽略。
3. 选择且只选择一个 action：
- CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/personal/<slug>.md`。
- CORROBORATE：规则被再次确认；追加 derived_from，可选强化置信度。
- REFINE：scope、条件、例外或例子变化；扩展 `How to apply:`。
- CORRECT：用户/团队改变主意或证据冲突；收紧到新旧证据都支持的表述，或用
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
## Wikilink 图
- 来源 provenance 从 digest 指回材料：
`derived_from:: [[daily/<date>/<session>.md]]` 或 `derived_from:: [[resource/<path>]]`。
- Personal 节点可以链接任意 digest bucket：
`[[{digest_dir}/personal/<slug>.md]]`、`[[{digest_dir}/procedure/<slug>.md]]`、
`[[{digest_dir}/wiki/<slug>.md]]`。
- 常用 predicate：`derived_from::`、`applies_to::`、`relates_to::`、`depends_on::`、
`contradicts::`；predicate 词表开放，写在方括号外。
- Wikilink 必须是带 `.md` 的 vault-relative path。
- CREATE 写入 `{digest_dir}/personal/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。
- `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。
- `frontmatter_update` 才能改 frontmatter，例如 description 或 `kind: preference`。
- 每次 session 只处理一个 target。不要顺手编辑其他节点。
只返回一个 YAML 或 JSON object：
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <写入或编辑的 digest path>
note: <简短落地总结>
integrate_system_prompt_wiki: |
You are the dream unit integration agent for the wiki bucket. The unit is
general knowledge: definition, principle, observation, decision-as-precedent,
factual claim, or mental model. Wiki is also the catch-all when nothing more
specific fits. Integrate exactly one unit into digest memory. Unit -> digest
node is 1:1; no SKIP outcome because the incoming unit has already been
selected as worth remembering.
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest is the abstract memory layer
Digest is not a faithful copy of the material. Details stay in the daily /
resource files; digest holds the definition, principle, observation, or
precedent a future agent should recall later.
- Body is short and abstract, usually 50-200 words unless the concept
genuinely needs more.
- If the draft starts copying paragraphs or narrating the event, it belongs
in the source material, not digest.
- Provenance edges carry details: cite every relevant unit_paths entry with
`derived_from:: [[<path>]]`.
- Digest-to-digest wikilinks carry the conceptual graph.
## Wiki Body Shape
Encyclopedia-flavored, not narrative:
- First line: one-sentence definition or claim.
- Body: short paragraphs or tight bullets with properties, sub-claims,
distinctions, and one-line examples.
- Relations: typed wikilinks where the relation has semantic weight; most
cross-node links can stay bare.
- `derived_from:: [[<material-path>]]`: at least one, and normally every
relevant path in unit_paths. Plain-prose provenance does NOT count; only
wikilinks survive future updates.
## Workflow
1. Recall first with node_search across digest buckets. Use noun phrases,
synonyms, abbreviations, super/sub-concepts, and contrasting terms. Use
`limit=20-30` for broad coverage; issue more calls when the abstraction
has multiple aspects worth querying separately. Recall feeds both dedup
and synapse decisions.
2. Use read/frontmatter_read only for likely matches. Internally classify
each recalled node:
- same_abstraction: same definition, principle, observation, or precedent;
this is the UPDATE target.
- related: adjacent concept, contrast, super/sub-concept, instance,
procedure, or preference worth linking in the body.
- unrelated: ignore.
3. Choose exactly one action:
- CREATE: no same_abstraction hit; write `{digest_dir}/wiki/<slug>.md`.
- CORROBORATE: the principle is reaffirmed by a new instance; append
derived_from and optionally strengthen wording.
- REFINE: nuance, scope, edge case, or framing changes; tighten the
relevant span. Body grows in precision, not detail volume.
- CORRECT: factual contradiction or overstatement; tighten to the narrower
supported form, or annotate inline with
`> note: contradicted by [[<path>]] - <one-line>`.
4. Weave related digest nodes into the body as wikilinks on both CREATE and
UPDATE. UPDATE must be additive: never remove existing wikilinks or
derived_from entries. Default to weaving more, not less; this is the only
chance to attach recalled related nodes.
## Wikilink Graph
- Source provenance links point from digest back to material:
`derived_from:: [[daily/<date>/<session>.md]]` or
`derived_from:: [[resource/<path>]]`.
- Wiki nodes may link to any digest bucket:
`[[{digest_dir}/wiki/<slug>.md]]`,
`[[{digest_dir}/procedure/<slug>.md]]`, or
`[[{digest_dir}/personal/<slug>.md]]`.
- Useful predicates include `derived_from::`, `is_a::`, `extends::`,
`depends_on::`, `relates_to::`, and `contradicts::`; predicates are open
and live outside the brackets. Most cross-node links can stay bare unless
the relation has clear semantic weight.
- Wikilinks must be full vault-relative paths with `.md`.
- CREATE writes inside `{digest_dir}/wiki/`. UPDATE may target any bucket if
recall legitimately found the same abstraction.
- Use `edit` for body-only changes, and make edits only-add whenever possible:
never remove wikilinks the old span contained.
- Use `frontmatter_update` for frontmatter changes such as description,
`kind: concept`, or `kind: observation`.
- One target per session. Never edit other nodes sideways.
Return only one YAML or JSON object:
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <digest path written or edited>
note: <short landing summary>
integrate_system_prompt_wiki_zh: |
你是 dream 的 wiki bucket unit 整合 agent。这个 unit 是通用知识：定义、原则、观察、作为先例的决策、
事实主张或 mental model。Wiki 也是无法更具体分类时的兜底。把且只把一个 unit 整合进 digest memory。
Unit -> digest node 是 1:1；没有 SKIP，因为传入的 unit 已经被判定为值得记忆。
vault_dir: {vault_dir}
digest_dir: {digest_dir}
## Digest 是抽象记忆层
Digest 不是材料复刻。细节留在 daily / resource 文件；digest 保存未来 agent 应该回忆的定义、原则、
观察或先例。
- 正文短且抽象，通常 50-200 words；只有概念本身确实需要时才更长。
- 如果草稿开始复制段落或叙述事件，说明细节放错层了。
- Provenance edge 承载细节：用 `derived_from:: [[<path>]]` 引用 unit_paths 中每个相关来源。
- Digest 之间的 wikilink 承载概念图。
## Wiki 正文形态
写 encyclopedia 风格，不写 narrative：
- First line：一句话定义或主张。
- Body：短段落或紧凑 bullets，写属性、子主张、区分和一行例子。
- Relations：有明确语义重量时使用 typed wikilink；大多数 cross-node link 可以裸写。
- `derived_from:: [[<material-path>]]`：至少一条，通常覆盖 unit_paths 中每个相关 path。
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
## 工作流
1. 先跨 digest bucket 使用 node_search 召回。查询要覆盖名词短语、同义词、缩写、上位/下位概念和对比词；
使用 `limit=20-30` 做较宽召回。如果抽象有多个值得分别查询的 aspect，可以多次召回。Recall 同时服务
dedup 和 synapse。
2. 只对明显可能匹配的节点使用 read/frontmatter_read。内部把召回节点分类：
- same_abstraction：同一定义、原则、观察或先例；这是 UPDATE 目标。
- related：相邻概念、对比概念、上位/下位概念、实例、流程或偏好，值得写入正文 wikilink。
- unrelated：忽略。
3. 选择且只选择一个 action：
- CREATE：没有 same_abstraction 命中；写入 `{digest_dir}/wiki/<slug>.md`。
- CORROBORATE：原则被新实例再次确认；追加 derived_from，可选强化措辞。
- REFINE：nuance、scope、edge case 或 framing 改变；收紧相关段落。正文增长的是精度，不是细节量。
- CORRECT：事实冲突或过度概括；收紧到更窄且有支持的表述，或用
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
wikilink 或 derived_from。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
## Wikilink 图
- 来源 provenance 从 digest 指回材料：
`derived_from:: [[daily/<date>/<session>.md]]` 或 `derived_from:: [[resource/<path>]]`。
- Wiki 节点可以链接任意 digest bucket：
`[[{digest_dir}/wiki/<slug>.md]]`、`[[{digest_dir}/procedure/<slug>.md]]`、
`[[{digest_dir}/personal/<slug>.md]]`。
- 常用 predicate：`derived_from::`、`is_a::`、`extends::`、`depends_on::`、
`relates_to::`、`contradicts::`；predicate 词表开放，写在方括号外。有明确语义重量时才加
predicate，大多数 cross-node link 可以裸写。
- Wikilink 必须是带 `.md` 的 vault-relative path。
- CREATE 写入 `{digest_dir}/wiki/`。如果 recall 合法命中同一抽象，UPDATE 可以跨 bucket。
- `edit` 只改正文，并尽量只增不删：不要删除 old span 中已有的 wikilink。
- `frontmatter_update` 才能改 frontmatter，例如 description、`kind: concept` 或 `kind: observation`。
- 每次 session 只处理一个 target。不要顺手编辑其他节点。
只返回一个 YAML 或 JSON object：
action: CREATE|CORROBORATE|REFINE|CORRECT
target_path: <写入或编辑的 digest path>
note: <简短落地总结>
integrate_user_message: |
hint: {hint}
unit_name: {unit_name}
unit_bucket: {unit_bucket}
unit_summary: {unit_summary}
unit_paths:
{unit_paths_json}
# Evidence
{material_blob}
Integrate this single unit into digest memory. Cite every relevant unit_paths
entry with `derived_from:: [[<path>]]`, recall related digest nodes, and weave
useful digest wikilinks into the target node.
integrate_user_message_zh: |
提示：{hint}
unit_name: {unit_name}
unit_bucket: {unit_bucket}
unit_summary: {unit_summary}
unit_paths:
{unit_paths_json}
# 证据
{material_blob}
将这个单独的 unit 整合进 digest memory。用 `derived_from:: [[<path>]]` 引用 unit_paths 中每个相关来源，
召回相关 digest 节点，并把有用的 digest wikilink 织入目标节点。

43
reme4/steps/evolve/dream/proactive.py Normal file
View file

@ -0,0 +1,43 @@
"""Read daily interests.yaml for proactive use."""
from ...base_step import BaseStep
from ....components import R
from .schema import ProactiveResult
from .utils import load_yaml_topics, today, vault_dir
@R.register("proactive_step")
class ProactiveStep(BaseStep):
"""Read ``daily/<date>/interests.yaml``."""
def __init__(self, include_content: bool = True, **kwargs):
super().__init__(**kwargs)
self.include_content = include_content
async def execute(self):
assert self.context is not None
day = today(self, str(self.context.get("date", "") or ""))
include_content = bool(self.context.get("include_content", self.include_content))
daily = self.config_value("daily_dir")
rel_path, abs_path = f"{daily}/{day}/interests.yaml", vault_dir(self) / daily / day / "interests.yaml"
result = ProactiveResult(date=day, path=rel_path)
if not abs_path.is_file():
result.skipped, result.summary = True, f"Skipped: interests file not found at {rel_path}"
return self._finish(True, result)
try:
result.content = abs_path.read_text(encoding="utf-8") if include_content else ""
result.topics = load_yaml_topics(abs_path)
except Exception as e: # noqa: BLE001
result.error, result.summary = f"{type(e).__name__}: {e}", ""
return self._finish(False, result)
result.summary = f"Read {len(result.topics)} proactive topic(s) from {rel_path}"
return self._finish(True, result)
def _finish(self, success: bool, result: ProactiveResult):
assert self.context is not None
self.context.response.success = success
self.context.response.answer = result.summary if success else f"Error: {result.error}"
self.context.response.metadata.update(result.model_dump())
return self.context.response

92
reme4/steps/evolve/dream/schema.py Normal file
View file

@ -0,0 +1,92 @@
"""Shared auto-dream schemas."""
from typing import Literal
from pydantic import BaseModel, Field
BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki")
Bucket = Literal["procedure", "personal", "wiki"]
class DreamUnit(BaseModel):
"""One cross-file memory unit emitted by global extract."""
name: str = Field(description="Short kebab-case handle for the abstraction.")
bucket: str = Field(description="procedure, personal, or wiki; unknown values route to wiki.")
summary: str = Field(description="Grounded abstraction summary with evidence pointers.")
paths: list[str] = Field(default_factory=list, description="Vault-relative source paths.")
class DreamTopic(BaseModel):
"""One topic candidate emitted by global extract."""
title: str = Field(description="Specific user-interest topic title.")
reason: str = Field(description="Why this topic may interest the user.")
evidence: str = Field(description="Grounded evidence pointer.")
keywords: list[str] = Field(default_factory=list, description="Keywords for de-duplication.")
paths: list[str] = Field(default_factory=list, description="Vault-relative source paths.")
class DreamExtractOutput(BaseModel):
"""Structured output for ``dream_extract_step``."""
units: list[DreamUnit] = Field(default_factory=list)
topics: list[DreamTopic] = Field(default_factory=list)
class IntegrateOutcome(BaseModel):
"""Structured output for one unit integration."""
action: Literal["CREATE", "CORROBORATE", "REFINE", "CORRECT"] = Field(description="Write decision.")
target_path: str = Field(description="Digest path written or edited.")
note: str = Field(default="", description="Short summary of what landed.")
class TopicSelectionOutput(BaseModel):
"""Structured output for daily topic selection."""
topics: list[DreamTopic] = Field(default_factory=list)
class ProactiveResult(BaseModel):
"""Result of reading daily interest topics."""
date: str = ""
path: str = ""
topics: list[dict] = Field(default_factory=list)
content: str = ""
skipped: bool = False
error: str = ""
summary: str = ""
class DreamState(BaseModel):
"""Shared state passed across the four dream steps."""
date: str = ""
hint: str = ""
daily_dir: str = ""
vault: str = ""
files_scanned: int = 0
files_unchanged: int = 0
files_changed: int = 0
files_deleted: int = 0
changed_paths: list[str] = Field(default_factory=list)
unchanged_paths: list[str] = Field(default_factory=list)
deleted_paths: list[str] = Field(default_factory=list)
existing: dict[str, float] = Field(default_factory=dict)
indexed: dict[str, float] = Field(default_factory=dict)
units: list[dict] = Field(default_factory=list)
topics: list[dict] = Field(default_factory=list)
extract_summary: str = ""
integrate_results: list[dict] = Field(default_factory=list)
nodes_created: list[str] = Field(default_factory=list)
nodes_updated: list[str] = Field(default_factory=list)
failed_units: list[dict] = Field(default_factory=list)
failed_paths: list[str] = Field(default_factory=list)
interests_path: str = ""
topics_written: int = 0
topic_error: str = ""
checkpoint_paths: list[str] = Field(default_factory=list)
errors: list[str] = Field(default_factory=list)
summary: str = ""

132
reme4/steps/evolve/dream/topics.py Normal file
View file

@ -0,0 +1,132 @@
"""Daily interests.yaml step."""
import json
from pathlib import Path
from ...base_step import BaseStep
from ...file_io import refresh_day_index
from ....components import R
from .utils import (
load_yaml_topics,
llm_available,
normalize_topic,
parse_structured_reply,
previous_dates,
state_from_context,
store_state,
vault_dir,
write_yaml,
)
@R.register("dream_topics_step")
class DreamTopicsStep(BaseStep):
"""Write ``daily/<date>/interests.yaml`` with same-day and recent de-dup."""
def __init__(self, topic_count: int = 3, topic_diversity_days: int = 7, **kwargs):
super().__init__(**kwargs)
self.topic_count = topic_count
self.topic_diversity_days = topic_diversity_days
async def execute(self):
assert self.context is not None
state = state_from_context(self)
topic_count = int(self.context.get("topic_count", self.topic_count) or self.topic_count)
raw_days = self.context.get("topic_diversity_days", self.topic_diversity_days)
diversity_days = int(raw_days or self.topic_diversity_days)
vault = Path(state.vault).resolve() if state.vault else vault_dir(self)
rel_path = f"{state.daily_dir}/{state.date}/interests.yaml"
abs_path = vault / state.daily_dir / state.date / "interests.yaml"
same_day = load_yaml_topics(abs_path)
if not state.topics:
state.interests_path = rel_path if abs_path.is_file() else ""
state.topics_written = len(same_day)
answer = f"Kept {len(same_day)} existing interest topic(s) at {rel_path}"
answer = answer if abs_path.is_file() else "Skipped interests.yaml write: no new topic candidates"
return self._finish(state, True, answer)
recent = [
topic
for day in previous_dates(state.date, diversity_days)
for topic in load_yaml_topics(vault / state.daily_dir / day / "interests.yaml")
]
try:
topics, _used_llm = await self._select_topics(state, same_day, recent, topic_count, diversity_days)
payload = {
"date": state.date,
"topic_count": topic_count,
"diversity_days": diversity_days,
"topics": topics,
}
write_yaml(abs_path, payload)
await refresh_day_index(self.file_store, state.date, state.daily_dir)
state.interests_path, state.topics_written = rel_path, len(topics)
return self._finish(state, True, f"Wrote {len(topics)} interest topic(s) to {rel_path}")
except Exception as e: # noqa: BLE001
state.topic_error = f"{type(e).__name__}: {e}"
state.errors.append(state.topic_error)
return self._finish(state, False, f"Error: {state.topic_error}")
async def _select_topics(self, state, same_day: list[dict], recent: list[dict], count: int, days: int):
if not state.topics:
return self._dedupe([], same_day, recent, count), False
if not llm_available(self):
return self._dedupe(state.topics, same_day, recent, count), False
result = await self.agent_wrapper.reply(
self.prompt_format(
"topics_user_message",
date=state.date,
topic_count=count,
diversity_days=days,
candidates_json=json.dumps(state.topics, ensure_ascii=False, indent=2),
same_day_json=json.dumps(same_day, ensure_ascii=False, indent=2),
recent_topics_json=json.dumps(recent, ensure_ascii=False, indent=2),
),
system_prompt=self.prompt_format("topics_system_prompt"),
)
meta = parse_structured_reply(str(result.get("result") or ""))
selected = [self._clean_topic(t) for t in meta.get("topics") or []]
if not any(selected):
selected = state.topics
return self._dedupe(selected, same_day, recent, count), True
@staticmethod
def _clean_topic(raw) -> dict:
if not isinstance(raw, dict):
return {}
title, reason = str(raw.get("title") or "").strip(), str(raw.get("reason") or "").strip()
if not title or not reason:
return {}
keywords, paths = raw.get("keywords") or [], raw.get("paths") or []
cleaned_keywords = [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else []
cleaned_paths = [str(p).strip() for p in paths if str(p).strip()] if isinstance(paths, list) else []
return {
"title": title,
"reason": reason,
"evidence": str(raw.get("evidence") or "").strip(),
"keywords": cleaned_keywords,
"paths": cleaned_paths,
}
@staticmethod
def _dedupe(topics: list[dict], same_day: list[dict], recent: list[dict], count: int) -> list[dict]:
recent_norm = {normalize_topic(t.get("title", "")) for t in recent}
seen = {normalize_topic(t.get("title", "")) for t in same_day}
out = list(same_day)
for topic in [t for t in topics if t]:
title_norm = normalize_topic(topic.get("title", ""))
if title_norm and title_norm not in seen and title_norm not in recent_norm:
seen.add(title_norm)
out.append(topic)
if len(out) >= count:
break
return out[:count]
def _finish(self, state, success: bool, answer: str):
assert self.context is not None
state.summary = answer
store_state(self, state)
self.context.response.success = success
self.context.response.answer = answer
return self.context.response

107
reme4/steps/evolve/dream/topics.yaml Normal file
View file

@ -0,0 +1,107 @@
topics_system_prompt: |
You select final daily user-interest topics from dream candidates for
daily/<date>/interests.yaml.
## Goals
- Preserve existing same-day topics unless they are clear duplicates.
- Avoid duplicates within today.
- Avoid topics already covered in recent interests.yaml files.
- Prefer concrete, recurring, or actionable interests over file summaries.
- Return no more than topic_count topics.
## Topic Quality
A good topic is something the user may want surfaced again: a live area of
attention, a recurring concern, a research direction, a project thread, or a
practical follow-up. Avoid generic labels, daily log summaries, and topics
whose only value is restating a file name.
## Output Rules
- Keep titles concise and specific.
- Keep reason grounded in candidate evidence.
- Keep keywords short and retrieval-friendly.
- Keep paths limited to the paths supplied by the candidates.
- Quote scalar strings that contain punctuation such as `:` or use block
scalars (`>`). Write paths as block lists, not flow lists, so
`daily/<date>/...` stays parseable.
Return only one YAML or JSON object:
topics:
- title: <specific topic>
reason: <why it matters>
evidence: <short evidence pointer>
keywords: [<keyword>, ...]
paths: [<source path>, ...]
topics_system_prompt_zh: |
你负责从 dream 候选中选择最终 daily user-interest topics，用于写入 daily/<date>/interests.yaml。
## 目标
- 保留同一天已有的 topics，除非它们明显重复。
- 避免当天内部重复。
- 避免重复最近 interests.yaml 已经覆盖过的 topics。
- 优先选择具体、反复出现、可行动的兴趣，而不是文件摘要。
- 返回数量不能超过 topic_count。
## Topic 质量
好的 topic 应该是用户未来可能希望再次看到的内容：持续关注的方向、反复出现的问题、研究方向、
项目线索或实际 follow-up。避免泛泛标签、daily log 摘要，以及只是在复述文件名的 topic。
## 输出规则
- title 要简短、具体。
- reason 要基于候选证据。
- keywords 要短，方便检索。
- paths 只能使用候选中提供的 paths。
- 如果字符串包含 `:` 等标点，要加引号或使用 block scalar (`>`)。
paths 使用 block list，不要用 flow list，避免 `daily/<date>/...` 解析失败。
只返回一个 YAML 或 JSON object：
topics:
- title: <具体 topic>
reason: <为什么重要>
evidence: <简短证据指针>
keywords: [<关键词>, ...]
paths: [<source path>, ...]
topics_user_message: |
date: {date}
topic_count: {topic_count}
diversity_days: {diversity_days}
# Candidate topics
{candidates_json}
# Existing same-day interests.yaml topics
{same_day_json}
# Recent interests.yaml topics to avoid repeating
{recent_topics_json}
Select the final topics for daily interests.yaml.
topics_user_message_zh: |
日期：{date}
topic_count: {topic_count}
diversity_days: {diversity_days}
# 候选 topics
{candidates_json}
# 同一天已有的 interests.yaml topics
{same_day_json}
# 需要避免重复的最近 interests.yaml topics
{recent_topics_json}
选择最终要写入 daily interests.yaml 的 topics。

174
reme4/steps/evolve/dream/utils.py Normal file
View file

@ -0,0 +1,174 @@
"""Shared auto-dream helpers."""
import datetime as dt
import re
from pathlib import Path
import yaml
from .._evolve import now
from ...base_step import BaseStep
from .schema import DreamState
def state_from_context(step: BaseStep) -> DreamState:
"""Get dream state from context."""
assert step.context is not None
raw = step.context.get("dream") or step.context.response.metadata.get("dream") or {}
state = DreamState.model_validate(raw)
if not state.daily_dir:
state.daily_dir = step.config_value("daily_dir")
return state
def store_state(step: BaseStep, state: DreamState) -> None:
"""Store dream state in context."""
assert step.context is not None
data = state.model_dump()
step.context["dream"] = data
step.context.response.metadata["dream"] = data
def vault_dir(step: BaseStep) -> Path:
"""Get vault directory."""
vr = getattr(step.file_store, "vault_path", None)
return Path(vr).resolve() if vr else Path.cwd().resolve()
def daily_dir(step: BaseStep) -> str:
"""Get daily directory."""
return step.config_value("daily_dir")
def today(step: BaseStep, explicit: str = "") -> str:
"""Get today's date."""
if explicit.strip():
return explicit.strip()
tz = step.app_context.app_config.timezone if step.app_context is not None else None
return now(tz).strftime("%Y-%m-%d")
def llm_available(step: BaseStep) -> bool:
"""Check if LLM is available."""
try:
return step.as_llm is not None and step.agent_wrapper is not None
except Exception:
return False
def scan_day_files(vault: Path, day: str, daily: str, interests_name: str = "interests.yaml") -> list[str]:
"""Scan day files."""
out: list[str] = []
day_index = vault / daily / f"{day}.md"
if day_index.is_file():
out.append(str(day_index.relative_to(vault)))
daily_root = vault / daily / day
if daily_root.is_dir():
out.extend(str(p.relative_to(vault)) for p in sorted(daily_root.rglob("*.md")) if p.is_file())
return [p for p in out if p != f"{daily}/{day}/{interests_name}"]
def pack_paths(vault: Path, paths: list[str], *, limit_per_file: int = 60000) -> str:
"""Pack paths into a single string."""
blocks: list[str] = []
for rel in paths:
target = vault / rel
if not target.is_file():
blocks.append(f"### {rel}\n(file not found)\n")
continue
try:
text = target.read_text(encoding="utf-8")
except Exception as e: # noqa: BLE001
blocks.append(f"### {rel}\n(error reading: {type(e).__name__}: {e})\n")
continue
suffix = "\n\n[truncated]\n" if len(text) > limit_per_file else ""
blocks.append(f"### {rel}\n{text[:limit_per_file]}{suffix}\n")
return "\n".join(blocks)
def clean_paths(raw_paths, allowed: set[str]) -> list[str]:
"""Clean paths."""
if not isinstance(raw_paths, list):
return []
out: list[str] = []
for item in raw_paths:
path = str(item or "").strip()
if path in allowed and path not in out:
out.append(path)
return out
def normalize_topic(text: str) -> str:
"""Normalize topic."""
return re.sub(r"[^a-z0-9\u4e00-\u9fff]+", " ", text.lower()).strip()
def previous_dates(day: str, n_days: int) -> list[str]:
"""Get previous dates."""
try:
base = dt.date.fromisoformat(day)
except ValueError:
return []
return [(base - dt.timedelta(days=i)).isoformat() for i in range(1, max(n_days, 0) + 1)]
def load_yaml_topics(path: Path) -> list[dict]:
"""Load YAML topics."""
if not path.is_file():
return []
try:
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
except Exception:
return []
topics = data.get("topics") if isinstance(data, dict) else None
if not isinstance(topics, list):
return []
return [cleaned for t in topics if isinstance(t, dict) and (cleaned := clean_topic(t))]
def clean_topic(raw: dict) -> dict:
"""Clean topic."""
title, reason = str(raw.get("title") or "").strip(), str(raw.get("reason") or "").strip()
if not title or not reason:
return {}
keywords = raw.get("keywords") or []
paths = raw.get("paths") or []
return {
"title": title,
"reason": reason,
"evidence": str(raw.get("evidence") or "").strip(),
"keywords": [str(k).strip() for k in keywords if str(k).strip()] if isinstance(keywords, list) else [],
"paths": [str(p).strip() for p in paths if str(p).strip()] if isinstance(paths, list) else [],
}
def write_yaml(path: Path, payload: dict) -> None:
"""Write YAML."""
path.parent.mkdir(parents=True, exist_ok=True)
rendered = yaml.safe_dump(payload, allow_unicode=True, sort_keys=False)
path.write_text(rendered if rendered.endswith("\n") else f"{rendered}\n", encoding="utf-8")
def parse_structured_reply(text: str) -> dict:
"""Parse a JSON/YAML object from an agent reply, including fenced blocks."""
candidates = [text.strip()]
candidates.extend(m.group(1).strip() for m in re.finditer(r"```(?:json|ya?ml)?\s*(.*?)```", text, re.S | re.I))
for raw in candidates:
if not raw:
continue
try:
data = yaml.safe_load(raw)
except yaml.YAMLError:
data = _parse_scalar_mapping(raw)
if isinstance(data, dict) and data:
return data
return {}
def _parse_scalar_mapping(raw: str) -> dict:
"""Parse a scalar mapping."""
out: dict[str, str] = {}
for line in raw.splitlines():
if match := re.match(r"^\s*(action|target_path|note)\s*:\s*(.+?)\s*$", line):
out[match.group(1)] = match.group(2).strip().strip("\"'")
return out

38
reme4/steps/file_io/__init__.py
View file

@ -0,0 +1,38 @@
"""File I/O step helpers."""
from ._daily_index import refresh_day_index, validate_session_id
from ._file_io import write_file_safe
from .daily_create import DailyCreateStep
from .daily_list import DailyListStep
from .daily_reindex import DailyReindexStep
from .delete import DeleteStep
from .edit import EditStep
from .frontmatter_delete import FrontmatterDeleteStep
from .frontmatter_read import FrontmatterReadStep
from .frontmatter_update import FrontmatterUpdateStep
from .list import ListStep
from .move import MoveStep
from .read import ReadStep
from .read_image import ReadImageStep
from .stat import StatStep
from .write import WriteStep
__all__ = [
"refresh_day_index",
"validate_session_id",
"write_file_safe",
"DailyCreateStep",
"DailyListStep",
"DailyReindexStep",
"DeleteStep",
"EditStep",
"FrontmatterDeleteStep",
"FrontmatterReadStep",
"FrontmatterUpdateStep",
"ListStep",
"MoveStep",
"ReadStep",
"ReadImageStep",
"StatStep",
"WriteStep",
]

42
reme4/steps/file_io/_daily_index.py
View file

@ -4,7 +4,7 @@ from pathlib import Path
import frontmatter
from ._path import validate_filename_component
from ._path import resolve_path, validate_filename_component
from ...utils import get_logger
logger = get_logger()
@ -49,15 +49,15 @@ def scan_notes(vault_dir: Path, date: str, daily_dir: str) -> list[dict]:
{"session_id": str, "path": str, "metadata": dict}
"""
date_dir = vault_dir / daily_dir / date
date_dir, err = resolve_path(vault_dir, f"{daily_dir}/{date}")
if err or date_dir is None:
logger.info(f"scan_notes skipped invalid daily path date={date!r} daily_dir={daily_dir!r} error={err!r}")
return []
if not date_dir.is_dir():
return []
out: list[dict] = []
prefix = "session_agent_"
for md_path in sorted(
p for p in date_dir.iterdir() if p.is_file() and p.suffix == ".md" and p.stem.startswith(prefix)
):
session_id = md_path.stem[len(prefix) :]
for md_path in sorted(p for p in date_dir.iterdir() if p.is_file() and p.suffix == ".md"):
session_id = md_path.stem
try:
post = frontmatter.loads(md_path.read_text(encoding="utf-8"))
except Exception:
@ -79,7 +79,16 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict:
"""
vault_dir = Path(file_store.vault_path or ".").resolve()
index_rel = f"{daily_dir}/{date}.md"
index_abs = vault_dir / index_rel
index_abs, err = resolve_path(vault_dir, index_rel)
if err or index_abs is None:
return {
"date": date,
"path": index_rel,
"notes": [],
"created": False,
"changed": False,
"error": err or "invalid path",
}
notes = scan_notes(vault_dir, date, daily_dir)
notes_payload = [{"path": n["path"], "session_id": n["session_id"], "metadata": n["metadata"]} for n in notes]
@ -90,6 +99,7 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict:
"path": index_rel,
"notes": notes_payload,
"created": False,
"changed": False,
}
notes_block = _render_notes_block(notes)
@ -97,13 +107,15 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict:
n = len(notes)
fm = {"name": date, "description": "No notes today." if n == 0 else f"{n} note(s) today."}
existing_text = ""
if index_abs.is_file():
post = frontmatter.loads(index_abs.read_text(encoding="utf-8"))
existing_text = index_abs.read_text(encoding="utf-8")
post = frontmatter.loads(existing_text)
new_body = _rebuild_body(post.content, notes_block)
merged = dict(post.metadata or {})
for key, value in fm.items():
if not merged.get(key):
merged[key] = value
if not merged.get("name"):
merged["name"] = fm["name"]
merged["description"] = fm["description"]
fm = merged
was_created = False
else:
@ -111,11 +123,15 @@ async def refresh_day_index(file_store, date: str, daily_dir: str) -> dict:
new_body = f"{_NOTES_OPEN}\n{notes_block}\n{_NOTES_CLOSE}\n"
was_created = True
out = frontmatter.Post(new_body, **fm)
index_abs.write_text(frontmatter.dumps(out), encoding="utf-8")
rendered = frontmatter.dumps(out)
changed = not index_abs.is_file() or existing_text != rendered
if changed:
index_abs.write_text(rendered, encoding="utf-8")
return {
"date": date,
"path": index_rel,
"notes": notes_payload,
"created": was_created,
"changed": changed,
}

26
reme4/steps/file_io/_file_io.py
View file

@ -113,6 +113,32 @@ async def read_file_safe(file_path, max_bytes: int = MAX_FILE_READ_BYTES) -> tup
return _decode_known_file(data, Path(file_path).suffix)
async def read_file_lines_safe(
file_path,
start_line: int,
end_line: int | None,
*,
max_collect_bytes: int = DEFAULT_MAX_BYTES * 2,
) -> tuple[str, int, str]:
"""Read a 1-based inclusive line range without loading the full file.
Returns ``(text, total_lines, encoding)``.
"""
encoding = await detect_file_encoding(file_path)
lines: list[str] = []
collected_bytes = 0
total = 0
async with aiofiles.open(str(file_path), "r", encoding=encoding, errors="replace") as f:
async for line in f:
total += 1
if total >= start_line and (end_line is None or total <= end_line):
if collected_bytes < max_collect_bytes:
cleaned = line.rstrip("\n")
lines.append(cleaned)
collected_bytes += len(cleaned.encode(encoding, errors="replace")) + 1
return "\n".join(lines), total, encoding
async def detect_file_encoding(file_path, sniff_bytes: int = 8192) -> str:
"""Detect the encoding of an existing file so writes can preserve it."""
try:

36
reme4/steps/file_io/_path.py
View file

@ -39,10 +39,13 @@ _RESERVED_NAMES = {
}
# pylint: disable=too-many-return-statements
def validate_filename_component(name: str, *, kind: str = "filename") -> str | None:
"""Return an error message, or ``None`` when ``name`` is a safe filename component."""
if not name:
return f"{kind} is required"
if name in (".", ".."):
return f"{kind} cannot be '.' or '..': {name!r}"
if name != name.strip():
return f"{kind} cannot have leading or trailing whitespace: {name!r}"
if _INVALID_CHARS.search(name):
@ -54,25 +57,44 @@ def validate_filename_component(name: str, *, kind: str = "filename") -> str | N
return None
def resolve_path(vault_path: Path, raw: str) -> tuple[Path | None, str | None]:
def is_relative_to(path: Path, parent: Path) -> bool:
"""Return True when ``path`` is equal to or nested under ``parent``."""
try:
path.relative_to(parent)
return True
except ValueError:
return False
# pylint: disable=too-many-return-statements
def resolve_path(
vault_path: Path,
raw: str,
*,
allow_empty: bool = False,
) -> tuple[Path | None, str | None]:
"""Resolve a `path=` argument against ``vault_path``.
Returns ``(abs_path, None)`` on success, or ``(None, error_message)`` on failure.
"""
if not raw or not str(raw).strip():
if allow_empty:
return vault_path.resolve(), None
return None, "`path` is required"
s = str(raw).strip()
p = Path(s)
if p.is_absolute():
logger.info("absolute path detected, recommending relative paths")
return p.resolve(), None
for part in p.parts:
if part == p.anchor:
continue
err = validate_filename_component(part, kind="path component")
if err:
return None, err
if p.is_absolute():
logger.info("absolute path detected, recommending relative paths")
return p, None
return vault_path / p, None
vault = vault_path.resolve()
target = (vault / p).resolve()
if not is_relative_to(target, vault):
return None, "`path` must stay inside the vault"
return target, None
def gate_md(target: Path) -> tuple[Path, bool]:

12
reme4/steps/file_io/daily_create.py
View file

@ -15,7 +15,7 @@ The caller fills the body via ``file_write`` / ``file_edit`` /
does not accept a body.
Inputs:
session_id (optional) — the note's session identifier (also the file stem);
session_id (optional) — the note's session identifier;
empty string → day-level file
date (optional, ``YYYY-MM-DD``; empty = today)
@ -30,6 +30,7 @@ import frontmatter
from ._daily_index import refresh_day_index, validate_session_id
from ._file_io import write_file_safe
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
from ...steps.evolve import now
@ -53,7 +54,7 @@ class DailyCreateStep(BaseStep):
session_id = self.context.get("session_id", "")
tz = self.app_context.app_config.timezone if self.app_context is not None else None
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
daily_dir = self.config_value("daily_dir")
return session_id, day, daily_dir
@staticmethod
@ -86,13 +87,16 @@ class DailyCreateStep(BaseStep):
if err:
self._fail(err)
return None
path_rel = f"{daily_dir}/{day}/session_agent_{session_id}.md"
name = session_id
path_rel = f"{daily_dir}/{day}/{session_id}.md"
else:
path_rel = f"{daily_dir}/{day}.md"
name = day
path_abs = (self.vault_path / path_rel).resolve()
path_abs, err = resolve_path(self.vault_path, path_rel)
if err or path_abs is None:
self._fail(err or "invalid path", date=day, session_id=session_id, path=path_rel)
return None
try:
created = await self._create_if_missing(path_abs, name)
except Exception as e: # pylint: disable=broad-except

10
reme4/steps/file_io/daily_list.py
View file

@ -15,6 +15,7 @@ to today.
from pathlib import Path
from ._daily_index import scan_notes
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
from ...steps.evolve import now
@ -29,7 +30,7 @@ class DailyListStep(BaseStep):
assert self.context is not None
tz = self.app_context.app_config.timezone if self.app_context is not None else None
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
daily_dir = self.config_value("daily_dir")
vault_dir = Path(self.file_store.vault_path or ".").resolve()
return day, daily_dir, vault_dir
@ -57,6 +58,13 @@ class DailyListStep(BaseStep):
"""Scan ``<daily_dir>/<date>/`` and emit one projected record per note."""
assert self.context is not None
day, daily_dir, vault_dir = self._collect_params()
_target_dir, err = resolve_path(vault_dir, f"{daily_dir}/{day}")
if err:
self.context.response.success = False
self.context.response.answer = f"Error: {err}"
self.context.response.metadata.update({"date": day, "error": err})
self.logger.info(f"[{self.name}] date={day} error={err!r}")
return
notes = [self._project(n) for n in scan_notes(vault_dir, day, daily_dir)]
self.context.response.success = True
lines = [self._format_note_line(n) for n in notes]

2
reme4/steps/file_io/daily_reindex.py
View file

@ -34,7 +34,7 @@ class DailyReindexStep(BaseStep):
assert self.context is not None
tz = self.app_context.app_config.timezone if self.app_context is not None else None
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
daily_dir = self.config_value("daily_dir")
return day, daily_dir
def _apply_result(self, refreshed: dict) -> None:

5
reme4/steps/file_io/delete.py
View file

@ -24,6 +24,7 @@ citing prose, or accept dangling.
import shutil
from pathlib import Path
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
from ...utils.wikilink_handler import WikilinkHandler
@ -70,7 +71,9 @@ class DeleteStep(BaseStep):
if not path:
return {"path": path, "error": "not found"}
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target = (vault_dir / path).resolve()
target, err = resolve_path(vault_dir, path)
if err or target is None:
return {"path": path, "error": err or "invalid path"}
if target.is_file():
inbound = await WikilinkHandler.find_inbound(self.file_store, target=path)
target.unlink()

54
reme4/steps/file_io/frontmatter_delete.py
View file

@ -13,6 +13,8 @@ from pathlib import Path
import frontmatter
from ._file_io import get_path_lock
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
@ -30,31 +32,37 @@ class FrontmatterDeleteStep(BaseStep):
keys = [keys]
keys = list(keys)
target = (Path(self.file_store.vault_path or ".") / path).resolve()
if not target.is_file():
payload: dict = {"path": path, "error": "not found"}
elif target.suffix != ".md":
payload = {"path": path, "error": "not markdown"}
elif not keys:
payload = {"path": path, "error": "keys is empty"}
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target, err = resolve_path(vault_dir, path)
if err or target is None:
payload: dict = {"path": path, "error": err or "invalid path"}
else:
post = frontmatter.loads(target.read_text(encoding="utf-8"))
deleted: list[str] = []
missing: list[str] = []
for k in keys:
if k in post.metadata:
del post.metadata[k]
deleted.append(k)
lock = await get_path_lock(target)
async with lock:
if not target.is_file():
payload = {"path": path, "error": "not found"}
elif target.suffix != ".md":
payload = {"path": path, "error": "not markdown"}
elif not keys:
payload = {"path": path, "error": "keys is empty"}
else:
missing.append(k)
if deleted:
target.write_text(frontmatter.dumps(post), encoding="utf-8")
payload = {
"path": path,
"deleted": deleted,
"missing": missing,
"frontmatter": dict(post.metadata),
}
post = frontmatter.loads(target.read_text(encoding="utf-8"))
deleted: list[str] = []
missing: list[str] = []
for k in keys:
if k in post.metadata:
del post.metadata[k]
deleted.append(k)
else:
missing.append(k)
if deleted:
target.write_text(frontmatter.dumps(post), encoding="utf-8")
payload = {
"path": path,
"deleted": deleted,
"missing": missing,
"frontmatter": dict(post.metadata),
}
if "error" in payload:
self.context.response.success = False

10
reme4/steps/file_io/frontmatter_read.py
View file

@ -13,6 +13,7 @@ from pathlib import Path
import frontmatter
import yaml
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
@ -26,7 +27,14 @@ class FrontmatterReadStep(BaseStep):
path: str = self.context.get("path") or ""
assert path, "path is required"
target = (Path(self.file_store.vault_path or ".") / path).resolve()
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target, err = resolve_path(vault_dir, path)
if err or target is None:
self.context.response.success = False
self.context.response.answer = f"Error: {err or 'invalid path'}"
self.context.response.metadata.update({"path": path, "exists": False, "error": err or "invalid path"})
self.logger.info(f"[{self.name}] path={path} error={err!r}")
return
if not target.is_file():
self.context.response.success = False
self.context.response.answer = f"Error: {path} not found"

30
reme4/steps/file_io/frontmatter_update.py
View file

@ -16,6 +16,8 @@ from pathlib import Path
import frontmatter
from ._file_io import get_path_lock
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
@ -31,18 +33,24 @@ class FrontmatterUpdateStep(BaseStep):
metadata = self.context.get("metadata") or {}
assert isinstance(metadata, dict), "metadata must be a dict"
target = (Path(self.file_store.vault_path or ".") / path).resolve()
if not target.is_file():
payload: dict = {"path": path, "error": "not found"}
elif target.suffix != ".md":
payload = {"path": path, "error": "not markdown"}
elif not metadata:
payload = {"path": path, "error": "no fields to update"}
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target, err = resolve_path(vault_dir, path)
if err or target is None:
payload: dict = {"path": path, "error": err or "invalid path"}
else:
post = frontmatter.loads(target.read_text(encoding="utf-8"))
post.metadata.update(metadata)
target.write_text(frontmatter.dumps(post), encoding="utf-8")
payload = {"path": path, "updated": metadata}
lock = await get_path_lock(target)
async with lock:
if not target.is_file():
payload = {"path": path, "error": "not found"}
elif target.suffix != ".md":
payload = {"path": path, "error": "not markdown"}
elif not metadata:
payload = {"path": path, "error": "no fields to update"}
else:
post = frontmatter.loads(target.read_text(encoding="utf-8"))
post.metadata.update(metadata)
target.write_text(frontmatter.dumps(post), encoding="utf-8")
payload = {"path": path, "updated": metadata}
if "error" in payload:
self.context.response.success = False

14
reme4/steps/file_io/list.py
View file

@ -16,6 +16,7 @@ should iterate the result and call ``frontmatter_read`` per candidate.
from pathlib import Path
from typing import Iterable
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
@ -48,14 +49,6 @@ class ListStep(BaseStep):
limit = DEFAULT_LIMIT
return path, recursive, limit if limit > 0 else DEFAULT_LIMIT
@staticmethod
def _resolve_target_dir(vault_dir: Path, path: str) -> Path:
"""Empty → vault root; absolute → as-is; relative → joined under vault_dir."""
if not path:
return vault_dir
candidate = Path(path)
return candidate.resolve() if candidate.is_absolute() else (vault_dir / candidate).resolve()
@staticmethod
def _walk_files(target_dir: Path, recursive: bool, limit: int) -> list[Path]:
"""Return up to ``limit`` regular files under ``target_dir``; short-circuits at the cap."""
@ -84,7 +77,10 @@ class ListStep(BaseStep):
assert self.context is not None
path, recursive, limit = self._collect_params()
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target_dir = self._resolve_target_dir(vault_dir, path)
target_dir, err = resolve_path(vault_dir, path, allow_empty=True)
if err or target_dir is None:
self._fail(err or "invalid path", path=path)
return None
if not target_dir.exists():
self._fail(f"directory {target_dir} does not exist", path=str(target_dir))

11
reme4/steps/file_io/move.py
View file

@ -32,6 +32,7 @@ original is still removed in that case (move semantics, not copy).
import shutil
from pathlib import Path
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
from ...utils.wikilink_handler import WikilinkHandler
@ -65,9 +66,13 @@ class MoveStep(BaseStep):
self.context.response.metadata.update(payload)
async def _move(self, src_path: str, dst_path: str, overwrite: bool, retarget: bool) -> dict:
vault_dir = Path(self.file_store.vault_path or ".")
src_abs = (vault_dir / src_path).resolve() if src_path else None
dst_abs = (vault_dir / dst_path).resolve() if not Path(dst_path).is_absolute() else None
vault_dir = Path(self.file_store.vault_path or ".").resolve()
src_abs, src_err = resolve_path(vault_dir, src_path) if src_path else (None, "src_path is required")
dst_abs, dst_err = resolve_path(vault_dir, dst_path) if dst_path else (None, "dst_path is required")
if src_err:
return {"src_path": src_path, "error": src_err}
if dst_err:
return {"dst_path": dst_path, "error": dst_err}
precheck_error = _precheck_move(src_path, dst_path, src_abs, dst_abs, overwrite)
if precheck_error:
return precheck_error

51
reme4/steps/file_io/read.py
View file

@ -2,10 +2,11 @@
from pathlib import Path
from ._file_io import read_file_safe, truncate_text_output
from ._file_io import read_file_lines_safe, read_file_safe, truncate_text_output
from ._path import NON_MD_WARNING, gate_md, resolve_path
from ..base_step import BaseStep
from ...components import R
from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES
from ...utils import expand_links, render_expansion_lines
@ -88,6 +89,7 @@ class ReadStep(BaseStep):
self._fail(f"read failed: {e}", path=str(target))
return None
# pylint: disable=too-many-return-statements
async def execute(self):
assert self.context is not None
raw = str(self.context.get("path") or "")
@ -104,23 +106,40 @@ class ReadStep(BaseStep):
if not self._check_file(target):
return None
content = await self._load_content(target)
if content is None:
return None
if target.stat().st_size <= MAX_FILE_READ_BYTES:
content = await self._load_content(target)
if content is None:
return None
all_lines = content.split("\n")
total = len(all_lines)
bounds = self._resolve_range(total, start_line, end_line, target)
if bounds is None:
return None
s, e = bounds
all_lines = content.split("\n")
total = len(all_lines)
bounds = self._resolve_range(total, start_line, end_line, target)
if bounds is None:
return None
s, e = bounds
excerpt = "\n".join(all_lines[s - 1 : e])
else:
s = max(1, int(start_line) if start_line is not None else 1)
requested_end = int(end_line) if end_line is not None else None
if requested_end is not None and s > requested_end:
self._fail(f"start_line ({s}) > end_line ({requested_end})", path=str(target))
return None
try:
excerpt, total, _encoding = await read_file_lines_safe(
target,
s,
requested_end,
max_collect_bytes=DEFAULT_MAX_BYTES * 2,
)
except Exception as e: # pylint: disable=broad-except
self._fail(f"read failed: {e}", path=str(target))
return None
if s > total:
self._fail(f"start_line {s} exceeds file length ({total} lines)", path=str(target), total_lines=total)
return None
e = min(total, requested_end if requested_end is not None else total)
text = truncate_text_output(
"\n".join(all_lines[s - 1 : e]),
start_line=s,
total_lines=total,
file_path=str(target),
)
text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target))
self.context.response.success = True
self.context.response.answer = text

10
reme4/steps/file_io/stat.py
View file

@ -24,6 +24,7 @@ from pathlib import Path
import frontmatter
from ._path import resolve_path
from ..base_step import BaseStep
from ...components import R
@ -37,7 +38,14 @@ class StatStep(BaseStep):
path: str = self.context.get("path", "") or ""
assert path, "path is required"
target = (Path(self.file_store.vault_path or ".") / path).resolve()
vault_dir = Path(self.file_store.vault_path or ".").resolve()
target, err = resolve_path(vault_dir, path)
if err or target is None:
self.context.response.success = False
self.context.response.answer = f"Error: {err or 'invalid path'}"
self.context.response.metadata.update({"path": path, "exists": False, "error": err or "invalid path"})
self.logger.info(f"[{self.name}] path={path} error={err!r}")
return
if not target.exists():
self.context.response.success = False
self.context.response.answer = f"stat: {path} not found"

31
reme4/steps/index/__init__.py
View file

@ -0,0 +1,31 @@
"""Index steps."""
from .clear_store import ClearStoreStep
from .log_changes import LogChangesStep
from .node_search import NodeSearchStep
from .init_changes import InitChangesStep
from .search import SearchStep
from .traverse import TraverseStep
from .update_changes import ChangeApplyStep, UpdateCatalogStep, UpdateIndexStep
from .watch_changes import (
DEFAULT_LOW_POWER_POLL_MS,
DEFAULT_WATCH_DEBOUNCE_MS,
DEFAULT_WATCH_STEP_MS,
WatchChangesStep,
)
__all__ = [
"ChangeApplyStep",
"ClearStoreStep",
"DEFAULT_LOW_POWER_POLL_MS",
"DEFAULT_WATCH_DEBOUNCE_MS",
"DEFAULT_WATCH_STEP_MS",
"InitChangesStep",
"LogChangesStep",
"NodeSearchStep",
"SearchStep",
"TraverseStep",
"UpdateCatalogStep",
"UpdateIndexStep",
"WatchChangesStep",
]

50
reme4/steps/index/_change_batch.py Normal file
View file

@ -0,0 +1,50 @@
"""Helpers for the common file change batch shape."""
from collections import OrderedDict
from collections.abc import Callable
from pathlib import Path
from watchfiles import Change
def _normalize_change(raw) -> Change | None:
if isinstance(raw, Change):
return raw
if isinstance(raw, str):
return Change.__members__.get(raw)
return None
def coalesce_changes(changes: list[dict], path_exists: Callable[[str], bool] | None = None) -> list[dict]:
"""Collapse duplicate path events to the final on-disk state."""
by_path: OrderedDict[str, set[Change]] = OrderedDict()
for item in changes:
if not isinstance(item, dict) or "path" not in item:
continue
change = _normalize_change(item.get("change"))
if change not in (Change.added, Change.modified, Change.deleted):
continue
by_path.setdefault(item["path"], set()).add(change)
def exists(path: str) -> bool:
return path_exists(path) if path_exists is not None else Path(path).is_file()
result: list[dict] = []
for path, seen in by_path.items():
if not exists(path):
result.append({"change": Change.deleted.name, "path": path})
elif seen == {Change.added}:
result.append({"change": Change.added.name, "path": path})
else:
result.append({"change": Change.modified.name, "path": path})
return result
def bucket_changes(changes: list[dict], path_exists: Callable[[str], bool] | None = None) -> dict[Change, list[str]]:
"""Group changes by watchfiles.Change."""
buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []}
for item in coalesce_changes(changes, path_exists=path_exists):
change = _normalize_change(item["change"])
if isinstance(change, Change) and change in buckets:
buckets[change].append(item["path"])
return buckets

18
reme4/steps/index/_watch_rules.py
View file

@ -1,4 +1,4 @@
"""Shared watch-rule logic for scan_changes and watch_changes steps."""
"""Shared watch-rule logic for init_changes and watch_changes steps."""
from dataclasses import dataclass, field
from pathlib import Path
@ -6,6 +6,7 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from ...schema import ApplicationConfig
from ...components.runtime_context import RuntimeContext
@dataclass
@ -31,6 +32,21 @@ def build_watch_rules(
return rules
def build_context_watch_rules(
app_config: "ApplicationConfig | None",
vault_path: Path,
context: "RuntimeContext",
) -> list[WatchRule]:
"""Build watch rules from context-level watch_dirs/watch_suffixes."""
if app_config is None:
return []
watch_dirs: list[str] = context.get("watch_dirs", [])
watch_suffixes: list[str] = context.get("watch_suffixes", [])
if not watch_dirs:
return []
return build_watch_rules(app_config, vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes)
def collect_existing(rules: list[WatchRule], recursive: bool) -> dict[str, float]:
"""Walk rule paths and return {abs_path: st_mtime} for matching files."""
existing: dict[str, float] = {}

30
reme4/steps/index/clear_and_scan.py
View file

@ -1,30 +0,0 @@
"""Wipe the file store and emit every vault file as an ``added`` change.
Designed to be chained before ``update_index_step`` so that the second step
performs the actual re-indexing and persistence.
"""
from ..base_step import BaseStep
from ...components import R
@R.register("clear_and_scan_step")
class ClearAndScanStep(BaseStep):
"""Clear the file store, walk the vault, and write changes into the context."""
async def execute(self):
assert self.context is not None
suffixes = tuple("." + s.strip(".") for s in self.context.get("suffix_filters", ["md"]))
await self.file_store.clear()
paths = [
str(p.absolute())
for p in self.vault_path.rglob("*")
if p.is_file() and (not suffixes or str(p).endswith(suffixes))
]
self.context["changes"] = [{"change": "added", "path": p} for p in paths]
counts = {"added": len(paths), "modified": 0, "deleted": 0}
self.context.response.metadata["counts"] = counts
self.logger.info(f"[{self.name}] cleared store and scanned {len(paths)} file(s)")
return self.context.response

16
reme4/steps/index/clear_store.py Normal file
View file

@ -0,0 +1,16 @@
"""Clear the file store before reusing the standard init change producer."""
from ..base_step import BaseStep
from ...components import R
@R.register("clear_store_step")
class ClearStoreStep(BaseStep):
"""Wipe ``file_store`` so ``init_changes_step(store=file_store)`` sees all files as added."""
async def execute(self):
assert self.context is not None
await self.file_store.clear()
self.context.response.metadata["cleared_store"] = True
self.logger.info(f"[{self.name}] cleared file_store")
return self.context.response

27
reme4/steps/index/foreach_dispatch.py
View file

@ -1,27 +0,0 @@
"""Foreach dispatch: iterate changes and call a configured job per file."""
from ..base_step import BaseStep
from ...components import R
@R.register("foreach_dispatch_step")
class ForeachDispatchStep(BaseStep):
"""For each change item, call ``dispatch_job`` with the vault-relative path."""
async def execute(self):
assert self.context is not None
changes: list[dict] = self.context.get("changes") or []
dispatch_job: str = self.context.get("dispatch_job", "")
if not dispatch_job:
self.logger.warning(f"[{self.name}] no dispatch_job configured, skip")
self.context.response.success = True
return self.context.response
for item in changes:
rel_path = self.to_vault_relative(item["path"])
try:
await self.run_job(dispatch_job, file_path=rel_path, change=item["change"])
except Exception:
self.logger.exception(f"[{self.name}] dispatch {dispatch_job} failed: {rel_path}")
self.context.response.success = True
self.context.response.metadata["dispatched"] = len(changes)
return self.context.response

80
reme4/steps/index/init_changes.py Normal file
View file

@ -0,0 +1,80 @@
"""One-shot change producer: diff watched files against file_store/file_catalog."""
from pathlib import Path
from typing import Iterable
from ._change_batch import coalesce_changes
from ._watch_rules import WatchRule, build_context_watch_rules, collect_existing
from ..base_step import BaseStep
from ...components import R
from ...schema import FileNode
@R.register("init_changes_step")
class InitChangesStep(BaseStep):
"""Scan once, write ``context["changes"]``, then dispatch change handlers."""
def __init__(
self,
monitor_type: str | None = None,
monitor_name: str = "default",
store: str | None = None,
recursive: bool = True,
**kwargs,
):
super().__init__(**kwargs)
self.monitor_type = monitor_type or store
self.monitor_name = monitor_name
self.recursive = recursive
if self.monitor_type in {"file_store", "file_catalog"}:
self.kwargs.setdefault(self.monitor_type, monitor_name)
def _get_watch_rules(self) -> list[WatchRule]:
assert self.context is not None
app_config = self.app_context.app_config if self.app_context else None
return build_context_watch_rules(app_config, self.vault_path, self.context)
async def _load_indexed_nodes(self) -> Iterable[FileNode]:
if self.monitor_type == "file_store":
return await self.file_store.get_nodes()
if self.monitor_type == "file_catalog":
if self.file_catalog is None:
raise RuntimeError("file_catalog is not initialized!")
return await self.file_catalog.get_nodes()
raise ValueError("init_changes_step.monitor_type must be 'file_store' or 'file_catalog'")
@staticmethod
def diff(existing: dict[str, float], nodes: Iterable[FileNode], vault_path: Path) -> tuple[
list[dict],
dict[str, int],
]:
"""Compute added/modified/deleted vs ``nodes`` and return (changes, counts)."""
indexed: dict[str, float] = {
str(Path(n.path) if Path(n.path).is_absolute() else vault_path / n.path): n.st_mtime for n in nodes
}
to_delete = list(indexed.keys() - existing.keys())
to_add = list(existing.keys() - indexed.keys())
to_modify = [p for p in existing.keys() & indexed.keys() if existing[p] != indexed[p]]
changes: list[dict] = (
[{"change": "added", "path": p} for p in to_add]
+ [{"change": "modified", "path": p} for p in to_modify]
+ [{"change": "deleted", "path": p} for p in to_delete]
)
counts = {"added": len(to_add), "modified": len(to_modify), "deleted": len(to_delete)}
return changes, counts
async def execute(self):
assert self.context is not None
rules = self._get_watch_rules()
existing = collect_existing(rules, recursive=self.recursive)
nodes = await self._load_indexed_nodes()
changes, counts = self.diff(existing, nodes, self.vault_path)
changes = coalesce_changes(changes)
self.context["changes"] = changes
if changes:
self.logger.info(f"[{self.name}] scan {self.monitor_type}:{self.monitor_name}: {counts}")
await self.dispatch_steps(self.dispatch_step_specs, changes=changes)
else:
self.logger.info(f"[{self.name}] {self.monitor_type}:{self.monitor_name} is up to date")
self.context.response.metadata["counts"] = counts
return self.context.response

2
reme4/steps/index/node_search.py
View file

@ -72,7 +72,7 @@ class NodeSearchStep(BaseStep):
return self.context.response
assert limit > 0, f"limit must be positive, got {limit}"
digest_dir = getattr(self.app_context.app_config, "digest_dir", "digest") or "digest"
digest_dir = self.config_value("digest_dir")
digest_prefix = digest_dir.rstrip("/") + "/"
# Over-fetch — digest filter drops a lot of raw hits.

97
reme4/steps/index/scan_changes.py
View file

@ -1,97 +0,0 @@
"""One-shot scan: diff watch_paths vs an indexed-state source, write changes into context.
Two symmetric variants — pick the one whose state source matches the loop's
write target so the index loop and the dream loop never contend on the
same component:
* :class:`ScanStoreChangesStep` (``scan_store_changes_step``) — diffs
against ``file_store``; used by ``index_update_loop`` (sole writer of
``file_store``).
* :class:`ScanCatalogChangesStep` (``scan_catalog_changes_step``) — diffs
against ``file_catalog``; used by ``resource_watch_loop`` and
``digest_watch_loop`` (sole writers of ``file_catalog``).
Both share the same diff vocabulary (``added`` / ``modified`` / ``deleted``)
and write into ``context["changes"]`` in the same shape, so downstream
steps don't care which variant produced the batch.
"""
from pathlib import Path
from typing import Iterable
from ._watch_rules import WatchRule, build_watch_rules, collect_existing
from ..base_step import BaseStep
from ...components import R
from ...schema import FileNode
def _diff(existing: dict[str, float], nodes: Iterable[FileNode], vault_path: Path) -> tuple[list[dict], dict[str, int]]:
"""Compute added/modified/deleted vs ``nodes`` and return (changes, counts)."""
indexed: dict[str, float] = {
str(Path(n.path) if Path(n.path).is_absolute() else vault_path / n.path): n.st_mtime for n in nodes
}
to_delete = list(indexed.keys() - existing.keys())
to_add = list(existing.keys() - indexed.keys())
to_modify = [p for p in existing.keys() & indexed.keys() if existing[p] != indexed[p]]
changes: list[dict] = (
[{"change": "added", "path": p} for p in to_add]
+ [{"change": "modified", "path": p} for p in to_modify]
+ [{"change": "deleted", "path": p} for p in to_delete]
)
counts = {"added": len(to_add), "modified": len(to_modify), "deleted": len(to_delete)}
return changes, counts
class _ScanChangesBase(BaseStep):
"""Shared scaffolding: collect on-disk state, defer node-loading to the subclass."""
def __init__(self, recursive: bool = True, **kwargs):
super().__init__(**kwargs)
self.recursive: bool = recursive
def _get_watch_rules(self) -> list[WatchRule]:
"""Build watch rules from context-level watch_dirs/watch_suffixes."""
assert self.context is not None
app_config = self.app_context.app_config if self.app_context else None
if app_config is None:
return []
watch_dirs: list[str] = self.context.get("watch_dirs", [])
watch_suffixes: list[str] = self.context.get("watch_suffixes", [])
if not watch_dirs:
return []
return build_watch_rules(app_config, self.vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes)
async def _load_indexed_nodes(self) -> Iterable[FileNode]:
raise NotImplementedError
async def execute(self):
assert self.context is not None
rules = self._get_watch_rules()
existing = collect_existing(rules, recursive=self.recursive)
nodes = await self._load_indexed_nodes()
changes, counts = _diff(existing, nodes, self.vault_path)
self.context["changes"] = changes
if changes:
self.logger.info(f"[{self.name}] scan: {counts}")
else:
self.logger.info(f"[{self.name}] store is up to date")
self.context.response.metadata["counts"] = counts
return self.context.response
@R.register("scan_store_changes_step")
class ScanStoreChangesStep(_ScanChangesBase):
"""Diff vault against ``file_store``; used by the index loop."""
async def _load_indexed_nodes(self) -> Iterable[FileNode]:
if self.file_store is None:
raise RuntimeError("file_store is not initialized!")
return await self.file_store.get_nodes()
@R.register("scan_catalog_changes_step")
class ScanCatalogChangesStep(_ScanChangesBase):
"""Diff vault against ``file_catalog``; used by resource/digest loops."""
async def _load_indexed_nodes(self) -> Iterable[FileNode]:
return await self.file_catalog.get_nodes()

83
reme4/steps/index/update_catalog.py
View file

@ -1,83 +0,0 @@
"""Update file catalog with a batch of file changes."""
from pathlib import Path
from watchfiles import Change
from ..base_step import BaseStep
from ...components import R
from ...schema import FileNode
@R.register("update_catalog_step")
class UpdateCatalogStep(BaseStep):
"""Classify raw watcher changes and update the file_catalog."""
def __init__(self, persist: bool = False, **kwargs):
super().__init__(**kwargs)
self._persist: bool = persist
async def execute(self):
assert self.context is not None
changes: list[dict] = self.context.get("changes") or []
persist: bool = self._persist or bool(self.context.get("persist", False))
buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []}
for item in changes:
c = item["change"]
if isinstance(c, str):
c = Change.__members__.get(c)
if isinstance(c, Change) and c in buckets:
buckets[c].append(item["path"])
results: list[dict] = []
for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")):
paths = buckets[change]
if not paths:
continue
self.logger.info(f"Detected {len(paths)} {change.name} file(s)")
nodes: list[FileNode] = []
ok_paths: list[str] = []
for path in paths:
abs_path = Path(path)
if not abs_path.is_file():
results.append({"change": change.name, "path": path, "success": False, "error": "not a file"})
continue
self.logger.info(f"{action} file: {path}")
try:
stat = abs_path.stat()
nodes.append(FileNode(path=self.to_vault_relative(abs_path), st_mtime=stat.st_mtime))
ok_paths.append(path)
except Exception as e:
self.logger.exception(f"Failed to stat {path}")
results.append({"change": change.name, "path": path, "success": False, "error": str(e)})
if nodes:
try:
await self.file_catalog.delete([n.path for n in nodes])
await self.file_catalog.upsert(nodes)
results.extend({"change": change.name, "path": p, "success": True} for p in ok_paths)
except Exception as e:
self.logger.exception(f"Failed to upsert {len(nodes)} {change.name} file(s)")
results.extend(
{"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths
)
if deleted := buckets[Change.deleted]:
if self.file_catalog is None:
raise RuntimeError("file_catalog is not initialized!")
self.logger.info(f"Detected {len(deleted)} deleted file(s)")
rel_deleted = [self.to_vault_relative(p) for p in deleted]
try:
await self.file_catalog.delete(rel_deleted)
results.extend({"change": "deleted", "path": p, "success": True} for p in deleted)
except Exception as e:
self.logger.exception(f"Failed to delete {len(deleted)} file(s)")
results.extend({"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted)
if persist and results:
await self.file_catalog.dump()
self.context.response.answer = results
self.context.response.success = all(r["success"] for r in results) if results else True
return self.context.response

179
reme4/steps/index/update_changes.py Normal file
View file

@ -0,0 +1,179 @@
"""Apply file change batches to file_catalog or file_store."""
from abc import abstractmethod
from pathlib import Path
from typing import Any
from watchfiles import Change
from ._change_batch import bucket_changes
from ..base_step import BaseStep
from ...components import R
from ...components.file_chunker import BaseFileChunker
from ...enumeration import ComponentEnum
from ...schema import FileChunk, FileNode
class ChangeApplyStep(BaseStep):
"""Shared added/modified/deleted handling for index update targets."""
target_name = "target"
def __init__(self, persist: bool | None = None, **kwargs):
super().__init__(**kwargs)
self.persist = persist
@abstractmethod
async def build_item(self, path: Path) -> Any:
"""Parse one existing file into the target item shape."""
@abstractmethod
def item_path(self, item: Any) -> str:
"""Return the target-relative path for an upsert item."""
@abstractmethod
async def upsert_items(self, items: list[Any]) -> None:
"""Upsert parsed items into the target."""
@abstractmethod
async def delete_paths(self, paths: list[str]) -> None:
"""Delete target-relative paths from the target."""
@abstractmethod
async def dump_target(self) -> None:
"""Persist the target."""
async def execute(self):
assert self.context is not None
changes: list[dict] = self.context.get("changes") or []
persist = bool(self.context.get("persist", True)) if self.persist is None else self.persist
buckets = bucket_changes(changes, path_exists=lambda p: self._to_abs_path(p).is_file())
results = await self._apply_existing(buckets)
results.extend(await self._apply_deleted(buckets[Change.deleted]))
if persist and results:
await self.dump_target()
self.context.response.answer = results
self.context.response.success = all(r["success"] for r in results) if results else True
return self.context.response
async def _apply_existing(self, buckets: dict[Change, list[str]]) -> list[dict]:
results: list[dict] = []
for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")):
paths = buckets[change]
if not paths:
continue
self.logger.info(f"Detected {len(paths)} {change.name} file(s)")
items, ok_paths = [], []
for path in paths:
item = await self._try_build_item(change, action, path, results)
if item is not None:
items.append(item)
ok_paths.append(path)
if items:
results.extend(await self._try_upsert(change, items, ok_paths))
return results
async def _try_build_item(self, change: Change, action: str, path: str, results: list[dict]):
abs_path = self._to_abs_path(path)
if not abs_path.is_file():
results.append({"change": change.name, "path": path, "success": False, "error": "not a file"})
return None
self.logger.info(f"{action} file: {path}")
try:
return await self.build_item(abs_path)
except Exception as e:
self.logger.exception(f"Failed to parse {path}")
results.append({"change": change.name, "path": path, "success": False, "error": str(e)})
return None
async def _try_upsert(self, change: Change, items: list[Any], ok_paths: list[str]) -> list[dict]:
try:
await self.delete_paths([self.item_path(item) for item in items])
await self.upsert_items(items)
return [{"change": change.name, "path": p, "success": True} for p in ok_paths]
except Exception as e:
self.logger.exception(f"Failed to upsert {len(items)} {change.name} file(s) into {self.target_name}")
return [{"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths]
async def _apply_deleted(self, deleted: list[str]) -> list[dict]:
if not deleted:
return []
self.logger.info(f"Detected {len(deleted)} deleted file(s)")
try:
await self.delete_paths([self.to_vault_relative(p) for p in deleted])
return [{"change": "deleted", "path": p, "success": True} for p in deleted]
except Exception as e:
self.logger.exception(f"Failed to delete {len(deleted)} file(s) from {self.target_name}")
return [{"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted]
def _to_abs_path(self, path: str | Path) -> Path:
p = Path(path)
return p if p.is_absolute() else self.vault_path / p
@R.register("update_catalog_step")
class UpdateCatalogStep(ChangeApplyStep):
"""Update file_catalog with a batch of file changes."""
target_name = "file_catalog"
async def build_item(self, path: Path) -> FileNode:
stat = path.stat()
return FileNode(path=self.to_vault_relative(path), st_mtime=stat.st_mtime)
def item_path(self, item: FileNode) -> str:
return item.path
async def upsert_items(self, items: list[FileNode]) -> None:
if self.file_catalog is None:
raise RuntimeError("file_catalog is not initialized!")
await self.file_catalog.upsert(items)
async def delete_paths(self, paths: list[str]) -> None:
if self.file_catalog is None:
raise RuntimeError("file_catalog is not initialized!")
await self.file_catalog.delete(paths)
async def dump_target(self) -> None:
if self.file_catalog is not None:
await self.file_catalog.dump()
@R.register("update_index_step")
class UpdateIndexStep(ChangeApplyStep):
"""Update file_store with a batch of file changes."""
target_name = "file_store"
async def build_item(self, path: Path) -> tuple[FileNode, list[FileChunk]]:
return await self.chunk_file(path)
def item_path(self, item: tuple[FileNode, list[FileChunk]]) -> str:
return item[0].path
async def upsert_items(self, items: list[tuple[FileNode, list[FileChunk]]]) -> None:
await self.file_store.upsert(items)
async def delete_paths(self, paths: list[str]) -> None:
await self.file_store.delete(paths)
async def dump_target(self) -> None:
await self.file_store.dump()
async def chunk_file(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]:
"""Chunk a file into (node, chunks)."""
if self.app_context is None:
raise RuntimeError("app_context is not set when resolving file chunker")
chunker = self._resolve_chunker(Path(path))
return await chunker.chunk(path)
def _resolve_chunker(self, path: Path) -> BaseFileChunker:
"""Resolve a file chunker for a given path."""
chunkers: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER]
suffix = path.suffix.lstrip(".").lower()
for candidate in chunkers.values():
if suffix and suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}:
return candidate
if default := chunkers.get("default"):
return default
raise RuntimeError(f"No file chunker supports {path} (suffix={suffix!r}) and no default chunker is configured")

119
reme4/steps/index/update_index.py
View file

@ -1,119 +0,0 @@
"""Update index with a batch of file changes."""
from pathlib import Path
from watchfiles import Change
from ..base_step import BaseStep
from ...components import R
from ...components.file_chunker import BaseFileChunker
from ...enumeration import ComponentEnum
from ...schema import FileChunk, FileNode
@R.register("update_index_step")
class UpdateIndexStep(BaseStep):
"""Classify raw watcher changes and update the file_store index."""
def __init__(self, persist: bool = False, **kwargs):
super().__init__(**kwargs)
self.persist: bool = persist
async def chunk_file(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]:
"""Chunk ``path`` using the file chunker whose ``supported_extensions`` claims its suffix.
First registered match wins (config insertion order). Falls back to the
``default`` chunker when no chunker claims the suffix.
"""
if self.app_context is None:
raise RuntimeError("app_context is not set when resolving file chunker")
chunker_dict: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER]
suffix = Path(path).suffix.lstrip(".").lower()
chunker: BaseFileChunker | None = None
if suffix:
for candidate in chunker_dict.values():
if suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}:
chunker = candidate
break
if chunker is None:
chunker = chunker_dict.get("default")
if chunker is None:
raise RuntimeError(
f"No file chunker supports {path} (suffix={suffix!r}) and no 'default' chunker is configured",
)
return await chunker.chunk(path)
async def execute(self):
assert self.context is not None
# Each item: {"change": Change | "added"|"modified"|"deleted", "path": absolute path}
changes: list[dict] = self.context.get("changes") or []
buckets: dict[Change, list[str]] = {Change.added: [], Change.modified: [], Change.deleted: []}
for item in changes:
c = item["change"]
if isinstance(c, str):
c = Change.__members__.get(c)
if isinstance(c, Change) and c in buckets:
buckets[c].append(item["path"])
results: list[dict] = []
for change, action in ((Change.added, "Adding"), (Change.modified, "Updating")):
paths = buckets[change]
if not paths:
continue
self.logger.info(f"Detected {len(paths)} {change.name} file(s)")
parsed: list[tuple[FileNode, list[FileChunk]]] = []
ok_paths: list[str] = []
for path in paths:
abs_path = Path(path)
if not abs_path.is_file():
results.append({"change": change.name, "path": path, "success": False, "error": "not a file"})
continue
self.logger.info(f"{action} file: {path}")
try:
parsed.append(await self.chunk_file(abs_path))
ok_paths.append(path)
except Exception as e:
self.logger.exception(f"Failed to parse {path}")
results.append({"change": change.name, "path": path, "success": False, "error": str(e)})
if parsed:
try:
await self.file_store.delete([n.path for n, _ in parsed])
await self.file_store.upsert(parsed)
results.extend({"change": change.name, "path": p, "success": True} for p in ok_paths)
except Exception as e:
self.logger.exception(f"Failed to persist {len(parsed)} {change.name} file(s)")
results.extend(
{"change": change.name, "path": p, "success": False, "error": str(e)} for p in ok_paths
)
if deleted := buckets[Change.deleted]:
if self.file_store is None:
raise RuntimeError("file_store is not initialized!")
self.logger.info(f"Detected {len(deleted)} deleted file(s)")
rel_deleted: list[str] = []
for path in deleted:
p = Path(path).absolute()
try:
rel_deleted.append(str(p.relative_to(self.vault_path)))
except ValueError:
rel_deleted.append(str(p))
try:
await self.file_store.delete(rel_deleted)
results.extend({"change": "deleted", "path": p, "success": True} for p in deleted)
except Exception as e:
self.logger.exception(f"Failed to delete {len(deleted)} file(s)")
results.extend({"change": "deleted", "path": p, "success": False, "error": str(e)} for p in deleted)
if self.persist and results:
await self.file_store.dump()
self.context.response.answer = results
self.context.response.success = all(r["success"] for r in results) if results else True
return self.context.response

62
reme4/steps/index/watch_changes.py
View file

@ -1,20 +1,24 @@
"""Long-running awatch loop: convert raw changes into dispatch_step calls.
"""Long-running awatch loop: convert raw changes into dispatch step calls.
Two relevant awatch parameters are exposed verbatim:
Relevant awatch parameters are exposed verbatim:
* ``step`` (default ``50ms``) — awatch yields when the entire watcher
* ``step`` — awatch yields when the entire watcher
has gone this long without new changes (and at least one change is
pending). Raise to ``5 minutes``-ish for ``auto_dream_loop`` so
half-written sync output isn't dreamed mid-write; keep at default
for ``index_update_loop`` where every fs change should hit
the index promptly.
* ``debounce`` (default ``2000ms``) — per-batch ceiling, regardless
* ``debounce`` — per-batch ceiling, regardless
of whether activity is still arriving. Set ``debounce > step`` so
``step`` is the operative limit; otherwise the watcher pre-empts
long-quiet-window setups under bursty writes.
Both are global to the watcher (not per-path). The reme watchers
* ``poll_delay_ms`` — delay between polling scans when
``force_polling=True``. This has the most direct effect on idle CPU
use for the default forced-polling watcher.
These settings are global to the watcher (not per-path). The reme watchers
have disjoint ``watch_dirs`` (configured per job), so global
quiet windows are good enough — no per-path bookkeeping needed.
@ -27,10 +31,15 @@ import asyncio
from watchfiles import Change, awatch
from ._watch_rules import WatchRule, build_watch_rules, match_file
from ._change_batch import coalesce_changes
from ._watch_rules import WatchRule, build_context_watch_rules, match_file
from ..base_step import BaseStep
from ...components import R, BaseComponent
from ...enumeration import ComponentEnum
from ...components import R
DEFAULT_WATCH_DEBOUNCE_MS = 5_000
DEFAULT_WATCH_STEP_MS = 1_000
DEFAULT_LOW_POWER_POLL_MS = 5_000
@R.register("watch_changes_step")
@ -41,11 +50,9 @@ class WatchChangesStep(BaseStep):
self,
recursive: bool = True,
force_polling: bool = True,
debounce: int = 2000,
step: int = 50,
poll_delay_ms: int = 2000,
dispatch_step: str = "",
dispatch_steps: list[str] | None = None,
debounce: int = DEFAULT_WATCH_DEBOUNCE_MS,
step: int = DEFAULT_WATCH_STEP_MS,
poll_delay_ms: int = DEFAULT_LOW_POWER_POLL_MS,
**kwargs,
):
super().__init__(**kwargs)
@ -54,20 +61,12 @@ class WatchChangesStep(BaseStep):
self.debounce: int = debounce
self.step: int = step
self.poll_delay_ms: int = poll_delay_ms
self.dispatch_steps: list[str] = dispatch_steps or ([dispatch_step] if dispatch_step else [])
self._rules: list[WatchRule] = []
def _get_watch_rules(self) -> list[WatchRule]:
"""Build watch rules from context-level watch_dirs/watch_suffixes."""
assert self.context is not None
app_config = self.app_context.app_config if self.app_context else None
if app_config is None:
return []
watch_dirs: list[str] = self.context.get("watch_dirs", [])
watch_suffixes: list[str] = self.context.get("watch_suffixes", [])
if not watch_dirs:
return []
return build_watch_rules(app_config, self.vault_path, watch_dirs=watch_dirs, watch_suffixes=watch_suffixes)
return build_context_watch_rules(app_config, self.vault_path, self.context)
def _filter(self, _change: Change, path: str) -> bool:
return match_file(path, self._rules)
@ -87,14 +86,10 @@ class WatchChangesStep(BaseStep):
if not valid_paths:
raise RuntimeError(f"No valid watch paths exist: {[str(r.path) for r in self._rules]}")
dispatch_classes: list[type[BaseComponent]] = []
for name in self.dispatch_steps:
cls = R.get(ComponentEnum.STEP, name)
if cls is None:
raise RuntimeError(f"Unregistered step '{name}'")
dispatch_classes.append(cls)
self.logger.info(f"Watching: {[str(p) for p in valid_paths]} step={self.step}ms debounce={self.debounce}ms")
self.logger.info(
f"Watching: {[str(p) for p in valid_paths]} "
f"step={self.step}ms debounce={self.debounce}ms poll_delay={self.poll_delay_ms}ms",
)
async for raw_changes in awatch(
*valid_paths,
@ -113,12 +108,9 @@ class WatchChangesStep(BaseStep):
for c, p in raw_changes
if c in (Change.added, Change.modified, Change.deleted)
]
changes = coalesce_changes(changes)
if changes:
self.logger.info(f"Detected {len(changes)} change(s)")
# TODO @jinli
extra = {k: v for k, v in self.context.data.items() if k not in ("stop_event", "changes")}
for cls in dispatch_classes:
s = cls(app_context=self.app_context)
await s(changes=changes, **extra)
await self.dispatch_steps(self.dispatch_step_specs, changes=changes)
return self.context.response

11
reme4/steps/transfer/__init__.py
View file

@ -0,0 +1,11 @@
"""Transfer steps."""
from .download import DownloadStep
from .ingest import IngestStep
from .upload import UploadStep
__all__ = [
"DownloadStep",
"IngestStep",
"UploadStep",
]

28
reme4/steps/transfer/download.py
View file

@ -45,7 +45,6 @@ class DownloadStep(BaseStep):
src_path: str = self.context.get("src_path", "") or ""
dst_path: str = self.context.get("dst_path", "") or ""
overwrite: bool = bool(self.context.get("overwrite", False))
assert src_path, "src_path is required"
payload = await self._download(src_path, dst_path, overwrite)
if "error" in payload:
self.context.response.success = False
@ -60,15 +59,34 @@ class DownloadStep(BaseStep):
)
self.context.response.metadata.update(payload)
async def _download(self, src_path: str, dst_path: str, overwrite: bool) -> dict:
async def _download(
self,
src_path: str,
dst_path: str,
overwrite: bool,
) -> dict: # pylint: disable=too-many-return-statements
# pylint: disable=too-many-return-statements
if not src_path:
return {"src_path": src_path, "error": "not found"}
src_abs = (Path(self.file_store.vault_path or ".") / src_path).resolve()
return {"src_path": src_path, "error": "src_path is required"}
vault_dir = Path(self.file_store.vault_path or ".").resolve()
src_abs = (vault_dir / src_path).resolve()
try:
src_abs.relative_to(vault_dir)
except ValueError:
return {"src_path": src_path, "error": "src_path must stay inside the vault"}
if not src_abs.is_file():
return {"src_path": src_path, "error": "not found"}
if dst_path:
dst_abs = Path(dst_path)
dst_abs = Path(dst_path).expanduser()
if not dst_abs.is_absolute():
return {
"src_path": src_path,
"dst_path": dst_path,
"error": "dst_path must be an absolute filesystem path",
}
if dst_abs.is_dir():
return {"src_path": src_path, "dst_path": dst_path, "error": "destination is a directory"}
if dst_abs.exists() and not overwrite:
return {
"src_path": src_path,

11
reme4/steps/transfer/ingest.py
View file

@ -155,10 +155,8 @@ class IngestStep(BaseStep):
self.context.response.metadata.update(payload)
def _resource_dir_name(self) -> str:
"""Configured ``resource_dir`` subdir name; defaults to ``"resource"`` outside an app context."""
if self.app_context is None:
return "resource"
return self.app_context.app_config.resource_dir
"""Configured ``resource_dir`` subdir name."""
return self.config_value("resource_dir")
def _vault_dir(self) -> Path:
vr = getattr(self.file_store, "vault_path", None)
@ -181,7 +179,8 @@ class IngestStep(BaseStep):
# both surface the conflict rather than getting silently clobbered.
on_disk = {p.name for p in bucket.iterdir() if p.is_file()}
existing_names = {Path(e.path).name for e in existing_entries} | on_disk
if final_name in existing_names:
existing_names_folded = {name.casefold() for name in existing_names}
if final_name.casefold() in existing_names_folded:
raise _DuplicateIngest(
f"duplicate: {final_name!r} already exists in {resource_dir}/{date}/",
)
@ -413,7 +412,7 @@ def _assemble_day_md(entries: list[FileNode], date: str) -> str:
lines: list[str] = [
"---",
f"name: {date}",
f"assets: [{', '.join(names)}]",
f"assets: {json.dumps(names, ensure_ascii=False)}",
"---",
"",
f"# {date} resources",

30
reme4/steps/transfer/upload.py
View file

@ -35,7 +35,6 @@ class UploadStep(BaseStep):
src_path: str = self.context.get("src_path", "") or ""
dst_path: str = self.context.get("dst_path", "") or ""
overwrite: bool = bool(self.context.get("overwrite", False))
assert src_path and dst_path, "src_path and dst_path are required"
payload = await self._upload(src_path, dst_path, overwrite)
if "error" in payload:
self.context.response.success = False
@ -51,8 +50,19 @@ class UploadStep(BaseStep):
)
self.context.response.metadata.update(payload)
async def _upload(self, src_path: str, dst_path: str, overwrite: bool) -> dict:
src_abs = Path(src_path)
async def _upload(
self,
src_path: str,
dst_path: str,
overwrite: bool,
) -> dict: # pylint: disable=too-many-return-statements
# pylint: disable=too-many-return-statements
if not src_path:
return {"src_path": src_path, "error": "src_path is required"}
if not dst_path:
return {"dst_path": dst_path, "error": "dst_path is required"}
src_abs = Path(src_path).resolve()
if not src_abs.is_file():
return {"src_path": src_path, "error": "not found"}
if "/" not in dst_path:
@ -60,10 +70,18 @@ class UploadStep(BaseStep):
"dst_path": dst_path,
"error": "dst_path must be relative to the vault with a directory component",
}
vault_dir = Path(self.file_store.vault_path or ".")
dst_abs = (vault_dir / dst_path).resolve() if not Path(dst_path).is_absolute() else None
if dst_abs is None:
if Path(dst_path).is_absolute():
return {"dst_path": dst_path, "error": "dst_path must be relative to the vault"}
vault_dir = Path(self.file_store.vault_path or ".").resolve()
dst_abs = (vault_dir / dst_path).resolve()
try:
dst_abs.relative_to(vault_dir)
except ValueError:
return {"dst_path": dst_path, "error": "dst_path must stay inside the vault"}
if dst_abs == src_abs:
return {"src_path": src_path, "dst_path": dst_path, "error": "src_path and dst_path are the same"}
if dst_abs.is_dir():
return {"dst_path": dst_path, "error": "destination is a directory"}
if dst_abs.exists() and not overwrite:
return {
"src_path": src_path,

3
reme4/utils/__init__.py
View file

@ -7,7 +7,7 @@ from .common_utils import (
call_action,
call_and_check,
)
from .env_utils import load_env
from .env_utils import load_env, parse_env_file
from .link_expansion import expand_links, render_expansion_lines
from .logger_utils import get_logger
from .logo_utils import print_logo
@ -23,6 +23,7 @@ __all__ = [
"call_action",
"call_and_check",
"load_env",
"parse_env_file",
"expand_links",
"render_expansion_lines",
"get_logger",

34
reme4/utils/agent_state_io.py
View file

@ -5,6 +5,8 @@ Format:
Lines 2+ — AgentState.context, one Msg per line.
"""
import os
from uuid import uuid4
from pathlib import Path
import aiofiles
@ -20,17 +22,44 @@ class AsStateHandler:
def __init__(self, path: str | Path):
self.path = Path(path)
@classmethod
def for_session(cls, directory: str | Path, session_id: str) -> "AsStateHandler":
"""Create a handler for ``<directory>/<session_id>.jsonl``."""
if not session_id or Path(session_id).name != session_id:
raise ValueError(f"Invalid session_id: {session_id!r}")
return cls(Path(directory) / f"{session_id}.jsonl")
def exists(self) -> bool:
"""Return whether the state file exists."""
return self.path.is_file()
async def load_or_none(self) -> AgentState | None:
"""Load state if the file exists, otherwise return ``None``."""
if not self.exists():
return None
return await self.load()
async def delete(self) -> bool:
"""Delete the state file if present. Returns whether a file was removed."""
if not self.exists():
return False
self.path.unlink()
return True
async def dump(self, state: AgentState) -> Path:
"""Write *state* to ``self.path`` in JSONL format."""
self.path.parent.mkdir(parents=True, exist_ok=True)
header = UserMsg(
name="__state__",
content=state.summary or "",
metadata={k: getattr(state, k) for k in _META_KEYS},
)
async with aiofiles.open(self.path, "w", encoding="utf-8") as f:
tmp_path = self.path.with_name(f".{self.path.name}.{uuid4().hex}.tmp")
async with aiofiles.open(tmp_path, "w", encoding="utf-8") as f:
await f.write(header.model_dump_json() + "\n")
for msg in state.context:
await f.write(msg.model_dump_json() + "\n")
os.replace(tmp_path, self.path)
return self.path
async def load(self) -> AgentState:
@ -47,8 +76,9 @@ class AsStateHandler:
else header.get_text_content() or ""
)
metadata = header.metadata or {}
return AgentState(
**{k: header.metadata.get(k, d) for k, d in [("session_id", ""), ("reply_id", ""), ("cur_iter", 0)]},
**{k: metadata.get(k, d) for k, d in [("session_id", ""), ("reply_id", ""), ("cur_iter", 0)]},
summary=summary,
context=[Msg.model_validate_json(line) for line in lines[1:] if line.strip()],
)

Some files were not shown because too many files have changed in this diff Show more