ReMe/reme4/config/default.yaml
jinliyl 83831ec90c
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper(统一 Agent 后端抽象)
- **`base_agent_wrapper.py`**:`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`(含 `session_id` / `last_message` / `result` / 可选 `structured_output`);`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`,改为 `add_job_tools(names: list[str])`(按名解析 BaseJob)与 `add_skills()`;新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。
- **`as_agent_wrapper.py`(AgentScope 后端)**:
  - 会话持久化重写:`session_path` 落地到 `<vault>/<session_dir>/agentscope/`,`_load_state` 支持 `resume` / `session_id` / `fork_session`,并做 UUID 校验(`_validate_session_id`);`_cleanup_expired_sessions` 按天数清理过期会话。
  - 新增内置工具集(`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write),`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效;`_resolve_skills()` 把配置的 skill 暴露给后端,`_load_tool_env()` 注入项目 `.env`。
  - `_event_to_chunk()` 把 20+ 种 AgentScope 事件(Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters)归一化为 `StreamChunk`。
- **`cc_agent_wrapper.py`(Claude Code SDK 后端,+551 行)**:
  - 新增 `_CcFileSessionStore`:基于 vault 的文件型会话存储,实现 append(按 uuid 去重)/ load / list / delete / list_subkeys,并对路径做 `_safe_parts` + `resolve()` 防越界校验。
  - `_build_options()`:统一构建 `ClaudeAgentOptions`,处理 skills、disallowed_tools(默认禁 `WebSearch`)、`.env` 注入、Claude Code 的 API 凭据解析(`_claude_code_api_env`,多级 base_url/api_key 回退)、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接(`_ensure_claude_skill_dir`)。
  - `_raw_event_to_chunk()` / `_message_content_to_chunks()`:把 Anthropic 流式事件(message_start/delta/stop、content_block_*)与 SDK 消息块(AssistantMessage/UserMessage/ResultMessage/RateLimitEvent)转换为统一 `StreamChunk`;跟踪 block_id/block_type/tool_call_name 做关联;处理尾部 `"success"` 误报异常的吞掉逻辑。

### 2. 统一流式协议(StreamChunk / ChunkEnum)
- **`stream_chunk.py`**:`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构,新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段,纯文本流仍保持轻量。
- **`chunk_enum.py`**:补全生命周期标记 `REPLY_START` / `REPLY_END`,并文档化两套后端事件 → ChunkEnum 的映射。

### 3. Index 模块重构(变化批次化 + dispatch)
- 新增 `_change_batch.py`:`coalesce_changes()` 把同路径多次事件折叠为最终状态(结合 path 存在性判定),`bucket_changes()` 按 watchfiles.Change 分桶。
- 新增 `init_changes.py`(`InitChangesStep`):一次性扫描,对比 file_store / file_catalog 已索引节点计算 added/modified/deleted,写入 `context["changes"]` 后 dispatch。
- 新增 `update_changes.py`:抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集;`UpdateCatalogStep`(写 file_catalog)、`UpdateIndexStep`(写 file_store,含按后缀解析 chunker)。
- **`watch_changes.py`**:改用 `dispatch_step_specs`(基类提供的 `dispatch_steps()`),每批先 `coalesce_changes` 再 dispatch;默认参数调整(debounce 5000ms / step 1000ms / poll 5000ms)并暴露常量。
- 删除旧步骤:`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧);`clear_store.py` 取代 clear_and_scan。

### 4. Evolve / Dream 模块(拆分为多步 pipeline)
- 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`,新增 `dream/` 子包,按 5 个步骤组织:
  - **`extract.py`**:扫描当日 day-index + daily 笔记,对比 file_catalog 找出 changed/deleted,调用 LLM 全局抽取 `units`(procedure/personal/wiki 三桶)与 `topics`,路径与桶做清洗/路由。
  - **`integrate.py`**:逐个 unit 调用 LLM 写入 digest,结构化输出 `IntegrateOutcome`(CREATE/CORROBORATE/REFINE/CORRECT),失败 unit/路径收集回写。
  - **`topics.py`**:写 `daily/<date>/interests.yaml`,结合当天已有 + 近 N 天做去重(`normalize_topic`),可走 LLM 或纯规则去重两条路径。
  - **`proactive.py`**:读取当日 `interests.yaml`,作为主动推荐话题的入口。
  - **`finish.py`**:把变更路径落盘到 dream file_catalog(checkpoint),渲染最终汇总摘要。
- 新增 `schema.py`(`DreamState` 等跨步骤共享状态与结构化输出模型)与 `utils.py`(状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数)。
- `evolve/__init__.py` 导出全部新 step。

### 5. auto_memory / auto_resource(适配新 Agent API)
- **`auto_memory.py`**:会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`;改用 `job_tools`;新增 `source_conversation` frontmatter 反向链接(`_session_link`);执行后刷新 day 索引(`refresh_day_index`),并对 session_id 做合法性校验。
- **`auto_resource.py`**:资源改用「同名 daily note」方案(`_compute_note_stem` 取文件 stem);批量处理 `changes: list[dict]`(`_handle_change` 逐项处理,返回逐项结果摘要);agent 会话 id 用稳定的 `uuid5`;同样刷新 day 索引。

### 6. BaseStep 基类增强
- 新增 `dispatch_steps` / `dispatch_step_specs` 机制:`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec,`dispatch_steps()` 复用当前 context 调用下游 step。
- 新增 `config_value()`:按 key 取 app config,缺失时回退 `ApplicationConfig` 默认值。
- 小幅清理:`language` 初始化、`copy()`、`Ref.__init__` 签名精简。

### 7. Components 改动
- **`file_store/local_file_store.py`**:持久化改用 zstd 压缩(`.jsonl.zst`,通过新 `utils/jsonl_zst.py`);upsert 时先删除旧 chunk 的 keyword 文档;embedding 复用改为 `(text, embedding)` 键控,要求文本一致才复用;新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。
- **`keyword_index/bm25_index.py`**:索引文件名加入组件名 + tokenizer 指纹(sha256 前 12 位),快照/恢复时校验指纹防配置漂移;空索引 dump 时删除文件,加载失败抛错而非静默。
- **`file_chunker/markdown_file_chunker.py`**:弃用 `python-frontmatter`,改用内置 YAML 解析(非法 YAML 不阻断正文索引),并修正因 frontmatter 占用行号导致的 AST 行号偏移(`line_offset`)。
- **`cron_job.py`**:大幅简化(-187 行),由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」;`Application` 启动顺序随之调整为 base > stream > background > cron。
- 其余小调整:service(base/http/mcp)、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。

### 8. Application 生命周期
- `_start()` 启动顺序明确为 components → base → stream → background → cron,启动失败会触发 `_close()` 回滚并 re-raise(不再吞异常)。
- 启动时创建 `session_dir` 目录;新增 `update_component()`(按类型/名就地更新已存在组件,不存在则报错)。

### 9. File IO / 路径安全
- **`_path.py`**:`resolve_path` 增加 vault 越界防护(`is_relative_to` 校验),禁止 `.` / `..` 路径分量,支持 `allow_empty`。
- **`read.py`**:大文件(超过 `MAX_FILE_READ_BYTES`)走按行读取 `read_file_lines_safe`,避免一次性载入内存。
- **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐(如 `refresh_day_index`、`read_file_lines_safe`)。
- **`env_utils.py`**:新增 `parse_env_file()`,`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。

### 10. Config
- `ApplicationConfig` 新增 `session_dir`(默认 `reme_session`)。
- `config_parser.py`:环境变量展开后做类型转换(`_convert_value`)、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。
- `default.yaml`:作业编排改用 `init_changes_step` + `dispatch_steps`(index/resource/digest 三个 watch loop 与 reindex);新增 `auto_dream`(4 步)、`proactive` 作业,移除旧 `dream`;file_catalog 增配 `resource` / `digest` / `dream` 实例;LLM 默认值与 Claude Code 凭据配置调整(tool_result_limit 50000、thinking_enable=false 等)。

### 11. 其它
- 新增 `steps/common/add.py`(`AddStep` 算术 demo)、`channel/__init__.py` 与 common `__init__` 导出整理。
- 新增 4 篇文档:`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`,以及 `reme_design.md` 更新。
**
2026-06-19 01:35:31 +08:00

614 lines
15 KiB
YAML

service:
backend: http
jobs:
index_update_loop:
backend: background
# watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_dirs: [daily_dir, digest_dir]
# watch_suffixes: [md, jsonl]
watch_suffixes: [md]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
- backend: watch_changes_step
dispatch_steps: [update_index_step]
resource_watch_loop:
backend: background
watch_dirs: [resource_dir]
watch_suffixes: [md, txt, json, jsonl, csv, yaml, html]
steps:
- backend: init_changes_step
monitor_type: file_catalog
monitor_name: resource
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
- backend: watch_changes_step
dispatch_steps:
- backend: update_catalog_step
file_catalog: resource
- backend: auto_resource_step
digest_watch_loop:
backend: background
watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [md]
steps:
- backend: init_changes_step
monitor_type: file_catalog
monitor_name: digest
dispatch_steps:
- backend: update_catalog_step
file_catalog: digest
- backend: log_changes_step
- backend: watch_changes_step
dispatch_steps:
- backend: update_catalog_step
file_catalog: digest
- backend: log_changes_step
version:
backend: base
description: "return reme package version"
parameters:
type: object
properties: { }
steps:
- backend: version_step
health_check:
backend: base
description: "return a concise health-check snapshot of reme components"
parameters:
type: object
properties: { }
steps:
- backend: health_check_step
help:
backend: base
description: "list all registered jobs with their metadata"
parameters:
type: object
properties: { }
steps:
- backend: help_step
traverse:
backend: base
description: "Walk the wikilink graph from a path."
parameters:
type: object
properties:
path:
type: string
description: "path"
depth:
type: integer
description: "hop limit"
default: 1
direction:
type: string
enum:
- forward
- backward
- both
default: both
required:
- path
steps:
- backend: traverse_step
reindex:
backend: base
description: "wipe the file store and rebuild it from the existing files"
watch_dirs: [daily_dir, digest_dir, resource_dir]
watch_suffixes: [md, jsonl]
parameters:
type: object
properties: { }
steps:
- backend: clear_store_step
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [update_index_step]
search:
backend: base
description: "Hybrid vault search (vector + BM25, RRF-fused)."
parameters:
type: object
properties:
query:
type: string
description: "search query"
limit:
type: integer
description: "max results"
default: 5
min_score:
type: number
description: "min fused score"
default: 0.0
required:
- query
steps:
- backend: search_step
vector_weight: 0.7
candidate_multiplier: 3.0
expand_links: true
max_links_per_direction: 10
node_search:
backend: base
description: "Digest node recall — given a candidate abstraction's name+description, surface existing digest nodes similar enough to either dedup against or link to as related."
parameters:
type: object
properties:
query:
type: string
description: "search query"
limit:
type: integer
description: "max digest nodes to return"
default: 20
required:
- query
steps:
- backend: node_search_step
vector_weight: 0.7
candidate_multiplier: 5.0
daily_create:
backend: base
description: "Provision a session note under a daily folder: daily/<date>/<session_id>.md or daily/<date>.md"
parameters:
type: object
properties:
session_id:
type: string
description: "the session identifier (also the file stem); empty = day-level file"
default: ""
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_create_step
daily_list:
backend: base
description: "List notes under a single day."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_list_step
daily_reindex:
backend: base
description: "Rebuild the day-index page daily/<date>.md."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD; empty = today"
default: ""
steps:
- backend: daily_reindex_step
frontmatter_delete:
backend: base
description: "Drop keys from a file's frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
keys:
type: array
description: "keys to remove"
items:
type: string
required:
- path
- keys
steps:
- backend: frontmatter_delete_step
frontmatter_read:
backend: base
description: "Read a file's frontmatter as a dict."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: frontmatter_read_step
frontmatter_update:
backend: base
description: "Merge key-values into a file's frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
metadata:
type: object
description: "key-values to merge"
required:
- path
- metadata
steps:
- backend: frontmatter_update_step
stat:
backend: base
description: "Stat path (size, mtime, exists, is_dir, is_file)."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: stat_step
list:
backend: base
description: "List files under a vault path."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative dir; empty = root"
default: ""
recursive:
type: boolean
description: "recurse"
default: false
limit:
type: integer
description: "max results"
default: 100
steps:
- backend: list_step
move:
backend: base
description: "Move / rename a vault file; rewrites inbound wikilinks by default."
parameters:
type: object
properties:
src_path:
type: string
description: "vault-relative source"
dst_path:
type: string
description: "vault-relative destination"
overwrite:
type: boolean
description: "overwrite if dst exists"
default: false
retarget:
type: boolean
description: "rewrite [[src]] → [[dst]] across the vault"
default: true
required:
- src_path
- dst_path
steps:
- backend: move_step
delete:
backend: base
description: "Delete a vault file or folder; returns surviving inbound wikilinks."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
required:
- path
steps:
- backend: delete_step
read:
backend: base
description: "Read a markdown file under the vault."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path; markdown only"
start_line:
type: integer
description: "first line (1-based, inclusive)"
end_line:
type: integer
description: "last line (1-based, inclusive)"
required:
- path
steps:
- backend: read_step
with_neighbors: false
max_neighbors_per_direction: 10
read_image:
backend: base
description: "Read an image file as base64 (vault-relative path)."
parameters:
type: object
properties:
path:
type: string
description: >-
vault-relative path; common image formats supported
(png/jpg/jpeg/webp/gif/bmp/tiff/heic)
required:
- path
steps:
- backend: read_image_step
max_bytes: 5242880
write:
backend: base
description: "Write a markdown file (create or overwrite) with name/description frontmatter."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path; markdown only"
name:
type: string
description: "frontmatter name"
description:
type: string
description: "frontmatter description"
content:
type: string
description: "body"
metadata:
type: object
description: "Optional extra frontmatter fields (md only)."
required:
- path
- name
- description
- content
steps:
- backend: write_step
edit:
backend: base
description: "Find-and-replace in a markdown file (all occurrences)."
parameters:
type: object
properties:
path:
type: string
description: "vault-relative path"
old:
type: string
description: "text to find"
new:
type: string
description: "replacement"
default: ""
required:
- path
- old
- new
steps:
- backend: edit_step
auto_dream:
backend: base
description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
default: ""
hint:
type: string
description: "caller guidance passed through to dream extract/integrate"
default: ""
topic_count:
type: integer
description: "maximum number of final daily interest topics"
default: 3
topic_diversity_days:
type: integer
description: "number of previous interests.yaml days to avoid repeating"
default: 7
steps:
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
topic_diversity_days: 7
- backend: dream_finish_step
file_catalog: dream
proactive:
backend: base
description: "Proactive: read daily/<date>/interests.yaml and expose the latest user-interest topics."
parameters:
type: object
properties:
date:
type: string
description: "YYYY-MM-DD to read; defaults to today in the dreamer's timezone"
default: ""
include_content:
type: boolean
description: "whether to include the raw YAML content in response metadata"
default: true
steps:
- backend: proactive_step
auto_memory:
backend: base
description: "Auto-memory: record conversation facts into a daily note"
parameters:
type: object
properties:
messages:
type: array
description: "messages"
items:
type: object
session_id:
type: string
description: "session identifier passed to daily_create"
default: ""
memory_hint:
type: string
description: "optional hint"
required:
- messages
steps:
- backend: auto_memory_step
auto_resource:
backend: base
description: "Auto-resource: interpret resource files into daily notes"
parameters:
type: object
properties:
changes:
type: array
description: "resource change batch, each item has path/file_path and change"
items:
type: object
properties:
path:
type: string
file_path:
type: string
change:
type: string
description: "added/modified/deleted"
required:
- changes
steps:
- backend: auto_resource_step
components:
tokenizer:
default:
backend: regex
as_embedding:
default:
backend: ${EMBEDDING_BACKEND:-openai}
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
credential:
api_key: ${EMBEDDING_API_KEY:-}
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
parameters:
dimensions: 1024
embedding_store:
default:
backend: local
as_embedding: default
as_llm:
default:
backend: ${LLM_BACKEND:-openai}
model: ${LLM_MODEL_NAME:-qwen3.7-plus}
stream: true
context_size: 200000
max_retries: 3
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: false
agent_wrapper:
default:
backend: agentscope
as_llm: default
permission_mode: bypass
react_config:
max_iters: 30
context_config:
trigger_ratio: 0.8
reserve_ratio: 0.1
tool_result_limit: 50000
model_config:
max_retries: 1
claude_code:
backend: claude_code
model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.1}
api_key: ${CLAUDE_CODE_API_KEY:-}
base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic}
permission_mode: bypassPermissions
file_graph:
default:
backend: local
file_catalog:
default:
backend: local
resource:
backend: local
digest:
backend: local
dream:
backend: local
file_chunker:
markdown:
backend: markdown
supported_extensions: [ "md" ]
default:
backend: default
supported_extensions: [ "jsonl" ]
keyword_index:
default:
backend: bm25
tokenizer: default
file_store:
default:
backend: local
store_name: local
# embedding_store: default
embedding_store: ""
keyword_index: default
file_graph: default