ReMe/reme4/components/base_component.py
jinliyl 83831ec90c
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper(统一 Agent 后端抽象)
- **`base_agent_wrapper.py`**:`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`(含 `session_id` / `last_message` / `result` / 可选 `structured_output`);`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`,改为 `add_job_tools(names: list[str])`(按名解析 BaseJob)与 `add_skills()`;新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。
- **`as_agent_wrapper.py`(AgentScope 后端)**:
  - 会话持久化重写:`session_path` 落地到 `<vault>/<session_dir>/agentscope/`,`_load_state` 支持 `resume` / `session_id` / `fork_session`,并做 UUID 校验(`_validate_session_id`);`_cleanup_expired_sessions` 按天数清理过期会话。
  - 新增内置工具集(`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write),`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效;`_resolve_skills()` 把配置的 skill 暴露给后端,`_load_tool_env()` 注入项目 `.env`。
  - `_event_to_chunk()` 把 20+ 种 AgentScope 事件(Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters)归一化为 `StreamChunk`。
- **`cc_agent_wrapper.py`(Claude Code SDK 后端,+551 行)**:
  - 新增 `_CcFileSessionStore`:基于 vault 的文件型会话存储,实现 append(按 uuid 去重)/ load / list / delete / list_subkeys,并对路径做 `_safe_parts` + `resolve()` 防越界校验。
  - `_build_options()`:统一构建 `ClaudeAgentOptions`,处理 skills、disallowed_tools(默认禁 `WebSearch`)、`.env` 注入、Claude Code 的 API 凭据解析(`_claude_code_api_env`,多级 base_url/api_key 回退)、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接(`_ensure_claude_skill_dir`)。
  - `_raw_event_to_chunk()` / `_message_content_to_chunks()`:把 Anthropic 流式事件(message_start/delta/stop、content_block_*)与 SDK 消息块(AssistantMessage/UserMessage/ResultMessage/RateLimitEvent)转换为统一 `StreamChunk`;跟踪 block_id/block_type/tool_call_name 做关联;处理尾部 `"success"` 误报异常的吞掉逻辑。

### 2. 统一流式协议(StreamChunk / ChunkEnum)
- **`stream_chunk.py`**:`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构,新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段,纯文本流仍保持轻量。
- **`chunk_enum.py`**:补全生命周期标记 `REPLY_START` / `REPLY_END`,并文档化两套后端事件 → ChunkEnum 的映射。

### 3. Index 模块重构(变化批次化 + dispatch)
- 新增 `_change_batch.py`:`coalesce_changes()` 把同路径多次事件折叠为最终状态(结合 path 存在性判定),`bucket_changes()` 按 watchfiles.Change 分桶。
- 新增 `init_changes.py`(`InitChangesStep`):一次性扫描,对比 file_store / file_catalog 已索引节点计算 added/modified/deleted,写入 `context["changes"]` 后 dispatch。
- 新增 `update_changes.py`:抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集;`UpdateCatalogStep`(写 file_catalog)、`UpdateIndexStep`(写 file_store,含按后缀解析 chunker)。
- **`watch_changes.py`**:改用 `dispatch_step_specs`(基类提供的 `dispatch_steps()`),每批先 `coalesce_changes` 再 dispatch;默认参数调整(debounce 5000ms / step 1000ms / poll 5000ms)并暴露常量。
- 删除旧步骤:`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧);`clear_store.py` 取代 clear_and_scan。

### 4. Evolve / Dream 模块(拆分为多步 pipeline)
- 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`,新增 `dream/` 子包,按 5 个步骤组织:
  - **`extract.py`**:扫描当日 day-index + daily 笔记,对比 file_catalog 找出 changed/deleted,调用 LLM 全局抽取 `units`(procedure/personal/wiki 三桶)与 `topics`,路径与桶做清洗/路由。
  - **`integrate.py`**:逐个 unit 调用 LLM 写入 digest,结构化输出 `IntegrateOutcome`(CREATE/CORROBORATE/REFINE/CORRECT),失败 unit/路径收集回写。
  - **`topics.py`**:写 `daily/<date>/interests.yaml`,结合当天已有 + 近 N 天做去重(`normalize_topic`),可走 LLM 或纯规则去重两条路径。
  - **`proactive.py`**:读取当日 `interests.yaml`,作为主动推荐话题的入口。
  - **`finish.py`**:把变更路径落盘到 dream file_catalog(checkpoint),渲染最终汇总摘要。
- 新增 `schema.py`(`DreamState` 等跨步骤共享状态与结构化输出模型)与 `utils.py`(状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数)。
- `evolve/__init__.py` 导出全部新 step。

### 5. auto_memory / auto_resource(适配新 Agent API)
- **`auto_memory.py`**:会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`;改用 `job_tools`;新增 `source_conversation` frontmatter 反向链接(`_session_link`);执行后刷新 day 索引(`refresh_day_index`),并对 session_id 做合法性校验。
- **`auto_resource.py`**:资源改用「同名 daily note」方案(`_compute_note_stem` 取文件 stem);批量处理 `changes: list[dict]`(`_handle_change` 逐项处理,返回逐项结果摘要);agent 会话 id 用稳定的 `uuid5`;同样刷新 day 索引。

### 6. BaseStep 基类增强
- 新增 `dispatch_steps` / `dispatch_step_specs` 机制:`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec,`dispatch_steps()` 复用当前 context 调用下游 step。
- 新增 `config_value()`:按 key 取 app config,缺失时回退 `ApplicationConfig` 默认值。
- 小幅清理:`language` 初始化、`copy()`、`Ref.__init__` 签名精简。

### 7. Components 改动
- **`file_store/local_file_store.py`**:持久化改用 zstd 压缩(`.jsonl.zst`,通过新 `utils/jsonl_zst.py`);upsert 时先删除旧 chunk 的 keyword 文档;embedding 复用改为 `(text, embedding)` 键控,要求文本一致才复用;新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。
- **`keyword_index/bm25_index.py`**:索引文件名加入组件名 + tokenizer 指纹(sha256 前 12 位),快照/恢复时校验指纹防配置漂移;空索引 dump 时删除文件,加载失败抛错而非静默。
- **`file_chunker/markdown_file_chunker.py`**:弃用 `python-frontmatter`,改用内置 YAML 解析(非法 YAML 不阻断正文索引),并修正因 frontmatter 占用行号导致的 AST 行号偏移(`line_offset`)。
- **`cron_job.py`**:大幅简化(-187 行),由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」;`Application` 启动顺序随之调整为 base > stream > background > cron。
- 其余小调整:service(base/http/mcp)、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。

### 8. Application 生命周期
- `_start()` 启动顺序明确为 components → base → stream → background → cron,启动失败会触发 `_close()` 回滚并 re-raise(不再吞异常)。
- 启动时创建 `session_dir` 目录;新增 `update_component()`(按类型/名就地更新已存在组件,不存在则报错)。

### 9. File IO / 路径安全
- **`_path.py`**:`resolve_path` 增加 vault 越界防护(`is_relative_to` 校验),禁止 `.` / `..` 路径分量,支持 `allow_empty`。
- **`read.py`**:大文件(超过 `MAX_FILE_READ_BYTES`)走按行读取 `read_file_lines_safe`,避免一次性载入内存。
- **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐(如 `refresh_day_index`、`read_file_lines_safe`)。
- **`env_utils.py`**:新增 `parse_env_file()`,`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。

### 10. Config
- `ApplicationConfig` 新增 `session_dir`(默认 `reme_session`)。
- `config_parser.py`:环境变量展开后做类型转换(`_convert_value`)、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。
- `default.yaml`:作业编排改用 `init_changes_step` + `dispatch_steps`(index/resource/digest 三个 watch loop 与 reindex);新增 `auto_dream`(4 步)、`proactive` 作业,移除旧 `dream`;file_catalog 增配 `resource` / `digest` / `dream` 实例;LLM 默认值与 Claude Code 凭据配置调整(tool_result_limit 50000、thinking_enable=false 等)。

### 11. 其它
- 新增 `steps/common/add.py`(`AddStep` 算术 demo)、`channel/__init__.py` 与 common `__init__` 导出整理。
- 新增 4 篇文档:`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`,以及 `reme_design.md` 更新。
**
2026-06-19 01:35:31 +08:00

254 lines
9 KiB
Python

"""Base class for components with async lifecycle and dependency injection."""
import asyncio
from abc import ABC
from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable, TypeVar, cast
from ..enumeration import ComponentEnum
from ..utils import get_logger
if TYPE_CHECKING:
from .application_context import ApplicationContext
T = TypeVar("T", bound="BaseComponent")
class ComponentMixin:
"""Shared state for components and steps: identity, config, vault paths."""
component_type = ComponentEnum.BASE
def __init__(
self,
name: str | None = None,
backend: str = "",
app_context: "ApplicationContext | None" = None,
**kwargs,
) -> None:
self.name: str = name or self.__class__.__name__
self.backend: str = backend
self.app_context: "ApplicationContext | None" = app_context
self.kwargs: dict = dict(kwargs)
logger = get_logger()
self.logger = logger.bind(component=self.name) if hasattr(logger, "bind") else logger
@property
def vault_path(self) -> Path:
"""Absolute vault root directory (cwd when no app_context is attached)."""
if self.app_context is None:
return Path.cwd()
return Path(self.app_context.app_config.vault_dir).absolute()
def to_vault_relative(self, path: str | Path) -> str:
"""Convert `path` to a vault-relative string; return absolute path when outside."""
abs_path = Path(path).absolute()
try:
return str(abs_path.relative_to(self.vault_path))
except ValueError:
return str(abs_path)
class Dependency:
"""Placeholder returned by ``BaseComponent.bind`` for an unresolved dependency.
Resolved into a real component (or None) when the owning component starts.
Accessing any attribute before resolution raises a clear error.
"""
__slots__ = ("ctype", "name", "default_factory", "optional")
def __init__(
self,
ctype: ComponentEnum,
name: str,
default_factory: Callable[[], Any] | None = None,
optional: bool = True,
) -> None:
self.ctype = ctype
self.name = name
self.default_factory = default_factory
self.optional = optional
def __repr__(self) -> str:
suffix = "?" if self.optional else ""
return f"<unresolved {self.ctype.value}:{self.name}{suffix}>"
def __getattr__(self, item: str) -> Any:
# Catches accidental use of the placeholder before start() resolves it.
raise RuntimeError(
f"Dependency {self.ctype.value}:{self.name} accessed before start() " f"(attribute '{item}')",
)
class BaseComponent(ComponentMixin, ABC):
"""Async lifecycle base class with bind-based dependency injection."""
component_type = ComponentEnum.BASE
def __init__(
self,
name: str | None = None,
backend: str = "",
app_context: "ApplicationContext | None" = None,
**kwargs,
) -> None:
super().__init__(name=name, backend=backend, app_context=app_context, **kwargs)
self._is_started: bool = False
self._lock: asyncio.Lock = asyncio.Lock()
# Components created via bind() default_factory in standalone mode;
# their lifecycle is owned by this component.
self._owned: list["BaseComponent"] = []
@property
def is_started(self) -> bool:
"""Whether the component has been started."""
return self._is_started
# ----- Dependency declaration ----------------------------------------
@staticmethod
def bind(
name: str | None,
base_cls: type[T],
*,
default_factory: Callable[[], T] | None = None,
optional: bool = True,
) -> T | None:
"""Declare a dependency on another component.
Returns a ``Dependency`` placeholder resolved into the real component
(or None / a factory-produced instance) when ``start`` runs. An empty
`name` short-circuits to None so callers can skip optional wiring.
"""
if not name:
return None
ctype = getattr(base_cls, "component_type", None)
if not isinstance(ctype, ComponentEnum) or ctype is ComponentEnum.BASE:
raise TypeError(
f"{base_cls.__name__} must declare a non-BASE ComponentEnum 'component_type'",
)
return cast(T, Dependency(ctype, name, default_factory, optional))
@property
def dependencies(self) -> list[Dependency]:
"""All unresolved dependency placeholders on this instance."""
return [v for v in self.__dict__.values() if isinstance(v, Dependency)]
async def _resolve_bindings(self) -> None:
"""Replace every ``Dependency`` attribute with its resolved target."""
for attr, dep in list(self.__dict__.items()):
if isinstance(dep, Dependency):
self._resolve_one(attr, dep)
def _resolve_one(self, attr: str, dep: Dependency) -> None:
"""Resolve a single dependency, dispatching by mode."""
if self.app_context is None:
self._resolve_standalone(attr, dep)
else:
self._resolve_from_context(attr, dep)
def _resolve_standalone(self, attr: str, dep: Dependency) -> None:
"""Standalone mode: use default_factory, or fall back to None when optional.
Required dependencies without a factory keep the placeholder so any
attribute access surfaces a clear error at the call site.
"""
if dep.default_factory is not None:
instance = dep.default_factory()
setattr(self, attr, instance)
if isinstance(instance, BaseComponent):
self._owned.append(instance)
elif dep.optional:
setattr(self, attr, None)
def _resolve_from_context(self, attr: str, dep: Dependency) -> None:
"""Context-bound mode: look up the component from ``app_context.components``."""
target = self.app_context.components.get(dep.ctype, {}).get(dep.name)
if target is not None:
setattr(self, attr, target)
elif dep.optional:
setattr(self, attr, None)
else:
raise ValueError(f"{dep.ctype.value} '{dep.name}' not found.")
# ----- Vault path helpers --------------------------------------------
@property
def vault_metadata_path(self) -> Path:
"""Vault metadata directory: ``<vault>/<metadata_dir>``."""
if self.app_context is None:
return Path.cwd() / "metadata"
return self.vault_path / self.app_context.app_config.metadata_dir
@property
def component_metadata_path(self) -> Path:
"""Per-component metadata directory under the vault."""
return self.vault_metadata_path / self.component_type.value
# ----- Lifecycle hooks (override in subclasses) ----------------------
async def _start(self) -> None:
"""Subclass hook called once after dependencies are resolved."""
async def _close(self) -> None:
"""Subclass hook called once during ``close``."""
async def dump(self) -> None:
"""Persist in-memory state to disk. Override when persistence is needed."""
async def load(self) -> None:
"""Restore in-memory state from disk. Override when persistence is needed."""
# ----- Lifecycle control --------------------------------------------
async def start(self) -> None:
"""Start the component once: resolve deps → start owned → run _start."""
async with self._lock:
if self._is_started:
return
await self._resolve_bindings()
for owned in self._owned:
await owned.start()
await self._start()
self._is_started = True
async def close(self) -> None:
"""Close the component once: run _close → close owned in reverse order."""
async with self._lock:
if not self._is_started:
return
first_error: BaseException | None = None
try:
await self._close()
except BaseException as exc:
first_error = exc
finally:
for owned in reversed(self._owned):
try:
await owned.close()
except BaseException as exc:
if first_error is None:
first_error = exc
else:
self.logger.exception(f"Failed to close owned component {owned.name}: {exc}")
self._is_started = False
if first_error is not None:
raise first_error
async def restart(self) -> None:
"""Close then start the component."""
await self.close()
await self.start()
async def __call__(self, **kwargs):
raise NotImplementedError
async def __aenter__(self) -> "BaseComponent":
await self.start()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
await self.close()