mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-05 08:06:15 +00:00
### 1. Agent Wrapper(统一 Agent 后端抽象) - **`base_agent_wrapper.py`**:`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`(含 `session_id` / `last_message` / `result` / 可选 `structured_output`);`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`,改为 `add_job_tools(names: list[str])`(按名解析 BaseJob)与 `add_skills()`;新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`(AgentScope 后端)**: - 会话持久化重写:`session_path` 落地到 `<vault>/<session_dir>/agentscope/`,`_load_state` 支持 `resume` / `session_id` / `fork_session`,并做 UUID 校验(`_validate_session_id`);`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集(`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write),`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效;`_resolve_skills()` 把配置的 skill 暴露给后端,`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件(Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters)归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`(Claude Code SDK 后端,+551 行)**: - 新增 `_CcFileSessionStore`:基于 vault 的文件型会话存储,实现 append(按 uuid 去重)/ load / list / delete / list_subkeys,并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`:统一构建 `ClaudeAgentOptions`,处理 skills、disallowed_tools(默认禁 `WebSearch`)、`.env` 注入、Claude Code 的 API 凭据解析(`_claude_code_api_env`,多级 base_url/api_key 回退)、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接(`_ensure_claude_skill_dir`)。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`:把 Anthropic 流式事件(message_start/delta/stop、content_block_*)与 SDK 消息块(AssistantMessage/UserMessage/ResultMessage/RateLimitEvent)转换为统一 `StreamChunk`;跟踪 block_id/block_type/tool_call_name 做关联;处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议(StreamChunk / ChunkEnum) - **`stream_chunk.py`**:`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构,新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段,纯文本流仍保持轻量。 - **`chunk_enum.py`**:补全生命周期标记 `REPLY_START` / `REPLY_END`,并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构(变化批次化 + dispatch) - 新增 `_change_batch.py`:`coalesce_changes()` 把同路径多次事件折叠为最终状态(结合 path 存在性判定),`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`(`InitChangesStep`):一次性扫描,对比 file_store / file_catalog 已索引节点计算 added/modified/deleted,写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`:抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集;`UpdateCatalogStep`(写 file_catalog)、`UpdateIndexStep`(写 file_store,含按后缀解析 chunker)。 - **`watch_changes.py`**:改用 `dispatch_step_specs`(基类提供的 `dispatch_steps()`),每批先 `coalesce_changes` 再 dispatch;默认参数调整(debounce 5000ms / step 1000ms / poll 5000ms)并暴露常量。 - 删除旧步骤:`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧);`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块(拆分为多步 pipeline) - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`,新增 `dream/` 子包,按 5 个步骤组织: - **`extract.py`**:扫描当日 day-index + daily 笔记,对比 file_catalog 找出 changed/deleted,调用 LLM 全局抽取 `units`(procedure/personal/wiki 三桶)与 `topics`,路径与桶做清洗/路由。 - **`integrate.py`**:逐个 unit 调用 LLM 写入 digest,结构化输出 `IntegrateOutcome`(CREATE/CORROBORATE/REFINE/CORRECT),失败 unit/路径收集回写。 - **`topics.py`**:写 `daily/<date>/interests.yaml`,结合当天已有 + 近 N 天做去重(`normalize_topic`),可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**:读取当日 `interests.yaml`,作为主动推荐话题的入口。 - **`finish.py`**:把变更路径落盘到 dream file_catalog(checkpoint),渲染最终汇总摘要。 - 新增 `schema.py`(`DreamState` 等跨步骤共享状态与结构化输出模型)与 `utils.py`(状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数)。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource(适配新 Agent API) - **`auto_memory.py`**:会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`;改用 `job_tools`;新增 `source_conversation` frontmatter 反向链接(`_session_link`);执行后刷新 day 索引(`refresh_day_index`),并对 session_id 做合法性校验。 - **`auto_resource.py`**:资源改用「同名 daily note」方案(`_compute_note_stem` 取文件 stem);批量处理 `changes: list[dict]`(`_handle_change` 逐项处理,返回逐项结果摘要);agent 会话 id 用稳定的 `uuid5`;同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制:`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec,`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`:按 key 取 app config,缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理:`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**:持久化改用 zstd 压缩(`.jsonl.zst`,通过新 `utils/jsonl_zst.py`);upsert 时先删除旧 chunk 的 keyword 文档;embedding 复用改为 `(text, embedding)` 键控,要求文本一致才复用;新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**:索引文件名加入组件名 + tokenizer 指纹(sha256 前 12 位),快照/恢复时校验指纹防配置漂移;空索引 dump 时删除文件,加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**:弃用 `python-frontmatter`,改用内置 YAML 解析(非法 YAML 不阻断正文索引),并修正因 frontmatter 占用行号导致的 AST 行号偏移(`line_offset`)。 - **`cron_job.py`**:大幅简化(-187 行),由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」;`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整:service(base/http/mcp)、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron,启动失败会触发 `_close()` 回滚并 re-raise(不再吞异常)。 - 启动时创建 `session_dir` 目录;新增 `update_component()`(按类型/名就地更新已存在组件,不存在则报错)。 ### 9. File IO / 路径安全 - **`_path.py`**:`resolve_path` 增加 vault 越界防护(`is_relative_to` 校验),禁止 `.` / `..` 路径分量,支持 `allow_empty`。 - **`read.py`**:大文件(超过 `MAX_FILE_READ_BYTES`)走按行读取 `read_file_lines_safe`,避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐(如 `refresh_day_index`、`read_file_lines_safe`)。 - **`env_utils.py`**:新增 `parse_env_file()`,`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`(默认 `reme_session`)。 - `config_parser.py`:环境变量展开后做类型转换(`_convert_value`)、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`:作业编排改用 `init_changes_step` + `dispatch_steps`(index/resource/digest 三个 watch loop 与 reindex);新增 `auto_dream`(4 步)、`proactive` 作业,移除旧 `dream`;file_catalog 增配 `resource` / `digest` / `dream` 实例;LLM 默认值与 Claude Code 凭据配置调整(tool_result_limit 50000、thinking_enable=false 等)。 ### 11. 其它 - 新增 `steps/common/add.py`(`AddStep` 算术 demo)、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档:`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`,以及 `reme_design.md` 更新。 **
254 lines
9 KiB
Python
254 lines
9 KiB
Python
"""Base class for components with async lifecycle and dependency injection."""
|
|
|
|
import asyncio
|
|
from abc import ABC
|
|
from pathlib import Path
|
|
from typing import TYPE_CHECKING, Any, Callable, TypeVar, cast
|
|
|
|
from ..enumeration import ComponentEnum
|
|
from ..utils import get_logger
|
|
|
|
if TYPE_CHECKING:
|
|
from .application_context import ApplicationContext
|
|
|
|
T = TypeVar("T", bound="BaseComponent")
|
|
|
|
|
|
class ComponentMixin:
|
|
"""Shared state for components and steps: identity, config, vault paths."""
|
|
|
|
component_type = ComponentEnum.BASE
|
|
|
|
def __init__(
|
|
self,
|
|
name: str | None = None,
|
|
backend: str = "",
|
|
app_context: "ApplicationContext | None" = None,
|
|
**kwargs,
|
|
) -> None:
|
|
self.name: str = name or self.__class__.__name__
|
|
self.backend: str = backend
|
|
self.app_context: "ApplicationContext | None" = app_context
|
|
self.kwargs: dict = dict(kwargs)
|
|
|
|
logger = get_logger()
|
|
self.logger = logger.bind(component=self.name) if hasattr(logger, "bind") else logger
|
|
|
|
@property
|
|
def vault_path(self) -> Path:
|
|
"""Absolute vault root directory (cwd when no app_context is attached)."""
|
|
if self.app_context is None:
|
|
return Path.cwd()
|
|
return Path(self.app_context.app_config.vault_dir).absolute()
|
|
|
|
def to_vault_relative(self, path: str | Path) -> str:
|
|
"""Convert `path` to a vault-relative string; return absolute path when outside."""
|
|
abs_path = Path(path).absolute()
|
|
try:
|
|
return str(abs_path.relative_to(self.vault_path))
|
|
except ValueError:
|
|
return str(abs_path)
|
|
|
|
|
|
class Dependency:
|
|
"""Placeholder returned by ``BaseComponent.bind`` for an unresolved dependency.
|
|
|
|
Resolved into a real component (or None) when the owning component starts.
|
|
Accessing any attribute before resolution raises a clear error.
|
|
"""
|
|
|
|
__slots__ = ("ctype", "name", "default_factory", "optional")
|
|
|
|
def __init__(
|
|
self,
|
|
ctype: ComponentEnum,
|
|
name: str,
|
|
default_factory: Callable[[], Any] | None = None,
|
|
optional: bool = True,
|
|
) -> None:
|
|
self.ctype = ctype
|
|
self.name = name
|
|
self.default_factory = default_factory
|
|
self.optional = optional
|
|
|
|
def __repr__(self) -> str:
|
|
suffix = "?" if self.optional else ""
|
|
return f"<unresolved {self.ctype.value}:{self.name}{suffix}>"
|
|
|
|
def __getattr__(self, item: str) -> Any:
|
|
# Catches accidental use of the placeholder before start() resolves it.
|
|
raise RuntimeError(
|
|
f"Dependency {self.ctype.value}:{self.name} accessed before start() " f"(attribute '{item}')",
|
|
)
|
|
|
|
|
|
class BaseComponent(ComponentMixin, ABC):
|
|
"""Async lifecycle base class with bind-based dependency injection."""
|
|
|
|
component_type = ComponentEnum.BASE
|
|
|
|
def __init__(
|
|
self,
|
|
name: str | None = None,
|
|
backend: str = "",
|
|
app_context: "ApplicationContext | None" = None,
|
|
**kwargs,
|
|
) -> None:
|
|
super().__init__(name=name, backend=backend, app_context=app_context, **kwargs)
|
|
|
|
self._is_started: bool = False
|
|
self._lock: asyncio.Lock = asyncio.Lock()
|
|
# Components created via bind() default_factory in standalone mode;
|
|
# their lifecycle is owned by this component.
|
|
self._owned: list["BaseComponent"] = []
|
|
|
|
@property
|
|
def is_started(self) -> bool:
|
|
"""Whether the component has been started."""
|
|
return self._is_started
|
|
|
|
# ----- Dependency declaration ----------------------------------------
|
|
|
|
@staticmethod
|
|
def bind(
|
|
name: str | None,
|
|
base_cls: type[T],
|
|
*,
|
|
default_factory: Callable[[], T] | None = None,
|
|
optional: bool = True,
|
|
) -> T | None:
|
|
"""Declare a dependency on another component.
|
|
|
|
Returns a ``Dependency`` placeholder resolved into the real component
|
|
(or None / a factory-produced instance) when ``start`` runs. An empty
|
|
`name` short-circuits to None so callers can skip optional wiring.
|
|
"""
|
|
if not name:
|
|
return None
|
|
ctype = getattr(base_cls, "component_type", None)
|
|
if not isinstance(ctype, ComponentEnum) or ctype is ComponentEnum.BASE:
|
|
raise TypeError(
|
|
f"{base_cls.__name__} must declare a non-BASE ComponentEnum 'component_type'",
|
|
)
|
|
return cast(T, Dependency(ctype, name, default_factory, optional))
|
|
|
|
@property
|
|
def dependencies(self) -> list[Dependency]:
|
|
"""All unresolved dependency placeholders on this instance."""
|
|
return [v for v in self.__dict__.values() if isinstance(v, Dependency)]
|
|
|
|
async def _resolve_bindings(self) -> None:
|
|
"""Replace every ``Dependency`` attribute with its resolved target."""
|
|
for attr, dep in list(self.__dict__.items()):
|
|
if isinstance(dep, Dependency):
|
|
self._resolve_one(attr, dep)
|
|
|
|
def _resolve_one(self, attr: str, dep: Dependency) -> None:
|
|
"""Resolve a single dependency, dispatching by mode."""
|
|
if self.app_context is None:
|
|
self._resolve_standalone(attr, dep)
|
|
else:
|
|
self._resolve_from_context(attr, dep)
|
|
|
|
def _resolve_standalone(self, attr: str, dep: Dependency) -> None:
|
|
"""Standalone mode: use default_factory, or fall back to None when optional.
|
|
|
|
Required dependencies without a factory keep the placeholder so any
|
|
attribute access surfaces a clear error at the call site.
|
|
"""
|
|
if dep.default_factory is not None:
|
|
instance = dep.default_factory()
|
|
setattr(self, attr, instance)
|
|
if isinstance(instance, BaseComponent):
|
|
self._owned.append(instance)
|
|
elif dep.optional:
|
|
setattr(self, attr, None)
|
|
|
|
def _resolve_from_context(self, attr: str, dep: Dependency) -> None:
|
|
"""Context-bound mode: look up the component from ``app_context.components``."""
|
|
target = self.app_context.components.get(dep.ctype, {}).get(dep.name)
|
|
if target is not None:
|
|
setattr(self, attr, target)
|
|
elif dep.optional:
|
|
setattr(self, attr, None)
|
|
else:
|
|
raise ValueError(f"{dep.ctype.value} '{dep.name}' not found.")
|
|
|
|
# ----- Vault path helpers --------------------------------------------
|
|
|
|
@property
|
|
def vault_metadata_path(self) -> Path:
|
|
"""Vault metadata directory: ``<vault>/<metadata_dir>``."""
|
|
if self.app_context is None:
|
|
return Path.cwd() / "metadata"
|
|
return self.vault_path / self.app_context.app_config.metadata_dir
|
|
|
|
@property
|
|
def component_metadata_path(self) -> Path:
|
|
"""Per-component metadata directory under the vault."""
|
|
return self.vault_metadata_path / self.component_type.value
|
|
|
|
# ----- Lifecycle hooks (override in subclasses) ----------------------
|
|
|
|
async def _start(self) -> None:
|
|
"""Subclass hook called once after dependencies are resolved."""
|
|
|
|
async def _close(self) -> None:
|
|
"""Subclass hook called once during ``close``."""
|
|
|
|
async def dump(self) -> None:
|
|
"""Persist in-memory state to disk. Override when persistence is needed."""
|
|
|
|
async def load(self) -> None:
|
|
"""Restore in-memory state from disk. Override when persistence is needed."""
|
|
|
|
# ----- Lifecycle control --------------------------------------------
|
|
|
|
async def start(self) -> None:
|
|
"""Start the component once: resolve deps → start owned → run _start."""
|
|
async with self._lock:
|
|
if self._is_started:
|
|
return
|
|
await self._resolve_bindings()
|
|
for owned in self._owned:
|
|
await owned.start()
|
|
await self._start()
|
|
self._is_started = True
|
|
|
|
async def close(self) -> None:
|
|
"""Close the component once: run _close → close owned in reverse order."""
|
|
async with self._lock:
|
|
if not self._is_started:
|
|
return
|
|
first_error: BaseException | None = None
|
|
try:
|
|
await self._close()
|
|
except BaseException as exc:
|
|
first_error = exc
|
|
finally:
|
|
for owned in reversed(self._owned):
|
|
try:
|
|
await owned.close()
|
|
except BaseException as exc:
|
|
if first_error is None:
|
|
first_error = exc
|
|
else:
|
|
self.logger.exception(f"Failed to close owned component {owned.name}: {exc}")
|
|
self._is_started = False
|
|
if first_error is not None:
|
|
raise first_error
|
|
|
|
async def restart(self) -> None:
|
|
"""Close then start the component."""
|
|
await self.close()
|
|
await self.start()
|
|
|
|
async def __call__(self, **kwargs):
|
|
raise NotImplementedError
|
|
|
|
async def __aenter__(self) -> "BaseComponent":
|
|
await self.start()
|
|
return self
|
|
|
|
async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
|
|
await self.close()
|