mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
### 1. Agent Wrapper(统一 Agent 后端抽象) - **`base_agent_wrapper.py`**:`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`(含 `session_id` / `last_message` / `result` / 可选 `structured_output`);`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`,改为 `add_job_tools(names: list[str])`(按名解析 BaseJob)与 `add_skills()`;新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`(AgentScope 后端)**: - 会话持久化重写:`session_path` 落地到 `<vault>/<session_dir>/agentscope/`,`_load_state` 支持 `resume` / `session_id` / `fork_session`,并做 UUID 校验(`_validate_session_id`);`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集(`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write),`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效;`_resolve_skills()` 把配置的 skill 暴露给后端,`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件(Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters)归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`(Claude Code SDK 后端,+551 行)**: - 新增 `_CcFileSessionStore`:基于 vault 的文件型会话存储,实现 append(按 uuid 去重)/ load / list / delete / list_subkeys,并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`:统一构建 `ClaudeAgentOptions`,处理 skills、disallowed_tools(默认禁 `WebSearch`)、`.env` 注入、Claude Code 的 API 凭据解析(`_claude_code_api_env`,多级 base_url/api_key 回退)、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接(`_ensure_claude_skill_dir`)。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`:把 Anthropic 流式事件(message_start/delta/stop、content_block_*)与 SDK 消息块(AssistantMessage/UserMessage/ResultMessage/RateLimitEvent)转换为统一 `StreamChunk`;跟踪 block_id/block_type/tool_call_name 做关联;处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议(StreamChunk / ChunkEnum) - **`stream_chunk.py`**:`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构,新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段,纯文本流仍保持轻量。 - **`chunk_enum.py`**:补全生命周期标记 `REPLY_START` / `REPLY_END`,并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构(变化批次化 + dispatch) - 新增 `_change_batch.py`:`coalesce_changes()` 把同路径多次事件折叠为最终状态(结合 path 存在性判定),`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`(`InitChangesStep`):一次性扫描,对比 file_store / file_catalog 已索引节点计算 added/modified/deleted,写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`:抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集;`UpdateCatalogStep`(写 file_catalog)、`UpdateIndexStep`(写 file_store,含按后缀解析 chunker)。 - **`watch_changes.py`**:改用 `dispatch_step_specs`(基类提供的 `dispatch_steps()`),每批先 `coalesce_changes` 再 dispatch;默认参数调整(debounce 5000ms / step 1000ms / poll 5000ms)并暴露常量。 - 删除旧步骤:`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧);`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块(拆分为多步 pipeline) - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`,新增 `dream/` 子包,按 5 个步骤组织: - **`extract.py`**:扫描当日 day-index + daily 笔记,对比 file_catalog 找出 changed/deleted,调用 LLM 全局抽取 `units`(procedure/personal/wiki 三桶)与 `topics`,路径与桶做清洗/路由。 - **`integrate.py`**:逐个 unit 调用 LLM 写入 digest,结构化输出 `IntegrateOutcome`(CREATE/CORROBORATE/REFINE/CORRECT),失败 unit/路径收集回写。 - **`topics.py`**:写 `daily/<date>/interests.yaml`,结合当天已有 + 近 N 天做去重(`normalize_topic`),可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**:读取当日 `interests.yaml`,作为主动推荐话题的入口。 - **`finish.py`**:把变更路径落盘到 dream file_catalog(checkpoint),渲染最终汇总摘要。 - 新增 `schema.py`(`DreamState` 等跨步骤共享状态与结构化输出模型)与 `utils.py`(状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数)。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource(适配新 Agent API) - **`auto_memory.py`**:会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`;改用 `job_tools`;新增 `source_conversation` frontmatter 反向链接(`_session_link`);执行后刷新 day 索引(`refresh_day_index`),并对 session_id 做合法性校验。 - **`auto_resource.py`**:资源改用「同名 daily note」方案(`_compute_note_stem` 取文件 stem);批量处理 `changes: list[dict]`(`_handle_change` 逐项处理,返回逐项结果摘要);agent 会话 id 用稳定的 `uuid5`;同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制:`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec,`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`:按 key 取 app config,缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理:`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**:持久化改用 zstd 压缩(`.jsonl.zst`,通过新 `utils/jsonl_zst.py`);upsert 时先删除旧 chunk 的 keyword 文档;embedding 复用改为 `(text, embedding)` 键控,要求文本一致才复用;新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**:索引文件名加入组件名 + tokenizer 指纹(sha256 前 12 位),快照/恢复时校验指纹防配置漂移;空索引 dump 时删除文件,加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**:弃用 `python-frontmatter`,改用内置 YAML 解析(非法 YAML 不阻断正文索引),并修正因 frontmatter 占用行号导致的 AST 行号偏移(`line_offset`)。 - **`cron_job.py`**:大幅简化(-187 行),由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」;`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整:service(base/http/mcp)、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron,启动失败会触发 `_close()` 回滚并 re-raise(不再吞异常)。 - 启动时创建 `session_dir` 目录;新增 `update_component()`(按类型/名就地更新已存在组件,不存在则报错)。 ### 9. File IO / 路径安全 - **`_path.py`**:`resolve_path` 增加 vault 越界防护(`is_relative_to` 校验),禁止 `.` / `..` 路径分量,支持 `allow_empty`。 - **`read.py`**:大文件(超过 `MAX_FILE_READ_BYTES`)走按行读取 `read_file_lines_safe`,避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐(如 `refresh_day_index`、`read_file_lines_safe`)。 - **`env_utils.py`**:新增 `parse_env_file()`,`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`(默认 `reme_session`)。 - `config_parser.py`:环境变量展开后做类型转换(`_convert_value`)、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`:作业编排改用 `init_changes_step` + `dispatch_steps`(index/resource/digest 三个 watch loop 与 reindex);新增 `auto_dream`(4 步)、`proactive` 作业,移除旧 `dream`;file_catalog 增配 `resource` / `digest` / `dream` 实例;LLM 默认值与 Claude Code 凭据配置调整(tool_result_limit 50000、thinking_enable=false 等)。 ### 11. 其它 - 新增 `steps/common/add.py`(`AddStep` 算术 demo)、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档:`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`,以及 `reme_design.md` 更新。 **
372 lines
10 KiB
Python
372 lines
10 KiB
Python
"""Tests for BaseComponent, Dependency, and ComponentMixin."""
|
|
|
|
# pylint: disable=protected-access,missing-function-docstring,missing-class-docstring,attribute-defined-outside-init
|
|
|
|
import asyncio
|
|
import os
|
|
import tempfile
|
|
|
|
import pytest
|
|
|
|
from reme4.components.base_component import BaseComponent, ComponentMixin, Dependency
|
|
from reme4.enumeration import ComponentEnum
|
|
|
|
|
|
# -- Test subclasses ----------------------------------------------------------
|
|
|
|
|
|
class StubComponent(BaseComponent):
|
|
component_type = ComponentEnum.FILE_CHUNKER
|
|
|
|
def __init__(self, **kwargs):
|
|
super().__init__(**kwargs)
|
|
self.start_count = 0
|
|
self.close_count = 0
|
|
|
|
async def _start(self):
|
|
self.start_count += 1
|
|
|
|
async def _close(self):
|
|
self.close_count += 1
|
|
|
|
|
|
class DepTarget(BaseComponent):
|
|
component_type = ComponentEnum.KEYWORD_INDEX
|
|
|
|
|
|
class RequiredDepTarget(BaseComponent):
|
|
component_type = ComponentEnum.FILE_GRAPH
|
|
|
|
|
|
class FailCloseComponent(StubComponent):
|
|
async def _close(self):
|
|
await super()._close()
|
|
raise RuntimeError("close failed")
|
|
|
|
|
|
# -- Dependency ---------------------------------------------------------------
|
|
|
|
|
|
def test_dependency_repr_optional():
|
|
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=True)
|
|
assert "?" in repr(dep)
|
|
assert "file_chunker" in repr(dep)
|
|
|
|
|
|
def test_dependency_repr_required():
|
|
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=False)
|
|
assert "?" not in repr(dep)
|
|
|
|
|
|
def test_dependency_getattr_raises():
|
|
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser")
|
|
with pytest.raises(RuntimeError, match="accessed before start"):
|
|
_ = dep.some_method
|
|
|
|
|
|
# -- bind ---------------------------------------------------------------------
|
|
|
|
|
|
def test_bind_returns_none_for_empty_name():
|
|
result = BaseComponent.bind(None, DepTarget)
|
|
assert result is None
|
|
|
|
result = BaseComponent.bind("", DepTarget)
|
|
assert result is None
|
|
|
|
|
|
def test_bind_returns_dependency_placeholder():
|
|
result = BaseComponent.bind("my_index", DepTarget)
|
|
assert isinstance(result, Dependency)
|
|
assert result.ctype == ComponentEnum.KEYWORD_INDEX
|
|
assert result.name == "my_index"
|
|
|
|
|
|
def test_bind_rejects_base_component_type():
|
|
class BadTarget(BaseComponent):
|
|
component_type = ComponentEnum.BASE
|
|
|
|
with pytest.raises(TypeError, match="non-BASE"):
|
|
BaseComponent.bind("x", BadTarget)
|
|
|
|
|
|
def test_bind_rejects_no_component_type():
|
|
class NoType:
|
|
pass
|
|
|
|
with pytest.raises(TypeError, match="non-BASE"):
|
|
BaseComponent.bind("x", NoType)
|
|
|
|
|
|
def test_bind_with_default_factory():
|
|
def factory():
|
|
return DepTarget(name="default")
|
|
|
|
result = BaseComponent.bind("idx", DepTarget, default_factory=factory)
|
|
assert isinstance(result, Dependency)
|
|
assert result.default_factory is factory
|
|
|
|
|
|
def test_bind_optional_flag():
|
|
dep = BaseComponent.bind("idx", DepTarget, optional=False)
|
|
assert dep.optional is False
|
|
|
|
|
|
# -- dependencies property ----------------------------------------------------
|
|
|
|
|
|
def test_dependencies_lists_unresolved():
|
|
comp = StubComponent()
|
|
comp.dep1 = Dependency(ComponentEnum.KEYWORD_INDEX, "a")
|
|
comp.dep2 = Dependency(ComponentEnum.FILE_GRAPH, "b")
|
|
comp.normal_attr = "not a dep"
|
|
deps = comp.dependencies
|
|
assert len(deps) == 2
|
|
|
|
|
|
# -- lifecycle ----------------------------------------------------------------
|
|
|
|
|
|
def test_start_close_idempotent():
|
|
async def run():
|
|
comp = StubComponent()
|
|
await comp.start()
|
|
await comp.start()
|
|
assert comp.start_count == 1
|
|
assert comp.is_started is True
|
|
|
|
await comp.close()
|
|
await comp.close()
|
|
assert comp.close_count == 1
|
|
assert comp.is_started is False
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_restart():
|
|
async def run():
|
|
comp = StubComponent()
|
|
await comp.start()
|
|
await comp.restart()
|
|
assert comp.start_count == 2
|
|
assert comp.close_count == 1
|
|
assert comp.is_started is True
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_async_context_manager():
|
|
async def run():
|
|
comp = StubComponent()
|
|
async with comp as c:
|
|
assert c is comp
|
|
assert comp.is_started is True
|
|
assert comp.is_started is False
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_close_closes_owned_when_parent_close_fails():
|
|
async def run():
|
|
owned = StubComponent(name="owned")
|
|
parent = FailCloseComponent(name="parent")
|
|
parent.dep = BaseComponent.bind(
|
|
"sub",
|
|
StubComponent,
|
|
default_factory=lambda: owned,
|
|
)
|
|
await parent.start()
|
|
|
|
with pytest.raises(RuntimeError, match="close failed"):
|
|
await parent.close()
|
|
|
|
assert owned.is_started is False
|
|
assert owned.close_count == 1
|
|
assert parent.is_started is False
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
# -- standalone resolution ----------------------------------------------------
|
|
|
|
|
|
def test_resolve_standalone_optional_becomes_none():
|
|
async def run():
|
|
comp = StubComponent()
|
|
comp.dep = BaseComponent.bind("idx", DepTarget)
|
|
await comp.start()
|
|
assert comp.dep is None
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_resolve_standalone_with_default_factory():
|
|
async def run():
|
|
comp = StubComponent()
|
|
comp.dep = BaseComponent.bind(
|
|
"idx",
|
|
DepTarget,
|
|
default_factory=lambda: DepTarget(name="auto"),
|
|
)
|
|
await comp.start()
|
|
assert isinstance(comp.dep, DepTarget)
|
|
assert comp.dep.name == "auto"
|
|
assert comp.dep in comp._owned
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_resolve_standalone_required_no_factory_keeps_placeholder():
|
|
async def run():
|
|
comp = StubComponent()
|
|
comp.dep = BaseComponent.bind("idx", DepTarget, optional=False)
|
|
await comp.start()
|
|
assert isinstance(comp.dep, Dependency)
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
# -- owned component lifecycle cascade ----------------------------------------
|
|
|
|
|
|
def test_owned_components_started_and_closed():
|
|
async def run():
|
|
owned = StubComponent(name="owned")
|
|
parent = StubComponent(name="parent")
|
|
parent.dep = BaseComponent.bind(
|
|
"sub",
|
|
StubComponent,
|
|
default_factory=lambda: owned,
|
|
)
|
|
await parent.start()
|
|
assert owned.is_started is True
|
|
|
|
await parent.close()
|
|
assert owned.is_started is False
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
# -- context-bound resolution -------------------------------------------------
|
|
|
|
|
|
def test_resolve_from_context():
|
|
async def run():
|
|
from reme4.components.application_context import ApplicationContext
|
|
|
|
target = DepTarget(name="real_index")
|
|
ctx = ApplicationContext()
|
|
ctx.components = {ComponentEnum.KEYWORD_INDEX: {"real_index": target}}
|
|
|
|
comp = StubComponent(app_context=ctx)
|
|
comp.dep = BaseComponent.bind("real_index", DepTarget)
|
|
await comp.start()
|
|
assert comp.dep is target
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_resolve_from_context_optional_missing():
|
|
async def run():
|
|
from reme4.components.application_context import ApplicationContext
|
|
|
|
ctx = ApplicationContext()
|
|
ctx.components = {}
|
|
comp = StubComponent(app_context=ctx)
|
|
comp.dep = BaseComponent.bind("missing", DepTarget, optional=True)
|
|
await comp.start()
|
|
assert comp.dep is None
|
|
await comp.close()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
def test_resolve_from_context_required_missing_raises():
|
|
async def run():
|
|
from reme4.components.application_context import ApplicationContext
|
|
|
|
ctx = ApplicationContext()
|
|
ctx.components = {}
|
|
comp = StubComponent(app_context=ctx)
|
|
comp.dep = BaseComponent.bind("missing", RequiredDepTarget, optional=False)
|
|
with pytest.raises(ValueError, match="not found"):
|
|
await comp.start()
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
# -- ComponentMixin paths -----------------------------------------------------
|
|
|
|
|
|
def test_vault_path_no_context():
|
|
mixin = ComponentMixin()
|
|
from pathlib import Path
|
|
|
|
assert mixin.vault_path == Path.cwd()
|
|
|
|
|
|
def test_to_vault_relative_inside_vault():
|
|
with tempfile.TemporaryDirectory() as tmpdir:
|
|
old_cwd = os.getcwd()
|
|
os.chdir(tmpdir)
|
|
try:
|
|
mixin = ComponentMixin()
|
|
abs_path = mixin.vault_path / "sub" / "file.md"
|
|
rel = mixin.to_vault_relative(abs_path)
|
|
assert rel == str(abs_path.relative_to(mixin.vault_path))
|
|
finally:
|
|
os.chdir(old_cwd)
|
|
|
|
|
|
def test_to_vault_relative_outside_vault():
|
|
mixin = ComponentMixin()
|
|
result = mixin.to_vault_relative("/some/other/path")
|
|
assert result == "/some/other/path"
|
|
|
|
|
|
# -- vault metadata paths -----------------------------------------------------
|
|
|
|
|
|
def test_vault_metadata_path_no_context():
|
|
comp = StubComponent()
|
|
assert comp.vault_metadata_path.name == "metadata"
|
|
|
|
|
|
def test_component_metadata_path():
|
|
comp = StubComponent()
|
|
assert comp.component_metadata_path.name == ComponentEnum.FILE_CHUNKER.value
|
|
|
|
|
|
if __name__ == "__main__":
|
|
print("\n=== BaseComponent Tests ===")
|
|
test_dependency_repr_optional()
|
|
test_dependency_repr_required()
|
|
test_dependency_getattr_raises()
|
|
test_bind_returns_none_for_empty_name()
|
|
test_bind_returns_dependency_placeholder()
|
|
test_bind_rejects_base_component_type()
|
|
test_bind_rejects_no_component_type()
|
|
test_bind_with_default_factory()
|
|
test_bind_optional_flag()
|
|
test_dependencies_lists_unresolved()
|
|
test_start_close_idempotent()
|
|
test_restart()
|
|
test_async_context_manager()
|
|
test_close_closes_owned_when_parent_close_fails()
|
|
test_resolve_standalone_optional_becomes_none()
|
|
test_resolve_standalone_with_default_factory()
|
|
test_resolve_standalone_required_no_factory_keeps_placeholder()
|
|
test_owned_components_started_and_closed()
|
|
test_resolve_from_context()
|
|
test_resolve_from_context_optional_missing()
|
|
test_resolve_from_context_required_missing_raises()
|
|
test_vault_path_no_context()
|
|
test_to_vault_relative_outside_vault()
|
|
test_vault_metadata_path_no_context()
|
|
test_component_metadata_path()
|
|
print("\n所有测试通过!")
|