diff --git a/docs4/old/background_backup.md b/docs4/old/background_backup.md index 1fde4440..4810d7c3 100644 --- a/docs4/old/background_backup.md +++ b/docs4/old/background_backup.md @@ -15,7 +15,7 @@ ReMe新版本V4 - 支持memory-self-evolving - [❌ 待补充] 没有发现自进化相关的 step/job 实现,目前只有基础的 search/reindex 等 common steps(`reme4/steps/common/`)。需要新增 auto-memory/auto-dream 等 step。 - 支持markdown之间的链接,构建graph,更好的渐进式展开 - - [✅ 已实现 → `reme4/components/file_parser/linked_file_parser.py`(wikilink 解析 + Dataview 谓词)、`reme4/components/file_graph/`(local/nx/neo4j 三种 graph 后端)] + - [✅ 已实现 → `reme4/components/file_chunker/markdown_file_chunker.py`(wikilink 解析 + Dataview 谓词)、`reme4/components/file_graph/`(local/nx/neo4j 三种 graph 后端)] - [✅ 已实现 → `reme4/steps/common/search.py:109` `_expand_links`、`reme4/config/default.yaml:86` `expand_links` 参数(搜索结果可附 outlinks/inlinks 邻居元数据)] 3. 工程实现: 1. components @@ -71,21 +71,21 @@ ReMe新版本V4 5. Markdown 格式 & Build Graph 1. obsidian格式的Markdown文件格式 - front matter格式 - - [✅ 已实现 → `reme4/components/file_parser/linked_file_parser.py:313` `frontmatter.loads(...)`;`reme4/schema/file_front_matter.py`] + - [✅ 已实现 → `reme4/components/file_chunker/markdown_file_chunker.py:313` `frontmatter.loads(...)`;`reme4/schema/file_front_matter.py`] - file link格式 4种格式 - - [⚠️ 部分实现] `linked_file_parser.py:88` `_WIKILINK_RE` 已支持 `[[target]]` / `[[target#anchor]]` / `[[target|alias]]` / `![[target]]`(嵌入),并支持 Dataview `predicate:: [[X]]` 与 inline `[predicate:: [[X]]]`。但 [❌ 待补充] 标准 Markdown `[text](url.md)` 链接尚未被解析为 graph 边。 + - [⚠️ 部分实现] `markdown_file_chunker.py:88` `_WIKILINK_RE` 已支持 `[[target]]` / `[[target#anchor]]` / `[[target|alias]]` / `![[target]]`(嵌入),并支持 Dataview `predicate:: [[X]]` 与 inline `[predicate:: [[X]]]`。但 [❌ 待补充] 标准 Markdown `[text](url.md)` 链接尚未被解析为 graph 边。 2. 更好的文件chunking机制 - 旧版 类似rag 带overlap的chunking机制 - [📌 历史] V3 旧逻辑,对照说明用,无需在 reme4 中实现。 - 解析 Markdown Ast - - [✅ 已实现 → `linked_file_parser.py:308` 使用 `mistletoe` 的 `Document`/`MarkdownRenderer`;`:335` `_build_tree` 把扁平 children 折叠成 section 嵌套树(`MdNode`)] + - [✅ 已实现 → `markdown_file_chunker.py:308` 使用 `mistletoe` 的 `Document`/`MarkdownRenderer`;`:335` `_build_tree` 把扁平 children 折叠成 section 嵌套树(`MdNode`)] - 每一个chunk都带全部标题 - - [✅ 已实现 → `linked_file_parser.py:381` `_chunk_node`(`before` 累积已经过的标题、`after` 拼剩余 desc_toc);`:712` `_make_chunk` 用 `_toc_join(before, content, after)` 把全文目录骨架前后包裹] + - [✅ 已实现 → `markdown_file_chunker.py:381` `_chunk_node`(`before` 累积已经过的标题、`after` 拼剩余 desc_toc);`:712` `_make_chunk` 用 `_toc_join(before, content, after)` 把全文目录骨架前后包裹] 3. 通过link构建graph索引,同时构建反向link索引 - [✅ 已实现 → `reme4/components/file_graph/base_file_graph.py`、`reme4/components/file_graph/local_file_graph.py`(含 `get_outlinks`、`get_inlinks` 双向索引);nx/neo4j 后端同 API;`reme4/steps/common/search.py:114-129` 使用双向 link] 4. link的生成有两种,一种是主agent在生成link;另一种是通过后台任务,自动构建文档之间的link - 介绍如何auto-link - - [⚠️ 部分实现] 主 agent 显式写 `[[link]]` 已经会被 parser 抓为边(`linked_file_parser.py:152` `_extract_links`)。但 [❌ 待补充] "后台任务自动补 link" 的实现(实体抽取 / 候选文档相似度匹配 / link 写回 markdown)尚不存在,需要单独的 step/job。 + - [⚠️ 部分实现] 主 agent 显式写 `[[link]]` 已经会被 parser 抓为边(`markdown_file_chunker.py:152` `_extract_links`)。但 [❌ 待补充] "后台任务自动补 link" 的实现(实体抽取 / 候选文档相似度匹配 / link 写回 markdown)尚不存在,需要单独的 step/job。 6. 如何做memory自进化 Auto-memory auto-dream @@ -120,7 +120,7 @@ V4更加高效的底层记忆索引 - 不支持关键词检索,这里需要Keyword倒排索引,对中文的支持较差 - [📌 历史] 描述 V3 痛点,不需要代码。 - V4版本我们重写了file parser,file store,file graph,file watcher,手写了支持增量更新倒排索引 - - file parser → [✅ `reme4/components/file_parser/`(base/default/chunked/linked 四种)] + - file parser → [✅ `reme4/components/file_chunker/`(base/default/chunked/linked 四种)] - file store → [✅ `reme4/components/file_store/local_file_store.py`] - file graph → [✅ `reme4/components/file_graph/`(local/nx/neo4j)] - file watcher → [✅ `reme4/components/file_watcher/lite_file_watcher.py` 基于 watchfiles awatch;`base_file_watcher.py` 抽象接口] @@ -146,7 +146,7 @@ xxxx 1. **组件框架**:backend 注册(`component_registry.py`)、生命周期(`base_component.py`)、依赖声明 + 拓扑启动(`application.py:80`)。 2. **Job/Step 体系**:`components/job/base_job.py`、`components/job/stream_job.py`、`steps/base_step.py` 与 `steps/common/*`。 3. **服务/客户端**:HTTP(`service/http_service.py` + `client/http_client.py`)、MCP(`service/mcp_service.py` + `client/mcp_client.py`),CLI 入口 `reme.py:main`。 -4. **Markdown 解析**:`file_parser/linked_file_parser.py`,含 frontmatter、wikilink + Dataview 谓词、AST 树、带全标题骨架的 chunking。 +4. **Markdown 解析**:`file_chunker/markdown_file_chunker.py`,含 frontmatter、wikilink + Dataview 谓词、AST 树、带全标题骨架的 chunking。 5. **Graph**:`file_graph/{local,nx,neo4j}_file_graph.py`,双向链接索引。 6. **存储 / 索引**:`file_store/local_file_store.py` + `keyword_index/bm25_index.py`(增量 BM25)+ `tokenizer/{regex,jieba}_tokenizer.py`。 7. **文件监听**:`file_watcher/lite_file_watcher.py`(watchfiles 轮询)。 diff --git a/docs4/old/reme4_index.md b/docs4/old/reme4_index.md index 00bf6c1e..8917232a 100644 --- a/docs4/old/reme4_index.md +++ b/docs4/old/reme4_index.md @@ -81,14 +81,13 @@ | `NxFileGraph` (`@R "nx"`) | `nx_file_graph.py` | networkx `MultiDiGraph` + pickle 持久化,虚节点用「无 node 属性」标识。 | | `Neo4jFileGraph` (`@R "neo4j"`) | `neo4j_file_graph.py` | Neo4j 后端(bolt 驱动),`(:File)-[:LINKS]->(:File)`,支持升降级虚节点、`rebuild_links` 修复重建。 | -### 4.5 File Parser — `reme4/components/file_parser/` +### 4.5 File Chunker — `reme4/components/file_chunker/` | 类 | 文件 | 说明 | | --- | --- | --- | -| `BaseFileParser` | `base_file_parser.py` | 抽象接口:`parse(path) -> (FileNode, list[FileChunk])`,提供 `_get_relative_path`。 | -| `DefaultFileParser` (`@R "default"`) | `default_file_parser.py` | 仅 stat:附件/二进制不读内容、不切块、不抽链接;作为无 `supported_extensions` 命中时的兜底 parser。 | -| `ChunkedFileParser` (`@R "chunked"`) | `chunked_file_parser.py` | 字节级带 overlap 切片 + YAML front matter + wikilink 抽取(含 Dataview `predicate::`)。 | -| `LinkedFileParser` (`@R "linked"`) | `linked_file_parser.py` | Markdown 专用:mistletoe AST → MdNode 树 → 章节递归分块;每个 chunk 携带完整 heading skeleton(TOC);wikilink 解析支持隐式 `.md`、folder-note、短路径歧义扇出,需注入 `file_graph` 解析目标。 | +| `BaseFileChunker` | `base_file_chunker.py` | 抽象接口:`parse(path) -> (FileNode, list[FileChunk])`,提供 `_get_relative_path`。 | +| `DefaultFileChunker` (`@R "default"`) | `default_file_chunker.py` | 字节级带 overlap 切片 + YAML front matter + wikilink 抽取(含 Dataview `predicate::`)。 | +| `MarkdownFileChunker` (`@R "markdown"`) | `markdown_file_chunker.py` | Markdown 专用:mistletoe AST → MdNode 树 → 章节递归分块;每个 chunk 携带完整 heading skeleton(TOC);wikilink 解析支持隐式 `.md`、folder-note、短路径歧义扇出,需注入 `file_graph` 解析目标。 | ### 4.6 File Store — `reme4/components/file_store/` @@ -104,7 +103,7 @@ | 类 | 文件 | 说明 | | --- | --- | --- | | `BaseFileWatcher` | `base_file_watcher.py` | 抽象接口:`watch_loop/update_store/on_added/on_modified/on_deleted`;启动后台任务先做一次全量同步再进入监听循环。 | -| `LiteFileWatcher` (`@R "lite"`) | `lite_file_watcher.py` | 基于 `watchfiles.awatch` 的轮询监听;变更分类后调用 file_parser 解析、写 file_store;`update_store` 通过 mtime 对比做增量。 | +| `LiteFileWatcher` (`@R "lite"`) | `lite_file_watcher.py` | 基于 `watchfiles.awatch` 的轮询监听;变更分类后调用 file_chunker 解析、写 file_store;`update_store` 通过 mtime 对比做增量。 | ### 4.8 Job — `reme4/components/job/` @@ -153,7 +152,7 @@ | 类 | 注册名 | 文件 | 作用 | | --- | --- | --- | --- | -| `BaseStep` | — | `base_step.py` | 抽象基类:`execute()` + `RuntimeContext` 注入 + `input/output_mapping` + 通过 `_resolve` 自动取组件(`as_llm/as_llm_formatter/as_token_counter/file_parser/file_store/embedding/file_watcher`);`add_as_tool(toolkit, job_name)` 把 job 包成 AgentScope tool。 | +| `BaseStep` | — | `base_step.py` | 抽象基类:`execute()` + `RuntimeContext` 注入 + `input/output_mapping` + 通过 `_resolve` 自动取组件(`as_llm/as_llm_formatter/as_token_counter/file_chunker/file_store/embedding/file_watcher`);`add_as_tool(toolkit, job_name)` 把 job 包成 AgentScope tool。 | | `DemoEchoStep1/2` | `demo_echo_step1` / `demo_echo_step2` | `common/demo.py` | 烟雾测试:query 处理 + 应答。 | | `HealthCheckStep` | `health_check_step` | `common/health_check.py` | 各组件健康/规模快照(embedding/file_graph/file_store/file_watcher/keyword_index)+ 内存深度估算。 | | `HelpStep` | `help_step` | `common/help.py` | 一行式列出全部 job 元信息(含参数 schema)。 | @@ -191,9 +190,9 @@ tokenizer (regex) ──┐ embedding_model (openai) ──┤── file_store (local) ── file_watcher (lite) file_graph (local) ──┤ (持有 embedding/keyword_index/file_graph) -file_parser (default) ──┘ │ +file_chunker (default) ──┘ │ keyword_index (bm25) ── tokenizer ──────────────┘ │ - file_parser ──┘ + file_chunker ──┘ ``` 启动时由 `Application._topological_order()`(Kahn 算法)按依赖拓扑序启动;关闭时反向。 @@ -205,7 +204,7 @@ keyword_index (bm25) ── tokenizer ───────────── | 新增检索后端 | 实现 `BaseFileStore` 子类,`@R.register("xxx")` | | 新增图后端 | 实现 `BaseFileGraph` 子类(参考 `Neo4jFileGraph` 处理虚节点) | | 新增分词器 | 实现 `BaseTokenizer.tokenize` | -| 新增解析器 | 实现 `BaseFileParser.parse`(返回 `(FileNode, list[FileChunk])`) | +| 新增解析器 | 实现 `BaseFileChunker.parse`(返回 `(FileNode, list[FileChunk])`) | | 新增 Job | 在 `default.yaml`(或自定义 yaml)`jobs:` 段声明 + 写步骤实现 | | 新增 Step | 继承 `BaseStep`,实现 `execute()` 并 `@R.register("xxx_step")` | | 暴露新协议 | 实现 `BaseService`(参考 `HttpService` / `MCPService`) | diff --git a/docs4/old/reme4_report.md b/docs4/old/reme4_report.md index 9b66aa28..b3e3a4bc 100644 --- a/docs4/old/reme4_report.md +++ b/docs4/old/reme4_report.md @@ -40,7 +40,7 @@ │ search · auto_memory · auto_dream · auto_link … │ ├─────────────────────────────────────────────────────────┤ │ Markdown 知识内核(本地文件即数据库) │ -│ FileParser · FileStore · FileGraph · FileWatcher │ +│ FileChunker · FileStore · FileGraph · FileWatcher │ │ BM25 倒排 · 向量索引 · Wiki Link 图谱 │ ├─────────────────────────────────────────────────────────┤ │ 文件目录约定 │ @@ -377,7 +377,7 @@ ReMe 自研增量 BM25 倒排索引,配合 jieba 中文分词: ReMe 把所有能力封装为 Component: ``` -embedding · file_store · file_graph · file_parser · file_watcher +embedding · file_store · file_graph · file_chunker · file_watcher tokenizer · keyword_index · LLM 适配 · service · client ``` @@ -909,7 +909,7 @@ type: personalization 新版本重写了记忆引擎的核心模块: -- **file parser** —— Markdown AST + 章节切片 + wikilink 抽取 +- **file chunker** —— Markdown AST + 章节切片 + wikilink 抽取 - **file store** —— 内存 chunk 字典 + JSONL 持久化 - **file graph** —— 双向链接索引,多 backend - **file watcher** —— 基于 watchfiles 的轻量监听 diff --git a/docs4/old/reme_design.md b/docs4/old/reme_design.md index d2e69904..39811721 100644 --- a/docs4/old/reme_design.md +++ b/docs4/old/reme_design.md @@ -123,7 +123,7 @@ traverse path=xxx direction=xxx depth=xxx | 主题dream + 生成链接 @sen | daily/xxx | knowledge/xxx | /dream | 把 daily 目录的内容按主题聚类合并到 topic 目录, 主动在文档中建立 [[link]] 关联 | | 主动proactive @wangce | daily / topic | proactive_query | pre_query | 思考 daily / topic 信息,主动决定推送给用户的消息 | -2. file_parser +2. file_chunker a. 抽象基类 parse: @jinli ⅰ. 输入是path:相对路径 ⅱ. 输出是FileMetadata & list[FileChunks] & list[FileEdge] diff --git a/pyproject.toml b/pyproject.toml index 7f4d7352..1543caa4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -37,7 +37,7 @@ dependencies = [ "prompt_toolkit>=3.0.52", "rich>=14.2.0", "asyncpg>=0.31.0", - "chromadb>=1.3.5", + "chromadb>=1.5.7", "dashscope>=1.25.1", "elasticsearch>=9.2.0", "fastapi>=0.121.3", diff --git a/reme4/application.py b/reme4/application.py index 7d737ec5..a26b56ba 100644 --- a/reme4/application.py +++ b/reme4/application.py @@ -48,7 +48,7 @@ class Application(BaseComponent): cfg = self.config vault_path = Path(cfg.vault_dir).absolute() vault_path.mkdir(parents=True, exist_ok=True) - for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.dialog_dir, cfg.daily_dir, cfg.digest_dir]: + for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]: if subdir: (vault_path / subdir).mkdir(parents=True, exist_ok=True) diff --git a/reme4/components/__init__.py b/reme4/components/__init__.py index f58e6740..a54e4a86 100644 --- a/reme4/components/__init__.py +++ b/reme4/components/__init__.py @@ -1,12 +1,13 @@ """Components""" -from . import llm +from . import agent_wrapper +from . import as_llm from . import client -from . import embedding +from . import as_embedding from . import embedding_store from . import file_catalog from . import file_graph -from . import file_parser +from . import file_chunker from . import file_store from . import job from . import keyword_index @@ -27,13 +28,14 @@ __all__ = [ "PromptHandler", "RuntimeContext", # base components - "llm", + "agent_wrapper", + "as_llm", "client", - "embedding", + "as_embedding", "embedding_store", "file_catalog", "file_graph", - "file_parser", + "file_chunker", "file_store", "job", "keyword_index", diff --git a/reme4/components/agent_wrapper/__init__.py b/reme4/components/agent_wrapper/__init__.py new file mode 100644 index 00000000..7e942e97 --- /dev/null +++ b/reme4/components/agent_wrapper/__init__.py @@ -0,0 +1,11 @@ +"""Unified agent wrapper component with swappable backends.""" + +from .base_agent_wrapper import BaseAgentWrapper +from .as_agent_wrapper import AsAgentWrapper +from .cc_agent_wrapper import CcAgentWrapper + +__all__ = [ + "BaseAgentWrapper", + "AsAgentWrapper", + "CcAgentWrapper", +] diff --git a/reme4/components/agent_wrapper/as_agent_wrapper.py b/reme4/components/agent_wrapper/as_agent_wrapper.py new file mode 100644 index 00000000..3c29952e --- /dev/null +++ b/reme4/components/agent_wrapper/as_agent_wrapper.py @@ -0,0 +1,84 @@ +"""AgentScope backend for the unified agent wrapper.""" + +from typing import Any, TYPE_CHECKING + +from agentscope.agent import Agent +from agentscope.message import TextBlock, ToolResultState, UserMsg, SystemMsg +from agentscope.tool import FunctionTool, ToolChunk, Toolkit + +from .base_agent_wrapper import BaseAgentWrapper +from ..as_llm import BaseAsLLM +from ..component_registry import R + +if TYPE_CHECKING: + from ..job.base_job import BaseJob + + +@R.register("agentscope") +class AsAgentWrapper(BaseAgentWrapper): + """Agent wrapper backed by AgentScope framework. + + Args: + as_llm: Name of the bound as_llm component (resolved via app_context). + Kwargs: + system_prompt: System prompt for the agent. + tools: list[BaseJob] to register as agent wrapper tools. + """ + + def __init__(self, as_llm: str = "default", **kwargs): + super().__init__(**kwargs) + self.as_llm = self.bind(as_llm, BaseAsLLM, optional=False) + + @staticmethod + def _make_tool(job: "BaseJob") -> FunctionTool: + async def run_job(**kwargs) -> ToolChunk: + response = await job(**kwargs) + return ToolChunk( + content=[TextBlock(text=str(response.answer))], + state=ToolResultState.SUCCESS if response.success else ToolResultState.ERROR, + ) + + tool = FunctionTool(func=run_job, name=job.name, description=job.description) + if job.parameters: + tool.input_schema = job.parameters + return tool + + async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]: + model = self.as_llm.model if self.as_llm else None + if model is None: + raise ValueError("AsAgentWrapper requires a bound as_llm component with a valid model.") + + for k, v in self.kwargs.items(): + kwargs.setdefault(k, v) + + output_schema: dict | None = kwargs.get("output_schema") + + tools: list["BaseJob"] = kwargs.get("tools", []) + toolkit = Toolkit(tools=[self._make_tool(job) for job in tools]) if tools else Toolkit() + + system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.") + + agent = Agent( + name=self.name, + system_prompt=system_prompt, + model=model, + toolkit=toolkit, + ) + + if isinstance(inputs, str): + inputs = UserMsg(name="user", content=inputs) + + if output_schema: + messages = [ + SystemMsg(name="system", content=system_prompt), + inputs, + ] + res = await model.generate_structured_output( + messages=messages, + structured_model=output_schema, + ) + return agent.state.session_id, res.content + + await agent.observe(inputs) + await agent.reply() + return agent.state.session_id, agent.state.context[-1] diff --git a/reme4/components/agent_wrapper/base_agent_wrapper.py b/reme4/components/agent_wrapper/base_agent_wrapper.py new file mode 100644 index 00000000..c3f83660 --- /dev/null +++ b/reme4/components/agent_wrapper/base_agent_wrapper.py @@ -0,0 +1,39 @@ +"""Base agent wrapper component.""" + +from abc import abstractmethod +from typing import Any, TYPE_CHECKING + +from ..base_component import BaseComponent +from ...enumeration import ComponentEnum + +if TYPE_CHECKING: + from ..job.base_job import BaseJob + + +class BaseAgentWrapper(BaseComponent): + """Abstract base for agent wrapper components with swappable backends. + + Subclasses implement reply() which returns (session_id, last_message). + Supports fluent configuration via set_system_prompt() and add_tools(). + """ + + component_type = ComponentEnum.AGENT_WRAPPER + + def set_system_prompt(self, prompt: str) -> "BaseAgentWrapper": + """Set the agent's system prompt. Returns self for chaining.""" + self.kwargs["system_prompt"] = prompt + return self + + def add_tools(self, tools: list["BaseJob"]) -> "BaseAgentWrapper": + """Append callable tools to the agent. Returns self for chaining.""" + self.kwargs.setdefault("tools", []).extend(tools) + return self + + def set_output_schema(self, schema: dict) -> "BaseAgentWrapper": + """Set a JSON schema for structured output. Returns self for chaining.""" + self.kwargs["output_schema"] = schema + return self + + @abstractmethod + async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]: + """Send inputs to the agent and return (session_id, last_message).""" diff --git a/reme4/components/agent_wrapper/cc_agent_wrapper.py b/reme4/components/agent_wrapper/cc_agent_wrapper.py new file mode 100644 index 00000000..d2d11d43 --- /dev/null +++ b/reme4/components/agent_wrapper/cc_agent_wrapper.py @@ -0,0 +1,86 @@ +"""Claude Code SDK backend for the unified agent wrapper.""" + +from typing import Any, TYPE_CHECKING + +from .base_agent_wrapper import BaseAgentWrapper +from ..component_registry import R + +if TYPE_CHECKING: + from ..job.base_job import BaseJob + + +@R.register("claude_code") +class CcAgentWrapper(BaseAgentWrapper): + """Agent wrapper backed by Claude Code SDK. + + Kwargs: + system_prompt: System prompt for the agent. + model: Claude model to use. + permission_mode: Permission mode for tool execution. + max_turns: Maximum conversation turns. + tools: list[BaseJob] to register as agent wrapper tools. + """ + + @staticmethod + def _make_tool(job: "BaseJob"): + from claude_agent_sdk import SdkMcpTool + + async def run_job(args): + response = await job(**args) + return { + "content": [{"type": "text", "text": str(response.answer)}], + "is_error": not response.success, + } + + return SdkMcpTool( + name=job.name, + description=job.description, + input_schema=job.parameters, + handler=run_job, + ) + + async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]: + from claude_agent_sdk import query, ResultMessage, create_sdk_mcp_server + from claude_agent_sdk.types import ClaudeAgentOptions + + for k, v in self.kwargs.items(): + kwargs.setdefault(k, v) + + opts = ClaudeAgentOptions() + opts.system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.") + opts.model = kwargs["model"] + opts.permission_mode = kwargs.get("permission_mode", None) + opts.max_turns = kwargs.get("max_turns", 50) + if session_id: + opts.session_id = session_id + opts.fork_session = True + + tools: list["BaseJob"] = kwargs.get("tools", []) + if tools: + sdk_tools = [self._make_tool(job) for job in tools] + server = create_sdk_mcp_server(name="reme_tools", tools=sdk_tools) + if isinstance(opts.mcp_servers, dict): + opts.mcp_servers["reme"] = server + else: + opts.mcp_servers = {"reme": server} + opts.allowed_tools.extend(job.name for job in tools) + + output_schema = kwargs.get("output_schema") + if output_schema: + opts.output_format = {"type": "json_schema", "schema": output_schema} + + if isinstance(inputs, str): + prompt = inputs + else: + raise NotImplementedError("Only string input is supported for Claude Code.") + + last_msg = None + async for msg in query(prompt=prompt, options=opts): + if isinstance(msg, ResultMessage): + last_msg = msg + + if last_msg is None: + raise ValueError("No message received from Claude Code.") + + result = last_msg.structured_output if output_schema and last_msg.structured_output else last_msg + return last_msg.session_id or "", result diff --git a/reme4/components/embedding/__init__.py b/reme4/components/as_embedding/__init__.py similarity index 77% rename from reme4/components/embedding/__init__.py rename to reme4/components/as_embedding/__init__.py index c8c4289b..995a3dfe 100644 --- a/reme4/components/embedding/__init__.py +++ b/reme4/components/as_embedding/__init__.py @@ -1,7 +1,7 @@ """AgentScope embedding model wrappers.""" from agentscope.embedding import ( - DashScopeMultiModalEmbedding as _AsDashScopeMultiModalEmbedding, + DashScopeMultiModalEmbedding, DashScopeTextEmbedding, EmbeddingModelBase, GeminiTextEmbedding, @@ -14,10 +14,10 @@ from ..component_registry import R from ...enumeration import ComponentEnum -class BaseEmbedding(BaseComponent): +class BaseAsEmbedding(BaseComponent): """Base wrapper for AgentScope embedding models. Builds ``self.model`` in ``_start``.""" - component_type = ComponentEnum.EMBEDDING + component_type = ComponentEnum.AS_EMBEDDING def __init__(self, **kwargs) -> None: super().__init__(**kwargs) @@ -39,7 +39,7 @@ class BaseEmbedding(BaseComponent): @R.register("openai") -class OpenAIEmbedding(BaseEmbedding): +class OpenAIAsEmbedding(BaseAsEmbedding): """OpenAI embedding model wrapper.""" async def _start(self) -> None: @@ -52,7 +52,7 @@ class OpenAIEmbedding(BaseEmbedding): @R.register("dashscope") -class DashScopeEmbedding(BaseEmbedding): +class DashScopeAsEmbedding(BaseAsEmbedding): """DashScope text embedding model wrapper.""" async def _start(self) -> None: @@ -60,15 +60,15 @@ class DashScopeEmbedding(BaseEmbedding): @R.register("dashscope_multimodal") -class DashScopeMultiModalEmbedding(BaseEmbedding): +class DashScopeMultiModalAsEmbedding(BaseAsEmbedding): """DashScope multimodal embedding model wrapper.""" async def _start(self) -> None: - self.model = _AsDashScopeMultiModalEmbedding(**self.kwargs) + self.model = DashScopeMultiModalEmbedding(**self.kwargs) @R.register("gemini") -class GeminiEmbedding(BaseEmbedding): +class GeminiAsEmbedding(BaseAsEmbedding): """Gemini embedding model wrapper.""" async def _start(self) -> None: @@ -76,7 +76,7 @@ class GeminiEmbedding(BaseEmbedding): @R.register("ollama") -class OllamaEmbedding(BaseEmbedding): +class OllamaAsEmbedding(BaseAsEmbedding): """Ollama embedding model wrapper.""" async def _start(self) -> None: @@ -84,10 +84,10 @@ class OllamaEmbedding(BaseEmbedding): __all__ = [ - "BaseEmbedding", - "OpenAIEmbedding", - "DashScopeEmbedding", - "DashScopeMultiModalEmbedding", - "GeminiEmbedding", - "OllamaEmbedding", + "BaseAsEmbedding", + "OpenAIAsEmbedding", + "DashScopeAsEmbedding", + "DashScopeMultiModalAsEmbedding", + "GeminiAsEmbedding", + "OllamaAsEmbedding", ] diff --git a/reme4/components/llm/__init__.py b/reme4/components/as_llm/__init__.py similarity index 81% rename from reme4/components/llm/__init__.py rename to reme4/components/as_llm/__init__.py index e2d212b4..7608986b 100644 --- a/reme4/components/llm/__init__.py +++ b/reme4/components/as_llm/__init__.py @@ -18,13 +18,13 @@ from ..component_registry import R from ...enumeration import ComponentEnum -class BaseLLM(BaseComponent): +class BaseAsLLM(BaseComponent): """Base wrapper for AgentScope chat models. Subclasses set ``credential_cls`` and inherit ``_start`` / ``_close``. """ - component_type = ComponentEnum.LLM + component_type = ComponentEnum.AS_LLM credential_cls: type[CredentialBase] def __init__(self, **kwargs) -> None: @@ -44,69 +44,69 @@ class BaseLLM(BaseComponent): @R.register("openai") -class OpenAILLM(BaseLLM): +class OpenAIAsLLM(BaseAsLLM): """OpenAI chat model wrapper.""" credential_cls = OpenAICredential @R.register("anthropic") -class AnthropicLLM(BaseLLM): +class AnthropicAsLLM(BaseAsLLM): """Anthropic chat model wrapper.""" credential_cls = AnthropicCredential @R.register("dashscope") -class DashScopeLLM(BaseLLM): +class DashScopeAsLLM(BaseAsLLM): """DashScope chat model wrapper.""" credential_cls = DashScopeCredential @R.register("deepseek") -class DeepSeekLLM(BaseLLM): +class DeepSeekAsLLM(BaseAsLLM): """DeepSeek chat model wrapper.""" credential_cls = DeepSeekCredential @R.register("gemini") -class GeminiLLM(BaseLLM): +class GeminiAsLLM(BaseAsLLM): """Gemini chat model wrapper.""" credential_cls = GeminiCredential @R.register("moonshot") -class MoonshotLLM(BaseLLM): +class MoonshotAsLLM(BaseAsLLM): """Moonshot chat model wrapper.""" credential_cls = MoonshotCredential @R.register("ollama") -class OllamaLLM(BaseLLM): +class OllamaAsLLM(BaseAsLLM): """Ollama chat model wrapper.""" credential_cls = OllamaCredential @R.register("xai") -class XAILLM(BaseLLM): +class XAIAsLLM(BaseAsLLM): """xAI chat model wrapper.""" credential_cls = XAICredential __all__ = [ - "BaseLLM", - "OpenAILLM", - "AnthropicLLM", - "DashScopeLLM", - "DeepSeekLLM", - "GeminiLLM", - "MoonshotLLM", - "OllamaLLM", - "XAILLM", + "BaseAsLLM", + "OpenAIAsLLM", + "AnthropicAsLLM", + "DashScopeAsLLM", + "DeepSeekAsLLM", + "GeminiAsLLM", + "MoonshotAsLLM", + "OllamaAsLLM", + "XAIAsLLM", ] diff --git a/reme4/components/embedding_store/local_embedding_store.py b/reme4/components/embedding_store/local_embedding_store.py index 186a4c43..1e10413e 100644 --- a/reme4/components/embedding_store/local_embedding_store.py +++ b/reme4/components/embedding_store/local_embedding_store.py @@ -9,7 +9,7 @@ import numpy as np from .base_embedding_store import BaseEmbeddingStore from ..component_registry import R -from ..embedding import BaseEmbedding +from ..as_embedding import BaseAsEmbedding Miss = tuple[int, str, str] # (result_index, text, cache_key) @@ -18,19 +18,19 @@ Miss = tuple[int, str, str] # (result_index, text, cache_key) class LocalEmbeddingStore(BaseEmbeddingStore): """Embedding store with LRU cache, disk persistence, and serial batching. - Delegates actual embedding computation to a bound ``embedding`` component. + Delegates actual embedding computation to a bound ``as_embedding`` component. """ def __init__( self, - embedding: str = "default", + as_embedding: str = "default", max_cache_size: int = 10000, enable_cache: bool = True, cache_version: str = "v1", **kwargs, ): super().__init__(**kwargs) - self.embedding = self.bind(embedding, BaseEmbedding, optional=False) + self.as_embedding = self.bind(as_embedding, BaseAsEmbedding, optional=False) self.max_cache_size = max_cache_size self.enable_cache = enable_cache self.cache_version = cache_version @@ -40,8 +40,8 @@ class LocalEmbeddingStore(BaseEmbeddingStore): @property def dimensions(self) -> int: """Return the embedding dimension size.""" - assert self.embedding is not None, "embedding component not bound" - return self.embedding.dimensions + assert self.as_embedding is not None, "embedding component not bound" + return self.as_embedding.dimensions @property def cache_path(self) -> Path: @@ -58,7 +58,7 @@ class LocalEmbeddingStore(BaseEmbeddingStore): async def health_check(self, timeout: float = 2.0) -> bool: tag = f"[EMBEDDING HEALTH CHECK] name={self.name}" try: - result = await asyncio.wait_for(self.embedding(["ping"]), timeout=timeout) + result = await asyncio.wait_for(self.as_embedding(["ping"]), timeout=timeout) if not result or result[0] is None: raise RuntimeError("empty embedding") self.is_healthy = True @@ -121,7 +121,7 @@ class LocalEmbeddingStore(BaseEmbeddingStore): async def _call_with_retry(self, texts: list[str], **kwargs) -> list[list[float] | None] | None: for attempt in range(self.max_retries): try: - result = await self.embedding(texts, **kwargs) + result = await self.as_embedding(texts, **kwargs) if result and len(result) == len(texts): return result except (TimeoutError, ConnectionError, OSError): diff --git a/reme4/components/file_chunker/__init__.py b/reme4/components/file_chunker/__init__.py new file mode 100644 index 00000000..d395a98a --- /dev/null +++ b/reme4/components/file_chunker/__init__.py @@ -0,0 +1,7 @@ +"""File chunker components.""" + +from .base_file_chunker import BaseFileChunker +from .default_file_chunker import DefaultFileChunker +from .markdown_file_chunker import MarkdownFileChunker + +__all__ = ["BaseFileChunker", "DefaultFileChunker", "MarkdownFileChunker"] diff --git a/reme4/components/file_parser/base_file_parser.py b/reme4/components/file_chunker/base_file_chunker.py similarity index 72% rename from reme4/components/file_parser/base_file_parser.py rename to reme4/components/file_chunker/base_file_chunker.py index 4d1621f4..53c59369 100644 --- a/reme4/components/file_parser/base_file_parser.py +++ b/reme4/components/file_chunker/base_file_chunker.py @@ -1,4 +1,4 @@ -"""Abstract base for file parsers.""" +"""Abstract base for file chunkers.""" from abc import abstractmethod from pathlib import Path @@ -8,10 +8,10 @@ from ...enumeration import ComponentEnum from ...schema import FileChunk, FileNode -class BaseFileParser(BaseComponent): - """Abstract base for file parsers. Subclasses implement `parse`.""" +class BaseFileChunker(BaseComponent): + """Abstract base for file chunkers. Subclasses implement `parse`.""" - component_type = ComponentEnum.FILE_PARSER + component_type = ComponentEnum.FILE_CHUNKER def __init__(self, supported_extensions: list[str] | None = None, **kwargs): super().__init__(**kwargs) diff --git a/reme4/components/file_parser/chunked_file_parser.py b/reme4/components/file_chunker/default_file_chunker.py similarity index 95% rename from reme4/components/file_parser/chunked_file_parser.py rename to reme4/components/file_chunker/default_file_chunker.py index 1725d3e0..2790ba60 100644 --- a/reme4/components/file_parser/chunked_file_parser.py +++ b/reme4/components/file_chunker/default_file_chunker.py @@ -1,4 +1,4 @@ -"""File parser with byte-based overlapping chunking.""" +"""Default file chunker — byte-based overlapping chunking.""" from bisect import bisect_right from pathlib import Path @@ -6,15 +6,15 @@ from pathlib import Path import aiofiles import yaml -from .base_file_parser import BaseFileParser +from .base_file_chunker import BaseFileChunker from ..component_registry import R from ...schema import FileChunk, FileFrontMatter, FileNode from ...utils.wikilink_handler import WikilinkHandler -@R.register("chunked") -class ChunkedFileParser(BaseFileParser): - """Parser that splits files into byte-based overlapping chunks.""" +@R.register("default") +class DefaultFileChunker(BaseFileChunker): + """Default chunker that splits files into byte-based overlapping chunks.""" def __init__(self, encoding: str = "utf-8", chunk_byte_size: int = 10000, overlap_byte_size: int = 100, **kwargs): super().__init__(**kwargs) diff --git a/reme4/components/file_parser/linked_file_parser.py b/reme4/components/file_chunker/markdown_file_chunker.py similarity index 98% rename from reme4/components/file_parser/linked_file_parser.py rename to reme4/components/file_chunker/markdown_file_chunker.py index bfd7522a..6439fbd6 100644 --- a/reme4/components/file_parser/linked_file_parser.py +++ b/reme4/components/file_chunker/markdown_file_chunker.py @@ -1,4 +1,4 @@ -"""Markdown file parser — frontmatter + wikilink graph + AST tree chunks. +"""Markdown file chunker — frontmatter + wikilink graph + AST tree chunks. Each chunk carries the **complete heading skeleton** of the document with its content inlined under the section that owns it; other sections @@ -21,7 +21,7 @@ from typing import Any import frontmatter -from .base_file_parser import BaseFileParser +from .base_file_chunker import BaseFileChunker from ..component_registry import R from ...schema import ( FileChunk, @@ -103,12 +103,12 @@ def _subtree_toc(n: MdNode) -> str: return f"{heading}\n\n{n.desc_toc}" if n.desc_toc else heading -# -- Parser --------------------------------------------------------------- +# -- Chunker -------------------------------------------------------------- -@R.register("linked") -class LinkedFileParser(BaseFileParser): - """Markdown parser: frontmatter + wikilink edges + full-skeleton chunks.""" +@R.register("markdown") +class MarkdownFileChunker(BaseFileChunker): + """Markdown chunker: frontmatter + wikilink edges + full-skeleton chunks.""" def __init__( self, diff --git a/reme4/components/file_graph/neo4j_file_graph.py b/reme4/components/file_graph/neo4j_file_graph.py index ec22a7ef..c2fbd13e 100644 --- a/reme4/components/file_graph/neo4j_file_graph.py +++ b/reme4/components/file_graph/neo4j_file_graph.py @@ -33,6 +33,7 @@ import error fires at ``_start`` (boot), not at first call. """ import json +import os from typing import Any from .base_file_graph import BaseFileGraph @@ -64,7 +65,7 @@ class Neo4jFileGraph(BaseFileGraph): Connection params (constructor kwargs): uri: bolt URL, e.g. ``bolt://localhost:7687`` user: auth user (default ``neo4j``) - password: auth password + password: auth password (required; falls back to ``NEO4J_PASSWORD`` env var) database: target db name (default ``neo4j``) """ @@ -72,14 +73,19 @@ class Neo4jFileGraph(BaseFileGraph): self, uri: str = "bolt://localhost:7687", user: str = "neo4j", - password: str = "neo4j", + password: str | None = None, database: str = "neo4j", **kwargs, ): super().__init__(**kwargs) self._uri: str = uri self._user: str = user - self._password: str = password + self._password: str = password or os.environ.get("NEO4J_PASSWORD") or "" + if not self._password: + raise ValueError( + "Neo4j password must be provided via the 'password' argument " + "or the NEO4J_PASSWORD environment variable.", + ) self._database: str = database self._driver = None diff --git a/reme4/components/file_parser/__init__.py b/reme4/components/file_parser/__init__.py deleted file mode 100644 index bf70a0cd..00000000 --- a/reme4/components/file_parser/__init__.py +++ /dev/null @@ -1,8 +0,0 @@ -"""File parser components.""" - -from .base_file_parser import BaseFileParser -from .chunked_file_parser import ChunkedFileParser -from .default_file_parser import DefaultFileParser -from .linked_file_parser import LinkedFileParser - -__all__ = ["BaseFileParser", "ChunkedFileParser", "DefaultFileParser", "LinkedFileParser"] diff --git a/reme4/components/file_parser/default_file_parser.py b/reme4/components/file_parser/default_file_parser.py deleted file mode 100644 index c799a9c3..00000000 --- a/reme4/components/file_parser/default_file_parser.py +++ /dev/null @@ -1,22 +0,0 @@ -"""Stat-only parser for attachment/binary files.""" - -from pathlib import Path - -from .base_file_parser import BaseFileParser -from ..component_registry import R -from ...schema import FileChunk, FileNode - - -@R.register("default") -class DefaultFileParser(BaseFileParser): - """Stat-only parser for attachment/binary files. - - No content read, no chunking, no link extraction. The resulting FileNode - has empty links and chunk_ids; front_matter carries mime and size so - retrieval can filter by file type without reopening the file. - """ - - async def parse(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]: - file_path = Path(path) - stat = file_path.stat() - return FileNode(path=self.to_vault_relative(path), st_mtime=stat.st_mtime, links=[], chunk_ids=[]), [] diff --git a/reme4/components/service/http_service.py b/reme4/components/service/http_service.py index 95ed5e72..11c7e711 100644 --- a/reme4/components/service/http_service.py +++ b/reme4/components/service/http_service.py @@ -46,10 +46,11 @@ class HttpService(BaseService): title=app.config.app_name, lifespan=self._lifespan(app, self.host, self.port), ) + cors_origins = ["*"] self.service.add_middleware( CORSMiddleware, # type: ignore[arg-type] - allow_origins=["*"], - allow_credentials=True, + allow_origins=cors_origins, + allow_credentials="*" not in cors_origins, allow_methods=["*"], allow_headers=["*"], ) diff --git a/reme4/config/default.yaml b/reme4/config/default.yaml index 782d5bcb..fab0b081 100644 --- a/reme4/config/default.yaml +++ b/reme4/config/default.yaml @@ -14,6 +14,7 @@ jobs: suffix_filters: [ "md" ] steps: - backend: scan_store_changes_step + recursive: true - backend: update_index_step persist: true - backend: watch_changes_step @@ -456,7 +457,7 @@ components: default: backend: regex - embedding: + as_embedding: default: backend: ${EMBEDDING_BACKEND:-openai} api_key: ${EMBEDDING_API_KEY:-} @@ -467,9 +468,9 @@ components: embedding_store: default: backend: local - embedding: default + as_embedding: default - llm: + as_llm: default: backend: ${LLM_BACKEND:-anthropic} model: ${LLM_MODEL_NAME:-glm-5.1} @@ -483,6 +484,16 @@ components: max_tokens: 65536 thinking_enable: true + agent_wrapper: + default: + backend: agentscope + as_llm: default + claude_code: + backend: claude_code + model: ${LLM_MODEL_NAME:-claude-opus-4-6} + permission_mode: bypassPermissions + max_turns: 10 + file_graph: default: backend: local @@ -491,15 +502,13 @@ components: default: backend: local - file_parser: - linked: - backend: linked + file_chunker: + markdown: + backend: markdown supported_extensions: [ "md" ] - chunked: - backend: chunked - supported_extensions: [ "txt", "html", "json", "jsonl", "yaml", "py" ] default: backend: default + supported_extensions: [ "txt", "html", "json", "jsonl", "yaml", "py" ] keyword_index: default: diff --git a/reme4/enumeration/component_enum.py b/reme4/enumeration/component_enum.py index db07dad3..61085465 100644 --- a/reme4/enumeration/component_enum.py +++ b/reme4/enumeration/component_enum.py @@ -8,13 +8,13 @@ class ComponentEnum(str, Enum): BASE = "base" - LLM = "llm" + AS_LLM = "as_llm" - EMBEDDING = "embedding" + AS_EMBEDDING = "as_embedding" EMBEDDING_STORE = "embedding_store" - FILE_PARSER = "file_parser" + FILE_CHUNKER = "file_chunker" FILE_STORE = "file_store" @@ -33,3 +33,5 @@ class ComponentEnum(str, Enum): JOB = "job" TOKENIZER = "tokenizer" + + AGENT_WRAPPER = "agent_wrapper" diff --git a/reme4/schema/application_config.py b/reme4/schema/application_config.py index a3b07d34..25e6e8b3 100644 --- a/reme4/schema/application_config.py +++ b/reme4/schema/application_config.py @@ -31,10 +31,10 @@ class ApplicationConfig(BaseModel): vault_dir: str = Field(default=".reme", description="Vault root directory for runtime files") metadata_dir: str = Field(default="reme_metadata", description="Subdirectory for ReMe persistent state") resource_dir: str = Field(default="resource", description="Subdirectory for external assets") - dialog_dir: str = Field(default="dialog", description="Subdirectory for dialog memory") daily_dir: str = Field(default="daily", description="Subdirectory for daily memory") digest_dir: str = Field(default="digest", description="Subdirectory for digest memory") enable_logo: bool = Field(default=True, description="Show ASCII logo on startup") + timezone: str | None = Field(default=None, description="IANA timezone (e.g. 'Asia/Shanghai'); None uses local time") language: str = Field(default="", description="Default language for LLM interactions") log_to_console: bool = Field(default=True, description="Log to console") log_to_file: bool = Field(default=True, description="Log to file") diff --git a/reme4/steps/base_step.py b/reme4/steps/base_step.py index ec507114..beeeb443 100644 --- a/reme4/steps/base_step.py +++ b/reme4/steps/base_step.py @@ -10,7 +10,7 @@ from agentscope.model import ChatModelBase from agentscope.tool import Toolkit, FunctionTool, ToolChunk from ..components.base_component import ComponentMixin -from ..components.file_parser import BaseFileParser +from ..components.file_chunker import BaseFileChunker from ..components.file_store import BaseFileStore from ..components.prompt_handler import PromptHandler from ..components.runtime_context import RuntimeContext @@ -32,7 +32,7 @@ class Ref: Replaces the ``@property`` + ``_resolve()`` boilerplate with a single class-level declaration:: - llm = Ref(ChatModelBase, ComponentEnum.LLM, "model") + as_llm = Ref(ChatModelBase, ComponentEnum.AS_LLM, "model") file_store = Ref(BaseFileStore, ComponentEnum.FILE_STORE) Resolution follows a 3-source fallback identical to the old ``_resolve``: @@ -102,7 +102,7 @@ class BaseStep(ComponentMixin, ABC): component_type = ComponentEnum.STEP - llm: ChatModelBase = Ref(ChatModelBase, ComponentEnum.LLM, "model") + as_llm: ChatModelBase = Ref(ChatModelBase, ComponentEnum.AS_LLM, "model") file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE) def __new__(cls, *args, **kwargs): @@ -165,24 +165,24 @@ class BaseStep(ComponentMixin, ABC): how attachments / binaries / unknown types still produce a FileNode. """ if self.app_context is None: - raise RuntimeError("app_context is not set when resolving file parser") - file_parser_dict: dict[str, BaseFileParser] = self.app_context.components[ComponentEnum.FILE_PARSER] + raise RuntimeError("app_context is not set when resolving file chunker") + file_chunker_dict: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER] suffix = Path(path).suffix.lstrip(".").lower() - parser: BaseFileParser | None = None + parser: BaseFileChunker | None = None if suffix: - for candidate in file_parser_dict.values(): + for candidate in file_chunker_dict.values(): if suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}: parser = candidate break if parser is None: - parser = file_parser_dict.get("default") + parser = file_chunker_dict.get("default") if parser is None: raise RuntimeError( - f"No file parser supports {path} (suffix={suffix!r}) and no 'default' parser is configured", + f"No file chunker supports {path} (suffix={suffix!r}) and no 'default' chunker is configured", ) return await parser.parse(path) diff --git a/reme4/steps/common/llm_demo.py b/reme4/steps/common/llm_demo.py index 06ec6403..55bf419f 100644 --- a/reme4/steps/common/llm_demo.py +++ b/reme4/steps/common/llm_demo.py @@ -1,4 +1,4 @@ -"""Demo step that drives an Agent via BaseStep.llm.""" +"""Demo step that drives an Agent via BaseStep.as_llm.""" from typing import Type @@ -25,7 +25,7 @@ def add(a: float, b: float) -> str: @R.register("llm_demo_step") class LLMDemoStep(BaseStep): - """Drive an Agent powered by ``self.llm``. + """Drive an Agent powered by ``self.as_llm``. Inputs (from RuntimeContext): query (str, required): user message content. @@ -55,7 +55,7 @@ class LLMDemoStep(BaseStep): agent = Agent( name=self.name, system_prompt=sys_prompt, - model=self.llm, + model=self.as_llm, toolkit=toolkit, state=AgentState( permission_context=PermissionContext( @@ -72,7 +72,7 @@ class LLMDemoStep(BaseStep): structured_content: dict | None = None if structured_model is not None: - structured_resp = await self.llm.generate_structured_output( + structured_resp = await self.as_llm.generate_structured_output( agent.state.context, structured_model=structured_model, ) diff --git a/reme4/steps/common/stream_llm_demo.py b/reme4/steps/common/stream_llm_demo.py index 3ff647fb..7513840b 100644 --- a/reme4/steps/common/stream_llm_demo.py +++ b/reme4/steps/common/stream_llm_demo.py @@ -1,4 +1,4 @@ -"""Demo step that drives an Agent via BaseStep.llm with streaming output.""" +"""Demo step that drives an Agent via BaseStep.as_llm with streaming output.""" import json @@ -34,7 +34,7 @@ def add(a: float, b: float) -> str: @R.register("stream_llm_demo_step") class StreamLLMDemoStep(BaseStep): - """Drive an Agent powered by ``self.llm`` with streaming output. + """Drive an Agent powered by ``self.as_llm`` with streaming output. When streaming is enabled on the context, text/thinking/tool events are pushed chunk-by-chunk via ``self.context.add_stream_string``. @@ -67,7 +67,7 @@ class StreamLLMDemoStep(BaseStep): agent = Agent( name=self.name, system_prompt=sys_prompt, - model=self.llm, + model=self.as_llm, toolkit=toolkit, state=AgentState( permission_context=PermissionContext( diff --git a/reme4/steps/evolve/__init__.py b/reme4/steps/evolve/__init__.py index e69de29b..9895edab 100644 --- a/reme4/steps/evolve/__init__.py +++ b/reme4/steps/evolve/__init__.py @@ -0,0 +1,5 @@ +"""Evolve steps.""" + +from ._evolve import now + +__all__ = ["now"] diff --git a/reme4/steps/evolve/auto_memory.py b/reme4/steps/evolve/auto_memory.py index 7806f885..3b0b4800 100644 --- a/reme4/steps/evolve/auto_memory.py +++ b/reme4/steps/evolve/auto_memory.py @@ -108,7 +108,7 @@ class AutoMemoryStep(BaseStep): agent = Agent( name="auto_memory", - model=self.llm, + model=self.as_llm, system_prompt=self.prompt_format("system_prompt"), toolkit=toolkit, state=AgentState( diff --git a/reme4/steps/evolve/dream.py b/reme4/steps/evolve/dream.py index f4218c41..52008848 100644 --- a/reme4/steps/evolve/dream.py +++ b/reme4/steps/evolve/dream.py @@ -269,7 +269,7 @@ class DreamStep(BaseStep): def _llm_available(self) -> bool: try: - return self.llm is not None + return self.as_llm is not None except Exception: return False @@ -303,7 +303,7 @@ class DreamStep(BaseStep): toolkit = self._build_extract_toolkit() agent = Agent( name="reme_dreamer_extract", - model=self.llm, + model=self.as_llm, system_prompt=self.prompt_format( "extract_system_prompt", vault_dir=str(vault_dir), @@ -326,7 +326,7 @@ class DreamStep(BaseStep): Msg(name="reme", role="user", content=[TextBlock(text=user_message)]), ) - structured_resp = await self.llm.generate_structured_output( + structured_resp = await self.as_llm.generate_structured_output( agent.state.context, structured_model=ExtractedUnits, ) @@ -362,7 +362,7 @@ class DreamStep(BaseStep): digest_dir = getattr(self.app_context.app_config, "digest_dir", "") agent = Agent( name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}", - model=self.llm, + model=self.as_llm, system_prompt=self.prompt_format( f"integrate_system_prompt_{bucket}", vault_dir=str(vault_dir), @@ -387,7 +387,7 @@ class DreamStep(BaseStep): await agent.reply( Msg(name="reme", role="user", content=[TextBlock(text=user_message)]), ) - structured_resp = await self.llm.generate_structured_output( + structured_resp = await self.as_llm.generate_structured_output( agent.state.context, structured_model=IntegrateOutcome, ) diff --git a/reme4/steps/file_io/daily_create.py b/reme4/steps/file_io/daily_create.py index 3e63cac5..7f1ba1f8 100644 --- a/reme4/steps/file_io/daily_create.py +++ b/reme4/steps/file_io/daily_create.py @@ -24,7 +24,6 @@ Outputs: metadata = {date, session_id, path, created, index?} """ -from datetime import date as _date from pathlib import Path import frontmatter @@ -33,6 +32,7 @@ from ._daily_index import refresh_day_index, validate_session_id from ._file_io import write_file_safe from ..base_step import BaseStep from ...components import R +from ...steps.evolve import now @R.register("daily_create_step") @@ -51,7 +51,8 @@ class DailyCreateStep(BaseStep): """Read ``session_id`` + ``date`` from context; default ``date`` today, ``daily_dir`` from app config.""" assert self.context is not None session_id = self.context.get("session_id", "") - day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d") + tz = self.app_context.app_config.timezone if self.app_context is not None else None + day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" return session_id, day, daily_dir diff --git a/reme4/steps/file_io/daily_list.py b/reme4/steps/file_io/daily_list.py index 549a6404..02408a67 100644 --- a/reme4/steps/file_io/daily_list.py +++ b/reme4/steps/file_io/daily_list.py @@ -12,12 +12,12 @@ Input is a single optional ``date`` (``YYYY-MM-DD``); falls back to today. """ -from datetime import date as _date from pathlib import Path from ._daily_index import scan_notes from ..base_step import BaseStep from ...components import R +from ...steps.evolve import now @R.register("daily_list_step") @@ -27,7 +27,8 @@ class DailyListStep(BaseStep): def _collect_params(self) -> tuple[str, str, Path]: """Read ``date`` (default today, ``YYYY-MM-DD``), resolve ``daily_dir``, locate the vault root on disk.""" assert self.context is not None - day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d") + tz = self.app_context.app_config.timezone if self.app_context is not None else None + day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" vault_dir = Path(self.file_store.vault_path or ".").resolve() return day, daily_dir, vault_dir diff --git a/reme4/steps/file_io/daily_reindex.py b/reme4/steps/file_io/daily_reindex.py index e09cf66e..6264fb13 100644 --- a/reme4/steps/file_io/daily_reindex.py +++ b/reme4/steps/file_io/daily_reindex.py @@ -19,11 +19,10 @@ today. Always idempotent and safe to re-run. """ -from datetime import date as _date - from ._daily_index import refresh_day_index from ..base_step import BaseStep from ...components import R +from ...steps.evolve import now @R.register("daily_reindex_step") @@ -33,7 +32,8 @@ class DailyReindexStep(BaseStep): def _collect_params(self) -> tuple[str, str]: """Read ``date`` (default today) and ``daily_dir`` (default ``daily``) from context/app config.""" assert self.context is not None - day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d") + tz = self.app_context.app_config.timezone if self.app_context is not None else None + day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d") daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily" return day, daily_dir diff --git a/reme4/utils/wikilink_handler.py b/reme4/utils/wikilink_handler.py index ba93bf33..f654d727 100644 --- a/reme4/utils/wikilink_handler.py +++ b/reme4/utils/wikilink_handler.py @@ -4,7 +4,7 @@ One class, :class:`WikilinkHandler`, owning every wikilink concern: * **Pure text** — regex, Dataview predicate inference, validation: :meth:`~WikilinkHandler.extract_links` (used by - :mod:`reme.components.file_parser.linked_file_parser`), + :mod:`reme.components.file_chunker.markdown_file_chunker`), :meth:`~WikilinkHandler.scan_and_rewrite`, :meth:`~WikilinkHandler.validate_src_dst` / :meth:`~WikilinkHandler.validate_scope` / diff --git a/tests4/unit/test_background_steps.py b/tests4/unit/test_background_steps.py index afc83dff..ef329831 100644 --- a/tests4/unit/test_background_steps.py +++ b/tests4/unit/test_background_steps.py @@ -1,7 +1,7 @@ """Tests for background steps: ScanStoreChangesStep + WatchChangesStep. Both steps are subclasses of BaseStep. To exercise them without spinning up the -full ApplicationContext, we pass real (started) file_store/file_parser via the +full ApplicationContext, we pass real (started) file_store/file_chunker via the step's kwargs (so the BaseStep _resolve() machinery returns them). ScanStoreChangesStep writes its result into ``context["changes"]`` for a downstream @@ -22,7 +22,7 @@ from pathlib import Path from watchfiles import Change -from reme4.components.file_parser import ChunkedFileParser +from reme4.components.file_chunker import DefaultFileChunker from reme4.components.file_store import LocalFileStore from reme4.components.runtime_context import RuntimeContext from reme4.steps import ScanStoreChangesStep, WatchChangesStep @@ -63,15 +63,15 @@ async def _make_scan_step( watch_paths: list[str] | str = "vault", suffix_filters: list[str] | None = None, recursive: bool = True, -) -> tuple[ScanStoreChangesStep, RuntimeContext, LocalFileStore, ChunkedFileParser]: +) -> tuple[ScanStoreChangesStep, RuntimeContext, LocalFileStore, DefaultFileChunker]: fs = LocalFileStore(name="test_store", embedding_store="") - parser = ChunkedFileParser() + parser = DefaultFileChunker() await fs.start() await parser.start() step = ScanStoreChangesStep( recursive=recursive, file_store=fs, - file_parser=parser, + file_chunker=parser, ) context = RuntimeContext( watch_paths=watch_paths, @@ -80,7 +80,7 @@ async def _make_scan_step( return step, context, fs, parser -async def _teardown(fs: LocalFileStore, parser: ChunkedFileParser) -> None: +async def _teardown(fs: LocalFileStore, parser: DefaultFileChunker) -> None: await parser.close() await fs.close() diff --git a/tests4/unit/test_base_component.py b/tests4/unit/test_base_component.py index 667de251..ba64dea7 100644 --- a/tests4/unit/test_base_component.py +++ b/tests4/unit/test_base_component.py @@ -16,7 +16,7 @@ from reme4.enumeration import ComponentEnum class StubComponent(BaseComponent): - component_type = ComponentEnum.FILE_PARSER + component_type = ComponentEnum.FILE_CHUNKER def __init__(self, **kwargs): super().__init__(**kwargs) @@ -42,18 +42,18 @@ class RequiredDepTarget(BaseComponent): def test_dependency_repr_optional(): - dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser", optional=True) + dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=True) assert "?" in repr(dep) - assert "file_parser" in repr(dep) + assert "file_chunker" in repr(dep) def test_dependency_repr_required(): - dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser", optional=False) + dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=False) assert "?" not in repr(dep) def test_dependency_getattr_raises(): - dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser") + dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser") with pytest.raises(RuntimeError, match="accessed before start"): _ = dep.some_method @@ -312,7 +312,7 @@ def test_vault_metadata_path_no_context(): def test_component_metadata_path(): comp = StubComponent() - assert comp.component_metadata_path.name == ComponentEnum.FILE_PARSER.value + assert comp.component_metadata_path.name == ComponentEnum.FILE_CHUNKER.value if __name__ == "__main__": diff --git a/tests4/unit/test_component_registry.py b/tests4/unit/test_component_registry.py index 31e39bfb..33a0ced2 100644 --- a/tests4/unit/test_component_registry.py +++ b/tests4/unit/test_component_registry.py @@ -10,7 +10,7 @@ from reme4.enumeration import ComponentEnum class _DummyComponent(BaseComponent): - component_type = ComponentEnum.FILE_PARSER + component_type = ComponentEnum.FILE_CHUNKER class _AnotherComponent(BaseComponent): @@ -31,13 +31,13 @@ class _BaseComponentType(BaseComponent): def test_register_direct_with_explicit_name(): reg = ComponentRegistry() reg.register(_DummyComponent, "my_parser") - assert reg.get(ComponentEnum.FILE_PARSER, "my_parser") is _DummyComponent + assert reg.get(ComponentEnum.FILE_CHUNKER, "my_parser") is _DummyComponent def test_register_direct_defaults_to_class_name(): reg = ComponentRegistry() reg.register(_DummyComponent) - assert reg.get(ComponentEnum.FILE_PARSER, "_DummyComponent") is _DummyComponent + assert reg.get(ComponentEnum.FILE_CHUNKER, "_DummyComponent") is _DummyComponent def test_register_decorator(): @@ -45,16 +45,16 @@ def test_register_decorator(): @reg.register("alias") class MyParser(BaseComponent): - component_type = ComponentEnum.FILE_PARSER + component_type = ComponentEnum.FILE_CHUNKER - assert reg.get(ComponentEnum.FILE_PARSER, "alias") is MyParser + assert reg.get(ComponentEnum.FILE_CHUNKER, "alias") is MyParser def test_register_overwrite_warns(caplog): reg = ComponentRegistry() reg.register(_DummyComponent, "dup") reg.register(_DummyComponent, "dup") - assert reg.get(ComponentEnum.FILE_PARSER, "dup") is _DummyComponent + assert reg.get(ComponentEnum.FILE_CHUNKER, "dup") is _DummyComponent def test_register_rejects_missing_component_type(): @@ -83,7 +83,7 @@ def test_get_all_returns_copy(): reg.register(_DummyComponent, "a") reg.register(_AnotherComponent, "b") - parsers = reg.get_all(ComponentEnum.FILE_PARSER) + parsers = reg.get_all(ComponentEnum.FILE_CHUNKER) assert parsers == {"a": _DummyComponent} indexes = reg.get_all(ComponentEnum.KEYWORD_INDEX) @@ -91,12 +91,12 @@ def test_get_all_returns_copy(): # Mutating the copy doesn't affect the registry. parsers["hacked"] = _DummyComponent - assert "hacked" not in reg.get_all(ComponentEnum.FILE_PARSER) + assert "hacked" not in reg.get_all(ComponentEnum.FILE_CHUNKER) def test_get_all_unknown_type_returns_empty(): reg = ComponentRegistry() - assert not reg.get_all(ComponentEnum.LLM) + assert not reg.get_all(ComponentEnum.AS_LLM) # -- get (miss) --------------------------------------------------------------- @@ -104,7 +104,7 @@ def test_get_all_unknown_type_returns_empty(): def test_get_nonexistent_returns_none(): reg = ComponentRegistry() - assert reg.get(ComponentEnum.FILE_PARSER, "nope") is None + assert reg.get(ComponentEnum.FILE_CHUNKER, "nope") is None # -- unregister --------------------------------------------------------------- @@ -113,13 +113,13 @@ def test_get_nonexistent_returns_none(): def test_unregister_existing(): reg = ComponentRegistry() reg.register(_DummyComponent, "x") - assert reg.unregister(ComponentEnum.FILE_PARSER, "x") is True - assert reg.get(ComponentEnum.FILE_PARSER, "x") is None + assert reg.unregister(ComponentEnum.FILE_CHUNKER, "x") is True + assert reg.get(ComponentEnum.FILE_CHUNKER, "x") is None def test_unregister_missing_returns_false(): reg = ComponentRegistry() - assert reg.unregister(ComponentEnum.FILE_PARSER, "nope") is False + assert reg.unregister(ComponentEnum.FILE_CHUNKER, "nope") is False # -- clear -------------------------------------------------------------------- @@ -130,7 +130,7 @@ def test_clear(): reg.register(_DummyComponent, "a") reg.register(_AnotherComponent, "b") reg.clear() - assert not reg.get_all(ComponentEnum.FILE_PARSER) + assert not reg.get_all(ComponentEnum.FILE_CHUNKER) assert not reg.get_all(ComponentEnum.KEYWORD_INDEX) diff --git a/tests4/unit/test_chunked_file_parser.py b/tests4/unit/test_default_file_chunker.py similarity index 94% rename from tests4/unit/test_chunked_file_parser.py rename to tests4/unit/test_default_file_chunker.py index e9b488f9..7c46526c 100644 --- a/tests4/unit/test_chunked_file_parser.py +++ b/tests4/unit/test_default_file_chunker.py @@ -1,10 +1,10 @@ -"""Tests for ChunkedFileParser.""" +"""Tests for DefaultFileChunker.""" import asyncio import os import tempfile -from reme4.components.file_parser import ChunkedFileParser +from reme4.components.file_chunker import DefaultFileChunker from reme4.utils.wikilink_handler import WikilinkHandler @@ -19,7 +19,7 @@ def test_parse_empty_file(): temp_path = f.name try: - parser = ChunkedFileParser() + parser = DefaultFileChunker() file_node, chunks = await parser.parse(temp_path) assert file_node.path == temp_path assert len(chunks) == 0 @@ -40,7 +40,7 @@ def test_parse_small_file(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=10000) + parser = DefaultFileChunker(chunk_byte_size=10000) _, chunks = await parser.parse(temp_path) assert len(chunks) == 1 assert chunks[0].start_line == 1 @@ -64,7 +64,7 @@ def test_parse_multiline_file(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=10000) + parser = DefaultFileChunker(chunk_byte_size=10000) _, chunks = await parser.parse(temp_path) assert len(chunks) == 1 assert chunks[0].start_line == 1 @@ -88,7 +88,7 @@ def test_parse_chunked_file(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=5000, overlap_byte_size=100) + parser = DefaultFileChunker(chunk_byte_size=5000, overlap_byte_size=100) _, chunks = await parser.parse(temp_path) assert len(chunks) > 1, f"Expected multiple chunks, got {len(chunks)}" # Verify overlap by checking that consecutive chunks share some content @@ -110,7 +110,7 @@ def test_parse_with_custom_encoding(): temp_path = f.name try: - parser = ChunkedFileParser(encoding="utf-8") + parser = DefaultFileChunker(encoding="utf-8") _, chunks = await parser.parse(temp_path) assert len(chunks) >= 1 assert "你好世界" in chunks[0].text @@ -131,7 +131,7 @@ def test_file_node_properties(): temp_path = f.name try: - parser = ChunkedFileParser() + parser = DefaultFileChunker() file_node, _ = await parser.parse(temp_path) assert hasattr(file_node, "path") assert hasattr(file_node, "st_mtime") @@ -153,7 +153,7 @@ def test_file_chunk_properties(): temp_path = f.name try: - parser = ChunkedFileParser() + parser = DefaultFileChunker() _, chunks = await parser.parse(temp_path) chunk = chunks[0] assert hasattr(chunk, "path") @@ -298,7 +298,7 @@ def test_parse_links_in_file(): temp_path = f.name try: - parser = ChunkedFileParser() + parser = DefaultFileChunker() file_node, _ = await parser.parse(temp_path) triples = {(link.predicate, link.target_path, link.target_anchor) for link in file_node.links} assert (None, "alpha", None) in triples @@ -326,7 +326,7 @@ def test_parse_links_empty_when_no_content(): fm_only_path = f.name try: - parser = ChunkedFileParser() + parser = DefaultFileChunker() node1, _ = await parser.parse(empty_path) node2, _ = await parser.parse(fm_only_path) assert node1.links == [] @@ -354,7 +354,7 @@ def test_chunk_does_not_split_wikilink_at_boundary(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=10) + parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=10) _, chunks = await parser.parse(temp_path) # The first chunk must NOT contain a partial link. first = chunks[0].text @@ -383,7 +383,7 @@ def test_chunk_does_not_split_wikilink_in_overlap(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=20) + parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=20) _, chunks = await parser.parse(temp_path) # No chunk should start mid-link. for c in chunks: @@ -413,7 +413,7 @@ def test_chunk_falls_back_for_oversize_link(): temp_path = f.name try: - parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=10) + parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=10) _, chunks = await parser.parse(temp_path) # Must terminate (not hang) and cover the whole file. assert len(chunks) >= 2 @@ -429,7 +429,7 @@ def test_min_chunk_and_overlap_size(): async def run(): # These values should be clamped to minimums - parser = ChunkedFileParser(chunk_byte_size=1, overlap_byte_size=0) + parser = DefaultFileChunker(chunk_byte_size=1, overlap_byte_size=0) assert parser.chunk_byte_size == 100 # minimum assert parser.overlap_byte_size == 4 # minimum diff --git a/tests4/unit/test_linked_file_parser.py b/tests4/unit/test_markdown_file_chunker.py similarity index 91% rename from tests4/unit/test_linked_file_parser.py rename to tests4/unit/test_markdown_file_chunker.py index ce82427c..ad6a87bb 100644 --- a/tests4/unit/test_linked_file_parser.py +++ b/tests4/unit/test_markdown_file_chunker.py @@ -1,4 +1,4 @@ -"""Tests for LinkedFileParser (markdown parser + wikilink extraction). +"""Tests for MarkdownFileChunker (markdown parser + wikilink extraction). Wikilink convention here is strict: targets are taken literally, no short-form basename search, no implicit ``.md``, no folder-note @@ -12,7 +12,7 @@ import asyncio import os import tempfile -from reme4.components.file_parser import LinkedFileParser +from reme4.components.file_chunker import MarkdownFileChunker class temp_chdir: @@ -46,7 +46,7 @@ def test_parse_empty_file(): async def run(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): path = _write_md(tmp, "x.md", "") - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, chunks = await parser.parse(path) assert node.path == "x.md" assert chunks == [] @@ -62,7 +62,7 @@ def test_parse_frontmatter_only(): async def run(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): path = _write_md(tmp, "fm.md", "---\nname: t\n---\n") - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, chunks = await parser.parse(path) assert node.front_matter.name == "t" assert chunks == [] @@ -79,7 +79,7 @@ def test_parse_small_body_one_chunk(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "# Hello\n\nthis is a small body." path = _write_md(tmp, "small.md", body) - parser = LinkedFileParser(chunk_chars=500) + parser = MarkdownFileChunker(chunk_chars=500) node, chunks = await parser.parse(path) assert len(chunks) == 1 assert "this is a small body" in chunks[0].text @@ -97,7 +97,7 @@ def test_parse_oversized_body_splits(): paras = "\n\n".join(f"paragraph {i} with some content text here." for i in range(50)) body = "# H\n\n" + paras path = _write_md(tmp, "big.md", body) - parser = LinkedFileParser(chunk_chars=200) + parser = MarkdownFileChunker(chunk_chars=200) _, chunks = await parser.parse(path) assert len(chunks) > 1 print("✓ test_parse_oversized_body_splits passed") @@ -113,7 +113,7 @@ def test_parse_chunk_ids_match_node_chunk_ids(): paras = "\n\n".join(f"para {i} body content here." for i in range(40)) body = "# H\n\n" + paras path = _write_md(tmp, "p.md", body) - parser = LinkedFileParser(chunk_chars=200) + parser = MarkdownFileChunker(chunk_chars=200) node, chunks = await parser.parse(path) assert node.chunk_ids == [c.id for c in chunks] print("✓ test_parse_chunk_ids_match_node_chunk_ids passed") @@ -128,7 +128,7 @@ def test_parse_links_literal_targets(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "see [[topics/Alice.md]] and [[topics/Bob.md#sec]]" path = _write_md(tmp, "note.md", body) - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, _ = await parser.parse(path) triples = {(link.target_path, link.target_anchor, link.predicate) for link in node.links} assert ("topics/Alice.md", None, None) in triples @@ -154,7 +154,7 @@ def test_parse_links_short_and_no_ext_kept_literally(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "see [[Alice]] and [[topics/Alice]] but also [[topics/Alice.md]]" path = _write_md(tmp, "note.md", body) - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, _ = await parser.parse(path) targets = {link.target_path for link in node.links} assert targets == {"Alice", "topics/Alice", "topics/Alice.md"} @@ -170,7 +170,7 @@ def test_parse_links_predicate_inline_and_line(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "extends:: [[A.md]]\n\nsome [concerns:: [[B.md]]] inline\n" path = _write_md(tmp, "note.md", body) - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, _ = await parser.parse(path) pairs = {(link.target_path, link.predicate) for link in node.links} assert ("A.md", "extends") in pairs @@ -187,7 +187,7 @@ def test_parse_links_deduped(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "[[A.md]] again [[A.md]] and [[A.md]]" path = _write_md(tmp, "note.md", body) - parser = LinkedFileParser() + parser = MarkdownFileChunker() node, _ = await parser.parse(path) assert len([link for link in node.links if link.target_path == "A.md"]) == 1 print("✓ test_parse_links_deduped passed") @@ -197,7 +197,7 @@ def test_parse_links_deduped(): def test_parse_min_chunk_chars_clamped(): """chunk_chars below 100 should be clamped to 100.""" - parser = LinkedFileParser(chunk_chars=10) + parser = MarkdownFileChunker(chunk_chars=10) assert parser.chunk_chars == 100 print("✓ test_parse_min_chunk_chars_clamped passed") @@ -209,7 +209,7 @@ def test_parse_embed_toc_prefixes_chunk_text(): with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): body = "# Top\n\n## Sub\n\nbody-content" path = _write_md(tmp, "toc.md", body) - parser = LinkedFileParser(chunk_chars=200, embed_toc=True) + parser = MarkdownFileChunker(chunk_chars=200, embed_toc=True) _, chunks = await parser.parse(path) # Single small section fits; check that the heading appears in text. assert any("Top" in c.text for c in chunks) @@ -219,7 +219,7 @@ def test_parse_embed_toc_prefixes_chunk_text(): if __name__ == "__main__": - print("\n=== LinkedFileParser tests ===") + print("\n=== MarkdownFileChunker tests ===") test_parse_empty_file() test_parse_frontmatter_only() test_parse_small_body_one_chunk()