mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-12 23:01:15 +00:00
refactor(file_chunker): replace file parser with file chunker component (#276)
* refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes
This commit is contained in:
jinliyl
GitHub
parent
a2d76cc034
commit
8eaa96390a
40 changed files with 443 additions and 220 deletions
|
|
@ -15,7 +15,7 @@ ReMe新版本V4
|
|||
- 支持memory-self-evolving
|
||||
- [❌ 待补充] 没有发现自进化相关的 step/job 实现，目前只有基础的 search/reindex 等 common steps（`reme4/steps/common/`）。需要新增 auto-memory/auto-dream 等 step。
|
||||
- 支持markdown之间的链接，构建graph，更好的渐进式展开
|
||||
- [✅ 已实现 → `reme4/components/file_parser/linked_file_parser.py`（wikilink 解析 + Dataview 谓词）、`reme4/components/file_graph/`（local/nx/neo4j 三种 graph 后端）]
|
||||
- [✅ 已实现 → `reme4/components/file_chunker/markdown_file_chunker.py`（wikilink 解析 + Dataview 谓词）、`reme4/components/file_graph/`（local/nx/neo4j 三种 graph 后端）]
|
||||
- [✅ 已实现 → `reme4/steps/common/search.py:109` `_expand_links`、`reme4/config/default.yaml:86` `expand_links` 参数（搜索结果可附 outlinks/inlinks 邻居元数据）]
|
||||
3. 工程实现：
|
||||
1. components
|
||||
|
|
@ -71,21 +71,21 @@ ReMe新版本V4
|
|||
5. Markdown 格式 & Build Graph
|
||||
1. obsidian格式的Markdown文件格式
|
||||
- front matter格式
|
||||
- [✅ 已实现 → `reme4/components/file_parser/linked_file_parser.py:313` `frontmatter.loads(...)`；`reme4/schema/file_front_matter.py`]
|
||||
- [✅ 已实现 → `reme4/components/file_chunker/markdown_file_chunker.py:313` `frontmatter.loads(...)`；`reme4/schema/file_front_matter.py`]
|
||||
- file link格式 4种格式
|
||||
- [⚠️ 部分实现] `linked_file_parser.py:88` `_WIKILINK_RE` 已支持 `[[target]]` / `[[target#anchor]]` / `[[target|alias]]` / `![[target]]`（嵌入），并支持 Dataview `predicate:: [[X]]` 与 inline `[predicate:: [[X]]]`。但 [❌ 待补充] 标准 Markdown `[text](url.md)` 链接尚未被解析为 graph 边。
|
||||
- [⚠️ 部分实现] `markdown_file_chunker.py:88` `_WIKILINK_RE` 已支持 `[[target]]` / `[[target#anchor]]` / `[[target|alias]]` / `![[target]]`（嵌入），并支持 Dataview `predicate:: [[X]]` 与 inline `[predicate:: [[X]]]`。但 [❌ 待补充] 标准 Markdown `[text](url.md)` 链接尚未被解析为 graph 边。
|
||||
2. 更好的文件chunking机制
|
||||
- 旧版 类似rag 带overlap的chunking机制
|
||||
- [📌 历史] V3 旧逻辑，对照说明用，无需在 reme4 中实现。
|
||||
- 解析 Markdown Ast
|
||||
- [✅ 已实现 → `linked_file_parser.py:308` 使用 `mistletoe` 的 `Document`/`MarkdownRenderer`；`:335` `_build_tree` 把扁平 children 折叠成 section 嵌套树（`MdNode`）]
|
||||
- [✅ 已实现 → `markdown_file_chunker.py:308` 使用 `mistletoe` 的 `Document`/`MarkdownRenderer`；`:335` `_build_tree` 把扁平 children 折叠成 section 嵌套树（`MdNode`）]
|
||||
- 每一个chunk都带全部标题
|
||||
- [✅ 已实现 → `linked_file_parser.py:381` `_chunk_node`（`before` 累积已经过的标题、`after` 拼剩余 desc_toc）；`:712` `_make_chunk` 用 `_toc_join(before, content, after)` 把全文目录骨架前后包裹]
|
||||
- [✅ 已实现 → `markdown_file_chunker.py:381` `_chunk_node`（`before` 累积已经过的标题、`after` 拼剩余 desc_toc）；`:712` `_make_chunk` 用 `_toc_join(before, content, after)` 把全文目录骨架前后包裹]
|
||||
3. 通过link构建graph索引，同时构建反向link索引
|
||||
- [✅ 已实现 → `reme4/components/file_graph/base_file_graph.py`、`reme4/components/file_graph/local_file_graph.py`（含 `get_outlinks`、`get_inlinks` 双向索引）；nx/neo4j 后端同 API；`reme4/steps/common/search.py:114-129` 使用双向 link]
|
||||
4. link的生成有两种，一种是主agent在生成link；另一种是通过后台任务，自动构建文档之间的link
|
||||
- 介绍如何auto-link
|
||||
- [⚠️ 部分实现] 主 agent 显式写 `[[link]]` 已经会被 parser 抓为边（`linked_file_parser.py:152` `_extract_links`）。但 [❌ 待补充] "后台任务自动补 link" 的实现（实体抽取 / 候选文档相似度匹配 / link 写回 markdown）尚不存在，需要单独的 step/job。
|
||||
- [⚠️ 部分实现] 主 agent 显式写 `[[link]]` 已经会被 parser 抓为边（`markdown_file_chunker.py:152` `_extract_links`）。但 [❌ 待补充] "后台任务自动补 link" 的实现（实体抽取 / 候选文档相似度匹配 / link 写回 markdown）尚不存在，需要单独的 step/job。
|
||||
6. 如何做memory自进化
|
||||
Auto-memory
|
||||
auto-dream
|
||||
|
|
@ -120,7 +120,7 @@ V4更加高效的底层记忆索引
|
|||
- 不支持关键词检索，这里需要Keyword倒排索引，对中文的支持较差
|
||||
- [📌 历史] 描述 V3 痛点，不需要代码。
|
||||
- V4版本我们重写了file parser，file store，file graph，file watcher，手写了支持增量更新倒排索引
|
||||
- file parser → [✅ `reme4/components/file_parser/`（base/default/chunked/linked 四种）]
|
||||
- file parser → [✅ `reme4/components/file_chunker/`（base/default/chunked/linked 四种）]
|
||||
- file store → [✅ `reme4/components/file_store/local_file_store.py`]
|
||||
- file graph → [✅ `reme4/components/file_graph/`（local/nx/neo4j）]
|
||||
- file watcher → [✅ `reme4/components/file_watcher/lite_file_watcher.py` 基于 watchfiles awatch；`base_file_watcher.py` 抽象接口]
|
||||
|
|
@ -146,7 +146,7 @@ xxxx
|
|||
1. **组件框架**：backend 注册（`component_registry.py`）、生命周期（`base_component.py`）、依赖声明 + 拓扑启动（`application.py:80`）。
|
||||
2. **Job/Step 体系**：`components/job/base_job.py`、`components/job/stream_job.py`、`steps/base_step.py` 与 `steps/common/*`。
|
||||
3. **服务/客户端**：HTTP（`service/http_service.py` + `client/http_client.py`）、MCP（`service/mcp_service.py` + `client/mcp_client.py`），CLI 入口 `reme.py:main`。
|
||||
4. **Markdown 解析**：`file_parser/linked_file_parser.py`，含 frontmatter、wikilink + Dataview 谓词、AST 树、带全标题骨架的 chunking。
|
||||
4. **Markdown 解析**：`file_chunker/markdown_file_chunker.py`，含 frontmatter、wikilink + Dataview 谓词、AST 树、带全标题骨架的 chunking。
|
||||
5. **Graph**：`file_graph/{local,nx,neo4j}_file_graph.py`，双向链接索引。
|
||||
6. **存储 / 索引**：`file_store/local_file_store.py` + `keyword_index/bm25_index.py`（增量 BM25）+ `tokenizer/{regex,jieba}_tokenizer.py`。
|
||||
7. **文件监听**：`file_watcher/lite_file_watcher.py`（watchfiles 轮询）。
|
||||
|
|
|
|||
|
|
@ -81,14 +81,13 @@
|
|||
| `NxFileGraph` (`@R "nx"`) | `nx_file_graph.py` | networkx `MultiDiGraph` + pickle 持久化，虚节点用「无 node 属性」标识。 |
|
||||
| `Neo4jFileGraph` (`@R "neo4j"`) | `neo4j_file_graph.py` | Neo4j 后端（bolt 驱动），`(:File)-[:LINKS]->(:File)`，支持升降级虚节点、`rebuild_links` 修复重建。 |
|
||||
|
||||
### 4.5 File Parser — `reme4/components/file_parser/`
|
||||
### 4.5 File Chunker — `reme4/components/file_chunker/`
|
||||
|
||||
| 类 | 文件 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `BaseFileParser` | `base_file_parser.py` | 抽象接口：`parse(path) -> (FileNode, list[FileChunk])`，提供 `_get_relative_path`。 |
|
||||
| `DefaultFileParser` (`@R "default"`) | `default_file_parser.py` | 仅 stat：附件/二进制不读内容、不切块、不抽链接；作为无 `supported_extensions` 命中时的兜底 parser。 |
|
||||
| `ChunkedFileParser` (`@R "chunked"`) | `chunked_file_parser.py` | 字节级带 overlap 切片 + YAML front matter + wikilink 抽取（含 Dataview `predicate::`）。 |
|
||||
| `LinkedFileParser` (`@R "linked"`) | `linked_file_parser.py` | Markdown 专用：mistletoe AST → MdNode 树 → 章节递归分块；每个 chunk 携带完整 heading skeleton（TOC）；wikilink 解析支持隐式 `.md`、folder-note、短路径歧义扇出，需注入 `file_graph` 解析目标。 |
|
||||
| `BaseFileChunker` | `base_file_chunker.py` | 抽象接口：`parse(path) -> (FileNode, list[FileChunk])`，提供 `_get_relative_path`。 |
|
||||
| `DefaultFileChunker` (`@R "default"`) | `default_file_chunker.py` | 字节级带 overlap 切片 + YAML front matter + wikilink 抽取（含 Dataview `predicate::`）。 |
|
||||
| `MarkdownFileChunker` (`@R "markdown"`) | `markdown_file_chunker.py` | Markdown 专用：mistletoe AST → MdNode 树 → 章节递归分块；每个 chunk 携带完整 heading skeleton（TOC）；wikilink 解析支持隐式 `.md`、folder-note、短路径歧义扇出，需注入 `file_graph` 解析目标。 |
|
||||
|
||||
### 4.6 File Store — `reme4/components/file_store/`
|
||||
|
||||
|
|
@ -104,7 +103,7 @@
|
|||
| 类 | 文件 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `BaseFileWatcher` | `base_file_watcher.py` | 抽象接口：`watch_loop/update_store/on_added/on_modified/on_deleted`；启动后台任务先做一次全量同步再进入监听循环。 |
|
||||
| `LiteFileWatcher` (`@R "lite"`) | `lite_file_watcher.py` | 基于 `watchfiles.awatch` 的轮询监听；变更分类后调用 file_parser 解析、写 file_store；`update_store` 通过 mtime 对比做增量。 |
|
||||
| `LiteFileWatcher` (`@R "lite"`) | `lite_file_watcher.py` | 基于 `watchfiles.awatch` 的轮询监听；变更分类后调用 file_chunker 解析、写 file_store；`update_store` 通过 mtime 对比做增量。 |
|
||||
|
||||
### 4.8 Job — `reme4/components/job/`
|
||||
|
||||
|
|
@ -153,7 +152,7 @@
|
|||
|
||||
| 类 | 注册名 | 文件 | 作用 |
|
||||
| --- | --- | --- | --- |
|
||||
| `BaseStep` | — | `base_step.py` | 抽象基类：`execute()` + `RuntimeContext` 注入 + `input/output_mapping` + 通过 `_resolve` 自动取组件（`as_llm/as_llm_formatter/as_token_counter/file_parser/file_store/embedding/file_watcher`）；`add_as_tool(toolkit, job_name)` 把 job 包成 AgentScope tool。 |
|
||||
| `BaseStep` | — | `base_step.py` | 抽象基类：`execute()` + `RuntimeContext` 注入 + `input/output_mapping` + 通过 `_resolve` 自动取组件（`as_llm/as_llm_formatter/as_token_counter/file_chunker/file_store/embedding/file_watcher`）；`add_as_tool(toolkit, job_name)` 把 job 包成 AgentScope tool。 |
|
||||
| `DemoEchoStep1/2` | `demo_echo_step1` / `demo_echo_step2` | `common/demo.py` | 烟雾测试：query 处理 + 应答。 |
|
||||
| `HealthCheckStep` | `health_check_step` | `common/health_check.py` | 各组件健康/规模快照（embedding/file_graph/file_store/file_watcher/keyword_index）+ 内存深度估算。 |
|
||||
| `HelpStep` | `help_step` | `common/help.py` | 一行式列出全部 job 元信息（含参数 schema）。 |
|
||||
|
|
@ -191,9 +190,9 @@
|
|||
tokenizer (regex) ──┐
|
||||
embedding_model (openai) ──┤── file_store (local) ── file_watcher (lite)
|
||||
file_graph (local) ──┤ (持有 embedding/keyword_index/file_graph)
|
||||
file_parser (default) ──┘ │
|
||||
file_chunker (default) ──┘ │
|
||||
keyword_index (bm25) ── tokenizer ──────────────┘ │
|
||||
file_parser ──┘
|
||||
file_chunker ──┘
|
||||
```
|
||||
|
||||
启动时由 `Application._topological_order()`（Kahn 算法）按依赖拓扑序启动；关闭时反向。
|
||||
|
|
@ -205,7 +204,7 @@ keyword_index (bm25) ── tokenizer ─────────────
|
|||
| 新增检索后端 | 实现 `BaseFileStore` 子类，`@R.register("xxx")` |
|
||||
| 新增图后端 | 实现 `BaseFileGraph` 子类（参考 `Neo4jFileGraph` 处理虚节点） |
|
||||
| 新增分词器 | 实现 `BaseTokenizer.tokenize` |
|
||||
| 新增解析器 | 实现 `BaseFileParser.parse`（返回 `(FileNode, list[FileChunk])`） |
|
||||
| 新增解析器 | 实现 `BaseFileChunker.parse`（返回 `(FileNode, list[FileChunk])`） |
|
||||
| 新增 Job | 在 `default.yaml`（或自定义 yaml）`jobs:` 段声明 + 写步骤实现 |
|
||||
| 新增 Step | 继承 `BaseStep`，实现 `execute()` 并 `@R.register("xxx_step")` |
|
||||
| 暴露新协议 | 实现 `BaseService`（参考 `HttpService` / `MCPService`） |
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@
|
|||
│ search · auto_memory · auto_dream · auto_link … │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ Markdown 知识内核（本地文件即数据库） │
|
||||
│ FileParser · FileStore · FileGraph · FileWatcher │
|
||||
│ FileChunker · FileStore · FileGraph · FileWatcher │
|
||||
│ BM25 倒排 · 向量索引 · Wiki Link 图谱 │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ 文件目录约定 │
|
||||
|
|
@ -377,7 +377,7 @@ ReMe 自研增量 BM25 倒排索引，配合 jieba 中文分词：
|
|||
ReMe 把所有能力封装为 Component：
|
||||
|
||||
```
|
||||
embedding · file_store · file_graph · file_parser · file_watcher
|
||||
embedding · file_store · file_graph · file_chunker · file_watcher
|
||||
tokenizer · keyword_index · LLM 适配 · service · client
|
||||
```
|
||||
|
||||
|
|
@ -909,7 +909,7 @@ type: personalization
|
|||
|
||||
新版本重写了记忆引擎的核心模块：
|
||||
|
||||
- **file parser** —— Markdown AST + 章节切片 + wikilink 抽取
|
||||
- **file chunker** —— Markdown AST + 章节切片 + wikilink 抽取
|
||||
- **file store** —— 内存 chunk 字典 + JSONL 持久化
|
||||
- **file graph** —— 双向链接索引，多 backend
|
||||
- **file watcher** —— 基于 watchfiles 的轻量监听
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ traverse path=xxx direction=xxx depth=xxx
|
|||
| 主题dream + 生成链接 @sen | daily/xxx | knowledge/xxx | /dream | 把 daily 目录的内容按主题聚类合并到 topic 目录, 主动在文档中建立 [[link]] 关联 |
|
||||
| 主动proactive @wangce | daily / topic | proactive_query | pre_query | 思考 daily / topic 信息，主动决定推送给用户的消息 |
|
||||
|
||||
2. file_parser
|
||||
2. file_chunker
|
||||
a. 抽象基类 parse: @jinli
|
||||
ⅰ. 输入是path：相对路径
|
||||
ⅱ. 输出是FileMetadata & list[FileChunks] & list[FileEdge]
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ dependencies = [
|
|||
"prompt_toolkit>=3.0.52",
|
||||
"rich>=14.2.0",
|
||||
"asyncpg>=0.31.0",
|
||||
"chromadb>=1.3.5",
|
||||
"chromadb>=1.5.7",
|
||||
"dashscope>=1.25.1",
|
||||
"elasticsearch>=9.2.0",
|
||||
"fastapi>=0.121.3",
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ class Application(BaseComponent):
|
|||
cfg = self.config
|
||||
vault_path = Path(cfg.vault_dir).absolute()
|
||||
vault_path.mkdir(parents=True, exist_ok=True)
|
||||
for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.dialog_dir, cfg.daily_dir, cfg.digest_dir]:
|
||||
for subdir in [cfg.metadata_dir, cfg.resource_dir, cfg.daily_dir, cfg.digest_dir]:
|
||||
if subdir:
|
||||
(vault_path / subdir).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,13 @@
|
|||
"""Components"""
|
||||
|
||||
from . import llm
|
||||
from . import agent_wrapper
|
||||
from . import as_llm
|
||||
from . import client
|
||||
from . import embedding
|
||||
from . import as_embedding
|
||||
from . import embedding_store
|
||||
from . import file_catalog
|
||||
from . import file_graph
|
||||
from . import file_parser
|
||||
from . import file_chunker
|
||||
from . import file_store
|
||||
from . import job
|
||||
from . import keyword_index
|
||||
|
|
@ -27,13 +28,14 @@ __all__ = [
|
|||
"PromptHandler",
|
||||
"RuntimeContext",
|
||||
# base components
|
||||
"llm",
|
||||
"agent_wrapper",
|
||||
"as_llm",
|
||||
"client",
|
||||
"embedding",
|
||||
"as_embedding",
|
||||
"embedding_store",
|
||||
"file_catalog",
|
||||
"file_graph",
|
||||
"file_parser",
|
||||
"file_chunker",
|
||||
"file_store",
|
||||
"job",
|
||||
"keyword_index",
|
||||
|
|
|
|||
11
reme4/components/agent_wrapper/__init__.py
Normal file
11
reme4/components/agent_wrapper/__init__.py
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
"""Unified agent wrapper component with swappable backends."""
|
||||
|
||||
from .base_agent_wrapper import BaseAgentWrapper
|
||||
from .as_agent_wrapper import AsAgentWrapper
|
||||
from .cc_agent_wrapper import CcAgentWrapper
|
||||
|
||||
__all__ = [
|
||||
"BaseAgentWrapper",
|
||||
"AsAgentWrapper",
|
||||
"CcAgentWrapper",
|
||||
]
|
||||
84
reme4/components/agent_wrapper/as_agent_wrapper.py
Normal file
84
reme4/components/agent_wrapper/as_agent_wrapper.py
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
"""AgentScope backend for the unified agent wrapper."""
|
||||
|
||||
from typing import Any, TYPE_CHECKING
|
||||
|
||||
from agentscope.agent import Agent
|
||||
from agentscope.message import TextBlock, ToolResultState, UserMsg, SystemMsg
|
||||
from agentscope.tool import FunctionTool, ToolChunk, Toolkit
|
||||
|
||||
from .base_agent_wrapper import BaseAgentWrapper
|
||||
from ..as_llm import BaseAsLLM
|
||||
from ..component_registry import R
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..job.base_job import BaseJob
|
||||
|
||||
|
||||
@R.register("agentscope")
|
||||
class AsAgentWrapper(BaseAgentWrapper):
|
||||
"""Agent wrapper backed by AgentScope framework.
|
||||
|
||||
Args:
|
||||
as_llm: Name of the bound as_llm component (resolved via app_context).
|
||||
Kwargs:
|
||||
system_prompt: System prompt for the agent.
|
||||
tools: list[BaseJob] to register as agent wrapper tools.
|
||||
"""
|
||||
|
||||
def __init__(self, as_llm: str = "default", **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.as_llm = self.bind(as_llm, BaseAsLLM, optional=False)
|
||||
|
||||
@staticmethod
|
||||
def _make_tool(job: "BaseJob") -> FunctionTool:
|
||||
async def run_job(**kwargs) -> ToolChunk:
|
||||
response = await job(**kwargs)
|
||||
return ToolChunk(
|
||||
content=[TextBlock(text=str(response.answer))],
|
||||
state=ToolResultState.SUCCESS if response.success else ToolResultState.ERROR,
|
||||
)
|
||||
|
||||
tool = FunctionTool(func=run_job, name=job.name, description=job.description)
|
||||
if job.parameters:
|
||||
tool.input_schema = job.parameters
|
||||
return tool
|
||||
|
||||
async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]:
|
||||
model = self.as_llm.model if self.as_llm else None
|
||||
if model is None:
|
||||
raise ValueError("AsAgentWrapper requires a bound as_llm component with a valid model.")
|
||||
|
||||
for k, v in self.kwargs.items():
|
||||
kwargs.setdefault(k, v)
|
||||
|
||||
output_schema: dict | None = kwargs.get("output_schema")
|
||||
|
||||
tools: list["BaseJob"] = kwargs.get("tools", [])
|
||||
toolkit = Toolkit(tools=[self._make_tool(job) for job in tools]) if tools else Toolkit()
|
||||
|
||||
system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.")
|
||||
|
||||
agent = Agent(
|
||||
name=self.name,
|
||||
system_prompt=system_prompt,
|
||||
model=model,
|
||||
toolkit=toolkit,
|
||||
)
|
||||
|
||||
if isinstance(inputs, str):
|
||||
inputs = UserMsg(name="user", content=inputs)
|
||||
|
||||
if output_schema:
|
||||
messages = [
|
||||
SystemMsg(name="system", content=system_prompt),
|
||||
inputs,
|
||||
]
|
||||
res = await model.generate_structured_output(
|
||||
messages=messages,
|
||||
structured_model=output_schema,
|
||||
)
|
||||
return agent.state.session_id, res.content
|
||||
|
||||
await agent.observe(inputs)
|
||||
await agent.reply()
|
||||
return agent.state.session_id, agent.state.context[-1]
|
||||
39
reme4/components/agent_wrapper/base_agent_wrapper.py
Normal file
39
reme4/components/agent_wrapper/base_agent_wrapper.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
"""Base agent wrapper component."""
|
||||
|
||||
from abc import abstractmethod
|
||||
from typing import Any, TYPE_CHECKING
|
||||
|
||||
from ..base_component import BaseComponent
|
||||
from ...enumeration import ComponentEnum
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..job.base_job import BaseJob
|
||||
|
||||
|
||||
class BaseAgentWrapper(BaseComponent):
|
||||
"""Abstract base for agent wrapper components with swappable backends.
|
||||
|
||||
Subclasses implement reply() which returns (session_id, last_message).
|
||||
Supports fluent configuration via set_system_prompt() and add_tools().
|
||||
"""
|
||||
|
||||
component_type = ComponentEnum.AGENT_WRAPPER
|
||||
|
||||
def set_system_prompt(self, prompt: str) -> "BaseAgentWrapper":
|
||||
"""Set the agent's system prompt. Returns self for chaining."""
|
||||
self.kwargs["system_prompt"] = prompt
|
||||
return self
|
||||
|
||||
def add_tools(self, tools: list["BaseJob"]) -> "BaseAgentWrapper":
|
||||
"""Append callable tools to the agent. Returns self for chaining."""
|
||||
self.kwargs.setdefault("tools", []).extend(tools)
|
||||
return self
|
||||
|
||||
def set_output_schema(self, schema: dict) -> "BaseAgentWrapper":
|
||||
"""Set a JSON schema for structured output. Returns self for chaining."""
|
||||
self.kwargs["output_schema"] = schema
|
||||
return self
|
||||
|
||||
@abstractmethod
|
||||
async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]:
|
||||
"""Send inputs to the agent and return (session_id, last_message)."""
|
||||
86
reme4/components/agent_wrapper/cc_agent_wrapper.py
Normal file
86
reme4/components/agent_wrapper/cc_agent_wrapper.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
"""Claude Code SDK backend for the unified agent wrapper."""
|
||||
|
||||
from typing import Any, TYPE_CHECKING
|
||||
|
||||
from .base_agent_wrapper import BaseAgentWrapper
|
||||
from ..component_registry import R
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..job.base_job import BaseJob
|
||||
|
||||
|
||||
@R.register("claude_code")
|
||||
class CcAgentWrapper(BaseAgentWrapper):
|
||||
"""Agent wrapper backed by Claude Code SDK.
|
||||
|
||||
Kwargs:
|
||||
system_prompt: System prompt for the agent.
|
||||
model: Claude model to use.
|
||||
permission_mode: Permission mode for tool execution.
|
||||
max_turns: Maximum conversation turns.
|
||||
tools: list[BaseJob] to register as agent wrapper tools.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _make_tool(job: "BaseJob"):
|
||||
from claude_agent_sdk import SdkMcpTool
|
||||
|
||||
async def run_job(args):
|
||||
response = await job(**args)
|
||||
return {
|
||||
"content": [{"type": "text", "text": str(response.answer)}],
|
||||
"is_error": not response.success,
|
||||
}
|
||||
|
||||
return SdkMcpTool(
|
||||
name=job.name,
|
||||
description=job.description,
|
||||
input_schema=job.parameters,
|
||||
handler=run_job,
|
||||
)
|
||||
|
||||
async def reply(self, inputs: Any, session_id: str | None = None, **kwargs) -> tuple[str, Any]:
|
||||
from claude_agent_sdk import query, ResultMessage, create_sdk_mcp_server
|
||||
from claude_agent_sdk.types import ClaudeAgentOptions
|
||||
|
||||
for k, v in self.kwargs.items():
|
||||
kwargs.setdefault(k, v)
|
||||
|
||||
opts = ClaudeAgentOptions()
|
||||
opts.system_prompt = kwargs.get("system_prompt", "You are a helpful assistant.")
|
||||
opts.model = kwargs["model"]
|
||||
opts.permission_mode = kwargs.get("permission_mode", None)
|
||||
opts.max_turns = kwargs.get("max_turns", 50)
|
||||
if session_id:
|
||||
opts.session_id = session_id
|
||||
opts.fork_session = True
|
||||
|
||||
tools: list["BaseJob"] = kwargs.get("tools", [])
|
||||
if tools:
|
||||
sdk_tools = [self._make_tool(job) for job in tools]
|
||||
server = create_sdk_mcp_server(name="reme_tools", tools=sdk_tools)
|
||||
if isinstance(opts.mcp_servers, dict):
|
||||
opts.mcp_servers["reme"] = server
|
||||
else:
|
||||
opts.mcp_servers = {"reme": server}
|
||||
opts.allowed_tools.extend(job.name for job in tools)
|
||||
|
||||
output_schema = kwargs.get("output_schema")
|
||||
if output_schema:
|
||||
opts.output_format = {"type": "json_schema", "schema": output_schema}
|
||||
|
||||
if isinstance(inputs, str):
|
||||
prompt = inputs
|
||||
else:
|
||||
raise NotImplementedError("Only string input is supported for Claude Code.")
|
||||
|
||||
last_msg = None
|
||||
async for msg in query(prompt=prompt, options=opts):
|
||||
if isinstance(msg, ResultMessage):
|
||||
last_msg = msg
|
||||
|
||||
if last_msg is None:
|
||||
raise ValueError("No message received from Claude Code.")
|
||||
|
||||
result = last_msg.structured_output if output_schema and last_msg.structured_output else last_msg
|
||||
return last_msg.session_id or "", result
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
"""AgentScope embedding model wrappers."""
|
||||
|
||||
from agentscope.embedding import (
|
||||
DashScopeMultiModalEmbedding as _AsDashScopeMultiModalEmbedding,
|
||||
DashScopeMultiModalEmbedding,
|
||||
DashScopeTextEmbedding,
|
||||
EmbeddingModelBase,
|
||||
GeminiTextEmbedding,
|
||||
|
|
@ -14,10 +14,10 @@ from ..component_registry import R
|
|||
from ...enumeration import ComponentEnum
|
||||
|
||||
|
||||
class BaseEmbedding(BaseComponent):
|
||||
class BaseAsEmbedding(BaseComponent):
|
||||
"""Base wrapper for AgentScope embedding models. Builds ``self.model`` in ``_start``."""
|
||||
|
||||
component_type = ComponentEnum.EMBEDDING
|
||||
component_type = ComponentEnum.AS_EMBEDDING
|
||||
|
||||
def __init__(self, **kwargs) -> None:
|
||||
super().__init__(**kwargs)
|
||||
|
|
@ -39,7 +39,7 @@ class BaseEmbedding(BaseComponent):
|
|||
|
||||
|
||||
@R.register("openai")
|
||||
class OpenAIEmbedding(BaseEmbedding):
|
||||
class OpenAIAsEmbedding(BaseAsEmbedding):
|
||||
"""OpenAI embedding model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
|
|
@ -52,7 +52,7 @@ class OpenAIEmbedding(BaseEmbedding):
|
|||
|
||||
|
||||
@R.register("dashscope")
|
||||
class DashScopeEmbedding(BaseEmbedding):
|
||||
class DashScopeAsEmbedding(BaseAsEmbedding):
|
||||
"""DashScope text embedding model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
|
|
@ -60,15 +60,15 @@ class DashScopeEmbedding(BaseEmbedding):
|
|||
|
||||
|
||||
@R.register("dashscope_multimodal")
|
||||
class DashScopeMultiModalEmbedding(BaseEmbedding):
|
||||
class DashScopeMultiModalAsEmbedding(BaseAsEmbedding):
|
||||
"""DashScope multimodal embedding model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
self.model = _AsDashScopeMultiModalEmbedding(**self.kwargs)
|
||||
self.model = DashScopeMultiModalEmbedding(**self.kwargs)
|
||||
|
||||
|
||||
@R.register("gemini")
|
||||
class GeminiEmbedding(BaseEmbedding):
|
||||
class GeminiAsEmbedding(BaseAsEmbedding):
|
||||
"""Gemini embedding model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
|
|
@ -76,7 +76,7 @@ class GeminiEmbedding(BaseEmbedding):
|
|||
|
||||
|
||||
@R.register("ollama")
|
||||
class OllamaEmbedding(BaseEmbedding):
|
||||
class OllamaAsEmbedding(BaseAsEmbedding):
|
||||
"""Ollama embedding model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
|
|
@ -84,10 +84,10 @@ class OllamaEmbedding(BaseEmbedding):
|
|||
|
||||
|
||||
__all__ = [
|
||||
"BaseEmbedding",
|
||||
"OpenAIEmbedding",
|
||||
"DashScopeEmbedding",
|
||||
"DashScopeMultiModalEmbedding",
|
||||
"GeminiEmbedding",
|
||||
"OllamaEmbedding",
|
||||
"BaseAsEmbedding",
|
||||
"OpenAIAsEmbedding",
|
||||
"DashScopeAsEmbedding",
|
||||
"DashScopeMultiModalAsEmbedding",
|
||||
"GeminiAsEmbedding",
|
||||
"OllamaAsEmbedding",
|
||||
]
|
||||
|
|
@ -18,13 +18,13 @@ from ..component_registry import R
|
|||
from ...enumeration import ComponentEnum
|
||||
|
||||
|
||||
class BaseLLM(BaseComponent):
|
||||
class BaseAsLLM(BaseComponent):
|
||||
"""Base wrapper for AgentScope chat models.
|
||||
|
||||
Subclasses set ``credential_cls`` and inherit ``_start`` / ``_close``.
|
||||
"""
|
||||
|
||||
component_type = ComponentEnum.LLM
|
||||
component_type = ComponentEnum.AS_LLM
|
||||
credential_cls: type[CredentialBase]
|
||||
|
||||
def __init__(self, **kwargs) -> None:
|
||||
|
|
@ -44,69 +44,69 @@ class BaseLLM(BaseComponent):
|
|||
|
||||
|
||||
@R.register("openai")
|
||||
class OpenAILLM(BaseLLM):
|
||||
class OpenAIAsLLM(BaseAsLLM):
|
||||
"""OpenAI chat model wrapper."""
|
||||
|
||||
credential_cls = OpenAICredential
|
||||
|
||||
|
||||
@R.register("anthropic")
|
||||
class AnthropicLLM(BaseLLM):
|
||||
class AnthropicAsLLM(BaseAsLLM):
|
||||
"""Anthropic chat model wrapper."""
|
||||
|
||||
credential_cls = AnthropicCredential
|
||||
|
||||
|
||||
@R.register("dashscope")
|
||||
class DashScopeLLM(BaseLLM):
|
||||
class DashScopeAsLLM(BaseAsLLM):
|
||||
"""DashScope chat model wrapper."""
|
||||
|
||||
credential_cls = DashScopeCredential
|
||||
|
||||
|
||||
@R.register("deepseek")
|
||||
class DeepSeekLLM(BaseLLM):
|
||||
class DeepSeekAsLLM(BaseAsLLM):
|
||||
"""DeepSeek chat model wrapper."""
|
||||
|
||||
credential_cls = DeepSeekCredential
|
||||
|
||||
|
||||
@R.register("gemini")
|
||||
class GeminiLLM(BaseLLM):
|
||||
class GeminiAsLLM(BaseAsLLM):
|
||||
"""Gemini chat model wrapper."""
|
||||
|
||||
credential_cls = GeminiCredential
|
||||
|
||||
|
||||
@R.register("moonshot")
|
||||
class MoonshotLLM(BaseLLM):
|
||||
class MoonshotAsLLM(BaseAsLLM):
|
||||
"""Moonshot chat model wrapper."""
|
||||
|
||||
credential_cls = MoonshotCredential
|
||||
|
||||
|
||||
@R.register("ollama")
|
||||
class OllamaLLM(BaseLLM):
|
||||
class OllamaAsLLM(BaseAsLLM):
|
||||
"""Ollama chat model wrapper."""
|
||||
|
||||
credential_cls = OllamaCredential
|
||||
|
||||
|
||||
@R.register("xai")
|
||||
class XAILLM(BaseLLM):
|
||||
class XAIAsLLM(BaseAsLLM):
|
||||
"""xAI chat model wrapper."""
|
||||
|
||||
credential_cls = XAICredential
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BaseLLM",
|
||||
"OpenAILLM",
|
||||
"AnthropicLLM",
|
||||
"DashScopeLLM",
|
||||
"DeepSeekLLM",
|
||||
"GeminiLLM",
|
||||
"MoonshotLLM",
|
||||
"OllamaLLM",
|
||||
"XAILLM",
|
||||
"BaseAsLLM",
|
||||
"OpenAIAsLLM",
|
||||
"AnthropicAsLLM",
|
||||
"DashScopeAsLLM",
|
||||
"DeepSeekAsLLM",
|
||||
"GeminiAsLLM",
|
||||
"MoonshotAsLLM",
|
||||
"OllamaAsLLM",
|
||||
"XAIAsLLM",
|
||||
]
|
||||
|
|
@ -9,7 +9,7 @@ import numpy as np
|
|||
|
||||
from .base_embedding_store import BaseEmbeddingStore
|
||||
from ..component_registry import R
|
||||
from ..embedding import BaseEmbedding
|
||||
from ..as_embedding import BaseAsEmbedding
|
||||
|
||||
Miss = tuple[int, str, str] # (result_index, text, cache_key)
|
||||
|
||||
|
|
@ -18,19 +18,19 @@ Miss = tuple[int, str, str] # (result_index, text, cache_key)
|
|||
class LocalEmbeddingStore(BaseEmbeddingStore):
|
||||
"""Embedding store with LRU cache, disk persistence, and serial batching.
|
||||
|
||||
Delegates actual embedding computation to a bound ``embedding`` component.
|
||||
Delegates actual embedding computation to a bound ``as_embedding`` component.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
embedding: str = "default",
|
||||
as_embedding: str = "default",
|
||||
max_cache_size: int = 10000,
|
||||
enable_cache: bool = True,
|
||||
cache_version: str = "v1",
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
self.embedding = self.bind(embedding, BaseEmbedding, optional=False)
|
||||
self.as_embedding = self.bind(as_embedding, BaseAsEmbedding, optional=False)
|
||||
self.max_cache_size = max_cache_size
|
||||
self.enable_cache = enable_cache
|
||||
self.cache_version = cache_version
|
||||
|
|
@ -40,8 +40,8 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
|
|||
@property
|
||||
def dimensions(self) -> int:
|
||||
"""Return the embedding dimension size."""
|
||||
assert self.embedding is not None, "embedding component not bound"
|
||||
return self.embedding.dimensions
|
||||
assert self.as_embedding is not None, "embedding component not bound"
|
||||
return self.as_embedding.dimensions
|
||||
|
||||
@property
|
||||
def cache_path(self) -> Path:
|
||||
|
|
@ -58,7 +58,7 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
|
|||
async def health_check(self, timeout: float = 2.0) -> bool:
|
||||
tag = f"[EMBEDDING HEALTH CHECK] name={self.name}"
|
||||
try:
|
||||
result = await asyncio.wait_for(self.embedding(["ping"]), timeout=timeout)
|
||||
result = await asyncio.wait_for(self.as_embedding(["ping"]), timeout=timeout)
|
||||
if not result or result[0] is None:
|
||||
raise RuntimeError("empty embedding")
|
||||
self.is_healthy = True
|
||||
|
|
@ -121,7 +121,7 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
|
|||
async def _call_with_retry(self, texts: list[str], **kwargs) -> list[list[float] | None] | None:
|
||||
for attempt in range(self.max_retries):
|
||||
try:
|
||||
result = await self.embedding(texts, **kwargs)
|
||||
result = await self.as_embedding(texts, **kwargs)
|
||||
if result and len(result) == len(texts):
|
||||
return result
|
||||
except (TimeoutError, ConnectionError, OSError):
|
||||
|
|
|
|||
7
reme4/components/file_chunker/__init__.py
Normal file
7
reme4/components/file_chunker/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
"""File chunker components."""
|
||||
|
||||
from .base_file_chunker import BaseFileChunker
|
||||
from .default_file_chunker import DefaultFileChunker
|
||||
from .markdown_file_chunker import MarkdownFileChunker
|
||||
|
||||
__all__ = ["BaseFileChunker", "DefaultFileChunker", "MarkdownFileChunker"]
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""Abstract base for file parsers."""
|
||||
"""Abstract base for file chunkers."""
|
||||
|
||||
from abc import abstractmethod
|
||||
from pathlib import Path
|
||||
|
|
@ -8,10 +8,10 @@ from ...enumeration import ComponentEnum
|
|||
from ...schema import FileChunk, FileNode
|
||||
|
||||
|
||||
class BaseFileParser(BaseComponent):
|
||||
"""Abstract base for file parsers. Subclasses implement `parse`."""
|
||||
class BaseFileChunker(BaseComponent):
|
||||
"""Abstract base for file chunkers. Subclasses implement `parse`."""
|
||||
|
||||
component_type = ComponentEnum.FILE_PARSER
|
||||
component_type = ComponentEnum.FILE_CHUNKER
|
||||
|
||||
def __init__(self, supported_extensions: list[str] | None = None, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""File parser with byte-based overlapping chunking."""
|
||||
"""Default file chunker — byte-based overlapping chunking."""
|
||||
|
||||
from bisect import bisect_right
|
||||
from pathlib import Path
|
||||
|
|
@ -6,15 +6,15 @@ from pathlib import Path
|
|||
import aiofiles
|
||||
import yaml
|
||||
|
||||
from .base_file_parser import BaseFileParser
|
||||
from .base_file_chunker import BaseFileChunker
|
||||
from ..component_registry import R
|
||||
from ...schema import FileChunk, FileFrontMatter, FileNode
|
||||
from ...utils.wikilink_handler import WikilinkHandler
|
||||
|
||||
|
||||
@R.register("chunked")
|
||||
class ChunkedFileParser(BaseFileParser):
|
||||
"""Parser that splits files into byte-based overlapping chunks."""
|
||||
@R.register("default")
|
||||
class DefaultFileChunker(BaseFileChunker):
|
||||
"""Default chunker that splits files into byte-based overlapping chunks."""
|
||||
|
||||
def __init__(self, encoding: str = "utf-8", chunk_byte_size: int = 10000, overlap_byte_size: int = 100, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""Markdown file parser — frontmatter + wikilink graph + AST tree chunks.
|
||||
"""Markdown file chunker — frontmatter + wikilink graph + AST tree chunks.
|
||||
|
||||
Each chunk carries the **complete heading skeleton** of the document
|
||||
with its content inlined under the section that owns it; other sections
|
||||
|
|
@ -21,7 +21,7 @@ from typing import Any
|
|||
import frontmatter
|
||||
|
||||
|
||||
from .base_file_parser import BaseFileParser
|
||||
from .base_file_chunker import BaseFileChunker
|
||||
from ..component_registry import R
|
||||
from ...schema import (
|
||||
FileChunk,
|
||||
|
|
@ -103,12 +103,12 @@ def _subtree_toc(n: MdNode) -> str:
|
|||
return f"{heading}\n\n{n.desc_toc}" if n.desc_toc else heading
|
||||
|
||||
|
||||
# -- Parser ---------------------------------------------------------------
|
||||
# -- Chunker --------------------------------------------------------------
|
||||
|
||||
|
||||
@R.register("linked")
|
||||
class LinkedFileParser(BaseFileParser):
|
||||
"""Markdown parser: frontmatter + wikilink edges + full-skeleton chunks."""
|
||||
@R.register("markdown")
|
||||
class MarkdownFileChunker(BaseFileChunker):
|
||||
"""Markdown chunker: frontmatter + wikilink edges + full-skeleton chunks."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
|
@ -33,6 +33,7 @@ import error fires at ``_start`` (boot), not at first call.
|
|||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from .base_file_graph import BaseFileGraph
|
||||
|
|
@ -64,7 +65,7 @@ class Neo4jFileGraph(BaseFileGraph):
|
|||
Connection params (constructor kwargs):
|
||||
uri: bolt URL, e.g. ``bolt://localhost:7687``
|
||||
user: auth user (default ``neo4j``)
|
||||
password: auth password
|
||||
password: auth password (required; falls back to ``NEO4J_PASSWORD`` env var)
|
||||
database: target db name (default ``neo4j``)
|
||||
"""
|
||||
|
||||
|
|
@ -72,14 +73,19 @@ class Neo4jFileGraph(BaseFileGraph):
|
|||
self,
|
||||
uri: str = "bolt://localhost:7687",
|
||||
user: str = "neo4j",
|
||||
password: str = "neo4j",
|
||||
password: str | None = None,
|
||||
database: str = "neo4j",
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
self._uri: str = uri
|
||||
self._user: str = user
|
||||
self._password: str = password
|
||||
self._password: str = password or os.environ.get("NEO4J_PASSWORD") or ""
|
||||
if not self._password:
|
||||
raise ValueError(
|
||||
"Neo4j password must be provided via the 'password' argument "
|
||||
"or the NEO4J_PASSWORD environment variable.",
|
||||
)
|
||||
self._database: str = database
|
||||
self._driver = None
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +0,0 @@
|
|||
"""File parser components."""
|
||||
|
||||
from .base_file_parser import BaseFileParser
|
||||
from .chunked_file_parser import ChunkedFileParser
|
||||
from .default_file_parser import DefaultFileParser
|
||||
from .linked_file_parser import LinkedFileParser
|
||||
|
||||
__all__ = ["BaseFileParser", "ChunkedFileParser", "DefaultFileParser", "LinkedFileParser"]
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
"""Stat-only parser for attachment/binary files."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from .base_file_parser import BaseFileParser
|
||||
from ..component_registry import R
|
||||
from ...schema import FileChunk, FileNode
|
||||
|
||||
|
||||
@R.register("default")
|
||||
class DefaultFileParser(BaseFileParser):
|
||||
"""Stat-only parser for attachment/binary files.
|
||||
|
||||
No content read, no chunking, no link extraction. The resulting FileNode
|
||||
has empty links and chunk_ids; front_matter carries mime and size so
|
||||
retrieval can filter by file type without reopening the file.
|
||||
"""
|
||||
|
||||
async def parse(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]:
|
||||
file_path = Path(path)
|
||||
stat = file_path.stat()
|
||||
return FileNode(path=self.to_vault_relative(path), st_mtime=stat.st_mtime, links=[], chunk_ids=[]), []
|
||||
|
|
@ -46,10 +46,11 @@ class HttpService(BaseService):
|
|||
title=app.config.app_name,
|
||||
lifespan=self._lifespan(app, self.host, self.port),
|
||||
)
|
||||
cors_origins = ["*"]
|
||||
self.service.add_middleware(
|
||||
CORSMiddleware, # type: ignore[arg-type]
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_origins=cors_origins,
|
||||
allow_credentials="*" not in cors_origins,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ jobs:
|
|||
suffix_filters: [ "md" ]
|
||||
steps:
|
||||
- backend: scan_store_changes_step
|
||||
recursive: true
|
||||
- backend: update_index_step
|
||||
persist: true
|
||||
- backend: watch_changes_step
|
||||
|
|
@ -456,7 +457,7 @@ components:
|
|||
default:
|
||||
backend: regex
|
||||
|
||||
embedding:
|
||||
as_embedding:
|
||||
default:
|
||||
backend: ${EMBEDDING_BACKEND:-openai}
|
||||
api_key: ${EMBEDDING_API_KEY:-}
|
||||
|
|
@ -467,9 +468,9 @@ components:
|
|||
embedding_store:
|
||||
default:
|
||||
backend: local
|
||||
embedding: default
|
||||
as_embedding: default
|
||||
|
||||
llm:
|
||||
as_llm:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-anthropic}
|
||||
model: ${LLM_MODEL_NAME:-glm-5.1}
|
||||
|
|
@ -483,6 +484,16 @@ components:
|
|||
max_tokens: 65536
|
||||
thinking_enable: true
|
||||
|
||||
agent_wrapper:
|
||||
default:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
claude_code:
|
||||
backend: claude_code
|
||||
model: ${LLM_MODEL_NAME:-claude-opus-4-6}
|
||||
permission_mode: bypassPermissions
|
||||
max_turns: 10
|
||||
|
||||
file_graph:
|
||||
default:
|
||||
backend: local
|
||||
|
|
@ -491,15 +502,13 @@ components:
|
|||
default:
|
||||
backend: local
|
||||
|
||||
file_parser:
|
||||
linked:
|
||||
backend: linked
|
||||
file_chunker:
|
||||
markdown:
|
||||
backend: markdown
|
||||
supported_extensions: [ "md" ]
|
||||
chunked:
|
||||
backend: chunked
|
||||
supported_extensions: [ "txt", "html", "json", "jsonl", "yaml", "py" ]
|
||||
default:
|
||||
backend: default
|
||||
supported_extensions: [ "txt", "html", "json", "jsonl", "yaml", "py" ]
|
||||
|
||||
keyword_index:
|
||||
default:
|
||||
|
|
|
|||
|
|
@ -8,13 +8,13 @@ class ComponentEnum(str, Enum):
|
|||
|
||||
BASE = "base"
|
||||
|
||||
LLM = "llm"
|
||||
AS_LLM = "as_llm"
|
||||
|
||||
EMBEDDING = "embedding"
|
||||
AS_EMBEDDING = "as_embedding"
|
||||
|
||||
EMBEDDING_STORE = "embedding_store"
|
||||
|
||||
FILE_PARSER = "file_parser"
|
||||
FILE_CHUNKER = "file_chunker"
|
||||
|
||||
FILE_STORE = "file_store"
|
||||
|
||||
|
|
@ -33,3 +33,5 @@ class ComponentEnum(str, Enum):
|
|||
JOB = "job"
|
||||
|
||||
TOKENIZER = "tokenizer"
|
||||
|
||||
AGENT_WRAPPER = "agent_wrapper"
|
||||
|
|
|
|||
|
|
@ -31,10 +31,10 @@ class ApplicationConfig(BaseModel):
|
|||
vault_dir: str = Field(default=".reme", description="Vault root directory for runtime files")
|
||||
metadata_dir: str = Field(default="reme_metadata", description="Subdirectory for ReMe persistent state")
|
||||
resource_dir: str = Field(default="resource", description="Subdirectory for external assets")
|
||||
dialog_dir: str = Field(default="dialog", description="Subdirectory for dialog memory")
|
||||
daily_dir: str = Field(default="daily", description="Subdirectory for daily memory")
|
||||
digest_dir: str = Field(default="digest", description="Subdirectory for digest memory")
|
||||
enable_logo: bool = Field(default=True, description="Show ASCII logo on startup")
|
||||
timezone: str | None = Field(default=None, description="IANA timezone (e.g. 'Asia/Shanghai'); None uses local time")
|
||||
language: str = Field(default="", description="Default language for LLM interactions")
|
||||
log_to_console: bool = Field(default=True, description="Log to console")
|
||||
log_to_file: bool = Field(default=True, description="Log to file")
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from agentscope.model import ChatModelBase
|
|||
from agentscope.tool import Toolkit, FunctionTool, ToolChunk
|
||||
|
||||
from ..components.base_component import ComponentMixin
|
||||
from ..components.file_parser import BaseFileParser
|
||||
from ..components.file_chunker import BaseFileChunker
|
||||
from ..components.file_store import BaseFileStore
|
||||
from ..components.prompt_handler import PromptHandler
|
||||
from ..components.runtime_context import RuntimeContext
|
||||
|
|
@ -32,7 +32,7 @@ class Ref:
|
|||
Replaces the ``@property`` + ``_resolve()`` boilerplate with a single
|
||||
class-level declaration::
|
||||
|
||||
llm = Ref(ChatModelBase, ComponentEnum.LLM, "model")
|
||||
as_llm = Ref(ChatModelBase, ComponentEnum.AS_LLM, "model")
|
||||
file_store = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
|
||||
|
||||
Resolution follows a 3-source fallback identical to the old ``_resolve``:
|
||||
|
|
@ -102,7 +102,7 @@ class BaseStep(ComponentMixin, ABC):
|
|||
|
||||
component_type = ComponentEnum.STEP
|
||||
|
||||
llm: ChatModelBase = Ref(ChatModelBase, ComponentEnum.LLM, "model")
|
||||
as_llm: ChatModelBase = Ref(ChatModelBase, ComponentEnum.AS_LLM, "model")
|
||||
file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
|
||||
|
||||
def __new__(cls, *args, **kwargs):
|
||||
|
|
@ -165,24 +165,24 @@ class BaseStep(ComponentMixin, ABC):
|
|||
how attachments / binaries / unknown types still produce a FileNode.
|
||||
"""
|
||||
if self.app_context is None:
|
||||
raise RuntimeError("app_context is not set when resolving file parser")
|
||||
file_parser_dict: dict[str, BaseFileParser] = self.app_context.components[ComponentEnum.FILE_PARSER]
|
||||
raise RuntimeError("app_context is not set when resolving file chunker")
|
||||
file_chunker_dict: dict[str, BaseFileChunker] = self.app_context.components[ComponentEnum.FILE_CHUNKER]
|
||||
|
||||
suffix = Path(path).suffix.lstrip(".").lower()
|
||||
|
||||
parser: BaseFileParser | None = None
|
||||
parser: BaseFileChunker | None = None
|
||||
if suffix:
|
||||
for candidate in file_parser_dict.values():
|
||||
for candidate in file_chunker_dict.values():
|
||||
if suffix in {ext.lower().lstrip(".") for ext in candidate.supported_extensions}:
|
||||
parser = candidate
|
||||
break
|
||||
|
||||
if parser is None:
|
||||
parser = file_parser_dict.get("default")
|
||||
parser = file_chunker_dict.get("default")
|
||||
|
||||
if parser is None:
|
||||
raise RuntimeError(
|
||||
f"No file parser supports {path} (suffix={suffix!r}) and no 'default' parser is configured",
|
||||
f"No file chunker supports {path} (suffix={suffix!r}) and no 'default' chunker is configured",
|
||||
)
|
||||
|
||||
return await parser.parse(path)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Demo step that drives an Agent via BaseStep.llm."""
|
||||
"""Demo step that drives an Agent via BaseStep.as_llm."""
|
||||
|
||||
from typing import Type
|
||||
|
||||
|
|
@ -25,7 +25,7 @@ def add(a: float, b: float) -> str:
|
|||
|
||||
@R.register("llm_demo_step")
|
||||
class LLMDemoStep(BaseStep):
|
||||
"""Drive an Agent powered by ``self.llm``.
|
||||
"""Drive an Agent powered by ``self.as_llm``.
|
||||
|
||||
Inputs (from RuntimeContext):
|
||||
query (str, required): user message content.
|
||||
|
|
@ -55,7 +55,7 @@ class LLMDemoStep(BaseStep):
|
|||
agent = Agent(
|
||||
name=self.name,
|
||||
system_prompt=sys_prompt,
|
||||
model=self.llm,
|
||||
model=self.as_llm,
|
||||
toolkit=toolkit,
|
||||
state=AgentState(
|
||||
permission_context=PermissionContext(
|
||||
|
|
@ -72,7 +72,7 @@ class LLMDemoStep(BaseStep):
|
|||
|
||||
structured_content: dict | None = None
|
||||
if structured_model is not None:
|
||||
structured_resp = await self.llm.generate_structured_output(
|
||||
structured_resp = await self.as_llm.generate_structured_output(
|
||||
agent.state.context,
|
||||
structured_model=structured_model,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Demo step that drives an Agent via BaseStep.llm with streaming output."""
|
||||
"""Demo step that drives an Agent via BaseStep.as_llm with streaming output."""
|
||||
|
||||
import json
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ def add(a: float, b: float) -> str:
|
|||
|
||||
@R.register("stream_llm_demo_step")
|
||||
class StreamLLMDemoStep(BaseStep):
|
||||
"""Drive an Agent powered by ``self.llm`` with streaming output.
|
||||
"""Drive an Agent powered by ``self.as_llm`` with streaming output.
|
||||
|
||||
When streaming is enabled on the context, text/thinking/tool events are
|
||||
pushed chunk-by-chunk via ``self.context.add_stream_string``.
|
||||
|
|
@ -67,7 +67,7 @@ class StreamLLMDemoStep(BaseStep):
|
|||
agent = Agent(
|
||||
name=self.name,
|
||||
system_prompt=sys_prompt,
|
||||
model=self.llm,
|
||||
model=self.as_llm,
|
||||
toolkit=toolkit,
|
||||
state=AgentState(
|
||||
permission_context=PermissionContext(
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
"""Evolve steps."""
|
||||
|
||||
from ._evolve import now
|
||||
|
||||
__all__ = ["now"]
|
||||
|
|
@ -108,7 +108,7 @@ class AutoMemoryStep(BaseStep):
|
|||
|
||||
agent = Agent(
|
||||
name="auto_memory",
|
||||
model=self.llm,
|
||||
model=self.as_llm,
|
||||
system_prompt=self.prompt_format("system_prompt"),
|
||||
toolkit=toolkit,
|
||||
state=AgentState(
|
||||
|
|
|
|||
|
|
@ -269,7 +269,7 @@ class DreamStep(BaseStep):
|
|||
|
||||
def _llm_available(self) -> bool:
|
||||
try:
|
||||
return self.llm is not None
|
||||
return self.as_llm is not None
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
|
@ -303,7 +303,7 @@ class DreamStep(BaseStep):
|
|||
toolkit = self._build_extract_toolkit()
|
||||
agent = Agent(
|
||||
name="reme_dreamer_extract",
|
||||
model=self.llm,
|
||||
model=self.as_llm,
|
||||
system_prompt=self.prompt_format(
|
||||
"extract_system_prompt",
|
||||
vault_dir=str(vault_dir),
|
||||
|
|
@ -326,7 +326,7 @@ class DreamStep(BaseStep):
|
|||
Msg(name="reme", role="user", content=[TextBlock(text=user_message)]),
|
||||
)
|
||||
|
||||
structured_resp = await self.llm.generate_structured_output(
|
||||
structured_resp = await self.as_llm.generate_structured_output(
|
||||
agent.state.context,
|
||||
structured_model=ExtractedUnits,
|
||||
)
|
||||
|
|
@ -362,7 +362,7 @@ class DreamStep(BaseStep):
|
|||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
agent = Agent(
|
||||
name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}",
|
||||
model=self.llm,
|
||||
model=self.as_llm,
|
||||
system_prompt=self.prompt_format(
|
||||
f"integrate_system_prompt_{bucket}",
|
||||
vault_dir=str(vault_dir),
|
||||
|
|
@ -387,7 +387,7 @@ class DreamStep(BaseStep):
|
|||
await agent.reply(
|
||||
Msg(name="reme", role="user", content=[TextBlock(text=user_message)]),
|
||||
)
|
||||
structured_resp = await self.llm.generate_structured_output(
|
||||
structured_resp = await self.as_llm.generate_structured_output(
|
||||
agent.state.context,
|
||||
structured_model=IntegrateOutcome,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@ Outputs:
|
|||
metadata = {date, session_id, path, created, index?}
|
||||
"""
|
||||
|
||||
from datetime import date as _date
|
||||
from pathlib import Path
|
||||
|
||||
import frontmatter
|
||||
|
|
@ -33,6 +32,7 @@ from ._daily_index import refresh_day_index, validate_session_id
|
|||
from ._file_io import write_file_safe
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
from ...steps.evolve import now
|
||||
|
||||
|
||||
@R.register("daily_create_step")
|
||||
|
|
@ -51,7 +51,8 @@ class DailyCreateStep(BaseStep):
|
|||
"""Read ``session_id`` + ``date`` from context; default ``date`` today, ``daily_dir`` from app config."""
|
||||
assert self.context is not None
|
||||
session_id = self.context.get("session_id", "")
|
||||
day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d")
|
||||
tz = self.app_context.app_config.timezone if self.app_context is not None else None
|
||||
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
|
||||
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
|
||||
return session_id, day, daily_dir
|
||||
|
||||
|
|
|
|||
|
|
@ -12,12 +12,12 @@ Input is a single optional ``date`` (``YYYY-MM-DD``); falls back
|
|||
to today.
|
||||
"""
|
||||
|
||||
from datetime import date as _date
|
||||
from pathlib import Path
|
||||
|
||||
from ._daily_index import scan_notes
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
from ...steps.evolve import now
|
||||
|
||||
|
||||
@R.register("daily_list_step")
|
||||
|
|
@ -27,7 +27,8 @@ class DailyListStep(BaseStep):
|
|||
def _collect_params(self) -> tuple[str, str, Path]:
|
||||
"""Read ``date`` (default today, ``YYYY-MM-DD``), resolve ``daily_dir``, locate the vault root on disk."""
|
||||
assert self.context is not None
|
||||
day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d")
|
||||
tz = self.app_context.app_config.timezone if self.app_context is not None else None
|
||||
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
|
||||
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
|
||||
vault_dir = Path(self.file_store.vault_path or ".").resolve()
|
||||
return day, daily_dir, vault_dir
|
||||
|
|
|
|||
|
|
@ -19,11 +19,10 @@ today.
|
|||
Always idempotent and safe to re-run.
|
||||
"""
|
||||
|
||||
from datetime import date as _date
|
||||
|
||||
from ._daily_index import refresh_day_index
|
||||
from ..base_step import BaseStep
|
||||
from ...components import R
|
||||
from ...steps.evolve import now
|
||||
|
||||
|
||||
@R.register("daily_reindex_step")
|
||||
|
|
@ -33,7 +32,8 @@ class DailyReindexStep(BaseStep):
|
|||
def _collect_params(self) -> tuple[str, str]:
|
||||
"""Read ``date`` (default today) and ``daily_dir`` (default ``daily``) from context/app config."""
|
||||
assert self.context is not None
|
||||
day = self.context.get("date", "") or _date.today().strftime("%Y-%m-%d")
|
||||
tz = self.app_context.app_config.timezone if self.app_context is not None else None
|
||||
day = self.context.get("date", "") or now(tz).strftime("%Y-%m-%d")
|
||||
daily_dir = self.app_context.app_config.daily_dir if self.app_context is not None else "daily"
|
||||
return day, daily_dir
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ One class, :class:`WikilinkHandler`, owning every wikilink concern:
|
|||
|
||||
* **Pure text** — regex, Dataview predicate inference, validation:
|
||||
:meth:`~WikilinkHandler.extract_links` (used by
|
||||
:mod:`reme.components.file_parser.linked_file_parser`),
|
||||
:mod:`reme.components.file_chunker.markdown_file_chunker`),
|
||||
:meth:`~WikilinkHandler.scan_and_rewrite`,
|
||||
:meth:`~WikilinkHandler.validate_src_dst` /
|
||||
:meth:`~WikilinkHandler.validate_scope` /
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""Tests for background steps: ScanStoreChangesStep + WatchChangesStep.
|
||||
|
||||
Both steps are subclasses of BaseStep. To exercise them without spinning up the
|
||||
full ApplicationContext, we pass real (started) file_store/file_parser via the
|
||||
full ApplicationContext, we pass real (started) file_store/file_chunker via the
|
||||
step's kwargs (so the BaseStep _resolve() machinery returns them).
|
||||
|
||||
ScanStoreChangesStep writes its result into ``context["changes"]`` for a downstream
|
||||
|
|
@ -22,7 +22,7 @@ from pathlib import Path
|
|||
|
||||
from watchfiles import Change
|
||||
|
||||
from reme4.components.file_parser import ChunkedFileParser
|
||||
from reme4.components.file_chunker import DefaultFileChunker
|
||||
from reme4.components.file_store import LocalFileStore
|
||||
from reme4.components.runtime_context import RuntimeContext
|
||||
from reme4.steps import ScanStoreChangesStep, WatchChangesStep
|
||||
|
|
@ -63,15 +63,15 @@ async def _make_scan_step(
|
|||
watch_paths: list[str] | str = "vault",
|
||||
suffix_filters: list[str] | None = None,
|
||||
recursive: bool = True,
|
||||
) -> tuple[ScanStoreChangesStep, RuntimeContext, LocalFileStore, ChunkedFileParser]:
|
||||
) -> tuple[ScanStoreChangesStep, RuntimeContext, LocalFileStore, DefaultFileChunker]:
|
||||
fs = LocalFileStore(name="test_store", embedding_store="")
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
await fs.start()
|
||||
await parser.start()
|
||||
step = ScanStoreChangesStep(
|
||||
recursive=recursive,
|
||||
file_store=fs,
|
||||
file_parser=parser,
|
||||
file_chunker=parser,
|
||||
)
|
||||
context = RuntimeContext(
|
||||
watch_paths=watch_paths,
|
||||
|
|
@ -80,7 +80,7 @@ async def _make_scan_step(
|
|||
return step, context, fs, parser
|
||||
|
||||
|
||||
async def _teardown(fs: LocalFileStore, parser: ChunkedFileParser) -> None:
|
||||
async def _teardown(fs: LocalFileStore, parser: DefaultFileChunker) -> None:
|
||||
await parser.close()
|
||||
await fs.close()
|
||||
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ from reme4.enumeration import ComponentEnum
|
|||
|
||||
|
||||
class StubComponent(BaseComponent):
|
||||
component_type = ComponentEnum.FILE_PARSER
|
||||
component_type = ComponentEnum.FILE_CHUNKER
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
|
|
@ -42,18 +42,18 @@ class RequiredDepTarget(BaseComponent):
|
|||
|
||||
|
||||
def test_dependency_repr_optional():
|
||||
dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser", optional=True)
|
||||
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=True)
|
||||
assert "?" in repr(dep)
|
||||
assert "file_parser" in repr(dep)
|
||||
assert "file_chunker" in repr(dep)
|
||||
|
||||
|
||||
def test_dependency_repr_required():
|
||||
dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser", optional=False)
|
||||
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser", optional=False)
|
||||
assert "?" not in repr(dep)
|
||||
|
||||
|
||||
def test_dependency_getattr_raises():
|
||||
dep = Dependency(ComponentEnum.FILE_PARSER, "my_parser")
|
||||
dep = Dependency(ComponentEnum.FILE_CHUNKER, "my_parser")
|
||||
with pytest.raises(RuntimeError, match="accessed before start"):
|
||||
_ = dep.some_method
|
||||
|
||||
|
|
@ -312,7 +312,7 @@ def test_vault_metadata_path_no_context():
|
|||
|
||||
def test_component_metadata_path():
|
||||
comp = StubComponent()
|
||||
assert comp.component_metadata_path.name == ComponentEnum.FILE_PARSER.value
|
||||
assert comp.component_metadata_path.name == ComponentEnum.FILE_CHUNKER.value
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from reme4.enumeration import ComponentEnum
|
|||
|
||||
|
||||
class _DummyComponent(BaseComponent):
|
||||
component_type = ComponentEnum.FILE_PARSER
|
||||
component_type = ComponentEnum.FILE_CHUNKER
|
||||
|
||||
|
||||
class _AnotherComponent(BaseComponent):
|
||||
|
|
@ -31,13 +31,13 @@ class _BaseComponentType(BaseComponent):
|
|||
def test_register_direct_with_explicit_name():
|
||||
reg = ComponentRegistry()
|
||||
reg.register(_DummyComponent, "my_parser")
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "my_parser") is _DummyComponent
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "my_parser") is _DummyComponent
|
||||
|
||||
|
||||
def test_register_direct_defaults_to_class_name():
|
||||
reg = ComponentRegistry()
|
||||
reg.register(_DummyComponent)
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "_DummyComponent") is _DummyComponent
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "_DummyComponent") is _DummyComponent
|
||||
|
||||
|
||||
def test_register_decorator():
|
||||
|
|
@ -45,16 +45,16 @@ def test_register_decorator():
|
|||
|
||||
@reg.register("alias")
|
||||
class MyParser(BaseComponent):
|
||||
component_type = ComponentEnum.FILE_PARSER
|
||||
component_type = ComponentEnum.FILE_CHUNKER
|
||||
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "alias") is MyParser
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "alias") is MyParser
|
||||
|
||||
|
||||
def test_register_overwrite_warns(caplog):
|
||||
reg = ComponentRegistry()
|
||||
reg.register(_DummyComponent, "dup")
|
||||
reg.register(_DummyComponent, "dup")
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "dup") is _DummyComponent
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "dup") is _DummyComponent
|
||||
|
||||
|
||||
def test_register_rejects_missing_component_type():
|
||||
|
|
@ -83,7 +83,7 @@ def test_get_all_returns_copy():
|
|||
reg.register(_DummyComponent, "a")
|
||||
reg.register(_AnotherComponent, "b")
|
||||
|
||||
parsers = reg.get_all(ComponentEnum.FILE_PARSER)
|
||||
parsers = reg.get_all(ComponentEnum.FILE_CHUNKER)
|
||||
assert parsers == {"a": _DummyComponent}
|
||||
|
||||
indexes = reg.get_all(ComponentEnum.KEYWORD_INDEX)
|
||||
|
|
@ -91,12 +91,12 @@ def test_get_all_returns_copy():
|
|||
|
||||
# Mutating the copy doesn't affect the registry.
|
||||
parsers["hacked"] = _DummyComponent
|
||||
assert "hacked" not in reg.get_all(ComponentEnum.FILE_PARSER)
|
||||
assert "hacked" not in reg.get_all(ComponentEnum.FILE_CHUNKER)
|
||||
|
||||
|
||||
def test_get_all_unknown_type_returns_empty():
|
||||
reg = ComponentRegistry()
|
||||
assert not reg.get_all(ComponentEnum.LLM)
|
||||
assert not reg.get_all(ComponentEnum.AS_LLM)
|
||||
|
||||
|
||||
# -- get (miss) ---------------------------------------------------------------
|
||||
|
|
@ -104,7 +104,7 @@ def test_get_all_unknown_type_returns_empty():
|
|||
|
||||
def test_get_nonexistent_returns_none():
|
||||
reg = ComponentRegistry()
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "nope") is None
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "nope") is None
|
||||
|
||||
|
||||
# -- unregister ---------------------------------------------------------------
|
||||
|
|
@ -113,13 +113,13 @@ def test_get_nonexistent_returns_none():
|
|||
def test_unregister_existing():
|
||||
reg = ComponentRegistry()
|
||||
reg.register(_DummyComponent, "x")
|
||||
assert reg.unregister(ComponentEnum.FILE_PARSER, "x") is True
|
||||
assert reg.get(ComponentEnum.FILE_PARSER, "x") is None
|
||||
assert reg.unregister(ComponentEnum.FILE_CHUNKER, "x") is True
|
||||
assert reg.get(ComponentEnum.FILE_CHUNKER, "x") is None
|
||||
|
||||
|
||||
def test_unregister_missing_returns_false():
|
||||
reg = ComponentRegistry()
|
||||
assert reg.unregister(ComponentEnum.FILE_PARSER, "nope") is False
|
||||
assert reg.unregister(ComponentEnum.FILE_CHUNKER, "nope") is False
|
||||
|
||||
|
||||
# -- clear --------------------------------------------------------------------
|
||||
|
|
@ -130,7 +130,7 @@ def test_clear():
|
|||
reg.register(_DummyComponent, "a")
|
||||
reg.register(_AnotherComponent, "b")
|
||||
reg.clear()
|
||||
assert not reg.get_all(ComponentEnum.FILE_PARSER)
|
||||
assert not reg.get_all(ComponentEnum.FILE_CHUNKER)
|
||||
assert not reg.get_all(ComponentEnum.KEYWORD_INDEX)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
"""Tests for ChunkedFileParser."""
|
||||
"""Tests for DefaultFileChunker."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from reme4.components.file_parser import ChunkedFileParser
|
||||
from reme4.components.file_chunker import DefaultFileChunker
|
||||
from reme4.utils.wikilink_handler import WikilinkHandler
|
||||
|
||||
|
||||
|
|
@ -19,7 +19,7 @@ def test_parse_empty_file():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
file_node, chunks = await parser.parse(temp_path)
|
||||
assert file_node.path == temp_path
|
||||
assert len(chunks) == 0
|
||||
|
|
@ -40,7 +40,7 @@ def test_parse_small_file():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=10000)
|
||||
parser = DefaultFileChunker(chunk_byte_size=10000)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0].start_line == 1
|
||||
|
|
@ -64,7 +64,7 @@ def test_parse_multiline_file():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=10000)
|
||||
parser = DefaultFileChunker(chunk_byte_size=10000)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0].start_line == 1
|
||||
|
|
@ -88,7 +88,7 @@ def test_parse_chunked_file():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=5000, overlap_byte_size=100)
|
||||
parser = DefaultFileChunker(chunk_byte_size=5000, overlap_byte_size=100)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
assert len(chunks) > 1, f"Expected multiple chunks, got {len(chunks)}"
|
||||
# Verify overlap by checking that consecutive chunks share some content
|
||||
|
|
@ -110,7 +110,7 @@ def test_parse_with_custom_encoding():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(encoding="utf-8")
|
||||
parser = DefaultFileChunker(encoding="utf-8")
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
assert len(chunks) >= 1
|
||||
assert "你好世界" in chunks[0].text
|
||||
|
|
@ -131,7 +131,7 @@ def test_file_node_properties():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
file_node, _ = await parser.parse(temp_path)
|
||||
assert hasattr(file_node, "path")
|
||||
assert hasattr(file_node, "st_mtime")
|
||||
|
|
@ -153,7 +153,7 @@ def test_file_chunk_properties():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
chunk = chunks[0]
|
||||
assert hasattr(chunk, "path")
|
||||
|
|
@ -298,7 +298,7 @@ def test_parse_links_in_file():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
file_node, _ = await parser.parse(temp_path)
|
||||
triples = {(link.predicate, link.target_path, link.target_anchor) for link in file_node.links}
|
||||
assert (None, "alpha", None) in triples
|
||||
|
|
@ -326,7 +326,7 @@ def test_parse_links_empty_when_no_content():
|
|||
fm_only_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser()
|
||||
parser = DefaultFileChunker()
|
||||
node1, _ = await parser.parse(empty_path)
|
||||
node2, _ = await parser.parse(fm_only_path)
|
||||
assert node1.links == []
|
||||
|
|
@ -354,7 +354,7 @@ def test_chunk_does_not_split_wikilink_at_boundary():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=10)
|
||||
parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=10)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
# The first chunk must NOT contain a partial link.
|
||||
first = chunks[0].text
|
||||
|
|
@ -383,7 +383,7 @@ def test_chunk_does_not_split_wikilink_in_overlap():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=20)
|
||||
parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=20)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
# No chunk should start mid-link.
|
||||
for c in chunks:
|
||||
|
|
@ -413,7 +413,7 @@ def test_chunk_falls_back_for_oversize_link():
|
|||
temp_path = f.name
|
||||
|
||||
try:
|
||||
parser = ChunkedFileParser(chunk_byte_size=100, overlap_byte_size=10)
|
||||
parser = DefaultFileChunker(chunk_byte_size=100, overlap_byte_size=10)
|
||||
_, chunks = await parser.parse(temp_path)
|
||||
# Must terminate (not hang) and cover the whole file.
|
||||
assert len(chunks) >= 2
|
||||
|
|
@ -429,7 +429,7 @@ def test_min_chunk_and_overlap_size():
|
|||
|
||||
async def run():
|
||||
# These values should be clamped to minimums
|
||||
parser = ChunkedFileParser(chunk_byte_size=1, overlap_byte_size=0)
|
||||
parser = DefaultFileChunker(chunk_byte_size=1, overlap_byte_size=0)
|
||||
assert parser.chunk_byte_size == 100 # minimum
|
||||
assert parser.overlap_byte_size == 4 # minimum
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""Tests for LinkedFileParser (markdown parser + wikilink extraction).
|
||||
"""Tests for MarkdownFileChunker (markdown parser + wikilink extraction).
|
||||
|
||||
Wikilink convention here is strict: targets are taken literally, no
|
||||
short-form basename search, no implicit ``.md``, no folder-note
|
||||
|
|
@ -12,7 +12,7 @@ import asyncio
|
|||
import os
|
||||
import tempfile
|
||||
|
||||
from reme4.components.file_parser import LinkedFileParser
|
||||
from reme4.components.file_chunker import MarkdownFileChunker
|
||||
|
||||
|
||||
class temp_chdir:
|
||||
|
|
@ -46,7 +46,7 @@ def test_parse_empty_file():
|
|||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
path = _write_md(tmp, "x.md", "")
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, chunks = await parser.parse(path)
|
||||
assert node.path == "x.md"
|
||||
assert chunks == []
|
||||
|
|
@ -62,7 +62,7 @@ def test_parse_frontmatter_only():
|
|||
async def run():
|
||||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
path = _write_md(tmp, "fm.md", "---\nname: t\n---\n")
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, chunks = await parser.parse(path)
|
||||
assert node.front_matter.name == "t"
|
||||
assert chunks == []
|
||||
|
|
@ -79,7 +79,7 @@ def test_parse_small_body_one_chunk():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "# Hello\n\nthis is a small body."
|
||||
path = _write_md(tmp, "small.md", body)
|
||||
parser = LinkedFileParser(chunk_chars=500)
|
||||
parser = MarkdownFileChunker(chunk_chars=500)
|
||||
node, chunks = await parser.parse(path)
|
||||
assert len(chunks) == 1
|
||||
assert "this is a small body" in chunks[0].text
|
||||
|
|
@ -97,7 +97,7 @@ def test_parse_oversized_body_splits():
|
|||
paras = "\n\n".join(f"paragraph {i} with some content text here." for i in range(50))
|
||||
body = "# H\n\n" + paras
|
||||
path = _write_md(tmp, "big.md", body)
|
||||
parser = LinkedFileParser(chunk_chars=200)
|
||||
parser = MarkdownFileChunker(chunk_chars=200)
|
||||
_, chunks = await parser.parse(path)
|
||||
assert len(chunks) > 1
|
||||
print("✓ test_parse_oversized_body_splits passed")
|
||||
|
|
@ -113,7 +113,7 @@ def test_parse_chunk_ids_match_node_chunk_ids():
|
|||
paras = "\n\n".join(f"para {i} body content here." for i in range(40))
|
||||
body = "# H\n\n" + paras
|
||||
path = _write_md(tmp, "p.md", body)
|
||||
parser = LinkedFileParser(chunk_chars=200)
|
||||
parser = MarkdownFileChunker(chunk_chars=200)
|
||||
node, chunks = await parser.parse(path)
|
||||
assert node.chunk_ids == [c.id for c in chunks]
|
||||
print("✓ test_parse_chunk_ids_match_node_chunk_ids passed")
|
||||
|
|
@ -128,7 +128,7 @@ def test_parse_links_literal_targets():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "see [[topics/Alice.md]] and [[topics/Bob.md#sec]]"
|
||||
path = _write_md(tmp, "note.md", body)
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, _ = await parser.parse(path)
|
||||
triples = {(link.target_path, link.target_anchor, link.predicate) for link in node.links}
|
||||
assert ("topics/Alice.md", None, None) in triples
|
||||
|
|
@ -154,7 +154,7 @@ def test_parse_links_short_and_no_ext_kept_literally():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "see [[Alice]] and [[topics/Alice]] but also [[topics/Alice.md]]"
|
||||
path = _write_md(tmp, "note.md", body)
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, _ = await parser.parse(path)
|
||||
targets = {link.target_path for link in node.links}
|
||||
assert targets == {"Alice", "topics/Alice", "topics/Alice.md"}
|
||||
|
|
@ -170,7 +170,7 @@ def test_parse_links_predicate_inline_and_line():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "extends:: [[A.md]]\n\nsome [concerns:: [[B.md]]] inline\n"
|
||||
path = _write_md(tmp, "note.md", body)
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, _ = await parser.parse(path)
|
||||
pairs = {(link.target_path, link.predicate) for link in node.links}
|
||||
assert ("A.md", "extends") in pairs
|
||||
|
|
@ -187,7 +187,7 @@ def test_parse_links_deduped():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "[[A.md]] again [[A.md]] and [[A.md]]"
|
||||
path = _write_md(tmp, "note.md", body)
|
||||
parser = LinkedFileParser()
|
||||
parser = MarkdownFileChunker()
|
||||
node, _ = await parser.parse(path)
|
||||
assert len([link for link in node.links if link.target_path == "A.md"]) == 1
|
||||
print("✓ test_parse_links_deduped passed")
|
||||
|
|
@ -197,7 +197,7 @@ def test_parse_links_deduped():
|
|||
|
||||
def test_parse_min_chunk_chars_clamped():
|
||||
"""chunk_chars below 100 should be clamped to 100."""
|
||||
parser = LinkedFileParser(chunk_chars=10)
|
||||
parser = MarkdownFileChunker(chunk_chars=10)
|
||||
assert parser.chunk_chars == 100
|
||||
print("✓ test_parse_min_chunk_chars_clamped passed")
|
||||
|
||||
|
|
@ -209,7 +209,7 @@ def test_parse_embed_toc_prefixes_chunk_text():
|
|||
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
|
||||
body = "# Top\n\n## Sub\n\nbody-content"
|
||||
path = _write_md(tmp, "toc.md", body)
|
||||
parser = LinkedFileParser(chunk_chars=200, embed_toc=True)
|
||||
parser = MarkdownFileChunker(chunk_chars=200, embed_toc=True)
|
||||
_, chunks = await parser.parse(path)
|
||||
# Single small section fits; check that the heading appears in text.
|
||||
assert any("Top" in c.text for c in chunks)
|
||||
|
|
@ -219,7 +219,7 @@ def test_parse_embed_toc_prefixes_chunk_text():
|
|||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("\n=== LinkedFileParser tests ===")
|
||||
print("\n=== MarkdownFileChunker tests ===")
|
||||
test_parse_empty_file()
|
||||
test_parse_frontmatter_only()
|
||||
test_parse_small_body_one_chunk()
|
||||
Loading…
Add table
Reference in a new issue