mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
856 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Diego C
|
2f37a159b7
|
为文件型记忆(ReMeLight)新增 LoCoMo 评测支持。在此之前仅向量型记忆系统(ReMe)有评测脚本 (#283)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
* feat: 新增 ReMeLight 文件版记忆的 LoCoMo 评测脚本 * feat: add LoCoMo eval for ReMeLight file-based memory * fix: resolve pre-commit lint issues * fix: align answer model with eval_reme.py (qwen3-30b-a3b-instruct-2507) * fix: add --disable=E0611 |
||
|
jinliyl
|
26cb5ca62f
|
Dev/0618 (#282)
* feat(core): enhance agent wrapper with session management and stream chunks
- Add AddStep for simple arithmetic operations
- Implement session persistence with AsStateHandler for AgentScope
- Introduce stream chunk conversion for unified event handling
- Add file session store for Claude Code agent backend
- Support skill integration and permission handling in agent wrappers
- Add proactive and daily topics features to auto dream step
- Refactor auto memory and auto resource steps to use job tools
- Update application shutdown sequence for proper resource cleanup
- Enhance environment loading utilities with parse_env_file function
- Add comprehensive error handling and validation for session IDs
* refactor(embedding): simplify model initialization and update component management
- Replace individual _start methods with shared model_cls attribute pattern
- Remove redundant _close methods from embedding model wrappers
- Add application-level component update capability via update_component method
- Simplify BaseAsLLM _start method with early return when model exists
- Remove unused skill instruction template from agent wrapper
- Restructure CronJob to execute its own steps instead of dispatching external jobs
- Update cron job configuration format to use steps instead of dispatch targets
- Refactor cron job tests to match new execution model
- Remove deprecated dispatch_step/dispatch_job functionality from cron job
* feat(agent): add session management and cleanup functionality
- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class
* feat(agent): add session management and cleanup functionality
- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class
* refactor(application): restructure job startup order and improve error handling
- Change job startup sequence from background-last to base-stream-background-cron
- Import specific job types (BackgroundJob, CronJob, StreamJob) instead of generic BaseJob
- Update isinstance checks for proper job type identification
- Implement robust error handling during component closure with preserved exceptions
- Modify job merging to combine config and call-time kwargs in BaseJob and StreamJob
- Update service job registration to return boolean success indicators
- Add timezone support for cron job scheduling
- Enhance keyword index persistence with component-specific filenames
- Add comprehensive file store consistency tests and search filtering capabilities
- Include tokenizer stopwords in package distribution
- Fix prompt handler validation behavior and error messages
* fix(core): handle exceptions during application startup and improve validation
- Add exception handling around component startup to close started components on failure
- Replace assertions with runtime checks in claim_channel step for Python -O compatibility
- Add input validation for config parser including empty keys and non-mapping roots
- Enhance environment variable expansion to convert scalar types
- Add support for relative config file paths by searching in config directory
- Validate 2D array requirements in batch cosine similarity function
- Update channel notify step to return proper response objects
- Pass client-specific arguments through CLI to HTTP client initialization
- Add comprehensive tests for error conditions and edge cases
* feat(graph): add Neo4j backend support with enhanced health monitoring
- Implement Neo4jFileGraph component with connection constraints and async operations
- Add cached count tracking for nodes, edges, and virtual nodes in Neo4j backend
- Update health check to include Neo4j status reporting with memory usage
- Modify LLM demo steps to always register add tool without conditional flag
- Enhance AddStep to handle numeric string conversion and input validation
- Add comprehensive unit tests for Neo4j integration and error handling scenarios
- Remove deprecated use_add_tool parameter from LLM demo components
- Update integration tests to reflect simplified tool registration approach
* feat(file_io): enhance file I/O operations with path validation and large file handling
- Add resolve_path function with comprehensive path validation and security checks
- Implement read_file_lines_safe for efficient reading of large files by line ranges
- Integrate path validation across all file I/O operations to prevent directory traversal
- Add proper error handling for invalid paths and file access issues
- Enhance daily index operations with path resolution and error reporting
- Add 'changed' field to index responses to track file modification status
- Update file listing operations to use secure path resolution
- Add support for JSONL files in default scanning operations
- Improve move and delete operations with proper path validation
- Add comprehensive path validation tests and security checks
* refactor(steps): update file I/O and prompt handling implementations
- Add module docstring to file_io/__init__.py
- Remove unused validate parameter from prompt_format method
- Update import path from reme.reme to reme4.reme in common_utils.py
- Add missing docstrings to test classes and methods
- Remove deprecated test_format_missing_variable_no_validate test
- Simplify assertion in test_job.py using not operator
- Update import statement in test_utils.py for common_utils
- Add docstrings to dummy classes and functions in tests
- Rename variable in get_node_embeddings for clarity
* refactor(steps): restructure step modules and update change handling
- Split monolithic steps module into channel, common, evolve, file_io, index, and transfer submodules
- Replace ScanStoreChangesStep and ScanCatalogChangesStep with unified InitChangesStep
- Remove ForeachDispatchStep and replace with direct dispatch_steps mechanism in InitChangesStep
- Update configuration to use new init_changes_step with dispatch_steps pattern
- Add coalesce_changes utility for collapsing duplicate file change events
- Enhance AutoResourceStep to handle batch changes instead of single file operations
- Introduce ClearStoreStep to replace ClearAndScanStep functionality
- Add async locks to LocalFileCatalog for thread-safe operations
- Update documentation to reflect new directory structure and session organization
* test(steps): add comprehensive unit tests for background steps and search functionality
- Add new test_background_steps.py with initialization and dispatch update tests
- Add test_index_update_loop_init_dispatch_updates_store_across_batches function
- Add test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs function
- Create new test_search_step.py with complete SearchStep unit test coverage
- Implement FakeSearchStore for isolated SearchStep testing without external dependencies
- Add hybrid search RRF merging test with vector and keyword result fusion
- Include keyword-only search test with min_score filtering functionality
- Add empty query validation test with early failure mechanism
- Test vector and keyword search method calls with proper parameter passing
- Verify score handling and result ranking in hybrid search scenarios
* refactor(file_io): remove session_agent prefix from daily note filenames
- Removed 'session_agent_' prefix from daily note file naming pattern
- Updated all references in auto_dream, auto_memory, auto_resource, and daily_steps
- Modified config documentation to reflect new filename pattern
- Changed session file storage location in auto_memory to reme_session/dialog/
- Added validate_session_id and write_file_safe imports to file_io module
- Updated tests to match new filename convention without 'session_agent_' prefix
- Fixed day index refresh logic to properly update note count descriptions
- Adjusted proactive step to use new file path pattern for session notes
* feat(auto_resource): change resource processing to use same-name daily notes
- Replace MD5-based session ID generation with UUID5 for agent sessions
- Compute note stem from resource filename instead of hashing for daily note naming
- Update delete handler to use note stem instead of session ID for file lookup
- Modify upsert handler to use note stem as session ID parameter
- Change execute method to require changes as list of dictionaries
- Update test cases to use changes array instead of individual file_path and change parameters
- Adjust test assertions to verify same-name daily note creation and modification
- Refactor session state storage to use AgentScope format and location
- Remove deprecated session_state file handling in favor of new note system
* docs(structure): update resource naming convention in documentation
- Change resource naming from hash-based to stem-based format
- Update file path references from resource_{hash(resource_name)}.md to {resource_stem}.md
- Modify documentation to reflect new resource storage structure
- Adjust auto-resource saving location to use resource stem instead of hashed name
* refactor(evolve): split auto_dream into multi-step pipeline with dedicated dream modules
- Replace single AutoDreamStep with 4-step pipeline: extract, integrate, topics, finish
- Create new dream module structure under reme4/steps/evolve/dream/
- Remove old dream.py, auto_dream.py, and daily_topics.py files
- Add DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep
- Update evolve init to import new dream step classes instead of old modules
- Document complete auto_dream logic breakdown and refactoring plan in markdown
- Consolidate dream-related functionality into focused, testable components
- Maintain LLM integration while improving step separation and error handling
* refactor(dream): update system prompts and integration logic
- Replace 'Phase 1/2' terminology with descriptive agent names in prompts
- Update extraction and integration prompts to clarify unit processing flow
- Add explicit instructions about provenance tracking and wikilink handling
- Enhance validation for source material references in personal preferences
- Add fallback mechanism to preserve topics when selection fails
- Include punctuation handling guidelines for YAML parsing
- Add comprehensive tests for wikilink graph relationships
- Create new configuration file with complete service definitions
* refactor(steps): remove auto_dream and dream steps, update config access
- Removed auto_dream step implementation and its associated logic
- Removed dream step implementation including extract and integrate phases
- Replaced direct app_context.app_config access with config_value method
- Added config_value helper method to BaseStep for unified config access
- Updated auto_memory to use session_dir config and add source conversation links
- Modified auto_resource to use config_value for directory paths
- Updated daily_* steps to use config_value for daily directory
- Added exception raising in bm25_index save method on failure
- Introduced SOURCE_CONVERSATION_KEY constant for tracking source sessions
* refactor(embedding): update embedding component to use credential-based initialization
- Replace model_cls with credential_cls for authentication handling
- Update configuration structure to separate credential and parameters
- Modify health check to access model name through new attribute path
- Change input parameter from 'text' to 'inputs' for generic handling
- Add credential initialization and parameter parsing in _start method
- Update agentscope dependency to version 2.0.2
- Remove direct model class references in favor of credential-based lookup
|
||
|
jinliyl
|
83831ec90c
|
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper（统一 Agent 后端抽象） - **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`（AgentScope 后端）**： - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**： - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议（StreamChunk / ChunkEnum） - **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。 - **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构（变化批次化 + dispatch） - 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。 - **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。 - 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块（拆分为多步 pipeline） - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织： - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。 - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。 - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。 - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。 - 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource（适配新 Agent API） - **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。 - **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。 - **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。 - 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。 ### 9. File IO / 路径安全 - **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。 - **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。 - **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。 - `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。 ### 11. 其它 - 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。 ** |
||
|
Sen Huang
|
f458566e2c
|
feat: add cron scheduling support and enhance Claude Code integration (#278)
* feat: add cron scheduling support and enhance Claude Code integration - Introduce CronStep for periodic job execution with support for cron expressions, daily schedules, and fixed intervals - Add automatic session management to Claude Code agent wrapper with cache-friendly defaults for system prompts and setting sources - Implement fork session support with proper validation - Enhance auto-dream functionality to dispatch per-file jobs instead of direct method calls for better backend agnosticism - Add session ID tracking to auto-resource operations - Remove deprecated download step component - Update auto-dream job naming from auto-dream to auto_dream - Add croniter dependency and update package data to include markdown files * feat: add CronJob component and rename cron step to cron job |
||
|
jinliyl
|
c3fb825af0
|
feat(agent): refactor agent wrapper, add session persistence, auto_resource step, and watch-loop improvements (#277)
* refactor(agent_wrapper): update agent wrapper implementations and config defaults
- Set default timezone to Asia/Shanghai in application config
- Add AgentScope imports and configure ReAct, context, and model configs
- Simplify __all__ export formatting in agent wrapper init
- Remove redundant docstring details from agent wrapper classes
- Optimize tool result handling with state assignment simplification
- Add permission context and state management for AgentScope backend
- Update Claude Code wrapper tool creation and server registration logic
- Configure default agent settings including permission mode and retry limits
- Remove obsolete comments and streamline code structure
* fix(agent): add output schema validation and BaseModel support
- Added type assertion to ensure output_schema is a dict in as_agent_wrapper
- Imported BaseModel from pydantic in base_agent_wrapper
- Modified set_output_schema to accept both dict and BaseModel types
- Added automatic conversion of BaseModel to JSON schema
- Updated method documentation to reflect new type support
* refactor(agent): replace direct agent instantiation with agent wrapper component
- Removed manual Agent creation and initialization in llm_demo step
- Integrated agent_wrapper component as dependency in base step
- Updated llm_demo step to use agent_wrapper.reply method instead of direct agent calls
- Modified structured output handling to work with new agent wrapper interface
- Simplified agent configuration by using wrapper's built-in functionality
- Updated documentation to reflect agent wrapper usage instead of direct as_llm access
- Removed redundant imports related to manual agent management
* feat(agent): add streaming support and refactor agent wrapper components
- Introduce reply_stream method in base agent wrapper with fallback implementation
- Add _build_agent helper method to AsAgentWrapper for agent instantiation
- Implement structured output generation with proper model assertions
- Update StreamLLMDemoStep to use agent_wrapper instead of direct Agent calls
- Replace manual streaming logic with execute_stream_task utility function
- Change default system prompt to provide detailed responses instead of concise ones
- Add colored output support for different chunk types in streaming demos
- Refactor test cases to use async task execution with streaming verification
* refactor(agent): remove session_id parameter from reply methods
- Removed session_id parameter from ASAgentWrapper.reply method signature
- Removed session_id parameter from BaseAgentWrapper.reply abstract method
- Removed session_id parameter from CCAgentWrapper.reply method signature
- Updated reply_stream methods to remove session_id parameter across all wrappers
- Modified CCAgentWrapper to use dynamic options assignment instead of hardcoded properties
- Set default system_prompt in config instead of hardcoded in code
- Increased default max_turns from 10 to 50 in configuration
* config: update default configuration and script entry point
- Change resource_dir from empty string to 'resource'
- Update command line entry point from 'reme4' to 'reme'
* feat(agent): add session state persistence and forking support
- Implement AsStateHandler for AgentState JSONL serialization
- Add session_id parameter to AsAgentWrapper.reply method
- Create timestamp-based session file paths with timezone support
- Load existing session state from JSONL files when session_id provided
- Save updated session state after each agent interaction
- Support session forking with UUID generation for new sessions
- Add integration tests for session persistence and forking scenarios
- Include temporary directory utilities for testing isolated sessions
- Ensure parent directories are created for session files automatically
* refactor(auto_memory): replace transcript parsing with direct message handling
- Remove transcript loading logic and related dependencies
- Add session message saving functionality with deduplication
- Use agent wrapper instead of direct AgentScope agent instantiation
- Simplify timezone handling using shared now utility
- Update logging and response metadata structure
- Remove unused imports and toolkit management methods
- Change session file naming from session_{id}.jsonl to session_agent_{id}.jsonl
* refactor(steps): move channel steps from index to channel module
- Move ChannelNotifyStep from .index.channel_notify to .channel.channel_notify
- Move ClaimChannelStep from .index.claim_channel to .channel.claim_channel
- Update __init__.py imports to reflect new module structure
- Reorganize steps list in __init__.py with channel section before index
- Add proper file prefix handling in daily index processing
- Update test imports to use new channel module location
* feat(evolve): add auto_resource step for interpreting resource files
- Add AutoResourceStep to interpret resource files into daily notes via an agent
- Implement resource file parsing with date and filename extraction logic
- Add session ID computation using MD5 hash of filename
- Create delete and upsert handlers for resource file operations
- Add truncation and sanitization functions for tool output in auto_memory
- Register auto_resource step with proper parameter validation
- Add configuration for resource watch loop with file extension filters
- Update default YAML config to include resource watch and digest watch loops
- Add shared watch-rule logic for scan_changes and watch_changes steps
- Implement foreach_dispatch and log_changes steps for change processing
- Rename update_store_index_loop to index_update_loop in configuration
- Refactor file chunking interface from parse to chunk method
- Remove unused imports and dependencies in auto_dream step
- Fix path iteration formatting in daily_index utility function
- Add comprehensive integration tests for auto_resource functionality
* refactor(auto_resource): format function call with multi-line parameters
- Reformatted await _handle_upsert call to use multiple lines for better readability
- Removed unused imports from scan_changes.py including BaseFileCatalog and ComponentEnum
- Added date parameter to RuntimeContext initialization in test cases
- Updated expected file paths in test assertions to include session_agent prefix
- Formatted long assertion statements across multiple lines to maintain character limit
- Corrected wikilink references from generic names to session_agent prefixed names
|
||
|
jinliyl
|
8eaa96390a
|
refactor(file_chunker): replace file parser with file chunker component (#276)
* refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes |
||
|
Sen Huang
|
a2d76cc034
|
refactor(auto_dream): improve recall workflow and documentation (#272)
* feat(file_store): add concurrency protection to LocalFileStore.dump() * feat(file_catalog): replace file_store with file_catalog in DreamStep * feat: add ChannelSink for Claude Code channel notifications * feat(auto-memory): add transcript_path support and enhance metadata |
||
|
jinliyl
|
a91b08f701
|
Revise index descriptions in reme_design.md
Updated the descriptions of the inverted index and vector index to clarify their implementations. |
||
|
jinliyl
|
cee2c3e338
|
Add JSONL support, reorganize vault structure, and update design docs (#274)
* feat(config): add jsonl support and update LLM integration tests - Added jsonl extension to supported extensions in chunked backend - Refactored LLM integration tests to use async functions instead of nested runs - Created helper functions _run_basic_chat, _run_with_tool, _run_structured_output, _run_structured_output_enum - Implemented _run_all function to execute all test scenarios sequentially - Updated main execution block to use asyncio.run with consolidated test runner - Maintained all original test functionality while improving code structure * feat(config): add dialog directory configuration and reorganize vault structure - Add new dialog_dir field for dialog memory storage - Reorder directory initialization sequence in application setup - Simplify vault_dir description in configuration schema - Update thread_pool_max_workers description to be more concise - Move resource_dir definition earlier in the configuration schema - Remove redundant text from digest_dir description * docs(reme): update design documentation with layered memory architecture - Replace quick test section with comprehensive layered memory structure - Add detailed explanation of three-tier memory organization (resource, daily, digest) - Document Obsidian-compatible Markdown format with YAML front matter - Describe four types of wikilink syntax and semantic linking features - Explain AST-aware semantic chunking for document parsing - Detail self-evolving system with auto-resource, auto-memory, and auto-dream - Document directory structure and lifecycle characteristics - Add comprehensive table showing content nature, triggers, and examples - Include semantic link extraction and knowledge graph formation processes - Describe automated indexing and relationship building workflows * docs(reme): update design documentation with simplified structure and clearer explanations - Simplified directory structure overview with cleaner formatting - Updated memory layering explanation with more concise descriptions - Improved table layouts for better readability - Clarified auto-resource, auto-memory, and auto-dream processes - Streamlined indexing and search mechanism descriptions - Enhanced component system documentation with clearer backend options - Refined job list with more precise functional descriptions - Modernized layout diagrams and process flows - Consolidated repetitive content while maintaining comprehensive coverage * docs(reme): add application scenario documentation for financial industry use case - Document comprehensive example of ReMe usage in新能源 industry research - Detail the week-long process of automatic knowledge graph construction - Explain the auto-memory and auto-dream pipeline with concrete examples - Describe the extract and integrate phases for creating wiki nodes - Illustrate cross-file linking through relates_to and derived_from predicates - Show progressive graph growth from daily sessions to complete ecosystem - Demonstrate hybrid retrieval with vector and keyword search capabilities - Provide detailed directory structure and file organization patterns - Explain the three-phase workflow: ingestion, processing, and retrieval - Document the financial analyst persona and their information management needs * style(config): fix spacing in thread_pool_max_workers field definition - Corrected spacing around description parameter in Field definition - Simplified multi-line assertion to single line in LLM integration test |
||
|
jinliyl
|
36a5512fc8
|
refactor(vector_store): make obvec and zvec vector stores optional dependencies (#273)
* refactor(vector_store): make obvec and zvec vector stores optional dependencies - Removed direct imports of ObVecVectorStore and ZvecVectorStore from init file - Added try-except blocks for conditional importing of optional vector stores - Updated error handling to check for both pyobvector and sqlalchemy in ObVecVectorStore - Renamed _OBVECTOR_IMPORT_ERROR to _OBVEC_IMPORT_ERROR for consistency - Moved pyobvector and related dependencies to optional 'obvec' extra - Added separate 'zvec' optional dependency group - Updated package configuration to exclude reme4 module patterns - Removed reme4 entry point from console scripts - Bumped version from 0.3.1.9 to 0.3.1.10 * refactor(dependencies): reorganize project dependencies and add optional seekdb support - Move sqlite-vec, prompt_toolkit, and rich to earlier in dependencies list - Remove pyseekdb from main dependencies and create separate seekdb optional dependency group - Reorder pyyaml to later in the dependencies list - Maintain all existing dependency versions while improving organization * chore(deps): remove faiss-cpu dependency from pyproject.toml - Removed faiss-cpu>=1.7.4 from the faiss dependency group - Cleaned up unused faiss dependency configuration - Updated project dependencies to exclude faiss-cpu package |
||
|
jinliyl
|
d8086039dc
|
refactor(Agentscope2.0): llm & embedding & agent (#271)
* refactor(embedding): replace embedding model with embedding store architecture - Remove as_token_counter component and its estimated token counter implementation - Replace BaseEmbeddingModel with BaseEmbedding that wraps AgentScope embedding models - Add support for multiple embedding providers (OpenAI, DashScope, Gemini, Ollama) - Introduce BaseEmbeddingStore and LocalEmbeddingStore for caching and persistence - Update component registry to use new embedding and embedding_store types - Modify file stores to use embedding_store instead of embedding_model - Update health check to monitor embedding_store instead of embedding_model - Change default config to use embedding_store with local backend - Add estimate_token_count utility function to utils module * refactor(llm): replace as_llm components with unified llm implementation - Remove deprecated as_llm and as_llm_formatter modules - Add new llm module with BaseLLM and provider-specific implementations - Update component registry to use LLM instead of AS_LLM - Replace all as_llm/as_llm_formatter references with llm in steps - Update configuration schema to use llm instead of as_llm - Rename integration test file from test_as_llm to test_llm - Add proper docstrings to embedding store dimension property - Add pylint disable comment for embedding model call - Remove unused FormatterBase import in base_step - Update token_utils with function docstring * refactor(evolve): replace ReActAgent with Agent and update message handling - Removed FlexReActAgent class and direct ReActAgent imports - Updated Agent instantiation to use new constructor parameters - Changed message content to use TextBlock format instead of plain strings - Modified timestamp access from msg.timestamp to msg.created_at - Updated metadata access pattern for structured outputs - Replaced Msg.from_dict with Msg.model_validate in auto_memory.py - Updated test mocks to patch Agent instead of ReActAgent - Changed message serialization from to_dict to model_dump in tests - Moved component references to base class definition - Updated demo tools to return strings instead of ToolResponse objects * feat(step): migrate to FunctionTool and add streaming support - Replace deprecated ToolResponse with FunctionTool in base_step.py - Remove unused TextBlock import from base_step.py - Update job registration to use new FunctionTool API - Add thinking_budget parameter to llm_demo configuration - Introduce StreamLLMDemoStep with streaming output capability - Add structured output support to LLMDemoStep via generate_structured_output - Implement streaming event handling for text/thinking/tool calls - Add integration tests for embedding functionality - Add integration tests for structured output and streaming features - Update tool usage in demo steps to use new function naming convention * fix(ci): correct package installation path in unittest workflow - Updated pip install command to use proper package path "./reme4[dev,core]" - Fixed dependency installation step in CI workflow configuration * chore(workflow): update python versions in unittest workflow - Remove Python 3.10 from test matrix - Add Python 3.11 to test matrix - Add Python 3.12 to test matrix - Keep Python 3.13 in test matrix - Update matrix configuration for better version coverage * fix(health): handle missing dimensions attribute in embedding status - Wrap dimensions access in try-except to prevent AttributeError - Return None when dimensions attribute is not available - Maintain backward compatibility for components without dimensions test(component): add comprehensive tests for BaseComponent and related classes - Add tests for Dependency class including repr and attribute access - Add tests for bind method with various scenarios and edge cases - Add tests for lifecycle management and async context handling - Add tests for standalone and context-bound dependency resolution - Add tests for ComponentMixin path utilities test(common): update LocalFileStore initialization parameter - Change embedding_model parameter to embedding_store in test setup - Update all affected test files consistently test(registry): add complete test suite for ComponentRegistry - Add tests for register method with explicit names and defaults - Add tests for decorator registration pattern - Add tests for get_all method returning copies - Add tests for unregister and clear operations - Add tests for error handling of invalid registrations test(job): add comprehensive tests for BaseJob and BackgroundJob - Add tests for step resolution and exception handling - Add tests for backoff delay calculation with jitter - Add tests for supervisor loop restart behavior - Add tests for task shutdown and cancellation test(prompt): add complete test suite for PromptHandler - Add tests for prompt loading from dictionaries and files - Add tests for internationalization and language fallback - Add tests for flag filtering and variable substitution - Add tests for format validation and error handling test(runtime): add basic tests for RuntimeContext dictionary access - Add tests for item getting, setting and containment checks - Add tests for missing key error handling * feat(evolve): add permission context and agent state management - Import PermissionContext, PermissionMode and AgentState modules - Add state configuration with bypass permission mode to AutoDream agents - Add state configuration with bypass permission mode to AutoMemory agents - Implement static _to_msg method for message validation and formatting - Refactor message processing to use the new _to_msg method - Ensure proper content structure for text blocks in message conversion * style(tests): update test files with linting rules and code improvements - Add missing pylint disable directives for docstring and attribute warnings - Replace lambda expressions with proper function definitions in test cases - Import Path directly instead of using lambda with __import__ - Simplify assertion checks by using truthiness instead of equality to empty dict - Remove unused imports and reorder imports consistently - Format dictionary literals with proper indentation and line breaks |
||
|
jinliyl
|
2c35d31762
|
feat(application): add thread pool support and background job threading capabilities (#270)
* feat(application): add thread pool support and background job threading capabilities - Integrate ThreadPoolExecutor for shared thread pool management in application context - Add thread_pool_max_workers configuration option with default value of 0 (disabled) - Implement thread pool creation and shutdown in application lifecycle methods - Add use_thread_pool option to background job configuration for thread-based execution - Support both asyncio event loop and threading event for background job stop mechanism - Implement _run_in_thread method for running supervisors in dedicated threads - Modify embedding models to use serial batching instead of concurrent batching - Remove max_concurrency parameter from base embedding model component - Update embedding model documentation to reflect serial batching implementation - Add pyproject.toml with project metadata, dependencies, and build configuration * refactor(job): simplify background job thread execution - Replace separate _run_in_thread method with direct lambda execution - Remove unnecessary ensure_future wrapper for thread pool execution - Simplify asyncio event loop usage in thread pool mode - Maintain same background job functionality with cleaner implementation - Remove redundant method definition and streamline execution flow |
||
|
Sen Huang
|
16d2d84431
|
feat(dream): replace digester with abstraction-layer dreamer pipeline (#264)
* feat(dream): replace digester with abstraction-layer dreamer pipeline
Reframe digest as the abstract memory layer (details stay in the daily/
resource material; digest holds principles, patterns, precedents reachable
via derived_from provenance edges). Replaces the old digester with a
2-phase ReAct workflow + a daily-tick wrapper:
- Phase 1 (Dreamer extract): clusters material into orthogonal memory
sub-units; each sub-unit maps 1:1 to a digest node (no inner atom
enumeration). Biases toward fewer / richer sub-units.
- Phase 2 (Dreamer integrate per sub-unit): cross-bucket recall +
exactly one write decision (CREATE / UPDATE / SKIP); UPDATE shapes
surfaced explicitly (corroborate / refine / correct).
- CronDreamer: scans <daily_dir>/<today>.md + <daily_dir>/<today>/**
+ <resource_dir>/<today>/** and runs dream_one per file.
Write tools are proper subclasses of the canonical file_io WriteStep /
EditStep with only path-shape + bucket + E-1 edge-conservation rules
layered on top:
- DigestWriteStep(WriteStep): path = <digest_dir>/<bucket>/<slug>.md,
must-not-exist, schema mirrors `write` (path / name / description /
content) so frontmatter lands automatically.
- DigestEditStep(EditStep): body-only find-and-replace + must-exist +
E-1 conservation preflight (refuses if any outbound wikilink would
be dropped).
Configuration:
- Bucket vocabulary structured in code (tuple[{name, description}]);
prompt renders the heuristic block at runtime via {buckets}.
- digest_dir / daily_dir / resource_dir come from app config (not tool
params); prompts use {digest_dir} placeholder.
- BaseStep walks class MRO when loading prompts, so subclasses inherit
parent yaml without duplication.
Tooling: agentscope register_tool_function schemas now wrap in the
proper {"type":"function","function":{...}} envelope. OpenAIAsLLM
routes base_url through client_kwargs so non-default endpoints work.
Smoke: tests4/smoke/{_dreamer_fixture.py,test_dreamer_inproc.py,
test_dreamer_cli.sh} drive the end-to-end pipeline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(dreamer): split long description string across multiple lines
* refactor(dream): remove hardcoded DEFAULT_DIGEST_DIR and use app_config
* docs(auto-cognition): add comprehensive design document for auto-cognition system
* refactor(steps): remove deprecated digest edit/write steps
* refactor(config): remove redundant LLM formatter backend configuration
* refactor(dreamer): improve code formatting and line breaks
* feat(auto-dream): implement three-bucket classification system for knowledge organization
* feat: rename dream_today step to auto-dream and refactor extraction logic
|
||
|
jinliyl
|
aa87f4fdea
|
feat(base_step): set default language from app context when not provided (#269)
- Initialize language attribute with app context language if not explicitly set - Add conditional logic to check for existing language value before assignment - Ensure proper fallback behavior when language parameter is empty or None |
||
|
jinliyl
|
041f957a7f
|
refactor(components) components and file I/O, fix method calls and validation (#268)
* refactor(components): extract shared component state into mixin - Introduce ComponentMixin class with shared state for components and steps - Move identity, config, and vault path functionality to ComponentMixin - Update BaseComponent to inherit from ComponentMixin - Update BaseStep to inherit from ComponentMixin - Consolidate vault path helper methods in ComponentMixin - Remove duplicate vault path implementations from BaseComponent and BaseStep - Add ComponentMixin to components module exports * refactor(file_io): implement path locks cache eviction mechanism - Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit - Implement cache eviction logic when locks exceed maximum capacity - Remove half of unlocked entries when cache limit is reached - Use list comprehension to identify unlocked locks for removal - Maintain existing path normalization and locking behavior fix(edit): correct method call from public to private fail method - Change self.fail to self._fail for internal error handling - Maintain consistent private method usage within class fix(mcp_client): change pop to get for optional command and args - Replace kwargs.pop with kwargs.get to avoid removing keys - Preserve original kwargs dictionary contents - Maintain default empty string and list values feat(reme): add client backend validation with error raising - Check if client_cls is None before instantiation - Raise ValueError with descriptive message for unknown backends - Provide clear error feedback for invalid backend configurations * fix(components): move directory creation to start method - Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index - Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph - Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store - Ensures directory creation happens after component initialization - Prevents potential issues with path creation during object construction * fix(steps): replace assertions with runtime errors for app_context validation - Replace assert statements with explicit RuntimeError exceptions when app_context is None - Add descriptive error messages for better debugging when resolving components - Replace assert in resolve_component method with proper exception handling - Replace assert in get_file_parser method with proper exception handling - Maintain same functionality while improving error reporting clarity * refactor(file_io): split file IO utilities into modular components - Move daily note helpers to separate _daily_index module - Extract path validation and resolution to new _path module - Remove unused code and imports from _file_io module - Update import statements across affected modules - Introduce WikilinkHandler utility for link parsing - Replace regex-based link extraction with WikilinkHandler - Add integration JSONL files to gitignore - Consolidate file locking mechanism in _file_io module * style(formatter): fix spacing issues in file IO and chunked file parser - Fixed whitespace around colon in slice notation in file_io.py - Corrected spacing around colon in slice notation in chunked_file_parser.py - Applied consistent formatting for array slicing operations - Improved code readability by standardizing space placement in ranges * refactor(steps): replace property-based component resolution with Ref descriptor - Introduce Ref descriptor class for lazy component dependency resolution - Replace _resolve method and individual properties with Ref descriptors - Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes - Remove legacy property methods and resolve logic from BaseStep - Add cache clearing mechanism for Ref values during step calls - Update UpdateCatalogStep to use Ref instead of property-based resolution |
||
|
jinliyl
|
c4ca617992
|
refactor(evolve): consolidate auto memory planner and writer into single step (#267)
* refactor(evolve): consolidate auto memory planner and writer into single step - Removed separate AutoMemoryPlannerStep and AutoMemoryWriterStep classes - Combined functionality into new AutoMemoryStep class in auto_memory.py - Migrated prompt templates from separate YAML files to unified auto_memory.yaml - Updated module imports to reference new consolidated step - Simplified memory recording process using single ReAct agent instead of two-stage planning/writing - Maintained same input/output contract with messages, session_id, and memory_hint parameters - Preserved all original functionality for creating/updating daily notes with conversation facts * fix(daily): update empty session_id handling to create day-level file - Changed test to verify empty session_id creates day-level file daily/<date>.md - Updated assertion to check response success instead of rejection - Modified metadata verification to include path, session_id and created status - Added file existence check for the generated daily markdown file - Updated test name and print statement to reflect new behavior - Fixed test registration to use updated function name |
||
|
jinliyl
|
9ee2f0f7ab
|
refactor(daily): replace slug with session_id for daily note identification (#266)
- Rename slug parameter to session_id across daily note operations - Update validation function from validate_slug to validate_session_id - Change data structure keys from slug to session_id in note objects - Modify file paths to use session_id instead of slug in daily folder - Update documentation and comments to reflect session_id terminology - Adjust test cases to use session_id parameter instead of slug - Change default frontmatter to include empty description field - Update configuration files to use session_id parameter name - Modify scan_notes function to return session_id instead of slug |
||
|
jinliyl
|
ef22bfb071
|
refactor(evolve): replace ReActAgent with FlexReActAgent to allow structured output (#265)
- Create FlexReActAgent subclass that overrides _reasoning method to handle tool_choice parameter - Modify auto_memory_planner to use FlexReActAgent instead of ReActAgent - Update base_step.py to accept additional kwargs in add_as_tool method - Change run_job function to merge kwargs properly when calling jobs - Remove redundant imports and constants from file_io.py - Simplify _render_notes_block and rename _replace_or_append_notes to _rebuild_body - Update method calls to use new function names in file_io operations |
||
|
jinliyl
|
3cb2579ff7
|
refactor(steps): update auto-memory (#263)
* refactor(steps): update naming conventions in components and configuration Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates. Key changes: - Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations - Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`) - Added new step definitions such as `submit_slug_updates` and `auto_memory` - Updated relevant documentation - Modified log output format for improved readability * refactor(evolve): Refactor the auto-memory module and update related configurations - Remove the old slug update commit step file - Add new auto-memory planner and writer steps - Update __init__.py to export the new step classes - Modify the auto_memory configuration structure in default.yaml - Update the slug field description for clearer explanation of its purpose * up * up * refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit` Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists. Specific changes include: - Updating workflow configurations to align with the new test directory structure - Modifying assertions across multiple test methods to make them more flexible and maintainable - Cleaning up and optimizing parts of the test code structure This is a comprehensive test refactoring effort aimed at improving test readability and robustness. * Refactor(steps): Update memory writing logic and optimize JSON schema structure Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`. Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format. Also corrected grammatical issues in the related documentation. * Fix: Improve frontend data parsing error handling and update test files Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file. |
||
|
imrewce
|
2ed2e89e24
|
port orthogonal steps (#262)
* feat(file_io): port orthogonal crud_steps features onto upstream restructure * refactor(file_io): expose with_neighbors/max_neighbors_per_direction/max_bytes as step kwargs (not LLM params) Match search_step's convention: tuning knobs that are config-like (not part of the LLM-facing schema) live in the yaml steps: block and are read via self.kwargs.get(...) — not exposed under parameters.properties. Also simplify the write step metadata field description. |
||
|
Sen Huang
|
8c48798164
|
feat(jobs): add digester step for knowledge distillation from daily notes (#261)
* feat(jobs): add digester step for knowledge distillation from daily notes * feat: add initial implementation * refactor(jobs): remove unnecessary blank lines in digester and synchronizer * feat: add initial implementation |
||
|
jinliyl
|
a4efc0f776
|
refactor(reme4): restructure steps packages (#258)
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题 修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。 * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * refactor(steps): Rename and adjust indexing step logic - Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py` - Update implementation details of `ScanChangesStep` and `ClearAndScanStep` - Modify the scheduling mechanism in `WatchChangesStep` - Adjust step registration and parameter configuration in config files - Update related tests to align with the new interface changes * up * feat(daily): replace daily CRUD operations with slug provisioning approach * refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore * up * up * up * up --------- Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com> |
||
|
Sen Huang
|
83bfddb4a4
|
refactor(config): streamline job descriptions and parameter docs (#257)
* refactor(config): streamline job descriptions and parameter docs - Simplify descriptions for search, traverse, list, read, stat, frontmatter:read, write, edit, append, frontmatter:update, frontmatter:delete, move, delete, upload, upload_resource, and download jobs - Shorten parameter descriptions to be more concise - Maintain essential information while reducing verbosity refactor(steps): rename daily steps and consolidate functionality - Rename daily_resolve_step to daily_read_step - Rename daily_create_step to daily_write_step - Update __init__.py imports to reflect new step names - Consolidate daily operations documentation refactor(daily): extract helper functions and improve structure - Rename _day_index.py to _daily_io.py - Extract validate_slug function for Windows-safe filename validation - Move scan_notes function to public interface - Add comprehensive docstrings explaining slug validation and day-index rebuild concerns feat(daily): decouple list operation from index refresh - Remove automatic day index refresh from daily_list_step - Change daily_list_step to pure read operation with no side effects - Sort notes by slug for stable output - Update documentation to clarify read/write separation refactor(daily): remove deprecated create step - Remove unused daily/create.py module - Simplify daily operations to focus on CRUD patterns * refactor(daily): replace module imports with explicit step class imports |
||
|
Sen Huang
|
bb354cc580
|
refactor(steps): reorganize step modules and remove demo steps (#255)
* feat(config): add comprehensive job definitions for vault operations - Add utility jobs like version, search, traverse, list, read, stat - Include file operations like move, delete, upload, download - Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex - Update descriptions to reflect vault-based operations instead of working_dir - Add proper section headers and documentation for each job category refactor(steps): reorganize step modules and remove demo steps - Move steps into categorized packages: common, crud, frontmatter, daily, jobs - Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2) - Add new steps: InitStep for vault initialization, TraverseStep for graph traversal - Update __init__.py to auto-import all step modules - Organize imports by functionality (common, CRUD operations, frontmatter, daily) feat(vault): implement vault-centric file operations and configuration - Change default config to use vault_dir instead of working_dir - Add environment variable support for embedding configuration - Implement file watcher with lite backend for daily/digest directories - Update search step to use 'name' instead of 'title' from frontmatter - Create ResourceEntry schema for tracking uploaded assets docs(steps): add comprehensive documentation for all step categories - Document file-I/O split by blast radius (crud vs frontmatter packages) - Add detailed descriptions for each step category and functionality - Explain the purpose and usage patterns for different types of file operations - Provide clear parameter documentation for all new job configurations * fix(config): correct vault directory path and remove unused job configurations - Fix vault_dir from 'vaultd' to 'vault' in default configuration - Remove deprecated traverse and list job configurations - Remove unused tag tooling configurations - Remove background watch_file job configuration refactor(steps): remove unused jobs module import - Comment out jobs module import in steps/__init__.py - This removes unused synchronizer and digester step registrations refactor(tests): update import path and add pylint directive - Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta - Add pylint disable directive for unused argument in test datetime mocks * efactor(steps): remove unused modules from __all__ - Remove "background" module from __all__ list - Remove "jobs" module from __all__ list - These modules were no longer being used in the steps package * feat(config): update vault directory structure and remove file watcher - Change vault_dir reference from ./vault to ./vault in CLI example - Add daily_dir, digest_dir, and resource_dir configuration options - Remove file_watcher component configuration as it's no longer needed - Update comment to reflect correct module name (reme4vault) refactor(steps): add background step and remove deprecated init step - Import and register background step module - Remove deprecated InitStep from common steps - Update __all__ export list to include background step refactor(reindex): improve reindex step to scan vault directly - Update docstring to reflect vault scanning instead of watcher sync - Replace file watcher stop/start logic with direct vault path walking - Add support for suffix filtering during reindex operation - Use index_changes job to process found files refactor(wikilink_utils): enhance inbound source lookup with link scope - Import LinkScopeEnum for proper type handling - Update get_inlinks call to use ALL scope for virtual targets - Improve documentation for reverse-index lookup behavior test(refactor): clean up test suite removing deprecated functionality - Remove test_init_job and test_demo_job unit tests - Update help job assertion to check for literal command format - Change test directory from .reme to vault in CRUD tests - Remove init and demo job calls from integration test BREAKING CHANGE: Removes file_watcher component and init step * style(steps): fix import formatting in __init__.py Add proper spacing in the background module import statement to maintain consistent code style and readability. * refactor(config): change default vault directory from vault to .reme Default dev config now points vault_dir at ./.reme so `python -m reme4 start` can be run from the repo root and exercise the full atomic-tool surface against the seeded test data. BREAKING CHANGE: The default vault directory has been changed from 'vault' to '.reme' in the configuration. * docs(reme4_report): fix markdown formatting and remove extra content * refactor(file_parser): delegate wikilink extraction to WikilinkHandler * fix(search): handle empty query case gracefully - Replace assertion with conditional check for empty query - Set response success to false when query is empty - Return error message instead of throwing assertion error - Maintain existing validation for other parameters |
||
|
Sen Huang
|
7d0bec60be
|
feat: rename working_dir to vault_dir and update documentation (#254)
* feat: rename working_dir to vault_dir and update documentation - Rename working_dir to vault_dir across the application - Update documentation to reflect vault_dir instead of working_dir - Change FileFrontMatter title field to name field - Update .gitignore to include vault directory - Modify file path descriptions to reference vault instead of working_dir - Update related configuration and property names accordingly * refactor(steps): rename working_path to vault_path in CRUD operations - Rename parameter from `working_path` to `vault_path` in `resolve_path` function - Update all usages in append, edit, read, and write steps to use `self.vault_path` - Update documentation comments to reflect the new parameter name - Update docstring in read.py to mention `vault_dir` instead of `vault` test(chunked_file_parser): update frontmatter field from title to name - Change frontmatter field from `title` to `name` in test cases - Update comment in background steps test to reference `vault_path` instead of `working_path` * refactor(schema): remove unused ResourceEntry import * feat(file_graph): add link scope filtering to get_inlinks/get_outlinks * feat(file-store): add scope parameter to link methods |
||
|
jinliyl
|
24cff10d46
|
feat(file_store): add FAISS-backed local file store implementation (#253)
* feat(file_store): add FAISS-backed local file store implementation - Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP - Implement FAISS index persistence with binary format and JSON id-map sidecar - Add automatic index rebuilding when sidecar files are missing or corrupted - Support tombstone mechanism for efficient deletion and compaction - Register 'faiss' component type in the registry system - Add faiss-cpu dependency requirement to pyproject.toml - Update configuration schema to use simplified parameter structure - Enhance search step to support parameter override from runtime context - Add comprehensive unit tests for FAISS store functionality - Implement fallback to parent methods for basic CRUD operations * refactor(search): simplify parameter retrieval logic - Removed _param method that checked context and kwargs - Directly use self.kwargs.get for all parameter retrievals - Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction - Reduced code complexity by eliminating redundant context checking logic |
||
|
jinliyl
|
ee94d3ec8b
|
docs(reme4): update report with detailed architecture sections (#252)
- Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context |
||
|
诸岳
|
0dff85b9b5
|
feat(store): seekdb file and vector stores via pyseekdb (embedded + remote) (#207)
* feat(seekdb): add Seekdb file and vector stores with pyseekdb>=1.2.0 * refactor(seekdb): add pyseekdb_conn and remote-only host/port config * refactor(embedding): remove env fallbacks from BaseEmbeddingModel; pass credentials in tests * refactor(seekdb): drop tenant from client kwargs; default database test and empty password * fix(deps): gate pyseekdb to Python >=3.11 for CI 3.10 compatibility * fix(seekdb): satisfy pre-commit pylint and formatting for seekdb stores |
||
|
jinliyl
|
71e42dbad0
|
refactor(reme4): replace file_watcher component with background steps pipeline (#251)
* up * up * up * up * up * up * up * up * up * up * up * feat(config): add daily_dir configuration and background job logging - Added daily_dir setting with default value 'memory' to config - Implemented logging for background job startup events - Enhanced component start logic to handle background backend type - Updated default YAML configuration structure * refactor(file_parser): replace _get_relative_path with to_vault_relative method - Remove redundant working_dir property from base file parser - Add to_vault_relative method to base component for path resolution - Update bare_file_parser to use new to_vault_relative method - Update default_file_parser to use new to_vault_relative method - Update linked_file_parser to use new to_vault_relative method - Make working_path absolute in base_component and steps - Simplify index_changes step by removing redundant base variable - Consolidate path relative logic in single shared method * docs(reme4): update report with detailed architecture sections - Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context |
||
|
imrewce
|
3285934f34
|
create/ append/ edit steps (#249)
* feat(core): adding create append edit steps for md crud * fix(core): changing frontmatter args scope for create step * fix(core): chang write frontmatter schema; fix edit logic; fix passing name payload to http client * refractor(steps): changing step compatibility for non-md files |
||
|
Sen Huang
|
db35cf792c
|
refactor(mcp-client): update MCPClient to support runtime action (#250)
Update MCPClient to accept action parameter during method calls instead of requiring it during instantiation, making it consistent with the new client interface. fix(reme): update client usage patterns Update all client usages to pass action parameter during method calls instead of during client instantiation. |
||
|
jinliyl
|
0c9b8ca852
|
doc[reme4] (#244)
* up * up * up * up * up * up * up * up * up * feat: implementation of the read step for reme (markdown) (#245) * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix(core): correct tool results directory naming (#246) - Changed directory name from 'tool_result' to 'tool_results' in documentation - Updated path variable assignment to use correct plural form 'tool_results' - Ensured consistent directory naming throughout initialization logic * fix: recent unittest inconsistency (#248) * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix: fix test inconsistency * up --------- Co-authored-by: imrewce <wce@pku.edu.cn> |
||
|
imrewce
|
e8592fc930
|
fix: recent unittest inconsistency (#248)
* feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix: fix test inconsistency |
||
|
jinliyl
|
40feaa9150
|
fix(core): correct tool results directory naming (#246)
- Changed directory name from 'tool_result' to 'tool_results' in documentation - Updated path variable assignment to use correct plural form 'tool_results' - Ensured consistent directory naming throughout initialization logic |
||
|
imrewce
|
bee2648ad1
|
feat: implementation of the read step for reme (markdown) (#245)
* feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception |
||
|
jinliyl
|
b001c06086
|
Dev/connect as tool (#243)
* refactor(steps): Add job management methods and support registering them as tools Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class. * fix doc * chore(pyproject.toml): Update dependency versions and adjust package configuration Bump agentscope version to 1.0.19 and reorganize the core dependency configuration structure. |
||
|
Joshua
|
0e5a9f5034
|
fix(reme_light): dedupe default watch paths on case-insensitive filesystems (#234)
* fix(reme_light): dedupe default watch paths on case-insensitive filesystems On Windows NTFS and macOS HFS+, ``MEMORY.md`` and ``memory.md`` resolve to the same physical file. ``ReMeLight.__init__`` hardcoded both spellings in the default ``watch_paths`` list, so the memory markdown file was indexed twice on those filesystems, wasting embedding calls and producing duplicate search hits. Dedupe the default candidate list using ``os.path.normcase`` as the comparison key. On case-sensitive filesystems normcase is the identity function, so both spellings continue to be watched there. The original path strings are preserved, the caller-supplied ``watch_paths`` path is untouched, and only the built-in fallback is affected. Fixes #228 * refactor(reme_light): simplify watch path dedup via existence check Replace the os.path.normcase-based dedup loop with a direct exists() check that picks one of MEMORY.md / memory.md. On case-insensitive filesystems both spellings resolve to the same file so exists() returns true for both, naturally avoiding a duplicate watch — including on macOS where os.path.normcase is the identity function and the previous approach silently did nothing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
jinliyl
|
68bd95b494
|
refactor(steps): Add job management methods and support registering t… (#242)
* refactor(steps): Add job management methods and support registering them as tools Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class. * fix doc |
||
|
Sen Huang
|
357415dd49
|
feat: add Neo4j file graph support and markdown parser with wikilink extraction (#240)
* feat: add Neo4j file graph support and markdown parser with wikilink extraction - Add Neo4jFileGraph implementation for property-graph storage with virtual/real node handling and link management - Introduce LinkedFileParser for markdown files with frontmatter, wikilink graph extraction, and full-skeleton chunking - Update pyproject.toml to include pyyaml, mistletoe, and neo4j dependencies - Modify .gitignore to exclude /vault and structure.md - Change reme CLI entry point from reme_ai.main to remecli.reme - Register new neo4j and md components in respective registries * refactor(file-graph): add chunk_ids support to Neo4jFileGraph Add chunk_ids field to File node properties in Neo4jFileGraph to enable better content chunk tracking and management. BREAKING CHANGE: File node schema now includes chunk_ids property which may affect existing integrations. feat(parser): implement wikilink resolution logic Move path resolution logic from utils/path_resolver to linked_file_parser module and enhance wikilink resolution with folder-note rule support and improved error handling. fix(tests): update test assertions and variable names Update test cases to reflect changes in data structures and variable naming conventions across various components. chore(config): update package entry point reference Change reme CLI entry point from remecli.reme:main to reme_ai.reme:main in pyproject.toml. refactor(utils): remove deprecated path_resolver module Remove the old path_resolver utility module as its functionality has been moved to linked_file_parser. docs(file-graph): update Neo4jFileGraph documentation Update class docstrings and comments to reflect new chunk_ids property and other structural changes. style(formatting): adjust code formatting and line breaks Minor formatting improvements including line length optimization and consistent spacing adjustments throughout the codebase. * fix(pyproject.toml): correct entry point for reme command Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main" to fix the module reference for the reme command in project scripts. |
||
|
jinliyl
|
fdc36a22bc
|
docs(cli): add comprehensive CLI commands documentation (#237)
* docs(cli): add comprehensive CLI commands documentation - Document CLI entry point and argument parsing mechanism - Add detailed command reference with parameters and behaviors - Include usage examples for common operations like start, search, and reindex - Describe backend options and service configuration overrides - Explain local vs server-side command execution patterns - Provide table format documentation for all available actions * docs(reme_design): update CLI command documentation with detailed action descriptions - Rename section from "CLI 指令" to "基础Job" and add author attribution - Add comprehensive table documenting all available actions with parameters and behaviors - Include detailed explanations for input/output parameters, defaults, and internal workflows - Update example usage commands with proper parameter passing syntax - Add metadata information for each action including health checks and component details - Clarify the difference between local actions and server-forwarded actions - Document the new list action that intercepts at client side without forwarding to server |
||
|
jinliyl
|
e411eeb4c0
|
dev/reme4 init merge (#236) | ||
|
jinli.yl
|
20b37414cb |
fix(file-watcher): enable force polling for file watcher
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
- Set force_polling=True in awatch to improve file detection reliability - Bump version from 0.3.1.8 to 0.3.1.9 |
||
|
Aqil Aziz
|
ccadf1d3f9
|
fix(file_watcher): reset stop event on restart (#233) | ||
|
yangtiancheng-ali
|
d72f5fc581
|
feat(vector_store): add Hologres vector store implementation (#226)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
|
||
|
lichen2015
|
f42cf60706
|
add zvec vector/file store (#218) | ||
|
Zhouwk
|
72eabfa858
|
fix(user profile): update locomo benchmark and update vector based profile code (#225)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
* feat(reme): 添加配置选项以启用或禁用个人资料功能 - 在 ReMe 初始化方法中添加 enable_profile 参数，默认值为 True - 根据 enable_profile 设置决定是否创建 profile 目录和设置 profile_dir - 在 PersonalSummarizer 中根据 enable_profile 条件性地添加个人资料相关工具 - 在 PersonalRetriever 中根据 enable_profile 条件性地添加 ReadAllProfiles 工具 - 修改 profile_path 属性以在禁用个人资料时返回 None - 修改 get_profile_handler 方法以在禁用个人资料时返回 None - 为 enable_profile 参数添加文档说明其用于云向量存储场景 * refactor(benchmark): 重构LongMemEval基准测试中的ReMe实例管理 - 移除未使用的shutil导入 - 将固定的ReMe实例改为每个问题创建独立实例以实现隔离 - 更新LLM配置名称从qwen3-max-think到qwen-max-t - 修改模型调用逻辑使用正确的model_name参数 - 添加qwen-flash和GPT-4o-mini等新模型配置 - 统一使用"User"作为用户名，通过集合名实现隔离 - 调整并发处理数从4降至1，批处理大小从10增至30 - 每个问题类型采样数从2增至4 - 添加异步上下文管理确保资源正确释放 * reformat 2 files * refactor(benchmark): 重构长记忆评估中的模型配置 - 将原有的 eval_model_name 替换为专门的 retrieve_model_name 用于检索操作 - 添加对 qwen-max 模型配置的支持 - 更新参数解析器以支持新的检索模型参数 - 修改最大并发数默认值从 1 提升到 4 - 调整样本数量默认值从 4 减少到 1 - 统一模型参数命名规范，区分摘要、检索和评估模型 - 优化内存处理器初始化逻辑，支持独立的检索模型配置 * fix(benchmark): 移除数据路径默认值并设为必填参数 - 将LongMemEval评估脚本中的data_path参数改为必需参数 - 将HaluMem评估脚本中的data_path参数改为必需参数 - 删除了硬编码的默认文件路径配置 - 强制用户显式指定数据集文件路径以避免路径错误 * Update __init__.py * Update __init__.py * fix(benchmark): 修复ReMe评估中的模型配置和空值处理问题 - 移除了retrieve_memory调用中不需要的llm_config_name参数 - 修复了长字符串打印的换行格式问题 - 添加了eval_result为空时的初始化处理 - 在accuracy评估中加入了eval_model_name参数传递 * style(benchmark): 格式化模型名称打印输出 - 移除了多行字符串中的换行符和多余空格 - 将模型名称信息合并为单行连续显示 - 保持了原有的打印格式和信息完整性 * docs(readme): 更新文档添加实验结果表格 - 在英文版 README 中添加 🧪 Experiments 章节 - 添加 LoCoMo 和 HaluMem 两个基准测试的结果表格 - 在中文版 README_ZH 中添加 🧪 实验 章节 - 添加 LoCoMo 和 HaluMem 测试集的实验配置说明 - 添加完整的实验数据对比表格和评估协议说明 * docs(readme): 更新文档中的内存系统链接 - 为基于文件的记忆系统添加锚点链接 - 为基于向量库的记忆系统添加锚点链接 - 修复英文文档中的链接格式 - 修复中文文档中的链接格式和空行问题 * docs(readme): update experimental results section in documentation - Remove outdated experimental data placeholder "Coming soon..." - Add complete evaluation results for LoCoMo and HaluMem benchmarks - Include detailed performance metrics tables for all memory methods - Update experimental settings description with ReMe backbone details - Align evaluation protocol information with LLM-as-a-Judge approach - Maintain consistent formatting between English and Chinese documentation * docs(benchmark): add quick start guides for halumem and longmemeval experiments - Created HaluMem experiment quick start guide with ReMe integration setup - Added detailed steps for installing ReMe environment using conda - Included repository cloning instructions for HaluMem benchmark - Provided complete command examples for running HaluMem experiments - Created LongMeMEval quick start guide with data download procedures - Added wget commands for downloading cleaned dataset files - Included evaluation script instructions for computing experiment statistics - Documented parameter configurations for different model types and batch sizes * docs(longmemeval): update quickstart guide documentation - Changed project name from Halumem to Longmemeval in title - Updated description to reference Longmemeval experiments instead of Halumem - Maintained existing ReMe integration instructions unchanged * chore(logger): add test comment to logger configuration - Added test comment in logger utility function - Removed duplicate log handling by keeping the remove() call * chore(logger): add test comment to logger configuration - Added test comment in logger utility function - Removed duplicate log handling by keeping the remove() call * feat(core): add file logging capability to application - Added log_to_file parameter to Application class constructor - Integrated log_to_file option in logger initialization - Updated ServiceContext to support file logging configuration - Modified init_logger function to conditionally enable file logging - Added log_to_file field to ServiceConfig schema - Updated ReMe class to include file logging option - Wrapped file logging setup in conditional check to prevent unnecessary operations * docs(benchmark): update HaluMem quickstart guide with dataset download instructions - Replace repository cloning with direct dataset download using curl - Add commands to download HaluMem-Medium.jsonl and HaluMem-Long.jsonl files - Include both official Hugging Face and mirror download sources - Update data path reference from nested directory to local data folder - Add dataset page link and mirror usage instructions for mainland China access * feat(memory): add profile retrieval tool and refactor profile management - Introduce RetrieveProfile tool for fetching specific user profiles - Refactor ProfileHandler to support both filesystem and vector backends - Add async methods to ProfileHandler with synchronous fallbacks - Update PersonalRetriever to support two-stage profile and memory retrieval - Enhance PersonalSummarizer with improved tool partitioning logic - Add profile_backend, profile_store_name, and profile_max_capacity configuration options - Replace direct ProfileHandler imports with get_profile_handler method - Implement profile search functionality with dedicated prompts and workflows - Add FileProfileBackend and VectorProfileBackend implementations - Update base memory tool with new profile configuration parameters * feat(profile): add custom profile collection name support - Add profile_collection_name parameter to Application constructor - Allow custom database collection name for vector profiles instead of default suffix - Update profile vector store configuration logic to use custom collection name - Modify _ensure_profile_vector_store_config to handle custom collection names - Update docstring with detailed parameter descriptions for profile configuration options * test(history): add single history id acceptance test for multiple mode - Add test case to verify multiple-mode history lookup accepts a single history_id string - Create FakeVectorStore stub with minimal implementation for ReadHistory tests - Return requested history node from vector store mock - Initialize ReadHistory tool with multiple mode enabled - Add pylint disable comment for protected access to vector store property * refactor(memory): update profile handler and vector tools with improved formatting and error handling - Add module docstring to profiles/__init__.py - Add pylint disable comments for no-name-in-module and missing-function-docstring - Format long error message in ProfileHandler.sync_run method for better readability - Reformat parameters in ProfileHandler.aadd method to separate lines - Update model_copy call in reme.py to span multiple lines for better readability - Format aadd_batch call in update_profile.py to span multiple lines * feat(profiles): add profile management system with file and vector storage backends - Add FileProfileBackend for filesystem-based profile persistence - Add VectorProfileBackend for vector store-based profile management - Create abstract BaseProfileBackend interface for profile operations - Implement ProfileVectorHandler for vector-backed profile storage - Add RetrieveProfile tool for semantic profile retrieval - Update eval_reme.py to use user_message_s2 for retriever prompt - Modify eval_reme.yaml to use {profiles} instead of {user_profile} - Implement complete CRUD operations for profile management - Add batch operations for efficient profile handling - Include search functionality with semantic matching capabilities - Add capacity limits and automatic cleanup for profile storage * docs(profiles): add comprehensive docstrings for profile backend and handler methods - Added documentation for get_all_sync, get_by_sync, delete_sync, delete_all_sync methods - Documented add_sync and add_batch_sync functionality with deduping behavior - Added docstrings for update_sync and search_sync operations - Updated ProfileHandler.format_node method with proper documentation - Refactored private _format_node to public format_node method - Added comprehensive documentation for profile vector handler operations - Documented _vector_profile_matches, _get_by_profile_id, _get_by_profile_key helper methods - Added docstrings for retrieve_profile functionality and formatting methods |
||
|
Zhouwk
|
e0d0e3e568
|
提供支持向量数据库的profile功能 (#221)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
* feat(reme): 添加配置选项以启用或禁用个人资料功能 - 在 ReMe 初始化方法中添加 enable_profile 参数，默认值为 True - 根据 enable_profile 设置决定是否创建 profile 目录和设置 profile_dir - 在 PersonalSummarizer 中根据 enable_profile 条件性地添加个人资料相关工具 - 在 PersonalRetriever 中根据 enable_profile 条件性地添加 ReadAllProfiles 工具 - 修改 profile_path 属性以在禁用个人资料时返回 None - 修改 get_profile_handler 方法以在禁用个人资料时返回 None - 为 enable_profile 参数添加文档说明其用于云向量存储场景 * refactor(benchmark): 重构LongMemEval基准测试中的ReMe实例管理 - 移除未使用的shutil导入 - 将固定的ReMe实例改为每个问题创建独立实例以实现隔离 - 更新LLM配置名称从qwen3-max-think到qwen-max-t - 修改模型调用逻辑使用正确的model_name参数 - 添加qwen-flash和GPT-4o-mini等新模型配置 - 统一使用"User"作为用户名，通过集合名实现隔离 - 调整并发处理数从4降至1，批处理大小从10增至30 - 每个问题类型采样数从2增至4 - 添加异步上下文管理确保资源正确释放 * reformat 2 files * refactor(benchmark): 重构长记忆评估中的模型配置 - 将原有的 eval_model_name 替换为专门的 retrieve_model_name 用于检索操作 - 添加对 qwen-max 模型配置的支持 - 更新参数解析器以支持新的检索模型参数 - 修改最大并发数默认值从 1 提升到 4 - 调整样本数量默认值从 4 减少到 1 - 统一模型参数命名规范，区分摘要、检索和评估模型 - 优化内存处理器初始化逻辑，支持独立的检索模型配置 * fix(benchmark): 移除数据路径默认值并设为必填参数 - 将LongMemEval评估脚本中的data_path参数改为必需参数 - 将HaluMem评估脚本中的data_path参数改为必需参数 - 删除了硬编码的默认文件路径配置 - 强制用户显式指定数据集文件路径以避免路径错误 * Update __init__.py * Update __init__.py * fix(benchmark): 修复ReMe评估中的模型配置和空值处理问题 - 移除了retrieve_memory调用中不需要的llm_config_name参数 - 修复了长字符串打印的换行格式问题 - 添加了eval_result为空时的初始化处理 - 在accuracy评估中加入了eval_model_name参数传递 * style(benchmark): 格式化模型名称打印输出 - 移除了多行字符串中的换行符和多余空格 - 将模型名称信息合并为单行连续显示 - 保持了原有的打印格式和信息完整性 * docs(readme): 更新文档添加实验结果表格 - 在英文版 README 中添加 🧪 Experiments 章节 - 添加 LoCoMo 和 HaluMem 两个基准测试的结果表格 - 在中文版 README_ZH 中添加 🧪 实验 章节 - 添加 LoCoMo 和 HaluMem 测试集的实验配置说明 - 添加完整的实验数据对比表格和评估协议说明 * docs(readme): 更新文档中的内存系统链接 - 为基于文件的记忆系统添加锚点链接 - 为基于向量库的记忆系统添加锚点链接 - 修复英文文档中的链接格式 - 修复中文文档中的链接格式和空行问题 * docs(readme): update experimental results section in documentation - Remove outdated experimental data placeholder "Coming soon..." - Add complete evaluation results for LoCoMo and HaluMem benchmarks - Include detailed performance metrics tables for all memory methods - Update experimental settings description with ReMe backbone details - Align evaluation protocol information with LLM-as-a-Judge approach - Maintain consistent formatting between English and Chinese documentation * docs(benchmark): add quick start guides for halumem and longmemeval experiments - Created HaluMem experiment quick start guide with ReMe integration setup - Added detailed steps for installing ReMe environment using conda - Included repository cloning instructions for HaluMem benchmark - Provided complete command examples for running HaluMem experiments - Created LongMeMEval quick start guide with data download procedures - Added wget commands for downloading cleaned dataset files - Included evaluation script instructions for computing experiment statistics - Documented parameter configurations for different model types and batch sizes * docs(longmemeval): update quickstart guide documentation - Changed project name from Halumem to Longmemeval in title - Updated description to reference Longmemeval experiments instead of Halumem - Maintained existing ReMe integration instructions unchanged * chore(logger): add test comment to logger configuration - Added test comment in logger utility function - Removed duplicate log handling by keeping the remove() call * chore(logger): add test comment to logger configuration - Added test comment in logger utility function - Removed duplicate log handling by keeping the remove() call * feat(core): add file logging capability to application - Added log_to_file parameter to Application class constructor - Integrated log_to_file option in logger initialization - Updated ServiceContext to support file logging configuration - Modified init_logger function to conditionally enable file logging - Added log_to_file field to ServiceConfig schema - Updated ReMe class to include file logging option - Wrapped file logging setup in conditional check to prevent unnecessary operations * docs(benchmark): update HaluMem quickstart guide with dataset download instructions - Replace repository cloning with direct dataset download using curl - Add commands to download HaluMem-Medium.jsonl and HaluMem-Long.jsonl files - Include both official Hugging Face and mirror download sources - Update data path reference from nested directory to local data folder - Add dataset page link and mirror usage instructions for mainland China access * feat(memory): add profile retrieval tool and refactor profile management - Introduce RetrieveProfile tool for fetching specific user profiles - Refactor ProfileHandler to support both filesystem and vector backends - Add async methods to ProfileHandler with synchronous fallbacks - Update PersonalRetriever to support two-stage profile and memory retrieval - Enhance PersonalSummarizer with improved tool partitioning logic - Add profile_backend, profile_store_name, and profile_max_capacity configuration options - Replace direct ProfileHandler imports with get_profile_handler method - Implement profile search functionality with dedicated prompts and workflows - Add FileProfileBackend and VectorProfileBackend implementations - Update base memory tool with new profile configuration parameters * feat(profile): add custom profile collection name support - Add profile_collection_name parameter to Application constructor - Allow custom database collection name for vector profiles instead of default suffix - Update profile vector store configuration logic to use custom collection name - Modify _ensure_profile_vector_store_config to handle custom collection names - Update docstring with detailed parameter descriptions for profile configuration options * test(history): add single history id acceptance test for multiple mode - Add test case to verify multiple-mode history lookup accepts a single history_id string - Create FakeVectorStore stub with minimal implementation for ReadHistory tests - Return requested history node from vector store mock - Initialize ReadHistory tool with multiple mode enabled - Add pylint disable comment for protected access to vector store property * refactor(memory): update profile handler and vector tools with improved formatting and error handling - Add module docstring to profiles/__init__.py - Add pylint disable comments for no-name-in-module and missing-function-docstring - Format long error message in ProfileHandler.sync_run method for better readability - Reformat parameters in ProfileHandler.aadd method to separate lines - Update model_copy call in reme.py to span multiple lines for better readability - Format aadd_batch call in update_profile.py to span multiple lines |
||
|
Zhouwk
|
625d184ca1
|
添加log_to_file的开关 (#205)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
|
||
|
jinliyl
|
9663ee3dbc
|
Update references from CoPaw to QwenPaw in README
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
|
||
|
Chojan Shang
|
f3d09aaa38
|
feat(vector_store): add OceanBase/seekdb vector store implementation (#201)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
* feat(vector_store): add OceanBase as a VectorStore * refactor(obvec): make it cleaner * docs: add obvec related info * refactor: minor update * refactor: clean code and pass lint * docs: remove unrelated edit * docs: minor update |