Commit graph

29 commits

Author SHA1 Message Date
jinli.yl
d28f4af5d0 up 2026-05-27 15:24:58 +08:00
jinli.yl
4f620324e5 up 2026-05-27 14:47:09 +08:00
jinli.yl
11b32e64de up 2026-05-27 09:55:53 +08:00
jinli.yl
b3031c6df1 up 2026-05-27 01:10:58 +08:00
jinli.yl
96043b0eb8 up 2026-05-26 19:49:17 +08:00
jinli.yl
aef40098bf up 2026-05-26 19:09:36 +08:00
jinli.yl
964137b474 up 2026-05-26 18:27:18 +08:00
jinli.yl
0c7310795f up 2026-05-26 17:41:39 +08:00
jinli.yl
4052ddbe6b up 2026-05-26 17:35:15 +08:00
jinli.yl
17919ebffa up 2026-05-26 17:26:37 +08:00
jinli.yl
18c0c84367 up 2026-05-26 16:58:52 +08:00
jinli.yl
71e8ec1b46 up 2026-05-26 16:23:42 +08:00
jinli.yl
9f90c448f1 up 2026-05-26 16:03:55 +08:00
jinli.yl
6ad02f0a8f up 2026-05-26 15:55:42 +08:00
jinli.yl
9a7877a3a4 up 2026-05-26 15:37:15 +08:00
jinli.yl
6915417a19 fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题
修复了在计算BM25相似度时对文档长度进行不正确归一化的bug,确保所有查询都能得到准确的相关性评分。
2026-05-26 00:13:50 +08:00
Sen Huang
83bfddb4a4
refactor(config): streamline job descriptions and parameter docs (#257)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.10 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
* refactor(config): streamline job descriptions and parameter docs

- Simplify descriptions for search, traverse, list, read, stat,
  frontmatter:read, write, edit, append, frontmatter:update,
  frontmatter:delete, move, delete, upload, upload_resource, and
  download jobs
- Shorten parameter descriptions to be more concise
- Maintain essential information while reducing verbosity

refactor(steps): rename daily steps and consolidate functionality

- Rename daily_resolve_step to daily_read_step
- Rename daily_create_step to daily_write_step
- Update __init__.py imports to reflect new step names
- Consolidate daily operations documentation

refactor(daily): extract helper functions and improve structure

- Rename _day_index.py to _daily_io.py
- Extract validate_slug function for Windows-safe filename validation
- Move scan_notes function to public interface
- Add comprehensive docstrings explaining slug validation and day-index
  rebuild concerns

feat(daily): decouple list operation from index refresh

- Remove automatic day index refresh from daily_list_step
- Change daily_list_step to pure read operation with no side effects
- Sort notes by slug for stable output
- Update documentation to clarify read/write separation

refactor(daily): remove deprecated create step

- Remove unused daily/create.py module
- Simplify daily operations to focus on CRUD patterns

* refactor(daily): replace module imports with explicit step class imports
2026-05-25 19:20:50 +08:00
Sen Huang
bb354cc580
refactor(steps): reorganize step modules and remove demo steps (#255)
* feat(config): add comprehensive job definitions for vault operations

- Add utility jobs like version, search, traverse, list, read, stat
- Include file operations like move, delete, upload, download
- Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex
- Update descriptions to reflect vault-based operations instead of working_dir
- Add proper section headers and documentation for each job category

refactor(steps): reorganize step modules and remove demo steps

- Move steps into categorized packages: common, crud, frontmatter, daily, jobs
- Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2)
- Add new steps: InitStep for vault initialization, TraverseStep for graph traversal
- Update __init__.py to auto-import all step modules
- Organize imports by functionality (common, CRUD operations, frontmatter, daily)

feat(vault): implement vault-centric file operations and configuration

- Change default config to use vault_dir instead of working_dir
- Add environment variable support for embedding configuration
- Implement file watcher with lite backend for daily/digest directories
- Update search step to use 'name' instead of 'title' from frontmatter
- Create ResourceEntry schema for tracking uploaded assets

docs(steps): add comprehensive documentation for all step categories

- Document file-I/O split by blast radius (crud vs frontmatter packages)
- Add detailed descriptions for each step category and functionality
- Explain the purpose and usage patterns for different types of file operations
- Provide clear parameter documentation for all new job configurations

* fix(config): correct vault directory path and remove unused job configurations

- Fix vault_dir from 'vaultd' to 'vault' in default configuration
- Remove deprecated traverse and list job configurations
- Remove unused tag tooling configurations
- Remove background watch_file job configuration

refactor(steps): remove unused jobs module import

- Comment out jobs module import in steps/__init__.py
- This removes unused synchronizer and digester step registrations

refactor(tests): update import path and add pylint directive

- Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta
- Add pylint disable directive for unused argument in test datetime mocks

* efactor(steps): remove unused modules from __all__

- Remove "background" module from __all__ list
- Remove "jobs" module from __all__ list
- These modules were no longer being used in the steps package

* feat(config): update vault directory structure and remove file watcher

- Change vault_dir reference from ./vault to ./vault in CLI example
- Add daily_dir, digest_dir, and resource_dir configuration options
- Remove file_watcher component configuration as it's no longer needed
- Update comment to reflect correct module name (reme4vault)

refactor(steps): add background step and remove deprecated init step

- Import and register background step module
- Remove deprecated InitStep from common steps
- Update __all__ export list to include background step

refactor(reindex): improve reindex step to scan vault directly

- Update docstring to reflect vault scanning instead of watcher sync
- Replace file watcher stop/start logic with direct vault path walking
- Add support for suffix filtering during reindex operation
- Use index_changes job to process found files

refactor(wikilink_utils): enhance inbound source lookup with link scope

- Import LinkScopeEnum for proper type handling
- Update get_inlinks call to use ALL scope for virtual targets
- Improve documentation for reverse-index lookup behavior

test(refactor): clean up test suite removing deprecated functionality

- Remove test_init_job and test_demo_job unit tests
- Update help job assertion to check for literal command format
- Change test directory from .reme to vault in CRUD tests
- Remove init and demo job calls from integration test

BREAKING CHANGE: Removes file_watcher component and init step

* style(steps): fix import formatting in __init__.py

Add proper spacing in the background module import statement
to maintain consistent code style and readability.

* refactor(config): change default vault directory from vault to .reme

Default dev config now points vault_dir at ./.reme so `python -m
reme4 start` can be run from the repo root and exercise the full
atomic-tool surface against the seeded test data.

BREAKING CHANGE: The default vault directory has been changed from
'vault' to '.reme' in the configuration.

* docs(reme4_report): fix markdown formatting and remove extra content

* refactor(file_parser): delegate wikilink extraction to WikilinkHandler

* fix(search): handle empty query case gracefully

- Replace assertion with conditional check for empty query
- Set response success to false when query is empty
- Return error message instead of throwing assertion error
- Maintain existing validation for other parameters
2026-05-25 17:52:51 +08:00
Sen Huang
7d0bec60be
feat: rename working_dir to vault_dir and update documentation (#254)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.10 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
* feat: rename working_dir to vault_dir and update documentation

- Rename working_dir to vault_dir across the application
- Update documentation to reflect vault_dir instead of working_dir
- Change FileFrontMatter title field to name field
- Update .gitignore to include vault directory
- Modify file path descriptions to reference vault instead of working_dir
- Update related configuration and property names accordingly

* refactor(steps): rename working_path to vault_path in CRUD operations

- Rename parameter from `working_path` to `vault_path` in `resolve_path` function
- Update all usages in append, edit, read, and write steps to use `self.vault_path`
- Update documentation comments to reflect the new parameter name
- Update docstring in read.py to mention `vault_dir` instead of `vault`

test(chunked_file_parser): update frontmatter field from title to name

- Change frontmatter field from `title` to `name` in test cases
- Update comment in background steps test to reference `vault_path` instead of `working_path`

* refactor(schema): remove unused ResourceEntry import

* feat(file_graph): add link scope filtering to get_inlinks/get_outlinks

* feat(file-store): add scope parameter to link methods
2026-05-22 18:10:29 +08:00
jinliyl
24cff10d46
feat(file_store): add FAISS-backed local file store implementation (#253)
* feat(file_store): add FAISS-backed local file store implementation

- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations

* refactor(search): simplify parameter retrieval logic

- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
2026-05-22 16:14:38 +08:00
jinliyl
ee94d3ec8b
docs(reme4): update report with detailed architecture sections (#252)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.10 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
2026-05-22 14:58:37 +08:00
jinliyl
71e42dbad0
refactor(reme4): replace file_watcher component with background steps pipeline (#251)
* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* feat(config): add daily_dir configuration and background job logging

- Added daily_dir setting with default value 'memory' to config
- Implemented logging for background job startup events
- Enhanced component start logic to handle background backend type
- Updated default YAML configuration structure

* refactor(file_parser): replace _get_relative_path with to_vault_relative method

- Remove redundant working_dir property from base file parser
- Add to_vault_relative method to base component for path resolution
- Update bare_file_parser to use new to_vault_relative method
- Update default_file_parser to use new to_vault_relative method
- Update linked_file_parser to use new to_vault_relative method
- Make working_path absolute in base_component and steps
- Simplify index_changes step by removing redundant base variable
- Consolidate path relative logic in single shared method

* docs(reme4): update report with detailed architecture sections

- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
2026-05-22 10:26:36 +08:00
imrewce
3285934f34
create/ append/ edit steps (#249)
* feat(core): adding create append edit steps for md crud

* fix(core): changing frontmatter args scope for create step

* fix(core): chang write frontmatter schema; fix edit logic; fix passing name payload to http client

* refractor(steps): changing step compatibility for non-md files
2026-05-21 15:57:37 +08:00
Sen Huang
db35cf792c
refactor(mcp-client): update MCPClient to support runtime action (#250)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.10 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Update MCPClient to accept action parameter during method calls
instead of requiring it during instantiation, making it consistent
with the new client interface.

fix(reme): update client usage patterns

Update all client usages to pass action parameter during method
calls instead of during client instantiation.
2026-05-21 11:09:35 +08:00
imrewce
e8592fc930
fix: recent unittest inconsistency (#248)
* feat: implementation of the read step for reme (markdown)

* fix(step): markdown read step fixing pr comments

* fix(step): more  fixes for pr comments

* further fix for better review adaptation

* accept absolute path

* fixing base job exception

* fix: fix test inconsistency
2026-05-20 09:58:15 +08:00
imrewce
bee2648ad1
feat: implementation of the read step for reme (markdown) (#245)
* feat: implementation of the read step for reme (markdown)

* fix(step): markdown read step fixing pr comments

* fix(step): more  fixes for pr comments

* further fix for better review adaptation

* accept absolute path

* fixing base job exception
2026-05-19 16:19:13 +08:00
jinliyl
68bd95b494
refactor(steps): Add job management methods and support registering t… (#242)
* refactor(steps): Add job management methods and support registering them as tools

Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class.

* fix doc
2026-05-18 15:00:02 +08:00
Sen Huang
357415dd49
feat: add Neo4j file graph support and markdown parser with wikilink extraction (#240)
* feat: add Neo4j file graph support and markdown parser with wikilink extraction

- Add Neo4jFileGraph implementation for property-graph storage with
  virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
  wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
  dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries

* refactor(file-graph): add chunk_ids support to Neo4jFileGraph

Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.

BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.

feat(parser): implement wikilink resolution logic

Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.

fix(tests): update test assertions and variable names

Update test cases to reflect changes in data structures and
variable naming conventions across various components.

chore(config): update package entry point reference

Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.

refactor(utils): remove deprecated path_resolver module

Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.

docs(file-graph): update Neo4jFileGraph documentation

Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.

style(formatting): adjust code formatting and line breaks

Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.

* fix(pyproject.toml): correct entry point for reme command

Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
2026-05-18 14:25:14 +08:00
jinliyl
e411eeb4c0
dev/reme4 init merge (#236) 2026-05-17 14:14:43 +08:00