Commit graph

878 commits

Author SHA1 Message Date
jinli.yl
92defa0949 up 2026-05-15 10:31:21 +08:00
jinli.yl
e713fcd804 up 2026-05-15 09:44:44 +08:00
jinli.yl
f16bdb0383 up 2026-05-15 01:34:55 +08:00
jinli.yl
76adc00552 up 2026-05-15 01:32:45 +08:00
jinli.yl
0391158e66 up 2026-05-15 00:48:08 +08:00
jinli.yl
7176ec8ece up 2026-05-15 00:43:27 +08:00
jinli.yl
f0c9ed0105 up 2026-05-15 00:30:23 +08:00
jinli.yl
d0b538fbc8 up 2026-05-14 20:51:14 +08:00
jinli.yl
c150f46065 up 2026-05-14 20:19:24 +08:00
huangsen
b72690d954 feat: add Neo4j backend and refactor file graph architecture
- add neo4j dependency to project requirements
- introduce NetworkXFileGraph to replace LocalFileGraph implementation
- rename local_file_graph.py to networkx_file_graph.py with updated
  component registration as 'networkx'
- remove pickle persistence logic from NetworkX backend, simplify
  initialization
- update Neo4jFileGraph to return FileLink objects instead of
  (FileNode, FileLink) tuples from get_inlinks/get_outlinks methods
- remove unused AsyncIterator import and adjust method signatures
- add BareFileParser for handling binary/attachment files without
  content parsing
- move wikilink resolution utilities to dedicated utility module
- refactor memory I/O to use file graph's link resolution methods
  directly
- remove link extraction utilities from schema module, consolidate
  in utils.wikilink_resolver
2026-05-14 19:46:18 +08:00
jinli.yl
d6c51e2173 refactor(file_graph): replace local file graph with pure Python implementation
- Rename local_file_graph.py to nx_file_graph.py and update class name to NxFileGraph
- Add proper import error handling for networkx dependency in NxFileGraph
- Create new LocalFileGraph implementation using pure Python dict-based storage
- Update base_file_graph to include abstract methods for node/link operations
- Modify upsert_nodes to handle multiple nodes and edge management
- Implement proper edge bookkeeping with inverse and pending edge tracking
- Add rebuild_links functionality to reconstruct graph relationships
- Update logging to show real vs virtual nodes and edge counts
- Maintain backward compatibility with existing graph persistence mechanisms
2026-05-14 18:01:48 +08:00
jinli.yl
e138987112 refactor(file_graph): replace local file graph with pure Python implementation
- Rename local_file_graph.py to nx_file_graph.py and update class name to NxFileGraph
- Add proper import error handling for networkx dependency in NxFileGraph
- Create new LocalFileGraph implementation using pure Python dict-based storage
- Update base_file_graph to include abstract methods for node/link operations
- Modify upsert_nodes to handle multiple nodes and edge management
- Implement proper edge bookkeeping with inverse and pending edge tracking
- Add rebuild_links functionality to reconstruct graph relationships
- Update logging to show real vs virtual nodes and edge counts
- Maintain backward compatibility with existing graph persistence mechanisms
2026-05-14 18:01:43 +08:00
jinli.yl
84b149d96a up 2026-05-14 14:37:28 +08:00
jinli.yl
c3c763a29b up
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
2026-05-14 12:19:33 +08:00
jinli.yl
cca6cce0e0 up 2026-05-14 12:17:51 +08:00
jinli.yl
d0aae68164 up 2026-05-14 11:37:25 +08:00
huangsen
d9114469e3 refactor(file_graph): remove unused iter_nodes abstract method
BREAKING CHANGE: Removed the iter_nodes abstract method from BaseFileGraph
and its implementations in LocalFileGraph and Neo4jFileGraph as it was
no longer being used in the codebase.
2026-05-14 11:36:39 +08:00
huangsen
6a5561f86a feat: add file_graph component and integrate with LinkedFileParser
- Add file_graph import to component registry
- Register FILE_GRAPH enum in ComponentEnum
- Implement BaseFileGraph integration in LinkedFileParser
- Replace FileEdge with FileLink for better semantic clarity
- Add lazy resolution of file_graph from app_context
- Update file watcher logging to reflect links instead of edges

refactor: streamline MCP transport layer architecture

- Remove redundant step shells from reme2/mcp/steps/
- Consolidate all @R.register components to reme2.memory package
- Update server.py to import reme2.memory directly
- Revise README.md to document new architecture
- Simplify module dependencies and import structure
2026-05-14 11:35:02 +08:00
jinli.yl
56b94c5952 up 2026-05-14 11:10:58 +08:00
jinli.yl
dacbf52637 up 2026-05-14 11:05:17 +08:00
jinli.yl
3a573d9122 up 2026-05-14 10:51:26 +08:00
jinli.yl
dce86f5b99 up 2026-05-14 10:51:14 +08:00
jinli.yl
ef0e11d237 up 2026-05-14 10:25:53 +08:00
jinli.yl
5f84ffc2f6 up 2026-05-14 10:05:41 +08:00
jinli.yl
5792221d3f up 2026-05-14 02:01:27 +08:00
huangsen
48137f7f3f refactor(file_parser): simplify chunk ID generation in LinkedFileParser
Remove path and line numbers from chunk ID calculation to improve
cache stability. The chunk identity now uses only the content hash
instead of including path::start::end coordinates.

This change removes the documentation about chunk identity format
and updates the hash_text call to use only the text content.
2026-05-13 16:30:53 +08:00
jinli.yl
81e8f417b9 up 2026-05-13 16:09:15 +08:00
huangsen
37f0037e6b feat(parser): add mistletoe dependency and refactor markdown parsing
Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.

The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
2026-05-13 16:00:36 +08:00
huangsen
51ec09d98f feat(parser): implement AST-based markdown chunking with full document TOC
- Replace legacy line-based chunking with AST tree approach that builds
  a complete document skeleton with content inlined under relevant
  sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
  (default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
  boundaries (code lines, table rows, list items) and prevents splits
  inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
  require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
  preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
  parse_wikilinks for consistency

BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
2026-05-13 14:32:00 +08:00
jinli.yl
90da850200 up 2026-05-13 14:25:52 +08:00
jinli.yl
5c28dd8be2 up
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
2026-05-13 12:22:59 +08:00
jinli.yl
703c43be23 up 2026-05-12 23:49:21 +08:00
jinli.yl
7cd2838c98 up 2026-05-12 20:47:43 +08:00
jinli.yl
ef0b74b3e7 up 2026-05-12 20:46:40 +08:00
jinli.yl
602622c9f6 up 2026-05-12 20:32:59 +08:00
jinli.yl
0b7825a8fd up 2026-05-12 20:28:49 +08:00
jinli.yl
4e089753b9 up 2026-05-12 16:39:35 +08:00
huangsen
701f8f6e3a refactor(file_store): simplify file store architecture and remove unused features
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Remove SqliteFileStore backend and simplify the base file store
interface. The file store now focuses on in-memory + JSONL persistence
for the (file → chunks) graph with reduced complexity.

BREAKING CHANGE: SqliteFileStore is removed, only LocalFileStore remains
available.
2026-05-11 20:57:10 +08:00
jinli.yl
93e356f899 up 2026-05-11 20:31:41 +08:00
huangsen
8465f6d06e ```
docs(protocol): add typed edge link protocol documentation

Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.

fix(memory): update path reference from vault_root to working_dir

Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.

refactor(components): remove edge_extractor module and simplify parsing

Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.

feat(parser): update parse method signature and simplify edge extraction

Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
2026-05-11 19:45:53 +08:00
jinli.yl
0adb7f231c up 2026-05-11 19:35:01 +08:00
jinli.yl
926b8b0b36 up 2026-05-11 18:23:39 +08:00
jinli.yl
7ff53d3acf up 2026-05-11 18:20:55 +08:00
jinli.yl
1265b7656b up 2026-05-11 17:54:24 +08:00
jinli.yl
662e1768dc up
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
2026-05-11 17:18:07 +08:00
huangsen
ad6368fdf0 ```
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
feat: add Claude Code marketplace plugin with service and expert tiers

Add comprehensive Claude Code marketplace plugin supporting two paradigms
for managing markdown vaults:

- reme-service: Service-tier plugin with high-level MCP tools
  (retrieve/remember/maintain) where reme2 internals handle R-M-W loop
- reme-expert: Expert-tier plugin where Claude Code agent runs R-M-W loop
  directly using raw memory_* primitives guided by reme protocol skill

Both plugins implement identical 4-phase work paradigm:
- Recall: Retrieve relevant context with ranking by relevance/proximity
- Log: Record event facts and raw materials via idempotent event-folder upsert
- Distill: Promote events to topic graph via R-M-W loop
- Maintain: Vault hygiene sweep with lint and decay operations

Include marketplace configuration, documentation, MCP server setup,
subagents (reme-distiller, reme-curator), hooks (PreCompact, SessionEnd,
Stop), and slash commands (/reme-distill, /reme-recall, /reme-clean).

Remove outdated plugin entries from gitignore.
```
2026-05-11 10:58:40 +08:00
huangsen
dd2de16481 ```
feat(file_watcher): add directory deletion support with descendant indexing

Add support for deleting entire directories and their indexed descendants
in the file watcher. Previously only individual file deletions were
handled properly. Now when a directory is deleted, the system finds all
indexed files beneath that directory path and removes them along with
their metadata and chunks.

The implementation includes:
- New `_descendant_indexed_paths` method to find all indexed files
  under a given directory path
- Updated `_on_deleted` method to process both the target path and
  all its indexed descendants
- Proper handling of symlinks and path resolution differences
- Enhanced logging to show directory deletion with child count

Also adds necessary os import for path operations.

refactor(config): restructure configuration profiles for clarity

Rename curated.yaml to remove outdated configuration file and
rename full.yaml to expert.yaml with updated documentation.
Add new service.yaml configuration profile that provides a
service-aligned MCP surface with three main tools:
- retrieve: graph-aware hybrid retrieval
- remember: single write entry point with log/distill modes
- maintain: vault hygiene sweep

The expert configuration now excludes the ingest tool since
cold-path operations are handled by external agents, and adds
memory_lint tool for structural issue detection.

Updated documentation to clarify the different configuration
profiles and their intended usage patterns.
```
2026-05-11 10:52:25 +08:00
huangsen
514bf35050 ```
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
docs: add ReMe2 architecture design documentation

- Add comprehensive design document (reme2.md) detailing the
  three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target

refactor: fix typo in personal retriever class

- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
  in PersonalRetriever.result assignment

chore: update gitignore with vault-related patterns

- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
2026-05-08 16:14:42 +08:00
huangsen
c2e43c398b ```
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
refactor(component): rename file_store to chunk_store and update interfaces

- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
2026-04-28 11:37:20 +08:00
jinli.yl
98632dfc24 feat(service): add MCP service to component service module
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
- Import MCPService from .mcp_service module
- Add MCPService to __all__ exports list
- Make MCPService available in component service package
2026-04-28 00:24:25 +08:00