Commit graph

884 commits

Author SHA1 Message Date
huangsen
c9345455a9 feat(crud): enhance file_download with configurable local path and overwrite option
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Symmetric counterpart to file_upload: source is in the vault,
target is on the local filesystem.

path (source) accepts a short link or vault-relative path;
resolution goes through path_resolver.resolve_to_absolute.
Short-link ambiguity is surfaced as error="ambiguous" with the
candidate list — the download is **not** executed in that case.

local_path (target) is a plain filesystem path. **Optional** —
when empty, the file lands in a session-scoped temp dir (lazy,
auto-cleaned on process exit; each call gets its own subdirectory
so concurrent agents don't trample each other). overwrite=True
is the default since most download flows are intentional refreshes.

BREAKING CHANGE: renamed step from file_download to download
and updated method signature to accept local_path and overwrite parameters.

refactor(list): switch from file_store index to direct filesystem walk

Reads directly from the filesystem (Path.iterdir / Path.rglob),
**not** the file_store index. The store may lag behind disk during
indexing or after rapid mutations; for the most current view,
the on-disk walk is the source of truth.

BREAKING CHANGE: renamed step from file_list to list and removed
dependency on file_store.graph for enumeration.

feat(property): restrict property operations to markdown files only

Property steps now operate only on .md files; non-markdown targets
get error="not markdown" and the call is **not** executed.

BREAKING CHANGE: property steps (read, update, delete) now validate
file extension and reject non-markdown files.

refactor(stat): rename file_stat to stat

BREAKING CHANGE: step name changed from file_stat to stat.

refactor(upload): rename file_upload to upload

BREAKING CHANGE: step name changed from file_upload to upload.

feat(traverse): support multiple seeds in graph traversal

Allow path parameter to accept either a single seed (str) or
a list of seeds for batch traversal operations.
2026-05-15 16:51:08 +08:00
huangsen
2e8a27ec5f refactor(parser): rename wikilink_resolver to link_parser module
BREAKING CHANGE: renamed `utils.wikilink_resolver` to `utils.link_parser`
2026-05-15 16:19:18 +08:00
huangsen
1852d05f7b feat(file_graph): add optional paths parameter to get_nodes method
Add support for `paths=None` in `get_nodes` method to return all real
nodes as the graph's "scan everything" entry point. When `paths` is
explicitly passed as an empty list, returns empty list. Virtual
placeholders are filtered out in both cases.

BREAKING CHANGE: The `get_nodes` method signature changed from
requiring a list of paths to accepting an optional list or None.

refactor(memory): move runtime_response imports to steps package

Move runtime_response imports from memory package to steps package to
improve code organization and reduce circular dependencies.

refactor(config): update obsidian configuration table structure

Update the obsidian configuration markdown table to improve clarity
and fix incorrect section headings. Change 'stat/list' to 'file' and
'return specific tag info' to 'tags' section with proper commands.

chore(memory): remove deprecated toolkit modules

Remove deprecated agent_toolkit.py, file_toolkit.py, and
graph_toolkit.py modules as functionality has been moved to steps
package.

feat(steps): add new CRUD file operations

Add new steps for file operations including file_download and
file_list operations with proper path resolution and filtering
capabilities.
2026-05-15 14:55:08 +08:00
huangsen
30971af5ec refactor(memory): restructure tool surface architecture
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
BREAKING CHANGE: Split monolithic agent_toolkit into modular components:
- memory_toolkit.py: memory_get, memory_create, memory_update_body,
  memory_update_meta, memory_search + schema policy helpers
- file_toolkit.py: file_download, file_upload, file_delete, file_list,
  file_move + path resolution and session temp dir
- graph_toolkit.py: graph_traverse + adjacency BFS
- event_toolkit.py: event_open, event_complete (using memory schema gates)
- lint_toolkit.py: check_* atomic primitives remain separate

Agent toolkit now composes these modules instead of containing all logic.

- Remove sync job from EXPECTED_JOBS since sync.py is no longer imported
- Add event_open and event_complete jobs to test expectations
- Update module imports to reference new toolkit structure
- Maintain backward compatibility through composition layer
2026-05-15 12:15:17 +08:00
jinli.yl
f4a9fac67e up 2026-05-15 12:12:31 +08:00
jinli.yl
02a77305ed up 2026-05-15 11:01:42 +08:00
jinli.yl
92defa0949 up 2026-05-15 10:31:21 +08:00
jinli.yl
e713fcd804 up 2026-05-15 09:44:44 +08:00
jinli.yl
f16bdb0383 up 2026-05-15 01:34:55 +08:00
jinli.yl
76adc00552 up 2026-05-15 01:32:45 +08:00
jinli.yl
0391158e66 up 2026-05-15 00:48:08 +08:00
jinli.yl
7176ec8ece up 2026-05-15 00:43:27 +08:00
jinli.yl
f0c9ed0105 up 2026-05-15 00:30:23 +08:00
jinli.yl
d0b538fbc8 up 2026-05-14 20:51:14 +08:00
jinli.yl
c150f46065 up 2026-05-14 20:19:24 +08:00
huangsen
b72690d954 feat: add Neo4j backend and refactor file graph architecture
- add neo4j dependency to project requirements
- introduce NetworkXFileGraph to replace LocalFileGraph implementation
- rename local_file_graph.py to networkx_file_graph.py with updated
  component registration as 'networkx'
- remove pickle persistence logic from NetworkX backend, simplify
  initialization
- update Neo4jFileGraph to return FileLink objects instead of
  (FileNode, FileLink) tuples from get_inlinks/get_outlinks methods
- remove unused AsyncIterator import and adjust method signatures
- add BareFileParser for handling binary/attachment files without
  content parsing
- move wikilink resolution utilities to dedicated utility module
- refactor memory I/O to use file graph's link resolution methods
  directly
- remove link extraction utilities from schema module, consolidate
  in utils.wikilink_resolver
2026-05-14 19:46:18 +08:00
jinli.yl
d6c51e2173 refactor(file_graph): replace local file graph with pure Python implementation
- Rename local_file_graph.py to nx_file_graph.py and update class name to NxFileGraph
- Add proper import error handling for networkx dependency in NxFileGraph
- Create new LocalFileGraph implementation using pure Python dict-based storage
- Update base_file_graph to include abstract methods for node/link operations
- Modify upsert_nodes to handle multiple nodes and edge management
- Implement proper edge bookkeeping with inverse and pending edge tracking
- Add rebuild_links functionality to reconstruct graph relationships
- Update logging to show real vs virtual nodes and edge counts
- Maintain backward compatibility with existing graph persistence mechanisms
2026-05-14 18:01:48 +08:00
jinli.yl
e138987112 refactor(file_graph): replace local file graph with pure Python implementation
- Rename local_file_graph.py to nx_file_graph.py and update class name to NxFileGraph
- Add proper import error handling for networkx dependency in NxFileGraph
- Create new LocalFileGraph implementation using pure Python dict-based storage
- Update base_file_graph to include abstract methods for node/link operations
- Modify upsert_nodes to handle multiple nodes and edge management
- Implement proper edge bookkeeping with inverse and pending edge tracking
- Add rebuild_links functionality to reconstruct graph relationships
- Update logging to show real vs virtual nodes and edge counts
- Maintain backward compatibility with existing graph persistence mechanisms
2026-05-14 18:01:43 +08:00
jinli.yl
84b149d96a up 2026-05-14 14:37:28 +08:00
jinli.yl
c3c763a29b up
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
2026-05-14 12:19:33 +08:00
jinli.yl
cca6cce0e0 up 2026-05-14 12:17:51 +08:00
jinli.yl
d0aae68164 up 2026-05-14 11:37:25 +08:00
huangsen
d9114469e3 refactor(file_graph): remove unused iter_nodes abstract method
BREAKING CHANGE: Removed the iter_nodes abstract method from BaseFileGraph
and its implementations in LocalFileGraph and Neo4jFileGraph as it was
no longer being used in the codebase.
2026-05-14 11:36:39 +08:00
huangsen
6a5561f86a feat: add file_graph component and integrate with LinkedFileParser
- Add file_graph import to component registry
- Register FILE_GRAPH enum in ComponentEnum
- Implement BaseFileGraph integration in LinkedFileParser
- Replace FileEdge with FileLink for better semantic clarity
- Add lazy resolution of file_graph from app_context
- Update file watcher logging to reflect links instead of edges

refactor: streamline MCP transport layer architecture

- Remove redundant step shells from reme2/mcp/steps/
- Consolidate all @R.register components to reme2.memory package
- Update server.py to import reme2.memory directly
- Revise README.md to document new architecture
- Simplify module dependencies and import structure
2026-05-14 11:35:02 +08:00
jinli.yl
56b94c5952 up 2026-05-14 11:10:58 +08:00
jinli.yl
dacbf52637 up 2026-05-14 11:05:17 +08:00
jinli.yl
3a573d9122 up 2026-05-14 10:51:26 +08:00
jinli.yl
dce86f5b99 up 2026-05-14 10:51:14 +08:00
jinli.yl
ef0e11d237 up 2026-05-14 10:25:53 +08:00
jinli.yl
5f84ffc2f6 up 2026-05-14 10:05:41 +08:00
jinli.yl
5792221d3f up 2026-05-14 02:01:27 +08:00
huangsen
48137f7f3f refactor(file_parser): simplify chunk ID generation in LinkedFileParser
Remove path and line numbers from chunk ID calculation to improve
cache stability. The chunk identity now uses only the content hash
instead of including path::start::end coordinates.

This change removes the documentation about chunk identity format
and updates the hash_text call to use only the text content.
2026-05-13 16:30:53 +08:00
jinli.yl
81e8f417b9 up 2026-05-13 16:09:15 +08:00
huangsen
37f0037e6b feat(parser): add mistletoe dependency and refactor markdown parsing
Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.

The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
2026-05-13 16:00:36 +08:00
huangsen
51ec09d98f feat(parser): implement AST-based markdown chunking with full document TOC
- Replace legacy line-based chunking with AST tree approach that builds
  a complete document skeleton with content inlined under relevant
  sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
  (default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
  boundaries (code lines, table rows, list items) and prevents splits
  inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
  require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
  preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
  parse_wikilinks for consistency

BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
2026-05-13 14:32:00 +08:00
jinli.yl
90da850200 up 2026-05-13 14:25:52 +08:00
jinli.yl
5c28dd8be2 up
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
2026-05-13 12:22:59 +08:00
jinli.yl
703c43be23 up 2026-05-12 23:49:21 +08:00
jinli.yl
7cd2838c98 up 2026-05-12 20:47:43 +08:00
jinli.yl
ef0b74b3e7 up 2026-05-12 20:46:40 +08:00
jinli.yl
602622c9f6 up 2026-05-12 20:32:59 +08:00
jinli.yl
0b7825a8fd up 2026-05-12 20:28:49 +08:00
jinli.yl
4e089753b9 up 2026-05-12 16:39:35 +08:00
huangsen
701f8f6e3a refactor(file_store): simplify file store architecture and remove unused features
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Remove SqliteFileStore backend and simplify the base file store
interface. The file store now focuses on in-memory + JSONL persistence
for the (file → chunks) graph with reduced complexity.

BREAKING CHANGE: SqliteFileStore is removed, only LocalFileStore remains
available.
2026-05-11 20:57:10 +08:00
jinli.yl
93e356f899 up 2026-05-11 20:31:41 +08:00
huangsen
8465f6d06e ```
docs(protocol): add typed edge link protocol documentation

Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.

fix(memory): update path reference from vault_root to working_dir

Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.

refactor(components): remove edge_extractor module and simplify parsing

Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.

feat(parser): update parse method signature and simplify edge extraction

Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
2026-05-11 19:45:53 +08:00
jinli.yl
0adb7f231c up 2026-05-11 19:35:01 +08:00
jinli.yl
926b8b0b36 up 2026-05-11 18:23:39 +08:00
jinli.yl
7ff53d3acf up 2026-05-11 18:20:55 +08:00
jinli.yl
1265b7656b up 2026-05-11 17:54:24 +08:00