mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-21 00:22:45 +00:00
Add mistletoe as a project dependency for enhanced markdown parsing capabilities. Refactor the LinkedFileParser to use a proper AST-based approach with MdNode tree structure, replacing the previous flat token processing method. The new implementation provides better handling of markdown elements including tables, code fences, lists, and headings, with improved chunking logic that maintains document structure in generated content segments. The changes include: - Add mistletoe dependency to pyproject.toml - Implement proper AST node representation with MdNode class - Create recursive chunking algorithm with TOC preservation - Add support for frontmatter extraction with FileFrontMatter schema - Optimize leaf node splitting with proper boundary detection - Include part numbering for split content pieces |
||
|---|---|---|
| .. | ||
| inspect_md_parser.py | ||
| test_file_edge.py | ||
| test_md_chunker.py | ||
| test_reme2.py | ||