mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-22 00:32:49 +00:00
- Replace legacy line-based chunking with AST tree approach that builds a complete document skeleton with content inlined under relevant sections - Add new chunking parameters: chunk_chars (default 2000) and embed_toc (default True) to control content size and TOC inclusion - Implement recursive chunking algorithm that respects structural boundaries (code lines, table rows, list items) and prevents splits inside blocks - Introduce part markers [Part X/N] for oversized leaf blocks that require splitting - Add CLI tool for inspecting parsed chunks and edges with options for preview and configuration - Refactor edge extraction to use FileEdge.from_text instead of parse_wikilinks for consistency BREAKING CHANGE: Chunk format changes significantly with full TOC skeleton wrapping content, affecting embedding models expecting breadcrumb prefixes. |
||
|---|---|---|
| .. | ||
| as_llm | ||
| as_llm_formatter | ||
| as_token_counter | ||
| client | ||
| embedding | ||
| file_parser | ||
| file_store | ||
| file_watcher | ||
| job | ||
| service | ||
| tokenizer | ||
| __init__.py | ||
| application_context.py | ||
| base_component.py | ||
| base_step.py | ||
| component_registry.py | ||
| prompt_handler.py | ||
| runtime_context.py | ||
| runtime_response.py | ||