mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-08 03:10:24 +00:00
82971ac5b0
* feat(file_chunker): add dedicated JSON and JSONL file chunkers - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * chore(config): update default chunker supported_extensions to txt/log * refactor(json_chunker): optimize _build_tree O(n²) serialization and rewrite tests - Fix O(n²) redundant json.dumps in _build_tree: * Empty containers handled directly as leaves (0 serialization) * Non-empty containers recurse first, then reconstruct+dump once * Only containers that become leaves pay serialization cost - Add _reconstruct_object/_reconstruct_array helpers - Remove dead code: _merge_json method - Apply user changes: min_element_size formula 0.01->0.05, threshold < to <= - Use indent=None for compact output (consistent with _SizeNode estimation) - Remove unused _text_size from JsonlFileChunker Test rewrite: - Replace try/finally boilerplate with make_json fixture - Group tests into TestXxx classes with pytest.mark.parametrize - Add TestOutputValidation: 9 parametrized scenarios verifying: * All chunks are valid JSON * Text length <= chunk_chars (with single-leaf tolerance) * Leaf-value concatenation matches original data (dict + array roots) - Add TestSizeNode: incremental size accuracy tests - Add TestDfsAlgorithm: path wrapping, DFS order, calibration tests - Update test_min_element_size_formula for new 0.05 multiplier - Update test_build_tree_structure for larger min_element_size * chore: apply black formatting to test files |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_auto_dream.py | ||
| test_background_steps.py | ||
| test_base_component.py | ||
| test_bm25_index_perf.py | ||
| test_channel_notify.py | ||
| test_channel_sink.py | ||
| test_claim_channel.py | ||
| test_common_steps.py | ||
| test_component_registry.py | ||
| test_config_parser.py | ||
| test_cron_job.py | ||
| test_crud_steps.py | ||
| test_daily_steps.py | ||
| test_default_file_chunker.py | ||
| test_evolve_utils.py | ||
| test_file_catalog.py | ||
| test_file_graph.py | ||
| test_file_store_consistency.py | ||
| test_http_client.py | ||
| test_job.py | ||
| test_json_file_chunker.py | ||
| test_jsonl_file_chunker.py | ||
| test_keyword_index.py | ||
| test_link_expansion.py | ||
| test_local_embedding_store.py | ||
| test_markdown_file_chunker.py | ||
| test_packaging.py | ||
| test_prompt_handler.py | ||
| test_read_image_steps.py | ||
| test_read_with_neighbors.py | ||
| test_reme_cli.py | ||
| test_runtime_context.py | ||
| test_search_step.py | ||
| test_service.py | ||
| test_service_utils.py | ||
| test_tokenizer.py | ||
| test_utils.py | ||
| test_wikilink_utils.py | ||
| test_write_metadata_lock.py | ||