mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
4eb2adf961
* feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint - Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff - Add dynamic efSearch (limit * 5) scaled to query request size - Add async_reindex option: background rebuild with generation-based invalidation - Extract _delete_nodes() in LocalFileStore for subclass reuse - Add unit tests for file store consistency * fix: resolve pylint warnings in faiss store and test file * refactor(file_store): replace generation-based reindex with event-flag worker - Replace _reindex_generation/lock/task with a single long-lived worker coroutine consuming an asyncio.Event flag; repeated submissions coalesce - Use local index reference in vector_search to avoid TOCTOU on self._faiss_index - Pass index explicitly to _set_ef_search for consistency - Track _index_writes to re-arm reindex after concurrent writes - Update tests to match new internal API * fix: resolve pylint too-many-return-statements and implicit-booleaness warnings * feat(file_store): add refine maintenance hook and incremental embedding backfill - Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore - FaissLocalFileStore: incremental vector add on backfill instead of full rebuild - Dynamic tombstone compaction threshold scaled by index size - Add RefineStoreStep with daily cron job (refine_store_cron) - Enable faiss backend and embedding_store by default in default.yaml - Add unit tests for faiss index maintenance * chore(deps): promote faiss-cpu to core dependencies faiss backend is now the default file_store, so faiss-cpu moves from the optional [core] extra to the base dependencies list. * feat: rename refine_store to optimize_index and add vecdb_path_constraint - Rename refine_store step to optimize_index with cron job scheduling - Add vecdb_path_constraint to file_store components - Update default.yaml with optimize_index_cron and faiss backend comment - Update memory_search docs (en/zh) for FAISS vector management - Update unit tests for index maintenance * feat(faiss): add embedding digest to reject stale sidecar after partial dump Add _chunks_embedding_digest() that computes an order-independent SHA-256 over (chunk_id, float16 embedding) pairs. The digest is written into the idmap sidecar at dump time and verified at load time. A mismatch means the sidecar vectors belong to a different chunk generation than the authoritative JSONL — detectable even when the live-ID set is unchanged (same-ID in-place update crash window). Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the crash-between-writes scenario and asserting digest-based rejection. Compress verbose docstrings/comments in existing tests for pylint line budget. --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
78 lines
1,008 B
Text
78 lines
1,008 B
Text
# OS / editor
|
|
.DS_Store
|
|
.idea/
|
|
.vscode/
|
|
.qoder/
|
|
*.code-workspace
|
|
|
|
# Local environment
|
|
.env
|
|
.env.*
|
|
!.env.example
|
|
!example.env
|
|
.venv/
|
|
venv/
|
|
env/
|
|
private*/
|
|
|
|
# Python caches / test artifacts
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
.ipynb_checkpoints/
|
|
.pytest_cache/
|
|
.ruff_cache/
|
|
.mypy_cache/
|
|
.coverage
|
|
coverage.xml
|
|
htmlcov/
|
|
|
|
# Packaging / build outputs
|
|
build/
|
|
dist/
|
|
*.egg-info/
|
|
|
|
# Logs / temporary files
|
|
*.log
|
|
nohup.out
|
|
nohup*.out
|
|
log/
|
|
logs/
|
|
runs/
|
|
tmp*/
|
|
temp*/
|
|
.trash/
|
|
|
|
# ReMe runtime data
|
|
.reme/
|
|
reme_workspace/
|
|
vault/
|
|
*.db
|
|
*.sqlite
|
|
*.sqlite3
|
|
|
|
# Documentation build outputs
|
|
docs/_build/
|
|
site/
|
|
|
|
evaluation/
|
|
datasets/
|
|
|
|
# Claude Code skills (local only)
|
|
.claude/skills/
|
|
|
|
# Memory workspaces (keep dirs, ignore contents)
|
|
benchmark/memory_workspaces/
|
|
|
|
# LongMemEval raw data (download via download.py)
|
|
benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json
|
|
|
|
# BEAM dataset (clone via git)
|
|
benchmark/datasets/BEAM/
|
|
|
|
# Benchmark outputs
|
|
benchmark/results/
|
|
|
|
# integration tests outputs
|
|
tests/integration/logs/
|
|
daily/
|