Compare commits

...

58 commits

Author SHA1 Message Date
jinliyl
99afc2604f
fix(release): harden embedding store and plugins for ReMe 0.4.1.9 (#503)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* chore(release): prepare ReMe 0.4.1.9

* refactor(config): remove daily_cookbook and streamline plugin configs

- Delete the entire daily_cookbook.yaml standalone application config
- Remove qwenpaw dependencies verification and related CI workflow steps
- Simplify release workflows by removing qwenpaw verification and enforcing reme-ai >=0.4.1.9
- Update plugin start commands and examples to use 'default' or 'demo' configs instead of daily_cookbook
- Adjust imports and tests related to daily_cookbook removal and injected_job_kwargs enhancements
- Refactor agent wrapper to support injected_job_kwargs for job parameter injection in auto-fin and daily-paper
- Improve daily_paper digest prompt to include configured daily directory and correct historical search constraints
- Update dependency versions in pyproject.toml files to require reme-ai >=0.4.1.9 and remove qwenpaw optional dependencies
- Clean up unused environment variables and obsolete test cases related to daily_cookbook and verification steps

* fix(local_embedding_store): retry batch computation on vector space changes

- Add up to 3 attempts to recompute embedding batch if vector space changes during processing
- Log warnings when maximum retries reached and discard stale results
- Prevent caching results from outdated vector spaces to maintain consistency
- Add tests to verify retry behavior and abort after continuous vector space churn

fix(daily_paper): update digest search logic and tests

- Change search to query existing memory, not only previous articles in daily_dir
- Allow multiple searches outside daily_dir but limit links to dated markdown in daily_dir before today
- Update test assertions to reflect revised search and linking rules

* fix(embedding): retry vector space changes per request
2026-08-28 11:35:04 +08:00
jinliyl
2dd2255760
ci: update release workflow actions and smoke checks (#502)
* ci: update artifact actions for Node 24

* ci: validate Auto Fin package manifest
2026-08-27 18:03:44 +08:00
jinliyl
d8d667c6ac
docs: refresh ReMe Studio preview image (#501) 2026-08-27 17:50:48 +08:00
jinliyl
940a923f06
ci: allow bootstrap release before qwenpaw plugins (#499) 2026-08-27 17:23:53 +08:00
jinliyl
3d2ecc60d2
feat(service): expose MCP through HTTP backend (#498)
* feat(service): expose MCP through HTTP backend

Serve JSON/SSE job endpoints and streamable HTTP MCP from one FastAPI application, sharing the same jobs and application lifecycle. Preserve the standalone MCP backend, add configurable MCP HTTP settings, update startup metadata and integration docs, and cover routing, lifecycle, configuration, and compatibility behavior with unit tests.

* fix(service): preserve MCP request protections

Route the exact MCP path through the complete FastMCP ASGI application so its middleware and state remain active. Reject non-literal MCP paths and validate reserved Job conflicts before tolerant service registration. Add regression coverage for middleware preservation, route syntax, and startup failure.

* fix(service): reject encoded MCP paths

Reject percent signs in mcp_path so ASGI path decoding cannot turn an accepted configuration into an unreachable route. Cover encoded slash, space, and double-encoded slash inputs.
2026-08-27 17:23:11 +08:00
jinliyl
6f38d201b6
ci: harden build and release workflows (#497) 2026-08-27 16:43:05 +08:00
jinliyl
ef3f99f019
refactor(packaging): reorganize published packages (#495)
* refactor(packaging): reorganize published packages

* fix(packaging): install AgentScope extra in wheel smoke

* docs: align package guides and documentation site

* ci(workflow): add core dependency verification step in Python package build

- Add a workflow step to verify released core dependencies by installing the wheel with core extras
- Assert the presence of the static index.html file to ensure proper package contents
- Create and use a temporary virtual environment for isolation during verification
- Keep existing artifacts upload step intact and conditional on inputs.upload_artifacts flag

* fix(ci): update package installation dependencies in Windows workflow

- Change pip install from editable reme_studio and core to only dev and as extras
- Remove installation of reme_studio and core to streamline dependency setup
- Ensure Windows CI uses the correct extras for testing environment

* fix(tests): add missing commas in toml file reads in package version tests

- Added trailing commas in the tomllib.loads calls for auto-fin and daily_paper configs
- Ensured consistent syntax to prevent potential tuple misinterpretation
- Improved readability and correctness of the test setup code

* fix(packaging): protect qwenpaw releases and test Studio health
2026-08-27 14:02:09 +08:00
jinliyl
b78e32ef03
feat(openclaw): align ReMe plugin with current SDK (#493)
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / TypeScript integrations / Type-check, test, and pack (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
Adopt definePluginEntry, before_prompt_build, current manifest contracts, and official OpenClaw SDK types.

Add DSH-aligned memory batching, retryable shutdown flushing, daily Auto Dream scheduling, updated documentation, tests, ClawHub validation, and optional release publishing.
2026-08-26 19:57:57 +08:00
jinliyl
6a6e0b3c29
fix(dsh): deduplicate pending memory guidance (#494) 2026-08-26 19:37:24 +08:00
jinliyl
a457bf7542
docs: reorganize readme around agent integrations (#492) 2026-08-26 19:30:16 +08:00
jinliyl
513fb5b7f4
feat: extract Daily Paper into an independently packaged plugin (#491)
* feat: extract Daily Paper into a plugin

* fix: satisfy clean-environment quality checks

* fix: address daily paper review feedback
2026-08-26 17:32:46 +08:00
jinliyl
1a6b584274
fix(persistence): avoid duplicate index dumps (#489)
* fix(persistence): avoid duplicate index dumps

* fix(persistence): align dumps with component ownership

* fix(persistence): preserve subclass dump hooks
2026-08-26 16:51:00 +08:00
jinliyl
15d12be6b6
fix(index): tolerate invalid text encoding (#490)
* fix(index): tolerate invalid text encoding

* fix(index): preserve text chunker compatibility
2026-08-26 16:16:26 +08:00
jinliyl
626c850ccb
fix(daily-paper): sanitize Unicode surrogates (#487)
Some checks failed
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* fix(daily-paper): sanitize Unicode surrogates

* fix(daily-paper): sanitize analysis workflow state
2026-08-24 18:59:53 +08:00
jinliyl
01ef1a6efb
ci: use trusted publishing for TypeScript package (#486)
* ci: use trusted publishing for TypeScript package

* docs: scope npm announcement to DeepSeek Harness
2026-08-24 15:21:28 +08:00
jinliyl
efcc2b34d1
feat: simplify plugin setup and add management CLI (#485)
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
* feat: simplify plugin setup and add management CLI

* fix: isolate plugin CLI import side effects

* refactor: streamline plugin validation

* fix: route plugin CLI arguments independently

* fix: support standard plugin source layouts
2026-08-23 17:58:20 +08:00
jinliyl
c8e1248769
fix: recover embeddings after transient health check failure (#471)
Some checks failed
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* fix: recover embedding after transient health failure

* refactor(embedding_store): remove provider_success_count and simplify health recovery logic

- Deleted provider_success_count attribute and related methods across embedding and file stores
- Updated _recover_after_real_request to rely solely on is_healthy flag for recovery decisions
- Removed redundant counting logic for provider successes during embedding operations
- Cleaned up health status management to streamline provider recovery detection
- Adjusted unit tests to align with removal of provider_success_count and maintain health checks consistency

* refactor(embedding_store): use default health check timeout

* fix(embedding_store): ensure is_healthy remains unchanged on cache hits

- Updated get_embeddings docstring to clarify cache hits must not alter is_healthy state
- Improved code comment for embedding dimension matching method

* fix(file_store): make embedding recovery race-safe

* ci: use default CodeQL query suite

* fix(file_store): preserve queued embedding rebuilds

* fix(file_store): preserve verified recovery without chunks
2026-08-21 13:58:51 +08:00
jinliyl
8416fd3ac9
feat: add unified TypeScript agent integrations (#483)
* feat: add unified TypeScript agent integrations

* fix: normalize endpoints without regex backtracking

* fix: address TypeScript integration review feedback

* fix: preserve original OpenClaw prompts

* fix: bound pending OpenClaw prompts
2026-08-21 13:57:14 +08:00
jinliyl
f44f52d919
fix(embedding): exclude provider init from health timeout (#484)
Some checks failed
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Website / Website checks (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
2026-08-21 11:23:14 +08:00
jinliyl
ebcb154e37
fix(search): isolate range dedup state (#465) 2026-08-20 16:15:30 +08:00
jinliyl
39233f4e62
ci: remove Dependabot configuration (#482) 2026-08-20 16:14:52 +08:00
jinliyl
94b7dedc26
Delete .github/README.md 2026-08-20 16:03:55 +08:00
jinliyl
87187c1d25
ci: organize GitHub automation (#466) 2026-08-20 15:56:30 +08:00
jinliyl
f5ec230fef
feat: add DSH memory integration and organize extensions (#461)
* feat: add DSH memory integration and organize extensions

* fix: support newer DSH release candidates

* fix: address DSH integration review feedback

* fix: handle DSH cross-day retry edge cases
2026-08-20 15:31:51 +08:00
jinliyl
2f5fd46b44
refactor: deduplicate entry-point loading (#460)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
GitHub Pages Check / test-and-build (push) Has been cancelled
Package Check / distributions (push) Has been cancelled
Deploy ReMe documentation / build (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Deploy ReMe documentation / deploy (push) Has been cancelled
* refactor: deduplicate entry-point loading

* fix: preserve config conflict error priority
2026-08-19 19:57:53 +08:00
jinliyl
618e8cec66
feat: add entry-point plugin system and extract Auto Fin (#459)
* feat: add entry-point plugin system

* fix: harden plugin config and client loading

* docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI

- Provide step-by-step instructions for updating project.version and merging branches
- Explain dependency verification for reme-ai on PyPI during build
- Specify requirements for GitHub Actions secret configuration and version uniqueness
- Describe manual workflow triggering and input of version number
- Recommend publishing order for related projects
- Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag

* feat: support plugin-defined component types

* refactor: simplify plugin configuration

* fix: isolate plugin loading and defer client fallback

* refactor: freeze built-in component registry

* fix: isolate config entry point loading

* fix: complete auto-fin package metadata
2026-08-19 17:23:23 +08:00
jinliyl
d3aee1adf5
feat(evolve): report auto-dream file changes (#458)
* feat(evolve): report auto-dream content changes

* perf(evolve): use lightweight dream snapshots
2026-08-19 15:44:06 +08:00
jinliyl
6b9a75267b
fix(ci): support AgentScope 2.0.6 initialization (#457)
Some checks are pending
Package Check / distributions (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-19 11:33:52 +08:00
jinliyl
c792fd197c
Update agentscope version to 2.0.6
Some checks are pending
Package Check / distributions (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-18 11:04:26 +08:00
jinliyl
fd2894f939
fix: harden the 0.4.1.7 release configuration (#456)
Some checks failed
NPM Format / Website checks (push) Has been cancelled
GitHub Pages Check / test-and-build (push) Has been cancelled
Package Check / distributions (push) Has been cancelled
Deploy ReMe documentation / build (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Deploy ReMe documentation / deploy (push) Has been cancelled
* fix(packaging): harden the Studio release workflow

* chore(daily-paper): tune scheduled discovery defaults

* fix(docs): link the ReMe blog to GitHub Pages

* fix(docs): increase Chinese hero title spacing

* refactor(docs): share hero title line spacing

* fix(docs): keep desktop hero copy on two lines

* fix(docs): widen the home hero description

* style(docs): loosen hero title line height

* fix(docs): hide Markdown frontmatter in rendered pages

* docs(readme): simplify installation and remove standalone ReMe Studio instructions

- Remove references to separate ReMe Studio package and static build steps
- Clarify that `core` extra includes common integrations including Studio
- Update installation instructions to use `pip install -e ".[core]"`
- Remove detailed Studio usage and frontend development instructions
- Note that Studio is included with `core` and optional via `web` extra
- Simplify Quick Start guide by removing Studio usage step
- Remove mentions of serving Studio with HTTP service when using extras
- Update both English and Chinese README files accordingly

* docs(readme): streamline and clarify memory design and operations

- Remove redundant explanations about core extra installation
- Simplify memory processing flow description for clarity
- Clarify memory workspace directory default and customization
- Condense automatic memory flow to emphasize rebuildable metadata
- Refine search functionality explanation with RRF fusion details
- Shorten and clarify agent integration description, removing redundancy
- Update and simplify the operations command list, removing less common commands
- Revise community and support section for conciseness and clarity
- Maintain parallel updates in both English and Chinese README files

* test(bump_version): add tests for version bumping and consistency checks

- Add dynamic loading of bump_version and package_studio scripts for testing
- Test that studio package and dependencies have matching versions
- Implement fixtures to write temporary version files for testing
- Add test ensuring bump_version updates all relevant files and dependencies
- Add test to reject inconsistent version sources before writing
- Refactor tests to use common REPOSITORY path variable
- Include imports and setup for pytest in test file

feat(bump_version): create script to update ReMe and Studio versions

- Implement version reading from __init__.py and pyproject.toml files
- Validate current versions are consistent across files before updating
- Update version strings atomically to avoid partial writes
- Ensure exact pinning of studio dependency in main package extras
- Validate new version format against a safe pattern
- Provide CLI interface to bump versions from command line
- Raise errors if expected version declarations or pins are missing or duplicated

* fix(release): validate split package publishing

* fix(release): improve validation diagnostics

* fix(release): sync docs and workflow inputs

* fix(release): split PyPI publish jobs
2026-08-13 17:22:00 +08:00
jinliyl
2a05914150
feat: distribute Studio as an optional package (#454) 2026-08-13 11:07:37 +08:00
jinliyl
29eb51d7ba
fix: show resolved service URL and shrink Studio preview asset (#453)
Some checks are pending
NPM Format / Website checks (push) Waiting to run
Deploy ReMe documentation / build (push) Waiting to run
Deploy ReMe documentation / deploy (push) Blocked by required conditions
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* fix: show resolved service address in startup banner

* perf(website): reduce social preview image size

* fix: resolve MCP transport in startup banner
2026-08-12 19:37:31 +08:00
jinliyl
da9a8b7810
fix(docs): make homepage cards use direct links (#452) 2026-08-12 13:28:23 +08:00
jinliyl
dbf2a17da6
docs: expand ReMe documentation site (#451) 2026-08-12 13:18:40 +08:00
jinliyl
64249873ce
fix(website): resolve Dependabot dependency alerts (#450) 2026-08-12 12:20:41 +08:00
jinliyl
28fa636506
fix(docs): use public npm registry for Pages (#449) 2026-08-12 12:03:19 +08:00
jinliyl
52fdd446fb
docs: add standalone GitHub Pages site (#448) 2026-08-12 11:55:35 +08:00
jinliyl
ab66f2bb56
docs: refresh ReMe guides, diagrams, and Studio documentation (#447)
* docs: update ReMe documentation URL

* docs: localize ReMe Studio social image

* docs(AGENTS): update agent guidelines and repository documentation structure

- Clarify coding agent guidance for keeping changes small and consistent
- Revise project principle descriptions for clarity and modern terminology
- Expand repository map with detailed component and folder explanations
- Add configuration and CLI usage instructions, including syntax and merging rules
- Elaborate on component, step registration, and application lifecycle processes
- Define jobs, steps, and state handling conventions for stateless design
- Specify workspace and file safety policies, including path restrictions and locking
- Update validation commands and testing environment recommendations
- Clarify coding and test conventions, including style and dependency policies
- Distinguish documentation boundaries and update website content contribution notes
- Reinforce change guardrails to avoid breaking backward compatibility and data loss
- Improve svg diagram formatting and textual details in auto dream and proactive flow image

* style(docs): fix font-family syntax in SVG style definitions

- Correct quotation marks around font-family names in memory-as-file.svg
- Standardize font-family formatting by removing unnecessary quotes in reme-blog-architecture.svg
- Ensure consistent CSS style formatting within SVG files for better rendering fidelity

* docs: add ReMe blog to news

* style(docs): inline svg styles and improve text formatting

- Convert multiline SVG style tags into single-line for compactness in multiple figures
- Remove redundant line breaks in subtitle text elements for consistency
- Shorten descriptive texts in SVG figures for clarity and conciseness
- Adjust font sizes and text for better readability in SVG elements
- Correct whitespace issues in Chinese markdown document for improved formatting
- Remove unused style blocks from framework structure SVG for cleaner code
2026-08-12 10:59:03 +08:00
jinliyl
215c1f72f2
feat: refine local-first research and memory workflows (#444)
Some checks are pending
NPM Format / Website checks (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* feat: refine local-first research workflows

* fix: delegate structured output tool choice

* refactor(auto-fin): fetch and filter rolling CLS news

* fix(auto-fin): keep imports portable across platforms

* feat(auto-fin): expose CLS fetch controls

* fix(auto-fin): propagate configurable news window

* feat(auto_fin): normalize hybrid wikilinks in report body

- Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations
- Use regex to identify hybrid wikilinks with optional destinations
- Replace redundant destinations with simpler wikilink format for clarity
- Ensure normalization is failure-safe with exception handling and logging
- Update report body normalization process to apply hybrid wikilink fix
- Add unit tests to verify correct normalization and failure safety behavior

* fix(dream): serialize integration with application-wide asyncio lock

- Add application-wide asyncio.Lock to serialize digest writes during integration
- Update _snapshot_digest to capture metadata per bucket
- Validate bucket association when recovering from file changes
- Add tests ensuring recovery only from the correct bucket
- Add tests confirming integration lock is shared across application context
- Enhance strict topic YAML loading validation in dream utils
- Add tests for strict topic loading rejecting invalid or lossy fields

* fix(cookbook): enable configurable job_tools for digest and merge steps

- Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps
- Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list
- Modify AutoFinMergeStep to similarly read job_tools from kwargs
- Update tests to pass job_tools explicitly when invoking these steps
- Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling

* fix: retry incomplete dream receipts

* perf(pdf): increase max PDF pages limit from 20 to 35

- Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml
- Modified code to extract up to 35 pages instead of 20 in analyze.py
- Updated README and README_ZH to document the increased max_pdf_pages
- Adjusted unit test assertions to reflect new max_pdf_pages limit of 35

* fix memory integration and daily paper links

* docs clarify cookbook tool usage
2026-08-11 23:32:34 +08:00
jinliyl
9533c17d51
feat(web): serve workspace from HTTP service (#446)
* feat(web): add the ReMe workspace frontend

* feat(web): serve workspace from HTTP service

* test(web): satisfy pylint docstring checks

* fix(web): use same-origin API safely

* fix(web): preserve API route semantics
2026-08-11 23:32:24 +08:00
jinliyl
b8f48c8004
feat(web): add the ReMe Studio frontend (#418)
* feat(web): add the ReMe workspace frontend

* fix(web): use public npm registry in lockfile

* fix(web): address workspace review feedback

* fix(web): protect drafts and report file limits

* fix(web): finish chat streams after tab switches

* feat(web): rename frontend to ReMe Studio
2026-08-11 19:47:59 +08:00
imrewce
3924f89bb4
feat(bench): adding eval adapter for proactiveness on Pi-Bench (#439)
* feat(bench): adding eval adapter for proactiveness on Pi-Bench

* Revise README for π-Bench evaluation suite

Updated the README to reflect the new project name and description.

* fix(bench): refining pi-bench scripts according to cr comments

* fix(bench): restore agent builtin tools in prebuilt toolkit
2026-08-11 16:37:54 +08:00
jinliyl
c7dbf31c3f
docs: expand ReMe guides and agent integrations (#445) 2026-08-11 13:31:11 +08:00
imrewce
58276f740b
fix(file_io): auto appending suffix for all related steps (#430)
* fix(file_io): auto appending suffix for all related steps

* fix(file_io): covering boundary cases of potential directory input
2026-08-11 11:09:28 +08:00
imrewce
3095564313
docs: Adding pi-bench related proc performance to blog draft (#443)
* docs(blog): refine proactive section wording in zh reme-blog

* chore(doc): supplementing proc related performance
2026-08-11 10:59:51 +08:00
jinliyl
21057931a9
fix(embedding): isolate caches by vector space (#442)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* fix(embedding): isolate caches by vector space

* fix(embedding): stabilize cache space switching

* Revert "fix(embedding): stabilize cache space switching"

This reverts commit 74193c9a0a.

* fix(embedding): include resolved OpenAI endpoint in cache ID

* fix(embedding): stabilize cache space switching

* fix(embedding): isolate Ollama endpoint caches
2026-08-10 22:42:16 +08:00
Zhaoyang Liu
5a5855f5ff
docs: refine ReMe launch blog (#440)
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-08-10 18:45:06 +08:00
jinliyl
072cb6a55b
feat(daily-paper): add opt-in Hugging Face mirror support (#437)
* feat(daily-paper): add Hugging Face mirror switch

* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env

The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.

HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(daily-paper): address mirror configuration feedback

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 15:57:24 +08:00
imrewce
fca42f4e6c
docs(blog): refine proactive section wording in zh reme-blog (#438) 2026-08-10 15:57:16 +08:00
jinliyl
d5e0d2837b
refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents (#432)
Some checks failed
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
* refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents

Rework the auto-fin and daily-paper cookbooks to run on structured-output
LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with
data-source mirrors, and rewrite the affected unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(auto_fin): unify JSON output serialization and writing

- Extracted _write_output static method to serialize and write Pydantic models as compact JSON
- Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps
- Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep
- Updated auto_fin merge step to write output via new _write_output method
- Enhanced news reading with caching in AutoFinHistoryStep
- Refined returns calculation to handle events before close on non-trading days correctly

feat(daily_paper): improve note path resolution and metadata handling

- Introduced iter_note_metadata generator for safe Markdown frontmatter iteration
- Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles
- Updated analyze, collect, digest, and select steps to use centralized constants and helpers
- Used utc_now_iso for consistent timestamping in metadata
- Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps
- Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places
- Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management
- Improved error messages for filename validation in daily paper title normalization

feat(auto_fin): add multi-run cron schedules for intraday refinement

- Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates
- Each intraday run adds evidence cumulatively instead of replacing prior output wholly
- Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron

refactor(auto_fin_data): clean ETF code handling and page limits

- Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes"
- Added constants for major news and fund page limits to control pagination
- Improved ETF name extraction logic to handle missing fields consistently

fix(auto_fin_merge): fix report retrieval and merging logic

- Added support for getting current intra-day report in addition to previous day's report
- Modified merge template to include prior and current report sections for better context
- Adjusted report path handling to consistently use Path objects

test(auto_fin): add coverage for returns calculation and report retrieval

- Added test for returns when event occurs before close on non-trading day, checking next session entry
- Added test for previous and current report retrieval feeding merge context with disk files
- Extended test asserts for auto_fin cron schedule changes in config

style(daily_paper): reorder and cleanup imports

- Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import
- Cleaned up commented and unused imports across daily_paper steps

* feat: add configurable upstream mirror proxy

* style: format auto-fin data step

* fix: align cookbook mirrors and contracts

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-07 23:53:14 +08:00
jinliyl
e05b201da9
feat(backend): improve workspace support for web clients (#420)
* feat(backend): improve workspace support for web clients

* fix(config): preserve the default workspace directory

* chore(reme): bump version to 0.4.1.5

- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file

* fix(chat): disable builtin tools in read-only mode

* fix(agent): make builtin tools opt-in

* fix(list): tolerate files removed during mtime sort

* fix(chat): expose complete read-only job set
2026-08-07 23:52:56 +08:00
jinliyl
765103a597
docs(blog): add reme blog (#436)
* docs: add Chinese ReMe blog article

* fix(docs): update wiki links and adjust SVG path coordinates

- Removed file extensions from wiki link texts for consistency
- Modified path coordinates for relation lines in SVG illustration
- Added an arrow path with fill color to indicate direction in SVG diagram

* docs(blog): expand ReMe user guide and invite community contributions

- Add detailed descriptions for different ReMe user groups including intelligent agents,
  developers, researchers, engineers, and analysts
- Emphasize user control over data as editable Markdown files instead of black-box storage
- Introduce ReMe's long-term memory infrastructure accessible via multiple interfaces
- Highlight how ReMe can turn scattered information into personal knowledge networks
- Include a new "Welcome Contributions" section encouraging community involvement
- List areas for contribution such as integration, data sources, features, applications,
  documentation, and issue feedback
2026-08-07 17:36:02 +08:00
jinliyl
168b7194ab
docs: add Chinese ReMe blog article (#435) 2026-08-07 17:19:01 +08:00
lichen2015
e7b9274190
fix(stat): return text/markdown for .md files regardless of OS mime registry (#433)
On macOS, mimetypes.guess_type() may not recognize .md files, causing
stat to report application/octet-stream and breaking test assertions.
Explicitly map .md files to text/markdown so the behavior is stable
across platforms.
2026-08-07 16:31:03 +08:00
jinliyl
c5d92a24ab
feat: weave dream wikilinks into contextual prose (#428)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-06 17:40:01 +08:00
jinliyl
9218a2d0e3
refactor: derive dialog paths from session_dir (#421)
* refactor: derive dialog paths from session directory

* fix: normalize configured session paths

* fix: align dialog watch paths with writers

* fix: reject absolute session directories
2026-08-06 17:07:14 +08:00
Ziyang Guo
6503e1271c
fix(prompt): default omitted conditional flags to false (#424)
Always apply conditional-line filtering so tagged prompt lines are removed unless the corresponding boolean flag is explicitly true. Add regressions for omitted flags with and without format variables.

Test: pytest tests/unit/test_prompt_handler.py -q
2026-08-06 16:22:59 +08:00
xyf2020
23d4c96c15
refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt (#422)
* chore(benchmark): isolate dataset/workspaces/results per benchmark

- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py

* feat(benchmark): add read tool to agentic answer and update BEAM results

- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)

* feat(auto_memory): add source line-number markers for note traceability

- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure

* feat(beam): enhance auto_memory with source markers and pilot ingest tooling

* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts

* feat: add CompressorStep and search_v2 dual-mode session compression

- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
  text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
  session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
  join_chunk_entries; session chunks now render line-aligned with
  L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
  per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
  query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths

* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results

* refactor(benchmark): make session compression config-driven via compress_session flag

Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.

* feat(lme/auto_memory): add source attribution markers with line numbers

Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.

* feat: improve agentic answer prompt and update beam 100K results

- Strengthen abstention rule: prohibit extrapolation from related but
  non-direct evidence
- Add multi-angle search after preliminary answer to check for
  conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
  including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
  (+0.140), abstention: 0.500 → 0.525 (+0.025)

* feat(read): add session-aware formatting for read tool and update BEAM eval

- Add truncate_session_output in _file_io.py to render jsonl session
  lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
  job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
  so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
  after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
  1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results

* chore(config): disable expand_links in beam and lme search_v2 configs

* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script

* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions

- Add compressor job and compressor as_llm component to reme/config/beam.yaml
  (aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
  when the compressor job is missing from the active config, log a warning
  and skip compression instead of raising 'Job compressor not found'.
  Skipped when there is no app_context so unit tests mocking run_job still
  drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
  before cd benchmark/beam/dataset (the directory is gitignored and absent
  in a fresh clone)

* fix(steps): guard compressor exceptions and fix ReadStep boolean override

1. search_v2: catch per-entry exceptions from run_job('compressor') inside
   compress() so asyncio.gather never propagates a compressor failure (e.g.
   temporary LLM outage). The failing entry keeps its original body while
   remaining entries are still compressed, preserving already-retrieved
   search results.

2. read: replace 'context_value or yaml_value' with an existence check so
   that a runtime-injected False can explicitly disable a YAML-true
   read_step_format_session flag.

Add focused unit tests for both paths.

* fix(search_v2): use existence check for strict_date_filter boolean override

Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.

* refactor(search): simplify strict_date_filter fallback to truthiness-or

* style(test): rename unused param to satisfy pylint W0613

* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt

- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
  per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
  benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
  description, keep search strategy, draft tool, and answer rules concise.

* docs(benchmark): update LME README_ZH results to latest eval run

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
2026-08-06 15:13:52 +08:00
412 changed files with 57871 additions and 9670 deletions

97
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View file

@ -0,0 +1,97 @@
name: Bug report
description: Report reproducible incorrect or unexpected ReMe behavior
title: "[Bug]: "
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve ReMe. Please remove secrets, API keys, and private memory content before submitting.
- type: textarea
id: description
attributes:
label: Description
description: What happened, and what did you expect instead?
placeholder: Describe the observed and expected behavior.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Provide the smallest configuration and command sequence that reproduces the problem.
placeholder: |
1. Configure ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: config
attributes:
label: Relevant configuration
description: Include only relevant values and redact credentials, tokens, endpoints, and private paths.
render: yaml
- type: textarea
id: logs
attributes:
label: Logs or traceback
description: Paste relevant output after removing secrets and private workspace content.
render: shell
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
validations:
required: true
- type: input
id: python-version
attributes:
label: Python version
placeholder: e.g. 3.11.9
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- Linux
- macOS
- Windows
- Other
validations:
required: true
- type: dropdown
id: area
attributes:
label: Affected area
options:
- CLI or configuration
- HTTP, MCP, or local service
- Memory or workspace files
- Search, catalog, graph, or index
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Packaging or installation
- Other
validations:
required: true
- type: checkboxes
id: safety
attributes:
label: Data safety
options:
- label: I removed credentials and private memory content from this report.
required: true

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View file

@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: ReMe documentation
url: https://reme.agentscope.io
about: Read the installation, configuration, and usage guides.
- name: Existing issues
url: https://github.com/agentscope-ai/ReMe/issues
about: Search for existing reports and discussions before opening a new issue.

View file

@ -0,0 +1,64 @@
name: Feature request
description: Propose a focused enhancement to ReMe
title: "[Feature]: "
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What user problem or limitation should this change address?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed behavior
description: Describe the desired behavior and its user-visible contract.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- CLI or configuration
- Jobs or steps
- Memory or workspace files
- Search, catalog, graph, or index
- Service or client
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Documentation
- Other
validations:
required: true
- type: textarea
id: ownership
attributes:
label: Local-first and compatibility considerations
description: Explain any effect on user-owned files, rebuildable state, configuration, schemas, or service interfaces.
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe workarounds or alternative designs you considered.
- type: textarea
id: examples
attributes:
label: Example usage
description: Show the proposed CLI, configuration, API, or UI behavior when useful.
render: shell
- type: checkboxes
id: contribution
attributes:
label: Contribution
options:
- label: I am willing to help implement or test this feature.

53
.github/ISSUE_TEMPLATE/question.yml vendored Normal file
View file

@ -0,0 +1,53 @@
name: Usage question
description: Ask for help using or configuring ReMe
title: "[Question]: "
labels: [question]
body:
- type: markdown
attributes:
value: Please check the documentation and existing issues before asking a new question.
- type: textarea
id: goal
attributes:
label: What are you trying to achieve?
validations:
required: true
- type: textarea
id: attempted
attributes:
label: What have you tried?
description: Include relevant commands or configuration, with secrets and private memory content removed.
validations:
required: true
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
- type: dropdown
id: area
attributes:
label: Area
options:
- Installation
- Configuration
- CLI or service usage
- Memory and workspace management
- Search and retrieval
- ReMe Studio
- Plugin or integration
- Other
- type: checkboxes
id: checked
attributes:
label: Before submitting
options:
- label: I checked the [ReMe documentation](https://reme.agentscope.io) and searched existing issues.
required: true
- label: I removed credentials and private memory content.
required: true

35
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,35 @@
## Summary
<!-- Explain the problem and the smallest coherent change that addresses it. -->
## Related issue
<!-- Use "Fixes #123" when applicable. -->
## Contract and data impact
- [ ] No public configuration, schema, CLI, endpoint, streaming, or workspace-layout contract changes
- [ ] No user-owned memory files are deleted or rewritten
- [ ] Derived indexes, catalogs, graphs, caches, and metadata remain rebuildable
<!-- If any item is unchecked, describe the impact and migration or recovery path. -->
## Validation
<!-- List the exact checks run and their results. Explain relevant checks that were not run. -->
- [ ] Focused tests pass
- [ ] Unit tests pass, or omitted tests are explained below
- [ ] `pre-commit run --all-files` passes, or omitted checks are explained below
- [ ] Frontend checks were run when `reme_studio/` changed
## Checklist
- [ ] I reviewed the diff for unrelated changes and sensitive data
- [ ] Tests cover intentional behavior changes
- [ ] Defaults, schemas, and concise documentation were updated together when required
- [ ] Long-lived clients, tasks, services, and executors follow the application lifecycle
## Screenshots or additional notes
<!-- Include UI screenshots, compatibility notes, or follow-up work when relevant. -->

58
.github/workflows/_build-docs.yml vendored Normal file
View file

@ -0,0 +1,58 @@
name: _Build documentation
on:
workflow_call:
inputs:
run_tests:
description: Run the documentation test suite before building
required: false
default: true
type: boolean
upload_pages_artifact:
description: Upload the build for a later GitHub Pages deployment job
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
build:
name: Build documentation
runs-on: ubuntu-latest
defaults:
run:
working-directory: github-pages
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
cache: npm
cache-dependency-path: github-pages/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
if: inputs.run_tests
run: npm test
- name: Build documentation
run: npm run build
- name: Configure Pages
if: inputs.upload_pages_artifact
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
- name: Upload Pages artifact
if: inputs.upload_pages_artifact
uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4
with:
path: github-pages/dist

View file

@ -0,0 +1,88 @@
name: _Build Python packages
on:
workflow_call:
inputs:
expected_version:
description: Expected release version; omit for a consistency-only check
required: false
default: ''
type: string
upload_artifacts:
description: Upload distributions for later publish jobs
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
distributions:
name: Build Python distributions
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest twine
- name: Validate package versions
if: inputs.expected_version == ''
run: python scripts/bump_version.py --check
- name: Validate release version
if: inputs.expected_version != ''
env:
EXPECTED_VERSION: ${{ inputs.expected_version }}
run: python scripts/bump_version.py --check --expected-version "${EXPECTED_VERSION}"
- name: Run package tests
run: PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
- name: Build and check distributions
run: |
mkdir -p dist/reme
python -m build --outdir dist/reme
python -m twine check dist/reme/*
- name: Verify distributions and isolated installation
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme/web/')
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme_studio/')
python -m venv "${RUNNER_TEMP}/reme-package-smoke"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -m pip install "${REME_WHEEL}[as]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -c "import reme"
- name: Verify released core dependencies
if: inputs.expected_version != ''
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m venv "${RUNNER_TEMP}/reme-core-package-smoke"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" -m pip install "${REME_WHEEL}[core]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- name: Upload ReMe distributions
if: inputs.upload_artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-distributions
path: dist/reme/
if-no-files-found: error

48
.github/workflows/ci-docs.yml vendored Normal file
View file

@ -0,0 +1,48 @@
name: CI / Documentation
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-docs.yml'
- '.github/workflows/_build-docs.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'reme_studio/README*.md'
- 'reme_studio/public/og.jpg'
- 'typescript/README*.md'
- 'plugins/*/README*.md'
- 'benchmark/*/README*.md'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-docs.yml'
- '.github/workflows/_build-docs.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'reme_studio/README*.md'
- 'reme_studio/public/og.jpg'
- 'typescript/README*.md'
- 'plugins/*/README*.md'
- 'benchmark/*/README*.md'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
documentation:
name: Test and build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true

40
.github/workflows/ci-packages.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: CI / Python packages
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
distributions:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml

40
.github/workflows/ci-python-quality.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: CI / Python quality
on:
push:
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
pre-commit:
name: Pre-commit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
cache: pip
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e reme_studio -e ".[dev,core]"
pip install -q --no-deps -e plugins/auto-fin -e plugins/daily_paper
- name: Pre-commit starts
run: pre-commit run --all-files

View file

@ -1,4 +1,4 @@
name: Tests ReMe
name: CI / Python tests
on:
push:
@ -11,6 +11,9 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
unit-tests:
name: Unit Tests - py${{ matrix.python-version }}
@ -21,10 +24,12 @@ jobs:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
@ -32,12 +37,14 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
pip install -e ".[dev,core]"
pip install -e reme_studio -e ".[dev,core]"
pip install --no-deps -e plugins/auto-fin
pip install -e plugins/daily_paper
pip install coverage
- name: Run unit tests
run: |
coverage run -m pytest tests/unit \
coverage run -m pytest tests/unit plugins/auto-fin plugins/daily_paper \
-v \
--tb=long \
-s \

90
.github/workflows/ci-reme-studio.yml vendored Normal file
View file

@ -0,0 +1,90 @@
name: CI / ReMe Studio
on:
push:
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
pull_request:
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
studio:
name: Studio checks
runs-on: ubuntu-latest
defaults:
run:
working-directory: reme_studio
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Setup Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run format check
run: npm run format:check
- name: Run lint
run: npm run lint
- name: Run tests
run: npm test
- name: Verify npm package
run: |
npm pack --pack-destination "${RUNNER_TEMP}"
tar -tzf "${RUNNER_TEMP}"/agentscope-ai-reme_studio-*.tgz | grep '^package/dist-static/index.html$'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.11"
- name: Build and verify Python package
working-directory: .
run: |
python -m pip install build packaging pytest twine
PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio
python -m twine check dist/studio/*
STUDIO_WHEEL="$(pwd)/$(ls dist/studio/reme_studio-*.whl)"
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY

51
.github/workflows/ci-typescript.yml vendored Normal file
View file

@ -0,0 +1,51 @@
name: CI / TypeScript integrations
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-typescript.yml'
- '.github/workflows/release-typescript.yml'
- 'typescript/**'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-typescript.yml'
- '.github/workflows/release-typescript.yml'
- 'typescript/**'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
package:
name: Type-check, test, and pack
runs-on: ubuntu-latest
defaults:
run:
working-directory: typescript
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
cache: npm
cache-dependency-path: typescript/package-lock.json
- run: npm ci
- run: npm run format:check
- run: npm run lint
- run: npm run typecheck
- run: npm test
- run: npm run test:package
- name: Validate OpenClaw package contract
run: npx --yes clawhub@0.23.3 package validate . --json

View file

@ -1,4 +1,4 @@
name: Windows Smoke
name: CI / Windows
on:
push:
@ -11,6 +11,9 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
cli-smoke:
name: CLI smoke - py${{ matrix.python-version }}
@ -21,10 +24,12 @@ jobs:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
@ -32,10 +37,10 @@ jobs:
- name: Install package
run: |
python -m pip install --upgrade pip setuptools wheel
pip install -e ".[dev,core]"
pip install -e ".[dev,as]"
- name: Run version job
run: reme start service.backend=cli job=version
run: reme start config=tests/fixtures/config/version-smoke.yaml job=version
- name: Run Windows path tests
run: |

52
.github/workflows/deploy-docs.yml vendored Normal file
View file

@ -0,0 +1,52 @@
name: Deploy / Documentation
on:
push:
branches: [main]
paths:
- "github-pages/**"
- "docs/**"
- "README.md"
- "README_ZH.md"
- "reme_studio/README*.md"
- "reme_studio/public/og.jpg"
- "typescript/README*.md"
- "plugins/*/README*.md"
- "benchmark/*/README*.md"
- "AGENTS.md"
- ".github/workflows/deploy-docs.yml"
- ".github/workflows/_build-docs.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
name: Build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true
upload_pages_artifact: true
permissions:
contents: read
pages: write
id-token: write
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
permissions:
pages: write
id-token: write
steps:
- name: Deploy
id: deployment
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5

View file

@ -1,16 +1,20 @@
name: PR Title Check
name: Policy / PR title
on:
pull_request:
branches: [main, master, dev, develop]
types: [opened, edited, synchronize, reopened]
permissions:
contents: read
pull-requests: read
jobs:
check-pr-title:
runs-on: ubuntu-latest
steps:
- name: Check PR title format
uses: amannn/action-semantic-pull-request@v6.1.1
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:

View file

@ -1,38 +0,0 @@
name: Pre-commit
on: [ push, pull_request ]
jobs:
run:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: True
matrix:
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
PYTHON: '3.11'
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e ".[dev,core]"
- name: Install pre-commit
run: |
pre-commit install
- name: Pre-commit starts
run: |
pre-commit run --all-files > pre-commit.log 2>&1 || true
cat pre-commit.log
if grep -q Failed pre-commit.log; then
echo -e "\e[41m [**FAIL**] Please install pre-commit and format your code first. \e[0m"
exit 1
fi
echo -e "\e[46m ********************************Passed******************************** \e[0m"

View file

@ -1,45 +0,0 @@
# This workflow will upload a Python Package using Twine when a release is created
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Publish Python Package to Pypi
on:
workflow_dispatch:
release:
types: [published]
permissions:
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel build
- name: Build package
run: python -m build
- name: Test installation
run: |
WHEEL="$(ls dist/*.whl)"
pip install "${WHEEL}[core]"
python -c "import reme; print(reme.__version__)"
- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}

157
.github/workflows/release-auto-fin.yml vendored Normal file
View file

@ -0,0 +1,157 @@
# 发布操作手册:
# 1. 先将 plugins/auto-fin/pyproject.toml 中的 project.version 更新为待发布版本并合入目标分支。
# 2. 确认插件依赖的 reme-ai 版本已经发布到 PyPI本工作流会在构建阶段验证该依赖可下载。
# 3. 确认 PyPI Trusted Publisher 已绑定本仓库、此工作流和 pypi environment且 PyPI 上不存在相同版本。
# 4. 在 GitHub 仓库的 Actions 页面选择“Release / Auto Fin plugin”点击“Run workflow”。
# 5. 输入与 project.version 完全一致的版本号(例如 0.1.0)后运行;版本也可以带 v 前缀。
#
# 推荐发布顺序reme-ai -> reme-auto-fin -> QwenPaw 更新依赖并通过 plugins: [auto-fin] 启用。
# 当前仅支持 workflow_dispatch 手动触发,不会因 push、tag 或 release 自动发布。
name: Release / Auto Fin plugin
run-name: Publish reme-auto-fin ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/auto-fin/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-auto-fin
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install --no-deps -e plugins/auto-fin
- name: Validate package name and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/auto-fin/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-auto-fin":
raise SystemExit(f"Expected project name 'reme-auto-fin', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [requirement for requirement in project["dependencies"] if requirement.startswith("reme-ai")]
if len(requirements) != 1:
raise SystemExit(f"Expected one reme-ai dependency, found {requirements!r}")
reme_requirement = Requirement(requirements[0])
if reme_requirement.name != "reme-ai" or reme_requirement.extras:
raise SystemExit(f"Expected a base reme-ai dependency, found {requirements[0]!r}")
if Version("0.4.1.8") in reme_requirement.specifier or Version("0.4.1.9") not in reme_requirement.specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.9, found {requirements[0]!r}")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirement}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Auto Fin tests
run: python -m pytest plugins/auto-fin -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-auto-fin-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/auto-fin
python -m build plugins/auto-fin --outdir dist/auto-fin
python -m twine check dist/auto-fin/*
- name: Verify distributions and isolated installation
run: |
AUTO_FIN_WHEEL="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.whl)"
AUTO_FIN_SDIST="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.tar.gz)"
python -m zipfile -l "${AUTO_FIN_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${AUTO_FIN_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-auto-fin-smoke"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" -m pip install \
"agentscope[model-ollama]==2.0.7" "${AUTO_FIN_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-auto-fin")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["auto-fin"].value == "reme_auto_fin"
manifest = load_package_manifest("reme_auto_fin", plugin_name="auto-fin")
assert set(manifest.backends) == {
"auto_fin_data_step",
"auto_fin_topic_step",
"auto_fin_merge_step",
}
assert set(manifest.application_defaults["jobs"]) == {
"auto_fin",
"auto_fin_cron",
}
PY
- name: Upload distributions
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin
- name: Publish reme-auto-fin
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/auto-fin

View file

@ -0,0 +1,157 @@
# Release checklist:
# 1. Update project.version in plugins/daily_paper/pyproject.toml and merge it into the target branch.
# 2. Publish the required reme-ai version before this plugin; the build verifies that dependency on PyPI.
# 3. Configure PyPI Trusted Publishing for this repository/workflow and its pypi environment.
# 4. Run "Release / Daily Paper plugin" from GitHub Actions with the exact project version (a v prefix is accepted).
#
# Recommended order: reme-ai -> reme-daily-paper -> downstream applications enabling plugins: [daily-paper].
# This workflow is intentionally manual and never publishes from a push, tag, or GitHub release event.
name: Release / Daily Paper plugin
run-name: Publish reme-daily-paper ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/daily_paper/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-daily-paper
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install -e plugins/daily_paper
- name: Validate package name, dependencies, and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/daily_paper/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-daily-paper":
raise SystemExit(f"Expected project name 'reme-daily-paper', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [Requirement(value) for value in project["dependencies"]]
reme_requirements = [requirement for requirement in requirements if requirement.name == "reme-ai"]
if len(reme_requirements) != 1 or reme_requirements[0].extras:
raise SystemExit(f"Expected one base reme-ai dependency, found {reme_requirements!r}")
if Version("0.4.1.8") in reme_requirements[0].specifier or Version("0.4.1.9") not in reme_requirements[0].specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.9, found {reme_requirements!r}")
if sum(requirement.name == "pypdf" for requirement in requirements) != 1:
raise SystemExit("Expected exactly one pypdf dependency")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirements[0]}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Daily Paper tests
run: python -m pytest plugins/daily_paper -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-daily-paper-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/daily-paper
python -m build plugins/daily_paper --outdir dist/daily-paper
python -m twine check dist/daily-paper/*
- name: Verify distributions and isolated installation
run: |
DAILY_PAPER_WHEEL="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.whl)"
DAILY_PAPER_SDIST="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.tar.gz)"
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/plugin.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/analyze.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${DAILY_PAPER_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-daily-paper-smoke"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" -m pip install \
"agentscope[model-ollama]==2.0.7" "${DAILY_PAPER_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-daily-paper")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["daily-paper"].value == "reme_daily_paper"
manifest = load_package_manifest("reme_daily_paper", plugin_name="daily-paper")
assert set(manifest.backends) == {
"daily_paper_collect_step",
"daily_paper_rank_step",
"daily_paper_select_step",
"daily_paper_analyze_step",
"daily_paper_digest_step",
}
assert set(manifest.application_defaults["jobs"]) == {"daily_paper", "daily_paper_cron"}
PY
- name: Upload distributions
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper
- name: Publish reme-daily-paper
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/daily-paper

47
.github/workflows/release-python.yml vendored Normal file
View file

@ -0,0 +1,47 @@
name: Release / Python packages
# Configure a PyPI Trusted Publisher for this repository, workflow, and its
# pypi environment before running the manual release.
on:
workflow_dispatch:
inputs:
version:
description: Release version
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-ai
cancel-in-progress: false
jobs:
build:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml
with:
expected_version: ${{ inputs.version }}
upload_artifacts: true
publish-reme:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download ReMe distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-distributions
path: dist/reme
- name: Publish ReMe
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/reme
skip-existing: true

View file

@ -0,0 +1,158 @@
# Release checklist:
# 1. Update reme_studio/pyproject.toml, package.json, and package-lock.json to the same Studio version.
# 2. Configure npm Trusted Publishing and PyPI Trusted Publishing with the pypi environment.
# 3. Run this workflow manually with the exact Studio version.
name: Release / ReMe Studio
run-name: Publish ReMe Studio ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Version from the Studio Python and npm manifests
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
permissions:
contents: read
concurrency:
group: publish-reme-studio
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.11"
- name: Validate Studio package names and version
run: |
python - <<'PY'
import json
import os
import tomllib
from pathlib import Path
studio = Path("reme_studio")
python_manifest = tomllib.loads((studio / "pyproject.toml").read_text(encoding="utf-8"))["project"]
npm_manifest = json.loads((studio / "package.json").read_text(encoding="utf-8"))
expected = os.environ["RELEASE_VERSION"].removeprefix("v")
if python_manifest["name"] != "reme_studio":
raise SystemExit(f"Unexpected Python package name: {python_manifest['name']}")
if npm_manifest["name"] != "@agentscope-ai/reme_studio":
raise SystemExit(f"Unexpected npm package name: {npm_manifest['name']}")
if python_manifest["version"] != expected or npm_manifest["version"] != expected:
raise SystemExit(
f"Studio manifests are {python_manifest['version']} and {npm_manifest['version']}; "
f"workflow input is {expected}",
)
prerelease = "-" in expected
if prerelease != (os.environ["NPM_TAG"] == "next"):
raise SystemExit("Prereleases must use next; stable releases must use latest")
PY
- name: Install dependencies and run checks
working-directory: reme_studio
run: |
npm ci
npm run format:check
npm run lint
npm test
- name: Build Studio distributions
run: |
python -m pip install build twine
mkdir -p dist/studio-python dist/studio-npm
npm pack ./reme_studio --pack-destination dist/studio-npm
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio-python
python -m twine check dist/studio-python/*
- name: Verify Studio distributions and isolated installation
run: |
STUDIO_WHEEL="$(pwd)/$(ls dist/studio-python/reme_studio-*.whl)"
tar -tzf dist/studio-npm/*.tgz | grep '^package/dist-static/index.html$'
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-studio-${{ inputs.version }}
path: |
dist/studio-python/*
dist/studio-npm/*
if-no-files-found: error
publish-python:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Publish ReMe Studio to PyPI
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/studio-python
skip-existing: true
publish-npm:
needs: build
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "24"
registry-url: https://registry.npmjs.org
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Publish ReMe Studio to npm
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/studio-npm/*.tgz --access public --tag "${NPM_TAG}" --provenance

167
.github/workflows/release-typescript.yml vendored Normal file
View file

@ -0,0 +1,167 @@
# Release checklist:
# 1. Update typescript/package.json and package-lock.json to the release version and merge them.
# 2. Configure npm Trusted Publishing for agentscope-ai/ReMe and this workflow file.
# 3. Run this workflow manually with the exact package version (an optional v prefix is accepted).
# 4. Configure ClawHub Trusted Publishing or CLAWHUB_TOKEN before enabling ClawHub publication.
# 5. Use the `next` tag for prereleases and `latest` only for stable releases.
name: Release / TypeScript integrations
run-name: Publish @agentscope-ai/reme ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Version from typescript/package.json (for example, 0.1.0)
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
publish_clawhub:
description: Also publish the verified tarball to ClawHub
required: true
default: false
type: boolean
permissions:
contents: read
concurrency:
group: publish-agentscope-ai-reme
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.validate.outputs.version }}
env:
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
- name: Validate package name and release version
id: validate
working-directory: typescript
run: |
node --input-type=module <<'JS'
import { appendFileSync, readFileSync } from 'node:fs';
const manifest = JSON.parse(readFileSync('package.json', 'utf8'));
const expected = process.env.RELEASE_VERSION.replace(/^v/, '');
if (manifest.name !== '@agentscope-ai/reme') {
throw new Error(`Unexpected package name: ${manifest.name}`);
}
if (manifest.version !== expected) {
throw new Error(`package.json is ${manifest.version}, workflow input is ${expected}`);
}
const prerelease = manifest.version.includes('-');
const npmTag = process.env.NPM_TAG;
if (prerelease !== (npmTag === 'next')) {
throw new Error(prerelease
? 'Prerelease versions must use the next npm tag'
: 'Stable versions must use the latest npm tag');
}
console.log(`Preparing ${manifest.name}@${manifest.version}`);
appendFileSync(process.env.GITHUB_OUTPUT, `version=${manifest.version}\n`);
JS
- name: Install dependencies
working-directory: typescript
run: npm ci
- name: Type-check and test
working-directory: typescript
run: |
npm run format:check
npm run lint
npm run typecheck
npm test
npm run test:package
npx --yes clawhub@0.23.3 package validate . --json
- name: Pack npm tarball
working-directory: typescript
run: |
mkdir -p "${RUNNER_TEMP}/reme-typescript-package"
npm pack --pack-destination "${RUNNER_TEMP}/reme-typescript-package"
- name: Upload npm tarball
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: agentscope-ai-reme-${{ inputs.version }}
path: ${{ runner.temp }}/reme-typescript-package/*.tgz
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- name: Set up Node for npm
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '24'
registry-url: https://registry.npmjs.org
- name: Download npm tarball
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: agentscope-ai-reme-${{ inputs.version }}
path: dist/typescript
- name: Reject an existing package version
env:
PACKAGE_VERSION: ${{ inputs.version }}
run: |
PACKAGE_VERSION="${PACKAGE_VERSION#v}"
if npm view "@agentscope-ai/reme@${PACKAGE_VERSION}" version >/dev/null 2>&1; then
echo "@agentscope-ai/reme@${PACKAGE_VERSION} already exists" >&2
exit 1
fi
- name: Publish to npm
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/typescript/*.tgz --access public --tag "${NPM_TAG}" --provenance
publish-clawhub:
if: ${{ inputs.publish_clawhub }}
needs: build
permissions:
actions: read
contents: read
id-token: write
uses: openclaw/clawhub/.github/workflows/package-publish.yml@87ca030c30f3cfb78ab15c8e66b5ff1469c8f9c8 # v0.23.3
with:
owner: agentscope-ai
family: code-plugin
version: ${{ needs.build.outputs.version }}
tags: ${{ inputs.npm_tag }}
source_repo: ${{ github.repository }}
source_commit: ${{ github.sha }}
source_ref: ${{ github.ref }}
source_path: typescript
package_artifact_name: agentscope-ai-reme-${{ inputs.version }}
wait_for_publication: true
secrets:
clawhub_token: ${{ secrets.CLAWHUB_TOKEN }}

46
.github/workflows/security-codeql.yml vendored Normal file
View file

@ -0,0 +1,46 @@
name: Security / CodeQL
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 1 * * 1'
workflow_dispatch:
permissions:
actions: read
contents: read
packages: read
security-events: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
analyze:
name: Analyze ${{ matrix.language }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- name: Checkout repository
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
languages: ${{ matrix.language }}
build-mode: none
- name: Perform CodeQL analysis
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
category: /language:${{ matrix.language }}

7
.gitignore vendored
View file

@ -30,7 +30,9 @@ htmlcov/
# Packaging / build outputs
build/
dist/
node_modules/
*.egg-info/
typescript/reports/
# Logs / temporary files
*.log
@ -46,6 +48,7 @@ temp*/
# ReMe runtime data
.reme/
reme_workspace/
reme_workspace_auto_fin_real_test*/
vault/
*.db
*.sqlite
@ -56,6 +59,10 @@ docs/_build/
site/
evaluation/
# The pi-Bench suite ships its own trace-history render config, which must
# stay in git even though it lives under an evaluation/ directory.
!benchmark/pibench/config/bench/evaluation/
!benchmark/pibench/config/bench/evaluation/**
datasets/
# Claude Code skills (local only)

273
AGENTS.md
View file

@ -1,20 +1,19 @@
# AGENTS.md
This file guides coding agents working in the ReMe repository. Keep changes small,
testable, and consistent with the contracts already expressed by the code.
This file guides coding agents working in the ReMe repository. Keep changes small, testable, and consistent with the
contracts expressed by the current code.
## Project Principles
ReMe is a local-first, file-native memory system for agents.
- User-owned memory files are the source of truth.
- Indexes, caches, metadata, and generated state must be rebuildable.
- Prefer transparent formats and behavior over hidden state.
- Preserve user control over storage, configuration, and service boundaries.
- User-owned workspace files are the durable source of truth.
- Indexes, catalogs, graphs, caches, and generated metadata must remain rebuildable.
- Prefer transparent formats and predictable behavior over hidden state.
- Preserve user control over workspace paths, configuration, and service boundaries.
- Keep concepts focused on project intent; let code and schemas describe implementation.
When a proposed convenience conflicts with these principles, favor data ownership,
recoverability, and predictable behavior.
When convenience conflicts with these principles, favor data ownership, recoverability, and explicit behavior.
## Sources of Truth
@ -22,166 +21,194 @@ Use this order when documentation and implementation disagree:
1. Current code and public Pydantic schemas.
2. Tests that describe supported behavior.
3. CLI help and the built-in configuration.
4. Development documentation and historical notes.
3. CLI behavior and the built-in configuration.
4. README files and other development documentation.
Do not copy large implementation descriptions into documentation. Link to the relevant
module or express the stable contract instead. If behavior changes intentionally, update
the code, schema, tests, configuration, and concise documentation together as needed.
Do not duplicate large implementation descriptions in documentation. Express the stable contract and link to the
relevant module where useful. When behavior changes intentionally, update the implementation, schemas, tests, defaults,
and concise documentation together.
## Repository Map
- `reme/reme.py`: CLI entry point and client/server dispatch.
- `reme/application.py`: application assembly, dependency ordering, and lifecycle.
- `reme/components/application_context.py`: application-wide wiring and shared in-memory metadata.
- `reme/components/runtime_context.py`: scratch state shared by steps within one execution.
- `reme/config/default.yaml`: built-in jobs, components, and defaults.
- `reme/schema/`: public and runtime Pydantic contracts.
- `reme/components/`: services, stores, clients, jobs, and component registration.
- `reme/steps/`: executable job steps.
- `tests/unit/`: primary fast validation suite.
- `tests/integration/`: tests that may require real credentials or services.
- `tests/vector/` and `tests/light/`: specialized suites.
- `plugins/reme/`: Claude Code integration.
- `skills/reme_memory/`: skill that communicates with the ReMe service.
- `skills/qwenpaw_memory/`: separate direct-file memory convention; it does not call ReMe.
- `docs/`: pages and assets that support the repository README; not the deployed docs site.
- `reme/reme.py`: CLI entry point; dispatches `start`, `find_reme`, and client calls.
- `reme/application.py`: application assembly, dependency ordering, job execution, and lifecycle.
- `reme/config/config_parser.py`: YAML/JSON loading, environment expansion, dot-notation parsing, and deep config
merging.
- `reme/config/default.yaml`: default service, jobs, steps, and components. Other files in
`reme/config/` are named configuration variants.
- `reme/schema/application_config.py`: typed application, component, and job configuration.
- `reme/schema/`: request, response, streaming, memory, graph, and file contracts.
- `reme/components/application_context.py`: application-wide wiring and in-memory shared state.
- `reme/components/runtime_context.py`: request-scoped data, response, streaming queue, and stop event.
- `reme/components/base_component.py`: component lifecycle, dependency binding, and workspace helpers.
- `reme/components/component_registry.py`: the frozen built-in registry template and application-local registry factory.
- `reme/components/job/`: base, stream, background, and cron job implementations.
- `reme/components/service/`: local CLI, HTTP, and MCP service backends.
- `reme/components/`: agent wrappers, model adapters, stores, catalogs, graphs, indexes, clients, tokenizers, and
outbound proxies.
- `reme/steps/`: registered job steps grouped by common, file I/O, index, evolve, cookbook, benchmark, and transfer
concerns.
- `reme/utils/`: shared utilities, including service discovery, logging, web-static resolution, session I/O, token
accounting, and wikilink handling.
- `tests/unit/`: primary fast, isolated validation suite.
- `tests/integration/`: service/model tests that may need credentials or external processes.
- `reme_studio/`: ReMe Studio frontend source plus the independently published `reme_studio` Python package and
`@agentscope-ai/reme_studio` npm static distribution.
- `typescript/`: the independently published `@agentscope-ai/reme` package, including the shared TypeScript client and
DeepSeek Harness and OpenClaw adapters.
- `plugins/`: installable ReMe extensions, such as Auto Fin.
- `integrations/`: adapters that connect ReMe to external agent hosts, such as Claude Code, DSH, and Hermes Agent.
- `skills/`: standalone skills; `reme_memory` calls ReMe, while other skills may use separate tools or direct-file
conventions.
- `benchmark/` and `cookbook/`: runnable evaluations and example workflows.
- `docs/`: README-linked supporting pages and figures.
## Development Setup
ReMe requires Python 3.11 or newer.
ReMe requires Python 3.11 or newer. Install the editable development environment with:
```bash
pip install -e ".[dev,core]"
pip install -e reme_studio -e ".[dev,core]"
```
Before changing behavior, inspect the adjacent implementation, schemas, configuration,
and focused tests. Follow existing patterns unless the task explicitly calls for a new
contract or architecture.
Before changing behavior, inspect the adjacent implementation, schema, built-in config, and focused tests. Follow
existing async and typing patterns unless the task explicitly requires a new contract.
## Change Workflow
## Configuration and CLI Contracts
1. Identify the narrowest supported contract affected by the request.
2. Read the relevant implementation and tests before editing.
3. Make the smallest coherent change; avoid unrelated cleanup.
4. Update related schemas, defaults, registrations, and imports when required.
5. Add or adjust focused tests for observable behavior.
6. Run proportionate validation and report anything not run.
- CLI syntax is `reme ACTION key=value ...`; leading `-` or `--` on arguments is accepted.
- Nested overrides use dot notation. Values support null, booleans, numbers, JSON collections, and quoted JSON strings;
leading-zero numeric-looking values remain strings.
- `config=<name-or-path>` loads a discovered config name or a `.yaml`, `.yml`, or `.json` file. With no explicit config
path, `default` is loaded when available.
- Config files expand `${VAR}` and `${VAR:-default}` recursively. An undefined variable without a default is an error.
- CLI/config overrides are deep-merged over the loaded file. Do not silently change this merge behavior or stable
configuration keys.
- `ApplicationConfig` normalizes `workspace_dir` to an expanded absolute path. `session_dir`
must remain workspace-relative; standard transcripts live under `{session_dir}/dialog`.
- `reme start` runs the configured service. `reme start job=<name> ...` switches to the one-shot CLI service and runs
the job through the normal application lifecycle.
- Other actions use a client selected from the running service configuration when discoverable, otherwise from local
config. Client-selection arguments must not leak into the job payload.
Component and step discovery depends on registration imports:
## Registration and Application Lifecycle
- Components use `R.register(...)` in `reme/components/component_registry.py`.
- Component packages must be reachable through `reme/components/__init__.py`.
- Step modules must be reachable through `reme/steps/__init__.py`.
Component and Step discovery is import-driven:
Adding an implementation without its registration import can leave it undiscoverable at
runtime. Treat the implementation, registry entry, and import side effect as one change.
- Implementations declare a non-`BASE` `component_type` and register with `@R.register("backend")`
or `R.register(Class, "backend")`.
- Component packages must be imported through `reme/components/__init__.py`.
- Step packages/modules must be reachable through their package `__init__.py` chain and ultimately
`reme/steps/__init__.py`.
- Adding an implementation without its registration import leaves it undiscoverable at runtime. Treat implementation,
registration, import side effect, defaults, and tests as one change.
Do not silently change stable CLI flags, configuration keys, workspace layouts, serialized
schemas, or service interfaces. When such a change is required, preserve compatibility
where practical and make the migration explicit.
`Application` validates config through `ApplicationContext`, creates workspace directories, instantiates the service,
configured components, and jobs, and then manages lifecycle as follows:
## Step State Model
- Components start in topological dependency order. Missing required dependencies and cycles fail explicitly; optional
dependencies may resolve to `None`.
- Jobs start after components in this order: base jobs, stream jobs, background jobs, then cron jobs.
- Shutdown closes everything in reverse start order and then shuts down the optional thread pool.
- If startup fails, already-started resources are closed.
- `BaseComponent.start()` and `close()` are lock-protected and idempotent. Dependencies created by a standalone
`default_factory` are owned and closed by the parent component.
Treat every Step as stateless. `BaseJob` stores Step specifications and builds fresh Step
instances for each Job invocation. A Step instance must not use `self` or class variables to
retain mutable runtime state between calls.
Keep async clients, tasks, executors, and services under this lifecycle. Do not introduce an untracked long-lived
resource.
Place state according to its lifetime:
## Jobs, Steps, and State
- Constructor fields on `self`: immutable Step configuration and resolved dependencies only.
- `self.context` (`RuntimeContext`): request data and intermediate results for one Job
execution; sequential Steps share this context.
- `self.app_context.metadata`: in-memory state that must be shared across Step or Job
invocations for the lifetime of the Application.
- Workspace files or a dedicated Component/store: durable state that must survive an
Application restart.
`BaseJob` resolves configured Step classes during job startup and constructs fresh Step instances for every invocation.
Job-level kwargs are merged into each `RuntimeContext`, with call-time kwargs taking precedence. Sequential Steps in one
invocation share the same `RuntimeContext` and `Response`.
Use narrow, namespaced keys in `app_context.metadata`, following existing patterns such as
`tool_contexts`. The ApplicationContext is shared, so account for
concurrent access when values are mutable. New Step code must not fall back to `self.kwargs`
or another Step field to emulate shared state when `app_context` is absent; tests of shared
state should construct an `ApplicationContext`. If shared state grows into a stable
service-level contract or needs its own lifecycle, locking, or persistence, promote it to a
typed ApplicationContext field or a dedicated Component instead of expanding an ad hoc
metadata bucket.
Treat Step instances as invocation-scoped:
Do not use `Response.metadata` as a state store. It is request-scoped output for callers and
diagnostics, distinct from `ApplicationContext.metadata`.
- Constructor fields and `self.kwargs` hold Step configuration and resolved dependencies. They may be cached or adjusted
during that one invocation, but must not be relied on across Job calls.
- `self.context.data` holds request inputs and intermediate values shared by sequential Steps.
- `self.context.response.answer`, `success`, and `metadata` are request-scoped output. Because the same response travels
through the Step chain, later Steps may consume metadata produced earlier, but it is not application-lifetime or
durable storage.
- `self.app_context.metadata` holds in-memory state shared across Job/Step invocations for the life of one
`Application`, such as counters, tool-context state, session maps, or locks.
- Workspace files or a dedicated Component/store hold durable state that must survive restart.
Use narrow, namespaced keys in `app_context.metadata` and protect shared mutable values against concurrent access. The
search/draft helpers intentionally mirror tool-context state into
`self.kwargs` only when no `ApplicationContext` exists for standalone use and unit tests; do not generalize that
compatibility fallback into persistent runtime state. If shared state becomes a stable service contract or needs
dedicated lifecycle, locking, or persistence, promote it to a typed context field or Component.
Additional Step contracts:
- `Ref` dependencies resolve in this order: Step kwargs, current `RuntimeContext`, then the named application component.
The value is cached only on the current Step instance and cleared before each call.
- `input_mapping` and `output_mapping` copy keys within `RuntimeContext.data`; missing sources are ignored.
- Dispatched Steps receive the current `RuntimeContext`, so their data and response are shared.
- Base jobs convert uncaught Step errors into `Response(success=False)`; stream jobs emit an error chunk and always a
terminal `DONE`; background jobs let errors reach their supervisor.
- Background jobs are never service-exposed. MCP also skips stream jobs. Respect `enable_serve`
and any configured service job allowlist.
## Workspace and File Safety
- Application startup creates the workspace plus configured metadata, session, memory-session, resource, daily, and
digest directories.
- File-operation paths are resolved against the workspace and must stay inside it. Home-relative paths are unsupported,
traversal escapes are rejected, and `_allowed_paths` restrictions fail closed when invalid.
- Preserve per-path locking, encoding detection, byte limits, truncation behavior, and optimistic
`expected_mtime` checks when modifying file operations.
- Do not bypass the existing file steps or stores in a way that weakens workspace containment.
- Never write test state into the repository's `.reme/`; use `tmp_path` or another isolated workspace.
- Do not delete or rewrite user memory to repair an index or make a test pass. Rebuild derived state from source files
instead.
## Validation
Use the narrowest useful check while iterating, then broaden it according to risk.
Run a focused test:
Focused test:
```bash
pytest tests/unit/path/to/test_file.py -v
```
Run the main unit suite:
Main unit suite:
```bash
pytest tests/unit -v --tb=long -s --log-cli-level=WARNING
```
Run repository formatting and lint checks when the change warrants it:
Repository formatting and lint checks:
```bash
pre-commit run --all-files
```
Formatting and lint configuration is authoritative. Python code currently uses a maximum
line length of 120 for Black and Flake8, with Pylint also run by pre-commit.
Black and Flake8 use a 120-character line limit and Python 3.11 formatting; Pylint is also run by pre-commit. If
`reme_studio/` changes, use its Node 22.13+ scripts and run the proportionate checks from that directory, such as
`npm run format:check`, `npm run lint`, or `npm test`.
Integration tests may contact real services and require credentials such as
`LLM_API_KEY` or `EMBEDDING_API_KEY`. Do not run credentialed or externally mutating tests
automatically. Run them only when the task requires them and the user has supplied or
authorized the necessary environment.
Integration tests may contact real model providers, services, or agent subprocesses and can require credentials. Do not
run credentialed or externally mutating tests automatically; run them only when the task requires them and the necessary
environment has been supplied or authorized. Mock network, model, and subprocess boundaries in unit tests.
## Coding and Test Conventions
- Target Python 3.11+ and follow the surrounding typing and async style.
- Steps are stateless. If a step needs to persist state, store it in
`self.app_context.metadata` rather than on the step instance.
- Keep public schemas explicit and backward-compatible where practical.
- Close async clients, services, tasks, and other lifecycle resources deterministically.
- Prefer clear failures over silently falling back to corrupt or ambiguous state.
- Keep indexes and caches derivable from user-owned source files.
- Use `tmp_path` or another isolated temporary workspace in tests.
- Never write test state into the repository's `.reme/` directory.
- Mock network or model boundaries in unit tests.
- Do not commit `.env` files, credentials, runtime memory, logs, indexes, or caches.
## Documentation Boundaries
ReMe's local docs and the deployed documentation site have separate responsibilities.
- Keep `docs/` focused on content and assets used by `README.md` and `README_ZH.md`.
- Preserve README-linked pages under `docs/en/` and `docs/zh/`, including their relative
paths, unless the README is updated in the same change.
- Keep README-required images under `docs/figure/`.
- Keep the README's main documentation index pointed at `docs.agentscope.io` or the
`agentscope-ai/docs` repository, following the existing link style.
- Do not treat local README-supporting pages as the source for the deployed website.
The separate `agentscope-ai/docs` repository owns website content, navigation, versioning,
and deployment. Public ReMe pages live there under `reme/<version>/`. Make website changes
in that repository and follow its existing version-management conventions.
Do not add website build configuration or deployment workflows to ReMe unless the task
explicitly changes this repository boundary.
## Agent Guardrails
## Change Guardrails
- Preserve unrelated user changes in a dirty working tree.
- Do not edit generated output when the source can be changed instead.
- Do not delete or rewrite user data to make a test pass.
- Avoid broad refactors unless they are necessary for the requested outcome.
- Do not introduce dependencies without a concrete need and repository-level justification.
- Treat network access, real credentials, and external service mutations as opt-in.
- State which validations passed and which were not run in the final handoff.
- Make the smallest coherent change and avoid unrelated cleanup or broad refactors.
- Do not edit generated output when the source can be changed instead. The publish workflow builds
`reme_studio/dist-static` and stages it under `reme_studio/src/reme_studio/static`; change `reme_studio/` source for
frontend work.
- Do not silently change CLI flags, configuration keys, workspace layouts, serialized schemas, endpoint shapes,
streaming termination, or service interfaces. Preserve compatibility where practical and document intentional
migrations.
- Do not introduce dependencies without a concrete repository-level need.
- Do not commit `.env` files, credentials, runtime memory, logs, indexes, caches, benchmark outputs, or generated
Studio distributions.
- State which validations passed and which relevant checks were not run in the final handoff.
If a requirement is ambiguous, first infer intent from nearby code, tests, and schemas. Ask
the user only when the remaining choice would materially alter a public contract, user data,
or external system.
If a requirement is ambiguous, infer intent from nearby code, schemas, defaults, and tests. Ask the user only when the
remaining choice would materially alter a public contract, user data, or an external system.

392
README.md
View file

@ -1,5 +1,5 @@
<p align="center">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p>
<p align="center">
@ -8,7 +8,7 @@
<a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a>
<a href="https://docs.agentscope.io/reme"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="Documentation"></a>
<a href="https://reme.agentscope.io"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="Documentation"></a>
<a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a>
<a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a>
@ -20,50 +20,44 @@
</p>
<p align="center">
<strong>An agent memory layer that turns conversations and resources into readable, editable, searchable Markdown memory.</strong><br>
<strong>A local-first, self-evolving personal knowledge base for AI agents.</strong><br>
</p>
> Previous versions: [0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) ·
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
🧠 ReMe is a local-first memory layer for **AI agents**. It turns conversations and resources into file-based long-term
memory, then continuously indexes, links, and consolidates that memory for future recall.
## ✨ Why ReMe?
## ✨ Core Ideas
🧠 ReMe turns conversations and resources into readable, editable, searchable, and interconnected Markdown memory. Agents
such as QwenPaw and DeepSeek Harness can share the same workspace to retrieve, maintain, and evolve knowledge, while
users retain control of the durable files.
- **Memory as File**: Markdown files with frontmatter and wikilinks serve as memory nodes that both users and agents can
read and write directly.
- **Self-evolving knowledge base**: Auto Memory, Auto Resource, and Auto Dream progressively transform conversations and
resources into long-term memories, while automatically building wikilink relationships.
- **Progressive hybrid search**: ReMe combines wikilinks, BM25, and embeddings for hybrid retrieval across keyword
matching, semantic recall, and relationship expansion.
- **Agent-friendly integration**: SKILL.md + CLI integration makes it easy for different agents to read, write,
maintain, and reuse memory.
- **Memory as File, File as Memory**: ReMe stores durable memory as ordinary Markdown with frontmatter and wikilinks.
Users and agents can inspect, edit, move, sync, and back it up with familiar tools, while indexes and generated
metadata remain rebuildable.
- **Self-evolving knowledge base**: ReMe progressively turns conversations and resources into daily notes and long-term
knowledge, preserving sources while refining facts, preferences, procedures, and relationships over time.
- **Recall is precise and context-aware.** BM25, optional embeddings, and wikilink expansion retrieve relevant
line-level passages and their relationships without loading the entire knowledge base into the agent context.
- **One memory workspace works across agents.** Personal assistants, coding agents, and other agent runtimes can share
the same local workspace through native integrations, SKILL.md, CLI, HTTP, MCP, or Python APIs.
<p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe Design Philosophy" width="92%">
</p>
## 🔭 Use Cases
## 📰 Latest Updates
- **Personal assistants**: Give personal assistants such as
[QwenPaw](https://github.com/agentscope-ai/QwenPaw), [OpenClaw](https://github.com/openclaw/openclaw), and
[Hermes](https://github.com/nousresearch/hermes-agent) a user-editable long-term memory layer.
- **Coding agents**: Preserve coding style, project background, repository decisions, and workflow experience across
sessions when integrating with coding agents such as [Claude Code](plugins/reme).
- **LLM Wiki**: Turn conversations, notes, and resources into a searchable, traceable, and linked Markdown knowledge
base that both users and agents can maintain.
- **Self-evolving agents**: Support agents that learn from experience by saving successful paths, failed attempts,
reusable procedures, and periodic reflections as memory.
## 📰 News
- [2026.08] - [Experience-driven enhancement method](benchmark/toolmemory/README.md) of agent tool-use
execution built on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403).
- [2026.07] - Introduced optional Cookbooks: [Daily Paper](cookbook/daily_paper/README.md) for paper discovery and
analysis, and [Auto Fin](cookbook/auto-fin/README.md) for file-native ETF event research based on CLS news and
historical market reactions.
- [2026.08] - Published [`@agentscope-ai/reme`](https://www.npmjs.com/package/@agentscope-ai/reme), providing native
ReMe memory integrations for DeepSeek Harness and OpenClaw plus a shared TypeScript HTTP client.
- [2026.08] - Published the [ReMe blog](https://agentscope-ai.github.io/ReMe/?doc=en-reme-blog), an end-to-end introduction to its local-first memory
architecture, self-evolving workflows, hybrid search, proactive discovery, and benchmark results.
- [2026.08] - [Experience-driven enhancement method](https://reme.agentscope.io/?doc=toolmemory-en) of agent tool-use execution built
on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403).
- [2026.07] - Introduced optional plugins: [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) for paper discovery and
analysis, and [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-en) for researching the latest 24 hours of topic-related CLS news
with local-memory search and validated historical wikilinks.
- [2026.07] - Our
paper [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/)
has been accepted to Findings of ACL 2026.
@ -85,33 +79,14 @@ Install from source:
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"
pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
```
### Environment Variables
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are
disabled by default, so the default setup does not start an embedding model or require an embedding API key.
```bash
cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.
> [!NOTE]
> To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and
> `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change
> `components.file_store.default.embedding_store` from `""` to `default`. See the
> [memory search guide](docs/en/memory_search.md) for details.
The static build requires Node.js 22.13 or newer and makes Studio available from the source tree.
### Start the Service
@ -126,10 +101,10 @@ reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181
```
After startup, check the service status. If you use a custom port, replace `2333` in the URL below with that port.
```bash
reme version
reme health_check
reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
```
@ -167,105 +142,51 @@ ReMe stores agent memory as readable Markdown.
Related: [[digest/wiki/memory-as-file.md]]
```
## 🧑‍🍳 Cookbooks
### ReMe Studio (Optional)
Cookbooks are optional, end-to-end workflows assembled from ReMe jobs and steps. They are not enabled by the default
configuration; select the cookbook's standalone configuration when starting ReMe. Each new cookbook will be added as
another row in this table.
The `core` installation includes Studio. After starting ReMe, open <http://127.0.0.1:2333/> to browse, edit, and search
the workspace. To add Studio to a base installation, use `pip install "reme-ai[web]"`. See the
[ReMe Studio guide](https://reme.agentscope.io/?doc=studio-en) for source builds, configuration, and development.
| Cookbook | Capability |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| [Daily Paper](cookbook/daily_paper/README.md) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](cookbook/auto-fin/README.md) | Match CLS events to liquid ETFs, study historical reactions, and generate file-native research reports. |
### Optional Model Configuration
## 📁 Memory System
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are
disabled by default, so the default setup does not start an embedding model or require an embedding API key.
> Memory as File, File as Memory.
```bash
cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
ReMe treats **memory as files**, progressively processing raw conversations and external resources from `session/` and
`resource/` into `daily/`, then consolidating them into reusable long-term memory nodes under `digest/`.
### Directory Structure
```text
<workspace_dir>/
├── metadata/ # Persistent system state such as indexes, graphs, and catalogs
├── session/ # Raw conversations and agent sessions
│ ├── dialog/
│ │ └── <session_id>.jsonl
│ ├── agentscope/
│ └── claude_code/
├── resource/ # External raw materials
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # Lightly processed memory: daily facts, conversation summaries, resource readings
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <session_event>.md
│ ├── <resource_stem>.md
│ └── interests.yaml
└── digest/ # Long-term memory: personal facts, procedural experience, knowledge nodes
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe file-based memory system overview" width="92%">
</p>
Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.
## 🧭 Memory Design Philosophy
> [!NOTE]
> To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and
> `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change
> `components.file_store.default.embedding_store` from `""` to `default`. See the
> [memory search guide](docs/en/memory_search.md) for details.
> Capture raw dialogs and resources, refine them into long-term preferences, reusable experience, and valuable
> knowledge,
> while keeping the result editable by humans and agents.
## 🤝 Use ReMe with Your Agent
### Automatic Memory Flow
ReMe can run as a local memory service accessed through the CLI, HTTP API, or MCP server, or it can be embedded in the
host process through its Python API. Host integrations can add memory guidance, recall, and capture to the agent
lifecycle according to the capabilities of each runtime.
ReMe follows a capture → index → consolidate → recall loop. Conversations and resources first become daily memory cards;
background jobs keep files searchable; `auto_dream` distills stable knowledge into `digest/`; agents recall memory
through search, wikilinks, or proactive topics.
| Capability | Entry point | What it does | Output |
|---------------------------------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| [`auto_memory`](docs/en/auto_memory.md) | Agent hook or `reme auto_memory` | Distills useful conversation facts while preserving the raw session. | `session/dialog/*.jsonl`, `daily/<date>/<session>.md` |
| [`auto_resource`](docs/en/auto_resource.md) | Resource watcher or `reme auto_resource` | Turns files under `resource/<date>/` into source-linked daily cards. | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/en/memory_search.md) | Background watcher or `reme reindex` | Maintains chunks, the BM25 index, the wikilink graph, and the optional embedding index. | Searchable `daily/`, `digest/`, and `resource/` content |
| [`auto_dream`](docs/en/auto_dream.md) | `dream_cron` or `reme auto_dream` | Consolidates changed daily cards into long-term personal, procedure, and wiki memory. | `digest/**`, `daily/<date>/interests.yaml` |
| [`proactive`](docs/en/proactive.md) | `reme proactive` before an agent decides to act | Reads topics generated by `auto_dream`; the host agent decides whether and how to mention them. | Structured topics from `daily/<date>/interests.yaml` |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
## 🤝 Agent-friendly Integration
ReMe runs as a local memory service and offers multiple integration paths: CLI, HTTP API, MCP server, and SDK. Different
agents can choose the path that fits their runtime while sharing the same local memory workspace.
| Agents | Recommended path | What works out of the box |
|------------------------------------------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
| **QwenPaw** | Embed ReMe via the Python SDK. | Reuse the app's own lifecycle and model config while keeping memory local and file-based. |
| **Claude Code** | Start ReMe as an MCP service and install [plugins/reme](plugins/reme). | MCP recall tools, a `reme-memory` skill, and a Stop hook that records sessions automatically. |
| **Other CLI-capable agents (OpenClaw/Hermes/Codex)** | Copy or install [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md). | Search/read/write memory and call `auto_memory`, `auto_dream`, and `proactive` via the CLI. |
| Agent | Recommended path | Available after integration |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **DeepSeek Harness** | Install [`@agentscope-ai/reme`](typescript/README.md#deepseek-harness) with `dsh plugin --profile web add @agentscope-ai/reme`. | Long-term memory guidance, the `reme_search` tool, and automatic capture of completed main-agent turns. |
| **OpenClaw** | Install [`@agentscope-ai/reme`](typescript/README.md#openclaw) with `openclaw plugins install @agentscope-ai/reme`. | Native memory tools, recall before user-triggered runs, and automatic turn capture. |
| **QwenPaw** | Embed ReMe in-process through its Python API. | Reuse the host lifecycle and model config while keeping memory local and file-based. |
| **Claude Code** | Start the streamable HTTP MCP service and install [the ReMe plugin](integrations/claude_code/reme). | MCP recall tools, the `reme-memory` skill, and a Stop hook that records sessions automatically. |
| **Hermes** | Start the HTTP service and install [the ReMe provider](integrations/hermes_agent). | Recall before model calls and asynchronous `auto_memory` after each completed turn. |
| **Codex and other CLI agents** | Install or copy the [ReMe Memory skill](skills/reme_memory/SKILL.md). | Search, read, and write memory through the CLI; automatic capture requires host lifecycle integration. |
<p align="center"><b>Integration demos</b></p>
@ -295,39 +216,166 @@ agents can choose the path that fits their runtime while sharing the same local
</tr>
</table>
## 🛠️ ReMe Operations
## 🧠 How ReMe Works
ReMe operates the workspace through a unified job interface exposed by the CLI. Agents usually only need retrieval,
reading, writing, editing, and automatic memory commands. Lower-level indexing, frontmatter, and file operation commands
are mainly for maintenance, debugging, or advanced integration. Run `reme help` for the full job list.
> Memory as File, File as Memory.
| Command | Purpose |
|-------------------------------------------|----------------------------------------------------------------------------------------|
| `reme start` | Start the local ReMe service. |
| `reme version` / `reme health_check` | Check package and component status. |
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](docs/en/memory_search.md) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme auto_memory` | Turn conversation messages into daily memory cards. Requires LLM credentials. |
| `reme auto_resource` | Interpret files under `resource/` into daily resource cards. Requires LLM credentials. |
| `reme auto_dream` / `reme proactive` | Consolidate daily memory into long-term digest and surface topics worth attention. |
| `reme reindex` | Rebuild search and wikilink indexes from existing files. |
ReMe treats **memory as files**, progressively processing filtered conversation source records and external resources
from `session/` and `resource/` into `daily/`, then `digest/`. The default workspace is `.reme/` under the current
directory; `workspace_dir=...` selects a different user-owned location.
## 🤝 Community and Support
### Workspace Layout
- **Issues and requests**: Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) first. If there is no
related discussion, open a new issue with background, expected behavior, and impact scope.
- **Code contributions**: Before making changes, read
the [contribution guide](https://docs.agentscope.io/reme/latest/en/contribution). Source, schemas, and tests are the
authoritative architecture and extension guide.
- **Documentation contributions**: Submit user-facing documentation changes to the
[unified documentation repository](https://github.com/agentscope-ai/docs) under `reme/<version>/{en,zh}/`.
```text
<workspace_dir>/
├── metadata/ # Rebuildable indexes, graphs, catalogs, and caches
├── session/ # Conversation source records and agent sessions
│ ├── dialog/
│ │ └── <session_id>.jsonl # Source messages saved by auto_memory
│ └── claude_code/
│ └── <session_id>.jsonl # ReMe copy used by auto_memory_cc
├── mem_session/ # Generated agent-wrapper sessions/config, not user memory
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # External raw materials
│ ├── <resource>.<ext> # Root-level files enter today's daily layer
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # Lightly processed memory: daily facts, conversation summaries, resource readings
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <generated_name>.md # Topic-named conversation or resource card
│ └── interests.yaml
└── digest/ # Long-term memory: personal facts, procedural experience, knowledge nodes
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe file-based memory system overview" width="92%">
</p>
### Memory Lifecycle
ReMe follows a capture → index → consolidate → recall loop. Workspace files remain the durable source of truth;
everything under `metadata/` is rebuildable.
| Capability | Entry point | What it does | Output |
| ------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [`auto_memory`](docs/en/auto_memory.md) | Agent hook or `reme auto_memory` | Distills useful conversation facts while preserving a filtered conversation source record. | `session/dialog/*.jsonl`, `daily/<date>/<generated-name>.md` |
| [`auto_resource`](docs/en/auto_resource.md) | Resource watcher or `reme auto_resource` | Turns files under `resource/` into source-linked, content-named daily cards. | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/en/memory_search.md) | Background watcher or `reme reindex` | Live-indexes Markdown in `daily/` and `digest/`; a full rebuild also scans `resource/` and JSONL. | Searchable chunks, BM25, wikilink graph, and optional vectors |
| [`auto_dream`](docs/en/auto_dream.md) | `dream_cron` or `reme auto_dream` | By default, extracts up to five reusable units from changed files in the latest two-day window, then creates, corroborates, refines, or corrects digest nodes. | `digest/**`, `daily/<date>/interests.yaml` |
| [`proactive`](docs/en/proactive.md) | `reme proactive` before an agent decides to act | Reads topics generated by `auto_dream`; the host agent decides whether and how to mention them. | Structured topics from `daily/<date>/interests.yaml` |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
Search returns matching chunks with line ranges and bounded wikilink neighbors. Optional vector results are fused with
BM25 through reciprocal rank fusion (RRF).
> [!IMPORTANT]
>
> `proactive` only reads and exposes interest topics produced by Auto Dream. It does not independently browse the web,
> send notifications, or rewrite the knowledge base; the host agent decides whether and how to act on a topic.
## 📊 Benchmarks
ReMe evaluates multi-session and long-context memory with agentic search-and-read workflows. The figures below are the
published reference runs in this repository; model, prompt, dataset, and judging details are documented with each
benchmark.
| Benchmark | Setting | Sample size | Agentic score | Focus |
| --------------------------------------------------------------------------- | ------------ | -----------------------: | ------------: | ------------------------------------------------------------------ |
| **[LongMemEval cleaned-s](https://reme.agentscope.io/?doc=longmemeval-en)** | **Overall** | **500 questions** | **89.4%** | Cross-session retrieval, knowledge updates, and temporal reasoning |
| [BEAM](https://reme.agentscope.io/?doc=beam-en) | 100K context | 20 cases / 400 questions | 66.1% | Ten types of long-context memory tasks |
| [BEAM](https://reme.agentscope.io/?doc=beam-en) | 1M context | 35 cases / 700 questions | 65.0% | Ultra-long conversation settings |
ReMe also achieved a **0.580 PROC score across five user personas** in the repository's
[π-Bench evaluation](https://reme.agentscope.io/?doc=pibench-en), 2.4% above NanoBot under the same test-model configuration. PROC
measures proactive handling of hidden intent, clarification, cross-session preferences and conventions, task
dependencies, and underspecified requests.
## 🧩 Extensions and Plugins
Plugins are optional Python distributions that contribute Component, Step, or Job backends and configuration. They are
installed separately and enabled explicitly by configuration. Daily Paper and Auto Fin are independently packaged
plugins; see the source distributions and their documentation for [Daily Paper](plugins/daily_paper/README.md) and
[Auto Fin](plugins/auto-fin/README.md).
| Plugin | Capability |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-en) | Fetch topic-related CLS news, search ReMe history, and generate wikilink-backed Markdown reports. |
See [Plugin Management](docs/en/plugin_management.md) to install, inspect, validate, enable, and uninstall ReMe plugins.
## 📚 Documentation
These guides cover the main user workflows and the runtime contracts implemented by the current code.
| Guide | What you will learn |
| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [Quick Start](docs/en/quick_start.md) | Install ReMe, start the service, and run the first file and memory operations. |
| [Memory as File](docs/en/memory_as_file.md) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. |
| [Auto Memory](docs/en/auto_memory.md) | Preserve source conversations and distill reusable daily memory cards. |
| [Auto Resource](docs/en/auto_resource.md) | Import supported text resources and turn them into source-linked daily cards. |
| [Auto Dream](docs/en/auto_dream.md) and [Auto Link](docs/en/auto_link.md) | Consolidate daily notes into evolving digest nodes and readable wikilink relationships. |
| [Memory Search](docs/en/memory_search.md) | Use BM25, optional vectors, RRF fusion, line-range recall, and progressive link expansion. |
| [Proactive](docs/en/proactive.md) | Read interest topics safely and integrate them into a host agent's decision flow. |
| [Application Scenarios](docs/en/reme_scene.md) | Follow concrete financial research, coding-memory, and personal knowledge-base examples. |
| [Framework](docs/en/framework.md) | Understand Application, Job, Step, Component, service, configuration, and lifecycle boundaries. |
| [TypeScript integrations](typescript/README.md) | Configure the shared client and native DeepSeek Harness and OpenClaw adapters. |
| [ReMe Blog](https://agentscope-ai.github.io/ReMe/?doc=en-reme-blog) | Read the product story, design rationale, examples, and benchmark summary. |
## 🛠️ Common Commands
Run `reme help` for the full job list. Common workspace and maintenance commands are:
| Command | Purpose |
| ----------------------------------------- | --------------------------------------------------------------------------------- |
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](docs/en/memory_search.md) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme traverse` / `reme graph_snapshot` | Explore wikilink neighborhoods or the category-rooted digest graph. |
| `reme chat` | Stream a read-only, workspace-aware agent conversation. Requires LLM credentials. |
| `reme reindex` | Rebuild search and wikilink indexes from existing files. |
## 🤝 Community and Contributing
- **Issues, requests, and help**: Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) first. If there is no
related discussion, open one with the background, expected behavior, and impact scope.
- **Code contributions**: Before making changes, read the repository's
[contribution guide](docs/en/contributing.md). Source, schemas, and tests are the authoritative architecture and
extension guide.
- **Documentation contributions**: Update the canonical files under `docs/en/`, `docs/zh/`, or the relevant package
directory in this repository. The documentation site is generated from these files.
- **Commit convention**: Conventional Commits are recommended, for example `feat(search): add link expansion option` or
`docs(zh): update quick start`.
- **Pre-submit checks**: Before submitting a PR, try to run `pre-commit run --all-files` and `pytest`. If tests that
depend on LLMs, embeddings, or external services cannot run, explain that in the PR.
- **Get help**: Use [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues) for bugs and feature requests. Project
documentation is available at [https://docs.agentscope.io/reme](https://docs.agentscope.io/reme).
- **Documentation**: Visit [reme.agentscope.io](https://reme.agentscope.io).
### Contributors

View file

@ -1,5 +1,5 @@
<p align="center">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p>
<p align="center">
@ -8,7 +8,7 @@
<a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a>
<a href="https://docs.agentscope.io/reme"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="文档"></a>
<a href="https://reme.agentscope.io"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="文档"></a>
<a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a>
<a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a>
@ -20,43 +20,43 @@
</p>
<p align="center">
<strong>一个将对话和资料转化为可读、可编辑、可检索 Markdown 记忆的 Agent 记忆层</strong><br>
<strong>面向 AI Agent 的 local-first 自进化个人知识库</strong><br>
</p>
> 历史版本:[0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) ·
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
🧠 ReMe 是一个面向 **AI 智能体** 的 local-first 记忆层。它把对话和资料沉淀为文件化长期记忆,并持续完成索引、链接和整理,让后续
Agent 能够可靠召回。
## ✨ 为什么选择 ReMe
## ✨ 核心创新
🧠 ReMe 将对话和资料持续沉淀为可读、可编辑、可检索、相互链接的 Markdown 记忆。QwenPaw、DeepSeek Harness 等 Agent
可以共享同一个 workspace共同检索、维护和演化知识而持久文件始终由用户掌控。
- **Memory as File**:以带 frontmatter 和 wikilink 的 Markdown 作为记忆节点,让用户和 Agent 都能直接读写。
- **自进化知识库**:通过 Auto Memory、Auto Resource 和 Auto Dream把对话与资料逐步加工为长期记忆并自动建立 wikilink 关系。
- **渐进式混合搜索**:融合 wikilink、BM25 和 embedding支持从关键词匹配到语义召回、关系扩展的混合检索。
- **Agent 友好集成**:通过 SKILL.md + CLI 接入,方便不同 Agent 读写、维护与复用记忆。
- **Memory as File, File as Memory**ReMe 使用带 frontmatter 和 wikilink 的普通 Markdown 保存持久记忆。用户和 Agent
都可以使用熟悉的工具查看、编辑、移动、同步和备份;索引及生成的元数据均可重建。
- **自进化知识库**ReMe 将对话和资料逐步加工为 daily note 与长期知识,在保留来源的同时,持续提炼事实、偏好、
流程经验及其关系。
- **精准召回所需上下文。** ReMe 结合 BM25、可选 embedding 和 wikilink 展开,召回带行号的相关片段及其关系,无需把整个知识库塞入
Agent 上下文。
- **一个 workspace可供不同 Agent 共同使用。** 个人助理、coding agent 和其他 Agent runtime 可以通过原生集成、SKILL.md、CLI、
HTTP、MCP 或 Python API 共享同一个本地记忆空间。
<p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe 设计理念" width="92%">
</p>
## 🔭 适用场景
- **Personal assistants**:为 [QwenPaw](https://github.com/agentscope-ai/QwenPaw)、
[OpenClaw](https://github.com/openclaw/openclaw)、[Hermes](https://github.com/nousresearch/hermes-agent)
等个人助理提供用户可编辑的长期记忆层。
- **Coding agents**:在接入 [Claude Code](plugins/reme) 等 coding agent 时,跨会话保留代码风格、项目背景、仓库决策和流程经验。
- **LLM Wiki**:把对话、笔记和资料转化为可检索、可追溯、可链接的 Markdown 知识库,由用户和 Agent 共同维护。
- **Self-evolving agents**:帮助 Agent 从经验中学习,把成功路径、失败尝试、可复用流程和阶段性反思沉淀为记忆。
## 📰 新闻
## 📰 最新动态
- [2026.08] - 发布 [`@agentscope-ai/reme`](https://www.npmjs.com/package/@agentscope-ai/reme),提供统一 TypeScript HTTP
client以及 DeepSeek Harness 和 OpenClaw 的原生 ReMe 记忆集成。
- [2026.08] - 发布 [ReMe 博客](https://agentscope-ai.github.io/ReMe/?doc=zh-reme-blog),系统介绍本地优先的记忆架构、自进化工作流、混合检索、
主动发现与评测结果。
- [2026.08] - 基于 ReMe 的智能体工具使用
[经验驱动增强方法](benchmark/toolmemory/README_ZH.md)已发布,见
[经验驱动增强方法](https://reme.agentscope.io/?doc=toolmemory-zh)已发布,见
[arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。
- [2026.07] - 新增可选 Cookbook 工作流:[每日论文](cookbook/daily_paper/README_ZH.md)用于论文发现与解析,
[Auto Fin](cookbook/auto-fin/README_ZH.md)用于结合财联社新闻和历史行情开展文件化 ETF 事件研究。
- [2026.07] - 新增可选插件:[每日论文](https://reme.agentscope.io/?doc=daily-paper-zh)用于论文发现与解析,
[Auto Fin](https://reme.agentscope.io/?doc=auto-fin-zh)用于研究最近 24 小时的主题相关财联社新闻,通过本地记忆搜索回顾历史材料并构建
wikilink。
- [2026.07] -
我们的论文 [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/)
已被 Findings of ACL 2026 接收。
@ -78,33 +78,14 @@ pip install "reme-ai[core]"
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"
pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
```
### 环境变量
如果需要 LLM 驱动的记忆演化或 embedding 检索可以配置环境变量。embedding 默认关闭,因此默认配置不会启动 embedding 模型,也不需要
embedding API key。
```bash
cat > .env <<'EOF'
# 可选:仅在配置中显式启用 embedding 组件后使用。
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# 必须auto_memory、auto_resource 和 auto_dream 需要 LLM。
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取可以先不配置 LLM 凭证。
> [!NOTE]
> 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中
> `components.as_embedding``components.embedding_store` 的注释,并将
> `components.file_store.default.embedding_store``""` 改为 `default`。完整说明见
> [记忆检索文档](docs/zh/memory_search.md)。
静态构建要求 Node.js 22.13 或更高版本,并让源码安装可以直接使用 Studio。
### 启动服务
@ -119,10 +100,10 @@ reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181
```
启动后可以检查服务状态;如果使用了自定义端口,请将下面 URL 中的 `2333` 替换为对应端口。
```bash
reme version
reme health_check
reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
```
@ -160,101 +141,50 @@ ReMe 会把 Agent 记忆保存为可读的 Markdown。
相关链接:[[digest/wiki/memory-as-file.md]]
```
## 🧑‍🍳 Cookbooks
### ReMe Studio可选
Cookbook 是由 ReMe jobs 和 steps 组装而成的可选端到端工作流。默认配置不会开启它们;启动 ReMe 时选择对应的 独立配置即可启用。后续新增的
cookbook 会继续在表格中按行追加。
上面的 `core` 安装已包含 Studio。启动 ReMe 后,打开 <http://127.0.0.1:2333/> 即可浏览、编辑和搜索 workspace。
如需为基础安装单独添加 Studio可使用 `pip install "reme-ai[web]"`。源码构建、配置和开发说明见
[ReMe Studio 指南](https://reme.agentscope.io/?doc=studio-zh)。
| Cookbook | 能力 |
|-----------------------------------------------|-----------------------------------------------------------------------|
| [每日论文](cookbook/daily_paper/README_ZH.md) | 发现并排序论文,使用 Agent 解读 PDF生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](cookbook/auto-fin/README_ZH.md) | 将财联社事件匹配到高流动性 ETF研究历史反应并生成文件化研究报告。 |
### 可选模型配置
## 📁 记忆系统
如果需要 LLM 驱动的记忆演化或 embedding 检索可以配置环境变量。embedding 默认关闭,因此默认配置不会启动 embedding 模型,也不需要
embedding API key。
> Memory as File, File as Memory.
```bash
cat > .env <<'EOF'
# 可选:仅在配置中显式启用 embedding 组件后使用。
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
ReMe 将 **记忆视为文件**,让原始对话和外部资料从 `session/``resource/` 渐进加工到 `daily/`,再沉淀为 `digest/`
中可长期复用的知识节点。
### 目录结构
```text
<workspace_dir>/
├── metadata/ # 系统索引、图谱、catalog 等持久状态
├── session/ # 原始对话和 Agent session
│ ├── dialog/
│ │ └── <session_id>.jsonl
│ ├── agentscope/
│ └── claude_code/
├── resource/ # 外部原始材料
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # 浅加工记忆:当天事实、对话摘要、资源解读
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <session_event>.md
│ ├── <resource_stem>.md
│ └── interests.yaml
└── digest/ # 长期记忆:个人事实、流程经验、知识节点
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
# 必须auto_memory、auto_resource 和 auto_dream 需要 LLM。
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe 文件化记忆系统总览" width="92%">
</p>
基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取可以先不配置 LLM 凭证。
## 🧭 记忆设计理念
> [!NOTE]
> 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中
> `components.as_embedding``components.embedding_store` 的注释,并将
> `components.file_store.default.embedding_store``""` 改为 `default`。完整说明见
> [记忆检索文档](docs/zh/memory_search.md)。
> 捕获原始对话和资料,将其整理为长期偏好、可复用经验和有价值的知识,并让结果始终能被用户和 Agent 直接编辑。
## 🤝 将 ReMe 接入你的 Agent
### 自动记忆流程
ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP server 接入,也可以通过 Python API 嵌入宿主进程。宿主集成可根据不同
runtime 的能力,将记忆指引、召回和捕获接入 Agent 生命周期。
ReMe 遵循 capture → index → consolidate → recall 的循环。对话和资料先变成 daily 记忆卡片;后台任务保持文件可检索;
`auto_dream` 将稳定知识沉淀到 `digest/`Agent 再通过搜索、wikilink 或 proactive topics 召回记忆。
| 能力 | 入口 | 作用 | 输出 |
|---------------------------------------------|-------------------------------------------|--------------------------------------------------------------------------|-------------------------------------------------------|
| [`auto_memory`](docs/zh/auto_memory.md) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留原始 session。 | `session/dialog/*.jsonl``daily/<date>/<session>.md` |
| [`auto_resource`](docs/zh/auto_resource.md) | 资源监听或 `reme auto_resource` | 将 `resource/<date>/` 下的文件转为带来源链接的 daily 卡片。 | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/zh/memory_search.md) | 后台监听或 `reme reindex` | 维护 chunks、BM25 索引、wikilink 图谱及可选的 embedding 索引。 | 可检索的 `daily/``digest/``resource/` 内容 |
| [`auto_dream`](docs/zh/auto_dream.md) | `dream_cron``reme auto_dream` | 将变化的 daily 卡片整理为长期 personal、procedure 和 wiki 记忆。 | `digest/**``daily/<date>/interests.yaml` |
| [`proactive`](docs/zh/proactive.md) | Agent 决定主动行动前调用 `reme proactive` | 读取 `auto_dream` 生成的 topics是否以及如何提醒用户由宿主 Agent 决定。 | 来自 `daily/<date>/interests.yaml` 的结构化 topics |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
## 🤝 Agent-friendly Integration
ReMe 作为本地记忆服务运行,并提供 CLI、HTTP API、MCP server 和 SDK 等多种接入方式。不同 Agent 可以选择适合自身 runtime
的路径,同时共享同一个本地 memory workspace。
| Agent | 推荐接入方式 | 开箱可用能力 |
|------------------------------------------------------|-------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| **QwenPaw** | 通过 Python SDK 嵌入 ReMe。 | 复用应用自身生命周期和模型配置,同时保持 memory 本地、文件化。 |
| **Claude Code** | 以 MCP service 启动 ReMe并安装 [plugins/reme](plugins/reme)。 | MCP recall tools、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Other CLI-capable agents (OpenClaw/Hermes/Codex)** | 复制或安装 [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索/读取/写入记忆,并调用 `auto_memory``auto_dream``proactive`。 |
| Agent | 推荐接入方式 | 接入后能力 |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **DeepSeek Harness** | 使用 `dsh plugin --profile web add @agentscope-ai/reme` 安装 [`@agentscope-ai/reme`](typescript/README_ZH.md#deepseek-harness)。 | 长期记忆指引、`reme_search` 工具,以及自动捕获已完成的主 Agent 对话。 |
| **OpenClaw** | 使用 `openclaw plugins install @agentscope-ai/reme` 安装 [`@agentscope-ai/reme`](typescript/README_ZH.md#openclaw)。 | 原生记忆工具、用户触发运行前召回和自动对话捕获。 |
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主生命周期和模型配置,同时保持记忆本地、文件化。 |
| **Claude Code** | 启动 streamable HTTP MCP service并安装 [ReMe 插件](integrations/claude_code/reme)。 | MCP 召回工具、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP service并安装 [ReMe provider](integrations/hermes_agent)。 | 模型调用前召回,每轮对话完成后异步执行 `auto_memory`。 |
| **Codex 及其他 CLI Agent** | 安装或复制 [ReMe Memory skill](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动捕获需要显式接入宿主生命周期。 |
<p align="center"><b>集成演示</b></p>
@ -284,36 +214,155 @@ ReMe 作为本地记忆服务运行,并提供 CLI、HTTP API、MCP server 和
</tr>
</table>
## 🛠️ ReMe Operations
## 🧠 ReMe 如何工作
ReMe 通过 CLI 暴露的统一 job interface 操作 workspace。Agent 通常只需要使用检索、读取、写入、编辑和自动记忆相关命令;更底层的索引、
frontmatter 和文件操作接口主要用于维护、调试或高级集成。完整 job 列表可以运行 `reme help` 查看。
> Memory as File, File as Memory.
ReMe 将 **记忆视为文件**,让过滤后的对话来源记录和外部资料从 `session/``resource/` 渐进加工到 `daily/`,再沉淀为
`digest/`。默认 workspace 是当前目录下的 `.reme/`;可通过 `workspace_dir=...` 选择其他由用户控制的位置。
### Workspace 结构
```text
<workspace_dir>/
├── metadata/ # 可重建的索引、图谱、catalog 和缓存
├── session/ # 对话来源记录和 Agent session
│ ├── dialog/
│ │ └── <session_id>.jsonl # auto_memory 保存的来源消息
│ └── claude_code/
│ └── <session_id>.jsonl # auto_memory_cc 使用的 ReMe 副本
├── mem_session/ # Agent wrapper 生成的 session/配置,不是用户记忆
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # 外部原始材料
│ ├── <resource>.<ext> # 根目录文件进入当天 daily 层
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # 浅加工记忆:当天事实、对话摘要、资源解读
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <generated_name>.md # 按主题命名的对话或资源卡片
│ └── interests.yaml
└── digest/ # 长期记忆:个人事实、流程经验、知识节点
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe 文件化记忆系统总览" width="92%">
</p>
### 记忆生命周期
ReMe 遵循 capture → index → consolidate → recall 的循环。workspace 文件是持久化的事实来源,`metadata/` 中的内容均可重建。
| 能力 | 入口 | 作用 | 输出 |
| ------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`auto_memory`](docs/zh/auto_memory.md) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留过滤后的对话来源记录。 | `session/dialog/*.jsonl``daily/<date>/<generated-name>.md` |
| [`auto_resource`](docs/zh/auto_resource.md) | 资源监听或 `reme auto_resource` | 将 `resource/` 下的文件转为带来源链接、按内容命名的 daily 卡片。 | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/zh/memory_search.md) | 后台监听或 `reme reindex` | 实时索引 `daily/``digest/` 中的 Markdown全量重建还会扫描 `resource/` 和 JSONL。 | 可检索的 chunks、BM25、wikilink 图谱和可选向量 |
| [`auto_dream`](docs/zh/auto_dream.md) | `dream_cron``reme auto_dream` | 默认从最近两天内变化的文件中最多提取 5 个可复用 unit再创建、印证、补充或修正 digest 节点。 | `digest/**``daily/<date>/interests.yaml` |
| [`proactive`](docs/zh/proactive.md) | Agent 决定主动行动前调用 `reme proactive` | 读取 `auto_dream` 生成的 topics是否以及如何提醒用户由宿主 Agent 决定。 | 来自 `daily/<date>/interests.yaml` 的结构化 topics |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
搜索返回带行号范围的相关 chunks 和数量受限的 wikilink 邻居;可选向量结果通过 RRF 与 BM25 融合。
> [!IMPORTANT]
>
> `proactive` 只读取并暴露 Auto Dream 生成的兴趣主题,不会自行联网、发送通知或改写知识库;是否以及如何使用主题,由宿主 Agent
> 决定。
## 📊 评测结果
ReMe 通过 Agent 多轮搜索与读取的方式评测多会话和超长上下文中的记忆能力。下表为仓库中已公开的参考实验结果模型、prompt、数据集和评判细节见各评测文档。
| 基准 | 设置 | 样本量 | Agentic 得分 | 主要检验内容 |
| --------------------------------------------------------------------------- | ----------- | ----------------: | -----------: | ------------------------------ |
| **[LongMemEval cleaned-s](https://reme.agentscope.io/?doc=longmemeval-zh)** | **整体** | **500 题** | **89.4%** | 跨会话检索、知识更新与时间推理 |
| [BEAM](https://reme.agentscope.io/?doc=beam-zh) | 100K 上下文 | 20 cases / 400 题 | 66.1% | 十类长上下文记忆任务 |
| [BEAM](https://reme.agentscope.io/?doc=beam-zh) | 1M 上下文 | 35 cases / 700 题 | 65.0% | 超长对话设置 |
在仓库的 [π-Bench 评测](https://reme.agentscope.io/?doc=pibench-zh)中ReMe Agent 在 5 种用户角色上的平均 **PROC 得分为 0.580**
,比相同测试模型配置的 NanoBot 高 2.4%。PROC 用于评估隐藏意图完成、针对性澄清、跨会话偏好和规范复用、跨任务依赖推断以及欠规格请求推进等主动性能力。
## 🧩 扩展与插件
插件是可选的独立 Python distribution可以贡献 Component、Step、Job backend 和配置,并通过配置显式启用。每日论文与 Auto Fin
均已独立打包,源码 distribution 及说明分别见[每日论文](plugins/daily_paper/README_ZH.md)和
[Auto Fin](plugins/auto-fin/README_ZH.md)。
| 插件 | 能力 |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [每日论文](https://reme.agentscope.io/?doc=daily-paper-zh) | 发现并排序论文,使用 Agent 解读 PDF生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-zh) | 拉取主题相关财联社新闻,搜索 ReMe 历史材料并生成带 wikilink 的 Markdown 报告。 |
安装、查看、校验、启用和卸载 ReMe 插件的方法见[插件管理](docs/zh/plugin_management.md)。
## 📚 文档
下列文档覆盖主要使用流程,并以当前代码的运行时契约为准。
| 文档 | 主要内容 |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| [快速开始](docs/zh/quick_start.md) | 安装 ReMe、启动服务并执行首次文件和记忆操作。 |
| [Memory as File](docs/zh/memory_as_file.md) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 |
| [Auto Memory](docs/zh/auto_memory.md) | 保留过滤后的对话来源记录,并提炼可复用的 daily 记忆卡片。 |
| [Auto Resource](docs/zh/auto_resource.md) | 导入支持的文本资料,转换为可追溯来源的 daily 卡片。 |
| [Auto Dream](docs/zh/auto_dream.md) 与 [Auto Link](docs/zh/auto_link.md) | 将 daily 记忆整理为持续演化的 digest 节点和可读 wikilink 关系。 |
| [记忆检索](docs/zh/memory_search.md) | 使用 BM25、可选向量、RRF 融合、行号范围召回和渐进式链接扩展。 |
| [Proactive](docs/zh/proactive.md) | 安全读取兴趣主题,并将其接入宿主 Agent 的决策流程。 |
| [应用场景](docs/zh/reme_scene.md) | 查看金融研究、研发记忆和个人知识库的完整使用示例。 |
| [框架说明](docs/zh/framework.md) | 理解 Application、Job、Step、Component、service、配置和生命周期边界。 |
| [TypeScript 集成](typescript/README_ZH.md) | 配置统一 client以及 DeepSeek Harness 和 OpenClaw 原生适配器。 |
| [ReMe 博客](https://agentscope-ai.github.io/ReMe/?doc=zh-reme-blog) | 了解完整产品故事、设计动机、使用示例和评测摘要。 |
## 🛠️ 常用命令
运行 `reme help` 可查看完整 job 列表。常用 workspace 与维护命令如下:
| 命令 | 作用 |
|-------------------------------------------|---------------------------------------------------------------|
| `reme start` | 启动本地 ReMe 服务。 |
| `reme version` / `reme health_check` | 检查包版本和组件状态。 |
| ----------------------------------------- | ------------------------------------------------------------- |
| `reme status` | 查看有状态数据组件的内存估算及进程 RSS。 |
| [`reme search`](docs/zh/memory_search.md) | 默认使用 BM25 和 wikilink 检索,启用后增加向量检索。 |
| `reme read` / `reme write` / `reme edit` | 检查和维护 Markdown 记忆文件。 |
| `reme auto_memory` | 将对话 messages 转为 daily 记忆卡片;需要 LLM 凭证。 |
| `reme auto_resource` | 将 `resource/` 下的文件解读为 daily 资料卡片;需要 LLM 凭证。 |
| `reme auto_dream` / `reme proactive` | 将 daily 记忆整理为长期 digest并暴露值得关注的主题。 |
| `reme traverse` / `reme graph_snapshot` | 浏览 wikilink 邻域或按类别组织的 digest 图。 |
| `reme chat` | 与可感知 workspace 的只读 Agent 进行流式对话;需要 LLM 凭证。 |
| `reme reindex` | 基于已有文件重建检索和 wikilink 索引。 |
## 🤝 社区与支持
## 🤝 社区与贡献
- **问题反馈与需求**:请先查看 [Open Issues](https://github.com/agentscope-ai/ReMe/issues);如无相关讨论,可新建 Issue
- **问题反馈、需求与帮助**:请先查看 [Open Issues](https://github.com/agentscope-ai/ReMe/issues);如无相关讨论,可新建 Issue
说明背景、目标行为和影响范围。
- **代码贡献**:改动前建议阅读 [贡献指南](https://docs.agentscope.io/reme/latest/zh/contribution)。架构与扩展方式以源码、schema
和测试为准。
- **文档贡献**:用户可见文档请提交到[统一文档仓库](https://github.com/agentscope-ai/docs)的 `reme/<version>/{en,zh}/` 目录。
- **代码贡献**:改动前建议阅读仓库内的[贡献指南](docs/zh/contributing.md)。架构与扩展方式以源码、schema 和测试为准。
- **文档贡献**:请直接更新本仓库 `docs/en/``docs/zh/` 或对应 package 目录中的规范源文件;文档站点会从这些文件生成。
- **提交规范**:建议使用 Conventional Commits例如 `feat(search): add link expansion option`
`docs(zh): update quick start`
- **提交前检查**:提交 PR 前请尽量运行 `pre-commit run --all-files``pytest`;如有依赖 LLM、embedding 或外部服务的测试无法运行,请在
PR 中说明。
- **获取帮助**:如需反馈 Bug 或功能请求,请使用 [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues);项目文档见
[https://docs.agentscope.io/reme](https://docs.agentscope.io/reme)。
- **项目文档**:访问 [reme.agentscope.io](https://reme.agentscope.io)。
### 贡献者

View file

@ -1,155 +0,0 @@
[中文版 / Chinese version](./README_ZH.md)
# ReMe Benchmarks
Reproduction guide for the two memory benchmarks shipped with ReMe:
- **LongMemEval** — long-term memory over multi-session chat histories.
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
per-item workspace, answer probing questions via an agentic (ReAct) mode,
then score answers with an LLM-as-judge.
## 1. Prerequisites
Install ReMe with dev + core extras (Python 3.11+):
```bash
pip install -e ".[dev,core]"
```
Configure model credentials in a project-root `.env` file (copied from `example.env`).
The runners auto-load `.env` from the repository root. Required variables typically include:
```bash
LLM_API_KEY=...
LLM_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_BASE_URL=...
```
Model names and component wiring live in the ReMe configs referenced by each benchmark
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
## 2. Download Datasets
Each benchmark keeps its own data under its directory:
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
All three are excluded from Git.
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
modify `BASE_URL` in `download.py`):
```bash
cd benchmark/longmemeval
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
```
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
`topics/` and other subdirectories.
## 3. Run LongMemEval
From the repository root:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
```
### Pipeline
1. Load the dataset (ground truth is embedded in the data file).
2. For each item, create an isolated workspace and ingest sessions in chronological order.
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
4. Answer each question via agentic (ReAct) mode.
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
### Key config — `benchmark/longmemeval/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
| `dataset.question_types` | Filter by question type; empty = all. |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
| `reme.config` | ReMe config used (`lme.yaml`). |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
## 4. Run BEAM
From the repository root:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # quiet
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
```
### Pipeline
1. For each case, load `chat.json` and convert each batch into a ReMe session.
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
3. Answer each probing question via agentic (ReAct) mode.
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
### Key config — `benchmark/beam/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
| `reme.config` | ReMe config used (`beam.yaml`). |
| `output.dir` | Results directory (`benchmark/beam/results`). |
## 5. Outputs & Logs
- **Results**: JSON files written to `output.dir`
(`results_<timestamp>.json` for LongMemEval,
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
accuracy/score is also printed to the console.
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
per worker process).
## 6. Stopping a Run
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
```bash
bash benchmark/kill.sh <PID>
```
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
`SIGKILL` for any process that does not exit within 5 seconds.
## 7. Reference Results
Recorded evaluation results are available in:
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

View file

@ -1,149 +0,0 @@
# ReMe 评测复现说明
ReMe 内置两个记忆能力评测基准的复现指南:
- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。
- **BEAM** —— 面向长上下文对话场景、基于评分细则rubric打分的记忆能力评测。
每个基准都会运行完整的端到端流程:将会话摄入独立的按条目隔离的工作区,
以 agenticReAct模式回答探测问题
最后由 LLM-as-judge 对答案进行打分。
## 1. 环境准备
安装 ReMe 及 dev + core 附加依赖Python 3.11+
```bash
pip install -e ".[dev,core]"
```
在项目根目录配置 `.env`(可从 `example.env` 复制),运行脚本会自动从仓库根目录加载 `.env`
通常需要以下变量:
```bash
LLM_API_KEY=...
LLM_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_BASE_URL=...
```
模型名称与组件装配位于各基准引用的 ReMe 配置中
`reme/config/lme.yaml``reme/config/beam.yaml`)。
## 2. 下载数据集
每个基准的数据都存放在各自目录下:`benchmark/<name>/dataset`(输入数据)、
`benchmark/<name>/workspaces`(按条目隔离的记忆工作区)、
`benchmark/<name>/results`(评测输出)。三者均不纳入 Git 版本管理。
**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本,数据托管在 HuggingFace
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(下载脚本经 hf-mirror.com 镜像源获取,如需更换源请修改 `download.py` 中的 `BASE_URL`
```bash
cd benchmark/longmemeval
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json已存在则自动跳过
```
**BEAM**公开仓库clone 到 `benchmark/beam/dataset/` 下):
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后,`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/``src/``topics/` 等子目录。
## 3. 运行 LongMemEval
在仓库根目录执行:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # 安静模式:仅评测级日志
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
### 流程
1. 加载数据集ground truth 已内嵌在数据文件中)。
2. 为每个条目创建独立工作区,按时间顺序摄入会话。
3. 当相邻会话跨越配置的时刻(默认 23:00时触发 `auto_dream`
4. 以 agenticReAct模式回答每个问题。
5. 通过 `answer_judge` 任务对答案做二元yes/no评判并输出各类型准确率。
### 关键配置 —— `benchmark/longmemeval/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.path` | 待评测的数据集文件(如 `longmemeval_s_reme_cleaned.json`),已包含 ground truth。 |
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
| `dataset.question_types` | 按问题类型过滤,空表示全部。 |
| `dataset.workspace_root` | 条目工作区根目录(`benchmark/longmemeval/workspaces/longmemeval-s`)。 |
| `evaluation.num_workers` | `0` = 自动cpu-2`1` = 串行,`>1` = 并行。 |
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
| `reme.config` | 使用的 ReMe 配置(`lme.yaml`)。 |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
| `output.dir` | 结果目录(`benchmark/longmemeval/results`)。 |
## 4. 运行 BEAM
在仓库根目录执行:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # 安静模式
python benchmark/beam/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
### 流程
1. 为每个 case 加载 `chat.json`,将每个 batch 转换为一个 ReMe 会话。
2. 按时间顺序将会话摄入独立工作区,随后执行 `digest_update`
3. 以 agenticReAct模式回答每个探测问题。
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分,并输出各类型平均分。
### 关键配置 —— `benchmark/beam/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.beam_root` | BEAM 数据集根目录(`benchmark/beam/dataset/BEAM`)。 |
| `dataset.chat_size` | 运行的变体:`100K` / `500K` / `1M` / `10M`。 |
| `dataset.case_ids` | 指定 case`["1","2"]`),空表示全部。 |
| `dataset.start_index` / `num_items` | case 分页(`num_items``0` 表示全部)。 |
| `dataset.workspace_root` | case 工作区根目录(`benchmark/beam/workspaces/beam`)。 |
| `evaluation.num_workers` | `0` = 自动,`1` = 串行,`>1` = 并行。 |
| `reme.config` | 使用的 ReMe 配置(`beam.yaml`)。 |
| `output.dir` | 结果目录(`benchmark/beam/results`)。 |
## 5. 输出与日志
- **结果**JSON 文件写入 `output.dir`
LongMemEval 为 `results_<timestamp>.json`
BEAM 为 `results_<chat_size>_<timestamp>.json`)。同时控制台会打印含各类型
准确率/分数的汇总。
- **日志**:当 `output.log_to_file` 开启时,每次运行的日志写入
`logs/<log_prefix>_<timestamp>/`(包含一个 `runner.log` 及每个 worker 进程的
`worker-<pid>.log`)。
## 6. 终止运行
并行运行会派生进程树。若要干净地终止某次运行及其全部 worker
```bash
bash benchmark/kill.sh <PID>
```
该脚本会先向整个进程树发送 `SIGTERM` 优雅终止,对 5 秒内未退出的进程再升级为 `SIGKILL`
## 7. 参考结果
已记录的评测结果见:
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

124
benchmark/beam/README.md Normal file
View file

@ -0,0 +1,124 @@
[中文版 / Chinese version](./README_ZH.md)
# BEAM Benchmark
BEAM is a benchmark for **memory capability over long-context chat cases**. Each
case contains a very long chat history split into batches; ReMe converts each
batch into a session, ingests them in chronological order, then answers probing
questions via an agentic (ReAct) mode. Answers are scored with BEAM's
rubric-based `answer_judge` job, which produces both a graded score and a binary
verdict, and per-type averages are reported.
BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so
memory systems can be stressed at different context lengths. Question types
include abstention, contradiction resolution, event ordering, information
extraction, instruction following, knowledge update, multi-session reasoning,
preference following, summarization, and temporal reasoning.
> For the shared setup (dependencies, credentials, log conventions) see the
> [top-level benchmark README](../README.md).
## 1. Get the Dataset
BEAM is a public repository, cloned into `benchmark/beam/dataset/`:
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
`topics/` and other subdirectories.
## 2. Run
From the repository root:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # quiet
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
```
## 3. Pipeline
1. For each case, load `chat.json` and convert each batch into a ReMe session.
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
3. Answer each probing question via agentic (ReAct) mode.
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
## 4. Key config — `benchmark/beam/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
| `reme.config` | ReMe config used (`beam.yaml`). |
| `output.dir` | Results directory (`benchmark/beam/results`). |
## 5. Outputs
Results are JSON files written to `output.dir` as
`results_<chat_size>_<timestamp>.json`, with a per-type score summary also
printed to the console. Logging conventions are shared across benchmarks — see
the [top-level README](../README.md#outputs--logs).
## 6. Reference Results
> The results below use the longmemeval-version prompt.
### 100K
agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory)
(2026-08-05, 20 cases / 400 Qs, total 46.0 min)
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction average token consumption (default agent, full build over 20 cases):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build
(2026-08-05, 35 cases / 700 Qs, total 459.2 min)
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction average token consumption (default agent, full build over 35 cases):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

119
benchmark/beam/README_ZH.md Normal file
View file

@ -0,0 +1,119 @@
# BEAM 评测
[English version](./README.md)
BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个
batch 的超长对话ReMe 将每个 batch 转换为一个会话,按时间顺序摄入后,以 agenticReAct
模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分,同时给出分级分数与二元
判定,并输出各类型平均分。
BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`,可在不同上下文长度下
压测记忆系统。题型包括 abstention拒答、contradiction resolution矛盾消解、event
ordering事件排序、information extraction信息抽取、instruction following指令遵循
knowledge update知识更新、multi-session reasoning多会话推理、preference following
偏好遵循、summarization摘要与 temporal reasoning时间推理
> 公共设置(依赖、凭据、日志约定)见[总评测说明](../README_ZH.md)。
## 1. 获取数据集
BEAM 是公开仓库clone 到 `benchmark/beam/dataset/` 下:
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后,`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/``src/``topics/` 等子目录。
## 2. 运行
在仓库根目录执行:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # 安静模式
python benchmark/beam/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
## 3. 流程
1. 为每个 case 加载 `chat.json`,将每个 batch 转换为一个 ReMe 会话。
2. 按时间顺序将会话摄入独立工作区,随后执行 `digest_update`
3. 以 agenticReAct模式回答每个探测问题。
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分,并输出各类型平均分。
## 4. 关键配置 —— `benchmark/beam/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.beam_root` | BEAM 数据集根目录(`benchmark/beam/dataset/BEAM`)。 |
| `dataset.chat_size` | 运行的变体:`100K` / `500K` / `1M` / `10M`。 |
| `dataset.case_ids` | 指定 case`["1","2"]`),空表示全部。 |
| `dataset.start_index` / `num_items` | case 分页(`num_items``0` 表示全部)。 |
| `dataset.workspace_root` | case 工作区根目录(`benchmark/beam/workspaces/beam`)。 |
| `evaluation.num_workers` | `0` = 自动,`1` = 串行,`>1` = 并行。 |
| `reme.config` | 使用的 ReMe 配置(`beam.yaml`)。 |
| `output.dir` | 结果目录(`benchmark/beam/results`)。 |
## 5. 输出
结果以 JSON 文件写入 `output.dir`,文件名为 `results_<chat_size>_<timestamp>.json`
同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用,见
[总说明](../README_ZH.md#输出与日志)。
## 6. 参考结果
> 以下结果使用 longmemeval 版本的 prompt。
### 100K
agentscope==2.0.4.post1conda reme 环境20 并发eval-only复用已构建 memory
2026-08-0520 cases / 400 Qs总耗时 46.0 min
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction 平均 token 消耗default agent20 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1conda reme 环境20 并发,全量构建 memory
2026-08-0535 cases / 700 Qs总耗时 459.2 min
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction 平均 token 消耗default agent35 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

View file

@ -0,0 +1,96 @@
[中文版 / Chinese version](./README_ZH.md)
# LongMemEval Benchmark
LongMemEval is a benchmark for **long-term memory over multi-session chat
histories**. Each item provides a chronologically ordered set of chat sessions
between a user and an assistant, followed by a probing question whose answer is
only recoverable by reasoning over the user-owned memory. ReMe ingests the
sessions into an isolated per-item workspace, answers the question via an
agentic (ReAct) mode, and scores the answer with an LLM-as-judge.
Question types include single-session (user / assistant / preference),
multi-session reasoning, knowledge update, and temporal reasoning.
> For the shared setup (dependencies, credentials, log conventions) see the
> [top-level benchmark README](../README.md).
## 1. Get the Dataset
ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2).
The download script fetches it via the hf-mirror.com mirror; to use a different
mirror, modify `BASE_URL` in [`download.py`](./download.py).
```bash
cd benchmark/longmemeval
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
```
Ground truth is embedded in the data file.
## 2. Run
From the repository root:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
```
## 3. Pipeline
1. Load the dataset (ground truth is embedded in the data file).
2. For each item, create an isolated workspace and ingest sessions in chronological order.
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
4. Answer each question via agentic (ReAct) mode.
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
## 4. Key config — `benchmark/longmemeval/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
| `dataset.question_types` | Filter by question type; empty = all. |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
| `reme.config` | ReMe config used (`lme.yaml`). |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
## 5. Outputs
Results are JSON files written to `output.dir` as `results_<timestamp>.json`,
with a per-type accuracy summary also printed to the console. Logging
conventions are shared across benchmarks — see the
[top-level README](../README.md#outputs--logs).
## 6. Reference Results
### cleaned-s
**Basic settings**
1. Modified auto-memory prompt, auto-dream disabled.
2. All sessions in reme-memory are strictly earlier than the question time.
**Results**
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory)
(2026-08-06, 500 items, total 10.0 min)
| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |

View file

@ -0,0 +1,90 @@
# LongMemEval 评测
[English version](./README.md)
LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间
顺序排列的用户与助手之间的会话以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe
将会话摄入按条目隔离的工作区,以 agenticReAct模式回答问题最后由 LLM-as-judge 打分。
题型包括单会话user / assistant / preference、多会话推理、知识更新与时间推理等。
> 公共设置(依赖、凭据、日志约定)见[总评测说明](../README_ZH.md)。
## 1. 获取数据集
ReMe 仅使用 **cleaned-S** 版本,数据托管在 HuggingFace
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。
下载脚本经 hf-mirror.com 镜像源获取,如需更换源请修改 [`download.py`](./download.py) 中的
`BASE_URL`
```bash
cd benchmark/longmemeval
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json已存在则自动跳过
```
ground truth 已内嵌在数据文件中。
## 2. 运行
在仓库根目录执行:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # 安静模式:仅评测级日志
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
## 3. 流程
1. 加载数据集ground truth 已内嵌在数据文件中)。
2. 为每个条目创建独立工作区,按时间顺序摄入会话。
3. 当相邻会话跨越配置的时刻(默认 23:00时触发 `auto_dream`
4. 以 agenticReAct模式回答每个问题。
5. 通过 `answer_judge` 任务对答案做二元yes/no评判并输出各类型准确率。
## 4. 关键配置 —— `benchmark/longmemeval/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.path` | 待评测的数据集文件(如 `longmemeval_s_reme_cleaned.json`),已包含 ground truth。 |
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
| `dataset.question_types` | 按问题类型过滤,空表示全部。 |
| `dataset.workspace_root` | 条目工作区根目录(`benchmark/longmemeval/workspaces/longmemeval-s`)。 |
| `evaluation.num_workers` | `0` = 自动cpu-2`1` = 串行,`>1` = 并行。 |
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
| `reme.config` | 使用的 ReMe 配置(`lme.yaml`)。 |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
| `output.dir` | 结果目录(`benchmark/longmemeval/results`)。 |
## 5. 输出
结果以 JSON 文件写入 `output.dir`,文件名为 `results_<timestamp>.json`
同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用,见
[总说明](../README_ZH.md#输出与日志)。
## 6. 参考结果
### cleaned-s
**基础设置**
1. 使用修改后的 auto-memory prompt关闭 auto-dream 机制
2. reme-memory 中的全部 session 的时间一定早于 question 的时间
**结果**
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only复用预构建记忆
2026-08-06500 题,总计 10.0 min
| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |

View file

@ -0,0 +1,76 @@
#!/bin/bash
# 杀死指定进程及其所有子进程
# Usage: bash kill.sh <PID>
if [ -z "$1" ]; then
echo "Usage: bash kill.sh <PID>"
echo " 杀死指定进程及其所有子进程"
exit 1
fi
PID=$1
# 检查进程是否存在
if ! kill -0 "$PID" 2>/dev/null; then
echo "进程 $PID 不存在"
exit 1
fi
# 递归收集所有子进程(包括子进程的子进程)
collect_children() {
local parent=$1
local children
children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
for child in $children; do
collect_children "$child"
done
echo "$parent"
}
# 收集进程树(子进程在前,父进程在后,保证先杀子再杀父)
PROCESS_TREE=$(collect_children "$PID")
TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ')
echo "进程树(共 $TOTAL 个进程):"
while read -r p; do
cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80)
printf " PID=%-8s %s\n" "$p" "$cmd"
done <<< "$PROCESS_TREE"
# 先 SIGTERM 优雅终止
echo ""
echo "发送 SIGTERM..."
while read -r p; do
kill "$p" 2>/dev/null
done <<< "$PROCESS_TREE"
# 等待最多 5 秒
for i in $(seq 1 5); do
alive=false
while read -r p; do
if kill -0 "$p" 2>/dev/null; then
alive=true
fi
done <<< "$PROCESS_TREE"
if [ "$alive" = false ]; then
break
fi
sleep 1
done
# 检查是否还有残留,强制 SIGKILL
remaining=false
while read -r p; do
if kill -0 "$p" 2>/dev/null; then
remaining=true
fi
done <<< "$PROCESS_TREE"
if [ "$remaining" = true ]; then
echo "部分进程未响应,发送 SIGKILL..."
while read -r p; do
kill -9 "$p" 2>/dev/null
done <<< "$PROCESS_TREE"
fi
echo "已终止进程树(根 PID=$PID,共 $TOTAL 个进程)"

14
benchmark/pibench/.gitignore vendored Normal file
View file

@ -0,0 +1,14 @@
# 含真实 API key绝不入库
env.sh
# 运行时产物(含对话内容,勿入库)
logs/
outputs/
reme_workspace/
nanobot_workspace/
# 数据符号链接(指向外部 π-Bench 仓库)
data
__pycache__/
*.pyc

327
benchmark/pibench/README.md Normal file
View file

@ -0,0 +1,327 @@
[中文版 / Chinese version](./README_ZH.md)
# π-Bench Evaluation Suite
A glue layer that connects the **ReMe agent (with persistent memory)** to
**π-Bench** (Proactive Personal Assistant Benchmark). This directory contains
only the minimal code and configuration needed for the integration: the
π-Bench framework (`src/`), evaluation data (`data/`), the AppWorld tool
environment, and ReMe itself are all **external third-party dependencies**,
referenced in place via symlink and environment variables and never bundled
with this suite.
- π-Bench: https://github.com/Simplified-Reasoning/Pi-Bench (arXiv: 2605.14678)
- ReMe: the root of the ReMe repository this suite lives in (recommended
location: `ReMe/benchmark/pibench/`)
## 1. Architecture
```
π-Bench runner (src.main --mode run)
│ user_agent (simulated-user LLM) walks data/{persona}/episode.yaml
│ task by task, chatting with the agent over multiple turns and judging
│ hidden intents (PROC) during the run phase
test server (π-Bench scripts/test_server.py, HTTP long-polling)
▲ /send │ /poll
│ ▼
bridge_reme.py ──────────────► ReMe Application (embedded as a library)
│ ├─ agent_wrapper: agent under test (AgentScope)
│ ├─ jobs: search / auto_memory / daily_write
│ └─ workspace: reme_workspace/{persona}/
│ (isolated persistent memory per persona)
└──── MCP ────► AppWorld MCP ────► AppWorld APIs (tool/app environment)
π-Bench runner (src.main --mode eval)
judger (judge LLM) reads the traces and scores each checklist item (COMP)
```
Key points:
- The bridge runs on **ReMe's own venv python** and uses ReMe as a library
(`resolve_app_config` + `Application`); **no ReMe source modification** is
required.
- Every incoming user message automatically triggers a ReMe memory `search`
and injects the matched memories (tuning knobs in §8); on task end (reset)
the session is distilled into daily notes by `auto_memory`.
- Tool calls executed by the agent (AppWorld MCP + ReMe job tools) are
captured per turn into the trace as `tool_steps`, so π-Bench
`tools_evaluation_path` scripts can score tool behavior (§7).
- π-Bench's `data/`, `src/` and AppWorld are not part of this suite; install
π-Bench first (§3.1).
## 2. Directory layout
```
pibench/
├── README.md / README_ZH.md # this document (English / Chinese)
├── env.sh.example # environment template (copy to env.sh, fill TODOs)
├── bridge_reme.py # ReMe ↔ test server bridge (memory inject/save,
│ # profile injection, tool-trace capture)
├── run_persona.sh # full pipeline for ONE persona (5 services + run + eval)
├── run_all.sh # batch over 5 personas (fresh/resume, default parallel=2)
├── resume.py # checkpoint resume: completion detection + surgical
│ # cleanup of interrupted tasks' residual memory
├── fix_trace_logs.py # run outputs → ~/.nanobot/trace_logs conversion,
│ # merging tool sidecars into turn files (pre-eval)
├── .gitignore # excludes env.sh and all runtime artifacts
└── config/
├── models/reme.yaml # runner model config (model_id=reme)
└── bench/evaluation/trace_history.yaml # trace render policy (shipped with
# the suite; passed via --history-config-path)
```
Generated at runtime (all git-ignored): `data` (symlink), `logs/`, `outputs/`,
`reme_workspace/`, `nanobot_workspace/`.
## 3. Prerequisites (third-party, install first)
### 3.1 π-Bench repository (with AppWorld)
```bash
git clone https://github.com/Simplified-Reasoning/Pi-Bench.git <pi-bench-dir>
cd <pi-bench-dir>
python3.11 -m venv .venv # scripts expect exactly this venv name
source .venv/bin/activate
pip install -e . # pibench runner (src.main)
bash scripts/setup_appworld.sh # install AppWorld and download its data (large)
```
Post-install sanity checks:
```bash
ls data/ # should contain researcher marketer pharmacist law_trainee Financier
.venv/bin/python -c "import src" && echo OK
.venv/bin/appworld --help >/dev/null && echo OK
```
### 3.2 ReMe repository
```bash
cd <reme-dir> # ReMe repository root (contains the reme/ package)
python3.11 -m venv .venv # scripts expect exactly this venv name
source .venv/bin/activate
pip install -e . # or ReMe's own install flow; `import reme` must work
```
Sanity check: `.venv/bin/python -c "import reme; print('ok')"`
## 4. Install this suite (step by step)
1. **Place the suite** (recommended inside the ReMe repo so `REME_DIR` is
inferred automatically):
```bash
cp -r pibench <reme-dir>/benchmark/pibench
cd <reme-dir>/benchmark/pibench
```
If placed elsewhere, set `REME_DIR` explicitly in env.sh later.
2. **Create the environment file and fill in the custom parameters**:
```bash
cp env.sh.example env.sh
```
Open `env.sh`; required items (marked TODO):
| Variable | Description |
|---|---|
| `PI_BENCH_ROOT` | π-Bench repo root (contains `src/` `data/` `.venv` `third_party/appworld`) |
| `USER_API_KEY` | API key of the simulated-user LLM (run phase, hidden-intent judging) |
| `JUDGER_API_KEY` | API key of the judger LLM (eval phase, checklist scoring) |
| `BRAVE_SEARCH_API_KEY` | optional; for the agent's web_search tool, `dummy` when unused |
Optional tuning: `REME_MODEL_NAME` (base model of the agent under test),
`REME_DIR`, `REME_LLM_BASE_URL` (default: DashScope OpenAI-compatible
endpoint).
3. **Link the evaluation data** (referenced in place, never copied):
```bash
ln -s "$PI_BENCH_ROOT/data" data
```
4. **(Optional) adjust model config** `config/models/reme.yaml`:
- `user_agent.model` / `judger.model`: model names for the simulated user
and the judger (literal values; π-Bench only expands `${ENV}` in
base_url/api_key).
- `run.turn_timeout`, `max_tool_iterations`, etc. as needed.
5. **Smoke check** (does not start the evaluation):
```bash
bash -n run_all.sh && bash -n run_persona.sh
source env.sh && "$REME_DIR/.venv/bin/python" -c "import reme; print('reme ok')"
```
## 5. Run the evaluation
> ⚠️ For long runs use `screen`, **not nohup** (nohup loses the permission
> context in sandboxed/restricted environments and breaks child processes).
```bash
# Full official run: wipe ALL personas' memory/outputs/traces first (default
# fresh mode, parallel=2)
mkdir -p logs # on a fresh deployment logs/ does not exist yet
screen -dmS pibench_suite bash -c "cd $(pwd) && bash run_all.sh > logs/run_all_master.log 2>&1"
# Checkpoint continuation (after an interruption; no wipe, completed tasks skipped)
bash run_all.sh --resume
# Other usages
bash run_all.sh --parallel 1 # sequential
bash run_all.sh --resume --skip-eval # run phase only
bash run_persona.sh researcher # single persona (default --resume semantics)
bash run_persona.sh researcher --fresh
```
Time reference: 5 personas × 20 tasks, parallel=2, fresh full run ≈ 1214 hours.
`run_all.sh` exits non-zero when any persona fails, so upstream automation
cannot mistake a partially failed suite run for a success.
## 6. Port allocation (parallel personas never collide)
| persona | AppWorld API | AppWorld MCP | Test Server | ReMe internal service |
|-------------|------|-------|------|-------|
| marketer | 9001 | 10001 | 9998 | 18766 |
| law_trainee | 9002 | 10002 | 9997 | 18767 |
| pharmacist | 9003 | 10003 | 9996 | 18768 |
| researcher | 9004 | 10004 | 9995 | 18765 |
| Financier | 9005 | 10005 | 9994 | 18769 |
## 7. Outputs and scores
- **Results**: `outputs/reme/{persona}/{task}/eval/results/*_result.json`
- `overall_average_score`: checklist completeness (COMP; the judger scores
each criterion YES/NO, weighted across dependency groups)
- `overall_proactiveness_average_score`: proactiveness (PROC; the
user_agent judges hidden-intent coverage during the run phase; each task
file also carries the global average)
- **Traces**: `~/.nanobot/trace_logs/reme/{persona}/{task}/...` (the scoring
input of the eval phase)
- **Logs**: `logs/` (`suite_<persona>.log` per persona; `bridge_*`,
`runner_run/eval_*`, `appworld_*`, `test_server_*` per service)
- **Memory store**: `reme_workspace/{persona}/` (daily/digest notes, raw
session dialogs, BM25 index, etc.; persistent across runs, wiped only in
fresh mode)
Score summary:
```bash
grep -h "overall_average_score\|overall_proactiveness" \
outputs/reme/*/*/eval/results/*_result.json | head
```
### Tool-trace capture (tools_evaluation support)
Some tasks define `objectives.tools_evaluation_path`: Python scripts that
score tool behavior (e.g. "the temporary Todoist board was created and
removed"). They need the executed tool calls in the trace. The pipeline:
1. During `reply()`, the bridge reads the persisted AgentScope session state
after each turn and extracts the new `tool_call` / `tool_result` blocks
(tool name, arguments, result).
2. Records are appended to
`outputs/reme/{persona}/{task}/history/{ts}-tools.jsonl`, tagged with the
turn number; AgentScope MCP names (`mcp__AppWorld__<tool>`) are normalized
to the π-Bench convention (`mcp_appworld_<tool>`).
3. `fix_trace_logs.py` pairs each `{ts}-messages.jsonl` run with the
temporally closest tools sidecar and merges the records into the generated
`turn_N.json` files under the `tool_steps` key — one of the two
tool-history formats understood by π-Bench's `collect_tool_history()`.
4. The eval phase then feeds `tool_steps` to both the tools_evaluation
scripts and the rendered `<tool_trace_extracts>` seen by the judger.
## 8. Memory mechanism (core design of this suite)
- **Persona isolation**: each persona has its own workspace
(`reme_workspace/{persona}/`); the bridge takes an exclusive
`.bridge.lock` on it at startup, so two bridges can never share one memory
store, and one persona's memory search can never reach another's memories.
- **Writes**: on task end (runner sends reset), the session is distilled by
the `auto_memory` job into daily notes and indexed by the background
watcher (BM25). Saves are non-blocking background tasks; the first message
of a new session waits for in-flight writes before searching.
- **Reads**: on every incoming user message the bridge runs one `search` and
injects matched memories (`[Relevant memories from previous sessions]`
prefix); without matches the message passes through unchanged. Retrieval
tuning (bridge CLI flags, adjustable in run_persona.sh):
- `--search-limit 3`: at most 3 memory chunks injected per message;
- `--search-min-score 2.0`: weak BM25 hits are filtered out;
- `tool_context_id` rotates per task: chunks already injected within the
same task are not re-injected (ReMe's seen-chunk dedup, 24h TTL); normal
recall resumes after task boundaries.
- **No self-leakage**: the in-progress session is not in the store yet
(saves happen on reset), so a task can never retrieve its own unfinished
content.
- The agent also holds `search`/`daily_write` tools and can retrieve/record
proactively.
- **System prompt**: `bridge_reme.py:build_system_prompt()` embeds the
HIDDEN-NEEDS protocol (proactiveness-oriented) and injects the persona
profile from `data/{persona}/profile.yaml` into every turn's system prompt.
## 9. Checkpoint resume and memory-cleanup semantics
- **Completion detection** (resume.py): scans
`outputs/reme/{persona}/**/history/*-log.jsonl` and
`outputs/reme/{persona}/run/*-log.jsonl` for
`Task finished task_id=X status=Y`. The status with the **newest event
timestamp** wins per task (record `timestamp`, falling back to
`timestamp_iso`, then to the timestamp embedded in the log file name) —
file category and read order alone can never override a newer record, so an
old run-level SUCCESS cannot mask a newer per-task ERROR. `SUCCESS /
MAX_TURNS / TIMEOUT` count as completed; `ERROR` and never-started tasks
are re-run (passed to the runner as repeated `--task-id` flags in episode
order).
- **Answer-leak prevention**: an interrupted task may already have been
distilled into daily notes during graceful shutdown; re-running it with
that memory injected would inflate scores. Before resuming,
`resume.py cleanup` therefore removes residual memory **only for tasks
about to be re-run** (daily/digest notes, session/dialog, mem_session;
matched via `session_id = pibench_{task}_*`). Completed tasks' memories are
never touched. Daily index files are refreshed **only for the dates that
lost notes**, by full workspace-relative wikilink path — and when the ReMe
package is importable, the refresh reuses ReMe's own daily-index rebuild
logic (`refresh_day_index`), so same-named notes on other dates are never
modified.
- **fresh vs resume are mutually exclusive**: a full memory wipe belongs to
fresh mode only (`run_all.sh` default, executed before any service starts);
resume never wipes.
## 10. Customization entry points
| Goal | Location |
|---|---|
| Base model of the agent under test | `REME_MODEL_NAME` in `env.sh` |
| user_agent / judger models | `config/models/reme.yaml` |
| Agent system prompt | `bridge_reme.py` `build_system_prompt()` |
| Memory retrieval limit/threshold | `--search-limit/--search-min-score` on the bridge command in `run_persona.sh` |
| ReMe internal parameters | **Do not modify ReMe source**; write a dedicated config modeled on `reme/config/beam.yaml` and override via `resolve_app_config(config=...)` (see bridge `_init_reme_app`) |
| Turn timeout / tool iteration cap | `config/models/reme.yaml` `run.turn_timeout`, `model.max_tool_iterations` |
## 11. Troubleshooting
- **Port already in use**: the scripts auto-kill residual processes on the
four port groups above; if another suite (e.g. a different π-Bench
experiment) holds them, stop it first or change the port table in
run_persona.sh.
- **Bridge exits immediately with workspace locked**: another bridge already
holds the same workspace; make sure each persona uses its own
`--workspace-dir` (the scripts allocate one per persona).
- **Runner reports `${USER_API_KEY} ... empty`**: env.sh is unfilled or not
sourced; run_persona.sh sources env.sh automatically — when running the
runner manually, `source env.sh` first.
- **`Cannot import 'reme'`**: the bridge must run with
`${REME_DIR}/.venv/bin/python` (run_persona.sh already does); otherwise
check that `REME_DIR` points at the ReMe repository root.
- **AppWorld fails to start**: run `bash scripts/setup_appworld.sh` in the
π-Bench repo first (downloads data); inspect
`logs/appworld_*_<persona>.log`.
- **trace_history.yaml not found**: the runner needs
`config/bench/evaluation/trace_history.yaml`; this suite ships the file and
passes it explicitly via `--history-config-path`, and run_persona.sh fails
fast with a clear error if it is missing. Always launch run_persona.sh /
run_all.sh from the suite directory.
## 12. Privacy and security
- The suite code and config templates contain **no real API keys, user names
or absolute paths**; real keys live only in your local `env.sh`
(git-ignored).
- `logs/`, `outputs/`, `reme_workspace/` and `nanobot_workspace/` contain
full conversations and model outputs; never commit or share them.
- The `data` symlink points at the official π-Bench evaluation data; respect
its data license terms.

View file

@ -0,0 +1,284 @@
# π-Bench 评测说明
[English version](./README.md)
**ReMe agent带持久记忆** 接入 **π-Bench**Proactive Personal Assistant
Benchmark的胶水层评测套件。只含对接所需的最小代码与配置π-Bench 框架
`src/`)、评测数据(`data/`、AppWorld 工具环境、ReMe 本体均为**外部第三方
依赖**,通过符号链接与环境变量原位引用,不随本套件分发。
- π-Bench: https://github.com/Simplified-Reasoning/Pi-Bench arXiv: 2605.14678
- ReMe: 你所在 ReMe 仓库的根目录(本套件推荐放在 `ReMe/benchmark/pibench/`
## 1. 架构总览
```
π-Bench runner (src.main --mode run)
│ user_agent模拟用户 LLM按 data/{persona}/episode.yaml 顺序
│ 逐任务、多轮地与 agent 对话,并在 run 阶段判定隐藏意图(PROC)
test server (π-Bench scripts/test_server.py, HTTP 长轮询)
▲ /send │ /poll
│ ▼
bridge_reme.py ──────────────► ReMe Application以库方式内嵌启动
│ ├─ agent_wrapper: 被测 agentAgentScope
│ ├─ jobs: search / auto_memory / daily_write
│ └─ workspace: reme_workspace/{persona}/
│ (每 persona 独立持久记忆库,互不可见)
└──── MCP ────► AppWorld MCP ────► AppWorld API工具/应用环境)
π-Bench runner (src.main --mode eval)
judger裁判 LLM读取 trace按 checklist 逐条 YES/NO 打分(COMP)
```
要点:
- bridge 用 **ReMe 自己的 venv python** 运行,把 ReMe 当库用(`resolve_app_config`
+ `Application`**ReMe 源码零改动**。
- 每条用户消息都会自动触发一次 ReMe memory `search` 并把命中记忆注入当前消息
(参数见 §8任务结束reset时会话被 `auto_memory` 提炼为 daily 笔记落盘。
- agent 执行的每一轮工具调用AppWorld MCP + ReMe job 工具)都会被采集并以
`tool_steps` 形式写入 trace供 π-Bench 的 `tools_evaluation_path` 脚本
对工具行为评分§7
- π-Bench 的 `data/``src/`、AppWorld 均不属于本套件,需先装好 π-Bench§3.1)。
## 2. 目录结构
```
pibench/
├── README.md / README_ZH.md # 本文档(英文 / 中文)
├── env.sh.example # 环境配置模板(复制为 env.sh 后填写 TODO 项)
├── bridge_reme.py # ReMe ↔ test server 桥接(记忆注入/保存、
│ # profile 注入、工具调用轨迹采集)
├── run_persona.sh # 单 persona 全流程5 个服务 + run + eval
├── run_all.sh # 5 个 persona 批跑fresh/resume默认 2 并行)
├── resume.py # 断点续跑:完成判定 + 中断任务残留记忆的外科清理
├── fix_trace_logs.py # run 输出 → ~/.nanobot/trace_logs 转换,
│ # 并把工具轨迹合并进 turn 文件eval 前置)
├── .gitignore # 排除 env.sh 与全部运行产物
└── config/
├── models/reme.yaml # runner 模型配置model_id=reme
└── bench/evaluation/trace_history.yaml # trace 渲染策略(随套件提供,
# 经 --history-config-path 显式传入)
```
运行时自动生成(均被 .gitignore 排除):`data`(符号链接)、`logs/`
`outputs/``reme_workspace/``nanobot_workspace/`
## 3. 前置依赖(第三方,先装好)
### 3.1 π-Bench 仓库(含 AppWorld
```bash
git clone https://github.com/Simplified-Reasoning/Pi-Bench.git <pi-bench-dir>
cd <pi-bench-dir>
python3.11 -m venv .venv # 脚本约定使用 .venv 这个目录名
source .venv/bin/activate
pip install -e . # pibench runnersrc.main
bash scripts/setup_appworld.sh # 安装 AppWorld 并下载其数据(体积较大,需网络)
```
装完自检:
```bash
ls data/ # 应含 researcher marketer pharmacist law_trainee Financier
.venv/bin/python -c "import src" && echo OK
.venv/bin/appworld --help >/dev/null && echo OK
```
### 3.2 ReMe 仓库
```bash
cd <reme-dir> # ReMe 仓库根目录(含 reme/ 包)
python3.11 -m venv .venv # 脚本约定使用 .venv 这个目录名
source .venv/bin/activate
pip install -e . # 或按 ReMe 自身安装方式,保证 `import reme` 可用
```
自检:`.venv/bin/python -c "import reme; print('ok')"`
## 4. 安装本套件(逐步)
1. **放置套件**(推荐放进 ReMe 仓库,`REME_DIR` 可自动推断):
```bash
cp -r pibench <reme-dir>/benchmark/pibench
cd <reme-dir>/benchmark/pibench
```
若放在其他位置,稍后在 env.sh 中显式设置 `REME_DIR`
2. **创建环境文件并填写自定义参数**
```bash
cp env.sh.example env.sh
```
打开 `env.sh`,必填项(标 TODO 的):
| 变量 | 说明 |
|---|---|
| `PI_BENCH_ROOT` | π-Bench 仓库根目录(含 `src/` `data/` `.venv` `third_party/appworld` |
| `USER_API_KEY` | 模拟用户 LLM 的 API keyrun 阶段判定隐藏意图) |
| `JUDGER_API_KEY` | 裁判 LLM 的 API keyeval 阶段 checklist 打分) |
| `BRAVE_SEARCH_API_KEY` | 可选agent 的 web_search 工具用,不用填 `dummy` |
可选调整:`REME_MODEL_NAME`(被测 agent 基模)、`REME_DIR`
`REME_LLM_BASE_URL`(默认 DashScope OpenAI 兼容端点)。
3. **链接评测数据**(π-Bench 数据原位引用,不复制):
```bash
ln -s "$PI_BENCH_ROOT/data" data
```
4. **(可选)调整模型配置** `config/models/reme.yaml`
- `user_agent.model` / `judger.model`:模拟用户与裁判的模型名(字面量,
π-Bench 仅对 base_url/api_key 做 `${ENV}` 展开)。
- `run.turn_timeout``max_tool_iterations` 等按需。
5. **冒烟自检**(不启动评测):
```bash
bash -n run_all.sh && bash -n run_persona.sh
source env.sh && "$REME_DIR/.venv/bin/python" -c "import reme; print('reme ok')"
```
## 5. 运行评测
> ⚠️ 长时间运行请放进 `screen`**不要用 nohup**nohup 在沙箱/受限环境下
> 会丢失权限上下文导致子进程异常)。
```bash
# 完整正式评测:先清空全部 persona 的记忆/输出/trace再从头跑默认 fresh2 并行)
mkdir -p logs # 全新部署时 logs/ 尚不存在,先建再重定向
screen -dmS pibench_suite bash -c "cd $(pwd) && bash run_all.sh > logs/run_all_master.log 2>&1"
# 断点续跑(中断后继续;不清记忆,跳过已完成任务)
bash run_all.sh --resume
# 其他用法
bash run_all.sh --parallel 1 # 串行
bash run_all.sh --resume --skip-eval # 只跑 run 阶段
bash run_persona.sh researcher # 单 persona默认 --resume 语义)
bash run_persona.sh researcher --fresh
```
耗时参考5 persona × 20 任务、2 并行fresh 全量约 1214 小时。
任一 persona 失败时 `run_all.sh` 以非零状态退出,上层自动化不会把部分失败
的评测误判为成功。
## 6. 端口分配(多 persona 并行互不冲突)
| persona | AppWorld API | AppWorld MCP | Test Server | ReMe 内部服务 |
|-------------|------|-------|------|-------|
| marketer | 9001 | 10001 | 9998 | 18766 |
| law_trainee | 9002 | 10002 | 9997 | 18767 |
| pharmacist | 9003 | 10003 | 9996 | 18768 |
| researcher | 9004 | 10004 | 9995 | 18765 |
| Financier | 9005 | 10005 | 9994 | 18769 |
## 7. 输出与分数
- **结果**`outputs/reme/{persona}/{task}/eval/results/*_result.json`
- `overall_average_score`checklist 完整度COMPjudger 逐条 YES/NO 按依赖组加权)
- `overall_proactiveness_average_score`主动性PROCrun 阶段 user_agent
判定隐藏意图覆盖率;每个任务文件同时携带全局均值)
- **trace**`~/.nanobot/trace_logs/reme/{persona}/{task}/...`eval 的判分输入)
- **日志**`logs/``suite_<persona>.log` 为每 persona 总日志,`bridge_*`
`runner_run/eval_*``appworld_*``test_server_*` 分服务)
- **记忆库**`reme_workspace/{persona}/`daily/digest 笔记、session 原始对话、
BM25 索引等跨运行持久fresh 才清空)
查看汇总:
```bash
grep -h "overall_average_score\|overall_proactiveness" \
outputs/reme/*/*/eval/results/*_result.json | head
```
### 工具轨迹采集tools_evaluation 支持)
部分任务定义了 `objectives.tools_evaluation_path`:用 Python 脚本对工具行为
打分(例如"临时 Todoist 看板已创建并被删除")。这些脚本需要 trace 里有真实
的工具调用记录。采集链路:
1. 每轮 `reply()` 之后bridge 读取 AgentScope 落盘的会话状态,提取本轮新增
`tool_call` / `tool_result` 块(工具名、参数、结果)。
2. 记录按 turn 编号追加写入
`outputs/reme/{persona}/{task}/history/{ts}-tools.jsonl`AgentScope 的
MCP 工具名(`mcp__AppWorld__<tool>`)会规范化为 π-Bench 约定
`mcp_appworld_<tool>`)。
3. `fix_trace_logs.py` 将每个 `{ts}-messages.jsonl` 运行与时间上最接近的
tools 旁路文件配对,把记录合并进生成的 `turn_N.json``tool_steps`
字段——这是 π-Bench `collect_tool_history()` 支持的两种工具轨迹格式之一。
4. eval 阶段 `tool_steps` 既提供给 tools_evaluation 脚本,也会被渲染为
judger 可见的 `<tool_trace_extracts>`
## 8. 记忆机制(本套件的核心设计)
- **persona 隔离**:每个 persona 独立 workspace`reme_workspace/{persona}/`
bridge 启动时对 workspace 加 `.bridge.lock` 排他锁,两个 bridge 不可能共用
同一记忆库;一个 persona 的 memory search 永远接触不到其他 persona 的记忆。
- **写入**任务结束runner 发送 reset会话经 `auto_memory` job 提炼为
daily 笔记落盘,后台 watcher 建 BM25 索引。保存为非阻塞后台任务,
新会话首条消息会先等待在途写入完成再检索。
- **读取**bridge 每收到一条用户消息自动 `search` 一次并注入命中记忆
`[Relevant memories from previous sessions]` 前缀),无命中则原样透传。
检索参数bridge 命令行,可在 run_persona.sh 中调整):
- `--search-limit 3`:每条消息最多注入 3 个记忆块;
- `--search-min-score 2.0`:过滤弱 BM25 命中;
- `tool_context_id` 按任务轮换:同一任务内已注入的记忆块不重复注入
ReMe 自带 seen-chunk 去重24h TTL任务边界后恢复正常召回。
- **无自泄漏**进行中的会话尚未入库save 发生在 reset任务不会检索到
自己未完成的内容。
- agent 同时持有 `search`/`daily_write` 工具,可主动检索/记录。
- **system prompt**`bridge_reme.py:build_system_prompt()` 内置
HIDDEN-NEEDS 协议(面向 proactiveness并把 `data/{persona}/profile.yaml`
的 persona profile 注入每轮 system prompt。
## 9. 断点续跑与记忆清理语义
- **完成判定**resume.py扫描 `outputs/reme/{persona}/**/history/*-log.jsonl`
`outputs/reme/{persona}/run/*-log.jsonl` 中的
`Task finished task_id=X status=Y`。每个任务以**事件时间最新**的记录为准
(优先取记录的 `timestamp`,回退 `timestamp_iso`,再回退日志文件名中的
时间戳)——文件类别与读取顺序本身不能覆盖更新的记录,因此旧的 run 级
SUCCESS 不会掩盖更新的 per-task ERROR。`SUCCESS/MAX_TURNS/TIMEOUT` 记为
完成,`ERROR`/未开始的任务重跑(按 episode 顺序以 `--task-id` 传给 runner
- **防答案泄漏**:被中断的任务可能已在优雅退出时提炼成 daily 笔记,直接重跑会
把答案注入、抬高分数。因此 resume 启动前 `resume.py cleanup` **只删除待重跑
任务**的残留记忆daily/digest 笔记、session/dialog、mem_session
`session_id = pibench_{task}_*` 匹配已完成任务的记忆一律不动。daily
索引**只刷新实际发生删除的日期**,按完整的 workspace 相对 wikilink 路径
匹配;当 ReMe 包可导入时,刷新直接复用 ReMe 自带的 daily 索引重建逻辑
`refresh_day_index`),不会误改其他日期下的同名笔记条目。
- **fresh vs resume 互斥**:全量清记忆只属于 fresh 模式(`run_all.sh` 默认,
在任何服务启动前执行resume 永不清全量。
## 10. 自定义与调优入口
| 目标 | 位置 |
|---|---|
| 被测 agent 基模 | `env.sh``REME_MODEL_NAME` |
| user_agent / judger 模型 | `config/models/reme.yaml` |
| agent system prompt | `bridge_reme.py` `build_system_prompt()` |
| 记忆检索条数/阈值 | `run_persona.sh` bridge 启动命令的 `--search-limit/--search-min-score` |
| ReMe 内部参数 | **不要改 ReMe 源码**;仿照 `reme/config/beam.yaml` 写专有配置,经 `resolve_app_config(config=...)` 覆盖(见 bridge `_init_reme_app` |
| 轮超时/工具迭代上限 | `config/models/reme.yaml` `run.turn_timeout``model.max_tool_iterations` |
## 11. 故障排查
- **端口被占用**:脚本会自动 kill 上述 4 组端口上的残留进程;若与其他套件
(如别的 π-Bench 实验)冲突,请先停掉对方或改 run_persona.sh 的端口表。
- **bridge 启动即退出,提示 workspace locked**:另一个 bridge 正占用同一
workspace确认每个 persona 用各自的 `--workspace-dir`(脚本已按 persona 分配)。
- **runner 报 `${USER_API_KEY} ... empty`**env.sh 未填写或未生效;
run_persona.sh 会自动 source env.sh手动运行 runner 时请先 `source env.sh`
- **`Cannot import 'reme'`**bridge 必须用 `${REME_DIR}/.venv/bin/python` 运行
run_persona.sh 已如此),或检查 `REME_DIR` 是否指向 ReMe 仓库根目录。
- **AppWorld 启动失败**:先在 π-Bench 仓库执行 `bash scripts/setup_appworld.sh`
下载数据;查看 `logs/appworld_*_<persona>.log`
- **trace_history.yaml 找不到**runner 需要
`config/bench/evaluation/trace_history.yaml`;本套件已随附该文件并通过
`--history-config-path` 显式传入run_persona.sh 启动前会做存在性检查,
缺失时立即报出清晰错误。请始终从套件目录启动 run_persona.sh / run_all.sh。
## 12. 隐私与安全
- 套件代码与配置模板中**不含任何真实 API key、用户名或绝对路径**
真实 key 只存在于你本地的 `env.sh`(已被 .gitignore 排除)。
- `logs/``outputs/``reme_workspace/``nanobot_workspace/` 含完整对话内容
与模型输出,请勿提交仓库或外传。
- `data` 符号链接指向 π-Bench 官方评测数据,请遵守其数据许可条款。

1039
benchmark/pibench/bridge_reme.py Executable file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,53 @@
version: 1
format:
root_tag: trace
turn_tag: turn
message_tag: message
file_tag: file
tool_call_tag_prefix: tool_call
tool_result_tag_prefix: tool_result
text_policy:
default:
truncate_chars: 1200
mask_newlines: false
field_overrides:
files_read:
truncate_chars: 40000
assistant_content:
truncate_chars: 40000
tool_result_content:
truncate_chars: 40000
fields:
turn:
include_session_key: false
files:
enabled: true
messages:
enabled: true
include_message_role_attr: true
include_message_index_attr: false
include_system: false
include_user: true
include_assistant_thinking_content: false
include_assistant_thinking_reasoning: false
include_assistant_content: true
include_assistant_reasoning: false
include_assistant_tool_calls: false
require_matching_tool_call: true
tool_calls:
include_tool_call_id: false
tools:
web_fetch:
enabled: true
include_tool_call_keys: [url]
include_tool_result: false
web_search:
enabled: true
include_tool_call_keys: [query]
include_tool_result: false

View file

@ -0,0 +1,40 @@
# ReMe model configuration for Pi-Bench
# Uses ReMe's AgentScope agent with Dashscope as the LLM backend
model:
model: reme
base_url: "http://localhost:8088"
api_key: "dummy"
provider: custom
max_tokens: 16384
max_tool_iterations: 120
memory_window: 100
user_agent:
model: qwen3.8-max
base_url: "${USER_BASE_URL}"
api_key: "${USER_API_KEY}"
temperature: 0.0
request_timeout: 360.0
judger:
model: qwen3.8-max
base_url: "${JUDGER_BASE_URL}"
api_key: "${JUDGER_API_KEY}"
temperature: 0.0
request_timeout: 360.0
tools:
brave_search_api_key: "${BRAVE_SEARCH_API_KEY}"
web_search_max_results: 10
nanobot:
trace_logs_dir: "~/.nanobot/trace_logs"
workspace_dir: "~/.nanobot/workspace"
copy_task_assets_to_workspace: true
run:
output_dir: outputs
log_level: INFO
user_mode: llm
turn_timeout: 2400.0

View file

@ -0,0 +1,57 @@
#!/bin/bash
# ═══════════════════════════════════════════════════════════════════════
# pibench evaluation suite - environment configuration template
# Usage: cp env.sh.example env.sh, then fill in the TODO items below.
# ⚠️ env.sh contains real API keys; never commit or share it
# (already excluded via .gitignore).
# ═══════════════════════════════════════════════════════════════════════
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ─── TODO: π-Bench repository root ────────────────────────────────────
# Must contain src/, data/, scripts/test_server.py, third_party/appworld
# and .venv (see README setup).
export PI_BENCH_ROOT=""
# ─── ReMe repository ──────────────────────────────────────────────────
# Defaults to two levels above this directory (the layout this suite uses
# when placed at ReMe/benchmark/pibench); point it at the actual ReMe
# repository root if the suite lives elsewhere.
export REME_DIR="${REME_DIR:-$(cd "${SUITE_DIR}/../.." && pwd)}"
# ─── Base model of the agent under test (LLM used by the ReMe agent) ──
export REME_MODEL_NAME="${REME_MODEL_NAME:-qwen3.6-plus}"
# ─── LLM service endpoint (default: DashScope OpenAI-compatible; any
# OpenAI-compatible endpoint works) ────────────────────────────────
DASHSCOPE_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export REME_LLM_BASE_URL="${REME_LLM_BASE_URL:-${DASHSCOPE_BASE_URL}}"
# ─── TODO: API keys ───────────────────────────────────────────────────
# USER_API_KEY : drives the simulated user LLM (run phase; judges whether
# hidden intents are satisfied and asks follow-ups)
# JUDGER_API_KEY: drives the judger LLM (eval phase; scores the checklist)
# The two may be identical; one strong model is recommended for both.
export USER_BASE_URL="${DASHSCOPE_BASE_URL}"
export USER_API_KEY="TODO-fill-in-user-agent-api-key"
export JUDGER_BASE_URL="${DASHSCOPE_BASE_URL}"
export JUDGER_API_KEY="TODO-fill-in-judger-api-key"
# The ReMe agent's key reuses USER_API_KEY by default (no need to repeat
# it when both use the same service and key).
export REME_LLM_API_KEY="${REME_LLM_API_KEY:-${USER_API_KEY}}"
# Brave Search (optional; used by the agent's web_search tool - use
# "dummy" when not needed).
export BRAVE_SEARCH_API_KEY="TODO-optional-brave-search-key-or-dummy"
# ─── Persistent memory workspaces (one subdirectory per persona,
# created automatically) ───────────────────────────────────────────
export REME_WORKSPACE_ROOT="${REME_WORKSPACE_ROOT:-${SUITE_DIR}/reme_workspace}"
# ─── Variables consumed by ReMe's default.yaml model config expansion;
# do not remove ────────────────────────────────────────────────────
export LLM_MODEL_NAME="${REME_MODEL_NAME}"
export LLM_BASE_URL="${REME_LLM_BASE_URL}"
export LLM_API_KEY="${REME_LLM_API_KEY}"

View file

@ -0,0 +1,198 @@
#!/usr/bin/env python3
"""Convert reme_eval run outputs into eval-compatible trace logs.
outputs/{model_id}/{user_id}/{task_id}/history/{ts}-messages.jsonl
-> ~/.nanobot/trace_logs/{model_id}/{user_id}/{task_id}/{ts}/turn_N.json
The bridge additionally writes {ts}-tools.jsonl sidecar files next to the
message histories: one JSON object per executed tool call with fields
{turn, name, arguments, result}. Each messages run is paired with the
temporally closest sidecar, and the records are merged into the generated
turn files under the "tool_steps" key, which is one of the tool-history
formats π-Bench's collect_tool_history() understands. Without this step,
tools_evaluation scripts would see no tool evidence at all.
Usage: python fix_trace_logs.py [user_id ...] (no args = all users)
"""
import json
import re
import sys
from datetime import datetime
from pathlib import Path
SUITE_DIR = Path(__file__).resolve().parent
OUTPUTS_DIR = SUITE_DIR / "outputs"
TRACE_LOGS_DIR = Path.home() / ".nanobot" / "trace_logs"
MESSAGES_FILE_RE = re.compile(r"^(\d{8}_\d{6})-messages\.jsonl$")
TOOLS_FILE_RE = re.compile(r"^(\d{8}_\d{6})-tools\.jsonl$")
TIME_FORMAT = "%Y%m%d_%H%M%S"
# A tool sidecar belongs to the messages run that started at most this many
# seconds earlier (the bridge stamps the sidecar when the task's first user
# message arrives, shortly after the runner opened the messages file).
MAX_PAIR_DELTA_SECONDS = 6 * 3600
def _to_epoch(timestamp: str) -> float:
"""Parse a YYYYMMDD_HHMMSS timestamp into epoch seconds."""
try:
return datetime.strptime(timestamp, TIME_FORMAT).timestamp()
except ValueError:
return 0.0
def load_tool_records(tools_file: Path) -> dict:
"""Group sidecar tool records by turn number."""
by_turn: dict = {}
try:
with open(tools_file, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if not isinstance(record, dict) or not record.get("name"):
continue
turn = int(record.get("turn") or 0)
by_turn.setdefault(turn, []).append(
{
"name": record["name"],
"arguments": record.get("arguments", {}),
"result": record.get("result", ""),
},
)
except OSError as exc:
print(f" WARNING: cannot read tool sidecar {tools_file}: {exc}")
return by_turn
def pair_tool_sidecars(message_runs: list, tool_runs: list) -> dict:
"""Pair each messages run with the temporally closest unused tool sidecar.
Fresh runs produce exactly one messages file and one sidecar per task;
re-runs append matching pairs, so sorted greedy nearest-timestamp
matching is stable. Sidecars farther away than MAX_PAIR_DELTA_SECONDS
(e.g. leftovers of a crashed bridge) stay unpaired.
"""
pairing: dict = {}
unused = list(tool_runs)
for msg_ts, _ in message_runs:
best_delta = None
best_item = None
for tool_ts, tool_path in unused:
delta = abs(_to_epoch(tool_ts) - _to_epoch(msg_ts))
if best_delta is None or delta < best_delta:
best_delta = delta
best_item = (tool_ts, tool_path)
if best_delta is not None and best_item is not None and best_delta <= MAX_PAIR_DELTA_SECONDS:
pairing[msg_ts] = best_item[1]
unused.remove(best_item)
return pairing
def build_turns(messages: list) -> list:
"""Split the flat message list into per-turn [user, assistant] groups."""
turns = []
i = 0
while i < len(messages):
turn_msgs = []
if messages[i]["role"] == "user":
turn_msgs.append({"role": "user", "content": messages[i]["message"]})
i += 1
if i < len(messages) and messages[i]["role"] == "assistant":
turn_msgs.append({"role": "assistant", "content": messages[i]["message"]})
i += 1
if not turn_msgs:
i += 1 # defensive: never spin on unexpected roles
continue
turns.append(turn_msgs)
return turns
def convert_task(model_id: str, user_id: str, task_dir: Path) -> None:
"""Convert one task's history dir into trace turn files with tool_steps."""
history_dir = task_dir / "history"
if not history_dir.is_dir():
return
message_runs = []
tool_runs = []
for msg_file in history_dir.glob("*-messages.jsonl"):
match = MESSAGES_FILE_RE.match(msg_file.name)
if match:
message_runs.append((match.group(1), msg_file))
for tools_file in history_dir.glob("*-tools.jsonl"):
match = TOOLS_FILE_RE.match(tools_file.name)
if match:
tool_runs.append((match.group(1), tools_file))
if not message_runs:
return
message_runs.sort(key=lambda item: item[0])
tool_runs.sort(key=lambda item: item[0])
pairing = pair_tool_sidecars(message_runs, tool_runs)
print(f"\n{model_id}/{user_id}/{task_dir.name}")
for timestamp, msg_file in message_runs:
trace_dir = TRACE_LOGS_DIR / model_id / user_id / task_dir.name / timestamp
trace_dir.mkdir(parents=True, exist_ok=True)
messages = []
with open(msg_file, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
msg = json.loads(line)
if msg.get("role") == "user" and msg.get("message") == "/new":
continue
messages.append(msg)
tools_file = pairing.get(timestamp)
tools_by_turn = load_tool_records(tools_file) if tools_file else {}
if tools_file is not None:
print(f" {timestamp}: paired tool sidecar {tools_file.name}")
turns = build_turns(messages)
for turn_idx, turn_msgs in enumerate(turns, start=1):
turn_data = {"messages": turn_msgs}
tool_steps = tools_by_turn.get(turn_idx)
if tool_steps:
turn_data["tool_steps"] = tool_steps
turn_file = trace_dir / f"turn_{turn_idx}.json"
with open(turn_file, "w", encoding="utf-8") as f:
json.dump(turn_data, f, indent=2, ensure_ascii=False)
tool_total = sum(len(steps) for steps in tools_by_turn.values())
print(f" {timestamp}: {len(turns)} turns, {tool_total} tool step(s) -> {trace_dir}")
def convert_outputs(user_filter=None):
"""Convert message history JSONL files into per-turn trace JSON files."""
if not OUTPUTS_DIR.exists():
print(f"outputs dir not found: {OUTPUTS_DIR}")
return
for model_dir in sorted(OUTPUTS_DIR.iterdir()):
if not model_dir.is_dir():
continue
model_id = model_dir.name
for user_dir in sorted(model_dir.iterdir()):
if not user_dir.is_dir():
continue
user_id = user_dir.name
if user_filter and user_id not in user_filter:
continue
for task_dir in sorted(user_dir.iterdir()):
if task_dir.is_dir():
convert_task(model_id, user_id, task_dir)
if __name__ == "__main__":
convert_outputs(set(sys.argv[1:]) or None)
print("\ndone")

332
benchmark/pibench/resume.py Executable file
View file

@ -0,0 +1,332 @@
#!/usr/bin/env python3
"""Checkpoint-resume support for the reme_eval suite.
Completion source of truth:
- outputs/reme/<persona>/<task_id>/history/*-log.jsonl (per-task logs,
flushed incrementally, survive mid-run kills)
- outputs/reme/<persona>/run/*-log.jsonl (run-level logs,
may be truncated if the process was killed before flush)
lines: "Task finished task_id=<id> status=<STATUS>"
A task counts as COMPLETED when its latest terminal status is one of
SUCCESS / MAX_TURNS / TIMEOUT. ERROR or never-started tasks stay pending.
"Latest" is decided by EVENT TIME, not by file category or read order:
each record's "timestamp" (epoch seconds, or "timestamp_iso" as fallback)
is compared across per-task and run-level logs alike, with the timestamp
embedded in the log file name as a last-resort fallback. This keeps an
old run-level SUCCESS from overriding a newer per-task ERROR when the
re-run died before the new run-level log captured the task.
Commands:
remaining <persona> [--json]
Print task_ids still to run, in data/<persona>/episode.yaml order
(one per line; --json prints {"completed": [...], "remaining": [...]}).
cleanup <persona> [--dry-run]
Surgically remove residual memory artifacts of tasks that are about
to be RE-RUN (i.e. pending tasks that left partial state because a
previous run was interrupted). This prevents answer leakage: an
interrupted task's conversation may already have been distilled into
daily notes during graceful shutdown, and re-running the task with
that memory injected would inflate scores.
Removed artifacts (only for pending tasks with residual state):
- daily/<date>/<note>.md whose frontmatter session_id matches
pibench_<task_id>_*, plus a refresh of ONLY the daily index of
the affected date(s) (daily/<date>.md), matched by the full
workspace-relative note path, never by bare file name
- digest notes with matching session_id
- session/dialog/pibench_<task_id>_*.jsonl
- mem_session/**.jsonl files containing pibench_<task_id>_
When the ReMe package is importable, the daily index refresh reuses
ReMe's own rebuild logic (reme.steps.file_io._daily_index.
refresh_day_index); otherwise index lines are dropped by exact
wikilink path match. Either way, indexes of other dates are never
touched. The ReMe watcher (init_changes_step) detects the deleted
daily notes on next bridge startup and removes them from the BM25
index itself.
Completed tasks' memories are NEVER touched by this command.
Design note (resume vs memory-wipe conflict):
A full memory wipe is a suite-level action of fresh mode (run_all.sh
without --resume) and happens before any service starts. Resume mode
never wipes; it only performs the surgical cleanup above. The two modes
are mutually exclusive, so a resumed run can never lose the cross-session
memory accumulated by completed tasks.
"""
import asyncio
import json
import os
import re
import sys
from datetime import datetime
from pathlib import Path
import yaml
try: # Reuse ReMe's daily-index rebuild when running inside the ReMe venv.
from reme.steps.file_io._daily_index import refresh_day_index
except ImportError: # pragma: no cover - depends on runtime venv
refresh_day_index = None
SUITE_DIR = Path(__file__).resolve().parent
DATA_DIR = Path(os.environ.get("REME_EVAL_DATA_DIR", SUITE_DIR / "data")).resolve()
OUTPUTS_DIR = Path(os.environ.get("REME_EVAL_OUTPUTS_DIR", SUITE_DIR / "outputs")) / "reme"
WORKSPACE_ROOT = Path(
os.environ.get("REME_WORKSPACE_ROOT", SUITE_DIR / "reme_workspace"),
).resolve()
COMPLETED_STATUSES = {"SUCCESS", "MAX_TURNS", "TIMEOUT"}
TASK_FINISHED_RE = re.compile(r"Task finished task_id=(\S+) status=(\S+)")
SESSION_ID_RE = re.compile(r"^session_id:\s*(\S+)", re.MULTILINE)
NOTE_COUNT_RE = re.compile(r"(description:\s*)\d+(\s*note\(s\) today)")
LOG_FILE_TS_RE = re.compile(r"^(\d{8}_\d{6})-log\.jsonl$")
TIME_FORMAT = "%Y%m%d_%H%M%S"
def log(msg: str) -> None:
"""Print a status message to stderr."""
print(msg, file=sys.stderr)
def episode_task_order(persona: str) -> list[str]:
"""Return the ordered task ids from the persona's episode.yaml."""
episode_path = DATA_DIR / persona / "episode.yaml"
with open(episode_path, "r", encoding="utf-8") as f:
episode = yaml.safe_load(f)
return [task["task_id"] for task in episode.get("tasks", [])]
def _event_time(record: dict, file_ts: str) -> float:
"""Best-effort event time (epoch seconds) of one log record.
Prefers the record's own timestamp fields; falls back to the timestamp
embedded in the log file name so that even stripped records keep a
meaningful order. Returns 0.0 when nothing is parseable.
"""
timestamp = record.get("timestamp")
if isinstance(timestamp, (int, float)) and not isinstance(timestamp, bool):
return float(timestamp)
iso = record.get("timestamp_iso")
if isinstance(iso, str):
try:
return datetime.fromisoformat(iso).timestamp()
except ValueError:
pass
if file_ts:
try:
return datetime.strptime(file_ts, TIME_FORMAT).timestamp()
except ValueError:
pass
return 0.0
def latest_task_statuses(persona: str) -> dict[str, str]:
"""Scan per-task and run-level logs; the newest EVENT TIME wins per task.
Every "Task finished" record across both log categories is keyed by
(event_time, file timestamp, file order, line number); the record with
the highest key decides the task's status. File category and read order
alone can never override a newer record from the other category.
"""
persona_dir = OUTPUTS_DIR / persona
if not persona_dir.is_dir():
return {}
log_files = sorted(persona_dir.glob("*/history/*-log.jsonl"))
log_files += sorted(persona_dir.glob("run/*-log.jsonl"))
best: dict[str, tuple[tuple, str]] = {}
for file_order, log_file in enumerate(log_files):
ts_match = LOG_FILE_TS_RE.match(log_file.name)
file_ts = ts_match.group(1) if ts_match else ""
try:
with open(log_file, "r", encoding="utf-8") as f:
for line_no, line in enumerate(f):
if "Task finished" not in line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
match = TASK_FINISHED_RE.search(str(record.get("message", "")))
if not match:
continue
task_id, status = match.group(1), match.group(2)
sort_key = (_event_time(record, file_ts), file_ts, file_order, line_no)
current = best.get(task_id)
if current is None or sort_key > current[0]:
best[task_id] = (sort_key, status)
except OSError:
continue
return {task_id: status for task_id, (_, status) in best.items()}
def split_tasks(persona: str) -> tuple[list[str], list[str]]:
"""Split the episode task order into completed and remaining tasks."""
order = episode_task_order(persona)
statuses = latest_task_statuses(persona)
completed = [t for t in order if statuses.get(t) in COMPLETED_STATUSES]
remaining = [t for t in order if t not in set(completed)]
return completed, remaining
def _daily_note_session_id(note_path: Path) -> str:
try:
text = note_path.read_text(encoding="utf-8")
except OSError:
return ""
match = SESSION_ID_RE.search(text)
return match.group(1) if match else ""
class _WorkspaceFileStoreShim:
"""Structural stand-in for ReMe's file store; only workspace_path is read."""
def __init__(self, workspace_path: Path):
self.workspace_path = workspace_path
def _refresh_daily_indexes(
workspace: Path,
removed_by_date: dict[str, set[str]],
removed: list[str],
) -> None:
"""Rebuild the daily index of each affected date via ReMe's own logic."""
for date in sorted(removed_by_date):
result = asyncio.run(
refresh_day_index(_WorkspaceFileStoreShim(workspace), date, "daily"),
)
if result.get("error"):
log(f"[resume] WARNING: daily index refresh failed for {date}: {result['error']}")
continue
removed.append(f"daily/{date}.md (refreshed, {len(removed_by_date[date])} note(s) removed)")
def _strip_index_lines(
workspace: Path,
removed_by_date: dict[str, set[str]],
removed: list[str],
dry_run: bool,
) -> None:
"""Fallback index edit: drop lines that reference removed notes by full
workspace-relative wikilink path, and fix the note count. Only the index
files of affected dates are touched."""
for date in sorted(removed_by_date):
index_path = workspace / "daily" / f"{date}.md"
if not index_path.is_file():
continue
wikilinks = [f"[[{rel_path}]]" for rel_path in sorted(removed_by_date[date])]
lines = index_path.read_text(encoding="utf-8").splitlines()
kept = [line for line in lines if not any(link in line for link in wikilinks)]
if len(kept) == len(lines):
continue
note_count = sum(1 for line in kept if line.startswith("- [[daily/"))
kept = [NOTE_COUNT_RE.sub(rf"\g<1>{note_count}\2", line) for line in kept]
removed.append(f"{index_path.relative_to(workspace)} (rewritten)")
if not dry_run:
index_path.write_text("\n".join(kept) + "\n", encoding="utf-8")
def cleanup_partial_memory(persona: str, remaining: list[str], dry_run: bool = False) -> list[str]:
"""Remove partial memory artifacts of remaining tasks so they can be re-run cleanly."""
workspace = WORKSPACE_ROOT / persona
removed: list[str] = []
if not workspace.is_dir() or not remaining:
return removed
prefixes = tuple(f"pibench_{task_id}_" for task_id in remaining)
def act(path: Path, label: str) -> None:
removed.append(label)
if not dry_run:
path.unlink()
# 1) daily / digest notes distilled from interrupted sessions. For daily
# notes, remember the full workspace-relative path grouped by date so only
# the affected daily indexes are refreshed below.
removed_by_date: dict[str, set[str]] = {}
for section in ("daily", "digest"):
section_root = workspace / section
if not section_root.is_dir():
continue
for note_path in section_root.rglob("*.md"):
if note_path.parent == section_root:
continue # index files handled below
session_id = _daily_note_session_id(note_path)
if session_id.startswith(prefixes):
rel_path = note_path.relative_to(workspace).as_posix()
act(note_path, rel_path)
if section == "daily":
removed_by_date.setdefault(note_path.parent.name, set()).add(rel_path)
# 2) daily index files: refresh only the dates that lost notes, matching
# notes by their full wikilink path instead of their bare file name.
if removed_by_date:
if dry_run:
for date in sorted(removed_by_date):
removed.append(f"daily/{date}.md (would refresh index)")
elif refresh_day_index is not None:
_refresh_daily_indexes(workspace, removed_by_date, removed)
else:
_strip_index_lines(workspace, removed_by_date, removed, dry_run)
# 3) raw dialog logs of interrupted sessions
dialog_dir = workspace / "session" / "dialog"
if dialog_dir.is_dir():
for task_id in remaining:
for dialog_path in dialog_dir.glob(f"pibench_{task_id}_*.jsonl"):
act(dialog_path, str(dialog_path.relative_to(workspace)))
# 4) agent-scope session states that contain interrupted-task sessions
mem_session_dir = workspace / "mem_session"
if mem_session_dir.is_dir():
for session_path in mem_session_dir.rglob("*.jsonl"):
try:
content = session_path.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
if any(prefix in content for prefix in prefixes):
act(session_path, str(session_path.relative_to(workspace)))
return removed
def main() -> int:
"""CLI entrypoint: run 'remaining' or 'cleanup' action for a persona."""
args = sys.argv[1:]
if len(args) < 2 or args[0] not in {"remaining", "cleanup"}:
print(__doc__, file=sys.stderr)
return 2
command, persona = args[0], args[1]
completed, remaining = split_tasks(persona)
if command == "remaining":
if "--json" in args:
print(json.dumps({"completed": completed, "remaining": remaining}))
else:
for task_id in remaining:
print(task_id)
log(
f"[resume] {persona}: completed={len(completed)} "
f"({', '.join(completed) if completed else '-'}) remaining={len(remaining)}",
)
return 0
dry_run = "--dry-run" in args
removed = cleanup_partial_memory(persona, remaining, dry_run=dry_run)
if removed:
verb = "would remove" if dry_run else "removed"
log(f"[resume] {persona}: {verb} {len(removed)} partial-memory artifact(s):")
for item in removed:
log(f" - {item}")
else:
log(f"[resume] {persona}: no partial-memory artifacts to clean")
return 0
if __name__ == "__main__":
sys.exit(main())

119
benchmark/pibench/run_all.sh Executable file
View file

@ -0,0 +1,119 @@
#!/bin/bash
# Run all 5 personas with the ReMe agent, PARALLEL at a time (default 2).
# Each persona's tasks follow data/{persona}/episode.yaml order.
#
# Usage:
# bash run_all.sh # FRESH official run: wipes ALL personas'
# # ReMe memory/outputs/trace logs first,
# # then runs everything from scratch.
# bash run_all.sh --resume # Checkpoint continuation: no wipe; every
# # persona skips already-completed tasks.
# bash run_all.sh --parallel 1 # sequential (original behavior)
# bash run_all.sh --skip-eval # run phase only
#
# Memory-wipe vs resume conflict resolution:
# The full ReMe memory wipe happens ONLY here, ONLY in fresh mode (the
# default), and ONLY before any service/bridge starts. --resume never
# wipes; run_persona.sh then additionally performs a surgical cleanup of
# residual memory belonging to interrupted (to-be-re-run) tasks, so a
# resumed run keeps all completed-task memory but never inherits a partial
# task's own answer. The two modes are mutually exclusive.
set -uo pipefail
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PERSONAS=(researcher marketer law_trainee pharmacist Financier)
TRACE_ROOT="${HOME}/.nanobot/trace_logs"
PARALLEL=2
MODE="fresh"
PASS_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
--parallel)
PARALLEL="${2:-}"; shift 2 || true
case "$PARALLEL" in (""|*[!0-9]*) echo "--parallel needs a positive integer"; exit 2 ;; esac
[ "$PARALLEL" -lt 1 ] && PARALLEL=1
[ "$PARALLEL" -gt ${#PERSONAS[@]} ] && PARALLEL=${#PERSONAS[@]}
;;
--resume)
if [ "$MODE" = "fresh_set" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="resume"; shift ;;
--fresh)
if [ "$MODE" = "resume" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="fresh_set"; shift ;;
--skip-eval) PASS_ARGS+=(--skip-eval); shift ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
[ "$MODE" = "fresh_set" ] && MODE="fresh"
START_TS=$(date +%Y%m%d_%H%M%S)
SUMMARY_LOG="${SUITE_DIR}/logs/run_all_${START_TS}.summary"
mkdir -p "${SUITE_DIR}/logs"
echo "############################################################"
echo "# reme_eval suite | mode=${MODE} parallel=${PARALLEL} | ${START_TS}"
echo "############################################################"
# ─── Fresh mode: suite-level wipe BEFORE anything starts ──────────────
if [ "$MODE" = "fresh" ]; then
echo "[fresh] wiping ALL personas' memory workspaces, outputs and trace logs..."
for persona in "${PERSONAS[@]}"; do
rm -rf "${SUITE_DIR}/reme_workspace/${persona}"
rm -rf "${SUITE_DIR}/outputs/reme/${persona}"
rm -rf "${TRACE_ROOT}/reme/${persona}"
rm -rf "${SUITE_DIR}/nanobot_workspace/${persona}"
done
echo "[fresh] wipe done."
else
echo "[resume] no memory wipe; personas resume after their last completed task."
fi
# ─── Run personas in batches of PARALLEL ──────────────────────────────
STATUS_LIST=()
ANY_FAILED=0
OVERALL_START=$(date +%s)
TOTAL=${#PERSONAS[@]}
for ((i = 0; i < TOTAL; i += PARALLEL)); do
BATCH=("${PERSONAS[@]:i:PARALLEL}")
BATCH_PIDS=()
BATCH_NAMES=()
echo ""
echo "============================================================"
echo "# BATCH $(( i / PARALLEL + 1 )): ${BATCH[*]} started $(date '+%F %T')"
echo "============================================================"
for persona in "${BATCH[@]}"; do
bash "${SUITE_DIR}/run_persona.sh" "${persona}" --resume ${PASS_ARGS[@]+"${PASS_ARGS[@]}"} \
> "${SUITE_DIR}/logs/suite_${persona}.log" 2>&1 &
BATCH_PIDS+=($!)
BATCH_NAMES+=("$persona")
done
for j in $(seq 0 $(( ${#BATCH[@]} - 1 ))); do
pid=${BATCH_PIDS[$j]}
persona=${BATCH_NAMES[$j]}
if wait "$pid"; then
STATUS_LIST+=("${persona}: OK")
else
rc=$?
ANY_FAILED=1
STATUS_LIST+=("${persona}: FAILED rc=${rc}")
echo "[run_all] ${persona} FAILED (rc=${rc}); see logs/suite_${persona}.log"
fi
done
done
total=$(( $(date +%s) - OVERALL_START ))
echo ""
echo "================ FINAL SUMMARY (${total}s total) ================" | tee -a "${SUMMARY_LOG}"
for line in "${STATUS_LIST[@]}"; do
echo " ${line}" | tee -a "${SUMMARY_LOG}"
done
echo "Summary: ${SUMMARY_LOG}"
if [ "${ANY_FAILED}" -ne 0 ]; then
FAILED_COUNT=$(printf '%s\n' "${STATUS_LIST[@]}" | grep -c "FAILED")
echo "[run_all] ${FAILED_COUNT} persona(s) FAILED; suite run is marked as failed." | tee -a "${SUMMARY_LOG}"
exit 1
fi
exit 0

301
benchmark/pibench/run_persona.sh Executable file
View file

@ -0,0 +1,301 @@
#!/bin/bash
# Run the full pi-bench evaluation for ONE persona with the ReMe agent.
# Tasks follow data/{persona}/episode.yaml order (runner-native).
#
# Usage: bash run_persona.sh <persona> [--fresh|--resume] [--skip-eval]
#
# Modes (default: --resume):
# --resume Checkpoint continuation. Never wipes memory. Tasks already
# finished (SUCCESS/MAX_TURNS/TIMEOUT in the task history logs)
# are skipped via repeated --task-id flags. Before starting, any
# residual memory of tasks that are about to be RE-RUN (partial
# sessions from an interrupted run) is surgically removed by
# resume.py cleanup, so re-runs don't inherit leaked answers.
# --fresh Wipes THIS persona's ReMe memory, outputs and trace logs first,
# then runs all tasks from scratch.
# The two flags are mutually exclusive. A full multi-persona memory wipe is a
# suite-level action of `run_all.sh` (fresh mode), never done here implicitly.
set -uo pipefail
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TRACE_ROOT="${HOME}/.nanobot/trace_logs"
# ─── External dependencies (pi-bench / ReMe are NOT bundled; see README) ──
if [ ! -f "${SUITE_DIR}/env.sh" ]; then
echo "env.sh not found. Run: cp env.sh.example env.sh (then fill in the TODO items)"
exit 1
fi
source "${SUITE_DIR}/env.sh"
PIBENCH_DIR="${PI_BENCH_ROOT:-}"
if [ -z "${PIBENCH_DIR}" ] || [ ! -f "${PIBENCH_DIR}/src/main.py" ]; then
echo "PI_BENCH_ROOT is unset or invalid (src/main.py not found). Set it in env.sh."
exit 1
fi
if [ ! -x "${PIBENCH_DIR}/.venv/bin/python" ] || [ ! -x "${PIBENCH_DIR}/.venv/bin/appworld" ]; then
echo "pi-bench venv incomplete: ${PIBENCH_DIR}/.venv must provide python + appworld (see README setup)."
exit 1
fi
if [ ! -x "${REME_DIR}/.venv/bin/python" ]; then
echo "ReMe venv not found: ${REME_DIR}/.venv/bin/python (check REME_DIR in env.sh)"
exit 1
fi
if [ ! -e "${SUITE_DIR}/data" ]; then
echo 'Benchmark data not linked. Run: ln -s "$PI_BENCH_ROOT/data" data'
exit 1
fi
# ─── Pre-flight: files the runner needs before any service starts ─────
MODEL_CONFIG="${SUITE_DIR}/config/models/reme.yaml"
HISTORY_CONFIG="${SUITE_DIR}/config/bench/evaluation/trace_history.yaml"
if [ ! -f "${MODEL_CONFIG}" ]; then
echo "Model config not found: ${MODEL_CONFIG} (see README directory layout)."
exit 1
fi
if [ ! -f "${HISTORY_CONFIG}" ]; then
echo "Trace history config not found: ${HISTORY_CONFIG}"
echo "pi-bench requires config/bench/evaluation/trace_history.yaml; see README."
exit 1
fi
APPWORLD_DIR="${PIBENCH_DIR}/third_party/appworld"
PI_PYTHON="${PIBENCH_DIR}/.venv/bin/python"
APPWORLD_BIN="${PIBENCH_DIR}/.venv/bin/appworld"
# resume.py runs on the ReMe venv so it can reuse ReMe's daily-index rebuild.
REME_PYTHON="${REME_DIR}/.venv/bin/python"
PERSONA="${1:-}"
if [ -z "$PERSONA" ]; then
echo "Usage: $0 <persona> [--fresh|--resume] [--skip-eval]"
exit 1
fi
shift
MODE="resume"
SKIP_EVAL=false
while [[ $# -gt 0 ]]; do
case $1 in
--fresh)
if [ "$MODE" = "resume_set" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="fresh"; shift ;;
--resume)
if [ "$MODE" = "fresh" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="resume_set"; shift ;;
--skip-eval) SKIP_EVAL=true; shift ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
[ "$MODE" = "resume_set" ] && MODE="resume"
# ─── Per-persona ports (pi-bench AGENTS.md convention) ────────────────
# REME_PORT: ReMe's internal HTTP service; must be unique per concurrent bridge.
case "$PERSONA" in
marketer) API_PORT=9001; MCP_PORT=10001; TEST_PORT=9998; REME_PORT=18766 ;;
law_trainee) API_PORT=9002; MCP_PORT=10002; TEST_PORT=9997; REME_PORT=18767 ;;
pharmacist) API_PORT=9003; MCP_PORT=10003; TEST_PORT=9996; REME_PORT=18768 ;;
researcher) API_PORT=9004; MCP_PORT=10004; TEST_PORT=9995; REME_PORT=18765 ;;
Financier) API_PORT=9005; MCP_PORT=10005; TEST_PORT=9994; REME_PORT=18769 ;;
*) echo "Unknown persona: $PERSONA"; exit 1 ;;
esac
API_URL="http://127.0.0.1:${API_PORT}"
MCP_URL="http://127.0.0.1:${MCP_PORT}/mcp"
TEST_URL="http://127.0.0.1:${TEST_PORT}"
LOG_DIR="${SUITE_DIR}/logs"
mkdir -p "${LOG_DIR}"
# ─── Environment (env.sh already sourced at the top) ──────────────────
WORKSPACE_DIR="${REME_WORKSPACE_ROOT}/${PERSONA}"
NANOBOT_WORKSPACE_DIR="${SUITE_DIR}/nanobot_workspace/${PERSONA}"
mkdir -p "${WORKSPACE_DIR}" "${NANOBOT_WORKSPACE_DIR}"
echo "========================================="
echo "ReMe x Pi-Bench | persona=${PERSONA} | mode=${MODE}"
echo " api=${API_PORT} mcp=${MCP_PORT} test=${TEST_PORT} reme=${REME_PORT}"
echo " model=${REME_MODEL_NAME}"
echo " memory workspace=${WORKSPACE_DIR} (persistent)"
echo "========================================="
# ─── Fresh mode: wipe this persona's state ────────────────────────────
if [ "$MODE" = "fresh" ]; then
echo "[fresh] wiping persona state: memory workspace, outputs, trace logs"
rm -rf "${WORKSPACE_DIR}"
rm -rf "${SUITE_DIR}/outputs/reme/${PERSONA}"
rm -rf "${TRACE_ROOT}/reme/${PERSONA}"
rm -rf "${NANOBOT_WORKSPACE_DIR}"
mkdir -p "${WORKSPACE_DIR}" "${NANOBOT_WORKSPACE_DIR}"
fi
# ─── Resume: determine remaining tasks + clean partial memories ───────
TASK_ARGS=()
RUN_PHASE_NEEDED=true
if [ "$MODE" = "resume" ]; then
REMAINING_JSON="$("${REME_PYTHON}" "${SUITE_DIR}/resume.py" remaining "${PERSONA}" --json)"
if [ -z "$REMAINING_JSON" ]; then
echo "Failed to compute remaining tasks"; exit 1
fi
echo "[resume] ${REMAINING_JSON}"
REMAINING_TASKS=()
while IFS= read -r tid_line; do
[ -n "$tid_line" ] && REMAINING_TASKS+=("$tid_line")
done < <("${REME_PYTHON}" "${SUITE_DIR}/resume.py" remaining "${PERSONA}" 2>/dev/null)
if [ ${#REMAINING_TASKS[@]} -eq 0 ]; then
RUN_PHASE_NEEDED=false
echo "[resume] all tasks already completed; skipping run phase"
else
# Remove residual memory of interrupted (to-be-re-run) tasks so
# re-runs don't get their own partial answers injected.
"${REME_PYTHON}" "${SUITE_DIR}/resume.py" cleanup "${PERSONA}"
for tid in "${REMAINING_TASKS[@]}"; do
TASK_ARGS+=(--task-id "$tid")
done
echo "[resume] running ${#REMAINING_TASKS[@]} remaining task(s): ${REMAINING_TASKS[*]}"
fi
fi
# ─── Port cleanup from previous runs ──────────────────────────────────
for port in ${API_PORT} ${MCP_PORT} ${TEST_PORT} ${REME_PORT}; do
pids=$(lsof -ti :${port} 2>/dev/null || true)
if [ -n "$pids" ]; then
echo "Killing stale processes on port ${port}: ${pids}"
kill -9 $pids 2>/dev/null || true
fi
done
sleep 2
PIDS=()
cleanup() {
echo "[${PERSONA}] cleaning up services..."
for pid in "${PIDS[@]:-}"; do
kill "$pid" 2>/dev/null || true
done
wait 2>/dev/null || true
}
trap cleanup EXIT INT TERM
wait_for_service() {
local url="$1" name="$2" port="$3" timeout="${4:-180}"
echo -n " waiting for ${name}..."
local start=$(date +%s)
while true; do
if curl -sf --max-time 5 "${url}" > /dev/null 2>&1; then
echo " ready"; return 0
fi
if [ -n "$port" ] && lsof -ti :${port} > /dev/null 2>&1; then
local elapsed=$(( $(date +%s) - start ))
if [ "$elapsed" -ge 10 ]; then echo " ready (port)"; return 0; fi
fi
if [ $(( $(date +%s) - start )) -ge "$timeout" ]; then
echo " TIMEOUT"; return 1
fi
sleep 2
done
}
# ─── [1/5] AppWorld API ────────────────────────────────────────────────
echo "[1/5] AppWorld API (:${API_PORT})"
(cd "${APPWORLD_DIR}" && exec "${APPWORLD_BIN}" serve apis --root . \
--port ${API_PORT}) > "${LOG_DIR}/appworld_api_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${API_URL}/docs" "AppWorld API" "${API_PORT}" 180; then
tail -20 "${LOG_DIR}/appworld_api_${PERSONA}.log"; exit 1
fi
# ─── [2/5] AppWorld MCP ────────────────────────────────────────────────
echo "[2/5] AppWorld MCP (:${MCP_PORT})"
TOOLS_CONFIG="${SUITE_DIR}/data/${PERSONA}/tools.yaml"
(cd "${APPWORLD_DIR}" && exec "${APPWORLD_BIN}" serve mcp http --root . \
--remote-apis-url "${API_URL}" --port ${MCP_PORT} \
--tools-config-file "${TOOLS_CONFIG}") > "${LOG_DIR}/appworld_mcp_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${MCP_URL}" "AppWorld MCP" "${MCP_PORT}" 180; then
tail -20 "${LOG_DIR}/appworld_mcp_${PERSONA}.log"; exit 1
fi
# ─── [3/5] Test Server ─────────────────────────────────────────────────
echo "[3/5] Test Server (:${TEST_PORT})"
PORT=${TEST_PORT} "${PI_PYTHON}" "${PIBENCH_DIR}/scripts/test_server.py" \
> "${LOG_DIR}/test_server_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${TEST_URL}/sent?after=-1" "Test Server" "${TEST_PORT}" 30; then
tail -20 "${LOG_DIR}/test_server_${PERSONA}.log"; exit 1
fi
# ─── [4/5] ReMe Bridge (ReMe venv) ─────────────────────────────────────
echo "[4/5] ReMe Bridge (reme service port ${REME_PORT})"
"${REME_DIR}/.venv/bin/python" "${SUITE_DIR}/bridge_reme.py" \
--test-server-url "${TEST_URL}" \
--appworld-mcp-url "${MCP_URL}" \
--reme-dir "${REME_DIR}" \
--data-root "${SUITE_DIR}/data" \
--user-id "${PERSONA}" \
--workspace-dir "${WORKSPACE_DIR}" \
--reme-port "${REME_PORT}" \
--model-name "${REME_MODEL_NAME}" \
--model-base-url "${REME_LLM_BASE_URL}" \
--model-api-key "${REME_LLM_API_KEY}" \
> "${LOG_DIR}/bridge_${PERSONA}.log" 2>&1 &
BRIDGE_PID=$!
PIDS+=(${BRIDGE_PID})
sleep 5
if ! kill -0 "${BRIDGE_PID}" 2>/dev/null; then
echo "Bridge failed to start:"; tail -30 "${LOG_DIR}/bridge_${PERSONA}.log"; exit 1
fi
for i in $(seq 1 12); do
if grep -q "Bridge started:" "${LOG_DIR}/bridge_${PERSONA}.log" 2>/dev/null; then
echo " bridge initialized"; break
fi
sleep 5
done
grep -q "Bridge started:" "${LOG_DIR}/bridge_${PERSONA}.log" 2>/dev/null || {
echo "WARNING: bridge may not be ready:"; tail -20 "${LOG_DIR}/bridge_${PERSONA}.log"; }
# ─── [5/5] Runner (run phase) ──────────────────────────────────────────
if [ "$RUN_PHASE_NEEDED" = true ]; then
echo "[5/5] Runner: run phase (episode order from data/${PERSONA}/episode.yaml)"
cd "${SUITE_DIR}"
BENCH_TEST_SERVER_URL="${TEST_URL}" PYTHONPATH="${PIBENCH_DIR}" \
"${PI_PYTHON}" -m src.main \
--model-config "${MODEL_CONFIG}" \
--history-config-path "${HISTORY_CONFIG}" \
--mode run --user-id "${PERSONA}" \
--workspace-dir "${NANOBOT_WORKSPACE_DIR}" \
${TASK_ARGS[@]+"${TASK_ARGS[@]}"} \
2>&1 | tee "${LOG_DIR}/runner_run_${PERSONA}.log"
RUN_EXIT=${PIPESTATUS[0]}
if [ ${RUN_EXIT} -ne 0 ]; then
echo "Run phase failed (exit ${RUN_EXIT}). Logs: ${LOG_DIR}/"
exit ${RUN_EXIT}
fi
else
echo "[5/5] Runner: run phase skipped (all tasks completed)"
fi
if [ "$SKIP_EVAL" = true ]; then
echo "Skipping eval (--skip-eval)"
exit 0
fi
# ─── Trace conversion + eval phase (always over all available traces) ──
echo "Converting trace logs..."
"${PI_PYTHON}" "${SUITE_DIR}/fix_trace_logs.py" "${PERSONA}"
echo "Runner: eval phase"
cd "${SUITE_DIR}"
BENCH_TEST_SERVER_URL="${TEST_URL}" PYTHONPATH="${PIBENCH_DIR}" \
"${PI_PYTHON}" -m src.main \
--model-config "${MODEL_CONFIG}" \
--history-config-path "${HISTORY_CONFIG}" \
--mode eval --user-id "${PERSONA}" \
--workspace-dir "${NANOBOT_WORKSPACE_DIR}" \
2>&1 | tee "${LOG_DIR}/runner_eval_${PERSONA}.log"
EVAL_EXIT=${PIPESTATUS[0]}
echo ""
echo "========================================="
echo "persona=${PERSONA} finished (eval exit=${EVAL_EXIT})"
echo " results : ${SUITE_DIR}/outputs/reme/${PERSONA}/"
echo " memory : ${WORKSPACE_DIR}/"
echo " logs : ${LOG_DIR}/"
echo "========================================="
exit ${EVAL_EXIT}

View file

@ -1,54 +0,0 @@
# beam result
## longmemeval版本的prompt
### 100K
agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only复用已构建 memory
(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction 平均 token 消耗default agent, 20 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory
(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction 平均 token 消耗default agent, 35 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

View file

@ -1,92 +0,0 @@
# LongMemEval 数据集测试结果
## cleaned-s
**basic settings**
1. 使用修改后的auto-memory prompt关闭auto-dream机制
2. reme-memory中的全部session的时间一定早于question的时间
**results **
1. Agentic answer框架回答每次最多调用5次search
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-user | 70 | 66 | 4 | 94.3% |
| single-session-assistant | 56 | 52 | 4 | 92.9% |
| knowledge-update | 78 | 60 | 18 | 76.9% |
| multi-session | 133 | 93 | 40 | 69.9% |
| temporal-reasoning | 133 | 78 | 55 | 58.6% |
| single-session-preference | 30 | 8 | 22 | 26.7% |
| **Overall** | **500** | **357** | **143** | **71.4%** |
2. prompted-based amswer每次固定使用原始query召回10个fileChunk
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-assistant | 56 | 56 | 0 | 100.0% |
| single-session-user | 70 | 67 | 3 | 95.7% |
| knowledge-update | 78 | 69 | 9 | 88.5% |
| multi-session | 133 | 99 | 34 | 74.4% |
| temporal-reasoning | 133 | 83 | 50 | 62.4% |
| single-session-preference | 30 | 16 | 14 | 53.3% |
| **Overall** | **500** | **390** | **110** | **78.0%** |
3. golden session。 使用与prompt-based answer相似的方法唯一区别是输入的chunk是longMemEval提供的golden session。
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-assistant | 56 | 56 | 0 | 100.0% |
| single-session-user | 70 | 69 | 1 | 98.6% |
| knowledge-update | 78 | 74 | 4 | 94.9% |
| temporal-reasoning | 133 | 124 | 9 | 93.2% |
| multi-session | 133 | 117 | 16 | 88.0% |
| single-session-preference | 30 | 17 | 13 | 56.7% |
| **Overall** | **500** | **457** | **43** | **91.4%** |
4. golden session + time filter. 和上面一个实验的区别是输入的golden被过滤了一次要求输入session的时间戳必须早于question的时间才行。
一共被过滤掉了75个session44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空一共有424个case。
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| knowledge-update | 78 | 75 | 3 | 96.2% |
| single-session-user | 70 | 67 | 3 | 95.7% |
| multi-session | 133 | 122 | 11 | 91.7% |
| single-session-assistant | 56 | 55 | 1 | 98.2% |
| temporal-reasoning | 133 | 91 | 42 | 68.4% |
| single-session-preference | 30 | 16 | 14 | 53.3% |
| **Overall** | **500** | **426** | **74** | **85.2%** |
5. 关闭auto-memory机制根据原始query一次性混合检索召回原始session计算recall.
| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes |
|---|---|---|---|---|
| knowledge-update | 78 | 75 | 99.3% | 100% |
| single-session-user | 70 | 67 | 100% | 100% |
| multi-session | 133 | 122 | 91.8% | 95.8% |
| single-session-assistant | 56 | 55 | 100% | 100% |
| temporal-reasoning | 133 | 91 | 87.6% | 94.2% |
| single-session-preference | 30 | 16 | 100% | 100% |
| **Overall** | **500** | **426** | **87.6%** | **94.2%** |
## 最终groundtruth
### agentic + prompted最终GT2026-07-16
| Category | Total | Agentic | Prompted limit=15 |
|---|---|---|---|
| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) |
| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) |
| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) |
| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) |
| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) |
| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) |
| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** |
Prompted token 消耗:总 input 13,111,421 (平均 26,275/题),总 output 313,370 (平均 628/题)。
平均 sessions_ingested: 44.8dreams_triggered: 0。

View file

@ -1,324 +0,0 @@
# Auto Fin Cookbook
[中文](README_ZH.md)
Auto Fin is a local-first, file-native ETF event-research workflow. It identifies market events in CLS news, selects
related liquid ETFs, studies similar historical events and subsequent returns, and produces a Chinese research report.
> Auto Fin provides event research and holding-period references only. It is not investment advice, does not
> connect to a broker, and does not place or simulate trades.
## Capabilities
- Download CLS news through Tushare and maintain up to 360 days of traceable local news records.
- Rank ETF candidates by previous-trading-day turnover, then select representative ETFs related to current events.
- Search ReMe memory and local news files for similar historical events, with strict source-path and news-ID checks.
- Calculate adjusted D1D10 historical returns in deterministic code instead of asking an Agent to invent numbers.
- Let an Agent judge event similarity, then calculate weights, expected returns, and a reference holding period in code.
- Save readable Markdown and structured JSON/JSONL artifacts, refresh the daily index, and optionally deliver the report
to DingTalk.
The workflow is assembled by
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml). Its public schemas are in
[`reme/schema/auto_fin.py`](../../reme/schema/auto_fin.py), and its steps are in
[`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/).
## Quick start
Auto Fin requires Python 3.11 or newer, the `core` dependencies, a Tushare token, and credentials for the configured
Claude Code-compatible endpoint.
From the repository root:
```bash
python -m pip install -e ".[core]"
export TUSHARE_TOKEN="your-tushare-token"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=auto_fin
```
The built-in configuration uses `qwen3.7-max` through DashScope's Anthropic-compatible endpoint. Override these
variables to use another compatible model or provider:
```bash
export CLAUDE_CODE_MODEL_NAME="your-model"
export CLAUDE_CODE_BASE_URL="https://your-anthropic-compatible-endpoint"
```
The default workspace is `reme_workspace/`. This standalone cookbook shares its workspace setting with the daily-paper
workflow:
```bash
export DAILY_PAPER_WORKSPACE_DIR="/absolute/path/to/reme-workspace"
```
To deliver the final Markdown report to DingTalk, set:
```bash
export DINGTALK_APP_KEY="your-app-key"
export DINGTALK_APP_SECRET="your-app-secret"
export DINGTALK_ROBOT_CODE="your-robot-code"
export DINGTALK_CONVERSATION_IDS="conversation-id-1,conversation-id-2"
```
DingTalk delivery is skipped when the required values are empty.
Dates and times use `Asia/Shanghai`. The optional `date` must be the current date:
```bash
reme start config=daily_cookbook job=auto_fin date=2026-07-25
```
To refresh every configured news day instead of reusing valid historical files:
```bash
reme start config=daily_cookbook job=auto_fin force=true
```
This may issue many Tushare requests. A normal run reuses valid historical news files and always refreshes today's file.
### Optional SSH proxy
The outbound proxy is disabled by default. To enable it, uncomment `components.outbound_proxy.default` in
`daily_cookbook.yaml`, configure non-interactive SSH authentication, and set:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## How it works
```mermaid
flowchart LR
A[Resolve run date and cutoff] --> B[Maintain CLS news files]
B --> C[Resolve previous A-share trading day]
C --> D[Build current event window]
D --> E[Filter liquid ETF candidates]
E --> F[Agent selects related ETFs]
F --> G{For each ETF}
G --> H[Agent searches historical events]
H --> I[Code resolves original news]
I --> J[Code calculates adjusted D1-D10 returns]
J --> K[Agent judges similarity]
K --> L[Code calculates weighted forecast]
L --> G
G --> M[Agent writes the combined report]
M --> N[Write artifacts and refresh daily index]
N --> O[Optional DingTalk delivery]
```
The top-level job contains four Auto Fin steps:
| Step | Responsibility | Agent |
|-------------------------|-------------------------------------------------------------|-------|
| `auto_fin_data_step` | Maintain news files and resolve the previous trading day | No |
| `auto_fin_topic_step` | Build inputs and select related ETFs and current events | Yes |
| `auto_fin_history_step` | Orchestrate historical research and market analysis per ETF | Yes |
| `auto_fin_merge_step` | Validate results and produce the final Markdown report | Yes |
For each selected ETF, `auto_fin_history_step` dispatches:
- `auto_fin_history_search_step`, which asks the Agent for historical news references and then resolves the original
records and calculates their returns in code.
- `auto_fin_market_step`, which asks the Agent only for similarity judgments and then calculates weights and forecasts
in code.
Agents handle semantic judgments; deterministic code handles source validation and financial calculations.
## Data and time boundaries
### News history
`auto_fin_data_step` reads CLS news from Tushare's `major_news` endpoint:
- The default lookback is 360 calendar days, including the run date.
- A valid historical file is reused unless `force=true`.
- Today's file is refreshed through the current `decision_at` on every run.
- Large responses are fetched through recursively split time windows.
- Records are ordered and deduplicated before being written with a stable `news_id`.
The current event window is:
```text
(previous A-share trading day at 15:00, decision_at]
```
Each run rebuilds this complete window; midday and evening runs do not use only the increment since the previous run.
### ETF candidates
The candidate universe combines:
- `etf_basic` for currently listed ETFs and their tracked indexes.
- `fund_daily` for turnover on the previous A-share trading day.
Code sorts candidates by turnover, removes duplicates by ETF name and index identity, and provides at most 150
candidates to the Topic Agent. The Agent may return at most 20 ETFs and must copy every ETF code, name, and news ID from
the generated candidate files.
Turnover is used only to narrow the research universe; it is not a trading signal.
## Historical research and forecasting
### Source resolution
The History Agent searches by event type, entities, transmission mechanism, and expected direction. It first uses
`memory_search` and may then scan:
```text
daily/YYYY-MM-DD/auto_fin_news_data.jsonl
```
Its output contains only a reason, `news_id`, and workspace-relative `source_path`. Code rejects:
- Current-window news presented as historical evidence.
- Absolute paths, `..` traversal, or paths outside the workspace.
- Sources not named `auto_fin_news_data.jsonl`.
- Missing files or IDs that do not resolve exactly once.
- Records without a usable publication time, title, or body.
Historical Markdown may guide retrieval, but the original news JSONL is the source of truth.
### Adjusted returns
For every resolved historical event, code reads `fund_daily` and `fund_adj` and calculates up to ten future closes:
- Before 09:30 on a trading day: enter at that day's open.
- From 09:30 until before 15:00: enter at that day's close.
- At or after 15:00, or on a non-trading day: enter at the next trading day's open.
- A daily close later than the current `decision_at` is excluded.
```text
adjusted_entry = raw_entry × entry_adjustment_factor
adjusted_close = raw_close × close_adjustment_factor
cumulative_return = adjusted_close / adjusted_entry - 1
```
Missing prices, factors, trading days, or horizons become explicit limitations. They are never filled with Agent-made
values.
### Similarity and forecast
The Market Agent returns semantic similarity in `[-1, 1]`:
- Positive values mean a similar mechanism and direction.
- Negative values mean a comparable mechanism but opposite direction.
- Zero means no useful relationship.
Code clamps out-of-range values, ignores zero-similarity events, normalizes weights from absolute similarity, and
reverses the historical return direction for negative matches. Each D1D10 horizon is calculated from the samples
available at that horizon. The suggested holding period is the positive-return horizon with the highest expected return,
or empty when none is positive.
The result also records limited samples, missing horizons, conflicting return directions, and other data limitations. It
is a comparison with a small historical sample, not evidence of statistical significance.
## Output layout
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── auto_fin_news_data.jsonl
│ ├── auto_fin_analysis.jsonl
│ └── auto_fin.md
└── resource/
└── YYYY-MM-DD/
├── filtered_news.jsonl
├── filtered_etf.jsonl
├── auto_fin_topic_output.jsonl
├── auto_fin_history_<index>_<ETF-code>_output.json
├── auto_fin_market_<index>_<ETF-code>_output.json
├── auto_fin_history_output.jsonl
└── auto_fin_merge_output.json
```
Important artifacts:
- `auto_fin_news_data.jsonl` is the user-owned source used to resolve historical news.
- `filtered_news.jsonl` and `filtered_etf.jsonl` are bounded inputs for the Topic Agent.
- Per-ETF history files contain resolved source news and code-calculated return paths.
- Per-ETF market files contain code-calculated matches, weights, and D1D10 forecasts.
- `auto_fin_analysis.jsonl` contains the final structured analysis for every selected ETF.
- `auto_fin.md` is the readable report and DingTalk payload.
- `daily/YYYY-MM-DD.md` is refreshed after report generation so the report is discoverable from the daily index.
News and reports remain ordinary user-owned files. Resource artifacts and search indexes can be rebuilt.
## Configuration
### Job parameters
| Parameter | Default | Meaning |
|-----------|-------------:|---------------------------------------------------------|
| `date` | Current date | Strict `YYYY-MM-DD`; only the current date is supported |
| `force` | `false` | Refresh all configured news days |
### Environment variables
| Variable | Required | Meaning |
|-----------------------------|----------|---------------------------------------------------------|
| `TUSHARE_TOKEN` | Yes | News, calendar, ETF daily data, and adjustment factors |
| `CLAUDE_CODE_API_KEY` | Yes | Auto Fin Agent credentials |
| `CLAUDE_CODE_MODEL_NAME` | No | Defaults to `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | No | Anthropic-compatible endpoint |
| `AUTO_FIN_AGENT_BACKEND` | No | Defaults to `claude_code` |
| `AUTO_FIN_PROJECT_PATH` | No | Agent project path; defaults to `..` |
| `REME_PROXY_IP` | No | SSH proxy host; used only when `ssh_http` is enabled |
| `REME_PROXY_ACCOUNT` | No | SSH proxy account; used only when `ssh_http` is enabled |
| `DAILY_PAPER_WORKSPACE_DIR` | No | Standalone cookbook workspace |
| `DINGTALK_*` | No | DingTalk application, robot, and conversation settings |
Unit tests can inject `tushare_provider` through the runtime context and do not require real credentials.
### Scheduled jobs
`daily_cookbook.yaml` defines:
| Job | Cron | Asia/Shanghai |
|----------------------|---------------|----------------|
| `auto_fin_0930_cron` | `30 9 * * *` | Daily at 09:30 |
| `auto_fin_1145_cron` | `45 11 * * *` | Daily at 11:45 |
| `auto_fin_1800_cron` | `0 18 * * *` | Daily at 18:00 |
These cron expressions do not exclude weekends or market holidays. The workflow resolves the previous A-share trading
day but does not currently skip a run merely because the run date is not a trading day.
## Agent and security boundaries
The Auto Fin wrapper loads the `tushare-data` skill, exposes the `memory_search` job tool, and defaults to
`bypassPermissions`. Prompts constrain each Agent's role, while code revalidates schemas, ETF identities, source paths,
news references, and calculated values.
The standalone cookbook does not configure an embedding store by default, so `memory_search` normally uses BM25 recall.
Vector and BM25 fusion becomes available only when an embedding store is configured.
`bypassPermissions` is not an operating-system sandbox. Review the configured project path, workspace, credentials, and
network boundary before deployment.
## Reruns and limitations
- Valid historical news files are reused; today's news is always refreshed.
- Outputs use stable per-day paths, so a later same-day run replaces the previous report and resource outputs.
- Auto Fin intentionally has no “report exists, skip” shortcut because its scheduled runs analyze updated news.
- Every successful run attempts DingTalk delivery when configured; notification deduplication is not implemented.
- Missing historical market horizons degrade one sample and are recorded as limitations.
- Invalid dates, missing required services, invalid Agent schemas, unknown ETFs or news IDs, unsafe paths, and
inconsistent cross-step ETF identities fail the job.
The current implementation does not include stocks, US-market correlation, portfolio accounting, BUY/SELL/HOLD actions,
T+1 execution rules, fees, slippage, broker integration, or real/simulated order execution.
## Development
Install development dependencies and run the focused suite:
```bash
python -m pip install -e ".[dev,core]"
PYTHONPATH=. pytest tests/unit/test_auto_fin.py -v
```
The unit suite mocks model and market-data boundaries. Tests requiring real Tushare, model, or DingTalk credentials
should be run separately and only with explicit authorization.

View file

@ -1,304 +0,0 @@
# Auto Fin Cookbook
[English](README.md)
Auto Fin 是一个本地优先、文件原生的 ETF 事件研究工作流。它从财联社新闻中识别市场事件,选择相关且流动性较好的
ETF研究相似历史事件及其后续收益并生成中文研究报告。
> Auto Fin 只提供事件研究和持有时间参考,不构成投资建议。当前实现不连接券商、不提交委托,也不执行模拟交易。
## 能力
- 通过 Tushare 获取财联社新闻,并维护最多 360 天可追溯的本地新闻记录。
- 按上一交易日成交额筛选 ETF 候选,再选择与当前事件直接相关的代表性 ETF。
- 通过 ReMe 记忆检索和本地新闻文件查找相似历史事件,并严格校验来源路径和新闻 ID。
- 由确定性代码计算复权后的 D1D10 历史收益,不让 Agent 编造行情数值。
- 由 Agent 判断事件相似度,再由代码计算权重、预期收益和参考持有时间。
- 保存可读 Markdown 和结构化 JSON/JSONL刷新每日索引并支持可选钉钉投递。
工作流由 [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配,公共 schema 位于
[`reme/schema/auto_fin.py`](../../reme/schema/auto_fin.py),各步骤位于
[`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/)。
## 快速开始
Auto Fin 要求 Python 3.11 或更高版本、`core` 依赖、Tushare token以及所配置 Claude Code 兼容 endpoint 的凭据。
在仓库根目录运行:
```bash
python -m pip install -e ".[core]"
export TUSHARE_TOKEN="your-tushare-token"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=auto_fin
```
内置配置默认通过 DashScope 的 Anthropic 兼容 endpoint 使用 `qwen3.7-max`。如需更换兼容模型或服务商:
```bash
export CLAUDE_CODE_MODEL_NAME="your-model"
export CLAUDE_CODE_BASE_URL="https://your-anthropic-compatible-endpoint"
```
默认 workspace 是 `reme_workspace/`。该 standalone cookbook 与每日论文工作流共用 workspace 配置:
```bash
export DAILY_PAPER_WORKSPACE_DIR="/absolute/path/to/reme-workspace"
```
如需把最终 Markdown 报告发送到钉钉:
```bash
export DINGTALK_APP_KEY="your-app-key"
export DINGTALK_APP_SECRET="your-app-secret"
export DINGTALK_ROBOT_CODE="your-robot-code"
export DINGTALK_CONVERSATION_IDS="conversation-id-1,conversation-id-2"
```
相关配置为空时会跳过钉钉投递。
日期和时间均使用 `Asia/Shanghai`。可选的 `date` 必须是当天:
```bash
reme start config=daily_cookbook job=auto_fin date=2026-07-25
```
如需刷新全部新闻日期,而不是复用有效历史文件:
```bash
reme start config=daily_cookbook job=auto_fin force=true
```
这可能产生大量 Tushare 请求。普通运行会复用有效历史新闻,并始终刷新当天文件。
### 可选 SSH 代理
出站代理默认关闭。如需启用,请取消 `daily_cookbook.yaml`
`components.outbound_proxy.default` 的注释,配置免交互 SSH 认证,并设置:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## 工作原理
```mermaid
flowchart LR
A[确定运行日和截止时间] --> B[维护财联社新闻文件]
B --> C[确定上一 A 股交易日]
C --> D[构建当前事件窗口]
D --> E[筛选高流动性 ETF 候选]
E --> F[Agent 选择相关 ETF]
F --> G{逐只 ETF}
G --> H[Agent 搜索历史事件]
H --> I[代码回查原始新闻]
I --> J[代码计算复权 D1-D10 收益]
J --> K[Agent 判断相似度]
K --> L[代码计算加权预测]
L --> G
G --> M[Agent 编写合并报告]
M --> N[写入产物并刷新每日索引]
N --> O[可选钉钉投递]
```
顶层 Job 包含四个 Auto Fin Step
| Step | 职责 | Agent |
|-------------------------|-----------------------------------|-------|
| `auto_fin_data_step` | 维护新闻文件并确定上一交易日 | 否 |
| `auto_fin_topic_step` | 构建输入并选择相关 ETF 和当前事件 | 是 |
| `auto_fin_history_step` | 逐只 ETF 编排历史研究和行情分析 | 是 |
| `auto_fin_merge_step` | 校验结果并生成最终 Markdown 报告 | 是 |
对于每只已选 ETF`auto_fin_history_step` 会派发:
- `auto_fin_history_search_step`Agent 返回历史新闻引用,然后由代码解析原始记录并计算收益。
- `auto_fin_market_step`Agent 只判断相似度,然后由代码计算权重和预测。
Agent 负责语义判断,确定性代码负责来源校验和金融数值计算。
## 数据和时间边界
### 新闻历史
`auto_fin_data_step` 使用 Tushare `major_news` 接口读取财联社新闻:
- 默认回看包含运行日在内的 360 个自然日。
- 有效历史文件会复用,除非设置 `force=true`
- 每次运行都会把当天文件刷新到当前 `decision_at`
- 数据量接近接口上限时会递归拆分时间窗口。
- 写入前会排序和去重,并生成稳定 `news_id`
当前事件窗口为:
```text
(上一 A 股交易日 15:00, decision_at]
```
每次运行都会重建完整窗口;午间和晚间运行不是只读取上次运行后的增量。
### ETF 候选
候选池组合使用:
- `etf_basic`:当前上市 ETF 及其跟踪指数。
- `fund_daily`:上一 A 股交易日成交额。
代码按成交额排序,并按 ETF 名称和指数标识去重,最多向 Topic Agent 提供 150 个候选。Agent 最多返回 20 只 ETF且 ETF 代码、名称和新闻
ID 都必须逐字来自候选文件。
成交额仅用于缩小研究范围,不是交易信号。
## 历史研究和预测
### 来源回查
History Agent 按事件类型、关键实体、传导机制和预期方向搜索。它优先使用 `memory_search`,必要时扫描:
```text
daily/YYYY-MM-DD/auto_fin_news_data.jsonl
```
Agent 只返回理由、`news_id` 和 workspace 相对 `source_path`。代码会拒绝:
- 把当前事件窗口内的新闻当作历史证据;
- 绝对路径、`..` 路径穿越或 workspace 外路径;
- 文件名不是 `auto_fin_news_data.jsonl` 的来源;
- 不存在的文件或不能唯一解析的 ID
- 缺少有效发布时间、标题或正文的记录。
历史 Markdown 只能作为检索线索,原始新闻 JSONL 才是事实来源。
### 复权收益
对每条已回查的历史事件,代码读取 `fund_daily``fund_adj`,计算最多十个未来收盘点:
- 交易日 09:30 前发生:以当日开盘价为 entry。
- 09:30 至 15:00 前发生:以当日收盘价为 entry。
- 15:00 或之后、以及非交易日发生:以下一交易日开盘价为 entry。
- 晚于当前 `decision_at` 的日线收盘数据不会参与计算。
```text
adjusted_entry = raw_entry × entry_adjustment_factor
adjusted_close = raw_close × close_adjustment_factor
cumulative_return = adjusted_close / adjusted_entry - 1
```
缺少价格、复权因子、交易日或 horizon 时会记录明确限制,不会用 Agent 生成的数值补齐。
### 相似度和预测
Market Agent 返回 `[-1, 1]` 范围内的语义相似度:
- 正值表示机制和方向相似。
- 负值表示机制可比但方向相反。
- `0` 表示没有有效关系。
代码会截断越界值、忽略零相似度事件,并按相似度绝对值归一化权重。负相似度样本会反转历史收益方向。 每个 D1D10 horizon 只使用该
horizon 有数据的样本。参考持有时间取正预期收益中最高的 horizon 没有正值时留空。
结果还会记录样本不足、horizon 缺失、收益方向冲突等限制。它只是有限历史样本比较,不代表统计显著性。
## 输出布局
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── auto_fin_news_data.jsonl
│ ├── auto_fin_analysis.jsonl
│ └── auto_fin.md
└── resource/
└── YYYY-MM-DD/
├── filtered_news.jsonl
├── filtered_etf.jsonl
├── auto_fin_topic_output.jsonl
├── auto_fin_history_<序号>_<ETF代码>_output.json
├── auto_fin_market_<序号>_<ETF代码>_output.json
├── auto_fin_history_output.jsonl
└── auto_fin_merge_output.json
```
主要产物:
- `auto_fin_news_data.jsonl`:用户拥有的历史新闻回查来源。
- `filtered_news.jsonl``filtered_etf.jsonl`:边界明确的 Topic Agent 输入。
- 各 ETF history 文件:已回查的原始新闻和代码计算的收益路径。
- 各 ETF market 文件:代码计算的匹配、权重和 D1D10 预测。
- `auto_fin_analysis.jsonl`:全部已选 ETF 的最终结构化分析。
- `auto_fin.md`:可读报告和钉钉投递内容。
- `daily/YYYY-MM-DD.md`:报告生成后会刷新,确保每日索引能够发现 Auto Fin 报告。
新闻和报告都是用户拥有的普通文件resource 中间产物和搜索索引均可重建。
## 配置
### Job 参数
| 参数 | 默认值 | 说明 |
|---------|--------:|-----------------------------------|
| `date` | 当天 | 严格 `YYYY-MM-DD`;当前只支持当天 |
| `force` | `false` | 是否刷新全部已配置新闻日期 |
### 环境变量
| 变量 | 必需 | 说明 |
|-----------------------------|------|-------------------------------------|
| `TUSHARE_TOKEN` | 是 | 新闻、交易日历、ETF 日线和复权因子 |
| `CLAUDE_CODE_API_KEY` | 是 | Auto Fin Agent 凭据 |
| `CLAUDE_CODE_MODEL_NAME` | 否 | 默认 `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | 否 | Anthropic 兼容 endpoint |
| `AUTO_FIN_AGENT_BACKEND` | 否 | 默认 `claude_code` |
| `AUTO_FIN_PROJECT_PATH` | 否 | Agent project path默认 `..` |
| `REME_PROXY_IP` | 否 | 仅启用 `ssh_http` 时使用的 SSH 主机 |
| `REME_PROXY_ACCOUNT` | 否 | 仅启用 `ssh_http` 时使用的 SSH 账户 |
| `DAILY_PAPER_WORKSPACE_DIR` | 否 | standalone cookbook workspace |
| `DINGTALK_*` | 否 | 钉钉应用、机器人和会话设置 |
单元测试可通过 RuntimeContext 注入 `tushare_provider`,不需要真实凭据。
### 定时任务
`daily_cookbook.yaml` 定义:
| Job | Cron | Asia/Shanghai |
|----------------------|---------------|---------------|
| `auto_fin_0930_cron` | `30 9 * * *` | 每天 09:30 |
| `auto_fin_1145_cron` | `45 11 * * *` | 每天 11:45 |
| `auto_fin_1800_cron` | `0 18 * * *` | 每天 18:00 |
这些 cron 表达式不会排除周末或休市日。工作流会确定上一 A 股交易日,但当前不会仅因为运行日不是交易日而跳过。
## Agent 和安全边界
Auto Fin wrapper 会加载 `tushare-data` skill、暴露 `memory_search` Job 工具,并默认使用
`bypassPermissions`。Prompt 会约束各 Agent 的职责,代码则再次校验 schema、ETF 身份、来源路径、 新闻引用和计算值。
standalone cookbook 默认没有配置 embedding store因此 `memory_search` 通常使用 BM25 召回; 配置 embedding store 后才能使用向量与
BM25 融合。
`bypassPermissions` 不是操作系统沙箱。部署前应检查 project path、workspace、凭据和网络边界。
## 重跑和限制
- 有效历史新闻会复用,当天新闻始终刷新。
- 输出使用稳定的每日路径,因此同一天后一次运行会替换前一次报告和 resource 输出。
- Auto Fin 不使用“报告已存在则跳过”,因为定时运行需要分析更新后的新闻。
- 配置钉钉后,每次成功运行都会尝试投递;当前没有通知去重。
- 历史样本缺少部分 horizon 时会降级该样本并记录限制。
- 非法日期、缺少必要服务、Agent schema 错误、未知 ETF/新闻、危险路径或跨步骤 ETF 身份不一致会使 Job 失败。
当前没有实现个股、美股关联、组合账本、BUY/SELL/HOLD、T+1 执行、手续费、滑点、券商连接, 也不会执行真实或模拟委托。
## 开发
安装开发依赖并运行聚焦测试:
```bash
python -m pip install -e ".[dev,core]"
PYTHONPATH=. pytest tests/unit/test_auto_fin.py -v
```
单元测试会 mock 模型和行情数据边界。需要真实 Tushare、模型或钉钉凭据的测试应单独运行且需要显式授权。

View file

@ -1,376 +0,0 @@
# Daily Paper Cookbook
[中文](README_ZH.md)
Daily Paper is a local-first, file-native workflow for turning research rankings into a daily reading package.
## Capabilities
- Collect papers from the Hugging Face weekly and monthly rankings while excluding yesterday's papers and recent
recommendations.
- Rank and select candidates, then use Claude Code to produce detailed Chinese notes and a five-minute Chinese brief.
- Keep PDFs, notes, and memories as ordinary user-owned files; indexes and caches remain rebuildable.
- Support daily scheduling, optional DingTalk delivery, conversation memory, auto-dream consolidation, and BM25 recall
for the background DingTalk agent.
The workflow is assembled by [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml). Its schemas live in
[`reme/schema/daily_paper.py`](../../reme/schema/daily_paper.py), and its steps live in
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/).
## Quick start
Daily Paper requires Python 3.11 or later, the `core` dependencies, network access to Hugging Face and arXiv, and
credentials for the configured Claude Code endpoint. Auto-memory and auto-dream additionally require the AgentScope LLM
credentials.
From the repository root:
```bash
python -m pip install -e ".[core]"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=daily_paper
```
The built-in configuration uses `qwen3.7-max` through DashScope's Anthropic-compatible endpoint. Override
`CLAUDE_CODE_MODEL_NAME` and `CLAUDE_CODE_BASE_URL` when using another compatible model or provider.
This is enough to generate paper notes and the daily brief. To use auto-memory and auto-dream, also configure:
```bash
export LLM_API_KEY="your-api-key"
```
By default, outputs are written under `reme_workspace/` in the directory where ReMe starts.
### Optional SSH proxy
The outbound proxy is disabled by default. To enable it, uncomment `components.outbound_proxy.default` in
`daily_cookbook.yaml`, configure non-interactive SSH authentication, and set:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## What it creates
A successful run writes ordinary PDFs and Markdown files beneath `workspace_dir`:
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── daily-paper-brief.md
│ ├── paper-<arxiv-id>.md
│ └── ...
├── resource/
│ └── papers/
│ ├── <arxiv-id>.pdf
│ └── ...
├── digest/
│ ├── personal/
│ ├── project/
│ ├── resource/
│ └── wiki/
├── metadata/
│ └── ... derived catalogs, indexes, and caches
└── mem_session/
├── agentscope/
└── claude_config/
```
- `paper-<arxiv-id>.md` is a detailed Chinese reading note with YAML frontmatter linking back to the source PDF and
paper pages.
- `daily-paper-brief.md` is a roughly five-minute Chinese digest with wikilinks to every selected paper note.
- `daily/YYYY-MM-DD.md` is a derived day index rebuilt from the Markdown files for that date.
- `resource/papers/` holds reusable source PDFs.
- `digest/` contains durable auto-dream output; files there remain ordinary user-owned Markdown.
- `metadata/` and search caches are derived state. `reindex` rebuilds the file store, BM25 index, and graph from source
files.
The paper notes are the source of truth for recommendation history: their frontmatter contains the `arxiv_id` values
used for future deduplication. The day index is derived and can be rebuilt. The workflow does not currently write a
separate run manifest.
## How the workflow works
```mermaid
flowchart LR
HF[Hugging Face<br/>weekly + monthly] --> C[1. Collect]
Y[Yesterday's papers] --> C
H[Recent local notes] --> C
C --> R[2. Rank]
R --> S[3. Select]
S --> A[4. Analyze PDFs]
A --> D[5. Build brief]
D --> N[6. Notify DingTalk]
A --> P[PDFs + paper notes]
D --> B[Brief + day index]
```
### 1. Collect and deduplicate
The Collect step fetches the weekly ranking for the run date's ISO week, the monthly ranking for its calendar month, and
the Hugging Face Daily Papers IDs for exactly the previous calendar day. It merges weekly and monthly metadata by arXiv
ID and preserves each list's display rank.
It then scans `daily/<prior-date>/paper-*.md` over the configured history window and excludes IDs found in note
frontmatter. The job fails clearly if no eligible papers remain.
### 2. Rank candidates
The Rank step uses reciprocal-rank fusion:
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
A missing rank contributes zero. Candidates are ordered by fused score, upvotes, and arXiv ID. The bounded candidate
pool also reserves several positions for papers whose titles or summaries match memory-related terms such as agent
memory, memory retrieval, continual learning, context compression, knowledge graphs, and RAG. This reserve is a simple
keyword heuristic, not a semantic classifier.
### 3. Select papers
Claude Code receives the bounded candidate pool and returns a structured `PaperSelection`. The implementation requires
exactly `top_k` unique in-pool IDs with consecutive ranks. Invalid output is returned to the agent once as validation
feedback; a second invalid response fails the job.
### 4. Download and analyze PDFs
Selected papers are processed sequentially. For each paper, the workflow:
1. validates the modern arXiv ID format;
2. downloads and validates the PDF, or reuses an existing file with a valid `%PDF-` header;
3. extracts text with `pypdf`, adding page markers and applying page and character limits;
4. asks Claude Code for a structured detailed reading; and
5. writes normalized frontmatter plus the generated Markdown body.
The current extractor requires a usable PDF text layer. Scanned or image-only PDFs fail because there is no OCR
fallback. If extraction exceeds a configured limit, the note records that the input was truncated.
### 5. Build the brief and index
Claude Code reads every detailed note and produces the daily brief. The code verifies that each source-note wikilink is
present and appends any missing links before writing the file. It then rebuilds `daily/YYYY-MM-DD.md` from that day's
Markdown frontmatter.
### 6. Optionally notify DingTalk
The final step sends the brief body, without YAML frontmatter, to each configured DingTalk group in order. With no
conversation IDs it is a no-op. If one group fails, the step still attempts the remaining groups and reports the
combined failure afterward.
## Memory and search
The standalone configuration separates agent wrappers by responsibility:
- `daily_paper` selects papers, analyzes them, and builds the brief. It keeps Claude Code's normal local tools and
disables `WebSearch`, but currently has no memory-retrieval job configured.
- `dingtalk_wait` runs the background DingTalk agent and exposes `memory_search` as a callable tool.
- `memory` runs auto-memory and the LLM-backed auto-dream steps through AgentScope. Its built-in shell and file tools
are disabled; memory changes go through the narrower ReMe jobs such as `daily_write`, `read`, `edit`, and `write`.
The built-in `memory_search` job uses BM25 over Markdown under `daily/` and `digest/`. ReMe's search step can fuse
vector results, but this cookbook does not configure an embedding store by default, so vector retrieval is not run.
`node_search` is a narrower digest recall tool used internally by auto-dream.
`index_update_loop` indexes existing memory files when the service starts and watches those directories for later
changes. Run `reindex` when recovering the derived file store or forcing a complete index rebuild. Source Markdown and
PDFs are not deleted by `reindex`.
`auto_memory` writes or updates one daily note from caller-supplied conversation messages and a stable `session_id`.
`auto_dream` scans recent daily notes, integrates durable units under `digest/`, and writes interest topics. Both are
on-demand jobs in this cookbook; no auto-dream cron is configured. The DingTalk agent can recall through
`memory_search`, but it does not automatically call `auto_memory` after a conversation.
## Dates, reruns, and idempotency
- `date` must be an exact `YYYY-MM-DD` value. When omitted, the job uses today in the application timezone, which is
`Asia/Shanghai` in the built-in configuration.
- “Yesterday” means `date - 1 day`, not the previous 24 hours.
- `history_days` considers prior dated note directories only; the current run date is never part of its history scan.
- If `daily/<date>/daily-paper-brief.md` already exists and `force=false`, collection, ranking, model calls, PDF work,
and digest generation are skipped. The existing brief remains available to the DingTalk notification step.
- `force=true` regenerates the notes and brief. Existing valid PDFs are still reused.
Each PDF, detailed note, and final brief uses a temporary file followed by replacement so callers do not see a partially
written file. The complete multi-file workflow is not transactional, and there is no global lock for two concurrent runs
of the same date.
## Running the cookbook
The main jobs in the standalone configuration are:
| Job | Behavior |
|---------------------|-----------------------------------------------------------------|
| `daily_paper` | On-demand generation through the CLI or HTTP service |
| `daily_paper_cron` | The same pipeline every day at 08:00 in `Asia/Shanghai` |
| `dingtalk_wait` | A supervised background DingTalk agent with `memory_search` |
| `auto_memory` | Write or update a daily note from conversation messages |
| `auto_dream` | Consolidate recent daily notes into digest memory and interests |
| `memory_search` | BM25 recall over daily and digest Markdown |
| `reindex` | Rebuild derived search state from existing memory files |
| `index_update_loop` | Initialize and continuously update search state in service mode |
Supporting jobs such as `node_search`, `daily_list`, `daily_write`, `read`, `write`, `edit`, and frontmatter updates
provide the constrained tools used by the memory agent.
### One-time runs
The quick-start command generates today's brief. To generate a specific date with selected overrides:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-07-21 \
top_k=3 \
history_days=30
```
Regenerate a date whose brief already exists:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-07-21 force=true
```
Add `service.show_metadata=true` to a one-time command when the response metadata is useful for diagnostics.
### Long-running service and cron
Start the standalone HTTP service and its scheduled/background jobs:
```bash
reme start config=daily_cookbook
```
It listens on `127.0.0.1:8001` by default, so it can run beside the default ReMe service. Call the on-demand job from
another terminal with either the ReMe client or HTTP:
```bash
reme daily_paper host=127.0.0.1 port=8001
```
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-07-21","top_k":3,"force":false}'
```
Recall memory, record a conversation, consolidate it, or explicitly rebuild the search index:
```bash
reme memory_search host=127.0.0.1 port=8001 query="agent memory" limit=5
reme auto_memory host=127.0.0.1 port=8001 \
session_id=example-session \
messages='[{"name":"user","role":"user","content":"I prefer concise paper summaries."}]'
reme auto_dream host=127.0.0.1 port=8001 date=2026-07-21
reme reindex host=127.0.0.1 port=8001
```
Service and schedule settings can be overridden at startup:
```bash
reme start \
config=daily_cookbook \
service.host=0.0.0.0 \
service.port=8101 \
jobs.daily_paper_cron.cron="30 7 * * *"
```
## Configuration
The most useful job settings are:
| Setting | Default | Purpose |
|-------------------|-------------:|--------------------------------------------------------------|
| `candidate_limit` | `20` | Maximum number of papers sent to selection |
| `memory_reserve` | `5` | Candidate positions reserved by the memory-keyword heuristic |
| `top_k` | `3` | Number of papers selected and analyzed |
| `rrf_k` | `60` | Reciprocal-rank fusion constant |
| `weekly_weight` | `0.7` | Weight of the weekly ranking in fusion |
| `history_days` | `30` | Prior recommendation window excluded by arXiv ID |
| `hf_timeout` | `30` seconds | Hugging Face request timeout |
| `hf_max_retries` | `3` | Maximum Hugging Face request attempts |
| `pdf_timeout` | `90` seconds | arXiv download timeout |
| `max_pdf_bytes` | `52428800` | Maximum PDF size (50 MiB) |
| `max_pdf_pages` | `80` | Maximum pages extracted for analysis |
| `max_pdf_chars` | `240000` | Maximum extracted characters sent for one paper |
The public job parameters are `date`, `force`, `top_k`, `weekly_weight`, and `history_days`. Explicit invocation values
take precedence over the job defaults.
The standalone application also accepts these environment variables:
| Variable | Purpose |
|-----------------------------------------|------------------------------------------------|
| `DAILY_PAPER_WORKSPACE_DIR` | Overrides the default `reme_workspace` |
| `DAILY_PAPER_PROJECT_PATH` | Repository/project path visible to Claude Code |
| `REME_PROXY_IP` | Optional SSH proxy host |
| `REME_PROXY_ACCOUNT` | Optional SSH proxy account |
| `DAILY_PAPER_HOST` / `DAILY_PAPER_PORT` | HTTP bind address |
| `CLAUDE_CODE_API_KEY` | API key for the Claude Code endpoint |
| `CLAUDE_CODE_MODEL_NAME` | Claude Code model; default `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | Claude Code Anthropic-compatible endpoint |
| `LLM_API_KEY` | API key for the AgentScope memory model |
| `LLM_MODEL_NAME` | Memory model; default `qwen3.7-max` |
| `LLM_BASE_URL` | Memory model's Anthropic-compatible endpoint |
`DAILY_PAPER_PROJECT_PATH` defaults to `..` relative to the workspace. With the default `reme_workspace`, starting from
the repository root resolves it back to the repository. If the workspace lives elsewhere, set both paths explicitly.
ReMe loads an uncommitted `.env` file found from the current directory upward, so the same values may be placed there
instead of exported in the shell.
## DingTalk configuration
DingTalk is optional. Configure it only when brief delivery or the background DingTalk agent is needed:
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
`DINGTALK_CONVERSATION_IDS` is required only for proactive brief delivery. The background `dingtalk_wait` job uses the
first three credentials but not the conversation list.
## Failure recovery and boundaries
| Situation | Behavior |
|--------------------------------------|----------------------------------------------------------------|
| Temporary Hugging Face failure | Retries with exponential delay up to `hf_max_retries` attempts |
| No eligible papers | Fails before ranking |
| Invalid `top_k` or selection output | Fails after validation; selection output gets one retry |
| Oversized, invalid, or textless PDF | Stops during analysis |
| PDF exceeds page or character limits | Continues with truncated text and records the truncation |
| One paper analysis fails | Stops the job; earlier PDFs and notes remain on disk |
| Brief misses a source-note link | Appends the missing wikilink before writing |
| Auto-dream partial integration | Successful units remain; failed paths are not checkpointed |
To recover, inspect the date's notes and PDFs, fix the network, credential, model, or PDF issue, then rerun the same
date with `force=true`. Valid cached PDFs will be reused.
The built-in Claude Code components run with `permission_mode: bypassPermissions` and disable `WebSearch`.
`dingtalk_wait` can call the local `memory_search` job; `daily_paper` currently has no job tools. The analysis and brief
prompts constrain what the agent should read, but these steps do not set a strict per-call tool allowlist or an
operating-system sandbox. The AgentScope memory wrapper disables its built-in shell and filesystem tools, but runs its
ReMe job tools in bypass permission mode. Run the cookbook only with a trusted project and workspace, and tighten the
agent configuration before shared or production use.
## Tests
The focused unit suite mocks Hugging Face, arXiv, Claude Code, and DingTalk boundaries:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```
Real runs access external services and may incur model costs; they should not be used as ordinary unit tests.

View file

@ -1,351 +0,0 @@
# 每日论文 Cookbook
[English](README.md)
每日论文是一个本地优先、文件原生的研究资讯工作流,用于把研究榜单转化为每日阅读材料。
## 能力
- 从 Hugging Face 周榜和月榜采集论文,并排除昨日论文和近期已经推荐过的论文。
- 对候选论文进行排序和精选,再使用 Claude Code 生成中文详细论文笔记和约五分钟可读完的中文简报。
- 将 PDF、笔记和记忆保存为由用户拥有的普通文件索引和缓存均可重建。
- 支持每日定时运行、可选钉钉投递、对话记忆、auto-dream 整理,以及供后台钉钉 Agent 使用的 BM25 检索。
工作流由 [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配,公共 schema 位于
[`reme/schema/daily_paper.py`](../../reme/schema/daily_paper.py),各步骤位于
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/)。
## 快速开始
每日论文要求 Python 3.11 或更高版本、`core` 依赖、可访问 Hugging Face 和 arXiv 的网络,以及所配置 Claude Code endpoint
的凭据。auto-memory 和 auto-dream 还需要 AgentScope LLM 凭据。
在仓库根目录运行:
```bash
python -m pip install -e ".[core]"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=daily_paper
```
内置配置默认通过 DashScope 的 Anthropic 兼容 endpoint 使用 `qwen3.7-max`。如需使用其他兼容模型或服务商, 请覆盖
`CLAUDE_CODE_MODEL_NAME``CLAUDE_CODE_BASE_URL`
以上配置足以生成论文笔记和每日简报。要使用 auto-memory 和 auto-dream还需配置
```bash
export LLM_API_KEY="your-api-key"
```
默认情况下,产物写入 ReMe 启动目录下的 `reme_workspace/`
### 可选 SSH 代理
出站代理默认关闭。如需启用,请取消 `daily_cookbook.yaml`
`components.outbound_proxy.default` 的注释,配置免交互 SSH 认证,并设置:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## 文件产物
一次成功运行会在 `workspace_dir` 下写入普通 PDF 和 Markdown 文件:
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── daily-paper-brief.md
│ ├── paper-<arxiv-id>.md
│ └── ...
├── resource/
│ └── papers/
│ ├── <arxiv-id>.pdf
│ └── ...
├── digest/
│ ├── personal/
│ ├── project/
│ ├── resource/
│ └── wiki/
├── metadata/
│ └── ... 派生 catalog、索引和缓存
└── mem_session/
├── agentscope/
└── claude_config/
```
- `paper-<arxiv-id>.md` 是中文详细论文解读,其 YAML frontmatter 会链接原始 PDF 和论文页面。
- `daily-paper-brief.md` 是约五分钟可读完的中文简报,并包含每篇入选论文笔记的 wikilink。
- `daily/YYYY-MM-DD.md` 是从当日 Markdown 文件重建的派生日索引。
- `resource/papers/` 保存可复用的原始 PDF。
- `digest/` 保存持久的 auto-dream 产物,其中仍然是由用户拥有的普通 Markdown 文件。
- `metadata/` 和搜索缓存属于派生状态;`reindex` 会根据源文件重建 file store、BM25 索引和图。
论文笔记是推荐历史的事实来源:后续排重会读取其 frontmatter 中的 `arxiv_id`。日索引属于可重建的派生文件。 当前工作流不会另外写入运行
manifest。
## 工作流程
```mermaid
flowchart LR
HF[Hugging Face<br/>周榜 + 月榜] --> C[1. Collect]
Y[昨日论文] --> C
H[近期本地笔记] --> C
C --> R[2. Rank]
R --> S[3. Select]
S --> A[4. Analyze PDFs]
A --> D[5. Build brief]
D --> N[6. Notify DingTalk]
A --> P[PDF + 论文笔记]
D --> B[简报 + 日索引]
```
### 1. 采集与排重
Collect 会获取运行日所在 ISO week 的周榜、所在自然月的月榜,以及严格前一个自然日的 Hugging Face Daily Papers ID。周榜和月榜元数据按
arXiv ID 合并,同时保留两个榜单各自的展示排名。
随后,它会在配置的历史窗口内扫描 `daily/<prior-date>/paper-*.md`,排除笔记 frontmatter 中已有的 ID。如果排重后 没有任何可选论文Job
会明确失败。
### 2. 候选排序
Rank 使用 reciprocal-rank fusionRRF
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
论文缺少某个榜单排名时该项贡献为零。候选按融合分、upvotes 和 arXiv ID 排序。有界候选池还会为标题或摘要命中 Agent
memory、memory retrieval、continual learning、context compression、knowledge graph、RAG 等记忆相关关键词的
论文保留若干位置。这个保留策略只是关键词启发式,不是语义分类器。
### 3. 精选论文
Claude Code 接收有界候选池,并返回结构化的 `PaperSelection`。实现要求恰好选择 `top_k` 个候选池内的唯一 ID 且 rank
必须连续。输出不合法时,校验错误会反馈给 Agent 并重试一次;第二次仍不合法则 Job 失败。
### 4. 下载并解读 PDF
入选论文按顺序逐篇处理。每篇论文都会经过:
1. 校验当前支持的新版 arXiv ID 格式;
2. 下载并校验 PDF或复用文件头为 `%PDF-` 的已有文件;
3. 使用 `pypdf` 提取文本、插入页码标记,并应用页数和字符数限制;
4. 请求 Claude Code 返回结构化的详细解读;
5. 写入规范化 frontmatter 和生成的 Markdown 正文。
当前提取器依赖可用的 PDF 文本层。扫描版或纯图片 PDF 会失败,因为没有 OCR fallback。提取内容超过配置限制时 笔记会记录输入已被截断。
### 5. 生成简报与索引
Claude Code 会读取全部详细笔记并生成当日简报。代码会检查每篇源笔记的 wikilink如有遗漏会在写入前自动补齐。 随后,工作流根据当日
Markdown frontmatter 重建 `daily/YYYY-MM-DD.md`
### 6. 可选的钉钉通知
最后一步会去掉 YAML frontmatter把简报正文按顺序发送到每个已配置的钉钉群。未配置群会话 ID 时,该步骤无副作用
跳过。某个群发送失败不会阻止继续尝试其他群,所有发送完成后再汇总报告失败。
## 记忆与检索
独立配置按职责分离 agent wrapper
- `daily_paper` 执行论文精选、解读和简报生成。它保留 Claude Code 的常规本地工具并禁用 `WebSearch`,但当前没有 配置记忆检索
Job。
- `dingtalk_wait` 运行后台钉钉 Agent并把 `memory_search` 作为可调用工具。
- `memory` 通过 AgentScope 执行 auto-memory 和 auto-dream 中依赖 LLM 的步骤。它禁用内置 shell 和文件工具; 记忆变更只能经过
`daily_write``read``edit``write` 等更窄的 ReMe job。
内置配置的 `memory_search` 使用 BM25 检索 `daily/``digest/` 下的 Markdown。ReMe 的搜索步骤支持融合向量结果, 但本
cookbook 默认没有配置 embedding store因此不会执行向量检索。`node_search` 是 auto-dream 内部使用的 digest 节点检索工具。
`index_update_loop` 会在服务启动时索引已有记忆文件,并持续监听这些目录的后续变化。修复派生 file store 或需要强制
完整重建索引时,可运行 `reindex``reindex` 不会删除源 Markdown 或 PDF。
`auto_memory` 根据调用方传入的对话消息和稳定 `session_id` 写入或更新一篇 daily note。`auto_dream` 扫描近期 daily
notes把持久记忆单元整合到 `digest/`,并生成兴趣主题。这两个 job 在本 cookbook 中都是按需执行;当前没有配置 auto-dream
cron。钉钉 Agent 可以通过 `memory_search` 召回记忆,但对话结束后不会自动调用 `auto_memory`
## 日期、重跑与幂等
- `date` 必须严格符合 `YYYY-MM-DD`。省略时使用应用配置时区中的今天;内置配置为 `Asia/Shanghai`
- “昨日”表示 `date - 1 day`,不是模糊的最近 24 小时。
- `history_days` 只扫描此前的日期目录,不会把本次运行日纳入历史窗口。
- 如果 `daily/<date>/daily-paper-brief.md` 已存在且 `force=false`采集、排序、模型调用、PDF 处理和简报生成都会
跳过;已有简报仍会交给钉钉通知步骤。
- `force=true` 会重新生成笔记和简报,但仍会复用已有且有效的 PDF。
每个 PDF、详细笔记和最终简报都会先写临时文件再替换避免读取方看到半写状态。整个多文件工作流不是事务 同一日期的并发运行也没有全局锁。
## 运行方式
独立配置中的主要 Job 如下:
| Job | 行为 |
|---------------------|---------------------------------------------------------------|
| `daily_paper` | 通过 CLI 或 HTTP 服务按需生成 |
| `daily_paper_cron` | 每天 08:00`Asia/Shanghai`)执行相同 pipeline |
| `dingtalk_wait` | 由 supervisor 管理、具有 `memory_search` 能力的后台钉钉 Agent |
| `auto_memory` | 根据对话消息写入或更新 daily note |
| `auto_dream` | 把近期 daily notes 整理为 digest 记忆和兴趣主题 |
| `memory_search` | 对 daily 和 digest Markdown 执行 BM25 检索 |
| `reindex` | 根据已有记忆文件重建派生检索状态 |
| `index_update_loop` | 在服务模式下初始化并持续更新检索状态 |
`node_search``daily_list``daily_write``read``write``edit` 和 frontmatter 更新等辅助 job 构成 memory agent 使用的受约束工具。
### 一次性运行
快速开始中的命令会生成今天的简报。要生成指定日期并覆盖部分参数:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-07-21 \
top_k=3 \
history_days=30
```
重新生成已有简报的日期:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-07-21 force=true
```
需要查看响应 metadata 进行诊断时,可在一次性命令中加入 `service.show_metadata=true`
### 常驻服务与 cron
启动独立 HTTP 服务以及定时、后台 Job
```bash
reme start config=daily_cookbook
```
服务默认监听 `127.0.0.1:8001`,因此可以和默认 ReMe 服务并行运行。在另一个终端中通过 ReMe client 或 HTTP 调用按需任务:
```bash
reme daily_paper host=127.0.0.1 port=8001
```
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-07-21","top_k":3,"force":false}'
```
检索记忆、记录对话、整理记忆,或显式重建检索索引:
```bash
reme memory_search host=127.0.0.1 port=8001 query="agent memory" limit=5
reme auto_memory host=127.0.0.1 port=8001 \
session_id=example-session \
messages='[{"name":"user","role":"user","content":"I prefer concise paper summaries."}]'
reme auto_dream host=127.0.0.1 port=8001 date=2026-07-21
reme reindex host=127.0.0.1 port=8001
```
监听地址和调度时间可以在启动时覆盖:
```bash
reme start \
config=daily_cookbook \
service.host=0.0.0.0 \
service.port=8101 \
jobs.daily_paper_cron.cron="30 7 * * *"
```
## 配置
最常用的 Job 配置如下:
| 配置项 | 默认值 | 用途 |
|-------------------|-----------:|------------------------------------|
| `candidate_limit` | `20` | 送入精选阶段的最大论文数 |
| `memory_reserve` | `5` | 记忆关键词启发式保留的候选位置数 |
| `top_k` | `3` | 最终精选和解读的论文数 |
| `rrf_k` | `60` | RRF 常数 |
| `weekly_weight` | `0.7` | 周榜在融合排序中的权重 |
| `history_days` | `30` | 按 arXiv ID 排除近期推荐的时间窗口 |
| `hf_timeout` | `30` 秒 | Hugging Face 请求 timeout |
| `hf_max_retries` | `3` | Hugging Face 请求最多尝试次数 |
| `pdf_timeout` | `90` 秒 | arXiv 下载 timeout |
| `max_pdf_bytes` | `52428800` | PDF 大小上限50 MiB |
| `max_pdf_pages` | `80` | 最多提取的 PDF 页数 |
| `max_pdf_chars` | `240000` | 单篇论文送入模型的最大提取字符数 |
公开 Job 参数为 `date``force``top_k``weekly_weight``history_days`。调用时显式传入的值优先于 Job 默认值。
独立应用还支持以下环境变量:
| 环境变量 | 用途 |
|-----------------------------------------|----------------------------------------|
| `DAILY_PAPER_WORKSPACE_DIR` | 覆盖默认 `reme_workspace` |
| `DAILY_PAPER_PROJECT_PATH` | Claude Code 可见的仓库或项目路径 |
| `REME_PROXY_IP` | 可选 SSH 代理主机 |
| `REME_PROXY_ACCOUNT` | 可选 SSH 代理账户 |
| `DAILY_PAPER_HOST` / `DAILY_PAPER_PORT` | HTTP 监听地址 |
| `CLAUDE_CODE_API_KEY` | Claude Code endpoint 的 API key |
| `CLAUDE_CODE_MODEL_NAME` | Claude Code 模型;默认 `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | Claude Code 的 Anthropic 兼容 endpoint |
| `LLM_API_KEY` | AgentScope memory 模型的 API key |
| `LLM_MODEL_NAME` | memory 模型;默认 `qwen3.7-max` |
| `LLM_BASE_URL` | memory 模型的 Anthropic 兼容 endpoint |
`DAILY_PAPER_PROJECT_PATH` 默认是相对于 workspace 的 `..`。使用默认 `reme_workspace` 并从仓库根目录启动时, 它会解析回仓库根目录。如果
workspace 位于其他位置,请显式设置这两个路径。
ReMe 会从当前目录向上查找未提交的 `.env`,因此也可以把相同变量放在其中,而不是在 shell 中导出。
## 钉钉配置
钉钉是可选能力。仅在需要投递简报或运行后台钉钉 Agent 时配置:
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
只有主动投递简报需要 `DINGTALK_CONVERSATION_IDS`。后台 `dingtalk_wait` Job 使用前三项凭据,不使用群会话列表。
## 故障恢复与边界
| 场景 | 当前行为 |
|----------------------------|----------------------------------------------|
| Hugging Face 暂时失败 | 按指数间隔重试,最多尝试 `hf_max_retries` 次 |
| 没有 eligible 论文 | 在排序前失败 |
| `top_k` 或精选结果不合法 | 校验后失败;精选结果可重试一次 |
| PDF 太大、无效或没有文本层 | 在解读阶段停止 |
| PDF 超过页数或字符数限制 | 使用截断文本继续,并记录截断状态 |
| 某篇论文解读失败 | Job 停止;此前写入的 PDF 和笔记保留 |
| 简报遗漏源笔记链接 | 写入前自动补齐 wikilink |
| auto-dream 部分整合失败 | 成功单元保留,失败路径不会被 checkpoint |
恢复时,先检查该日期已有的笔记和 PDF修复网络、凭据、模型或 PDF 问题,再使用相同日期和 `force=true` 重跑。 有效的缓存 PDF
会被复用。
内置 Claude Code 组件使用 `permission_mode: bypassPermissions`,并禁用 `WebSearch``dingtalk_wait` 可以调用本地
`memory_search``daily_paper` 当前没有配置 Job 工具。Analyze 和 Brief prompt 会限制 Agent 应读取的内容,但这些
步骤没有设置严格的逐次调用工具 allowlist也不是操作系统级沙箱。AgentScope memory wrapper 禁用了内置 shell 和 文件系统工具,但其
ReMe job tools 运行于 bypass permission mode。请只在可信的项目和 workspace 中运行;用于共享 或生产环境前,应进一步收紧配置。
## 测试
聚焦的单元测试会 mock Hugging Face、arXiv、Claude Code 和钉钉边界:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```
真实运行会访问外部服务并可能产生模型费用,不应把它当作普通单元测试执行。

View file

@ -1,16 +1,17 @@
# Auto Dream
`auto_dream` is ReMe's long-term memory distillation flow from daily to digest. It scans daily inputs for a specified date,
processes only files that changed since the previous dream, extracts content worth retaining as memory units, integrates those
units into `digest/`, and generates the day's `interests.yaml` for proactive use.
`auto_dream` is ReMe's long-term memory distillation flow from daily to digest. By default it scans the target date and
the previous day, processes only files changed since the previous dream, extracts a small set of high-value memory units
across that window, integrates them into `digest/`, and writes the target day's `interests.yaml` for proactive use.
<p align="center">
<img src="../figure/auto-dream-and-proactive.svg" alt="ReMe Auto Dream and Proactive flow from daily to digest to proactive" width="92%">
</p>
Its daily inputs usually come from [Auto Memory](./auto_memory.md) and [Auto Resource](./auto_resource.md). For the file
semantics of `digest/`, Sources sections, and wikilinks, see [Memory as File](./memory_as_file.md). For the linking strategy
used during Integrate, see [Auto Link](./auto_link.md). To read `interests.yaml`, use [Proactive](./proactive.md).
semantics of `digest/`, Sources sections, and wikilinks, see [Memory as File](./memory_as_file.md). For the linking
strategy used during Integrate, see [Auto Link](./auto_link.md). To read `interests.yaml`,
use [Proactive](./proactive.md).
## Configuration
@ -26,6 +27,12 @@ auto_dream:
hint:
type: string
default: ""
scan_days:
type: integer
default: 2
max_units:
type: integer
default: 5
topic_count:
type: integer
default: 3
@ -36,6 +43,8 @@ auto_dream:
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
scan_days: 2
max_units: 5
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
@ -46,34 +55,39 @@ auto_dream:
Parameters:
| Parameter | Purpose |
|---|---|
| `date` | Date to process in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `hint` | Additional guidance from the caller for the Extract and Integrate stages. |
| `topic_count` | Maximum number of topics written to `interests.yaml`. Defaults to 3. |
| Parameter | Purpose |
|------------------------|---------------------------------------------------------------------------------------------------------|
| `date` | Date to process in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `hint` | Additional guidance from the caller for the Extract and Integrate stages. |
| `scan_days` | Recent-date window ending at `date`; defaults to 2 and has a minimum of 1. |
| `max_units` | Maximum reusable units extracted in one run; defaults to 5. |
| `topic_count` | Maximum number of topics written to `interests.yaml`. Defaults to 3. |
| `topic_diversity_days` | Number of past days of `interests.yaml` files considered when avoiding duplicate topics. Defaults to 7. |
## Inputs and Outputs
Inputs are daily Markdown files for the specified date:
Inputs are daily Markdown files from the most recent `scan_days` ending at the specified date. For example,
`date=2026-06-20` with `scan_days=2` scans:
```text
daily/<date>.md
daily/<date>/**/*.md
daily/2026-06-19.md
daily/2026-06-19/**/*.md
daily/2026-06-20.md
daily/2026-06-20/**/*.md
```
`daily/<date>/interests.yaml` is excluded from extraction input so topics from the previous run do not feed back into the
next extraction.
Every `daily/<date>/interests.yaml` in the scan window is excluded from extraction so previous proactive output cannot
feed back into the next run. Final topics are written only for the target date.
The main outputs are:
| Output | Description |
|---|---|
| `digest/procedure/*.md` | Methods, workflows, runbooks, and executable experience. |
| `digest/personal/*.md` | User-, team-, and project-related preferences, facts, and long-term context. |
| `digest/wiki/*.md` | General knowledge, concepts, observations, and decision precedents. |
| `daily/<date>/interests.yaml` | Topics worth proactive attention from the host agent that day. |
| `metadata/file_catalog/dream*` | Dream-specific catalog used to detect changes in daily inputs. |
| Output | Description |
|--------------------------------|------------------------------------------------------------------------------|
| `digest/procedure/*.md` | Methods, workflows, runbooks, and executable experience. |
| `digest/personal/*.md` | User-, team-, and project-related preferences, facts, and long-term context. |
| `digest/wiki/*.md` | General knowledge, concepts, observations, and decision precedents. |
| `daily/<date>/interests.yaml` | Topics worth proactive attention from the host agent that day. |
| `metadata/file_catalog/dream*` | Dream-specific catalog used to detect changes in daily inputs. |
## Four Stages
@ -81,43 +95,50 @@ The main outputs are:
`dream_extract_step` performs three tasks:
1. Refresh the day's index page at `daily/<date>.md`.
2. Scan `daily/<date>.md` and `daily/<date>/**/*.md` and compare their mtimes with `file_catalog: dream`.
3. Send only changed files to the LLM and globally extract two structured result types: `units` and `topics`.
1. Refresh each `daily/<date>.md` in the scan window.
2. Scan those day indexes and `daily/<date>/**/*.md`, comparing mtimes with `file_catalog: dream`.
3. Send all changed files together to the LLM and globally extract two structured result types: `units` and `topics`.
`units` are long-term memory units ready to be distilled into digest. Each has `name`, `bucket`, `summary`, and `paths`.
`bucket` may only be `procedure`, `personal`, or `wiki`; unknown values are routed to `wiki`.
A run returns at most `max_units`; extraction merges cross-file evidence for the same abstraction and drops passing
mentions, per-file summaries, and weak candidates without reusable value. `bucket` may only be `procedure`, `personal`,
or `wiki`; unknown values are routed to `wiki`.
`topics` are proactive-interest candidates for the day. They contain `title`, `reason`, `evidence`, `keywords`, and
`paths` and are filtered again in the Topics stage.
If there are no changed files, the flow ends early with success and skips later extraction work. If files changed but no LLM
is configured, Extract fails because extraction requires an LLM.
If there are no changed files, Extract succeeds with no units; Integrate then has no unit work, Topics preserves any
existing target-day topics, and Finish still performs its normal catalog summary. If files changed but no LLM is
configured, Extract fails because extraction requires an LLM.
### 2. Integrate
`dream_integrate_step` invokes an agent independently for each unit and integrates that unit into one digest node. It exposes
these tools to the agent:
`dream_integrate_step` invokes an agent independently for each unit and integrates that unit into one digest node. It
exposes these tools to the agent:
```text
node_search, read, frontmatter_read, write, edit, frontmatter_update
```
This stage carries the core responsibility of `auto_link`. It first uses `node_search` to recall similar or related nodes at
digest-node granularity, decides whether to create or update a node, and finally writes sources and related digest nodes as
wikilinks. See [Auto Link](./auto_link.md) for the recall, deduplication, and edge-writing rules.
This stage carries the core responsibility of `auto_link`. It first uses `node_search` to recall similar or related
nodes at digest-node granularity, decides whether to create or update a node, and finally writes sources and related
digest nodes as wikilinks. See [Auto Link](./auto_link.md) for the recall, deduplication, and edge-writing rules.
Extract is the gate for deciding whether material is worth remembering, so Integrate has no `SKIP` action: each admitted
unit must land in exactly one digest node. Creates and updates must retain provenance and weave related digest links
into contextual sentences; bare wikilinks and standalone relationship fields are not valid output.
There are four integration actions:
| Action | Meaning |
|---|---|
| `CREATE` | No equivalent abstraction exists; create a new digest node. |
| Action | Meaning |
|---------------|--------------------------------------------------------------------------------|
| `CREATE` | No equivalent abstraction exists; create a new digest node. |
| `CORROBORATE` | The same memory appeared again; append a source or strengthen the description. |
| `REFINE` | New material adds boundaries, steps, prerequisites, applicability, or detail. |
| `CORRECT` | New material corrects errors, omissions, or conflicts in the existing node. |
| `REFINE` | New material adds boundaries, steps, prerequisites, applicability, or detail. |
| `CORRECT` | New material corrects errors, omissions, or conflicts in the existing node. |
Successfully integrated units are recorded in `integrate_results`. Failed units enter `failed_units`, and their source paths
enter `failed_paths`. The Finish stage does not checkpoint failed paths, ensuring that they can be retried later.
Successfully integrated units are recorded in `integrate_results`. Failed units enter `failed_units`, and their source
paths enter `failed_paths`. The Finish stage does not checkpoint failed paths, ensuring that they can be retried later.
### 3. Topics
@ -127,7 +148,7 @@ It reads:
```text
daily/<date>/interests.yaml
daily/<previous-date>/interests.yaml
daily/<each of the previous topic_diversity_days dates>/interests.yaml
```
Existing topics from the same day are preserved, while similar topics from the previous `topic_diversity_days` days are
@ -156,7 +177,8 @@ topics:
`dream_finish_step` completes the run:
1. Write successfully processed changed paths to `file_catalog: dream`.
2. Also write `daily/<date>/interests.yaml` and `daily/<date>.md` to the catalog.
2. Also write the target `daily/<date>/interests.yaml` and every refreshed day-index page in the scan window to the
catalog.
3. Persist the dream catalog if there were upserts or deletions.
4. Return a summary containing counts for scanned, changed, integrated, topics, checkpoints, and related values.
@ -177,6 +199,12 @@ With caller guidance:
reme auto_dream date=2026-06-20 hint="Prioritize engineering decisions and long-term preferences"
```
Override the default scan window and unit cap:
```bash
reme auto_dream date=2026-06-20 scan_days=3 max_units=8
```
The same set of steps can also be placed in a `cron` Job, for example to run every morning:
```yaml
@ -195,15 +223,16 @@ jobs:
## Important Boundaries
`auto_dream` consumes only daily inputs and does not rewrite daily bodies. Daily preserves facts and the original situation;
digest is the abstracted long-term memory layer.
`auto_dream` consumes only daily inputs and does not rewrite daily bodies. Daily preserves facts and the original
situation; digest is the abstracted long-term memory layer.
`digest` is not a copy of the source text. Its body should preserve reusable abstractions, while a Sources section points
back with entries such as `- [[daily/<date>/...]]`. Links follow the workspace-relative wikilink semantics described in
`digest` is not a copy of the source text. Its body should preserve reusable abstractions, while a Sources section
points back with contextual sentences such as `The decision was recorded in [[daily/<date>/decision.md]].` Links follow
the workspace-relative wikilink semantics described in
[Memory as File](./memory_as_file.md).
`auto_dream` does not invent an overview from nothing. Only content that actually appears in daily input and is extracted as
a unit or topic can enter digest or `interests.yaml`.
`auto_dream` does not invent an overview from nothing. Only content that actually appears in daily input and is
extracted as a unit or topic can enter digest or `interests.yaml`.
The complete flow depends on an LLM for Extract and Integrate. Topics can perform local deduplication without an LLM, but that
does not mean the full dream flow can run offline.
The complete flow depends on an LLM for Extract and Integrate. Topics can perform local deduplication without an LLM,
but that does not mean the full dream flow can run offline.

View file

@ -1,11 +1,12 @@
# Auto Link
In the current implementation, `auto_link` is not a separately registered Job. It is a capability of the Integrate stage in
In the current implementation, `auto_link` is not a separately registered Job. It is a capability of the Integrate stage
in
`auto_dream`: when `dream_integrate_step` writes a memory unit to `digest/`, it also recalls digest nodes, makes a
deduplication decision, links sources, and weaves wikilinks to related nodes into the result.
For the complete dream flow, see [Auto Dream](./auto_dream.md). For general wikilink, frontmatter, and workspace-relative
path semantics, see [Memory as File](./memory_as_file.md). For question-answering retrieval, see
For the complete dream flow, see [Auto Dream](./auto_dream.md). For general wikilink, frontmatter, and
workspace-relative path semantics, see [Memory as File](./memory_as_file.md). For question-answering retrieval, see
[Memory Search](./memory_search.md).
## Where It Runs
@ -21,19 +22,19 @@ auto_dream:
- dream_finish_step
```
The Integrate stage processes each unit independently. A unit is written to exactly one target digest node, but that node may
link to multiple sources and multiple related digest nodes.
The Integrate stage processes each unit independently. A unit is written to exactly one target digest node, but that
node may link to multiple sources and multiple related digest nodes.
## Goals
`auto_link` addresses graph quality at write time:
| Problem | Handling |
|---|---|
| The same memory already exists | Recall and update the existing node instead of creating a duplicate. |
| New and existing material are related | Write workspace-relative wikilinks into the body. |
| A digest node is disconnected from its sources | Add daily/resource links under a `## Sources` section. |
| A node contains only isolated prose | Add links to related digest nodes on both CREATE and UPDATE. |
| Problem | Handling |
|------------------------------------------------|----------------------------------------------------------------------|
| The same memory already exists | Recall and update the existing node instead of creating a duplicate. |
| New and existing material are related | Write workspace-relative wikilinks into the body. |
| A digest node is disconnected from its sources | Add daily/resource links under a `## Sources` section. |
| A node contains only isolated prose | Add links to related digest nodes on both CREATE and UPDATE. |
## Toolchain
@ -48,41 +49,42 @@ edit
frontmatter_update
```
`node_search` is digest-only node retrieval designed for dream integration. It returns node-level signals such as the digest
node's `path` and the `name` and `description` from frontmatter. It does not expand the body and does not perform the link
expansion used by ordinary search.
`node_search` is digest-only node retrieval designed for dream integration. It returns node-level signals such as the
digest node's `path` and the `name` and `description` from frontmatter. It does not expand the body and does not perform
the link expansion used by ordinary search.
`read` and `frontmatter_read` are used only for candidates that may be relevant, avoiding expansion of every recalled result
into a large context.
`read` and `frontmatter_read` are used only for candidates that may be relevant, avoiding expansion of every recalled
result into a large context.
## Linking Flow
### 1. Recall candidate nodes
The agent first calls `node_search` with the unit's triggers, verbs, nouns, synonyms, and possible failure modes. Broad recall,
for example `limit=20-30`, is recommended by default because this step serves both deduplication and link discovery.
The agent first calls `node_search` with the unit's triggers, verbs, nouns, synonyms, and possible failure modes. Broad
recall, for example `limit=20-30`, is recommended by default because this step serves both deduplication and link
discovery.
Recalled results are internally classified into three groups:
| Classification | Meaning | Next action |
|---|---|---|
| `same_abstraction` | The trigger or underlying abstraction is the same, with substantial content overlap. | Use as the UPDATE target. |
| `related` | An adjacent process, prerequisite, failure mode, concept, preference, or upstream/downstream knowledge. | Write a body wikilink. |
| `unrelated` | Only superficially similar or unrelated. | Ignore. |
| Classification | Meaning | Next action |
|--------------------|---------------------------------------------------------------------------------------------------------|---------------------------|
| `same_abstraction` | The trigger or underlying abstraction is the same, with substantial content overlap. | Use as the UPDATE target. |
| `related` | An adjacent process, prerequisite, failure mode, concept, preference, or upstream/downstream knowledge. | Write a body wikilink. |
| `unrelated` | Only superficially similar or unrelated. | Ignore. |
### 2. Choose a write action
Every unit must select one action:
| Action | Linking semantics |
|---|---|
| `CREATE` | Write a new `digest/<bucket>/<slug>.md` and add source and related-node links to its body. |
| `CORROBORATE` | The same abstraction appeared again; append its source link and strengthen the description when needed. |
| `REFINE` | New material extends the existing node; insert the additional content in the appropriate section and preserve existing links. |
| `CORRECT` | New material corrects the existing node; use source links to identify the basis for the correction. |
| Action | Linking semantics |
|---------------|-------------------------------------------------------------------------------------------------------------------------------|
| `CREATE` | Write a new `digest/<bucket>/<slug>.md` and add source and related-node links to its body. |
| `CORROBORATE` | The same abstraction appeared again; append its source link and strengthen the description when needed. |
| `REFINE` | New material extends the existing node; insert the additional content in the appropriate section and preserve existing links. |
| `CORRECT` | New material corrects the existing node; use source links to identify the basis for the correction. |
An UPDATE should be additive whenever possible: do not delete existing wikilinks or source entries. This prevents
later graph indexing and retrieval from losing edges.
An UPDATE should be additive whenever possible: do not delete existing wikilinks or source entries. This prevents later
graph indexing and retrieval from losing edges.
### 3. Write source edges
@ -91,12 +93,13 @@ Source edges are ordinary wikilinks grouped under a Markdown heading:
```markdown
## Sources
- [[daily/2026-06-20/session.md]]
- [[resource/2026-06-20/paper.md]]
The decision was recorded in [[daily/2026-06-20/session.md]], while the supporting technical evidence comes from
[[resource/2026-06-20/paper.md]].
```
These edges represent the evidence behind a digest node. Plain-text descriptions do not count as source edges because only
wikilinks can be parsed reliably by the file graph. For the complete parsing rules, see
These edges represent the evidence behind a digest node. Plain-text descriptions do not count as source edges because
only wikilinks can be parsed reliably by the file graph. The surrounding sentence must explain what each source
supports; a bare wikilink line is not valid Integrate output. For the complete parsing rules, see
[Memory as File](./memory_as_file.md#wikilink).
### 4. Write relationships between digest nodes
@ -113,11 +116,11 @@ This design extends [[digest/wiki/hybrid-search.md]] and uses
`auto_link` adjusts the shape of its output according to the unit bucket:
| Bucket | Writing focus |
|---|---|
| `procedure` | Write a runbook with triggers, steps, inputs, and failure modes. Link prerequisites, substeps, and related preferences. |
| `personal` | Write user-, team-, or project-specific facts and preferences. Link related projects, habits, and decision context. |
| `wiki` | Write general knowledge, principles, observations, and decision precedents. Link concepts, methods, and adjacent knowledge. |
| Bucket | Writing focus |
|-------------|-----------------------------------------------------------------------------------------------------------------------------|
| `procedure` | Write a runbook with triggers, steps, inputs, and failure modes. Link prerequisites, substeps, and related preferences. |
| `personal` | Write user-, team-, or project-specific facts and preferences. Link related projects, habits, and decision context. |
| `wiki` | Write general knowledge, principles, observations, and decision precedents. Link concepts, methods, and adjacent knowledge. |
Regardless of bucket, preserve source edges and weave recalled related digest nodes into the body whenever possible.
@ -125,19 +128,19 @@ Regardless of bucket, preserve source edges and weave recalled related digest no
`auto_link` uses `node_search`, not the question-answering `search`.
| Capability | Purpose |
|---|---|
| `search` | External question answering; returns chunks and can expand upstream/downstream link context. |
| Capability | Purpose |
|---------------|-----------------------------------------------------------------------------------------------------------|
| `search` | External question answering; returns chunks and can expand upstream/downstream link context. |
| `node_search` | Dream integration; recalls only digest node-level summaries for deduplication and related-link decisions. |
This boundary matters. The Integrate stage needs to decide whether the same abstraction already exists and which nodes should
be linked; it should not load large numbers of body chunks into context. [Memory Search](./memory_search.md) handles
question-oriented chunk retrieval, RRF fusion, and link expansion.
This boundary matters. The Integrate stage needs to decide whether the same abstraction already exists and which nodes
should be linked; it should not load large numbers of body chunks into context. [Memory Search](./memory_search.md)
handles question-oriented chunk retrieval, RRF fusion, and link expansion.
## Failure and Retry
If integration of a unit fails, `dream_integrate_step` records `failed_units` and `failed_paths`.
`dream_finish_step` does not checkpoint those source paths, so the next `auto_dream` run processes them again.
This makes auto_link writes retryable: a failure does not mark the input as complete or silently discard digest edges that
should have been created.
This makes auto_link writes retryable: a failure does not mark the input as complete or silently discard digest edges
that should have been created.

View file

@ -1,8 +1,9 @@
# Auto Memory
Auto Memory is ReMe's entry point for conversational memory. Each conversation is first distilled into a daily memory card
identified by `session_id`, and the day's `YYYY-MM-DD.md` page then indexes all of those cards. It turns "we talked about it"
into "it was remembered" while preserving the original conversation as evidence.
Auto Memory is ReMe's entry point for conversational memory. Within a target date, it uses `session_id` to find or update at
most one daily memory card, whose filename is a concise topic or event name chosen by the Agent. The day's `YYYY-MM-DD.md`
page indexes those cards. It turns "we talked about it" into "it was remembered" while retaining a source conversation record
as evidence.
<p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%">
@ -13,9 +14,9 @@ For the general file semantics of `daily/`, `session/`, frontmatter, and wikilin
```text
Conversation
├─ step 1: daily/YYYY-MM-DD/<session_id>.md # one card per conversation
├─ step 2: daily/YYYY-MM-DD.md # daily index linking the cards
└─ source: session/dialog/<session_id>.jsonl # original conversation
├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # one topic-named card per session
├─ step 2: daily/YYYY-MM-DD.md # daily index linking the cards
└─ source: session/dialog/<session_id>.jsonl # source conversation record
```
## What It Records
@ -39,29 +40,33 @@ workspace/
daily/
2026-06-20.md
2026-06-20/
session-a.md
session-b.md
login-refactor-decision.md
retrieval-regression.md
```
`daily/2026-06-20/session-a.md` and `daily/2026-06-20/session-b.md` are memory cards distilled from different
conversations. `daily/2026-06-20.md` is the index page for that day. Resource files enter the same daily memory layer; see
The two files under the date directory are topic-named cards distilled from different conversations.
`daily/2026-06-20.md` is the index page for that day. Resource files enter the same daily memory layer; see
[Auto Resource](./auto_resource.md).
When a call includes `session_id`, Auto Memory records that conversation separately under the given ID:
When a call includes `session_id`, Auto Memory uses it to find the corresponding card through frontmatter, while the Agent
chooses a readable filename through `name`:
```text
daily/2026-06-20/session-a.md
```yaml
name: login-refactor-decision
session_id: session-a
source_conversation: "[[session/dialog/session-a.jsonl]]"
```
This keeps different conversations separate. A requirements discussion, a debugging session, and a documentation update can
each have their own memory card. To see what happened on a particular day, start with `YYYY-MM-DD.md`. To inspect what was
distilled from one conversation, open the corresponding `<session_id>.md`.
This keeps different conversations separate without forcing opaque IDs into filenames. An update locates the existing note by
`session_id` or `source_conversation`; if the Agent supplies a better frontmatter `name`, the system can rename the note and
retarget inbound wikilinks. To see what happened on a day, start with `YYYY-MM-DD.md`.
## Preserving the Original Information
The distilled daily note is optimized for readability; the original conversation is retained for trust and verification.
The distilled daily note is optimized for readability; a filtered source conversation record is retained for trust and
verification.
While generating memory cards, Auto Memory also saves the raw sessions:
While generating memory cards, Auto Memory also saves the source messages:
```text
session/
@ -70,12 +75,12 @@ session/
session-b.jsonl
```
Each daily note points to its corresponding original conversation. When a memory needs verification, follow that link back to
the complete context in which it was created.
Each daily note points to its corresponding conversation record. Saved messages omit tool-result blocks and base64 data
blocks, preventing recalled memory and binary payloads from being mistaken for user-provided evidence later.
## Message Timestamps
Auto Memory preserves each message's `created_at` in both the prompt and the raw session JSONL. When importing historical
Auto Memory preserves each retained message's `created_at` in both the prompt and the source conversation JSONL. When importing historical
conversations or benchmark data, provide the actual occurrence time for every message so the model does not confuse event
time with execution time:
@ -92,8 +97,8 @@ For compatibility with common dataset schemas, `auto_memory` also checks `time_c
`timeCreated`, and `created_time` when `created_at` is absent. These fields may appear either at the top level of a message
or inside `metadata`.
When a call does not explicitly provide `date`, Auto Memory uses the date of the earliest valid `created_at` value in the
messages. If no message contains a valid timestamp, it falls back to the current date. Historical imports may also specify the
When a call does not explicitly provide `date`, Auto Memory uses the latest valid `created_at` date in the messages. If no
message contains a valid timestamp, it falls back to the current date. Historical imports may also specify the
target date directly:
```bash

View file

@ -1,8 +1,8 @@
# Auto Resource `Beta`
Auto Resource is ReMe's entry point for interpreting resources and is currently in **Beta**. Resource files first enter
`resource/` by date and are then interpreted into daily resource cards. Each card's filename comes from the LLM-generated
frontmatter `name`, and `source_resource` links the card back to its original file.
`resource/`, preferably under a date directory, and are then interpreted into daily resource cards. Each card's filename
comes from the LLM-generated frontmatter `name`, and `source_resource` links the card back to its original file.
<p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%">
@ -13,7 +13,7 @@ For the general file semantics of workspace layers, `resource/`, and `daily/`, s
[Auto Memory](./auto_memory.md).
```text
resource/YYYY-MM-DD/<resource_file>
resource/[YYYY-MM-DD/]<resource_file>
├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # interpreted resource card
├─ step 2: source_resource points to the original resource
└─ step 3: daily/YYYY-MM-DD.md # daily index linking the cards
@ -21,8 +21,8 @@ resource/YYYY-MM-DD/<resource_file>
## What It Records
Auto Resource does more than copy file content. It extracts information that will make the resource easier to retrieve and
understand later:
Auto Resource does more than copy file content. It extracts information that will make the resource easier to retrieve
and understand later:
- Core content: what the resource is mainly about.
- Structure: its sections, tables, fields, and data organization.
@ -34,26 +34,28 @@ In short, it turns "a file was archived" into "the resource is usable."
## Original Resource Entry Point
Auto Resource uses `resource/` as the entry point for source material. Resources must be placed under a date, which determines
the day whose daily memory layer receives the interpreted card.
Auto Resource uses `resource/` as the entry point for source material. Date directories are recommended, and their date
determines which daily memory layer receives the interpreted card. A file directly under `resource/` is also supported
and uses today in the application timezone.
Example directory:
```text
workspace/
resource/
quick-note.txt # enters today's daily layer
2026-06-20/
market-report.md
meeting-notes.csv
```
The current Beta version is best suited to text-based resources such as `md`, `txt`, `json`, `jsonl`, `csv`, `yaml`,
and `html`.
The current Beta version is best suited to text-based resources such as `md`, `txt`, `json`, `jsonl`, `csv`, `yaml`, and
`html`.
## Resource Cards
Each resource file produces one daily resource card. The system initially uses the resource file's stem as a temporary path.
After the agent writes the card, the file is renamed according to its frontmatter `name`:
Each resource file produces one daily resource card. The system initially uses the resource file's stem as a temporary
path. After the agent writes the card, the file is renamed according to its frontmatter `name`:
```text
resource/2026-06-20/market-report.md
@ -67,14 +69,14 @@ The resource card links to the original file through frontmatter:
source_resource: "[[resource/2026-06-20/market-report.md]]"
```
When a resource changes, Auto Resource finds and updates the corresponding card through `source_resource`. When a resource is
deleted, its daily note is also removed. The older `daily/YYYY-MM-DD/<resource_stem>.md` naming convention remains supported
as a fallback.
When a resource changes, Auto Resource finds and updates the corresponding card through `source_resource`. When a
resource is deleted, its daily note is also removed. The older `daily/YYYY-MM-DD/<resource_stem>.md` naming convention
remains supported as a fallback.
## Daily Index
Resource cards enter the same daily memory layer as Auto Memory cards. The day's `YYYY-MM-DD.md` page acts as an index and
organizes those resource cards:
Resource cards enter the same daily memory layer as Auto Memory cards. The day's `YYYY-MM-DD.md` page acts as an index
and organizes those resource cards:
```text
daily/
@ -91,11 +93,13 @@ resource, open its corresponding resource card.
The interpreted daily note is optimized for readability; the original resource is retained for trust and verification.
Auto Resource does not move the original file. It remains under `resource/YYYY-MM-DD/`. Text resources can therefore enter
the daily memory flow while their source files stay in their original location.
Auto Resource does not move the original file. It remains at its original path under `resource/`. Text resources can
therefore enter the daily memory flow while their source files stay in their original location.
## What Happens Next
Auto Resource only creates resource interpretations in the daily layer. To distill long-term knowledge from resources into
`digest/`, use [Auto Dream](./auto_dream.md). To search original resources, daily cards, and digest nodes, use
[Memory Search](./memory_search.md).
Auto Resource only creates resource interpretations in the daily layer. To distill long-term knowledge from resources
into
`digest/`, use [Auto Dream](./auto_dream.md). The default live index covers daily cards and digest nodes. Run
`reme reindex`
when original resource files must also be directly searchable; see [Memory Search](./memory_search.md).

View file

@ -8,12 +8,12 @@ ReMe is open source and hosted on GitHub:
## How to Contribute
Thank you for your interest in ReMe. ReMe is a file-first, self-evolving memory system for agents. Contributions are welcome
through issue reports, documentation improvements, additional tests, bug fixes, and new capabilities.
Thank you for your interest in ReMe. ReMe is a file-first, self-evolving memory system for agents. Contributions are
welcome through issue reports, documentation improvements, additional tests, bug fixes, and new capabilities.
If this is your first time running ReMe locally, start with [Quick Start](./quick_start.md). If your change affects runtime
layers, Jobs, Steps, or components, read [ReMe Framework](./framework.md). If it affects workspace directories, frontmatter,
wikilinks, or chunking, read [Memory as File](./memory_as_file.md).
If this is your first time running ReMe locally, start with [Quick Start](./quick_start.md). If your change affects
runtime layers, Jobs, Steps, or components, read [ReMe Framework](./framework.md). If it affects workspace directories,
frontmatter, wikilinks, or chunking, read [Memory as File](./memory_as_file.md).
### 1. Before You Begin
@ -21,9 +21,10 @@ Before investing in an implementation:
- Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) for an existing issue or discussion.
- If a related issue is still open, comment that you would like to work on it to avoid duplicate effort.
- If no issue exists, create one describing the context, expected behavior, possible implementation, and scope of impact.
- For larger feature changes, align with maintainers on interfaces, configuration, compatibility, and test strategy before
submitting an implementation.
- If no issue exists, create one describing the context, expected behavior, possible implementation, and scope of
impact.
- For larger feature changes, align with maintainers on interfaces, configuration, compatibility, and test strategy
before submitting an implementation.
### 2. Local Development Environment
@ -38,7 +39,11 @@ The project requires Python 3.11 or later. A virtual environment is recommended:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,full]"
pip install -e reme_studio -e ".[dev,full]"
cd reme_studio
npm ci
npm run build:static
cd ..
pre-commit install
```
@ -53,8 +58,8 @@ CLI / Client -> Service -> Application -> Job -> Step -> Component / Workspace
In practice:
- Capabilities exposed to users or external systems should normally be orchestrated by a Job, then exposed by a Service as a
CLI-, HTTP-, or MCP-callable interface.
- Capabilities exposed to users or external systems should normally be orchestrated by a Job, then exposed by a Service
as a CLI-, HTTP-, or MCP-callable interface.
- Reusable infrastructure belongs in `reme/components/`, with dependencies declared through `BaseComponent.bind()`.
- Atomic business operations belong in `reme/steps/` and access the file store, agent wrapper, catalog, LLM, and other
components through `BaseStep.Ref`.
@ -65,31 +70,33 @@ In practice:
When adding a Step or Job, pay particular attention to these conventions:
- Register implementations with `@R.register("<backend_name>")`. Registration names should be stable, clear, and match the
configured `backend`.
- After adding a Step file, make sure its package `__init__.py` imports the module; otherwise, the registry will not load it.
- Register implementations with `@R.register("<backend_name>")`. Registration names should be stable, clear, and match
the configured `backend`.
- After adding a Step file, make sure its package `__init__.py` imports the module; otherwise, the registry will not
load it.
- A Step should perform one atomic business operation. Cross-step flows belong in Job configuration or a dedicated
orchestration Step.
- A Job composes Steps and selects normal, streaming, background, or scheduled execution. `enable_serve` controls whether it
is externally exposed.
- A Job composes Steps and selects normal, streaming, background, or scheduled execution. `enable_serve` controls
whether it is externally exposed.
- When a Step needs components, prefer `BaseStep.Ref`. Do not reconstruct global components inside a Step or bypass
`ApplicationContext`.
- File, index, graph, frontmatter, and wikilink behavior must preserve consistent workspace-relative path semantics.
- Add fast tests under `tests/unit/` for new capabilities. Put cross-component, LLM, embedding, or service behavior under
- Add fast tests under `tests/unit/` for new capabilities. Put cross-component, LLM, embedding, or service behavior
under
`tests/integration/` when appropriate.
### 4. Code and Documentation Changes
Choose the appropriate entry point for the type of change:
| Change type | Primary location | Guidance |
|---|---|---|
| Configuration or startup behavior | `reme/config/`, `reme/application.py`, `reme/reme.py` | Keep the default configuration runnable and avoid breaking existing CLI, HTTP, and MCP entry points. |
| Component capability | `reme/components/` | Reuse `BaseComponent`, the registry, and context objects. |
| Job or Step | `reme/components/job/`, `reme/steps/` | Follow the Job -> Step model in [ReMe Framework](./framework.md), keep request and response schemas clear, and add corresponding tests. |
| Data structure | `reme/schema/`, `reme/enumeration/` | Preserve serialization compatibility and existing frontmatter and wikilink semantics. |
| Utility | `reme/utils/` | Keep function boundaries small and cover edge cases with unit tests. |
| User documentation | `docs/en/`, `README.md` | Update documentation when user-visible behavior changes. |
| Change type | Primary location | Guidance |
|-----------------------------------|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| Configuration or startup behavior | `reme/config/`, `reme/application.py`, `reme/reme.py` | Keep the default configuration runnable and avoid breaking existing CLI, HTTP, and MCP entry points. |
| Component capability | `reme/components/` | Reuse `BaseComponent`, the registry, and context objects. |
| Job or Step | `reme/components/job/`, `reme/steps/` | Follow the Job -> Step model in [ReMe Framework](./framework.md), keep request and response schemas clear, and add corresponding tests. |
| Data structure | `reme/schema/`, `reme/enumeration/` | Preserve serialization compatibility and existing frontmatter and wikilink semantics. |
| Utility | `reme/utils/` | Keep function boundaries small and cover edge cases with unit tests. |
| User documentation | `docs/en/`, `README.md` | Update documentation when user-visible behavior changes. |
If a change involves an LLM, embeddings, an external service, file watching, or a background task, also describe its
dependencies, failure behavior, and local validation method.
@ -165,15 +172,16 @@ pytest tests/unit/test_reme_cli.py
If `pre-commit` modifies files automatically, commit those changes and rerun the checks until everything passes.
The current pre-commit configuration includes YAML/TOML/JSON validation, private-key detection, trailing-whitespace checks,
The current pre-commit configuration includes YAML/TOML/JSON validation, private-key detection, trailing-whitespace
checks,
`black`, `flake8`, `pylint`, and `pyroma`. The main formatting rules are:
- `black --line-length=120`
- `flake8 --max-line-length=120`
- `pylint --max-line-length=120`
Some integration tests may require an LLM, embeddings, or external service configuration. If you cannot run them locally,
state why they were skipped and what alternative validation you completed in the PR description.
Some integration tests may require an LLM, embeddings, or external service configuration. If you cannot run them
locally, state why they were skipped and what alternative validation you completed in the PR description.
### 8. Testing Requirements
@ -183,7 +191,8 @@ Add tests according to the risk of the change:
- For a new Step, Job, or component, cover at least the main path and a failure path.
- For changes to shared logic such as indexes, graphs, wikilinks, frontmatter, or file operations, add edge cases.
- For changes to the CLI, services, or configuration parsing, cover the user-visible entry point.
- Documentation-only changes usually do not require new tests, but running `pre-commit run --all-files` is still recommended.
- Documentation-only changes usually do not require new tests, but running `pre-commit run --all-files` is still
recommended.
Place tests according to the existing structure:
@ -213,9 +222,9 @@ Documentation should:
- Bugs and feature requests: [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues)
- Project home: [GitHub Repository](https://github.com/agentscope-ai/ReMe)
- Documentation site: [https://reme.agentscope.io/](https://reme.agentscope.io/)
- Documentation site: [https://reme.agentscope.io](https://reme.agentscope.io)
---
Thank you for contributing to ReMe. Your improvements help make long-term memory for agents more readable, controllable, and
maintainable.
Thank you for contributing to ReMe. Your improvements help make long-term memory for agents more readable, controllable,
and maintainable.

View file

@ -16,13 +16,13 @@ To run and use ReMe first, see [Quick Start](./quick_start.md). For workspace fi
### Capability Boundary
ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into `digest/`,
and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP.
ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into
`digest/`, and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP.
Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current conversation,
injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those capabilities belong in
the host agent framework. ReMe accepts conversations, resources, and file changes that have already occurred and persists the
information with long-term value.
Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current
conversation, injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those
capabilities belong in the host agent framework. ReMe accepts conversations, resources, and file changes that have
already occurred and persists the information with long-term value.
```mermaid
flowchart LR
@ -38,16 +38,16 @@ flowchart LR
Core layers:
| Layer | Main location | Responsibility |
|---|---|---|
| CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. |
| Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. |
| Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. |
| Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. |
| Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. |
| Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. |
| Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. |
| Config | `reme/config/` | Default YAML configuration and command-line override parsing. |
| Layer | Main location | Responsibility |
|-------------|----------------------------|---------------------------------------------------------------------------------------------------|
| CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. |
| Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. |
| Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. |
| Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. |
| Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. |
| Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. |
| Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. |
| Config | `reme/config/` | Default YAML configuration and command-line override parsing. |
## 2. Directory Structure
@ -55,11 +55,12 @@ Core layers:
reme/
reme.py # CLI entry point
application.py # Application assembly and lifecycle
plugin.py # installed plugin contract and entry-point loader
config/
default.yaml # default service / jobs / components
config_parser.py # config=, dot notation, and env placeholder parsing
components/
component_registry.py # global registry R
component_registry.py # backend registry and application-local copies
base_component.py # ComponentMixin / BaseComponent / bind dependency declarations
runtime_context.py # context for one Job execution
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
@ -68,18 +69,25 @@ reme/
file_store/ # file-index coordination layer
file_graph/ # wikilink graph
keyword_index/ # BM25 and other keyword indexes
file_chunker/ # Markdown / default text chunking
file_chunker/ # Markdown / JSON / JSONL / generic text chunking
file_catalog/ # change checkpoints
as_llm/, as_embedding/ # model wrappers
agent_wrapper/ # AgentScope / Claude Code wrappers
agent_wrapper/ # AgentScope / Claude Code / Codex wrappers
steps/
base_step.py # BaseStep, Ref, dispatch_steps
common/ # version, help, health_check, demo
common/ # version, help, health_check, status, chat
benchmark/ # LongMemEval / BEAM evaluation steps
cookbook/ # built-in cookbook support steps
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory, auto_resource, auto_dream, proactive
transfer/ # upload/download/ingest
channel/ # MCP channel tools
transfer/ # upload/download
plugins/
auto-fin/ # independent example plugin distribution
daily_paper/ # independent paper-research plugin distribution
integrations/
claude_code/ # Claude Code adapter and marketplace
hermes_agent/ # Hermes Agent memory-provider adapter
```
The default workspace directories are defined by `ApplicationConfig`:
@ -87,7 +95,8 @@ The default workspace directories are defined by `ApplicationConfig`:
```text
<workspace_dir>/
metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state
session/ # agent sessions and original conversations
session/ # source conversations used by memory workflows
mem_session/ # generated Agent wrapper sessions and configuration
resource/ # external resources
daily/ # lightly processed memory
digest/ # long-term digest memory
@ -127,13 +136,13 @@ reme search query="memory" backend=mcp
Configuration parsing supports:
| Capability | Source | Description |
|---|---|---|
| Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. |
| Explicit configuration | `config=<name-or-path>` | Accept a built-in configuration name or a YAML/JSON file path. |
| Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. |
| Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. |
| Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. |
| Capability | Source | Description |
|------------------------|-------------------------|--------------------------------------------------------------------------|
| Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. |
| Explicit configuration | `config=<name-or-path>` | Accept a built-in configuration name or a YAML/JSON file path. |
| Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. |
| Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. |
| Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. |
### 3.2 Service
@ -157,22 +166,28 @@ flowchart LR
HTTP service behavior:
| Job type | HTTP exposure |
|---|---|
| Non-`StreamJob` with `enable_serve: true` | `POST /<job.name>` returning `Response` JSON. |
| `StreamJob` | `POST /<job.name>` returning `text/event-stream`. |
| `enable_serve: false` | No endpoint is registered. |
| Job type | HTTP exposure |
|-------------------------------------------|---------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | `POST /<job.name>` returning `Response` JSON. |
| `StreamJob` | `POST /<job.name>` returning `text/event-stream`. |
| `enable_serve: false` | No endpoint is registered. |
After registering Job endpoints, the HTTP service can also mount the ReMe Studio single-page application. The default is
`service.web_enabled=true`. Builds are resolved from `service.web_static_dir`, `REME_WEB_STATIC_DIR`, the optional
`reme_studio` package installed by the `web` and `core` extras, and source-tree locations such as
`reme_studio/dist-static`. If no `index.html` is found, only the frontend is skipped and the Job API remains available. The
Studio `GET` fallback does not replace existing `POST /<job.name>` routes.
MCP service behavior:
| Job type | MCP exposure |
|---|---|
| Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. |
| `StreamJob` | Currently skipped and not registered. |
| `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. |
| Job type | MCP exposure |
|-------------------------------------------|-------------------------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. |
| `StreamJob` | Currently skipped and not registered. |
| `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. |
MCP services can inject server-owned arguments with `injected_job_kwargs`; callers cannot override those arguments.
Set `tool_error_on_failure: true` to expose an unsuccessful ReMe `Response` as an MCP tool error.
MCP services can inject server-owned arguments with `injected_job_kwargs`; callers cannot override those arguments. Set
`tool_error_on_failure: true` to expose an unsuccessful ReMe `Response` as an MCP tool error.
## 4. Registry and Dependency Injection
@ -198,24 +213,51 @@ The registry key is:
`component_type` comes from a class attribute:
| Type | Class attribute |
|---|---|
| Step | `BaseStep.component_type = ComponentEnum.STEP` |
| Job | `BaseJob.component_type = ComponentEnum.JOB` |
| Service | `BaseService.component_type = ComponentEnum.SERVICE` |
| Type | Class attribute |
|-----------|-----------------------------------------------------------|
| Step | `BaseStep.component_type = ComponentEnum.STEP` |
| Job | `BaseJob.component_type = ComponentEnum.JOB` |
| Service | `BaseService.component_type = ComponentEnum.SERVICE` |
| FileStore | `BaseFileStore.component_type = ComponentEnum.FILE_STORE` |
The same backend name can therefore exist under different component types. For example, `http` can be both a service backend
and a client backend.
The same backend name can therefore exist under different component types. For example, `http` can be both a service
backend and a client backend.
### 4.2 Registration Through Module Imports
`ComponentEnum` provides the built-in identifiers, but installed plugins may declare a new type with a namespaced
string such as `example.reranker`. Custom identifiers use lowercase letters and numbers separated by `.`, `_`, or `-`.
They are configured under `components` and participate in the same dependency ordering and lifecycle as built-ins.
Registration happens when a module is imported. `reme/components/__init__.py` imports component packages, while
`reme/steps/__init__.py` imports `channel/common/evolve/file_io/index/transfer`. Each package's `__init__.py` then imports
its concrete modules, causing `@R.register(...)` to execute.
### 4.2 Built-in and Plugin Registration
After adding a Step file, make sure the package's `__init__.py` imports it. Otherwise, the backend will not appear in the
registry.
Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point
group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to
`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The
entry-point name is the plugin's identity.
Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override.
Plugin registration therefore stays local to one application;
duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other.
The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration
files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. See the independently
packaged [Auto Fin](../../plugins/auto-fin/README.md) and [Daily Paper](../../plugins/daily_paper/README.md) plugins.
Plugin packages are managed locally and remain separate from per-application activation:
```bash
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
```
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
### 4.3 Component.bind
@ -234,18 +276,18 @@ flowchart LR
Rules for `BaseComponent.bind(name, BaseClass, optional=True)`:
| Scenario | Behavior |
|---|---|
| `name` is empty | Return `None` and skip the dependency. |
| `app_context` exists | Look up `app_context.components[ctype][name]`. |
| Dependency missing and `optional=True` | Resolve to `None`. |
| Dependency missing and `optional=False` | Fail at startup. |
| Standalone mode | A private component can be created with `default_factory`. |
| Scenario | Behavior |
|-----------------------------------------|------------------------------------------------------------|
| `name` is empty | Return `None` and skip the dependency. |
| `app_context` exists | Look up `app_context.components[ctype][name]`. |
| Dependency missing and `optional=True` | Resolve to `None`. |
| Dependency missing and `optional=False` | Fail at startup. |
| Standalone mode | A private component can be created with `default_factory`. |
### 4.4 Step.Ref
Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps access
components primarily through `BaseStep.Ref`:
Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps
access components primarily through `BaseStep.Ref`:
```python
file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
@ -301,11 +343,13 @@ flowchart LR
F --> G["start CronJob"]
```
During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their dependencies.
During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their
dependencies.
## 6. Job Model
A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under `jobs:`
A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under
`jobs:`
in `reme/config/default.yaml`.
### 6.1 BaseJob
@ -326,23 +370,23 @@ flowchart LR
Important source behavior:
| Source | Behavior |
|---|---|
| `_start()` | Parse each Step config from YAML into `(step_cls, params)`. |
| `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. |
| `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. |
| Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. |
| Source | Behavior |
|--------------------|----------------------------------------------------------------------------------|
| `_start()` | Parse each Step config from YAML into `(step_cls, params)`. |
| `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. |
| `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. |
| Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. |
### 6.2 StreamJob
`StreamJob` extends `BaseJob` but returns streaming chunks:
| Behavior | Description |
|---|---|
| Context | Includes `stream_queue`. |
| Behavior | Description |
|-------------|------------------------------------------------------------|
| Context | Includes `stream_queue`. |
| Step output | Call `context.add_stream_string(text, ChunkEnum.CONTENT)`. |
| Exception | Write `ChunkEnum.ERROR`. |
| Completion | Always send a `DONE` chunk. |
| Exception | Write `ChunkEnum.ERROR`. |
| Completion | Always send a `DONE` chunk. |
### 6.3 BackgroundJob
@ -362,8 +406,8 @@ flowchart LR
J --> K["wait close_timeout; cancel on timeout"]
```
The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions, which
allows the supervisor to restart the task.
The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions,
which allows the supervisor to restart the task.
### 6.4 CronJob
@ -389,7 +433,9 @@ The current implementation uses `croniter` to calculate the next trigger time. T
```mermaid
flowchart LR
Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"]
Jobs --> Base["base<br/>version / help / health_check<br/>search / node_search / traverse / reindex<br/>read / write / edit / delete / move / list / stat<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_resource / auto_dream / proactive"]
Jobs --> Cron["cron<br/>dream_cron<br/>optimize_index_cron"]
Jobs --> Stream["stream<br/>chat"]
Jobs --> Base["base<br/>version / help / health_check / status / app_config<br/>search / node_search / traverse / graph_snapshot / reindex<br/>read / load / read_image / write / save / edit / delete / move / list / stat / frontmatter_*<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_memory_cc / auto_resource / auto_dream / proactive"]
```
## 7. Step Model
@ -413,12 +459,12 @@ flowchart LR
`RuntimeContext` is shared by all Steps within one Job invocation:
| Field | Description |
|---|---|
| `response` | Final `Response(answer, success, metadata)`. |
| `data` | Free-form dictionary containing input parameters and intermediate results. |
| `stream_queue` | Output queue for streaming Jobs. |
| `stop_event` | Stop signal for background Jobs. |
| Field | Description |
|----------------|----------------------------------------------------------------------------|
| `response` | Final `Response(answer, success, metadata)`. |
| `data` | Free-form dictionary containing input parameters and intermediate results. |
| `stream_queue` | Output queue for streaming Jobs. |
| `stop_event` | Stop signal for background Jobs. |
Common Step code:
@ -482,21 +528,22 @@ flowchart LR
Current default components in `reme/config/default.yaml`:
| ComponentEnum | Name | Backend | Description |
|---|---|---|---|
| `service` | singleton | `http` | Default HTTP service. |
| `tokenizer` | `default` | `regex` | BM25 tokenizer. |
| `as_embedding` | `default` | `${EMBEDDING_BACKEND:-openai}` | Embedding model wrapper. |
| `embedding_store` | `default` | `local` | Embedding store depending on `as_embedding: default`. |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. |
| `file_graph` | `default` | `local` | Wikilink graph. |
| `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. |
| `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. |
| `file_chunker` | `default` | `default` | Default text chunking, currently supporting `jsonl`. |
| `keyword_index` | `default` | `bm25` | BM25 keyword index. |
| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. |
| ComponentEnum | Name | Backend | Description |
|-------------------|---------------------------------|--------------------------------------------------|--------------------------------------------------------------------------------|
| `service` | singleton | `http` | Default HTTP service. |
| `tokenizer` | `default` | `regex` | BM25 tokenizer. |
| `as_embedding` | `default` | Not configured by default; example uses `openai` | Provides the embedding model wrapper after uncommenting the example config. |
| `embedding_store` | `default` | Not configured by default; example uses `local` | Depends on `as_embedding: default` after uncommenting the example config. |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. |
| `agent_wrapper` | `codex/codex_oauth` | `codex` | Codex wrappers for API-key and OAuth authentication. |
| `file_graph` | `default` | `local` | Wikilink graph. |
| `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. |
| `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. |
| `file_chunker` | `json/jsonl/default` | `json/jsonl/default` | JSON, JSONL, and generic text chunkers; generic text supports `txt` and `log`. |
| `keyword_index` | `default` | `bm25` | BM25 keyword index. |
| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. |
Note that the `search` Step configuration contains `vector_weight`, but `file_store.default.embedding_store` is empty by
default. Vector retrieval is available only when the runtime configuration enables an embedding store.
@ -554,12 +601,12 @@ class MySearchStep(BaseStep):
Common attributes available directly:
| Attribute | Component resolved by default |
|---|---|
| `self.as_llm` | `.model` from `as_llm: default`. |
| Attribute | Component resolved by default |
|----------------------|-------------------------------------|
| `self.as_llm` | `.model` from `as_llm: default`. |
| `self.agent_wrapper` | `agent_wrapper: default`; optional. |
| `self.file_catalog` | `file_catalog: default`; optional. |
| `self.file_store` | `file_store: default`. |
| `self.file_catalog` | `file_catalog: default`; optional. |
| `self.file_store` | `file_store: default`. |
To select a non-default component from Job configuration:
@ -571,13 +618,13 @@ steps:
### 9.4 Step Design Guidance
| Guidance | Reason |
|---|---|
| Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. |
| Write the final result to `context.response`. | Services and clients consume the standard `Response`. |
| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. |
| A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. |
| Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. |
| Guidance | Reason |
|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
| Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. |
| Write the final result to `context.response`. | Services and clients consume the standard `Response`. |
| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. |
| A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. |
| Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. |
### 9.5 Unit Test Example
@ -600,8 +647,8 @@ async def test_uppercase_step():
## 10. Adding a Job
A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new execution
model is required.
A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new
execution model is required.
### 10.1 Adding a Normal Request Job
@ -736,11 +783,11 @@ jobs:
Characteristics of a background Job:
| Characteristic | Description |
|---|---|
| Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. |
| Has a supervisor | Restarts with exponential backoff after an exception by default. |
| Has a stop event | Notifies the loop to exit during close. |
| Characteristic | Description |
|---------------------------------|--------------------------------------------------------------------|
| Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. |
| Has a supervisor | Restarts with exponential backoff after an exception by default. |
| Has a stop event | Notifies the loop to exit during close. |
| Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. |
### 10.5 Adding a Cron Job
@ -767,14 +814,14 @@ An invalid `cron` expression fails at startup.
Most use cases require only a new Step plus a YAML Job. Consider adding `reme/components/job/*.py` only in these cases:
| Requirement | New Job class? |
|---|---|
| Add a business command | No; use `backend: base`. |
| Chain existing steps | No; use `steps:`. |
| Need SSE/streaming output | No; use `backend: stream`. |
| Need a background loop | No; use `backend: background`. |
| Need cron scheduling | No; use `backend: cron`. |
| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. |
| Requirement | New Job class? |
|---------------------------------------------------------------------|--------------------------------|
| Add a business command | No; use `backend: base`. |
| Chain existing steps | No; use `steps:`. |
| Need SSE/streaming output | No; use `backend: stream`. |
| Need a background loop | No; use `backend: background`. |
| Need cron scheduling | No; use `backend: cron`. |
| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. |
Minimal shape of a new Job backend:

View file

@ -6,37 +6,39 @@ ReMe's core idea is **Memory as File, File as Memory**.
<img src="../figure/memory-as-file.svg" alt="ReMe Memory as File model" width="92%">
</p>
**Memory as File**: long-term memory is not hidden in a black-box database. It lives in Markdown files, resource files, and
index snapshots under the workspace. Users and agents can directly read, write, move, and delete those files.
**Memory as File**: long-term memory is not hidden in a black-box database. Its source material and readable memories
live in user-owned files under the workspace. Users and agents can directly read, write, move, and delete those files;
indexes and snapshots under `metadata/` are derived state that can be rebuilt.
**File as Memory**: each file is more than ordinary text. It is an indexable, linkable, and evolvable memory node. ReMe parses
frontmatter, body chunks, and wikilink edges from files and organizes them into retrieval indexes and a graph.
**File as Memory**: each file is more than ordinary text. It is an indexable, linkable, and evolvable memory node. ReMe
parses frontmatter, body chunks, and wikilink edges from files and organizes them into retrieval indexes and a graph.
In other words, files are both a human-readable interface and an operational interface for agents. Directory structure
carries the memory layers, while Markdown syntax expresses content, metadata, and relationships.
## Design Goals
ReMe represents memory as files not merely for convenient storage, but to give long-term memory several essential properties:
ReMe represents memory as files not merely for convenient storage, but to give long-term memory several essential
properties:
| Goal | Meaning |
|---|---|
| Readable | Users can open the workspace directly and read daily notes, digest nodes, and source material like ordinary notes. |
| Editable | Users and agents can correct, extend, move, or delete memory with file operations, without a specialized database client. |
| Traceable | Long-term conclusions in digest can point back to daily, resource, or session files from a Sources section. |
| Portable | The workspace is an ordinary directory. Markdown, JSONL, YAML, and resource files can be backed up, synchronized, versioned, or moved to other tools. |
| Indexable | Although the files are plain text, ReMe parses frontmatter, chunks, and wikilinks to build a retrieval index and file graph. |
| Collaborative | Humans judge and correct; agents organize, link, and retrieve. Both operate on the same files. |
| Goal | Meaning |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| Readable | Users can open the workspace directly and read daily notes, digest nodes, and source material like ordinary notes. |
| Editable | Users and agents can correct, extend, move, or delete memory with file operations, without a specialized database client. |
| Traceable | Long-term conclusions in digest can point back to daily, resource, or session files from a Sources section. |
| Portable | The workspace is an ordinary directory. Markdown, JSONL, YAML, and resource files can be backed up, synchronized, versioned, or moved to other tools. |
| Indexable | Although the files are plain text, ReMe parses frontmatter, chunks, and wikilinks to build a retrieval index and file graph. |
| Collaborative | Humans judge and correct; agents organize, link, and retrieve. Both operate on the same files. |
ReMe memory is therefore neither a hidden database record nor a prompt fragment visible only to an LLM. It is first a file
owned by the user and only then indexed by the system for retrieval.
ReMe memory is therefore neither a hidden database record nor a prompt fragment visible only to an LLM. It is first a
file owned by the user and only then indexed by the system for retrieval.
## Memory Layers
A ReMe workspace divides memory into four layers:
```text
raw input -> session/ + resource/
source records -> session/ + resource/
working memory -> daily/
long memory -> digest/
system state -> metadata/
@ -44,19 +46,23 @@ system state -> metadata/
Each layer solves a different problem.
`session/` and `resource/` preserve raw input. Their purpose is to retain the original situation: conversations, agent
sessions, uploaded material, web pages, and reports remain intact as evidence for later verification.
`session/` and `resource/` preserve source records. Files under `resource/` remain unchanged at their original path.
Standard Auto Memory records retain conversation messages while intentionally omitting tool-result and base64 data
blocks; this keeps recalled output and binary payloads from masquerading as user-provided evidence. Generated Agent
runtime state instead lives under `mem_session/`.
`daily/` is the lightly processed layer. It organizes the day's conversations and resources into more readable daily notes:
what happened, which conclusions were reached, which follow-up tasks remain, and where the source material lives. Daily does
not aim for final abstraction; it is closer to a workbench for the day.
`daily/` is the lightly processed layer. It organizes the day's conversations and resources into more readable daily
notes:
what happened, which conclusions were reached, which follow-up tasks remain, and where the source material lives. Daily
does not aim for final abstraction; it is closer to a workbench for the day.
`digest/` is the deeply processed layer. It stores memory nodes that can be reused over time, such as user preferences,
project background, procedural experience, conceptual knowledge, and decision precedents. Digest should not merely copy
daily. It should merge recurring facts, methods, and relationships into more stable descriptions.
`metadata/` is the system index layer. It stores runtime state such as the file catalog, chunk index, and graph snapshots.
Users normally do not edit this content manually. The actual editing surface is `daily/`, `digest/`, and, when necessary,
`metadata/` is the system index layer. It stores runtime state such as the file catalog, chunk index, and graph
snapshots. Users normally do not edit this content manually. The actual editing surface is `daily/`, `digest/`, and,
when necessary,
`resource/`.
These layers let ReMe preserve both the original situation and its abstraction: daily reconstructs what happened, while
@ -73,21 +79,23 @@ The corresponding automatic flows are [Auto Memory](./auto_memory.md), [Auto Res
```text
<workspace_dir>/
├── metadata/ # system index layer; persistent indexes, graph, catalogs; not a manual editing surface
├── session/ # raw input layer; original conversations and agent sessions
├── session/ # source-record layer; source conversations
│ ├── dialog/
│ │ └── <session_id>.jsonl # conversation messages saved by auto_memory
│ ├── agentscope/
│ │ └── <session_id>.jsonl
│ │ └── <session_id>.jsonl # source messages saved by auto_memory
│ └── claude_code/
│ └── <session_id>.jsonl
├── resource/ # raw input layer; original external material
│ └── <session_id>.jsonl # ReMe copy used by auto_memory_cc
├── mem_session/ # generated Agent wrapper sessions/config, not user memory
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # source-record layer; original external material
│ ├── <resource>.<ext> # root-level input uses today's date
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
│ └── <resource>.<ext> # dated input uses the directory date
├── daily/ # lightly processed layer; facts, conversation summaries, and resource interpretations by date
│ ├── YYYY-MM-DD.md # index page for the day
│ └── YYYY-MM-DD/
│ ├── <session_id>.md # daily note distilled from a conversation
│ ├── <resource_stem>.md # daily note distilled from a resource
│ ├── <generated_name>.md # topic-named conversation or resource card
│ └── interests.yaml # proactive interest topics generated by auto_dream
└── digest/ # deeply processed layer; reusable personal facts, procedures, and knowledge nodes
├── personal/
@ -103,17 +111,21 @@ Typical flows:
```text
conversation
-> session/dialog/<session_id>.jsonl
-> daily/YYYY-MM-DD/<session_id>.md
-> daily/YYYY-MM-DD/<generated_name>.md
-> digest/personal | digest/procedure | digest/wiki
external resource
-> resource/YYYY-MM-DD/<resource>.<ext>
-> daily/YYYY-MM-DD/<resource_stem>.md
-> resource/[YYYY-MM-DD/]<resource>.<ext>
-> daily/YYYY-MM-DD/<generated_name>.md
-> digest/wiki | digest/procedure
```
The first two steps focus on recording and organizing; the final step focuses on long-term distillation. `auto_memory` and
`auto_resource` generate daily notes from raw input, and `auto_dream` extracts and integrates digest nodes from daily.
The first two steps focus on recording and organizing; the final step focuses on long-term distillation. `auto_memory`
and
`auto_resource` generate daily notes from source input, and `auto_dream` extracts and integrates digest nodes from
daily. The generated daily filename comes from validated frontmatter `name`; `session_id`, `source_conversation`, and
`source_resource`
provide stable provenance and lookup identity instead of determining the filename.
## Markdown Format
@ -146,8 +158,8 @@ source_conversation: [[session/dialog/abc.jsonl]]
---
```
The current code recognizes `name` and `description` explicitly. Other fields are preserved as additional metadata. The write
interface merges `name`, `description`, and `metadata` into frontmatter.
The current code recognizes `name` and `description` explicitly. Other fields are preserved as additional metadata. The
write interface merges `name`, `description`, and `metadata` into frontmatter.
Treat frontmatter as a node-level summary and the body as evidence, explanation, and relationships. For example:
@ -165,7 +177,7 @@ Apply this preference when following [[digest/procedure/technical-documentation.
## Sources
- [[daily/2026-06-20/session-a.md]]
This preference was recorded in [[daily/2026-06-20/documentation-style.md]], which captures the user's repeated guidance.
```
This has three benefits:
@ -174,8 +186,8 @@ This has three benefits:
2. The body can carry fuller facts, conditions, counterexamples, and sources.
3. Ordinary wikilinks can be parsed by the graph and maintained when files move.
Frontmatter is best for stable, short, structured fields; the body is best for explanations meant for people. Do not put long
body text into YAML fields.
Frontmatter is best for stable, short, structured fields; the body is best for explanations meant for people. Do not put
long body text into YAML fields.
### Wikilink
@ -197,13 +209,13 @@ ReMe wikilinks use **literal path semantics**:
ReMe does not append `.md` automatically, search by filename, or automatically resolve folder notes. Use complete
workspace-relative paths with their extensions.
Ordinary Markdown links such as `[label](../wiki/example.md)` do not create `FileLink` edges and are not rewritten by move or
retarget operations.
Ordinary Markdown links such as `[label](../wiki/example.md)` do not create `FileLink` edges and are not rewritten by
move or retarget operations.
Anchors such as `#L9`, `#L9-L10`, and `#L9-L10,L15-L20` remain ordinary `target_anchor` strings in the graph. The graph
parser does not validate line-anchor syntax, so values such as `#L0`, `#L10-L9`, and `#L9,` are also stored. The `read` job
does not interpret an anchor appended to `path`; use the separate 1-based, inclusive `start_line` and `end_line` arguments to
read a range, for example `read(path="digest/wiki/solar.md", start_line=9, end_line=10)`.
parser does not validate line-anchor syntax, so values such as `#L0`, `#L10-L9`, and `#L9,` are also stored. The `read`
job does not interpret an anchor appended to `path`; use the separate 1-based, inclusive `start_line` and `end_line`
arguments to read a range, for example `read(path="digest/wiki/solar.md", start_line=9, end_line=10)`.
Wikilinks support these behaviors:
@ -224,9 +236,8 @@ FileLink
```
Older documents containing wrappers such as `related:: [[path]]`,
`- related:: [[path]]`, or `[related:: [[path]]]` remain readable. ReMe
ignores the surrounding text and indexes the inner `[[path]]` as an ordinary
link. After upgrading from a version that stored typed links, run `reme reindex`
`- related:: [[path]]`, or `[related:: [[path]]]` remain readable. ReMe ignores the surrounding text and indexes the
inner `[[path]]` as an ordinary link. After upgrading from a version that stored typed links, run `reme reindex`
once to rebuild the derived graph without the removed relationship field.
### Sources and Relationships
@ -238,8 +249,8 @@ A Sources section records where a long-term memory came from:
```markdown
## Sources
- [[daily/2026-06-20/session-a.md]]
- [[resource/2026-06-20/report.pdf]]
The preference was observed in [[daily/2026-06-20/documentation-style.md]], and the supporting report evidence is retained in
[[resource/2026-06-20/report.pdf]].
```
A conceptual relationship link explains which other long-term memories relate to the node. Weave it into natural prose:
@ -255,16 +266,16 @@ This analysis extends [[digest/wiki/solar-supply-chain.md]], follows
Because memory is stored as files, users can edit the workspace directly, while agents can read and write the same files
through ReMe's file tools. Both follow the same conventions:
| Operation | Guidance |
|---|---|
| Add memory | Write to the appropriate directory, use frontmatter for Markdown, and prefer complete workspace-relative wikilinks. |
| Edit a body | Preserve existing sources and important wikilinks. When correcting an old conclusion, explain how the new material changes the previous judgment. |
| Move a file | ReMe's move tool rewrites old paths in inbound edges by default. After a manual move, inspect inbound links again. |
| Operation | Guidance |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
| Add memory | Write to the appropriate directory, use frontmatter for Markdown, and prefer complete workspace-relative wikilinks. |
| Edit a body | Preserve existing sources and important wikilinks. When correcting an old conclusion, explain how the new material changes the previous judgment. |
| Move a file | ReMe's move tool rewrites old paths in inbound edges by default. After a manual move, inspect inbound links again. |
| Delete a file | Check inbound links first. ReMe's delete tool returns source files that still point to the target, making dangling references easier to clean up. |
| Edit metadata | Use frontmatter for short fields. When the body changes substantially, update `description` as well. |
| Edit metadata | Use frontmatter for short fields. When the body changes substantially, update `description` as well. |
A practical rule is: **an agent may rewrite the wording, but it must not lose evidence edges**. In particular, Sources entries
and existing digest-to-digest wikilinks are the basis for traceable and extensible long-term memory.
A practical rule is: **an agent may rewrite the wording, but it must not lose evidence edges**. In particular, Sources
entries and existing digest-to-digest wikilinks are the basis for traceable and extensible long-term memory.
## Path Semantics
@ -272,7 +283,7 @@ All file tools and wikilinks use workspace-relative paths as their basic unit:
```text
digest/wiki/solar.md
daily/2026-06-20/session-a.md
daily/2026-06-20/documentation-style.md
resource/2026-06-20/report.pdf
```
@ -284,16 +295,16 @@ Recommended practices:
1. Include `.md` when linking a Markdown file.
2. Use the complete source path when linking from digest to daily or resource.
3. Rename or move files through ReMe's move tool whenever possible to avoid stale paths.
4. Put external source material under `resource/YYYY-MM-DD/...` and long-term abstractions under `digest/...`. Do not put
raw source material directly into digest.
4. Put external source material under `resource/YYYY-MM-DD/...` and long-term abstractions under `digest/...`. Do not
put raw source material directly into digest.
Explicit path semantics sacrifice a little convenience when writing by hand, but provide predictability, portability, and
automatic maintainability.
Explicit path semantics sacrifice a little convenience when writing by hand, but provide predictability, portability,
and automatic maintainability.
## Memory Chunking
Memory chunking divides a file into retrievable fragments. ReMe does not split Markdown at fixed lengths by default; it tries
to preserve semantic structure.
Memory chunking divides a file into retrievable fragments. ReMe does not split Markdown at fixed lengths by default; it
tries to preserve semantic structure.
This section explains how files become retrieval chunks. For index updates, BM25, vector recall, and link expansion, see
[Memory Search](./memory_search.md).
@ -308,8 +319,8 @@ Document
chunk 1 | chunk 2 | chunk 3 | ...
```
This is simple, but it can cut headings, tables, code blocks, lists, and `[[wikilinks]]` in the middle. After a match, the
agent often sees only an isolated fragment without knowing its section or relationship to other memory nodes.
This is simple, but it can cut headings, tables, code blocks, lists, and `[[wikilinks]]` in the middle. After a match,
the agent often sees only an isolated fragment without knowing its section or relationship to other memory nodes.
ReMe chunking is closer to splitting memory by file structure:
@ -375,5 +386,10 @@ Matched body fragment
This lets the agent see not only an isolated paragraph but also its structural position in the source file.
Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For Markdown,
the chunker also avoids cutting `[[wikilinks]]` in the middle.
Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For
Markdown, the chunker also avoids cutting `[[wikilinks]]` in the middle.
`DefaultFileChunker` and `MarkdownFileChunker` decode files with their configured `encoding` and normalize platform
newlines to LF before indexing. Their default `invalid_encoding_policy: replace` keeps decodable content searchable
when a source contains invalid bytes, without modifying the source file. Set `invalid_encoding_policy: strict` on a
chunker component to reject such files instead.

View file

@ -1,8 +1,10 @@
# Memory Search
Memory Search is ReMe's memory retrieval entry point. It continuously builds files under `daily/`, `digest/`, and `resource/`
into a searchable chunk index and wikilink graph. At query time, it first recalls the most relevant fragments and then expands
context along the bidirectional links of the files containing those fragments.
Memory Search is ReMe's memory retrieval entry point. The default background loop continuously builds Markdown under
`daily/` and `digest/` into a searchable chunk index and wikilink graph. At query time, it first recalls the most
relevant fragments and then expands context along the bidirectional links of the files containing those fragments.
`reme reindex` has a broader rebuild scope that also scans `resource/` and JSONL; it is intentionally different from the
live watcher.
<p align="center">
<img src="../figure/auto-index-and-memory-search.svg" alt="ReMe Auto Index and Memory Search indexing, recall, fusion, and link expansion" width="92%">
@ -21,14 +23,17 @@ workspace files
## What It Searches
The default `index_update_loop` watches three memory directories:
The default `index_update_loop` watches two memory directories:
- `daily_dir`: daily working memory and session memory cards generated by Auto Memory.
- `digest_dir`: long-term distilled digest nodes.
- `resource_dir`: external resources or imported material.
The default suffixes are `md` and `jsonl`. Markdown uses the `markdown` chunker, which parses frontmatter, heading structure,
and `[[wikilinks]]`. JSONL uses the `default` chunker and creates overlapping chunks by byte size.
The live watcher handles only the `md` suffix. A separate `resource_watch_loop` watches `resource_dir`, and Auto
Resource turns those inputs into daily cards that enter the live index. When `reme reindex` is run manually, its
configuration scans
`daily_dir`, `digest_dir`, and `resource_dir` for `md` and `jsonl`; Markdown uses the `markdown` chunker and JSONL uses
the
`jsonl` chunker.
## How the Index Is Built
@ -39,8 +44,8 @@ The background Job `index_update_loop` maintains the index using configuration f
```yaml
index_update_loop:
backend: background
watch_dirs: [ daily_dir, digest_dir, resource_dir ]
watch_suffixes: [ md, jsonl ]
watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [md]
steps:
- backend: init_changes_step
monitor_type: file_store
@ -54,9 +59,9 @@ index_update_loop:
`FileNode.st_mtime` values already stored in `file_store`, calculates added, modified, and deleted changes, and passes
`context["changes"]` to `update_index_step`.
While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same directories,
groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the same path into one
stable batch of changes.
While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same
directories, groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the
same path into one stable batch of changes.
`update_index_step` performs the actual index writes:
@ -66,13 +71,14 @@ stable batch of changes.
4. For a deleted file, remove its records from `file_store`, `keyword_index`, and `file_graph`.
5. When changes exist, dump state to `metadata/` so it can be restored on the next startup.
The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into `FileNode`, `FileChunk`, and `FileLink`
The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into `FileNode`, `FileChunk`, and
`FileLink`
objects. For detailed chunking rules, see [Memory as File](./memory_as_file.md#memory-chunking).
### Index Optimization
Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes;
too many tombstones degrade search performance. An idle-time optimization mechanism is built in—the `optimize_index_cron`
Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes; too
many tombstones degrade search performance. An idle-time optimization mechanism is built in—the `optimize_index_cron`
scheduled job compacts tombstones and rebuilds indexes during off-peak hours:
```yaml
@ -100,17 +106,25 @@ file_store:
It combines three kinds of capability:
| Part | Default state | Purpose |
|---|---|---|
| `file_chunks` | Enabled | Store `FileChunk` text, line numbers, scores, and optional embeddings. |
| `keyword_index.default` | Enabled | BM25 inverted index where chunk ID is the document ID. |
| `file_graph.default` | Enabled | Store `FileNode` objects and wikilink edges. |
| `embedding_store` | Disabled | When enabled, generate embeddings for chunks and support vector recall. |
| Part | Default state | Purpose |
|-------------------------|---------------|-------------------------------------------------------------------------|
| `file_chunks` | Enabled | Store `FileChunk` text, line numbers, scores, and optional embeddings. |
| `keyword_index.default` | Enabled | BM25 inverted index where chunk ID is the document ID. |
| `file_graph.default` | Enabled | Store `FileNode` objects and wikilink edges. |
| `embedding_store` | Disabled | When enabled, generate embeddings for chunks and support vector recall. |
Out of the box, search therefore uses primarily BM25 plus link expansion. After setting `embedding_store: default`,
`SearchStep` runs vector and keyword recall together. Additionally, switching the `file_store` `backend` from `local` to
`faiss` upgrades vector retrieval from a linear scan to a FAISS HNSW index, offering faster recall at scale.
The embedding store accepts `health_check_timeout` for its startup probe. A temporary failure skips the current vector
backfill while keeping BM25 available; a later successful provider request resumes the missing-vector backfill
automatically.
Embedded integrations that have already verified a provider can call `resume_embedding(verified=True)`. When changing
the embedding vector space, pass `rebuild=True`; persisted vectors are invalidated before a serial background rebuild,
and vector search remains unavailable until the rebuilt vectors are safely persisted.
## How to Search
The `search` Job is also configured in `default.yaml`:
@ -123,10 +137,12 @@ search:
query: string
limit: integer
min_score: number
start_date: string
end_date: string
steps:
- backend: search_step
vector_weight: 0.7
candidate_multiplier: 3.0
candidate_multiplier: 5.0
expand_links: true
max_links_per_direction: 10
```
@ -137,11 +153,17 @@ Call it with:
reme search query="recent discussions about indexing" limit=5
```
Use `start_date` and `end_date` for inclusive `YYYY-MM-DD` filtering:
```bash
reme search query="index regression" start_date=2026-06-01 end_date=2026-06-20 limit=10
```
`search_step` executes in this order:
```mermaid
flowchart LR
A["query + limit"] --> B["candidates = limit * candidate_multiplier"]
A["query + limit"] --> B["candidates = min(200, limit * candidate_multiplier)"]
B --> C["file_store.vector_search(...)"]
B --> D["file_store.keyword_search(...)"]
C --> E["RRF fusion"]
@ -152,17 +174,17 @@ flowchart LR
H --> I["Response.answer + metadata"]
```
If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking is
returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores directly; it
compares ranks in the two result lists:
If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking
is returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores
directly; it compares ranks in the two result lists:
```text
fused_score = vector_weight / (60 + vector_rank)
+ keyword_weight / (60 + keyword_rank)
```
The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can still
promote chunks with exact term matches.
The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can
still promote chunks with exact term matches.
## How BM25 Works
@ -174,12 +196,14 @@ promote chunks with exact term matches.
- The inverted index records which chunks contain each token and its term frequency within each chunk.
- A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs.
When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old `chunk_ids`
When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old
`chunk_ids`
and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize.
## Progressive Expansion
"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three layers:
"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three
layers:
1. Chunk recall: return only the `limit` most relevant text fragments.
2. File location: each result includes `path:start_line-end_line`. Pass the path and line bounds separately as `path`,
@ -198,8 +222,8 @@ matched chunk
-> render neighbor path, name, description, and anchor
```
This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory connects
to. If a result is worth pursuing, use `read path=...` to open the source or
This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory
connects to. If a result is worth pursuing, use `read path=...` to open the source or
`traverse path=... depth=2` to continue along the wikilink graph.
## Return Format
@ -213,7 +237,7 @@ to. If a result is worth pursuing, use `read path=...` to open the source or
Typical text structure:
```text
========== daily/2026-06-20/session-a.md:12-28 [score=0.0317 keyword=4.8120] ==========
========== daily/2026-06-20/retrieval-regression.md:12-28 [score=0.0317 keyword=4.8120] ==========
...matched memory fragment...
outlinks (2):
-> digest/indexing.md name="Indexing" description="..."
@ -221,5 +245,5 @@ Typical text structure:
<- daily/2026-06-19.md name="..."
```
`counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned. With
embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`.
`counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned.
With embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`.

View file

@ -0,0 +1,225 @@
# Plugin Management
ReMe plugins are ordinary Python distributions discovered through the `reme.plugins` entry-point group. Installing a
plugin makes it available to the current Python environment; it does not enable the plugin in every ReMe application.
Keep these two operations separate:
```text
reme plugins install ... install a package into the current Python environment
plugins: [auto-fin] enable an installed plugin for one Application
```
Plugin package management is local-only. It does not run through a ReMe HTTP or MCP service and never edits application
configuration files automatically.
A typical plugin workflow has three stages:
1. Install ReMe and the plugin distribution.
2. Configure the plugin's runtime environment as described in the
[ReMe model-configuration guide](../../README.md#optional-model-configuration).
3. Start an Application with the plugin explicitly enabled, for example
`reme start plugins='["auto-fin"]'`.
## List installed plugins
```bash
reme plugins list
```
The table shows the plugin entry-point name, Python distribution, version, and plugin contract:
```text
PLUGIN DISTRIBUTION VERSION FORMAT
-------- ------------- ------- --------
auto-fin reme-auto-fin X.Y.Z manifest
```
`manifest` plugins use the current package-level `plugin.yaml` contract. `legacy` plugins use the compatible Python
descriptor contract.
A manifest separates backend registration from application configuration:
```yaml
backends:
example_step: example_plugin.steps:ExampleStep
application_defaults:
jobs:
example:
backend: base
steps:
- backend: example_step
```
`application_defaults` is a partial `ApplicationConfig`. It is kept below the manifest's `backends` namespace because
backend import declarations are part of plugin discovery and are not application configuration.
Use JSON when another local tool needs structured output:
```bash
reme plugins list --json
```
To compare installed plugins with one application config:
```bash
reme plugins list --config default
```
The optional `ENABLED` column reflects only the `plugins` list resolved from that config. A command-line override used
by another running process is not a global enable state.
## Install a plugin package
Install a published distribution:
```bash
reme plugins install reme-auto-fin
```
Install or upgrade a pinned version:
```bash
reme plugins install 'reme-auto-fin==X.Y.Z'
reme plugins install reme-auto-fin --upgrade
```
Install a local plugin project:
```bash
reme plugins install ./plugins/auto-fin
```
Use editable mode while developing it:
```bash
reme plugins install ./plugins/auto-fin --editable
```
ReMe invokes pip through the same Python interpreter that runs the `reme` command. Pip remains responsible for package
resolution, downloads, dependency changes, and build execution. Install only packages and local projects you trust.
After installation, confirm the discovered plugin name:
```bash
reme plugins list
reme plugins validate auto-fin
```
## Inspect a plugin
```bash
reme plugins show auto-fin
```
For a manifest plugin, the result includes its registered backend names and default Job names. JSON output is also
available:
```bash
reme plugins show auto-fin --json
```
`show` identifies the package contract without constructing a ReMe Application.
## Validate a plugin
Validate an installed plugin:
```bash
reme plugins validate auto-fin
```
Validate a local project before installation:
```bash
reme plugins validate ./plugins/auto-fin
```
Validation checks the entry point, `plugin.yaml`, backend imports and component types, registry collisions, merged
`application_defaults`, and the resulting `ApplicationConfig`. Validation imports plugin backend modules, so run it
only for trusted code.
## Enable a plugin in a service
Installation alone does not load plugin code into an Application. Enable plugins explicitly in configuration:
```yaml
plugins:
- auto-fin
```
Or add them for one service launch:
```bash
reme start plugins='["auto-fin"]'
```
When `config` is omitted, ReMe loads `default.yaml`. The plugin's `application_defaults` are merged below that config,
so explicit config values and CLI overrides win. This mapping is an `ApplicationConfig` fragment, not a separate
configuration schema. The plugin backends are registered only in that Application's local registry.
After the default HTTP service starts, access plugin Jobs through ReMe's CLI client or HTTP:
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
When the application uses an MCP service, service-enabled plugin Jobs appear as MCP tools instead.
Custom application configs must provide the plugin's runtime dependencies, including an `agent_wrapper.default` and
the `search` and `read` Jobs used by Auto Fin.
## Uninstall a plugin
Use the plugin entry-point name, not necessarily the distribution name:
```bash
reme plugins uninstall auto-fin
```
Skip pip's confirmation prompt when needed:
```bash
reme plugins uninstall auto-fin --yes
```
ReMe resolves `auto-fin` to the distribution that provides it, such as `reme-auto-fin`. If one distribution provides
multiple plugin entry points, the command lists the other plugins that will also be removed.
Uninstallation does not rewrite user configuration. Remove the plugin from relevant `plugins` lists yourself;
otherwise the next Application startup fails explicitly because the configured plugin is no longer installed. Restart
already-running ReMe processes after installing, upgrading, or uninstalling packages.
## Troubleshooting
### Plugin is installed but unavailable
Check that the `reme` command and pip package share one Python interpreter:
```bash
reme plugins list
python -c 'import sys; print(sys.executable)'
```
Using `reme plugins install` avoids the most common interpreter mismatch because it runs `python -m pip` with ReMe's
own interpreter.
### Plugin is installed but not loaded
Add its entry-point name to the Application's `plugins` list. ReMe intentionally has no global enable/disable state.
### Startup reports that the plugin is not installed
The active config still enables a missing plugin. Reinstall it or remove the corresponding name from `plugins`.
### Changes are not visible in a running service
Plugin discovery and backend registration happen during Application construction. Restart the service after changing
installed packages.

View file

@ -1,17 +1,17 @@
# Proactive
`proactive` is ReMe's interface for reading proactive memory. It does not reanalyze daily notes or call an LLM. It only reads
the current day's interest topics written by `auto_dream`:
`proactive` is ReMe's interface for reading proactive memory. It does not reanalyze daily notes or call an LLM. It only
reads the current day's interest topics written by `auto_dream`:
```text
daily/<date>/interests.yaml
```
A host agent can use it to learn "what is worth proactive attention today," then decide whether to remind the user, ask a
follow-up question, recommend a next step, or produce a proactive insight.
A host agent can use it to learn "what is worth proactive attention today," then decide whether to remind the user, ask
a follow-up question, recommend a next step, or produce a proactive insight.
`interests.yaml` is generated by the Topics stage of [Auto Dream](./auto_dream.md). `proactive` only reads and exposes the
result.
`interests.yaml` is generated by the Topics stage of [Auto Dream](./auto_dream.md). `proactive` only reads and exposes
the result.
## Configuration
@ -34,10 +34,10 @@ proactive:
Parameters:
| Parameter | Purpose |
|---|---|
| `date` | Date to read in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `include_content` | Whether to return the raw YAML in the answer and metadata. Defaults to `true`. |
| Parameter | Purpose |
|-------------------|-------------------------------------------------------------------------------------------|
| `date` | Date to read in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `include_content` | Whether to return the raw YAML in the answer and metadata. Defaults to `true`. |
## Input Contract
@ -67,15 +67,15 @@ When the file is read successfully, `proactive_step` returns `summary` and `topi
`include_content=true`, the answer also contains `content`. The same result fields remain available in standard response
metadata:
| Field | Description |
|---|---|
| `date` | The date actually read. |
| `path` | `daily/<date>/interests.yaml`. |
| `topics` | Parsed topic list. |
| Field | Description |
|-----------|------------------------------------------------------|
| `date` | The date actually read. |
| `path` | `daily/<date>/interests.yaml`. |
| `topics` | Parsed topic list. |
| `content` | Raw YAML; returned only when `include_content=true`. |
| `skipped` | `true` when the file does not exist. |
| `error` | Read or parse error. |
| `summary` | Short summary. |
| `skipped` | `true` when the file does not exist. |
| `error` | Read or parse error. |
| `summary` | Short summary. |
When the file exists and parses successfully, the answer is structured data. For example:
@ -135,21 +135,21 @@ daily notes
The responsibilities are divided as follows. For the complete Extract, Integrate, Topics, and Finish flow, see
[Auto Dream](./auto_dream.md):
| Module | Responsibility |
|---|---|
| `dream_extract_step` | Extract topic candidates from changed daily inputs. |
| `dream_topics_step` | Deduplicate, select, and write `interests.yaml`. |
| `proactive_step` | Read `interests.yaml` and expose it to the host agent. |
| Module | Responsibility |
|----------------------|--------------------------------------------------------|
| `dream_extract_step` | Extract topic candidates from changed daily inputs. |
| `dream_topics_step` | Deduplicate, select, and write `interests.yaml`. |
| `proactive_step` | Read `interests.yaml` and expose it to the host agent. |
`proactive` does not modify files, update a catalog, or decide whether the user should be interrupted. It only provides the
day's topic material. The caller's product policy determines whether, when, and in what tone to push it to the user.
`proactive` does not modify files, update a catalog, or decide whether the user should be interrupted. It only provides
the day's topic material. The caller's product policy determines whether, when, and in what tone to push it to the user.
## Failure Modes
| Scenario | Behavior |
|---|---|
| `interests.yaml` does not exist | `success=true`, `skipped=true`, `topics=[]`. |
| YAML cannot be read or parsed | `success=false`; the answer contains an error summary. |
| YAML exists but has no valid topics | `success=true`, `topics=[]`. |
| Scenario | Behavior |
|-------------------------------------|--------------------------------------------------------|
| `interests.yaml` does not exist | `success=true`, `skipped=true`, `topics=[]`. |
| YAML cannot be read or parsed | `success=false`; the answer contains an error summary. |
| YAML exists but has no valid topics | `success=true`, `topics=[]`. |
Callers should therefore check `success` first, then `skipped`, and finally whether `topics` is empty.

View file

@ -15,11 +15,17 @@ Install from source:
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"
pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
```
Installing the `core` extra is recommended. The current code imports the AgentScope wrapper, and self-evolving memory also
depends on it.
The static build step requires Node.js 22.13 or newer and makes Studio available when running ReMe from the source tree.
Installing the `core` extra is recommended. The current code imports the AgentScope wrapper, and self-evolving memory
also depends on it.
To use agent workflows such as `auto_memory`, `auto_resource`, and `auto_dream`, configure an LLM:
@ -51,10 +57,16 @@ reme start service.port=8181
```bash
reme version
reme health_check
reme list
reme help
```
`reme list` lists server actions. Ordinary commands invoke server Jobs over HTTP.
`reme help` lists server actions. Ordinary commands invoke server Jobs over HTTP.
The base `reme-ai` package does not include frontend assets. Install `reme-ai[web]` or `reme-ai[core]`, then open
<http://127.0.0.1:2333/> for ReMe Studio. It uses the same service to
browse, edit, and search the workspace and inspect the digest wikilink graph. Disable it with
`service.web_enabled=false`, or provide a custom build with `service.web_static_dir` / `REME_WEB_STATIC_DIR`. The Job
API still starts if no web build is found.
---
@ -65,7 +77,8 @@ The default workspace is `.reme/` under the current directory. It is created aut
```text
.reme/
├── metadata/ # persistent indexes, graph, catalogs, and related state
├── session/ # agent sessions and original conversations
├── session/ # source conversation records
├── mem_session/ # generated Agent wrapper sessions/config
├── resource/ # external resources
├── daily/ # daily notes
└── digest/ # long-term memory
@ -91,12 +104,13 @@ reme write \
description="Example memory for the quick start" \
content="# Quick Start Demo
ReMe indexes Markdown under the daily, digest, and resource directories.
The default live watcher indexes Markdown under the daily and digest directories.
Related link: [[digest/wiki/search-demo.md]]"
```
`path` is relative to the workspace. A missing suffix is automatically completed with `.md`. For Markdown files, `name` and
`path` is relative to the workspace. A missing suffix is automatically completed with `.md`. For Markdown files, `name`
and
`description` are written to frontmatter.
The background watcher builds the index automatically. You can also rebuild it manually:
@ -117,8 +131,8 @@ Read:
reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20
```
With the default configuration, retrieval is primarily BM25 plus wikilink graph expansion. Vector retrieval is supported by
the code, but the embedding store is disabled by default. For the full retrieval flow, see
With the default configuration, retrieval is primarily BM25 plus wikilink graph expansion. Vector retrieval is supported
by the code, but the embedding store is disabled by default. For the full retrieval flow, see
[Memory Search](./memory_search.md).
---
@ -132,7 +146,13 @@ reme frontmatter_read path=digest/wiki/quick-start-demo
reme frontmatter_update path=digest/wiki/quick-start-demo metadata='{"tags":["demo"]}'
```
The name `list` is used by the CLI to list actions, so the file-listing Job must be called over HTTP:
The file-listing Job can be called directly from the CLI:
```bash
reme list path=digest recursive=true limit=50
```
The equivalent HTTP call is:
```bash
curl -s http://127.0.0.1:2333/list \
@ -161,7 +181,8 @@ reme auto_memory \
memory_hint="Record the user's preference"
```
After placing external material under `resource/YYYY-MM-DD/`, the default background task watches
After placing external material under `resource/YYYY-MM-DD/` or directly under `resource/`, the default background task
watches
`md/txt/json/jsonl/csv/yaml/html`. You can also trigger processing manually:
```bash
@ -175,7 +196,8 @@ reme auto_dream date=2026-06-20
reme proactive date=2026-06-20
```
These flows require a working LLM. Without an LLM configuration, start with basic capabilities such as `write`, `read`, and
These flows require a working LLM. Without an LLM configuration, start with basic capabilities such as `write`, `read`,
and
`search`.
For more detail, see [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md),

339
docs/en/reme-blog.md Normal file
View file

@ -0,0 +1,339 @@
# ReMe: A Personal Knowledge Base That Keeps Growing After Every Conversation
Every day, we talk with AI.
It helps us analyze projects, read papers, and troubleshoot problems. We also tell it about our preferences, plans, and ideas we have not fully worked out yet.
But most of the time, when a conversation ends, its value is locked away in the chat history. The next time we open a new window, the AI may remember a conclusion but not where it came from. It may find an old conversation but fail to connect it with materials we read or decisions we made later.
Useful long-term memory should do more than preserve what once happened. It should keep organizing information, building connections, and bringing past knowledge back into future reasoning when needed.
That is exactly what ReMe sets out to do.
> **ReMe is a local-first, self-evolving personal knowledge base for AI agents. It continuously turns conversations and resources into readable, editable, searchable, and interconnected Markdown memories, while surfacing threads worth following.**
GitHub: [https://github.com/agentscope-ai/ReMe](https://github.com/agentscope-ai/ReMe)
Documentation: [https://reme.agentscope.io](https://reme.agentscope.io)
<p align="center">
<img src="../figure/reme-blog/reme-blog-cover-benchmark.png" alt="ReMe self-evolving personal knowledge base and public benchmark results" width="100%">
</p>
## A Memory Loop That Keeps Growing
<p align="center">
<img src="../figure/reme-blog/reme-blog-architecture.svg" alt="ReMe self-evolving memory loop" width="100%">
</p>
ReMe is not another chatbot, nor does it try to replace the agents you already use. It is a local memory layer that agents such as QwenPaw, OpenClaw, Hermes, and Claude Code can share.
Built around a set of ordinary files, it does four things:
- Auto Memory extracts information worth keeping from conversations;
- Auto Resource turns external materials into traceable memories;
- Auto Dream consolidates daily memories into long-term knowledge;
- Index, Search, and Proactive bring old memories back into new tasks.
Together, they form a `capture → index → consolidate → recall` loop:
- Conversations and external resources are preserved first;
- Valuable information is organized into daily memories;
- Scattered events are consolidated into long-term knowledge nodes;
- Search, knowledge links, and interest discovery bring old memories back into future reasoning.
Most importantly, this loop is centered not on an opaque database, but on files owned by the user. Indexes, graphs, and caches are merely derived state that can always be rebuilt.
## Memory as File: Your Memories Are Your Files
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-as-file.svg" alt="ReMe Memory as File" width="100%">
</p>
ReMe's core design is called **Memory as File, File as Memory.**
“Memory as File” means long-term memories are not hidden inside a product. They live in Markdown, JSONL, YAML, and original resource files within your workspace. You can open them directly in VS Code, Typora, or Obsidian, and back them up or move them with Git, cloud storage, or your own synchronization setup.
“File as Memory” means each file is more than plain text. With YAML frontmatter, section structure, line ranges, and Wikilinks, it becomes a memory node that can be indexed, connected, and continuously evolved.
For example, a long-term memory about writing preferences might look like this:
```markdown
---
name: "User preference: technical writing style"
description: Prefers stating the problem and outcome first, followed by technical details and examples.
kind: preference
---
The user wants technical articles to have a clear narrative and avoid unnecessary jargon.
When writing an article, refer to [[digest/procedure/Technical content writing process.md]].
## Sources
This preference was observed in [[daily/2026-08-07/content-discussion.md]], which records the user's writing guidance.
```
Months later, even if you have forgotten the conversation, the agent can still read the preference, find the related process, and follow `Sources` back to the original context.
This is also the key difference between ReMe and “black-box memory”: agents can organize memories, but users always retain the right to inspect, correct, move, and delete them.
## Auto Memory: Turning Conversations into a Daily Journal
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-memory.svg" alt="ReMe Auto Memory turns conversations into daily memories" width="100%">
</p>
A great deal of valuable information does not begin with “please remember this.”
For example, you might say in a conversation:
> “Let's not refactor the login module this week. We can do it after the customer demo. Upgrading dependencies directly caused compatibility issues last time, so let's add regression tests first.”
This short passage contains project status, a time constraint, a lesson from a previous failure, and a next action. Auto Memory extracts these details from the conversation stream and writes them into a daily memory card, while retaining a source conversation record in `session/dialog/`.
```text
session/dialog/project-a.jsonl Source conversation record
daily/2026-08-07/login-refactor-decision.md Content-named memory card
daily/2026-08-07.md Daily index, providing an overview
```
`session_id` remains in the card's frontmatter for stable lookup and provenance; the filename comes from the Agent-generated
topic/event `name`, so it does not have to match the session ID.
The next time the login module comes up, the agent does not need to search through the entire chat history. It can immediately see why the refactor was postponed, what went wrong before, and what should happen next.
It is like having a recorder who is always present—not one that mechanically transcribes every word, but one that organizes what will still matter later.
## Auto Resource: Bringing External Materials into the Same Memory System
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-resource.svg" alt="ReMe Auto Resource turns external materials into traceable personal memories" width="100%">
</p>
Not all valuable information comes from conversations. Research materials, project documents, meeting notes, archived web pages, and structured data may all become part of a personal knowledge base.
Auto Resource provides a general entry point for external materials. After a resource enters `resource/`, ReMe preserves the original and organizes its topics, key facts, and actionable information into daily cards with `source_resource` links. It currently supports text-based resources including Markdown, plain text, JSON, JSONL, CSV, YAML, and HTML.
In other words, Auto Memory builds personal knowledge from conversations, while Auto Resource builds it from non-conversational materials. Both streams flow into the same daily memory layer, where ReMe indexes, consolidates, and retrieves them together.
### Daily Paper: An Example External-Resource Workflow
Daily Paper is an optional plugin built on this file-based memory system. It collects papers from the weekly and monthly Hugging Face Papers rankings, removes items recommended recently, ranks the remaining papers, selects three, saves their PDFs, and generates Chinese paper notes and a briefing that takes about five minutes to read.
Imagine that you regularly follow research on agent memory. Each morning, instead of receiving only three links, you get three detailed notes already saved locally. The briefing points to the original notes through Wikilinks, and each note links back to its PDF. A month later, when you ask, “What recent methods compress long-term memory?”, those materials are already in the same retrieval system. There is no need to search through browser history again.
Daily Paper demonstrates how Auto Resource can be composed into a concrete workflow, but the external-resource pipeline is not limited to papers.
## Auto Dream: Growing Daily Notes into Connected Long-Term Knowledge
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-dream.svg" alt="ReMe Auto Dream extracts, classifies, and consolidates long-term knowledge from daily memories while adding Wikilinks" width="100%">
</p>
As daily notes accumulate, a new problem emerges: the information is all there, but it remains scattered across different dates.
Suppose conversations and external materials give you three pieces of information about the same problem:
- The first time a build hung, clearing the cache did not help;
- A project document later confirmed that insufficient Node.js memory was the root cause;
- A third note added that the issue occurs more often in large TypeScript projects.
By default, Auto Dream looks at the two most recent days ending at the target date and sends only daily files changed since
the previous run to extraction. It merges cross-file evidence for the same abstraction and keeps only the strongest reusable
memories within a default cap of five units, then writes them into three categories of long-term memory:
- `Personal`: preferences, conventions, and constraints specific to a user, team, or project;
- `Procedure`: repeatable processes, methods, and troubleshooting guides;
- `Wiki`: general definitions, principles, observations, and knowledge.
For example, the information above would become `digest/procedure/Troubleshooting frozen frontend builds.md`, which records the triggering conditions, diagnostic sequence, failed attempts, solution, and scope of applicability—instead of simply concatenating several daily notes.
When consolidating each memory unit, Auto Dream first searches existing nodes across `personal`, `procedure`, and `wiki`, distinguishing between the “same abstraction” and “related knowledge.” The same abstraction determines how the target node evolves:
- `CREATE`: no equivalent memory exists, so create a new node;
- `CORROBORATE`: the same conclusion appears again, so add its source and strengthen confidence;
- `REFINE`: new material adds conditions, steps, or details;
- `CORRECT`: new information corrects an earlier conclusion.
Related knowledge is written into the body as Wikilinks during the same consolidation process. This is Auto Link. For example, “Troubleshooting frozen frontend builds” can connect general knowledge, team preferences, and original evidence at once:
```markdown
This issue often occurs in [[digest/wiki/Large TypeScript projects.md]]. When resolving it,
follow the “add regression tests first” convention in [[digest/personal/Team change preferences.md]].
## Sources
The root cause and applicable scenarios were documented in
[[daily/2026-08-07/build-debug.md|Build troubleshooting record]].
```
Knowledge evolves and links are created in the same workflow. Relationships are not invisible edges hidden in a graph database; they are readable, editable content in the files themselves. The files can rebuild the graph—the graph never takes control of the files.
## Memory Index: Turning Ordinary Files into a Searchable Memory Network
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-index.svg" alt="ReMe Memory Index build process" width="100%">
</p>
Markdown is easy for people to read, but if files are merely piled into directories, agents still struggle to find them
quickly. The default live index watches Markdown under `daily/` and `digest/`. A separate resource workflow watches
`resource/` and turns those files into daily cards that enter the same index. For a full rebuild from existing files,
`reme reindex` also scans `resource/` and JSONL.
A Markdown file is parsed into:
- One file node containing file-level information such as its path and frontmatter;
- Multiple semantic chunks split, wherever possible, along the boundaries of headings, paragraphs, lists, and code blocks, while retaining section structure and line numbers;
- Multiple Wikilink edges recording what the file points to and what points back to it.
For retrieval, ReMe can combine three types of signals:
| Retrieval signal | Problem it solves | Example |
|------------------|-------------------|---------|
| BM25 keywords | Exact names, terms, and identifiers must not be missed | “CATL”, “issue #184” |
| Embedding vectors | Semantically similar wording should still match | “build frozen” and “packaging stage not responding” |
| Wikilink graph | Reveal upstream and downstream relationships after finding a node | From “cobalt” to “ternary cathodes” and related research notes |
The default configuration enables BM25 and Wikilink expansion out of the box. Embeddings are optional and participate in vector retrieval only when enabled. Indexes, graphs, and caches are stored in `metadata/`; even if deleted, they can be rebuilt from the user's source files.
## Memory Search: Find the Answer First, Then Expand Relationships Progressively
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-search.svg" alt="ReMe hybrid search and progressive expansion" width="100%">
</p>
Many RAG systems put all Top-K passages into the context at once. This is simple, but it creates two problems: isolated chunks lack context, while expanding every neighbor's full text quickly consumes tokens.
ReMe's hybrid search lets BM25 and optional vector retrieval produce their own candidates, then fuses the rankings with RRF. Instead of directly comparing BM25 scores with cosine similarities—two different scales—RRF combines where each result appears in the two ranked lists.
After retrieval, information expands progressively in three layers:
1. **Start with the matching passage**: return the most relevant chunk, file path, and line numbers;
2. **Then inspect the relationship directory**: show the file's outgoing and incoming links, including only each neighbor's path, name, description, and anchor rather than loading all of its content immediately;
3. **Finally, go deeper as needed**: the agent decides which relationship is genuinely relevant, then reads the original file or continues traversing the graph.
For example, you ask: “What was the name of the book about attention that Alice recommended last time?”
The first step may find a dinner note that says only, “The title contains the word deep.’” The result also shows that the note links to Alice's personal node and is backlinked by reading notes for *Deep Work*.
The agent does not need to load Alice's entire profile, every reading note, and a whole month of journal entries into its context. It only needs to follow the most relevant link and read once more before answering:
> It was *Deep Work*. Alice recommended it at that dinner, and you later read Chapter 3 and left notes.
This resembles human association: first recall a fragment, then follow the trail to recover the full context.
## Proactive: Discovering Needs You Have Not Yet Put into Words
<p align="center">
<img src="../figure/reme-blog/reme-blog-proactive.svg" alt="ReMe Proactive's two-way memory loop" width="100%">
</p>
At this point, ReMe has two input streams that continuously enrich the knowledge base:
- Auto Memory distills personal context from ongoing conversations;
- Auto Resource adds new knowledge from external materials.
Proactive reverses the direction. From accumulated conversations and materials, it discovers topics you have not yet resolved or may want to pursue, along with information you have not noticed but that closely relates to your recent work. These discoveries can then guide what external knowledge enters the system next.
For example, over the past week you separately mentioned that:
- Search results lack sources;
- Long documents lose section context after chunking;
- You want to compare several agent-memory evaluation methods.
Even though you never explicitly said, “Help me systematically study the explainability of memory retrieval,” Auto Dream can distill an interest topic from these daily memories:
```yaml
title: Evaluating the explainability of memory retrieval
reason: The user has recently focused on source tracing, structure-aware chunking, and memory evaluation.
evidence: daily/2026-08-07/search-discussion.md
keywords:
- memory search
- source attribution
- benchmark
```
In a future beta release, after reading this topic through Proactive, a host agent could ask at an appropriate moment, “Would you like me to turn the retrieval issues we discussed recently into an evaluation plan?” It could also use the topic to initiate a user-authorized research workflow. Users would not need to identify and explicitly specify their interests and scope in advance; external resources related to needs implicit in their conversations could continue flowing into the knowledge base.
There is an important boundary: **ReMe's Proactive feature only reads and exposes interest topics. It does not independently access the internet, send notifications, or rewrite the knowledge base.**
It does not guess your interests from nowhere. It surfaces clues that already appeared in your behavior and conversations but have not yet been explicitly stated.
## Performance: Can It Retrieve Information from Very Long Histories?
ReMe uses LongMemEval and BEAM to evaluate memory across multiple sessions and extremely long conversations. During evaluation, the agent can use ReAct to search and read over multiple rounds, generate an answer, and then receive an LLM-as-judge score.
| Benchmark | Setting | Sample size | Agentic score | Primary capabilities tested |
|-----------|---------|------------:|---------------:|-----------------------------|
| **LongMemEval cleaned-s** | **Overall** | **500 questions** | **89.4%** | Cross-session retrieval, knowledge updates, and temporal reasoning |
| BEAM | 100K context | 20 cases / 400 questions | 66.1% | Ten types of long-context memory tasks |
| BEAM | 1M context | 35 cases / 700 questions | 65.0% | Larger-scale, ultra-long conversation settings |
LongMemEval cleaned-s includes single-session facts, preferences, multi-session reasoning, knowledge updates, temporal reasoning, and other question types. ReMe achieved an overall Agentic score of 89.4% across 500 questions. See the [LongMemEval evaluation guide](../../benchmark/longmemeval/README.md) for the complete workflow and breakdown.
BEAM covers ten categories of tasks, including contradiction resolution, event ordering, information extraction, knowledge updates, multi-session reasoning, preference following, summarization, and temporal reasoning. ReMe scored 66.1% on 20 cases / 400 questions with a 100K context and 65.0% on 35 cases / 700 questions with a 1M context. See the [BEAM evaluation guide](../../benchmark/beam/README.md) for the complete setup.
ReMe also uses $\pi$-Bench to evaluate the potential of multi-session reasoning to improve agent proactivity. The PROC score in $\pi$-Bench evaluates capabilities including directly fulfilling hidden intent, guiding targeted clarification, recovering cross-session preferences, reusing cross-session conventions, inferring cross-task dependencies, and advancing underspecified requests. Across five user personas, ReMe Agent achieved an average PROC score of 0.580, outperforming NanoBot by 2.4% under the same test-model configuration. See the [$\pi$-Bench paper](https://arxiv.org/abs/2605.14678) for details about the benchmark.
## Who Is ReMe For?
### People Who Use Agents Directly
If you want AI to understand you continuously throughout a long-term collaboration, ReMe lets your personal assistant stop starting from scratch. Your preferences, project context, important materials, and past decisions accumulate through ongoing conversations and can be found again when they are genuinely relevant.
Researchers, engineers, analysts, and other knowledge workers all fall into this category. Researchers can connect papers, discussions, and reading notes; engineers can preserve project decisions and cross-session troubleshooting experience; analysts can build an evolving record of events, perspectives, and sources. Their professions differ, but they share the same need: AI that can understand the past, accumulate experience, and recover supporting context for the next task.
### Developers Who Build Agents
If you are building an agent, harness, or AI product, ReMe provides an independent long-term memory layer. Through its CLI, HTTP API, MCP Server, or Python API, you can let multiple agents share the same file-based workspace without reimplementing memory extraction, knowledge organization, hybrid retrieval, and relationship expansion for every application.
Files remain the source of truth, while indexes and caches can be rebuilt at any time. This also makes it easier to determine whether an incorrect retrieval originated in the source material, memory consolidation, or the retrieval pipeline.
Ultimately, ReMe is for users and developers who want AI to do more than “answer this one request”: they want it to understand the past, accumulate experience, and know them better over the course of a long-term collaboration. We want agents to understand you better the more you use them—but that understanding should not live in a black box that you cannot inspect, correct, or take with you.
ReMe's answer is straightforward:
- Memories are files owned by the user;
- Original information preserves what happened, while long-term knowledge preserves the abstraction;
- New conversations and resources keep flowing in, while existing knowledge is continuously supplemented and corrected;
- Every conclusion can be traced to relationships and sources through Wikilinks;
- Indexes and caches serve the files rather than replace them;
- Agents can remember, organize, search, and discover, but users always retain ultimate control.
When these mechanisms come together, a personal knowledge base is no longer a repository you must maintain by hand.
It remembers a little more after every conversation and understands a little more after every new resource. At night, it reorganizes scattered experiences. When a future question arises, it follows the connections between pieces of knowledge and brings back the memory you actually need.
That is what ReMe sets out to do: **make memory not only persistent, but continuously evolving.**
## Integrate ReMe with the Agents You Already Use
ReMe can run as a local memory service accessed through its CLI, HTTP API, or MCP Server, or it can be embedded in a host
process through its Python API. The default HTTP service can also serve ReMe Studio at the same address for browsing,
editing, and searching the workspace and inspecting the digest wikilink graph. Different agents can choose the integration
that best fits their runtime environment and share the same local memory workspace when needed.
| Agent | Recommended integration | Capabilities after integration |
|-------|-------------------------|--------------------------------|
| **DeepSeek Harness** | Install [`@agentscope-ai/reme`](../../typescript/README.md#deepseek-harness) as a DSH profile bundle. | Long-term memory guidance, `reme_search`, automatic capture of completed main-agent turns, and scheduled Auto Dream. |
| **OpenClaw** | Install [`@agentscope-ai/reme`](../../typescript/README.md#openclaw) as the native memory plugin. | Recall before conversational root-agent runs, explicit search, automatic turn capture, and scheduled Auto Dream. |
| **QwenPaw** | Embed ReMe in-process through the Python API. | Reuse the host application's lifecycle and model configuration while keeping memories local and file-based. |
| **Claude Code** | Start the streamable HTTP MCP Service and install [`integrations/claude_code/reme`](../../integrations/claude_code/reme). | MCP memory-recall tools, the `reme-memory` skill, and a Stop hook that automatically records sessions. |
| **Hermes** | Start the HTTP Service and install [`integrations/hermes_agent`](../../integrations/hermes_agent). | Automatically recall relevant memories before model calls and invoke `auto_memory` asynchronously after each conversation turn. |
| **Codex and other CLI-capable agents** | Copy or install [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md). | Search, read, and write memories through the CLI; automatic recording requires the host agent to integrate explicitly with the conversation lifecycle. |
For installation, configuration, and integration demos, see the [README](../../README.md).
## Contributions Welcome
ReMe is open source, and we welcome the community's help in making this self-evolving memory system more complete:
- Integrate more agents and harnesses so different runtime environments can use the same user-owned long-term memory;
- Contribute new Auto Resource sources and workflows so papers, news, and other public materials can continuously enter the knowledge base;
- Improve Auto Memory, Auto Dream, Auto Link, hybrid search, and Proactive so memories are organized more accurately, relationships are clearer, and retrieval is more reliable;
- Add application examples, evaluation tasks, and diagnostic reports to help us understand successes and failures in real long-term use;
- Improve documentation and tests, or share your needs and ideas for personal AI memory through an Issue.
Whether it is a code contribution, a use case, a bug report, or a new memory workflow, every contribution can bring ReMe closer to a truly readable, controllable, and continuously evolving personal knowledge base.
Contribution guide: [https://docs.agentscope.io/reme/latest/en/contribution](https://docs.agentscope.io/reme/latest/en/contribution)

View file

@ -54,20 +54,22 @@ session/
daily/
├── 2026-05-18.md
└── 2026-05-18/
├── 2026-05-18-close.md
├── glencore-q3.md
├── cobalt-policy.md
├── cathode-trend.md
├── cobalt-supply-risk.md
├── glencore-output-update.md
├── drc-cobalt-policy.md
├── high-nickel-cathode-trend.md
└── interests.yaml # generated after auto_dream
```
The corresponding flow is:
- `auto_memory` saves the original conversation to `session/dialog/<session_id>.jsonl`, then asks the agent to write
important facts to `daily/<date>/<session_id>.md`.
- `resource_watch_loop` watches text-file changes under `resource/` and triggers `auto_resource_step` to write a
same-named daily note.
- `daily_create` maintains `daily/<date>.md` as the index page for that day.
- `auto_memory` saves a filtered source conversation record to `session/dialog/<session_id>.jsonl`, then asks the agent to write
important facts to a topic-named `daily/<date>/<generated_name>.md`. The note keeps `session_id` and
`source_conversation` in frontmatter for stable lookup and provenance.
- `resource_watch_loop` watches text-file changes under `resource/` and triggers `auto_resource_step` to write a daily note
with `source_resource`. The agent suggests a content-based filename, which the system sanitizes and de-duplicates; it is
not guaranteed to match the resource filename.
- Auto Memory, Auto Resource, and Auto Dream refresh `daily/<date>.md` after writing.
### Day 1 evening: Auto Dream writes to Digest
@ -81,8 +83,8 @@ reme auto_dream date=2026-05-18
```text
dream_extract_step
scan daily/2026-05-18.md and changed files under daily/2026-05-18/
output units and topics
scan the daily window from 2026-05-17 through 2026-05-18 by default
output at most 5 units plus topics from changed files
dream_integrate_step
recall existing digest nodes with node_search for each unit
decide CREATE / CORROBORATE / REFINE / CORRECT
@ -122,7 +124,7 @@ Changes to mining-rights policy in the DRC may affect KFM mine operations and sh
## Sources
- [[daily/2026-05-18/2026-05-18-close.md]]
The production decline and policy risk were recorded in [[daily/2026-05-18/cobalt-supply-risk.md]].
```
Note that wikilinks use literal path semantics. Prefer complete workspace-relative paths with the `.md` extension. ReMe
@ -235,7 +237,7 @@ topics:
reason: The user repeatedly mentioned KFM and cobalt-price risk today
keywords: [cobalt, DRC, CMOC, KFM]
paths:
- daily/2026-05-18/2026-05-18-close.md
- daily/2026-05-18/cobalt-supply-risk.md
```
Call:
@ -321,7 +323,7 @@ The build stalls near the end. CPU usage is low, but memory keeps growing.
## Sources
- [[daily/2026-03-10/build-oom-2026-03-10.md]]
The failed attempts and successful memory adjustment were recorded in [[daily/2026-03-10/build-oom-2026-03-10.md]].
```
Example `digest/personal/code-style.md`:
@ -374,7 +376,7 @@ and upgrading the minification plugin did not help last time.
- `digest/procedure/` stores both "how to do it" and "which paths failed," letting the agent reuse diagnostic experience.
- `digest/personal/` stores user preferences so the agent can follow the same engineering style across sessions.
- The original conversation remains under `session/dialog/`; daily records stay traceable, and digest is only the
- The source conversation record remains under `session/dialog/`; daily records stay traceable, and digest is only the
long-term distilled result.
## Scenario 3: A Personal Second Brain
@ -423,7 +425,7 @@ At lunch on 2026-04-20, Alice recommended [[digest/wiki/deep-work.md]], a book a
## Sources
- [[daily/2026-04-20/lunch-with-alice.md]]
The recommendation was recorded in [[daily/2026-04-20/lunch-with-alice.md]].
```
### An associative recall

View file

@ -1,129 +1,117 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img" aria-labelledby="title desc">
<title id="title">ReMe auto dream and proactive flow</title>
<desc id="desc">A left-to-right flow from changed daily notes to digest integration, interest topic writing, catalog checkpointing, and proactive reads.</desc>
<defs>
<style>
.bg { fill: #fffdf8; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; }
.step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; }
.panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; }
.chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; }
.badge { fill: #44546a; }
.arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
.line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img"
aria-labelledby="title desc">
<title id="title">ReMe auto dream and proactive flow</title>
<desc id="desc">A left-to-right flow from a recent changed-daily window to digest integration, interest topic
writing, catalog checkpointing, and proactive reads.
</desc>
<defs>
<style>.bg { fill: #fffdf8; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; } .step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; } .panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; } .chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; } .badge { fill: #44546a; } .arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); } .line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Dream and Proactive</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Scan changed daily memory, integrate reusable units into digest, then expose proactive topics.</text>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Dream and Proactive</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Scan a recent daily window, integrate a compact set of reusable units, then expose proactive topics.</text>
<rect class="panel" x="38" y="132" width="196" height="300"/>
<circle class="badge" cx="72" cy="170" r="15"/>
<text class="step-num" x="72" y="174" text-anchor="middle">1</text>
<text class="step-title" x="100" y="176">Extract</text>
<text class="step-subtitle" x="66" y="206">dream_extract_step</text>
<rect class="chip" x="66" y="232" width="140" height="44"/>
<text class="chip-title" x="136" y="251" text-anchor="middle">refresh index</text>
<text class="chip-text" x="136" y="269" text-anchor="middle">daily/&lt;date&gt;.md</text>
<rect class="chip" x="66" y="296" width="140" height="44"/>
<text class="chip-title" x="136" y="315" text-anchor="middle">compare catalog</text>
<text class="chip-text" x="136" y="333" text-anchor="middle">changed daily</text>
<rect class="chip" x="66" y="360" width="140" height="44"/>
<text class="chip-title" x="136" y="379" text-anchor="middle">LLM extract</text>
<text class="chip-text" x="136" y="397" text-anchor="middle">units + topics</text>
<rect class="panel" x="38" y="132" width="196" height="300"/>
<circle class="badge" cx="72" cy="170" r="15"/>
<text class="step-num" x="72" y="174" text-anchor="middle">1</text>
<text class="step-title" x="100" y="176">Extract</text>
<text class="step-subtitle" x="66" y="206">dream_extract_step</text>
<rect class="chip" x="66" y="232" width="140" height="44"/>
<text class="chip-title" x="136" y="251" text-anchor="middle">refresh index</text>
<text class="chip-text" x="136" y="269" text-anchor="middle">recent 2 days</text>
<rect class="chip" x="66" y="296" width="140" height="44"/>
<text class="chip-title" x="136" y="315" text-anchor="middle">compare catalog</text>
<text class="chip-text" x="136" y="333" text-anchor="middle">changed daily</text>
<rect class="chip" x="66" y="360" width="140" height="44"/>
<text class="chip-title" x="136" y="379" text-anchor="middle">LLM extract</text>
<text class="chip-text" x="136" y="397" text-anchor="middle">≤ 5 units + topics</text>
<rect class="panel" x="270" y="132" width="196" height="300"/>
<circle class="badge" cx="304" cy="170" r="15"/>
<text class="step-num" x="304" y="174" text-anchor="middle">2</text>
<text class="step-title" x="332" y="176">Integrate</text>
<text class="step-subtitle" x="298" y="206">dream_integrate_step</text>
<rect class="chip" x="298" y="232" width="140" height="44"/>
<text class="chip-title" x="368" y="251" text-anchor="middle">node_search</text>
<text class="chip-text" x="368" y="269" text-anchor="middle">recall digest</text>
<rect class="chip" x="298" y="296" width="140" height="44"/>
<text class="chip-title" x="368" y="315" text-anchor="middle">auto link</text>
<text class="chip-text" x="368" y="333" text-anchor="middle">dedup + links</text>
<rect class="chip" x="298" y="360" width="140" height="44"/>
<text class="chip-title" x="368" y="379" text-anchor="middle">write digest</text>
<text class="chip-text" x="368" y="397" text-anchor="middle">create / update</text>
<rect class="panel" x="270" y="132" width="196" height="300"/>
<circle class="badge" cx="304" cy="170" r="15"/>
<text class="step-num" x="304" y="174" text-anchor="middle">2</text>
<text class="step-title" x="332" y="176">Integrate</text>
<text class="step-subtitle" x="298" y="206">dream_integrate_step</text>
<rect class="chip" x="298" y="232" width="140" height="44"/>
<text class="chip-title" x="368" y="251" text-anchor="middle">node_search</text>
<text class="chip-text" x="368" y="269" text-anchor="middle">recall digest</text>
<rect class="chip" x="298" y="296" width="140" height="44"/>
<text class="chip-title" x="368" y="315" text-anchor="middle">auto link</text>
<text class="chip-text" x="368" y="333" text-anchor="middle">dedup + links</text>
<rect class="chip" x="298" y="360" width="140" height="44"/>
<text class="chip-title" x="368" y="379" text-anchor="middle">write digest</text>
<text class="chip-text" x="368" y="397" text-anchor="middle">create / update</text>
<rect class="panel" x="502" y="132" width="196" height="300"/>
<circle class="badge" cx="536" cy="170" r="15"/>
<text class="step-num" x="536" y="174" text-anchor="middle">3</text>
<text class="step-title" x="564" y="176">Topics</text>
<text class="step-subtitle" x="530" y="206">dream_topics_step</text>
<rect class="chip" x="530" y="232" width="140" height="44"/>
<text class="chip-title" x="600" y="251" text-anchor="middle">merge topics</text>
<text class="chip-text" x="600" y="269" text-anchor="middle">same day kept</text>
<rect class="chip" x="530" y="296" width="140" height="44"/>
<text class="chip-title" x="600" y="315" text-anchor="middle">avoid repeats</text>
<text class="chip-text" x="600" y="333" text-anchor="middle">last 7 days</text>
<rect class="chip" x="530" y="360" width="140" height="44"/>
<text class="chip-title" x="600" y="379" text-anchor="middle">write YAML</text>
<text class="chip-text" x="600" y="397" text-anchor="middle">interests.yaml</text>
<rect class="panel" x="502" y="132" width="196" height="300"/>
<circle class="badge" cx="536" cy="170" r="15"/>
<text class="step-num" x="536" y="174" text-anchor="middle">3</text>
<text class="step-title" x="564" y="176">Topics</text>
<text class="step-subtitle" x="530" y="206">dream_topics_step</text>
<rect class="chip" x="530" y="232" width="140" height="44"/>
<text class="chip-title" x="600" y="251" text-anchor="middle">merge topics</text>
<text class="chip-text" x="600" y="269" text-anchor="middle">same day kept</text>
<rect class="chip" x="530" y="296" width="140" height="44"/>
<text class="chip-title" x="600" y="315" text-anchor="middle">avoid repeats</text>
<text class="chip-text" x="600" y="333" text-anchor="middle">last 7 days</text>
<rect class="chip" x="530" y="360" width="140" height="44"/>
<text class="chip-title" x="600" y="379" text-anchor="middle">write YAML</text>
<text class="chip-text" x="600" y="397" text-anchor="middle">interests.yaml</text>
<rect class="panel" x="734" y="132" width="196" height="300"/>
<circle class="badge" cx="768" cy="170" r="15"/>
<text class="step-num" x="768" y="174" text-anchor="middle">4</text>
<text class="step-title" x="796" y="176">Finish</text>
<text class="step-subtitle" x="762" y="206">dream_finish_step</text>
<rect class="chip" x="762" y="232" width="140" height="44"/>
<text class="chip-title" x="832" y="251" text-anchor="middle">checkpoint</text>
<text class="chip-text" x="832" y="269" text-anchor="middle">skip failures</text>
<rect class="chip" x="762" y="296" width="140" height="44"/>
<text class="chip-title" x="832" y="315" text-anchor="middle">persist catalog</text>
<text class="chip-text" x="832" y="333" text-anchor="middle">file_catalog</text>
<rect class="chip" x="762" y="360" width="140" height="44"/>
<text class="chip-title" x="832" y="379" text-anchor="middle">return summary</text>
<text class="chip-text" x="832" y="397" text-anchor="middle">counts + errors</text>
<rect class="panel" x="734" y="132" width="196" height="300"/>
<circle class="badge" cx="768" cy="170" r="15"/>
<text class="step-num" x="768" y="174" text-anchor="middle">4</text>
<text class="step-title" x="796" y="176">Finish</text>
<text class="step-subtitle" x="762" y="206">dream_finish_step</text>
<rect class="chip" x="762" y="232" width="140" height="44"/>
<text class="chip-title" x="832" y="251" text-anchor="middle">checkpoint</text>
<text class="chip-text" x="832" y="269" text-anchor="middle">skip failures</text>
<rect class="chip" x="762" y="296" width="140" height="44"/>
<text class="chip-title" x="832" y="315" text-anchor="middle">persist catalog</text>
<text class="chip-text" x="832" y="333" text-anchor="middle">file_catalog</text>
<rect class="chip" x="762" y="360" width="140" height="44"/>
<text class="chip-title" x="832" y="379" text-anchor="middle">return summary</text>
<text class="chip-text" x="832" y="397" text-anchor="middle">counts + errors</text>
<rect class="panel" x="966" y="132" width="196" height="300"/>
<circle class="badge" cx="1000" cy="170" r="15"/>
<text class="step-num" x="1000" y="174" text-anchor="middle">5</text>
<text class="step-title" x="1028" y="176">Proactive</text>
<text class="step-subtitle" x="994" y="206">proactive_step</text>
<rect class="chip" x="994" y="232" width="140" height="44"/>
<text class="chip-title" x="1064" y="251" text-anchor="middle">read YAML</text>
<text class="chip-text" x="1064" y="269" text-anchor="middle">interests.yaml</text>
<rect class="chip" x="994" y="296" width="140" height="44"/>
<text class="chip-title" x="1064" y="315" text-anchor="middle">return topics</text>
<text class="chip-text" x="1064" y="333" text-anchor="middle">metadata</text>
<rect class="chip" x="994" y="360" width="140" height="44"/>
<text class="chip-title" x="1064" y="379" text-anchor="middle">agent decides</text>
<text class="chip-text" x="1064" y="397" text-anchor="middle">whether to notify</text>
<rect class="panel" x="966" y="132" width="196" height="300"/>
<circle class="badge" cx="1000" cy="170" r="15"/>
<text class="step-num" x="1000" y="174" text-anchor="middle">5</text>
<text class="step-title" x="1028" y="176">Proactive</text>
<text class="step-subtitle" x="994" y="206">proactive_step</text>
<rect class="chip" x="994" y="232" width="140" height="44"/>
<text class="chip-title" x="1064" y="251" text-anchor="middle">read YAML</text>
<text class="chip-text" x="1064" y="269" text-anchor="middle">interests.yaml</text>
<rect class="chip" x="994" y="296" width="140" height="44"/>
<text class="chip-title" x="1064" y="315" text-anchor="middle">return topics</text>
<text class="chip-text" x="1064" y="333" text-anchor="middle">metadata</text>
<rect class="chip" x="994" y="360" width="140" height="44"/>
<text class="chip-title" x="1064" y="379" text-anchor="middle">agent decides</text>
<text class="chip-text" x="1064" y="397" text-anchor="middle">whether to notify</text>
<path class="arrow" d="M234 282 H270"/>
<path class="arrow" d="M466 282 H502"/>
<path class="arrow" d="M698 282 H734"/>
<path class="arrow" d="M930 282 H966"/>
<path class="arrow" d="M234 282 H270"/>
<path class="arrow" d="M466 282 H502"/>
<path class="arrow" d="M698 282 H734"/>
<path class="arrow" d="M930 282 H966"/>
<rect class="panel" x="80" y="502" width="1040" height="82"/>
<text class="note" x="126" y="532">Inputs</text>
<text class="chip-text" x="126" y="554">daily/&lt;date&gt;.md and daily/&lt;date&gt;/**/*.md</text>
<line class="line" x1="358" y1="518" x2="358" y2="566"/>
<text class="chip-title" x="402" y="532">Long-term memory</text>
<text class="chip-text" x="402" y="554">digest/procedure, digest/personal, digest/wiki</text>
<line class="line" x1="704" y1="518" x2="704" y2="566"/>
<text class="chip-title" x="748" y="532">Proactive material</text>
<text class="chip-text" x="748" y="554">daily/&lt;date&gt;/interests.yaml</text>
<line class="line" x1="942" y1="518" x2="942" y2="566"/>
<text class="chip-title" x="972" y="532">Boundary</text>
<text class="chip-text" x="972" y="554">Read only; caller decides</text>
<rect class="panel" x="80" y="502" width="1040" height="82"/>
<text class="note" x="126" y="532">Inputs</text>
<text class="chip-text" x="126" y="554">recent scan_days; changed files</text>
<line class="line" x1="358" y1="518" x2="358" y2="566"/>
<text class="chip-title" x="402" y="532">Long-term memory</text>
<text class="chip-text" x="402" y="554">digest/{procedure, personal, wiki}</text>
<line class="line" x1="704" y1="518" x2="704" y2="566"/>
<text class="chip-title" x="748" y="532">Proactive material</text>
<text class="chip-text" x="748" y="554">daily/&lt;date&gt;/interests.yaml</text>
<line class="line" x1="942" y1="518" x2="942" y2="566"/>
<text class="chip-title" x="972" y="532">Boundary</text>
<text class="chip-text" x="972" y="554" style="font-size:11px">Read-only; caller decides</text>
<path class="soft-arrow" d="M1064 432 C1064 476 600 472 600 432"/>
<text class="note" x="834" y="474" text-anchor="middle">proactive reads interests.yaml after auto_dream writes it</text>
<path class="soft-arrow" d="M1064 432 C1064 476 600 472 600 432"/>
<text class="note" x="834" y="474" text-anchor="middle">proactive reads interests.yaml after auto_dream writes it</text>
</svg>

Before

Width:  |  Height:  |  Size: 9 KiB

After

Width:  |  Height:  |  Size: 9.2 KiB

View file

@ -1,98 +1,86 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img" aria-labelledby="title desc">
<title id="title">ReMe auto index and memory search flow</title>
<desc id="desc">A concise left-to-right flow from automatic workspace indexing to memory search and progressive link expansion.</desc>
<defs>
<style>
.bg { fill: #fffdf8; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; }
.step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; }
.panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; }
.chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; }
.badge { fill: #44546a; }
.arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
.line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img"
aria-labelledby="title desc">
<title id="title">ReMe auto index and memory search flow</title>
<desc id="desc">A concise left-to-right flow from live daily and digest Markdown indexing to memory search and
progressive link expansion.
</desc>
<defs>
<style>.bg { fill: #fffdf8; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; } .step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; } .panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; } .chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; } .badge { fill: #44546a; } .arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); } .line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Index and Memory Search</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Keep workspace indexes fresh, recall relevant chunks, then expand nearby wikilink context.</text>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Index and Memory Search</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Keep workspace indexes fresh, recall relevant chunks, then expand nearby wikilink context.</text>
<rect class="panel" x="64" y="150" width="232" height="260"/>
<circle class="badge" cx="100" cy="188" r="15"/>
<text class="step-num" x="100" y="192" text-anchor="middle">1</text>
<text class="step-title" x="128" y="194">Watch memory</text>
<text class="step-subtitle" x="92" y="224">index_update_loop</text>
<rect class="chip" x="92" y="250" width="176" height="44"/>
<text class="chip-title" x="180" y="269" text-anchor="middle">daily / digest / resource</text>
<text class="chip-text" x="180" y="287" text-anchor="middle">md and jsonl files</text>
<rect class="chip" x="92" y="314" width="176" height="44"/>
<text class="chip-title" x="180" y="333" text-anchor="middle">init + watch changes</text>
<text class="chip-text" x="180" y="351" text-anchor="middle">added / modified / deleted</text>
<rect class="panel" x="64" y="150" width="232" height="260"/>
<circle class="badge" cx="100" cy="188" r="15"/>
<text class="step-num" x="100" y="192" text-anchor="middle">1</text>
<text class="step-title" x="128" y="194">Watch memory</text>
<text class="step-subtitle" x="92" y="224">index_update_loop</text>
<rect class="chip" x="92" y="250" width="176" height="44"/>
<text class="chip-title" x="180" y="269" text-anchor="middle">daily / digest</text>
<text class="chip-text" x="180" y="287" text-anchor="middle">Markdown live watch</text>
<rect class="chip" x="92" y="314" width="176" height="44"/>
<text class="chip-title" x="180" y="333" text-anchor="middle">init + watch changes</text>
<text class="chip-text" x="180" y="351" text-anchor="middle">added / modified / deleted</text>
<rect class="panel" x="348" y="150" width="232" height="260"/>
<circle class="badge" cx="384" cy="188" r="15"/>
<text class="step-num" x="384" y="192" text-anchor="middle">2</text>
<text class="step-title" x="412" y="194">Build index</text>
<text class="step-subtitle" x="376" y="224">update_index_step</text>
<rect class="chip" x="376" y="250" width="176" height="44"/>
<text class="chip-title" x="464" y="269" text-anchor="middle">chunk file</text>
<text class="chip-text" x="464" y="287" text-anchor="middle">FileNode + FileChunk[]</text>
<rect class="chip" x="376" y="314" width="176" height="44"/>
<text class="chip-title" x="464" y="333" text-anchor="middle">store structures</text>
<text class="chip-text" x="464" y="351" text-anchor="middle">BM25 + graph + chunks</text>
<rect class="panel" x="348" y="150" width="232" height="260"/>
<circle class="badge" cx="384" cy="188" r="15"/>
<text class="step-num" x="384" y="192" text-anchor="middle">2</text>
<text class="step-title" x="412" y="194">Build index</text>
<text class="step-subtitle" x="376" y="224">update_index_step</text>
<rect class="chip" x="376" y="250" width="176" height="44"/>
<text class="chip-title" x="464" y="269" text-anchor="middle">chunk file</text>
<text class="chip-text" x="464" y="287" text-anchor="middle">FileNode + FileChunk[]</text>
<rect class="chip" x="376" y="314" width="176" height="44"/>
<text class="chip-title" x="464" y="333" text-anchor="middle">store structures</text>
<text class="chip-text" x="464" y="351" text-anchor="middle">BM25 + graph + chunks</text>
<rect class="panel" x="632" y="150" width="232" height="260"/>
<circle class="badge" cx="668" cy="188" r="15"/>
<text class="step-num" x="668" y="192" text-anchor="middle">3</text>
<text class="step-title" x="696" y="194">Recall chunks</text>
<text class="step-subtitle" x="660" y="224">search_step</text>
<rect class="chip" x="660" y="250" width="176" height="44"/>
<text class="chip-title" x="748" y="269" text-anchor="middle">BM25 search</text>
<text class="chip-text" x="748" y="287" text-anchor="middle">keyword-ranked chunks</text>
<rect class="chip" x="660" y="314" width="176" height="44"/>
<text class="chip-title" x="748" y="333" text-anchor="middle">optional vector search</text>
<text class="chip-text" x="748" y="351" text-anchor="middle">RRF fusion when enabled</text>
<rect class="panel" x="632" y="150" width="232" height="260"/>
<circle class="badge" cx="668" cy="188" r="15"/>
<text class="step-num" x="668" y="192" text-anchor="middle">3</text>
<text class="step-title" x="696" y="194">Recall chunks</text>
<text class="step-subtitle" x="660" y="224">search_step</text>
<rect class="chip" x="660" y="250" width="176" height="44"/>
<text class="chip-title" x="748" y="269" text-anchor="middle">BM25 search</text>
<text class="chip-text" x="748" y="287" text-anchor="middle">keyword-ranked chunks</text>
<rect class="chip" x="660" y="314" width="176" height="44"/>
<text class="chip-title" x="748" y="333" text-anchor="middle">optional vector search</text>
<text class="chip-text" x="748" y="351" text-anchor="middle">RRF fusion when enabled</text>
<rect class="panel" x="916" y="150" width="220" height="260"/>
<circle class="badge" cx="952" cy="188" r="15"/>
<text class="step-num" x="952" y="192" text-anchor="middle">4</text>
<text class="step-title" x="980" y="194">Expand context</text>
<text class="step-subtitle" x="944" y="224">expand_links</text>
<rect class="chip" x="944" y="250" width="164" height="44"/>
<text class="chip-title" x="1026" y="269" text-anchor="middle">top chunks</text>
<text class="chip-text" x="1026" y="287" text-anchor="middle">path + line range</text>
<rect class="chip" x="944" y="314" width="164" height="44"/>
<text class="chip-title" x="1026" y="333" text-anchor="middle">outlinks + inlinks</text>
<text class="chip-text" x="1026" y="351" text-anchor="middle">name, description, via</text>
<rect class="panel" x="916" y="150" width="220" height="260"/>
<circle class="badge" cx="952" cy="188" r="15"/>
<text class="step-num" x="952" y="192" text-anchor="middle">4</text>
<text class="step-title" x="980" y="194">Expand context</text>
<text class="step-subtitle" x="944" y="224">expand_links</text>
<rect class="chip" x="944" y="250" width="164" height="44"/>
<text class="chip-title" x="1026" y="269" text-anchor="middle">top chunks</text>
<text class="chip-text" x="1026" y="287" text-anchor="middle">path + line range</text>
<rect class="chip" x="944" y="314" width="164" height="44"/>
<text class="chip-title" x="1026" y="333" text-anchor="middle">outlinks + inlinks</text>
<text class="chip-text" x="1026" y="351" text-anchor="middle">name, description, via</text>
<path class="arrow" d="M296 280 H348"/>
<path class="arrow" d="M580 280 H632"/>
<path class="arrow" d="M864 280 H916"/>
<path class="arrow" d="M296 280 H348"/>
<path class="arrow" d="M580 280 H632"/>
<path class="arrow" d="M864 280 H916"/>
<rect class="panel" x="142" y="484" width="916" height="76"/>
<text class="note" x="190" y="514">Default path</text>
<text class="chip-text" x="190" y="536">BM25 first, vector optional.</text>
<line class="line" x1="410" y1="500" x2="410" y2="544"/>
<text class="chip-title" x="454" y="514">BM25 is enabled by default</text>
<text class="chip-text" x="454" y="536">embedding_store is empty unless configured.</text>
<line class="line" x1="720" y1="500" x2="720" y2="544"/>
<text class="chip-title" x="764" y="514">Results stay compact first</text>
<text class="chip-text" x="764" y="536">Use read or traverse for deeper expansion.</text>
<rect class="panel" x="142" y="484" width="916" height="76"/>
<text class="note" x="190" y="514">Rebuild scope</text>
<text class="chip-text" x="190" y="536">reindex adds resource + JSONL.</text>
<line class="line" x1="410" y1="500" x2="410" y2="544"/>
<text class="chip-title" x="454" y="514">BM25 is enabled by default</text>
<text class="chip-text" x="454" y="536">Recall cap: 200; embeddings opt-in.</text>
<line class="line" x1="720" y1="500" x2="720" y2="544"/>
<text class="chip-title" x="764" y="514">Results stay compact first</text>
<text class="chip-text" x="764" y="536">Use read or traverse for deeper expansion.</text>
<path class="soft-arrow" d="M1026 410 C1026 456 748 452 748 410"/>
<text class="note" x="888" y="454" text-anchor="middle">link expansion is contextual, not a full-workspace dump</text>
<path class="soft-arrow" d="M1026 410 C1026 456 748 452 748 410"/>
<text class="note" x="888" y="454" text-anchor="middle">link expansion is contextual, not a full-workspace dump</text>
</svg>

Before

Width:  |  Height:  |  Size: 7 KiB

After

Width:  |  Height:  |  Size: 7 KiB

View file

@ -1,91 +1,79 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img" aria-labelledby="title desc">
<title id="title">ReMe auto memory and auto resource flow</title>
<desc id="desc">A concise flow where auto memory and auto resource both create daily cards, which are indexed in the daily memory layer and then used by dream and search.</desc>
<defs>
<style>
.bg { fill: #fffdf8; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; }
.step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; }
.panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; }
.chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; }
.badge { fill: #44546a; }
.arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
.line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img"
aria-labelledby="title desc">
<title id="title">ReMe auto memory and auto resource flow</title>
<desc id="desc">A concise flow where auto memory and auto resource both create daily cards, which are indexed in the
daily memory layer and then used by dream and search.
</desc>
<defs>
<style>.bg { fill: #fffdf8; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .step-num { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; } .step-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .step-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; } .panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; } .chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; } .badge { fill: #44546a; } .arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); } .line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Memory &amp; Auto Resource</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Two ingestion paths create daily cards, then share one daily index and downstream memory flow.</text>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="54" text-anchor="middle">Auto Memory &amp; Auto Resource</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Two ingestion paths create daily cards, then share one daily index and downstream memory flow.</text>
<rect class="panel" x="84" y="132" width="300" height="300"/>
<circle class="badge" cx="118" cy="170" r="15"/>
<text class="step-num" x="118" y="174" text-anchor="middle">1</text>
<text class="step-title" x="146" y="176">Inputs</text>
<text class="step-subtitle" x="112" y="206">memory + resource</text>
<rect class="chip" x="112" y="234" width="244" height="54"/>
<text class="chip-title" x="234" y="255" text-anchor="middle">Auto Memory</text>
<text class="chip-text" x="234" y="273" text-anchor="middle">conversation -> session daily card</text>
<rect class="chip" x="112" y="312" width="244" height="54"/>
<text class="chip-title" x="234" y="333" text-anchor="middle">Auto Resource</text>
<text class="chip-text" x="234" y="351" text-anchor="middle">dated file -> resource daily card</text>
<rect class="chip" x="112" y="388" width="244" height="28"/>
<text class="chip-text" x="234" y="407" text-anchor="middle">original sources remain verifiable</text>
<rect class="panel" x="84" y="132" width="300" height="300"/>
<circle class="badge" cx="118" cy="170" r="15"/>
<text class="step-num" x="118" y="174" text-anchor="middle">1</text>
<text class="step-title" x="146" y="176">Inputs</text>
<text class="step-subtitle" x="112" y="206">memory + resource</text>
<rect class="chip" x="112" y="234" width="244" height="54"/>
<text class="chip-title" x="234" y="255" text-anchor="middle">Auto Memory</text>
<text class="chip-text" x="234" y="273" text-anchor="middle">conversation -> topic-named card</text>
<rect class="chip" x="112" y="312" width="244" height="54"/>
<text class="chip-title" x="234" y="333" text-anchor="middle">Auto Resource</text>
<text class="chip-text" x="234" y="351" text-anchor="middle">source file -> topic-named card</text>
<rect class="chip" x="112" y="388" width="244" height="28"/>
<text class="chip-text" x="234" y="407" text-anchor="middle">original sources remain verifiable</text>
<rect class="panel" x="450" y="132" width="300" height="300"/>
<circle class="badge" cx="484" cy="170" r="15"/>
<text class="step-num" x="484" y="174" text-anchor="middle">2</text>
<text class="step-title" x="512" y="176">Daily Memory Layer</text>
<text class="step-subtitle" x="478" y="206">daily/&lt;date&gt;/</text>
<rect class="chip" x="478" y="234" width="244" height="54"/>
<text class="chip-title" x="600" y="255" text-anchor="middle">daily cards</text>
<text class="chip-text" x="600" y="273" text-anchor="middle">facts, context, actions</text>
<rect class="chip" x="478" y="312" width="244" height="54"/>
<text class="chip-title" x="600" y="333" text-anchor="middle">day index</text>
<text class="chip-text" x="600" y="351" text-anchor="middle">daily/&lt;date&gt;.md links cards</text>
<rect class="chip" x="478" y="388" width="244" height="28"/>
<text class="chip-text" x="600" y="407" text-anchor="middle">one readable daily memory stream</text>
<rect class="panel" x="450" y="132" width="300" height="300"/>
<circle class="badge" cx="484" cy="170" r="15"/>
<text class="step-num" x="484" y="174" text-anchor="middle">2</text>
<text class="step-title" x="512" y="176">Daily Memory Layer</text>
<text class="step-subtitle" x="478" y="206">daily/&lt;date&gt;/</text>
<rect class="chip" x="478" y="234" width="244" height="54"/>
<text class="chip-title" x="600" y="255" text-anchor="middle">daily cards</text>
<text class="chip-text" x="600" y="273" text-anchor="middle">facts, context, actions</text>
<rect class="chip" x="478" y="312" width="244" height="54"/>
<text class="chip-title" x="600" y="333" text-anchor="middle">day index</text>
<text class="chip-text" x="600" y="351" text-anchor="middle">daily/&lt;date&gt;.md links cards</text>
<rect class="chip" x="478" y="388" width="244" height="28"/>
<text class="chip-text" x="600" y="407" text-anchor="middle">one readable daily memory stream</text>
<rect class="panel" x="816" y="132" width="300" height="300"/>
<circle class="badge" cx="850" cy="170" r="15"/>
<text class="step-num" x="850" y="174" text-anchor="middle">3</text>
<text class="step-title" x="878" y="176">Downstream</text>
<text class="step-subtitle" x="844" y="206">dream + search</text>
<rect class="chip" x="844" y="234" width="244" height="54"/>
<text class="chip-title" x="966" y="255" text-anchor="middle">Auto Dream</text>
<text class="chip-text" x="966" y="273" text-anchor="middle">daily material -> digest memory</text>
<rect class="chip" x="844" y="312" width="244" height="54"/>
<text class="chip-title" x="966" y="333" text-anchor="middle">Memory Search</text>
<text class="chip-text" x="966" y="351" text-anchor="middle">retrieve daily and digest nodes</text>
<rect class="chip" x="844" y="388" width="244" height="28"/>
<text class="chip-text" x="966" y="407" text-anchor="middle">daily is the shared ingestion layer</text>
<rect class="panel" x="816" y="132" width="300" height="300"/>
<circle class="badge" cx="850" cy="170" r="15"/>
<text class="step-num" x="850" y="174" text-anchor="middle">3</text>
<text class="step-title" x="878" y="176">Downstream</text>
<text class="step-subtitle" x="844" y="206">dream + search</text>
<rect class="chip" x="844" y="234" width="244" height="54"/>
<text class="chip-title" x="966" y="255" text-anchor="middle">Auto Dream</text>
<text class="chip-text" x="966" y="273" text-anchor="middle">daily material -> digest memory</text>
<rect class="chip" x="844" y="312" width="244" height="54"/>
<text class="chip-title" x="966" y="333" text-anchor="middle">Memory Search</text>
<text class="chip-text" x="966" y="351" text-anchor="middle">retrieve daily and digest nodes</text>
<rect class="chip" x="844" y="388" width="244" height="28"/>
<text class="chip-text" x="966" y="407" text-anchor="middle">daily is the shared ingestion layer</text>
<path class="arrow" d="M384 282 H450"/>
<path class="arrow" d="M750 282 H816"/>
<path class="arrow" d="M384 282 H450"/>
<path class="arrow" d="M750 282 H816"/>
<rect class="panel" x="126" y="502" width="948" height="82"/>
<text class="note" x="176" y="532">Conversation source</text>
<text class="chip-text" x="176" y="554">session/dialog/&lt;session_id&gt;.jsonl</text>
<line class="line" x1="454" y1="518" x2="454" y2="566"/>
<text class="chip-title" x="498" y="532">Resource source</text>
<text class="chip-text" x="498" y="554">resource/&lt;date&gt;/&lt;resource_file&gt;</text>
<line class="line" x1="796" y1="518" x2="796" y2="566"/>
<text class="chip-title" x="838" y="532">Daily output</text>
<text class="chip-text" x="838" y="554">daily cards plus daily/&lt;date&gt;.md</text>
<rect class="panel" x="126" y="502" width="948" height="82"/>
<text class="note" x="176" y="532">Conversation source</text>
<text class="chip-text" x="176" y="554">session/dialog/&lt;session_id&gt;.jsonl</text>
<line class="line" x1="454" y1="518" x2="454" y2="566"/>
<text class="chip-title" x="498" y="532">Resource source</text>
<text class="chip-text" x="498" y="554">resource/[&lt;date&gt;/]&lt;resource_file&gt;</text>
<line class="line" x1="796" y1="518" x2="796" y2="566"/>
<text class="chip-title" x="838" y="532">Daily output</text>
<text class="chip-text" x="838" y="554">daily/&lt;date&gt;/&lt;name&gt;.md</text>
<path class="soft-arrow" d="M966 432 C966 474 600 474 600 432"/>
<text class="note" x="782" y="478" text-anchor="middle">downstream flows read the same daily memory layer</text>
<path class="soft-arrow" d="M966 432 C966 474 600 474 600 432"/>
<text class="note" x="782" y="478" text-anchor="middle">downstream flows read the same daily memory layer</text>
</svg>

Before

Width:  |  Height:  |  Size: 6.7 KiB

After

Width:  |  Height:  |  Size: 6.8 KiB

View file

@ -1,97 +1,83 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="520" viewBox="0 0 1200 520" role="img" aria-labelledby="title desc">
<title id="title">ReMe design philosophy</title>
<desc id="desc">A simple diagram showing ReMe's design philosophy: file-native memory, progressive abstraction, and explicit relationship graph for agents.</desc>
<defs>
<style>
.bg { fill: #ffffff; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; }
.panel-blue { fill: #eef7ff; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; }
.panel-green { fill: #f0fbf5; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; }
.panel-yellow { fill: #fff7e5; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; }
.label { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.small { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.tiny { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.chip { fill: #ffffff; stroke: #1f2430; stroke-width: 1.6; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; }
.chip-text { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { stroke: #a3adbd; stroke-width: 1.25; fill: none; stroke-dasharray: 6 6; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
.node { fill: #ffffff; stroke: #1f2430; stroke-width: 1.6; }
.link { stroke: #a3adbd; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; }
.file-line { stroke: #7f8b9d; stroke-width: 1.45; stroke-linecap: round; }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="520" viewBox="0 0 1200 520" role="img"
aria-labelledby="title desc">
<title id="title">ReMe design philosophy</title>
<desc id="desc">A simple diagram showing ReMe's design philosophy: file-native memory, progressive abstraction, and
explicit relationship graph for agents.
</desc>
<defs>
<style>.bg { fill: #ffffff; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; } .panel-blue { fill: #eef7ff; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; } .panel-green { fill: #f0fbf5; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; } .panel-yellow { fill: #fff7e5; stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; } .label { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .small { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .tiny { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .chip { fill: #ffffff; stroke: #1f2430; stroke-width: 1.6; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; } .chip-text { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { stroke: #a3adbd; stroke-width: 1.25; fill: none; stroke-dasharray: 6 6; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); } .node { fill: #ffffff; stroke: #1f2430; stroke-width: 1.6; } .link { stroke: #a3adbd; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; } .file-line { stroke: #7f8b9d; stroke-width: 1.45; stroke-linecap: round; }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="520"/>
<text class="title" x="600" y="54" text-anchor="middle">ReMe Design Philosophy</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">File-native memory, progressive abstraction, and explicit links for reusable agent context.</text>
<rect class="bg" x="0" y="0" width="1200" height="520"/>
<text class="title" x="600" y="54" text-anchor="middle">ReMe Design Philosophy</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">File-native memory, progressive abstraction, and explicit links for reusable agent context.</text>
<rect class="panel-blue" x="64" y="134" width="304" height="220" rx="14"/>
<text class="label" x="216" y="170" text-anchor="middle">Foundation: Memory as File</text>
<text class="small" x="216" y="196" text-anchor="middle">Readable, editable knowledge assets</text>
<g transform="translate(126 224)">
<rect class="panel" x="0" y="0" width="120" height="82" rx="8"/>
<path class="file-line" d="M22 26 H98"/>
<path class="file-line" d="M22 43 H86"/>
<path class="file-line" d="M22 60 H72"/>
<rect class="chip" x="136" y="6" width="96" height="28" rx="8"/>
<text class="chip-text" x="184" y="25" text-anchor="middle">Markdown</text>
<rect class="chip" x="136" y="48" width="96" height="28" rx="8"/>
<text class="chip-text" x="184" y="67" text-anchor="middle">YAML/JSONL</text>
</g>
<text class="tiny" x="216" y="332" text-anchor="middle">Human-readable. Agent-operable.</text>
<rect class="panel-blue" x="64" y="134" width="304" height="220" rx="14"/>
<text class="label" x="216" y="170" text-anchor="middle">Foundation: Memory as File</text>
<text class="small" x="216" y="196" text-anchor="middle">Readable, editable knowledge assets</text>
<g transform="translate(126 224)">
<rect class="panel" x="0" y="0" width="120" height="82" rx="8"/>
<path class="file-line" d="M22 26 H98"/>
<path class="file-line" d="M22 43 H86"/>
<path class="file-line" d="M22 60 H72"/>
<rect class="chip" x="136" y="6" width="96" height="28" rx="8"/>
<text class="chip-text" x="184" y="25" text-anchor="middle">Markdown</text>
<rect class="chip" x="136" y="48" width="96" height="28" rx="8"/>
<text class="chip-text" x="184" y="67" text-anchor="middle">YAML/JSONL</text>
</g>
<text class="tiny" x="216" y="332" text-anchor="middle">Human-readable. Agent-operable.</text>
<rect class="panel-yellow" x="448" y="134" width="304" height="220" rx="14"/>
<text class="label" x="600" y="170" text-anchor="middle">Core: Progressive Abstraction</text>
<text class="small" x="600" y="196" text-anchor="middle">From raw material to durable memory</text>
<g transform="translate(472 235)">
<rect class="chip" x="0" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="33" y="26" text-anchor="middle">Raw</text>
<rect class="chip" x="95" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="128" y="26" text-anchor="middle">Daily</text>
<rect class="chip" x="190" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="223" y="26" text-anchor="middle">Digest</text>
<path class="arrow" d="M68 21 H89"/>
<path class="arrow" d="M163 21 H184"/>
</g>
<text class="tiny" x="600" y="310" text-anchor="middle">session/ + resource/ -> daily/ -> digest/</text>
<text class="tiny" x="600" y="332" text-anchor="middle">Continuous consolidation, not one-shot summary.</text>
<rect class="panel-yellow" x="448" y="134" width="304" height="220" rx="14"/>
<text class="label" x="600" y="170" text-anchor="middle">Core: Progressive Abstraction</text>
<text class="small" x="600" y="196" text-anchor="middle">From raw material to durable memory</text>
<g transform="translate(472 235)">
<rect class="chip" x="0" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="33" y="26" text-anchor="middle">Raw</text>
<rect class="chip" x="95" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="128" y="26" text-anchor="middle">Daily</text>
<rect class="chip" x="190" y="0" width="66" height="42" rx="10"/>
<text class="chip-text" x="223" y="26" text-anchor="middle">Digest</text>
<path class="arrow" d="M68 21 H89"/>
<path class="arrow" d="M163 21 H184"/>
</g>
<text class="tiny" x="600" y="310" text-anchor="middle">session/resource -> daily -> digest</text>
<text class="tiny" x="600" y="332" text-anchor="middle">Continuous consolidation; not one-shot.</text>
<rect class="panel-green" x="832" y="134" width="304" height="220" rx="14"/>
<text class="label" x="984" y="170" text-anchor="middle">Innovation: Link Graph</text>
<text class="small" x="984" y="196" text-anchor="middle">Explicit links expand context</text>
<g transform="translate(930 207) scale(0.74)">
<path class="link" d="M74 40 L26 90"/>
<path class="link" d="M74 40 L130 84"/>
<path class="link" d="M26 90 L130 84"/>
<path class="link" d="M74 40 L152 26"/>
<circle class="node" cx="74" cy="40" r="24"/>
<circle class="node" cx="26" cy="90" r="20"/>
<circle class="node" cx="130" cy="84" r="20"/>
<circle class="node" cx="152" cy="26" r="18"/>
<text class="chip-text" x="74" y="38" text-anchor="middle">[[ ]]</text>
<text class="tiny" x="74" y="55" text-anchor="middle">Node</text>
<text class="tiny" x="26" y="95" text-anchor="middle">Src</text>
<text class="tiny" x="130" y="89" text-anchor="middle">Risk</text>
<text class="tiny" x="152" y="31" text-anchor="middle">Dep</text>
</g>
<text class="tiny" x="984" y="318" text-anchor="middle">Beyond similarity: source, cause, dependency, risk.</text>
<text class="tiny" x="984" y="338" text-anchor="middle">Wikilinks guide expansion across nodes.</text>
<rect class="panel-green" x="832" y="134" width="304" height="220" rx="14"/>
<text class="label" x="984" y="170" text-anchor="middle">Innovation: Link Graph</text>
<text class="small" x="984" y="196" text-anchor="middle">Explicit links expand context</text>
<g transform="translate(930 207) scale(0.74)">
<path class="link" d="M74 40 L26 90"/>
<path class="link" d="M74 40 L130 84"/>
<path class="link" d="M26 90 L130 84"/>
<path class="link" d="M74 40 L152 26"/>
<circle class="node" cx="74" cy="40" r="24"/>
<circle class="node" cx="26" cy="90" r="20"/>
<circle class="node" cx="130" cy="84" r="20"/>
<circle class="node" cx="152" cy="26" r="18"/>
<text class="chip-text" x="74" y="38" text-anchor="middle">[[ ]]</text>
<text class="tiny" x="74" y="55" text-anchor="middle">Node</text>
<text class="tiny" x="26" y="95" text-anchor="middle">Src</text>
<text class="tiny" x="130" y="89" text-anchor="middle">Risk</text>
<text class="tiny" x="152" y="31" text-anchor="middle">Dep</text>
</g>
<text class="tiny" x="984" y="318" text-anchor="middle">Source · cause · dependency · risk</text>
<text class="tiny" x="984" y="338" text-anchor="middle">Wikilinks guide expansion across nodes.</text>
<path class="arrow" d="M368 244 H448"/>
<path class="arrow" d="M752 244 H832"/>
<path class="arrow" d="M368 244 H448"/>
<path class="arrow" d="M752 244 H832"/>
<rect class="panel" x="220" y="414" width="760" height="56" rx="14"/>
<text class="small" x="600" y="438" text-anchor="middle">One shared file tree acts as both memory store and operating surface.</text>
<text class="chip-text" x="600" y="460" text-anchor="middle">capture -> refine -> link -> retrieve -> expand</text>
<path class="soft-arrow" d="M216 354 C216 392 428 392 454 414"/>
<path class="soft-arrow" d="M600 354 V414"/>
<path class="soft-arrow" d="M984 354 C984 392 772 392 746 414"/>
<rect class="panel" x="220" y="414" width="760" height="56" rx="14"/>
<text class="small" x="600" y="438" text-anchor="middle">One shared file tree acts as both memory store and operating surface.</text>
<text class="chip-text" x="600" y="460" text-anchor="middle">capture -> refine -> link -> retrieve -> expand</text>
<path class="soft-arrow" d="M216 354 C216 392 428 392 454 414"/>
<path class="soft-arrow" d="M600 354 V414"/>
<path class="soft-arrow" d="M984 354 C984 392 772 392 746 414"/>
</svg>

Before

Width:  |  Height:  |  Size: 6.8 KiB

After

Width:  |  Height:  |  Size: 6.9 KiB

View file

@ -1,158 +1,149 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="920" viewBox="0 0 1200 920" role="img" aria-labelledby="title desc">
<title id="title">ReMe framework structure</title>
<desc id="desc">An architectural map of ReMe from file-backed workspace storage through knowledge kernel, workflows, application wiring, and external service surfaces.</desc>
<defs>
<linearGradient id="paper" x1="0" y1="0" x2="1" y2="1">
<stop offset="0" stop-color="#fbfaf6"/>
<stop offset="0.55" stop-color="#f5f7fb"/>
<stop offset="1" stop-color="#eef7f4"/>
</linearGradient>
<linearGradient id="serviceGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#d9f3ec"/>
<stop offset="1" stop-color="#dcecf7"/>
</linearGradient>
<linearGradient id="appGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#e5e8fb"/>
<stop offset="1" stop-color="#f1e4f5"/>
</linearGradient>
<linearGradient id="jobGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#fae4d8"/>
<stop offset="1" stop-color="#f4eccd"/>
</linearGradient>
<linearGradient id="kernelGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#d9f1f4"/>
<stop offset="1" stop-color="#e1e8f7"/>
</linearGradient>
<linearGradient id="workspaceGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#f0e5d3"/>
<stop offset="1" stop-color="#e3efd9"/>
</linearGradient>
<filter id="softShadow" x="-8%" y="-18%" width="116%" height="145%">
<feDropShadow dx="0" dy="16" stdDeviation="16" flood-color="#1c2b3a" flood-opacity="0.13"/>
</filter>
<filter id="nodeShadow" x="-15%" y="-30%" width="130%" height="170%">
<feDropShadow dx="0" dy="8" stdDeviation="8" flood-color="#1c2b3a" flood-opacity="0.12"/>
</filter>
<style>
.bg { fill: url(#paper); }
.grain { fill: none; stroke: #d9ded8; stroke-width: 1; opacity: 0.38; }
text { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Arial, sans-serif; }
.title { font-size: 30px; font-weight: 800; fill: #172033; }
.subtitle { font-size: 14px; fill: #667489; }
.layer-title { font-size: 18px; font-weight: 800; fill: #263548; }
.layer-note { font-size: 13px; fill: rgba(38,53,72,0.72); }
.section-label { font-size: 11px; font-weight: 800; fill: #647187; letter-spacing: 1.8px; }
.node-title { font-size: 14px; font-weight: 700; fill: #1d2838; }
.node-text { font-size: 12px; fill: #667489; }
.micro { font-size: 11px; fill: #728096; }
.plate { filter: url(#softShadow); }
.plate-bg { fill: #ffffff; stroke: rgba(52, 65, 84, 0.12); stroke-width: 1; }
.side { opacity: 0.18; }
.node { fill: #ffffff; stroke: rgba(90, 105, 128, 0.22); stroke-width: 1; filter: url(#nodeShadow); }
</style>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="920" viewBox="0 0 1200 920" role="img"
aria-labelledby="title desc">
<title id="title">ReMe framework structure</title>
<desc id="desc">An architectural map of ReMe from file-backed workspace storage through knowledge kernel, workflows,
application wiring, and external service surfaces.
</desc>
<defs>
<linearGradient id="paper" x1="0" y1="0" x2="1" y2="1">
<stop offset="0" stop-color="#fbfaf6"/>
<stop offset="0.55" stop-color="#f5f7fb"/>
<stop offset="1" stop-color="#eef7f4"/>
</linearGradient>
<linearGradient id="serviceGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#d9f3ec"/>
<stop offset="1" stop-color="#dcecf7"/>
</linearGradient>
<linearGradient id="appGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#e5e8fb"/>
<stop offset="1" stop-color="#f1e4f5"/>
</linearGradient>
<linearGradient id="jobGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#fae4d8"/>
<stop offset="1" stop-color="#f4eccd"/>
</linearGradient>
<linearGradient id="kernelGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#d9f1f4"/>
<stop offset="1" stop-color="#e1e8f7"/>
</linearGradient>
<linearGradient id="workspaceGrad" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#f0e5d3"/>
<stop offset="1" stop-color="#e3efd9"/>
</linearGradient>
<filter id="softShadow" x="-8%" y="-18%" width="116%" height="145%">
<feDropShadow dx="0" dy="16" stdDeviation="16" flood-color="#1c2b3a" flood-opacity="0.13"/>
</filter>
<filter id="nodeShadow" x="-15%" y="-30%" width="130%" height="170%">
<feDropShadow dx="0" dy="8" stdDeviation="8" flood-color="#1c2b3a" flood-opacity="0.12"/>
</filter>
<style>.bg { fill: url(#paper); } .grain { fill: none; stroke: #d9ded8; stroke-width: 1; opacity: 0.38; } text { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Arial, sans-serif; } .title { font-size: 30px; font-weight: 800; fill: #172033; } .subtitle { font-size: 14px; fill: #667489; } .layer-title { font-size: 18px; font-weight: 800; fill: #263548; } .layer-note { font-size: 13px; fill: rgba(38,53,72,0.72); } .section-label { font-size: 11px; font-weight: 800; fill: #647187; letter-spacing: 1.8px; } .node-title { font-size: 14px; font-weight: 700; fill: #1d2838; } .node-text { font-size: 12px; fill: #667489; } .micro { font-size: 11px; fill: #728096; } .plate { filter: url(#softShadow); } .plate-bg { fill: #ffffff; stroke: rgba(52, 65, 84, 0.12); stroke-width: 1; } .side { opacity: 0.18; } .node { fill: #ffffff; stroke: rgba(90, 105, 128, 0.22); stroke-width: 1; filter: url(#nodeShadow); }</style>
</defs>
<rect class="bg" width="1200" height="920"/>
<path class="grain" d="M98 138 C246 104 337 155 475 121 C616 86 752 124 890 96 C1019 70 1082 98 1136 126"/>
<path class="grain" d="M64 768 C218 720 374 790 514 742 C660 692 812 740 966 706 C1066 684 1118 708 1146 732"/>
<rect class="bg" width="1200" height="920"/>
<path class="grain" d="M98 138 C246 104 337 155 475 121 C616 86 752 124 890 96 C1019 70 1082 98 1136 126"/>
<path class="grain" d="M64 768 C218 720 374 790 514 742 C660 692 812 740 966 706 C1066 684 1118 708 1146 732"/>
<text class="title" x="600" y="56" text-anchor="middle">ReMe Framework Structure</text>
<text class="subtitle" x="600" y="83" text-anchor="middle">File-backed memory, searchable knowledge kernel, composable jobs, app wiring, and public service surfaces.</text>
<text class="title" x="600" y="56" text-anchor="middle">ReMe Framework Structure</text>
<text class="subtitle" x="600" y="83" text-anchor="middle">File-backed memory, searchable knowledge kernel, composable jobs, app wiring, and public service surfaces.</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 196 L1074 196 L1042 226 L158 226 Z" fill="#83cbbb"/>
<rect class="plate-bg" x="126" y="128" width="948" height="96" rx="18"/>
<rect x="126" y="128" width="280" height="96" rx="18" fill="url(#serviceGrad)"/>
<text class="layer-title" x="178" y="166">Service</text>
<text class="layer-note" x="178" y="188">Public interfaces</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 196 L1074 196 L1042 226 L158 226 Z" fill="#83cbbb"/>
<rect class="plate-bg" x="126" y="128" width="948" height="96" rx="18"/>
<rect x="126" y="128" width="280" height="96" rx="18" fill="url(#serviceGrad)"/>
<text class="layer-title" x="178" y="166">Service</text>
<text class="layer-note" x="178" y="188">Public interfaces</text>
<rect class="node" x="436" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="505" y="172" text-anchor="middle">HTTP API</text>
<text class="node-text" x="505" y="190" text-anchor="middle">server routes</text>
<rect class="node" x="626" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="695" y="172" text-anchor="middle">MCP Tools</text>
<text class="node-text" x="695" y="190" text-anchor="middle">agent actions</text>
<rect class="node" x="816" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="885" y="172" text-anchor="middle">CLI Client</text>
<text class="node-text" x="885" y="190" text-anchor="middle">local access</text>
</g>
<rect class="node" x="436" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="505" y="172" text-anchor="middle">HTTP API</text>
<text class="node-text" x="505" y="190" text-anchor="middle">API + Studio</text>
<rect class="node" x="626" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="695" y="172" text-anchor="middle">MCP Tools</text>
<text class="node-text" x="695" y="190" text-anchor="middle">agent actions</text>
<rect class="node" x="816" y="151" width="138" height="50" rx="13"/>
<text class="node-title" x="885" y="172" text-anchor="middle">CLI Client</text>
<text class="node-text" x="885" y="190" text-anchor="middle">service calls</text>
</g>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M94 336 L1106 336 L1074 366 L126 366 Z" fill="#aab5ec"/>
<rect class="plate-bg" x="94" y="264" width="1012" height="100" rx="18"/>
<rect x="94" y="264" width="280" height="100" rx="18" fill="url(#appGrad)"/>
<text class="layer-title" x="146" y="303">Application</text>
<text class="layer-note" x="146" y="325">Config, wiring, lifecycle</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M94 336 L1106 336 L1074 366 L126 366 Z" fill="#aab5ec"/>
<rect class="plate-bg" x="94" y="264" width="1012" height="100" rx="18"/>
<rect x="94" y="264" width="280" height="100" rx="18" fill="url(#appGrad)"/>
<text class="layer-title" x="146" y="303">Application</text>
<text class="layer-note" x="146" y="325">Config, wiring, lifecycle</text>
<rect class="node" x="424" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="485" y="319" text-anchor="middle">Context</text>
<rect class="node" x="574" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="635" y="319" text-anchor="middle">Wiring</text>
<rect class="node" x="724" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="785" y="319" text-anchor="middle">Lifecycle</text>
<rect class="node" x="874" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="935" y="319" text-anchor="middle">Job APIs</text>
</g>
<rect class="node" x="424" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="485" y="319" text-anchor="middle">Context</text>
<rect class="node" x="574" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="635" y="319" text-anchor="middle">Wiring</text>
<rect class="node" x="724" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="785" y="319" text-anchor="middle">Lifecycle</text>
<rect class="node" x="874" y="290" width="122" height="48" rx="12"/>
<text class="node-title" x="935" y="319" text-anchor="middle">Job APIs</text>
</g>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 500 L1074 500 L1042 530 L158 530 Z" fill="#e9ad8f"/>
<rect class="plate-bg" x="126" y="404" width="948" height="124" rx="18"/>
<rect x="126" y="404" width="280" height="124" rx="18" fill="url(#jobGrad)"/>
<text class="layer-title" x="178" y="448">Steps / Jobs</text>
<text class="layer-note" x="178" y="470">Composable workflows</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 500 L1074 500 L1042 530 L158 530 Z" fill="#e9ad8f"/>
<rect class="plate-bg" x="126" y="404" width="948" height="124" rx="18"/>
<rect x="126" y="404" width="280" height="124" rx="18" fill="url(#jobGrad)"/>
<text class="layer-title" x="178" y="448">Steps / Jobs</text>
<text class="layer-note" x="178" y="470">Composable workflows</text>
<rect class="node" x="424" y="428" width="232" height="76" rx="14"/>
<text class="node-title" x="448" y="456">Jobs</text>
<text class="node-text" x="448" y="479">base · stream · background · cron</text>
<rect class="node" x="704" y="428" width="300" height="76" rx="14"/>
<text class="node-title" x="728" y="456">Step Modules</text>
<text class="node-text" x="728" y="479">file_io · index · evolve · transfer · channel · common</text>
</g>
<rect class="node" x="424" y="428" width="232" height="76" rx="14"/>
<text class="node-title" x="448" y="456">Jobs</text>
<text class="node-text" x="448" y="479">base · stream · background · cron</text>
<rect class="node" x="704" y="428" width="300" height="76" rx="14"/>
<text class="node-title" x="728" y="456">Step Modules</text>
<text class="node-text" x="728" y="479">file_io · index · evolve · common</text>
<text class="node-text" x="728" y="497">transfer · cookbook · benchmark</text>
</g>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M94 674 L1106 674 L1074 704 L126 704 Z" fill="#8fcbd5"/>
<rect class="plate-bg" x="94" y="568" width="1012" height="134" rx="18"/>
<rect x="94" y="568" width="280" height="134" rx="18" fill="url(#kernelGrad)"/>
<text class="layer-title" x="146" y="615">Knowledge Kernel</text>
<text class="layer-note" x="146" y="637">Index, watch, schema</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M94 674 L1106 674 L1074 704 L126 704 Z" fill="#8fcbd5"/>
<rect class="plate-bg" x="94" y="568" width="1012" height="134" rx="18"/>
<rect x="94" y="568" width="280" height="134" rx="18" fill="url(#kernelGrad)"/>
<text class="layer-title" x="146" y="615">Knowledge Kernel</text>
<text class="layer-note" x="146" y="637">Index, watch, schema</text>
<rect class="node" x="424" y="594" width="228" height="82" rx="14"/>
<text class="node-title" x="448" y="623">Index Stores</text>
<text class="node-text" x="448" y="646">file_store · keyword_index</text>
<text class="node-text" x="448" y="664">embedding_store · file_graph</text>
<rect class="node" x="698" y="594" width="172" height="82" rx="14"/>
<text class="node-title" x="722" y="623">File Watcher</text>
<text class="node-text" x="722" y="646">scanner · chunker</text>
<text class="node-text" x="722" y="664">catalog</text>
<rect class="node" x="916" y="594" width="142" height="82" rx="14"/>
<text class="node-title" x="940" y="623">Memory</text>
<text class="node-text" x="940" y="646">FileNode</text>
<text class="node-text" x="940" y="664">FileChunk · FileLink</text>
</g>
<rect class="node" x="424" y="594" width="228" height="82" rx="14"/>
<text class="node-title" x="448" y="623">Index Stores</text>
<text class="node-text" x="448" y="646">file_store · keyword_index</text>
<text class="node-text" x="448" y="664">embedding_store · file_graph</text>
<rect class="node" x="698" y="594" width="172" height="82" rx="14"/>
<text class="node-title" x="722" y="623">File Watcher</text>
<text class="node-text" x="722" y="646">scanner · chunker</text>
<text class="node-text" x="722" y="664">catalog</text>
<rect class="node" x="916" y="594" width="142" height="82" rx="14"/>
<text class="node-title" x="940" y="623">Memory</text>
<text class="node-text" x="940" y="646">FileNode</text>
<text class="node-text" x="940" y="664">FileChunk · FileLink</text>
</g>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 818 L1074 818 L1042 848 L158 848 Z" fill="#d0b98f"/>
<rect class="plate-bg" x="126" y="738" width="948" height="108" rx="18"/>
<rect x="126" y="738" width="280" height="108" rx="18" fill="url(#workspaceGrad)"/>
<text class="layer-title" x="178" y="777">Workspace Layout</text>
<text class="layer-note" x="178" y="799">File-backed memory</text>
<g class="plate" transform="translate(0,0)">
<path class="side" d="M126 818 L1074 818 L1042 848 L158 848 Z" fill="#d0b98f"/>
<rect class="plate-bg" x="126" y="738" width="948" height="108" rx="18"/>
<rect x="126" y="738" width="280" height="108" rx="18" fill="url(#workspaceGrad)"/>
<text class="layer-title" x="178" y="777">Workspace Layout</text>
<text class="layer-note" x="178" y="799">File-backed memory</text>
<rect class="node" x="424" y="767" width="126" height="50" rx="13"/>
<text class="node-title" x="487" y="788" text-anchor="middle">daily/</text>
<text class="node-text" x="487" y="806" text-anchor="middle">working notes</text>
<rect class="node" x="584" y="767" width="126" height="50" rx="13"/>
<text class="node-title" x="647" y="788" text-anchor="middle">digest/</text>
<text class="node-text" x="647" y="806" text-anchor="middle">long-term</text>
<rect class="node" x="744" y="767" width="126" height="50" rx="13"/>
<text class="node-title" x="807" y="788" text-anchor="middle">resource/</text>
<text class="node-text" x="807" y="806" text-anchor="middle">resources</text>
<rect class="node" x="904" y="767" width="126" height="50" rx="13"/>
<text class="node-title" x="967" y="788" text-anchor="middle">metadata/</text>
<text class="node-text" x="967" y="806" text-anchor="middle">state</text>
</g>
<rect class="node" x="400" y="767" width="160" height="50" rx="13"/>
<text class="node-title" x="480" y="788" text-anchor="middle">session directories</text>
<text class="node-text" x="480" y="806" text-anchor="middle">sources · agent runtime</text>
<rect class="node" x="572" y="767" width="100" height="50" rx="13"/>
<text class="node-title" x="622" y="788" text-anchor="middle">daily/</text>
<text class="node-text" x="622" y="806" text-anchor="middle">working</text>
<rect class="node" x="684" y="767" width="100" height="50" rx="13"/>
<text class="node-title" x="734" y="788" text-anchor="middle">digest/</text>
<text class="node-text" x="734" y="806" text-anchor="middle">long-term</text>
<rect class="node" x="796" y="767" width="100" height="50" rx="13"/>
<text class="node-title" x="846" y="788" text-anchor="middle">resource/</text>
<text class="node-text" x="846" y="806" text-anchor="middle">sources</text>
<rect class="node" x="908" y="767" width="122" height="50" rx="13"/>
<text class="node-title" x="969" y="788" text-anchor="middle">metadata/</text>
<text class="node-text" x="969" y="806" text-anchor="middle">derived state</text>
</g>
<text class="section-label" x="92" y="118">EXTERNAL SURFACES</text>
<text class="section-label" x="92" y="734">PERSISTENT MEMORY BASE</text>
<text class="micro" x="600" y="878" text-anchor="middle">Structure reads bottom-up during boot and top-down during use.</text>
<text class="micro" x="600" y="896" text-anchor="middle">Workspace files feed the kernel; workflows compose operations; application wiring exposes stable service entry points.</text>
<text class="section-label" x="92" y="118">EXTERNAL SURFACES</text>
<text class="section-label" x="92" y="734">PERSISTENT MEMORY BASE</text>
<text class="micro" x="600" y="878" text-anchor="middle">Structure reads bottom-up during boot and top-down during use.</text>
<text class="micro" x="600" y="896" text-anchor="middle">Workspace files feed the kernel; workflows compose operations; application wiring exposes stable service entry points.</text>
</svg>

Before

Width:  |  Height:  |  Size: 9.5 KiB

After

Width:  |  Height:  |  Size: 10 KiB

View file

@ -1,110 +1,97 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="670" viewBox="0 0 1200 670" role="img" aria-labelledby="title desc">
<title id="title">ReMe memory as file model</title>
<desc id="desc">A diagram showing ReMe workspace files as both a human readable memory interface and an agent operable memory graph, flowing from raw input to daily notes, digest nodes, and metadata indexes.</desc>
<defs>
<style>
.bg { fill: #fffdf8; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.section-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.section-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; }
.label { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; }
.panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; }
.chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; }
.badge { fill: #44546a; }
.soft { fill: #eef7ff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; }
.arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
.line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="670" viewBox="0 0 1200 670" role="img"
aria-labelledby="title desc">
<title id="title">ReMe memory as file model</title>
<desc id="desc">A diagram showing ReMe workspace files as both a human readable memory interface and an agent
operable memory graph, flowing from raw input to daily notes, digest nodes, and metadata indexes.
</desc>
<defs>
<style>.bg { fill: #fffdf8; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .section-title { font: 700 18px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, Segoe UI, sans-serif; fill: #1f2430; } .section-subtitle { font: 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, Segoe UI, sans-serif; fill: #556276; } .chip-title { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .chip-text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .note { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; } .label { font: 700 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #ffffff; } .panel { fill: #ffffff; stroke: #1f2430; stroke-width: 2.2; rx: 18; ry: 18; stroke-linecap: round; stroke-linejoin: round; } .chip { fill: #f8fbff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; stroke-dasharray: 6 5; } .badge { fill: #44546a; } .soft { fill: #eef7ff; stroke: #1f2430; stroke-width: 1.6; rx: 11; ry: 11; stroke-linecap: round; stroke-linejoin: round; } .arrow { stroke: #7f8b9d; stroke-width: 1.45; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; fill: none; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); } .line { stroke: #a3adbd; stroke-width: 1.25; stroke-linecap: round; }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="670"/>
<text class="title" x="600" y="54" text-anchor="middle">Memory as File</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Workspace files are the readable memory surface and the operable graph/index substrate.</text>
<rect class="bg" x="0" y="0" width="1200" height="670"/>
<text class="title" x="600" y="54" text-anchor="middle">Memory as File</text>
<text class="subtitle" x="600" y="80" text-anchor="middle">Workspace files are the readable memory surface and the operable graph/index substrate.</text>
<rect class="panel" x="72" y="124" width="280" height="168"/>
<rect class="badge" x="102" y="150" width="94" height="24" rx="12" ry="12"/>
<text class="label" x="149" y="167" text-anchor="middle">Human</text>
<text class="section-title" x="102" y="202">Read and edit files</text>
<text class="section-subtitle" x="102" y="226">Markdown, YAML, JSONL, resources</text>
<rect class="chip" x="102" y="248" width="220" height="28"/>
<text class="chip-text" x="212" y="267" text-anchor="middle">open, revise, move, delete</text>
<rect class="panel" x="72" y="124" width="280" height="168"/>
<rect class="badge" x="102" y="150" width="94" height="24" rx="12" ry="12"/>
<text class="label" x="149" y="167" text-anchor="middle">Human</text>
<text class="section-title" x="102" y="202">Read and edit files</text>
<text class="section-subtitle" x="102" y="226" style="font-size:11px">Markdown · YAML · JSONL · resources</text>
<rect class="chip" x="102" y="248" width="220" height="28"/>
<text class="chip-text" x="212" y="267" text-anchor="middle">open, revise, move, delete</text>
<rect class="panel" x="460" y="120" width="280" height="176"/>
<text class="section-title" x="600" y="152" text-anchor="middle">Workspace directory</text>
<text class="section-subtitle" x="600" y="176" text-anchor="middle">the shared memory interface</text>
<rect class="soft" x="504" y="204" width="192" height="30"/>
<text class="chip-title" x="600" y="224" text-anchor="middle">Memory as File</text>
<rect class="soft" x="504" y="248" width="192" height="30"/>
<text class="chip-title" x="600" y="268" text-anchor="middle">File as Memory</text>
<rect class="panel" x="460" y="120" width="280" height="176"/>
<text class="section-title" x="600" y="152" text-anchor="middle">Workspace directory</text>
<text class="section-subtitle" x="600" y="176" text-anchor="middle">the shared memory interface</text>
<rect class="soft" x="504" y="204" width="192" height="30"/>
<text class="chip-title" x="600" y="224" text-anchor="middle">Memory as File</text>
<rect class="soft" x="504" y="248" width="192" height="30"/>
<text class="chip-title" x="600" y="268" text-anchor="middle">File as Memory</text>
<rect class="panel" x="848" y="124" width="280" height="168"/>
<rect class="badge" x="878" y="150" width="94" height="24" rx="12" ry="12"/>
<text class="label" x="925" y="167" text-anchor="middle">Agent</text>
<text class="section-title" x="878" y="202">Parse and operate graph</text>
<text class="section-subtitle" x="878" y="226">frontmatter, chunks, wikilinks</text>
<rect class="chip" x="878" y="248" width="220" height="28"/>
<text class="chip-text" x="988" y="267" text-anchor="middle">search, link, rewrite, index</text>
<rect class="panel" x="848" y="124" width="280" height="168"/>
<rect class="badge" x="878" y="150" width="94" height="24" rx="12" ry="12"/>
<text class="label" x="925" y="167" text-anchor="middle">Agent</text>
<text class="section-title" x="878" y="202">Parse and operate graph</text>
<text class="section-subtitle" x="878" y="226">frontmatter, chunks, wikilinks</text>
<rect class="chip" x="878" y="248" width="220" height="28"/>
<text class="chip-text" x="988" y="267" text-anchor="middle">search, link, rewrite, index</text>
<path class="arrow" d="M352 208 H460"/>
<path class="arrow" d="M740 208 H848"/>
<path class="soft-arrow" d="M848 250 C740 326 460 326 352 250"/>
<text class="note" x="600" y="336" text-anchor="middle">people and agents see the same file tree, so edits and evidence links stay inspectable</text>
<path class="arrow" d="M352 208 H460"/>
<path class="arrow" d="M740 208 H848"/>
<path class="soft-arrow" d="M848 250 C740 326 460 326 352 250"/>
<text class="note" x="600" y="336" text-anchor="middle">people and agents see the same file tree, so edits and evidence links stay inspectable</text>
<rect class="panel" x="54" y="398" width="224" height="144"/>
<circle class="badge" cx="88" cy="432" r="15"/>
<text class="label" x="88" y="436" text-anchor="middle">1</text>
<text class="section-title" x="116" y="438">Raw input</text>
<text class="section-subtitle" x="82" y="468">keep the original scene</text>
<rect class="chip" x="82" y="494" width="168" height="28"/>
<text class="chip-text" x="166" y="513" text-anchor="middle">session/ + resource/</text>
<rect class="panel" x="54" y="398" width="224" height="144"/>
<circle class="badge" cx="88" cy="432" r="15"/>
<text class="label" x="88" y="436" text-anchor="middle">1</text>
<text class="section-title" x="116" y="438">Raw input</text>
<text class="section-subtitle" x="82" y="468">keep the original scene</text>
<rect class="chip" x="82" y="494" width="168" height="28"/>
<text class="chip-text" x="166" y="513" text-anchor="middle">session/ + resource/</text>
<rect class="panel" x="326" y="398" width="224" height="144"/>
<circle class="badge" cx="360" cy="432" r="15"/>
<text class="label" x="360" y="436" text-anchor="middle">2</text>
<text class="section-title" x="388" y="438">Daily</text>
<text class="section-subtitle" x="354" y="468">shallow working memory</text>
<rect class="chip" x="354" y="494" width="168" height="28"/>
<text class="chip-text" x="438" y="513" text-anchor="middle">daily/YYYY-MM-DD/*.md</text>
<rect class="panel" x="326" y="398" width="224" height="144"/>
<circle class="badge" cx="360" cy="432" r="15"/>
<text class="label" x="360" y="436" text-anchor="middle">2</text>
<text class="section-title" x="388" y="438">Daily</text>
<text class="section-subtitle" x="354" y="468">shallow working memory</text>
<rect class="chip" x="354" y="494" width="168" height="28"/>
<text class="chip-text" x="438" y="513" text-anchor="middle">daily/&lt;date&gt;/&lt;name&gt;.md</text>
<rect class="panel" x="598" y="398" width="224" height="144"/>
<circle class="badge" cx="632" cy="432" r="15"/>
<text class="label" x="632" y="436" text-anchor="middle">3</text>
<text class="section-title" x="660" y="438">Digest</text>
<text class="section-subtitle" x="626" y="468">long-term reusable nodes</text>
<rect class="chip" x="626" y="494" width="168" height="28"/>
<text class="chip-text" x="710" y="513" text-anchor="middle">personal / procedure / wiki</text>
<rect class="panel" x="598" y="398" width="224" height="144"/>
<circle class="badge" cx="632" cy="432" r="15"/>
<text class="label" x="632" y="436" text-anchor="middle">3</text>
<text class="section-title" x="660" y="438">Digest</text>
<text class="section-subtitle" x="626" y="468">long-term reusable nodes</text>
<rect class="chip" x="626" y="494" width="168" height="28"/>
<text class="chip-text" x="710" y="513" text-anchor="middle">personal / procedure / wiki</text>
<rect class="panel" x="870" y="398" width="224" height="144"/>
<circle class="badge" cx="904" cy="432" r="15"/>
<text class="label" x="904" y="436" text-anchor="middle">4</text>
<text class="section-title" x="932" y="438">Metadata</text>
<text class="section-subtitle" x="898" y="468">system state and indexes</text>
<rect class="chip" x="898" y="494" width="168" height="28"/>
<text class="chip-text" x="982" y="513" text-anchor="middle">catalog + chunks + links</text>
<rect class="panel" x="870" y="398" width="224" height="144"/>
<circle class="badge" cx="904" cy="432" r="15"/>
<text class="label" x="904" y="436" text-anchor="middle">4</text>
<text class="section-title" x="932" y="438">Metadata</text>
<text class="section-subtitle" x="898" y="468">system state and indexes</text>
<rect class="chip" x="898" y="494" width="168" height="28"/>
<text class="chip-text" x="982" y="513" text-anchor="middle">catalog + chunks + links</text>
<path class="arrow" d="M278 470 H326"/>
<path class="arrow" d="M550 470 H598"/>
<path class="arrow" d="M822 470 H870"/>
<path class="soft-arrow" d="M982 398 C982 356 710 356 710 398"/>
<path class="soft-arrow" d="M982 542 C982 592 438 592 438 542"/>
<path class="arrow" d="M278 470 H326"/>
<path class="arrow" d="M550 470 H598"/>
<path class="arrow" d="M822 470 H870"/>
<path class="soft-arrow" d="M982 398 C982 356 710 356 710 398"/>
<path class="soft-arrow" d="M982 542 C982 592 438 592 438 542"/>
<rect class="panel" x="126" y="594" width="948" height="46"/>
<text class="note" x="176" y="622">Stable paths</text>
<line class="line" x1="308" y1="606" x2="308" y2="628"/>
<text class="chip-text" x="360" y="622">workspace-relative wikilinks</text>
<line class="line" x1="548" y1="606" x2="548" y2="628"/>
<text class="chip-text" x="600" y="622">derived_from evidence edges</text>
<line class="line" x1="814" y1="606" x2="814" y2="628"/>
<text class="chip-text" x="862" y="622">search expands structure and links</text>
<rect class="panel" x="126" y="594" width="948" height="46"/>
<text class="note" x="176" y="622">Stable paths</text>
<line class="line" x1="308" y1="606" x2="308" y2="628"/>
<text class="chip-text" x="360" y="622">workspace-relative wikilinks</text>
<line class="line" x1="548" y1="606" x2="548" y2="628"/>
<text class="chip-text" x="600" y="622">derived_from evidence edges</text>
<line class="line" x1="814" y1="606" x2="814" y2="628"/>
<text class="chip-text" x="862" y="622">search expands linked context</text>
</svg>

Before

Width:  |  Height:  |  Size: 7.6 KiB

After

Width:  |  Height:  |  Size: 7.7 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 47 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 385 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 46 KiB

View file

@ -0,0 +1,80 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img"
aria-labelledby="title desc">
<title id="title">Auto Resource</title>
<desc id="desc">A general workflow that imports external information, interprets it with traceable sources, and
integrates it into a user-owned personal knowledge base.
</desc>
<defs>
<style>.bg{fill:#fffdf8}.title{font:800 30px Arial,sans-serif;fill:#1f2430}.sub{font:14px Arial,sans-serif;fill:#667085}.panel{fill:#fff;stroke:#1f2430;stroke-width:2}.head{font:700 18px Arial,sans-serif;fill:#1f2430}.text{font:13px Arial,sans-serif;fill:#5e6a7c}.small{font:12px Arial,sans-serif;fill:#667085}.label{font:700 13px Arial,sans-serif;fill:#1f2430}.path{font:700 13px Arial,sans-serif;fill:#e66f11}.chip{fill:#f8fbff;stroke:#1f2430;stroke-width:1.3;stroke-dasharray:6 5}.orange{fill:#fff3e7}.blue{fill:#eef7ff}.green{fill:#eef9f4}.arrow{fill:none;stroke:#8793a3;stroke-width:2.5;marker-end:url(#a)}</style>
<marker id="a" markerWidth="9" markerHeight="9" refX="8" refY="4.5" orient="auto">
<path d="M0 0v9l9-4.5z" fill="#8793a3"/>
</marker>
</defs>
<rect class="bg" width="1200" height="640"/>
<text text-anchor="middle"><tspan class="title" x="600" y="50">Auto Resource</tspan><tspan class="sub" x="600" y="77">Bring external information into a traceable, user-owned knowledge base</tspan></text>
<text x="1080" y="51" style="font:800 25px Arial,sans-serif;fill:#19bfc3">Re</text>
<text x="1111" y="51" style="font:800 25px Arial,sans-serif;fill:#3274d9">Me</text>
<rect class="panel" x="46" y="112" width="304" height="458" rx="18"/>
<circle cx="92" cy="159" r="24" fill="#ff963d"/>
<path d="M80 149h21l7 7v19H80z" fill="#fff"/>
<path d="M101 149v8h8" fill="none" stroke="#ff963d" stroke-width="2"/>
<text class="head" x="128" y="155">External Information</text>
<text class="text" x="128" y="179" style="font-size:13px">Originals remain verifiable</text>
<rect class="chip orange" x="76" y="214" width="244" height="68" rx="11"/>
<text class="label" x="198" y="240" text-anchor="middle">Documents · notes · web archives</text>
<text class="small" x="198" y="263" text-anchor="middle">research · reports · structured data</text>
<rect class="chip" x="76" y="302" width="244" height="68" rx="11"/>
<text class="label" x="198" y="328" text-anchor="middle">md · txt · json · jsonl</text>
<text class="small" x="198" y="351" text-anchor="middle">csv · yaml · html</text>
<rect class="chip" x="76" y="390" width="244" height="68" rx="11"/>
<text class="label" x="198" y="416" text-anchor="middle">Create · update · delete</text>
<text class="small" x="198" y="439" text-anchor="middle" style="font-size:12px">Changes stay visible and recoverable</text>
<text class="path" x="198" y="505" text-anchor="middle">resource/</text>
<text class="text" x="198" y="532" text-anchor="middle">Original files remain user-owned</text>
<path class="arrow" d="M350 341h62"/>
<rect class="panel" x="412" y="112" width="376" height="458" rx="18"/>
<circle cx="458" cy="159" r="24" fill="#5b91dc"/>
<path d="M445 149h26v22h-26zM451 143h14v6" fill="#fff" stroke="#fff" stroke-width="3" stroke-linejoin="round"/>
<text class="head" x="494" y="155">Auto Resource Ingestion</text>
<text class="text" x="494" y="179">Parse, interpret, and preserve provenance</text>
<rect class="chip blue" x="442" y="214" width="316" height="68" rx="11"/>
<text class="label" x="468" y="240">1 Parse content and structure</text>
<text class="small" x="468" y="263">Read text, sections, fields, and metadata</text>
<rect class="chip orange" x="442" y="302" width="316" height="68" rx="11"/>
<text class="label" x="468" y="328">2 Interpret reusable information</text>
<text class="small" x="468" y="351" style="font-size:12px">Extract themes, facts, and actionable details</text>
<rect class="chip green" x="442" y="390" width="316" height="68" rx="11"/>
<text class="label" x="468" y="416">3 Attach traceable provenance</text>
<text class="small" x="468" y="439" style="font-size:12px">Link each memory with source_resource</text>
<text class="path" x="600" y="505" text-anchor="middle">daily/</text>
<text class="text" x="600" y="532" text-anchor="middle">Write readable resource memory</text>
<path class="arrow" d="M788 341h62"/>
<rect class="panel" x="850" y="112" width="304" height="458" rx="18"/>
<circle cx="896" cy="159" r="24" fill="#ff963d"/>
<path d="M883 171c7-19 18-26 27-27-1 12-6 25-27 27z" fill="#fff"/>
<path d="M896 151v23" stroke="#fff" stroke-width="3" stroke-linecap="round"/>
<text class="head" x="932" y="155">Knowledge Base</text>
<text class="text" x="932" y="179" style="font-size:13px">One memory system per source</text>
<rect class="chip blue" x="880" y="214" width="244" height="68" rx="11"/>
<text class="label" x="1002" y="240" text-anchor="middle">daily/</text>
<text class="small" x="1002" y="263" text-anchor="middle" style="font-size:12px">Readable, traceable resource memory</text>
<rect class="chip green" x="880" y="302" width="244" height="68" rx="11"/>
<text class="label" x="1002" y="328" text-anchor="middle">digest/</text>
<text class="small" x="1002" y="351" text-anchor="middle" style="font-size:12px">Long-term memory via Auto Dream</text>
<rect class="chip" x="880" y="390" width="244" height="68" rx="11"/>
<text class="label" x="1002" y="416" text-anchor="middle">metadata/</text>
<text class="small" x="1002" y="439" text-anchor="middle" style="font-size:12px">Rebuildable index and link graph</text>
<text class="path" x="1002" y="505" text-anchor="middle">Search · link · reuse</text>
<text class="text" x="1002" y="532" text-anchor="middle">External knowledge returns to future tasks</text>
<text class="small" x="600" y="610" text-anchor="middle">User files stay authoritative · derived indexes remain rebuildable</text>
</svg>

After

Width:  |  Height:  |  Size: 6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

View file

@ -0,0 +1,44 @@
<svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" width="1680" height="973"
viewBox="0 0 1680 973" role="img" aria-labelledby="title desc">
<title id="title">ReMe Hero and Public Benchmark Results</title>
<desc id="desc">ReMe self-evolving personal knowledge base hero with LongMemEval 89.4%, BEAM 100K 66.1%, and BEAM 1M
65.0% results.
</desc>
<defs>
<linearGradient id="glass" x1="0" y1="0" x2="1" y2="0">
<stop offset="0" stop-color="#171b25" stop-opacity="0.96"/>
<stop offset="0.58" stop-color="#202530" stop-opacity="0.94"/>
<stop offset="1" stop-color="#252b36" stop-opacity="0.92"/>
</linearGradient>
<filter id="shadow" x="-20%" y="-30%" width="140%" height="180%">
<feDropShadow dx="0" dy="-8" stdDeviation="16" flood-color="#000" flood-opacity="0.16"/>
</filter>
<style>.sans{font-family:Arial,sans-serif} .eyebrow{font-size:17px;font-weight:750;letter-spacing:2px;fill:#ffad61} .metric{font-size:49px;font-weight:900;letter-spacing:-1.5px;fill:#fff} .label{font-size:17px;font-weight:720;fill:#fff} .small{font-size:14px;font-weight:520;fill:#d0d5dd}</style>
</defs>
<image xlink:href="reme-blog-cover.png" x="0" y="0" width="1680" height="973" preserveAspectRatio="xMidYMid slice"/>
<rect x="0" y="711" width="1680" height="262" fill="url(#glass)" filter="url(#shadow)"/>
<g class="sans">
<text class="eyebrow" x="84" y="765">PUBLIC BENCHMARK RESULTS</text>
<text class="small" x="84" y="798">Agentic scores across multi-session and long-context settings</text>
<line x1="84" y1="829" x2="1596" y2="829" stroke="#fff" stroke-opacity="0.17"/>
<text class="metric" x="84" y="898">89.4%</text>
<text class="label" x="277" y="875">LongMemEval cleaned-s</text>
<text class="small" x="277" y="903">500 questions · cross-session retrieval</text>
<line x1="590" y1="852" x2="590" y2="923" stroke="#fff" stroke-opacity="0.18"/>
<text class="metric" x="650" y="898">66.1%</text>
<text class="label" x="833" y="875">BEAM 100K</text>
<text class="small" x="833" y="903">20 cases · 400 questions</text>
<line x1="1080" y1="852" x2="1080" y2="923" stroke="#fff" stroke-opacity="0.18"/>
<text class="metric" x="1140" y="898">65.0%</text>
<text class="label" x="1323" y="875">BEAM 1M</text>
<text class="small" x="1323" y="903">35 cases · 700 questions</text>
<text x="1596" y="950" text-anchor="end" style="font-size:12px;font-weight:520;fill:#98a2b3">Scores from different benchmarks are not compared directly</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 382 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 46 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 46 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 47 KiB

View file

@ -1,204 +1,184 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img" aria-labelledby="title desc">
<title id="title">ReMe overview</title>
<desc id="desc">A hand-drawn style overview of ReMe, showing Auto Memory plus Auto Resource, Auto Dream plus Proactive, Auto Index plus Memory Search, and Memory as File.</desc>
<defs>
<style>
.bg { fill: #fffdf8; }
.ink { stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; }
.thin { stroke-width: 1.6; }
.dash { stroke-dasharray: 8 7; }
.title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; }
.head { font: 700 17px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.label { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; }
.text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; }
.tiny { font: 11px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; }
.box { fill: #ffffff; }
.blue { fill: #eef7ff; }
.green { fill: #f0fbf5; }
.yellow { fill: #fff7e5; }
.pink { fill: #fff2f6; }
.violet { fill: #f4f1ff; }
.mint { fill: #eefafa; }
.peach { fill: #fff2ea; }
.paper { fill: #f8fbff; }
.tab { fill: #ffffff; }
.step { stroke-dasharray: 6 5; }
.arrow { fill: none; stroke: #7f8b9d; stroke-width: 1.45; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); }
.soft-arrow { fill: none; stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }
</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="640" viewBox="0 0 1200 640" role="img"
aria-labelledby="title desc">
<title id="title">ReMe overview</title>
<desc id="desc">A hand-drawn style overview of ReMe, showing Auto Memory plus Auto Resource, Auto Dream plus
Proactive, Auto Index plus Memory Search, and Memory as File.
</desc>
<defs>
<style>.bg { fill: #fffdf8; } .ink { stroke: #1f2430; stroke-width: 2.2; stroke-linecap: round; stroke-linejoin: round; } .thin { stroke-width: 1.6; } .dash { stroke-dasharray: 8 7; } .title { font: 700 30px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .subtitle { font: 14px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #556276; } .head { font: 700 17px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .label { font: 700 13px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #1f2430; } .text { font: 12px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #4f5c6f; } .tiny { font: 11px "Comic Sans MS", "Bradley Hand", -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #5e6a7c; } .box { fill: #ffffff; } .blue { fill: #eef7ff; } .green { fill: #f0fbf5; } .yellow { fill: #fff7e5; } .pink { fill: #fff2f6; } .violet { fill: #f4f1ff; } .mint { fill: #eefafa; } .peach { fill: #fff2ea; } .paper { fill: #f8fbff; } .tab { fill: #ffffff; } .step { stroke-dasharray: 6 5; } .arrow { fill: none; stroke: #7f8b9d; stroke-width: 1.45; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow); } .soft-arrow { fill: none; stroke: #a3adbd; stroke-width: 1.25; stroke-dasharray: 6 6; stroke-linecap: round; stroke-linejoin: round; marker-end: url(#arrow-soft); }</style>
<marker id="arrow" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto" markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#7f8b9d"/>
</marker>
<marker id="arrow-soft" markerWidth="6" markerHeight="6" refX="5" refY="2" orient="auto"
markerUnits="strokeWidth">
<path d="M0,0 L0,4 L5,2 z" fill="#a3adbd"/>
</marker>
</defs>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="44" text-anchor="middle">ReMe</text>
<text class="subtitle" x="600" y="68" text-anchor="middle">A file-native memory loop: capture, consolidate, link, search, and proactively surface what matters.</text>
<rect class="bg" x="0" y="0" width="1200" height="640"/>
<text class="title" x="600" y="44" text-anchor="middle">ReMe</text>
<text class="subtitle" x="600" y="68" text-anchor="middle">A file-native memory loop: capture, consolidate, link, search, and proactively surface what matters.</text>
<!-- Top workflow -->
<g transform="translate(40 120)">
<rect class="blue ink thin" x="0" y="0" width="300" height="260" rx="18"/>
<text class="head" x="150" y="31" text-anchor="middle">Auto Memory + Auto Resource</text>
<text class="text" x="150" y="56" text-anchor="middle">Capture sessions and resources into daily cards.</text>
<!-- Top workflow -->
<g transform="translate(40 120)">
<rect class="blue ink thin" x="0" y="0" width="300" height="260" rx="18"/>
<text class="head" x="150" y="31" text-anchor="middle">Auto Memory + Auto Resource</text>
<text class="text" x="150" y="56" text-anchor="middle" style="font-size:12px">Capture sessions and resources into daily cards.</text>
<g transform="translate(24 78)">
<rect class="blue ink thin dash" x="0" y="0" width="252" height="64" rx="13"/>
<text class="label" x="54" y="27" text-anchor="middle">Auto</text>
<text class="label" x="54" y="45" text-anchor="middle">Memory</text>
<g transform="translate(24 78)">
<rect class="blue ink thin dash" x="0" y="0" width="252" height="64" rx="13"/>
<text class="label" x="54" y="27" text-anchor="middle">Auto</text>
<text class="label" x="54" y="45" text-anchor="middle">Memory</text>
<rect class="box ink thin step" x="100" y="12" width="62" height="40" rx="9"/>
<text class="label" x="131" y="29" text-anchor="middle">Capture</text>
<text class="tiny" x="131" y="44" text-anchor="middle">chat</text>
<rect class="box ink thin step" x="100" y="12" width="62" height="40" rx="9"/>
<text class="label" x="131" y="29" text-anchor="middle">Capture</text>
<text class="tiny" x="131" y="44" text-anchor="middle">chat</text>
<rect class="yellow ink thin step" x="178" y="12" width="56" height="40" rx="9"/>
<text class="label" x="206" y="29" text-anchor="middle">Write</text>
<text class="tiny" x="206" y="44" text-anchor="middle">daily</text>
<rect class="yellow ink thin step" x="178" y="12" width="56" height="40" rx="9"/>
<text class="label" x="206" y="29" text-anchor="middle">Write</text>
<text class="tiny" x="206" y="44" text-anchor="middle">daily</text>
<path class="arrow" d="M164 32 H176"/>
<path class="arrow" d="M164 32 H176"/>
</g>
<g transform="translate(24 162)">
<rect class="green ink thin dash" x="0" y="0" width="252" height="64" rx="13"/>
<text class="label" x="54" y="27" text-anchor="middle">Auto</text>
<text class="label" x="54" y="45" text-anchor="middle">Resource</text>
<rect class="box ink thin step" x="100" y="12" width="62" height="40" rx="9"/>
<text class="label" x="131" y="29" text-anchor="middle">Read</text>
<text class="tiny" x="131" y="44" text-anchor="middle">source</text>
<rect class="yellow ink thin step" x="178" y="12" width="56" height="40" rx="9"/>
<text class="label" x="206" y="29" text-anchor="middle">Write</text>
<text class="tiny" x="206" y="44" text-anchor="middle">daily</text>
<path class="arrow" d="M164 32 H176"/>
</g>
</g>
<g transform="translate(24 162)">
<rect class="green ink thin dash" x="0" y="0" width="252" height="64" rx="13"/>
<text class="label" x="54" y="27" text-anchor="middle">Auto</text>
<text class="label" x="54" y="45" text-anchor="middle">Resource</text>
<g transform="translate(380 120)">
<rect class="yellow ink thin" x="0" y="0" width="320" height="260" rx="18"/>
<text class="head" x="160" y="32" text-anchor="middle">Auto Dream + Proactive</text>
<text class="text" x="160" y="58" text-anchor="middle" style="font-size:12px">Consolidate notes, link knowledge, surface interests.</text>
<rect class="box ink thin step" x="100" y="12" width="62" height="40" rx="9"/>
<text class="label" x="131" y="29" text-anchor="middle">Read</text>
<text class="tiny" x="131" y="44" text-anchor="middle">source</text>
<rect class="box ink thin step" x="38" y="84" width="112" height="56" rx="12"/>
<text class="label" x="94" y="108" text-anchor="middle">Extract</text>
<text class="tiny" x="94" y="126" text-anchor="middle">changed files</text>
<rect class="yellow ink thin step" x="178" y="12" width="56" height="40" rx="9"/>
<text class="label" x="206" y="29" text-anchor="middle">Write</text>
<text class="tiny" x="206" y="44" text-anchor="middle">daily</text>
<rect class="peach ink thin step" x="170" y="84" width="112" height="56" rx="12"/>
<text class="label" x="226" y="108" text-anchor="middle">Auto Link</text>
<text class="tiny" x="226" y="126" text-anchor="middle">dedupe + edges</text>
<path class="arrow" d="M164 32 H176"/>
</g>
</g>
<rect class="pink ink thin step" x="38" y="168" width="112" height="56" rx="12"/>
<text class="label" x="94" y="192" text-anchor="middle">Integrate</text>
<text class="tiny" x="94" y="210" text-anchor="middle">write digest</text>
<g transform="translate(380 120)">
<rect class="yellow ink thin" x="0" y="0" width="320" height="260" rx="18"/>
<text class="head" x="160" y="32" text-anchor="middle">Auto Dream + Proactive</text>
<text class="text" x="160" y="58" text-anchor="middle">Consolidate notes, link knowledge, surface interests.</text>
<rect class="mint ink thin step" x="170" y="168" width="112" height="56" rx="12"/>
<text class="label" x="226" y="192" text-anchor="middle">Proactive</text>
<text class="tiny" x="226" y="210" text-anchor="middle">interests.yaml</text>
<rect class="box ink thin step" x="38" y="84" width="112" height="56" rx="12"/>
<text class="label" x="94" y="108" text-anchor="middle">Extract</text>
<text class="tiny" x="94" y="126" text-anchor="middle">changed files</text>
<rect class="peach ink thin step" x="170" y="84" width="112" height="56" rx="12"/>
<text class="label" x="226" y="108" text-anchor="middle">Auto Link</text>
<text class="tiny" x="226" y="126" text-anchor="middle">dedupe + edges</text>
<rect class="pink ink thin step" x="38" y="168" width="112" height="56" rx="12"/>
<text class="label" x="94" y="192" text-anchor="middle">Integrate</text>
<text class="tiny" x="94" y="210" text-anchor="middle">write digest</text>
<rect class="mint ink thin step" x="170" y="168" width="112" height="56" rx="12"/>
<text class="label" x="226" y="192" text-anchor="middle">Proactive</text>
<text class="tiny" x="226" y="210" text-anchor="middle">interests.yaml</text>
<path class="arrow" d="M152 112 H168"/>
<path class="arrow" d="M226 142 V166"/>
<path class="arrow" d="M168 196 H152"/>
<path class="arrow" d="M94 166 V142"/>
</g>
<g transform="translate(730 120)">
<rect class="violet ink thin" x="0" y="0" width="430" height="260" rx="18"/>
<text class="head" x="215" y="31" text-anchor="middle">Auto Index + Memory Search</text>
<text class="text" x="215" y="56" text-anchor="middle">Keep indexes fresh, recall chunks, then follow links.</text>
<g transform="translate(24 86)">
<rect class="blue ink thin dash" x="0" y="0" width="178" height="136" rx="14"/>
<text class="label" x="89" y="24" text-anchor="middle">Auto Index</text>
<rect class="box ink thin step" x="18" y="46" width="64" height="58" rx="10"/>
<text class="label" x="50" y="70" text-anchor="middle">Watch</text>
<text class="tiny" x="50" y="88" text-anchor="middle">changes</text>
<rect class="yellow ink thin step" x="98" y="46" width="64" height="58" rx="10"/>
<text class="label" x="130" y="70" text-anchor="middle">Build</text>
<text class="tiny" x="130" y="88" text-anchor="middle">index</text>
<path class="arrow" d="M84 75 H96"/>
<path class="arrow" d="M152 112 H168"/>
<path class="arrow" d="M226 142 V166"/>
<path class="arrow" d="M168 196 H152"/>
<path class="arrow" d="M94 166 V142"/>
</g>
<g transform="translate(228 86)">
<rect class="green ink thin dash" x="0" y="0" width="178" height="136" rx="14"/>
<text class="label" x="89" y="24" text-anchor="middle">Memory Search</text>
<g transform="translate(730 120)">
<rect class="violet ink thin" x="0" y="0" width="430" height="260" rx="18"/>
<text class="head" x="215" y="31" text-anchor="middle">Auto Index + Memory Search</text>
<text class="text" x="215" y="56" text-anchor="middle" style="font-size:12px">Keep indexes fresh, recall chunks, then follow links.</text>
<rect class="box ink thin step" x="18" y="46" width="64" height="58" rx="10"/>
<text class="label" x="50" y="70" text-anchor="middle">Recall</text>
<text class="tiny" x="50" y="88" text-anchor="middle">BM25 + vec</text>
<g transform="translate(24 86)">
<rect class="blue ink thin dash" x="0" y="0" width="178" height="136" rx="14"/>
<text class="label" x="89" y="24" text-anchor="middle">Auto Index</text>
<rect class="mint ink thin step" x="98" y="46" width="64" height="58" rx="10"/>
<text class="label" x="130" y="70" text-anchor="middle">Expand</text>
<text class="tiny" x="130" y="88" text-anchor="middle">links</text>
<rect class="box ink thin step" x="18" y="46" width="64" height="58" rx="10"/>
<text class="label" x="50" y="70" text-anchor="middle">Watch</text>
<text class="tiny" x="50" y="88" text-anchor="middle">changes</text>
<path class="arrow" d="M84 75 H96"/>
<rect class="yellow ink thin step" x="98" y="46" width="64" height="58" rx="10"/>
<text class="label" x="130" y="70" text-anchor="middle">Build</text>
<text class="tiny" x="130" y="88" text-anchor="middle">index</text>
<path class="arrow" d="M84 75 H96"/>
</g>
<g transform="translate(228 86)">
<rect class="green ink thin dash" x="0" y="0" width="178" height="136" rx="14"/>
<text class="label" x="89" y="24" text-anchor="middle">Memory Search</text>
<rect class="box ink thin step" x="18" y="46" width="64" height="58" rx="10"/>
<text class="label" x="50" y="70" text-anchor="middle">Recall</text>
<text class="tiny" x="50" y="88" text-anchor="middle" style="font-size:10px">BM25 + vec</text>
<rect class="mint ink thin step" x="98" y="46" width="64" height="58" rx="10"/>
<text class="label" x="130" y="70" text-anchor="middle">Expand</text>
<text class="tiny" x="130" y="88" text-anchor="middle">links</text>
<path class="arrow" d="M84 75 H96"/>
</g>
<path class="arrow" d="M204 161 H226"/>
</g>
<path class="arrow" d="M204 161 H226"/>
</g>
<path class="arrow" d="M340 272 H380"/>
<path class="arrow" d="M340 272 H380"/>
<!-- Bottom foundation -->
<rect class="box ink" x="40" y="408" width="1120" height="202" rx="24"/>
<text class="head" x="600" y="442" text-anchor="middle">Memory as File</text>
<text class="text" x="600" y="466" text-anchor="middle">Every memory is readable, editable, indexable, linkable, and auditable as files.</text>
<!-- Bottom foundation -->
<rect class="box ink" x="40" y="408" width="1120" height="202" rx="24"/>
<text class="head" x="600" y="442" text-anchor="middle">Memory as File</text>
<text class="text" x="600" y="466" text-anchor="middle">Every memory is readable, editable, indexable, linkable, and auditable as files.</text>
<g transform="translate(94 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-12" y="14" width="44" height="22"/>
<text class="label" x="10" y="30" text-anchor="middle">.jsonl</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">session/</text>
<text class="tiny" x="106" y="39">source conversations</text>
</g>
<g transform="translate(94 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-12" y="14" width="44" height="22"/>
<text class="label" x="10" y="30" text-anchor="middle">.json</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">session/</text>
<text class="tiny" x="106" y="39">raw session logs</text>
</g>
<g transform="translate(344 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.src</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">resource/</text>
<text class="tiny" x="106" y="39">original source material</text>
</g>
<g transform="translate(344 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.md</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">resource/</text>
<text class="tiny" x="106" y="39">raw material with source</text>
</g>
<g transform="translate(604 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.md</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">daily/</text>
<text class="tiny" x="106" y="39">working memory cards</text>
</g>
<g transform="translate(604 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.md</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">daily/</text>
<text class="tiny" x="106" y="39">working memory cards</text>
</g>
<g transform="translate(850 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.md</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">digest/</text>
<text class="tiny" x="106" y="39">long-term knowledge nodes</text>
</g>
<g transform="translate(850 500)">
<rect class="paper ink thin" x="10" y="6" width="62" height="52"/>
<rect class="paper ink thin" x="5" y="3" width="62" height="52"/>
<rect class="paper ink thin" x="0" y="0" width="62" height="52"/>
<rect class="tab ink thin" x="-8" y="14" width="34" height="22"/>
<text class="label" x="9" y="30" text-anchor="middle">.md</text>
<line class="ink thin" x1="26" y1="28" x2="50" y2="28"/>
<line class="ink thin" x1="22" y1="42" x2="52" y2="42"/>
<text class="label" x="106" y="19">digest/</text>
<text class="tiny" x="106" y="39">long-term knowledge nodes</text>
</g>
<path class="soft-arrow" d="M190 380 V408"/>
<path class="soft-arrow" d="M540 380 V408"/>
<path class="soft-arrow" d="M945 380 V408"/>
<path class="soft-arrow" d="M190 380 V408"/>
<path class="soft-arrow" d="M540 380 V408"/>
<path class="soft-arrow" d="M945 380 V408"/>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

View file

@ -1,7 +1,8 @@
# Auto Dream
`auto_dream` 是 ReMe 的 daily 到 digest 的长期记忆沉淀流程。它扫描指定日期的 daily 输入,只处理相对上次 dream
发生变化的文件,把值得长期保留的内容抽取成 memory units整合进 `digest/`,再生成当天可供主动提醒使用的 `interests.yaml`
`auto_dream` 是 ReMe 的 daily 到 digest 的长期记忆沉淀流程。它默认扫描目标日期及前一天的 daily 输入,只处理相对上次 dream
发生变化的文件,从整个扫描窗口中抽取少量高价值 memory units整合进 `digest/`,再生成目标日期可供主动提醒使用的
`interests.yaml`
<p align="center">
<img src="../figure/auto-dream-and-proactive.svg" alt="ReMe Auto Dream and Proactive 从 daily 到 digest 再到 proactive 的流程" width="92%">
@ -25,6 +26,12 @@ auto_dream:
hint:
type: string
default: ""
scan_days:
type: integer
default: 2
max_units:
type: integer
default: 5
topic_count:
type: integer
default: 3
@ -35,6 +42,8 @@ auto_dream:
- backend: dream_extract_step
file_catalog: dream
topic_session_id: interests
scan_days: 2
max_units: 5
- backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
@ -45,32 +54,36 @@ auto_dream:
参数含义:
| 参数 | 作用 |
|------------------------|------------------------------------------------|
| `date` | 要处理的日期,格式为 `YYYY-MM-DD`。为空时使用应用时区中的今天。 |
| `hint` | 调用方给抽取和整合阶段的额外指导。 |
| `topic_count` | 最终写入 `interests.yaml` 的 topic 上限,默认 3。 |
| 参数 | 作用 |
|------------------------|-------------------------------------------------------------------|
| `date` | 要处理的日期,格式为 `YYYY-MM-DD`。为空时使用应用时区中的今天。 |
| `hint` | 调用方给抽取和整合阶段的额外指导。 |
| `scan_days` | 以 `date` 结尾的最近日期窗口;默认扫描 2 天,最小为 1。 |
| `max_units` | 一次最多抽取多少个可复用 unit默认 5。 |
| `topic_count` | 最终写入 `interests.yaml` 的 topic 上限,默认 3。 |
| `topic_diversity_days` | 选择 topic 时参考过去多少天的 `interests.yaml` 避免重复,默认 7。 |
## 输入和输出
输入来自指定日期的 daily markdown
输入来自以指定日期结尾的最近 `scan_days` 天 daily markdown。例如 `date=2026-06-20``scan_days=2` 时会扫描
```text
daily/<date>.md
daily/<date>/**/*.md
daily/2026-06-19.md
daily/2026-06-19/**/*.md
daily/2026-06-20.md
daily/2026-06-20/**/*.md
```
`daily/<date>/interests.yaml` 不作为抽取输入,避免上一轮主动主题反过来污染下一轮抽取。
扫描窗口内的 `daily/<date>/interests.yaml` 不作为抽取输入,避免上一轮主动主题反过来污染下一轮抽取。最终 topic 只写入目标日期。
主要输出有三类:
| 输出 | 说明 |
|-------------------------------------|-------------------------------------|
| `digest/procedure/*.md` | 方法、流程、runbook、可执行经验。 |
| `digest/personal/*.md` | 用户、团队、项目相关的偏好、事实、长期上下文。 |
| `digest/wiki/*.md` | 通用知识、概念、观察、决策先例。 |
| `daily/<date>/interests.yaml` | 当天值得上层 Agent 主动关注的兴趣主题。 |
| 输出 | 说明 |
|--------------------------------|---------------------------------------------------|
| `digest/procedure/*.md` | 方法、流程、runbook、可执行经验。 |
| `digest/personal/*.md` | 用户、团队、项目相关的偏好、事实、长期上下文。 |
| `digest/wiki/*.md` | 通用知识、概念、观察、决策先例。 |
| `daily/<date>/interests.yaml` | 当天值得上层 Agent 主动关注的兴趣主题。 |
| `metadata/file_catalog/dream*` | dream 专用 catalog用于判断 daily 输入是否变化。 |
## 四个阶段
@ -79,16 +92,18 @@ daily/<date>/**/*.md
`dream_extract_step` 做三件事:
1. 刷新当天索引页 `daily/<date>.md`
2. 扫描 `daily/<date>.md` `daily/<date>/**/*.md`,与 `file_catalog: dream` 中记录的 mtime 对比。
3. 只把 changed files 交给 LLM全局抽取两类结构化结果`units``topics`
1. 刷新扫描窗口内每天的索引页 `daily/<date>.md`
2. 扫描这些日期的索引页`daily/<date>/**/*.md`,与 `file_catalog: dream` 中记录的 mtime 对比。
3. 只把 changed files 一起交给 LLM全局抽取两类结构化结果`units``topics`
`units` 是准备沉淀进 digest 的长期记忆单元,包含 `name``bucket``summary``paths``bucket` 只允许 `procedure`
`personal``wiki`;未知值会路由到 `wiki`
`units` 是准备沉淀进 digest 的长期记忆单元,包含 `name``bucket``summary``paths`。一次最多返回 `max_units`
个,抽取器会优先合并指向同一抽象的跨文件证据,并丢弃短暂提及、逐文件摘要和缺少复用价值的弱候选。`bucket` 只允许
`procedure``personal``wiki`;未知值会路由到 `wiki`
`topics` 是当天主动兴趣候选,包含 `title``reason``evidence``keywords``paths`,后续由 Topics 阶段再筛选。
如果没有 changed files流程会提前成功结束后续抽取工作如果有变化但没有配置 LLMExtract 会失败,因为抽取依赖 LLM。
如果没有 changed filesExtract 会成功返回空 unitsIntegrate 随后没有 unit 可处理Topics 保留目标日期已有的 topicsFinish
仍会正常汇总 catalog。如果有变化但没有配置 LLMExtract 会失败,因为抽取依赖 LLM。
### 2. Integrate
@ -101,14 +116,17 @@ node_search, read, frontmatter_read, write, edit, frontmatter_update
这一阶段承担 `auto_link` 的核心职责:先用 `node_search` 在 digest 节点级召回相似或相关节点,再判断是新建还是更新,最后把来源和相关
digest 节点写成 wikilink。具体召回、去重和写边规则见 [Auto Link](./auto_link.md)。
Extract 已经承担“是否值得长期记住”的过滤,因此 Integrate 不提供 `SKIP` 动作:每个进入本阶段的 unit 都应落到且只落到一个
digest 节点。新增与更新都必须保留来源,并把相关 digest 链接写进有上下文的句子;不能只写裸 Wikilink 或独立的关系字段。
整合动作只有四种:
| 动作 | 含义 |
|---------------|-------------------------|
| `CREATE` | 没有相同抽象,创建新的 digest 节点。 |
| `CORROBORATE` | 同一记忆再次出现,追加来源或强化表述。 |
| 动作 | 含义 |
|---------------|------------------------------------------------|
| `CREATE` | 没有相同抽象,创建新的 digest 节点。 |
| `CORROBORATE` | 同一记忆再次出现,追加来源或强化表述。 |
| `REFINE` | 新材料补充了边界、步骤、前提、适用范围或细节。 |
| `CORRECT` | 新材料修正了旧节点的错误、遗漏或冲突。 |
| `CORRECT` | 新材料修正了旧节点的错误、遗漏或冲突。 |
Integrate 成功的 unit 会记录到 `integrate_results`;失败的 unit 会进入 `failed_units`,其来源路径会进入 `failed_paths`
Finish 阶段不会 checkpoint 失败路径,保证下次还能重试。
@ -121,7 +139,7 @@ Finish 阶段不会 checkpoint 失败路径,保证下次还能重试。
```text
daily/<date>/interests.yaml
daily/<previous-date>/interests.yaml
daily/<过去 topic_diversity_days 天中的每一天>/interests.yaml
```
同一天已有 topics 会被保留,最近 `topic_diversity_days` 天出现过的相似主题会被去重。默认最多写 3 个 topic。配置了 LLM 时会让
@ -149,7 +167,7 @@ topics:
`dream_finish_step` 负责收尾:
1. 将成功处理的 changed paths 写入 `file_catalog: dream`
2. 将 `daily/<date>/interests.yaml``daily/<date>.md` 也写入 catalog。
2. 将目标日期的 `daily/<date>/interests.yaml` 和扫描窗口内每个已刷新的 day-index 页也写入 catalog。
3. 如果有 upsert 或 delete持久化 dream catalog。
4. 返回包含 scanned、changed、integrated、topics、checkpoint 等计数的摘要。
@ -169,6 +187,12 @@ reme auto_dream date=2026-06-20
reme auto_dream date=2026-06-20 hint="优先沉淀工程决策和长期偏好"
```
覆盖默认扫描窗口和 unit 上限:
```bash
reme auto_dream date=2026-06-20 scan_days=3 max_units=8
```
也可以在配置中把同一组 step 放进 `cron` job例如每天凌晨运行
```yaml
@ -189,7 +213,8 @@ jobs:
`auto_dream` 只消费 daily 输入,不改写 daily 正文。daily 是事实和现场记录digest 才是抽象后的长期记忆层。
`digest` 不是原文复制。正文应保留可复用抽象,细节通过 Sources 章节中的 `- [[daily/<date>/...]]` 指回来源。链接写法遵循
`digest` 不是原文复制。正文应保留可复用抽象Sources 章节用带上下文的完整句子指回来源,例如
`该决策记录在 [[daily/<date>/decision.md]] 中。`链接写法遵循
[Memory as File](./memory_as_file.md) 中的 workspace-relative wikilink 语义。
`auto_dream` 不凭空生成总览。只有 daily 输入中确实出现、并被抽取为 unit 或 topic 的内容,才会进入 digest 或

View file

@ -26,12 +26,12 @@ Integrate 阶段对每个 unit 独立运行。一个 unit 只落到一个目标
`auto_link` 解决的是写入时的图谱质量问题:
| 问题 | 处理方式 |
|--------------|----------------------------------------------------|
| 已有相同记忆 | 召回后更新旧节点,而不是重复创建。 |
| 新旧材料有关联 | 在正文中写入 workspace-relative wikilink。 |
| 问题 | 处理方式 |
|-------------------|------------------------------------------------------|
| 已有相同记忆 | 召回后更新旧节点,而不是重复创建。 |
| 新旧材料有关联 | 在正文中写入 workspace-relative wikilink。 |
| digest 与来源断开 | 在 `## Sources` 章节加入指向 daily/resource 的链接。 |
| 节点只有孤立正文 | 在 CREATE 和 UPDATE 时都补充相关 digest 节点链接。 |
| 节点只有孤立正文 | 在 CREATE 和 UPDATE 时都补充相关 digest 节点链接。 |
## 工具链
@ -60,22 +60,22 @@ Agent 先用 unit 的触发条件、动词、名词、同义词和可能的 fail
召回结果会被内部分成三类:
| 分类 | 含义 | 后续动作 |
|--------------------|-----------------------------|----------------|
| `same_abstraction` | 触发条件或抽象本质相同,内容实质重叠。 | 作为 UPDATE 目标。 |
| 分类 | 含义 | 后续动作 |
|--------------------|--------------------------------------------------------|---------------------|
| `same_abstraction` | 触发条件或抽象本质相同,内容实质重叠。 | 作为 UPDATE 目标。 |
| `related` | 相邻流程、前置条件、失败模式、概念、偏好或上下游知识。 | 写入正文 wikilink。 |
| `unrelated` | 只是表面相似或无关。 | 忽略。 |
| `unrelated` | 只是表面相似或无关。 | 忽略。 |
### 2. 选择写入动作
每个 unit 必须选择一个动作:
| 动作 | 链接含义 |
|---------------|-----------------------------------------------------|
| 动作 | 链接含义 |
|---------------|----------------------------------------------------------------------------|
| `CREATE` | 写入新的 `digest/<bucket>/<slug>.md`,并在新正文里加入来源和相关节点链接。 |
| `CORROBORATE` | 同一抽象再次出现,追加来源链接,必要时强化描述。 |
| `REFINE` | 新材料扩展了旧节点,把补充内容插入合适段落,并保留旧链接。 |
| `CORRECT` | 新材料修正旧节点,用来源链接标出修正依据。 |
| `CORROBORATE` | 同一抽象再次出现,追加来源链接,必要时强化描述。 |
| `REFINE` | 新材料扩展了旧节点,把补充内容插入合适段落,并保留旧链接。 |
| `CORRECT` | 新材料修正旧节点,用来源链接标出修正依据。 |
UPDATE 必须尽量只增不删:不要删除已有 wikilink 或来源条目。这是为了让后续图谱索引和检索不会丢边。
@ -86,11 +86,12 @@ UPDATE 必须尽量只增不删:不要删除已有 wikilink 或来源条目。
```markdown
## Sources
- [[daily/2026-06-20/session.md]]
- [[resource/2026-06-20/paper.md]]
该决策记录在 [[daily/2026-06-20/session.md]] 中,支撑它的技术证据来自
[[resource/2026-06-20/paper.md]]
```
这些边表示 digest 节点的证据来源。纯文本描述不算来源边,因为只有 wikilink 能被 file graph 稳定解析。更完整的 wikilink
这些边表示 digest 节点的证据来源。纯文本描述不算来源边,因为只有 wikilink 能被 file graph 稳定解析;外层完整句子还必须说明每个来源支持什么,
裸 Wikilink 单独成行不是合法的 Integrate 输出。更完整的 wikilink
解析规则见 [Memory as File](./memory_as_file.md#wikilink)。
### 4. 写 digest 关联边
@ -107,11 +108,11 @@ digest 之间的关联使用完整 workspace-relative 路径,并自然织入
`auto_link` 的规则会随 unit bucket 调整写入形态:
| Bucket | 写入重点 |
|-------------|---------------------------------------------|
| Bucket | 写入重点 |
|-------------|--------------------------------------------------------------------------------|
| `procedure` | 写成 runbook触发条件、步骤、输入、失败模式。链接前置流程、子步骤、相关偏好。 |
| `personal` | 写用户、团队、项目特定事实或偏好。链接相关项目、习惯、决策背景。 |
| `wiki` | 写通用知识、原则、观察、决策先例。链接概念、方法、相邻知识。 |
| `personal` | 写用户、团队、项目特定事实或偏好。链接相关项目、习惯、决策背景。 |
| `wiki` | 写通用知识、原则、观察、决策先例。链接概念、方法、相邻知识。 |
无论 bucket 是什么,都要保留来源边,并尽量把召回到的相关 digest 节点织入正文。
@ -119,9 +120,9 @@ digest 之间的关联使用完整 workspace-relative 路径,并自然织入
`auto_link` 使用的是 `node_search`,不是面向问答的 `search`
| 能力 | 用途 |
|---------------|-------------------------------------------|
| `search` | 面向外部问答,返回 chunk并可展开上下游 link context。 |
| 能力 | 用途 |
|---------------|---------------------------------------------------------------------|
| `search` | 面向外部问答,返回 chunk并可展开上下游 link context。 |
| `node_search` | 面向 dream 集成,只召回 digest 节点级摘要,用来判断去重和相关链接。 |
这个边界很重要Integrate 阶段需要的是“是否已有相同抽象,以及应该链接哪些节点”,而不是直接把大量正文片段塞进上下文。

View file

@ -1,7 +1,7 @@
# Auto Memory
Auto Memory 是 ReMe 的对话记忆入口:每段对话先按 `session_id` 沉淀成一张 daily 记忆卡片,再由当天的 `YYYY-MM-DD.md`
统一索引。它负责把“聊过”变成“记住”,并把原始对话留好出处
Auto Memory 是 ReMe 的对话记忆入口:在目标日期内,它用 `session_id` 定位或更新最多一张 daily 记忆卡片,文件名由 Agent
根据内容生成简洁的主题或事件名,再由当天的 `YYYY-MM-DD.md` 统一索引。它负责把“聊过”变成“记住”,并保留可追溯的对话记录
<p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory 与 Auto Resource 写入 daily 记忆卡片的流程" width="92%">
@ -11,9 +11,9 @@ Auto Memory 是 ReMe 的对话记忆入口:每段对话先按 `session_id` 沉
```text
Conversation
├─ step 1: daily/YYYY-MM-DD/<session_id>.md # 每段对话先成卡片
├─ step 2: daily/YYYY-MM-DD.md # 当天索引再串起来
└─ source: session/dialog/<session_id>.jsonl # 原始对话
├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # 每个 session 一张主题卡片
├─ step 2: daily/YYYY-MM-DD.md # 当天索引再串起来
└─ source: session/dialog/<session_id>.jsonl # 对话来源记录
```
## 它记录什么
@ -37,27 +37,29 @@ workspace/
daily/
2026-06-20.md
2026-06-20/
session-a.md
session-b.md
login-refactor-decision.md
retrieval-regression.md
```
其中 `daily/2026-06-20/session-a.md``daily/2026-06-20/session-b.md` 是不同对话整理出的记忆卡片,
`daily/2026-06-20.md` 是当天索引页。资源文件也会进入同一个 daily 记忆层,见 [Auto Resource](./auto_resource.md)。
日期目录下的两个文件是不同对话整理出的主题卡片,`daily/2026-06-20.md` 是当天索引页。资源文件也会进入
同一个 daily 记忆层,见 [Auto Resource](./auto_resource.md)。
当调用时带上 `session_id`Auto Memory 会按这个 id 单独记录这段对话
当调用时带上 `session_id`Auto Memory 会通过 frontmatter 用它定位卡片Agent 则通过 `name` 决定可读文件名
```text
daily/2026-06-20/session-a.md
```yaml
name: login-refactor-decision
session_id: session-a
source_conversation: "[[session/dialog/session-a.jsonl]]"
```
这样不同对话不会混在一起。一次需求讨论、一次问题排查、一次文档修改,都可以拥有自己的记忆卡片。以后想知道这一天发生了什么,先看
`YYYY-MM-DD.md`;想看某段对话沉淀了什么,再进入对应的 `<session_id>.md`
这样既能分开不同对话,又不必把不透明的 ID 当文件名。更新时会按 `session_id``source_conversation` 找到旧卡片;如果 Agent
提供了更好的 frontmatter `name`,系统可重命名并重定向入链。查看某天内容时从 `YYYY-MM-DD.md` 开始
## 同时保存原始信息
整理后的 daily note 负责“好读”,原始对话负责“可信”。
整理后的 daily note 负责“好读”,过滤后的对话来源记录负责“可信”。
Auto Memory 在生成记忆卡片的同时,也会保存原始会话
Auto Memory 在生成记忆卡片的同时,也会保存对话来源消息
```text
session/
@ -66,11 +68,12 @@ session/
session-b.jsonl
```
daily note 会指向对应的原始对话。需要核对某条记忆时,可以顺着链接回到当时的完整上下文。
daily note 会指向对应的对话记录。持久化时会排除 tool-result block 和 base64 data block避免召回记忆或二进制负载在后续流程中被误当成
用户提供的证据。
## 消息时间
Auto Memory 会在 prompt 和原始会话 JSONL 中保留每条消息的 `created_at`。导入历史对话或 benchmark 数据时,建议为每条
Auto Memory 会在 prompt 和对话来源 JSONL 中保留每条已保留消息的 `created_at`。导入历史对话或 benchmark 数据时,建议为每条
message 提供真实发生时间,避免模型把事件时间误解为运行时间:
```bash
@ -85,7 +88,7 @@ reme auto_memory \
为了兼容常见数据集字段,`auto_memory` 也会在缺少 `created_at` 时读取 `time_created``timestamp``createdAt`
`timeCreated``created_time`。这些字段可以放在 message 顶层,也可以放在 `metadata` 中。
当调用没有显式传入 `date`Auto Memory 会使用消息中最的有效 `created_at` 日期作为 daily note 日期;如果消息没有有效时间,
当调用没有显式传入 `date`Auto Memory 会使用消息中最的有效 `created_at` 日期作为 daily note 日期;如果消息没有有效时间,
则回退到当前日期。历史导入也可以显式指定目标日期:
```bash

View file

@ -1,7 +1,8 @@
# Auto Resource `Beta`
Auto Resource 是 ReMe 的资源解读入口,目前处于 **Beta**。资源文件先按日期进入 `resource/`,再被解读成 daily
资源卡片;卡片文件名由 LLM 生成的 frontmatter `name` 决定,并通过 `source_resource` 追溯原始文件。
Auto Resource 是 ReMe 的资源解读入口,目前处于 **Beta**。资源文件先进入 `resource/`(推荐按日期放置),再被解读成 daily
资源卡片;卡片文件名由 LLM 生成的 frontmatter `name` 决定,并通过 `source_resource`
追溯原始文件。
<p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory 与 Auto Resource 写入 daily 记忆卡片的流程" width="92%">
@ -11,7 +12,7 @@ Auto Resource 是 ReMe 的资源解读入口,目前处于 **Beta**。资源文
[Auto Memory](./auto_memory.md)。
```text
resource/YYYY-MM-DD/<resource_file>
resource/[YYYY-MM-DD/]<resource_file>
├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # 资源解读卡片
├─ step 2: source_resource 指回原始资源
└─ step 3: daily/YYYY-MM-DD.md # 当天索引再串起来
@ -31,13 +32,15 @@ resource/YYYY-MM-DD/<resource_file>
## 原始资料入口
Auto Resource 以 `resource/` 作为原始资料入口。资源需要按日期放置,这个日期会决定它进入哪一天的 daily 记忆层。
Auto Resource 以 `resource/` 作为原始资料入口。推荐按日期放置,目录日期会决定它进入哪一天的 daily 记忆层;也支持直接放在
`resource/` 根目录,此时使用应用时区中的今天。
示例目录:
```text
workspace/
resource/
quick-note.txt # 进入今天的 daily
2026-06-20/
market-report.md
meeting-notes.csv
@ -47,8 +50,8 @@ workspace/
## 资源卡片
每个资源文件会生成一张 daily 资源卡片。创建时先使用资源文件 stem 作为临时路径Agent 写入后,系统会根据
frontmatter `name` 重命名文件:
每个资源文件会生成一张 daily 资源卡片。创建时先使用资源文件 stem 作为临时路径Agent 写入后,系统会根据 frontmatter `name`
重命名文件:
```text
resource/2026-06-20/market-report.md
@ -62,8 +65,8 @@ daily/2026-06-20/市场报告要点.md
source_resource: "[[resource/2026-06-20/market-report.md]]"
```
如果资源文件更新Auto Resource 会通过 `source_resource` 找到对应卡片并更新;如果资源文件删除,对应的 daily note
也会被清理。旧版本按 stem 生成的 `daily/YYYY-MM-DD/<resource_stem>.md` 仍作为 fallback 兼容。
如果资源文件更新Auto Resource 会通过 `source_resource` 找到对应卡片并更新;如果资源文件删除,对应的 daily note 也会被清理。旧版本按
stem 生成的 `daily/YYYY-MM-DD/<resource_stem>.md` 仍作为 fallback 兼容。
## 当天索引
@ -83,9 +86,9 @@ daily/
解读后的 daily note 负责“好读”,原始资源负责“可信”。
Auto Resource 不会把原始文件挪走:它仍然留在 `resource/YYYY-MM-DD/`。这样,文本资料会进入 daily 记忆流,原始文件也始终保留在它来时的位置。
Auto Resource 不会把原始文件挪走:它仍然留在 `resource/` 下的原路径。这样,文本资料会进入 daily 记忆流,原始文件也始终保留在它来时的位置。
## 后续流向
Auto Resource 只生成 daily 层的资源解读。要把资源中的长期知识沉淀进 `digest/`,使用 [Auto Dream](./auto_dream.md)要检索原始资源、
daily 卡片和 digest 节点,使用 [Memory Search](./memory_search.md)。
Auto Resource 只生成 daily 层的资源解读。要把资源中的长期知识沉淀进 `digest/`,使用 [Auto Dream](./auto_dream.md)默认实时检索会
索引 daily 卡片和 digest 节点。若还要直接检索原始资源文件,可运行 `reme reindex`,详见 [Memory Search](./memory_search.md)。

View file

@ -37,7 +37,11 @@ ReMe 的核心代码位于:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,full]"
pip install -e reme_studio -e ".[dev,full]"
cd reme_studio
npm ci
npm run build:static
cd ..
pre-commit install
```
@ -71,14 +75,14 @@ CLI / Client -> Service -> Application -> Job -> Step -> Component / Workspace
根据改动类型选择合适的入口:
| 改动类型 | 主要位置 | 建议 |
|------------|-----------------------------------------------------|---------------------------------------------------------------------------|
| 配置或启动行为 | `reme/config/``reme/application.py``reme/reme.py` | 保持默认配置可运行,避免破坏现有 CLI、HTTP 和 MCP 入口 |
| 组件能力 | `reme/components/` | 优先复用 `BaseComponent`、registry 和上下文对象 |
| Job 或 Step | `reme/components/job/``reme/steps/` | 遵照 [ReMe 代码框架](./framework.md) 的 Job -> Step 范式,保持请求、响应 schema 清晰,并补充对应测试 |
| 数据结构 | `reme/schema/``reme/enumeration/` | 注意序列化兼容性和已有 front matter、wikilink 语义 |
| 工具函数 | `reme/utils/` | 保持函数边界小,并用单元测试覆盖边界情况 |
| 用户文档 | `docs/zh/``README.md` | 当用户可见行为变化时同步更新文档 |
| 改动类型 | 主要位置 | 建议 |
|----------------|-------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| 配置或启动行为 | `reme/config/``reme/application.py``reme/reme.py` | 保持默认配置可运行,避免破坏现有 CLI、HTTP 和 MCP 入口 |
| 组件能力 | `reme/components/` | 优先复用 `BaseComponent`、registry 和上下文对象 |
| Job 或 Step | `reme/components/job/``reme/steps/` | 遵照 [ReMe 代码框架](./framework.md) 的 Job -> Step 范式,保持请求、响应 schema 清晰,并补充对应测试 |
| 数据结构 | `reme/schema/``reme/enumeration/` | 注意序列化兼容性和已有 front matter、wikilink 语义 |
| 工具函数 | `reme/utils/` | 保持函数边界小,并用单元测试覆盖边界情况 |
| 用户文档 | `docs/zh/``README.md` | 当用户可见行为变化时同步更新文档 |
如果改动涉及 LLM、embedding、外部服务、文件监听或后台任务请同时说明依赖条件、失败行为和本地验证方式。
@ -199,7 +203,7 @@ docs/
- Bugs 和功能请求:[GitHub Issues](https://github.com/agentscope-ai/ReMe/issues)
- 项目主页:[GitHub Repository](https://github.com/agentscope-ai/ReMe)
- 文档站点:[https://reme.agentscope.io/](https://reme.agentscope.io/)
- 文档站点:[https://reme.agentscope.io](https://reme.agentscope.io)
---

View file

@ -2,8 +2,8 @@
## 1. 总览
ReMe 的运行时可以理解为:**配置驱动的 Application 把组件和 Job 装配起来Service 把可服务的 Job 暴露给 CLI、HTTP 或 MCPJob
再按顺序执行 Step**。
ReMe 的运行时可以理解为:**配置驱动的 Application 把组件和 Job 装配起来Service 把可服务的 Job 暴露给 CLI、HTTP 或
MCPJob 再按顺序执行 Step**。
<p align="center">
<img src="../figure/framework-structure.svg" alt="ReMe 代码框架结构CLI、Service、Application、Job、Step 与 Component" width="92%">
@ -33,16 +33,16 @@ flowchart LR
核心分层:
| 层 | 主要目录 | 职责 |
|-------------|----------------------------|--------------------------------------------------------------|
| CLI | `reme/reme.py` | 解析命令;`start` 启动服务;其他 action 通过 client 调用服务 |
| Service | `reme/components/service/` | 把 Job 注册成 HTTP endpoint 或 MCP tool |
| Application | `reme/application.py` | 读取配置后的对象装配、依赖拓扑启动、关闭、Job 调用 |
| Job | `reme/components/job/` | 编排一组 Step决定同步、流式、后台、定时运行方式 |
| Step | `reme/steps/` | 业务原子操作,例如读写文件、检索、索引、自进化 |
| 层 | 主要目录 | 职责 |
|-------------|----------------------------|---------------------------------------------------------------------------|
| CLI | `reme/reme.py` | 解析命令;`start` 启动服务;其他 action 通过 client 调用服务 |
| Service | `reme/components/service/` | 把 Job 注册成 HTTP endpoint 或 MCP tool |
| Application | `reme/application.py` | 读取配置后的对象装配、依赖拓扑启动、关闭、Job 调用 |
| Job | `reme/components/job/` | 编排一组 Step决定同步、流式、后台、定时运行方式 |
| Step | `reme/steps/` | 业务原子操作,例如读写文件、检索、索引、自进化 |
| Component | `reme/components/` | 可复用基础设施,例如 file_store、file_graph、keyword_index、agent_wrapper |
| Schema | `reme/schema/` | `Request``Response``FileChunk``FileNode`、配置模型等数据结构 |
| Config | `reme/config/` | 默认 YAML 配置和命令行覆盖解析 |
| Config | `reme/config/` | 默认 YAML 配置和命令行覆盖解析 |
## 2. 目录结构
@ -50,11 +50,12 @@ flowchart LR
reme/
reme.py # CLI 入口
application.py # Application 装配与生命周期
plugin.py # 已安装插件契约与 entry-point loader
config/
default.yaml # 默认 service / jobs / components
config_parser.py # config=、dot notation、env 占位符解析
components/
component_registry.py # 全局注册表 R
component_registry.py # backend 注册表及 Application 局部副本
base_component.py # ComponentMixin / BaseComponent / bind 依赖声明
runtime_context.py # 单次 Job 执行上下文
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
@ -63,18 +64,25 @@ reme/
file_store/ # 文件索引协调层
file_graph/ # wikilink 图谱
keyword_index/ # BM25 等关键词索引
file_chunker/ # Markdown / 默认文本分块
file_chunker/ # Markdown / JSON / JSONL / 通用文本分块
file_catalog/ # 变更 checkpoint
as_llm/, as_embedding/ # 模型封装
agent_wrapper/ # AgentScope / Claude Code wrapper
agent_wrapper/ # AgentScope / Claude Code / Codex wrapper
steps/
base_step.py # BaseStep、Ref、dispatch_steps
common/ # version、help、health_check、demo
common/ # version、help、health_check、status、chat
benchmark/ # LongMemEval / BEAM 评测步骤
cookbook/ # 内置 cookbook 支持步骤
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory、auto_resource、auto_dream、proactive
transfer/ # upload/download/ingest
channel/ # MCP channel 工具
transfer/ # upload/download
plugins/
auto-fin/ # 独立发布的示例插件
daily_paper/ # 独立发布的论文研究插件
integrations/
claude_code/ # Claude Code 适配器及 marketplace
hermes_agent/ # Hermes Agent memory provider 适配器
```
默认 workspace 目录由 `ApplicationConfig` 定义:
@ -82,7 +90,8 @@ reme/
```text
<workspace_dir>/
metadata/ # file_store、file_graph、keyword_index、file_catalog 等持久状态
session/ # Agent session 与原始对话
session/ # 记忆工作流使用的对话来源记录
mem_session/ # Agent wrapper 生成的 session 和配置
resource/ # 外部资源
daily/ # 浅加工记忆
digest/ # 长期 digest 记忆
@ -122,21 +131,21 @@ reme search query="memory" backend=mcp
配置解析支持:
| 能力 | 源码 | 说明 |
|--------------|-------------------------|---------------------------------------------|
| 默认配置 | `resolve_app_config()` | 未指定 `config` 时加载 `reme/config/default.yaml` |
| 指定配置 | `config=<name-or-path>` | 可传内置配置名或 YAML/JSON 文件路径 |
| dot notation | `parse_dot_notation()` | 例如 `service.port=8181` |
| 环境变量 | `_expand_env_vars()` | 支持 `${VAR}``${VAR:-default}` |
| 值转换 | `_convert_value()` | bool、int、float、JSON list/dict/null 会自动转换 |
| 能力 | 源码 | 说明 |
|--------------|-------------------------|---------------------------------------------------|
| 默认配置 | `resolve_app_config()` | 未指定 `config` 时加载 `reme/config/default.yaml` |
| 指定配置 | `config=<name-or-path>` | 可传内置配置名或 YAML/JSON 文件路径 |
| dot notation | `parse_dot_notation()` | 例如 `service.port=8181` |
| 环境变量 | `_expand_env_vars()` | 支持 `${VAR}``${VAR:-default}` |
| 值转换 | `_convert_value()` | bool、int、float、JSON list/dict/null 会自动转换 |
### 3.2 Service
`BaseService.run_app()` 的顺序:
可通过可选的 `service.jobs` 列表将 HTTP 或 MCP 仅暴露给指定 Job。未配置时所有 `enable_serve: true` 的 Job
仍可被暴露;配置为空列表时不暴露任何 Job。该白名单不会覆盖 `enable_serve: false`
配置该列表后,缺失、禁用、不受支持或无效的已选 Job 会导致服务启动失败。
仍可被暴露;配置为空列表时不暴露任何 Job。该白名单不会覆盖 `enable_serve: false`配置该列表后,缺失、禁用、不受支持或无效的已选
Job 会导致服务启动失败。
```mermaid
flowchart LR
@ -152,19 +161,24 @@ flowchart LR
HTTP service 行为:
| Job 类型 | HTTP 暴露方式 |
|--------------------------------------|-------------------------------------------|
| Job 类型 | HTTP 暴露方式 |
|----------------------------------------|----------------------------------------------|
| 非 `StreamJob``enable_serve: true` | `POST /<job.name>`,返回 `Response` JSON |
| `StreamJob` | `POST /<job.name>`,返回 `text/event-stream` |
| `enable_serve: false` | 不注册 endpoint |
| `StreamJob` | `POST /<job.name>`,返回 `text/event-stream` |
| `enable_serve: false` | 不注册 endpoint |
HTTP service 还可以在所有 Job endpoint 注册完成后挂载 ReMe Studio 单页应用。默认 `service.web_enabled=true`;构建产物按
`service.web_static_dir``REME_WEB_STATIC_DIR`、由 `web``core` extra 安装的可选 `reme_studio` 包,以及源码树
`reme_studio/dist-static` 等候选位置解析。找不到 `index.html` 时只跳过前端Job API 仍然可用。Studio 的 `GET` fallback 不会覆盖
已有的 `POST /<job.name>`
MCP service 行为:
| Job 类型 | MCP 暴露方式 |
|--------------------------------------|---------------------------------|
| 非 `StreamJob``enable_serve: true` | 注册为 MCP tool |
| `StreamJob` | 当前跳过,不注册 |
| `BackgroundJob` | 构造时强制 `enable_serve=False`,不会暴露 |
| Job 类型 | MCP 暴露方式 |
|----------------------------------------|-------------------------------------------|
| 非 `StreamJob``enable_serve: true` | 注册为 MCP tool |
| `StreamJob` | 当前跳过,不注册 |
| `BackgroundJob` | 构造时强制 `enable_serve=False`,不会暴露 |
MCP 服务可通过 `injected_job_kwargs` 注入由服务端管理的参数,调用方不能覆盖这些参数。设置
`tool_error_on_failure: true` 后,不成功的 ReMe `Response` 会作为 MCP tool error 返回。
@ -192,7 +206,7 @@ class VersionStep(BaseStep):
其中 `component_type` 来自类属性,例如:
| 类型 | 类属性 |
| 类型 | 类属性 |
|-----------|-----------------------------------------------------------|
| Step | `BaseStep.component_type = ComponentEnum.STEP` |
| Job | `BaseJob.component_type = ComponentEnum.JOB` |
@ -201,12 +215,38 @@ class VersionStep(BaseStep):
所以同名 backend 在不同 component type 下可以共存。例如 `http` 同时可以是 service backend 和 client backend。
### 4.2 模块导入触发注册
`ComponentEnum` 提供内置类型标识;已安装插件也可以用 `example.reranker` 这样的命名空间字符串声明新类型。自定义标识仅使用
小写字母和数字,并以 `.``_``-` 分隔。它们配置在 `components` 下,与内置组件参与相同的依赖排序和生命周期。
注册发生在模块 import 时。`reme/components/__init__.py` 会 import 各组件包,`reme/steps/__init__.py` 会 import
`channel/common/evolve/file_io/index/transfer`。这些包的 `__init__.py` 再 import 具体模块,从而执行 `@R.register(...)`
### 4.2 内置注册与插件注册
新增 Step 文件后,必须保证它所在包的 `__init__.py` 会 import 该模块,否则注册表里找不到这个 backend。
内置实现通过 package import 填充内置注册表bootstrap 完成后 ReMe 会冻结这个模板,并为每个 `Application` 创建可写副本。
运行期代码通过当前 Application 的注册表解析 backend不能修改进程级模板。随后只加载最终配置中 `plugins` 明确启用的已安装插件。
插件通过 Python `reme.plugins` entry-point group 暴露其 package。package 内的 `plugin.yaml` 只有两个可选 mapping
`backends` 将注册名映射到 `module:Class``application_defaults` 提供低优先级的 `ApplicationConfig` 配置片段。
entry-point 名称就是插件标识;使用
应用配置的 `plugins` 列表或 CLI 的 `plugins=[...]` override 显式启用插件。插件注册因此只影响当前 Application两个插件提供相同
`(component_type, backend)` 时会在装配阶段失败,
不会互相覆盖。
迁移期间仍兼容旧的 Python `Plugin` descriptor 和 `reme.configs` entry point。配置的 `extends` 可以继承内置配置、
旧插件配置或文件配置。独立打包示例见 [Auto Fin](../../plugins/auto-fin/README_ZH.md) 与
[每日论文](../../plugins/daily_paper/README_ZH.md) 插件。
插件包的本地管理与单个应用是否启用插件相互独立:
```bash
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
```
这些管理命令使用当前 Python 解释器对应的 pip不通过 HTTP 或 MCP service 执行。
### 4.3 Component.bind
@ -225,13 +265,13 @@ flowchart LR
`BaseComponent.bind(name, BaseClass, optional=True)` 的规则:
| 场景 | 行为 |
|------------------------|--------------------------------------------|
| `name` 为空 | 返回 `None`,跳过依赖 |
| `app_context` 存在 | 从 `app_context.components[ctype][name]` 查找 |
| 场景 | 行为 |
|-----------------------------|-----------------------------------------------|
| `name` 为空 | 返回 `None`,跳过依赖 |
| `app_context` 存在 | 从 `app_context.components[ctype][name]` 查找 |
| 依赖缺失且 `optional=True` | 解析为 `None` |
| 依赖缺失且 `optional=False` | 启动时报错 |
| standalone 模式 | 可用 `default_factory` 创建自有组件 |
| 依赖缺失且 `optional=False` | 启动时报错 |
| standalone 模式 | 可用 `default_factory` 创建自有组件 |
### 4.4 Step.Ref
@ -315,23 +355,23 @@ flowchart LR
关键源码行为:
| 源码 | 行为 |
|------------------|-------------------------------------------------|
| 源码 | 行为 |
|------------------|--------------------------------------------------------|
| `_start()` | 把 YAML 中每个 step config 解析成 `(step_cls, params)` |
| `_build_steps()` | 每次调用都创建新的 Step 实例,避免跨请求共享状态 |
| `__call__()` | 创建 `RuntimeContext`,按顺序执行 step |
| 异常处理 | 捕获异常,`response.success=False``answer=str(e)` |
| `_build_steps()` | 每次调用都创建新的 Step 实例,避免跨请求共享状态 |
| `__call__()` | 创建 `RuntimeContext`,按顺序执行 step |
| 异常处理 | 捕获异常,`response.success=False``answer=str(e)` |
### 6.2 StreamJob
`StreamJob` 继承 `BaseJob`,但返回流式 chunk
| 行为 | 说明 |
|---------|---------------------------------------------------------|
| context | 带 `stream_queue` |
|-----------|-----------------------------------------------------------|
| context | 带 `stream_queue` |
| Step 输出 | 调用 `context.add_stream_string(text, ChunkEnum.CONTENT)` |
| 异常 | 写入 `ChunkEnum.ERROR` |
| 结束 | 总是发送 `DONE` chunk |
| 结束 | 总是发送 `DONE` chunk |
### 6.3 BackgroundJob
@ -376,7 +416,9 @@ jobs:
```mermaid
flowchart LR
Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"]
Jobs --> Base["base<br/>version / help / health_check<br/>search / node_search / traverse / reindex<br/>read / write / edit / delete / move / list / stat<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_resource / auto_dream / proactive"]
Jobs --> Cron["cron<br/>dream_cron<br/>optimize_index_cron"]
Jobs --> Stream["stream<br/>chat"]
Jobs --> Base["base<br/>version / help / health_check / status / app_config<br/>search / node_search / traverse / graph_snapshot / reindex<br/>read / load / read_image / write / save / edit / delete / move / list / stat / frontmatter_*<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_memory_cc / auto_resource / auto_dream / proactive"]
```
## 7. Step 模型
@ -400,12 +442,12 @@ flowchart LR
`RuntimeContext` 是一次 Job 调用内所有 Step 共享的上下文:
| 字段 | 说明 |
|----------------|---------------------------------------------|
| 字段 | 说明 |
|----------------|--------------------------------------------------|
| `response` | 最终返回的 `Response(answer, success, metadata)` |
| `data` | 自由字典,保存输入参数和中间结果 |
| `stream_queue` | 流式 Job 的输出队列 |
| `stop_event` | 后台 Job 的停止信号 |
| `data` | 自由字典,保存输入参数和中间结果 |
| `stream_queue` | 流式 Job 的输出队列 |
| `stop_event` | 后台 Job 的停止信号 |
Step 里常见写法:
@ -468,21 +510,22 @@ flowchart LR
`reme/config/default.yaml` 当前默认组件:
| ComponentEnum | 名称 | backend | 说明 |
|-------------------|---------------------------------|--------------------------------|------------------------------------------------------|
| `service` | 单例 | `http` | 默认 HTTP 服务 |
| `tokenizer` | `default` | `regex` | BM25 分词器 |
| `as_embedding` | `default` | `${EMBEDDING_BACKEND:-openai}` | embedding 模型封装 |
| `embedding_store` | `default` | `local` | embedding 存储,依赖 `as_embedding: default` |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM 模型封装 |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper |
| `file_graph` | `default` | `local` | wikilink 图谱 |
| `file_catalog` | `default/resource/digest/dream` | `local` | 文件变更 checkpoint |
| `file_chunker` | `markdown` | `markdown` | Markdown AST 分块 |
| `file_chunker` | `default` | `default` | 默认文本分块,当前支持 `jsonl` |
| `keyword_index` | `default` | `bm25` | BM25 关键词索引 |
| `file_store` | `default` | `local` | 组合 file_graph、keyword_index默认 `embedding_store: ""` |
| ComponentEnum | 名称 | backend | 说明 |
|-------------------|---------------------------------|-----------------------------|------------------------------------------------------------|
| `service` | 单例 | `http` | 默认 HTTP 服务 |
| `tokenizer` | `default` | `regex` | BM25 分词器 |
| `as_embedding` | `default` | 默认未配置;示例为 `openai` | 取消配置注释后提供 embedding 模型封装 |
| `embedding_store` | `default` | 默认未配置;示例为 `local` | 取消配置注释后依赖 `as_embedding: default` |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM 模型封装 |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper |
| `agent_wrapper` | `codex/codex_oauth` | `codex` | 分别使用 API key 与 OAuth 的 Codex wrapper |
| `file_graph` | `default` | `local` | wikilink 图谱 |
| `file_catalog` | `default/resource/digest/dream` | `local` | 文件变更 checkpoint |
| `file_chunker` | `markdown` | `markdown` | Markdown AST 分块 |
| `file_chunker` | `json/jsonl/default` | `json/jsonl/default` | JSON、JSONL 与通用文本分块;默认通用分块支持 `txt``log` |
| `keyword_index` | `default` | `bm25` | BM25 关键词索引 |
| `file_store` | `default` | `local` | 组合 file_graph、keyword_index默认 `embedding_store: ""` |
注意:`search` step 的配置含 `vector_weight`,但默认 `file_store.default.embedding_store` 为空,因此实际是否有向量检索取决于运行配置是否启用
embedding store。
@ -540,12 +583,12 @@ class MySearchStep(BaseStep):
可直接用的常见属性:
| 属性 | 默认解析的组件 |
|----------------------|------------------------------|
| `self.as_llm` | `as_llm: default``.model` |
| `self.agent_wrapper` | `agent_wrapper: default`,可选 |
| `self.file_catalog` | `file_catalog: default`,可选 |
| `self.file_store` | `file_store: default` |
| 属性 | 默认解析的组件 |
|----------------------|--------------------------------|
| `self.as_llm` | `as_llm: default``.model` |
| `self.agent_wrapper` | `agent_wrapper: default`,可选 |
| `self.file_catalog` | `file_catalog: default`,可选 |
| `self.file_store` | `file_store: default` |
如果希望 Job 配置指定非 default 组件:
@ -557,13 +600,13 @@ steps:
### 9.4 Step 设计建议
| 建议 | 原因 |
|--------------------------------------------|--------------------------------------------|
| 从 `context` 读取输入,向 `context` 写中间结果 | 多 Step Job 依赖同一个上下文传递数据 |
| 最终结果写到 `context.response` | Service 和 client 只关心标准 `Response` |
| 不在 Step 实例上保存请求级状态 | 每次 Job 调用会重建 Step但保持无状态更容易测试 |
| 需要中断的后台循环检查 `context.stop_event` | `BackgroundJob.close()` 依赖 stop_event 优雅退出 |
| 流式输出只在 StreamJob 中调用 `add_stream_string()` | 普通 Job 没有 stream queue |
| 建议 | 原因 |
|-----------------------------------------------------|--------------------------------------------------|
| 从 `context` 读取输入,向 `context` 写中间结果 | 多 Step Job 依赖同一个上下文传递数据 |
| 最终结果写到 `context.response` | Service 和 client 只关心标准 `Response` |
| 不在 Step 实例上保存请求级状态 | 每次 Job 调用会重建 Step但保持无状态更容易测试 |
| 需要中断的后台循环检查 `context.stop_event` | `BackgroundJob.close()` 依赖 stop_event 优雅退出 |
| 流式输出只在 StreamJob 中调用 `add_stream_string()` | 普通 Job 没有 stream queue |
### 9.5 单测示例
@ -721,12 +764,12 @@ jobs:
后台 Job 的特点:
| 特点 | 说明 |
|--------------|----------------------------------------------------|
| 不对外暴露 | `BackgroundJob.__init__()` 强制 `enable_serve=False` |
| 有 supervisor | 默认异常后指数退避重启 |
| 有 stop_event | close 时通知循环退出 |
| 适合监听/消费 | 文件监听、队列消费、周期性长循环 |
| 特点 | 说明 |
|---------------|------------------------------------------------------|
| 不对外暴露 | `BackgroundJob.__init__()` 强制 `enable_serve=False` |
| 有 supervisor | 默认异常后指数退避重启 |
| 有 stop_event | close 时通知循环退出 |
| 适合监听/消费 | 文件监听、队列消费、周期性长循环 |
### 10.5 新增 Cron Job
@ -752,14 +795,14 @@ jobs:
大多数场景只需要新增 Step + YAML Job。只有这些情况才考虑新增 `reme/components/job/*.py`
| 需求 | 是否需要新 Job 类 |
|----------------|---------------------------|
| 新增一个业务命令 | 否,用 `backend: base` |
| 串联多个已有步骤 | 否,用 `steps:` |
| 要 SSE/流式输出 | 否,用 `backend: stream` |
| 要后台循环 | 否,用 `backend: background` |
| 要 cron 定时 | 否,用 `backend: cron` |
| 要全新的调度/并发/事务语义 | 是,新增 Job backend |
| 需求 | 是否需要新 Job 类 |
|----------------------------|------------------------------|
| 新增一个业务命令 | 否,用 `backend: base` |
| 串联多个已有步骤 | 否,用 `steps:` |
| 要 SSE/流式输出 | 否,用 `backend: stream` |
| 要后台循环 | 否,用 `backend: background` |
| 要 cron 定时 | 否,用 `backend: cron` |
| 要全新的调度/并发/事务语义 | 是,新增 Job backend |
新增 Job backend 的最小形态:

View file

@ -6,8 +6,8 @@ ReMe 的核心思想是:**Memory as File, File as Memory**。
<img src="../figure/memory-as-file.svg" alt="ReMe Memory as File 文件化记忆模型" width="92%">
</p>
**Memory as File**:长期记忆不是藏在黑盒数据库里,而是落在 workspace 目录中的 Markdown 文件、资源文件和索引快照里。用户和 Agent
都可以直接读、写、移动、删除这些文件
**Memory as File**:长期记忆不是藏在黑盒数据库里,原始材料和可读记忆都落在 workspace 内由用户拥有的文件中。用户和 Agent
可以直接读、写、移动、删除它们;`metadata/` 里的索引和快照是可重建的派生状态
**File as Memory**每个文件不只是普通文本也是一个可索引、可链接、可演化的记忆节点。ReMe 会从文件中解析 frontmatter、正文
chunk、wikilink 边,并把它们组织成检索和图谱。
@ -18,14 +18,14 @@ chunk、wikilink 边,并把它们组织成检索和图谱。
ReMe 把记忆设计成文件,不只是为了“方便存储”,而是为了让长期记忆具备几个基本性质:
| 目标 | 含义 |
|----------|----------------------------------------------------------------------|
| 可读 | 用户可以直接打开 workspace像读普通笔记一样读 daily、digest 和原始材料。 |
| 可编辑 | 用户和 Agent 都能用文件操作修正、补充、移动或删除记忆,不必依赖专用数据库客户端。 |
| 可追溯 | digest 中的长期结论可以通过 Sources 章节回到 daily、resource 或 session 原文。 |
| 可迁移 | workspace 是普通目录Markdown、JSONL、YAML 和资源文件可以被备份、同步、版本管理或迁移到其他工具。 |
| 可索引 | 文件虽然是普通文本,但 ReMe 会解析 frontmatter、chunk、wikilink构建检索索引和文件图谱。 |
| 可协作 | 人负责判断和修正Agent 负责整理、链接和检索;二者看到和操作的是同一套文件。 |
| 目标 | 含义 |
|--------|----------------------------------------------------------------------------------------------------|
| 可读 | 用户可以直接打开 workspace像读普通笔记一样读 daily、digest 和原始材料。 |
| 可编辑 | 用户和 Agent 都能用文件操作修正、补充、移动或删除记忆,不必依赖专用数据库客户端。 |
| 可追溯 | digest 中的长期结论可以通过 Sources 章节回到 daily、resource 或 session 原文。 |
| 可迁移 | workspace 是普通目录Markdown、JSONL、YAML 和资源文件可以被备份、同步、版本管理或迁移到其他工具。 |
| 可索引 | 文件虽然是普通文本,但 ReMe 会解析 frontmatter、chunk、wikilink构建检索索引和文件图谱。 |
| 可协作 | 人负责判断和修正Agent 负责整理、链接和检索;二者看到和操作的是同一套文件。 |
因此ReMe 的记忆不是“数据库里的一条隐藏记录”,也不是“只给 LLM 看的 prompt 片段”。它首先是用户拥有的文件,其次才被系统索引成可召回的记忆。
@ -34,7 +34,7 @@ ReMe 把记忆设计成文件,不只是为了“方便存储”,而是为了
ReMe 的 workspace 把记忆分成四层:
```text
raw input -> session/ + resource/
source records -> session/ + resource/
working memory -> daily/
long memory -> digest/
system state -> metadata/
@ -42,13 +42,14 @@ system state -> metadata/
这四层解决的是不同问题。
`session/``resource/` 保存原始输入。它们强调“不要丢现场”对话、Agent session、上传资料、网页或报告先原样留下作为以后核对的证据。
`session/``resource/` 保存来源记录。`resource/` 文件保持原路径和原内容;标准 Auto Memory 保留对话消息,但会有意排除
tool-result 和 base64 data block防止召回结果和二进制负载被误当成用户证据。Agent 运行时生成状态则放在 `mem_session/`
`daily/` 是浅加工层。它把当天发生的对话和资源整理成更适合阅读的 daily note什么事情发生了、有哪些结论、留下了哪些后续任务、对应原文在哪里。
daily 不追求最终抽象,它更像当天工作台。
`digest/` 是深加工层。这里保存的是可以长期复用的记忆节点例如用户偏好、项目背景、流程经验、概念知识、决策先例。digest
不应该只是复制 daily而应该把多次出现的事实、方法和关系合并成更稳定的表述。
`digest/` 是深加工层。这里保存的是可以长期复用的记忆节点例如用户偏好、项目背景、流程经验、概念知识、决策先例。digest 不应该只是复制
daily而应该把多次出现的事实、方法和关系合并成更稳定的表述。
`metadata/` 是系统索引层。它保存 file catalog、chunk 索引、图谱快照等运行状态。用户通常不需要手写这里的内容;真正的人工编辑入口是
`daily/``digest/` 和必要时的 `resource/`
@ -60,27 +61,29 @@ daily 不追求最终抽象,它更像当天工作台。
ReMe 用目录表达记忆组织和记忆分层。原始材料先进入 `resource/``session/`,再沉淀到 `daily/`,最后由 `auto_dream`
整合到 `digest/`
对应的自动流程分别是 [Auto Memory](./auto_memory.md)、[Auto Resource](./auto_resource.md) 和 [Auto Dream](./auto_dream.md)。
检索这些文件时使用 [Memory Search](./memory_search.md)。
对应的自动流程分别是 [Auto Memory](./auto_memory.md)、[Auto Resource](./auto_resource.md)
和 [Auto Dream](./auto_dream.md)。检索这些文件时使用 [Memory Search](./memory_search.md)。
```text
<workspace_dir>/
├── metadata/ # 系统索引层ReMe 索引、图谱、catalog 等持久状态,不作为人工编辑入口
├── session/ # 原始输入层;原始对话和 Agent session
├── session/ # 来源记录层;对话来源记录
│ ├── dialog/
│ │ └── <session_id>.jsonl # auto_memory 保存的对话消息
│ ├── agentscope/
│ │ └── <session_id>.jsonl
│ │ └── <session_id>.jsonl # auto_memory 保存的来源消息
│ └── claude_code/
│ └── <session_id>.jsonl
├── resource/ # 原始输入层;外部原始材料
│ └── <session_id>.jsonl # auto_memory_cc 使用的 ReMe 副本
├── mem_session/ # Agent wrapper 生成的 session/配置,不是用户记忆
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # 来源记录层;外部原始材料
│ ├── <resource>.<ext> # 根目录文件使用今天日期
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
│ └── <resource>.<ext> # 按目录日期进入 daily
├── daily/ # 浅加工层;按日期组织当天事实、对话摘要、资源解读
│ ├── YYYY-MM-DD.md # 当天索引页
│ └── YYYY-MM-DD/
│ ├── <session_id>.md # 对话加工后的 daily note
│ ├── <resource_stem>.md # 资源加工后的 daily note
│ ├── <generated_name>.md # 按主题命名的对话或资源卡片
│ └── interests.yaml # auto_dream 产出的主动兴趣主题
└── digest/ # 深加工层;可长期复用的个人事实、流程经验、知识节点
├── personal/
@ -96,17 +99,18 @@ ReMe 用目录表达记忆组织和记忆分层。原始材料先进入 `resourc
```text
对话
-> session/dialog/<session_id>.jsonl
-> daily/YYYY-MM-DD/<session_id>.md
-> daily/YYYY-MM-DD/<generated_name>.md
-> digest/personal | digest/procedure | digest/wiki
外部资料
-> resource/YYYY-MM-DD/<resource>.<ext>
-> daily/YYYY-MM-DD/<resource_stem>.md
-> resource/[YYYY-MM-DD/]<resource>.<ext>
-> daily/YYYY-MM-DD/<generated_name>.md
-> digest/wiki | digest/procedure
```
前两步偏向记录和整理,最后一步偏向长期沉淀。`auto_memory``auto_resource` 负责从原始输入生成 daily`auto_dream`
负责从 daily 抽取并整合 digest。
负责从 daily 抽取并整合 digest。daily 文件名来自经校验的 frontmatter `name``session_id``source_conversation`
`source_resource` 负责稳定追溯与定位,不用来强制决定文件名。
## Markdown 格式
@ -158,7 +162,7 @@ confidence: observed
## Sources
- [[daily/2026-06-20/session-a.md]]
该偏好记录于 [[daily/2026-06-20/文档说明风格.md]],其中保留了用户多次提出的指导。
```
这样做有三个好处:
@ -180,7 +184,7 @@ Wikilink 用 `[[...]]` 表达文件之间的关系:
[[notes/example.md#L9-L10,L15-L20]]
```
ReMe 的 wikilink 是**字面路径语义**
ReMe 的 wikilink 是 **字面路径语义**
```text
[[X]] -> target_path = "X"
@ -190,9 +194,9 @@ ReMe 的 wikilink 是**字面路径语义**
`[label](../wiki/example.md)` 这类普通 Markdown 链接不会建立 `FileLink`move 或 retarget 操作也不会改写它们。
`#L9``#L9-L10``#L9-L10,L15-L20` 这类锚点会作为普通 `target_anchor` 字符串保存在图谱中。图谱解析器
不会校验行号锚点,因此 `#L0``#L10-L9``#L9,` 也会被保存。`read` 不会解析追加在 `path` 后的锚点;读取指定
范围时需要分别传入从 1 开始、首尾均包含的 `start_line``end_line`,例如
`#L9``#L9-L10``#L9-L10,L15-L20` 这类锚点会作为普通 `target_anchor` 字符串保存在图谱中。图谱解析器不会校验行号锚点,因此
`#L0``#L10-L9``#L9,` 也会被保存。`read` 不会解析追加在 `path` 后的锚点;读取指定范围时需要分别传入从 1 开始、首尾均包含的
`start_line``end_line`,例如
`read(path="digest/wiki/光伏.md", start_line=9, end_line=10)`
Wikilink 的作用:
@ -215,8 +219,7 @@ FileLink
旧文档中的 `related:: [[path]]``- related:: [[path]]`
`[related:: [[path]]]` 仍然可以读取。ReMe 会忽略外围文本,把内部 `[[path]]`
作为普通链接建立索引。从曾存储 typed link 的版本升级后,应执行一次 `reme reindex`
用源文件重建不含旧关系字段的派生图索引。
作为普通链接建立索引。从曾存储 typed link 的版本升级后,应执行一次 `reme reindex`,用源文件重建不含旧关系字段的派生图索引。
### 来源和关系
@ -227,8 +230,8 @@ Sources 章节说明“这条长期记忆从哪里来”:
```markdown
## Sources
- [[daily/2026-06-20/session-a.md]]
- [[resource/2026-06-20/report.pdf]]
该偏好观察自 [[daily/2026-06-20/文档说明风格.md]],支撑它的报告证据保留在
[[resource/2026-06-20/report.pdf]] 中。
```
概念关系链接说明“这个节点和哪些长期记忆有关”,并自然织入正文:
@ -243,13 +246,13 @@ Sources 章节说明“这条长期记忆从哪里来”:
因为记忆就是文件,用户可以直接在编辑器里改 workspaceAgent 也可以通过 ReMe 的文件工具读写同一批文件。两者遵守同一套约定:
| 操作 | 建议 |
|--------|--------------------------------------------------------------------|
| 新增记忆 | 写入合适目录Markdown 使用 frontmatter并尽量写完整 workspace-relative wikilink。 |
| 修改正文 | 保留已有来源和关键 wikilink如果是修正旧结论在正文里说明新材料如何改变旧判断。 |
| 移动文件 | 使用 ReMe 的 move 工具时会默认改写入边中的旧路径;手工移动后建议重新检查入链。 |
| 删除文件 | 删除前检查入链ReMe 的 delete 会返回仍然指向目标的来源文件,方便清理悬空引用。 |
| 修改元数据 | 用 frontmatter 表达短字段;正文发生实质变化时同步更新 `description` |
| 操作 | 建议 |
|------------|-------------------------------------------------------------------------------------|
| 新增记忆 | 写入合适目录Markdown 使用 frontmatter并尽量写完整 workspace-relative wikilink。 |
| 修改正文 | 保留已有来源和关键 wikilink如果是修正旧结论在正文里说明新材料如何改变旧判断。 |
| 移动文件 | 使用 ReMe 的 move 工具时会默认改写入边中的旧路径;手工移动后建议重新检查入链。 |
| 删除文件 | 删除前检查入链ReMe 的 delete 会返回仍然指向目标的来源文件,方便清理悬空引用。 |
| 修改元数据 | 用 frontmatter 表达短字段;正文发生实质变化时同步更新 `description`。 |
一个实用规则是:**可以让 Agent 重写表达,但不要让它丢掉证据边**。尤其是 digest 节点中的 Sources 条目和已有
digest-to-digest Wikilink是长期记忆可追溯和可扩展的基础。
@ -260,7 +263,7 @@ digest-to-digest Wikilink是长期记忆可追溯和可扩展的基础。
```text
digest/wiki/光伏.md
daily/2026-06-20/session-a.md
daily/2026-06-20/文档说明风格.md
resource/2026-06-20/report.pdf
```
@ -360,3 +363,7 @@ FileChunk[]
这样检索命中时Agent 不只看到孤立段落,还能看到它在原文件中的结构位置。
非 Markdown 默认走 `DefaultFileChunker`:按字节大小切分,并保留少量 overlap对 Markdown 则会避免把 `[[wikilink]]` 从中间切开。
`DefaultFileChunker``MarkdownFileChunker` 使用各自配置的 `encoding` 解码文件,并在索引前将平台换行符统一为
LF。默认的 `invalid_encoding_policy: replace` 会在源文件含无效字节时保留其中可解码的内容用于检索,但不会修改源
文件;如需拒绝此类文件,可在 chunker 组件上设置 `invalid_encoding_policy: strict`

View file

@ -1,7 +1,8 @@
# Memory Search
Memory Search 是 ReMe 的记忆检索入口。它先把 `daily/``digest/``resource/` 里的文件持续构建成可搜索的 chunk 索引和
wikilink 图谱;查询时先召回最相关的片段,再沿着片段所在文件的双向链接展开上下文。
Memory Search 是 ReMe 的记忆检索入口。默认后台持续把 `daily/``digest/` 里的 Markdown 构建成可搜索的 chunk 索引和
wikilink 图谱;查询时先召回最相关的片段,再沿着片段所在文件的双向链接展开上下文。`reme reindex` 的重建范围更宽,会额外扫描
`resource/` 和 JSONL这与实时 watcher 的默认范围不同。
<p align="center">
<img src="../figure/auto-index-and-memory-search.svg" alt="ReMe Auto Index and Memory Search 索引、召回、融合与链接展开流程" width="92%">
@ -19,14 +20,14 @@ workspace files
## 它搜索什么
默认配置里的 `index_update_loop` 监听类记忆目录:
默认配置里的 `index_update_loop` 监听类记忆目录:
- `daily_dir`Auto Memory 生成的每日工作记忆和 session 记忆卡片。
- `digest_dir`:长期沉淀后的 digest 节点。
- `resource_dir`:外部资源或导入资料。
默认后缀是 `md``jsonl`。其中 Markdown 用 `markdown` chunker能解析 frontmatter、标题结构和 `[[wikilink]]``jsonl`
`default` chunker按字节大小做重叠切块。
默认实时后缀只有 `md``resource_dir` 由独立的 `resource_watch_loop` 监听,并经 Auto Resource 转换成 daily 卡片后进入实时索引。
如果手动运行 `reme reindex`,其配置会扫描 `daily_dir``digest_dir``resource_dir` 下的 `md``jsonl`Markdown 用
`markdown` chunkerJSONL 用 `jsonl` chunker。
## 索引怎么构建
@ -37,8 +38,8 @@ workspace files
```yaml
index_update_loop:
backend: background
watch_dirs: [ daily_dir, digest_dir, resource_dir ]
watch_suffixes: [ md, jsonl ]
watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [md]
steps:
- backend: init_changes_step
monitor_type: file_store
@ -67,7 +68,8 @@ Markdown chunker 会解析 YAML frontmatter、标题结构和 wikilink产出
### 索引优化
BM25 和 FAISS HNSW 向量索引在删除节点时都采用墓碑tombstone标记而非物理移除积累过多会拖慢搜索。为此内置了闲暇时间索引优化机制——`optimize_index_cron` 定时任务在低峰期压缩墓碑并重建索引:
BM25 和 FAISS HNSW 向量索引在删除节点时都采用墓碑tombstone标记而非物理移除积累过多会拖慢搜索。为此内置了闲暇时间索引优化机制——
`optimize_index_cron` 定时任务在低峰期压缩墓碑并重建索引:
```yaml
optimize_index_cron:
@ -94,14 +96,21 @@ file_store:
它组合三类能力:
| 部件 | 默认状态 | 作用 |
|-------------------------|------|--------------------------------------|
| `file_chunks` | 启用 | 保存 `FileChunk` 文本、行号、分数、可选 embedding |
| `keyword_index.default` | 启用 | BM25 倒排索引chunk id 是 doc id |
| `file_graph.default` | 启用 | 保存 `FileNode` 和 wikilink 边 |
| `embedding_store` | 默认关闭 | 开启后为 chunk 生成 embedding并支持向量召回 |
| 部件 | 默认状态 | 作用 |
|-------------------------|----------|---------------------------------------------------|
| `file_chunks` | 启用 | 保存 `FileChunk` 文本、行号、分数、可选 embedding |
| `keyword_index.default` | 启用 | BM25 倒排索引chunk id 是 doc id |
| `file_graph.default` | 启用 | 保存 `FileNode` 和 wikilink 边 |
| `embedding_store` | 默认关闭 | 开启后为 chunk 生成 embedding并支持向量召回 |
所以开箱搜索主要是 BM25 + 链接展开。把 `embedding_store: default` 打开后,`SearchStep` 会同时跑向量召回和关键词召回。此时若将 `file_store``backend``local` 改为 `faiss`,向量检索会从线性扫描升级为 FAISS HNSW 索引,在大规模 chunk 场景下召回效率更高。
所以开箱搜索主要是 BM25 + 链接展开。把 `embedding_store: default` 打开后,`SearchStep` 会同时跑向量召回和关键词召回。此时若将
`file_store``backend``local` 改为 `faiss`,向量检索会从线性扫描升级为 FAISS HNSW 索引,在大规模 chunk 场景下召回效率更高。
Embedding store 可通过 `health_check_timeout` 配置启动探测。临时失败只会跳过本次向量回填BM25 仍可使用;
后续真实请求成功后会自动恢复缺失向量的回填。
已经完成真实服务验证的嵌入式集成可以调用 `resume_embedding(verified=True)`。切换 Embedding 向量空间时应同时传入
`rebuild=True`ReMe 会先使旧向量失效,再串行后台重建,并在新向量安全持久化前暂停向量搜索。
## 怎么搜索
@ -115,10 +124,12 @@ search:
query: string
limit: integer
min_score: number
start_date: string
end_date: string
steps:
- backend: search_step
vector_weight: 0.7
candidate_multiplier: 3.0
candidate_multiplier: 5.0
expand_links: true
max_links_per_direction: 10
```
@ -129,11 +140,17 @@ search:
reme search query="最近关于索引的讨论" limit=5
```
`start_date``end_date` 可以按 `YYYY-MM-DD` 做包含边界的日期过滤:
```bash
reme search query="索引回归" start_date=2026-06-01 end_date=2026-06-20 limit=10
```
`search_step` 的执行顺序是:
```mermaid
flowchart LR
A["query + limit"] --> B["candidates = limit * candidate_multiplier"]
A["query + limit"] --> B["candidates = min(200, limit * candidate_multiplier)"]
B --> C["file_store.vector_search(...)"]
B --> D["file_store.keyword_search(...)"]
C --> E["RRF 融合"]
@ -200,7 +217,7 @@ Memory Search 的“渐进式”不是一次把全库内容塞进结果,而是
典型文本结构:
```text
========== daily/2026-06-20/session-a.md:12-28 [score=0.0317 keyword=4.8120] ==========
========== daily/2026-06-20/retrieval-regression.md:12-28 [score=0.0317 keyword=4.8120] ==========
...命中的记忆片段...
outlinks (2):
-> digest/indexing.md name="Indexing" description="..."

View file

@ -0,0 +1,215 @@
# 插件管理
ReMe 插件是通过 `reme.plugins` entry-point group 发现的普通 Python distribution。安装插件只表示它在当前 Python
环境中可用,并不会让所有 ReMe Application 自动启用该插件。
需要区分两个操作:
```text
reme plugins install ... 将插件包安装到当前 Python 环境
plugins: [auto-fin] 为一个 Application 启用已安装插件
```
插件包管理仅在本地 CLI 执行,不经过 ReMe HTTP 或 MCP service也不会自动修改应用配置文件。
典型的插件使用流程分为三个阶段:
1. 安装 ReMe 和插件 distribution。
2. 按照 [ReMe 可选模型配置说明](../../README_ZH.md#可选模型配置)配置插件运行所需的环境变量。
3. 启动 Application 时显式启用插件,例如 `reme start plugins='["auto-fin"]'`
## 查看已安装插件
```bash
reme plugins list
```
输出包含插件 entry-point 名称、Python distribution、版本和插件契约
```text
PLUGIN DISTRIBUTION VERSION FORMAT
-------- ------------- ------- --------
auto-fin reme-auto-fin X.Y.Z manifest
```
`manifest` 表示插件使用当前的 package-level `plugin.yaml` 契约;`legacy` 表示插件使用仍然兼容的 Python descriptor
契约。
manifest 将 backend 注册与应用配置分开:
```yaml
backends:
example_step: example_plugin.steps:ExampleStep
application_defaults:
jobs:
example:
backend: base
steps:
- backend: example_step
```
`application_defaults` 是一段不完整的 `ApplicationConfig`。它与 manifest 的 `backends` 命名空间分开,因为 backend
导入声明属于插件发现协议,并不是应用配置。
本地工具需要结构化结果时可以使用 JSON
```bash
reme plugins list --json
```
对照某个应用配置查看启用状态:
```bash
reme plugins list --config default
```
可选的 `ENABLED` 列只反映该配置解析出的 `plugins` 列表。其他运行中进程使用的 CLI override 不是全局启用状态。
## 安装插件包
安装已发布的 distribution
```bash
reme plugins install reme-auto-fin
```
安装指定版本或升级:
```bash
reme plugins install 'reme-auto-fin==X.Y.Z'
reme plugins install reme-auto-fin --upgrade
```
安装本地插件项目:
```bash
reme plugins install ./plugins/auto-fin
```
开发插件时使用 editable 模式:
```bash
reme plugins install ./plugins/auto-fin --editable
```
ReMe 会通过运行 `reme` 命令的同一个 Python 解释器调用 pip。包解析、下载、依赖变更和构建执行仍由 pip 负责。请只安装
可信的包和本地项目。
安装后确认 ReMe 实际发现的插件名:
```bash
reme plugins list
reme plugins validate auto-fin
```
## 查看插件详情
```bash
reme plugins show auto-fin
```
对于 manifest 插件,结果包含注册的 backend 名称和默认 Job 名称。也可以输出 JSON
```bash
reme plugins show auto-fin --json
```
`show` 只检查包契约,不构造 ReMe Application。
## 校验插件
校验已安装插件:
```bash
reme plugins validate auto-fin
```
安装前校验本地插件项目:
```bash
reme plugins validate ./plugins/auto-fin
```
校验范围包括 entry point、`plugin.yaml`、backend 导入和组件类型、registry 冲突、`application_defaults` 合并以及最终的
`ApplicationConfig`。校验过程会导入插件 backend 模块,因此只能对可信代码执行。
## 在服务中启用插件
只安装插件不会将插件代码加载到 Application。需要在配置中显式启用
```yaml
plugins:
- auto-fin
```
也可以只为本次服务启动追加插件:
```bash
reme start plugins='["auto-fin"]'
```
未传入 `config`ReMe 加载 `default.yaml`。插件的 `application_defaults` 合并在该配置之下,因此显式配置和 CLI
override 优先。这个 mapping 是 `ApplicationConfig` 配置片段,并不是另一套配置 schema。插件 backend 只注册到该
Application 的局部 registry。
默认 HTTP service 启动后,可以通过 ReMe CLI client 或 HTTP 访问插件 Job
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
当应用使用 MCP service 时,允许对外服务的插件 Job 会显示为 MCP tool。
自定义应用配置需要提供插件的运行依赖,包括 `agent_wrapper.default`,以及 Auto Fin 使用的 `search``read` Jobs。
## 卸载插件
这里使用插件 entry-point 名称,它不一定等于 distribution 名称:
```bash
reme plugins uninstall auto-fin
```
需要跳过 pip 确认时:
```bash
reme plugins uninstall auto-fin --yes
```
ReMe 会将 `auto-fin` 解析为提供它的 distribution例如 `reme-auto-fin`。如果一个 distribution 提供多个插件 entry
point命令会列出同时被移除的其他插件。
卸载不会重写用户配置。请自行从相关 `plugins` 列表中删除插件,否则下一次启动 Application 时会因为配置的插件未安装而明确
失败。安装、升级或卸载包后,需要重启已经运行的 ReMe 进程。
## 常见问题
### 插件已经安装,但 ReMe 找不到
检查 `reme` 命令与安装插件使用的 pip 是否属于同一个 Python 解释器:
```bash
reme plugins list
python -c 'import sys; print(sys.executable)'
```
使用 `reme plugins install` 可以避免最常见的解释器不一致问题,因为它通过 ReMe 自己的解释器运行 `python -m pip`
### 插件已经安装,但没有加载
将插件 entry-point 名称加入 Application 的 `plugins` 列表。ReMe 刻意不提供全局 enable/disable 状态。
### 启动时报插件未安装
当前配置仍然启用了缺失插件。请重新安装插件,或者从 `plugins` 中删除对应名称。
### 运行中的服务看不到插件变化
插件发现和 backend 注册发生在 Application 构造阶段。修改已安装包后需要重启服务。

View file

@ -31,10 +31,10 @@ proactive:
参数含义:
| 参数 | 作用 |
|-------------------|----------------------------------------|
| 参数 | 作用 |
|-------------------|-----------------------------------------------------------------|
| `date` | 要读取的日期,格式为 `YYYY-MM-DD`。为空时使用应用时区中的今天。 |
| `include_content` | 是否在 answer 和 metadata 中返回 YAML 原文,默认 `true`。 |
| `include_content` | 是否在 answer 和 metadata 中返回 YAML 原文,默认 `true` |
## 输入契约
@ -62,15 +62,15 @@ topics:
成功读取时,`proactive_step` 会在主要 answer 中返回 `summary``topics`;当 `include_content=true` 时还会返回
`content`。相同的结果字段也会保留在标准 response metadata 中:
| 字段 | 说明 |
|-----------|----------------------------------------|
| `date` | 实际读取的日期。 |
| `path` | `daily/<date>/interests.yaml`。 |
| `topics` | 解析后的 topic 列表。 |
| 字段 | 说明 |
|-----------|-------------------------------------------------|
| `date` | 实际读取的日期。 |
| `path` | `daily/<date>/interests.yaml` |
| `topics` | 解析后的 topic 列表。 |
| `content` | YAML 原文;仅在 `include_content=true` 时返回。 |
| `skipped` | 文件不存在时为 `true`。 |
| `error` | 读取或解析异常。 |
| `summary` | 简短摘要。 |
| `skipped` | 文件不存在时为 `true` |
| `error` | 读取或解析异常。 |
| `summary` | 简短摘要。 |
文件存在且解析成功时answer 是结构化数据,例如:
@ -129,20 +129,20 @@ daily notes
职责边界如下。更完整的 Extract、Integrate、Topics、Finish 说明见 [Auto Dream](./auto_dream.md)
| 模块 | 职责 |
|----------------------|----------------------------------------|
| 模块 | 职责 |
|----------------------|----------------------------------------------|
| `dream_extract_step` | 从 changed daily 输入抽取 topic candidates。 |
| `dream_topics_step` | 去重、筛选并写入 `interests.yaml` |
| `proactive_step` | 读取 `interests.yaml`,暴露给上层 Agent。 |
| `dream_topics_step` | 去重、筛选并写入 `interests.yaml`。 |
| `proactive_step` | 读取 `interests.yaml`,暴露给上层 Agent。 |
`proactive` 不修改任何文件,不更新 catalog也不负责判断是否应该主动打扰用户。它只提供当天主题材料是否推送、何时推送、用什么语气推送应由调用方根据产品策略决定。
## 失败模式
| 场景 | 行为 |
|----------------------|--------------------------------------------|
| `interests.yaml` 不存在 | `success=true``skipped=true``topics=[]`。 |
| YAML 无法读取或解析异常 | `success=false`answer 返回错误摘要。 |
| YAML 存在但没有合法 topics | `success=true``topics=[]`。 |
| 场景 | 行为 |
|----------------------------|-----------------------------------------------|
| `interests.yaml` 不存在 | `success=true``skipped=true``topics=[]`。 |
| YAML 无法读取或解析异常 | `success=false`answer 返回错误摘要。 |
| YAML 存在但没有合法 topics | `success=true``topics=[]` |
因此推荐调用方先检查 `success`,再检查 `skipped`,最后检查 `topics` 是否为空。

View file

@ -15,9 +15,15 @@ pip install "reme-ai[core]"
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"
pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
```
静态构建步骤需要 Node.js 22.13 或更高版本,用于在从源码运行 ReMe 时提供 Studio。
`core` extra 建议安装:当前代码会导入 AgentScope wrapper自进化记忆也依赖它。
如果要使用 `auto_memory``auto_resource``auto_dream` 这类 Agent 流程,再配置 LLM
@ -50,10 +56,16 @@ reme start service.port=8181
```bash
reme version
reme health_check
reme list
reme help
```
`reme list` 会列出服务端 action。普通命令会通过 HTTP 调用服务端 Job。
`reme help` 会列出服务端 action。普通命令会通过 HTTP 调用服务端 Job。
基础 `reme-ai` 包不包含前端资源。安装 `reme-ai[web]``reme-ai[core]` 后,浏览器打开
<http://127.0.0.1:2333/> 即可进入 ReMe Studio在同一服务中浏览、编辑和搜索
workspace并查看 digest Wikilink 图。可用 `service.web_enabled=false` 关闭,或通过 `service.web_static_dir` /
`REME_WEB_STATIC_DIR`
指定自定义静态目录找不到构建产物时Job API 仍会正常启动。
---
@ -64,7 +76,8 @@ reme list
```text
.reme/
├── metadata/ # 索引、图谱、catalog 等持久状态
├── session/ # Agent session 与原始对话
├── session/ # 对话来源记录
├── mem_session/ # Agent wrapper 生成的 session/配置
├── resource/ # 外部资料
├── daily/ # daily note
└── digest/ # 长期记忆
@ -89,7 +102,7 @@ reme write \
description="快速开始示例记忆" \
content="# Quick Start Demo
ReMe 会索引 daily、digest 和 resource 目录中的 Markdown。
默认实时 watcher 会索引 daily 和 digest 目录中的 Markdown。
相关链接:[[digest/wiki/search-demo.md]]"
```
@ -128,7 +141,13 @@ reme frontmatter_read path=digest/wiki/quick-start-demo
reme frontmatter_update path=digest/wiki/quick-start-demo metadata='{"tags":["demo"]}'
```
`list` 这个名字在 CLI 中用于 action 列表,所以文件列表 Job 需要用 HTTP 调:
文件列表 Job 可以直接通过 CLI 调用:
```bash
reme list path=digest recursive=true limit=50
```
等价的 HTTP 调用是:
```bash
curl -s http://127.0.0.1:2333/list \
@ -157,7 +176,8 @@ reme auto_memory \
memory_hint="记录用户偏好"
```
外部资料放入 `resource/YYYY-MM-DD/` 后,默认后台会监听 `md/txt/json/jsonl/csv/yaml/html`。也可以手动触发:
外部资料放入 `resource/YYYY-MM-DD/` 或直接放在 `resource/` 下后,默认后台会监听 `md/txt/json/jsonl/csv/yaml/html`
也可以手动触发:
```bash
reme auto_resource changes='[{"path":"resource/2026-06-20/report.md","change":"added"}]'

358
docs/zh/reme-blog.md Normal file
View file

@ -0,0 +1,358 @@
# ReMe让你的个人知识库在每一次对话后继续生长
我们每天都在和 AI 对话。
它帮我们分析项目、阅读论文、排查问题,也听我们讲过偏好、计划和那些还没完全想清楚的念头。
但大多数时候对话结束价值也被关进了历史记录。下一次打开新窗口AI 可能还记得一句结论,却不知道结论从哪里来;可能搜到一段旧对话,却无法把它和后来读过的资料、做过的决定连起来。
真正有用的长期记忆,不应该只是“存过”,而应该能够持续整理、建立联系,并在未来需要时重新参与思考。
这正是 ReMe 想解决的事情。
> **ReMe 是一个面向 AI Agent 的、local-first 的自进化个人知识库。它让对话与资料持续沉淀为可读、可编辑、可检索、相互链接的
Markdown 记忆,并从中提炼值得继续关注的线索。**
项目地址:[https://github.com/agentscope-ai/ReMe](https://github.com/agentscope-ai/ReMe)
项目文档:[https://reme.agentscope.io](https://reme.agentscope.io)
<p align="center">
<img src="../figure/reme-blog/reme-blog-cover-benchmark.png" alt="ReMe 自进化个人知识库与公开基准结果" width="100%">
</p>
## 一个会持续生长的记忆循环
<p align="center">
<img src="../figure/reme-blog/reme-blog-architecture.svg" alt="ReMe 自进化记忆循环" width="100%">
</p>
ReMe 不是另一个聊天机器人,也不试图替代你正在使用的 Agent。它更像一个可以被 QwenPaw、OpenClaw、Hermes、Claude Code 等 Agent
共享的本地记忆层。
它围绕一套普通文件,完成四件事:
- Auto Memory 从对话中提炼值得保留的信息;
- Auto Resource 将外部资料整理为可追溯的记忆;
- Auto Dream 把每日记忆进一步整合为长期知识;
- Index、Search 和 Proactive 让旧记忆重新回到新的任务中。
这形成了一个 `capture → index → consolidate → recall` 的循环:
- 对话和外部资料先被保留下来;
- 有价值的信息被整理为每日记忆;
- 零散事件进一步合并为长期知识节点;
- 搜索、知识链接和兴趣发现,让旧记忆重新参与未来的思考。
更重要的是,这个循环的中心不是一个用户看不见的黑盒数据库,而是用户自己拥有的文件。索引、图谱和缓存都只是可重建的派生状态。
## Memory as File记忆首先是你的文件
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-as-file.svg" alt="ReMe Memory as File" width="100%">
</p>
ReMe 的核心设计称为 **Memory as File, File as Memory。**
“Memory as File”意味着长期记忆不是藏在产品内部而是落在 workspace 里的 Markdown、JSONL、YAML 和原始资源文件中。你可以用 VS
Code、Typora 或 Obsidian 直接打开,也可以用 Git、网盘或自己的同步方案备份和迁移。
“File as Memory”意味着每个文件又不只是普通文本。它可以带有 YAML frontmatter、章节结构、行号范围和
Wikilink成为一个可索引、可连接、可继续演化的记忆节点。
例如,一条关于写作偏好的长期记忆可以是:
```markdown
---
name: 用户偏好:技术文章风格
description: 喜欢先讲问题和结果,再解释技术细节,并用例子帮助理解。
kind: preference
---
用户希望技术文章有清晰主线,避免堆砌术语。
撰写文章时可参考 [[digest/procedure/技术内容写作流程.md]]。
## Sources
该偏好观察自 [[daily/2026-08-07/content-discussion.md]],其中记录了用户对写作方式的要求。
```
几个月后即使你已经忘了这次对话Agent 仍然能读到偏好、找到关联流程,并顺着 `Sources` 回到当时的上下文。
这也是 ReMe 与“黑盒记忆”的关键区别Agent 可以整理记忆,但用户永远保留查看、修正、移动和删除它的权利。
## Auto Memory把聊过的事写成每天的日记
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-memory.svg" alt="ReMe Auto Memory 将对话整理为 daily 记忆" width="100%">
</p>
很多有价值的信息并不是以“请记住”开头的。
比如你在一次对话里说:
> “这周先不要重构登录模块,客户演示之后再做。上次直接升级依赖导致兼容问题,这次先补回归测试。”
这段话里同时包含了项目状态、时间约束、一次失败经验和后续行动。Auto Memory 会把它从聊天流水中提炼出来,写成当天的一张 daily
记忆卡片;可追溯的对话来源记录则保存在 `session/dialog/` 中。
```text
session/dialog/project-a.jsonl 对话来源记录
daily/2026-08-07/login-refactor-decision.md 按内容命名的记忆卡片
daily/2026-08-07.md 当天索引,负责总览
```
`session_id` 仍保留在卡片 frontmatter 中,用于稳定定位和追溯;文件名来自 Agent 生成的主题/事件 `name`,不必与 session ID
相同。
以后再讨论登录模块Agent 不必翻遍聊天记录,就能先看到:当前为什么没有重构、曾经踩过什么坑、下一步应该先做什么。
它像一位一直在场的记录者,但不是机械地抄写逐字稿,而是把“以后还会用到什么”整理出来。
## Auto Resource让外部资料进入同一套记忆系统
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-resource.svg" alt="ReMe Auto Resource 将外部资料整理为可追溯的个人记忆" width="100%">
</p>
并不是所有有价值的信息都来自对话。研究资料、项目文档、会议纪要、网页存档和结构化数据,同样可能成为个人知识库的一部分。
Auto Resource 提供了一条更通用的外部资料入口。资料进入 `resource/`ReMe 保留原文,再把主题、关键事实和可行动信息整理为带有
`source_resource` 链接的 daily 卡片。当前可以处理 Markdown、纯文本、JSON、JSONL、CSV、YAML 和 HTML 等文本类资料。
这意味着Auto Memory 负责从对话建立个人知识Auto Resource 负责从非对话资料建立个人知识。两条输入最终进入同一个 daily
记忆层,再由 ReMe 统一索引、整合和检索。
### Daily Paper外部资料工作流的一个例子
Daily Paper 是建立在这套文件化记忆之上的可选插件。它会从 Hugging Face Papers 的周榜和月榜收集论文,去除近期已经推荐过的内容,排序后精选三篇,保存
PDF并生成中文论文笔记与一份约五分钟可读完的简报。
想象一下,你持续关注 Agent Memory每天早上收到的不只是三个论文链接而是三篇已经保存到本地的详细笔记。简报通过 Wikilink
指向原始笔记,原始笔记又能回到 PDF。一个月后再问“最近有哪些方法在做长期记忆压缩”这些材料已经进入同一套检索系统不需要重新从浏览器历史里寻找。
Daily Paper 展示了 Auto Resource 可以怎样被组合成具体工作流,但外部资料入口并不局限于论文。
## Auto Dream让日记长成相互连接的长期知识
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-dream.svg" alt="ReMe Auto Dream 从 daily 抽取、分类并整合长期知识,同时写入 Wikilink" width="100%">
</p>
日记多了,新的问题也会出现:信息虽然都在,却仍然散落在不同日期里。
假设你先后从对话和外部资料中得到三条关于同一个问题的信息:
- 第一次排查构建卡死,清缓存无效;
- 第二次在项目文档中确认根因是 Node 内存不足;
- 第三次又补充了大型 TypeScript 项目下更容易触发这个问题。
Auto Dream 默认查看以目标日期结尾的最近两天,只把相对上次运行发生变化的 daily 文件一起交给抽取器。它合并指向同一抽象的跨文件
证据,并在默认最多五个 unit 的额度内只保留最值得复用的记忆,再按内容写入三类长期记忆:
- `Personal`:用户、团队或项目特定的偏好、约定和约束;
- `Procedure`:可以再次执行的流程、方法和排查手册;
- `Wiki`:通用的定义、原则、观察和知识。
例如,上面的信息会被整理为 `digest/procedure/前端构建卡死排查.md`,包含触发条件、排查顺序、无效尝试、解决方法和适用边界,而不是几篇日记的简单拼接。
整合每个记忆单元时Auto Dream 会先跨 `personal``procedure``wiki` 搜索已有节点,区分“同一抽象”和“相关知识”。同一抽象决定如何演化目标节点:
- `CREATE`:没有相同记忆,创建新节点;
- `CORROBORATE`:同一结论再次出现,补充来源并增强可信度;
- `REFINE`:新材料补全条件、步骤或细节;
- `CORRECT`:新信息修正旧结论。
相关知识则在同一次整合中以 Wikilink 写进正文,这就是 Auto Link。比如“前端构建卡死排查”可以同时连接通用知识、团队偏好和原始证据
```markdown
这个问题常见于 [[digest/wiki/大型 TypeScript 项目.md]],处理时遵循
[[digest/personal/团队变更偏好.md]] 中“先补回归测试”的约定。
## Sources
根因与适用场景记录在 [[daily/2026-08-07/build-debug.md|构建排查记录]] 中。
```
知识的演化与链接发生在同一条流程里。关系不是藏在图数据库里的不可见边,而是正文中可读、可改的内容;文件可以重建图,图不会反过来绑架文件。
## Memory Index普通文件如何变成可搜索的记忆网络
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-index.svg" alt="ReMe Memory Index 构建过程" width="100%">
</p>
Markdown 适合人读但如果只是把文件堆进目录Agent 仍然很难快速找到它们。默认实时索引持续监听 `daily/``digest/` 中的
Markdown`resource/` 由独立资源流程监听,转成 daily 卡片后进入同一索引。需要从现有文件完整重建时,`reme reindex` 还会扫描
`resource/` 与 JSONL。
一份 Markdown 会被解析为:
- 一个文件节点包含路径、frontmatter 等文件级信息;
- 多个语义 chunk尽量沿标题、段落、列表和代码块边界切分并保留章节骨架与行号
- 多条 Wikilink 边:记录它指向谁,以及谁又指向它。
在检索侧ReMe 可以组合三类信号:
| 检索信号 | 解决的问题 | 例子 |
|----------------|------------------------------------|------------------------------------|
| BM25 关键词 | 精确名称、术语和编号不能丢 | “宁德时代”“issue #184” |
| Embedding 向量 | 用户换了一种说法,也要理解语义接近 | “构建卡死”与“打包阶段没有响应” |
| Wikilink 图谱 | 命中一个节点后,看到它的上下游关系 | 从“钴”找到“三元正极”和相关调研记录 |
默认配置开箱启用 BM25 与 Wikilink 展开Embedding 是可选能力,开启后才会参与向量召回。索引、图谱和缓存都写在 `metadata/`
即使删除也可以根据用户的源文件重新构建。
## Memory Search先找到答案再沿着关系渐进展开
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-search.svg" alt="ReMe 混合搜索与渐进式展开" width="100%">
</p>
很多 RAG 系统会一次性把 Top-K 文本全部塞进上下文。这样做简单,却容易带来两个问题:孤立切片缺少上下文,而把邻居正文全部展开又会迅速消耗
token。
ReMe 的混合搜索先让 BM25 与可选的向量检索各自召回候选,再使用 RRF 按排名融合。RRF 不强行比较 BM25
分数与余弦相似度这两种不同量纲,而是综合一个结果在两张榜单中的位置。
召回之后,信息按三层渐进式展开:
1. **先看命中片段**:返回最相关的 chunk、文件路径和行号
2. **再看关系目录**:展示该文件的出链与入链,只给邻居的路径、名称、描述和锚点,不急着加载全文;
3. **最后按需深入**Agent 判断哪条关系真正相关,再读取原文或沿图谱继续遍历。
例如,你问:“上次 Alice 推荐的那本讲注意力的书叫什么?”
第一步可能命中一张聚餐日记,其中只写着“标题里有‘深度’两个字”;结果同时显示,这张日记链接到了 Alice 的个人节点,也被《深度工作》的阅读笔记反向引用。
Agent 不需要把 Alice 的全部档案、所有读书笔记和整个月的日记都塞进上下文。它只要沿着最相关的链接再读一次,就能回答:
> 是《深度工作》。Alice 在那次聚餐时推荐了它,你后来还读了第三章并留下了笔记。
这更像人的联想过程:先想起一个片段,再顺藤摸瓜找到完整上下文。
## Proactive从你的记忆里发现那些尚未说出口的需要
<p align="center">
<img src="../figure/reme-blog/reme-blog-proactive.svg" alt="ReMe Proactive 内外双向记忆循环" width="100%">
</p>
到这里ReMe 已经有了两条不断丰富知识库的输入流:
- Auto Memory 从持续发生的对话中沉淀个人上下文;
- Auto Resource 从外部资料中补充新的知识。
Proactive 则把方向反过来:它从已经积累的对话和资料中,发现你仍未解决或可能希望继续推进的主题,以及你尚未关注、但与近期工作紧密相关的信息。这些发现会反哺输入流,为补充外部知识指明新的路径。
例如,你最近一周分别聊过:
- 搜索结果里缺少来源;
- 长文档切片后容易失去章节上下文;
- 想比较几种 Agent Memory 评测方法。
尽管你从未明确说“帮我系统研究记忆检索的可解释性”,但 Auto Dream 可以从这些 daily 记忆中提炼出一个兴趣主题:
```yaml
title: 记忆检索的可解释性评估
reason: 用户近期持续关注来源追溯、结构化切片和记忆评测。
evidence: daily/2026-08-07/search-discussion.md
keywords:
- memory search
- source attribution
- benchmark
```
在未来的 beta 版本中,上层 Agent 通过 Proactive
读取这个主题后,可以选择在合适的时机追问:“要不要把最近讨论过的检索问题整理成一份评测方案?”也可以据此启动一个经过用户授权的资料收集流程。用户不需要预先梳理并显式指定自己的兴趣关键词和范围;与对话中潜在需求相关的外部资料,也可以持续进入知识库。
这里有一个重要边界:**ReMe 的 Proactive 本身只读取并暴露兴趣主题,不会擅自联网、推送或改写知识库。**
它不是凭空猜测你的兴趣,而是让那些已经出现在行为和对话中、却还没有被明确表达的线索浮出水面。
## Performance它真的能从很长的历史里找回信息吗
ReMe 使用 LongMemEval 和 BEAM 验证多会话与超长对话中的记忆能力。评测时Agent 可以用 ReAct 方式进行多轮搜索和读取,生成答案后再由
LLM-as-judge 评分。
| 基准 | 设置 | 样本量 | Agentic 得分 | 主要检验内容 |
|---------------------------|-------------|------------------:|-------------:|--------------------------------|
| **LongMemEval cleaned-s** | **整体** | **500 题** | **89.4%** | 跨会话检索、知识更新与时间推理 |
| BEAM | 100K 上下文 | 20 cases / 400 题 | 66.1% | 十类长上下文记忆任务 |
| BEAM | 1M 上下文 | 35 cases / 700 题 | 65.0% | 更大规模的超长对话设置 |
LongMemEval cleaned-s 包含单会话事实、偏好、多会话推理、知识更新和时间推理等题型。ReMe 在 500 道问题上取得 89.4% 的整体
Agentic 得分。完整流程和分项结果见 [LongMemEval 评测说明](../../benchmark/longmemeval/README_ZH.md)。
BEAM 覆盖矛盾消解、事件排序、信息抽取、知识更新、多会话推理、偏好遵循、摘要和时间推理等十类任务。ReMe 在 100K 设置下的 20
cases / 400 题上取得 66.1%,在 1M 设置下的 35 cases / 700 题上取得
65.0%。完整设置见 [BEAM 评测说明](../../benchmark/beam/README_ZH.md)。
此外ReMe 还使用 $\pi$-Bench 验证了基于多会话推理提升 Agent 主动性的潜力。$\pi$-Bench 中的 PROC 分数旨在评估 Agent
在隐藏意图直接完成、针对性澄清引导、跨会话偏好恢复、跨会话规范复用、跨任务依赖推断以及欠规格请求推进等方面的主动性能力。ReMe
Agent 在 5 种用户角色User Persona上平均取得 0.580 的 PROC 分数,超出相同测试模型配置的 NanoBot
2.4%。关于该基准的详细介绍见 [$\pi$-Bench 论文](https://arxiv.org/abs/2605.14678)。
## ReMe 能帮谁?
### 直接使用 Agent 的人
如果你希望 AI 在长期协作中持续了解你ReMe 可以让个人助理不再每次都从零开始。你的偏好、项目背景、重要资料和过去做过的决定,会在持续对话中沉淀下来,并在真正相关的时候被重新找到。
研究者、工程师、分析师和其他知识工作者都属于这一类直接用户。研究者可以让论文、讨论和阅读笔记彼此连接;工程师可以保留项目决定与跨会话排障经验;分析师可以持续积累事件、观点与来源。职业不同,共同需求都是让
AI 能够理解过去、积累经验,并在下一次任务中找回依据。
### 构建 Agent 的开发者
如果你正在构建 Agent、Harness 或 AI 产品ReMe 提供了一层可以独立接入的长期记忆基础设施。你可以通过 CLI、HTTP API、MCP Server
或 Python API让不同 Agent 共享同一个文件化 workspace而不必为每个应用重新实现记忆抽取、知识整理、混合检索和关系展开。
文件是事实来源,索引与缓存可以随时重建,也更容易定位一次错误召回究竟来自原始资料、记忆整理还是检索链路。
归根结底ReMe 适合那些希望 AI 不只“回答这一次”,还能够理解过去、积累经验,并在长期协作中越来越懂自己的用户和开发者。我们希望
Agent 越用越懂你,但“懂”不应该建立在一个无法查看、无法修正、无法带走的黑盒上。
ReMe 给出的答案很朴素:
- 记忆是用户拥有的文件;
- 原始信息保留现场,长期知识保留抽象;
- 新对话和新资料持续进入,旧知识也持续被补充和修正;
- 每个结论都可以通过 Wikilink 找到关系和来源;
- 索引与缓存服务于文件,而不是取代文件;
- Agent 可以记住、整理、搜索和发现,但最终控制权始终属于用户。
当这些机制连接起来,个人知识库就不再是一座需要你手工维护的仓库。
它会在每一次对话后多记住一点,在每一份资料到来后多理解一点,在夜晚把零散经验重新整理,在未来的某个问题出现时,再沿着知识之间的联系,把真正需要的那段记忆带回来。
这就是 ReMe 想做的事:**让记忆不只被保存,也能持续进化。**
## 接入你正在使用的 Agent
ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP Server 接入,也可以通过 Python API 嵌入宿主进程。默认 HTTP
服务还可在同一地址提供 ReMe Studio用于浏览、编辑、搜索 workspace 和查看 digest Wikilink 图。不同 Agent 可以选择适合自身
运行环境的路径,并按需共享同一个本地 memory workspace。
| Agent | 推荐接入方式 | 接入后能力 |
|----------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| **DeepSeek Harness** | 将 [`@agentscope-ai/reme`](../../typescript/README_ZH.md#deepseek-harness) 安装为 DSH profile bundle。 | 长期记忆指引、`reme_search`、自动捕获主 Agent 已完成的对话,以及定时 Auto Dream。 |
| **OpenClaw** | 将 [`@agentscope-ai/reme`](../../typescript/README_ZH.md#openclaw) 安装为原生 memory plugin。 | 根 Agent 对话运行前召回、显式搜索、自动捕获对话,以及定时 Auto Dream。 |
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主应用的生命周期和模型配置,同时保持记忆本地、文件化。 |
| **Claude Code** | 启动 streamable HTTP MCP Service并安装 [`integrations/claude_code/reme`](../../integrations/claude_code/reme)。 | MCP 记忆召回工具、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP Service并安装 [`integrations/hermes_agent`](../../integrations/hermes_agent)。 | 在模型调用前自动召回相关记忆,并在每轮对话完成后异步调用 `auto_memory`。 |
| **Codex 等支持 CLI 的 Agent** | 复制或安装 [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动记录需要宿主 Agent 显式接入会话生命周期。 |
安装、配置与集成演示可查看 [README 中文版](../../README_ZH.md)。
## 欢迎贡献
ReMe 已经开源,我们也欢迎社区一起把这套自进化记忆系统做得更完整:
- 接入更多 Agent 与 Harness让不同运行环境都能使用同一套用户拥有的长期记忆
- 贡献新的 Auto Resource 数据源和工作流,让论文、新闻及其他公开资料能够持续进入知识库;
- 改进 Auto Memory、Auto Dream、Auto Link、混合搜索与 Proactive让记忆整理得更准确、关系更清晰、召回更可靠
- 补充新的应用案例、评测任务和诊断报告,帮助我们看见真实长期使用中的成功与失败;
- 完善文档、测试,或通过 Issue 分享你对个人 AI 记忆的需求与想法。
无论是一段代码、一份使用案例、一次问题反馈,还是一个新的记忆工作流,都可能帮助 ReMe 更接近真正可读、可控、可持续进化的个人知识库。
贡献指南:[https://docs.agentscope.io/reme/latest/en/contribution](https://docs.agentscope.io/reme/latest/en/contribution)

View file

@ -50,18 +50,20 @@ session/
daily/
├── 2026-05-18.md
└── 2026-05-18/
├── 2026-05-18-close.md
├── glencore-q3.md
├── cobalt-policy.md
├── cathode-trend.md
├── cobalt-supply-risk.md
├── glencore-output-update.md
├── drc-cobalt-policy.md
├── high-nickel-cathode-trend.md
└── interests.yaml # auto_dream 后生成
```
对应链路:
- `auto_memory` 保存原始对话到 `session/dialog/<session_id>.jsonl`,再让 Agent 把重要事实写入 `daily/<date>/<session_id>.md`
- `resource_watch_loop` 监听 `resource/` 文本文件变化,并触发 `auto_resource_step` 写同名 daily note。
- `daily_create` 会维护 `daily/<date>.md` 当天索引页。
- `auto_memory` 保存对话来源消息到 `session/dialog/<session_id>.jsonl`,再让 Agent 把重要事实写入按主题命名的
`daily/<date>/<generated_name>.md`;卡片 frontmatter 保留 `session_id``source_conversation` 用于稳定定位和追溯。
- `resource_watch_loop` 监听 `resource/` 文本文件变化,并触发 `auto_resource_step` 写带 `source_resource` 的 daily note文件名由
Agent 根据内容建议,再由系统清洗并处理冲突,不保证与资源同名。
- Auto Memory、Auto Resource 和 Auto Dream 都会在写入后刷新 `daily/<date>.md` 当天索引页。
### Day 1 晚上Auto Dream 进入 Digest
@ -75,8 +77,8 @@ reme auto_dream date=2026-05-18
```text
dream_extract_step
扫描 daily/2026-05-18.md 和 daily/2026-05-18/ 下 changed 文件
输出 units 和 topics
默认扫描 2026-05-17 至 2026-05-18 的 daily 窗口
从 changed 文件输出最多 5 个 units 和 topics
dream_integrate_step
每个 unit 用 node_search 召回已有 digest 节点
@ -114,12 +116,12 @@ description: 锂电正极材料关键原料,主产区集中于刚果(金)
## 供给端
嘉能可三季度钴产量同比下滑 18%,需要继续跟踪供给收缩对价格的影响。
## Sources
- [[daily/2026-05-18/2026-05-18-close.md]]
## 政策风险
刚果(金)矿权政策变化可能影响 KFM 矿运营,需联动跟踪洛阳钼业。
## Sources
产量下滑与政策风险的证据记录在 [[daily/2026-05-18/cobalt-supply-risk.md]] 中。
```
注意wikilink 是字面路径语义,推荐写完整 workspace-relative 路径和 `.md` 扩展名。ReMe 不会自动把 `[[钴]]` 解析成某个文件。
@ -227,7 +229,7 @@ topics:
reason: 用户当天多次提到 KFM 矿和钴价风险
keywords: [钴, 刚果金, 洛阳钼业, KFM]
paths:
- daily/2026-05-18/2026-05-18-close.md
- daily/2026-05-18/cobalt-supply-risk.md
```
调用:
@ -311,7 +313,7 @@ description: build 卡住且内存上涨时,优先检查类型检查进程内
## Sources
- [[daily/2026-03-10/build-oom-2026-03-10.md]]
失败尝试和有效的内存调整记录在 [[daily/2026-03-10/build-oom-2026-03-10.md]] 中。
```
示例 `digest/personal/code-style.md`
@ -363,7 +365,7 @@ Agent 回复可以直接跳过低价值路径:
- `digest/procedure/` 保存“怎么做”和“哪些路径无效”,让 Agent 复用排查经验。
- `digest/personal/` 保存用户偏好,让 Agent 跨会话遵守同一工程风格。
- 原始对话仍在 `session/dialog/`daily 记录可追溯digest 只是长期提炼结果。
- 对话来源记录仍在 `session/dialog/`daily 记录可追溯digest 只是长期提炼结果。
## 场景三:个人第二大脑
@ -410,7 +412,7 @@ description: 用户朋友,常推荐阅读材料
## Sources
- [[daily/2026-04-20/lunch-with-alice.md]]
这次推荐记录在 [[daily/2026-04-20/lunch-with-alice.md]] 中。
```
### 一次联想式回忆

View file

@ -3,6 +3,7 @@ EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# Optional SSH jump host for Hugging Face daily-paper downloads.
# REME_PROXY_IP=proxy.example.com
# REME_PROXY_ACCOUNT=your-ssh-account
# Optional data-source mirrors. Unset variables use the official services.
# HF_MIRROR_URL=https://hf-mirror.com
# ARXIV_MIRROR_URL=https://export.arxiv.org
# TUSHARE_MIRROR_URL=http://112.124.63.173:4000/tushare

3
github-pages/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
node_modules/
dist/
.generated/

89
github-pages/README.md Normal file
View file

@ -0,0 +1,89 @@
# ReMe GitHub Pages
This directory contains the standalone Vite documentation site published at <https://reme.agentscope.io>. The
GitHub Pages fallback is <https://agentscope-ai.github.io/ReMe/>. It does not depend on the ReMe Studio application in
`reme_studio/`.
## Requirements
- Node.js 22.13 or newer
- npm
## Local development
From the repository root:
```bash
cd github-pages
npm install
npm run dev
```
Open the URL printed by Vite, normally <http://localhost:5173/>. The development server watches the frontend source.
When a repository Markdown file changes, restart the development command to regenerate the documentation content.
For subsequent installs or CI-compatible dependency installation, use:
```bash
npm ci
```
## Preview the production build
Build and start the preview server:
```bash
npm run build
npm run preview
```
Open the URL printed by Vite, normally <http://localhost:4173/>. Production assets use relative paths so the same build
works on both the custom domain and the GitHub Pages project path.
The generated `dist/` and `.generated/` directories are disposable build output and are excluded from Git.
## Documentation sources
The build script reads the canonical repository files directly. Do not edit generated copies under `.generated/` or
`dist/`.
- `README.md` and `README_ZH.md`: project introductions
- `docs/en/` and `docs/zh/`: English and Chinese guides
- `docs/figure/`: documentation images
- `reme_studio/README.md` and `reme_studio/README_ZH.md`: ReMe Studio guide
- `typescript/README.md` and `typescript/README_ZH.md`: TypeScript client, DeepSeek Harness, and OpenClaw integration guide
- `plugins/*/README*.md`: plugin guides
- `benchmark/{beam,longmemeval,pibench,toolmemory}/README*.md`: benchmark guides and results
- `AGENTS.md`: repository development guide
To add or reorganize a document in the site navigation, update
[`scripts/generate-content.mjs`](./scripts/generate-content.mjs). Presentation and interaction code lives in `src/`.
## Project structure
```text
github-pages/
├── index.html
├── package.json
├── scripts/
│ └── generate-content.mjs
├── src/
│ ├── main.js
│ └── styles.css
└── vite.config.js
```
## Deployment
The repository workflow `.github/workflows/deploy-docs.yml` builds this directory and publishes `dist/` to GitHub Pages.
It runs after relevant documentation or site files change on `main`, and it can also be started manually from the
GitHub Actions page.
The repository's **Settings → Pages → Build and deployment → Source** must be set to **GitHub Actions**. Its custom
domain must be set to `reme.agentscope.io`; `public/CNAME` preserves that domain in the published artifact.
Useful links:
- ReMe documentation: <https://reme.agentscope.io>
- GitHub Pages fallback: <https://agentscope-ai.github.io/ReMe/>
- ReMe repository: <https://github.com/agentscope-ai/ReMe>

Some files were not shown because too many files have changed in this diff Show more