Compare commits

...

124 commits

Author SHA1 Message Date
Ziyang Guo
084c02e43a
fix(docs): publish OpenClaw and Hermes integration figures (#586)
Some checks failed
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* fix(docs): publish OpenClaw and Hermes integration figures

The generated /en/integrations/openclaw, /zh/integrations/openclaw and
/en/integrations/hermes pages are built from the plugin READMEs, which
reference local figures/. Only the dsh and Studio READMEs had their
./figures/ links rewritten to site paths and their figure directories
copied into the build, so every screenshot on those three pages resolved
to /<language>/integrations/figures/*.png and 404'd. verify-build.mjs
only checks <a> links, never <img> sources, so the build stayed green.

Mirror the dsh pattern: rewrite the figure links to /figures/openclaw/
and /figures/hermes/, copy both figure directories into public/figures/,
add the missing language-cross link for the Hermes page, and cover all of
it in the generated-content tests.

* ci(docs): watch the new integration figure directories

The figures this PR adds under integrations/hermes_agent/figures/ and
integrations/openclaw/figures/ were not in either docs workflow's path
filter, so a follow-up commit that only touches a PNG would skip the docs
build and the broken-reference tests. Add both globs to the pull_request
filter in ci-docs.yml and the push filter in deploy-docs.yml, matching how
integrations/dsh/figures/** and reme_studio/figures/** are already listed.

Requested in review on #586.

* test(docs): validate all integration screenshot references

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-10-06 14:32:09 +08:00
pei711
dc12798526
fix(agent): preserve Unicode separators in saved state (#594)
Some checks failed
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / OpenClaw plugin / Validate OpenClaw plugin (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI and Release / Docker / Build and test / amd64 (push) Has been cancelled
CI and Release / Docker / Build and test / arm64 (push) Has been cancelled
CI and Release / Docker / Publish multi-platform tags (push) Has been cancelled
Co-authored-by: pei711 <199601925+pei711@users.noreply.github.com>
2026-10-05 11:25:37 +08:00
dependabot[bot]
090c8c24ab
chore(deps): bump openclaw/clawhub/.github/workflows/package-publish.yml (#596)
Bumps [openclaw/clawhub/.github/workflows/package-publish.yml](https://github.com/openclaw/clawhub) from 0.23.3 to 0.24.0.
- [Release notes](https://github.com/openclaw/clawhub/releases)
- [Changelog](https://github.com/openclaw/clawhub/blob/main/CHANGELOG.md)
- [Commits](87ca030c30...cacf5ec1b0)

---
updated-dependencies:
- dependency-name: openclaw/clawhub/.github/workflows/package-publish.yml
  dependency-version: 0.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-05 11:11:40 +08:00
jinliyl
eb2f466a30
docs: announce Studio Playground in latest updates (#595)
Some checks failed
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
CI and Release / Docker / Build and test / amd64 (push) Waiting to run
CI and Release / Docker / Build and test / arm64 (push) Waiting to run
CI and Release / Docker / Publish multi-platform tags (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
Deploy / Documentation / Build documentation (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* docs: announce Studio Playground in latest updates

* docs: center Playground preview and collapse older updates
2026-10-04 17:53:06 +08:00
jinliyl
4c54c2b650
feat(docker): add container deployment and multi-platform release workflow (#582)
Some checks failed
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
CI and Release / Docker / Build and test / amd64 (push) Waiting to run
CI and Release / Docker / Build and test / arm64 (push) Waiting to run
CI and Release / Docker / Publish multi-platform tags (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / ReMe Studio / Studio checks (push) Has been cancelled
* feat(docker): add container deployment and multi-platform release workflow

* fix(llm): initialize providers on model access and preserve runtime injection

* test(llm): keep rejected model updates independent of credentials
2026-10-03 23:54:58 +08:00
machaoxin0407
49bbbc93ff
perf(bm25): maintain live document length incrementally (#590)
Co-authored-by: machaoxin0407 <221922045+machaoxin0407@users.noreply.github.com>
2026-10-03 22:38:11 +08:00
jinliyl
d529ec5256
feat(studio): add bilingual browser demo to documentation site (#581)
* feat(studio): add bilingual browser demo to documentation site

* fix(studio): preserve literal wikilinks during Markdown parsing

* fix(studio): preserve Markdown references and escaped wikilinks
2026-10-03 22:34:11 +08:00
jinliyl
1648b7ce93
Revert "docs: show main update time and CI status in READMEs (#583)" (#589)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
This reverts commit 65b48ec5fc.
2026-10-01 10:07:00 +08:00
jinliyl
65b48ec5fc
docs: show main update time and CI status in READMEs (#583)
Some checks failed
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / DSH plugin / Validate DSH plugin (push) Has been cancelled
CI / OpenClaw plugin / Validate OpenClaw plugin (push) Has been cancelled
2026-10-01 09:01:54 +08:00
jinliyl
81c16d3e2c
feat(jobs): add a unified enabled switch for all job types (#580)
* feat(jobs): allow disabling cron schedules independently

* refactor(jobs): unify job activation with enabled flag
2026-09-30 18:30:36 +08:00
jinliyl
67135cfc57
fix(dsh): support DeepSeek Harness 0.1.7-rc.2 (#571)
* fix(dsh): support DeepSeek Harness 0.1.7-rc.2

* fix(dsh): preserve settings and validate live configuration

* fix(dsh): recognize legacy ReMe guidance on resumed sessions

* fix(dsh): keep live settings schema browser compatible

* fix(dsh): preserve exact host validation with serializable forms
2026-09-30 14:53:42 +08:00
xyf2020
67936d5a43
refactor(plugins): move BaseAgenticAnswerStep out of the core benchmark steps package into the lme/beam plugins (#576)
* refactor(plugins): move BaseAgenticAnswerStep into lme/beam plugins

Move the shared benchmark answer base class from the core
reme.steps.benchmark package into each plugin's own src tree
(reme_lme.base_agentic_answer / reme_beam.base_agentic_answer) with
absolute imports, drop the core benchmark steps package, and update
plugin READMEs accordingly.

* refactor(plugins): simplify benchmark answer steps

* docs(plugins): clarify benchmark answer step migration

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-30 12:19:32 +08:00
dependabot[bot]
30227e509f
chore(deps-dev): bump agentscope from 2.0.8 to 2.0.9 (#578)
Some checks failed
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / ReMe Studio / Studio checks (push) Has been cancelled
Bumps [agentscope](https://github.com/agentscope-ai/agentscope) from 2.0.8 to 2.0.9.
- [Release notes](https://github.com/agentscope-ai/agentscope/releases)
- [Changelog](https://github.com/agentscope-ai/agentscope/blob/main/docs/changelog.md)
- [Commits](https://github.com/agentscope-ai/agentscope/compare/v2.0.8...v2.0.9)

---
updated-dependencies:
- dependency-name: agentscope
  dependency-version: 2.0.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 14:31:11 +08:00
dependabot[bot]
bebad36745
chore(deps-dev): bump agentscope from 2.0.7.post1 to 2.0.8 (#531)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Bumps [agentscope](https://github.com/agentscope-ai/agentscope) from 2.0.7.post1 to 2.0.8.
- [Release notes](https://github.com/agentscope-ai/agentscope/releases)
- [Changelog](https://github.com/agentscope-ai/agentscope/blob/main/docs/changelog.md)
- [Commits](https://github.com/agentscope-ai/agentscope/compare/v2.0.7.post1...v2.0.8)

---
updated-dependencies:
- dependency-name: agentscope
  dependency-version: 2.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-26 00:19:03 +08:00
jinliyl
8b5456641f
docs: show ReMe ecosystem on homepage (#568)
Some checks failed
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
Deploy / Documentation / Build documentation (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* docs: show ReMe ecosystem on homepage

* fix(docs): prevent hero title from overlapping ecosystem column

At 1366x768 the English headline used white-space: pre inside a
left column squeezed to ~538px by the 650px right column, so it
overflowed onto the ecosystem map (review comment on #568).

- use white-space: pre-wrap so the headline wraps within its column
- stack the hero to a single column below 1680px (benchmark/traffic
  sections keep their original 1320px breakpoint)
- pin the ecosystem column to 650px and cap the headline at 76px so
  the two-line headline fits beside it at 1681px and wider

Co-Authored-By: Claude Code <noreply@anthropic.com>

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-24 11:58:05 +08:00
jinliyl
c9a9728164
docs: link readmes to documentation site (#567)
Some checks failed
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Python packages / Build and verify distributions (push) Has been cancelled
2026-09-23 11:11:32 +08:00
WQS
ce386f5391
feat(auto-resource): unify text and image auto-resource using agentwrapper (#555)
* feat(auto_resource): unify text and image agent workflows

* test(auto_resource): streamline coverage and clarify image prompts

* refactor(auto_resource): simplify shared agent interpretation

* test(codex): isolate stdio startup budgets and teardown

* refactor(auto-resource): align image options and wrapper backend checks

* fix(auto-resource): finalize image notes after agent errors

* fix(auto-resource): complete image agent review fixes

* refactor(auto-resource): simplify shared reply failure finalization

* refactor(auto-resource): compose shared resource instructions

* fix(auto-resource): preserve prompt configuration compatibility

* refactor(auto-resource): remove legacy image prompt aliases
2026-09-23 10:20:35 +08:00
jinliyl
873bcee220
feat(scripts): plot cumulative star growth over a configurable period (#562)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
* feat(scripts): plot cumulative star growth over a configurable period

github_star_growth.py only reported daily new stars. Reconstruct the running
star total from the repository's current count and the daily increments, and
render it as an SVG line chart so the growth trend is readable at a glance.

- add --period (default 90d) accepting d/w/m/y units, keep --days as an alias
- crop the vertical axis near the data instead of anchoring it at zero
- label the chart with the plotted date range and the final star count
- report the star count at the start of the window on stderr

Co-Authored-By: Claude <noreply@anthropic.com>

* style(scripts): add trailing comma expected by pre-commit

The add-trailing-comma hook rewrites this call, so CI failed on the first
push of #562. No behaviour change.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-21 12:06:28 +08:00
jinliyl
19472233be
fix(docs): repair localized blog navigation (#560)
* fix(docs): repair localized blog navigation

* fix(docs): contain memory tag diagram labels

* docs: publish English memory tags article

* docs: align English memory tags title
2026-09-21 11:28:19 +08:00
dependabot[bot]
59e4ed7d3b
chore(deps): bump the codeql group with 2 updates (#561)
Bumps the codeql group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.38.0 to 4.38.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](b96794f015...1c5b675653)

Updates `github/codeql-action/analyze` from 4.38.0 to 4.38.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](b96794f015...1c5b675653)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.38.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.38.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-21 10:21:06 +08:00
jinliyl
07d4d6e838
docs: add Memory Tags blog (#559)
Some checks are pending
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze python (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
* docs: add Memory Tags blog

* docs: clarify Memory Tags behavior

* docs: name automatic tag workflows
2026-09-20 18:03:20 +08:00
WQS
6125fc197d
feat(auto-memory): add session image support (#532)
* feat(auto-memory): add opt-in image input

* feat(auto-memory): caption session images into source-linked notes

* fix(auto-memory): version Pillow 10 compatible image preparation

* fix(auto-memory): harden image evidence and retry boundaries

* fix(auto-memory): preserve image evidence across replay and concurrent writes

Keep persisted image positions through disabled history backfills and transcript filtering. Merge note links atomically, map 16-bit grayscale without clipping, recheck restored caption owners, and reuse unchanged identity metadata during batch publication. Add regressions and document conservative custom-rename behavior.

* refactor(auto-memory): restore main baseline for image modes v2

* refactor(images): share resource caption preprocessing and model calls

* feat(watch): support scoped exclusions for managed session images

* feat(auto-memory): add opt-in resource and caption-only image input

* refactor(auto-memory): keep caption-only mode with text fallback

* refactor(auto-memory): make caption-only mode dispatch explicit

* docs(auto-memory): focus image guide on caption-only mode

* feat(auto-memory): add direct multimodal image extraction

* refactor(auto-memory): route direct images through the vision model

* test(auto-memory): consolidate overlapping image regressions

* feat(auto-memory): interleave direct images with conversation text

* docs(auto-memory): clarify history rendering scope for direct inputs

* feat(auto-memory): require vision declaration for direct-only images

* refactor(auto-memory): keep bound model and native image helpers

* refactor(auto-memory): keep image inputs native and opt-in

* refactor(auto-memory): simplify image docs and tests

* refactor(auto-memory): confine image adaptation to image requests

* refactor(auto-memory): remove image switch type validation
2026-09-20 15:35:00 +08:00
jinliyl
cae613d1c4
refactor(auto-fin): one note per topic, merged by a separate digest step (#558)
* fix(auto-fin): parse topic IDs from fenced JSON replies

* fix(auto-fin): limit report agent tool calls in prompt

* refactor(auto-fin): research news by topic before market open

* fix(config): update default model version for claude_code backend

- Change model version from qwen3.8-max to qwen3.7-plus
- Use environment variable LLM_MODEL_NAME to allow override
- Ensure backend configuration reflects updated model setting

* refactor(auto-fin): write one note per topic before the daily digest

The merge step did two jobs at once: it researched every topic and
combined the results into a single report. Split it the way daily-paper
separates analysis from its brief, so each topic earns a durable note of
its own.

- auto_fin_research_step writes one note per topic that had relevant
  news, tagged `kind: auto-fin-topic` and `topic` in frontmatter
- auto_fin_digest_step merges those notes into the day's brief with no
  tools of its own and appends a `## 主题详解` section linking back to
  each note
- a same-day rerun finds a topic's note by its `topic` frontmatter and
  replaces it in place, deleting the old file when the title changed
- base.py now owns the shared Markdown layer: title sanitizing, report
  normalizing, wikilink validation, note lookup, atomic frontmatter
  writes, and change tracking, so both steps share one write path
- the DingTalk step maps `auto_fin_digest_path` to `markdown_path`
  explicitly instead of relying on whichever step ran last
- drop the unused AutoFinTopicOutput schema and read `job_tools` from
  the step config rather than hardcoding `search`

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(dingtalk): surface rejection details when delivery fails

A failed group send only reported HTTPStatusError, so an operator had to
reproduce the request by hand to learn why DingTalk refused it. Include
the status code and the whitelisted error keys from the response body in
both the log line and the raised RuntimeError.

Only `code`, `message`, and `requestid` are reported: the request body
carries the message content and credentials, so an error response that
echoes it back must not reach the log. Detail is truncated to 200 chars.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(test): make the suite green on CI

- Point the cookbook claude_code model assertion at qwen3.7-plus, the
  default commit 9404e600 set, so the pre-existing red stops blocking
- Satisfy pylint on the auto-fin tests: prefer implicit booleaness for
  the recorded Agent calls and drop an unused tmp_path fixture

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(auto-fin): name notes after the topic, not the Agent title

The research Agent returned a whole paragraph as its title; that became a
filename and blew past the filesystem's 255-byte name limit, failing with
ENAMETOOLONG inside resolve_note_path. Topics are configured values, so
they are short and predictable - use them for file names and keep the
Agent title in frontmatter.

- Name topic notes after the topic and the digest after the run date
- Fold a byte budget into normalize_title as a safety net for long topics
- Take an AutoFinReportOutput in _write_report instead of loose fields
- Ask both prompts for a short title now that it is display-only

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: isolate per-topic research failures and scope frontmatter reads

A single failing topic used to fail the whole cron job and discard the news
already gathered for the topics that had not run yet -- the 09-19 09:24 run
lost its robot notes that way. Research now logs the failure, continues with
the remaining topics, and only fails the run when no topic produced a note.

frontmatter_read was the only frontmatter step without the _allowed_paths
check that read, write, edit, and frontmatter_update already honour, so an
Agent scoped to one file could still read another file's metadata.

- Isolate per-topic research failures and report them as failed_topics
- Fail loudly when every topic fails so an empty brief is never sent
- Apply _check_path_permission in FrontmatterReadStep

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(auto-fin): keep hand-edited notes and wikilink delimiters from breaking the run

Addresses three review findings on the topic-per-note rework.

- `find_note` and `read_note` now skip a note whose YAML frontmatter does
  not parse. A hand-edited note in the day directory raised
  `yaml.parser.ParserError`, which per-topic isolation surfaced as
  "Auto Fin research failed for every topic" and took the run down with it.
- `normalize_title` also strips `[`, `]` and `#`, which `WikilinkHandler`
  treats as target delimiters. `AI[算力]` used to emit a trailer link the
  parser could not read at all, and `C#` resolved to `.../C` plus an anchor.
- `_write_report` returns the body it actually wrote, and both callers
  propagate it, so the digest answer and the note handed to the digest Agent
  no longer carry links that validation had already downgraded on disk.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-20 14:52:56 +08:00
jinliyl
5231f3970c
fix(hermes): preserve profile context and reset invalid provider state (#554)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* Fix Hermes provider profile thread compatibility and reinitialization

* docs(hermes): clarify compatibility checks were manual
2026-09-16 16:01:15 +08:00
jinliyl
9ebe17a89d
Fix cookbook embedding store config test (#553) 2026-09-16 11:09:50 +08:00
jinliyl
16269c9a76
Add max_retries and quota_retry_delay to cookbook.yaml
Some checks failed
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / DSH plugin / Validate DSH plugin (push) Has been cancelled
CI / OpenClaw plugin / Validate OpenClaw plugin (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
2026-09-15 20:32:34 +08:00
jinliyl
6c5d2194c0
docs(plugins): use local install paths (#552) 2026-09-15 20:15:52 +08:00
jinliyl
5f2c693ddb
feat(plugins): extract DingTalk integration (#551)
* feat(plugins): extract DingTalk integration

* feat(config): complete cookbook model setup

* docs(config): add cookbook startup guide

* fix(config): clarify DingTalk Claude tool access
2026-09-15 20:05:52 +08:00
jinliyl
dab56fc794
chore: bump ReMe and OpenClaw versions (#549) 2026-09-15 18:29:25 +08:00
jinliyl
6f4bdfd416
refactor(plugins): decouple DingTalk notifications (#550) 2026-09-15 18:29:06 +08:00
jinliyl
b9caae1e50
feat(service): support explicit wildcard network binds (#547)
* feat(service): bind network services to all interfaces

* fix(service): keep network listeners local by default

* fix(service): make remote access explicit
2026-09-15 14:26:33 +08:00
jinliyl
d67f1490f5
fix(dsh): support latest host prerelease (#548) 2026-09-15 13:31:45 +08:00
jinliyl
fe336da566
feat(studio): align branding with documentation site (#546)
* feat(studio): align branding with documentation site

* fix(studio): refine brand lockups

* fix(studio): simplify welcome branding

* docs(studio): add comprehensive visual guide

* docs: promote Studio to top-level navigation

* fix(studio): address review feedback
2026-09-15 13:29:55 +08:00
jinliyl
fd6337fe42
docs: refresh agent integration guides (#545)
Some checks are pending
CI / OpenClaw plugin / Validate OpenClaw plugin (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Python tests / Unit Tests - py3.14 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
CI / DSH plugin / Validate DSH plugin (push) Waiting to run
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / GitHub Actions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
* docs: refresh agent integration guides

* docs: clarify Claude Code transcript access
2026-09-14 17:55:08 +08:00
WQS
6ed97f033b
feat(auto_resource): reuse parsed resource ownership per batch (#529)
* perf(auto_resource): reuse historical ownership lookup per batch

Share a lazy ownership index across routed processors and expire it at the end of each resource batch. Refresh changed days and their safe aliases after writes, deletes, and partial failures.

Add regression coverage for scan counts, ownership, lookup scope isolation, and retry behavior.

* fix(auto_resource): reconcile shared lookup across active batches

* fix(auto_resource): refresh batch ownership from filesystem metadata

* refactor(auto_resource): simplify batch ownership lookup

Keep per-day ownership caches paired with pre-read metadata and refresh changed days before subsequent resources. Remove reverse-index bookkeeping, post-read stabilization retries, and recursive scope fallback.

Handle late ELOOP errors on Python 3.13 while preserving other OS errors. Update regression tests for cross-resource freshness and retain safety and lifecycle coverage.

* refactor(auto_resource): streamline batch lookup refresh

* refactor(auto_resource): scope ownership cache to batch mutations

Replace filesystem snapshots with one lazy historical lookup shared by resource processors. Reload only days changed by this invocation and discard the cache when it ends.

Reuse canonical daily-note scans for symlink safety and Python 3.13 compatibility. Consolidate regression tests around the narrowed batch contract.

Validation: Python 3.11, 3.12 and 3.13 core/plugin suites each passed 1407 tests; pre-commit --all-files and offline lookup/deletion comparison passed.

* refactor(auto_resource): reuse initial ownership without refreshes

* test(auto_resource): streamline ownership lookup regression coverage
2026-09-14 16:55:42 +08:00
jinliyl
46eca95bb9
fix(job): preserve nested exception details (#544) 2026-09-14 16:54:54 +08:00
jinliyl
4f7c8786e3
fix(integrations): correct Hermes plugin version (#543) 2026-09-14 13:03:00 +08:00
jinliyl
a518dd168b
chore(deps): bump CodeQL action to 4.38.0 (#542) 2026-09-14 12:02:54 +08:00
jinliyl
9ad3dafce5
chore(ci): harden and split workflows (#539)
Some checks failed
CI / DSH plugin / Validate DSH plugin (push) Has been cancelled
CI / OpenClaw plugin / Validate OpenClaw plugin (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / GitHub Actions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.14 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* chore(ci): harden and split workflows

* fix(ci): support token-based npm publishing

* test: make disappearing resource check portable

* fix(ci): make Studio releases recoverable

* fix(ci): stop Studio publishing on cancellation
2026-09-11 18:21:49 +08:00
Sen Huang
05958d4d8b
feat(evolve): tag auto-dream outputs (#538) 2026-09-11 14:08:29 +08:00
jinliyl
dff2d33cec
fix(daily-paper): keep digest generation after empty search (#537) 2026-09-11 13:22:37 +08:00
jinliyl
6cb81e7921
refactor(integrations): split TypeScript host plugins (#536)
* refactor(integrations): split TypeScript host plugins

* feat(integrations): refresh host compatibility and status UI

* fix(openclaw): secure status diagnostics
2026-09-11 13:06:02 +08:00
jinliyl
1be61b1e4c
feat(integrations): add dual-mode Hermes memory provider (#533)
* feat(integrations): add dual-mode Hermes memory provider

* style(integrations): apply repository formatting

* fix(integrations): address Hermes provider review

* fix(integrations): bound embedded recall startup cleanup

* fix(integrations): finish embedded application cleanup

* docs(integrations): expand Hermes verification guide
2026-09-11 12:25:18 +08:00
jinliyl
7e25d4679b
feat: auto-tag plugin-generated reports (#534)
* feat(plugins): auto-tag generated reports

* fix(logging): forward host records on Python 3.13

* refactor(tags): decouple auto tagging from index updates

* fix(tags): bind auto tagging to configured index

* fix(tags): preserve standalone default index

* fix(tags): make auto tagging best effort
2026-09-11 11:52:52 +08:00
xyf2020
9975bb37b9
Separate benchmark judge plugins (#535)
Some checks failed
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Documentation / Test and build documentation (push) Has been cancelled
2026-09-10 20:16:46 +08:00
xyf2020
06fb46fa48
feat(tags): add configurable tag indexing and filtered hybrid search (#530)
* Add optional tag generation and normalization to auto memory

* Add tag index components and clean up temporary JSONL files

* Preserve tag index state when reconciliation fails

* Refactor and streamline application implementation

* Fix pylint C1803 warnings in tag normalization tests

* Document optional tag index configuration

* Make tag index failures non-blocking and disable auto-memory tags

* Add configurable tag indexing and tag listing

* Add tag-filtered hybrid search with exact candidate ranking

* Remove obsolete generated files

* Rename tag index key to tag_key and reject reserved fields

* Extract automatic tagging into a dedicated step

* Restrict frontmatter updates to authorized keys

* Refine tag filtering and automatic memory tagging

* Require underscore-separated tags in auto-tag prompts

- Forbid spaces in tags and require underscores (e.g. sam_altman) in both
  English and Chinese auto_tag prompts, with English examples switched to
  English entities (OpenAI, gold)
- Drop prompt-string assertions superseded by the new tagging rule
- Merge construction/runtime tag_key validation tests into one parametrized case

* Make max_tags_per_file configurable in auto-tag step

* Consolidate tag index tests

* Fix search test fixture lint warnings

* Align tag contracts and index health behavior

* Fall back when tag index is unavailable

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-10 18:01:09 +08:00
jinliyl
1f67a6ce29
fix(docs): repair dark mode styling (#526)
Some checks failed
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
2026-09-07 17:33:36 +08:00
imrewce
354837f9af
feat(proactive): separate proactive refresh from auto dream (#488)
* refractor(proactive): upgrade proactive feature with disentangled job and steps

* refactor(proactive): apply audit fixes

- rename read-side job 'proactive' -> 'proactive_read' (less confusing vs the refresh pipeline)
- drop dedicated agent_wrapper.proactive; extraction reuses the default wrapper
- simplify schema: remove unused ProactiveExtractOutput/TopicUpdate, drop resource_paths
- extract no longer scans resource/ directly (daily notes already carry resource content)
- update tests and docs accordingly

* feat(proactive): strict extract-output gate and prompt total budget

- parse_extract_reply now requires a contract section (follow_ups/extends/updates
  as a list); non-empty replies with misspelled section names trigger the
  existing one-shot retry instead of silently checkpointing changed files
- pack_paths gains max_total_chars; extract packs newest daily material first,
  keeps the first file on overflow, and records omitted files in a trailer
  (default budget 300000 chars, configurable via max_total_chars)
- tests: schema gate unit, schema-error retry e2e, budget unit + e2e

* feat(proactive): add scenario-card plan step and generative agenda step

* feat(proactive): digest-personal profile personalization and leaner LLM contract

- extract/plan/agenda now draw a user profile block from <digest_dir>/personal/*.md
  (frontmatter description + body excerpt, per-file budget, profile.md fallback)
- all daily access honours the configured daily_dir (prompt paths parameterized,
  config-driven fallbacks) so workspaces using e.g. memory/ work unchanged
- schema trim: drop dead fields errors/material_paths, carry_forward_all -> count
- shrink LLM output contract: new topics emit title/reason/confidence/paths only;
  keywords removed end-to-end, evidence derived from paths[0] (updates keep it)

* fix(proactive): skip checkpoint when extract reply stays unusable after retry

Two consecutive unparseable replies now short-circuit the round without
checkpointing, so the same material is retried next round instead of being
silently consumed (closes the residual audit #1 gap: the structural gate
detected schema-wrong output but a double failure still checkpointed).

* fix(proactive): replace running bool with reference-counted job activity tracker for the idle gate

* refactor(proactive): remove job activity tracking and idle gate, restore job tree to upstream

* fix(proactive): address second audit round (readonly reader, mtime checkpoint, wider fallbacks, profile containment, horizon content, expiry boundary)

* refactor(dream): strip interests.yaml ownership from dream, proactive is now the sole writer

* refactor(dream): separate proactive topic generation

* ci: update renamed auto dream smoke test

* fix(proactive): complete refresh migration and docs

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-07 17:23:37 +08:00
xyf2020
f04eedb3ab
feat(Tag filter): Add an optional rebuildable tag index for file frontmatter (#517)
Some checks failed
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* Add optional tag generation and normalization to auto memory

* Add tag index components and clean up temporary JSONL files

* Preserve tag index state when reconciliation fails

* Refactor and streamline application implementation

* Fix pylint C1803 warnings in tag normalization tests

* Document optional tag index configuration

* Make tag index failures non-blocking and disable auto-memory tags

* fix(tag-index): fail closed and support reindexing

* fix(tag-index): preserve complete query expressions

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-07 15:52:17 +08:00
Amir Fathi
36e3a87c75
fix(embedding): retry 429 rate-limit errors instead of dropping the batch (#525)
* fix(embedding): retry 429 rate-limit errors instead of dropping the batch

openai.RateLimitError is not a TimeoutError/ConnectionError/OSError, so
_call_with_retry's except Exception branch caught it and returned None on
the first attempt with zero backoff. Add _is_rate_limited, mirroring the
existing _is_insufficient_quota duck-typed check, and retry a 429 with the
same exponential backoff used for network errors.

* fix(embedding): insufficient_quota errors carrying status_code=429 no longer bypass quota handling

_is_rate_limited() checked status_code == 429 first, so an OpenAI-compatible
insufficient_quota error (which also carries status_code=429) matched the
generic rate-limit branch before the code=insufficient_quota check ever ran.
That meant a real quota exhaustion retried on the wrong backoff (or not at
all, when quota_retry_delay is unset) instead of the dedicated quota_retry_delay
wait.

_is_rate_limited() now defers to _is_insufficient_quota() first. The existing
quota test double now sets status_code=429 to match the real OpenAI error
shape, which is what exposes the regression without the fix.

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>

* fix(embedding): log rate-limit retries

---------

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-07 14:31:48 +08:00
WQS
5c17874f73
feat(auto_resource): interpret image resources into daily notes (#500)
* feat: add auto_image step for image resource caption notes

* feat: wire image resources into the resource watch loop

* refactor: split auto resource processors behind router

* refactor: align resource processor module names

* refactor: preserve auto resource compatibility

* refactor: clarify auto resource routing structure

* fix: address auto resource review concerns

* test: scope auto resource fixtures

* docs: align auto resource processor wording

* test: cover image resize failures

* fix: harden image resource lifecycle

* fix: preserve resource image detail and linked daily ownership

* style(file-graph): stabilize multiline docstring formatting
2026-09-07 12:00:38 +08:00
jinliyl
0eba6ea831
feat(docs): redesign homepage and add traffic dashboard (#522)
Some checks failed
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* feat(docs): redesign homepage and add traffic dashboard

* feat(docs): publish readmes as project overview
2026-09-05 00:53:32 +08:00
jinliyl
193fd418fb
fix(docs): use uncached navigation icon (#521) 2026-09-04 17:00:29 +08:00
jinliyl
8c3d3016e3
docs: refresh ReMe homepage branding (#520) 2026-09-04 16:43:29 +08:00
jinliyl
dc28e62526
feat(typescript): update DSH adapter and publish integration guides (#519)
* feat(typescript): update DSH integration guides

* docs(typescript): address DSH review feedback
2026-09-04 16:16:01 +08:00
jinliyl
88ed21165b
docs: rebuild documentation site with VitePress (#518)
* docs: rebuild documentation site with VitePress

* docs: address VitePress review feedback

* docs: complete locale redirect validation
2026-09-04 14:50:50 +08:00
xyf2020
3f2eb6235f
feat(benchmark): extract BEAM and LongMemEval into standalone plugins (#512)
* Simplify project implementation

* Centralize benchmark agentic answer base class

* Remove bundled ReMe source snapshots

* Preserve local benchmark configs and plugin discovery behavior

* docs: enrich job parameter descriptions in beam and lme plugin configs

* Simplify project structure and remove obsolete code

* Move benchmark search step configuration into BEAM and LME plugins

* Rename benchmark judge packages to avoid import collisions

* Remove explicit plugin package loading in favor of entry-point discovery

* Export benchmark plugin Steps from public packages
2026-09-03 14:06:39 +08:00
jinliyl
8c4898999d
fix: pin ClawHub source ref to publish SHA (#515) 2026-09-01 22:02:27 +08:00
jinliyl
c1b85f8241
fix: let ClawHub infer trusted publisher owner (#514) 2026-09-01 21:43:39 +08:00
jinliyl
f9a45a319a
fix: publish an OpenClaw-specific ClawHub artifact (#513)
* fix: publish an OpenClaw-specific ClawHub artifact

* fix: enable ClawHub publication
2026-09-01 20:27:47 +08:00
jinliyl
65cb4ebdd6
fix: skip dependency bindings in status memory walk (#510)
* fix: skip dependency bindings in status memory walk

* chore: bump version to 0.4.1.11
2026-08-31 16:24:41 +08:00
jinliyl
bec7e48772
docs: move environment setup before service start (#509) 2026-08-31 16:18:46 +08:00
jinliyl
fc4a5398a8
fix(index): make embedding rebuild explicit and scoped (#508)
* fix(index): make embedding rebuild explicit and scoped

* fix(index): harden scoped reindex completion

* fix(index): guard embedding space transitions

* fix(index): serialize reindex with mutations

* docs(index): clarify scoped reindex semantics

* docs(index): explain synchronous checkpoint snapshots

* fix(index): serialize checkpoint publication
2026-08-28 23:25:17 +08:00
jinliyl
21f7757c80
ci: publish reme-ai from GitHub releases (#507) 2026-08-28 17:01:43 +08:00
jinliyl
c85917a812
feat(core): support live component replacement (#505)
* feat(core): support live component replacement

* fix(core): serialize replacement with app lifecycle

* fix(core): preserve state during component replacement

* fix(core): roll back partial component startup

* chore(deps): add 'web' extra to reme-ai and remove reme_studio from core dependencies

- Updated reme-ai dependency to include 'as' and 'web' extras
- Removed reme_studio from core dependency list to avoid duplication or unnecessary install

* fix(core): preserve persisted state on startup failure
2026-08-28 16:04:02 +08:00
jinliyl
157b096448
ci(deps): automate AgentScope updates (#504)
* ci(deps): automate AgentScope updates

* fix(ci): enforce AgentScope release pin contract
2026-08-28 13:27:21 +08:00
jinliyl
99afc2604f
fix(release): harden embedding store and plugins for ReMe 0.4.1.9 (#503)
Some checks failed
CI / Python packages / Build and verify distributions (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / ReMe Studio / Studio checks (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* chore(release): prepare ReMe 0.4.1.9

* refactor(config): remove daily_cookbook and streamline plugin configs

- Delete the entire daily_cookbook.yaml standalone application config
- Remove qwenpaw dependencies verification and related CI workflow steps
- Simplify release workflows by removing qwenpaw verification and enforcing reme-ai >=0.4.1.9
- Update plugin start commands and examples to use 'default' or 'demo' configs instead of daily_cookbook
- Adjust imports and tests related to daily_cookbook removal and injected_job_kwargs enhancements
- Refactor agent wrapper to support injected_job_kwargs for job parameter injection in auto-fin and daily-paper
- Improve daily_paper digest prompt to include configured daily directory and correct historical search constraints
- Update dependency versions in pyproject.toml files to require reme-ai >=0.4.1.9 and remove qwenpaw optional dependencies
- Clean up unused environment variables and obsolete test cases related to daily_cookbook and verification steps

* fix(local_embedding_store): retry batch computation on vector space changes

- Add up to 3 attempts to recompute embedding batch if vector space changes during processing
- Log warnings when maximum retries reached and discard stale results
- Prevent caching results from outdated vector spaces to maintain consistency
- Add tests to verify retry behavior and abort after continuous vector space churn

fix(daily_paper): update digest search logic and tests

- Change search to query existing memory, not only previous articles in daily_dir
- Allow multiple searches outside daily_dir but limit links to dated markdown in daily_dir before today
- Update test assertions to reflect revised search and linking rules

* fix(embedding): retry vector space changes per request
2026-08-28 11:35:04 +08:00
jinliyl
2dd2255760
ci: update release workflow actions and smoke checks (#502)
* ci: update artifact actions for Node 24

* ci: validate Auto Fin package manifest
2026-08-27 18:03:44 +08:00
jinliyl
d8d667c6ac
docs: refresh ReMe Studio preview image (#501) 2026-08-27 17:50:48 +08:00
jinliyl
940a923f06
ci: allow bootstrap release before qwenpaw plugins (#499) 2026-08-27 17:23:53 +08:00
jinliyl
3d2ecc60d2
feat(service): expose MCP through HTTP backend (#498)
* feat(service): expose MCP through HTTP backend

Serve JSON/SSE job endpoints and streamable HTTP MCP from one FastAPI application, sharing the same jobs and application lifecycle. Preserve the standalone MCP backend, add configurable MCP HTTP settings, update startup metadata and integration docs, and cover routing, lifecycle, configuration, and compatibility behavior with unit tests.

* fix(service): preserve MCP request protections

Route the exact MCP path through the complete FastMCP ASGI application so its middleware and state remain active. Reject non-literal MCP paths and validate reserved Job conflicts before tolerant service registration. Add regression coverage for middleware preservation, route syntax, and startup failure.

* fix(service): reject encoded MCP paths

Reject percent signs in mcp_path so ASGI path decoding cannot turn an accepted configuration into an unreachable route. Cover encoded slash, space, and double-encoded slash inputs.
2026-08-27 17:23:11 +08:00
jinliyl
6f38d201b6
ci: harden build and release workflows (#497) 2026-08-27 16:43:05 +08:00
jinliyl
ef3f99f019
refactor(packaging): reorganize published packages (#495)
* refactor(packaging): reorganize published packages

* fix(packaging): install AgentScope extra in wheel smoke

* docs: align package guides and documentation site

* ci(workflow): add core dependency verification step in Python package build

- Add a workflow step to verify released core dependencies by installing the wheel with core extras
- Assert the presence of the static index.html file to ensure proper package contents
- Create and use a temporary virtual environment for isolation during verification
- Keep existing artifacts upload step intact and conditional on inputs.upload_artifacts flag

* fix(ci): update package installation dependencies in Windows workflow

- Change pip install from editable reme_studio and core to only dev and as extras
- Remove installation of reme_studio and core to streamline dependency setup
- Ensure Windows CI uses the correct extras for testing environment

* fix(tests): add missing commas in toml file reads in package version tests

- Added trailing commas in the tomllib.loads calls for auto-fin and daily_paper configs
- Ensured consistent syntax to prevent potential tuple misinterpretation
- Improved readability and correctness of the test setup code

* fix(packaging): protect qwenpaw releases and test Studio health
2026-08-27 14:02:09 +08:00
jinliyl
b78e32ef03
feat(openclaw): align ReMe plugin with current SDK (#493)
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / TypeScript integrations / Type-check, test, and pack (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
Adopt definePluginEntry, before_prompt_build, current manifest contracts, and official OpenClaw SDK types.

Add DSH-aligned memory batching, retryable shutdown flushing, daily Auto Dream scheduling, updated documentation, tests, ClawHub validation, and optional release publishing.
2026-08-26 19:57:57 +08:00
jinliyl
6a6e0b3c29
fix(dsh): deduplicate pending memory guidance (#494) 2026-08-26 19:37:24 +08:00
jinliyl
a457bf7542
docs: reorganize readme around agent integrations (#492) 2026-08-26 19:30:16 +08:00
jinliyl
513fb5b7f4
feat: extract Daily Paper into an independently packaged plugin (#491)
* feat: extract Daily Paper into a plugin

* fix: satisfy clean-environment quality checks

* fix: address daily paper review feedback
2026-08-26 17:32:46 +08:00
jinliyl
1a6b584274
fix(persistence): avoid duplicate index dumps (#489)
* fix(persistence): avoid duplicate index dumps

* fix(persistence): align dumps with component ownership

* fix(persistence): preserve subclass dump hooks
2026-08-26 16:51:00 +08:00
jinliyl
15d12be6b6
fix(index): tolerate invalid text encoding (#490)
* fix(index): tolerate invalid text encoding

* fix(index): preserve text chunker compatibility
2026-08-26 16:16:26 +08:00
jinliyl
626c850ccb
fix(daily-paper): sanitize Unicode surrogates (#487)
Some checks failed
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* fix(daily-paper): sanitize Unicode surrogates

* fix(daily-paper): sanitize analysis workflow state
2026-08-24 18:59:53 +08:00
jinliyl
01ef1a6efb
ci: use trusted publishing for TypeScript package (#486)
* ci: use trusted publishing for TypeScript package

* docs: scope npm announcement to DeepSeek Harness
2026-08-24 15:21:28 +08:00
jinliyl
efcc2b34d1
feat: simplify plugin setup and add management CLI (#485)
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
* feat: simplify plugin setup and add management CLI

* fix: isolate plugin CLI import side effects

* refactor: streamline plugin validation

* fix: route plugin CLI arguments independently

* fix: support standard plugin source layouts
2026-08-23 17:58:20 +08:00
jinliyl
c8e1248769
fix: recover embeddings after transient health check failure (#471)
Some checks failed
CI / Python tests / Unit Tests - py3.12 (push) Has been cancelled
CI / Python tests / Unit Tests - py3.13 (push) Has been cancelled
CI / TypeScript integrations / Type-check, test, and pack (push) Has been cancelled
CI / Windows / CLI smoke - py3.11 (push) Has been cancelled
Deploy / Documentation / Build documentation (push) Has been cancelled
Security / CodeQL / Analyze javascript-typescript (push) Has been cancelled
Security / CodeQL / Analyze python (push) Has been cancelled
CI / Documentation / Test and build documentation (push) Has been cancelled
CI / Python quality / Pre-commit (push) Has been cancelled
CI / Python tests / Unit Tests - py3.11 (push) Has been cancelled
Deploy / Documentation / deploy (push) Has been cancelled
* fix: recover embedding after transient health failure

* refactor(embedding_store): remove provider_success_count and simplify health recovery logic

- Deleted provider_success_count attribute and related methods across embedding and file stores
- Updated _recover_after_real_request to rely solely on is_healthy flag for recovery decisions
- Removed redundant counting logic for provider successes during embedding operations
- Cleaned up health status management to streamline provider recovery detection
- Adjusted unit tests to align with removal of provider_success_count and maintain health checks consistency

* refactor(embedding_store): use default health check timeout

* fix(embedding_store): ensure is_healthy remains unchanged on cache hits

- Updated get_embeddings docstring to clarify cache hits must not alter is_healthy state
- Improved code comment for embedding dimension matching method

* fix(file_store): make embedding recovery race-safe

* ci: use default CodeQL query suite

* fix(file_store): preserve queued embedding rebuilds

* fix(file_store): preserve verified recovery without chunks
2026-08-21 13:58:51 +08:00
jinliyl
8416fd3ac9
feat: add unified TypeScript agent integrations (#483)
* feat: add unified TypeScript agent integrations

* fix: normalize endpoints without regex backtracking

* fix: address TypeScript integration review feedback

* fix: preserve original OpenClaw prompts

* fix: bound pending OpenClaw prompts
2026-08-21 13:57:14 +08:00
jinliyl
f44f52d919
fix(embedding): exclude provider init from health timeout (#484)
Some checks failed
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Website / Website checks (push) Has been cancelled
CI / Python packages / Build and verify distributions (push) Has been cancelled
2026-08-21 11:23:14 +08:00
jinliyl
ebcb154e37
fix(search): isolate range dedup state (#465) 2026-08-20 16:15:30 +08:00
jinliyl
39233f4e62
ci: remove Dependabot configuration (#482) 2026-08-20 16:14:52 +08:00
jinliyl
94b7dedc26
Delete .github/README.md 2026-08-20 16:03:55 +08:00
jinliyl
87187c1d25
ci: organize GitHub automation (#466) 2026-08-20 15:56:30 +08:00
jinliyl
f5ec230fef
feat: add DSH memory integration and organize extensions (#461)
* feat: add DSH memory integration and organize extensions

* fix: support newer DSH release candidates

* fix: address DSH integration review feedback

* fix: handle DSH cross-day retry edge cases
2026-08-20 15:31:51 +08:00
jinliyl
2f5fd46b44
refactor: deduplicate entry-point loading (#460)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
GitHub Pages Check / test-and-build (push) Has been cancelled
Package Check / distributions (push) Has been cancelled
Deploy ReMe documentation / build (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Deploy ReMe documentation / deploy (push) Has been cancelled
* refactor: deduplicate entry-point loading

* fix: preserve config conflict error priority
2026-08-19 19:57:53 +08:00
jinliyl
618e8cec66
feat: add entry-point plugin system and extract Auto Fin (#459)
* feat: add entry-point plugin system

* fix: harden plugin config and client loading

* docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI

- Provide step-by-step instructions for updating project.version and merging branches
- Explain dependency verification for reme-ai on PyPI during build
- Specify requirements for GitHub Actions secret configuration and version uniqueness
- Describe manual workflow triggering and input of version number
- Recommend publishing order for related projects
- Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag

* feat: support plugin-defined component types

* refactor: simplify plugin configuration

* fix: isolate plugin loading and defer client fallback

* refactor: freeze built-in component registry

* fix: isolate config entry point loading

* fix: complete auto-fin package metadata
2026-08-19 17:23:23 +08:00
jinliyl
d3aee1adf5
feat(evolve): report auto-dream file changes (#458)
* feat(evolve): report auto-dream content changes

* perf(evolve): use lightweight dream snapshots
2026-08-19 15:44:06 +08:00
jinliyl
6b9a75267b
fix(ci): support AgentScope 2.0.6 initialization (#457)
Some checks are pending
Package Check / distributions (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-19 11:33:52 +08:00
jinliyl
c792fd197c
Update agentscope version to 2.0.6
Some checks are pending
Package Check / distributions (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-18 11:04:26 +08:00
jinliyl
fd2894f939
fix: harden the 0.4.1.7 release configuration (#456)
Some checks failed
NPM Format / Website checks (push) Has been cancelled
GitHub Pages Check / test-and-build (push) Has been cancelled
Package Check / distributions (push) Has been cancelled
Deploy ReMe documentation / build (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Deploy ReMe documentation / deploy (push) Has been cancelled
* fix(packaging): harden the Studio release workflow

* chore(daily-paper): tune scheduled discovery defaults

* fix(docs): link the ReMe blog to GitHub Pages

* fix(docs): increase Chinese hero title spacing

* refactor(docs): share hero title line spacing

* fix(docs): keep desktop hero copy on two lines

* fix(docs): widen the home hero description

* style(docs): loosen hero title line height

* fix(docs): hide Markdown frontmatter in rendered pages

* docs(readme): simplify installation and remove standalone ReMe Studio instructions

- Remove references to separate ReMe Studio package and static build steps
- Clarify that `core` extra includes common integrations including Studio
- Update installation instructions to use `pip install -e ".[core]"`
- Remove detailed Studio usage and frontend development instructions
- Note that Studio is included with `core` and optional via `web` extra
- Simplify Quick Start guide by removing Studio usage step
- Remove mentions of serving Studio with HTTP service when using extras
- Update both English and Chinese README files accordingly

* docs(readme): streamline and clarify memory design and operations

- Remove redundant explanations about core extra installation
- Simplify memory processing flow description for clarity
- Clarify memory workspace directory default and customization
- Condense automatic memory flow to emphasize rebuildable metadata
- Refine search functionality explanation with RRF fusion details
- Shorten and clarify agent integration description, removing redundancy
- Update and simplify the operations command list, removing less common commands
- Revise community and support section for conciseness and clarity
- Maintain parallel updates in both English and Chinese README files

* test(bump_version): add tests for version bumping and consistency checks

- Add dynamic loading of bump_version and package_studio scripts for testing
- Test that studio package and dependencies have matching versions
- Implement fixtures to write temporary version files for testing
- Add test ensuring bump_version updates all relevant files and dependencies
- Add test to reject inconsistent version sources before writing
- Refactor tests to use common REPOSITORY path variable
- Include imports and setup for pytest in test file

feat(bump_version): create script to update ReMe and Studio versions

- Implement version reading from __init__.py and pyproject.toml files
- Validate current versions are consistent across files before updating
- Update version strings atomically to avoid partial writes
- Ensure exact pinning of studio dependency in main package extras
- Validate new version format against a safe pattern
- Provide CLI interface to bump versions from command line
- Raise errors if expected version declarations or pins are missing or duplicated

* fix(release): validate split package publishing

* fix(release): improve validation diagnostics

* fix(release): sync docs and workflow inputs

* fix(release): split PyPI publish jobs
2026-08-13 17:22:00 +08:00
jinliyl
2a05914150
feat: distribute Studio as an optional package (#454) 2026-08-13 11:07:37 +08:00
jinliyl
29eb51d7ba
fix: show resolved service URL and shrink Studio preview asset (#453)
Some checks are pending
NPM Format / Website checks (push) Waiting to run
Deploy ReMe documentation / build (push) Waiting to run
Deploy ReMe documentation / deploy (push) Blocked by required conditions
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* fix: show resolved service address in startup banner

* perf(website): reduce social preview image size

* fix: resolve MCP transport in startup banner
2026-08-12 19:37:31 +08:00
jinliyl
da9a8b7810
fix(docs): make homepage cards use direct links (#452) 2026-08-12 13:28:23 +08:00
jinliyl
dbf2a17da6
docs: expand ReMe documentation site (#451) 2026-08-12 13:18:40 +08:00
jinliyl
64249873ce
fix(website): resolve Dependabot dependency alerts (#450) 2026-08-12 12:20:41 +08:00
jinliyl
28fa636506
fix(docs): use public npm registry for Pages (#449) 2026-08-12 12:03:19 +08:00
jinliyl
52fdd446fb
docs: add standalone GitHub Pages site (#448) 2026-08-12 11:55:35 +08:00
jinliyl
ab66f2bb56
docs: refresh ReMe guides, diagrams, and Studio documentation (#447)
* docs: update ReMe documentation URL

* docs: localize ReMe Studio social image

* docs(AGENTS): update agent guidelines and repository documentation structure

- Clarify coding agent guidance for keeping changes small and consistent
- Revise project principle descriptions for clarity and modern terminology
- Expand repository map with detailed component and folder explanations
- Add configuration and CLI usage instructions, including syntax and merging rules
- Elaborate on component, step registration, and application lifecycle processes
- Define jobs, steps, and state handling conventions for stateless design
- Specify workspace and file safety policies, including path restrictions and locking
- Update validation commands and testing environment recommendations
- Clarify coding and test conventions, including style and dependency policies
- Distinguish documentation boundaries and update website content contribution notes
- Reinforce change guardrails to avoid breaking backward compatibility and data loss
- Improve svg diagram formatting and textual details in auto dream and proactive flow image

* style(docs): fix font-family syntax in SVG style definitions

- Correct quotation marks around font-family names in memory-as-file.svg
- Standardize font-family formatting by removing unnecessary quotes in reme-blog-architecture.svg
- Ensure consistent CSS style formatting within SVG files for better rendering fidelity

* docs: add ReMe blog to news

* style(docs): inline svg styles and improve text formatting

- Convert multiline SVG style tags into single-line for compactness in multiple figures
- Remove redundant line breaks in subtitle text elements for consistency
- Shorten descriptive texts in SVG figures for clarity and conciseness
- Adjust font sizes and text for better readability in SVG elements
- Correct whitespace issues in Chinese markdown document for improved formatting
- Remove unused style blocks from framework structure SVG for cleaner code
2026-08-12 10:59:03 +08:00
jinliyl
215c1f72f2
feat: refine local-first research and memory workflows (#444)
Some checks are pending
NPM Format / Website checks (push) Waiting to run
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* feat: refine local-first research workflows

* fix: delegate structured output tool choice

* refactor(auto-fin): fetch and filter rolling CLS news

* fix(auto-fin): keep imports portable across platforms

* feat(auto-fin): expose CLS fetch controls

* fix(auto-fin): propagate configurable news window

* feat(auto_fin): normalize hybrid wikilinks in report body

- Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations
- Use regex to identify hybrid wikilinks with optional destinations
- Replace redundant destinations with simpler wikilink format for clarity
- Ensure normalization is failure-safe with exception handling and logging
- Update report body normalization process to apply hybrid wikilink fix
- Add unit tests to verify correct normalization and failure safety behavior

* fix(dream): serialize integration with application-wide asyncio lock

- Add application-wide asyncio.Lock to serialize digest writes during integration
- Update _snapshot_digest to capture metadata per bucket
- Validate bucket association when recovering from file changes
- Add tests ensuring recovery only from the correct bucket
- Add tests confirming integration lock is shared across application context
- Enhance strict topic YAML loading validation in dream utils
- Add tests for strict topic loading rejecting invalid or lossy fields

* fix(cookbook): enable configurable job_tools for digest and merge steps

- Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps
- Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list
- Modify AutoFinMergeStep to similarly read job_tools from kwargs
- Update tests to pass job_tools explicitly when invoking these steps
- Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling

* fix: retry incomplete dream receipts

* perf(pdf): increase max PDF pages limit from 20 to 35

- Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml
- Modified code to extract up to 35 pages instead of 20 in analyze.py
- Updated README and README_ZH to document the increased max_pdf_pages
- Adjusted unit test assertions to reflect new max_pdf_pages limit of 35

* fix memory integration and daily paper links

* docs clarify cookbook tool usage
2026-08-11 23:32:34 +08:00
jinliyl
9533c17d51
feat(web): serve workspace from HTTP service (#446)
* feat(web): add the ReMe workspace frontend

* feat(web): serve workspace from HTTP service

* test(web): satisfy pylint docstring checks

* fix(web): use same-origin API safely

* fix(web): preserve API route semantics
2026-08-11 23:32:24 +08:00
jinliyl
b8f48c8004
feat(web): add the ReMe Studio frontend (#418)
* feat(web): add the ReMe workspace frontend

* fix(web): use public npm registry in lockfile

* fix(web): address workspace review feedback

* fix(web): protect drafts and report file limits

* fix(web): finish chat streams after tab switches

* feat(web): rename frontend to ReMe Studio
2026-08-11 19:47:59 +08:00
imrewce
3924f89bb4
feat(bench): adding eval adapter for proactiveness on Pi-Bench (#439)
* feat(bench): adding eval adapter for proactiveness on Pi-Bench

* Revise README for π-Bench evaluation suite

Updated the README to reflect the new project name and description.

* fix(bench): refining pi-bench scripts according to cr comments

* fix(bench): restore agent builtin tools in prebuilt toolkit
2026-08-11 16:37:54 +08:00
jinliyl
c7dbf31c3f
docs: expand ReMe guides and agent integrations (#445) 2026-08-11 13:31:11 +08:00
imrewce
58276f740b
fix(file_io): auto appending suffix for all related steps (#430)
* fix(file_io): auto appending suffix for all related steps

* fix(file_io): covering boundary cases of potential directory input
2026-08-11 11:09:28 +08:00
imrewce
3095564313
docs: Adding pi-bench related proc performance to blog draft (#443)
* docs(blog): refine proactive section wording in zh reme-blog

* chore(doc): supplementing proc related performance
2026-08-11 10:59:51 +08:00
jinliyl
21057931a9
fix(embedding): isolate caches by vector space (#442)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* fix(embedding): isolate caches by vector space

* fix(embedding): stabilize cache space switching

* Revert "fix(embedding): stabilize cache space switching"

This reverts commit 74193c9a0a.

* fix(embedding): include resolved OpenAI endpoint in cache ID

* fix(embedding): stabilize cache space switching

* fix(embedding): isolate Ollama endpoint caches
2026-08-10 22:42:16 +08:00
Zhaoyang Liu
5a5855f5ff
docs: refine ReMe launch blog (#440)
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-08-10 18:45:06 +08:00
jinliyl
072cb6a55b
feat(daily-paper): add opt-in Hugging Face mirror support (#437)
* feat(daily-paper): add Hugging Face mirror switch

* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env

The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.

HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(daily-paper): address mirror configuration feedback

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 15:57:24 +08:00
imrewce
fca42f4e6c
docs(blog): refine proactive section wording in zh reme-blog (#438) 2026-08-10 15:57:16 +08:00
jinliyl
d5e0d2837b
refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents (#432)
Some checks failed
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
Windows Smoke / CLI smoke - py3.11 (push) Has been cancelled
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
* refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents

Rework the auto-fin and daily-paper cookbooks to run on structured-output
LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with
data-source mirrors, and rewrite the affected unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(auto_fin): unify JSON output serialization and writing

- Extracted _write_output static method to serialize and write Pydantic models as compact JSON
- Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps
- Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep
- Updated auto_fin merge step to write output via new _write_output method
- Enhanced news reading with caching in AutoFinHistoryStep
- Refined returns calculation to handle events before close on non-trading days correctly

feat(daily_paper): improve note path resolution and metadata handling

- Introduced iter_note_metadata generator for safe Markdown frontmatter iteration
- Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles
- Updated analyze, collect, digest, and select steps to use centralized constants and helpers
- Used utc_now_iso for consistent timestamping in metadata
- Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps
- Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places
- Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management
- Improved error messages for filename validation in daily paper title normalization

feat(auto_fin): add multi-run cron schedules for intraday refinement

- Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates
- Each intraday run adds evidence cumulatively instead of replacing prior output wholly
- Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron

refactor(auto_fin_data): clean ETF code handling and page limits

- Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes"
- Added constants for major news and fund page limits to control pagination
- Improved ETF name extraction logic to handle missing fields consistently

fix(auto_fin_merge): fix report retrieval and merging logic

- Added support for getting current intra-day report in addition to previous day's report
- Modified merge template to include prior and current report sections for better context
- Adjusted report path handling to consistently use Path objects

test(auto_fin): add coverage for returns calculation and report retrieval

- Added test for returns when event occurs before close on non-trading day, checking next session entry
- Added test for previous and current report retrieval feeding merge context with disk files
- Extended test asserts for auto_fin cron schedule changes in config

style(daily_paper): reorder and cleanup imports

- Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import
- Cleaned up commented and unused imports across daily_paper steps

* feat: add configurable upstream mirror proxy

* style: format auto-fin data step

* fix: align cookbook mirrors and contracts

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-07 23:53:14 +08:00
jinliyl
e05b201da9
feat(backend): improve workspace support for web clients (#420)
* feat(backend): improve workspace support for web clients

* fix(config): preserve the default workspace directory

* chore(reme): bump version to 0.4.1.5

- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file

* fix(chat): disable builtin tools in read-only mode

* fix(agent): make builtin tools opt-in

* fix(list): tolerate files removed during mtime sort

* fix(chat): expose complete read-only job set
2026-08-07 23:52:56 +08:00
jinliyl
765103a597
docs(blog): add reme blog (#436)
* docs: add Chinese ReMe blog article

* fix(docs): update wiki links and adjust SVG path coordinates

- Removed file extensions from wiki link texts for consistency
- Modified path coordinates for relation lines in SVG illustration
- Added an arrow path with fill color to indicate direction in SVG diagram

* docs(blog): expand ReMe user guide and invite community contributions

- Add detailed descriptions for different ReMe user groups including intelligent agents,
  developers, researchers, engineers, and analysts
- Emphasize user control over data as editable Markdown files instead of black-box storage
- Introduce ReMe's long-term memory infrastructure accessible via multiple interfaces
- Highlight how ReMe can turn scattered information into personal knowledge networks
- Include a new "Welcome Contributions" section encouraging community involvement
- List areas for contribution such as integration, data sources, features, applications,
  documentation, and issue feedback
2026-08-07 17:36:02 +08:00
jinliyl
168b7194ab
docs: add Chinese ReMe blog article (#435) 2026-08-07 17:19:01 +08:00
lichen2015
e7b9274190
fix(stat): return text/markdown for .md files regardless of OS mime registry (#433)
On macOS, mimetypes.guess_type() may not recognize .md files, causing
stat to report application/octet-stream and breaking test assertions.
Explicitly map .md files to text/markdown so the behavior is stable
across platforms.
2026-08-07 16:31:03 +08:00
jinliyl
c5d92a24ab
feat: weave dream wikilinks into contextual prose (#428)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
2026-08-06 17:40:01 +08:00
jinliyl
9218a2d0e3
refactor: derive dialog paths from session_dir (#421)
* refactor: derive dialog paths from session directory

* fix: normalize configured session paths

* fix: align dialog watch paths with writers

* fix: reject absolute session directories
2026-08-06 17:07:14 +08:00
Ziyang Guo
6503e1271c
fix(prompt): default omitted conditional flags to false (#424)
Always apply conditional-line filtering so tagged prompt lines are removed unless the corresponding boolean flag is explicitly true. Add regressions for omitted flags with and without format variables.

Test: pytest tests/unit/test_prompt_handler.py -q
2026-08-06 16:22:59 +08:00
xyf2020
23d4c96c15
refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt (#422)
* chore(benchmark): isolate dataset/workspaces/results per benchmark

- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py

* feat(benchmark): add read tool to agentic answer and update BEAM results

- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)

* feat(auto_memory): add source line-number markers for note traceability

- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure

* feat(beam): enhance auto_memory with source markers and pilot ingest tooling

* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts

* feat: add CompressorStep and search_v2 dual-mode session compression

- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
  text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
  session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
  join_chunk_entries; session chunks now render line-aligned with
  L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
  per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
  query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths

* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results

* refactor(benchmark): make session compression config-driven via compress_session flag

Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.

* feat(lme/auto_memory): add source attribution markers with line numbers

Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.

* feat: improve agentic answer prompt and update beam 100K results

- Strengthen abstention rule: prohibit extrapolation from related but
  non-direct evidence
- Add multi-angle search after preliminary answer to check for
  conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
  including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
  (+0.140), abstention: 0.500 → 0.525 (+0.025)

* feat(read): add session-aware formatting for read tool and update BEAM eval

- Add truncate_session_output in _file_io.py to render jsonl session
  lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
  job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
  so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
  after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
  1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results

* chore(config): disable expand_links in beam and lme search_v2 configs

* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script

* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions

- Add compressor job and compressor as_llm component to reme/config/beam.yaml
  (aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
  when the compressor job is missing from the active config, log a warning
  and skip compression instead of raising 'Job compressor not found'.
  Skipped when there is no app_context so unit tests mocking run_job still
  drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
  before cd benchmark/beam/dataset (the directory is gitignored and absent
  in a fresh clone)

* fix(steps): guard compressor exceptions and fix ReadStep boolean override

1. search_v2: catch per-entry exceptions from run_job('compressor') inside
   compress() so asyncio.gather never propagates a compressor failure (e.g.
   temporary LLM outage). The failing entry keeps its original body while
   remaining entries are still compressed, preserving already-retrieved
   search results.

2. read: replace 'context_value or yaml_value' with an existence check so
   that a runtime-injected False can explicitly disable a YAML-true
   read_step_format_session flag.

Add focused unit tests for both paths.

* fix(search_v2): use existence check for strict_date_filter boolean override

Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.

* refactor(search): simplify strict_date_filter fallback to truthiness-or

* style(test): rename unused param to satisfy pylint W0613

* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt

- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
  per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
  benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
  description, keep search strategy, draft tool, and answer rules concise.

* docs(benchmark): update LME README_ZH results to latest eval run

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
2026-08-06 15:13:52 +08:00
718 changed files with 92666 additions and 13037 deletions

40
.dockerignore Normal file
View file

@ -0,0 +1,40 @@
.git
**/.DS_Store
**/.ssh
**/.codex
**/.claude
**/private*
**/.env
**/.env.*
**/.reme
**/.venv
**/venv
**/__pycache__
**/*.py[cod]
**/.pytest_cache
**/.mypy_cache
**/.ruff_cache
**/.coverage*
**/htmlcov
**/node_modules
**/dist
**/dist-static
**/.generated
**/.next
**/.vite
**/.wrangler
**/.cache
**/*.egg-info
**/build
**/logs
**/*.log
**/*.tmp
reme_studio/src/reme_studio/static
benchmark
cookbook
docs
github-pages
integrations
plugins
skills
tests

97
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View file

@ -0,0 +1,97 @@
name: Bug report
description: Report reproducible incorrect or unexpected ReMe behavior
title: "[Bug]: "
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve ReMe. Please remove secrets, API keys, and private memory content before submitting.
- type: textarea
id: description
attributes:
label: Description
description: What happened, and what did you expect instead?
placeholder: Describe the observed and expected behavior.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Provide the smallest configuration and command sequence that reproduces the problem.
placeholder: |
1. Configure ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: config
attributes:
label: Relevant configuration
description: Include only relevant values and redact credentials, tokens, endpoints, and private paths.
render: yaml
- type: textarea
id: logs
attributes:
label: Logs or traceback
description: Paste relevant output after removing secrets and private workspace content.
render: shell
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
validations:
required: true
- type: input
id: python-version
attributes:
label: Python version
placeholder: e.g. 3.11.9
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- Linux
- macOS
- Windows
- Other
validations:
required: true
- type: dropdown
id: area
attributes:
label: Affected area
options:
- CLI or configuration
- HTTP, MCP, or local service
- Memory or workspace files
- Search, catalog, graph, or index
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Packaging or installation
- Other
validations:
required: true
- type: checkboxes
id: safety
attributes:
label: Data safety
options:
- label: I removed credentials and private memory content from this report.
required: true

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View file

@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: ReMe documentation
url: https://reme.agentscope.io
about: Read the installation, configuration, and usage guides.
- name: Existing issues
url: https://github.com/agentscope-ai/ReMe/issues
about: Search for existing reports and discussions before opening a new issue.

View file

@ -0,0 +1,64 @@
name: Feature request
description: Propose a focused enhancement to ReMe
title: "[Feature]: "
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What user problem or limitation should this change address?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed behavior
description: Describe the desired behavior and its user-visible contract.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- CLI or configuration
- Jobs or steps
- Memory or workspace files
- Search, catalog, graph, or index
- Service or client
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Documentation
- Other
validations:
required: true
- type: textarea
id: ownership
attributes:
label: Local-first and compatibility considerations
description: Explain any effect on user-owned files, rebuildable state, configuration, schemas, or service interfaces.
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe workarounds or alternative designs you considered.
- type: textarea
id: examples
attributes:
label: Example usage
description: Show the proposed CLI, configuration, API, or UI behavior when useful.
render: shell
- type: checkboxes
id: contribution
attributes:
label: Contribution
options:
- label: I am willing to help implement or test this feature.

53
.github/ISSUE_TEMPLATE/question.yml vendored Normal file
View file

@ -0,0 +1,53 @@
name: Usage question
description: Ask for help using or configuring ReMe
title: "[Question]: "
labels: [question]
body:
- type: markdown
attributes:
value: Please check the documentation and existing issues before asking a new question.
- type: textarea
id: goal
attributes:
label: What are you trying to achieve?
validations:
required: true
- type: textarea
id: attempted
attributes:
label: What have you tried?
description: Include relevant commands or configuration, with secrets and private memory content removed.
validations:
required: true
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
- type: dropdown
id: area
attributes:
label: Area
options:
- Installation
- Configuration
- CLI or service usage
- Memory and workspace management
- Search and retrieval
- ReMe Studio
- Plugin or integration
- Other
- type: checkboxes
id: checked
attributes:
label: Before submitting
options:
- label: I checked the [ReMe documentation](https://reme.agentscope.io) and searched existing issues.
required: true
- label: I removed credentials and private memory content.
required: true

35
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,35 @@
## Summary
<!-- Explain the problem and the smallest coherent change that addresses it. -->
## Related issue
<!-- Use "Fixes #123" when applicable. -->
## Contract and data impact
- [ ] No public configuration, schema, CLI, endpoint, streaming, or workspace-layout contract changes
- [ ] No user-owned memory files are deleted or rewritten
- [ ] Derived indexes, catalogs, graphs, caches, and metadata remain rebuildable
<!-- If any item is unchecked, describe the impact and migration or recovery path. -->
## Validation
<!-- List the exact checks run and their results. Explain relevant checks that were not run. -->
- [ ] Focused tests pass
- [ ] Unit tests pass, or omitted tests are explained below
- [ ] `pre-commit run --all-files` passes, or omitted checks are explained below
- [ ] Frontend checks were run when `reme_studio/` changed
## Checklist
- [ ] I reviewed the diff for unrelated changes and sensitive data
- [ ] Tests cover intentional behavior changes
- [ ] Defaults, schemas, and concise documentation were updated together when required
- [ ] Long-lived clients, tasks, services, and executors follow the application lifecycle
## Screenshots or additional notes
<!-- Include UI screenshots, compatibility notes, or follow-up work when relevant. -->

35
.github/dependabot.yml vendored Normal file
View file

@ -0,0 +1,35 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
target-branch: "main"
schedule:
interval: "weekly"
day: "monday"
time: "09:30"
timezone: "Asia/Shanghai"
groups:
codeql:
patterns:
- "github/codeql-action/*"
open-pull-requests-limit: 5
commit-message:
prefix: "chore"
include: "scope"
- package-ecosystem: "pip"
directory: "/"
target-branch: "main"
schedule:
interval: "cron"
cronjob: "30 9 * * *"
timezone: "Asia/Shanghai"
allow:
- dependency-name: "agentscope"
cooldown:
exclude:
- "agentscope"
open-pull-requests-limit: 1
commit-message:
prefix: "chore"
include: "scope"

80
.github/workflows/_build-docs.yml vendored Normal file
View file

@ -0,0 +1,80 @@
name: _Build documentation
on:
workflow_call:
inputs:
run_tests:
description: Run the documentation test suite before building
required: false
default: true
type: boolean
upload_pages_artifact:
description: Upload the build for a later GitHub Pages deployment job
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
build:
name: Build documentation
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
working-directory: github-pages
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Set up Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '22.22.3'
cache: npm
cache-dependency-path: |
github-pages/package-lock.json
reme_studio/package-lock.json
- name: Install dependencies
run: npm ci
- name: Install Studio dependencies
run: npm ci --prefix ../reme_studio
- name: Check Studio types and lint
if: inputs.run_tests
working-directory: reme_studio
run: npm run format:check && npm run lint
- name: Test browser demo behavior
if: inputs.run_tests
working-directory: reme_studio
run: node --test tests/demo-workspace.test.mjs tests/wikilinks.test.mjs
- name: Run tests
if: inputs.run_tests
run: npm test
- name: Build documentation
run: npm run build
- name: Verify browser demo bundle
if: inputs.run_tests
working-directory: reme_studio
run: node --test tests/demo-build.test.mjs
- name: Configure Pages
if: inputs.upload_pages_artifact
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
- name: Upload Pages artifact
if: inputs.upload_pages_artifact
uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
with:
path: github-pages/dist

View file

@ -0,0 +1,89 @@
name: _Build Python packages
on:
workflow_call:
inputs:
expected_version:
description: Expected release version; omit for a consistency-only check
required: false
default: ''
type: string
upload_artifacts:
description: Upload distributions for later publish jobs
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
distributions:
name: Build Python distributions
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.11'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest twine
- name: Validate package versions
if: inputs.expected_version == ''
run: python scripts/bump_version.py --check
- name: Validate release version
if: inputs.expected_version != ''
env:
EXPECTED_VERSION: ${{ inputs.expected_version }}
run: python scripts/bump_version.py --check --expected-version "${EXPECTED_VERSION}"
- name: Run package tests
run: PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
- name: Build and check distributions
run: |
mkdir -p dist/reme
python -m build --outdir dist/reme
python -m twine check dist/reme/*
- name: Verify distributions and isolated installation
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme/web/')
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme_studio/')
python -m venv "${RUNNER_TEMP}/reme-package-smoke"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -m pip install "${REME_WHEEL}[as]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -c "import reme"
- name: Verify released core dependencies
if: inputs.expected_version != ''
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m venv "${RUNNER_TEMP}/reme-core-package-smoke"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" -m pip install "${REME_WHEEL}[core]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- name: Upload ReMe distributions
if: inputs.upload_artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reme-distributions
path: dist/reme/
if-no-files-found: error

View file

@ -0,0 +1,155 @@
name: _Release npm plugin
on:
workflow_call:
inputs:
directory:
description: Repository-relative package directory
required: true
type: string
package_name:
description: Exact public npm package name
required: true
type: string
artifact_name:
description: Prefix for the packed package artifact
required: true
type: string
version:
description: Exact package.json version; an optional v prefix is accepted
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
type: string
use_npm_token:
description: Use the npm environment NPM_TOKEN instead of Trusted Publishing
required: false
default: false
type: boolean
validate_clawhub:
description: Validate the package against the ClawHub contract
required: false
default: false
type: boolean
outputs:
version:
description: Normalized package version
value: ${{ jobs.build.outputs.version }}
secrets:
NPM_TOKEN:
description: Optional bootstrap or recovery token for npm publishing
required: false
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
version: ${{ steps.validate.outputs.version }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24.16.0"
cache: npm
cache-dependency-path: ${{ inputs.directory }}/package-lock.json
- name: Validate package identity and version
id: validate
working-directory: ${{ inputs.directory }}
env:
EXPECTED_NAME: ${{ inputs.package_name }}
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
run: |
node --input-type=module <<'JS'
import { appendFileSync, readFileSync } from 'node:fs';
const manifest = JSON.parse(readFileSync('package.json', 'utf8'));
const expected = process.env.RELEASE_VERSION.replace(/^v/, '');
if (manifest.name !== process.env.EXPECTED_NAME) {
throw new Error(`Expected ${process.env.EXPECTED_NAME}, found ${manifest.name}`);
}
if (manifest.version !== expected) throw new Error(`package.json is ${manifest.version}, workflow input is ${expected}`);
if (manifest.version.includes('-') !== (process.env.NPM_TAG === 'next')) {
throw new Error('Prereleases must use next; stable releases must use latest');
}
appendFileSync(process.env.GITHUB_OUTPUT, `version=${manifest.version}\n`);
JS
- run: npm ci
working-directory: ${{ inputs.directory }}
- name: Validate package
working-directory: ${{ inputs.directory }}
run: |
npm run format:check
npm run lint
npm run typecheck
npm test
npm run test:package
- name: Validate ClawHub contract
if: inputs.validate_clawhub
working-directory: ${{ inputs.directory }}
run: npx --yes clawhub@0.23.3 package validate . --json
- name: Pack
working-directory: ${{ inputs.directory }}
run: |
mkdir -p "$RUNNER_TEMP/plugin-package"
npm pack --pack-destination "$RUNNER_TEMP/plugin-package"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ inputs.artifact_name }}-${{ steps.validate.outputs.version }}
path: ${{ runner.temp }}/plugin-package/*.tgz
if-no-files-found: error
publish:
if: github.ref == 'refs/heads/main'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment: npm
permissions:
contents: read
id-token: write
steps:
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24"
registry-url: https://registry.npmjs.org
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ inputs.artifact_name }}-${{ needs.build.outputs.version }}
path: dist/plugin
- name: Reject an existing package version
env:
PACKAGE_NAME: ${{ inputs.package_name }}
PACKAGE_VERSION: ${{ needs.build.outputs.version }}
run: |
if npm view "${PACKAGE_NAME}@${PACKAGE_VERSION}" version >/dev/null 2>&1; then
echo "${PACKAGE_NAME}@${PACKAGE_VERSION} already exists" >&2
exit 1
fi
- name: Publish to npm with Trusted Publishing
if: ${{ !inputs.use_npm_token }}
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/plugin/*.tgz --access public --tag "$NPM_TAG" --provenance
- name: Publish to npm with NPM_TOKEN
if: inputs.use_npm_token
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
NPM_TAG: ${{ inputs.npm_tag }}
run: |
if [[ -z "${NODE_AUTH_TOKEN}" ]]; then
echo "NPM_TOKEN is required when use_npm_token is enabled" >&2
exit 1
fi
npm publish dist/plugin/*.tgz --access public --tag "$NPM_TAG" --provenance

40
.github/workflows/ci-docs.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: CI / Documentation
on:
pull_request:
branches: [main]
paths:
- '.github/workflows/ci-docs.yml'
- '.github/workflows/_build-docs.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'reme/config/default.yaml'
- 'integrations/claude_code/README.md'
- 'integrations/hermes_agent/README.md'
- 'integrations/hermes_agent/figures/**'
- 'reme_studio/**'
- 'integrations/dsh/README*.md'
- 'integrations/dsh/figures/**'
- 'integrations/openclaw/README*.md'
- 'integrations/openclaw/figures/**'
- 'plugins/*/README*.md'
- 'benchmark/*/README*.md'
- 'benchmark/toolmemory/gitcha.png'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
documentation:
name: Test and build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true

52
.github/workflows/ci-dsh-plugin.yml vendored Normal file
View file

@ -0,0 +1,52 @@
name: CI / DSH plugin
on:
push:
branches: [main]
paths:
- ".github/workflows/ci-dsh-plugin.yml"
- ".github/workflows/_release-npm-plugin.yml"
- ".github/workflows/release-dsh-plugin.yml"
- "integrations/dsh/**"
pull_request:
branches: [main]
paths:
- ".github/workflows/ci-dsh-plugin.yml"
- ".github/workflows/_release-npm-plugin.yml"
- ".github/workflows/release-dsh-plugin.yml"
- "integrations/dsh/**"
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
package:
name: Validate DSH plugin
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
working-directory: integrations/dsh
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24.16.0"
cache: npm
cache-dependency-path: integrations/dsh/package-lock.json
- run: npm ci
- run: npm run format:check
- run: npm run lint
- run: npm run typecheck
- run: npm test
- run: npm run test:package

View file

@ -0,0 +1,54 @@
name: CI / OpenClaw plugin
on:
push:
branches: [main]
paths:
- ".github/workflows/ci-openclaw-plugin.yml"
- ".github/workflows/_release-npm-plugin.yml"
- ".github/workflows/release-openclaw-plugin.yml"
- "integrations/openclaw/**"
pull_request:
branches: [main]
paths:
- ".github/workflows/ci-openclaw-plugin.yml"
- ".github/workflows/_release-npm-plugin.yml"
- ".github/workflows/release-openclaw-plugin.yml"
- "integrations/openclaw/**"
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
package:
name: Validate OpenClaw plugin
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
working-directory: integrations/openclaw
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24.16.0"
cache: npm
cache-dependency-path: integrations/openclaw/package-lock.json
- run: npm ci
- run: npm run format:check
- run: npm run lint
- run: npm run typecheck
- run: npm test
- run: npm run test:package
- name: Validate ClawHub contract
run: npx --yes clawhub@0.23.3 package validate . --json

40
.github/workflows/ci-packages.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: CI / Python packages
on:
push:
branches: [main]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
pull_request:
branches: [main]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
distributions:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml

65
.github/workflows/ci-python-quality.yml vendored Normal file
View file

@ -0,0 +1,65 @@
name: CI / Python quality
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
actionlint:
name: GitHub Actions
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Validate workflows with actionlint
env:
ACTIONLINT_VERSION: 1.7.12
ACTIONLINT_SHA256: 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
run: |
archive="actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz"
curl --fail --location --proto '=https' --retry 3 --silent --show-error \
--output "${RUNNER_TEMP}/${archive}" \
"https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/${archive}"
echo "${ACTIONLINT_SHA256} ${RUNNER_TEMP}/${archive}" | sha256sum --check
tar -xzf "${RUNNER_TEMP}/${archive}" -C "${RUNNER_TEMP}" actionlint
"${RUNNER_TEMP}/actionlint" .github/workflows/*.yml
pre-commit:
name: Pre-commit
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.11'
cache: pip
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e reme_studio -e ".[dev,core]"
pip install -q --no-deps -e plugins/auto-fin -e plugins/daily_paper -e plugins/lme -e plugins/beam
- name: Pre-commit starts
run: pre-commit run --all-files

View file

@ -1,30 +1,36 @@
name: Tests ReMe name: CI / Python tests
on: on:
push: push:
branches: [main, master, dev, develop] branches: [main]
pull_request: pull_request:
branches: [main, master, dev, develop] branches: [main]
workflow_dispatch: workflow_dispatch:
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
unit-tests: unit-tests:
name: Unit Tests - py${{ matrix.python-version }} name: Unit Tests - py${{ matrix.python-version }}
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 90
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
python-version: ["3.11", "3.12", "3.13"] python-version: ["3.11", "3.12", "3.13", "3.14"]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }} - name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5 uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
cache: 'pip' cache: 'pip'
@ -32,12 +38,15 @@ jobs:
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip setuptools wheel python -m pip install --upgrade pip setuptools wheel
pip install -e ".[dev,core]" pip install -e reme_studio -e ".[dev,core]"
pip install --no-deps -e plugins/auto-fin
pip install -e plugins/daily_paper
pip install -e plugins/lme -e plugins/beam
pip install coverage pip install coverage
- name: Run unit tests - name: Run unit tests
run: | run: |
coverage run -m pytest tests/unit \ coverage run -m pytest tests/unit plugins/auto-fin plugins/daily_paper plugins/lme plugins/beam \
-v \ -v \
--tb=long \ --tb=long \
-s \ -s \

93
.github/workflows/ci-reme-studio.yml vendored Normal file
View file

@ -0,0 +1,93 @@
name: CI / ReMe Studio
on:
push:
branches: [main]
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
pull_request:
branches: [main]
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
studio:
name: Studio checks
runs-on: ubuntu-latest
timeout-minutes: 30
defaults:
run:
working-directory: reme_studio
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run format check
run: npm run format:check
- name: Run lint
run: npm run lint
- name: Run tests
run: npm test
- name: Verify npm package
run: |
npm pack --pack-destination "${RUNNER_TEMP}"
tar -tzf "${RUNNER_TEMP}"/agentscope-ai-reme_studio-*.tgz | grep '^package/dist-static/index.html$'
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
- name: Build and verify Python package
working-directory: .
run: |
python -m pip install build packaging pytest twine
PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio
python -m twine check dist/studio/*
STUDIO_WHEEL="$(pwd)/$(ls dist/studio/reme_studio-*.whl)"
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY

View file

@ -1,30 +1,36 @@
name: Windows Smoke name: CI / Windows
on: on:
push: push:
branches: [main, master, dev, develop] branches: [main]
pull_request: pull_request:
branches: [main, master, dev, develop] branches: [main]
workflow_dispatch: workflow_dispatch:
concurrency: concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true cancel-in-progress: true
permissions:
contents: read
jobs: jobs:
cli-smoke: cli-smoke:
name: CLI smoke - py${{ matrix.python-version }} name: CLI smoke - py${{ matrix.python-version }}
runs-on: windows-latest runs-on: windows-latest
timeout-minutes: 30
strategy: strategy:
fail-fast: false fail-fast: false
matrix: matrix:
python-version: ["3.11"] python-version: ["3.11"]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }} - name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5 uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
cache: 'pip' cache: 'pip'
@ -32,15 +38,15 @@ jobs:
- name: Install package - name: Install package
run: | run: |
python -m pip install --upgrade pip setuptools wheel python -m pip install --upgrade pip setuptools wheel
pip install -e ".[dev,core]" pip install -e ".[dev,as]"
- name: Run version job - name: Run version job
run: reme start service.backend=cli job=version run: reme start config=tests/fixtures/config/version-smoke.yaml job=version
- name: Run Windows path tests - name: Run Windows path tests
run: | run: |
python -m pytest ` python -m pytest `
tests/unit/test_auto_dream.py::test_scan_day_files_includes_nested_md_and_excludes_interests ` tests/unit/test_auto_dream.py::test_scan_day_files_includes_only_markdown_day_files `
tests/unit/test_auto_dream.py::test_dream_extract_matches_posix_catalog_paths ` tests/unit/test_auto_dream.py::test_dream_extract_matches_posix_catalog_paths `
tests/unit/test_read_with_neighbors.py::test_read_with_neighbors_uses_posix_nested_path ` tests/unit/test_read_with_neighbors.py::test_read_with_neighbors_uses_posix_nested_path `
-v -v

60
.github/workflows/deploy-docs.yml vendored Normal file
View file

@ -0,0 +1,60 @@
name: Deploy / Documentation
on:
push:
branches: [main]
paths:
- "github-pages/**"
- "docs/**"
- "reme/config/default.yaml"
- "integrations/claude_code/README.md"
- "integrations/hermes_agent/README.md"
- "integrations/hermes_agent/figures/**"
- "README.md"
- "README_ZH.md"
- "reme_studio/**"
- "integrations/dsh/README*.md"
- "integrations/dsh/figures/**"
- "integrations/openclaw/README*.md"
- "integrations/openclaw/figures/**"
- "plugins/*/README*.md"
- "benchmark/*/README*.md"
- "benchmark/toolmemory/gitcha.png"
- "AGENTS.md"
- ".github/workflows/deploy-docs.yml"
- ".github/workflows/_build-docs.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
name: Build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true
upload_pages_artifact: true
permissions:
contents: read
pages: write
id-token: write
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
timeout-minutes: 10
permissions:
pages: write
id-token: write
steps:
- name: Deploy
id: deployment
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1

221
.github/workflows/docker.yml vendored Normal file
View file

@ -0,0 +1,221 @@
name: CI and Release / Docker
on:
pull_request:
branches: [main]
paths:
- '.github/workflows/docker.yml'
- 'Dockerfile'
- '.dockerignore'
- 'docker-compose.yml'
- 'deploy/docker/**'
- 'pyproject.toml'
- 'README.md'
- 'LICENSE'
- 'reme/**'
- 'reme_studio/**'
- 'scripts/package_studio.py'
- 'scripts/test_docker_image.py'
push:
branches: [main]
paths:
- '.github/workflows/docker.yml'
- 'Dockerfile'
- '.dockerignore'
- 'docker-compose.yml'
- 'deploy/docker/**'
- 'pyproject.toml'
- 'README.md'
- 'LICENSE'
- 'reme/**'
- 'reme_studio/**'
- 'scripts/package_studio.py'
- 'scripts/test_docker_image.py'
release:
types: [published]
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
PUBLISH_IMAGE: ${{ github.event_name == 'release' || (github.event_name != 'pull_request' && github.ref == 'refs/heads/main') }}
jobs:
build:
name: Build and test / ${{ matrix.arch }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: ubuntu-24.04
- arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.11'
- name: Validate package version
env:
RELEASE_VERSION: ${{ github.event.release.tag_name }}
run: |
python -m pip install packaging
if [ -n "$RELEASE_VERSION" ]; then
python scripts/bump_version.py --check --expected-version "$RELEASE_VERSION"
else
python scripts/bump_version.py --check
fi
- name: Normalize image name
id: image
env:
REPOSITORY: ${{ github.repository }}
run: echo "name=ghcr.io/${REPOSITORY,,}" >> "$GITHUB_OUTPUT"
- name: Validate Compose
run: docker compose config --quiet
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
id: metadata
with:
images: ${{ steps.image.outputs.name }}
flavor: latest=${{ github.event_name == 'release' && !github.event.release.prerelease && 'auto' || 'false' }}
tags: |
type=raw,value=main,enable=${{ github.ref == 'refs/heads/main' }}
type=pep440,pattern={{version}},value=${{ github.event.release.tag_name }},enable=${{ github.event_name == 'release' }}
type=sha
- name: Build local image
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
with:
context: .
platforms: linux/${{ matrix.arch }}
load: true
tags: reme:smoke
labels: ${{ steps.metadata.outputs.labels }}
cache-from: type=gha,scope=reme-${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=reme-${{ matrix.arch }}
- name: Test installed image and persistent workspace
run: python scripts/test_docker_image.py --image reme:smoke
- name: Log in to GHCR
if: env.PUBLISH_IMAGE == 'true'
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Publish tested architecture by digest
id: publish
if: env.PUBLISH_IMAGE == 'true'
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
with:
context: .
platforms: linux/${{ matrix.arch }}
outputs: type=image,name=${{ steps.image.outputs.name }},push-by-digest=true,name-canonical=true,push=true
labels: ${{ steps.metadata.outputs.labels }}
cache-from: type=gha,scope=reme-${{ matrix.arch }}
provenance: mode=max
sbom: true
- name: Record image digest
if: env.PUBLISH_IMAGE == 'true'
env:
IMAGE_DIGEST: ${{ steps.publish.outputs.digest }}
run: |
mkdir -p "$RUNNER_TEMP/digests"
touch "$RUNNER_TEMP/digests/${IMAGE_DIGEST#sha256:}"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: env.PUBLISH_IMAGE == 'true'
with:
name: docker-digest-${{ matrix.arch }}
path: ${{ runner.temp }}/digests/*
if-no-files-found: error
retention-days: 1
manifest:
name: Publish multi-platform tags
needs: build
if: github.event_name == 'release' || (github.event_name != 'pull_request' && github.ref == 'refs/heads/main')
runs-on: ubuntu-24.04
timeout-minutes: 15
permissions:
contents: read
packages: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: docker-digest-*
merge-multiple: true
path: ${{ runner.temp }}/digests
- name: Normalize image name
id: image
env:
REPOSITORY: ${{ github.repository }}
run: echo "name=ghcr.io/${REPOSITORY,,}" >> "$GITHUB_OUTPUT"
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
id: metadata
with:
images: ${{ steps.image.outputs.name }}
flavor: latest=${{ github.event_name == 'release' && !github.event.release.prerelease && 'auto' || 'false' }}
tags: |
type=raw,value=main,enable=${{ github.ref == 'refs/heads/main' }}
type=pep440,pattern={{version}},value=${{ github.event.release.tag_name }},enable=${{ github.event_name == 'release' }}
type=sha
- name: Assemble and verify tags
env:
IMAGE_NAME: ${{ steps.image.outputs.name }}
IMAGE_TAGS: ${{ steps.metadata.outputs.tags }}
run: |
set -euo pipefail
image_refs=()
for digest_file in "$RUNNER_TEMP"/digests/*; do
image_refs+=("${IMAGE_NAME}@sha256:$(basename "$digest_file")")
done
[ "${#image_refs[@]}" -eq 2 ]
tag_args=()
while IFS= read -r tag; do
tag_args+=(--tag "$tag")
done <<< "$IMAGE_TAGS"
docker buildx imagetools create "${tag_args[@]}" "${image_refs[@]}"
while IFS= read -r tag; do
manifest=$(docker buildx imagetools inspect "$tag" --raw)
IMAGE_MANIFEST="$manifest" python - <<'PY'
import json
import os
manifest = json.loads(os.environ["IMAGE_MANIFEST"])
platforms = {(item["platform"]["os"], item["platform"]["architecture"]) for item in manifest["manifests"]}
assert {("linux", "amd64"), ("linux", "arm64")} <= platforms, platforms
PY
done <<< "$IMAGE_TAGS"

View file

@ -1,16 +1,21 @@
name: PR Title Check name: Policy / PR title
on: on:
pull_request: pull_request:
branches: [main, master, dev, develop] branches: [main]
types: [opened, edited, synchronize, reopened] types: [opened, edited, reopened]
permissions:
contents: read
pull-requests: read
jobs: jobs:
check-pr-title: check-pr-title:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10
steps: steps:
- name: Check PR title format - name: Check PR title format
uses: amannn/action-semantic-pull-request@v6.1.1 uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with: with:

View file

@ -1,38 +0,0 @@
name: Pre-commit
on: [ push, pull_request ]
jobs:
run:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: True
matrix:
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
PYTHON: '3.11'
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e ".[dev,core]"
- name: Install pre-commit
run: |
pre-commit install
- name: Pre-commit starts
run: |
pre-commit run --all-files > pre-commit.log 2>&1 || true
cat pre-commit.log
if grep -q Failed pre-commit.log; then
echo -e "\e[41m [**FAIL**] Please install pre-commit and format your code first. \e[0m"
exit 1
fi
echo -e "\e[46m ********************************Passed******************************** \e[0m"

View file

@ -1,45 +0,0 @@
# This workflow will upload a Python Package using Twine when a release is created
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
name: Publish Python Package to Pypi
on:
workflow_dispatch:
release:
types: [published]
permissions:
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel build
- name: Build package
run: python -m build
- name: Test installation
run: |
WHEEL="$(ls dist/*.whl)"
pip install "${WHEEL}[core]"
python -c "import reme; print(reme.__version__)"
- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}

178
.github/workflows/release-auto-fin.yml vendored Normal file
View file

@ -0,0 +1,178 @@
# 发布操作手册:
# 1. 先将 plugins/auto-fin/pyproject.toml 中的 project.version 更新为待发布版本并合入目标分支。
# 2. 确认插件依赖的 reme-ai 版本已经发布到 PyPI;本工作流会在构建阶段验证该依赖可下载。
# 3. 确认 PyPI Trusted Publisher 已绑定本仓库、此工作流和 pypi environment,且 PyPI 上不存在相同版本。
# 4. 在 GitHub 仓库的 Actions 页面选择“Release / Auto Fin plugin”,点击“Run workflow”。
# 5. 输入与 project.version 完全一致的版本号(例如 0.1.0)后运行;版本也可以带 v 前缀。
#
# 推荐发布顺序:reme-ai -> reme-auto-fin -> QwenPaw 更新依赖并通过 plugins: [auto-fin] 启用。
# 当前仅支持 workflow_dispatch 手动触发,不会因 push、tag 或 release 自动发布。
name: Release / Auto Fin plugin
run-name: Publish reme-auto-fin ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/auto-fin/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-auto-fin
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 45
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install --no-deps -e plugins/auto-fin
- name: Validate package name and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/auto-fin/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-auto-fin":
raise SystemExit(f"Expected project name 'reme-auto-fin', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [requirement for requirement in project["dependencies"] if requirement.startswith("reme-ai")]
if len(requirements) != 1:
raise SystemExit(f"Expected one reme-ai dependency, found {requirements!r}")
reme_requirement = Requirement(requirements[0])
if reme_requirement.name != "reme-ai" or reme_requirement.extras:
raise SystemExit(f"Expected a base reme-ai dependency, found {requirements[0]!r}")
if Version("0.4.1.11") in reme_requirement.specifier or Version("0.4.1.12") not in reme_requirement.specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.12, found {requirements[0]!r}")
root_project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
agentscope_requirements = [
Requirement(value) for value in root_project["optional-dependencies"]["as"]
]
if len(agentscope_requirements) != 1 or agentscope_requirements[0].name != "agentscope":
raise SystemExit(f"Expected one agentscope dependency, found {agentscope_requirements!r}")
agentscope_requirement = agentscope_requirements[0]
agentscope_specifiers = list(agentscope_requirement.specifier)
if (
agentscope_requirement.extras != {"model-ollama"}
or agentscope_requirement.marker is not None
or len(agentscope_specifiers) != 1
or agentscope_specifiers[0].operator != "=="
or Version(agentscope_specifiers[0].version).is_prerelease
):
raise SystemExit(f"Expected a stable exact agentscope[model-ollama] pin, found {agentscope_requirement}")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirement}", file=output)
print(f"agentscope_requirement={agentscope_requirement}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Auto Fin tests
run: python -m pytest plugins/auto-fin -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-auto-fin-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/auto-fin
python -m build plugins/auto-fin --outdir dist/auto-fin
python -m twine check dist/auto-fin/*
- name: Verify distributions and isolated installation
env:
AGENTSCOPE_REQUIREMENT: ${{ steps.package.outputs.agentscope_requirement }}
run: |
AUTO_FIN_WHEEL="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.whl)"
AUTO_FIN_SDIST="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.tar.gz)"
python -m zipfile -l "${AUTO_FIN_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${AUTO_FIN_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-auto-fin-smoke"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" -m pip install \
"${AGENTSCOPE_REQUIREMENT}" "${AUTO_FIN_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-auto-fin")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["auto-fin"].value == "reme_auto_fin"
manifest = load_package_manifest("reme_auto_fin", plugin_name="auto-fin")
assert set(manifest.backends) == {
"auto_fin_data_step",
"auto_fin_topic_step",
"auto_fin_merge_step",
}
assert set(manifest.application_defaults["jobs"]) == {
"auto_fin",
"auto_fin_cron",
}
PY
- name: Upload distributions
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin
- name: Publish reme-auto-fin
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/auto-fin

View file

@ -0,0 +1,178 @@
# Release checklist:
# 1. Update project.version in plugins/daily_paper/pyproject.toml and merge it into the target branch.
# 2. Publish the required reme-ai version before this plugin; the build verifies that dependency on PyPI.
# 3. Configure PyPI Trusted Publishing for this repository/workflow and its pypi environment.
# 4. Run "Release / Daily Paper plugin" from GitHub Actions with the exact project version (a v prefix is accepted).
#
# Recommended order: reme-ai -> reme-daily-paper -> downstream applications enabling plugins: [daily-paper].
# This workflow is intentionally manual and never publishes from a push, tag, or GitHub release event.
name: Release / Daily Paper plugin
run-name: Publish reme-daily-paper ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/daily_paper/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-daily-paper
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 45
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install -e plugins/daily_paper
- name: Validate package name, dependencies, and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/daily_paper/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-daily-paper":
raise SystemExit(f"Expected project name 'reme-daily-paper', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [Requirement(value) for value in project["dependencies"]]
reme_requirements = [requirement for requirement in requirements if requirement.name == "reme-ai"]
if len(reme_requirements) != 1 or reme_requirements[0].extras:
raise SystemExit(f"Expected one base reme-ai dependency, found {reme_requirements!r}")
if Version("0.4.1.11") in reme_requirements[0].specifier or Version("0.4.1.12") not in reme_requirements[0].specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.12, found {reme_requirements!r}")
if sum(requirement.name == "pypdf" for requirement in requirements) != 1:
raise SystemExit("Expected exactly one pypdf dependency")
root_project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
agentscope_requirements = [
Requirement(value) for value in root_project["optional-dependencies"]["as"]
]
if len(agentscope_requirements) != 1 or agentscope_requirements[0].name != "agentscope":
raise SystemExit(f"Expected one agentscope dependency, found {agentscope_requirements!r}")
agentscope_requirement = agentscope_requirements[0]
agentscope_specifiers = list(agentscope_requirement.specifier)
if (
agentscope_requirement.extras != {"model-ollama"}
or agentscope_requirement.marker is not None
or len(agentscope_specifiers) != 1
or agentscope_specifiers[0].operator != "=="
or Version(agentscope_specifiers[0].version).is_prerelease
):
raise SystemExit(f"Expected a stable exact agentscope[model-ollama] pin, found {agentscope_requirement}")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirements[0]}", file=output)
print(f"agentscope_requirement={agentscope_requirement}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Daily Paper tests
run: python -m pytest plugins/daily_paper -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-daily-paper-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/daily-paper
python -m build plugins/daily_paper --outdir dist/daily-paper
python -m twine check dist/daily-paper/*
- name: Verify distributions and isolated installation
env:
AGENTSCOPE_REQUIREMENT: ${{ steps.package.outputs.agentscope_requirement }}
run: |
DAILY_PAPER_WHEEL="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.whl)"
DAILY_PAPER_SDIST="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.tar.gz)"
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/plugin.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/analyze.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${DAILY_PAPER_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-daily-paper-smoke"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" -m pip install \
"${AGENTSCOPE_REQUIREMENT}" "${DAILY_PAPER_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-daily-paper")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["daily-paper"].value == "reme_daily_paper"
manifest = load_package_manifest("reme_daily_paper", plugin_name="daily-paper")
assert set(manifest.backends) == {
"daily_paper_collect_step",
"daily_paper_rank_step",
"daily_paper_select_step",
"daily_paper_analyze_step",
"daily_paper_digest_step",
}
assert set(manifest.application_defaults["jobs"]) == {"daily_paper", "daily_paper_cron"}
PY
- name: Upload distributions
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper
- name: Publish reme-daily-paper
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/daily-paper

View file

@ -0,0 +1,49 @@
# Configure npm Trusted Publishing for this caller filename and the npm environment.
name: Release / DSH plugin
run-name: Publish ReMe DSH plugin ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Exact package.json version; an optional v prefix is accepted
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
use_npm_token:
description: Use the npm environment NPM_TOKEN instead of Trusted Publishing
required: true
default: false
type: boolean
permissions:
contents: read
concurrency:
group: publish-reme-dsh-plugin
cancel-in-progress: false
jobs:
release:
if: github.ref == 'refs/heads/main'
permissions:
contents: read
id-token: write
uses: ./.github/workflows/_release-npm-plugin.yml
secrets:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
with:
directory: integrations/dsh
package_name: '@agentscope-ai/reme-dsh-plugin'
artifact_name: agentscope-ai-reme-dsh-plugin
version: ${{ inputs.version }}
npm_tag: ${{ inputs.npm_tag }}
use_npm_token: ${{ inputs.use_npm_token }}

View file

@ -0,0 +1,75 @@
# Configure npm Trusted Publishing for this caller filename and the npm environment.
name: Release / OpenClaw plugin
run-name: Publish ReMe OpenClaw plugin ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Exact package.json version; an optional v prefix is accepted
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
use_npm_token:
description: Use the npm environment NPM_TOKEN instead of Trusted Publishing
required: true
default: false
type: boolean
publish_clawhub:
description: Also publish the package to ClawHub
required: true
default: false
type: boolean
permissions:
contents: read
concurrency:
group: publish-reme-openclaw-plugin
cancel-in-progress: false
jobs:
release:
if: github.ref == 'refs/heads/main'
permissions:
contents: read
id-token: write
uses: ./.github/workflows/_release-npm-plugin.yml
secrets:
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
with:
directory: integrations/openclaw
package_name: '@agentscope-ai/reme-openclaw-plugin'
artifact_name: agentscope-ai-reme-openclaw-plugin
version: ${{ inputs.version }}
npm_tag: ${{ inputs.npm_tag }}
use_npm_token: ${{ inputs.use_npm_token }}
validate_clawhub: true
publish-clawhub:
if: ${{ inputs.publish_clawhub && github.ref == 'refs/heads/main' }}
needs: release
permissions:
actions: read
contents: read
id-token: write
uses: openclaw/clawhub/.github/workflows/package-publish.yml@cacf5ec1b0ee3cb532ab4555a68c1db9e0c1aac7 # v0.24.0
with:
family: code-plugin
version: ${{ needs.release.outputs.version }}
tags: ${{ inputs.npm_tag }}
source_repo: ${{ github.repository }}
source_commit: ${{ github.sha }}
source_ref: ${{ github.sha }}
source_path: integrations/openclaw
package_artifact_name: agentscope-ai-reme-openclaw-plugin-${{ needs.release.outputs.version }}
dry_run: false
wait_for_publication: true

52
.github/workflows/release-python.yml vendored Normal file
View file

@ -0,0 +1,52 @@
name: Release / ReMe Python package
# Publishing a GitHub Release is the primary release trigger for reme-ai; workflow_dispatch is the recovery path.
# Configure a PyPI Trusted Publisher for this repository, workflow, and its pypi environment first.
run-name: Publish reme-ai ${{ github.event.release.tag_name || inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Exact reme-ai version; an optional v prefix is accepted
required: true
type: string
release:
types: [published]
permissions:
contents: read
concurrency:
group: publish-reme-ai
cancel-in-progress: false
jobs:
build:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml
with:
expected_version: ${{ github.event.release.tag_name || inputs.version }}
upload_artifacts: true
publish-reme:
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download ReMe distributions
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reme-distributions
path: dist/reme
- name: Publish ReMe
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/reme
skip-existing: true

View file

@ -0,0 +1,243 @@
# Release checklist:
# 1. Update reme_studio/pyproject.toml, package.json, and package-lock.json to the same Studio version.
# 2. Configure npm Trusted Publishing with the npm environment and PyPI Trusted Publishing with the pypi environment.
# 3. Run this workflow manually with the exact Studio version.
name: Release / ReMe Studio
run-name: Publish ReMe Studio ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Version from the Studio Python and npm manifests
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
use_npm_token:
description: Use the npm environment NPM_TOKEN instead of Trusted Publishing
required: true
default: false
type: boolean
publish_target:
description: Packages to publish; single-package modes are for release recovery
required: true
default: both
type: choice
options:
- both
- pypi
- npm
permissions:
contents: read
concurrency:
group: publish-reme-studio
cancel-in-progress: false
jobs:
build:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
env:
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
- name: Validate Studio package names and version
run: |
python - <<'PY'
import json
import os
import tomllib
from pathlib import Path
studio = Path("reme_studio")
python_manifest = tomllib.loads((studio / "pyproject.toml").read_text(encoding="utf-8"))["project"]
npm_manifest = json.loads((studio / "package.json").read_text(encoding="utf-8"))
expected = os.environ["RELEASE_VERSION"].removeprefix("v")
if python_manifest["name"] != "reme_studio":
raise SystemExit(f"Unexpected Python package name: {python_manifest['name']}")
if npm_manifest["name"] != "@agentscope-ai/reme_studio":
raise SystemExit(f"Unexpected npm package name: {npm_manifest['name']}")
if python_manifest["version"] != expected or npm_manifest["version"] != expected:
raise SystemExit(
f"Studio manifests are {python_manifest['version']} and {npm_manifest['version']}; "
f"workflow input is {expected}",
)
prerelease = "-" in expected
if prerelease != (os.environ["NPM_TAG"] == "next"):
raise SystemExit("Prereleases must use next; stable releases must use latest")
PY
- name: Install dependencies and run checks
working-directory: reme_studio
run: |
npm ci
npm run format:check
npm run lint
npm test
- name: Build Studio distributions
run: |
python -m pip install build twine
mkdir -p dist/studio-python dist/studio-npm
npm pack ./reme_studio --pack-destination dist/studio-npm
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio-python
python -m twine check dist/studio-python/*
- name: Verify Studio distributions and isolated installation
run: |
STUDIO_WHEEL="$(pwd)/$(ls dist/studio-python/reme_studio-*.whl)"
tar -tzf dist/studio-npm/*.tgz | grep '^package/dist-static/index.html$'
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reme-studio-${{ inputs.version }}
path: |
dist/studio-python/*
dist/studio-npm/*
if-no-files-found: error
publish-python:
if: inputs.publish_target != 'npm'
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
environment: pypi
permissions:
contents: read
id-token: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Publish ReMe Studio to PyPI
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/studio-python
skip-existing: true
publish-npm:
if: >-
!cancelled() &&
needs.build.result == 'success' &&
(inputs.publish_target == 'npm' ||
(inputs.publish_target == 'both' && needs.publish-python.result == 'success'))
needs: [build, publish-python]
runs-on: ubuntu-latest
timeout-minutes: 10
environment: npm
permissions:
contents: read
id-token: write
steps:
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "24"
registry-url: https://registry.npmjs.org
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Verify the matching PyPI release for npm-only recovery
if: inputs.publish_target == 'npm'
env:
PACKAGE_VERSION: ${{ inputs.version }}
run: |
python - <<'PY'
import os
import urllib.error
import urllib.request
version = os.environ["PACKAGE_VERSION"].removeprefix("v")
url = f"https://pypi.org/pypi/reme-studio/{version}/json"
try:
with urllib.request.urlopen(url, timeout=30) as response:
if response.status != 200:
raise SystemExit(f"Unexpected PyPI response for reme-studio {version}: {response.status}")
except urllib.error.HTTPError as exc:
raise SystemExit(f"reme-studio {version} must exist on PyPI before npm-only recovery") from exc
PY
- name: Check for an identical existing npm package
id: npm-version
env:
PACKAGE_VERSION: ${{ inputs.version }}
run: |
package_file=$(find dist/studio-npm -maxdepth 1 -name '*.tgz' -print -quit)
if [[ -z "${package_file}" ]]; then
echo "Studio npm artifact is missing" >&2
exit 1
fi
local_integrity=$(node --input-type=module - "${package_file}" <<'JS'
import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';
const digest = createHash('sha512').update(readFileSync(process.argv[2])).digest('base64');
console.log(`sha512-${digest}`);
JS
)
if remote_integrity=$(npm view "@agentscope-ai/reme_studio@${PACKAGE_VERSION#v}" dist.integrity 2>/dev/null); then
if [[ "${remote_integrity}" != "${local_integrity}" ]]; then
echo "Existing npm package has different contents" >&2
exit 1
fi
echo "exists=true" >> "${GITHUB_OUTPUT}"
echo "The identical npm package already exists; nothing to publish"
else
echo "exists=false" >> "${GITHUB_OUTPUT}"
fi
- name: Publish ReMe Studio to npm with Trusted Publishing
if: ${{ steps.npm-version.outputs.exists != 'true' && !inputs.use_npm_token }}
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/studio-npm/*.tgz --access public --tag "${NPM_TAG}" --provenance
- name: Publish ReMe Studio to npm with NPM_TOKEN
if: ${{ steps.npm-version.outputs.exists != 'true' && inputs.use_npm_token }}
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
NPM_TAG: ${{ inputs.npm_tag }}
run: |
if [[ -z "${NODE_AUTH_TOKEN}" ]]; then
echo "NPM_TOKEN is required when use_npm_token is enabled" >&2
exit 1
fi
npm publish dist/studio-npm/*.tgz --access public --tag "${NPM_TAG}" --provenance

45
.github/workflows/security-codeql.yml vendored Normal file
View file

@ -0,0 +1,45 @@
name: Security / CodeQL
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 1 * * 1'
workflow_dispatch:
permissions:
contents: read
security-events: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
analyze:
name: Analyze ${{ matrix.language }}
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4
with:
languages: ${{ matrix.language }}
build-mode: none
- name: Perform CodeQL analysis
uses: github/codeql-action/analyze@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4
with:
category: /language:${{ matrix.language }}

7
.gitignore vendored
View file

@ -30,7 +30,9 @@ htmlcov/
# Packaging / build outputs # Packaging / build outputs
build/ build/
dist/ dist/
node_modules/
*.egg-info/ *.egg-info/
integrations/*/reports/
# Logs / temporary files # Logs / temporary files
*.log *.log
@ -46,6 +48,7 @@ temp*/
# ReMe runtime data # ReMe runtime data
.reme/ .reme/
reme_workspace/ reme_workspace/
reme_workspace_auto_fin_real_test*/
vault/ vault/
*.db *.db
*.sqlite *.sqlite
@ -56,6 +59,10 @@ docs/_build/
site/ site/
evaluation/ evaluation/
# The pi-Bench suite ships its own trace-history render config, which must
# stay in git even though it lives under an evaluation/ directory.
!benchmark/pibench/config/bench/evaluation/
!benchmark/pibench/config/bench/evaluation/**
datasets/ datasets/
# Claude Code skills (local only) # Claude Code skills (local only)

275
AGENTS.md
View file

@ -1,20 +1,19 @@
# AGENTS.md # AGENTS.md
This file guides coding agents working in the ReMe repository. Keep changes small, This file guides coding agents working in the ReMe repository. Keep changes small, testable, and consistent with the
testable, and consistent with the contracts already expressed by the code. contracts expressed by the current code.
## Project Principles ## Project Principles
ReMe is a local-first, file-native memory system for agents. ReMe is a local-first, file-native memory system for agents.
- User-owned memory files are the source of truth. - User-owned workspace files are the durable source of truth.
- Indexes, caches, metadata, and generated state must be rebuildable. - Indexes, catalogs, graphs, caches, and generated metadata must remain rebuildable.
- Prefer transparent formats and behavior over hidden state. - Prefer transparent formats and predictable behavior over hidden state.
- Preserve user control over storage, configuration, and service boundaries. - Preserve user control over workspace paths, configuration, and service boundaries.
- Keep concepts focused on project intent; let code and schemas describe implementation. - Keep concepts focused on project intent; let code and schemas describe implementation.
When a proposed convenience conflicts with these principles, favor data ownership, When convenience conflicts with these principles, favor data ownership, recoverability, and explicit behavior.
recoverability, and predictable behavior.
## Sources of Truth ## Sources of Truth
@ -22,166 +21,198 @@ Use this order when documentation and implementation disagree:
1. Current code and public Pydantic schemas. 1. Current code and public Pydantic schemas.
2. Tests that describe supported behavior. 2. Tests that describe supported behavior.
3. CLI help and the built-in configuration. 3. CLI behavior and the built-in configuration.
4. Development documentation and historical notes. 4. README files and other development documentation.
Do not copy large implementation descriptions into documentation. Link to the relevant Do not duplicate large implementation descriptions in documentation. Express the stable contract and link to the
module or express the stable contract instead. If behavior changes intentionally, update relevant module where useful. When behavior changes intentionally, update the implementation, schemas, tests, defaults,
the code, schema, tests, configuration, and concise documentation together as needed. and concise documentation together.
## Repository Map ## Repository Map
- `reme/reme.py`: CLI entry point and client/server dispatch. - `reme/reme.py`: CLI entry point; dispatches `start`, `find_reme`, and client calls.
- `reme/application.py`: application assembly, dependency ordering, and lifecycle. - `reme/application.py`: application assembly, dependency ordering, job execution, and lifecycle.
- `reme/components/application_context.py`: application-wide wiring and shared in-memory metadata. - `reme/config/config_parser.py`: YAML/JSON loading, environment expansion, dot-notation parsing, and deep config
- `reme/components/runtime_context.py`: scratch state shared by steps within one execution. merging.
- `reme/config/default.yaml`: built-in jobs, components, and defaults. - `reme/config/default.yaml`: default service, jobs, steps, and components. Other files in
- `reme/schema/`: public and runtime Pydantic contracts. `reme/config/` are named configuration variants.
- `reme/components/`: services, stores, clients, jobs, and component registration. - `reme/schema/application_config.py`: typed application, component, and job configuration.
- `reme/steps/`: executable job steps. - `reme/schema/`: request, response, streaming, memory, graph, and file contracts.
- `tests/unit/`: primary fast validation suite. - `reme/components/application_context.py`: application-wide wiring and in-memory shared state.
- `tests/integration/`: tests that may require real credentials or services. - `reme/components/runtime_context.py`: request-scoped data, response, streaming queue, and stop event.
- `tests/vector/` and `tests/light/`: specialized suites. - `reme/components/base_component.py`: component lifecycle, dependency binding, and workspace helpers.
- `plugins/reme/`: Claude Code integration. - `reme/components/component_registry.py`: the frozen built-in registry template and application-local registry factory.
- `skills/reme_memory/`: skill that communicates with the ReMe service. - `reme/components/job/`: base, stream, background, and cron job implementations.
- `skills/qwenpaw_memory/`: separate direct-file memory convention; it does not call ReMe. - `reme/components/service/`: local CLI, HTTP, and MCP service backends.
- `docs/`: pages and assets that support the repository README; not the deployed docs site. - `reme/components/`: agent wrappers, model adapters, stores, catalogs, graphs, indexes, clients, tokenizers, and
outbound proxies.
- `reme/steps/`: registered job steps grouped by common, file I/O, index, evolve, cookbook, and transfer
concerns.
- `reme/utils/`: shared utilities, including service discovery, logging, web-static resolution, session I/O, token
accounting, and wikilink handling.
- `tests/unit/`: primary fast, isolated validation suite.
- `tests/integration/`: service/model tests that may need credentials or external processes.
- `reme_studio/`: ReMe Studio frontend source plus the independently published `reme_studio` Python package and
`@agentscope-ai/reme_studio` npm static distribution.
- `plugins/`: installable ReMe extensions, including Auto Fin and LME/BEAM plugins.
- `integrations/`: adapters that connect ReMe to external agent hosts, including the independent, self-contained DSH
and OpenClaw TypeScript plugins plus the Claude Code and Hermes Agent integrations.
- `skills/`: standalone skills; `reme_memory` calls ReMe, while other skills may use separate tools or direct-file
conventions.
- `benchmark/` and `cookbook/`: runnable evaluations and example workflows.
- `docs/`: README-linked supporting pages and figures.
- `github-pages/`: VitePress build shell, generated-content assembly, documentation checks, and GitHub Pages output. The
canonical theme and guides remain under `docs/`; `.generated/` and `dist/` are disposable.
## Development Setup ## Development Setup
ReMe requires Python 3.11 or newer. ReMe requires Python 3.11 or newer. Install the editable development environment with:
```bash ```bash
pip install -e ".[dev,core]" pip install -e reme_studio -e ".[dev,core]"
``` ```
Before changing behavior, inspect the adjacent implementation, schemas, configuration, Before changing behavior, inspect the adjacent implementation, schema, built-in config, and focused tests. Follow
and focused tests. Follow existing patterns unless the task explicitly calls for a new existing async and typing patterns unless the task explicitly requires a new contract.
contract or architecture.
## Change Workflow ## Configuration and CLI Contracts
1. Identify the narrowest supported contract affected by the request. - CLI syntax is `reme ACTION key=value ...`; leading `-` or `--` on arguments is accepted.
2. Read the relevant implementation and tests before editing. - Nested overrides use dot notation. Values support null, booleans, numbers, JSON collections, and quoted JSON strings;
3. Make the smallest coherent change; avoid unrelated cleanup. leading-zero numeric-looking values remain strings.
4. Update related schemas, defaults, registrations, and imports when required. - `config=<name-or-path>` loads a discovered config name or a `.yaml`, `.yml`, or `.json` file. With no explicit config
5. Add or adjust focused tests for observable behavior. path, `default` is loaded when available.
6. Run proportionate validation and report anything not run. - Config files expand `${VAR}` and `${VAR:-default}` recursively. An undefined variable without a default is an error.
- CLI/config overrides are deep-merged over the loaded file. Do not silently change this merge behavior or stable
configuration keys.
- `ApplicationConfig` normalizes `workspace_dir` to an expanded absolute path. `session_dir`
must remain workspace-relative; standard transcripts live under `{session_dir}/dialog`.
- `reme start` runs the configured service. `reme start job=<name> ...` switches to the one-shot CLI service and runs
the job through the normal application lifecycle.
- Other actions use a client selected from the running service configuration when discoverable, otherwise from local
config. Client-selection arguments must not leak into the job payload.
Component and step discovery depends on registration imports: ## Registration and Application Lifecycle
- Components use `R.register(...)` in `reme/components/component_registry.py`. Component and Step discovery is import-driven:
- Component packages must be reachable through `reme/components/__init__.py`.
- Step modules must be reachable through `reme/steps/__init__.py`.
Adding an implementation without its registration import can leave it undiscoverable at - Implementations declare a non-`BASE` `component_type` and register with `@R.register("backend")`
runtime. Treat the implementation, registry entry, and import side effect as one change. or `R.register(Class, "backend")`.
- Component packages must be imported through `reme/components/__init__.py`.
- Step packages/modules must be reachable through their package `__init__.py` chain and ultimately
`reme/steps/__init__.py`.
- Adding an implementation without its registration import leaves it undiscoverable at runtime. Treat implementation,
registration, import side effect, defaults, and tests as one change.
Do not silently change stable CLI flags, configuration keys, workspace layouts, serialized `Application` validates config through `ApplicationContext`, creates workspace directories, instantiates the service,
schemas, or service interfaces. When such a change is required, preserve compatibility configured components, and jobs, and then manages lifecycle as follows:
where practical and make the migration explicit.
## Step State Model - Components start in topological dependency order. Missing required dependencies and cycles fail explicitly; optional
dependencies may resolve to `None`.
- Jobs start after components in this order: base jobs, stream jobs, background jobs, then cron jobs.
- Shutdown closes everything in reverse start order and then shuts down the optional thread pool.
- If startup fails, already-started resources are closed.
- `BaseComponent.start()` and `close()` are lock-protected and idempotent. Dependencies created by a standalone
`default_factory` are owned and closed by the parent component.
Treat every Step as stateless. `BaseJob` stores Step specifications and builds fresh Step Keep async clients, tasks, executors, and services under this lifecycle. Do not introduce an untracked long-lived
instances for each Job invocation. A Step instance must not use `self` or class variables to resource.
retain mutable runtime state between calls.
Place state according to its lifetime: ## Jobs, Steps, and State
- Constructor fields on `self`: immutable Step configuration and resolved dependencies only. `BaseJob` resolves configured Step classes during job startup and constructs fresh Step instances for every invocation.
- `self.context` (`RuntimeContext`): request data and intermediate results for one Job Job-level kwargs are merged into each `RuntimeContext`, with call-time kwargs taking precedence. Sequential Steps in one
execution; sequential Steps share this context. invocation share the same `RuntimeContext` and `Response`.
- `self.app_context.metadata`: in-memory state that must be shared across Step or Job
invocations for the lifetime of the Application.
- Workspace files or a dedicated Component/store: durable state that must survive an
Application restart.
Use narrow, namespaced keys in `app_context.metadata`, following existing patterns such as Treat Step instances as invocation-scoped:
`tool_contexts`. The ApplicationContext is shared, so account for
concurrent access when values are mutable. New Step code must not fall back to `self.kwargs`
or another Step field to emulate shared state when `app_context` is absent; tests of shared
state should construct an `ApplicationContext`. If shared state grows into a stable
service-level contract or needs its own lifecycle, locking, or persistence, promote it to a
typed ApplicationContext field or a dedicated Component instead of expanding an ad hoc
metadata bucket.
Do not use `Response.metadata` as a state store. It is request-scoped output for callers and - Constructor fields and `self.kwargs` hold Step configuration and resolved dependencies. They may be cached or adjusted
diagnostics, distinct from `ApplicationContext.metadata`. during that one invocation, but must not be relied on across Job calls.
- `self.context.data` holds request inputs and intermediate values shared by sequential Steps.
- `self.context.response.answer`, `success`, and `metadata` are request-scoped output. Because the same response travels
through the Step chain, later Steps may consume metadata produced earlier, but it is not application-lifetime or
durable storage.
- `self.app_context.metadata` holds in-memory state shared across Job/Step invocations for the life of one
`Application`, such as counters, tool-context state, session maps, or locks.
- Workspace files or a dedicated Component/store hold durable state that must survive restart.
Use narrow, namespaced keys in `app_context.metadata` and protect shared mutable values against concurrent access. The
search/draft helpers intentionally mirror tool-context state into
`self.kwargs` only when no `ApplicationContext` exists for standalone use and unit tests; do not generalize that
compatibility fallback into persistent runtime state. If shared state becomes a stable service contract or needs
dedicated lifecycle, locking, or persistence, promote it to a typed context field or Component.
Additional Step contracts:
- `Ref` dependencies resolve in this order: Step kwargs, current `RuntimeContext`, then the named application component.
The value is cached only on the current Step instance and cleared before each call.
- `input_mapping` and `output_mapping` copy keys within `RuntimeContext.data`; missing sources are ignored.
- Dispatched Steps receive the current `RuntimeContext`, so their data and response are shared.
- Base jobs convert uncaught Step errors into `Response(success=False)`; stream jobs emit an error chunk and always a
terminal `DONE`; background jobs let errors reach their supervisor.
- Background jobs are never service-exposed. MCP also skips stream jobs. Respect `enable_serve`
and any configured service job allowlist.
## Workspace and File Safety
- Application startup creates the workspace plus configured metadata, session, memory-session, resource, daily, and
digest directories.
- File-operation paths are resolved against the workspace and must stay inside it. Home-relative paths are unsupported,
traversal escapes are rejected, and `_allowed_paths` restrictions fail closed when invalid.
- Preserve per-path locking, encoding detection, byte limits, truncation behavior, and optimistic
`expected_mtime` checks when modifying file operations.
- Do not bypass the existing file steps or stores in a way that weakens workspace containment.
- Never write test state into the repository's `.reme/`; use `tmp_path` or another isolated workspace.
- Do not delete or rewrite user memory to repair an index or make a test pass. Rebuild derived state from source files
instead.
## Validation ## Validation
Use the narrowest useful check while iterating, then broaden it according to risk. Use the narrowest useful check while iterating, then broaden it according to risk.
Run a focused test: Focused test:
```bash ```bash
pytest tests/unit/path/to/test_file.py -v pytest tests/unit/path/to/test_file.py -v
``` ```
Run the main unit suite: Main unit suite:
```bash ```bash
pytest tests/unit -v --tb=long -s --log-cli-level=WARNING pytest tests/unit -v --tb=long -s --log-cli-level=WARNING
``` ```
Run repository formatting and lint checks when the change warrants it: Repository formatting and lint checks:
```bash ```bash
pre-commit run --all-files pre-commit run --all-files
``` ```
Formatting and lint configuration is authoritative. Python code currently uses a maximum Black and Flake8 use a 120-character line limit and Python 3.11 formatting; Pylint is also run by pre-commit. If
line length of 120 for Black and Flake8, with Pylint also run by pre-commit. `reme_studio/` changes, use its Node 22.13+ scripts and run the proportionate checks from that directory, such as
`npm run format:check`, `npm run lint`, or `npm test`.
Integration tests may contact real services and require credentials such as Integration tests may contact real model providers, services, or agent subprocesses and can require credentials. Do not
`LLM_API_KEY` or `EMBEDDING_API_KEY`. Do not run credentialed or externally mutating tests run credentialed or externally mutating tests automatically; run them only when the task requires them and the necessary
automatically. Run them only when the task requires them and the user has supplied or environment has been supplied or authorized. Mock network, model, and subprocess boundaries in unit tests.
authorized the necessary environment.
## Coding and Test Conventions If documentation or the documentation theme changes, run `npm test` and `npm run build` from `github-pages/`. The Job
reference is generated from `reme/config/default.yaml`; do not edit generated pages directly.
- Target Python 3.11+ and follow the surrounding typing and async style. ## Change Guardrails
- Steps are stateless. If a step needs to persist state, store it in
`self.app_context.metadata` rather than on the step instance.
- Keep public schemas explicit and backward-compatible where practical.
- Close async clients, services, tasks, and other lifecycle resources deterministically.
- Prefer clear failures over silently falling back to corrupt or ambiguous state.
- Keep indexes and caches derivable from user-owned source files.
- Use `tmp_path` or another isolated temporary workspace in tests.
- Never write test state into the repository's `.reme/` directory.
- Mock network or model boundaries in unit tests.
- Do not commit `.env` files, credentials, runtime memory, logs, indexes, or caches.
## Documentation Boundaries
ReMe's local docs and the deployed documentation site have separate responsibilities.
- Keep `docs/` focused on content and assets used by `README.md` and `README_ZH.md`.
- Preserve README-linked pages under `docs/en/` and `docs/zh/`, including their relative
paths, unless the README is updated in the same change.
- Keep README-required images under `docs/figure/`.
- Keep the README's main documentation index pointed at `docs.agentscope.io` or the
`agentscope-ai/docs` repository, following the existing link style.
- Do not treat local README-supporting pages as the source for the deployed website.
The separate `agentscope-ai/docs` repository owns website content, navigation, versioning,
and deployment. Public ReMe pages live there under `reme/<version>/`. Make website changes
in that repository and follow its existing version-management conventions.
Do not add website build configuration or deployment workflows to ReMe unless the task
explicitly changes this repository boundary.
## Agent Guardrails
- Preserve unrelated user changes in a dirty working tree. - Preserve unrelated user changes in a dirty working tree.
- Do not edit generated output when the source can be changed instead. - Make the smallest coherent change and avoid unrelated cleanup or broad refactors.
- Do not delete or rewrite user data to make a test pass. - Do not edit generated output when the source can be changed instead. The publish workflow builds
- Avoid broad refactors unless they are necessary for the requested outcome. `reme_studio/dist-static` and stages it under `reme_studio/src/reme_studio/static`; change `reme_studio/` source for
- Do not introduce dependencies without a concrete need and repository-level justification. frontend work.
- Treat network access, real credentials, and external service mutations as opt-in. - Do not silently change CLI flags, configuration keys, workspace layouts, serialized schemas, endpoint shapes,
- State which validations passed and which were not run in the final handoff. streaming termination, or service interfaces. Preserve compatibility where practical and document intentional
migrations.
- Do not introduce dependencies without a concrete repository-level need.
- Do not commit `.env` files, credentials, runtime memory, logs, indexes, caches, benchmark outputs, or generated
Studio distributions.
- State which validations passed and which relevant checks were not run in the final handoff.
If a requirement is ambiguous, first infer intent from nearby code, tests, and schemas. Ask If a requirement is ambiguous, infer intent from nearby code, schemas, defaults, and tests. Ask the user only when the
the user only when the remaining choice would materially alter a public contract, user data, remaining choice would materially alter a public contract, user data, or an external system.
or external system.

54
Dockerfile Normal file
View file

@ -0,0 +1,54 @@
# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS studio-builder
WORKDIR /build/reme_studio
COPY reme_studio/package.json reme_studio/package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY reme_studio/ ./
RUN npm run build:static && test -f dist-static/index.html
FROM python:3.11-slim-bookworm AS python-builder
RUN apt-get update \
&& apt-get install -y --no-install-recommends build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
COPY pyproject.toml README.md LICENSE ./
COPY reme/ reme/
COPY reme_studio/pyproject.toml reme_studio/README.md reme_studio/LICENSE reme_studio/
COPY reme_studio/src/ reme_studio/src/
COPY --from=studio-builder /build/reme_studio/dist-static/ reme_studio/dist-static/
COPY scripts/package_studio.py scripts/package_studio.py
RUN python scripts/package_studio.py && python -m venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"
RUN --mount=type=cache,target=/root/.cache/pip \
python -m pip install --upgrade pip \
&& python -m pip install ./reme_studio ".[core,image-heif]" \
&& python -m pip check
# Check installed resources away from the checkout, so source files cannot mask
# incomplete wheels or a Studio package fetched accidentally from PyPI.
WORKDIR /tmp
RUN python -I -c "import reme; from reme_studio import static_dir; from reme.config import resolve_app_config; assert (static_dir() / 'index.html').is_file(); assert resolve_app_config(log_config=False)['service']['backend'] == 'http'"
FROM python:3.11-slim-bookworm AS runtime
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates git libgomp1 libstdc++6 tini tzdata \
&& rm -rf /var/lib/apt/lists/* \
&& groupadd --gid 1000 reme \
&& useradd --uid 1000 --gid reme --no-create-home reme \
&& mkdir -p /app /data \
&& chown reme:reme /app /data
COPY --from=python-builder /opt/venv /opt/venv
COPY deploy/docker/reme_container.py /usr/local/lib/reme_container.py
ENV PATH="/opt/venv/bin:${PATH}" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
HOME=/tmp/reme-home \
REME_WORKSPACE_DIR=/data \
REME_HOST=0.0.0.0
WORKDIR /app
USER reme
EXPOSE 2333
HEALTHCHECK --interval=30s --timeout=5s --start-period=120s --retries=3 \
CMD ["python", "/usr/local/lib/reme_container.py", "--healthcheck"]
ENTRYPOINT ["/usr/bin/tini", "--", "python", "/usr/local/lib/reme_container.py"]
CMD ["start"]

407
README.md
View file

@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%"> <img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p> </p>
<p align="center"> <p align="center">
@ -8,7 +8,7 @@
<a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a> <a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a> <a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a> <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a>
<a href="https://docs.agentscope.io/reme"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="Documentation"></a> <a href="https://reme.agentscope.io"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="Documentation"></a>
<a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a> <a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a>
<a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a> <a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a> <a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a>
@ -20,54 +20,79 @@
</p> </p>
<p align="center"> <p align="center">
<strong>An agent memory layer that turns conversations and resources into readable, editable, searchable Markdown memory.</strong><br> <strong>A local-first, self-evolving personal knowledge base for AI agents.</strong><br>
</p> </p>
> Previous versions: [0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) · > Previous versions: [0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) ·
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) · > [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch) > [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
🧠 ReMe is a local-first memory layer for **AI agents**. It turns conversations and resources into file-based long-term ## ✨ Why ReMe?
memory, then continuously indexes, links, and consolidates that memory for future recall.
## ✨ Core Ideas 🧠 ReMe turns conversations and resources into readable, editable, searchable, and interconnected Markdown memory. Agents
such as QwenPaw and DeepSeek Harness can share the same workspace to retrieve, maintain, and evolve knowledge, while
users retain control of the durable files.
- **Memory as File**: Markdown files with frontmatter and wikilinks serve as memory nodes that both users and agents can - **Memory as File, File as Memory**: ReMe stores durable memory as ordinary Markdown with frontmatter and wikilinks.
read and write directly. Users and agents can inspect, edit, move, sync, and back it up with familiar tools, while indexes and generated
- **Self-evolving knowledge base**: Auto Memory, Auto Resource, and Auto Dream progressively transform conversations and metadata remain rebuildable.
resources into long-term memories, while automatically building wikilink relationships. - **Self-evolving knowledge base**: ReMe progressively turns conversations and resources into daily notes and long-term
- **Progressive hybrid search**: ReMe combines wikilinks, BM25, and embeddings for hybrid retrieval across keyword knowledge, preserving sources while refining facts, preferences, procedures, and relationships over time.
matching, semantic recall, and relationship expansion. - **Recall is precise and context-aware.** BM25, optional embeddings, and wikilink expansion retrieve relevant
- **Agent-friendly integration**: SKILL.md + CLI integration makes it easy for different agents to read, write, line-level passages and their relationships without loading the entire knowledge base into the agent context.
maintain, and reuse memory. - **One memory workspace works across agents.** Personal assistants, coding agents, and other agent runtimes can share
the same local workspace through native integrations, SKILL.md, CLI, HTTP, MCP, or Python APIs.
<p align="center"> <p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe Design Philosophy" width="92%"> <img src="docs/figure/design-philosophy.svg" alt="ReMe Design Philosophy" width="92%">
</p> </p>
## 🔭 Use Cases ## 📰 Latest Updates
- **Personal assistants**: Give personal assistants such as - [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=en) is live**: explore example memory
[QwenPaw](https://github.com/agentscope-ai/QwenPaw), [OpenClaw](https://github.com/openclaw/openclaw), and files, edit Markdown, and browse linked memory graphs right in your browser—no installation or backend required.
[Hermes](https://github.com/nousresearch/hermes-agent) a user-editable long-term memory layer. Everyone is welcome to [try it out](https://reme.agentscope.io/studio/?lang=en)!
- **Coding agents**: Preserve coding style, project background, repository decisions, and workflow experience across
sessions when integrating with coding agents such as [Claude Code](plugins/reme).
- **LLM Wiki**: Turn conversations, notes, and resources into a searchable, traceable, and linked Markdown knowledge
base that both users and agents can maintain.
- **Self-evolving agents**: Support agents that learn from experience by saving successful paths, failed attempts,
reusable procedures, and periodic reflections as memory.
## 📰 News <p align="center">
<a href="https://reme.agentscope.io/studio/?lang=en">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/reme_studio/figures/studio-overview.png" alt="ReMe Studio workspace preview — click to try the Playground" width="480" style="margin: 0 auto;">
</a>
</p>
- [2026.08] - [Experience-driven enhancement method](benchmark/toolmemory/README.md) of agent tool-use - [2026.09] - **[ReMe Memory Tags](https://reme.agentscope.io/en/blog_20260920) published**: an introduction
execution built on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403). to file-native entity tags, rebuildable tag indexes, and tag-filtered memory search.
- [2026.07] - Introduced optional Cookbooks: [Daily Paper](cookbook/daily_paper/README.md) for paper discovery and - [2026.09] - **[Hermes Agent memory provider](https://reme.agentscope.io/en/integrations/hermes) available**: choose HTTP or embedded
analysis, and [Auto Fin](cookbook/auto-fin/README.md) for file-native ETF event research based on CLS news and mode for automatic recall before model calls and asynchronous `auto_memory` after completed turns. The integration
historical market reactions. supports Hermes Agent 0.21+ and includes profile-aware background work.
- [2026.09] - **[OpenClaw plugin](https://reme.agentscope.io/en/integrations/openclaw) released**: install it from
[ClawHub](https://clawhub.ai/agentscope-ai/plugins/reme-openclaw-plugin) or
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-openclaw-plugin) to add native memory recall, automatic
conversation capture, and scheduled consolidation to OpenClaw.
- [2026.09] - **[DeepSeek Harness plugin](https://reme.agentscope.io/en/integrations/dsh) released**: install it from
[Awesome DSH Plugin](https://awesome-dsh-plugin.com/p/agentscope-ai/ReMe--integrations-dsh/) or
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-dsh-plugin) for long-term-memory guidance, `reme_search`,
automatic memory, Auto Dream, and ReMe Status.
<details>
<summary>More updates</summary>
- [2026.08] - **ReMe blog published**: the [ReMe blog](https://reme.agentscope.io/en/reme-blog) introduces the
local-first memory architecture, self-evolving workflows, hybrid search, proactive discovery, and benchmark results.
- [2026.08] - **New ReMe ecosystem plugins**: [Daily Paper](https://reme.agentscope.io/en/plugins/daily-paper)
discovers and analyzes papers and generates file-native briefs, while
[Auto Fin](https://reme.agentscope.io/en/plugins/auto-fin) researches the latest 24 hours of topic-related CLS news
and builds traceable reports with local memory. Try them out.
- [2026.08] - **Plugin development support released**: use [Plugin Development](https://reme.agentscope.io/en/plugin_development) and
[Plugin Management](https://reme.agentscope.io/en/plugin_management) to extend ReMe with Components, Steps, and Jobs. Contributions and
new community plugins are welcome.
- [2026.08] - ReMe's [experience-driven enhancement method](https://reme.agentscope.io/en/benchmarks/toolmemory) for
agent tool use is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403).
- [2026.07] - Our - [2026.07] - Our
paper [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/) paper [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/)
has been accepted to Findings of ACL 2026. has been accepted to Findings of ACL 2026.
</details>
## 🚀 Quick Start ## 🚀 Quick Start
### Installation ### Installation
@ -85,9 +110,28 @@ Install from source:
```bash ```bash
git clone https://github.com/agentscope-ai/ReMe.git git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe cd ReMe
pip install -e ".[core]" pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
``` ```
The static build requires Node.js 22.13 or newer and makes Studio available from the source tree.
### Docker
With Docker and Compose 2.24.0+, build and start ReMe with the bundled Studio:
```bash
mkdir -p .reme
docker compose up --build -d
```
Open <http://127.0.0.1:2333>. The complete workspace persists in `./.reme`. On Linux, set `REME_UID` and `REME_GID` to your
user's IDs when they differ from 1000. See [Docker deployment](https://reme.agentscope.io/en/docker) for model credentials,
custom paths, published images, and upgrades.
### Environment Variables ### Environment Variables
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are
@ -99,7 +143,7 @@ cat > .env <<'EOF'
# EMBEDDING_API_KEY=sk-xxx # EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 # EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# Required for auto_memory, auto_resource, and auto_dream. # Required for auto_memory, auto_resource, auto_dream, and proactive refresh.
LLM_API_KEY=sk-xxx LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF EOF
@ -111,7 +155,7 @@ Basic file operations, BM25 search, wikilink traversal, and reading proactive to
> To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and > To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and
> `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change > `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change
> `components.file_store.default.embedding_store` from `""` to `default`. See the > `components.file_store.default.embedding_store` from `""` to `default`. See the
> [memory search guide](docs/en/memory_search.md) for details. > [memory search guide](https://reme.agentscope.io/en/memory_search) for details.
### Start the Service ### Start the Service
@ -126,10 +170,10 @@ reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181 # reme start workspace_dir=/tmp/reme-demo service.port=8181
``` ```
After startup, check the service status. If you use a custom port, replace `2333` in the URL below with that port.
```bash ```bash
reme version reme version
reme health_check
reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}' curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
``` ```
@ -167,105 +211,26 @@ ReMe stores agent memory as readable Markdown.
Related: [[digest/wiki/memory-as-file.md]] Related: [[digest/wiki/memory-as-file.md]]
``` ```
## 🧑‍🍳 Cookbooks ### ReMe Studio (Optional)
Cookbooks are optional, end-to-end workflows assembled from ReMe jobs and steps. They are not enabled by the default The `core` installation includes Studio. After starting ReMe, open <http://127.0.0.1:2333/> to browse, edit, and search
configuration; select the cookbook's standalone configuration when starting ReMe. Each new cookbook will be added as the workspace. To add Studio to a base installation, use `pip install "reme-ai[web]"`. See the
another row in this table. [ReMe Studio guide](https://reme.agentscope.io/en/workspace/studio) for source builds, configuration, and development.
| Cookbook | Capability | ## 🤝 Use ReMe with Your Agent
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| [Daily Paper](cookbook/daily_paper/README.md) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](cookbook/auto-fin/README.md) | Match CLS events to liquid ETFs, study historical reactions, and generate file-native research reports. |
## 📁 Memory System ReMe can run as a local memory service accessed through the CLI, HTTP API, or MCP server, or it can be embedded in the
host process through its Python API. Host integrations can add memory guidance, recall, and capture to the agent
lifecycle according to the capabilities of each runtime.
> Memory as File, File as Memory. | Agent | Recommended path | Available after integration |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
ReMe treats **memory as files**, progressively processing raw conversations and external resources from `session/` and | **DeepSeek Harness** | Install [`@agentscope-ai/reme-dsh-plugin`](https://reme.agentscope.io/en/integrations/dsh) with `dsh plugin --profile web add @agentscope-ai/reme-dsh-plugin`. | Configurable memory guidance, `reme_search`, automatic turn capture, scheduled Auto Dream, and ReMe Status. |
`resource/` into `daily/`, then consolidating them into reusable long-term memory nodes under `digest/`. | **OpenClaw** | Install [`@agentscope-ai/reme-openclaw-plugin`](https://reme.agentscope.io/en/integrations/openclaw) with `openclaw plugins install clawhub:@agentscope-ai/reme-openclaw-plugin`. | Native memory tools, recall before user-triggered runs, and automatic turn capture. |
| **QwenPaw** | Embed ReMe in-process through its Python API. | Reuse the host lifecycle and model config while keeping memory local and file-based. |
### Directory Structure | **Claude Code** | Start the shared streamable HTTP MCP service and install [the ReMe plugin](https://reme.agentscope.io/en/integrations/claude-code). | Semantic, graph, and state recall through MCP, plus asynchronous session capture through a Stop hook. |
| **Hermes** | Install [the ReMe provider](https://reme.agentscope.io/en/integrations/hermes) and choose HTTP or embedded mode. | Recall before model calls and asynchronous `auto_memory` after each completed turn. |
```text | **Codex and other CLI agents** | Install or copy the [ReMe Memory skill](skills/reme_memory/SKILL.md). | Search, read, and write memory through the CLI; automatic capture requires host lifecycle integration. |
<workspace_dir>/
├── metadata/ # Persistent system state such as indexes, graphs, and catalogs
├── session/ # Raw conversations and agent sessions
│ ├── dialog/
│ │ └── <session_id>.jsonl
│ ├── agentscope/
│ └── claude_code/
├── resource/ # External raw materials
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # Lightly processed memory: daily facts, conversation summaries, resource readings
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <session_event>.md
│ ├── <resource_stem>.md
│ └── interests.yaml
└── digest/ # Long-term memory: personal facts, procedural experience, knowledge nodes
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe file-based memory system overview" width="92%">
</p>
## 🧭 Memory Design Philosophy
> Capture raw dialogs and resources, refine them into long-term preferences, reusable experience, and valuable
> knowledge,
> while keeping the result editable by humans and agents.
### Automatic Memory Flow
ReMe follows a capture → index → consolidate → recall loop. Conversations and resources first become daily memory cards;
background jobs keep files searchable; `auto_dream` distills stable knowledge into `digest/`; agents recall memory
through search, wikilinks, or proactive topics.
| Capability | Entry point | What it does | Output |
|---------------------------------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| [`auto_memory`](docs/en/auto_memory.md) | Agent hook or `reme auto_memory` | Distills useful conversation facts while preserving the raw session. | `session/dialog/*.jsonl`, `daily/<date>/<session>.md` |
| [`auto_resource`](docs/en/auto_resource.md) | Resource watcher or `reme auto_resource` | Turns files under `resource/<date>/` into source-linked daily cards. | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/en/memory_search.md) | Background watcher or `reme reindex` | Maintains chunks, the BM25 index, the wikilink graph, and the optional embedding index. | Searchable `daily/`, `digest/`, and `resource/` content |
| [`auto_dream`](docs/en/auto_dream.md) | `dream_cron` or `reme auto_dream` | Consolidates changed daily cards into long-term personal, procedure, and wiki memory. | `digest/**`, `daily/<date>/interests.yaml` |
| [`proactive`](docs/en/proactive.md) | `reme proactive` before an agent decides to act | Reads topics generated by `auto_dream`; the host agent decides whether and how to mention them. | Structured topics from `daily/<date>/interests.yaml` |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
## 🤝 Agent-friendly Integration
ReMe runs as a local memory service and offers multiple integration paths: CLI, HTTP API, MCP server, and SDK. Different
agents can choose the path that fits their runtime while sharing the same local memory workspace.
| Agents | Recommended path | What works out of the box |
|------------------------------------------------------|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|
| **QwenPaw** | Embed ReMe via the Python SDK. | Reuse the app's own lifecycle and model config while keeping memory local and file-based. |
| **Claude Code** | Start ReMe as an MCP service and install [plugins/reme](plugins/reme). | MCP recall tools, a `reme-memory` skill, and a Stop hook that records sessions automatically. |
| **Other CLI-capable agents (OpenClaw/Hermes/Codex)** | Copy or install [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md). | Search/read/write memory and call `auto_memory`, `auto_dream`, and `proactive` via the CLI. |
<p align="center"><b>Integration demos</b></p> <p align="center"><b>Integration demos</b></p>
@ -295,39 +260,171 @@ agents can choose the path that fits their runtime while sharing the same local
</tr> </tr>
</table> </table>
## 🛠️ ReMe Operations ## 🧠 How ReMe Works
ReMe operates the workspace through a unified job interface exposed by the CLI. Agents usually only need retrieval, > Memory as File, File as Memory.
reading, writing, editing, and automatic memory commands. Lower-level indexing, frontmatter, and file operation commands
are mainly for maintenance, debugging, or advanced integration. Run `reme help` for the full job list.
| Command | Purpose | ReMe treats **memory as files**, progressively processing filtered conversation source records and external resources
|-------------------------------------------|----------------------------------------------------------------------------------------| from `session/` and `resource/` into `daily/`, then `digest/`. The default workspace is `.reme/` under the current
| `reme start` | Start the local ReMe service. | directory; `workspace_dir=...` selects a different user-owned location.
| `reme version` / `reme health_check` | Check package and component status. |
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](docs/en/memory_search.md) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme auto_memory` | Turn conversation messages into daily memory cards. Requires LLM credentials. |
| `reme auto_resource` | Interpret files under `resource/` into daily resource cards. Requires LLM credentials. |
| `reme auto_dream` / `reme proactive` | Consolidate daily memory into long-term digest and surface topics worth attention. |
| `reme reindex` | Rebuild search and wikilink indexes from existing files. |
## 🤝 Community and Support ### Workspace Layout
- **Issues and requests**: Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) first. If there is no ```text
related discussion, open a new issue with background, expected behavior, and impact scope. <workspace_dir>/
- **Code contributions**: Before making changes, read ├── metadata/ # Rebuildable indexes, graphs, catalogs, and caches
the [contribution guide](https://docs.agentscope.io/reme/latest/en/contribution). Source, schemas, and tests are the ├── session/ # Conversation source records and agent sessions
authoritative architecture and extension guide. │ ├── dialog/
- **Documentation contributions**: Submit user-facing documentation changes to the │ │ └── <session_id>.jsonl # Source messages saved by auto_memory
[unified documentation repository](https://github.com/agentscope-ai/docs) under `reme/<version>/{en,zh}/`. │ └── claude_code/
│ └── <session_id>.jsonl # ReMe copy used by auto_memory_cc
├── mem_session/ # Generated agent-wrapper sessions/config, not user memory
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # External raw materials
│ ├── <resource>.<ext> # Root-level files enter today's daily layer
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # Lightly processed memory: daily facts, conversation summaries, resource readings
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <generated_name>.md # Topic-named conversation or resource card
│ └── interests.yaml
└── digest/ # Long-term memory: personal facts, procedural experience, knowledge nodes
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe file-based memory system overview" width="92%">
</p>
### Memory Lifecycle
ReMe follows a capture → index → consolidate → recall loop. Workspace files remain the durable source of truth;
everything under `metadata/` is rebuildable.
| Capability | Entry point | What it does | Output |
| ------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [`auto_memory`](https://reme.agentscope.io/en/auto_memory) | Agent hook or `reme auto_memory` | Distills useful conversation facts while preserving a filtered conversation source record. | `session/dialog/*.jsonl`, `daily/<date>/<generated-name>.md` |
| [`auto_resource`](https://reme.agentscope.io/en/auto_resource) | Resource watcher or `reme auto_resource` | Turns files under `resource/` into source-linked, content-named daily cards. | `daily/<date>/<resource-card>.md` |
| [`auto_index`](https://reme.agentscope.io/en/memory_search) | Background watcher or `reme reindex` | The watcher ingests Markdown from `daily/` and `digest/`; `reindex` only rebuilds BM25 and embeddings from already-ingested chunks. | Searchable chunks, BM25, wikilink graph, and optional vectors |
| [`auto_dream`](https://reme.agentscope.io/en/auto_dream) | `dream_cron` or `reme auto_dream` | By default, extracts up to five reusable units from changed files in the latest two-day window, then creates, corroborates, refines, or corrects digest nodes. | `digest/**` |
| [`proactive_read`](https://reme.agentscope.io/en/proactive) | `reme proactive_read` before an agent decides to act | Reads topics generated by the independent proactive refresh flow; the host agent decides whether and how to mention them. | Structured topics from `daily/<date>/interests.yaml` |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
Search returns matching chunks with line ranges and bounded wikilink neighbors. Optional vector results are fused with
BM25 through reciprocal rank fusion (RRF).
> [!IMPORTANT]
>
> `proactive_read` only reads and exposes interest topics produced by proactive refresh. It does not independently browse the web,
> send notifications, or rewrite the knowledge base; the host agent decides whether and how to act on a topic.
## 📊 Benchmarks
ReMe evaluates multi-session and long-context memory with agentic search-and-read workflows. The figures below are the
published reference runs in this repository; model, prompt, dataset, and judging details are documented with each
benchmark.
| Benchmark | Setting | Sample size | Agentic score | Focus |
| --------------------------------------------------------------------------- | ------------ | -----------------------: | ------------: | ------------------------------------------------------------------ |
| **[LongMemEval cleaned-s](https://reme.agentscope.io/en/benchmarks/longmemeval)** | **Overall** | **500 questions** | **89.4%** | Cross-session retrieval, knowledge updates, and temporal reasoning |
| [BEAM](https://reme.agentscope.io/en/benchmarks/beam) | 100K context | 20 cases / 400 questions | 66.1% | Ten types of long-context memory tasks |
| [BEAM](https://reme.agentscope.io/en/benchmarks/beam) | 1M context | 35 cases / 700 questions | 65.0% | Ultra-long conversation settings |
ReMe also achieved a **0.580 PROC score across five user personas** in the repository's
[π-Bench evaluation](https://reme.agentscope.io/en/benchmarks/pibench), 2.4% above NanoBot under the same test-model configuration. PROC
measures proactive handling of hidden intent, clarification, cross-session preferences and conventions, task
dependencies, and underspecified requests.
## 🧩 Extensions and Plugins
Plugins are optional Python distributions that contribute Component, Step, or Job backends and configuration. They are
installed separately and enabled explicitly by configuration. Daily Paper and Auto Fin are independently packaged
plugins; see their documentation for [Daily Paper](https://reme.agentscope.io/en/plugins/daily-paper) and
[Auto Fin](https://reme.agentscope.io/en/plugins/auto-fin).
| Plugin | Capability |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| [Daily Paper](https://reme.agentscope.io/en/plugins/daily-paper) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](https://reme.agentscope.io/en/plugins/auto-fin) | Fetch topic-related CLS news, search ReMe history, and generate wikilink-backed Markdown reports. |
See [Plugin Management](https://reme.agentscope.io/en/plugin_management) to install, inspect, validate, enable, and uninstall ReMe plugins.
## 📚 Documentation
These guides cover the main user workflows and the runtime contracts implemented by the current code.
| Guide | What you will learn |
| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [Quick Start](https://reme.agentscope.io/en/quick_start) | Install ReMe, start the service, and run the first file and memory operations. |
| [Configuration](https://reme.agentscope.io/en/configuration) | Configure the workspace, models, Service, Jobs, Components, plugins, and CLI overrides. |
| [Services and Deployment](https://reme.agentscope.io/en/services) | Use HTTP, SSE, MCP, and Studio while respecting the default security boundary. |
| [Memory as File](https://reme.agentscope.io/en/memory_as_file) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. |
| [Auto Memory](https://reme.agentscope.io/en/auto_memory) | Preserve source conversations and distill reusable daily memory cards. |
| [Auto Resource](https://reme.agentscope.io/en/auto_resource) | Import supported text and image resources as source-linked daily cards. |
| [Auto Dream](https://reme.agentscope.io/en/auto_dream) and [Auto Link](https://reme.agentscope.io/en/auto_link) | Consolidate daily notes into evolving digest nodes and readable wikilink relationships. |
| [Memory Search](https://reme.agentscope.io/en/memory_search) | Use BM25, optional vectors, RRF fusion, line-range recall, and progressive link expansion. |
| [Proactive](https://reme.agentscope.io/en/proactive) | Read interest topics safely and integrate them into a host agent's decision flow. |
| [Application Scenarios](https://reme.agentscope.io/en/reme_scene) | Follow concrete financial research, coding-memory, and personal knowledge-base examples. |
| [Framework](https://reme.agentscope.io/en/framework) | Understand Application, Job, Step, Component, service, configuration, and lifecycle boundaries. |
| [Agent Integrations](https://reme.agentscope.io/en/integrations) | Choose an interface and connect DSH, Claude Code, OpenClaw, Hermes, Codex, or another agent. |
| [DSH plugin](https://reme.agentscope.io/en/integrations/dsh) and [Claude Code plugin](https://reme.agentscope.io/en/integrations/claude-code) | Configure host-native recall, automatic capture, consolidation, and diagnostics. |
| [CLI and Job API](https://reme.agentscope.io/en/reference/cli) | Learn command syntax and use the generated default Job parameter reference. |
| [Operations and Recovery](https://reme.agentscope.io/en/operations) | Diagnose services, maintain indexes, and back up, migrate, or recover a workspace. |
| [ReMe Blog](https://reme.agentscope.io/en/reme-blog) | Read the product story, design rationale, examples, and benchmark summary. |
## 🛠️ Common Commands
Run `reme help` for the full job list. Common workspace and maintenance commands are:
| Command | Purpose |
| ----------------------------------------- | --------------------------------------------------------------------------------- |
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](https://reme.agentscope.io/en/memory_search) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme traverse` / `reme graph_snapshot` | Explore wikilink neighborhoods or the category-rooted digest graph. |
| `reme chat` | Stream a read-only, workspace-aware agent conversation. Requires LLM credentials. |
| `reme reindex` | Rebuild BM25 and embedding indexes from already-ingested chunks. |
## 🤝 Community and Contributing
- **Issues, requests, and help**: Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) first. If there is no
related discussion, open one with the background, expected behavior, and impact scope.
- **Code contributions**: Before making changes, read the repository's
[contribution guide](https://reme.agentscope.io/en/contributing). Source, schemas, and tests are the authoritative architecture and
extension guide.
- **Documentation contributions**: Update the canonical files under `docs/en/`, `docs/zh/`, or the relevant package
directory in this repository. The documentation site is generated from these files.
- **Commit convention**: Conventional Commits are recommended, for example `feat(search): add link expansion option` or - **Commit convention**: Conventional Commits are recommended, for example `feat(search): add link expansion option` or
`docs(zh): update quick start`. `docs(zh): update quick start`.
- **Pre-submit checks**: Before submitting a PR, try to run `pre-commit run --all-files` and `pytest`. If tests that - **Pre-submit checks**: Before submitting a PR, try to run `pre-commit run --all-files` and `pytest`. If tests that
depend on LLMs, embeddings, or external services cannot run, explain that in the PR. depend on LLMs, embeddings, or external services cannot run, explain that in the PR.
- **Get help**: Use [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues) for bugs and feature requests. Project - **Documentation**: Visit [reme.agentscope.io](https://reme.agentscope.io).
documentation is available at [https://docs.agentscope.io/reme](https://docs.agentscope.io/reme).
### Contributors ### Contributors

View file

@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%"> <img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p> </p>
<p align="center"> <p align="center">
@ -8,7 +8,7 @@
<a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a> <a href="https://pepy.tech/project/reme-ai/"><img src="https://img.shields.io/pypi/dm/reme-ai" alt="PyPI Downloads"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a> <a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/commit-activity/m/agentscope-ai/ReMe?style=flat-square" alt="GitHub commit activity"></a>
<a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a> <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-black" alt="License"></a>
<a href="https://docs.agentscope.io/reme"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="文档"></a> <a href="https://reme.agentscope.io"><img src="https://img.shields.io/badge/docs-ReMe-blue" alt="文档"></a>
<a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a> <a href="./README.md"><img src="https://img.shields.io/badge/English-Click-yellow" alt="English"></a>
<a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a> <a href="./README_ZH.md"><img src="https://img.shields.io/badge/简体中文-点击查看-orange" alt="简体中文"></a>
<a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a> <a href="https://github.com/agentscope-ai/ReMe"><img src="https://img.shields.io/github/stars/agentscope-ai/ReMe?style=social" alt="GitHub Stars"></a>
@ -20,47 +20,71 @@
</p> </p>
<p align="center"> <p align="center">
<strong>一个将对话和资料转化为可读、可编辑、可检索 Markdown 记忆的 Agent 记忆层。</strong><br> <strong>面向 AI Agent 的 local-first 自进化个人知识库。</strong><br>
</p> </p>
> 历史版本:[0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) · > 历史版本:[0.3.x](https://github.com/agentscope-ai/ReMe/tree/reme_v3) ·
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) · > [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch) > [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
🧠 ReMe 是一个面向 **AI 智能体** 的 local-first 记忆层。它把对话和资料沉淀为文件化长期记忆,并持续完成索引、链接和整理,让后续 ## ✨ 为什么选择 ReMe?
Agent 能够可靠召回。
## ✨ 核心创新 🧠 ReMe 将对话和资料持续沉淀为可读、可编辑、可检索、相互链接的 Markdown 记忆。QwenPaw、DeepSeek Harness 等 Agent
可以共享同一个 workspace,共同检索、维护和演化知识,而持久文件始终由用户掌控。
- **Memory as File**:以带 frontmatter 和 wikilink 的 Markdown 作为记忆节点,让用户和 Agent 都能直接读写。 - **Memory as File, File as Memory**:ReMe 使用带 frontmatter 和 wikilink 的普通 Markdown 保存持久记忆。用户和 Agent
- **自进化知识库**:通过 Auto Memory、Auto Resource 和 Auto Dream,把对话与资料逐步加工为长期记忆,并自动建立 wikilink 关系。 都可以使用熟悉的工具查看、编辑、移动、同步和备份;索引及生成的元数据均可重建。
- **渐进式混合搜索**:融合 wikilink、BM25 和 embedding,支持从关键词匹配到语义召回、关系扩展的混合检索。 - **自进化知识库**:ReMe 将对话和资料逐步加工为 daily note 与长期知识,在保留来源的同时,持续提炼事实、偏好、
- **Agent 友好集成**:通过 SKILL.md + CLI 接入,方便不同 Agent 读写、维护与复用记忆。 流程经验及其关系。
- **精准召回所需上下文。** ReMe 结合 BM25、可选 embedding 和 wikilink 展开,召回带行号的相关片段及其关系,无需把整个知识库塞入
Agent 上下文。
- **一个 workspace,可供不同 Agent 共同使用。** 个人助理、coding agent 和其他 Agent runtime 可以通过原生集成、SKILL.md、CLI、
HTTP、MCP 或 Python API 共享同一个本地记忆空间。
<p align="center"> <p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe 设计理念" width="92%"> <img src="docs/figure/design-philosophy.svg" alt="ReMe 设计理念" width="92%">
</p> </p>
## 🔭 适用场景 ## 📰 最新动态
- **Personal assistants**:为 [QwenPaw](https://github.com/agentscope-ai/QwenPaw)、 - [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=zh) 上线**:无需安装或启动后端,
[OpenClaw](https://github.com/openclaw/openclaw)、[Hermes](https://github.com/nousresearch/hermes-agent) 即可在浏览器中浏览示例记忆文件、编辑 Markdown、探索记忆关联图谱。欢迎大家[来体验](https://reme.agentscope.io/studio/?lang=zh)!
等个人助理提供用户可编辑的长期记忆层。
- **Coding agents**:在接入 [Claude Code](plugins/reme) 等 coding agent 时,跨会话保留代码风格、项目背景、仓库决策和流程经验。
- **LLM Wiki**:把对话、笔记和资料转化为可检索、可追溯、可链接的 Markdown 知识库,由用户和 Agent 共同维护。
- **Self-evolving agents**:帮助 Agent 从经验中学习,把成功路径、失败尝试、可复用流程和阶段性反思沉淀为记忆。
## 📰 新闻 <p align="center">
<a href="https://reme.agentscope.io/studio/?lang=zh">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/reme_studio/figures/studio-overview.png" alt="ReMe Studio 工作区预览,点击体验 Playground" width="480" style="margin: 0 auto;">
</a>
</p>
- [2026.09] - **[给记忆加上“标签”](https://reme.agentscope.io/zh/blog_20260920)发布**:介绍基于 Markdown 的实体标签、
可重建 Tag Index 与标签过滤检索。
- [2026.09] - **[Hermes Agent 记忆 Provider](https://reme.agentscope.io/zh/integrations/hermes) 已可使用**:支持 HTTP 和 Embedded
两种模式,在模型调用前自动召回、每轮对话结束后异步执行 `auto_memory`。集成支持 Hermes Agent 0.21 及以上版本,后台任务也会继承当前 profile 上下文。
- [2026.09] - **[OpenClaw 插件](https://reme.agentscope.io/zh/integrations/openclaw) 发布**:可通过
[ClawHub](https://clawhub.ai/agentscope-ai/plugins/reme-openclaw-plugin) 或
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-openclaw-plugin) 安装,为 OpenClaw 提供原生记忆召回、自动对话捕获和定时整理能力。
- [2026.09] - **[DeepSeek Harness 插件](https://reme.agentscope.io/zh/integrations/dsh) 发布**:可通过
[Awesome DSH Plugin](https://awesome-dsh-plugin.com/p/agentscope-ai/ReMe--integrations-dsh/) 或
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-dsh-plugin) 安装,提供长期记忆指引、`reme_search`、自动记忆、Auto Dream 和 ReMe Status。
<details>
<summary>更多更新</summary>
- [2026.08] - **ReMe 博客发布**:[ReMe 博客](https://reme.agentscope.io/zh/reme-blog) 系统介绍了本地优先的记忆架构、
自进化工作流、混合检索、主动发现与评测结果。
- [2026.08] - **新增 ReMe 生态插件**:[每日论文](https://reme.agentscope.io/zh/plugins/daily-paper) 可自动发现、解析论文并生成文件化简报;
[Auto Fin](https://reme.agentscope.io/zh/plugins/auto-fin) 可研究最近 24 小时的主题相关财联社新闻,并结合本地记忆构建可追溯报告。欢迎体验。
- [2026.08] - **插件开发能力上线**:参考 [插件开发](https://reme.agentscope.io/zh/plugin_development) 与 [插件管理](https://reme.agentscope.io/zh/plugin_management),
为 ReMe 扩展 Component、Step 和 Job;欢迎开发并分享你的插件。
- [2026.08] - 基于 ReMe 的智能体工具使用 - [2026.08] - 基于 ReMe 的智能体工具使用
[经验驱动增强方法](benchmark/toolmemory/README_ZH.md)已发布,见 [经验驱动增强方法](https://reme.agentscope.io/zh/benchmarks/toolmemory) 已发布,见
[arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。 [arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。
- [2026.07] - 新增可选 Cookbook 工作流:[每日论文](cookbook/daily_paper/README_ZH.md)用于论文发现与解析,
[Auto Fin](cookbook/auto-fin/README_ZH.md)用于结合财联社新闻和历史行情开展文件化 ETF 事件研究。
- [2026.07] - - [2026.07] -
我们的论文 [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/) 我们的论文 [Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution](https://aclanthology.org/2026.findings-acl.829/)
已被 Findings of ACL 2026 接收。 已被 Findings of ACL 2026 接收。
</details>
## 🚀 快速开始 ## 🚀 快速开始
### 安装 ### 安装
@ -78,13 +102,31 @@ pip install "reme-ai[core]"
```bash ```bash
git clone https://github.com/agentscope-ai/ReMe.git git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe cd ReMe
pip install -e ".[core]" pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
``` ```
### 环境变量 静态构建要求 Node.js 22.13 或更高版本,并让源码安装可以直接使用 Studio。
如果需要 LLM 驱动的记忆演化或 embedding 检索,可以配置环境变量。embedding 默认关闭,因此默认配置不会启动 embedding 模型,也不需要 ### Docker
embedding API key。
使用 Docker 和 Compose 2.24.0+,构建并启动包含 Studio 的 ReMe:
```bash
mkdir -p .reme
docker compose up --build -d
```
打开 <http://127.0.0.1:2333>,完整工作区保存在宿主机的 `./.reme`。Linux 用户的 UID/GID 不是 1000 时,请设置对应的
`REME_UID` 和 `REME_GID`。模型凭证、自定义路径、发布镜像和升级方式见 [Docker 部署](https://reme.agentscope.io/zh/docker)。
### 环境变量配置
如果需要 LLM 驱动的记忆演化或 embedding 检索,请在启动服务前配置环境变量。embedding 默认关闭,因此默认配置不会启动
embedding 模型,也不需要 embedding API key。
```bash ```bash
cat > .env <<'EOF' cat > .env <<'EOF'
@ -92,19 +134,19 @@ cat > .env <<'EOF'
# EMBEDDING_API_KEY=sk-xxx # EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 # EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# 必须:auto_memory、auto_resource 和 auto_dream 需要 LLM。 # 必须:auto_memory、auto_resource、auto_dream 和 proactive refresh 需要 LLM。
LLM_API_KEY=sk-xxx LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF EOF
``` ```
基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取可以先不配置 LLM 凭证。 基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取不需要 LLM 凭证,可以跳过此步骤并直接启动服务。
> [!NOTE] > [!NOTE]
> 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中 > 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中
> `components.as_embedding` 和 `components.embedding_store` 的注释,并将 > `components.as_embedding` 和 `components.embedding_store` 的注释,并将
> `components.file_store.default.embedding_store` 从 `""` 改为 `default`。完整说明见 > `components.file_store.default.embedding_store` 从 `""` 改为 `default`。完整说明见
> [记忆检索文档](docs/zh/memory_search.md)。 > [记忆检索文档](https://reme.agentscope.io/zh/memory_search)。
### 启动服务 ### 启动服务
@ -119,10 +161,10 @@ reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181 # reme start workspace_dir=/tmp/reme-demo service.port=8181
``` ```
启动后可以检查服务状态;如果使用了自定义端口,请将下面 URL 中的 `2333` 替换为对应端口。
```bash ```bash
reme version reme version
reme health_check
reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}' curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
``` ```
@ -160,101 +202,25 @@ ReMe 会把 Agent 记忆保存为可读的 Markdown。
相关链接:[[digest/wiki/memory-as-file.md]] 相关链接:[[digest/wiki/memory-as-file.md]]
``` ```
## 🧑‍🍳 Cookbooks ### ReMe Studio(可选)
Cookbook 是由 ReMe jobs 和 steps 组装而成的可选端到端工作流。默认配置不会开启它们;启动 ReMe 时选择对应的 独立配置即可启用。后续新增的 上面的 `core` 安装已包含 Studio。启动 ReMe 后,打开 <http://127.0.0.1:2333/> 即可浏览、编辑和搜索 workspace。
cookbook 会继续在表格中按行追加。 如需为基础安装单独添加 Studio,可使用 `pip install "reme-ai[web]"`。源码构建、配置和开发说明见
[ReMe Studio 指南](https://reme.agentscope.io/zh/workspace/studio)。
| Cookbook | 能力 | ## 🤝 将 ReMe 接入你的 Agent
|-----------------------------------------------|-----------------------------------------------------------------------|
| [每日论文](cookbook/daily_paper/README_ZH.md) | 发现并排序论文,使用 Agent 解读 PDF,生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](cookbook/auto-fin/README_ZH.md) | 将财联社事件匹配到高流动性 ETF,研究历史反应并生成文件化研究报告。 |
## 📁 记忆系统 ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP server 接入,也可以通过 Python API 嵌入宿主进程。宿主集成可根据不同
runtime 的能力,将记忆指引、召回和捕获接入 Agent 生命周期。
> Memory as File, File as Memory. | Agent | 推荐接入方式 | 接入后能力 |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
ReMe 将 **记忆视为文件**,让原始对话和外部资料从 `session/`、`resource/` 渐进加工到 `daily/`,再沉淀为 `digest/` | **DeepSeek Harness** | 使用 `dsh plugin --profile web add @agentscope-ai/reme-dsh-plugin` 安装 [`@agentscope-ai/reme-dsh-plugin`](https://reme.agentscope.io/zh/integrations/dsh)。 | 可配置记忆指引、`reme_search`、自动对话捕获、定时 Auto Dream 和 ReMe Status。 |
中可长期复用的知识节点。 | **OpenClaw** | 使用 `openclaw plugins install clawhub:@agentscope-ai/reme-openclaw-plugin` 安装 [`@agentscope-ai/reme-openclaw-plugin`](https://reme.agentscope.io/zh/integrations/openclaw)。 | 原生记忆工具、用户触发运行前召回和自动对话捕获。 |
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主生命周期和模型配置,同时保持记忆本地、文件化。 |
### 目录结构 | **Claude Code** | 启动共享的 streamable HTTP MCP service,并安装 [ReMe 插件](https://reme.agentscope.io/zh/integrations/claude-code)。 | 通过 MCP 进行语义、图关系和状态召回,并由 Stop Hook 异步捕获会话。 |
| **Hermes** | 安装 [ReMe provider](https://reme.agentscope.io/zh/integrations/hermes),并选择 HTTP 或 Embedded 模式。 | 模型调用前召回,每轮对话完成后异步执行 `auto_memory`。 |
```text | **Codex 及其他 CLI Agent** | 安装或复制 [ReMe Memory skill](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动捕获需要显式接入宿主生命周期。 |
<workspace_dir>/
├── metadata/ # 系统索引、图谱、catalog 等持久状态
├── session/ # 原始对话和 Agent session
│ ├── dialog/
│ │ └── <session_id>.jsonl
│ ├── agentscope/
│ └── claude_code/
├── resource/ # 外部原始材料
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # 浅加工记忆:当天事实、对话摘要、资源解读
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <session_event>.md
│ ├── <resource_stem>.md
│ └── interests.yaml
└── digest/ # 长期记忆:个人事实、流程经验、知识节点
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe 文件化记忆系统总览" width="92%">
</p>
## 🧭 记忆设计理念
> 捕获原始对话和资料,将其整理为长期偏好、可复用经验和有价值的知识,并让结果始终能被用户和 Agent 直接编辑。
### 自动记忆流程
ReMe 遵循 capture → index → consolidate → recall 的循环。对话和资料先变成 daily 记忆卡片;后台任务保持文件可检索;
`auto_dream` 将稳定知识沉淀到 `digest/`;Agent 再通过搜索、wikilink 或 proactive topics 召回记忆。
| 能力 | 入口 | 作用 | 输出 |
|---------------------------------------------|-------------------------------------------|--------------------------------------------------------------------------|-------------------------------------------------------|
| [`auto_memory`](docs/zh/auto_memory.md) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留原始 session。 | `session/dialog/*.jsonl`、`daily/<date>/<session>.md` |
| [`auto_resource`](docs/zh/auto_resource.md) | 资源监听或 `reme auto_resource` | 将 `resource/<date>/` 下的文件转为带来源链接的 daily 卡片。 | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/zh/memory_search.md) | 后台监听或 `reme reindex` | 维护 chunks、BM25 索引、wikilink 图谱及可选的 embedding 索引。 | 可检索的 `daily/`、`digest/`、`resource/` 内容 |
| [`auto_dream`](docs/zh/auto_dream.md) | `dream_cron` 或 `reme auto_dream` | 将变化的 daily 卡片整理为长期 personal、procedure 和 wiki 记忆。 | `digest/**`、`daily/<date>/interests.yaml` |
| [`proactive`](docs/zh/proactive.md) | Agent 决定主动行动前调用 `reme proactive` | 读取 `auto_dream` 生成的 topics;是否以及如何提醒用户由宿主 Agent 决定。 | 来自 `daily/<date>/interests.yaml` 的结构化 topics |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
## 🤝 Agent-friendly Integration
ReMe 作为本地记忆服务运行,并提供 CLI、HTTP API、MCP server 和 SDK 等多种接入方式。不同 Agent 可以选择适合自身 runtime
的路径,同时共享同一个本地 memory workspace。
| Agent | 推荐接入方式 | 开箱可用能力 |
|------------------------------------------------------|-------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| **QwenPaw** | 通过 Python SDK 嵌入 ReMe。 | 复用应用自身生命周期和模型配置,同时保持 memory 本地、文件化。 |
| **Claude Code** | 以 MCP service 启动 ReMe,并安装 [plugins/reme](plugins/reme)。 | MCP recall tools、`reme-memory` skill,以及自动记录会话的 Stop hook。 |
| **Other CLI-capable agents (OpenClaw/Hermes/Codex)** | 复制或安装 [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索/读取/写入记忆,并调用 `auto_memory`、`auto_dream` 和 `proactive`。 |
<p align="center"><b>集成演示</b></p> <p align="center"><b>集成演示</b></p>
@ -284,36 +250,160 @@ ReMe 作为本地记忆服务运行,并提供 CLI、HTTP API、MCP server 和
</tr> </tr>
</table> </table>
## 🛠️ ReMe Operations ## 🧠 ReMe 如何工作
ReMe 通过 CLI 暴露的统一 job interface 操作 workspace。Agent 通常只需要使用检索、读取、写入、编辑和自动记忆相关命令;更底层的索引、 > Memory as File, File as Memory.
frontmatter 和文件操作接口主要用于维护、调试或高级集成。完整 job 列表可以运行 `reme help` 查看。
ReMe 将 **记忆视为文件**,让过滤后的对话来源记录和外部资料从 `session/`、`resource/` 渐进加工到 `daily/`,再沉淀为
`digest/`。默认 workspace 是当前目录下的 `.reme/`;可通过 `workspace_dir=...` 选择其他由用户控制的位置。
### Workspace 结构
```text
<workspace_dir>/
├── metadata/ # 可重建的索引、图谱、catalog 和缓存
├── session/ # 对话来源记录和 Agent session
│ ├── dialog/
│ │ └── <session_id>.jsonl # auto_memory 保存的来源消息
│ └── claude_code/
│ └── <session_id>.jsonl # auto_memory_cc 使用的 ReMe 副本
├── mem_session/ # Agent wrapper 生成的 session/配置,不是用户记忆
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # 外部原始材料
│ ├── <resource>.<ext> # 根目录文件进入当天 daily 层
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # 浅加工记忆:当天事实、对话摘要、资源解读
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <generated_name>.md # 按主题命名的对话或资源卡片
│ └── interests.yaml
└── digest/ # 长期记忆:个人事实、流程经验、知识节点
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
```
<p align="center">
<img src="docs/figure/reme-overview.svg" alt="ReMe 文件化记忆系统总览" width="92%">
</p>
### 记忆生命周期
ReMe 遵循 capture → index → consolidate → recall 的循环。workspace 文件是持久化的事实来源,`metadata/` 中的内容均可重建。
| 能力 | 入口 | 作用 | 输出 |
| ------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`auto_memory`](https://reme.agentscope.io/zh/auto_memory) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留过滤后的对话来源记录。 | `session/dialog/*.jsonl`、`daily/<date>/<generated-name>.md` |
| [`auto_resource`](https://reme.agentscope.io/zh/auto_resource) | 资源监听或 `reme auto_resource` | 将 `resource/` 下的文件转为带来源链接、按内容命名的 daily 卡片。 | `daily/<date>/<resource-card>.md` |
| [`auto_index`](https://reme.agentscope.io/zh/memory_search) | 后台监听或 `reme reindex` | watcher 摄取 `daily/` 和 `digest/` 中的 Markdown;`reindex` 只基于已摄取的 chunks 重建 BM25 和 Embedding。 | 可检索的 chunks、BM25、wikilink 图谱和可选向量 |
| [`auto_dream`](https://reme.agentscope.io/zh/auto_dream) | `dream_cron` 或 `reme auto_dream` | 默认从最近两天内变化的文件中最多提取 5 个可复用 unit,再创建、印证、补充或修正 digest 节点。 | `digest/**` |
| [`proactive_read`](https://reme.agentscope.io/zh/proactive) | Agent 决定主动行动前调用 `reme proactive_read` | 读取独立 proactive refresh 流程生成的 topics;是否以及如何提醒用户由宿主 Agent 决定。 | 来自 `daily/<date>/interests.yaml` 的结构化 topics |
<table>
<tr>
<td align="center" width="50%">
<img src="docs/figure/memory-as-file.svg" alt="Memory as File" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-memory-resource.svg" alt="Auto Memory and Resource" width="92%">
</td>
</tr>
<tr>
<td align="center" width="50%">
<img src="docs/figure/auto-dream-and-proactive.svg" alt="Auto Dream and Proactive" width="92%">
</td>
<td align="center" width="50%">
<img src="docs/figure/auto-index-and-memory-search.svg" alt="Auto Index and Memory Search" width="92%">
</td>
</tr>
</table>
搜索返回带行号范围的相关 chunks 和数量受限的 wikilink 邻居;可选向量结果通过 RRF 与 BM25 融合。
> [!IMPORTANT]
>
> `proactive_read` 只读取并暴露 proactive refresh 生成的兴趣主题,不会自行联网、发送通知或改写知识库;是否以及如何使用主题,由宿主 Agent
> 决定。
## 📊 评测结果
ReMe 通过 Agent 多轮搜索与读取的方式,评测多会话和超长上下文中的记忆能力。下表为仓库中已公开的参考实验结果;模型、prompt、数据集和评判细节见各评测文档。
| 基准 | 设置 | 样本量 | Agentic 得分 | 主要检验内容 |
| --------------------------------------------------------------------------- | ----------- | ----------------: | -----------: | ------------------------------ |
| **[LongMemEval cleaned-s](https://reme.agentscope.io/zh/benchmarks/longmemeval)** | **整体** | **500 题** | **89.4%** | 跨会话检索、知识更新与时间推理 |
| [BEAM](https://reme.agentscope.io/zh/benchmarks/beam) | 100K 上下文 | 20 cases / 400 题 | 66.1% | 十类长上下文记忆任务 |
| [BEAM](https://reme.agentscope.io/zh/benchmarks/beam) | 1M 上下文 | 35 cases / 700 题 | 65.0% | 超长对话设置 |
在仓库的 [π-Bench 评测](https://reme.agentscope.io/zh/benchmarks/pibench)中,ReMe Agent 在 5 种用户角色上的平均 **PROC 得分为 0.580**
,比相同测试模型配置的 NanoBot 高 2.4%。PROC 用于评估隐藏意图完成、针对性澄清、跨会话偏好和规范复用、跨任务依赖推断以及欠规格请求推进等主动性能力。
## 🧩 扩展与插件
插件是可选的独立 Python distribution,可以贡献 Component、Step、Job backend 和配置,并通过配置显式启用。每日论文与 Auto Fin
均已独立打包,使用说明分别见[每日论文](https://reme.agentscope.io/zh/plugins/daily-paper)和
[Auto Fin](https://reme.agentscope.io/zh/plugins/auto-fin)。
| 插件 | 能力 |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [每日论文](https://reme.agentscope.io/zh/plugins/daily-paper) | 发现并排序论文,使用 Agent 解读 PDF,生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](https://reme.agentscope.io/zh/plugins/auto-fin) | 拉取主题相关财联社新闻,搜索 ReMe 历史材料并生成带 wikilink 的 Markdown 报告。 |
安装、查看、校验、启用和卸载 ReMe 插件的方法见[插件管理](https://reme.agentscope.io/zh/plugin_management)。
## 📚 文档
下列文档覆盖主要使用流程,并以当前代码的运行时契约为准。
| 文档 | 主要内容 |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| [快速开始](https://reme.agentscope.io/zh/quick_start) | 安装 ReMe、启动服务,并执行首次文件和记忆操作。 |
| [基础配置](https://reme.agentscope.io/zh/configuration) | 配置 workspace、模型、Service、Job、Component、插件和命令行覆盖。 |
| [服务与部署](https://reme.agentscope.io/zh/services) | 使用 HTTP、SSE、MCP 和 Studio,并理解默认安全边界。 |
| [Memory as File](https://reme.agentscope.io/zh/memory_as_file) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 |
| [Auto Memory](https://reme.agentscope.io/zh/auto_memory) | 保留过滤后的对话来源记录,并提炼可复用的 daily 记忆卡片。 |
| [Auto Resource](https://reme.agentscope.io/zh/auto_resource) | 导入支持的文本与图像资料,转换为可追溯来源的 daily 卡片。 |
| [Auto Dream](https://reme.agentscope.io/zh/auto_dream) 与 [Auto Link](https://reme.agentscope.io/zh/auto_link) | 将 daily 记忆整理为持续演化的 digest 节点和可读 wikilink 关系。 |
| [记忆检索](https://reme.agentscope.io/zh/memory_search) | 使用 BM25、可选向量、RRF 融合、行号范围召回和渐进式链接扩展。 |
| [Proactive](https://reme.agentscope.io/zh/proactive) | 安全读取兴趣主题,并将其接入宿主 Agent 的决策流程。 |
| [应用场景](https://reme.agentscope.io/zh/reme_scene) | 查看金融研究、研发记忆和个人知识库的完整使用示例。 |
| [框架说明](https://reme.agentscope.io/zh/framework) | 理解 Application、Job、Step、Component、service、配置和生命周期边界。 |
| [Agent 集成](https://reme.agentscope.io/zh/integrations) | 选择接口,并将 DSH、Claude Code、OpenClaw、Hermes、Codex 或其他 Agent 接入 ReMe。 |
| [DSH 插件](https://reme.agentscope.io/zh/integrations/dsh) 与 [Claude Code 插件](https://reme.agentscope.io/zh/integrations/claude-code) | 配置宿主原生召回、自动捕获、记忆整理与诊断。 |
| [CLI 与 Job API](https://reme.agentscope.io/zh/reference/cli) | 查询命令语法,以及由默认配置自动生成的 Job 参数参考。 |
| [运维与恢复](https://reme.agentscope.io/zh/operations) | 诊断服务、维护索引,并备份、迁移和恢复 workspace。 |
| [ReMe 博客](https://reme.agentscope.io/zh/reme-blog) | 了解完整产品故事、设计动机、使用示例和评测摘要。 |
## 🛠️ 常用命令
运行 `reme help` 可查看完整 job 列表。常用 workspace 与维护命令如下:
| 命令 | 作用 | | 命令 | 作用 |
|-------------------------------------------|---------------------------------------------------------------| | ----------------------------------------- | ------------------------------------------------------------- |
| `reme start` | 启动本地 ReMe 服务。 |
| `reme version` / `reme health_check` | 检查包版本和组件状态。 |
| `reme status` | 查看有状态数据组件的内存估算及进程 RSS。 | | `reme status` | 查看有状态数据组件的内存估算及进程 RSS。 |
| [`reme search`](docs/zh/memory_search.md) | 默认使用 BM25 和 wikilink 检索,启用后增加向量检索。 | | [`reme search`](https://reme.agentscope.io/zh/memory_search) | 默认使用 BM25 和 wikilink 检索,启用后增加向量检索。 |
| `reme read` / `reme write` / `reme edit` | 检查和维护 Markdown 记忆文件。 | | `reme read` / `reme write` / `reme edit` | 检查和维护 Markdown 记忆文件。 |
| `reme auto_memory` | 将对话 messages 转为 daily 记忆卡片;需要 LLM 凭证。 | | `reme traverse` / `reme graph_snapshot` | 浏览 wikilink 邻域或按类别组织的 digest 图。 |
| `reme auto_resource` | 将 `resource/` 下的文件解读为 daily 资料卡片;需要 LLM 凭证。 | | `reme chat` | 与可感知 workspace 的只读 Agent 进行流式对话;需要 LLM 凭证。 |
| `reme auto_dream` / `reme proactive` | 将 daily 记忆整理为长期 digest,并暴露值得关注的主题。 | | `reme reindex` | 基于已摄取的 chunks 重建 BM25 和 Embedding 索引。 |
| `reme reindex` | 基于已有文件重建检索和 wikilink 索引。 |
## 🤝 社区与支持 ## 🤝 社区与贡献
- **问题反馈与需求**:请先查看 [Open Issues](https://github.com/agentscope-ai/ReMe/issues);如无相关讨论,可新建 Issue - **问题反馈、需求与帮助**:请先查看 [Open Issues](https://github.com/agentscope-ai/ReMe/issues);如无相关讨论,可新建 Issue
说明背景、目标行为和影响范围。 说明背景、目标行为和影响范围。
- **代码贡献**:改动前建议阅读 [贡献指南](https://docs.agentscope.io/reme/latest/zh/contribution)。架构与扩展方式以源码、schema - **代码贡献**:改动前建议阅读[贡献指南](https://reme.agentscope.io/zh/contributing)。架构与扩展方式以源码、schema 和测试为准。
和测试为准。 - **文档贡献**:请直接更新本仓库 `docs/en/`、`docs/zh/` 或对应 package 目录中的规范源文件;文档站点会从这些文件生成。
- **文档贡献**:用户可见文档请提交到[统一文档仓库](https://github.com/agentscope-ai/docs)的 `reme/<version>/{en,zh}/` 目录。
- **提交规范**:建议使用 Conventional Commits,例如 `feat(search): add link expansion option`、 - **提交规范**:建议使用 Conventional Commits,例如 `feat(search): add link expansion option`、
`docs(zh): update quick start`。 `docs(zh): update quick start`。
- **提交前检查**:提交 PR 前请尽量运行 `pre-commit run --all-files` 和 `pytest`;如有依赖 LLM、embedding 或外部服务的测试无法运行,请在 - **提交前检查**:提交 PR 前请尽量运行 `pre-commit run --all-files` 和 `pytest`;如有依赖 LLM、embedding 或外部服务的测试无法运行,请在
PR 中说明。 PR 中说明。
- **获取帮助**:如需反馈 Bug 或功能请求,请使用 [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues);项目文档见 - **项目文档**:访问 [reme.agentscope.io](https://reme.agentscope.io)。
[https://docs.agentscope.io/reme](https://docs.agentscope.io/reme)。
### 贡献者 ### 贡献者

View file

@ -1,155 +0,0 @@
[中文版 / Chinese version](./README_ZH.md)
# ReMe Benchmarks
Reproduction guide for the two memory benchmarks shipped with ReMe:
- **LongMemEval** — long-term memory over multi-session chat histories.
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
per-item workspace, answer probing questions via an agentic (ReAct) mode,
then score answers with an LLM-as-judge.
## 1. Prerequisites
Install ReMe with dev + core extras (Python 3.11+):
```bash
pip install -e ".[dev,core]"
```
Configure model credentials in a project-root `.env` file (copied from `example.env`).
The runners auto-load `.env` from the repository root. Required variables typically include:
```bash
LLM_API_KEY=...
LLM_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_BASE_URL=...
```
Model names and component wiring live in the ReMe configs referenced by each benchmark
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
## 2. Download Datasets
Each benchmark keeps its own data under its directory:
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
All three are excluded from Git.
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
modify `BASE_URL` in `download.py`):
```bash
cd benchmark/longmemeval
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
```
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
`topics/` and other subdirectories.
## 3. Run LongMemEval
From the repository root:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
```
### Pipeline
1. Load the dataset (ground truth is embedded in the data file).
2. For each item, create an isolated workspace and ingest sessions in chronological order.
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
4. Answer each question via agentic (ReAct) mode.
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
### Key config — `benchmark/longmemeval/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
| `dataset.question_types` | Filter by question type; empty = all. |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
| `reme.config` | ReMe config used (`lme.yaml`). |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
## 4. Run BEAM
From the repository root:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # quiet
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
```
### Pipeline
1. For each case, load `chat.json` and convert each batch into a ReMe session.
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
3. Answer each probing question via agentic (ReAct) mode.
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
### Key config — `benchmark/beam/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
| `reme.config` | ReMe config used (`beam.yaml`). |
| `output.dir` | Results directory (`benchmark/beam/results`). |
## 5. Outputs & Logs
- **Results**: JSON files written to `output.dir`
(`results_<timestamp>.json` for LongMemEval,
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
accuracy/score is also printed to the console.
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
per worker process).
## 6. Stopping a Run
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
```bash
bash benchmark/kill.sh <PID>
```
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
`SIGKILL` for any process that does not exit within 5 seconds.
## 7. Reference Results
Recorded evaluation results are available in:
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

View file

@ -1,149 +0,0 @@
# ReMe 评测复现说明
ReMe 内置两个记忆能力评测基准的复现指南:
- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。
- **BEAM** —— 面向长上下文对话场景、基于评分细则(rubric)打分的记忆能力评测。
每个基准都会运行完整的端到端流程:将会话摄入独立的按条目隔离的工作区,
以 agentic(ReAct)模式回答探测问题,
最后由 LLM-as-judge 对答案进行打分。
## 1. 环境准备
安装 ReMe 及 dev + core 附加依赖(Python 3.11+):
```bash
pip install -e ".[dev,core]"
```
在项目根目录配置 `.env`(可从 `example.env` 复制),运行脚本会自动从仓库根目录加载 `.env`。
通常需要以下变量:
```bash
LLM_API_KEY=...
LLM_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_BASE_URL=...
```
模型名称与组件装配位于各基准引用的 ReMe 配置中
(`reme/config/lme.yaml` 与 `reme/config/beam.yaml`)。
## 2. 下载数据集
每个基准的数据都存放在各自目录下:`benchmark/<name>/dataset`(输入数据)、
`benchmark/<name>/workspaces`(按条目隔离的记忆工作区)、
`benchmark/<name>/results`(评测输出)。三者均不纳入 Git 版本管理。
**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本,数据托管在 HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(下载脚本经 hf-mirror.com 镜像源获取,如需更换源请修改 `download.py` 中的 `BASE_URL`):
```bash
cd benchmark/longmemeval
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json,已存在则自动跳过
```
**BEAM**(公开仓库,clone 到 `benchmark/beam/dataset/` 下):
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后,`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
## 3. 运行 LongMemEval
在仓库根目录执行:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # 安静模式:仅评测级日志
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
### 流程
1. 加载数据集(ground truth 已内嵌在数据文件中)。
2. 为每个条目创建独立工作区,按时间顺序摄入会话。
3. 当相邻会话跨越配置的时刻(默认 23:00)时触发 `auto_dream`。
4. 以 agentic(ReAct)模式回答每个问题。
5. 通过 `answer_judge` 任务对答案做二元(yes/no)评判,并输出各类型准确率。
### 关键配置 —— `benchmark/longmemeval/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.path` | 待评测的数据集文件(如 `longmemeval_s_reme_cleaned.json`),已包含 ground truth。 |
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
| `dataset.question_types` | 按问题类型过滤,空表示全部。 |
| `dataset.workspace_root` | 条目工作区根目录(`benchmark/longmemeval/workspaces/longmemeval-s`)。 |
| `evaluation.num_workers` | `0` = 自动(cpu-2),`1` = 串行,`>1` = 并行。 |
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
| `reme.config` | 使用的 ReMe 配置(`lme.yaml`)。 |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
| `output.dir` | 结果目录(`benchmark/longmemeval/results`)。 |
## 4. 运行 BEAM
在仓库根目录执行:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # 安静模式
python benchmark/beam/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
### 流程
1. 为每个 case 加载 `chat.json`,将每个 batch 转换为一个 ReMe 会话。
2. 按时间顺序将会话摄入独立工作区,随后执行 `digest_update`。
3. 以 agentic(ReAct)模式回答每个探测问题。
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分,并输出各类型平均分。
### 关键配置 —— `benchmark/beam/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.beam_root` | BEAM 数据集根目录(`benchmark/beam/dataset/BEAM`)。 |
| `dataset.chat_size` | 运行的变体:`100K` / `500K` / `1M` / `10M`。 |
| `dataset.case_ids` | 指定 case(如 `["1","2"]`),空表示全部。 |
| `dataset.start_index` / `num_items` | case 分页(`num_items` 为 `0` 表示全部)。 |
| `dataset.workspace_root` | case 工作区根目录(`benchmark/beam/workspaces/beam`)。 |
| `evaluation.num_workers` | `0` = 自动,`1` = 串行,`>1` = 并行。 |
| `reme.config` | 使用的 ReMe 配置(`beam.yaml`)。 |
| `output.dir` | 结果目录(`benchmark/beam/results`)。 |
## 5. 输出与日志
- **结果**:JSON 文件写入 `output.dir`
(LongMemEval 为 `results_<timestamp>.json`,
BEAM 为 `results_<chat_size>_<timestamp>.json`)。同时控制台会打印含各类型
准确率/分数的汇总。
- **日志**:当 `output.log_to_file` 开启时,每次运行的日志写入
`logs/<log_prefix>_<timestamp>/`(包含一个 `runner.log` 及每个 worker 进程的
`worker-<pid>.log`)。
## 6. 终止运行
并行运行会派生进程树。若要干净地终止某次运行及其全部 worker:
```bash
bash benchmark/kill.sh <PID>
```
该脚本会先向整个进程树发送 `SIGTERM` 优雅终止,对 5 秒内未退出的进程再升级为 `SIGKILL`。
## 7. 参考结果
已记录的评测结果见:
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

137
benchmark/beam/README.md Normal file
View file

@ -0,0 +1,137 @@
[中文版 / Chinese version](./README_ZH.md)
# BEAM Benchmark
BEAM is a benchmark for **memory capability over long-context chat cases**. Each
case contains a very long chat history split into batches; ReMe converts each
batch into a session, ingests them in chronological order, then answers probing
questions via an agentic (ReAct) mode. Answers are scored with BEAM's
rubric-based `answer_judge` job, which produces both a graded score and a binary
verdict, and per-type averages are reported.
BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so
memory systems can be stressed at different context lengths. Question types
include abstention, contradiction resolution, event ordering, information
extraction, instruction following, knowledge update, multi-session reasoning,
preference following, summarization, and temporal reasoning.
Install ReMe and the BEAM plugin in editable mode from the repository root:
```bash
python -m pip install -e ".[as]"
reme plugins install ./plugins/beam --editable
reme plugins install ./plugins/beam-judge --editable
reme plugins validate beam
```
The runner explicitly enables the installed `beam` plugin and combines its defaults with
ReMe's built-in `benchmark` preset. Editable installation keeps changes under
[`plugins/beam`](../../plugins/beam/README.md) visible without reinstalling the plugin.
Custom application config paths still work through `reme.config` and can use `extends: benchmark`.
This directory continues to own the runner, evaluation settings, dataset and outputs.
Model credentials use the environment variables declared by the shared benchmark configuration.
## 1. Get the Dataset
BEAM is a public repository, cloned into `benchmark/beam/dataset/`:
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
`topics/` and other subdirectories.
## 2. Run
From the repository root:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # quiet
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
```
## 3. Pipeline
1. For each case, load `chat.json` and convert each batch into a ReMe session.
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
3. Answer each probing question via agentic (ReAct) mode.
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
## 4. Key config — `benchmark/beam/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
| `reme.config` | ReMe config used (`benchmark`). |
| `output.dir` | Results directory (`benchmark/beam/results`). |
## 5. Outputs
Results are JSON files written to `output.dir` as
`results_<chat_size>_<timestamp>.json`, with a per-type score summary also
printed to the console. Logging conventions are shared across benchmarks — see
the [top-level README](../README.md#outputs--logs).
## 6. Reference Results
> The results below use the longmemeval-version prompt.
### 100K
agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory)
(2026-08-05, 20 cases / 400 Qs, total 46.0 min)
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction average token consumption (default agent, full build over 20 cases):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build
(2026-08-05, 35 cases / 700 Qs, total 459.2 min)
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction average token consumption (default agent, full build over 35 cases):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

132
benchmark/beam/README_ZH.md Normal file
View file

@ -0,0 +1,132 @@
# BEAM 评测
[English version](./README.md)
BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个
batch 的超长对话;ReMe 将每个 batch 转换为一个会话,按时间顺序摄入后,以 agentic(ReAct)
模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分,同时给出分级分数与二元
判定,并输出各类型平均分。
BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`,可在不同上下文长度下
压测记忆系统。题型包括 abstention(拒答)、contradiction resolution(矛盾消解)、event
ordering(事件排序)、information extraction(信息抽取)、instruction following(指令遵循)、
knowledge update(知识更新)、multi-session reasoning(多会话推理)、preference following
(偏好遵循)、summarization(摘要)与 temporal reasoning(时间推理)。
在仓库根目录以 editable 模式安装 ReMe 和 BEAM 插件:
```bash
python -m pip install -e ".[as]"
reme plugins install ./plugins/beam --editable
reme plugins install ./plugins/beam-judge --editable
reme plugins validate beam
```
runner 显式启用已安装的 `beam` 插件,并将插件默认配置与 ReMe 内置的 `benchmark` 配置组合。
editable 安装会让 [`plugins/beam`](../../plugins/beam/README_ZH.md) 下的源码修改直接生效,无需重复安装。
本目录继续保留评测参数、数据集及输出。自定义完整应用配置路径仍可通过 `reme.config` 指定,
并可使用 `extends: benchmark`。
模型凭据通过公共 benchmark 配置中声明的环境变量设置。
## 1. 获取数据集
BEAM 是公开仓库,clone 到 `benchmark/beam/dataset/` 下:
```bash
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后,`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
## 2. 运行
在仓库根目录执行:
```bash
python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q # 安静模式
python benchmark/beam/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
## 3. 流程
1. 为每个 case 加载 `chat.json`,将每个 batch 转换为一个 ReMe 会话。
2. 按时间顺序将会话摄入独立工作区,随后执行 `digest_update`。
3. 以 agentic(ReAct)模式回答每个探测问题。
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分,并输出各类型平均分。
## 4. 关键配置 —— `benchmark/beam/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.beam_root` | BEAM 数据集根目录(`benchmark/beam/dataset/BEAM`)。 |
| `dataset.chat_size` | 运行的变体:`100K` / `500K` / `1M` / `10M`。 |
| `dataset.case_ids` | 指定 case(如 `["1","2"]`),空表示全部。 |
| `dataset.start_index` / `num_items` | case 分页(`num_items` 为 `0` 表示全部)。 |
| `dataset.workspace_root` | case 工作区根目录(`benchmark/beam/workspaces/beam`)。 |
| `evaluation.num_workers` | `0` = 自动,`1` = 串行,`>1` = 并行。 |
| `reme.config` | 使用的 ReMe 配置(`benchmark`)。 |
| `output.dir` | 结果目录(`benchmark/beam/results`)。 |
## 5. 输出
结果以 JSON 文件写入 `output.dir`,文件名为 `results_<chat_size>_<timestamp>.json`,
同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用,见
[总说明](../README_ZH.md#输出与日志)。
## 6. 参考结果
> 以下结果使用 longmemeval 版本的 prompt。
### 100K
agentscope==2.0.4.post1,conda reme 环境,20 并发,eval-only(复用已构建 memory)
(2026-08-05,20 cases / 400 Qs,总耗时 46.0 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction 平均 token 消耗(default agent,20 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1,conda reme 环境,20 并发,全量构建 memory
(2026-08-05,35 cases / 700 Qs,总耗时 459.2 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction 平均 token 消耗(default agent,35 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

View file

@ -14,7 +14,8 @@ evaluation:
compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression
reme: reme:
config: "beam.yaml" # reme config (in reme/config/) config: "benchmark" # shared ReMe benchmark preset
plugins: [beam, beam-judge]
output: output:
dir: "benchmark/beam/results" dir: "benchmark/beam/results"

View file

@ -29,7 +29,7 @@ import yaml
from dotenv import load_dotenv from dotenv import load_dotenv
# Load .env from project root # Load .env from project root
_PROJECT_ROOT = Path(__file__).parent.parent.parent _PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
load_dotenv(_PROJECT_ROOT / ".env") load_dotenv(_PROJECT_ROOT / ".env")
# Workspace root — read from config.yaml (dataset.workspace_root) # Workspace root — read from config.yaml (dataset.workspace_root)
@ -151,6 +151,23 @@ def load_eval_config(config_path: str | None = None) -> dict:
return yaml.safe_load(raw) return yaml.safe_load(raw)
def create_reme_app(config: str = "benchmark", **overrides):
"""Create an app with the BEAM candidate and judge plugins enabled.
Plugin discovery remains environment-based; editable installation keeps local
plugin source changes visible to every multiprocessing worker.
"""
from reme import Application
from reme.config import resolve_app_config
enabled_plugins = list(overrides.pop("plugins", ()) or ())
for plugin in ("beam", "beam-judge"):
if plugin not in enabled_plugins:
enabled_plugins.append(plugin)
app_config = resolve_app_config(config=config, plugins=enabled_plugins, **overrides)
return Application(**app_config)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# BEAM data loading # BEAM data loading
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -319,8 +336,6 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
Returns: Returns:
A results dict with all questions, answers, and judgments. A results dict with all questions, answers, and judgments.
""" """
from reme import Application
from reme.config import resolve_app_config
dataset_cfg = eval_config["dataset"] dataset_cfg = eval_config["dataset"]
chat_size = dataset_cfg["chat_size"] chat_size = dataset_cfg["chat_size"]
@ -375,15 +390,15 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
force_init=True, force_init=True,
) )
cfg = resolve_app_config( app = create_reme_app(
config=eval_config["reme"]["config"], config=eval_config["reme"]["config"],
plugins=eval_config["reme"].get("plugins", ()),
workspace_dir=workspace_dir, workspace_dir=workspace_dir,
log_to_console=output_cfg.get("log_to_console", True), log_to_console=output_cfg.get("log_to_console", True),
log_to_file=output_cfg.get("log_to_file", False), log_to_file=output_cfg.get("log_to_file", False),
enable_logo=False, enable_logo=False,
) )
app = Application(**cfg)
await app.start() await app.start()
from reme.utils.evaluation_interface import check_agent_token_usage # noqa: E402 from reme.utils.evaluation_interface import check_agent_token_usage # noqa: E402

View file

@ -0,0 +1,110 @@
[中文版 / Chinese version](./README_ZH.md)
# LongMemEval Benchmark
LongMemEval is a benchmark for **long-term memory over multi-session chat
histories**. Each item provides a chronologically ordered set of chat sessions
between a user and an assistant, followed by a probing question whose answer is
only recoverable by reasoning over the user-owned memory. ReMe ingests the
sessions into an isolated per-item workspace, answers the question via an
agentic (ReAct) mode, and scores the answer with an LLM-as-judge.
Question types include single-session (user / assistant / preference),
multi-session reasoning, knowledge update, and temporal reasoning.
Install ReMe and the LongMemEval plugin in editable mode from the repository root:
```bash
python -m pip install -e ".[as]"
reme plugins install ./plugins/lme --editable
reme plugins install ./plugins/lme-judge --editable
reme plugins validate lme
```
The runner explicitly enables the installed `lme` plugin and combines its defaults with
ReMe's built-in `benchmark` preset. Editable installation keeps changes under
[`plugins/lme`](../../plugins/lme/README.md) visible without reinstalling the plugin.
Custom application config paths still work through `reme.config` and can use `extends: benchmark`.
This directory continues to own the runner, evaluation settings, dataset and outputs.
Model credentials use the environment variables declared by the shared benchmark configuration.
## 1. Get the Dataset
ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2).
The download script fetches it via the hf-mirror.com mirror; to use a different
mirror, modify `BASE_URL` in [`download.py`](./download.py).
```bash
cd benchmark/longmemeval
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
```
Ground truth is embedded in the data file.
## 2. Run
From the repository root:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
```
## 3. Pipeline
1. Load the dataset (ground truth is embedded in the data file).
2. For each item, create an isolated workspace and ingest sessions in chronological order.
3. If a custom application configuration enables `auto_dream`, trigger it when sessions cross the configured hour
(default 23:00). The packaged preset leaves it disabled.
4. Answer each question via agentic (ReAct) mode.
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
## 4. Key config — `benchmark/longmemeval/config.yaml`
| Key | Meaning |
| --- | --- |
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
| `dataset.question_types` | Filter by question type; empty = all. |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
| `reme.config` | ReMe config used (`benchmark`). |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
## 5. Outputs
Results are JSON files written to `output.dir` as `results_<timestamp>.json`,
with a per-type accuracy summary also printed to the console. Logging
conventions are shared across benchmarks — see the
[top-level README](../README.md#outputs--logs).
## 6. Reference Results
### cleaned-s
**Basic settings**
1. Modified auto-memory prompt, auto-dream disabled.
2. All sessions in reme-memory are strictly earlier than the question time.
**Results**
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory)
(2026-08-06, 500 items, total 10.0 min)
| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |

View file

@ -0,0 +1,103 @@
# LongMemEval 评测
[English version](./README.md)
LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间
顺序排列的用户与助手之间的会话,以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe
将会话摄入按条目隔离的工作区,以 agentic(ReAct)模式回答问题,最后由 LLM-as-judge 打分。
题型包括单会话(user / assistant / preference)、多会话推理、知识更新与时间推理等。
在仓库根目录以 editable 模式安装 ReMe 和 LongMemEval 插件:
```bash
python -m pip install -e ".[as]"
reme plugins install ./plugins/lme --editable
reme plugins install ./plugins/lme-judge --editable
reme plugins validate lme
```
runner 显式启用已安装的 `lme` 插件,并将插件默认配置与 ReMe 内置的 `benchmark` 配置组合。
editable 安装会让 [`plugins/lme`](../../plugins/lme/README_ZH.md) 下的源码修改直接生效,无需重复安装。
本目录继续保留评测参数、数据集及输出。自定义完整应用配置路径仍可通过 `reme.config` 指定,
并可使用 `extends: benchmark`。
模型凭据通过公共 benchmark 配置中声明的环境变量设置。
## 1. 获取数据集
ReMe 仅使用 **cleaned-S** 版本,数据托管在 HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。
下载脚本经 hf-mirror.com 镜像源获取,如需更换源请修改 [`download.py`](./download.py) 中的
`BASE_URL`。
```bash
cd benchmark/longmemeval
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json,已存在则自动跳过
```
ground truth 已内嵌在数据文件中。
## 2. 运行
在仓库根目录执行:
```bash
python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q # 安静模式:仅评测级日志
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区,仅执行查询 + 评判
```
## 3. 流程
1. 加载数据集(ground truth 已内嵌在数据文件中)。
2. 为每个条目创建独立工作区,按时间顺序摄入会话。
3. 若自定义应用配置启用了 `auto_dream`,在相邻会话跨越配置时刻(默认 23:00)时触发;插件预设保持关闭。
4. 以 agentic(ReAct)模式回答每个问题。
5. 通过 `answer_judge` 任务对答案做二元(yes/no)评判,并输出各类型准确率。
## 4. 关键配置 —— `benchmark/longmemeval/config.yaml`
| 配置项 | 含义 |
| --- | --- |
| `dataset.path` | 待评测的数据集文件(如 `longmemeval_s_reme_cleaned.json`),已包含 ground truth。 |
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
| `dataset.question_types` | 按问题类型过滤,空表示全部。 |
| `dataset.workspace_root` | 条目工作区根目录(`benchmark/longmemeval/workspaces/longmemeval-s`)。 |
| `evaluation.num_workers` | `0` = 自动(cpu-2),`1` = 串行,`>1` = 并行。 |
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
| `reme.config` | 使用的 ReMe 配置(`benchmark`)。 |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
| `output.dir` | 结果目录(`benchmark/longmemeval/results`)。 |
## 5. 输出
结果以 JSON 文件写入 `output.dir`,文件名为 `results_<timestamp>.json`,
同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用,见
[总说明](../README_ZH.md#输出与日志)。
## 6. 参考结果
### cleaned-s
**基础设置**
1. 使用修改后的 auto-memory prompt,关闭 auto-dream 机制
2. reme-memory 中的全部 session 的时间一定早于 question 的时间
**结果**
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only(复用预构建记忆)
(2026-08-06,500 题,总计 10.0 min)
| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |

View file

@ -10,7 +10,7 @@ dataset:
workspace_root: "benchmark/longmemeval/workspaces/longmemeval-s" # workspace root for item workspaces workspace_root: "benchmark/longmemeval/workspaces/longmemeval-s" # workspace root for item workspaces
evaluation: evaluation:
# LLM-as-judge uses the 'judge' as_llm component defined in lme.yaml # LLM-as-judge uses the 'judge' as_llm component defined in benchmark.yaml
# Model and credentials are configured there (reading from .env) # Model and credentials are configured there (reading from .env)
# Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml # Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml
num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel
@ -18,7 +18,8 @@ evaluation:
compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression
reme: reme:
config: "lme.yaml" # reme config to use (in reme/config/) config: "benchmark" # runner enables both plugins below
plugins: [lme, lme-judge]
# Dream trigger: when gap between consecutive sessions crosses this hour (23:00) # Dream trigger: when gap between consecutive sessions crosses this hour (23:00)
dream_trigger_hour: 23 dream_trigger_hour: 23
# Dream scan_days for each trigger # Dream scan_days for each trigger

View file

@ -0,0 +1,76 @@
#!/bin/bash
# 杀死指定进程及其所有子进程
# Usage: bash kill.sh <PID>
if [ -z "$1" ]; then
echo "Usage: bash kill.sh <PID>"
echo " 杀死指定进程及其所有子进程"
exit 1
fi
PID=$1
# 检查进程是否存在
if ! kill -0 "$PID" 2>/dev/null; then
echo "进程 $PID 不存在"
exit 1
fi
# 递归收集所有子进程(包括子进程的子进程)
collect_children() {
local parent=$1
local children
children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
for child in $children; do
collect_children "$child"
done
echo "$parent"
}
# 收集进程树(子进程在前,父进程在后,保证先杀子再杀父)
PROCESS_TREE=$(collect_children "$PID")
TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ')
echo "进程树(共 $TOTAL 个进程):"
while read -r p; do
cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80)
printf " PID=%-8s %s\n" "$p" "$cmd"
done <<< "$PROCESS_TREE"
# 先 SIGTERM 优雅终止
echo ""
echo "发送 SIGTERM..."
while read -r p; do
kill "$p" 2>/dev/null
done <<< "$PROCESS_TREE"
# 等待最多 5 秒
for i in $(seq 1 5); do
alive=false
while read -r p; do
if kill -0 "$p" 2>/dev/null; then
alive=true
fi
done <<< "$PROCESS_TREE"
if [ "$alive" = false ]; then
break
fi
sleep 1
done
# 检查是否还有残留,强制 SIGKILL
remaining=false
while read -r p; do
if kill -0 "$p" 2>/dev/null; then
remaining=true
fi
done <<< "$PROCESS_TREE"
if [ "$remaining" = true ]; then
echo "部分进程未响应,发送 SIGKILL..."
while read -r p; do
kill -9 "$p" 2>/dev/null
done <<< "$PROCESS_TREE"
fi
echo "已终止进程树(根 PID=$PID,共 $TOTAL 个进程)"

View file

@ -28,7 +28,7 @@ import yaml
from dotenv import load_dotenv from dotenv import load_dotenv
# Load .env from project root # Load .env from project root
_PROJECT_ROOT = Path(__file__).parent.parent.parent _PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
load_dotenv(_PROJECT_ROOT / ".env") load_dotenv(_PROJECT_ROOT / ".env")
# Workspace root for evaluation items — read from config.yaml (dataset.workspace_root) # Workspace root for evaluation items — read from config.yaml (dataset.workspace_root)
@ -150,6 +150,23 @@ def load_eval_config(config_path: str | None = None) -> dict:
return yaml.safe_load(raw) return yaml.safe_load(raw)
def create_reme_app(config: str = "benchmark", **overrides):
"""Create an app with the LongMemEval candidate and judge plugins enabled.
Plugin discovery remains environment-based; editable installation keeps local
plugin source changes visible to every multiprocessing worker.
"""
from reme import Application
from reme.config import resolve_app_config
enabled_plugins = list(overrides.pop("plugins", ()) or ())
for plugin in ("lme", "lme-judge"):
if plugin not in enabled_plugins:
enabled_plugins.append(plugin)
app_config = resolve_app_config(config=config, plugins=enabled_plugins, **overrides)
return Application(**app_config)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Date utilities # Date utilities
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -257,8 +274,6 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl
using the existing workspace. Useful for re-evaluating different query using the existing workspace. Useful for re-evaluating different query
configurations without re-ingesting sessions. configurations without re-ingesting sessions.
""" """
from reme import Application
from reme.config import resolve_app_config
from reme.utils.evaluation_interface import track_agent_token_usage, track_job_counts from reme.utils.evaluation_interface import track_agent_token_usage, track_job_counts
reme_cfg = eval_config["reme"] reme_cfg = eval_config["reme"]
@ -325,15 +340,15 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl
force_init=True, force_init=True,
) )
cfg = resolve_app_config( app = create_reme_app(
config=reme_cfg["config"], config=reme_cfg["config"],
plugins=reme_cfg.get("plugins", ()),
workspace_dir=workspace_dir, workspace_dir=workspace_dir,
log_to_console=output_cfg.get("log_to_console", True), log_to_console=output_cfg.get("log_to_console", True),
log_to_file=output_cfg.get("log_to_file", False), log_to_file=output_cfg.get("log_to_file", False),
enable_logo=False, enable_logo=False,
) )
app = Application(**cfg)
await app.start() await app.start()
try: try:

14
benchmark/pibench/.gitignore vendored Normal file
View file

@ -0,0 +1,14 @@
# 含真实 API key,绝不入库
env.sh
# 运行时产物(含对话内容,勿入库)
logs/
outputs/
reme_workspace/
nanobot_workspace/
# 数据符号链接(指向外部 π-Bench 仓库)
data
__pycache__/
*.pyc

327
benchmark/pibench/README.md Normal file
View file

@ -0,0 +1,327 @@
[中文版 / Chinese version](./README_ZH.md)
# π-Bench Evaluation Suite
A glue layer that connects the **ReMe agent (with persistent memory)** to
**π-Bench** (Proactive Personal Assistant Benchmark). This directory contains
only the minimal code and configuration needed for the integration: the
π-Bench framework (`src/`), evaluation data (`data/`), the AppWorld tool
environment, and ReMe itself are all **external third-party dependencies**,
referenced in place via symlink and environment variables and never bundled
with this suite.
- π-Bench: https://github.com/Simplified-Reasoning/Pi-Bench (arXiv: 2605.14678)
- ReMe: the root of the ReMe repository this suite lives in (recommended
location: `ReMe/benchmark/pibench/`)
## 1. Architecture
```
π-Bench runner (src.main --mode run)
│ user_agent (simulated-user LLM) walks data/{persona}/episode.yaml
│ task by task, chatting with the agent over multiple turns and judging
│ hidden intents (PROC) during the run phase
▼
test server (π-Bench scripts/test_server.py, HTTP long-polling)
▲ /send │ /poll
│ ▼
bridge_reme.py ──────────────► ReMe Application (embedded as a library)
│ ├─ agent_wrapper: agent under test (AgentScope)
│ ├─ jobs: search / auto_memory / daily_write
│ └─ workspace: reme_workspace/{persona}/
│ (isolated persistent memory per persona)
└──── MCP ────► AppWorld MCP ────► AppWorld APIs (tool/app environment)
π-Bench runner (src.main --mode eval)
judger (judge LLM) reads the traces and scores each checklist item (COMP)
```
Key points:
- The bridge runs on **ReMe's own venv python** and uses ReMe as a library
(`resolve_app_config` + `Application`); **no ReMe source modification** is
required.
- Every incoming user message automatically triggers a ReMe memory `search`
and injects the matched memories (tuning knobs in §8); on task end (reset)
the session is distilled into daily notes by `auto_memory`.
- Tool calls executed by the agent (AppWorld MCP + ReMe job tools) are
captured per turn into the trace as `tool_steps`, so π-Bench
`tools_evaluation_path` scripts can score tool behavior (§7).
- π-Bench's `data/`, `src/` and AppWorld are not part of this suite; install
π-Bench first (§3.1).
## 2. Directory layout
```
pibench/
├── README.md / README_ZH.md # this document (English / Chinese)
├── env.sh.example # environment template (copy to env.sh, fill TODOs)
├── bridge_reme.py # ReMe ↔ test server bridge (memory inject/save,
│ # profile injection, tool-trace capture)
├── run_persona.sh # full pipeline for ONE persona (5 services + run + eval)
├── run_all.sh # batch over 5 personas (fresh/resume, default parallel=2)
├── resume.py # checkpoint resume: completion detection + surgical
│ # cleanup of interrupted tasks' residual memory
├── fix_trace_logs.py # run outputs → ~/.nanobot/trace_logs conversion,
│ # merging tool sidecars into turn files (pre-eval)
├── .gitignore # excludes env.sh and all runtime artifacts
└── config/
├── models/reme.yaml # runner model config (model_id=reme)
└── bench/evaluation/trace_history.yaml # trace render policy (shipped with
# the suite; passed via --history-config-path)
```
Generated at runtime (all git-ignored): `data` (symlink), `logs/`, `outputs/`,
`reme_workspace/`, `nanobot_workspace/`.
## 3. Prerequisites (third-party, install first)
### 3.1 π-Bench repository (with AppWorld)
```bash
git clone https://github.com/Simplified-Reasoning/Pi-Bench.git <pi-bench-dir>
cd <pi-bench-dir>
python3.11 -m venv .venv # scripts expect exactly this venv name
source .venv/bin/activate
pip install -e . # pibench runner (src.main)
bash scripts/setup_appworld.sh # install AppWorld and download its data (large)
```
Post-install sanity checks:
```bash
ls data/ # should contain researcher marketer pharmacist law_trainee Financier
.venv/bin/python -c "import src" && echo OK
.venv/bin/appworld --help >/dev/null && echo OK
```
### 3.2 ReMe repository
```bash
cd <reme-dir> # ReMe repository root (contains the reme/ package)
python3.11 -m venv .venv # scripts expect exactly this venv name
source .venv/bin/activate
pip install -e . # or ReMe's own install flow; `import reme` must work
```
Sanity check: `.venv/bin/python -c "import reme; print('ok')"`
## 4. Install this suite (step by step)
1. **Place the suite** (recommended inside the ReMe repo so `REME_DIR` is
inferred automatically):
```bash
cp -r pibench <reme-dir>/benchmark/pibench
cd <reme-dir>/benchmark/pibench
```
If placed elsewhere, set `REME_DIR` explicitly in env.sh later.
2. **Create the environment file and fill in the custom parameters**:
```bash
cp env.sh.example env.sh
```
Open `env.sh`; required items (marked TODO):
| Variable | Description |
|---|---|
| `PI_BENCH_ROOT` | π-Bench repo root (contains `src/` `data/` `.venv` `third_party/appworld`) |
| `USER_API_KEY` | API key of the simulated-user LLM (run phase, hidden-intent judging) |
| `JUDGER_API_KEY` | API key of the judger LLM (eval phase, checklist scoring) |
| `BRAVE_SEARCH_API_KEY` | optional; for the agent's web_search tool, `dummy` when unused |
Optional tuning: `REME_MODEL_NAME` (base model of the agent under test),
`REME_DIR`, `REME_LLM_BASE_URL` (default: DashScope OpenAI-compatible
endpoint).
3. **Link the evaluation data** (referenced in place, never copied):
```bash
ln -s "$PI_BENCH_ROOT/data" data
```
4. **(Optional) adjust model config** `config/models/reme.yaml`:
- `user_agent.model` / `judger.model`: model names for the simulated user
and the judger (literal values; π-Bench only expands `${ENV}` in
base_url/api_key).
- `run.turn_timeout`, `max_tool_iterations`, etc. as needed.
5. **Smoke check** (does not start the evaluation):
```bash
bash -n run_all.sh && bash -n run_persona.sh
source env.sh && "$REME_DIR/.venv/bin/python" -c "import reme; print('reme ok')"
```
## 5. Run the evaluation
> ⚠️ For long runs use `screen`, **not nohup** (nohup loses the permission
> context in sandboxed/restricted environments and breaks child processes).
```bash
# Full official run: wipe ALL personas' memory/outputs/traces first (default
# fresh mode, parallel=2)
mkdir -p logs # on a fresh deployment logs/ does not exist yet
screen -dmS pibench_suite bash -c "cd $(pwd) && bash run_all.sh > logs/run_all_master.log 2>&1"
# Checkpoint continuation (after an interruption; no wipe, completed tasks skipped)
bash run_all.sh --resume
# Other usages
bash run_all.sh --parallel 1 # sequential
bash run_all.sh --resume --skip-eval # run phase only
bash run_persona.sh researcher # single persona (default --resume semantics)
bash run_persona.sh researcher --fresh
```
Time reference: 5 personas × 20 tasks, parallel=2, fresh full run ≈ 12–14 hours.
`run_all.sh` exits non-zero when any persona fails, so upstream automation
cannot mistake a partially failed suite run for a success.
## 6. Port allocation (parallel personas never collide)
| persona | AppWorld API | AppWorld MCP | Test Server | ReMe internal service |
|-------------|------|-------|------|-------|
| marketer | 9001 | 10001 | 9998 | 18766 |
| law_trainee | 9002 | 10002 | 9997 | 18767 |
| pharmacist | 9003 | 10003 | 9996 | 18768 |
| researcher | 9004 | 10004 | 9995 | 18765 |
| Financier | 9005 | 10005 | 9994 | 18769 |
## 7. Outputs and scores
- **Results**: `outputs/reme/{persona}/{task}/eval/results/*_result.json`
- `overall_average_score`: checklist completeness (COMP; the judger scores
each criterion YES/NO, weighted across dependency groups)
- `overall_proactiveness_average_score`: proactiveness (PROC; the
user_agent judges hidden-intent coverage during the run phase; each task
file also carries the global average)
- **Traces**: `~/.nanobot/trace_logs/reme/{persona}/{task}/...` (the scoring
input of the eval phase)
- **Logs**: `logs/` (`suite_<persona>.log` per persona; `bridge_*`,
`runner_run/eval_*`, `appworld_*`, `test_server_*` per service)
- **Memory store**: `reme_workspace/{persona}/` (daily/digest notes, raw
session dialogs, BM25 index, etc.; persistent across runs, wiped only in
fresh mode)
Score summary:
```bash
grep -h "overall_average_score\|overall_proactiveness" \
outputs/reme/*/*/eval/results/*_result.json | head
```
### Tool-trace capture (tools_evaluation support)
Some tasks define `objectives.tools_evaluation_path`: Python scripts that
score tool behavior (e.g. "the temporary Todoist board was created and
removed"). They need the executed tool calls in the trace. The pipeline:
1. During `reply()`, the bridge reads the persisted AgentScope session state
after each turn and extracts the new `tool_call` / `tool_result` blocks
(tool name, arguments, result).
2. Records are appended to
`outputs/reme/{persona}/{task}/history/{ts}-tools.jsonl`, tagged with the
turn number; AgentScope MCP names (`mcp__AppWorld__<tool>`) are normalized
to the π-Bench convention (`mcp_appworld_<tool>`).
3. `fix_trace_logs.py` pairs each `{ts}-messages.jsonl` run with the
temporally closest tools sidecar and merges the records into the generated
`turn_N.json` files under the `tool_steps` key — one of the two
tool-history formats understood by π-Bench's `collect_tool_history()`.
4. The eval phase then feeds `tool_steps` to both the tools_evaluation
scripts and the rendered `<tool_trace_extracts>` seen by the judger.
## 8. Memory mechanism (core design of this suite)
- **Persona isolation**: each persona has its own workspace
(`reme_workspace/{persona}/`); the bridge takes an exclusive
`.bridge.lock` on it at startup, so two bridges can never share one memory
store, and one persona's memory search can never reach another's memories.
- **Writes**: on task end (runner sends reset), the session is distilled by
the `auto_memory` job into daily notes and indexed by the background
watcher (BM25). Saves are non-blocking background tasks; the first message
of a new session waits for in-flight writes before searching.
- **Reads**: on every incoming user message the bridge runs one `search` and
injects matched memories (`[Relevant memories from previous sessions]`
prefix); without matches the message passes through unchanged. Retrieval
tuning (bridge CLI flags, adjustable in run_persona.sh):
- `--search-limit 3`: at most 3 memory chunks injected per message;
- `--search-min-score 2.0`: weak BM25 hits are filtered out;
- `tool_context_id` rotates per task: chunks already injected within the
same task are not re-injected (ReMe's seen-chunk dedup, 24h TTL); normal
recall resumes after task boundaries.
- **No self-leakage**: the in-progress session is not in the store yet
(saves happen on reset), so a task can never retrieve its own unfinished
content.
- The agent also holds `search`/`daily_write` tools and can retrieve/record
proactively.
- **System prompt**: `bridge_reme.py:build_system_prompt()` embeds the
HIDDEN-NEEDS protocol (proactiveness-oriented) and injects the persona
profile from `data/{persona}/profile.yaml` into every turn's system prompt.
## 9. Checkpoint resume and memory-cleanup semantics
- **Completion detection** (resume.py): scans
`outputs/reme/{persona}/**/history/*-log.jsonl` and
`outputs/reme/{persona}/run/*-log.jsonl` for
`Task finished task_id=X status=Y`. The status with the **newest event
timestamp** wins per task (record `timestamp`, falling back to
`timestamp_iso`, then to the timestamp embedded in the log file name) —
file category and read order alone can never override a newer record, so an
old run-level SUCCESS cannot mask a newer per-task ERROR. `SUCCESS /
MAX_TURNS / TIMEOUT` count as completed; `ERROR` and never-started tasks
are re-run (passed to the runner as repeated `--task-id` flags in episode
order).
- **Answer-leak prevention**: an interrupted task may already have been
distilled into daily notes during graceful shutdown; re-running it with
that memory injected would inflate scores. Before resuming,
`resume.py cleanup` therefore removes residual memory **only for tasks
about to be re-run** (daily/digest notes, session/dialog, mem_session;
matched via `session_id = pibench_{task}_*`). Completed tasks' memories are
never touched. Daily index files are refreshed **only for the dates that
lost notes**, by full workspace-relative wikilink path — and when the ReMe
package is importable, the refresh reuses ReMe's own daily-index rebuild
logic (`refresh_day_index`), so same-named notes on other dates are never
modified.
- **fresh vs resume are mutually exclusive**: a full memory wipe belongs to
fresh mode only (`run_all.sh` default, executed before any service starts);
resume never wipes.
## 10. Customization entry points
| Goal | Location |
|---|---|
| Base model of the agent under test | `REME_MODEL_NAME` in `env.sh` |
| user_agent / judger models | `config/models/reme.yaml` |
| Agent system prompt | `bridge_reme.py` `build_system_prompt()` |
| Memory retrieval limit/threshold | `--search-limit/--search-min-score` on the bridge command in `run_persona.sh` |
| ReMe internal parameters | **Do not modify ReMe source**; extend the built-in `benchmark` config and override via `resolve_app_config(config=...)` (see bridge `_init_reme_app`) |
| Turn timeout / tool iteration cap | `config/models/reme.yaml` `run.turn_timeout`, `model.max_tool_iterations` |
## 11. Troubleshooting
- **Port already in use**: the scripts auto-kill residual processes on the
four port groups above; if another suite (e.g. a different π-Bench
experiment) holds them, stop it first or change the port table in
run_persona.sh.
- **Bridge exits immediately with workspace locked**: another bridge already
holds the same workspace; make sure each persona uses its own
`--workspace-dir` (the scripts allocate one per persona).
- **Runner reports `${USER_API_KEY} ... empty`**: env.sh is unfilled or not
sourced; run_persona.sh sources env.sh automatically — when running the
runner manually, `source env.sh` first.
- **`Cannot import 'reme'`**: the bridge must run with
`${REME_DIR}/.venv/bin/python` (run_persona.sh already does); otherwise
check that `REME_DIR` points at the ReMe repository root.
- **AppWorld fails to start**: run `bash scripts/setup_appworld.sh` in the
π-Bench repo first (downloads data); inspect
`logs/appworld_*_<persona>.log`.
- **trace_history.yaml not found**: the runner needs
`config/bench/evaluation/trace_history.yaml`; this suite ships the file and
passes it explicitly via `--history-config-path`, and run_persona.sh fails
fast with a clear error if it is missing. Always launch run_persona.sh /
run_all.sh from the suite directory.
## 12. Privacy and security
- The suite code and config templates contain **no real API keys, user names
or absolute paths**; real keys live only in your local `env.sh`
(git-ignored).
- `logs/`, `outputs/`, `reme_workspace/` and `nanobot_workspace/` contain
full conversations and model outputs; never commit or share them.
- The `data` symlink points at the official π-Bench evaluation data; respect
its data license terms.

View file

@ -0,0 +1,284 @@
# π-Bench 评测说明
[English version](./README.md)
将 **ReMe agent(带持久记忆)** 接入 **π-Bench**(Proactive Personal Assistant
Benchmark)的胶水层评测套件。只含对接所需的最小代码与配置;π-Bench 框架
(`src/`)、评测数据(`data/`)、AppWorld 工具环境、ReMe 本体均为**外部第三方
依赖**,通过符号链接与环境变量原位引用,不随本套件分发。
- π-Bench: https://github.com/Simplified-Reasoning/Pi-Bench (arXiv: 2605.14678)
- ReMe: 你所在 ReMe 仓库的根目录(本套件推荐放在 `ReMe/benchmark/pibench/`)
## 1. 架构总览
```
π-Bench runner (src.main --mode run)
│ user_agent(模拟用户 LLM)按 data/{persona}/episode.yaml 顺序
│ 逐任务、多轮地与 agent 对话,并在 run 阶段判定隐藏意图(PROC)
▼
test server (π-Bench scripts/test_server.py, HTTP 长轮询)
▲ /send │ /poll
│ ▼
bridge_reme.py ──────────────► ReMe Application(以库方式内嵌启动)
│ ├─ agent_wrapper: 被测 agent(AgentScope)
│ ├─ jobs: search / auto_memory / daily_write
│ └─ workspace: reme_workspace/{persona}/
│ (每 persona 独立持久记忆库,互不可见)
└──── MCP ────► AppWorld MCP ────► AppWorld API(工具/应用环境)
π-Bench runner (src.main --mode eval)
judger(裁判 LLM)读取 trace,按 checklist 逐条 YES/NO 打分(COMP)
```
要点:
- bridge 用 **ReMe 自己的 venv python** 运行,把 ReMe 当库用(`resolve_app_config`
+ `Application`),**ReMe 源码零改动**。
- 每条用户消息都会自动触发一次 ReMe memory `search` 并把命中记忆注入当前消息
(参数见 §8);任务结束(reset)时会话被 `auto_memory` 提炼为 daily 笔记落盘。
- agent 执行的每一轮工具调用(AppWorld MCP + ReMe job 工具)都会被采集并以
`tool_steps` 形式写入 trace,供 π-Bench 的 `tools_evaluation_path` 脚本
对工具行为评分(§7)。
- π-Bench 的 `data/`、`src/`、AppWorld 均不属于本套件,需先装好 π-Bench(§3.1)。
## 2. 目录结构
```
pibench/
├── README.md / README_ZH.md # 本文档(英文 / 中文)
├── env.sh.example # 环境配置模板(复制为 env.sh 后填写 TODO 项)
├── bridge_reme.py # ReMe ↔ test server 桥接(记忆注入/保存、
│ # profile 注入、工具调用轨迹采集)
├── run_persona.sh # 单 persona 全流程(5 个服务 + run + eval)
├── run_all.sh # 5 个 persona 批跑(fresh/resume,默认 2 并行)
├── resume.py # 断点续跑:完成判定 + 中断任务残留记忆的外科清理
├── fix_trace_logs.py # run 输出 → ~/.nanobot/trace_logs 转换,
│ # 并把工具轨迹合并进 turn 文件(eval 前置)
├── .gitignore # 排除 env.sh 与全部运行产物
└── config/
├── models/reme.yaml # runner 模型配置(model_id=reme)
└── bench/evaluation/trace_history.yaml # trace 渲染策略(随套件提供,
# 经 --history-config-path 显式传入)
```
运行时自动生成(均被 .gitignore 排除):`data`(符号链接)、`logs/`、
`outputs/`、`reme_workspace/`、`nanobot_workspace/`。
## 3. 前置依赖(第三方,先装好)
### 3.1 π-Bench 仓库(含 AppWorld)
```bash
git clone https://github.com/Simplified-Reasoning/Pi-Bench.git <pi-bench-dir>
cd <pi-bench-dir>
python3.11 -m venv .venv # 脚本约定使用 .venv 这个目录名
source .venv/bin/activate
pip install -e . # pibench runner(src.main)
bash scripts/setup_appworld.sh # 安装 AppWorld 并下载其数据(体积较大,需网络)
```
装完自检:
```bash
ls data/ # 应含 researcher marketer pharmacist law_trainee Financier
.venv/bin/python -c "import src" && echo OK
.venv/bin/appworld --help >/dev/null && echo OK
```
### 3.2 ReMe 仓库
```bash
cd <reme-dir> # ReMe 仓库根目录(含 reme/ 包)
python3.11 -m venv .venv # 脚本约定使用 .venv 这个目录名
source .venv/bin/activate
pip install -e . # 或按 ReMe 自身安装方式,保证 `import reme` 可用
```
自检:`.venv/bin/python -c "import reme; print('ok')"`
## 4. 安装本套件(逐步)
1. **放置套件**(推荐放进 ReMe 仓库,`REME_DIR` 可自动推断):
```bash
cp -r pibench <reme-dir>/benchmark/pibench
cd <reme-dir>/benchmark/pibench
```
若放在其他位置,稍后在 env.sh 中显式设置 `REME_DIR`。
2. **创建环境文件并填写自定义参数**:
```bash
cp env.sh.example env.sh
```
打开 `env.sh`,必填项(标 TODO 的):
| 变量 | 说明 |
|---|---|
| `PI_BENCH_ROOT` | π-Bench 仓库根目录(含 `src/` `data/` `.venv` `third_party/appworld`) |
| `USER_API_KEY` | 模拟用户 LLM 的 API key(run 阶段判定隐藏意图) |
| `JUDGER_API_KEY` | 裁判 LLM 的 API key(eval 阶段 checklist 打分) |
| `BRAVE_SEARCH_API_KEY` | 可选;agent 的 web_search 工具用,不用填 `dummy` |
可选调整:`REME_MODEL_NAME`(被测 agent 基模)、`REME_DIR`、
`REME_LLM_BASE_URL`(默认 DashScope OpenAI 兼容端点)。
3. **链接评测数据**(π-Bench 数据原位引用,不复制):
```bash
ln -s "$PI_BENCH_ROOT/data" data
```
4. **(可选)调整模型配置** `config/models/reme.yaml`:
- `user_agent.model` / `judger.model`:模拟用户与裁判的模型名(字面量,
π-Bench 仅对 base_url/api_key 做 `${ENV}` 展开)。
- `run.turn_timeout`、`max_tool_iterations` 等按需。
5. **冒烟自检**(不启动评测):
```bash
bash -n run_all.sh && bash -n run_persona.sh
source env.sh && "$REME_DIR/.venv/bin/python" -c "import reme; print('reme ok')"
```
## 5. 运行评测
> ⚠️ 长时间运行请放进 `screen`,**不要用 nohup**(nohup 在沙箱/受限环境下
> 会丢失权限上下文导致子进程异常)。
```bash
# 完整正式评测:先清空全部 persona 的记忆/输出/trace,再从头跑(默认 fresh,2 并行)
mkdir -p logs # 全新部署时 logs/ 尚不存在,先建再重定向
screen -dmS pibench_suite bash -c "cd $(pwd) && bash run_all.sh > logs/run_all_master.log 2>&1"
# 断点续跑(中断后继续;不清记忆,跳过已完成任务)
bash run_all.sh --resume
# 其他用法
bash run_all.sh --parallel 1 # 串行
bash run_all.sh --resume --skip-eval # 只跑 run 阶段
bash run_persona.sh researcher # 单 persona(默认 --resume 语义)
bash run_persona.sh researcher --fresh
```
耗时参考:5 persona × 20 任务、2 并行,fresh 全量约 12–14 小时。
任一 persona 失败时 `run_all.sh` 以非零状态退出,上层自动化不会把部分失败
的评测误判为成功。
## 6. 端口分配(多 persona 并行互不冲突)
| persona | AppWorld API | AppWorld MCP | Test Server | ReMe 内部服务 |
|-------------|------|-------|------|-------|
| marketer | 9001 | 10001 | 9998 | 18766 |
| law_trainee | 9002 | 10002 | 9997 | 18767 |
| pharmacist | 9003 | 10003 | 9996 | 18768 |
| researcher | 9004 | 10004 | 9995 | 18765 |
| Financier | 9005 | 10005 | 9994 | 18769 |
## 7. 输出与分数
- **结果**:`outputs/reme/{persona}/{task}/eval/results/*_result.json`
- `overall_average_score`:checklist 完整度(COMP,judger 逐条 YES/NO 按依赖组加权)
- `overall_proactiveness_average_score`:主动性(PROC,run 阶段 user_agent
判定隐藏意图覆盖率;每个任务文件同时携带全局均值)
- **trace**:`~/.nanobot/trace_logs/reme/{persona}/{task}/...`(eval 的判分输入)
- **日志**:`logs/`(`suite_<persona>.log` 为每 persona 总日志,`bridge_*`、
`runner_run/eval_*`、`appworld_*`、`test_server_*` 分服务)
- **记忆库**:`reme_workspace/{persona}/`(daily/digest 笔记、session 原始对话、
BM25 索引等;跨运行持久,fresh 才清空)
查看汇总:
```bash
grep -h "overall_average_score\|overall_proactiveness" \
outputs/reme/*/*/eval/results/*_result.json | head
```
### 工具轨迹采集(tools_evaluation 支持)
部分任务定义了 `objectives.tools_evaluation_path`:用 Python 脚本对工具行为
打分(例如"临时 Todoist 看板已创建并被删除")。这些脚本需要 trace 里有真实
的工具调用记录。采集链路:
1. 每轮 `reply()` 之后,bridge 读取 AgentScope 落盘的会话状态,提取本轮新增
的 `tool_call` / `tool_result` 块(工具名、参数、结果)。
2. 记录按 turn 编号追加写入
`outputs/reme/{persona}/{task}/history/{ts}-tools.jsonl`;AgentScope 的
MCP 工具名(`mcp__AppWorld__<tool>`)会规范化为 π-Bench 约定
(`mcp_appworld_<tool>`)。
3. `fix_trace_logs.py` 将每个 `{ts}-messages.jsonl` 运行与时间上最接近的
tools 旁路文件配对,把记录合并进生成的 `turn_N.json` 的 `tool_steps`
字段——这是 π-Bench `collect_tool_history()` 支持的两种工具轨迹格式之一。
4. eval 阶段 `tool_steps` 既提供给 tools_evaluation 脚本,也会被渲染为
judger 可见的 `<tool_trace_extracts>`。
## 8. 记忆机制(本套件的核心设计)
- **persona 隔离**:每个 persona 独立 workspace(`reme_workspace/{persona}/`),
bridge 启动时对 workspace 加 `.bridge.lock` 排他锁,两个 bridge 不可能共用
同一记忆库;一个 persona 的 memory search 永远接触不到其他 persona 的记忆。
- **写入**:任务结束(runner 发送 reset)时,会话经 `auto_memory` job 提炼为
daily 笔记落盘,后台 watcher 建 BM25 索引。保存为非阻塞后台任务,
新会话首条消息会先等待在途写入完成再检索。
- **读取**:bridge 每收到一条用户消息自动 `search` 一次并注入命中记忆
(`[Relevant memories from previous sessions]` 前缀),无命中则原样透传。
检索参数(bridge 命令行,可在 run_persona.sh 中调整):
- `--search-limit 3`:每条消息最多注入 3 个记忆块;
- `--search-min-score 2.0`:过滤弱 BM25 命中;
- `tool_context_id` 按任务轮换:同一任务内已注入的记忆块不重复注入
(ReMe 自带 seen-chunk 去重,24h TTL),任务边界后恢复正常召回。
- **无自泄漏**:进行中的会话尚未入库(save 发生在 reset),任务不会检索到
自己未完成的内容。
- agent 同时持有 `search`/`daily_write` 工具,可主动检索/记录。
- **system prompt**:`bridge_reme.py:build_system_prompt()` 内置
HIDDEN-NEEDS 协议(面向 proactiveness),并把 `data/{persona}/profile.yaml`
的 persona profile 注入每轮 system prompt。
## 9. 断点续跑与记忆清理语义
- **完成判定**(resume.py):扫描 `outputs/reme/{persona}/**/history/*-log.jsonl`
与 `outputs/reme/{persona}/run/*-log.jsonl` 中的
`Task finished task_id=X status=Y`。每个任务以**事件时间最新**的记录为准
(优先取记录的 `timestamp`,回退 `timestamp_iso`,再回退日志文件名中的
时间戳)——文件类别与读取顺序本身不能覆盖更新的记录,因此旧的 run 级
SUCCESS 不会掩盖更新的 per-task ERROR。`SUCCESS/MAX_TURNS/TIMEOUT` 记为
完成,`ERROR`/未开始的任务重跑(按 episode 顺序以 `--task-id` 传给 runner)。
- **防答案泄漏**:被中断的任务可能已在优雅退出时提炼成 daily 笔记,直接重跑会
把答案注入、抬高分数。因此 resume 启动前 `resume.py cleanup` **只删除待重跑
任务**的残留记忆(daily/digest 笔记、session/dialog、mem_session,按
`session_id = pibench_{task}_*` 匹配),已完成任务的记忆一律不动。daily
索引**只刷新实际发生删除的日期**,按完整的 workspace 相对 wikilink 路径
匹配;当 ReMe 包可导入时,刷新直接复用 ReMe 自带的 daily 索引重建逻辑
(`refresh_day_index`),不会误改其他日期下的同名笔记条目。
- **fresh vs resume 互斥**:全量清记忆只属于 fresh 模式(`run_all.sh` 默认,
在任何服务启动前执行);resume 永不清全量。
## 10. 自定义与调优入口
| 目标 | 位置 |
|---|---|
| 被测 agent 基模 | `env.sh` 的 `REME_MODEL_NAME` |
| user_agent / judger 模型 | `config/models/reme.yaml` |
| agent system prompt | `bridge_reme.py` `build_system_prompt()` |
| 记忆检索条数/阈值 | `run_persona.sh` bridge 启动命令的 `--search-limit/--search-min-score` |
| ReMe 内部参数 | **不要改 ReMe 源码**;继承内置 `benchmark` 配置,并经 `resolve_app_config(config=...)` 覆盖(见 bridge `_init_reme_app`) |
| 轮超时/工具迭代上限 | `config/models/reme.yaml` `run.turn_timeout`、`model.max_tool_iterations` |
## 11. 故障排查
- **端口被占用**:脚本会自动 kill 上述 4 组端口上的残留进程;若与其他套件
(如别的 π-Bench 实验)冲突,请先停掉对方或改 run_persona.sh 的端口表。
- **bridge 启动即退出,提示 workspace locked**:另一个 bridge 正占用同一
workspace;确认每个 persona 用各自的 `--workspace-dir`(脚本已按 persona 分配)。
- **runner 报 `${USER_API_KEY} ... empty`**:env.sh 未填写或未生效;
run_persona.sh 会自动 source env.sh,手动运行 runner 时请先 `source env.sh`。
- **`Cannot import 'reme'`**:bridge 必须用 `${REME_DIR}/.venv/bin/python` 运行
(run_persona.sh 已如此),或检查 `REME_DIR` 是否指向 ReMe 仓库根目录。
- **AppWorld 启动失败**:先在 π-Bench 仓库执行 `bash scripts/setup_appworld.sh`
下载数据;查看 `logs/appworld_*_<persona>.log`。
- **trace_history.yaml 找不到**:runner 需要
`config/bench/evaluation/trace_history.yaml`;本套件已随附该文件并通过
`--history-config-path` 显式传入,run_persona.sh 启动前会做存在性检查,
缺失时立即报出清晰错误。请始终从套件目录启动 run_persona.sh / run_all.sh。
## 12. 隐私与安全
- 套件代码与配置模板中**不含任何真实 API key、用户名或绝对路径**;
真实 key 只存在于你本地的 `env.sh`(已被 .gitignore 排除)。
- `logs/`、`outputs/`、`reme_workspace/`、`nanobot_workspace/` 含完整对话内容
与模型输出,请勿提交仓库或外传。
- `data` 符号链接指向 π-Bench 官方评测数据,请遵守其数据许可条款。

1039
benchmark/pibench/bridge_reme.py Executable file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,53 @@
version: 1
format:
root_tag: trace
turn_tag: turn
message_tag: message
file_tag: file
tool_call_tag_prefix: tool_call
tool_result_tag_prefix: tool_result
text_policy:
default:
truncate_chars: 1200
mask_newlines: false
field_overrides:
files_read:
truncate_chars: 40000
assistant_content:
truncate_chars: 40000
tool_result_content:
truncate_chars: 40000
fields:
turn:
include_session_key: false
files:
enabled: true
messages:
enabled: true
include_message_role_attr: true
include_message_index_attr: false
include_system: false
include_user: true
include_assistant_thinking_content: false
include_assistant_thinking_reasoning: false
include_assistant_content: true
include_assistant_reasoning: false
include_assistant_tool_calls: false
require_matching_tool_call: true
tool_calls:
include_tool_call_id: false
tools:
web_fetch:
enabled: true
include_tool_call_keys: [url]
include_tool_result: false
web_search:
enabled: true
include_tool_call_keys: [query]
include_tool_result: false

View file

@ -0,0 +1,40 @@
# ReMe model configuration for Pi-Bench
# Uses ReMe's AgentScope agent with Dashscope as the LLM backend
model:
model: reme
base_url: "http://localhost:8088"
api_key: "dummy"
provider: custom
max_tokens: 16384
max_tool_iterations: 120
memory_window: 100
user_agent:
model: qwen3.8-max
base_url: "${USER_BASE_URL}"
api_key: "${USER_API_KEY}"
temperature: 0.0
request_timeout: 360.0
judger:
model: qwen3.8-max
base_url: "${JUDGER_BASE_URL}"
api_key: "${JUDGER_API_KEY}"
temperature: 0.0
request_timeout: 360.0
tools:
brave_search_api_key: "${BRAVE_SEARCH_API_KEY}"
web_search_max_results: 10
nanobot:
trace_logs_dir: "~/.nanobot/trace_logs"
workspace_dir: "~/.nanobot/workspace"
copy_task_assets_to_workspace: true
run:
output_dir: outputs
log_level: INFO
user_mode: llm
turn_timeout: 2400.0

View file

@ -0,0 +1,57 @@
#!/bin/bash
# ═══════════════════════════════════════════════════════════════════════
# pibench evaluation suite - environment configuration template
# Usage: cp env.sh.example env.sh, then fill in the TODO items below.
# ⚠️ env.sh contains real API keys; never commit or share it
# (already excluded via .gitignore).
# ═══════════════════════════════════════════════════════════════════════
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ─── TODO: π-Bench repository root ────────────────────────────────────
# Must contain src/, data/, scripts/test_server.py, third_party/appworld
# and .venv (see README setup).
export PI_BENCH_ROOT=""
# ─── ReMe repository ──────────────────────────────────────────────────
# Defaults to two levels above this directory (the layout this suite uses
# when placed at ReMe/benchmark/pibench); point it at the actual ReMe
# repository root if the suite lives elsewhere.
export REME_DIR="${REME_DIR:-$(cd "${SUITE_DIR}/../.." && pwd)}"
# ─── Base model of the agent under test (LLM used by the ReMe agent) ──
export REME_MODEL_NAME="${REME_MODEL_NAME:-qwen3.6-plus}"
# ─── LLM service endpoint (default: DashScope OpenAI-compatible; any
# OpenAI-compatible endpoint works) ────────────────────────────────
DASHSCOPE_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
export REME_LLM_BASE_URL="${REME_LLM_BASE_URL:-${DASHSCOPE_BASE_URL}}"
# ─── TODO: API keys ───────────────────────────────────────────────────
# USER_API_KEY : drives the simulated user LLM (run phase; judges whether
# hidden intents are satisfied and asks follow-ups)
# JUDGER_API_KEY: drives the judger LLM (eval phase; scores the checklist)
# The two may be identical; one strong model is recommended for both.
export USER_BASE_URL="${DASHSCOPE_BASE_URL}"
export USER_API_KEY="TODO-fill-in-user-agent-api-key"
export JUDGER_BASE_URL="${DASHSCOPE_BASE_URL}"
export JUDGER_API_KEY="TODO-fill-in-judger-api-key"
# The ReMe agent's key reuses USER_API_KEY by default (no need to repeat
# it when both use the same service and key).
export REME_LLM_API_KEY="${REME_LLM_API_KEY:-${USER_API_KEY}}"
# Brave Search (optional; used by the agent's web_search tool - use
# "dummy" when not needed).
export BRAVE_SEARCH_API_KEY="TODO-optional-brave-search-key-or-dummy"
# ─── Persistent memory workspaces (one subdirectory per persona,
# created automatically) ───────────────────────────────────────────
export REME_WORKSPACE_ROOT="${REME_WORKSPACE_ROOT:-${SUITE_DIR}/reme_workspace}"
# ─── Variables consumed by ReMe's default.yaml model config expansion;
# do not remove ────────────────────────────────────────────────────
export LLM_MODEL_NAME="${REME_MODEL_NAME}"
export LLM_BASE_URL="${REME_LLM_BASE_URL}"
export LLM_API_KEY="${REME_LLM_API_KEY}"

View file

@ -0,0 +1,198 @@
#!/usr/bin/env python3
"""Convert reme_eval run outputs into eval-compatible trace logs.
outputs/{model_id}/{user_id}/{task_id}/history/{ts}-messages.jsonl
-> ~/.nanobot/trace_logs/{model_id}/{user_id}/{task_id}/{ts}/turn_N.json
The bridge additionally writes {ts}-tools.jsonl sidecar files next to the
message histories: one JSON object per executed tool call with fields
{turn, name, arguments, result}. Each messages run is paired with the
temporally closest sidecar, and the records are merged into the generated
turn files under the "tool_steps" key, which is one of the tool-history
formats π-Bench's collect_tool_history() understands. Without this step,
tools_evaluation scripts would see no tool evidence at all.
Usage: python fix_trace_logs.py [user_id ...] (no args = all users)
"""
import json
import re
import sys
from datetime import datetime
from pathlib import Path
SUITE_DIR = Path(__file__).resolve().parent
OUTPUTS_DIR = SUITE_DIR / "outputs"
TRACE_LOGS_DIR = Path.home() / ".nanobot" / "trace_logs"
MESSAGES_FILE_RE = re.compile(r"^(\d{8}_\d{6})-messages\.jsonl$")
TOOLS_FILE_RE = re.compile(r"^(\d{8}_\d{6})-tools\.jsonl$")
TIME_FORMAT = "%Y%m%d_%H%M%S"
# A tool sidecar belongs to the messages run that started at most this many
# seconds earlier (the bridge stamps the sidecar when the task's first user
# message arrives, shortly after the runner opened the messages file).
MAX_PAIR_DELTA_SECONDS = 6 * 3600
def _to_epoch(timestamp: str) -> float:
"""Parse a YYYYMMDD_HHMMSS timestamp into epoch seconds."""
try:
return datetime.strptime(timestamp, TIME_FORMAT).timestamp()
except ValueError:
return 0.0
def load_tool_records(tools_file: Path) -> dict:
"""Group sidecar tool records by turn number."""
by_turn: dict = {}
try:
with open(tools_file, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if not isinstance(record, dict) or not record.get("name"):
continue
turn = int(record.get("turn") or 0)
by_turn.setdefault(turn, []).append(
{
"name": record["name"],
"arguments": record.get("arguments", {}),
"result": record.get("result", ""),
},
)
except OSError as exc:
print(f" WARNING: cannot read tool sidecar {tools_file}: {exc}")
return by_turn
def pair_tool_sidecars(message_runs: list, tool_runs: list) -> dict:
"""Pair each messages run with the temporally closest unused tool sidecar.
Fresh runs produce exactly one messages file and one sidecar per task;
re-runs append matching pairs, so sorted greedy nearest-timestamp
matching is stable. Sidecars farther away than MAX_PAIR_DELTA_SECONDS
(e.g. leftovers of a crashed bridge) stay unpaired.
"""
pairing: dict = {}
unused = list(tool_runs)
for msg_ts, _ in message_runs:
best_delta = None
best_item = None
for tool_ts, tool_path in unused:
delta = abs(_to_epoch(tool_ts) - _to_epoch(msg_ts))
if best_delta is None or delta < best_delta:
best_delta = delta
best_item = (tool_ts, tool_path)
if best_delta is not None and best_item is not None and best_delta <= MAX_PAIR_DELTA_SECONDS:
pairing[msg_ts] = best_item[1]
unused.remove(best_item)
return pairing
def build_turns(messages: list) -> list:
"""Split the flat message list into per-turn [user, assistant] groups."""
turns = []
i = 0
while i < len(messages):
turn_msgs = []
if messages[i]["role"] == "user":
turn_msgs.append({"role": "user", "content": messages[i]["message"]})
i += 1
if i < len(messages) and messages[i]["role"] == "assistant":
turn_msgs.append({"role": "assistant", "content": messages[i]["message"]})
i += 1
if not turn_msgs:
i += 1 # defensive: never spin on unexpected roles
continue
turns.append(turn_msgs)
return turns
def convert_task(model_id: str, user_id: str, task_dir: Path) -> None:
"""Convert one task's history dir into trace turn files with tool_steps."""
history_dir = task_dir / "history"
if not history_dir.is_dir():
return
message_runs = []
tool_runs = []
for msg_file in history_dir.glob("*-messages.jsonl"):
match = MESSAGES_FILE_RE.match(msg_file.name)
if match:
message_runs.append((match.group(1), msg_file))
for tools_file in history_dir.glob("*-tools.jsonl"):
match = TOOLS_FILE_RE.match(tools_file.name)
if match:
tool_runs.append((match.group(1), tools_file))
if not message_runs:
return
message_runs.sort(key=lambda item: item[0])
tool_runs.sort(key=lambda item: item[0])
pairing = pair_tool_sidecars(message_runs, tool_runs)
print(f"\n{model_id}/{user_id}/{task_dir.name}")
for timestamp, msg_file in message_runs:
trace_dir = TRACE_LOGS_DIR / model_id / user_id / task_dir.name / timestamp
trace_dir.mkdir(parents=True, exist_ok=True)
messages = []
with open(msg_file, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
msg = json.loads(line)
if msg.get("role") == "user" and msg.get("message") == "/new":
continue
messages.append(msg)
tools_file = pairing.get(timestamp)
tools_by_turn = load_tool_records(tools_file) if tools_file else {}
if tools_file is not None:
print(f" {timestamp}: paired tool sidecar {tools_file.name}")
turns = build_turns(messages)
for turn_idx, turn_msgs in enumerate(turns, start=1):
turn_data = {"messages": turn_msgs}
tool_steps = tools_by_turn.get(turn_idx)
if tool_steps:
turn_data["tool_steps"] = tool_steps
turn_file = trace_dir / f"turn_{turn_idx}.json"
with open(turn_file, "w", encoding="utf-8") as f:
json.dump(turn_data, f, indent=2, ensure_ascii=False)
tool_total = sum(len(steps) for steps in tools_by_turn.values())
print(f" {timestamp}: {len(turns)} turns, {tool_total} tool step(s) -> {trace_dir}")
def convert_outputs(user_filter=None):
"""Convert message history JSONL files into per-turn trace JSON files."""
if not OUTPUTS_DIR.exists():
print(f"outputs dir not found: {OUTPUTS_DIR}")
return
for model_dir in sorted(OUTPUTS_DIR.iterdir()):
if not model_dir.is_dir():
continue
model_id = model_dir.name
for user_dir in sorted(model_dir.iterdir()):
if not user_dir.is_dir():
continue
user_id = user_dir.name
if user_filter and user_id not in user_filter:
continue
for task_dir in sorted(user_dir.iterdir()):
if task_dir.is_dir():
convert_task(model_id, user_id, task_dir)
if __name__ == "__main__":
convert_outputs(set(sys.argv[1:]) or None)
print("\ndone")

332
benchmark/pibench/resume.py Executable file
View file

@ -0,0 +1,332 @@
#!/usr/bin/env python3
"""Checkpoint-resume support for the reme_eval suite.
Completion source of truth:
- outputs/reme/<persona>/<task_id>/history/*-log.jsonl (per-task logs,
flushed incrementally, survive mid-run kills)
- outputs/reme/<persona>/run/*-log.jsonl (run-level logs,
may be truncated if the process was killed before flush)
lines: "Task finished task_id=<id> status=<STATUS>"
A task counts as COMPLETED when its latest terminal status is one of
SUCCESS / MAX_TURNS / TIMEOUT. ERROR or never-started tasks stay pending.
"Latest" is decided by EVENT TIME, not by file category or read order:
each record's "timestamp" (epoch seconds, or "timestamp_iso" as fallback)
is compared across per-task and run-level logs alike, with the timestamp
embedded in the log file name as a last-resort fallback. This keeps an
old run-level SUCCESS from overriding a newer per-task ERROR when the
re-run died before the new run-level log captured the task.
Commands:
remaining <persona> [--json]
Print task_ids still to run, in data/<persona>/episode.yaml order
(one per line; --json prints {"completed": [...], "remaining": [...]}).
cleanup <persona> [--dry-run]
Surgically remove residual memory artifacts of tasks that are about
to be RE-RUN (i.e. pending tasks that left partial state because a
previous run was interrupted). This prevents answer leakage: an
interrupted task's conversation may already have been distilled into
daily notes during graceful shutdown, and re-running the task with
that memory injected would inflate scores.
Removed artifacts (only for pending tasks with residual state):
- daily/<date>/<note>.md whose frontmatter session_id matches
pibench_<task_id>_*, plus a refresh of ONLY the daily index of
the affected date(s) (daily/<date>.md), matched by the full
workspace-relative note path, never by bare file name
- digest notes with matching session_id
- session/dialog/pibench_<task_id>_*.jsonl
- mem_session/**.jsonl files containing pibench_<task_id>_
When the ReMe package is importable, the daily index refresh reuses
ReMe's own rebuild logic (reme.steps.file_io._daily_index.
refresh_day_index); otherwise index lines are dropped by exact
wikilink path match. Either way, indexes of other dates are never
touched. The ReMe watcher (init_changes_step) detects the deleted
daily notes on next bridge startup and removes them from the BM25
index itself.
Completed tasks' memories are NEVER touched by this command.
Design note (resume vs memory-wipe conflict):
A full memory wipe is a suite-level action of fresh mode (run_all.sh
without --resume) and happens before any service starts. Resume mode
never wipes; it only performs the surgical cleanup above. The two modes
are mutually exclusive, so a resumed run can never lose the cross-session
memory accumulated by completed tasks.
"""
import asyncio
import json
import os
import re
import sys
from datetime import datetime
from pathlib import Path
import yaml
try: # Reuse ReMe's daily-index rebuild when running inside the ReMe venv.
from reme.steps.file_io._daily_index import refresh_day_index
except ImportError: # pragma: no cover - depends on runtime venv
refresh_day_index = None
SUITE_DIR = Path(__file__).resolve().parent
DATA_DIR = Path(os.environ.get("REME_EVAL_DATA_DIR", SUITE_DIR / "data")).resolve()
OUTPUTS_DIR = Path(os.environ.get("REME_EVAL_OUTPUTS_DIR", SUITE_DIR / "outputs")) / "reme"
WORKSPACE_ROOT = Path(
os.environ.get("REME_WORKSPACE_ROOT", SUITE_DIR / "reme_workspace"),
).resolve()
COMPLETED_STATUSES = {"SUCCESS", "MAX_TURNS", "TIMEOUT"}
TASK_FINISHED_RE = re.compile(r"Task finished task_id=(\S+) status=(\S+)")
SESSION_ID_RE = re.compile(r"^session_id:\s*(\S+)", re.MULTILINE)
NOTE_COUNT_RE = re.compile(r"(description:\s*)\d+(\s*note\(s\) today)")
LOG_FILE_TS_RE = re.compile(r"^(\d{8}_\d{6})-log\.jsonl$")
TIME_FORMAT = "%Y%m%d_%H%M%S"
def log(msg: str) -> None:
"""Print a status message to stderr."""
print(msg, file=sys.stderr)
def episode_task_order(persona: str) -> list[str]:
"""Return the ordered task ids from the persona's episode.yaml."""
episode_path = DATA_DIR / persona / "episode.yaml"
with open(episode_path, "r", encoding="utf-8") as f:
episode = yaml.safe_load(f)
return [task["task_id"] for task in episode.get("tasks", [])]
def _event_time(record: dict, file_ts: str) -> float:
"""Best-effort event time (epoch seconds) of one log record.
Prefers the record's own timestamp fields; falls back to the timestamp
embedded in the log file name so that even stripped records keep a
meaningful order. Returns 0.0 when nothing is parseable.
"""
timestamp = record.get("timestamp")
if isinstance(timestamp, (int, float)) and not isinstance(timestamp, bool):
return float(timestamp)
iso = record.get("timestamp_iso")
if isinstance(iso, str):
try:
return datetime.fromisoformat(iso).timestamp()
except ValueError:
pass
if file_ts:
try:
return datetime.strptime(file_ts, TIME_FORMAT).timestamp()
except ValueError:
pass
return 0.0
def latest_task_statuses(persona: str) -> dict[str, str]:
"""Scan per-task and run-level logs; the newest EVENT TIME wins per task.
Every "Task finished" record across both log categories is keyed by
(event_time, file timestamp, file order, line number); the record with
the highest key decides the task's status. File category and read order
alone can never override a newer record from the other category.
"""
persona_dir = OUTPUTS_DIR / persona
if not persona_dir.is_dir():
return {}
log_files = sorted(persona_dir.glob("*/history/*-log.jsonl"))
log_files += sorted(persona_dir.glob("run/*-log.jsonl"))
best: dict[str, tuple[tuple, str]] = {}
for file_order, log_file in enumerate(log_files):
ts_match = LOG_FILE_TS_RE.match(log_file.name)
file_ts = ts_match.group(1) if ts_match else ""
try:
with open(log_file, "r", encoding="utf-8") as f:
for line_no, line in enumerate(f):
if "Task finished" not in line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
match = TASK_FINISHED_RE.search(str(record.get("message", "")))
if not match:
continue
task_id, status = match.group(1), match.group(2)
sort_key = (_event_time(record, file_ts), file_ts, file_order, line_no)
current = best.get(task_id)
if current is None or sort_key > current[0]:
best[task_id] = (sort_key, status)
except OSError:
continue
return {task_id: status for task_id, (_, status) in best.items()}
def split_tasks(persona: str) -> tuple[list[str], list[str]]:
"""Split the episode task order into completed and remaining tasks."""
order = episode_task_order(persona)
statuses = latest_task_statuses(persona)
completed = [t for t in order if statuses.get(t) in COMPLETED_STATUSES]
remaining = [t for t in order if t not in set(completed)]
return completed, remaining
def _daily_note_session_id(note_path: Path) -> str:
try:
text = note_path.read_text(encoding="utf-8")
except OSError:
return ""
match = SESSION_ID_RE.search(text)
return match.group(1) if match else ""
class _WorkspaceFileStoreShim:
"""Structural stand-in for ReMe's file store; only workspace_path is read."""
def __init__(self, workspace_path: Path):
self.workspace_path = workspace_path
def _refresh_daily_indexes(
workspace: Path,
removed_by_date: dict[str, set[str]],
removed: list[str],
) -> None:
"""Rebuild the daily index of each affected date via ReMe's own logic."""
for date in sorted(removed_by_date):
result = asyncio.run(
refresh_day_index(_WorkspaceFileStoreShim(workspace), date, "daily"),
)
if result.get("error"):
log(f"[resume] WARNING: daily index refresh failed for {date}: {result['error']}")
continue
removed.append(f"daily/{date}.md (refreshed, {len(removed_by_date[date])} note(s) removed)")
def _strip_index_lines(
workspace: Path,
removed_by_date: dict[str, set[str]],
removed: list[str],
dry_run: bool,
) -> None:
"""Fallback index edit: drop lines that reference removed notes by full
workspace-relative wikilink path, and fix the note count. Only the index
files of affected dates are touched."""
for date in sorted(removed_by_date):
index_path = workspace / "daily" / f"{date}.md"
if not index_path.is_file():
continue
wikilinks = [f"[[{rel_path}]]" for rel_path in sorted(removed_by_date[date])]
lines = index_path.read_text(encoding="utf-8").splitlines()
kept = [line for line in lines if not any(link in line for link in wikilinks)]
if len(kept) == len(lines):
continue
note_count = sum(1 for line in kept if line.startswith("- [[daily/"))
kept = [NOTE_COUNT_RE.sub(rf"\g<1>{note_count}\2", line) for line in kept]
removed.append(f"{index_path.relative_to(workspace)} (rewritten)")
if not dry_run:
index_path.write_text("\n".join(kept) + "\n", encoding="utf-8")
def cleanup_partial_memory(persona: str, remaining: list[str], dry_run: bool = False) -> list[str]:
"""Remove partial memory artifacts of remaining tasks so they can be re-run cleanly."""
workspace = WORKSPACE_ROOT / persona
removed: list[str] = []
if not workspace.is_dir() or not remaining:
return removed
prefixes = tuple(f"pibench_{task_id}_" for task_id in remaining)
def act(path: Path, label: str) -> None:
removed.append(label)
if not dry_run:
path.unlink()
# 1) daily / digest notes distilled from interrupted sessions. For daily
# notes, remember the full workspace-relative path grouped by date so only
# the affected daily indexes are refreshed below.
removed_by_date: dict[str, set[str]] = {}
for section in ("daily", "digest"):
section_root = workspace / section
if not section_root.is_dir():
continue
for note_path in section_root.rglob("*.md"):
if note_path.parent == section_root:
continue # index files handled below
session_id = _daily_note_session_id(note_path)
if session_id.startswith(prefixes):
rel_path = note_path.relative_to(workspace).as_posix()
act(note_path, rel_path)
if section == "daily":
removed_by_date.setdefault(note_path.parent.name, set()).add(rel_path)
# 2) daily index files: refresh only the dates that lost notes, matching
# notes by their full wikilink path instead of their bare file name.
if removed_by_date:
if dry_run:
for date in sorted(removed_by_date):
removed.append(f"daily/{date}.md (would refresh index)")
elif refresh_day_index is not None:
_refresh_daily_indexes(workspace, removed_by_date, removed)
else:
_strip_index_lines(workspace, removed_by_date, removed, dry_run)
# 3) raw dialog logs of interrupted sessions
dialog_dir = workspace / "session" / "dialog"
if dialog_dir.is_dir():
for task_id in remaining:
for dialog_path in dialog_dir.glob(f"pibench_{task_id}_*.jsonl"):
act(dialog_path, str(dialog_path.relative_to(workspace)))
# 4) agent-scope session states that contain interrupted-task sessions
mem_session_dir = workspace / "mem_session"
if mem_session_dir.is_dir():
for session_path in mem_session_dir.rglob("*.jsonl"):
try:
content = session_path.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
if any(prefix in content for prefix in prefixes):
act(session_path, str(session_path.relative_to(workspace)))
return removed
def main() -> int:
"""CLI entrypoint: run 'remaining' or 'cleanup' action for a persona."""
args = sys.argv[1:]
if len(args) < 2 or args[0] not in {"remaining", "cleanup"}:
print(__doc__, file=sys.stderr)
return 2
command, persona = args[0], args[1]
completed, remaining = split_tasks(persona)
if command == "remaining":
if "--json" in args:
print(json.dumps({"completed": completed, "remaining": remaining}))
else:
for task_id in remaining:
print(task_id)
log(
f"[resume] {persona}: completed={len(completed)} "
f"({', '.join(completed) if completed else '-'}) remaining={len(remaining)}",
)
return 0
dry_run = "--dry-run" in args
removed = cleanup_partial_memory(persona, remaining, dry_run=dry_run)
if removed:
verb = "would remove" if dry_run else "removed"
log(f"[resume] {persona}: {verb} {len(removed)} partial-memory artifact(s):")
for item in removed:
log(f" - {item}")
else:
log(f"[resume] {persona}: no partial-memory artifacts to clean")
return 0
if __name__ == "__main__":
sys.exit(main())

119
benchmark/pibench/run_all.sh Executable file
View file

@ -0,0 +1,119 @@
#!/bin/bash
# Run all 5 personas with the ReMe agent, PARALLEL at a time (default 2).
# Each persona's tasks follow data/{persona}/episode.yaml order.
#
# Usage:
# bash run_all.sh # FRESH official run: wipes ALL personas'
# # ReMe memory/outputs/trace logs first,
# # then runs everything from scratch.
# bash run_all.sh --resume # Checkpoint continuation: no wipe; every
# # persona skips already-completed tasks.
# bash run_all.sh --parallel 1 # sequential (original behavior)
# bash run_all.sh --skip-eval # run phase only
#
# Memory-wipe vs resume conflict resolution:
# The full ReMe memory wipe happens ONLY here, ONLY in fresh mode (the
# default), and ONLY before any service/bridge starts. --resume never
# wipes; run_persona.sh then additionally performs a surgical cleanup of
# residual memory belonging to interrupted (to-be-re-run) tasks, so a
# resumed run keeps all completed-task memory but never inherits a partial
# task's own answer. The two modes are mutually exclusive.
set -uo pipefail
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PERSONAS=(researcher marketer law_trainee pharmacist Financier)
TRACE_ROOT="${HOME}/.nanobot/trace_logs"
PARALLEL=2
MODE="fresh"
PASS_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
--parallel)
PARALLEL="${2:-}"; shift 2 || true
case "$PARALLEL" in (""|*[!0-9]*) echo "--parallel needs a positive integer"; exit 2 ;; esac
[ "$PARALLEL" -lt 1 ] && PARALLEL=1
[ "$PARALLEL" -gt ${#PERSONAS[@]} ] && PARALLEL=${#PERSONAS[@]}
;;
--resume)
if [ "$MODE" = "fresh_set" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="resume"; shift ;;
--fresh)
if [ "$MODE" = "resume" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="fresh_set"; shift ;;
--skip-eval) PASS_ARGS+=(--skip-eval); shift ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
[ "$MODE" = "fresh_set" ] && MODE="fresh"
START_TS=$(date +%Y%m%d_%H%M%S)
SUMMARY_LOG="${SUITE_DIR}/logs/run_all_${START_TS}.summary"
mkdir -p "${SUITE_DIR}/logs"
echo "############################################################"
echo "# reme_eval suite | mode=${MODE} parallel=${PARALLEL} | ${START_TS}"
echo "############################################################"
# ─── Fresh mode: suite-level wipe BEFORE anything starts ──────────────
if [ "$MODE" = "fresh" ]; then
echo "[fresh] wiping ALL personas' memory workspaces, outputs and trace logs..."
for persona in "${PERSONAS[@]}"; do
rm -rf "${SUITE_DIR}/reme_workspace/${persona}"
rm -rf "${SUITE_DIR}/outputs/reme/${persona}"
rm -rf "${TRACE_ROOT}/reme/${persona}"
rm -rf "${SUITE_DIR}/nanobot_workspace/${persona}"
done
echo "[fresh] wipe done."
else
echo "[resume] no memory wipe; personas resume after their last completed task."
fi
# ─── Run personas in batches of PARALLEL ──────────────────────────────
STATUS_LIST=()
ANY_FAILED=0
OVERALL_START=$(date +%s)
TOTAL=${#PERSONAS[@]}
for ((i = 0; i < TOTAL; i += PARALLEL)); do
BATCH=("${PERSONAS[@]:i:PARALLEL}")
BATCH_PIDS=()
BATCH_NAMES=()
echo ""
echo "============================================================"
echo "# BATCH $(( i / PARALLEL + 1 )): ${BATCH[*]} started $(date '+%F %T')"
echo "============================================================"
for persona in "${BATCH[@]}"; do
bash "${SUITE_DIR}/run_persona.sh" "${persona}" --resume ${PASS_ARGS[@]+"${PASS_ARGS[@]}"} \
> "${SUITE_DIR}/logs/suite_${persona}.log" 2>&1 &
BATCH_PIDS+=($!)
BATCH_NAMES+=("$persona")
done
for j in $(seq 0 $(( ${#BATCH[@]} - 1 ))); do
pid=${BATCH_PIDS[$j]}
persona=${BATCH_NAMES[$j]}
if wait "$pid"; then
STATUS_LIST+=("${persona}: OK")
else
rc=$?
ANY_FAILED=1
STATUS_LIST+=("${persona}: FAILED rc=${rc}")
echo "[run_all] ${persona} FAILED (rc=${rc}); see logs/suite_${persona}.log"
fi
done
done
total=$(( $(date +%s) - OVERALL_START ))
echo ""
echo "================ FINAL SUMMARY (${total}s total) ================" | tee -a "${SUMMARY_LOG}"
for line in "${STATUS_LIST[@]}"; do
echo " ${line}" | tee -a "${SUMMARY_LOG}"
done
echo "Summary: ${SUMMARY_LOG}"
if [ "${ANY_FAILED}" -ne 0 ]; then
FAILED_COUNT=$(printf '%s\n' "${STATUS_LIST[@]}" | grep -c "FAILED")
echo "[run_all] ${FAILED_COUNT} persona(s) FAILED; suite run is marked as failed." | tee -a "${SUMMARY_LOG}"
exit 1
fi
exit 0

301
benchmark/pibench/run_persona.sh Executable file
View file

@ -0,0 +1,301 @@
#!/bin/bash
# Run the full pi-bench evaluation for ONE persona with the ReMe agent.
# Tasks follow data/{persona}/episode.yaml order (runner-native).
#
# Usage: bash run_persona.sh <persona> [--fresh|--resume] [--skip-eval]
#
# Modes (default: --resume):
# --resume Checkpoint continuation. Never wipes memory. Tasks already
# finished (SUCCESS/MAX_TURNS/TIMEOUT in the task history logs)
# are skipped via repeated --task-id flags. Before starting, any
# residual memory of tasks that are about to be RE-RUN (partial
# sessions from an interrupted run) is surgically removed by
# resume.py cleanup, so re-runs don't inherit leaked answers.
# --fresh Wipes THIS persona's ReMe memory, outputs and trace logs first,
# then runs all tasks from scratch.
# The two flags are mutually exclusive. A full multi-persona memory wipe is a
# suite-level action of `run_all.sh` (fresh mode), never done here implicitly.
set -uo pipefail
SUITE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TRACE_ROOT="${HOME}/.nanobot/trace_logs"
# ─── External dependencies (pi-bench / ReMe are NOT bundled; see README) ──
if [ ! -f "${SUITE_DIR}/env.sh" ]; then
echo "env.sh not found. Run: cp env.sh.example env.sh (then fill in the TODO items)"
exit 1
fi
source "${SUITE_DIR}/env.sh"
PIBENCH_DIR="${PI_BENCH_ROOT:-}"
if [ -z "${PIBENCH_DIR}" ] || [ ! -f "${PIBENCH_DIR}/src/main.py" ]; then
echo "PI_BENCH_ROOT is unset or invalid (src/main.py not found). Set it in env.sh."
exit 1
fi
if [ ! -x "${PIBENCH_DIR}/.venv/bin/python" ] || [ ! -x "${PIBENCH_DIR}/.venv/bin/appworld" ]; then
echo "pi-bench venv incomplete: ${PIBENCH_DIR}/.venv must provide python + appworld (see README setup)."
exit 1
fi
if [ ! -x "${REME_DIR}/.venv/bin/python" ]; then
echo "ReMe venv not found: ${REME_DIR}/.venv/bin/python (check REME_DIR in env.sh)"
exit 1
fi
if [ ! -e "${SUITE_DIR}/data" ]; then
echo 'Benchmark data not linked. Run: ln -s "$PI_BENCH_ROOT/data" data'
exit 1
fi
# ─── Pre-flight: files the runner needs before any service starts ─────
MODEL_CONFIG="${SUITE_DIR}/config/models/reme.yaml"
HISTORY_CONFIG="${SUITE_DIR}/config/bench/evaluation/trace_history.yaml"
if [ ! -f "${MODEL_CONFIG}" ]; then
echo "Model config not found: ${MODEL_CONFIG} (see README directory layout)."
exit 1
fi
if [ ! -f "${HISTORY_CONFIG}" ]; then
echo "Trace history config not found: ${HISTORY_CONFIG}"
echo "pi-bench requires config/bench/evaluation/trace_history.yaml; see README."
exit 1
fi
APPWORLD_DIR="${PIBENCH_DIR}/third_party/appworld"
PI_PYTHON="${PIBENCH_DIR}/.venv/bin/python"
APPWORLD_BIN="${PIBENCH_DIR}/.venv/bin/appworld"
# resume.py runs on the ReMe venv so it can reuse ReMe's daily-index rebuild.
REME_PYTHON="${REME_DIR}/.venv/bin/python"
PERSONA="${1:-}"
if [ -z "$PERSONA" ]; then
echo "Usage: $0 <persona> [--fresh|--resume] [--skip-eval]"
exit 1
fi
shift
MODE="resume"
SKIP_EVAL=false
while [[ $# -gt 0 ]]; do
case $1 in
--fresh)
if [ "$MODE" = "resume_set" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="fresh"; shift ;;
--resume)
if [ "$MODE" = "fresh" ]; then echo "--fresh and --resume are mutually exclusive"; exit 2; fi
MODE="resume_set"; shift ;;
--skip-eval) SKIP_EVAL=true; shift ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
[ "$MODE" = "resume_set" ] && MODE="resume"
# ─── Per-persona ports (pi-bench AGENTS.md convention) ────────────────
# REME_PORT: ReMe's internal HTTP service; must be unique per concurrent bridge.
case "$PERSONA" in
marketer) API_PORT=9001; MCP_PORT=10001; TEST_PORT=9998; REME_PORT=18766 ;;
law_trainee) API_PORT=9002; MCP_PORT=10002; TEST_PORT=9997; REME_PORT=18767 ;;
pharmacist) API_PORT=9003; MCP_PORT=10003; TEST_PORT=9996; REME_PORT=18768 ;;
researcher) API_PORT=9004; MCP_PORT=10004; TEST_PORT=9995; REME_PORT=18765 ;;
Financier) API_PORT=9005; MCP_PORT=10005; TEST_PORT=9994; REME_PORT=18769 ;;
*) echo "Unknown persona: $PERSONA"; exit 1 ;;
esac
API_URL="http://127.0.0.1:${API_PORT}"
MCP_URL="http://127.0.0.1:${MCP_PORT}/mcp"
TEST_URL="http://127.0.0.1:${TEST_PORT}"
LOG_DIR="${SUITE_DIR}/logs"
mkdir -p "${LOG_DIR}"
# ─── Environment (env.sh already sourced at the top) ──────────────────
WORKSPACE_DIR="${REME_WORKSPACE_ROOT}/${PERSONA}"
NANOBOT_WORKSPACE_DIR="${SUITE_DIR}/nanobot_workspace/${PERSONA}"
mkdir -p "${WORKSPACE_DIR}" "${NANOBOT_WORKSPACE_DIR}"
echo "========================================="
echo "ReMe x Pi-Bench | persona=${PERSONA} | mode=${MODE}"
echo " api=${API_PORT} mcp=${MCP_PORT} test=${TEST_PORT} reme=${REME_PORT}"
echo " model=${REME_MODEL_NAME}"
echo " memory workspace=${WORKSPACE_DIR} (persistent)"
echo "========================================="
# ─── Fresh mode: wipe this persona's state ────────────────────────────
if [ "$MODE" = "fresh" ]; then
echo "[fresh] wiping persona state: memory workspace, outputs, trace logs"
rm -rf "${WORKSPACE_DIR}"
rm -rf "${SUITE_DIR}/outputs/reme/${PERSONA}"
rm -rf "${TRACE_ROOT}/reme/${PERSONA}"
rm -rf "${NANOBOT_WORKSPACE_DIR}"
mkdir -p "${WORKSPACE_DIR}" "${NANOBOT_WORKSPACE_DIR}"
fi
# ─── Resume: determine remaining tasks + clean partial memories ───────
TASK_ARGS=()
RUN_PHASE_NEEDED=true
if [ "$MODE" = "resume" ]; then
REMAINING_JSON="$("${REME_PYTHON}" "${SUITE_DIR}/resume.py" remaining "${PERSONA}" --json)"
if [ -z "$REMAINING_JSON" ]; then
echo "Failed to compute remaining tasks"; exit 1
fi
echo "[resume] ${REMAINING_JSON}"
REMAINING_TASKS=()
while IFS= read -r tid_line; do
[ -n "$tid_line" ] && REMAINING_TASKS+=("$tid_line")
done < <("${REME_PYTHON}" "${SUITE_DIR}/resume.py" remaining "${PERSONA}" 2>/dev/null)
if [ ${#REMAINING_TASKS[@]} -eq 0 ]; then
RUN_PHASE_NEEDED=false
echo "[resume] all tasks already completed; skipping run phase"
else
# Remove residual memory of interrupted (to-be-re-run) tasks so
# re-runs don't get their own partial answers injected.
"${REME_PYTHON}" "${SUITE_DIR}/resume.py" cleanup "${PERSONA}"
for tid in "${REMAINING_TASKS[@]}"; do
TASK_ARGS+=(--task-id "$tid")
done
echo "[resume] running ${#REMAINING_TASKS[@]} remaining task(s): ${REMAINING_TASKS[*]}"
fi
fi
# ─── Port cleanup from previous runs ──────────────────────────────────
for port in ${API_PORT} ${MCP_PORT} ${TEST_PORT} ${REME_PORT}; do
pids=$(lsof -ti :${port} 2>/dev/null || true)
if [ -n "$pids" ]; then
echo "Killing stale processes on port ${port}: ${pids}"
kill -9 $pids 2>/dev/null || true
fi
done
sleep 2
PIDS=()
cleanup() {
echo "[${PERSONA}] cleaning up services..."
for pid in "${PIDS[@]:-}"; do
kill "$pid" 2>/dev/null || true
done
wait 2>/dev/null || true
}
trap cleanup EXIT INT TERM
wait_for_service() {
local url="$1" name="$2" port="$3" timeout="${4:-180}"
echo -n " waiting for ${name}..."
local start=$(date +%s)
while true; do
if curl -sf --max-time 5 "${url}" > /dev/null 2>&1; then
echo " ready"; return 0
fi
if [ -n "$port" ] && lsof -ti :${port} > /dev/null 2>&1; then
local elapsed=$(( $(date +%s) - start ))
if [ "$elapsed" -ge 10 ]; then echo " ready (port)"; return 0; fi
fi
if [ $(( $(date +%s) - start )) -ge "$timeout" ]; then
echo " TIMEOUT"; return 1
fi
sleep 2
done
}
# ─── [1/5] AppWorld API ────────────────────────────────────────────────
echo "[1/5] AppWorld API (:${API_PORT})"
(cd "${APPWORLD_DIR}" && exec "${APPWORLD_BIN}" serve apis --root . \
--port ${API_PORT}) > "${LOG_DIR}/appworld_api_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${API_URL}/docs" "AppWorld API" "${API_PORT}" 180; then
tail -20 "${LOG_DIR}/appworld_api_${PERSONA}.log"; exit 1
fi
# ─── [2/5] AppWorld MCP ────────────────────────────────────────────────
echo "[2/5] AppWorld MCP (:${MCP_PORT})"
TOOLS_CONFIG="${SUITE_DIR}/data/${PERSONA}/tools.yaml"
(cd "${APPWORLD_DIR}" && exec "${APPWORLD_BIN}" serve mcp http --root . \
--remote-apis-url "${API_URL}" --port ${MCP_PORT} \
--tools-config-file "${TOOLS_CONFIG}") > "${LOG_DIR}/appworld_mcp_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${MCP_URL}" "AppWorld MCP" "${MCP_PORT}" 180; then
tail -20 "${LOG_DIR}/appworld_mcp_${PERSONA}.log"; exit 1
fi
# ─── [3/5] Test Server ─────────────────────────────────────────────────
echo "[3/5] Test Server (:${TEST_PORT})"
PORT=${TEST_PORT} "${PI_PYTHON}" "${PIBENCH_DIR}/scripts/test_server.py" \
> "${LOG_DIR}/test_server_${PERSONA}.log" 2>&1 &
PIDS+=($!)
if ! wait_for_service "${TEST_URL}/sent?after=-1" "Test Server" "${TEST_PORT}" 30; then
tail -20 "${LOG_DIR}/test_server_${PERSONA}.log"; exit 1
fi
# ─── [4/5] ReMe Bridge (ReMe venv) ─────────────────────────────────────
echo "[4/5] ReMe Bridge (reme service port ${REME_PORT})"
"${REME_DIR}/.venv/bin/python" "${SUITE_DIR}/bridge_reme.py" \
--test-server-url "${TEST_URL}" \
--appworld-mcp-url "${MCP_URL}" \
--reme-dir "${REME_DIR}" \
--data-root "${SUITE_DIR}/data" \
--user-id "${PERSONA}" \
--workspace-dir "${WORKSPACE_DIR}" \
--reme-port "${REME_PORT}" \
--model-name "${REME_MODEL_NAME}" \
--model-base-url "${REME_LLM_BASE_URL}" \
--model-api-key "${REME_LLM_API_KEY}" \
> "${LOG_DIR}/bridge_${PERSONA}.log" 2>&1 &
BRIDGE_PID=$!
PIDS+=(${BRIDGE_PID})
sleep 5
if ! kill -0 "${BRIDGE_PID}" 2>/dev/null; then
echo "Bridge failed to start:"; tail -30 "${LOG_DIR}/bridge_${PERSONA}.log"; exit 1
fi
for i in $(seq 1 12); do
if grep -q "Bridge started:" "${LOG_DIR}/bridge_${PERSONA}.log" 2>/dev/null; then
echo " bridge initialized"; break
fi
sleep 5
done
grep -q "Bridge started:" "${LOG_DIR}/bridge_${PERSONA}.log" 2>/dev/null || {
echo "WARNING: bridge may not be ready:"; tail -20 "${LOG_DIR}/bridge_${PERSONA}.log"; }
# ─── [5/5] Runner (run phase) ──────────────────────────────────────────
if [ "$RUN_PHASE_NEEDED" = true ]; then
echo "[5/5] Runner: run phase (episode order from data/${PERSONA}/episode.yaml)"
cd "${SUITE_DIR}"
BENCH_TEST_SERVER_URL="${TEST_URL}" PYTHONPATH="${PIBENCH_DIR}" \
"${PI_PYTHON}" -m src.main \
--model-config "${MODEL_CONFIG}" \
--history-config-path "${HISTORY_CONFIG}" \
--mode run --user-id "${PERSONA}" \
--workspace-dir "${NANOBOT_WORKSPACE_DIR}" \
${TASK_ARGS[@]+"${TASK_ARGS[@]}"} \
2>&1 | tee "${LOG_DIR}/runner_run_${PERSONA}.log"
RUN_EXIT=${PIPESTATUS[0]}
if [ ${RUN_EXIT} -ne 0 ]; then
echo "Run phase failed (exit ${RUN_EXIT}). Logs: ${LOG_DIR}/"
exit ${RUN_EXIT}
fi
else
echo "[5/5] Runner: run phase skipped (all tasks completed)"
fi
if [ "$SKIP_EVAL" = true ]; then
echo "Skipping eval (--skip-eval)"
exit 0
fi
# ─── Trace conversion + eval phase (always over all available traces) ──
echo "Converting trace logs..."
"${PI_PYTHON}" "${SUITE_DIR}/fix_trace_logs.py" "${PERSONA}"
echo "Runner: eval phase"
cd "${SUITE_DIR}"
BENCH_TEST_SERVER_URL="${TEST_URL}" PYTHONPATH="${PIBENCH_DIR}" \
"${PI_PYTHON}" -m src.main \
--model-config "${MODEL_CONFIG}" \
--history-config-path "${HISTORY_CONFIG}" \
--mode eval --user-id "${PERSONA}" \
--workspace-dir "${NANOBOT_WORKSPACE_DIR}" \
2>&1 | tee "${LOG_DIR}/runner_eval_${PERSONA}.log"
EVAL_EXIT=${PIPESTATUS[0]}
echo ""
echo "========================================="
echo "persona=${PERSONA} finished (eval exit=${EVAL_EXIT})"
echo " results : ${SUITE_DIR}/outputs/reme/${PERSONA}/"
echo " memory : ${WORKSPACE_DIR}/"
echo " logs : ${LOG_DIR}/"
echo "========================================="
exit ${EVAL_EXIT}

View file

@ -1,54 +0,0 @@
# beam result
## longmemeval版本的prompt
### 100K
agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only(复用已构建 memory)
(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction 平均 token 消耗(default agent, 20 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory
(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction 平均 token 消耗(default agent, 35 cases 全量构建):
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

View file

@ -1,92 +0,0 @@
# LongMemEval 数据集测试结果
## cleaned-s
**basic settings**
1. 使用修改后的auto-memory prompt,关闭auto-dream机制
2. reme-memory中的全部session的时间一定早于question的时间
**results **
1. Agentic answer框架回答,每次最多调用5次search
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-user | 70 | 66 | 4 | 94.3% |
| single-session-assistant | 56 | 52 | 4 | 92.9% |
| knowledge-update | 78 | 60 | 18 | 76.9% |
| multi-session | 133 | 93 | 40 | 69.9% |
| temporal-reasoning | 133 | 78 | 55 | 58.6% |
| single-session-preference | 30 | 8 | 22 | 26.7% |
| **Overall** | **500** | **357** | **143** | **71.4%** |
2. prompted-based amswer,每次固定使用原始query召回10个fileChunk
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-assistant | 56 | 56 | 0 | 100.0% |
| single-session-user | 70 | 67 | 3 | 95.7% |
| knowledge-update | 78 | 69 | 9 | 88.5% |
| multi-session | 133 | 99 | 34 | 74.4% |
| temporal-reasoning | 133 | 83 | 50 | 62.4% |
| single-session-preference | 30 | 16 | 14 | 53.3% |
| **Overall** | **500** | **390** | **110** | **78.0%** |
3. golden session。 使用与prompt-based answer相似的方法,唯一区别是,输入的chunk是longMemEval提供的golden session。
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| single-session-assistant | 56 | 56 | 0 | 100.0% |
| single-session-user | 70 | 69 | 1 | 98.6% |
| knowledge-update | 78 | 74 | 4 | 94.9% |
| temporal-reasoning | 133 | 124 | 9 | 93.2% |
| multi-session | 133 | 117 | 16 | 88.0% |
| single-session-preference | 30 | 17 | 13 | 56.7% |
| **Overall** | **500** | **457** | **43** | **91.4%** |
4. golden session + time filter. 和上面一个实验的区别是,输入的golden被过滤了一次,要求输入session的时间戳必须早于question的时间才行。
一共被过滤掉了75个session,44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空,一共有424个case。
| Category | Total | Correct | Wrong | Accuracy |
|---|---|---|---|---|
| knowledge-update | 78 | 75 | 3 | 96.2% |
| single-session-user | 70 | 67 | 3 | 95.7% |
| multi-session | 133 | 122 | 11 | 91.7% |
| single-session-assistant | 56 | 55 | 1 | 98.2% |
| temporal-reasoning | 133 | 91 | 42 | 68.4% |
| single-session-preference | 30 | 16 | 14 | 53.3% |
| **Overall** | **500** | **426** | **74** | **85.2%** |
5. 关闭auto-memory机制,根据原始query一次性混合检索召回原始session,计算recall.
| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes |
|---|---|---|---|---|
| knowledge-update | 78 | 75 | 99.3% | 100% |
| single-session-user | 70 | 67 | 100% | 100% |
| multi-session | 133 | 122 | 91.8% | 95.8% |
| single-session-assistant | 56 | 55 | 100% | 100% |
| temporal-reasoning | 133 | 91 | 87.6% | 94.2% |
| single-session-preference | 30 | 16 | 100% | 100% |
| **Overall** | **500** | **426** | **87.6%** | **94.2%** |
## 最终groundtruth
### agentic + prompted(最终GT,2026-07-16)
| Category | Total | Agentic | Prompted limit=15 |
|---|---|---|---|
| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) |
| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) |
| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) |
| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) |
| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) |
| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) |
| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** |
Prompted token 消耗:总 input 13,111,421 (平均 26,275/题),总 output 313,370 (平均 628/题)。
平均 sessions_ingested: 44.8,dreams_triggered: 0。

View file

@ -6,7 +6,7 @@
> Code: [https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) > Code: [https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)
<p align="center"> <p align="center">
<img src="gitcha.png" alt="ExpG challenges and overview" width="85%"> <img src="./gitcha.png" alt="ExpG challenges and overview" width="85%">
</p> </p>
### Overview ### Overview

View file

@ -6,7 +6,7 @@
> 代码:[https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) > 代码:[https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)
<p align="center"> <p align="center">
<img src="gitcha.png" alt="ExpG 挑战与概览" width="85%"> <img src="./gitcha.png" alt="ExpG 挑战与概览" width="85%">
</p> </p>
### 简介 ### 简介

View file

@ -1,324 +0,0 @@
# Auto Fin Cookbook
[中文](README_ZH.md)
Auto Fin is a local-first, file-native ETF event-research workflow. It identifies market events in CLS news, selects
related liquid ETFs, studies similar historical events and subsequent returns, and produces a Chinese research report.
> Auto Fin provides event research and holding-period references only. It is not investment advice, does not
> connect to a broker, and does not place or simulate trades.
## Capabilities
- Download CLS news through Tushare and maintain up to 360 days of traceable local news records.
- Rank ETF candidates by previous-trading-day turnover, then select representative ETFs related to current events.
- Search ReMe memory and local news files for similar historical events, with strict source-path and news-ID checks.
- Calculate adjusted D1–D10 historical returns in deterministic code instead of asking an Agent to invent numbers.
- Let an Agent judge event similarity, then calculate weights, expected returns, and a reference holding period in code.
- Save readable Markdown and structured JSON/JSONL artifacts, refresh the daily index, and optionally deliver the report
to DingTalk.
The workflow is assembled by
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml). Its public schemas are in
[`reme/schema/auto_fin.py`](../../reme/schema/auto_fin.py), and its steps are in
[`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/).
## Quick start
Auto Fin requires Python 3.11 or newer, the `core` dependencies, a Tushare token, and credentials for the configured
Claude Code-compatible endpoint.
From the repository root:
```bash
python -m pip install -e ".[core]"
export TUSHARE_TOKEN="your-tushare-token"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=auto_fin
```
The built-in configuration uses `qwen3.7-max` through DashScope's Anthropic-compatible endpoint. Override these
variables to use another compatible model or provider:
```bash
export CLAUDE_CODE_MODEL_NAME="your-model"
export CLAUDE_CODE_BASE_URL="https://your-anthropic-compatible-endpoint"
```
The default workspace is `reme_workspace/`. This standalone cookbook shares its workspace setting with the daily-paper
workflow:
```bash
export DAILY_PAPER_WORKSPACE_DIR="/absolute/path/to/reme-workspace"
```
To deliver the final Markdown report to DingTalk, set:
```bash
export DINGTALK_APP_KEY="your-app-key"
export DINGTALK_APP_SECRET="your-app-secret"
export DINGTALK_ROBOT_CODE="your-robot-code"
export DINGTALK_CONVERSATION_IDS="conversation-id-1,conversation-id-2"
```
DingTalk delivery is skipped when the required values are empty.
Dates and times use `Asia/Shanghai`. The optional `date` must be the current date:
```bash
reme start config=daily_cookbook job=auto_fin date=2026-07-25
```
To refresh every configured news day instead of reusing valid historical files:
```bash
reme start config=daily_cookbook job=auto_fin force=true
```
This may issue many Tushare requests. A normal run reuses valid historical news files and always refreshes today's file.
### Optional SSH proxy
The outbound proxy is disabled by default. To enable it, uncomment `components.outbound_proxy.default` in
`daily_cookbook.yaml`, configure non-interactive SSH authentication, and set:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## How it works
```mermaid
flowchart LR
A[Resolve run date and cutoff] --> B[Maintain CLS news files]
B --> C[Resolve previous A-share trading day]
C --> D[Build current event window]
D --> E[Filter liquid ETF candidates]
E --> F[Agent selects related ETFs]
F --> G{For each ETF}
G --> H[Agent searches historical events]
H --> I[Code resolves original news]
I --> J[Code calculates adjusted D1-D10 returns]
J --> K[Agent judges similarity]
K --> L[Code calculates weighted forecast]
L --> G
G --> M[Agent writes the combined report]
M --> N[Write artifacts and refresh daily index]
N --> O[Optional DingTalk delivery]
```
The top-level job contains four Auto Fin steps:
| Step | Responsibility | Agent |
|-------------------------|-------------------------------------------------------------|-------|
| `auto_fin_data_step` | Maintain news files and resolve the previous trading day | No |
| `auto_fin_topic_step` | Build inputs and select related ETFs and current events | Yes |
| `auto_fin_history_step` | Orchestrate historical research and market analysis per ETF | Yes |
| `auto_fin_merge_step` | Validate results and produce the final Markdown report | Yes |
For each selected ETF, `auto_fin_history_step` dispatches:
- `auto_fin_history_search_step`, which asks the Agent for historical news references and then resolves the original
records and calculates their returns in code.
- `auto_fin_market_step`, which asks the Agent only for similarity judgments and then calculates weights and forecasts
in code.
Agents handle semantic judgments; deterministic code handles source validation and financial calculations.
## Data and time boundaries
### News history
`auto_fin_data_step` reads CLS news from Tushare's `major_news` endpoint:
- The default lookback is 360 calendar days, including the run date.
- A valid historical file is reused unless `force=true`.
- Today's file is refreshed through the current `decision_at` on every run.
- Large responses are fetched through recursively split time windows.
- Records are ordered and deduplicated before being written with a stable `news_id`.
The current event window is:
```text
(previous A-share trading day at 15:00, decision_at]
```
Each run rebuilds this complete window; midday and evening runs do not use only the increment since the previous run.
### ETF candidates
The candidate universe combines:
- `etf_basic` for currently listed ETFs and their tracked indexes.
- `fund_daily` for turnover on the previous A-share trading day.
Code sorts candidates by turnover, removes duplicates by ETF name and index identity, and provides at most 150
candidates to the Topic Agent. The Agent may return at most 20 ETFs and must copy every ETF code, name, and news ID from
the generated candidate files.
Turnover is used only to narrow the research universe; it is not a trading signal.
## Historical research and forecasting
### Source resolution
The History Agent searches by event type, entities, transmission mechanism, and expected direction. It first uses
`memory_search` and may then scan:
```text
daily/YYYY-MM-DD/auto_fin_news_data.jsonl
```
Its output contains only a reason, `news_id`, and workspace-relative `source_path`. Code rejects:
- Current-window news presented as historical evidence.
- Absolute paths, `..` traversal, or paths outside the workspace.
- Sources not named `auto_fin_news_data.jsonl`.
- Missing files or IDs that do not resolve exactly once.
- Records without a usable publication time, title, or body.
Historical Markdown may guide retrieval, but the original news JSONL is the source of truth.
### Adjusted returns
For every resolved historical event, code reads `fund_daily` and `fund_adj` and calculates up to ten future closes:
- Before 09:30 on a trading day: enter at that day's open.
- From 09:30 until before 15:00: enter at that day's close.
- At or after 15:00, or on a non-trading day: enter at the next trading day's open.
- A daily close later than the current `decision_at` is excluded.
```text
adjusted_entry = raw_entry × entry_adjustment_factor
adjusted_close = raw_close × close_adjustment_factor
cumulative_return = adjusted_close / adjusted_entry - 1
```
Missing prices, factors, trading days, or horizons become explicit limitations. They are never filled with Agent-made
values.
### Similarity and forecast
The Market Agent returns semantic similarity in `[-1, 1]`:
- Positive values mean a similar mechanism and direction.
- Negative values mean a comparable mechanism but opposite direction.
- Zero means no useful relationship.
Code clamps out-of-range values, ignores zero-similarity events, normalizes weights from absolute similarity, and
reverses the historical return direction for negative matches. Each D1–D10 horizon is calculated from the samples
available at that horizon. The suggested holding period is the positive-return horizon with the highest expected return,
or empty when none is positive.
The result also records limited samples, missing horizons, conflicting return directions, and other data limitations. It
is a comparison with a small historical sample, not evidence of statistical significance.
## Output layout
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── auto_fin_news_data.jsonl
│ ├── auto_fin_analysis.jsonl
│ └── auto_fin.md
└── resource/
└── YYYY-MM-DD/
├── filtered_news.jsonl
├── filtered_etf.jsonl
├── auto_fin_topic_output.jsonl
├── auto_fin_history_<index>_<ETF-code>_output.json
├── auto_fin_market_<index>_<ETF-code>_output.json
├── auto_fin_history_output.jsonl
└── auto_fin_merge_output.json
```
Important artifacts:
- `auto_fin_news_data.jsonl` is the user-owned source used to resolve historical news.
- `filtered_news.jsonl` and `filtered_etf.jsonl` are bounded inputs for the Topic Agent.
- Per-ETF history files contain resolved source news and code-calculated return paths.
- Per-ETF market files contain code-calculated matches, weights, and D1–D10 forecasts.
- `auto_fin_analysis.jsonl` contains the final structured analysis for every selected ETF.
- `auto_fin.md` is the readable report and DingTalk payload.
- `daily/YYYY-MM-DD.md` is refreshed after report generation so the report is discoverable from the daily index.
News and reports remain ordinary user-owned files. Resource artifacts and search indexes can be rebuilt.
## Configuration
### Job parameters
| Parameter | Default | Meaning |
|-----------|-------------:|---------------------------------------------------------|
| `date` | Current date | Strict `YYYY-MM-DD`; only the current date is supported |
| `force` | `false` | Refresh all configured news days |
### Environment variables
| Variable | Required | Meaning |
|-----------------------------|----------|---------------------------------------------------------|
| `TUSHARE_TOKEN` | Yes | News, calendar, ETF daily data, and adjustment factors |
| `CLAUDE_CODE_API_KEY` | Yes | Auto Fin Agent credentials |
| `CLAUDE_CODE_MODEL_NAME` | No | Defaults to `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | No | Anthropic-compatible endpoint |
| `AUTO_FIN_AGENT_BACKEND` | No | Defaults to `claude_code` |
| `AUTO_FIN_PROJECT_PATH` | No | Agent project path; defaults to `..` |
| `REME_PROXY_IP` | No | SSH proxy host; used only when `ssh_http` is enabled |
| `REME_PROXY_ACCOUNT` | No | SSH proxy account; used only when `ssh_http` is enabled |
| `DAILY_PAPER_WORKSPACE_DIR` | No | Standalone cookbook workspace |
| `DINGTALK_*` | No | DingTalk application, robot, and conversation settings |
Unit tests can inject `tushare_provider` through the runtime context and do not require real credentials.
### Scheduled jobs
`daily_cookbook.yaml` defines:
| Job | Cron | Asia/Shanghai |
|----------------------|---------------|----------------|
| `auto_fin_0930_cron` | `30 9 * * *` | Daily at 09:30 |
| `auto_fin_1145_cron` | `45 11 * * *` | Daily at 11:45 |
| `auto_fin_1800_cron` | `0 18 * * *` | Daily at 18:00 |
These cron expressions do not exclude weekends or market holidays. The workflow resolves the previous A-share trading
day but does not currently skip a run merely because the run date is not a trading day.
## Agent and security boundaries
The Auto Fin wrapper loads the `tushare-data` skill, exposes the `memory_search` job tool, and defaults to
`bypassPermissions`. Prompts constrain each Agent's role, while code revalidates schemas, ETF identities, source paths,
news references, and calculated values.
The standalone cookbook does not configure an embedding store by default, so `memory_search` normally uses BM25 recall.
Vector and BM25 fusion becomes available only when an embedding store is configured.
`bypassPermissions` is not an operating-system sandbox. Review the configured project path, workspace, credentials, and
network boundary before deployment.
## Reruns and limitations
- Valid historical news files are reused; today's news is always refreshed.
- Outputs use stable per-day paths, so a later same-day run replaces the previous report and resource outputs.
- Auto Fin intentionally has no “report exists, skip” shortcut because its scheduled runs analyze updated news.
- Every successful run attempts DingTalk delivery when configured; notification deduplication is not implemented.
- Missing historical market horizons degrade one sample and are recorded as limitations.
- Invalid dates, missing required services, invalid Agent schemas, unknown ETFs or news IDs, unsafe paths, and
inconsistent cross-step ETF identities fail the job.
The current implementation does not include stocks, US-market correlation, portfolio accounting, BUY/SELL/HOLD actions,
T+1 execution rules, fees, slippage, broker integration, or real/simulated order execution.
## Development
Install development dependencies and run the focused suite:
```bash
python -m pip install -e ".[dev,core]"
PYTHONPATH=. pytest tests/unit/test_auto_fin.py -v
```
The unit suite mocks model and market-data boundaries. Tests requiring real Tushare, model, or DingTalk credentials
should be run separately and only with explicit authorization.

View file

@ -1,304 +0,0 @@
# Auto Fin Cookbook
[English](README.md)
Auto Fin 是一个本地优先、文件原生的 ETF 事件研究工作流。它从财联社新闻中识别市场事件,选择相关且流动性较好的
ETF,研究相似历史事件及其后续收益,并生成中文研究报告。
> Auto Fin 只提供事件研究和持有时间参考,不构成投资建议。当前实现不连接券商、不提交委托,也不执行模拟交易。
## 能力
- 通过 Tushare 获取财联社新闻,并维护最多 360 天可追溯的本地新闻记录。
- 按上一交易日成交额筛选 ETF 候选,再选择与当前事件直接相关的代表性 ETF。
- 通过 ReMe 记忆检索和本地新闻文件查找相似历史事件,并严格校验来源路径和新闻 ID。
- 由确定性代码计算复权后的 D1–D10 历史收益,不让 Agent 编造行情数值。
- 由 Agent 判断事件相似度,再由代码计算权重、预期收益和参考持有时间。
- 保存可读 Markdown 和结构化 JSON/JSONL,刷新每日索引,并支持可选钉钉投递。
工作流由 [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配,公共 schema 位于
[`reme/schema/auto_fin.py`](../../reme/schema/auto_fin.py),各步骤位于
[`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/)。
## 快速开始
Auto Fin 要求 Python 3.11 或更高版本、`core` 依赖、Tushare token,以及所配置 Claude Code 兼容 endpoint 的凭据。
在仓库根目录运行:
```bash
python -m pip install -e ".[core]"
export TUSHARE_TOKEN="your-tushare-token"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=auto_fin
```
内置配置默认通过 DashScope 的 Anthropic 兼容 endpoint 使用 `qwen3.7-max`。如需更换兼容模型或服务商:
```bash
export CLAUDE_CODE_MODEL_NAME="your-model"
export CLAUDE_CODE_BASE_URL="https://your-anthropic-compatible-endpoint"
```
默认 workspace 是 `reme_workspace/`。该 standalone cookbook 与每日论文工作流共用 workspace 配置:
```bash
export DAILY_PAPER_WORKSPACE_DIR="/absolute/path/to/reme-workspace"
```
如需把最终 Markdown 报告发送到钉钉:
```bash
export DINGTALK_APP_KEY="your-app-key"
export DINGTALK_APP_SECRET="your-app-secret"
export DINGTALK_ROBOT_CODE="your-robot-code"
export DINGTALK_CONVERSATION_IDS="conversation-id-1,conversation-id-2"
```
相关配置为空时会跳过钉钉投递。
日期和时间均使用 `Asia/Shanghai`。可选的 `date` 必须是当天:
```bash
reme start config=daily_cookbook job=auto_fin date=2026-07-25
```
如需刷新全部新闻日期,而不是复用有效历史文件:
```bash
reme start config=daily_cookbook job=auto_fin force=true
```
这可能产生大量 Tushare 请求。普通运行会复用有效历史新闻,并始终刷新当天文件。
### 可选 SSH 代理
出站代理默认关闭。如需启用,请取消 `daily_cookbook.yaml` 中
`components.outbound_proxy.default` 的注释,配置免交互 SSH 认证,并设置:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## 工作原理
```mermaid
flowchart LR
A[确定运行日和截止时间] --> B[维护财联社新闻文件]
B --> C[确定上一 A 股交易日]
C --> D[构建当前事件窗口]
D --> E[筛选高流动性 ETF 候选]
E --> F[Agent 选择相关 ETF]
F --> G{逐只 ETF}
G --> H[Agent 搜索历史事件]
H --> I[代码回查原始新闻]
I --> J[代码计算复权 D1-D10 收益]
J --> K[Agent 判断相似度]
K --> L[代码计算加权预测]
L --> G
G --> M[Agent 编写合并报告]
M --> N[写入产物并刷新每日索引]
N --> O[可选钉钉投递]
```
顶层 Job 包含四个 Auto Fin Step:
| Step | 职责 | Agent |
|-------------------------|-----------------------------------|-------|
| `auto_fin_data_step` | 维护新闻文件并确定上一交易日 | 否 |
| `auto_fin_topic_step` | 构建输入并选择相关 ETF 和当前事件 | 是 |
| `auto_fin_history_step` | 逐只 ETF 编排历史研究和行情分析 | 是 |
| `auto_fin_merge_step` | 校验结果并生成最终 Markdown 报告 | 是 |
对于每只已选 ETF,`auto_fin_history_step` 会派发:
- `auto_fin_history_search_step`:Agent 返回历史新闻引用,然后由代码解析原始记录并计算收益。
- `auto_fin_market_step`:Agent 只判断相似度,然后由代码计算权重和预测。
Agent 负责语义判断,确定性代码负责来源校验和金融数值计算。
## 数据和时间边界
### 新闻历史
`auto_fin_data_step` 使用 Tushare `major_news` 接口读取财联社新闻:
- 默认回看包含运行日在内的 360 个自然日。
- 有效历史文件会复用,除非设置 `force=true`。
- 每次运行都会把当天文件刷新到当前 `decision_at`。
- 数据量接近接口上限时会递归拆分时间窗口。
- 写入前会排序和去重,并生成稳定 `news_id`。
当前事件窗口为:
```text
(上一 A 股交易日 15:00, decision_at]
```
每次运行都会重建完整窗口;午间和晚间运行不是只读取上次运行后的增量。
### ETF 候选
候选池组合使用:
- `etf_basic`:当前上市 ETF 及其跟踪指数。
- `fund_daily`:上一 A 股交易日成交额。
代码按成交额排序,并按 ETF 名称和指数标识去重,最多向 Topic Agent 提供 150 个候选。Agent 最多返回 20 只 ETF,且 ETF 代码、名称和新闻
ID 都必须逐字来自候选文件。
成交额仅用于缩小研究范围,不是交易信号。
## 历史研究和预测
### 来源回查
History Agent 按事件类型、关键实体、传导机制和预期方向搜索。它优先使用 `memory_search`,必要时扫描:
```text
daily/YYYY-MM-DD/auto_fin_news_data.jsonl
```
Agent 只返回理由、`news_id` 和 workspace 相对 `source_path`。代码会拒绝:
- 把当前事件窗口内的新闻当作历史证据;
- 绝对路径、`..` 路径穿越或 workspace 外路径;
- 文件名不是 `auto_fin_news_data.jsonl` 的来源;
- 不存在的文件或不能唯一解析的 ID;
- 缺少有效发布时间、标题或正文的记录。
历史 Markdown 只能作为检索线索,原始新闻 JSONL 才是事实来源。
### 复权收益
对每条已回查的历史事件,代码读取 `fund_daily` 和 `fund_adj`,计算最多十个未来收盘点:
- 交易日 09:30 前发生:以当日开盘价为 entry。
- 09:30 至 15:00 前发生:以当日收盘价为 entry。
- 15:00 或之后、以及非交易日发生:以下一交易日开盘价为 entry。
- 晚于当前 `decision_at` 的日线收盘数据不会参与计算。
```text
adjusted_entry = raw_entry × entry_adjustment_factor
adjusted_close = raw_close × close_adjustment_factor
cumulative_return = adjusted_close / adjusted_entry - 1
```
缺少价格、复权因子、交易日或 horizon 时会记录明确限制,不会用 Agent 生成的数值补齐。
### 相似度和预测
Market Agent 返回 `[-1, 1]` 范围内的语义相似度:
- 正值表示机制和方向相似。
- 负值表示机制可比但方向相反。
- `0` 表示没有有效关系。
代码会截断越界值、忽略零相似度事件,并按相似度绝对值归一化权重。负相似度样本会反转历史收益方向。 每个 D1–D10 horizon 只使用该
horizon 有数据的样本。参考持有时间取正预期收益中最高的 horizon; 没有正值时留空。
结果还会记录样本不足、horizon 缺失、收益方向冲突等限制。它只是有限历史样本比较,不代表统计显著性。
## 输出布局
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── auto_fin_news_data.jsonl
│ ├── auto_fin_analysis.jsonl
│ └── auto_fin.md
└── resource/
└── YYYY-MM-DD/
├── filtered_news.jsonl
├── filtered_etf.jsonl
├── auto_fin_topic_output.jsonl
├── auto_fin_history_<序号>_<ETF代码>_output.json
├── auto_fin_market_<序号>_<ETF代码>_output.json
├── auto_fin_history_output.jsonl
└── auto_fin_merge_output.json
```
主要产物:
- `auto_fin_news_data.jsonl`:用户拥有的历史新闻回查来源。
- `filtered_news.jsonl` 和 `filtered_etf.jsonl`:边界明确的 Topic Agent 输入。
- 各 ETF history 文件:已回查的原始新闻和代码计算的收益路径。
- 各 ETF market 文件:代码计算的匹配、权重和 D1–D10 预测。
- `auto_fin_analysis.jsonl`:全部已选 ETF 的最终结构化分析。
- `auto_fin.md`:可读报告和钉钉投递内容。
- `daily/YYYY-MM-DD.md`:报告生成后会刷新,确保每日索引能够发现 Auto Fin 报告。
新闻和报告都是用户拥有的普通文件;resource 中间产物和搜索索引均可重建。
## 配置
### Job 参数
| 参数 | 默认值 | 说明 |
|---------|--------:|-----------------------------------|
| `date` | 当天 | 严格 `YYYY-MM-DD`;当前只支持当天 |
| `force` | `false` | 是否刷新全部已配置新闻日期 |
### 环境变量
| 变量 | 必需 | 说明 |
|-----------------------------|------|-------------------------------------|
| `TUSHARE_TOKEN` | 是 | 新闻、交易日历、ETF 日线和复权因子 |
| `CLAUDE_CODE_API_KEY` | 是 | Auto Fin Agent 凭据 |
| `CLAUDE_CODE_MODEL_NAME` | 否 | 默认 `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | 否 | Anthropic 兼容 endpoint |
| `AUTO_FIN_AGENT_BACKEND` | 否 | 默认 `claude_code` |
| `AUTO_FIN_PROJECT_PATH` | 否 | Agent project path,默认 `..` |
| `REME_PROXY_IP` | 否 | 仅启用 `ssh_http` 时使用的 SSH 主机 |
| `REME_PROXY_ACCOUNT` | 否 | 仅启用 `ssh_http` 时使用的 SSH 账户 |
| `DAILY_PAPER_WORKSPACE_DIR` | 否 | standalone cookbook workspace |
| `DINGTALK_*` | 否 | 钉钉应用、机器人和会话设置 |
单元测试可通过 RuntimeContext 注入 `tushare_provider`,不需要真实凭据。
### 定时任务
`daily_cookbook.yaml` 定义:
| Job | Cron | Asia/Shanghai |
|----------------------|---------------|---------------|
| `auto_fin_0930_cron` | `30 9 * * *` | 每天 09:30 |
| `auto_fin_1145_cron` | `45 11 * * *` | 每天 11:45 |
| `auto_fin_1800_cron` | `0 18 * * *` | 每天 18:00 |
这些 cron 表达式不会排除周末或休市日。工作流会确定上一 A 股交易日,但当前不会仅因为运行日不是交易日而跳过。
## Agent 和安全边界
Auto Fin wrapper 会加载 `tushare-data` skill、暴露 `memory_search` Job 工具,并默认使用
`bypassPermissions`。Prompt 会约束各 Agent 的职责,代码则再次校验 schema、ETF 身份、来源路径、 新闻引用和计算值。
standalone cookbook 默认没有配置 embedding store,因此 `memory_search` 通常使用 BM25 召回; 配置 embedding store 后才能使用向量与
BM25 融合。
`bypassPermissions` 不是操作系统沙箱。部署前应检查 project path、workspace、凭据和网络边界。
## 重跑和限制
- 有效历史新闻会复用,当天新闻始终刷新。
- 输出使用稳定的每日路径,因此同一天后一次运行会替换前一次报告和 resource 输出。
- Auto Fin 不使用“报告已存在则跳过”,因为定时运行需要分析更新后的新闻。
- 配置钉钉后,每次成功运行都会尝试投递;当前没有通知去重。
- 历史样本缺少部分 horizon 时会降级该样本并记录限制。
- 非法日期、缺少必要服务、Agent schema 错误、未知 ETF/新闻、危险路径或跨步骤 ETF 身份不一致会使 Job 失败。
当前没有实现个股、美股关联、组合账本、BUY/SELL/HOLD、T+1 执行、手续费、滑点、券商连接, 也不会执行真实或模拟委托。
## 开发
安装开发依赖并运行聚焦测试:
```bash
python -m pip install -e ".[dev,core]"
PYTHONPATH=. pytest tests/unit/test_auto_fin.py -v
```
单元测试会 mock 模型和行情数据边界。需要真实 Tushare、模型或钉钉凭据的测试应单独运行,且需要显式授权。

View file

@ -1,376 +0,0 @@
# Daily Paper Cookbook
[中文](README_ZH.md)
Daily Paper is a local-first, file-native workflow for turning research rankings into a daily reading package.
## Capabilities
- Collect papers from the Hugging Face weekly and monthly rankings while excluding yesterday's papers and recent
recommendations.
- Rank and select candidates, then use Claude Code to produce detailed Chinese notes and a five-minute Chinese brief.
- Keep PDFs, notes, and memories as ordinary user-owned files; indexes and caches remain rebuildable.
- Support daily scheduling, optional DingTalk delivery, conversation memory, auto-dream consolidation, and BM25 recall
for the background DingTalk agent.
The workflow is assembled by [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml). Its schemas live in
[`reme/schema/daily_paper.py`](../../reme/schema/daily_paper.py), and its steps live in
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/).
## Quick start
Daily Paper requires Python 3.11 or later, the `core` dependencies, network access to Hugging Face and arXiv, and
credentials for the configured Claude Code endpoint. Auto-memory and auto-dream additionally require the AgentScope LLM
credentials.
From the repository root:
```bash
python -m pip install -e ".[core]"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=daily_paper
```
The built-in configuration uses `qwen3.7-max` through DashScope's Anthropic-compatible endpoint. Override
`CLAUDE_CODE_MODEL_NAME` and `CLAUDE_CODE_BASE_URL` when using another compatible model or provider.
This is enough to generate paper notes and the daily brief. To use auto-memory and auto-dream, also configure:
```bash
export LLM_API_KEY="your-api-key"
```
By default, outputs are written under `reme_workspace/` in the directory where ReMe starts.
### Optional SSH proxy
The outbound proxy is disabled by default. To enable it, uncomment `components.outbound_proxy.default` in
`daily_cookbook.yaml`, configure non-interactive SSH authentication, and set:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## What it creates
A successful run writes ordinary PDFs and Markdown files beneath `workspace_dir`:
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── daily-paper-brief.md
│ ├── paper-<arxiv-id>.md
│ └── ...
├── resource/
│ └── papers/
│ ├── <arxiv-id>.pdf
│ └── ...
├── digest/
│ ├── personal/
│ ├── project/
│ ├── resource/
│ └── wiki/
├── metadata/
│ └── ... derived catalogs, indexes, and caches
└── mem_session/
├── agentscope/
└── claude_config/
```
- `paper-<arxiv-id>.md` is a detailed Chinese reading note with YAML frontmatter linking back to the source PDF and
paper pages.
- `daily-paper-brief.md` is a roughly five-minute Chinese digest with wikilinks to every selected paper note.
- `daily/YYYY-MM-DD.md` is a derived day index rebuilt from the Markdown files for that date.
- `resource/papers/` holds reusable source PDFs.
- `digest/` contains durable auto-dream output; files there remain ordinary user-owned Markdown.
- `metadata/` and search caches are derived state. `reindex` rebuilds the file store, BM25 index, and graph from source
files.
The paper notes are the source of truth for recommendation history: their frontmatter contains the `arxiv_id` values
used for future deduplication. The day index is derived and can be rebuilt. The workflow does not currently write a
separate run manifest.
## How the workflow works
```mermaid
flowchart LR
HF[Hugging Face<br/>weekly + monthly] --> C[1. Collect]
Y[Yesterday's papers] --> C
H[Recent local notes] --> C
C --> R[2. Rank]
R --> S[3. Select]
S --> A[4. Analyze PDFs]
A --> D[5. Build brief]
D --> N[6. Notify DingTalk]
A --> P[PDFs + paper notes]
D --> B[Brief + day index]
```
### 1. Collect and deduplicate
The Collect step fetches the weekly ranking for the run date's ISO week, the monthly ranking for its calendar month, and
the Hugging Face Daily Papers IDs for exactly the previous calendar day. It merges weekly and monthly metadata by arXiv
ID and preserves each list's display rank.
It then scans `daily/<prior-date>/paper-*.md` over the configured history window and excludes IDs found in note
frontmatter. The job fails clearly if no eligible papers remain.
### 2. Rank candidates
The Rank step uses reciprocal-rank fusion:
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
A missing rank contributes zero. Candidates are ordered by fused score, upvotes, and arXiv ID. The bounded candidate
pool also reserves several positions for papers whose titles or summaries match memory-related terms such as agent
memory, memory retrieval, continual learning, context compression, knowledge graphs, and RAG. This reserve is a simple
keyword heuristic, not a semantic classifier.
### 3. Select papers
Claude Code receives the bounded candidate pool and returns a structured `PaperSelection`. The implementation requires
exactly `top_k` unique in-pool IDs with consecutive ranks. Invalid output is returned to the agent once as validation
feedback; a second invalid response fails the job.
### 4. Download and analyze PDFs
Selected papers are processed sequentially. For each paper, the workflow:
1. validates the modern arXiv ID format;
2. downloads and validates the PDF, or reuses an existing file with a valid `%PDF-` header;
3. extracts text with `pypdf`, adding page markers and applying page and character limits;
4. asks Claude Code for a structured detailed reading; and
5. writes normalized frontmatter plus the generated Markdown body.
The current extractor requires a usable PDF text layer. Scanned or image-only PDFs fail because there is no OCR
fallback. If extraction exceeds a configured limit, the note records that the input was truncated.
### 5. Build the brief and index
Claude Code reads every detailed note and produces the daily brief. The code verifies that each source-note wikilink is
present and appends any missing links before writing the file. It then rebuilds `daily/YYYY-MM-DD.md` from that day's
Markdown frontmatter.
### 6. Optionally notify DingTalk
The final step sends the brief body, without YAML frontmatter, to each configured DingTalk group in order. With no
conversation IDs it is a no-op. If one group fails, the step still attempts the remaining groups and reports the
combined failure afterward.
## Memory and search
The standalone configuration separates agent wrappers by responsibility:
- `daily_paper` selects papers, analyzes them, and builds the brief. It keeps Claude Code's normal local tools and
disables `WebSearch`, but currently has no memory-retrieval job configured.
- `dingtalk_wait` runs the background DingTalk agent and exposes `memory_search` as a callable tool.
- `memory` runs auto-memory and the LLM-backed auto-dream steps through AgentScope. Its built-in shell and file tools
are disabled; memory changes go through the narrower ReMe jobs such as `daily_write`, `read`, `edit`, and `write`.
The built-in `memory_search` job uses BM25 over Markdown under `daily/` and `digest/`. ReMe's search step can fuse
vector results, but this cookbook does not configure an embedding store by default, so vector retrieval is not run.
`node_search` is a narrower digest recall tool used internally by auto-dream.
`index_update_loop` indexes existing memory files when the service starts and watches those directories for later
changes. Run `reindex` when recovering the derived file store or forcing a complete index rebuild. Source Markdown and
PDFs are not deleted by `reindex`.
`auto_memory` writes or updates one daily note from caller-supplied conversation messages and a stable `session_id`.
`auto_dream` scans recent daily notes, integrates durable units under `digest/`, and writes interest topics. Both are
on-demand jobs in this cookbook; no auto-dream cron is configured. The DingTalk agent can recall through
`memory_search`, but it does not automatically call `auto_memory` after a conversation.
## Dates, reruns, and idempotency
- `date` must be an exact `YYYY-MM-DD` value. When omitted, the job uses today in the application timezone, which is
`Asia/Shanghai` in the built-in configuration.
- “Yesterday” means `date - 1 day`, not the previous 24 hours.
- `history_days` considers prior dated note directories only; the current run date is never part of its history scan.
- If `daily/<date>/daily-paper-brief.md` already exists and `force=false`, collection, ranking, model calls, PDF work,
and digest generation are skipped. The existing brief remains available to the DingTalk notification step.
- `force=true` regenerates the notes and brief. Existing valid PDFs are still reused.
Each PDF, detailed note, and final brief uses a temporary file followed by replacement so callers do not see a partially
written file. The complete multi-file workflow is not transactional, and there is no global lock for two concurrent runs
of the same date.
## Running the cookbook
The main jobs in the standalone configuration are:
| Job | Behavior |
|---------------------|-----------------------------------------------------------------|
| `daily_paper` | On-demand generation through the CLI or HTTP service |
| `daily_paper_cron` | The same pipeline every day at 08:00 in `Asia/Shanghai` |
| `dingtalk_wait` | A supervised background DingTalk agent with `memory_search` |
| `auto_memory` | Write or update a daily note from conversation messages |
| `auto_dream` | Consolidate recent daily notes into digest memory and interests |
| `memory_search` | BM25 recall over daily and digest Markdown |
| `reindex` | Rebuild derived search state from existing memory files |
| `index_update_loop` | Initialize and continuously update search state in service mode |
Supporting jobs such as `node_search`, `daily_list`, `daily_write`, `read`, `write`, `edit`, and frontmatter updates
provide the constrained tools used by the memory agent.
### One-time runs
The quick-start command generates today's brief. To generate a specific date with selected overrides:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-07-21 \
top_k=3 \
history_days=30
```
Regenerate a date whose brief already exists:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-07-21 force=true
```
Add `service.show_metadata=true` to a one-time command when the response metadata is useful for diagnostics.
### Long-running service and cron
Start the standalone HTTP service and its scheduled/background jobs:
```bash
reme start config=daily_cookbook
```
It listens on `127.0.0.1:8001` by default, so it can run beside the default ReMe service. Call the on-demand job from
another terminal with either the ReMe client or HTTP:
```bash
reme daily_paper host=127.0.0.1 port=8001
```
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-07-21","top_k":3,"force":false}'
```
Recall memory, record a conversation, consolidate it, or explicitly rebuild the search index:
```bash
reme memory_search host=127.0.0.1 port=8001 query="agent memory" limit=5
reme auto_memory host=127.0.0.1 port=8001 \
session_id=example-session \
messages='[{"name":"user","role":"user","content":"I prefer concise paper summaries."}]'
reme auto_dream host=127.0.0.1 port=8001 date=2026-07-21
reme reindex host=127.0.0.1 port=8001
```
Service and schedule settings can be overridden at startup:
```bash
reme start \
config=daily_cookbook \
service.host=0.0.0.0 \
service.port=8101 \
jobs.daily_paper_cron.cron="30 7 * * *"
```
## Configuration
The most useful job settings are:
| Setting | Default | Purpose |
|-------------------|-------------:|--------------------------------------------------------------|
| `candidate_limit` | `20` | Maximum number of papers sent to selection |
| `memory_reserve` | `5` | Candidate positions reserved by the memory-keyword heuristic |
| `top_k` | `3` | Number of papers selected and analyzed |
| `rrf_k` | `60` | Reciprocal-rank fusion constant |
| `weekly_weight` | `0.7` | Weight of the weekly ranking in fusion |
| `history_days` | `30` | Prior recommendation window excluded by arXiv ID |
| `hf_timeout` | `30` seconds | Hugging Face request timeout |
| `hf_max_retries` | `3` | Maximum Hugging Face request attempts |
| `pdf_timeout` | `90` seconds | arXiv download timeout |
| `max_pdf_bytes` | `52428800` | Maximum PDF size (50 MiB) |
| `max_pdf_pages` | `80` | Maximum pages extracted for analysis |
| `max_pdf_chars` | `240000` | Maximum extracted characters sent for one paper |
The public job parameters are `date`, `force`, `top_k`, `weekly_weight`, and `history_days`. Explicit invocation values
take precedence over the job defaults.
The standalone application also accepts these environment variables:
| Variable | Purpose |
|-----------------------------------------|------------------------------------------------|
| `DAILY_PAPER_WORKSPACE_DIR` | Overrides the default `reme_workspace` |
| `DAILY_PAPER_PROJECT_PATH` | Repository/project path visible to Claude Code |
| `REME_PROXY_IP` | Optional SSH proxy host |
| `REME_PROXY_ACCOUNT` | Optional SSH proxy account |
| `DAILY_PAPER_HOST` / `DAILY_PAPER_PORT` | HTTP bind address |
| `CLAUDE_CODE_API_KEY` | API key for the Claude Code endpoint |
| `CLAUDE_CODE_MODEL_NAME` | Claude Code model; default `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | Claude Code Anthropic-compatible endpoint |
| `LLM_API_KEY` | API key for the AgentScope memory model |
| `LLM_MODEL_NAME` | Memory model; default `qwen3.7-max` |
| `LLM_BASE_URL` | Memory model's Anthropic-compatible endpoint |
`DAILY_PAPER_PROJECT_PATH` defaults to `..` relative to the workspace. With the default `reme_workspace`, starting from
the repository root resolves it back to the repository. If the workspace lives elsewhere, set both paths explicitly.
ReMe loads an uncommitted `.env` file found from the current directory upward, so the same values may be placed there
instead of exported in the shell.
## DingTalk configuration
DingTalk is optional. Configure it only when brief delivery or the background DingTalk agent is needed:
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
`DINGTALK_CONVERSATION_IDS` is required only for proactive brief delivery. The background `dingtalk_wait` job uses the
first three credentials but not the conversation list.
## Failure recovery and boundaries
| Situation | Behavior |
|--------------------------------------|----------------------------------------------------------------|
| Temporary Hugging Face failure | Retries with exponential delay up to `hf_max_retries` attempts |
| No eligible papers | Fails before ranking |
| Invalid `top_k` or selection output | Fails after validation; selection output gets one retry |
| Oversized, invalid, or textless PDF | Stops during analysis |
| PDF exceeds page or character limits | Continues with truncated text and records the truncation |
| One paper analysis fails | Stops the job; earlier PDFs and notes remain on disk |
| Brief misses a source-note link | Appends the missing wikilink before writing |
| Auto-dream partial integration | Successful units remain; failed paths are not checkpointed |
To recover, inspect the date's notes and PDFs, fix the network, credential, model, or PDF issue, then rerun the same
date with `force=true`. Valid cached PDFs will be reused.
The built-in Claude Code components run with `permission_mode: bypassPermissions` and disable `WebSearch`.
`dingtalk_wait` can call the local `memory_search` job; `daily_paper` currently has no job tools. The analysis and brief
prompts constrain what the agent should read, but these steps do not set a strict per-call tool allowlist or an
operating-system sandbox. The AgentScope memory wrapper disables its built-in shell and filesystem tools, but runs its
ReMe job tools in bypass permission mode. Run the cookbook only with a trusted project and workspace, and tighten the
agent configuration before shared or production use.
## Tests
The focused unit suite mocks Hugging Face, arXiv, Claude Code, and DingTalk boundaries:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```
Real runs access external services and may incur model costs; they should not be used as ordinary unit tests.

View file

@ -1,351 +0,0 @@
# 每日论文 Cookbook
[English](README.md)
每日论文是一个本地优先、文件原生的研究资讯工作流,用于把研究榜单转化为每日阅读材料。
## 能力
- 从 Hugging Face 周榜和月榜采集论文,并排除昨日论文和近期已经推荐过的论文。
- 对候选论文进行排序和精选,再使用 Claude Code 生成中文详细论文笔记和约五分钟可读完的中文简报。
- 将 PDF、笔记和记忆保存为由用户拥有的普通文件;索引和缓存均可重建。
- 支持每日定时运行、可选钉钉投递、对话记忆、auto-dream 整理,以及供后台钉钉 Agent 使用的 BM25 检索。
工作流由 [`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配,公共 schema 位于
[`reme/schema/daily_paper.py`](../../reme/schema/daily_paper.py),各步骤位于
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/)。
## 快速开始
每日论文要求 Python 3.11 或更高版本、`core` 依赖、可访问 Hugging Face 和 arXiv 的网络,以及所配置 Claude Code endpoint
的凭据。auto-memory 和 auto-dream 还需要 AgentScope LLM 凭据。
在仓库根目录运行:
```bash
python -m pip install -e ".[core]"
export CLAUDE_CODE_API_KEY="your-api-key"
reme start config=daily_cookbook job=daily_paper
```
内置配置默认通过 DashScope 的 Anthropic 兼容 endpoint 使用 `qwen3.7-max`。如需使用其他兼容模型或服务商, 请覆盖
`CLAUDE_CODE_MODEL_NAME` 和 `CLAUDE_CODE_BASE_URL`。
以上配置足以生成论文笔记和每日简报。要使用 auto-memory 和 auto-dream,还需配置:
```bash
export LLM_API_KEY="your-api-key"
```
默认情况下,产物写入 ReMe 启动目录下的 `reme_workspace/`。
### 可选 SSH 代理
出站代理默认关闭。如需启用,请取消 `daily_cookbook.yaml` 中
`components.outbound_proxy.default` 的注释,配置免交互 SSH 认证,并设置:
```bash
export REME_PROXY_IP="your-ssh-proxy-host"
export REME_PROXY_ACCOUNT="your-ssh-account"
```
## 文件产物
一次成功运行会在 `workspace_dir` 下写入普通 PDF 和 Markdown 文件:
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── daily-paper-brief.md
│ ├── paper-<arxiv-id>.md
│ └── ...
├── resource/
│ └── papers/
│ ├── <arxiv-id>.pdf
│ └── ...
├── digest/
│ ├── personal/
│ ├── project/
│ ├── resource/
│ └── wiki/
├── metadata/
│ └── ... 派生 catalog、索引和缓存
└── mem_session/
├── agentscope/
└── claude_config/
```
- `paper-<arxiv-id>.md` 是中文详细论文解读,其 YAML frontmatter 会链接原始 PDF 和论文页面。
- `daily-paper-brief.md` 是约五分钟可读完的中文简报,并包含每篇入选论文笔记的 wikilink。
- `daily/YYYY-MM-DD.md` 是从当日 Markdown 文件重建的派生日索引。
- `resource/papers/` 保存可复用的原始 PDF。
- `digest/` 保存持久的 auto-dream 产物,其中仍然是由用户拥有的普通 Markdown 文件。
- `metadata/` 和搜索缓存属于派生状态;`reindex` 会根据源文件重建 file store、BM25 索引和图。
论文笔记是推荐历史的事实来源:后续排重会读取其 frontmatter 中的 `arxiv_id`。日索引属于可重建的派生文件。 当前工作流不会另外写入运行
manifest。
## 工作流程
```mermaid
flowchart LR
HF[Hugging Face<br/>周榜 + 月榜] --> C[1. Collect]
Y[昨日论文] --> C
H[近期本地笔记] --> C
C --> R[2. Rank]
R --> S[3. Select]
S --> A[4. Analyze PDFs]
A --> D[5. Build brief]
D --> N[6. Notify DingTalk]
A --> P[PDF + 论文笔记]
D --> B[简报 + 日索引]
```
### 1. 采集与排重
Collect 会获取运行日所在 ISO week 的周榜、所在自然月的月榜,以及严格前一个自然日的 Hugging Face Daily Papers ID。周榜和月榜元数据按
arXiv ID 合并,同时保留两个榜单各自的展示排名。
随后,它会在配置的历史窗口内扫描 `daily/<prior-date>/paper-*.md`,排除笔记 frontmatter 中已有的 ID。如果排重后 没有任何可选论文,Job
会明确失败。
### 2. 候选排序
Rank 使用 reciprocal-rank fusion(RRF):
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
论文缺少某个榜单排名时,该项贡献为零。候选按融合分、upvotes 和 arXiv ID 排序。有界候选池还会为标题或摘要命中 Agent
memory、memory retrieval、continual learning、context compression、knowledge graph、RAG 等记忆相关关键词的
论文保留若干位置。这个保留策略只是关键词启发式,不是语义分类器。
### 3. 精选论文
Claude Code 接收有界候选池,并返回结构化的 `PaperSelection`。实现要求恰好选择 `top_k` 个候选池内的唯一 ID, 且 rank
必须连续。输出不合法时,校验错误会反馈给 Agent 并重试一次;第二次仍不合法则 Job 失败。
### 4. 下载并解读 PDF
入选论文按顺序逐篇处理。每篇论文都会经过:
1. 校验当前支持的新版 arXiv ID 格式;
2. 下载并校验 PDF,或复用文件头为 `%PDF-` 的已有文件;
3. 使用 `pypdf` 提取文本、插入页码标记,并应用页数和字符数限制;
4. 请求 Claude Code 返回结构化的详细解读;
5. 写入规范化 frontmatter 和生成的 Markdown 正文。
当前提取器依赖可用的 PDF 文本层。扫描版或纯图片 PDF 会失败,因为没有 OCR fallback。提取内容超过配置限制时, 笔记会记录输入已被截断。
### 5. 生成简报与索引
Claude Code 会读取全部详细笔记并生成当日简报。代码会检查每篇源笔记的 wikilink;如有遗漏,会在写入前自动补齐。 随后,工作流根据当日
Markdown frontmatter 重建 `daily/YYYY-MM-DD.md`。
### 6. 可选的钉钉通知
最后一步会去掉 YAML frontmatter,把简报正文按顺序发送到每个已配置的钉钉群。未配置群会话 ID 时,该步骤无副作用
跳过。某个群发送失败不会阻止继续尝试其他群,所有发送完成后再汇总报告失败。
## 记忆与检索
独立配置按职责分离 agent wrapper:
- `daily_paper` 执行论文精选、解读和简报生成。它保留 Claude Code 的常规本地工具并禁用 `WebSearch`,但当前没有 配置记忆检索
Job。
- `dingtalk_wait` 运行后台钉钉 Agent,并把 `memory_search` 作为可调用工具。
- `memory` 通过 AgentScope 执行 auto-memory 和 auto-dream 中依赖 LLM 的步骤。它禁用内置 shell 和文件工具; 记忆变更只能经过
`daily_write`、`read`、`edit`、`write` 等更窄的 ReMe job。
内置配置的 `memory_search` 使用 BM25 检索 `daily/` 和 `digest/` 下的 Markdown。ReMe 的搜索步骤支持融合向量结果, 但本
cookbook 默认没有配置 embedding store,因此不会执行向量检索。`node_search` 是 auto-dream 内部使用的 digest 节点检索工具。
`index_update_loop` 会在服务启动时索引已有记忆文件,并持续监听这些目录的后续变化。修复派生 file store 或需要强制
完整重建索引时,可运行 `reindex`。`reindex` 不会删除源 Markdown 或 PDF。
`auto_memory` 根据调用方传入的对话消息和稳定 `session_id` 写入或更新一篇 daily note。`auto_dream` 扫描近期 daily
notes,把持久记忆单元整合到 `digest/`,并生成兴趣主题。这两个 job 在本 cookbook 中都是按需执行;当前没有配置 auto-dream
cron。钉钉 Agent 可以通过 `memory_search` 召回记忆,但对话结束后不会自动调用 `auto_memory`。
## 日期、重跑与幂等
- `date` 必须严格符合 `YYYY-MM-DD`。省略时使用应用配置时区中的今天;内置配置为 `Asia/Shanghai`。
- “昨日”表示 `date - 1 day`,不是模糊的最近 24 小时。
- `history_days` 只扫描此前的日期目录,不会把本次运行日纳入历史窗口。
- 如果 `daily/<date>/daily-paper-brief.md` 已存在且 `force=false`,采集、排序、模型调用、PDF 处理和简报生成都会
跳过;已有简报仍会交给钉钉通知步骤。
- `force=true` 会重新生成笔记和简报,但仍会复用已有且有效的 PDF。
每个 PDF、详细笔记和最终简报都会先写临时文件再替换,避免读取方看到半写状态。整个多文件工作流不是事务, 同一日期的并发运行也没有全局锁。
## 运行方式
独立配置中的主要 Job 如下:
| Job | 行为 |
|---------------------|---------------------------------------------------------------|
| `daily_paper` | 通过 CLI 或 HTTP 服务按需生成 |
| `daily_paper_cron` | 每天 08:00(`Asia/Shanghai`)执行相同 pipeline |
| `dingtalk_wait` | 由 supervisor 管理、具有 `memory_search` 能力的后台钉钉 Agent |
| `auto_memory` | 根据对话消息写入或更新 daily note |
| `auto_dream` | 把近期 daily notes 整理为 digest 记忆和兴趣主题 |
| `memory_search` | 对 daily 和 digest Markdown 执行 BM25 检索 |
| `reindex` | 根据已有记忆文件重建派生检索状态 |
| `index_update_loop` | 在服务模式下初始化并持续更新检索状态 |
`node_search`、`daily_list`、`daily_write`、`read`、`write`、`edit` 和 frontmatter 更新等辅助 job 构成 memory agent 使用的受约束工具。
### 一次性运行
快速开始中的命令会生成今天的简报。要生成指定日期并覆盖部分参数:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-07-21 \
top_k=3 \
history_days=30
```
重新生成已有简报的日期:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-07-21 force=true
```
需要查看响应 metadata 进行诊断时,可在一次性命令中加入 `service.show_metadata=true`。
### 常驻服务与 cron
启动独立 HTTP 服务以及定时、后台 Job:
```bash
reme start config=daily_cookbook
```
服务默认监听 `127.0.0.1:8001`,因此可以和默认 ReMe 服务并行运行。在另一个终端中通过 ReMe client 或 HTTP 调用按需任务:
```bash
reme daily_paper host=127.0.0.1 port=8001
```
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-07-21","top_k":3,"force":false}'
```
检索记忆、记录对话、整理记忆,或显式重建检索索引:
```bash
reme memory_search host=127.0.0.1 port=8001 query="agent memory" limit=5
reme auto_memory host=127.0.0.1 port=8001 \
session_id=example-session \
messages='[{"name":"user","role":"user","content":"I prefer concise paper summaries."}]'
reme auto_dream host=127.0.0.1 port=8001 date=2026-07-21
reme reindex host=127.0.0.1 port=8001
```
监听地址和调度时间可以在启动时覆盖:
```bash
reme start \
config=daily_cookbook \
service.host=0.0.0.0 \
service.port=8101 \
jobs.daily_paper_cron.cron="30 7 * * *"
```
## 配置
最常用的 Job 配置如下:
| 配置项 | 默认值 | 用途 |
|-------------------|-----------:|------------------------------------|
| `candidate_limit` | `20` | 送入精选阶段的最大论文数 |
| `memory_reserve` | `5` | 记忆关键词启发式保留的候选位置数 |
| `top_k` | `3` | 最终精选和解读的论文数 |
| `rrf_k` | `60` | RRF 常数 |
| `weekly_weight` | `0.7` | 周榜在融合排序中的权重 |
| `history_days` | `30` | 按 arXiv ID 排除近期推荐的时间窗口 |
| `hf_timeout` | `30` 秒 | Hugging Face 请求 timeout |
| `hf_max_retries` | `3` | Hugging Face 请求最多尝试次数 |
| `pdf_timeout` | `90` 秒 | arXiv 下载 timeout |
| `max_pdf_bytes` | `52428800` | PDF 大小上限(50 MiB) |
| `max_pdf_pages` | `80` | 最多提取的 PDF 页数 |
| `max_pdf_chars` | `240000` | 单篇论文送入模型的最大提取字符数 |
公开 Job 参数为 `date`、`force`、`top_k`、`weekly_weight` 和 `history_days`。调用时显式传入的值优先于 Job 默认值。
独立应用还支持以下环境变量:
| 环境变量 | 用途 |
|-----------------------------------------|----------------------------------------|
| `DAILY_PAPER_WORKSPACE_DIR` | 覆盖默认 `reme_workspace` |
| `DAILY_PAPER_PROJECT_PATH` | Claude Code 可见的仓库或项目路径 |
| `REME_PROXY_IP` | 可选 SSH 代理主机 |
| `REME_PROXY_ACCOUNT` | 可选 SSH 代理账户 |
| `DAILY_PAPER_HOST` / `DAILY_PAPER_PORT` | HTTP 监听地址 |
| `CLAUDE_CODE_API_KEY` | Claude Code endpoint 的 API key |
| `CLAUDE_CODE_MODEL_NAME` | Claude Code 模型;默认 `qwen3.7-max` |
| `CLAUDE_CODE_BASE_URL` | Claude Code 的 Anthropic 兼容 endpoint |
| `LLM_API_KEY` | AgentScope memory 模型的 API key |
| `LLM_MODEL_NAME` | memory 模型;默认 `qwen3.7-max` |
| `LLM_BASE_URL` | memory 模型的 Anthropic 兼容 endpoint |
`DAILY_PAPER_PROJECT_PATH` 默认是相对于 workspace 的 `..`。使用默认 `reme_workspace` 并从仓库根目录启动时, 它会解析回仓库根目录。如果
workspace 位于其他位置,请显式设置这两个路径。
ReMe 会从当前目录向上查找未提交的 `.env`,因此也可以把相同变量放在其中,而不是在 shell 中导出。
## 钉钉配置
钉钉是可选能力。仅在需要投递简报或运行后台钉钉 Agent 时配置:
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
只有主动投递简报需要 `DINGTALK_CONVERSATION_IDS`。后台 `dingtalk_wait` Job 使用前三项凭据,不使用群会话列表。
## 故障恢复与边界
| 场景 | 当前行为 |
|----------------------------|----------------------------------------------|
| Hugging Face 暂时失败 | 按指数间隔重试,最多尝试 `hf_max_retries` 次 |
| 没有 eligible 论文 | 在排序前失败 |
| `top_k` 或精选结果不合法 | 校验后失败;精选结果可重试一次 |
| PDF 太大、无效或没有文本层 | 在解读阶段停止 |
| PDF 超过页数或字符数限制 | 使用截断文本继续,并记录截断状态 |
| 某篇论文解读失败 | Job 停止;此前写入的 PDF 和笔记保留 |
| 简报遗漏源笔记链接 | 写入前自动补齐 wikilink |
| auto-dream 部分整合失败 | 成功单元保留,失败路径不会被 checkpoint |
恢复时,先检查该日期已有的笔记和 PDF,修复网络、凭据、模型或 PDF 问题,再使用相同日期和 `force=true` 重跑。 有效的缓存 PDF
会被复用。
内置 Claude Code 组件使用 `permission_mode: bypassPermissions`,并禁用 `WebSearch`。`dingtalk_wait` 可以调用本地
`memory_search`;`daily_paper` 当前没有配置 Job 工具。Analyze 和 Brief prompt 会限制 Agent 应读取的内容,但这些
步骤没有设置严格的逐次调用工具 allowlist,也不是操作系统级沙箱。AgentScope memory wrapper 禁用了内置 shell 和 文件系统工具,但其
ReMe job tools 运行于 bypass permission mode。请只在可信的项目和 workspace 中运行;用于共享 或生产环境前,应进一步收紧配置。
## 测试
聚焦的单元测试会 mock Hugging Face、arXiv、Claude Code 和钉钉边界:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```
真实运行会访问外部服务并可能产生模型费用,不应把它当作普通单元测试执行。

18
deploy/docker/example.env Normal file
View file

@ -0,0 +1,18 @@
# Copy to the repository's .env for Docker Compose; keep real credentials private.
# File operations and BM25 search work without model credentials.
LLM_API_KEY=
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_MODEL_NAME=qwen3.7-plus
# Host settings: create this directory before starting Compose.
REME_DATA_DIR=./.reme
REME_PUBLISHED_PORT=2333
# On Linux, set these to the outputs of `id -u` and `id -g` so files remain yours.
REME_UID=1000
REME_GID=1000
# Optional container settings.
# REME_TIMEZONE=Asia/Shanghai
# REME_CONFIG=/etc/reme/config.yaml
# REME_IMAGE=ghcr.io/agentscope-ai/reme:main

View file

@ -0,0 +1,100 @@
"""Container startup and health checks using ReMe's existing CLI contract."""
from __future__ import annotations
import json
import os
from pathlib import Path
import sys
from urllib.request import ProxyHandler, Request, build_opener
HEALTH_STATE_PATH = Path("/tmp/reme-health.json")
_ENV_OVERRIDES = {
"REME_CONFIG": "config",
"REME_WORKSPACE_DIR": "workspace_dir",
"REME_HOST": "service.host",
"REME_PORT": "service.port",
"REME_TIMEZONE": "timezone",
}
def start_command(arguments: list[str], environment: dict[str, str]) -> list[str]:
"""Add container defaults only to `start`; explicit CLI arguments win."""
from reme.config import deep_merge_config, parse_kwargs
defaults = parse_kwargs(
"log_to_file=false",
*[f"{key}={json.dumps(environment[name])}" for name, key in _ENV_OVERRIDES.items() if environment.get(name)],
)
# Docker environment values are strings; the HTTP service expects an integer port.
if environment.get("REME_PORT"):
defaults["service"]["port"] = int(environment["REME_PORT"])
overrides = deep_merge_config(defaults, parse_kwargs(*arguments))
return ["reme", "start", *[f"{key}={json.dumps(value, ensure_ascii=False)}" for key, value in overrides.items()]]
def write_health_state(command: list[str], state_path: Path = HEALTH_STATE_PATH) -> None:
"""Record only the effective HTTP address, never credentials or user data."""
from reme.components.service.cli_service import prepare_start_config
from reme.config import parse_kwargs
from reme.constants import REME_DEFAULT_HOST, REME_DEFAULT_PORT, normalize_connect_host
from reme.plugin import resolve_plugin_runtime
config = resolve_plugin_runtime(prepare_start_config(parse_kwargs(*command[2:]))).config
service = config.get("service") or {}
if service.get("backend") != "http":
return
host = normalize_connect_host(service.get("host") or REME_DEFAULT_HOST)
if host == "::":
host = "::1"
port = int(service.get("port", REME_DEFAULT_PORT))
if not 1 <= port <= 65535:
raise ValueError("service.port must be between 1 and 65535")
host = f"[{host}]" if ":" in host else host
with state_path.open("w", encoding="utf-8") as state_file:
os.chmod(state_path, 0o600)
json.dump({"url": f"http://{host}:{port}/health_check"}, state_file)
def healthcheck(state_path: Path = HEALTH_STATE_PATH) -> int:
"""Require both a successful Job and a healthy component snapshot."""
try:
state = json.loads(state_path.read_text(encoding="utf-8"))
request = Request(state["url"], data=b"{}", headers={"Content-Type": "application/json"}, method="POST")
# A deployment's outbound proxy must not intercept its local probe.
with build_opener(ProxyHandler({})).open(request, timeout=4) as response:
payload = json.load(response)
healthy = payload.get("metadata", {}).get("health", {}).get("healthy")
if payload.get("success") is True and healthy is True:
return 0
except (OSError, ValueError, KeyError, TypeError, AttributeError):
pass
print("ReMe HTTP health check failed; inspect the container logs and POST /health_check", file=sys.stderr)
return 1
def main() -> int:
"""Prepare startup, then replace this process so signals reach ReMe."""
arguments = sys.argv[1:]
if arguments == ["--healthcheck"]:
return healthcheck()
HEALTH_STATE_PATH.unlink(missing_ok=True)
Path(os.environ.get("HOME", "/tmp/reme-home")).mkdir(parents=True, exist_ok=True)
if not arguments:
arguments = ["start"]
start_actions = {"start", "-start", "--start"}
if len(arguments) >= 2 and arguments[0] == "reme" and arguments[1] in start_actions:
arguments = arguments[1:]
if arguments[0] in start_actions:
from reme.utils import load_env
load_env()
arguments = start_command(arguments[1:], dict(os.environ))
write_health_state(arguments)
os.execvp(arguments[0], arguments)
return 0
if __name__ == "__main__":
raise SystemExit(main())

24
docker-compose.yml Normal file
View file

@ -0,0 +1,24 @@
services:
reme:
image: ${REME_IMAGE:-reme:local}
build:
context: .
init: false # The image already runs tini.
user: "${REME_UID:-1000}:${REME_GID:-1000}"
ports:
- "${REME_BIND_ADDRESS:-127.0.0.1}:${REME_PUBLISHED_PORT:-2333}:${REME_PORT:-2333}"
volumes:
- type: bind
source: ${REME_DATA_DIR:-./.reme}
target: /data
bind:
create_host_path: false
env_file:
- path: ${REME_ENV_FILE:-.env}
required: false
environment:
REME_HOST: 0.0.0.0
REME_PORT: ${REME_PORT:-2333}
REME_WORKSPACE_DIR: /data
restart: unless-stopped
stop_grace_period: 60s

396
docs/.vitepress/config.mts Normal file
View file

@ -0,0 +1,396 @@
import fs from "node:fs";
import path from "node:path";
import { execFileSync } from "node:child_process";
import { fileURLToPath } from "node:url";
import { defineConfig, type DefaultTheme } from "vitepress";
import { legacyRoutes } from "./legacy-routes.mjs";
const sourceRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
const repositoryRoot = path.resolve(sourceRoot, "../../..");
const repository = "https://github.com/agentscope-ai/ReMe";
const base = process.env.DOCS_BASE || "/";
function readSourceMap(): Record<string, string> {
try {
return JSON.parse(fs.readFileSync(path.join(sourceRoot, ".source-map.json"), "utf8"));
} catch {
return {};
}
}
const sourceMap = readSourceMap();
function collectMarkdown(directory: string, root = directory): string[] {
const files: string[] = [];
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
if (entry.name.startsWith(".") || entry.name === "public" || entry.name === "figure") continue;
const absolute = path.join(directory, entry.name);
if (entry.isDirectory()) files.push(...collectMarkdown(absolute, root));
else if (entry.name.endsWith(".md")) files.push(path.relative(root, absolute).replaceAll(path.sep, "/"));
}
return files.sort();
}
function buildLlmsFiles(outDir: string) {
const pages = collectMarkdown(sourceRoot);
const index = [
"# ReMe Documentation",
"",
"> Local-first, file-native memory for agents.",
"",
...pages.map((relativePath) => {
const source = fs.readFileSync(path.join(sourceRoot, relativePath), "utf8");
const title = source.match(/^#\s+(.+)$/m)?.[1]
|| source.match(/^title:\s*(.+)$/m)?.[1]
|| path.basename(relativePath, ".md");
const route = relativePath.replace(/(?:^|\/)index\.md$/, "").replace(/\.md$/, "");
return `- [${title}](https://reme.agentscope.io/${route})`;
}),
"",
];
fs.writeFileSync(path.join(outDir, "llms.txt"), index.join("\n"), "utf8");
const full = ["# ReMe Documentation", ""];
for (const relativePath of pages) {
const source = fs.readFileSync(path.join(sourceRoot, relativePath), "utf8");
full.push(`<!-- source: ${sourcePathFor(relativePath)} -->`, "", source, "", "---", "");
const pageDir = path.join(outDir, relativePath.replace(/\.md$/, ""));
fs.mkdirSync(pageDir, { recursive: true });
fs.writeFileSync(path.join(pageDir, "llms.txt"), source, "utf8");
}
fs.writeFileSync(path.join(outDir, "llms-full.txt"), full.join("\n"), "utf8");
}
function sourcePathFor(relativePath: string) {
return sourceMap[relativePath] || `docs/${relativePath}`;
}
function sourceLastUpdated(relativePath: string): number | undefined {
const sourcePath = sourcePathFor(relativePath);
try {
const timestamp = execFileSync("git", ["log", "-1", "--format=%ct", "--", sourcePath], {
cwd: repositoryRoot,
encoding: "utf8",
}).trim();
if (timestamp) return Number(timestamp) * 1000;
} catch {
// Fall back to the canonical file timestamp outside a Git checkout.
}
try {
return fs.statSync(path.join(repositoryRoot, sourcePath)).mtimeMs;
} catch {
return undefined;
}
}
const legacyRedirectScript = `(() => {
const routes = ${JSON.stringify(legacyRoutes)};
const id = new URLSearchParams(window.location.search).get("doc");
const target = id && routes[id];
const base = ${JSON.stringify(base)};
if (target) {
const destination = /^https?:/.test(target)
? target
: base.replace(/\\/$/, "") + target;
window.location.replace(destination + window.location.hash);
return;
}
const root = base.endsWith("/") ? base : base + "/";
if (window.location.pathname === root) {
window.location.replace(root + "zh/" + window.location.hash);
}
})();`;
function nav(language: "zh" | "en"): DefaultTheme.NavItem[] {
const zh = language === "zh";
return [
{ text: zh ? "首页" : "Home", link: `/${language}/` },
{ text: zh ? "文档" : "Docs", link: `/${language}/quick_start` },
{ text: zh ? "体验Studio" : "Try Studio", link: `/studio/?lang=${language}`, target: "_self" },
{ text: zh ? "集成" : "Integrations", link: `/${language}/integrations` },
{ text: zh ? "插件" : "Plugins", link: `/${language}/plugin_management` },
{ text: zh ? "评测" : "Benchmarks", link: `/${language}/benchmarks/longmemeval` },
{ text: zh ? "博客" : "Blog", link: `/${language}/reme-blog` },
{ text: zh ? "常见问题" : "FAQ", link: `/${language}/faq` },
];
}
function docsSidebar(language: "zh" | "en"): DefaultTheme.SidebarItem[] {
const zh = language === "zh";
return [
{
text: zh ? "开始使用" : "Get Started",
collapsed: false,
items: [
{ text: zh ? "项目介绍" : "Introduction", link: `/${language}/overview` },
{ text: zh ? "快速开始" : "Quick Start", link: `/${language}/quick_start` },
{ text: zh ? "基础配置" : "Configuration", link: `/${language}/configuration` },
{ text: zh ? "服务与部署" : "Services and Deployment", link: `/${language}/services` },
{ text: zh ? "Docker 部署" : "Docker Deployment", link: `/${language}/docker` },
{ text: "ReMe Studio", link: `/${language}/workspace/studio` },
],
},
{
text: zh ? "核心概念" : "Core Concepts",
collapsed: false,
items: [
{ text: zh ? "文件即记忆" : "Memory as File", link: `/${language}/memory_as_file` },
{ text: zh ? "记忆检索" : "Memory Search", link: `/${language}/memory_search` },
{ text: zh ? "自动关联" : "Auto Link", link: `/${language}/auto_link` },
{ text: zh ? "应用场景" : "Application Scenarios", link: `/${language}/reme_scene` },
],
},
{
text: zh ? "记忆工作流" : "Memory Workflows",
collapsed: false,
items: [
{ text: "Auto Memory", link: `/${language}/auto_memory` },
{ text: "Auto Resource", link: `/${language}/auto_resource` },
{ text: "Auto Dream", link: `/${language}/auto_dream` },
{ text: "Proactive", link: `/${language}/proactive` },
],
},
{
text: zh ? "API 与运维" : "API and Operations",
collapsed: true,
items: [
{ text: "CLI", link: `/${language}/reference/cli` },
{ text: zh ? "Job API" : "Job API", link: `/${language}/reference/jobs` },
{ text: "HTTP / MCP", link: `/${language}/services#http-api` },
{ text: zh ? "诊断、备份与恢复" : "Diagnostics, Backup, and Recovery", link: `/${language}/operations` },
],
},
{
text: zh ? "开发者" : "Development",
collapsed: true,
items: [
{ text: zh ? "代码框架" : "Framework", link: `/${language}/framework` },
{ text: zh ? "开源与贡献" : "Contributing", link: `/${language}/contributing` },
],
},
];
}
function integrationsSidebar(language: "zh" | "en"): DefaultTheme.SidebarItem[] {
const zh = language === "zh";
return [{
text: zh ? "Agent 集成" : "Agent Integrations",
collapsed: false,
items: [
{ text: zh ? "集成总览" : "Overview", link: `/${language}/integrations` },
{ text: "Claude Code", link: `/${language}/integrations/claude-code` },
{ text: "Hermes Agent", link: `/${language}/integrations/hermes` },
{ text: "DeepSeek Harness", link: `/${language}/integrations/dsh` },
{ text: "OpenClaw", link: `/${language}/integrations/openclaw` },
],
}];
}
function pluginsSidebar(language: "zh" | "en"): DefaultTheme.SidebarItem[] {
const zh = language === "zh";
return [{
text: zh ? "插件" : "Plugins",
collapsed: false,
items: [
{ text: zh ? "插件管理" : "Plugin Management", link: `/${language}/plugin_management` },
{ text: zh ? "插件开发" : "Plugin Development", link: `/${language}/plugin_development` },
{ text: zh ? "每日论文" : "Daily Paper", link: `/${language}/plugins/daily-paper` },
{ text: "Auto Fin", link: `/${language}/plugins/auto-fin` },
{ text: "LME", link: `/${language}/plugins/lme` },
{ text: "BEAM", link: `/${language}/plugins/beam` },
],
}];
}
function benchmarksSidebar(language: "zh" | "en"): DefaultTheme.SidebarItem[] {
const zh = language === "zh";
return [{
text: zh ? "记忆能力评测" : "Memory Benchmarks",
collapsed: false,
items: [
{ text: "LongMemEval", link: `/${language}/benchmarks/longmemeval` },
{ text: "BEAM", link: `/${language}/benchmarks/beam` },
{ text: "π-Bench", link: `/${language}/benchmarks/pibench` },
{ text: "Tool Memory / ExpG", link: `/${language}/benchmarks/toolmemory` },
],
}];
}
function singlePageSidebar(language: "zh" | "en", page: "blog" | "faq"): DefaultTheme.SidebarItem[] {
const zh = language === "zh";
if (page === "blog") {
return [{
text: zh ? "ReMe 博客" : "ReMe Blog",
link: `/${language}/reme-blog`,
collapsed: false,
items: [
{ text: zh ? "ReMe介绍" : "About ReMe", link: `/${language}/reme-blog` },
{ text: zh ? "记忆标签" : "Memory Tags", link: `/${language}/blog_20260920` },
],
}];
}
return [{
text: zh ? "帮助" : "Help",
collapsed: false,
items: [{
text: zh ? "常见问题" : "Frequently Asked Questions",
link: `/${language}/faq`,
}],
}];
}
function sidebars(language: "zh" | "en"): DefaultTheme.SidebarMulti {
return {
[`/${language}/integrations`]: integrationsSidebar(language),
[`/${language}/workspace/`]: [],
[`/${language}/plugins/`]: pluginsSidebar(language),
[`/${language}/plugin_management`]: pluginsSidebar(language),
[`/${language}/plugin_development`]: pluginsSidebar(language),
[`/${language}/benchmarks/`]: benchmarksSidebar(language),
[`/${language}/reme-blog`]: singlePageSidebar(language, "blog"),
[`/${language}/blog_20260920`]: singlePageSidebar(language, "blog"),
[`/${language}/faq`]: singlePageSidebar(language, "faq"),
[`/${language}/`]: docsSidebar(language),
};
}
function configureRepositoryLinks(md: any) {
for (const ruleName of ["link_open", "image"] as const) {
const original = md.renderer.rules[ruleName];
md.renderer.rules[ruleName] = (tokens: any[], index: number, options: any, env: any, self: any) => {
const attribute = ruleName === "image" ? "src" : "href";
const token = tokens[index];
const attributeIndex = token.attrIndex(attribute);
const target = attributeIndex >= 0 ? token.attrs[attributeIndex][1] : "";
if (target && !/^(?:[a-z]+:|#|\/)/i.test(target)) {
const cleanTarget = target.split("#")[0].split("?")[0];
const generatedTarget = path.resolve(sourceRoot, path.dirname(env.relativePath), cleanTarget);
if (!fs.existsSync(generatedTarget)) {
const originalPage = sourcePathFor(env.relativePath);
const originalTarget = path.posix.normalize(path.posix.join(path.posix.dirname(originalPage), cleanTarget));
const suffix = target.slice(cleanTarget.length);
const url = ruleName === "image"
? `https://raw.githubusercontent.com/agentscope-ai/ReMe/main/${originalTarget}${suffix}`
: `${repository}/blob/main/${originalTarget}${suffix}`;
token.attrs[attributeIndex][1] = url;
}
}
return original ? original(tokens, index, options, env, self) : self.renderToken(tokens, index, options);
};
}
}
export default defineConfig({
lang: "zh-CN",
title: "ReMe",
description: "Local-first, file-native memory for agents",
base,
cleanUrls: true,
lastUpdated: true,
ignoreDeadLinks: [/^http:\/\/localhost(?::\d+)?(?:\/|$)/],
sitemap: {
hostname: "https://reme.agentscope.io",
transformItems(items) {
const isRoot = (url: string) => url.replace(/^\/+|\/+$/g, "") === "";
return items.filter((item) => !isRoot(item.url)).map((item) => {
const route = item.url.replace(/^\/+/, "");
const relativePath = !route || route.endsWith("/") ? `${route}index.md` : `${route}.md`;
const links = item.links?.filter((link) => !isRoot(link.url));
return { ...item, links, lastmod: sourceLastUpdated(relativePath) };
});
},
},
head: [
["link", { rel: "icon", type: "image/svg+xml", href: `${base}reme-icon.svg` }],
["meta", { name: "theme-color", content: "#087f6a", media: "(prefers-color-scheme: light)" }],
["meta", { name: "theme-color", content: "#0d1512", media: "(prefers-color-scheme: dark)" }],
["script", {
defer: "",
src: "https://cloud.umami.is/script.js",
"data-website-id": "8cafe9df-d883-4046-b5e9-36dfd21a4884",
"data-domains": "reme.agentscope.io",
}],
["script", {}, legacyRedirectScript],
],
markdown: {
config: configureRepositoryLinks,
},
transformPageData(pageData, { siteConfig }) {
const sourcePath = path.join(siteConfig.srcDir, pageData.relativePath);
pageData.frontmatter._sourcePath = sourcePathFor(pageData.relativePath);
pageData.lastUpdated = sourceLastUpdated(pageData.relativePath);
try {
pageData.frontmatter._rawMarkdown = fs.readFileSync(sourcePath, "utf8");
} catch {
pageData.frontmatter._rawMarkdown = "";
}
},
buildEnd(siteConfig) {
buildLlmsFiles(siteConfig.outDir);
},
themeConfig: {
logo: "/reme-icon.svg",
siteTitle: "ReMe",
nav: [
...nav("zh"),
{
text: "语言",
items: [
{ text: "简体中文", link: "/zh/" },
{ text: "English", link: "/en/" },
],
},
],
outline: { label: "页面导航", level: [2, 3] },
search: {
provider: "local",
options: {
locales: {
zh: {
translations: {
button: { buttonText: "搜索文档", buttonAriaLabel: "搜索文档" },
modal: {
noResultsText: "没有找到相关内容",
resetButtonTitle: "清除查询",
footer: { selectText: "选择", navigateText: "切换", closeText: "关闭" },
},
},
},
},
},
},
socialLinks: [{ icon: "github", link: repository }],
footer: {
message: "Released under the Apache-2.0 License.",
copyright: "Copyright ReMe contributors",
},
},
locales: {
zh: {
label: "简体中文",
lang: "zh-CN",
link: "/zh/",
themeConfig: {
nav: nav("zh"),
sidebar: sidebars("zh"),
outline: { label: "页面导航", level: [2, 3] },
docFooter: { prev: "上一页", next: "下一页" },
darkModeSwitchLabel: "外观",
sidebarMenuLabel: "菜单",
returnToTopLabel: "返回顶部",
langMenuLabel: "切换语言",
},
},
en: {
label: "English",
lang: "en-US",
link: "/en/",
themeConfig: {
nav: nav("en"),
sidebar: sidebars("en"),
outline: { label: "On this page", level: [2, 3] },
docFooter: { prev: "Previous page", next: "Next page" },
},
},
},
});

View file

@ -0,0 +1,47 @@
export const legacyRoutes = {
"readme-zh": "/zh/",
"readme-en": "/en/",
"zh-quick_start": "/zh/quick_start",
"en-quick_start": "/en/quick_start",
"zh-plugin_management": "/zh/plugin_management",
"en-plugin_management": "/en/plugin_management",
"zh-memory_as_file": "/zh/memory_as_file",
"en-memory_as_file": "/en/memory_as_file",
"zh-memory_search": "/zh/memory_search",
"en-memory_search": "/en/memory_search",
"zh-auto_memory": "/zh/auto_memory",
"en-auto_memory": "/en/auto_memory",
"zh-auto_resource": "/zh/auto_resource",
"en-auto_resource": "/en/auto_resource",
"zh-auto_link": "/zh/auto_link",
"en-auto_link": "/en/auto_link",
"zh-auto_dream": "/zh/auto_dream",
"en-auto_dream": "/en/auto_dream",
"zh-proactive": "/zh/proactive",
"en-proactive": "/en/proactive",
"zh-reme_scene": "/zh/reme_scene",
"en-reme_scene": "/en/reme_scene",
"zh-framework": "/zh/framework",
"en-framework": "/en/framework",
"zh-reme-blog": "/zh/reme-blog",
"en-reme-blog": "/en/reme-blog",
"zh-contributing": "/zh/contributing",
"en-contributing": "/en/contributing",
"typescript-zh": "/zh/integrations",
"typescript-en": "/en/integrations",
"studio-zh": "/zh/workspace/studio",
"studio-en": "/en/workspace/studio",
"daily-paper-zh": "/zh/plugins/daily-paper",
"daily-paper-en": "/en/plugins/daily-paper",
"auto-fin-zh": "/zh/plugins/auto-fin",
"auto-fin-en": "/en/plugins/auto-fin",
"beam-zh": "/zh/benchmarks/beam",
"beam-en": "/en/benchmarks/beam",
"longmemeval-zh": "/zh/benchmarks/longmemeval",
"longmemeval-en": "/en/benchmarks/longmemeval",
"pibench-zh": "/zh/benchmarks/pibench",
"pibench-en": "/en/benchmarks/pibench",
"toolmemory-zh": "/zh/benchmarks/toolmemory",
"toolmemory-en": "/en/benchmarks/toolmemory",
"agents-guide": "https://github.com/agentscope-ai/ReMe/blob/main/AGENTS.md",
};

View file

@ -0,0 +1,32 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import { useData } from "vitepress";
const { frontmatter, lang } = useData();
const copied = ref(false);
const label = computed(() => {
if (copied.value) return lang.value.startsWith("zh") ? "已复制" : "Copied";
return lang.value.startsWith("zh") ? "复制 Markdown" : "Copy Markdown";
});
async function copyMarkdown() {
const markdown = String(frontmatter.value._rawMarkdown || "");
if (!markdown) return;
await navigator.clipboard.writeText(markdown);
copied.value = true;
window.setTimeout(() => { copied.value = false; }, 1800);
}
</script>
<template>
<div class="copy-markdown-wrap">
<button class="copy-markdown" type="button" :class="{ copied }" @click="copyMarkdown">
<svg v-if="!copied" viewBox="0 0 24 24" aria-hidden="true">
<rect x="9" y="9" width="13" height="13" rx="2" />
<path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1" />
</svg>
<svg v-else viewBox="0 0 24 24" aria-hidden="true"><path d="m5 12 4 4L19 6" /></svg>
{{ label }}
</button>
</div>
</template>

View file

@ -0,0 +1,523 @@
<script setup lang="ts">
import { computed, onMounted, reactive } from "vue";
import { useData, withBase } from "vitepress";
const props = defineProps<{ lang: "zh" | "en" }>();
const repository = "https://github.com/agentscope-ai/ReMe";
const trafficShareBase = "https://cloud.umami.is/analytics/us/share/S1OZK1PSDLEpyiU5?date=30day&page=1";
const { isDark } = useData();
const stats = reactive({ stars: "3.4K+", forks: "293" });
const translations = {
zh: {
eyebrow: "LOCAL-FIRST · FILE-NATIVE",
title: "让 Agent 真正记住,\n让记忆始终属于你。",
lead: "ReMe 将对话、资料与经验沉淀为可读、可编辑、可检索、相互链接的本地文件,让不同 Agent 共享同一套长期记忆。",
quickStart: "快速开始",
learnMore: "了解 ReMe",
stars: "GitHub Stars",
forks: "Forks",
mapLabel: "OPEN ECOSYSTEM",
mapTitle: "ReMe 与开源生态",
capabilityTitle: "核心能力",
capabilities: [
{ mark: "▣", title: "文件即记忆", href: "/zh/memory_as_file", tone: "mint" },
{ mark: "✦", title: "自动记忆", href: "/zh/auto_memory", tone: "cyan" },
{ mark: "⌕", title: "混合检索", href: "/zh/memory_search", tone: "blue" },
{ mark: "⌁", title: "图谱关联", href: "/zh/auto_link", tone: "amber" },
],
agentTitle: "ReMe 接入 Agent",
agentGuide: "接入指南",
backendTitle: "检索引擎接入 ReMe",
backendGuide: "配置指南",
backendNote: "可选向量索引",
integrations: [
{ logo: "/ecosystem/qwenpaw.png", title: "QwenPaw", href: "https://github.com/agentscope-ai/QwenPaw", tone: "blue" },
{ logo: "/ecosystem/deepseek-harness.svg", title: "DeepSeek Harness", href: "https://github.com/deepseek-ai/deepseek-harness", tone: "mint" },
{ logo: "/ecosystem/openclaw.svg", title: "OpenClaw", href: "https://github.com/openclaw/openclaw", tone: "cyan" },
{ logo: "/ecosystem/claude-code.png", title: "Claude Code", href: "https://github.com/anthropics/claude-code", tone: "violet" },
{ logo: "/ecosystem/hermes.svg", title: "Hermes Agent", href: "https://github.com/NousResearch/hermes-agent", tone: "amber" },
],
backends: [
{ logo: "/ecosystem/zvec.ico", title: "Zvec", href: "https://github.com/alibaba/zvec", tone: "mint" },
{ logo: "/ecosystem/faiss.png", title: "FAISS", href: "https://github.com/facebookresearch/faiss", tone: "blue" },
],
benchmarkLabel: "02 / BENCHMARKS",
benchmarkTitle: "用真实评测,\n验证长期记忆",
benchmarkLead: "从跨会话检索到百万级上下文,ReMe 用可复现的公开基准验证长期记忆。",
benchmarkAction: "查看全部评测",
benchmarkNote: "仓库已发布参考结果 · Agentic score",
benchmarks: [
{ name: "LongMemEval", setting: "500 题 · cleaned-s", score: 89.4 },
{ name: "BEAM 100K", setting: "20 cases · 400 题", score: 66.1 },
{ name: "BEAM 1M", setting: "35 cases · 700 题", score: 65.0 },
],
piLabel: "π-Bench 主动性",
piDetail: "5 类用户画像的平均 PROC 得分",
piDelta: "较 NanoBot +2.4%",
sectionLabel: "03 / PRODUCTS & PLUGINS",
sectionTitle: "从记忆工作区,到自动研究",
sectionLead: "三个完整入口,把 ReMe 用到真实工作流中。",
products: [
{ mark: "▣", label: "WORKSPACE", title: "ReMe Studio", detail: "在本地 Web 工作区中浏览、编辑、搜索记忆,并探索 wikilink 图谱。", href: "/studio/?lang=zh", tone: "mint" },
{ mark: "◌", label: "DISCOVER", title: "Daily Paper", detail: "筛选值得阅读的论文,分析 PDF,并生成文件化笔记与五分钟简报。", href: "/zh/plugins/daily-paper", tone: "cyan" },
{ mark: "↗", label: "RESEARCH", title: "Auto Fin", detail: "连接最新财联社新闻与本地历史记忆,生成带 wikilink 的研究报告。", href: "/zh/plugins/auto-fin", tone: "amber" },
],
trafficLabel: "04 / OPEN METRICS",
trafficTitle: "公开、透明的访问趋势",
trafficDetail: "最近 30 天的页面浏览量与访问趋势,由 Umami 提供匿名统计。",
trafficAction: "打开完整数据页",
trafficFrameTitle: "ReMe 最近 30 天访问数据",
},
en: {
eyebrow: "LOCAL-FIRST · FILE-NATIVE",
title: "Memory for AI agents.\nFiles that remain yours.",
lead: "ReMe turns conversations, resources, and experience into readable, editable, searchable, interconnected local files—a shared long-term memory layer for every agent.",
quickStart: "Quick Start",
learnMore: "Meet ReMe",
stars: "GitHub Stars",
forks: "Forks",
mapLabel: "OPEN ECOSYSTEM",
mapTitle: "ReMe and the open ecosystem",
capabilityTitle: "CORE CAPABILITIES",
capabilities: [
{ mark: "▣", title: "Memory as files", href: "/en/memory_as_file", tone: "mint" },
{ mark: "✦", title: "Auto memory", href: "/en/auto_memory", tone: "cyan" },
{ mark: "⌕", title: "Hybrid search", href: "/en/memory_search", tone: "blue" },
{ mark: "⌁", title: "Linked graph", href: "/en/auto_link", tone: "amber" },
],
agentTitle: "ReMe for agents",
agentGuide: "Integration guide",
backendTitle: "Retrieval for ReMe",
backendGuide: "Configuration guide",
backendNote: "Optional vector indexes",
integrations: [
{ logo: "/ecosystem/qwenpaw.png", title: "QwenPaw", href: "https://github.com/agentscope-ai/QwenPaw", tone: "blue" },
{ logo: "/ecosystem/deepseek-harness.svg", title: "DeepSeek Harness", href: "https://github.com/deepseek-ai/deepseek-harness", tone: "mint" },
{ logo: "/ecosystem/openclaw.svg", title: "OpenClaw", href: "https://github.com/openclaw/openclaw", tone: "cyan" },
{ logo: "/ecosystem/claude-code.png", title: "Claude Code", href: "https://github.com/anthropics/claude-code", tone: "violet" },
{ logo: "/ecosystem/hermes.svg", title: "Hermes Agent", href: "https://github.com/NousResearch/hermes-agent", tone: "amber" },
],
backends: [
{ logo: "/ecosystem/zvec.ico", title: "Zvec", href: "https://github.com/alibaba/zvec", tone: "mint" },
{ logo: "/ecosystem/faiss.png", title: "FAISS", href: "https://github.com/facebookresearch/faiss", tone: "blue" },
],
benchmarkLabel: "02 / BENCHMARKS",
benchmarkTitle: "Memory that holds up\nunder pressure",
benchmarkLead: "From cross-session retrieval to million-token context, ReMe validates long-term memory with reproducible public benchmarks.",
benchmarkAction: "Explore all benchmarks",
benchmarkNote: "Published reference runs · Agentic score",
benchmarks: [
{ name: "LongMemEval", setting: "500 questions · cleaned-s", score: 89.4 },
{ name: "BEAM 100K", setting: "20 cases · 400 questions", score: 66.1 },
{ name: "BEAM 1M", setting: "35 cases · 700 questions", score: 65.0 },
],
piLabel: "π-Bench proactivity",
piDetail: "Average PROC score across five personas",
piDelta: "+2.4% over NanoBot",
sectionLabel: "03 / PRODUCTS & PLUGINS",
sectionTitle: "From memory workspace to automated research",
sectionLead: "Three complete paths for putting ReMe into real workflows.",
products: [
{ mark: "▣", label: "WORKSPACE", title: "ReMe Studio", detail: "Browse, edit, and search memory in a local web workspace, then explore its wikilink graph.", href: "/studio/?lang=en", tone: "mint" },
{ mark: "◌", label: "DISCOVER", title: "Daily Paper", detail: "Select useful papers, analyze PDFs, and create file-native notes plus a five-minute brief.", href: "/en/plugins/daily-paper", tone: "cyan" },
{ mark: "↗", label: "RESEARCH", title: "Auto Fin", detail: "Connect recent CLS news with local memory to create traceable, wikilink-backed reports.", href: "/en/plugins/auto-fin", tone: "amber" },
],
trafficLabel: "04 / OPEN METRICS",
trafficTitle: "Public, transparent traffic",
trafficDetail: "Page views and traffic trends from the last 30 days, measured anonymously with Umami.",
trafficAction: "Open the full report",
trafficFrameTitle: "ReMe traffic for the last 30 days",
},
} as const;
const text = computed(() => translations[props.lang]);
const trafficShareUrl = computed(() => `${trafficShareBase}&theme=${isDark.value ? "dark" : "light"}`);
const localLink = (href: string) => withBase(href);
function normalizeCompactCount(value: string) {
const normalized = value.trim().toUpperCase();
return /[KMB]$/.test(normalized) ? `${normalized}+` : normalized;
}
async function readBadge(metric: "stars" | "forks") {
const response = await fetch(`https://img.shields.io/github/${metric}/agentscope-ai/ReMe.json`);
if (!response.ok) throw new Error(`Unable to load ${metric}`);
const payload = await response.json();
return normalizeCompactCount(String(payload.message || payload.value || ""));
}
onMounted(async () => {
const [stars, forks] = await Promise.allSettled([readBadge("stars"), readBadge("forks")]);
if (stars.status === "fulfilled" && stars.value) stats.stars = stars.value;
if (forks.status === "fulfilled" && forks.value) stats.forks = forks.value;
});
</script>
<template>
<div class="reme-home" :class="{ 'is-zh': lang === 'zh' }">
<section class="home-stage">
<div class="hero-copy">
<p class="eyebrow">{{ text.eyebrow }}</p>
<h1>{{ text.title }}</h1>
<p class="hero-lead">{{ text.lead }}</p>
<div class="hero-actions">
<a class="action primary" :href="localLink(`/${lang}/quick_start`)">{{ text.quickStart }} <span>→</span></a>
<a class="action secondary" :href="localLink(`/${lang}/overview`)">{{ text.learnMore }} <span>↗</span></a>
</div>
<div class="repo-stats" aria-live="polite">
<a :href="`${repository}/stargazers`" target="_blank" rel="noreferrer" :aria-label="`${stats.stars} ${text.stars}`">
<span class="stat-icon">☆</span>
<span><strong>{{ stats.stars }}</strong><small>{{ text.stars }}</small></span>
</a>
<a :href="`${repository}/forks`" target="_blank" rel="noreferrer" :aria-label="`${stats.forks} ${text.forks}`">
<span class="stat-icon fork-icon">⑂</span>
<span><strong>{{ stats.forks }}</strong><small>{{ text.forks }}</small></span>
</a>
</div>
</div>
<div class="ecosystem-map">
<div class="map-heading">
<div>
<span>{{ text.mapLabel }}</span>
<strong>{{ text.mapTitle }}</strong>
</div>
</div>
<div class="ecosystem-network">
<div class="network-brands">
<div class="network-group-label">
<strong>{{ text.agentTitle }}</strong>
<a :href="localLink(`/${lang}/integrations`)">{{ text.agentGuide }} ↗</a>
</div>
<div class="brand-viewport">
<div class="brand-reel">
<a
v-for="integration in text.integrations"
:key="integration.title"
class="brand-link"
:class="integration.tone"
:href="integration.href"
target="_blank"
rel="noopener noreferrer"
:aria-label="`${integration.title} GitHub`"
>
<span class="brand-mark" aria-hidden="true"><img :src="localLink(integration.logo)" alt="" /></span>
<strong>{{ integration.title }}</strong>
</a>
<div v-for="integration in text.integrations" :key="`${integration.title}-clone`" class="brand-link reel-clone" :class="integration.tone" aria-hidden="true">
<span class="brand-mark"><img :src="localLink(integration.logo)" alt="" /></span>
<strong>{{ integration.title }}</strong>
</div>
</div>
</div>
<div class="network-group-label backend-label">
<strong>{{ text.backendTitle }}</strong>
<a :href="localLink(`/${lang}/memory_search#${lang === 'zh' ? '向量索引后端' : 'vector-index-backends'}`)">{{ text.backendGuide }} ↗</a>
</div>
<div class="backend-links">
<a v-for="backend in text.backends" :key="backend.title" class="brand-link" :class="backend.tone" :href="backend.href" target="_blank" rel="noopener noreferrer" :aria-label="`${backend.title} GitHub`">
<span class="brand-mark" aria-hidden="true"><img :src="localLink(backend.logo)" alt="" /></span>
<strong>{{ backend.title }}</strong>
</a>
</div>
</div>
<div class="network-center" aria-hidden="true">
<span class="network-ring"></span>
<span class="network-core"><img :src="localLink('/reme-icon.svg')" alt="" /></span>
<strong>ReMe</strong>
</div>
<div class="network-capabilities">
<strong class="capability-label">{{ text.capabilityTitle }}</strong>
<a v-for="capability in text.capabilities" :key="capability.title" class="capability-link" :class="capability.tone" :href="localLink(capability.href)">
<span aria-hidden="true">{{ capability.mark }}</span>
<strong>{{ capability.title }}</strong>
<span aria-hidden="true">↗</span>
</a>
</div>
</div>
</div>
</section>
<section class="benchmark-section page-panel">
<div class="benchmark-intro">
<p class="section-label">{{ text.benchmarkLabel }}</p>
<h2>{{ text.benchmarkTitle }}</h2>
<p>{{ text.benchmarkLead }}</p>
<a :href="localLink(`/${lang}/benchmarks/longmemeval`)">{{ text.benchmarkAction }} <span>→</span></a>
</div>
<div class="benchmark-board">
<div class="benchmark-board-head">
<span>{{ text.benchmarkNote }}</span>
<span>0—100%</span>
</div>
<div class="benchmark-chart">
<div v-for="benchmark in text.benchmarks" :key="benchmark.name" class="benchmark-row">
<div class="benchmark-name">
<strong>{{ benchmark.name }}</strong>
<small>{{ benchmark.setting }}</small>
</div>
<div class="benchmark-track" aria-hidden="true">
<span :style="{ width: `${benchmark.score}%` }"></span>
</div>
<strong class="benchmark-score">{{ benchmark.score.toFixed(1) }}%</strong>
</div>
</div>
<a class="pi-score" :href="localLink(`/${lang}/benchmarks/pibench`)" :aria-label="`${text.piLabel}: 0.580`">
<span class="pi-symbol">π</span>
<span>
<small>{{ text.piLabel }}</small>
<strong>0.580</strong>
<em>{{ text.piDetail }}</em>
</span>
<b>{{ text.piDelta }} ↗</b>
</a>
</div>
</section>
<section class="product-section">
<div class="section-heading">
<p class="section-label">{{ text.sectionLabel }}</p>
<div>
<h2>{{ text.sectionTitle }}</h2>
<p>{{ text.sectionLead }}</p>
</div>
</div>
<div class="product-grid">
<a
v-for="product in text.products"
:key="product.title"
class="product-card"
:class="product.tone"
:href="localLink(product.href)"
:target="product.href.startsWith('/studio/') ? '_self' : undefined"
>
<span class="product-mark">{{ product.mark }}</span>
<span class="product-label">{{ product.label }}</span>
<strong>{{ product.title }}</strong>
<p>{{ product.detail }}</p>
<span class="product-arrow">→</span>
</a>
</div>
</section>
<section class="traffic-section page-panel">
<div class="traffic-heading">
<p class="section-label">{{ text.trafficLabel }}</p>
<h2>{{ text.trafficTitle }}</h2>
<p>{{ text.trafficDetail }}</p>
<a :href="localLink(`/${lang}/traffic`)">{{ text.trafficAction }} <span>→</span></a>
</div>
<div class="traffic-window">
<div class="traffic-window-bar" aria-hidden="true">
<span></span><span></span><span></span><b>reme.agentscope.io · 30 days</b>
</div>
<iframe :src="trafficShareUrl" :title="text.trafficFrameTitle" loading="lazy" referrerpolicy="no-referrer" />
</div>
</section>
</div>
</template>
<style scoped>
.reme-home {
--home-ink: #17231e;
--home-muted: #66736d;
--home-line: #d8e2dc;
--home-accent: #087f6a;
--home-surface: #ffffff;
--home-surface-soft: #f7f9f7;
--home-glass: rgba(255, 255, 255, 0.72);
--home-tile: rgba(255, 255, 255, 0.88);
--home-primary-bg: #17241e;
--home-primary-text: #ffffff;
--home-shadow: rgba(28, 57, 45, 0.12);
--section-light: #f8faf8;
--section-tint: #edf4f1;
max-width: 1720px;
margin: 0 auto;
padding: 0 clamp(24px, 4.5vw, 72px) 80px;
color: var(--home-ink);
}
.home-stage {
position: relative;
display: grid;
grid-template-columns: minmax(0, 1fr) 650px;
gap: clamp(42px, 4vw, 68px);
align-items: center;
min-height: calc(100vh - 64px);
padding: 72px 0 82px;
}
.home-stage::before {
position: absolute;
z-index: -1;
inset: 0 calc(50% - 50vw);
background:
radial-gradient(ellipse 70% 105% at -8% 18%, rgba(21, 158, 126, 0.14), transparent 72%),
radial-gradient(ellipse 68% 105% at 108% 10%, rgba(77, 103, 211, 0.13), transparent 73%),
linear-gradient(115deg, #f7fbf8 0%, #fbfaf6 49%, #f7f8fd 100%);
content: "";
}
.eyebrow, .section-label { margin: 0; color: var(--home-accent); font: 750 13px/1.4 var(--vp-font-family-mono); letter-spacing: 0.16em; }
.hero-copy h1 { max-width: 100%; margin: 23px 0 0; color: var(--home-ink); font: 760 clamp(52px, 4.2vw, 76px)/1.04 Georgia, "Times New Roman", serif; white-space: pre-wrap; letter-spacing: -0.052em; }
.is-zh .hero-copy h1 { max-width: 760px; font-size: clamp(52px, 3.6vw, 64px); white-space: pre-line; word-break: keep-all; }
.hero-lead { max-width: 650px; margin: 28px 0 0; color: var(--home-muted); font-size: clamp(17px, 1.3vw, 20px); line-height: 1.75; }
.hero-actions { display: flex; flex-wrap: wrap; gap: 12px; margin-top: 34px; }
.action { display: inline-flex; align-items: center; justify-content: space-between; gap: 28px; min-width: 166px; min-height: 54px; padding: 0 19px; border: 1px solid var(--home-line); border-radius: 12px; color: var(--home-ink); background: var(--home-glass); text-decoration: none; font-weight: 720; box-shadow: 0 8px 22px color-mix(in srgb, var(--home-shadow) 50%, transparent); transition: transform 160ms ease, box-shadow 160ms ease; }
.action.primary { border-color: var(--home-primary-bg); color: var(--home-primary-text); background: var(--home-primary-bg); box-shadow: 0 12px 26px color-mix(in srgb, var(--home-primary-bg) 28%, transparent); }
.action:hover { transform: translateY(-2px); box-shadow: 0 15px 28px var(--home-shadow); }
.repo-stats { display: flex; flex-wrap: wrap; gap: 34px; margin-top: 40px; }
.repo-stats a { display: flex; gap: 12px; align-items: flex-start; color: inherit; text-decoration: none; }
.stat-icon { color: var(--home-accent); font-size: 30px; line-height: 1; }
.fork-icon { transform: rotate(90deg); }
.repo-stats strong { display: block; font: 740 28px/1 var(--vp-font-family-mono); letter-spacing: -0.04em; }
.repo-stats small { display: block; margin-top: 8px; color: var(--home-muted); font-size: 13px; }
.ecosystem-map { position: relative; min-width: 0; }
.map-heading { margin-bottom: 25px; }
.map-heading > div { display: flex; min-width: 0; flex-direction: column; gap: 8px; }
.map-heading span { color: var(--home-accent); font: 700 10px/1.4 var(--vp-font-family-mono); letter-spacing: 0.12em; }
.map-heading strong { font-size: 18px; line-height: 1.3; white-space: nowrap; }
.ecosystem-network { position: relative; display: grid; grid-template-columns: minmax(0, 1.15fr) minmax(100px, 0.62fr) minmax(0, 1fr); gap: 12px; align-items: center; min-height: 370px; }
.ecosystem-network::before, .ecosystem-network::after { position: absolute; z-index: 0; top: 50%; width: 19%; border-top: 1px dashed color-mix(in srgb, var(--home-accent) 58%, var(--home-line)); content: ""; }
.ecosystem-network::before { left: 30%; }.ecosystem-network::after { right: 27%; }
.network-brands, .network-center, .network-capabilities { position: relative; z-index: 1; min-width: 0; }
.network-group-label { display: flex; align-items: baseline; justify-content: space-between; gap: 5px; margin-bottom: 7px; }
.network-group-label strong, .capability-label { color: var(--home-muted); font: 750 10px/1.3 var(--vp-font-family-mono); letter-spacing: 0.04em; }
.network-group-label a { flex: none; color: var(--home-accent); font-size: 10px; font-weight: 700; text-decoration: none; white-space: nowrap; }
.network-group-label a:hover { text-decoration: underline; }
.brand-viewport { height: 184px; overflow: hidden; mask-image: linear-gradient(transparent, #000 12%, #000 88%, transparent); }
.brand-reel { display: grid; grid-auto-rows: 46px; gap: 6px; animation: brand-scroll 19s linear infinite; }
.brand-viewport:hover .brand-reel { animation-play-state: paused; }
.brand-viewport:focus-within { overflow-y: auto; mask-image: none; }
.brand-viewport:focus-within .brand-reel { animation: none; }
.brand-link { display: flex; min-width: 0; height: 46px; align-items: center; gap: 7px; padding: 5px; border: 1px solid color-mix(in srgb, var(--card-accent) 26%, var(--home-line)); border-radius: 9px; color: var(--home-ink); background: var(--home-tile); text-decoration: none; transition: border-color 160ms ease, transform 160ms ease; }
.brand-link:hover { border-color: var(--card-accent); transform: translateX(2px); }
.brand-mark { display: grid; width: 32px; height: 32px; flex: none; place-items: center; overflow: hidden; border: 1px solid var(--home-line); border-radius: 7px; background: #fff; }
.brand-mark img { width: 27px; height: 27px; margin: 0; object-fit: contain; }
.brand-link strong { min-width: 0; font-size: 13px; line-height: 1.15; overflow-wrap: anywhere; }
.reel-clone { pointer-events: none; }
.backend-label { margin-top: 13px; }
.backend-links { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 5px; }
.backend-links .brand-link { height: 42px; gap: 5px; }
.backend-links .brand-mark { width: 28px; height: 28px; }
.backend-links .brand-mark img { width: 23px; height: 23px; }
.network-center { display: flex; min-height: 164px; flex-direction: column; align-items: center; justify-content: center; gap: 16px; }
.network-ring { position: absolute; top: 50%; left: 50%; width: 108px; height: 108px; border: 1px solid color-mix(in srgb, var(--home-accent) 30%, var(--home-line)); border-radius: 50%; transform: translate(-50%, -64%); animation: hub-pulse 3.6s ease-in-out infinite; }
.network-ring::after { position: absolute; inset: 10px; border: 1px solid var(--home-line); border-radius: 50%; content: ""; }
.network-core { z-index: 1; display: grid; width: 64px; height: 64px; place-items: center; border: 1px solid var(--home-line); border-radius: 50%; background: var(--home-surface); box-shadow: 0 10px 28px var(--home-shadow); }
.network-core img { width: 42px; height: 42px; margin: 0; }
.network-center strong { z-index: 1; color: var(--home-accent); font: 750 12px/1 var(--vp-font-family-mono); }
.network-capabilities { display: grid; gap: 8px; }
.capability-label { display: block; margin-bottom: 2px; }
.capability-link { display: flex; min-height: 53px; align-items: center; gap: 7px; padding: 7px; border: 1px solid color-mix(in srgb, var(--card-accent) 30%, var(--home-line)); border-radius: 10px; color: var(--home-ink); background: radial-gradient(circle at 100% 0, color-mix(in srgb, var(--card-accent) 11%, transparent), transparent 70%), var(--home-tile); text-decoration: none; transition: transform 160ms ease, border-color 160ms ease; }
.capability-link:hover { border-color: var(--card-accent); transform: translateX(2px); }
.capability-link span:first-child { display: grid; width: 27px; height: 27px; flex: none; place-items: center; border-radius: 7px; color: var(--card-accent); background: color-mix(in srgb, var(--card-accent) 12%, var(--home-surface)); font-size: 16px; }
.capability-link strong { min-width: 0; font-size: 13px; line-height: 1.2; }
.capability-link span:last-child { margin-left: auto; color: var(--card-accent); font-size: 13px; }
.brand-link:focus-visible, .capability-link:focus-visible, .network-group-label a:focus-visible { outline: 2px solid var(--home-accent); outline-offset: 2px; }
.mint { --card-accent: #059b7f; }.cyan { --card-accent: #169cc4; }.blue { --card-accent: #536bd8; }.amber { --card-accent: #c48324; }.violet { --card-accent: #7654c2; }
@keyframes brand-scroll { to { transform: translateY(-260px); } }
@keyframes hub-pulse { 50% { transform: translate(-50%, -64%) scale(1.12); opacity: 0.55; } }
.page-panel { position: relative; min-height: calc(100vh - 64px); }
.benchmark-section { display: grid; grid-template-columns: minmax(310px, 0.72fr) minmax(560px, 1.28fr); gap: clamp(54px, 7vw, 110px); align-items: center; padding: 112px 0 120px; color: var(--home-ink); }
.benchmark-section::before { position: absolute; z-index: -1; inset: 0 calc(50% - 50vw); border-top: 1px solid var(--home-line); background: var(--section-tint); content: ""; }
.benchmark-intro .section-label { color: var(--home-accent); }
.benchmark-intro h2 { max-width: 620px; margin: 23px 0 0; color: var(--home-ink); font-size: clamp(42px, 4.2vw, 68px); line-height: 1.06; white-space: pre-line; letter-spacing: -0.052em; }
.is-zh .benchmark-intro h2 { word-break: keep-all; }
.benchmark-intro > p:not(.section-label) { max-width: 560px; margin: 24px 0 0; color: var(--home-muted); font-size: 17px; line-height: 1.75; }
.benchmark-intro > a, .traffic-heading > a { display: inline-flex; align-items: center; gap: 32px; min-height: 50px; margin-top: 34px; padding: 0 18px; border: 1px solid color-mix(in srgb, var(--home-ink) 24%, transparent); border-radius: 11px; color: var(--home-ink); text-decoration: none; font-weight: 720; transition: background 160ms ease, transform 160ms ease; }
.benchmark-intro > a:hover, .traffic-heading > a:hover { background: color-mix(in srgb, var(--home-ink) 6%, transparent); transform: translateY(-2px); }
.benchmark-board { padding: clamp(24px, 3vw, 38px); border: 1px solid rgba(255, 255, 255, 0.14); border-radius: 26px; color: white; background: #15352b; box-shadow: 0 28px 64px rgba(24, 58, 45, 0.2); }
.benchmark-board-head { display: flex; justify-content: space-between; gap: 20px; padding-bottom: 22px; border-bottom: 1px solid rgba(255, 255, 255, 0.13); color: #94aca3; font: 700 11px/1.4 var(--vp-font-family-mono); letter-spacing: 0.1em; text-transform: uppercase; }
.benchmark-chart { display: grid; gap: 28px; padding: 32px 0; }
.benchmark-row { display: grid; grid-template-columns: 150px minmax(140px, 1fr) 72px; gap: 20px; align-items: center; }
.benchmark-name strong { display: block; color: white; font-size: 16px; }
.benchmark-name small { display: block; margin-top: 5px; color: #8fa69d; font-size: 12px; }
.benchmark-track { height: 10px; overflow: hidden; border-radius: 99px; background: rgba(255, 255, 255, 0.09); }
.benchmark-track span { display: block; height: 100%; border-radius: inherit; background: linear-gradient(90deg, #38d3ae, #85ead2); box-shadow: 0 0 22px rgba(66, 220, 182, 0.28); }
.benchmark-score { color: white; font: 740 20px/1 var(--vp-font-family-mono); text-align: right; }
.pi-score { display: grid; grid-template-columns: 58px minmax(0, 1fr) auto; gap: 18px; align-items: center; padding: 20px; border: 1px solid rgba(142, 160, 255, 0.27); border-radius: 18px; color: white; background: linear-gradient(110deg, rgba(82, 105, 216, 0.23), rgba(82, 105, 216, 0.08)); text-decoration: none; }
.pi-symbol { display: grid; place-items: center; width: 58px; height: 58px; border-radius: 15px; color: #b9c6ff; background: rgba(107, 129, 236, 0.18); font: 700 31px/1 Georgia, serif; }
.pi-score small, .pi-score em { display: block; color: #aabbb4; font-size: 12px; font-style: normal; }
.pi-score strong { display: block; margin: 4px 0; font: 750 25px/1 var(--vp-font-family-mono); }
.pi-score b { color: #a9b7ff; font-size: 13px; white-space: nowrap; }
.product-section { position: relative; display: flex; min-height: calc(100vh - 64px); flex-direction: column; justify-content: center; padding: 112px 0 120px; }
.product-section::before { position: absolute; z-index: -1; inset: 0 calc(50% - 50vw); border-top: 1px solid var(--home-line); background: var(--section-light); content: ""; }
.section-heading { display: grid; grid-template-columns: minmax(190px, 0.38fr) minmax(0, 1fr); gap: 40px; align-items: start; margin-bottom: 36px; }
.section-heading h2, .traffic-heading h2 { max-width: 820px; margin: 0; color: var(--home-ink); font-size: clamp(38px, 3.6vw, 58px); line-height: 1.1; letter-spacing: -0.045em; }
.section-heading p:not(.section-label) { margin: 15px 0 0; color: var(--home-muted); font-size: 16px; }
.product-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 18px; }
.product-card { position: relative; display: flex; min-height: 360px; flex-direction: column; padding: 30px; overflow: hidden; border: 1px solid color-mix(in srgb, var(--card-accent) 30%, var(--home-line)); border-radius: 22px; color: var(--home-ink); background: radial-gradient(circle at 80% 0, color-mix(in srgb, var(--card-accent) 14%, transparent), transparent 50%), linear-gradient(150deg, var(--home-surface), color-mix(in srgb, var(--card-accent) 5%, var(--home-surface-soft))); text-decoration: none; transition: transform 170ms ease, box-shadow 170ms ease; }
.product-card:hover { transform: translateY(-4px); box-shadow: 0 20px 38px color-mix(in srgb, var(--card-accent) 14%, transparent); }
.product-mark { display: grid; place-items: center; width: 46px; height: 46px; border-radius: 13px; color: var(--card-accent); background: color-mix(in srgb, var(--card-accent) 13%, var(--home-surface)); font: 650 24px/1 var(--vp-font-family-mono); }
.product-label { margin-top: 50px; color: var(--card-accent); font: 720 11px/1.3 var(--vp-font-family-mono); letter-spacing: 0.12em; }
.product-card strong { margin-top: 10px; font-size: 23px; }
.product-card p { max-width: 390px; margin: 13px 0 0; color: var(--home-muted); font-size: 15px; line-height: 1.65; }
.product-arrow { position: absolute; right: 23px; bottom: 20px; color: var(--card-accent); font-size: 22px; }
.traffic-section { display: grid; grid-template-columns: minmax(300px, 0.64fr) minmax(600px, 1.36fr); gap: clamp(48px, 6vw, 94px); align-items: center; padding: 112px 0 116px; color: var(--home-ink); }
.traffic-section::before { position: absolute; z-index: -1; inset: 0 calc(50% - 50vw); border-top: 1px solid var(--home-line); background: var(--section-tint); content: ""; }
.traffic-heading .section-label { color: var(--home-accent); }
.traffic-heading h2 { margin-top: 22px; color: var(--home-ink); }
.traffic-heading > p:not(.section-label) { max-width: 460px; margin: 22px 0 0; color: var(--home-muted); font-size: 17px; line-height: 1.7; }
.traffic-window { overflow: hidden; height: min(650px, calc(100vh - 170px)); min-height: 520px; border: 1px solid rgba(255, 255, 255, 0.2); border-radius: 24px; background: white; box-shadow: 0 34px 80px rgba(0, 0, 0, 0.3); }
.traffic-window-bar { display: flex; align-items: center; gap: 8px; height: 46px; padding: 0 16px; border-bottom: 1px solid #e5e8e7; background: #f6f8f7; }
.traffic-window-bar span { width: 9px; height: 9px; border-radius: 50%; background: #a8b4af; }
.traffic-window-bar span:first-child { background: #f08d78; }
.traffic-window-bar span:nth-child(2) { background: #e6bf67; }
.traffic-window-bar span:nth-child(3) { background: #69bd9a; }
.traffic-window-bar b { margin-left: 8px; color: #7b8983; font: 650 11px/1 var(--vp-font-family-mono); }
.traffic-window iframe { width: 100%; height: calc(100% - 46px); border: 0; }
:global(html.dark .reme-home) { --home-ink: #edf7f3; --home-muted: #a8bbb3; --home-line: #2d4038; --home-accent: #57dfc3; --home-surface: #14201b; --home-surface-soft: #101a16; --home-glass: rgba(17, 28, 23, 0.78); --home-tile: rgba(20, 32, 27, 0.92); --home-primary-bg: #57dfc3; --home-primary-text: #07120e; --home-shadow: rgba(0, 0, 0, 0.3); --section-light: #0d1512; --section-tint: #14201b; color-scheme: dark; }
:global(html.dark .home-stage::before) { background: radial-gradient(ellipse 70% 105% at -8% 18%, rgba(24, 169, 143, 0.13), transparent 72%), radial-gradient(ellipse 68% 105% at 108% 10%, rgba(74, 100, 218, 0.15), transparent 73%), linear-gradient(115deg, #0d1713 0%, #101713 49%, #10131c 100%); }
:global(html.dark .benchmark-board) { background: #0b1712; box-shadow: 0 28px 64px rgba(0, 0, 0, 0.32); }
@media (max-width: 1680px) {
.home-stage { grid-template-columns: 1fr; min-height: auto; }
.hero-copy { max-width: 800px; padding-top: 26px; }
.hero-copy h1 { white-space: pre-wrap; }
.ecosystem-map { max-width: 850px; }
}
@media (max-width: 1320px) {
.benchmark-section, .traffic-section { grid-template-columns: 1fr; min-height: auto; }
.benchmark-intro, .traffic-heading { max-width: 720px; }
.traffic-window { width: 100%; max-width: 1000px; }
}
@media (max-width: 1000px) {
.product-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
}
@media (max-width: 700px) {
.reme-home { padding-right: 20px; padding-left: 20px; }
.home-stage { gap: 42px; padding: 52px 0 62px; }
.hero-copy h1 { font-size: clamp(43px, 13vw, 62px); }
.hero-lead { font-size: 16px; }
.map-heading strong { white-space: normal; }
.section-heading { grid-template-columns: 1fr; gap: 18px; }
.product-grid { grid-template-columns: 1fr; }
.product-card { min-height: 280px; }
.benchmark-section, .product-section, .traffic-section { padding-top: 78px; padding-bottom: 84px; }
.benchmark-row { grid-template-columns: minmax(0, 1fr) auto; gap: 12px; }
.benchmark-track { grid-column: 1 / -1; grid-row: 2; }
.benchmark-score { grid-column: 2; grid-row: 1; }
.pi-score { grid-template-columns: 48px minmax(0, 1fr); }
.pi-symbol { width: 48px; height: 48px; }
.pi-score b { grid-column: 2; }
.traffic-window { height: 660px; min-height: 0; border-radius: 17px; }
}
@media (max-width: 520px) {
.ecosystem-network { grid-template-columns: minmax(0, 1.1fr) 62px minmax(0, 1fr); gap: 4px; }
.network-group-label { flex-wrap: wrap; }
.brand-link strong, .capability-link strong { font-size: 10px; }
.network-ring { width: 78px; height: 78px; }
.network-core { width: 52px; height: 52px; }
.network-core img { width: 34px; height: 34px; }
.capability-link { gap: 3px; padding: 5px; }
.capability-link span:first-child { width: 22px; height: 22px; font-size: 13px; }
.backend-links { grid-template-columns: 1fr; }
}
@media (prefers-reduced-motion: reduce) {
.brand-viewport { height: auto; mask-image: none; }
.brand-reel, .network-ring { animation: none; }
.reel-clone { display: none; }
}
</style>

View file

@ -0,0 +1,15 @@
<script setup lang="ts">
import { computed } from "vue";
import { useData } from "vitepress";
const { frontmatter, lang } = useData();
const sourcePath = computed(() => String(frontmatter.value._sourcePath || ""));
const label = computed(() => lang.value.startsWith("zh") ? "在 GitHub 查看源文件" : "View source on GitHub");
const href = computed(() => `https://github.com/agentscope-ai/ReMe/blob/main/${sourcePath.value}`);
</script>
<template>
<div v-if="sourcePath" class="source-link-wrap">
<a :href="href" target="_blank" rel="noreferrer">{{ label }} ↗</a>
</div>
</template>

View file

@ -0,0 +1,55 @@
<script setup lang="ts">
import { computed } from "vue";
import { useData } from "vitepress";
const props = defineProps<{ lang: "zh" | "en" }>();
const shareBase = "https://cloud.umami.is/analytics/us/share/S1OZK1PSDLEpyiU5?date=30day&page=1";
const { isDark } = useData();
const shareUrl = computed(() => `${shareBase}&theme=${isDark.value ? "dark" : "light"}`);
const text = computed(() => props.lang === "zh" ? {
eyebrow: "OPEN METRICS",
title: "ReMe 访问数据",
description: "最近 30 天的页面浏览量与访问趋势,由 Umami 提供隐私友好的匿名统计。",
action: "在 Umami 中打开完整页面",
frameTitle: "ReMe 最近 30 天访问数据",
} : {
eyebrow: "OPEN METRICS",
title: "ReMe traffic",
description: "Page views and traffic trends from the last 30 days, measured anonymously with privacy-friendly Umami analytics.",
action: "Open the full report in Umami",
frameTitle: "ReMe traffic for the last 30 days",
});
</script>
<template>
<main class="traffic-page">
<header>
<div>
<p>{{ text.eyebrow }}</p>
<h1>{{ text.title }}</h1>
<span>{{ text.description }}</span>
</div>
<a :href="shareUrl" target="_blank" rel="noreferrer">{{ text.action }} ↗</a>
</header>
<div class="traffic-frame-wrap">
<iframe :src="shareUrl" :title="text.frameTitle" loading="eager" referrerpolicy="no-referrer" />
</div>
</main>
</template>
<style scoped>
.traffic-page { max-width: 1440px; margin: 0 auto; padding: clamp(54px, 7vw, 96px) clamp(22px, 5vw, 74px) 90px; }
.traffic-page header { display: flex; align-items: flex-end; justify-content: space-between; gap: 40px; margin-bottom: 34px; }
.traffic-page header p { margin: 0 0 15px; color: var(--vp-c-brand-1); font: 750 12px/1.4 var(--vp-font-family-mono); letter-spacing: 0.16em; }
.traffic-page h1 { margin: 0; color: var(--vp-c-text-1); font-size: clamp(42px, 5.5vw, 68px); line-height: 1.05; letter-spacing: -0.05em; }
.traffic-page header span { display: block; max-width: 720px; margin-top: 17px; color: var(--vp-c-text-2); font-size: 17px; line-height: 1.65; }
.traffic-page header a { flex: none; padding: 11px 15px; border: 1px solid var(--vp-c-divider); border-radius: 10px; color: var(--vp-c-text-1); background: var(--vp-c-bg-soft); text-decoration: none; font-size: 14px; font-weight: 700; }
.traffic-page header a:hover { border-color: var(--vp-c-brand-1); color: var(--vp-c-brand-1); }
.traffic-frame-wrap { height: min(880px, calc(100vh - 210px)); min-height: 650px; overflow: hidden; border: 1px solid var(--vp-c-divider); border-radius: 20px; background: white; box-shadow: 0 24px 58px rgba(26, 62, 47, 0.12); }
.traffic-frame-wrap iframe { width: 100%; height: 100%; border: 0; }
@media (max-width: 700px) {
.traffic-page { padding-top: 42px; }
.traffic-page header { align-items: flex-start; flex-direction: column; gap: 20px; }
.traffic-frame-wrap { height: 720px; min-height: 0; border-radius: 14px; }
}
</style>

View file

@ -0,0 +1,370 @@
:root {
--vp-layout-max-width: 1560px;
--vp-c-brand-1: #087f6a;
--vp-c-brand-2: #086554;
--vp-c-brand-3: #19a98f;
--vp-c-brand-soft: rgba(8, 127, 106, 0.14);
--vp-c-bg: #ffffff;
--vp-c-bg-alt: #f4f7f5;
--vp-c-bg-elv: #ffffff;
--vp-c-bg-soft: #f1f6f3;
--vp-c-text-1: #17221d;
--vp-c-text-2: #526159;
--vp-c-text-3: #718078;
--vp-c-divider: #dce5e0;
--vp-font-family-base: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
--vp-font-family-mono: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
--reme-blue: #3156d9;
--reme-green: #087f6a;
}
.dark {
--vp-c-brand-1: #57dfc3;
--vp-c-brand-2: #35c6a9;
--vp-c-brand-3: #087f6a;
--vp-c-brand-soft: rgba(87, 223, 195, 0.14);
--vp-c-bg: #0d1512;
--vp-c-bg-alt: #09100d;
--vp-c-bg-elv: #14201b;
--vp-c-bg-soft: #17251f;
--vp-c-text-1: #edf7f3;
--vp-c-text-2: #bacbc4;
--vp-c-text-3: #91a49c;
--vp-c-divider: #283a33;
}
body {
background:
radial-gradient(circle at 8% 8%, rgba(25, 201, 176, 0.055), transparent 28rem),
var(--vp-c-bg);
}
.VPNav {
border-bottom: 1px solid color-mix(in srgb, var(--vp-c-divider) 84%, transparent);
background: color-mix(in srgb, var(--vp-c-bg) 88%, transparent);
backdrop-filter: blur(18px) saturate(140%);
}
.VPNavBarTitle .logo {
width: 30px;
height: 30px;
}
.VPNavBarTitle .title {
font-weight: 780;
letter-spacing: -0.02em;
}
.VPNavBarSearch .DocSearch-Button,
.VPNavBarSearch button {
min-width: 190px;
border: 1px solid var(--vp-c-divider);
border-radius: 10px;
background: var(--vp-c-bg-alt);
}
@media (min-width: 960px) {
.VPNavBar.has-sidebar .container > .title {
width: 113px !important;
padding-right: 0 !important;
}
.VPNavBar.has-sidebar .content {
padding-left: 113px !important;
}
.VPNavBarSearch {
padding-left: 24px !important;
}
}
@media (min-width: 1496px) {
.VPNavBar.has-sidebar .container {
position: relative !important;
max-width: 1496px !important;
margin: 0 auto !important;
}
.VPNavBar.has-sidebar .container > .title {
width: 81px !important;
padding-left: 0 !important;
}
.VPNavBar.has-sidebar .content {
padding-right: 0 !important;
padding-left: 81px !important;
}
}
.VPSidebar {
border-right: 1px solid var(--vp-c-divider);
background: color-mix(in srgb, var(--vp-c-bg-alt) 82%, transparent);
}
.VPSidebarItem .text {
font-size: 14px;
}
.VPSidebarItem.level-0 > .item > .text {
color: var(--vp-c-text-1);
font-weight: 750;
}
.VPSidebarItem.is-active > .item .link > .text {
color: var(--vp-c-brand-1);
}
.VPDocAsideOutline {
border-left-color: var(--vp-c-divider);
}
.VPDoc .container > .content {
min-width: 0;
}
.VPDoc .content-container {
max-width: 900px !important;
}
.vp-doc {
color: var(--vp-c-text-1);
font-size: 16px;
line-height: 1.78;
}
.vp-doc h1 {
margin-bottom: 28px;
font-size: clamp(36px, 5vw, 52px);
line-height: 1.08;
letter-spacing: -0.045em;
}
.vp-doc h2 {
margin-top: 52px;
border-top-color: var(--vp-c-divider);
font-size: 27px;
letter-spacing: -0.025em;
}
.vp-doc h3 {
margin-top: 34px;
font-size: 20px;
}
.vp-doc :not(pre) > code {
border-radius: 5px;
color: color-mix(in srgb, var(--vp-c-brand-1) 82%, var(--vp-c-text-1));
}
.vp-doc div[class*="language-"] {
border: 1px solid var(--vp-c-divider);
border-radius: 12px;
box-shadow: inset 3px 0 0 color-mix(in srgb, var(--vp-c-brand-1) 62%, transparent);
}
.copy-markdown-wrap {
display: flex;
justify-content: flex-end;
margin-bottom: 18px;
}
.copy-markdown {
display: inline-flex;
align-items: center;
gap: 7px;
min-height: 34px;
padding: 6px 12px;
border: 1px solid var(--vp-c-divider);
border-radius: 9px;
color: var(--vp-c-text-2);
background: var(--vp-c-bg-soft);
cursor: pointer;
font-size: 13px;
font-weight: 650;
}
.copy-markdown:hover,
.copy-markdown.copied {
border-color: var(--vp-c-brand-1);
color: var(--vp-c-brand-1);
}
.copy-markdown svg {
width: 15px;
height: 15px;
fill: none;
stroke: currentColor;
stroke-linecap: round;
stroke-linejoin: round;
stroke-width: 2;
}
.source-link-wrap {
margin-top: 48px;
padding-top: 20px;
border-top: 1px solid var(--vp-c-divider);
font-size: 13px;
}
.source-link-wrap a {
color: var(--vp-c-text-3);
text-decoration: none;
}
.source-link-wrap a:hover {
color: var(--vp-c-brand-1);
}
.VPHome {
overflow: hidden;
background:
radial-gradient(circle at 14% 14%, rgba(25, 201, 176, 0.14), transparent 30rem),
radial-gradient(circle at 86% 10%, rgba(49, 86, 217, 0.11), transparent 28rem);
}
.VPHero .name {
background: linear-gradient(120deg, var(--reme-green), var(--reme-blue));
background-clip: text;
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
.VPHero .name,
.VPHero .text {
letter-spacing: -0.05em;
}
.VPHero .image-bg {
width: min(88%, 420px);
height: 220px;
border-radius: 42%;
background: linear-gradient(125deg, rgba(25, 201, 176, 0.34), rgba(49, 86, 217, 0.25));
filter: blur(52px);
}
.VPHero .image-container {
isolation: isolate;
perspective: 900px;
}
.VPHero .image-container::before,
.VPHero .image-container::after {
position: absolute;
content: "";
pointer-events: none;
}
.VPHero .image-container::before {
z-index: 0;
top: 50%;
left: 50%;
width: min(88%, 430px);
height: 210px;
border: 1px solid color-mix(in srgb, var(--vp-c-bg-elv) 64%, var(--reme-blue));
border-radius: 30px;
background:
linear-gradient(135deg, color-mix(in srgb, var(--vp-c-bg-elv) 92%, transparent), color-mix(in srgb, var(--vp-c-bg-soft) 76%, transparent)),
radial-gradient(circle at 15% 15%, rgba(25, 201, 176, 0.14), transparent 42%);
box-shadow:
0 30px 70px rgba(19, 70, 91, 0.16),
inset 0 1px 0 color-mix(in srgb, white 72%, transparent);
backdrop-filter: blur(22px) saturate(135%);
transform: translate(-50%, -50%) rotate(-1.5deg);
}
.VPHero .image-container::after {
z-index: -1;
top: 50%;
left: 50%;
width: min(78%, 380px);
height: 210px;
border: 1px solid rgba(49, 86, 217, 0.17);
border-radius: 30px;
background: linear-gradient(135deg, rgba(25, 201, 176, 0.1), rgba(49, 86, 217, 0.11));
transform: translate(-46%, -48%) rotate(7deg);
}
.VPHero .image-src {
z-index: 1;
width: min(76%, 380px);
max-width: 380px !important;
max-height: 150px !important;
object-fit: contain;
filter: drop-shadow(0 12px 20px rgba(18, 78, 105, 0.16));
}
.VPHomeFeatures .item:nth-child(1) { --feature-accent: #18b99e; }
.VPHomeFeatures .item:nth-child(2) { --feature-accent: #25a8dc; }
.VPHomeFeatures .item:nth-child(3) { --feature-accent: #6575e8; }
.VPHomeFeatures .item:nth-child(4) { --feature-accent: #e69a42; }
.VPHomeFeatures .VPFeature {
border-color: color-mix(in srgb, var(--feature-accent) 28%, var(--vp-c-divider));
border-radius: 16px;
background:
radial-gradient(circle at 10% 4%, color-mix(in srgb, var(--feature-accent) 16%, transparent), transparent 52%),
linear-gradient(150deg, var(--vp-c-bg-elv), color-mix(in srgb, var(--feature-accent) 7%, var(--vp-c-bg-soft)));
box-shadow:
inset 0 1px 0 color-mix(in srgb, white 76%, transparent),
0 8px 24px color-mix(in srgb, var(--feature-accent) 7%, transparent);
transition: transform 160ms ease, border-color 160ms ease, box-shadow 160ms ease;
}
.VPHomeFeatures .VPFeature .box {
display: grid;
grid-template-columns: auto minmax(0, 1fr);
grid-template-rows: auto 1fr;
column-gap: 12px;
align-items: center;
}
.VPHomeFeatures .VPFeature .icon {
grid-column: 1;
grid-row: 1;
width: auto;
height: auto;
margin: 0;
border: 0;
background: transparent;
box-shadow: none;
font-size: 25px;
}
.VPHomeFeatures .VPFeature .title {
grid-column: 2;
grid-row: 1;
color: color-mix(in srgb, var(--feature-accent) 22%, var(--vp-c-text-1));
}
.VPHomeFeatures .VPFeature .details {
grid-column: 1 / -1;
grid-row: 2;
align-self: start;
padding-top: 18px;
}
.VPHomeFeatures .VPFeature .link-text {
grid-column: 1 / -1;
}
.VPHomeFeatures .VPFeature:hover {
transform: translateY(-3px);
border-color: color-mix(in srgb, var(--feature-accent) 48%, var(--vp-c-divider));
box-shadow: 0 18px 38px color-mix(in srgb, var(--feature-accent) 15%, transparent);
}
.dark .VPHomeFeatures .VPFeature {
background:
radial-gradient(circle at 10% 4%, color-mix(in srgb, var(--feature-accent) 18%, transparent), transparent 54%),
linear-gradient(150deg, var(--vp-c-bg-elv), color-mix(in srgb, var(--feature-accent) 8%, var(--vp-c-bg-soft)));
box-shadow: inset 0 1px 0 rgba(255, 255, 255, 0.06);
}
@media (max-width: 768px) {
.vp-doc h1 { font-size: 34px; }
.vp-doc h2 { margin-top: 44px; font-size: 24px; }
.copy-markdown-wrap { margin-top: -8px; }
.VPHero .image-container::before,
.VPHero .image-container::after { height: 176px; border-radius: 24px; }
.VPHero .image-src { width: 72%; max-height: 120px !important; }
}

View file

@ -0,0 +1,21 @@
import { h } from "vue";
import DefaultTheme from "vitepress/theme";
import CopyMarkdownButton from "./CopyMarkdownButton.vue";
import HomePage from "./HomePage.vue";
import SourceLink from "./SourceLink.vue";
import TrafficPage from "./TrafficPage.vue";
import "./custom.css";
export default {
extends: DefaultTheme,
enhanceApp({ app }) {
app.component("HomePage", HomePage);
app.component("TrafficPage", TrafficPage);
},
Layout() {
return h(DefaultTheme.Layout, null, {
"doc-before": () => h(CopyMarkdownButton),
"doc-after": () => h(SourceLink),
});
},
};

View file

@ -1,16 +1,13 @@
# Auto Dream # Auto Dream
`auto_dream` is ReMe's long-term memory distillation flow from daily to digest. It scans daily inputs for a specified date, `auto_dream` is ReMe's long-term memory distillation flow from daily to digest. By default it scans the target date and
processes only files that changed since the previous dream, extracts content worth retaining as memory units, integrates those the previous day, processes only files changed since the previous dream, extracts a small set of high-value memory units
units into `digest/`, and generates the day's `interests.yaml` for proactive use. across that window, and integrates them into `digest/`.
<p align="center">
<img src="../figure/auto-dream-and-proactive.svg" alt="ReMe Auto Dream and Proactive flow from daily to digest to proactive" width="92%">
</p>
Its daily inputs usually come from [Auto Memory](./auto_memory.md) and [Auto Resource](./auto_resource.md). For the file Its daily inputs usually come from [Auto Memory](./auto_memory.md) and [Auto Resource](./auto_resource.md). For the file
semantics of `digest/`, Sources sections, and wikilinks, see [Memory as File](./memory_as_file.md). For the linking strategy semantics of `digest/`, Sources sections, and wikilinks, see [Memory as File](./memory_as_file.md). For the linking
used during Integrate, see [Auto Link](./auto_link.md). To read `interests.yaml`, use [Proactive](./proactive.md). strategy used during Integrate, see [Auto Link](./auto_link.md). Proactive discovery is a separate flow; see
[Proactive](./proactive.md).
## Configuration ## Configuration
@ -26,54 +23,54 @@ auto_dream:
hint: hint:
type: string type: string
default: "" default: ""
topic_count: scan_days:
type: integer type: integer
default: 3 default: 2
topic_diversity_days: max_units:
type: integer type: integer
default: 7 default: 5
steps: steps:
- backend: dream_extract_step - backend: dream_extract_step
file_catalog: dream file_catalog: dream
topic_session_id: interests scan_days: 2
max_units: 5
- backend: dream_integrate_step - backend: dream_integrate_step
- backend: dream_topics_step
topic_count: 3
topic_diversity_days: 7
- backend: dream_finish_step - backend: dream_finish_step
file_catalog: dream file_catalog: dream
- backend: auto_tag_step
``` ```
Parameters: Parameters:
| Parameter | Purpose | | Parameter | Purpose |
|---|---| |------------------------|---------------------------------------------------------------------------------------------------------|
| `date` | Date to process in `YYYY-MM-DD` format. When empty, use today in the application's timezone. | | `date` | Date to process in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `hint` | Additional guidance from the caller for the Extract and Integrate stages. | | `hint` | Additional guidance from the caller for the Extract and Integrate stages. |
| `topic_count` | Maximum number of topics written to `interests.yaml`. Defaults to 3. | | `scan_days` | Recent-date window ending at `date`; defaults to 2 and has a minimum of 1. |
| `topic_diversity_days` | Number of past days of `interests.yaml` files considered when avoiding duplicate topics. Defaults to 7. | | `max_units` | Maximum reusable units extracted in one run; defaults to 5. |
## Inputs and Outputs ## Inputs and Outputs
Inputs are daily Markdown files for the specified date: Inputs are daily Markdown files from the most recent `scan_days` ending at the specified date. For example,
`date=2026-06-20` with `scan_days=2` scans:
```text ```text
daily/<date>.md daily/2026-06-19.md
daily/<date>/**/*.md daily/2026-06-19/**/*.md
daily/2026-06-20.md
daily/2026-06-20/**/*.md
``` ```
`daily/<date>/interests.yaml` is excluded from extraction input so topics from the previous run do not feed back into the Only Markdown day indexes and notes are scanned. Proactive state and `interests.yaml` are not Auto Dream inputs.
next extraction.
The main outputs are: The main outputs are:
| Output | Description | | Output | Description |
|---|---| |--------------------------------|------------------------------------------------------------------------------|
| `digest/procedure/*.md` | Methods, workflows, runbooks, and executable experience. | | `digest/procedure/*.md` | Methods, workflows, runbooks, and executable experience. |
| `digest/personal/*.md` | User-, team-, and project-related preferences, facts, and long-term context. | | `digest/personal/*.md` | User-, team-, and project-related preferences, facts, and long-term context. |
| `digest/wiki/*.md` | General knowledge, concepts, observations, and decision precedents. | | `digest/wiki/*.md` | General knowledge, concepts, observations, and decision precedents. |
| `daily/<date>/interests.yaml` | Topics worth proactive attention from the host agent that day. | | `metadata/file_catalog/dream*` | Dream-specific catalog used to detect changes in daily inputs. |
| `metadata/file_catalog/dream*` | Dream-specific catalog used to detect changes in daily inputs. |
## Four Stages ## Four Stages
@ -81,88 +78,74 @@ The main outputs are:
`dream_extract_step` performs three tasks: `dream_extract_step` performs three tasks:
1. Refresh the day's index page at `daily/<date>.md`. 1. Refresh each `daily/<date>.md` in the scan window.
2. Scan `daily/<date>.md` and `daily/<date>/**/*.md` and compare their mtimes with `file_catalog: dream`. 2. Scan those day indexes and `daily/<date>/**/*.md`, comparing mtimes with `file_catalog: dream`.
3. Send only changed files to the LLM and globally extract two structured result types: `units` and `topics`. 3. Send all changed files together to the LLM and globally extract structured memory `units`.
`units` are long-term memory units ready to be distilled into digest. Each has `name`, `bucket`, `summary`, and `paths`. `units` are long-term memory units ready to be distilled into digest. Each has `name`, `bucket`, `summary`, and `paths`.
`bucket` may only be `procedure`, `personal`, or `wiki`; unknown values are routed to `wiki`. A run returns at most `max_units`; extraction merges cross-file evidence for the same abstraction and drops passing
mentions, per-file summaries, and weak candidates without reusable value. `bucket` may only be `procedure`, `personal`,
or `wiki`; unknown values are routed to `wiki`.
`topics` are proactive-interest candidates for the day. They contain `title`, `reason`, `evidence`, `keywords`, and If there are no changed files, Extract succeeds with no units; Integrate then has no unit work, and Finish still
`paths` and are filtered again in the Topics stage. performs its normal catalog summary. If files changed but no LLM is
configured, Extract fails because extraction requires an LLM.
If there are no changed files, the flow ends early with success and skips later extraction work. If files changed but no LLM
is configured, Extract fails because extraction requires an LLM.
### 2. Integrate ### 2. Integrate
`dream_integrate_step` invokes an agent independently for each unit and integrates that unit into one digest node. It exposes `dream_integrate_step` invokes an agent independently for each unit and integrates that unit into one digest node. It
these tools to the agent: exposes these tools to the agent:
```text ```text
node_search, read, frontmatter_read, write, edit, frontmatter_update node_search, read, frontmatter_read, write, edit, frontmatter_update
``` ```
This stage carries the core responsibility of `auto_link`. It first uses `node_search` to recall similar or related nodes at This stage carries the core responsibility of `auto_link`. It first uses `node_search` to recall similar or related
digest-node granularity, decides whether to create or update a node, and finally writes sources and related digest nodes as nodes at digest-node granularity, decides whether to create or update a node, and finally writes sources and related
wikilinks. See [Auto Link](./auto_link.md) for the recall, deduplication, and edge-writing rules. digest nodes as wikilinks. See [Auto Link](./auto_link.md) for the recall, deduplication, and edge-writing rules.
Extract is the gate for deciding whether material is worth remembering, so Integrate has no `SKIP` action: each admitted
unit must land in exactly one digest node. Creates and updates must retain provenance and weave related digest links
into contextual sentences; bare wikilinks and standalone relationship fields are not valid output.
There are four integration actions: There are four integration actions:
| Action | Meaning | | Action | Meaning |
|---|---| |---------------|--------------------------------------------------------------------------------|
| `CREATE` | No equivalent abstraction exists; create a new digest node. | | `CREATE` | No equivalent abstraction exists; create a new digest node. |
| `CORROBORATE` | The same memory appeared again; append a source or strengthen the description. | | `CORROBORATE` | The same memory appeared again; append a source or strengthen the description. |
| `REFINE` | New material adds boundaries, steps, prerequisites, applicability, or detail. | | `REFINE` | New material adds boundaries, steps, prerequisites, applicability, or detail. |
| `CORRECT` | New material corrects errors, omissions, or conflicts in the existing node. | | `CORRECT` | New material corrects errors, omissions, or conflicts in the existing node. |
Successfully integrated units are recorded in `integrate_results`. Failed units enter `failed_units`, and their source paths Successfully integrated units are recorded in `integrate_results`. Failed units enter `failed_units`, and their source
enter `failed_paths`. The Finish stage does not checkpoint failed paths, ensuring that they can be retried later. paths enter `failed_paths`. The Finish stage does not checkpoint failed paths, ensuring that they can be retried later.
### 3. Topics ### 3. Finish
`dream_topics_step` turns topic candidates from Extract into the final `daily/<date>/interests.yaml` for the day.
It reads:
```text
daily/<date>/interests.yaml
daily/<previous-date>/interests.yaml
```
Existing topics from the same day are preserved, while similar topics from the previous `topic_diversity_days` days are
deduplicated. At most three topics are written by default. With an LLM configured, the LLM selects topics that are more
specific, actionable, and non-repetitive. Without an LLM, the step falls back to local normalization and deduplication.
Example output format. See [Proactive](./proactive.md) for the interface that reads this file:
```yaml
date: 2026-06-20
topic_count: 3
diversity_days: 7
topics:
- title: Quality regression in the memory retrieval pipeline
reason: The user has recently made repeated changes to search, node_search, and dream integration.
evidence: daily/2026-06-20/session.md
keywords:
- memory search
- auto dream
paths:
- daily/2026-06-20/session.md
```
### 4. Finish
`dream_finish_step` completes the run: `dream_finish_step` completes the run:
1. Write successfully processed changed paths to `file_catalog: dream`. 1. Write successfully processed changed paths to `file_catalog: dream`.
2. Also write `daily/<date>/interests.yaml` and `daily/<date>.md` to the catalog. 2. Also write every refreshed day-index page in the scan window to the catalog.
3. Persist the dream catalog if there were upserts or deletions. 3. Persist the dream catalog if there were upserts or deletions.
4. Return a summary containing counts for scanned, changed, integrated, topics, checkpoints, and related values. 4. Return a summary containing counts for scanned, changed, integrated, checkpoints, and related values.
Auto Dream neither reads nor writes proactive state or `interests.yaml`. Those files are owned by the proactive refresh
pipeline; see [Proactive](./proactive.md).
Failed paths are not checkpointed. The next `auto_dream` run therefore continues to treat them as changed inputs until Failed paths are not checkpointed. The next `auto_dream` run therefore continues to treat them as changed inputs until
integration succeeds. integration succeeds.
### 4. Auto Tag
After Finish, both `auto_dream` and `dream_cron` run `auto_tag_step` on Markdown digest files actually created or modified
during integration, including writes recovered after agent errors. Repeated writes to one file are tagged once. The
Step uses the same request-scoped `changes` contract as [Auto Memory](./auto_memory.md) and writes entity tags to the
configured frontmatter key, `memory_tags` by default. Unchanged files and daily source notes are not tagged by Dream.
Tagging diagnostics appear in `metadata.auto_tag`. Per-file tagging failures preserve the dream answer, success status,
and checkpoint decisions. A later run without file changes does not automatically retry failed tagging. Tag-index
updates follow the existing asynchronous file watcher.
## Running Auto Dream ## Running Auto Dream
CLI: CLI:
@ -177,6 +160,12 @@ With caller guidance:
reme auto_dream date=2026-06-20 hint="Prioritize engineering decisions and long-term preferences" reme auto_dream date=2026-06-20 hint="Prioritize engineering decisions and long-term preferences"
``` ```
Override the default scan window and unit cap:
```bash
reme auto_dream date=2026-06-20 scan_days=3 max_units=8
```
The same set of steps can also be placed in a `cron` Job, for example to run every morning: The same set of steps can also be placed in a `cron` Job, for example to run every morning:
```yaml ```yaml
@ -188,22 +177,22 @@ jobs:
- backend: dream_extract_step - backend: dream_extract_step
file_catalog: dream file_catalog: dream
- backend: dream_integrate_step - backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step - backend: dream_finish_step
file_catalog: dream file_catalog: dream
- backend: auto_tag_step
``` ```
## Important Boundaries ## Important Boundaries
`auto_dream` consumes only daily inputs and does not rewrite daily bodies. Daily preserves facts and the original situation; `auto_dream` consumes only daily inputs and does not rewrite daily bodies. Daily preserves facts and the original
digest is the abstracted long-term memory layer. situation; digest is the abstracted long-term memory layer.
`digest` is not a copy of the source text. Its body should preserve reusable abstractions, while a Sources section points `digest` is not a copy of the source text. Its body should preserve reusable abstractions, while a Sources section
back with entries such as `- [[daily/<date>/...]]`. Links follow the workspace-relative wikilink semantics described in points back with contextual sentences such as `The decision was recorded in [[daily/<date>/decision.md]].` Links follow
the workspace-relative wikilink semantics described in
[Memory as File](./memory_as_file.md). [Memory as File](./memory_as_file.md).
`auto_dream` does not invent an overview from nothing. Only content that actually appears in daily input and is extracted as `auto_dream` does not invent an overview from nothing. Only content that actually appears in daily input and is
a unit or topic can enter digest or `interests.yaml`. extracted as a memory unit can enter digest.
The complete flow depends on an LLM for Extract and Integrate. Topics can perform local deduplication without an LLM, but that The complete flow depends on an LLM for Extract, Integrate, and Auto Tag.
does not mean the full dream flow can run offline.

View file

@ -1,11 +1,12 @@
# Auto Link # Auto Link
In the current implementation, `auto_link` is not a separately registered Job. It is a capability of the Integrate stage in In the current implementation, `auto_link` is not a separately registered Job. It is a capability of the Integrate stage
in
`auto_dream`: when `dream_integrate_step` writes a memory unit to `digest/`, it also recalls digest nodes, makes a `auto_dream`: when `dream_integrate_step` writes a memory unit to `digest/`, it also recalls digest nodes, makes a
deduplication decision, links sources, and weaves wikilinks to related nodes into the result. deduplication decision, links sources, and weaves wikilinks to related nodes into the result.
For the complete dream flow, see [Auto Dream](./auto_dream.md). For general wikilink, frontmatter, and workspace-relative For the complete dream flow, see [Auto Dream](./auto_dream.md). For general wikilink, frontmatter, and
path semantics, see [Memory as File](./memory_as_file.md). For question-answering retrieval, see workspace-relative path semantics, see [Memory as File](./memory_as_file.md). For question-answering retrieval, see
[Memory Search](./memory_search.md). [Memory Search](./memory_search.md).
## Where It Runs ## Where It Runs
@ -17,23 +18,23 @@ auto_dream:
steps: steps:
- dream_extract_step - dream_extract_step
- dream_integrate_step # where auto_link actually happens - dream_integrate_step # where auto_link actually happens
- dream_topics_step
- dream_finish_step - dream_finish_step
- auto_tag_step
``` ```
The Integrate stage processes each unit independently. A unit is written to exactly one target digest node, but that node may The Integrate stage processes each unit independently. A unit is written to exactly one target digest node, but that
link to multiple sources and multiple related digest nodes. node may link to multiple sources and multiple related digest nodes.
## Goals ## Goals
`auto_link` addresses graph quality at write time: `auto_link` addresses graph quality at write time:
| Problem | Handling | | Problem | Handling |
|---|---| |------------------------------------------------|----------------------------------------------------------------------|
| The same memory already exists | Recall and update the existing node instead of creating a duplicate. | | The same memory already exists | Recall and update the existing node instead of creating a duplicate. |
| New and existing material are related | Write workspace-relative wikilinks into the body. | | New and existing material are related | Write workspace-relative wikilinks into the body. |
| A digest node is disconnected from its sources | Add daily/resource links under a `## Sources` section. | | A digest node is disconnected from its sources | Add daily/resource links under a `## Sources` section. |
| A node contains only isolated prose | Add links to related digest nodes on both CREATE and UPDATE. | | A node contains only isolated prose | Add links to related digest nodes on both CREATE and UPDATE. |
## Toolchain ## Toolchain
@ -48,41 +49,42 @@ edit
frontmatter_update frontmatter_update
``` ```
`node_search` is digest-only node retrieval designed for dream integration. It returns node-level signals such as the digest `node_search` is digest-only node retrieval designed for dream integration. It returns node-level signals such as the
node's `path` and the `name` and `description` from frontmatter. It does not expand the body and does not perform the link digest node's `path` and the `name` and `description` from frontmatter. It does not expand the body and does not perform
expansion used by ordinary search. the link expansion used by ordinary search.
`read` and `frontmatter_read` are used only for candidates that may be relevant, avoiding expansion of every recalled result `read` and `frontmatter_read` are used only for candidates that may be relevant, avoiding expansion of every recalled
into a large context. result into a large context.
## Linking Flow ## Linking Flow
### 1. Recall candidate nodes ### 1. Recall candidate nodes
The agent first calls `node_search` with the unit's triggers, verbs, nouns, synonyms, and possible failure modes. Broad recall, The agent first calls `node_search` with the unit's triggers, verbs, nouns, synonyms, and possible failure modes. Broad
for example `limit=20-30`, is recommended by default because this step serves both deduplication and link discovery. recall, for example `limit=20-30`, is recommended by default because this step serves both deduplication and link
discovery.
Recalled results are internally classified into three groups: Recalled results are internally classified into three groups:
| Classification | Meaning | Next action | | Classification | Meaning | Next action |
|---|---|---| |--------------------|---------------------------------------------------------------------------------------------------------|---------------------------|
| `same_abstraction` | The trigger or underlying abstraction is the same, with substantial content overlap. | Use as the UPDATE target. | | `same_abstraction` | The trigger or underlying abstraction is the same, with substantial content overlap. | Use as the UPDATE target. |
| `related` | An adjacent process, prerequisite, failure mode, concept, preference, or upstream/downstream knowledge. | Write a body wikilink. | | `related` | An adjacent process, prerequisite, failure mode, concept, preference, or upstream/downstream knowledge. | Write a body wikilink. |
| `unrelated` | Only superficially similar or unrelated. | Ignore. | | `unrelated` | Only superficially similar or unrelated. | Ignore. |
### 2. Choose a write action ### 2. Choose a write action
Every unit must select one action: Every unit must select one action:
| Action | Linking semantics | | Action | Linking semantics |
|---|---| |---------------|-------------------------------------------------------------------------------------------------------------------------------|
| `CREATE` | Write a new `digest/<bucket>/<slug>.md` and add source and related-node links to its body. | | `CREATE` | Write a new `digest/<bucket>/<slug>.md` and add source and related-node links to its body. |
| `CORROBORATE` | The same abstraction appeared again; append its source link and strengthen the description when needed. | | `CORROBORATE` | The same abstraction appeared again; append its source link and strengthen the description when needed. |
| `REFINE` | New material extends the existing node; insert the additional content in the appropriate section and preserve existing links. | | `REFINE` | New material extends the existing node; insert the additional content in the appropriate section and preserve existing links. |
| `CORRECT` | New material corrects the existing node; use source links to identify the basis for the correction. | | `CORRECT` | New material corrects the existing node; use source links to identify the basis for the correction. |
An UPDATE should be additive whenever possible: do not delete existing wikilinks or source entries. This prevents An UPDATE should be additive whenever possible: do not delete existing wikilinks or source entries. This prevents later
later graph indexing and retrieval from losing edges. graph indexing and retrieval from losing edges.
### 3. Write source edges ### 3. Write source edges
@ -91,12 +93,13 @@ Source edges are ordinary wikilinks grouped under a Markdown heading:
```markdown ```markdown
## Sources ## Sources
- [[daily/2026-06-20/session.md]] The decision was recorded in [[daily/2026-06-20/session.md]], while the supporting technical evidence comes from
- [[resource/2026-06-20/paper.md]] [[resource/2026-06-20/paper.md]].
``` ```
These edges represent the evidence behind a digest node. Plain-text descriptions do not count as source edges because only These edges represent the evidence behind a digest node. Plain-text descriptions do not count as source edges because
wikilinks can be parsed reliably by the file graph. For the complete parsing rules, see only wikilinks can be parsed reliably by the file graph. The surrounding sentence must explain what each source
supports; a bare wikilink line is not valid Integrate output. For the complete parsing rules, see
[Memory as File](./memory_as_file.md#wikilink). [Memory as File](./memory_as_file.md#wikilink).
### 4. Write relationships between digest nodes ### 4. Write relationships between digest nodes
@ -113,11 +116,11 @@ This design extends [[digest/wiki/hybrid-search.md]] and uses
`auto_link` adjusts the shape of its output according to the unit bucket: `auto_link` adjusts the shape of its output according to the unit bucket:
| Bucket | Writing focus | | Bucket | Writing focus |
|---|---| |-------------|-----------------------------------------------------------------------------------------------------------------------------|
| `procedure` | Write a runbook with triggers, steps, inputs, and failure modes. Link prerequisites, substeps, and related preferences. | | `procedure` | Write a runbook with triggers, steps, inputs, and failure modes. Link prerequisites, substeps, and related preferences. |
| `personal` | Write user-, team-, or project-specific facts and preferences. Link related projects, habits, and decision context. | | `personal` | Write user-, team-, or project-specific facts and preferences. Link related projects, habits, and decision context. |
| `wiki` | Write general knowledge, principles, observations, and decision precedents. Link concepts, methods, and adjacent knowledge. | | `wiki` | Write general knowledge, principles, observations, and decision precedents. Link concepts, methods, and adjacent knowledge. |
Regardless of bucket, preserve source edges and weave recalled related digest nodes into the body whenever possible. Regardless of bucket, preserve source edges and weave recalled related digest nodes into the body whenever possible.
@ -125,19 +128,19 @@ Regardless of bucket, preserve source edges and weave recalled related digest no
`auto_link` uses `node_search`, not the question-answering `search`. `auto_link` uses `node_search`, not the question-answering `search`.
| Capability | Purpose | | Capability | Purpose |
|---|---| |---------------|-----------------------------------------------------------------------------------------------------------|
| `search` | External question answering; returns chunks and can expand upstream/downstream link context. | | `search` | External question answering; returns chunks and can expand upstream/downstream link context. |
| `node_search` | Dream integration; recalls only digest node-level summaries for deduplication and related-link decisions. | | `node_search` | Dream integration; recalls only digest node-level summaries for deduplication and related-link decisions. |
This boundary matters. The Integrate stage needs to decide whether the same abstraction already exists and which nodes should This boundary matters. The Integrate stage needs to decide whether the same abstraction already exists and which nodes
be linked; it should not load large numbers of body chunks into context. [Memory Search](./memory_search.md) handles should be linked; it should not load large numbers of body chunks into context. [Memory Search](./memory_search.md)
question-oriented chunk retrieval, RRF fusion, and link expansion. handles question-oriented chunk retrieval, RRF fusion, and link expansion.
## Failure and Retry ## Failure and Retry
If integration of a unit fails, `dream_integrate_step` records `failed_units` and `failed_paths`. If integration of a unit fails, `dream_integrate_step` records `failed_units` and `failed_paths`.
`dream_finish_step` does not checkpoint those source paths, so the next `auto_dream` run processes them again. `dream_finish_step` does not checkpoint those source paths, so the next `auto_dream` run processes them again.
This makes auto_link writes retryable: a failure does not mark the input as complete or silently discard digest edges that This makes auto_link writes retryable: a failure does not mark the input as complete or silently discard digest edges
should have been created. that should have been created.

View file

@ -1,8 +1,9 @@
# Auto Memory # Auto Memory
Auto Memory is ReMe's entry point for conversational memory. Each conversation is first distilled into a daily memory card Auto Memory is ReMe's entry point for conversational memory. Within a target date, it uses `session_id` to find or update at
identified by `session_id`, and the day's `YYYY-MM-DD.md` page then indexes all of those cards. It turns "we talked about it" most one daily memory card, whose filename is a concise topic or event name chosen by the Agent. The day's `YYYY-MM-DD.md`
into "it was remembered" while preserving the original conversation as evidence. page indexes those cards. It turns "we talked about it" into "it was remembered" while retaining a source conversation record
as evidence.
<p align="center"> <p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%"> <img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%">
@ -13,9 +14,9 @@ For the general file semantics of `daily/`, `session/`, frontmatter, and wikilin
```text ```text
Conversation Conversation
├─ step 1: daily/YYYY-MM-DD/<session_id>.md # one card per conversation ├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # one topic-named card per session
├─ step 2: daily/YYYY-MM-DD.md # daily index linking the cards ├─ step 2: daily/YYYY-MM-DD.md # daily index linking the cards
└─ source: session/dialog/<session_id>.jsonl # original conversation └─ source: session/dialog/<session_id>.jsonl # source conversation record
``` ```
## What It Records ## What It Records
@ -39,29 +40,33 @@ workspace/
daily/ daily/
2026-06-20.md 2026-06-20.md
2026-06-20/ 2026-06-20/
session-a.md login-refactor-decision.md
session-b.md retrieval-regression.md
``` ```
`daily/2026-06-20/session-a.md` and `daily/2026-06-20/session-b.md` are memory cards distilled from different The two files under the date directory are topic-named cards distilled from different conversations.
conversations. `daily/2026-06-20.md` is the index page for that day. Resource files enter the same daily memory layer; see `daily/2026-06-20.md` is the index page for that day. Resource files enter the same daily memory layer; see
[Auto Resource](./auto_resource.md). [Auto Resource](./auto_resource.md).
When a call includes `session_id`, Auto Memory records that conversation separately under the given ID: When a call includes `session_id`, Auto Memory uses it to find the corresponding card through frontmatter, while the Agent
chooses a readable filename through `name`:
```text ```yaml
daily/2026-06-20/session-a.md name: login-refactor-decision
session_id: session-a
source_conversation: "[[session/dialog/session-a.jsonl]]"
``` ```
This keeps different conversations separate. A requirements discussion, a debugging session, and a documentation update can This keeps different conversations separate without forcing opaque IDs into filenames. An update locates the existing note by
each have their own memory card. To see what happened on a particular day, start with `YYYY-MM-DD.md`. To inspect what was `session_id` or `source_conversation`; if the Agent supplies a better frontmatter `name`, the system can rename the note and
distilled from one conversation, open the corresponding `<session_id>.md`. retarget inbound wikilinks. To see what happened on a day, start with `YYYY-MM-DD.md`.
## Preserving the Original Information ## Preserving the Original Information
The distilled daily note is optimized for readability; the original conversation is retained for trust and verification. The distilled daily note is optimized for readability; a filtered source conversation record is retained for trust and
verification.
While generating memory cards, Auto Memory also saves the raw sessions: While generating memory cards, Auto Memory also saves the source messages:
```text ```text
session/ session/
@ -70,12 +75,47 @@ session/
session-b.jsonl session-b.jsonl
``` ```
Each daily note points to its corresponding original conversation. When a memory needs verification, follow that link back to Each daily note points to its corresponding conversation record. Saved messages omit tool-result blocks and base64 data
the complete context in which it was created. blocks, preventing recalled memory and binary payloads from being mistaken for user-provided evidence later.
## Images in Conversations
Auto Memory can read images together with the surrounding conversation. Images are disabled by default; enable them for a
call with `include_images=true`.
Image input requires an `agentscope` wrapper with a vision-capable `as_llm` model and compatible formatter.
Auto Memory uses that model to read the conversation, without generating captions first. When images are disabled or no
image blocks are present, the existing text-only behavior is unchanged, including support for other wrappers.
Pass images as top-level AgentScope `DataBlock` values in `messages`, with an `image/` media type. Text and images stay in
their original order, with speaker and timestamp boundaries preserved. Base64 sources and HTTP(S) URLs pass unchanged to
the formatter; Auto Memory does not download or preprocess the images. URLs must be accessible to the model provider. For local
files, submit Base64 instead of a `file://` URL; other URL schemes are also unsupported.
The wrapper's `context_config.max_image_num` limits the number of images per call; Auto Memory rejects excess images rather
than increasing the limit. The AgentScope default is 5. To use a higher limit, set it when starting the service:
```bash
reme start components.agent_wrapper.default.context_config.max_image_num=20
```
Then call the running service from another terminal, using the same workspace:
```bash
reme auto_memory session_id=session-a include_images=true messages='[...]'
```
Model and formatter limits still apply. When image input is enabled and images are present, Auto Memory checks the wrapper
backend, URL schemes and image count before saving the conversation. Later formatter or provider errors are returned
without retrying as text-only. As with text-only calls, those errors do not roll back an already saved conversation.
Source JSONL saving follows the filtering rules above, including the omission of Base64 blocks. To process those images
again, resubmit the original messages rather than the saved JSONL. No separate image files or caption cards are created,
though the wrapper's internal Agent state under `mem_session/agentscope` can contain image inputs.
## Message Timestamps ## Message Timestamps
Auto Memory preserves each message's `created_at` in both the prompt and the raw session JSONL. When importing historical Auto Memory preserves each retained message's `created_at` in both the prompt and the source conversation JSONL. When importing historical
conversations or benchmark data, provide the actual occurrence time for every message so the model does not confuse event conversations or benchmark data, provide the actual occurrence time for every message so the model does not confuse event
time with execution time: time with execution time:
@ -92,8 +132,8 @@ For compatibility with common dataset schemas, `auto_memory` also checks `time_c
`timeCreated`, and `created_time` when `created_at` is absent. These fields may appear either at the top level of a message `timeCreated`, and `created_time` when `created_at` is absent. These fields may appear either at the top level of a message
or inside `metadata`. or inside `metadata`.
When a call does not explicitly provide `date`, Auto Memory uses the date of the earliest valid `created_at` value in the When a call does not explicitly provide `date`, Auto Memory uses the latest valid `created_at` date in the messages. If no
messages. If no message contains a valid timestamp, it falls back to the current date. Historical imports may also specify the message contains a valid timestamp, it falls back to the current date. Historical imports may also specify the
target date directly: target date directly:
```bash ```bash
@ -105,5 +145,13 @@ reme auto_memory \
## What Happens Next ## What Happens Next
The default `auto_memory` and `auto_memory_cc` jobs run `auto_tag_step` after recording memory. Only a daily note that
was actually created or modified is tagged, using its final path after any rename. Claude Code callers still pass only
`session_id`; repeated Stop events with no new messages skip both memory generation and tagging.
Tags describe the document's central entities and are stored in the configured frontmatter key (`memory_tags` by default).
Per-file tagging failures are reported in `metadata.auto_tag` while preserving the memory response. Calls without note
changes do not automatically retry failed tagging; the existing file watcher updates the tag index asynchronously.
Auto Memory only creates memory in the daily layer. To distill this material further into long-term `digest/` nodes, use Auto Memory only creates memory in the daily layer. To distill this material further into long-term `digest/` nodes, use
[Auto Dream](./auto_dream.md). To search daily and digest content, use [Memory Search](./memory_search.md). [Auto Dream](./auto_dream.md). To search daily and digest content, use [Memory Search](./memory_search.md).

View file

@ -1,8 +1,8 @@
# Auto Resource `Beta` # Auto Resource `Beta`
Auto Resource is ReMe's entry point for interpreting resources and is currently in **Beta**. Resource files first enter Auto Resource is ReMe's entry point for interpreting resources and is currently in **Beta**. Resource files first enter
`resource/` by date and are then interpreted into daily resource cards. Each card's filename comes from the LLM-generated `resource/`, preferably under a date directory, and are then interpreted into daily resource cards. Each card's filename
frontmatter `name`, and `source_resource` links the card back to its original file. comes from the LLM-generated frontmatter `name`, and `source_resource` links the card back to its original file.
<p align="center"> <p align="center">
<img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%"> <img src="../figure/auto-memory-resource.svg" alt="ReMe Auto Memory and Auto Resource writing daily memory cards" width="92%">
@ -13,7 +13,7 @@ For the general file semantics of workspace layers, `resource/`, and `daily/`, s
[Auto Memory](./auto_memory.md). [Auto Memory](./auto_memory.md).
```text ```text
resource/YYYY-MM-DD/<resource_file> resource/[YYYY-MM-DD/]<resource_file>
├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # interpreted resource card ├─ step 1: daily/YYYY-MM-DD/<generated_name>.md # interpreted resource card
├─ step 2: source_resource points to the original resource ├─ step 2: source_resource points to the original resource
└─ step 3: daily/YYYY-MM-DD.md # daily index linking the cards └─ step 3: daily/YYYY-MM-DD.md # daily index linking the cards
@ -21,8 +21,8 @@ resource/YYYY-MM-DD/<resource_file>
## What It Records ## What It Records
Auto Resource does more than copy file content. It extracts information that will make the resource easier to retrieve and Auto Resource does more than copy file content. It extracts information that will make the resource easier to retrieve
understand later: and understand later:
- Core content: what the resource is mainly about. - Core content: what the resource is mainly about.
- Structure: its sections, tables, fields, and data organization. - Structure: its sections, tables, fields, and data organization.
@ -34,26 +34,93 @@ In short, it turns "a file was archived" into "the resource is usable."
## Original Resource Entry Point ## Original Resource Entry Point
Auto Resource uses `resource/` as the entry point for source material. Resources must be placed under a date, which determines Auto Resource uses `resource/` as the entry point for source material. Date directories are recommended, and their date
the day whose daily memory layer receives the interpreted card. determines which daily memory layer receives the interpreted card. A file directly under `resource/` is also supported
and uses today in the application timezone when it is first processed. On later days, an exact `source_resource` match
keeps updates and deletion tied to that original daily card instead of creating a new card or leaving an orphan.
Example directory: Example directory:
```text ```text
workspace/ workspace/
resource/ resource/
quick-note.txt # enters today's daily layer
2026-06-20/ 2026-06-20/
market-report.md market-report.md
meeting-notes.csv meeting-notes.csv
``` ```
The current Beta version is best suited to text-based resources such as `md`, `txt`, `json`, `jsonl`, `csv`, `yaml`, Text resources such as `md`, `txt`, `json`, `jsonl`, `csv`, `yaml`, and `html` are the primary fit. Image resources
and `html`. (`png`, `jpg`, `jpeg`, `webp`, `gif`, `bmp`, `tiff`, `heic`) produce caption cards as described in
[Image Resources](#image-resources).
Internally, one `AutoResourceStep` receives each change batch and sends every item to the first configured processor
whose class-level matcher accepts it. `AutoImageResourceStep` handles image suffixes and `AutoTextResourceStep` is the
final fallback. A new modality can therefore add a registered processor, its prompt, and one `dispatch_steps` entry
without changing the router.
## Image Resources
Text and image resources share the same agent-wrapper and note-writing tools. Image inputs add a native AgentScope
image block alongside the interpretation instructions; the agent writes a caption card linked to the original image.
The card body starts with an `![[resource/...]]` embed link and the frontmatter carries `kind: image` and `media_type`,
so text search reaches image content through the caption.
Image processing is enabled by default (`include_images=true`) and requires an AgentScope wrapper bound to a compatible
model and formatter. Configure the model through `components.agent_wrapper.<name>.as_llm`, selecting the wrapper with
`agent_wrapper` on the resource Step. The former image-Step `as_llm` override and automatic `as_llm.vision` selection
are replaced by that binding. There is no separate caption model, schema-extraction call, or text-only retry after an
agent failure. An agent workflow can make multiple model requests while using its tools.
Each image interpretation starts a new session. Use the returned `agent_session_id` to find its processing record;
reprocessing the same image still updates the original card. The body should contain the image embed followed by a
description or transcription under `## Caption`, not an empty caption or a JSON response. Leave `status` to later
processing steps and keep its existing value when updating the card.
Customize image instructions with `prompt_dict.resource_instructions` (`resource_instructions_zh` for Chinese).
Rename existing `user_message` / `user_message_zh` settings accordingly.
Shared create/update templates insert these instructions at `{resource_instructions}`; older templates
without the placeholder receive them at the end.
Set `include_images=false` on an `auto_resource` call or as a Job default to skip **all** image events, including
deletions. Call-time values override Job defaults; when neither is set, image processing is enabled. For the watcher, use
`jobs.resource_watch_loop.include_images=false`; for manual calls, use `jobs.auto_resource.include_images=false`.
The image processor reports each skip in the existing result and warning log; text processing is unchanged. Existing
image cards are left untouched, even if their source image is deleted. Re-enabling images does not replay skipped
events; explicitly submit the affected paths to `auto_resource` when compensation is needed. The wrapper's configured
image-count limit is respected and must allow at least one image per resource call; it is not increased automatically.
Configure the wrapper when starting the persistent service. For example, to allow one image per agent context:
```bash
reme start components.agent_wrapper.default.context_config.max_image_num=1
```
The watcher processes resource changes automatically. To explicitly reprocess an existing `resource/photo.png`, run
the client in another terminal using the same workspace:
```bash
reme auto_resource include_images=true changes='[{"path":"resource/photo.png","change":"modified"}]'
```
Images wider or taller than 2048px are downscaled,
and provider-unfriendly formats are re-encoded, in memory for the request only; the original file under
`resource/` is never modified. Before a full decode, image dimensions are checked against a default limit of 40,000,000
pixels; images over the limit and Pillow decompression-bomb warnings fail only that resource. EXIF orientation is
applied to the in-memory request copy before resizing or conversion. Oversized JPEGs first use decoder-level
downsampling, followed by a final thumbnail pass when needed. The VLM request MIME and the card's frontmatter
`media_type` use the format Pillow detects from the image bytes, rather than trusting the filename extension. When an
image changes, its card is updated in place; when the image is deleted, the card is removed with it, provided image
processing is enabled.
Image preprocessing uses Pillow from the `core` extra. HEIC resources additionally require the optional
`image-heif` extra: `pip install "reme-ai[image-heif]"`. Other supported image formats do not load or require the HEIF
plugin.
## Resource Cards ## Resource Cards
Each resource file produces one daily resource card. The system initially uses the resource file's stem as a temporary path. Each resource file produces one daily resource card. The system initially uses the resource file's stem as a temporary
After the agent writes the card, the file is renamed according to its frontmatter `name`: path. After the matching processor writes the card, the file is renamed according to its frontmatter `name`:
```text ```text
resource/2026-06-20/market-report.md resource/2026-06-20/market-report.md
@ -67,14 +134,19 @@ The resource card links to the original file through frontmatter:
source_resource: "[[resource/2026-06-20/market-report.md]]" source_resource: "[[resource/2026-06-20/market-report.md]]"
``` ```
When a resource changes, Auto Resource finds and updates the corresponding card through `source_resource`. When a resource is When a resource changes, Auto Resource finds and updates the corresponding card through an exact `source_resource`
deleted, its daily note is also removed. The older `daily/YYYY-MM-DD/<resource_stem>.md` naming convention remains supported match. When an enabled resource is deleted, only the explicitly linked daily note is removed. A same-stem note without that
as a fallback. provenance marker is treated as user-owned and left untouched; new resource cards use a collision-free path instead.
A failed call may still have changed a card; `modified` records whether the file changed. If the agent writes the card
and then fails or is cancelled, the written content stays on disk. ReMe tries to complete metadata and update the day's
index for the card linked through `source_resource`, while preserving the original error or cancellation. A failed
image-note format check also leaves the written content in place. Failed calls are not retried automatically.
## Daily Index ## Daily Index
Resource cards enter the same daily memory layer as Auto Memory cards. The day's `YYYY-MM-DD.md` page acts as an index and Resource cards enter the same daily memory layer as Auto Memory cards. The day's `YYYY-MM-DD.md` page acts as an index
organizes those resource cards: and organizes those resource cards:
```text ```text
daily/ daily/
@ -91,11 +163,12 @@ resource, open its corresponding resource card.
The interpreted daily note is optimized for readability; the original resource is retained for trust and verification. The interpreted daily note is optimized for readability; the original resource is retained for trust and verification.
Auto Resource does not move the original file. It remains under `resource/YYYY-MM-DD/`. Text resources can therefore enter Auto Resource does not move the original file. It remains at its original path under `resource/`. Resources can
the daily memory flow while their source files stay in their original location. therefore enter the daily memory flow while their source files stay in their original location.
## What Happens Next ## What Happens Next
Auto Resource only creates resource interpretations in the daily layer. To distill long-term knowledge from resources into Auto Resource only creates resource interpretations in the daily layer. To distill long-term knowledge from resources
`digest/`, use [Auto Dream](./auto_dream.md). To search original resources, daily cards, and digest nodes, use into `digest/`, use [Auto Dream](./auto_dream.md). The default live index covers daily cards and digest nodes. Manual
[Memory Search](./memory_search.md). `reindex` only rebuilds search indexes from chunks already accepted by an ingestion path; it does not add the original
resource files to search. See [Memory Search](./memory_search.md).

172
docs/en/blog_20260920.md Normal file
View file

@ -0,0 +1,172 @@
# ReMe Memory Tags
Any memory system used over the long term eventually runs into a deceptively simple problem: **as memories accumulate, how do you search only the right subset?**
Suppose you and an agent have discussed three projects, all involving a launch, a budget, and an owner. Six months later, you ask:
> "What else do we need to confirm before launch?"
There is nothing wrong with the question, but it provides too few cues. Keyword search may retrieve every document that mentions "launch," while semantic search may blend experiences from several similar projects. Both find memories with similar content, but neither necessarily knows which project, company, or person you mean right now.
Human recall rarely works this way. We seldom run a full-text search across every experience at once. Instead, we begin with a few cues: **the ones about Alice, Project A, or that discussion from last year.** Once the scope narrows, the details begin to surface.
That is why ReMe adds memory tags. Each Markdown memory can express not only what it says, but also who or what it is mainly about—and that cue can participate directly in retrieval.
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-tags.svg" alt="ReMe builds an index from Markdown tags and filters the search scope" width="100%">
</p>
## Why Memory Tags?
ReMe already uses BM25 for keyword search, optional embeddings for semantic similarity, and Wikilinks for traversing relationships between memories. Memory tags do not replace any of them. They add another dimension: **retrieval scope.**
Think of the three mechanisms as answering different questions:
- The query answers, "What am I looking for now?"
- A Wikilink answers, "Which memories are related to this one?"
- A memory tag answers, "Which memories should I search first?"
For example, "How did we handle the budget overrun?" may apply to many projects. If the search also includes `Project_A`, the agent can first narrow the scope to files related to Project A, then look for the specific details about the overrun.
Directories cannot fully solve this problem. A meeting note may concern Alice, Project A, and a customer at the same time, but a file normally occupies only one place on disk. Tags give the same memory multiple entry points without changing its original directory structure.
## Let Each Memory Say Who or What It Is About
ReMe memories remain plain Markdown. Tags live directly in YAML frontmatter, for example:
```markdown
---
name: Project A pre-launch checklist
description: Alice confirmed the launch window, rollback conditions, and customer notification order.
memory_tags:
- Alice
- Project_A
---
Project A is scheduled to launch on Thursday evening. Complete regression
testing first and have Alice confirm the customer notification. Roll back if
the error rate exceeds the agreed threshold.
```
The default field is named `memory_tags`. The name is intentional: this is not a loose collection of broad article keywords. It answers a more stable question:
> **Which real-world person or thing is this Markdown memory about?**
An entity can be a person, organization, company, project, or asset—for example, `Alice`, `CATL`, `Project_A`, or `Gold`. Compared with broad topics such as "work," "important," or "meeting," entities make better anchors for long-term memory because people, organizations, and projects tend to recur across many conversations.
In the default configuration, Auto Memory (`auto_memory`, `auto_memory_cc`) and Auto Dream (`auto_dream`, `dream_cron`) generate these tags for daily and digest Markdown files actually added or modified during the current run. Before tagging, the workflow reads the full document and its existing frontmatter, then checks tags already used in the workspace. It prefers an existing spelling for the same entity so that `Project_A`, `project a`, and `项目A` do not silently become three separate tags. Manual imports and edits do not trigger automatic tagging; existing `memory_tags` values are synchronized to the Tag Index by the file-watching workflow.
By default, a file receives only its most important entity. Multiple tags are used only when the document genuinely centers on multiple independent entities, and the total remains limited. A document without a clear core entity can use an empty list:
```yaml
memory_tags: []
```
This matters more than tagging for its own sake. More tags do not make a memory richer; too many broad tags only turn every filtered search back into a workspace-wide search.
Of course, `memory_tags` is only ReMe's default convention. The frontmatter field read by the tag index is configurable, and tag values remain under the user's control. Teams that already use `entities`, `people`, or another field can adapt the index to their files instead of migrating Markdown into a closed format.
## How Does the Tag Index Work?
After reading frontmatter, ReMe builds two simple cue maps: which files belong to a tag, and which tags belong to a file. For example:
```text
Alice -> daily/project-a-launch.md
Project_A -> daily/project-a-launch.md
daily/project-a-launch.md -> Alice, Project_A
```
This is a bidirectional index derived from Markdown files. Relationships update when memories are created or modified, and stale relationships disappear when files are deleted. Tag comparison is case-insensitive and normalizes details such as whitespace, reducing accidental splits caused by spelling variations.
The index does not replace files or become a new source of truth. The real tags remain in user-visible, editable frontmatter. If the index is lost, it can be rebuilt from the Markdown metadata in the current file graph:
```bash
reme reindex scope=tag
```
This follows ReMe's usual principle: **files belong to the user, indexes serve the files, and indexes are always rebuildable.**
To inspect the tags in a workspace and see how many files each tag covers, list them directly:
```bash
reme list_tags order_by=file_count order=desc
```
Besides supporting search, this makes the structure of the memory workspace observable. You can quickly see that a project has accumulated many memories, or notice that one person's name has been split across several near-duplicate spellings.
## How Do Tags Participate in Search?
The most important role of memory tags is not display, but filtering.
Consider the earlier example. A natural-language query by itself looks like this:
```bash
reme search query="What else do we need to confirm before launch?"
```
That searches the entire searchable memory scope. Add a tag:
```bash
reme search \
query="What else do we need to confirm before launch?" \
tags='["Project_A"]'
```
ReMe first uses the Tag Index to find files tagged `Project_A`. BM25 and optional vector retrieval then produce direct matches only from those files, after which ranking fusion proceeds as usual.
There is one important boundary: tag filtering constrains direct retrieval hits, but it does not cut off Wikilink relationships. Default link expansion may still list the paths, names, and descriptions of neighboring memories outside the tag scope so the agent can decide whether to read further. Those neighbors do not become direct keyword or vector-search hits merely because they were listed.
The flow can be summarized as follows:
```text
Natural-language question + tag cue
↓
Tag Index identifies candidate files
↓
Keyword / semantic search within those files
↓
Return direct matching passages and optionally list relationships
(related neighbors may fall outside the tag scope)
```
Tag filtering can also be combined with date conditions—for example, to inspect memories created for a project during the last month. Each condition narrows a separate dimension: the entity specifies who or what, the date specifies when, and the query specifies what you want to know.
When several tags are supplied, ReMe currently keeps files that match any of them. For example, `tags=[Alice, Project_A]` retrieves memories about Alice or Project A, then lets the query determine which results rank first. This lets an agent widen the candidate set with several plausible entity cues without returning to a workspace-wide search.
## What Changes in Practice?
Memory tags do not make a tag mandatory for every search. Searches without tags continue to work as before. The real change is that when a user or agent already knows part of the context, that context no longer has to remain hidden inside a vague query.
### 1. The Same Question Is Less Likely to Drift into Another Project
"Why was it delayed last time?", "Who approved the budget?", and "What remains before launch?" all depend heavily on context. Tags establish the project or person first, reducing the chance that memories with similar names or content enter the candidate set.
### 2. Memories About the Same Entity Can Accumulate Across Time
Alice may appear in meeting notes, project decisions, personal preferences, and retrospectives. Those files do not need to move into one directory. A shared tag creates an entity view across directories and dates.
### 3. Memory Structure Is Visible to Both People and Agents
Tags are not internal fields hidden in a specialized database. Users can open, edit, and review them in Markdown. An agent can inspect the tags that exist before deciding which entity cue to include in a search. Incorrect tags can be found, and naming can converge over time.
### 4. Search Becomes Easier to Explain
When a result is unexpected, the pipeline can be inspected step by step: does the document contain the right `memory_tags`, does the Tag Index include the path, or did keyword and semantic ranking fail to match it? This chain is easier to diagnose and correct than one opaque relevance score.
## Tags Are Retrieval Cues, Not a Taxonomy
The goal is not to turn a personal knowledge base into a carefully maintained classification tree. Real memories naturally overlap: one conversation may involve both a person and a project, while one decision may belong to today's meeting and shape a retrospective months later.
Memory tags are closer to the retrieval cues used by human memory. Seeing a person's name reminds us of shared experiences; thinking about a project brings related decisions, problems, and commitments to mind. A cue is not the memory itself, but it helps us enter the right context faster.
What ReMe does is deliberately simple:
- Preserve complete, readable memories in Markdown.
- Use `memory_tags` to express who or what a memory is about.
- Connect entities and files through a rebuildable Tag Index.
- Narrow the scope by tag before using keywords, semantics, and links to find the answer.
In this way, memory becomes more than a collection of full-text-searchable documents. It begins to acquire a structure that better matches how people associate ideas.
When you say, "That Alice project from last time," the agent receives more than a sentence. It receives a cue it can actually follow back into the past.

178
docs/en/configuration.md Normal file
View file

@ -0,0 +1,178 @@
---
title: Configuration
description: ReMe configuration files, environment expansion, command-line overrides, and core components.
---
# Configuration
ReMe uses YAML or JSON to describe its Service, Jobs, and Components. The built-in default is `reme/config/default.yaml`. Select another configuration at startup and apply command-line overrides when needed.
## Precedence
Configuration is merged in this order, with later values winning:
1. `application_defaults` from enabled plugins.
2. The selected file; `default` is used when none is specified.
3. CLI dot-notation overrides.
```bash
reme start
reme start config=demo
reme start config=cookbook
reme start config=/absolute/path/to/app.yaml
reme start service.port=8181 workspace_dir=/data/reme
```
`config` accepts a built-in name or a `.yaml`, `.yml`, or `.json` file. Overrides are deep-merged, so changing `service.port` preserves sibling service settings.
The optional `cookbook` variant extends `default` and composes the separately installed Auto Fin, Daily Paper, and
DingTalk plugins. It requires the three DingTalk application credential environment variables before configuration
loading. It also enables `text-embedding-v4` vector retrieval, uses AgentScope with
`${LLM_MODEL_NAME:-qwen3.8-max}` by default, and runs the DingTalk bridge through Claude Code with the same
`LLM_MODEL_NAME` and `LLM_API_KEY`.
## CLI values
Arguments use `key=value`; leading `-` or `--` is accepted:
```bash
reme start --service.port=8181 --service.web_enabled=false
```
Values support null, booleans, numbers, JSON arrays and objects, quoted JSON strings, and plain strings. Numeric-looking values with leading zeroes, such as `007`, remain strings. Quote values such as `"true"` in JSON when they must remain strings.
## Environment variables
Configuration recursively expands:
```yaml
api_key: ${LLM_API_KEY}
base_url: ${LLM_BASE_URL:-https://example.com/v1}
```
`${VAR}` fails when undefined; `${VAR:-default}` uses its fallback. ReMe also searches for `.env` from the command's working directory through at most five parents.
Keep secrets in `.env` or the process environment, never in committed configuration.
## Application fields
| Field | Default | Purpose |
|---|---|---|
| `app_name` | `ReMe` | Display name |
| `workspace_dir` | `.reme` | User-owned workspace root, normalized to an absolute path |
| `metadata_dir` | `metadata` | Rebuildable indexes, graphs, and catalogs |
| `session_dir` | `session` | Agent sessions; standard transcripts use `session/dialog` |
| `mem_session_dir` | `mem_session` | Agent-wrapper sessions and configuration |
| `resource_dir` | `resource` | External resources |
| `daily_dir` | `daily` | Daily memory |
| `digest_dir` | `digest` | Consolidated long-term memory |
| `timezone` | `Asia/Shanghai` | IANA timezone used for dates and cron jobs |
| `language` | empty | Default language for LLM interactions |
| `plugins` | `[]` | Installed plugins enabled for this Application |
| `service` | HTTP | Service configuration |
| `jobs` | default Jobs | Job configurations by name |
| `components` | defaults | Components grouped by type and name |
`session_dir` must remain workspace-relative.
## Enabling jobs
`jobs.<name>.enabled` defaults to `true` for all Job types. Disabled jobs retain their configuration but do not
start, expose service interfaces, or accept calls through `Application.run_job()` / `run_stream_job()`.
For example, disable ReMe's Dream cron when a host plugin owns the schedule:
```bash
reme start jobs.dream_cron.enabled=false
```
Restart the service to apply the override. The separate `auto_dream` API remains available, and other jobs continue running.
`enable_serve` independently controls service exposure: `enabled=true, enable_serve=false` keeps a Job available for
local calls. Background and cron jobs are never service-exposed.
## LLM
The default LLM uses an OpenAI-compatible interface:
```yaml
components:
as_llm:
default:
backend: openai
model: qwen3.7-plus
context_size: 200000
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
```
Built-in registrations include `openai`, `anthropic`, `dashscope`, `deepseek`, `gemini`, `moonshot`, `ollama`, and `xai`. Their detailed model fields follow the corresponding AgentScope wrappers.
File operations, BM25 search, wikilink traversal, and `proactive_read` do not require an LLM. Evolution workflows such
as `auto_memory`, `auto_resource`, `auto_dream`, and proactive refresh do.
## Embeddings
Vector retrieval is disabled by default. Credentials alone do not enable it: configure `as_embedding`, `embedding_store`, and connect the store to `file_store`.
```yaml
components:
as_embedding:
default:
backend: openai
model: text-embedding-v4
dimensions: 1024
credential:
api_key: ${EMBEDDING_API_KEY}
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
embedding_store:
default:
backend: local
as_embedding: default
file_store:
default:
backend: local
embedding_store: default
keyword_index: default
file_graph: default
```
Rebuild the embedding index after changing the model or dimensions.
## Service and Jobs
Minimal HTTP configuration:
```yaml
service:
backend: http
host: 127.0.0.1
port: 2333
web_enabled: true
mcp_enabled: true
mcp_path: /mcp
```
A Job declares a backend, parameter schema, and ordered Steps:
```yaml
jobs:
example:
backend: base
description: Example job
parameters:
type: object
properties:
text: { type: string }
required: [text]
steps:
- backend: example_step
```
Set `enable_serve: false` to keep a Job internal. Background and cron Jobs are never service-exposed.
## Inspect the effective configuration
```bash
reme app_config
```
The result is the merged, validated configuration with secrets redacted. Use it when diagnosing plugin or override precedence. The authoritative contracts remain `reme/schema/application_config.py` and `reme/config/default.yaml`.

View file

@ -8,12 +8,12 @@ ReMe is open source and hosted on GitHub:
## How to Contribute ## How to Contribute
Thank you for your interest in ReMe. ReMe is a file-first, self-evolving memory system for agents. Contributions are welcome Thank you for your interest in ReMe. ReMe is a file-first, self-evolving memory system for agents. Contributions are
through issue reports, documentation improvements, additional tests, bug fixes, and new capabilities. welcome through issue reports, documentation improvements, additional tests, bug fixes, and new capabilities.
If this is your first time running ReMe locally, start with [Quick Start](./quick_start.md). If your change affects runtime If this is your first time running ReMe locally, start with [Quick Start](./quick_start.md). If your change affects
layers, Jobs, Steps, or components, read [ReMe Framework](./framework.md). If it affects workspace directories, frontmatter, runtime layers, Jobs, Steps, or components, read [ReMe Framework](./framework.md). If it affects workspace directories,
wikilinks, or chunking, read [Memory as File](./memory_as_file.md). frontmatter, wikilinks, or chunking, read [Memory as File](./memory_as_file.md).
### 1. Before You Begin ### 1. Before You Begin
@ -21,9 +21,10 @@ Before investing in an implementation:
- Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) for an existing issue or discussion. - Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) for an existing issue or discussion.
- If a related issue is still open, comment that you would like to work on it to avoid duplicate effort. - If a related issue is still open, comment that you would like to work on it to avoid duplicate effort.
- If no issue exists, create one describing the context, expected behavior, possible implementation, and scope of impact. - If no issue exists, create one describing the context, expected behavior, possible implementation, and scope of
- For larger feature changes, align with maintainers on interfaces, configuration, compatibility, and test strategy before impact.
submitting an implementation. - For larger feature changes, align with maintainers on interfaces, configuration, compatibility, and test strategy
before submitting an implementation.
### 2. Local Development Environment ### 2. Local Development Environment
@ -38,7 +39,11 @@ The project requires Python 3.11 or later. A virtual environment is recommended:
```bash ```bash
python -m venv .venv python -m venv .venv
source .venv/bin/activate source .venv/bin/activate
pip install -e ".[dev,full]" pip install -e reme_studio -e ".[dev,full]"
cd reme_studio
npm ci
npm run build:static
cd ..
pre-commit install pre-commit install
``` ```
@ -53,8 +58,8 @@ CLI / Client -> Service -> Application -> Job -> Step -> Component / Workspace
In practice: In practice:
- Capabilities exposed to users or external systems should normally be orchestrated by a Job, then exposed by a Service as a - Capabilities exposed to users or external systems should normally be orchestrated by a Job, then exposed by a Service
CLI-, HTTP-, or MCP-callable interface. as a CLI-, HTTP-, or MCP-callable interface.
- Reusable infrastructure belongs in `reme/components/`, with dependencies declared through `BaseComponent.bind()`. - Reusable infrastructure belongs in `reme/components/`, with dependencies declared through `BaseComponent.bind()`.
- Atomic business operations belong in `reme/steps/` and access the file store, agent wrapper, catalog, LLM, and other - Atomic business operations belong in `reme/steps/` and access the file store, agent wrapper, catalog, LLM, and other
components through `BaseStep.Ref`. components through `BaseStep.Ref`.
@ -65,31 +70,33 @@ In practice:
When adding a Step or Job, pay particular attention to these conventions: When adding a Step or Job, pay particular attention to these conventions:
- Register implementations with `@R.register("<backend_name>")`. Registration names should be stable, clear, and match the - Register implementations with `@R.register("<backend_name>")`. Registration names should be stable, clear, and match
configured `backend`. the configured `backend`.
- After adding a Step file, make sure its package `__init__.py` imports the module; otherwise, the registry will not load it. - After adding a Step file, make sure its package `__init__.py` imports the module; otherwise, the registry will not
load it.
- A Step should perform one atomic business operation. Cross-step flows belong in Job configuration or a dedicated - A Step should perform one atomic business operation. Cross-step flows belong in Job configuration or a dedicated
orchestration Step. orchestration Step.
- A Job composes Steps and selects normal, streaming, background, or scheduled execution. `enable_serve` controls whether it - A Job composes Steps and selects normal, streaming, background, or scheduled execution. `enable_serve` controls
is externally exposed. whether it is externally exposed.
- When a Step needs components, prefer `BaseStep.Ref`. Do not reconstruct global components inside a Step or bypass - When a Step needs components, prefer `BaseStep.Ref`. Do not reconstruct global components inside a Step or bypass
`ApplicationContext`. `ApplicationContext`.
- File, index, graph, frontmatter, and wikilink behavior must preserve consistent workspace-relative path semantics. - File, index, graph, frontmatter, and wikilink behavior must preserve consistent workspace-relative path semantics.
- Add fast tests under `tests/unit/` for new capabilities. Put cross-component, LLM, embedding, or service behavior under - Add fast tests under `tests/unit/` for new capabilities. Put cross-component, LLM, embedding, or service behavior
under
`tests/integration/` when appropriate. `tests/integration/` when appropriate.
### 4. Code and Documentation Changes ### 4. Code and Documentation Changes
Choose the appropriate entry point for the type of change: Choose the appropriate entry point for the type of change:
| Change type | Primary location | Guidance | | Change type | Primary location | Guidance |
|---|---|---| |-----------------------------------|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| Configuration or startup behavior | `reme/config/`, `reme/application.py`, `reme/reme.py` | Keep the default configuration runnable and avoid breaking existing CLI, HTTP, and MCP entry points. | | Configuration or startup behavior | `reme/config/`, `reme/application.py`, `reme/reme.py` | Keep the default configuration runnable and avoid breaking existing CLI, HTTP, and MCP entry points. |
| Component capability | `reme/components/` | Reuse `BaseComponent`, the registry, and context objects. | | Component capability | `reme/components/` | Reuse `BaseComponent`, the registry, and context objects. |
| Job or Step | `reme/components/job/`, `reme/steps/` | Follow the Job -> Step model in [ReMe Framework](./framework.md), keep request and response schemas clear, and add corresponding tests. | | Job or Step | `reme/components/job/`, `reme/steps/` | Follow the Job -> Step model in [ReMe Framework](./framework.md), keep request and response schemas clear, and add corresponding tests. |
| Data structure | `reme/schema/`, `reme/enumeration/` | Preserve serialization compatibility and existing frontmatter and wikilink semantics. | | Data structure | `reme/schema/`, `reme/enumeration/` | Preserve serialization compatibility and existing frontmatter and wikilink semantics. |
| Utility | `reme/utils/` | Keep function boundaries small and cover edge cases with unit tests. | | Utility | `reme/utils/` | Keep function boundaries small and cover edge cases with unit tests. |
| User documentation | `docs/en/`, `README.md` | Update documentation when user-visible behavior changes. | | User documentation | `docs/en/`, `README.md` | Update documentation when user-visible behavior changes. |
If a change involves an LLM, embeddings, an external service, file watching, or a background task, also describe its If a change involves an LLM, embeddings, an external service, file watching, or a background task, also describe its
dependencies, failure behavior, and local validation method. dependencies, failure behavior, and local validation method.
@ -165,15 +172,16 @@ pytest tests/unit/test_reme_cli.py
If `pre-commit` modifies files automatically, commit those changes and rerun the checks until everything passes. If `pre-commit` modifies files automatically, commit those changes and rerun the checks until everything passes.
The current pre-commit configuration includes YAML/TOML/JSON validation, private-key detection, trailing-whitespace checks, The current pre-commit configuration includes YAML/TOML/JSON validation, private-key detection, trailing-whitespace
checks,
`black`, `flake8`, `pylint`, and `pyroma`. The main formatting rules are: `black`, `flake8`, `pylint`, and `pyroma`. The main formatting rules are:
- `black --line-length=120` - `black --line-length=120`
- `flake8 --max-line-length=120` - `flake8 --max-line-length=120`
- `pylint --max-line-length=120` - `pylint --max-line-length=120`
Some integration tests may require an LLM, embeddings, or external service configuration. If you cannot run them locally, Some integration tests may require an LLM, embeddings, or external service configuration. If you cannot run them
state why they were skipped and what alternative validation you completed in the PR description. locally, state why they were skipped and what alternative validation you completed in the PR description.
### 8. Testing Requirements ### 8. Testing Requirements
@ -183,7 +191,8 @@ Add tests according to the risk of the change:
- For a new Step, Job, or component, cover at least the main path and a failure path. - For a new Step, Job, or component, cover at least the main path and a failure path.
- For changes to shared logic such as indexes, graphs, wikilinks, frontmatter, or file operations, add edge cases. - For changes to shared logic such as indexes, graphs, wikilinks, frontmatter, or file operations, add edge cases.
- For changes to the CLI, services, or configuration parsing, cover the user-visible entry point. - For changes to the CLI, services, or configuration parsing, cover the user-visible entry point.
- Documentation-only changes usually do not require new tests, but running `pre-commit run --all-files` is still recommended. - Documentation-only changes usually do not require new tests, but running `pre-commit run --all-files` is still
recommended.
Place tests according to the existing structure: Place tests according to the existing structure:
@ -200,6 +209,13 @@ Documentation lives under:
docs/ docs/
``` ```
User guides should have matching `docs/zh/` and `docs/en/` versions and appear in the corresponding navigation in
`docs/.vitepress/config.mts`. The ReMe Studio, TypeScript, plugin, and benchmark READMEs remain canonical in their own
directories; `github-pages/scripts/generate-content.mjs` mirrors them during builds. Never edit `.generated/` or `dist/`.
The Job API reference is generated from `reme/config/default.yaml`. Update that YAML and its tests when a default Job
contract changes rather than editing generated pages.
Documentation should: Documentation should:
- Use clear titles that directly identify a capability or flow. - Use clear titles that directly identify a capability or flow.
@ -207,15 +223,24 @@ Documentation should:
- Use real repository paths such as `reme/config/default.yaml`, `reme/steps/`, and `tests/unit/`. - Use real repository paths such as `reme/config/default.yaml`, `reme/steps/`, and `tests/unit/`.
- Describe default behavior according to the current code, `pyproject.toml`, and default configuration. - Describe default behavior according to the current code, `pyproject.toml`, and default configuration.
Validate the documentation site with:
```bash
cd github-pages
npm ci
npm test
npm run build
```
--- ---
## Getting Help ## Getting Help
- Bugs and feature requests: [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues) - Bugs and feature requests: [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues)
- Project home: [GitHub Repository](https://github.com/agentscope-ai/ReMe) - Project home: [GitHub Repository](https://github.com/agentscope-ai/ReMe)
- Documentation site: [https://reme.agentscope.io/](https://reme.agentscope.io/) - Documentation site: [https://reme.agentscope.io](https://reme.agentscope.io)
--- ---
Thank you for contributing to ReMe. Your improvements help make long-term memory for agents more readable, controllable, and Thank you for contributing to ReMe. Your improvements help make long-term memory for agents more readable, controllable,
maintainable. and maintainable.

156
docs/en/docker.md Normal file
View file

@ -0,0 +1,156 @@
---
title: Docker Deployment
description: Run ReMe and Studio in Docker with a user-owned persistent workspace.
---
# Docker Deployment
The image includes ReMe's `core` and HEIF image dependencies and the Studio static frontend. One HTTP process serves the API,
Studio at `/`, and MCP at `/mcp`. The default configuration keeps embeddings disabled; file operations and BM25 search do
not require model credentials.
## Build and run with Compose
Use Docker Engine or Docker Desktop with Compose **2.24.0 or newer**. From the repository root:
```bash
mkdir -p .reme
docker compose up --build -d
docker compose logs -f reme
```
Open <http://127.0.0.1:2333>. Compose binds the host port to loopback and mounts `./.reme` at `/data`. The source checkout and
Studio assets are not mounted over the installed application.
On Linux, if your workspace is not owned by UID/GID 1000, set the process identity before starting:
```bash
export REME_UID=$(id -u)
export REME_GID=$(id -g)
docker compose up --build -d
```
For model-powered memory evolution, copy `deploy/docker/example.env` to `.env` if you do not already have one, then fill in
your model credentials. Compose injects this optional file at runtime; Docker builds exclude `.env` files. Compose's
`--env-file` controls variable interpolation; `REME_ENV_FILE` selects the file injected into the container.
## Use a published image
The Docker workflow publishes `ghcr.io/agentscope-ai/reme:main` after successful main-branch checks. Stable GitHub releases
publish their package version and `latest`; prereleases do not update `latest`. Both Linux amd64 and arm64 images are tested
before their combined tags are published. Publication starts when the workflow is enabled in the repository.
```bash
mkdir -p "$HOME/.reme"
docker run -d --name reme \
--user "$(id -u):$(id -g)" \
-p 127.0.0.1:2333:2333 \
--mount "type=bind,source=$HOME/.reme,target=/data" \
--restart unless-stopped \
ghcr.io/agentscope-ai/reme:main
```
Add `--env-file /path/to/model.env` before the image name when using model credentials. For reproducible deployments,
replace `main` with a released version or image digest. To use the image with Compose, set `REME_IMAGE`, then run
`docker compose pull` and `docker compose up -d --no-build`.
## Paths, configuration, and ports
The image runs as UID/GID 1000 by default. `/data` contains the entire workspace: source sessions, resources, daily notes,
digest notes, and rebuildable metadata. Create the host directory yourself and make it writable by the configured user.
Mounting only `metadata/` does not preserve the source memories. Paths in a custom configuration refer to the container's
filesystem; additional paths need additional mounts.
| Setting | Meaning |
|---|---|
| `REME_WORKSPACE_DIR` | Container workspace; image default `/data`, fixed to `/data` by Compose |
| `REME_CONFIG` | Existing config name or mounted YAML/JSON path; unset uses the built-in default |
| `REME_HOST` | HTTP bind address; image and Compose use `0.0.0.0` |
| `REME_PORT` | Container port override; Compose defaults to `2333` |
| `REME_TIMEZONE` | Optional application timezone override; otherwise the application default applies |
| `REME_DATA_DIR` | Compose host workspace directory; default `./.reme` |
| `REME_PUBLISHED_PORT` | Compose host port; default `2333`, independent of the container port |
| `REME_BIND_ADDRESS` | Compose host bind address; default `127.0.0.1` |
| `REME_UID`, `REME_GID` | Compose process identity; both default to `1000` |
| `REME_ENV_FILE` | Optional Compose runtime environment file; default `.env` |
Explicit `start key=value` arguments override container environment settings, which override the loaded configuration for
those keys. Other keys retain ReMe's normal deep merge behavior. File logging defaults to off in the image; use container
logs. `log_to_file=true` explicitly enables file logs under `/app/logs`, which requires a separate mount to persist them.
The temporary home directory `/tmp/reme-home` and probe address file are disposable, not workspace storage.
To customize the full job/component configuration, copy `reme/config/default.yaml` to `reme.yaml`, edit it, set
`REME_CONFIG=/etc/reme/config.yaml` in `.env`, and add `compose.override.yaml`:
```yaml
services:
reme:
volumes:
- ./reme.yaml:/etc/reme/config.yaml:ro
```
Keep the `health_check` Job enabled and in `service.jobs` if you use an allowlist. The image's probe uses HTTP; when
overriding the service to CLI or MCP stdio, disable the Docker health check with `--no-healthcheck` (or Compose
`healthcheck: {disable: true}`).
To override startup without changing the image:
```bash
docker run --rm -p 127.0.0.1:2444:2444 \
--mount "type=bind,source=$HOME/.reme,target=/data" \
reme:local start service.port=2444 timezone=UTC
docker compose exec reme reme health_check
docker compose exec reme reme status
```
Other commands pass through unchanged, including `reme start job=version` for a one-shot job or `python` for diagnostics.
From a host CLI, supply the published address explicitly, for example `reme health_check host=127.0.0.1 port=2444`.
Host process discovery cannot reconstruct a container's startup arguments. Configure agent integrations to use the
published HTTP or MCP endpoint instead of starting a second native ReMe on the same workspace.
## Networking and optional tools
`127.0.0.1` inside a container refers to that container. A model server on the host needs a reachable host address, such as
`host.docker.internal` on Docker Desktop. On Linux, add `extra_hosts: ["host.docker.internal:host-gateway"]` to the service
and configure the model URL accordingly. Another Compose service is reachable by its service name.
The HTTP action API has no built-in authentication and includes write/delete operations. Keep the default loopback port
publication. For access from another machine, place an authenticated TLS proxy in front of the service and restrict direct
access to its port; the same restriction must cover Studio, HTTP Jobs, and MCP.
The image includes the configured agent SDK dependencies, but host OAuth files, transcripts, plugins, external MCP
executables, and host workspace paths are not automatically available. Mount required data explicitly and install extra
plugins/tools in a derived image so that recreating the container preserves the installation. Keep credentials out of
Docker build arguments and layers. Optional FAISS/zvec backends also depend on the capabilities of the target machine.
## Health, upgrade, and recovery
Docker posts to the existing `/health_check` Job and requires both `success=true` and `metadata.health.healthy=true`.
The probe address follows effective configuration and CLI port overrides, and bypasses outbound proxy settings.
Initialization has a 120-second health grace period; larger workspaces may need a longer Compose `healthcheck.start_period`.
This reports component health, not whether a remote model will accept a future request. An unhealthy Docker status alone
does not trigger `restart: unless-stopped`; that policy restarts exited processes.
Before upgrading, stop writes and back up the **complete** host workspace and your deployment configuration. Then:
```bash
docker compose stop
# Back up the configured host workspace here.
docker compose pull
docker compose up -d --no-build
docker compose exec reme reme health_check
```
For a locally built deployment, replace `pull` and `up --no-build` with `docker compose up --build -d`. Compose allows
60 seconds for orderly shutdown. Recreating containers leaves the bind-mounted workspace intact; use one ReMe writer
process per workspace. See [backup and recovery](./operations.md) for restoring derived state without deleting memory.
For container validation after a local build:
```bash
docker build -t reme:local .
python scripts/test_docker_image.py --image reme:local
```
The smoke check uses a disposable workspace and no model credentials. It verifies Studio, HTTP and MCP, file containment,
non-root execution, graceful shutdown, and memory search after replacing a container on a different port.

72
docs/en/faq.md Normal file
View file

@ -0,0 +1,72 @@
---
title: Frequently Asked Questions
description: Quick answers for ReMe installation, services, models, retrieval, files, and plugins.
---
# Frequently Asked Questions
## Do basic file operations require a model API key?
No. `write`, `read`, `list`, `stat`, BM25 search, wikilink traversal, and `proactive_read` work without model
credentials. `auto_memory`, `auto_resource`, `auto_dream`, and proactive refresh require an LLM.
## Why is search still BM25-only after setting an embedding key?
Embeddings are disabled by default. Configure `as_embedding` and `embedding_store`, then connect `file_store.default.embedding_store` to that component. See [Configuration](./configuration.md#embeddings).
## Why did `reme reindex` not discover a new file?
`reindex` rebuilds indexes from current `file_chunks`; it does not scan the workspace. Check `index_update_loop`, the watched directory and extension, and `health_check`.
## How do I use another workspace?
```bash
reme start workspace_dir=/absolute/path/to/memory
```
Ordinary CLI calls discover the running service, so they do not need the workspace argument again.
## What if port 2333 is occupied?
Do not stop an unknown listener. Select another port:
```bash
reme start service.port=8181
```
Then confirm it with `reme find_reme`.
## Why is an installed plugin missing its Jobs?
Installation only makes the distribution discoverable in the active Python environment. Enable it for the Application:
```bash
reme start plugins='["auto-fin"]'
```
Restart a running service after changing package or enablement state.
## May I edit workspace Markdown directly?
Yes. Files are the source of truth and watchers ingest changes. Keep frontmatter valid, use complete workspace-relative wikilinks, and avoid unconditional concurrent saves.
## May I expose ReMe publicly?
Not with the default configuration alone. Jobs can write and delete, HTTP CORS is permissive, and there is no general authentication layer. Use a controlled network or authenticated TLS reverse proxy and restrict `service.jobs`.
## How should I back up and migrate memory?
Stop writes and back up the complete workspace. `session/`, `resource/`, `daily/`, and `digest/` are the key sources; `metadata/` can be backed up or rebuilt. See [Diagnostics, Backup, and Recovery](./operations.md).
## Why is Studio unavailable?
The base `reme-ai` package has no frontend assets. Install `reme-ai[web]` or `reme-ai[core]`, or set `service.web_static_dir`. Missing Studio assets do not disable the Job API.
## Which capabilities does the running service expose?
```bash
reme help
reme app_config
```
Static documentation describes defaults; plugins and custom configuration may change the active service.

View file

@ -16,13 +16,13 @@ To run and use ReMe first, see [Quick Start](./quick_start.md). For workspace fi
### Capability Boundary ### Capability Boundary
ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into `digest/`, ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into
and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP. `digest/`, and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP.
Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current conversation, Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current
injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those capabilities belong in conversation, injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those
the host agent framework. ReMe accepts conversations, resources, and file changes that have already occurred and persists the capabilities belong in the host agent framework. ReMe accepts conversations, resources, and file changes that have
information with long-term value. already occurred and persists the information with long-term value.
```mermaid ```mermaid
flowchart LR flowchart LR
@ -38,16 +38,16 @@ flowchart LR
Core layers: Core layers:
| Layer | Main location | Responsibility | | Layer | Main location | Responsibility |
|---|---|---| |-------------|----------------------------|---------------------------------------------------------------------------------------------------|
| CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. | | CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. |
| Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. | | Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. |
| Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. | | Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. |
| Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. | | Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. |
| Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. | | Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. |
| Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. | | Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. |
| Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. | | Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. |
| Config | `reme/config/` | Default YAML configuration and command-line override parsing. | | Config | `reme/config/` | Default YAML configuration and command-line override parsing. |
## 2. Directory Structure ## 2. Directory Structure
@ -55,11 +55,13 @@ Core layers:
reme/ reme/
reme.py # CLI entry point reme.py # CLI entry point
application.py # Application assembly and lifecycle application.py # Application assembly and lifecycle
plugin.py # installed plugin contract and entry-point loader
config/ config/
default.yaml # default service / jobs / components default.yaml # default service / jobs / components
cookbook.yaml # Auto Fin + Daily Paper + DingTalk composition
config_parser.py # config=, dot notation, and env placeholder parsing config_parser.py # config=, dot notation, and env placeholder parsing
components/ components/
component_registry.py # global registry R component_registry.py # backend registry and application-local copies
base_component.py # ComponentMixin / BaseComponent / bind dependency declarations base_component.py # ComponentMixin / BaseComponent / bind dependency declarations
runtime_context.py # context for one Job execution runtime_context.py # context for one Job execution
job/ # BaseJob / StreamJob / BackgroundJob / CronJob job/ # BaseJob / StreamJob / BackgroundJob / CronJob
@ -68,18 +70,24 @@ reme/
file_store/ # file-index coordination layer file_store/ # file-index coordination layer
file_graph/ # wikilink graph file_graph/ # wikilink graph
keyword_index/ # BM25 and other keyword indexes keyword_index/ # BM25 and other keyword indexes
file_chunker/ # Markdown / default text chunking file_chunker/ # Markdown / JSON / JSONL / generic text chunking
file_catalog/ # change checkpoints file_catalog/ # change checkpoints
as_llm/, as_embedding/ # model wrappers as_llm/, as_embedding/ # model wrappers
agent_wrapper/ # AgentScope / Claude Code wrappers agent_wrapper/ # AgentScope / Claude Code / Codex wrappers
steps/ steps/
base_step.py # BaseStep, Ref, dispatch_steps base_step.py # BaseStep, Ref, dispatch_steps
common/ # version, help, health_check, demo common/ # version, help, health_check, status, chat
file_io/ # read/write/edit/delete/move/frontmatter/daily file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse index/ # watch/init/update/search/traverse
evolve/ # auto_memory, auto_resource, auto_dream, proactive evolve/ # auto_memory, auto_resource, auto_dream, proactive
transfer/ # upload/download/ingest transfer/ # upload/download
channel/ # MCP channel tools plugins/
dingtalk/ # independent DingTalk integration plugin distribution
auto-fin/ # independent example plugin distribution
daily_paper/ # independent paper-research plugin distribution
integrations/
claude_code/ # Claude Code adapter and marketplace
hermes_agent/ # Hermes Agent memory-provider adapter
``` ```
The default workspace directories are defined by `ApplicationConfig`: The default workspace directories are defined by `ApplicationConfig`:
@ -87,7 +95,8 @@ The default workspace directories are defined by `ApplicationConfig`:
```text ```text
<workspace_dir>/ <workspace_dir>/
metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state
session/ # agent sessions and original conversations session/ # source conversations used by memory workflows
mem_session/ # generated Agent wrapper sessions and configuration
resource/ # external resources resource/ # external resources
daily/ # lightly processed memory daily/ # lightly processed memory
digest/ # long-term digest memory digest/ # long-term digest memory
@ -127,13 +136,13 @@ reme search query="memory" backend=mcp
Configuration parsing supports: Configuration parsing supports:
| Capability | Source | Description | | Capability | Source | Description |
|---|---|---| |------------------------|-------------------------|--------------------------------------------------------------------------|
| Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. | | Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. |
| Explicit configuration | `config=<name-or-path>` | Accept a built-in configuration name or a YAML/JSON file path. | | Explicit configuration | `config=<name-or-path>` | Accept a built-in configuration name or a YAML/JSON file path. |
| Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. | | Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. |
| Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. | | Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. |
| Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. | | Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. |
### 3.2 Service ### 3.2 Service
@ -157,22 +166,28 @@ flowchart LR
HTTP service behavior: HTTP service behavior:
| Job type | HTTP exposure | | Job type | HTTP exposure |
|---|---| |-------------------------------------------|---------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | `POST /<job.name>` returning `Response` JSON. | | Non-`StreamJob` with `enable_serve: true` | `POST /<job.name>` returning `Response` JSON. |
| `StreamJob` | `POST /<job.name>` returning `text/event-stream`. | | `StreamJob` | `POST /<job.name>` returning `text/event-stream`. |
| `enable_serve: false` | No endpoint is registered. | | `enable_serve: false` | No endpoint is registered. |
After registering Job endpoints, the HTTP service can also mount the ReMe Studio single-page application. The default is
`service.web_enabled=true`. Builds are resolved from `service.web_static_dir`, `REME_WEB_STATIC_DIR`, the optional
`reme_studio` package installed by the `web` and `core` extras, and source-tree locations such as
`reme_studio/dist-static`. If no `index.html` is found, only the frontend is skipped and the Job API remains available. The
Studio `GET` fallback does not replace existing `POST /<job.name>` routes.
MCP service behavior: MCP service behavior:
| Job type | MCP exposure | | Job type | MCP exposure |
|---|---| |-------------------------------------------|-------------------------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. | | Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. |
| `StreamJob` | Currently skipped and not registered. | | `StreamJob` | Currently skipped and not registered. |
| `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. | | `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. |
MCP services can inject server-owned arguments with `injected_job_kwargs`; callers cannot override those arguments. MCP services can inject server-owned arguments with `injected_job_kwargs`; callers cannot override those arguments. Set
Set `tool_error_on_failure: true` to expose an unsuccessful ReMe `Response` as an MCP tool error. `tool_error_on_failure: true` to expose an unsuccessful ReMe `Response` as an MCP tool error.
## 4. Registry and Dependency Injection ## 4. Registry and Dependency Injection
@ -198,24 +213,56 @@ The registry key is:
`component_type` comes from a class attribute: `component_type` comes from a class attribute:
| Type | Class attribute | | Type | Class attribute |
|---|---| |-----------|-----------------------------------------------------------|
| Step | `BaseStep.component_type = ComponentEnum.STEP` | | Step | `BaseStep.component_type = ComponentEnum.STEP` |
| Job | `BaseJob.component_type = ComponentEnum.JOB` | | Job | `BaseJob.component_type = ComponentEnum.JOB` |
| Service | `BaseService.component_type = ComponentEnum.SERVICE` | | Service | `BaseService.component_type = ComponentEnum.SERVICE` |
| FileStore | `BaseFileStore.component_type = ComponentEnum.FILE_STORE` | | FileStore | `BaseFileStore.component_type = ComponentEnum.FILE_STORE` |
The same backend name can therefore exist under different component types. For example, `http` can be both a service backend The same backend name can therefore exist under different component types. For example, `http` can be both a service
and a client backend. backend and a client backend.
### 4.2 Registration Through Module Imports `ComponentEnum` provides the built-in identifiers, but installed plugins may declare a new type with a namespaced
string such as `example.reranker`. Custom identifiers use lowercase letters and numbers separated by `.`, `_`, or `-`.
They are configured under `components` and participate in the same dependency ordering and lifecycle as built-ins.
Registration happens when a module is imported. `reme/components/__init__.py` imports component packages, while ### 4.2 Built-in and Plugin Registration
`reme/steps/__init__.py` imports `channel/common/evolve/file_io/index/transfer`. Each package's `__init__.py` then imports
its concrete modules, causing `@R.register(...)` to execute.
After adding a Step file, make sure the package's `__init__.py` imports it. Otherwise, the backend will not appear in the Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
registry. bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point
group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to
`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The
entry-point name is the plugin's identity.
Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override.
Plugin registration therefore stays local to one application;
duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other.
The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration
files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. See the independently
packaged [DingTalk](../../plugins/dingtalk/README.md), [Auto Fin](../../plugins/auto-fin/README.md), and
[Daily Paper](../../plugins/daily_paper/README.md) plugins.
Plugin packages are managed locally and remain separate from per-application activation:
```bash
reme plugins list
reme plugins install plugins/dingtalk
reme plugins install plugins/auto-fin
reme plugins install plugins/daily_paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start config=cookbook
```
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
The built-in `cookbook` configuration composes the three plugins, adds DingTalk delivery to the two report pipelines,
and starts the DingTalk Agent bridge as a background Job. Enabling Auto Fin or Daily Paper alone keeps it independent
from DingTalk.
### 4.3 Component.bind ### 4.3 Component.bind
@ -234,18 +281,18 @@ flowchart LR
Rules for `BaseComponent.bind(name, BaseClass, optional=True)`: Rules for `BaseComponent.bind(name, BaseClass, optional=True)`:
| Scenario | Behavior | | Scenario | Behavior |
|---|---| |-----------------------------------------|------------------------------------------------------------|
| `name` is empty | Return `None` and skip the dependency. | | `name` is empty | Return `None` and skip the dependency. |
| `app_context` exists | Look up `app_context.components[ctype][name]`. | | `app_context` exists | Look up `app_context.components[ctype][name]`. |
| Dependency missing and `optional=True` | Resolve to `None`. | | Dependency missing and `optional=True` | Resolve to `None`. |
| Dependency missing and `optional=False` | Fail at startup. | | Dependency missing and `optional=False` | Fail at startup. |
| Standalone mode | A private component can be created with `default_factory`. | | Standalone mode | A private component can be created with `default_factory`. |
### 4.4 Step.Ref ### 4.4 Step.Ref
Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps access Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps
components primarily through `BaseStep.Ref`: access components primarily through `BaseStep.Ref`:
```python ```python
file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE) file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
@ -301,11 +348,13 @@ flowchart LR
F --> G["start CronJob"] F --> G["start CronJob"]
``` ```
During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their dependencies. During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their
dependencies.
## 6. Job Model ## 6. Job Model
A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under `jobs:` A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under
`jobs:`
in `reme/config/default.yaml`. in `reme/config/default.yaml`.
### 6.1 BaseJob ### 6.1 BaseJob
@ -326,23 +375,23 @@ flowchart LR
Important source behavior: Important source behavior:
| Source | Behavior | | Source | Behavior |
|---|---| |--------------------|----------------------------------------------------------------------------------|
| `_start()` | Parse each Step config from YAML into `(step_cls, params)`. | | `_start()` | Parse each Step config from YAML into `(step_cls, params)`. |
| `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. | | `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. |
| `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. | | `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. |
| Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. | | Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. |
### 6.2 StreamJob ### 6.2 StreamJob
`StreamJob` extends `BaseJob` but returns streaming chunks: `StreamJob` extends `BaseJob` but returns streaming chunks:
| Behavior | Description | | Behavior | Description |
|---|---| |-------------|------------------------------------------------------------|
| Context | Includes `stream_queue`. | | Context | Includes `stream_queue`. |
| Step output | Call `context.add_stream_string(text, ChunkEnum.CONTENT)`. | | Step output | Call `context.add_stream_string(text, ChunkEnum.CONTENT)`. |
| Exception | Write `ChunkEnum.ERROR`. | | Exception | Write `ChunkEnum.ERROR`. |
| Completion | Always send a `DONE` chunk. | | Completion | Always send a `DONE` chunk. |
### 6.3 BackgroundJob ### 6.3 BackgroundJob
@ -362,8 +411,8 @@ flowchart LR
J --> K["wait close_timeout; cancel on timeout"] J --> K["wait close_timeout; cancel on timeout"]
``` ```
The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions, which The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions,
allows the supervisor to restart the task. which allows the supervisor to restart the task.
### 6.4 CronJob ### 6.4 CronJob
@ -377,7 +426,6 @@ jobs:
steps: steps:
- backend: dream_extract_step - backend: dream_extract_step
- backend: dream_integrate_step - backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step - backend: dream_finish_step
``` ```
@ -389,7 +437,9 @@ The current implementation uses `croniter` to calculate the next trigger time. T
```mermaid ```mermaid
flowchart LR flowchart LR
Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"] Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"]
Jobs --> Base["base<br/>version / help / health_check<br/>search / node_search / traverse / reindex<br/>read / write / edit / delete / move / list / stat<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_resource / auto_dream / proactive"] Jobs --> Cron["cron<br/>dream_cron<br/>proactive_refresh_cron<br/>optimize_index_cron"]
Jobs --> Stream["stream<br/>chat"]
Jobs --> Base["base<br/>version / help / health_check / status / app_config<br/>search / node_search / traverse / graph_snapshot / reindex<br/>read / load / read_image / write / save / edit / delete / move / list / stat / frontmatter_*<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_memory_cc / auto_resource / auto_dream / proactive_refresh / proactive_read"]
``` ```
## 7. Step Model ## 7. Step Model
@ -413,12 +463,12 @@ flowchart LR
`RuntimeContext` is shared by all Steps within one Job invocation: `RuntimeContext` is shared by all Steps within one Job invocation:
| Field | Description | | Field | Description |
|---|---| |----------------|----------------------------------------------------------------------------|
| `response` | Final `Response(answer, success, metadata)`. | | `response` | Final `Response(answer, success, metadata)`. |
| `data` | Free-form dictionary containing input parameters and intermediate results. | | `data` | Free-form dictionary containing input parameters and intermediate results. |
| `stream_queue` | Output queue for streaming Jobs. | | `stream_queue` | Output queue for streaming Jobs. |
| `stop_event` | Stop signal for background Jobs. | | `stop_event` | Stop signal for background Jobs. |
Common Step code: Common Step code:
@ -482,21 +532,22 @@ flowchart LR
Current default components in `reme/config/default.yaml`: Current default components in `reme/config/default.yaml`:
| ComponentEnum | Name | Backend | Description | | ComponentEnum | Name | Backend | Description |
|---|---|---|---| |-------------------|---------------------------------|--------------------------------------------------|--------------------------------------------------------------------------------|
| `service` | singleton | `http` | Default HTTP service. | | `service` | singleton | `http` | Default HTTP service. |
| `tokenizer` | `default` | `regex` | BM25 tokenizer. | | `tokenizer` | `default` | `regex` | BM25 tokenizer. |
| `as_embedding` | `default` | `${EMBEDDING_BACKEND:-openai}` | Embedding model wrapper. | | `as_embedding` | `default` | Not configured by default; example uses `openai` | Provides the embedding model wrapper after uncommenting the example config. |
| `embedding_store` | `default` | `local` | Embedding store depending on `as_embedding: default`. | | `embedding_store` | `default` | Not configured by default; example uses `local` | Depends on `as_embedding: default` after uncommenting the example config. |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. | | `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. | | `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. | | `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. |
| `file_graph` | `default` | `local` | Wikilink graph. | | `agent_wrapper` | `codex/codex_oauth` | `codex` | Codex wrappers for API-key and OAuth authentication. |
| `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. | | `file_graph` | `default` | `local` | Wikilink graph. |
| `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. | | `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. |
| `file_chunker` | `default` | `default` | Default text chunking, currently supporting `jsonl`. | | `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. |
| `keyword_index` | `default` | `bm25` | BM25 keyword index. | | `file_chunker` | `json/jsonl/default` | `json/jsonl/default` | JSON, JSONL, and generic text chunkers; generic text supports `txt` and `log`. |
| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. | | `keyword_index` | `default` | `bm25` | BM25 keyword index. |
| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. |
Note that the `search` Step configuration contains `vector_weight`, but `file_store.default.embedding_store` is empty by Note that the `search` Step configuration contains `vector_weight`, but `file_store.default.embedding_store` is empty by
default. Vector retrieval is available only when the runtime configuration enables an embedding store. default. Vector retrieval is available only when the runtime configuration enables an embedding store.
@ -554,12 +605,12 @@ class MySearchStep(BaseStep):
Common attributes available directly: Common attributes available directly:
| Attribute | Component resolved by default | | Attribute | Component resolved by default |
|---|---| |----------------------|-------------------------------------|
| `self.as_llm` | `.model` from `as_llm: default`. | | `self.as_llm` | `.model` from `as_llm: default`. |
| `self.agent_wrapper` | `agent_wrapper: default`; optional. | | `self.agent_wrapper` | `agent_wrapper: default`; optional. |
| `self.file_catalog` | `file_catalog: default`; optional. | | `self.file_catalog` | `file_catalog: default`; optional. |
| `self.file_store` | `file_store: default`. | | `self.file_store` | `file_store: default`. |
To select a non-default component from Job configuration: To select a non-default component from Job configuration:
@ -571,13 +622,13 @@ steps:
### 9.4 Step Design Guidance ### 9.4 Step Design Guidance
| Guidance | Reason | | Guidance | Reason |
|---|---| |---------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
| Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. | | Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. |
| Write the final result to `context.response`. | Services and clients consume the standard `Response`. | | Write the final result to `context.response`. | Services and clients consume the standard `Response`. |
| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. | | Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. |
| A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. | | A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. |
| Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. | | Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. |
### 9.5 Unit Test Example ### 9.5 Unit Test Example
@ -600,8 +651,8 @@ async def test_uppercase_step():
## 10. Adding a Job ## 10. Adding a Job
A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new execution A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new
model is required. execution model is required.
### 10.1 Adding a Normal Request Job ### 10.1 Adding a Normal Request Job
@ -736,11 +787,11 @@ jobs:
Characteristics of a background Job: Characteristics of a background Job:
| Characteristic | Description | | Characteristic | Description |
|---|---| |---------------------------------|--------------------------------------------------------------------|
| Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. | | Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. |
| Has a supervisor | Restarts with exponential backoff after an exception by default. | | Has a supervisor | Restarts with exponential backoff after an exception by default. |
| Has a stop event | Notifies the loop to exit during close. | | Has a stop event | Notifies the loop to exit during close. |
| Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. | | Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. |
### 10.5 Adding a Cron Job ### 10.5 Adding a Cron Job
@ -756,7 +807,6 @@ jobs:
- backend: dream_extract_step - backend: dream_extract_step
file_catalog: dream file_catalog: dream
- backend: dream_integrate_step - backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step - backend: dream_finish_step
file_catalog: dream file_catalog: dream
``` ```
@ -767,14 +817,14 @@ An invalid `cron` expression fails at startup.
Most use cases require only a new Step plus a YAML Job. Consider adding `reme/components/job/*.py` only in these cases: Most use cases require only a new Step plus a YAML Job. Consider adding `reme/components/job/*.py` only in these cases:
| Requirement | New Job class? | | Requirement | New Job class? |
|---|---| |---------------------------------------------------------------------|--------------------------------|
| Add a business command | No; use `backend: base`. | | Add a business command | No; use `backend: base`. |
| Chain existing steps | No; use `steps:`. | | Chain existing steps | No; use `steps:`. |
| Need SSE/streaming output | No; use `backend: stream`. | | Need SSE/streaming output | No; use `backend: stream`. |
| Need a background loop | No; use `backend: background`. | | Need a background loop | No; use `backend: background`. |
| Need cron scheduling | No; use `backend: cron`. | | Need cron scheduling | No; use `backend: cron`. |
| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. | | Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. |
Minimal shape of a new Job backend: Minimal shape of a new Job backend:

9
docs/en/index.md Normal file
View file

@ -0,0 +1,9 @@
---
layout: home
markdownStyles: false
title: ReMe
titleTemplate: false
description: ReMe is a local-first, file-native memory system for agents.
---
<HomePage lang="en" />

100
docs/en/integrations.md Normal file
View file

@ -0,0 +1,100 @@
---
title: Agent Integrations
description: Connect ReMe to agents through the CLI, HTTP, MCP, Skills, and host adapters.
---
# Agent Integrations
ReMe keeps memory in an independent service and a user-owned workspace. Multiple agents can call the same memory system without binding storage to one model or host.
## Choose an interface
| Scenario | Recommended interface |
|---|---|
| Local script or hook | ReMe CLI |
| Application backend | HTTP Client |
| Tool-protocol host | MCP |
| DeepSeek Harness | [`@agentscope-ai/reme-dsh-plugin`](./integrations/dsh.md) profile bundle |
| OpenClaw | [`@agentscope-ai/reme-openclaw-plugin`](./integrations/openclaw.md) |
| Claude Code | [Shared HTTP MCP + Skill + Stop Hook](./integrations/claude-code.md) |
| Hermes Agent | Memory provider adapter |
| Codex or another coding agent | `reme_memory` Skill or MCP |
## General memory loop
1. Before answering, call `search` for relevant memory.
2. Use `read` on high-value results and `traverse` when relationships matter.
3. Retain workspace-relative source paths in the answer.
4. At session end, pass source messages to `auto_memory`.
5. Let background or scheduled workflows consolidate daily notes into digest memory.
An empty search result must remain empty; do not present model inference as recalled history.
## MCP
The default HTTP service exposes streamable HTTP MCP at `http://127.0.0.1:2333/mcp`. Common tools include `search`,
`read`, `traverse`, `list`, `auto_memory`, and `proactive_read`.
Use `service.jobs` to expose a read-only subset or keep write tools in a separate configuration.
## CLI and Skill
`skills/reme_memory/SKILL.md` defines a general workflow for agents that can run local commands: installation checks, service discovery, retrieval, reading, and persistence boundaries.
It deliberately avoids silently modifying Python environments, stopping unknown processes on port conflicts, writing recalled tool output back as conversation source, or persisting credentials.
## DeepSeek Harness
Install the self-contained [DeepSeek Harness plugin](./integrations/dsh.md):
```bash
dsh plugin --profile web add @agentscope-ai/reme-dsh-plugin
```
Release links: [Awesome DSH Plugin](https://awesome-dsh-plugin.com/p/agentscope-ai/ReMe--integrations-dsh/) and
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-dsh-plugin).
It injects long-term-memory usage guidance into new root-agent sessions and exposes the read-only `reme_search` tool;
it does not preload the full memory history into the prompt. Completed user/assistant turns can be submitted to
`auto_memory` in background batches, while a timezone-aware schedule runs `auto_dream` to consolidate daily notes.
DSH settings configure the endpoint, guidance language, search limits, capture interval, root-agent filtering, and
consolidation schedule. The ReMe Status page exposes Overview, Auto Memory, Memory Consolidation, Components, Journal,
and Personal Knowledge Base views. Runtime counters are diagnostic state; workspace Markdown remains the durable source
of truth.
## OpenClaw
Install the independently published [OpenClaw plugin](./integrations/openclaw.md):
```bash
openclaw plugins install clawhub:@agentscope-ai/reme-openclaw-plugin
```
Release links: [ClawHub](https://clawhub.ai/agentscope-ai/plugins/reme-openclaw-plugin) and
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-openclaw-plugin). The plugin provides its own host-specific
ReMe HTTP boundary and release lifecycle.
## Claude Code
The [Claude Code plugin](./integrations/claude-code.md) connects every Claude Code window to one ReMe HTTP process at
`http://127.0.0.1:2333/mcp` by default. The `reme-memory` Skill selects among semantic `search`, topological `traverse`,
and state-oriented `daily_list` / `frontmatter_read`, then reads and cites the relevant workspace paths.
On Stop, the hook passes only the Claude Code `session_id` to the server-side `auto_memory_cc` job. On POSIX systems it
detaches the potentially long model call so Claude Code can stop immediately; unreachable-service and other best-effort
failures are written to the plugin log instead of blocking the host. ReMe resolves the local transcript, and repeated
Stop events with no new messages do not create duplicate memory.
## Hermes Agent
`integrations/hermes_agent/` provides a memory provider with HTTP and embedded modes. It recalls context before model calls and asynchronously invokes `auto_memory` after each turn. Its `config_schema.py` is rendered by Hermes' generic memory settings UI.
## Production guidance
- choose a stable absolute `workspace_dir`;
- reuse a service discovered by `reme find_reme`;
- treat `reme help` as the active Job contract;
- apply timeouts and failure logging to writes;
- do not block the host's core response path when memory is temporarily unavailable;
- use authentication, TLS, and a minimal Job allowlist for remote access.

View file

@ -6,37 +6,39 @@ ReMe's core idea is **Memory as File, File as Memory**.
<img src="../figure/memory-as-file.svg" alt="ReMe Memory as File model" width="92%"> <img src="../figure/memory-as-file.svg" alt="ReMe Memory as File model" width="92%">
</p> </p>
**Memory as File**: long-term memory is not hidden in a black-box database. It lives in Markdown files, resource files, and **Memory as File**: long-term memory is not hidden in a black-box database. Its source material and readable memories
index snapshots under the workspace. Users and agents can directly read, write, move, and delete those files. live in user-owned files under the workspace. Users and agents can directly read, write, move, and delete those files;
indexes and snapshots under `metadata/` are derived state that can be rebuilt.
**File as Memory**: each file is more than ordinary text. It is an indexable, linkable, and evolvable memory node. ReMe parses **File as Memory**: each file is more than ordinary text. It is an indexable, linkable, and evolvable memory node. ReMe
frontmatter, body chunks, and wikilink edges from files and organizes them into retrieval indexes and a graph. parses frontmatter, body chunks, and wikilink edges from files and organizes them into retrieval indexes and a graph.
In other words, files are both a human-readable interface and an operational interface for agents. Directory structure In other words, files are both a human-readable interface and an operational interface for agents. Directory structure
carries the memory layers, while Markdown syntax expresses content, metadata, and relationships. carries the memory layers, while Markdown syntax expresses content, metadata, and relationships.
## Design Goals ## Design Goals
ReMe represents memory as files not merely for convenient storage, but to give long-term memory several essential properties: ReMe represents memory as files not merely for convenient storage, but to give long-term memory several essential
properties:
| Goal | Meaning | | Goal | Meaning |
|---|---| |---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| Readable | Users can open the workspace directly and read daily notes, digest nodes, and source material like ordinary notes. | | Readable | Users can open the workspace directly and read daily notes, digest nodes, and source material like ordinary notes. |
| Editable | Users and agents can correct, extend, move, or delete memory with file operations, without a specialized database client. | | Editable | Users and agents can correct, extend, move, or delete memory with file operations, without a specialized database client. |
| Traceable | Long-term conclusions in digest can point back to daily, resource, or session files from a Sources section. | | Traceable | Long-term conclusions in digest can point back to daily, resource, or session files from a Sources section. |
| Portable | The workspace is an ordinary directory. Markdown, JSONL, YAML, and resource files can be backed up, synchronized, versioned, or moved to other tools. | | Portable | The workspace is an ordinary directory. Markdown, JSONL, YAML, and resource files can be backed up, synchronized, versioned, or moved to other tools. |
| Indexable | Although the files are plain text, ReMe parses frontmatter, chunks, and wikilinks to build a retrieval index and file graph. | | Indexable | Although the files are plain text, ReMe parses frontmatter, chunks, and wikilinks to build a retrieval index and file graph. |
| Collaborative | Humans judge and correct; agents organize, link, and retrieve. Both operate on the same files. | | Collaborative | Humans judge and correct; agents organize, link, and retrieve. Both operate on the same files. |
ReMe memory is therefore neither a hidden database record nor a prompt fragment visible only to an LLM. It is first a file ReMe memory is therefore neither a hidden database record nor a prompt fragment visible only to an LLM. It is first a
owned by the user and only then indexed by the system for retrieval. file owned by the user and only then indexed by the system for retrieval.
## Memory Layers ## Memory Layers
A ReMe workspace divides memory into four layers: A ReMe workspace divides memory into four layers:
```text ```text
raw input -> session/ + resource/ source records -> session/ + resource/
working memory -> daily/ working memory -> daily/
long memory -> digest/ long memory -> digest/
system state -> metadata/ system state -> metadata/
@ -44,19 +46,23 @@ system state -> metadata/
Each layer solves a different problem. Each layer solves a different problem.
`session/` and `resource/` preserve raw input. Their purpose is to retain the original situation: conversations, agent `session/` and `resource/` preserve source records. Files under `resource/` remain unchanged at their original path.
sessions, uploaded material, web pages, and reports remain intact as evidence for later verification. Standard Auto Memory records retain conversation messages while intentionally omitting tool-result and base64 data
blocks; this keeps recalled output and binary payloads from masquerading as user-provided evidence. Generated Agent
runtime state instead lives under `mem_session/`.
`daily/` is the lightly processed layer. It organizes the day's conversations and resources into more readable daily notes: `daily/` is the lightly processed layer. It organizes the day's conversations and resources into more readable daily
what happened, which conclusions were reached, which follow-up tasks remain, and where the source material lives. Daily does notes:
not aim for final abstraction; it is closer to a workbench for the day. what happened, which conclusions were reached, which follow-up tasks remain, and where the source material lives. Daily
does not aim for final abstraction; it is closer to a workbench for the day.
`digest/` is the deeply processed layer. It stores memory nodes that can be reused over time, such as user preferences, `digest/` is the deeply processed layer. It stores memory nodes that can be reused over time, such as user preferences,
project background, procedural experience, conceptual knowledge, and decision precedents. Digest should not merely copy project background, procedural experience, conceptual knowledge, and decision precedents. Digest should not merely copy
daily. It should merge recurring facts, methods, and relationships into more stable descriptions. daily. It should merge recurring facts, methods, and relationships into more stable descriptions.
`metadata/` is the system index layer. It stores runtime state such as the file catalog, chunk index, and graph snapshots. `metadata/` is the system index layer. It stores runtime state such as the file catalog, chunk index, and graph
Users normally do not edit this content manually. The actual editing surface is `daily/`, `digest/`, and, when necessary, snapshots. Users normally do not edit this content manually. The actual editing surface is `daily/`, `digest/`, and,
when necessary,
`resource/`. `resource/`.
These layers let ReMe preserve both the original situation and its abstraction: daily reconstructs what happened, while These layers let ReMe preserve both the original situation and its abstraction: daily reconstructs what happened, while
@ -73,22 +79,24 @@ The corresponding automatic flows are [Auto Memory](./auto_memory.md), [Auto Res
```text ```text
<workspace_dir>/ <workspace_dir>/
├── metadata/ # system index layer; persistent indexes, graph, catalogs; not a manual editing surface ├── metadata/ # system index layer; persistent indexes, graph, catalogs; not a manual editing surface
├── session/ # raw input layer; original conversations and agent sessions ├── session/ # source-record layer; source conversations
│ ├── dialog/ │ ├── dialog/
│ │ └── <session_id>.jsonl # conversation messages saved by auto_memory │ │ └── <session_id>.jsonl # source messages saved by auto_memory
│ ├── agentscope/
│ │ └── <session_id>.jsonl
│ └── claude_code/ │ └── claude_code/
│ └── <session_id>.jsonl │ └── <session_id>.jsonl # ReMe copy used by auto_memory_cc
├── resource/ # raw input layer; original external material ├── mem_session/ # generated Agent wrapper sessions/config, not user memory
│ ├── agentscope/
│ ├── claude_config/
│ └── codex/
├── resource/ # source-record layer; original external material
│ ├── <resource>.<ext> # root-level input uses today's date
│ └── YYYY-MM-DD/ │ └── YYYY-MM-DD/
│ └── <resource>.<ext> │ └── <resource>.<ext> # dated input uses the directory date
├── daily/ # lightly processed layer; facts, conversation summaries, and resource interpretations by date ├── daily/ # lightly processed layer; facts, conversation summaries, and resource interpretations by date
│ ├── YYYY-MM-DD.md # index page for the day │ ├── YYYY-MM-DD.md # index page for the day
│ └── YYYY-MM-DD/ │ └── YYYY-MM-DD/
│ ├── <session_id>.md # daily note distilled from a conversation │ ├── <generated_name>.md # topic-named conversation or resource card
│ ├── <resource_stem>.md # daily note distilled from a resource │ └── interests.yaml # proactive interest topics generated by proactive refresh
│ └── interests.yaml # proactive interest topics generated by auto_dream
└── digest/ # deeply processed layer; reusable personal facts, procedures, and knowledge nodes └── digest/ # deeply processed layer; reusable personal facts, procedures, and knowledge nodes
├── personal/ ├── personal/
│ └── <memory>.md # user profile, preferences, and durable personal facts │ └── <memory>.md # user profile, preferences, and durable personal facts
@ -103,17 +111,21 @@ Typical flows:
```text ```text
conversation conversation
-> session/dialog/<session_id>.jsonl -> session/dialog/<session_id>.jsonl
-> daily/YYYY-MM-DD/<session_id>.md -> daily/YYYY-MM-DD/<generated_name>.md
-> digest/personal | digest/procedure | digest/wiki -> digest/personal | digest/procedure | digest/wiki
external resource external resource
-> resource/YYYY-MM-DD/<resource>.<ext> -> resource/[YYYY-MM-DD/]<resource>.<ext>
-> daily/YYYY-MM-DD/<resource_stem>.md -> daily/YYYY-MM-DD/<generated_name>.md
-> digest/wiki | digest/procedure -> digest/wiki | digest/procedure
``` ```
The first two steps focus on recording and organizing; the final step focuses on long-term distillation. `auto_memory` and The first two steps focus on recording and organizing; the final step focuses on long-term distillation. `auto_memory`
`auto_resource` generate daily notes from raw input, and `auto_dream` extracts and integrates digest nodes from daily. and
`auto_resource` generate daily notes from source input, and `auto_dream` extracts and integrates digest nodes from
daily. The generated daily filename comes from validated frontmatter `name`; `session_id`, `source_conversation`, and
`source_resource`
provide stable provenance and lookup identity instead of determining the filename.
## Markdown Format ## Markdown Format
@ -146,8 +158,8 @@ source_conversation: [[session/dialog/abc.jsonl]]
--- ---
``` ```
The current code recognizes `name` and `description` explicitly. Other fields are preserved as additional metadata. The write The current code recognizes `name` and `description` explicitly. Other fields are preserved as additional metadata. The
interface merges `name`, `description`, and `metadata` into frontmatter. write interface merges `name`, `description`, and `metadata` into frontmatter.
Treat frontmatter as a node-level summary and the body as evidence, explanation, and relationships. For example: Treat frontmatter as a node-level summary and the body as evidence, explanation, and relationships. For example:
@ -165,7 +177,7 @@ Apply this preference when following [[digest/procedure/technical-documentation.
## Sources ## Sources
- [[daily/2026-06-20/session-a.md]] This preference was recorded in [[daily/2026-06-20/documentation-style.md]], which captures the user's repeated guidance.
``` ```
This has three benefits: This has three benefits:
@ -174,8 +186,8 @@ This has three benefits:
2. The body can carry fuller facts, conditions, counterexamples, and sources. 2. The body can carry fuller facts, conditions, counterexamples, and sources.
3. Ordinary wikilinks can be parsed by the graph and maintained when files move. 3. Ordinary wikilinks can be parsed by the graph and maintained when files move.
Frontmatter is best for stable, short, structured fields; the body is best for explanations meant for people. Do not put long Frontmatter is best for stable, short, structured fields; the body is best for explanations meant for people. Do not put
body text into YAML fields. long body text into YAML fields.
### Wikilink ### Wikilink
@ -197,13 +209,13 @@ ReMe wikilinks use **literal path semantics**:
ReMe does not append `.md` automatically, search by filename, or automatically resolve folder notes. Use complete ReMe does not append `.md` automatically, search by filename, or automatically resolve folder notes. Use complete
workspace-relative paths with their extensions. workspace-relative paths with their extensions.
Ordinary Markdown links such as `[label](../wiki/example.md)` do not create `FileLink` edges and are not rewritten by move or Ordinary Markdown links such as `[label](../wiki/example.md)` do not create `FileLink` edges and are not rewritten by
retarget operations. move or retarget operations.
Anchors such as `#L9`, `#L9-L10`, and `#L9-L10,L15-L20` remain ordinary `target_anchor` strings in the graph. The graph Anchors such as `#L9`, `#L9-L10`, and `#L9-L10,L15-L20` remain ordinary `target_anchor` strings in the graph. The graph
parser does not validate line-anchor syntax, so values such as `#L0`, `#L10-L9`, and `#L9,` are also stored. The `read` job parser does not validate line-anchor syntax, so values such as `#L0`, `#L10-L9`, and `#L9,` are also stored. The `read`
does not interpret an anchor appended to `path`; use the separate 1-based, inclusive `start_line` and `end_line` arguments to job does not interpret an anchor appended to `path`; use the separate 1-based, inclusive `start_line` and `end_line`
read a range, for example `read(path="digest/wiki/solar.md", start_line=9, end_line=10)`. arguments to read a range, for example `read(path="digest/wiki/solar.md", start_line=9, end_line=10)`.
Wikilinks support these behaviors: Wikilinks support these behaviors:
@ -224,10 +236,10 @@ FileLink
``` ```
Older documents containing wrappers such as `related:: [[path]]`, Older documents containing wrappers such as `related:: [[path]]`,
`- related:: [[path]]`, or `[related:: [[path]]]` remain readable. ReMe `- related:: [[path]]`, or `[related:: [[path]]]` remain readable. ReMe ignores the surrounding text and indexes the
ignores the surrounding text and indexes the inner `[[path]]` as an ordinary inner `[[path]]` as an ordinary link. Graph changes are applied when source files pass through the normal ingestion
link. After upgrading from a version that stored typed links, run `reme reindex` path. `reme reindex` only rebuilds BM25 and embedding indexes from existing chunks; it does not reparse files or rebuild
once to rebuild the derived graph without the removed relationship field. the derived graph.
### Sources and Relationships ### Sources and Relationships
@ -238,8 +250,8 @@ A Sources section records where a long-term memory came from:
```markdown ```markdown
## Sources ## Sources
- [[daily/2026-06-20/session-a.md]] The preference was observed in [[daily/2026-06-20/documentation-style.md]], and the supporting report evidence is retained in
- [[resource/2026-06-20/report.pdf]] [[resource/2026-06-20/report.pdf]].
``` ```
A conceptual relationship link explains which other long-term memories relate to the node. Weave it into natural prose: A conceptual relationship link explains which other long-term memories relate to the node. Weave it into natural prose:
@ -255,16 +267,16 @@ This analysis extends [[digest/wiki/solar-supply-chain.md]], follows
Because memory is stored as files, users can edit the workspace directly, while agents can read and write the same files Because memory is stored as files, users can edit the workspace directly, while agents can read and write the same files
through ReMe's file tools. Both follow the same conventions: through ReMe's file tools. Both follow the same conventions:
| Operation | Guidance | | Operation | Guidance |
|---|---| |---------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
| Add memory | Write to the appropriate directory, use frontmatter for Markdown, and prefer complete workspace-relative wikilinks. | | Add memory | Write to the appropriate directory, use frontmatter for Markdown, and prefer complete workspace-relative wikilinks. |
| Edit a body | Preserve existing sources and important wikilinks. When correcting an old conclusion, explain how the new material changes the previous judgment. | | Edit a body | Preserve existing sources and important wikilinks. When correcting an old conclusion, explain how the new material changes the previous judgment. |
| Move a file | ReMe's move tool rewrites old paths in inbound edges by default. After a manual move, inspect inbound links again. | | Move a file | ReMe's move tool rewrites old paths in inbound edges by default. After a manual move, inspect inbound links again. |
| Delete a file | Check inbound links first. ReMe's delete tool returns source files that still point to the target, making dangling references easier to clean up. | | Delete a file | Check inbound links first. ReMe's delete tool returns source files that still point to the target, making dangling references easier to clean up. |
| Edit metadata | Use frontmatter for short fields. When the body changes substantially, update `description` as well. | | Edit metadata | Use frontmatter for short fields. When the body changes substantially, update `description` as well. |
A practical rule is: **an agent may rewrite the wording, but it must not lose evidence edges**. In particular, Sources entries A practical rule is: **an agent may rewrite the wording, but it must not lose evidence edges**. In particular, Sources
and existing digest-to-digest wikilinks are the basis for traceable and extensible long-term memory. entries and existing digest-to-digest wikilinks are the basis for traceable and extensible long-term memory.
## Path Semantics ## Path Semantics
@ -272,7 +284,7 @@ All file tools and wikilinks use workspace-relative paths as their basic unit:
```text ```text
digest/wiki/solar.md digest/wiki/solar.md
daily/2026-06-20/session-a.md daily/2026-06-20/documentation-style.md
resource/2026-06-20/report.pdf resource/2026-06-20/report.pdf
``` ```
@ -284,16 +296,16 @@ Recommended practices:
1. Include `.md` when linking a Markdown file. 1. Include `.md` when linking a Markdown file.
2. Use the complete source path when linking from digest to daily or resource. 2. Use the complete source path when linking from digest to daily or resource.
3. Rename or move files through ReMe's move tool whenever possible to avoid stale paths. 3. Rename or move files through ReMe's move tool whenever possible to avoid stale paths.
4. Put external source material under `resource/YYYY-MM-DD/...` and long-term abstractions under `digest/...`. Do not put 4. Put external source material under `resource/YYYY-MM-DD/...` and long-term abstractions under `digest/...`. Do not
raw source material directly into digest. put raw source material directly into digest.
Explicit path semantics sacrifice a little convenience when writing by hand, but provide predictability, portability, and Explicit path semantics sacrifice a little convenience when writing by hand, but provide predictability, portability,
automatic maintainability. and automatic maintainability.
## Memory Chunking ## Memory Chunking
Memory chunking divides a file into retrievable fragments. ReMe does not split Markdown at fixed lengths by default; it tries Memory chunking divides a file into retrievable fragments. ReMe does not split Markdown at fixed lengths by default; it
to preserve semantic structure. tries to preserve semantic structure.
This section explains how files become retrieval chunks. For index updates, BM25, vector recall, and link expansion, see This section explains how files become retrieval chunks. For index updates, BM25, vector recall, and link expansion, see
[Memory Search](./memory_search.md). [Memory Search](./memory_search.md).
@ -308,8 +320,8 @@ Document
chunk 1 | chunk 2 | chunk 3 | ... chunk 1 | chunk 2 | chunk 3 | ...
``` ```
This is simple, but it can cut headings, tables, code blocks, lists, and `[[wikilinks]]` in the middle. After a match, the This is simple, but it can cut headings, tables, code blocks, lists, and `[[wikilinks]]` in the middle. After a match,
agent often sees only an isolated fragment without knowing its section or relationship to other memory nodes. the agent often sees only an isolated fragment without knowing its section or relationship to other memory nodes.
ReMe chunking is closer to splitting memory by file structure: ReMe chunking is closer to splitting memory by file structure:
@ -375,5 +387,10 @@ Matched body fragment
This lets the agent see not only an isolated paragraph but also its structural position in the source file. This lets the agent see not only an isolated paragraph but also its structural position in the source file.
Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For Markdown, Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For
the chunker also avoids cutting `[[wikilinks]]` in the middle. Markdown, the chunker also avoids cutting `[[wikilinks]]` in the middle.
`DefaultFileChunker` and `MarkdownFileChunker` decode files with their configured `encoding` and normalize platform
newlines to LF before indexing. Their default `invalid_encoding_policy: replace` keeps decodable content searchable
when a source contains invalid bytes, without modifying the source file. Set `invalid_encoding_policy: strict` on a
chunker component to reject such files instead.

View file

@ -1,8 +1,10 @@
# Memory Search # Memory Search
Memory Search is ReMe's memory retrieval entry point. It continuously builds files under `daily/`, `digest/`, and `resource/` Memory Search is ReMe's memory retrieval entry point. The default background loop continuously builds Markdown under
into a searchable chunk index and wikilink graph. At query time, it first recalls the most relevant fragments and then expands `daily/` and `digest/` into a searchable chunk index and wikilink graph. At query time, it first recalls the most
context along the bidirectional links of the files containing those fragments. relevant fragments and then expands context along the bidirectional links of the files containing those fragments.
`reme reindex` rebuilds derived BM25 and embedding indexes from the authoritative in-memory `file_chunks`; it does not
rescan workspace files, rechunk content, or rewrite the wikilink graph.
<p align="center"> <p align="center">
<img src="../figure/auto-index-and-memory-search.svg" alt="ReMe Auto Index and Memory Search indexing, recall, fusion, and link expansion" width="92%"> <img src="../figure/auto-index-and-memory-search.svg" alt="ReMe Auto Index and Memory Search indexing, recall, fusion, and link expansion" width="92%">
@ -21,14 +23,14 @@ workspace files
## What It Searches ## What It Searches
The default `index_update_loop` watches three memory directories: The default `index_update_loop` watches two memory directories:
- `daily_dir`: daily working memory and session memory cards generated by Auto Memory. - `daily_dir`: daily working memory and session memory cards generated by Auto Memory.
- `digest_dir`: long-term distilled digest nodes. - `digest_dir`: long-term distilled digest nodes.
- `resource_dir`: external resources or imported material.
The default suffixes are `md` and `jsonl`. Markdown uses the `markdown` chunker, which parses frontmatter, heading structure, The live watcher handles only the `md` suffix. A separate `resource_watch_loop` watches `resource_dir`, and Auto
and `[[wikilinks]]`. JSONL uses the `default` chunker and creates overlapping chunks by byte size. Resource turns those inputs into daily cards that enter the live index. Manual `reindex` operates on chunks already
accepted by those ingestion paths and therefore does not expand the set of searched files.
## How the Index Is Built ## How the Index Is Built
@ -39,8 +41,8 @@ The background Job `index_update_loop` maintains the index using configuration f
```yaml ```yaml
index_update_loop: index_update_loop:
backend: background backend: background
watch_dirs: [ daily_dir, digest_dir, resource_dir ] watch_dirs: [daily_dir, digest_dir]
watch_suffixes: [ md, jsonl ] watch_suffixes: [md]
steps: steps:
- backend: init_changes_step - backend: init_changes_step
monitor_type: file_store monitor_type: file_store
@ -54,9 +56,9 @@ index_update_loop:
`FileNode.st_mtime` values already stored in `file_store`, calculates added, modified, and deleted changes, and passes `FileNode.st_mtime` values already stored in `file_store`, calculates added, modified, and deleted changes, and passes
`context["changes"]` to `update_index_step`. `context["changes"]` to `update_index_step`.
While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same directories, While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same
groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the same path into one directories, groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the
stable batch of changes. same path into one stable batch of changes.
`update_index_step` performs the actual index writes: `update_index_step` performs the actual index writes:
@ -66,13 +68,14 @@ stable batch of changes.
4. For a deleted file, remove its records from `file_store`, `keyword_index`, and `file_graph`. 4. For a deleted file, remove its records from `file_store`, `keyword_index`, and `file_graph`.
5. When changes exist, dump state to `metadata/` so it can be restored on the next startup. 5. When changes exist, dump state to `metadata/` so it can be restored on the next startup.
The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into `FileNode`, `FileChunk`, and `FileLink` The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into `FileNode`, `FileChunk`, and
`FileLink`
objects. For detailed chunking rules, see [Memory as File](./memory_as_file.md#memory-chunking). objects. For detailed chunking rules, see [Memory as File](./memory_as_file.md#memory-chunking).
### Index Optimization ### Index Optimization
Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes; Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes; too
too many tombstones degrade search performance. An idle-time optimization mechanism is built in—the `optimize_index_cron` many tombstones degrade search performance. An idle-time optimization mechanism is built in—the `optimize_index_cron`
scheduled job compacts tombstones and rebuilds indexes during off-peak hours: scheduled job compacts tombstones and rebuilds indexes during off-peak hours:
```yaml ```yaml
@ -100,16 +103,37 @@ file_store:
It combines three kinds of capability: It combines three kinds of capability:
| Part | Default state | Purpose | | Part | Default state | Purpose |
|---|---|---| |-------------------------|---------------|-------------------------------------------------------------------------|
| `file_chunks` | Enabled | Store `FileChunk` text, line numbers, scores, and optional embeddings. | | `file_chunks` | Enabled | Store `FileChunk` text, line numbers, scores, and optional embeddings. |
| `keyword_index.default` | Enabled | BM25 inverted index where chunk ID is the document ID. | | `keyword_index.default` | Enabled | BM25 inverted index where chunk ID is the document ID. |
| `file_graph.default` | Enabled | Store `FileNode` objects and wikilink edges. | | `file_graph.default` | Enabled | Store `FileNode` objects and wikilink edges. |
| `embedding_store` | Disabled | When enabled, generate embeddings for chunks and support vector recall. | | `embedding_store` | Disabled | When enabled, generate embeddings for chunks and support vector recall. |
Out of the box, search therefore uses primarily BM25 plus link expansion. After setting `embedding_store: default`, Out of the box, search therefore uses primarily BM25 plus link expansion. After setting `embedding_store: default`,
`SearchStep` runs vector and keyword recall together. Additionally, switching the `file_store` `backend` from `local` to `SearchStep` runs vector and keyword recall together.
`faiss` upgrades vector retrieval from a linear scan to a FAISS HNSW index, offering faster recall at scale.
The embedding store accepts `health_check_timeout` for its startup probe. A temporary failure skips the current vector
backfill while keeping BM25 available; a later successful provider request resumes the missing-vector backfill
automatically.
Embedded integrations that have already verified a provider can call `resume_embedding(verified=True)` to repair
missing vectors in the same vector space. Vector-space changes must use the explicit `reindex` job with
`scope: embedding`; vector search remains unavailable until that job finishes successfully.
Use `scope: bm25` to rebuild only keyword search, or `scope: tag` to rebuild the optional tag index from the current
file graph. `scope: all` rebuilds BM25 first, then embeddings, and finally tags. BM25 and embedding rebuilds use the
current `file_chunks` snapshot; the tag rebuild uses `FileNode` frontmatter from the file graph.
## Vector Index Backends
With embeddings enabled, `file_store.default.backend` can be `local`, `zvec`, or `faiss`. `local` scans vectors linearly;
`zvec` uses an in-process HNSW index with native vector updates and deletes; `faiss` uses a FAISS HNSW index. The latter
two store rebuildable vector indexes. Memory files and ReMe's chunk data remain the source of truth.
To use Zvec or FAISS, first configure `as_embedding` and `embedding_store` as shown in
[Configuration](./configuration.md#embeddings). Then set `file_store.default.embedding_store` to `default` and
`file_store.default.backend` to `zvec` or `faiss`. The `core` installation includes both dependencies; the default
configuration still uses `local` with embeddings disabled.
## How to Search ## How to Search
@ -123,10 +147,12 @@ search:
query: string query: string
limit: integer limit: integer
min_score: number min_score: number
start_date: string
end_date: string
steps: steps:
- backend: search_step - backend: search_step
vector_weight: 0.7 vector_weight: 0.7
candidate_multiplier: 3.0 candidate_multiplier: 5.0
expand_links: true expand_links: true
max_links_per_direction: 10 max_links_per_direction: 10
``` ```
@ -137,11 +163,17 @@ Call it with:
reme search query="recent discussions about indexing" limit=5 reme search query="recent discussions about indexing" limit=5
``` ```
Use `start_date` and `end_date` for inclusive `YYYY-MM-DD` filtering:
```bash
reme search query="index regression" start_date=2026-06-01 end_date=2026-06-20 limit=10
```
`search_step` executes in this order: `search_step` executes in this order:
```mermaid ```mermaid
flowchart LR flowchart LR
A["query + limit"] --> B["candidates = limit * candidate_multiplier"] A["query + limit"] --> B["candidates = min(200, limit * candidate_multiplier)"]
B --> C["file_store.vector_search(...)"] B --> C["file_store.vector_search(...)"]
B --> D["file_store.keyword_search(...)"] B --> D["file_store.keyword_search(...)"]
C --> E["RRF fusion"] C --> E["RRF fusion"]
@ -152,17 +184,17 @@ flowchart LR
H --> I["Response.answer + metadata"] H --> I["Response.answer + metadata"]
``` ```
If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking is If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking
returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores directly; it is returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores
compares ranks in the two result lists: directly; it compares ranks in the two result lists:
```text ```text
fused_score = vector_weight / (60 + vector_rank) fused_score = vector_weight / (60 + vector_rank)
+ keyword_weight / (60 + keyword_rank) + keyword_weight / (60 + keyword_rank)
``` ```
The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can still The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can
promote chunks with exact term matches. still promote chunks with exact term matches.
## How BM25 Works ## How BM25 Works
@ -174,12 +206,14 @@ promote chunks with exact term matches.
- The inverted index records which chunks contain each token and its term frequency within each chunk. - The inverted index records which chunks contain each token and its term frequency within each chunk.
- A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs. - A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs.
When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old `chunk_ids` When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old
`chunk_ids`
and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize. and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize.
## Progressive Expansion ## Progressive Expansion
"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three layers: "Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three
layers:
1. Chunk recall: return only the `limit` most relevant text fragments. 1. Chunk recall: return only the `limit` most relevant text fragments.
2. File location: each result includes `path:start_line-end_line`. Pass the path and line bounds separately as `path`, 2. File location: each result includes `path:start_line-end_line`. Pass the path and line bounds separately as `path`,
@ -198,8 +232,8 @@ matched chunk
-> render neighbor path, name, description, and anchor -> render neighbor path, name, description, and anchor
``` ```
This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory connects This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory
to. If a result is worth pursuing, use `read path=...` to open the source or connects to. If a result is worth pursuing, use `read path=...` to open the source or
`traverse path=... depth=2` to continue along the wikilink graph. `traverse path=... depth=2` to continue along the wikilink graph.
## Return Format ## Return Format
@ -213,7 +247,7 @@ to. If a result is worth pursuing, use `read path=...` to open the source or
Typical text structure: Typical text structure:
```text ```text
========== daily/2026-06-20/session-a.md:12-28 [score=0.0317 keyword=4.8120] ========== ========== daily/2026-06-20/retrieval-regression.md:12-28 [score=0.0317 keyword=4.8120] ==========
...matched memory fragment... ...matched memory fragment...
outlinks (2): outlinks (2):
-> digest/indexing.md name="Indexing" description="..." -> digest/indexing.md name="Indexing" description="..."
@ -221,5 +255,5 @@ Typical text structure:
<- daily/2026-06-19.md name="..." <- daily/2026-06-19.md name="..."
``` ```
`counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned. With `counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned.
embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`. With embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`.

102
docs/en/operations.md Normal file
View file

@ -0,0 +1,102 @@
---
title: Diagnostics, Backup, and Recovery
description: ReMe health checks, logs, index maintenance, workspace backup, migration, and recovery.
---
# Diagnostics, Backup, and Recovery
ReMe recovery protects user-owned workspace files and rebuilds catalogs, indexes, and graphs from those sources. Never delete or rewrite user memory merely to repair derived state.
## Quick diagnosis
Run these in order:
```bash
reme find_reme
reme version
reme health_check
reme status
reme app_config
```
- `find_reme` confirms the actual host, port, and PID;
- `version` verifies that the CLI reaches the service;
- `health_check` reports component health;
- `status` estimates stateful component memory and process RSS;
- `app_config` returns the effective configuration with secrets redacted.
## Logs and common symptoms
`log_to_console` and `log_to_file` control logging. For startup failures, inspect the first exception rather than later client connection errors.
| Symptom | Check first |
|---|---|
| CLI cannot find ReMe | `reme find_reme`, process state, startup directory, and port |
| Automatic memory fails | LLM backend, model, API key, and base URL |
| Search is BM25-only | Whether an embedding store is connected to `file_store` |
| New files are absent | Directory, extension, watcher, and `health_check` |
| Studio fails but API works | Installed web extra, static path, and browser console |
| Installed plugin is unavailable | Python interpreter, `plugins` configuration, and service restart |
## Index maintenance
```bash
reme reindex scope=all
reme reindex scope=bm25
reme reindex scope=embedding
```
`reindex` rebuilds BM25 and/or embedding indexes from the current `file_chunks`. It does not scan the workspace, rechunk files, or rebuild the wikilink graph. Diagnose the watcher first when ingestion is the problem.
Rebuild a daily index page separately:
```bash
reme daily_reindex date=2026-09-04
```
## Backup
Stop writes or stop the service, then back up the complete workspace. The most important sources are:
- `session/` for conversation sources;
- `resource/` for external resources;
- `daily/` for daily memory;
- `digest/` for consolidated memory.
`metadata/` contains indexes, graphs, and catalogs. Backing it up accelerates restoration, but it is not the sole source of truth.
Use an explicit, stable absolute `workspace_dir` for durable deployments rather than relying on an incidental `.reme/` under the current directory.
## Migrate a workspace
1. Stop the old service to prevent writes during the copy.
2. Copy the complete workspace while preserving timestamps.
3. Start with the new absolute path:
```bash
reme start workspace_dir=/new/location/reme-memory
```
4. Run `health_check`, `status`, and a representative `search`.
5. Rebuild embeddings if their model or dimensions changed.
Do not push a workspace containing private conversations to a public repository.
## Recover derived state
Do not remove anything until a backup exists. Then:
1. preserve `session/`, `resource/`, `daily/`, and `digest/`;
2. record the effective configuration and component backends;
3. verify that the failure is limited to `metadata/`;
4. move suspect derived state to an isolated backup location;
5. restart with the same configuration and let watchers rebuild;
6. validate search, graph traversal, and daily indexes.
The internal layout of metadata files is not a public automation contract.
## Concurrent editing
When Studio or an editor saves a complete file, pass the mtime from `stat` as `save.expected_mtime`. A save then fails if another actor changed the file after it was opened, avoiding silent overwrites.
File Jobs enforce workspace containment and per-path locking. Do not bypass them to write arbitrary absolute paths.

View file

@ -0,0 +1,101 @@
---
title: Plugin Development
description: Create, register, configure, test, and publish a ReMe plugin.
---
# Plugin Development
A ReMe plugin is a regular Python distribution exposed through the `reme.plugins` entry-point group. Its package-level `plugin.yaml` can register Step and Component backends and provide default Application configuration.
## Minimal structure
```text
my-plugin/
├── pyproject.toml
└── src/my_plugin/
├── __init__.py
├── plugin.yaml
└── steps.py
```
`pyproject.toml`:
```toml
[project.entry-points."reme.plugins"]
my-plugin = "my_plugin"
```
`plugin.yaml`:
```yaml
name: my-plugin
backends:
my_step: my_plugin.steps:MyStep
application_defaults:
jobs:
my_action:
backend: base
description: Run my plugin action
parameters:
type: object
properties:
text: { type: string }
required: [text]
steps:
- backend: my_step
```
## Implement a Step
```python
from reme.components.component_registry import R
from reme.steps.base_step import BaseStep
@R.register("my_step")
class MyStep(BaseStep):
async def execute(self):
self.context.response.answer = self.context.data["text"]
```
Step instances belong to one Job invocation. Put shared in-memory state under a narrow `app_context.metadata` key. Promote state that needs lifecycle, locking, or persistence to a Component or workspace file.
## Configuration merge
`application_defaults` is a partial `ApplicationConfig`:
```text
plugin defaults < selected/default config < CLI overrides
```
Plugins must not rewrite user configuration. Their backends enter an Application-local registry only when the plugin appears in that Application's `plugins` list.
## Local validation
```bash
reme plugins validate ./path/to/my-plugin
reme plugins install ./path/to/my-plugin --editable
reme plugins list
reme plugins show my-plugin
reme start plugins='["my-plugin"]'
reme my_action text=hello
```
Validation imports plugin code, so run it only for trusted sources.
## Test boundaries
- create workspaces with `tmp_path`;
- mock network, model, and subprocess boundaries;
- verify disabled plugins do not mutate the built-in registry;
- verify plugin defaults and explicit configuration precedence;
- keep tasks, clients, and executors under Component lifecycle;
- never delete or rewrite user source files to repair derived state.
The repository's Daily Paper, Auto Fin, LME, and BEAM plugins are complete examples.
## Compatibility
Legacy Python Plugin descriptors and the `reme.configs` entry point remain supported during migration, but new plugins should use `plugin.yaml`. Enablement always belongs to an Application rather than a process-global switch.
See [Plugin Management](./plugin_management.md) for installation, upgrades, and removal.

View file

@ -0,0 +1,236 @@
# Plugin Management
ReMe plugins are ordinary Python distributions discovered through the `reme.plugins` entry-point group. Installing a
plugin makes it available to the current Python environment; it does not enable the plugin in every ReMe application.
Keep these two operations separate:
```text
reme plugins install ... install a package into the current Python environment
plugins: [auto-fin] enable an installed plugin for one Application
```
Plugin package management is local-only. It does not run through a ReMe HTTP or MCP service and never edits application
configuration files automatically.
A typical plugin workflow has three stages:
1. Install ReMe and the plugin distribution.
2. Configure the plugin's runtime environment as described in the
[ReMe model-configuration guide](../../README.md#optional-model-configuration).
3. Start an Application with the plugin explicitly enabled, for example
`reme start plugins='["auto-fin"]'`.
## List installed plugins
```bash
reme plugins list
```
The table shows the plugin entry-point name, Python distribution, version, and plugin contract:
```text
PLUGIN DISTRIBUTION VERSION FORMAT
-------- ------------- ------- --------
auto-fin reme-auto-fin X.Y.Z manifest
```
`manifest` plugins use the current package-level `plugin.yaml` contract. `legacy` plugins use the compatible Python
descriptor contract.
A manifest separates backend registration from application configuration:
```yaml
backends:
example_step: example_plugin.steps:ExampleStep
application_defaults:
jobs:
example:
backend: base
steps:
- backend: example_step
```
`application_defaults` is a partial `ApplicationConfig`. It is kept below the manifest's `backends` namespace because
backend import declarations are part of plugin discovery and are not application configuration.
Use JSON when another local tool needs structured output:
```bash
reme plugins list --json
```
To compare installed plugins with one application config:
```bash
reme plugins list --config default
```
The optional `ENABLED` column reflects only the `plugins` list resolved from that config. A command-line override used
by another running process is not a global enable state.
## Install a plugin package
Install a published distribution by its package name:
```bash
reme plugins install your-plugin-package
```
Install or upgrade a pinned version:
```bash
reme plugins install 'your-plugin-package==X.Y.Z'
reme plugins install your-plugin-package --upgrade
```
Install a local plugin project:
```bash
reme plugins install ./plugins/auto-fin
```
Use editable mode while developing it:
```bash
reme plugins install ./plugins/auto-fin --editable
```
ReMe invokes pip through the same Python interpreter that runs the `reme` command. Pip remains responsible for package
resolution, downloads, dependency changes, and build execution. Install only packages and local projects you trust.
After installation, confirm the discovered plugin name:
```bash
reme plugins list
reme plugins validate auto-fin
```
## Inspect a plugin
```bash
reme plugins show auto-fin
```
For a manifest plugin, the result includes its registered backend names and default Job names. JSON output is also
available:
```bash
reme plugins show auto-fin --json
```
`show` identifies the package contract without constructing a ReMe Application.
## Validate a plugin
Validate an installed plugin:
```bash
reme plugins validate auto-fin
```
Validate a local project before installation:
```bash
reme plugins validate ./plugins/auto-fin
```
Validation checks the entry point, `plugin.yaml`, backend imports and component types, registry collisions, merged
`application_defaults`, and the resulting `ApplicationConfig`. Validation imports plugin backend modules, so run it
only for trusted code.
## Enable a plugin in a service
Installation alone does not load plugin code into an Application. Enable plugins explicitly in configuration:
```yaml
plugins:
- auto-fin
```
Or add them for one service launch:
```bash
reme start plugins='["auto-fin"]'
```
When `config` is omitted, ReMe loads `default.yaml`. The plugin's `application_defaults` are merged below that config,
so explicit config values and CLI overrides win. This mapping is an `ApplicationConfig` fragment, not a separate
configuration schema. The plugin backends are registered only in that Application's local registry.
After the default HTTP service starts, access plugin Jobs through ReMe's CLI client or HTTP:
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
When the application uses an MCP service, service-enabled plugin Jobs appear as MCP tools instead.
Custom application configs must provide the plugin's runtime dependencies, including an `agent_wrapper.default`, a
`file_store.default` with an enabled tag index, and the `search`, `read`, `list_tags`, `frontmatter_read`, and
`frontmatter_update` Jobs used by Auto Fin and automatic tagging.
## Benchmark application presets
The [LME](../../plugins/lme/README.md) and [BEAM](../../plugins/beam/README.md) plugins
register their backends and plugin-owned Jobs in `plugin.yaml`. ReMe's built-in `benchmark`
preset provides the shared core Jobs and components without inheriting `default`, so default
background and cron jobs are not included. Install the selected benchmark plugin, then use
`config=benchmark` together with `plugins=["lme"]` or `plugins=["beam"]`. The repository's
benchmark runners enable the corresponding installed plugin automatically; editable installation
keeps local plugin changes visible. Dataset runners remain under `benchmark/`.
## Uninstall a plugin
Use the plugin entry-point name, not necessarily the distribution name:
```bash
reme plugins uninstall auto-fin
```
Skip pip's confirmation prompt when needed:
```bash
reme plugins uninstall auto-fin --yes
```
ReMe resolves `auto-fin` to the distribution that provides it, such as `reme-auto-fin`. If one distribution provides
multiple plugin entry points, the command lists the other plugins that will also be removed.
Uninstallation does not rewrite user configuration. Remove the plugin from relevant `plugins` lists yourself;
otherwise the next Application startup fails explicitly because the configured plugin is no longer installed. Restart
already-running ReMe processes after installing, upgrading, or uninstalling packages.
## Troubleshooting
### Plugin is installed but unavailable
Check that the `reme` command and pip package share one Python interpreter:
```bash
reme plugins list
python -c 'import sys; print(sys.executable)'
```
Using `reme plugins install` avoids the most common interpreter mismatch because it runs `python -m pip` with ReMe's
own interpreter.
### Plugin is installed but not loaded
Add its entry-point name to the Application's `plugins` list. ReMe intentionally has no global enable/disable state.
### Startup reports that the plugin is not installed
The active config still enables a missing plugin. Reinstall it or remove the corresponding name from `plugins`.
### Changes are not visible in a running service
Plugin discovery and backend registration happen during Application construction. Restart the service after changing
installed packages.

View file

@ -1,65 +1,140 @@
# Proactive # Proactive
`proactive` is ReMe's interface for reading proactive memory. It does not reanalyze daily notes or call an LLM. It only reads `proactive_read` is ReMe's interface for reading proactive memory. It does not reanalyze daily notes or call an LLM. It
the current day's interest topics written by `auto_dream`: reads interest topics written by the independent proactive refresh flow:
```text ```text
daily/<date>/interests.yaml daily/<date>/interests.yaml
``` ```
A host agent can use it to learn "what is worth proactive attention today," then decide whether to remind the user, ask a A host agent can use it to learn "what is worth proactive attention today," then decide whether to remind the user, ask
follow-up question, recommend a next step, or produce a proactive insight. a follow-up question, recommend a next step, or produce a proactive insight.
`interests.yaml` is generated by the Topics stage of [Auto Dream](./auto_dream.md). `proactive` only reads and exposes the `interests.yaml` is generated by the proactive refresh pipeline, scheduled by `proactive_refresh_cron` by default.
result. `proactive_read` only reads and exposes the result; Auto Dream is an independent daily-to-digest flow and does not read
or write proactive state.
## Configuration ## Configuration
The default configuration is in `reme/config/default.yaml`: The default configuration is in `reme/config/default.yaml`. It defines the same refresh steps twice: a local-only
one-shot job for maintenance and debugging, and the scheduled job that runs every day at 18:00 in the application
timezone:
```yaml ```yaml
proactive: proactive_refresh:
backend: base
enable_serve: false
steps: &proactive_refresh_steps
- backend: proactive_extract_step
file_catalog: proactive
scan_days: 2
carry_forward_days: 14
max_carry_forward_topics: 20
llm_timeout_seconds: 300
max_chars_per_file: 60000
max_total_chars: 300000
- backend: proactive_topics_step
known_threshold: 0.85
known_threshold_calibrated_for: text-embedding-v4@1024
min_push_confidence: 0.5
max_topics: 10
- backend: proactive_plan_step
- backend: proactive_agenda_step
- backend: proactive_finish_step
file_catalog: proactive
proactive_refresh_cron:
backend: cron
cron: "0 18 * * *"
steps: *proactive_refresh_steps
```
The anchor above is only a compact illustration; `default.yaml` spells out both step lists explicitly. The read job is:
```yaml
proactive_read:
backend: base backend: base
description: "Proactive: read daily/<date>/interests.yaml and expose the latest user-interest topics." description: "Proactive: read daily/<date>/interests.yaml and expose the latest user-interest topics."
parameters: parameters:
date: type: object
type: string properties:
default: "" date:
include_content: type: string
type: boolean default: ""
default: true include_content:
type: boolean
default: true
horizon_days:
type: integer
default: 1
min_confidence:
type: number
default: 0.4
steps: steps:
- backend: proactive_step - backend: proactive_step
min_confidence: 0.4
``` ```
Parameters: Parameters:
| Parameter | Purpose | | Parameter | Purpose |
|---|---| |-------------------|------------------------------------------------------------------------------------------------------|
| `date` | Date to read in `YYYY-MM-DD` format. When empty, use today in the application's timezone. | | `date` | Date to read in `YYYY-MM-DD` format. When empty, use today in the application's timezone. |
| `include_content` | Whether to return the raw YAML in the answer and metadata. Defaults to `true`. | | `include_content` | Whether to return the raw YAML in the answer and metadata. Defaults to `true`. |
| `horizon_days` | Read one day's exposure file, or use the truth source for a wider evidence horizon. Defaults to `1`. |
| `min_confidence` | Minimum topic confidence to return. Defaults to `0.4`; legacy topics use `0.5`. |
### Refresh cost, files, and opt-out
When no daily Markdown note changed, refresh exits before calling an LLM and does not create a new exposure file. With
changed material, extraction normally makes one LLM call and may retry once after an unusable reply. If push candidates
remain, planning makes one additional call; agenda generation makes one more when there are multiple candidates. A
refresh therefore makes at most four LLM calls with the default chain.
The refresh pipeline maintains the rebuildable `daily/_proactive.yaml` truth source, writes
`daily/<date>/interests.yaml`, and advances the independent `proactive` file catalog. Auto Dream does not read or write
any of those proactive artifacts.
To disable automatic refresh, use an explicit application config that omits the `proactive_refresh_cron` job. Keep the
local-only `proactive_refresh` job if you still want on-demand maintenance. Because it has `enable_serve: false`, it is
not exposed through HTTP or MCP.
## Input Contract ## Input Contract
A typical file looks like this: A current proactive-refresh file looks like this:
```yaml ```yaml
version: 2
date: 2026-06-20 date: 2026-06-20
topic_count: 3 generated_at: 2026-06-20T18:00:00+08:00
diversity_days: 7 push: true
topics: topics:
- title: Quality regression in the memory retrieval pipeline - id: baa88ad49cb2
title: Quality regression in the memory retrieval pipeline
reason: The user has recently made repeated changes to search, node_search, and dream integration. reason: The user has recently made repeated changes to search, node_search, and dream integration.
kind: follow_up
confidence: 0.86
first_seen: 2026-06-20
last_evidence_at: 2026-06-20
evidence: daily/2026-06-20/session.md evidence: daily/2026-06-20/session.md
keywords:
- memory search
- auto dream
paths: paths:
- daily/2026-06-20/session.md - daily/2026-06-20/session.md
agenda:
- topic_id: baa88ad49cb2
title: Quality regression in the memory retrieval pipeline
scenario_type: resume_task
opener: Review the latest retrieval regression before the next release.
next_action: Compare the failing query against the previous index snapshot.
preconditions: []
delivery: in_conversation
linked_memory: []
order_reason: Recent evidence and a concrete next action.
suppressed: []
``` ```
Only the `topics` list is parsed into structured results. Every topic requires at least `title` and `reason`; Current v2 topics include stable identity, kind, confidence, evidence dates, and source paths. The reader also accepts
`evidence`, `keywords`, and `paths` are supporting fields. legacy v1 files containing `title`, `reason`, `evidence`, `keywords`, and `paths`; missing v2 confidence falls back to
`0.5`.
## Return Value ## Return Value
@ -67,15 +142,23 @@ When the file is read successfully, `proactive_step` returns `summary` and `topi
`include_content=true`, the answer also contains `content`. The same result fields remain available in standard response `include_content=true`, the answer also contains `content`. The same result fields remain available in standard response
metadata: metadata:
| Field | Description | | Field | Description |
|---|---| |-----------|------------------------------------------------------|
| `date` | The date actually read. | | `date` | The date actually read. |
| `path` | `daily/<date>/interests.yaml`. | | `path` | `daily/<date>/interests.yaml`. |
| `topics` | Parsed topic list. | | `topics` | Parsed topic list. |
| `content` | Raw YAML; returned only when `include_content=true`. | | `content` | Raw YAML; returned only when `include_content=true`. |
| `skipped` | `true` when the file does not exist. | | `skipped` | `true` when the file does not exist. |
| `error` | Read or parse error. | | `error` | Read or parse error. |
| `summary` | Short summary. | | `summary` | Short summary. |
| `agenda` | Today's proactive agenda (optional, v2 files only). |
When today's `interests.yaml` was produced by the proactive refresh chain with an agenda,
the answer also carries an `agenda` field: the ordered agenda items, each with `topic_id`,
`title`, `scenario_type`, `opener` (a natural conversation opener), `next_action` (the
minimal executable step), `preconditions`, `delivery`, `linked_memory` and `order_reason`.
Agenda items whose topic is resolved or below `min_confidence` are filtered out on read;
the field is absent when the file has no agenda.
When the file exists and parses successfully, the answer is structured data. For example: When the file exists and parses successfully, the answer is structured data. For example:
@ -84,13 +167,30 @@ When the file exists and parses successfully, the answer is structured data. For
"summary": "Read 1 proactive topic(s) from daily/2026-06-20/interests.yaml", "summary": "Read 1 proactive topic(s) from daily/2026-06-20/interests.yaml",
"topics": [ "topics": [
{ {
"id": "baa88ad49cb2",
"title": "Quality regression in the memory retrieval pipeline", "title": "Quality regression in the memory retrieval pipeline",
"reason": "The user has recently made repeated changes to search, node_search, and dream integration.", "reason": "The user has recently made repeated changes to search, node_search, and dream integration.",
"kind": "follow_up",
"confidence": 0.86,
"first_seen": "2026-06-20",
"last_evidence_at": "2026-06-20",
"evidence": "daily/2026-06-20/session.md", "evidence": "daily/2026-06-20/session.md",
"keywords": ["memory search", "auto dream"],
"paths": ["daily/2026-06-20/session.md"] "paths": ["daily/2026-06-20/session.md"]
} }
], ],
"agenda": [
{
"topic_id": "baa88ad49cb2",
"title": "Quality regression in the memory retrieval pipeline",
"scenario_type": "resume_task",
"opener": "Review the latest retrieval regression before the next release.",
"next_action": "Compare the failing query against the previous index snapshot.",
"preconditions": [],
"delivery": "in_conversation",
"linked_memory": [],
"order_reason": "Recent evidence and a concrete next action."
}
],
"content": "date: 2026-06-20\n..." "content": "date: 2026-06-20\n..."
} }
``` ```
@ -104,52 +204,57 @@ A missing file is not an error. The call succeeds with a skipped result:
Skipped: interests file not found at daily/2026-06-20/interests.yaml Skipped: interests file not found at daily/2026-06-20/interests.yaml
``` ```
This lets a host agent treat "there is no dream result for today yet" as a normal empty state. This lets a host agent treat "there is no proactive refresh result for today yet" as a normal empty state.
## Running Proactive ## Running Proactive
CLI: Run one refresh immediately through the normal application lifecycle:
```bash ```bash
reme proactive date=2026-06-20 reme start job=proactive_refresh date=2026-06-20
```
This command may call the configured LLM and may update `_proactive.yaml`, `interests.yaml`, and the proactive catalog.
It does not run Auto Dream.
Read the generated topics:
```bash
reme proactive_read date=2026-06-20
``` ```
Omit the raw YAML content: Omit the raw YAML content:
```bash ```bash
reme proactive date=2026-06-20 include_content=false reme proactive_read date=2026-06-20 include_content=false
``` ```
## Relationship to auto_dream ## Relationship to auto_dream
`proactive` is the downstream read step for `auto_dream`: Proactive refresh and Auto Dream consume daily notes independently:
```text ```text
daily notes daily notes -> auto_dream -> digest
-> auto_dream daily notes -> proactive_refresh_cron -> daily/<date>/interests.yaml -> proactive_read -> host agent
-> daily/<date>/interests.yaml
-> proactive
-> host agent
``` ```
The responsibilities are divided as follows. For the complete Extract, Integrate, Topics, and Finish flow, see The proactive responsibilities are divided as follows:
[Auto Dream](./auto_dream.md):
| Module | Responsibility | | Module | Responsibility |
|---|---| |--------------------------|--------------------------------------------------------|
| `dream_extract_step` | Extract topic candidates from changed daily inputs. | | `proactive_refresh` | Run the refresh pipeline once from the local CLI. |
| `dream_topics_step` | Deduplicate, select, and write `interests.yaml`. | | `proactive_refresh_cron` | Run the same writer pipeline every day at 18:00. |
| `proactive_step` | Read `interests.yaml` and expose it to the host agent. | | `proactive_step` | Read `interests.yaml` and expose it to the host agent. |
`proactive` does not modify files, update a catalog, or decide whether the user should be interrupted. It only provides the `proactive_read` does not modify files, update a catalog, or decide whether the user should be interrupted. It only provides
day's topic material. The caller's product policy determines whether, when, and in what tone to push it to the user. the day's topic material. The caller's product policy determines whether, when, and in what tone to push it to the user.
## Failure Modes ## Failure Modes
| Scenario | Behavior | | Scenario | Behavior |
|---|---| |-------------------------------------|--------------------------------------------------------|
| `interests.yaml` does not exist | `success=true`, `skipped=true`, `topics=[]`. | | `interests.yaml` does not exist | `success=true`, `skipped=true`, `topics=[]`. |
| YAML cannot be read or parsed | `success=false`; the answer contains an error summary. | | YAML cannot be read or parsed | `success=false`; the answer contains an error summary. |
| YAML exists but has no valid topics | `success=true`, `topics=[]`. | | YAML exists but has no valid topics | `success=true`, `topics=[]`. |
Callers should therefore check `success` first, then `skipped`, and finally whether `topics` is empty. Callers should therefore check `success` first, then `skipped`, and finally whether `topics` is empty.

View file

@ -1,5 +1,12 @@
---
title: Quick Start
description: Install and start ReMe, then complete a first file, retrieval, and automatic-memory workflow.
---
# Quick Start # Quick Start
This page gets one working loop running. See [Configuration](./configuration.md) for the full configuration contract and [Services and Deployment](./services.md) for HTTP or MCP integration.
## Installation ## Installation
ReMe requires Python 3.11+. ReMe requires Python 3.11+.
@ -15,13 +22,19 @@ Install from source:
```bash ```bash
git clone https://github.com/agentscope-ai/ReMe.git git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe cd ReMe
pip install -e ".[core]" pip install -e reme_studio -e ".[core]"
cd reme_studio
npm ci
npm run build:static
cd ..
``` ```
Installing the `core` extra is recommended. The current code imports the AgentScope wrapper, and self-evolving memory also The static build step requires Node.js 22.13 or newer and makes Studio available when running ReMe from the source tree.
depends on it.
To use agent workflows such as `auto_memory`, `auto_resource`, and `auto_dream`, configure an LLM: Installing the `core` extra is recommended. The current code imports the AgentScope wrapper, and self-evolving memory
also depends on it.
To use agent workflows such as `auto_memory`, `auto_resource`, `auto_dream`, and proactive refresh, configure an LLM:
```bash ```bash
cat > .env <<'EOF' cat > .env <<'EOF'
@ -51,10 +64,16 @@ reme start service.port=8181
```bash ```bash
reme version reme version
reme health_check reme health_check
reme list reme help
``` ```
`reme list` lists server actions. Ordinary commands invoke server Jobs over HTTP. `reme help` lists server actions. Ordinary commands invoke server Jobs over HTTP.
The base `reme-ai` package does not include frontend assets. Install `reme-ai[web]` or `reme-ai[core]`, then open
<http://127.0.0.1:2333/> for ReMe Studio. It uses the same service to
browse, edit, and search the workspace and inspect the digest wikilink graph. Disable it with
`service.web_enabled=false`, or provide a custom build with `service.web_static_dir` / `REME_WEB_STATIC_DIR`. The Job
API still starts if no web build is found.
--- ---
@ -65,7 +84,8 @@ The default workspace is `.reme/` under the current directory. It is created aut
```text ```text
.reme/ .reme/
├── metadata/ # persistent indexes, graph, catalogs, and related state ├── metadata/ # persistent indexes, graph, catalogs, and related state
├── session/ # agent sessions and original conversations ├── session/ # source conversation records
├── mem_session/ # generated Agent wrapper sessions/config
├── resource/ # external resources ├── resource/ # external resources
├── daily/ # daily notes ├── daily/ # daily notes
└── digest/ # long-term memory └── digest/ # long-term memory
@ -91,20 +111,24 @@ reme write \
description="Example memory for the quick start" \ description="Example memory for the quick start" \
content="# Quick Start Demo content="# Quick Start Demo
ReMe indexes Markdown under the daily, digest, and resource directories. The default live watcher indexes Markdown under the daily and digest directories.
Related link: [[digest/wiki/search-demo.md]]" Related link: [[digest/wiki/search-demo.md]]"
``` ```
`path` is relative to the workspace. A missing suffix is automatically completed with `.md`. For Markdown files, `name` and `path` is relative to the workspace. A missing suffix is automatically completed with `.md`. For Markdown files, `name`
and
`description` are written to frontmatter. `description` are written to frontmatter.
The background watcher builds the index automatically. You can also rebuild it manually: The background watcher ingests workspace files automatically. You can manually rebuild the derived BM25 and embedding
indexes from the chunks it has already ingested:
```bash ```bash
reme reindex reme reindex
``` ```
This command does not scan workspace files, rechunk content, or rebuild the wikilink graph.
Search: Search:
```bash ```bash
@ -117,8 +141,8 @@ Read:
reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20 reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20
``` ```
With the default configuration, retrieval is primarily BM25 plus wikilink graph expansion. Vector retrieval is supported by With the default configuration, retrieval is primarily BM25 plus wikilink graph expansion. Vector retrieval is supported
the code, but the embedding store is disabled by default. For the full retrieval flow, see by the code, but the embedding store is disabled by default. For the full retrieval flow, see
[Memory Search](./memory_search.md). [Memory Search](./memory_search.md).
--- ---
@ -132,7 +156,13 @@ reme frontmatter_read path=digest/wiki/quick-start-demo
reme frontmatter_update path=digest/wiki/quick-start-demo metadata='{"tags":["demo"]}' reme frontmatter_update path=digest/wiki/quick-start-demo metadata='{"tags":["demo"]}'
``` ```
The name `list` is used by the CLI to list actions, so the file-listing Job must be called over HTTP: The file-listing Job can be called directly from the CLI:
```bash
reme list path=digest recursive=true limit=50
```
The equivalent HTTP call is:
```bash ```bash
curl -s http://127.0.0.1:2333/list \ curl -s http://127.0.0.1:2333/list \
@ -161,8 +191,9 @@ reme auto_memory \
memory_hint="Record the user's preference" memory_hint="Record the user's preference"
``` ```
After placing external material under `resource/YYYY-MM-DD/`, the default background task watches After placing external material under `resource/YYYY-MM-DD/` or directly under `resource/`, the default background task
`md/txt/json/jsonl/csv/yaml/html`. You can also trigger processing manually: watches text resources (`md/txt/json/jsonl/csv/yaml/html`) and image resources
(`png/jpg/jpeg/webp/gif/bmp/tiff/heic`). You can also trigger processing manually:
```bash ```bash
reme auto_resource changes='[{"path":"resource/2026-06-20/report.md","change":"added"}]' reme auto_resource changes='[{"path":"resource/2026-06-20/report.md","change":"added"}]'
@ -172,10 +203,11 @@ Distill daily notes into long-term digest memory:
```bash ```bash
reme auto_dream date=2026-06-20 reme auto_dream date=2026-06-20
reme proactive date=2026-06-20 reme proactive_read date=2026-06-20
``` ```
These flows require a working LLM. Without an LLM configuration, start with basic capabilities such as `write`, `read`, and These flows require a working LLM. Without an LLM configuration, start with basic capabilities such as `write`, `read`,
and
`search`. `search`.
For more detail, see [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md), For more detail, see [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md),
@ -212,3 +244,5 @@ You can also specify a YAML or JSON configuration file:
```bash ```bash
reme start config=/path/to/custom.yaml reme start config=/path/to/custom.yaml
``` ```
Continue with the [CLI Reference](./reference/cli.md), [Job API Reference](./reference/jobs.md), or [Diagnostics, Backup, and Recovery](./operations.md).

87
docs/en/reference/cli.md Normal file
View file

@ -0,0 +1,87 @@
---
title: CLI Reference
description: ReMe command syntax, service invocation, configuration overrides, and plugin commands.
---
# CLI Reference
The basic syntax is:
```text
reme ACTION key=value ...
```
## Start an Application
```bash
reme start
reme start config=demo
reme start workspace_dir=/data/reme service.port=8181
reme start job=search query="keywords" limit=5
```
`start job=<name>` runs one Job through a one-shot service; plain `start` runs the configured Service.
## Call Jobs
Once a service is running, each action name is a Job name:
```bash
reme help
reme health_check
reme search query="project decision" limit=10
reme read path=digest/wiki/project.md start_line=1 end_line=80
```
Use JSON for structured values:
```bash
reme auto_memory \
session_id=example \
messages='[{"role":"user","content":"Remember this preference"}]'
```
Client-selection arguments—`backend`, `transport`, `host`, `port`, `timeout`, `command`, `args`, and `show_metadata`—configure the client and never leak into the Job payload.
## Configuration overrides
```bash
reme start \
config=/path/to/custom.yaml \
service.port=8181 \
service.web_enabled=false \
plugins='["auto-fin"]'
```
Leading `-` or `--` is optional. Use dots for nested keys and JSON for arrays and objects.
## Service discovery
```bash
reme find_reme
```
This reports a discovered service but never starts or replaces a process.
## Plugin commands
Package management runs locally rather than through HTTP or MCP:
```bash
reme plugins list
reme plugins show auto-fin
reme plugins validate auto-fin
reme plugins install plugins/auto-fin
reme plugins uninstall auto-fin
```
See [Plugin Management](../plugin_management.md) for the complete workflow.
## Discover active capabilities
The [Job API Reference](./jobs.md) describes the default configuration. Plugins and custom YAML may change the running service, so automation should prefer:
```bash
reme help
reme app_config
```

340
docs/en/reme-blog.md Normal file
View file

@ -0,0 +1,340 @@
# ReMe: A Personal Knowledge Base That Keeps Growing After Every Conversation
Every day, we talk with AI.
It helps us analyze projects, read papers, and troubleshoot problems. We also tell it about our preferences, plans, and ideas we have not fully worked out yet.
But most of the time, when a conversation ends, its value is locked away in the chat history. The next time we open a new window, the AI may remember a conclusion but not where it came from. It may find an old conversation but fail to connect it with materials we read or decisions we made later.
Useful long-term memory should do more than preserve what once happened. It should keep organizing information, building connections, and bringing past knowledge back into future reasoning when needed.
That is exactly what ReMe sets out to do.
> **ReMe is a local-first, self-evolving personal knowledge base for AI agents. It continuously turns conversations and resources into readable, editable, searchable, and interconnected Markdown memories, while surfacing threads worth following.**
GitHub: [https://github.com/agentscope-ai/ReMe](https://github.com/agentscope-ai/ReMe)
Documentation: [https://reme.agentscope.io](https://reme.agentscope.io)
<p align="center">
<img src="../figure/reme-blog/reme-blog-cover-benchmark.png" alt="ReMe self-evolving personal knowledge base and public benchmark results" width="100%">
</p>
## A Memory Loop That Keeps Growing
<p align="center">
<img src="../figure/reme-blog/reme-blog-architecture.svg" alt="ReMe self-evolving memory loop" width="100%">
</p>
ReMe is not another chatbot, nor does it try to replace the agents you already use. It is a local memory layer that agents such as QwenPaw, OpenClaw, Hermes, and Claude Code can share.
Built around a set of ordinary files, it does four things:
- Auto Memory extracts information worth keeping from conversations;
- Auto Resource turns external materials into traceable memories;
- Auto Dream consolidates daily memories into long-term knowledge;
- Index, Search, and Proactive bring old memories back into new tasks.
Together, they form a `capture → index → consolidate → recall` loop:
- Conversations and external resources are preserved first;
- Valuable information is organized into daily memories;
- Scattered events are consolidated into long-term knowledge nodes;
- Search, knowledge links, and interest discovery bring old memories back into future reasoning.
Most importantly, this loop is centered not on an opaque database, but on files owned by the user. Indexes, graphs, and caches are merely derived state that can always be rebuilt.
## Memory as File: Your Memories Are Your Files
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-as-file.svg" alt="ReMe Memory as File" width="100%">
</p>
ReMe's core design is called **Memory as File, File as Memory.**
“Memory as File” means long-term memories are not hidden inside a product. They live in Markdown, JSONL, YAML, and original resource files within your workspace. You can open them directly in VS Code, Typora, or Obsidian, and back them up or move them with Git, cloud storage, or your own synchronization setup.
“File as Memory” means each file is more than plain text. With YAML frontmatter, section structure, line ranges, and Wikilinks, it becomes a memory node that can be indexed, connected, and continuously evolved.
For example, a long-term memory about writing preferences might look like this:
```markdown
---
name: "User preference: technical writing style"
description: Prefers stating the problem and outcome first, followed by technical details and examples.
kind: preference
---
The user wants technical articles to have a clear narrative and avoid unnecessary jargon.
When writing an article, refer to [[digest/procedure/Technical content writing process.md]].
## Sources
This preference was observed in [[daily/2026-08-07/content-discussion.md]], which records the user's writing guidance.
```
Months later, even if you have forgotten the conversation, the agent can still read the preference, find the related process, and follow `Sources` back to the original context.
This is also the key difference between ReMe and “black-box memory”: agents can organize memories, but users always retain the right to inspect, correct, move, and delete them.
## Auto Memory: Turning Conversations into a Daily Journal
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-memory.svg" alt="ReMe Auto Memory turns conversations into daily memories" width="100%">
</p>
A great deal of valuable information does not begin with “please remember this.”
For example, you might say in a conversation:
> “Let's not refactor the login module this week. We can do it after the customer demo. Upgrading dependencies directly caused compatibility issues last time, so let's add regression tests first.”
This short passage contains project status, a time constraint, a lesson from a previous failure, and a next action. Auto Memory extracts these details from the conversation stream and writes them into a daily memory card, while retaining a source conversation record in `session/dialog/`.
```text
session/dialog/project-a.jsonl Source conversation record
daily/2026-08-07/login-refactor-decision.md Content-named memory card
daily/2026-08-07.md Daily index, providing an overview
```
`session_id` remains in the card's frontmatter for stable lookup and provenance; the filename comes from the Agent-generated
topic/event `name`, so it does not have to match the session ID.
The next time the login module comes up, the agent does not need to search through the entire chat history. It can immediately see why the refactor was postponed, what went wrong before, and what should happen next.
It is like having a recorder who is always present—not one that mechanically transcribes every word, but one that organizes what will still matter later.
## Auto Resource: Bringing External Materials into the Same Memory System
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-resource.svg" alt="ReMe Auto Resource turns external materials into traceable personal memories" width="100%">
</p>
Not all valuable information comes from conversations. Research materials, project documents, meeting notes, archived web pages, and structured data may all become part of a personal knowledge base.
Auto Resource provides a general entry point for external materials. After a resource enters `resource/`, ReMe preserves the original and organizes its topics, key facts, and actionable information into daily cards with `source_resource` links. It supports text resources including Markdown, plain text, JSON, JSONL, CSV, YAML, and HTML, plus image resources that a vision model turns into caption cards.
In other words, Auto Memory builds personal knowledge from conversations, while Auto Resource builds it from non-conversational materials. Both streams flow into the same daily memory layer, where ReMe indexes, consolidates, and retrieves them together.
### Daily Paper: An Example External-Resource Workflow
Daily Paper is an optional plugin built on this file-based memory system. It collects papers from the weekly and monthly Hugging Face Papers rankings, removes items recommended recently, ranks the remaining papers, selects three, saves their PDFs, and generates Chinese paper notes and a briefing that takes about five minutes to read.
Imagine that you regularly follow research on agent memory. Each morning, instead of receiving only three links, you get three detailed notes already saved locally. The briefing points to the original notes through Wikilinks, and each note links back to its PDF. A month later, when you ask, “What recent methods compress long-term memory?”, those materials are already in the same retrieval system. There is no need to search through browser history again.
Daily Paper demonstrates how Auto Resource can be composed into a concrete workflow, but the external-resource pipeline is not limited to papers.
## Auto Dream: Growing Daily Notes into Connected Long-Term Knowledge
<p align="center">
<img src="../figure/reme-blog/reme-blog-auto-dream.svg" alt="ReMe Auto Dream extracts, classifies, and consolidates long-term knowledge from daily memories while adding Wikilinks" width="100%">
</p>
As daily notes accumulate, a new problem emerges: the information is all there, but it remains scattered across different dates.
Suppose conversations and external materials give you three pieces of information about the same problem:
- The first time a build hung, clearing the cache did not help;
- A project document later confirmed that insufficient Node.js memory was the root cause;
- A third note added that the issue occurs more often in large TypeScript projects.
By default, Auto Dream looks at the two most recent days ending at the target date and sends only daily files changed since
the previous run to extraction. It merges cross-file evidence for the same abstraction and keeps only the strongest reusable
memories within a default cap of five units, then writes them into three categories of long-term memory:
- `Personal`: preferences, conventions, and constraints specific to a user, team, or project;
- `Procedure`: repeatable processes, methods, and troubleshooting guides;
- `Wiki`: general definitions, principles, observations, and knowledge.
For example, the information above would become `digest/procedure/Troubleshooting frozen frontend builds.md`, which records the triggering conditions, diagnostic sequence, failed attempts, solution, and scope of applicability—instead of simply concatenating several daily notes.
When consolidating each memory unit, Auto Dream first searches existing nodes across `personal`, `procedure`, and `wiki`, distinguishing between the “same abstraction” and “related knowledge.” The same abstraction determines how the target node evolves:
- `CREATE`: no equivalent memory exists, so create a new node;
- `CORROBORATE`: the same conclusion appears again, so add its source and strengthen confidence;
- `REFINE`: new material adds conditions, steps, or details;
- `CORRECT`: new information corrects an earlier conclusion.
Related knowledge is written into the body as Wikilinks during the same consolidation process. This is Auto Link. For example, “Troubleshooting frozen frontend builds” can connect general knowledge, team preferences, and original evidence at once:
```markdown
This issue often occurs in [[digest/wiki/Large TypeScript projects.md]]. When resolving it,
follow the “add regression tests first” convention in [[digest/personal/Team change preferences.md]].
## Sources
The root cause and applicable scenarios were documented in
[[daily/2026-08-07/build-debug.md|Build troubleshooting record]].
```
Knowledge evolves and links are created in the same workflow. Relationships are not invisible edges hidden in a graph database; they are readable, editable content in the files themselves. The files can rebuild the graph—the graph never takes control of the files.
## Memory Index: Turning Ordinary Files into a Searchable Memory Network
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-index.svg" alt="ReMe Memory Index build process" width="100%">
</p>
Markdown is easy for people to read, but if files are merely piled into directories, agents still struggle to find them
quickly. The default live index watches Markdown under `daily/` and `digest/`. A separate resource workflow watches
`resource/` and turns those files into daily cards that enter the same index. Manual `reindex` rebuilds BM25 and
embedding indexes from the chunks those ingestion paths have already accepted; it does not rescan files or rebuild the
Wikilink graph.
A Markdown file is parsed into:
- One file node containing file-level information such as its path and frontmatter;
- Multiple semantic chunks split, wherever possible, along the boundaries of headings, paragraphs, lists, and code blocks, while retaining section structure and line numbers;
- Multiple Wikilink edges recording what the file points to and what points back to it.
For retrieval, ReMe can combine three types of signals:
| Retrieval signal | Problem it solves | Example |
|------------------|-------------------|---------|
| BM25 keywords | Exact names, terms, and identifiers must not be missed | “CATL”, “issue #184” |
| Embedding vectors | Semantically similar wording should still match | “build frozen” and “packaging stage not responding” |
| Wikilink graph | Reveal upstream and downstream relationships after finding a node | From “cobalt” to “ternary cathodes” and related research notes |
The default configuration enables BM25 and Wikilink expansion out of the box. Embeddings are optional and participate in vector retrieval only when enabled. Indexes, graphs, and caches are stored in `metadata/`; even if deleted, they can be rebuilt from the user's source files.
## Memory Search: Find the Answer First, Then Expand Relationships Progressively
<p align="center">
<img src="../figure/reme-blog/reme-blog-memory-search.svg" alt="ReMe hybrid search and progressive expansion" width="100%">
</p>
Many RAG systems put all Top-K passages into the context at once. This is simple, but it creates two problems: isolated chunks lack context, while expanding every neighbor's full text quickly consumes tokens.
ReMe's hybrid search lets BM25 and optional vector retrieval produce their own candidates, then fuses the rankings with RRF. Instead of directly comparing BM25 scores with cosine similarities—two different scales—RRF combines where each result appears in the two ranked lists.
After retrieval, information expands progressively in three layers:
1. **Start with the matching passage**: return the most relevant chunk, file path, and line numbers;
2. **Then inspect the relationship directory**: show the file's outgoing and incoming links, including only each neighbor's path, name, description, and anchor rather than loading all of its content immediately;
3. **Finally, go deeper as needed**: the agent decides which relationship is genuinely relevant, then reads the original file or continues traversing the graph.
For example, you ask: “What was the name of the book about attention that Alice recommended last time?”
The first step may find a dinner note that says only, “The title contains the word ‘deep.’” The result also shows that the note links to Alice's personal node and is backlinked by reading notes for *Deep Work*.
The agent does not need to load Alice's entire profile, every reading note, and a whole month of journal entries into its context. It only needs to follow the most relevant link and read once more before answering:
> It was *Deep Work*. Alice recommended it at that dinner, and you later read Chapter 3 and left notes.
This resembles human association: first recall a fragment, then follow the trail to recover the full context.
## Proactive: Discovering Needs You Have Not Yet Put into Words
<p align="center">
<img src="../figure/reme-blog/reme-blog-proactive.svg" alt="ReMe Proactive's two-way memory loop" width="100%">
</p>
At this point, ReMe has two input streams that continuously enrich the knowledge base:
- Auto Memory distills personal context from ongoing conversations;
- Auto Resource adds new knowledge from external materials.
Proactive reverses the direction. From accumulated conversations and materials, it discovers topics you have not yet resolved or may want to pursue, along with information you have not noticed but that closely relates to your recent work. These discoveries can then guide what external knowledge enters the system next.
For example, over the past week you separately mentioned that:
- Search results lack sources;
- Long documents lose section context after chunking;
- You want to compare several agent-memory evaluation methods.
Even though you never explicitly said, “Help me systematically study the explainability of memory retrieval,” Auto Dream can distill an interest topic from these daily memories:
```yaml
title: Evaluating the explainability of memory retrieval
reason: The user has recently focused on source tracing, structure-aware chunking, and memory evaluation.
evidence: daily/2026-08-07/search-discussion.md
keywords:
- memory search
- source attribution
- benchmark
```
In a future beta release, after reading this topic through Proactive, a host agent could ask at an appropriate moment, “Would you like me to turn the retrieval issues we discussed recently into an evaluation plan?” It could also use the topic to initiate a user-authorized research workflow. Users would not need to identify and explicitly specify their interests and scope in advance; external resources related to needs implicit in their conversations could continue flowing into the knowledge base.
There is an important boundary: **ReMe's Proactive feature only reads and exposes interest topics. It does not independently access the internet, send notifications, or rewrite the knowledge base.**
It does not guess your interests from nowhere. It surfaces clues that already appeared in your behavior and conversations but have not yet been explicitly stated.
## Performance: Can It Retrieve Information from Very Long Histories?
ReMe uses LongMemEval and BEAM to evaluate memory across multiple sessions and extremely long conversations. During evaluation, the agent can use ReAct to search and read over multiple rounds, generate an answer, and then receive an LLM-as-judge score.
| Benchmark | Setting | Sample size | Agentic score | Primary capabilities tested |
|-----------|---------|------------:|---------------:|-----------------------------|
| **LongMemEval cleaned-s** | **Overall** | **500 questions** | **89.4%** | Cross-session retrieval, knowledge updates, and temporal reasoning |
| BEAM | 100K context | 20 cases / 400 questions | 66.1% | Ten types of long-context memory tasks |
| BEAM | 1M context | 35 cases / 700 questions | 65.0% | Larger-scale, ultra-long conversation settings |
LongMemEval cleaned-s includes single-session facts, preferences, multi-session reasoning, knowledge updates, temporal reasoning, and other question types. ReMe achieved an overall Agentic score of 89.4% across 500 questions. See the [LongMemEval evaluation guide](../../benchmark/longmemeval/README.md) for the complete workflow and breakdown.
BEAM covers ten categories of tasks, including contradiction resolution, event ordering, information extraction, knowledge updates, multi-session reasoning, preference following, summarization, and temporal reasoning. ReMe scored 66.1% on 20 cases / 400 questions with a 100K context and 65.0% on 35 cases / 700 questions with a 1M context. See the [BEAM evaluation guide](../../benchmark/beam/README.md) for the complete setup.
ReMe also uses $\pi$-Bench to evaluate the potential of multi-session reasoning to improve agent proactivity. The PROC score in $\pi$-Bench evaluates capabilities including directly fulfilling hidden intent, guiding targeted clarification, recovering cross-session preferences, reusing cross-session conventions, inferring cross-task dependencies, and advancing underspecified requests. Across five user personas, ReMe Agent achieved an average PROC score of 0.580, outperforming NanoBot by 2.4% under the same test-model configuration. See the [$\pi$-Bench paper](https://arxiv.org/abs/2605.14678) for details about the benchmark.
## Who Is ReMe For?
### People Who Use Agents Directly
If you want AI to understand you continuously throughout a long-term collaboration, ReMe lets your personal assistant stop starting from scratch. Your preferences, project context, important materials, and past decisions accumulate through ongoing conversations and can be found again when they are genuinely relevant.
Researchers, engineers, analysts, and other knowledge workers all fall into this category. Researchers can connect papers, discussions, and reading notes; engineers can preserve project decisions and cross-session troubleshooting experience; analysts can build an evolving record of events, perspectives, and sources. Their professions differ, but they share the same need: AI that can understand the past, accumulate experience, and recover supporting context for the next task.
### Developers Who Build Agents
If you are building an agent, harness, or AI product, ReMe provides an independent long-term memory layer. Through its CLI, HTTP API, MCP Server, or Python API, you can let multiple agents share the same file-based workspace without reimplementing memory extraction, knowledge organization, hybrid retrieval, and relationship expansion for every application.
Files remain the source of truth, while indexes and caches can be rebuilt at any time. This also makes it easier to determine whether an incorrect retrieval originated in the source material, memory consolidation, or the retrieval pipeline.
Ultimately, ReMe is for users and developers who want AI to do more than “answer this one request”: they want it to understand the past, accumulate experience, and know them better over the course of a long-term collaboration. We want agents to understand you better the more you use them—but that understanding should not live in a black box that you cannot inspect, correct, or take with you.
ReMe's answer is straightforward:
- Memories are files owned by the user;
- Original information preserves what happened, while long-term knowledge preserves the abstraction;
- New conversations and resources keep flowing in, while existing knowledge is continuously supplemented and corrected;
- Every conclusion can be traced to relationships and sources through Wikilinks;
- Indexes and caches serve the files rather than replace them;
- Agents can remember, organize, search, and discover, but users always retain ultimate control.
When these mechanisms come together, a personal knowledge base is no longer a repository you must maintain by hand.
It remembers a little more after every conversation and understands a little more after every new resource. At night, it reorganizes scattered experiences. When a future question arises, it follows the connections between pieces of knowledge and brings back the memory you actually need.
That is what ReMe sets out to do: **make memory not only persistent, but continuously evolving.**
## Integrate ReMe with the Agents You Already Use
ReMe can run as a local memory service accessed through its CLI, HTTP API, or MCP Server, or it can be embedded in a host
process through its Python API. The default HTTP service can also serve ReMe Studio at the same address for browsing,
editing, and searching the workspace and inspecting the digest wikilink graph. Different agents can choose the integration
that best fits their runtime environment and share the same local memory workspace when needed.
| Agent | Recommended integration | Capabilities after integration |
|-------|-------------------------|--------------------------------|
| **DeepSeek Harness** | Install [`@agentscope-ai/reme-dsh-plugin`](https://reme.agentscope.io/en/integrations/dsh) as a DSH profile bundle. | Long-term memory guidance, `reme_search`, automatic capture of completed main-agent turns, scheduled Auto Dream, and ReMe Status. |
| **OpenClaw** | Install [`@agentscope-ai/reme-openclaw-plugin`](https://reme.agentscope.io/en/integrations/openclaw) as the native memory plugin. | Recall before conversational root-agent runs, explicit search, automatic turn capture, scheduled Auto Dream, and status diagnostics. |
| **QwenPaw** | Embed ReMe in-process through the Python API. | Reuse the host application's lifecycle and model configuration while keeping memories local and file-based. |
| **Claude Code** | Start the shared streamable HTTP MCP Service and install the [Claude Code plugin](https://reme.agentscope.io/en/integrations/claude-code). | Semantic, graph, and state recall through MCP, plus asynchronous session capture through a Stop hook. |
| **Hermes** | Install [`integrations/hermes_agent`](../../integrations/hermes_agent) and choose HTTP or embedded mode. | Automatically recall relevant memories before model calls and invoke `auto_memory` asynchronously after each conversation turn. |
| **Codex and other CLI-capable agents** | Copy or install [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md). | Search, read, and write memories through the CLI; automatic recording requires the host agent to integrate explicitly with the conversation lifecycle. |
For installation, configuration, and integration demos, see the [README](../../README.md).
## Contributions Welcome
ReMe is open source, and we welcome the community's help in making this self-evolving memory system more complete:
- Integrate more agents and harnesses so different runtime environments can use the same user-owned long-term memory;
- Contribute new Auto Resource sources and workflows so papers, news, and other public materials can continuously enter the knowledge base;
- Improve Auto Memory, Auto Dream, Auto Link, hybrid search, and Proactive so memories are organized more accurately, relationships are clearer, and retrieval is more reliable;
- Add application examples, evaluation tasks, and diagnostic reports to help us understand successes and failures in real long-term use;
- Improve documentation and tests, or share your needs and ideas for personal AI memory through an Issue.
Whether it is a code contribution, a use case, a bug report, or a new memory workflow, every contribution can bring ReMe closer to a truly readable, controllable, and continuously evolving personal knowledge base.
Contribution guide: [https://docs.agentscope.io/reme/latest/en/contribution](https://docs.agentscope.io/reme/latest/en/contribution)

View file

@ -13,9 +13,11 @@ Conversations / external resources
| |
+--> auto_dream +--> auto_dream
| distill daily/ into digest/{personal,procedure,wiki}/ | distill daily/ into digest/{personal,procedure,wiki}/
| and write daily/<date>/interests.yaml
| |
+--> search / node_search / read / traverse / proactive +--> proactive_refresh_cron
| write daily/<date>/interests.yaml
|
+--> search / node_search / read / traverse / proactive_read
let agents retrieve, associate, read, and inspect interest topics let agents retrieve, associate, read, and inspect interest topics
``` ```
@ -54,20 +56,22 @@ session/
daily/ daily/
├── 2026-05-18.md ├── 2026-05-18.md
└── 2026-05-18/ └── 2026-05-18/
├── 2026-05-18-close.md ├── cobalt-supply-risk.md
├── glencore-q3.md ├── glencore-output-update.md
├── cobalt-policy.md ├── drc-cobalt-policy.md
├── cathode-trend.md ├── high-nickel-cathode-trend.md
└── interests.yaml # generated after auto_dream └── interests.yaml # generated by proactive refresh
``` ```
The corresponding flow is: The corresponding flow is:
- `auto_memory` saves the original conversation to `session/dialog/<session_id>.jsonl`, then asks the agent to write - `auto_memory` saves a filtered source conversation record to `session/dialog/<session_id>.jsonl`, then asks the agent to write
important facts to `daily/<date>/<session_id>.md`. important facts to a topic-named `daily/<date>/<generated_name>.md`. The note keeps `session_id` and
- `resource_watch_loop` watches text-file changes under `resource/` and triggers `auto_resource_step` to write a `source_conversation` in frontmatter for stable lookup and provenance.
same-named daily note. - `resource_watch_loop` watches supported text and image changes under `resource/` and triggers `auto_resource_step` to
- `daily_create` maintains `daily/<date>.md` as the index page for that day. write a daily note with `source_resource`. Text resources use the agent, while images use a vision model. The generated
content-based filename is sanitized and de-duplicated; it is not guaranteed to match the resource filename.
- Auto Memory, Auto Resource, and Auto Dream refresh `daily/<date>.md` after writing.
### Day 1 evening: Auto Dream writes to Digest ### Day 1 evening: Auto Dream writes to Digest
@ -81,15 +85,15 @@ reme auto_dream date=2026-05-18
```text ```text
dream_extract_step dream_extract_step
scan daily/2026-05-18.md and changed files under daily/2026-05-18/ scan the daily window from 2026-05-17 through 2026-05-18 by default
output units and topics output at most 5 memory units from changed files
dream_integrate_step dream_integrate_step
recall existing digest nodes with node_search for each unit recall existing digest nodes with node_search for each unit
decide CREATE / CORROBORATE / REFINE / CORRECT decide CREATE / CORROBORATE / REFINE / CORRECT
dream_topics_step
write daily/2026-05-18/interests.yaml
dream_finish_step dream_finish_step
checkpoint successfully processed daily inputs checkpoint successfully processed daily inputs
auto_tag_step
tag the entities in created or modified digest notes
``` ```
Outputs in this scenario: Outputs in this scenario:
@ -122,7 +126,7 @@ Changes to mining-rights policy in the DRC may affect KFM mine operations and sh
## Sources ## Sources
- [[daily/2026-05-18/2026-05-18-close.md]] The production decline and policy risk were recorded in [[daily/2026-05-18/cobalt-supply-risk.md]].
``` ```
Note that wikilinks use literal path semantics. Prefer complete workspace-relative paths with the `.md` extension. ReMe Note that wikilinks use literal path semantics. Prefer complete workspace-relative paths with the `.md` extension. ReMe
@ -218,7 +222,7 @@ the CATL interview record on 2026-05-19.
### Proactive: Read the day's interest topics ### Proactive: Read the day's interest topics
`auto_dream` writes: The independent proactive refresh flow writes:
```text ```text
daily/2026-05-18/interests.yaml daily/2026-05-18/interests.yaml
@ -227,24 +231,41 @@ daily/2026-05-18/interests.yaml
Example: Example:
```yaml ```yaml
version: 2
date: 2026-05-18 date: 2026-05-18
topic_count: 3 generated_at: 2026-05-18T18:00:00+08:00
diversity_days: 7 push: true
topics: topics:
- title: Impact of DRC mining-rights policy on cobalt supply - id: 9c2aa7bd21bf
title: Impact of DRC mining-rights policy on cobalt supply
reason: The user repeatedly mentioned KFM and cobalt-price risk today reason: The user repeatedly mentioned KFM and cobalt-price risk today
keywords: [cobalt, DRC, CMOC, KFM] kind: follow_up
confidence: 0.7
first_seen: 2026-05-18
last_evidence_at: 2026-05-18
evidence: daily/2026-05-18/cobalt-supply-risk.md
paths: paths:
- daily/2026-05-18/2026-05-18-close.md - daily/2026-05-18/cobalt-supply-risk.md
agenda:
- topic_id: 9c2aa7bd21bf
title: Impact of DRC mining-rights policy on cobalt supply
scenario_type: resume_task
opener: Review the KFM policy update before the next cobalt-supply decision.
next_action: Compare the latest policy note with the existing supply-risk assessment.
preconditions: []
delivery: in_conversation
linked_memory: [daily/2026-05-18/cobalt-supply-risk.md]
order_reason: Recent evidence and a concrete next step.
suppressed: []
``` ```
Call: Call:
```bash ```bash
reme proactive date=2026-05-18 reme proactive_read date=2026-05-18
``` ```
The `proactive` Job returns the topics from `interests.yaml` and, optionally, the raw YAML content. The `proactive_read` Job returns the topics from `interests.yaml` and, optionally, the raw YAML content.
### Value of this scenario ### Value of this scenario
@ -321,7 +342,7 @@ The build stalls near the end. CPU usage is low, but memory keeps growing.
## Sources ## Sources
- [[daily/2026-03-10/build-oom-2026-03-10.md]] The failed attempts and successful memory adjustment were recorded in [[daily/2026-03-10/build-oom-2026-03-10.md]].
``` ```
Example `digest/personal/code-style.md`: Example `digest/personal/code-style.md`:
@ -374,7 +395,7 @@ and upgrading the minification plugin did not help last time.
- `digest/procedure/` stores both "how to do it" and "which paths failed," letting the agent reuse diagnostic experience. - `digest/procedure/` stores both "how to do it" and "which paths failed," letting the agent reuse diagnostic experience.
- `digest/personal/` stores user preferences so the agent can follow the same engineering style across sessions. - `digest/personal/` stores user preferences so the agent can follow the same engineering style across sessions.
- The original conversation remains under `session/dialog/`; daily records stay traceable, and digest is only the - The source conversation record remains under `session/dialog/`; daily records stay traceable, and digest is only the
long-term distilled result. long-term distilled result.
## Scenario 3: A Personal Second Brain ## Scenario 3: A Personal Second Brain
@ -423,7 +444,7 @@ At lunch on 2026-04-20, Alice recommended [[digest/wiki/deep-work.md]], a book a
## Sources ## Sources
- [[daily/2026-04-20/lunch-with-alice.md]] The recommendation was recorded in [[daily/2026-04-20/lunch-with-alice.md]].
``` ```
### An associative recall ### An associative recall

118
docs/en/services.md Normal file
View file

@ -0,0 +1,118 @@
---
title: Services and Deployment
description: Run ReMe through HTTP, SSE, MCP, the CLI, and ReMe Studio while respecting its local security boundary.
---
# Services and Deployment
ReMe can run as a local HTTP service, a standalone MCP server, or a one-shot CLI Job. By default HTTP binds to
`127.0.0.1:2333`; one process serves JSON, SSE, streamable HTTP MCP, and optional ReMe Studio.
## HTTP API
```bash
reme start
reme start service.host=127.0.0.1 service.port=8181
```
Regular Jobs become `POST /<job-name>`:
```bash
curl -s http://127.0.0.1:2333/search \
-H 'Content-Type: application/json' \
-d '{"query":"user preferences","limit":5}'
```
Job arguments live at the request body's top level. A regular response follows the `Response` schema:
```json
{"success": true, "answer": "...", "metadata": {}}
```
Unhandled Step failures become unsuccessful responses.
## Streaming and SSE
Jobs with `backend: stream` also use `POST /<job-name>`, returning `text/event-stream`. Error paths emit an error chunk and always terminate the stream. The default `chat` Job is streaming:
```bash
curl -N http://127.0.0.1:2333/chat \
-H 'Content-Type: application/json' \
-d '{"query":"Summarize my long-term preferences"}'
```
MCP does not expose Stream Jobs.
## MCP
The default HTTP service mounts streamable HTTP MCP at `/mcp`:
```yaml
service:
backend: http
mcp_enabled: true
mcp_path: /mcp
mcp_stateless_http: false
```
For a standalone MCP service:
```bash
reme start service.backend=mcp service.transport=stdio
reme start service.backend=mcp service.transport=sse service.port=2333
reme start service.backend=mcp service.transport=streamable-http service.port=2333
```
MCP tools come from non-stream Jobs with `enable_serve: true`. Use `service.jobs` as an allowlist. `injected_job_kwargs` adds server-managed values that callers cannot override.
```yaml
service:
backend: http
jobs: [search, read, traverse, auto_memory]
injected_job_kwargs:
tenant_id: local-user
tool_error_on_failure: true
```
## ReMe Studio
After installing `reme-ai[web]` or `reme-ai[core]`, the default HTTP origin also serves Studio:
```text
http://127.0.0.1:2333/
```
Disable it with `service.web_enabled=false` or select a build with `service.web_static_dir`. Missing static assets do not prevent the Job API from starting.
## One-shot Jobs
```bash
reme start job=search query="user preferences" limit=5
```
This selects the one-shot CLI Service while retaining the normal Application, Component, and Job lifecycle.
## Service discovery
```bash
reme find_reme
```
Ordinary `reme <action>` commands prefer the running service's actual backend, host, port, and transport. They fall back to local configuration only when no service is discovered.
## Security boundary
ReMe is local-first:
- the default binds to `127.0.0.1`;
- HTTP CORS allows any origin;
- Jobs may write, move, or delete files;
- the service layer has no general-purpose user authentication.
Remote access must be enabled explicitly with `reme start service.host=0.0.0.0`. Do not expose the service directly to
the public internet. Place it on a controlled network or behind an authenticated TLS reverse proxy, apply access
controls and request-size limits, and expose only necessary Jobs.
## OpenAPI
FastAPI exposes the active endpoints through `/docs`, `/redoc`, and `/openapi.json`. The Studio SPA fallback preserves these reserved paths.

Some files were not shown because too many files have changed in this diff Show more