diff --git a/AGENTS.md b/AGENTS.md index e5990bef..4953bd66 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,8 @@ the code, schema, tests, configuration, and concise documentation together as ne - `reme/reme.py`: CLI entry point and client/server dispatch. - `reme/application.py`: application assembly, dependency ordering, and lifecycle. +- `reme/components/application_context.py`: application-wide wiring and shared in-memory metadata. +- `reme/components/runtime_context.py`: scratch state shared by steps within one execution. - `reme/config/default.yaml`: built-in jobs, components, and defaults. - `reme/schema/`: public and runtime Pydantic contracts. - `reme/components/`: services, stores, clients, jobs, and component registration. @@ -79,6 +81,34 @@ Do not silently change stable CLI flags, configuration keys, workspace layouts, schemas, or service interfaces. When such a change is required, preserve compatibility where practical and make the migration explicit. +## Step State Model + +Treat every Step as stateless. `BaseJob` stores Step specifications and builds fresh Step +instances for each Job invocation. A Step instance must not use `self` or class variables to +retain mutable runtime state between calls. + +Place state according to its lifetime: + +- Constructor fields on `self`: immutable Step configuration and resolved dependencies only. +- `self.context` (`RuntimeContext`): request data and intermediate results for one Job + execution; sequential Steps share this context. +- `self.app_context.metadata`: in-memory state that must be shared across Step or Job + invocations for the lifetime of the Application. +- Workspace files or a dedicated Component/store: durable state that must survive an + Application restart. + +Use narrow, namespaced keys in `app_context.metadata`, following existing patterns such as +`tool_contexts` and `channel_sink`. The ApplicationContext is shared, so account for +concurrent access when values are mutable. New Step code must not fall back to `self.kwargs` +or another Step field to emulate shared state when `app_context` is absent; tests of shared +state should construct an `ApplicationContext`. If shared state grows into a stable +service-level contract or needs its own lifecycle, locking, or persistence, promote it to a +typed ApplicationContext field or a dedicated Component instead of expanding an ad hoc +metadata bucket. + +Do not use `Response.metadata` as a state store. It is request-scoped output for callers and +diagnostics, distinct from `ApplicationContext.metadata`. + ## Validation Use the narrowest useful check while iterating, then broaden it according to risk. diff --git a/docs/en/auto_dream.md b/docs/en/auto_dream.md index b6b5177f..d84a24ab 100644 --- a/docs/en/auto_dream.md +++ b/docs/en/auto_dream.md @@ -1,7 +1,209 @@ # Auto Dream -```{note} -πŸ“– The English version of this page is in progress. +`auto_dream` is ReMe's long-term memory distillation flow from daily to digest. It scans daily inputs for a specified date, +processes only files that changed since the previous dream, extracts content worth retaining as memory units, integrates those +units into `digest/`, and generates the day's `interests.yaml` for proactive use. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +

+ ReMe Auto Dream and Proactive flow from daily to digest to proactive +

+ +Its daily inputs usually come from [Auto Memory](./auto_memory.md) and [Auto Resource](./auto_resource.md). For the file +semantics of `digest/`, `derived_from::`, and wikilinks, see [Memory as File](./memory_as_file.md). For the linking strategy +used during Integrate, see [Auto Link](./auto_link.md). To read `interests.yaml`, use [Proactive](./proactive.md). + +## Configuration + +The default configuration is in `reme/config/default.yaml`: + +```yaml +auto_dream: + backend: base + parameters: + date: + type: string + default: "" + hint: + type: string + default: "" + topic_count: + type: integer + default: 3 + topic_diversity_days: + type: integer + default: 7 + steps: + - backend: dream_extract_step + file_catalog: dream + topic_session_id: interests + - backend: dream_integrate_step + - backend: dream_topics_step + topic_count: 3 + topic_diversity_days: 7 + - backend: dream_finish_step + file_catalog: dream ``` + +Parameters: + +| Parameter | Purpose | +|---|---| +| `date` | Date to process in `YYYY-MM-DD` format. When empty, use today in the application's timezone. | +| `hint` | Additional guidance from the caller for the Extract and Integrate stages. | +| `topic_count` | Maximum number of topics written to `interests.yaml`. Defaults to 3. | +| `topic_diversity_days` | Number of past days of `interests.yaml` files considered when avoiding duplicate topics. Defaults to 7. | + +## Inputs and Outputs + +Inputs are daily Markdown files for the specified date: + +```text +daily/.md +daily//**/*.md +``` + +`daily//interests.yaml` is excluded from extraction input so topics from the previous run do not feed back into the +next extraction. + +The main outputs are: + +| Output | Description | +|---|---| +| `digest/procedure/*.md` | Methods, workflows, runbooks, and executable experience. | +| `digest/personal/*.md` | User-, team-, and project-related preferences, facts, and long-term context. | +| `digest/wiki/*.md` | General knowledge, concepts, observations, and decision precedents. | +| `daily//interests.yaml` | Topics worth proactive attention from the host agent that day. | +| `metadata/file_catalog/dream*` | Dream-specific catalog used to detect changes in daily inputs. | + +## Four Stages + +### 1. Extract + +`dream_extract_step` performs three tasks: + +1. Refresh the day's index page at `daily/.md`. +2. Scan `daily/.md` and `daily//**/*.md` and compare their mtimes with `file_catalog: dream`. +3. Send only changed files to the LLM and globally extract two structured result types: `units` and `topics`. + +`units` are long-term memory units ready to be distilled into digest. Each has `name`, `bucket`, `summary`, and `paths`. +`bucket` may only be `procedure`, `personal`, or `wiki`; unknown values are routed to `wiki`. + +`topics` are proactive-interest candidates for the day. They contain `title`, `reason`, `evidence`, `keywords`, and +`paths` and are filtered again in the Topics stage. + +If there are no changed files, the flow ends early with success and skips later extraction work. If files changed but no LLM +is configured, Extract fails because extraction requires an LLM. + +### 2. Integrate + +`dream_integrate_step` invokes an agent independently for each unit and integrates that unit into one digest node. It exposes +these tools to the agent: + +```text +node_search, read, frontmatter_read, write, edit, frontmatter_update +``` + +This stage carries the core responsibility of `auto_link`. It first uses `node_search` to recall similar or related nodes at +digest-node granularity, decides whether to create or update a node, and finally writes sources and related digest nodes as +wikilinks. See [Auto Link](./auto_link.md) for the recall, deduplication, and edge-writing rules. + +There are four integration actions: + +| Action | Meaning | +|---|---| +| `CREATE` | No equivalent abstraction exists; create a new digest node. | +| `CORROBORATE` | The same memory appeared again; append a source or strengthen the description. | +| `REFINE` | New material adds boundaries, steps, prerequisites, applicability, or detail. | +| `CORRECT` | New material corrects errors, omissions, or conflicts in the existing node. | + +Successfully integrated units are recorded in `integrate_results`. Failed units enter `failed_units`, and their source paths +enter `failed_paths`. The Finish stage does not checkpoint failed paths, ensuring that they can be retried later. + +### 3. Topics + +`dream_topics_step` turns topic candidates from Extract into the final `daily//interests.yaml` for the day. + +It reads: + +```text +daily//interests.yaml +daily//interests.yaml +``` + +Existing topics from the same day are preserved, while similar topics from the previous `topic_diversity_days` days are +deduplicated. At most three topics are written by default. With an LLM configured, the LLM selects topics that are more +specific, actionable, and non-repetitive. Without an LLM, the step falls back to local normalization and deduplication. + +Example output format. See [Proactive](./proactive.md) for the interface that reads this file: + +```yaml +date: 2026-06-20 +topic_count: 3 +diversity_days: 7 +topics: + - title: Quality regression in the memory retrieval pipeline + reason: The user has recently made repeated changes to search, node_search, and dream integration. + evidence: daily/2026-06-20/session.md + keywords: + - memory search + - auto dream + paths: + - daily/2026-06-20/session.md +``` + +### 4. Finish + +`dream_finish_step` completes the run: + +1. Write successfully processed changed paths to `file_catalog: dream`. +2. Also write `daily//interests.yaml` and `daily/.md` to the catalog. +3. Persist the dream catalog if there were upserts or deletions. +4. Return a summary containing counts for scanned, changed, integrated, topics, checkpoints, and related values. + +Failed paths are not checkpointed. The next `auto_dream` run therefore continues to treat them as changed inputs until +integration succeeds. + +## Running Auto Dream + +CLI: + +```bash +reme auto_dream date=2026-06-20 +``` + +With caller guidance: + +```bash +reme auto_dream date=2026-06-20 hint="Prioritize engineering decisions and long-term preferences" +``` + +The same set of steps can also be placed in a `cron` Job, for example to run every morning: + +```yaml +jobs: + daily_auto_dream: + backend: cron + cron: "30 3 * * *" + steps: + - backend: dream_extract_step + file_catalog: dream + - backend: dream_integrate_step + - backend: dream_topics_step + - backend: dream_finish_step + file_catalog: dream +``` + +## Important Boundaries + +`auto_dream` consumes only daily inputs and does not rewrite daily bodies. Daily preserves facts and the original situation; +digest is the abstracted long-term memory layer. + +`digest` is not a copy of the source text. Its body should preserve reusable abstractions, while details point back to sources +through `derived_from:: [[daily//...]]`. Links follow the workspace-relative wikilink semantics described in +[Memory as File](./memory_as_file.md). + +`auto_dream` does not invent an overview from nothing. Only content that actually appears in daily input and is extracted as +a unit or topic can enter digest or `interests.yaml`. + +The complete flow depends on an LLM for Extract and Integrate. Topics can perform local deduplication without an LLM, but that +does not mean the full dream flow can run offline. diff --git a/docs/en/auto_link.md b/docs/en/auto_link.md index d83ed691..dde6afcb 100644 --- a/docs/en/auto_link.md +++ b/docs/en/auto_link.md @@ -1,7 +1,144 @@ # Auto Link -```{note} -πŸ“– The English version of this page is in progress. +In the current implementation, `auto_link` is not a separately registered Job. It is a capability of the Integrate stage in +`auto_dream`: when `dream_integrate_step` writes a memory unit to `digest/`, it also recalls digest nodes, makes a +deduplication decision, links sources, and weaves wikilinks to related nodes into the result. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +For the complete dream flow, see [Auto Dream](./auto_dream.md). For general wikilink, frontmatter, and workspace-relative +path semantics, see [Memory as File](./memory_as_file.md). For question-answering retrieval, see +[Memory Search](./memory_search.md). + +## Where It Runs + +The default `auto_dream` flow is: + +```yaml +auto_dream: + steps: + - dream_extract_step + - dream_integrate_step # where auto_link actually happens + - dream_topics_step + - dream_finish_step ``` + +The Integrate stage processes each unit independently. A unit is written to exactly one target digest node, but that node may +link to multiple sources and multiple related digest nodes. + +## Goals + +`auto_link` addresses graph quality at write time: + +| Problem | Handling | +|---|---| +| The same memory already exists | Recall and update the existing node instead of creating a duplicate. | +| New and existing material are related | Write workspace-relative wikilinks into the body. | +| A digest node is disconnected from its sources | Point back to daily/resource source material with `derived_from:: [[...]]`. | +| A node contains only isolated prose | Add links to related digest nodes on both CREATE and UPDATE. | + +## Toolchain + +`dream_integrate_step` exposes these tools to the agent: + +```text +node_search +read +frontmatter_read +write +edit +frontmatter_update +``` + +`node_search` is digest-only node retrieval designed for dream integration. It returns node-level signals such as the digest +node's `path` and the `name` and `description` from frontmatter. It does not expand the body and does not perform the link +expansion used by ordinary search. + +`read` and `frontmatter_read` are used only for candidates that may be relevant, avoiding expansion of every recalled result +into a large context. + +## Linking Flow + +### 1. Recall candidate nodes + +The agent first calls `node_search` with the unit's triggers, verbs, nouns, synonyms, and possible failure modes. Broad recall, +for example `limit=20-30`, is recommended by default because this step serves both deduplication and link discovery. + +Recalled results are internally classified into three groups: + +| Classification | Meaning | Next action | +|---|---|---| +| `same_abstraction` | The trigger or underlying abstraction is the same, with substantial content overlap. | Use as the UPDATE target. | +| `related` | An adjacent process, prerequisite, failure mode, concept, preference, or upstream/downstream knowledge. | Write a body wikilink. | +| `unrelated` | Only superficially similar or unrelated. | Ignore. | + +### 2. Choose a write action + +Every unit must select one action: + +| Action | Linking semantics | +|---|---| +| `CREATE` | Write a new `digest//.md` and add source and related-node links to its body. | +| `CORROBORATE` | The same abstraction appeared again; append a new `derived_from:: [[...]]` and strengthen the description when needed. | +| `REFINE` | New material extends the existing node; insert the additional content in the appropriate section and preserve existing links. | +| `CORRECT` | New material corrects the existing node; use source links to identify the basis for the correction. | + +An UPDATE should be additive whenever possible: do not delete existing wikilinks or `derived_from` entries. This prevents +later graph indexing and retrieval from losing edges. + +### 3. Write source edges + +Source edges use Markdown wikilinks: + +```markdown +derived_from:: [[daily/2026-06-20/session.md]] +derived_from:: [[resource/2026-06-20/paper.md]] +``` + +These edges represent the evidence behind a digest node. Plain-text descriptions do not count as source edges because only +wikilinks can be parsed reliably by the file graph. For the complete parsing rules, see +[Memory as File](./memory_as_file.md#wikilink). + +### 4. Write relationships between digest nodes + +Relationships between digest nodes also use complete workspace-relative paths: + +```markdown +relates_to:: [[digest/wiki/hybrid-search.md]] +depends_on:: [[digest/procedure/rebuild-index.md]] +blocks_on:: [[digest/personal/team-review-preference.md]] +``` + +Predicates are open-ended. Common forms include `relates_to::`, `depends_on::`, and `blocks_on::`. The predicate sits outside +the brackets, while the target path goes inside `[[...]]` and should include the `.md` suffix. + +## Bucket Differences + +`auto_link` adjusts the shape of its output according to the unit bucket: + +| Bucket | Writing focus | +|---|---| +| `procedure` | Write a runbook with triggers, steps, inputs, and failure modes. Link prerequisites, substeps, and related preferences. | +| `personal` | Write user-, team-, or project-specific facts and preferences. Link related projects, habits, and decision context. | +| `wiki` | Write general knowledge, principles, observations, and decision precedents. Link concepts, methods, and adjacent knowledge. | + +Regardless of bucket, preserve source edges and weave recalled related digest nodes into the body whenever possible. + +## Relationship to Search + +`auto_link` uses `node_search`, not the question-answering `search`. + +| Capability | Purpose | +|---|---| +| `search` | External question answering; returns chunks and can expand upstream/downstream link context. | +| `node_search` | Dream integration; recalls only digest node-level summaries for deduplication and related-link decisions. | + +This boundary matters. The Integrate stage needs to decide whether the same abstraction already exists and which nodes should +be linked; it should not load large numbers of body chunks into context. [Memory Search](./memory_search.md) handles +question-oriented chunk retrieval, RRF fusion, and link expansion. + +## Failure and Retry + +If integration of a unit fails, `dream_integrate_step` records `failed_units` and `failed_paths`. +`dream_finish_step` does not checkpoint those source paths, so the next `auto_dream` run processes them again. + +This makes auto_link writes retryable: a failure does not mark the input as complete or silently discard digest edges that +should have been created. diff --git a/docs/en/auto_memory.md b/docs/en/auto_memory.md index 29a36f00..c29c1e4a 100644 --- a/docs/en/auto_memory.md +++ b/docs/en/auto_memory.md @@ -1,7 +1,109 @@ # Auto Memory -```{note} -πŸ“– The English version of this page is in progress. +Auto Memory is ReMe's entry point for conversational memory. Each conversation is first distilled into a daily memory card +identified by `session_id`, and the day's `YYYY-MM-DD.md` page then indexes all of those cards. It turns "we talked about it" +into "it was remembered" while preserving the original conversation as evidence. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +

+ ReMe Auto Memory and Auto Resource writing daily memory cards +

+ +For the general file semantics of `daily/`, `session/`, frontmatter, and wikilinks, see +[Memory as File](./memory_as_file.md). + +```text +Conversation + β”œβ”€ step 1: daily/YYYY-MM-DD/.md # one card per conversation + β”œβ”€ step 2: daily/YYYY-MM-DD.md # daily index linking the cards + └─ source: session/dialog/.jsonl # original conversation ``` + +## What It Records + +Auto Memory does not preserve a chat transcript as a running summary. It records information that may remain useful later: + +- User preferences: preferred style, collaboration habits, and long-term requirements. +- Key facts: project background, important numbers, explicit conclusions, and constraints. +- Process decisions: what happened, why a choice was made, and which alternatives were rejected. +- Current state: what has been completed, what is blocked, and what comes next. +- Reusable experience: commands, workflows, diagnostic methods, and solutions. + +## Write Location + +Auto Memory writes distilled memories to `daily/`. Conversations from the same day first become individual cards: + +Example directory: + +```text +workspace/ + daily/ + 2026-06-20.md + 2026-06-20/ + session-a.md + session-b.md +``` + +`daily/2026-06-20/session-a.md` and `daily/2026-06-20/session-b.md` are memory cards distilled from different +conversations. `daily/2026-06-20.md` is the index page for that day. Resource files enter the same daily memory layer; see +[Auto Resource](./auto_resource.md). + +When a call includes `session_id`, Auto Memory records that conversation separately under the given ID: + +```text +daily/2026-06-20/session-a.md +``` + +This keeps different conversations separate. A requirements discussion, a debugging session, and a documentation update can +each have their own memory card. To see what happened on a particular day, start with `YYYY-MM-DD.md`. To inspect what was +distilled from one conversation, open the corresponding `.md`. + +## Preserving the Original Information + +The distilled daily note is optimized for readability; the original conversation is retained for trust and verification. + +While generating memory cards, Auto Memory also saves the raw sessions: + +```text +session/ + dialog/ + session-a.jsonl + session-b.jsonl +``` + +Each daily note points to its corresponding original conversation. When a memory needs verification, follow that link back to +the complete context in which it was created. + +## Message Timestamps + +Auto Memory preserves each message's `created_at` in both the prompt and the raw session JSONL. When importing historical +conversations or benchmark data, provide the actual occurrence time for every message so the model does not confuse event +time with execution time: + +```bash +reme auto_memory \ + session_id=locomo-session \ + messages='[ + {"role":"user","content":"Jon lost his job today.","created_at":"2023-01-19T08:00:00"}, + {"role":"assistant","content":"I am sorry to hear that.","created_at":"2023-01-19T08:01:00"} + ]' +``` + +For compatibility with common dataset schemas, `auto_memory` also checks `time_created`, `timestamp`, `createdAt`, +`timeCreated`, and `created_time` when `created_at` is absent. These fields may appear either at the top level of a message +or inside `metadata`. + +When a call does not explicitly provide `date`, Auto Memory uses the date of the earliest valid `created_at` value in the +messages. If no message contains a valid timestamp, it falls back to the current date. Historical imports may also specify the +target date directly: + +```bash +reme auto_memory \ + session_id=locomo-session \ + date=2023-01-19 \ + messages='[{"role":"user","content":"Jon lost his job today."}]' +``` + +## What Happens Next + +Auto Memory only creates memory in the daily layer. To distill this material further into long-term `digest/` nodes, use +[Auto Dream](./auto_dream.md). To search daily and digest content, use [Memory Search](./memory_search.md). diff --git a/docs/en/auto_resource.md b/docs/en/auto_resource.md index 2e6790ba..a3ae3f19 100644 --- a/docs/en/auto_resource.md +++ b/docs/en/auto_resource.md @@ -1,7 +1,101 @@ -# Auto Resource +# Auto Resource `Beta` -```{note} -πŸ“– The English version of this page is in progress. +Auto Resource is ReMe's entry point for interpreting resources and is currently in **Beta**. Resource files first enter +`resource/` by date and are then interpreted into daily resource cards. Each card's filename comes from the LLM-generated +frontmatter `name`, and `source_resource` links the card back to its original file. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +

+ ReMe Auto Memory and Auto Resource writing daily memory cards +

+ +For the general file semantics of workspace layers, `resource/`, and `daily/`, see +[Memory as File](./memory_as_file.md). For the flow that writes conversations to daily, see +[Auto Memory](./auto_memory.md). + +```text +resource/YYYY-MM-DD/ + β”œβ”€ step 1: daily/YYYY-MM-DD/.md # interpreted resource card + β”œβ”€ step 2: source_resource points to the original resource + └─ step 3: daily/YYYY-MM-DD.md # daily index linking the cards ``` + +## What It Records + +Auto Resource does more than copy file content. It extracts information that will make the resource easier to retrieve and +understand later: + +- Core content: what the resource is mainly about. +- Structure: its sections, tables, fields, and data organization. +- Key details: important numbers, names, dates, and conclusions. +- Context and purpose: why the resource exists and how it relates to current work. +- Actionable items: tasks, deadlines, and follow-up work. + +In short, it turns "a file was archived" into "the resource is usable." + +## Original Resource Entry Point + +Auto Resource uses `resource/` as the entry point for source material. Resources must be placed under a date, which determines +the day whose daily memory layer receives the interpreted card. + +Example directory: + +```text +workspace/ + resource/ + 2026-06-20/ + market-report.md + meeting-notes.csv +``` + +The current Beta version is best suited to text-based resources such as `md`, `txt`, `json`, `jsonl`, `csv`, `yaml`, +and `html`. + +## Resource Cards + +Each resource file produces one daily resource card. The system initially uses the resource file's stem as a temporary path. +After the agent writes the card, the file is renamed according to its frontmatter `name`: + +```text +resource/2026-06-20/market-report.md + ↓ +daily/2026-06-20/market-report-highlights.md +``` + +The resource card links to the original file through frontmatter: + +```yaml +source_resource: "[[resource/2026-06-20/market-report.md]]" +``` + +When a resource changes, Auto Resource finds and updates the corresponding card through `source_resource`. When a resource is +deleted, its daily note is also removed. The older `daily/YYYY-MM-DD/.md` naming convention remains supported +as a fallback. + +## Daily Index + +Resource cards enter the same daily memory layer as Auto Memory cards. The day's `YYYY-MM-DD.md` page acts as an index and +organizes those resource cards: + +```text +daily/ + 2026-06-20.md + 2026-06-20/ + market-report-highlights.md + meeting-notes-summary.md +``` + +To review which resources were processed on a day, start with `YYYY-MM-DD.md`. To inspect what was distilled from one +resource, open its corresponding resource card. + +## Preserving the Original Resource + +The interpreted daily note is optimized for readability; the original resource is retained for trust and verification. + +Auto Resource does not move the original file. It remains under `resource/YYYY-MM-DD/`. Text resources can therefore enter +the daily memory flow while their source files stay in their original location. + +## What Happens Next + +Auto Resource only creates resource interpretations in the daily layer. To distill long-term knowledge from resources into +`digest/`, use [Auto Dream](./auto_dream.md). To search original resources, daily cards, and digest nodes, use +[Memory Search](./memory_search.md). diff --git a/docs/en/contributing.md b/docs/en/contributing.md index c8309f05..b4fea323 100644 --- a/docs/en/contributing.md +++ b/docs/en/contributing.md @@ -1,7 +1,221 @@ -# Contributing +# Open Source and Contributing -```{note} -πŸ“– The English version of this page is in progress. +ReMe is open source and hosted on GitHub: -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +**https://github.com/agentscope-ai/ReMe** + +--- + +## How to Contribute + +Thank you for your interest in ReMe. ReMe is a file-first, self-evolving memory system for agents. Contributions are welcome +through issue reports, documentation improvements, additional tests, bug fixes, and new capabilities. + +If this is your first time running ReMe locally, start with [Quick Start](./quick_start.md). If your change affects runtime +layers, Jobs, Steps, or components, read [ReMe Framework](./framework.md). If it affects workspace directories, frontmatter, +wikilinks, or chunking, read [Memory as File](./memory_as_file.md). + +### 1. Before You Begin + +Before investing in an implementation: + +- Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) for an existing issue or discussion. +- If a related issue is still open, comment that you would like to work on it to avoid duplicate effort. +- If no issue exists, create one describing the context, expected behavior, possible implementation, and scope of impact. +- For larger feature changes, align with maintainers on interfaces, configuration, compatibility, and test strategy before + submitting an implementation. + +### 2. Local Development Environment + +The core ReMe code is located in: + +- `reme/`: Python package source, including configuration, components, services, Jobs, Steps, schemas, and utilities. +- `pyproject.toml`: project metadata, dependencies, optional dependencies, command entry points, and test configuration. +- `tests/`: unit and integration tests. + +The project requires Python 3.11 or later. A virtual environment is recommended: + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -e ".[dev,full]" +pre-commit install ``` + +### 3. Development Model + +Before developing ReMe code, read [ReMe Framework](./framework.md). New or modified core capabilities should follow the +layers and call chain described there: + +```text +CLI / Client -> Service -> Application -> Job -> Step -> Component / Workspace +``` + +In practice: + +- Capabilities exposed to users or external systems should normally be orchestrated by a Job, then exposed by a Service as a + CLI-, HTTP-, or MCP-callable interface. +- Reusable infrastructure belongs in `reme/components/`, with dependencies declared through `BaseComponent.bind()`. +- Atomic business operations belong in `reme/steps/` and access the file store, agent wrapper, catalog, LLM, and other + components through `BaseStep.Ref`. +- Request, response, and persistent data structures belong in `reme/schema/` or `reme/enumeration/`. Do not scatter + implicit structures through Step implementations. +- Configuration-driven defaults belong in `reme/config/default.yaml`, and the default configuration must remain runnable + and testable. + +When adding a Step or Job, pay particular attention to these conventions: + +- Register implementations with `@R.register("")`. Registration names should be stable, clear, and match the + configured `backend`. +- After adding a Step file, make sure its package `__init__.py` imports the module; otherwise, the registry will not load it. +- A Step should perform one atomic business operation. Cross-step flows belong in Job configuration or a dedicated + orchestration Step. +- A Job composes Steps and selects normal, streaming, background, or scheduled execution. `enable_serve` controls whether it + is externally exposed. +- When a Step needs components, prefer `BaseStep.Ref`. Do not reconstruct global components inside a Step or bypass + `ApplicationContext`. +- File, index, graph, frontmatter, and wikilink behavior must preserve consistent workspace-relative path semantics. +- Add fast tests under `tests/unit/` for new capabilities. Put cross-component, LLM, embedding, or service behavior under + `tests/integration/` when appropriate. + +### 4. Code and Documentation Changes + +Choose the appropriate entry point for the type of change: + +| Change type | Primary location | Guidance | +|---|---|---| +| Configuration or startup behavior | `reme/config/`, `reme/application.py`, `reme/reme.py` | Keep the default configuration runnable and avoid breaking existing CLI, HTTP, and MCP entry points. | +| Component capability | `reme/components/` | Reuse `BaseComponent`, the registry, and context objects. | +| Job or Step | `reme/components/job/`, `reme/steps/` | Follow the Job -> Step model in [ReMe Framework](./framework.md), keep request and response schemas clear, and add corresponding tests. | +| Data structure | `reme/schema/`, `reme/enumeration/` | Preserve serialization compatibility and existing frontmatter and wikilink semantics. | +| Utility | `reme/utils/` | Keep function boundaries small and cover edge cases with unit tests. | +| User documentation | `docs/en/`, `README.md` | Update documentation when user-visible behavior changes. | + +If a change involves an LLM, embeddings, an external service, file watching, or a background task, also describe its +dependencies, failure behavior, and local validation method. + +### 5. Commit Message Format + +Use [Conventional Commits](https://www.conventionalcommits.org/) to keep history clear. + +Format: + +```text +(): +``` + +Common types: + +- `feat`: new feature +- `fix`: bug fix +- `docs`: documentation only +- `style`: code-style change with no behavior change +- `refactor`: refactoring that neither fixes a bug nor adds a feature +- `perf`: performance improvement +- `test`: add or update tests +- `chore`: build, tooling, or maintenance work + +Examples: + +```bash +feat(search): add link expansion option +fix(file-graph): handle pending wikilinks after move +docs(memory): update auto memory guide +test(config): cover default yaml parsing +chore(pre-commit): update lint hooks +``` + +### 6. Pull Request Titles + +PR titles should use the same format: + +```text +(): +``` + +Requirements: + +- Use `feat`, `fix`, `docs`, `test`, `refactor`, `chore`, `perf`, `style`, `build`, or `revert` as the type. +- Use lowercase letters, numbers, hyphens, or underscores for the scope. +- Keep the description short and state the actual effect of the PR. + +Examples: + +```text +feat(auto-memory): persist source conversation metadata +fix(markdown): keep wikilink aliases during edit +docs(en): add contribution guide +``` + +### 7. Pre-submit Checks + +Before committing or opening a PR, run at least: + +```bash +pre-commit run --all-files +pytest +``` + +For a localized code change, start with a narrower test set: + +```bash +pytest tests/unit/test_search_step.py +pytest tests/unit/test_reme_cli.py +``` + +If `pre-commit` modifies files automatically, commit those changes and rerun the checks until everything passes. + +The current pre-commit configuration includes YAML/TOML/JSON validation, private-key detection, trailing-whitespace checks, +`black`, `flake8`, `pylint`, and `pyroma`. The main formatting rules are: + +- `black --line-length=120` +- `flake8 --max-line-length=120` +- `pylint --max-line-length=120` + +Some integration tests may require an LLM, embeddings, or external service configuration. If you cannot run them locally, +state why they were skipped and what alternative validation you completed in the PR description. + +### 8. Testing Requirements + +Add tests according to the risk of the change: + +- For a bug fix, first add a regression test that reproduces the issue. +- For a new Step, Job, or component, cover at least the main path and a failure path. +- For changes to shared logic such as indexes, graphs, wikilinks, frontmatter, or file operations, add edge cases. +- For changes to the CLI, services, or configuration parsing, cover the user-visible entry point. +- Documentation-only changes usually do not require new tests, but running `pre-commit run --all-files` is still recommended. + +Place tests according to the existing structure: + +- `tests/unit/`: fast tests that require no real external service. +- `tests/integration/`: integration tests spanning components or requiring external configuration. + +### 9. Documentation Contributions + +When a change affects how users install, configure, invoke, or understand ReMe, update the documentation as well. + +Documentation lives under: + +```text +docs/ +``` + +Documentation should: + +- Use clear titles that directly identify a capability or flow. +- Provide commands that can be copied and run. +- Use real repository paths such as `reme/config/default.yaml`, `reme/steps/`, and `tests/unit/`. +- Describe default behavior according to the current code, `pyproject.toml`, and default configuration. + +--- + +## Getting Help + +- Bugs and feature requests: [GitHub Issues](https://github.com/agentscope-ai/ReMe/issues) +- Project home: [GitHub Repository](https://github.com/agentscope-ai/ReMe) +- Documentation site: [https://reme.agentscope.io/](https://reme.agentscope.io/) + +--- + +Thank you for contributing to ReMe. Your improvements help make long-term memory for agents more readable, controllable, and +maintainable. diff --git a/docs/en/framework.md b/docs/en/framework.md index 24313440..b4bc11a7 100644 --- a/docs/en/framework.md +++ b/docs/en/framework.md @@ -1,7 +1,786 @@ -# Framework +# ReMe Framework -```{note} -πŸ“– The English version of this page is in progress. +## 1. Overview -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +The ReMe runtime can be understood as follows: **a configuration-driven Application assembles components and Jobs; the +Service exposes service-enabled Jobs to the CLI, HTTP, or MCP; and each Job executes its Steps in sequence**. + +

+ ReMe framework structure: CLI, Service, Application, Job, Step, and Component +

+ +To run and use ReMe first, see [Quick Start](./quick_start.md). For workspace file semantics, see +[Memory as File](./memory_as_file.md). User-facing guides for retrieval, automatic memory, and proactive reading are +[Memory Search](./memory_search.md), [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md), +[Auto Dream](./auto_dream.md), and [Proactive](./proactive.md). + +### Capability Boundary + +ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into `digest/`, +and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP. + +Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current conversation, +injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those capabilities belong in +the host agent framework. ReMe accepts conversations, resources, and file changes that have already occurred and persists the +information with long-term value. + +```mermaid +flowchart LR + CLI["reme CLI
reme/reme.py"] --> Client["Client
http / mcp"] + Client --> Service["Service
HTTP / MCP"] + Service --> App["Application
reme/application.py"] + App --> Jobs["Jobs
base / stream / background / cron"] + Jobs --> Steps["Steps
reme/steps/**"] + Steps --> Ctx["RuntimeContext
data + Response + stream queue"] + Steps --> Components["Components
store / graph / index / llm / agent / catalog"] + Components --> Workspace["Workspace
daily / digest / resource / metadata"] ``` + +Core layers: + +| Layer | Main location | Responsibility | +|---|---|---| +| CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. | +| Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. | +| Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. | +| Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. | +| Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. | +| Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. | +| Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. | +| Config | `reme/config/` | Default YAML configuration and command-line override parsing. | + +## 2. Directory Structure + +```text +reme/ + reme.py # CLI entry point + application.py # Application assembly and lifecycle + config/ + default.yaml # default service / jobs / components + config_parser.py # config=, dot notation, and env placeholder parsing + components/ + component_registry.py # global registry R + base_component.py # ComponentMixin / BaseComponent / bind dependency declarations + runtime_context.py # context for one Job execution + job/ # BaseJob / StreamJob / BackgroundJob / CronJob + service/ # HTTP / MCP services + client/ # HTTP / MCP clients + file_store/ # file-index coordination layer + file_graph/ # wikilink graph + keyword_index/ # BM25 and other keyword indexes + file_chunker/ # Markdown / default text chunking + file_catalog/ # change checkpoints + as_llm/, as_embedding/ # model wrappers + agent_wrapper/ # AgentScope / Claude Code wrappers + steps/ + base_step.py # BaseStep, Ref, dispatch_steps + common/ # version, help, health_check, demo + file_io/ # read/write/edit/delete/move/frontmatter/daily + index/ # watch/init/update/search/traverse + evolve/ # auto_memory, auto_resource, auto_dream, proactive + transfer/ # upload/download/ingest + channel/ # MCP channel tools +``` + +The default workspace directories are defined by `ApplicationConfig`: + +```text +/ + metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state + session/ # agent sessions and original conversations + resource/ # external resources + daily/ # lightly processed memory + digest/ # long-term digest memory +``` + +`Application.__init__()` first ensures that these directories exist, then initializes the service, components, and Jobs. + +## 3. Startup and Call Chain + +### 3.1 CLI + +The entry point is `reme/reme.py::main()`: + +```mermaid +flowchart LR + A["main()"] --> B["parse_args(*sys.argv[1:])"] + B --> C{action} + C -->|" start "| D["load_env()"] + D --> E["resolve_app_config(**kwargs)"] + E --> F["precheck_start(service)"] + F --> G["ReMe(**config).run_app()"] + C -->|" find_reme "| H["cli_find_reme()"] + C -->|" other actions "| I["call_server(action, **kwargs)"] + I --> J["R.get(ComponentEnum.CLIENT, backend)"] + J --> K["client(action=action, **kwargs)"] +``` + +Common commands: + +```bash +reme start +reme start service.port=8181 +reme version +reme search query="memory" limit=5 +reme search query="memory" backend=mcp +``` + +Configuration parsing supports: + +| Capability | Source | Description | +|---|---|---| +| Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. | +| Explicit configuration | `config=` | Accept a built-in configuration name or a YAML/JSON file path. | +| Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. | +| Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. | +| Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. | + +### 3.2 Service + +`BaseService.run_app()` executes in this order: + +```mermaid +flowchart LR + A["Service.build_service(app)"] --> B["read app.context.jobs"] + B --> C{"job.enable_serve == true?"} + C -->|yes| D["Service.add_job(job)"] + C -->|no| E["skip registration"] + D --> F["Service.start_service(app)"] + E --> F + F --> G["app.start() during lifespan"] + G --> H["Application starts jobs"] +``` + +HTTP service behavior: + +| Job type | HTTP exposure | +|---|---| +| Non-`StreamJob` with `enable_serve: true` | `POST /` returning `Response` JSON. | +| `StreamJob` | `POST /` returning `text/event-stream`. | +| `enable_serve: false` | No endpoint is registered. | + +MCP service behavior: + +| Job type | MCP exposure | +|---|---| +| Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. | +| `StreamJob` | Currently skipped and not registered. | +| `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. | + +## 4. Registry and Dependency Injection + +### 4.1 Global Registry R + +ReMe uses the process-wide singleton `R = ComponentRegistry()`. Every component, Job, and Step is registered with +`@R.register("name")`. + +```python +from ...components import R + + +@R.register("version_step") +class VersionStep(BaseStep): + ... +``` + +The registry key is: + +```text +(component_type, register_name) -> class +``` + +`component_type` comes from a class attribute: + +| Type | Class attribute | +|---|---| +| Step | `BaseStep.component_type = ComponentEnum.STEP` | +| Job | `BaseJob.component_type = ComponentEnum.JOB` | +| Service | `BaseService.component_type = ComponentEnum.SERVICE` | +| FileStore | `BaseFileStore.component_type = ComponentEnum.FILE_STORE` | + +The same backend name can therefore exist under different component types. For example, `http` can be both a service backend +and a client backend. + +### 4.2 Registration Through Module Imports + +Registration happens when a module is imported. `reme/components/__init__.py` imports component packages, while +`reme/steps/__init__.py` imports `channel/common/evolve/file_io/index/transfer`. Each package's `__init__.py` then imports +its concrete modules, causing `@R.register(...)` to execute. + +After adding a Step file, make sure the package's `__init__.py` imports it. Otherwise, the backend will not appear in the +registry. + +### 4.3 Component.bind + +Dependencies between components are declared with `BaseComponent.bind()`. At startup, +`Application._topological_order()` reads every component's `dependencies` and starts them in topological order. + +```mermaid +flowchart LR + A["Component.__init__
self.keyword_index = self.bind(...)"] --> B["Dependency placeholder"] + B --> C["Application._topological_order()"] + C --> D["component.start()"] + D --> E["_resolve_bindings()"] + E --> F["self.keyword_index = app_context.components[type][name]"] + F --> G["component._start()"] +``` + +Rules for `BaseComponent.bind(name, BaseClass, optional=True)`: + +| Scenario | Behavior | +|---|---| +| `name` is empty | Return `None` and skip the dependency. | +| `app_context` exists | Look up `app_context.components[ctype][name]`. | +| Dependency missing and `optional=True` | Resolve to `None`. | +| Dependency missing and `optional=False` | Fail at startup. | +| Standalone mode | A private component can be created with `default_factory`. | + +### 4.4 Step.Ref + +Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps access +components primarily through `BaseStep.Ref`: + +```python +file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE) +agent_wrapper: BaseAgentWrapper = Ref(BaseAgentWrapper, ComponentEnum.AGENT_WRAPPER, optional=True) +``` + +Resolution priority: + +```mermaid +flowchart LR + A["access self.file_store"] --> B{"same-named object in kwargs?"} + B -->|yes| C["use kwargs object"] + B -->|no| D{"same-named object in context.data?"} + D -->|yes| E["use context object"] + D -->|no| F["read name from kwargs['file_store']; default is default"] + F --> G["app_context.components[FILE_STORE][name]"] +``` + +A Step configuration can therefore specify: + +```yaml +steps: + - backend: update_catalog_step + file_catalog: resource +``` + +Here, `file_catalog: resource` means to resolve the `file_catalog` component named `resource`. + +## 5. Application Lifecycle + +The Application converts configuration into runtime objects and starts and closes them in order. + +```mermaid +flowchart LR + A["Application(**kwargs)"] --> B["ApplicationContext(**kwargs)
parse ApplicationConfig"] + B --> C["_setup_workspace_directories()"] + C --> D["_init_service()"] + D --> E["_init_components()"] + E --> F["_init_jobs()"] + F --> G["run_app()"] + G --> H["service.run_app(app)"] +``` + +Startup order in `Application._start()`: + +```mermaid +flowchart LR + A["create optional thread_pool"] --> B["topologically sort components"] + B --> C["start components"] + C --> D["start BaseJob"] + D --> E["start StreamJob"] + E --> F["start BackgroundJob"] + F --> G["start CronJob"] +``` + +During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their dependencies. + +## 6. Job Model + +A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under `jobs:` +in `reme/config/default.yaml`. + +### 6.1 BaseJob + +`BaseJob` is the most common request-oriented Job: + +```mermaid +flowchart LR + Caller["Caller"] --> Job["BaseJob
job(**kwargs)"] + Job --> Ctx["RuntimeContext
merged_kwargs"] + Ctx --> S1["Step 1
await step(context)"] + S1 --> D1["read/write context.data / response"] + D1 --> S2["Step 2
await step(context)"] + S2 --> D2["read/write context.data / response"] + D2 --> Resp["context.response"] + Resp --> Caller +``` + +Important source behavior: + +| Source | Behavior | +|---|---| +| `_start()` | Parse each Step config from YAML into `(step_cls, params)`. | +| `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. | +| `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. | +| Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. | + +### 6.2 StreamJob + +`StreamJob` extends `BaseJob` but returns streaming chunks: + +| Behavior | Description | +|---|---| +| Context | Includes `stream_queue`. | +| Step output | Call `context.add_stream_string(text, ChunkEnum.CONTENT)`. | +| Exception | Write `ChunkEnum.ERROR`. | +| Completion | Always send a `DONE` chunk. | + +### 6.3 BackgroundJob + +`BackgroundJob` runs long-lived loops such as file watchers. Its constructor forces `enable_serve=False`. + +```mermaid +flowchart LR + A["Application starts BackgroundJob"] --> B["_start() creates stop_event and task"] + B --> C["_run_with_supervisor()"] + C --> D["await self()"] + D --> E{"exception?"} + E -->|no, returned normally| F["finish"] + E -->|yes and supervisor = True| G["exponential backoff + jitter"] + G --> C + E -->|yes and supervisor = False| H["raise exception"] + I["close()"] --> J["stop_event.set()"] + J --> K["wait close_timeout; cancel on timeout"] +``` + +The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions, which +allows the supervisor to restart the task. + +### 6.4 CronJob + +`CronJob` extends `BackgroundJob` with a `cron` expression: + +```yaml +jobs: + nightly_dream: + backend: cron + cron: "0 3 * * *" + steps: + - backend: dream_extract_step + - backend: dream_integrate_step + - backend: dream_topics_step + - backend: dream_finish_step +``` + +The current implementation uses `croniter` to calculate the next trigger time. The timezone comes from +`app_config.timezone`. + +### 6.5 Default Job Types + +```mermaid +flowchart LR + Jobs["default.yaml jobs"] --> BG["background
index_update_loop
resource_watch_loop
digest_watch_loop"] + Jobs --> Base["base
version / help / health_check
search / node_search / traverse / reindex
read / write / edit / delete / move / list / stat
daily_list / daily_reindex / daily_write
auto_memory / auto_resource / auto_dream / proactive"] +``` + +## 7. Step Model + +A Step is a concrete business action. Every Step extends `BaseStep` and implements `execute()`. + +```mermaid +flowchart LR + A["Job._build_steps()"] --> B["Step.__init__()"] + B --> C["load prompt
class-named YAML + prompt_dict override"] + C --> D["Step.__call__(context, **kwargs)"] + D --> E["clear Ref cache"] + E --> F["RuntimeContext.from_context()"] + F --> G["input_mapping"] + G --> H["execute()"] + H --> I["output_mapping"] + I --> J["return result"] +``` + +### 7.1 RuntimeContext + +`RuntimeContext` is shared by all Steps within one Job invocation: + +| Field | Description | +|---|---| +| `response` | Final `Response(answer, success, metadata)`. | +| `data` | Free-form dictionary containing input parameters and intermediate results. | +| `stream_queue` | Output queue for streaming Jobs. | +| `stop_event` | Stop signal for background Jobs. | + +Common Step code: + +```python +assert self.context is not None +query = self.context.get("query", "") +self.context["processed_query"] = query.strip().lower() +self.context.response.answer = "..." +self.context.response.metadata["key"] = "value" +return self.context.response +``` + +### 7.2 input_mapping / output_mapping + +`BaseStep.__call__()` invokes `RuntimeContext.apply_mapping()` before and after execution: + +```yaml +steps: + - backend: some_step + input_mapping: + user_query: query + output_mapping: + result: final_result +``` + +The semantics are to copy `context.data[source]` to `context.data[target]`. + +### 7.3 dispatch_steps + +Some Steps produce batches of events and dispatch them to other Steps. `BaseStep.dispatch_steps()` resolves and executes +child Steps according to configuration. + +Example from the default configuration: + +```yaml +index_update_loop: + backend: background + watch_dirs: [ daily_dir, digest_dir ] + watch_suffixes: [ md ] + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [ update_index_step ] + - backend: watch_changes_step + dispatch_steps: [ update_index_step ] +``` + +Flow: + +```mermaid +flowchart LR + Init["init_changes_step"] --> Batch["changes batch"] + Watch["watch_changes_step"] --> Batch + Batch --> Dispatch["dispatch_steps(...)"] + Dispatch --> Update["update_index_step"] + Update --> Store["file_store"] +``` + +## 8. Components in the Default Configuration + +Current default components in `reme/config/default.yaml`: + +| ComponentEnum | Name | Backend | Description | +|---|---|---|---| +| `service` | singleton | `http` | Default HTTP service. | +| `tokenizer` | `default` | `regex` | BM25 tokenizer. | +| `as_embedding` | `default` | `${EMBEDDING_BACKEND:-openai}` | Embedding model wrapper. | +| `embedding_store` | `default` | `local` | Embedding store depending on `as_embedding: default`. | +| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. | +| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. | +| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. | +| `file_graph` | `default` | `local` | Wikilink graph. | +| `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. | +| `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. | +| `file_chunker` | `default` | `default` | Default text chunking, currently supporting `jsonl`. | +| `keyword_index` | `default` | `bm25` | BM25 keyword index. | +| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. | + +Note that the `search` Step configuration contains `vector_weight`, but `file_store.default.embedding_store` is empty by +default. Vector retrieval is available only when the runtime configuration enables an embedding store. + +## 9. Adding a Step + +### 9.1 Minimal Step + +Suppose you want to add a Step that converts input text to uppercase. + +Create a file such as `reme/steps/common/uppercase.py`: + +```python +from ..base_step import BaseStep +from ...components import R + + +@R.register("uppercase_step") +class UppercaseStep(BaseStep): + async def execute(self): + assert self.context is not None + text = self.context.get("text", "") + result = str(text).upper() + + self.context["uppercase_text"] = result + self.context.response.answer = result + self.context.response.metadata["length"] = len(result) + return self.context.response +``` + +### 9.2 Registering the Step + +Make sure `reme/steps/common/__init__.py` imports the new module. Add: + +```python +from . import uppercase +``` + +The reason is that `@R.register("uppercase_step")` only executes after the module is imported. + +### 9.3 Accessing Components + +If a Step needs an existing component, prefer the Refs provided by `BaseStep`: + +```python +class MySearchStep(BaseStep): + async def execute(self): + assert self.context is not None + results = await self.file_store.keyword_search( + self.context.get("query", ""), + limit=5, + ) + ... +``` + +Common attributes available directly: + +| Attribute | Component resolved by default | +|---|---| +| `self.as_llm` | `.model` from `as_llm: default`. | +| `self.agent_wrapper` | `agent_wrapper: default`; optional. | +| `self.file_catalog` | `file_catalog: default`; optional. | +| `self.file_store` | `file_store: default`. | + +To select a non-default component from Job configuration: + +```yaml +steps: + - backend: my_step + file_catalog: dream +``` + +### 9.4 Step Design Guidance + +| Guidance | Reason | +|---|---| +| Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. | +| Write the final result to `context.response`. | Services and clients consume the standard `Response`. | +| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. | +| A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. | +| Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. | + +### 9.5 Unit Test Example + +A Step can be instantiated directly and passed a `RuntimeContext`: + +```python +import pytest + +from reme.components.runtime_context import RuntimeContext +from reme.steps.common.uppercase import UppercaseStep + + +@pytest.mark.asyncio +async def test_uppercase_step(): + ctx = RuntimeContext(text="hello") + resp = await UppercaseStep()(ctx) + assert resp.answer == "HELLO" + assert ctx["uppercase_text"] == "HELLO" +``` + +## 10. Adding a Job + +A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new execution +model is required. + +### 10.1 Adding a Normal Request Job + +Add the Job under `jobs:` in a YAML configuration: + +```yaml +jobs: + uppercase: + backend: base + description: "Convert text to uppercase." + parameters: + type: object + properties: + text: + type: string + description: "input text" + required: + - text + steps: + - backend: uppercase_step +``` + +Start and call it: + +```bash +reme start +reme uppercase text="hello" +``` + +Call chain: + +```mermaid +flowchart LR + CLI["CLI
reme uppercase text=hello"] --> HTTP["HTTP Client"] + HTTP --> Req["POST /uppercase"] + Req --> S["HttpService"] + S --> J["uppercase BaseJob
job(text='hello')"] + J --> Step["uppercase_step
await step(context)"] + Step --> Resp["context.response.answer = HELLO"] + Resp --> JSON["Response JSON"] + JSON --> CLIOut["CLI prints answer"] +``` + +### 10.2 Adding a Multi-Step Job + +A Job can chain multiple Steps: + +```yaml +jobs: + demo_echo: + backend: base + description: "Normalize query, then echo it." + parameters: + type: object + properties: + query: + type: string + default: "" + min_score: + type: number + default: 0.5 + steps: + - backend: demo_echo_step1 + - backend: demo_echo_step2 +``` + +The first Step writes: + +```text +context["processed_query"] +context["adjusted_min_score"] +``` + +The second Step reads those fields and writes the final `response`. + +### 10.3 Adding a Stream Job + +Use `backend: stream` in configuration: + +```yaml +jobs: + stream_uppercase: + backend: stream + description: "Stream uppercase text." + parameters: + type: object + properties: + text: + type: string + required: + - text + steps: + - backend: uppercase_prepare_step + - backend: uppercase_stream_step +``` + +Example streaming Step: + +```python +from ..base_step import BaseStep +from ...components import R +from ...enumeration import ChunkEnum + + +@R.register("uppercase_stream_step") +class UppercaseStreamStep(BaseStep): + async def execute(self): + assert self.context is not None + for ch in self.context.get("uppercase_text", ""): + await self.context.add_stream_string(ch, ChunkEnum.CONTENT) + return self.context.response +``` + +### 10.4 Adding a Background Job + +Use `backend: background` in configuration: + +```yaml +jobs: + my_watch_loop: + backend: background + watch_dirs: [ daily_dir ] + watch_suffixes: [ md ] + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [ update_index_step ] + - backend: watch_changes_step + dispatch_steps: [ update_index_step ] +``` + +Characteristics of a background Job: + +| Characteristic | Description | +|---|---| +| Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. | +| Has a supervisor | Restarts with exponential backoff after an exception by default. | +| Has a stop event | Notifies the loop to exit during close. | +| Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. | + +### 10.5 Adding a Cron Job + +Use `backend: cron` in configuration: + +```yaml +jobs: + daily_auto_dream: + backend: cron + cron: "30 3 * * *" + steps: + - backend: dream_extract_step + file_catalog: dream + - backend: dream_integrate_step + - backend: dream_topics_step + - backend: dream_finish_step + file_catalog: dream +``` + +An invalid `cron` expression fails at startup. + +### 10.6 When a New Job Backend Is Needed + +Most use cases require only a new Step plus a YAML Job. Consider adding `reme/components/job/*.py` only in these cases: + +| Requirement | New Job class? | +|---|---| +| Add a business command | No; use `backend: base`. | +| Chain existing steps | No; use `steps:`. | +| Need SSE/streaming output | No; use `backend: stream`. | +| Need a background loop | No; use `backend: background`. | +| Need cron scheduling | No; use `backend: cron`. | +| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. | + +Minimal shape of a new Job backend: + +```python +from .base_job import BaseJob +from ..component_registry import R + + +@R.register("my_job_backend") +class MyJob(BaseJob): + async def __call__(self, **kwargs): + # custom scheduling logic + return await super().__call__(**kwargs) +``` + +Also ensure the module is imported by `reme/components/job/__init__.py`. diff --git a/docs/en/memory_as_file.md b/docs/en/memory_as_file.md index 5f73af91..6be4d56d 100644 --- a/docs/en/memory_as_file.md +++ b/docs/en/memory_as_file.md @@ -1,7 +1,373 @@ # Memory as File -```{note} -πŸ“– The English version of this page is in progress. +ReMe's core idea is **Memory as File, File as Memory**. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +

+ ReMe Memory as File model +

+ +**Memory as File**: long-term memory is not hidden in a black-box database. It lives in Markdown files, resource files, and +index snapshots under the workspace. Users and agents can directly read, write, move, and delete those files. + +**File as Memory**: each file is more than ordinary text. It is an indexable, linkable, and evolvable memory node. ReMe parses +frontmatter, body chunks, and wikilink edges from files and organizes them into retrieval indexes and a graph. + +In other words, files are both a human-readable interface and an operational interface for agents. Directory structure +carries the memory layers, while Markdown syntax expresses content, metadata, and relationships. + +## Design Goals + +ReMe represents memory as files not merely for convenient storage, but to give long-term memory several essential properties: + +| Goal | Meaning | +|---|---| +| Readable | Users can open the workspace directly and read daily notes, digest nodes, and source material like ordinary notes. | +| Editable | Users and agents can correct, extend, move, or delete memory with file operations, without a specialized database client. | +| Traceable | Long-term conclusions in digest can point back to daily, resource, or session sources through `derived_from:: [[...]]`. | +| Portable | The workspace is an ordinary directory. Markdown, JSONL, YAML, and resource files can be backed up, synchronized, versioned, or moved to other tools. | +| Indexable | Although the files are plain text, ReMe parses frontmatter, chunks, and wikilinks to build a retrieval index and file graph. | +| Collaborative | Humans judge and correct; agents organize, link, and retrieve. Both operate on the same files. | + +ReMe memory is therefore neither a hidden database record nor a prompt fragment visible only to an LLM. It is first a file +owned by the user and only then indexed by the system for retrieval. + +## Memory Layers + +A ReMe workspace divides memory into four layers: + +```text +raw input -> session/ + resource/ +working memory -> daily/ +long memory -> digest/ +system state -> metadata/ ``` + +Each layer solves a different problem. + +`session/` and `resource/` preserve raw input. Their purpose is to retain the original situation: conversations, agent +sessions, uploaded material, web pages, and reports remain intact as evidence for later verification. + +`daily/` is the lightly processed layer. It organizes the day's conversations and resources into more readable daily notes: +what happened, which conclusions were reached, which follow-up tasks remain, and where the source material lives. Daily does +not aim for final abstraction; it is closer to a workbench for the day. + +`digest/` is the deeply processed layer. It stores memory nodes that can be reused over time, such as user preferences, +project background, procedural experience, conceptual knowledge, and decision precedents. Digest should not merely copy +daily. It should merge recurring facts, methods, and relationships into more stable descriptions. + +`metadata/` is the system index layer. It stores runtime state such as the file catalog, chunk index, and graph snapshots. +Users normally do not edit this content manually. The actual editing surface is `daily/`, `digest/`, and, when necessary, +`resource/`. + +These layers let ReMe preserve both the original situation and its abstraction: daily reconstructs what happened, while +digest answers what remains reusable later. + +## Directory Layout + +ReMe uses directories to express memory organization and layers. Source material first enters `resource/` or `session/`, +then flows into `daily/`, and is finally integrated into `digest/` by `auto_dream`. + +The corresponding automatic flows are [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md), and +[Auto Dream](./auto_dream.md). Use [Memory Search](./memory_search.md) to retrieve these files. + +```text +/ +β”œβ”€β”€ metadata/ # system index layer; persistent indexes, graph, catalogs; not a manual editing surface +β”œβ”€β”€ session/ # raw input layer; original conversations and agent sessions +β”‚ β”œβ”€β”€ dialog/ +β”‚ β”‚ └── .jsonl # conversation messages saved by auto_memory +β”‚ β”œβ”€β”€ agentscope/ +β”‚ β”‚ └── .jsonl +β”‚ └── claude_code/ +β”‚ └── .jsonl +β”œβ”€β”€ resource/ # raw input layer; original external material +β”‚ └── YYYY-MM-DD/ +β”‚ └── . +β”œβ”€β”€ daily/ # lightly processed layer; facts, conversation summaries, and resource interpretations by date +β”‚ β”œβ”€β”€ YYYY-MM-DD.md # index page for the day +β”‚ └── YYYY-MM-DD/ +β”‚ β”œβ”€β”€ .md # daily note distilled from a conversation +β”‚ β”œβ”€β”€ .md # daily note distilled from a resource +β”‚ └── interests.yaml # proactive interest topics generated by auto_dream +└── digest/ # deeply processed layer; reusable personal facts, procedures, and knowledge nodes + β”œβ”€β”€ personal/ + β”‚ └── .md # user profile, preferences, and durable personal facts + β”œβ”€β”€ procedure/ + β”‚ └── .md # procedures, methods, and operational experience + └── wiki/ + └── .md # general knowledge, concepts, and decision precedents +``` + +Typical flows: + +```text +conversation + -> session/dialog/.jsonl + -> daily/YYYY-MM-DD/.md + -> digest/personal | digest/procedure | digest/wiki + +external resource + -> resource/YYYY-MM-DD/. + -> daily/YYYY-MM-DD/.md + -> digest/wiki | digest/procedure +``` + +The first two steps focus on recording and organizing; the final step focuses on long-term distillation. `auto_memory` and +`auto_resource` generate daily notes from raw input, and `auto_dream` extracts and integrates digest nodes from daily. + +## Markdown Format + +ReMe favors Markdown for memory because it works well for human reading, agent editing, and programmatic parsing. + +A typical memory file: + +```markdown +--- +name: Solar Supply Chain Research +description: An end-to-end view from polysilicon to modules +tags: [new energy, solar] +--- + +# Conclusions + +The solar supply chain consists of [[digest/wiki/polysilicon.md]], wafers, cells, and modules. + +upstream:: [[digest/wiki/polysilicon.md]] +[company:: [[digest/wiki/longi.md|LONGi]]] +``` + +### Frontmatter + +Frontmatter is a YAML block at the beginning of a file, enclosed by `---`: + +```markdown +--- +name: Document name +description: Document description +source_conversation: [[session/dialog/abc.jsonl]] +--- +``` + +The current code recognizes `name` and `description` explicitly. Other fields are preserved as additional metadata. The write +interface merges `name`, `description`, and `metadata` into frontmatter. + +Treat frontmatter as a node-level summary and the body as evidence, explanation, and relationships. For example: + +```markdown +--- +name: "User preference: documentation style" +description: The user prefers direct, engineering-oriented technical explanations with context but without unnecessary length. +kind: preference +confidence: observed +--- + +The user repeatedly asks documentation to explain motivation, boundaries, and examples while avoiding marketing language. + +derived_from:: [[daily/2026-06-20/session-a.md]] +related:: [[digest/procedure/technical-documentation.md]] +``` + +This has three benefits: + +1. `name` and `description` serve as lightweight summaries in lists, recall results, and agent decisions. +2. The body can carry fuller facts, conditions, counterexamples, and sources. +3. Typed wikilinks such as `derived_from::` and `related::` can be parsed by the graph and maintained when files move. + +Frontmatter is best for stable, short, structured fields; the body is best for explanations meant for people. Do not put long +body text into YAML fields. + +### Wikilink + +Wikilinks express relationships between files with `[[...]]`: + +```text +[[digest/wiki/solar.md]] +[[digest/wiki/solar.md#supply-chain]] +[[digest/wiki/solar.md|solar]] +![[resource/2026-06-01/report.md]] +``` + +ReMe wikilinks use **literal path semantics**: + +```text +[[X]] -> target_path = "X" +``` + +ReMe does not append `.md` automatically, search by filename, or automatically resolve folder notes. Use complete +workspace-relative paths with their extensions. + +Wikilinks support these behaviors: + +```text +body link -> create a FileLink +predicate:: link -> create a FileLink with a relationship name +move a file -> rewrite [[old path]] in inbound edges by default +delete a file -> return remaining inbound edges so references can be cleaned up +search match -> expand inbound and outbound links to provide context +``` + +Supported relationship forms: + +```markdown +industry:: [[digest/wiki/new-energy.md]] +[competitor:: [[digest/wiki/byd.md]]] +``` + +Parsed result: + +```text +FileLink + source_path = current file + target_path = digest/wiki/new-energy.md + predicate = industry +``` + +### Sources and Relationships + +The two most important link types in ReMe are source links and conceptual relationship links. + +A source link explains where a long-term memory came from: + +```markdown +derived_from:: [[daily/2026-06-20/session-a.md]] +derived_from:: [[resource/2026-06-20/report.pdf]] +``` + +A conceptual relationship link explains which other long-term memories relate to the node: + +```markdown +related:: [[digest/wiki/solar-supply-chain.md]] +depends_on:: [[digest/procedure/research-report-analysis.md]] +contrasts_with:: [[digest/wiki/central-inverter.md]] +``` + +Ordinary body wikilinks also create graph edges, but when the relationship itself has semantic value, prefer +`predicate:: [[path]]`. This makes the meaning of links clearer to search, graph traversal, and later agent integration. + +## Human and Agent Editing + +Because memory is stored as files, users can edit the workspace directly, while agents can read and write the same files +through ReMe's file tools. Both follow the same conventions: + +| Operation | Guidance | +|---|---| +| Add memory | Write to the appropriate directory, use frontmatter for Markdown, and prefer complete workspace-relative wikilinks. | +| Edit a body | Preserve existing sources and important wikilinks. When correcting an old conclusion, explain how the new material changes the previous judgment. | +| Move a file | ReMe's move tool rewrites old paths in inbound edges by default. After a manual move, inspect inbound links again. | +| Delete a file | Check inbound links first. ReMe's delete tool returns source files that still point to the target, making dangling references easier to clean up. | +| Edit metadata | Use frontmatter for short fields. When the body changes substantially, update `description` as well. | + +A practical rule is: **an agent may rewrite the wording, but it must not lose evidence edges**. In particular, +`derived_from:: [[...]]` and existing digest-to-digest wikilinks are the basis for traceable and extensible long-term memory. + +## Path Semantics + +All file tools and wikilinks use workspace-relative paths as their basic unit: + +```text +digest/wiki/solar.md +daily/2026-06-20/session-a.md +resource/2026-06-20/report.pdf +``` + +This creates a clear boundary: ReMe does not treat `[[solar]]` as a repository-wide title search and does not assume +Obsidian-style same-name resolution. `[[digest/wiki/solar.md]]` points to that exact path. + +Recommended practices: + +1. Include `.md` when linking a Markdown file. +2. Use the complete source path when linking from digest to daily or resource. +3. Rename or move files through ReMe's move tool whenever possible to avoid stale paths. +4. Put external source material under `resource/YYYY-MM-DD/...` and long-term abstractions under `digest/...`. Do not put + raw source material directly into digest. + +Explicit path semantics sacrifice a little convenience when writing by hand, but provide predictability, portability, and +automatic maintainability. + +## Memory Chunking + +Memory chunking divides a file into retrievable fragments. ReMe does not split Markdown at fixed lengths by default; it tries +to preserve semantic structure. + +This section explains how files become retrieval chunks. For index updates, BM25, vector recall, and link expansion, see +[Memory Search](./memory_search.md). + +Traditional RAG often uses fixed-window splitting: + +```text +Document + | + | every N tokens + overlap + v +chunk 1 | chunk 2 | chunk 3 | ... +``` + +This is simple, but it can cut headings, tables, code blocks, lists, and `[[wikilinks]]` in the middle. After a match, the +agent often sees only an isolated fragment without knowing its section or relationship to other memory nodes. + +ReMe chunking is closer to splitting memory by file structure: + +```text +Markdown file + | + | frontmatter + headings + blocks + wikilinks + v +semantic chunks with document skeleton +``` + +Comparison: + +```text +traditional RAG chunk + = fixed-length text fragment + overlap + +ReMe memory chunk + = section structure + body fragment + line range + wikilink relationship context +``` + +Markdown files use `MarkdownFileChunker`: + +```text +Markdown + | + | mistletoe AST + v +Document + └─ H1 section + β”œβ”€ paragraph / list / table / code + └─ H2 section + └─ ... + | + v +FileChunk[] +``` + +Chunking rules: + +```text +1. Parse frontmatter first; send the body to the chunker separately. +2. Build a section tree from heading levels. +3. Prefer one complete section per chunk. +4. When a section is too long, recursively split its subsections and body blocks. +5. Repeat table headers when splitting tables. +6. Repeat the fence when splitting code blocks. +7. Pack lists by item. +8. Only then split greedily by line and add [Part X/N]. +``` + +By default, every chunk includes its heading skeleton: + +```text +# Top-level heading + +## Current section + +Matched body fragment + +## Following section heading +``` + +This lets the agent see not only an isolated paragraph but also its structural position in the source file. + +Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For Markdown, +the chunker also avoids cutting `[[wikilinks]]` in the middle. diff --git a/docs/en/memory_search.md b/docs/en/memory_search.md index 0a6dc3bc..529035ae 100644 --- a/docs/en/memory_search.md +++ b/docs/en/memory_search.md @@ -1,7 +1,207 @@ # Memory Search -```{note} -πŸ“– The English version of this page is in progress. +Memory Search is ReMe's memory retrieval entry point. It continuously builds files under `daily/`, `digest/`, and `resource/` +into a searchable chunk index and wikilink graph. At query time, it first recalls the most relevant fragments and then expands +context along the bidirectional links of the files containing those fragments. -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +

+ ReMe Auto Index and Memory Search indexing, recall, fusion, and link expansion +

+ +For the general semantics of file layers, frontmatter, wikilinks, and chunking, see +[Memory as File](./memory_as_file.md). This page focuses on index maintenance and query execution. + +```text +workspace files + β”œβ”€ index_update_loop: detect added / modified / deleted + β”œβ”€ update_index_step: file -> FileNode + FileChunk[] + β”œβ”€ file_store: store chunks, BM25, optional embeddings, and the wikilink graph + └─ search_step: BM25 / vector recall -> RRF fusion -> link expansion ``` + +## What It Searches + +The default `index_update_loop` watches three memory directories: + +- `daily_dir`: daily working memory and session memory cards generated by Auto Memory. +- `digest_dir`: long-term distilled digest nodes. +- `resource_dir`: external resources or imported material. + +The default suffixes are `md` and `jsonl`. Markdown uses the `markdown` chunker, which parses frontmatter, heading structure, +and `[[wikilinks]]`. JSONL uses the `default` chunker and creates overlapping chunks by byte size. + +## How the Index Is Built + +The background Job `index_update_loop` maintains the index using configuration from `reme/config/default.yaml`: + +```yaml +index_update_loop: + backend: background + watch_dirs: [ daily_dir, digest_dir, resource_dir ] + watch_suffixes: [ md, jsonl ] + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [ update_index_step ] + - backend: watch_changes_step + dispatch_steps: [ update_index_step ] +``` + +`init_changes_step` runs at startup. It scans the watched directories, compares file mtimes on disk with +`FileNode.st_mtime` values already stored in `file_store`, calculates added, modified, and deleted changes, and passes +`context["changes"]` to `update_index_step`. + +While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same directories, +groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the same path into one +stable batch of changes. + +`update_index_step` performs the actual index writes: + +1. Select a file chunker by suffix. +2. Parse the file into one `FileNode` and multiple `FileChunk` objects. +3. For an added or modified file, delete its old chunks before upserting the new chunks. +4. For a deleted file, remove its records from `file_store`, `keyword_index`, and `file_graph`. +5. When changes exist, dump state to `metadata/` so it can be restored on the next startup. + +The Markdown chunker parses YAML frontmatter, heading structure, and `[[...]]` into `FileNode`, `FileChunk`, and `FileLink` +objects. For detailed chunking rules, see [Memory as File](./memory_as_file.md#memory-chunking). + +## What file_store Contains + +The default `file_store.default` backend is `local`: + +```yaml +file_store: + default: + backend: local + embedding_store: "" + keyword_index: default + file_graph: default +``` + +It combines three kinds of capability: + +| Part | Default state | Purpose | +|---|---|---| +| `file_chunks` | Enabled | Store `FileChunk` text, line numbers, scores, and optional embeddings. | +| `keyword_index.default` | Enabled | BM25 inverted index where chunk ID is the document ID. | +| `file_graph.default` | Enabled | Store `FileNode` objects and wikilink edges. | +| `embedding_store` | Disabled | When enabled, generate embeddings for chunks and support vector recall. | + +Out of the box, search therefore uses primarily BM25 plus link expansion. After setting `embedding_store: default`, +`SearchStep` runs vector and keyword recall together. + +## How to Search + +The `search` Job is also configured in `default.yaml`: + +```yaml +search: + backend: base + description: "Hybrid workspace search (vector + BM25, RRF-fused)." + parameters: + query: string + limit: integer + min_score: number + steps: + - backend: search_step + vector_weight: 0.7 + candidate_multiplier: 3.0 + expand_links: true + max_links_per_direction: 10 +``` + +Call it with: + +```bash +reme search query="recent discussions about indexing" limit=5 +``` + +`search_step` executes in this order: + +```mermaid +flowchart LR + A["query + limit"] --> B["candidates = limit * candidate_multiplier"] + B --> C["file_store.vector_search(...)"] + B --> D["file_store.keyword_search(...)"] + C --> E["RRF fusion"] + D --> E + E --> F["min_score filter"] + F --> G["truncate to limit"] + G --> H["expand_links(...)"] + H --> I["Response.answer + metadata"] +``` + +If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking is +returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores directly; it +compares ranks in the two result lists: + +```text +fused_score = vector_weight / (60 + vector_rank) + + keyword_weight / (60 + keyword_rank) +``` + +The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can still +promote chunks with exact term matches. + +## How BM25 Works + +`keyword_search()` calls `keyword_index.retrieve(query, limit)`. Each chunk is a document in the BM25 index: + +- `doc_id` is `FileChunk.id`. +- `content` is `FileChunk.text`. +- The tokenizer splits text into tokens. +- The inverted index records which chunks contain each token and its term frequency within each chunk. +- A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs. + +When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old `chunk_ids` +and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize. + +## Progressive Expansion + +"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three layers: + +1. Chunk recall: return only the `limit` most relevant text fragments. +2. File location: each result includes `path:start_line-end_line`, allowing the caller to read the source precisely with `read`. +3. Link neighbors: call `expand_links()` for each matched file and expand at most `max_links_per_direction` outlinks and + inlinks. + +Expansion data comes from `file_graph` rather than rescanning files: + +```text +matched chunk + -> chunk.path + -> file_store.get_outlinks(path) + -> file_store.get_inlinks(path) + -> file_store.get_nodes(neighbor_paths) + -> render neighbor path, name, description, predicate, and anchor +``` + +This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory connects +to. If a result is worth pursuing, use `read path=...` to open the source or +`traverse path=... depth=2` to continue along the wikilink graph. + +## Return Format + +`SearchStep` writes results in two places: + +- `response.answer`: human-readable text. Each matched block contains its path, line numbers, score, and chunk content, + followed by outlinks and inlinks. +- `response.metadata`: structured programmatic results containing `results`, `link_expansion`, and `counts`. + +Typical text structure: + +```text +========== daily/2026-06-20/session-a.md:12-28 [score=0.0317 keyword=4.8120] ========== +...matched memory fragment... + outlinks (2): + -> digest/indexing.md name="Indexing" description="..." + via predicate=related + inlinks (1): + <- daily/2026-06-19.md name="..." + via plain +``` + +`counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned. With +embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`. diff --git a/docs/en/proactive.md b/docs/en/proactive.md index 499fba83..764db886 100644 --- a/docs/en/proactive.md +++ b/docs/en/proactive.md @@ -1,7 +1,138 @@ # Proactive -```{note} -πŸ“– The English version of this page is in progress. +`proactive` is ReMe's interface for reading proactive memory. It does not reanalyze daily notes or call an LLM. It only reads +the current day's interest topics written by `auto_dream`: -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +```text +daily//interests.yaml ``` + +A host agent can use it to learn "what is worth proactive attention today," then decide whether to remind the user, ask a +follow-up question, recommend a next step, or produce a proactive insight. + +`interests.yaml` is generated by the Topics stage of [Auto Dream](./auto_dream.md). `proactive` only reads and exposes the +result. + +## Configuration + +The default configuration is in `reme/config/default.yaml`: + +```yaml +proactive: + backend: base + description: "Proactive: read daily//interests.yaml and expose the latest user-interest topics." + parameters: + date: + type: string + default: "" + include_content: + type: boolean + default: true + steps: + - backend: proactive_step +``` + +Parameters: + +| Parameter | Purpose | +|---|---| +| `date` | Date to read in `YYYY-MM-DD` format. When empty, use today in the application's timezone. | +| `include_content` | Whether to return the raw YAML in metadata. Defaults to `true`. | + +## Input Contract + +A typical file looks like this: + +```yaml +date: 2026-06-20 +topic_count: 3 +diversity_days: 7 +topics: + - title: Quality regression in the memory retrieval pipeline + reason: The user has recently made repeated changes to search, node_search, and dream integration. + evidence: daily/2026-06-20/session.md + keywords: + - memory search + - auto dream + paths: + - daily/2026-06-20/session.md +``` + +Only the `topics` list is parsed into structured results. Every topic requires at least `title` and `reason`; +`evidence`, `keywords`, and `paths` are supporting fields. + +## Return Value + +When the file is read successfully, `proactive_step` writes these values to standard response metadata: + +| Field | Description | +|---|---| +| `date` | The date actually read. | +| `path` | `daily//interests.yaml`. | +| `topics` | Parsed topic list. | +| `content` | Raw YAML; returned only when `include_content=true`. | +| `skipped` | `true` when the file does not exist. | +| `error` | Read or parse error. | +| `summary` | Short summary. | + +When the file exists and parses successfully, the answer looks like: + +```text +Read 3 proactive topic(s) from daily/2026-06-20/interests.yaml +``` + +A missing file is not an error. The call succeeds with a skipped result: + +```text +Skipped: interests file not found at daily/2026-06-20/interests.yaml +``` + +This lets a host agent treat "there is no dream result for today yet" as a normal empty state. + +## Running Proactive + +CLI: + +```bash +reme proactive date=2026-06-20 +``` + +Omit the raw YAML content: + +```bash +reme proactive date=2026-06-20 include_content=false +``` + +## Relationship to auto_dream + +`proactive` is the downstream read step for `auto_dream`: + +```text +daily notes + -> auto_dream + -> daily//interests.yaml + -> proactive + -> host agent +``` + +The responsibilities are divided as follows. For the complete Extract, Integrate, Topics, and Finish flow, see +[Auto Dream](./auto_dream.md): + +| Module | Responsibility | +|---|---| +| `dream_extract_step` | Extract topic candidates from changed daily inputs. | +| `dream_topics_step` | Deduplicate, select, and write `interests.yaml`. | +| `proactive_step` | Read `interests.yaml` and expose it to the host agent. | + +`proactive` does not modify files, update a catalog, or decide whether the user should be interrupted. It only provides the +day's topic material. The caller's product policy determines whether, when, and in what tone to push it to the user. + +## Failure Modes + +| Scenario | Behavior | +|---|---| +| `interests.yaml` does not exist | `success=true`, `skipped=true`, `topics=[]`. | +| YAML cannot be read or parsed | `success=false`; the answer contains an error summary. | +| YAML exists but has no valid topics | `success=true`, `topics=[]`. | + +Callers should therefore check `success` first, then `skipped`, and finally whether `topics` is empty. diff --git a/docs/en/quick_start.md b/docs/en/quick_start.md index 26fa3603..d0e31036 100644 --- a/docs/en/quick_start.md +++ b/docs/en/quick_start.md @@ -1,7 +1,214 @@ # Quick Start -```{note} -πŸ“– The English version of this page is in progress. +## Installation -In the meantime, please read the δΈ­ζ–‡η‰ˆζœ¬. +ReMe requires Python 3.11+. + +Install from pip: + +```bash +pip install "reme-ai[core]" +``` + +Install from source: + +```bash +git clone https://github.com/agentscope-ai/ReMe.git +cd ReMe +pip install -e ".[core]" +``` + +Installing the `core` extra is recommended. The current code imports the AgentScope wrapper, and self-evolving memory also +depends on it. + +To use agent workflows such as `auto_memory`, `auto_resource`, and `auto_dream`, configure an LLM: + +```bash +cat > .env <<'EOF' +LLM_BACKEND=openai +LLM_MODEL_NAME=qwen3.7-plus +LLM_API_KEY=your_api_key +LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 +EOF +``` + +You can initially omit the LLM configuration if you only need basic file operations and BM25 retrieval. + +--- + +## Start the Service + +```bash +reme start +``` + +The default service address is `127.0.0.1:2333`. If the port is already in use: + +```bash +reme start service.port=8181 +``` + +```bash +reme version +reme health_check +reme list +``` + +`reme list` lists server actions. Ordinary commands invoke server Jobs over HTTP. + +--- + +## Workspace Layout + +The default workspace is `.reme/` under the current directory. It is created automatically at startup: + +```text +.reme/ +β”œβ”€β”€ metadata/ # persistent indexes, graph, catalogs, and related state +β”œβ”€β”€ session/ # agent sessions and original conversations +β”œβ”€β”€ resource/ # external resources +β”œβ”€β”€ daily/ # daily notes +└── digest/ # long-term memory +``` + +For directory layers, Markdown frontmatter, and wikilink semantics, see +[Memory as File](./memory_as_file.md). + +You can also specify the workspace at startup: + +```bash +reme start workspace_dir=/tmp/reme-demo service.port=8181 +``` + +--- + +## Write, Index, and Search + +```bash +reme write \ + path=digest/wiki/quick-start-demo \ + name="Quick Start Demo" \ + description="Example memory for the quick start" \ + content="# Quick Start Demo + +ReMe indexes Markdown under the daily, digest, and resource directories. + +Related link: [[digest/wiki/search-demo.md]]" +``` + +`path` is relative to the workspace. A missing suffix is automatically completed with `.md`. For Markdown files, `name` and +`description` are written to frontmatter. + +The background watcher builds the index automatically. You can also rebuild it manually: + +```bash +reme reindex +``` + +Search: + +```bash +reme search query="quick start example memory" limit=5 +``` + +Read: + +```bash +reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20 +``` + +With the default configuration, retrieval is primarily BM25 plus wikilink graph expansion. Vector retrieval is supported by +the code, but the embedding store is disabled by default. For the full retrieval flow, see +[Memory Search](./memory_search.md). + +--- + +## Files and Daily Notes + +```bash +reme stat path=digest/wiki/quick-start-demo +reme edit path=digest/wiki/quick-start-demo old="indexes" new="continuously indexes" +reme frontmatter_read path=digest/wiki/quick-start-demo +reme frontmatter_update path=digest/wiki/quick-start-demo metadata='{"tags":["demo"]}' +``` + +The name `list` is used by the CLI to list actions, so the file-listing Job must be called over HTTP: + +```bash +curl -s http://127.0.0.1:2333/list \ + -H 'Content-Type: application/json' \ + -d '{"path":"digest","recursive":true,"limit":50}' +``` + +Daily notes: + +```bash +reme write path=daily/2026-06-20/demo-session.md name=demo-session description="Demo session" content="Recorded content" +reme daily_list +reme daily_reindex +``` + +`write` can create a daily note directly. Run `daily_reindex` when the day's index needs to be refreshed. + +--- + +## Automatic Memory + +```bash +reme auto_memory \ + session_id=chat-demo \ + messages='[{"role":"user","content":"I prefer to preserve project experience as Markdown."},{"role":"assistant","content":"Recorded."}]' \ + memory_hint="Record the user's preference" +``` + +After placing external material under `resource/YYYY-MM-DD/`, the default background task watches +`md/txt/json/jsonl/csv/yaml/html`. You can also trigger processing manually: + +```bash +reme auto_resource changes='[{"path":"resource/2026-06-20/report.md","change":"added"}]' +``` + +Distill daily notes into long-term digest memory: + +```bash +reme auto_dream date=2026-06-20 +reme proactive date=2026-06-20 +``` + +These flows require a working LLM. Without an LLM configuration, start with basic capabilities such as `write`, `read`, and +`search`. + +For more detail, see [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md), +[Auto Dream](./auto_dream.md), and [Proactive](./proactive.md). + +--- + +## HTTP and Configuration + +Every service-enabled Job is exposed as `POST /`: + +```bash +curl -s http://127.0.0.1:2333/version \ + -H 'Content-Type: application/json' \ + -d '{}' + +curl -s http://127.0.0.1:2333/search \ + -H 'Content-Type: application/json' \ + -d '{"query":"quick start","limit":5}' +``` + +The default configuration comes from `reme/config/default.yaml`. Override it at startup with dot notation: + +```bash +reme start \ + workspace_dir=/tmp/reme-demo \ + service.host=127.0.0.1 \ + service.port=8181 \ + enable_logo=false +``` + +You can also specify a YAML or JSON configuration file: + +```bash +reme start config=/path/to/custom.yaml ``` diff --git a/plugins/.claude-plugin/marketplace.json b/plugins/claude_code/.claude-plugin/marketplace.json similarity index 100% rename from plugins/.claude-plugin/marketplace.json rename to plugins/claude_code/.claude-plugin/marketplace.json diff --git a/plugins/README.md b/plugins/claude_code/README.md similarity index 100% rename from plugins/README.md rename to plugins/claude_code/README.md diff --git a/plugins/reme/.claude-plugin/plugin.json b/plugins/claude_code/reme/.claude-plugin/plugin.json similarity index 100% rename from plugins/reme/.claude-plugin/plugin.json rename to plugins/claude_code/reme/.claude-plugin/plugin.json diff --git a/plugins/reme/.mcp.json b/plugins/claude_code/reme/.mcp.json similarity index 100% rename from plugins/reme/.mcp.json rename to plugins/claude_code/reme/.mcp.json diff --git a/plugins/reme/hooks/auto_memory.py b/plugins/claude_code/reme/hooks/auto_memory.py similarity index 100% rename from plugins/reme/hooks/auto_memory.py rename to plugins/claude_code/reme/hooks/auto_memory.py diff --git a/plugins/reme/hooks/hooks.json b/plugins/claude_code/reme/hooks/hooks.json similarity index 100% rename from plugins/reme/hooks/hooks.json rename to plugins/claude_code/reme/hooks/hooks.json diff --git a/plugins/reme/skills/reme-memory/SKILL.md b/plugins/claude_code/reme/skills/reme-memory/SKILL.md similarity index 100% rename from plugins/reme/skills/reme-memory/SKILL.md rename to plugins/claude_code/reme/skills/reme-memory/SKILL.md