From 643d36a4b6c5df2ab569da72322c35787dbc8f79 Mon Sep 17 00:00:00 2001 From: huangsen Date: Mon, 1 Jun 2026 16:39:52 +0800 Subject: [PATCH] refactor(steps): remove deprecated digest edit/write steps --- .../reme-service/skills/reme-service/SKILL.md | 147 -- reme4/components/as_llm/__init__.py | 3 +- reme4/components/job/base_job.py | 3 - reme4/steps/__init__.py | 4 - reme4/steps/evolve/dream/__init__.py | 24 +- reme4/steps/evolve/dream/digest_edit.py | 125 -- reme4/steps/evolve/dream/digest_write.py | 140 -- reme4/steps/evolve/dream/dreamer.py | 410 ++--- reme4/steps/evolve/dream/dreamer.yaml | 1449 +++++++++++------ tests4/integration/_dreamer_fixture.py | 57 +- tests4/integration/test_dreamer_inproc.py | 25 +- 11 files changed, 1206 insertions(+), 1181 deletions(-) delete mode 100644 reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md delete mode 100644 reme4/steps/evolve/dream/digest_edit.py delete mode 100644 reme4/steps/evolve/dream/digest_write.py diff --git a/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md b/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md deleted file mode 100644 index d29b98a8..00000000 --- a/reme-plugin/plugins/reme-service/skills/reme-service/SKILL.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -name: reme-service -description: Use this skill whenever the user references their personal vault (markdown notes managed by the `reme` MCP, service-tier surface, backed by reme4), or when there's a meaningful session outcome to record / a question that prior work might answer. Triggers include "what do I know about X", "did I work on Y before", "save this", "记下", "落盘", "提炼", "vault", any mention of resource/ / daily/ / digest/ files, or recognizing that a non-trivial session outcome should be recorded. Skill follows a 3-phase paradigm (Recall / Log / Distill) over a 4-tier lifecycle (external channel → resource → daily → digest). Log + Distill route to two SERVICE LAYER MCP tools — `synchronizer` and `digester` — whose internal ReActAgents run the LLM loop INSIDE reme4. Inbound assets from external channels land via `upload` into `resource//` (service-only). All other tools (search / traverse / file_list / file_read / file_write / file_append / file_stat / frontmatter_*) are shared atomic primitives — whole-file CRUD covers all body changes; `frontmatter` is the one sliced RUD surface (YAML is structured data — surgical key edits cannot be safely emulated with string-substitution on the body). ---- - -# vault — service tier (reme4) - -The vault is a personal markdown knowledge base managed by the `reme` MCP server. **Service tier**: two service-layer LLM-driven tools (`synchronizer`, `digester`) + the service-only resource-ingest primitive (`upload`) + the full shared atomic primitive surface. Log + Distill phases hand off to the service layer; the R-M-W loops run **inside reme4** in those tools' internal ReActAgents. - -## Business objects - -- **Resource bucket** — passive ingest from external channels. `resource//` is a flat folder keyed by the day the asset was received, containing the assets themselves (any file type), a `meta.json` array of provenance rows (channel / source / received_at / description), and a derived `.md` view assembled from meta.json. **One ingest path only**: the `upload` tool. Read-only for everything else (synchronizer / digester / hand-edits never write here). -- **Daily note** — hot, streaming fact log of one thread. Single file `daily//.md`; everything worth keeping (verbatim user prompt, key tool output, intermediate data) inlined inside the body. One upstream writer per note; every other consumer treats it as read-only. Inbound channel assets do NOT live here — those go to the resource bucket and are referenced via `[[resource//]]` wikilinks in the note's `## References` section when the task consumes them. -- **digest node** — cold, curated long-lived cognition. `digest//.md` (or nested at any depth: `digest///.md`). Each scope folder must contain `/.md` as its canonical entry. **Slugs are globally unique under `digest/`** — a folder name appears at most once anywhere in the tree. - -Lifecycle: **external channel → `upload` → resource// → session work + daily folder → distill → digest node**. Each tier is one-way downstream. Inbound assets are passive (someone sends you a file); daily materials are active (you fetched / produced them during a task); digest entries are distilled cognition. The distill marker is the daily's `status` frontmatter — a **daily-tier convention owned by the Digester** (reme core reserves only `name` / `description`; `status` is an extra used exclusively by Sync/Digester). Convention: absent ≡ `pending`; the Digester flips it to `completed` (or `skipped`) once it has processed the daily. Find unprocessed work by listing `daily/` and `frontmatter_read`-ing each summary — the ones with no `status` are pending. - -## Tool surface (service tier) - -| Group | Tools | Where the work runs | -|---|---|---| -| **Service layer (LLM-driven)** | `synchronizer`, `digester` | **Inside reme4** — internal ReActAgent | -| **Service-only ingest** | `upload` (external channel → `resource//`) | reme4 thin primitive (no LLM) | -| Shared retrieve | `search`, `traverse` | reme4 thin primitive (no LLM) | -| Shared read | `file_list`, `file_read`, `file_stat`, `frontmatter_read` | reme4 thin primitive (no LLM) | -| Shared write | `file_write`, `file_append`, `file_edit`, `frontmatter_update`, `frontmatter_delete` | reme4 thin primitive (no LLM) | -| Shared file ops | `file_move`, `file_delete`, `file_download` | reme4 thin primitive (no LLM) | -| Shared daily | `daily_read`, `daily_write`, `daily_list`, `daily_reindex` | reme4 thin primitive (no LLM) | - -The shared block is identical to expert tier; what makes this **service** tier is the two service-layer tools at the top plus the service-only `upload` ingest primitive. - -## 3-Phase paradigm - -### Phase 1: Recall - -**What** — retrieve relevant context (chunks ranked by RRF-fused vector + BM25 score, with optional wikilink expansion via the file graph). -**Triggers** — intent-driven only: "what do I know about X" / "did I work on Y" / "what's connected to [[Z]]" / task needs prior methodology. -**How** — -- `search(query, limit?, expand_links?, ...)` for hybrid chunk retrieval. -- `traverse(path, depth?, direction?)` to chase a seed file's wikilink neighborhood. -- `file_list` / `file_read` / `file_stat` / `frontmatter_read` for primary-key reads. - -``` -search query="auth refactor decisions" limit=5 -search query="see [[张三.md]]" -traverse path="digest/zhang-san/zhang-san.md" depth=1 -``` - -### Phase 2: Log (service layer) - -**What** — digest the recent conversation slice into a daily note. The Synchronizer's internal ReActAgent picks a slug, writes the note (everything inlined into a single file), and handles continuation (same slug → same file). -**Triggers** — (a) intent-driven: meaningful fact / output / decision just landed; (b) **PreCompact hook**: prompt fires to dump volatile state. -**How** — `synchronizer(messages, note?)`. - -``` -synchronizer - messages: - - {role: user, content: "let's design the auth refactor"} - - {role: assistant, content: "two options: JWT vs session..."} - - {role: user, content: "go with JWT + refresh token rotation"} - note: "auth refactor" # optional hint to bias the slug -``` - -The Synchronizer reads the conversation, picks a stable slug (or reuses an existing one when `note` matches), writes `daily//.md`, and returns a `SynchronizerResult` audit (`note` path, `summary` of the just-written note, `actions`). Surface the summary verbatim if the user wants to see what landed. - -**Surgical edits** (without going through Synchronizer's LLM loop): -- `daily_read(slug, date?)` to probe / merge — returns body in `answer` and the parsed frontmatter dict in metadata. `exists: false` = fresh, `exists: true` = upsert. -- `daily_write(slug, body, frontmatter?, date?, overwrite?)` for a full-note write — `overwrite=false` (default, idempotent skip-if-exists; mirrors the old `daily_resolve` probe) for fresh threads, `overwrite=true` for UPDATE after a `daily_read`. Auto-mkdirs the day folder and refreshes the day index. -- `file_append(path, content)` for cheap end-of-file extensions to trailing sections (`## Progress`, `## Findings`, `## Decisions`) — saves the read-modify-write round-trip. -- `frontmatter_update(path, metadata={key: value, ...})` to merge one or more frontmatter keys (call `daily_reindex` afterward if you touched `name` / `description`). -- `frontmatter_delete(path, keys=[...])` to drop frontmatter keys. -- For mid-body edits on a daily note, `daily_read` then `daily_write overwrite=true`. For non-daily paths, `file_read` then `file_edit` (string substitution) or `file_write` (full body) — there's no body/section slice tool; YAML is the only structured surface that earns its own RUD package. - -Use these when you know exactly what to write; use `synchronizer` when you want the service layer to decide what's worth keeping from the conversation. - -### Phase 3: Distill (service layer) - -**What** — promote daily notes into the digest knowledge graph. The Digester's internal ReActAgent reads each daily note (single-file inline content), looks up existing digest nodes (globally unique slugs — same slug at any nesting depth is the same node), applies the R-M-W decision rules (CREATE / UPDATE / MOVE; mere mentions with no own-node substance are left as-is). Relations are recorded as typed wikilinks in the source node's body (`predicate:: [[X.md]]`); target bodies are never edited (backlinks come from `traverse direction=backward` at query time). After each daily note is processed, the Digester flips its `status` frontmatter to `completed` (or `skipped` if nothing was lifted) — **that flip IS the distill marker** (a daily-tier convention the Digester owns; absent ≡ pending). The Digester scans `daily/` and `frontmatter_read`s each note, picking the ones whose `status` is absent. -**Triggers** — (a) intent-driven: task wraps and the working set is ready; (b) **SessionEnd hook**: prompt fires to call `digester` once. -**How** — `digester(daily_paths, hint?)`. - -``` -digester - daily_paths: - - daily/2026-05-17/auth-refactor - - daily/2026-05-17/perf-bench - hint: "End-of-task distillation — focus on the auth decisions; perf-bench is a methodology dump." -``` - -Returns a `DistillResult` (`used_llm`, `skipped`, `daily_read`, `summary`, `error`). Surface the `summary` verbatim. - -**Cold-path rule**: handoff once at task wrap, not per turn. - -## Inbound channel ingest (outside the 3-phase loop) - -When the user hands you an externally-received asset (file from wechat / email / browser / api / ...), land it in the resource bucket before doing anything else: - -``` -upload - path: /tmp/report-q1.pdf - channel: wechat - source: design-group - description: Q1 sales report -``` - -The tool copies the file into `resource//`, appends a `ResourceEntry` to that day's `meta.json`, and regenerates `resource//.md`. Returns `{date, name, path}` — surface `path` so the user knows where the asset landed. If a downstream task consumes the asset, reference it from the daily note's References section with `[[resource//]]` (the canonical resource path) rather than inlining the file. - -Triggers — user phrases like "save this file", "上传这个", "存一下刚收到的", or a channel hook that hands you an inbound payload. - -## Trigger → Phase quick reference - -| Trigger | Phase | What you do | -|---|---|---| -| User hands you an inbound asset from an external channel | Ingest | `upload(path=..., channel=..., source=?, description=?)` | -| User asks about prior work / [[X]] | Recall | `search` / `traverse` | -| Fact lands during task | Log | `synchronizer(messages=[...])` | -| Surgical edit needed | Log | `daily_read` / `daily_write` / `file_append` / `frontmatter_update` / `frontmatter_delete` / `daily_reindex` | -| **PreCompact hook** fires | Log (urgent dump) | `synchronizer(messages=[...], note=...)` | -| Task wraps | Distill | `digester(daily_paths=[...])` | -| **SessionEnd hook** fires | Log + Distill | `synchronizer` then `digester` | - -## Protocol (the rules every write must respect) - -@../../../../protocol.md - -## Anti-patterns - -- ❌ Picking a fresh `note` slug on every `synchronizer` call within the same logical thread → fragments the thread. **Reuse the slug.** -- ❌ Calling `digester` per turn → it's a handoff tool, not a per-turn tool. Once at end-of-task is the rule. -- ❌ Calling `digester` on a daily whose `status` is already `completed` or `skipped` → it'll be a no-op; don't keep re-pushing. (The Digester's own pending scan — `file_list` + per-item `frontmatter_read` — filters those out for you.) -- ❌ Creating a new `digest/X/x.md` when `X` already exists somewhere else under `digest/` (e.g. `digest/people/X/x.md`) — slugs are globally unique; reuse the existing node and fold the new facts in. -- ❌ Manually `file_write`-ing under `digest/` instead of going through `digester` — digest nodes are the Digester's domain. (You can still do it for one-off corrections; just don't bypass the service layer for routine distillation.) -- ❌ Writing `status` from outside the Digester — `status` is a Digester-owned daily-tier convention (enum `pending` / `completed` / `skipped`; absent ≡ pending). Flipping it from a hand-written tool call makes the note look already-processed (the Digester's pending scan skips it) and the next `digester` invocation never picks it up. -- ❌ Using `file_write` (full-file replacement) to flip one frontmatter key — use `frontmatter_update`. -- ❌ Using `file_write` to extend trailing sections like `## Progress` — use `file_append`; saves the R-M-W round-trip and the prompt tokens of echoing the whole body back. -- ❌ Writing under `resource/` from anything other than `upload` — that bucket is the passive ingest contract. Hand-edits / `synchronizer` / `digester` must never touch it. -- ❌ Inlining an inbound asset into the daily note body — leave it in `resource//` and reference it via `[[resource//]]` in the note's `## References` section. Daily notes are single-file; inbound assets stay in `resource/`. -- ❌ Writing short-form (`[[Alice]]`) or no-extension (`[[k/x]]`) wikilinks — they don't resolve. Always full path relative to the vault with `.md`: `[[digest/alice/alice.md]]`. - -## What you DON'T have to think about - -- Slug uniqueness within a thread — `synchronizer` / `digester` pick paths and reuse existing notes; wikilinks are literal full paths, so two different paths never silently merge. -- Status flips — the Digester writes `status=completed` (or `skipped`) per processed daily note; that frontmatter flag IS the distill marker, so it's never optional but you never write it yourself. -- Frontmatter schema — only `name` / `description` / `status` are reserved (all optional); the protocol defines opinionated default axes (`lifecycle` / `scope` / `source` / `role`) but enforcement is caller-side, not protocol-side. -- Pending-vs-digest bookkeeping — the digester scans `daily/` and `frontmatter_read`s each note to find ones whose `status` is absent; it maintains the queue. - -If you need fine-grained control over every R-M-W decision visible in the main session's tool log, switch to [reme-expert](../reme-expert) — same shared tools, no service layer, plus a subagent that owns the Distill LLM loop in its own context window. diff --git a/reme4/components/as_llm/__init__.py b/reme4/components/as_llm/__init__.py index 0d4b0054..82ae553f 100644 --- a/reme4/components/as_llm/__init__.py +++ b/reme4/components/as_llm/__init__.py @@ -25,8 +25,7 @@ class OpenAIAsLLM(BaseAsLLM): """OpenAI chat model wrapper.""" async def _start(self) -> None: - kwargs = dict(self.kwargs) - self.model = OpenAIChatModel(**kwargs) + self.model = OpenAIChatModel(**self.kwargs) async def _close(self) -> None: if self.model is not None: diff --git a/reme4/components/job/base_job.py b/reme4/components/job/base_job.py index 38a70371..3a414bee 100644 --- a/reme4/components/job/base_job.py +++ b/reme4/components/job/base_job.py @@ -53,9 +53,6 @@ class BaseJob(BaseComponent): raise ValueError(f"Unregistered backend '{config.backend}' of type '{ComponentEnum.STEP}'") params = config.model_dump() params["app_context"] = self.app_context - # Inherit app-level language unless the step's own config overrides it. - if not params.get("language") and self.app_context is not None: - params["language"] = getattr(self.app_context.app_config, "language", "") or "" return step_cls, params def _build_steps(self) -> list["BaseStep"]: diff --git a/reme4/steps/__init__.py b/reme4/steps/__init__.py index dfd361f2..428bd470 100644 --- a/reme4/steps/__init__.py +++ b/reme4/steps/__init__.py @@ -9,8 +9,6 @@ from .common.stream_demo import StreamDemoStep1, StreamDemoStep2 from .common.version import VersionStep from .evolve.auto_memory import AutoMemoryStep from .evolve.dream.cron_dreamer import CronDreamer -from .evolve.dream.digest_edit import DigestEditStep -from .evolve.dream.digest_write import DigestWriteStep from .evolve.dream.dreamer import Dreamer from .file_io.daily_create import DailyCreateStep from .file_io.daily_list import DailyListStep @@ -77,8 +75,6 @@ __all__ = [ "WatchChangesStep", # evolve.dream "CronDreamer", - "DigestEditStep", - "DigestWriteStep", "Dreamer", # transfer "DownloadStep", diff --git a/reme4/steps/evolve/dream/__init__.py b/reme4/steps/evolve/dream/__init__.py index 0565afe0..2691474e 100644 --- a/reme4/steps/evolve/dream/__init__.py +++ b/reme4/steps/evolve/dream/__init__.py @@ -1,19 +1,17 @@ -"""dream — auto-dream pipeline: classify + integrate via constrained digest tools. +"""dream — auto-dream pipeline: extract abstractions, then integrate per +sub-unit using bucket-specific Phase 2 prompts. -Four steps: +Two steps: - dreamer — 2-phase ReAct workflow (extract memory sub-units, - then integrate per sub-unit via the digest tools). - cron_dreamer — daily wrapper around dreamer; scans today's - daily/ + resource/ files and runs dream_one on each. - digest_write_step — constrained WriteStep that creates a new - digest//.md. - digest_edit_step — constrained EditStep that find-and-replaces in - an existing digest node, enforcing E-1 edge - conservation. + dreamer — 2-phase ReAct workflow (extract memory sub-units + tagged with bucket, then integrate per sub-unit + via the canonical write/edit tools). + cron_dreamer — daily wrapper around dreamer; scans today's + daily/ + resource/ files and runs dream_one on each. + +Phase 2 uses the canonical ``write`` / ``edit`` jobs directly — no +constrained variants. Bucket placement is prompt-level discipline. """ from . import cron_dreamer # noqa: F401 -- @R.register("cron_dreamer_step") -from . import digest_edit # noqa: F401 -- @R.register("digest_edit_step") -from . import digest_write # noqa: F401 -- @R.register("digest_write_step") from . import dreamer # noqa: F401 -- @R.register("dreamer_step") diff --git a/reme4/steps/evolve/dream/digest_edit.py b/reme4/steps/evolve/dream/digest_edit.py deleted file mode 100644 index bbb2bb49..00000000 --- a/reme4/steps/evolve/dream/digest_edit.py +++ /dev/null @@ -1,125 +0,0 @@ -"""``digest_edit_step`` — constrained EditStep that targets ``digest//.md``. - -Subclasses :class:`EditStep`. On top of the generic find-and-replace, -this step adds: - -* **Path-shape validation** — the target must be ``digest//.md`` - with ```` in the configured set. -* **Existence check** — the file must already exist (use - :class:`DigestWriteStep` for new nodes). -* **E-1 strong edge conservation** — the wikilink set in the file BEFORE - the replacement must be a subset of the wikilink set AFTER. The check - runs as a preflight: the replacement is simulated, links are compared, - and the actual write is delegated to ``super().execute()`` only if - conservation holds. On violation the step returns - ``REJECT_CONSERVATION`` (with the missing edges) and the file on disk - is left untouched. -""" - -from pathlib import Path - -import frontmatter - -from .digest_write import _validate_digest_path, bucket_names, normalize_buckets -from ...file_io._file_io import read_file_safe -from ...file_io.edit import EditStep -from ....components import R -from ....utils.wikilink_handler import WikilinkHandler - - -@R.register("digest_edit_step") -class DigestEditStep(EditStep): - """EditStep variant that enforces digest path shape + E-1 edge conservation. - - ``digest_dir`` is sourced from ``app_context.app_config.digest_dir`` at - execute time — same convention as the ``daily_*`` steps. - """ - - def __init__(self, buckets=None, **kwargs): - super().__init__(**kwargs) - self.buckets = normalize_buckets(buckets) - - def _reject(self, message: str, **meta) -> None: - assert self.context is not None - self.context.response.success = False - self.context.response.answer = f"REJECT: {message}" - if meta: - self.context.response.metadata.update(meta) - - async def execute(self): - assert self.context is not None - raw = str(self.context.get("path") or "") - old = self.context.get("old") - new = self.context.get("new") - digest_dir = getattr(self.app_context.app_config, "digest_dir", "") - - err = _validate_digest_path(raw, bucket_names(self.buckets), digest_dir) - if err: - self._reject(err) - return None - - abs_path = (Path(self.vault_path) / raw).resolve() - if not abs_path.exists(): - self._reject(f"{raw} does not exist; use digest_write instead") - return None - - # Preflight conservation check: simulate the find-and-replace, compare - # wikilink sets before vs after, refuse the write if any edge is dropped. - # We let super().execute() re-validate `old in body` and surface its - # own error if old is empty or missing. - if old is not None and new is not None and str(old) != "": - preview_text = await _preview_replacement(abs_path, str(old), str(new)) - if preview_text is not None: - missing = _missing_edges( - await _read_text(abs_path), - preview_text, - raw, - ) - if missing: - missing_repr = sorted(f"[[{t}]]" + (f" (predicate={p})" if p else "") for t, p in missing) - self._reject( - ( - f"REJECT_CONSERVATION: replacement drops {len(missing)} edge(s) " - "the old body had. E-1 strong-conservation: every outbound wikilink " - "in the old body MUST appear in the new body. " - f"Missing: {', '.join(missing_repr)}. " - "Adjust `new` to keep the missing links and retry." - ), - conservation_violation={ - "target_path": raw, - "missing": sorted(list(missing)), - }, - ) - return None - - return await super().execute() - - -async def _read_text(abs_path: Path) -> str: - text, _ = await read_file_safe(abs_path) - return text - - -async def _preview_replacement(abs_path: Path, old: str, new: str) -> str | None: - """Return the full file text as it WOULD look after EditStep's replace. - - Mirrors EditStep's body-only replacement (frontmatter is left untouched). - Returns ``None`` if the replacement isn't possible (``old`` not in body) so - the caller can defer the error to super().execute().""" - raw_text, _ = await read_file_safe(abs_path) - post = frontmatter.loads(raw_text) - body = post.content - if old not in body: - return None - post.content = body.replace(old, new) - new_text = frontmatter.dumps(post) if post.metadata else post.content - if not new_text.endswith("\n"): - new_text += "\n" - return new_text - - -def _missing_edges(old_text: str, new_text: str, path: str) -> set[tuple[str, str | None]]: - """Edges present in ``old_text`` but absent from ``new_text`` (E-1 deltas).""" - old_set = {(link.target_path, link.predicate) for link in WikilinkHandler.extract_links(old_text, path)} - new_set = {(link.target_path, link.predicate) for link in WikilinkHandler.extract_links(new_text, path)} - return old_set - new_set diff --git a/reme4/steps/evolve/dream/digest_write.py b/reme4/steps/evolve/dream/digest_write.py deleted file mode 100644 index 271bfbf9..00000000 --- a/reme4/steps/evolve/dream/digest_write.py +++ /dev/null @@ -1,140 +0,0 @@ -"""``digest_write_step`` — constrained WriteStep that targets ``digest//.md``. - -Subclasses :class:`WriteStep`. The only thing this step adds on top of the -generic file write is path-shape validation: - -* ``path`` must look like ``digest//.md`` (depth 1, ``.md`` suffix). -* ```` must be one of the configured ``buckets`` (defaults to the - dreamer's :data:`DEFAULT_BUCKETS`). -* The target must NOT already exist — for in-place updates the agent uses - :class:`DigestEditStep` instead. - -Everything else (atomic write semantics, encoding, parent-dir creation, -frontmatter handling) is inherited from :class:`WriteStep`. - -Bucket vocabulary lives in this module (NOT in the dreamer prompt template) -so it can be swapped / extended without touching the prompt. Future direction: -externalize via app-config injection; the ``Dreamer`` constructor already -accepts an override. -""" - -from pathlib import Path - -from ...file_io.write import WriteStep -from ....components import R - - -# Each bucket carries a name (the filesystem folder under ``digest/``) and a -# one-line description that the Phase 2 prompt renders into the bucket-picking -# heuristic block at runtime. -DEFAULT_BUCKETS: tuple[dict[str, str], ...] = ( - { - "name": "concept", - "description": 'definitions, principles, mental models ("what IS X?")', - }, - { - "name": "procedure", - "description": 'steps, methods, recipes ("how do I do X?")', - }, - { - "name": "entity", - "description": "specific named things (person, system, tool, project)", - }, - { - "name": "observation", - "description": 'findings, results, decisions with rationale ("what happened / was decided?")', - }, - { - "name": "preference", - "description": 'user / team / agent collaboration rules ("how does X like to work / what to avoid?")', - }, - { - "name": "unknown", - "description": "fallback when no specialized bucket fits (first-class, not a failure state)", - }, -) - - -def normalize_buckets(buckets) -> tuple[dict[str, str], ...]: - """Normalize a caller-supplied bucket spec into ``tuple[{name, description}, ...]``. - - Accepts ``None`` (→ :data:`DEFAULT_BUCKETS`), tuple/list of dicts (returned - as-is), or tuple/list of strings (legacy — each wrapped with an empty - description). - """ - if not buckets: - return DEFAULT_BUCKETS - first = next(iter(buckets)) - if isinstance(first, dict): - return tuple(buckets) - return tuple({"name": str(b), "description": ""} for b in buckets) - - -def bucket_names(buckets) -> tuple[str, ...]: - """Extract just the name field of each bucket — used for path-membership checks.""" - if not buckets: - return () - first = next(iter(buckets)) - if isinstance(first, dict): - return tuple(b["name"] for b in buckets) - return tuple(buckets) - - -@R.register("digest_write_step") -class DigestWriteStep(WriteStep): - """WriteStep variant that enforces the ``//.md`` layout. - - ``digest_dir`` is sourced from ``app_context.app_config.digest_dir`` at - execute time — same convention as ``daily_create`` / ``daily_list`` / - ``daily_reindex`` for their ``daily_dir``. - """ - - def __init__(self, buckets=None, **kwargs): - super().__init__(**kwargs) - self.buckets = normalize_buckets(buckets) - - def _reject(self, message: str) -> None: - assert self.context is not None - self.context.response.success = False - self.context.response.answer = f"REJECT: {message}" - - async def execute(self): - assert self.context is not None - raw = str(self.context.get("path") or "") - digest_dir = getattr(self.app_context.app_config, "digest_dir", "") - - err = _validate_digest_path(raw, bucket_names(self.buckets), digest_dir) - if err: - self._reject(err) - return None - - abs_path = (Path(self.vault_path) / raw).resolve() - if abs_path.exists(): - self._reject(f"{raw} already exists; use digest_edit instead") - return None - - return await super().execute() - - -def _validate_digest_path( - raw: str, - allowed_names: tuple[str, ...], - digest_dir: str, -) -> str | None: - """Return an error message, or ``None`` if ``raw`` matches the digest shape - and uses one of the allowed bucket names. ``digest_dir`` is the configured - digest root (e.g. ``"digest"``).""" - prefix = f"{digest_dir}/" - expected = f"{digest_dir}//.md" - if not raw.startswith(prefix) or not raw.endswith(".md"): - return f"path must be {expected!r}, got {raw!r}" - parts = raw.split("/") - if len(parts) != 3: - return f"digest is a shallow bucket layout ({expected}, depth 1); got {raw!r}" - bucket = parts[1] - if bucket not in allowed_names: - return ( - f"bucket {bucket!r} not in allowed set {list(allowed_names)}. " - "Use 'unknown' when no specialized bucket fits." - ) - return None diff --git a/reme4/steps/evolve/dream/dreamer.py b/reme4/steps/evolve/dream/dreamer.py index 61332a61..c3e37566 100644 --- a/reme4/steps/evolve/dream/dreamer.py +++ b/reme4/steps/evolve/dream/dreamer.py @@ -1,88 +1,46 @@ """Dreamer — auto-dream's create_or_update step. Reads one daily-event note or resource file at the given vault-relative -``path``, identifies the ABSTRACTIONS the material teaches in Phase 1, -then in Phase 2 makes ONE cognitive write decision (CREATE or one of -the three UPDATE flavors: CORROBORATE / REFINE / CORRECT) per -abstraction. See ``docs4/auto_dream_design.md`` for the model -contract (buckets / nodes / edges / evolution) and ``§4.2`` for the -pipeline. +``path``, identifies the ABSTRACTIONS the material teaches in Phase 1 +(each tagged with one of the three buckets), then in Phase 2 makes +ONE cognitive write decision (CREATE or one of the three UPDATE +flavors: CORROBORATE / REFINE / CORRECT) per abstraction using a +**bucket-specific** integrate prompt. -**Digest is the abstract memory layer** — analogous to a prefrontal -cortex aggregating cognition. Raw details (timestamps, full -procedures, who-said-what, numbers) stay in the material; digest -holds the principle, pattern, or precedent that should survive -once the details fade. Provenance wikilinks (``derived_from::``) -let readers drill back down to the source on demand. +**Digest is the abstract memory layer** — raw details stay in the +material; digest holds the principle, pattern, or precedent worth +recalling once the specifics fade. Provenance wikilinks +(``derived_from::``) let readers drill back down to the source. Pipeline (external loop in Python, two distinct ReAct agent invocations, **light Phase 1 / heavy Phase 2**): execute(): _extract(material_blob) # 1× ReAct: identify abstractions - # agent emits ExtractedUnits structured output - # ({units: [{name, summary}, ...]}) - for unit in self._units: # Python loop, K iterations (K = num abstractions) - _integrate_unit(unit) # 1× ReAct per abstraction: agent sees full material + - # the sub-unit's name/summary, recalls, decides - # bucket, makes ONE write decision (CREATE or - # one of the UPDATE flavors). Sub-unit ↔ digest - # node is 1:1. + # agent emits ExtractedUnits + # ({units: [{name, bucket, summary}, ...]}) + for unit in self._units: # Python loop, K iterations + _integrate_unit(unit) # 1× ReAct per abstraction, dispatched + # to integrate_system_prompt_; + # recalls cross-bucket, decides write, + # uses canonical write/edit tools. -* **Phase 1 (extract / abstract)** uses a read-only toolkit - and emits an :class:`ExtractedUnits` Pydantic model as its - final structured answer (no tool call needed for the unit - list — agentscope's ``structured_model`` enforces the shape). - The agent identifies the abstractions the material teaches — - principles, patterns, precedents worth carrying forward once - specifics fade. Multiple raw facts that illustrate the same - abstraction collapse into ONE sub-unit. Prompt biases toward - fewer / coarser sub-units; filing detail under a digest - sub-unit is the wrong layer. No event-level umbrella node is - manufactured — the material itself plays that role via - ``derived_from`` provenance edges. +The bucket vocabulary is hard-coded (:data:`BUCKETS`) — three buckets, +each with a dedicated Phase 2 prompt: -* **Phase 2 (integrate per abstraction)** runs once per declared - sub-unit with a fresh ReAct session (clean context) and the full - read + write toolkit (``search``, ``traverse``, ``read``, - ``frontmatter_read``, ``digest_write``, ``digest_edit``). Three - UPDATE shapes are surfaced explicitly - in the prompt: +* ``procedure`` — how-to-do-X: steps, methods, recipes, workflows. +* ``personal`` — user/team specific: identity, preferences, + conventions, things they avoid. +* ``wiki`` — general knowledge: definitions, principles, + observations, decisions-as-precedent. Default catch-all. - - **corroborate** (most common): the abstraction already - exists; the material is one more instance → append a - ``derived_from::`` provenance wikilink so confidence - accumulates; body unchanged in substance. - - **refine**: the material reveals nuance / scope / edge - cases the abstraction under-specified → tighten the - relevant span + add the new provenance link. - - **correct**: the material contradicts the abstraction → - tighten to the narrower form both old and new support, - or annotate the contradiction inline + add provenance. +There is no SKIP outcome in Phase 2: Phase 1 is the gate for "not +worth memorizing"; anything reaching Phase 2 warrants a write. - CREATE is reserved for genuinely new abstractions not yet in - the vault — even thin first-encounter seeds, which grow via - CORROBORATE / REFINE on later passes. There is no SKIP outcome: - Phase 1 is the gate for "not worth memorizing"; anything that - reaches Phase 2 warrants a write. - -The trade-off vs heavy Phase 1: full material is sent to LLM K -times in Phase 2 (one per abstraction). The advantages: no -information loss in summary, focused reasoning per call, and -granularity tuned at a single prompt (Phase 1) rather than two. - -Mechanical guardrails at the write boundary: - -* ``digest_write`` (subclass of WriteStep) rejects paths outside - ``//.md`` (where ``digest_dir`` comes - from app config and ``bucket`` is in the fixed bucket set), and - refuses if the path already exists. -* ``digest_edit`` (subclass of EditStep) is a body-only find-and-replace - on an existing digest node, gated by E-1 strong-conservation: the - outbound link set BEFORE the replacement must be a subset of the link - set AFTER. If any edge would be dropped the tool returns - ``REJECT_CONSERVATION`` and the agent must adjust ``new`` to keep - the missing links before retrying. +Phase 2 uses the **canonical** ``write`` / ``edit`` jobs (no +constrained variants). Bucket placement and edge conservation are +prompt-level discipline; the tools themselves perform no path-shape +or conservation validation. Invocation form (CLI / MCP): reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md @@ -98,13 +56,21 @@ from agentscope.message import Msg, TextBlock from agentscope.tool import Toolkit, ToolResponse from pydantic import BaseModel, Field -from .digest_edit import DigestEditStep -from .digest_write import DigestWriteStep, bucket_names, normalize_buckets from .._evolve import FlexReActAgent from ...base_step import BaseStep from ....components import R +# Hard-coded bucket vocabulary. Phase 1 classifies each sub-unit into +# one of these; Phase 2 dispatches to the bucket-specific prompt. +# Order matters for prompt rendering — keep procedure/personal/wiki. +BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki") + +# Bucket = Literal of BUCKETS. Pydantic Literal must be a static type; +# update both BUCKETS and Bucket together if the vocabulary changes. +Bucket = Literal["procedure", "personal", "wiki"] + + _EXTRACT_READ_TOOLS: tuple[str, ...] = ("read",) _INTEGRATE_READ_TOOLS: tuple[str, ...] = ( @@ -114,6 +80,11 @@ _INTEGRATE_READ_TOOLS: tuple[str, ...] = ( "frontmatter_read", ) +_INTEGRATE_WRITE_TOOLS: tuple[str, ...] = ( + "write", + "edit", +) + def _pack_material(file_store, path: str) -> str: """Render one daily-event note or resource file into a prompt block.""" @@ -139,7 +110,18 @@ class MemoryUnit(BaseModel): "Short kebab-case identifier for the abstraction " "(e.g. 'jwt-rotation-decision', 'pr-size-pref'). " "Agent-internal handle — NOT the eventual digest slug; " - "Phase 2 picks the actual filing path + bucket." + "Phase 2 picks the actual filing path." + ), + ) + bucket: Bucket = Field( + description=( + "Which bucket this abstraction belongs in — Phase 2 dispatches " + "to a bucket-specific prompt based on this. Pick exactly one: " + "`procedure` (how-to-do-X — steps, methods, recipes, workflows), " + "`personal` (user/team-specific — identity, preferences, " + "conventions, things they avoid), `wiki` (general knowledge — " + "definitions, principles, observations, decisions-as-precedent; " + "default catch-all when nothing else fits)." ), ) summary: str = Field( @@ -161,24 +143,18 @@ class ExtractedUnits(BaseModel): description=( "Memory sub-units identified in the material — orthogonal " "abstractions (principles / patterns / precedents) worth " - "lifting into long-term memory. Empty list = nothing worth " - "lifting (Phase 2 is skipped)." + "lifting into long-term memory. Each is tagged with its " + "bucket. Empty list = nothing worth lifting (Phase 2 is skipped)." ), ) -def _render_outcome_line(unit_name: str, o: "IntegrateOutcome") -> str: +def _render_outcome_line(unit_name: str, bucket: str, o: "IntegrateOutcome") -> str: """Format one IntegrateOutcome as a one-line summary entry.""" - if o.action == "CREATE": - body = f"CREATE {o.target_path}" - if o.note: - body += f" — {o.note}" - else: # CORROBORATE / REFINE / CORRECT (all UPDATE-flavored) - recovered = " (recovered from REJECT_CONSERVATION)" if o.recovered_from_conservation else "" - body = f"{o.action} {o.target_path}{recovered}" - if o.note: - body += f" — {o.note}" - return f"[{unit_name}] {body}" + body = f"{o.action} {o.target_path}" + if o.note: + body += f" — {o.note}" + return f"[{unit_name}/{bucket}] {body}" class IntegrateOutcome(BaseModel): @@ -203,7 +179,8 @@ class IntegrateOutcome(BaseModel): ) target_path: str = Field( description=( - "The digest path you wrote to — must match what your " "`digest_write` / `digest_edit` call(s) targeted." + "The digest path you wrote to — must match what your `write` / " + "`edit` call(s) targeted." ), ) note: str = Field( @@ -216,14 +193,6 @@ class IntegrateOutcome(BaseModel): "outcome note." ), ) - recovered_from_conservation: bool = Field( - default=False, - description=( - "Set to true if `digest_edit` initially returned " - "REJECT_CONSERVATION and you re-composed `new` to preserve the " - "missing links. Only meaningful for CORROBORATE / REFINE / CORRECT." - ), - ) class DreamResult(BaseModel): @@ -231,9 +200,8 @@ class DreamResult(BaseModel): Per-tool audit lives in the toolkit layer (not exposed back to the orchestrator). Structured outcome here is the input path the call - processed, the memory sub-units the agent declared in Phase 1, - what got created / updated in Phase 2, and any conservation - rejections that occurred along the way. + processed, the memory sub-units the agent declared in Phase 1, and + what got created / updated in Phase 2. """ used_llm: bool = False @@ -242,7 +210,6 @@ class DreamResult(BaseModel): units: list[dict] = Field(default_factory=list) nodes_created: list[str] = Field(default_factory=list) nodes_updated: list[str] = Field(default_factory=list) - conservation_violations: list[dict] = Field(default_factory=list) summary: str = "" error: str = "" @@ -257,8 +224,6 @@ class Dreamer(BaseStep): empty string to no-op. hint (str, optional): caller guidance to the LLM (e.g. "focus on the auth-related decisions"). - buckets (list[str], optional): override the fixed bucket - set; default ``DEFAULT_BUCKETS``. Output (written to context.response.answer): ``DreamResult`` JSON in ``metadata``; LLM summary in ``answer``. @@ -272,20 +237,16 @@ class Dreamer(BaseStep): toolkit: Toolkit | None = None, console_enabled: bool = False, timezone: str | None = None, - buckets: list[str] | tuple[str, ...] | None = None, **kwargs, ): super().__init__(**kwargs) self.toolkit = toolkit self.console_enabled = console_enabled self.timezone = timezone - self.buckets = normalize_buckets(buckets) - assert "unknown" in bucket_names(self.buckets), "bucket set must include 'unknown' as the unclassified fallback" # Per-invocation outcome trackers, populated by tool callbacks. self._units: list[dict] = [] self._created: list[str] = [] self._updated: list[str] = [] - self._violations: list[dict] = [] def _now(self) -> datetime.datetime: if self.timezone: @@ -305,46 +266,43 @@ class Dreamer(BaseStep): except Exception: return False - def _make_digest_write_tool(self): - """Tool closure: wraps :class:`DigestWriteStep` and tracks creates.""" + def _make_write_tool(self): + """Wrap the canonical ``write`` job with create-tracking. - async def digest_write(path: str, name: str, description: str, content: str) -> ToolResponse: - step = DigestWriteStep( - file_store=self.file_store, - buckets=self.buckets, - app_context=self.app_context, + Same shape as the underlying job; the wrapper just records + successful paths into ``self._created`` so the dreamer can + reconstruct outcomes when the LLM drops its structured emission. + """ + job = self.get_job("write") + if job is None: + raise RuntimeError("write job not registered") + + async def write(path: str, name: str, description: str, content: str) -> ToolResponse: + resp = await job( + path=path, + name=name, + description=description, + content=content, ) - await step(path=path, name=name, description=description, content=content) - assert step.context is not None - resp = step.context.response - if not resp.success: - return ToolResponse(content=[TextBlock(type="text", text=resp.answer)]) - self._created.append(path) - return ToolResponse(content=[TextBlock(type="text", text=f"OK: created {path}")]) + if resp.success: + self._created.append(path) + return ToolResponse(content=[TextBlock(type="text", text=resp.answer)]) - return digest_write + return write, job - def _make_digest_edit_tool(self): - """Tool closure: wraps :class:`DigestEditStep` and tracks updates / conservation violations.""" + def _make_edit_tool(self): + """Wrap the canonical ``edit`` job with update-tracking.""" + job = self.get_job("edit") + if job is None: + raise RuntimeError("edit job not registered") - async def digest_edit(path: str, old: str, new: str) -> ToolResponse: - step = DigestEditStep( - file_store=self.file_store, - buckets=self.buckets, - app_context=self.app_context, - ) - await step(path=path, old=old, new=new) - assert step.context is not None - resp = step.context.response - if not resp.success: - violation = (resp.metadata or {}).get("conservation_violation") - if violation: - self._violations.append(violation) - return ToolResponse(content=[TextBlock(type="text", text=resp.answer)]) - self._updated.append(path) - return ToolResponse(content=[TextBlock(type="text", text=f"OK: updated {path}")]) + async def edit(path: str, old: str, new: str) -> ToolResponse: + resp = await job(path=path, old=old, new=new) + if resp.success: + self._updated.append(path) + return ToolResponse(content=[TextBlock(type="text", text=resp.answer)]) - return digest_edit + return edit, job def _build_extract_toolkit(self) -> Toolkit: """Read-only toolkit for the extract agent. Sub-units come back via @@ -355,97 +313,43 @@ class Dreamer(BaseStep): return toolkit def _build_integrate_toolkit(self) -> Toolkit: - """Full read + conservation-aware write toolkit for the integrate agent.""" + """Full read + canonical write/edit toolkit for the integrate agent. + + write/edit are wrapped in tracker closures (created/updated paths) + so the outer loop can reconstruct outcomes when an LLM call drops + its structured emission. Read-only tools go through ``add_as_tool`` + unchanged. + """ toolkit = self.toolkit or Toolkit() for job_name in _INTEGRATE_READ_TOOLS: self.add_as_tool(toolkit, job_name) - digest_dir = getattr(self.app_context.app_config, "digest_dir", "") - path_shape = f"'{digest_dir}//.md'" - digest_write_desc = ( - "Create a NEW digest node — same shape as the canonical `write` job, plus " - f"path-shape validation: `path` must be {path_shape} where " - f"bucket is one of {list(bucket_names(self.buckets))} (use 'unknown' when " - "no specialized bucket fits — it is a first-class bucket, not a failure " - "state). `name` and `description` go into the YAML frontmatter; `content` " - "is the body. Fails if the path already exists; use `digest_edit` then." - ) + + write_tool, write_job = self._make_write_tool() toolkit.register_tool_function( - tool_func=self._make_digest_write_tool(), - func_name="digest_write", - func_description=digest_write_desc, + tool_func=write_tool, + func_name="write", + func_description=write_job.description, json_schema={ "type": "function", "function": { - "name": "digest_write", - "description": digest_write_desc, - "parameters": { - "type": "object", - "properties": { - "path": { - "type": "string", - "description": f"vault-relative path; must match {path_shape}", - }, - "name": { - "type": "string", - "description": "frontmatter name (usually the slug)", - }, - "description": { - "type": "string", - "description": "frontmatter description — one-line summary of the abstraction", - }, - "content": { - "type": "string", - "description": "body (markdown; no frontmatter " - "— name/description go in the fields above)", - }, - }, - "required": ["path", "name", "description", "content"], - }, + "name": "write", + "description": write_job.description, + "parameters": write_job.parameters, }, }, ) - digest_edit_desc = ( - "Find-and-replace inside an existing digest node's body — same shape as " - "the canonical `edit` job, plus path-shape validation (`path` must be " - f"{path_shape}, file must exist) and E-1 strong edge conservation: every " - "outbound wikilink present BEFORE the replacement must still be present " - "AFTER. If you drop any edge the tool returns REJECT_CONSERVATION and you " - "must adjust `new` to keep the missing links. Operates on body only; " - "frontmatter is untouched. Prefer narrow `old` spans." - ) + + edit_tool, edit_job = self._make_edit_tool() toolkit.register_tool_function( - tool_func=self._make_digest_edit_tool(), - func_name="digest_edit", - func_description=digest_edit_desc, + tool_func=edit_tool, + func_name="edit", + func_description=edit_job.description, json_schema={ "type": "function", "function": { - "name": "digest_edit", - "description": digest_edit_desc, - "parameters": { - "type": "object", - "properties": { - "path": { - "type": "string", - "description": f"vault-relative path; must match {path_shape} and exist", - }, - "old": { - "type": "string", - "description": ( - "Substring to locate in the EXISTING body (frontmatter excluded). " - "Must match verbatim. Pick a span large enough to be unique." - ), - }, - "new": { - "type": "string", - "description": ( - "Replacement text. Should weave new material into the existing " - "wording without dropping any wikilinks the `old` span contained." - ), - }, - }, - "required": ["path", "old", "new"], - }, + "name": "edit", + "description": edit_job.description, + "parameters": edit_job.parameters, }, }, ) @@ -460,7 +364,7 @@ class Dreamer(BaseStep): sys_prompt=self.prompt_format( "extract_system_prompt", vault_dir=str(vault_dir), - buckets=", ".join(bucket_names(self.buckets)), + buckets=", ".join(BUCKETS), ), formatter=self.as_llm_formatter, toolkit=toolkit, @@ -486,36 +390,44 @@ class Dreamer(BaseStep): continue name = str(raw.get("name") or "").strip() summary = str(raw.get("summary") or "").strip() - if name and summary: - cleaned.append({"name": name, "summary": summary}) + bucket = str(raw.get("bucket") or "").strip() + if not name or not summary: + continue + if bucket not in BUCKETS: + # Defensive: structured_model should already reject this, + # but if it slips through we route to wiki (the catch-all). + self.logger.warning( + f"[{self.name}] unit {name!r} emitted bucket {bucket!r} " + f"not in {list(BUCKETS)}; routing to 'wiki'", + ) + bucket = "wiki" + cleaned.append({"name": name, "summary": summary, "bucket": bucket}) self._units = cleaned return (msg.get_text_content() or "").strip() async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome: - """One ReAct invocation per memory sub-unit. Returns the parsed + """One ReAct invocation per memory sub-unit, dispatched to the + bucket-specific system prompt. Returns the parsed :class:`IntegrateOutcome` reported by the agent. - File writes happen as side effects via the ``digest_write`` / - ``digest_edit`` tool calls during the ReAct loop (which populate - ``self._created`` / ``self._updated`` / ``self._violations``); the - structured outcome here is the agent's own summary of what it - decided — useful for rendering and for catching hallucinations - (action=CREATE without the matching write call landing in trackers). + File writes happen as side effects via the canonical ``write`` / + ``edit`` tool calls (which populate ``self._created`` / + ``self._updated`` via the tracker closures); the structured + outcome here is the agent's own summary of what it decided — + useful for rendering and for catching hallucinations (action= + CREATE without the matching write call landing in trackers). """ + bucket = unit.get("bucket") or "wiki" toolkit = self._build_integrate_toolkit() digest_dir = getattr(self.app_context.app_config, "digest_dir", "") - buckets_block = "\n".join( - f" - `{digest_dir}/{b['name']}/`" + (f" — {b['description']}" if b.get("description") else "") - for b in self.buckets - ) agent = FlexReActAgent( name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}", model=self.as_llm, sys_prompt=self.prompt_format( - "integrate_system_prompt", + f"integrate_system_prompt_{bucket}", vault_dir=str(vault_dir), digest_dir=digest_dir, - buckets=buckets_block, + bucket=bucket, ), formatter=self.as_llm_formatter, toolkit=toolkit, @@ -525,6 +437,7 @@ class Dreamer(BaseStep): "integrate_user_message", hint=hint or "(none)", unit_name=unit.get("name", ""), + unit_bucket=bucket, unit_summary=unit.get("summary", ""), material_blob=material_blob, ) @@ -580,12 +493,11 @@ class Dreamer(BaseStep): self._units.clear() self._created.clear() self._updated.clear() - self._violations.clear() vault_dir = self._vault_dir() # Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the - # memory sub-units worth lifting. + # memory sub-units worth lifting. Each unit carries its own bucket. self.logger.info(f"[{self.name}] extract phase: path={path!r}") extract_summary = await self._extract(material_blob, hint, vault_dir) @@ -599,31 +511,36 @@ class Dreamer(BaseStep): self.logger.info( f"[{self.name}] integrate phase: {len(self._units)} sub-unit(s): " - f"{', '.join(u['name'] for u in self._units)}", + + ", ".join(f"{u['name']}/{u['bucket']}" for u in self._units), ) - # Phase 2 — integrate, one fresh ReAct per sub-unit. Python-level - # loop, not agent loop. Each session emits a structured - # IntegrateOutcome; file writes happen as side effects via - # digest_write / digest_edit tool calls. + # Phase 2 — integrate, one fresh ReAct per sub-unit, dispatched to + # the bucket-specific system prompt. Python-level loop, not agent + # loop. Each session emits a structured IntegrateOutcome; file + # writes happen as side effects via the canonical write / edit + # tool calls. per_unit_lines: list[str] = [] for i, unit in enumerate(self._units, start=1): name = unit.get("name", "?") + bucket = unit.get("bucket", "?") try: outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir) except Exception as e: self.logger.error( - f"[{self.name}] integrate {i}/{len(self._units)} (unit={name}) " f"failed: {type(e).__name__}: {e}", + f"[{self.name}] integrate {i}/{len(self._units)} " + f"(unit={name}, bucket={bucket}) failed: {type(e).__name__}: {e}", ) - per_unit_lines.append(f"[{name}] FAILED: {type(e).__name__}: {e}") + per_unit_lines.append(f"[{name}/{bucket}] FAILED: {type(e).__name__}: {e}") continue - per_unit_lines.append(_render_outcome_line(name, outcome)) + per_unit_lines.append(_render_outcome_line(name, bucket, outcome)) summary = ( f"Declared {len(self._units)} sub-unit(s) " - f"({', '.join(u['name'] for u in self._units)}); " - f"created {len(self._created)}, updated {len(self._updated)}, " - f"conservation violations {len(self._violations)}.\n" + "\n".join(per_unit_lines) + + "(" + + ", ".join(f"{u['name']}/{u['bucket']}" for u in self._units) + + "); " + f"created {len(self._created)}, updated {len(self._updated)}.\n" + + "\n".join(per_unit_lines) ) return DreamResult( @@ -632,7 +549,6 @@ class Dreamer(BaseStep): units=list(self._units), nodes_created=list(self._created), nodes_updated=list(self._updated), - conservation_violations=list(self._violations), summary=summary, skipped=False, ) diff --git a/reme4/steps/evolve/dream/dreamer.yaml b/reme4/steps/evolve/dream/dreamer.yaml index 44f1ac7c..64bc3731 100644 --- a/reme4/steps/evolve/dream/dreamer.yaml +++ b/reme4/steps/evolve/dream/dreamer.yaml @@ -1,13 +1,13 @@ extract_system_prompt: | - You are the **dreamer** — EXTRACT phase. Your ONLY job here - is to read the material - and identify the ABSTRACTIONS it teaches — the principles, - patterns, decisions-as-precedent, cognitive takeaways — that - belong in long-term memory. You commit them via the structured - output schema attached to this call (an `ExtractedUnits` - object). You do NOT do recall, integrate, or write. A separate - downstream invocation processes each unit with the full material - in context. + You are the **dreamer** — EXTRACT phase. Your ONLY job here is to + read the material, identify the ABSTRACTIONS it teaches — the + principles, patterns, decisions-as-precedent, cognitive takeaways + that belong in long-term memory — and **classify each into one + of three buckets**. You commit the result via the structured + output schema attached to this call (an `ExtractedUnits` object). + You do NOT do recall, integrate, or write. A separate downstream + invocation processes each unit (with its bucket determining which + Phase 2 prompt runs). vault_dir: {vault_dir} @@ -46,19 +46,8 @@ extract_system_prompt: | Sub-units are NOT bucket names, NOT kinds, NOT the eventual digest slug — they are an agent-internal handle for the - abstraction you've identified. Phase 2 picks the bucket / - slug / write decision per sub-unit. - - Typical abstractions, by material shape: - - * Analysis / decision notes: the underlying principle the - decision rests on; a pattern the analysis surfaces; - a constraint that will recur in similar problems. - * Discussion notes: a preference / convention that should - shape future work; a stable concept the discussion - crystallizes; an open question worth carrying forward. - * Resource content: a foundational concept; a procedure - that generalizes beyond this resource. + abstraction you've identified. Phase 2 picks the slug + write + decision per sub-unit; YOU pick the bucket here in Phase 1. ### Bias: fewer, richer sub-units over many narrow ones @@ -75,23 +64,58 @@ extract_system_prompt: | When in doubt, KEEP TOGETHER (or DROP one of them entirely). - Counter-example for splitting: "preference: small PRs" + - "preference: no trailing summary in replies" → TWO sub-units. - Different situations of invocation (code review vs response - style), independent evolution. - ### What NOT to declare - - Passing mentions with no new abstraction (e.g. an OAuth - recap that restates a known concept) — daily-note indexing - already covers detail-level recall. + - Passing mentions with no new abstraction — daily-note + indexing already covers detail-level recall. - Facts whose only audience is the material itself (one-off timestamps, single meeting attendance) — not an abstraction. - Event-level umbrella sub-units (e.g. "X-event-summary") — every sub-unit already carries a `derived_from:: [[]]` - wikilink, so the material itself is the fan-out point linking - to all its derived digest nodes; the provenance graph - already provides that view. + wikilink, so the material itself is the fan-out point. + + ## Bucket vocabulary (HARD-CODED — pick exactly one per unit) + + The bucket determines which specialized Phase 2 prompt processes + this sub-unit. Three buckets, picked by *what kind of abstraction* + this is — NOT by the material's surface topic. + + - **`procedure`** — *how to do X*. Steps, methods, recipes, + workflows, runbooks, executable patterns. The reader's question + is "how do I accomplish Y?" Pick this when the abstraction + is an actionable sequence or technique. + Examples: "key-rotation procedure", "incident triage flow", + "how to wire up a new MCP tool". + + - **`personal`** — *user/team-specific facts about how WE work*. + Identity ("who is X"), preferences ("user prefers terse replies"), + conventions ("we use kebab-case for slug names"), things to + avoid ("don't run schema migrations on Friday"), collaboration + style. The reader's question is "what does THIS user / team + want / do / dislike?" Pick this when the abstraction is only + valid in the context of this user / team / project. + Examples: "huangsen prefers short PRs", "team avoids + mocking the DB in integration tests", "we don't write + `status` frontmatter". + + - **`wiki`** — *general knowledge*. Definitions, principles, + observations, decisions-as-precedent, factual claims, mental + models. The reader's question is "what IS X / what happened + / what was decided?" Pick this when the abstraction is true + independent of which user is reading. Also the **default + catch-all** when nothing else fits cleanly. + Examples: "JWT is a signed token format", "short-credential + compliance drives auth cadence", "moving to 24h refresh + reduced p99 latency by 12%". + + Picking heuristic when a unit straddles two buckets: pick by + CENTER OF GRAVITY — what would a future reader most likely be + searching for, and from which mindset? "User prefers small PRs" + is *personal*, not *wiki*, because the lesson only applies to + this user. "Small PRs are easier to review" is *wiki* — it's a + general claim. "Steps to split a large PR" is *procedure*. + + Available buckets: {buckets} ## What to do @@ -107,12 +131,13 @@ extract_system_prompt: | would I still want to recall?* That lesson is a sub-unit candidate. - 3. **Emit the surviving list** as your structured output. Each + 3. **Tag each unit with a bucket** (procedure / personal / + wiki). This routes Phase 2 to the right specialized prompt. + + 4. **Emit the surviving list** as your structured output. Each entry's `summary` should be concrete about WHERE in the - material the supporting evidence lives (e.g. "the 30→24h - decision in the 'Decision' section backed by the SOC2 CC6.1 - criticism in the 'Observation' section"), so Phase 2 can - cite it as provenance without re-reading. Field shapes are + material the supporting evidence lives, so Phase 2 can cite + it as provenance without re-reading. Field shapes are enforced by the schema attached to this call. If the material teaches no new abstraction worth long-term @@ -121,12 +146,10 @@ extract_system_prompt: | ## Boundaries - - You CANNOT write to digest in this phase (no - digest_write / digest_edit tools here). + - You CANNOT write to digest in this phase (no write/edit tools + here). - You CANNOT do recall in this phase (no search/traverse here). - You declare ABSTRACTIONS (sub-units), not detail copies. - Phase 2 handles recall + the single write decision per - sub-unit. - The structured output you emit is the final scope for this dream call. @@ -138,193 +161,151 @@ extract_user_message: | {material_blob} - Identify the ABSTRACTIONS this material teaches (lessons / - principles / patterns worth recalling after the details fade). - Collapse multiple supporting facts into one sub-unit when they - illustrate the same abstraction. Emit the result via the - structured output schema attached to this call. Use an empty - unit list when the material teaches no new abstraction. + Identify the ABSTRACTIONS this material teaches, classify each + into one of {{procedure, personal, wiki}}, and emit via the + structured output schema. Empty list if nothing new is taught. -integrate_system_prompt: | - You are the **dreamer** — INTEGRATE phase. This invocation - processes ONE MEMORY - SUB-UNIT against the full material. You see the entire material - in the user message; Phase 1 told you which abstraction to - focus on and pointed you at the supporting evidence. Your job: - recall existing digest nodes (cross-bucket), decide between - CREATE and the three UPDATE flavors (CORROBORATE / REFINE / - CORRECT), and write. +# ============================================================ +# Phase 2 — bucket-specific INTEGRATE prompts. +# Each bucket has its OWN fully self-contained system prompt. +# Dispatcher in dreamer.py picks integrate_system_prompt_ +# based on the bucket Phase 1 assigned to the unit. +# ============================================================ - **Sub-unit maps 1:1 to a digest node.** Exactly ONE write - per session — there is no "no-write" outcome; Phase 1 is the - gate for "not worth memorizing". +integrate_system_prompt_procedure: | + You are the **dreamer** — INTEGRATE phase, **procedure bucket**. - ## Digest is the abstract memory layer + This invocation processes ONE memory sub-unit whose abstraction + is a *procedure*: a how-to-do-X — steps, methods, recipes, + workflows, runbooks. The reader's question against this node + later will be "how do I accomplish Y?" The completed material + is in the user message; Phase 1 already pointed you at the + supporting evidence. Your job: recall existing digest nodes + (cross-bucket), decide between CREATE and the three UPDATE + flavors (CORROBORATE / REFINE / CORRECT), and write — **with + a procedure-shaped body**. - Digest is **not** a faithful copy of the material — it is the - cognitive aggregation (think prefrontal cortex). The details - stay in the daily / resource file; digest holds the principle, - pattern, or precedent the agent should recall later. So: + **Sub-unit maps 1:1 to a digest node.** Exactly ONE write per + session — there is no "no-write" outcome. - - **Body should be SHORT and abstract** (≈ 50-200 words for - most nodes; longer only when the concept genuinely needs it). - If your draft starts copying paragraphs from the material, - you're filing detail in the wrong layer. - - **Provenance edges carry the details.** Whenever this - abstraction is illustrated by a specific material, add a - `derived_from:: [[daily/...]]` or `[[resource/...]]` - wikilink — readers drill down through the edge, not through - re-stated facts in the body. - - **Wikilinks between digest nodes** carry the conceptual - graph: `relates_to::`, `depends_on::`, `is_a::`, etc. + ## Procedure-bucket body shape + + A procedure node body should read like a runbook, not a recap. + Keep it short and actionable: + + - **Trigger / when to use**: 1 line — under what conditions + does the reader reach for this procedure? + - **Steps**: a numbered or terse bulleted list. Each step is + one verb-led imperative. Optional inline justification + ("because X locks the row before Y commits"). + - **Pre-conditions / inputs**: one short list, not prose. + - **Failure modes / caveats**: brief — "if step 3 returns + ROLLBACK, restart from step 1" type notes. NOT a transcript + of every observed failure. + - **At least one `derived_from:: [[]]`** so + the procedure is traceable to the material that taught it. + + Body is short (≈ 50-200 words). If your draft starts copying + paragraphs of conversation or full code blocks from the + material, you're filing detail in the wrong layer — those + belong in the daily / resource file; the digest only carries + the generalizable runbook. ## What to do - Two-stage flow: **RECALL** (assemble candidate paths) → **HIT** - (confirm whether any candidate carries this sub-unit's - abstraction). The decision falls out of stage 2: + Two-stage flow: **RECALL** (cross-bucket; assemble candidate + paths) → **HIT** (confirm whether any candidate carries this + procedure): - hit set empty ⇒ CREATE - hit set non-empty ⇒ UPDATE the best match - (CORROBORATE / REFINE / CORRECT) + hit set empty ⇒ CREATE under {digest_dir}/procedure/ + hit set non-empty ⇒ UPDATE the best match + (CORROBORATE / REFINE / CORRECT) - ### Stage 1 — RECALL (search + traverse) + ### Stage 1 — RECALL (search + traverse; CROSS-BUCKET) Goal: surface candidate paths under `{digest_dir}/`. Recall is - intentionally cross-bucket; UPDATE may target any bucket. + intentionally cross-bucket — the same procedure may have been + filed elsewhere by an earlier dream pass, and updating it in + place beats creating a duplicate. - - **`search`** — keyword + vector hits. Call with the sub-unit's - likely slug + its summary. Returns top-K matched chunks plus - a one-hop wikilink expansion. + - **`search`** — keyword + vector hits using the sub-unit's + likely slug + summary. Include verb stems ("rotate", + "migrate", "deploy") since procedure slugs lean that way. - - **`traverse path= depth=2 direction=both`** — graph - expansion. Run this whenever `search` returned ANY hit under - `{digest_dir}/`, even if the top hit looks unrelated by - snippet alone. Search is keyword-based and routinely misses - semantically close abstractions filed under different - terminology — those live one wikilink away from a noisy hit. - Skipping traverse is the main failure mode that produces - duplicate nodes under different bucket / slug. - - If `search` returns nothing under `{digest_dir}/`, there is no - anchor to traverse from. Recall ends with an empty candidate - set; proceed to CREATE. + - **`traverse path= depth=2 direction=both`** — + graph expansion. Run this whenever `search` returned ANY + hit under `{digest_dir}/`, even if the top hit looks unrelated + by snippet alone — adjacent procedures often link from + related concept nodes. ### Stage 2 — HIT (frontmatter_read + read) - Goal: for each candidate path, decide whether it carries the - same abstraction as your sub-unit. Progressive disclosure — - cheap triage first. + Goal: for each candidate, decide whether it carries the same + procedure as your sub-unit. - - **`frontmatter_read path=`** — peek `name` + - `description`. If they clearly refer to a DIFFERENT - abstraction, drop the candidate without paying for the body. + - **`frontmatter_read`** — peek `name` + `description`. Drop + candidates that are clearly different procedures (different + domain, different trigger). + - **`read`** — full body for survivors. Same procedure means: + same trigger AND substantially overlapping steps. Slight + variations in wording or one extra step is REFINE territory, + not "different procedure". - - **`read path=`** — full body for every survivor. - Do NOT decide UPDATE on chunk snippets or frontmatter alone. - The body is what you compare your sub-unit against. - - Hit set = candidates whose body confirms the same abstraction. + Hit set = candidates whose body confirms the same procedure. ### Decision - - **Hit set empty** ⇒ CREATE a new digest node. + - **Hit set empty** ⇒ CREATE a new digest node under + `{digest_dir}/procedure/.md`. - **Hit set non-empty** ⇒ UPDATE the best-matching hit: - same instance restated → CORROBORATE; nuance/scope added → - REFINE; contradiction or overstatement → CORRECT. + - **CORROBORATE**: same procedure observed again → + append a `derived_from::` link, optionally strengthen + wording ("consistently used across N runs"); steps + unchanged. + - **REFINE**: procedure has new pre-condition, edge case, + or failure-mode addition → expand the relevant span; + new step or guard goes into the right slot of the + ordering; add provenance. + - **CORRECT**: procedure as previously stated has a + wrong order, missing critical step, or bad outcome → + tighten or annotate inline (`> note: contradicted by + [[new-material]] — `); add provenance. - ### b. Decide bucket + write — exactly one of: + ### Tools - - **`digest_write(path, name, description, content)`** — for CREATE. - Same shape as the canonical `write` job; the digest variant - only adds path-shape validation. Use ONLY when no existing - digest node captures this abstraction. - - `path` must be `{digest_dir}//.md` where `bucket` - is one of the FIXED bucket vocabulary below (pick the - one a human would browse for this abstraction; use - `unknown` only as a last resort). + - **`write(path, name, description, content)`** — for CREATE. + Canonical write job (no path-shape validation; you are + responsible for placing it correctly). + - `path` MUST be `{digest_dir}/procedure/.md`. Do + NOT write outside `procedure/` — your prompt is bucket- + specific because Phase 1 classified this unit as a + procedure. - `name` is the frontmatter name (usually the slug). - - `description` is the one-line summary of the abstraction - (lands in YAML frontmatter; downstream search relies on it). - - `content` is the body — short (≈ 50-200 words), abstract, - principle-oriented — NOT a transcript of the material. - Do NOT prepend `---` frontmatter into `content`; the step - composes the frontmatter from `name` + `description` - automatically. Include at least one - `derived_from:: [[]]` provenance wikilink - in the body so the abstraction can be traced back to its - source. - Fails if path exists; if so, this is actually an UPDATE — - re-do recall and switch to `digest_edit`. + - `description` is the one-line summary of the procedure. + - `content` is the body (no leading `---` frontmatter — + the step composes frontmatter from `name` + `description` + automatically). + If the path already exists, the write will overwrite — but + that should never happen for a CREATE, because RECALL would + have surfaced it as a hit. Re-do RECALL in that case. - - **`digest_edit(path, old, new)`** — for the three - update-flavored actions (CORROBORATE / REFINE / CORRECT). - This is the cognitive engagement step. The existing digest - captures an earlier version of the abstraction; the new - material **corroborates, corrects, or refines** it: - - 1. **CORROBORATE** (most common). The material is one - more instance of an abstraction already captured. - Body usually unchanged in substance — append a new - `derived_from:: [[]]` provenance - wikilink so the supporting evidence accumulates. - Optionally strengthen wording ("consistently - observed across N sources" / replace "appears to" with - "does"). One small `digest_edit` call is enough. - 2. **REFINE** (frequent). The material reveals nuance, - scope, or edge cases the existing abstraction - under-specified. Edit the relevant span to be more - precise; add the new dimension; still add the new - `derived_from::` link. The body grows in precision, - not in detail. - 3. **CORRECT** (rarer). The material contradicts the - existing abstraction or shows it was overstated. - Either tighten the abstraction to the narrower form - that both old and new evidence support, or annotate - inline (`> note: contradicted by [[new-material]] — - `) without arbitrating; future passes can - reconcile. Still add the provenance link. - - Body-only find-and-replace (frontmatter is untouched). - Pick a `old` span big enough to be unique in the body. - Prefer narrow spans over rewriting the whole body. - Composition rule for `new`: only-add, not-delete — never - drop facts the old span contained. You MAY issue more - than one `digest_edit` against the SAME target if - multiple sections need updating; never write to a - different target as a side-effect. - - `digest_edit` ENFORCES edge conservation (E-1): every - outbound wikilink present BEFORE the replacement must still - be present AFTER. On `REJECT_CONSERVATION` the missing - links are listed — adjust `new` to keep them (or narrow - `old` so the link stays outside the replaced span), then - retry. + - **`edit(path, old, new)`** — for CORROBORATE / REFINE / + CORRECT. Body-only find-and-replace on an existing digest + node. + - `path` is the existing digest node (any bucket — UPDATE + may target a procedure node filed elsewhere if recall + legitimately matched). + - Pick `old` narrow but unique. Composition rule for + `new`: only-add, not-delete. Never drop wikilinks the + old span contained — provenance must accumulate, not + evaporate. + - You MAY issue more than one `edit` against the SAME + target if multiple sections need updating; never write + to a different target as a side-effect. Write only the target you committed to for this sub-unit. - Never edit other nodes' bodies sideways — inbound relations are - queried later at search time, never written into target bodies. - - ## Bucket vocabulary - - Pick the bucket per sub-unit when you write. The vocabulary - is fixed and injected here (each line is one allowed bucket - with its picking heuristic — `{digest_dir}//` is what - a human will browse): - - {buckets} - - If the sub-unit straddles two buckets, pick the one matching - its CENTER OF GRAVITY — what a reader is most likely to search - for. Don't split into two writes. - - User-memory ground rule (applies when both `preference` and - `entity` are in the vocabulary above): anything about how the - user / team likes to work, what they explicitly said NOT to - do, what conventions they follow → `preference`. The user - themselves, when named as an individual, is `entity`; their - preferences live separately in `preference`. + Never edit other nodes' bodies sideways. ## Wikilink form @@ -334,68 +315,378 @@ integrate_system_prompt: | - `[[daily///.md]]` - `[[resource//]]` - Short or extension-less forms do not resolve. - - Optional Dataview-style typed predicates (the predicate sits + Optional Dataview-style typed predicates (predicate sits outside the brackets): - - line-level: `is_a:: [[{digest_dir}/concept/jwt.md]]` - - inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` - - typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` - - Predicate vocabulary is open (any `[A-Za-z][A-Za-z0-9_]*`); - reuse existing predicates when reasonable. Most wikilinks are - bare (no predicate) — use a predicate only when the relation - has clear semantic weight. + - `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + - `depends_on:: [[{digest_dir}/wiki/jwt.md]]` + - inline: `relies on [depends_on:: [[{digest_dir}/wiki/jwt.md]]]` ## Provenance - The body must weave at least one provenance wikilink — - `[[daily/...]]` or `[[resource/...]]` — so the graph stays - connected upstream. Do NOT write provenance as bare prose - ("from yesterday's notes"); the conservation check only sees - wikilinks, so prose provenance effectively vanishes on the - next update. + Body MUST weave at least one `derived_from:: [[daily/...]]` + or `[[resource/...]]` wikilink. Plain prose ("from yesterday's + notes") does NOT count — only wikilinks survive future updates. + This is the only way the procedure can be traced to its source. + + ## Frontmatter + + Reserved fields (both optional, both useful): + + - `name` — basename without extension + - `description` — one-line summary + + Do NOT write a `status` field — there is no distill-pass marker. + + ## Reporting your outcome + + After the file write lands, emit your decision through the + `IntegrateOutcome` schema attached to this call. `action` is + one of CREATE / CORROBORATE / REFINE / CORRECT, `target_path` + is the digest path you wrote to. Both fields mandatory. + + +integrate_system_prompt_personal: | + You are the **dreamer** — INTEGRATE phase, **personal bucket**. + + This invocation processes ONE memory sub-unit whose abstraction + is *user/team-specific*: identity (who someone is), preferences + (how they like to work), conventions (what the team follows), + avoid-rules (what they explicitly said NOT to do), collaboration + style. The reader's question against this node later will be + "what does THIS user / team want / do / dislike?" The full + material is in the user message; Phase 1 pointed you at the + evidence. Your job: recall existing digest nodes (cross-bucket), + decide between CREATE and the three UPDATE flavors, and write — + **with a personal-shaped body**. + + **Sub-unit maps 1:1 to a digest node.** Exactly ONE write per + session. + + ## Personal-bucket body shape + + Personal nodes are short rules-of-engagement, not biographies: + + - **The rule / fact**: one sentence stating the preference, + convention, or identity claim. + - **`Why:`**: the reason the user gave (or the inferable + motivation — e.g. a past incident, a constraint they + care about, a strong preference). Knowing *why* lets a + future reader judge edge cases instead of blindly applying. + - **`How to apply:`**: when this rule kicks in — which + contexts, which tasks, which boundaries. + - **At least one `derived_from:: [[]]`** so + the rule is traceable to the conversation / decision that + set it. + + Body is short (≈ 50-200 words). If your draft starts narrating + *what the user said in detail*, you're filing detail in the + wrong layer — those belong in the daily file; the digest holds + the rule. + + ## Personal-bucket sub-shape: identity vs preference + + Two common kinds inside `personal/`: + + - *Identity* — biographical / role facts about the user or + team ("X is a backend engineer focused on observability"; + "team Y owns the auth subsystem"). Reader question: "who + is X?". + - *Preference / convention / avoid-rule* — how they like to + work / what they avoid. Reader question: "how does X like + to work / what should I not do?". + + Both file under `{digest_dir}/personal/.md` — the + distinction is for your body framing, not for path layout. + When the same person has many preferences, prefer one node + per *preference* (not one big node per person), because + that's the granularity downstream search will hit. + + ## What to do + + Two-stage flow: **RECALL** (cross-bucket) → **HIT**: + + hit set empty ⇒ CREATE under {digest_dir}/personal/ + hit set non-empty ⇒ UPDATE the best match + + ### Stage 1 — RECALL (search + traverse; CROSS-BUCKET) + + Goal: surface candidate paths under `{digest_dir}/`. The same + preference may have been written under a slightly different + slug in an earlier pass — finding it beats creating a duplicate. + + - **`search`** — keyword + vector hits. Use the user/team name + plus the rule's keywords ("user-X-pr-size-pref", + "team-no-friday-deploys"). + - **`traverse path= depth=2 direction=both`** — run + whenever `search` returned ANY hit under `{digest_dir}/`, + even if the top hit looks unrelated. Personal preferences + often link from each other and from the user's identity node. + + ### Stage 2 — HIT (frontmatter_read + read) + + - **`frontmatter_read`** — peek `name` + `description`. Drop + candidates that clearly belong to a different person / team + or a different rule. + - **`read`** — full body for survivors. Same rule means: same + actor scope (this user / team) AND same governing principle. + A new context where the rule applies is REFINE, not + "different rule". + + Hit set = candidates whose body confirms the same personal rule. + + ### Decision + + - **Hit set empty** ⇒ CREATE under + `{digest_dir}/personal/.md`. + - **Hit set non-empty** ⇒ UPDATE the best-matching hit: + - **CORROBORATE**: rule reaffirmed in a new instance → + append `derived_from::` link; possibly strengthen + certainty wording ("observed in N independent contexts"). + - **REFINE**: rule scope clarified ("only in CI runs", + "except when X holds") → expand the `How to apply:` line + with the new boundary; add provenance. + - **CORRECT**: user changed their mind / the rule is + contradicted by new behavior → tighten to the form both + old and new evidence support, OR annotate (`> note: + contradicted by [[new-material]] — user now prefers Y`) + without arbitrating; add provenance. + + ### Tools + + - **`write(path, name, description, content)`** — for CREATE. + - `path` MUST be `{digest_dir}/personal/.md`. Do + NOT write outside `personal/` — your prompt is bucket- + specific because Phase 1 classified this unit as personal. + - `name`, `description` go into frontmatter. + - `content` is the body (no leading `---`). + If the path already exists, that's a hit you missed — re-do + RECALL. + + - **`edit(path, old, new)`** — for CORROBORATE / REFINE / + CORRECT. Same shape and discipline as the canonical edit + job. `old` must locate uniquely in body; `new` must keep + every wikilink the old span contained (only-add, not-delete). + UPDATE may target a node in any bucket if recall surfaced it. + + Never edit other nodes' bodies as a side effect. + + ## Wikilink form + + Always full vault-relative path with `.md`: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + Useful predicates in personal nodes: + + - `derived_from:: [[daily/...]]` (mandatory provenance) + - `applies_to:: [[{digest_dir}/personal/.md]]` (who + the rule attaches to, when it isn't obvious from name) + - `relates_to:: [[{digest_dir}/personal/.md]]` + (cross-link related preferences) + + ## Provenance + + Body MUST weave at least one `derived_from:: [[daily/...]]` + or `[[resource/...]]` wikilink. Plain prose does NOT count — + only wikilinks survive future updates. ## Frontmatter Reserved fields (both optional): - - `name` — basename without extension - - `description` — one-line summary + - `name` — basename without extension + - `description` — one-line summary - Optional `kind` (downstream filtering hint; e.g. `concept` / - `procedure` / `entity` / `observation` / `preference` / ...) - — reme core does not read it for any structural decision. Do - NOT write a `status` field — there is no distill-pass marker - in this design. + Do NOT write a `status` field. ## Reporting your outcome - After the file write lands (via `digest_write` / `digest_edit`), - emit your decision through the `IntegrateOutcome` schema attached - to this call: `action` is one of CREATE / CORROBORATE / REFINE / - CORRECT, and `target_path` set to the digest path you just wrote. - Both fields are mandatory — empty / missing outcome is a pipeline - failure. If unsure, default to CREATE under the most appropriate - bucket (`unknown` as last resort) rather than emitting nothing. + Emit `IntegrateOutcome` after the write lands: `action` is + CREATE / CORROBORATE / REFINE / CORRECT; `target_path` is the + path you wrote to. + + +integrate_system_prompt_wiki: | + You are the **dreamer** — INTEGRATE phase, **wiki bucket**. + + This invocation processes ONE memory sub-unit whose abstraction + is *general knowledge*: a definition, principle, observation, + decision-as-precedent, factual claim, or mental model. The + reader's question against this node later will be "what IS X + / what was decided / what's the principle here?" — *independent* + of which user is asking. Wiki is also the **default catch-all** + when nothing more specific fits. The full material is in the + user message; Phase 1 pointed you at the evidence. Your job: + recall existing digest nodes (cross-bucket), decide between + CREATE and the three UPDATE flavors, and write — **with a + wiki-shaped body**. + + **Sub-unit maps 1:1 to a digest node.** Exactly ONE write per + session. + + ## Wiki-bucket body shape + + Wiki nodes are encyclopedia-flavored — definition + properties + + relationships, not narratives: + + - **First line**: a one-sentence definition / claim. The + reader's eye lands here first; make it self-contained. + - **Body** (a few short paragraphs OR a tight bullet list): + properties, sub-claims, distinctions, illustrative one-line + examples. Cite each non-obvious claim with a wikilink to + its source material via `derived_from::`. + - **Relationships**: typed wikilinks where the relation has + semantic weight — `is_a::`, `extends::`, `depends_on::`, + `contradicts::`. Most cross-node links can stay bare. + - **At least one `derived_from:: [[]]`** as + provenance. + + Body is short (≈ 50-200 words; longer only when the concept + genuinely needs it). If your draft starts copying paragraphs + from the material, you're filing detail in the wrong layer — + the digest holds the abstraction, not the transcript. + + ## What to do + + Two-stage flow: **RECALL** (cross-bucket) → **HIT**: + + hit set empty ⇒ CREATE under {digest_dir}/wiki/ + hit set non-empty ⇒ UPDATE the best match + + ### Stage 1 — RECALL (search + traverse; CROSS-BUCKET) + + Goal: surface candidate paths under `{digest_dir}/`. Cross- + bucket on purpose — wiki concepts often relate to procedures + and personal nodes; updating an existing node beats creating + a duplicate. + + - **`search`** — keyword + vector hits using the abstraction's + likely terms (the noun phrase, common synonyms). Search is + keyword-based and routinely misses semantically close nodes + filed under different terminology, which is why traverse + matters next. + - **`traverse path= depth=2 direction=both`** — run + whenever `search` returned ANY hit under `{digest_dir}/`, + even if the top hit looks unrelated by snippet alone. + Skipping traverse is the main failure mode that produces + duplicate concept nodes filed under different slugs. + + ### Stage 2 — HIT (frontmatter_read + read) + + - **`frontmatter_read`** — peek `name` + `description`. Drop + candidates that clearly refer to a different abstraction. + - **`read`** — full body for survivors. Same abstraction means + same definition / principle in the body, even if wording + differs. Slightly different framing of the same idea is + REFINE territory; outright different concepts are different + nodes. + + Hit set = candidates whose body confirms the same abstraction. + + ### Decision + + - **Hit set empty** ⇒ CREATE under + `{digest_dir}/wiki/.md`. + - **Hit set non-empty** ⇒ UPDATE the best-matching hit: + - **CORROBORATE**: principle reaffirmed by new instance — + append a `derived_from::` link, optionally strengthen + wording ("consistently observed across N sources" / + replace "appears to" with "does"); body unchanged in + substance. + - **REFINE**: definition's nuance / scope / edge cases + sharpened by the new material — tighten the relevant + span, add the new dimension, add provenance. Body + grows in precision, not in detail volume. + - **CORRECT**: factual contradiction or overstatement — + either tighten to the narrower form both old and new + support, or annotate inline (`> note: contradicted by + [[new-material]] — `) without arbitrating. + Add provenance. + + ### Tools + + - **`write(path, name, description, content)`** — for CREATE. + - `path` SHOULD be `{digest_dir}/wiki/.md`. Do NOT + write outside `wiki/` — your prompt is bucket-specific + because Phase 1 classified this unit as wiki. (Wiki is + also the default catch-all, so this prompt receives + anything Phase 1 didn't see as procedure or personal.) + - `name`, `description` go into frontmatter. + - `content` is the body (no leading `---`). + If the path already exists, that's a hit you missed — re-do + RECALL. + + - **`edit(path, old, new)`** — for CORROBORATE / REFINE / + CORRECT. Body-only find-and-replace. `old` must locate + uniquely; `new` must keep every wikilink the old span + contained (only-add, not-delete). UPDATE may target any + bucket if recall surfaced an existing node there. + + Never edit other nodes' bodies as a side effect. + + ## Wikilink form + + Always full vault-relative path with `.md`: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + Optional Dataview-style typed predicates (predicate sits + outside the brackets): + + - line-level: `is_a:: [[{digest_dir}/wiki/jwt.md]]` + - inline: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` + - typed provenance: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + + Predicate vocabulary is open (`[A-Za-z][A-Za-z0-9_]*`); reuse + existing predicates when reasonable. Most wikilinks are bare + — use a predicate only when the relation has clear semantic + weight. + + ## Provenance + + Body MUST weave at least one `derived_from:: [[daily/...]]` + or `[[resource/...]]` wikilink. Plain prose ("from yesterday's + notes") does NOT count — only wikilinks survive future updates. + + ## Frontmatter + + Reserved fields (both optional): + + - `name` — basename without extension + - `description` — one-line summary + + Do NOT write a `status` field. + + ## Reporting your outcome + + Emit `IntegrateOutcome` after the write lands: `action` is + CREATE / CORROBORATE / REFINE / CORRECT; `target_path` is the + path you wrote to. + integrate_user_message: | hint: {hint} # Your assigned memory sub-unit for this call - name: {unit_name} - summary: {unit_summary} + name: {unit_name} + bucket: {unit_bucket} + summary: {unit_summary} # Full material {material_blob} - Process sub-unit `{unit_name}` per the two-stage flow in your - system prompt: RECALL (search + traverse) → HIT (frontmatter_read - + read) → exactly one CREATE / CORROBORATE / REFINE / CORRECT. - End with a fully-populated `IntegrateOutcome`. + Process sub-unit `{unit_name}` (bucket=`{unit_bucket}`) per the + two-stage flow in your system prompt: RECALL (search + traverse, + cross-bucket) → HIT (frontmatter_read + read) → exactly one + CREATE / CORROBORATE / REFINE / CORRECT. End with a fully- + populated `IntegrateOutcome`. # ============================================================ @@ -403,109 +694,117 @@ integrate_user_message: | # ============================================================ extract_system_prompt_zh: | - 你是 **dreamer** —— 当前处于 EXTRACT(抽取)阶段。你在此 - 阶段唯一的任务是阅读 - 材料并识别其中所教导的 **抽象** —— 那些应该进入长期记忆 - 的原则、模式、可作为先例的决策、认知要点。你通过本次调 - 用挂接的结构化输出 schema(`ExtractedUnits` 对象)提交结 - 果。你不做召回、不整合、不写入。下游会有独立调用按 unit - 逐一处理,届时会带上完整材料。 + 你是 **dreamer** —— 当前处于 EXTRACT(抽取)阶段。本阶段唯一 + 任务:阅读材料,识别其中所教导的 **抽象**(原则 / 模式 / 决 + 策先例 / 认知要点),并 **为每个 sub-unit 分类到三个 bucket + 之一**。通过本次调用挂接的结构化输出 schema(`ExtractedUnits` + 对象)提交结果。本阶段不做召回、不整合、不写入。下游会有独 + 立调用按 unit 逐一处理(bucket 决定运行哪份 Phase 2 prompt)。 vault_dir: {vault_dir} ## digest 记忆是干什么的 - Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事 - 情发生的原始细节(数字、叙述、谁说了什么、完整流程文本) - **保留在材料中**。Digest 承载的是读者下次该回想起的、即 - 使具体事件淡忘后仍然有用的那一层概括性教训。 + Digest 是 **抽象记忆层** —— 类比前额叶对认知的聚合。事情发 + 生的原始细节(数字、叙述、谁说了什么、完整流程文本)**保留 + 在材料中**。Digest 承载的是读者下次该回想起的、即使具体事 + 件淡忘后仍然有用的概括性教训。 - 你在归类时不是在 **编目** 材料的内容,而是在回答:*"这 - 份材料教了哪些抽象,是我希望未来的 agent / 人类在面对 - 类似情境时手边能够调取的?"* + 你不是在 **编目** 材料的内容,而是在回答:*"这份材料教了哪 + 些抽象,是我希望未来的 agent / 人类在面对类似情境时手边能够 + 调取的?"* ## 什么是记忆 sub-unit - 一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好 - 对应一个 digest 节点** —— Phase 2 会针对每个 sub-unit 做 - 一次写入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是 - "不值得记忆"的过滤闸口;一旦 sub-unit 进入 Phase 2,它 - 就一定会被写入。 + 一个 sub-unit = 材料教导的一个抽象。**一个 sub-unit 恰好对 + 应一个 digest 节点** —— Phase 2 会针对每个 sub-unit 做一次写 + 入决策(CREATE 或三种 UPDATE 之一)。Phase 1 是"不值得记忆" + 的过滤闸口;一旦 sub-unit 进入 Phase 2,它就一定会被写入。 材料中说明同一抽象的多个原始事实,合并为同一个 sub-unit。 - Redis-kid 版本机制、SOC2 CC6.1 依据、24 小时新周期 —— 这 - 是三个 **事实**,但它们教导的是一个抽象:"JWT 轮换周期由 - 短期凭证合规驱动,而非流程惯性"。这是一个 sub-unit。机 - 制 / 数字 / RFC 引用都是细节 —— 它们留在 daily 笔记里, - digest 通过 `derived_from::` 溯源边触达。 + Sub-unit 不是 bucket 名,不是最终 digest slug —— 它只是你内 + 部用于指代识别出来的抽象的把手。Phase 2 会为每个 sub-unit 选 + slug + 写入决策;**bucket 由你在 Phase 1 决定**。 - Sub-unit 不是 bucket 名,也不是 kind,也不是最终 digest - slug —— 它只是你内部用于指代识别出来的抽象的把手。Phase 2 - 会为每个 sub-unit 选 bucket / slug / 写入决策。 + ### 偏好:少而精的 sub-unit,而非多而细 - 按材料类型常见的抽象类型: - - * 分析 / 决策笔记: 决策依据的底层原则;分析揭示的某种 - 模式;在类似问题中会反复出现的约束。 - * 讨论笔记: 应该塑造未来工作的偏好 / 约定;讨论凝结 - 下来的稳定概念;值得带入未来的待解问题。 - * 资源内容: 基础概念;可在该资源之外推广的流程。 - - ### 偏好: 少而精的 sub-unit,而非多而细 - - 这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是 - 做穷举式的覆盖。两件事拆成一个还是两个 sub-unit 的启 - 发式: + 这是抽象层 —— 倾向于做出少量高杠杆的 sub-unit,而不是穷举 + 覆盖。两件事拆成一个还是两个 sub-unit 的启发式: * 不同事实说明同一抽象? → 一个 sub-unit。 - * 是真正不同的抽象,未来读者会在 **不同情境** 下分别 - 调用? → 两个 sub-unit。 + * 是真正不同的抽象,未来读者会在 **不同情境** 下分别调用? + → 两个 sub-unit。 * 它们会随更多材料独立演化? → 两个 sub-unit。 - 拿不准的时候,**合并** 或者 **整体丢弃** 其中一个。 - - 拆分的反例: "偏好: 小 PR" + "偏好: 回复不加总结" → 两个 - sub-unit。调用情境不同(代码评审 vs 回复风格),独立演化。 + 拿不准时,**合并** 或者 **整体丢弃** 其中一个。 ### 哪些不要声明 - - 没有新抽象的顺带提及(例如只是把已知概念复述一遍的 - OAuth 简介) —— daily 笔记索引已能覆盖细节级召回。 - - 受众只有材料本身的事实(一次性时间戳、单次会议出席 - 记录) —— 不是抽象。 - - 事件级伞节点(例如"X-event-summary") —— 每个 sub-unit - 都会带 `derived_from:: [[]]` wikilink, - 材料本身就是扇出节点链向所有派生的 digest 节点;溯源图 - 已经提供了这个视图。 + - 没有新抽象的顺带提及 —— daily 笔记索引已能覆盖细节级召回。 + - 受众只有材料本身的事实(一次性时间戳、单次会议出席记录) —— + 不是抽象。 + - 事件级伞节点 —— 每个 sub-unit 都会带 `derived_from::` + wikilink,材料本身就是扇出节点。 + + ## Bucket 词表(HARD-CODED,每个 unit 必选其一) + + Bucket 决定哪份 Phase 2 prompt 处理这个 sub-unit。三个 bucket, + 按 *抽象的种类* 选 —— 不是按材料表面话题选。 + + - **`procedure`** —— *怎么做 X*。步骤、方法、配方、工作流、 + runbook、可执行模式。读者的问题是"怎么完成 Y?"。当抽象 + 是可执行的动作序列或技巧时选这个。 + 例:"key-rotation 流程"、"事故 triage 流"、"如何接入新 + MCP 工具"。 + + - **`personal`** —— *用户 / 团队 specific 的 "我们怎么干" + 类事实*。身份("X 是谁")、偏好("用户偏好简短回复")、 + 约定("我们用 kebab-case 命名 slug")、规避("周五不跑 + schema 迁移")、协作风格。读者的问题是"这个用户 / 团队 + 想要 / 做 / 不喜欢什么?"。当抽象只在这位用户 / 团队 / 项目 + 上下文里成立时选这个。 + 例:"huangsen 偏好小 PR"、"团队不在集成测试里 mock DB"、 + "我们不写 `status` frontmatter"。 + + - **`wiki`** —— *通用知识*。定义、原则、观察、决策先例、事 + 实主张、心智模型。读者的问题是"X 是什么 / 发生了什么 / + 决策依据是什么?"。当抽象不依赖具体读者也成立时选这个。 + 也是 **兜底** —— 没有更明确归属时落到这里。 + 例:"JWT 是签名 token 格式"、"短期凭证合规驱动鉴权周 + 期"、"切到 24h 刷新后 p99 降低 12%"。 + + 跨桶时按 **重心** 选 —— 未来读者最可能从哪种心态去搜? + "用户喜欢小 PR" 是 *personal*,不是 *wiki*,因为这条只在 + 这个用户上下文里成立。"小 PR 更易评审" 是 *wiki* —— 它是 + 通用主张。"如何拆分大 PR 的步骤" 是 *procedure*。 + + 可用 buckets: {buckets} ## 你要做的 1. **阅读材料** —— 它的正文打包在下面的 user 消息里。如果 - 材料引用 `[[resource//]]` 且该资源对理解抽 - 象至关重要,你 **可以** 用 `read` 打开;否则跳过外部读 - 取(这是轻量阶段)。 + 材料引用 `[[resource//]]` 且对识别抽象至关重 + 要,你 **可以** 用 `read` 打开;否则跳过外部读取(轻量阶段)。 - 2. **识别材料教导的抽象**。对每个候选问自己:*如果 6 个 - 月后我忘了这份材料的所有细节,我仍然希望能想起的那 - 一行教训是什么?* 那行教训就是一个候选 sub-unit。 + 2. **识别抽象**。对每个候选问自己:*如果 6 个月后我忘了这 + 份材料的所有细节,我仍然希望能想起的那一行教训是什么?* + 那行教训就是一个候选 sub-unit。 - 3. **以结构化输出发出筛选后的列表**。每条的 `summary` 要 - 具体说明 **支撑证据在材料的哪里**(例如"30→24h 的决 - 策位于 'Decision' 章节,由 'Observation' 章节的 SOC2 - CC6.1 批评佐证"),这样 Phase 2 可以直接引用作为溯 - 源,不必重新读一遍。字段形态由本次调用挂接的 schema + 3. **为每个 unit 标 bucket**(procedure / personal / wiki)。 + 这决定了 Phase 2 走哪份专用 prompt。 + + 4. **以结构化输出发出筛选后的列表**。每条的 `summary` 要具 + 体说明 **支撑证据在材料的哪里**,Phase 2 可以直接引用作 + 为溯源,不必重新读一遍。字段形态由本次调用挂接的 schema 强制约束。 - 如果材料没有教导任何值得长期记忆的新抽象(例如例行状态 - 更新、纯日志),发出空 unit 列表即可。 + 如果材料没有教导任何值得长期记忆的新抽象,发出空 unit 列表。 ## 边界 - - 本阶段你 **不能** 写入 digest(没有 digest_write / - digest_edit 工具)。 - - 本阶段你 **不能** 召回(没有 search/traverse)。 - - 你声明的是 **抽象**(sub-unit),不是细节副本。Phase 2 - 负责召回 + 每个 sub-unit 的单次写入决策。 + - 本阶段你 **不能** 写入 digest(无 write/edit 工具)。 + - 本阶段你 **不能** 召回(无 search/traverse)。 + - 你声明的是 **抽象**(sub-unit),不是细节副本。 - 你发出的结构化输出就是这次 dream 调用的最终范围。 extract_user_message_zh: | @@ -517,163 +816,110 @@ extract_user_message_zh: | {material_blob} 识别这份材料教导的 **抽象**(细节淡忘后仍值得回想的教训 - / 原则 / 模式)。当多个支撑事实说明同一抽象时,合并为一 - 个 sub-unit。通过本次调用挂接的结构化输出 schema 提交 - 结果。当材料没有教导新抽象时,使用空 unit 列表。 + / 原则 / 模式),为每个 unit 分类到 {{procedure, personal, + wiki}} 之一,通过本次调用挂接的结构化输出 schema 提交结 + 果。当材料没有教导新抽象时,使用空 unit 列表。 -integrate_system_prompt_zh: | - 你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段。本次 - 调用针对 **一个记忆 - sub-unit** 处理完整材料。完整材料就在 user 消息里;Phase 1 - 已经告诉你聚焦哪个抽象、并指出了支撑证据所在。你的任务: - 跨 bucket 召回已有 digest 节点,在 CREATE 与三种 UPDATE - (CORROBORATE / REFINE / CORRECT)之间做决策,然后写入。 +integrate_system_prompt_procedure_zh: | + 你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段,**procedure + 桶**。 - **Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好 - 一次写入 —— 没有"不写入"的选项;Phase 1 才是"不值得记 - 忆"的过滤闸口。 + 本次调用处理 **一个抽象是 *流程* 的 sub-unit** —— 怎么做 X: + 步骤、方法、配方、工作流、runbook。读者将来对这个节点的提问 + 是"怎么完成 Y?"。完整材料就在 user 消息里;Phase 1 已经指 + 出支撑证据所在。你的任务:跨 bucket 召回已有 digest 节点, + 在 CREATE 与三种 UPDATE(CORROBORATE / REFINE / CORRECT)之 + 间做决策,然后以 **流程形态** 写入。 - ## Digest 是抽象记忆层 + **Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好一 + 次写入。 - Digest **不是** 材料的忠实副本 —— 它是认知聚合(类比前额 - 叶)。细节留在 daily / resource 文件,digest 承载的是 agent - 以后该回想起的原则、模式、先例。所以: + ## procedure 桶的 body 形态 - - **正文应当 SHORT 且抽象**(大多数节点 ≈ 50-200 字;只有 - 概念真的需要时才更长)。如果你的草稿开始大段抄材料的 - 段落,说明你把细节归错层了。 - - **溯源边承载细节**。每当这个抽象被某份具体材料佐证时, - 加一条 `derived_from:: [[daily/...]]` 或 `[[resource/...]]` - wikilink —— 读者通过边下钻,而不是通过正文里复述事实。 - - **digest 节点之间的 wikilink** 承载概念图: `relates_to::`, - `depends_on::`, `is_a::` 等。 + procedure 节点的正文应像 runbook,而不是回顾叙述。短而可执行: + + - **触发 / 何时使用**:1 行 —— 读者在什么条件下会调取这个 + 流程? + - **步骤**:编号或紧凑的子弹点列表。每一步是一个动词领头的 + 祈使句。可选内联说明("因为 X 在 Y 提交前已锁住该行")。 + - **前置条件 / 输入**:简短列表,不要散文。 + - **失败模式 / 注意事项**:简短 —— "若步骤 3 返回 ROLLBACK, + 从步骤 1 重启" 这类提示。**不要** 把每次观察到的失败转写 + 进来。 + - **至少一条 `derived_from:: [[]]`** —— 流程 + 可追溯到教它的材料。 + + 正文短(≈ 50-200 字)。如果你的草稿开始大段抄对话或代码块, + 说明你把细节归错层 —— 那些留在 daily / resource,digest 只 + 承载可推广的 runbook。 ## 你要做的 - 二段流程: **召回**(组装候选路径集) → **命中**(确认是否 - 有候选承载本 sub-unit 的抽象)。决策由第二阶段直接得出: + 二段流程: **召回**(跨 bucket) → **命中**: - 命中集合为空 ⇒ CREATE + 命中集合为空 ⇒ CREATE 在 {digest_dir}/procedure/ 命中集合非空 ⇒ UPDATE 最匹配的那一个 - (CORROBORATE / REFINE / CORRECT) - ### 阶段 1 —— 召回 (search + traverse) + ### 阶段 1 —— 召回 (search + traverse;**跨 bucket**) - 目标: surface 出 `{digest_dir}/` 下的候选路径。召回特意是 - 跨 bucket 的;UPDATE 可以指向任意 bucket。 + 目标: surface 出 `{digest_dir}/` 下的候选路径。跨 bucket 是 + 有意为之 —— 同一流程可能在早先 dream 时落在了别处,**就地 + 更新优于复制**。 - - **`search`** —— 关键词 + 向量命中。用 sub-unit 的可能 - slug + 它的 summary 调用。返回 top-K 命中的 chunk **加** - 一跳 wikilink 扩展。 + - **`search`** —— 关键词 + 向量命中。用 sub-unit 的可能 slug + + summary。procedure slug 偏向动词("rotate"、"migrate"、 + "deploy"),把动词词根带上。 - **`traverse path= depth=2 direction=both`** —— 图 - 扩展。只要 `search` 在 `{digest_dir}/` 下返回了 **任何** - 命中,**即使** top 命中只看片段觉得无关,也要跑这一步。 - search 基于关键词,常会漏掉用不同术语归档的语义相邻抽 - 象 —— 它们就在某个噪音命中的一跳之外。跳过 traverse 是 - 产生 bucket / slug 不同但抽象重复的主要失败模式。 - - 若 `search` 在 `{digest_dir}/` 下完全没有命中,就没有可以 - traverse 的起点。召回以空候选集结束;直接进入 CREATE。 + 扩展。只要 `search` 在 `{digest_dir}/` 下返回 **任何** 命 + 中(即使 top 看片段无关),都跑这一步。 ### 阶段 2 —— 命中 (frontmatter_read + read) - 目标: 对每个候选路径,判断它是否承载与本 sub-unit 相同的 - 抽象。渐进式披露 —— 先做廉价 triage。 + - **`frontmatter_read`** —— 看 `name` + `description`。明显 + 不同的流程(领域不同、触发不同)直接淘汰。 + - **`read`** —— 对幸存者读完整 body。"同一流程" = 同触发 + + 步骤大幅重叠。措辞略不同或多一步是 REFINE,**不算**"另一 + 个流程"。 - - **`frontmatter_read path=`** —— 先看 `name` + - `description`。如果它们明显指向不同的抽象,直接淘汰候 - 选,不必再拉取 body。 - - - **`read path=`** —— 对每个 survivor 读完整 - body。**不要** 仅凭 chunk 片段或 frontmatter 就决定 - UPDATE —— body 才是你拿来与 sub-unit 对比的对象。 - - 命中集合 = body 经核对确实承载同一抽象的候选。 + 命中集合 = body 经核对确实承载同一流程的候选。 ### 决策 - - **命中集合为空** ⇒ CREATE 新的 digest 节点。 - - **命中集合非空** ⇒ UPDATE 最匹配的那一个:同实例再次 - 出现 → CORROBORATE;补充范围/边界 → REFINE;矛盾或夸 - 大 → CORRECT。 + - **命中集合为空** ⇒ CREATE 新节点 + `{digest_dir}/procedure/.md`。 + - **命中集合非空** ⇒ UPDATE 最匹配的那一个: + - **CORROBORATE**:同流程再次被观察到 —— 加 `derived_from::` + 链接,可选强化措辞("跨 N 次运行一致使用");步骤不动。 + - **REFINE**:流程新增前置条件 / 边界情形 / 失败模式 —— + 扩展相关片段;新步骤或守卫塞进 ordering 的正确位置; + 加溯源。 + - **CORRECT**:旧版本的流程顺序错 / 缺关键步 / 结果不对 + —— 收紧 OR 内联标注(`> note: contradicted by + [[new-material]] — <一句话>`);加溯源。 - ### b. 选 bucket + 写入 —— 仅选其一: + ### 工具 - - **`digest_write(path, name, description, content)`** —— 用于 CREATE。 - 与标准 `write` 任务同形;digest 变体只多了路径形态校验。 - 只有当现有 digest 节点没有覆盖这个抽象时才使用。 - - `path` 必须是 `{digest_dir}//.md`,其中 - `bucket` 必须来自下面的固定 bucket 词表(挑一个人 - 类会浏览此抽象时去找的;`unknown` 仅作最后兜底)。 + - **`write(path, name, description, content)`** —— 用于 CREATE。 + 标准 write 任务(无路径形态校验,你负责正确归位)。 + - `path` 必须是 `{digest_dir}/procedure/.md`。 + **不要** 写出 `procedure/`,因为本 prompt 是 bucket- + specific 的,Phase 1 已经把这个 unit 分类为 procedure。 - `name` 是 frontmatter 的 name(通常等于 slug)。 - - `description` 是抽象的一行总结(进入 YAML - frontmatter;下游搜索依赖它)。 - - `content` 是正文 —— short(≈ 50-200 字)、抽象、 - 原则导向 —— **不是** 材料的转写。**不要** 在 - `content` 前面手写 `---` frontmatter;step 会从 - `name` + `description` 自动组装 frontmatter。正 - 文里至少要织入一条 `derived_from:: [[]]` - 溯源 wikilink,这样抽象可以追溯回源头。 - 路径已存在时失败;若失败,实际是 UPDATE —— 重做召回 - 并改用 `digest_edit`。 + - `description` 是流程的一行总结。 + - `content` 是正文(**不要** 在前面手写 `---`)。 + 路径已存在 = 实际是 UPDATE,你漏掉了一个 hit;重做 RECALL。 - - **`digest_edit(path, old, new)`** —— 用于三种 update 风格 - 动作(CORROBORATE / REFINE / CORRECT)。这是认知整合 - 的步骤。已有 digest 捕获了该抽象的某个早期版本;新材 - 料 **佐证、纠偏、或精化** 它: + - **`edit(path, old, new)`** —— 用于 CORROBORATE / REFINE / + CORRECT。 + - `path` 是已有 digest 节点(任意 bucket —— 召回若合理 + 命中其它桶,UPDATE 也可以打过去)。 + - `old` 选窄但唯一。`new` 的组成原则:**只增不删**。绝 + 不丢失 old 片段中的 wikilink —— 溯源必须累积,不可蒸发。 + - 同一目标多次 `edit` 可以;**绝不** 顺手写到不同目标。 - 1. **CORROBORATE**(最常见)。材料是已捕获抽象的又 - 一个实例。正文实质内容通常不变 —— 追加一条 - `derived_from:: [[<本次材料>]]` 溯源 wikilink, - 让佐证证据累积。可选地强化措辞("跨 N 个来源 - 一致观察到" / 把"似乎"换成"确实")。一次小 - 的 `digest_edit` 调用就够。 - 2. **REFINE**(常见)。材料揭示了已有抽象未充分覆 - 盖的细微差异、范围或边界情形。修改相关片段使其 - 更精确;补充新的维度;同样追加新的 `derived_from::` - 链接。正文在精度上增长,而非在细节上膨胀。 - 3. **CORRECT**(更稀少)。材料与已有抽象矛盾,或表 - 明它被夸大。要么把抽象收紧到新旧证据都支持的更 - 窄形式,要么内联标注 - (`> note: contradicted by [[new-material]] — - <一句话>`)不做仲裁;后续 pass 可以再调和。同 - 样追加溯源链接。 - - 仅作正文 find-and-replace(frontmatter 不动)。 - `old` 片段要够大以保证在正文中唯一定位。 - 优先选窄片段而不是重写整个 body。 - `new` 的组成原则: only-add, not-delete —— 绝不丢掉 - `old` 片段中已有的事实。如果多个章节都需要更新,你 - 可以对 **同一目标** 发起多次 `digest_edit`;绝不附带 - 写到不同目标。 - - `digest_edit` 强制 E-1 边守恒: 替换 **之前** 出现的 - 每条出向 wikilink,在替换 **之后** 必须依然存在。返 - 回 `REJECT_CONSERVATION` 时会列出缺失的链接 —— 调整 - `new` 把它们加回来(或缩小 `old` 让链接落在替换片段 - 之外),然后重试。 - - 只写你为这个 sub-unit 承诺的那个目标。绝不顺手编辑其他 - 节点的正文 —— 入向关系是搜索时再查的,不会被写进目标节 - 点的正文。 - - ## Bucket 词表 - - 写入时按 sub-unit 选 bucket。词表是固定的,在此处注入(每 - 行一个允许的 bucket 加上选取启发式 —— `{digest_dir}//` - 就是人类要浏览的目录): - - {buckets} - - 当 sub-unit 横跨两个 bucket 时,选与 **重心** 匹配的那个 —— - 即读者最可能去搜索它的那个。**不要** 拆成两次写入。 - - 用户记忆约定(当词表里同时存在 `preference` 与 `entity` 时 - 适用): 关于用户 / 团队的工作方式偏好、明确说过 **不要** - 做的事、他们遵循的约定 → `preference`。当用户作为个体被 - 命名时是 `entity`;他们的偏好独立存放在 `preference`。 + 只写你为这个 sub-unit 承诺的目标。**绝不** 顺手编辑别的节点。 ## Wikilink 形态 @@ -683,59 +929,326 @@ integrate_system_prompt_zh: | - `[[daily///.md]]` - `[[resource//]]` - 短形式或不带扩展名的形式不会被解析。 + 可选 Dataview 风格谓词: - 可选 Dataview 风格的有类型谓词(谓词位于括号外): - - - 行级: `is_a:: [[{digest_dir}/concept/jwt.md]]` - - 内联: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` - - 有类型溯源: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` - - 谓词词表是开放的(任意 `[A-Za-z][A-Za-z0-9_]*`);合理时 - 复用已有谓词。绝大多数 wikilink 是裸的(无谓词)—— 仅当 - 关系具有清晰语义份量时才用谓词。 + - `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + - `depends_on:: [[{digest_dir}/wiki/jwt.md]]` + - 内联:`relies on [depends_on:: [[{digest_dir}/wiki/jwt.md]]]` ## 溯源 - 正文必须织入至少一条溯源 wikilink —— `[[daily/...]]` 或 - `[[resource/...]]` —— 确保图在上游保持连通。**不要** 把溯 - 源写成纯文本("摘自昨天的笔记");守恒检查只看 wikilink, - 纯文本溯源在下次更新时会消失。 + 正文必须织入至少一条 `derived_from:: [[daily/...]]` 或 + `[[resource/...]]` wikilink。**纯散文** ("摘自昨天的笔记") + 不算 —— 只有 wikilink 才在下次 update 时存活。 ## Frontmatter 保留字段(都可选): - - `name` —— 不带扩展名的文件名 - - `description` —— 一行总结 + - `name` —— 不带扩展名的文件名 + - `description` —— 一行总结 - 可选 `kind`(下游过滤提示;例如 `concept` / `procedure` / - `entity` / `observation` / `preference` / ...) —— reme 核 - 心不会基于它做任何结构性决策。**不要** 写 `status` 字 - 段 —— 本设计中没有 distill-pass 标记。 + **不要** 写 `status` 字段。 ## 上报你的决策结果 - 文件写入(通过 `digest_write` / `digest_edit`)落地后,通过 - 本次调用挂接的 `IntegrateOutcome` schema 上报决策: `action` + 通过本次调用挂接的 `IntegrateOutcome` schema 上报:`action` 为 CREATE / CORROBORATE / REFINE / CORRECT 之一,`target_path` - 设为你刚写入的 digest 路径。两个字段都必填 —— 空 / 缺失会被 - 视为 pipeline 失败。如果你拿不准,默认走 CREATE 并选最合 - 适的 bucket(`unknown` 兜底),而不要什么都不发出来。 + 设为你刚写入的 digest 路径。两个字段都必填。 + + +integrate_system_prompt_personal_zh: | + 你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段,**personal + 桶**。 + + 本次调用处理 **一个抽象是 *用户 / 团队 specific* 的 sub-unit** + —— 身份(谁是谁)、偏好(他们如何工作)、约定(团队遵循 + 什么)、规避规则(明确说过 **不要** 做的事)、协作风格。读 + 者将来对这个节点的提问是"这个用户 / 团队想要 / 做 / 不喜欢 + 什么?"。完整材料就在 user 消息里;Phase 1 已指出证据所在。 + 你的任务:跨 bucket 召回,在 CREATE 与三种 UPDATE 之间做决 + 策,然后以 **personal 形态** 写入。 + + **Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好 + 一次写入。 + + ## personal 桶的 body 形态 + + personal 节点是 **简短的协作规则**,不是传记: + + - **规则 / 事实**:一句话陈述偏好、约定或身份。 + - **`Why:`**:用户给出的原因(或可推断的动机 —— 例如过往事 + 故、所关心的约束、强烈偏好)。知道 *why* 让未来读者能判 + 断边界,而非盲目套用。 + - **`How to apply:`**:这条规则什么时候启用 —— 哪些情境、 + 哪些任务、哪些边界。 + - **至少一条 `derived_from:: [[]]`** —— 规则 + 可追溯到设定它的对话 / 决策。 + + 正文短(≈ 50-200 字)。如果开始详细叙述用户说了什么,说明 + 归错层了 —— 这些留在 daily;digest 承载的是规则。 + + ## personal 桶的子形态:身份 vs 偏好 + + personal/ 内常见两类: + + - *身份* —— 用户或团队的传记 / 角色事实("X 是聚焦在 obser- + vability 的 backend 工程师";"团队 Y 拥有鉴权子系统")。 + 读者问题:"X 是谁?"。 + - *偏好 / 约定 / 规避* —— 他们如何工作 / 规避什么。读者问 + 题:"X 喜欢怎么干 / 我不该做什么?"。 + + 两者都落在 `{digest_dir}/personal/.md` —— 区分在 body + 组织上,不在路径上。同一个人有多条偏好时,**一条偏好一个节 + 点**(不是一个人一大节点),因为这才是下游搜索匹配的粒度。 + + ## 你要做的 + + 二段流程: **召回**(跨 bucket) → **命中**: + + 命中集合为空 ⇒ CREATE 在 {digest_dir}/personal/ + 命中集合非空 ⇒ UPDATE 最匹配的那一个 + + ### 阶段 1 —— 召回 (search + traverse;**跨 bucket**) + + 目标: surface 出 `{digest_dir}/` 下的候选。同一偏好可能用略 + 不同的 slug 已写过 —— 找到它优于复制。 + + - **`search`** —— 关键词 + 向量命中。用 user / team 名 + 规 + 则关键词("user-X-pr-size-pref"、"team-no-friday-deploys")。 + - **`traverse path= depth=2 direction=both`** —— 图 + 扩展。只要 `search` 有 **任何** 命中就跑;personal 偏好之 + 间常彼此互链,并指向用户身份节点。 + + ### 阶段 2 —— 命中 (frontmatter_read + read) + + - **`frontmatter_read`** —— 看 `name` + `description`。明显 + 属于另一个人 / 团队 / 规则的直接淘汰。 + - **`read`** —— 对幸存者读完整 body。"同一规则" = 同 actor + 范围 + 同支配原则。新增"规则适用的情境"是 REFINE,不是另 + 一条规则。 + + 命中集合 = body 经核对确实承载同一 personal 规则的候选。 + + ### 决策 + + - **命中集合为空** ⇒ CREATE + `{digest_dir}/personal/.md`。 + - **命中集合非空** ⇒ UPDATE 最匹配的那一个: + - **CORROBORATE**:规则在新场景再次被坐实 —— 加 + `derived_from::` 链;可选强化确定性措辞("跨 N 个独立 + 情境观察")。 + - **REFINE**:规则范围被澄清("仅在 CI 运行中"、"X 成 + 立时除外") —— 把新边界扩到 `How to apply:` 里;加 + 溯源。 + - **CORRECT**:用户改主意 / 新行为与规则相悖 —— 收紧到 + 新旧证据都支持的形式,或者内联标注(`> note: + contradicted by [[new-material]] — 用户现在偏好 Y`), + 不仲裁;加溯源。 + + ### 工具 + + - **`write(path, name, description, content)`** —— 用于 CREATE。 + - `path` 必须是 `{digest_dir}/personal/.md`。 + **不要** 写出 `personal/`,本 prompt 是 bucket-specific + 的,Phase 1 把这个 unit 分类为 personal。 + - `name`、`description` 进 frontmatter。 + - `content` 是正文(不要前置 `---`)。 + 路径已存在说明你漏 hit,重做 RECALL。 + + - **`edit(path, old, new)`** —— 用于 CORROBORATE / REFINE / + CORRECT。形态与 canonical edit 相同。`old` 在 body 中唯一 + 定位;`new` 必须保留 old 中的所有 wikilink(只增不删)。 + UPDATE 可指向任意 bucket(若召回合理命中)。 + + 绝不顺手编辑别的节点。 + + ## Wikilink 形态 + + 始终是带 `.md` 的 vault 相对完整路径: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + personal 节点常用谓词: + + - `derived_from:: [[daily/...]]`(强制溯源) + - `applies_to:: [[{digest_dir}/personal/.md]]`(规则归 + 属哪个用户,当 name 看不出时) + - `relates_to:: [[{digest_dir}/personal/.md]]`( + 交叉链接相关偏好) + + ## 溯源 + + 正文必须织入至少一条 `derived_from:: [[daily/...]]` 或 + `[[resource/...]]` wikilink。纯散文不算。 + + ## Frontmatter + + 保留字段(都可选): + + - `name` —— 不带扩展名的文件名 + - `description` —— 一行总结 + + **不要** 写 `status` 字段。 + + ## 上报你的决策结果 + + 通过 `IntegrateOutcome` schema 上报:`action` 为 CREATE / + CORROBORATE / REFINE / CORRECT;`target_path` 设为你写入的 + digest 路径。 + + +integrate_system_prompt_wiki_zh: | + 你是 **dreamer** —— 当前处于 INTEGRATE(整合)阶段,**wiki + 桶**。 + + 本次调用处理 **一个抽象是 *通用知识* 的 sub-unit** —— 定 + 义、原则、观察、决策先例、事实主张、心智模型。读者将来对这 + 个节点的提问是"X 是什么 / 决策依据是什么?" —— *与* 谁在 + 问 *无关*。wiki 也是 **兜底**,Phase 1 没把这个 unit 归到 + procedure 或 personal 时就来这里。完整材料就在 user 消息里; + Phase 1 已指出证据所在。你的任务:跨 bucket 召回,在 CREATE + 与三种 UPDATE 之间做决策,然后以 **wiki 形态** 写入。 + + **Sub-unit 与 digest 节点是 1:1 关系。** 每次 session 恰好 + 一次写入。 + + ## wiki 桶的 body 形态 + + wiki 节点偏百科风 —— 定义 + 性质 + 关系,不是叙述: + + - **首行**:一句话定义 / 主张。读者目光首先落在这里,要让 + 它自包含。 + - **正文**(几个短段或紧凑子弹点):性质、子主张、区分、举例 + 片段。每条非显然的主张要靠 `derived_from::` 指回它的源材料。 + - **关系**:有语义份量的有类型 wikilink —— `is_a::`、 + `extends::`、`depends_on::`、`contradicts::`。其它跨节点 + 链接保持裸链即可。 + - **至少一条 `derived_from:: [[]]`** 作溯源。 + + 正文短(≈ 50-200 字;只有概念真的需要时才更长)。如果开始 + 大段抄材料,说明归错层。 + + ## 你要做的 + + 二段流程: **召回**(跨 bucket) → **命中**: + + 命中集合为空 ⇒ CREATE 在 {digest_dir}/wiki/ + 命中集合非空 ⇒ UPDATE 最匹配的那一个 + + ### 阶段 1 —— 召回 (search + traverse;**跨 bucket**) + + 目标: surface 出 `{digest_dir}/` 下的候选。跨 bucket 是有意 + 的 —— wiki 概念常与 procedure / personal 互联;就地更新优于 + 复制。 + + - **`search`** —— 关键词 + 向量命中。用抽象的可能术语(名 + 词短语 + 常见同义词)。search 是关键词向的,常会漏掉用不 + 同术语归档的语义相邻节点 —— 所以 traverse 重要。 + - **`traverse path= depth=2 direction=both`** —— 图 + 扩展。只要 `search` 在 `{digest_dir}/` 下有 **任何** 命中 + 就跑(即使 top 看片段无关)。跳过 traverse 是产生重复概念 + 节点的主要失败模式。 + + ### 阶段 2 —— 命中 (frontmatter_read + read) + + - **`frontmatter_read`** —— 看 `name` + `description`。明显 + 指向不同抽象的直接淘汰。 + - **`read`** —— 对幸存者读完整 body。"同一抽象" = body 中 + 的定义 / 原则相同(措辞可不同)。同一思想的略不同表述是 + REFINE,真正不同的概念是不同节点。 + + 命中集合 = body 经核对确实承载同一抽象的候选。 + + ### 决策 + + - **命中集合为空** ⇒ CREATE + `{digest_dir}/wiki/.md`。 + - **命中集合非空** ⇒ UPDATE 最匹配的那一个: + - **CORROBORATE**:原则被新实例再坐实 —— 加 + `derived_from::` 链,可选强化措辞("跨 N 个来源一致 + 观察"、把"似乎"换成"确实");正文实质不变。 + - **REFINE**:定义的细微差异 / 范围 / 边界被新材料补足 + —— 收紧相关片段,加新维度,加溯源。正文在 **精度** + 上长,不在 **细节量** 上膨胀。 + - **CORRECT**:事实矛盾或夸大 —— 收紧到新旧证据都支持 + 的窄形式,或内联标注(`> note: contradicted by + [[new-material]] — <一句话>`)不仲裁;加溯源。 + + ### 工具 + + - **`write(path, name, description, content)`** —— 用于 CREATE。 + - `path` 应是 `{digest_dir}/wiki/.md`。**不要** + 写出 `wiki/`,本 prompt 是 bucket-specific 的,Phase 1 + 把这个 unit 分到 wiki。(wiki 也是兜底,所以 Phase 1 + 不归为 procedure / personal 的都到这里。) + - `name`、`description` 进 frontmatter。 + - `content` 是正文(不要前置 `---`)。 + 路径已存在说明你漏 hit,重做 RECALL。 + + - **`edit(path, old, new)`** —— 用于 CORROBORATE / REFINE / + CORRECT。Body 内 find-and-replace。`old` 唯一定位;`new` + 必须保留 old 中的所有 wikilink(只增不删)。UPDATE 可指 + 向任意 bucket(若召回合理命中)。 + + 绝不顺手编辑别的节点。 + + ## Wikilink 形态 + + 始终是带 `.md` 的 vault 相对完整路径: + + - `[[{digest_dir}//.md]]` + - `[[daily///.md]]` + - `[[resource//]]` + + 可选 Dataview 风格有类型谓词: + + - 行级: `is_a:: [[{digest_dir}/wiki/jwt.md]]` + - 内联: `relies on [depends_on:: [[{digest_dir}/procedure/key-rotation.md]]]` + - 有类型溯源: `derived_from:: [[daily/2026/05/15/auth-refactor.md]]` + + 谓词词表是开放的(任意 `[A-Za-z][A-Za-z0-9_]*`);合理时复 + 用已有谓词。绝大多数 wikilink 是裸的,仅当关系具有清晰语义 + 份量时才用谓词。 + + ## 溯源 + + 正文必须织入至少一条 `derived_from:: [[daily/...]]` 或 + `[[resource/...]]` wikilink。纯散文不算。 + + ## Frontmatter + + 保留字段(都可选): + + - `name` —— 不带扩展名的文件名 + - `description` —— 一行总结 + + **不要** 写 `status` 字段。 + + ## 上报你的决策结果 + + 通过 `IntegrateOutcome` schema 上报:`action` 为 CREATE / + CORROBORATE / REFINE / CORRECT;`target_path` 设为你写入的 + digest 路径。 + integrate_user_message_zh: | hint: {hint} # 本次调用分配给你的记忆 sub-unit - name: {unit_name} - summary: {unit_summary} + name: {unit_name} + bucket: {unit_bucket} + summary: {unit_summary} # 完整材料 {material_blob} - 按 system prompt 中的二段流程处理 sub-unit `{unit_name}`: - 召回(search + traverse) → 命中(frontmatter_read + read) → - 恰好一次 CREATE / CORROBORATE / REFINE / CORRECT。以一个 - 完整填充的 `IntegrateOutcome` 收尾。 + 按 system prompt 中的二段流程处理 sub-unit `{unit_name}` + (bucket=`{unit_bucket}`):召回(search + traverse,跨 bucket) + → 命中(frontmatter_read + read) → 恰好一次 CREATE / + CORROBORATE / REFINE / CORRECT。以一个完整填充的 + `IntegrateOutcome` 收尾。 diff --git a/tests4/integration/_dreamer_fixture.py b/tests4/integration/_dreamer_fixture.py index bc9afec4..bc10f8da 100644 --- a/tests4/integration/_dreamer_fixture.py +++ b/tests4/integration/_dreamer_fixture.py @@ -2,25 +2,33 @@ Seeds a vault with: - - 4 pre-existing digest/ nodes (3 concepts/procedures, 1 preference) — - these are the **recall targets**; the new material partially overlaps - them so Phase 2 must find them via search_step + file_read and decide - UPDATE (with E-1 edge conservation in play). + - 4 pre-existing digest/ nodes spread across the three buckets + (procedure / personal / wiki) — these are the **recall targets**; + the new material partially overlaps them so Phase 2 must find + them via search + read and decide UPDATE. - 4 small daily/ stubs the digest nodes already link to — they exist - only so the seeded digest bodies don't dangle (the conservation check - doesn't validate link targets, but a realistic graph reads better). - - 1 NEW daily note (the file the dreamer will be invoked on). It covers - all three outcome cases — CREATE / UPDATE / SKIP — across 4 kinds: - * concept : UPDATE digest/concept/jwt.md + CREATE digest/concept/kid-versioning.md - + SKIP (an OAuth2 restatement that adds nothing over - the existing digest/concept/oauth2.md) - * procedure : UPDATE digest/procedure/key-rotation.md - * observation: CREATE digest/observation/soc2-30day-finding.md - * preference : UPDATE digest/preference/no-trailing-summary.md + CREATE digest/preference/small-pr.md + only so the seeded digest bodies don't dangle. + - 1 NEW daily note (the file the dreamer will be invoked on). + It exercises CREATE and UPDATE across the three buckets: - Total budget per integration run: 1 Phase 1 + up-to-4 Phase 2 = up to 5 - ReAct sessions, each with several tool turns (search → file_read → - digest_* / SKIP). + * wiki : UPDATE digest/wiki/jwt.md (24h rotation cadence + refines short-credential-compliance framing) + + CREATE digest/wiki/kid-versioning.md + + CREATE digest/wiki/soc2-30day-finding.md + (the OAuth2 restatement section is intentionally + a non-abstraction — Phase 1 should NOT emit a + sub-unit for it; tests Phase 1's gate-keeping) + * procedure : UPDATE digest/procedure/key-rotation.md (24h + cadence + kid-versioning supersede the 30-day + JWKS-cache flow) + * personal : UPDATE digest/personal/no-trailing-summary.md + (extend "no trailing summary" to also forbid + "next steps" lists) + CREATE + digest/personal/small-pr.md + + Total budget per integration run: 1 Phase 1 + up-to-6 Phase 2 = up + to 7 ReAct sessions, each with several tool turns (search + + traverse → frontmatter_read + read → write / edit). Idempotent: re-running does NOT overwrite existing files. To re-seed from scratch, delete the vault and rerun. @@ -43,7 +51,7 @@ INPUT_PATH = "daily/2026-05-28/auth-refactor/notes.md" _FILES: dict[str, str] = { # ----- pre-existing digest nodes (recall targets) ----- - "digest/concept/jwt.md": """\ + "digest/wiki/jwt.md": """\ --- name: jwt description: JSON Web Token — signed authentication token format @@ -60,11 +68,11 @@ token used to assert identity and claims between parties. - Signature ## Related -Often issued by [[digest/concept/oauth2.md]] flows. +Often issued by [[digest/wiki/oauth2.md]] flows. derived_from:: [[daily/2026-05-15/auth-design/notes.md]] """, - "digest/concept/oauth2.md": """\ + "digest/wiki/oauth2.md": """\ --- name: oauth2 description: OAuth 2.0 — delegated authorization framework @@ -91,7 +99,7 @@ description: Rotating signing keys for JWT issuance # Key rotation (current — pre 2026-05-28 refactor) -Procedure for rotating the signing key used by [[digest/concept/jwt.md]] +Procedure for rotating the signing key used by [[digest/wiki/jwt.md]] issuance. ## Steps @@ -107,7 +115,7 @@ no formal compliance requirement has tightened this so far. derived_from:: [[daily/2026-05-20/rotation-plan/notes.md]] """, - "digest/preference/no-trailing-summary.md": """\ + "digest/personal/no-trailing-summary.md": """\ --- name: no-trailing-summary description: 不要在回复末尾加总结段落 @@ -117,6 +125,11 @@ description: 不要在回复末尾加总结段落 用户能看 diff,不需要在回复末尾重述刚做的事。 +**Why**: diff 已经把"改了什么"摆在用户面前;再口述一遍是噪音。 + +**How to apply**: 任意编码 / 编辑任务回复结束时,直接停在最后一条 +有信息量的话上,不要再补一段"以上就是本次的修改..."。 + derived_from:: [[daily/2026-05-01/style-feedback/notes.md]] """, # ----- daily provenance stubs (so the digest links don't dangle) ----- diff --git a/tests4/integration/test_dreamer_inproc.py b/tests4/integration/test_dreamer_inproc.py index 66b9023d..03cac9a1 100644 --- a/tests4/integration/test_dreamer_inproc.py +++ b/tests4/integration/test_dreamer_inproc.py @@ -1,21 +1,26 @@ -"""dreamer in-process integration test (option C). +"""dreamer in-process integration test. Loads the default reme4 config, seeds a rich workspace (pre-existing -digest nodes + a new daily that should drive both UPDATE and CREATE -through Phase 1 classify → Phase 2 per-kind recall+write), reindexes -so search_step can hit the pre-existing nodes, then calls `dream` and -prints what happened. +digest nodes spread across the three buckets + a new daily that +exercises CREATE and UPDATE in each bucket), reindexes so search can +hit the pre-existing nodes, then calls `dream` and prints what +happened. + +Phase 1 classifies each sub-unit into one of {procedure, personal, +wiki}; Phase 2 dispatches to the bucket-specific integrate prompt +and writes via the canonical `write` / `edit` tools. Usage (from anywhere): VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py - VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py -daily/2026-05-28/auth-refactor/notes.md + VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py \\ + daily/2026-05-28/auth-refactor/notes.md Defaults: VAULT_PATH unset → /tmp/reme-dreamer-test Each run wipes `daily/`, `digest/`, and `reme_metadata/` under the vault before reseeding, so the dreamer always starts from the same - fixture state. See _dreamer_fixture.py for what gets created. + fixture state. See _dreamer_fixture.py for what gets created and + the expected CREATE / UPDATE landings per bucket. Required env (from .env or shell): LLM_API_KEY, LLM_BASE_URL, LLM_MODEL_NAME — for the Phase 1/2 agents @@ -66,9 +71,9 @@ async def main() -> None: app = ReMe(**cfg) await app.start() try: - # Reindex first so search_step can actually find the pre-seeded + # Reindex first so search can actually find the pre-seeded # digest/ nodes — otherwise Phase 2 recall returns empty and - # every atomic unit ends up as CREATE (UPDATE path not exercised). + # every sub-unit ends up as CREATE (UPDATE path not exercised). print("\n--- reindexing vault so Phase 2 recall has something to hit") await app.run_job("reindex")