mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-29 01:41:38 +00:00
refactor(steps): remove deprecated digest edit/write steps
This commit is contained in:
huangsen
parent
87eca2a6de
commit
643d36a4b6
11 changed files with 1206 additions and 1181 deletions
|
|
@ -1,147 +0,0 @@
|
|||
---
|
||||
name: reme-service
|
||||
description: Use this skill whenever the user references their personal vault (markdown notes managed by the `reme` MCP, service-tier surface, backed by reme4), or when there's a meaningful session outcome to record / a question that prior work might answer. Triggers include "what do I know about X", "did I work on Y before", "save this", "记下", "落盘", "提炼", "vault", any mention of resource/ / daily/ / digest/ files, or recognizing that a non-trivial session outcome should be recorded. Skill follows a 3-phase paradigm (Recall / Log / Distill) over a 4-tier lifecycle (external channel → resource → daily → digest). Log + Distill route to two SERVICE LAYER MCP tools — `synchronizer` and `digester` — whose internal ReActAgents run the LLM loop INSIDE reme4. Inbound assets from external channels land via `upload` into `resource/<date>/` (service-only). All other tools (search / traverse / file_list / file_read / file_write / file_append / file_stat / frontmatter_*) are shared atomic primitives — whole-file CRUD covers all body changes; `frontmatter` is the one sliced RUD surface (YAML is structured data — surgical key edits cannot be safely emulated with string-substitution on the body).
|
||||
---
|
||||
|
||||
# vault — service tier (reme4)
|
||||
|
||||
The vault is a personal markdown knowledge base managed by the `reme` MCP server. **Service tier**: two service-layer LLM-driven tools (`synchronizer`, `digester`) + the service-only resource-ingest primitive (`upload`) + the full shared atomic primitive surface. Log + Distill phases hand off to the service layer; the R-M-W loops run **inside reme4** in those tools' internal ReActAgents.
|
||||
|
||||
## Business objects
|
||||
|
||||
- **Resource bucket** — passive ingest from external channels. `resource/<YYYY-MM-DD>/` is a flat folder keyed by the day the asset was received, containing the assets themselves (any file type), a `meta.json` array of provenance rows (channel / source / received_at / description), and a derived `<date>.md` view assembled from meta.json. **One ingest path only**: the `upload` tool. Read-only for everything else (synchronizer / digester / hand-edits never write here).
|
||||
- **Daily note** — hot, streaming fact log of one thread. Single file `daily/<YYYY-MM-DD>/<slug>.md`; everything worth keeping (verbatim user prompt, key tool output, intermediate data) inlined inside the body. One upstream writer per note; every other consumer treats it as read-only. Inbound channel assets do NOT live here — those go to the resource bucket and are referenced via `[[resource/<date>/<name>]]` wikilinks in the note's `## References` section when the task consumes them.
|
||||
- **digest node** — cold, curated long-lived cognition. `digest/<slug>/<slug>.md` (or nested at any depth: `digest/<scope>/<slug>/<slug>.md`). Each scope folder must contain `<folder>/<folder>.md` as its canonical entry. **Slugs are globally unique under `digest/`** — a folder name appears at most once anywhere in the tree.
|
||||
|
||||
Lifecycle: **external channel → `upload` → resource/<date>/ → session work + daily folder → distill → digest node**. Each tier is one-way downstream. Inbound assets are passive (someone sends you a file); daily materials are active (you fetched / produced them during a task); digest entries are distilled cognition. The distill marker is the daily's `status` frontmatter — a **daily-tier convention owned by the Digester** (reme core reserves only `name` / `description`; `status` is an extra used exclusively by Sync/Digester). Convention: absent ≡ `pending`; the Digester flips it to `completed` (or `skipped`) once it has processed the daily. Find unprocessed work by listing `daily/` and `frontmatter_read`-ing each summary — the ones with no `status` are pending.
|
||||
|
||||
## Tool surface (service tier)
|
||||
|
||||
| Group | Tools | Where the work runs |
|
||||
|---|---|---|
|
||||
| **Service layer (LLM-driven)** | `synchronizer`, `digester` | **Inside reme4** — internal ReActAgent |
|
||||
| **Service-only ingest** | `upload` (external channel → `resource/<date>/`) | reme4 thin primitive (no LLM) |
|
||||
| Shared retrieve | `search`, `traverse` | reme4 thin primitive (no LLM) |
|
||||
| Shared read | `file_list`, `file_read`, `file_stat`, `frontmatter_read` | reme4 thin primitive (no LLM) |
|
||||
| Shared write | `file_write`, `file_append`, `file_edit`, `frontmatter_update`, `frontmatter_delete` | reme4 thin primitive (no LLM) |
|
||||
| Shared file ops | `file_move`, `file_delete`, `file_download` | reme4 thin primitive (no LLM) |
|
||||
| Shared daily | `daily_read`, `daily_write`, `daily_list`, `daily_reindex` | reme4 thin primitive (no LLM) |
|
||||
|
||||
The shared block is identical to expert tier; what makes this **service** tier is the two service-layer tools at the top plus the service-only `upload` ingest primitive.
|
||||
|
||||
## 3-Phase paradigm
|
||||
|
||||
### Phase 1: Recall
|
||||
|
||||
**What** — retrieve relevant context (chunks ranked by RRF-fused vector + BM25 score, with optional wikilink expansion via the file graph).
|
||||
**Triggers** — intent-driven only: "what do I know about X" / "did I work on Y" / "what's connected to [[Z]]" / task needs prior methodology.
|
||||
**How** —
|
||||
- `search(query, limit?, expand_links?, ...)` for hybrid chunk retrieval.
|
||||
- `traverse(path, depth?, direction?)` to chase a seed file's wikilink neighborhood.
|
||||
- `file_list` / `file_read` / `file_stat` / `frontmatter_read` for primary-key reads.
|
||||
|
||||
```
|
||||
search query="auth refactor decisions" limit=5
|
||||
search query="see [[张三.md]]"
|
||||
traverse path="digest/zhang-san/zhang-san.md" depth=1
|
||||
```
|
||||
|
||||
### Phase 2: Log (service layer)
|
||||
|
||||
**What** — digest the recent conversation slice into a daily note. The Synchronizer's internal ReActAgent picks a slug, writes the note (everything inlined into a single file), and handles continuation (same slug → same file).
|
||||
**Triggers** — (a) intent-driven: meaningful fact / output / decision just landed; (b) **PreCompact hook**: prompt fires to dump volatile state.
|
||||
**How** — `synchronizer(messages, note?)`.
|
||||
|
||||
```
|
||||
synchronizer
|
||||
messages:
|
||||
- {role: user, content: "let's design the auth refactor"}
|
||||
- {role: assistant, content: "two options: JWT vs session..."}
|
||||
- {role: user, content: "go with JWT + refresh token rotation"}
|
||||
note: "auth refactor" # optional hint to bias the slug
|
||||
```
|
||||
|
||||
The Synchronizer reads the conversation, picks a stable slug (or reuses an existing one when `note` matches), writes `daily/<today>/<slug>.md`, and returns a `SynchronizerResult` audit (`note` path, `summary` of the just-written note, `actions`). Surface the summary verbatim if the user wants to see what landed.
|
||||
|
||||
**Surgical edits** (without going through Synchronizer's LLM loop):
|
||||
- `daily_read(slug, date?)` to probe / merge — returns body in `answer` and the parsed frontmatter dict in metadata. `exists: false` = fresh, `exists: true` = upsert.
|
||||
- `daily_write(slug, body, frontmatter?, date?, overwrite?)` for a full-note write — `overwrite=false` (default, idempotent skip-if-exists; mirrors the old `daily_resolve` probe) for fresh threads, `overwrite=true` for UPDATE after a `daily_read`. Auto-mkdirs the day folder and refreshes the day index.
|
||||
- `file_append(path, content)` for cheap end-of-file extensions to trailing sections (`## Progress`, `## Findings`, `## Decisions`) — saves the read-modify-write round-trip.
|
||||
- `frontmatter_update(path, metadata={key: value, ...})` to merge one or more frontmatter keys (call `daily_reindex` afterward if you touched `name` / `description`).
|
||||
- `frontmatter_delete(path, keys=[...])` to drop frontmatter keys.
|
||||
- For mid-body edits on a daily note, `daily_read` then `daily_write overwrite=true`. For non-daily paths, `file_read` then `file_edit` (string substitution) or `file_write` (full body) — there's no body/section slice tool; YAML is the only structured surface that earns its own RUD package.
|
||||
|
||||
Use these when you know exactly what to write; use `synchronizer` when you want the service layer to decide what's worth keeping from the conversation.
|
||||
|
||||
### Phase 3: Distill (service layer)
|
||||
|
||||
**What** — promote daily notes into the digest knowledge graph. The Digester's internal ReActAgent reads each daily note (single-file inline content), looks up existing digest nodes (globally unique slugs — same slug at any nesting depth is the same node), applies the R-M-W decision rules (CREATE / UPDATE / MOVE; mere mentions with no own-node substance are left as-is). Relations are recorded as typed wikilinks in the source node's body (`predicate:: [[X.md]]`); target bodies are never edited (backlinks come from `traverse direction=backward` at query time). After each daily note is processed, the Digester flips its `status` frontmatter to `completed` (or `skipped` if nothing was lifted) — **that flip IS the distill marker** (a daily-tier convention the Digester owns; absent ≡ pending). The Digester scans `daily/` and `frontmatter_read`s each note, picking the ones whose `status` is absent.
|
||||
**Triggers** — (a) intent-driven: task wraps and the working set is ready; (b) **SessionEnd hook**: prompt fires to call `digester` once.
|
||||
**How** — `digester(daily_paths, hint?)`.
|
||||
|
||||
```
|
||||
digester
|
||||
daily_paths:
|
||||
- daily/2026-05-17/auth-refactor
|
||||
- daily/2026-05-17/perf-bench
|
||||
hint: "End-of-task distillation — focus on the auth decisions; perf-bench is a methodology dump."
|
||||
```
|
||||
|
||||
Returns a `DistillResult` (`used_llm`, `skipped`, `daily_read`, `summary`, `error`). Surface the `summary` verbatim.
|
||||
|
||||
**Cold-path rule**: handoff once at task wrap, not per turn.
|
||||
|
||||
## Inbound channel ingest (outside the 3-phase loop)
|
||||
|
||||
When the user hands you an externally-received asset (file from wechat / email / browser / api / ...), land it in the resource bucket before doing anything else:
|
||||
|
||||
```
|
||||
upload
|
||||
path: /tmp/report-q1.pdf
|
||||
channel: wechat
|
||||
source: design-group
|
||||
description: Q1 sales report
|
||||
```
|
||||
|
||||
The tool copies the file into `resource/<today>/<basename>`, appends a `ResourceEntry` to that day's `meta.json`, and regenerates `resource/<today>/<today>.md`. Returns `{date, name, path}` — surface `path` so the user knows where the asset landed. If a downstream task consumes the asset, reference it from the daily note's References section with `[[resource/<date>/<name>]]` (the canonical resource path) rather than inlining the file.
|
||||
|
||||
Triggers — user phrases like "save this file", "上传这个", "存一下刚收到的", or a channel hook that hands you an inbound payload.
|
||||
|
||||
## Trigger → Phase quick reference
|
||||
|
||||
| Trigger | Phase | What you do |
|
||||
|---|---|---|
|
||||
| User hands you an inbound asset from an external channel | Ingest | `upload(path=..., channel=..., source=?, description=?)` |
|
||||
| User asks about prior work / [[X]] | Recall | `search` / `traverse` |
|
||||
| Fact lands during task | Log | `synchronizer(messages=[...])` |
|
||||
| Surgical edit needed | Log | `daily_read` / `daily_write` / `file_append` / `frontmatter_update` / `frontmatter_delete` / `daily_reindex` |
|
||||
| **PreCompact hook** fires | Log (urgent dump) | `synchronizer(messages=[...], note=...)` |
|
||||
| Task wraps | Distill | `digester(daily_paths=[...])` |
|
||||
| **SessionEnd hook** fires | Log + Distill | `synchronizer` then `digester` |
|
||||
|
||||
## Protocol (the rules every write must respect)
|
||||
|
||||
@../../../../protocol.md
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- ❌ Picking a fresh `note` slug on every `synchronizer` call within the same logical thread → fragments the thread. **Reuse the slug.**
|
||||
- ❌ Calling `digester` per turn → it's a handoff tool, not a per-turn tool. Once at end-of-task is the rule.
|
||||
- ❌ Calling `digester` on a daily whose `status` is already `completed` or `skipped` → it'll be a no-op; don't keep re-pushing. (The Digester's own pending scan — `file_list` + per-item `frontmatter_read` — filters those out for you.)
|
||||
- ❌ Creating a new `digest/X/x.md` when `X` already exists somewhere else under `digest/` (e.g. `digest/people/X/x.md`) — slugs are globally unique; reuse the existing node and fold the new facts in.
|
||||
- ❌ Manually `file_write`-ing under `digest/` instead of going through `digester` — digest nodes are the Digester's domain. (You can still do it for one-off corrections; just don't bypass the service layer for routine distillation.)
|
||||
- ❌ Writing `status` from outside the Digester — `status` is a Digester-owned daily-tier convention (enum `pending` / `completed` / `skipped`; absent ≡ pending). Flipping it from a hand-written tool call makes the note look already-processed (the Digester's pending scan skips it) and the next `digester` invocation never picks it up.
|
||||
- ❌ Using `file_write` (full-file replacement) to flip one frontmatter key — use `frontmatter_update`.
|
||||
- ❌ Using `file_write` to extend trailing sections like `## Progress` — use `file_append`; saves the R-M-W round-trip and the prompt tokens of echoing the whole body back.
|
||||
- ❌ Writing under `resource/` from anything other than `upload` — that bucket is the passive ingest contract. Hand-edits / `synchronizer` / `digester` must never touch it.
|
||||
- ❌ Inlining an inbound asset into the daily note body — leave it in `resource/<date>/<name>` and reference it via `[[resource/<date>/<name>]]` in the note's `## References` section. Daily notes are single-file; inbound assets stay in `resource/`.
|
||||
- ❌ Writing short-form (`[[Alice]]`) or no-extension (`[[k/x]]`) wikilinks — they don't resolve. Always full path relative to the vault with `.md`: `[[digest/alice/alice.md]]`.
|
||||
|
||||
## What you DON'T have to think about
|
||||
|
||||
- Slug uniqueness within a thread — `synchronizer` / `digester` pick paths and reuse existing notes; wikilinks are literal full paths, so two different paths never silently merge.
|
||||
- Status flips — the Digester writes `status=completed` (or `skipped`) per processed daily note; that frontmatter flag IS the distill marker, so it's never optional but you never write it yourself.
|
||||
- Frontmatter schema — only `name` / `description` / `status` are reserved (all optional); the protocol defines opinionated default axes (`lifecycle` / `scope` / `source` / `role`) but enforcement is caller-side, not protocol-side.
|
||||
- Pending-vs-digest bookkeeping — the digester scans `daily/` and `frontmatter_read`s each note to find ones whose `status` is absent; it maintains the queue.
|
||||
|
||||
If you need fine-grained control over every R-M-W decision visible in the main session's tool log, switch to [reme-expert](../reme-expert) — same shared tools, no service layer, plus a subagent that owns the Distill LLM loop in its own context window.
|
||||
|
|
@ -25,8 +25,7 @@ class OpenAIAsLLM(BaseAsLLM):
|
|||
"""OpenAI chat model wrapper."""
|
||||
|
||||
async def _start(self) -> None:
|
||||
kwargs = dict(self.kwargs)
|
||||
self.model = OpenAIChatModel(**kwargs)
|
||||
self.model = OpenAIChatModel(**self.kwargs)
|
||||
|
||||
async def _close(self) -> None:
|
||||
if self.model is not None:
|
||||
|
|
|
|||
|
|
@ -53,9 +53,6 @@ class BaseJob(BaseComponent):
|
|||
raise ValueError(f"Unregistered backend '{config.backend}' of type '{ComponentEnum.STEP}'")
|
||||
params = config.model_dump()
|
||||
params["app_context"] = self.app_context
|
||||
# Inherit app-level language unless the step's own config overrides it.
|
||||
if not params.get("language") and self.app_context is not None:
|
||||
params["language"] = getattr(self.app_context.app_config, "language", "") or ""
|
||||
return step_cls, params
|
||||
|
||||
def _build_steps(self) -> list["BaseStep"]:
|
||||
|
|
|
|||
|
|
@ -9,8 +9,6 @@ from .common.stream_demo import StreamDemoStep1, StreamDemoStep2
|
|||
from .common.version import VersionStep
|
||||
from .evolve.auto_memory import AutoMemoryStep
|
||||
from .evolve.dream.cron_dreamer import CronDreamer
|
||||
from .evolve.dream.digest_edit import DigestEditStep
|
||||
from .evolve.dream.digest_write import DigestWriteStep
|
||||
from .evolve.dream.dreamer import Dreamer
|
||||
from .file_io.daily_create import DailyCreateStep
|
||||
from .file_io.daily_list import DailyListStep
|
||||
|
|
@ -77,8 +75,6 @@ __all__ = [
|
|||
"WatchChangesStep",
|
||||
# evolve.dream
|
||||
"CronDreamer",
|
||||
"DigestEditStep",
|
||||
"DigestWriteStep",
|
||||
"Dreamer",
|
||||
# transfer
|
||||
"DownloadStep",
|
||||
|
|
|
|||
|
|
@ -1,19 +1,17 @@
|
|||
"""dream — auto-dream pipeline: classify + integrate via constrained digest tools.
|
||||
"""dream — auto-dream pipeline: extract abstractions, then integrate per
|
||||
sub-unit using bucket-specific Phase 2 prompts.
|
||||
|
||||
Four steps:
|
||||
Two steps:
|
||||
|
||||
dreamer — 2-phase ReAct workflow (extract memory sub-units,
|
||||
then integrate per sub-unit via the digest tools).
|
||||
cron_dreamer — daily wrapper around dreamer; scans today's
|
||||
daily/ + resource/ files and runs dream_one on each.
|
||||
digest_write_step — constrained WriteStep that creates a new
|
||||
digest/<bucket>/<slug>.md.
|
||||
digest_edit_step — constrained EditStep that find-and-replaces in
|
||||
an existing digest node, enforcing E-1 edge
|
||||
conservation.
|
||||
dreamer — 2-phase ReAct workflow (extract memory sub-units
|
||||
tagged with bucket, then integrate per sub-unit
|
||||
via the canonical write/edit tools).
|
||||
cron_dreamer — daily wrapper around dreamer; scans today's
|
||||
daily/ + resource/ files and runs dream_one on each.
|
||||
|
||||
Phase 2 uses the canonical ``write`` / ``edit`` jobs directly — no
|
||||
constrained variants. Bucket placement is prompt-level discipline.
|
||||
"""
|
||||
|
||||
from . import cron_dreamer # noqa: F401 -- @R.register("cron_dreamer_step")
|
||||
from . import digest_edit # noqa: F401 -- @R.register("digest_edit_step")
|
||||
from . import digest_write # noqa: F401 -- @R.register("digest_write_step")
|
||||
from . import dreamer # noqa: F401 -- @R.register("dreamer_step")
|
||||
|
|
|
|||
|
|
@ -1,125 +0,0 @@
|
|||
"""``digest_edit_step`` — constrained EditStep that targets ``digest/<bucket>/<slug>.md``.
|
||||
|
||||
Subclasses :class:`EditStep`. On top of the generic find-and-replace,
|
||||
this step adds:
|
||||
|
||||
* **Path-shape validation** — the target must be ``digest/<bucket>/<slug>.md``
|
||||
with ``<bucket>`` in the configured set.
|
||||
* **Existence check** — the file must already exist (use
|
||||
:class:`DigestWriteStep` for new nodes).
|
||||
* **E-1 strong edge conservation** — the wikilink set in the file BEFORE
|
||||
the replacement must be a subset of the wikilink set AFTER. The check
|
||||
runs as a preflight: the replacement is simulated, links are compared,
|
||||
and the actual write is delegated to ``super().execute()`` only if
|
||||
conservation holds. On violation the step returns
|
||||
``REJECT_CONSERVATION`` (with the missing edges) and the file on disk
|
||||
is left untouched.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import frontmatter
|
||||
|
||||
from .digest_write import _validate_digest_path, bucket_names, normalize_buckets
|
||||
from ...file_io._file_io import read_file_safe
|
||||
from ...file_io.edit import EditStep
|
||||
from ....components import R
|
||||
from ....utils.wikilink_handler import WikilinkHandler
|
||||
|
||||
|
||||
@R.register("digest_edit_step")
|
||||
class DigestEditStep(EditStep):
|
||||
"""EditStep variant that enforces digest path shape + E-1 edge conservation.
|
||||
|
||||
``digest_dir`` is sourced from ``app_context.app_config.digest_dir`` at
|
||||
execute time — same convention as the ``daily_*`` steps.
|
||||
"""
|
||||
|
||||
def __init__(self, buckets=None, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.buckets = normalize_buckets(buckets)
|
||||
|
||||
def _reject(self, message: str, **meta) -> None:
|
||||
assert self.context is not None
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"REJECT: {message}"
|
||||
if meta:
|
||||
self.context.response.metadata.update(meta)
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
raw = str(self.context.get("path") or "")
|
||||
old = self.context.get("old")
|
||||
new = self.context.get("new")
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
|
||||
err = _validate_digest_path(raw, bucket_names(self.buckets), digest_dir)
|
||||
if err:
|
||||
self._reject(err)
|
||||
return None
|
||||
|
||||
abs_path = (Path(self.vault_path) / raw).resolve()
|
||||
if not abs_path.exists():
|
||||
self._reject(f"{raw} does not exist; use digest_write instead")
|
||||
return None
|
||||
|
||||
# Preflight conservation check: simulate the find-and-replace, compare
|
||||
# wikilink sets before vs after, refuse the write if any edge is dropped.
|
||||
# We let super().execute() re-validate `old in body` and surface its
|
||||
# own error if old is empty or missing.
|
||||
if old is not None and new is not None and str(old) != "":
|
||||
preview_text = await _preview_replacement(abs_path, str(old), str(new))
|
||||
if preview_text is not None:
|
||||
missing = _missing_edges(
|
||||
await _read_text(abs_path),
|
||||
preview_text,
|
||||
raw,
|
||||
)
|
||||
if missing:
|
||||
missing_repr = sorted(f"[[{t}]]" + (f" (predicate={p})" if p else "") for t, p in missing)
|
||||
self._reject(
|
||||
(
|
||||
f"REJECT_CONSERVATION: replacement drops {len(missing)} edge(s) "
|
||||
"the old body had. E-1 strong-conservation: every outbound wikilink "
|
||||
"in the old body MUST appear in the new body. "
|
||||
f"Missing: {', '.join(missing_repr)}. "
|
||||
"Adjust `new` to keep the missing links and retry."
|
||||
),
|
||||
conservation_violation={
|
||||
"target_path": raw,
|
||||
"missing": sorted(list(missing)),
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
return await super().execute()
|
||||
|
||||
|
||||
async def _read_text(abs_path: Path) -> str:
|
||||
text, _ = await read_file_safe(abs_path)
|
||||
return text
|
||||
|
||||
|
||||
async def _preview_replacement(abs_path: Path, old: str, new: str) -> str | None:
|
||||
"""Return the full file text as it WOULD look after EditStep's replace.
|
||||
|
||||
Mirrors EditStep's body-only replacement (frontmatter is left untouched).
|
||||
Returns ``None`` if the replacement isn't possible (``old`` not in body) so
|
||||
the caller can defer the error to super().execute()."""
|
||||
raw_text, _ = await read_file_safe(abs_path)
|
||||
post = frontmatter.loads(raw_text)
|
||||
body = post.content
|
||||
if old not in body:
|
||||
return None
|
||||
post.content = body.replace(old, new)
|
||||
new_text = frontmatter.dumps(post) if post.metadata else post.content
|
||||
if not new_text.endswith("\n"):
|
||||
new_text += "\n"
|
||||
return new_text
|
||||
|
||||
|
||||
def _missing_edges(old_text: str, new_text: str, path: str) -> set[tuple[str, str | None]]:
|
||||
"""Edges present in ``old_text`` but absent from ``new_text`` (E-1 deltas)."""
|
||||
old_set = {(link.target_path, link.predicate) for link in WikilinkHandler.extract_links(old_text, path)}
|
||||
new_set = {(link.target_path, link.predicate) for link in WikilinkHandler.extract_links(new_text, path)}
|
||||
return old_set - new_set
|
||||
|
|
@ -1,140 +0,0 @@
|
|||
"""``digest_write_step`` — constrained WriteStep that targets ``digest/<bucket>/<slug>.md``.
|
||||
|
||||
Subclasses :class:`WriteStep`. The only thing this step adds on top of the
|
||||
generic file write is path-shape validation:
|
||||
|
||||
* ``path`` must look like ``digest/<bucket>/<slug>.md`` (depth 1, ``.md`` suffix).
|
||||
* ``<bucket>`` must be one of the configured ``buckets`` (defaults to the
|
||||
dreamer's :data:`DEFAULT_BUCKETS`).
|
||||
* The target must NOT already exist — for in-place updates the agent uses
|
||||
:class:`DigestEditStep` instead.
|
||||
|
||||
Everything else (atomic write semantics, encoding, parent-dir creation,
|
||||
frontmatter handling) is inherited from :class:`WriteStep`.
|
||||
|
||||
Bucket vocabulary lives in this module (NOT in the dreamer prompt template)
|
||||
so it can be swapped / extended without touching the prompt. Future direction:
|
||||
externalize via app-config injection; the ``Dreamer`` constructor already
|
||||
accepts an override.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from ...file_io.write import WriteStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
# Each bucket carries a name (the filesystem folder under ``digest/``) and a
|
||||
# one-line description that the Phase 2 prompt renders into the bucket-picking
|
||||
# heuristic block at runtime.
|
||||
DEFAULT_BUCKETS: tuple[dict[str, str], ...] = (
|
||||
{
|
||||
"name": "concept",
|
||||
"description": 'definitions, principles, mental models ("what IS X?")',
|
||||
},
|
||||
{
|
||||
"name": "procedure",
|
||||
"description": 'steps, methods, recipes ("how do I do X?")',
|
||||
},
|
||||
{
|
||||
"name": "entity",
|
||||
"description": "specific named things (person, system, tool, project)",
|
||||
},
|
||||
{
|
||||
"name": "observation",
|
||||
"description": 'findings, results, decisions with rationale ("what happened / was decided?")',
|
||||
},
|
||||
{
|
||||
"name": "preference",
|
||||
"description": 'user / team / agent collaboration rules ("how does X like to work / what to avoid?")',
|
||||
},
|
||||
{
|
||||
"name": "unknown",
|
||||
"description": "fallback when no specialized bucket fits (first-class, not a failure state)",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def normalize_buckets(buckets) -> tuple[dict[str, str], ...]:
|
||||
"""Normalize a caller-supplied bucket spec into ``tuple[{name, description}, ...]``.
|
||||
|
||||
Accepts ``None`` (→ :data:`DEFAULT_BUCKETS`), tuple/list of dicts (returned
|
||||
as-is), or tuple/list of strings (legacy — each wrapped with an empty
|
||||
description).
|
||||
"""
|
||||
if not buckets:
|
||||
return DEFAULT_BUCKETS
|
||||
first = next(iter(buckets))
|
||||
if isinstance(first, dict):
|
||||
return tuple(buckets)
|
||||
return tuple({"name": str(b), "description": ""} for b in buckets)
|
||||
|
||||
|
||||
def bucket_names(buckets) -> tuple[str, ...]:
|
||||
"""Extract just the name field of each bucket — used for path-membership checks."""
|
||||
if not buckets:
|
||||
return ()
|
||||
first = next(iter(buckets))
|
||||
if isinstance(first, dict):
|
||||
return tuple(b["name"] for b in buckets)
|
||||
return tuple(buckets)
|
||||
|
||||
|
||||
@R.register("digest_write_step")
|
||||
class DigestWriteStep(WriteStep):
|
||||
"""WriteStep variant that enforces the ``<digest_dir>/<bucket>/<slug>.md`` layout.
|
||||
|
||||
``digest_dir`` is sourced from ``app_context.app_config.digest_dir`` at
|
||||
execute time — same convention as ``daily_create`` / ``daily_list`` /
|
||||
``daily_reindex`` for their ``daily_dir``.
|
||||
"""
|
||||
|
||||
def __init__(self, buckets=None, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.buckets = normalize_buckets(buckets)
|
||||
|
||||
def _reject(self, message: str) -> None:
|
||||
assert self.context is not None
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"REJECT: {message}"
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
raw = str(self.context.get("path") or "")
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
|
||||
err = _validate_digest_path(raw, bucket_names(self.buckets), digest_dir)
|
||||
if err:
|
||||
self._reject(err)
|
||||
return None
|
||||
|
||||
abs_path = (Path(self.vault_path) / raw).resolve()
|
||||
if abs_path.exists():
|
||||
self._reject(f"{raw} already exists; use digest_edit instead")
|
||||
return None
|
||||
|
||||
return await super().execute()
|
||||
|
||||
|
||||
def _validate_digest_path(
|
||||
raw: str,
|
||||
allowed_names: tuple[str, ...],
|
||||
digest_dir: str,
|
||||
) -> str | None:
|
||||
"""Return an error message, or ``None`` if ``raw`` matches the digest shape
|
||||
and uses one of the allowed bucket names. ``digest_dir`` is the configured
|
||||
digest root (e.g. ``"digest"``)."""
|
||||
prefix = f"{digest_dir}/"
|
||||
expected = f"{digest_dir}/<bucket>/<slug>.md"
|
||||
if not raw.startswith(prefix) or not raw.endswith(".md"):
|
||||
return f"path must be {expected!r}, got {raw!r}"
|
||||
parts = raw.split("/")
|
||||
if len(parts) != 3:
|
||||
return f"digest is a shallow bucket layout ({expected}, depth 1); got {raw!r}"
|
||||
bucket = parts[1]
|
||||
if bucket not in allowed_names:
|
||||
return (
|
||||
f"bucket {bucket!r} not in allowed set {list(allowed_names)}. "
|
||||
"Use 'unknown' when no specialized bucket fits."
|
||||
)
|
||||
return None
|
||||
|
|
@ -1,88 +1,46 @@
|
|||
"""Dreamer — auto-dream's create_or_update step.
|
||||
|
||||
Reads one daily-event note or resource file at the given vault-relative
|
||||
``path``, identifies the ABSTRACTIONS the material teaches in Phase 1,
|
||||
then in Phase 2 makes ONE cognitive write decision (CREATE or one of
|
||||
the three UPDATE flavors: CORROBORATE / REFINE / CORRECT) per
|
||||
abstraction. See ``docs4/auto_dream_design.md`` for the model
|
||||
contract (buckets / nodes / edges / evolution) and ``§4.2`` for the
|
||||
pipeline.
|
||||
``path``, identifies the ABSTRACTIONS the material teaches in Phase 1
|
||||
(each tagged with one of the three buckets), then in Phase 2 makes
|
||||
ONE cognitive write decision (CREATE or one of the three UPDATE
|
||||
flavors: CORROBORATE / REFINE / CORRECT) per abstraction using a
|
||||
**bucket-specific** integrate prompt.
|
||||
|
||||
**Digest is the abstract memory layer** — analogous to a prefrontal
|
||||
cortex aggregating cognition. Raw details (timestamps, full
|
||||
procedures, who-said-what, numbers) stay in the material; digest
|
||||
holds the principle, pattern, or precedent that should survive
|
||||
once the details fade. Provenance wikilinks (``derived_from::``)
|
||||
let readers drill back down to the source on demand.
|
||||
**Digest is the abstract memory layer** — raw details stay in the
|
||||
material; digest holds the principle, pattern, or precedent worth
|
||||
recalling once the specifics fade. Provenance wikilinks
|
||||
(``derived_from::``) let readers drill back down to the source.
|
||||
|
||||
Pipeline (external loop in Python, two distinct ReAct agent invocations,
|
||||
**light Phase 1 / heavy Phase 2**):
|
||||
|
||||
execute():
|
||||
_extract(material_blob) # 1× ReAct: identify abstractions
|
||||
# agent emits ExtractedUnits structured output
|
||||
# ({units: [{name, summary}, ...]})
|
||||
for unit in self._units: # Python loop, K iterations (K = num abstractions)
|
||||
_integrate_unit(unit) # 1× ReAct per abstraction: agent sees full material +
|
||||
# the sub-unit's name/summary, recalls, decides
|
||||
# bucket, makes ONE write decision (CREATE or
|
||||
# one of the UPDATE flavors). Sub-unit ↔ digest
|
||||
# node is 1:1.
|
||||
# agent emits ExtractedUnits
|
||||
# ({units: [{name, bucket, summary}, ...]})
|
||||
for unit in self._units: # Python loop, K iterations
|
||||
_integrate_unit(unit) # 1× ReAct per abstraction, dispatched
|
||||
# to integrate_system_prompt_<bucket>;
|
||||
# recalls cross-bucket, decides write,
|
||||
# uses canonical write/edit tools.
|
||||
|
||||
* **Phase 1 (extract / abstract)** uses a read-only toolkit
|
||||
and emits an :class:`ExtractedUnits` Pydantic model as its
|
||||
final structured answer (no tool call needed for the unit
|
||||
list — agentscope's ``structured_model`` enforces the shape).
|
||||
The agent identifies the abstractions the material teaches —
|
||||
principles, patterns, precedents worth carrying forward once
|
||||
specifics fade. Multiple raw facts that illustrate the same
|
||||
abstraction collapse into ONE sub-unit. Prompt biases toward
|
||||
fewer / coarser sub-units; filing detail under a digest
|
||||
sub-unit is the wrong layer. No event-level umbrella node is
|
||||
manufactured — the material itself plays that role via
|
||||
``derived_from`` provenance edges.
|
||||
The bucket vocabulary is hard-coded (:data:`BUCKETS`) — three buckets,
|
||||
each with a dedicated Phase 2 prompt:
|
||||
|
||||
* **Phase 2 (integrate per abstraction)** runs once per declared
|
||||
sub-unit with a fresh ReAct session (clean context) and the full
|
||||
read + write toolkit (``search``, ``traverse``, ``read``,
|
||||
``frontmatter_read``, ``digest_write``, ``digest_edit``). Three
|
||||
UPDATE shapes are surfaced explicitly
|
||||
in the prompt:
|
||||
* ``procedure`` — how-to-do-X: steps, methods, recipes, workflows.
|
||||
* ``personal`` — user/team specific: identity, preferences,
|
||||
conventions, things they avoid.
|
||||
* ``wiki`` — general knowledge: definitions, principles,
|
||||
observations, decisions-as-precedent. Default catch-all.
|
||||
|
||||
- **corroborate** (most common): the abstraction already
|
||||
exists; the material is one more instance → append a
|
||||
``derived_from::`` provenance wikilink so confidence
|
||||
accumulates; body unchanged in substance.
|
||||
- **refine**: the material reveals nuance / scope / edge
|
||||
cases the abstraction under-specified → tighten the
|
||||
relevant span + add the new provenance link.
|
||||
- **correct**: the material contradicts the abstraction →
|
||||
tighten to the narrower form both old and new support,
|
||||
or annotate the contradiction inline + add provenance.
|
||||
There is no SKIP outcome in Phase 2: Phase 1 is the gate for "not
|
||||
worth memorizing"; anything reaching Phase 2 warrants a write.
|
||||
|
||||
CREATE is reserved for genuinely new abstractions not yet in
|
||||
the vault — even thin first-encounter seeds, which grow via
|
||||
CORROBORATE / REFINE on later passes. There is no SKIP outcome:
|
||||
Phase 1 is the gate for "not worth memorizing"; anything that
|
||||
reaches Phase 2 warrants a write.
|
||||
|
||||
The trade-off vs heavy Phase 1: full material is sent to LLM K
|
||||
times in Phase 2 (one per abstraction). The advantages: no
|
||||
information loss in summary, focused reasoning per call, and
|
||||
granularity tuned at a single prompt (Phase 1) rather than two.
|
||||
|
||||
Mechanical guardrails at the write boundary:
|
||||
|
||||
* ``digest_write`` (subclass of WriteStep) rejects paths outside
|
||||
``<digest_dir>/<bucket>/<slug>.md`` (where ``digest_dir`` comes
|
||||
from app config and ``bucket`` is in the fixed bucket set), and
|
||||
refuses if the path already exists.
|
||||
* ``digest_edit`` (subclass of EditStep) is a body-only find-and-replace
|
||||
on an existing digest node, gated by E-1 strong-conservation: the
|
||||
outbound link set BEFORE the replacement must be a subset of the link
|
||||
set AFTER. If any edge would be dropped the tool returns
|
||||
``REJECT_CONSERVATION`` and the agent must adjust ``new`` to keep
|
||||
the missing links before retrying.
|
||||
Phase 2 uses the **canonical** ``write`` / ``edit`` jobs (no
|
||||
constrained variants). Bucket placement and edge conservation are
|
||||
prompt-level discipline; the tools themselves perform no path-shape
|
||||
or conservation validation.
|
||||
|
||||
Invocation form (CLI / MCP):
|
||||
reme dream path=daily/2026-05-28/auth-refactor/auth-refactor.md
|
||||
|
|
@ -98,13 +56,21 @@ from agentscope.message import Msg, TextBlock
|
|||
from agentscope.tool import Toolkit, ToolResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .digest_edit import DigestEditStep
|
||||
from .digest_write import DigestWriteStep, bucket_names, normalize_buckets
|
||||
from .._evolve import FlexReActAgent
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
# Hard-coded bucket vocabulary. Phase 1 classifies each sub-unit into
|
||||
# one of these; Phase 2 dispatches to the bucket-specific prompt.
|
||||
# Order matters for prompt rendering — keep procedure/personal/wiki.
|
||||
BUCKETS: tuple[str, ...] = ("procedure", "personal", "wiki")
|
||||
|
||||
# Bucket = Literal of BUCKETS. Pydantic Literal must be a static type;
|
||||
# update both BUCKETS and Bucket together if the vocabulary changes.
|
||||
Bucket = Literal["procedure", "personal", "wiki"]
|
||||
|
||||
|
||||
_EXTRACT_READ_TOOLS: tuple[str, ...] = ("read",)
|
||||
|
||||
_INTEGRATE_READ_TOOLS: tuple[str, ...] = (
|
||||
|
|
@ -114,6 +80,11 @@ _INTEGRATE_READ_TOOLS: tuple[str, ...] = (
|
|||
"frontmatter_read",
|
||||
)
|
||||
|
||||
_INTEGRATE_WRITE_TOOLS: tuple[str, ...] = (
|
||||
"write",
|
||||
"edit",
|
||||
)
|
||||
|
||||
|
||||
def _pack_material(file_store, path: str) -> str:
|
||||
"""Render one daily-event note or resource file into a prompt block."""
|
||||
|
|
@ -139,7 +110,18 @@ class MemoryUnit(BaseModel):
|
|||
"Short kebab-case identifier for the abstraction "
|
||||
"(e.g. 'jwt-rotation-decision', 'pr-size-pref'). "
|
||||
"Agent-internal handle — NOT the eventual digest slug; "
|
||||
"Phase 2 picks the actual filing path + bucket."
|
||||
"Phase 2 picks the actual filing path."
|
||||
),
|
||||
)
|
||||
bucket: Bucket = Field(
|
||||
description=(
|
||||
"Which bucket this abstraction belongs in — Phase 2 dispatches "
|
||||
"to a bucket-specific prompt based on this. Pick exactly one: "
|
||||
"`procedure` (how-to-do-X — steps, methods, recipes, workflows), "
|
||||
"`personal` (user/team-specific — identity, preferences, "
|
||||
"conventions, things they avoid), `wiki` (general knowledge — "
|
||||
"definitions, principles, observations, decisions-as-precedent; "
|
||||
"default catch-all when nothing else fits)."
|
||||
),
|
||||
)
|
||||
summary: str = Field(
|
||||
|
|
@ -161,24 +143,18 @@ class ExtractedUnits(BaseModel):
|
|||
description=(
|
||||
"Memory sub-units identified in the material — orthogonal "
|
||||
"abstractions (principles / patterns / precedents) worth "
|
||||
"lifting into long-term memory. Empty list = nothing worth "
|
||||
"lifting (Phase 2 is skipped)."
|
||||
"lifting into long-term memory. Each is tagged with its "
|
||||
"bucket. Empty list = nothing worth lifting (Phase 2 is skipped)."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _render_outcome_line(unit_name: str, o: "IntegrateOutcome") -> str:
|
||||
def _render_outcome_line(unit_name: str, bucket: str, o: "IntegrateOutcome") -> str:
|
||||
"""Format one IntegrateOutcome as a one-line summary entry."""
|
||||
if o.action == "CREATE":
|
||||
body = f"CREATE {o.target_path}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
else: # CORROBORATE / REFINE / CORRECT (all UPDATE-flavored)
|
||||
recovered = " (recovered from REJECT_CONSERVATION)" if o.recovered_from_conservation else ""
|
||||
body = f"{o.action} {o.target_path}{recovered}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
return f"[{unit_name}] {body}"
|
||||
body = f"{o.action} {o.target_path}"
|
||||
if o.note:
|
||||
body += f" — {o.note}"
|
||||
return f"[{unit_name}/{bucket}] {body}"
|
||||
|
||||
|
||||
class IntegrateOutcome(BaseModel):
|
||||
|
|
@ -203,7 +179,8 @@ class IntegrateOutcome(BaseModel):
|
|||
)
|
||||
target_path: str = Field(
|
||||
description=(
|
||||
"The digest path you wrote to — must match what your " "`digest_write` / `digest_edit` call(s) targeted."
|
||||
"The digest path you wrote to — must match what your `write` / "
|
||||
"`edit` call(s) targeted."
|
||||
),
|
||||
)
|
||||
note: str = Field(
|
||||
|
|
@ -216,14 +193,6 @@ class IntegrateOutcome(BaseModel):
|
|||
"outcome note."
|
||||
),
|
||||
)
|
||||
recovered_from_conservation: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Set to true if `digest_edit` initially returned "
|
||||
"REJECT_CONSERVATION and you re-composed `new` to preserve the "
|
||||
"missing links. Only meaningful for CORROBORATE / REFINE / CORRECT."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class DreamResult(BaseModel):
|
||||
|
|
@ -231,9 +200,8 @@ class DreamResult(BaseModel):
|
|||
|
||||
Per-tool audit lives in the toolkit layer (not exposed back to the
|
||||
orchestrator). Structured outcome here is the input path the call
|
||||
processed, the memory sub-units the agent declared in Phase 1,
|
||||
what got created / updated in Phase 2, and any conservation
|
||||
rejections that occurred along the way.
|
||||
processed, the memory sub-units the agent declared in Phase 1, and
|
||||
what got created / updated in Phase 2.
|
||||
"""
|
||||
|
||||
used_llm: bool = False
|
||||
|
|
@ -242,7 +210,6 @@ class DreamResult(BaseModel):
|
|||
units: list[dict] = Field(default_factory=list)
|
||||
nodes_created: list[str] = Field(default_factory=list)
|
||||
nodes_updated: list[str] = Field(default_factory=list)
|
||||
conservation_violations: list[dict] = Field(default_factory=list)
|
||||
summary: str = ""
|
||||
error: str = ""
|
||||
|
||||
|
|
@ -257,8 +224,6 @@ class Dreamer(BaseStep):
|
|||
empty string to no-op.
|
||||
hint (str, optional): caller guidance to the LLM
|
||||
(e.g. "focus on the auth-related decisions").
|
||||
buckets (list[str], optional): override the fixed bucket
|
||||
set; default ``DEFAULT_BUCKETS``.
|
||||
|
||||
Output (written to context.response.answer):
|
||||
``DreamResult`` JSON in ``metadata``; LLM summary in ``answer``.
|
||||
|
|
@ -272,20 +237,16 @@ class Dreamer(BaseStep):
|
|||
toolkit: Toolkit | None = None,
|
||||
console_enabled: bool = False,
|
||||
timezone: str | None = None,
|
||||
buckets: list[str] | tuple[str, ...] | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
self.toolkit = toolkit
|
||||
self.console_enabled = console_enabled
|
||||
self.timezone = timezone
|
||||
self.buckets = normalize_buckets(buckets)
|
||||
assert "unknown" in bucket_names(self.buckets), "bucket set must include 'unknown' as the unclassified fallback"
|
||||
# Per-invocation outcome trackers, populated by tool callbacks.
|
||||
self._units: list[dict] = []
|
||||
self._created: list[str] = []
|
||||
self._updated: list[str] = []
|
||||
self._violations: list[dict] = []
|
||||
|
||||
def _now(self) -> datetime.datetime:
|
||||
if self.timezone:
|
||||
|
|
@ -305,46 +266,43 @@ class Dreamer(BaseStep):
|
|||
except Exception:
|
||||
return False
|
||||
|
||||
def _make_digest_write_tool(self):
|
||||
"""Tool closure: wraps :class:`DigestWriteStep` and tracks creates."""
|
||||
def _make_write_tool(self):
|
||||
"""Wrap the canonical ``write`` job with create-tracking.
|
||||
|
||||
async def digest_write(path: str, name: str, description: str, content: str) -> ToolResponse:
|
||||
step = DigestWriteStep(
|
||||
file_store=self.file_store,
|
||||
buckets=self.buckets,
|
||||
app_context=self.app_context,
|
||||
Same shape as the underlying job; the wrapper just records
|
||||
successful paths into ``self._created`` so the dreamer can
|
||||
reconstruct outcomes when the LLM drops its structured emission.
|
||||
"""
|
||||
job = self.get_job("write")
|
||||
if job is None:
|
||||
raise RuntimeError("write job not registered")
|
||||
|
||||
async def write(path: str, name: str, description: str, content: str) -> ToolResponse:
|
||||
resp = await job(
|
||||
path=path,
|
||||
name=name,
|
||||
description=description,
|
||||
content=content,
|
||||
)
|
||||
await step(path=path, name=name, description=description, content=content)
|
||||
assert step.context is not None
|
||||
resp = step.context.response
|
||||
if not resp.success:
|
||||
return ToolResponse(content=[TextBlock(type="text", text=resp.answer)])
|
||||
self._created.append(path)
|
||||
return ToolResponse(content=[TextBlock(type="text", text=f"OK: created {path}")])
|
||||
if resp.success:
|
||||
self._created.append(path)
|
||||
return ToolResponse(content=[TextBlock(type="text", text=resp.answer)])
|
||||
|
||||
return digest_write
|
||||
return write, job
|
||||
|
||||
def _make_digest_edit_tool(self):
|
||||
"""Tool closure: wraps :class:`DigestEditStep` and tracks updates / conservation violations."""
|
||||
def _make_edit_tool(self):
|
||||
"""Wrap the canonical ``edit`` job with update-tracking."""
|
||||
job = self.get_job("edit")
|
||||
if job is None:
|
||||
raise RuntimeError("edit job not registered")
|
||||
|
||||
async def digest_edit(path: str, old: str, new: str) -> ToolResponse:
|
||||
step = DigestEditStep(
|
||||
file_store=self.file_store,
|
||||
buckets=self.buckets,
|
||||
app_context=self.app_context,
|
||||
)
|
||||
await step(path=path, old=old, new=new)
|
||||
assert step.context is not None
|
||||
resp = step.context.response
|
||||
if not resp.success:
|
||||
violation = (resp.metadata or {}).get("conservation_violation")
|
||||
if violation:
|
||||
self._violations.append(violation)
|
||||
return ToolResponse(content=[TextBlock(type="text", text=resp.answer)])
|
||||
self._updated.append(path)
|
||||
return ToolResponse(content=[TextBlock(type="text", text=f"OK: updated {path}")])
|
||||
async def edit(path: str, old: str, new: str) -> ToolResponse:
|
||||
resp = await job(path=path, old=old, new=new)
|
||||
if resp.success:
|
||||
self._updated.append(path)
|
||||
return ToolResponse(content=[TextBlock(type="text", text=resp.answer)])
|
||||
|
||||
return digest_edit
|
||||
return edit, job
|
||||
|
||||
def _build_extract_toolkit(self) -> Toolkit:
|
||||
"""Read-only toolkit for the extract agent. Sub-units come back via
|
||||
|
|
@ -355,97 +313,43 @@ class Dreamer(BaseStep):
|
|||
return toolkit
|
||||
|
||||
def _build_integrate_toolkit(self) -> Toolkit:
|
||||
"""Full read + conservation-aware write toolkit for the integrate agent."""
|
||||
"""Full read + canonical write/edit toolkit for the integrate agent.
|
||||
|
||||
write/edit are wrapped in tracker closures (created/updated paths)
|
||||
so the outer loop can reconstruct outcomes when an LLM call drops
|
||||
its structured emission. Read-only tools go through ``add_as_tool``
|
||||
unchanged.
|
||||
"""
|
||||
toolkit = self.toolkit or Toolkit()
|
||||
for job_name in _INTEGRATE_READ_TOOLS:
|
||||
self.add_as_tool(toolkit, job_name)
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
path_shape = f"'{digest_dir}/<bucket>/<slug>.md'"
|
||||
digest_write_desc = (
|
||||
"Create a NEW digest node — same shape as the canonical `write` job, plus "
|
||||
f"path-shape validation: `path` must be {path_shape} where "
|
||||
f"bucket is one of {list(bucket_names(self.buckets))} (use 'unknown' when "
|
||||
"no specialized bucket fits — it is a first-class bucket, not a failure "
|
||||
"state). `name` and `description` go into the YAML frontmatter; `content` "
|
||||
"is the body. Fails if the path already exists; use `digest_edit` then."
|
||||
)
|
||||
|
||||
write_tool, write_job = self._make_write_tool()
|
||||
toolkit.register_tool_function(
|
||||
tool_func=self._make_digest_write_tool(),
|
||||
func_name="digest_write",
|
||||
func_description=digest_write_desc,
|
||||
tool_func=write_tool,
|
||||
func_name="write",
|
||||
func_description=write_job.description,
|
||||
json_schema={
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "digest_write",
|
||||
"description": digest_write_desc,
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": f"vault-relative path; must match {path_shape}",
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "frontmatter name (usually the slug)",
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "frontmatter description — one-line summary of the abstraction",
|
||||
},
|
||||
"content": {
|
||||
"type": "string",
|
||||
"description": "body (markdown; no frontmatter "
|
||||
"— name/description go in the fields above)",
|
||||
},
|
||||
},
|
||||
"required": ["path", "name", "description", "content"],
|
||||
},
|
||||
"name": "write",
|
||||
"description": write_job.description,
|
||||
"parameters": write_job.parameters,
|
||||
},
|
||||
},
|
||||
)
|
||||
digest_edit_desc = (
|
||||
"Find-and-replace inside an existing digest node's body — same shape as "
|
||||
"the canonical `edit` job, plus path-shape validation (`path` must be "
|
||||
f"{path_shape}, file must exist) and E-1 strong edge conservation: every "
|
||||
"outbound wikilink present BEFORE the replacement must still be present "
|
||||
"AFTER. If you drop any edge the tool returns REJECT_CONSERVATION and you "
|
||||
"must adjust `new` to keep the missing links. Operates on body only; "
|
||||
"frontmatter is untouched. Prefer narrow `old` spans."
|
||||
)
|
||||
|
||||
edit_tool, edit_job = self._make_edit_tool()
|
||||
toolkit.register_tool_function(
|
||||
tool_func=self._make_digest_edit_tool(),
|
||||
func_name="digest_edit",
|
||||
func_description=digest_edit_desc,
|
||||
tool_func=edit_tool,
|
||||
func_name="edit",
|
||||
func_description=edit_job.description,
|
||||
json_schema={
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "digest_edit",
|
||||
"description": digest_edit_desc,
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": f"vault-relative path; must match {path_shape} and exist",
|
||||
},
|
||||
"old": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Substring to locate in the EXISTING body (frontmatter excluded). "
|
||||
"Must match verbatim. Pick a span large enough to be unique."
|
||||
),
|
||||
},
|
||||
"new": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"Replacement text. Should weave new material into the existing "
|
||||
"wording without dropping any wikilinks the `old` span contained."
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["path", "old", "new"],
|
||||
},
|
||||
"name": "edit",
|
||||
"description": edit_job.description,
|
||||
"parameters": edit_job.parameters,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
|
@ -460,7 +364,7 @@ class Dreamer(BaseStep):
|
|||
sys_prompt=self.prompt_format(
|
||||
"extract_system_prompt",
|
||||
vault_dir=str(vault_dir),
|
||||
buckets=", ".join(bucket_names(self.buckets)),
|
||||
buckets=", ".join(BUCKETS),
|
||||
),
|
||||
formatter=self.as_llm_formatter,
|
||||
toolkit=toolkit,
|
||||
|
|
@ -486,36 +390,44 @@ class Dreamer(BaseStep):
|
|||
continue
|
||||
name = str(raw.get("name") or "").strip()
|
||||
summary = str(raw.get("summary") or "").strip()
|
||||
if name and summary:
|
||||
cleaned.append({"name": name, "summary": summary})
|
||||
bucket = str(raw.get("bucket") or "").strip()
|
||||
if not name or not summary:
|
||||
continue
|
||||
if bucket not in BUCKETS:
|
||||
# Defensive: structured_model should already reject this,
|
||||
# but if it slips through we route to wiki (the catch-all).
|
||||
self.logger.warning(
|
||||
f"[{self.name}] unit {name!r} emitted bucket {bucket!r} "
|
||||
f"not in {list(BUCKETS)}; routing to 'wiki'",
|
||||
)
|
||||
bucket = "wiki"
|
||||
cleaned.append({"name": name, "summary": summary, "bucket": bucket})
|
||||
self._units = cleaned
|
||||
return (msg.get_text_content() or "").strip()
|
||||
|
||||
async def _integrate_unit(self, unit: dict, material_blob: str, hint: str, vault_dir: Path) -> IntegrateOutcome:
|
||||
"""One ReAct invocation per memory sub-unit. Returns the parsed
|
||||
"""One ReAct invocation per memory sub-unit, dispatched to the
|
||||
bucket-specific system prompt. Returns the parsed
|
||||
:class:`IntegrateOutcome` reported by the agent.
|
||||
|
||||
File writes happen as side effects via the ``digest_write`` /
|
||||
``digest_edit`` tool calls during the ReAct loop (which populate
|
||||
``self._created`` / ``self._updated`` / ``self._violations``); the
|
||||
structured outcome here is the agent's own summary of what it
|
||||
decided — useful for rendering and for catching hallucinations
|
||||
(action=CREATE without the matching write call landing in trackers).
|
||||
File writes happen as side effects via the canonical ``write`` /
|
||||
``edit`` tool calls (which populate ``self._created`` /
|
||||
``self._updated`` via the tracker closures); the structured
|
||||
outcome here is the agent's own summary of what it decided —
|
||||
useful for rendering and for catching hallucinations (action=
|
||||
CREATE without the matching write call landing in trackers).
|
||||
"""
|
||||
bucket = unit.get("bucket") or "wiki"
|
||||
toolkit = self._build_integrate_toolkit()
|
||||
digest_dir = getattr(self.app_context.app_config, "digest_dir", "")
|
||||
buckets_block = "\n".join(
|
||||
f" - `{digest_dir}/{b['name']}/`" + (f" — {b['description']}" if b.get("description") else "")
|
||||
for b in self.buckets
|
||||
)
|
||||
agent = FlexReActAgent(
|
||||
name=f"reme_dreamer_integrate_{unit.get('name', 'unit')}",
|
||||
model=self.as_llm,
|
||||
sys_prompt=self.prompt_format(
|
||||
"integrate_system_prompt",
|
||||
f"integrate_system_prompt_{bucket}",
|
||||
vault_dir=str(vault_dir),
|
||||
digest_dir=digest_dir,
|
||||
buckets=buckets_block,
|
||||
bucket=bucket,
|
||||
),
|
||||
formatter=self.as_llm_formatter,
|
||||
toolkit=toolkit,
|
||||
|
|
@ -525,6 +437,7 @@ class Dreamer(BaseStep):
|
|||
"integrate_user_message",
|
||||
hint=hint or "(none)",
|
||||
unit_name=unit.get("name", ""),
|
||||
unit_bucket=bucket,
|
||||
unit_summary=unit.get("summary", ""),
|
||||
material_blob=material_blob,
|
||||
)
|
||||
|
|
@ -580,12 +493,11 @@ class Dreamer(BaseStep):
|
|||
self._units.clear()
|
||||
self._created.clear()
|
||||
self._updated.clear()
|
||||
self._violations.clear()
|
||||
|
||||
vault_dir = self._vault_dir()
|
||||
|
||||
# Phase 1 — extract (light). Agent emits ExtractedUnits structured output to commit the
|
||||
# memory sub-units worth lifting.
|
||||
# memory sub-units worth lifting. Each unit carries its own bucket.
|
||||
self.logger.info(f"[{self.name}] extract phase: path={path!r}")
|
||||
extract_summary = await self._extract(material_blob, hint, vault_dir)
|
||||
|
||||
|
|
@ -599,31 +511,36 @@ class Dreamer(BaseStep):
|
|||
|
||||
self.logger.info(
|
||||
f"[{self.name}] integrate phase: {len(self._units)} sub-unit(s): "
|
||||
f"{', '.join(u['name'] for u in self._units)}",
|
||||
+ ", ".join(f"{u['name']}/{u['bucket']}" for u in self._units),
|
||||
)
|
||||
|
||||
# Phase 2 — integrate, one fresh ReAct per sub-unit. Python-level
|
||||
# loop, not agent loop. Each session emits a structured
|
||||
# IntegrateOutcome; file writes happen as side effects via
|
||||
# digest_write / digest_edit tool calls.
|
||||
# Phase 2 — integrate, one fresh ReAct per sub-unit, dispatched to
|
||||
# the bucket-specific system prompt. Python-level loop, not agent
|
||||
# loop. Each session emits a structured IntegrateOutcome; file
|
||||
# writes happen as side effects via the canonical write / edit
|
||||
# tool calls.
|
||||
per_unit_lines: list[str] = []
|
||||
for i, unit in enumerate(self._units, start=1):
|
||||
name = unit.get("name", "?")
|
||||
bucket = unit.get("bucket", "?")
|
||||
try:
|
||||
outcome = await self._integrate_unit(unit, material_blob, hint, vault_dir)
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"[{self.name}] integrate {i}/{len(self._units)} (unit={name}) " f"failed: {type(e).__name__}: {e}",
|
||||
f"[{self.name}] integrate {i}/{len(self._units)} "
|
||||
f"(unit={name}, bucket={bucket}) failed: {type(e).__name__}: {e}",
|
||||
)
|
||||
per_unit_lines.append(f"[{name}] FAILED: {type(e).__name__}: {e}")
|
||||
per_unit_lines.append(f"[{name}/{bucket}] FAILED: {type(e).__name__}: {e}")
|
||||
continue
|
||||
per_unit_lines.append(_render_outcome_line(name, outcome))
|
||||
per_unit_lines.append(_render_outcome_line(name, bucket, outcome))
|
||||
|
||||
summary = (
|
||||
f"Declared {len(self._units)} sub-unit(s) "
|
||||
f"({', '.join(u['name'] for u in self._units)}); "
|
||||
f"created {len(self._created)}, updated {len(self._updated)}, "
|
||||
f"conservation violations {len(self._violations)}.\n" + "\n".join(per_unit_lines)
|
||||
+ "("
|
||||
+ ", ".join(f"{u['name']}/{u['bucket']}" for u in self._units)
|
||||
+ "); "
|
||||
f"created {len(self._created)}, updated {len(self._updated)}.\n"
|
||||
+ "\n".join(per_unit_lines)
|
||||
)
|
||||
|
||||
return DreamResult(
|
||||
|
|
@ -632,7 +549,6 @@ class Dreamer(BaseStep):
|
|||
units=list(self._units),
|
||||
nodes_created=list(self._created),
|
||||
nodes_updated=list(self._updated),
|
||||
conservation_violations=list(self._violations),
|
||||
summary=summary,
|
||||
skipped=False,
|
||||
)
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -2,25 +2,33 @@
|
|||
|
||||
Seeds a vault with:
|
||||
|
||||
- 4 pre-existing digest/ nodes (3 concepts/procedures, 1 preference) —
|
||||
these are the **recall targets**; the new material partially overlaps
|
||||
them so Phase 2 must find them via search_step + file_read and decide
|
||||
UPDATE (with E-1 edge conservation in play).
|
||||
- 4 pre-existing digest/ nodes spread across the three buckets
|
||||
(procedure / personal / wiki) — these are the **recall targets**;
|
||||
the new material partially overlaps them so Phase 2 must find
|
||||
them via search + read and decide UPDATE.
|
||||
- 4 small daily/ stubs the digest nodes already link to — they exist
|
||||
only so the seeded digest bodies don't dangle (the conservation check
|
||||
doesn't validate link targets, but a realistic graph reads better).
|
||||
- 1 NEW daily note (the file the dreamer will be invoked on). It covers
|
||||
all three outcome cases — CREATE / UPDATE / SKIP — across 4 kinds:
|
||||
* concept : UPDATE digest/concept/jwt.md + CREATE digest/concept/kid-versioning.md
|
||||
+ SKIP (an OAuth2 restatement that adds nothing over
|
||||
the existing digest/concept/oauth2.md)
|
||||
* procedure : UPDATE digest/procedure/key-rotation.md
|
||||
* observation: CREATE digest/observation/soc2-30day-finding.md
|
||||
* preference : UPDATE digest/preference/no-trailing-summary.md + CREATE digest/preference/small-pr.md
|
||||
only so the seeded digest bodies don't dangle.
|
||||
- 1 NEW daily note (the file the dreamer will be invoked on).
|
||||
It exercises CREATE and UPDATE across the three buckets:
|
||||
|
||||
Total budget per integration run: 1 Phase 1 + up-to-4 Phase 2 = up to 5
|
||||
ReAct sessions, each with several tool turns (search → file_read →
|
||||
digest_* / SKIP).
|
||||
* wiki : UPDATE digest/wiki/jwt.md (24h rotation cadence
|
||||
refines short-credential-compliance framing) +
|
||||
CREATE digest/wiki/kid-versioning.md +
|
||||
CREATE digest/wiki/soc2-30day-finding.md
|
||||
(the OAuth2 restatement section is intentionally
|
||||
a non-abstraction — Phase 1 should NOT emit a
|
||||
sub-unit for it; tests Phase 1's gate-keeping)
|
||||
* procedure : UPDATE digest/procedure/key-rotation.md (24h
|
||||
cadence + kid-versioning supersede the 30-day
|
||||
JWKS-cache flow)
|
||||
* personal : UPDATE digest/personal/no-trailing-summary.md
|
||||
(extend "no trailing summary" to also forbid
|
||||
"next steps" lists) + CREATE
|
||||
digest/personal/small-pr.md
|
||||
|
||||
Total budget per integration run: 1 Phase 1 + up-to-6 Phase 2 = up
|
||||
to 7 ReAct sessions, each with several tool turns (search +
|
||||
traverse → frontmatter_read + read → write / edit).
|
||||
|
||||
Idempotent: re-running does NOT overwrite existing files. To re-seed
|
||||
from scratch, delete the vault and rerun.
|
||||
|
|
@ -43,7 +51,7 @@ INPUT_PATH = "daily/2026-05-28/auth-refactor/notes.md"
|
|||
|
||||
_FILES: dict[str, str] = {
|
||||
# ----- pre-existing digest nodes (recall targets) -----
|
||||
"digest/concept/jwt.md": """\
|
||||
"digest/wiki/jwt.md": """\
|
||||
---
|
||||
name: jwt
|
||||
description: JSON Web Token — signed authentication token format
|
||||
|
|
@ -60,11 +68,11 @@ token used to assert identity and claims between parties.
|
|||
- Signature
|
||||
|
||||
## Related
|
||||
Often issued by [[digest/concept/oauth2.md]] flows.
|
||||
Often issued by [[digest/wiki/oauth2.md]] flows.
|
||||
|
||||
derived_from:: [[daily/2026-05-15/auth-design/notes.md]]
|
||||
""",
|
||||
"digest/concept/oauth2.md": """\
|
||||
"digest/wiki/oauth2.md": """\
|
||||
---
|
||||
name: oauth2
|
||||
description: OAuth 2.0 — delegated authorization framework
|
||||
|
|
@ -91,7 +99,7 @@ description: Rotating signing keys for JWT issuance
|
|||
|
||||
# Key rotation (current — pre 2026-05-28 refactor)
|
||||
|
||||
Procedure for rotating the signing key used by [[digest/concept/jwt.md]]
|
||||
Procedure for rotating the signing key used by [[digest/wiki/jwt.md]]
|
||||
issuance.
|
||||
|
||||
## Steps
|
||||
|
|
@ -107,7 +115,7 @@ no formal compliance requirement has tightened this so far.
|
|||
|
||||
derived_from:: [[daily/2026-05-20/rotation-plan/notes.md]]
|
||||
""",
|
||||
"digest/preference/no-trailing-summary.md": """\
|
||||
"digest/personal/no-trailing-summary.md": """\
|
||||
---
|
||||
name: no-trailing-summary
|
||||
description: 不要在回复末尾加总结段落
|
||||
|
|
@ -117,6 +125,11 @@ description: 不要在回复末尾加总结段落
|
|||
|
||||
用户能看 diff,不需要在回复末尾重述刚做的事。
|
||||
|
||||
**Why**: diff 已经把"改了什么"摆在用户面前;再口述一遍是噪音。
|
||||
|
||||
**How to apply**: 任意编码 / 编辑任务回复结束时,直接停在最后一条
|
||||
有信息量的话上,不要再补一段"以上就是本次的修改..."。
|
||||
|
||||
derived_from:: [[daily/2026-05-01/style-feedback/notes.md]]
|
||||
""",
|
||||
# ----- daily provenance stubs (so the digest links don't dangle) -----
|
||||
|
|
|
|||
|
|
@ -1,21 +1,26 @@
|
|||
"""dreamer in-process integration test (option C).
|
||||
"""dreamer in-process integration test.
|
||||
|
||||
Loads the default reme4 config, seeds a rich workspace (pre-existing
|
||||
digest nodes + a new daily that should drive both UPDATE and CREATE
|
||||
through Phase 1 classify → Phase 2 per-kind recall+write), reindexes
|
||||
so search_step can hit the pre-existing nodes, then calls `dream` and
|
||||
prints what happened.
|
||||
digest nodes spread across the three buckets + a new daily that
|
||||
exercises CREATE and UPDATE in each bucket), reindexes so search can
|
||||
hit the pre-existing nodes, then calls `dream` and prints what
|
||||
happened.
|
||||
|
||||
Phase 1 classifies each sub-unit into one of {procedure, personal,
|
||||
wiki}; Phase 2 dispatches to the bucket-specific integrate prompt
|
||||
and writes via the canonical `write` / `edit` tools.
|
||||
|
||||
Usage (from anywhere):
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py
|
||||
daily/2026-05-28/auth-refactor/notes.md
|
||||
VAULT_PATH=/tmp/reme-dreamer-test python tests4/integration/test_dreamer_inproc.py \\
|
||||
daily/2026-05-28/auth-refactor/notes.md
|
||||
|
||||
Defaults:
|
||||
VAULT_PATH unset → /tmp/reme-dreamer-test
|
||||
Each run wipes `daily/`, `digest/`, and `reme_metadata/` under the
|
||||
vault before reseeding, so the dreamer always starts from the same
|
||||
fixture state. See _dreamer_fixture.py for what gets created.
|
||||
fixture state. See _dreamer_fixture.py for what gets created and
|
||||
the expected CREATE / UPDATE landings per bucket.
|
||||
|
||||
Required env (from .env or shell):
|
||||
LLM_API_KEY, LLM_BASE_URL, LLM_MODEL_NAME — for the Phase 1/2 agents
|
||||
|
|
@ -66,9 +71,9 @@ async def main() -> None:
|
|||
app = ReMe(**cfg)
|
||||
await app.start()
|
||||
try:
|
||||
# Reindex first so search_step can actually find the pre-seeded
|
||||
# Reindex first so search can actually find the pre-seeded
|
||||
# digest/ nodes — otherwise Phase 2 recall returns empty and
|
||||
# every atomic unit ends up as CREATE (UPDATE path not exercised).
|
||||
# every sub-unit ends up as CREATE (UPDATE path not exercised).
|
||||
print("\n--- reindexing vault so Phase 2 recall has something to hit")
|
||||
await app.run_job("reindex")
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue