ReMe/tests/unit/test_frontmatter_steps.py
jinliyl cae613d1c4
refactor(auto-fin): one note per topic, merged by a separate digest step (#558)
* fix(auto-fin): parse topic IDs from fenced JSON replies

* fix(auto-fin): limit report agent tool calls in prompt

* refactor(auto-fin): research news by topic before market open

* fix(config): update default model version for claude_code backend

- Change model version from qwen3.8-max to qwen3.7-plus
- Use environment variable LLM_MODEL_NAME to allow override
- Ensure backend configuration reflects updated model setting

* refactor(auto-fin): write one note per topic before the daily digest

The merge step did two jobs at once: it researched every topic and
combined the results into a single report. Split it the way daily-paper
separates analysis from its brief, so each topic earns a durable note of
its own.

- auto_fin_research_step writes one note per topic that had relevant
  news, tagged `kind: auto-fin-topic` and `topic` in frontmatter
- auto_fin_digest_step merges those notes into the day's brief with no
  tools of its own and appends a `## 主题详解` section linking back to
  each note
- a same-day rerun finds a topic's note by its `topic` frontmatter and
  replaces it in place, deleting the old file when the title changed
- base.py now owns the shared Markdown layer: title sanitizing, report
  normalizing, wikilink validation, note lookup, atomic frontmatter
  writes, and change tracking, so both steps share one write path
- the DingTalk step maps `auto_fin_digest_path` to `markdown_path`
  explicitly instead of relying on whichever step ran last
- drop the unused AutoFinTopicOutput schema and read `job_tools` from
  the step config rather than hardcoding `search`

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(dingtalk): surface rejection details when delivery fails

A failed group send only reported HTTPStatusError, so an operator had to
reproduce the request by hand to learn why DingTalk refused it. Include
the status code and the whitelisted error keys from the response body in
both the log line and the raised RuntimeError.

Only `code`, `message`, and `requestid` are reported: the request body
carries the message content and credentials, so an error response that
echoes it back must not reach the log. Detail is truncated to 200 chars.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(test): make the suite green on CI

- Point the cookbook claude_code model assertion at qwen3.7-plus, the
  default commit 9404e600 set, so the pre-existing red stops blocking
- Satisfy pylint on the auto-fin tests: prefer implicit booleaness for
  the recorded Agent calls and drop an unused tmp_path fixture

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(auto-fin): name notes after the topic, not the Agent title

The research Agent returned a whole paragraph as its title; that became a
filename and blew past the filesystem's 255-byte name limit, failing with
ENAMETOOLONG inside resolve_note_path. Topics are configured values, so
they are short and predictable - use them for file names and keep the
Agent title in frontmatter.

- Name topic notes after the topic and the digest after the run date
- Fold a byte budget into normalize_title as a safety net for long topics
- Take an AutoFinReportOutput in _write_report instead of loose fields
- Ask both prompts for a short title now that it is display-only

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: isolate per-topic research failures and scope frontmatter reads

A single failing topic used to fail the whole cron job and discard the news
already gathered for the topics that had not run yet -- the 09-19 09:24 run
lost its robot notes that way. Research now logs the failure, continues with
the remaining topics, and only fails the run when no topic produced a note.

frontmatter_read was the only frontmatter step without the _allowed_paths
check that read, write, edit, and frontmatter_update already honour, so an
Agent scoped to one file could still read another file's metadata.

- Isolate per-topic research failures and report them as failed_topics
- Fail loudly when every topic fails so an empty brief is never sent
- Apply _check_path_permission in FrontmatterReadStep

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(auto-fin): keep hand-edited notes and wikilink delimiters from breaking the run

Addresses three review findings on the topic-per-note rework.

- `find_note` and `read_note` now skip a note whose YAML frontmatter does
  not parse. A hand-edited note in the day directory raised
  `yaml.parser.ParserError`, which per-topic isolation surfaced as
  "Auto Fin research failed for every topic" and took the run down with it.
- `normalize_title` also strips `[`, `]` and `#`, which `WikilinkHandler`
  treats as target delimiters. `AI[算力]` used to emit a trailer link the
  parser could not read at all, and `C#` resolved to `.../C` plus an anchor.
- `_write_report` returns the body it actually wrote, and both callers
  propagate it, so the digest answer and the note handed to the digest Agent
  no longer carry links that validation had already downgraded on disk.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-20 14:52:56 +08:00

264 lines
9.9 KiB
Python

"""Tests for the frontmatter-only CRUD steps (``frontmatter_read`` /
``frontmatter_update`` / ``frontmatter_delete``).
Covers the suffix-less path contract shared with ``read`` / ``write`` /
``edit``: a bare path with no suffix auto-appends ``.md``. All three
siblings must agree on path handling, disclose the substitution via
``resolved_path`` in the response metadata, and report the actually
probed path in not-found errors instead of the caller's raw input.
"""
# pylint: disable=protected-access
import os
import tempfile
from pathlib import Path
import frontmatter
import pytest
from reme.components.file_store import LocalFileStore
from reme.steps.file_io.frontmatter_delete import FrontmatterDeleteStep
from reme.steps.file_io.frontmatter_read import FrontmatterReadStep
from reme.steps.file_io.frontmatter_update import FrontmatterUpdateStep
class temp_chdir:
"""Context manager to temporarily chdir into a path and restore on exit."""
def __init__(self, path):
self.path = path
self.old = None
def __enter__(self):
self.old = os.getcwd()
os.chdir(self.path)
return self
def __exit__(self, *exc):
os.chdir(self.old)
def _seed(workspace: Path, rel: str, body: str) -> Path:
target = workspace / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
return target
async def _make_store() -> LocalFileStore:
store = LocalFileStore(name="t_fm", embedding_store="")
await store.start()
return store
async def _run(step_cls, store: LocalFileStore, **kwargs):
step = step_cls(file_store=store)
await step(**kwargs)
return step.context.response
NOTE = "notes/post.md"
BODY = "---\nname: n\ntags:\n- a\n---\nbody\n"
@pytest.mark.asyncio
async def test_read_no_suffix_autoappends_md():
"""frontmatter_read on a suffix-less path resolves to the ``.md`` file."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
_seed(Path(tmp), NOTE, BODY)
store = await _make_store()
resp = await _run(FrontmatterReadStep, store, path="notes/post")
assert resp.success is True
assert resp.metadata["frontmatter"] == {"name": "n", "tags": ["a"]}
assert resp.metadata["path"] == "notes/post"
assert resp.metadata["resolved_path"] == "notes/post.md"
await store.close()
@pytest.mark.asyncio
async def test_read_honors_injected_path_scope():
"""An injected allowed-path scope denies frontmatter reads outside of it."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
_seed(Path(tmp), NOTE, BODY)
store = await _make_store()
allowed = await _run(FrontmatterReadStep, store, path=NOTE, _allowed_paths=[NOTE])
assert allowed.success is True
assert allowed.metadata["frontmatter"] == {"name": "n", "tags": ["a"]}
denied = await _run(FrontmatterReadStep, store, path=NOTE, _allowed_paths=["notes/other.md"])
assert denied.success is False
assert denied.metadata["error"] == "no permission to access this file"
await store.close()
@pytest.mark.asyncio
async def test_update_no_suffix_autoappends_md():
"""frontmatter_update on a suffix-less path resolves to the ``.md`` file."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
note = _seed(Path(tmp), NOTE, BODY)
store = await _make_store()
resp = await _run(FrontmatterUpdateStep, store, path="notes/post", metadata={"name": "renamed"})
assert resp.success is True
assert resp.metadata["updated"] == {"name": "renamed"}
assert resp.metadata["resolved_path"] == "notes/post.md"
assert "name: renamed" in note.read_text(encoding="utf-8")
await store.close()
@pytest.mark.asyncio
async def test_update_honors_injected_frontmatter_key_scope():
"""Injected key scope rejects the entire update when any key is outside it."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
note = _seed(Path(tmp), NOTE, BODY)
store = await _make_store()
before = note.read_bytes()
resp = await _run(
FrontmatterUpdateStep,
store,
path=NOTE,
metadata={"tags": ["new"], "name": "renamed"},
_allowed_frontmatter_keys=["tags"],
)
assert resp.success is False
assert resp.metadata["error"] == "frontmatter key(s) not allowed: 'name'"
assert note.read_bytes() == before
resp = await _run(
FrontmatterUpdateStep,
store,
path=NOTE,
metadata={"tags": ["new"]},
_allowed_frontmatter_keys=["tags"],
)
assert resp.success is True
assert frontmatter.loads(note.read_text(encoding="utf-8")).metadata == {"name": "n", "tags": ["new"]}
await store.close()
@pytest.mark.asyncio
async def test_update_empty_frontmatter_key_scope_denies_all_updates():
"""An empty injected list is restrictive, unlike an omitted constraint."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
note = _seed(Path(tmp), NOTE, BODY)
store = await _make_store()
before = note.read_bytes()
resp = await _run(
FrontmatterUpdateStep,
store,
path=NOTE,
metadata={"tags": ["new"]},
_allowed_frontmatter_keys=[],
)
assert resp.success is False
assert note.read_bytes() == before
await store.close()
@pytest.mark.asyncio
async def test_delete_no_suffix_autoappends_md():
"""frontmatter_delete on a suffix-less path resolves to the ``.md`` file."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
note = _seed(Path(tmp), NOTE, BODY)
store = await _make_store()
resp = await _run(FrontmatterDeleteStep, store, path="notes/post", keys=["tags"])
assert resp.success is True
assert resp.metadata["deleted"] == ["tags"]
assert resp.metadata["missing"] == []
assert resp.metadata["resolved_path"] == "notes/post.md"
assert "tags" not in note.read_text(encoding="utf-8")
await store.close()
@pytest.mark.asyncio
async def test_family_consistent_on_suffixless_path():
"""read / update / delete all succeed on the same suffix-less path."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
_seed(Path(tmp), NOTE, BODY)
store = await _make_store()
resp = await _run(FrontmatterReadStep, store, path="notes/post")
assert resp.success is True
resp = await _run(FrontmatterUpdateStep, store, path="notes/post", metadata={"status": "done"})
assert resp.success is True
resp = await _run(FrontmatterDeleteStep, store, path="notes/post", keys=["status"])
assert resp.success is True
assert resp.metadata["deleted"] == ["status"]
await store.close()
@pytest.mark.asyncio
async def test_read_missing_reports_probed_path():
"""not-found errors surface the probed ``.md`` path, not the raw input."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
store = await _make_store()
resp = await _run(FrontmatterReadStep, store, path="digest/missing")
assert resp.success is False
assert resp.metadata["exists"] is False
assert resp.metadata["path"] == "digest/missing"
assert resp.metadata["resolved_path"] == "digest/missing.md"
assert "digest/missing.md" in str(resp.answer)
await store.close()
@pytest.mark.asyncio
async def test_update_missing_reports_probed_path():
"""frontmatter_update not-found errors mention the probed ``.md`` path."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
store = await _make_store()
resp = await _run(FrontmatterUpdateStep, store, path="digest/missing", metadata={"x": 1})
assert resp.success is False
assert resp.metadata["error"] == "digest/missing.md not found"
assert resp.metadata["resolved_path"] == "digest/missing.md"
assert "digest/missing.md" in str(resp.answer)
await store.close()
@pytest.mark.asyncio
async def test_delete_missing_reports_probed_path():
"""frontmatter_delete not-found errors mention the probed ``.md`` path."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
store = await _make_store()
resp = await _run(FrontmatterDeleteStep, store, path="digest/missing", keys=["x"])
assert resp.success is False
assert resp.metadata["error"] == "digest/missing.md not found"
assert resp.metadata["resolved_path"] == "digest/missing.md"
assert "digest/missing.md" in str(resp.answer)
await store.close()
@pytest.mark.asyncio
async def test_non_md_rejected_without_resolved_path():
"""Non-markdown targets are rejected by all three siblings; no substitution."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
_seed(Path(tmp), "notes/data.txt", "plain\n")
store = await _make_store()
for step_cls, kwargs in (
(FrontmatterReadStep, {}),
(FrontmatterUpdateStep, {"metadata": {"x": 1}}),
(FrontmatterDeleteStep, {"keys": ["x"]}),
):
resp = await _run(step_cls, store, path="notes/data.txt", **kwargs)
assert resp.success is False
assert "not markdown" in str(resp.answer).lower()
assert "resolved_path" not in resp.metadata
await store.close()
@pytest.mark.asyncio
async def test_explicit_md_path_has_no_resolved_path():
"""Explicit ``.md`` paths need no substitution, so no ``resolved_path``."""
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
_seed(Path(tmp), NOTE, BODY)
store = await _make_store()
resp = await _run(FrontmatterReadStep, store, path=NOTE)
assert resp.success is True
assert "resolved_path" not in resp.metadata
await store.close()