* fix(auto-memory): serialize session transcript writes per session
_save_session_messages ran its read-merge-write on the per-session
JSONL transcript without any lock. Concurrent HTTP requests saving the
same session could interleave: two writers both appending against a
stale snapshot write overlapping history twice, and an append racing
a full rewrite can be overwritten by the rewriter's stale merge,
losing messages.
Hold get_path_lock(path) across the whole read, merge, and write,
consistent with the existing session image handling in the same file.
Merge, ordering, and can_append logic are unchanged.
Add deterministic regression tests that gate one writer's read between
snapshot capture and write: concurrent appends must merge disjoint
messages in created_at order, and an appended message must survive a
concurrent full rewrite. Both tests fail without the lock.
Closes#622
* test(auto-memory): make session-lock regression tests deterministic
Replace fixed sleeps with explicit synchronization: a shared operation
log records lock/read/write progress per writer, and writer A holds its
post-read pause until writer B is queued on the transcript lock. The
tests now assert the serialization order from the log (B reads only
after A's write completed) in addition to transcript content, so they
no longer depend on IO timing.
Also fix pylint findings from CI: use from-import for the auto_memory
module and drop unused tmp_path parameters.
* fix(search): accept CLI queries that look like numbers
The CLI parses key=value arguments into typed values, so query=404 reached the search steps as an int and failed on .strip(), and query=NaN or query=Infinity became non-finite floats that the HTTP client cannot encode as JSON and the MCP client sends as null.
Read the query as text in the four search steps, as ChatStep already does, and keep values that float() reads as NaN or infinity as strings.
* fix(search): treat only a missing query as empty
str(query or "") turned query=0 and query=0.0 into an empty string, so those searches still failed with "query cannot be empty". Check for None before converting instead.
The BEAM and LME plugins replace the default search job with their own SearchV2Step, which still called .strip() on the raw value, so query=404 raised AttributeError when either plugin was enabled. Read the query the same way there.
The numeric-query test is now parametrized over the six steps and passes 404, 0 and 0.0 through parse_kwargs first.
* fix(stat): put mtime in the answer where the job description promises it
The ``stat`` job is advertised to agents as "Stat path (size, mtime, exists,
is_dir, is_file)", but the answer only ever carried the path, type and size —
``mtime`` lived solely in metadata, and MCP returns only ``response.answer``.
So an agent asking how fresh a file is got no answer, even though the tool
description told it the field was there.
Render mtime in both branches of the answer:
stat: topics/n.md (file, 20 bytes, mtime 2026-09-21T15:50:52.880316)
stat: topics (dir, mtime 2026-09-21T15:50:52.880316)
The value is hoisted into a local so the answer and ``metadata["mtime"]`` are
guaranteed to come from the same ``stat()`` call rather than two reads. The
module docstring claimed the agent receives "size, mtime, mime type, and ...
frontmatter"; it now states that size and mtime are in the answer while mime
and frontmatter stay in metadata, matching reality.
Metadata is unchanged, so programmatic consumers see no difference. This is
the same answer-completeness class as #564, found while auditing which
advertised job fields actually reach the LLM.
* docs(stat): say which consumers actually see mime and frontmatter
Review feedback on #566: the module docstring told the agent it gets
"mime type and ... the parsed frontmatter alongside in metadata", which
reads as though an agent can use those fields to decide whether to
download or parse a file. It cannot: the MCP service returns only
`response.answer` (`mcp_tools.py` -> `return response.answer`), so
`metadata["mime"]` and `metadata["frontmatter"]` reach direct `Response`
consumers but never an MCP caller.
Spell the split out instead of implying one audience: the answer carries
path, type, size and mtime, and an agent that needs the frontmatter has
to read the file rather than stat it. The envelope description below is
unchanged, since it already lists the full metadata shape.
* fix(reindex): return a readable summary so Studio stops showing a placeholder
`ReindexStep` assigned the store's report dict straight to
`response.answer`. ReMe Studio's settings panel already anticipates a string
answer:
typeof result.answer === "string" && result.answer ? result.answer : t("indexRebuilt")
(`reme_studio/app/settings-center.tsx`), so with a dict answer that branch was
unreachable and every rebuild reported the generic "index rebuilt" text. The
operator never saw which scopes ran or how many entries each indexed.
Render the report as one summary line — `Reindexed scope=all: bm25=7,
embedding=5, tag=2` for the composite scope and `Reindexed scope=bm25: 3
indexed` for a single one. The report itself still lands in metadata, so
programmatic consumers are unaffected, and Studio needs no change because its
string branch now fires.
`reindex` is the only answer-carrying consumer of this step; no test asserted
the answer content, only metadata, so nothing else moved.
* chore(reindex): tighten the renderer and deduplicate the summary test
Review on #565: the helper's docstring carried the Studio background
that already lives in the PR description; the intermediate per_scope
dict collapses into the join generator; and the all-scope delegation
test now uses distinct counts (7, 5, 2) so it alone pins both the exact
summary answer and the full metadata report, replacing the overlapping
summary test.
* refactor(reindex): share summary return formatting
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
benchmark.yaml's python_execute job advertised a `timeout` parameter, but
`timeout` is one of the CLI's client-selection kwargs (reme/reme.py pops it
before the payload is sent), so `reme python_execute code=... timeout=300`
never delivered the timeout to the job. The value silently became the HTTP
client's request timeout while the subprocess kept the 60s default, so
benchmark code that needs longer was killed at 60s with no hint why. HTTP
and MCP callers were unaffected.
Rename the parameter to `python_timeout`, mirroring how the shell job
already avoids the same collision with `shell_timeout`. The step prefers
the new name and still accepts the legacy `timeout` context key so
existing benchmark configs keep working. A new guard test fails if any
job in reme/config/*.yaml or a plugin plugin.yaml declares a
client-reserved parameter name again.
* refactor(plugins): move BaseAgenticAnswerStep into lme/beam plugins
Move the shared benchmark answer base class from the core
reme.steps.benchmark package into each plugin's own src tree
(reme_lme.base_agentic_answer / reme_beam.base_agentic_answer) with
absolute imports, drop the core benchmark steps package, and update
plugin READMEs accordingly.
* refactor(plugins): simplify benchmark answer steps
* docs(plugins): clarify benchmark answer step migration
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* fix(auto-fin): parse topic IDs from fenced JSON replies
* fix(auto-fin): limit report agent tool calls in prompt
* refactor(auto-fin): research news by topic before market open
* fix(config): update default model version for claude_code backend
- Change model version from qwen3.8-max to qwen3.7-plus
- Use environment variable LLM_MODEL_NAME to allow override
- Ensure backend configuration reflects updated model setting
* refactor(auto-fin): write one note per topic before the daily digest
The merge step did two jobs at once: it researched every topic and
combined the results into a single report. Split it the way daily-paper
separates analysis from its brief, so each topic earns a durable note of
its own.
- auto_fin_research_step writes one note per topic that had relevant
news, tagged `kind: auto-fin-topic` and `topic` in frontmatter
- auto_fin_digest_step merges those notes into the day's brief with no
tools of its own and appends a `## 主题详解` section linking back to
each note
- a same-day rerun finds a topic's note by its `topic` frontmatter and
replaces it in place, deleting the old file when the title changed
- base.py now owns the shared Markdown layer: title sanitizing, report
normalizing, wikilink validation, note lookup, atomic frontmatter
writes, and change tracking, so both steps share one write path
- the DingTalk step maps `auto_fin_digest_path` to `markdown_path`
explicitly instead of relying on whichever step ran last
- drop the unused AutoFinTopicOutput schema and read `job_tools` from
the step config rather than hardcoding `search`
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(dingtalk): surface rejection details when delivery fails
A failed group send only reported HTTPStatusError, so an operator had to
reproduce the request by hand to learn why DingTalk refused it. Include
the status code and the whitelisted error keys from the response body in
both the log line and the raised RuntimeError.
Only `code`, `message`, and `requestid` are reported: the request body
carries the message content and credentials, so an error response that
echoes it back must not reach the log. Detail is truncated to 200 chars.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(test): make the suite green on CI
- Point the cookbook claude_code model assertion at qwen3.7-plus, the
default commit 9404e600 set, so the pre-existing red stops blocking
- Satisfy pylint on the auto-fin tests: prefer implicit booleaness for
the recorded Agent calls and drop an unused tmp_path fixture
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(auto-fin): name notes after the topic, not the Agent title
The research Agent returned a whole paragraph as its title; that became a
filename and blew past the filesystem's 255-byte name limit, failing with
ENAMETOOLONG inside resolve_note_path. Topics are configured values, so
they are short and predictable - use them for file names and keep the
Agent title in frontmatter.
- Name topic notes after the topic and the digest after the run date
- Fold a byte budget into normalize_title as a safety net for long topics
- Take an AutoFinReportOutput in _write_report instead of loose fields
- Ask both prompts for a short title now that it is display-only
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: isolate per-topic research failures and scope frontmatter reads
A single failing topic used to fail the whole cron job and discard the news
already gathered for the topics that had not run yet -- the 09-19 09:24 run
lost its robot notes that way. Research now logs the failure, continues with
the remaining topics, and only fails the run when no topic produced a note.
frontmatter_read was the only frontmatter step without the _allowed_paths
check that read, write, edit, and frontmatter_update already honour, so an
Agent scoped to one file could still read another file's metadata.
- Isolate per-topic research failures and report them as failed_topics
- Fail loudly when every topic fails so an empty brief is never sent
- Apply _check_path_permission in FrontmatterReadStep
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-fin): keep hand-edited notes and wikilink delimiters from breaking the run
Addresses three review findings on the topic-per-note rework.
- `find_note` and `read_note` now skip a note whose YAML frontmatter does
not parse. A hand-edited note in the day directory raised
`yaml.parser.ParserError`, which per-topic isolation surfaced as
"Auto Fin research failed for every topic" and took the run down with it.
- `normalize_title` also strips `[`, `]` and `#`, which `WikilinkHandler`
treats as target delimiters. `AI[算力]` used to emit a trailer link the
parser could not read at all, and `C#` resolved to `.../C` plus an anchor.
- `_write_report` returns the body it actually wrote, and both callers
propagate it, so the digest answer and the note handed to the digest Agent
no longer carry links that validation had already downgraded on disk.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf(auto_resource): reuse historical ownership lookup per batch
Share a lazy ownership index across routed processors and expire it at the end of each resource batch. Refresh changed days and their safe aliases after writes, deletes, and partial failures.
Add regression coverage for scan counts, ownership, lookup scope isolation, and retry behavior.
* fix(auto_resource): reconcile shared lookup across active batches
* fix(auto_resource): refresh batch ownership from filesystem metadata
* refactor(auto_resource): simplify batch ownership lookup
Keep per-day ownership caches paired with pre-read metadata and refresh changed days before subsequent resources. Remove reverse-index bookkeeping, post-read stabilization retries, and recursive scope fallback.
Handle late ELOOP errors on Python 3.13 while preserving other OS errors. Update regression tests for cross-resource freshness and retain safety and lifecycle coverage.
* refactor(auto_resource): streamline batch lookup refresh
* refactor(auto_resource): scope ownership cache to batch mutations
Replace filesystem snapshots with one lazy historical lookup shared by resource processors. Reload only days changed by this invocation and discard the cache when it ends.
Reuse canonical daily-note scans for symlink safety and Python 3.13 compatibility. Consolidate regression tests around the narrowed batch contract.
Validation: Python 3.11, 3.12 and 3.13 core/plugin suites each passed 1407 tests; pre-commit --all-files and offline lookup/deletion comparison passed.
* refactor(auto_resource): reuse initial ownership without refreshes
* test(auto_resource): streamline ownership lookup regression coverage
* feat(plugins): auto-tag generated reports
* fix(logging): forward host records on Python 3.13
* refactor(tags): decouple auto tagging from index updates
* fix(tags): bind auto tagging to configured index
* fix(tags): preserve standalone default index
* fix(tags): make auto tagging best effort
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* Add configurable tag indexing and tag listing
* Add tag-filtered hybrid search with exact candidate ranking
* Remove obsolete generated files
* Rename tag index key to tag_key and reject reserved fields
* Extract automatic tagging into a dedicated step
* Restrict frontmatter updates to authorized keys
* Refine tag filtering and automatic memory tagging
* Require underscore-separated tags in auto-tag prompts
- Forbid spaces in tags and require underscores (e.g. sam_altman) in both
English and Chinese auto_tag prompts, with English examples switched to
English entities (OpenAI, gold)
- Drop prompt-string assertions superseded by the new tagging rule
- Merge construction/runtime tag_key validation tests into one parametrized case
* Make max_tags_per_file configurable in auto-tag step
* Consolidate tag index tests
* Fix search test fixture lint warnings
* Align tag contracts and index health behavior
* Fall back when tag index is unavailable
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* refractor(proactive): upgrade proactive feature with disentangled job and steps
* refactor(proactive): apply audit fixes
- rename read-side job 'proactive' -> 'proactive_read' (less confusing vs the refresh pipeline)
- drop dedicated agent_wrapper.proactive; extraction reuses the default wrapper
- simplify schema: remove unused ProactiveExtractOutput/TopicUpdate, drop resource_paths
- extract no longer scans resource/ directly (daily notes already carry resource content)
- update tests and docs accordingly
* feat(proactive): strict extract-output gate and prompt total budget
- parse_extract_reply now requires a contract section (follow_ups/extends/updates
as a list); non-empty replies with misspelled section names trigger the
existing one-shot retry instead of silently checkpointing changed files
- pack_paths gains max_total_chars; extract packs newest daily material first,
keeps the first file on overflow, and records omitted files in a trailer
(default budget 300000 chars, configurable via max_total_chars)
- tests: schema gate unit, schema-error retry e2e, budget unit + e2e
* feat(proactive): add scenario-card plan step and generative agenda step
* feat(proactive): digest-personal profile personalization and leaner LLM contract
- extract/plan/agenda now draw a user profile block from <digest_dir>/personal/*.md
(frontmatter description + body excerpt, per-file budget, profile.md fallback)
- all daily access honours the configured daily_dir (prompt paths parameterized,
config-driven fallbacks) so workspaces using e.g. memory/ work unchanged
- schema trim: drop dead fields errors/material_paths, carry_forward_all -> count
- shrink LLM output contract: new topics emit title/reason/confidence/paths only;
keywords removed end-to-end, evidence derived from paths[0] (updates keep it)
* fix(proactive): skip checkpoint when extract reply stays unusable after retry
Two consecutive unparseable replies now short-circuit the round without
checkpointing, so the same material is retried next round instead of being
silently consumed (closes the residual audit #1 gap: the structural gate
detected schema-wrong output but a double failure still checkpointed).
* fix(proactive): replace running bool with reference-counted job activity tracker for the idle gate
* refactor(proactive): remove job activity tracking and idle gate, restore job tree to upstream
* fix(proactive): address second audit round (readonly reader, mtime checkpoint, wider fallbacks, profile containment, horizon content, expiry boundary)
* refactor(dream): strip interests.yaml ownership from dream, proactive is now the sole writer
* refactor(dream): separate proactive topic generation
* ci: update renamed auto dream smoke test
* fix(proactive): complete refresh migration and docs
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* fix(tag-index): fail closed and support reindexing
* fix(tag-index): preserve complete query expressions
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* feat: add entry-point plugin system
* fix: harden plugin config and client loading
* docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI
- Provide step-by-step instructions for updating project.version and merging branches
- Explain dependency verification for reme-ai on PyPI during build
- Specify requirements for GitHub Actions secret configuration and version uniqueness
- Describe manual workflow triggering and input of version number
- Recommend publishing order for related projects
- Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag
* feat: support plugin-defined component types
* refactor: simplify plugin configuration
* fix: isolate plugin loading and defer client fallback
* refactor: freeze built-in component registry
* fix: isolate config entry point loading
* fix: complete auto-fin package metadata
* feat: refine local-first research workflows
* fix: delegate structured output tool choice
* refactor(auto-fin): fetch and filter rolling CLS news
* fix(auto-fin): keep imports portable across platforms
* feat(auto-fin): expose CLS fetch controls
* fix(auto-fin): propagate configurable news window
* feat(auto_fin): normalize hybrid wikilinks in report body
- Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations
- Use regex to identify hybrid wikilinks with optional destinations
- Replace redundant destinations with simpler wikilink format for clarity
- Ensure normalization is failure-safe with exception handling and logging
- Update report body normalization process to apply hybrid wikilink fix
- Add unit tests to verify correct normalization and failure safety behavior
* fix(dream): serialize integration with application-wide asyncio lock
- Add application-wide asyncio.Lock to serialize digest writes during integration
- Update _snapshot_digest to capture metadata per bucket
- Validate bucket association when recovering from file changes
- Add tests ensuring recovery only from the correct bucket
- Add tests confirming integration lock is shared across application context
- Enhance strict topic YAML loading validation in dream utils
- Add tests for strict topic loading rejecting invalid or lossy fields
* fix(cookbook): enable configurable job_tools for digest and merge steps
- Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps
- Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list
- Modify AutoFinMergeStep to similarly read job_tools from kwargs
- Update tests to pass job_tools explicitly when invoking these steps
- Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling
* fix: retry incomplete dream receipts
* perf(pdf): increase max PDF pages limit from 20 to 35
- Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml
- Modified code to extract up to 35 pages instead of 20 in analyze.py
- Updated README and README_ZH to document the increased max_pdf_pages
- Adjusted unit test assertions to reflect new max_pdf_pages limit of 35
* fix memory integration and daily paper links
* docs clarify cookbook tool usage
* feat(daily-paper): add Hugging Face mirror switch
* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env
The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.
HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(daily-paper): address mirror configuration feedback
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents
Rework the auto-fin and daily-paper cookbooks to run on structured-output
LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with
data-source mirrors, and rewrite the affected unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(auto_fin): unify JSON output serialization and writing
- Extracted _write_output static method to serialize and write Pydantic models as compact JSON
- Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps
- Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep
- Updated auto_fin merge step to write output via new _write_output method
- Enhanced news reading with caching in AutoFinHistoryStep
- Refined returns calculation to handle events before close on non-trading days correctly
feat(daily_paper): improve note path resolution and metadata handling
- Introduced iter_note_metadata generator for safe Markdown frontmatter iteration
- Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles
- Updated analyze, collect, digest, and select steps to use centralized constants and helpers
- Used utc_now_iso for consistent timestamping in metadata
- Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps
- Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places
- Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management
- Improved error messages for filename validation in daily paper title normalization
feat(auto_fin): add multi-run cron schedules for intraday refinement
- Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates
- Each intraday run adds evidence cumulatively instead of replacing prior output wholly
- Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron
refactor(auto_fin_data): clean ETF code handling and page limits
- Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes"
- Added constants for major news and fund page limits to control pagination
- Improved ETF name extraction logic to handle missing fields consistently
fix(auto_fin_merge): fix report retrieval and merging logic
- Added support for getting current intra-day report in addition to previous day's report
- Modified merge template to include prior and current report sections for better context
- Adjusted report path handling to consistently use Path objects
test(auto_fin): add coverage for returns calculation and report retrieval
- Added test for returns when event occurs before close on non-trading day, checking next session entry
- Added test for previous and current report retrieval feeding merge context with disk files
- Extended test asserts for auto_fin cron schedule changes in config
style(daily_paper): reorder and cleanup imports
- Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import
- Cleaned up commented and unused imports across daily_paper steps
* feat: add configurable upstream mirror proxy
* style: format auto-fin data step
* fix: align cookbook mirrors and contracts
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(backend): improve workspace support for web clients
* fix(config): preserve the default workspace directory
* chore(reme): bump version to 0.4.1.5
- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file
* fix(chat): disable builtin tools in read-only mode
* fix(agent): make builtin tools opt-in
* fix(list): tolerate files removed during mtime sort
* fix(chat): expose complete read-only job set
On macOS, mimetypes.guess_type() may not recognize .md files, causing
stat to report application/octet-stream and breaking test assertions.
Explicitly map .md files to text/markdown so the behavior is stable
across platforms.
