Compare commits

..

No commits in common. "main" and "v0.4.1.7" have entirely different histories.

300 changed files with 3050 additions and 23467 deletions

View file

@ -1,97 +0,0 @@
name: Bug report
description: Report reproducible incorrect or unexpected ReMe behavior
title: "[Bug]: "
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve ReMe. Please remove secrets, API keys, and private memory content before submitting.
- type: textarea
id: description
attributes:
label: Description
description: What happened, and what did you expect instead?
placeholder: Describe the observed and expected behavior.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Provide the smallest configuration and command sequence that reproduces the problem.
placeholder: |
1. Configure ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: config
attributes:
label: Relevant configuration
description: Include only relevant values and redact credentials, tokens, endpoints, and private paths.
render: yaml
- type: textarea
id: logs
attributes:
label: Logs or traceback
description: Paste relevant output after removing secrets and private workspace content.
render: shell
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
validations:
required: true
- type: input
id: python-version
attributes:
label: Python version
placeholder: e.g. 3.11.9
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- Linux
- macOS
- Windows
- Other
validations:
required: true
- type: dropdown
id: area
attributes:
label: Affected area
options:
- CLI or configuration
- HTTP, MCP, or local service
- Memory or workspace files
- Search, catalog, graph, or index
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Packaging or installation
- Other
validations:
required: true
- type: checkboxes
id: safety
attributes:
label: Data safety
options:
- label: I removed credentials and private memory content from this report.
required: true

View file

@ -1,8 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: ReMe documentation
url: https://reme.agentscope.io
about: Read the installation, configuration, and usage guides.
- name: Existing issues
url: https://github.com/agentscope-ai/ReMe/issues
about: Search for existing reports and discussions before opening a new issue.

View file

@ -1,64 +0,0 @@
name: Feature request
description: Propose a focused enhancement to ReMe
title: "[Feature]: "
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What user problem or limitation should this change address?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed behavior
description: Describe the desired behavior and its user-visible contract.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- CLI or configuration
- Jobs or steps
- Memory or workspace files
- Search, catalog, graph, or index
- Service or client
- Model or agent integration
- ReMe Studio
- Plugin or external integration
- Documentation
- Other
validations:
required: true
- type: textarea
id: ownership
attributes:
label: Local-first and compatibility considerations
description: Explain any effect on user-owned files, rebuildable state, configuration, schemas, or service interfaces.
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe workarounds or alternative designs you considered.
- type: textarea
id: examples
attributes:
label: Example usage
description: Show the proposed CLI, configuration, API, or UI behavior when useful.
render: shell
- type: checkboxes
id: contribution
attributes:
label: Contribution
options:
- label: I am willing to help implement or test this feature.

View file

@ -1,53 +0,0 @@
name: Usage question
description: Ask for help using or configuring ReMe
title: "[Question]: "
labels: [question]
body:
- type: markdown
attributes:
value: Please check the documentation and existing issues before asking a new question.
- type: textarea
id: goal
attributes:
label: What are you trying to achieve?
validations:
required: true
- type: textarea
id: attempted
attributes:
label: What have you tried?
description: Include relevant commands or configuration, with secrets and private memory content removed.
validations:
required: true
- type: input
id: reme-version
attributes:
label: ReMe version
placeholder: e.g. 0.4.1.8 or a commit SHA
- type: dropdown
id: area
attributes:
label: Area
options:
- Installation
- Configuration
- CLI or service usage
- Memory and workspace management
- Search and retrieval
- ReMe Studio
- Plugin or integration
- Other
- type: checkboxes
id: checked
attributes:
label: Before submitting
options:
- label: I checked the [ReMe documentation](https://reme.agentscope.io) and searched existing issues.
required: true
- label: I removed credentials and private memory content.
required: true

View file

@ -1,35 +0,0 @@
## Summary
<!-- Explain the problem and the smallest coherent change that addresses it. -->
## Related issue
<!-- Use "Fixes #123" when applicable. -->
## Contract and data impact
- [ ] No public configuration, schema, CLI, endpoint, streaming, or workspace-layout contract changes
- [ ] No user-owned memory files are deleted or rewritten
- [ ] Derived indexes, catalogs, graphs, caches, and metadata remain rebuildable
<!-- If any item is unchecked, describe the impact and migration or recovery path. -->
## Validation
<!-- List the exact checks run and their results. Explain relevant checks that were not run. -->
- [ ] Focused tests pass
- [ ] Unit tests pass, or omitted tests are explained below
- [ ] `pre-commit run --all-files` passes, or omitted checks are explained below
- [ ] Frontend checks were run when `reme_studio/` changed
## Checklist
- [ ] I reviewed the diff for unrelated changes and sensitive data
- [ ] Tests cover intentional behavior changes
- [ ] Defaults, schemas, and concise documentation were updated together when required
- [ ] Long-lived clients, tasks, services, and executors follow the application lifecycle
## Screenshots or additional notes
<!-- Include UI screenshots, compatibility notes, or follow-up work when relevant. -->

View file

@ -1,58 +0,0 @@
name: _Build documentation
on:
workflow_call:
inputs:
run_tests:
description: Run the documentation test suite before building
required: false
default: true
type: boolean
upload_pages_artifact:
description: Upload the build for a later GitHub Pages deployment job
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
build:
name: Build documentation
runs-on: ubuntu-latest
defaults:
run:
working-directory: github-pages
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
cache: npm
cache-dependency-path: github-pages/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
if: inputs.run_tests
run: npm test
- name: Build documentation
run: npm run build
- name: Configure Pages
if: inputs.upload_pages_artifact
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
- name: Upload Pages artifact
if: inputs.upload_pages_artifact
uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4
with:
path: github-pages/dist

View file

@ -1,88 +0,0 @@
name: _Build Python packages
on:
workflow_call:
inputs:
expected_version:
description: Expected release version; omit for a consistency-only check
required: false
default: ''
type: string
upload_artifacts:
description: Upload distributions for later publish jobs
required: false
default: false
type: boolean
permissions:
contents: read
jobs:
distributions:
name: Build Python distributions
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest twine
- name: Validate package versions
if: inputs.expected_version == ''
run: python scripts/bump_version.py --check
- name: Validate release version
if: inputs.expected_version != ''
env:
EXPECTED_VERSION: ${{ inputs.expected_version }}
run: python scripts/bump_version.py --check --expected-version "${EXPECTED_VERSION}"
- name: Run package tests
run: PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
- name: Build and check distributions
run: |
mkdir -p dist/reme
python -m build --outdir dist/reme
python -m twine check dist/reme/*
- name: Verify distributions and isolated installation
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme/web/')
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme_studio/')
python -m venv "${RUNNER_TEMP}/reme-package-smoke"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -m pip install "${REME_WHEEL}[as]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -c "import reme"
- name: Verify released core dependencies
if: inputs.expected_version != ''
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
python -m venv "${RUNNER_TEMP}/reme-core-package-smoke"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" -m pip install "${REME_WHEEL}[core]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-core-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- name: Upload ReMe distributions
if: inputs.upload_artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-distributions
path: dist/reme/
if-no-files-found: error

View file

@ -1,48 +0,0 @@
name: CI / Documentation
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-docs.yml'
- '.github/workflows/_build-docs.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'reme_studio/README*.md'
- 'reme_studio/public/og.jpg'
- 'typescript/README*.md'
- 'plugins/*/README*.md'
- 'benchmark/*/README*.md'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-docs.yml'
- '.github/workflows/_build-docs.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'reme_studio/README*.md'
- 'reme_studio/public/og.jpg'
- 'typescript/README*.md'
- 'plugins/*/README*.md'
- 'benchmark/*/README*.md'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
documentation:
name: Test and build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true

View file

@ -1,40 +0,0 @@
name: CI / Python packages
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-packages.yml'
- '.github/workflows/_build-python-packages.yml'
- '.github/workflows/release-python.yml'
- 'pyproject.toml'
- 'README.md'
- 'reme/**'
- 'scripts/bump_version.py'
- 'tests/unit/test_package_versions.py'
- 'LICENSE'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
distributions:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml

View file

@ -1,40 +0,0 @@
name: CI / Python quality
on:
push:
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
pre-commit:
name: Pre-commit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
cache: pip
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e reme_studio -e ".[dev,core]"
pip install -q --no-deps -e plugins/auto-fin -e plugins/daily_paper
- name: Pre-commit starts
run: pre-commit run --all-files

View file

@ -1,90 +0,0 @@
name: CI / ReMe Studio
on:
push:
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
pull_request:
paths:
- "reme_studio/**"
- ".github/workflows/ci-reme-studio.yml"
- ".github/workflows/release-reme-studio.yml"
- "scripts/package_studio.py"
- "tests/unit/test_package_versions.py"
- "pyproject.toml"
- "LICENSE"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
studio:
name: Studio checks
runs-on: ubuntu-latest
defaults:
run:
working-directory: reme_studio
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Setup Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run format check
run: npm run format:check
- name: Run lint
run: npm run lint
- name: Run tests
run: npm test
- name: Verify npm package
run: |
npm pack --pack-destination "${RUNNER_TEMP}"
tar -tzf "${RUNNER_TEMP}"/agentscope-ai-reme_studio-*.tgz | grep '^package/dist-static/index.html$'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.11"
- name: Build and verify Python package
working-directory: .
run: |
python -m pip install build packaging pytest twine
PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio
python -m twine check dist/studio/*
STUDIO_WHEEL="$(pwd)/$(ls dist/studio/reme_studio-*.whl)"
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY

View file

@ -1,51 +0,0 @@
name: CI / TypeScript integrations
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-typescript.yml'
- '.github/workflows/release-typescript.yml'
- 'typescript/**'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/ci-typescript.yml'
- '.github/workflows/release-typescript.yml'
- 'typescript/**'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
package:
name: Type-check, test, and pack
runs-on: ubuntu-latest
defaults:
run:
working-directory: typescript
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
cache: npm
cache-dependency-path: typescript/package-lock.json
- run: npm ci
- run: npm run format:check
- run: npm run lint
- run: npm run typecheck
- run: npm test
- run: npm run test:package
- name: Validate OpenClaw package contract
run: npx --yes clawhub@0.23.3 package validate . --json

View file

@ -1,52 +0,0 @@
name: Deploy / Documentation
on:
push:
branches: [main]
paths:
- "github-pages/**"
- "docs/**"
- "README.md"
- "README_ZH.md"
- "reme_studio/README*.md"
- "reme_studio/public/og.jpg"
- "typescript/README*.md"
- "plugins/*/README*.md"
- "benchmark/*/README*.md"
- "AGENTS.md"
- ".github/workflows/deploy-docs.yml"
- ".github/workflows/_build-docs.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
name: Build documentation
uses: ./.github/workflows/_build-docs.yml
with:
run_tests: true
upload_pages_artifact: true
permissions:
contents: read
pages: write
id-token: write
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
permissions:
pages: write
id-token: write
steps:
- name: Deploy
id: deployment
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5

View file

@ -0,0 +1,61 @@
name: GitHub Pages Check
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/github-pages-check.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'website/README*.md'
- 'website/public/og.jpg'
- 'cookbook/*/README*.md'
- 'benchmark/*/README*.md'
- 'skills/reme_memory/SKILL.md'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/github-pages-check.yml'
- 'AGENTS.md'
- 'README.md'
- 'README_ZH.md'
- 'docs/**'
- 'github-pages/**'
- 'website/README*.md'
- 'website/public/og.jpg'
- 'cookbook/*/README*.md'
- 'benchmark/*/README*.md'
- 'skills/reme_memory/SKILL.md'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
test-and-build:
runs-on: ubuntu-latest
defaults:
run:
working-directory: github-pages
steps:
- uses: actions/checkout@v6
- name: Set up Node
uses: actions/setup-node@v6
with:
node-version: '22.13'
cache: npm
cache-dependency-path: github-pages/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Build documentation
run: npm run build

41
.github/workflows/npm-format.yml vendored Normal file
View file

@ -0,0 +1,41 @@
name: NPM Format
on:
push:
paths:
- "website/**"
- ".github/workflows/npm-format.yml"
pull_request:
paths:
- "website/**"
- ".github/workflows/npm-format.yml"
jobs:
website:
name: Website checks
runs-on: ubuntu-latest
defaults:
run:
working-directory: website
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: website/package-lock.json
- name: Install dependencies
run: npm ci
- name: Run format check
run: npm run format:check
- name: Run lint
run: npm run lint
- name: Run tests
run: npm test

97
.github/workflows/package-check.yml vendored Normal file
View file

@ -0,0 +1,97 @@
name: Package Check
on:
push:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/package-check.yml'
- '.github/workflows/python-publish.yml'
- 'packages/reme_ai_studio/**'
- 'pyproject.toml'
- 'reme/__init__.py'
- 'reme/utils/web_static.py'
- 'scripts/bump_version.py'
- 'scripts/package_studio.py'
- 'tests/unit/test_package_versions.py'
- 'website/**'
- 'LICENSE'
pull_request:
branches: [main, master, dev, develop]
paths:
- '.github/workflows/package-check.yml'
- '.github/workflows/python-publish.yml'
- 'packages/reme_ai_studio/**'
- 'pyproject.toml'
- 'reme/__init__.py'
- 'reme/utils/web_static.py'
- 'scripts/bump_version.py'
- 'scripts/package_studio.py'
- 'tests/unit/test_package_versions.py'
- 'website/**'
- 'LICENSE'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
distributions:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: '22.13'
cache: npm
cache-dependency-path: website/package-lock.json
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest twine
- name: Validate release versions
run: python scripts/bump_version.py --check
- name: Run package tests
run: PYTHONPATH=. python -m pytest tests/unit/test_package_versions.py -q
- name: Build Studio static workspace
working-directory: website
run: |
npm ci
npm run build:static
- name: Build and check distributions
run: |
python scripts/package_studio.py
mkdir -p dist/reme dist/studio
python -m build --outdir dist/reme
python -m build packages/reme_ai_studio --outdir dist/studio
python -m twine check dist/reme/* dist/studio/*
- name: Verify distributions and isolated installation
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
STUDIO_WHEEL="$(pwd)/$(ls dist/studio/reme_ai_studio-*.whl)"
STUDIO_SDIST="$(pwd)/$(ls dist/studio/reme_ai_studio-*.tar.gz)"
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme/web/')
python -m zipfile -l "${STUDIO_WHEEL}" | grep 'reme_ai_studio/static/index.html'
python -m zipfile -l "${STUDIO_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${STUDIO_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-package-smoke"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -m pip install \
--find-links "$(pwd)/dist/studio" "${REME_WHEEL}[core]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -c \
"import reme; from reme_ai_studio import static_dir; assert (static_dir() / 'index.html').is_file()"
"${RUNNER_TEMP}/reme-package-smoke/bin/python" -c \
"from reme.utils import resolve_web_static_dir; assert (resolve_web_static_dir() / 'index.html').is_file()"

68
.github/workflows/pages.yml vendored Normal file
View file

@ -0,0 +1,68 @@
name: Deploy ReMe documentation
on:
push:
branches: [main]
paths:
- "github-pages/**"
- "docs/**"
- "README.md"
- "README_ZH.md"
- "website/README*.md"
- "website/public/og.jpg"
- "cookbook/*/README*.md"
- "benchmark/*/README*.md"
- "skills/reme_memory/SKILL.md"
- "AGENTS.md"
- ".github/workflows/pages.yml"
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: pages
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "22.13"
cache: npm
cache-dependency-path: github-pages/package-lock.json
- name: Install dependencies
working-directory: github-pages
run: npm ci
- name: Build documentation
working-directory: github-pages
run: npm run build
- name: Configure Pages
uses: actions/configure-pages@v5
- name: Upload Pages artifact
uses: actions/upload-pages-artifact@v4
with:
path: github-pages/dist
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy
id: deployment
uses: actions/deploy-pages@v4

View file

@ -1,20 +1,16 @@
name: Policy / PR title
name: PR Title Check
on:
pull_request:
branches: [main, master, dev, develop]
types: [opened, edited, synchronize, reopened]
permissions:
contents: read
pull-requests: read
jobs:
check-pr-title:
runs-on: ubuntu-latest
steps:
- name: Check PR title format
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
uses: amannn/action-semantic-pull-request@v6.1.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:

38
.github/workflows/pre-commit.yml vendored Normal file
View file

@ -0,0 +1,38 @@
name: Pre-commit
on: [ push, pull_request ]
jobs:
run:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: True
matrix:
os: [ ubuntu-latest ]
env:
OS: ${{ matrix.os }}
PYTHON: '3.11'
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Update setuptools
run: |
pip install -U setuptools wheel
- name: Install
run: |
pip install -q -e packages/reme_ai_studio -e ".[dev,core]"
- name: Install pre-commit
run: |
pre-commit install
- name: Pre-commit starts
run: |
pre-commit run --all-files > pre-commit.log 2>&1 || true
cat pre-commit.log
if grep -q Failed pre-commit.log; then
echo -e "\e[41m [**FAIL**] Please install pre-commit and format your code first. \e[0m"
exit 1
fi
echo -e "\e[46m ********************************Passed******************************** \e[0m"

123
.github/workflows/python-publish.yml vendored Normal file
View file

@ -0,0 +1,123 @@
name: Publish Python packages to PyPI
on:
workflow_dispatch:
inputs:
version:
description: Release version
required: true
type: string
release:
types: [published]
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ github.event_name == 'release' && github.event.release.tag_name || inputs.version }}
steps:
- uses: actions/checkout@v6
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: '22.13'
cache: npm
cache-dependency-path: website/package-lock.json
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.11'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging twine
- name: Validate release version
run: python scripts/bump_version.py --check --expected-version "${RELEASE_VERSION}"
- name: Build Studio static workspace
working-directory: website
run: |
npm ci
npm run build:static
- name: Prepare and build distributions
run: |
python scripts/package_studio.py
mkdir -p dist/reme dist/studio
python -m build --outdir dist/reme
python -m build packages/reme_ai_studio --outdir dist/studio
python -m twine check dist/reme/* dist/studio/*
- name: Verify distributions and isolated installation
run: |
REME_WHEEL="$(pwd)/$(ls dist/reme/reme_ai-[0-9]*.whl)"
STUDIO_WHEEL="$(pwd)/$(ls dist/studio/reme_ai_studio-*.whl)"
STUDIO_SDIST="$(pwd)/$(ls dist/studio/reme_ai_studio-*.tar.gz)"
python -m zipfile -l "${REME_WHEEL}" | (! grep 'reme/web/')
python -m zipfile -l "${STUDIO_WHEEL}" | grep 'reme_ai_studio/static/index.html'
python -m zipfile -l "${STUDIO_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${STUDIO_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-release-smoke"
"${RUNNER_TEMP}/reme-release-smoke/bin/python" -m pip install \
--find-links "$(pwd)/dist/studio" "${REME_WHEEL}[core]"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-release-smoke/bin/python" -c \
"import reme; from reme_ai_studio import static_dir; assert (static_dir() / 'index.html').is_file()"
"${RUNNER_TEMP}/reme-release-smoke/bin/python" -c \
"from reme.utils import resolve_web_static_dir; assert (resolve_web_static_dir() / 'index.html').is_file()"
- name: Upload ReMe Studio distributions
uses: actions/upload-artifact@v4
with:
name: reme-studio-distributions
path: dist/studio/
- name: Upload ReMe distributions
uses: actions/upload-artifact@v4
with:
name: reme-distributions
path: dist/reme/
publish-studio:
needs: build
runs-on: ubuntu-latest
steps:
- name: Download ReMe Studio distributions
uses: actions/download-artifact@v4
with:
name: reme-studio-distributions
path: dist/studio
- name: Publish ReMe Studio
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}
packages-dir: dist/studio
skip-existing: true
publish-reme:
needs: publish-studio
runs-on: ubuntu-latest
steps:
- name: Download ReMe distributions
uses: actions/download-artifact@v4
with:
name: reme-distributions
path: dist/reme
- name: Publish ReMe
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}
packages-dir: dist/reme
skip-existing: true

View file

@ -1,157 +0,0 @@
# 发布操作手册:
# 1. 先将 plugins/auto-fin/pyproject.toml 中的 project.version 更新为待发布版本并合入目标分支。
# 2. 确认插件依赖的 reme-ai 版本已经发布到 PyPI本工作流会在构建阶段验证该依赖可下载。
# 3. 确认 PyPI Trusted Publisher 已绑定本仓库、此工作流和 pypi environment且 PyPI 上不存在相同版本。
# 4. 在 GitHub 仓库的 Actions 页面选择“Release / Auto Fin plugin”点击“Run workflow”。
# 5. 输入与 project.version 完全一致的版本号(例如 0.1.0)后运行;版本也可以带 v 前缀。
#
# 推荐发布顺序reme-ai -> reme-auto-fin -> QwenPaw 更新依赖并通过 plugins: [auto-fin] 启用。
# 当前仅支持 workflow_dispatch 手动触发,不会因 push、tag 或 release 自动发布。
name: Release / Auto Fin plugin
run-name: Publish reme-auto-fin ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/auto-fin/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-auto-fin
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install --no-deps -e plugins/auto-fin
- name: Validate package name and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/auto-fin/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-auto-fin":
raise SystemExit(f"Expected project name 'reme-auto-fin', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [requirement for requirement in project["dependencies"] if requirement.startswith("reme-ai")]
if len(requirements) != 1:
raise SystemExit(f"Expected one reme-ai dependency, found {requirements!r}")
reme_requirement = Requirement(requirements[0])
if reme_requirement.name != "reme-ai" or reme_requirement.extras:
raise SystemExit(f"Expected a base reme-ai dependency, found {requirements[0]!r}")
if Version("0.4.1.8") in reme_requirement.specifier or Version("0.4.1.9") not in reme_requirement.specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.9, found {requirements[0]!r}")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirement}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Auto Fin tests
run: python -m pytest plugins/auto-fin -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-auto-fin-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/auto-fin
python -m build plugins/auto-fin --outdir dist/auto-fin
python -m twine check dist/auto-fin/*
- name: Verify distributions and isolated installation
run: |
AUTO_FIN_WHEEL="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.whl)"
AUTO_FIN_SDIST="$(pwd)/$(ls dist/auto-fin/reme_auto_fin-*.tar.gz)"
python -m zipfile -l "${AUTO_FIN_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${AUTO_FIN_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-auto-fin-smoke"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" -m pip install \
"agentscope[model-ollama]==2.0.7" "${AUTO_FIN_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-auto-fin-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-auto-fin")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["auto-fin"].value == "reme_auto_fin"
manifest = load_package_manifest("reme_auto_fin", plugin_name="auto-fin")
assert set(manifest.backends) == {
"auto_fin_data_step",
"auto_fin_topic_step",
"auto_fin_merge_step",
}
assert set(manifest.application_defaults["jobs"]) == {
"auto_fin",
"auto_fin_cron",
}
PY
- name: Upload distributions
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-auto-fin-${{ inputs.version }}
path: dist/auto-fin
- name: Publish reme-auto-fin
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/auto-fin

View file

@ -1,157 +0,0 @@
# Release checklist:
# 1. Update project.version in plugins/daily_paper/pyproject.toml and merge it into the target branch.
# 2. Publish the required reme-ai version before this plugin; the build verifies that dependency on PyPI.
# 3. Configure PyPI Trusted Publishing for this repository/workflow and its pypi environment.
# 4. Run "Release / Daily Paper plugin" from GitHub Actions with the exact project version (a v prefix is accepted).
#
# Recommended order: reme-ai -> reme-daily-paper -> downstream applications enabling plugins: [daily-paper].
# This workflow is intentionally manual and never publishes from a push, tag, or GitHub release event.
name: Release / Daily Paper plugin
run-name: Publish reme-daily-paper ${{ inputs.version }}
on:
workflow_dispatch:
inputs:
version:
description: Version from plugins/daily_paper/pyproject.toml (for example, 0.1.0)
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-daily-paper
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.11'
- name: Install test and build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build packaging pytest pytest-asyncio twine
python -m pip install -e ".[core]"
python -m pip install -e plugins/daily_paper
- name: Validate package name, dependencies, and release version
id: package
run: |
python - "${RELEASE_VERSION}" <<'PY'
import os
import sys
import tomllib
from pathlib import Path
from packaging.requirements import Requirement
from packaging.version import Version
project = tomllib.loads(Path("plugins/daily_paper/pyproject.toml").read_text(encoding="utf-8"))["project"]
expected = Version(sys.argv[1].removeprefix("v"))
actual = Version(project["version"])
if project["name"] != "reme-daily-paper":
raise SystemExit(f"Expected project name 'reme-daily-paper', found {project['name']!r}")
if actual != expected:
raise SystemExit(f"Package version is {actual}, but workflow input is {expected}")
requirements = [Requirement(value) for value in project["dependencies"]]
reme_requirements = [requirement for requirement in requirements if requirement.name == "reme-ai"]
if len(reme_requirements) != 1 or reme_requirements[0].extras:
raise SystemExit(f"Expected one base reme-ai dependency, found {reme_requirements!r}")
if Version("0.4.1.8") in reme_requirements[0].specifier or Version("0.4.1.9") not in reme_requirements[0].specifier:
raise SystemExit(f"Expected reme-ai>=0.4.1.9, found {reme_requirements!r}")
if sum(requirement.name == "pypdf" for requirement in requirements) != 1:
raise SystemExit("Expected exactly one pypdf dependency")
with Path(os.environ["GITHUB_OUTPUT"]).open("a", encoding="utf-8") as output:
print(f"reme_requirement={reme_requirements[0]}", file=output)
print(f"Publishing {project['name']} {actual}")
PY
- name: Run Daily Paper tests
run: python -m pytest plugins/daily_paper -q
- name: Require the plugin-enabled ReMe release on PyPI
env:
REME_REQUIREMENT: ${{ steps.package.outputs.reme_requirement }}
run: |
python -m pip download --no-deps \
--dest "${RUNNER_TEMP}/reme-daily-paper-base" \
"${REME_REQUIREMENT}"
- name: Build and check distributions
run: |
mkdir -p dist/daily-paper
python -m build plugins/daily_paper --outdir dist/daily-paper
python -m twine check dist/daily-paper/*
- name: Verify distributions and isolated installation
run: |
DAILY_PAPER_WHEEL="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.whl)"
DAILY_PAPER_SDIST="$(pwd)/$(ls dist/daily-paper/reme_daily_paper-*.tar.gz)"
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/plugin.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'reme_daily_paper/analyze.yaml'
python -m zipfile -l "${DAILY_PAPER_WHEEL}" | grep 'dist-info/licenses/LICENSE'
python -m tarfile -l "${DAILY_PAPER_SDIST}" | grep '/LICENSE'
python -m venv "${RUNNER_TEMP}/reme-daily-paper-smoke"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" -m pip install \
"agentscope[model-ollama]==2.0.7" "${DAILY_PAPER_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-daily-paper-smoke/bin/python" - <<'PY'
from importlib.metadata import distribution
from reme.plugin_manifest import load_package_manifest
package = distribution("reme-daily-paper")
plugins = {entry.name: entry for entry in package.entry_points if entry.group == "reme.plugins"}
assert plugins["daily-paper"].value == "reme_daily_paper"
manifest = load_package_manifest("reme_daily_paper", plugin_name="daily-paper")
assert set(manifest.backends) == {
"daily_paper_collect_step",
"daily_paper_rank_step",
"daily_paper_select_step",
"daily_paper_analyze_step",
"daily_paper_digest_step",
}
assert set(manifest.application_defaults["jobs"]) == {"daily_paper", "daily_paper_cron"}
PY
- name: Upload distributions
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper/
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-daily-paper-${{ inputs.version }}
path: dist/daily-paper
- name: Publish reme-daily-paper
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/daily-paper

View file

@ -1,47 +0,0 @@
name: Release / Python packages
# Configure a PyPI Trusted Publisher for this repository, workflow, and its
# pypi environment before running the manual release.
on:
workflow_dispatch:
inputs:
version:
description: Release version
required: true
type: string
permissions:
contents: read
concurrency:
group: publish-reme-ai
cancel-in-progress: false
jobs:
build:
name: Build and verify distributions
uses: ./.github/workflows/_build-python-packages.yml
with:
expected_version: ${{ inputs.version }}
upload_artifacts: true
publish-reme:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- name: Download ReMe distributions
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-distributions
path: dist/reme
- name: Publish ReMe
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/reme
skip-existing: true

View file

@ -1,158 +0,0 @@
# Release checklist:
# 1. Update reme_studio/pyproject.toml, package.json, and package-lock.json to the same Studio version.
# 2. Configure npm Trusted Publishing and PyPI Trusted Publishing with the pypi environment.
# 3. Run this workflow manually with the exact Studio version.
name: Release / ReMe Studio
run-name: Publish ReMe Studio ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Version from the Studio Python and npm manifests
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
permissions:
contents: read
concurrency:
group: publish-reme-studio
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
env:
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "22.22.3"
cache: npm
cache-dependency-path: reme_studio/package-lock.json
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.11"
- name: Validate Studio package names and version
run: |
python - <<'PY'
import json
import os
import tomllib
from pathlib import Path
studio = Path("reme_studio")
python_manifest = tomllib.loads((studio / "pyproject.toml").read_text(encoding="utf-8"))["project"]
npm_manifest = json.loads((studio / "package.json").read_text(encoding="utf-8"))
expected = os.environ["RELEASE_VERSION"].removeprefix("v")
if python_manifest["name"] != "reme_studio":
raise SystemExit(f"Unexpected Python package name: {python_manifest['name']}")
if npm_manifest["name"] != "@agentscope-ai/reme_studio":
raise SystemExit(f"Unexpected npm package name: {npm_manifest['name']}")
if python_manifest["version"] != expected or npm_manifest["version"] != expected:
raise SystemExit(
f"Studio manifests are {python_manifest['version']} and {npm_manifest['version']}; "
f"workflow input is {expected}",
)
prerelease = "-" in expected
if prerelease != (os.environ["NPM_TAG"] == "next"):
raise SystemExit("Prereleases must use next; stable releases must use latest")
PY
- name: Install dependencies and run checks
working-directory: reme_studio
run: |
npm ci
npm run format:check
npm run lint
npm test
- name: Build Studio distributions
run: |
python -m pip install build twine
mkdir -p dist/studio-python dist/studio-npm
npm pack ./reme_studio --pack-destination dist/studio-npm
python scripts/package_studio.py
python -m build reme_studio --outdir dist/studio-python
python -m twine check dist/studio-python/*
- name: Verify Studio distributions and isolated installation
run: |
STUDIO_WHEEL="$(pwd)/$(ls dist/studio-python/reme_studio-*.whl)"
tar -tzf dist/studio-npm/*.tgz | grep '^package/dist-static/index.html$'
python -m venv "${RUNNER_TEMP}/reme-studio-package-smoke"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" -m pip install "${STUDIO_WHEEL}"
cd "${RUNNER_TEMP}"
"${RUNNER_TEMP}/reme-studio-package-smoke/bin/python" - <<'PY'
from reme_studio import static_dir
assert (static_dir() / "index.html").is_file()
PY
- uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: reme-studio-${{ inputs.version }}
path: |
dist/studio-python/*
dist/studio-npm/*
if-no-files-found: error
publish-python:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Publish ReMe Studio to PyPI
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/studio-python
skip-existing: true
publish-npm:
needs: build
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: "24"
registry-url: https://registry.npmjs.org
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: reme-studio-${{ inputs.version }}
path: dist
- name: Publish ReMe Studio to npm
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/studio-npm/*.tgz --access public --tag "${NPM_TAG}" --provenance

View file

@ -1,167 +0,0 @@
# Release checklist:
# 1. Update typescript/package.json and package-lock.json to the release version and merge them.
# 2. Configure npm Trusted Publishing for agentscope-ai/ReMe and this workflow file.
# 3. Run this workflow manually with the exact package version (an optional v prefix is accepted).
# 4. Configure ClawHub Trusted Publishing or CLAWHUB_TOKEN before enabling ClawHub publication.
# 5. Use the `next` tag for prereleases and `latest` only for stable releases.
name: Release / TypeScript integrations
run-name: Publish @agentscope-ai/reme ${{ inputs.version }} (${{ inputs.npm_tag }})
on:
workflow_dispatch:
inputs:
version:
description: Version from typescript/package.json (for example, 0.1.0)
required: true
type: string
npm_tag:
description: npm distribution tag
required: true
default: latest
type: choice
options:
- next
- latest
publish_clawhub:
description: Also publish the verified tarball to ClawHub
required: true
default: false
type: boolean
permissions:
contents: read
concurrency:
group: publish-agentscope-ai-reme
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.validate.outputs.version }}
env:
RELEASE_VERSION: ${{ inputs.version }}
NPM_TAG: ${{ inputs.npm_tag }}
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Set up Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '22.22.3'
- name: Validate package name and release version
id: validate
working-directory: typescript
run: |
node --input-type=module <<'JS'
import { appendFileSync, readFileSync } from 'node:fs';
const manifest = JSON.parse(readFileSync('package.json', 'utf8'));
const expected = process.env.RELEASE_VERSION.replace(/^v/, '');
if (manifest.name !== '@agentscope-ai/reme') {
throw new Error(`Unexpected package name: ${manifest.name}`);
}
if (manifest.version !== expected) {
throw new Error(`package.json is ${manifest.version}, workflow input is ${expected}`);
}
const prerelease = manifest.version.includes('-');
const npmTag = process.env.NPM_TAG;
if (prerelease !== (npmTag === 'next')) {
throw new Error(prerelease
? 'Prerelease versions must use the next npm tag'
: 'Stable versions must use the latest npm tag');
}
console.log(`Preparing ${manifest.name}@${manifest.version}`);
appendFileSync(process.env.GITHUB_OUTPUT, `version=${manifest.version}\n`);
JS
- name: Install dependencies
working-directory: typescript
run: npm ci
- name: Type-check and test
working-directory: typescript
run: |
npm run format:check
npm run lint
npm run typecheck
npm test
npm run test:package
npx --yes clawhub@0.23.3 package validate . --json
- name: Pack npm tarball
working-directory: typescript
run: |
mkdir -p "${RUNNER_TEMP}/reme-typescript-package"
npm pack --pack-destination "${RUNNER_TEMP}/reme-typescript-package"
- name: Upload npm tarball
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: agentscope-ai-reme-${{ inputs.version }}
path: ${{ runner.temp }}/reme-typescript-package/*.tgz
if-no-files-found: error
publish:
needs: build
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
steps:
- name: Set up Node for npm
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: '24'
registry-url: https://registry.npmjs.org
- name: Download npm tarball
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: agentscope-ai-reme-${{ inputs.version }}
path: dist/typescript
- name: Reject an existing package version
env:
PACKAGE_VERSION: ${{ inputs.version }}
run: |
PACKAGE_VERSION="${PACKAGE_VERSION#v}"
if npm view "@agentscope-ai/reme@${PACKAGE_VERSION}" version >/dev/null 2>&1; then
echo "@agentscope-ai/reme@${PACKAGE_VERSION} already exists" >&2
exit 1
fi
- name: Publish to npm
env:
NPM_TAG: ${{ inputs.npm_tag }}
run: npm publish dist/typescript/*.tgz --access public --tag "${NPM_TAG}" --provenance
publish-clawhub:
if: ${{ inputs.publish_clawhub }}
needs: build
permissions:
actions: read
contents: read
id-token: write
uses: openclaw/clawhub/.github/workflows/package-publish.yml@87ca030c30f3cfb78ab15c8e66b5ff1469c8f9c8 # v0.23.3
with:
owner: agentscope-ai
family: code-plugin
version: ${{ needs.build.outputs.version }}
tags: ${{ inputs.npm_tag }}
source_repo: ${{ github.repository }}
source_commit: ${{ github.sha }}
source_ref: ${{ github.ref }}
source_path: typescript
package_artifact_name: agentscope-ai-reme-${{ inputs.version }}
wait_for_publication: true
secrets:
clawhub_token: ${{ secrets.CLAWHUB_TOKEN }}

View file

@ -1,46 +0,0 @@
name: Security / CodeQL
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 1 * * 1'
workflow_dispatch:
permissions:
actions: read
contents: read
packages: read
security-events: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
analyze:
name: Analyze ${{ matrix.language }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- name: Checkout repository
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
languages: ${{ matrix.language }}
build-mode: none
- name: Perform CodeQL analysis
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
category: /language:${{ matrix.language }}

View file

@ -1,4 +1,4 @@
name: CI / Python tests
name: Tests ReMe
on:
push:
@ -11,9 +11,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
unit-tests:
name: Unit Tests - py${{ matrix.python-version }}
@ -24,12 +21,10 @@ jobs:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
persist-credentials: false
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
@ -37,14 +32,12 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
pip install -e reme_studio -e ".[dev,core]"
pip install --no-deps -e plugins/auto-fin
pip install -e plugins/daily_paper
pip install -e packages/reme_ai_studio -e ".[dev,core]"
pip install coverage
- name: Run unit tests
run: |
coverage run -m pytest tests/unit plugins/auto-fin plugins/daily_paper \
coverage run -m pytest tests/unit \
-v \
--tb=long \
-s \

View file

@ -1,4 +1,4 @@
name: CI / Windows
name: Windows Smoke
on:
push:
@ -11,9 +11,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
cli-smoke:
name: CLI smoke - py${{ matrix.python-version }}
@ -24,12 +21,17 @@ jobs:
python-version: ["3.11"]
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
- uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
persist-credentials: false
node-version: '22'
cache: npm
cache-dependency-path: website/package-lock.json
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
@ -37,10 +39,23 @@ jobs:
- name: Install package
run: |
python -m pip install --upgrade pip setuptools wheel
pip install -e ".[dev,as]"
pip install -e packages/reme_ai_studio -e ".[dev,core]"
- name: Build Studio static workspace
working-directory: website
run: |
npm ci
npm run build:static
- name: Verify editable source installation serves Studio
shell: pwsh
run: |
Push-Location $env:RUNNER_TEMP
python -c "from reme.utils import resolve_web_static_dir; assert (resolve_web_static_dir() / 'index.html').is_file()"
Pop-Location
- name: Run version job
run: reme start config=tests/fixtures/config/version-smoke.yaml job=version
run: reme start service.backend=cli job=version
- name: Run Windows path tests
run: |

6
.gitignore vendored
View file

@ -30,9 +30,11 @@ htmlcov/
# Packaging / build outputs
build/
dist/
node_modules/
*.egg-info/
typescript/reports/
# Website build integration source (not generated output)
!website/build/
!website/build/**
# Logs / temporary files
*.log

View file

@ -41,7 +41,7 @@ and concise documentation together.
- `reme/components/application_context.py`: application-wide wiring and in-memory shared state.
- `reme/components/runtime_context.py`: request-scoped data, response, streaming queue, and stop event.
- `reme/components/base_component.py`: component lifecycle, dependency binding, and workspace helpers.
- `reme/components/component_registry.py`: the frozen built-in registry template and application-local registry factory.
- `reme/components/component_registry.py`: the process-wide `(component type, backend)` registry.
- `reme/components/job/`: base, stream, background, and cron job implementations.
- `reme/components/service/`: local CLI, HTTP, and MCP service backends.
- `reme/components/`: agent wrappers, model adapters, stores, catalogs, graphs, indexes, clients, tokenizers, and
@ -52,15 +52,11 @@ and concise documentation together.
accounting, and wikilink handling.
- `tests/unit/`: primary fast, isolated validation suite.
- `tests/integration/`: service/model tests that may need credentials or external processes.
- `reme_studio/`: ReMe Studio frontend source plus the independently published `reme_studio` Python package and
`@agentscope-ai/reme_studio` npm static distribution.
- `typescript/`: the independently published `@agentscope-ai/reme` package, including the shared TypeScript client and
DeepSeek Harness and OpenClaw adapters.
- `plugins/`: installable ReMe extensions, such as Auto Fin.
- `integrations/`: adapters that connect ReMe to external agent hosts, such as Claude Code, DSH, and Hermes Agent.
- `website/`: ReMe Workspace frontend source; its static build can be served by the HTTP service.
- `plugins/claude_code/` and `plugins/hermes_agent/`: agent integrations.
- `skills/`: standalone skills; `reme_memory` calls ReMe, while other skills may use separate tools or direct-file
conventions.
- `benchmark/` and `cookbook/`: runnable evaluations and example workflows.
- `benchmark/` and `cookbook/`: runnable evaluation and example workflows.
- `docs/`: README-linked supporting pages and figures.
## Development Setup
@ -68,7 +64,7 @@ and concise documentation together.
ReMe requires Python 3.11 or newer. Install the editable development environment with:
```bash
pip install -e reme_studio -e ".[dev,core]"
pip install -e packages/reme_ai_studio -e ".[dev,core]"
```
Before changing behavior, inspect the adjacent implementation, schema, built-in config, and focused tests. Follow
@ -188,7 +184,7 @@ pre-commit run --all-files
```
Black and Flake8 use a 120-character line limit and Python 3.11 formatting; Pylint is also run by pre-commit. If
`reme_studio/` changes, use its Node 22.13+ scripts and run the proportionate checks from that directory, such as
`website/` changes, use its Node 22.13+ scripts and run the proportionate checks from that directory, such as
`npm run format:check`, `npm run lint`, or `npm test`.
Integration tests may contact real model providers, services, or agent subprocesses and can require credentials. Do not
@ -200,14 +196,13 @@ environment has been supplied or authorized. Mock network, model, and subprocess
- Preserve unrelated user changes in a dirty working tree.
- Make the smallest coherent change and avoid unrelated cleanup or broad refactors.
- Do not edit generated output when the source can be changed instead. The publish workflow builds
`reme_studio/dist-static` and stages it under `reme_studio/src/reme_studio/static`; change `reme_studio/` source for
frontend work.
`website/dist-static` and copies it into `reme/web`; change `website/` source for frontend work.
- Do not silently change CLI flags, configuration keys, workspace layouts, serialized schemas, endpoint shapes,
streaming termination, or service interfaces. Preserve compatibility where practical and document intentional
migrations.
- Do not introduce dependencies without a concrete repository-level need.
- Do not commit `.env` files, credentials, runtime memory, logs, indexes, caches, benchmark outputs, or generated
Studio distributions.
website distributions.
- State which validations passed and which relevant checks were not run in the final handoff.
If a requirement is ambiguous, infer intent from nearby code, schemas, defaults, and tests. Ask the user only when the

278
README.md
View file

@ -1,5 +1,5 @@
<p align="center">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p>
<p align="center">
@ -27,35 +27,45 @@
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
## ✨ Why ReMe?
🧠 ReMe turns conversations and resources into readable, editable, searchable, and interconnected Markdown memory. It
works alongside agents such as QwenPaw, OpenClaw, Hermes, and Claude Code, continuously organizing what they learn while
keeping the files under the user's control.
🧠 ReMe turns conversations and resources into readable, editable, searchable, and interconnected Markdown memory. Agents
such as QwenPaw and DeepSeek Harness can share the same workspace to retrieve, maintain, and evolve knowledge, while
users retain control of the durable files.
## ✨ Core Ideas
- **Memory as File, File as Memory**: ReMe stores durable memory as ordinary Markdown with frontmatter and wikilinks.
Users and agents can inspect, edit, move, sync, and back it up with familiar tools, while indexes and generated
metadata remain rebuildable.
- **Self-evolving knowledge base**: ReMe progressively turns conversations and resources into daily notes and long-term
knowledge, preserving sources while refining facts, preferences, procedures, and relationships over time.
- **Recall is precise and context-aware.** BM25, optional embeddings, and wikilink expansion retrieve relevant
line-level passages and their relationships without loading the entire knowledge base into the agent context.
- **One memory workspace works across agents.** Personal assistants, coding agents, and other agent runtimes can share
the same local workspace through native integrations, SKILL.md, CLI, HTTP, MCP, or Python APIs.
- **Memory as File, File as Memory**: Markdown files with frontmatter and wikilinks serve as memory nodes that both
users and agents can inspect, edit, move, and back up directly.
- **Self-evolving knowledge base**: Auto Memory, Auto Resource, and Auto Dream progressively transform conversations and
resources into daily notes and long-term knowledge, while Auto Link writes relationships and sources back into the
files.
- **Progressive hybrid search**: ReMe combines wikilinks, BM25, and embeddings for hybrid retrieval across keyword
matching, optional semantic recall, and relationship expansion without loading every neighboring file into context.
- **Agent-friendly integration**: SKILL.md + CLI integration makes it easy for different agents to read, write,
maintain, and reuse the same local workspace. HTTP, MCP, and Python integrations are also available.
<p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe Design Philosophy" width="92%">
</p>
## 📰 Latest Updates
## 🔭 Use Cases
- **Personal assistants**: Give personal assistants such as
[QwenPaw](https://github.com/agentscope-ai/QwenPaw), [OpenClaw](https://github.com/openclaw/openclaw), and
[Hermes](https://github.com/nousresearch/hermes-agent) a user-editable long-term memory layer.
- **Coding agents**: Preserve coding style, project background, repository decisions, and workflow experience across
sessions when integrating with coding agents such as [Claude Code](plugins/claude_code/reme).
- **LLM Wiki**: Turn conversations, notes, and resources into a searchable, traceable, and linked Markdown knowledge
base that both users and agents can maintain.
- **Self-evolving agents**: Support agents that learn from experience by saving successful paths, failed attempts,
reusable procedures, and periodic reflections as memory.
## 📰 News
- [2026.08] - Published [`@agentscope-ai/reme`](https://www.npmjs.com/package/@agentscope-ai/reme), providing native
ReMe memory integrations for DeepSeek Harness and OpenClaw plus a shared TypeScript HTTP client.
- [2026.08] - Published the [ReMe blog](https://agentscope-ai.github.io/ReMe/?doc=en-reme-blog), an end-to-end introduction to its local-first memory
architecture, self-evolving workflows, hybrid search, proactive discovery, and benchmark results.
- [2026.08] - [Experience-driven enhancement method](https://reme.agentscope.io/?doc=toolmemory-en) of agent tool-use execution built
on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403).
- [2026.07] - Introduced optional plugins: [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) for paper discovery and
- [2026.07] - Introduced optional Cookbooks: [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) for paper discovery and
analysis, and [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-en) for researching the latest 24 hours of topic-related CLS news
with local-memory search and validated historical wikilinks.
- [2026.07] - Our
@ -79,8 +89,8 @@ Install from source:
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e reme_studio -e ".[core]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[core]"
cd website
npm ci
npm run build:static
cd ..
@ -88,6 +98,31 @@ cd ..
The static build requires Node.js 22.13 or newer and makes Studio available from the source tree.
### Environment Variables
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are
disabled by default, so the default setup does not start an embedding model or require an embedding API key.
```bash
cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.
> [!NOTE]
> To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and
> `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change
> `components.file_store.default.embedding_store` from `""` to `default`. See the
> [memory search guide](docs/en/memory_search.md) for details.
### Start the Service
```bash
@ -108,6 +143,12 @@ reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
```
### ReMe Studio (Optional)
The `core` installation above includes Studio. After starting ReMe, open <http://127.0.0.1:2333/> to browse, edit, and
search the workspace. To add Studio to a base installation, use `pip install "reme-ai[web]"`. See the
[ReMe Studio guide](https://reme.agentscope.io/?doc=studio-en) for source builds, configuration, and development.
### 5-Minute Memory Demo
With the service running, write a memory node, let ReMe index it, then retrieve it:
@ -142,81 +183,35 @@ ReMe stores agent memory as readable Markdown.
Related: [[digest/wiki/memory-as-file.md]]
```
### ReMe Studio (Optional)
## 📚 Usage Guides
The `core` installation includes Studio. After starting ReMe, open <http://127.0.0.1:2333/> to browse, edit, and search
the workspace. To add Studio to a base installation, use `pip install "reme-ai[web]"`. See the
[ReMe Studio guide](https://reme.agentscope.io/?doc=studio-en) for source builds, configuration, and development.
These Markdown guides cover the main user workflows and the runtime contracts implemented by the current code.
### Optional Model Configuration
| Guide | What you will learn |
|-------|---------------------|
| [Quick Start](docs/en/quick_start.md) | Install ReMe, start the service, and run the first file and memory operations. |
| [Memory as File](docs/en/memory_as_file.md) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. |
| [Auto Memory](docs/en/auto_memory.md) | Preserve source conversations and distill reusable daily memory cards. |
| [Auto Resource](docs/en/auto_resource.md) | Import supported text resources and turn them into source-linked daily cards. |
| [Auto Dream](docs/en/auto_dream.md) and [Auto Link](docs/en/auto_link.md) | Consolidate daily notes into evolving digest nodes and readable wikilink relationships. |
| [Memory Search](docs/en/memory_search.md) | Use BM25, optional vectors, RRF fusion, line-range recall, and progressive link expansion. |
| [Proactive](docs/en/proactive.md) | Read interest topics safely and integrate them into a host agent's decision flow. |
| [Agent Integration Scenarios](docs/en/reme_scene.md) | Choose among CLI/SKILL.md, HTTP, MCP, and embedded Python integration. |
| [Framework](docs/en/framework.md) | Understand Application, Job, Step, Component, service, configuration, and lifecycle boundaries. |
| [ReMe Blog](https://agentscope-ai.github.io/ReMe/?doc=en-reme-blog) | Read the product story, design rationale, examples, and benchmark summary. |
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are
disabled by default, so the default setup does not start an embedding model or require an embedding API key.
## 🧑‍🍳 Cookbooks
```bash
cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
Cookbooks are optional, end-to-end workflows assembled from ReMe jobs and steps. They are not enabled by the default
configuration; select the cookbook's standalone configuration when starting ReMe. Each new cookbook will be added as
another row in this table.
# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
| Cookbook | Capability |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------|
| [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-en) | Fetch topic-related CLS news, search ReMe history, and generate wikilink-backed Markdown reports. |
Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.
> [!NOTE]
> To enable embedding-based semantic retrieval, uncomment `components.as_embedding` and
> `components.embedding_store` in [`reme/config/default.yaml`](reme/config/default.yaml), then change
> `components.file_store.default.embedding_store` from `""` to `default`. See the
> [memory search guide](docs/en/memory_search.md) for details.
## 🤝 Use ReMe with Your Agent
ReMe can run as a local memory service accessed through the CLI, HTTP API, or MCP server, or it can be embedded in the
host process through its Python API. Host integrations can add memory guidance, recall, and capture to the agent
lifecycle according to the capabilities of each runtime.
| Agent | Recommended path | Available after integration |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **DeepSeek Harness** | Install [`@agentscope-ai/reme`](typescript/README.md#deepseek-harness) with `dsh plugin --profile web add @agentscope-ai/reme`. | Long-term memory guidance, the `reme_search` tool, and automatic capture of completed main-agent turns. |
| **OpenClaw** | Install [`@agentscope-ai/reme`](typescript/README.md#openclaw) with `openclaw plugins install @agentscope-ai/reme`. | Native memory tools, recall before user-triggered runs, and automatic turn capture. |
| **QwenPaw** | Embed ReMe in-process through its Python API. | Reuse the host lifecycle and model config while keeping memory local and file-based. |
| **Claude Code** | Start the streamable HTTP MCP service and install [the ReMe plugin](integrations/claude_code/reme). | MCP recall tools, the `reme-memory` skill, and a Stop hook that records sessions automatically. |
| **Hermes** | Start the HTTP service and install [the ReMe provider](integrations/hermes_agent). | Recall before model calls and asynchronous `auto_memory` after each completed turn. |
| **Codex and other CLI agents** | Install or copy the [ReMe Memory skill](skills/reme_memory/SKILL.md). | Search, read, and write memory through the CLI; automatic capture requires host lifecycle integration. |
<p align="center"><b>Integration demos</b></p>
<table>
<tr>
<td align="center"></td>
<td width="45%" align="center"><b>Auto Memory</b></td>
<td width="45%" align="center"><b>Auto Dream</b></td>
</tr>
<tr>
<td align="center"><b>QwenPaw</b></td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-memory.gif" alt="QwenPaw Auto Memory demo" width="100%">
</td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-dream.gif" alt="QwenPaw Auto Dream demo" width="100%">
</td>
</tr>
<tr>
<td align="center"><b>Claude Code</b></td>
<td width="45%">
<img src="docs/figure/cc-auto-memory.gif" alt="Claude Code Auto Memory demo" width="100%">
</td>
<td width="45%">
<img src="docs/figure/cc-auto-dream.gif" alt="Claude Code Auto Dream demo" width="100%">
</td>
</tr>
</table>
## 🧠 How ReMe Works
## 📁 Memory System
> Memory as File, File as Memory.
@ -224,7 +219,7 @@ ReMe treats **memory as files**, progressively processing filtered conversation
from `session/` and `resource/` into `daily/`, then `digest/`. The default workspace is `.reme/` under the current
directory; `workspace_dir=...` selects a different user-owned location.
### Workspace Layout
### Directory Structure
```text
<workspace_dir>/
@ -260,13 +255,13 @@ directory; `workspace_dir=...` selects a different user-owned location.
<img src="docs/figure/reme-overview.svg" alt="ReMe file-based memory system overview" width="92%">
</p>
### Memory Lifecycle
## 🧭 Memory Design Philosophy
ReMe follows a capture → index → consolidate → recall loop. Workspace files remain the durable source of truth;
everything under `metadata/` is rebuildable.
| Capability | Entry point | What it does | Output |
| ------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
|---------------------------------------------|-------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
| [`auto_memory`](docs/en/auto_memory.md) | Agent hook or `reme auto_memory` | Distills useful conversation facts while preserving a filtered conversation source record. | `session/dialog/*.jsonl`, `daily/<date>/<generated-name>.md` |
| [`auto_resource`](docs/en/auto_resource.md) | Resource watcher or `reme auto_resource` | Turns files under `resource/` into source-linked, content-named daily cards. | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/en/memory_search.md) | Background watcher or `reme reindex` | Live-indexes Markdown in `daily/` and `digest/`; a full rebuild also scans `resource/` and JSONL. | Searchable chunks, BM25, wikilink graph, and optional vectors |
@ -296,18 +291,17 @@ Search returns matching chunks with line ranges and bounded wikilink neighbors.
BM25 through reciprocal rank fusion (RRF).
> [!IMPORTANT]
>
> `proactive` only reads and exposes interest topics produced by Auto Dream. It does not independently browse the web,
> send notifications, or rewrite the knowledge base; the host agent decides whether and how to act on a topic.
## 📊 Benchmarks
## 📊 Performance
ReMe evaluates multi-session and long-context memory with agentic search-and-read workflows. The figures below are the
published reference runs in this repository; model, prompt, dataset, and judging details are documented with each
benchmark.
| Benchmark | Setting | Sample size | Agentic score | Focus |
| --------------------------------------------------------------------------- | ------------ | -----------------------: | ------------: | ------------------------------------------------------------------ |
| Benchmark | Setting | Sample size | Agentic score | Focus |
|--------------------------------------------------------------|--------------|-------------------------:|--------------:|--------------------------------------------------------------------|
| **[LongMemEval cleaned-s](https://reme.agentscope.io/?doc=longmemeval-en)** | **Overall** | **500 questions** | **89.4%** | Cross-session retrieval, knowledge updates, and temporal reasoning |
| [BEAM](https://reme.agentscope.io/?doc=beam-en) | 100K context | 20 cases / 400 questions | 66.1% | Ten types of long-context memory tasks |
| [BEAM](https://reme.agentscope.io/?doc=beam-en) | 1M context | 35 cases / 700 questions | 65.0% | Ultra-long conversation settings |
@ -317,60 +311,68 @@ ReMe also achieved a **0.580 PROC score across five user personas** in the repos
measures proactive handling of hidden intent, clarification, cross-session preferences and conventions, task
dependencies, and underspecified requests.
## 🧩 Extensions and Plugins
## 🤝 Agent-friendly Integration
Plugins are optional Python distributions that contribute Component, Step, or Job backends and configuration. They are
installed separately and enabled explicitly by configuration. Daily Paper and Auto Fin are independently packaged
plugins; see the source distributions and their documentation for [Daily Paper](plugins/daily_paper/README.md) and
[Auto Fin](plugins/auto-fin/README.md).
ReMe can run as a local memory service accessed through the CLI, HTTP API, or MCP server, or it can be embedded in the
host process through its Python API.
| Plugin | Capability |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| [Daily Paper](https://reme.agentscope.io/?doc=daily-paper-en) | Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief. |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-en) | Fetch topic-related CLS news, search ReMe history, and generate wikilink-backed Markdown reports. |
| Agents | Recommended path | Available after integration |
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| **QwenPaw** | Embed ReMe in-process through its Python API. | Reuse the host application's lifecycle and model config while keeping memory local and file-based. |
| **Claude Code** | Start the streamable HTTP MCP service and install [plugins/claude_code/reme](plugins/claude_code/reme). | MCP recall tools, a `reme-memory` skill, and a Stop hook that records sessions automatically. |
| **Hermes** | Start the HTTP service and install [plugins/hermes_agent](plugins/hermes_agent). | Recall relevant memory before model calls and enqueue `auto_memory` after each completed turn. |
| **Other CLI-capable agents (OpenClaw/Codex)** | Copy or install [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md). | Search, read, and write memory via the CLI; automatic recording requires explicit host lifecycle hooks. |
See [Plugin Management](docs/en/plugin_management.md) to install, inspect, validate, enable, and uninstall ReMe plugins.
<p align="center"><b>Integration demos</b></p>
## 📚 Documentation
<table>
<tr>
<td align="center"></td>
<td width="45%" align="center"><b>Auto Memory</b></td>
<td width="45%" align="center"><b>Auto Dream</b></td>
</tr>
<tr>
<td align="center"><b>QwenPaw</b></td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-memory.gif" alt="QwenPaw Auto Memory demo" width="100%">
</td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-dream.gif" alt="QwenPaw Auto Dream demo" width="100%">
</td>
</tr>
<tr>
<td align="center"><b>Claude Code</b></td>
<td width="45%">
<img src="docs/figure/cc-auto-memory.gif" alt="Claude Code Auto Memory demo" width="100%">
</td>
<td width="45%">
<img src="docs/figure/cc-auto-dream.gif" alt="Claude Code Auto Dream demo" width="100%">
</td>
</tr>
</table>
These guides cover the main user workflows and the runtime contracts implemented by the current code.
| Guide | What you will learn |
| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [Quick Start](docs/en/quick_start.md) | Install ReMe, start the service, and run the first file and memory operations. |
| [Memory as File](docs/en/memory_as_file.md) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. |
| [Auto Memory](docs/en/auto_memory.md) | Preserve source conversations and distill reusable daily memory cards. |
| [Auto Resource](docs/en/auto_resource.md) | Import supported text resources and turn them into source-linked daily cards. |
| [Auto Dream](docs/en/auto_dream.md) and [Auto Link](docs/en/auto_link.md) | Consolidate daily notes into evolving digest nodes and readable wikilink relationships. |
| [Memory Search](docs/en/memory_search.md) | Use BM25, optional vectors, RRF fusion, line-range recall, and progressive link expansion. |
| [Proactive](docs/en/proactive.md) | Read interest topics safely and integrate them into a host agent's decision flow. |
| [Application Scenarios](docs/en/reme_scene.md) | Follow concrete financial research, coding-memory, and personal knowledge-base examples. |
| [Framework](docs/en/framework.md) | Understand Application, Job, Step, Component, service, configuration, and lifecycle boundaries. |
| [TypeScript integrations](typescript/README.md) | Configure the shared client and native DeepSeek Harness and OpenClaw adapters. |
| [ReMe Blog](https://agentscope-ai.github.io/ReMe/?doc=en-reme-blog) | Read the product story, design rationale, examples, and benchmark summary. |
## 🛠️ Common Commands
## 🛠️ ReMe Operations
Run `reme help` for the full job list. Common workspace and maintenance commands are:
| Command | Purpose |
| ----------------------------------------- | --------------------------------------------------------------------------------- |
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](docs/en/memory_search.md) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme traverse` / `reme graph_snapshot` | Explore wikilink neighborhoods or the category-rooted digest graph. |
| `reme chat` | Stream a read-only, workspace-aware agent conversation. Requires LLM credentials. |
| `reme reindex` | Rebuild search and wikilink indexes from existing files. |
| Command | Purpose |
|-------------------------------------------|----------------------------------------------------------------------------------------|
| `reme status` | Show stateful data-component memory estimates and process RSS. |
| [`reme search`](docs/en/memory_search.md) | Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
| `reme read` / `reme write` / `reme edit` | Inspect and maintain Markdown memory files. |
| `reme traverse` / `reme graph_snapshot` | Explore wikilink neighborhoods or the category-rooted digest graph. |
| `reme chat` | Stream a read-only, workspace-aware agent conversation. Requires LLM credentials. |
| `reme reindex` | Rebuild search and wikilink indexes from existing files. |
## 🤝 Community and Contributing
## 🤝 Community and Support
- **Issues, requests, and help**: Check [Open Issues](https://github.com/agentscope-ai/ReMe/issues) first. If there is no
related discussion, open one with the background, expected behavior, and impact scope.
- **Code contributions**: Before making changes, read the repository's
[contribution guide](docs/en/contributing.md). Source, schemas, and tests are the authoritative architecture and
extension guide.
- **Documentation contributions**: Update the canonical files under `docs/en/`, `docs/zh/`, or the relevant package
directory in this repository. The documentation site is generated from these files.
- **Code contributions**: Before making changes, read
the [contribution guide](https://docs.agentscope.io/reme/latest/en/contribution). Source, schemas, and tests are the
authoritative architecture and extension guide.
- **Documentation contributions**: Submit user-facing documentation changes to the
[unified documentation repository](https://github.com/agentscope-ai/docs) under `reme/<version>/{en,zh}/`.
- **Commit convention**: Conventional Commits are recommended, for example `feat(search): add link expansion option` or
`docs(zh): update quick start`.
- **Pre-submit checks**: Before submitting a PR, try to run `pre-commit run --all-files` and `pytest`. If tests that

View file

@ -1,5 +1,5 @@
<p align="center">
<img src="https://raw.githubusercontent.com/agentscope-ai/ReMe/main/docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
<img src="docs/figure/reme_logo.png" alt="ReMe Logo" width="50%">
</p>
<p align="center">
@ -27,34 +27,38 @@
> [0.2.x](https://github.com/agentscope-ai/ReMe/tree/v0.2.0.6) ·
> [MemoryScope](https://github.com/agentscope-ai/ReMe/tree/memoryscope_branch)
## ✨ 为什么选择 ReMe
🧠 ReMe 将对话和资料持续沉淀为可读、可编辑、可检索、相互链接的 Markdown 记忆。它可以与 QwenPaw、OpenClaw、Hermes 和 Claude
Code 等 Agent 协作,在持续整理知识的同时,始终把文件控制权留给用户。
🧠 ReMe 将对话和资料持续沉淀为可读、可编辑、可检索、相互链接的 Markdown 记忆。QwenPaw、DeepSeek Harness 等 Agent
可以共享同一个 workspace共同检索、维护和演化知识而持久文件始终由用户掌控。
## ✨ 核心创新
- **Memory as File, File as Memory**ReMe 使用带 frontmatter 和 wikilink 的普通 Markdown 保存持久记忆。用户和 Agent
都可以使用熟悉的工具查看、编辑、移动、同步和备份;索引及生成的元数据均可重建。
- **自进化知识库**ReMe 将对话和资料逐步加工为 daily note 与长期知识,在保留来源的同时,持续提炼事实、偏好、
流程经验及其关系。
- **精准召回所需上下文。** ReMe 结合 BM25、可选 embedding 和 wikilink 展开,召回带行号的相关片段及其关系,无需把整个知识库塞入
Agent 上下文。
- **一个 workspace可供不同 Agent 共同使用。** 个人助理、coding agent 和其他 Agent runtime 可以通过原生集成、SKILL.md、CLI、
HTTP、MCP 或 Python API 共享同一个本地记忆空间。
- **Memory as File, File as Memory**:以带 frontmatter 和 wikilink 的 Markdown 作为记忆节点,用户和 Agent 都能直接查看、编辑、移动和备份。
- **自进化知识库**Auto Memory、Auto Resource 和 Auto Dream 把对话与资料逐步加工为 daily 记忆和长期知识Auto Link
再将关系与来源写回文件。
- **渐进式混合搜索**:融合 wikilink、BM25 和可选 embedding从关键词匹配、语义召回到关系扩展避免一次性将所有邻居全文塞入上下文。
- **Agent 友好集成**:可通过 SKILL.md + CLI 读写和维护同一个本地 workspace也支持 HTTP、MCP 和 Python API 接入。
<p align="center">
<img src="docs/figure/design-philosophy.svg" alt="ReMe 设计理念" width="92%">
</p>
## 📰 最新动态
## 🔭 适用场景
- **Personal assistants**:为 [QwenPaw](https://github.com/agentscope-ai/QwenPaw)、
[OpenClaw](https://github.com/openclaw/openclaw)、[Hermes](https://github.com/nousresearch/hermes-agent)
等个人助理提供用户可编辑的长期记忆层。
- **Coding agents**:在接入 [Claude Code](plugins/claude_code/reme) 等 coding agent 时,跨会话保留代码风格、项目背景、仓库决策和流程经验。
- **LLM Wiki**:把对话、笔记和资料转化为可检索、可追溯、可链接的 Markdown 知识库,由用户和 Agent 共同维护。
- **Self-evolving agents**:帮助 Agent 从经验中学习,把成功路径、失败尝试、可复用流程和阶段性反思沉淀为记忆。
## 📰 新闻
- [2026.08] - 发布 [`@agentscope-ai/reme`](https://www.npmjs.com/package/@agentscope-ai/reme),提供统一 TypeScript HTTP
client以及 DeepSeek Harness 和 OpenClaw 的原生 ReMe 记忆集成。
- [2026.08] - 发布 [ReMe 博客](https://agentscope-ai.github.io/ReMe/?doc=zh-reme-blog),系统介绍本地优先的记忆架构、自进化工作流、混合检索、
主动发现与评测结果。
- [2026.08] - 基于 ReMe 的智能体工具使用
[经验驱动增强方法](https://reme.agentscope.io/?doc=toolmemory-zh)已发布,见
[arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。
- [2026.07] - 新增可选插件:[每日论文](https://reme.agentscope.io/?doc=daily-paper-zh)用于论文发现与解析,
- [2026.07] - 新增可选 Cookbook 工作流[每日论文](https://reme.agentscope.io/?doc=daily-paper-zh)用于论文发现与解析,
[Auto Fin](https://reme.agentscope.io/?doc=auto-fin-zh)用于研究最近 24 小时的主题相关财联社新闻,通过本地记忆搜索回顾历史材料并构建
wikilink。
- [2026.07] -
@ -78,8 +82,8 @@ pip install "reme-ai[core]"
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e reme_studio -e ".[core]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[core]"
cd website
npm ci
npm run build:static
cd ..
@ -87,6 +91,31 @@ cd ..
静态构建要求 Node.js 22.13 或更高版本,并让源码安装可以直接使用 Studio。
### 环境变量
如果需要 LLM 驱动的记忆演化或 embedding 检索可以配置环境变量。embedding 默认关闭,因此默认配置不会启动 embedding 模型,也不需要
embedding API key。
```bash
cat > .env <<'EOF'
# 可选:仅在配置中显式启用 embedding 组件后使用。
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# 必须auto_memory、auto_resource 和 auto_dream 需要 LLM。
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取可以先不配置 LLM 凭证。
> [!NOTE]
> 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中
> `components.as_embedding``components.embedding_store` 的注释,并将
> `components.file_store.default.embedding_store``""` 改为 `default`。完整说明见
> [记忆检索文档](docs/zh/memory_search.md)。
### 启动服务
```bash
@ -107,6 +136,12 @@ reme help
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
```
### ReMe Studio可选
上面的 `core` 安装已包含 Studio。启动 ReMe 后,打开 <http://127.0.0.1:2333/> 即可浏览、编辑和搜索 workspace。
如需为基础安装单独添加 Studio可使用 `pip install "reme-ai[web]"`。源码构建、配置和开发说明见
[ReMe Studio 指南](https://reme.agentscope.io/?doc=studio-zh)。
### 5 分钟记忆 Demo
服务运行后,可以写入一个记忆节点,让 ReMe 索引并检索它:
@ -141,87 +176,41 @@ ReMe 会把 Agent 记忆保存为可读的 Markdown。
相关链接:[[digest/wiki/memory-as-file.md]]
```
### ReMe Studio可选
## 📚 使用指南
上面的 `core` 安装已包含 Studio。启动 ReMe 后,打开 <http://127.0.0.1:2333/> 即可浏览、编辑和搜索 workspace。
如需为基础安装单独添加 Studio可使用 `pip install "reme-ai[web]"`。源码构建、配置和开发说明见
[ReMe Studio 指南](https://reme.agentscope.io/?doc=studio-zh)。
下列 Markdown 文档覆盖主要使用流程,并以当前代码的运行时契约为准。
### 可选模型配置
| 文档 | 主要内容 |
|------|----------|
| [快速开始](docs/zh/quick_start.md) | 安装 ReMe、启动服务并执行首次文件和记忆操作。 |
| [Memory as File](docs/zh/memory_as_file.md) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 |
| [Auto Memory](docs/zh/auto_memory.md) | 保留过滤后的对话来源记录,并提炼可复用的 daily 记忆卡片。 |
| [Auto Resource](docs/zh/auto_resource.md) | 导入支持的文本资料,转换为可追溯来源的 daily 卡片。 |
| [Auto Dream](docs/zh/auto_dream.md) 与 [Auto Link](docs/zh/auto_link.md) | 将 daily 记忆整理为持续演化的 digest 节点和可读 wikilink 关系。 |
| [记忆检索](docs/zh/memory_search.md) | 使用 BM25、可选向量、RRF 融合、行号范围召回和渐进式链接扩展。 |
| [Proactive](docs/zh/proactive.md) | 安全读取兴趣主题,并将其接入宿主 Agent 的决策流程。 |
| [Agent 接入场景](docs/zh/reme_scene.md) | 在 CLI/SKILL.md、HTTP、MCP 和嵌入式 Python 集成之间选择。 |
| [框架说明](docs/zh/framework.md) | 理解 Application、Job、Step、Component、service、配置和生命周期边界。 |
| [ReMe 博客](https://agentscope-ai.github.io/ReMe/?doc=zh-reme-blog) | 了解完整产品故事、设计动机、使用示例和评测摘要。 |
如果需要 LLM 驱动的记忆演化或 embedding 检索可以配置环境变量。embedding 默认关闭,因此默认配置不会启动 embedding 模型,也不需要
embedding API key。
## 🧑‍🍳 Cookbooks
```bash
cat > .env <<'EOF'
# 可选:仅在配置中显式启用 embedding 组件后使用。
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
Cookbook 是由 ReMe jobs 和 steps 组装而成的可选端到端工作流。默认配置不会开启它们;启动 ReMe 时选择对应的独立配置即可启用。后续新增的
cookbook 会继续在表格中按行追加。
# 必须auto_memory、auto_resource 和 auto_dream 需要 LLM。
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
```
| Cookbook | 能力 |
|-----------------------------------------------|--------------------------------------------------------------------------------|
| [每日论文](https://reme.agentscope.io/?doc=daily-paper-zh) | 发现并排序论文,使用 Agent 解读 PDF生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-zh) | 拉取主题相关财联社新闻,搜索 ReMe 历史材料并生成带 wikilink 的 Markdown 报告。 |
基础文件读写、BM25 检索、wikilink 遍历和 proactive topics 读取可以先不配置 LLM 凭证。
> [!NOTE]
> 如需启用基于 embedding 的语义检索,请取消 [`reme/config/default.yaml`](reme/config/default.yaml) 中
> `components.as_embedding``components.embedding_store` 的注释,并将
> `components.file_store.default.embedding_store``""` 改为 `default`。完整说明见
> [记忆检索文档](docs/zh/memory_search.md)。
## 🤝 将 ReMe 接入你的 Agent
ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP server 接入,也可以通过 Python API 嵌入宿主进程。宿主集成可根据不同
runtime 的能力,将记忆指引、召回和捕获接入 Agent 生命周期。
| Agent | 推荐接入方式 | 接入后能力 |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **DeepSeek Harness** | 使用 `dsh plugin --profile web add @agentscope-ai/reme` 安装 [`@agentscope-ai/reme`](typescript/README_ZH.md#deepseek-harness)。 | 长期记忆指引、`reme_search` 工具,以及自动捕获已完成的主 Agent 对话。 |
| **OpenClaw** | 使用 `openclaw plugins install @agentscope-ai/reme` 安装 [`@agentscope-ai/reme`](typescript/README_ZH.md#openclaw)。 | 原生记忆工具、用户触发运行前召回和自动对话捕获。 |
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主生命周期和模型配置,同时保持记忆本地、文件化。 |
| **Claude Code** | 启动 streamable HTTP MCP service并安装 [ReMe 插件](integrations/claude_code/reme)。 | MCP 召回工具、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP service并安装 [ReMe provider](integrations/hermes_agent)。 | 模型调用前召回,每轮对话完成后异步执行 `auto_memory`。 |
| **Codex 及其他 CLI Agent** | 安装或复制 [ReMe Memory skill](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动捕获需要显式接入宿主生命周期。 |
<p align="center"><b>集成演示</b></p>
<table>
<tr>
<td align="center"></td>
<td width="45%" align="center"><b>Auto Memory</b></td>
<td width="45%" align="center"><b>Auto Dream</b></td>
</tr>
<tr>
<td align="center"><b>QwenPaw</b></td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-memory.gif" alt="QwenPaw Auto Memory 演示" width="100%">
</td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-dream.gif" alt="QwenPaw Auto Dream 演示" width="100%">
</td>
</tr>
<tr>
<td align="center"><b>Claude Code</b></td>
<td width="45%">
<img src="docs/figure/cc-auto-memory.gif" alt="Claude Code Auto Memory 演示" width="100%">
</td>
<td width="45%">
<img src="docs/figure/cc-auto-dream.gif" alt="Claude Code Auto Dream 演示" width="100%">
</td>
</tr>
</table>
## 🧠 ReMe 如何工作
## 📁 记忆系统
> Memory as File, File as Memory.
ReMe 将 **记忆视为文件**,让过滤后的对话来源记录和外部资料从 `session/``resource/` 渐进加工到 `daily/`,再沉淀为
`digest/`。默认 workspace 是当前目录下的 `.reme/`;可通过 `workspace_dir=...` 选择其他由用户控制的位置。
### Workspace 结构
### 目录结构
```text
<workspace_dir>/
@ -257,13 +246,13 @@ ReMe 将 **记忆视为文件**,让过滤后的对话来源记录和外部资
<img src="docs/figure/reme-overview.svg" alt="ReMe 文件化记忆系统总览" width="92%">
</p>
### 记忆生命周期
## 🧭 记忆设计理念
ReMe 遵循 capture → index → consolidate → recall 的循环。workspace 文件是持久化的事实来源,`metadata/` 中的内容均可重建。
| 能力 | 入口 | 作用 | 输出 |
| ------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`auto_memory`](docs/zh/auto_memory.md) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留过滤后的对话来源记录。 | `session/dialog/*.jsonl``daily/<date>/<generated-name>.md` |
|---------------------------------------------|-------------------------------------------|----------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| [`auto_memory`](docs/zh/auto_memory.md) | Agent hook 或 `reme auto_memory` | 提炼有长期价值的对话事实,同时保留过滤后的对话来源记录。 | `session/dialog/*.jsonl``daily/<date>/<generated-name>.md` |
| [`auto_resource`](docs/zh/auto_resource.md) | 资源监听或 `reme auto_resource` | 将 `resource/` 下的文件转为带来源链接、按内容命名的 daily 卡片。 | `daily/<date>/<resource-card>.md` |
| [`auto_index`](docs/zh/memory_search.md) | 后台监听或 `reme reindex` | 实时索引 `daily/``digest/` 中的 Markdown全量重建还会扫描 `resource/` 和 JSONL。 | 可检索的 chunks、BM25、wikilink 图谱和可选向量 |
| [`auto_dream`](docs/zh/auto_dream.md) | `dream_cron``reme auto_dream` | 默认从最近两天内变化的文件中最多提取 5 个可复用 unit再创建、印证、补充或修正 digest 节点。 | `digest/**``daily/<date>/interests.yaml` |
@ -291,16 +280,15 @@ ReMe 遵循 capture → index → consolidate → recall 的循环。workspace
搜索返回带行号范围的相关 chunks 和数量受限的 wikilink 邻居;可选向量结果通过 RRF 与 BM25 融合。
> [!IMPORTANT]
>
> `proactive` 只读取并暴露 Auto Dream 生成的兴趣主题,不会自行联网、发送通知或改写知识库;是否以及如何使用主题,由宿主 Agent
> 决定。
决定。
## 📊 评测结果
## 📊 性能表现
ReMe 通过 Agent 多轮搜索与读取的方式评测多会话和超长上下文中的记忆能力。下表为仓库中已公开的参考实验结果模型、prompt、数据集和评判细节见各评测文档。
| 基准 | 设置 | 样本量 | Agentic 得分 | 主要检验内容 |
| --------------------------------------------------------------------------- | ----------- | ----------------: | -----------: | ------------------------------ |
| 基准 | 设置 | 样本量 | Agentic 得分 | 主要检验内容 |
|-----------------------------------------------------------------|-------------|------------------:|-------------:|--------------------------------|
| **[LongMemEval cleaned-s](https://reme.agentscope.io/?doc=longmemeval-zh)** | **整体** | **500 题** | **89.4%** | 跨会话检索、知识更新与时间推理 |
| [BEAM](https://reme.agentscope.io/?doc=beam-zh) | 100K 上下文 | 20 cases / 400 题 | 66.1% | 十类长上下文记忆任务 |
| [BEAM](https://reme.agentscope.io/?doc=beam-zh) | 1M 上下文 | 35 cases / 700 题 | 65.0% | 超长对话设置 |
@ -308,43 +296,52 @@ ReMe 通过 Agent 多轮搜索与读取的方式,评测多会话和超长上
在仓库的 [π-Bench 评测](https://reme.agentscope.io/?doc=pibench-zh)中ReMe Agent 在 5 种用户角色上的平均 **PROC 得分为 0.580**
,比相同测试模型配置的 NanoBot 高 2.4%。PROC 用于评估隐藏意图完成、针对性澄清、跨会话偏好和规范复用、跨任务依赖推断以及欠规格请求推进等主动性能力。
## 🧩 扩展与插件
## 🤝 Agent-friendly Integration
插件是可选的独立 Python distribution可以贡献 Component、Step、Job backend 和配置,并通过配置显式启用。每日论文与 Auto Fin
均已独立打包,源码 distribution 及说明分别见[每日论文](plugins/daily_paper/README_ZH.md)和
[Auto Fin](plugins/auto-fin/README_ZH.md)。
ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP server 接入,也可以通过 Python API 嵌入宿主进程。不同 Agent 可以选择适合自身
runtime 的路径。
| 插件 | 能力 |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [每日论文](https://reme.agentscope.io/?doc=daily-paper-zh) | 发现并排序论文,使用 Agent 解读 PDF生成文件化论文笔记和五分钟简报。 |
| [Auto Fin](https://reme.agentscope.io/?doc=auto-fin-zh) | 拉取主题相关财联社新闻,搜索 ReMe 历史材料并生成带 wikilink 的 Markdown 报告。 |
| Agent | 推荐接入方式 | 接入后能力 |
|-----------------------------------------------|-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主应用的生命周期和模型配置,同时保持 memory 本地、文件化。 |
| **Claude Code** | 启动 streamable HTTP MCP service并安装 [plugins/claude_code/reme](plugins/claude_code/reme)。 | MCP recall tools、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP service并安装 [plugins/hermes_agent](plugins/hermes_agent)。 | 在模型调用前自动召回相关记忆,并在每轮对话完成后异步调用 `auto_memory`。 |
| **Other CLI-capable agents (OpenClaw/Codex)** | 复制或安装 [skills/reme_memory/SKILL.md](skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动记录需要宿主 Agent 显式接入会话生命周期。 |
安装、查看、校验、启用和卸载 ReMe 插件的方法见[插件管理](docs/zh/plugin_management.md)。
<p align="center"><b>集成演示</b></p>
## 📚 文档
<table>
<tr>
<td align="center"></td>
<td width="45%" align="center"><b>Auto Memory</b></td>
<td width="45%" align="center"><b>Auto Dream</b></td>
</tr>
<tr>
<td align="center"><b>QwenPaw</b></td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-memory.gif" alt="QwenPaw Auto Memory 演示" width="100%">
</td>
<td width="45%">
<img src="docs/figure/qwenpaw-auto-dream.gif" alt="QwenPaw Auto Dream 演示" width="100%">
</td>
</tr>
<tr>
<td align="center"><b>Claude Code</b></td>
<td width="45%">
<img src="docs/figure/cc-auto-memory.gif" alt="Claude Code Auto Memory 演示" width="100%">
</td>
<td width="45%">
<img src="docs/figure/cc-auto-dream.gif" alt="Claude Code Auto Dream 演示" width="100%">
</td>
</tr>
</table>
下列文档覆盖主要使用流程,并以当前代码的运行时契约为准。
| 文档 | 主要内容 |
| ------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| [快速开始](docs/zh/quick_start.md) | 安装 ReMe、启动服务并执行首次文件和记忆操作。 |
| [Memory as File](docs/zh/memory_as_file.md) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 |
| [Auto Memory](docs/zh/auto_memory.md) | 保留过滤后的对话来源记录,并提炼可复用的 daily 记忆卡片。 |
| [Auto Resource](docs/zh/auto_resource.md) | 导入支持的文本资料,转换为可追溯来源的 daily 卡片。 |
| [Auto Dream](docs/zh/auto_dream.md) 与 [Auto Link](docs/zh/auto_link.md) | 将 daily 记忆整理为持续演化的 digest 节点和可读 wikilink 关系。 |
| [记忆检索](docs/zh/memory_search.md) | 使用 BM25、可选向量、RRF 融合、行号范围召回和渐进式链接扩展。 |
| [Proactive](docs/zh/proactive.md) | 安全读取兴趣主题,并将其接入宿主 Agent 的决策流程。 |
| [应用场景](docs/zh/reme_scene.md) | 查看金融研究、研发记忆和个人知识库的完整使用示例。 |
| [框架说明](docs/zh/framework.md) | 理解 Application、Job、Step、Component、service、配置和生命周期边界。 |
| [TypeScript 集成](typescript/README_ZH.md) | 配置统一 client以及 DeepSeek Harness 和 OpenClaw 原生适配器。 |
| [ReMe 博客](https://agentscope-ai.github.io/ReMe/?doc=zh-reme-blog) | 了解完整产品故事、设计动机、使用示例和评测摘要。 |
## 🛠️ 常用命令
## 🛠️ ReMe Operations
运行 `reme help` 可查看完整 job 列表。常用 workspace 与维护命令如下:
| 命令 | 作用 |
| ----------------------------------------- | ------------------------------------------------------------- |
|-------------------------------------------|---------------------------------------------------------------|
| `reme status` | 查看有状态数据组件的内存估算及进程 RSS。 |
| [`reme search`](docs/zh/memory_search.md) | 默认使用 BM25 和 wikilink 检索,启用后增加向量检索。 |
| `reme read` / `reme write` / `reme edit` | 检查和维护 Markdown 记忆文件。 |
@ -352,12 +349,13 @@ ReMe 通过 Agent 多轮搜索与读取的方式,评测多会话和超长上
| `reme chat` | 与可感知 workspace 的只读 Agent 进行流式对话;需要 LLM 凭证。 |
| `reme reindex` | 基于已有文件重建检索和 wikilink 索引。 |
## 🤝 社区与贡献
## 🤝 社区与支持
- **问题反馈、需求与帮助**:请先查看 [Open Issues](https://github.com/agentscope-ai/ReMe/issues);如无相关讨论,可新建 Issue
说明背景、目标行为和影响范围。
- **代码贡献**:改动前建议阅读仓库内的[贡献指南](docs/zh/contributing.md)。架构与扩展方式以源码、schema 和测试为准。
- **文档贡献**:请直接更新本仓库 `docs/en/``docs/zh/` 或对应 package 目录中的规范源文件;文档站点会从这些文件生成。
- **代码贡献**:改动前建议阅读 [贡献指南](https://docs.agentscope.io/reme/latest/zh/contribution)。架构与扩展方式以源码、schema
和测试为准。
- **文档贡献**:用户可见文档请提交到[统一文档仓库](https://github.com/agentscope-ai/docs)的 `reme/<version>/{en,zh}/` 目录。
- **提交规范**:建议使用 Conventional Commits例如 `feat(search): add link expansion option`
`docs(zh): update quick start`
- **提交前检查**:提交 PR 前请尽量运行 `pre-commit run --all-files``pytest`;如有依赖 LLM、embedding 或外部服务的测试无法运行,请在

View file

@ -1,65 +1,36 @@
# Auto Fin Plugin
# Auto Fin Cookbook
[中文](README_ZH.md)
Auto Fin fetches a rolling window of CLS telegraph news (24 hours by default), selects items related to configured
topics, searches ReMe for useful historical context, and writes one Chinese Markdown report with validated wikilinks.
Current news and topic selection stay in runtime memory; only the final report becomes durable memory. This directory
is an independent Python distribution. Its single `reme.plugins` entry point exposes a `plugin.yaml` containing the
three Step backends and their Job configuration under `application_defaults`. Enable the installed plugin explicitly
through `plugins=["auto-fin"]`.
Current news and topic selection stay in runtime memory; only the final report becomes durable memory. The
implementation lives in [`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/) and is assembled by
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml).
> Auto Fin has no reliable market-price feed. It does not calculate returns, targets, or entry points and is not
> investment advice.
## Quick start
### 1. Install ReMe and Auto Fin
```bash
python -m pip install "reme-ai[core]>=0.4.1.9"
reme plugins install reme-auto-fin
python -m pip install -e ".[core]"
export LLM_API_KEY="your-api-key"
export LLM_MODEL_NAME="qwen3.7-plus"
export LLM_BASE_URL="https://your-provider.example/v1"
reme start config=daily_cookbook job=auto_fin
```
### 2. Configure the model environment
`LLM_MODEL_NAME` defaults to `qwen3.7-plus`. There is no built-in `LLM_BASE_URL`, so set the OpenAI-compatible endpoint
required by the selected provider.
Configure the LLM environment variables as described in the
[ReMe model-configuration guide](../../README.md#optional-model-configuration). Other compatible models and providers
can also be used.
### 3. Start ReMe with the plugin
The default topics are `黄金,机器人,半导体`. Override them per run:
```bash
reme start plugins='["auto-fin"]'
reme start config=daily_cookbook job=auto_fin topics="黄金,AI,存储芯片"
```
With no explicit `config`, ReMe loads `default.yaml` and adds the plugin to that service.
From another terminal, call the running HTTP service through ReMe's CLI client:
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
Or call its HTTP endpoint directly:
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
The HTTP service also exposes the same Job as the `auto_fin` MCP tool at `/mcp`. The default topics are
`黄金,机器人,半导体`; an empty value also uses these defaults.
To host the application with both JSON and MCP access:
```bash
reme start plugins='["auto-fin"]' \
service.backend=http
```
Custom application configs must provide `agent_wrapper.default` and the `search` and `read` Jobs used by Auto Fin.
An empty value also uses the defaults.
## Pipeline
@ -70,7 +41,7 @@ normalize and deduplicate in RuntimeContext
topic Agent selects real news IDs in bounded batches
research Agent uses search + read on historical memory
research Agent uses memory_search + read on historical memory
validate historical wikilinks in code
@ -85,8 +56,8 @@ records outside the window are discarded.
IDs and deduplicates repeated IDs, then preserves the source-news order. If nothing is relevant, the job succeeds as a
skip without writing or sending a report.
`auto_fin_merge_step` receives only selected current news. It exposes `search` and `read`, and keeps current CLS IDs,
times, and titles as plain evidence. The prompt limits
`auto_fin_merge_step` receives only selected current news. It exposes `memory_search` and `read`, instructs the Agent to
search no later than yesterday, and keeps current CLS IDs, times, and titles as plain evidence. The prompt limits
wikilinks to historical Markdown actually used by the Agent; the code-level boundary independently keeps only existing,
workspace-relative Markdown targets. Missing, absolute, escaping, backslash, and self-referential targets are degraded
to their readable aliases.
@ -105,22 +76,23 @@ refreshes the daily index. No JSONL, intermediate Markdown, or structured Agent
| `request_interval` | `10` | Minimum delay in seconds after every CLS request attempt; may be zero |
| `max_retries` | `3` | Maximum attempts for each CLS page request; must be at least one |
The plugin cron Job starts with the application and runs daily at 18:00 in the application timezone.
The built-in schedules run daily at 09:30, 11:30, and 18:00 in `Asia/Shanghai`.
## Output
```text
.reme/daily/YYYY-MM-DD/auto_fin.md
reme_workspace/daily/YYYY-MM-DD/auto_fin.md
```
The report includes a title, description, current CLS evidence, historical analysis, contextual wikilinks, and a fixed
non-investment disclaimer. Network errors and invalid Agent output fail explicitly; no relevant current news is a
successful skip.
successful skip. If `DINGTALK_CONVERSATION_IDS` is empty, delivery is a no-op. If it is set, the DingTalk credentials
described in the [Daily Paper cookbook](../daily_paper/README.md#6-dingtalk) are required.
## Validation
```bash
python -m pytest plugins/auto-fin -v
pytest tests/unit/test_auto_fin.py -v
```
Unit tests mock the CLS and Agent boundaries and do not contact external services.

View file

@ -1,60 +1,33 @@
# Auto Fin 插件
# Auto Fin Cookbook
[English](README.md)
Auto Fin 自动拉取一个滚动时间窗口内的财联社电报(默认 24 小时),按配置 topics 筛选相关新闻,搜索 ReMe 中有回顾价值的历史材料,最后写入一份带校验
wikilink 的中文 Markdown 报告。当前新闻和筛选结果只存在于本次运行内存中,只有最终报告成为持久记忆。本目录是一个独立 Python
distribution单个 `reme.plugins` entry point 暴露 `plugin.yaml`,其中声明三个 Step backend并在
`application_defaults` 下提供 Job 配置;通过 `plugins=["auto-fin"]` 显式启用这个已安装插件
wikilink 的中文 Markdown 报告。当前新闻和筛选结果只存在于本次运行内存中,只有最终报告成为持久记忆。实现位于
[`reme/steps/cookbook/auto_fin/`](../../reme/steps/cookbook/auto_fin/),并由
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配
> Auto Fin 没有可靠行情数据,不计算收益、目标价或买卖点,也不提供投资建议。
## 快速开始
### 1. 安装 ReMe 和 Auto Fin
```bash
python -m pip install "reme-ai[core]>=0.4.1.9"
reme plugins install reme-auto-fin
python -m pip install -e ".[core]"
export LLM_API_KEY="your-api-key"
export LLM_MODEL_NAME="qwen3.7-plus"
export LLM_BASE_URL="https://your-provider.example/v1"
reme start config=daily_cookbook job=auto_fin
```
### 2. 配置模型环境变量
`LLM_MODEL_NAME` 默认是 `qwen3.7-plus`。代码没有内置 `LLM_BASE_URL`,请设置所选服务商提供的 OpenAI 兼容 endpoint。
按照 ReMe README 的[可选模型配置说明](../../README_ZH.md#可选模型配置)配置 LLM 环境变量,也可以使用其他兼容的模型和服务商。
### 3. 带插件启动 ReMe
默认 topics 是 `黄金,机器人,半导体`。可在运行时覆盖:
```bash
reme start plugins='["auto-fin"]'
reme start config=daily_cookbook job=auto_fin topics="黄金,AI,存储芯片"
```
未显式传入 `config`ReMe 会加载 `default.yaml`,并将插件叠加到该服务上。
在另一个终端中,通过 ReMe CLI client 调用正在运行的 HTTP 服务:
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
也可以直接调用 HTTP endpoint
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
HTTP service 也会在 `/mcp` 中将同一个 Job 暴露为 `auto_fin` MCP tool。默认 topics 是 `黄金,机器人,半导体`
传入空值也会使用默认值。
如果需要同时通过 JSON 和 MCP 访问同一个应用:
```bash
reme start plugins='["auto-fin"]' \
service.backend=http
```
自定义应用配置需要提供 `agent_wrapper.default`,以及 Auto Fin 使用的 `search``read` Jobs。
传入空值也会使用默认 topics。
## 流程
@ -65,7 +38,7 @@ reme start plugins='["auto-fin"]' \
Topic Agent 分批选择真实 news_id
Research Agent 使用 search + read 检索历史记忆
Research Agent 使用 memory_search + read 检索历史记忆
代码校验历史 wikilink
@ -78,8 +51,8 @@ daily/YYYY-MM-DD/auto_fin.md
`auto_fin_topic_step` 分批接收当前新闻,只返回相关的 `news_id`。代码会忽略未知 ID、去除重复 ID并保持源新闻顺序。如果没有相关新闻Job
会成功跳过,不写报告也不发送通知。
`auto_fin_merge_step` 只接收筛选后的当前新闻,并向 Agent 开放 `search` 和 `read`。当前新闻以 CLS ID、时间和标题作为普通证据。
Prompt 要求 Agent 只链接实际使用过的历史 Markdown代码边界则独立保证只保留真实存在、相对
`auto_fin_merge_step` 只接收筛选后的当前新闻,并向 Agent 开放 `memory_search` 和 `read`历史检索截止到昨天;当前新闻以
CLS ID、时间和标题作为普通证据。Prompt 要求 Agent 只链接实际使用过的历史 Markdown代码边界则独立保证只保留真实存在、相对
workspace 的 Markdown 目标。不存在、绝对路径、越界、带反斜杠和自引用的目标都会降级为可读 alias。
同日重跑会参考当天已有报告并覆盖为修订结果。最终写入使用原子替换并刷新当天索引;流程不会写入 JSONL、中间 Markdown 或 Agent
@ -96,21 +69,22 @@ workspace 的 Markdown 目标。不存在、绝对路径、越界、带反斜杠
| `request_interval` | `10` | 每次财联社请求尝试后的最小等待秒数,可设为 0 |
| `max_retries` | `3` | 每页财联社请求的最大尝试次数,至少为 1 |
插件的 cron Job 随应用启动,并按应用配置的时区在每天 18:00 运行。
内置定时任务每天按 `Asia/Shanghai` 在 09:30、11:30 和 18:00 运行。
## 产物
```text
.reme/daily/YYYY-MM-DD/auto_fin.md
reme_workspace/daily/YYYY-MM-DD/auto_fin.md
```
报告包含标题、说明、当前 CLS 证据、历史分析、上下文 wikilink 和固定非投资建议声明。网络错误与无效 Agent 输出
会明确失败;没有相关当前新闻则成功跳过。
会明确失败;没有相关当前新闻则成功跳过。`DINGTALK_CONVERSATION_IDS` 为空时发送步骤无副作用;设置该变量后,
还必须提供[每日论文 Cookbook](../daily_paper/README_ZH.md#6-dingtalk)中列出的钉钉凭据。
## 验证
```bash
python -m pytest plugins/auto-fin -v
pytest tests/unit/test_auto_fin.py -v
```
单元测试 mock CLS 与 Agent 边界,不访问外部服务。

View file

@ -0,0 +1,262 @@
# Daily Paper Cookbook
[中文](README_ZH.md)
Daily Paper selects three papers from the Hugging Face Papers weekly and monthly rankings, downloads their arXiv PDFs,
and produces detailed Chinese reading notes plus a roughly five-minute Chinese brief. The implementation lives in
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/) and is assembled by
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml).
## Quick start
The workflow requires Python 3.11 or later, the `core` dependencies, an available AgentScope LLM, and network access to
Hugging Face Papers and arXiv.
```bash
python -m pip install -e ".[core]"
export LLM_API_KEY="your-api-key"
export LLM_MODEL_NAME="qwen3.7-plus"
export LLM_BASE_URL="https://your-provider.example/v1"
reme start config=daily_cookbook job=daily_paper
```
The built-in LLM component defaults to:
- model: `qwen3.7-plus`
- endpoint: no built-in `LLM_BASE_URL`; set the OpenAI-compatible endpoint required by your provider
- environment variables: `LLM_API_KEY`, `LLM_MODEL_NAME`, and `LLM_BASE_URL`
Auto Fin and Daily Paper share this single `default` LLM and the `default` AgentScope wrapper. Daily Paper Select and
Analyze call the wrapper without tools, while Daily Paper Digest and Auto Fin Merge receive the read-only
`memory_search` and `read` ReMe job tools. The interactive `dingtalk_wait` step separately overrides the wrapper per
call with AgentScope `bash` and an explicit ReMe job allowlist.
The default workspace is `reme_workspace/` beneath the process working directory. Override it with
`DAILY_PAPER_WORKSPACE_DIR`.
## Pipeline
```text
Hugging Face weekly/monthly rankings
Collect ──► Rank ──► Select 3 ──► Analyze PDFs ──► Digest ──► DingTalk (optional)
│ │
├─ PDFs ├─ daily brief
└─ paper notes └─ day index
```
### 1. Collect
`daily_paper_collect_step` concurrently fetches:
- the Hugging Face weekly ranking for the run date's ISO week;
- the monthly ranking for the run date's calendar month; and
- Hugging Face Daily Papers for exactly the previous calendar day.
The weekly and monthly results are merged by arXiv ID while preserving both ranks. The step then excludes papers found
in yesterday's list or in the `arxiv_id` frontmatter of `daily/<date>/*.md` within the previous `history_days`.
If a Markdown file with `kind: daily-paper-brief` already exists and `force=false`, generation is skipped; the saved
brief can still proceed to DingTalk delivery. The job fails when no eligible papers remain.
### 2. Rank
`daily_paper_rank_step` uses reciprocal-rank fusion:
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
A missing rank contributes zero. Papers are ordered by fused score, upvotes, and arXiv ID. The pool is capped at
`candidate_limit`, and Rank applies no topic preference.
### 3. Select
`daily_paper_select_step` sends candidate metadata to a tool-free AgentScope agent and requires exactly three items:
```json
{"papers": [{"arxiv_id": "2601.01234", "reasoning": "A specific, verifiable reason"}]}
```
All IDs must be unique and belong to the candidate pool, and every reason must be non-empty. A validation failure is
returned to the agent for one retry. Only a non-empty `topics` value injects a personalized subject preference into the
selection prompt; it does not change the fixed count of three papers.
### 4. Analyze
`daily_paper_analyze_step` processes the selected papers in order:
1. validates a modern `YYYY.NNNN` or `YYYY.NNNNN` arXiv ID;
2. downloads the PDF to `resource/papers/<arxiv-id>.pdf`;
3. reuses an existing target whose header is `%PDF-`;
4. extracts paginated text with `pypdf`, bounded by `max_pdf_pages` and `max_pdf_chars`;
5. sends metadata, selection reasoning, and PDF text to a tool-free agent; and
6. writes a Chinese note to `daily/<date>/<Chinese-title>.md`.
Downloads use a temporary file and atomically replace the target only after validating the PDF header. They are also
bounded by `max_pdf_bytes`. There is no OCR fallback, so scanned or textless PDFs fail. When extraction is truncated,
the note records `pdf_text_truncated: true` in its frontmatter.
### 5. Digest
`daily_paper_digest_step` uses the three in-memory analyses as the factual source for the Chinese brief. It also
searches and, when needed, reads earlier daily notes to identify related coverage; those notes may only support
contextual wikilinks, not add facts about the current papers. The agent returns `title`, `desc`, and `body`. The code
then:
- strips model-generated YAML frontmatter if present;
- normalizes the Chinese title for use as a filename;
- keeps model-generated wikilinks only when they point to existing `daily/` Markdown files dated before the run date;
- deterministically appends wikilinks to all three source notes;
- writes `daily/<date>/<Chinese-brief-title>.md`; and
- rebuilds the `daily/<date>.md` day index.
Final response metadata includes the date, week/month scopes, selected arXiv IDs, selection reasons, note/PDF/brief
paths, source counts, and exclusion counts.
### 6. DingTalk
The final `dingtalk_markdown_send_step` is optional. With no conversation IDs it is a no-op. When configured, it strips
frontmatter and sends the brief body to each group in order:
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
A failed recipient does not prevent later attempts; the step reports a combined failure after trying every group.
## Outputs
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <Chinese-paper-title>.md # three, kind: daily-paper-analysis
│ └── <Chinese-brief-title>.md # one, kind: daily-paper-brief
└── resource/
└── papers/
└── <arxiv-id>.pdf
```
Each successful generation writes three analysis notes and one brief. A forced rerun can leave unrelated or previously
selected analysis notes in the same day directory; ReMe does not delete them as cleanup. Filenames come from the agent's
Chinese titles. The implementation removes unsafe path characters and resolves title collisions. Markdown and PDF
outputs are written through same-directory temporary files and atomic replacement.
## Parameters and defaults
Public job parameters:
| Parameter | Default | Purpose |
|-----------------|--------:|-----------------------------------------------------------------------------------------|
| `date` | `""` | Run date; empty uses today in the app timezone, otherwise requires `YYYY-MM-DD` |
| `force` | `false` | Regenerate even when the day's brief exists |
| `use_hf_mirror` | `false` | Use the Hugging Face mirror from `HF_MIRROR_URL`, or `https://hf-mirror.com` when unset |
| `topics` | `""` | Topics to prioritize during selection |
| `weekly_weight` | `0.7` | Weekly contribution to RRF |
| `history_days` | `30` | Prior recommendation exclusion window |
Step-level settings on the `daily_paper` job:
| Setting | Default | Purpose |
|-------------------|--------------:|----------------------------------------------------|
| `candidate_limit` | `20` | Maximum candidates sent to Select |
| `rrf_k` | `60` | RRF constant |
| `hf_timeout` | `600` seconds | Timeout for one Hugging Face request |
| `hf_max_retries` | `3` | Maximum Hugging Face attempts |
| `pdf_timeout` | `600` seconds | arXiv PDF download timeout |
| `max_pdf_bytes` | `52428800` | PDF limit, 50 MiB |
| `max_pdf_pages` | `35` | Maximum extracted pages |
| `max_pdf_chars` | `300000` | Maximum extracted PDF characters sent to the agent |
## Mirrors
The data clients use httpx's default environment handling, so `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` take effect
when present. The two data sources reach a mirror differently: Hugging Face is gated on the `use_hf_mirror` job
parameter, while arXiv is driven by its environment variable alone.
```dotenv
# The built-in daily_paper_cron job enables the mirror by default; set false to use the official service
DAILY_PAPER_USE_HF_MIRROR=false
# Read only when the manual or scheduled job enables the mirror; defaults to https://hf-mirror.com when unset
HF_MIRROR_URL=https://hf-mirror.com
# Optional override; the code defaults to https://arxiv.org when unset
ARXIV_MIRROR_URL=https://export.arxiv.org
# Path-prefixed relay URLs are also supported
# HF_MIRROR_URL=http://relay-host:18080/hf
# ARXIV_MIRROR_URL=http://relay-host:18080/arxiv
```
`HF_MIRROR_URL` must implement the `/papers/...`, `/api/daily_papers`, and `/api/papers/...` routes used by the current
client. `ARXIV_MIRROR_URL` must implement `/pdf/<arxiv-id>`. A path prefix in either base URL is preserved, and a
trailing slash is optional. There is no fallback chain: whichever base URL a client selects is the only one it tries.
> **Behavior change:** `HF_MIRROR_URL` used to redirect Hugging Face traffic on its own. It is now read only when the
> job runs with `use_hf_mirror=true`; otherwise the official service is used and the client logs a warning that the
> variable was ignored. Pass `use_hf_mirror=true` for manual requests. The built-in `daily_paper_cron` job enables the
> mirror by default; set `DAILY_PAPER_USE_HF_MIRROR=false` to make that scheduled job use the official service.
## Running the workflow
Generate a brief for a specific date:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-08-06 \
topics="Agent memory" \
history_days=30
```
Force a rerun; valid local PDFs are still reused:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-08-06 force=true
```
Start the HTTP service and scheduled jobs:
```bash
reme start config=daily_cookbook
```
The built-in service listens on `127.0.0.1:8001`. `daily_paper_cron` runs every day at 08:00 in the
`Asia/Shanghai` timezone, prioritizes the topic `大模型长期记忆`, and uses the Hugging Face mirror by default. Set
`DAILY_PAPER_USE_HF_MIRROR=false` to use the official service. Override the bind address with `DAILY_PAPER_HOST`,
`DAILY_PAPER_PORT`, or startup arguments.
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-08-06","force":false,"topics":"Agent memory"}'
```
## Failures and reruns
- Hugging Face HTTP failures use exponential backoff up to `hf_max_retries` attempts; invalid response payloads fail
immediately.
- Fewer than three candidates, invalid agent selection, invalid/oversized/textless PDFs, or empty agent output stop the
job.
- Papers are analyzed sequentially; PDFs and notes completed before a failure remain on disk.
- `force=true` regenerates the selected notes and the brief while reusing valid PDFs; it does not remove other notes
already present in that day's directory.
- The multi-file workflow is not transactional and has no global per-date execution lock.
## Tests
The focused unit tests mock Hugging Face, arXiv, AgentScope, and DingTalk boundaries and do not call real services:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```

View file

@ -0,0 +1,248 @@
# 每日论文 Cookbook
[English](README.md)
每日论文工作流从 Hugging Face Papers 的周榜和月榜中筛选三篇论文,下载 arXiv PDF生成中文论文解读和一篇约五分钟可读完的中文简报。当前实现位于
[`reme/steps/cookbook/daily_paper/`](../../reme/steps/cookbook/daily_paper/),由
[`daily_cookbook.yaml`](../../reme/config/daily_cookbook.yaml) 装配。
## 快速开始
要求 Python 3.11 或更高版本、`core` 依赖、可用的 AgentScope LLM以及能访问 Hugging Face Papers 和 arXiv 的网络。
```bash
python -m pip install -e ".[core]"
export LLM_API_KEY="your-api-key"
export LLM_MODEL_NAME="qwen3.7-plus"
export LLM_BASE_URL="https://your-provider.example/v1"
reme start config=daily_cookbook job=daily_paper
```
内置 LLM 组件默认配置为:
- 模型:`qwen3.7-plus`
- endpoint无内置 `LLM_BASE_URL`;请设置服务商要求的 OpenAI 兼容 endpoint
- 环境变量:`LLM_API_KEY``LLM_MODEL_NAME``LLM_BASE_URL`
Auto Fin 和 Daily Paper 共用这一个 `default` LLM 和 `default` AgentScope wrapper。Daily Paper 的 Select 和 Analyze
调用不带工具Daily Paper Digest 与 Auto Fin Merge 使用只读的 ReMe Job 工具 `memory_search``read`。交互式
`dingtalk_wait` Step 则会在调用时单独覆盖 wrapper启用 AgentScope `bash` 和明确的 ReMe Job allowlist。
默认 workspace 是启动目录下的 `reme_workspace/`,可通过 `DAILY_PAPER_WORKSPACE_DIR` 覆盖。
## 工作流
```text
Hugging Face 周榜/月榜
Collect ──► Rank ──► Select 3 篇 ──► Analyze PDF ──► Digest ──► DingTalk可选
│ │
├─ PDF ├─ 每日简报
└─ 论文解读 └─ 当日索引
```
### 1. Collect
`daily_paper_collect_step` 根据运行日期并发读取:
- 该日期所在 ISO week 的 Hugging Face 周榜;
- 该日期所在自然月的 Hugging Face 月榜;
- 严格前一个自然日的 Hugging Face Daily Papers。
周榜和月榜按 arXiv ID 合并,并保留各自排名。随后排除:
- 昨日 Daily Papers 中的论文;
- `history_days` 窗口内,已出现在 `daily/<date>/*.md` frontmatter `arxiv_id` 中的论文。
如果当天已经存在 `kind: daily-paper-brief` 的 Markdown 且 `force=false`整个生成流程会跳过已有简报仍可进入钉钉发送步骤。没有剩余候选论文时Job
直接失败。
### 2. Rank
`daily_paper_rank_step` 使用 reciprocal-rank fusion
```text
score = 1 / (rrf_k + monthly_rank)
+ weekly_weight / (rrf_k + weekly_rank)
```
缺失的榜单排名贡献为零。论文按融合分、upvotes、arXiv ID 排序,候选池最多保留 `candidate_limit` 篇。Rank 阶段不应用任何主题倾向。
### 3. Select
`daily_paper_select_step` 将候选元数据交给无工具的 AgentScope Agent并要求返回恰好三项
```json
{"papers": [{"arxiv_id": "2601.01234", "reasoning": "具体且可核验的选择理由"}]}
```
三个 ID 必须唯一且都属于候选池,理由不能为空。校验失败后,错误信息会反馈给 Agent 并重试一次。只有非空 `topics`
会向精选提示注入个性化主题,且不会改变固定的三篇数量。
### 4. Analyze
`daily_paper_analyze_step` 按精选顺序逐篇处理:
1. 校验新版 arXiv ID 格式 `YYYY.NNNN``YYYY.NNNNN`
2. 下载 PDF 到 `resource/papers/<arxiv-id>.pdf`
3. 如果目标文件已存在且以 `%PDF-` 开头,直接复用;
4. 用 `pypdf` 提取分页文本,受 `max_pdf_pages``max_pdf_chars` 限制;
5. 将论文元数据、选择理由和 PDF 文本交给无工具 Agent
6. 将中文解读写入 `daily/<date>/<中文标题>.md`
下载采用临时文件并在校验 PDF 文件头后原子替换,同时限制 `max_pdf_bytes`。当前没有 OCR扫描版或无文本层 PDF 会失败。提取被截断时,笔记
frontmatter 中的 `pdf_text_truncated` 会记录为 `true`
### 5. Digest
`daily_paper_digest_step` 以内存中的三篇解读作为本期事实来源生成中文简报,同时搜索并按需读取较早的 daily
文章来识别相关报道;历史文章只能用于建立上下文 wikilink不能用于补充本期论文事实。输出必须包含 `title``desc``body`
。代码会:
- 去掉模型可能生成的 YAML frontmatter
- 规范化中文标题并用作文件名;
- 只保留指向真实存在、日期早于运行日期的 `daily/` Markdown 文件的模型生成 wikilink
- 确定性追加三篇源笔记的 wikilink
- 写入 `daily/<date>/<中文简报标题>.md`
- 重建 `daily/<date>.md` 当日索引。
最终响应 metadata 包含日期、周/月范围、入选 arXiv ID、选择理由、笔记/PDF/简报路径、源榜单数量和排重数量。
### 6. DingTalk
最后的 `dingtalk_markdown_send_step` 是可选步骤。未设置群会话 ID 时无副作用跳过;配置后会去掉 frontmatter并把简报正文依次发送给所有群
```dotenv
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
任一群发送失败不会阻止继续尝试后续群,全部尝试结束后统一报告失败。
## 产物
```text
reme_workspace/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <中文论文标题>.md # 三篇kind: daily-paper-analysis
│ └── <中文简报标题>.md # 一篇kind: daily-paper-brief
└── resource/
└── papers/
└── <arxiv-id>.pdf
```
每次成功生成会写入三篇论文解读和一篇简报。强制重跑后当日目录中可能保留其他内容或此前入选论文的解读ReMe
不会把它们作为清理对象删除。文件名来自 Agent 返回的中文标题。代码会清理路径不安全字符并处理同名文件。Markdown 和 PDF
都通过同目录临时文件写入后原子替换。
## 参数与默认值
可在调用时传入的 Job 参数:
| 参数 | 默认值 | 作用 |
|-----------------|--------:|----------------------------------------------------------------------------------------------|
| `date` | `""` | 运行日期;空值使用应用时区当天,非空值必须为 `YYYY-MM-DD` |
| `force` | `false` | 已有当日简报时仍重新生成 |
| `use_hf_mirror` | `false` | 是否使用 Hugging Face 镜像站;优先读取 `HF_MIRROR_URL`,未配置时使用 `https://hf-mirror.com` |
| `topics` | `""` | 精选论文时优先考虑的主题 |
| `weekly_weight` | `0.7` | RRF 中周榜权重 |
| `history_days` | `30` | 历史推荐排重窗口 |
`daily_paper` Job 的步骤级配置:
| 配置 | 默认值 | 作用 |
|-------------------|-----------:|------------------------------|
| `candidate_limit` | `20` | 送入 Select 的最大候选数 |
| `rrf_k` | `60` | RRF 常数 |
| `hf_timeout` | `600` 秒 | Hugging Face 单次请求超时 |
| `hf_max_retries` | `3` | Hugging Face 最大尝试次数 |
| `pdf_timeout` | `600` 秒 | arXiv PDF 下载超时 |
| `max_pdf_bytes` | `52428800` | PDF 上限50 MiB |
| `max_pdf_pages` | `35` | 最多提取页数 |
| `max_pdf_chars` | `300000` | 最多送入 Agent 的 PDF 字符数 |
## 镜像站
数据客户端使用 httpx 默认的环境处理,因此存在 `HTTP_PROXY``HTTPS_PROXY``NO_PROXY` 时会自动生效。两个数据源启用镜像的方式不同Hugging
Face 由 `use_hf_mirror` 任务参数控制arXiv 仅由环境变量驱动。
```dotenv
# 内置 daily_paper_cron 定时任务默认启用镜像站;设为 false 可改用官方服务
DAILY_PAPER_USE_HF_MIRROR=false
# 仅在手动任务或定时任务启用镜像时读取;未配置时使用 https://hf-mirror.com
HF_MIRROR_URL=https://hf-mirror.com
# 可选覆盖;未设置时代码使用 https://arxiv.org
ARXIV_MIRROR_URL=https://export.arxiv.org
# 也支持带路径前缀的中转地址
# HF_MIRROR_URL=http://relay-host:18080/hf
# ARXIV_MIRROR_URL=http://relay-host:18080/arxiv
```
`HF_MIRROR_URL` 必须提供当前代码使用的 `/papers/...``/api/daily_papers``/api/papers/...` 路径。`ARXIV_MIRROR_URL`
必须支持 `/pdf/<arxiv-id>`。两种 base URL 都会保留路径前缀,末尾 `/` 可有可无。不存在备用地址回退:客户端选定哪个 base
URL就只访问该地址。
> **行为变更:** 以往只要设置 `HF_MIRROR_URL` 就会改变 Hugging Face
> 的访问地址;现在该变量仅在任务启用镜像时才会读取,否则直接访问官方站点,并输出一条“已忽略该变量”的告警日志。手动调用需传入
> `use_hf_mirror=true`。内置 `daily_paper_cron` 定时任务默认启用镜像;设置 `DAILY_PAPER_USE_HF_MIRROR=false`
> 可让该定时任务改用官方服务。
## 运行方式
生成指定日期的简报:
```bash
reme start \
config=daily_cookbook \
job=daily_paper \
date=2026-08-06 \
topics="Agent memory" \
history_days=30
```
强制重跑;有效的本地 PDF 仍会复用:
```bash
reme start config=daily_cookbook job=daily_paper date=2026-08-06 force=true
```
启动 HTTP 服务和定时任务:
```bash
reme start config=daily_cookbook
```
内置服务监听 `127.0.0.1:8001``daily_paper_cron``Asia/Shanghai` 时区每天 08:00 运行,默认优先关注
`大模型长期记忆`,并使用 Hugging Face 镜像站。设置 `DAILY_PAPER_USE_HF_MIRROR=false` 可改用官方服务。可通过
`DAILY_PAPER_HOST``DAILY_PAPER_PORT` 或启动参数覆盖监听地址和端口。
```bash
curl -s http://127.0.0.1:8001/daily_paper \
-H 'Content-Type: application/json' \
-d '{"date":"2026-08-06","force":false,"topics":"Agent memory"}'
```
## 失败与重跑
- Hugging Face HTTP 请求失败会指数退避重试,最多尝试 `hf_max_retries` 次;响应数据格式无效时立即失败。
- 候选少于三篇、Agent 精选不合法、PDF 无效/过大/无文本或 Agent 输出为空都会终止 Job。
- 三篇论文按顺序处理;中途失败时,之前已完成的 PDF 和笔记会保留。
- `force=true` 会重新生成本次入选论文的解读和简报,并复用有效 PDF不会删除当日目录中已有的其他笔记。
- 多文件流程不是事务,也没有同一日期的全局运行锁。
## 测试
单元测试会 mock Hugging Face、arXiv、AgentScope 和 DingTalk 边界,不访问真实服务:
```bash
python -m pip install -e ".[dev,core]"
pytest tests/unit/test_daily_paper.py -v
```

View file

@ -39,8 +39,8 @@ The project requires Python 3.11 or later. A virtual environment is recommended:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e reme_studio -e ".[dev,full]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[dev,full]"
cd website
npm ci
npm run build:static
cd ..

View file

@ -55,12 +55,11 @@ Core layers:
reme/
reme.py # CLI entry point
application.py # Application assembly and lifecycle
plugin.py # installed plugin contract and entry-point loader
config/
default.yaml # default service / jobs / components
config_parser.py # config=, dot notation, and env placeholder parsing
components/
component_registry.py # backend registry and application-local copies
component_registry.py # global registry R
base_component.py # ComponentMixin / BaseComponent / bind dependency declarations
runtime_context.py # context for one Job execution
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
@ -77,17 +76,11 @@ reme/
base_step.py # BaseStep, Ref, dispatch_steps
common/ # version, help, health_check, status, chat
benchmark/ # LongMemEval / BEAM evaluation steps
cookbook/ # built-in cookbook support steps
cookbook/ # optional research workflow steps
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory, auto_resource, auto_dream, proactive
transfer/ # upload/download
plugins/
auto-fin/ # independent example plugin distribution
daily_paper/ # independent paper-research plugin distribution
integrations/
claude_code/ # Claude Code adapter and marketplace
hermes_agent/ # Hermes Agent memory-provider adapter
```
The default workspace directories are defined by `ApplicationConfig`:
@ -174,8 +167,8 @@ HTTP service behavior:
After registering Job endpoints, the HTTP service can also mount the ReMe Studio single-page application. The default is
`service.web_enabled=true`. Builds are resolved from `service.web_static_dir`, `REME_WEB_STATIC_DIR`, the optional
`reme_studio` package installed by the `web` and `core` extras, and source-tree locations such as
`reme_studio/dist-static`. If no `index.html` is found, only the frontend is skipped and the Job API remains available. The
`reme-ai-studio` package installed by the `web` and `core` extras, and source-tree locations such as
`website/dist-static`. If no `index.html` is found, only the frontend is skipped and the Job API remains available. The
Studio `GET` fallback does not replace existing `POST /<job.name>` routes.
MCP service behavior:
@ -223,41 +216,14 @@ The registry key is:
The same backend name can therefore exist under different component types. For example, `http` can be both a service
backend and a client backend.
`ComponentEnum` provides the built-in identifiers, but installed plugins may declare a new type with a namespaced
string such as `example.reranker`. Custom identifiers use lowercase letters and numbers separated by `.`, `_`, or `-`.
They are configured under `components` and participate in the same dependency ordering and lifecycle as built-ins.
### 4.2 Registration Through Module Imports
### 4.2 Built-in and Plugin Registration
Registration happens when a module is imported. `reme/components/__init__.py` imports component packages, while
`reme/steps/__init__.py` imports `benchmark/common/cookbook/evolve/file_io/index/transfer`. Each package's `__init__.py`
then imports its concrete modules, causing `@R.register(...)` to execute.
Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point
group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to
`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The
entry-point name is the plugin's identity.
Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override.
Plugin registration therefore stays local to one application;
duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other.
The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration
files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. See the independently
packaged [Auto Fin](../../plugins/auto-fin/README.md) and [Daily Paper](../../plugins/daily_paper/README.md) plugins.
Plugin packages are managed locally and remain separate from per-application activation:
```bash
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
```
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
After adding a Step file, make sure the package's `__init__.py` imports it. Otherwise, the backend will not appear in
the registry.
### 4.3 Component.bind

View file

@ -388,8 +388,3 @@ This lets the agent see not only an isolated paragraph but also its structural p
Non-Markdown files use `DefaultFileChunker` by default. It splits by byte size and preserves a small overlap. For
Markdown, the chunker also avoids cutting `[[wikilinks]]` in the middle.
`DefaultFileChunker` and `MarkdownFileChunker` decode files with their configured `encoding` and normalize platform
newlines to LF before indexing. Their default `invalid_encoding_policy: replace` keeps decodable content searchable
when a source contains invalid bytes, without modifying the source file. Set `invalid_encoding_policy: strict` on a
chunker component to reject such files instead.

View file

@ -117,14 +117,6 @@ Out of the box, search therefore uses primarily BM25 plus link expansion. After
`SearchStep` runs vector and keyword recall together. Additionally, switching the `file_store` `backend` from `local` to
`faiss` upgrades vector retrieval from a linear scan to a FAISS HNSW index, offering faster recall at scale.
The embedding store accepts `health_check_timeout` for its startup probe. A temporary failure skips the current vector
backfill while keeping BM25 available; a later successful provider request resumes the missing-vector backfill
automatically.
Embedded integrations that have already verified a provider can call `resume_embedding(verified=True)`. When changing
the embedding vector space, pass `rebuild=True`; persisted vectors are invalidated before a serial background rebuild,
and vector search remains unavailable until the rebuilt vectors are safely persisted.
## How to Search
The `search` Job is also configured in `default.yaml`:

View file

@ -1,225 +0,0 @@
# Plugin Management
ReMe plugins are ordinary Python distributions discovered through the `reme.plugins` entry-point group. Installing a
plugin makes it available to the current Python environment; it does not enable the plugin in every ReMe application.
Keep these two operations separate:
```text
reme plugins install ... install a package into the current Python environment
plugins: [auto-fin] enable an installed plugin for one Application
```
Plugin package management is local-only. It does not run through a ReMe HTTP or MCP service and never edits application
configuration files automatically.
A typical plugin workflow has three stages:
1. Install ReMe and the plugin distribution.
2. Configure the plugin's runtime environment as described in the
[ReMe model-configuration guide](../../README.md#optional-model-configuration).
3. Start an Application with the plugin explicitly enabled, for example
`reme start plugins='["auto-fin"]'`.
## List installed plugins
```bash
reme plugins list
```
The table shows the plugin entry-point name, Python distribution, version, and plugin contract:
```text
PLUGIN DISTRIBUTION VERSION FORMAT
-------- ------------- ------- --------
auto-fin reme-auto-fin X.Y.Z manifest
```
`manifest` plugins use the current package-level `plugin.yaml` contract. `legacy` plugins use the compatible Python
descriptor contract.
A manifest separates backend registration from application configuration:
```yaml
backends:
example_step: example_plugin.steps:ExampleStep
application_defaults:
jobs:
example:
backend: base
steps:
- backend: example_step
```
`application_defaults` is a partial `ApplicationConfig`. It is kept below the manifest's `backends` namespace because
backend import declarations are part of plugin discovery and are not application configuration.
Use JSON when another local tool needs structured output:
```bash
reme plugins list --json
```
To compare installed plugins with one application config:
```bash
reme plugins list --config default
```
The optional `ENABLED` column reflects only the `plugins` list resolved from that config. A command-line override used
by another running process is not a global enable state.
## Install a plugin package
Install a published distribution:
```bash
reme plugins install reme-auto-fin
```
Install or upgrade a pinned version:
```bash
reme plugins install 'reme-auto-fin==X.Y.Z'
reme plugins install reme-auto-fin --upgrade
```
Install a local plugin project:
```bash
reme plugins install ./plugins/auto-fin
```
Use editable mode while developing it:
```bash
reme plugins install ./plugins/auto-fin --editable
```
ReMe invokes pip through the same Python interpreter that runs the `reme` command. Pip remains responsible for package
resolution, downloads, dependency changes, and build execution. Install only packages and local projects you trust.
After installation, confirm the discovered plugin name:
```bash
reme plugins list
reme plugins validate auto-fin
```
## Inspect a plugin
```bash
reme plugins show auto-fin
```
For a manifest plugin, the result includes its registered backend names and default Job names. JSON output is also
available:
```bash
reme plugins show auto-fin --json
```
`show` identifies the package contract without constructing a ReMe Application.
## Validate a plugin
Validate an installed plugin:
```bash
reme plugins validate auto-fin
```
Validate a local project before installation:
```bash
reme plugins validate ./plugins/auto-fin
```
Validation checks the entry point, `plugin.yaml`, backend imports and component types, registry collisions, merged
`application_defaults`, and the resulting `ApplicationConfig`. Validation imports plugin backend modules, so run it
only for trusted code.
## Enable a plugin in a service
Installation alone does not load plugin code into an Application. Enable plugins explicitly in configuration:
```yaml
plugins:
- auto-fin
```
Or add them for one service launch:
```bash
reme start plugins='["auto-fin"]'
```
When `config` is omitted, ReMe loads `default.yaml`. The plugin's `application_defaults` are merged below that config,
so explicit config values and CLI overrides win. This mapping is an `ApplicationConfig` fragment, not a separate
configuration schema. The plugin backends are registered only in that Application's local registry.
After the default HTTP service starts, access plugin Jobs through ReMe's CLI client or HTTP:
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
When the application uses an MCP service, service-enabled plugin Jobs appear as MCP tools instead.
Custom application configs must provide the plugin's runtime dependencies, including an `agent_wrapper.default` and
the `search` and `read` Jobs used by Auto Fin.
## Uninstall a plugin
Use the plugin entry-point name, not necessarily the distribution name:
```bash
reme plugins uninstall auto-fin
```
Skip pip's confirmation prompt when needed:
```bash
reme plugins uninstall auto-fin --yes
```
ReMe resolves `auto-fin` to the distribution that provides it, such as `reme-auto-fin`. If one distribution provides
multiple plugin entry points, the command lists the other plugins that will also be removed.
Uninstallation does not rewrite user configuration. Remove the plugin from relevant `plugins` lists yourself;
otherwise the next Application startup fails explicitly because the configured plugin is no longer installed. Restart
already-running ReMe processes after installing, upgrading, or uninstalling packages.
## Troubleshooting
### Plugin is installed but unavailable
Check that the `reme` command and pip package share one Python interpreter:
```bash
reme plugins list
python -c 'import sys; print(sys.executable)'
```
Using `reme plugins install` avoids the most common interpreter mismatch because it runs `python -m pip` with ReMe's
own interpreter.
### Plugin is installed but not loaded
Add its entry-point name to the Application's `plugins` list. ReMe intentionally has no global enable/disable state.
### Startup reports that the plugin is not installed
The active config still enables a missing plugin. Reinstall it or remove the corresponding name from `plugins`.
### Changes are not visible in a running service
Plugin discovery and backend registration happen during Application construction. Restart the service after changing
installed packages.

View file

@ -15,8 +15,8 @@ Install from source:
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e reme_studio -e ".[core]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[core]"
cd website
npm ci
npm run build:static
cd ..

View file

@ -119,7 +119,7 @@ In other words, Auto Memory builds personal knowledge from conversations, while
### Daily Paper: An Example External-Resource Workflow
Daily Paper is an optional plugin built on this file-based memory system. It collects papers from the weekly and monthly Hugging Face Papers rankings, removes items recommended recently, ranks the remaining papers, selects three, saves their PDFs, and generates Chinese paper notes and a briefing that takes about five minutes to read.
Daily Paper is an optional cookbook built on this file-based memory system. It collects papers from the weekly and monthly Hugging Face Papers rankings, removes items recommended recently, ranks the remaining papers, selects three, saves their PDFs, and generates Chinese paper notes and a briefing that takes about five minutes to read.
Imagine that you regularly follow research on agent memory. Each morning, instead of receiving only three links, you get three detailed notes already saved locally. The briefing points to the original notes through Wikilinks, and each note links back to its PDF. A month later, when you ask, “What recent methods compress long-term memory?”, those materials are already in the same retrieval system. There is no need to search through browser history again.
@ -315,12 +315,10 @@ that best fits their runtime environment and share the same local memory workspa
| Agent | Recommended integration | Capabilities after integration |
|-------|-------------------------|--------------------------------|
| **DeepSeek Harness** | Install [`@agentscope-ai/reme`](../../typescript/README.md#deepseek-harness) as a DSH profile bundle. | Long-term memory guidance, `reme_search`, automatic capture of completed main-agent turns, and scheduled Auto Dream. |
| **OpenClaw** | Install [`@agentscope-ai/reme`](../../typescript/README.md#openclaw) as the native memory plugin. | Recall before conversational root-agent runs, explicit search, automatic turn capture, and scheduled Auto Dream. |
| **QwenPaw** | Embed ReMe in-process through the Python API. | Reuse the host application's lifecycle and model configuration while keeping memories local and file-based. |
| **Claude Code** | Start the streamable HTTP MCP Service and install [`integrations/claude_code/reme`](../../integrations/claude_code/reme). | MCP memory-recall tools, the `reme-memory` skill, and a Stop hook that automatically records sessions. |
| **Hermes** | Start the HTTP Service and install [`integrations/hermes_agent`](../../integrations/hermes_agent). | Automatically recall relevant memories before model calls and invoke `auto_memory` asynchronously after each conversation turn. |
| **Codex and other CLI-capable agents** | Copy or install [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md). | Search, read, and write memories through the CLI; automatic recording requires the host agent to integrate explicitly with the conversation lifecycle. |
| **Claude Code** | Start the streamable HTTP MCP Service and install [`plugins/claude_code/reme`](../../plugins/claude_code/reme). | MCP memory-recall tools, the `reme-memory` skill, and a Stop hook that automatically records sessions. |
| **Hermes** | Start the HTTP Service and install [`plugins/hermes_agent`](../../plugins/hermes_agent). | Automatically recall relevant memories before model calls and invoke `auto_memory` asynchronously after each conversation turn. |
| **OpenClaw, Codex, and other CLI-capable agents** | Copy or install [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md). | Search, read, and write memories through the CLI; automatic recording requires the host agent to integrate explicitly with the conversation lifecycle. |
For installation, configuration, and integration demos, see the [README](../../README.md).

View file

@ -37,8 +37,8 @@ ReMe 的核心代码位于:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e reme_studio -e ".[dev,full]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[dev,full]"
cd website
npm ci
npm run build:static
cd ..

View file

@ -50,12 +50,11 @@ flowchart LR
reme/
reme.py # CLI 入口
application.py # Application 装配与生命周期
plugin.py # 已安装插件契约与 entry-point loader
config/
default.yaml # 默认 service / jobs / components
config_parser.py # config=、dot notation、env 占位符解析
components/
component_registry.py # backend 注册表及 Application 局部副本
component_registry.py # 全局注册表 R
base_component.py # ComponentMixin / BaseComponent / bind 依赖声明
runtime_context.py # 单次 Job 执行上下文
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
@ -72,17 +71,11 @@ reme/
base_step.py # BaseStep、Ref、dispatch_steps
common/ # version、help、health_check、status、chat
benchmark/ # LongMemEval / BEAM 评测步骤
cookbook/ # 内置 cookbook 支持步骤
cookbook/ # 可选研究工作流步骤
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory、auto_resource、auto_dream、proactive
transfer/ # upload/download
plugins/
auto-fin/ # 独立发布的示例插件
daily_paper/ # 独立发布的论文研究插件
integrations/
claude_code/ # Claude Code 适配器及 marketplace
hermes_agent/ # Hermes Agent memory provider 适配器
```
默认 workspace 目录由 `ApplicationConfig` 定义:
@ -168,8 +161,8 @@ HTTP service 行为:
| `enable_serve: false` | 不注册 endpoint |
HTTP service 还可以在所有 Job endpoint 注册完成后挂载 ReMe Studio 单页应用。默认 `service.web_enabled=true`;构建产物按
`service.web_static_dir``REME_WEB_STATIC_DIR`、由 `web``core` extra 安装的可选 `reme_studio` 包,以及源码树
`reme_studio/dist-static` 等候选位置解析。找不到 `index.html` 时只跳过前端Job API 仍然可用。Studio 的 `GET` fallback 不会覆盖
`service.web_static_dir``REME_WEB_STATIC_DIR`、由 `web``core` extra 安装的可选 `reme-ai-studio` 包,以及源码树
`website/dist-static` 等候选位置解析。找不到 `index.html` 时只跳过前端Job API 仍然可用。Studio 的 `GET` fallback 不会覆盖
已有的 `POST /<job.name>`
MCP service 行为:
@ -215,38 +208,13 @@ class VersionStep(BaseStep):
所以同名 backend 在不同 component type 下可以共存。例如 `http` 同时可以是 service backend 和 client backend。
`ComponentEnum` 提供内置类型标识;已安装插件也可以用 `example.reranker` 这样的命名空间字符串声明新类型。自定义标识仅使用
小写字母和数字,并以 `.``_``-` 分隔。它们配置在 `components` 下,与内置组件参与相同的依赖排序和生命周期。
### 4.2 模块导入触发注册
### 4.2 内置注册与插件注册
注册发生在模块 import 时。`reme/components/__init__.py` 会 import 各组件包,`reme/steps/__init__.py` 会 import
`benchmark/common/cookbook/evolve/file_io/index/transfer`。这些包的 `__init__.py` 再 import 具体模块,从而执行
`@R.register(...)`
内置实现通过 package import 填充内置注册表bootstrap 完成后 ReMe 会冻结这个模板,并为每个 `Application` 创建可写副本。
运行期代码通过当前 Application 的注册表解析 backend不能修改进程级模板。随后只加载最终配置中 `plugins` 明确启用的已安装插件。
插件通过 Python `reme.plugins` entry-point group 暴露其 package。package 内的 `plugin.yaml` 只有两个可选 mapping
`backends` 将注册名映射到 `module:Class``application_defaults` 提供低优先级的 `ApplicationConfig` 配置片段。
entry-point 名称就是插件标识;使用
应用配置的 `plugins` 列表或 CLI 的 `plugins=[...]` override 显式启用插件。插件注册因此只影响当前 Application两个插件提供相同
`(component_type, backend)` 时会在装配阶段失败,
不会互相覆盖。
迁移期间仍兼容旧的 Python `Plugin` descriptor 和 `reme.configs` entry point。配置的 `extends` 可以继承内置配置、
旧插件配置或文件配置。独立打包示例见 [Auto Fin](../../plugins/auto-fin/README_ZH.md) 与
[每日论文](../../plugins/daily_paper/README_ZH.md) 插件。
插件包的本地管理与单个应用是否启用插件相互独立:
```bash
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
```
这些管理命令使用当前 Python 解释器对应的 pip不通过 HTTP 或 MCP service 执行。
新增 Step 文件后,必须保证它所在包的 `__init__.py` 会 import 该模块,否则注册表里找不到这个 backend。
### 4.3 Component.bind

View file

@ -363,7 +363,3 @@ FileChunk[]
这样检索命中时Agent 不只看到孤立段落,还能看到它在原文件中的结构位置。
非 Markdown 默认走 `DefaultFileChunker`:按字节大小切分,并保留少量 overlap对 Markdown 则会避免把 `[[wikilink]]` 从中间切开。
`DefaultFileChunker``MarkdownFileChunker` 使用各自配置的 `encoding` 解码文件,并在索引前将平台换行符统一为
LF。默认的 `invalid_encoding_policy: replace` 会在源文件含无效字节时保留其中可解码的内容用于检索,但不会修改源
文件;如需拒绝此类文件,可在 chunker 组件上设置 `invalid_encoding_policy: strict`

View file

@ -106,12 +106,6 @@ file_store:
所以开箱搜索主要是 BM25 + 链接展开。把 `embedding_store: default` 打开后,`SearchStep` 会同时跑向量召回和关键词召回。此时若将
`file_store``backend``local` 改为 `faiss`,向量检索会从线性扫描升级为 FAISS HNSW 索引,在大规模 chunk 场景下召回效率更高。
Embedding store 可通过 `health_check_timeout` 配置启动探测。临时失败只会跳过本次向量回填BM25 仍可使用;
后续真实请求成功后会自动恢复缺失向量的回填。
已经完成真实服务验证的嵌入式集成可以调用 `resume_embedding(verified=True)`。切换 Embedding 向量空间时应同时传入
`rebuild=True`ReMe 会先使旧向量失效,再串行后台重建,并在新向量安全持久化前暂停向量搜索。
## 怎么搜索
`search` Job 也是在 `default.yaml` 中配置:

View file

@ -1,215 +0,0 @@
# 插件管理
ReMe 插件是通过 `reme.plugins` entry-point group 发现的普通 Python distribution。安装插件只表示它在当前 Python
环境中可用,并不会让所有 ReMe Application 自动启用该插件。
需要区分两个操作:
```text
reme plugins install ... 将插件包安装到当前 Python 环境
plugins: [auto-fin] 为一个 Application 启用已安装插件
```
插件包管理仅在本地 CLI 执行,不经过 ReMe HTTP 或 MCP service也不会自动修改应用配置文件。
典型的插件使用流程分为三个阶段:
1. 安装 ReMe 和插件 distribution。
2. 按照 [ReMe 可选模型配置说明](../../README_ZH.md#可选模型配置)配置插件运行所需的环境变量。
3. 启动 Application 时显式启用插件,例如 `reme start plugins='["auto-fin"]'`
## 查看已安装插件
```bash
reme plugins list
```
输出包含插件 entry-point 名称、Python distribution、版本和插件契约
```text
PLUGIN DISTRIBUTION VERSION FORMAT
-------- ------------- ------- --------
auto-fin reme-auto-fin X.Y.Z manifest
```
`manifest` 表示插件使用当前的 package-level `plugin.yaml` 契约;`legacy` 表示插件使用仍然兼容的 Python descriptor
契约。
manifest 将 backend 注册与应用配置分开:
```yaml
backends:
example_step: example_plugin.steps:ExampleStep
application_defaults:
jobs:
example:
backend: base
steps:
- backend: example_step
```
`application_defaults` 是一段不完整的 `ApplicationConfig`。它与 manifest 的 `backends` 命名空间分开,因为 backend
导入声明属于插件发现协议,并不是应用配置。
本地工具需要结构化结果时可以使用 JSON
```bash
reme plugins list --json
```
对照某个应用配置查看启用状态:
```bash
reme plugins list --config default
```
可选的 `ENABLED` 列只反映该配置解析出的 `plugins` 列表。其他运行中进程使用的 CLI override 不是全局启用状态。
## 安装插件包
安装已发布的 distribution
```bash
reme plugins install reme-auto-fin
```
安装指定版本或升级:
```bash
reme plugins install 'reme-auto-fin==X.Y.Z'
reme plugins install reme-auto-fin --upgrade
```
安装本地插件项目:
```bash
reme plugins install ./plugins/auto-fin
```
开发插件时使用 editable 模式:
```bash
reme plugins install ./plugins/auto-fin --editable
```
ReMe 会通过运行 `reme` 命令的同一个 Python 解释器调用 pip。包解析、下载、依赖变更和构建执行仍由 pip 负责。请只安装
可信的包和本地项目。
安装后确认 ReMe 实际发现的插件名:
```bash
reme plugins list
reme plugins validate auto-fin
```
## 查看插件详情
```bash
reme plugins show auto-fin
```
对于 manifest 插件,结果包含注册的 backend 名称和默认 Job 名称。也可以输出 JSON
```bash
reme plugins show auto-fin --json
```
`show` 只检查包契约,不构造 ReMe Application。
## 校验插件
校验已安装插件:
```bash
reme plugins validate auto-fin
```
安装前校验本地插件项目:
```bash
reme plugins validate ./plugins/auto-fin
```
校验范围包括 entry point、`plugin.yaml`、backend 导入和组件类型、registry 冲突、`application_defaults` 合并以及最终的
`ApplicationConfig`。校验过程会导入插件 backend 模块,因此只能对可信代码执行。
## 在服务中启用插件
只安装插件不会将插件代码加载到 Application。需要在配置中显式启用
```yaml
plugins:
- auto-fin
```
也可以只为本次服务启动追加插件:
```bash
reme start plugins='["auto-fin"]'
```
未传入 `config`ReMe 加载 `default.yaml`。插件的 `application_defaults` 合并在该配置之下,因此显式配置和 CLI
override 优先。这个 mapping 是 `ApplicationConfig` 配置片段,并不是另一套配置 schema。插件 backend 只注册到该
Application 的局部 registry。
默认 HTTP service 启动后,可以通过 ReMe CLI client 或 HTTP 访问插件 Job
```bash
reme auto_fin topics="黄金,AI,存储芯片"
```
```bash
curl -s http://127.0.0.1:2333/auto_fin \
-H 'Content-Type: application/json' \
-d '{"topics":"黄金,AI,存储芯片"}'
```
当应用使用 MCP service 时,允许对外服务的插件 Job 会显示为 MCP tool。
自定义应用配置需要提供插件的运行依赖,包括 `agent_wrapper.default`,以及 Auto Fin 使用的 `search``read` Jobs。
## 卸载插件
这里使用插件 entry-point 名称,它不一定等于 distribution 名称:
```bash
reme plugins uninstall auto-fin
```
需要跳过 pip 确认时:
```bash
reme plugins uninstall auto-fin --yes
```
ReMe 会将 `auto-fin` 解析为提供它的 distribution例如 `reme-auto-fin`。如果一个 distribution 提供多个插件 entry
point命令会列出同时被移除的其他插件。
卸载不会重写用户配置。请自行从相关 `plugins` 列表中删除插件,否则下一次启动 Application 时会因为配置的插件未安装而明确
失败。安装、升级或卸载包后,需要重启已经运行的 ReMe 进程。
## 常见问题
### 插件已经安装,但 ReMe 找不到
检查 `reme` 命令与安装插件使用的 pip 是否属于同一个 Python 解释器:
```bash
reme plugins list
python -c 'import sys; print(sys.executable)'
```
使用 `reme plugins install` 可以避免最常见的解释器不一致问题,因为它通过 ReMe 自己的解释器运行 `python -m pip`
### 插件已经安装,但没有加载
将插件 entry-point 名称加入 Application 的 `plugins` 列表。ReMe 刻意不提供全局 enable/disable 状态。
### 启动时报插件未安装
当前配置仍然启用了缺失插件。请重新安装插件,或者从 `plugins` 中删除对应名称。
### 运行中的服务看不到插件变化
插件发现和 backend 注册发生在 Application 构造阶段。修改已安装包后需要重启服务。

View file

@ -15,8 +15,8 @@ pip install "reme-ai[core]"
```bash
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e reme_studio -e ".[core]"
cd reme_studio
pip install -e packages/reme_ai_studio -e ".[core]"
cd website
npm ci
npm run build:static
cd ..

View file

@ -126,7 +126,7 @@ Auto Resource 提供了一条更通用的外部资料入口。资料进入 `reso
### Daily Paper外部资料工作流的一个例子
Daily Paper 是建立在这套文件化记忆之上的可选插件。它会从 Hugging Face Papers 的周榜和月榜收集论文,去除近期已经推荐过的内容,排序后精选三篇,保存
Daily Paper 是建立在这套文件化记忆之上的可选 Cookbook。它会从 Hugging Face Papers 的周榜和月榜收集论文,去除近期已经推荐过的内容,排序后精选三篇,保存
PDF并生成中文论文笔记与一份约五分钟可读完的简报。
想象一下,你持续关注 Agent Memory每天早上收到的不只是三个论文链接而是三篇已经保存到本地的详细笔记。简报通过 Wikilink
@ -334,12 +334,10 @@ ReMe 既可以作为本地记忆服务,通过 CLI、HTTP API 或 MCP Server
| Agent | 推荐接入方式 | 接入后能力 |
|----------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| **DeepSeek Harness** | 将 [`@agentscope-ai/reme`](../../typescript/README_ZH.md#deepseek-harness) 安装为 DSH profile bundle。 | 长期记忆指引、`reme_search`、自动捕获主 Agent 已完成的对话,以及定时 Auto Dream。 |
| **OpenClaw** | 将 [`@agentscope-ai/reme`](../../typescript/README_ZH.md#openclaw) 安装为原生 memory plugin。 | 根 Agent 对话运行前召回、显式搜索、自动捕获对话,以及定时 Auto Dream。 |
| **QwenPaw** | 通过 Python API 在进程内嵌入 ReMe。 | 复用宿主应用的生命周期和模型配置,同时保持记忆本地、文件化。 |
| **Claude Code** | 启动 streamable HTTP MCP Service并安装 [`integrations/claude_code/reme`](../../integrations/claude_code/reme)。 | MCP 记忆召回工具、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP Service并安装 [`integrations/hermes_agent`](../../integrations/hermes_agent)。 | 在模型调用前自动召回相关记忆,并在每轮对话完成后异步调用 `auto_memory`。 |
| **Codex 等支持 CLI 的 Agent** | 复制或安装 [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动记录需要宿主 Agent 显式接入会话生命周期。 |
| **Claude Code** | 启动 streamable HTTP MCP Service并安装 [`plugins/claude_code/reme`](../../plugins/claude_code/reme)。 | MCP 记忆召回工具、`reme-memory` skill以及自动记录会话的 Stop hook。 |
| **Hermes** | 启动 HTTP Service并安装 [`plugins/hermes_agent`](../../plugins/hermes_agent)。 | 在模型调用前自动召回相关记忆,并在每轮对话完成后异步调用 `auto_memory`。 |
| **OpenClaw、Codex 等支持 CLI 的 Agent** | 复制或安装 [`skills/reme_memory/SKILL.md`](../../skills/reme_memory/SKILL.md)。 | 通过 CLI 搜索、读取和写入记忆;自动记录需要宿主 Agent 显式接入会话生命周期。 |
安装、配置与集成演示可查看 [README 中文版](../../README_ZH.md)。

View file

@ -2,7 +2,7 @@
This directory contains the standalone Vite documentation site published at <https://reme.agentscope.io>. The
GitHub Pages fallback is <https://agentscope-ai.github.io/ReMe/>. It does not depend on the ReMe Studio application in
`reme_studio/`.
`website/`.
## Requirements
@ -50,10 +50,10 @@ The build script reads the canonical repository files directly. Do not edit gene
- `README.md` and `README_ZH.md`: project introductions
- `docs/en/` and `docs/zh/`: English and Chinese guides
- `docs/figure/`: documentation images
- `reme_studio/README.md` and `reme_studio/README_ZH.md`: ReMe Studio guide
- `typescript/README.md` and `typescript/README_ZH.md`: TypeScript client, DeepSeek Harness, and OpenClaw integration guide
- `plugins/*/README*.md`: plugin guides
- `website/README.md` and `website/README_ZH.md`: ReMe Studio guide
- `cookbook/*/README*.md`: research workflow guides
- `benchmark/{beam,longmemeval,pibench,toolmemory}/README*.md`: benchmark guides and results
- `skills/reme_memory/SKILL.md`: ReMe Memory skill guide
- `AGENTS.md`: repository development guide
To add or reorganize a document in the site navigation, update
@ -75,7 +75,7 @@ github-pages/
## Deployment
The repository workflow `.github/workflows/deploy-docs.yml` builds this directory and publishes `dist/` to GitHub Pages.
The repository workflow `.github/workflows/pages.yml` builds this directory and publishes `dist/` to GitHub Pages.
It runs after relevant documentation or site files change on `main`, and it can also be started manually from the
GitHub Actions page.

View file

@ -10,7 +10,7 @@
"dev": "node scripts/generate-content.mjs && vite",
"build": "node scripts/generate-content.mjs && vite build",
"preview": "vite preview",
"test": "node scripts/generate-content.mjs && node --test tests/*.test.mjs"
"test": "node --test tests/*.test.mjs"
},
"dependencies": {
"dompurify": "^3.2.6",

View file

@ -6,22 +6,8 @@ const siteDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..")
const repoDir = path.resolve(siteDir, "..");
const outputDir = path.join(siteDir, ".generated", "content");
const navigationGroupOrder = [
"overview",
"start",
"integration",
"fundamentals",
"automation",
"concepts",
"workspace",
"plugins",
"benchmarks",
"development",
];
const topicOrder = [
"quick_start",
"plugin_management",
"memory_as_file",
"memory_search",
"auto_memory",
@ -37,7 +23,6 @@ const topicOrder = [
const groups = {
quick_start: "start",
plugin_management: "start",
memory_as_file: "fundamentals",
memory_search: "fundamentals",
auto_memory: "automation",
@ -53,7 +38,6 @@ const groups = {
const localizedTitles = {
quick_start: { zh: "快速开始", en: "Quick Start" },
plugin_management: { zh: "插件管理", en: "Plugin Management" },
memory_as_file: { zh: "文件即记忆", en: "Memory as File" },
memory_search: { zh: "记忆检索", en: "Memory Search" },
auto_memory: { zh: "自动记忆", en: "Auto Memory" },
@ -68,19 +52,9 @@ const localizedTitles = {
};
const productDocuments = [
{
slug: "typescript",
source: "typescript",
titles: { zh: "TypeScript Agent 集成", en: "TypeScript Agent Integrations" },
descriptions: {
zh: "配置统一 HTTP client以及 DeepSeek Harness 和 OpenClaw 原生适配器。",
en: "Configure the shared HTTP client and native DeepSeek Harness and OpenClaw adapters.",
},
group: "integration",
},
{
slug: "studio",
source: "reme_studio",
source: "website",
titles: { zh: "ReMe 工作台", en: "ReMe Studio" },
descriptions: {
zh: "浏览、编辑和搜索本地记忆,并探索记忆图谱。",
@ -90,23 +64,23 @@ const productDocuments = [
},
{
slug: "daily-paper",
source: "plugins/daily_paper",
titles: { zh: "每日论文插件", en: "Daily Paper Plugin" },
source: "cookbook/daily_paper",
titles: { zh: "每日论文", en: "Daily Paper" },
descriptions: {
zh: "发现论文、解析 PDF并生成阅读笔记与每日简报。",
en: "Discover papers, analyze PDFs, and produce reading notes and a daily brief.",
},
group: "plugins",
group: "cookbooks",
},
{
slug: "auto-fin",
source: "plugins/auto-fin",
titles: { zh: "Auto Fin 插件", en: "Auto Fin Plugin" },
source: "cookbook/auto-fin",
titles: { zh: "财经研究", en: "Auto Fin" },
descriptions: {
zh: "结合最新财联社新闻与本地历史记忆生成研究报告。",
en: "Research recent CLS news with historical context from local memory.",
},
group: "plugins",
group: "cookbooks",
},
{
slug: "beam",
@ -151,6 +125,18 @@ const productDocuments = [
];
const sharedDocuments = [
{
id: "reme-memory-skill",
path: "skills/reme_memory/SKILL.md",
sourcePath: "skills/reme_memory/SKILL.md",
titles: {
zh: "ReMe 记忆技能",
en: "ReMe Memory Skill",
},
description: "Bootstrap, retrieve, write, and consolidate memory from an agent.",
group: "integration",
language: "shared",
},
{
id: "agents-guide",
path: "AGENTS.md",
@ -220,9 +206,7 @@ async function buildManifest() {
}
}
return [...documents, ...sharedDocuments].sort(
(left, right) => navigationGroupOrder.indexOf(left.group) - navigationGroupOrder.indexOf(right.group),
);
return [...documents, ...sharedDocuments];
}
await rm(path.join(siteDir, ".generated"), { recursive: true, force: true });
@ -243,6 +227,14 @@ for (const product of productDocuments) {
await cp(path.join(repoDir, product.source, filename), path.join(outputDir, product.source, filename));
}
}
await mkdir(path.join(outputDir, "website", "public"), { recursive: true });
await cp(path.join(repoDir, "website", "public", "og.jpg"), path.join(outputDir, "website", "public", "og.jpg"));
await mkdir(path.join(outputDir, "skills", "reme_memory"), { recursive: true });
await cp(
path.join(repoDir, "skills", "reme_memory", "SKILL.md"),
path.join(outputDir, "skills", "reme_memory", "SKILL.md"),
);
await writeFile(
path.join(outputDir, "manifest.json"),
`${JSON.stringify({ documents: await buildManifest() }, null, 2)}\n`,

View file

@ -24,7 +24,7 @@ const copy = {
concepts: "架构与场景",
integration: "Agent 集成",
workspace: "工作区",
plugins: "插件",
cookbooks: "研究工作流",
benchmarks: "评测",
development: "开发者规范",
},
@ -46,7 +46,7 @@ const copy = {
concepts: "Architecture & scenarios",
integration: "Agent integration",
workspace: "Workspace",
plugins: "Plugins",
cookbooks: "Research workflows",
benchmarks: "Benchmarks",
development: "Development",
},
@ -71,8 +71,8 @@ const homeCopy = {
exploreDescription: "选择你现在想完成的事情。每个入口都直接连接到对应的完整文档。",
cards: [
{ id: "studio-zh", icon: "◫", label: "管理记忆", title: "ReMe 工作台", description: "在本地 Web 工作区中浏览、编辑、搜索记忆,并探索 wikilink 图谱。", tone: "mint" },
{ id: "daily-paper-zh", icon: "◌", label: "发现与分析", title: "每日论文插件", description: "从论文榜单筛选值得阅读的工作,解析 PDF并生成笔记与五分钟简报。", tone: "blue" },
{ id: "auto-fin-zh", icon: "↗", label: "主题研究", title: "Auto Fin 插件", description: "连接最新财联社新闻和本地历史记忆,生成可追溯的研究报告。", tone: "amber" },
{ id: "daily-paper-zh", icon: "◌", label: "发现与分析", title: "每日论文", description: "从论文榜单筛选值得阅读的工作,解析 PDF并生成笔记与五分钟简报。", tone: "blue" },
{ id: "auto-fin-zh", icon: "↗", label: "主题研究", title: "财经研究", description: "连接最新财联社新闻和本地历史记忆,生成可追溯的研究报告。", tone: "amber" },
],
benchmark: "验证记忆能力",
benchmarkDescription: "从检索规模、跨会话问答、个人智能体到工具经验,查看 ReMe 的四套评测与复现实验。",
@ -88,8 +88,8 @@ const homeCopy = {
exploreDescription: "Start with what you want to accomplish. Every entry opens the complete guide.",
cards: [
{ id: "studio-en", icon: "◫", label: "Manage memory", title: "ReMe Studio", description: "Browse, edit, and search memory in a local web workspace, then explore its wikilink graph.", tone: "mint" },
{ id: "daily-paper-en", icon: "◌", label: "Discover & analyze", title: "Daily Paper Plugin", description: "Select useful papers from rankings, analyze PDFs, and create notes plus a five-minute brief.", tone: "blue" },
{ id: "auto-fin-en", icon: "↗", label: "Research topics", title: "Auto Fin Plugin", description: "Connect recent CLS news with historical local memory to produce traceable research reports.", tone: "amber" },
{ id: "daily-paper-en", icon: "◌", label: "Discover & analyze", title: "Daily Paper", description: "Select useful papers from rankings, analyze PDFs, and create notes plus a five-minute brief.", tone: "blue" },
{ id: "auto-fin-en", icon: "↗", label: "Research topics", title: "Auto Fin", description: "Connect recent CLS news with historical local memory to produce traceable research reports.", tone: "amber" },
],
benchmark: "Validate memory systems",
benchmarkDescription: "Explore four reproducible evaluations covering retrieval scale, cross-session QA, personal agents, and tool-use experience.",
@ -284,7 +284,7 @@ function renderHome(pushHistory = true) {
</div>
</section>
<section class="home-explore">
<p class="section-kicker">01 / PRODUCT & PLUGINS</p>
<p class="section-kicker">01 / PRODUCT & WORKFLOWS</p>
<h2>${labels.explore}</h2>
<p class="section-lead">${labels.exploreDescription}</p>
<div class="feature-grid">

View file

@ -1,23 +0,0 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import test from "node:test";
const manifestUrl = new URL("../.generated/content/manifest.json", import.meta.url);
test("omits retired Agent documents and places Agent integration after getting started", async () => {
const manifest = JSON.parse(await readFile(manifestUrl, "utf8"));
const documents = manifest.documents;
const groups = [...new Set(documents.map((document) => document.group))];
assert.equal(documents.some((document) => document.id === "reme-memory-skill"), false);
assert.equal(documents.some((document) => document.sourcePath.endsWith("agent_integration_plan.md")), false);
assert.deepEqual(
documents
.filter((document) => document.group === "plugins")
.map((document) => document.title || document.titles?.en),
["每日论文插件", "Auto Fin 插件", "Daily Paper Plugin", "Auto Fin Plugin"],
);
assert.equal(documents.some((document) => document.group === "cookbooks"), false);
assert.ok(groups.indexOf("integration") > groups.indexOf("start"));
assert.ok(groups.indexOf("integration") < groups.indexOf("fundamentals"));
});

View file

@ -1,11 +0,0 @@
# Agent Integrations
This directory contains host-specific adapters that connect external agents to ReMe. An integration may use the host's
plugin API, hooks, MCP configuration, or client interface, but it does not extend ReMe's runtime through the
`reme.plugins` entry-point group.
The shared TypeScript client and the DeepSeek Harness and OpenClaw adapters live in
[`../typescript`](../typescript/README.md).
Installable extensions of ReMe itself include [Auto Fin](../plugins/auto-fin/README.md) and
[Daily Paper](../plugins/daily_paper/README.md).

1
packages/reme_ai_studio/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
/LICENSE

View file

@ -0,0 +1,203 @@
# ReMe Studio
English | [简体中文](#简体中文)
ReMe Studio is the local web workspace for ReMe. It lets you browse and edit user-owned workspace files, explore memory
links, and chat with the ReMe Agent without moving durable memory into a separate application database. Search indexes,
graphs, and other derived metadata remain rebuildable from the source files.
![ReMe Studio workspace](https://raw.githubusercontent.com/agentscope-ai/ReMe/main/website/public/og.jpg)
## Installation
Install Studio together with ReMe's optional integrations:
```bash
pip install "reme-ai[core]"
```
For Studio without the other optional integrations, use `pip install "reme-ai[web]"`. The base `reme-ai` package is
headless and does not include the frontend assets.
## Features
- **Workspace browsing**: browse the full workspace or focus on journal and knowledge files through dedicated views. The
navigator refreshes as files change on disk.
- **Markdown editing and preview**: open multiple files in tabs, render Markdown front matter and GitHub Flavored
Markdown, edit with Monaco, save with optimistic modification-time checks, and download files locally.
- **Memory graph**: inspect indexed wikilinks under the `wiki`, `personal`, and
`procedure` knowledge roots, follow inbound and outbound links, and open the corresponding Markdown source.
- **Agent chat**: stream conversations with the read-only workspace Agent, see tool calls and token usage, and drag
workspace files into the conversation as references.
- **Service management**: inspect service and component memory usage, review the effective redacted configuration and
version, and rebuild derived indexes without modifying source memory files.
- **Personalization**: switch between English and Chinese, and use light, dark, or system appearance.
## Requirements
- Python 3.11 or newer with ReMe installed.
- A running ReMe HTTP service. Agent chat additionally requires a working Agent and model configuration.
- Node.js 22.13 or newer is required only when developing or building Studio from source.
See the [repository README](https://github.com/agentscope-ai/ReMe#readme) for ReMe installation and backend configuration.
## Development
Start ReMe from the repository root, then run the frontend in another terminal:
```bash
# Terminal 1, from the repository root
reme start
# Terminal 2
cd website
npm install
npm run dev
```
Open <http://localhost:3000>. The frontend connects to
`http://127.0.0.1:2333` by default. Override it when needed:
```bash
NEXT_PUBLIC_REME_API_URL=http://127.0.0.1:8000 npm run dev
```
## ReMe-hosted static build
ReMe can serve Studio from the same FastAPI process as its HTTP API. Build the static variant and restart ReMe:
```bash
cd website
npm ci
npm run build:static
cd ..
reme start
```
Open <http://127.0.0.1:2333>. The static build uses same-origin requests by default. For standalone static development,
run `npm run dev:static` and set
`VITE_REME_API_URL` to the running ReMe service URL when necessary.
The regular `npm run build` command remains the vinext/Sites deployment build;
`npm run build:static` creates `dist-static/` exclusively for FastAPI and Python package distribution.
## Configuration
The workspace hides dotfiles and dot-directories. It displays only Markdown and text files by default. Configure the
allowed extensions as a comma-separated list in `.env.local`:
```bash
NEXT_PUBLIC_REME_WORKSPACE_EXTENSIONS=md,txt,mdx
```
The memory graph requires an index built by ReMe. Rebuilding the index from the Studio settings regenerates derived data
from workspace files and does not modify the source memory.
## Checks
```bash
npm run format:check
npm run lint
npm run build
npm run build:static
npm test
```
---
# 简体中文
[English](#reme-studio) | 简体中文
ReMe Studio 是 ReMe 的本地 Web 工作区。你可以在这里浏览和编辑自己拥有的工作区文件、探索记忆之间的联系,并与 ReMe Agent
对话,而无需将持久记忆迁移到独立的应用数据库中。搜索索引、图谱和其他派生元数据均可根据源文件重建。
![ReMe Studio 工作区](https://raw.githubusercontent.com/agentscope-ai/ReMe/main/website/public/og.jpg)
## 安装
安装 Studio 和 ReMe 的可选集成功能:
```bash
pip install "reme-ai[core]"
```
如果只需要 Studio不需要其他可选集成可以使用 `pip install "reme-ai[web]"`。基础 `reme-ai` 包以无界面模式分发,
不包含前端资源。
## 功能
- **浏览工作区**:浏览完整工作区,或通过独立视图聚焦日记和知识文件;磁盘中的文件发生变化后,导航器会自动刷新。
- **Markdown 编辑与预览**:在多个标签页中打开文件,渲染 Markdown front matter 和 GitHub Flavored Markdown使用 Monaco
编辑器编辑,通过修改时间检查安全保存,并可将文件下载到本地。
- **记忆图谱**:查看知识库 `wiki``personal``procedure` 目录中已索引的 wikilink检查入链和出链并从图谱打开对应的
Markdown 源文件。
- **Agent 对话**:与只读工作区 Agent 进行流式对话,查看工具调用和 token 用量,还可将工作区文件拖入对话作为引用。
- **服务管理**:查看服务及组件的内存使用情况、当前生效的脱敏配置和版本,并在不修改记忆源文件的情况下重建派生索引。
- **个性化设置**:切换中英文界面,并使用浅色、深色或跟随系统的外观。
## 环境要求
- Python 3.11 或更高版本,并已安装 ReMe。
- 正在运行的 ReMe HTTP 服务。Agent 对话还需要可用的 Agent 和模型配置。
- 只有从源码开发或构建 Studio 时才需要 Node.js 22.13 或更高版本。
ReMe 的安装和后端配置请参阅[仓库中文 README](https://github.com/agentscope-ai/ReMe/blob/main/README_ZH.md)。
## 本地开发
先在仓库根目录启动 ReMe然后在另一个终端运行前端
```bash
# 终端 1仓库根目录
reme start
# 终端 2
cd website
npm install
npm run dev
```
打开 <http://localhost:3000>。前端默认连接 `http://127.0.0.1:2333`,需要时可覆盖该地址:
```bash
NEXT_PUBLIC_REME_API_URL=http://127.0.0.1:8000 npm run dev
```
## 由 ReMe 托管的静态构建
ReMe 可以通过提供 HTTP API 的同一个 FastAPI 进程托管 Studio。构建静态版本并重启 ReMe
```bash
cd website
npm ci
npm run build:static
cd ..
reme start
```
打开 <http://127.0.0.1:2333>。静态构建默认使用同源请求。进行独立的静态开发时,运行
`npm run dev:static`;如有需要,将 `VITE_REME_API_URL` 设置为正在运行的 ReMe 服务地址。
常规的 `npm run build` 命令仍用于 vinext/Sites 部署构建;`npm run build:static` 仅为 FastAPI 和 Python 包分发生成
`dist-static/`
## 配置
工作区会隐藏点文件和点目录,并且默认只显示 Markdown 和文本文件。可以在 `.env.local` 中通过逗号分隔的列表配置允许显示的扩展名:
```bash
NEXT_PUBLIC_REME_WORKSPACE_EXTENSIONS=md,txt,mdx
```
记忆图谱依赖 ReMe 构建的索引。在 Studio 设置中重建索引时,只会根据工作区文件重新生成派生数据,不会修改记忆源文件。
## 检查
```bash
npm run format:check
npm run lint
npm run build
npm run build:static
npm test
```

View file

@ -1,6 +1,6 @@
[project]
name = "reme_studio"
version = "0.1.1"
name = "reme-ai-studio"
version = "0.4.1.7"
description = "Optional ReMe Studio static frontend."
readme = "README.md"
license = "Apache-2.0"
@ -8,17 +8,17 @@ license-files = ["LICENSE"]
requires-python = ">=3.11"
[project.urls]
Homepage = "https://github.com/agentscope-ai/ReMe/tree/main/reme_studio"
Homepage = "https://github.com/agentscope-ai/ReMe/tree/main/website"
Documentation = "https://reme.agentscope.io"
Repository = "https://github.com/agentscope-ai/ReMe"
[tool.setuptools]
package-dir = { "" = "src" }
packages = ["reme_studio"]
packages = ["reme_ai_studio"]
include-package-data = false
[tool.setuptools.package-data]
"reme_studio" = ["static/**"]
"reme_ai_studio" = ["static/**"]
[build-system]
requires = ["setuptools>=77", "wheel"]

View file

@ -1,201 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2025 Alibaba Group
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View file

@ -1,31 +0,0 @@
[project]
name = "reme-auto-fin"
version = "0.1.2"
description = "Auto Fin example plugin for ReMe."
readme = "README.md"
license = "Apache-2.0"
license-files = ["LICENSE"]
requires-python = ">=3.11"
dependencies = [
"reme-ai>=0.4.1.9",
]
[project.entry-points."reme.plugins"]
auto-fin = "reme_auto_fin"
[tool.setuptools]
package-dir = { "" = "src" }
packages = ["reme_auto_fin"]
include-package-data = true
[tool.setuptools.package-data]
reme_auto_fin = ["*.yaml"]
[tool.pytest.ini_options]
asyncio_default_fixture_loop_scope = "function"
pythonpath = ["src", "../.."]
testpaths = ["tests"]
[build-system]
requires = ["setuptools>=77", "wheel"]
build-backend = "setuptools.build_meta"

View file

@ -1,50 +0,0 @@
backends:
auto_fin_data_step: reme_auto_fin.data:AutoFinDataStep
auto_fin_topic_step: reme_auto_fin.topic:AutoFinTopicStep
auto_fin_merge_step: reme_auto_fin.merge:AutoFinMergeStep
application_defaults:
jobs:
auto_fin:
backend: base
description: "Fetch and research recent topic-related CLS news."
parameters:
type: object
properties:
date:
type: string
description: "Current date in YYYY-MM-DD; empty means today in Asia/Shanghai."
default: ""
now:
type: string
description: "Optional simulated current time in ISO 8601 format; empty means the real current time."
default: ""
topics:
type: string
description: "Comma-separated topics used to filter current CLS news."
default: "黄金,机器人,半导体"
window_hours:
type: number
exclusiveMinimum: 0
description: "Rolling number of hours of CLS news to fetch."
default: 24
request_interval:
type: number
minimum: 0
description: "Minimum delay in seconds after each CLS request attempt."
default: 10
max_retries:
type: integer
minimum: 1
description: "Maximum attempts for each CLS page request."
default: 3
steps: &auto_fin_steps
- backend: auto_fin_data_step
- backend: auto_fin_topic_step
- backend: auto_fin_merge_step
job_tools: [search, read]
auto_fin_cron:
backend: cron
cron: "0 18 * * *"
steps: *auto_fin_steps

View file

@ -40,19 +40,19 @@ server means one set of background watchers / dream cron across all your Claude
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
```
3. Start the ReMe HTTP server (one time, leave it running):
3. Start the ReMe MCP server (one time, leave it running):
```bash
reme start service.backend=http
reme start service.backend=mcp service.transport=streamable-http
```
The same process serves the JSON Job API and MCP at `http://127.0.0.1:2333/mcp`. To use a different port, start with
It serves `http://127.0.0.1:2333/mcp`. To use a different port, start with
`service.port=<port>` and update the `url` in `.mcp.json` to match.
## Install the plugin
```
/plugin marketplace add ./integrations/claude_code
/plugin marketplace add ./plugins/claude_code
/plugin install reme@reme-marketplace
```
@ -62,10 +62,10 @@ recall memory and report server health.
## Notes
- The plugin's MCP server URL lives in `integrations/claude_code/reme/.mcp.json`. Keep it in sync with how you start
- The plugin's MCP server URL lives in `plugins/claude_code/reme/.mcp.json`. Keep it in sync with how you start
ReMe (host/port). The Stop hook reads this same file to find the server (override with `REME_HOST`
/ `REME_PORT` env vars).
- The Stop hook needs `python3` on `PATH` and resolves transcripts under `~/.claude/projects`
(override the base with `CLAUDE_CONFIG_DIR`). It logs to `integrations/claude_code/reme/logs/auto_memory_hook.log`.
(override the base with `CLAUDE_CONFIG_DIR`). It logs to `plugins/claude_code/reme/logs/auto_memory_hook.log`.
- The MCP tool-name prefix (`mcp__reme__…`) may include the server segment depending on your Claude
Code version; the skill uses the `mcp__reme__*` wildcard so it works either way.

View file

@ -14,7 +14,7 @@ The recall tools come from the `reme` MCP server (surfaced as `mcp__reme__…`):
running:
```
reme start service.backend=http
reme start service.backend=mcp service.transport=streamable-http
```
If the tools are missing, that server is not running — tell the user the command above instead of
@ -44,7 +44,7 @@ If nothing useful comes back, say so plainly rather than guessing.
To check ReMe is up: call `version` and `health_check`, then summarize the version and the health
snapshot (components, workspace). If the `mcp__reme__…` tools are not available at all, the server
is not running — tell the user to start it with the command above. The plugin connects at
`http://127.0.0.1:2333/mcp`; a different host/port must match the `url` in the bundled `.mcp.json`.
`http://127.0.0.1:2333/mcp`; a different host/port must match the `url` in `plugins/reme/.mcp.json`.
## Workspace model

View file

@ -1,201 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2025 Alibaba Group
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View file

@ -1,137 +0,0 @@
# Daily Paper Plugin
[中文](README_ZH.md)
Daily Paper selects three papers from the Hugging Face Papers weekly and monthly rankings, downloads their arXiv PDFs,
and produces detailed Chinese reading notes plus a roughly five-minute Chinese brief. This directory is an independent
Python distribution. Its single `reme.plugins` entry point exposes a `plugin.yaml` containing five Step backends and
their Job configuration under `application_defaults`. Enable the installed plugin explicitly through
`plugins=["daily-paper"]`.
## Quick start
### 1. Install ReMe and Daily Paper
```bash
python -m pip install "reme-ai[core]>=0.4.1.9"
reme plugins install reme-daily-paper
```
### 2. Configure the model environment
Configure the LLM environment variables as described in the
[ReMe model-configuration guide](../../README.md#optional-model-configuration). Other compatible models and providers
can also be used. The workflow also requires network access to Hugging Face Papers and arXiv.
### 3. Start ReMe with the plugin
```bash
reme start plugins='["daily-paper"]'
```
With no explicit `config`, ReMe loads `default.yaml` and adds the plugin to that service. The plugin starts
`daily_paper_cron`, which runs daily at 08:00. From another terminal, generate a brief manually through ReMe's CLI
client:
```bash
reme daily_paper topics="Agent memory"
```
Or call its HTTP endpoint directly:
```bash
curl -s http://127.0.0.1:2333/daily_paper \
-H 'Content-Type: application/json' \
-d '{"topics":"Agent memory"}'
```
To run the Job once without starting a long-lived service:
```bash
reme start plugins='["daily-paper"]' job=daily_paper topics="Agent memory"
```
## Pipeline
```text
Hugging Face weekly/monthly rankings
merge ranks and exclude yesterday's and recently recommended papers
rank with RRF and let an Agent select three papers
download and parse arXiv PDFs, then write three Chinese analyses
use search + read to connect prior memory and generate a brief
refresh the daily index and optionally send the brief to DingTalk
```
`daily_paper_collect_step` concurrently reads the weekly and monthly rankings for the run date plus the strictly
preceding day's Daily Papers. It merges candidates by arXiv ID and excludes both yesterday's list and papers recommended
within `history_days`.
`daily_paper_rank_step` combines weekly and monthly positions with reciprocal-rank fusion and retains at most
`candidate_limit` papers. `daily_paper_select_step` then asks a tool-free Agent to select three unique candidate IDs.
Non-empty `topics` affect selection preference but not the fixed count.
`daily_paper_analyze_step` downloads PDFs into `resource/papers/`, reuses existing valid files, and extracts text within
the configured page, character, and file-size limits. It writes the three Chinese analyses in selection order. Scanned
PDFs and files without a text layer fail explicitly.
`daily_paper_digest_step` treats those three analyses as the factual source and receives only the read-only
`search` and `read` tools for linking earlier memory. Code validates historical wikilinks, appends links to all
three source notes, and rebuilds the daily index. The optional `dingtalk_markdown_send_step` sends the final brief when
conversation IDs are configured and otherwise skips without side effects.
## Parameters
| Parameter | Default | Purpose |
|-----------------|--------:|-----------------------------------------------------------------------------------|
| `date` | `""` | Empty uses today in the application timezone; otherwise use `YYYY-MM-DD` |
| `force` | `false` | Regenerate when that day's final brief already exists |
| `use_hf_mirror` | `false` | Use `HF_MIRROR_URL`, or `https://hf-mirror.com` when it is unset |
| `topics` | `""` | Optional topics to prioritize during selection |
| `weekly_weight` | `0.7` | Weekly contribution in reciprocal-rank fusion |
| `history_days` | `30` | Prior recommendation window excluded by arXiv ID |
Step-level defaults are `candidate_limit=20`, `rrf_k=60`, `hf_timeout=600`, `hf_max_retries=3`, `pdf_timeout=600`,
`max_pdf_bytes=52428800`, `max_pdf_pages=35`, and `max_pdf_chars=300000`.
The data clients automatically honor `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`. Manual runs enable the Hugging Face
mirror with `use_hf_mirror=true`; the cron Job enables it by default and can use the official service with
`DAILY_PAPER_USE_HF_MIRROR=false`. These environment variables override data sources and DingTalk settings:
```dotenv
HF_MIRROR_URL=https://hf-mirror.com
ARXIV_MIRROR_URL=https://export.arxiv.org
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
## Output
```text
.reme/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <Chinese-paper-title>.md # three, kind: daily-paper-analysis
│ └── <Chinese-brief-title>.md # one, kind: daily-paper-brief
└── resource/papers/
└── <arxiv-id>.pdf
```
Markdown and PDF files are written atomically through temporary files in the same directory. `force=true` regenerates
the selected analyses and brief while reusing valid PDFs; it does not delete other notes already present for that day.
Network errors, too few candidates, invalid Agent output, and unparseable PDFs fail explicitly.
## Validation
```bash
python -m pytest plugins/daily_paper -v
```
Unit tests mock the Hugging Face, arXiv, AgentScope, and DingTalk boundaries and do not contact external services.

View file

@ -1,127 +0,0 @@
# 每日论文插件
[English](README.md)
每日论文从 Hugging Face Papers 的周榜和月榜中筛选三篇论文,下载 arXiv PDF生成中文论文解读和一篇约五分钟可读完的
中文简报。本目录是一个独立 Python distribution单个 `reme.plugins` entry point 暴露 `plugin.yaml`,其中声明五个
Step backend并在 `application_defaults` 下提供 Job 配置;通过 `plugins=["daily-paper"]` 显式启用这个已安装插件。
## 快速开始
### 1. 安装 ReMe 和每日论文插件
```bash
python -m pip install "reme-ai[core]>=0.4.1.9"
reme plugins install reme-daily-paper
```
### 2. 配置模型环境变量
按照 ReMe README 的[可选模型配置说明](../../README_ZH.md#可选模型配置)配置 LLM 环境变量,也可以使用其他兼容的模型和
服务商。工作流还需要能够访问 Hugging Face Papers 和 arXiv。
### 3. 带插件启动 ReMe
```bash
reme start plugins='["daily-paper"]'
```
未显式传入 `config`ReMe 会加载 `default.yaml`,并将插件叠加到该服务上。插件随应用启动每天 08:00 运行的
`daily_paper_cron`;在另一个终端中,也可以通过 ReMe CLI client 手动生成简报:
```bash
reme daily_paper topics="Agent memory"
```
也可以直接调用 HTTP endpoint
```bash
curl -s http://127.0.0.1:2333/daily_paper \
-H 'Content-Type: application/json' \
-d '{"topics":"Agent memory"}'
```
如果只需运行一次 Job无需启动长期服务
```bash
reme start plugins='["daily-paper"]' job=daily_paper topics="Agent memory"
```
## 流程
```text
Hugging Face 周榜/月榜
合并排名并排除昨日及近期已推荐论文
RRF 排序后由 Agent 精选三篇
下载并解析 arXiv PDF生成三篇中文解读
使用 search + read 关联历史记忆并生成简报
写入当日索引,并按需发送到钉钉
```
`daily_paper_collect_step` 并发读取运行日期所在周和所在月的榜单,以及严格前一日的 Daily Papers。候选按 arXiv ID
合并,并排除昨日榜单和 `history_days` 窗口内已经推荐的论文。
`daily_paper_rank_step` 使用 reciprocal-rank fusion 合并周榜和月榜排名,最多保留 `candidate_limit` 篇;
`daily_paper_select_step` 再让无工具 Agent 精选三个唯一的候选 ID。非空 `topics` 只影响精选偏好,不改变固定数量。
`daily_paper_analyze_step` 下载 PDF 到 `resource/papers/`,复用已有的有效文件,并在页数、字符数和文件大小限制内提取
文本。三篇中文解读按精选顺序写入当天目录;扫描版或没有文本层的 PDF 会明确失败。
`daily_paper_digest_step` 以本次生成的三篇解读为事实来源,只开放只读的 `search``read` 来关联较早记忆。
代码会校验历史 wikilink、追加三篇源笔记链接并重建当日索引。可选的 `dingtalk_markdown_send_step` 在配置群会话后
发送最终简报;未配置时无副作用跳过。
## 参数
| 参数 | 默认值 | 作用 |
|-----------------|--------:|----------------------------------------------------------------------------------------------|
| `date` | `""` | 运行日期;空值使用应用时区当天,非空值必须为 `YYYY-MM-DD` |
| `force` | `false` | 已有当日简报时仍重新生成 |
| `use_hf_mirror` | `false` | 使用 `HF_MIRROR_URL`;未配置时使用 `https://hf-mirror.com` |
| `topics` | `""` | 精选论文时优先考虑的主题 |
| `weekly_weight` | `0.7` | RRF 中周榜权重 |
| `history_days` | `30` | 历史推荐排重窗口 |
步骤级默认值包括:`candidate_limit=20``rrf_k=60``hf_timeout=600``hf_max_retries=3`
`pdf_timeout=600``max_pdf_bytes=52428800``max_pdf_pages=35``max_pdf_chars=300000`
数据客户端自动使用 `HTTP_PROXY``HTTPS_PROXY``NO_PROXY`。手动任务通过 `use_hf_mirror=true` 启用 Hugging Face
镜像;定时任务默认启用,可设置 `DAILY_PAPER_USE_HF_MIRROR=false` 改用官方服务。以下环境变量可覆盖数据源和钉钉配置:
```dotenv
HF_MIRROR_URL=https://hf-mirror.com
ARXIV_MIRROR_URL=https://export.arxiv.org
DINGTALK_APP_KEY=your-app-key
DINGTALK_APP_SECRET=your-app-secret
DINGTALK_ROBOT_CODE=your-robot-code
DINGTALK_CONVERSATION_IDS=cid-group-one,cid-group-two
```
## 产物
```text
.reme/
├── daily/
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <中文论文标题>.md # 三篇kind: daily-paper-analysis
│ └── <中文简报标题>.md # 一篇kind: daily-paper-brief
└── resource/papers/
└── <arxiv-id>.pdf
```
Markdown 和 PDF 都通过同目录临时文件原子写入。`force=true` 会重新生成本次入选论文的解读和简报,并复用有效 PDF
不会删除当天已有的其他笔记。网络错误、候选不足、无效 Agent 输出和无法解析的 PDF 都会明确失败。
## 验证
```bash
python -m pytest plugins/daily_paper -v
```
单元测试 mock Hugging Face、arXiv、AgentScope 和钉钉边界,不访问外部服务。

View file

@ -1,32 +0,0 @@
[project]
name = "reme-daily-paper"
version = "0.1.2"
description = "Daily Paper research and reading-note plugin for ReMe."
readme = "README.md"
license = "Apache-2.0"
license-files = ["LICENSE"]
requires-python = ">=3.11"
dependencies = [
"pypdf>=5.0.0",
"reme-ai>=0.4.1.9",
]
[project.entry-points."reme.plugins"]
daily-paper = "reme_daily_paper"
[tool.setuptools]
package-dir = { "" = "src" }
packages = ["reme_daily_paper"]
include-package-data = true
[tool.setuptools.package-data]
reme_daily_paper = ["*.yaml"]
[tool.pytest.ini_options]
asyncio_default_fixture_loop_scope = "function"
pythonpath = ["src", "../.."]
testpaths = ["tests"]
[build-system]
requires = ["setuptools>=77", "wheel"]
build-backend = "setuptools.build_meta"

View file

@ -1,82 +0,0 @@
backends:
daily_paper_collect_step: reme_daily_paper.collect:DailyPaperCollectStep
daily_paper_rank_step: reme_daily_paper.rank:DailyPaperRankStep
daily_paper_select_step: reme_daily_paper.select:DailyPaperSelectStep
daily_paper_analyze_step: reme_daily_paper.analyze:DailyPaperAnalyzeStep
daily_paper_digest_step: reme_daily_paper.digest:DailyPaperDigestStep
application_defaults:
jobs:
daily_paper:
backend: base
description: "Build detailed readings and a five-minute brief from Hugging Face weekly/monthly papers."
candidate_limit: &candidate_limit 20
rrf_k: &rrf_k 60
weekly_weight: &weekly_weight 0.7
history_days: &history_days 30
hf_timeout: &hf_timeout 600
hf_max_retries: &hf_max_retries 3
pdf_timeout: &pdf_timeout 600
max_pdf_bytes: &max_pdf_bytes 52428800
max_pdf_pages: &max_pdf_pages 35
max_pdf_chars: &max_pdf_chars 300000
parameters:
type: object
properties:
date:
type: string
description: "Run date in YYYY-MM-DD; empty means today in the application timezone."
default: ""
force:
type: boolean
description: "Regenerate even when that day's final brief already exists."
default: false
use_hf_mirror:
type: boolean
description: "Use the Hugging Face mirror configured by HF_MIRROR_URL, or hf-mirror.com when unset."
default: false
topics:
type: string
description: "Optional topics to prioritize when selecting papers."
default: ""
weekly_weight:
type: number
description: "Weekly contribution in reciprocal-rank fusion."
default: 0.7
history_days:
type: integer
description: "Prior recommendation window excluded by arXiv ID."
default: 30
steps: &daily_paper_steps
- backend: daily_paper_collect_step
- backend: daily_paper_rank_step
- backend: daily_paper_select_step
- backend: daily_paper_analyze_step
- backend: daily_paper_digest_step
job_tools: [search, read]
- backend: dingtalk_markdown_send_step
input_mapping:
daily_paper_digest_path: markdown_path
app_key: ${DINGTALK_APP_KEY:-}
app_secret: ${DINGTALK_APP_SECRET:-}
robot_code: ${DINGTALK_ROBOT_CODE:-}
conversation_ids: ${DINGTALK_CONVERSATION_IDS:-}
title: ReMe Daily Paper
timeout: 15
daily_paper_cron:
backend: cron
cron: "0 8 * * *"
topics: "大模型长期记忆"
use_hf_mirror: ${DAILY_PAPER_USE_HF_MIRROR:-true}
candidate_limit: *candidate_limit
rrf_k: *rrf_k
weekly_weight: *weekly_weight
history_days: *history_days
hf_timeout: *hf_timeout
hf_max_retries: *hf_max_retries
pdf_timeout: *pdf_timeout
max_pdf_bytes: *max_pdf_bytes
max_pdf_pages: *max_pdf_pages
max_pdf_chars: *max_pdf_chars
steps: *daily_paper_steps

View file

@ -38,7 +38,7 @@ retrieval is enabled. Keep the service running while Hermes is active.
Hermes supports installing a plugin from a repository subdirectory:
```bash
hermes plugins install agentscope-ai/ReMe/integrations/hermes_agent
hermes plugins install agentscope-ai/ReMe/plugins/hermes_agent
hermes memory setup
```

View file

@ -38,17 +38,15 @@ dependencies = [
"uvicorn>=0.41.0",
"watchfiles>=1.1.1",
"zstandard>=0.23.0",
"pypdf>=5.0.0",
]
[project.optional-dependencies]
as = [
"agentscope[model-ollama]==2.0.7",
]
web = [
"reme_studio",
"reme-ai-studio==0.4.1.7",
]
core = [
"reme-ai[as]",
"agentscope==2.0.4.post1",
"claude-agent-sdk>=0.2.126",
"dingtalk-stream>=0.24.3",
"openai-codex>=0.144.4",
@ -60,7 +58,7 @@ core = [
"neo4j>=6.2.0",
"networkx>=3.4.2",
"polars>=1.43.0",
"reme_studio",
"reme-ai-studio==0.4.1.7",
]
dev = [
"packaging>=24.2",
@ -82,7 +80,7 @@ Repository = "https://github.com/agentscope-ai/ReMe"
reme = "reme.reme:main"
[tool.setuptools]
packages = { find = { where = ["."], include = ["reme", "reme.*"], exclude = ["reme_studio*"] } }
packages = { find = { where = ["."], include = ["reme*"] } }
include-package-data = true
[tool.setuptools.package-data]

View file

@ -1,6 +1,6 @@
"""ReMe CLI package."""
__version__ = "0.4.1.9"
__version__ = "0.4.1.7"
from . import config
from . import constants
@ -8,20 +8,13 @@ from . import enumeration
from . import schema
from . import steps
from . import utils
from .components import BaseComponent, R
from .application import Application
from .plugin import Backend, Plugin
from .components import BaseComponent
from .reme import ReMe
# Component and Step packages above have completed their decorator-driven
# bootstrap. Runtime code receives mutable copies of this immutable template.
R.freeze()
__all__ = [
"Application",
"BaseComponent",
"Backend",
"Plugin",
"ReMe",
# submodules
"config",

View file

@ -7,24 +7,22 @@ from pathlib import Path
from typing import AsyncGenerator, TypeVar
from . import __version__
from .components import ApplicationContext, BaseComponent
from .components import BaseComponent, ApplicationContext
from .components.job import BackgroundJob, BaseJob, CronJob, StreamJob
from .components.service import BaseService
from .enumeration import ComponentEnum, ComponentType, component_type_name
from .plugin import resolve_plugin_runtime
from .enumeration import ComponentEnum
from .schema import ComponentConfig, Response, StreamChunk
from .utils import execute_stream_task, print_logo, get_logger
T = TypeVar("T", bound=BaseComponent)
_NodeKey = tuple[str, str]
_NodeKey = tuple[ComponentEnum, str]
class Application(BaseComponent):
"""Wires components from config and runs jobs against them."""
def __init__(self, **kwargs) -> None:
runtime = resolve_plugin_runtime(kwargs)
self.context = ApplicationContext(registry=runtime.registry, **runtime.config)
self.context = ApplicationContext(**kwargs)
self._started_components: list[BaseComponent] = []
self._setup_workspace_directories()
@ -100,7 +98,7 @@ class Application(BaseComponent):
def _instantiate(
self,
ctype: ComponentType,
ctype: ComponentEnum,
cfg: ComponentConfig,
*,
label: str,
@ -111,13 +109,16 @@ class Application(BaseComponent):
`label` is the human-readable identifier used only in error messages.
`expected_type` narrows the return type and guards against a backend
registered under the wrong component type.
registered under the wrong ComponentEnum.
`name` is forwarded to the constructor for named components/jobs;
leave it None for the service, which is keyed solely by type.
"""
# Lazy import: the registry self-populates as component modules load.
from .components import R
if not cfg.backend:
raise ValueError(f"{label} is missing the required 'backend' field")
backend_cls = self.context.registry.get(ctype, cfg.backend)
backend_cls = R.get(ctype, cfg.backend)
if backend_cls is None:
raise ValueError(f"Unregistered backend '{cfg.backend}' for {label}")
@ -152,7 +153,7 @@ class Application(BaseComponent):
heapq.heappush(ready, downstream)
if len(ordered) != len(nodes):
unresolved = [f"{k[0]}:{k[1]}" for k, d in in_degree.items() if d > 0]
unresolved = [f"{k[0].value}:{k[1]}" for k, d in in_degree.items() if d > 0]
raise ValueError(f"Circular dependency detected among: {unresolved}")
return ordered
@ -171,7 +172,7 @@ class Application(BaseComponent):
in_degree[key] += 1
elif not dep.optional:
raise ValueError(
f"Component {key[0]}:{key[1]} depends on unregistered {dep.ctype}:{dep.name}",
f"Component {key[0].value}:{key[1]} depends on unregistered {dep.ctype.value}:{dep.name}",
)
return in_degree, dependents
@ -204,7 +205,7 @@ class Application(BaseComponent):
await c.start()
self._started_components.append(c)
except Exception as e:
self.logger.exception(f"Failed to start {component_type_name(c.component_type)}:{c.name}: {e}")
self.logger.exception(f"Failed to start {c.component_type.value}:{c.name}: {e}")
raise
async def _close(self) -> None:
@ -213,23 +214,23 @@ class Application(BaseComponent):
try:
await c.close()
except Exception as e:
self.logger.exception(f"Failed to close {component_type_name(c.component_type)}:{c.name}: {e}")
self.logger.exception(f"Failed to close {c.component_type.value}:{c.name}: {e}")
self._started_components.clear()
if self.context.thread_pool is not None:
self.context.thread_pool.shutdown(wait=True)
self.context.thread_pool = None
async def update_component(self, component_enum: ComponentType, name: str, /, **kwargs) -> BaseComponent:
async def update_component(self, component_enum: ComponentEnum | str, name: str, /, **kwargs) -> BaseComponent:
"""Update an existing component by type/name; never creates missing components."""
component_type = component_type_name(component_enum)
group = self.context.components.get(component_type)
component_enum = ComponentEnum(component_enum)
group = self.context.components.get(component_enum)
if not group or name not in group:
raise KeyError(f"Component '{name}' not found in {component_type}")
raise KeyError(f"Component '{name}' not found in {component_enum.value}")
component = group[name]
for key, value in kwargs.items():
if not hasattr(component, key):
raise AttributeError(f"Component {component_type}:{name} has no attribute '{key}'")
raise AttributeError(f"Component {component_enum.value}:{name} has no attribute '{key}'")
setattr(component, key, value)
return component

View file

@ -16,7 +16,7 @@ from . import service
from . import tokenizer
from .application_context import ApplicationContext
from .base_component import BaseComponent, ComponentMixin
from .component_registry import ComponentRegistry, R, create_application_registry
from .component_registry import ComponentRegistry, R
from .prompt_handler import PromptHandler
from .runtime_context import RuntimeContext
@ -26,7 +26,6 @@ __all__ = [
"ComponentMixin",
"ComponentRegistry",
"R",
"create_application_registry",
"PromptHandler",
"RuntimeContext",
# base components

View file

@ -3,8 +3,8 @@
from concurrent.futures import ThreadPoolExecutor
from typing import TYPE_CHECKING, Any
from ..enumeration import ComponentEnum
from ..schema import ApplicationConfig
from .component_registry import ComponentRegistry, create_application_registry
if TYPE_CHECKING:
from .base_component import BaseComponent
@ -20,14 +20,13 @@ class ApplicationContext:
components, jobs, and the service can find each other at runtime.
"""
def __init__(self, *, registry: ComponentRegistry | None = None, **kwargs):
def __init__(self, **kwargs):
# Parse raw kwargs into a typed, validated config object.
self.app_config: ApplicationConfig = ApplicationConfig(**kwargs)
self.registry = registry or create_application_registry()
# Populated by Application during initialization.
self.service: "BaseService | None" = None
self.components: dict[str, dict[str, "BaseComponent"]] = {}
self.components: dict[ComponentEnum, dict[str, "BaseComponent"]] = {}
self.jobs: dict[str, "BaseJob"] = {}
self.thread_pool: ThreadPoolExecutor | None = None

View file

@ -121,15 +121,6 @@ class BaseAsEmbedding(BaseComponent):
response = await self.model(inputs, **kwargs) # pylint: disable=not-callable
return response.embeddings
def initialize_model(self) -> None:
"""Construct the provider without making a remote request.
Callers that apply their own request timeout can initialize first so
one-time SDK imports and client construction do not consume that
timeout budget. Normal embedding calls remain lazily initialized.
"""
self._ensure_model()
async def _start(self) -> None:
"""Defer provider construction until the first remote embedding call."""
return None

View file

@ -5,7 +5,7 @@ from abc import ABC
from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable, TypeVar, cast
from ..enumeration import ComponentEnum, ComponentType, component_type_name
from ..enumeration import ComponentEnum
from ..utils import get_logger
if TYPE_CHECKING:
@ -61,24 +61,24 @@ class Dependency:
def __init__(
self,
ctype: ComponentType,
ctype: ComponentEnum,
name: str,
default_factory: Callable[[], Any] | None = None,
optional: bool = True,
) -> None:
self.ctype = component_type_name(ctype)
self.ctype = ctype
self.name = name
self.default_factory = default_factory
self.optional = optional
def __repr__(self) -> str:
suffix = "?" if self.optional else ""
return f"<unresolved {self.ctype}:{self.name}{suffix}>"
return f"<unresolved {self.ctype.value}:{self.name}{suffix}>"
def __getattr__(self, item: str) -> Any:
# Catches accidental use of the placeholder before start() resolves it.
raise RuntimeError(
f"Dependency {self.ctype}:{self.name} accessed before start() " f"(attribute '{item}')",
f"Dependency {self.ctype.value}:{self.name} accessed before start() " f"(attribute '{item}')",
)
@ -126,14 +126,10 @@ class BaseComponent(ComponentMixin, ABC):
if not name:
return None
ctype = getattr(base_cls, "component_type", None)
try:
ctype = component_type_name(ctype)
except (TypeError, ValueError) as exc:
if not isinstance(ctype, ComponentEnum) or ctype is ComponentEnum.BASE:
raise TypeError(
f"{base_cls.__name__} must declare a non-BASE string 'component_type'",
) from exc
if ctype == ComponentEnum.BASE.value:
raise TypeError(f"{base_cls.__name__} must declare a non-BASE 'component_type'")
f"{base_cls.__name__} must declare a non-BASE ComponentEnum 'component_type'",
)
return cast(T, Dependency(ctype, name, default_factory, optional))
@property
@ -176,7 +172,7 @@ class BaseComponent(ComponentMixin, ABC):
elif dep.optional:
setattr(self, attr, None)
else:
raise ValueError(f"{dep.ctype} '{dep.name}' not found.")
raise ValueError(f"{dep.ctype.value} '{dep.name}' not found.")
# ----- Workspace path helpers --------------------------------------------
@ -190,7 +186,7 @@ class BaseComponent(ComponentMixin, ABC):
@property
def component_metadata_path(self) -> Path:
"""Per-component metadata directory under the workspace."""
return self.workspace_metadata_path / component_type_name(self.component_type)
return self.workspace_metadata_path / self.component_type.value
# ----- Lifecycle hooks (override in subclasses) ----------------------

View file

@ -1,14 +1,12 @@
"""Registry mapping ``(component type, backend)`` to implementation classes."""
"""Global registry mapping ``(ComponentEnum, name) -> component class``."""
from collections.abc import Iterator
from contextlib import contextmanager
from threading import RLock
from typing import Callable, TypeVar, cast
from .base_component import ComponentMixin
from ..enumeration import ComponentType, component_type_name
from .base_component import BaseComponent
from ..enumeration import ComponentEnum
from ..utils import get_logger
T = TypeVar("T", bound=ComponentMixin)
T = TypeVar("T", bound=BaseComponent)
class ComponentRegistry:
@ -19,46 +17,27 @@ class ComponentRegistry:
"""
def __init__(self) -> None:
self._registry: dict[str, dict[str, type[ComponentMixin]]] = {}
self._owners: dict[tuple[str, str], str] = {}
self._lock = RLock()
self._frozen = False
self._registry: dict[ComponentEnum, dict[str, type[BaseComponent]]] = {}
self.logger = get_logger(log_to_file=False)
def _ensure_mutable(self) -> None:
"""Reject changes after this registry becomes an immutable template."""
if self._frozen:
raise RuntimeError("Component registry is frozen")
def _do_register(self, cls: type[T], name: str, *, owner: str | None = None) -> type[T]:
"""Insert ``cls`` under its component type and reject ambiguous providers."""
try:
component_type = component_type_name(getattr(cls, "component_type", None))
except (TypeError, ValueError) as exc:
raise TypeError(f"{cls.__name__} must have a non-empty string 'component_type' attribute") from exc
def _do_register(self, cls: type[T], name: str) -> type[T]:
"""Insert `cls` under its ``component_type`` group; warn on overwrite."""
component_type = getattr(cls, "component_type", None)
if not isinstance(component_type, ComponentEnum):
raise TypeError(
f"{cls.__name__} must have a ComponentEnum 'component_type' attribute",
)
if not name:
raise ValueError("Component name cannot be empty")
with self._lock:
self._ensure_mutable()
group = self._registry.setdefault(component_type, {})
key = (component_type, name)
if name in group:
existing = group[name]
existing_owner = self._owners[key]
new_owner = owner or cls.__module__
if existing is cls and existing_owner == new_owner:
return cls
raise ValueError(
f"Backend '{component_type}:{name}' is provided by both " f"'{existing_owner}' and '{new_owner}'",
)
group[name] = cls
self._owners[key] = owner or cls.__module__
group = self._registry.setdefault(component_type, {})
if name in group:
self.logger.warning(
f"Component '{name}' already registered for {component_type}, overwriting",
)
group[name] = cls
return cls
def add(self, name: str, cls: type[T], *, owner: str) -> type[T]:
"""Register one explicitly owned plugin contribution."""
return self._do_register(cls, name, owner=owner)
def register(
self,
cls_or_name: type[T] | str,
@ -81,76 +60,25 @@ class ComponentRegistry:
return decorator
def get(self, component_type: ComponentType, name: str) -> type[ComponentMixin] | None:
def get(self, component_type: ComponentEnum, name: str) -> type[BaseComponent] | None:
"""Look up a registered class; return None if not found."""
with self._lock:
return self._registry.get(component_type_name(component_type), {}).get(name)
return self._registry.get(component_type, {}).get(name)
def get_all(self, component_type: ComponentType) -> dict[str, type[ComponentMixin]]:
def get_all(self, component_type: ComponentEnum) -> dict[str, type[BaseComponent]]:
"""Return a shallow copy of all classes registered under `component_type`."""
with self._lock:
return dict(self._registry.get(component_type_name(component_type), {}))
return dict(self._registry.get(component_type, {}))
def unregister(self, component_type: ComponentType, name: str) -> bool:
def unregister(self, component_type: ComponentEnum, name: str) -> bool:
"""Remove an entry; return True if it existed, False otherwise."""
component_type = component_type_name(component_type)
with self._lock:
self._ensure_mutable()
if (group := self._registry.get(component_type)) and name in group:
del group[name]
self._owners.pop((component_type, name), None)
return True
return False
if (group := self._registry.get(component_type)) and name in group:
del group[name]
return True
return False
def clear(self) -> None:
"""Drop every registered entry."""
with self._lock:
self._ensure_mutable()
self._registry.clear()
self._owners.clear()
def freeze(self) -> None:
"""Make this registry an immutable template for future copies."""
with self._lock:
self._frozen = True
@property
def frozen(self) -> bool:
"""Whether mutating operations are disabled."""
with self._lock:
return self._frozen
def copy(self) -> "ComponentRegistry":
"""Return an independent registry containing the same providers."""
copied = ComponentRegistry()
with self._lock:
for component_type, group in self._registry.items():
for name, implementation in group.items():
copied.add(name, implementation, owner=self._owners[(component_type, name)])
return copied
@contextmanager
def preserve(self, *, allow_mutation: bool = False) -> Iterator[None]:
"""Restore the registry after code that may register through import side effects."""
with self._lock:
registry = {component_type: dict(group) for component_type, group in self._registry.items()}
owners = dict(self._owners)
frozen = self._frozen
if allow_mutation:
self._frozen = False
try:
yield
finally:
self._registry = registry
self._owners = owners
self._frozen = frozen
self._registry.clear()
# Import-time registry for built-in implementations. Runtime code should use
# ``create_application_registry`` rather than mutate this template.
# Process-wide singleton used throughout the codebase.
R = ComponentRegistry()
def create_application_registry() -> ComponentRegistry:
"""Return a mutable registry initialized from the frozen built-in template."""
return R.copy()

View file

@ -25,7 +25,6 @@ class BaseEmbeddingStore(BaseComponent):
max_input_length: int = 8192,
max_retries: int = 3,
quota_retry_delay: float | None = None,
health_check_timeout: float = 15.0,
**kwargs,
):
super().__init__(**kwargs)
@ -33,7 +32,6 @@ class BaseEmbeddingStore(BaseComponent):
self.max_input_length = max_input_length
self.max_retries = max_retries
self.quota_retry_delay = quota_retry_delay
self.health_check_timeout = health_check_timeout
self.is_healthy: bool = True
def _truncate(self, text: str) -> str:
@ -59,7 +57,7 @@ class BaseEmbeddingStore(BaseComponent):
return text
@abstractmethod
async def health_check(self, timeout: float | None = None) -> bool:
async def health_check(self, timeout: float = 2.0) -> bool:
"""Probe the provider; sets and returns is_healthy."""
async def get_embedding(self, input_text: str, **kwargs) -> np.ndarray | None:
@ -69,7 +67,7 @@ class BaseEmbeddingStore(BaseComponent):
@abstractmethod
async def get_embeddings(self, input_text: list[str], **kwargs) -> list[np.ndarray | None]:
"""Get embeddings; cache hits must not change is_healthy."""
"""Get embeddings for texts."""
def _embedding_dim_matches(self, embedding: np.ndarray | None) -> bool:
"""Return whether an embedding matches the configured model dimension."""

View file

@ -12,7 +12,6 @@ from ..component_registry import R
from ..as_embedding import BaseAsEmbedding
Miss = tuple[int, str, str] # (result_index, text, cache_key)
_MAX_VECTOR_SPACE_ATTEMPTS = 3
@R.register("local")
@ -69,60 +68,33 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
async def _close(self) -> None:
await self.dump()
async def health_check(self, timeout: float | None = None) -> bool:
timeout = self.health_check_timeout if timeout is None else timeout
if not isinstance(timeout, (int, float)) or not np.isfinite(timeout) or timeout <= 0:
raise ValueError("timeout must be finite and greater than 0")
async def health_check(self, timeout: float = 5.0) -> bool:
tag = f"[EMBEDDING HEALTH CHECK] name={self.name} workspace_dir={self.workspace_path}"
started_at = asyncio.get_running_loop().time()
try:
# Provider construction may synchronously import an SDK and build
# its HTTP client. Keep that one-time work outside the request
# timeout so the full budget applies to the initialized provider
# call instead of being consumed before a request can be sent.
self.as_embedding.initialize_model()
result = await asyncio.wait_for(self.as_embedding(["ping"]), timeout=timeout)
if not result or result[0] is None:
raise RuntimeError("empty embedding")
if len(result[0]) != self.dimensions:
raise RuntimeError(f"embedding dimension mismatch: {len(result[0])} != {self.dimensions}")
self.is_healthy = True
elapsed = asyncio.get_running_loop().time() - started_at
self.logger.info(f"{tag} -> OK timeout={timeout}s elapsed={elapsed:.3f}s")
self.logger.info(f"{tag} -> OK")
except asyncio.TimeoutError:
self.is_healthy = False
elapsed = asyncio.get_running_loop().time() - started_at
self.logger.error(f"{tag} -> FAIL timeout={timeout}s elapsed={elapsed:.3f}s error=timeout({timeout}s)")
except Exception as exc: # Provider SDKs expose many exception types.
self.logger.error(f"{tag} -> FAIL timeout({timeout}s)")
except Exception as e:
self.is_healthy = False
elapsed = asyncio.get_running_loop().time() - started_at
self.logger.error(
f"{tag} -> FAIL timeout={timeout}s elapsed={elapsed:.3f}s error={type(exc).__name__}: {exc}",
)
self.logger.error(f"{tag} -> FAIL {type(e).__name__}: {e}")
return self.is_healthy
# -- Public API --
async def get_embeddings(self, input_text: list[str], **kwargs) -> list[np.ndarray | None]:
await self._sync_cache_space()
texts = [self._truncate(t) for t in input_text]
for attempt in range(1, _MAX_VECTOR_SPACE_ATTEMPTS + 1):
await self._sync_cache_space()
vector_space_id = self._cache_space
results, misses = self._partition_by_cache(texts)
stable = not misses or await self._fill_misses(misses, results, vector_space_id, **kwargs)
if stable and vector_space_id == self.vector_space_id == self._cache_space:
return results
if attempt == _MAX_VECTOR_SPACE_ATTEMPTS:
self.logger.warning(
f"Embedding vector space kept changing while computing a request; "
f"discarding all result(s) after {attempt} attempts",
)
else:
self.logger.info(
f"Embedding vector space changed while computing a request; "
f"discarding all result(s) and retrying ({attempt}/{_MAX_VECTOR_SPACE_ATTEMPTS})",
)
return [None] * len(texts)
results, misses = self._partition_by_cache(texts)
if misses:
await self._fill_misses(misses, results, **kwargs)
return results
# -- Batching --
@ -138,26 +110,15 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
misses.append((idx, text, key))
return results, misses
async def _fill_misses(
self,
misses: list[Miss],
results: list[np.ndarray | None],
vector_space_id: str,
**kwargs,
) -> bool:
"""Fill every miss only while the request remains in one vector space."""
async def _fill_misses(self, misses: list[Miss], results: list[np.ndarray | None], **kwargs) -> None:
vector_space_id = self._cache_space
size = self.max_batch_size
for start in range(0, len(misses), size):
if vector_space_id != self.vector_space_id or vector_space_id != self._cache_space:
return False
batch = misses[start : start + size]
computed = await self._compute_batch(batch, **kwargs)
if vector_space_id != self.vector_space_id or vector_space_id != self._cache_space:
return False
for idx, key, emb in computed:
for idx, key, emb in await self._compute_batch(batch, **kwargs):
results[idx] = emb
self._cache_put(key, emb)
return True
if vector_space_id == self.vector_space_id == self._cache_space:
self._cache_put(key, emb)
async def _compute_batch(self, batch: list[Miss], **kwargs) -> list[tuple[int, str, np.ndarray]]:
texts = [text for _, text, _ in batch]
@ -177,10 +138,6 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
if bad_dims:
details = ", ".join(f"{count} with dim {dim}" for dim, count in sorted(bad_dims.items()))
self.logger.error(f"Embedding dimension mismatch in batch: expected {self.dimensions}; rejected {details}")
if out:
self.is_healthy = True
else:
self.is_healthy = False
return out
async def _call_with_retry(self, texts: list[str], **kwargs) -> list[list[float] | None] | None:
@ -204,9 +161,7 @@ class LocalEmbeddingStore(BaseEmbeddingStore):
await asyncio.sleep(self.quota_retry_delay)
continue
self.logger.exception("Embedding request failed")
self.is_healthy = False
return None
self.is_healthy = False
return None
@staticmethod

View file

@ -2,7 +2,6 @@
from bisect import bisect_right
from pathlib import Path
from typing import Literal
import aiofiles
import yaml
@ -12,29 +11,14 @@ from ..component_registry import R
from ...schema import FileChunk, FileFrontMatter, FileNode
from ...utils.wikilink_handler import WikilinkHandler
InvalidEncodingPolicy = Literal["replace", "strict"]
@R.register("default")
class DefaultFileChunker(BaseFileChunker):
"""Default chunker that splits files into byte-based overlapping chunks."""
def __init__(
self,
encoding: str = "utf-8",
chunk_byte_size: int = 10000,
overlap_byte_size: int = 100,
*,
invalid_encoding_policy: InvalidEncodingPolicy = "replace",
**kwargs,
):
def __init__(self, encoding: str = "utf-8", chunk_byte_size: int = 10000, overlap_byte_size: int = 100, **kwargs):
super().__init__(**kwargs)
if invalid_encoding_policy not in {"replace", "strict"}:
raise ValueError(
f"invalid_encoding_policy must be 'replace' or 'strict', got {invalid_encoding_policy!r}",
)
self.encoding = encoding
self.invalid_encoding_policy = invalid_encoding_policy
self.chunk_byte_size = max(100, chunk_byte_size)
self.overlap_byte_size = max(4, overlap_byte_size)
@ -53,35 +37,13 @@ class DefaultFileChunker(BaseFileChunker):
front_matter = FileFrontMatter()
return front_matter, text[end_idx + 4 :].lstrip("\n")
async def _read_text_for_indexing(self, file_path: Path) -> str:
"""Decode text for a derived index without modifying the source file."""
async with aiofiles.open(file_path, "rb") as f:
data = await f.read()
try:
text = data.decode(self.encoding)
except UnicodeDecodeError as exc:
if self.invalid_encoding_policy == "strict":
raise
invalid_bytes = data[exc.start : exc.end].hex(" ")
self.logger.warning(
f"Invalid {self.encoding} in {file_path} at byte {exc.start} (bytes: {invalid_bytes}); "
"indexed with replacement characters; source file unchanged",
)
text = data.decode(self.encoding, errors="replace")
# Some codecs (for example ASCII) cannot encode U+FFFD. Convert the
# decoded fallback to that codec's own replacement representation so
# later byte-based chunking remains safe.
text = text.encode(self.encoding, errors="replace").decode(self.encoding)
# Match the universal-newline behavior of the previous text-mode reads.
return text.replace("\r\n", "\n").replace("\r", "\n")
async def chunk(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]:
file_path = Path(path)
stat = file_path.stat()
rel_path = self.to_workspace_relative(path)
text = await self._read_text_for_indexing(file_path)
async with aiofiles.open(file_path, encoding=self.encoding) as f:
text = await f.read()
if not text:
return FileNode(path=rel_path, st_mtime=stat.st_mtime), []

View file

@ -24,7 +24,7 @@ import yaml
from pydantic import ValidationError
from .default_file_chunker import DefaultFileChunker, InvalidEncodingPolicy
from .default_file_chunker import DefaultFileChunker
from ..component_registry import R
from ...schema import (
FileChunk,
@ -108,16 +108,9 @@ class MarkdownFileChunker(DefaultFileChunker):
max_ast_sections: int | None = 100,
include_frontmatter_in_metadata: bool = False,
include_frontmatter_keys_in_metadata: list[str] | None = None,
*,
invalid_encoding_policy: InvalidEncodingPolicy = "replace",
**kwargs,
):
super().__init__(
encoding=encoding,
invalid_encoding_policy=invalid_encoding_policy,
chunk_byte_size=chunk_byte_size,
**kwargs,
)
super().__init__(encoding=encoding, chunk_byte_size=chunk_byte_size, **kwargs)
self.embed_toc = embed_toc
self.max_ast_sections = max(0, max_ast_sections) if max_ast_sections is not None else None
self.include_frontmatter_in_metadata = include_frontmatter_in_metadata
@ -126,8 +119,7 @@ class MarkdownFileChunker(DefaultFileChunker):
async def chunk(self, path: str | Path) -> tuple[FileNode, list[FileChunk]]:
file_path = Path(path)
rel_path = self.to_workspace_relative(path)
text = await self._read_text_for_indexing(file_path)
front_matter, content, line_offset = self._parse_front_matter(text)
front_matter, content, line_offset = self._parse_front_matter(file_path.read_text(encoding=self.encoding))
chunks: list[FileChunk] = []
if content and content.strip():

View file

@ -252,11 +252,6 @@ class FaissLocalFileStore(LocalFileStore):
self._add_to_index([c.id for c in to_add], vectors)
self._compact_if_needed()
async def _reset_vector_index(self) -> None:
"""Discard all vectors before rebuilding a changed vector space."""
await self._stop_reindex_worker()
self._rebuild_index()
# -- async reindex ----------------------------------------------------
def _submit_reindex(self) -> None:
@ -496,10 +491,10 @@ class FaissLocalFileStore(LocalFileStore):
self.faiss_idmap_path.unlink(missing_ok=True)
return False
async def _dump_owned_state(self) -> None:
"""Persist chunks and the FAISS sidecar, excluding dependency snapshots."""
async def dump(self) -> None:
"""Persist chunks JSONL via the parent, then write the FAISS sidecar atomically."""
async with self._faiss_dump_lock:
await super()._dump_owned_state()
await super().dump()
if self._faiss_index is None or self.embedding_store is None:
return
try:
@ -630,29 +625,26 @@ class FaissLocalFileStore(LocalFileStore):
# -- search -----------------------------------------------------------
async def vector_search(self, query: str, limit: int, search_filter: dict) -> list[FileChunk]:
index_empty = self._faiss_index is None or self._faiss_index.ntotal == 0
embedding_unavailable = self.embedding_store is None or self._embedding_rebuild_pending
if (
embedding_unavailable
self.embedding_store is None
or not query
or limit <= 0
or (index_empty and getattr(self.embedding_store, "is_healthy", True))
or self._faiss_index is None
or self._faiss_index.ntotal == 0
):
return []
query_embedding = None
was_healthy = bool(getattr(self.embedding_store, "is_healthy", True))
try:
query_embedding = await self.embedding_store.get_embedding(query)
except Exception as e:
self._mark_embedding_unhealthy(f"search: {type(e).__name__}: {e}")
self._disable_embedding(f"search: {type(e).__name__}: {e}")
if query_embedding is None or not self._embedding_dim_matches(query_embedding):
if query_embedding is not None:
self._mark_embedding_unhealthy(
self._disable_embedding(
f"search: query embedding dimension {len(query_embedding)} != {self.embedding_store.dimensions}",
)
return []
await self._recover_after_real_request(was_healthy)
# get_embedding above yielded control; a concurrent clear() drops the
# index to None once embedding is disabled, and a reindex may have swapped

View file

@ -67,14 +67,10 @@ class LocalFileStore(BaseFileStore):
self.file_chunks: dict[str, FileChunk] = {}
self.chunks_path = self.component_metadata_path / f"file_chunks_{self.name}_{self.store_version}.jsonl.zst"
self._embedding_backfill_task: asyncio.Task | None = None
self._embedding_backfill_pending: tuple[bool, bool] | None = None
self._embedding_rebuild_pending = False
self._closing = False
# -- lifecycle ------------------------------------------------------------
async def _start(self) -> None:
self._closing = False
started_at = time.monotonic()
self.component_metadata_path.mkdir(parents=True, exist_ok=True)
await super()._start()
@ -98,53 +94,17 @@ class LocalFileStore(BaseFileStore):
)
async def _close(self) -> None:
self._closing = True
await self._cancel_embedding_backfill()
# Dependencies are closed separately by Application (reverse
# topological order) or BaseComponent (owned standalone dependencies).
# Persist only this store's local state here so each component writes
# exactly once during shutdown. Preserve the historical dump() hook for
# third-party subclasses that override it to write additional state.
if type(self).dump is LocalFileStore.dump:
await self._dump_owned_state()
else:
await self.dump()
await self.dump()
self.file_chunks.clear()
await super()._close()
def _mark_embedding_unhealthy(self, reason: str) -> None:
"""Record a temporary provider failure while preserving the component."""
def _disable_embedding(self, reason: str) -> None:
"""Drop embedding after a runtime failure; keyword search still works."""
if self.embedding_store is None:
return
self.embedding_store.is_healthy = False
self.logger.error(f"{self.name}: embedding unavailable, {reason}; keyword search remains active")
async def _recover_after_real_request(self, was_healthy: bool) -> None:
"""Schedule repair when a real, non-cache provider request recovers."""
if self.embedding_store is None or was_healthy or not getattr(self.embedding_store, "is_healthy", True):
return
self.logger.info(f"{self.name}: embedding provider recovered; scheduling missing-vector backfill")
await self.resume_embedding(verified=True)
async def resume_embedding(self, *, verified: bool = False, rebuild: bool = False) -> bool:
"""Resume a configured provider and schedule a deduplicated repair.
Embedded applications may pass ``verified=True`` after they have already
made a successful real provider request, avoiding a redundant ping. Pass
``rebuild=True`` when the active vector space changed; existing vectors
are derived data and are discarded before a full background rebuild.
"""
if self.embedding_store is None or self._closing:
return False
if verified:
self.embedding_store.is_healthy = True
if rebuild:
await self._prepare_embedding_rebuild()
if not self.file_chunks:
self._embedding_rebuild_pending = False
return True
self._start_embedding_backfill(skip_health_check=verified, rebuild=rebuild)
return True
self.logger.error(f"{self.name}: embedding disabled, {reason}")
self.embedding_store = None
def _embedding_dim_matches(self, embedding: np.ndarray | None) -> bool:
"""Return whether an index embedding matches the active embedding model."""
@ -268,40 +228,29 @@ class LocalFileStore(BaseFileStore):
return
self._drop_stale_embeddings(self.file_chunks.values(), "load")
def _start_embedding_backfill(self, *, skip_health_check: bool = False, rebuild: bool = False) -> None:
def _start_embedding_backfill(self) -> None:
"""Schedule startup embedding repair without delaying component readiness."""
started_at = time.monotonic()
if self._closing:
self.logger.info(f"{self.name}: embedding backfill skipped: reason=closing")
return
if not self.embedding_store:
self.logger.info(
f"{self.name}: embedding backfill skipped: reason=embedding_disabled, "
f"elapsed={time.monotonic() - started_at:.3f}s",
)
return
if self._embedding_backfill_task is not None and not self._embedding_backfill_task.done():
pending_verified = skip_health_check or bool(
self._embedding_backfill_pending and self._embedding_backfill_pending[0],
)
pending_rebuild = rebuild or bool(
self._embedding_backfill_pending and self._embedding_backfill_pending[1],
)
if pending_verified or pending_rebuild:
self._embedding_backfill_pending = (pending_verified, pending_rebuild)
self.logger.info(
f"{self.name}: embedding backfill scheduling skipped: reason=already_running, "
f"elapsed={time.monotonic() - started_at:.3f}s",
)
return
if not self.file_chunks:
self.logger.info(
f"{self.name}: embedding backfill skipped: reason=no_chunks, "
f"elapsed={time.monotonic() - started_at:.3f}s",
)
return
if self._embedding_backfill_task is not None and not self._embedding_backfill_task.done():
self.logger.info(
f"{self.name}: embedding backfill scheduling skipped: reason=already_running, "
f"elapsed={time.monotonic() - started_at:.3f}s",
)
return
self._embedding_backfill_task = asyncio.create_task(
self._run_embedding_backfill(skip_health_check=skip_health_check, rebuild=rebuild),
self._backfill_missing_embeddings(),
name=f"embedding-backfill:{self.name}",
)
self.logger.info(
@ -309,37 +258,10 @@ class LocalFileStore(BaseFileStore):
f"elapsed={time.monotonic() - started_at:.3f}s",
)
async def _run_embedding_backfill(self, *, skip_health_check: bool, rebuild: bool) -> None:
"""Run one repair and honor a verified request queued behind it."""
current_task = asyncio.current_task()
try:
if rebuild:
# A task that was already running when rebuild was requested
# may have written a stale provider result after the first
# invalidation. Clear once more at the queue boundary.
await self._prepare_embedding_rebuild()
await self._backfill_missing_embeddings(skip_health_check=skip_health_check)
finally:
if self._embedding_backfill_task is current_task:
self._embedding_backfill_task = None
pending = self._embedding_backfill_pending
self._embedding_backfill_pending = None
if pending is not None and not self._closing and self.embedding_store is not None:
pending_verified, pending_rebuild = pending
if pending_verified:
self.embedding_store.is_healthy = True
if pending_rebuild:
self._embedding_rebuild_pending = True
self._start_embedding_backfill(
skip_health_check=pending_verified,
rebuild=pending_rebuild,
)
async def _cancel_embedding_backfill(self) -> None:
"""Cancel and collect the startup repair task during component shutdown."""
task = self._embedding_backfill_task
self._embedding_backfill_task = None
self._embedding_backfill_pending = None
if task is None:
return
if not task.done():
@ -363,22 +285,9 @@ class LocalFileStore(BaseFileStore):
next_percent += _PROGRESS_LOG_PERCENT_STEP
return next_percent
async def _backfill_missing_embeddings(self, *, skip_health_check: bool = False) -> None:
async def _backfill_missing_embeddings(self) -> None:
"""Background-repair persisted chunks that do not have usable vectors."""
started_at = time.monotonic()
try:
await self._backfill_missing_embeddings_inner(skip_health_check=skip_health_check, started_at=started_at)
finally:
if self._embedding_rebuild_pending and not self._closing:
try:
await self._after_embedding_backfill()
await self.dump()
self._embedding_rebuild_pending = False
except Exception:
self.logger.exception(f"{self.name}: failed to finalize embedding rebuild")
async def _backfill_missing_embeddings_inner(self, *, skip_health_check: bool, started_at: float) -> None:
"""Perform one backfill pass; the caller owns rebuild finalization."""
if not self.embedding_store or not self.file_chunks:
self.logger.info(
f"{self.name}: embedding backfill finished without work: "
@ -404,18 +313,18 @@ class LocalFileStore(BaseFileStore):
batch_size = max(1, int(getattr(self.embedding_store, "max_batch_size", 10)))
self.logger.info(f"{self.name}: embedding backfill started: total={total}, batch_size={batch_size}")
try:
if not skip_health_check:
health_check_started_at = time.monotonic()
is_healthy = await self.embedding_store.health_check()
self.logger.info(
f"{self.name}: embedding health check complete: healthy={is_healthy}, "
f"elapsed={time.monotonic() - health_check_started_at:.3f}s",
health_check_started_at = time.monotonic()
is_healthy = await self.embedding_store.health_check()
self.logger.info(
f"{self.name}: embedding health check complete: healthy={is_healthy}, "
f"elapsed={time.monotonic() - health_check_started_at:.3f}s",
)
if not is_healthy:
self._disable_embedding("backfill health check failed")
self.logger.warning(
f"{self.name}: embedding backfill failed: processed=0/{total}, reason=health check failed",
)
if not is_healthy:
self.logger.warning(
f"{self.name}: embedding backfill skipped: processed=0/{total}, reason=health check failed",
)
return
return
processed = 0
batch_count = 0
@ -439,9 +348,8 @@ class LocalFileStore(BaseFileStore):
f"{total}, elapsed={elapsed:.2f}s",
)
raise
except Exception as e:
self._mark_embedding_unhealthy(f"backfill: {type(e).__name__}: {e}")
self._disable_embedding(f"backfill: {type(e).__name__}: {e}")
elapsed = time.monotonic() - started_at
self.logger.exception(
f"{self.name}: embedding backfill failed: processed={processed if 'processed' in locals() else 0}/"
@ -454,24 +362,13 @@ class LocalFileStore(BaseFileStore):
self.logger.info(
f"{self.name}: embedding backfill complete: filled={filled}/{total}, elapsed={elapsed:.2f}s",
)
if filled and not self._embedding_rebuild_pending:
if filled:
try:
await self._after_embedding_backfill()
await self.dump()
except Exception:
self.logger.exception(f"{self.name}: failed to persist completed embedding backfill")
async def _prepare_embedding_rebuild(self) -> None:
"""Invalidate and persist vectors from the previous vector space."""
self._embedding_rebuild_pending = True
for chunk in self.file_chunks.values():
chunk.embedding = None
await self._reset_vector_index()
await self.dump()
async def _reset_vector_index(self) -> None:
"""Drop a derived vector index before rebuilding a changed vector space."""
async def _after_embedding_backfill(self) -> None:
"""Backend hook for refreshing derived vector indexes after backfill."""
@ -537,8 +434,9 @@ class LocalFileStore(BaseFileStore):
elapsed = time.monotonic() - started_at
self.logger.info(f"{self.name}: keyword index rebuild complete: total={total}, elapsed={elapsed:.2f}s")
async def _dump_owned_state(self) -> None:
"""Persist state owned by this store, excluding dependency snapshots."""
async def dump(self) -> None:
"""Atomically rewrite the JSONL, then cascade dump into keyword_index and file_graph."""
assert self.file_graph is not None
try:
write_jsonl_zst(
self.chunks_path,
@ -548,11 +446,6 @@ class LocalFileStore(BaseFileStore):
self.logger.info(f"Saved {len(self.file_chunks)} chunks to {self.chunks_path}")
except Exception as e:
self.logger.exception(f"Failed to write {self.chunks_path}: {e}")
async def dump(self) -> None:
"""Persist a complete store/index/graph consistency checkpoint."""
assert self.file_graph is not None
await self._dump_owned_state()
if self.keyword_index:
await self.keyword_index.dump()
await self.file_graph.dump()
@ -647,15 +540,12 @@ class LocalFileStore(BaseFileStore):
async def _embed_pending(self, chunks: list[FileChunk]) -> None:
if not (chunks and self.embedding_store):
return
was_healthy = bool(getattr(self.embedding_store, "is_healthy", True))
try:
await self.embedding_store.get_node_embeddings(chunks)
except Exception as e:
self._mark_embedding_unhealthy(f"upsert: {type(e).__name__}: {e}")
self._disable_embedding(f"upsert: {type(e).__name__}: {e}")
return
self._drop_stale_embeddings(chunks, "upsert")
if any(chunk.embedding is not None for chunk in chunks):
await self._recover_after_real_request(was_healthy)
async def delete(self, path: str | list[str]) -> None:
assert self.file_graph is not None
@ -711,23 +601,21 @@ class LocalFileStore(BaseFileStore):
# -- search ---------------------------------------------------------------
async def vector_search(self, query: str, limit: int, search_filter: dict) -> list[FileChunk]:
if self.embedding_store is None or self._embedding_rebuild_pending or not query or limit <= 0:
if self.embedding_store is None or not query or limit <= 0:
return []
was_healthy = bool(getattr(self.embedding_store, "is_healthy", True))
try:
query_embedding = await self.embedding_store.get_embedding(query)
except Exception as e:
self._mark_embedding_unhealthy(f"search: {type(e).__name__}: {e}")
self._disable_embedding(f"search: {type(e).__name__}: {e}")
return []
if query_embedding is None:
return []
if not self._embedding_dim_matches(query_embedding):
self._mark_embedding_unhealthy(
self._disable_embedding(
f"search: query embedding dimension {len(query_embedding)} != {self.embedding_store.dimensions}",
)
return []
await self._recover_after_real_request(was_healthy)
top: list[tuple[float, int, FileChunk]] = []
candidates: list[FileChunk] = []

View file

@ -176,10 +176,6 @@ class ZvecLocalFileStore(LocalFileStore):
]
self._upsert_docs(to_add)
async def _reset_vector_index(self) -> None:
"""Discard all vectors before rebuilding a changed vector space."""
self._collection = self._create_collection()
# -- maintenance ------------------------------------------------------
async def optimize_index(self) -> None:
@ -319,9 +315,9 @@ class ZvecLocalFileStore(LocalFileStore):
f"elapsed={time.monotonic() - started_at:.3f}s",
)
async def _dump_owned_state(self) -> None:
"""Persist chunks and zvec state, excluding dependency snapshots."""
await super()._dump_owned_state()
async def dump(self) -> None:
"""Persist chunks JSONL via the parent, then flush zvec and write the sidecar."""
await super().dump()
if self._collection is None or self.embedding_store is None:
return
try:
@ -416,25 +412,20 @@ class ZvecLocalFileStore(LocalFileStore):
# -- search -----------------------------------------------------------
async def vector_search(self, query: str, limit: int, search_filter: dict) -> list[FileChunk]:
if self.embedding_store is None or self._embedding_rebuild_pending or not query or limit <= 0:
return []
index_empty = self._collection is None or not self._indexed_ids
if index_empty and getattr(self.embedding_store, "is_healthy", True):
if self.embedding_store is None or not query or limit <= 0 or self._collection is None or not self._indexed_ids:
return []
query_embedding = None
was_healthy = bool(getattr(self.embedding_store, "is_healthy", True))
try:
query_embedding = await self.embedding_store.get_embedding(query)
except Exception as e:
self._mark_embedding_unhealthy(f"search: {type(e).__name__}: {e}")
self._disable_embedding(f"search: {type(e).__name__}: {e}")
if query_embedding is None or not self._embedding_dim_matches(query_embedding):
if query_embedding is not None:
self._mark_embedding_unhealthy(
self._disable_embedding(
f"search: query embedding dimension {len(query_embedding)} != {self.embedding_store.dimensions}",
)
return []
await self._recover_after_real_request(was_healthy)
# get_embedding above yielded control; a concurrent clear() may have
# swapped or dropped the collection. Re-read before dereferencing.

View file

@ -47,7 +47,7 @@ class BaseJob(BaseComponent):
config = raw if isinstance(raw, ComponentConfig) else ComponentConfig(**raw)
if not config.backend:
raise ValueError("Step is missing the required 'backend' field")
step_cls = self.app_context.registry.get(ComponentEnum.STEP, config.backend)
step_cls = R.get(ComponentEnum.STEP, config.backend)
if not step_cls:
raise ValueError(f"Unregistered backend '{config.backend}' of type '{ComponentEnum.STEP}'")
params = config.model_dump()

View file

@ -1,17 +1,16 @@
"""HTTP service: expose jobs through JSON/SSE endpoints and MCP tools."""
"""HTTP service: exposes jobs as FastAPI endpoints (JSON, or SSE for stream jobs)."""
import asyncio
import warnings
from collections.abc import AsyncGenerator
from pathlib import Path
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING
import uvicorn
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, StreamingResponse
from fastapi.staticfiles import StaticFiles
from starlette.routing import Route
from .base_service import BaseService
from ..component_registry import R
@ -19,7 +18,6 @@ from ..job import BaseJob, StreamJob
from ...constants import REME_DEFAULT_HOST, REME_DEFAULT_PORT
from ...schema import Request, Response
from ...utils import execute_stream_task, resolve_web_static_dir
from .mcp_tools import add_mcp_job
if TYPE_CHECKING:
from ...application import Application
@ -35,7 +33,7 @@ _WEBSOCKET_DEPRECATION_PATTERNS = (
@R.register("http")
class HttpService(BaseService):
"""Expose jobs through JSON/SSE endpoints and streamable HTTP MCP."""
"""Map non-stream jobs to JSON POST endpoints and StreamJobs to SSE endpoints."""
def __init__(
self,
@ -43,11 +41,6 @@ class HttpService(BaseService):
port: int = REME_DEFAULT_PORT,
web_enabled: bool = True,
web_static_dir: str | None = None,
mcp_enabled: bool = True,
mcp_path: str = "/mcp",
mcp_stateless_http: bool = False,
injected_job_kwargs: dict[str, Any] | None = None,
tool_error_on_failure: bool = False,
**kwargs,
):
super().__init__(**kwargs)
@ -55,34 +48,14 @@ class HttpService(BaseService):
self.port: int = port
self.web_enabled = web_enabled
self.web_static_dir = web_static_dir
self.mcp_enabled = mcp_enabled
self.mcp_path = self._validate_mcp_path(mcp_path)
self.mcp_stateless_http = mcp_stateless_http
self.injected_job_kwargs = dict(injected_job_kwargs or {})
self.tool_error_on_failure = tool_error_on_failure
self.mcp_server = None
self.mcp_app = None
# ----- BaseService contract ------------------------------------------
def build_service(self, app: "Application") -> None:
"""Create one FastAPI app containing JSON/SSE and optional MCP routes."""
lifespan = self._lifespan(app, self.host, self.port)
if self.mcp_enabled:
from fastmcp import FastMCP
from fastmcp.utilities.lifespan import combine_lifespans
self.mcp_server = FastMCP(name=app.config.app_name)
self.mcp_app = self.mcp_server.http_app(
path=self.mcp_path,
transport="streamable-http",
stateless_http=self.mcp_stateless_http,
)
lifespan = combine_lifespans(lifespan, self.mcp_app.lifespan)
"""Create the FastAPI app with permissive CORS and an app-managed lifespan."""
self.service = FastAPI(
title=app.config.app_name,
lifespan=lifespan,
lifespan=self._lifespan(app, self.host, self.port),
)
cors_origins = ["*"]
self.service.add_middleware(
@ -92,60 +65,19 @@ class HttpService(BaseService):
allow_methods=["*"],
allow_headers=["*"],
)
if self.mcp_app is not None:
# Forward the exact path to the complete FastMCP ASGI app. Copying
# only its routes would bypass its middleware and application state;
# mounting it would make the trailing-slash path canonical instead.
self.service.router.routes.append(
Route(
self.mcp_path,
endpoint=self.mcp_app,
include_in_schema=False,
),
)
def add_jobs(self, app: "Application") -> None:
"""Validate reserved routes before the shared tolerant registration loop."""
if self.mcp_enabled:
conflicts = sorted(
job.name
for name, job in app.context.jobs.items()
if job.enable_serve and (self.jobs is None or name in self.jobs) and f"/{job.name}" == self.mcp_path
)
if conflicts:
names = ", ".join(conflicts)
raise ValueError(
f"Job name conflicts with the MCP endpoint {self.mcp_path!r}: {names}",
)
super().add_jobs(app)
def add_job(self, job: BaseJob) -> bool:
"""Register HTTP routes for every job and MCP tools for non-stream jobs."""
if self.mcp_enabled and f"/{job.name}" == self.mcp_path:
raise ValueError(
f"Job name '{job.name}' conflicts with the MCP endpoint {self.mcp_path!r}",
)
"""Dispatch to streaming or non-streaming registration based on job type."""
if isinstance(job, StreamJob):
self._add_stream_job(job)
else:
self._add_json_job(job)
if self.mcp_server is not None:
add_mcp_job(
self.mcp_server,
job,
injected_job_kwargs=self.injected_job_kwargs,
tool_error_on_failure=self.tool_error_on_failure,
)
return True
def start_service(self, app: "Application") -> None:
"""Run uvicorn, suppressing unrelated websocket deprecation noise."""
for pattern in _WEBSOCKET_DEPRECATION_PATTERNS:
warnings.filterwarnings(
"ignore",
category=DeprecationWarning,
message=pattern,
)
warnings.filterwarnings("ignore", category=DeprecationWarning, message=pattern)
uvicorn.run(self.service, host=self.host, port=self.port, **self.kwargs)
def finalize_service(self, app: "Application") -> None:
@ -196,23 +128,6 @@ class HttpService(BaseService):
# ----- Endpoint factories --------------------------------------------
@staticmethod
def _validate_mcp_path(path: str) -> str:
"""Return a canonical, non-reserved absolute path for the MCP endpoint."""
if not path.startswith("/") or path == "/" or path.endswith("/"):
raise ValueError(
"mcp_path must start with '/', must not be '/', and must not end with '/'",
)
if "//" in path or any(segment in {".", ".."} for segment in path.split("/")):
raise ValueError("mcp_path must use non-empty literal path segments")
if any(char in path for char in "{}?#%\\") or any(
char.isspace() or ord(char) < 32 or ord(char) == 127 for char in path
):
raise ValueError("mcp_path must be a literal URL path without route, query, or fragment syntax")
if path in {"/assets", "/docs", "/redoc", "/openapi.json"}:
raise ValueError(f"mcp_path conflicts with reserved HTTP path {path!r}")
return path
def _add_json_job(self, job: BaseJob) -> None:
"""Register a job as POST /{job.name} returning a JSON Response."""

View file

@ -4,9 +4,8 @@ from typing import TYPE_CHECKING, Any
from .base_service import BaseService
from ..component_registry import R
from ..job import BaseJob
from ..job import BaseJob, StreamJob
from ...constants import REME_DEFAULT_HOST, REME_DEFAULT_PORT
from .mcp_tools import add_mcp_job
if TYPE_CHECKING:
from fastmcp.server.server import Transport
@ -46,12 +45,41 @@ class MCPService(BaseService):
def add_job(self, job: BaseJob) -> bool:
"""Register a non-stream job as an MCP tool; StreamJobs are unsupported."""
return add_mcp_job(
self.service,
job,
injected_job_kwargs=self.injected_job_kwargs,
tool_error_on_failure=self.tool_error_on_failure,
from fastmcp.exceptions import ToolError
from fastmcp.tools import FunctionTool
if isinstance(job, StreamJob):
return False
async def execute_tool(**kwargs):
conflicts = sorted(self.injected_job_kwargs.keys() & kwargs.keys())
if conflicts:
names = ", ".join(conflicts)
raise ToolError(f"{names} injected by the MCP server and cannot be provided by the caller")
kwargs.update(self.injected_job_kwargs)
response = await job(**kwargs)
if self.tool_error_on_failure and not response.success:
raise ToolError(str(response.answer))
return response.answer
parameters = dict(job.parameters or {})
injected_names = self.injected_job_kwargs.keys()
if "properties" in parameters:
parameters["properties"] = {
name: schema for name, schema in parameters["properties"].items() if name not in injected_names
}
if "required" in parameters:
parameters["required"] = [name for name in parameters["required"] if name not in injected_names]
self.service.add_tool(
FunctionTool(
name=job.name,
description=job.description,
fn=execute_tool,
parameters=parameters,
),
)
return True
def start_service(self, app: "Application") -> None:
"""Run the MCP server; bind host/port only for network transports."""
@ -59,8 +87,4 @@ class MCPService(BaseService):
if self.transport != "stdio":
transport_kwargs["host"] = self.host
transport_kwargs["port"] = self.port
self.service.run(
transport=self.transport,
show_banner=False,
**transport_kwargs,
)
self.service.run(transport=self.transport, show_banner=False, **transport_kwargs)

View file

@ -1,52 +0,0 @@
"""Shared MCP tool registration for services that expose ReMe jobs."""
from typing import Any
from ..job import BaseJob, StreamJob
def add_mcp_job(
server: Any,
job: BaseJob,
*,
injected_job_kwargs: dict[str, Any],
tool_error_on_failure: bool,
) -> bool:
"""Register a non-stream job as an MCP tool on ``server``."""
from fastmcp.exceptions import ToolError
from fastmcp.tools import FunctionTool
if isinstance(job, StreamJob):
return False
async def execute_tool(**kwargs):
conflicts = sorted(injected_job_kwargs.keys() & kwargs.keys())
if conflicts:
names = ", ".join(conflicts)
raise ToolError(
f"{names} injected by the MCP server and cannot be provided by the caller",
)
kwargs.update(injected_job_kwargs)
response = await job(**kwargs)
if tool_error_on_failure and not response.success:
raise ToolError(str(response.answer))
return response.answer
parameters = dict(job.parameters or {})
injected_names = injected_job_kwargs.keys()
if "properties" in parameters:
parameters["properties"] = {
name: schema for name, schema in parameters["properties"].items() if name not in injected_names
}
if "required" in parameters:
parameters["required"] = [name for name in parameters["required"] if name not in injected_names]
server.add_tool(
FunctionTool(
name=job.name,
description=job.description,
fn=execute_tool,
parameters=parameters,
),
)
return True

View file

@ -1,19 +1,8 @@
"""Config"""
from .config_parser import (
deep_merge_config,
expand_env_vars,
parse_action,
parse_args,
parse_kwargs,
resolve_app_config,
)
from .config_parser import parse_args, resolve_app_config
__all__ = [
"deep_merge_config",
"expand_env_vars",
"parse_action",
"parse_args",
"parse_kwargs",
"resolve_app_config",
]

Some files were not shown because too many files have changed in this diff Show more