diff --git a/README.md b/README.md index b9d1180a..e95df6f3 100644 --- a/README.md +++ b/README.md @@ -190,6 +190,7 @@ These Markdown guides cover the main user workflows and the runtime contracts im | Guide | What you will learn | |-------|---------------------| | [Quick Start](docs/en/quick_start.md) | Install ReMe, start the service, and run the first file and memory operations. | +| [Plugin Management](docs/en/plugin_management.md) | Install, inspect, validate, enable, and uninstall local ReMe plugins. | | [Memory as File](docs/en/memory_as_file.md) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. | | [Auto Memory](docs/en/auto_memory.md) | Preserve source conversations and distill reusable daily memory cards. | | [Auto Resource](docs/en/auto_resource.md) | Import supported text resources and turn them into source-linked daily cards. | diff --git a/README_ZH.md b/README_ZH.md index 9d59344d..81279a7b 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -183,6 +183,7 @@ ReMe 会把 Agent 记忆保存为可读的 Markdown。 | 文档 | 主要内容 | |------|----------| | [快速开始](docs/zh/quick_start.md) | 安装 ReMe、启动服务，并执行首次文件和记忆操作。 | +| [插件管理](docs/zh/plugin_management.md) | 安装、查看、校验、启用和卸载本地 ReMe 插件。 | | [Memory as File](docs/zh/memory_as_file.md) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 | | [Auto Memory](docs/zh/auto_memory.md) | 保留过滤后的对话来源记录，并提炼可复用的 daily 记忆卡片。 | | [Auto Resource](docs/zh/auto_resource.md) | 导入支持的文本资料，转换为可追溯来源的 daily 卡片。 | diff --git a/docs/en/framework.md b/docs/en/framework.md index 3b80ce47..c0f0ce52 100644 --- a/docs/en/framework.md +++ b/docs/en/framework.md @@ -231,14 +231,31 @@ They are configured under `components` and participate in the same dependency or Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by -`plugins` in the resolved configuration. A plugin -is exposed through the `reme.plugins` Python entry-point group and returns a declarative `reme.plugin.Plugin` containing -its named backend classes and default configuration. Plugin registration therefore stays local to one application; +`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point +group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to +`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The +entry-point name is the plugin's identity. +Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override. +Plugin registration therefore stays local to one application; duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other. -Installed plugins may also expose named configuration files through `reme.configs`. Configuration can use `extends` to -inherit another built-in, plugin, or file-based configuration. The [Auto Fin plugin](../../plugins/auto-fin/README.md) -is the complete packaging example. +The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration +files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. The +[Auto Fin plugin](../../plugins/auto-fin/README.md) is the current packaging example. + +Plugin packages are managed locally and remain separate from per-application activation: + +```bash +reme plugins list +reme plugins install reme-auto-fin +reme plugins show auto-fin +reme plugins validate auto-fin +reme plugins uninstall auto-fin + +reme start plugins='["auto-fin"]' +``` + +These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service. ### 4.3 Component.bind diff --git a/docs/en/plugin_management.md b/docs/en/plugin_management.md new file mode 100644 index 00000000..9575aae8 --- /dev/null +++ b/docs/en/plugin_management.md @@ -0,0 +1,228 @@ +# Plugin Management + +ReMe plugins are ordinary Python distributions discovered through the `reme.plugins` entry-point group. Installing a +plugin makes it available to the current Python environment; it does not enable the plugin in every ReMe application. + +Keep these two operations separate: + +```text +reme plugins install ... install a package into the current Python environment +plugins: [auto-fin] enable an installed plugin for one Application +``` + +Plugin package management is local-only. It does not run through a ReMe HTTP or MCP service and never edits application +configuration files automatically. + +A typical plugin workflow has three stages: + +1. Install ReMe and the plugin distribution. +2. Configure the plugin's runtime environment as described in the + [ReMe environment-variable guide](../../README.md#environment-variables). +3. Start an Application with the plugin explicitly enabled, for example + `reme start plugins='["auto-fin"]'`. + +## List installed plugins + +```bash +reme plugins list +``` + +The table shows the plugin entry-point name, Python distribution, version, and plugin contract: + +```text +PLUGIN DISTRIBUTION VERSION FORMAT +-------- ------------- ------- -------- +auto-fin reme-auto-fin 0.1.0 manifest +``` + +`manifest` plugins use the current package-level `plugin.yaml` contract. `legacy` plugins use the compatible Python +descriptor contract. + +A manifest separates backend registration from application configuration: + +```yaml +backends: + example_step: example_plugin.steps:ExampleStep + +application_defaults: + jobs: + example: + backend: base + steps: + - backend: example_step +``` + +`application_defaults` is a partial `ApplicationConfig`. It is kept below the manifest's `backends` namespace because +backend import declarations are part of plugin discovery and are not application configuration. + +Use JSON when another local tool needs structured output: + +```bash +reme plugins list --json +``` + +To compare installed plugins with one application config: + +```bash +reme plugins list --config daily_cookbook +``` + +The optional `ENABLED` column reflects only the `plugins` list resolved from that config. A command-line override used +by another running process is not a global enable state. + +## Install a plugin package + +Install a published distribution: + +```bash +reme plugins install reme-auto-fin +``` + +Install or upgrade a pinned version: + +```bash +reme plugins install 'reme-auto-fin==0.1.0' +reme plugins install reme-auto-fin --upgrade +``` + +Install a local plugin project: + +```bash +reme plugins install ./plugins/auto-fin +``` + +Use editable mode while developing it: + +```bash +reme plugins install ./plugins/auto-fin --editable +``` + +ReMe invokes pip through the same Python interpreter that runs the `reme` command. Pip remains responsible for package +resolution, downloads, dependency changes, and build execution. Install only packages and local projects you trust. + +After installation, confirm the discovered plugin name: + +```bash +reme plugins list +reme plugins validate auto-fin +``` + +## Inspect a plugin + +```bash +reme plugins show auto-fin +``` + +For a manifest plugin, the result includes its registered backend names and default Job names. JSON output is also +available: + +```bash +reme plugins show auto-fin --json +``` + +`show` identifies the package contract without constructing a ReMe Application. + +## Validate a plugin + +Validate an installed plugin: + +```bash +reme plugins validate auto-fin +``` + +Validate a local project before installation: + +```bash +reme plugins validate ./plugins/auto-fin +``` + +Validation checks the entry point, `plugin.yaml`, backend imports and component types, registry collisions, merged +`application_defaults`, and the resulting `ApplicationConfig`. Validation imports plugin backend modules, so run it +only for trusted code. + +## Enable a plugin in a service + +Installation alone does not load plugin code into an Application. Enable plugins explicitly in configuration: + +```yaml +plugins: + - auto-fin +``` + +Or add them for one service launch: + +```bash +reme start plugins='["auto-fin"]' +``` + +When `config` is omitted, ReMe loads `default.yaml`. The plugin's `application_defaults` are merged below that config, +so explicit config values and CLI overrides win. This mapping is an `ApplicationConfig` fragment, not a separate +configuration schema. The plugin backends are registered only in that Application's local registry. + +After the default HTTP service starts, access plugin Jobs through ReMe's CLI client or HTTP: + +```bash +reme auto_fin topics="黄金,AI,存储芯片" +``` + +```bash +curl -s http://127.0.0.1:2333/auto_fin \ + -H 'Content-Type: application/json' \ + -d '{"topics":"黄金,AI,存储芯片"}' +``` + +When the application uses an MCP service, service-enabled plugin Jobs appear as MCP tools instead. + +To add the plugin to another application config, select it explicitly: + +```bash +reme start config=daily_cookbook plugins='["auto-fin"]' +``` + +## Uninstall a plugin + +Use the plugin entry-point name, not necessarily the distribution name: + +```bash +reme plugins uninstall auto-fin +``` + +Skip pip's confirmation prompt when needed: + +```bash +reme plugins uninstall auto-fin --yes +``` + +ReMe resolves `auto-fin` to the distribution that provides it, such as `reme-auto-fin`. If one distribution provides +multiple plugin entry points, the command lists the other plugins that will also be removed. + +Uninstallation does not rewrite user configuration. Remove the plugin from relevant `plugins` lists yourself; +otherwise the next Application startup fails explicitly because the configured plugin is no longer installed. Restart +already-running ReMe processes after installing, upgrading, or uninstalling packages. + +## Troubleshooting + +### Plugin is installed but unavailable + +Check that the `reme` command and pip package share one Python interpreter: + +```bash +reme plugins list +python -c 'import sys; print(sys.executable)' +``` + +Using `reme plugins install` avoids the most common interpreter mismatch because it runs `python -m pip` with ReMe's +own interpreter. + +### Plugin is installed but not loaded + +Add its entry-point name to the Application's `plugins` list. ReMe intentionally has no global enable/disable state. + +### Startup reports that the plugin is not installed + +The active config still enables a missing plugin. Reinstall it or remove the corresponding name from `plugins`. + +### Changes are not visible in a running service + +Plugin discovery and backend registration happen during Application construction. Restart the service after changing +installed packages. diff --git a/docs/zh/framework.md b/docs/zh/framework.md index cb955261..619eb145 100644 --- a/docs/zh/framework.md +++ b/docs/zh/framework.md @@ -221,12 +221,29 @@ class VersionStep(BaseStep): 内置实现通过 package import 填充内置注册表；bootstrap 完成后 ReMe 会冻结这个模板，并为每个 `Application` 创建可写副本。 运行期代码通过当前 Application 的注册表解析 backend，不能修改进程级模板。随后只加载最终配置中 `plugins` 明确启用的已安装插件。 -插件通过 Python `reme.plugins` entry-point group 暴露，返回声明式 `reme.plugin.Plugin`，其中包含命名 -backend class 和默认配置。插件注册因此只影响当前 Application；两个插件提供相同 `(component_type, backend)` 时会在装配阶段失败， +插件通过 Python `reme.plugins` entry-point group 暴露其 package。package 内的 `plugin.yaml` 只有两个可选 mapping： +`backends` 将注册名映射到 `module:Class`，`application_defaults` 提供低优先级的 `ApplicationConfig` 配置片段。 +entry-point 名称就是插件标识；使用 +应用配置的 `plugins` 列表或 CLI 的 `plugins=[...]` override 显式启用插件。插件注册因此只影响当前 Application；两个插件提供相同 +`(component_type, backend)` 时会在装配阶段失败， 不会互相覆盖。 -插件还可以通过 `reme.configs` 暴露命名配置；配置的 `extends` 可以继承内置配置、插件配置或文件配置。完整打包示例见 -[Auto Fin 插件](../../plugins/auto-fin/README_ZH.md)。 +迁移期间仍兼容旧的 Python `Plugin` descriptor 和 `reme.configs` entry point。配置的 `extends` 可以继承内置配置、 +旧插件配置或文件配置。当前完整打包示例见 [Auto Fin 插件](../../plugins/auto-fin/README_ZH.md)。 + +插件包的本地管理与单个应用是否启用插件相互独立： + +```bash +reme plugins list +reme plugins install reme-auto-fin +reme plugins show auto-fin +reme plugins validate auto-fin +reme plugins uninstall auto-fin + +reme start plugins='["auto-fin"]' +``` + +这些管理命令使用当前 Python 解释器对应的 pip，不通过 HTTP 或 MCP service 执行。 ### 4.3 Component.bind diff --git a/docs/zh/plugin_management.md b/docs/zh/plugin_management.md new file mode 100644 index 00000000..0e1abf40 --- /dev/null +++ b/docs/zh/plugin_management.md @@ -0,0 +1,219 @@ +# 插件管理 + +ReMe 插件是通过 `reme.plugins` entry-point group 发现的普通 Python distribution。安装插件只表示它在当前 Python +环境中可用，并不会让所有 ReMe Application 自动启用该插件。 + +需要区分两个操作： + +```text +reme plugins install ... 将插件包安装到当前 Python 环境 +plugins: [auto-fin] 为一个 Application 启用已安装插件 +``` + +插件包管理仅在本地 CLI 执行，不经过 ReMe HTTP 或 MCP service，也不会自动修改应用配置文件。 + +典型的插件使用流程分为三个阶段： + +1. 安装 ReMe 和插件 distribution。 +2. 按照 [ReMe 环境变量说明](../../README_ZH.md#环境变量)配置插件运行所需的环境变量。 +3. 启动 Application 时显式启用插件，例如 `reme start plugins='["auto-fin"]'`。 + +## 查看已安装插件 + +```bash +reme plugins list +``` + +输出包含插件 entry-point 名称、Python distribution、版本和插件契约： + +```text +PLUGIN DISTRIBUTION VERSION FORMAT +-------- ------------- ------- -------- +auto-fin reme-auto-fin 0.1.0 manifest +``` + +`manifest` 表示插件使用当前的 package-level `plugin.yaml` 契约；`legacy` 表示插件使用仍然兼容的 Python descriptor +契约。 + +manifest 将 backend 注册与应用配置分开： + +```yaml +backends: + example_step: example_plugin.steps:ExampleStep + +application_defaults: + jobs: + example: + backend: base + steps: + - backend: example_step +``` + +`application_defaults` 是一段不完整的 `ApplicationConfig`。它与 manifest 的 `backends` 命名空间分开，因为 backend +导入声明属于插件发现协议，并不是应用配置。 + +本地工具需要结构化结果时可以使用 JSON： + +```bash +reme plugins list --json +``` + +对照某个应用配置查看启用状态： + +```bash +reme plugins list --config daily_cookbook +``` + +可选的 `ENABLED` 列只反映该配置解析出的 `plugins` 列表。其他运行中进程使用的 CLI override 不是全局启用状态。 + +## 安装插件包 + +安装已发布的 distribution： + +```bash +reme plugins install reme-auto-fin +``` + +安装指定版本或升级： + +```bash +reme plugins install 'reme-auto-fin==0.1.0' +reme plugins install reme-auto-fin --upgrade +``` + +安装本地插件项目： + +```bash +reme plugins install ./plugins/auto-fin +``` + +开发插件时使用 editable 模式： + +```bash +reme plugins install ./plugins/auto-fin --editable +``` + +ReMe 会通过运行 `reme` 命令的同一个 Python 解释器调用 pip。包解析、下载、依赖变更和构建执行仍由 pip 负责。请只安装 +可信的包和本地项目。 + +安装后确认 ReMe 实际发现的插件名： + +```bash +reme plugins list +reme plugins validate auto-fin +``` + +## 查看插件详情 + +```bash +reme plugins show auto-fin +``` + +对于 manifest 插件，结果包含注册的 backend 名称和默认 Job 名称。也可以输出 JSON： + +```bash +reme plugins show auto-fin --json +``` + +`show` 只检查包契约，不构造 ReMe Application。 + +## 校验插件 + +校验已安装插件： + +```bash +reme plugins validate auto-fin +``` + +安装前校验本地插件项目： + +```bash +reme plugins validate ./plugins/auto-fin +``` + +校验范围包括 entry point、`plugin.yaml`、backend 导入和组件类型、registry 冲突、`application_defaults` 合并以及最终的 +`ApplicationConfig`。校验过程会导入插件 backend 模块，因此只能对可信代码执行。 + +## 在服务中启用插件 + +只安装插件不会将插件代码加载到 Application。需要在配置中显式启用： + +```yaml +plugins: + - auto-fin +``` + +也可以只为本次服务启动追加插件： + +```bash +reme start plugins='["auto-fin"]' +``` + +未传入 `config` 时，ReMe 加载 `default.yaml`。插件的 `application_defaults` 合并在该配置之下，因此显式配置和 CLI +override 优先。这个 mapping 是 `ApplicationConfig` 配置片段，并不是另一套配置 schema。插件 backend 只注册到该 +Application 的局部 registry。 + +默认 HTTP service 启动后，可以通过 ReMe CLI client 或 HTTP 访问插件 Job： + +```bash +reme auto_fin topics="黄金,AI,存储芯片" +``` + +```bash +curl -s http://127.0.0.1:2333/auto_fin \ + -H 'Content-Type: application/json' \ + -d '{"topics":"黄金,AI,存储芯片"}' +``` + +当应用使用 MCP service 时，允许对外服务的插件 Job 会显示为 MCP tool。 + +如果需要将插件叠加到其他应用配置，则显式选择该配置： + +```bash +reme start config=daily_cookbook plugins='["auto-fin"]' +``` + +## 卸载插件 + +这里使用插件 entry-point 名称，它不一定等于 distribution 名称： + +```bash +reme plugins uninstall auto-fin +``` + +需要跳过 pip 确认时： + +```bash +reme plugins uninstall auto-fin --yes +``` + +ReMe 会将 `auto-fin` 解析为提供它的 distribution，例如 `reme-auto-fin`。如果一个 distribution 提供多个插件 entry +point，命令会列出同时被移除的其他插件。 + +卸载不会重写用户配置。请自行从相关 `plugins` 列表中删除插件，否则下一次启动 Application 时会因为配置的插件未安装而明确 +失败。安装、升级或卸载包后，需要重启已经运行的 ReMe 进程。 + +## 常见问题 + +### 插件已经安装，但 ReMe 找不到 + +检查 `reme` 命令与安装插件使用的 pip 是否属于同一个 Python 解释器： + +```bash +reme plugins list +python -c 'import sys; print(sys.executable)' +``` + +使用 `reme plugins install` 可以避免最常见的解释器不一致问题，因为它通过 ReMe 自己的解释器运行 `python -m pip`。 + +### 插件已经安装，但没有加载 + +将插件 entry-point 名称加入 Application 的 `plugins` 列表。ReMe 刻意不提供全局 enable/disable 状态。 + +### 启动时报插件未安装 + +当前配置仍然启用了缺失插件。请重新安装插件，或者从 `plugins` 中删除对应名称。 + +### 运行中的服务看不到插件变化 + +插件发现和 backend 注册发生在 Application 构造阶段。修改已安装包后需要重启服务。 diff --git a/github-pages/scripts/generate-content.mjs b/github-pages/scripts/generate-content.mjs index 67506006..de8a17d4 100644 --- a/github-pages/scripts/generate-content.mjs +++ b/github-pages/scripts/generate-content.mjs @@ -8,6 +8,7 @@ const outputDir = path.join(siteDir, ".generated", "content"); const topicOrder = [ "quick_start", + "plugin_management", "memory_as_file", "memory_search", "auto_memory", @@ -23,6 +24,7 @@ const topicOrder = [ const groups = { quick_start: "start", + plugin_management: "start", memory_as_file: "fundamentals", memory_search: "fundamentals", auto_memory: "automation", @@ -38,6 +40,7 @@ const groups = { const localizedTitles = { quick_start: { zh: "快速开始", en: "Quick Start" }, + plugin_management: { zh: "插件管理", en: "Plugin Management" }, memory_as_file: { zh: "文件即记忆", en: "Memory as File" }, memory_search: { zh: "记忆检索", en: "Memory Search" }, auto_memory: { zh: "自动记忆", en: "Auto Memory" }, diff --git a/plugins/README.md b/plugins/README.md index bfe2086f..5f484bfa 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -1,6 +1,24 @@ # ReMe Plugins -This directory contains installable extensions of ReMe itself. A plugin may contribute components, steps, jobs, and -configuration through the `reme.plugins` and `reme.configs` Python entry-point groups. +This directory contains installable extensions of ReMe itself. A plugin exposes its package through the `reme.plugins` +Python entry-point group and declares backend classes plus a low-priority `ApplicationConfig` fragment under +`application_defaults` in that package's `plugin.yaml`. + +Manage plugin packages with the local CLI: + +```bash +reme plugins list +reme plugins show auto-fin +reme plugins install reme-auto-fin +reme plugins install ./plugins/auto-fin --editable +reme plugins validate auto-fin +reme plugins uninstall auto-fin +``` + +Installation and activation are separate. Enable an installed plugin for one application through its config or CLI: + +```bash +reme start plugins='["auto-fin"]' +``` Adapters for external agent hosts belong in [`../integrations`](../integrations/README.md). diff --git a/plugins/auto-fin/README.md b/plugins/auto-fin/README.md index 277b9b9b..d58c029b 100644 --- a/plugins/auto-fin/README.md +++ b/plugins/auto-fin/README.md @@ -5,32 +5,64 @@ Auto Fin fetches a rolling window of CLS telegraph news (24 hours by default), selects items related to configured topics, searches ReMe for useful historical context, and writes one Chinese Markdown report with validated wikilinks. Current news and topic selection stay in runtime memory; only the final report becomes durable memory. This directory -is an independent Python distribution. Its `reme.plugins` entry point contributes the three Step backends and their Job -configuration; its `reme.configs` entry point exposes the runnable `auto-fin` configuration. +is an independent Python distribution. Its single `reme.plugins` entry point exposes a `plugin.yaml` containing the +three Step backends and their Job configuration under `application_defaults`. Enable the installed plugin explicitly +through `plugins=["auto-fin"]`. > Auto Fin has no reliable market-price feed. It does not calculate returns, targets, or entry points and is not > investment advice. ## Quick start -```bash -python -m pip install "reme-ai[core]>=0.4.1.8" reme-auto-fin -export LLM_API_KEY="your-api-key" -export LLM_MODEL_NAME="qwen3.7-plus" -export LLM_BASE_URL="https://your-provider.example/v1" -reme start config=auto-fin job=auto_fin -``` - -`LLM_MODEL_NAME` defaults to `qwen3.7-plus`. There is no built-in `LLM_BASE_URL`, so set the OpenAI-compatible endpoint -required by the selected provider. - -The default topics are `黄金,机器人,半导体`. Override them per run: +### 1. Install ReMe and Auto Fin ```bash -reme start config=auto-fin job=auto_fin topics="黄金,AI,存储芯片" +python -m pip install "reme-ai[core]>=0.4.1.8" +reme plugins install reme-auto-fin ``` -An empty value also uses the defaults. +### 2. Configure the model environment + +Configure the LLM environment variables as described in the +[ReMe README](../../README.md#environment-variables). Other compatible models and providers can also be used. + +### 3. Start ReMe with the plugin + +```bash +reme start plugins='["auto-fin"]' +``` + +With no explicit `config`, ReMe loads `default.yaml` and adds the plugin to that service. + +From another terminal, call the running HTTP service through ReMe's CLI client: + +```bash +reme auto_fin topics="黄金,AI,存储芯片" +``` + +Or call its HTTP endpoint directly: + +```bash +curl -s http://127.0.0.1:2333/auto_fin \ + -H 'Content-Type: application/json' \ + -d '{"topics":"黄金,AI,存储芯片"}' +``` + +When enabled on an MCP service, the same Job is exposed as the `auto_fin` MCP tool. The default topics are +`黄金,机器人,半导体`; an empty value also uses these defaults. + +To host the same application as an MCP service instead: + +```bash +reme start plugins='["auto-fin"]' \ + service.backend=mcp service.transport=streamable-http +``` + +To add Auto Fin to another application instead, select that config explicitly, for example: + +```bash +reme start config=daily_cookbook plugins='["auto-fin"]' +``` ## Pipeline @@ -76,7 +108,7 @@ refreshes the daily index. No JSONL, intermediate Markdown, or structured Agent | `request_interval` | `10` | Minimum delay in seconds after every CLS request attempt; may be zero | | `max_retries` | `3` | Maximum attempts for each CLS page request; must be at least one | -The plugin-provided schedules run daily at 09:30, 11:30, and 18:00 in `Asia/Shanghai`. +The three plugin cron Jobs start with the application and run daily at 09:30, 11:30, and 18:00 in `Asia/Shanghai`. ## Output diff --git a/plugins/auto-fin/README_ZH.md b/plugins/auto-fin/README_ZH.md index 4394726f..76527b98 100644 --- a/plugins/auto-fin/README_ZH.md +++ b/plugins/auto-fin/README_ZH.md @@ -4,30 +4,61 @@ Auto Fin 自动拉取一个滚动时间窗口内的财联社电报（默认 24 小时），按配置 topics 筛选相关新闻，搜索 ReMe 中有回顾价值的历史材料，最后写入一份带校验 wikilink 的中文 Markdown 报告。当前新闻和筛选结果只存在于本次运行内存中，只有最终报告成为持久记忆。本目录是一个独立 Python -distribution：`reme.plugins` entry point 贡献三个 Step backend 及其 Job 配置，`reme.configs` entry point 暴露可直接运行的 -`auto-fin` 配置。 +distribution：单个 `reme.plugins` entry point 暴露 `plugin.yaml`，其中声明三个 Step backend，并在 +`application_defaults` 下提供 Job 配置；通过 `plugins=["auto-fin"]` 显式启用这个已安装插件。 > Auto Fin 没有可靠行情数据，不计算收益、目标价或买卖点，也不提供投资建议。 ## 快速开始 -```bash -python -m pip install "reme-ai[core]>=0.4.1.8" reme-auto-fin -export LLM_API_KEY="your-api-key" -export LLM_MODEL_NAME="qwen3.7-plus" -export LLM_BASE_URL="https://your-provider.example/v1" -reme start config=auto-fin job=auto_fin -``` - -`LLM_MODEL_NAME` 默认是 `qwen3.7-plus`。代码没有内置 `LLM_BASE_URL`，请设置所选服务商提供的 OpenAI 兼容 endpoint。 - -默认 topics 是 `黄金,机器人,半导体`。可在运行时覆盖： +### 1. 安装 ReMe 和 Auto Fin ```bash -reme start config=auto-fin job=auto_fin topics="黄金,AI,存储芯片" +python -m pip install "reme-ai[core]>=0.4.1.8" +reme plugins install reme-auto-fin ``` -传入空值也会使用默认 topics。 +### 2. 配置模型环境变量 + +按照 ReMe README 的[环境变量说明](../../README_ZH.md#环境变量)配置 LLM 环境变量，也可以使用其他兼容的模型和服务商。 + +### 3. 带插件启动 ReMe + +```bash +reme start plugins='["auto-fin"]' +``` + +未显式传入 `config` 时，ReMe 会加载 `default.yaml`，并将插件叠加到该服务上。 + +在另一个终端中，通过 ReMe CLI client 调用正在运行的 HTTP 服务： + +```bash +reme auto_fin topics="黄金,AI,存储芯片" +``` + +也可以直接调用 HTTP endpoint： + +```bash +curl -s http://127.0.0.1:2333/auto_fin \ + -H 'Content-Type: application/json' \ + -d '{"topics":"黄金,AI,存储芯片"}' +``` + +在 MCP service 中启用插件时，同一个 Job 会暴露为 `auto_fin` MCP tool。默认 topics 是 `黄金,机器人,半导体`， +传入空值也会使用默认值。 + +如果需要将同一个应用作为 MCP service 启动： + +```bash +reme start plugins='["auto-fin"]' \ + service.backend=mcp service.transport=streamable-http +``` + +如果需要将 Auto Fin 叠加到其他应用，则显式选择相应配置，例如： + +```bash +reme start config=daily_cookbook plugins='["auto-fin"]' +``` ## 流程 @@ -69,7 +100,7 @@ workspace 的 Markdown 目标。不存在、绝对路径、越界、带反斜杠 | `request_interval` | `10` | 每次财联社请求尝试后的最小等待秒数，可设为 0 | | `max_retries` | `3` | 每页财联社请求的最大尝试次数，至少为 1 | -插件提供的定时任务每天按 `Asia/Shanghai` 在 09:30、11:30 和 18:00 运行。 +插件的三个 cron Job 随应用启动，并按 `Asia/Shanghai` 时区在每天 09:30、11:30 和 18:00 运行。 ## 产物 diff --git a/plugins/auto-fin/pyproject.toml b/plugins/auto-fin/pyproject.toml index 8c222bfa..c36291e4 100644 --- a/plugins/auto-fin/pyproject.toml +++ b/plugins/auto-fin/pyproject.toml @@ -11,10 +11,7 @@ dependencies = [ ] [project.entry-points."reme.plugins"] -auto-fin = "reme_auto_fin:plugin" - -[project.entry-points."reme.configs"] -auto-fin = "reme_auto_fin:CONFIG_PATH" +auto-fin = "reme_auto_fin" [tool.setuptools] package-dir = { "" = "src" } diff --git a/plugins/auto-fin/src/reme_auto_fin/__init__.py b/plugins/auto-fin/src/reme_auto_fin/__init__.py index 3e226af3..2c820d0d 100644 --- a/plugins/auto-fin/src/reme_auto_fin/__init__.py +++ b/plugins/auto-fin/src/reme_auto_fin/__init__.py @@ -1,21 +1,14 @@ """Auto Fin news research workflow.""" -from pathlib import Path - from .data import AutoFinDataStep from .merge import AutoFinMergeStep -from .plugin import plugin from .schema import AutoFinReportOutput, AutoFinTopicOutput from .topic import AutoFinTopicStep -CONFIG_PATH = Path(__file__).with_name("config.yaml") - __all__ = [ "AutoFinReportOutput", "AutoFinDataStep", "AutoFinMergeStep", "AutoFinTopicOutput", "AutoFinTopicStep", - "CONFIG_PATH", - "plugin", ] diff --git a/plugins/auto-fin/src/reme_auto_fin/config.yaml b/plugins/auto-fin/src/reme_auto_fin/config.yaml deleted file mode 100644 index 35148071..00000000 --- a/plugins/auto-fin/src/reme_auto_fin/config.yaml +++ /dev/null @@ -1,2 +0,0 @@ -extends: default -plugins: [auto-fin] diff --git a/plugins/auto-fin/src/reme_auto_fin/defaults.yaml b/plugins/auto-fin/src/reme_auto_fin/defaults.yaml deleted file mode 100644 index 14773f4e..00000000 --- a/plugins/auto-fin/src/reme_auto_fin/defaults.yaml +++ /dev/null @@ -1,54 +0,0 @@ -jobs: - auto_fin: - backend: base - description: "Fetch and research recent topic-related CLS news." - parameters: - type: object - properties: - date: - type: string - description: "Current date in YYYY-MM-DD; empty means today in Asia/Shanghai." - default: "" - now: - type: string - description: "Optional simulated current time in ISO 8601 format; empty means the real current time." - default: "" - topics: - type: string - description: "Comma-separated topics used to filter current CLS news." - default: "黄金,机器人,半导体" - window_hours: - type: number - exclusiveMinimum: 0 - description: "Rolling number of hours of CLS news to fetch." - default: 24 - request_interval: - type: number - minimum: 0 - description: "Minimum delay in seconds after each CLS request attempt." - default: 10 - max_retries: - type: integer - minimum: 1 - description: "Maximum attempts for each CLS page request." - default: 3 - steps: &auto_fin_steps - - backend: auto_fin_data_step - - backend: auto_fin_topic_step - - backend: auto_fin_merge_step - job_tools: [memory_search, read] - - auto_fin_0930_cron: - backend: cron - cron: "30 9 * * *" - steps: *auto_fin_steps - - auto_fin_1130_cron: - backend: cron - cron: "30 11 * * *" - steps: *auto_fin_steps - - auto_fin_1800_cron: - backend: cron - cron: "0 18 * * *" - steps: *auto_fin_steps diff --git a/plugins/auto-fin/src/reme_auto_fin/plugin.py b/plugins/auto-fin/src/reme_auto_fin/plugin.py deleted file mode 100644 index 04d929d4..00000000 --- a/plugins/auto-fin/src/reme_auto_fin/plugin.py +++ /dev/null @@ -1,31 +0,0 @@ -"""Auto Fin plugin declaration.""" - -from pathlib import Path - -import yaml - -from reme.plugin import Backend, Plugin - -from .data import AutoFinDataStep -from .merge import AutoFinMergeStep -from .topic import AutoFinTopicStep - - -def _default_config() -> dict: - path = Path(__file__).with_name("defaults.yaml") - with path.open(encoding="utf-8") as stream: - config = yaml.safe_load(stream) - if not isinstance(config, dict): - raise ValueError(f"Auto Fin defaults must be a mapping: {path}") - return config - - -plugin = Plugin( - name="auto-fin", - backends=( - Backend("auto_fin_data_step", AutoFinDataStep), - Backend("auto_fin_topic_step", AutoFinTopicStep), - Backend("auto_fin_merge_step", AutoFinMergeStep), - ), - config=_default_config(), -) diff --git a/plugins/auto-fin/src/reme_auto_fin/plugin.yaml b/plugins/auto-fin/src/reme_auto_fin/plugin.yaml new file mode 100644 index 00000000..2a041d15 --- /dev/null +++ b/plugins/auto-fin/src/reme_auto_fin/plugin.yaml @@ -0,0 +1,60 @@ +backends: + auto_fin_data_step: reme_auto_fin.data:AutoFinDataStep + auto_fin_topic_step: reme_auto_fin.topic:AutoFinTopicStep + auto_fin_merge_step: reme_auto_fin.merge:AutoFinMergeStep + +application_defaults: + jobs: + auto_fin: + backend: base + description: "Fetch and research recent topic-related CLS news." + parameters: + type: object + properties: + date: + type: string + description: "Current date in YYYY-MM-DD; empty means today in Asia/Shanghai." + default: "" + now: + type: string + description: "Optional simulated current time in ISO 8601 format; empty means the real current time." + default: "" + topics: + type: string + description: "Comma-separated topics used to filter current CLS news." + default: "黄金,机器人,半导体" + window_hours: + type: number + exclusiveMinimum: 0 + description: "Rolling number of hours of CLS news to fetch." + default: 24 + request_interval: + type: number + minimum: 0 + description: "Minimum delay in seconds after each CLS request attempt." + default: 10 + max_retries: + type: integer + minimum: 1 + description: "Maximum attempts for each CLS page request." + default: 3 + steps: &auto_fin_steps + - backend: auto_fin_data_step + - backend: auto_fin_topic_step + - backend: auto_fin_merge_step + job_tools: [memory_search, read] + + auto_fin_0930_cron: + backend: cron + cron: "30 9 * * *" + steps: *auto_fin_steps + + auto_fin_1130_cron: + backend: cron + cron: "30 11 * * *" + steps: *auto_fin_steps + + auto_fin_1800_cron: + backend: cron + cron: "0 18 * * *" + steps: *auto_fin_steps diff --git a/plugins/auto-fin/tests/test_auto_fin.py b/plugins/auto-fin/tests/test_auto_fin.py index 6efe5b54..3a785804 100644 --- a/plugins/auto-fin/tests/test_auto_fin.py +++ b/plugins/auto-fin/tests/test_auto_fin.py @@ -7,11 +7,11 @@ from pathlib import Path from zoneinfo import ZoneInfo import pytest +import yaml from reme_auto_fin.base import _plain_text, _write from reme_auto_fin.data import AutoFinDataStep from reme_auto_fin.merge import AutoFinMergeStep -from reme_auto_fin.plugin import plugin from reme_auto_fin.schema import AutoFinReportOutput, AutoFinTopicOutput from reme_auto_fin.topic import AutoFinTopicStep from reme.components import ApplicationContext @@ -19,6 +19,9 @@ from reme.components.agent_wrapper.base_agent_wrapper import BaseAgentWrapper from reme.components.runtime_context import RuntimeContext SHANGHAI = ZoneInfo("Asia/Shanghai") +PLUGIN_MANIFEST = yaml.safe_load( + (Path(__file__).parents[1] / "src" / "reme_auto_fin" / "plugin.yaml").read_text(encoding="utf-8"), +) def _row(news_id: int, value: datetime, title: str = "新闻", content: str = "正文") -> dict: @@ -239,7 +242,8 @@ def test_hybrid_wikilink_normalization_is_conservative_and_failure_safe(tmp_path def test_plugin_config_has_default_topics_and_no_intermediate_index_step(): - job = plugin.config["jobs"]["auto_fin"] + jobs = PLUGIN_MANIFEST["application_defaults"]["jobs"] + job = jobs["auto_fin"] assert job["parameters"]["properties"]["topics"]["default"] == "黄金,机器人,半导体" assert job["parameters"]["properties"]["window_hours"]["default"] == 24 assert job["parameters"]["properties"]["request_interval"]["default"] == 10 @@ -256,8 +260,8 @@ def test_plugin_config_has_default_topics_and_no_intermediate_index_step(): "auto_fin_1130_cron": "30 11 * * *", "auto_fin_1800_cron": "0 18 * * *", }.items(): - assert plugin.config["jobs"][name]["cron"] == schedule - assert plugin.config["jobs"][name]["steps"] == job["steps"] + assert jobs[name]["cron"] == schedule + assert jobs[name]["steps"] == job["steps"] def test_agent_schemas_are_small_and_required(): diff --git a/reme/application.py b/reme/application.py index 57fae78b..fb811a9d 100644 --- a/reme/application.py +++ b/reme/application.py @@ -7,11 +7,11 @@ from pathlib import Path from typing import AsyncGenerator, TypeVar from . import __version__ -from .components import ApplicationContext, BaseComponent, create_application_registry +from .components import ApplicationContext, BaseComponent from .components.job import BackgroundJob, BaseJob, CronJob, StreamJob from .components.service import BaseService from .enumeration import ComponentEnum, ComponentType, component_type_name -from .plugin import PluginManager +from .plugin import resolve_plugin_runtime from .schema import ComponentConfig, Response, StreamChunk from .utils import execute_stream_task, print_logo, get_logger @@ -23,11 +23,8 @@ class Application(BaseComponent): """Wires components from config and runs jobs against them.""" def __init__(self, **kwargs) -> None: - plugin_manager = PluginManager.discover(kwargs.get("plugins") or ()) - resolved = plugin_manager.merge_config(kwargs) - registry = create_application_registry() - plugin_manager.register(registry) - self.context = ApplicationContext(registry=registry, **resolved) + runtime = resolve_plugin_runtime(kwargs) + self.context = ApplicationContext(registry=runtime.registry, **runtime.config) self._started_components: list[BaseComponent] = [] self._setup_workspace_directories() diff --git a/reme/config/__init__.py b/reme/config/__init__.py index ce106a77..642ab984 100644 --- a/reme/config/__init__.py +++ b/reme/config/__init__.py @@ -1,10 +1,19 @@ """Config""" -from .config_parser import deep_merge_config, expand_env_vars, parse_args, resolve_app_config +from .config_parser import ( + deep_merge_config, + expand_env_vars, + parse_action, + parse_args, + parse_kwargs, + resolve_app_config, +) __all__ = [ "deep_merge_config", "expand_env_vars", + "parse_action", "parse_args", + "parse_kwargs", "resolve_app_config", ] diff --git a/reme/config/config_parser.py b/reme/config/config_parser.py index 16cf8a6f..ee06edd5 100644 --- a/reme/config/config_parser.py +++ b/reme/config/config_parser.py @@ -10,13 +10,17 @@ from typing import Any import yaml -from ..entry_point import find_entry_points, load_entry_point, unique_entry_point +from ..entry_point import ( + CONFIG_ENTRY_POINT_GROUP, + find_entry_points, + load_entry_point, + unique_entry_point, +) # Config files are looked up relative to this module's directory _CONFIG_DIR = Path(__file__).parent # Extensions in priority order: yaml > yml > json when stems collide _SUPPORTED_EXTS = (".yaml", ".yml", ".json") -_CONFIG_ENTRY_POINT_GROUP = "reme.configs" _ENV_VAR_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-([^}]*))?}") # Strings like "007" / "00501" must stay as strings, not be coerced to numbers _LEADING_ZERO_RE = re.compile(r"^-?0\d") @@ -145,7 +149,7 @@ def _load_config(name_or_path: str, encoding: str = "utf-8", _stack: tuple[str, raise ValueError(f"Circular config inheritance: {chain}") built_in = _CONFIG_REGISTRY.get(name_or_path) - external_entries = find_entry_points(_CONFIG_ENTRY_POINT_GROUP, name_or_path) + external_entries = find_entry_points(CONFIG_ENTRY_POINT_GROUP, name_or_path) if built_in is not None and external_entries: raise ValueError(f"Config '{name_or_path}' is provided by both ReMe and an installed distribution") if built_in is not None: @@ -222,8 +226,29 @@ def _strip_arg_dashes(arg: str) -> str: return arg -def parse_args(*args) -> tuple[str, dict]: - """Parse CLI args: first arg is action, rest are key=value pairs. +def parse_action(arg: str) -> str: + """Parse and validate one top-level CLI action.""" + action = _strip_arg_dashes(arg) + if "=" in action: + raise ValueError(f"First argument must be action, got: {arg}") + return action + + +def parse_kwargs(*args: str) -> dict: + """Parse application-style ``key=value`` CLI arguments.""" + kvs: list[str] = [] + for raw in args: + arg = _strip_arg_dashes(raw) + if "=" in arg: + kvs.append(arg) + else: + raise ValueError(f"Invalid argument format (expected key=value): {raw}") + + return parse_dot_notation(kvs) if kvs else {} + + +def parse_args(*args: str) -> tuple[str, dict]: + """Parse an application CLI action followed by ``key=value`` arguments. Usage: reme app config=paw.yaml service.name=test Returns: (action, parsed_kv_dict) @@ -231,26 +256,16 @@ def parse_args(*args) -> tuple[str, dict]: if not args: raise ValueError("No arguments provided") - first = _strip_arg_dashes(args[0]) - if "=" in first: - raise ValueError(f"First argument must be action, got: {args[0]}") - - kvs: list[str] = [] - for raw in args[1:]: - arg = _strip_arg_dashes(raw) - if "=" in arg: - kvs.append(arg) - else: - raise ValueError(f"Invalid argument format (expected key=value): {raw}") - - parsed = parse_dot_notation(kvs) if kvs else {} - return first, parsed + return parse_action(args[0]), parse_kwargs(*args[1:]) def resolve_app_config(*, log_config: bool = True, **kwargs) -> dict: """Resolve full app-start config: load `config=path` file, fall back to `default`, then deep-merge with the remaining kwargs as overrides. + Therefore ``reme start plugins=[...]`` layers that plugin selection over + ``default.yaml`` without requiring an explicit ``config=default``. + Set ``log_config=False`` for user-facing client calls that should print only the requested job's output. """ diff --git a/reme/entry_point.py b/reme/entry_point.py index fb62b345..dff3e802 100644 --- a/reme/entry_point.py +++ b/reme/entry_point.py @@ -3,6 +3,9 @@ from importlib import metadata from typing import Any +PLUGIN_ENTRY_POINT_GROUP = "reme.plugins" +CONFIG_ENTRY_POINT_GROUP = "reme.configs" + def find_entry_points(group: str, name: str) -> list[metadata.EntryPoint]: """Return all matching entry points without importing their providers.""" diff --git a/reme/plugin.py b/reme/plugin.py index 43c69950..5b574961 100644 --- a/reme/plugin.py +++ b/reme/plugin.py @@ -1,17 +1,23 @@ -"""Installed plugin discovery and application-local registration.""" +"""Load explicitly enabled plugins into one application's config and registry. + +New plugins expose a package-only ``reme.plugins`` entry point and declare two +optional mappings in ``plugin.yaml``: ``backends`` and ``application_defaults``. +Python ``Plugin`` descriptors remain supported as a compatibility boundary. +""" from __future__ import annotations from collections.abc import Iterable, Mapping from dataclasses import dataclass, field +from importlib import import_module +from importlib.metadata import EntryPoint from typing import Any from .components.base_component import ComponentMixin -from .components.component_registry import ComponentRegistry +from .components.component_registry import ComponentRegistry, create_application_registry from .config import deep_merge_config, expand_env_vars -from .entry_point import find_entry_points, load_entry_point, unique_entry_point - -PLUGIN_ENTRY_POINT_GROUP = "reme.plugins" +from .entry_point import PLUGIN_ENTRY_POINT_GROUP, find_entry_points, load_entry_point, unique_entry_point +from .plugin_manifest import PluginManifest, load_package_manifest @dataclass(frozen=True) @@ -24,13 +30,62 @@ class Backend: @dataclass(frozen=True) class Plugin: - """Declarative plugin loaded from the ``reme.plugins`` entry-point group.""" + """Legacy Python descriptor accepted during the plugin manifest migration.""" name: str backends: tuple[Backend, ...] = () config: Mapping[str, Any] = field(default_factory=dict) +@dataclass(frozen=True) +class PluginRuntime: + """Application config and registry after applying enabled plugins.""" + + config: dict[str, Any] + registry: ComponentRegistry + + +def _load_backend(target: str, *, plugin_name: str) -> type[ComponentMixin]: + """Import one ``module:class`` backend target from a plugin manifest.""" + module_name, separator, attribute = target.partition(":") + if not separator or not module_name or not attribute or ":" in attribute: + raise ValueError(f"Plugin '{plugin_name}' has invalid backend target: {target!r}") + try: + value: Any = import_module(module_name) + for part in attribute.split("."): + value = getattr(value, part) + except (AttributeError, ImportError) as exc: + raise ValueError(f"Plugin '{plugin_name}' cannot load backend '{target}': {exc}") from exc + if not isinstance(value, type) or not issubclass(value, ComponentMixin): + raise TypeError(f"Plugin '{plugin_name}' backend '{target}' is not a ComponentMixin class") + return value + + +def _plugin_from_manifest(name: str, manifest: PluginManifest) -> Plugin: + """Convert a parsed manifest into one runtime plugin descriptor.""" + backends = tuple( + Backend(backend_name, _load_backend(target, plugin_name=name)) + for backend_name, target in manifest.backends.items() + ) + return Plugin(name=name, backends=backends, config=manifest.application_defaults) + + +def _load_plugin(name: str, entry: EntryPoint) -> Plugin: + """Load a package manifest, falling back to the legacy descriptor form.""" + if ":" not in entry.value: + # Package-only targets use plugin.yaml; package:object targets are the + # legacy Python descriptor contract. + from .components.component_registry import R + + with R.preserve(allow_mutation=True): + manifest = load_package_manifest(entry.value, plugin_name=name) + return _plugin_from_manifest(name, manifest) + plugin = load_entry_point(entry, invoke=True) + if not isinstance(plugin, Plugin): + raise TypeError(f"Plugin entry point '{name}' did not return reme.plugin.Plugin") + return plugin + + class PluginManager: """Resolve enabled plugins and apply their contributions to one application.""" @@ -53,9 +108,7 @@ class PluginManager: entry = unique_entry_point(entries, name, provider="Plugin") if entry is None: raise ValueError(f"Plugin '{name}' is not installed") - plugin = load_entry_point(entry, invoke=True) - if not isinstance(plugin, Plugin): - raise TypeError(f"Plugin entry point '{name}' did not return reme.plugin.Plugin") + plugin = _load_plugin(name, entry) if plugin.name != name: raise ValueError(f"Plugin entry point '{name}' returned plugin '{plugin.name}'") plugins.append(plugin) @@ -63,7 +116,7 @@ class PluginManager: return cls(plugins) def merge_config(self, application_config: Mapping[str, Any]) -> dict[str, Any]: - """Place plugin defaults below the user's resolved application config.""" + """Place plugin application defaults below the user's resolved config.""" merged: dict[str, Any] = {} for plugin in self.plugins: merged = deep_merge_config(merged, expand_env_vars(plugin.config)) @@ -74,3 +127,11 @@ class PluginManager: for plugin in self.plugins: for backend in plugin.backends: registry.add(backend.name, backend.implementation, owner=plugin.name) + + +def resolve_plugin_runtime(application_config: Mapping[str, Any]) -> PluginRuntime: + """Build one local registry, with user config overriding plugin application defaults.""" + manager = PluginManager.discover(application_config.get("plugins") or ()) + registry = create_application_registry() + manager.register(registry) + return PluginRuntime(config=manager.merge_config(application_config), registry=registry) diff --git a/reme/plugin_cli.py b/reme/plugin_cli.py new file mode 100644 index 00000000..bfe87dba --- /dev/null +++ b/reme/plugin_cli.py @@ -0,0 +1,358 @@ +"""Local CLI for inspecting and managing installed ReMe plugin packages.""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from importlib import metadata +from pathlib import Path +import subprocess +import sys +import tomllib +from typing import Sequence + +from .entry_point import PLUGIN_ENTRY_POINT_GROUP +from .plugin_manifest import PLUGIN_MANIFEST, PluginManifest, load_package_manifest, parse_plugin_manifest + + +@dataclass(frozen=True) +class InstalledPlugin: + """Package metadata for one installed ``reme.plugins`` entry point.""" + + name: str + target: str + distribution: str + version: str + entry: metadata.EntryPoint + + @property + def format(self) -> str: + """Return the declarative or compatibility contract used by the entry point.""" + return "manifest" if ":" not in self.target else "legacy" + + +def _installed_plugins() -> list[InstalledPlugin]: + """Discover installed plugins without importing their packages.""" + entries = metadata.entry_points().select(group=PLUGIN_ENTRY_POINT_GROUP) + plugins: list[InstalledPlugin] = [] + for entry in entries: + distribution = getattr(entry, "dist", None) + dist_name = distribution.metadata.get("Name", "") if distribution is not None else "" + version = distribution.version if distribution is not None else "" + plugins.append( + InstalledPlugin( + name=entry.name, + target=entry.value, + distribution=dist_name or "unknown", + version=version or "unknown", + entry=entry, + ), + ) + return sorted(plugins, key=lambda item: (item.name, item.distribution, item.target)) + + +def _select_plugin(name: str) -> InstalledPlugin: + matches = [plugin for plugin in _installed_plugins() if plugin.name == name] + if not matches: + raise ValueError(f"Plugin '{name}' is not installed") + if len(matches) > 1: + providers = ", ".join(f"{plugin.distribution} ({plugin.target})" for plugin in matches) + raise ValueError(f"Plugin '{name}' has multiple installed providers: {providers}") + return matches[0] + + +def _print_table(headers: Sequence[str], rows: Sequence[Sequence[str]]) -> None: + widths = [max(len(header), *(len(row[index]) for row in rows)) for index, header in enumerate(headers)] + print(" ".join(header.ljust(widths[index]) for index, header in enumerate(headers))) + print(" ".join("-" * width for width in widths)) + for row in rows: + print(" ".join(value.ljust(widths[index]) for index, value in enumerate(row))) + + +def _enabled_plugins(config: str | None) -> set[str] | None: + if config is None: + return None + from .config.config_parser import resolve_app_config + + value = resolve_app_config(config=config, log_config=False).get("plugins") or [] + if not isinstance(value, list) or not all(isinstance(name, str) for name in value): + raise TypeError("Application config 'plugins' must be a list of strings") + return set(value) + + +def _list_plugins(args: argparse.Namespace) -> int: + plugins = _installed_plugins() + enabled = _enabled_plugins(args.config) + records = [ + { + "name": plugin.name, + "distribution": plugin.distribution, + "version": plugin.version, + "format": plugin.format, + "target": plugin.target, + **({"enabled": plugin.name in enabled} if enabled is not None else {}), + } + for plugin in plugins + ] + if args.json: + print(json.dumps(records, ensure_ascii=False, indent=2)) + return 0 + if not records: + print("No ReMe plugins installed.") + return 0 + headers = ["PLUGIN", "DISTRIBUTION", "VERSION", "FORMAT"] + if enabled is not None: + headers.append("ENABLED") + rows = [ + [ + record["name"], + record["distribution"], + record["version"], + record["format"], + *(["yes" if record["enabled"] else "no"] if enabled is not None else []), + ] + for record in records + ] + _print_table(headers, rows) + return 0 + + +def _installed_manifest(plugin: InstalledPlugin) -> PluginManifest: + if plugin.format != "manifest": + raise ValueError(f"Plugin '{plugin.name}' uses the legacy Python descriptor format") + distribution = getattr(plugin.entry, "dist", None) + if distribution is not None: + relative = Path(*plugin.target.split(".")).joinpath(PLUGIN_MANIFEST) + path = Path(distribution.locate_file(relative)) + if path.is_file(): + return parse_plugin_manifest(path.read_text(encoding="utf-8"), plugin_name=plugin.name) + # Editable installs may not expose package data through ``locate_file``. + # Importlib resources then has to import the package, whose ``__init__`` + # may still contain legacy registration side effects. + from .components.component_registry import R + + with R.preserve(allow_mutation=True): + return load_package_manifest(plugin.target, plugin_name=plugin.name) + + +def _plugin_details(plugin: InstalledPlugin) -> dict: + details = { + "name": plugin.name, + "distribution": plugin.distribution, + "version": plugin.version, + "format": plugin.format, + "target": plugin.target, + "backends": [], + "default_jobs": [], + } + if plugin.format == "manifest": + manifest = _installed_manifest(plugin) + details["backends"] = list(manifest.backends) + jobs = manifest.application_defaults.get("jobs") or {} + details["default_jobs"] = list(jobs) if isinstance(jobs, dict) else [] + return details + + +def _show_plugin(args: argparse.Namespace) -> int: + details = _plugin_details(_select_plugin(args.plugin)) + if args.json: + print(json.dumps(details, ensure_ascii=False, indent=2)) + return 0 + for label, key in ( + ("Plugin", "name"), + ("Distribution", "distribution"), + ("Version", "version"), + ("Format", "format"), + ("Entry point", "target"), + ): + print(f"{label}: {details[key]}") + for label, key in (("Backends", "backends"), ("Default jobs", "default_jobs")): + values = details[key] + print(f"{label}:" if values else f"{label}: none") + for value in values: + print(f" {value}") + return 0 + + +def _run_pip(arguments: list[str]) -> int: + command = [sys.executable, "-m", "pip", *arguments] + try: + return subprocess.run(command, check=False).returncode + except OSError as exc: + raise RuntimeError(f"Unable to run pip with {sys.executable}: {exc}") from exc + + +def _install_plugin(args: argparse.Namespace) -> int: + command = ["install"] + if args.editable: + command.append("--editable") + if args.upgrade: + command.append("--upgrade") + command.append(args.target) + result = _run_pip(command) + if result == 0: + print("Package installed. Run 'reme plugins list' to verify it, then enable its plugin name in app config.") + return result + + +def _uninstall_plugin(args: argparse.Namespace) -> int: + plugin = _select_plugin(args.plugin) + if plugin.distribution == "unknown": + raise ValueError(f"Cannot determine the distribution that provides plugin '{plugin.name}'") + siblings = [ + candidate.name + for candidate in _installed_plugins() + if candidate.distribution == plugin.distribution and candidate.name != plugin.name + ] + if siblings: + print(f"Distribution '{plugin.distribution}' also provides: {', '.join(siblings)}") + command = ["uninstall"] + if args.yes: + command.append("--yes") + command.append(plugin.distribution) + result = _run_pip(command) + if result == 0: + print(f"Plugin package '{plugin.distribution}' uninstalled. Remove '{plugin.name}' from application configs.") + return result + + +def _validate_plugins(manager) -> None: + """Validate imports, registry ownership, and merged application schema.""" + from .components.component_registry import create_application_registry + from .config.config_parser import resolve_app_config + from .schema.application_config import ApplicationConfig + + registry = create_application_registry() + manager.register(registry) + ApplicationConfig(**manager.merge_config(resolve_app_config(log_config=False))) + + +def _validate_installed(name: str) -> list[str]: + from .plugin import PluginManager + + manager = PluginManager.discover([name]) + _validate_plugins(manager) + return [name] + + +def _local_source_roots(project_file: Path, project: dict) -> list[Path]: + """Return declared and conventional Python source roots for a local project.""" + project_root = project_file.parent + candidates: list[Path] = [] + + setuptools = project.get("tool", {}).get("setuptools", {}) + if isinstance(setuptools, dict): + package_dir = setuptools.get("package-dir", {}) + if isinstance(package_dir, dict) and isinstance(package_dir.get(""), str): + candidates.append(project_root / package_dir[""]) + + packages = setuptools.get("packages", {}) + package_find = packages.get("find", {}) if isinstance(packages, dict) else {} + if isinstance(package_find, dict): + where = package_find.get("where", []) + if isinstance(where, str): + where = [where] + if isinstance(where, list): + candidates.extend(project_root / item for item in where if isinstance(item, str)) + + # ``src`` is a build-backend-independent Python project convention used by + # Hatchling, Poetry, Flit, and setuptools. Root-layout projects remain the + # final fallback. + candidates.extend((project_root / "src", project_root)) + return list(dict.fromkeys(candidate.resolve() for candidate in candidates)) + + +def _validate_local(path: Path) -> list[str]: + from .components.component_registry import R + from .plugin import PluginManager, _plugin_from_manifest + + project_file = path if path.name == "pyproject.toml" else path / "pyproject.toml" + if not project_file.is_file(): + raise FileNotFoundError(f"pyproject.toml not found: {project_file}") + project = tomllib.loads(project_file.read_text(encoding="utf-8")) + entry_points = project.get("project", {}).get("entry-points", {}).get(PLUGIN_ENTRY_POINT_GROUP) + if not isinstance(entry_points, dict) or not entry_points: + raise ValueError(f"No {PLUGIN_ENTRY_POINT_GROUP} entry points found in {project_file}") + + source_roots = _local_source_roots(project_file, project) + plugins = [] + selected_roots: list[Path] = [] + manifests = [] + for name, package in entry_points.items(): + if not isinstance(name, str) or not isinstance(package, str) or ":" in package: + raise ValueError("Local validation requires package-only manifest entry points") + relative_manifest = Path(*package.split(".")).joinpath(PLUGIN_MANIFEST) + manifest_path = next( + (root / relative_manifest for root in source_roots if (root / relative_manifest).is_file()), + None, + ) + if manifest_path is None: + searched = ", ".join(str(root / relative_manifest) for root in source_roots) + raise FileNotFoundError(f"Plugin manifest not found; searched: {searched}") + selected_roots.append(manifest_path.parents[len(package.split("."))]) + manifests.append((name, manifest_path)) + + inserted_roots = list(dict.fromkeys(str(root) for root in selected_roots)) + for source_root in reversed(inserted_roots): + sys.path.insert(0, source_root) + try: + # Match installed-plugin loading: imports may execute compatibility + # decorators, but they must not mutate the frozen built-in template. + with R.preserve(allow_mutation=True): + for name, manifest_path in manifests: + manifest = parse_plugin_manifest(manifest_path.read_text(encoding="utf-8"), plugin_name=name) + plugins.append(_plugin_from_manifest(name, manifest)) + _validate_plugins(PluginManager(plugins)) + finally: + for source_root in inserted_roots: + sys.path.remove(source_root) + return [plugin.name for plugin in plugins] + + +def _validate_plugin(args: argparse.Namespace) -> int: + path = Path(args.target).expanduser() + names = _validate_local(path.resolve()) if path.exists() else _validate_installed(args.target) + print(f"Valid ReMe plugin: {', '.join(names)}") + return 0 + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="reme plugins", description="Manage ReMe plugin packages.") + commands = parser.add_subparsers(dest="command", required=True) + + list_parser = commands.add_parser("list", help="List installed ReMe plugins.") + list_parser.add_argument("--config", help="Show whether plugins are enabled by this config.") + list_parser.add_argument("--json", action="store_true", help="Print JSON output.") + list_parser.set_defaults(handler=_list_plugins) + + show_parser = commands.add_parser("show", help="Show one installed plugin.") + show_parser.add_argument("plugin") + show_parser.add_argument("--json", action="store_true", help="Print JSON output.") + show_parser.set_defaults(handler=_show_plugin) + + install_parser = commands.add_parser("install", help="Install a plugin package with this Python interpreter.") + install_parser.add_argument("target", help="Distribution specifier, wheel, VCS URL, or local path.") + install_parser.add_argument("--editable", action="store_true", help="Install a local project in editable mode.") + install_parser.add_argument("--upgrade", action="store_true", help="Upgrade an existing installation.") + install_parser.set_defaults(handler=_install_plugin) + + uninstall_parser = commands.add_parser("uninstall", help="Uninstall the distribution providing a plugin.") + uninstall_parser.add_argument("plugin", help="Plugin entry-point name, such as auto-fin.") + uninstall_parser.add_argument("--yes", action="store_true", help="Do not ask pip for confirmation.") + uninstall_parser.set_defaults(handler=_uninstall_plugin) + + validate_parser = commands.add_parser("validate", help="Validate an installed plugin or local plugin project.") + validate_parser.add_argument("target", help="Installed plugin name, project directory, or pyproject.toml.") + validate_parser.set_defaults(handler=_validate_plugin) + return parser + + +def plugin_cli(argv: Sequence[str]) -> int: + """Run the local-only plugin command group and return a process status.""" + args = _parser().parse_args(list(argv)) + try: + return args.handler(args) + except (FileNotFoundError, KeyError, ModuleNotFoundError, RuntimeError, TypeError, ValueError) as exc: + print(f"Error: {exc}", file=sys.stderr) + return 1 diff --git a/reme/plugin_manifest.py b/reme/plugin_manifest.py new file mode 100644 index 00000000..ab82c868 --- /dev/null +++ b/reme/plugin_manifest.py @@ -0,0 +1,56 @@ +"""Parse the small, declarative contract shared by plugin runtime and CLI.""" + +from dataclasses import dataclass +from importlib import resources +from typing import Any + +import yaml + +PLUGIN_MANIFEST = "plugin.yaml" + + +@dataclass(frozen=True) +class PluginManifest: + """The two contributions an installed plugin may declare.""" + + backends: dict[str, str] + application_defaults: dict[str, Any] + + +def parse_plugin_manifest(text: str, *, plugin_name: str) -> PluginManifest: + """Parse and validate plugin.yaml without importing backend modules.""" + try: + value = yaml.safe_load(text) + except yaml.YAMLError as exc: + raise ValueError(f"Plugin '{plugin_name}' manifest is invalid YAML") from exc + if not isinstance(value, dict): + raise ValueError(f"Plugin '{plugin_name}' manifest root must be a mapping") + + unknown = set(value) - {"backends", "application_defaults"} + if unknown: + raise ValueError(f"Plugin '{plugin_name}' manifest has unknown keys: {', '.join(sorted(unknown))}") + + backends = value.get("backends") + application_defaults = value.get("application_defaults") + backends = {} if backends is None else backends + application_defaults = {} if application_defaults is None else application_defaults + if not isinstance(backends, dict): + raise TypeError(f"Plugin '{plugin_name}' manifest 'backends' must be a mapping") + if not isinstance(application_defaults, dict): + raise TypeError(f"Plugin '{plugin_name}' manifest 'application_defaults' must be a mapping") + for name, target in backends.items(): + if not isinstance(name, str) or not name: + raise TypeError(f"Plugin '{plugin_name}' backend names must be non-empty strings") + if not isinstance(target, str) or not target: + raise TypeError(f"Plugin '{plugin_name}' backend target for '{name}' must be a non-empty string") + return PluginManifest(backends=dict(backends), application_defaults=dict(application_defaults)) + + +def load_package_manifest(package: str, *, plugin_name: str) -> PluginManifest: + """Read plugin.yaml from an importable package.""" + try: + path = resources.files(package).joinpath(PLUGIN_MANIFEST) + text = path.read_text(encoding="utf-8") + except (ModuleNotFoundError, FileNotFoundError, TypeError) as exc: + raise ValueError(f"Plugin '{plugin_name}' does not provide {package}/{PLUGIN_MANIFEST}") from exc + return parse_plugin_manifest(text, plugin_name=plugin_name) diff --git a/reme/reme.py b/reme/reme.py index 64166be7..401a61b7 100644 --- a/reme/reme.py +++ b/reme/reme.py @@ -1,14 +1,15 @@ """ReMe memory management application entry point.""" import asyncio +from collections.abc import Sequence +from dataclasses import dataclass import sys from .application import Application -from .components import create_application_registry from .components.service.cli_service import prepare_start_config, should_precheck_start -from .config import parse_args, resolve_app_config +from .config import parse_action, parse_kwargs, resolve_app_config from .enumeration import ComponentEnum -from .plugin import PluginManager +from .plugin import resolve_plugin_runtime from .utils import cli_find_reme, load_env, precheck_start, running_app_config _CLIENT_KWARGS = {"host", "port", "timeout", "transport", "command", "args", "show_metadata"} @@ -18,6 +19,21 @@ class ReMe(Application): """ReMe memory management application.""" +@dataclass(frozen=True) +class CliInvocation: + """A top-level CLI action with arguments in that action's own syntax.""" + + action: str + arguments: tuple[str, ...] + + +def parse_cli_invocation(argv: Sequence[str]) -> CliInvocation: + """Parse only the grammar shared by every CLI command family.""" + if not argv: + raise ValueError("No arguments provided") + return CliInvocation(action=parse_action(argv[0]), arguments=tuple(argv[1:])) + + async def call_server(action: str, **kwargs): """Call the running server with a client matching its *actual* service config. @@ -42,8 +58,8 @@ async def call_server(action: str, **kwargs): app_config = running_app_config() if app_config is None: app_config = resolve_app_config(log_config=False, **resolve_kwargs) - plugin_manager = PluginManager.discover(app_config.get("plugins") or ()) - app_config = plugin_manager.merge_config(app_config) + runtime = resolve_plugin_runtime(app_config) + app_config = runtime.config service = app_config.get("service") service = service if isinstance(service, dict) else {} @@ -55,9 +71,7 @@ async def call_server(action: str, **kwargs): client_kwargs = {k: seed[k] for k in _CLIENT_KWARGS if k in seed} client_kwargs.update({key: kwargs.pop(key) for key in list(kwargs) if key in _CLIENT_KWARGS}) - registry = create_application_registry() - plugin_manager.register(registry) - client_cls = registry.get(ComponentEnum.CLIENT, backend) + client_cls = runtime.registry.get(ComponentEnum.CLIENT, backend) if client_cls is None: raise ValueError(f"Unknown client backend: {backend!r}") async with client_cls(**client_kwargs) as client: @@ -66,16 +80,39 @@ async def call_server(action: str, **kwargs): print() -def main(): +def _run_plugin_command(argv: Sequence[str]) -> None: + """Run local package management without initializing the application.""" + from .plugin_cli import plugin_cli + + status = plugin_cli(argv) + if status: + raise SystemExit(status) + + +def _start_application(kwargs: dict, environment: dict) -> None: + """Resolve startup configuration and run the application.""" + kwargs = prepare_start_config(kwargs) + kwargs["environment"] = environment + if should_precheck_start(kwargs) and not precheck_start(kwargs.get("service")): + return + ReMe(**kwargs).run_app() + + +def main() -> None: """Parse CLI arguments and launch the appropriate mode.""" + invocation = parse_cli_invocation(sys.argv[1:]) + action = invocation.action + + if action == "plugins": + # Package management is local-only and must not load application config, + # environment files, or a running service. + _run_plugin_command(invocation.arguments) + return + environment = load_env() - action, kwargs = parse_args(*sys.argv[1:]) + kwargs = parse_kwargs(*invocation.arguments) if action == "start": - kwargs = prepare_start_config(kwargs) - kwargs["environment"] = environment - if should_precheck_start(kwargs) and not precheck_start(kwargs.get("service")): - return - ReMe(**kwargs).run_app() + _start_application(kwargs, environment) elif action == "find_reme": cli_find_reme() else: diff --git a/tests/unit/test_config_parser.py b/tests/unit/test_config_parser.py index 67ed2de0..22c51434 100644 --- a/tests/unit/test_config_parser.py +++ b/tests/unit/test_config_parser.py @@ -71,6 +71,14 @@ def test_resolve_app_config_can_suppress_config_log(monkeypatch): assert not messages +def test_resolve_app_config_layers_plugins_over_default(): + """A plugin-only start keeps the ordinary default application config.""" + config = resolve_app_config(log_config=False, plugins=["auto-fin"]) + + assert config["service"]["backend"] == "http" + assert config["plugins"] == ["auto-fin"] + + def test_default_config_registers_daily_write_job(): """``daily_write`` is exposed as a base job backed by ``daily_write_step``.""" cfg = _load_config("default.yaml") @@ -127,6 +135,14 @@ def test_parse_args_rejects_non_key_value_extra_argument(): parse_args("search", "hello") +def test_parse_args_separates_action_and_application_kwargs(): + """The shared action grammar is independent from application key/value parsing.""" + action, kwargs = parse_args("--search", "--query=hello", "limit=3") + + assert action == "search" + assert kwargs == {"query": "hello", "limit": 3} + + @pytest.mark.parametrize("item", ["=1", ".a=1", "a.=1", "a..b=1"]) def test_parse_dot_notation_rejects_empty_key_segments(item): """Dot notation keys cannot contain empty path segments.""" diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index d5854682..a86aaa15 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -1,6 +1,6 @@ """Tests for installed plugin discovery and application-local registration.""" -# pylint: disable=missing-class-docstring,missing-function-docstring +# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access from pathlib import Path @@ -11,7 +11,8 @@ from reme.components.base_component import BaseComponent, ComponentMixin from reme.components.component_registry import ComponentRegistry, R from reme.config.config_parser import _load_config from reme.enumeration import ComponentEnum -from reme.plugin import Backend, Plugin, PluginManager +from reme.plugin import Backend, Plugin, PluginManager, _load_backend +from reme.plugin_manifest import parse_plugin_manifest class _PluginStep(ComponentMixin): @@ -42,7 +43,7 @@ def _set_entry_points(monkeypatch, *entries): monkeypatch.setattr("reme.entry_point.metadata.entry_points", lambda: _FakeEntryPoints(entries)) -def test_plugin_defaults_are_below_application_config(): +def test_plugin_application_defaults_are_below_application_config(): manager = PluginManager( [ Plugin( @@ -57,7 +58,7 @@ def test_plugin_defaults_are_below_application_config(): assert merged["jobs"]["task"] == {"backend": "base", "value": 2} -def test_plugin_defaults_expand_environment(monkeypatch): +def test_plugin_application_defaults_expand_environment(monkeypatch): monkeypatch.setenv("PLUGIN_LIMIT", "12") manager = PluginManager([Plugin(name="example", config={"limit": "${PLUGIN_LIMIT}"})]) @@ -138,6 +139,57 @@ def test_plugin_manager_loads_explicit_entry_point(monkeypatch): assert manager.plugins == (descriptor,) +def test_plugin_manager_loads_package_manifest(monkeypatch, tmp_path): + package = tmp_path / "example_plugin" + package.mkdir() + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "backend.py").write_text( + "from reme.components.base_component import ComponentMixin\n" + "from reme.enumeration import ComponentEnum\n" + "class ExampleStep(ComponentMixin):\n" + " component_type = ComponentEnum.STEP\n", + encoding="utf-8", + ) + (package / "plugin.yaml").write_text( + "backends:\n" + " example_step: example_plugin.backend:ExampleStep\n" + "application_defaults:\n" + " jobs:\n" + " example:\n" + " backend: base\n", + encoding="utf-8", + ) + monkeypatch.syspath_prepend(str(tmp_path)) + _set_entry_points( + monkeypatch, + _FakeEntryPoint("example", "example_plugin", lambda: None, "reme.plugins"), + ) + + manager = PluginManager.discover(["example"]) + registry = ComponentRegistry() + manager.register(registry) + + backend = registry.get(ComponentEnum.STEP, "example_step") + assert backend is not None + assert backend.__name__ == "ExampleStep" + assert manager.merge_config({})["jobs"]["example"]["backend"] == "base" + + +def test_plugin_manifest_rejects_legacy_defaults_field(): + with pytest.raises(ValueError, match="unknown keys: defaults"): + parse_plugin_manifest("defaults: {}\n", plugin_name="example") + + +def test_plugin_manifest_requires_application_defaults_mapping(): + with pytest.raises(TypeError, match="manifest 'application_defaults' must be a mapping"): + parse_plugin_manifest("application_defaults: []\n", plugin_name="example") + + +def test_plugin_manifest_reports_missing_backend_attribute(): + with pytest.raises(ValueError, match="cannot load backend.*MissingStep"): + _load_backend("reme.plugin:MissingStep", plugin_name="missing") + + def test_plugin_manager_rejects_multiple_entry_point_providers(monkeypatch): descriptor = Plugin(name="example") _set_entry_points( diff --git a/tests/unit/test_plugin_cli.py b/tests/unit/test_plugin_cli.py new file mode 100644 index 00000000..ff06868b --- /dev/null +++ b/tests/unit/test_plugin_cli.py @@ -0,0 +1,297 @@ +"""Tests for local plugin package management commands.""" + +# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access + +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from reme import plugin_cli as plugin_cli_module +from reme import reme as reme_module +from reme.components import R +from reme.enumeration import ComponentEnum + + +class _FakeDistribution: + def __init__(self, root: Path, name: str = "reme-auto-fin", version: str = "0.1.0") -> None: + self.root = root + self.metadata = {"Name": name} + self.version = version + + def locate_file(self, relative: Path) -> Path: + return self.root / relative + + +class _FakeEntryPoint: + def __init__(self, name: str, value: str, distribution: _FakeDistribution) -> None: + self.name = name + self.value = value + self.dist = distribution + + +class _FakeEntryPoints(list): + def select(self, *, group): + assert group == "reme.plugins" + return self + + +def _install_manifest(tmp_path: Path) -> _FakeEntryPoint: + package = tmp_path / "reme_auto_fin" + package.mkdir() + (package / "plugin.yaml").write_text( + "backends:\n" + " auto_fin_step: reme_auto_fin.step:AutoFinStep\n" + "application_defaults:\n" + " jobs:\n" + " auto_fin:\n" + " backend: base\n", + encoding="utf-8", + ) + return _FakeEntryPoint("auto-fin", "reme_auto_fin", _FakeDistribution(tmp_path)) + + +def test_list_plugins_does_not_load_plugin_code(monkeypatch, tmp_path, capsys): + entry = _install_manifest(tmp_path) + monkeypatch.setattr( + plugin_cli_module.metadata, + "entry_points", + lambda: _FakeEntryPoints([entry]), + ) + + assert plugin_cli_module.plugin_cli(["list"]) == 0 + + output = capsys.readouterr().out + assert "auto-fin" in output + assert "reme-auto-fin" in output + assert "manifest" in output + + +def test_list_plugins_marks_configured_plugins(monkeypatch, tmp_path, capsys): + entry = _install_manifest(tmp_path) + monkeypatch.setattr( + plugin_cli_module.metadata, + "entry_points", + lambda: _FakeEntryPoints([entry]), + ) + monkeypatch.setattr(plugin_cli_module, "_enabled_plugins", lambda _config: {"auto-fin"}) + + assert plugin_cli_module.plugin_cli(["list", "--config", "daily_cookbook"]) == 0 + + output = capsys.readouterr().out + assert "ENABLED" in output + assert "yes" in output + + +def test_show_plugin_reads_manifest_without_importing_backends(monkeypatch, tmp_path, capsys): + entry = _install_manifest(tmp_path) + monkeypatch.setattr( + plugin_cli_module.metadata, + "entry_points", + lambda: _FakeEntryPoints([entry]), + ) + + assert plugin_cli_module.plugin_cli(["show", "auto-fin"]) == 0 + + output = capsys.readouterr().out + assert "auto_fin_step" in output + assert "auto_fin" in output + + +def test_show_plugin_preserves_registry_when_editable_fallback_imports_package(monkeypatch, tmp_path, capsys): + package = tmp_path / "fallback_plugin" + package.mkdir() + (package / "__init__.py").write_text("from .backend import FallbackStep\n", encoding="utf-8") + (package / "backend.py").write_text( + "from reme.components import ComponentMixin, R\n" + "from reme.enumeration import ComponentEnum\n" + "@R.register('fallback_import_side_effect')\n" + "class FallbackStep(ComponentMixin):\n" + " component_type = ComponentEnum.STEP\n", + encoding="utf-8", + ) + (package / "plugin.yaml").write_text( + """backends: + fallback_step: fallback_plugin.backend:FallbackStep +""", + encoding="utf-8", + ) + entry = _FakeEntryPoint("fallback", "fallback_plugin", _FakeDistribution(tmp_path / "missing")) + monkeypatch.syspath_prepend(str(tmp_path)) + monkeypatch.setattr(plugin_cli_module.metadata, "entry_points", lambda: _FakeEntryPoints([entry])) + + assert plugin_cli_module.plugin_cli(["show", "fallback"]) == 0 + + assert "fallback_step" in capsys.readouterr().out + assert R.get(ComponentEnum.STEP, "fallback_import_side_effect") is None + + +def test_install_uses_current_python_pip(monkeypatch, capsys): + commands = [] + monkeypatch.setattr(plugin_cli_module, "_run_pip", lambda command: commands.append(command) or 0) + + assert plugin_cli_module.plugin_cli(["install", ".", "--editable", "--upgrade"]) == 0 + + assert commands == [["install", "--editable", "--upgrade", "."]] + assert "plugins list" in capsys.readouterr().out + + +def test_pip_uses_current_python_interpreter(monkeypatch): + calls = [] + monkeypatch.setattr( + plugin_cli_module.subprocess, + "run", + lambda command, **kwargs: calls.append((command, kwargs)) or SimpleNamespace(returncode=7), + ) + + assert plugin_cli_module._run_pip(["install", "example"]) == 7 + + assert calls == [ + ( + [plugin_cli_module.sys.executable, "-m", "pip", "install", "example"], + {"check": False}, + ), + ] + + +def test_uninstall_resolves_plugin_to_distribution(monkeypatch, tmp_path, capsys): + entry = _install_manifest(tmp_path) + commands = [] + monkeypatch.setattr( + plugin_cli_module.metadata, + "entry_points", + lambda: _FakeEntryPoints([entry]), + ) + monkeypatch.setattr(plugin_cli_module, "_run_pip", lambda command: commands.append(command) or 0) + + assert plugin_cli_module.plugin_cli(["uninstall", "auto-fin", "--yes"]) == 0 + + assert commands == [["uninstall", "--yes", "reme-auto-fin"]] + assert "Remove 'auto-fin'" in capsys.readouterr().out + + +def test_validate_local_auto_fin_project(): + repository = Path(__file__).resolve().parents[2] + + names = plugin_cli_module._validate_local(repository / "plugins" / "auto-fin") + + assert names == ["auto-fin"] + + +@pytest.mark.parametrize( + "build_system", + [ + "requires = ['hatchling']\nbuild-backend = 'hatchling.build'\n", + "requires = ['poetry-core']\nbuild-backend = 'poetry.core.masonry.api'\n", + "requires = ['flit-core']\nbuild-backend = 'flit_core.buildapi'\n", + ], +) +def test_validate_local_supports_backend_independent_src_layout(tmp_path, build_system): + package = tmp_path / "src" / "src_layout_plugin" + package.mkdir(parents=True) + (tmp_path / "pyproject.toml").write_text( + "[project]\n" + "name = 'example-plugin'\n" + "version = '0.1.0'\n" + "[project.entry-points.'reme.plugins']\n" + "example = 'src_layout_plugin'\n" + "[build-system]\n" + f"{build_system}", + encoding="utf-8", + ) + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "backend.py").write_text( + "from reme.components import ComponentMixin\n" + "from reme.enumeration import ComponentEnum\n" + "class ExampleStep(ComponentMixin):\n" + " component_type = ComponentEnum.STEP\n", + encoding="utf-8", + ) + (package / "plugin.yaml").write_text( + "backends:\n example_step: src_layout_plugin.backend:ExampleStep\n", + encoding="utf-8", + ) + + assert plugin_cli_module._validate_local(tmp_path) == ["example"] + + +def test_validate_local_supports_setuptools_find_source_root(tmp_path): + package = tmp_path / "python" / "find_layout_plugin" + package.mkdir(parents=True) + (tmp_path / "pyproject.toml").write_text( + "[project]\n" + "name = 'example-plugin'\n" + "version = '0.1.0'\n" + "[project.entry-points.'reme.plugins']\n" + "example = 'find_layout_plugin'\n" + "[tool.setuptools.packages.find]\n" + "where = ['python']\n", + encoding="utf-8", + ) + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "backend.py").write_text( + "from reme.components import ComponentMixin\n" + "from reme.enumeration import ComponentEnum\n" + "class ExampleStep(ComponentMixin):\n" + " component_type = ComponentEnum.STEP\n", + encoding="utf-8", + ) + (package / "plugin.yaml").write_text( + "backends:\n example_step: find_layout_plugin.backend:ExampleStep\n", + encoding="utf-8", + ) + + assert plugin_cli_module._validate_local(tmp_path) == ["example"] + + +def test_validate_local_preserves_registry_during_backend_imports(tmp_path): + package = tmp_path / "src" / "decorated_plugin" + package.mkdir(parents=True) + (tmp_path / "pyproject.toml").write_text( + "[project]\n" + "name = 'decorated-plugin'\n" + "version = '0.1.0'\n" + "[project.entry-points.'reme.plugins']\n" + "decorated = 'decorated_plugin'\n" + "[tool.setuptools.package-dir]\n" + "'' = 'src'\n", + encoding="utf-8", + ) + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "backend.py").write_text( + "from reme.components import ComponentMixin, R\n" + "from reme.enumeration import ComponentEnum\n" + "@R.register('local_import_side_effect')\n" + "class DecoratedStep(ComponentMixin):\n" + " component_type = ComponentEnum.STEP\n", + encoding="utf-8", + ) + (package / "plugin.yaml").write_text( + """backends: + decorated_step: decorated_plugin.backend:DecoratedStep +""", + encoding="utf-8", + ) + + assert plugin_cli_module._validate_local(tmp_path) == ["decorated"] + assert R.get(ComponentEnum.STEP, "local_import_side_effect") is None + + +def test_plugin_command_errors_are_clean(monkeypatch, capsys): + monkeypatch.setattr(plugin_cli_module.metadata, "entry_points", _FakeEntryPoints) + + assert plugin_cli_module.plugin_cli(["show", "missing"]) == 1 + + assert "Plugin 'missing' is not installed" in capsys.readouterr().err + + +@pytest.mark.parametrize("action", ["plugins", "-plugins", "--plugins"]) +def test_main_routes_plugins_before_loading_environment(monkeypatch, action): + events = [] + monkeypatch.setattr("sys.argv", ["reme", action, "list"]) + monkeypatch.setattr(reme_module, "load_env", lambda: events.append("load_env")) + monkeypatch.setattr(plugin_cli_module, "plugin_cli", lambda argv: events.append(list(argv)) or 0) + + reme_module.main() + + assert events == [["list"]] diff --git a/tests/unit/test_reme_cli.py b/tests/unit/test_reme_cli.py index 7430aafc..13d74cbd 100644 --- a/tests/unit/test_reme_cli.py +++ b/tests/unit/test_reme_cli.py @@ -12,8 +12,9 @@ from reme.components.service import cli_service from reme.components.service.cli_service import CliService from reme import reme as reme_module from reme.components.base_component import ComponentMixin +from reme.components.component_registry import create_application_registry from reme.enumeration import ComponentEnum -from reme.plugin import Backend, Plugin, PluginManager +from reme.plugin import Backend, Plugin, PluginManager, PluginRuntime def _recording_client(seen, output="ok", base=object): @@ -42,8 +43,11 @@ def _recording_client(seen, output="ok", base=object): def _set_client_backend(monkeypatch, client_cls): monkeypatch.setattr( reme_module, - "create_application_registry", - lambda: SimpleNamespace(get=lambda component_type, backend: client_cls), + "resolve_plugin_runtime", + lambda config: PluginRuntime( + config=dict(config), + registry=SimpleNamespace(get=lambda component_type, backend: client_cls), + ), ) @@ -94,8 +98,8 @@ def test_main_loads_env_before_calling_server(monkeypatch): events = [] main_globals = reme_module.main.__globals__ + monkeypatch.setattr("sys.argv", ["reme", "shell", "cmd=pwd"]) monkeypatch.setitem(main_globals, "load_env", lambda: events.append("load_env")) - monkeypatch.setitem(main_globals, "parse_args", lambda *_args: ("shell", {"cmd": "pwd"})) async def fake_call_server(action, **kwargs): events.append(("call_server", action, kwargs)) @@ -123,8 +127,8 @@ def test_main_saves_loaded_environment_in_start_config(monkeypatch): observed["ran"] = True main_globals = reme_module.main.__globals__ + monkeypatch.setattr("sys.argv", ["reme", "start"]) monkeypatch.setitem(main_globals, "load_env", lambda: {"TOOL_ENV": "configured"}) - monkeypatch.setitem(main_globals, "parse_args", lambda *_args: ("start", {})) monkeypatch.setitem(main_globals, "prepare_start_config", lambda _kwargs: {"service": {"backend": "cli"}}) monkeypatch.setitem(main_globals, "ReMe", FakeReMe) @@ -354,7 +358,13 @@ def test_call_server_uses_running_plugins_and_their_service_defaults(monkeypatch lambda **_kwargs: {"service": {"backend": "http"}}, ) monkeypatch.setattr(reme_module, "running_app_config", lambda: {"plugins": ["example"]}) - monkeypatch.setattr(reme_module.PluginManager, "discover", lambda _specs: manager) + + def resolve_runtime(config): + registry = create_application_registry() + manager.register(registry) + return PluginRuntime(config=manager.merge_config(config), registry=registry) + + monkeypatch.setattr(reme_module, "resolve_plugin_runtime", resolve_runtime) asyncio.run(reme_module.call_server("search", query="hello"))