mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
feat: simplify plugin setup and add management CLI (#485)
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
Some checks are pending
CI / Documentation / Test and build documentation (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
* feat: simplify plugin setup and add management CLI * fix: isolate plugin CLI import side effects * refactor: streamline plugin validation * fix: route plugin CLI arguments independently * fix: support standard plugin source layouts
This commit is contained in:
parent
c8e1248769
commit
efcc2b34d1
29 changed files with 1654 additions and 209 deletions
|
|
@ -190,6 +190,7 @@ These Markdown guides cover the main user workflows and the runtime contracts im
|
|||
| Guide | What you will learn |
|
||||
|-------|---------------------|
|
||||
| [Quick Start](docs/en/quick_start.md) | Install ReMe, start the service, and run the first file and memory operations. |
|
||||
| [Plugin Management](docs/en/plugin_management.md) | Install, inspect, validate, enable, and uninstall local ReMe plugins. |
|
||||
| [Memory as File](docs/en/memory_as_file.md) | Understand workspace layers, frontmatter, wikilinks, chunks, and the file-as-source-of-truth model. |
|
||||
| [Auto Memory](docs/en/auto_memory.md) | Preserve source conversations and distill reusable daily memory cards. |
|
||||
| [Auto Resource](docs/en/auto_resource.md) | Import supported text resources and turn them into source-linked daily cards. |
|
||||
|
|
|
|||
|
|
@ -183,6 +183,7 @@ ReMe 会把 Agent 记忆保存为可读的 Markdown。
|
|||
| 文档 | 主要内容 |
|
||||
|------|----------|
|
||||
| [快速开始](docs/zh/quick_start.md) | 安装 ReMe、启动服务,并执行首次文件和记忆操作。 |
|
||||
| [插件管理](docs/zh/plugin_management.md) | 安装、查看、校验、启用和卸载本地 ReMe 插件。 |
|
||||
| [Memory as File](docs/zh/memory_as_file.md) | 理解 workspace 分层、frontmatter、wikilink、chunk 和文件事实来源模型。 |
|
||||
| [Auto Memory](docs/zh/auto_memory.md) | 保留过滤后的对话来源记录,并提炼可复用的 daily 记忆卡片。 |
|
||||
| [Auto Resource](docs/zh/auto_resource.md) | 导入支持的文本资料,转换为可追溯来源的 daily 卡片。 |
|
||||
|
|
|
|||
|
|
@ -231,14 +231,31 @@ They are configured under `components` and participate in the same dependency or
|
|||
Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
|
||||
bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's
|
||||
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
|
||||
`plugins` in the resolved configuration. A plugin
|
||||
is exposed through the `reme.plugins` Python entry-point group and returns a declarative `reme.plugin.Plugin` containing
|
||||
its named backend classes and default configuration. Plugin registration therefore stays local to one application;
|
||||
`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point
|
||||
group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to
|
||||
`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The
|
||||
entry-point name is the plugin's identity.
|
||||
Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override.
|
||||
Plugin registration therefore stays local to one application;
|
||||
duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other.
|
||||
|
||||
Installed plugins may also expose named configuration files through `reme.configs`. Configuration can use `extends` to
|
||||
inherit another built-in, plugin, or file-based configuration. The [Auto Fin plugin](../../plugins/auto-fin/README.md)
|
||||
is the complete packaging example.
|
||||
The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration
|
||||
files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. The
|
||||
[Auto Fin plugin](../../plugins/auto-fin/README.md) is the current packaging example.
|
||||
|
||||
Plugin packages are managed locally and remain separate from per-application activation:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
reme plugins install reme-auto-fin
|
||||
reme plugins show auto-fin
|
||||
reme plugins validate auto-fin
|
||||
reme plugins uninstall auto-fin
|
||||
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
|
||||
|
||||
### 4.3 Component.bind
|
||||
|
||||
|
|
|
|||
228
docs/en/plugin_management.md
Normal file
228
docs/en/plugin_management.md
Normal file
|
|
@ -0,0 +1,228 @@
|
|||
# Plugin Management
|
||||
|
||||
ReMe plugins are ordinary Python distributions discovered through the `reme.plugins` entry-point group. Installing a
|
||||
plugin makes it available to the current Python environment; it does not enable the plugin in every ReMe application.
|
||||
|
||||
Keep these two operations separate:
|
||||
|
||||
```text
|
||||
reme plugins install ... install a package into the current Python environment
|
||||
plugins: [auto-fin] enable an installed plugin for one Application
|
||||
```
|
||||
|
||||
Plugin package management is local-only. It does not run through a ReMe HTTP or MCP service and never edits application
|
||||
configuration files automatically.
|
||||
|
||||
A typical plugin workflow has three stages:
|
||||
|
||||
1. Install ReMe and the plugin distribution.
|
||||
2. Configure the plugin's runtime environment as described in the
|
||||
[ReMe environment-variable guide](../../README.md#environment-variables).
|
||||
3. Start an Application with the plugin explicitly enabled, for example
|
||||
`reme start plugins='["auto-fin"]'`.
|
||||
|
||||
## List installed plugins
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
```
|
||||
|
||||
The table shows the plugin entry-point name, Python distribution, version, and plugin contract:
|
||||
|
||||
```text
|
||||
PLUGIN DISTRIBUTION VERSION FORMAT
|
||||
-------- ------------- ------- --------
|
||||
auto-fin reme-auto-fin 0.1.0 manifest
|
||||
```
|
||||
|
||||
`manifest` plugins use the current package-level `plugin.yaml` contract. `legacy` plugins use the compatible Python
|
||||
descriptor contract.
|
||||
|
||||
A manifest separates backend registration from application configuration:
|
||||
|
||||
```yaml
|
||||
backends:
|
||||
example_step: example_plugin.steps:ExampleStep
|
||||
|
||||
application_defaults:
|
||||
jobs:
|
||||
example:
|
||||
backend: base
|
||||
steps:
|
||||
- backend: example_step
|
||||
```
|
||||
|
||||
`application_defaults` is a partial `ApplicationConfig`. It is kept below the manifest's `backends` namespace because
|
||||
backend import declarations are part of plugin discovery and are not application configuration.
|
||||
|
||||
Use JSON when another local tool needs structured output:
|
||||
|
||||
```bash
|
||||
reme plugins list --json
|
||||
```
|
||||
|
||||
To compare installed plugins with one application config:
|
||||
|
||||
```bash
|
||||
reme plugins list --config daily_cookbook
|
||||
```
|
||||
|
||||
The optional `ENABLED` column reflects only the `plugins` list resolved from that config. A command-line override used
|
||||
by another running process is not a global enable state.
|
||||
|
||||
## Install a plugin package
|
||||
|
||||
Install a published distribution:
|
||||
|
||||
```bash
|
||||
reme plugins install reme-auto-fin
|
||||
```
|
||||
|
||||
Install or upgrade a pinned version:
|
||||
|
||||
```bash
|
||||
reme plugins install 'reme-auto-fin==0.1.0'
|
||||
reme plugins install reme-auto-fin --upgrade
|
||||
```
|
||||
|
||||
Install a local plugin project:
|
||||
|
||||
```bash
|
||||
reme plugins install ./plugins/auto-fin
|
||||
```
|
||||
|
||||
Use editable mode while developing it:
|
||||
|
||||
```bash
|
||||
reme plugins install ./plugins/auto-fin --editable
|
||||
```
|
||||
|
||||
ReMe invokes pip through the same Python interpreter that runs the `reme` command. Pip remains responsible for package
|
||||
resolution, downloads, dependency changes, and build execution. Install only packages and local projects you trust.
|
||||
|
||||
After installation, confirm the discovered plugin name:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
reme plugins validate auto-fin
|
||||
```
|
||||
|
||||
## Inspect a plugin
|
||||
|
||||
```bash
|
||||
reme plugins show auto-fin
|
||||
```
|
||||
|
||||
For a manifest plugin, the result includes its registered backend names and default Job names. JSON output is also
|
||||
available:
|
||||
|
||||
```bash
|
||||
reme plugins show auto-fin --json
|
||||
```
|
||||
|
||||
`show` identifies the package contract without constructing a ReMe Application.
|
||||
|
||||
## Validate a plugin
|
||||
|
||||
Validate an installed plugin:
|
||||
|
||||
```bash
|
||||
reme plugins validate auto-fin
|
||||
```
|
||||
|
||||
Validate a local project before installation:
|
||||
|
||||
```bash
|
||||
reme plugins validate ./plugins/auto-fin
|
||||
```
|
||||
|
||||
Validation checks the entry point, `plugin.yaml`, backend imports and component types, registry collisions, merged
|
||||
`application_defaults`, and the resulting `ApplicationConfig`. Validation imports plugin backend modules, so run it
|
||||
only for trusted code.
|
||||
|
||||
## Enable a plugin in a service
|
||||
|
||||
Installation alone does not load plugin code into an Application. Enable plugins explicitly in configuration:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
- auto-fin
|
||||
```
|
||||
|
||||
Or add them for one service launch:
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
When `config` is omitted, ReMe loads `default.yaml`. The plugin's `application_defaults` are merged below that config,
|
||||
so explicit config values and CLI overrides win. This mapping is an `ApplicationConfig` fragment, not a separate
|
||||
configuration schema. The plugin backends are registered only in that Application's local registry.
|
||||
|
||||
After the default HTTP service starts, access plugin Jobs through ReMe's CLI client or HTTP:
|
||||
|
||||
```bash
|
||||
reme auto_fin topics="黄金,AI,存储芯片"
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:2333/auto_fin \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"topics":"黄金,AI,存储芯片"}'
|
||||
```
|
||||
|
||||
When the application uses an MCP service, service-enabled plugin Jobs appear as MCP tools instead.
|
||||
|
||||
To add the plugin to another application config, select it explicitly:
|
||||
|
||||
```bash
|
||||
reme start config=daily_cookbook plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
## Uninstall a plugin
|
||||
|
||||
Use the plugin entry-point name, not necessarily the distribution name:
|
||||
|
||||
```bash
|
||||
reme plugins uninstall auto-fin
|
||||
```
|
||||
|
||||
Skip pip's confirmation prompt when needed:
|
||||
|
||||
```bash
|
||||
reme plugins uninstall auto-fin --yes
|
||||
```
|
||||
|
||||
ReMe resolves `auto-fin` to the distribution that provides it, such as `reme-auto-fin`. If one distribution provides
|
||||
multiple plugin entry points, the command lists the other plugins that will also be removed.
|
||||
|
||||
Uninstallation does not rewrite user configuration. Remove the plugin from relevant `plugins` lists yourself;
|
||||
otherwise the next Application startup fails explicitly because the configured plugin is no longer installed. Restart
|
||||
already-running ReMe processes after installing, upgrading, or uninstalling packages.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Plugin is installed but unavailable
|
||||
|
||||
Check that the `reme` command and pip package share one Python interpreter:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
python -c 'import sys; print(sys.executable)'
|
||||
```
|
||||
|
||||
Using `reme plugins install` avoids the most common interpreter mismatch because it runs `python -m pip` with ReMe's
|
||||
own interpreter.
|
||||
|
||||
### Plugin is installed but not loaded
|
||||
|
||||
Add its entry-point name to the Application's `plugins` list. ReMe intentionally has no global enable/disable state.
|
||||
|
||||
### Startup reports that the plugin is not installed
|
||||
|
||||
The active config still enables a missing plugin. Reinstall it or remove the corresponding name from `plugins`.
|
||||
|
||||
### Changes are not visible in a running service
|
||||
|
||||
Plugin discovery and backend registration happen during Application construction. Restart the service after changing
|
||||
installed packages.
|
||||
|
|
@ -221,12 +221,29 @@ class VersionStep(BaseStep):
|
|||
|
||||
内置实现通过 package import 填充内置注册表;bootstrap 完成后 ReMe 会冻结这个模板,并为每个 `Application` 创建可写副本。
|
||||
运行期代码通过当前 Application 的注册表解析 backend,不能修改进程级模板。随后只加载最终配置中 `plugins` 明确启用的已安装插件。
|
||||
插件通过 Python `reme.plugins` entry-point group 暴露,返回声明式 `reme.plugin.Plugin`,其中包含命名
|
||||
backend class 和默认配置。插件注册因此只影响当前 Application;两个插件提供相同 `(component_type, backend)` 时会在装配阶段失败,
|
||||
插件通过 Python `reme.plugins` entry-point group 暴露其 package。package 内的 `plugin.yaml` 只有两个可选 mapping:
|
||||
`backends` 将注册名映射到 `module:Class`,`application_defaults` 提供低优先级的 `ApplicationConfig` 配置片段。
|
||||
entry-point 名称就是插件标识;使用
|
||||
应用配置的 `plugins` 列表或 CLI 的 `plugins=[...]` override 显式启用插件。插件注册因此只影响当前 Application;两个插件提供相同
|
||||
`(component_type, backend)` 时会在装配阶段失败,
|
||||
不会互相覆盖。
|
||||
|
||||
插件还可以通过 `reme.configs` 暴露命名配置;配置的 `extends` 可以继承内置配置、插件配置或文件配置。完整打包示例见
|
||||
[Auto Fin 插件](../../plugins/auto-fin/README_ZH.md)。
|
||||
迁移期间仍兼容旧的 Python `Plugin` descriptor 和 `reme.configs` entry point。配置的 `extends` 可以继承内置配置、
|
||||
旧插件配置或文件配置。当前完整打包示例见 [Auto Fin 插件](../../plugins/auto-fin/README_ZH.md)。
|
||||
|
||||
插件包的本地管理与单个应用是否启用插件相互独立:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
reme plugins install reme-auto-fin
|
||||
reme plugins show auto-fin
|
||||
reme plugins validate auto-fin
|
||||
reme plugins uninstall auto-fin
|
||||
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
这些管理命令使用当前 Python 解释器对应的 pip,不通过 HTTP 或 MCP service 执行。
|
||||
|
||||
### 4.3 Component.bind
|
||||
|
||||
|
|
|
|||
219
docs/zh/plugin_management.md
Normal file
219
docs/zh/plugin_management.md
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
# 插件管理
|
||||
|
||||
ReMe 插件是通过 `reme.plugins` entry-point group 发现的普通 Python distribution。安装插件只表示它在当前 Python
|
||||
环境中可用,并不会让所有 ReMe Application 自动启用该插件。
|
||||
|
||||
需要区分两个操作:
|
||||
|
||||
```text
|
||||
reme plugins install ... 将插件包安装到当前 Python 环境
|
||||
plugins: [auto-fin] 为一个 Application 启用已安装插件
|
||||
```
|
||||
|
||||
插件包管理仅在本地 CLI 执行,不经过 ReMe HTTP 或 MCP service,也不会自动修改应用配置文件。
|
||||
|
||||
典型的插件使用流程分为三个阶段:
|
||||
|
||||
1. 安装 ReMe 和插件 distribution。
|
||||
2. 按照 [ReMe 环境变量说明](../../README_ZH.md#环境变量)配置插件运行所需的环境变量。
|
||||
3. 启动 Application 时显式启用插件,例如 `reme start plugins='["auto-fin"]'`。
|
||||
|
||||
## 查看已安装插件
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
```
|
||||
|
||||
输出包含插件 entry-point 名称、Python distribution、版本和插件契约:
|
||||
|
||||
```text
|
||||
PLUGIN DISTRIBUTION VERSION FORMAT
|
||||
-------- ------------- ------- --------
|
||||
auto-fin reme-auto-fin 0.1.0 manifest
|
||||
```
|
||||
|
||||
`manifest` 表示插件使用当前的 package-level `plugin.yaml` 契约;`legacy` 表示插件使用仍然兼容的 Python descriptor
|
||||
契约。
|
||||
|
||||
manifest 将 backend 注册与应用配置分开:
|
||||
|
||||
```yaml
|
||||
backends:
|
||||
example_step: example_plugin.steps:ExampleStep
|
||||
|
||||
application_defaults:
|
||||
jobs:
|
||||
example:
|
||||
backend: base
|
||||
steps:
|
||||
- backend: example_step
|
||||
```
|
||||
|
||||
`application_defaults` 是一段不完整的 `ApplicationConfig`。它与 manifest 的 `backends` 命名空间分开,因为 backend
|
||||
导入声明属于插件发现协议,并不是应用配置。
|
||||
|
||||
本地工具需要结构化结果时可以使用 JSON:
|
||||
|
||||
```bash
|
||||
reme plugins list --json
|
||||
```
|
||||
|
||||
对照某个应用配置查看启用状态:
|
||||
|
||||
```bash
|
||||
reme plugins list --config daily_cookbook
|
||||
```
|
||||
|
||||
可选的 `ENABLED` 列只反映该配置解析出的 `plugins` 列表。其他运行中进程使用的 CLI override 不是全局启用状态。
|
||||
|
||||
## 安装插件包
|
||||
|
||||
安装已发布的 distribution:
|
||||
|
||||
```bash
|
||||
reme plugins install reme-auto-fin
|
||||
```
|
||||
|
||||
安装指定版本或升级:
|
||||
|
||||
```bash
|
||||
reme plugins install 'reme-auto-fin==0.1.0'
|
||||
reme plugins install reme-auto-fin --upgrade
|
||||
```
|
||||
|
||||
安装本地插件项目:
|
||||
|
||||
```bash
|
||||
reme plugins install ./plugins/auto-fin
|
||||
```
|
||||
|
||||
开发插件时使用 editable 模式:
|
||||
|
||||
```bash
|
||||
reme plugins install ./plugins/auto-fin --editable
|
||||
```
|
||||
|
||||
ReMe 会通过运行 `reme` 命令的同一个 Python 解释器调用 pip。包解析、下载、依赖变更和构建执行仍由 pip 负责。请只安装
|
||||
可信的包和本地项目。
|
||||
|
||||
安装后确认 ReMe 实际发现的插件名:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
reme plugins validate auto-fin
|
||||
```
|
||||
|
||||
## 查看插件详情
|
||||
|
||||
```bash
|
||||
reme plugins show auto-fin
|
||||
```
|
||||
|
||||
对于 manifest 插件,结果包含注册的 backend 名称和默认 Job 名称。也可以输出 JSON:
|
||||
|
||||
```bash
|
||||
reme plugins show auto-fin --json
|
||||
```
|
||||
|
||||
`show` 只检查包契约,不构造 ReMe Application。
|
||||
|
||||
## 校验插件
|
||||
|
||||
校验已安装插件:
|
||||
|
||||
```bash
|
||||
reme plugins validate auto-fin
|
||||
```
|
||||
|
||||
安装前校验本地插件项目:
|
||||
|
||||
```bash
|
||||
reme plugins validate ./plugins/auto-fin
|
||||
```
|
||||
|
||||
校验范围包括 entry point、`plugin.yaml`、backend 导入和组件类型、registry 冲突、`application_defaults` 合并以及最终的
|
||||
`ApplicationConfig`。校验过程会导入插件 backend 模块,因此只能对可信代码执行。
|
||||
|
||||
## 在服务中启用插件
|
||||
|
||||
只安装插件不会将插件代码加载到 Application。需要在配置中显式启用:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
- auto-fin
|
||||
```
|
||||
|
||||
也可以只为本次服务启动追加插件:
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
未传入 `config` 时,ReMe 加载 `default.yaml`。插件的 `application_defaults` 合并在该配置之下,因此显式配置和 CLI
|
||||
override 优先。这个 mapping 是 `ApplicationConfig` 配置片段,并不是另一套配置 schema。插件 backend 只注册到该
|
||||
Application 的局部 registry。
|
||||
|
||||
默认 HTTP service 启动后,可以通过 ReMe CLI client 或 HTTP 访问插件 Job:
|
||||
|
||||
```bash
|
||||
reme auto_fin topics="黄金,AI,存储芯片"
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:2333/auto_fin \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"topics":"黄金,AI,存储芯片"}'
|
||||
```
|
||||
|
||||
当应用使用 MCP service 时,允许对外服务的插件 Job 会显示为 MCP tool。
|
||||
|
||||
如果需要将插件叠加到其他应用配置,则显式选择该配置:
|
||||
|
||||
```bash
|
||||
reme start config=daily_cookbook plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
## 卸载插件
|
||||
|
||||
这里使用插件 entry-point 名称,它不一定等于 distribution 名称:
|
||||
|
||||
```bash
|
||||
reme plugins uninstall auto-fin
|
||||
```
|
||||
|
||||
需要跳过 pip 确认时:
|
||||
|
||||
```bash
|
||||
reme plugins uninstall auto-fin --yes
|
||||
```
|
||||
|
||||
ReMe 会将 `auto-fin` 解析为提供它的 distribution,例如 `reme-auto-fin`。如果一个 distribution 提供多个插件 entry
|
||||
point,命令会列出同时被移除的其他插件。
|
||||
|
||||
卸载不会重写用户配置。请自行从相关 `plugins` 列表中删除插件,否则下一次启动 Application 时会因为配置的插件未安装而明确
|
||||
失败。安装、升级或卸载包后,需要重启已经运行的 ReMe 进程。
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 插件已经安装,但 ReMe 找不到
|
||||
|
||||
检查 `reme` 命令与安装插件使用的 pip 是否属于同一个 Python 解释器:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
python -c 'import sys; print(sys.executable)'
|
||||
```
|
||||
|
||||
使用 `reme plugins install` 可以避免最常见的解释器不一致问题,因为它通过 ReMe 自己的解释器运行 `python -m pip`。
|
||||
|
||||
### 插件已经安装,但没有加载
|
||||
|
||||
将插件 entry-point 名称加入 Application 的 `plugins` 列表。ReMe 刻意不提供全局 enable/disable 状态。
|
||||
|
||||
### 启动时报插件未安装
|
||||
|
||||
当前配置仍然启用了缺失插件。请重新安装插件,或者从 `plugins` 中删除对应名称。
|
||||
|
||||
### 运行中的服务看不到插件变化
|
||||
|
||||
插件发现和 backend 注册发生在 Application 构造阶段。修改已安装包后需要重启服务。
|
||||
|
|
@ -8,6 +8,7 @@ const outputDir = path.join(siteDir, ".generated", "content");
|
|||
|
||||
const topicOrder = [
|
||||
"quick_start",
|
||||
"plugin_management",
|
||||
"memory_as_file",
|
||||
"memory_search",
|
||||
"auto_memory",
|
||||
|
|
@ -23,6 +24,7 @@ const topicOrder = [
|
|||
|
||||
const groups = {
|
||||
quick_start: "start",
|
||||
plugin_management: "start",
|
||||
memory_as_file: "fundamentals",
|
||||
memory_search: "fundamentals",
|
||||
auto_memory: "automation",
|
||||
|
|
@ -38,6 +40,7 @@ const groups = {
|
|||
|
||||
const localizedTitles = {
|
||||
quick_start: { zh: "快速开始", en: "Quick Start" },
|
||||
plugin_management: { zh: "插件管理", en: "Plugin Management" },
|
||||
memory_as_file: { zh: "文件即记忆", en: "Memory as File" },
|
||||
memory_search: { zh: "记忆检索", en: "Memory Search" },
|
||||
auto_memory: { zh: "自动记忆", en: "Auto Memory" },
|
||||
|
|
|
|||
|
|
@ -1,6 +1,24 @@
|
|||
# ReMe Plugins
|
||||
|
||||
This directory contains installable extensions of ReMe itself. A plugin may contribute components, steps, jobs, and
|
||||
configuration through the `reme.plugins` and `reme.configs` Python entry-point groups.
|
||||
This directory contains installable extensions of ReMe itself. A plugin exposes its package through the `reme.plugins`
|
||||
Python entry-point group and declares backend classes plus a low-priority `ApplicationConfig` fragment under
|
||||
`application_defaults` in that package's `plugin.yaml`.
|
||||
|
||||
Manage plugin packages with the local CLI:
|
||||
|
||||
```bash
|
||||
reme plugins list
|
||||
reme plugins show auto-fin
|
||||
reme plugins install reme-auto-fin
|
||||
reme plugins install ./plugins/auto-fin --editable
|
||||
reme plugins validate auto-fin
|
||||
reme plugins uninstall auto-fin
|
||||
```
|
||||
|
||||
Installation and activation are separate. Enable an installed plugin for one application through its config or CLI:
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
Adapters for external agent hosts belong in [`../integrations`](../integrations/README.md).
|
||||
|
|
|
|||
|
|
@ -5,32 +5,64 @@
|
|||
Auto Fin fetches a rolling window of CLS telegraph news (24 hours by default), selects items related to configured
|
||||
topics, searches ReMe for useful historical context, and writes one Chinese Markdown report with validated wikilinks.
|
||||
Current news and topic selection stay in runtime memory; only the final report becomes durable memory. This directory
|
||||
is an independent Python distribution. Its `reme.plugins` entry point contributes the three Step backends and their Job
|
||||
configuration; its `reme.configs` entry point exposes the runnable `auto-fin` configuration.
|
||||
is an independent Python distribution. Its single `reme.plugins` entry point exposes a `plugin.yaml` containing the
|
||||
three Step backends and their Job configuration under `application_defaults`. Enable the installed plugin explicitly
|
||||
through `plugins=["auto-fin"]`.
|
||||
|
||||
> Auto Fin has no reliable market-price feed. It does not calculate returns, targets, or entry points and is not
|
||||
> investment advice.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
python -m pip install "reme-ai[core]>=0.4.1.8" reme-auto-fin
|
||||
export LLM_API_KEY="your-api-key"
|
||||
export LLM_MODEL_NAME="qwen3.7-plus"
|
||||
export LLM_BASE_URL="https://your-provider.example/v1"
|
||||
reme start config=auto-fin job=auto_fin
|
||||
```
|
||||
|
||||
`LLM_MODEL_NAME` defaults to `qwen3.7-plus`. There is no built-in `LLM_BASE_URL`, so set the OpenAI-compatible endpoint
|
||||
required by the selected provider.
|
||||
|
||||
The default topics are `黄金,机器人,半导体`. Override them per run:
|
||||
### 1. Install ReMe and Auto Fin
|
||||
|
||||
```bash
|
||||
reme start config=auto-fin job=auto_fin topics="黄金,AI,存储芯片"
|
||||
python -m pip install "reme-ai[core]>=0.4.1.8"
|
||||
reme plugins install reme-auto-fin
|
||||
```
|
||||
|
||||
An empty value also uses the defaults.
|
||||
### 2. Configure the model environment
|
||||
|
||||
Configure the LLM environment variables as described in the
|
||||
[ReMe README](../../README.md#environment-variables). Other compatible models and providers can also be used.
|
||||
|
||||
### 3. Start ReMe with the plugin
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
With no explicit `config`, ReMe loads `default.yaml` and adds the plugin to that service.
|
||||
|
||||
From another terminal, call the running HTTP service through ReMe's CLI client:
|
||||
|
||||
```bash
|
||||
reme auto_fin topics="黄金,AI,存储芯片"
|
||||
```
|
||||
|
||||
Or call its HTTP endpoint directly:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:2333/auto_fin \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"topics":"黄金,AI,存储芯片"}'
|
||||
```
|
||||
|
||||
When enabled on an MCP service, the same Job is exposed as the `auto_fin` MCP tool. The default topics are
|
||||
`黄金,机器人,半导体`; an empty value also uses these defaults.
|
||||
|
||||
To host the same application as an MCP service instead:
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]' \
|
||||
service.backend=mcp service.transport=streamable-http
|
||||
```
|
||||
|
||||
To add Auto Fin to another application instead, select that config explicitly, for example:
|
||||
|
||||
```bash
|
||||
reme start config=daily_cookbook plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
## Pipeline
|
||||
|
||||
|
|
@ -76,7 +108,7 @@ refreshes the daily index. No JSONL, intermediate Markdown, or structured Agent
|
|||
| `request_interval` | `10` | Minimum delay in seconds after every CLS request attempt; may be zero |
|
||||
| `max_retries` | `3` | Maximum attempts for each CLS page request; must be at least one |
|
||||
|
||||
The plugin-provided schedules run daily at 09:30, 11:30, and 18:00 in `Asia/Shanghai`.
|
||||
The three plugin cron Jobs start with the application and run daily at 09:30, 11:30, and 18:00 in `Asia/Shanghai`.
|
||||
|
||||
## Output
|
||||
|
||||
|
|
|
|||
|
|
@ -4,30 +4,61 @@
|
|||
|
||||
Auto Fin 自动拉取一个滚动时间窗口内的财联社电报(默认 24 小时),按配置 topics 筛选相关新闻,搜索 ReMe 中有回顾价值的历史材料,最后写入一份带校验
|
||||
wikilink 的中文 Markdown 报告。当前新闻和筛选结果只存在于本次运行内存中,只有最终报告成为持久记忆。本目录是一个独立 Python
|
||||
distribution:`reme.plugins` entry point 贡献三个 Step backend 及其 Job 配置,`reme.configs` entry point 暴露可直接运行的
|
||||
`auto-fin` 配置。
|
||||
distribution:单个 `reme.plugins` entry point 暴露 `plugin.yaml`,其中声明三个 Step backend,并在
|
||||
`application_defaults` 下提供 Job 配置;通过 `plugins=["auto-fin"]` 显式启用这个已安装插件。
|
||||
|
||||
> Auto Fin 没有可靠行情数据,不计算收益、目标价或买卖点,也不提供投资建议。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
python -m pip install "reme-ai[core]>=0.4.1.8" reme-auto-fin
|
||||
export LLM_API_KEY="your-api-key"
|
||||
export LLM_MODEL_NAME="qwen3.7-plus"
|
||||
export LLM_BASE_URL="https://your-provider.example/v1"
|
||||
reme start config=auto-fin job=auto_fin
|
||||
```
|
||||
|
||||
`LLM_MODEL_NAME` 默认是 `qwen3.7-plus`。代码没有内置 `LLM_BASE_URL`,请设置所选服务商提供的 OpenAI 兼容 endpoint。
|
||||
|
||||
默认 topics 是 `黄金,机器人,半导体`。可在运行时覆盖:
|
||||
### 1. 安装 ReMe 和 Auto Fin
|
||||
|
||||
```bash
|
||||
reme start config=auto-fin job=auto_fin topics="黄金,AI,存储芯片"
|
||||
python -m pip install "reme-ai[core]>=0.4.1.8"
|
||||
reme plugins install reme-auto-fin
|
||||
```
|
||||
|
||||
传入空值也会使用默认 topics。
|
||||
### 2. 配置模型环境变量
|
||||
|
||||
按照 ReMe README 的[环境变量说明](../../README_ZH.md#环境变量)配置 LLM 环境变量,也可以使用其他兼容的模型和服务商。
|
||||
|
||||
### 3. 带插件启动 ReMe
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
未显式传入 `config` 时,ReMe 会加载 `default.yaml`,并将插件叠加到该服务上。
|
||||
|
||||
在另一个终端中,通过 ReMe CLI client 调用正在运行的 HTTP 服务:
|
||||
|
||||
```bash
|
||||
reme auto_fin topics="黄金,AI,存储芯片"
|
||||
```
|
||||
|
||||
也可以直接调用 HTTP endpoint:
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:2333/auto_fin \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"topics":"黄金,AI,存储芯片"}'
|
||||
```
|
||||
|
||||
在 MCP service 中启用插件时,同一个 Job 会暴露为 `auto_fin` MCP tool。默认 topics 是 `黄金,机器人,半导体`,
|
||||
传入空值也会使用默认值。
|
||||
|
||||
如果需要将同一个应用作为 MCP service 启动:
|
||||
|
||||
```bash
|
||||
reme start plugins='["auto-fin"]' \
|
||||
service.backend=mcp service.transport=streamable-http
|
||||
```
|
||||
|
||||
如果需要将 Auto Fin 叠加到其他应用,则显式选择相应配置,例如:
|
||||
|
||||
```bash
|
||||
reme start config=daily_cookbook plugins='["auto-fin"]'
|
||||
```
|
||||
|
||||
## 流程
|
||||
|
||||
|
|
@ -69,7 +100,7 @@ workspace 的 Markdown 目标。不存在、绝对路径、越界、带反斜杠
|
|||
| `request_interval` | `10` | 每次财联社请求尝试后的最小等待秒数,可设为 0 |
|
||||
| `max_retries` | `3` | 每页财联社请求的最大尝试次数,至少为 1 |
|
||||
|
||||
插件提供的定时任务每天按 `Asia/Shanghai` 在 09:30、11:30 和 18:00 运行。
|
||||
插件的三个 cron Job 随应用启动,并按 `Asia/Shanghai` 时区在每天 09:30、11:30 和 18:00 运行。
|
||||
|
||||
## 产物
|
||||
|
||||
|
|
|
|||
|
|
@ -11,10 +11,7 @@ dependencies = [
|
|||
]
|
||||
|
||||
[project.entry-points."reme.plugins"]
|
||||
auto-fin = "reme_auto_fin:plugin"
|
||||
|
||||
[project.entry-points."reme.configs"]
|
||||
auto-fin = "reme_auto_fin:CONFIG_PATH"
|
||||
auto-fin = "reme_auto_fin"
|
||||
|
||||
[tool.setuptools]
|
||||
package-dir = { "" = "src" }
|
||||
|
|
|
|||
|
|
@ -1,21 +1,14 @@
|
|||
"""Auto Fin news research workflow."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from .data import AutoFinDataStep
|
||||
from .merge import AutoFinMergeStep
|
||||
from .plugin import plugin
|
||||
from .schema import AutoFinReportOutput, AutoFinTopicOutput
|
||||
from .topic import AutoFinTopicStep
|
||||
|
||||
CONFIG_PATH = Path(__file__).with_name("config.yaml")
|
||||
|
||||
__all__ = [
|
||||
"AutoFinReportOutput",
|
||||
"AutoFinDataStep",
|
||||
"AutoFinMergeStep",
|
||||
"AutoFinTopicOutput",
|
||||
"AutoFinTopicStep",
|
||||
"CONFIG_PATH",
|
||||
"plugin",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,2 +0,0 @@
|
|||
extends: default
|
||||
plugins: [auto-fin]
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
jobs:
|
||||
auto_fin:
|
||||
backend: base
|
||||
description: "Fetch and research recent topic-related CLS news."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "Current date in YYYY-MM-DD; empty means today in Asia/Shanghai."
|
||||
default: ""
|
||||
now:
|
||||
type: string
|
||||
description: "Optional simulated current time in ISO 8601 format; empty means the real current time."
|
||||
default: ""
|
||||
topics:
|
||||
type: string
|
||||
description: "Comma-separated topics used to filter current CLS news."
|
||||
default: "黄金,机器人,半导体"
|
||||
window_hours:
|
||||
type: number
|
||||
exclusiveMinimum: 0
|
||||
description: "Rolling number of hours of CLS news to fetch."
|
||||
default: 24
|
||||
request_interval:
|
||||
type: number
|
||||
minimum: 0
|
||||
description: "Minimum delay in seconds after each CLS request attempt."
|
||||
default: 10
|
||||
max_retries:
|
||||
type: integer
|
||||
minimum: 1
|
||||
description: "Maximum attempts for each CLS page request."
|
||||
default: 3
|
||||
steps: &auto_fin_steps
|
||||
- backend: auto_fin_data_step
|
||||
- backend: auto_fin_topic_step
|
||||
- backend: auto_fin_merge_step
|
||||
job_tools: [memory_search, read]
|
||||
|
||||
auto_fin_0930_cron:
|
||||
backend: cron
|
||||
cron: "30 9 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
||||
auto_fin_1130_cron:
|
||||
backend: cron
|
||||
cron: "30 11 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
||||
auto_fin_1800_cron:
|
||||
backend: cron
|
||||
cron: "0 18 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""Auto Fin plugin declaration."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from reme.plugin import Backend, Plugin
|
||||
|
||||
from .data import AutoFinDataStep
|
||||
from .merge import AutoFinMergeStep
|
||||
from .topic import AutoFinTopicStep
|
||||
|
||||
|
||||
def _default_config() -> dict:
|
||||
path = Path(__file__).with_name("defaults.yaml")
|
||||
with path.open(encoding="utf-8") as stream:
|
||||
config = yaml.safe_load(stream)
|
||||
if not isinstance(config, dict):
|
||||
raise ValueError(f"Auto Fin defaults must be a mapping: {path}")
|
||||
return config
|
||||
|
||||
|
||||
plugin = Plugin(
|
||||
name="auto-fin",
|
||||
backends=(
|
||||
Backend("auto_fin_data_step", AutoFinDataStep),
|
||||
Backend("auto_fin_topic_step", AutoFinTopicStep),
|
||||
Backend("auto_fin_merge_step", AutoFinMergeStep),
|
||||
),
|
||||
config=_default_config(),
|
||||
)
|
||||
60
plugins/auto-fin/src/reme_auto_fin/plugin.yaml
Normal file
60
plugins/auto-fin/src/reme_auto_fin/plugin.yaml
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
backends:
|
||||
auto_fin_data_step: reme_auto_fin.data:AutoFinDataStep
|
||||
auto_fin_topic_step: reme_auto_fin.topic:AutoFinTopicStep
|
||||
auto_fin_merge_step: reme_auto_fin.merge:AutoFinMergeStep
|
||||
|
||||
application_defaults:
|
||||
jobs:
|
||||
auto_fin:
|
||||
backend: base
|
||||
description: "Fetch and research recent topic-related CLS news."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "Current date in YYYY-MM-DD; empty means today in Asia/Shanghai."
|
||||
default: ""
|
||||
now:
|
||||
type: string
|
||||
description: "Optional simulated current time in ISO 8601 format; empty means the real current time."
|
||||
default: ""
|
||||
topics:
|
||||
type: string
|
||||
description: "Comma-separated topics used to filter current CLS news."
|
||||
default: "黄金,机器人,半导体"
|
||||
window_hours:
|
||||
type: number
|
||||
exclusiveMinimum: 0
|
||||
description: "Rolling number of hours of CLS news to fetch."
|
||||
default: 24
|
||||
request_interval:
|
||||
type: number
|
||||
minimum: 0
|
||||
description: "Minimum delay in seconds after each CLS request attempt."
|
||||
default: 10
|
||||
max_retries:
|
||||
type: integer
|
||||
minimum: 1
|
||||
description: "Maximum attempts for each CLS page request."
|
||||
default: 3
|
||||
steps: &auto_fin_steps
|
||||
- backend: auto_fin_data_step
|
||||
- backend: auto_fin_topic_step
|
||||
- backend: auto_fin_merge_step
|
||||
job_tools: [memory_search, read]
|
||||
|
||||
auto_fin_0930_cron:
|
||||
backend: cron
|
||||
cron: "30 9 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
||||
auto_fin_1130_cron:
|
||||
backend: cron
|
||||
cron: "30 11 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
||||
auto_fin_1800_cron:
|
||||
backend: cron
|
||||
cron: "0 18 * * *"
|
||||
steps: *auto_fin_steps
|
||||
|
|
@ -7,11 +7,11 @@ from pathlib import Path
|
|||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from reme_auto_fin.base import _plain_text, _write
|
||||
from reme_auto_fin.data import AutoFinDataStep
|
||||
from reme_auto_fin.merge import AutoFinMergeStep
|
||||
from reme_auto_fin.plugin import plugin
|
||||
from reme_auto_fin.schema import AutoFinReportOutput, AutoFinTopicOutput
|
||||
from reme_auto_fin.topic import AutoFinTopicStep
|
||||
from reme.components import ApplicationContext
|
||||
|
|
@ -19,6 +19,9 @@ from reme.components.agent_wrapper.base_agent_wrapper import BaseAgentWrapper
|
|||
from reme.components.runtime_context import RuntimeContext
|
||||
|
||||
SHANGHAI = ZoneInfo("Asia/Shanghai")
|
||||
PLUGIN_MANIFEST = yaml.safe_load(
|
||||
(Path(__file__).parents[1] / "src" / "reme_auto_fin" / "plugin.yaml").read_text(encoding="utf-8"),
|
||||
)
|
||||
|
||||
|
||||
def _row(news_id: int, value: datetime, title: str = "新闻", content: str = "正文") -> dict:
|
||||
|
|
@ -239,7 +242,8 @@ def test_hybrid_wikilink_normalization_is_conservative_and_failure_safe(tmp_path
|
|||
|
||||
|
||||
def test_plugin_config_has_default_topics_and_no_intermediate_index_step():
|
||||
job = plugin.config["jobs"]["auto_fin"]
|
||||
jobs = PLUGIN_MANIFEST["application_defaults"]["jobs"]
|
||||
job = jobs["auto_fin"]
|
||||
assert job["parameters"]["properties"]["topics"]["default"] == "黄金,机器人,半导体"
|
||||
assert job["parameters"]["properties"]["window_hours"]["default"] == 24
|
||||
assert job["parameters"]["properties"]["request_interval"]["default"] == 10
|
||||
|
|
@ -256,8 +260,8 @@ def test_plugin_config_has_default_topics_and_no_intermediate_index_step():
|
|||
"auto_fin_1130_cron": "30 11 * * *",
|
||||
"auto_fin_1800_cron": "0 18 * * *",
|
||||
}.items():
|
||||
assert plugin.config["jobs"][name]["cron"] == schedule
|
||||
assert plugin.config["jobs"][name]["steps"] == job["steps"]
|
||||
assert jobs[name]["cron"] == schedule
|
||||
assert jobs[name]["steps"] == job["steps"]
|
||||
|
||||
|
||||
def test_agent_schemas_are_small_and_required():
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ from pathlib import Path
|
|||
from typing import AsyncGenerator, TypeVar
|
||||
|
||||
from . import __version__
|
||||
from .components import ApplicationContext, BaseComponent, create_application_registry
|
||||
from .components import ApplicationContext, BaseComponent
|
||||
from .components.job import BackgroundJob, BaseJob, CronJob, StreamJob
|
||||
from .components.service import BaseService
|
||||
from .enumeration import ComponentEnum, ComponentType, component_type_name
|
||||
from .plugin import PluginManager
|
||||
from .plugin import resolve_plugin_runtime
|
||||
from .schema import ComponentConfig, Response, StreamChunk
|
||||
from .utils import execute_stream_task, print_logo, get_logger
|
||||
|
||||
|
|
@ -23,11 +23,8 @@ class Application(BaseComponent):
|
|||
"""Wires components from config and runs jobs against them."""
|
||||
|
||||
def __init__(self, **kwargs) -> None:
|
||||
plugin_manager = PluginManager.discover(kwargs.get("plugins") or ())
|
||||
resolved = plugin_manager.merge_config(kwargs)
|
||||
registry = create_application_registry()
|
||||
plugin_manager.register(registry)
|
||||
self.context = ApplicationContext(registry=registry, **resolved)
|
||||
runtime = resolve_plugin_runtime(kwargs)
|
||||
self.context = ApplicationContext(registry=runtime.registry, **runtime.config)
|
||||
self._started_components: list[BaseComponent] = []
|
||||
|
||||
self._setup_workspace_directories()
|
||||
|
|
|
|||
|
|
@ -1,10 +1,19 @@
|
|||
"""Config"""
|
||||
|
||||
from .config_parser import deep_merge_config, expand_env_vars, parse_args, resolve_app_config
|
||||
from .config_parser import (
|
||||
deep_merge_config,
|
||||
expand_env_vars,
|
||||
parse_action,
|
||||
parse_args,
|
||||
parse_kwargs,
|
||||
resolve_app_config,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"deep_merge_config",
|
||||
"expand_env_vars",
|
||||
"parse_action",
|
||||
"parse_args",
|
||||
"parse_kwargs",
|
||||
"resolve_app_config",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -10,13 +10,17 @@ from typing import Any
|
|||
|
||||
import yaml
|
||||
|
||||
from ..entry_point import find_entry_points, load_entry_point, unique_entry_point
|
||||
from ..entry_point import (
|
||||
CONFIG_ENTRY_POINT_GROUP,
|
||||
find_entry_points,
|
||||
load_entry_point,
|
||||
unique_entry_point,
|
||||
)
|
||||
|
||||
# Config files are looked up relative to this module's directory
|
||||
_CONFIG_DIR = Path(__file__).parent
|
||||
# Extensions in priority order: yaml > yml > json when stems collide
|
||||
_SUPPORTED_EXTS = (".yaml", ".yml", ".json")
|
||||
_CONFIG_ENTRY_POINT_GROUP = "reme.configs"
|
||||
_ENV_VAR_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-([^}]*))?}")
|
||||
# Strings like "007" / "00501" must stay as strings, not be coerced to numbers
|
||||
_LEADING_ZERO_RE = re.compile(r"^-?0\d")
|
||||
|
|
@ -145,7 +149,7 @@ def _load_config(name_or_path: str, encoding: str = "utf-8", _stack: tuple[str,
|
|||
raise ValueError(f"Circular config inheritance: {chain}")
|
||||
|
||||
built_in = _CONFIG_REGISTRY.get(name_or_path)
|
||||
external_entries = find_entry_points(_CONFIG_ENTRY_POINT_GROUP, name_or_path)
|
||||
external_entries = find_entry_points(CONFIG_ENTRY_POINT_GROUP, name_or_path)
|
||||
if built_in is not None and external_entries:
|
||||
raise ValueError(f"Config '{name_or_path}' is provided by both ReMe and an installed distribution")
|
||||
if built_in is not None:
|
||||
|
|
@ -222,8 +226,29 @@ def _strip_arg_dashes(arg: str) -> str:
|
|||
return arg
|
||||
|
||||
|
||||
def parse_args(*args) -> tuple[str, dict]:
|
||||
"""Parse CLI args: first arg is action, rest are key=value pairs.
|
||||
def parse_action(arg: str) -> str:
|
||||
"""Parse and validate one top-level CLI action."""
|
||||
action = _strip_arg_dashes(arg)
|
||||
if "=" in action:
|
||||
raise ValueError(f"First argument must be action, got: {arg}")
|
||||
return action
|
||||
|
||||
|
||||
def parse_kwargs(*args: str) -> dict:
|
||||
"""Parse application-style ``key=value`` CLI arguments."""
|
||||
kvs: list[str] = []
|
||||
for raw in args:
|
||||
arg = _strip_arg_dashes(raw)
|
||||
if "=" in arg:
|
||||
kvs.append(arg)
|
||||
else:
|
||||
raise ValueError(f"Invalid argument format (expected key=value): {raw}")
|
||||
|
||||
return parse_dot_notation(kvs) if kvs else {}
|
||||
|
||||
|
||||
def parse_args(*args: str) -> tuple[str, dict]:
|
||||
"""Parse an application CLI action followed by ``key=value`` arguments.
|
||||
|
||||
Usage: reme app config=paw.yaml service.name=test
|
||||
Returns: (action, parsed_kv_dict)
|
||||
|
|
@ -231,26 +256,16 @@ def parse_args(*args) -> tuple[str, dict]:
|
|||
if not args:
|
||||
raise ValueError("No arguments provided")
|
||||
|
||||
first = _strip_arg_dashes(args[0])
|
||||
if "=" in first:
|
||||
raise ValueError(f"First argument must be action, got: {args[0]}")
|
||||
|
||||
kvs: list[str] = []
|
||||
for raw in args[1:]:
|
||||
arg = _strip_arg_dashes(raw)
|
||||
if "=" in arg:
|
||||
kvs.append(arg)
|
||||
else:
|
||||
raise ValueError(f"Invalid argument format (expected key=value): {raw}")
|
||||
|
||||
parsed = parse_dot_notation(kvs) if kvs else {}
|
||||
return first, parsed
|
||||
return parse_action(args[0]), parse_kwargs(*args[1:])
|
||||
|
||||
|
||||
def resolve_app_config(*, log_config: bool = True, **kwargs) -> dict:
|
||||
"""Resolve full app-start config: load `config=path` file, fall back to
|
||||
`default`, then deep-merge with the remaining kwargs as overrides.
|
||||
|
||||
Therefore ``reme start plugins=[...]`` layers that plugin selection over
|
||||
``default.yaml`` without requiring an explicit ``config=default``.
|
||||
|
||||
Set ``log_config=False`` for user-facing client calls that should print only
|
||||
the requested job's output.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -3,6 +3,9 @@
|
|||
from importlib import metadata
|
||||
from typing import Any
|
||||
|
||||
PLUGIN_ENTRY_POINT_GROUP = "reme.plugins"
|
||||
CONFIG_ENTRY_POINT_GROUP = "reme.configs"
|
||||
|
||||
|
||||
def find_entry_points(group: str, name: str) -> list[metadata.EntryPoint]:
|
||||
"""Return all matching entry points without importing their providers."""
|
||||
|
|
|
|||
|
|
@ -1,17 +1,23 @@
|
|||
"""Installed plugin discovery and application-local registration."""
|
||||
"""Load explicitly enabled plugins into one application's config and registry.
|
||||
|
||||
New plugins expose a package-only ``reme.plugins`` entry point and declare two
|
||||
optional mappings in ``plugin.yaml``: ``backends`` and ``application_defaults``.
|
||||
Python ``Plugin`` descriptors remain supported as a compatibility boundary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from importlib import import_module
|
||||
from importlib.metadata import EntryPoint
|
||||
from typing import Any
|
||||
|
||||
from .components.base_component import ComponentMixin
|
||||
from .components.component_registry import ComponentRegistry
|
||||
from .components.component_registry import ComponentRegistry, create_application_registry
|
||||
from .config import deep_merge_config, expand_env_vars
|
||||
from .entry_point import find_entry_points, load_entry_point, unique_entry_point
|
||||
|
||||
PLUGIN_ENTRY_POINT_GROUP = "reme.plugins"
|
||||
from .entry_point import PLUGIN_ENTRY_POINT_GROUP, find_entry_points, load_entry_point, unique_entry_point
|
||||
from .plugin_manifest import PluginManifest, load_package_manifest
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -24,13 +30,62 @@ class Backend:
|
|||
|
||||
@dataclass(frozen=True)
|
||||
class Plugin:
|
||||
"""Declarative plugin loaded from the ``reme.plugins`` entry-point group."""
|
||||
"""Legacy Python descriptor accepted during the plugin manifest migration."""
|
||||
|
||||
name: str
|
||||
backends: tuple[Backend, ...] = ()
|
||||
config: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PluginRuntime:
|
||||
"""Application config and registry after applying enabled plugins."""
|
||||
|
||||
config: dict[str, Any]
|
||||
registry: ComponentRegistry
|
||||
|
||||
|
||||
def _load_backend(target: str, *, plugin_name: str) -> type[ComponentMixin]:
|
||||
"""Import one ``module:class`` backend target from a plugin manifest."""
|
||||
module_name, separator, attribute = target.partition(":")
|
||||
if not separator or not module_name or not attribute or ":" in attribute:
|
||||
raise ValueError(f"Plugin '{plugin_name}' has invalid backend target: {target!r}")
|
||||
try:
|
||||
value: Any = import_module(module_name)
|
||||
for part in attribute.split("."):
|
||||
value = getattr(value, part)
|
||||
except (AttributeError, ImportError) as exc:
|
||||
raise ValueError(f"Plugin '{plugin_name}' cannot load backend '{target}': {exc}") from exc
|
||||
if not isinstance(value, type) or not issubclass(value, ComponentMixin):
|
||||
raise TypeError(f"Plugin '{plugin_name}' backend '{target}' is not a ComponentMixin class")
|
||||
return value
|
||||
|
||||
|
||||
def _plugin_from_manifest(name: str, manifest: PluginManifest) -> Plugin:
|
||||
"""Convert a parsed manifest into one runtime plugin descriptor."""
|
||||
backends = tuple(
|
||||
Backend(backend_name, _load_backend(target, plugin_name=name))
|
||||
for backend_name, target in manifest.backends.items()
|
||||
)
|
||||
return Plugin(name=name, backends=backends, config=manifest.application_defaults)
|
||||
|
||||
|
||||
def _load_plugin(name: str, entry: EntryPoint) -> Plugin:
|
||||
"""Load a package manifest, falling back to the legacy descriptor form."""
|
||||
if ":" not in entry.value:
|
||||
# Package-only targets use plugin.yaml; package:object targets are the
|
||||
# legacy Python descriptor contract.
|
||||
from .components.component_registry import R
|
||||
|
||||
with R.preserve(allow_mutation=True):
|
||||
manifest = load_package_manifest(entry.value, plugin_name=name)
|
||||
return _plugin_from_manifest(name, manifest)
|
||||
plugin = load_entry_point(entry, invoke=True)
|
||||
if not isinstance(plugin, Plugin):
|
||||
raise TypeError(f"Plugin entry point '{name}' did not return reme.plugin.Plugin")
|
||||
return plugin
|
||||
|
||||
|
||||
class PluginManager:
|
||||
"""Resolve enabled plugins and apply their contributions to one application."""
|
||||
|
||||
|
|
@ -53,9 +108,7 @@ class PluginManager:
|
|||
entry = unique_entry_point(entries, name, provider="Plugin")
|
||||
if entry is None:
|
||||
raise ValueError(f"Plugin '{name}' is not installed")
|
||||
plugin = load_entry_point(entry, invoke=True)
|
||||
if not isinstance(plugin, Plugin):
|
||||
raise TypeError(f"Plugin entry point '{name}' did not return reme.plugin.Plugin")
|
||||
plugin = _load_plugin(name, entry)
|
||||
if plugin.name != name:
|
||||
raise ValueError(f"Plugin entry point '{name}' returned plugin '{plugin.name}'")
|
||||
plugins.append(plugin)
|
||||
|
|
@ -63,7 +116,7 @@ class PluginManager:
|
|||
return cls(plugins)
|
||||
|
||||
def merge_config(self, application_config: Mapping[str, Any]) -> dict[str, Any]:
|
||||
"""Place plugin defaults below the user's resolved application config."""
|
||||
"""Place plugin application defaults below the user's resolved config."""
|
||||
merged: dict[str, Any] = {}
|
||||
for plugin in self.plugins:
|
||||
merged = deep_merge_config(merged, expand_env_vars(plugin.config))
|
||||
|
|
@ -74,3 +127,11 @@ class PluginManager:
|
|||
for plugin in self.plugins:
|
||||
for backend in plugin.backends:
|
||||
registry.add(backend.name, backend.implementation, owner=plugin.name)
|
||||
|
||||
|
||||
def resolve_plugin_runtime(application_config: Mapping[str, Any]) -> PluginRuntime:
|
||||
"""Build one local registry, with user config overriding plugin application defaults."""
|
||||
manager = PluginManager.discover(application_config.get("plugins") or ())
|
||||
registry = create_application_registry()
|
||||
manager.register(registry)
|
||||
return PluginRuntime(config=manager.merge_config(application_config), registry=registry)
|
||||
|
|
|
|||
358
reme/plugin_cli.py
Normal file
358
reme/plugin_cli.py
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
"""Local CLI for inspecting and managing installed ReMe plugin packages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from importlib import metadata
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tomllib
|
||||
from typing import Sequence
|
||||
|
||||
from .entry_point import PLUGIN_ENTRY_POINT_GROUP
|
||||
from .plugin_manifest import PLUGIN_MANIFEST, PluginManifest, load_package_manifest, parse_plugin_manifest
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InstalledPlugin:
|
||||
"""Package metadata for one installed ``reme.plugins`` entry point."""
|
||||
|
||||
name: str
|
||||
target: str
|
||||
distribution: str
|
||||
version: str
|
||||
entry: metadata.EntryPoint
|
||||
|
||||
@property
|
||||
def format(self) -> str:
|
||||
"""Return the declarative or compatibility contract used by the entry point."""
|
||||
return "manifest" if ":" not in self.target else "legacy"
|
||||
|
||||
|
||||
def _installed_plugins() -> list[InstalledPlugin]:
|
||||
"""Discover installed plugins without importing their packages."""
|
||||
entries = metadata.entry_points().select(group=PLUGIN_ENTRY_POINT_GROUP)
|
||||
plugins: list[InstalledPlugin] = []
|
||||
for entry in entries:
|
||||
distribution = getattr(entry, "dist", None)
|
||||
dist_name = distribution.metadata.get("Name", "") if distribution is not None else ""
|
||||
version = distribution.version if distribution is not None else ""
|
||||
plugins.append(
|
||||
InstalledPlugin(
|
||||
name=entry.name,
|
||||
target=entry.value,
|
||||
distribution=dist_name or "unknown",
|
||||
version=version or "unknown",
|
||||
entry=entry,
|
||||
),
|
||||
)
|
||||
return sorted(plugins, key=lambda item: (item.name, item.distribution, item.target))
|
||||
|
||||
|
||||
def _select_plugin(name: str) -> InstalledPlugin:
|
||||
matches = [plugin for plugin in _installed_plugins() if plugin.name == name]
|
||||
if not matches:
|
||||
raise ValueError(f"Plugin '{name}' is not installed")
|
||||
if len(matches) > 1:
|
||||
providers = ", ".join(f"{plugin.distribution} ({plugin.target})" for plugin in matches)
|
||||
raise ValueError(f"Plugin '{name}' has multiple installed providers: {providers}")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _print_table(headers: Sequence[str], rows: Sequence[Sequence[str]]) -> None:
|
||||
widths = [max(len(header), *(len(row[index]) for row in rows)) for index, header in enumerate(headers)]
|
||||
print(" ".join(header.ljust(widths[index]) for index, header in enumerate(headers)))
|
||||
print(" ".join("-" * width for width in widths))
|
||||
for row in rows:
|
||||
print(" ".join(value.ljust(widths[index]) for index, value in enumerate(row)))
|
||||
|
||||
|
||||
def _enabled_plugins(config: str | None) -> set[str] | None:
|
||||
if config is None:
|
||||
return None
|
||||
from .config.config_parser import resolve_app_config
|
||||
|
||||
value = resolve_app_config(config=config, log_config=False).get("plugins") or []
|
||||
if not isinstance(value, list) or not all(isinstance(name, str) for name in value):
|
||||
raise TypeError("Application config 'plugins' must be a list of strings")
|
||||
return set(value)
|
||||
|
||||
|
||||
def _list_plugins(args: argparse.Namespace) -> int:
|
||||
plugins = _installed_plugins()
|
||||
enabled = _enabled_plugins(args.config)
|
||||
records = [
|
||||
{
|
||||
"name": plugin.name,
|
||||
"distribution": plugin.distribution,
|
||||
"version": plugin.version,
|
||||
"format": plugin.format,
|
||||
"target": plugin.target,
|
||||
**({"enabled": plugin.name in enabled} if enabled is not None else {}),
|
||||
}
|
||||
for plugin in plugins
|
||||
]
|
||||
if args.json:
|
||||
print(json.dumps(records, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if not records:
|
||||
print("No ReMe plugins installed.")
|
||||
return 0
|
||||
headers = ["PLUGIN", "DISTRIBUTION", "VERSION", "FORMAT"]
|
||||
if enabled is not None:
|
||||
headers.append("ENABLED")
|
||||
rows = [
|
||||
[
|
||||
record["name"],
|
||||
record["distribution"],
|
||||
record["version"],
|
||||
record["format"],
|
||||
*(["yes" if record["enabled"] else "no"] if enabled is not None else []),
|
||||
]
|
||||
for record in records
|
||||
]
|
||||
_print_table(headers, rows)
|
||||
return 0
|
||||
|
||||
|
||||
def _installed_manifest(plugin: InstalledPlugin) -> PluginManifest:
|
||||
if plugin.format != "manifest":
|
||||
raise ValueError(f"Plugin '{plugin.name}' uses the legacy Python descriptor format")
|
||||
distribution = getattr(plugin.entry, "dist", None)
|
||||
if distribution is not None:
|
||||
relative = Path(*plugin.target.split(".")).joinpath(PLUGIN_MANIFEST)
|
||||
path = Path(distribution.locate_file(relative))
|
||||
if path.is_file():
|
||||
return parse_plugin_manifest(path.read_text(encoding="utf-8"), plugin_name=plugin.name)
|
||||
# Editable installs may not expose package data through ``locate_file``.
|
||||
# Importlib resources then has to import the package, whose ``__init__``
|
||||
# may still contain legacy registration side effects.
|
||||
from .components.component_registry import R
|
||||
|
||||
with R.preserve(allow_mutation=True):
|
||||
return load_package_manifest(plugin.target, plugin_name=plugin.name)
|
||||
|
||||
|
||||
def _plugin_details(plugin: InstalledPlugin) -> dict:
|
||||
details = {
|
||||
"name": plugin.name,
|
||||
"distribution": plugin.distribution,
|
||||
"version": plugin.version,
|
||||
"format": plugin.format,
|
||||
"target": plugin.target,
|
||||
"backends": [],
|
||||
"default_jobs": [],
|
||||
}
|
||||
if plugin.format == "manifest":
|
||||
manifest = _installed_manifest(plugin)
|
||||
details["backends"] = list(manifest.backends)
|
||||
jobs = manifest.application_defaults.get("jobs") or {}
|
||||
details["default_jobs"] = list(jobs) if isinstance(jobs, dict) else []
|
||||
return details
|
||||
|
||||
|
||||
def _show_plugin(args: argparse.Namespace) -> int:
|
||||
details = _plugin_details(_select_plugin(args.plugin))
|
||||
if args.json:
|
||||
print(json.dumps(details, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
for label, key in (
|
||||
("Plugin", "name"),
|
||||
("Distribution", "distribution"),
|
||||
("Version", "version"),
|
||||
("Format", "format"),
|
||||
("Entry point", "target"),
|
||||
):
|
||||
print(f"{label}: {details[key]}")
|
||||
for label, key in (("Backends", "backends"), ("Default jobs", "default_jobs")):
|
||||
values = details[key]
|
||||
print(f"{label}:" if values else f"{label}: none")
|
||||
for value in values:
|
||||
print(f" {value}")
|
||||
return 0
|
||||
|
||||
|
||||
def _run_pip(arguments: list[str]) -> int:
|
||||
command = [sys.executable, "-m", "pip", *arguments]
|
||||
try:
|
||||
return subprocess.run(command, check=False).returncode
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Unable to run pip with {sys.executable}: {exc}") from exc
|
||||
|
||||
|
||||
def _install_plugin(args: argparse.Namespace) -> int:
|
||||
command = ["install"]
|
||||
if args.editable:
|
||||
command.append("--editable")
|
||||
if args.upgrade:
|
||||
command.append("--upgrade")
|
||||
command.append(args.target)
|
||||
result = _run_pip(command)
|
||||
if result == 0:
|
||||
print("Package installed. Run 'reme plugins list' to verify it, then enable its plugin name in app config.")
|
||||
return result
|
||||
|
||||
|
||||
def _uninstall_plugin(args: argparse.Namespace) -> int:
|
||||
plugin = _select_plugin(args.plugin)
|
||||
if plugin.distribution == "unknown":
|
||||
raise ValueError(f"Cannot determine the distribution that provides plugin '{plugin.name}'")
|
||||
siblings = [
|
||||
candidate.name
|
||||
for candidate in _installed_plugins()
|
||||
if candidate.distribution == plugin.distribution and candidate.name != plugin.name
|
||||
]
|
||||
if siblings:
|
||||
print(f"Distribution '{plugin.distribution}' also provides: {', '.join(siblings)}")
|
||||
command = ["uninstall"]
|
||||
if args.yes:
|
||||
command.append("--yes")
|
||||
command.append(plugin.distribution)
|
||||
result = _run_pip(command)
|
||||
if result == 0:
|
||||
print(f"Plugin package '{plugin.distribution}' uninstalled. Remove '{plugin.name}' from application configs.")
|
||||
return result
|
||||
|
||||
|
||||
def _validate_plugins(manager) -> None:
|
||||
"""Validate imports, registry ownership, and merged application schema."""
|
||||
from .components.component_registry import create_application_registry
|
||||
from .config.config_parser import resolve_app_config
|
||||
from .schema.application_config import ApplicationConfig
|
||||
|
||||
registry = create_application_registry()
|
||||
manager.register(registry)
|
||||
ApplicationConfig(**manager.merge_config(resolve_app_config(log_config=False)))
|
||||
|
||||
|
||||
def _validate_installed(name: str) -> list[str]:
|
||||
from .plugin import PluginManager
|
||||
|
||||
manager = PluginManager.discover([name])
|
||||
_validate_plugins(manager)
|
||||
return [name]
|
||||
|
||||
|
||||
def _local_source_roots(project_file: Path, project: dict) -> list[Path]:
|
||||
"""Return declared and conventional Python source roots for a local project."""
|
||||
project_root = project_file.parent
|
||||
candidates: list[Path] = []
|
||||
|
||||
setuptools = project.get("tool", {}).get("setuptools", {})
|
||||
if isinstance(setuptools, dict):
|
||||
package_dir = setuptools.get("package-dir", {})
|
||||
if isinstance(package_dir, dict) and isinstance(package_dir.get(""), str):
|
||||
candidates.append(project_root / package_dir[""])
|
||||
|
||||
packages = setuptools.get("packages", {})
|
||||
package_find = packages.get("find", {}) if isinstance(packages, dict) else {}
|
||||
if isinstance(package_find, dict):
|
||||
where = package_find.get("where", [])
|
||||
if isinstance(where, str):
|
||||
where = [where]
|
||||
if isinstance(where, list):
|
||||
candidates.extend(project_root / item for item in where if isinstance(item, str))
|
||||
|
||||
# ``src`` is a build-backend-independent Python project convention used by
|
||||
# Hatchling, Poetry, Flit, and setuptools. Root-layout projects remain the
|
||||
# final fallback.
|
||||
candidates.extend((project_root / "src", project_root))
|
||||
return list(dict.fromkeys(candidate.resolve() for candidate in candidates))
|
||||
|
||||
|
||||
def _validate_local(path: Path) -> list[str]:
|
||||
from .components.component_registry import R
|
||||
from .plugin import PluginManager, _plugin_from_manifest
|
||||
|
||||
project_file = path if path.name == "pyproject.toml" else path / "pyproject.toml"
|
||||
if not project_file.is_file():
|
||||
raise FileNotFoundError(f"pyproject.toml not found: {project_file}")
|
||||
project = tomllib.loads(project_file.read_text(encoding="utf-8"))
|
||||
entry_points = project.get("project", {}).get("entry-points", {}).get(PLUGIN_ENTRY_POINT_GROUP)
|
||||
if not isinstance(entry_points, dict) or not entry_points:
|
||||
raise ValueError(f"No {PLUGIN_ENTRY_POINT_GROUP} entry points found in {project_file}")
|
||||
|
||||
source_roots = _local_source_roots(project_file, project)
|
||||
plugins = []
|
||||
selected_roots: list[Path] = []
|
||||
manifests = []
|
||||
for name, package in entry_points.items():
|
||||
if not isinstance(name, str) or not isinstance(package, str) or ":" in package:
|
||||
raise ValueError("Local validation requires package-only manifest entry points")
|
||||
relative_manifest = Path(*package.split(".")).joinpath(PLUGIN_MANIFEST)
|
||||
manifest_path = next(
|
||||
(root / relative_manifest for root in source_roots if (root / relative_manifest).is_file()),
|
||||
None,
|
||||
)
|
||||
if manifest_path is None:
|
||||
searched = ", ".join(str(root / relative_manifest) for root in source_roots)
|
||||
raise FileNotFoundError(f"Plugin manifest not found; searched: {searched}")
|
||||
selected_roots.append(manifest_path.parents[len(package.split("."))])
|
||||
manifests.append((name, manifest_path))
|
||||
|
||||
inserted_roots = list(dict.fromkeys(str(root) for root in selected_roots))
|
||||
for source_root in reversed(inserted_roots):
|
||||
sys.path.insert(0, source_root)
|
||||
try:
|
||||
# Match installed-plugin loading: imports may execute compatibility
|
||||
# decorators, but they must not mutate the frozen built-in template.
|
||||
with R.preserve(allow_mutation=True):
|
||||
for name, manifest_path in manifests:
|
||||
manifest = parse_plugin_manifest(manifest_path.read_text(encoding="utf-8"), plugin_name=name)
|
||||
plugins.append(_plugin_from_manifest(name, manifest))
|
||||
_validate_plugins(PluginManager(plugins))
|
||||
finally:
|
||||
for source_root in inserted_roots:
|
||||
sys.path.remove(source_root)
|
||||
return [plugin.name for plugin in plugins]
|
||||
|
||||
|
||||
def _validate_plugin(args: argparse.Namespace) -> int:
|
||||
path = Path(args.target).expanduser()
|
||||
names = _validate_local(path.resolve()) if path.exists() else _validate_installed(args.target)
|
||||
print(f"Valid ReMe plugin: {', '.join(names)}")
|
||||
return 0
|
||||
|
||||
|
||||
def _parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(prog="reme plugins", description="Manage ReMe plugin packages.")
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
list_parser = commands.add_parser("list", help="List installed ReMe plugins.")
|
||||
list_parser.add_argument("--config", help="Show whether plugins are enabled by this config.")
|
||||
list_parser.add_argument("--json", action="store_true", help="Print JSON output.")
|
||||
list_parser.set_defaults(handler=_list_plugins)
|
||||
|
||||
show_parser = commands.add_parser("show", help="Show one installed plugin.")
|
||||
show_parser.add_argument("plugin")
|
||||
show_parser.add_argument("--json", action="store_true", help="Print JSON output.")
|
||||
show_parser.set_defaults(handler=_show_plugin)
|
||||
|
||||
install_parser = commands.add_parser("install", help="Install a plugin package with this Python interpreter.")
|
||||
install_parser.add_argument("target", help="Distribution specifier, wheel, VCS URL, or local path.")
|
||||
install_parser.add_argument("--editable", action="store_true", help="Install a local project in editable mode.")
|
||||
install_parser.add_argument("--upgrade", action="store_true", help="Upgrade an existing installation.")
|
||||
install_parser.set_defaults(handler=_install_plugin)
|
||||
|
||||
uninstall_parser = commands.add_parser("uninstall", help="Uninstall the distribution providing a plugin.")
|
||||
uninstall_parser.add_argument("plugin", help="Plugin entry-point name, such as auto-fin.")
|
||||
uninstall_parser.add_argument("--yes", action="store_true", help="Do not ask pip for confirmation.")
|
||||
uninstall_parser.set_defaults(handler=_uninstall_plugin)
|
||||
|
||||
validate_parser = commands.add_parser("validate", help="Validate an installed plugin or local plugin project.")
|
||||
validate_parser.add_argument("target", help="Installed plugin name, project directory, or pyproject.toml.")
|
||||
validate_parser.set_defaults(handler=_validate_plugin)
|
||||
return parser
|
||||
|
||||
|
||||
def plugin_cli(argv: Sequence[str]) -> int:
|
||||
"""Run the local-only plugin command group and return a process status."""
|
||||
args = _parser().parse_args(list(argv))
|
||||
try:
|
||||
return args.handler(args)
|
||||
except (FileNotFoundError, KeyError, ModuleNotFoundError, RuntimeError, TypeError, ValueError) as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
56
reme/plugin_manifest.py
Normal file
56
reme/plugin_manifest.py
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
"""Parse the small, declarative contract shared by plugin runtime and CLI."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from importlib import resources
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
PLUGIN_MANIFEST = "plugin.yaml"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PluginManifest:
|
||||
"""The two contributions an installed plugin may declare."""
|
||||
|
||||
backends: dict[str, str]
|
||||
application_defaults: dict[str, Any]
|
||||
|
||||
|
||||
def parse_plugin_manifest(text: str, *, plugin_name: str) -> PluginManifest:
|
||||
"""Parse and validate plugin.yaml without importing backend modules."""
|
||||
try:
|
||||
value = yaml.safe_load(text)
|
||||
except yaml.YAMLError as exc:
|
||||
raise ValueError(f"Plugin '{plugin_name}' manifest is invalid YAML") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError(f"Plugin '{plugin_name}' manifest root must be a mapping")
|
||||
|
||||
unknown = set(value) - {"backends", "application_defaults"}
|
||||
if unknown:
|
||||
raise ValueError(f"Plugin '{plugin_name}' manifest has unknown keys: {', '.join(sorted(unknown))}")
|
||||
|
||||
backends = value.get("backends")
|
||||
application_defaults = value.get("application_defaults")
|
||||
backends = {} if backends is None else backends
|
||||
application_defaults = {} if application_defaults is None else application_defaults
|
||||
if not isinstance(backends, dict):
|
||||
raise TypeError(f"Plugin '{plugin_name}' manifest 'backends' must be a mapping")
|
||||
if not isinstance(application_defaults, dict):
|
||||
raise TypeError(f"Plugin '{plugin_name}' manifest 'application_defaults' must be a mapping")
|
||||
for name, target in backends.items():
|
||||
if not isinstance(name, str) or not name:
|
||||
raise TypeError(f"Plugin '{plugin_name}' backend names must be non-empty strings")
|
||||
if not isinstance(target, str) or not target:
|
||||
raise TypeError(f"Plugin '{plugin_name}' backend target for '{name}' must be a non-empty string")
|
||||
return PluginManifest(backends=dict(backends), application_defaults=dict(application_defaults))
|
||||
|
||||
|
||||
def load_package_manifest(package: str, *, plugin_name: str) -> PluginManifest:
|
||||
"""Read plugin.yaml from an importable package."""
|
||||
try:
|
||||
path = resources.files(package).joinpath(PLUGIN_MANIFEST)
|
||||
text = path.read_text(encoding="utf-8")
|
||||
except (ModuleNotFoundError, FileNotFoundError, TypeError) as exc:
|
||||
raise ValueError(f"Plugin '{plugin_name}' does not provide {package}/{PLUGIN_MANIFEST}") from exc
|
||||
return parse_plugin_manifest(text, plugin_name=plugin_name)
|
||||
67
reme/reme.py
67
reme/reme.py
|
|
@ -1,14 +1,15 @@
|
|||
"""ReMe memory management application entry point."""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
import sys
|
||||
|
||||
from .application import Application
|
||||
from .components import create_application_registry
|
||||
from .components.service.cli_service import prepare_start_config, should_precheck_start
|
||||
from .config import parse_args, resolve_app_config
|
||||
from .config import parse_action, parse_kwargs, resolve_app_config
|
||||
from .enumeration import ComponentEnum
|
||||
from .plugin import PluginManager
|
||||
from .plugin import resolve_plugin_runtime
|
||||
from .utils import cli_find_reme, load_env, precheck_start, running_app_config
|
||||
|
||||
_CLIENT_KWARGS = {"host", "port", "timeout", "transport", "command", "args", "show_metadata"}
|
||||
|
|
@ -18,6 +19,21 @@ class ReMe(Application):
|
|||
"""ReMe memory management application."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CliInvocation:
|
||||
"""A top-level CLI action with arguments in that action's own syntax."""
|
||||
|
||||
action: str
|
||||
arguments: tuple[str, ...]
|
||||
|
||||
|
||||
def parse_cli_invocation(argv: Sequence[str]) -> CliInvocation:
|
||||
"""Parse only the grammar shared by every CLI command family."""
|
||||
if not argv:
|
||||
raise ValueError("No arguments provided")
|
||||
return CliInvocation(action=parse_action(argv[0]), arguments=tuple(argv[1:]))
|
||||
|
||||
|
||||
async def call_server(action: str, **kwargs):
|
||||
"""Call the running server with a client matching its *actual* service config.
|
||||
|
||||
|
|
@ -42,8 +58,8 @@ async def call_server(action: str, **kwargs):
|
|||
app_config = running_app_config()
|
||||
if app_config is None:
|
||||
app_config = resolve_app_config(log_config=False, **resolve_kwargs)
|
||||
plugin_manager = PluginManager.discover(app_config.get("plugins") or ())
|
||||
app_config = plugin_manager.merge_config(app_config)
|
||||
runtime = resolve_plugin_runtime(app_config)
|
||||
app_config = runtime.config
|
||||
service = app_config.get("service")
|
||||
service = service if isinstance(service, dict) else {}
|
||||
|
||||
|
|
@ -55,9 +71,7 @@ async def call_server(action: str, **kwargs):
|
|||
client_kwargs = {k: seed[k] for k in _CLIENT_KWARGS if k in seed}
|
||||
client_kwargs.update({key: kwargs.pop(key) for key in list(kwargs) if key in _CLIENT_KWARGS})
|
||||
|
||||
registry = create_application_registry()
|
||||
plugin_manager.register(registry)
|
||||
client_cls = registry.get(ComponentEnum.CLIENT, backend)
|
||||
client_cls = runtime.registry.get(ComponentEnum.CLIENT, backend)
|
||||
if client_cls is None:
|
||||
raise ValueError(f"Unknown client backend: {backend!r}")
|
||||
async with client_cls(**client_kwargs) as client:
|
||||
|
|
@ -66,16 +80,39 @@ async def call_server(action: str, **kwargs):
|
|||
print()
|
||||
|
||||
|
||||
def main():
|
||||
def _run_plugin_command(argv: Sequence[str]) -> None:
|
||||
"""Run local package management without initializing the application."""
|
||||
from .plugin_cli import plugin_cli
|
||||
|
||||
status = plugin_cli(argv)
|
||||
if status:
|
||||
raise SystemExit(status)
|
||||
|
||||
|
||||
def _start_application(kwargs: dict, environment: dict) -> None:
|
||||
"""Resolve startup configuration and run the application."""
|
||||
kwargs = prepare_start_config(kwargs)
|
||||
kwargs["environment"] = environment
|
||||
if should_precheck_start(kwargs) and not precheck_start(kwargs.get("service")):
|
||||
return
|
||||
ReMe(**kwargs).run_app()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Parse CLI arguments and launch the appropriate mode."""
|
||||
invocation = parse_cli_invocation(sys.argv[1:])
|
||||
action = invocation.action
|
||||
|
||||
if action == "plugins":
|
||||
# Package management is local-only and must not load application config,
|
||||
# environment files, or a running service.
|
||||
_run_plugin_command(invocation.arguments)
|
||||
return
|
||||
|
||||
environment = load_env()
|
||||
action, kwargs = parse_args(*sys.argv[1:])
|
||||
kwargs = parse_kwargs(*invocation.arguments)
|
||||
if action == "start":
|
||||
kwargs = prepare_start_config(kwargs)
|
||||
kwargs["environment"] = environment
|
||||
if should_precheck_start(kwargs) and not precheck_start(kwargs.get("service")):
|
||||
return
|
||||
ReMe(**kwargs).run_app()
|
||||
_start_application(kwargs, environment)
|
||||
elif action == "find_reme":
|
||||
cli_find_reme()
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -71,6 +71,14 @@ def test_resolve_app_config_can_suppress_config_log(monkeypatch):
|
|||
assert not messages
|
||||
|
||||
|
||||
def test_resolve_app_config_layers_plugins_over_default():
|
||||
"""A plugin-only start keeps the ordinary default application config."""
|
||||
config = resolve_app_config(log_config=False, plugins=["auto-fin"])
|
||||
|
||||
assert config["service"]["backend"] == "http"
|
||||
assert config["plugins"] == ["auto-fin"]
|
||||
|
||||
|
||||
def test_default_config_registers_daily_write_job():
|
||||
"""``daily_write`` is exposed as a base job backed by ``daily_write_step``."""
|
||||
cfg = _load_config("default.yaml")
|
||||
|
|
@ -127,6 +135,14 @@ def test_parse_args_rejects_non_key_value_extra_argument():
|
|||
parse_args("search", "hello")
|
||||
|
||||
|
||||
def test_parse_args_separates_action_and_application_kwargs():
|
||||
"""The shared action grammar is independent from application key/value parsing."""
|
||||
action, kwargs = parse_args("--search", "--query=hello", "limit=3")
|
||||
|
||||
assert action == "search"
|
||||
assert kwargs == {"query": "hello", "limit": 3}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("item", ["=1", ".a=1", "a.=1", "a..b=1"])
|
||||
def test_parse_dot_notation_rejects_empty_key_segments(item):
|
||||
"""Dot notation keys cannot contain empty path segments."""
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
"""Tests for installed plugin discovery and application-local registration."""
|
||||
|
||||
# pylint: disable=missing-class-docstring,missing-function-docstring
|
||||
# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -11,7 +11,8 @@ from reme.components.base_component import BaseComponent, ComponentMixin
|
|||
from reme.components.component_registry import ComponentRegistry, R
|
||||
from reme.config.config_parser import _load_config
|
||||
from reme.enumeration import ComponentEnum
|
||||
from reme.plugin import Backend, Plugin, PluginManager
|
||||
from reme.plugin import Backend, Plugin, PluginManager, _load_backend
|
||||
from reme.plugin_manifest import parse_plugin_manifest
|
||||
|
||||
|
||||
class _PluginStep(ComponentMixin):
|
||||
|
|
@ -42,7 +43,7 @@ def _set_entry_points(monkeypatch, *entries):
|
|||
monkeypatch.setattr("reme.entry_point.metadata.entry_points", lambda: _FakeEntryPoints(entries))
|
||||
|
||||
|
||||
def test_plugin_defaults_are_below_application_config():
|
||||
def test_plugin_application_defaults_are_below_application_config():
|
||||
manager = PluginManager(
|
||||
[
|
||||
Plugin(
|
||||
|
|
@ -57,7 +58,7 @@ def test_plugin_defaults_are_below_application_config():
|
|||
assert merged["jobs"]["task"] == {"backend": "base", "value": 2}
|
||||
|
||||
|
||||
def test_plugin_defaults_expand_environment(monkeypatch):
|
||||
def test_plugin_application_defaults_expand_environment(monkeypatch):
|
||||
monkeypatch.setenv("PLUGIN_LIMIT", "12")
|
||||
manager = PluginManager([Plugin(name="example", config={"limit": "${PLUGIN_LIMIT}"})])
|
||||
|
||||
|
|
@ -138,6 +139,57 @@ def test_plugin_manager_loads_explicit_entry_point(monkeypatch):
|
|||
assert manager.plugins == (descriptor,)
|
||||
|
||||
|
||||
def test_plugin_manager_loads_package_manifest(monkeypatch, tmp_path):
|
||||
package = tmp_path / "example_plugin"
|
||||
package.mkdir()
|
||||
(package / "__init__.py").write_text("", encoding="utf-8")
|
||||
(package / "backend.py").write_text(
|
||||
"from reme.components.base_component import ComponentMixin\n"
|
||||
"from reme.enumeration import ComponentEnum\n"
|
||||
"class ExampleStep(ComponentMixin):\n"
|
||||
" component_type = ComponentEnum.STEP\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "plugin.yaml").write_text(
|
||||
"backends:\n"
|
||||
" example_step: example_plugin.backend:ExampleStep\n"
|
||||
"application_defaults:\n"
|
||||
" jobs:\n"
|
||||
" example:\n"
|
||||
" backend: base\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.syspath_prepend(str(tmp_path))
|
||||
_set_entry_points(
|
||||
monkeypatch,
|
||||
_FakeEntryPoint("example", "example_plugin", lambda: None, "reme.plugins"),
|
||||
)
|
||||
|
||||
manager = PluginManager.discover(["example"])
|
||||
registry = ComponentRegistry()
|
||||
manager.register(registry)
|
||||
|
||||
backend = registry.get(ComponentEnum.STEP, "example_step")
|
||||
assert backend is not None
|
||||
assert backend.__name__ == "ExampleStep"
|
||||
assert manager.merge_config({})["jobs"]["example"]["backend"] == "base"
|
||||
|
||||
|
||||
def test_plugin_manifest_rejects_legacy_defaults_field():
|
||||
with pytest.raises(ValueError, match="unknown keys: defaults"):
|
||||
parse_plugin_manifest("defaults: {}\n", plugin_name="example")
|
||||
|
||||
|
||||
def test_plugin_manifest_requires_application_defaults_mapping():
|
||||
with pytest.raises(TypeError, match="manifest 'application_defaults' must be a mapping"):
|
||||
parse_plugin_manifest("application_defaults: []\n", plugin_name="example")
|
||||
|
||||
|
||||
def test_plugin_manifest_reports_missing_backend_attribute():
|
||||
with pytest.raises(ValueError, match="cannot load backend.*MissingStep"):
|
||||
_load_backend("reme.plugin:MissingStep", plugin_name="missing")
|
||||
|
||||
|
||||
def test_plugin_manager_rejects_multiple_entry_point_providers(monkeypatch):
|
||||
descriptor = Plugin(name="example")
|
||||
_set_entry_points(
|
||||
|
|
|
|||
297
tests/unit/test_plugin_cli.py
Normal file
297
tests/unit/test_plugin_cli.py
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
"""Tests for local plugin package management commands."""
|
||||
|
||||
# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access
|
||||
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from reme import plugin_cli as plugin_cli_module
|
||||
from reme import reme as reme_module
|
||||
from reme.components import R
|
||||
from reme.enumeration import ComponentEnum
|
||||
|
||||
|
||||
class _FakeDistribution:
|
||||
def __init__(self, root: Path, name: str = "reme-auto-fin", version: str = "0.1.0") -> None:
|
||||
self.root = root
|
||||
self.metadata = {"Name": name}
|
||||
self.version = version
|
||||
|
||||
def locate_file(self, relative: Path) -> Path:
|
||||
return self.root / relative
|
||||
|
||||
|
||||
class _FakeEntryPoint:
|
||||
def __init__(self, name: str, value: str, distribution: _FakeDistribution) -> None:
|
||||
self.name = name
|
||||
self.value = value
|
||||
self.dist = distribution
|
||||
|
||||
|
||||
class _FakeEntryPoints(list):
|
||||
def select(self, *, group):
|
||||
assert group == "reme.plugins"
|
||||
return self
|
||||
|
||||
|
||||
def _install_manifest(tmp_path: Path) -> _FakeEntryPoint:
|
||||
package = tmp_path / "reme_auto_fin"
|
||||
package.mkdir()
|
||||
(package / "plugin.yaml").write_text(
|
||||
"backends:\n"
|
||||
" auto_fin_step: reme_auto_fin.step:AutoFinStep\n"
|
||||
"application_defaults:\n"
|
||||
" jobs:\n"
|
||||
" auto_fin:\n"
|
||||
" backend: base\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return _FakeEntryPoint("auto-fin", "reme_auto_fin", _FakeDistribution(tmp_path))
|
||||
|
||||
|
||||
def test_list_plugins_does_not_load_plugin_code(monkeypatch, tmp_path, capsys):
|
||||
entry = _install_manifest(tmp_path)
|
||||
monkeypatch.setattr(
|
||||
plugin_cli_module.metadata,
|
||||
"entry_points",
|
||||
lambda: _FakeEntryPoints([entry]),
|
||||
)
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["list"]) == 0
|
||||
|
||||
output = capsys.readouterr().out
|
||||
assert "auto-fin" in output
|
||||
assert "reme-auto-fin" in output
|
||||
assert "manifest" in output
|
||||
|
||||
|
||||
def test_list_plugins_marks_configured_plugins(monkeypatch, tmp_path, capsys):
|
||||
entry = _install_manifest(tmp_path)
|
||||
monkeypatch.setattr(
|
||||
plugin_cli_module.metadata,
|
||||
"entry_points",
|
||||
lambda: _FakeEntryPoints([entry]),
|
||||
)
|
||||
monkeypatch.setattr(plugin_cli_module, "_enabled_plugins", lambda _config: {"auto-fin"})
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["list", "--config", "daily_cookbook"]) == 0
|
||||
|
||||
output = capsys.readouterr().out
|
||||
assert "ENABLED" in output
|
||||
assert "yes" in output
|
||||
|
||||
|
||||
def test_show_plugin_reads_manifest_without_importing_backends(monkeypatch, tmp_path, capsys):
|
||||
entry = _install_manifest(tmp_path)
|
||||
monkeypatch.setattr(
|
||||
plugin_cli_module.metadata,
|
||||
"entry_points",
|
||||
lambda: _FakeEntryPoints([entry]),
|
||||
)
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["show", "auto-fin"]) == 0
|
||||
|
||||
output = capsys.readouterr().out
|
||||
assert "auto_fin_step" in output
|
||||
assert "auto_fin" in output
|
||||
|
||||
|
||||
def test_show_plugin_preserves_registry_when_editable_fallback_imports_package(monkeypatch, tmp_path, capsys):
|
||||
package = tmp_path / "fallback_plugin"
|
||||
package.mkdir()
|
||||
(package / "__init__.py").write_text("from .backend import FallbackStep\n", encoding="utf-8")
|
||||
(package / "backend.py").write_text(
|
||||
"from reme.components import ComponentMixin, R\n"
|
||||
"from reme.enumeration import ComponentEnum\n"
|
||||
"@R.register('fallback_import_side_effect')\n"
|
||||
"class FallbackStep(ComponentMixin):\n"
|
||||
" component_type = ComponentEnum.STEP\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "plugin.yaml").write_text(
|
||||
"""backends:
|
||||
fallback_step: fallback_plugin.backend:FallbackStep
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
entry = _FakeEntryPoint("fallback", "fallback_plugin", _FakeDistribution(tmp_path / "missing"))
|
||||
monkeypatch.syspath_prepend(str(tmp_path))
|
||||
monkeypatch.setattr(plugin_cli_module.metadata, "entry_points", lambda: _FakeEntryPoints([entry]))
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["show", "fallback"]) == 0
|
||||
|
||||
assert "fallback_step" in capsys.readouterr().out
|
||||
assert R.get(ComponentEnum.STEP, "fallback_import_side_effect") is None
|
||||
|
||||
|
||||
def test_install_uses_current_python_pip(monkeypatch, capsys):
|
||||
commands = []
|
||||
monkeypatch.setattr(plugin_cli_module, "_run_pip", lambda command: commands.append(command) or 0)
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["install", ".", "--editable", "--upgrade"]) == 0
|
||||
|
||||
assert commands == [["install", "--editable", "--upgrade", "."]]
|
||||
assert "plugins list" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_pip_uses_current_python_interpreter(monkeypatch):
|
||||
calls = []
|
||||
monkeypatch.setattr(
|
||||
plugin_cli_module.subprocess,
|
||||
"run",
|
||||
lambda command, **kwargs: calls.append((command, kwargs)) or SimpleNamespace(returncode=7),
|
||||
)
|
||||
|
||||
assert plugin_cli_module._run_pip(["install", "example"]) == 7
|
||||
|
||||
assert calls == [
|
||||
(
|
||||
[plugin_cli_module.sys.executable, "-m", "pip", "install", "example"],
|
||||
{"check": False},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def test_uninstall_resolves_plugin_to_distribution(monkeypatch, tmp_path, capsys):
|
||||
entry = _install_manifest(tmp_path)
|
||||
commands = []
|
||||
monkeypatch.setattr(
|
||||
plugin_cli_module.metadata,
|
||||
"entry_points",
|
||||
lambda: _FakeEntryPoints([entry]),
|
||||
)
|
||||
monkeypatch.setattr(plugin_cli_module, "_run_pip", lambda command: commands.append(command) or 0)
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["uninstall", "auto-fin", "--yes"]) == 0
|
||||
|
||||
assert commands == [["uninstall", "--yes", "reme-auto-fin"]]
|
||||
assert "Remove 'auto-fin'" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_validate_local_auto_fin_project():
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
|
||||
names = plugin_cli_module._validate_local(repository / "plugins" / "auto-fin")
|
||||
|
||||
assert names == ["auto-fin"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"build_system",
|
||||
[
|
||||
"requires = ['hatchling']\nbuild-backend = 'hatchling.build'\n",
|
||||
"requires = ['poetry-core']\nbuild-backend = 'poetry.core.masonry.api'\n",
|
||||
"requires = ['flit-core']\nbuild-backend = 'flit_core.buildapi'\n",
|
||||
],
|
||||
)
|
||||
def test_validate_local_supports_backend_independent_src_layout(tmp_path, build_system):
|
||||
package = tmp_path / "src" / "src_layout_plugin"
|
||||
package.mkdir(parents=True)
|
||||
(tmp_path / "pyproject.toml").write_text(
|
||||
"[project]\n"
|
||||
"name = 'example-plugin'\n"
|
||||
"version = '0.1.0'\n"
|
||||
"[project.entry-points.'reme.plugins']\n"
|
||||
"example = 'src_layout_plugin'\n"
|
||||
"[build-system]\n"
|
||||
f"{build_system}",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "__init__.py").write_text("", encoding="utf-8")
|
||||
(package / "backend.py").write_text(
|
||||
"from reme.components import ComponentMixin\n"
|
||||
"from reme.enumeration import ComponentEnum\n"
|
||||
"class ExampleStep(ComponentMixin):\n"
|
||||
" component_type = ComponentEnum.STEP\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "plugin.yaml").write_text(
|
||||
"backends:\n example_step: src_layout_plugin.backend:ExampleStep\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert plugin_cli_module._validate_local(tmp_path) == ["example"]
|
||||
|
||||
|
||||
def test_validate_local_supports_setuptools_find_source_root(tmp_path):
|
||||
package = tmp_path / "python" / "find_layout_plugin"
|
||||
package.mkdir(parents=True)
|
||||
(tmp_path / "pyproject.toml").write_text(
|
||||
"[project]\n"
|
||||
"name = 'example-plugin'\n"
|
||||
"version = '0.1.0'\n"
|
||||
"[project.entry-points.'reme.plugins']\n"
|
||||
"example = 'find_layout_plugin'\n"
|
||||
"[tool.setuptools.packages.find]\n"
|
||||
"where = ['python']\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "__init__.py").write_text("", encoding="utf-8")
|
||||
(package / "backend.py").write_text(
|
||||
"from reme.components import ComponentMixin\n"
|
||||
"from reme.enumeration import ComponentEnum\n"
|
||||
"class ExampleStep(ComponentMixin):\n"
|
||||
" component_type = ComponentEnum.STEP\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "plugin.yaml").write_text(
|
||||
"backends:\n example_step: find_layout_plugin.backend:ExampleStep\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert plugin_cli_module._validate_local(tmp_path) == ["example"]
|
||||
|
||||
|
||||
def test_validate_local_preserves_registry_during_backend_imports(tmp_path):
|
||||
package = tmp_path / "src" / "decorated_plugin"
|
||||
package.mkdir(parents=True)
|
||||
(tmp_path / "pyproject.toml").write_text(
|
||||
"[project]\n"
|
||||
"name = 'decorated-plugin'\n"
|
||||
"version = '0.1.0'\n"
|
||||
"[project.entry-points.'reme.plugins']\n"
|
||||
"decorated = 'decorated_plugin'\n"
|
||||
"[tool.setuptools.package-dir]\n"
|
||||
"'' = 'src'\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "__init__.py").write_text("", encoding="utf-8")
|
||||
(package / "backend.py").write_text(
|
||||
"from reme.components import ComponentMixin, R\n"
|
||||
"from reme.enumeration import ComponentEnum\n"
|
||||
"@R.register('local_import_side_effect')\n"
|
||||
"class DecoratedStep(ComponentMixin):\n"
|
||||
" component_type = ComponentEnum.STEP\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(package / "plugin.yaml").write_text(
|
||||
"""backends:
|
||||
decorated_step: decorated_plugin.backend:DecoratedStep
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert plugin_cli_module._validate_local(tmp_path) == ["decorated"]
|
||||
assert R.get(ComponentEnum.STEP, "local_import_side_effect") is None
|
||||
|
||||
|
||||
def test_plugin_command_errors_are_clean(monkeypatch, capsys):
|
||||
monkeypatch.setattr(plugin_cli_module.metadata, "entry_points", _FakeEntryPoints)
|
||||
|
||||
assert plugin_cli_module.plugin_cli(["show", "missing"]) == 1
|
||||
|
||||
assert "Plugin 'missing' is not installed" in capsys.readouterr().err
|
||||
|
||||
|
||||
@pytest.mark.parametrize("action", ["plugins", "-plugins", "--plugins"])
|
||||
def test_main_routes_plugins_before_loading_environment(monkeypatch, action):
|
||||
events = []
|
||||
monkeypatch.setattr("sys.argv", ["reme", action, "list"])
|
||||
monkeypatch.setattr(reme_module, "load_env", lambda: events.append("load_env"))
|
||||
monkeypatch.setattr(plugin_cli_module, "plugin_cli", lambda argv: events.append(list(argv)) or 0)
|
||||
|
||||
reme_module.main()
|
||||
|
||||
assert events == [["list"]]
|
||||
|
|
@ -12,8 +12,9 @@ from reme.components.service import cli_service
|
|||
from reme.components.service.cli_service import CliService
|
||||
from reme import reme as reme_module
|
||||
from reme.components.base_component import ComponentMixin
|
||||
from reme.components.component_registry import create_application_registry
|
||||
from reme.enumeration import ComponentEnum
|
||||
from reme.plugin import Backend, Plugin, PluginManager
|
||||
from reme.plugin import Backend, Plugin, PluginManager, PluginRuntime
|
||||
|
||||
|
||||
def _recording_client(seen, output="ok", base=object):
|
||||
|
|
@ -42,8 +43,11 @@ def _recording_client(seen, output="ok", base=object):
|
|||
def _set_client_backend(monkeypatch, client_cls):
|
||||
monkeypatch.setattr(
|
||||
reme_module,
|
||||
"create_application_registry",
|
||||
lambda: SimpleNamespace(get=lambda component_type, backend: client_cls),
|
||||
"resolve_plugin_runtime",
|
||||
lambda config: PluginRuntime(
|
||||
config=dict(config),
|
||||
registry=SimpleNamespace(get=lambda component_type, backend: client_cls),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -94,8 +98,8 @@ def test_main_loads_env_before_calling_server(monkeypatch):
|
|||
events = []
|
||||
|
||||
main_globals = reme_module.main.__globals__
|
||||
monkeypatch.setattr("sys.argv", ["reme", "shell", "cmd=pwd"])
|
||||
monkeypatch.setitem(main_globals, "load_env", lambda: events.append("load_env"))
|
||||
monkeypatch.setitem(main_globals, "parse_args", lambda *_args: ("shell", {"cmd": "pwd"}))
|
||||
|
||||
async def fake_call_server(action, **kwargs):
|
||||
events.append(("call_server", action, kwargs))
|
||||
|
|
@ -123,8 +127,8 @@ def test_main_saves_loaded_environment_in_start_config(monkeypatch):
|
|||
observed["ran"] = True
|
||||
|
||||
main_globals = reme_module.main.__globals__
|
||||
monkeypatch.setattr("sys.argv", ["reme", "start"])
|
||||
monkeypatch.setitem(main_globals, "load_env", lambda: {"TOOL_ENV": "configured"})
|
||||
monkeypatch.setitem(main_globals, "parse_args", lambda *_args: ("start", {}))
|
||||
monkeypatch.setitem(main_globals, "prepare_start_config", lambda _kwargs: {"service": {"backend": "cli"}})
|
||||
monkeypatch.setitem(main_globals, "ReMe", FakeReMe)
|
||||
|
||||
|
|
@ -354,7 +358,13 @@ def test_call_server_uses_running_plugins_and_their_service_defaults(monkeypatch
|
|||
lambda **_kwargs: {"service": {"backend": "http"}},
|
||||
)
|
||||
monkeypatch.setattr(reme_module, "running_app_config", lambda: {"plugins": ["example"]})
|
||||
monkeypatch.setattr(reme_module.PluginManager, "discover", lambda _specs: manager)
|
||||
|
||||
def resolve_runtime(config):
|
||||
registry = create_application_registry()
|
||||
manager.register(registry)
|
||||
return PluginRuntime(config=manager.merge_config(config), registry=registry)
|
||||
|
||||
monkeypatch.setattr(reme_module, "resolve_plugin_runtime", resolve_runtime)
|
||||
|
||||
asyncio.run(reme_module.call_server("search", query="hello"))
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue