scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-08-28 05:25:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11
ReMe/docs/en/framework.md
jinliyl ef3f99f019
refactor(packaging): reorganize published packages (#495) 
* refactor(packaging): reorganize published packages

* fix(packaging): install AgentScope extra in wheel smoke

* docs: align package guides and documentation site

* ci(workflow): add core dependency verification step in Python package build

- Add a workflow step to verify released core dependencies by installing the wheel with core extras
- Assert the presence of the static index.html file to ensure proper package contents
- Create and use a temporary virtual environment for isolation during verification
- Keep existing artifacts upload step intact and conditional on inputs.upload_artifacts flag

* fix(ci): update package installation dependencies in Windows workflow

- Change pip install from editable reme_studio and core to only dev and as extras
- Remove installation of reme_studio and core to streamline dependency setup
- Ensure Windows CI uses the correct extras for testing environment

* fix(tests): add missing commas in toml file reads in package version tests

- Added trailing commas in the tomllib.loads calls for auto-fin and daily_paper configs
- Ensured consistent syntax to prevent potential tuple misinterpretation
- Improved readability and correctness of the test setup code

* fix(packaging): protect qwenpaw releases and test Studio health
2026-08-27 14:02:09 +08:00

840 lines
34 KiB
Markdown
Raw Permalink Blame History

# ReMe Framework
## 1. Overview
The ReMe runtime can be understood as follows: **a configuration-driven Application assembles components and Jobs; the
Service exposes service-enabled Jobs to the CLI, HTTP, or MCP; and each Job executes its Steps in sequence**.
<p align="center">
<img src="../figure/framework-structure.svg" alt="ReMe framework structure: CLI, Service, Application, Job, Step, and Component" width="92%">
</p>
To run and use ReMe first, see [Quick Start](./quick_start.md). For workspace file semantics, see
[Memory as File](./memory_as_file.md). User-facing guides for retrieval, automatic memory, and proactive reading are
[Memory Search](./memory_search.md), [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md),
[Auto Dream](./auto_dream.md), and [Proactive](./proactive.md).
### Capability Boundary
ReMe v4 focuses on long-term memory: it distills conversations and resources into `daily/`, organizes them into
`digest/`, and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP.
Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current
conversation, injecting summaries, trimming tool output, or providing an independent `/compact` interface. Those
capabilities belong in the host agent framework. ReMe accepts conversations, resources, and file changes that have
already occurred and persists the information with long-term value.
```mermaid
flowchart LR
CLI["reme CLI<br/>reme/reme.py"] --> Client["Client<br/>http / mcp"]
Client --> Service["Service<br/>HTTP / MCP"]
Service --> App["Application<br/>reme/application.py"]
App --> Jobs["Jobs<br/>base / stream / background / cron"]
Jobs --> Steps["Steps<br/>reme/steps/**"]
Steps --> Ctx["RuntimeContext<br/>data + Response + stream queue"]
Steps --> Components["Components<br/>store / graph / index / llm / agent / catalog"]
Components --> Workspace["Workspace<br/>daily / digest / resource / metadata"]
```
Core layers:
| Layer | Main location | Responsibility |
|-------------|----------------------------|---------------------------------------------------------------------------------------------------|
| CLI | `reme/reme.py` | Parse commands; `start` launches the service; other actions call the service through a client. |
| Service | `reme/components/service/` | Register Jobs as HTTP endpoints or MCP tools. |
| Application | `reme/application.py` | Assemble configured objects, start them in dependency order, close them, and invoke Jobs. |
| Job | `reme/components/job/` | Orchestrate Steps and select normal, streaming, background, or scheduled execution. |
| Step | `reme/steps/` | Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. |
| Component | `reme/components/` | Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. |
| Schema | `reme/schema/` | Data structures such as `Request`, `Response`, `FileChunk`, `FileNode`, and configuration models. |
| Config | `reme/config/` | Default YAML configuration and command-line override parsing. |
## 2. Directory Structure
```text
reme/
reme.py # CLI entry point
application.py # Application assembly and lifecycle
plugin.py # installed plugin contract and entry-point loader
config/
default.yaml # default service / jobs / components
config_parser.py # config=, dot notation, and env placeholder parsing
components/
component_registry.py # backend registry and application-local copies
base_component.py # ComponentMixin / BaseComponent / bind dependency declarations
runtime_context.py # context for one Job execution
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
service/ # HTTP / MCP services
client/ # HTTP / MCP clients
file_store/ # file-index coordination layer
file_graph/ # wikilink graph
keyword_index/ # BM25 and other keyword indexes
file_chunker/ # Markdown / JSON / JSONL / generic text chunking
file_catalog/ # change checkpoints
as_llm/, as_embedding/ # model wrappers
agent_wrapper/ # AgentScope / Claude Code / Codex wrappers
steps/
base_step.py # BaseStep, Ref, dispatch_steps
common/ # version, help, health_check, status, chat
benchmark/ # LongMemEval / BEAM evaluation steps
cookbook/ # built-in cookbook support steps
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory, auto_resource, auto_dream, proactive
transfer/ # upload/download
plugins/
auto-fin/ # independent example plugin distribution
daily_paper/ # independent paper-research plugin distribution
integrations/
claude_code/ # Claude Code adapter and marketplace
hermes_agent/ # Hermes Agent memory-provider adapter
```
The default workspace directories are defined by `ApplicationConfig`:
```text
<workspace_dir>/
metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state
session/ # source conversations used by memory workflows
mem_session/ # generated Agent wrapper sessions and configuration
resource/ # external resources
daily/ # lightly processed memory
digest/ # long-term digest memory
```
`Application.__init__()` first ensures that these directories exist, then initializes the service, components, and Jobs.
## 3. Startup and Call Chain
### 3.1 CLI
The entry point is `reme/reme.py::main()`:
```mermaid
flowchart LR
A["main()"] --> B["parse_args(*sys.argv[1:])"]
B --> C{action}
C -->|" start "| D["load_env()"]
D --> E["resolve_app_config(**kwargs)"]
E --> F["precheck_start(service)"]
F --> G["ReMe(**config).run_app()"]
C -->|" find_reme "| H["cli_find_reme()"]
C -->|" other actions "| I["call_server(action, **kwargs)"]
I --> J["R.get(ComponentEnum.CLIENT, backend)"]
J --> K["client(action=action, **kwargs)"]
```
Common commands:
```bash
reme start
reme start service.port=8181
reme version
reme search query="memory" limit=5
reme search query="memory" backend=mcp
```
Configuration parsing supports:
| Capability | Source | Description |
|------------------------|-------------------------|--------------------------------------------------------------------------|
| Default configuration | `resolve_app_config()` | Load `reme/config/default.yaml` when `config` is not specified. |
| Explicit configuration | `config=<name-or-path>` | Accept a built-in configuration name or a YAML/JSON file path. |
| Dot notation | `parse_dot_notation()` | For example, `service.port=8181`. |
| Environment variables | `_expand_env_vars()` | Support `${VAR}` and `${VAR:-default}`. |
| Value conversion | `_convert_value()` | Convert bool, int, float, JSON list/dict, and null values automatically. |
### 3.2 Service
`BaseService.run_app()` executes in this order:
Set the optional `service.jobs` list to restrict HTTP or MCP exposure to those job names. If omitted, all jobs with
`enable_serve: true` remain eligible; an empty list exposes none. The whitelist does not override `enable_serve: false`.
When the list is configured, a missing, disabled, unsupported, or invalid selected job fails service startup.
```mermaid
flowchart LR
A["Service.build_service(app)"] --> B["read app.context.jobs"]
B --> C{"enabled and selected by service.jobs?"}
C -->|yes| D["Service.add_job(job)"]
C -->|no| E["skip registration"]
D --> F["Service.start_service(app)"]
E --> F
F --> G["app.start() during lifespan"]
G --> H["Application starts jobs"]
```
HTTP service behavior:
| Job type | HTTP exposure |
|-------------------------------------------|---------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | `POST /<job.name>` returning `Response` JSON. |
| `StreamJob` | `POST /<job.name>` returning `text/event-stream`. |
| `enable_serve: false` | No endpoint is registered. |
After registering Job endpoints, the HTTP service can also mount the ReMe Studio single-page application. The default is
`service.web_enabled=true`. Builds are resolved from `service.web_static_dir`, `REME_WEB_STATIC_DIR`, the optional
`reme_studio` package installed by the `web` and `core` extras, and source-tree locations such as
`reme_studio/dist-static`. If no `index.html` is found, only the frontend is skipped and the Job API remains available. The
Studio `GET` fallback does not replace existing `POST /<job.name>` routes.
MCP service behavior:
| Job type | MCP exposure |
|-------------------------------------------|-------------------------------------------------------------------|
| Non-`StreamJob` with `enable_serve: true` | Registered as an MCP tool. |
| `StreamJob` | Currently skipped and not registered. |
| `BackgroundJob` | Forces `enable_serve=False` at construction and is never exposed. |
MCP services can inject server-owned arguments with `injected_job_kwargs`; callers cannot override those arguments. Set
`tool_error_on_failure: true` to expose an unsuccessful ReMe `Response` as an MCP tool error.
## 4. Registry and Dependency Injection
### 4.1 Global Registry R
ReMe uses the process-wide singleton `R = ComponentRegistry()`. Every component, Job, and Step is registered with
`@R.register("name")`.
```python
from ...components import R
@R.register("version_step")
class VersionStep(BaseStep):
...
```
The registry key is:
```text
(component_type, register_name) -> class
```
`component_type` comes from a class attribute:
| Type | Class attribute |
|-----------|-----------------------------------------------------------|
| Step | `BaseStep.component_type = ComponentEnum.STEP` |
| Job | `BaseJob.component_type = ComponentEnum.JOB` |
| Service | `BaseService.component_type = ComponentEnum.SERVICE` |
| FileStore | `BaseFileStore.component_type = ComponentEnum.FILE_STORE` |
The same backend name can therefore exist under different component types. For example, `http` can be both a service
backend and a client backend.
`ComponentEnum` provides the built-in identifiers, but installed plugins may declare a new type with a namespaced
string such as `example.reranker`. Custom identifiers use lowercase letters and numbers separated by `.`, `_`, or `-`.
They are configured under `components` and participate in the same dependency ordering and lifecycle as built-ins.
### 4.2 Built-in and Plugin Registration
Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
bootstrap, and each `Application` receives a mutable copy. Runtime code resolves backends through the application's
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
`plugins` in the resolved configuration. A plugin exposes its package through the `reme.plugins` Python entry-point
group. The package's `plugin.yaml` has two optional mappings: `backends` maps registration names to
`module:Class` targets, and `application_defaults` contributes a low-priority `ApplicationConfig` fragment. The
entry-point name is the plugin's identity.
Plugins are enabled explicitly through the application config's `plugins` list or a `plugins=[...]` CLI override.
Plugin registration therefore stays local to one application;
duplicate `(component_type, backend)` providers fail during assembly instead of overwriting each other.
The legacy Python `Plugin` descriptor and `reme.configs` entry points remain accepted during migration. Configuration
files can use `extends` to inherit another built-in, legacy plugin, or file-based configuration. See the independently
packaged [Auto Fin](../../plugins/auto-fin/README.md) and [Daily Paper](../../plugins/daily_paper/README.md) plugins.
Plugin packages are managed locally and remain separate from per-application activation:
```bash
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
```
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
### 4.3 Component.bind
Dependencies between components are declared with `BaseComponent.bind()`. At startup,
`Application._topological_order()` reads every component's `dependencies` and starts them in topological order.
```mermaid
flowchart LR
A["Component.__init__<br/>self.keyword_index = self.bind(...)"] --> B["Dependency placeholder"]
B --> C["Application._topological_order()"]
C --> D["component.start()"]
D --> E["_resolve_bindings()"]
E --> F["self.keyword_index = app_context.components[type][name]"]
F --> G["component._start()"]
```
Rules for `BaseComponent.bind(name, BaseClass, optional=True)`:
| Scenario | Behavior |
|-----------------------------------------|------------------------------------------------------------|
| `name` is empty | Return `None` and skip the dependency. |
| `app_context` exists | Look up `app_context.components[ctype][name]`. |
| Dependency missing and `optional=True` | Resolve to `None`. |
| Dependency missing and `optional=False` | Fail at startup. |
| Standalone mode | A private component can be created with `default_factory`. |
### 4.4 Step.Ref
Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps
access components primarily through `BaseStep.Ref`:
```python
file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
agent_wrapper: BaseAgentWrapper = Ref(BaseAgentWrapper, ComponentEnum.AGENT_WRAPPER, optional=True)
```
Resolution priority:
```mermaid
flowchart LR
A["access self.file_store"] --> B{"same-named object in kwargs?"}
B -->|yes| C["use kwargs object"]
B -->|no| D{"same-named object in context.data?"}
D -->|yes| E["use context object"]
D -->|no| F["read name from kwargs['file_store']; default is default"]
F --> G["app_context.components[FILE_STORE][name]"]
```
A Step configuration can therefore specify:
```yaml
steps:
- backend: update_catalog_step
file_catalog: resource
```
Here, `file_catalog: resource` means to resolve the `file_catalog` component named `resource`.
## 5. Application Lifecycle
The Application converts configuration into runtime objects and starts and closes them in order.
```mermaid
flowchart LR
A["Application(**kwargs)"] --> B["ApplicationContext(**kwargs)<br/>parse ApplicationConfig"]
B --> C["_setup_workspace_directories()"]
C --> D["_init_service()"]
D --> E["_init_components()"]
E --> F["_init_jobs()"]
F --> G["run_app()"]
G --> H["service.run_app(app)"]
```
Startup order in `Application._start()`:
```mermaid
flowchart LR
A["create optional thread_pool"] --> B["topologically sort components"]
B --> C["start components"]
C --> D["start BaseJob"]
D --> E["start StreamJob"]
E --> F["start BackgroundJob"]
F --> G["start CronJob"]
```
During shutdown, objects in `_started_components` are closed in reverse order so dependents close before their
dependencies.
## 6. Job Model
A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under
`jobs:`
in `reme/config/default.yaml`.
### 6.1 BaseJob
`BaseJob` is the most common request-oriented Job:
```mermaid
flowchart LR
Caller["Caller"] --> Job["BaseJob<br/>job(**kwargs)"]
Job --> Ctx["RuntimeContext<br/>merged_kwargs"]
Ctx --> S1["Step 1<br/>await step(context)"]
S1 --> D1["read/write context.data / response"]
D1 --> S2["Step 2<br/>await step(context)"]
S2 --> D2["read/write context.data / response"]
D2 --> Resp["context.response"]
Resp --> Caller
```
Important source behavior:
| Source | Behavior |
|--------------------|----------------------------------------------------------------------------------|
| `_start()` | Parse each Step config from YAML into `(step_cls, params)`. |
| `_build_steps()` | Create new Step instances for every call, avoiding state shared across requests. |
| `__call__()` | Create a `RuntimeContext` and execute Steps sequentially. |
| Exception handling | Catch the exception, set `response.success=False`, and set `answer=str(e)`. |
### 6.2 StreamJob
`StreamJob` extends `BaseJob` but returns streaming chunks:
| Behavior | Description |
|-------------|------------------------------------------------------------|
| Context | Includes `stream_queue`. |
| Step output | Call `context.add_stream_string(text, ChunkEnum.CONTENT)`. |
| Exception | Write `ChunkEnum.ERROR`. |
| Completion | Always send a `DONE` chunk. |
### 6.3 BackgroundJob
`BackgroundJob` runs long-lived loops such as file watchers. Its constructor forces `enable_serve=False`.
```mermaid
flowchart LR
A["Application starts BackgroundJob"] --> B["_start() creates stop_event and task"]
B --> C["_run_with_supervisor()"]
C --> D["await self()"]
D --> E{"exception?"}
E -->|no, returned normally| F["finish"]
E -->|yes and supervisor = True| G["exponential backoff + jitter"]
G --> C
E -->|yes and supervisor = False| H["raise exception"]
I["close()"] --> J["stop_event.set()"]
J --> K["wait close_timeout; cancel on timeout"]
```
The default `BackgroundJob.__call__()` also executes configured Steps in sequence, but it does not swallow exceptions,
which allows the supervisor to restart the task.
### 6.4 CronJob
`CronJob` extends `BackgroundJob` with a `cron` expression:
```yaml
jobs:
nightly_dream:
backend: cron
cron: "0 3 * * *"
steps:
- backend: dream_extract_step
- backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step
```
The current implementation uses `croniter` to calculate the next trigger time. The timezone comes from
`app_config.timezone`.
### 6.5 Default Job Types
```mermaid
flowchart LR
Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"]
Jobs --> Cron["cron<br/>dream_cron<br/>optimize_index_cron"]
Jobs --> Stream["stream<br/>chat"]
Jobs --> Base["base<br/>version / help / health_check / status / app_config<br/>search / node_search / traverse / graph_snapshot / reindex<br/>read / load / read_image / write / save / edit / delete / move / list / stat / frontmatter_*<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_memory_cc / auto_resource / auto_dream / proactive"]
```
## 7. Step Model
A Step is a concrete business action. Every Step extends `BaseStep` and implements `execute()`.
```mermaid
flowchart LR
A["Job._build_steps()"] --> B["Step.__init__()"]
B --> C["load prompt<br/>class-named YAML + prompt_dict override"]
C --> D["Step.__call__(context, **kwargs)"]
D --> E["clear Ref cache"]
E --> F["RuntimeContext.from_context()"]
F --> G["input_mapping"]
G --> H["execute()"]
H --> I["output_mapping"]
I --> J["return result"]
```
### 7.1 RuntimeContext
`RuntimeContext` is shared by all Steps within one Job invocation:
| Field | Description |
|----------------|----------------------------------------------------------------------------|
| `response` | Final `Response(answer, success, metadata)`. |
| `data` | Free-form dictionary containing input parameters and intermediate results. |
| `stream_queue` | Output queue for streaming Jobs. |
| `stop_event` | Stop signal for background Jobs. |
Common Step code:
```python
assert self.context is not None
query = self.context.get("query", "")
self.context["processed_query"] = query.strip().lower()
self.context.response.answer = "..."
self.context.response.metadata["key"] = "value"
return self.context.response
```
### 7.2 input_mapping / output_mapping
`BaseStep.__call__()` invokes `RuntimeContext.apply_mapping()` before and after execution:
```yaml
steps:
- backend: some_step
input_mapping:
user_query: query
output_mapping:
result: final_result
```
The semantics are to copy `context.data[source]` to `context.data[target]`.
### 7.3 dispatch_steps
Some Steps produce batches of events and dispatch them to other Steps. `BaseStep.dispatch_steps()` resolves and executes
child Steps according to configuration.
Example from the default configuration:
```yaml
index_update_loop:
backend: background
watch_dirs: [ daily_dir, digest_dir ]
watch_suffixes: [ md ]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [ update_index_step ]
- backend: watch_changes_step
dispatch_steps: [ update_index_step ]
```
Flow:
```mermaid
flowchart LR
Init["init_changes_step"] --> Batch["changes batch"]
Watch["watch_changes_step"] --> Batch
Batch --> Dispatch["dispatch_steps(...)"]
Dispatch --> Update["update_index_step"]
Update --> Store["file_store"]
```
## 8. Components in the Default Configuration
Current default components in `reme/config/default.yaml`:
| ComponentEnum | Name | Backend | Description |
|-------------------|---------------------------------|--------------------------------------------------|--------------------------------------------------------------------------------|
| `service` | singleton | `http` | Default HTTP service. |
| `tokenizer` | `default` | `regex` | BM25 tokenizer. |
| `as_embedding` | `default` | Not configured by default; example uses `openai` | Provides the embedding model wrapper after uncommenting the example config. |
| `embedding_store` | `default` | Not configured by default; example uses `local` | Depends on `as_embedding: default` after uncommenting the example config. |
| `as_llm` | `default` | `${LLM_BACKEND:-openai}` | LLM model wrapper. |
| `agent_wrapper` | `default` | `agentscope` | AgentScope wrapper. |
| `agent_wrapper` | `claude_code` | `claude_code` | Claude Code wrapper. |
| `agent_wrapper` | `codex/codex_oauth` | `codex` | Codex wrappers for API-key and OAuth authentication. |
| `file_graph` | `default` | `local` | Wikilink graph. |
| `file_catalog` | `default/resource/digest/dream` | `local` | File-change checkpoints. |
| `file_chunker` | `markdown` | `markdown` | Markdown AST chunking. |
| `file_chunker` | `json/jsonl/default` | `json/jsonl/default` | JSON, JSONL, and generic text chunkers; generic text supports `txt` and `log`. |
| `keyword_index` | `default` | `bm25` | BM25 keyword index. |
| `file_store` | `default` | `local` | Combines file_graph and keyword_index; defaults to `embedding_store: ""`. |
Note that the `search` Step configuration contains `vector_weight`, but `file_store.default.embedding_store` is empty by
default. Vector retrieval is available only when the runtime configuration enables an embedding store.
## 9. Adding a Step
### 9.1 Minimal Step
Suppose you want to add a Step that converts input text to uppercase.
Create a file such as `reme/steps/common/uppercase.py`:
```python
from ..base_step import BaseStep
from ...components import R
@R.register("uppercase_step")
class UppercaseStep(BaseStep):
async def execute(self):
assert self.context is not None
text = self.context.get("text", "")
result = str(text).upper()
self.context["uppercase_text"] = result
self.context.response.answer = result
self.context.response.metadata["length"] = len(result)
return self.context.response
```
### 9.2 Registering the Step
Make sure `reme/steps/common/__init__.py` imports the new module. Add:
```python
from . import uppercase
```
The reason is that `@R.register("uppercase_step")` only executes after the module is imported.
### 9.3 Accessing Components
If a Step needs an existing component, prefer the Refs provided by `BaseStep`:
```python
class MySearchStep(BaseStep):
async def execute(self):
assert self.context is not None
results = await self.file_store.keyword_search(
self.context.get("query", ""),
limit=5,
)
...
```
Common attributes available directly:
| Attribute | Component resolved by default |
|----------------------|-------------------------------------|
| `self.as_llm` | `.model` from `as_llm: default`. |
| `self.agent_wrapper` | `agent_wrapper: default`; optional. |
| `self.file_catalog` | `file_catalog: default`; optional. |
| `self.file_store` | `file_store: default`. |
To select a non-default component from Job configuration:
```yaml
steps:
- backend: my_step
file_catalog: dream
```
### 9.4 Step Design Guidance
| Guidance | Reason |
|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
| Read input from `context` and write intermediate results to `context`. | A multi-Step Job passes data through the same context. |
| Write the final result to `context.response`. | Services and clients consume the standard `Response`. |
| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. |
| A background loop that supports interruption should check `context.stop_event`. | `BackgroundJob.close()` relies on the stop event for graceful shutdown. |
| Call `add_stream_string()` only from a StreamJob. | A normal Job has no stream queue. |
### 9.5 Unit Test Example
A Step can be instantiated directly and passed a `RuntimeContext`:
```python
import pytest
from reme.components.runtime_context import RuntimeContext
from reme.steps.common.uppercase import UppercaseStep
@pytest.mark.asyncio
async def test_uppercase_step():
ctx = RuntimeContext(text="hello")
resp = await UppercaseStep()(ctx)
assert resp.answer == "HELLO"
assert ctx["uppercase_text"] == "HELLO"
```
## 10. Adding a Job
A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new
execution model is required.
### 10.1 Adding a Normal Request Job
Add the Job under `jobs:` in a YAML configuration:
```yaml
jobs:
uppercase:
backend: base
description: "Convert text to uppercase."
parameters:
type: object
properties:
text:
type: string
description: "input text"
required:
- text
steps:
- backend: uppercase_step
```
Start and call it:
```bash
reme start
reme uppercase text="hello"
```
Call chain:
```mermaid
flowchart LR
CLI["CLI<br/>reme uppercase text=hello"] --> HTTP["HTTP Client"]
HTTP --> Req["POST /uppercase"]
Req --> S["HttpService"]
S --> J["uppercase BaseJob<br/>job(text='hello')"]
J --> Step["uppercase_step<br/>await step(context)"]
Step --> Resp["context.response.answer = HELLO"]
Resp --> JSON["Response JSON"]
JSON --> CLIOut["CLI prints answer"]
```
### 10.2 Adding a Multi-Step Job
A Job can chain multiple Steps:
```yaml
jobs:
demo_echo:
backend: base
description: "Normalize query, then echo it."
parameters:
type: object
properties:
query:
type: string
default: ""
min_score:
type: number
default: 0.5
steps:
- backend: demo_echo_step1
- backend: demo_echo_step2
```
The first Step writes:
```text
context["processed_query"]
context["adjusted_min_score"]
```
The second Step reads those fields and writes the final `response`.
### 10.3 Adding a Stream Job
Use `backend: stream` in configuration:
```yaml
jobs:
stream_uppercase:
backend: stream
description: "Stream uppercase text."
parameters:
type: object
properties:
text:
type: string
required:
- text
steps:
- backend: uppercase_prepare_step
- backend: uppercase_stream_step
```
Example streaming Step:
```python
from ..base_step import BaseStep
from ...components import R
from ...enumeration import ChunkEnum
@R.register("uppercase_stream_step")
class UppercaseStreamStep(BaseStep):
async def execute(self):
assert self.context is not None
for ch in self.context.get("uppercase_text", ""):
await self.context.add_stream_string(ch, ChunkEnum.CONTENT)
return self.context.response
```
### 10.4 Adding a Background Job
Use `backend: background` in configuration:
```yaml
jobs:
my_watch_loop:
backend: background
watch_dirs: [ daily_dir ]
watch_suffixes: [ md ]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [ update_index_step ]
- backend: watch_changes_step
dispatch_steps: [ update_index_step ]
```
Characteristics of a background Job:
| Characteristic | Description |
|---------------------------------|--------------------------------------------------------------------|
| Not externally exposed | `BackgroundJob.__init__()` forces `enable_serve=False`. |
| Has a supervisor | Restarts with exponential backoff after an exception by default. |
| Has a stop event | Notifies the loop to exit during close. |
| Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. |
### 10.5 Adding a Cron Job
Use `backend: cron` in configuration:
```yaml
jobs:
daily_auto_dream:
backend: cron
cron: "30 3 * * *"
steps:
- backend: dream_extract_step
file_catalog: dream
- backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step
file_catalog: dream
```
An invalid `cron` expression fails at startup.
### 10.6 When a New Job Backend Is Needed
Most use cases require only a new Step plus a YAML Job. Consider adding `reme/components/job/*.py` only in these cases:
| Requirement | New Job class? |
|---------------------------------------------------------------------|--------------------------------|
| Add a business command | No; use `backend: base`. |
| Chain existing steps | No; use `steps:`. |
| Need SSE/streaming output | No; use `backend: stream`. |
| Need a background loop | No; use `backend: background`. |
| Need cron scheduling | No; use `backend: cron`. |
| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. |
Minimal shape of a new Job backend:
```python
from .base_job import BaseJob
from ..component_registry import R
@R.register("my_job_backend")
class MyJob(BaseJob):
async def __call__(self, **kwargs):
# custom scheduling logic
return await super().__call__(**kwargs)
```
Also ensure the module is imported by `reme/components/job/__init__.py`.
Reference in a new issue View git blame Copy permalink