* refactor(packaging): reorganize published packages * fix(packaging): install AgentScope extra in wheel smoke * docs: align package guides and documentation site * ci(workflow): add core dependency verification step in Python package build - Add a workflow step to verify released core dependencies by installing the wheel with core extras - Assert the presence of the static index.html file to ensure proper package contents - Create and use a temporary virtual environment for isolation during verification - Keep existing artifacts upload step intact and conditional on inputs.upload_artifacts flag * fix(ci): update package installation dependencies in Windows workflow - Change pip install from editable reme_studio and core to only dev and as extras - Remove installation of reme_studio and core to streamline dependency setup - Ensure Windows CI uses the correct extras for testing environment * fix(tests): add missing commas in toml file reads in package version tests - Added trailing commas in the tomllib.loads calls for auto-fin and daily_paper configs - Ensured consistent syntax to prevent potential tuple misinterpretation - Improved readability and correctness of the test setup code * fix(packaging): protect qwenpaw releases and test Studio health
34 KiB
ReMe Framework
1. Overview
The ReMe runtime can be understood as follows: a configuration-driven Application assembles components and Jobs; the Service exposes service-enabled Jobs to the CLI, HTTP, or MCP; and each Job executes its Steps in sequence.
To run and use ReMe first, see Quick Start. For workspace file semantics, see Memory as File. User-facing guides for retrieval, automatic memory, and proactive reading are Memory Search, Auto Memory, Auto Resource, Auto Dream, and Proactive.
Capability Boundary
ReMe v4 focuses on long-term memory: it distills conversations and resources into daily/, organizes them into
digest/, and exposes write, retrieval, and proactive-read capabilities through the CLI, HTTP, and MCP.
Single-session context-window management is outside the scope of ReMe v4. This includes compressing the current
conversation, injecting summaries, trimming tool output, or providing an independent /compact interface. Those
capabilities belong in the host agent framework. ReMe accepts conversations, resources, and file changes that have
already occurred and persists the information with long-term value.
flowchart LR
CLI["reme CLI<br/>reme/reme.py"] --> Client["Client<br/>http / mcp"]
Client --> Service["Service<br/>HTTP / MCP"]
Service --> App["Application<br/>reme/application.py"]
App --> Jobs["Jobs<br/>base / stream / background / cron"]
Jobs --> Steps["Steps<br/>reme/steps/**"]
Steps --> Ctx["RuntimeContext<br/>data + Response + stream queue"]
Steps --> Components["Components<br/>store / graph / index / llm / agent / catalog"]
Components --> Workspace["Workspace<br/>daily / digest / resource / metadata"]
Core layers:
| Layer | Main location | Responsibility |
|---|---|---|
| CLI | reme/reme.py |
Parse commands; start launches the service; other actions call the service through a client. |
| Service | reme/components/service/ |
Register Jobs as HTTP endpoints or MCP tools. |
| Application | reme/application.py |
Assemble configured objects, start them in dependency order, close them, and invoke Jobs. |
| Job | reme/components/job/ |
Orchestrate Steps and select normal, streaming, background, or scheduled execution. |
| Step | reme/steps/ |
Atomic business operations such as file I/O, retrieval, indexing, and self-evolution. |
| Component | reme/components/ |
Reusable infrastructure such as file_store, file_graph, keyword_index, and agent_wrapper. |
| Schema | reme/schema/ |
Data structures such as Request, Response, FileChunk, FileNode, and configuration models. |
| Config | reme/config/ |
Default YAML configuration and command-line override parsing. |
2. Directory Structure
reme/
reme.py # CLI entry point
application.py # Application assembly and lifecycle
plugin.py # installed plugin contract and entry-point loader
config/
default.yaml # default service / jobs / components
config_parser.py # config=, dot notation, and env placeholder parsing
components/
component_registry.py # backend registry and application-local copies
base_component.py # ComponentMixin / BaseComponent / bind dependency declarations
runtime_context.py # context for one Job execution
job/ # BaseJob / StreamJob / BackgroundJob / CronJob
service/ # HTTP / MCP services
client/ # HTTP / MCP clients
file_store/ # file-index coordination layer
file_graph/ # wikilink graph
keyword_index/ # BM25 and other keyword indexes
file_chunker/ # Markdown / JSON / JSONL / generic text chunking
file_catalog/ # change checkpoints
as_llm/, as_embedding/ # model wrappers
agent_wrapper/ # AgentScope / Claude Code / Codex wrappers
steps/
base_step.py # BaseStep, Ref, dispatch_steps
common/ # version, help, health_check, status, chat
benchmark/ # LongMemEval / BEAM evaluation steps
cookbook/ # built-in cookbook support steps
file_io/ # read/write/edit/delete/move/frontmatter/daily
index/ # watch/init/update/search/traverse
evolve/ # auto_memory, auto_resource, auto_dream, proactive
transfer/ # upload/download
plugins/
auto-fin/ # independent example plugin distribution
daily_paper/ # independent paper-research plugin distribution
integrations/
claude_code/ # Claude Code adapter and marketplace
hermes_agent/ # Hermes Agent memory-provider adapter
The default workspace directories are defined by ApplicationConfig:
<workspace_dir>/
metadata/ # persistent file_store, file_graph, keyword_index, file_catalog, and related state
session/ # source conversations used by memory workflows
mem_session/ # generated Agent wrapper sessions and configuration
resource/ # external resources
daily/ # lightly processed memory
digest/ # long-term digest memory
Application.__init__() first ensures that these directories exist, then initializes the service, components, and Jobs.
3. Startup and Call Chain
3.1 CLI
The entry point is reme/reme.py::main():
flowchart LR
A["main()"] --> B["parse_args(*sys.argv[1:])"]
B --> C{action}
C -->|" start "| D["load_env()"]
D --> E["resolve_app_config(**kwargs)"]
E --> F["precheck_start(service)"]
F --> G["ReMe(**config).run_app()"]
C -->|" find_reme "| H["cli_find_reme()"]
C -->|" other actions "| I["call_server(action, **kwargs)"]
I --> J["R.get(ComponentEnum.CLIENT, backend)"]
J --> K["client(action=action, **kwargs)"]
Common commands:
reme start
reme start service.port=8181
reme version
reme search query="memory" limit=5
reme search query="memory" backend=mcp
Configuration parsing supports:
| Capability | Source | Description |
|---|---|---|
| Default configuration | resolve_app_config() |
Load reme/config/default.yaml when config is not specified. |
| Explicit configuration | config=<name-or-path> |
Accept a built-in configuration name or a YAML/JSON file path. |
| Dot notation | parse_dot_notation() |
For example, service.port=8181. |
| Environment variables | _expand_env_vars() |
Support ${VAR} and ${VAR:-default}. |
| Value conversion | _convert_value() |
Convert bool, int, float, JSON list/dict, and null values automatically. |
3.2 Service
BaseService.run_app() executes in this order:
Set the optional service.jobs list to restrict HTTP or MCP exposure to those job names. If omitted, all jobs with
enable_serve: true remain eligible; an empty list exposes none. The whitelist does not override enable_serve: false.
When the list is configured, a missing, disabled, unsupported, or invalid selected job fails service startup.
flowchart LR
A["Service.build_service(app)"] --> B["read app.context.jobs"]
B --> C{"enabled and selected by service.jobs?"}
C -->|yes| D["Service.add_job(job)"]
C -->|no| E["skip registration"]
D --> F["Service.start_service(app)"]
E --> F
F --> G["app.start() during lifespan"]
G --> H["Application starts jobs"]
HTTP service behavior:
| Job type | HTTP exposure |
|---|---|
Non-StreamJob with enable_serve: true |
POST /<job.name> returning Response JSON. |
StreamJob |
POST /<job.name> returning text/event-stream. |
enable_serve: false |
No endpoint is registered. |
After registering Job endpoints, the HTTP service can also mount the ReMe Studio single-page application. The default is
service.web_enabled=true. Builds are resolved from service.web_static_dir, REME_WEB_STATIC_DIR, the optional
reme_studio package installed by the web and core extras, and source-tree locations such as
reme_studio/dist-static. If no index.html is found, only the frontend is skipped and the Job API remains available. The
Studio GET fallback does not replace existing POST /<job.name> routes.
MCP service behavior:
| Job type | MCP exposure |
|---|---|
Non-StreamJob with enable_serve: true |
Registered as an MCP tool. |
StreamJob |
Currently skipped and not registered. |
BackgroundJob |
Forces enable_serve=False at construction and is never exposed. |
MCP services can inject server-owned arguments with injected_job_kwargs; callers cannot override those arguments. Set
tool_error_on_failure: true to expose an unsuccessful ReMe Response as an MCP tool error.
4. Registry and Dependency Injection
4.1 Global Registry R
ReMe uses the process-wide singleton R = ComponentRegistry(). Every component, Job, and Step is registered with
@R.register("name").
from ...components import R
@R.register("version_step")
class VersionStep(BaseStep):
...
The registry key is:
(component_type, register_name) -> class
component_type comes from a class attribute:
| Type | Class attribute |
|---|---|
| Step | BaseStep.component_type = ComponentEnum.STEP |
| Job | BaseJob.component_type = ComponentEnum.JOB |
| Service | BaseService.component_type = ComponentEnum.SERVICE |
| FileStore | BaseFileStore.component_type = ComponentEnum.FILE_STORE |
The same backend name can therefore exist under different component types. For example, http can be both a service
backend and a client backend.
ComponentEnum provides the built-in identifiers, but installed plugins may declare a new type with a namespaced
string such as example.reranker. Custom identifiers use lowercase letters and numbers separated by ., _, or -.
They are configured under components and participate in the same dependency ordering and lifecycle as built-ins.
4.2 Built-in and Plugin Registration
Built-in implementations populate the built-in registry through package imports. ReMe freezes that template after
bootstrap, and each Application receives a mutable copy. Runtime code resolves backends through the application's
registry rather than changing the process-wide template. ReMe then loads only the installed plugins explicitly named by
plugins in the resolved configuration. A plugin exposes its package through the reme.plugins Python entry-point
group. The package's plugin.yaml has two optional mappings: backends maps registration names to
module:Class targets, and application_defaults contributes a low-priority ApplicationConfig fragment. The
entry-point name is the plugin's identity.
Plugins are enabled explicitly through the application config's plugins list or a plugins=[...] CLI override.
Plugin registration therefore stays local to one application;
duplicate (component_type, backend) providers fail during assembly instead of overwriting each other.
The legacy Python Plugin descriptor and reme.configs entry points remain accepted during migration. Configuration
files can use extends to inherit another built-in, legacy plugin, or file-based configuration. See the independently
packaged Auto Fin and Daily Paper plugins.
Plugin packages are managed locally and remain separate from per-application activation:
reme plugins list
reme plugins install reme-auto-fin
reme plugins install reme-daily-paper
reme plugins show daily-paper
reme plugins validate daily-paper
reme plugins uninstall daily-paper
reme start plugins='["auto-fin","daily-paper"]'
These management commands use the current Python interpreter's pip and never run through an HTTP or MCP service.
4.3 Component.bind
Dependencies between components are declared with BaseComponent.bind(). At startup,
Application._topological_order() reads every component's dependencies and starts them in topological order.
flowchart LR
A["Component.__init__<br/>self.keyword_index = self.bind(...)"] --> B["Dependency placeholder"]
B --> C["Application._topological_order()"]
C --> D["component.start()"]
D --> E["_resolve_bindings()"]
E --> F["self.keyword_index = app_context.components[type][name]"]
F --> G["component._start()"]
Rules for BaseComponent.bind(name, BaseClass, optional=True):
| Scenario | Behavior |
|---|---|
name is empty |
Return None and skip the dependency. |
app_context exists |
Look up app_context.components[ctype][name]. |
Dependency missing and optional=True |
Resolve to None. |
Dependency missing and optional=False |
Fail at startup. |
| Standalone mode | A private component can be created with default_factory. |
4.4 Step.Ref
Steps do not participate in component topological startup. They are created temporarily for each Job invocation. Steps
access components primarily through BaseStep.Ref:
file_store: BaseFileStore = Ref(BaseFileStore, ComponentEnum.FILE_STORE)
agent_wrapper: BaseAgentWrapper = Ref(BaseAgentWrapper, ComponentEnum.AGENT_WRAPPER, optional=True)
Resolution priority:
flowchart LR
A["access self.file_store"] --> B{"same-named object in kwargs?"}
B -->|yes| C["use kwargs object"]
B -->|no| D{"same-named object in context.data?"}
D -->|yes| E["use context object"]
D -->|no| F["read name from kwargs['file_store']; default is default"]
F --> G["app_context.components[FILE_STORE][name]"]
A Step configuration can therefore specify:
steps:
- backend: update_catalog_step
file_catalog: resource
Here, file_catalog: resource means to resolve the file_catalog component named resource.
5. Application Lifecycle
The Application converts configuration into runtime objects and starts and closes them in order.
flowchart LR
A["Application(**kwargs)"] --> B["ApplicationContext(**kwargs)<br/>parse ApplicationConfig"]
B --> C["_setup_workspace_directories()"]
C --> D["_init_service()"]
D --> E["_init_components()"]
E --> F["_init_jobs()"]
F --> G["run_app()"]
G --> H["service.run_app(app)"]
Startup order in Application._start():
flowchart LR
A["create optional thread_pool"] --> B["topologically sort components"]
B --> C["start components"]
C --> D["start BaseJob"]
D --> E["start StreamJob"]
E --> F["start BackgroundJob"]
F --> G["start CronJob"]
During shutdown, objects in _started_components are closed in reverse order so dependents close before their
dependencies.
6. Job Model
A Job is the orchestration unit for an externally callable capability or background task. Jobs are configured under
jobs:
in reme/config/default.yaml.
6.1 BaseJob
BaseJob is the most common request-oriented Job:
flowchart LR
Caller["Caller"] --> Job["BaseJob<br/>job(**kwargs)"]
Job --> Ctx["RuntimeContext<br/>merged_kwargs"]
Ctx --> S1["Step 1<br/>await step(context)"]
S1 --> D1["read/write context.data / response"]
D1 --> S2["Step 2<br/>await step(context)"]
S2 --> D2["read/write context.data / response"]
D2 --> Resp["context.response"]
Resp --> Caller
Important source behavior:
| Source | Behavior |
|---|---|
_start() |
Parse each Step config from YAML into (step_cls, params). |
_build_steps() |
Create new Step instances for every call, avoiding state shared across requests. |
__call__() |
Create a RuntimeContext and execute Steps sequentially. |
| Exception handling | Catch the exception, set response.success=False, and set answer=str(e). |
6.2 StreamJob
StreamJob extends BaseJob but returns streaming chunks:
| Behavior | Description |
|---|---|
| Context | Includes stream_queue. |
| Step output | Call context.add_stream_string(text, ChunkEnum.CONTENT). |
| Exception | Write ChunkEnum.ERROR. |
| Completion | Always send a DONE chunk. |
6.3 BackgroundJob
BackgroundJob runs long-lived loops such as file watchers. Its constructor forces enable_serve=False.
flowchart LR
A["Application starts BackgroundJob"] --> B["_start() creates stop_event and task"]
B --> C["_run_with_supervisor()"]
C --> D["await self()"]
D --> E{"exception?"}
E -->|no, returned normally| F["finish"]
E -->|yes and supervisor = True| G["exponential backoff + jitter"]
G --> C
E -->|yes and supervisor = False| H["raise exception"]
I["close()"] --> J["stop_event.set()"]
J --> K["wait close_timeout; cancel on timeout"]
The default BackgroundJob.__call__() also executes configured Steps in sequence, but it does not swallow exceptions,
which allows the supervisor to restart the task.
6.4 CronJob
CronJob extends BackgroundJob with a cron expression:
jobs:
nightly_dream:
backend: cron
cron: "0 3 * * *"
steps:
- backend: dream_extract_step
- backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step
The current implementation uses croniter to calculate the next trigger time. The timezone comes from
app_config.timezone.
6.5 Default Job Types
flowchart LR
Jobs["default.yaml jobs"] --> BG["background<br/>index_update_loop<br/>resource_watch_loop<br/>digest_watch_loop"]
Jobs --> Cron["cron<br/>dream_cron<br/>optimize_index_cron"]
Jobs --> Stream["stream<br/>chat"]
Jobs --> Base["base<br/>version / help / health_check / status / app_config<br/>search / node_search / traverse / graph_snapshot / reindex<br/>read / load / read_image / write / save / edit / delete / move / list / stat / frontmatter_*<br/>daily_list / daily_reindex / daily_write<br/>auto_memory / auto_memory_cc / auto_resource / auto_dream / proactive"]
7. Step Model
A Step is a concrete business action. Every Step extends BaseStep and implements execute().
flowchart LR
A["Job._build_steps()"] --> B["Step.__init__()"]
B --> C["load prompt<br/>class-named YAML + prompt_dict override"]
C --> D["Step.__call__(context, **kwargs)"]
D --> E["clear Ref cache"]
E --> F["RuntimeContext.from_context()"]
F --> G["input_mapping"]
G --> H["execute()"]
H --> I["output_mapping"]
I --> J["return result"]
7.1 RuntimeContext
RuntimeContext is shared by all Steps within one Job invocation:
| Field | Description |
|---|---|
response |
Final Response(answer, success, metadata). |
data |
Free-form dictionary containing input parameters and intermediate results. |
stream_queue |
Output queue for streaming Jobs. |
stop_event |
Stop signal for background Jobs. |
Common Step code:
assert self.context is not None
query = self.context.get("query", "")
self.context["processed_query"] = query.strip().lower()
self.context.response.answer = "..."
self.context.response.metadata["key"] = "value"
return self.context.response
7.2 input_mapping / output_mapping
BaseStep.__call__() invokes RuntimeContext.apply_mapping() before and after execution:
steps:
- backend: some_step
input_mapping:
user_query: query
output_mapping:
result: final_result
The semantics are to copy context.data[source] to context.data[target].
7.3 dispatch_steps
Some Steps produce batches of events and dispatch them to other Steps. BaseStep.dispatch_steps() resolves and executes
child Steps according to configuration.
Example from the default configuration:
index_update_loop:
backend: background
watch_dirs: [ daily_dir, digest_dir ]
watch_suffixes: [ md ]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [ update_index_step ]
- backend: watch_changes_step
dispatch_steps: [ update_index_step ]
Flow:
flowchart LR
Init["init_changes_step"] --> Batch["changes batch"]
Watch["watch_changes_step"] --> Batch
Batch --> Dispatch["dispatch_steps(...)"]
Dispatch --> Update["update_index_step"]
Update --> Store["file_store"]
8. Components in the Default Configuration
Current default components in reme/config/default.yaml:
| ComponentEnum | Name | Backend | Description |
|---|---|---|---|
service |
singleton | http |
Default HTTP service. |
tokenizer |
default |
regex |
BM25 tokenizer. |
as_embedding |
default |
Not configured by default; example uses openai |
Provides the embedding model wrapper after uncommenting the example config. |
embedding_store |
default |
Not configured by default; example uses local |
Depends on as_embedding: default after uncommenting the example config. |
as_llm |
default |
${LLM_BACKEND:-openai} |
LLM model wrapper. |
agent_wrapper |
default |
agentscope |
AgentScope wrapper. |
agent_wrapper |
claude_code |
claude_code |
Claude Code wrapper. |
agent_wrapper |
codex/codex_oauth |
codex |
Codex wrappers for API-key and OAuth authentication. |
file_graph |
default |
local |
Wikilink graph. |
file_catalog |
default/resource/digest/dream |
local |
File-change checkpoints. |
file_chunker |
markdown |
markdown |
Markdown AST chunking. |
file_chunker |
json/jsonl/default |
json/jsonl/default |
JSON, JSONL, and generic text chunkers; generic text supports txt and log. |
keyword_index |
default |
bm25 |
BM25 keyword index. |
file_store |
default |
local |
Combines file_graph and keyword_index; defaults to embedding_store: "". |
Note that the search Step configuration contains vector_weight, but file_store.default.embedding_store is empty by
default. Vector retrieval is available only when the runtime configuration enables an embedding store.
9. Adding a Step
9.1 Minimal Step
Suppose you want to add a Step that converts input text to uppercase.
Create a file such as reme/steps/common/uppercase.py:
from ..base_step import BaseStep
from ...components import R
@R.register("uppercase_step")
class UppercaseStep(BaseStep):
async def execute(self):
assert self.context is not None
text = self.context.get("text", "")
result = str(text).upper()
self.context["uppercase_text"] = result
self.context.response.answer = result
self.context.response.metadata["length"] = len(result)
return self.context.response
9.2 Registering the Step
Make sure reme/steps/common/__init__.py imports the new module. Add:
from . import uppercase
The reason is that @R.register("uppercase_step") only executes after the module is imported.
9.3 Accessing Components
If a Step needs an existing component, prefer the Refs provided by BaseStep:
class MySearchStep(BaseStep):
async def execute(self):
assert self.context is not None
results = await self.file_store.keyword_search(
self.context.get("query", ""),
limit=5,
)
...
Common attributes available directly:
| Attribute | Component resolved by default |
|---|---|
self.as_llm |
.model from as_llm: default. |
self.agent_wrapper |
agent_wrapper: default; optional. |
self.file_catalog |
file_catalog: default; optional. |
self.file_store |
file_store: default. |
To select a non-default component from Job configuration:
steps:
- backend: my_step
file_catalog: dream
9.4 Step Design Guidance
| Guidance | Reason |
|---|---|
Read input from context and write intermediate results to context. |
A multi-Step Job passes data through the same context. |
Write the final result to context.response. |
Services and clients consume the standard Response. |
| Do not store request-scoped state on a Step instance. | A Step is rebuilt for every Job call, and stateless Steps are easier to test. |
A background loop that supports interruption should check context.stop_event. |
BackgroundJob.close() relies on the stop event for graceful shutdown. |
Call add_stream_string() only from a StreamJob. |
A normal Job has no stream queue. |
9.5 Unit Test Example
A Step can be instantiated directly and passed a RuntimeContext:
import pytest
from reme.components.runtime_context import RuntimeContext
from reme.steps.common.uppercase import UppercaseStep
@pytest.mark.asyncio
async def test_uppercase_step():
ctx = RuntimeContext(text="hello")
resp = await UppercaseStep()(ctx)
assert resp.answer == "HELLO"
assert ctx["uppercase_text"] == "HELLO"
10. Adding a Job
A Job usually requires no new Python class; configure existing Steps instead. Add a new Job backend only when a new execution model is required.
10.1 Adding a Normal Request Job
Add the Job under jobs: in a YAML configuration:
jobs:
uppercase:
backend: base
description: "Convert text to uppercase."
parameters:
type: object
properties:
text:
type: string
description: "input text"
required:
- text
steps:
- backend: uppercase_step
Start and call it:
reme start
reme uppercase text="hello"
Call chain:
flowchart LR
CLI["CLI<br/>reme uppercase text=hello"] --> HTTP["HTTP Client"]
HTTP --> Req["POST /uppercase"]
Req --> S["HttpService"]
S --> J["uppercase BaseJob<br/>job(text='hello')"]
J --> Step["uppercase_step<br/>await step(context)"]
Step --> Resp["context.response.answer = HELLO"]
Resp --> JSON["Response JSON"]
JSON --> CLIOut["CLI prints answer"]
10.2 Adding a Multi-Step Job
A Job can chain multiple Steps:
jobs:
demo_echo:
backend: base
description: "Normalize query, then echo it."
parameters:
type: object
properties:
query:
type: string
default: ""
min_score:
type: number
default: 0.5
steps:
- backend: demo_echo_step1
- backend: demo_echo_step2
The first Step writes:
context["processed_query"]
context["adjusted_min_score"]
The second Step reads those fields and writes the final response.
10.3 Adding a Stream Job
Use backend: stream in configuration:
jobs:
stream_uppercase:
backend: stream
description: "Stream uppercase text."
parameters:
type: object
properties:
text:
type: string
required:
- text
steps:
- backend: uppercase_prepare_step
- backend: uppercase_stream_step
Example streaming Step:
from ..base_step import BaseStep
from ...components import R
from ...enumeration import ChunkEnum
@R.register("uppercase_stream_step")
class UppercaseStreamStep(BaseStep):
async def execute(self):
assert self.context is not None
for ch in self.context.get("uppercase_text", ""):
await self.context.add_stream_string(ch, ChunkEnum.CONTENT)
return self.context.response
10.4 Adding a Background Job
Use backend: background in configuration:
jobs:
my_watch_loop:
backend: background
watch_dirs: [ daily_dir ]
watch_suffixes: [ md ]
steps:
- backend: init_changes_step
monitor_type: file_store
monitor_name: default
dispatch_steps: [ update_index_step ]
- backend: watch_changes_step
dispatch_steps: [ update_index_step ]
Characteristics of a background Job:
| Characteristic | Description |
|---|---|
| Not externally exposed | BackgroundJob.__init__() forces enable_serve=False. |
| Has a supervisor | Restarts with exponential backoff after an exception by default. |
| Has a stop event | Notifies the loop to exit during close. |
| Suitable for watching/consuming | File watching, queue consumption, and periodic long-running loops. |
10.5 Adding a Cron Job
Use backend: cron in configuration:
jobs:
daily_auto_dream:
backend: cron
cron: "30 3 * * *"
steps:
- backend: dream_extract_step
file_catalog: dream
- backend: dream_integrate_step
- backend: dream_topics_step
- backend: dream_finish_step
file_catalog: dream
An invalid cron expression fails at startup.
10.6 When a New Job Backend Is Needed
Most use cases require only a new Step plus a YAML Job. Consider adding reme/components/job/*.py only in these cases:
| Requirement | New Job class? |
|---|---|
| Add a business command | No; use backend: base. |
| Chain existing steps | No; use steps:. |
| Need SSE/streaming output | No; use backend: stream. |
| Need a background loop | No; use backend: background. |
| Need cron scheduling | No; use backend: cron. |
| Need entirely new scheduling, concurrency, or transaction semantics | Yes; add a Job backend. |
Minimal shape of a new Job backend:
from .base_job import BaseJob
from ..component_registry import R
@R.register("my_job_backend")
class MyJob(BaseJob):
async def __call__(self, **kwargs):
# custom scheduling logic
return await super().__call__(**kwargs)
Also ensure the module is imported by reme/components/job/__init__.py.