0b5bb64f26
* fix(search): accept CLI queries that look like numbers The CLI parses key=value arguments into typed values, so query=404 reached the search steps as an int and failed on .strip(), and query=NaN or query=Infinity became non-finite floats that the HTTP client cannot encode as JSON and the MCP client sends as null. Read the query as text in the four search steps, as ChatStep already does, and keep values that float() reads as NaN or infinity as strings. * fix(search): treat only a missing query as empty str(query or "") turned query=0 and query=0.0 into an empty string, so those searches still failed with "query cannot be empty". Check for None before converting instead. The BEAM and LME plugins replace the default search job with their own SearchV2Step, which still called .strip() on the raw value, so query=404 raised AttributeError when either plugin was enabled. Read the query the same way there. The numeric-query test is now parametrized over the six steps and passes 404, 0 and 0.0 through parse_kwargs first. |
||
|---|---|---|
| .. | ||
| src/reme_lme | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| README_ZH.md | ||
LongMemEval plugin
This plugin owns the LongMemEval memory and agentic-answer Steps and their Job defaults.
The trusted judge Step, prompts and answer_judge Job live in plugins/lme-judge. ReMe's built-in benchmark.yaml owns the
shared evaluation Jobs and components. Dataset handling, the runner and results remain
in benchmark/longmemeval.
From the repository root, install ReMe and this plugin in editable mode before running the benchmark:
python -m pip install -e ".[as]"
reme plugins install ./plugins/lme --editable
reme plugins install ./plugins/lme-judge --editable
reme plugins validate lme
reme plugins validate lme-judge
python benchmark/longmemeval/run.py
Editable installation registers the lme entry point while keeping source changes immediately
visible. The runner selects the built-in benchmark preset and explicitly enables lme and lme-judge for
each Application. Installing the plugin makes it discoverable but does not enable it globally.
plugin.yaml registers backends and contributes the plugin-owned auto_memory,
agentic_answer Job defaults. Start a full benchmark application with
reme start config=benchmark plugins='["lme", "lme-judge"]'. The shared preset does not inherit
default: only declared Jobs run, indexing is manual, and neither scheduled dream
nor the optional auto_dream Job is enabled.
The existing auto_memory, agentic_answer, answer_judge, bench and judge
names and model environment variables are unchanged. Explicit application/CLI overrides
still take precedence. Installing this plugin does not start an evaluation.
Custom Python callers should import memory, search and answer Steps from reme_lme, and install
lme-judge before importing the judge Step from judge_lme. After uninstalling,
Applications and CLI services must omit the plugin until it is installed again.
LmeAgenticAnswerStep now implements the answer behavior directly. The former
reme.steps.benchmark.BaseAgenticAnswerStep import is gone; custom subclasses can
extend reme_lme.LmeAgenticAnswerStep for LongMemEval behavior, or implement their
own Step using reme.steps.base_step.BaseStep.
Uninstallation never removes datasets, workspaces or results.
Restart an existing service after changing plugins.