Commit graph

4 commits

Author SHA1 Message Date
Zhengxu (Joshua) Jin
0b5bb64f26
fix(search): accept CLI queries that look like numbers (#612)
* fix(search): accept CLI queries that look like numbers

The CLI parses key=value arguments into typed values, so query=404 reached the search steps as an int and failed on .strip(), and query=NaN or query=Infinity became non-finite floats that the HTTP client cannot encode as JSON and the MCP client sends as null.

Read the query as text in the four search steps, as ChatStep already does, and keep values that float() reads as NaN or infinity as strings.

* fix(search): treat only a missing query as empty

str(query or "") turned query=0 and query=0.0 into an empty string, so those searches still failed with "query cannot be empty". Check for None before converting instead.

The BEAM and LME plugins replace the default search job with their own SearchV2Step, which still called .strip() on the raw value, so query=404 raised AttributeError when either plugin was enabled. Read the query the same way there.

The numeric-query test is now parametrized over the six steps and passes 404, 0 and 0.0 through parse_kwargs first.
2026-10-16 10:36:05 +08:00
xyf2020
67936d5a43
refactor(plugins): move BaseAgenticAnswerStep out of the core benchmark steps package into the lme/beam plugins (#576)
* refactor(plugins): move BaseAgenticAnswerStep into lme/beam plugins

Move the shared benchmark answer base class from the core
reme.steps.benchmark package into each plugin's own src tree
(reme_lme.base_agentic_answer / reme_beam.base_agentic_answer) with
absolute imports, drop the core benchmark steps package, and update
plugin READMEs accordingly.

* refactor(plugins): simplify benchmark answer steps

* docs(plugins): clarify benchmark answer step migration

---------

Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-09-30 12:19:32 +08:00
xyf2020
9975bb37b9
Separate benchmark judge plugins (#535)
Some checks failed
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Documentation / Test and build documentation (push) Has been cancelled
2026-09-10 20:16:46 +08:00
xyf2020
3f2eb6235f
feat(benchmark): extract BEAM and LongMemEval into standalone plugins (#512)
* Simplify project implementation

* Centralize benchmark agentic answer base class

* Remove bundled ReMe source snapshots

* Preserve local benchmark configs and plugin discovery behavior

* docs: enrich job parameter descriptions in beam and lme plugin configs

* Simplify project structure and remove obsolete code

* Move benchmark search step configuration into BEAM and LME plugins

* Rename benchmark judge packages to avoid import collisions

* Remove explicit plugin package loading in favor of entry-point discovery

* Export benchmark plugin Steps from public packages
2026-09-03 14:06:39 +08:00