mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-29 06:36:47 +00:00
docs: reference Hermes Jev compaction evaluation results
This commit is contained in:
parent
9a9b729b34
commit
b196764541
2 changed files with 46 additions and 0 deletions
|
|
@ -160,6 +160,30 @@ These are not ten additional long-context tasks. Top N saturates when fewer sour
|
|||
|
||||
</details>
|
||||
|
||||
## Related Hermes Agent experiment
|
||||
|
||||
[Hermes Agent PR #116246](https://github.com/NousResearch/hermes-agent/pull/116246) reports a different experiment:
|
||||
`fast-jev-compaction` replaces summaries with Jev-scored tool-call/result retention while keeping user/assistant text
|
||||
verbatim. On three real 500K-token lineage prefixes, with 15 recall questions each, the authors report:
|
||||
|
||||
| Method | Mean recall | Retained tokens |
|
||||
| --- | ---: | ---: |
|
||||
| Summary only | 43.3% | 55K |
|
||||
| Summary + one `session_search` recovery round | 78.9% | 55K |
|
||||
| Jev, plugin defaults | 75.5% | 115K |
|
||||
| Jev-ranked / recency-ranked, same 60K tool-evidence budget | 77.8% / 77.8% | 176K / 177K |
|
||||
|
||||
At the default threshold, all 851 candidate tool calls were dropped; the closed-book gain over summary-only therefore
|
||||
came from retaining user/assistant text, not selective tool retention. Repeated-compaction simulations also found that
|
||||
the growing text floor eventually prevents further savings. The authors recommend against adopting that strategy as
|
||||
Hermes' primary compaction mechanism; the merged PR adds evaluation, not production adoption.
|
||||
|
||||
Our experiment instead keeps a summary and appends selected original evidence, with tool-result bodies visible to Jev.
|
||||
It uses small synthetic tasks, a small-budget summary setting and exact-field scoring, without archive recovery or
|
||||
continuous compaction. Its scores cannot be compared directly with the Hermes recall percentages, and it does not
|
||||
establish superiority over Hermes' production summary-plus-recovery path. The experiments address different strategies;
|
||||
our missing equal-budget selector controls also prevent attributing the gains solely to Jev ranking.
|
||||
|
||||
## Reproduction
|
||||
|
||||
### Models and runtime
|
||||
|
|
|
|||
|
|
@ -148,6 +148,28 @@ Top 4~5 的点估计没有进一步提高,追加更多原文不保证更准
|
|||
|
||||
</details>
|
||||
|
||||
## 相关 Hermes Agent 实验
|
||||
|
||||
[Hermes Agent PR #116246](https://github.com/NousResearch/hermes-agent/pull/116246) 报告了另一种实验:
|
||||
`fast-jev-compaction` 用 Jev 对工具调用/结果的保留决策替代摘要,同时完整保留用户和助手文本。
|
||||
作者在三个真实会话谱系的 500K-token 前缀上,各使用 15 道召回题,报告结果如下:
|
||||
|
||||
| 方法 | 平均召回率 | 保留 token |
|
||||
| --- | ---: | ---: |
|
||||
| 仅摘要 | 43.3% | 55K |
|
||||
| 摘要+一轮 `session_search` 恢复 | 78.9% | 55K |
|
||||
| Jev,插件默认设置 | 75.5% | 115K |
|
||||
| Jev 排序/时间排序,相同 60K 工具证据预算 | 77.8% / 77.8% | 176K / 177K |
|
||||
|
||||
默认阈值下,851 个候选工具调用全部被删除;因此相对仅摘要的闭卷收益来自保留用户和助手文本,
|
||||
而非有选择地保留工具证据。连续压缩模拟还发现,不断增长的文本下限最终会阻止进一步释放空间。
|
||||
作者不建议将该策略作为 Hermes 的主要压缩机制;合并的是评测代码,并非在生产路径中采用该方案。
|
||||
|
||||
本实验保留摘要,再追加选出的原文,并向 Jev 提供工具结果正文。
|
||||
它使用小规模合成任务、小预算摘要设置和精确字段评分,不包含历史恢复或连续压缩。
|
||||
本实验得分不能与 Hermes 的召回百分比直接比较,也不能证明优于 Hermes 生产环境的摘要+恢复路径。
|
||||
两项实验考察不同策略;本实验尚缺少同预算选择器对照,因此也不能把收益全部归因于 Jev 排序。
|
||||
|
||||
## 复现
|
||||
|
||||
### 模型与运行环境
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue