From b196764541d4210b00ef585d4b0e85a3044d2e34 Mon Sep 17 00:00:00 2001 From: "jinli.yl" Date: Mon, 19 Oct 2026 01:41:24 +0800 Subject: [PATCH] docs: reference Hermes Jev compaction evaluation results --- benchmark/compaction/README.md | 24 ++++++++++++++++++++++++ benchmark/compaction/README_ZH.md | 22 ++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/benchmark/compaction/README.md b/benchmark/compaction/README.md index e17720bc..ca95f813 100644 --- a/benchmark/compaction/README.md +++ b/benchmark/compaction/README.md @@ -160,6 +160,30 @@ These are not ten additional long-context tasks. Top N saturates when fewer sour +## Related Hermes Agent experiment + +[Hermes Agent PR #116246](https://github.com/NousResearch/hermes-agent/pull/116246) reports a different experiment: +`fast-jev-compaction` replaces summaries with Jev-scored tool-call/result retention while keeping user/assistant text +verbatim. On three real 500K-token lineage prefixes, with 15 recall questions each, the authors report: + +| Method | Mean recall | Retained tokens | +| --- | ---: | ---: | +| Summary only | 43.3% | 55K | +| Summary + one `session_search` recovery round | 78.9% | 55K | +| Jev, plugin defaults | 75.5% | 115K | +| Jev-ranked / recency-ranked, same 60K tool-evidence budget | 77.8% / 77.8% | 176K / 177K | + +At the default threshold, all 851 candidate tool calls were dropped; the closed-book gain over summary-only therefore +came from retaining user/assistant text, not selective tool retention. Repeated-compaction simulations also found that +the growing text floor eventually prevents further savings. The authors recommend against adopting that strategy as +Hermes' primary compaction mechanism; the merged PR adds evaluation, not production adoption. + +Our experiment instead keeps a summary and appends selected original evidence, with tool-result bodies visible to Jev. +It uses small synthetic tasks, a small-budget summary setting and exact-field scoring, without archive recovery or +continuous compaction. Its scores cannot be compared directly with the Hermes recall percentages, and it does not +establish superiority over Hermes' production summary-plus-recovery path. The experiments address different strategies; +our missing equal-budget selector controls also prevent attributing the gains solely to Jev ranking. + ## Reproduction ### Models and runtime diff --git a/benchmark/compaction/README_ZH.md b/benchmark/compaction/README_ZH.md index d8203316..36669255 100644 --- a/benchmark/compaction/README_ZH.md +++ b/benchmark/compaction/README_ZH.md @@ -148,6 +148,28 @@ Top 4～5 的点估计没有进一步提高，追加更多原文不保证更准 +## 相关 Hermes Agent 实验 + +[Hermes Agent PR #116246](https://github.com/NousResearch/hermes-agent/pull/116246) 报告了另一种实验： +`fast-jev-compaction` 用 Jev 对工具调用／结果的保留决策替代摘要，同时完整保留用户和助手文本。 +作者在三个真实会话谱系的 500K-token 前缀上，各使用 15 道召回题，报告结果如下： + +| 方法 | 平均召回率 | 保留 token | +| --- | ---: | ---: | +| 仅摘要 | 43.3% | 55K | +| 摘要＋一轮 `session_search` 恢复 | 78.9% | 55K | +| Jev，插件默认设置 | 75.5% | 115K | +| Jev 排序／时间排序，相同 60K 工具证据预算 | 77.8% / 77.8% | 176K / 177K | + +默认阈值下，851 个候选工具调用全部被删除；因此相对仅摘要的闭卷收益来自保留用户和助手文本， +而非有选择地保留工具证据。连续压缩模拟还发现，不断增长的文本下限最终会阻止进一步释放空间。 +作者不建议将该策略作为 Hermes 的主要压缩机制；合并的是评测代码，并非在生产路径中采用该方案。 + +本实验保留摘要，再追加选出的原文，并向 Jev 提供工具结果正文。 +它使用小规模合成任务、小预算摘要设置和精确字段评分，不包含历史恢复或连续压缩。 +本实验得分不能与 Hermes 的召回百分比直接比较，也不能证明优于 Hermes 生产环境的摘要＋恢复路径。 +两项实验考察不同策略；本实验尚缺少同预算选择器对照，因此也不能把收益全部归因于 Jev 排序。 + ## 复现 ### 模型与运行环境