scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-10-23 05:35:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

docs: explain compaction experiment background setup and conclusions inline

Browse source
This commit is contained in:
jinli.yl 2026-10-16 17:56:12 +08:00
parent 52b065e4a8
commit 21f754b08a
2 changed files with 75 additions and 32 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

57
docs/en/context_compaction.md
View file

@ -119,25 +119,48 @@ latency and total cost alongside token reduction. ReMe does not automatically su
## Jev across two summary strategies
One 13-method experiment compares Full context with a generic summary S and Hermes compaction summary H
(small-budget setting), each plus Jev Top 0–5. Top 0 means summary only. Every condition preserves its frozen summary and shares the same
Jev-selected originals across mechanisms.
### Background
![Quality, downstream input and compression overhead](../figure/compaction-topn-results.svg)
Summaries shorten history but can omit exact identifiers, configuration values or log details needed for a later answer.
This experiment asks: **can a few Jev-selected original evidence blocks recover information missing from a fixed summary,
and does that benefit occur with both a generic summary and a Hermes summary?** Answer input and total fees show the
tradeoff between retained information, context length and cost.
On four primary task templates, S improves from **82.0% to 99.5%** at Top 3; H improves from **79.4% to 97.9%**.
Their mean answer inputs are **3,241 / 3,193 tokens**, versus Full's **4,014 / 100%**. Panel C shows **total fees for one compression plus one Qwen answer**:
summary generation + Jev ranking + downstream input/output. S increases **$0.014293 → $0.016429** from Top 0 to 5;
H increases **$0.010245 → $0.012388**. Full costs **$0.006992** for its answer.
Jev's fee stays flat after Top 1; additional retained evidence increases Qwen's input fee. These are uncached,
dated public-list-price estimates, not invoices or evidence of one-answer savings.
### Experimental setup
The panel has 14 predefined query/answer templates, 546 contexts and 1,092 direct-Qwen answers. Ten short/query-only
controls are reported separately. H reuses Hermes' compaction summary-generation pipeline with a smaller budget
and length constraints. The experiment invokes summary generation directly, bypassing automatic triggering and head/tail
selection. All H-only contexts shorten Full; at Top 3–5 only 9/12 primary contexts per method do so, with all cases retained
in scoring. This is an incremental, unequal-length comparison without agent continuation.
| Item | Recorded setting |
| --- | --- |
| Data | 14 synthetic task templates with predefined queries/answers: one existing payment trajectory and 13 actual read-only Hermes executions. Four primary tasks were selected by source length before collection: payment retries, checkout 504, server-error localization and build-log token lookup. Ten short/query-only controls are reported separately. |
| Framework and models | Hermes Agent `AIAgent`, pinned to `30463589eb54`; **`qwen3.8-max`** for collection, both summary mechanisms and answering. **`jev-1.13.0`** through AgentScope **2.0.10** ranks evidence. Hermes collects trajectories; evaluation calls Qwen directly. |
| Summary mechanisms | **S** uses a generic handoff-summary prompt. **H** is Hermes compaction summary under a small-budget setting: its `legacy` generation pipeline with adjusted budget/length constraints. Summary generation is invoked directly, bypassing automatic triggering and head/tail selection. |
| 13 methods | **Full context** retains all pre-evaluation history. **S + Top 0–5** and **H + Top 0–5** each contribute six methods. Top 0 is summary only; Top N appends N whole evidence blocks. Both curves share identical Jev selections; each generation freezes S/H byte-for-byte. |
| Budget | Forced small-budget simulation: summary-body allowance **64–256 tokens**, derived from evidence length; complete answer-input ceilings are task-specific. Larger N retains more evidence, so lengths differ. Full is an uncompressed reference; it need not exceed every case ceiling. |
| Evaluation | Summarization/ranking receives the original task and evidence, never the held-out questions or gold. Final collection reports are excluded. Answers return JSON and are scored with predefined type-strict field rules, **without an LLM judge**. Average within tasks, then weight tasks equally. |
| Repeats and size | Three summary/ranking generations per task, two answers per context: **546 contexts / 1,092 answers**. The figure shows four primary tasks, 24 answers per method; repeated answers are not additional tasks. Summary/answer temperature is 0, with thinking disabled. |
Jev scores all available evidence against the original task once; N changes the number retained, not ranking-call count.
This is a gold-answer **information-retention evaluation**, without resuming Hermes' tool loop to complete the task.
### Reading the figure and conclusions
![Answer quality, downstream input and total cost across two summary strategies](../figure/compaction-topn-results.svg)
- **A: selected originals help both summary mechanisms on this panel.** Full scores 100%; Top 3 improves S from
**82.0% to 99.5%**, and H from **79.4% to 97.9%**. This supports supplementing these summaries with original evidence;
it does not establish general superiority or statistical equivalence to Full.
- **B: higher retention consumes more context than summary alone.** Top 3 uses **3,241 / 3,193** mean answer-input tokens
for S/H, versus Full's **4,014**, reductions of **19.3% / 20.4%**. Both exceed their summary-only input. Top 4–5 do not
improve the scores: more original text does not guarantee better answers.
- **C: there is no total-cost saving over a single post-compression answer here.** Fees include **summary generation +
one Jev ranking + one Qwen answer, input and output**. At Top 3, S/H cost about **$0.015738 / $0.011697**, versus Full's
**$0.006992**. Ranking fees are identical at Top 1–5; reading additional evidence increases Qwen's fee. These are
public USD list-price estimates dated 2026-10-16, excluding cache discounts, free quotas and account offers, not invoices.
**Scope:** the primary result covers four synthetic task templates; ten controls are separate. All 42 H-only contexts
shorten Full, but Top 3–5 shorten only **9/12 primary contexts per method**, with every case still scored. Strict field scoring includes format compliance; semantically equivalent wording can still lose credit.
The result shows an information/length tradeoff. Without equal-length longer-summary or alternative-selector controls,
the entire gain cannot be attributed to Jev ranking itself. Future-turn cost amortization and agent continuation success
remain unmeasured.
<!-- Payment has 10 units (maximum Top 10); actual N is min(N, available units). -->
See [models, framework versions, parameters and reproduction steps](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/SETTINGS.md),
the [query/answer protocol](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/STUDY.md) and [complete results and token accounting](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/RESULTS.md).
[Full settings and reproduction](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/SETTINGS.md) · [Queries, answers and scoring](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/STUDY.md) · [All 13 methods and short-task controls](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/RESULTS.md)

50
docs/zh/context_compaction.md
View file

@ -110,23 +110,43 @@ ReMe 不会自动生成摘要或写入长期记忆。
## 两种摘要机制下的 Jev 增益
只保留一个 13 组实验：Full context，以及普通摘要 S、Hermes 上下文压缩摘要 H（小预算版） 各自追加 Jev Top 0～5。
Top 0 表示纯摘要；每轮摘要逐字固定，两种机制使用完全相同的 Jev 排名与原文块。
### 实验背景
![Quality, downstream input and compression overhead](../figure/compaction-topn-results.svg)
摘要可以缩短历史，但可能丢失后续回答需要的精确标识符、配置值或日志细节。本实验考察：**在摘要保持不变的基础上，
追加 Jev 选出的少量原文，能否补回这些信息？这一增益能否同时出现在普通摘要和 Hermes 摘要中？**
同时报告回答输入量和总费用，观察信息保留、上下文长度与开销之间的取舍。
4 个主任务中，Top 3 将 S 从 **82.0% 提升至 99.5%**，将 H 从 **79.4% 提升至 97.9%**。
平均回答输入分别为 **3241 / 3193 tokens**，Full 为 **4014 tokens / 100%**。
C 图纵轴为**一次压缩后回答的总费用（USD）**：摘要生成 + Jev 排序 + Qwen 回答输入／输出。
从 Top 0 到 Top 5，S 为 **$0.014293 → $0.016429**，H 为 **$0.010245 → $0.012388**；
Full 的回答费用为 **$0.006992**。Jev 排名费在 Top 1～5 不变，但保留更多原文会增加 Qwen 的输入费。
按公开标价、不计缓存折扣估算，不等同于实际账单；当前单次回答口径下，压缩组总费用均高于 Full。
### 实验设置
共 14 个预设 query/answer 模板、546 个上下文、1092 次直接 Qwen 回答；10 个短任务对照单独报告。
H 复用 Hermes 的上下文压缩摘要生成流程，并采用小预算与长度约束；实验直接生成摘要，不经过自动触发和首尾窗口选择。
纯 H 的 42 个上下文全部短于 Full；
Top 3～5 每组主任务只有 9/12 个上下文实现缩短，全部样例均保留评分。这是非等长的增量比较，未继续运行 agent。
| 项目 | 本次设置 |
| --- | --- |
| 数据 | 14 个预设 query/answer 的合成任务模板：1 条已有支付轨迹，另用 Hermes 实际采集 13 条只读轨迹。按来源长度预先确定 4 个主任务：支付重试、checkout 504、服务器错误定位、构建日志 token 查找；10 个短任务／query-only 对照单独报告。 |
| 框架与模型 | Hermes Agent `AIAgent`，版本固定为 `30463589eb54`；采集、两种摘要和最终回答均使用 **`qwen3.8-max`**。Jev 为 **`jev-1.13.0`**，通过 AgentScope **2.0.10** 调用。采集使用 Hermes，评分阶段直接调用 Qwen。 |
| 两种摘要 | **S**：通用任务交接摘要提示词。**H**：Hermes 上下文压缩摘要（小预算版），复用其 `legacy` 摘要生成流程，调整预算和长度约束；实验直接生成摘要，不经过自动触发和首尾窗口选择。 |
| 13 个对照组 | **Full context** 保留完整的评测前历史；**S + Top 0～5** 和 **H + Top 0～5** 各 6 组。Top 0 是纯摘要；Top N 在原摘要后追加 N 个完整证据块。两条曲线使用相同 Jev 排名和原文，每轮的 S／H 均逐字固定。 |
| 压缩预算 | 主动压缩的小预算模拟：摘要正文上限按证据长度设为 **64～256 tokens**；完整回答输入上限按任务计算。N 增大时保留更多原文，实际输入随之变化，各组并非等长比较；Full 是无压缩参考，不要求每个任务都超过该上限。 |
| 评测 | 原任务和证据用于摘要／排序，评测问题与标准答案不提供给它们；最终采集报告也排除。回答需返回 JSON，按预设字段的严格类型和值规则评分，**不用 LLM judge**。分数先在任务内平均，再对任务等权平均。 |
| 重复与规模 | 每个任务 3 轮摘要／排名，每个上下文回答 2 次；共 **546 个上下文、1092 次回答**。主图只展示 4 个主任务，每种方法 24 次回答；重复回答不等于新增任务。摘要和回答温度为 0，关闭 thinking。 |
Jev 先依据原任务给全部证据块打分，再选择 Top N；N 只影响保留数量，不增加排名调用次数。
这是预设标准答案的**信息保留评测**，没有让 Hermes 在压缩后继续调用工具完成任务。
### 读图与结论
![Answer quality, downstream input and total cost across two summary strategies](../figure/compaction-topn-results.svg)
- **A：两种摘要都能从追加原文中获益。** Full 的字段正确率为 100%；Top 3 将 S 从 **82.0% 提升至 99.5%**，
将 H 从 **79.4% 提升至 97.9%**。这支持“摘要 + 关键原文”在本组任务中补充摘要遗漏的信息，尚不能证明通用优势或与 Full 统计等价。
- **B：以更多上下文换取更高的信息保留率。** Top 3 的平均回答输入为 S **3241**、H **3193 tokens**，
相比 Full 的 **4014 tokens** 分别减少 **19.3%／20.4%**，但都高于各自纯摘要。Top 4～5 未继续提分，更多原文不保证效果更好。
- **C：按一次压缩后回答计算，当前没有总费用节省。** 总费用包含**摘要生成 + 一次 Jev 排名 + 一次 Qwen 回答的输入／输出**。
Top 3 的 S／H 费用约 **$0.015738／$0.011697**，Full 为 **$0.006992**。Jev 排名费在 Top 1～5 相同，
曲线的增长来自 Qwen 读取更多原文。费用按 2026-10-16 的公开 USD 标价估算，未计缓存折扣、免费额度或账户优惠；不是实际账单。
**结果边界：** 主结论来自 4 个合成任务模板；10 个短任务对照不混入主图。纯 H 的 42 个上下文均短于 Full，
但 Top 3～5 每种方法仅有 **9/12 个主任务上下文**实际缩短，全部样例均保留评分。严格字段评分包含格式要求，语义相同但格式不同也可能失分。
当前结果展示了信息保留与长度的取舍；尚未加入等长度的更长摘要或其他选择器对照，不能将全部增益归因于 Jev 排序本身。
多轮摊销后的费用收益和 agent 继续执行的成功率均未验证。
<!-- 支付轨迹有 10 个证据块，最多 Top 10；实际 N 为 min(N, 可用证据块数)。 -->
模型、框架版本、参数和复现步骤见[实验设置](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/SETTINGS.md)。
详见[query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/STUDY.md)及[完整结果与 token 开销](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/RESULTS.md)。
[完整实验设置与复现步骤](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/SETTINGS.md) · [query、answer 与评分规则](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/STUDY.md) · [13 组完整结果及短任务对照](https://github.com/jinliyl/ReMe/blob/8f796715ab107550a78d462f72f9e1b934ef7daa/benchmark/compaction/RESULTS.md)