From e6e38d943c55cd457739c9b33137ed264b58547a Mon Sep 17 00:00:00 2001 From: "jinli.yl" Date: Fri, 16 Oct 2026 12:11:34 +0800 Subject: [PATCH] docs: report summary-aware incremental evidence results --- docs/en/context_compaction.md | 19 +++++++++++++++++++ docs/zh/context_compaction.md | 16 ++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md index 3e534d57..ec5c7245 100644 --- a/docs/en/context_compaction.md +++ b/docs/en/context_compaction.md @@ -137,3 +137,22 @@ This run demonstrates input reduction and lower observed compaction latency, wit improvements over summarization. It uses one task and its synthetic variant, with two answers per context. Budgets use a local tokenizer proxy, input usage comes from the API, and total billing cost and archive recovery were not tested. See the benchmark for scoring, complete results, omission analysis and reproduction commands. + + +## Summary plus incremental evidence + +The [multi-arm protocol and results (Chinese)](https://github.com/jinliyl/ReMe/blob/ba57fc5cdb80f3eb8b53ea12588254f3212a2c6f/benchmark/compaction/HYBRID_RESULTS.md) compare fixed-budget allocation, additions to a frozen summary, +and summary-aware evidence selection. These are direct Qwen evaluations: payment input comes from a Hermes-collected +synthetic task, plus an independent replica fixture. Hermes does not continue its agent loop during evaluation. + +The summary-aware arm keeps every summary byte unchanged, then adds at most 400 evidence tokens selected at probability +0.5 or higher. A task-only selector uses the same threshold as a control. Across 16 valid frozen summary instances, strict fact +accuracy improved in 7 and tied in 9, with no observed regression; the mean gain was 12.2 percentage points. At a 300-token +base-summary allowance, the short payment case improved from 60% to 100%, and its long-log variant from 75% to 100%. + +![Summary-aware evidence additions to unchanged summaries](../figure/decision-compaction-summary-aware.svg) + +This post-hoc diagnostic covers two task families, including a noise variant and repeated generations. It is not held-out +validation or a monotonicity guarantee. Appending evidence increases input: the long-log 600-token baseline grew from +3070 to 3452 actual input tokens, with strict accuracy improving from 70% to 87.5%. The hybrid policy remains experimental +and does not change the production `compact_context` contract above. diff --git a/docs/zh/context_compaction.md b/docs/zh/context_compaction.md index b6be29c5..033651ff 100644 --- a/docs/zh/context_compaction.md +++ b/docs/zh/context_compaction.md @@ -125,3 +125,19 @@ Jev 分块筛选、首尾截取和普通 Qwen 摘要。证据预算为 600 / 100 这项实验显示了输入压缩和较短的压缩耗时，尚未证明稳定优于普通摘要。它只有一个任务及其合成变体， 每个上下文回答两次；预算使用本地 tokenizer 代理，总输入使用 API usage，未计算总费用或测试原文恢复。 具体评分方法、完整结果、遗漏分析与复现命令均见 Benchmark 文档。 + + +## 摘要与增量原文 + +[多组评测方案与结果](https://github.com/jinliyl/ReMe/blob/ba57fc5cdb80f3eb8b53ea12588254f3212a2c6f/benchmark/compaction/HYBRID_RESULTS.md)比较了固定预算配比、冻结摘要后追加原文，以及让决策模型看到摘要后选择互补证据。 +这些实验直接调用 Qwen；支付数据来自 Hermes 采集的合成任务，另有独立副本故障 fixture，没有运行 Hermes 代理继续执行。 + +摘要感知组保留同一份纯摘要，再追加最多 400 token 原文，只选择增量有用概率不低于 0.5 的片段。 +同阈值的任务相关性选择作为对照。16 份有效冻结摘要实例中，7 份严格得分提升、9 份持平，未观察到下降， +严格事实得分平均提高 12.2 个百分点。支付短样本的 300-token 基础摘要从 60% 到 100%，长日志同档从 75% 到 100%。 + +![Summary-aware evidence additions to unchanged summaries](../figure/decision-compaction-summary-aware.svg) + +这是两个任务族上的事后机制诊断，包含增噪变体和重复生成，不能视为独立测试集或单调性保证。 +追加原文增加总输入：长日志 600-token 基础摘要从 3070 到 3452 个实际输入 token，严格得分从 70% 到 87.5%。 +该策略仍是 benchmark 实验，没有改变上面的 `compact_context` 生产契约。