diff --git a/benchmark/compaction/RESULTS.md b/benchmark/compaction/RESULTS.md index 3ac769a8..92eb5989 100644 --- a/benchmark/compaction/RESULTS.md +++ b/benchmark/compaction/RESULTS.md @@ -24,13 +24,17 @@ each summary remains byte-identical across its six conditions. | H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 | | H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 | -![Quality, downstream input and compression overhead](../../docs/figure/compaction-topn-results.svg) +![Quality, downstream input and Jev overhead by Top N](../../docs/figure/compaction-topn-results.svg) Both summary mechanisms benefit on this panel: S rises from **82.0% to 99.5%** at Top 3; H rises from **79.4% to 97.9%**. Mean downstream inputs at Top 3 are **3,241 / 3,193**, versus Full's **4,014**: reductions of **19.3% / 20.4%**. Top 4–5 do not improve these point estimates. This is an exploratory result on four synthetic task templates, not a general optimal-N claim, equivalence test or proof that Jev beats other equal-budget selectors. +Panel C plots Jev overhead against Top N: zero at Top 0, then the same **4,162 input + 344 output tokens** +(4,506 total) at Top 1–5 for either summary mechanism. Ranking scores every available unit once, irrespective of how +many units are subsequently retained. Summary-generation usage remains in the table. + ## Token and cost accounting All table values are mean provider tokens. Answer input includes the original system/task, summary/evidence and held-out diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md index 2873bebf..7eca415b 100644 --- a/docs/en/context_compaction.md +++ b/docs/en/context_compaction.md @@ -127,7 +127,8 @@ Jev-selected originals across mechanisms. On four primary task templates, S improves from **82.0% to 99.5%** at Top 3; H improves from **79.4% to 97.9%**. Their mean answer inputs are **3,241 / 3,193 tokens**, versus Full's **4,014 / 100%**. The figure separates downstream -input from summary/ranking overhead; fewer answer tokens do not establish lower total monetary cost. +input from Jev ranking overhead; summary-generation usage is in the results table. Top 0 has no Jev call; Top 1–5 share +one complete ranking and have identical Jev overhead. Fewer answer tokens do not establish lower total monetary cost. The panel has 14 predefined query/answer templates, 546 contexts and 1,092 direct-Qwen answers. Ten short/query-only controls are reported separately. H reuses Hermes' compaction summary-generation pipeline with a smaller budget @@ -136,5 +137,5 @@ selection. All H-only contexts shorten Full; at Top 3–5 only 9/12 primary cont in scoring. This is an incremental, unequal-length comparison without agent continuation. -See [models, framework versions, parameters and reproduction steps](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/SETTINGS.md), -the [query/answer protocol](https://github.com/jinliyl/ReMe/blob/d2638b166f6daeec6f3e842750313526617786a1/benchmark/compaction/STUDY.md) and [complete results and token accounting](https://github.com/jinliyl/ReMe/blob/d2638b166f6daeec6f3e842750313526617786a1/benchmark/compaction/RESULTS.md). +See [models, framework versions, parameters and reproduction steps](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/SETTINGS.md), +the [query/answer protocol](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/STUDY.md) and [complete results and token accounting](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/RESULTS.md). diff --git a/docs/figure/compaction-topn-results.svg b/docs/figure/compaction-topn-results.svg index e4f279d9..adb8aa15 100644 --- a/docs/figure/compaction-topn-results.svg +++ b/docs/figure/compaction-topn-results.svg @@ -6,7 +6,7 @@ - 2026-10-16T17:18:34.287481 + 2026-10-16T17:38:17.442645 image/svg+xml @@ -40,47 +40,47 @@ z +" clip-path="url(#pf6d0324672)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/> - 0 + 0 - 1 + 1 - 2 + 2 - 3 + 3 - 4 + 4 - 5 + 5 - Jev Top N (0 = summary only) + Jev Top N (0 = summary only) @@ -88,81 +88,81 @@ L 420.832463 180.72 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 70 + 70 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 75 + 75 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 80 + 80 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 85 + 85 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 90 + 90 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 95 + 95 +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 100 + 100 - Exact-field score (%) + Exact-field score (%) @@ -177,9 +177,9 @@ L 217.646476 227.42625 L 279.217987 185.12625 L 340.789498 196.87625 L 402.361009 202.75125 -" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - - + + + + + + + - 82.0% + 82.0% - 99.5% + 99.5% - 97.4% + 97.4% +" clip-path="url(#pf6d0324672)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - - + + + + + + + - 79.4% + 79.4% - 97.9% + 97.9% - 96.9% + 96.9% - Full: 100% + Full: 100% - A Answer quality + A Answer quality @@ -267,47 +267,47 @@ z +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/> - 0 + 0 - 1 + 1 - 2 + 2 - 3 + 3 - 4 + 4 - 5 + 5 - Jev Top N (0 = summary only) + Jev Top N (0 = summary only) @@ -315,48 +315,48 @@ L 877.440222 195.184761 +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 2500 + 2500 +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 3000 + 3000 +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 3500 + 3500 +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - 4000 + 4000 - Mean answer input tokens + Mean answer input tokens @@ -371,24 +371,24 @@ L 674.254235 325.195957 L 735.825746 291.856813 L 797.397258 259.935026 L 858.968769 239.393778 -" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> + + + + + + + - 2,480 + 2,480 - 3,241 + 3,241 - 3,660 + 3,660 - - - - - - - +" clip-path="url(#p6f3dc61b1a)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> + + + + + + + - 2,432 + 2,432 - 3,193 + 3,193 - 3,613 + 3,613 - Full: 4,014 + Full: 4,014 - B Downstream context + B Downstream context @@ -434,218 +434,349 @@ z - - - - + - 0 + 0 - - - - + - 2k + 1 - - - - + - 4k + 2 - - - - + - 6k + 3 - - Mean provider tokens per generation + + + + 4 + + + + + + 5 + + + + Jev Top N (0 = summary only) + + + + + + + + + + 0 + + + + + + + + + 1k + + + + + + + + + 2k + + + + + + + + + 3k + + + + + + + + + 4k + + + + + + + + + 5k + + + + Mean Jev tokens per run - - +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - - - - S generation - - - 5,379 / 200 +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - - - - H generation - - - 3,120 / 152 +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - +" clip-path="url(#p327e181977)" style="fill: #1a9985"/> - - Jev ranking + + - - 4,162 / 344 + + - - C Compression overhead + + + + + + + + + + + + + + 0 + + + 4,506 + + + 4,506 + + + 4,506 + + + 4,506 + + + 4,506 + + + Same ranking cost for S and H + + + C Jev ranking overhead + + + + + + + + - - + +" style="fill: #1a9985"/> - - Input + + Input (4,162) - - + +" style="fill: #b3ddd6"/> - - Output + + Output (344) - - Jev across two summary strategies + + Jev across two summary strategies - - Same source trajectories. Frozen summaries. Identical Jev selections. + + Same source trajectories. Frozen summaries. Identical Jev selections. - - At Top 3 + + At Top 3 - - S: 99.5% | 3,241 input tokens H: 97.9% | 3,193 input tokens + + S: 99.5% | 3,241 input tokens H: 97.9% | 3,193 input tokens - - Primary cohort: 4 task templates, 24 answers per method. Another 10 short/query-only controls are reported separately. + + Primary cohort: 4 task templates, 24 answers per method; 10 separate controls. Top 3–5 shorten 9/12 primary contexts per method. - - Overhead includes recorded retries; one summary + one Jev ranking per standalone fusion run. Jev cost is unchanged for Top 1–5. + + Jev overhead is zero at Top 0 and identical at Top 1–5 for S and H. Each fused method is charged one complete ranking, including retries. - - Tokens are not currency. Top 3–5 shorten 9/12 primary contexts per method; all cases remain scored. Direct Qwen evaluation; no agent continuation. + + Tokens are not currency; summary-generation overhead is listed in the results table. Direct Qwen evaluation; no agent continuation. - + - + - - Generic summary S + + Generic summary S - + - + - - Hermes small-budget H + + Hermes small-budget H - - + - - Full context + + Full context - + - + - + diff --git a/docs/zh/context_compaction.md b/docs/zh/context_compaction.md index 5c91e973..7a1cdeb3 100644 --- a/docs/zh/context_compaction.md +++ b/docs/zh/context_compaction.md @@ -117,7 +117,8 @@ Top 0 表示纯摘要；每轮摘要逐字固定，两种机制使用完全相 4 个主任务中，Top 3 将 S 从 **82.0% 提升至 99.5%**，将 H 从 **79.4% 提升至 97.9%**。 平均回答输入分别为 **3241 / 3193 tokens**，Full 为 **4014 tokens / 100%**。 -图中分别展示效果、回答输入和摘要／排序开销；回答输入减少不等于总费用降低。 +图中分别展示效果、回答输入和 Jev 排序开销；摘要生成用量见结果表。Top 0 不调用 Jev，Top 1～5 共用完整排名， +因此 Jev 开销相同。回答输入减少不等于总费用降低。 共 14 个预设 query/answer 模板、546 个上下文、1092 次直接 Qwen 回答；10 个短任务对照单独报告。 H 复用 Hermes 的上下文压缩摘要生成流程，并采用小预算与长度约束；实验直接生成摘要，不经过自动触发和首尾窗口选择。 @@ -125,5 +126,5 @@ H 复用 Hermes 的上下文压缩摘要生成流程，并采用小预算与长 Top 3～5 每组主任务只有 9/12 个上下文实现缩短，全部样例均保留评分。这是非等长的增量比较，未继续运行 agent。 -模型、框架版本、参数和复现步骤见[实验设置](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/SETTINGS.md)。 -详见[query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/d2638b166f6daeec6f3e842750313526617786a1/benchmark/compaction/STUDY.md)及[完整结果与 token 开销](https://github.com/jinliyl/ReMe/blob/d2638b166f6daeec6f3e842750313526617786a1/benchmark/compaction/RESULTS.md)。 +模型、框架版本、参数和复现步骤见[实验设置](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/SETTINGS.md)。 +详见[query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/STUDY.md)及[完整结果与 token 开销](https://github.com/jinliyl/ReMe/blob/65ef7358417d21746cf9f95aaf9005aa521c5688/benchmark/compaction/RESULTS.md)。