From 7686be7baaddd4094b454c30c8fca61f5429df98 Mon Sep 17 00:00:00 2001 From: "jinli.yl" Date: Fri, 16 Oct 2026 18:34:50 +0800 Subject: [PATCH] docs: compare compaction-only costs and omit Full from cost panel --- benchmark/compaction/README.md | 4 +- benchmark/compaction/RESULTS.md | 91 +++++----- benchmark/compaction/SETTINGS.md | 20 +-- benchmark/compaction/STUDY.md | 8 +- benchmark/compaction/pricing.json | 2 +- docs/en/context_compaction.md | 14 +- docs/figure/compaction-topn-results.svg | 213 ++++++++++++------------ docs/zh/context_compaction.md | 12 +- 8 files changed, 178 insertions(+), 186 deletions(-) diff --git a/benchmark/compaction/README.md b/benchmark/compaction/README.md index 06abf31a..fa1a43ce 100644 --- a/benchmark/compaction/README.md +++ b/benchmark/compaction/README.md @@ -73,5 +73,5 @@ For the recorded format amendment, native summaries used the pre-amendment sourc and permits only the held-out `exam` field to differ; the summary generator never receives that field. `costs.json` reports standalone summary/ranking input and output overhead separately from answer usage, including -recorded retries. The chart converts summary + Jev + one Qwen answer usage to estimated USD with the dated rates in -[pricing.json](./pricing.json); cache discounts and trajectory collection are excluded. See [the formula](./SETTINGS.md#scoring-and-costs). +recorded retries. Panel C converts summary + Jev compaction usage to estimated USD with the dated rates in +[pricing.json](./pricing.json); cache discounts, downstream answering and trajectory collection are excluded. Full appears in A/B only. See [the formula](./SETTINGS.md#scoring-and-costs). diff --git a/benchmark/compaction/RESULTS.md b/benchmark/compaction/RESULTS.md index 61dbca3b..ab596748 100644 --- a/benchmark/compaction/RESULTS.md +++ b/benchmark/compaction/RESULTS.md @@ -8,46 +8,42 @@ each summary remains byte-identical across its six conditions. ## Four primary tasks -| Method | Fact score | Answer input | Summary in / out | Jev in / out | Total USD | +| Method | Fact score | Answer input | Summary in / out | Jev in / out | Compaction USD | | --- | ---: | ---: | ---: | ---: | ---: | -| Full context | 100.0% | 4,014 | 0 / 0 | 0 / 0 | $0.006992 | -| S | 82.0% | 2,480 | 5,379 / 200 | 0 / 0 | $0.014293 | -| S + Jev Top 1 | 83.0% | 2,753 | 5,379 / 200 | 4,162 / 344 | $0.014920 | -| S + Jev Top 2 | 94.5% | 2,974 | 5,379 / 200 | 4,162 / 344 | $0.015289 | -| S + Jev Top 3 | 99.5% | 3,241 | 5,379 / 200 | 4,162 / 344 | $0.015738 | -| S + Jev Top 4 | 98.1% | 3,496 | 5,379 / 200 | 4,162 / 344 | $0.016166 | -| S + Jev Top 5 | 97.4% | 3,660 | 5,379 / 200 | 4,162 / 344 | $0.016429 | -| H (Hermes, small budget) | 79.4% | 2,432 | 3,120 / 152 | 0 / 0 | $0.010245 | -| H + Jev Top 1 | 85.3% | 2,705 | 3,120 / 152 | 4,162 / 344 | $0.010877 | -| H + Jev Top 2 | 92.9% | 2,927 | 3,120 / 152 | 4,162 / 344 | $0.011244 | -| H + Jev Top 3 | 97.9% | 3,193 | 3,120 / 152 | 4,162 / 344 | $0.011697 | -| H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 | $0.012114 | -| H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 | $0.012388 | +| Full context | 100.0% | 4,014 | 0 / 0 | 0 / 0 | — | +| S | 82.0% | 2,480 | 5,379 / 200 | 0 / 0 | $0.009864 | +| S + Jev Top 1 | 83.0% | 2,753 | 5,379 / 200 | 4,162 / 344 | $0.010039 | +| S + Jev Top 2 | 94.5% | 2,974 | 5,379 / 200 | 4,162 / 344 | $0.010039 | +| S + Jev Top 3 | 99.5% | 3,241 | 5,379 / 200 | 4,162 / 344 | $0.010039 | +| S + Jev Top 4 | 98.1% | 3,496 | 5,379 / 200 | 4,162 / 344 | $0.010039 | +| S + Jev Top 5 | 97.4% | 3,660 | 5,379 / 200 | 4,162 / 344 | $0.010039 | +| H (Hermes, small budget) | 79.4% | 2,432 | 3,120 / 152 | 0 / 0 | $0.005902 | +| H + Jev Top 1 | 85.3% | 2,705 | 3,120 / 152 | 4,162 / 344 | $0.006077 | +| H + Jev Top 2 | 92.9% | 2,927 | 3,120 / 152 | 4,162 / 344 | $0.006077 | +| H + Jev Top 3 | 97.9% | 3,193 | 3,120 / 152 | 4,162 / 344 | $0.006077 | +| H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 | $0.006077 | +| H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 | $0.006077 | -![Quality, downstream input and estimated total fees by Top N](../../docs/figure/compaction-topn-results.svg) +![Quality, downstream input and compaction fees by Top N](../../docs/figure/compaction-topn-results.svg) Both summary mechanisms benefit on this panel: S rises from **82.0% to 99.5%** at Top 3; H rises from **79.4% to 97.9%**. Mean downstream inputs at Top 3 are **3,241 / 3,193**, versus Full's **4,014**: reductions of **19.3% / 20.4%**. Top 4–5 do not improve these point estimates. This is an exploratory result on four synthetic task templates, not a general optimal-N claim, equivalence test or proof that Jev beats other equal-budget selectors. -Panel C plots **one-compression + one-answer total fees** against Top N, separately for S and H. -S increases from **$0.014293** at Top 0 to **$0.016429** at Top 5; -H increases from **$0.010245** to **$0.012388**. -Full's answering fee is **$0.006992**. More selected evidence increases downstream Qwen input cost; -Jev's ranking fee itself stays **$0.000175** for every N > 0. +Panel C compares **compaction fees only: summary generation + Jev ranking**, with Full omitted. +Top 0 costs **$0.009864 for S / $0.005902 for H**. Top 1–5 each cost **$0.010039 / $0.006077**. +Jev adds **$0.000175** once per compaction, about **1.8% / 3.0%** above S/H summary generation respectively. +The plateau is expected: Jev scores all available evidence once, then different N values retain different score prefixes. ## Token and cost accounting -Token columns are mean actual provider usage. Answer input includes the original system/task, summary/evidence and -held-out question. **Total USD** includes one summary generation, one complete Jev ranking for N > 0, and **one Qwen -answer (input + output)**. Recorded summary/ranking retries are included; trajectory collection and calibration are -excluded. Full pays its answering fee and has no compression preprocessing. The two scoring repeats estimate the -mean cost of one answer; preprocessing is charged once, not amortized across them. - -For reference, preprocessing alone costs S $0.009864 / H $0.005902 at Top 0, -or S $0.010039 / H $0.006077 with Jev. Those preprocessing fees stay flat -at Top 1–5; the **overall** curve rises because the selected evidence must be read by Qwen when answering. +Token columns are mean actual provider usage. Answer input remains a separate context-length metric, including the +original system/task, compressed evidence and held-out question. **Compaction USD** includes one summary generation +and one complete Jev ranking for N > 0, including recorded retries. **Downstream answering, trajectory collection and +calibration are excluded.** Full does not perform compaction and has no fee entry in this comparison (shown as “—”); +it remains the reference for answer quality and input length in A/B. No zero-total-cost claim is made for Full. +The two scoring repeats do not amortize compaction fees. Prices checked **2026-10-16**, per million tokens: @@ -56,12 +52,11 @@ Prices checked **2026-10-16**, per million tokens: | Qwen `qwen3.8-max`, China (Beijing) | $1.65 | $4.951 | [Alibaba Cloud](https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max) | | Jev `jev-1.13.0`, direct TypeSafe | $0.042 | $0 | [TypeSafe](https://docs.typesafe.ai/models) | -`total_usd = ((summary_input + answer_input) × 1.65 + (summary_output + answer_output) × 4.951 -+ jev_input × 0.042) / 1,000,000`. +`compaction_usd = (summary_input × 1.65 + summary_output × 4.951 + jev_input × 0.042) / 1,000,000`. [pricing.json](./pricing.json) freezes the rates, sources and scope. These are **uncached public-list-price estimates**: -all prompt tokens are priced at the regular input rate, even where API usage reports cache hits. Free quotas, discounts, -credits, taxes and account billing differences are excluded. This is a one-answer list-price comparison, not invoice reconciliation. In this horizon every compressed method -costs more than Full; future-turn amortization or total savings have not been established. +all summary prompt tokens are priced at the regular input rate, even where API usage reports cache hits. Free quotas, +discounts, credits, taxes and account billing differences are excluded. This measures the incremental cost of adding +Jev to a summary, not end-to-end monetary savings or invoice reconciliation. One shared ranking serves Top 1–5 and both mechanisms in the experiment; each standalone fused method includes its full cost, neither multiplied by N nor divided across sweep conditions. @@ -90,21 +85,21 @@ the other 38 comprise 9 primary and 29 control contexts. All remain scored. Aver
Ten short/query-only controls -| Method | Fact score | Answer input | Summary in / out | Jev in / out | Total USD | +| Method | Fact score | Answer input | Summary in / out | Jev in / out | Compaction USD | | --- | ---: | ---: | ---: | ---: | ---: | -| Full context | 100.0% | 2,727 | 0 / 0 | 0 / 0 | $0.004692 | -| S | 67.1% | 2,315 | 1,099 / 74 | 0 / 0 | $0.006175 | -| S + Jev Top 1 | 84.7% | 2,493 | 1,099 / 74 | 929 / 50 | $0.006517 | -| S + Jev Top 2 | 87.5% | 2,573 | 1,099 / 74 | 929 / 50 | $0.006654 | -| S + Jev Top 3 | 92.9% | 2,687 | 1,099 / 74 | 929 / 50 | $0.006843 | -| S + Jev Top 4 | 92.9% | 2,722 | 1,099 / 74 | 929 / 50 | $0.006901 | -| S + Jev Top 5 | 92.5% | 2,722 | 1,099 / 74 | 929 / 50 | $0.006901 | -| H (Hermes, small budget) | 73.2% | 2,308 | 1,944 / 69 | 0 / 0 | $0.007532 | -| H + Jev Top 1 | 80.1% | 2,486 | 1,944 / 69 | 929 / 50 | $0.007871 | -| H + Jev Top 2 | 92.5% | 2,566 | 1,944 / 69 | 929 / 50 | $0.008019 | -| H + Jev Top 3 | 99.2% | 2,680 | 1,944 / 69 | 929 / 50 | $0.008204 | -| H + Jev Top 4 | 99.2% | 2,715 | 1,944 / 69 | 929 / 50 | $0.008263 | -| H + Jev Top 5 | 98.8% | 2,715 | 1,944 / 69 | 929 / 50 | $0.008264 | +| Full context | 100.0% | 2,727 | 0 / 0 | 0 / 0 | — | +| S | 67.1% | 2,315 | 1,099 / 74 | 0 / 0 | $0.002179 | +| S + Jev Top 1 | 84.7% | 2,493 | 1,099 / 74 | 929 / 50 | $0.002218 | +| S + Jev Top 2 | 87.5% | 2,573 | 1,099 / 74 | 929 / 50 | $0.002218 | +| S + Jev Top 3 | 92.9% | 2,687 | 1,099 / 74 | 929 / 50 | $0.002218 | +| S + Jev Top 4 | 92.9% | 2,722 | 1,099 / 74 | 929 / 50 | $0.002218 | +| S + Jev Top 5 | 92.5% | 2,722 | 1,099 / 74 | 929 / 50 | $0.002218 | +| H (Hermes, small budget) | 73.2% | 2,308 | 1,944 / 69 | 0 / 0 | $0.003550 | +| H + Jev Top 1 | 80.1% | 2,486 | 1,944 / 69 | 929 / 50 | $0.003589 | +| H + Jev Top 2 | 92.5% | 2,566 | 1,944 / 69 | 929 / 50 | $0.003589 | +| H + Jev Top 3 | 99.2% | 2,680 | 1,944 / 69 | 929 / 50 | $0.003589 | +| H + Jev Top 4 | 99.2% | 2,715 | 1,944 / 69 | 929 / 50 | $0.003589 | +| H + Jev Top 5 | 98.8% | 2,715 | 1,944 / 69 | 929 / 50 | $0.003589 | These controls are not ten additional long-context tasks. Their small source-unit counts can make Top N saturate. diff --git a/benchmark/compaction/SETTINGS.md b/benchmark/compaction/SETTINGS.md index 325d3f1b..671df12e 100644 --- a/benchmark/compaction/SETTINGS.md +++ b/benchmark/compaction/SETTINGS.md @@ -119,20 +119,18 @@ generations, generations within tasks, then tasks equally. Report primary/contro Answer input includes system/task, compressed history and exam. Standalone-method overhead includes one summary and, for N > 0, one whole Jev ranking, including recorded retries. Ranking cost is identical across N and both mechanisms; do not multiply by N or divide by sweep size. Evaluation repeats do not amortize overhead. Report input/output tokens -by model. Total fees use [pricing.json](./pricing.json), checked 2026-10-16: Qwen `qwen3.8-max` China (Beijing) +by model. Compaction fees use [pricing.json](./pricing.json), checked 2026-10-16: Qwen `qwen3.8-max` China (Beijing) input **$1.65/M**, output **$4.951/M** ([official rate](https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max)); Jev `jev-1.13.0` input **$0.042/M**, output free ([official rate](https://docs.typesafe.ai/models)). -`total_usd = ((summary_input + answer_input) * 1.65 + (summary_output + answer_output) * 4.951 -+ jev_input * 0.042) / 1e6`. -At Top 0 charge summary generation plus one Qwen answer; Top 1–5 add one identical whole Jev ranking -and charge the Qwen answer on the enlarged context. Mean usage is averaged across -the same tasks/generations; fees are computed from unrounded usage. Recorded retries are included. The plotted amount -includes one downstream answer and excludes collection/calibration. Full pays its actual mean answering fee and -no compression preprocessing. Two scoring repeats estimate the cost of one answer; they do not amortize overhead. -These are uncached list-price estimates: cache hits are deliberately charged at the regular input rate for comparison. -No free quotas, promotions, taxes or account-specific adjustments are applied. Prices are a dated snapshot, not an invoice; -no total monetary saving is claimed. +`compaction_usd = (summary_input * 1.65 + summary_output * 4.951 + jev_input * 0.042) / 1e6`. +At Top 0 charge only summary generation; Top 1–5 each add one identical complete Jev ranking. Mean usage is averaged +across the same tasks/generations, and fees are computed from unrounded usage. Recorded summary/ranking retries are +included. **Downstream answering, collection and calibration are excluded.** Full is excluded from panel C and has +“—” in the cost table; it remains the quality/input reference in A/B. Scoring repeats do not amortize preprocessing. +These are uncached list-price estimates: summary cache hits are charged at the regular input rate for comparison. +No free quotas, promotions, taxes or account-specific adjustments are applied. Prices are a dated snapshot, not an +invoice. This scope measures summary + Jev compaction overhead, not total execution cost or monetary savings. ## Reproduction levels and artifact availability diff --git a/benchmark/compaction/STUDY.md b/benchmark/compaction/STUDY.md index bcfcf354..502980dd 100644 --- a/benchmark/compaction/STUDY.md +++ b/benchmark/compaction/STUDY.md @@ -111,10 +111,10 @@ The extension keeps the existing per-task ceilings. Strict shrinkage versus Full filtering or removing conditions: Top 3–5 each shorten 9/12 primary contexts in both mechanisms. The H fusion arms shorten 172/210 contexts (51/60 primary, 121/150 controls). Summary-only H still shortens all 42/42. -For costs, each independent method includes one summary generation when applicable, one complete Jev ranking when -N > 0, and one downstream answer. Recorded retry usage is included. One ranking is shared across the Top-N sweep and +For compaction costs, each independent method includes one summary generation and one complete Jev ranking when +N > 0. Downstream answering is excluded. Recorded retry usage is included. One ranking is shared across the Top-N sweep and mechanisms in the actual experiment; its standalone-method cost is neither multiplied by N nor divided by sweep size. The two answer repeats do not amortize compression overhead. Report provider input/output tokens separately. -The current chart estimates summary + Jev + one downstream answer fees from the dated public USD rates in [pricing.json](./pricing.json), -using uncached input pricing and excluding trajectory collection. [Settings](./SETTINGS.md#scoring-and-costs) define +Panel C estimates summary + Jev compaction fees from the dated public USD rates in [pricing.json](./pricing.json), +using uncached input pricing and excluding downstream answering and trajectory collection. Full appears in A/B only. [Settings](./SETTINGS.md#scoring-and-costs) define the formula and exclusions; this is not an account invoice or a claim of total pipeline savings. diff --git a/benchmark/compaction/pricing.json b/benchmark/compaction/pricing.json index 1540a7ff..1f588509 100644 --- a/benchmark/compaction/pricing.json +++ b/benchmark/compaction/pricing.json @@ -13,6 +13,6 @@ "output": 0.0, "source": "https://docs.typesafe.ai/models" }, - "scope": "One summary generation plus one complete Jev ranking when N > 0, plus one downstream Qwen answer; recorded summary/ranking retries included; trajectory collection excluded", + "scope": "Compaction only: one summary generation plus one complete Jev ranking when N > 0; recorded retries included; downstream answering and trajectory collection excluded; Full omitted from the cost comparison", "billing_basis": "Public list-price estimate; all input charged at uncached rate; no credits, promotions, taxes, exchange conversion or invoice reconciliation" } diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md index 54d4f94b..2847e366 100644 --- a/docs/en/context_compaction.md +++ b/docs/en/context_compaction.md @@ -124,7 +124,7 @@ latency and total cost alongside token reduction. ReMe does not automatically su Summaries shorten history but can omit exact identifiers, configuration values or log details needed for a later answer. This experiment asks: **can a few Jev-selected original evidence blocks recover information missing from a fixed summary, and does that benefit occur with both a generic summary and a Hermes summary?** Answer input and total fees show the -tradeoff between retained information, context length and cost. +tradeoff between retained information, context length and compaction cost. ### Experimental setup @@ -143,7 +143,7 @@ This is a gold-answer **information-retention evaluation**, without resuming Her ### Reading the figure and conclusions -![Answer quality, downstream input and total cost across two summary strategies](../figure/compaction-topn-results.svg) +![Answer quality, downstream input and compaction cost across two summary strategies](../figure/compaction-topn-results.svg) - **A: selected originals help both summary mechanisms on this panel.** Full scores 100%; Top 3 improves S from **82.0% to 99.5%**, and H from **79.4% to 97.9%**. This supports supplementing these summaries with original evidence; @@ -151,10 +151,12 @@ This is a gold-answer **information-retention evaluation**, without resuming Her - **B: higher retention consumes more context than summary alone.** Top 3 uses **3,241 / 3,193** mean answer-input tokens for S/H, versus Full's **4,014**, reductions of **19.3% / 20.4%**. Both exceed their summary-only input. Top 4–5 do not improve the scores: more original text does not guarantee better answers. -- **C: there is no total-cost saving over a single post-compression answer here.** Fees include **summary generation + - one Jev ranking + one Qwen answer, input and output**. At Top 3, S/H cost about **$0.015738 / $0.011697**, versus Full's - **$0.006992**. Ranking fees are identical at Top 1–5; reading additional evidence increases Qwen's fee. These are - public USD list-price estimates dated 2026-10-16, excluding cache discounts, free quotas and account offers, not invoices. +- **C: compare compaction fees only.** Top 0 pays for Qwen summary generation: **$0.009864 / $0.005902** for S/H. + Top 1–5 each pay for summary generation plus one complete Jev ranking: **$0.010039 / $0.006077**. + Jev adds **$0.000175**, about **1.8% / 3.0%** of S/H summary fees. The curve plateaus because Jev scores all evidence + once before selecting Top N. **C excludes downstream answering and omits Full; Full remains the reference in A/B.** + Fees use public USD list prices dated 2026-10-16, excluding cache discounts, free quotas and account offers. + This is neither an invoice nor evidence of end-to-end savings. **Scope:** the primary result covers four synthetic task templates; ten controls are separate. All 42 H-only contexts shorten Full, but Top 3–5 shorten only **9/12 primary contexts per method**, with every case still scored. Strict field scoring includes format compliance; semantically equivalent wording can still lose credit. diff --git a/docs/figure/compaction-topn-results.svg b/docs/figure/compaction-topn-results.svg index 05be182d..9e3750f6 100644 --- a/docs/figure/compaction-topn-results.svg +++ b/docs/figure/compaction-topn-results.svg @@ -6,7 +6,7 @@ - 2026-10-16T17:47:05.316892 + 2026-10-16T18:18:39.704996 image/svg+xml @@ -40,7 +40,7 @@ z +" clip-path="url(#peabac2b509)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/> @@ -88,7 +88,7 @@ L 420.832463 180.72 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -99,7 +99,7 @@ L 420.832463 434.52 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -110,7 +110,7 @@ L 420.832463 392.22 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -121,7 +121,7 @@ L 420.832463 349.92 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -132,7 +132,7 @@ L 420.832463 307.62 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -143,7 +143,7 @@ L 420.832463 265.32 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -154,7 +154,7 @@ L 420.832463 223.02 +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -177,9 +177,9 @@ L 217.646476 227.42625 L 279.217987 185.12625 L 340.789498 196.87625 L 402.361009 202.75125 -" clip-path="url(#p766555bb3d)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - - + + + + + + + @@ -216,9 +216,9 @@ L 217.646476 240.645 L 279.217987 198.345 L 340.789498 207.1575 L 402.361009 207.1575 -" clip-path="url(#p766555bb3d)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> +" clip-path="url(#peabac2b509)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - - + + + + + + + @@ -267,7 +267,7 @@ z +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/> @@ -315,7 +315,7 @@ L 877.440222 195.184761 +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -326,7 +326,7 @@ L 877.440222 384.495652 +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -337,7 +337,7 @@ L 877.440222 321.965217 +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -348,7 +348,7 @@ L 877.440222 259.434783 +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -371,14 +371,14 @@ L 674.254235 325.195957 L 735.825746 291.856813 L 797.397258 259.935026 L 858.968769 239.393778 -" clip-path="url(#pbc4e44716b)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/> + + + + + + + @@ -397,14 +397,14 @@ L 674.254235 331.136348 L 735.825746 297.797204 L 797.397258 265.875417 L 858.968769 245.33417 -" clip-path="url(#pbc4e44716b)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> - - - - - - - +" clip-path="url(#p5ed38ffaf2)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/> + + + + + + + @@ -478,7 +478,7 @@ z +" clip-path="url(#p76a1f9e546)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> @@ -489,114 +489,109 @@ L 1347.84 434.52 +" clip-path="url(#p76a1f9e546)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - $0.005 + $0.003 +" clip-path="url(#p76a1f9e546)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - $0.010 + $0.006 +" clip-path="url(#p76a1f9e546)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - $0.015 + $0.009 +" clip-path="url(#p76a1f9e546)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/> - $0.020 + $0.012 - Estimated USD: compress + answer + Estimated USD per compaction - - - - - - - - - - - - + + + + + + + + + - $0.01429 + $0.009864 - $0.01643 + $0.010039 - - - - - - - - - + + + + + + + + + - $0.01025 + $0.005902 - $0.01239 + $0.006077 - Summary + Jev + one Qwen answer + Summary generation + Jev ranking - Full: $0.00699 + Jev adds $0.000175 once at N > 0 - C Total cost: one answer + C Compaction cost @@ -615,37 +610,37 @@ L 1320.489726 256.357166 Primary cohort: 4 task templates, 24 answers per method; 10 separate controls. Top 3–5 shorten 9/12 primary contexts per method. - C includes one summary, one Jev ranking at N > 0, and one Qwen answer (input + output). Full includes its Qwen answer fee. + C: Top 0 pays for summary generation; Top 1–5 add one complete Jev ranking. Answering is excluded. Full is shown in A/B only. Uncached list-price estimate (2026-10-16): Qwen $1.65 input / $4.951 output per M tokens; Jev $0.042 input / $0 output. Not an invoice. - + - + Generic summary S - + - + Hermes small-budget H - + + - + - + diff --git a/docs/zh/context_compaction.md b/docs/zh/context_compaction.md index 921ff014..088101e9 100644 --- a/docs/zh/context_compaction.md +++ b/docs/zh/context_compaction.md @@ -114,7 +114,7 @@ ReMe 不会自动生成摘要或写入长期记忆。 摘要可以缩短历史，但可能丢失后续回答需要的精确标识符、配置值或日志细节。本实验考察：**在摘要保持不变的基础上， 追加 Jev 选出的少量原文，能否补回这些信息？这一增益能否同时出现在普通摘要和 Hermes 摘要中？** -同时报告回答输入量和总费用，观察信息保留、上下文长度与开销之间的取舍。 +同时报告回答输入量和压缩费用，观察信息保留、上下文长度与开销之间的取舍。 ### 实验设置 @@ -133,15 +133,17 @@ Jev 先依据原任务给全部证据块打分，再选择 Top N；N 只影响 ### 读图与结论 -![Answer quality, downstream input and total cost across two summary strategies](../figure/compaction-topn-results.svg) +![Answer quality, downstream input and compaction cost across two summary strategies](../figure/compaction-topn-results.svg) - **A：两种摘要都能从追加原文中获益。** Full 的字段正确率为 100%；Top 3 将 S 从 **82.0% 提升至 99.5%**， 将 H 从 **79.4% 提升至 97.9%**。这支持“摘要 + 关键原文”在本组任务中补充摘要遗漏的信息，尚不能证明通用优势或与 Full 统计等价。 - **B：以更多上下文换取更高的信息保留率。** Top 3 的平均回答输入为 S **3241**、H **3193 tokens**， 相比 Full 的 **4014 tokens** 分别减少 **19.3%／20.4%**，但都高于各自纯摘要。Top 4～5 未继续提分，更多原文不保证效果更好。 -- **C：按一次压缩后回答计算，当前没有总费用节省。** 总费用包含**摘要生成 + 一次 Jev 排名 + 一次 Qwen 回答的输入／输出**。 - Top 3 的 S／H 费用约 **$0.015738／$0.011697**，Full 为 **$0.006992**。Jev 排名费在 Top 1～5 相同， - 曲线的增长来自 Qwen 读取更多原文。费用按 2026-10-16 的公开 USD 标价估算，未计缓存折扣、免费额度或账户优惠；不是实际账单。 +- **C：只比较压缩本身的费用。** Top 0 只计 Qwen 生成摘要的费用：S／H 约 **$0.009864／$0.005902**； + Top 1～5 均为“摘要 + 一次完整 Jev 排名”，约 **$0.010039／$0.006077**。Jev 额外增加约 **$0.000175**， + 即摘要费的 **1.8%／3.0%**。Jev 先给全部证据打分，再取 Top N，因此 Top 1～5 的压缩费用相同，曲线呈平台。 + **C 不计压缩后的回答费用，也不展示 Full；Full 仅在 A/B 作为效果和输入量参考。** + 费用按 2026-10-16 的公开 USD 标价估算，未计缓存折扣、免费额度或账户优惠；不代表实际账单或端到端费用节省。 **结果边界：** 主结论来自 4 个合成任务模板；10 个短任务对照不混入主图。纯 H 的 42 个上下文均短于 Full， 但 Top 3～5 每种方法仅有 **9/12 个主任务上下文**实际缩短，全部样例均保留评分。严格字段评分包含格式要求，语义相同但格式不同也可能失分。