diff --git a/benchmark/compaction/README.md b/benchmark/compaction/README.md index fa1a43ce..83df4579 100644 --- a/benchmark/compaction/README.md +++ b/benchmark/compaction/README.md @@ -71,7 +71,3 @@ See [the exact simulation settings and preflight disclosures](./STUDY.md#hermes- For the recorded format amendment, native summaries used the pre-amendment source files: pass `--native-evaluation /path/to/pre-amendment/evaluation` when extending that run. The extension verifies its source hash and permits only the held-out `exam` field to differ; the summary generator never receives that field. - -`costs.json` reports standalone summary/ranking input and output overhead separately from answer usage, including -recorded retries. Panel C converts summary + Jev compaction usage to estimated USD with the dated rates in -[pricing.json](./pricing.json); cache discounts, downstream answering and trajectory collection are excluded. Full appears in A/B only. See [the formula](./SETTINGS.md#scoring-and-costs). diff --git a/benchmark/compaction/RESULTS.md b/benchmark/compaction/RESULTS.md index ab596748..a52596d1 100644 --- a/benchmark/compaction/RESULTS.md +++ b/benchmark/compaction/RESULTS.md @@ -8,57 +8,32 @@ each summary remains byte-identical across its six conditions. ## Four primary tasks -| Method | Fact score | Answer input | Summary in / out | Jev in / out | Compaction USD | -| --- | ---: | ---: | ---: | ---: | ---: | -| Full context | 100.0% | 4,014 | 0 / 0 | 0 / 0 | — | -| S | 82.0% | 2,480 | 5,379 / 200 | 0 / 0 | $0.009864 | -| S + Jev Top 1 | 83.0% | 2,753 | 5,379 / 200 | 4,162 / 344 | $0.010039 | -| S + Jev Top 2 | 94.5% | 2,974 | 5,379 / 200 | 4,162 / 344 | $0.010039 | -| S + Jev Top 3 | 99.5% | 3,241 | 5,379 / 200 | 4,162 / 344 | $0.010039 | -| S + Jev Top 4 | 98.1% | 3,496 | 5,379 / 200 | 4,162 / 344 | $0.010039 | -| S + Jev Top 5 | 97.4% | 3,660 | 5,379 / 200 | 4,162 / 344 | $0.010039 | -| H (Hermes, small budget) | 79.4% | 2,432 | 3,120 / 152 | 0 / 0 | $0.005902 | -| H + Jev Top 1 | 85.3% | 2,705 | 3,120 / 152 | 4,162 / 344 | $0.006077 | -| H + Jev Top 2 | 92.9% | 2,927 | 3,120 / 152 | 4,162 / 344 | $0.006077 | -| H + Jev Top 3 | 97.9% | 3,193 | 3,120 / 152 | 4,162 / 344 | $0.006077 | -| H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 | $0.006077 | -| H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 | $0.006077 | +| Method | Fact score | Qwen answer input tokens | +| --- | ---: | ---: | +| Full context | 100.0% | 4,014 | +| S | 82.0% | 2,480 | +| S + Jev Top 1 | 83.0% | 2,753 | +| S + Jev Top 2 | 94.5% | 2,974 | +| S + Jev Top 3 | 99.5% | 3,241 | +| S + Jev Top 4 | 98.1% | 3,496 | +| S + Jev Top 5 | 97.4% | 3,660 | +| H (Hermes, small budget) | 79.4% | 2,432 | +| H + Jev Top 1 | 85.3% | 2,705 | +| H + Jev Top 2 | 92.9% | 2,927 | +| H + Jev Top 3 | 97.9% | 3,193 | +| H + Jev Top 4 | 96.9% | 3,448 | +| H + Jev Top 5 | 96.9% | 3,613 | -![Quality, downstream input and compaction fees by Top N](../../docs/figure/compaction-topn-results.svg) +![Answer quality and downstream input across two summary strategies](../../docs/figure/compaction-topn-results.svg) Both summary mechanisms benefit on this panel: S rises from **82.0% to 99.5%** at Top 3; H rises from **79.4% to 97.9%**. Mean downstream inputs at Top 3 are **3,241 / 3,193**, versus Full's **4,014**: reductions of **19.3% / 20.4%**. Top 4–5 do not improve these point estimates. This is an exploratory result on four synthetic task templates, not a general optimal-N claim, equivalence test or proof that Jev beats other equal-budget selectors. -Panel C compares **compaction fees only: summary generation + Jev ranking**, with Full omitted. -Top 0 costs **$0.009864 for S / $0.005902 for H**. Top 1–5 each cost **$0.010039 / $0.006077**. -Jev adds **$0.000175** once per compaction, about **1.8% / 3.0%** above S/H summary generation respectively. -The plateau is expected: Jev scores all available evidence once, then different N values retain different score prefixes. - -## Token and cost accounting - -Token columns are mean actual provider usage. Answer input remains a separate context-length metric, including the -original system/task, compressed evidence and held-out question. **Compaction USD** includes one summary generation -and one complete Jev ranking for N > 0, including recorded retries. **Downstream answering, trajectory collection and -calibration are excluded.** Full does not perform compaction and has no fee entry in this comparison (shown as “—”); -it remains the reference for answer quality and input length in A/B. No zero-total-cost claim is made for Full. -The two scoring repeats do not amortize compaction fees. - -Prices checked **2026-10-16**, per million tokens: - -| Model | Input USD | Output USD | Source | -| --- | ---: | ---: | --- | -| Qwen `qwen3.8-max`, China (Beijing) | $1.65 | $4.951 | [Alibaba Cloud](https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max) | -| Jev `jev-1.13.0`, direct TypeSafe | $0.042 | $0 | [TypeSafe](https://docs.typesafe.ai/models) | - -`compaction_usd = (summary_input × 1.65 + summary_output × 4.951 + jev_input × 0.042) / 1,000,000`. -[pricing.json](./pricing.json) freezes the rates, sources and scope. These are **uncached public-list-price estimates**: -all summary prompt tokens are priced at the regular input rate, even where API usage reports cache hits. Free quotas, -discounts, credits, taxes and account billing differences are excluded. This measures the incremental cost of adding -Jev to a summary, not end-to-end monetary savings or invoice reconciliation. -One shared ranking serves Top 1–5 and both mechanisms in the experiment; each standalone fused method includes its full -cost, neither multiplied by N nor divided across sweep conditions. +Answer input is actual downstream Qwen API usage, including the original system/task, compressed history and held-out +question. It does not include summary-generation or Jev-ranking requests. The figure reports only exact-field score +and downstream context length. ## Setup and boundaries @@ -85,21 +60,21 @@ the other 38 comprise 9 primary and 29 control contexts. All remain scored. Aver
Ten short/query-only controls -| Method | Fact score | Answer input | Summary in / out | Jev in / out | Compaction USD | -| --- | ---: | ---: | ---: | ---: | ---: | -| Full context | 100.0% | 2,727 | 0 / 0 | 0 / 0 | — | -| S | 67.1% | 2,315 | 1,099 / 74 | 0 / 0 | $0.002179 | -| S + Jev Top 1 | 84.7% | 2,493 | 1,099 / 74 | 929 / 50 | $0.002218 | -| S + Jev Top 2 | 87.5% | 2,573 | 1,099 / 74 | 929 / 50 | $0.002218 | -| S + Jev Top 3 | 92.9% | 2,687 | 1,099 / 74 | 929 / 50 | $0.002218 | -| S + Jev Top 4 | 92.9% | 2,722 | 1,099 / 74 | 929 / 50 | $0.002218 | -| S + Jev Top 5 | 92.5% | 2,722 | 1,099 / 74 | 929 / 50 | $0.002218 | -| H (Hermes, small budget) | 73.2% | 2,308 | 1,944 / 69 | 0 / 0 | $0.003550 | -| H + Jev Top 1 | 80.1% | 2,486 | 1,944 / 69 | 929 / 50 | $0.003589 | -| H + Jev Top 2 | 92.5% | 2,566 | 1,944 / 69 | 929 / 50 | $0.003589 | -| H + Jev Top 3 | 99.2% | 2,680 | 1,944 / 69 | 929 / 50 | $0.003589 | -| H + Jev Top 4 | 99.2% | 2,715 | 1,944 / 69 | 929 / 50 | $0.003589 | -| H + Jev Top 5 | 98.8% | 2,715 | 1,944 / 69 | 929 / 50 | $0.003589 | +| Method | Fact score | Qwen answer input tokens | +| --- | ---: | ---: | +| Full context | 100.0% | 2,727 | +| S | 67.1% | 2,315 | +| S + Jev Top 1 | 84.7% | 2,493 | +| S + Jev Top 2 | 87.5% | 2,573 | +| S + Jev Top 3 | 92.9% | 2,687 | +| S + Jev Top 4 | 92.9% | 2,722 | +| S + Jev Top 5 | 92.5% | 2,722 | +| H (Hermes, small budget) | 73.2% | 2,308 | +| H + Jev Top 1 | 80.1% | 2,486 | +| H + Jev Top 2 | 92.5% | 2,566 | +| H + Jev Top 3 | 99.2% | 2,680 | +| H + Jev Top 4 | 99.2% | 2,715 | +| H + Jev Top 5 | 98.8% | 2,715 | These controls are not ten additional long-context tasks. Their small source-unit counts can make Top N saturate. @@ -116,7 +91,5 @@ Artifacts under `/Users/yuli/workspace/trajectory-runs/` remain outside Git: `hermes-fixture14-20261016/` (corpus), `compaction-fixture14-20261016-v2/` (S/rankings), `compaction-fixture14-hermes-adapted-20261016/` (H), `compaction-fixture14-final-20261016/` (original eight arms), and `compaction-fixture14-dual-summary-20261016/` (current 13-arm results, usage and audit). -The extension used **1,170,946 input / 19,990 output Qwen tokens** for its 420 answers, with zero new Jev/summary calls. -Historical collection, calibration and complete monetary billing are not included in this incremental usage figure.
diff --git a/benchmark/compaction/SETTINGS.md b/benchmark/compaction/SETTINGS.md index 671df12e..b7814f7d 100644 --- a/benchmark/compaction/SETTINGS.md +++ b/benchmark/compaction/SETTINGS.md @@ -27,7 +27,6 @@ The Qwen alias is what the requests and captured answering responses identify; a was not supplied. Service-side updates and nondeterminism can change new-run results even at temperature zero. Use the same `LLM_BASE_URL` service for the closest rerun; this endpoint and API keys are supplied privately in `.env`. The endpoint was verified as the public China (Beijing) DashScope endpoint; API keys remain private. -The chart uses the published USD list-price schedule below rather than an account invoice. ## Runtime dependencies @@ -110,27 +109,14 @@ The retained pre-amendment payment `budget.json` says 4,459; its final question Every condition fits its recorded ceiling. All 42 H-only contexts shorten Full; at Top 3–5, each mechanism shortens only 9/12 primary contexts per method. All cases remain scored; mean savings do not imply every case saves tokens. -## Scoring and costs +## Scoring Each answer returns `{"facts": {...}}`. Field checks require exact recursive types/values, with only payment HTTP-code order ignored. Missing/malformed/failed planned conditions receive zero. Average fields within answers, answers within generations, generations within tasks, then tasks equally. Report primary/control cohorts separately. -Answer input includes system/task, compressed history and exam. Standalone-method overhead includes one summary and, -for N > 0, one whole Jev ranking, including recorded retries. Ranking cost is identical across N and both mechanisms; -do not multiply by N or divide by sweep size. Evaluation repeats do not amortize overhead. Report input/output tokens -by model. Compaction fees use [pricing.json](./pricing.json), checked 2026-10-16: Qwen `qwen3.8-max` China (Beijing) -input **$1.65/M**, output **$4.951/M** ([official rate](https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max)); -Jev `jev-1.13.0` input **$0.042/M**, output free ([official rate](https://docs.typesafe.ai/models)). - -`compaction_usd = (summary_input * 1.65 + summary_output * 4.951 + jev_input * 0.042) / 1e6`. -At Top 0 charge only summary generation; Top 1–5 each add one identical complete Jev ranking. Mean usage is averaged -across the same tasks/generations, and fees are computed from unrounded usage. Recorded summary/ranking retries are -included. **Downstream answering, collection and calibration are excluded.** Full is excluded from panel C and has -“—” in the cost table; it remains the quality/input reference in A/B. Scoring repeats do not amortize preprocessing. -These are uncached list-price estimates: summary cache hits are charged at the regular input rate for comparison. -No free quotas, promotions, taxes or account-specific adjustments are applied. Prices are a dated snapshot, not an -invoice. This scope measures summary + Jev compaction overhead, not total execution cost or monetary savings. +Answer input is the answering API's actual `prompt_tokens`, including system/task messages, compressed history and +the follow-up question. Summary-generation and Jev-ranking requests are separate and are not added to this metric. ## Reproduction levels and artifact availability diff --git a/benchmark/compaction/STUDY.md b/benchmark/compaction/STUDY.md index 502980dd..022084c7 100644 --- a/benchmark/compaction/STUDY.md +++ b/benchmark/compaction/STUDY.md @@ -58,7 +58,7 @@ no task is selected because a method wins. Coverage, pairing and collection fail - Summary allowance: 64–256 local Qwen tokens, equal to 20% of rendered evidence tokens clamped to those limits. The allowance is frozen from source length. - Before answering, calculate a per-task complete-input ceiling that fits the protected prompts, the frozen summary allowance, and the five largest whole units, plus a 64-token reserve and 32 extra framing tokens; the ceiling is enlarged before answers if needed to fit serialized S. This size-only ceiling permits the entire N sweep without changing S or truncating evidence; actual lengths are reported rather than forced equal. -- Report Full-to-fusion input reduction. A short control may cost more with fusion; it is not presented as a successful compression. +- Report Full-to-fusion input reduction. A short control may use more input tokens with fusion; it is not presented as a successful compression. - Save protocol, source hashes, code snapshots, requests/responses, nested selections and per-field grades outside the repository. No archive recovery. - The original payment-only table remains a development result; this new task panel is not an independent held-out confirmation because the payment task was already used. @@ -110,11 +110,3 @@ S/H use identical selected units per task/generation/N, restored to source order The extension keeps the existing per-task ceilings. Strict shrinkage versus Full is reported separately, without filtering or removing conditions: Top 3–5 each shorten 9/12 primary contexts in both mechanisms. The H fusion arms shorten 172/210 contexts (51/60 primary, 121/150 controls). Summary-only H still shortens all 42/42. - -For compaction costs, each independent method includes one summary generation and one complete Jev ranking when -N > 0. Downstream answering is excluded. Recorded retry usage is included. One ranking is shared across the Top-N sweep and -mechanisms in the actual experiment; its standalone-method cost is neither multiplied by N nor divided by sweep size. -The two answer repeats do not amortize compression overhead. Report provider input/output tokens separately. -Panel C estimates summary + Jev compaction fees from the dated public USD rates in [pricing.json](./pricing.json), -using uncached input pricing and excluding downstream answering and trajectory collection. Full appears in A/B only. [Settings](./SETTINGS.md#scoring-and-costs) define -the formula and exclusions; this is not an account invoice or a claim of total pipeline savings. diff --git a/benchmark/compaction/pricing.json b/benchmark/compaction/pricing.json deleted file mode 100644 index 1f588509..00000000 --- a/benchmark/compaction/pricing.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "checked_on": "2026-10-16", - "currency": "USD", - "unit": "per_1000000_tokens", - "qwen3.8-max": { - "input": 1.65, - "output": 4.951, - "region": "China (Beijing)", - "source": "https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max" - }, - "jev-1.13.0": { - "input": 0.042, - "output": 0.0, - "source": "https://docs.typesafe.ai/models" - }, - "scope": "Compaction only: one summary generation plus one complete Jev ranking when N > 0; recorded retries included; downstream answering and trajectory collection excluded; Full omitted from the cost comparison", - "billing_basis": "Public list-price estimate; all input charged at uncached rate; no credits, promotions, taxes, exchange conversion or invoice reconciliation" -} diff --git a/benchmark/compaction/suite_fusion.py b/benchmark/compaction/suite_fusion.py index 271b5290..68f222b7 100644 --- a/benchmark/compaction/suite_fusion.py +++ b/benchmark/compaction/suite_fusion.py @@ -7,7 +7,6 @@ import hashlib import json from pathlib import Path import shutil -import statistics from dotenv import dotenv_values from openai import AsyncOpenAI @@ -19,58 +18,6 @@ from benchmark.compaction.suite_native import validate_base from benchmark.compaction.suite_run import ARMS, load_cases, report -def costs(args, rows, cases): - """Standalone one-answer costs, including one summary/ranking and all recorded retries.""" - result = {} - for cohort in ("primary", "controls"): - ids = {k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")} - table = [] - for arm in ARMS + ["hermes_summary"] + [f"hermes_top{n}" for n in range(1, 6)]: - entries = [] - for row in rows: - if row["case"] not in ids or row["arm"] != arm: - continue - key, i = row["case"], row["repeat"] - summary = [] - if arm.startswith("hermes"): - paths = (args.native / key / "generation").glob(f"c{i}.a*.summary.response.json") - summary = [json.loads(p.read_text())["usage"] for p in paths] - elif arm != "full": - paths = (args.evaluation / key).glob(f"paired.c{i}.summary*.response.json") - summary = [json.loads(p.read_text())["usage"] for p in paths] - decision = [] - if arm.startswith(("fusion", "hermes_top")): - paths = (args.evaluation / key).glob(f"paired.c{i}.decision*.response.json") - decision = [json.loads(p.read_text())["usage"] for p in paths] - entries.append( - { - "summary_input": sum(u["prompt_tokens"] for u in summary), - "summary_output": sum(u["completion_tokens"] for u in summary), - "jev_input": sum(u["input_tokens"] for u in decision), - "jev_output": sum(u["output_tokens"] for u in decision), - "answer_input": ( - statistics.mean(t["usage"]["prompt_tokens"] for t in row["trials"]) - if row["trials"] - else None - ), - "answer_output": ( - statistics.mean(t["usage"]["completion_tokens"] for t in row["trials"]) - if row["trials"] - else None - ), - }, - ) - table.append( - { - "arm": arm, - **{k: statistics.mean(e[k] for e in entries if e[k] is not None) for k in entries[0]}, - }, - ) - result[cohort] = table - write(args.output / "costs.json", result) - return result - - def prepare(args, cases): """Verify artifacts and freeze all paired inputs before any new answer call.""" original = validate_base(args.evaluation, args.corpus, cases) @@ -216,7 +163,6 @@ async def run(args): finally: await client.close() load_cases(args.corpus) - costs(args, rows, cases) for path in output.rglob("*.json"): if values["LLM_API_KEY"].encode() in path.read_bytes(): raise ValueError("Configured API key in artifacts") diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md index 375d46ee..4a94bccf 100644 --- a/docs/en/context_compaction.md +++ b/docs/en/context_compaction.md @@ -115,7 +115,7 @@ REME_RUN_DECISION_INTEGRATION=1 pytest tests/integration/test_decision_compactio Only model requests run externally; fixture tool calls are not executed. These cases test behavior, not general task success. Evaluate real trajectories against recency and rule-based previews, measuring recovery, continuation quality, -latency and total cost alongside token reduction. ReMe does not automatically summarize or write persistent memory. +latency alongside token reduction. ReMe does not automatically summarize or write persistent memory. ## Jev across two summary strategies @@ -123,8 +123,8 @@ latency and total cost alongside token reduction. ReMe does not automatically su Summaries shorten history but can omit exact identifiers, configuration values or log details needed for a later answer. This experiment asks: **can a few Jev-selected original evidence blocks recover information missing from a fixed summary, -and does that benefit occur with both a generic summary and a Hermes summary?** Answer input and total fees show the -tradeoff between retained information, context length and compaction cost. +and does that benefit occur with both a generic summary and a Hermes summary?** Exact-field score and Qwen answer input show the +tradeoff between retained information and context length. ### Experimental setup @@ -143,7 +143,7 @@ This is a gold-answer **information-retention evaluation**, without resuming Her ### Reading the figure and conclusions -![Answer quality, downstream input and compaction cost across two summary strategies](../figure/compaction-topn-results.svg) +![Answer quality and downstream input across two summary strategies](../figure/compaction-topn-results.svg) - **A: selected originals help both summary mechanisms on this panel.** Full scores 100%; Top 3 improves S from **82.0% to 99.5%**, and H from **79.4% to 97.9%**. This supports supplementing these summaries with original evidence; @@ -151,18 +151,11 @@ This is a gold-answer **information-retention evaluation**, without resuming Her - **B: higher retention consumes more context than summary alone.** Top 3 uses **3,241 / 3,193** mean answer-input tokens for S/H, versus Full's **4,014**, reductions of **19.3% / 20.4%**. Both exceed their summary-only input. Top 4–5 do not improve the scores: more original text does not guarantee better answers. -- **C: compare compaction fees only.** Top 0 pays for Qwen summary generation: **$0.009864 / $0.005902** for S/H. - Top 1–5 each pay for summary generation plus one complete Jev ranking: **$0.010039 / $0.006077**. - Jev adds **$0.000175**, about **1.8% / 3.0%** of S/H summary fees. The curve plateaus because Jev scores all evidence - once before selecting Top N. **C excludes downstream answering and omits Full; Full remains the reference in A/B.** - Fees use public USD list prices dated 2026-10-16, excluding cache discounts, free quotas and account offers. - This is neither an invoice nor evidence of end-to-end savings. **Scope:** the primary result covers four synthetic task templates; ten controls are separate. All 42 H-only contexts shorten Full, but Top 3–5 shorten only **9/12 primary contexts per method**, with every case still scored. Strict field scoring includes format compliance; semantically equivalent wording can still lose credit. The result shows an information/length tradeoff. Without equal-length longer-summary or alternative-selector controls, -the entire gain cannot be attributed to Jev ranking itself. Future-turn cost amortization and agent continuation success -remain unmeasured. +the entire gain cannot be attributed to Jev ranking itself. Agent continuation success remains unmeasured. [Full settings and reproduction](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/SETTINGS.md) · [Queries, answers and scoring](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/STUDY.md) · [All 13 methods and short-task controls](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/RESULTS.md) diff --git a/docs/figure/compaction-topn-results.svg b/docs/figure/compaction-topn-results.svg index 9e3750f6..0e30e9b4 100644 --- a/docs/figure/compaction-topn-results.svg +++ b/docs/figure/compaction-topn-results.svg @@ -1,12 +1,12 @@ - + - 2026-10-16T18:18:39.704996 + 2026-10-16T21:37:27.862626 image/svg+xml @@ -22,164 +22,164 @@ - - + - 0 + 0 - 1 + 1 - 2 + 2 - 3 + 3 - 4 + 4 - 5 + 5 - Jev Top N (0 = summary only) + Jev Top N (0 = summary only) - + - 70 + 70 - + - 75 + 75 - + - 80 + 80 - + - 85 + 85 - + - 90 + 90 - + - 95 + 95 - + - 100 + 100 - Exact-field score (%) + Exact-field score (%) - - + - - - - - - - - + + + + + + + - 82.0% + 82.0% - 99.5% + 99.5% - 97.4% + 97.4% - + - - - - - - - - + + + + + + + - 79.4% + 79.4% - 97.9% + 97.9% - 96.9% + 96.9% - Full: 100% + Full: 100% - A Answer quality + A Answer quality - - + - 0 + 0 - 1 + 1 - 2 + 2 - 3 + 3 - 4 + 4 - 5 + 5 - Jev Top N (0 = summary only) + Jev Top N (0 = summary only) - + - 2500 + 2500 - + - 3000 + 3000 - + - 3500 + 3500 - + - 4000 + 4000 - Mean answer input tokens + Mean answer input tokens - - - - - - - - - + + + + + + + + - 2,480 + 2,480 - 3,241 + 3,241 - 3,660 + 3,660 - - - - - - - - + + + + + + + + - 2,432 + 2,432 - 3,193 + 3,193 - 3,613 + 3,613 - Full: 4,014 + Full: 4,014 - B Downstream context + B Downstream context - - - - - - - - - 0 - - - - - - 1 - - - - - - 2 - - - - - - 3 - - - - - - 4 - - - - - - 5 - - - - Jev Top N (0 = summary only) - - - - - - - - - - $0.000 - - - - - - - - - $0.003 - - - - - - - - - $0.006 - - - - - - - - - $0.009 - - - - - - - - - $0.012 - - - - Estimated USD per compaction - - - - - - - - - - - - - - - - - - $0.009864 - - - $0.010039 - - - - - - - - - - - - - - $0.005902 - - - $0.006077 - - - Summary generation + Jev ranking - - - Jev adds $0.000175 once at N > 0 - - - C Compaction cost - + + Jev across two summary strategies - - Jev across two summary strategies + + Same source trajectories. Frozen summaries. Identical Jev selections. - - Same source trajectories. Frozen summaries. Identical Jev selections. + + At Top 3 - - At Top 3 + + S: 99.5% | 3,241 input tokens H: 97.9% | 3,193 input tokens - - S: 99.5% | 3,241 input tokens H: 97.9% | 3,193 input tokens + + Primary cohort: 4 task templates, 24 answers per method. Ten short/query-only controls are reported separately. - - Primary cohort: 4 task templates, 24 answers per method; 10 separate controls. Top 3–5 shorten 9/12 primary contexts per method. + + Top 3–5 shorten 9/12 primary contexts per method; all cases remain scored. - - C: Top 0 pays for summary generation; Top 1–5 add one complete Jev ranking. Answering is excluded. Full is shown in A/B only. - - - Uncached list-price estimate (2026-10-16): Qwen $1.65 input / $4.951 output per M tokens; Jev $0.042 input / $0 output. Not an invoice. + + Frozen summaries and identical Jev selections; direct Qwen evaluation, no agent continuation. - - + - + - - Generic summary S + + Generic summary S - - + - + - - Hermes small-budget H + + Hermes small-budget H - - + - - Full context + + Full context - - + + - - - - - + + diff --git a/docs/zh/context_compaction.md b/docs/zh/context_compaction.md index 5d443bce..fb6a67e9 100644 --- a/docs/zh/context_compaction.md +++ b/docs/zh/context_compaction.md @@ -90,7 +90,7 @@ response = await step( | `evidence_chars` | `2000` | 每个输入、结果预览或近期文本的字符上限 | | `preview_chars` | `300` | 建议结果保留的头尾片段字符上限，不含恢复标记 | | `max_state_tokens` | `25000` | 决策状态的估算上限 | -| `max_request_tokens` | `30000` | 状态和问题的估算上限，含 256 token 开销预留 | +| `max_request_tokens` | `30000` | 状态和问题的估算上限，含 256 token 额外预留 | | `max_output_tokens` | `null` | 可选的宿主计数预算 | | `max_concurrency` | `4` | 最大并发请求数 | | `total_timeout_seconds` | `30` | 包含排队的总体请求超时 | @@ -105,7 +105,7 @@ REME_RUN_DECISION_INTEGRATION=1 pytest tests/integration/test_decision_compactio ``` 只有模型请求调用外部服务，历史中的工具不会执行。这些用例验证行为，不证明通用任务成功率。 -真实 trajectory 评测应对比近期保留和纯规则预览，同时测量恢复次数、继续执行的质量、时延、总成本和 token 节省。 +真实 trajectory 评测应对比近期保留和纯规则预览，同时测量恢复次数、继续执行的质量、时延和 token 节省。 ReMe 不会自动生成摘要或写入长期记忆。 ## 两种摘要机制下的 Jev 增益 @@ -114,7 +114,7 @@ ReMe 不会自动生成摘要或写入长期记忆。 摘要可以缩短历史，但可能丢失后续回答需要的精确标识符、配置值或日志细节。本实验考察：**在摘要保持不变的基础上， 追加 Jev 选出的少量原文，能否补回这些信息？这一增益能否同时出现在普通摘要和 Hermes 摘要中？** -同时报告回答输入量和压缩费用，观察信息保留、上下文长度与开销之间的取舍。 +同时报告字段正确率和 Qwen 回答输入量，观察信息保留与上下文长度之间的取舍。 ### 实验设置 @@ -133,22 +133,17 @@ Jev 先依据原任务给全部证据块打分，再选择 Top N；N 只影响 ### 读图与结论 -![Answer quality, downstream input and compaction cost across two summary strategies](../figure/compaction-topn-results.svg) +![Answer quality and downstream input across two summary strategies](../figure/compaction-topn-results.svg) - **A：两种摘要都能从追加原文中获益。** Full 的字段正确率为 100%；Top 3 将 S 从 **82.0% 提升至 99.5%**， 将 H 从 **79.4% 提升至 97.9%**。这支持“摘要 + 关键原文”在本组任务中补充摘要遗漏的信息，尚不能证明通用优势或与 Full 统计等价。 - **B：以更多上下文换取更高的信息保留率。** Top 3 的平均回答输入为 S **3241**、H **3193 tokens**， 相比 Full 的 **4014 tokens** 分别减少 **19.3%／20.4%**，但都高于各自纯摘要。Top 4～5 未继续提分，更多原文不保证效果更好。 -- **C：只比较压缩本身的费用。** Top 0 只计 Qwen 生成摘要的费用：S／H 约 **$0.009864／$0.005902**； - Top 1～5 均为“摘要 + 一次完整 Jev 排名”，约 **$0.010039／$0.006077**。Jev 额外增加约 **$0.000175**， - 即摘要费的 **1.8%／3.0%**。Jev 先给全部证据打分，再取 Top N，因此 Top 1～5 的压缩费用相同，曲线呈平台。 - **C 不计压缩后的回答费用，也不展示 Full；Full 仅在 A/B 作为效果和输入量参考。** - 费用按 2026-10-16 的公开 USD 标价估算，未计缓存折扣、免费额度或账户优惠；不代表实际账单或端到端费用节省。 **结果边界：** 主结论来自 4 个合成任务模板；10 个短任务对照不混入主图。纯 H 的 42 个上下文均短于 Full， 但 Top 3～5 每种方法仅有 **9/12 个主任务上下文**实际缩短，全部样例均保留评分。严格字段评分包含格式要求，语义相同但格式不同也可能失分。 当前结果展示了信息保留与长度的取舍；尚未加入等长度的更长摘要或其他选择器对照，不能将全部增益归因于 Jev 排序本身。 -多轮摊销后的费用收益和 agent 继续执行的成功率均未验证。 +agent 继续执行的成功率尚未验证。 [完整实验设置与复现步骤](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/SETTINGS.md) · [query、answer 与评分规则](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/STUDY.md) · [13 组完整结果及短任务对照](https://github.com/jinliyl/ReMe/blob/7686be7baaddd4094b454c30c8fca61f5429df98/benchmark/compaction/RESULTS.md)