diff --git a/benchmark/compaction/README.md b/benchmark/compaction/README.md
index 02883cf8..eb269d49 100644
--- a/benchmark/compaction/README.md
+++ b/benchmark/compaction/README.md
@@ -4,7 +4,7 @@ A **14-query fixture panel** with predefined answers and rule scoring: four prim
controls. See [queries, answers and frozen settings](./STUDY.md) and [measured results](./RESULTS.md).
Hermes collects the source through real `read_file` calls. ReMe's experiment then compares Full context, a frozen Qwen
-summary S, S + Jev Top 1–5, and a **Hermes small-window summary adapter**. Scored answering calls Qwen directly;
+summary S and forced **Hermes small-window summary H**, each with Jev Top 0–5. Scored answering calls Qwen directly;
there is no resumed Hermes tool loop or archive recovery. Production `compact_context` is separate.
## Run
@@ -39,15 +39,24 @@ python -m benchmark.compaction.suite_native --live \
--hermes-python /path/to/hermes-environment/bin/python \
--tokenizer /path/to/cached/qwen/tokenizer.json \
--output /path/outside/repository/combined-results
+
+# Reuse all frozen summaries/rankings and 672 answers; add only H + Top 1–5.
+python -m benchmark.compaction.suite_fusion --live \
+ --corpus /path/outside/repository/corpus/corpus.json \
+ --evaluation /path/outside/repository/evaluation \
+ --combined /path/outside/repository/combined-results \
+ --native /path/outside/repository/native-generations \
+ --tokenizer /path/to/cached/qwen/tokenizer.json \
+ --output /path/outside/repository/dual-summary-results
```
The task definitions use local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b` (MIT). The recorded study uses
-`qwen3.8-max`, three summary/ranking generations and two answers per context: **336 contexts, 672 answers**. Provider
-requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay
+`qwen3.8-max`, three summary/ranking generations and two answers per context: **546 contexts, 1,092 answers across thirteen methods**.
+Provider requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay
outside git; existing runs are never overwritten.
-S remains byte-identical across its five fusion variants. Top N means a strict descending-score prefix, without a
-probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer
+S and H each remain byte-identical across their five fusion variants; both use the same Jev selections.
+Top N means a strict descending-score prefix, without a probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer
than N units, all available units are retained and `actual_n` records the count. The original payment source has ten
units (maximum Top 10); this study sweeps Top 1–5 with task-specific unit counts.
@@ -55,3 +64,10 @@ The native baseline invokes Hermes' actual summary generator with an explicit sm
S-matched body budget, final word constraint, native provenance validation, and body-only downstream framing. It bypasses
normal triggers/head-tail selection and **must actually shorten input**; it is not the unchanged automatic Hermes policy.
See [the exact simulation settings and preflight disclosures](./STUDY.md#added-baseline-hermes-small-window-summary-adapter).
+
+For the recorded format amendment, native summaries used the pre-amendment source files: pass
+`--native-evaluation /path/to/pre-amendment/evaluation` when extending that run. The extension verifies its source hash
+and permits only the held-out `exam` field to differ; the summary generator never receives that field.
+
+`costs.json` reports standalone summary/ranking input and output overhead separately from answer usage, including
+recorded retries. Token counts do not apply model prices or cache discounts and are not monetary savings.
diff --git a/benchmark/compaction/RESULTS.md b/benchmark/compaction/RESULTS.md
index ab1a404b..942e7762 100644
--- a/benchmark/compaction/RESULTS.md
+++ b/benchmark/compaction/RESULTS.md
@@ -1,97 +1,105 @@
-# Fixed-summary compaction on fourteen fixture tasks
+# Jev across two summary strategies
-2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · **14 distinct queries with predefined answers**.
-Four source-length-qualified tasks form the primary cohort; ten short/query-only controls are reported separately.
-Cohorts were fixed before collection, with no accuracy-based task filtering.
+2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0`.
-## Primary cohort: four tasks
+One **13-method experiment**: Full context, generic summary **S + Top 0–5**, and forced Hermes small-window summary
+**H + Top 0–5**. Top 0 means summary only. Both curves share the same original evidence and frozen Jev rankings;
+each summary remains byte-identical across its six conditions.
-| Method | Mean input tokens | Fact score | Valid contexts |
-| --- | ---: | ---: | --- |
-| Full context | 4,014 | 100.0% | 12/12 |
-| S | 2,480 | 82.0% | 12/12 |
-| Hermes small-window adapter | 2,432 | 79.4% | 12/12 |
-| S + Jev Top 1 | 2,753 | 83.0% | 12/12 |
-| S + Jev Top 2 | 2,974 | 94.5% | 12/12 |
-| S + Jev Top 3 | 3,241 | 99.5% | 12/12 |
-| S + Jev Top 4 | 3,496 | 98.1% | 12/12 |
-| S + Jev Top 5 | 3,660 | 97.4% | 12/12 |
+## Four primary tasks
-Input tokens are actual downstream API usage, including the original system/task and follow-up. Fact score averages
-strict JSON field checks within each answer, then generations and tasks equally; invalid conditions score zero.
-Full context reaches 100%; S + Top 3 reaches **99.5% with 19.3% less downstream input** (3,241 versus 4,014 tokens).
-Top 4 and Top 5 do not improve this point estimate. No equivalence test or general optimal-N claim is made.
+| Method | Fact score | Answer input | Summary in / out | Jev in / out |
+| --- | ---: | ---: | ---: | ---: |
+| Full context | 100.0% | 4,014 | 0 / 0 | 0 / 0 |
+| S | 82.0% | 2,480 | 5,379 / 200 | 0 / 0 |
+| S + Jev Top 1 | 83.0% | 2,753 | 5,379 / 200 | 4,162 / 344 |
+| S + Jev Top 2 | 94.5% | 2,974 | 5,379 / 200 | 4,162 / 344 |
+| S + Jev Top 3 | 99.5% | 3,241 | 5,379 / 200 | 4,162 / 344 |
+| S + Jev Top 4 | 98.1% | 3,496 | 5,379 / 200 | 4,162 / 344 |
+| S + Jev Top 5 | 97.4% | 3,660 | 5,379 / 200 | 4,162 / 344 |
+| H (Hermes small-window) | 79.4% | 2,432 | 3,120 / 152 | 0 / 0 |
+| H + Jev Top 1 | 85.3% | 2,705 | 3,120 / 152 | 4,162 / 344 |
+| H + Jev Top 2 | 92.9% | 2,927 | 3,120 / 152 | 4,162 / 344 |
+| H + Jev Top 3 | 97.9% | 3,193 | 3,120 / 152 | 4,162 / 344 |
+| H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 |
+| H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 |
-
+
-## What was tested
+Both summary mechanisms benefit on this panel: S rises from **82.0% to 99.5%** at Top 3; H rises from **79.4% to 97.9%**.
+Mean downstream inputs at Top 3 are **3,241 / 3,193**, versus Full's **4,014**: reductions of **19.3% / 20.4%**.
+Top 4–5 do not improve these point estimates. This is an exploratory result on four synthetic task templates,
+not a general optimal-N claim, equivalence test or proof that Jev beats other equal-budget selectors.
-The [full query/answer table and protocol](./STUDY.md) define the oracle before collection. The four primary tasks are
-payment retry investigation, checkout 504 analysis, server-error localization and a build-output token lookup.
-The ten controls cover shorter configuration, code, dependency, notes, version and file-generation questions.
-All fourteen sources are synthetic fixtures: one existing payment trajectory plus thirteen genuine Hermes executions
-over explicitly read-only adaptations of upstream tasks. Long-file views are deterministic original excerpts, not
-added noise. These are not the maintainers' private production transcripts.
+## Token and cost accounting
-Each task has three frozen summary/ranking pairs and two answers per method: **336 contexts and 672 answers** across
-eight methods. S summarizes every source unit once. Each S + Top N preserves that S byte-for-byte and adds complete
-original units, ranked without a threshold and restored to source order. Actual N saturates at the available unit count.
-The original payment source has ten units, so its maximum is Top 10; the sweep tests Top 1–5.
+All table values are mean provider tokens. Answer input includes the original system/task, summary/evidence and held-out
+question. Summary and Jev columns are **one-time overhead per standalone condition**, including recorded length retries.
+One frozen Jev ranking serves every N and either summary mechanism: Top 1–5 therefore have the same ranking cost;
+it is neither multiplied by N nor divided by the number of sweep conditions. Summary-only and Full incur no Jev cost.
+The two scoring repeats estimate one downstream answer; overhead is not amortized across those evaluation repeats.
-S's evidence allowance is 20% of rendered-evidence tokens, clamped to 64–256 local Qwen tokens. A per-task complete-input
-ceiling is calculated from protected prompts, the summary allowance, the five largest whole units and reserve, before
-scored answers. This permits the incremental sweep; **it is not an equal-length S-versus-fusion comparison or a claim
-that every Full input exceeds its ceiling**. All actual inputs are checked. Short cases may gain no token savings.
+These are token counts, **not currency**: model prices and cached-token discounts are not applied. Compression increases
+one-off provider work here; lower downstream input alone does not demonstrate lower total cost. Repeated future turns
+may amortize overhead, but that has not been measured.
-The added **Hermes small-window adapter** calls the real native generator, with its legacy template, an overridden
-64–256-token body budget and an appended total word limit. It retains native redaction/provenance checks and strips
-only the native control prefix before the common summary carrier. It bypasses automatic triggering and full-compressor
-head/tail selection: **this is a forced simulation, not the unchanged default Hermes compressor**. All **42/42** native
-contexts fit their body/input budgets and use fewer actual API input tokens than the corresponding Full context.
-No original-history fallback is accepted.
+## Setup and boundaries
-Hermes collects the trajectories; answering calls Qwen directly. Gold and follow-up questions never enter collection,
-summary or decision requests. The final collection report is excluded from every history. There is no tool continuation,
-retrieval, LLM judge or generated answer key. Exact string/type compliance is part of this metric; it does not by itself
-prove semantic information loss or task-completion differences. The native and S methods have the same maximum body
-allowance, but their actual lengths differ. Four primary templates are too few to establish general superiority.
+[Queries, fixed answers and scoring rules](./STUDY.md) · [Reproduction commands](./README.md).
+
+The panel contains **14 task templates**, one existing payment trajectory and thirteen real read-only Hermes executions
+over synthetic upstream fixtures. Four primary tasks were selected by source length before collection; ten short/query-only
+controls remain separate. Three summary/ranking pairs and two answers per condition give **546 contexts / 1,092 answers**.
+The extension reused all 42 H summaries, rankings and 672 previous answers; it added only 420 answering calls.
+No new summary or Jev calls were needed. All 1,092 grades were replayed against captured requests/responses.
+
+Scoring uses predefined, recursive type-strict JSON field rules, with no LLM judge. Questions/gold/final collection reports
+are excluded from summary and decision inputs. Answering calls Qwen directly, without Hermes tool continuation or recovery.
+S and H share a 64–256-token maximum summary-body allowance, but use different prompts and actual lengths.
+H calls Hermes' real legacy generator through an explicit budget/word-limit adapter; it is **not the default automatic
+compressor**. All 42 H-only contexts strictly shorten Full, without original-history fallback.
+
+Top N adds complete source units without truncation, saturating at the available count. The payment source has ten units
+(maximum Top 10); this experiment sweeps Top 1–5. Existing per-task input ceilings are preserved, not tuned after answers.
+The comparison is incremental, **not equal-length**. All conditions fit their ceilings, but only **9/12 primary contexts
+per method at Top 3–5** are shorter than their paired Full. Across the new H fusion arms, 172/210 contexts shorten Full;
+the other 38 comprise 9 primary and 29 control contexts. All remain scored. Average savings do not imply per-case savings.
Ten short/query-only controls
-| Method | Mean input tokens | Fact score | Valid contexts |
-| --- | ---: | ---: | --- |
-| Full context | 2,727 | 100.0% | 30/30 |
-| S | 2,315 | 67.1% | 30/30 |
-| Hermes small-window adapter | 2,308 | 73.2% | 30/30 |
-| S + Jev Top 1 | 2,493 | 84.7% | 30/30 |
-| S + Jev Top 2 | 2,573 | 87.5% | 30/30 |
-| S + Jev Top 3 | 2,687 | 92.9% | 30/30 |
-| S + Jev Top 4 | 2,722 | 92.9% | 30/30 |
-| S + Jev Top 5 | 2,722 | 92.5% | 30/30 |
+| Method | Fact score | Answer input | Summary in / out | Jev in / out |
+| --- | ---: | ---: | ---: | ---: |
+| Full context | 100.0% | 2,727 | 0 / 0 | 0 / 0 |
+| S | 67.1% | 2,315 | 1,099 / 74 | 0 / 0 |
+| S + Jev Top 1 | 84.7% | 2,493 | 1,099 / 74 | 929 / 50 |
+| S + Jev Top 2 | 87.5% | 2,573 | 1,099 / 74 | 929 / 50 |
+| S + Jev Top 3 | 92.9% | 2,687 | 1,099 / 74 | 929 / 50 |
+| S + Jev Top 4 | 92.9% | 2,722 | 1,099 / 74 | 929 / 50 |
+| S + Jev Top 5 | 92.5% | 2,722 | 1,099 / 74 | 929 / 50 |
+| H (Hermes small-window) | 73.2% | 2,308 | 1,944 / 69 | 0 / 0 |
+| H + Jev Top 1 | 80.1% | 2,486 | 1,944 / 69 | 929 / 50 |
+| H + Jev Top 2 | 92.5% | 2,566 | 1,944 / 69 | 929 / 50 |
+| H + Jev Top 3 | 99.2% | 2,680 | 1,944 / 69 | 929 / 50 |
+| H + Jev Top 4 | 99.2% | 2,715 | 1,944 / 69 | 929 / 50 |
+| H + Jev Top 5 | 98.8% | 2,715 | 1,944 / 69 | 929 / 50 |
-All controls remain in their separate denominator. They are not presented as ten additional long-context tasks.
+These controls are not ten additional long-context tasks. Their small source-unit counts can make Top N saturate.
-## Audit and reproducibility
+
+Audit and artifact locations
-See [commands](./README.md). The audit replayed all **672** grades, reconstructed original source spans, verified source
-hashes and actual initial system prompts, checked frozen S and strict nested Top-N sets, protected prompts, question/gold
-isolation, actual token ceilings, native strict shrinkage and configured-key exclusion. All 336 contexts are valid.
+The audit verifies source hashes, frozen S/H, identical nested Top-N originals, exact captured inputs, all 1,092 grades,
+actual token ceilings, unchanged prior results and configured-key exclusion. All 546 conditions are valid.
+Prior answer-format amendments and native calibration disclosures are retained in [the protocol](./STUDY.md).
-The format preflight exposed two underspecified answer shapes: payment counts as arrays and note text with bullet
-markers. Those two follow-ups were clarified without changing gold or graders, and every seven-arm condition on both
-tasks was re-answered using the same S/rankings. Their 84 initial answers remain archived and excluded; all 504 answers
-from the other twelve tasks remain included. The native adapter adds 84 fresh answers. Unadapted native/length-only
-calibration attempts are also retained; they are not silently counted as successful compression.
+Artifacts under `/Users/yuli/workspace/trajectory-runs/` remain outside Git:
+`hermes-fixture14-20261016/` (corpus), `compaction-fixture14-20261016-v2/` (S/rankings),
+`compaction-fixture14-hermes-adapted-20261016/` (H), `compaction-fixture14-final-20261016/` (original eight arms),
+and `compaction-fixture14-dual-summary-20261016/` (current 13-arm results, usage and audit).
+The extension used **1,170,946 input / 19,990 output Qwen tokens** for its 420 answers, with zero new Jev/summary calls.
+Historical collection, calibration and complete monetary billing are not included in this incremental usage figure.
-Formal recorded usage: **757 Qwen calls** (672 answers, 42 S calls, 43 native calls including one length retry),
-2,027,184 input / 40,099 output tokens; **54 Jev calls**, 77,799 input / 5,634 output tokens. This excludes Hermes
-collection and preflights, including interrupted calibration attempts. No complete monetary-cost claim is made.
-Downstream input reduction is not total pipeline savings or a prompt-cache measurement.
-
-Artifacts remain outside the repository: `hermes-fixture14-20261016/` holds the frozen corpus and actual Hermes sessions;
-`compaction-fixture14-20261016-v2/` holds the seven-arm records; `compaction-fixture14-hermes-adapted-20261016/` holds native
-generation; `compaction-fixture14-final-20261016/` holds the combined results and `audit.json`. This is a development panel,
-including a previously used payment task, not independent held-out confirmation. Production `compact_context` is unchanged.
+
diff --git a/benchmark/compaction/STUDY.md b/benchmark/compaction/STUDY.md
index 2b9290ea..09c2326c 100644
--- a/benchmark/compaction/STUDY.md
+++ b/benchmark/compaction/STUDY.md
@@ -95,3 +95,22 @@ Unadapted native generation and length-calibration records remain outside the re
included its long control prefix and exceeded Full input on 40/42 contexts; it is retained as an integration preflight,
not silently treated as successful compression. The explicit body carrier and word-budget adapter above are fixed
before their new scored answers. No expected answer, cohort, Jev ranking or existing S is changed.
+
+
+## Current comparison: two frozen summary mechanisms
+
+The current experiment is one 13-arm panel: Full; S + Jev Top 0–5; H + Jev Top 0–5. Top 0 means summary only.
+H is the forced Hermes small-window adapter defined above. The extension reuses every original S/H summary, ranking,
+selected unit and scored answer; no new summarization, Jev request or trajectory collection occurs. It adds 210 contexts
+and 420 answers, for 546 contexts and 1,092 answers overall. All new inputs were frozen before answering.
+S/H use identical selected units per task/generation/N, restored to source order. No answer-based reranking is allowed.
+
+The extension keeps the existing per-task ceilings. Strict shrinkage versus Full is reported separately, without
+filtering or removing conditions: Top 3–5 each shorten 9/12 primary contexts in both mechanisms. The H fusion arms
+shorten 172/210 contexts (51/60 primary, 121/150 controls). Summary-only H still shortens all 42/42.
+
+For costs, each independent method includes one summary generation when applicable, one complete Jev ranking when
+N > 0, and one downstream answer. Recorded retry usage is included. One ranking is shared across the Top-N sweep and
+mechanisms in the actual experiment; its standalone-method cost is neither multiplied by N nor divided by sweep size.
+The two answer repeats do not amortize compression overhead. Report provider input/output tokens separately; do not
+claim currency or total savings without prices, cache billing and an explicit continuation horizon.
diff --git a/benchmark/compaction/suite_fusion.py b/benchmark/compaction/suite_fusion.py
new file mode 100644
index 00000000..271b5290
--- /dev/null
+++ b/benchmark/compaction/suite_fusion.py
@@ -0,0 +1,250 @@
+"""Reuse frozen Hermes summaries and Jev selections; score only H + Top 1..5."""
+
+import argparse
+import asyncio
+from copy import deepcopy
+import hashlib
+import json
+from pathlib import Path
+import shutil
+import statistics
+
+from dotenv import dotenv_values
+from openai import AsyncOpenAI
+
+from benchmark.compaction.cases import render, write
+from benchmark.compaction.run import Experiment, compose, messages, top_n
+from benchmark.compaction.suite import grade
+from benchmark.compaction.suite_native import validate_base
+from benchmark.compaction.suite_run import ARMS, load_cases, report
+
+
+def costs(args, rows, cases):
+ """Standalone one-answer costs, including one summary/ranking and all recorded retries."""
+ result = {}
+ for cohort in ("primary", "controls"):
+ ids = {k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")}
+ table = []
+ for arm in ARMS + ["hermes_summary"] + [f"hermes_top{n}" for n in range(1, 6)]:
+ entries = []
+ for row in rows:
+ if row["case"] not in ids or row["arm"] != arm:
+ continue
+ key, i = row["case"], row["repeat"]
+ summary = []
+ if arm.startswith("hermes"):
+ paths = (args.native / key / "generation").glob(f"c{i}.a*.summary.response.json")
+ summary = [json.loads(p.read_text())["usage"] for p in paths]
+ elif arm != "full":
+ paths = (args.evaluation / key).glob(f"paired.c{i}.summary*.response.json")
+ summary = [json.loads(p.read_text())["usage"] for p in paths]
+ decision = []
+ if arm.startswith(("fusion", "hermes_top")):
+ paths = (args.evaluation / key).glob(f"paired.c{i}.decision*.response.json")
+ decision = [json.loads(p.read_text())["usage"] for p in paths]
+ entries.append(
+ {
+ "summary_input": sum(u["prompt_tokens"] for u in summary),
+ "summary_output": sum(u["completion_tokens"] for u in summary),
+ "jev_input": sum(u["input_tokens"] for u in decision),
+ "jev_output": sum(u["output_tokens"] for u in decision),
+ "answer_input": (
+ statistics.mean(t["usage"]["prompt_tokens"] for t in row["trials"])
+ if row["trials"]
+ else None
+ ),
+ "answer_output": (
+ statistics.mean(t["usage"]["completion_tokens"] for t in row["trials"])
+ if row["trials"]
+ else None
+ ),
+ },
+ )
+ table.append(
+ {
+ "arm": arm,
+ **{k: statistics.mean(e[k] for e in entries if e[k] is not None) for k in entries[0]},
+ },
+ )
+ result[cohort] = table
+ write(args.output / "costs.json", result)
+ return result
+
+
+def prepare(args, cases):
+ """Verify artifacts and freeze all paired inputs before any new answer call."""
+ original = validate_base(args.evaluation, args.corpus, cases)
+ combined = json.loads((args.combined / "results.json").read_text())
+ expected = {(k, a, i) for k in cases for a in ARMS + ["hermes_summary"] for i in range(3)}
+ actual = {(r["case"], r["arm"], r["repeat"]) for r in combined}
+ if actual != expected or len(combined) != len(expected):
+ raise ValueError("Expected the complete frozen eight-method panel")
+ if [r for r in combined if r["arm"] != "hermes_summary"] != original:
+ raise ValueError("Combined panel changed the original seven-method results")
+ index = {(r["case"], r["arm"], r["repeat"]): r for r in combined}
+ prepared = []
+ for key, case in cases.items():
+ root = args.evaluation / key
+ source = root / "source.json"
+ saved = json.loads(source.read_text())
+ if saved["messages"] != case["messages"] or saved["exam"] != case["exam"] or saved["gold"] != case["gold"]:
+ raise ValueError("Corpus or evaluation questions changed")
+ generation = args.native / key / "generation"
+ validation = json.loads((generation / "validation.json").read_text())
+ native_source = (args.native_evaluation or args.evaluation) / key / "source.json"
+ if validation["source_sha256"] != hashlib.sha256(native_source.read_bytes()).hexdigest():
+ raise ValueError("Native summary source mismatch")
+ native_saved = json.loads(native_source.read_text())
+ # The format amendment changed only the held-out exam, never native generator input.
+ if {k: v for k, v in native_saved.items() if k != "exam"} != {k: v for k, v in saved.items() if k != "exam"}:
+ raise ValueError("Native source evidence changed")
+ summaries = json.loads((generation / "summaries.json").read_text())
+ if len(summaries) != 3 or {r["repeat"] for r in summaries} != set(range(3)):
+ raise ValueError("Native generations are incomplete")
+ for entry in summaries:
+ i = entry["repeat"]
+ native = index[key, "hermes_summary", i]
+ if not entry["valid"] or not native["valid"] or native["summary"] != entry["summary"]:
+ raise ValueError("Native summary is invalid or changed")
+ scores = {}
+ for path in sorted(root.glob(f"paired.c{i}.decision*.response.json")):
+ scores.update(json.loads(path.read_text())["scores"])
+ full_tokens = index[key, "full", i]["trials"][0]["usage"]["prompt_tokens"]
+ for n in range(1, 6):
+ paired = index[key, f"fusion_top{n}", i]
+ chosen = top_n(saved["units"], scores, min(n, len(saved["units"])))
+ if paired["selected"] != [u["id"] for u in chosen]:
+ raise ValueError("Frozen selection differs from recorded Jev ranking")
+ context = messages(case, compose(entry["summary"], render(chosen)))
+ prepared.append(
+ {
+ "case": key,
+ "arm": f"hermes_top{n}",
+ "repeat": i,
+ "summary": entry["summary"],
+ "selected": paired["selected"],
+ "actual_n": len(chosen),
+ "input_cap": paired["input_cap"],
+ "full_input_tokens": full_tokens,
+ "context": context,
+ "valid": False,
+ "trials": [],
+ },
+ )
+ return deepcopy(combined), prepared
+
+
+async def run(args):
+ """No summary or decision requests; every new trial only calls the answering model."""
+ from tokenizers import Tokenizer
+
+ repo = Path(__file__).resolve().parents[2]
+ output = args.output.resolve()
+ if output.exists() or output == repo or repo in output.parents:
+ raise ValueError("Use a new output directory outside repository")
+ _, cases = load_cases(args.corpus)
+ rows, prepared = prepare(args, cases)
+ tokenizer = Tokenizer.from_file(str(args.tokenizer))
+ values = dotenv_values(repo / ".env")
+ if not values.get("LLM_API_KEY") or not values.get("LLM_BASE_URL"):
+ raise ValueError("Missing Qwen credentials")
+ output.mkdir(parents=True)
+ for name in ("suite_fusion.py", "suite_run.py", "STUDY.md"):
+ shutil.copy2(Path(__file__).parent / name, output / name)
+ write(
+ output / "protocol.json",
+ {
+ "base_evaluation": str(args.evaluation.resolve()),
+ "combined_evaluation": str(args.combined.resolve()),
+ "native_generations": str(args.native.resolve()),
+ "native_source_evaluation": str((args.native_evaluation or args.evaluation).resolve()),
+ "model": "qwen3.8-max",
+ "methods": 13,
+ "tasks": 14,
+ "generations": 3,
+ "answers_per_context": 2,
+ "new_contexts": 210,
+ "new_answers": 420,
+ "summary_calls": 0,
+ "decision_calls": 0,
+ "acceptance": "same existing input ceilings; mark strict reduction separately without filtering tasks",
+ "ranking": "identical frozen Jev scores and selected originals for S and H",
+ "corpus_sha256": hashlib.sha256(args.corpus.read_bytes()).hexdigest(),
+ },
+ )
+ for row in prepared:
+ root = output / row["case"]
+ root.mkdir(exist_ok=True)
+ label = f"c{row['repeat']}.{row['arm']}"
+ write(root / f"{label}.context.json", {"messages": row.pop("context")})
+ rows.extend(prepared)
+ write(output / "frozen-selections.json", rows)
+ write(output / "results.json", rows)
+ client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1)
+ limit = asyncio.Semaphore(4)
+ work = asyncio.Semaphore(4)
+
+ async def answer(row):
+ async with work:
+ root = output / row["case"]
+ label = f"c{row['repeat']}.{row['arm']}"
+ exp = Experiment(client, None, root, tokenizer, limit, "qwen3.8-max", row["input_cap"])
+ try:
+ context = json.loads((root / f"{label}.context.json").read_text())["messages"]
+ trials = []
+ for j in range(2):
+ text, usage = await exp.ask(f"{label}.answer{j}", context)
+ prediction = json.loads(text)
+ trials.append(
+ {"answer": prediction, "usage": usage, "checks": grade(cases[row["case"]], prediction)},
+ )
+ row["trials"] = trials
+ row["valid"] = all(t["usage"]["prompt_tokens"] <= row["input_cap"] for t in trials)
+ row["shorter_than_full"] = all(t["usage"]["prompt_tokens"] < row["full_input_tokens"] for t in trials)
+ if not row["valid"]:
+ row["error"] = "input_ceiling_exceeded"
+ except Exception as exc:
+ row.update(valid=False, error=type(exc).__name__, trials=[])
+ write(output / "results.json", rows)
+ report(output, rows, cases, 2, include_native_fusion=True)
+ print(row["case"], label, "OK" if row["valid"] else row["error"], flush=True)
+
+ try:
+ async with asyncio.TaskGroup() as tasks:
+ for row in prepared:
+ tasks.create_task(answer(row))
+ finally:
+ await client.close()
+ load_cases(args.corpus)
+ costs(args, rows, cases)
+ for path in output.rglob("*.json"):
+ if values["LLM_API_KEY"].encode() in path.read_bytes():
+ raise ValueError("Configured API key in artifacts")
+ write(
+ output / "validation.json",
+ {
+ "contexts": len(rows),
+ "answers": sum(len(r["trials"]) for r in rows),
+ "new_valid_contexts": sum(r["valid"] for r in prepared),
+ "new_contexts_shorter_than_full": sum(r.get("shorter_than_full", False) for r in prepared),
+ "no_new_summary_or_decision_calls": True,
+ "no_api_keys": True,
+ },
+ )
+
+
+def main():
+ """Require explicit live opt-in; all artifacts must already exist except output."""
+ parser = argparse.ArgumentParser(description=__doc__)
+ for name in ("corpus", "evaluation", "combined", "native", "tokenizer", "output"):
+ parser.add_argument("--" + name, type=Path, required=True)
+ parser.add_argument("--live", action="store_true")
+ parser.add_argument("--native-evaluation", type=Path)
+ args = parser.parse_args()
+ if not args.live:
+ parser.error("Use --live to authorize the 420 answering calls")
+ asyncio.run(run(args))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/compaction/suite_run.py b/benchmark/compaction/suite_run.py
index ceb8267c..386339e3 100644
--- a/benchmark/compaction/suite_run.py
+++ b/benchmark/compaction/suite_run.py
@@ -65,11 +65,14 @@ def load_cases(corpus_path):
return manifest, result
-def metrics(rows, ids, repeats, include_native=False):
+def metrics(rows, ids, repeats, include_native=False, include_native_fusion=False):
"""Equal task weights; invalid planned contexts remain zero instead of disappearing."""
table = []
arms = ARMS[:2] + ["hermes_summary"] + ARMS[2:] if include_native else ARMS
labels = LABELS[:2] + ["Hermes small-window adapter"] + LABELS[2:] if include_native else LABELS
+ if include_native_fusion:
+ arms = ARMS + ["hermes_summary"] + [f"hermes_top{n}" for n in range(1, 6)]
+ labels = LABELS + ["H (Hermes small-window)"] + [f"H + Jev Top {n}" for n in range(1, 6)]
for arm, label in zip(arms, labels):
task_facts = []
task_exact = []
@@ -108,13 +111,13 @@ def metrics(rows, ids, repeats, include_native=False):
return table
-def report(output, rows, cases, repeats, include_native=False):
- """Keep primary and short/query-only controls separate in compact seven-row tables."""
+def report(output, rows, cases, repeats, include_native=False, include_native_fusion=False):
+ """Keep primary and short/query-only controls in separate tables."""
result = {}
lines = ["# Fourteen-task fixed-summary study", ""]
for cohort in ("primary", "controls"):
ids = [k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")]
- result[cohort] = metrics(rows, ids, repeats, include_native)
+ result[cohort] = metrics(rows, ids, repeats, include_native, include_native_fusion)
lines += [
f"## {cohort.title()} ({len(ids)} tasks)",
"",
@@ -128,7 +131,7 @@ def report(output, rows, cases, repeats, include_native=False):
f"{r['valid_contexts']}/{r['planned_contexts']} |",
)
lines += [""]
- result["per_task"] = {k: metrics(rows, [k], repeats, include_native) for k in cases}
+ result["per_task"] = {k: metrics(rows, [k], repeats, include_native, include_native_fusion) for k in cases}
write(output / "aggregate.json", result)
(output / "REPORT.md").write_text("\n".join(lines) + "\n")
diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md
index 582f780b..12a1af1b 100644
--- a/docs/en/context_compaction.md
+++ b/docs/en/context_compaction.md
@@ -117,27 +117,22 @@ Only model requests run externally; fixture tool calls are not executed. These c
success. Evaluate real trajectories against recency and rule-based previews, measuring recovery, continuation quality,
latency and total cost alongside token reduction. ReMe does not automatically summarize or write persistent memory.
-## Fixed summary + selected original evidence
+## Jev across two summary strategies
-The experiment uses **14 fixture queries with predefined answers**, collected through Hermes: four source-length-qualified
-primary tasks and ten short/query-only controls. Rule scoring checks exact JSON fields; no LLM judge creates or grades gold.
+One 13-method experiment compares Full context with a generic summary S and a forced Hermes small-window summary H,
+each plus Jev Top 0–5. Top 0 means summary only. Every condition preserves its frozen summary and shares the same
+Jev-selected originals across mechanisms.
-
+
-Across the four primary tasks, Full context reaches 100%, S reaches 82.0%, and S + Top 3 reaches **99.5% with 19.3% fewer
-input tokens** (3,241 versus 4,014). Top 4 and Top 5 score 98.1% and 97.4%, so more retained text is not automatically better.
-Each generation freezes S and the Jev ranking before answers; all five fusion methods keep byte-identical S.
+On four primary task templates, S improves from **82.0% to 99.5%** at Top 3; H improves from **79.4% to 97.9%**.
+Their mean answer inputs are **3,241 / 3,193 tokens**, versus Full's **4,014 / 100%**. The figure separates downstream
+input from summary/ranking overhead; fewer answer tokens do not establish lower total monetary cost.
-The **Hermes small-window adapter** scores 79.4% at 2,432 input tokens. It calls the real native summary generator with
-explicit small-window budget/word constraints and a body-only carrier; this is not the unchanged automatic compressor.
-All 42 native contexts fit their budgets and use strictly fewer actual input tokens than Full, with no original fallback.
+The panel has 14 predefined query/answer templates, 546 contexts and 1,092 direct-Qwen answers. Ten short/query-only
+controls are reported separately. H uses an explicit native-generator budget adapter, not Hermes' default automatic
+compressor. All H-only contexts shorten Full; at Top 3–5 only 9/12 primary contexts per method do so, with all cases
+retained in scoring. This is an incremental, unequal-length comparison without agent continuation.
-
-Top N counts complete source units and saturates at each task's available count. The original payment has ten units,
-so its maximum is Top 10; the sweep tests Top 1–5. Summary/body allowances and complete-input ceilings are task-specific;
-this is an incremental-evidence curve, not an equal-length comparison or proof that every Full exceeds its ceiling.
-
-The panel has 336 contexts and 672 answers, with the controls reported separately. Hermes collects source trajectories;
-answering calls Qwen directly without tool continuation or recovery. These few synthetic tasks do not establish general
-superiority or statistical equivalence. The hybrid and native adapter remain benchmark-only.
-See the [exact queries, answers and settings](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/STUDY.md) and [results, controls and audit](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/RESULTS.md).
+
+See the [query/answer protocol](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md) and [complete results and token accounting](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md).
diff --git a/docs/figure/compaction-topn-results.svg b/docs/figure/compaction-topn-results.svg
index 9a93cf1c..353f5308 100644
--- a/docs/figure/compaction-topn-results.svg
+++ b/docs/figure/compaction-topn-results.svg
@@ -1,11 +1,12 @@
-