scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-10-23 05:35:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

bench: compare Jev fusion across frozen summary strategies

Browse source
This commit is contained in:
jinli.yl 2026-10-16 17:21:53 +08:00
parent eb816361d8
commit d2638b166f
9 changed files with 980 additions and 404 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
benchmark/compaction/README.md
View file

@ -4,7 +4,7 @@ A **14-query fixture panel** with predefined answers and rule scoring: four prim
controls. See [queries, answers and frozen settings](./STUDY.md) and [measured results](./RESULTS.md).
Hermes collects the source through real `read_file` calls. ReMe's experiment then compares Full context, a frozen Qwen
summary S, S + Jev Top 1–5, and a **Hermes small-window summary adapter**. Scored answering calls Qwen directly;
summary S and forced **Hermes small-window summary H**, each with Jev Top 0–5. Scored answering calls Qwen directly;
there is no resumed Hermes tool loop or archive recovery. Production `compact_context` is separate.
## Run
@ -39,15 +39,24 @@ python -m benchmark.compaction.suite_native --live \
--hermes-python /path/to/hermes-environment/bin/python \
--tokenizer /path/to/cached/qwen/tokenizer.json \
--output /path/outside/repository/combined-results
# Reuse all frozen summaries/rankings and 672 answers; add only H + Top 1–5.
python -m benchmark.compaction.suite_fusion --live \
--corpus /path/outside/repository/corpus/corpus.json \
--evaluation /path/outside/repository/evaluation \
--combined /path/outside/repository/combined-results \
--native /path/outside/repository/native-generations \
--tokenizer /path/to/cached/qwen/tokenizer.json \
--output /path/outside/repository/dual-summary-results
```
The task definitions use local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b` (MIT). The recorded study uses
`qwen3.8-max`, three summary/ranking generations and two answers per context: **336 contexts, 672 answers**. Provider
requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay
`qwen3.8-max`, three summary/ranking generations and two answers per context: **546 contexts, 1,092 answers across thirteen methods**.
Provider requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay
outside git; existing runs are never overwritten.
S remains byte-identical across its five fusion variants. Top N means a strict descending-score prefix, without a
probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer
S and H each remain byte-identical across their five fusion variants; both use the same Jev selections.
Top N means a strict descending-score prefix, without a probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer
than N units, all available units are retained and `actual_n` records the count. The original payment source has ten
units (maximum Top 10); this study sweeps Top 1–5 with task-specific unit counts.
@ -55,3 +64,10 @@ The native baseline invokes Hermes' actual summary generator with an explicit sm
S-matched body budget, final word constraint, native provenance validation, and body-only downstream framing. It bypasses
normal triggers/head-tail selection and **must actually shorten input**; it is not the unchanged automatic Hermes policy.
See [the exact simulation settings and preflight disclosures](./STUDY.md#added-baseline-hermes-small-window-summary-adapter).
For the recorded format amendment, native summaries used the pre-amendment source files: pass
`--native-evaluation /path/to/pre-amendment/evaluation` when extending that run. The extension verifies its source hash
and permits only the held-out `exam` field to differ; the summary generator never receives that field.
`costs.json` reports standalone summary/ranking input and output overhead separately from answer usage, including
recorded retries. Token counts do not apply model prices or cache discounts and are not monetary savings.

158
benchmark/compaction/RESULTS.md
View file

@ -1,97 +1,105 @@
# Fixed-summary compaction on fourteen fixture tasks
# Jev across two summary strategies
2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · **14 distinct queries with predefined answers**.
Four source-length-qualified tasks form the primary cohort; ten short/query-only controls are reported separately.
Cohorts were fixed before collection, with no accuracy-based task filtering.
2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0`.
## Primary cohort: four tasks
One **13-method experiment**: Full context, generic summary **S + Top 0–5**, and forced Hermes small-window summary
**H + Top 0–5**. Top 0 means summary only. Both curves share the same original evidence and frozen Jev rankings;
each summary remains byte-identical across its six conditions.
| Method | Mean input tokens | Fact score | Valid contexts |
| --- | ---: | ---: | --- |
| Full context | 4,014 | 100.0% | 12/12 |
| S | 2,480 | 82.0% | 12/12 |
| Hermes small-window adapter | 2,432 | 79.4% | 12/12 |
| S + Jev Top 1 | 2,753 | 83.0% | 12/12 |
| S + Jev Top 2 | 2,974 | 94.5% | 12/12 |
| S + Jev Top 3 | 3,241 | 99.5% | 12/12 |
| S + Jev Top 4 | 3,496 | 98.1% | 12/12 |
| S + Jev Top 5 | 3,660 | 97.4% | 12/12 |
## Four primary tasks
Input tokens are actual downstream API usage, including the original system/task and follow-up. Fact score averages
strict JSON field checks within each answer, then generations and tasks equally; invalid conditions score zero.
Full context reaches 100%; S + Top 3 reaches **99.5% with 19.3% less downstream input** (3,241 versus 4,014 tokens).
Top 4 and Top 5 do not improve this point estimate. No equivalence test or general optimal-N claim is made.
| Method | Fact score | Answer input | Summary in / out | Jev in / out |
| --- | ---: | ---: | ---: | ---: |
| Full context | 100.0% | 4,014 | 0 / 0 | 0 / 0 |
| S | 82.0% | 2,480 | 5,379 / 200 | 0 / 0 |
| S + Jev Top 1 | 83.0% | 2,753 | 5,379 / 200 | 4,162 / 344 |
| S + Jev Top 2 | 94.5% | 2,974 | 5,379 / 200 | 4,162 / 344 |
| S + Jev Top 3 | 99.5% | 3,241 | 5,379 / 200 | 4,162 / 344 |
| S + Jev Top 4 | 98.1% | 3,496 | 5,379 / 200 | 4,162 / 344 |
| S + Jev Top 5 | 97.4% | 3,660 | 5,379 / 200 | 4,162 / 344 |
| H (Hermes small-window) | 79.4% | 2,432 | 3,120 / 152 | 0 / 0 |
| H + Jev Top 1 | 85.3% | 2,705 | 3,120 / 152 | 4,162 / 344 |
| H + Jev Top 2 | 92.9% | 2,927 | 3,120 / 152 | 4,162 / 344 |
| H + Jev Top 3 | 97.9% | 3,193 | 3,120 / 152 | 4,162 / 344 |
| H + Jev Top 4 | 96.9% | 3,448 | 3,120 / 152 | 4,162 / 344 |
| H + Jev Top 5 | 96.9% | 3,613 | 3,120 / 152 | 4,162 / 344 |
![Primary-cohort input and fact score](../../docs/figure/compaction-topn-results.svg)
![Quality, downstream input and compression overhead](../../docs/figure/compaction-topn-results.svg)
## What was tested
Both summary mechanisms benefit on this panel: S rises from **82.0% to 99.5%** at Top 3; H rises from **79.4% to 97.9%**.
Mean downstream inputs at Top 3 are **3,241 / 3,193**, versus Full's **4,014**: reductions of **19.3% / 20.4%**.
Top 4–5 do not improve these point estimates. This is an exploratory result on four synthetic task templates,
not a general optimal-N claim, equivalence test or proof that Jev beats other equal-budget selectors.
The [full query/answer table and protocol](./STUDY.md) define the oracle before collection. The four primary tasks are
payment retry investigation, checkout 504 analysis, server-error localization and a build-output token lookup.
The ten controls cover shorter configuration, code, dependency, notes, version and file-generation questions.
All fourteen sources are synthetic fixtures: one existing payment trajectory plus thirteen genuine Hermes executions
over explicitly read-only adaptations of upstream tasks. Long-file views are deterministic original excerpts, not
added noise. These are not the maintainers' private production transcripts.
## Token and cost accounting
Each task has three frozen summary/ranking pairs and two answers per method: **336 contexts and 672 answers** across
eight methods. S summarizes every source unit once. Each S + Top N preserves that S byte-for-byte and adds complete
original units, ranked without a threshold and restored to source order. Actual N saturates at the available unit count.
The original payment source has ten units, so its maximum is Top 10; the sweep tests Top 1–5.
All table values are mean provider tokens. Answer input includes the original system/task, summary/evidence and held-out
question. Summary and Jev columns are **one-time overhead per standalone condition**, including recorded length retries.
One frozen Jev ranking serves every N and either summary mechanism: Top 1–5 therefore have the same ranking cost;
it is neither multiplied by N nor divided by the number of sweep conditions. Summary-only and Full incur no Jev cost.
The two scoring repeats estimate one downstream answer; overhead is not amortized across those evaluation repeats.
S's evidence allowance is 20% of rendered-evidence tokens, clamped to 64–256 local Qwen tokens. A per-task complete-input
ceiling is calculated from protected prompts, the summary allowance, the five largest whole units and reserve, before
scored answers. This permits the incremental sweep; **it is not an equal-length S-versus-fusion comparison or a claim
that every Full input exceeds its ceiling**. All actual inputs are checked. Short cases may gain no token savings.
These are token counts, **not currency**: model prices and cached-token discounts are not applied. Compression increases
one-off provider work here; lower downstream input alone does not demonstrate lower total cost. Repeated future turns
may amortize overhead, but that has not been measured.
The added **Hermes small-window adapter** calls the real native generator, with its legacy template, an overridden
64–256-token body budget and an appended total word limit. It retains native redaction/provenance checks and strips
only the native control prefix before the common summary carrier. It bypasses automatic triggering and full-compressor
head/tail selection: **this is a forced simulation, not the unchanged default Hermes compressor**. All **42/42** native
contexts fit their body/input budgets and use fewer actual API input tokens than the corresponding Full context.
No original-history fallback is accepted.
## Setup and boundaries
Hermes collects the trajectories; answering calls Qwen directly. Gold and follow-up questions never enter collection,
summary or decision requests. The final collection report is excluded from every history. There is no tool continuation,
retrieval, LLM judge or generated answer key. Exact string/type compliance is part of this metric; it does not by itself
prove semantic information loss or task-completion differences. The native and S methods have the same maximum body
allowance, but their actual lengths differ. Four primary templates are too few to establish general superiority.
[Queries, fixed answers and scoring rules](./STUDY.md) · [Reproduction commands](./README.md).
The panel contains **14 task templates**, one existing payment trajectory and thirteen real read-only Hermes executions
over synthetic upstream fixtures. Four primary tasks were selected by source length before collection; ten short/query-only
controls remain separate. Three summary/ranking pairs and two answers per condition give **546 contexts / 1,092 answers**.
The extension reused all 42 H summaries, rankings and 672 previous answers; it added only 420 answering calls.
No new summary or Jev calls were needed. All 1,092 grades were replayed against captured requests/responses.
Scoring uses predefined, recursive type-strict JSON field rules, with no LLM judge. Questions/gold/final collection reports
are excluded from summary and decision inputs. Answering calls Qwen directly, without Hermes tool continuation or recovery.
S and H share a 64–256-token maximum summary-body allowance, but use different prompts and actual lengths.
H calls Hermes' real legacy generator through an explicit budget/word-limit adapter; it is **not the default automatic
compressor**. All 42 H-only contexts strictly shorten Full, without original-history fallback.
Top N adds complete source units without truncation, saturating at the available count. The payment source has ten units
(maximum Top 10); this experiment sweeps Top 1–5. Existing per-task input ceilings are preserved, not tuned after answers.
The comparison is incremental, **not equal-length**. All conditions fit their ceilings, but only **9/12 primary contexts
per method at Top 3–5** are shorter than their paired Full. Across the new H fusion arms, 172/210 contexts shorten Full;
the other 38 comprise 9 primary and 29 control contexts. All remain scored. Average savings do not imply per-case savings.
<details>
<summary>Ten short/query-only controls</summary>
| Method | Mean input tokens | Fact score | Valid contexts |
| --- | ---: | ---: | --- |
| Full context | 2,727 | 100.0% | 30/30 |
| S | 2,315 | 67.1% | 30/30 |
| Hermes small-window adapter | 2,308 | 73.2% | 30/30 |
| S + Jev Top 1 | 2,493 | 84.7% | 30/30 |
| S + Jev Top 2 | 2,573 | 87.5% | 30/30 |
| S + Jev Top 3 | 2,687 | 92.9% | 30/30 |
| S + Jev Top 4 | 2,722 | 92.9% | 30/30 |
| S + Jev Top 5 | 2,722 | 92.5% | 30/30 |
| Method | Fact score | Answer input | Summary in / out | Jev in / out |
| --- | ---: | ---: | ---: | ---: |
| Full context | 100.0% | 2,727 | 0 / 0 | 0 / 0 |
| S | 67.1% | 2,315 | 1,099 / 74 | 0 / 0 |
| S + Jev Top 1 | 84.7% | 2,493 | 1,099 / 74 | 929 / 50 |
| S + Jev Top 2 | 87.5% | 2,573 | 1,099 / 74 | 929 / 50 |
| S + Jev Top 3 | 92.9% | 2,687 | 1,099 / 74 | 929 / 50 |
| S + Jev Top 4 | 92.9% | 2,722 | 1,099 / 74 | 929 / 50 |
| S + Jev Top 5 | 92.5% | 2,722 | 1,099 / 74 | 929 / 50 |
| H (Hermes small-window) | 73.2% | 2,308 | 1,944 / 69 | 0 / 0 |
| H + Jev Top 1 | 80.1% | 2,486 | 1,944 / 69 | 929 / 50 |
| H + Jev Top 2 | 92.5% | 2,566 | 1,944 / 69 | 929 / 50 |
| H + Jev Top 3 | 99.2% | 2,680 | 1,944 / 69 | 929 / 50 |
| H + Jev Top 4 | 99.2% | 2,715 | 1,944 / 69 | 929 / 50 |
| H + Jev Top 5 | 98.8% | 2,715 | 1,944 / 69 | 929 / 50 |
All controls remain in their separate denominator. They are not presented as ten additional long-context tasks.
These controls are not ten additional long-context tasks. Their small source-unit counts can make Top N saturate.
</details>
## Audit and reproducibility
<details>
<summary>Audit and artifact locations</summary>
See [commands](./README.md). The audit replayed all **672** grades, reconstructed original source spans, verified source
hashes and actual initial system prompts, checked frozen S and strict nested Top-N sets, protected prompts, question/gold
isolation, actual token ceilings, native strict shrinkage and configured-key exclusion. All 336 contexts are valid.
The audit verifies source hashes, frozen S/H, identical nested Top-N originals, exact captured inputs, all 1,092 grades,
actual token ceilings, unchanged prior results and configured-key exclusion. All 546 conditions are valid.
Prior answer-format amendments and native calibration disclosures are retained in [the protocol](./STUDY.md).
The format preflight exposed two underspecified answer shapes: payment counts as arrays and note text with bullet
markers. Those two follow-ups were clarified without changing gold or graders, and every seven-arm condition on both
tasks was re-answered using the same S/rankings. Their 84 initial answers remain archived and excluded; all 504 answers
from the other twelve tasks remain included. The native adapter adds 84 fresh answers. Unadapted native/length-only
calibration attempts are also retained; they are not silently counted as successful compression.
Artifacts under `/Users/yuli/workspace/trajectory-runs/` remain outside Git:
`hermes-fixture14-20261016/` (corpus), `compaction-fixture14-20261016-v2/` (S/rankings),
`compaction-fixture14-hermes-adapted-20261016/` (H), `compaction-fixture14-final-20261016/` (original eight arms),
and `compaction-fixture14-dual-summary-20261016/` (current 13-arm results, usage and audit).
The extension used **1,170,946 input / 19,990 output Qwen tokens** for its 420 answers, with zero new Jev/summary calls.
Historical collection, calibration and complete monetary billing are not included in this incremental usage figure.
Formal recorded usage: **757 Qwen calls** (672 answers, 42 S calls, 43 native calls including one length retry),
2,027,184 input / 40,099 output tokens; **54 Jev calls**, 77,799 input / 5,634 output tokens. This excludes Hermes
collection and preflights, including interrupted calibration attempts. No complete monetary-cost claim is made.
Downstream input reduction is not total pipeline savings or a prompt-cache measurement.
Artifacts remain outside the repository: `hermes-fixture14-20261016/` holds the frozen corpus and actual Hermes sessions;
`compaction-fixture14-20261016-v2/` holds the seven-arm records; `compaction-fixture14-hermes-adapted-20261016/` holds native
generation; `compaction-fixture14-final-20261016/` holds the combined results and `audit.json`. This is a development panel,
including a previously used payment task, not independent held-out confirmation. Production `compact_context` is unchanged.
</details>

19
benchmark/compaction/STUDY.md
View file

@ -95,3 +95,22 @@ Unadapted native generation and length-calibration records remain outside the re
included its long control prefix and exceeded Full input on 40/42 contexts; it is retained as an integration preflight,
not silently treated as successful compression. The explicit body carrier and word-budget adapter above are fixed
before their new scored answers. No expected answer, cohort, Jev ranking or existing S is changed.
## Current comparison: two frozen summary mechanisms
The current experiment is one 13-arm panel: Full; S + Jev Top 0–5; H + Jev Top 0–5. Top 0 means summary only.
H is the forced Hermes small-window adapter defined above. The extension reuses every original S/H summary, ranking,
selected unit and scored answer; no new summarization, Jev request or trajectory collection occurs. It adds 210 contexts
and 420 answers, for 546 contexts and 1,092 answers overall. All new inputs were frozen before answering.
S/H use identical selected units per task/generation/N, restored to source order. No answer-based reranking is allowed.
The extension keeps the existing per-task ceilings. Strict shrinkage versus Full is reported separately, without
filtering or removing conditions: Top 3–5 each shorten 9/12 primary contexts in both mechanisms. The H fusion arms
shorten 172/210 contexts (51/60 primary, 121/150 controls). Summary-only H still shortens all 42/42.
For costs, each independent method includes one summary generation when applicable, one complete Jev ranking when
N > 0, and one downstream answer. Recorded retry usage is included. One ranking is shared across the Top-N sweep and
mechanisms in the actual experiment; its standalone-method cost is neither multiplied by N nor divided by sweep size.
The two answer repeats do not amortize compression overhead. Report provider input/output tokens separately; do not
claim currency or total savings without prices, cache billing and an explicit continuation horizon.

250
benchmark/compaction/suite_fusion.py Normal file
View file

@ -0,0 +1,250 @@
"""Reuse frozen Hermes summaries and Jev selections; score only H + Top 1..5."""
import argparse
import asyncio
from copy import deepcopy
import hashlib
import json
from pathlib import Path
import shutil
import statistics
from dotenv import dotenv_values
from openai import AsyncOpenAI
from benchmark.compaction.cases import render, write
from benchmark.compaction.run import Experiment, compose, messages, top_n
from benchmark.compaction.suite import grade
from benchmark.compaction.suite_native import validate_base
from benchmark.compaction.suite_run import ARMS, load_cases, report
def costs(args, rows, cases):
"""Standalone one-answer costs, including one summary/ranking and all recorded retries."""
result = {}
for cohort in ("primary", "controls"):
ids = {k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")}
table = []
for arm in ARMS + ["hermes_summary"] + [f"hermes_top{n}" for n in range(1, 6)]:
entries = []
for row in rows:
if row["case"] not in ids or row["arm"] != arm:
continue
key, i = row["case"], row["repeat"]
summary = []
if arm.startswith("hermes"):
paths = (args.native / key / "generation").glob(f"c{i}.a*.summary.response.json")
summary = [json.loads(p.read_text())["usage"] for p in paths]
elif arm != "full":
paths = (args.evaluation / key).glob(f"paired.c{i}.summary*.response.json")
summary = [json.loads(p.read_text())["usage"] for p in paths]
decision = []
if arm.startswith(("fusion", "hermes_top")):
paths = (args.evaluation / key).glob(f"paired.c{i}.decision*.response.json")
decision = [json.loads(p.read_text())["usage"] for p in paths]
entries.append(
{
"summary_input": sum(u["prompt_tokens"] for u in summary),
"summary_output": sum(u["completion_tokens"] for u in summary),
"jev_input": sum(u["input_tokens"] for u in decision),
"jev_output": sum(u["output_tokens"] for u in decision),
"answer_input": (
statistics.mean(t["usage"]["prompt_tokens"] for t in row["trials"])
if row["trials"]
else None
),
"answer_output": (
statistics.mean(t["usage"]["completion_tokens"] for t in row["trials"])
if row["trials"]
else None
),
},
)
table.append(
{
"arm": arm,
**{k: statistics.mean(e[k] for e in entries if e[k] is not None) for k in entries[0]},
},
)
result[cohort] = table
write(args.output / "costs.json", result)
return result
def prepare(args, cases):
"""Verify artifacts and freeze all paired inputs before any new answer call."""
original = validate_base(args.evaluation, args.corpus, cases)
combined = json.loads((args.combined / "results.json").read_text())
expected = {(k, a, i) for k in cases for a in ARMS + ["hermes_summary"] for i in range(3)}
actual = {(r["case"], r["arm"], r["repeat"]) for r in combined}
if actual != expected or len(combined) != len(expected):
raise ValueError("Expected the complete frozen eight-method panel")
if [r for r in combined if r["arm"] != "hermes_summary"] != original:
raise ValueError("Combined panel changed the original seven-method results")
index = {(r["case"], r["arm"], r["repeat"]): r for r in combined}
prepared = []
for key, case in cases.items():
root = args.evaluation / key
source = root / "source.json"
saved = json.loads(source.read_text())
if saved["messages"] != case["messages"] or saved["exam"] != case["exam"] or saved["gold"] != case["gold"]:
raise ValueError("Corpus or evaluation questions changed")
generation = args.native / key / "generation"
validation = json.loads((generation / "validation.json").read_text())
native_source = (args.native_evaluation or args.evaluation) / key / "source.json"
if validation["source_sha256"] != hashlib.sha256(native_source.read_bytes()).hexdigest():
raise ValueError("Native summary source mismatch")
native_saved = json.loads(native_source.read_text())
# The format amendment changed only the held-out exam, never native generator input.
if {k: v for k, v in native_saved.items() if k != "exam"} != {k: v for k, v in saved.items() if k != "exam"}:
raise ValueError("Native source evidence changed")
summaries = json.loads((generation / "summaries.json").read_text())
if len(summaries) != 3 or {r["repeat"] for r in summaries} != set(range(3)):
raise ValueError("Native generations are incomplete")
for entry in summaries:
i = entry["repeat"]
native = index[key, "hermes_summary", i]
if not entry["valid"] or not native["valid"] or native["summary"] != entry["summary"]:
raise ValueError("Native summary is invalid or changed")
scores = {}
for path in sorted(root.glob(f"paired.c{i}.decision*.response.json")):
scores.update(json.loads(path.read_text())["scores"])
full_tokens = index[key, "full", i]["trials"][0]["usage"]["prompt_tokens"]
for n in range(1, 6):
paired = index[key, f"fusion_top{n}", i]
chosen = top_n(saved["units"], scores, min(n, len(saved["units"])))
if paired["selected"] != [u["id"] for u in chosen]:
raise ValueError("Frozen selection differs from recorded Jev ranking")
context = messages(case, compose(entry["summary"], render(chosen)))
prepared.append(
{
"case": key,
"arm": f"hermes_top{n}",
"repeat": i,
"summary": entry["summary"],
"selected": paired["selected"],
"actual_n": len(chosen),
"input_cap": paired["input_cap"],
"full_input_tokens": full_tokens,
"context": context,
"valid": False,
"trials": [],
},
)
return deepcopy(combined), prepared
async def run(args):
"""No summary or decision requests; every new trial only calls the answering model."""
from tokenizers import Tokenizer
repo = Path(__file__).resolve().parents[2]
output = args.output.resolve()
if output.exists() or output == repo or repo in output.parents:
raise ValueError("Use a new output directory outside repository")
_, cases = load_cases(args.corpus)
rows, prepared = prepare(args, cases)
tokenizer = Tokenizer.from_file(str(args.tokenizer))
values = dotenv_values(repo / ".env")
if not values.get("LLM_API_KEY") or not values.get("LLM_BASE_URL"):
raise ValueError("Missing Qwen credentials")
output.mkdir(parents=True)
for name in ("suite_fusion.py", "suite_run.py", "STUDY.md"):
shutil.copy2(Path(__file__).parent / name, output / name)
write(
output / "protocol.json",
{
"base_evaluation": str(args.evaluation.resolve()),
"combined_evaluation": str(args.combined.resolve()),
"native_generations": str(args.native.resolve()),
"native_source_evaluation": str((args.native_evaluation or args.evaluation).resolve()),
"model": "qwen3.8-max",
"methods": 13,
"tasks": 14,
"generations": 3,
"answers_per_context": 2,
"new_contexts": 210,
"new_answers": 420,
"summary_calls": 0,
"decision_calls": 0,
"acceptance": "same existing input ceilings; mark strict reduction separately without filtering tasks",
"ranking": "identical frozen Jev scores and selected originals for S and H",
"corpus_sha256": hashlib.sha256(args.corpus.read_bytes()).hexdigest(),
},
)
for row in prepared:
root = output / row["case"]
root.mkdir(exist_ok=True)
label = f"c{row['repeat']}.{row['arm']}"
write(root / f"{label}.context.json", {"messages": row.pop("context")})
rows.extend(prepared)
write(output / "frozen-selections.json", rows)
write(output / "results.json", rows)
client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1)
limit = asyncio.Semaphore(4)
work = asyncio.Semaphore(4)
async def answer(row):
async with work:
root = output / row["case"]
label = f"c{row['repeat']}.{row['arm']}"
exp = Experiment(client, None, root, tokenizer, limit, "qwen3.8-max", row["input_cap"])
try:
context = json.loads((root / f"{label}.context.json").read_text())["messages"]
trials = []
for j in range(2):
text, usage = await exp.ask(f"{label}.answer{j}", context)
prediction = json.loads(text)
trials.append(
{"answer": prediction, "usage": usage, "checks": grade(cases[row["case"]], prediction)},
)
row["trials"] = trials
row["valid"] = all(t["usage"]["prompt_tokens"] <= row["input_cap"] for t in trials)
row["shorter_than_full"] = all(t["usage"]["prompt_tokens"] < row["full_input_tokens"] for t in trials)
if not row["valid"]:
row["error"] = "input_ceiling_exceeded"
except Exception as exc:
row.update(valid=False, error=type(exc).__name__, trials=[])
write(output / "results.json", rows)
report(output, rows, cases, 2, include_native_fusion=True)
print(row["case"], label, "OK" if row["valid"] else row["error"], flush=True)
try:
async with asyncio.TaskGroup() as tasks:
for row in prepared:
tasks.create_task(answer(row))
finally:
await client.close()
load_cases(args.corpus)
costs(args, rows, cases)
for path in output.rglob("*.json"):
if values["LLM_API_KEY"].encode() in path.read_bytes():
raise ValueError("Configured API key in artifacts")
write(
output / "validation.json",
{
"contexts": len(rows),
"answers": sum(len(r["trials"]) for r in rows),
"new_valid_contexts": sum(r["valid"] for r in prepared),
"new_contexts_shorter_than_full": sum(r.get("shorter_than_full", False) for r in prepared),
"no_new_summary_or_decision_calls": True,
"no_api_keys": True,
},
)
def main():
"""Require explicit live opt-in; all artifacts must already exist except output."""
parser = argparse.ArgumentParser(description=__doc__)
for name in ("corpus", "evaluation", "combined", "native", "tokenizer", "output"):
parser.add_argument("--" + name, type=Path, required=True)
parser.add_argument("--live", action="store_true")
parser.add_argument("--native-evaluation", type=Path)
args = parser.parse_args()
if not args.live:
parser.error("Use --live to authorize the 420 answering calls")
asyncio.run(run(args))
if __name__ == "__main__":
main()

13
benchmark/compaction/suite_run.py
View file

@ -65,11 +65,14 @@ def load_cases(corpus_path):
return manifest, result
def metrics(rows, ids, repeats, include_native=False):
def metrics(rows, ids, repeats, include_native=False, include_native_fusion=False):
"""Equal task weights; invalid planned contexts remain zero instead of disappearing."""
table = []
arms = ARMS[:2] + ["hermes_summary"] + ARMS[2:] if include_native else ARMS
labels = LABELS[:2] + ["Hermes small-window adapter"] + LABELS[2:] if include_native else LABELS
if include_native_fusion:
arms = ARMS + ["hermes_summary"] + [f"hermes_top{n}" for n in range(1, 6)]
labels = LABELS + ["H (Hermes small-window)"] + [f"H + Jev Top {n}" for n in range(1, 6)]
for arm, label in zip(arms, labels):
task_facts = []
task_exact = []
@ -108,13 +111,13 @@ def metrics(rows, ids, repeats, include_native=False):
return table
def report(output, rows, cases, repeats, include_native=False):
"""Keep primary and short/query-only controls separate in compact seven-row tables."""
def report(output, rows, cases, repeats, include_native=False, include_native_fusion=False):
"""Keep primary and short/query-only controls in separate tables."""
result = {}
lines = ["# Fourteen-task fixed-summary study", ""]
for cohort in ("primary", "controls"):
ids = [k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")]
result[cohort] = metrics(rows, ids, repeats, include_native)
result[cohort] = metrics(rows, ids, repeats, include_native, include_native_fusion)
lines += [
f"## {cohort.title()} ({len(ids)} tasks)",
"",
@ -128,7 +131,7 @@ def report(output, rows, cases, repeats, include_native=False):
f"{r['valid_contexts']}/{r['planned_contexts']} |",
)
lines += [""]
result["per_task"] = {k: metrics(rows, [k], repeats, include_native) for k in cases}
result["per_task"] = {k: metrics(rows, [k], repeats, include_native, include_native_fusion) for k in cases}
write(output / "aggregate.json", result)
(output / "REPORT.md").write_text("\n".join(lines) + "\n")

33
docs/en/context_compaction.md
View file

@ -117,27 +117,22 @@ Only model requests run externally; fixture tool calls are not executed. These c
success. Evaluate real trajectories against recency and rule-based previews, measuring recovery, continuation quality,
latency and total cost alongside token reduction. ReMe does not automatically summarize or write persistent memory.
## Fixed summary + selected original evidence
## Jev across two summary strategies
The experiment uses **14 fixture queries with predefined answers**, collected through Hermes: four source-length-qualified
primary tasks and ten short/query-only controls. Rule scoring checks exact JSON fields; no LLM judge creates or grades gold.
One 13-method experiment compares Full context with a generic summary S and a forced Hermes small-window summary H,
each plus Jev Top 0–5. Top 0 means summary only. Every condition preserves its frozen summary and shares the same
Jev-selected originals across mechanisms.
![Exact-field score and downstream input for four primary tasks](../figure/compaction-topn-results.svg)
![Quality, downstream input and compression overhead](../figure/compaction-topn-results.svg)
Across the four primary tasks, Full context reaches 100%, S reaches 82.0%, and S + Top 3 reaches **99.5% with 19.3% fewer
input tokens** (3,241 versus 4,014). Top 4 and Top 5 score 98.1% and 97.4%, so more retained text is not automatically better.
Each generation freezes S and the Jev ranking before answers; all five fusion methods keep byte-identical S.
On four primary task templates, S improves from **82.0% to 99.5%** at Top 3; H improves from **79.4% to 97.9%**.
Their mean answer inputs are **3,241 / 3,193 tokens**, versus Full's **4,014 / 100%**. The figure separates downstream
input from summary/ranking overhead; fewer answer tokens do not establish lower total monetary cost.
The **Hermes small-window adapter** scores 79.4% at 2,432 input tokens. It calls the real native summary generator with
explicit small-window budget/word constraints and a body-only carrier; this is not the unchanged automatic compressor.
All 42 native contexts fit their budgets and use strictly fewer actual input tokens than Full, with no original fallback.
The panel has 14 predefined query/answer templates, 546 contexts and 1,092 direct-Qwen answers. Ten short/query-only
controls are reported separately. H uses an explicit native-generator budget adapter, not Hermes' default automatic
compressor. All H-only contexts shorten Full; at Top 3–5 only 9/12 primary contexts per method do so, with all cases
retained in scoring. This is an incremental, unequal-length comparison without agent continuation.
<!-- Payment has 10 units (maximum Top 10). Other tasks have different counts; actual N is min(N, available units). -->
Top N counts complete source units and saturates at each task's available count. The original payment has ten units,
so its maximum is Top 10; the sweep tests Top 1–5. Summary/body allowances and complete-input ceilings are task-specific;
this is an incremental-evidence curve, not an equal-length comparison or proof that every Full exceeds its ceiling.
The panel has 336 contexts and 672 answers, with the controls reported separately. Hermes collects source trajectories;
answering calls Qwen directly without tool continuation or recovery. These few synthetic tasks do not establish general
superiority or statistical equivalence. The hybrid and native adapter remain benchmark-only.
See the [exact queries, answers and settings](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/STUDY.md) and [results, controls and audit](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/RESULTS.md).
<!-- Payment has 10 units (maximum Top 10); actual N is min(N, available units). -->
See the [query/answer protocol](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md) and [complete results and token accounting](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md).

789
docs/figure/compaction-topn-results.svg
View file

@ -1,11 +1,12 @@
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1008pt" height="403.2pt" viewBox="0 0 1008 403.2" xmlns="http://www.w3.org/2000/svg" version="1.1">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1382.4pt" height="612pt" viewBox="0 0 1382.4 612" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:date>2026-10-16T17:18:34.287481</dc:date>
<dc:format>image/svg+xml</dc:format>
<dc:creator>
<cc:Agent>
@ -20,410 +21,632 @@
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 403.2
L 1008 403.2
L 1008 0
<path d="M 0 612
L 1382.4 612
L 1382.4 0
L 0 0
z
" style="fill: #ffffff"/>
</g>
<g id="axes_1">
<g id="patch_2">
<path d="M 206.64 338.688
L 556.953901 338.688
L 556.953901 64.512
L 206.64 64.512
<path d="M 76.032 434.52
L 420.832463 434.52
L 420.832463 146.88
L 76.032 146.88
z
" style="fill: #ffffff"/>
</g>
<g id="line2d_1">
<path d="M 76.032 180.72
L 420.832463 180.72
" clip-path="url(#p260a3f0845)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/>
</g>
<g id="matplotlib.axis_1">
<g id="xtick_1">
<g id="line2d_1">
<path d="M 206.64 338.688
L 206.64 64.512
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_2"/>
<g id="text_1">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="206.64" y="350.545422" transform="rotate(-0 206.64 350.545422)">0</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="94.503453" y="450.637187" transform="rotate(-0 94.503453 450.637187)">0</text>
</g>
</g>
<g id="xtick_2">
<g id="line2d_3">
<path d="M 282.795196 338.688
L 282.795196 64.512
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_4"/>
<g id="line2d_3"/>
<g id="text_2">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="282.795196" y="350.545422" transform="rotate(-0 282.795196 350.545422)">25</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="156.074965" y="450.637187" transform="rotate(-0 156.074965 450.637187)">1</text>
</g>
</g>
<g id="xtick_3">
<g id="line2d_5">
<path d="M 358.950392 338.688
L 358.950392 64.512
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_6"/>
<g id="line2d_4"/>
<g id="text_3">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="358.950392" y="350.545422" transform="rotate(-0 358.950392 350.545422)">50</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="217.646476" y="450.637187" transform="rotate(-0 217.646476 450.637187)">2</text>
</g>
</g>
<g id="xtick_4">
<g id="line2d_7">
<path d="M 435.105588 338.688
L 435.105588 64.512
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_8"/>
<g id="line2d_5"/>
<g id="text_4">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="435.105588" y="350.545422" transform="rotate(-0 435.105588 350.545422)">75</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="279.217987" y="450.637187" transform="rotate(-0 279.217987 450.637187)">3</text>
</g>
</g>
<g id="xtick_5">
<g id="line2d_9">
<path d="M 511.260784 338.688
L 511.260784 64.512
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_10"/>
<g id="line2d_6"/>
<g id="text_5">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="511.260784" y="350.545422" transform="rotate(-0 511.260784 350.545422)">100</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="340.789498" y="450.637187" transform="rotate(-0 340.789498 450.637187)">4</text>
</g>
</g>
<g id="xtick_6">
<g id="line2d_7"/>
<g id="text_6">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="402.361009" y="450.637187" transform="rotate(-0 402.361009 450.637187)">5</text>
</g>
</g>
<g id="text_7">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #43536b" x="248.432231" y="475.638125" transform="rotate(-0 248.432231 475.638125)">Jev Top N (0 = summary only)</text>
</g>
</g>
<g id="matplotlib.axis_2">
<g id="ytick_1">
<g id="line2d_11"/>
<g id="text_6">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="91.593515" transform="rotate(-0 203.14 91.593515)">Full context</text>
<g id="line2d_8">
<path d="M 76.032 434.52
L 420.832463 434.52
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_9"/>
<g id="text_8">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="439.078594" transform="rotate(-0 69.032 439.078594)">70</text>
</g>
</g>
<g id="ytick_2">
<g id="line2d_12"/>
<g id="text_7">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="124.21755" transform="rotate(-0 203.14 124.21755)">Summary S</text>
<g id="line2d_10">
<path d="M 76.032 392.22
L 420.832463 392.22
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_11"/>
<g id="text_9">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="396.778594" transform="rotate(-0 69.032 396.778594)">75</text>
</g>
</g>
<g id="ytick_3">
<g id="line2d_12">
<path d="M 76.032 349.92
L 420.832463 349.92
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_13"/>
<g id="text_8">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="156.842444" transform="rotate(-0 203.14 156.842444)">Hermes small-window</text>
<g id="text_10">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="354.478594" transform="rotate(-0 69.032 354.478594)">80</text>
</g>
</g>
<g id="ytick_4">
<g id="line2d_14"/>
<g id="text_9">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="189.466479" transform="rotate(-0 203.14 189.466479)">S + Jev Top 1</text>
<g id="line2d_14">
<path d="M 76.032 307.62
L 420.832463 307.62
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_15"/>
<g id="text_11">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="312.178594" transform="rotate(-0 69.032 312.178594)">85</text>
</g>
</g>
<g id="ytick_5">
<g id="line2d_15"/>
<g id="text_10">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="222.090943" transform="rotate(-0 203.14 222.090943)">S + Jev Top 2</text>
<g id="line2d_16">
<path d="M 76.032 265.32
L 420.832463 265.32
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_17"/>
<g id="text_12">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="269.878594" transform="rotate(-0 69.032 269.878594)">90</text>
</g>
</g>
<g id="ytick_6">
<g id="line2d_16"/>
<g id="text_11">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="254.715408" transform="rotate(-0 203.14 254.715408)">S + Jev Top 3</text>
<g id="line2d_18">
<path d="M 76.032 223.02
L 420.832463 223.02
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_19"/>
<g id="text_13">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="227.578594" transform="rotate(-0 69.032 227.578594)">95</text>
</g>
</g>
<g id="ytick_7">
<g id="line2d_17"/>
<g id="text_12">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="287.339872" transform="rotate(-0 203.14 287.339872)">S + Jev Top 4</text>
<g id="line2d_20">
<path d="M 76.032 180.72
L 420.832463 180.72
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_21"/>
<g id="text_14">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="69.032" y="185.278594" transform="rotate(-0 69.032 185.278594)">100</text>
</g>
</g>
<g id="ytick_8">
<g id="line2d_18"/>
<g id="text_13">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="319.964337" transform="rotate(-0 203.14 319.964337)">S + Jev Top 5</text>
</g>
<g id="text_15">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #43536b" x="33.244188" y="290.7" transform="rotate(-90 33.244188 290.7)">Exact-field score (%)</text>
</g>
</g>
<g id="patch_3">
<path d="M 206.64 76.974545
L 511.260784 76.974545
L 511.260784 97.854203
L 206.64 97.854203
<path d="M 76.032 434.52
L 420.832463 434.52
" style="fill: none; stroke: #dbe2ea; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="line2d_22">
<path d="M 94.503453 332.8825
L 156.074965 324.95125
L 217.646476 227.42625
L 279.217987 185.12625
L 340.789498 196.87625
L 402.361009 202.75125
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/>
<defs>
<path id="m90dc9a1c0b" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" clip-path="url(#pa8067ffbdb)" style="fill: #718096"/>
</g>
<g id="patch_4">
<path d="M 206.64 109.59901
L 456.471351 109.59901
L 456.471351 130.478667
L 206.64 130.478667
z
" clip-path="url(#pa8067ffbdb)" style="fill: #e5a442"/>
</g>
<g id="patch_5">
<path d="M 206.64 142.223475
L 448.432747 142.223475
L 448.432747 163.103132
L 206.64 163.103132
z
" clip-path="url(#pa8067ffbdb)" style="fill: #9674ba"/>
</g>
<g id="patch_6">
<path d="M 206.64 174.847939
L 459.327171 174.847939
L 459.327171 195.727596
L 206.64 195.727596
z
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
</g>
<g id="patch_7">
<path d="M 206.64 207.472404
L 494.443178 207.472404
L 494.443178 228.352061
L 206.64 228.352061
z
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
</g>
<g id="patch_8">
<path d="M 206.64 240.096868
L 509.674217 240.096868
L 509.674217 260.976525
L 206.64 260.976525
z
" clip-path="url(#pa8067ffbdb)" style="fill: #20917d"/>
</g>
<g id="patch_9">
<path d="M 206.64 272.721333
L 505.443373 272.721333
L 505.443373 293.60099
L 206.64 293.60099
z
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
</g>
<g id="patch_10">
<path d="M 206.64 305.345797
L 503.327951 305.345797
L 503.327951 326.225455
L 206.64 326.225455
z
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
</g>
<g id="text_14">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="516.515492" y="90.01203" transform="rotate(-0 516.515492 90.01203)">100.0%</text>
</g>
<g id="text_15">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="461.72606" y="122.636495" transform="rotate(-0 461.72606 122.636495)">82.0%</text>
" style="stroke: #dc8f20"/>
</defs>
<g clip-path="url(#p260a3f0845)">
<use xlink:href="#m90dc9a1c0b" x="94.503453" y="332.8825" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="156.074965" y="324.95125" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="217.646476" y="227.42625" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="279.217987" y="185.12625" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="340.789498" y="196.87625" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="402.361009" y="202.75125" style="fill: #dc8f20; stroke: #dc8f20"/>
</g>
</g>
<g id="text_16">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="453.687456" y="155.260959" transform="rotate(-0 453.687456 155.260959)">79.4%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="94.503453" y="322.8825" transform="rotate(-0 94.503453 322.8825)">82.0%</text>
</g>
<g id="text_17">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="464.58188" y="187.885424" transform="rotate(-0 464.58188 187.885424)">83.0%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="279.217987" y="175.12625" transform="rotate(-0 279.217987 175.12625)">99.5%</text>
</g>
<g id="text_18">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="499.697887" y="220.509889" transform="rotate(-0 499.697887 220.509889)">94.5%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="402.361009" y="192.75125" transform="rotate(-0 402.361009 192.75125)">97.4%</text>
</g>
<g id="line2d_23">
<path d="M 94.503453 355.2075
L 156.074965 304.97625
L 217.646476 240.645
L 279.217987 198.345
L 340.789498 207.1575
L 402.361009 207.1575
" clip-path="url(#p260a3f0845)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/>
<defs>
<path id="m712a07e464" d="M 0 3
C 0.795609 3 1.55874 2.683901 2.12132 2.12132
C 2.683901 1.55874 3 0.795609 3 0
C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132
C 1.55874 -2.683901 0.795609 -3 0 -3
C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132
C -2.683901 -1.55874 -3 -0.795609 -3 0
C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132
C -1.55874 2.683901 -0.795609 3 0 3
z
" style="stroke: #596fc4"/>
</defs>
<g clip-path="url(#p260a3f0845)">
<use xlink:href="#m712a07e464" x="94.503453" y="355.2075" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="156.074965" y="304.97625" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="217.646476" y="240.645" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="279.217987" y="198.345" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="340.789498" y="207.1575" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="402.361009" y="207.1575" style="fill: #596fc4; stroke: #596fc4"/>
</g>
</g>
<g id="text_19">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="514.928926" y="253.134353" transform="rotate(-0 514.928926 253.134353)">99.5%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="94.503453" y="373.2075" transform="rotate(-0 94.503453 373.2075)">79.4%</text>
</g>
<g id="text_20">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="510.698081" y="285.758818" transform="rotate(-0 510.698081 285.758818)">98.1%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="279.217987" y="216.345" transform="rotate(-0 279.217987 216.345)">97.9%</text>
</g>
<g id="text_21">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="508.582659" y="318.383282" transform="rotate(-0 508.582659 318.383282)">97.4%</text>
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="402.361009" y="225.1575" transform="rotate(-0 402.361009 225.1575)">96.9%</text>
</g>
<g id="text_22">
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="206.64" y="52.512" transform="rotate(-0 206.64 52.512)">Exact-field score (%)</text>
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #7b8798" x="413.936453" y="163.867481" transform="rotate(-0 413.936453 163.867481)">Full: 100%</text>
</g>
<g id="text_23">
<text style="font-weight: 700; font-size: 15px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="76.032" y="123.88" transform="rotate(-0 76.032 123.88)">A Answer quality</text>
</g>
</g>
<g id="axes_2">
<g id="patch_11">
<path d="M 637.526099 338.688
L 987.84 338.688
L 987.84 64.512
L 637.526099 64.512
<g id="patch_4">
<path d="M 532.639759 434.52
L 877.440222 434.52
L 877.440222 146.88
L 532.639759 146.88
z
" style="fill: #ffffff"/>
</g>
<g id="line2d_24">
<path d="M 532.639759 195.184761
L 877.440222 195.184761
" clip-path="url(#pde7327f20e)" style="fill: none; stroke-dasharray: 7.5,6; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/>
</g>
<g id="matplotlib.axis_3">
<g id="xtick_6">
<g id="line2d_19">
<path d="M 637.526099 338.688
L 637.526099 64.512
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_20"/>
<g id="text_23">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="637.526099" y="350.545422" transform="rotate(-0 637.526099 350.545422)">0</text>
</g>
</g>
<g id="xtick_7">
<g id="line2d_21">
<path d="M 749.328408 338.688
L 749.328408 64.512
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_22"/>
<g id="line2d_25"/>
<g id="text_24">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="749.328408" y="350.545422" transform="rotate(-0 749.328408 350.545422)">1500</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="551.111213" y="450.637187" transform="rotate(-0 551.111213 450.637187)">0</text>
</g>
</g>
<g id="xtick_8">
<g id="line2d_23">
<path d="M 861.130717 338.688
L 861.130717 64.512
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_24"/>
<g id="line2d_26"/>
<g id="text_25">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="861.130717" y="350.545422" transform="rotate(-0 861.130717 350.545422)">3000</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="612.682724" y="450.637187" transform="rotate(-0 612.682724 450.637187)">1</text>
</g>
</g>
<g id="xtick_9">
<g id="line2d_25">
<path d="M 972.933025 338.688
L 972.933025 64.512
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
</g>
<g id="line2d_26"/>
<g id="line2d_27"/>
<g id="text_26">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="972.933025" y="350.545422" transform="rotate(-0 972.933025 350.545422)">4500</text>
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="674.254235" y="450.637187" transform="rotate(-0 674.254235 450.637187)">2</text>
</g>
</g>
<g id="xtick_10">
<g id="line2d_28"/>
<g id="text_27">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="735.825746" y="450.637187" transform="rotate(-0 735.825746 450.637187)">3</text>
</g>
</g>
<g id="xtick_11">
<g id="line2d_29"/>
<g id="text_28">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="797.397258" y="450.637187" transform="rotate(-0 797.397258 450.637187)">4</text>
</g>
</g>
<g id="xtick_12">
<g id="line2d_30"/>
<g id="text_29">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="858.968769" y="450.637187" transform="rotate(-0 858.968769 450.637187)">5</text>
</g>
</g>
<g id="text_30">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #43536b" x="705.039991" y="475.638125" transform="rotate(-0 705.039991 475.638125)">Jev Top N (0 = summary only)</text>
</g>
</g>
<g id="matplotlib.axis_4">
<g id="ytick_8">
<g id="line2d_31">
<path d="M 532.639759 384.495652
L 877.440222 384.495652
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_32"/>
<g id="text_31">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="525.639759" y="389.054246" transform="rotate(-0 525.639759 389.054246)">2500</text>
</g>
</g>
<g id="ytick_9">
<g id="line2d_27"/>
<g id="line2d_33">
<path d="M 532.639759 321.965217
L 877.440222 321.965217
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_34"/>
<g id="text_32">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="525.639759" y="326.523811" transform="rotate(-0 525.639759 326.523811)">3000</text>
</g>
</g>
<g id="ytick_10">
<g id="line2d_28"/>
<g id="line2d_35">
<path d="M 532.639759 259.434783
L 877.440222 259.434783
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_36"/>
<g id="text_33">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="525.639759" y="263.993376" transform="rotate(-0 525.639759 263.993376)">3500</text>
</g>
</g>
<g id="ytick_11">
<g id="line2d_29"/>
<g id="line2d_37">
<path d="M 532.639759 196.904348
L 877.440222 196.904348
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_38"/>
<g id="text_34">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #58677b" x="525.639759" y="201.462942" transform="rotate(-0 525.639759 201.462942)">4000</text>
</g>
</g>
<g id="ytick_12">
<g id="line2d_30"/>
<g id="text_35">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #43536b" x="482.216947" y="290.7" transform="rotate(-90 482.216947 290.7)">Mean answer input tokens</text>
</g>
<g id="ytick_13">
<g id="line2d_31"/>
</g>
<g id="patch_5">
<path d="M 532.639759 434.52
L 877.440222 434.52
" style="fill: none; stroke: #dbe2ea; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="line2d_39">
<path d="M 551.111213 387.007291
L 612.682724 352.896939
L 674.254235 325.195957
L 735.825746 291.856813
L 797.397258 259.935026
L 858.968769 239.393778
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #dc8f20; stroke-width: 2.6; stroke-linecap: square"/>
<g clip-path="url(#pde7327f20e)">
<use xlink:href="#m90dc9a1c0b" x="551.111213" y="387.007291" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="612.682724" y="352.896939" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="674.254235" y="325.195957" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="735.825746" y="291.856813" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="797.397258" y="259.935026" style="fill: #dc8f20; stroke: #dc8f20"/>
<use xlink:href="#m90dc9a1c0b" x="858.968769" y="239.393778" style="fill: #dc8f20; stroke: #dc8f20"/>
</g>
<g id="ytick_14">
<g id="line2d_32"/>
</g>
<g id="text_36">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="551.111213" y="377.007291" transform="rotate(-0 551.111213 377.007291)">2,480</text>
</g>
<g id="text_37">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="735.825746" y="281.856813" transform="rotate(-0 735.825746 281.856813)">3,241</text>
</g>
<g id="text_38">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #dc8f20" x="858.968769" y="229.393778" transform="rotate(-0 858.968769 229.393778)">3,660</text>
</g>
<g id="line2d_40">
<path d="M 551.111213 392.947683
L 612.682724 358.83733
L 674.254235 331.136348
L 735.825746 297.797204
L 797.397258 265.875417
L 858.968769 245.33417
" clip-path="url(#pde7327f20e)" style="fill: none; stroke: #596fc4; stroke-width: 2.6; stroke-linecap: square"/>
<g clip-path="url(#pde7327f20e)">
<use xlink:href="#m712a07e464" x="551.111213" y="392.947683" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="612.682724" y="358.83733" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="674.254235" y="331.136348" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="735.825746" y="297.797204" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="797.397258" y="265.875417" style="fill: #596fc4; stroke: #596fc4"/>
<use xlink:href="#m712a07e464" x="858.968769" y="245.33417" style="fill: #596fc4; stroke: #596fc4"/>
</g>
<g id="ytick_15">
<g id="line2d_33"/>
</g>
<g id="text_39">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="551.111213" y="410.947683" transform="rotate(-0 551.111213 410.947683)">2,432</text>
</g>
<g id="text_40">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="735.825746" y="315.797204" transform="rotate(-0 735.825746 315.797204)">3,193</text>
</g>
<g id="text_41">
<text style="font-weight: 700; font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #596fc4" x="858.968769" y="263.33417" transform="rotate(-0 858.968769 263.33417)">3,613</text>
</g>
<g id="text_42">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #7b8798" x="870.544213" y="163.867481" transform="rotate(-0 870.544213 163.867481)">Full: 4,014</text>
</g>
<g id="text_43">
<text style="font-weight: 700; font-size: 15px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="532.639759" y="123.88" transform="rotate(-0 532.639759 123.88)">B Downstream context</text>
</g>
</g>
<g id="axes_3">
<g id="patch_6">
<path d="M 989.247519 434.52
L 1347.84 434.52
L 1347.84 146.88
L 989.247519 146.88
z
" style="fill: #ffffff"/>
</g>
<g id="matplotlib.axis_5">
<g id="xtick_13">
<g id="line2d_41">
<path d="M 989.247519 434.52
L 989.247519 146.88
" clip-path="url(#p7ebe458925)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_42"/>
<g id="text_44">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="989.247519" y="450.637187" transform="rotate(-0 989.247519 450.637187)">0</text>
</g>
</g>
<g id="ytick_16">
<g id="line2d_34"/>
<g id="xtick_14">
<g id="line2d_43">
<path d="M 1083.613961 434.52
L 1083.613961 146.88
" clip-path="url(#p7ebe458925)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_44"/>
<g id="text_45">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="1083.613961" y="450.638125" transform="rotate(-0 1083.613961 450.638125)">2k</text>
</g>
</g>
<g id="xtick_15">
<g id="line2d_45">
<path d="M 1177.980404 434.52
L 1177.980404 146.88
" clip-path="url(#p7ebe458925)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_46"/>
<g id="text_46">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="1177.980404" y="450.638125" transform="rotate(-0 1177.980404 450.638125)">4k</text>
</g>
</g>
<g id="xtick_16">
<g id="line2d_47">
<path d="M 1272.346846 434.52
L 1272.346846 146.88
" clip-path="url(#p7ebe458925)" style="fill: none; stroke: #e8edf3; stroke-width: 0.9; stroke-linecap: square"/>
</g>
<g id="line2d_48"/>
<g id="text_47">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #58677b" x="1272.346846" y="450.638125" transform="rotate(-0 1272.346846 450.638125)">6k</text>
</g>
</g>
<g id="text_48">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #43536b" x="1168.543759" y="475.639063" transform="rotate(-0 1168.543759 475.639063)">Mean provider tokens per generation</text>
</g>
</g>
<g id="matplotlib.axis_6"/>
<g id="patch_7">
<path d="M 989.247519 434.52
L 1347.84 434.52
" style="fill: none; stroke: #dbe2ea; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square"/>
</g>
<g id="patch_8">
<path d="M 989.247519 193.569391
L 1243.046066 193.569391
L 1243.046066 233.58887
L 989.247519 233.58887
z
" clip-path="url(#p7ebe458925)" style="fill: #dc8f20"/>
</g>
<g id="patch_9">
<path d="M 1243.046066 193.569391
L 1252.466982 193.569391
L 1252.466982 233.58887
L 1243.046066 233.58887
z
" clip-path="url(#p7ebe458925)" style="fill: #dc8f20; opacity: 0.3"/>
</g>
<g id="text_49">
<text style="font-weight: 700; font-size: 11.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="989.247519" y="182.469305" transform="rotate(-0 989.247519 182.469305)">S generation</text>
</g>
<g id="text_50">
<text style="font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #43536b" x="1256.949388" y="216.306669" transform="rotate(-0 1256.949388 216.306669)">5,379 / 200</text>
</g>
<g id="patch_10">
<path d="M 989.247519 276.943304
L 1136.470965 276.943304
L 1136.470965 316.962783
L 989.247519 316.962783
z
" clip-path="url(#p7ebe458925)" style="fill: #596fc4"/>
</g>
<g id="patch_11">
<path d="M 1136.470965 276.943304
L 1143.650678 276.943304
L 1143.650678 316.962783
L 1136.470965 316.962783
z
" clip-path="url(#p7ebe458925)" style="fill: #596fc4; opacity: 0.3"/>
</g>
<g id="text_51">
<text style="font-weight: 700; font-size: 11.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="989.247519" y="265.843218" transform="rotate(-0 989.247519 265.843218)">H generation</text>
</g>
<g id="text_52">
<text style="font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #43536b" x="1148.133084" y="299.680583" transform="rotate(-0 1148.133084 299.680583)">3,120 / 152</text>
</g>
<g id="patch_12">
<path d="M 637.526099 76.974545
L 936.690444 76.974545
L 936.690444 97.854203
L 637.526099 97.854203
<path d="M 989.247519 360.317217
L 1185.61229 360.317217
L 1185.61229 400.336696
L 989.247519 400.336696
z
" clip-path="url(#p012de37c52)" style="fill: #718096"/>
" clip-path="url(#p7ebe458925)" style="fill: #1a9985"/>
</g>
<g id="patch_13">
<path d="M 637.526099 109.59901
L 822.366372 109.59901
L 822.366372 130.478667
L 637.526099 130.478667
<path d="M 1185.61229 360.317217
L 1201.855114 360.317217
L 1201.855114 400.336696
L 1185.61229 400.336696
z
" clip-path="url(#p012de37c52)" style="fill: #e5a442"/>
" clip-path="url(#p7ebe458925)" style="fill: #1a9985; opacity: 0.3"/>
</g>
<g id="patch_14">
<path d="M 637.526099 142.223475
L 818.825965 142.223475
L 818.825965 163.103132
L 637.526099 163.103132
<g id="text_53">
<text style="font-weight: 700; font-size: 11.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="989.247519" y="349.217131" transform="rotate(-0 989.247519 349.217131)">Jev ranking</text>
</g>
<g id="text_54">
<text style="font-size: 10.5px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #43536b" x="1206.33752" y="383.054496" transform="rotate(-0 1206.33752 383.054496)">4,162 / 344</text>
</g>
<g id="text_55">
<text style="font-weight: 700; font-size: 15px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="989.247519" y="123.88" transform="rotate(-0 989.247519 123.88)">C Compression overhead</text>
</g>
<g id="legend_1">
<g id="patch_14">
<path d="M 1202.210312 492.151256
L 1222.210312 492.151256
L 1222.210312 485.151256
L 1202.210312 485.151256
z
" clip-path="url(#p012de37c52)" style="fill: #9674ba"/>
</g>
<g id="patch_15">
<path d="M 637.526099 174.847939
L 842.695758 174.847939
L 842.695758 195.727596
L 637.526099 195.727596
" style="fill: #65758b"/>
</g>
<g id="text_56">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="1230.210312" y="492.151256" transform="rotate(-0 1230.210312 492.151256)">Input</text>
</g>
<g id="patch_15">
<path d="M 1276.104062 492.151256
L 1296.104062 492.151256
L 1296.104062 485.151256
L 1276.104062 485.151256
z
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
</g>
<g id="patch_16">
<path d="M 637.526099 207.472404
L 859.205232 207.472404
L 859.205232 228.352061
L 637.526099 228.352061
z
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
</g>
<g id="patch_17">
<path d="M 637.526099 240.096868
L 879.074987 240.096868
L 879.074987 260.976525
L 637.526099 260.976525
z
" clip-path="url(#p012de37c52)" style="fill: #20917d"/>
</g>
<g id="patch_18">
<path d="M 637.526099 272.721333
L 898.100013 272.721333
L 898.100013 293.60099
L 637.526099 293.60099
z
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
</g>
<g id="patch_19">
<path d="M 637.526099 305.345797
L 910.342366 305.345797
L 910.342366 326.225455
L 637.526099 326.225455
z
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
</g>
<g id="text_27">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="941.945152" y="90.01203" transform="rotate(-0 941.945152 90.01203)">4,014</text>
</g>
<g id="text_28">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="827.62108" y="122.636495" transform="rotate(-0 827.62108 122.636495)">2,480</text>
</g>
<g id="text_29">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="824.080674" y="155.260959" transform="rotate(-0 824.080674 155.260959)">2,432</text>
</g>
<g id="text_30">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="847.950467" y="187.885424" transform="rotate(-0 847.950467 187.885424)">2,753</text>
</g>
<g id="text_31">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="864.459941" y="220.509889" transform="rotate(-0 864.459941 220.509889)">2,974</text>
</g>
<g id="text_32">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="884.329696" y="253.134353" transform="rotate(-0 884.329696 253.134353)">3,241</text>
</g>
<g id="text_33">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="903.354722" y="285.758818" transform="rotate(-0 903.354722 285.758818)">3,496</text>
</g>
<g id="text_34">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="915.597075" y="318.383282" transform="rotate(-0 915.597075 318.383282)">3,660</text>
</g>
<g id="text_35">
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="637.526099" y="52.512" transform="rotate(-0 637.526099 52.512)">Mean downstream input tokens</text>
" style="fill: #65758b; opacity: 0.3"/>
</g>
<g id="text_57">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="1304.104062" y="492.151256" transform="rotate(-0 1304.104062 492.151256)">Output</text>
</g>
</g>
</g>
<g id="text_36">
<text style="font-weight: 700; font-size: 18px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #1a2f45" x="40.32" y="21.741188" transform="rotate(-0 40.32 21.741188)">Fixed summary + selected original evidence</text>
<g id="text_58">
<text style="font-weight: 700; font-size: 25px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="76.032" y="36.72" transform="rotate(-0 76.032 36.72)">Jev across two summary strategies</text>
</g>
<g id="text_37">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #45586b" x="40.32" y="379.008" transform="rotate(-0 40.32 379.008)">Primary cohort: 4 task templates · 24 answers per method · 14-query panel includes 10 separate controls</text>
<g id="text_59">
<text style="font-size: 14px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #58677b" x="76.032" y="70.38" transform="rotate(-0 76.032 70.38)">Same source trajectories. Frozen summaries. Identical Jev selections.</text>
</g>
<g id="text_38">
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #45586b" x="40.32" y="393.12" transform="rotate(-0 40.32 393.12)">Hermes uses an explicit small-window adapter. Evaluation calls Qwen directly; no LLM judge.</text>
<g id="text_60">
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="76.032" y="514.08" transform="rotate(-0 76.032 514.08)">At Top 3</text>
</g>
<g id="text_61">
<text style="font-size: 13px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="186.624" y="514.08" transform="rotate(-0 186.624 514.08)">S: 99.5% | 3,241 input tokens H: 97.9% | 3,193 input tokens</text>
</g>
<g id="text_62">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #58677b" x="76.032" y="547.74" transform="rotate(-0 76.032 547.74)">Primary cohort: 4 task templates, 24 answers per method. Another 10 short/query-only controls are reported separately.</text>
</g>
<g id="text_63">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #58677b" x="76.032" y="568.548" transform="rotate(-0 76.032 568.548)">Overhead includes recorded retries; one summary + one Jev ranking per standalone fusion run. Jev cost is unchanged for Top 1–5.</text>
</g>
<g id="text_64">
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #58677b" x="76.032" y="589.356" transform="rotate(-0 76.032 589.356)">Tokens are not currency. Top 3–5 shorten 9/12 primary contexts per method; all cases remain scored. Direct Qwen evaluation; no agent continuation.</text>
</g>
<g id="legend_2">
<g id="line2d_49">
<path d="M 78.5376 97.114125
L 90.5376 97.114125
L 102.5376 97.114125
" style="fill: none; stroke: #dc8f20; stroke-width: 2.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m90dc9a1c0b" x="90.5376" y="97.114125" style="fill: #dc8f20; stroke: #dc8f20"/>
</g>
</g>
<g id="text_65">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="112.1376" y="101.314125" transform="rotate(-0 112.1376 101.314125)">Generic summary S</text>
</g>
<g id="line2d_50">
<path d="M 254.541975 97.114125
L 266.541975 97.114125
L 278.541975 97.114125
" style="fill: none; stroke: #596fc4; stroke-width: 2.5; stroke-linecap: square"/>
<g>
<use xlink:href="#m712a07e464" x="266.541975" y="97.114125" style="fill: #596fc4; stroke: #596fc4"/>
</g>
</g>
<g id="text_66">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="288.141975" y="101.314125" transform="rotate(-0 288.141975 101.314125)">Hermes small-window H</text>
</g>
<g id="line2d_51">
<path d="M 457.06635 97.114125
L 469.06635 97.114125
L 481.06635 97.114125
" style="fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #7b8798; stroke-width: 1.5"/>
</g>
<g id="text_67">
<text style="font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #172d45" x="490.66635" y="101.314125" transform="rotate(-0 490.66635 101.314125)">Full context</text>
</g>
</g>
</g>
<defs>
<clipPath id="pa8067ffbdb">
<rect x="206.64" y="64.512" width="350.313901" height="274.176"/>
<clipPath id="p260a3f0845">
<rect x="76.032" y="146.88" width="344.800463" height="287.64"/>
</clipPath>
<clipPath id="p012de37c52">
<rect x="637.526099" y="64.512" width="350.313901" height="274.176"/>
<clipPath id="pde7327f20e">
<rect x="532.639759" y="146.88" width="344.800463" height="287.64"/>
</clipPath>
<clipPath id="p7ebe458925">
<rect x="989.247519" y="146.88" width="358.592481" height="287.64"/>
</clipPath>
</defs>
</svg>

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 30 KiB

29
docs/zh/context_compaction.md
View file

@ -108,25 +108,20 @@ REME_RUN_DECISION_INTEGRATION=1 pytest tests/integration/test_decision_compactio
真实 trajectory 评测应对比近期保留和纯规则预览，同时测量恢复次数、继续执行的质量、时延、总成本和 token 节省。
ReMe 不会自动生成摘要或写入长期记忆。
## 固定摘要与增量原文实验
## 两种摘要机制下的 Jev 增益
实验固定 **14 个有预设答案的 fixture query**，通过 Hermes 采集轨迹：4 个按来源长度筛出的主实验任务，
以及 10 个短任务或 query-only 对照。使用确定性 JSON 字段规则评分，不用 LLM 生成标准答案或担任 judge。
只保留一个 13 组实验：Full context，以及普通摘要 S、Hermes 小窗口摘要 H 各自追加 Jev Top 0～5。
Top 0 表示纯摘要；每轮摘要逐字固定，两种机制使用完全相同的 Jev 排名与原文块。
![Exact-field score and downstream input for four primary tasks](../figure/compaction-topn-results.svg)
![Quality, downstream input and compression overhead](../figure/compaction-topn-results.svg)
4 个主实验任务中，Full context 为 100%，S 为 82.0%，S + Top 3 达到 **99.5%**，平均输入从 4014 降至
3241 tokens，减少 **19.3%**。Top 4、Top 5 分别为 98.1%、97.4%，追加更多原文没有继续提分。
每轮在评分前固定 S 和 Jev 排名，五种融合始终保留逐字相同的 S。
4 个主任务中，Top 3 将 S 从 **82.0% 提升至 99.5%**，将 H 从 **79.4% 提升至 97.9%**。
平均回答输入分别为 **3241 / 3193 tokens**，Full 为 **4014 tokens / 100%**。
图中分别展示效果、回答输入和摘要／排序开销；回答输入减少不等于总费用降低。
新增的 **Hermes 小窗口摘要适配器** 得分为 79.4%，平均输入 2432 tokens。它实际调用原生摘要生成器，
但显式增加小窗口预算与字数约束，并使用摘要正文统一封装；不是默认自动压缩器。
42/42 个上下文均满足预算，且实际 API 输入严格短于对应 Full，没有用原文回退充当压缩成功。
共 14 个预设 query/answer 模板、546 个上下文、1092 次直接 Qwen 回答；10 个短任务对照单独报告。
H 是显式调整预算的原生生成器适配方案，不是 Hermes 默认自动压缩器。纯 H 的 42 个上下文全部短于 Full；
Top 3～5 每组主任务只有 9/12 个上下文实现缩短，全部样例均保留评分。这是非等长的增量比较，未继续运行 agent。
<!-- 支付轨迹有 10 个证据块，最多 Top 10；其他任务数量不同，实际 N 为 min(N, 可用证据块数)。 -->
Top N 按完整证据块计数，超过可用数量时保留全部。原支付轨迹共有 10 块，最多 Top 10；本次测试 Top 1～5。
摘要正文预算和完整输入上限按任务计算；这是逐步追加证据的曲线，不是等长比较，也不代表所有 Full 都超过上限。
共 336 个上下文、672 次回答，短任务对照单独报告。Hermes 负责采集，评分时直接调用 Qwen，没有继续工具循环或
恢复原文。少量合成任务不能证明通用优势或统计等价；融合和原生摘要适配器均仅用于 benchmark。
详见[完整 query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/STUDY.md)和[结果、对照与审计](https://github.com/jinliyl/ReMe/blob/11317634e5e255a7c7b13492f048f8ca0c90ab3a/benchmark/compaction/RESULTS.md)。
<!-- 支付轨迹有 10 个证据块，最多 Top 10；实际 N 为 min(N, 可用证据块数)。 -->
详见[query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md)及[完整结果与 token 开销](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md)。

67
tests/unit/test_compaction_study.py
View file

@ -2,11 +2,13 @@
import hashlib
import json
from argparse import Namespace
import pytest
from benchmark.compaction.cases import event_units, write
from benchmark.compaction.suite_native import validate_base
from benchmark.compaction.suite_fusion import prepare
from benchmark.compaction.suite import context, define, exam, grade
from benchmark.compaction.suite_run import ARMS, load_cases, metrics
@ -149,3 +151,68 @@ def test_native_rejects_incompatible_or_duplicate_base_panels(tmp_path):
write(tmp_path / "protocol.json", protocol)
with pytest.raises(ValueError, match="matching fourteen-task"):
validate_base(tmp_path, corpus, cases)
def test_hermes_fusion_reuses_summary_and_verified_jev_prefix(tmp_path, monkeypatch):
"""Appending evidence cannot rewrite the frozen native summary or rerank units."""
case = {
"messages": [{"role": "system", "content": "system"}, {"role": "user", "content": "task"}],
"exam": "held-out",
"gold": {"answer": 42},
}
units = [{"id": "u", "file": "source", "text": "evidence", "spans": []}]
base = []
native = []
for i in range(3):
for arm in ARMS:
base.append(
{
"case": "example",
"arm": arm,
"repeat": i,
"summary": "S",
"selected": ["u"] if arm.startswith("fusion") else [],
"input_cap": 1000,
"trials": [{"usage": {"prompt_tokens": 200}}],
},
)
native.append(
{
"case": "example",
"arm": "hermes_summary",
"repeat": i,
"summary": f"frozen H {i}",
"valid": True,
},
)
evaluation = tmp_path / "evaluation"
generation = tmp_path / "native" / "example" / "generation"
source = evaluation / "example" / "source.json"
source.parent.mkdir(parents=True)
generation.mkdir(parents=True)
write(source, {**case, "units": units})
for i in range(3):
write(source.parent / f"paired.c{i}.decision0.response.json", {"scores": {"u": 0.9}})
write(generation / "validation.json", {"source_sha256": hashlib.sha256(source.read_bytes()).hexdigest()})
write(generation / "summaries.json", [{"repeat": i, "summary": f"frozen H {i}", "valid": True} for i in range(3)])
combined = tmp_path / "combined"
combined.mkdir()
write(combined / "results.json", base + native)
monkeypatch.setattr("benchmark.compaction.suite_fusion.validate_base", lambda *_: base)
args = Namespace(
evaluation=evaluation,
native=tmp_path / "native",
combined=combined,
corpus=tmp_path / "corpus.json",
native_evaluation=None,
)
reused, prepared = prepare(args, {"example": case})
assert reused == base + native and len(prepared) == 15
for row in prepared:
assert row["summary"] == f"frozen H {row['repeat']}"
assert row["selected"] == ["u"] and row["actual_n"] == 1
assert row["context"][-1]["content"] == "held-out"
assert "evidence" in row["context"][-2]["content"]
write(source.parent / "paired.c0.decision0.response.json", {"scores": {"wrong": 0.9}})
with pytest.raises(ValueError, match="every unit needs a score"):
prepare(args, {"example": case})