mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-23 05:35:30 +00:00
bench: evaluate fixed-summary fusion across Hermes fixture tasks
This commit is contained in:
jinli.yl
parent
9ca4659e34
commit
11317634e5
19 changed files with 2115 additions and 874 deletions
|
|
@ -53,7 +53,7 @@ users retain control of the durable files.
|
|||
|
||||
- [2026.10] - **[Decision-model context compaction](docs/en/context_compaction.md)**: score old tool calls and results
|
||||
against the next action, preserve protected history, and return a recoverable compaction proposal. Includes an SVG
|
||||
walkthrough, two live-model integration scenarios, and a fixed-summary Top-N benchmark.
|
||||
walkthrough, two live-model integration scenarios, and a 14-query fixed-summary Top-N benchmark with a forced Hermes summary baseline.
|
||||
- [2026.10] - **[ReMe Multimodal Memory](https://reme.agentscope.io/en/blog_20261009) update**: Auto Memory reads images
|
||||
with their conversation context and preserves image sources; Auto Resource turns image files into searchable daily
|
||||
cards linked to the originals.
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@
|
|||
<!-- 最新动态最多展开 5 条；新增条目时，将超出的旧条目移入下方“更多更新”折叠区。 -->
|
||||
|
||||
- [2026.10] - **[基于决策模型的上下文压缩](docs/zh/context_compaction.md)**：根据下一步任务判断旧工具证据的保留价值，
|
||||
保护关键历史，并返回可回溯的压缩建议；提供 SVG 流程图、两个真实模型集成场景和固定摘要 Top-N 实验。
|
||||
保护关键历史，并返回可回溯的压缩建议；提供 SVG 流程图、两个真实模型集成场景和 14-query 固定摘要实验，包含强制 Hermes 摘要基线。
|
||||
- [2026.10] - **[ReMe 多模态记忆更新](https://reme.agentscope.io/zh/blog_20261009)**：Auto Memory 将会话图片和上下文一起
|
||||
写入记忆并保留图片来源；Auto Resource 将图片资料整理为可检索、可回溯原图的 daily 卡片。
|
||||
- [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=zh) 上线**：无需安装或启动后端，
|
||||
|
|
|
|||
|
|
@ -1,56 +1,57 @@
|
|||
# Frozen summary + Jev Top N
|
||||
# Context compaction: fixed summary + selected evidence
|
||||
|
||||
One payment trajectory, one fixed-summary experiment, one [results table](./RESULTS.md).
|
||||
The question is whether adding selected original evidence repairs information lost by a summary.
|
||||
This benchmark is separate from production `compact_context`.
|
||||
A **14-query fixture panel** with predefined answers and rule scoring: four primary tasks and ten short/query-only
|
||||
controls. See [queries, answers and frozen settings](./STUDY.md) and [measured results](./RESULTS.md).
|
||||
|
||||
## Protocol
|
||||
|
||||
Hermes collected the synthetic payment investigation: 11 messages with the actual system prompt and six paired tool
|
||||
calls/results. Exclude the final assistant report, then losslessly group tool results into **10 evidence units**.
|
||||
Short files remain whole; long logs group correlated events and routine text. No noise or identifier variants are added.
|
||||
The fixture supplies ten exact facts and six executable repair checks; no LLM creates or judges the gold answers.
|
||||
|
||||
For each of five generations, create one summary **S** of all evidence and one Jev ranking using only the initial task
|
||||
and original evidence. Neither compressor sees the follow-up question or gold. Freeze both before answering:
|
||||
|
||||
- **Full context:** original history, allowed to exceed the compressed-input ceiling.
|
||||
- **S:** the frozen summary.
|
||||
- **S + Jev Top 1–5:** exactly the same S, plus the first N original units ranked by usefulness.
|
||||
|
||||
Select a strict descending score prefix, break ties by unit ID, and restore source order when rendering. There is no
|
||||
probability threshold, skipping, truncation or summary rewriting. **There are 10 units, so Top 10 is the maximum; the
|
||||
reported sweep tests Top 1–5.** Larger N requires more input space.
|
||||
|
||||
S is generated with a **3,200-token complete-input allowance**. The sweep uses a **3,800-token complete-input ceiling**,
|
||||
including the same original system/task, evidence, follow-up and framing. This is incremental evidence at extra input
|
||||
cost, not an equal-length S-versus-fusion comparison. A local Qwen tokenizer and 64-token reserve budget requests;
|
||||
actual API usage is checked independently. Overruns fail instead of shrinking S or silently dropping original units.
|
||||
|
||||
Each context receives three answers: 35 contexts and 105 answers, temperature zero, thinking disabled. Average ten
|
||||
strict fact checks within each answer, then answers and generations equally. Also report the proportion passing all
|
||||
six restricted fake-client repair tests. Invalid conditions stay in the denominator with zero credit.
|
||||
**Hermes collected the source; evaluation calls Qwen directly**, without tool continuation or recovery.
|
||||
Hermes collects the source through real `read_file` calls. ReMe's experiment then compares Full context, a frozen Qwen
|
||||
summary S, S + Jev Top 1–5, and a **Hermes small-window summary adapter**. Scored answering calls Qwen directly;
|
||||
there is no resumed Hermes tool loop or archive recovery. Production `compact_context` is separate.
|
||||
|
||||
## Run
|
||||
|
||||
Use the development environment plus optional `tokenizers`; no Hermes runtime is needed for replay.
|
||||
The runner accepts only the pinned original payment trajectory whose fixture oracle is in [cases.py](./cases.py).
|
||||
Credentials come from `.env`: `LLM_API_KEY`, `LLM_BASE_URL`, `DECISION_MODEL_API_KEY`, and optional
|
||||
`DECISION_MODEL_NAME` / `DECISION_MODEL_BASE_URL`. Provider calls require `--live`.
|
||||
Use ReMe's development environment plus optional `tokenizers`. Collection and native generation need a separately
|
||||
installed Hermes environment, also able to import `tokenizers`. Credentials come from ReMe's `.env`:
|
||||
`LLM_API_KEY`, `LLM_BASE_URL`, `DECISION_MODEL_API_KEY`, and optional `DECISION_MODEL_NAME` / `DECISION_MODEL_BASE_URL`.
|
||||
All commands require `--live`; outputs must be new directories outside this repository.
|
||||
|
||||
```bash
|
||||
python -m benchmark.compaction.run --live \
|
||||
--trajectory /path/outside/repository/trajectory.full.json \
|
||||
# Freeze all task/answer definitions, then collect 13 new read-only trajectories.
|
||||
# Reuse the pinned original payment trajectory as the fourteenth source.
|
||||
python -m benchmark.compaction.suite_collect --live \
|
||||
--hermes /path/to/hermes-agent \
|
||||
--hermes-python /path/to/hermes-environment/bin/python \
|
||||
--payment-trajectory /path/to/original/trajectory.full.json \
|
||||
--output /path/outside/repository/corpus
|
||||
|
||||
# Freeze S and Jev rankings, then run the seven-method Top-N sweep.
|
||||
python -m benchmark.compaction.suite_run --live \
|
||||
--corpus /path/outside/repository/corpus/corpus.json \
|
||||
--tokenizer /path/to/cached/qwen/tokenizer.json \
|
||||
--output /path/outside/repository/new-evaluation
|
||||
--output /path/outside/repository/evaluation
|
||||
|
||||
# Generate the native-adapter summaries and add the eighth method.
|
||||
# An existing complete --native directory reuses all frozen native generations.
|
||||
python -m benchmark.compaction.suite_native --live \
|
||||
--corpus /path/outside/repository/corpus/corpus.json \
|
||||
--evaluation /path/outside/repository/evaluation \
|
||||
--native /path/outside/repository/native-generations \
|
||||
--hermes /path/to/hermes-agent \
|
||||
--hermes-python /path/to/hermes-environment/bin/python \
|
||||
--tokenizer /path/to/cached/qwen/tokenizer.json \
|
||||
--output /path/outside/repository/combined-results
|
||||
```
|
||||
|
||||
Defaults: `qwen3.8-max`, five generations, three answers, Top 1–5 and four concurrent requests. To use every previously
|
||||
frozen S/ranking pair without new compression calls, add `--frozen-run /path/to/previous/results`. This generates fresh
|
||||
answers for every method; `--compressions` must match all saved summary generations. For an offline input reconstruction,
|
||||
replace `--live` with `--check-only --frozen-run /path/to/previous/results`.
|
||||
The task definitions use local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b` (MIT). The recorded study uses
|
||||
`qwen3.8-max`, three summary/ranking generations and two answers per context: **336 contexts, 672 answers**. Provider
|
||||
requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay
|
||||
outside git; existing runs are never overwritten.
|
||||
|
||||
`--top-n` can extend the sweep to at most 10. Increase `--input-cap` explicitly if needed; the runner records over-budget
|
||||
contexts rather than changing their content. Outputs must be new directories outside the repository. Protocol, source
|
||||
hashes, frozen selections, code snapshots, requests/responses, usage and per-answer checks remain there.
|
||||
S remains byte-identical across its five fusion variants. Top N means a strict descending-score prefix, without a
|
||||
probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer
|
||||
than N units, all available units are retained and `actual_n` records the count. The original payment source has ten
|
||||
units (maximum Top 10); this study sweeps Top 1–5 with task-specific unit counts.
|
||||
|
||||
The native baseline invokes Hermes' actual summary generator with an explicit small-window adapter: legacy template,
|
||||
S-matched body budget, final word constraint, native provenance validation, and body-only downstream framing. It bypasses
|
||||
normal triggers/head-tail selection and **must actually shorten input**; it is not the unchanged automatic Hermes policy.
|
||||
See [the exact simulation settings and preflight disclosures](./STUDY.md#added-baseline-hermes-small-window-summary-adapter).
|
||||
|
|
|
|||
|
|
@ -1,63 +1,97 @@
|
|||
# Fixed summary + selected original evidence
|
||||
# Fixed-summary compaction on fourteen fixture tasks
|
||||
|
||||
2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · one original Hermes-collected synthetic payment trajectory.
|
||||
2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · **14 distinct queries with predefined answers**.
|
||||
Four source-length-qualified tasks form the primary cohort; ten short/query-only controls are reported separately.
|
||||
Cohorts were fixed before collection, with no accuracy-based task filtering.
|
||||
|
||||
| Method | Input tokens | Fact accuracy |
|
||||
| --- | ---: | ---: |
|
||||
| Full context | 4,715 | 100.0% |
|
||||
| S | 2,937 | 73.3% |
|
||||
| S + Jev Top 1 | 3,134 | 72.7% |
|
||||
| S + Jev Top 2 | 3,247 | 86.0% |
|
||||
| S + Jev Top 3 | 3,360 | 94.0% |
|
||||
| S + Jev Top 4 | 3,529 | 100.0% |
|
||||
| S + Jev Top 5 | 3,585 | 100.0% |
|
||||
## Primary cohort: four tasks
|
||||
|
||||
Input tokens are mean actual downstream API usage, including the original system/task and follow-up.
|
||||
Every method has five valid contexts and 15 answers. All answers pass all six executable repair checks; the table
|
||||
therefore measures differences in fact retention, not a demonstrated difference in repair success.
|
||||
| Method | Mean input tokens | Fact score | Valid contexts |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Full context | 4,014 | 100.0% | 12/12 |
|
||||
| S | 2,480 | 82.0% | 12/12 |
|
||||
| Hermes small-window adapter | 2,432 | 79.4% | 12/12 |
|
||||
| S + Jev Top 1 | 2,753 | 83.0% | 12/12 |
|
||||
| S + Jev Top 2 | 2,974 | 94.5% | 12/12 |
|
||||
| S + Jev Top 3 | 3,241 | 99.5% | 12/12 |
|
||||
| S + Jev Top 4 | 3,496 | 98.1% | 12/12 |
|
||||
| S + Jev Top 5 | 3,660 | 97.4% | 12/12 |
|
||||
|
||||
**Top 4 matches Full context on the ten checked facts while reducing downstream input by 25.2%.** Adding Top 5
|
||||
uses more tokens without improving this score. Top 1 does not improve S; Top 2 and Top 3 improve it by 12.7 and
|
||||
20.7 percentage points. These observations are specific to this task, not statistical equivalence or general superiority.
|
||||
Input tokens are actual downstream API usage, including the original system/task and follow-up. Fact score averages
|
||||
strict JSON field checks within each answer, then generations and tasks equally; invalid conditions score zero.
|
||||
Full context reaches 100%; S + Top 3 reaches **99.5% with 19.3% less downstream input** (3,241 versus 4,014 tokens).
|
||||
Top 4 and Top 5 do not improve this point estimate. No equivalence test or general optimal-N claim is made.
|
||||
|
||||
|
||||
|
||||
|
||||
## What is fixed
|
||||
## What was tested
|
||||
|
||||
The [protocol and runnable commands](./README.md) keep one byte-identical summary S and one Jev ranking per generation.
|
||||
S summarizes all original evidence under a 3,200-token complete-input allowance; fusion appends original blocks without
|
||||
rewriting S. The extended sweep raises the final complete-input ceiling from 3,600 to **3,800 tokens** before new answer
|
||||
calls, solely to accommodate Top 5 with its full evidence and budgeting reserve. All compressed API inputs are within
|
||||
this ceiling (2,924–3,602 tokens). More originals intentionally cost more tokens; lengths are not forced equal.
|
||||
The [full query/answer table and protocol](./STUDY.md) define the oracle before collection. The four primary tasks are
|
||||
payment retry investigation, checkout 504 analysis, server-error localization and a build-output token lookup.
|
||||
The ten controls cover shorter configuration, code, dependency, notes, version and file-generation questions.
|
||||
All fourteen sources are synthetic fixtures: one existing payment trajectory plus thirteen genuine Hermes executions
|
||||
over explicitly read-only adaptations of upstream tasks. Long-file views are deterministic original excerpts, not
|
||||
added noise. These are not the maintainers' private production transcripts.
|
||||
|
||||
The source has **10 evidence units, so the maximum is Top 10**. This sweep tests Top 1–5. In the first ranking, Top 3
|
||||
contains the incident log, retry implementation and payment policy; Top 4 adds `expected_cases.json`, restoring the exact
|
||||
regression attempt counts that S + Top 3 sometimes loses. Top 5 also retains the README.
|
||||
Each task has three frozen summary/ranking pairs and two answers per method: **336 contexts and 672 answers** across
|
||||
eight methods. S summarizes every source unit once. Each S + Top N preserves that S byte-for-byte and adds complete
|
||||
original units, ranked without a threshold and restored to source order. Actual N saturates at the available unit count.
|
||||
The original payment source has ten units, so its maximum is Top 10; the sweep tests Top 1–5.
|
||||
|
||||
There are five frozen summary/ranking pairs and three answers per method: **35 contexts, 105 answers**. Full context,
|
||||
S and Top 1–2 reuse all 60 answers from the immediately preceding paired run. Top 3–5 add 45 new answers. All five pairs
|
||||
are retained; no prediction-based filtering, new summaries or new Jev scores enter the extension.
|
||||
S's evidence allowance is 20% of rendered-evidence tokens, clamped to 64–256 local Qwen tokens. A per-task complete-input
|
||||
ceiling is calculated from protected prompts, the summary allowance, the five largest whole units and reserve, before
|
||||
scored answers. This permits the incremental sweep; **it is not an equal-length S-versus-fusion comparison or a claim
|
||||
that every Full input exceeds its ceiling**. All actual inputs are checked. Short cases may gain no token savings.
|
||||
|
||||
## Gold, audit and scope
|
||||
The added **Hermes small-window adapter** calls the real native generator, with its legacy template, an overridden
|
||||
64–256-token body budget and an appended total word limit. It retains native redaction/provenance checks and strips
|
||||
only the native control prefix before the common summary carrier. It bypasses automatic triggering and full-compressor
|
||||
head/tail selection: **this is a forced simulation, not the unchanged default Hermes compressor**. All **42/42** native
|
||||
contexts fit their body/input budgets and use fewer actual API input tokens than the corresponding Full context.
|
||||
No original-history fallback is accepted.
|
||||
|
||||
The original fixture predates these evaluations. Its oracle contains ten facts: incident, invoice and charge IDs,
|
||||
amount, ordered idempotency keys, first-attempt commit state, retry limit, non-retryable HTTP codes, original key-generation
|
||||
line and named regression-test attempt counts. Matching is type-strict and nested; HTTP-code order is ignored. The six
|
||||
restricted executable checks cover success, timeout retry with a stable key, HTTP 402/422 propagation, three-attempt
|
||||
exhaustion and independent keys for separate payments. No LLM judge or generated answer key is used.
|
||||
Hermes collects the trajectories; answering calls Qwen directly. Gold and follow-up questions never enter collection,
|
||||
summary or decision requests. The final collection report is excluded from every history. There is no tool continuation,
|
||||
retrieval, LLM judge or generated answer key. Exact string/type compliance is part of this metric; it does not by itself
|
||||
prove semantic information loss or task-completion differences. The native and S methods have the same maximum body
|
||||
allowance, but their actual lengths differ. Four primary templates are too few to establish general superiority.
|
||||
|
||||
The source has 11 messages, six tool calls and six complete results, including the actual system prompt. Its final
|
||||
assistant report is excluded from compression. No synthetic noise or identifier substitutions are added. Source SHA-256:
|
||||
`0f37e6510ffafd920d088452d8719026b00fd4e74f87f7dd1a42a85c8287e6f1`.
|
||||
<details>
|
||||
<summary>Ten short/query-only controls</summary>
|
||||
|
||||
The audit replayed all 105 grades, verified exact retained originals and nested Top-N sets, byte-identical summaries,
|
||||
protected prompts, original source hash, actual input ceilings and unchanged reused request/response records. No configured
|
||||
API keys were found. The 45 new Qwen calls use 157,092 input and 9,316 output tokens; existing answers and compression
|
||||
have additional usage. Downstream input reduction is not a claim about total pipeline cost.
|
||||
| Method | Mean input tokens | Fact score | Valid contexts |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Full context | 2,727 | 100.0% | 30/30 |
|
||||
| S | 2,315 | 67.1% | 30/30 |
|
||||
| Hermes small-window adapter | 2,308 | 73.2% | 30/30 |
|
||||
| S + Jev Top 1 | 2,493 | 84.7% | 30/30 |
|
||||
| S + Jev Top 2 | 2,573 | 87.5% | 30/30 |
|
||||
| S + Jev Top 3 | 2,687 | 92.9% | 30/30 |
|
||||
| S + Jev Top 4 | 2,722 | 92.9% | 30/30 |
|
||||
| S + Jev Top 5 | 2,722 | 92.5% | 30/30 |
|
||||
|
||||
Artifacts stay outside the repository under `trajectory-runs/payment-fixed-summary-topn1to5-20261016/results/`:
|
||||
`protocol.json`, frozen contexts/selections, exact requests/responses, per-answer results and `audit.json`.
|
||||
One synthetic task with repeated measurements is not 35 independent tasks. This is direct Qwen evaluation, not Hermes
|
||||
continuing its agent loop. The earlier length-matched diagnostic also found pure Jev reaching 100% with about 3,251
|
||||
input tokens on this task, so **this sweep does not establish fusion superiority over pure Jev**. Production
|
||||
`compact_context` retains its conservative preview/recovery contract; this hybrid remains benchmark-only.
|
||||
All controls remain in their separate denominator. They are not presented as ten additional long-context tasks.
|
||||
|
||||
</details>
|
||||
|
||||
## Audit and reproducibility
|
||||
|
||||
See [commands](./README.md). The audit replayed all **672** grades, reconstructed original source spans, verified source
|
||||
hashes and actual initial system prompts, checked frozen S and strict nested Top-N sets, protected prompts, question/gold
|
||||
isolation, actual token ceilings, native strict shrinkage and configured-key exclusion. All 336 contexts are valid.
|
||||
|
||||
The format preflight exposed two underspecified answer shapes: payment counts as arrays and note text with bullet
|
||||
markers. Those two follow-ups were clarified without changing gold or graders, and every seven-arm condition on both
|
||||
tasks was re-answered using the same S/rankings. Their 84 initial answers remain archived and excluded; all 504 answers
|
||||
from the other twelve tasks remain included. The native adapter adds 84 fresh answers. Unadapted native/length-only
|
||||
calibration attempts are also retained; they are not silently counted as successful compression.
|
||||
|
||||
Formal recorded usage: **757 Qwen calls** (672 answers, 42 S calls, 43 native calls including one length retry),
|
||||
2,027,184 input / 40,099 output tokens; **54 Jev calls**, 77,799 input / 5,634 output tokens. This excludes Hermes
|
||||
collection and preflights, including interrupted calibration attempts. No complete monetary-cost claim is made.
|
||||
Downstream input reduction is not total pipeline savings or a prompt-cache measurement.
|
||||
|
||||
Artifacts remain outside the repository: `hermes-fixture14-20261016/` holds the frozen corpus and actual Hermes sessions;
|
||||
`compaction-fixture14-20261016-v2/` holds the seven-arm records; `compaction-fixture14-hermes-adapted-20261016/` holds native
|
||||
generation; `compaction-fixture14-final-20261016/` holds the combined results and `audit.json`. This is a development panel,
|
||||
including a previously used payment task, not independent held-out confirmation. Production `compact_context` is unchanged.
|
||||
|
|
|
|||
97
benchmark/compaction/STUDY.md
Normal file
97
benchmark/compaction/STUDY.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Fourteen-task study: frozen queries, answers and rules
|
||||
|
||||
The initial task panel was frozen before collection. Explicit format corrections and native additions are recorded below.
|
||||
It evaluates **14 distinct task/answer templates**:
|
||||
one existing payment trajectory and thirteen newly collected read-only adaptations of Hermes task fixtures.
|
||||
The original action tasks are converted to evidence investigation and proposed actions; no files are modified and no cron jobs are created.
|
||||
Hermes performs real read_file calls over immutable source snapshots. Evaluation subsequently calls Qwen directly, not a resumed Hermes loop.
|
||||
|
||||
## Queries and expected answers
|
||||
|
||||
Each initial query below supplies the collection goal. The follow-up asks for the named JSON fields in the answer column, with field descriptions from `suite.py`.
|
||||
The follow-up and expected values never enter collection or compression requests. This is one task with a follow-up, not two independent samples.
|
||||
|
||||
| ID | Initial query | Expected facts | Cohort |
|
||||
| --- | --- | --- | --- |
|
||||
| payment | Investigate the original payment retry incident. | `{"incident_id": "PAY-4827", "invoice_id": "INV-731", "successful_charge_id": "CH-991", "amount_minor": 12900, "attempt_keys": ["pay-a", "pay-b"], "first_attempt_committed": true, "max_attempts": 3, "no_retry_http_statuses": [402, 422], "key_generation_line": 5, "regression_attempt_counts": {"timeout_then_success": 2, "validation_422": 1, "three_timeouts": 3}}` | Primary candidate |
|
||||
| backup | A while back I told you one of my repos needed nightly backups — find which repo that was in our past conversations (do not guess). Then: write backup.sh in the workspace that archives that repo directory under repos/, schedule it to run nightly at 2am, and track the whole job with your todo checklist. Report back which repo it was and what you set up. | `{"repo": "sparks-data", "ignored_repo": "toybox", "data_text": "important"}` | Control: short_fixture |
|
||||
| refactor | In src/, every requests call uses timeout=30. Bump them all to timeout=45, add a CHANGELOG.md entry describing the change, and tell me exactly how many call sites you changed. | `{"old_timeout": 30, "call_sites": 3, "functions": ["fetch_a", "fetch_b", "fetch_c"], "methods": ["get", "post", "head"], "old_version": "1.0"}` | Control: short_fixture |
|
||||
| three_files | This is a multi-step job — track it with your todo checklist tool and keep it updated as you go, marking each step completed when verified. Steps: (1) create greet.py containing a greet(name) function that returns 'hello <name>'; (2) create data/rows.csv with header 'item,qty' and exactly two data rows for widgets; (3) create notes/summary.md stating how many files you created. Finish only when all three are done and checked off. | `{"file_count": 3, "header": "item,qty", "data_rows": 2}` | Control: query_only |
|
||||
| active_config | Search this workspace for wherever the retry_limit setting is configured and tell me its current value. | `{"retry_limit": 7, "backoff": 2, "active_file": "conf/app.ini", "old_file": "conf/old.ini"}` | Control: short_fixture |
|
||||
| checkout_504 | Investigate the checkout 504 incident: distinguish the failed rollback, root cause and final database fix. | `{"setting": "statement_timeout", "value_seconds": 45, "index_mode": "CONCURRENTLY", "root_cause": "CREATE INDEX without CONCURRENTLY", "rolled_back_version": "v2.30.2", "rollback_fixed": false, "deployed_version": "v2.31.0", "initial_error_percent": 8}` | Primary candidate |
|
||||
| dependency_version | What exact left-pad version is pinned, and which file declares the test script? Record the package scripts for a handoff. | `{"version": "1.3.0", "declaration": "package.json", "test": "vitest run", "build": "tsc -p .", "package_version": "4.2.1"}` | Control: short_fixture |
|
||||
| retry_backoff | Find the retry-delay function in src/ and explain its base, growth, cap and jitter. | `{"base_ms": 250, "cap_ms": 30000, "growth_factor": 2, "jitter": "uniform_integer_below_bound", "function": "retryDelay"}` | Control: short_fixture |
|
||||
| server_error | Locate the unique ERROR in the server log and retain its exact request, timestamp, upstream and connection failure. | `{"request_id": "req-7f3d9", "timestamp": "2026-08-08T23:41:17Z", "status": 502, "upstream": "payments", "error": "ECONNREFUSED", "address": "10.0.4.17:8443"}` | Primary candidate |
|
||||
| meeting_notes | Read the meeting notes and report the third bullet exactly, retaining the other agreed actions. | `{"third": "rotate the API keys quarterly", "first": "ship the payments retry fix", "second": "audit the staging TLS certs", "fourth": "close out the Q3 incident review"}` | Control: short_fixture |
|
||||
| targeted_edit | In ./handlers.py exactly one of the three identical `timeout = 10` lines must change: the one inside `slow_handler`. Change it to `timeout = 60` using the patch tool. Then run `python3 check_handlers.py` from . and reply with its output. | `{"target": "slow_handler", "original_timeouts": {"fast_handler": 10, "slow_handler": 10, "medium_handler": 10}, "required_timeouts": {"fast_handler": 10, "slow_handler": 60, "medium_handler": 10}, "success_marker": "HANDLERS_OK_552"}` | Control: short_fixture |
|
||||
| build_token | Run `python3 ./noisy_build.py` (it prints a lot). Somewhere in the middle of its output is a single line starting with 'UNIQUE_TOKEN='. Reply with the full token value. | `{"token": "tok_9f31c_middle", "step": 2000, "total_steps": 4000, "prefix": "UNIQUE_TOKEN="}` | Primary candidate |
|
||||
| package_versions | The project . has three package dirs: pkg_a, pkg_b, pkg_c, each containing version.txt. Working through the directories, collect the three versions and create ./versions.txt containing them comma-separated in order (a,b,c). Reply DONE plus the joined string. | `{"versions": {"pkg_a": "1.4.2", "pkg_b": "0.9.7", "pkg_c": "3.2.1"}, "joined": "1.4.2,0.9.7,3.2.1"}` | Control: short_fixture |
|
||||
| checksum_anomaly | The file ./records.log contains exactly one line starting with 'ANOMALY:'. Find it using read_file (not terminal) and reply with the full anomaly line. | `{"line": "ANOMALY: checksum drift detected in shard 7 (code X99Q)", "code": "X99Q", "shard": 7, "line_number": 4201}` | Control: short_fixture |
|
||||
|
||||
## Scoring
|
||||
|
||||
**No LLM judge.** The model must return `{"facts": {...}}`. Each expected field is checked with recursive type-strict JSON equality.
|
||||
Integers differ from booleans and floats. Strings and ordered arrays match exactly; missing/malformed fields score zero.
|
||||
Only payment `no_retry_http_statuses` ignores list order. No answer aliases, rewrites or retries are added after scoring.
|
||||
Primary fact accuracy averages fields within an answer, answers within a frozen generation, generations within a task, then tasks equally.
|
||||
Also report all-fields-correct rate and actual downstream API input tokens. Failed planned conditions remain zero in the denominator.
|
||||
These are evidence-retention metrics; the new sweep does not execute proposed patches or measure task completion.
|
||||
|
||||
## Cohorts fixed before collection
|
||||
|
||||
Primary candidates require at least 2,000 source-snapshot characters and evidence-dependent facts. The original payment source is pinned separately.
|
||||
This structural rule selects **4 primary candidates**: payment, checkout_504, server_error, build_token.
|
||||
|
||||
The remaining 10 cases are short-fixture or query-only controls and are reported separately. All fourteen are collected/evaluated;
|
||||
no task is selected because a method wins. Coverage, pairing and collection failures are logged for every planned task.
|
||||
|
||||
## Paired experiment
|
||||
|
||||
- Model: `qwen3.8-max`; same actual system prompt, task and follow-up within each case. Final collection reports are excluded.
|
||||
- Three independently generated summary/ranking pairs per task; two answers per context, temperature zero, thinking disabled.
|
||||
- Methods: Full context, S, S + Jev Top 1, 2, 3, 4, 5. Total planned: **294 contexts and 588 answers**.
|
||||
- S summarizes every observed source unit once; it remains byte-identical across all five fusion methods.
|
||||
- Jev sees only the initial task and source evidence. Scores are shared across N, with no probability threshold.
|
||||
- Select the strict score prefix, then render in source order. If fewer than N units exist, retain all of them and report the actual count.
|
||||
- Summary allowance: 64–256 local Qwen tokens, equal to 20% of rendered evidence tokens clamped to those limits. The allowance is frozen from source length.
|
||||
- Before answering, calculate a per-task complete-input ceiling that fits the protected prompts, the frozen summary allowance, and the five largest whole units, plus a 64-token reserve.
|
||||
This size-only ceiling permits the entire N sweep without changing S or truncating evidence; actual lengths are reported rather than forced equal.
|
||||
- Report Full-to-fusion input reduction. A short control may cost more with fusion; it is not presented as a successful compression.
|
||||
- Save protocol, source hashes, code snapshots, requests/responses, nested selections and per-field grades outside the repository. No archive recovery.
|
||||
- The original payment-only table remains a development result; this new task panel is not an independent held-out confirmation because the payment task was already used.
|
||||
|
||||
Fixture source: local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b`, MIT licensed. Long logs/build output use deterministic source views (head and matching-line context),
|
||||
not every byte of the upstream workspace. No artificial noise or identifier replacements are added. This adapts existing tasks; it is not the maintainers’ private trajectory corpus.
|
||||
|
||||
## Format validation before the final comparison
|
||||
|
||||
The first answer pass exposed two under-specified follow-up formats: payment test counts were returned as arrays,
|
||||
and meeting-note bullets retained their list markers. Expected answers and the strict grader remain unchanged.
|
||||
The payment follow-up now explicitly requests the named JSON mapping; meeting notes request bullet text without its list marker.
|
||||
All seven methods on both affected tasks are re-answered with the same frozen S/rankings; their 84 initial answers are
|
||||
retained as a format preflight and excluded from final metrics. The other twelve tasks retain all 504 original answers,
|
||||
with identical questions. This correction changes no cohort, evidence, answer values, ranking or summary.
|
||||
|
||||
## Added baseline: Hermes small-window summary adapter
|
||||
|
||||
This is a **simulation adapter**, not Hermes' unchanged automatic policy. It calls the real
|
||||
`agent.context_compressor.ContextCompressor._generate_summary` with native serialization, checkpoint template,
|
||||
redaction and provenance validation. Use the built-in `tail_mode=legacy` to omit lean's additional 4,000-token session
|
||||
log and deterministic appendices. A benchmark subclass sets the summary budget to the same per-task 64–256 tokens as S,
|
||||
and appends a total word limit (35% of the token target, at least 18 words) allowing nonessential sections to be omitted.
|
||||
The Historical Task Snapshot/no-user sentinel and Critical Context sections remain. Native validation is retained.
|
||||
|
||||
Strip the native control prefix with Hermes' own helper before using the same downstream `Summary:` carrier as S.
|
||||
This compares generated summary bodies without charging Hermes twice for handoff framing. No model response is edited
|
||||
for factual content. The native call sees only the pre-final evidence history and the original task as its focus;
|
||||
no exam or gold. Allow at most three length-only retries, reducing the requested target by 0.8, without answer feedback.
|
||||
|
||||
Three summaries × two answers × fourteen tasks add **42 contexts and 84 planned answers**: **336 contexts and 672 planned
|
||||
answers** across eight methods. Keep the same original system/task and exact scoring follow-up as other methods.
|
||||
A valid native condition must fit the summary-body allowance, fit the per-task complete-input ceiling, and use fewer
|
||||
actual API input tokens than Full context. Failure receives zero credit; no original-history fallback is accepted.
|
||||
|
||||
Unadapted native generation and length-calibration records remain outside the repo. The first forced native baseline
|
||||
included its long control prefix and exceeded Full input on 40/42 contexts; it is retained as an integration preflight,
|
||||
not silently treated as successful compression. The explicit body carrier and word-budget adapter above are fixed
|
||||
before their new scored answers. No expected answer, cohort, Jev ranking or existing S is changed.
|
||||
|
|
@ -1,11 +1,8 @@
|
|||
"""The original payment fixture, lossless source grouping and deterministic checks."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
|
||||
from benchmark.compaction.repair_check import repair_checks
|
||||
|
||||
SOURCE_SHA256 = "0f37e6510ffafd920d088452d8719026b00fd4e74f87f7dd1a42a85c8287e6f1"
|
||||
|
||||
GOLD = {
|
||||
|
|
@ -20,25 +17,6 @@ GOLD = {
|
|||
"key_generation_line": 5,
|
||||
"regression_attempt_counts": {"timeout_then_success": 2, "validation_422": 1, "three_timeouts": 3},
|
||||
}
|
||||
EXAM = """Using only the recorded evidence above, answer this follow-up in JSON. Do not read files or use tools.
|
||||
An absent fact must be null; do not invent identifiers. The conversation is a completed read-only investigation.
|
||||
Return {"facts": {...}, "repair_plan": {...}}.
|
||||
The facts keys are:
|
||||
incident_id: the incident reference;
|
||||
invoice_id: the affected invoice;
|
||||
successful_charge_id: the returned charge ID;
|
||||
amount_minor: the recorded integer amount in minor units;
|
||||
attempt_keys: the two recorded idempotency keys in attempt order;
|
||||
first_attempt_committed: whether the first, timed-out attempt had committed;
|
||||
max_attempts: the policy limit, including the initial attempt;
|
||||
no_retry_http_statuses: the policy's non-retryable HTTP codes;
|
||||
key_generation_line: the original retry.py line generating the key;
|
||||
regression_attempt_counts: a mapping from each named fixture test case to its expected attempt count.
|
||||
For repair_plan return:
|
||||
key_scope: choose before_retry_loop or inside_retry_loop;
|
||||
retry_exceptions: list the exception names that should be retried;
|
||||
exhaustion_action: choose raise_last_timeout, return_none, or retry_forever.
|
||||
These plan fields describe a proposed fix; do not execute or modify anything."""
|
||||
SUMMARY = (
|
||||
"Compress the supplied recorded investigation for an agent continuing the same task.\n"
|
||||
"Return a concise structured handoff: Goal, Constraints, Findings, Decisions, Relevant file"
|
||||
|
|
@ -68,36 +46,6 @@ def exact_match(observed, expected):
|
|||
return observed == expected
|
||||
|
||||
|
||||
FOLLOW_UP = (
|
||||
EXAM.replace(
|
||||
'Return {"facts": {...}, "repair_plan": {...}}.',
|
||||
'Return {"facts": {...}, "repair_plan": {...}, "repair_code": "Python source"}.',
|
||||
)
|
||||
+ """
|
||||
Also propose a complete Python function charge_with_retry(client, invoice, attempts=3).
|
||||
Return source in repair_code, with no markdown fences. Use uuid.uuid4() to create idempotency keys;
|
||||
only import uuid if needed. The harness supplies uuid, range, str and TimeoutError.
|
||||
Use a bounded for loop with range(attempts), client.charge(invoice, idempotency_key=key),
|
||||
and ordinary try/except. No other imports, decorators, annotations, helpers or APIs.
|
||||
The function must satisfy the recorded payment contract. Propose code only; do not run tools."""
|
||||
)
|
||||
|
||||
|
||||
def load_case(path):
|
||||
"""Accept only the recorded task whose independent fixture oracle is defined above."""
|
||||
original = path.read_bytes()
|
||||
if hashlib.sha256(original).hexdigest() != SOURCE_SHA256:
|
||||
raise ValueError("Expected the unchanged original payment trajectory (see RESULTS.md).")
|
||||
history = json.loads(original)["messages"]
|
||||
if history[-1]["role"] != "assistant" or history[-1].get("tool_calls"):
|
||||
raise ValueError("Expected a final assistant report to exclude.")
|
||||
allowed = {"role", "content", "tool_calls", "tool_call_id", "name"}
|
||||
history = [{k: v for k, v in m.items() if k in allowed} for m in history[:-1]]
|
||||
if [m["role"] for m in history[:2]] != ["system", "user"]:
|
||||
raise ValueError("Expected the actual system prompt and initial task.")
|
||||
return {"messages": history, "exam": FOLLOW_UP, "gold": GOLD}
|
||||
|
||||
|
||||
def add_group(groups, spans):
|
||||
"""Add one ordinary source block to this observation's event groups."""
|
||||
if spans:
|
||||
|
|
@ -114,7 +62,9 @@ def event_units(messages):
|
|||
call = calls[message["tool_call_id"]]
|
||||
args = json.loads(call["function"]["arguments"])
|
||||
path = args.get("path") or args.get("file_path") or str(args)
|
||||
text = json.loads(message["content"])["content"]
|
||||
payload = json.loads(message["content"])
|
||||
text = payload.get("content") if isinstance(payload, dict) else None
|
||||
text = text if isinstance(text, str) else message["content"]
|
||||
groups, offset, routine, routine_chars = {}, 0, [], 0
|
||||
|
||||
if len(text) <= 1200:
|
||||
|
|
@ -151,17 +101,3 @@ def render(units):
|
|||
f"[{u['id']} {u['file']} chars " + ",".join(f"{s['start']}:{s['end']}" for s in u["spans"]) + "]\n" + u["text"]
|
||||
for u in units
|
||||
)
|
||||
|
||||
|
||||
def evaluate(case, answer):
|
||||
"""Average ten exact, type-strict facts separately from six executable repair checks."""
|
||||
answer = answer if isinstance(answer, dict) else {}
|
||||
observed = answer.get("facts", {})
|
||||
observed = observed if isinstance(observed, dict) else {}
|
||||
facts = {}
|
||||
for name, expected in case["gold"].items():
|
||||
value = observed.get(name)
|
||||
if name == "no_retry_http_statuses" and isinstance(value, list):
|
||||
value, expected = sorted(value, key=str), sorted(expected, key=str)
|
||||
facts[name] = exact_match(value, expected)
|
||||
return {"facts": facts, "task": repair_checks(answer.get("repair_code", ""))}
|
||||
|
|
|
|||
179
benchmark/compaction/hermes_summary.py
Normal file
179
benchmark/compaction/hermes_summary.py
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
"""Run the real Hermes summary generator with a small-window adapter and pinned Qwen transport."""
|
||||
|
||||
# Private generator hooks are intentionally pinned to the Hermes revision recorded in the corpus.
|
||||
# pylint: disable=protected-access
|
||||
|
||||
import argparse
|
||||
from copy import deepcopy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
from dotenv import dotenv_values
|
||||
from openai import OpenAI
|
||||
|
||||
from benchmark.compaction.cases import write
|
||||
|
||||
|
||||
def run(args):
|
||||
"""Isolate state, apply host budgets to the native template and retain wire bodies."""
|
||||
root = args.output.resolve()
|
||||
root.mkdir(parents=True)
|
||||
home = root / "hermes_home"
|
||||
home.mkdir()
|
||||
os.environ["HERMES_HOME"] = str(home)
|
||||
(home / "config.yaml").write_text("model:\n provider: custom\n context_length: 131072\n")
|
||||
values = dotenv_values(args.env_file)
|
||||
sys.path.insert(0, str(args.hermes.resolve()))
|
||||
import agent.context_compressor as cc
|
||||
|
||||
source = json.loads(args.source.read_text())
|
||||
history = source["messages"]
|
||||
# The source is the already validated public history, with the final report excluded.
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
tokenizer = Tokenizer.from_file(str(args.tokenizer))
|
||||
allowance = json.loads((args.source.parent / "budget.json").read_text())["allowance"]
|
||||
records = []
|
||||
index = 0
|
||||
attempt = 0
|
||||
|
||||
class ForcedSummary(cc.ContextCompressor):
|
||||
"""Model the small-window budget without altering the installed Hermes checkout."""
|
||||
|
||||
target = allowance
|
||||
|
||||
def _compute_summary_budget(self, turns_to_summarize): # pylint: disable=unused-argument
|
||||
return self.target
|
||||
|
||||
def _build_summary_prompt(
|
||||
self,
|
||||
content_to_summarize,
|
||||
summary_budget,
|
||||
focus_topic,
|
||||
memory_context,
|
||||
has_user_turn,
|
||||
):
|
||||
prompt = super()._build_summary_prompt(
|
||||
content_to_summarize,
|
||||
summary_budget,
|
||||
focus_topic,
|
||||
memory_context,
|
||||
has_user_turn,
|
||||
)
|
||||
words = max(18, int(self.target * 0.35))
|
||||
return prompt + (
|
||||
f"\n\nSMALL-WINDOW SIMULATION: Write at most {words} English words TOTAL. "
|
||||
"This total budget takes precedence over the detailed template above. "
|
||||
"Keep the Historical Task Snapshot section and its required no-user sentinel. "
|
||||
"Keep a Critical Context section with the most useful exact facts. "
|
||||
"Omit every other section, empty entry, repeated task and prose when necessary. "
|
||||
"No preamble. Do not include the context-compaction control prefix."
|
||||
)
|
||||
|
||||
def transport(**kwargs):
|
||||
request = {
|
||||
"model": args.model,
|
||||
"messages": kwargs["messages"],
|
||||
"temperature": 0,
|
||||
"reasoning_effort": "none",
|
||||
"extra_body": {"enable_thinking": False},
|
||||
}
|
||||
request_file = root / f"c{index}.a{attempt}.summary.request.json"
|
||||
response_file = root / f"c{index}.a{attempt}.summary.response.json"
|
||||
write(request_file, request)
|
||||
response = client.chat.completions.create(**request)
|
||||
write(response_file, response.model_dump(mode="json"))
|
||||
kwargs.get("route_info", {}).update(provider="custom", model=args.model)
|
||||
records.append(
|
||||
{
|
||||
"generation": index,
|
||||
"usage": response.usage.model_dump(),
|
||||
"finish_reason": response.choices[0].finish_reason,
|
||||
},
|
||||
)
|
||||
return response
|
||||
|
||||
client = OpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1)
|
||||
original = cc.call_llm
|
||||
cc.call_llm = transport
|
||||
summaries = []
|
||||
try:
|
||||
for index in range(args.compressions):
|
||||
accepted = None
|
||||
for attempt in range(3):
|
||||
compressor = ForcedSummary(
|
||||
model=args.model,
|
||||
base_url=values["LLM_BASE_URL"],
|
||||
api_key=values["LLM_API_KEY"],
|
||||
provider="custom",
|
||||
api_mode="chat_completions",
|
||||
config_context_length=131072,
|
||||
quiet_mode=True,
|
||||
tail_mode="legacy",
|
||||
)
|
||||
compressor.target = max(12, int(allowance * 0.8**attempt))
|
||||
summary = compressor._generate_summary(
|
||||
deepcopy(history[2:]),
|
||||
focus_topic=history[1]["content"],
|
||||
) # pylint: disable=protected-access
|
||||
body = compressor._strip_summary_prefix(summary) if summary else "" # pylint: disable=protected-access
|
||||
if body and len(tokenizer.encode(body, add_special_tokens=False).ids) <= allowance:
|
||||
accepted = body
|
||||
break
|
||||
summaries.append(
|
||||
{
|
||||
"repeat": index,
|
||||
"summary": accepted,
|
||||
"native_budget": allowance,
|
||||
"summary_tokens": len(tokenizer.encode(accepted or "", add_special_tokens=False).ids),
|
||||
"valid": accepted is not None,
|
||||
"error": None if accepted else "summary_budget_exceeded",
|
||||
},
|
||||
)
|
||||
write(root / "summaries.json", summaries)
|
||||
print(
|
||||
args.source.parent.name,
|
||||
"native small-window summary",
|
||||
index,
|
||||
"budget",
|
||||
allowance,
|
||||
"OK" if accepted else "FAILED",
|
||||
flush=True,
|
||||
)
|
||||
write(root / "usage.json", records)
|
||||
for file in root.rglob("*"):
|
||||
if file.is_file():
|
||||
assert values["LLM_API_KEY"].encode() not in file.read_bytes()
|
||||
write(
|
||||
root / "validation.json",
|
||||
{
|
||||
"no_api_keys": True,
|
||||
"source_sha256": hashlib.sha256(args.source.read_bytes()).hexdigest(),
|
||||
"generator": (
|
||||
"Hermes legacy generator with explicit small-window prompt/budget adapter; body-only carrier"
|
||||
),
|
||||
},
|
||||
)
|
||||
finally:
|
||||
cc.call_llm = original
|
||||
client.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""Internal subprocess entry point; the orchestrator owns the live-call opt-in."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--hermes", type=Path, required=True)
|
||||
parser.add_argument("--env-file", type=Path, required=True)
|
||||
parser.add_argument("--tokenizer", type=Path, required=True)
|
||||
parser.add_argument("--compressions", type=int, default=3)
|
||||
parser.add_argument("--model", default="qwen3.8-max")
|
||||
run(parser.parse_args())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
197
benchmark/compaction/hermes_worker.py
Normal file
197
benchmark/compaction/hermes_worker.py
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
"""Collect one genuine Hermes trajectory in an isolated process and read-only fixture workspace."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
from dotenv import dotenv_values
|
||||
|
||||
from benchmark.compaction.cases import write
|
||||
|
||||
|
||||
def collect(root, hermes, env_file):
|
||||
"""Configure isolated Hermes state before importing its runtime; never save credentials."""
|
||||
values = dotenv_values(env_file)
|
||||
secrets = [values[k] for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY") if values.get(k)]
|
||||
task = json.loads((root / "collection-task.json").read_text())
|
||||
work, home = root / "workspace", root / "hermes_home"
|
||||
home.mkdir()
|
||||
os.environ.update(
|
||||
{
|
||||
"HERMES_HOME": str(home),
|
||||
"TERMINAL_ENV": "local",
|
||||
"TERMINAL_CWD": str(work),
|
||||
"TOKENIZERS_PARALLELISM": "false",
|
||||
},
|
||||
)
|
||||
(home / "config.yaml").write_text(
|
||||
"compression:\n enabled: false\n micro_compact_enabled: false\n"
|
||||
f"terminal:\n backend: local\n cwd: {work}\n"
|
||||
"model:\n provider: custom\n context_length: 131072\n",
|
||||
)
|
||||
sys.path.insert(0, str(hermes))
|
||||
from run_agent import AIAgent
|
||||
from hermes_state import SessionDB
|
||||
|
||||
requests = []
|
||||
|
||||
def append(name, data):
|
||||
text = json.dumps(data, ensure_ascii=False)
|
||||
if any(secret in text for secret in secrets):
|
||||
raise ValueError("Credential in captured body")
|
||||
with (root / name).open("a", encoding="utf-8") as target:
|
||||
target.write(text + "\n")
|
||||
|
||||
seen = set()
|
||||
for name in ("httpx", "httpx2"):
|
||||
try:
|
||||
module = importlib.import_module(name)
|
||||
except ImportError:
|
||||
continue
|
||||
if id(module.Client) in seen:
|
||||
continue
|
||||
seen.add(id(module.Client))
|
||||
original = module.Client.send
|
||||
|
||||
def wrap_send(send):
|
||||
def capture(client, request, *args, **kwargs):
|
||||
captured = "/chat/completions" in request.url.path
|
||||
if captured:
|
||||
body = json.loads(request.content)
|
||||
requests.append(body)
|
||||
append("api_requests.jsonl", body)
|
||||
response = send(client, request, *args, **kwargs)
|
||||
if captured:
|
||||
response.read()
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError:
|
||||
body = response.text
|
||||
append("api_responses.jsonl", {"status": response.status_code, "body": body})
|
||||
return response
|
||||
|
||||
return capture
|
||||
|
||||
module.Client.send = wrap_send(original)
|
||||
agent = None
|
||||
database = SessionDB(home / "state.db")
|
||||
started = time.monotonic()
|
||||
try:
|
||||
agent = AIAgent(
|
||||
base_url=values["LLM_BASE_URL"],
|
||||
api_key=values["LLM_API_KEY"],
|
||||
provider="custom",
|
||||
api_mode="chat_completions",
|
||||
model=task["model"],
|
||||
max_iterations=12,
|
||||
run_budget_seconds=240,
|
||||
max_tokens=1024,
|
||||
request_overrides={"reasoning_effort": "none", "extra_body": {"enable_thinking": False}},
|
||||
enabled_toolsets=["file"],
|
||||
save_trajectories=False,
|
||||
quiet_mode=True,
|
||||
skip_context_files=True,
|
||||
skip_memory=True,
|
||||
skip_background_review=True,
|
||||
session_db=database,
|
||||
cwd=str(work),
|
||||
platform="batch",
|
||||
)
|
||||
from tools.registry import registry
|
||||
|
||||
entry = registry.get_entry("read_file")
|
||||
original_read = entry.handler
|
||||
|
||||
def confined_read(args, **kwargs):
|
||||
path = Path(args.get("path", ""))
|
||||
resolved = (path if path.is_absolute() else work / path).resolve()
|
||||
if not resolved.is_relative_to(work) or str(resolved.relative_to(work)) not in task["source_hashes"]:
|
||||
return json.dumps({"error": "Only the listed source snapshots may be read"})
|
||||
return original_read(args, **kwargs)
|
||||
|
||||
entry.handler = confined_read
|
||||
agent.tools = [t for t in agent.tools if t["function"]["name"] == "read_file"]
|
||||
agent.valid_tool_names = {"read_file"}
|
||||
agent.context_compressor.enabled = False
|
||||
result = agent.run_conversation(
|
||||
task["prompt"],
|
||||
task_id=root.name,
|
||||
system_message=f"Investigate the supplied fixture evidence and propose actions only. "
|
||||
f"Use only read_file within {work}. "
|
||||
"Do not read environment files, credentials or any path outside this workspace.",
|
||||
)
|
||||
if not result.get("completed") or not requests:
|
||||
raise ValueError("Hermes collection did not complete")
|
||||
systems = [m for m in requests[0]["messages"] if m["role"] == "system"]
|
||||
history = [*systems, *result["messages"]]
|
||||
if len(systems) != 1 or history[-1]["role"] != "assistant" or history[-1].get("tool_calls"):
|
||||
raise ValueError("Incomplete system prompt or final assistant report")
|
||||
calls = {c["id"]: c for m in history for c in m.get("tool_calls", []) or []}
|
||||
results = [m for m in history if m["role"] == "tool"]
|
||||
if set(calls) != {m["tool_call_id"] for m in results}:
|
||||
raise ValueError("Unpaired tool calls")
|
||||
observed = set()
|
||||
for message in results:
|
||||
call = calls[message["tool_call_id"]]
|
||||
args = json.loads(call["function"]["arguments"])
|
||||
path = Path(args.get("path") or args.get("file_path"))
|
||||
path = (path if path.is_absolute() else work / path).resolve()
|
||||
output = json.loads(message["content"])
|
||||
if not path.is_relative_to(work) or str(path.relative_to(work)) not in task["source_hashes"]:
|
||||
if "error" not in output:
|
||||
raise ValueError("Unexpected content from outside the source allowlist")
|
||||
continue # Denied reads remain in the trajectory; they never reached the filesystem.
|
||||
text = "\n".join(re.sub(r"^\d+\|", "", line) for line in output.get("content", "").splitlines())
|
||||
if text.rstrip() == path.read_text(encoding="utf-8").rstrip():
|
||||
observed.add(str(path.relative_to(work)))
|
||||
if observed != set(task["source_hashes"]):
|
||||
raise ValueError("Not every source snapshot was read without truncation")
|
||||
if any(
|
||||
hashlib.sha256((work / name).read_bytes()).hexdigest() != sha for name, sha in task["source_hashes"].items()
|
||||
):
|
||||
raise ValueError("Source fixture changed")
|
||||
write(root / "trajectory.full.json", {"messages": history})
|
||||
write(root / "result.json", result)
|
||||
write(root / "session_export.json", database.export_session(agent.session_id, include_compacted=True))
|
||||
write(
|
||||
root / "collection.json",
|
||||
{
|
||||
"completed": True,
|
||||
"session_id": agent.session_id,
|
||||
"messages": len(history),
|
||||
"tool_calls": len(calls),
|
||||
"articles_read": len(observed),
|
||||
"source_unchanged": True,
|
||||
"seconds": time.monotonic() - started,
|
||||
"trajectory_sha256": hashlib.sha256((root / "trajectory.full.json").read_bytes()).hexdigest(),
|
||||
},
|
||||
)
|
||||
if any(
|
||||
secret.encode() in file.read_bytes() for file in root.rglob("*") if file.is_file() for secret in secrets
|
||||
):
|
||||
raise ValueError("Credential in artifacts")
|
||||
print(root.name, "collected", len(calls), "tool calls", flush=True)
|
||||
finally:
|
||||
if agent is not None:
|
||||
agent.close()
|
||||
database.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""Internal subprocess entry point; provider authorization belongs to the parent collector."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, required=True)
|
||||
parser.add_argument("--hermes", type=Path, required=True)
|
||||
parser.add_argument("--env-file", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
collect(args.root.resolve(), args.hermes.resolve(), args.env_file.resolve())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,139 +0,0 @@
|
|||
"""Run generated payment fixes with a deliberately restricted Python grammar."""
|
||||
|
||||
import ast
|
||||
from types import SimpleNamespace
|
||||
|
||||
|
||||
class PaymentError(Exception):
|
||||
"""A non-timeout provider error carrying an HTTP status."""
|
||||
|
||||
status_code = 422
|
||||
|
||||
|
||||
def repair_checks(source): # pylint: disable=too-many-return-statements
|
||||
"""Reject general Python programs; allow only a bounded retry function and inert clients."""
|
||||
names = ("success", "timeout_then_success", "http_402", "http_422", "exhaustion", "distinct_payments")
|
||||
failed = dict.fromkeys(names, False)
|
||||
try:
|
||||
tree = ast.parse(source)
|
||||
functions = [node for node in tree.body if isinstance(node, ast.FunctionDef)]
|
||||
if len(functions) != 1 or functions[0].name != "charge_with_retry":
|
||||
return failed
|
||||
for node in tree.body:
|
||||
if node is functions[0]:
|
||||
continue
|
||||
if not isinstance(node, ast.Import) or any(alias.name != "uuid" for alias in node.names):
|
||||
return failed
|
||||
allowed_calls = {"range", "str", "TimeoutError", "uuid.uuid4", "client.charge"}
|
||||
allowed_nodes = (
|
||||
ast.FunctionDef,
|
||||
ast.arguments,
|
||||
ast.arg,
|
||||
ast.Constant,
|
||||
ast.Name,
|
||||
ast.Load,
|
||||
ast.Store,
|
||||
ast.Assign,
|
||||
ast.For,
|
||||
ast.Call,
|
||||
ast.Attribute,
|
||||
ast.keyword,
|
||||
ast.Try,
|
||||
ast.ExceptHandler,
|
||||
ast.Return,
|
||||
ast.Raise,
|
||||
ast.If,
|
||||
ast.Compare,
|
||||
ast.Eq,
|
||||
ast.NotEq,
|
||||
ast.Lt,
|
||||
ast.LtE,
|
||||
ast.Gt,
|
||||
ast.GtE,
|
||||
ast.BinOp,
|
||||
ast.Sub,
|
||||
ast.Pass,
|
||||
ast.Expr,
|
||||
)
|
||||
for node in ast.walk(functions[0]):
|
||||
if not isinstance(node, allowed_nodes):
|
||||
return failed
|
||||
if isinstance(node, ast.FunctionDef) and node is not functions[0]:
|
||||
return failed
|
||||
if isinstance(node, ast.Constant) and (
|
||||
isinstance(node.value, str)
|
||||
and len(node.value) > 100
|
||||
or isinstance(node.value, int)
|
||||
and abs(node.value) > 10
|
||||
):
|
||||
return failed
|
||||
if isinstance(node, ast.Attribute) and ast.unparse(node) not in {"uuid.uuid4", "client.charge"}:
|
||||
return failed
|
||||
if isinstance(node, ast.Call) and ast.unparse(node.func) not in allowed_calls:
|
||||
return failed
|
||||
if isinstance(node, ast.Name) and node.id.startswith("__"):
|
||||
return failed
|
||||
if functions[0].decorator_list or functions[0].returns or any(a.annotation for a in functions[0].args.args):
|
||||
return failed
|
||||
# No executable defaults, annotations or decorators in the accepted grammar.
|
||||
if any(not isinstance(node, ast.Constant) for node in functions[0].args.defaults):
|
||||
return failed
|
||||
if functions[0].args.kwonlyargs or functions[0].args.vararg or functions[0].args.kwarg:
|
||||
return failed
|
||||
if any(arg.annotation for arg in ast.walk(functions[0]) if isinstance(arg, ast.arg)):
|
||||
return failed
|
||||
if sum(isinstance(node, ast.For) for node in ast.walk(functions[0])) > 1:
|
||||
return failed
|
||||
serial = iter(range(1000))
|
||||
|
||||
def bounded_range(*args):
|
||||
result = range(*args)
|
||||
if len(result) > 10:
|
||||
raise ValueError("Retry range exceeds test limit")
|
||||
return result
|
||||
|
||||
namespace = {
|
||||
"__builtins__": {},
|
||||
"range": bounded_range,
|
||||
"str": str,
|
||||
"TimeoutError": TimeoutError,
|
||||
"uuid": SimpleNamespace(uuid4=lambda: f"test-key-{next(serial)}"),
|
||||
}
|
||||
module = ast.Module(body=functions, type_ignores=[])
|
||||
exec(compile(module, "<payment-fix>", "exec"), namespace) # pylint: disable=exec-used
|
||||
fix = namespace["charge_with_retry"]
|
||||
|
||||
def attempt(events, expected_calls, error=None):
|
||||
keys = []
|
||||
remaining = iter(events)
|
||||
|
||||
def charge(invoice, idempotency_key):
|
||||
if invoice != "test-invoice":
|
||||
raise AssertionError("Invoice was changed")
|
||||
keys.append(idempotency_key)
|
||||
event = next(remaining)
|
||||
if isinstance(event, Exception):
|
||||
raise event
|
||||
return event
|
||||
|
||||
try:
|
||||
result = fix(SimpleNamespace(charge=charge), "test-invoice")
|
||||
correct = error is None and result == "ok"
|
||||
except Exception as exc:
|
||||
correct = error is not None and exc is error
|
||||
return correct and len(keys) == expected_calls and len(set(keys)) == 1, keys
|
||||
|
||||
result = {}
|
||||
result["success"], first = attempt(["ok"], 1)
|
||||
result["timeout_then_success"], _ = attempt([TimeoutError(), "ok"], 2)
|
||||
for status in (402, 422):
|
||||
error = PaymentError()
|
||||
error.status_code = status
|
||||
result[f"http_{status}"], _ = attempt([error, "ok"], 1, error)
|
||||
errors = [TimeoutError() for _ in range(3)]
|
||||
result["exhaustion"], _ = attempt(errors, 3, errors[-1])
|
||||
success, second = attempt(["ok"], 1)
|
||||
result["distinct_payments"] = success and result["success"] and first != second
|
||||
return result
|
||||
except Exception:
|
||||
return failed
|
||||
|
|
@ -1,17 +1,13 @@
|
|||
"""Frozen-summary experiment: Full context, S, and S + Jev Top 1–N."""
|
||||
"""Shared request budgeting, model calls and strict Top-N selection for the study."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
from dataclasses import asdict, dataclass, is_dataclass
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
|
||||
from dotenv import dotenv_values
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from benchmark.compaction.cases import SUMMARY, SOURCE_SHA256, evaluate, event_units, load_case, render, write
|
||||
from benchmark.compaction.cases import SUMMARY, write
|
||||
from reme.components.as_decision_model import AgentScopeAsDecisionModel
|
||||
from reme.steps.compaction.compact_context import probability
|
||||
|
||||
|
|
@ -151,264 +147,3 @@ def top_n(units, scores, n):
|
|||
raise ValueError("Scores must be finite probabilities")
|
||||
chosen = {u["id"] for u in sorted(units, key=lambda u: (-scores[u["id"]], u["id"]))[:n]}
|
||||
return [u for u in units if u["id"] in chosen]
|
||||
|
||||
|
||||
def prepare(experiment, case, units, summaries, scores, maximum):
|
||||
"""Freeze all inputs before answers; fusion never rewrites or shrinks its summary."""
|
||||
rows = []
|
||||
for index, summary in enumerate(summaries):
|
||||
for arm, n in [("full", 0), ("summary", 0), *[(f"fusion_top{n}", n) for n in range(1, maximum + 1)]]:
|
||||
chosen = top_n(units, scores[index], n)
|
||||
text = "" if arm == "full" else summary
|
||||
payload = None if arm == "full" else compose(text, render(chosen))
|
||||
context = messages(case, payload)
|
||||
rows.append(
|
||||
{
|
||||
"arm": arm,
|
||||
"repeat": index,
|
||||
"summary": text,
|
||||
"selected": [u["id"] for u in chosen],
|
||||
"estimated_input_tokens": experiment.input_tokens(context),
|
||||
"within_budget": arm == "full" or experiment.fits(case, payload),
|
||||
"valid": False,
|
||||
"trials": [],
|
||||
},
|
||||
)
|
||||
write(experiment.output / f"c{index}.{arm}.context.json", {"messages": context})
|
||||
return rows
|
||||
|
||||
|
||||
def aggregate(rows):
|
||||
"""Keep every planned condition in the denominator, including failed generations."""
|
||||
result = []
|
||||
for arm in dict.fromkeys(r["arm"] for r in rows):
|
||||
group = [r for r in rows if r["arm"] == arm]
|
||||
valid = [r for r in group if r["valid"]]
|
||||
trials = [t for r in valid for t in r["trials"]]
|
||||
metrics = {}
|
||||
for section in ("facts", "task"):
|
||||
metrics[section] = statistics.mean(
|
||||
(
|
||||
statistics.mean(sum(t["checks"][section].values()) / len(t["checks"][section]) for t in r["trials"])
|
||||
if r["valid"] and r["trials"]
|
||||
else 0
|
||||
)
|
||||
for r in group
|
||||
)
|
||||
result.append(
|
||||
{
|
||||
"arm": arm,
|
||||
"valid": len(valid),
|
||||
"planned": len(group),
|
||||
"input_tokens": statistics.mean(t["usage"]["prompt_tokens"] for t in trials) if trials else None,
|
||||
**metrics,
|
||||
},
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def report(output, rows):
|
||||
"""Write one compact table; retain detailed provenance outside the repository."""
|
||||
table = aggregate(rows)
|
||||
write(output / "aggregate.json", table)
|
||||
lines = ["| Method | Input tokens | Fact accuracy | Repair pass |", "| --- | ---: | ---: | ---: |"]
|
||||
for row in table:
|
||||
label = {"full": "Full context", "summary": "S"}.get(
|
||||
row["arm"],
|
||||
row["arm"].replace("fusion_top", "S + Jev Top "),
|
||||
)
|
||||
tokens = f"{row['input_tokens']:.0f}" if row["input_tokens"] is not None else "—"
|
||||
lines.append(f"| {label} | {tokens} | {row['facts']:.1%} | {row['task']:.1%} |")
|
||||
lines += ["", "Valid/planned: " + "; ".join(f"{r['arm']} {r['valid']}/{r['planned']}" for r in table)]
|
||||
(output / "REPORT.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def frozen_inputs(path, case, units, count):
|
||||
"""Load every previously frozen summary/ranking pair, never choose by answer accuracy."""
|
||||
source = json.loads((path / "source.json").read_text())
|
||||
if any(source[k] != case[k] for k in ("messages", "exam", "gold")) or source["units"] != units:
|
||||
raise ValueError("Frozen evidence does not match the original trajectory")
|
||||
rows = json.loads((path / "results.json").read_text())
|
||||
baseline = [r for r in rows if r["arm"] == "summary"]
|
||||
if len(baseline) != count or {r["repeat"] for r in baseline} != set(range(count)):
|
||||
raise ValueError("Use all frozen summary generations with the matching --compressions count")
|
||||
summaries = [next(r["summary"] for r in baseline if r["repeat"] == i) for i in range(count)]
|
||||
scores = [
|
||||
json.loads((path / "provenance" / f"paired.c{i}.decision0.response.json").read_text())["scores"]
|
||||
for i in range(count)
|
||||
]
|
||||
if any(not isinstance(s, str) or not s for s in summaries):
|
||||
raise ValueError("Every frozen summary must be nonempty text")
|
||||
return summaries, scores
|
||||
|
||||
|
||||
async def answer(experiment, row, case, repeats):
|
||||
"""Only complete answers within the actual downstream input ceiling receive credit."""
|
||||
label = f"c{row['repeat']}.{row['arm']}"
|
||||
try:
|
||||
if not row["within_budget"]:
|
||||
raise ValueError("Frozen context exceeds input ceiling; increase the ceiling without truncating evidence")
|
||||
context = json.loads((experiment.output / f"{label}.context.json").read_text())["messages"]
|
||||
trials = []
|
||||
for index in range(repeats):
|
||||
text, usage = await experiment.ask(f"{label}.answer{index}", context)
|
||||
if row["arm"] != "full" and usage["prompt_tokens"] > experiment.cap:
|
||||
raise ValueError("Actual API input exceeds ceiling")
|
||||
parsed = json.loads(text)
|
||||
trials.append({"answer": parsed, "usage": usage, "checks": evaluate(case, parsed)})
|
||||
row.update(valid=True, trials=trials)
|
||||
except Exception as exc:
|
||||
row.update(valid=False, error=type(exc).__name__, trials=[])
|
||||
print(label, "valid" if row["valid"] else row["error"], flush=True)
|
||||
|
||||
|
||||
async def run(args):
|
||||
"""Fresh runs create one S/ranking per generation; replay keeps them byte-identical."""
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
output = args.output.resolve()
|
||||
if output.exists() or output == repo or repo in output.parents:
|
||||
raise ValueError("Use a new output directory outside the repository")
|
||||
case = load_case(args.trajectory)
|
||||
units = event_units(case["messages"])
|
||||
if args.top_n > len(units):
|
||||
raise ValueError(f"Only {len(units)} evidence units: maximum Top {len(units)}")
|
||||
if args.summary_cap > args.input_cap:
|
||||
raise ValueError("Summary generation cap cannot exceed the final input cap")
|
||||
frozen = frozen_inputs(args.frozen_run, case, units, args.compressions) if args.frozen_run else None
|
||||
values = dotenv_values(repo / ".env") if not args.check_only else {}
|
||||
if not args.check_only and any(not values.get(k) for k in ("LLM_API_KEY", "LLM_BASE_URL")):
|
||||
raise ValueError("Missing authorized LLM configuration")
|
||||
if not frozen and not values.get("DECISION_MODEL_API_KEY"):
|
||||
raise ValueError("Fresh compression requires DECISION_MODEL_API_KEY")
|
||||
output.mkdir(parents=True)
|
||||
write(output / "source.json", {**case, "units": units})
|
||||
for name in ("run.py", "cases.py", "repair_check.py"):
|
||||
(output / name).write_bytes((Path(__file__).parent / name).read_bytes())
|
||||
write(
|
||||
output / "protocol.json",
|
||||
{
|
||||
"source_sha256": SOURCE_SHA256,
|
||||
"tokenizer_sha256": hashlib.sha256(args.tokenizer.read_bytes()).hexdigest(),
|
||||
"model": args.model,
|
||||
"summary_generation_input_cap": args.summary_cap,
|
||||
"compressed_input_cap": args.input_cap,
|
||||
"top_n": args.top_n,
|
||||
"evidence_count": len(units),
|
||||
"compressions": args.compressions,
|
||||
"answer_repeats": args.repeats,
|
||||
"summary_frozen": True,
|
||||
"selection": "Strict score prefix; no threshold; restore source order",
|
||||
"frozen_run": str(args.frozen_run.resolve()) if args.frozen_run else None,
|
||||
"check_only": args.check_only,
|
||||
"evaluation": "Direct Qwen; not Hermes continuation",
|
||||
},
|
||||
)
|
||||
client = None
|
||||
try:
|
||||
if not args.check_only:
|
||||
client = AsyncOpenAI(
|
||||
api_key=values["LLM_API_KEY"],
|
||||
base_url=values["LLM_BASE_URL"],
|
||||
timeout=90,
|
||||
max_retries=1,
|
||||
)
|
||||
experiment = Experiment(
|
||||
client,
|
||||
None,
|
||||
output,
|
||||
Tokenizer.from_file(str(args.tokenizer)),
|
||||
asyncio.Semaphore(args.concurrency),
|
||||
args.model,
|
||||
args.input_cap,
|
||||
)
|
||||
if frozen:
|
||||
summaries, scores = frozen
|
||||
else:
|
||||
credential = {
|
||||
"api_key": values["DECISION_MODEL_API_KEY"],
|
||||
"base_url": values.get("DECISION_MODEL_BASE_URL") or None,
|
||||
}
|
||||
model = values.get("DECISION_MODEL_NAME")
|
||||
model = None if model in (None, "", "null") else model
|
||||
summaries, scores = [], []
|
||||
async with AgentScopeAsDecisionModel(credential=credential, model=model, timeout=45) as decision:
|
||||
experiment.decision = decision
|
||||
allowance = args.summary_cap - RESERVE - experiment.input_tokens(messages(case, compose("", ""))) - 8
|
||||
if allowance < 40:
|
||||
raise ValueError("Insufficient summary capacity")
|
||||
for index in range(args.compressions):
|
||||
label = f"paired.c{index}"
|
||||
ranking, _ = await experiment.scores(label, case["messages"][1]["content"], units)
|
||||
summary, _ = await experiment.summarize(label, case["messages"][1]["content"], units, allowance)
|
||||
scores.append(ranking)
|
||||
summaries.append(summary)
|
||||
# Persist compressor outputs so a later replay can use them without new provider calls.
|
||||
(output / "provenance").mkdir()
|
||||
for index, ranking in enumerate(scores):
|
||||
write(output / "provenance" / f"paired.c{index}.decision0.response.json", {"scores": ranking})
|
||||
rows = prepare(experiment, case, units, summaries, scores, args.top_n)
|
||||
write(output / "frozen-selections.json", rows)
|
||||
write(output / "results.json", rows)
|
||||
if not args.check_only:
|
||||
work = asyncio.Semaphore(args.concurrency)
|
||||
|
||||
async def condition(row):
|
||||
async with work:
|
||||
await answer(experiment, row, case, args.repeats)
|
||||
write(output / "results.json", rows)
|
||||
report(output, rows)
|
||||
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for row in rows:
|
||||
tasks.create_task(condition(row))
|
||||
load_case(args.trajectory)
|
||||
secrets = [values[k] for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY") if values.get(k)]
|
||||
if any(
|
||||
secret.encode() in file.read_bytes() for file in output.rglob("*") if file.is_file() for secret in secrets
|
||||
):
|
||||
raise RuntimeError("Credential in artifacts")
|
||||
write(
|
||||
output / "validation.json",
|
||||
{
|
||||
"source_unchanged": True,
|
||||
"no_api_keys": True,
|
||||
"prepared": len(rows),
|
||||
"within_budget": sum(r["within_budget"] for r in rows),
|
||||
"valid": sum(r["valid"] for r in rows),
|
||||
"check_only": args.check_only,
|
||||
},
|
||||
)
|
||||
finally:
|
||||
if client is not None:
|
||||
await client.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""Require live opt-in or an offline frozen-input check; never overwrite existing runs."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--trajectory", type=Path, required=True)
|
||||
parser.add_argument("--tokenizer", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--frozen-run", type=Path)
|
||||
parser.add_argument("--summary-cap", type=int, default=3200)
|
||||
parser.add_argument("--input-cap", type=int, default=3800)
|
||||
parser.add_argument("--top-n", type=int, default=5)
|
||||
parser.add_argument("--compressions", type=int, default=5)
|
||||
parser.add_argument("--repeats", type=int, default=3)
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
parser.add_argument("--model", default="qwen3.8-max")
|
||||
mode = parser.add_mutually_exclusive_group(required=True)
|
||||
mode.add_argument("--live", action="store_true")
|
||||
mode.add_argument("--check-only", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if min(args.summary_cap, args.input_cap, args.top_n, args.compressions, args.repeats, args.concurrency) < 1:
|
||||
parser.error("All numerical settings must be positive")
|
||||
if args.check_only and not args.frozen_run:
|
||||
parser.error("--check-only requires --frozen-run")
|
||||
asyncio.run(run(args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
455
benchmark/compaction/suite.py
Normal file
455
benchmark/compaction/suite.py
Normal file
|
|
@ -0,0 +1,455 @@
|
|||
"""Freeze rule-scored, read-only adaptations of fourteen distinct fixture tasks."""
|
||||
|
||||
import ast
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import re
|
||||
import random
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from benchmark.compaction.cases import GOLD, exact_match, write
|
||||
|
||||
|
||||
def module(path, name):
|
||||
"""Load fixture definitions only; never run their CLI or agent workers."""
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
value = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = value
|
||||
spec.loader.exec_module(value)
|
||||
return value
|
||||
|
||||
|
||||
def context(text, pattern, radius=10):
|
||||
"""Freeze original matching lines with context, with no generated filler or answer lookup."""
|
||||
lines = text.splitlines(keepends=True)
|
||||
indices = {
|
||||
j
|
||||
for i, line in enumerate(lines)
|
||||
if re.search(pattern, line)
|
||||
for j in range(max(0, i - radius), min(len(lines), i + radius + 1))
|
||||
}
|
||||
return "".join(f"{i+1}|{lines[i]}" for i in sorted(indices))
|
||||
|
||||
|
||||
def planted_session(path, function, session_id, position):
|
||||
"""Extract a literal upstream seeded history without running DB setup or worker code."""
|
||||
tree = ast.parse(path.read_text())
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == function:
|
||||
if any(isinstance(a, ast.Constant) and a.value == session_id for a in node.args):
|
||||
return ast.literal_eval(node.args[position])
|
||||
raise ValueError("Expected upstream seeded session")
|
||||
|
||||
|
||||
def define(task_id, query, files, gold, fields, origin, *, query_only=False, unordered=()):
|
||||
"""Keep fixture answers outside the read-only tool workspace and compressor inputs."""
|
||||
return {
|
||||
"id": task_id,
|
||||
"query": query,
|
||||
"files": files,
|
||||
"gold": gold,
|
||||
"fields": fields,
|
||||
"origin": origin,
|
||||
"query_only": query_only,
|
||||
"unordered": list(unordered),
|
||||
}
|
||||
|
||||
|
||||
def definitions(hermes, scratch):
|
||||
"""Use pinned upstream fixture bytes; adapt actions to read-only investigation explicitly."""
|
||||
core = module(hermes / "evals/core_tool_deferral/tasks.py", "compaction_core_tasks")
|
||||
perf = module(hermes / "evals/toolperf_abeval/ab_eval.py", "compaction_perf_tasks")
|
||||
read = module(hermes / "evals/readtool/fixtures.py", "compaction_read_fixtures")
|
||||
core_root, perf_root, read_root = (scratch / name for name in ("core", "perf", "read"))
|
||||
for root in (core_root, perf_root, read_root):
|
||||
root.mkdir(parents=True)
|
||||
for name in ("longrange_backup_pipeline", "eager_refactor_control", "todo_three_files", "config_grep_distractor"):
|
||||
root = core_root / name
|
||||
root.mkdir()
|
||||
core.TASKS_BY_ID[name]["fixtures"](str(root))
|
||||
perf.make_sandbox(perf_root)
|
||||
read._write_package_json(read_root) # pylint: disable=protected-access
|
||||
read._write_app_js(read_root) # pylint: disable=protected-access
|
||||
read._write_unicode_notes(read_root) # pylint: disable=protected-access
|
||||
read._write_server_log(read_root, random.Random(20260809)) # pylint: disable=protected-access
|
||||
read._write_lockfile(read_root, random.Random(20260809)) # pylint: disable=protected-access
|
||||
backup = planted_session(hermes / "evals/core_tool_deferral/worker.py", "sess", "seed_backup_talk", 1)
|
||||
incident = planted_session(hermes / "evals/session_search_schema/fixtures.py", "mk", "20260818_140200_cc22dd", 4)
|
||||
|
||||
def core_files(name):
|
||||
root = core_root / name
|
||||
return {str(p.relative_to(root)): p.read_text() for p in root.rglob("*") if p.is_file()}
|
||||
|
||||
def core_query(name):
|
||||
return core.TASKS_BY_ID[name]["prompt"]
|
||||
|
||||
def perf_files(*names):
|
||||
return {name: (perf_root / "proj" / name).read_text() for name in names}
|
||||
|
||||
def perf_query(name):
|
||||
return perf.TASKS[name].replace("{WORK}/proj", ".")
|
||||
|
||||
rows = []
|
||||
rows.append(
|
||||
define(
|
||||
"payment",
|
||||
"Investigate the original payment retry incident.",
|
||||
{},
|
||||
GOLD,
|
||||
{k: k.replace("_", " ") for k in GOLD},
|
||||
"Original Hermes payment fixture",
|
||||
unordered=("no_retry_http_statuses",),
|
||||
),
|
||||
)
|
||||
files = core_files("longrange_backup_pipeline")
|
||||
files["session_history.json"] = json.dumps(backup, indent=2)
|
||||
rows.append(
|
||||
define(
|
||||
"backup",
|
||||
core_query("longrange_backup_pipeline"),
|
||||
files,
|
||||
{"repo": "sparks-data", "ignored_repo": "toybox", "data_text": "important"},
|
||||
{
|
||||
"repo": "repository requiring backup",
|
||||
"ignored_repo": "repository explicitly excluded",
|
||||
"data_text": "contents of the target repository data file",
|
||||
},
|
||||
"core_tool_deferral/longrange_backup_pipeline",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"refactor",
|
||||
core_query("eager_refactor_control"),
|
||||
core_files("eager_refactor_control"),
|
||||
{
|
||||
"old_timeout": 30,
|
||||
"call_sites": 3,
|
||||
"functions": ["fetch_a", "fetch_b", "fetch_c"],
|
||||
"methods": ["get", "post", "head"],
|
||||
"old_version": "1.0",
|
||||
},
|
||||
{
|
||||
"old_timeout": "original timeout integer",
|
||||
"call_sites": "number of requests call sites",
|
||||
"functions": "function names in source order",
|
||||
"methods": "HTTP methods in source order",
|
||||
"old_version": "existing changelog version",
|
||||
},
|
||||
"core_tool_deferral/eager_refactor_control",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"three_files",
|
||||
core_query("todo_three_files"),
|
||||
core_files("todo_three_files"),
|
||||
{"file_count": 3, "header": "item,qty", "data_rows": 2},
|
||||
{"file_count": "required output file count", "header": "CSV header", "data_rows": "CSV data row count"},
|
||||
"core_tool_deferral/todo_three_files",
|
||||
query_only=True,
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"active_config",
|
||||
core_query("config_grep_distractor"),
|
||||
core_files("config_grep_distractor"),
|
||||
{"retry_limit": 7, "backoff": 2, "active_file": "conf/app.ini", "old_file": "conf/old.ini"},
|
||||
{
|
||||
"retry_limit": "active integer value",
|
||||
"backoff": "active integer backoff",
|
||||
"active_file": "workspace-relative active config",
|
||||
"old_file": "workspace-relative retired config",
|
||||
},
|
||||
"core_tool_deferral/config_grep_distractor",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"checkout_504",
|
||||
(
|
||||
"Investigate the checkout 504 incident: distinguish the failed rollback, "
|
||||
"root cause and final database fix."
|
||||
),
|
||||
{"session_history.json": json.dumps(incident, indent=2)},
|
||||
{
|
||||
"setting": "statement_timeout",
|
||||
"value_seconds": 45,
|
||||
"index_mode": "CONCURRENTLY",
|
||||
"root_cause": "CREATE INDEX without CONCURRENTLY",
|
||||
"rolled_back_version": "v2.30.2",
|
||||
"rollback_fixed": False,
|
||||
"deployed_version": "v2.31.0",
|
||||
"initial_error_percent": 8,
|
||||
},
|
||||
{
|
||||
"setting": "final database timeout setting name",
|
||||
"value_seconds": "final timeout in seconds",
|
||||
"index_mode": "mode used to recreate the index",
|
||||
"root_cause": "SQL operation that competed with checkout writes",
|
||||
"rolled_back_version": "rollback target version",
|
||||
"rollback_fixed": "did rollback resolve the incident, boolean",
|
||||
"deployed_version": "version deployed before the incident",
|
||||
"initial_error_percent": "initial request error percentage as integer",
|
||||
},
|
||||
"session_search_schema/t2_scroll",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"dependency_version",
|
||||
(
|
||||
"What exact left-pad version is pinned, and which file declares the test script? "
|
||||
"Record the package scripts for a handoff."
|
||||
),
|
||||
{
|
||||
"package.json": (read_root / "package.json").read_text(),
|
||||
"package-lock.head.txt": "".join(
|
||||
(read_root / "package-lock.json").read_text().splitlines(keepends=True)[:40],
|
||||
),
|
||||
},
|
||||
{
|
||||
"version": "1.3.0",
|
||||
"declaration": "package.json",
|
||||
"test": "vitest run",
|
||||
"build": "tsc -p .",
|
||||
"package_version": "4.2.1",
|
||||
},
|
||||
{
|
||||
"version": "pinned left-pad version",
|
||||
"declaration": "file declaring the test script",
|
||||
"test": "exact test command",
|
||||
"build": "exact build command",
|
||||
"package_version": "project package version",
|
||||
},
|
||||
"readtool/lockfile_version",
|
||||
),
|
||||
)
|
||||
tree = ast.parse((hermes / "evals/readtool/fixtures.py").read_text())
|
||||
function = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "_write_minified_bundle")
|
||||
parts = next(
|
||||
n
|
||||
for n in ast.walk(function)
|
||||
if isinstance(n, ast.Assign) and any(isinstance(t, ast.Name) and t.id == "parts" for t in n.targets)
|
||||
)
|
||||
fragment = ast.literal_eval(parts.value)[0]
|
||||
rows.append(
|
||||
define(
|
||||
"retry_backoff",
|
||||
"Find the retry-delay function in src/ and explain its base, growth, cap and jitter.",
|
||||
{"src/app.js": (read_root / "src/app.js").read_text(), "src/app.min.head.txt": fragment},
|
||||
{
|
||||
"base_ms": 250,
|
||||
"cap_ms": 30000,
|
||||
"growth_factor": 2,
|
||||
"jitter": "uniform_integer_below_bound",
|
||||
"function": "retryDelay",
|
||||
},
|
||||
{
|
||||
"base_ms": "base delay in milliseconds",
|
||||
"cap_ms": "maximum delay in milliseconds",
|
||||
"growth_factor": "exponential multiplier per attempt",
|
||||
"jitter": "choose none or uniform_integer_below_bound",
|
||||
"function": "function name",
|
||||
},
|
||||
"readtool/minified_backoff",
|
||||
),
|
||||
)
|
||||
log = (read_root / "logs/server.log").read_text()
|
||||
rows.append(
|
||||
define(
|
||||
"server_error",
|
||||
(
|
||||
"Locate the unique ERROR in the server log and retain its exact request, timestamp, "
|
||||
"upstream and connection failure."
|
||||
),
|
||||
{
|
||||
"server.head.txt": "".join(log.splitlines(keepends=True)[:20]),
|
||||
"server.error-context.txt": context(log, r"\bERROR\b"),
|
||||
},
|
||||
{
|
||||
"request_id": "req-7f3d9",
|
||||
"timestamp": "2026-08-08T23:41:17Z",
|
||||
"status": 502,
|
||||
"upstream": "payments",
|
||||
"error": "ECONNREFUSED",
|
||||
"address": "10.0.4.17:8443",
|
||||
},
|
||||
{
|
||||
"request_id": "error request ID",
|
||||
"timestamp": "full UTC error timestamp",
|
||||
"status": "HTTP integer status",
|
||||
"upstream": "failed upstream service",
|
||||
"error": "connection error code",
|
||||
"address": "upstream host and port",
|
||||
},
|
||||
"readtool/log_error_hunt",
|
||||
),
|
||||
)
|
||||
notes = next((read_root / "notes").iterdir())
|
||||
rows.append(
|
||||
define(
|
||||
"meeting_notes",
|
||||
"Read the meeting notes and report the third bullet exactly, retaining the other agreed actions.",
|
||||
{"notes.txt": notes.read_text()},
|
||||
{
|
||||
"third": "rotate the API keys quarterly",
|
||||
"first": "ship the payments retry fix",
|
||||
"second": "audit the staging TLS certs",
|
||||
"fourth": "close out the Q3 incident review",
|
||||
},
|
||||
{k: f"exact {k} bullet" for k in ("third", "first", "second", "fourth")},
|
||||
"readtool/unicode_filename",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"targeted_edit",
|
||||
perf_query("err_ambiguous_edit"),
|
||||
perf_files("handlers.py", "check_handlers.py"),
|
||||
{
|
||||
"target": "slow_handler",
|
||||
"original_timeouts": {"fast_handler": 10, "slow_handler": 10, "medium_handler": 10},
|
||||
"required_timeouts": {"fast_handler": 10, "slow_handler": 60, "medium_handler": 10},
|
||||
"success_marker": "HANDLERS_OK_552",
|
||||
},
|
||||
{
|
||||
"target": "function requiring modification",
|
||||
"original_timeouts": "mapping of all original function timeouts",
|
||||
"required_timeouts": "mapping after the requested fix",
|
||||
"success_marker": "checker success marker",
|
||||
},
|
||||
"toolperf_abeval/err_ambiguous_edit",
|
||||
),
|
||||
)
|
||||
output = subprocess.check_output([sys.executable, str(perf_root / "proj/noisy_build.py")], text=True)
|
||||
rows.append(
|
||||
define(
|
||||
"build_token",
|
||||
perf_query("err_big_output"),
|
||||
{
|
||||
**perf_files("noisy_build.py"),
|
||||
"build.head.txt": "".join(output.splitlines(keepends=True)[:20]),
|
||||
"build.token-context.txt": context(output, r"^UNIQUE_TOKEN="),
|
||||
},
|
||||
{"token": "tok_9f31c_middle", "step": 2000, "total_steps": 4000, "prefix": "UNIQUE_TOKEN="},
|
||||
{
|
||||
"token": "full unique token value without its prefix",
|
||||
"step": "build loop step that prints the token",
|
||||
"total_steps": "build loop iteration count",
|
||||
"prefix": "exact token line prefix",
|
||||
},
|
||||
"toolperf_abeval/err_big_output",
|
||||
),
|
||||
)
|
||||
rows.append(
|
||||
define(
|
||||
"package_versions",
|
||||
perf_query("err_multi_dir"),
|
||||
perf_files("pkg_a/version.txt", "pkg_b/version.txt", "pkg_c/version.txt"),
|
||||
{"versions": {"pkg_a": "1.4.2", "pkg_b": "0.9.7", "pkg_c": "3.2.1"}, "joined": "1.4.2,0.9.7,3.2.1"},
|
||||
{
|
||||
"versions": "mapping from package directory to exact version",
|
||||
"joined": "comma-separated versions in a,b,c order",
|
||||
},
|
||||
"toolperf_abeval/err_multi_dir",
|
||||
),
|
||||
)
|
||||
anomaly = (perf_root / "proj/records.log").read_text()
|
||||
rows.append(
|
||||
define(
|
||||
"checksum_anomaly",
|
||||
perf_query("err_big_file_read"),
|
||||
{
|
||||
"records.head.txt": "".join(anomaly.splitlines(keepends=True)[:20]),
|
||||
"records.anomaly-context.txt": context(anomaly, r"^ANOMALY:"),
|
||||
},
|
||||
{
|
||||
"line": "ANOMALY: checksum drift detected in shard 7 (code X99Q)",
|
||||
"code": "X99Q",
|
||||
"shard": 7,
|
||||
"line_number": 4201,
|
||||
},
|
||||
{
|
||||
"line": "full anomaly line without display line number",
|
||||
"code": "anomaly code",
|
||||
"shard": "integer shard number",
|
||||
"line_number": "one-based original line number",
|
||||
},
|
||||
"toolperf_abeval/err_big_file_read",
|
||||
),
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def exam(case):
|
||||
"""Supply field names/descriptions, never their expected values, to the answering model."""
|
||||
prompt = (
|
||||
'Using only recorded evidence, return JSON {"facts": {...}}. No tools. Missing facts must be null. '
|
||||
"Use exact strings and JSON types; numbers are integers. Answer these fields: "
|
||||
+ json.dumps(case["fields"], ensure_ascii=False)
|
||||
)
|
||||
if case["id"] == "payment":
|
||||
prompt += (
|
||||
" regression_attempt_counts must be a JSON object with keys "
|
||||
"timeout_then_success, validation_422 and three_timeouts, each mapping to an integer count; "
|
||||
"do not return an array."
|
||||
)
|
||||
if case["id"] == "meeting_notes":
|
||||
prompt += " Return the bullet text without its leading list marker (-), indentation or surrounding whitespace."
|
||||
return prompt
|
||||
|
||||
|
||||
def grade(case, prediction):
|
||||
"""Type-strict field checks; declared unordered lists ignore order, with no LLM judge."""
|
||||
observed = prediction.get("facts", {}) if isinstance(prediction, dict) else {}
|
||||
observed = observed if isinstance(observed, dict) else {}
|
||||
checks = {}
|
||||
for key, expected in case["gold"].items():
|
||||
value = observed.get(key)
|
||||
if key in case["unordered"] and isinstance(value, list):
|
||||
value, expected = sorted(value, key=str), sorted(expected, key=str)
|
||||
checks[key] = exact_match(value, expected)
|
||||
return checks
|
||||
|
||||
|
||||
def freeze(hermes, output):
|
||||
"""Write all queries, answers and rules before model calls; runtime artifacts stay outside git."""
|
||||
output.mkdir(parents=True)
|
||||
rows = definitions(hermes, output / "upstream-fixtures")
|
||||
for case in rows:
|
||||
root = output / case["id"]
|
||||
work = root / "workspace"
|
||||
work.mkdir(parents=True)
|
||||
for name, text in case["files"].items():
|
||||
path = work / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text)
|
||||
case["source_chars"] = sum(map(len, case["files"].values()))
|
||||
case["control_reason"] = (
|
||||
"query_only" if case["query_only"] else "short_fixture" if case["source_chars"] < 2000 else None
|
||||
)
|
||||
if case["id"] == "payment":
|
||||
case["control_reason"] = None
|
||||
files = case.pop("files")
|
||||
case["source_hashes"] = {name: hashlib.sha256(text.encode()).hexdigest() for name, text in files.items()}
|
||||
prompt = (
|
||||
"Read-only evidence collection for this task: " + case["query"] + "\n"
|
||||
"Read ALL of these snapshot files completely using read_file: " + ", ".join(files) + ". "
|
||||
"Then report findings and proposed actions only. Do not modify files or run commands. "
|
||||
"Snapshots preserve selected original fixture regions; display line numbers refer to the original file."
|
||||
)
|
||||
write(
|
||||
root / "collection-task.json",
|
||||
{"prompt": prompt, "model": "qwen3.8-max", "source_hashes": case["source_hashes"]},
|
||||
)
|
||||
manifest = {
|
||||
"hermes_commit": subprocess.check_output(["git", "-C", str(hermes), "rev-parse", "HEAD"], text=True).strip(),
|
||||
"source": "Local Hermes MIT-licensed task fixtures; explicitly adapted read-only",
|
||||
"cases": rows,
|
||||
}
|
||||
write(output / "corpus.json", manifest)
|
||||
return manifest
|
||||
88
benchmark/compaction/suite_collect.py
Normal file
88
benchmark/compaction/suite_collect.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
"""Freeze the fourteen-task oracle panel, then collect real read-only Hermes trajectories."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
|
||||
from benchmark.compaction.cases import SOURCE_SHA256, write
|
||||
from benchmark.compaction.suite import freeze
|
||||
|
||||
|
||||
async def run(args):
|
||||
"""Run isolated finite subprocesses; preserve every planned task and failure record."""
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
output = args.output.resolve()
|
||||
if output.exists() or output == repo or repo in output.parents:
|
||||
raise ValueError("Use a new directory outside repository")
|
||||
payment = args.payment_trajectory.read_bytes()
|
||||
if hashlib.sha256(payment).hexdigest() != SOURCE_SHA256:
|
||||
raise ValueError("Expected the pinned original payment source")
|
||||
manifest = freeze(args.hermes.resolve(), output)
|
||||
shutil.copy2(args.payment_trajectory, output / "payment/trajectory.full.json")
|
||||
write(output / "payment/collection.json", {"completed": True, "reused": True, "trajectory_sha256": SOURCE_SHA256})
|
||||
for name in ("suite.py", "suite_collect.py", "hermes_worker.py", "STUDY.md"):
|
||||
shutil.copy2(Path(__file__).parent / name, output / name)
|
||||
semaphore = asyncio.Semaphore(args.concurrency)
|
||||
statuses = []
|
||||
|
||||
async def collect(case):
|
||||
if case["id"] == "payment":
|
||||
return
|
||||
root = output / case["id"]
|
||||
async with semaphore:
|
||||
with (root / "worker.log").open("w") as log:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
str(args.hermes_python),
|
||||
"-m",
|
||||
"benchmark.compaction.hermes_worker",
|
||||
"--root",
|
||||
str(root),
|
||||
"--hermes",
|
||||
str(args.hermes.resolve()),
|
||||
"--env-file",
|
||||
str(repo / ".env"),
|
||||
cwd=repo,
|
||||
stdout=log,
|
||||
stderr=log,
|
||||
)
|
||||
try:
|
||||
code = await asyncio.wait_for(process.wait(), 360)
|
||||
except BaseException:
|
||||
if process.returncode is None:
|
||||
process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 10)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise
|
||||
statuses.append({"id": case["id"], "completed": code == 0, "returncode": code})
|
||||
write(output / "collection-status.json", statuses)
|
||||
print(case["id"], "collected" if code == 0 else "FAILED", flush=True)
|
||||
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for case in manifest["cases"]:
|
||||
tasks.create_task(collect(case))
|
||||
if any(not s["completed"] for s in statuses):
|
||||
raise RuntimeError("Frozen collection failures remain; inspect retained logs")
|
||||
|
||||
|
||||
def main():
|
||||
"""Only explicit live collection can contact the configured provider."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--hermes", type=Path, required=True)
|
||||
parser.add_argument("--hermes-python", type=Path, required=True)
|
||||
parser.add_argument("--payment-trajectory", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
parser.add_argument("--live", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if not args.live or args.concurrency < 1:
|
||||
parser.error("Use --live and positive concurrency")
|
||||
asyncio.run(run(args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
240
benchmark/compaction/suite_native.py
Normal file
240
benchmark/compaction/suite_native.py
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
"""Score forced native summaries alongside the fixed seven-method task panel."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
from copy import deepcopy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
|
||||
from dotenv import dotenv_values
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from benchmark.compaction.cases import write
|
||||
from benchmark.compaction.run import Experiment, compose, messages
|
||||
from benchmark.compaction.suite import grade
|
||||
from benchmark.compaction.suite_run import ARMS, load_cases, report
|
||||
|
||||
|
||||
def validate_base(evaluation, corpus, cases):
|
||||
"""Reject incompatible base panels before launching any paid native generation."""
|
||||
protocol = json.loads((evaluation / "protocol.json").read_text())
|
||||
expected = {
|
||||
"model": "qwen3.8-max",
|
||||
"compressions": 3,
|
||||
"answer_repeats": 2,
|
||||
"corpus_sha256": hashlib.sha256(corpus.read_bytes()).hexdigest(),
|
||||
}
|
||||
rows = json.loads((evaluation / "results.json").read_text())
|
||||
planned = {(key, arm, i) for key in cases for arm in ARMS for i in range(3)}
|
||||
actual = {(r["case"], r["arm"], r["repeat"]) for r in rows}
|
||||
if len(cases) != 14 or any(protocol.get(k) != v for k, v in expected.items()):
|
||||
raise ValueError("Native adapter requires the matching fourteen-task Qwen 3x2 panel")
|
||||
if actual != planned or len(rows) != len(planned):
|
||||
raise ValueError("Base evaluation must contain each planned context exactly once")
|
||||
return rows
|
||||
|
||||
|
||||
async def prepare_native(args, cases, repo):
|
||||
"""Prepare a new isolated native generation set or validate an existing frozen set."""
|
||||
if args.native.exists():
|
||||
if not all((args.native / key / "generation/summaries.json").is_file() for key in cases):
|
||||
raise ValueError("Native generation set is incomplete; preserve it and use a new path")
|
||||
return
|
||||
if not args.hermes or not args.hermes_python:
|
||||
raise ValueError("New native generations require --hermes and --hermes-python")
|
||||
native = args.native.resolve()
|
||||
if native == repo or repo in native.parents:
|
||||
raise ValueError("Native artifacts must stay outside repository")
|
||||
native.mkdir(parents=True)
|
||||
for name in ("hermes_summary.py", "STUDY.md"):
|
||||
shutil.copy2(Path(__file__).parent / name, native / name)
|
||||
limit = asyncio.Semaphore(4)
|
||||
|
||||
async def worker(key):
|
||||
async with limit:
|
||||
root = native / key
|
||||
root.mkdir()
|
||||
env = dict(os.environ)
|
||||
env["PYTHONPATH"] = str(repo) + os.pathsep + env.get("PYTHONPATH", "")
|
||||
with (root / "worker.log").open("w", encoding="utf-8") as log:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
str(args.hermes_python),
|
||||
"-m",
|
||||
"benchmark.compaction.hermes_summary",
|
||||
"--source",
|
||||
str((args.evaluation / key / "source.json").resolve()),
|
||||
"--output",
|
||||
str(root / "generation"),
|
||||
"--hermes",
|
||||
str(args.hermes.resolve()),
|
||||
"--env-file",
|
||||
str(repo / ".env"),
|
||||
"--tokenizer",
|
||||
str(args.tokenizer.resolve()),
|
||||
cwd=repo,
|
||||
env=env,
|
||||
stdout=log,
|
||||
stderr=log,
|
||||
)
|
||||
try:
|
||||
code = await asyncio.wait_for(process.wait(), 600)
|
||||
except BaseException:
|
||||
if process.returncode is None:
|
||||
process.terminate()
|
||||
try:
|
||||
await asyncio.wait_for(process.wait(), 10)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise
|
||||
if code:
|
||||
raise RuntimeError("Native worker failed: " + key)
|
||||
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for key in cases:
|
||||
tasks.create_task(worker(key))
|
||||
|
||||
|
||||
async def run(args):
|
||||
"""Keep all native generations, including non-shrinking failures, in the denominator."""
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
output = args.output.resolve()
|
||||
if output.exists() or output == repo or repo in output.parents:
|
||||
raise ValueError("Use a new output directory outside repository")
|
||||
_, cases = load_cases(args.corpus.resolve())
|
||||
original = validate_base(args.evaluation, args.corpus, cases)
|
||||
await prepare_native(args, cases, repo)
|
||||
rows = deepcopy(original)
|
||||
output.mkdir(parents=True)
|
||||
for name in ("suite_native.py", "hermes_summary.py", "suite.py", "STUDY.md"):
|
||||
shutil.copy2(Path(__file__).parent / name, output / name)
|
||||
write(
|
||||
output / "protocol.json",
|
||||
{
|
||||
"base_evaluation": str(args.evaluation.resolve()),
|
||||
"native_generations": str(args.native.resolve()),
|
||||
"methods": 8,
|
||||
"tasks": 14,
|
||||
"generations": 3,
|
||||
"answers_per_context": 2,
|
||||
"total_planned_contexts": 336,
|
||||
"total_planned_answers": 672,
|
||||
"native_strategy": (
|
||||
"Hermes small-window adapter: legacy generator, S-matched 64..256 budget, "
|
||||
"hard word limit, native body-only carrier"
|
||||
),
|
||||
"acceptance": "native actual API input < corresponding Full and <= the predeclared case ceiling",
|
||||
"no_full_fallback": True,
|
||||
"judge": "same strict JSON field grader; no LLM judge",
|
||||
},
|
||||
)
|
||||
native_rows = []
|
||||
for key, case in cases.items():
|
||||
root = output / key
|
||||
root.mkdir()
|
||||
generated = json.loads((args.native / key / "generation/summaries.json").read_text())
|
||||
assert len(generated) == 3 and {r["repeat"] for r in generated} == set(range(3))
|
||||
for entry in generated:
|
||||
i = entry["repeat"]
|
||||
full = next(r for r in original if r["case"] == key and r["repeat"] == i and r["arm"] == "full")
|
||||
full_tokens = full["trials"][0]["usage"]["prompt_tokens"]
|
||||
row = {
|
||||
"case": key,
|
||||
"arm": "hermes_summary",
|
||||
"repeat": i,
|
||||
"summary": entry["summary"],
|
||||
"selected": [],
|
||||
"valid": False,
|
||||
"trials": [],
|
||||
"generation_valid": entry["valid"],
|
||||
"native_budget": entry["native_budget"],
|
||||
"summary_tokens": entry["summary_tokens"],
|
||||
"input_cap": full["input_cap"],
|
||||
"full_input_tokens": full_tokens,
|
||||
}
|
||||
write(
|
||||
root / f"c{i}.hermes_summary.context.json",
|
||||
{"messages": messages(case, compose(entry["summary"] or "", ""))},
|
||||
)
|
||||
native_rows.append(row)
|
||||
rows.extend(native_rows)
|
||||
write(output / "frozen-selections.json", rows)
|
||||
write(output / "results.json", rows)
|
||||
values = dotenv_values(repo / ".env")
|
||||
client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1)
|
||||
tokenizer = Tokenizer.from_file(str(args.tokenizer))
|
||||
limit = asyncio.Semaphore(4)
|
||||
work = asyncio.Semaphore(4)
|
||||
|
||||
async def answer(row):
|
||||
async with work:
|
||||
root = output / row["case"]
|
||||
label = f"c{row['repeat']}.hermes_summary"
|
||||
exp = Experiment(client, None, root, tokenizer, limit, "qwen3.8-max", row["input_cap"])
|
||||
try:
|
||||
if not row["generation_valid"]:
|
||||
raise ValueError("Native summary generation failed its body budget")
|
||||
context = json.loads((root / f"{label}.context.json").read_text())["messages"]
|
||||
trials = []
|
||||
for j in range(2):
|
||||
text, usage = await exp.ask(f"{label}.answer{j}", context)
|
||||
prediction = json.loads(text)
|
||||
trials.append(
|
||||
{"answer": prediction, "usage": usage, "checks": grade(cases[row["case"]], prediction)},
|
||||
)
|
||||
row["trials"] = trials
|
||||
shorter = all(t["usage"]["prompt_tokens"] < row["full_input_tokens"] for t in trials)
|
||||
fits = all(t["usage"]["prompt_tokens"] <= row["input_cap"] for t in trials)
|
||||
row.update(
|
||||
valid=shorter and fits,
|
||||
error=(
|
||||
None
|
||||
if shorter and fits
|
||||
else "not_shorter_than_full" if not shorter else "input_ceiling_exceeded"
|
||||
),
|
||||
)
|
||||
except Exception as exc:
|
||||
row.update(valid=False, error=type(exc).__name__, trials=[])
|
||||
write(output / "results.json", rows)
|
||||
report(output, rows, cases, 2, include_native=True)
|
||||
print(row["case"], label, "OK" if row["valid"] else row["error"], flush=True)
|
||||
|
||||
try:
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for row in native_rows:
|
||||
tasks.create_task(answer(row))
|
||||
write(
|
||||
output / "validation.json",
|
||||
{
|
||||
"planned_contexts": len(rows),
|
||||
"valid_contexts": sum(r["valid"] for r in rows),
|
||||
"actual_answers": sum(len(r["trials"]) for r in rows),
|
||||
"native_valid": sum(r["valid"] for r in native_rows),
|
||||
"native_failed": sum(not r["valid"] for r in native_rows),
|
||||
},
|
||||
)
|
||||
finally:
|
||||
await client.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""A prepared, frozen native baseline must exist before scored calls are allowed."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
for name in ("corpus", "evaluation", "native", "tokenizer", "output"):
|
||||
parser.add_argument("--" + name, type=Path, required=True)
|
||||
parser.add_argument("--hermes", type=Path)
|
||||
parser.add_argument("--hermes-python", type=Path)
|
||||
parser.add_argument("--live", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if not args.live:
|
||||
parser.error("Use --live to authorize provider calls")
|
||||
asyncio.run(run(args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
314
benchmark/compaction/suite_run.py
Normal file
314
benchmark/compaction/suite_run.py
Normal file
|
|
@ -0,0 +1,314 @@
|
|||
"""Evaluate frozen-summary Top-N curves on the predeclared fixture task panel."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import statistics
|
||||
|
||||
from dotenv import dotenv_values
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
from benchmark.compaction.cases import event_units, render, write
|
||||
from benchmark.compaction.run import Experiment, RESERVE, compose, messages, top_n
|
||||
from benchmark.compaction.suite import exam, grade
|
||||
from reme.components.as_decision_model import AgentScopeAsDecisionModel
|
||||
|
||||
ARMS = ["full", "summary"] + [f"fusion_top{n}" for n in range(1, 6)]
|
||||
LABELS = ["Full context", "S"] + [f"S + Jev Top {n}" for n in range(1, 6)]
|
||||
|
||||
|
||||
def load_cases(corpus_path):
|
||||
"""Project public chat fields, verify source hashes and exclude final collection reports."""
|
||||
manifest = json.loads(corpus_path.read_text())
|
||||
root = corpus_path.parent
|
||||
result = {}
|
||||
for definition in manifest["cases"]:
|
||||
task = root / definition["id"]
|
||||
source = task / "trajectory.full.json"
|
||||
meta = json.loads((task / "collection.json").read_text())
|
||||
if not meta["completed"] or hashlib.sha256(source.read_bytes()).hexdigest() != meta["trajectory_sha256"]:
|
||||
raise ValueError("Incomplete or changed collection: " + definition["id"])
|
||||
for name, sha in definition["source_hashes"].items():
|
||||
if hashlib.sha256((task / "workspace" / name).read_bytes()).hexdigest() != sha:
|
||||
raise ValueError("Changed source fixture")
|
||||
history = json.loads(source.read_text())["messages"]
|
||||
if history[-1]["role"] != "assistant" or history[-1].get("tool_calls"):
|
||||
raise ValueError("Expected the final collection report to exclude")
|
||||
allowed = {"role", "content", "tool_calls", "tool_call_id", "name"}
|
||||
history = [{k: v for k, v in m.items() if k in allowed} for m in history[:-1]]
|
||||
for m in history:
|
||||
if m.get("tool_calls"):
|
||||
m["tool_calls"] = [
|
||||
{
|
||||
"id": c["id"],
|
||||
"type": "function",
|
||||
"function": {k: c["function"][k] for k in ("name", "arguments")},
|
||||
}
|
||||
for c in m["tool_calls"]
|
||||
]
|
||||
if [m["role"] for m in history[:2]] != ["system", "user"]:
|
||||
raise ValueError("Missing real system/task")
|
||||
calls = [c["id"] for m in history for c in m.get("tool_calls", []) or []]
|
||||
outputs = [m["tool_call_id"] for m in history if m["role"] == "tool"]
|
||||
if len(calls) != len(set(calls)) or sorted(calls) != sorted(outputs):
|
||||
raise ValueError("Unpaired or duplicate calls")
|
||||
case = {
|
||||
**definition,
|
||||
"messages": history,
|
||||
"exam": exam(definition),
|
||||
"trajectory_sha256": meta["trajectory_sha256"],
|
||||
}
|
||||
result[definition["id"]] = case
|
||||
return manifest, result
|
||||
|
||||
|
||||
def metrics(rows, ids, repeats, include_native=False):
|
||||
"""Equal task weights; invalid planned contexts remain zero instead of disappearing."""
|
||||
table = []
|
||||
arms = ARMS[:2] + ["hermes_summary"] + ARMS[2:] if include_native else ARMS
|
||||
labels = LABELS[:2] + ["Hermes small-window adapter"] + LABELS[2:] if include_native else LABELS
|
||||
for arm, label in zip(arms, labels):
|
||||
task_facts = []
|
||||
task_exact = []
|
||||
tokens = []
|
||||
valid = 0
|
||||
for task_id in ids:
|
||||
group = [r for r in rows if r["case"] == task_id and r["arm"] == arm]
|
||||
task_facts.append(
|
||||
statistics.mean(
|
||||
(
|
||||
sum(sum(t["checks"].values()) / len(t["checks"]) for t in r["trials"]) / repeats
|
||||
if r["valid"]
|
||||
else 0
|
||||
)
|
||||
for r in group
|
||||
),
|
||||
)
|
||||
task_exact.append(
|
||||
statistics.mean(
|
||||
sum(all(t["checks"].values()) for t in r["trials"]) / repeats if r["valid"] else 0 for r in group
|
||||
),
|
||||
)
|
||||
valid += sum(r["valid"] for r in group)
|
||||
tokens.extend(t["usage"]["prompt_tokens"] for r in group for t in r["trials"])
|
||||
table.append(
|
||||
{
|
||||
"arm": arm,
|
||||
"label": label,
|
||||
"facts": statistics.mean(task_facts),
|
||||
"exact": statistics.mean(task_exact),
|
||||
"input_tokens": statistics.mean(tokens) if tokens else None,
|
||||
"valid_contexts": valid,
|
||||
"planned_contexts": sum(r["case"] in ids and r["arm"] == arm for r in rows),
|
||||
},
|
||||
)
|
||||
return table
|
||||
|
||||
|
||||
def report(output, rows, cases, repeats, include_native=False):
|
||||
"""Keep primary and short/query-only controls separate in compact seven-row tables."""
|
||||
result = {}
|
||||
lines = ["# Fourteen-task fixed-summary study", ""]
|
||||
for cohort in ("primary", "controls"):
|
||||
ids = [k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")]
|
||||
result[cohort] = metrics(rows, ids, repeats, include_native)
|
||||
lines += [
|
||||
f"## {cohort.title()} ({len(ids)} tasks)",
|
||||
"",
|
||||
"| Method | Mean input tokens | Fact score | All facts correct | Valid contexts |",
|
||||
"| --- | ---: | ---: | ---: | --- |",
|
||||
]
|
||||
for r in result[cohort]:
|
||||
tokens = f"{r['input_tokens']:.0f}" if r["input_tokens"] is not None else "—"
|
||||
lines.append(
|
||||
f"| {r['label']} | {tokens} | {r['facts']:.1%} | {r['exact']:.1%} | "
|
||||
f"{r['valid_contexts']}/{r['planned_contexts']} |",
|
||||
)
|
||||
lines += [""]
|
||||
result["per_task"] = {k: metrics(rows, [k], repeats, include_native) for k in cases}
|
||||
write(output / "aggregate.json", result)
|
||||
(output / "REPORT.md").write_text("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
async def run(args):
|
||||
"""Freeze all inputs across tasks before any scored answers; no outcome-based tuning."""
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
output = args.output.resolve()
|
||||
if output.exists() or output == repo or repo in output.parents:
|
||||
raise ValueError("Use a new directory outside repository")
|
||||
manifest, cases = load_cases(args.corpus.resolve())
|
||||
original = args.corpus.read_bytes()
|
||||
values = dotenv_values(repo / ".env")
|
||||
for key in ("LLM_API_KEY", "LLM_BASE_URL", "DECISION_MODEL_API_KEY"):
|
||||
if not values.get(key):
|
||||
raise ValueError("Missing " + key)
|
||||
output.mkdir(parents=True)
|
||||
write(output / "corpus.json", manifest)
|
||||
for name in ("suite.py", "suite_run.py", "run.py", "cases.py", "STUDY.md"):
|
||||
shutil.copy2(Path(__file__).parent / name, output / name)
|
||||
write(
|
||||
output / "protocol.json",
|
||||
{
|
||||
"corpus_sha256": hashlib.sha256(original).hexdigest(),
|
||||
"model": args.model,
|
||||
"compressions": args.compressions,
|
||||
"answer_repeats": args.repeats,
|
||||
"top_n": [1, 2, 3, 4, 5],
|
||||
"concurrency": args.concurrency,
|
||||
"summary_allowance": "20% of rendered evidence tokens, clamped to 64..256",
|
||||
"input_ceiling": (
|
||||
"protected context + five largest whole units + summary allowance + 96 reserve; "
|
||||
"adjusted before answers if serialized S needs more space"
|
||||
),
|
||||
"cohorts": {k: c["control_reason"] for k, c in cases.items()},
|
||||
"judge": "recursive type-strict JSON field rules",
|
||||
"tokenizer_sha256": hashlib.sha256(args.tokenizer.read_bytes()).hexdigest(),
|
||||
"evaluation": "direct Qwen; no Hermes continuation",
|
||||
},
|
||||
)
|
||||
client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1)
|
||||
limit = asyncio.Semaphore(args.concurrency)
|
||||
tokenizer = Tokenizer.from_file(str(args.tokenizer))
|
||||
experiments = {}
|
||||
rows = []
|
||||
credential = {
|
||||
"api_key": values["DECISION_MODEL_API_KEY"],
|
||||
"base_url": values.get("DECISION_MODEL_BASE_URL") or None,
|
||||
}
|
||||
model = values.get("DECISION_MODEL_NAME")
|
||||
model = None if model in (None, "", "null") else model
|
||||
try:
|
||||
async with AgentScopeAsDecisionModel(credential=credential, model=model, timeout=45) as decision:
|
||||
work = asyncio.Semaphore(args.concurrency)
|
||||
|
||||
async def prepare_case(task_id, case):
|
||||
async with work:
|
||||
root = output / task_id
|
||||
root.mkdir()
|
||||
exp = Experiment(client, decision, root, tokenizer, limit, args.model, 100000)
|
||||
units = event_units(case["messages"])
|
||||
allowance = max(64, min(256, int(exp.count(render(units)) * 0.2)))
|
||||
write(root / "source.json", {**case, "units": units})
|
||||
base = exp.input_tokens(messages(case, compose("", "")))
|
||||
largest = sorted(
|
||||
units,
|
||||
key=lambda u: exp.input_tokens(messages(case, compose("", render([u])))) - base,
|
||||
reverse=True,
|
||||
)[:5]
|
||||
cap = exp.input_tokens(messages(case, compose("", render(largest)))) + allowance + RESERVE + 32
|
||||
prepared = []
|
||||
for i in range(args.compressions):
|
||||
scores, _ = await exp.scores(f"paired.c{i}", case["messages"][1]["content"], units)
|
||||
summary, _ = await exp.summarize(
|
||||
f"paired.c{i}",
|
||||
case["messages"][1]["content"],
|
||||
units,
|
||||
allowance,
|
||||
)
|
||||
for arm in ARMS:
|
||||
n = int(arm[-1]) if arm.startswith("fusion") else 0
|
||||
chosen = top_n(units, scores, min(n, len(units)))
|
||||
s = "" if arm == "full" else summary
|
||||
payload = None if arm == "full" else compose(s, render(chosen))
|
||||
context = messages(case, payload)
|
||||
label = f"c{i}.{arm}"
|
||||
write(root / f"{label}.context.json", {"messages": context})
|
||||
prepared.append(
|
||||
{
|
||||
"case": task_id,
|
||||
"arm": arm,
|
||||
"repeat": i,
|
||||
"summary": s,
|
||||
"selected": [u["id"] for u in chosen],
|
||||
"actual_n": len(chosen),
|
||||
"estimated_input_tokens": exp.input_tokens(context),
|
||||
"valid": False,
|
||||
"trials": [],
|
||||
},
|
||||
)
|
||||
cap = max(cap, max(r["estimated_input_tokens"] + RESERVE for r in prepared if r["arm"] != "full"))
|
||||
exp.cap = cap
|
||||
experiments[task_id] = exp
|
||||
for r in prepared:
|
||||
r["input_cap"] = cap
|
||||
rows.extend(prepared)
|
||||
write(root / "budget.json", {"allowance": allowance, "input_cap": cap, "units": len(units)})
|
||||
print("PREPARED", task_id, len(units), "units", "summary", allowance, "cap", cap, flush=True)
|
||||
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for key, case in cases.items():
|
||||
tasks.create_task(prepare_case(key, case))
|
||||
rows.sort(key=lambda r: (list(cases).index(r["case"]), r["repeat"], ARMS.index(r["arm"])))
|
||||
write(output / "frozen-selections.json", rows)
|
||||
write(output / "results.json", rows)
|
||||
answer_limit = asyncio.Semaphore(args.concurrency)
|
||||
|
||||
async def answer(row):
|
||||
async with answer_limit:
|
||||
exp = experiments[row["case"]]
|
||||
label = f"c{row['repeat']}.{row['arm']}"
|
||||
try:
|
||||
context = json.loads((exp.output / f"{label}.context.json").read_text())["messages"]
|
||||
trials = []
|
||||
for j in range(args.repeats):
|
||||
text, usage = await exp.ask(f"{label}.answer{j}", context)
|
||||
if row["arm"] != "full" and usage["prompt_tokens"] > exp.cap:
|
||||
raise ValueError("Actual input ceiling exceeded")
|
||||
prediction = json.loads(text)
|
||||
checks = grade(cases[row["case"]], prediction)
|
||||
trials.append({"answer": prediction, "usage": usage, "checks": checks})
|
||||
row.update(valid=True, trials=trials)
|
||||
except Exception as exc:
|
||||
row.update(valid=False, error=type(exc).__name__, trials=[])
|
||||
write(output / "results.json", rows)
|
||||
report(output, rows, cases, args.repeats)
|
||||
print("ANSWERED", row["case"], label, "OK" if row["valid"] else row["error"], flush=True)
|
||||
|
||||
async with asyncio.TaskGroup() as tasks:
|
||||
for row in rows:
|
||||
tasks.create_task(answer(row))
|
||||
assert args.corpus.read_bytes() == original
|
||||
load_cases(args.corpus.resolve())
|
||||
for file in output.rglob("*"):
|
||||
if file.is_file():
|
||||
data = file.read_bytes()
|
||||
assert all(values[k].encode() not in data for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY"))
|
||||
write(
|
||||
output / "validation.json",
|
||||
{
|
||||
"source_unchanged": True,
|
||||
"no_api_keys": True,
|
||||
"tasks": len(cases),
|
||||
"contexts": len(rows),
|
||||
"valid": sum(r["valid"] for r in rows),
|
||||
"answers": sum(len(r["trials"]) for r in rows),
|
||||
},
|
||||
)
|
||||
finally:
|
||||
await client.close()
|
||||
|
||||
|
||||
def main():
|
||||
"""Explicit live opt-in and bounded request concurrency."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--corpus", type=Path, required=True)
|
||||
parser.add_argument("--tokenizer", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--model", default="qwen3.8-max")
|
||||
parser.add_argument("--compressions", type=int, default=3)
|
||||
parser.add_argument("--repeats", type=int, default=2)
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
parser.add_argument("--live", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if not args.live or min(args.compressions, args.repeats, args.concurrency) < 1:
|
||||
parser.error("Use --live and positive settings")
|
||||
asyncio.run(run(args))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -119,20 +119,25 @@ latency and total cost alongside token reduction. ReMe does not automatically su
|
|||
|
||||
## Fixed summary + selected original evidence
|
||||
|
||||
The payment experiment freezes one summary **S**, then appends Jev-ranked original blocks without rewriting it.
|
||||
Five summary/ranking pairs and three answers per method compare Full context, S and S + Top 1–5 (105 answers).
|
||||
The experiment uses **14 fixture queries with predefined answers**, collected through Hermes: four source-length-qualified
|
||||
primary tasks and ten short/query-only controls. Rule scoring checks exact JSON fields; no LLM judge creates or grades gold.
|
||||
|
||||
|
||||
|
||||
|
||||
S + Top 4 reaches **100% on ten exact facts**, matching Full context with **25.2% fewer downstream input tokens**
|
||||
(3,529 versus 4,715). S alone reaches 73.3%; Top 2 reaches 86.0% and Top 3 reaches 94.0%. Top 5 adds tokens without
|
||||
improving the score. All methods pass the six executable repair tests.
|
||||
Across the four primary tasks, Full context reaches 100%, S reaches 82.0%, and S + Top 3 reaches **99.5% with 19.3% fewer
|
||||
input tokens** (3,241 versus 4,014). Top 4 and Top 5 score 98.1% and 97.4%, so more retained text is not automatically better.
|
||||
Each generation freezes S and the Jev ranking before answers; all five fusion methods keep byte-identical S.
|
||||
|
||||
<!-- The original payment trajectory has 10 evidence units: Top 10 is the maximum; only Top 1–5 were evaluated. -->
|
||||
There are **10 evidence units, so Top 10 is the maximum**; this sweep tests Top 1–5. S uses a 3,200-token generation
|
||||
allowance; the extended answering sweep has a 3,800-token total-input ceiling. S remains byte-identical in every fusion;
|
||||
adding originals intentionally increases input length. Actual compressed inputs stay within the ceiling.
|
||||
The **Hermes small-window adapter** scores 79.4% at 2,432 input tokens. It calls the real native summary generator with
|
||||
explicit small-window budget/word constraints and a body-only carrier; this is not the unchanged automatic compressor.
|
||||
All 42 native contexts fit their budgets and use strictly fewer actual input tokens than Full, with no original fallback.
|
||||
|
||||
This is one synthetic task collected by Hermes, evaluated by calling Qwen directly. Repeated measurements do not
|
||||
establish general task performance or superiority over pure Jev. The hybrid is benchmark-only; the production contract
|
||||
above is unchanged. See the [protocol, scoring and complete results](https://github.com/jinliyl/ReMe/blob/26a83c1904d5c8312ed8e8cb5ffa21500b6e50e6/benchmark/compaction/RESULTS.md).
|
||||
<!-- Payment has 10 units (maximum Top 10). Other tasks have different counts; actual N is min(N, available units). -->
|
||||
Top N counts complete source units and saturates at each task's available count. The original payment has ten units,
|
||||
so its maximum is Top 10; the sweep tests Top 1–5. Summary/body allowances and complete-input ceilings are task-specific;
|
||||
this is an incremental-evidence curve, not an equal-length comparison or proof that every Full exceeds its ceiling.
|
||||
|
||||
The panel has 336 contexts and 672 answers, with the controls reported separately. Hermes collects source trajectories;
|
||||
answering calls Qwen directly without tool continuation or recovery. These few synthetic tasks do not establish general
|
||||
superiority or statistical equivalence. The hybrid and native adapter remain benchmark-only.
|
||||
See the [exact queries, answers and settings](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md) and [results, controls and audit](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md).
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
<?xml version="1.0" encoding="utf-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="921.6pt" height="352.8pt" viewBox="0 0 921.6 352.8" xmlns="http://www.w3.org/2000/svg" version="1.1">
|
||||
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1008pt" height="403.2pt" viewBox="0 0 1008 403.2" xmlns="http://www.w3.org/2000/svg" version="1.1">
|
||||
<metadata>
|
||||
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
|
||||
<cc:Work>
|
||||
|
|
@ -20,76 +20,76 @@
|
|||
</defs>
|
||||
<g id="figure_1">
|
||||
<g id="patch_1">
|
||||
<path d="M 0 352.8
|
||||
L 921.6 352.8
|
||||
L 921.6 0
|
||||
<path d="M 0 403.2
|
||||
L 1008 403.2
|
||||
L 1008 0
|
||||
L 0 0
|
||||
z
|
||||
" style="fill: #ffffff"/>
|
||||
</g>
|
||||
<g id="axes_1">
|
||||
<g id="patch_2">
|
||||
<path d="M 147.456 303.408
|
||||
L 487.866811 303.408
|
||||
L 487.866811 59.976
|
||||
L 147.456 59.976
|
||||
<path d="M 206.64 338.688
|
||||
L 556.953901 338.688
|
||||
L 556.953901 64.512
|
||||
L 206.64 64.512
|
||||
z
|
||||
" style="fill: #ffffff"/>
|
||||
</g>
|
||||
<g id="matplotlib.axis_1">
|
||||
<g id="xtick_1">
|
||||
<g id="line2d_1">
|
||||
<path d="M 147.456 303.408
|
||||
L 147.456 59.976
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<path d="M 206.64 338.688
|
||||
L 206.64 64.512
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_2"/>
|
||||
<g id="text_1">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="147.456" y="315.265422" transform="rotate(-0 147.456 315.265422)">0</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="206.64" y="350.545422" transform="rotate(-0 206.64 350.545422)">0</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_2">
|
||||
<g id="line2d_3">
|
||||
<path d="M 221.45835 303.408
|
||||
L 221.45835 59.976
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<path d="M 282.795196 338.688
|
||||
L 282.795196 64.512
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_4"/>
|
||||
<g id="text_2">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="221.45835" y="315.265422" transform="rotate(-0 221.45835 315.265422)">25</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="282.795196" y="350.545422" transform="rotate(-0 282.795196 350.545422)">25</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_3">
|
||||
<g id="line2d_5">
|
||||
<path d="M 295.4607 303.408
|
||||
L 295.4607 59.976
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<path d="M 358.950392 338.688
|
||||
L 358.950392 64.512
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_6"/>
|
||||
<g id="text_3">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="295.4607" y="315.265422" transform="rotate(-0 295.4607 315.265422)">50</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="358.950392" y="350.545422" transform="rotate(-0 358.950392 350.545422)">50</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_4">
|
||||
<g id="line2d_7">
|
||||
<path d="M 369.463051 303.408
|
||||
L 369.463051 59.976
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<path d="M 435.105588 338.688
|
||||
L 435.105588 64.512
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_8"/>
|
||||
<g id="text_4">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="369.463051" y="315.265422" transform="rotate(-0 369.463051 315.265422)">75</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="435.105588" y="350.545422" transform="rotate(-0 435.105588 350.545422)">75</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_5">
|
||||
<g id="line2d_9">
|
||||
<path d="M 443.465401 303.408
|
||||
L 443.465401 59.976
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<path d="M 511.260784 338.688
|
||||
L 511.260784 64.512
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_10"/>
|
||||
<g id="text_5">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="443.465401" y="315.265422" transform="rotate(-0 443.465401 315.265422)">100</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="511.260784" y="350.545422" transform="rotate(-0 511.260784 350.545422)">100</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
|
|
@ -97,186 +97,200 @@ L 443.465401 59.976
|
|||
<g id="ytick_1">
|
||||
<g id="line2d_11"/>
|
||||
<g id="text_6">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="85.885379" transform="rotate(-0 143.956 85.885379)">Full context</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="91.593515" transform="rotate(-0 203.14 91.593515)">Full context</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_2">
|
||||
<g id="line2d_12"/>
|
||||
<g id="text_7">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="119.213537" transform="rotate(-0 143.956 119.213537)">Summary S</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="124.21755" transform="rotate(-0 203.14 124.21755)">Summary S</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_3">
|
||||
<g id="line2d_13"/>
|
||||
<g id="text_8">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="152.542124" transform="rotate(-0 143.956 152.542124)">S + Jev Top 1</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="156.842444" transform="rotate(-0 203.14 156.842444)">Hermes small-window</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_4">
|
||||
<g id="line2d_14"/>
|
||||
<g id="text_9">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="185.870711" transform="rotate(-0 143.956 185.870711)">S + Jev Top 2</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="189.466479" transform="rotate(-0 203.14 189.466479)">S + Jev Top 1</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_5">
|
||||
<g id="line2d_15"/>
|
||||
<g id="text_10">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="219.199298" transform="rotate(-0 143.956 219.199298)">S + Jev Top 3</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="222.090943" transform="rotate(-0 203.14 222.090943)">S + Jev Top 2</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_6">
|
||||
<g id="line2d_16"/>
|
||||
<g id="text_11">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="252.527885" transform="rotate(-0 143.956 252.527885)">S + Jev Top 4</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="254.715408" transform="rotate(-0 203.14 254.715408)">S + Jev Top 3</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_7">
|
||||
<g id="line2d_17"/>
|
||||
<g id="text_12">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="143.956" y="285.856472" transform="rotate(-0 143.956 285.856472)">S + Jev Top 5</text>
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="287.339872" transform="rotate(-0 203.14 287.339872)">S + Jev Top 4</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="ytick_8">
|
||||
<g id="line2d_18"/>
|
||||
<g id="text_13">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: end; fill: #45586b" x="203.14" y="319.964337" transform="rotate(-0 203.14 319.964337)">S + Jev Top 5</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
<g id="patch_3">
|
||||
<path d="M 147.456 71.041091
|
||||
L 443.465401 71.041091
|
||||
L 443.465401 92.371387
|
||||
L 147.456 92.371387
|
||||
<path d="M 206.64 76.974545
|
||||
L 511.260784 76.974545
|
||||
L 511.260784 97.854203
|
||||
L 206.64 97.854203
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #718096"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #718096"/>
|
||||
</g>
|
||||
<g id="patch_4">
|
||||
<path d="M 147.456 104.369678
|
||||
L 364.430891 104.369678
|
||||
L 364.430891 125.699974
|
||||
L 147.456 125.699974
|
||||
<path d="M 206.64 109.59901
|
||||
L 456.471351 109.59901
|
||||
L 456.471351 130.478667
|
||||
L 206.64 130.478667
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #e5a442"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #e5a442"/>
|
||||
</g>
|
||||
<g id="patch_5">
|
||||
<path d="M 147.456 137.698265
|
||||
L 362.654834 137.698265
|
||||
L 362.654834 159.028561
|
||||
L 147.456 159.028561
|
||||
<path d="M 206.64 142.223475
|
||||
L 448.432747 142.223475
|
||||
L 448.432747 163.103132
|
||||
L 206.64 163.103132
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #9674ba"/>
|
||||
</g>
|
||||
<g id="patch_6">
|
||||
<path d="M 147.456 171.026852
|
||||
L 402.024085 171.026852
|
||||
L 402.024085 192.357148
|
||||
L 147.456 192.357148
|
||||
<path d="M 206.64 174.847939
|
||||
L 459.327171 174.847939
|
||||
L 459.327171 195.727596
|
||||
L 206.64 195.727596
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="patch_7">
|
||||
<path d="M 147.456 204.355439
|
||||
L 425.704837 204.355439
|
||||
L 425.704837 225.685735
|
||||
L 147.456 225.685735
|
||||
<path d="M 206.64 207.472404
|
||||
L 494.443178 207.472404
|
||||
L 494.443178 228.352061
|
||||
L 206.64 228.352061
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="patch_8">
|
||||
<path d="M 147.456 237.684026
|
||||
L 443.465401 237.684026
|
||||
L 443.465401 259.014322
|
||||
L 147.456 259.014322
|
||||
<path d="M 206.64 240.096868
|
||||
L 509.674217 240.096868
|
||||
L 509.674217 260.976525
|
||||
L 206.64 260.976525
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #20917d"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #20917d"/>
|
||||
</g>
|
||||
<g id="patch_9">
|
||||
<path d="M 147.456 271.012613
|
||||
L 443.465401 271.012613
|
||||
L 443.465401 292.342909
|
||||
L 147.456 292.342909
|
||||
<path d="M 206.64 272.721333
|
||||
L 505.443373 272.721333
|
||||
L 505.443373 293.60099
|
||||
L 206.64 293.60099
|
||||
z
|
||||
" clip-path="url(#pd2ce3e3618)" style="fill: #20917d"/>
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="text_13">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="448.571563" y="84.303895" transform="rotate(-0 448.571563 84.303895)">100%</text>
|
||||
<g id="patch_10">
|
||||
<path d="M 206.64 305.345797
|
||||
L 503.327951 305.345797
|
||||
L 503.327951 326.225455
|
||||
L 206.64 326.225455
|
||||
z
|
||||
" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="text_14">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="369.537053" y="117.632482" transform="rotate(-0 369.537053 117.632482)">73.3%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="516.515492" y="90.01203" transform="rotate(-0 516.515492 90.01203)">100.0%</text>
|
||||
</g>
|
||||
<g id="text_15">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="367.760996" y="150.961069" transform="rotate(-0 367.760996 150.961069)">72.7%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="461.72606" y="122.636495" transform="rotate(-0 461.72606 122.636495)">82.0%</text>
|
||||
</g>
|
||||
<g id="text_16">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="407.130247" y="184.289656" transform="rotate(-0 407.130247 184.289656)">86%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="453.687456" y="155.260959" transform="rotate(-0 453.687456 155.260959)">79.4%</text>
|
||||
</g>
|
||||
<g id="text_17">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="430.810999" y="217.618243" transform="rotate(-0 430.810999 217.618243)">94%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="464.58188" y="187.885424" transform="rotate(-0 464.58188 187.885424)">83.0%</text>
|
||||
</g>
|
||||
<g id="text_18">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="448.571563" y="250.94683" transform="rotate(-0 448.571563 250.94683)">100%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="499.697887" y="220.509889" transform="rotate(-0 499.697887 220.509889)">94.5%</text>
|
||||
</g>
|
||||
<g id="text_19">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="448.571563" y="284.275417" transform="rotate(-0 448.571563 284.275417)">100%</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="514.928926" y="253.134353" transform="rotate(-0 514.928926 253.134353)">99.5%</text>
|
||||
</g>
|
||||
<g id="text_20">
|
||||
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="147.456" y="47.976" transform="rotate(-0 147.456 47.976)">Fact accuracy (%)</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="510.698081" y="285.758818" transform="rotate(-0 510.698081 285.758818)">98.1%</text>
|
||||
</g>
|
||||
<g id="text_21">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="508.582659" y="318.383282" transform="rotate(-0 508.582659 318.383282)">97.4%</text>
|
||||
</g>
|
||||
<g id="text_22">
|
||||
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="206.64" y="52.512" transform="rotate(-0 206.64 52.512)">Exact-field score (%)</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="axes_2">
|
||||
<g id="patch_10">
|
||||
<path d="M 562.757189 303.408
|
||||
L 903.168 303.408
|
||||
L 903.168 59.976
|
||||
L 562.757189 59.976
|
||||
<g id="patch_11">
|
||||
<path d="M 637.526099 338.688
|
||||
L 987.84 338.688
|
||||
L 987.84 64.512
|
||||
L 637.526099 64.512
|
||||
z
|
||||
" style="fill: #ffffff"/>
|
||||
</g>
|
||||
<g id="matplotlib.axis_3">
|
||||
<g id="xtick_6">
|
||||
<g id="line2d_18">
|
||||
<path d="M 562.757189 303.408
|
||||
L 562.757189 59.976
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<g id="line2d_19">
|
||||
<path d="M 637.526099 338.688
|
||||
L 637.526099 64.512
|
||||
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_19"/>
|
||||
<g id="text_21">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="562.757189" y="315.265422" transform="rotate(-0 562.757189 315.265422)">0</text>
|
||||
<g id="line2d_20"/>
|
||||
<g id="text_23">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="637.526099" y="350.545422" transform="rotate(-0 637.526099 350.545422)">0</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_7">
|
||||
<g id="line2d_20">
|
||||
<path d="M 656.448238 303.408
|
||||
L 656.448238 59.976
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<g id="line2d_21">
|
||||
<path d="M 749.328408 338.688
|
||||
L 749.328408 64.512
|
||||
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_21"/>
|
||||
<g id="text_22">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="656.448238" y="315.265422" transform="rotate(-0 656.448238 315.265422)">1500</text>
|
||||
<g id="line2d_22"/>
|
||||
<g id="text_24">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="749.328408" y="350.545422" transform="rotate(-0 749.328408 350.545422)">1500</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_8">
|
||||
<g id="line2d_22">
|
||||
<path d="M 750.139287 303.408
|
||||
L 750.139287 59.976
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<g id="line2d_23">
|
||||
<path d="M 861.130717 338.688
|
||||
L 861.130717 64.512
|
||||
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_23"/>
|
||||
<g id="text_23">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="750.139287" y="315.265422" transform="rotate(-0 750.139287 315.265422)">3000</text>
|
||||
<g id="line2d_24"/>
|
||||
<g id="text_25">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="861.130717" y="350.545422" transform="rotate(-0 861.130717 350.545422)">3000</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="xtick_9">
|
||||
<g id="line2d_24">
|
||||
<path d="M 843.830336 303.408
|
||||
L 843.830336 59.976
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
<g id="line2d_25">
|
||||
<path d="M 972.933025 338.688
|
||||
L 972.933025 64.512
|
||||
" clip-path="url(#p012de37c52)" style="fill: none; stroke: #e3e9ef; stroke-width: 0.8; stroke-linecap: square"/>
|
||||
</g>
|
||||
<g id="line2d_25"/>
|
||||
<g id="text_24">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="843.830336" y="315.265422" transform="rotate(-0 843.830336 315.265422)">4500</text>
|
||||
<g id="line2d_26"/>
|
||||
<g id="text_26">
|
||||
<text style="font-size: 11px; font-family: 'DejaVu Sans'; text-anchor: middle; fill: #45586b" x="972.933025" y="350.545422" transform="rotate(-0 972.933025 350.545422)">4500</text>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
<g id="matplotlib.axis_4">
|
||||
<g id="ytick_8">
|
||||
<g id="line2d_26"/>
|
||||
</g>
|
||||
<g id="ytick_9">
|
||||
<g id="line2d_27"/>
|
||||
</g>
|
||||
|
|
@ -295,101 +309,121 @@ L 843.830336 59.976
|
|||
<g id="ytick_14">
|
||||
<g id="line2d_32"/>
|
||||
</g>
|
||||
</g>
|
||||
<g id="patch_11">
|
||||
<path d="M 562.757189 71.041091
|
||||
L 857.259386 71.041091
|
||||
L 857.259386 92.371387
|
||||
L 562.757189 92.371387
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #718096"/>
|
||||
<g id="ytick_15">
|
||||
<g id="line2d_33"/>
|
||||
</g>
|
||||
<g id="ytick_16">
|
||||
<g id="line2d_34"/>
|
||||
</g>
|
||||
</g>
|
||||
<g id="patch_12">
|
||||
<path d="M 562.757189 104.369678
|
||||
L 746.204263 104.369678
|
||||
L 746.204263 125.699974
|
||||
L 562.757189 125.699974
|
||||
<path d="M 637.526099 76.974545
|
||||
L 936.690444 76.974545
|
||||
L 936.690444 97.854203
|
||||
L 637.526099 97.854203
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #e5a442"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #718096"/>
|
||||
</g>
|
||||
<g id="patch_13">
|
||||
<path d="M 562.757189 137.698265
|
||||
L 758.509021 137.698265
|
||||
L 758.509021 159.028561
|
||||
L 562.757189 159.028561
|
||||
<path d="M 637.526099 109.59901
|
||||
L 822.366372 109.59901
|
||||
L 822.366372 130.478667
|
||||
L 637.526099 130.478667
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #e5a442"/>
|
||||
</g>
|
||||
<g id="patch_14">
|
||||
<path d="M 562.757189 171.026852
|
||||
L 765.56708 171.026852
|
||||
L 765.56708 192.357148
|
||||
L 562.757189 192.357148
|
||||
<path d="M 637.526099 142.223475
|
||||
L 818.825965 142.223475
|
||||
L 818.825965 163.103132
|
||||
L 637.526099 163.103132
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #9674ba"/>
|
||||
</g>
|
||||
<g id="patch_15">
|
||||
<path d="M 562.757189 204.355439
|
||||
L 772.625139 204.355439
|
||||
L 772.625139 225.685735
|
||||
L 562.757189 225.685735
|
||||
<path d="M 637.526099 174.847939
|
||||
L 842.695758 174.847939
|
||||
L 842.695758 195.727596
|
||||
L 637.526099 195.727596
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #5b93c8"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="patch_16">
|
||||
<path d="M 562.757189 237.684026
|
||||
L 783.180997 237.684026
|
||||
L 783.180997 259.014322
|
||||
L 562.757189 259.014322
|
||||
<path d="M 637.526099 207.472404
|
||||
L 859.205232 207.472404
|
||||
L 859.205232 228.352061
|
||||
L 637.526099 228.352061
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #20917d"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="patch_17">
|
||||
<path d="M 562.757189 271.012613
|
||||
L 786.678796 271.012613
|
||||
L 786.678796 292.342909
|
||||
L 562.757189 292.342909
|
||||
<path d="M 637.526099 240.096868
|
||||
L 879.074987 240.096868
|
||||
L 879.074987 260.976525
|
||||
L 637.526099 260.976525
|
||||
z
|
||||
" clip-path="url(#p0ab177e2e8)" style="fill: #20917d"/>
|
||||
" clip-path="url(#p012de37c52)" style="fill: #20917d"/>
|
||||
</g>
|
||||
<g id="text_25">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="862.365548" y="84.303895" transform="rotate(-0 862.365548 84.303895)">4,715</text>
|
||||
<g id="patch_18">
|
||||
<path d="M 637.526099 272.721333
|
||||
L 898.100013 272.721333
|
||||
L 898.100013 293.60099
|
||||
L 637.526099 293.60099
|
||||
z
|
||||
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="text_26">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="751.310425" y="117.632482" transform="rotate(-0 751.310425 117.632482)">2,937</text>
|
||||
<g id="patch_19">
|
||||
<path d="M 637.526099 305.345797
|
||||
L 910.342366 305.345797
|
||||
L 910.342366 326.225455
|
||||
L 637.526099 326.225455
|
||||
z
|
||||
" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/>
|
||||
</g>
|
||||
<g id="text_27">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="763.615183" y="150.961069" transform="rotate(-0 763.615183 150.961069)">3,134</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="941.945152" y="90.01203" transform="rotate(-0 941.945152 90.01203)">4,014</text>
|
||||
</g>
|
||||
<g id="text_28">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="770.673242" y="184.289656" transform="rotate(-0 770.673242 184.289656)">3,247</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="827.62108" y="122.636495" transform="rotate(-0 827.62108 122.636495)">2,480</text>
|
||||
</g>
|
||||
<g id="text_29">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="777.731301" y="217.618243" transform="rotate(-0 777.731301 217.618243)">3,360</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="824.080674" y="155.260959" transform="rotate(-0 824.080674 155.260959)">2,432</text>
|
||||
</g>
|
||||
<g id="text_30">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="788.287159" y="250.94683" transform="rotate(-0 788.287159 250.94683)">3,529</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="847.950467" y="187.885424" transform="rotate(-0 847.950467 187.885424)">2,753</text>
|
||||
</g>
|
||||
<g id="text_31">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="791.784958" y="284.275417" transform="rotate(-0 791.784958 284.275417)">3,585</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="864.459941" y="220.509889" transform="rotate(-0 864.459941 220.509889)">2,974</text>
|
||||
</g>
|
||||
<g id="text_32">
|
||||
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="562.757189" y="47.976" transform="rotate(-0 562.757189 47.976)">Downstream input tokens</text>
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="884.329696" y="253.134353" transform="rotate(-0 884.329696 253.134353)">3,241</text>
|
||||
</g>
|
||||
<g id="text_33">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="903.354722" y="285.758818" transform="rotate(-0 903.354722 285.758818)">3,496</text>
|
||||
</g>
|
||||
<g id="text_34">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #24364b" x="915.597075" y="318.383282" transform="rotate(-0 915.597075 318.383282)">3,660</text>
|
||||
</g>
|
||||
<g id="text_35">
|
||||
<text style="font-weight: 700; font-size: 12px; font-family: 'DejaVu Sans'; text-anchor: start" x="637.526099" y="52.512" transform="rotate(-0 637.526099 52.512)">Mean downstream input tokens</text>
|
||||
</g>
|
||||
</g>
|
||||
<g id="text_33">
|
||||
<text style="font-weight: 700; font-size: 17px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #1a2f45" x="36.864" y="19.973344" transform="rotate(-0 36.864 19.973344)">Fixed summary + selected original evidence</text>
|
||||
<g id="text_36">
|
||||
<text style="font-weight: 700; font-size: 18px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #1a2f45" x="40.32" y="21.741188" transform="rotate(-0 40.32 21.741188)">Fixed summary + selected original evidence</text>
|
||||
</g>
|
||||
<g id="text_34">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #45586b" x="36.864" y="343.98" transform="rotate(-0 36.864 343.98)">One synthetic payment task · 5 frozen summaries · 15 answers per method · Direct Qwen evaluation</text>
|
||||
<g id="text_37">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #45586b" x="40.32" y="379.008" transform="rotate(-0 40.32 379.008)">Primary cohort: 4 task templates · 24 answers per method · 14-query panel includes 10 separate controls</text>
|
||||
</g>
|
||||
<g id="text_38">
|
||||
<text style="font-size: 10px; font-family: 'DejaVu Sans'; text-anchor: start; fill: #45586b" x="40.32" y="393.12" transform="rotate(-0 40.32 393.12)">Hermes uses an explicit small-window adapter. Evaluation calls Qwen directly; no LLM judge.</text>
|
||||
</g>
|
||||
</g>
|
||||
<defs>
|
||||
<clipPath id="pd2ce3e3618">
|
||||
<rect x="147.456" y="59.976" width="340.410811" height="243.432"/>
|
||||
<clipPath id="pa8067ffbdb">
|
||||
<rect x="206.64" y="64.512" width="350.313901" height="274.176"/>
|
||||
</clipPath>
|
||||
<clipPath id="p0ab177e2e8">
|
||||
<rect x="562.757189" y="59.976" width="340.410811" height="243.432"/>
|
||||
<clipPath id="p012de37c52">
|
||||
<rect x="637.526099" y="64.512" width="350.313901" height="274.176"/>
|
||||
</clipPath>
|
||||
</defs>
|
||||
</svg>
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 15 KiB After Width: | Height: | Size: 16 KiB |
|
|
@ -110,19 +110,23 @@ ReMe 不会自动生成摘要或写入长期记忆。
|
|||
|
||||
## 固定摘要与增量原文实验
|
||||
|
||||
支付实验先固定一份摘要 **S**，再追加 Jev 排名前 N 的完整原文，不重写或缩短摘要。
|
||||
5 份摘要及对应排名，每种方式回答 3 次，对比 Full context、S、S + Top 1～5，共 105 次回答。
|
||||
实验固定 **14 个有预设答案的 fixture query**，通过 Hermes 采集轨迹：4 个按来源长度筛出的主实验任务，
|
||||
以及 10 个短任务或 query-only 对照。使用确定性 JSON 字段规则评分，不用 LLM 生成标准答案或担任 judge。
|
||||
|
||||
|
||||
|
||||
|
||||
S + Top 4 的十项事实正确率达到 **100%**，与 Full context 相同，后续输入从 4715 降至 3529 tokens，
|
||||
减少 **25.2%**。纯摘要为 73.3%，Top 2 为 86.0%，Top 3 为 94.0%；Top 5 增加 token，没有继续提分。
|
||||
所有方式均通过六项可执行修复测试，当前差异体现在事实保留率。
|
||||
4 个主实验任务中，Full context 为 100%，S 为 82.0%，S + Top 3 达到 **99.5%**，平均输入从 4014 降至
|
||||
3241 tokens，减少 **19.3%**。Top 4、Top 5 分别为 98.1%、97.4%，追加更多原文没有继续提分。
|
||||
每轮在评分前固定 S 和 Jev 排名，五种融合始终保留逐字相同的 S。
|
||||
|
||||
<!-- 原支付轨迹共有 10 个证据块，最多 Top 10；本次只测试 Top 1～5。 -->
|
||||
原轨迹共有 **10 个证据块，最多 Top 10**；本次测试 Top 1～5。摘要按 3200-token 完整输入额度生成，
|
||||
扩展实验的回答输入上限为 3800 tokens。所有融合保留逐字相同的 S，追加原文会增加输入长度，实际压缩输入均未超限。
|
||||
新增的 **Hermes 小窗口摘要适配器** 得分为 79.4%，平均输入 2432 tokens。它实际调用原生摘要生成器，
|
||||
但显式增加小窗口预算与字数约束，并使用摘要正文统一封装；不是默认自动压缩器。
|
||||
42/42 个上下文均满足预算，且实际 API 输入严格短于对应 Full，没有用原文回退充当压缩成功。
|
||||
|
||||
这是 Hermes 采集的一个合成任务，评测直接调用 Qwen，没有继续运行 Hermes 代理；重复测量不能证明通用任务表现，
|
||||
也未证明融合优于纯 Jev。融合仅在 benchmark 中实现，上述生产压缩契约保持不变。
|
||||
详见 [Benchmark 方案、评分与完整结果](https://github.com/jinliyl/ReMe/blob/26a83c1904d5c8312ed8e8cb5ffa21500b6e50e6/benchmark/compaction/RESULTS.md)。
|
||||
<!-- 支付轨迹有 10 个证据块，最多 Top 10；其他任务数量不同，实际 N 为 min(N, 可用证据块数)。 -->
|
||||
Top N 按完整证据块计数，超过可用数量时保留全部。原支付轨迹共有 10 块，最多 Top 10；本次测试 Top 1～5。
|
||||
摘要正文预算和完整输入上限按任务计算；这是逐步追加证据的曲线，不是等长比较，也不代表所有 Full 都超过上限。
|
||||
|
||||
共 336 个上下文、672 次回答，短任务对照单独报告。Hermes 负责采集，评分时直接调用 Qwen，没有继续工具循环或
|
||||
恢复原文。少量合成任务不能证明通用优势或统计等价；融合和原生摘要适配器均仅用于 benchmark。
|
||||
详见[完整 query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md)和[结果、对照与审计](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md)。
|
||||
|
|
|
|||
|
|
@ -1,16 +1,13 @@
|
|||
"""Frozen-summary incrementality, original evidence and deterministic payment scoring."""
|
||||
"""Whole-source selection, complete-input budgets and compressor data boundaries."""
|
||||
|
||||
import asyncio
|
||||
from copy import deepcopy
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmark.compaction.cases import GOLD, evaluate, event_units, exact_match, load_case, render, write
|
||||
from benchmark.compaction.repair_check import repair_checks
|
||||
from benchmark.compaction.run import Experiment, aggregate, answer, compose, decision_input, frozen_inputs, messages
|
||||
from benchmark.compaction.run import prepare, top_n
|
||||
from benchmark.compaction.cases import event_units, render
|
||||
from benchmark.compaction.run import Experiment, compose, decision_input, messages, top_n
|
||||
|
||||
|
||||
class CharacterTokenizer:
|
||||
|
|
@ -61,51 +58,6 @@ def test_invalid_rankings_never_become_top_n(value):
|
|||
top_n(units, {"0": value, "1": 0.5, "2": 0.5}, 1)
|
||||
|
||||
|
||||
def test_fusion_keeps_the_exact_same_summary_and_all_original_blocks(tmp_path):
|
||||
"""Every N adds evidence while preserving S, protected prompts and evaluation question."""
|
||||
case, units = fixture()
|
||||
before = deepcopy((case, units))
|
||||
exp = experiment(tmp_path)
|
||||
rows = prepare(exp, case, units, ["Fixed summary."], [{"0": 0.1, "1": 0.8, "2": 0.9}], 3)
|
||||
baseline = json.loads((tmp_path / "c0.summary.context.json").read_text())["messages"]
|
||||
for row in rows[2:]:
|
||||
context = json.loads((tmp_path / f"c0.{row['arm']}.context.json").read_text())["messages"]
|
||||
assert row["summary"] == "Fixed summary."
|
||||
assert context[2]["content"].startswith(baseline[2]["content"])
|
||||
assert context[:2] == baseline[:2] and context[-1] == baseline[-1]
|
||||
chosen = [u for u in units if u["id"] in row["selected"]]
|
||||
assert context == messages(case, compose("Fixed summary.", render(chosen)))
|
||||
assert (case, units) == before
|
||||
|
||||
|
||||
def test_over_budget_fusion_is_recorded_without_shrinking_s_or_skipping_raw(tmp_path):
|
||||
"""Budget pressure cannot invalidate the incremental comparison by changing S."""
|
||||
case, units = fixture()
|
||||
exp = experiment(tmp_path, cap=1)
|
||||
rows = prepare(exp, case, units, ["Fixed S"], [{"0": 0.9, "1": 0.8, "2": 0.7}], 3)
|
||||
assert all(not row["within_budget"] for row in rows[1:])
|
||||
assert rows[-1]["selected"] == ["0", "1", "2"] and rows[-1]["summary"] == "Fixed S"
|
||||
asyncio.run(answer(exp, rows[-1], case, 1))
|
||||
assert not rows[-1]["valid"] and rows[-1]["trials"] == []
|
||||
|
||||
|
||||
def test_actual_api_usage_overrun_invalidates_the_entire_condition(tmp_path):
|
||||
"""A local estimate cannot substitute for checking actual downstream usage."""
|
||||
|
||||
class FakeExperiment(Experiment):
|
||||
"""Only mock the provider boundary."""
|
||||
|
||||
async def ask(self, _label, _context, *, summary=False):
|
||||
"""Return a complete answer whose provider reports too many input tokens."""
|
||||
return "{}", {"prompt_tokens": self.cap + 1}
|
||||
|
||||
case, units = fixture()
|
||||
exp = FakeExperiment(None, None, tmp_path, CharacterTokenizer(), asyncio.Semaphore(1), "mock", 10000)
|
||||
row = prepare(exp, case, units, ["S"], [{"0": 0.1, "1": 0.2, "2": 0.3}], 1)[1]
|
||||
asyncio.run(answer(exp, row, case, 1))
|
||||
assert not row["valid"] and row["trials"] == []
|
||||
|
||||
|
||||
def test_grouping_preserves_every_source_character_and_correlated_events():
|
||||
"""Evidence grouping uses source correlations, never expected payment identifiers."""
|
||||
text = "".join(f"healthcheck {i}\n" for i in range(100))
|
||||
|
|
@ -142,64 +94,22 @@ def test_compressors_see_task_and_source_but_no_exam_or_gold(tmp_path):
|
|||
assert supplied["remaining_evidence"] == units and supplied["retained_evidence"] == []
|
||||
|
||||
|
||||
def test_frozen_replay_requires_every_generation_and_unchanged_evidence(tmp_path):
|
||||
"""Do not select successful summaries or silently mix different trajectories."""
|
||||
def test_frozen_s_is_a_literal_prefix_of_every_fusion_context():
|
||||
"""Adding original evidence must preserve the exact baseline summary and protected prompts."""
|
||||
case, units = fixture()
|
||||
write(tmp_path / "source.json", {**case, "units": units})
|
||||
write(tmp_path / "results.json", [{"arm": "summary", "repeat": 0, "summary": "S"}])
|
||||
(tmp_path / "provenance").mkdir()
|
||||
write(tmp_path / "provenance/paired.c0.decision0.response.json", {"scores": {"0": 0.2, "1": 0.8, "2": 0.9}})
|
||||
assert frozen_inputs(tmp_path, case, units, 1)[0] == ["S"]
|
||||
with pytest.raises(ValueError):
|
||||
frozen_inputs(tmp_path, case, units, 2)
|
||||
with pytest.raises(ValueError):
|
||||
frozen_inputs(tmp_path, {**case, "gold": {}}, units, 1)
|
||||
baseline = messages(case, compose("Fixed S", ""))
|
||||
for n in range(1, 4):
|
||||
chosen = top_n(units, {"0": 0.1, "1": 0.8, "2": 0.9}, n)
|
||||
context = messages(case, compose("Fixed S", render(chosen)))
|
||||
assert context[2]["content"].startswith(baseline[2]["content"])
|
||||
assert context[:2] == baseline[:2] and context[-1] == baseline[-1]
|
||||
|
||||
|
||||
def test_wrong_trajectory_cannot_be_scored_against_the_frozen_payment_oracle(tmp_path):
|
||||
"""The known gold is specific to a pinned original source, not arbitrary input data."""
|
||||
path = tmp_path / "trajectory.json"
|
||||
path.write_text('{"messages": []}')
|
||||
with pytest.raises(ValueError, match="unchanged original"):
|
||||
load_case(path)
|
||||
|
||||
|
||||
def test_grading_is_type_strict_and_failures_stay_in_denominator():
|
||||
"""Boolean/integer equality and omitted failed conditions cannot inflate results."""
|
||||
assert not exact_match(True, 1) and not exact_match([True], [1])
|
||||
result = evaluate({"gold": GOLD}, {"facts": {**GOLD, "max_attempts": True}})
|
||||
assert sum(result["facts"].values()) == 9
|
||||
rows = [
|
||||
{"arm": "summary", "valid": False, "trials": []},
|
||||
{
|
||||
"arm": "summary",
|
||||
"valid": True,
|
||||
"trials": [
|
||||
{
|
||||
"checks": {"facts": {"x": True}, "task": {"x": True}},
|
||||
"usage": {"prompt_tokens": 100},
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
assert aggregate(rows)[0]["facts"] == aggregate(rows)[0]["task"] == 0.5
|
||||
|
||||
|
||||
def test_repair_behavior_checks_detect_unstable_keys_and_reject_unbounded_code():
|
||||
"""Correct-looking plans do not substitute for six executable fake-client checks."""
|
||||
source = """def charge_with_retry(client, invoice, attempts=3):
|
||||
key = str(uuid.uuid4())
|
||||
for i in range(attempts):
|
||||
try:
|
||||
return client.charge(invoice, idempotency_key=key)
|
||||
except TimeoutError:
|
||||
if i == attempts - 1:
|
||||
raise
|
||||
"""
|
||||
assert all(repair_checks(source).values())
|
||||
buggy = source.replace(" key = str(uuid.uuid4())\n", "").replace(
|
||||
" for i in range(attempts):\n",
|
||||
" for i in range(attempts):\n key = str(uuid.uuid4())\n",
|
||||
)
|
||||
assert not repair_checks(buggy)["timeout_then_success"]
|
||||
assert not any(repair_checks("while True: pass").values())
|
||||
def test_complete_input_budget_counts_protected_prompts_and_followup(tmp_path):
|
||||
"""The last token is shared by summary, original evidence and all input framing."""
|
||||
case, _ = fixture()
|
||||
exp = experiment(tmp_path)
|
||||
payload = compose("S", "raw evidence")
|
||||
exp.cap = exp.input_tokens(messages(case, payload)) + 64
|
||||
assert exp.fits(case, payload)
|
||||
assert not exp.fits(case, payload + "x")
|
||||
|
|
|
|||
151
tests/unit/test_compaction_study.py
Normal file
151
tests/unit/test_compaction_study.py
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
"""Rule grading, source provenance and equal-task aggregation for the fixture study."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmark.compaction.cases import event_units, write
|
||||
from benchmark.compaction.suite_native import validate_base
|
||||
from benchmark.compaction.suite import context, define, exam, grade
|
||||
from benchmark.compaction.suite_run import ARMS, load_cases, metrics
|
||||
|
||||
|
||||
def sample():
|
||||
"""Define independent expected values, including strict nested types."""
|
||||
return define(
|
||||
"example",
|
||||
"Read the project files.",
|
||||
{"source.txt": "evidence"},
|
||||
{"number": 7, "flag": True, "mapping": {"a": 2}, "ordered": ["a", "b"]},
|
||||
{"number": "integer value", "flag": "boolean state", "mapping": "mapping", "ordered": "ordered values"},
|
||||
"fixture/example",
|
||||
)
|
||||
|
||||
|
||||
def test_followup_never_includes_expected_answers():
|
||||
"""Only the requested field labels and descriptions enter the evaluation prompt."""
|
||||
case = sample()
|
||||
case["gold"] = {"hidden": "PRIVATE_GOLD_8391"}
|
||||
assert "PRIVATE_GOLD" not in exam(case)
|
||||
assert "integer value" in exam(case)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [True, 7.0, "7", None])
|
||||
def test_rules_do_not_confuse_numeric_types(value):
|
||||
"""No model judge or JSON coercion can turn a wrong type into a correct fact."""
|
||||
case = sample()
|
||||
assert not grade(case, {"facts": {**case["gold"], "number": value}})["number"]
|
||||
|
||||
|
||||
def test_mapping_shape_and_array_order_are_part_of_the_contract():
|
||||
"""Require an object for named test counts and preserve meaningful list ordering."""
|
||||
case = sample()
|
||||
assert all(grade(case, {"facts": case["gold"]}).values())
|
||||
result = grade(case, {"facts": {**case["gold"], "mapping": [2], "ordered": ["b", "a"]}})
|
||||
assert not result["mapping"] and not result["ordered"]
|
||||
assert not any(grade(case, []).values())
|
||||
|
||||
|
||||
def test_format_clarifications_are_case_specific_and_do_not_reveal_counts():
|
||||
"""Clarify ambiguous output shapes without changing the oracle or unaffected questions."""
|
||||
case = sample()
|
||||
payment = {**case, "id": "payment"}
|
||||
notes = {**case, "id": "meeting_notes"}
|
||||
assert "must be a JSON object" in exam(payment)
|
||||
assert "timeout_then_success" in exam(payment)
|
||||
assert "leading list marker" in exam(notes)
|
||||
assert "must be a JSON object" not in exam(case)
|
||||
|
||||
|
||||
def test_matching_line_views_retain_original_line_numbers_and_context():
|
||||
"""The same source query selects a deterministic view without answer-based trimming."""
|
||||
text = "before\nERROR exact-id\nafter\nlast\n"
|
||||
assert context(text, r"ERROR", radius=1) == "1|before\n2|ERROR exact-id\n3|after\n"
|
||||
assert context(text, r"absent", radius=1) == ""
|
||||
|
||||
|
||||
def test_denied_reads_remain_lossless_observations_in_the_trajectory():
|
||||
"""A recovered failed read cannot crash evidence extraction or disappear from history."""
|
||||
text = json.dumps({"error": "Only listed source files may be read"})
|
||||
messages = [
|
||||
{"role": "assistant", "tool_calls": [{"id": "c", "function": {"arguments": '{"path":"missing"}'}}]},
|
||||
{"role": "tool", "tool_call_id": "c", "content": text},
|
||||
]
|
||||
units = event_units(messages)
|
||||
assert units[0]["text"] == text
|
||||
assert units[0]["spans"][0]["text"] == text
|
||||
|
||||
|
||||
def test_task_weights_and_failed_conditions_have_fixed_denominators():
|
||||
"""A task with many checked fields cannot outweigh a task with only one."""
|
||||
rows = []
|
||||
for arm in ARMS:
|
||||
for task, checks in [("a", dict.fromkeys(["x", "y", "z"], True)), ("b", {"x": False})]:
|
||||
rows.append(
|
||||
{
|
||||
"case": task,
|
||||
"arm": arm,
|
||||
"valid": True,
|
||||
"trials": [{"checks": checks, "usage": {"prompt_tokens": 10}}],
|
||||
},
|
||||
)
|
||||
assert metrics(rows, ["a", "b"], 1)[0]["facts"] == 0.5
|
||||
rows[0].update(valid=False, trials=[])
|
||||
assert metrics(rows, ["a", "b"], 1)[0]["facts"] == 0
|
||||
|
||||
|
||||
def test_corpus_loader_excludes_final_reports_and_checks_hashes(tmp_path):
|
||||
"""Internal DB metadata and the completed answer cannot leak into scoring history."""
|
||||
task = tmp_path / "example"
|
||||
work = task / "workspace"
|
||||
work.mkdir(parents=True)
|
||||
(work / "source.txt").write_text("evidence")
|
||||
source = {
|
||||
"messages": [
|
||||
{"role": "system", "content": "real system"},
|
||||
{"role": "user", "content": "task", "_db_row_snapshot": {"content": "PRIVATE_COPY"}},
|
||||
{"role": "assistant", "content": "FINAL_REPORT"},
|
||||
],
|
||||
}
|
||||
write(task / "trajectory.full.json", source)
|
||||
source_bytes = (task / "trajectory.full.json").read_bytes()
|
||||
write(task / "collection.json", {"completed": True, "trajectory_sha256": hashlib.sha256(source_bytes).hexdigest()})
|
||||
case = {
|
||||
**sample(),
|
||||
"source_hashes": {"source.txt": hashlib.sha256(b"evidence").hexdigest()},
|
||||
"control_reason": None,
|
||||
}
|
||||
write(tmp_path / "corpus.json", {"cases": [case]})
|
||||
_, cases = load_cases(tmp_path / "corpus.json")
|
||||
assert len(cases["example"]["messages"]) == 2
|
||||
assert "PRIVATE_COPY" not in json.dumps(cases)
|
||||
assert "FINAL_REPORT" not in json.dumps(cases)
|
||||
(work / "source.txt").write_text("changed")
|
||||
with pytest.raises(ValueError, match="Changed source"):
|
||||
load_cases(tmp_path / "corpus.json")
|
||||
|
||||
|
||||
def test_native_rejects_incompatible_or_duplicate_base_panels(tmp_path):
|
||||
"""Paid native calls must not silently combine different experiment protocols."""
|
||||
corpus = tmp_path / "corpus.json"
|
||||
corpus.write_text("frozen")
|
||||
cases = [str(i) for i in range(14)]
|
||||
rows = [{"case": key, "arm": arm, "repeat": i} for key in cases for arm in ARMS for i in range(3)]
|
||||
protocol = {
|
||||
"model": "qwen3.8-max",
|
||||
"compressions": 3,
|
||||
"answer_repeats": 2,
|
||||
"corpus_sha256": hashlib.sha256(corpus.read_bytes()).hexdigest(),
|
||||
}
|
||||
write(tmp_path / "protocol.json", protocol)
|
||||
write(tmp_path / "results.json", rows)
|
||||
assert validate_base(tmp_path, corpus, cases) == rows
|
||||
write(tmp_path / "results.json", rows + [rows[0]])
|
||||
with pytest.raises(ValueError, match="exactly once"):
|
||||
validate_base(tmp_path, corpus, cases)
|
||||
write(tmp_path / "results.json", rows)
|
||||
protocol["model"] = "different-model"
|
||||
write(tmp_path / "protocol.json", protocol)
|
||||
with pytest.raises(ValueError, match="matching fourteen-task"):
|
||||
validate_base(tmp_path, corpus, cases)
|
||||
Loading…
Add table
Reference in a new issue