diff --git a/README.md b/README.md index 9abe45d3..c7b31cd0 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ users retain control of the durable files. - [2026.10] - **[Decision-model context compaction](docs/en/context_compaction.md)**: score old tool calls and results against the next action, preserve protected history, and return a recoverable compaction proposal. Includes an SVG - walkthrough, two live-model integration scenarios, and a fixed-summary Top-N benchmark. + walkthrough, two live-model integration scenarios, and a 14-query fixed-summary Top-N benchmark with a forced Hermes summary baseline. - [2026.10] - **[ReMe Multimodal Memory](https://reme.agentscope.io/en/blog_20261009) update**: Auto Memory reads images with their conversation context and preserves image sources; Auto Resource turns image files into searchable daily cards linked to the originals. diff --git a/README_ZH.md b/README_ZH.md index 7b3a99a4..27049c3f 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -50,7 +50,7 @@ - [2026.10] - **[基于决策模型的上下文压缩](docs/zh/context_compaction.md)**：根据下一步任务判断旧工具证据的保留价值， - 保护关键历史，并返回可回溯的压缩建议；提供 SVG 流程图、两个真实模型集成场景和固定摘要 Top-N 实验。 + 保护关键历史，并返回可回溯的压缩建议；提供 SVG 流程图、两个真实模型集成场景和 14-query 固定摘要实验，包含强制 Hermes 摘要基线。 - [2026.10] - **[ReMe 多模态记忆更新](https://reme.agentscope.io/zh/blog_20261009)**：Auto Memory 将会话图片和上下文一起 写入记忆并保留图片来源；Auto Resource 将图片资料整理为可检索、可回溯原图的 daily 卡片。 - [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=zh) 上线**：无需安装或启动后端， diff --git a/benchmark/compaction/README.md b/benchmark/compaction/README.md index f2d99c75..02883cf8 100644 --- a/benchmark/compaction/README.md +++ b/benchmark/compaction/README.md @@ -1,56 +1,57 @@ -# Frozen summary + Jev Top N +# Context compaction: fixed summary + selected evidence -One payment trajectory, one fixed-summary experiment, one [results table](./RESULTS.md). -The question is whether adding selected original evidence repairs information lost by a summary. -This benchmark is separate from production `compact_context`. +A **14-query fixture panel** with predefined answers and rule scoring: four primary tasks and ten short/query-only +controls. See [queries, answers and frozen settings](./STUDY.md) and [measured results](./RESULTS.md). -## Protocol - -Hermes collected the synthetic payment investigation: 11 messages with the actual system prompt and six paired tool -calls/results. Exclude the final assistant report, then losslessly group tool results into **10 evidence units**. -Short files remain whole; long logs group correlated events and routine text. No noise or identifier variants are added. -The fixture supplies ten exact facts and six executable repair checks; no LLM creates or judges the gold answers. - -For each of five generations, create one summary **S** of all evidence and one Jev ranking using only the initial task -and original evidence. Neither compressor sees the follow-up question or gold. Freeze both before answering: - -- **Full context:** original history, allowed to exceed the compressed-input ceiling. -- **S:** the frozen summary. -- **S + Jev Top 1–5:** exactly the same S, plus the first N original units ranked by usefulness. - -Select a strict descending score prefix, break ties by unit ID, and restore source order when rendering. There is no -probability threshold, skipping, truncation or summary rewriting. **There are 10 units, so Top 10 is the maximum; the -reported sweep tests Top 1–5.** Larger N requires more input space. - -S is generated with a **3,200-token complete-input allowance**. The sweep uses a **3,800-token complete-input ceiling**, -including the same original system/task, evidence, follow-up and framing. This is incremental evidence at extra input -cost, not an equal-length S-versus-fusion comparison. A local Qwen tokenizer and 64-token reserve budget requests; -actual API usage is checked independently. Overruns fail instead of shrinking S or silently dropping original units. - -Each context receives three answers: 35 contexts and 105 answers, temperature zero, thinking disabled. Average ten -strict fact checks within each answer, then answers and generations equally. Also report the proportion passing all -six restricted fake-client repair tests. Invalid conditions stay in the denominator with zero credit. -**Hermes collected the source; evaluation calls Qwen directly**, without tool continuation or recovery. +Hermes collects the source through real `read_file` calls. ReMe's experiment then compares Full context, a frozen Qwen +summary S, S + Jev Top 1–5, and a **Hermes small-window summary adapter**. Scored answering calls Qwen directly; +there is no resumed Hermes tool loop or archive recovery. Production `compact_context` is separate. ## Run -Use the development environment plus optional `tokenizers`; no Hermes runtime is needed for replay. -The runner accepts only the pinned original payment trajectory whose fixture oracle is in [cases.py](./cases.py). -Credentials come from `.env`: `LLM_API_KEY`, `LLM_BASE_URL`, `DECISION_MODEL_API_KEY`, and optional -`DECISION_MODEL_NAME` / `DECISION_MODEL_BASE_URL`. Provider calls require `--live`. +Use ReMe's development environment plus optional `tokenizers`. Collection and native generation need a separately +installed Hermes environment, also able to import `tokenizers`. Credentials come from ReMe's `.env`: +`LLM_API_KEY`, `LLM_BASE_URL`, `DECISION_MODEL_API_KEY`, and optional `DECISION_MODEL_NAME` / `DECISION_MODEL_BASE_URL`. +All commands require `--live`; outputs must be new directories outside this repository. ```bash -python -m benchmark.compaction.run --live \ - --trajectory /path/outside/repository/trajectory.full.json \ +# Freeze all task/answer definitions, then collect 13 new read-only trajectories. +# Reuse the pinned original payment trajectory as the fourteenth source. +python -m benchmark.compaction.suite_collect --live \ + --hermes /path/to/hermes-agent \ + --hermes-python /path/to/hermes-environment/bin/python \ + --payment-trajectory /path/to/original/trajectory.full.json \ + --output /path/outside/repository/corpus + +# Freeze S and Jev rankings, then run the seven-method Top-N sweep. +python -m benchmark.compaction.suite_run --live \ + --corpus /path/outside/repository/corpus/corpus.json \ --tokenizer /path/to/cached/qwen/tokenizer.json \ - --output /path/outside/repository/new-evaluation + --output /path/outside/repository/evaluation + +# Generate the native-adapter summaries and add the eighth method. +# An existing complete --native directory reuses all frozen native generations. +python -m benchmark.compaction.suite_native --live \ + --corpus /path/outside/repository/corpus/corpus.json \ + --evaluation /path/outside/repository/evaluation \ + --native /path/outside/repository/native-generations \ + --hermes /path/to/hermes-agent \ + --hermes-python /path/to/hermes-environment/bin/python \ + --tokenizer /path/to/cached/qwen/tokenizer.json \ + --output /path/outside/repository/combined-results ``` -Defaults: `qwen3.8-max`, five generations, three answers, Top 1–5 and four concurrent requests. To use every previously -frozen S/ranking pair without new compression calls, add `--frozen-run /path/to/previous/results`. This generates fresh -answers for every method; `--compressions` must match all saved summary generations. For an offline input reconstruction, -replace `--live` with `--check-only --frozen-run /path/to/previous/results`. +The task definitions use local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b` (MIT). The recorded study uses +`qwen3.8-max`, three summary/ranking generations and two answers per context: **336 contexts, 672 answers**. Provider +requests are bounded to four at a time. Raw files, sessions, wire bodies, code snapshots and per-field grades stay +outside git; existing runs are never overwritten. -`--top-n` can extend the sweep to at most 10. Increase `--input-cap` explicitly if needed; the runner records over-budget -contexts rather than changing their content. Outputs must be new directories outside the repository. Protocol, source -hashes, frozen selections, code snapshots, requests/responses, usage and per-answer checks remain there. +S remains byte-identical across its five fusion variants. Top N means a strict descending-score prefix, without a +probability threshold, skipping or truncation; selected units are rendered in source order. When a task has fewer +than N units, all available units are retained and `actual_n` records the count. The original payment source has ten +units (maximum Top 10); this study sweeps Top 1–5 with task-specific unit counts. + +The native baseline invokes Hermes' actual summary generator with an explicit small-window adapter: legacy template, +S-matched body budget, final word constraint, native provenance validation, and body-only downstream framing. It bypasses +normal triggers/head-tail selection and **must actually shorten input**; it is not the unchanged automatic Hermes policy. +See [the exact simulation settings and preflight disclosures](./STUDY.md#added-baseline-hermes-small-window-summary-adapter). diff --git a/benchmark/compaction/RESULTS.md b/benchmark/compaction/RESULTS.md index 11ca5d20..ab1a404b 100644 --- a/benchmark/compaction/RESULTS.md +++ b/benchmark/compaction/RESULTS.md @@ -1,63 +1,97 @@ -# Fixed summary + selected original evidence +# Fixed-summary compaction on fourteen fixture tasks -2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · one original Hermes-collected synthetic payment trajectory. +2026-10-16 · Qwen `qwen3.8-max` · Jev `jev-1.13.0` · **14 distinct queries with predefined answers**. +Four source-length-qualified tasks form the primary cohort; ten short/query-only controls are reported separately. +Cohorts were fixed before collection, with no accuracy-based task filtering. -| Method | Input tokens | Fact accuracy | -| --- | ---: | ---: | -| Full context | 4,715 | 100.0% | -| S | 2,937 | 73.3% | -| S + Jev Top 1 | 3,134 | 72.7% | -| S + Jev Top 2 | 3,247 | 86.0% | -| S + Jev Top 3 | 3,360 | 94.0% | -| S + Jev Top 4 | 3,529 | 100.0% | -| S + Jev Top 5 | 3,585 | 100.0% | +## Primary cohort: four tasks -Input tokens are mean actual downstream API usage, including the original system/task and follow-up. -Every method has five valid contexts and 15 answers. All answers pass all six executable repair checks; the table -therefore measures differences in fact retention, not a demonstrated difference in repair success. +| Method | Mean input tokens | Fact score | Valid contexts | +| --- | ---: | ---: | --- | +| Full context | 4,014 | 100.0% | 12/12 | +| S | 2,480 | 82.0% | 12/12 | +| Hermes small-window adapter | 2,432 | 79.4% | 12/12 | +| S + Jev Top 1 | 2,753 | 83.0% | 12/12 | +| S + Jev Top 2 | 2,974 | 94.5% | 12/12 | +| S + Jev Top 3 | 3,241 | 99.5% | 12/12 | +| S + Jev Top 4 | 3,496 | 98.1% | 12/12 | +| S + Jev Top 5 | 3,660 | 97.4% | 12/12 | -**Top 4 matches Full context on the ten checked facts while reducing downstream input by 25.2%.** Adding Top 5 -uses more tokens without improving this score. Top 1 does not improve S; Top 2 and Top 3 improve it by 12.7 and -20.7 percentage points. These observations are specific to this task, not statistical equivalence or general superiority. +Input tokens are actual downstream API usage, including the original system/task and follow-up. Fact score averages +strict JSON field checks within each answer, then generations and tasks equally; invalid conditions score zero. +Full context reaches 100%; S + Top 3 reaches **99.5% with 19.3% less downstream input** (3,241 versus 4,014 tokens). +Top 4 and Top 5 do not improve this point estimate. No equivalence test or general optimal-N claim is made. -![Frozen summary plus ranked original evidence](../../docs/figure/compaction-topn-results.svg) +![Primary-cohort input and fact score](../../docs/figure/compaction-topn-results.svg) -## What is fixed +## What was tested -The [protocol and runnable commands](./README.md) keep one byte-identical summary S and one Jev ranking per generation. -S summarizes all original evidence under a 3,200-token complete-input allowance; fusion appends original blocks without -rewriting S. The extended sweep raises the final complete-input ceiling from 3,600 to **3,800 tokens** before new answer -calls, solely to accommodate Top 5 with its full evidence and budgeting reserve. All compressed API inputs are within -this ceiling (2,924–3,602 tokens). More originals intentionally cost more tokens; lengths are not forced equal. +The [full query/answer table and protocol](./STUDY.md) define the oracle before collection. The four primary tasks are +payment retry investigation, checkout 504 analysis, server-error localization and a build-output token lookup. +The ten controls cover shorter configuration, code, dependency, notes, version and file-generation questions. +All fourteen sources are synthetic fixtures: one existing payment trajectory plus thirteen genuine Hermes executions +over explicitly read-only adaptations of upstream tasks. Long-file views are deterministic original excerpts, not +added noise. These are not the maintainers' private production transcripts. -The source has **10 evidence units, so the maximum is Top 10**. This sweep tests Top 1–5. In the first ranking, Top 3 -contains the incident log, retry implementation and payment policy; Top 4 adds `expected_cases.json`, restoring the exact -regression attempt counts that S + Top 3 sometimes loses. Top 5 also retains the README. +Each task has three frozen summary/ranking pairs and two answers per method: **336 contexts and 672 answers** across +eight methods. S summarizes every source unit once. Each S + Top N preserves that S byte-for-byte and adds complete +original units, ranked without a threshold and restored to source order. Actual N saturates at the available unit count. +The original payment source has ten units, so its maximum is Top 10; the sweep tests Top 1–5. -There are five frozen summary/ranking pairs and three answers per method: **35 contexts, 105 answers**. Full context, -S and Top 1–2 reuse all 60 answers from the immediately preceding paired run. Top 3–5 add 45 new answers. All five pairs -are retained; no prediction-based filtering, new summaries or new Jev scores enter the extension. +S's evidence allowance is 20% of rendered-evidence tokens, clamped to 64–256 local Qwen tokens. A per-task complete-input +ceiling is calculated from protected prompts, the summary allowance, the five largest whole units and reserve, before +scored answers. This permits the incremental sweep; **it is not an equal-length S-versus-fusion comparison or a claim +that every Full input exceeds its ceiling**. All actual inputs are checked. Short cases may gain no token savings. -## Gold, audit and scope +The added **Hermes small-window adapter** calls the real native generator, with its legacy template, an overridden +64–256-token body budget and an appended total word limit. It retains native redaction/provenance checks and strips +only the native control prefix before the common summary carrier. It bypasses automatic triggering and full-compressor +head/tail selection: **this is a forced simulation, not the unchanged default Hermes compressor**. All **42/42** native +contexts fit their body/input budgets and use fewer actual API input tokens than the corresponding Full context. +No original-history fallback is accepted. -The original fixture predates these evaluations. Its oracle contains ten facts: incident, invoice and charge IDs, -amount, ordered idempotency keys, first-attempt commit state, retry limit, non-retryable HTTP codes, original key-generation -line and named regression-test attempt counts. Matching is type-strict and nested; HTTP-code order is ignored. The six -restricted executable checks cover success, timeout retry with a stable key, HTTP 402/422 propagation, three-attempt -exhaustion and independent keys for separate payments. No LLM judge or generated answer key is used. +Hermes collects the trajectories; answering calls Qwen directly. Gold and follow-up questions never enter collection, +summary or decision requests. The final collection report is excluded from every history. There is no tool continuation, +retrieval, LLM judge or generated answer key. Exact string/type compliance is part of this metric; it does not by itself +prove semantic information loss or task-completion differences. The native and S methods have the same maximum body +allowance, but their actual lengths differ. Four primary templates are too few to establish general superiority. -The source has 11 messages, six tool calls and six complete results, including the actual system prompt. Its final -assistant report is excluded from compression. No synthetic noise or identifier substitutions are added. Source SHA-256: -`0f37e6510ffafd920d088452d8719026b00fd4e74f87f7dd1a42a85c8287e6f1`. +
+Ten short/query-only controls -The audit replayed all 105 grades, verified exact retained originals and nested Top-N sets, byte-identical summaries, -protected prompts, original source hash, actual input ceilings and unchanged reused request/response records. No configured -API keys were found. The 45 new Qwen calls use 157,092 input and 9,316 output tokens; existing answers and compression -have additional usage. Downstream input reduction is not a claim about total pipeline cost. +| Method | Mean input tokens | Fact score | Valid contexts | +| --- | ---: | ---: | --- | +| Full context | 2,727 | 100.0% | 30/30 | +| S | 2,315 | 67.1% | 30/30 | +| Hermes small-window adapter | 2,308 | 73.2% | 30/30 | +| S + Jev Top 1 | 2,493 | 84.7% | 30/30 | +| S + Jev Top 2 | 2,573 | 87.5% | 30/30 | +| S + Jev Top 3 | 2,687 | 92.9% | 30/30 | +| S + Jev Top 4 | 2,722 | 92.9% | 30/30 | +| S + Jev Top 5 | 2,722 | 92.5% | 30/30 | -Artifacts stay outside the repository under `trajectory-runs/payment-fixed-summary-topn1to5-20261016/results/`: -`protocol.json`, frozen contexts/selections, exact requests/responses, per-answer results and `audit.json`. -One synthetic task with repeated measurements is not 35 independent tasks. This is direct Qwen evaluation, not Hermes -continuing its agent loop. The earlier length-matched diagnostic also found pure Jev reaching 100% with about 3,251 -input tokens on this task, so **this sweep does not establish fusion superiority over pure Jev**. Production -`compact_context` retains its conservative preview/recovery contract; this hybrid remains benchmark-only. +All controls remain in their separate denominator. They are not presented as ten additional long-context tasks. + +
+ +## Audit and reproducibility + +See [commands](./README.md). The audit replayed all **672** grades, reconstructed original source spans, verified source +hashes and actual initial system prompts, checked frozen S and strict nested Top-N sets, protected prompts, question/gold +isolation, actual token ceilings, native strict shrinkage and configured-key exclusion. All 336 contexts are valid. + +The format preflight exposed two underspecified answer shapes: payment counts as arrays and note text with bullet +markers. Those two follow-ups were clarified without changing gold or graders, and every seven-arm condition on both +tasks was re-answered using the same S/rankings. Their 84 initial answers remain archived and excluded; all 504 answers +from the other twelve tasks remain included. The native adapter adds 84 fresh answers. Unadapted native/length-only +calibration attempts are also retained; they are not silently counted as successful compression. + +Formal recorded usage: **757 Qwen calls** (672 answers, 42 S calls, 43 native calls including one length retry), +2,027,184 input / 40,099 output tokens; **54 Jev calls**, 77,799 input / 5,634 output tokens. This excludes Hermes +collection and preflights, including interrupted calibration attempts. No complete monetary-cost claim is made. +Downstream input reduction is not total pipeline savings or a prompt-cache measurement. + +Artifacts remain outside the repository: `hermes-fixture14-20261016/` holds the frozen corpus and actual Hermes sessions; +`compaction-fixture14-20261016-v2/` holds the seven-arm records; `compaction-fixture14-hermes-adapted-20261016/` holds native +generation; `compaction-fixture14-final-20261016/` holds the combined results and `audit.json`. This is a development panel, +including a previously used payment task, not independent held-out confirmation. Production `compact_context` is unchanged. diff --git a/benchmark/compaction/STUDY.md b/benchmark/compaction/STUDY.md new file mode 100644 index 00000000..2b9290ea --- /dev/null +++ b/benchmark/compaction/STUDY.md @@ -0,0 +1,97 @@ +# Fourteen-task study: frozen queries, answers and rules + +The initial task panel was frozen before collection. Explicit format corrections and native additions are recorded below. +It evaluates **14 distinct task/answer templates**: +one existing payment trajectory and thirteen newly collected read-only adaptations of Hermes task fixtures. +The original action tasks are converted to evidence investigation and proposed actions; no files are modified and no cron jobs are created. +Hermes performs real read_file calls over immutable source snapshots. Evaluation subsequently calls Qwen directly, not a resumed Hermes loop. + +## Queries and expected answers + +Each initial query below supplies the collection goal. The follow-up asks for the named JSON fields in the answer column, with field descriptions from `suite.py`. +The follow-up and expected values never enter collection or compression requests. This is one task with a follow-up, not two independent samples. + +| ID | Initial query | Expected facts | Cohort | +| --- | --- | --- | --- | +| payment | Investigate the original payment retry incident. | `{"incident_id": "PAY-4827", "invoice_id": "INV-731", "successful_charge_id": "CH-991", "amount_minor": 12900, "attempt_keys": ["pay-a", "pay-b"], "first_attempt_committed": true, "max_attempts": 3, "no_retry_http_statuses": [402, 422], "key_generation_line": 5, "regression_attempt_counts": {"timeout_then_success": 2, "validation_422": 1, "three_timeouts": 3}}` | Primary candidate | +| backup | A while back I told you one of my repos needed nightly backups — find which repo that was in our past conversations (do not guess). Then: write backup.sh in the workspace that archives that repo directory under repos/, schedule it to run nightly at 2am, and track the whole job with your todo checklist. Report back which repo it was and what you set up. | `{"repo": "sparks-data", "ignored_repo": "toybox", "data_text": "important"}` | Control: short_fixture | +| refactor | In src/, every requests call uses timeout=30. Bump them all to timeout=45, add a CHANGELOG.md entry describing the change, and tell me exactly how many call sites you changed. | `{"old_timeout": 30, "call_sites": 3, "functions": ["fetch_a", "fetch_b", "fetch_c"], "methods": ["get", "post", "head"], "old_version": "1.0"}` | Control: short_fixture | +| three_files | This is a multi-step job — track it with your todo checklist tool and keep it updated as you go, marking each step completed when verified. Steps: (1) create greet.py containing a greet(name) function that returns 'hello '; (2) create data/rows.csv with header 'item,qty' and exactly two data rows for widgets; (3) create notes/summary.md stating how many files you created. Finish only when all three are done and checked off. | `{"file_count": 3, "header": "item,qty", "data_rows": 2}` | Control: query_only | +| active_config | Search this workspace for wherever the retry_limit setting is configured and tell me its current value. | `{"retry_limit": 7, "backoff": 2, "active_file": "conf/app.ini", "old_file": "conf/old.ini"}` | Control: short_fixture | +| checkout_504 | Investigate the checkout 504 incident: distinguish the failed rollback, root cause and final database fix. | `{"setting": "statement_timeout", "value_seconds": 45, "index_mode": "CONCURRENTLY", "root_cause": "CREATE INDEX without CONCURRENTLY", "rolled_back_version": "v2.30.2", "rollback_fixed": false, "deployed_version": "v2.31.0", "initial_error_percent": 8}` | Primary candidate | +| dependency_version | What exact left-pad version is pinned, and which file declares the test script? Record the package scripts for a handoff. | `{"version": "1.3.0", "declaration": "package.json", "test": "vitest run", "build": "tsc -p .", "package_version": "4.2.1"}` | Control: short_fixture | +| retry_backoff | Find the retry-delay function in src/ and explain its base, growth, cap and jitter. | `{"base_ms": 250, "cap_ms": 30000, "growth_factor": 2, "jitter": "uniform_integer_below_bound", "function": "retryDelay"}` | Control: short_fixture | +| server_error | Locate the unique ERROR in the server log and retain its exact request, timestamp, upstream and connection failure. | `{"request_id": "req-7f3d9", "timestamp": "2026-08-08T23:41:17Z", "status": 502, "upstream": "payments", "error": "ECONNREFUSED", "address": "10.0.4.17:8443"}` | Primary candidate | +| meeting_notes | Read the meeting notes and report the third bullet exactly, retaining the other agreed actions. | `{"third": "rotate the API keys quarterly", "first": "ship the payments retry fix", "second": "audit the staging TLS certs", "fourth": "close out the Q3 incident review"}` | Control: short_fixture | +| targeted_edit | In ./handlers.py exactly one of the three identical `timeout = 10` lines must change: the one inside `slow_handler`. Change it to `timeout = 60` using the patch tool. Then run `python3 check_handlers.py` from . and reply with its output. | `{"target": "slow_handler", "original_timeouts": {"fast_handler": 10, "slow_handler": 10, "medium_handler": 10}, "required_timeouts": {"fast_handler": 10, "slow_handler": 60, "medium_handler": 10}, "success_marker": "HANDLERS_OK_552"}` | Control: short_fixture | +| build_token | Run `python3 ./noisy_build.py` (it prints a lot). Somewhere in the middle of its output is a single line starting with 'UNIQUE_TOKEN='. Reply with the full token value. | `{"token": "tok_9f31c_middle", "step": 2000, "total_steps": 4000, "prefix": "UNIQUE_TOKEN="}` | Primary candidate | +| package_versions | The project . has three package dirs: pkg_a, pkg_b, pkg_c, each containing version.txt. Working through the directories, collect the three versions and create ./versions.txt containing them comma-separated in order (a,b,c). Reply DONE plus the joined string. | `{"versions": {"pkg_a": "1.4.2", "pkg_b": "0.9.7", "pkg_c": "3.2.1"}, "joined": "1.4.2,0.9.7,3.2.1"}` | Control: short_fixture | +| checksum_anomaly | The file ./records.log contains exactly one line starting with 'ANOMALY:'. Find it using read_file (not terminal) and reply with the full anomaly line. | `{"line": "ANOMALY: checksum drift detected in shard 7 (code X99Q)", "code": "X99Q", "shard": 7, "line_number": 4201}` | Control: short_fixture | + +## Scoring + +**No LLM judge.** The model must return `{"facts": {...}}`. Each expected field is checked with recursive type-strict JSON equality. +Integers differ from booleans and floats. Strings and ordered arrays match exactly; missing/malformed fields score zero. +Only payment `no_retry_http_statuses` ignores list order. No answer aliases, rewrites or retries are added after scoring. +Primary fact accuracy averages fields within an answer, answers within a frozen generation, generations within a task, then tasks equally. +Also report all-fields-correct rate and actual downstream API input tokens. Failed planned conditions remain zero in the denominator. +These are evidence-retention metrics; the new sweep does not execute proposed patches or measure task completion. + +## Cohorts fixed before collection + +Primary candidates require at least 2,000 source-snapshot characters and evidence-dependent facts. The original payment source is pinned separately. +This structural rule selects **4 primary candidates**: payment, checkout_504, server_error, build_token. + +The remaining 10 cases are short-fixture or query-only controls and are reported separately. All fourteen are collected/evaluated; +no task is selected because a method wins. Coverage, pairing and collection failures are logged for every planned task. + +## Paired experiment + +- Model: `qwen3.8-max`; same actual system prompt, task and follow-up within each case. Final collection reports are excluded. +- Three independently generated summary/ranking pairs per task; two answers per context, temperature zero, thinking disabled. +- Methods: Full context, S, S + Jev Top 1, 2, 3, 4, 5. Total planned: **294 contexts and 588 answers**. +- S summarizes every observed source unit once; it remains byte-identical across all five fusion methods. +- Jev sees only the initial task and source evidence. Scores are shared across N, with no probability threshold. +- Select the strict score prefix, then render in source order. If fewer than N units exist, retain all of them and report the actual count. +- Summary allowance: 64–256 local Qwen tokens, equal to 20% of rendered evidence tokens clamped to those limits. The allowance is frozen from source length. +- Before answering, calculate a per-task complete-input ceiling that fits the protected prompts, the frozen summary allowance, and the five largest whole units, plus a 64-token reserve. + This size-only ceiling permits the entire N sweep without changing S or truncating evidence; actual lengths are reported rather than forced equal. +- Report Full-to-fusion input reduction. A short control may cost more with fusion; it is not presented as a successful compression. +- Save protocol, source hashes, code snapshots, requests/responses, nested selections and per-field grades outside the repository. No archive recovery. +- The original payment-only table remains a development result; this new task panel is not an independent held-out confirmation because the payment task was already used. + +Fixture source: local Hermes commit `30463589eb54fa0f48724594cc0e98e6b995729b`, MIT licensed. Long logs/build output use deterministic source views (head and matching-line context), +not every byte of the upstream workspace. No artificial noise or identifier replacements are added. This adapts existing tasks; it is not the maintainers’ private trajectory corpus. + +## Format validation before the final comparison + +The first answer pass exposed two under-specified follow-up formats: payment test counts were returned as arrays, +and meeting-note bullets retained their list markers. Expected answers and the strict grader remain unchanged. +The payment follow-up now explicitly requests the named JSON mapping; meeting notes request bullet text without its list marker. +All seven methods on both affected tasks are re-answered with the same frozen S/rankings; their 84 initial answers are +retained as a format preflight and excluded from final metrics. The other twelve tasks retain all 504 original answers, +with identical questions. This correction changes no cohort, evidence, answer values, ranking or summary. + +## Added baseline: Hermes small-window summary adapter + +This is a **simulation adapter**, not Hermes' unchanged automatic policy. It calls the real +`agent.context_compressor.ContextCompressor._generate_summary` with native serialization, checkpoint template, +redaction and provenance validation. Use the built-in `tail_mode=legacy` to omit lean's additional 4,000-token session +log and deterministic appendices. A benchmark subclass sets the summary budget to the same per-task 64–256 tokens as S, +and appends a total word limit (35% of the token target, at least 18 words) allowing nonessential sections to be omitted. +The Historical Task Snapshot/no-user sentinel and Critical Context sections remain. Native validation is retained. + +Strip the native control prefix with Hermes' own helper before using the same downstream `Summary:` carrier as S. +This compares generated summary bodies without charging Hermes twice for handoff framing. No model response is edited +for factual content. The native call sees only the pre-final evidence history and the original task as its focus; +no exam or gold. Allow at most three length-only retries, reducing the requested target by 0.8, without answer feedback. + +Three summaries × two answers × fourteen tasks add **42 contexts and 84 planned answers**: **336 contexts and 672 planned +answers** across eight methods. Keep the same original system/task and exact scoring follow-up as other methods. +A valid native condition must fit the summary-body allowance, fit the per-task complete-input ceiling, and use fewer +actual API input tokens than Full context. Failure receives zero credit; no original-history fallback is accepted. + +Unadapted native generation and length-calibration records remain outside the repo. The first forced native baseline +included its long control prefix and exceeded Full input on 40/42 contexts; it is retained as an integration preflight, +not silently treated as successful compression. The explicit body carrier and word-budget adapter above are fixed +before their new scored answers. No expected answer, cohort, Jev ranking or existing S is changed. diff --git a/benchmark/compaction/cases.py b/benchmark/compaction/cases.py index 94a70a06..4d72e563 100644 --- a/benchmark/compaction/cases.py +++ b/benchmark/compaction/cases.py @@ -1,11 +1,8 @@ """The original payment fixture, lossless source grouping and deterministic checks.""" -import hashlib import json import re -from benchmark.compaction.repair_check import repair_checks - SOURCE_SHA256 = "0f37e6510ffafd920d088452d8719026b00fd4e74f87f7dd1a42a85c8287e6f1" GOLD = { @@ -20,25 +17,6 @@ GOLD = { "key_generation_line": 5, "regression_attempt_counts": {"timeout_then_success": 2, "validation_422": 1, "three_timeouts": 3}, } -EXAM = """Using only the recorded evidence above, answer this follow-up in JSON. Do not read files or use tools. -An absent fact must be null; do not invent identifiers. The conversation is a completed read-only investigation. -Return {"facts": {...}, "repair_plan": {...}}. -The facts keys are: -incident_id: the incident reference; -invoice_id: the affected invoice; -successful_charge_id: the returned charge ID; -amount_minor: the recorded integer amount in minor units; -attempt_keys: the two recorded idempotency keys in attempt order; -first_attempt_committed: whether the first, timed-out attempt had committed; -max_attempts: the policy limit, including the initial attempt; -no_retry_http_statuses: the policy's non-retryable HTTP codes; -key_generation_line: the original retry.py line generating the key; -regression_attempt_counts: a mapping from each named fixture test case to its expected attempt count. -For repair_plan return: -key_scope: choose before_retry_loop or inside_retry_loop; -retry_exceptions: list the exception names that should be retried; -exhaustion_action: choose raise_last_timeout, return_none, or retry_forever. -These plan fields describe a proposed fix; do not execute or modify anything.""" SUMMARY = ( "Compress the supplied recorded investigation for an agent continuing the same task.\n" "Return a concise structured handoff: Goal, Constraints, Findings, Decisions, Relevant file" @@ -68,36 +46,6 @@ def exact_match(observed, expected): return observed == expected -FOLLOW_UP = ( - EXAM.replace( - 'Return {"facts": {...}, "repair_plan": {...}}.', - 'Return {"facts": {...}, "repair_plan": {...}, "repair_code": "Python source"}.', - ) - + """ -Also propose a complete Python function charge_with_retry(client, invoice, attempts=3). -Return source in repair_code, with no markdown fences. Use uuid.uuid4() to create idempotency keys; -only import uuid if needed. The harness supplies uuid, range, str and TimeoutError. -Use a bounded for loop with range(attempts), client.charge(invoice, idempotency_key=key), -and ordinary try/except. No other imports, decorators, annotations, helpers or APIs. -The function must satisfy the recorded payment contract. Propose code only; do not run tools.""" -) - - -def load_case(path): - """Accept only the recorded task whose independent fixture oracle is defined above.""" - original = path.read_bytes() - if hashlib.sha256(original).hexdigest() != SOURCE_SHA256: - raise ValueError("Expected the unchanged original payment trajectory (see RESULTS.md).") - history = json.loads(original)["messages"] - if history[-1]["role"] != "assistant" or history[-1].get("tool_calls"): - raise ValueError("Expected a final assistant report to exclude.") - allowed = {"role", "content", "tool_calls", "tool_call_id", "name"} - history = [{k: v for k, v in m.items() if k in allowed} for m in history[:-1]] - if [m["role"] for m in history[:2]] != ["system", "user"]: - raise ValueError("Expected the actual system prompt and initial task.") - return {"messages": history, "exam": FOLLOW_UP, "gold": GOLD} - - def add_group(groups, spans): """Add one ordinary source block to this observation's event groups.""" if spans: @@ -114,7 +62,9 @@ def event_units(messages): call = calls[message["tool_call_id"]] args = json.loads(call["function"]["arguments"]) path = args.get("path") or args.get("file_path") or str(args) - text = json.loads(message["content"])["content"] + payload = json.loads(message["content"]) + text = payload.get("content") if isinstance(payload, dict) else None + text = text if isinstance(text, str) else message["content"] groups, offset, routine, routine_chars = {}, 0, [], 0 if len(text) <= 1200: @@ -151,17 +101,3 @@ def render(units): f"[{u['id']} {u['file']} chars " + ",".join(f"{s['start']}:{s['end']}" for s in u["spans"]) + "]\n" + u["text"] for u in units ) - - -def evaluate(case, answer): - """Average ten exact, type-strict facts separately from six executable repair checks.""" - answer = answer if isinstance(answer, dict) else {} - observed = answer.get("facts", {}) - observed = observed if isinstance(observed, dict) else {} - facts = {} - for name, expected in case["gold"].items(): - value = observed.get(name) - if name == "no_retry_http_statuses" and isinstance(value, list): - value, expected = sorted(value, key=str), sorted(expected, key=str) - facts[name] = exact_match(value, expected) - return {"facts": facts, "task": repair_checks(answer.get("repair_code", ""))} diff --git a/benchmark/compaction/hermes_summary.py b/benchmark/compaction/hermes_summary.py new file mode 100644 index 00000000..3c50e709 --- /dev/null +++ b/benchmark/compaction/hermes_summary.py @@ -0,0 +1,179 @@ +"""Run the real Hermes summary generator with a small-window adapter and pinned Qwen transport.""" + +# Private generator hooks are intentionally pinned to the Hermes revision recorded in the corpus. +# pylint: disable=protected-access + +import argparse +from copy import deepcopy +import hashlib +import json +import os +from pathlib import Path +import sys + +from dotenv import dotenv_values +from openai import OpenAI + +from benchmark.compaction.cases import write + + +def run(args): + """Isolate state, apply host budgets to the native template and retain wire bodies.""" + root = args.output.resolve() + root.mkdir(parents=True) + home = root / "hermes_home" + home.mkdir() + os.environ["HERMES_HOME"] = str(home) + (home / "config.yaml").write_text("model:\n provider: custom\n context_length: 131072\n") + values = dotenv_values(args.env_file) + sys.path.insert(0, str(args.hermes.resolve())) + import agent.context_compressor as cc + + source = json.loads(args.source.read_text()) + history = source["messages"] + # The source is the already validated public history, with the final report excluded. + from tokenizers import Tokenizer + + tokenizer = Tokenizer.from_file(str(args.tokenizer)) + allowance = json.loads((args.source.parent / "budget.json").read_text())["allowance"] + records = [] + index = 0 + attempt = 0 + + class ForcedSummary(cc.ContextCompressor): + """Model the small-window budget without altering the installed Hermes checkout.""" + + target = allowance + + def _compute_summary_budget(self, turns_to_summarize): # pylint: disable=unused-argument + return self.target + + def _build_summary_prompt( + self, + content_to_summarize, + summary_budget, + focus_topic, + memory_context, + has_user_turn, + ): + prompt = super()._build_summary_prompt( + content_to_summarize, + summary_budget, + focus_topic, + memory_context, + has_user_turn, + ) + words = max(18, int(self.target * 0.35)) + return prompt + ( + f"\n\nSMALL-WINDOW SIMULATION: Write at most {words} English words TOTAL. " + "This total budget takes precedence over the detailed template above. " + "Keep the Historical Task Snapshot section and its required no-user sentinel. " + "Keep a Critical Context section with the most useful exact facts. " + "Omit every other section, empty entry, repeated task and prose when necessary. " + "No preamble. Do not include the context-compaction control prefix." + ) + + def transport(**kwargs): + request = { + "model": args.model, + "messages": kwargs["messages"], + "temperature": 0, + "reasoning_effort": "none", + "extra_body": {"enable_thinking": False}, + } + request_file = root / f"c{index}.a{attempt}.summary.request.json" + response_file = root / f"c{index}.a{attempt}.summary.response.json" + write(request_file, request) + response = client.chat.completions.create(**request) + write(response_file, response.model_dump(mode="json")) + kwargs.get("route_info", {}).update(provider="custom", model=args.model) + records.append( + { + "generation": index, + "usage": response.usage.model_dump(), + "finish_reason": response.choices[0].finish_reason, + }, + ) + return response + + client = OpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1) + original = cc.call_llm + cc.call_llm = transport + summaries = [] + try: + for index in range(args.compressions): + accepted = None + for attempt in range(3): + compressor = ForcedSummary( + model=args.model, + base_url=values["LLM_BASE_URL"], + api_key=values["LLM_API_KEY"], + provider="custom", + api_mode="chat_completions", + config_context_length=131072, + quiet_mode=True, + tail_mode="legacy", + ) + compressor.target = max(12, int(allowance * 0.8**attempt)) + summary = compressor._generate_summary( + deepcopy(history[2:]), + focus_topic=history[1]["content"], + ) # pylint: disable=protected-access + body = compressor._strip_summary_prefix(summary) if summary else "" # pylint: disable=protected-access + if body and len(tokenizer.encode(body, add_special_tokens=False).ids) <= allowance: + accepted = body + break + summaries.append( + { + "repeat": index, + "summary": accepted, + "native_budget": allowance, + "summary_tokens": len(tokenizer.encode(accepted or "", add_special_tokens=False).ids), + "valid": accepted is not None, + "error": None if accepted else "summary_budget_exceeded", + }, + ) + write(root / "summaries.json", summaries) + print( + args.source.parent.name, + "native small-window summary", + index, + "budget", + allowance, + "OK" if accepted else "FAILED", + flush=True, + ) + write(root / "usage.json", records) + for file in root.rglob("*"): + if file.is_file(): + assert values["LLM_API_KEY"].encode() not in file.read_bytes() + write( + root / "validation.json", + { + "no_api_keys": True, + "source_sha256": hashlib.sha256(args.source.read_bytes()).hexdigest(), + "generator": ( + "Hermes legacy generator with explicit small-window prompt/budget adapter; body-only carrier" + ), + }, + ) + finally: + cc.call_llm = original + client.close() + + +def main(): + """Internal subprocess entry point; the orchestrator owns the live-call opt-in.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--hermes", type=Path, required=True) + parser.add_argument("--env-file", type=Path, required=True) + parser.add_argument("--tokenizer", type=Path, required=True) + parser.add_argument("--compressions", type=int, default=3) + parser.add_argument("--model", default="qwen3.8-max") + run(parser.parse_args()) + + +if __name__ == "__main__": + main() diff --git a/benchmark/compaction/hermes_worker.py b/benchmark/compaction/hermes_worker.py new file mode 100644 index 00000000..66c8a5fa --- /dev/null +++ b/benchmark/compaction/hermes_worker.py @@ -0,0 +1,197 @@ +"""Collect one genuine Hermes trajectory in an isolated process and read-only fixture workspace.""" + +import argparse +import hashlib +import importlib +import json +import os +from pathlib import Path +import re +import sys +import time + +from dotenv import dotenv_values + +from benchmark.compaction.cases import write + + +def collect(root, hermes, env_file): + """Configure isolated Hermes state before importing its runtime; never save credentials.""" + values = dotenv_values(env_file) + secrets = [values[k] for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY") if values.get(k)] + task = json.loads((root / "collection-task.json").read_text()) + work, home = root / "workspace", root / "hermes_home" + home.mkdir() + os.environ.update( + { + "HERMES_HOME": str(home), + "TERMINAL_ENV": "local", + "TERMINAL_CWD": str(work), + "TOKENIZERS_PARALLELISM": "false", + }, + ) + (home / "config.yaml").write_text( + "compression:\n enabled: false\n micro_compact_enabled: false\n" + f"terminal:\n backend: local\n cwd: {work}\n" + "model:\n provider: custom\n context_length: 131072\n", + ) + sys.path.insert(0, str(hermes)) + from run_agent import AIAgent + from hermes_state import SessionDB + + requests = [] + + def append(name, data): + text = json.dumps(data, ensure_ascii=False) + if any(secret in text for secret in secrets): + raise ValueError("Credential in captured body") + with (root / name).open("a", encoding="utf-8") as target: + target.write(text + "\n") + + seen = set() + for name in ("httpx", "httpx2"): + try: + module = importlib.import_module(name) + except ImportError: + continue + if id(module.Client) in seen: + continue + seen.add(id(module.Client)) + original = module.Client.send + + def wrap_send(send): + def capture(client, request, *args, **kwargs): + captured = "/chat/completions" in request.url.path + if captured: + body = json.loads(request.content) + requests.append(body) + append("api_requests.jsonl", body) + response = send(client, request, *args, **kwargs) + if captured: + response.read() + try: + body = response.json() + except ValueError: + body = response.text + append("api_responses.jsonl", {"status": response.status_code, "body": body}) + return response + + return capture + + module.Client.send = wrap_send(original) + agent = None + database = SessionDB(home / "state.db") + started = time.monotonic() + try: + agent = AIAgent( + base_url=values["LLM_BASE_URL"], + api_key=values["LLM_API_KEY"], + provider="custom", + api_mode="chat_completions", + model=task["model"], + max_iterations=12, + run_budget_seconds=240, + max_tokens=1024, + request_overrides={"reasoning_effort": "none", "extra_body": {"enable_thinking": False}}, + enabled_toolsets=["file"], + save_trajectories=False, + quiet_mode=True, + skip_context_files=True, + skip_memory=True, + skip_background_review=True, + session_db=database, + cwd=str(work), + platform="batch", + ) + from tools.registry import registry + + entry = registry.get_entry("read_file") + original_read = entry.handler + + def confined_read(args, **kwargs): + path = Path(args.get("path", "")) + resolved = (path if path.is_absolute() else work / path).resolve() + if not resolved.is_relative_to(work) or str(resolved.relative_to(work)) not in task["source_hashes"]: + return json.dumps({"error": "Only the listed source snapshots may be read"}) + return original_read(args, **kwargs) + + entry.handler = confined_read + agent.tools = [t for t in agent.tools if t["function"]["name"] == "read_file"] + agent.valid_tool_names = {"read_file"} + agent.context_compressor.enabled = False + result = agent.run_conversation( + task["prompt"], + task_id=root.name, + system_message=f"Investigate the supplied fixture evidence and propose actions only. " + f"Use only read_file within {work}. " + "Do not read environment files, credentials or any path outside this workspace.", + ) + if not result.get("completed") or not requests: + raise ValueError("Hermes collection did not complete") + systems = [m for m in requests[0]["messages"] if m["role"] == "system"] + history = [*systems, *result["messages"]] + if len(systems) != 1 or history[-1]["role"] != "assistant" or history[-1].get("tool_calls"): + raise ValueError("Incomplete system prompt or final assistant report") + calls = {c["id"]: c for m in history for c in m.get("tool_calls", []) or []} + results = [m for m in history if m["role"] == "tool"] + if set(calls) != {m["tool_call_id"] for m in results}: + raise ValueError("Unpaired tool calls") + observed = set() + for message in results: + call = calls[message["tool_call_id"]] + args = json.loads(call["function"]["arguments"]) + path = Path(args.get("path") or args.get("file_path")) + path = (path if path.is_absolute() else work / path).resolve() + output = json.loads(message["content"]) + if not path.is_relative_to(work) or str(path.relative_to(work)) not in task["source_hashes"]: + if "error" not in output: + raise ValueError("Unexpected content from outside the source allowlist") + continue # Denied reads remain in the trajectory; they never reached the filesystem. + text = "\n".join(re.sub(r"^\d+\|", "", line) for line in output.get("content", "").splitlines()) + if text.rstrip() == path.read_text(encoding="utf-8").rstrip(): + observed.add(str(path.relative_to(work))) + if observed != set(task["source_hashes"]): + raise ValueError("Not every source snapshot was read without truncation") + if any( + hashlib.sha256((work / name).read_bytes()).hexdigest() != sha for name, sha in task["source_hashes"].items() + ): + raise ValueError("Source fixture changed") + write(root / "trajectory.full.json", {"messages": history}) + write(root / "result.json", result) + write(root / "session_export.json", database.export_session(agent.session_id, include_compacted=True)) + write( + root / "collection.json", + { + "completed": True, + "session_id": agent.session_id, + "messages": len(history), + "tool_calls": len(calls), + "articles_read": len(observed), + "source_unchanged": True, + "seconds": time.monotonic() - started, + "trajectory_sha256": hashlib.sha256((root / "trajectory.full.json").read_bytes()).hexdigest(), + }, + ) + if any( + secret.encode() in file.read_bytes() for file in root.rglob("*") if file.is_file() for secret in secrets + ): + raise ValueError("Credential in artifacts") + print(root.name, "collected", len(calls), "tool calls", flush=True) + finally: + if agent is not None: + agent.close() + database.close() + + +def main(): + """Internal subprocess entry point; provider authorization belongs to the parent collector.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--hermes", type=Path, required=True) + parser.add_argument("--env-file", type=Path, required=True) + args = parser.parse_args() + collect(args.root.resolve(), args.hermes.resolve(), args.env_file.resolve()) + + +if __name__ == "__main__": + main() diff --git a/benchmark/compaction/repair_check.py b/benchmark/compaction/repair_check.py deleted file mode 100644 index c4ed36a7..00000000 --- a/benchmark/compaction/repair_check.py +++ /dev/null @@ -1,139 +0,0 @@ -"""Run generated payment fixes with a deliberately restricted Python grammar.""" - -import ast -from types import SimpleNamespace - - -class PaymentError(Exception): - """A non-timeout provider error carrying an HTTP status.""" - - status_code = 422 - - -def repair_checks(source): # pylint: disable=too-many-return-statements - """Reject general Python programs; allow only a bounded retry function and inert clients.""" - names = ("success", "timeout_then_success", "http_402", "http_422", "exhaustion", "distinct_payments") - failed = dict.fromkeys(names, False) - try: - tree = ast.parse(source) - functions = [node for node in tree.body if isinstance(node, ast.FunctionDef)] - if len(functions) != 1 or functions[0].name != "charge_with_retry": - return failed - for node in tree.body: - if node is functions[0]: - continue - if not isinstance(node, ast.Import) or any(alias.name != "uuid" for alias in node.names): - return failed - allowed_calls = {"range", "str", "TimeoutError", "uuid.uuid4", "client.charge"} - allowed_nodes = ( - ast.FunctionDef, - ast.arguments, - ast.arg, - ast.Constant, - ast.Name, - ast.Load, - ast.Store, - ast.Assign, - ast.For, - ast.Call, - ast.Attribute, - ast.keyword, - ast.Try, - ast.ExceptHandler, - ast.Return, - ast.Raise, - ast.If, - ast.Compare, - ast.Eq, - ast.NotEq, - ast.Lt, - ast.LtE, - ast.Gt, - ast.GtE, - ast.BinOp, - ast.Sub, - ast.Pass, - ast.Expr, - ) - for node in ast.walk(functions[0]): - if not isinstance(node, allowed_nodes): - return failed - if isinstance(node, ast.FunctionDef) and node is not functions[0]: - return failed - if isinstance(node, ast.Constant) and ( - isinstance(node.value, str) - and len(node.value) > 100 - or isinstance(node.value, int) - and abs(node.value) > 10 - ): - return failed - if isinstance(node, ast.Attribute) and ast.unparse(node) not in {"uuid.uuid4", "client.charge"}: - return failed - if isinstance(node, ast.Call) and ast.unparse(node.func) not in allowed_calls: - return failed - if isinstance(node, ast.Name) and node.id.startswith("__"): - return failed - if functions[0].decorator_list or functions[0].returns or any(a.annotation for a in functions[0].args.args): - return failed - # No executable defaults, annotations or decorators in the accepted grammar. - if any(not isinstance(node, ast.Constant) for node in functions[0].args.defaults): - return failed - if functions[0].args.kwonlyargs or functions[0].args.vararg or functions[0].args.kwarg: - return failed - if any(arg.annotation for arg in ast.walk(functions[0]) if isinstance(arg, ast.arg)): - return failed - if sum(isinstance(node, ast.For) for node in ast.walk(functions[0])) > 1: - return failed - serial = iter(range(1000)) - - def bounded_range(*args): - result = range(*args) - if len(result) > 10: - raise ValueError("Retry range exceeds test limit") - return result - - namespace = { - "__builtins__": {}, - "range": bounded_range, - "str": str, - "TimeoutError": TimeoutError, - "uuid": SimpleNamespace(uuid4=lambda: f"test-key-{next(serial)}"), - } - module = ast.Module(body=functions, type_ignores=[]) - exec(compile(module, "", "exec"), namespace) # pylint: disable=exec-used - fix = namespace["charge_with_retry"] - - def attempt(events, expected_calls, error=None): - keys = [] - remaining = iter(events) - - def charge(invoice, idempotency_key): - if invoice != "test-invoice": - raise AssertionError("Invoice was changed") - keys.append(idempotency_key) - event = next(remaining) - if isinstance(event, Exception): - raise event - return event - - try: - result = fix(SimpleNamespace(charge=charge), "test-invoice") - correct = error is None and result == "ok" - except Exception as exc: - correct = error is not None and exc is error - return correct and len(keys) == expected_calls and len(set(keys)) == 1, keys - - result = {} - result["success"], first = attempt(["ok"], 1) - result["timeout_then_success"], _ = attempt([TimeoutError(), "ok"], 2) - for status in (402, 422): - error = PaymentError() - error.status_code = status - result[f"http_{status}"], _ = attempt([error, "ok"], 1, error) - errors = [TimeoutError() for _ in range(3)] - result["exhaustion"], _ = attempt(errors, 3, errors[-1]) - success, second = attempt(["ok"], 1) - result["distinct_payments"] = success and result["success"] and first != second - return result - except Exception: - return failed diff --git a/benchmark/compaction/run.py b/benchmark/compaction/run.py index c0d15c3f..cd88fcbb 100644 --- a/benchmark/compaction/run.py +++ b/benchmark/compaction/run.py @@ -1,17 +1,13 @@ -"""Frozen-summary experiment: Full context, S, and S + Jev Top 1–N.""" +"""Shared request budgeting, model calls and strict Top-N selection for the study.""" -import argparse import asyncio from dataclasses import asdict, dataclass, is_dataclass -import hashlib import json from pathlib import Path -import statistics -from dotenv import dotenv_values from openai import AsyncOpenAI -from benchmark.compaction.cases import SUMMARY, SOURCE_SHA256, evaluate, event_units, load_case, render, write +from benchmark.compaction.cases import SUMMARY, write from reme.components.as_decision_model import AgentScopeAsDecisionModel from reme.steps.compaction.compact_context import probability @@ -151,264 +147,3 @@ def top_n(units, scores, n): raise ValueError("Scores must be finite probabilities") chosen = {u["id"] for u in sorted(units, key=lambda u: (-scores[u["id"]], u["id"]))[:n]} return [u for u in units if u["id"] in chosen] - - -def prepare(experiment, case, units, summaries, scores, maximum): - """Freeze all inputs before answers; fusion never rewrites or shrinks its summary.""" - rows = [] - for index, summary in enumerate(summaries): - for arm, n in [("full", 0), ("summary", 0), *[(f"fusion_top{n}", n) for n in range(1, maximum + 1)]]: - chosen = top_n(units, scores[index], n) - text = "" if arm == "full" else summary - payload = None if arm == "full" else compose(text, render(chosen)) - context = messages(case, payload) - rows.append( - { - "arm": arm, - "repeat": index, - "summary": text, - "selected": [u["id"] for u in chosen], - "estimated_input_tokens": experiment.input_tokens(context), - "within_budget": arm == "full" or experiment.fits(case, payload), - "valid": False, - "trials": [], - }, - ) - write(experiment.output / f"c{index}.{arm}.context.json", {"messages": context}) - return rows - - -def aggregate(rows): - """Keep every planned condition in the denominator, including failed generations.""" - result = [] - for arm in dict.fromkeys(r["arm"] for r in rows): - group = [r for r in rows if r["arm"] == arm] - valid = [r for r in group if r["valid"]] - trials = [t for r in valid for t in r["trials"]] - metrics = {} - for section in ("facts", "task"): - metrics[section] = statistics.mean( - ( - statistics.mean(sum(t["checks"][section].values()) / len(t["checks"][section]) for t in r["trials"]) - if r["valid"] and r["trials"] - else 0 - ) - for r in group - ) - result.append( - { - "arm": arm, - "valid": len(valid), - "planned": len(group), - "input_tokens": statistics.mean(t["usage"]["prompt_tokens"] for t in trials) if trials else None, - **metrics, - }, - ) - return result - - -def report(output, rows): - """Write one compact table; retain detailed provenance outside the repository.""" - table = aggregate(rows) - write(output / "aggregate.json", table) - lines = ["| Method | Input tokens | Fact accuracy | Repair pass |", "| --- | ---: | ---: | ---: |"] - for row in table: - label = {"full": "Full context", "summary": "S"}.get( - row["arm"], - row["arm"].replace("fusion_top", "S + Jev Top "), - ) - tokens = f"{row['input_tokens']:.0f}" if row["input_tokens"] is not None else "—" - lines.append(f"| {label} | {tokens} | {row['facts']:.1%} | {row['task']:.1%} |") - lines += ["", "Valid/planned: " + "; ".join(f"{r['arm']} {r['valid']}/{r['planned']}" for r in table)] - (output / "REPORT.md").write_text("\n".join(lines) + "\n", encoding="utf-8") - - -def frozen_inputs(path, case, units, count): - """Load every previously frozen summary/ranking pair, never choose by answer accuracy.""" - source = json.loads((path / "source.json").read_text()) - if any(source[k] != case[k] for k in ("messages", "exam", "gold")) or source["units"] != units: - raise ValueError("Frozen evidence does not match the original trajectory") - rows = json.loads((path / "results.json").read_text()) - baseline = [r for r in rows if r["arm"] == "summary"] - if len(baseline) != count or {r["repeat"] for r in baseline} != set(range(count)): - raise ValueError("Use all frozen summary generations with the matching --compressions count") - summaries = [next(r["summary"] for r in baseline if r["repeat"] == i) for i in range(count)] - scores = [ - json.loads((path / "provenance" / f"paired.c{i}.decision0.response.json").read_text())["scores"] - for i in range(count) - ] - if any(not isinstance(s, str) or not s for s in summaries): - raise ValueError("Every frozen summary must be nonempty text") - return summaries, scores - - -async def answer(experiment, row, case, repeats): - """Only complete answers within the actual downstream input ceiling receive credit.""" - label = f"c{row['repeat']}.{row['arm']}" - try: - if not row["within_budget"]: - raise ValueError("Frozen context exceeds input ceiling; increase the ceiling without truncating evidence") - context = json.loads((experiment.output / f"{label}.context.json").read_text())["messages"] - trials = [] - for index in range(repeats): - text, usage = await experiment.ask(f"{label}.answer{index}", context) - if row["arm"] != "full" and usage["prompt_tokens"] > experiment.cap: - raise ValueError("Actual API input exceeds ceiling") - parsed = json.loads(text) - trials.append({"answer": parsed, "usage": usage, "checks": evaluate(case, parsed)}) - row.update(valid=True, trials=trials) - except Exception as exc: - row.update(valid=False, error=type(exc).__name__, trials=[]) - print(label, "valid" if row["valid"] else row["error"], flush=True) - - -async def run(args): - """Fresh runs create one S/ranking per generation; replay keeps them byte-identical.""" - from tokenizers import Tokenizer - - repo = Path(__file__).resolve().parents[2] - output = args.output.resolve() - if output.exists() or output == repo or repo in output.parents: - raise ValueError("Use a new output directory outside the repository") - case = load_case(args.trajectory) - units = event_units(case["messages"]) - if args.top_n > len(units): - raise ValueError(f"Only {len(units)} evidence units: maximum Top {len(units)}") - if args.summary_cap > args.input_cap: - raise ValueError("Summary generation cap cannot exceed the final input cap") - frozen = frozen_inputs(args.frozen_run, case, units, args.compressions) if args.frozen_run else None - values = dotenv_values(repo / ".env") if not args.check_only else {} - if not args.check_only and any(not values.get(k) for k in ("LLM_API_KEY", "LLM_BASE_URL")): - raise ValueError("Missing authorized LLM configuration") - if not frozen and not values.get("DECISION_MODEL_API_KEY"): - raise ValueError("Fresh compression requires DECISION_MODEL_API_KEY") - output.mkdir(parents=True) - write(output / "source.json", {**case, "units": units}) - for name in ("run.py", "cases.py", "repair_check.py"): - (output / name).write_bytes((Path(__file__).parent / name).read_bytes()) - write( - output / "protocol.json", - { - "source_sha256": SOURCE_SHA256, - "tokenizer_sha256": hashlib.sha256(args.tokenizer.read_bytes()).hexdigest(), - "model": args.model, - "summary_generation_input_cap": args.summary_cap, - "compressed_input_cap": args.input_cap, - "top_n": args.top_n, - "evidence_count": len(units), - "compressions": args.compressions, - "answer_repeats": args.repeats, - "summary_frozen": True, - "selection": "Strict score prefix; no threshold; restore source order", - "frozen_run": str(args.frozen_run.resolve()) if args.frozen_run else None, - "check_only": args.check_only, - "evaluation": "Direct Qwen; not Hermes continuation", - }, - ) - client = None - try: - if not args.check_only: - client = AsyncOpenAI( - api_key=values["LLM_API_KEY"], - base_url=values["LLM_BASE_URL"], - timeout=90, - max_retries=1, - ) - experiment = Experiment( - client, - None, - output, - Tokenizer.from_file(str(args.tokenizer)), - asyncio.Semaphore(args.concurrency), - args.model, - args.input_cap, - ) - if frozen: - summaries, scores = frozen - else: - credential = { - "api_key": values["DECISION_MODEL_API_KEY"], - "base_url": values.get("DECISION_MODEL_BASE_URL") or None, - } - model = values.get("DECISION_MODEL_NAME") - model = None if model in (None, "", "null") else model - summaries, scores = [], [] - async with AgentScopeAsDecisionModel(credential=credential, model=model, timeout=45) as decision: - experiment.decision = decision - allowance = args.summary_cap - RESERVE - experiment.input_tokens(messages(case, compose("", ""))) - 8 - if allowance < 40: - raise ValueError("Insufficient summary capacity") - for index in range(args.compressions): - label = f"paired.c{index}" - ranking, _ = await experiment.scores(label, case["messages"][1]["content"], units) - summary, _ = await experiment.summarize(label, case["messages"][1]["content"], units, allowance) - scores.append(ranking) - summaries.append(summary) - # Persist compressor outputs so a later replay can use them without new provider calls. - (output / "provenance").mkdir() - for index, ranking in enumerate(scores): - write(output / "provenance" / f"paired.c{index}.decision0.response.json", {"scores": ranking}) - rows = prepare(experiment, case, units, summaries, scores, args.top_n) - write(output / "frozen-selections.json", rows) - write(output / "results.json", rows) - if not args.check_only: - work = asyncio.Semaphore(args.concurrency) - - async def condition(row): - async with work: - await answer(experiment, row, case, args.repeats) - write(output / "results.json", rows) - report(output, rows) - - async with asyncio.TaskGroup() as tasks: - for row in rows: - tasks.create_task(condition(row)) - load_case(args.trajectory) - secrets = [values[k] for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY") if values.get(k)] - if any( - secret.encode() in file.read_bytes() for file in output.rglob("*") if file.is_file() for secret in secrets - ): - raise RuntimeError("Credential in artifacts") - write( - output / "validation.json", - { - "source_unchanged": True, - "no_api_keys": True, - "prepared": len(rows), - "within_budget": sum(r["within_budget"] for r in rows), - "valid": sum(r["valid"] for r in rows), - "check_only": args.check_only, - }, - ) - finally: - if client is not None: - await client.close() - - -def main(): - """Require live opt-in or an offline frozen-input check; never overwrite existing runs.""" - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--trajectory", type=Path, required=True) - parser.add_argument("--tokenizer", type=Path, required=True) - parser.add_argument("--output", type=Path, required=True) - parser.add_argument("--frozen-run", type=Path) - parser.add_argument("--summary-cap", type=int, default=3200) - parser.add_argument("--input-cap", type=int, default=3800) - parser.add_argument("--top-n", type=int, default=5) - parser.add_argument("--compressions", type=int, default=5) - parser.add_argument("--repeats", type=int, default=3) - parser.add_argument("--concurrency", type=int, default=4) - parser.add_argument("--model", default="qwen3.8-max") - mode = parser.add_mutually_exclusive_group(required=True) - mode.add_argument("--live", action="store_true") - mode.add_argument("--check-only", action="store_true") - args = parser.parse_args() - if min(args.summary_cap, args.input_cap, args.top_n, args.compressions, args.repeats, args.concurrency) < 1: - parser.error("All numerical settings must be positive") - if args.check_only and not args.frozen_run: - parser.error("--check-only requires --frozen-run") - asyncio.run(run(args)) - - -if __name__ == "__main__": - main() diff --git a/benchmark/compaction/suite.py b/benchmark/compaction/suite.py new file mode 100644 index 00000000..086788ba --- /dev/null +++ b/benchmark/compaction/suite.py @@ -0,0 +1,455 @@ +"""Freeze rule-scored, read-only adaptations of fourteen distinct fixture tasks.""" + +import ast +import hashlib +import importlib.util +import json +import re +import random +import subprocess +import sys + +from benchmark.compaction.cases import GOLD, exact_match, write + + +def module(path, name): + """Load fixture definitions only; never run their CLI or agent workers.""" + spec = importlib.util.spec_from_file_location(name, path) + value = importlib.util.module_from_spec(spec) + sys.modules[name] = value + spec.loader.exec_module(value) + return value + + +def context(text, pattern, radius=10): + """Freeze original matching lines with context, with no generated filler or answer lookup.""" + lines = text.splitlines(keepends=True) + indices = { + j + for i, line in enumerate(lines) + if re.search(pattern, line) + for j in range(max(0, i - radius), min(len(lines), i + radius + 1)) + } + return "".join(f"{i+1}|{lines[i]}" for i in sorted(indices)) + + +def planted_session(path, function, session_id, position): + """Extract a literal upstream seeded history without running DB setup or worker code.""" + tree = ast.parse(path.read_text()) + for node in ast.walk(tree): + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == function: + if any(isinstance(a, ast.Constant) and a.value == session_id for a in node.args): + return ast.literal_eval(node.args[position]) + raise ValueError("Expected upstream seeded session") + + +def define(task_id, query, files, gold, fields, origin, *, query_only=False, unordered=()): + """Keep fixture answers outside the read-only tool workspace and compressor inputs.""" + return { + "id": task_id, + "query": query, + "files": files, + "gold": gold, + "fields": fields, + "origin": origin, + "query_only": query_only, + "unordered": list(unordered), + } + + +def definitions(hermes, scratch): + """Use pinned upstream fixture bytes; adapt actions to read-only investigation explicitly.""" + core = module(hermes / "evals/core_tool_deferral/tasks.py", "compaction_core_tasks") + perf = module(hermes / "evals/toolperf_abeval/ab_eval.py", "compaction_perf_tasks") + read = module(hermes / "evals/readtool/fixtures.py", "compaction_read_fixtures") + core_root, perf_root, read_root = (scratch / name for name in ("core", "perf", "read")) + for root in (core_root, perf_root, read_root): + root.mkdir(parents=True) + for name in ("longrange_backup_pipeline", "eager_refactor_control", "todo_three_files", "config_grep_distractor"): + root = core_root / name + root.mkdir() + core.TASKS_BY_ID[name]["fixtures"](str(root)) + perf.make_sandbox(perf_root) + read._write_package_json(read_root) # pylint: disable=protected-access + read._write_app_js(read_root) # pylint: disable=protected-access + read._write_unicode_notes(read_root) # pylint: disable=protected-access + read._write_server_log(read_root, random.Random(20260809)) # pylint: disable=protected-access + read._write_lockfile(read_root, random.Random(20260809)) # pylint: disable=protected-access + backup = planted_session(hermes / "evals/core_tool_deferral/worker.py", "sess", "seed_backup_talk", 1) + incident = planted_session(hermes / "evals/session_search_schema/fixtures.py", "mk", "20260818_140200_cc22dd", 4) + + def core_files(name): + root = core_root / name + return {str(p.relative_to(root)): p.read_text() for p in root.rglob("*") if p.is_file()} + + def core_query(name): + return core.TASKS_BY_ID[name]["prompt"] + + def perf_files(*names): + return {name: (perf_root / "proj" / name).read_text() for name in names} + + def perf_query(name): + return perf.TASKS[name].replace("{WORK}/proj", ".") + + rows = [] + rows.append( + define( + "payment", + "Investigate the original payment retry incident.", + {}, + GOLD, + {k: k.replace("_", " ") for k in GOLD}, + "Original Hermes payment fixture", + unordered=("no_retry_http_statuses",), + ), + ) + files = core_files("longrange_backup_pipeline") + files["session_history.json"] = json.dumps(backup, indent=2) + rows.append( + define( + "backup", + core_query("longrange_backup_pipeline"), + files, + {"repo": "sparks-data", "ignored_repo": "toybox", "data_text": "important"}, + { + "repo": "repository requiring backup", + "ignored_repo": "repository explicitly excluded", + "data_text": "contents of the target repository data file", + }, + "core_tool_deferral/longrange_backup_pipeline", + ), + ) + rows.append( + define( + "refactor", + core_query("eager_refactor_control"), + core_files("eager_refactor_control"), + { + "old_timeout": 30, + "call_sites": 3, + "functions": ["fetch_a", "fetch_b", "fetch_c"], + "methods": ["get", "post", "head"], + "old_version": "1.0", + }, + { + "old_timeout": "original timeout integer", + "call_sites": "number of requests call sites", + "functions": "function names in source order", + "methods": "HTTP methods in source order", + "old_version": "existing changelog version", + }, + "core_tool_deferral/eager_refactor_control", + ), + ) + rows.append( + define( + "three_files", + core_query("todo_three_files"), + core_files("todo_three_files"), + {"file_count": 3, "header": "item,qty", "data_rows": 2}, + {"file_count": "required output file count", "header": "CSV header", "data_rows": "CSV data row count"}, + "core_tool_deferral/todo_three_files", + query_only=True, + ), + ) + rows.append( + define( + "active_config", + core_query("config_grep_distractor"), + core_files("config_grep_distractor"), + {"retry_limit": 7, "backoff": 2, "active_file": "conf/app.ini", "old_file": "conf/old.ini"}, + { + "retry_limit": "active integer value", + "backoff": "active integer backoff", + "active_file": "workspace-relative active config", + "old_file": "workspace-relative retired config", + }, + "core_tool_deferral/config_grep_distractor", + ), + ) + rows.append( + define( + "checkout_504", + ( + "Investigate the checkout 504 incident: distinguish the failed rollback, " + "root cause and final database fix." + ), + {"session_history.json": json.dumps(incident, indent=2)}, + { + "setting": "statement_timeout", + "value_seconds": 45, + "index_mode": "CONCURRENTLY", + "root_cause": "CREATE INDEX without CONCURRENTLY", + "rolled_back_version": "v2.30.2", + "rollback_fixed": False, + "deployed_version": "v2.31.0", + "initial_error_percent": 8, + }, + { + "setting": "final database timeout setting name", + "value_seconds": "final timeout in seconds", + "index_mode": "mode used to recreate the index", + "root_cause": "SQL operation that competed with checkout writes", + "rolled_back_version": "rollback target version", + "rollback_fixed": "did rollback resolve the incident, boolean", + "deployed_version": "version deployed before the incident", + "initial_error_percent": "initial request error percentage as integer", + }, + "session_search_schema/t2_scroll", + ), + ) + rows.append( + define( + "dependency_version", + ( + "What exact left-pad version is pinned, and which file declares the test script? " + "Record the package scripts for a handoff." + ), + { + "package.json": (read_root / "package.json").read_text(), + "package-lock.head.txt": "".join( + (read_root / "package-lock.json").read_text().splitlines(keepends=True)[:40], + ), + }, + { + "version": "1.3.0", + "declaration": "package.json", + "test": "vitest run", + "build": "tsc -p .", + "package_version": "4.2.1", + }, + { + "version": "pinned left-pad version", + "declaration": "file declaring the test script", + "test": "exact test command", + "build": "exact build command", + "package_version": "project package version", + }, + "readtool/lockfile_version", + ), + ) + tree = ast.parse((hermes / "evals/readtool/fixtures.py").read_text()) + function = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "_write_minified_bundle") + parts = next( + n + for n in ast.walk(function) + if isinstance(n, ast.Assign) and any(isinstance(t, ast.Name) and t.id == "parts" for t in n.targets) + ) + fragment = ast.literal_eval(parts.value)[0] + rows.append( + define( + "retry_backoff", + "Find the retry-delay function in src/ and explain its base, growth, cap and jitter.", + {"src/app.js": (read_root / "src/app.js").read_text(), "src/app.min.head.txt": fragment}, + { + "base_ms": 250, + "cap_ms": 30000, + "growth_factor": 2, + "jitter": "uniform_integer_below_bound", + "function": "retryDelay", + }, + { + "base_ms": "base delay in milliseconds", + "cap_ms": "maximum delay in milliseconds", + "growth_factor": "exponential multiplier per attempt", + "jitter": "choose none or uniform_integer_below_bound", + "function": "function name", + }, + "readtool/minified_backoff", + ), + ) + log = (read_root / "logs/server.log").read_text() + rows.append( + define( + "server_error", + ( + "Locate the unique ERROR in the server log and retain its exact request, timestamp, " + "upstream and connection failure." + ), + { + "server.head.txt": "".join(log.splitlines(keepends=True)[:20]), + "server.error-context.txt": context(log, r"\bERROR\b"), + }, + { + "request_id": "req-7f3d9", + "timestamp": "2026-08-08T23:41:17Z", + "status": 502, + "upstream": "payments", + "error": "ECONNREFUSED", + "address": "10.0.4.17:8443", + }, + { + "request_id": "error request ID", + "timestamp": "full UTC error timestamp", + "status": "HTTP integer status", + "upstream": "failed upstream service", + "error": "connection error code", + "address": "upstream host and port", + }, + "readtool/log_error_hunt", + ), + ) + notes = next((read_root / "notes").iterdir()) + rows.append( + define( + "meeting_notes", + "Read the meeting notes and report the third bullet exactly, retaining the other agreed actions.", + {"notes.txt": notes.read_text()}, + { + "third": "rotate the API keys quarterly", + "first": "ship the payments retry fix", + "second": "audit the staging TLS certs", + "fourth": "close out the Q3 incident review", + }, + {k: f"exact {k} bullet" for k in ("third", "first", "second", "fourth")}, + "readtool/unicode_filename", + ), + ) + rows.append( + define( + "targeted_edit", + perf_query("err_ambiguous_edit"), + perf_files("handlers.py", "check_handlers.py"), + { + "target": "slow_handler", + "original_timeouts": {"fast_handler": 10, "slow_handler": 10, "medium_handler": 10}, + "required_timeouts": {"fast_handler": 10, "slow_handler": 60, "medium_handler": 10}, + "success_marker": "HANDLERS_OK_552", + }, + { + "target": "function requiring modification", + "original_timeouts": "mapping of all original function timeouts", + "required_timeouts": "mapping after the requested fix", + "success_marker": "checker success marker", + }, + "toolperf_abeval/err_ambiguous_edit", + ), + ) + output = subprocess.check_output([sys.executable, str(perf_root / "proj/noisy_build.py")], text=True) + rows.append( + define( + "build_token", + perf_query("err_big_output"), + { + **perf_files("noisy_build.py"), + "build.head.txt": "".join(output.splitlines(keepends=True)[:20]), + "build.token-context.txt": context(output, r"^UNIQUE_TOKEN="), + }, + {"token": "tok_9f31c_middle", "step": 2000, "total_steps": 4000, "prefix": "UNIQUE_TOKEN="}, + { + "token": "full unique token value without its prefix", + "step": "build loop step that prints the token", + "total_steps": "build loop iteration count", + "prefix": "exact token line prefix", + }, + "toolperf_abeval/err_big_output", + ), + ) + rows.append( + define( + "package_versions", + perf_query("err_multi_dir"), + perf_files("pkg_a/version.txt", "pkg_b/version.txt", "pkg_c/version.txt"), + {"versions": {"pkg_a": "1.4.2", "pkg_b": "0.9.7", "pkg_c": "3.2.1"}, "joined": "1.4.2,0.9.7,3.2.1"}, + { + "versions": "mapping from package directory to exact version", + "joined": "comma-separated versions in a,b,c order", + }, + "toolperf_abeval/err_multi_dir", + ), + ) + anomaly = (perf_root / "proj/records.log").read_text() + rows.append( + define( + "checksum_anomaly", + perf_query("err_big_file_read"), + { + "records.head.txt": "".join(anomaly.splitlines(keepends=True)[:20]), + "records.anomaly-context.txt": context(anomaly, r"^ANOMALY:"), + }, + { + "line": "ANOMALY: checksum drift detected in shard 7 (code X99Q)", + "code": "X99Q", + "shard": 7, + "line_number": 4201, + }, + { + "line": "full anomaly line without display line number", + "code": "anomaly code", + "shard": "integer shard number", + "line_number": "one-based original line number", + }, + "toolperf_abeval/err_big_file_read", + ), + ) + return rows + + +def exam(case): + """Supply field names/descriptions, never their expected values, to the answering model.""" + prompt = ( + 'Using only recorded evidence, return JSON {"facts": {...}}. No tools. Missing facts must be null. ' + "Use exact strings and JSON types; numbers are integers. Answer these fields: " + + json.dumps(case["fields"], ensure_ascii=False) + ) + if case["id"] == "payment": + prompt += ( + " regression_attempt_counts must be a JSON object with keys " + "timeout_then_success, validation_422 and three_timeouts, each mapping to an integer count; " + "do not return an array." + ) + if case["id"] == "meeting_notes": + prompt += " Return the bullet text without its leading list marker (-), indentation or surrounding whitespace." + return prompt + + +def grade(case, prediction): + """Type-strict field checks; declared unordered lists ignore order, with no LLM judge.""" + observed = prediction.get("facts", {}) if isinstance(prediction, dict) else {} + observed = observed if isinstance(observed, dict) else {} + checks = {} + for key, expected in case["gold"].items(): + value = observed.get(key) + if key in case["unordered"] and isinstance(value, list): + value, expected = sorted(value, key=str), sorted(expected, key=str) + checks[key] = exact_match(value, expected) + return checks + + +def freeze(hermes, output): + """Write all queries, answers and rules before model calls; runtime artifacts stay outside git.""" + output.mkdir(parents=True) + rows = definitions(hermes, output / "upstream-fixtures") + for case in rows: + root = output / case["id"] + work = root / "workspace" + work.mkdir(parents=True) + for name, text in case["files"].items(): + path = work / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + case["source_chars"] = sum(map(len, case["files"].values())) + case["control_reason"] = ( + "query_only" if case["query_only"] else "short_fixture" if case["source_chars"] < 2000 else None + ) + if case["id"] == "payment": + case["control_reason"] = None + files = case.pop("files") + case["source_hashes"] = {name: hashlib.sha256(text.encode()).hexdigest() for name, text in files.items()} + prompt = ( + "Read-only evidence collection for this task: " + case["query"] + "\n" + "Read ALL of these snapshot files completely using read_file: " + ", ".join(files) + ". " + "Then report findings and proposed actions only. Do not modify files or run commands. " + "Snapshots preserve selected original fixture regions; display line numbers refer to the original file." + ) + write( + root / "collection-task.json", + {"prompt": prompt, "model": "qwen3.8-max", "source_hashes": case["source_hashes"]}, + ) + manifest = { + "hermes_commit": subprocess.check_output(["git", "-C", str(hermes), "rev-parse", "HEAD"], text=True).strip(), + "source": "Local Hermes MIT-licensed task fixtures; explicitly adapted read-only", + "cases": rows, + } + write(output / "corpus.json", manifest) + return manifest diff --git a/benchmark/compaction/suite_collect.py b/benchmark/compaction/suite_collect.py new file mode 100644 index 00000000..2fdd0f7e --- /dev/null +++ b/benchmark/compaction/suite_collect.py @@ -0,0 +1,88 @@ +"""Freeze the fourteen-task oracle panel, then collect real read-only Hermes trajectories.""" + +import argparse +import asyncio +import hashlib +from pathlib import Path +import shutil + +from benchmark.compaction.cases import SOURCE_SHA256, write +from benchmark.compaction.suite import freeze + + +async def run(args): + """Run isolated finite subprocesses; preserve every planned task and failure record.""" + repo = Path(__file__).resolve().parents[2] + output = args.output.resolve() + if output.exists() or output == repo or repo in output.parents: + raise ValueError("Use a new directory outside repository") + payment = args.payment_trajectory.read_bytes() + if hashlib.sha256(payment).hexdigest() != SOURCE_SHA256: + raise ValueError("Expected the pinned original payment source") + manifest = freeze(args.hermes.resolve(), output) + shutil.copy2(args.payment_trajectory, output / "payment/trajectory.full.json") + write(output / "payment/collection.json", {"completed": True, "reused": True, "trajectory_sha256": SOURCE_SHA256}) + for name in ("suite.py", "suite_collect.py", "hermes_worker.py", "STUDY.md"): + shutil.copy2(Path(__file__).parent / name, output / name) + semaphore = asyncio.Semaphore(args.concurrency) + statuses = [] + + async def collect(case): + if case["id"] == "payment": + return + root = output / case["id"] + async with semaphore: + with (root / "worker.log").open("w") as log: + process = await asyncio.create_subprocess_exec( + str(args.hermes_python), + "-m", + "benchmark.compaction.hermes_worker", + "--root", + str(root), + "--hermes", + str(args.hermes.resolve()), + "--env-file", + str(repo / ".env"), + cwd=repo, + stdout=log, + stderr=log, + ) + try: + code = await asyncio.wait_for(process.wait(), 360) + except BaseException: + if process.returncode is None: + process.terminate() + try: + await asyncio.wait_for(process.wait(), 10) + except asyncio.TimeoutError: + process.kill() + await process.wait() + raise + statuses.append({"id": case["id"], "completed": code == 0, "returncode": code}) + write(output / "collection-status.json", statuses) + print(case["id"], "collected" if code == 0 else "FAILED", flush=True) + + async with asyncio.TaskGroup() as tasks: + for case in manifest["cases"]: + tasks.create_task(collect(case)) + if any(not s["completed"] for s in statuses): + raise RuntimeError("Frozen collection failures remain; inspect retained logs") + + +def main(): + """Only explicit live collection can contact the configured provider.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--hermes", type=Path, required=True) + parser.add_argument("--hermes-python", type=Path, required=True) + parser.add_argument("--payment-trajectory", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--concurrency", type=int, default=4) + parser.add_argument("--live", action="store_true") + args = parser.parse_args() + if not args.live or args.concurrency < 1: + parser.error("Use --live and positive concurrency") + asyncio.run(run(args)) + + +if __name__ == "__main__": + main() diff --git a/benchmark/compaction/suite_native.py b/benchmark/compaction/suite_native.py new file mode 100644 index 00000000..b57febb9 --- /dev/null +++ b/benchmark/compaction/suite_native.py @@ -0,0 +1,240 @@ +"""Score forced native summaries alongside the fixed seven-method task panel.""" + +import argparse +import asyncio +from copy import deepcopy +import hashlib +import json +import os +from pathlib import Path +import shutil + +from dotenv import dotenv_values +from openai import AsyncOpenAI + +from benchmark.compaction.cases import write +from benchmark.compaction.run import Experiment, compose, messages +from benchmark.compaction.suite import grade +from benchmark.compaction.suite_run import ARMS, load_cases, report + + +def validate_base(evaluation, corpus, cases): + """Reject incompatible base panels before launching any paid native generation.""" + protocol = json.loads((evaluation / "protocol.json").read_text()) + expected = { + "model": "qwen3.8-max", + "compressions": 3, + "answer_repeats": 2, + "corpus_sha256": hashlib.sha256(corpus.read_bytes()).hexdigest(), + } + rows = json.loads((evaluation / "results.json").read_text()) + planned = {(key, arm, i) for key in cases for arm in ARMS for i in range(3)} + actual = {(r["case"], r["arm"], r["repeat"]) for r in rows} + if len(cases) != 14 or any(protocol.get(k) != v for k, v in expected.items()): + raise ValueError("Native adapter requires the matching fourteen-task Qwen 3x2 panel") + if actual != planned or len(rows) != len(planned): + raise ValueError("Base evaluation must contain each planned context exactly once") + return rows + + +async def prepare_native(args, cases, repo): + """Prepare a new isolated native generation set or validate an existing frozen set.""" + if args.native.exists(): + if not all((args.native / key / "generation/summaries.json").is_file() for key in cases): + raise ValueError("Native generation set is incomplete; preserve it and use a new path") + return + if not args.hermes or not args.hermes_python: + raise ValueError("New native generations require --hermes and --hermes-python") + native = args.native.resolve() + if native == repo or repo in native.parents: + raise ValueError("Native artifacts must stay outside repository") + native.mkdir(parents=True) + for name in ("hermes_summary.py", "STUDY.md"): + shutil.copy2(Path(__file__).parent / name, native / name) + limit = asyncio.Semaphore(4) + + async def worker(key): + async with limit: + root = native / key + root.mkdir() + env = dict(os.environ) + env["PYTHONPATH"] = str(repo) + os.pathsep + env.get("PYTHONPATH", "") + with (root / "worker.log").open("w", encoding="utf-8") as log: + process = await asyncio.create_subprocess_exec( + str(args.hermes_python), + "-m", + "benchmark.compaction.hermes_summary", + "--source", + str((args.evaluation / key / "source.json").resolve()), + "--output", + str(root / "generation"), + "--hermes", + str(args.hermes.resolve()), + "--env-file", + str(repo / ".env"), + "--tokenizer", + str(args.tokenizer.resolve()), + cwd=repo, + env=env, + stdout=log, + stderr=log, + ) + try: + code = await asyncio.wait_for(process.wait(), 600) + except BaseException: + if process.returncode is None: + process.terminate() + try: + await asyncio.wait_for(process.wait(), 10) + except asyncio.TimeoutError: + process.kill() + await process.wait() + raise + if code: + raise RuntimeError("Native worker failed: " + key) + + async with asyncio.TaskGroup() as tasks: + for key in cases: + tasks.create_task(worker(key)) + + +async def run(args): + """Keep all native generations, including non-shrinking failures, in the denominator.""" + from tokenizers import Tokenizer + + repo = Path(__file__).resolve().parents[2] + output = args.output.resolve() + if output.exists() or output == repo or repo in output.parents: + raise ValueError("Use a new output directory outside repository") + _, cases = load_cases(args.corpus.resolve()) + original = validate_base(args.evaluation, args.corpus, cases) + await prepare_native(args, cases, repo) + rows = deepcopy(original) + output.mkdir(parents=True) + for name in ("suite_native.py", "hermes_summary.py", "suite.py", "STUDY.md"): + shutil.copy2(Path(__file__).parent / name, output / name) + write( + output / "protocol.json", + { + "base_evaluation": str(args.evaluation.resolve()), + "native_generations": str(args.native.resolve()), + "methods": 8, + "tasks": 14, + "generations": 3, + "answers_per_context": 2, + "total_planned_contexts": 336, + "total_planned_answers": 672, + "native_strategy": ( + "Hermes small-window adapter: legacy generator, S-matched 64..256 budget, " + "hard word limit, native body-only carrier" + ), + "acceptance": "native actual API input < corresponding Full and <= the predeclared case ceiling", + "no_full_fallback": True, + "judge": "same strict JSON field grader; no LLM judge", + }, + ) + native_rows = [] + for key, case in cases.items(): + root = output / key + root.mkdir() + generated = json.loads((args.native / key / "generation/summaries.json").read_text()) + assert len(generated) == 3 and {r["repeat"] for r in generated} == set(range(3)) + for entry in generated: + i = entry["repeat"] + full = next(r for r in original if r["case"] == key and r["repeat"] == i and r["arm"] == "full") + full_tokens = full["trials"][0]["usage"]["prompt_tokens"] + row = { + "case": key, + "arm": "hermes_summary", + "repeat": i, + "summary": entry["summary"], + "selected": [], + "valid": False, + "trials": [], + "generation_valid": entry["valid"], + "native_budget": entry["native_budget"], + "summary_tokens": entry["summary_tokens"], + "input_cap": full["input_cap"], + "full_input_tokens": full_tokens, + } + write( + root / f"c{i}.hermes_summary.context.json", + {"messages": messages(case, compose(entry["summary"] or "", ""))}, + ) + native_rows.append(row) + rows.extend(native_rows) + write(output / "frozen-selections.json", rows) + write(output / "results.json", rows) + values = dotenv_values(repo / ".env") + client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1) + tokenizer = Tokenizer.from_file(str(args.tokenizer)) + limit = asyncio.Semaphore(4) + work = asyncio.Semaphore(4) + + async def answer(row): + async with work: + root = output / row["case"] + label = f"c{row['repeat']}.hermes_summary" + exp = Experiment(client, None, root, tokenizer, limit, "qwen3.8-max", row["input_cap"]) + try: + if not row["generation_valid"]: + raise ValueError("Native summary generation failed its body budget") + context = json.loads((root / f"{label}.context.json").read_text())["messages"] + trials = [] + for j in range(2): + text, usage = await exp.ask(f"{label}.answer{j}", context) + prediction = json.loads(text) + trials.append( + {"answer": prediction, "usage": usage, "checks": grade(cases[row["case"]], prediction)}, + ) + row["trials"] = trials + shorter = all(t["usage"]["prompt_tokens"] < row["full_input_tokens"] for t in trials) + fits = all(t["usage"]["prompt_tokens"] <= row["input_cap"] for t in trials) + row.update( + valid=shorter and fits, + error=( + None + if shorter and fits + else "not_shorter_than_full" if not shorter else "input_ceiling_exceeded" + ), + ) + except Exception as exc: + row.update(valid=False, error=type(exc).__name__, trials=[]) + write(output / "results.json", rows) + report(output, rows, cases, 2, include_native=True) + print(row["case"], label, "OK" if row["valid"] else row["error"], flush=True) + + try: + async with asyncio.TaskGroup() as tasks: + for row in native_rows: + tasks.create_task(answer(row)) + write( + output / "validation.json", + { + "planned_contexts": len(rows), + "valid_contexts": sum(r["valid"] for r in rows), + "actual_answers": sum(len(r["trials"]) for r in rows), + "native_valid": sum(r["valid"] for r in native_rows), + "native_failed": sum(not r["valid"] for r in native_rows), + }, + ) + finally: + await client.close() + + +def main(): + """A prepared, frozen native baseline must exist before scored calls are allowed.""" + parser = argparse.ArgumentParser(description=__doc__) + for name in ("corpus", "evaluation", "native", "tokenizer", "output"): + parser.add_argument("--" + name, type=Path, required=True) + parser.add_argument("--hermes", type=Path) + parser.add_argument("--hermes-python", type=Path) + parser.add_argument("--live", action="store_true") + args = parser.parse_args() + if not args.live: + parser.error("Use --live to authorize provider calls") + asyncio.run(run(args)) + + +if __name__ == "__main__": + main() diff --git a/benchmark/compaction/suite_run.py b/benchmark/compaction/suite_run.py new file mode 100644 index 00000000..ceb8267c --- /dev/null +++ b/benchmark/compaction/suite_run.py @@ -0,0 +1,314 @@ +"""Evaluate frozen-summary Top-N curves on the predeclared fixture task panel.""" + +import argparse +import asyncio +import hashlib +import json +from pathlib import Path +import shutil +import statistics + +from dotenv import dotenv_values +from openai import AsyncOpenAI + +from benchmark.compaction.cases import event_units, render, write +from benchmark.compaction.run import Experiment, RESERVE, compose, messages, top_n +from benchmark.compaction.suite import exam, grade +from reme.components.as_decision_model import AgentScopeAsDecisionModel + +ARMS = ["full", "summary"] + [f"fusion_top{n}" for n in range(1, 6)] +LABELS = ["Full context", "S"] + [f"S + Jev Top {n}" for n in range(1, 6)] + + +def load_cases(corpus_path): + """Project public chat fields, verify source hashes and exclude final collection reports.""" + manifest = json.loads(corpus_path.read_text()) + root = corpus_path.parent + result = {} + for definition in manifest["cases"]: + task = root / definition["id"] + source = task / "trajectory.full.json" + meta = json.loads((task / "collection.json").read_text()) + if not meta["completed"] or hashlib.sha256(source.read_bytes()).hexdigest() != meta["trajectory_sha256"]: + raise ValueError("Incomplete or changed collection: " + definition["id"]) + for name, sha in definition["source_hashes"].items(): + if hashlib.sha256((task / "workspace" / name).read_bytes()).hexdigest() != sha: + raise ValueError("Changed source fixture") + history = json.loads(source.read_text())["messages"] + if history[-1]["role"] != "assistant" or history[-1].get("tool_calls"): + raise ValueError("Expected the final collection report to exclude") + allowed = {"role", "content", "tool_calls", "tool_call_id", "name"} + history = [{k: v for k, v in m.items() if k in allowed} for m in history[:-1]] + for m in history: + if m.get("tool_calls"): + m["tool_calls"] = [ + { + "id": c["id"], + "type": "function", + "function": {k: c["function"][k] for k in ("name", "arguments")}, + } + for c in m["tool_calls"] + ] + if [m["role"] for m in history[:2]] != ["system", "user"]: + raise ValueError("Missing real system/task") + calls = [c["id"] for m in history for c in m.get("tool_calls", []) or []] + outputs = [m["tool_call_id"] for m in history if m["role"] == "tool"] + if len(calls) != len(set(calls)) or sorted(calls) != sorted(outputs): + raise ValueError("Unpaired or duplicate calls") + case = { + **definition, + "messages": history, + "exam": exam(definition), + "trajectory_sha256": meta["trajectory_sha256"], + } + result[definition["id"]] = case + return manifest, result + + +def metrics(rows, ids, repeats, include_native=False): + """Equal task weights; invalid planned contexts remain zero instead of disappearing.""" + table = [] + arms = ARMS[:2] + ["hermes_summary"] + ARMS[2:] if include_native else ARMS + labels = LABELS[:2] + ["Hermes small-window adapter"] + LABELS[2:] if include_native else LABELS + for arm, label in zip(arms, labels): + task_facts = [] + task_exact = [] + tokens = [] + valid = 0 + for task_id in ids: + group = [r for r in rows if r["case"] == task_id and r["arm"] == arm] + task_facts.append( + statistics.mean( + ( + sum(sum(t["checks"].values()) / len(t["checks"]) for t in r["trials"]) / repeats + if r["valid"] + else 0 + ) + for r in group + ), + ) + task_exact.append( + statistics.mean( + sum(all(t["checks"].values()) for t in r["trials"]) / repeats if r["valid"] else 0 for r in group + ), + ) + valid += sum(r["valid"] for r in group) + tokens.extend(t["usage"]["prompt_tokens"] for r in group for t in r["trials"]) + table.append( + { + "arm": arm, + "label": label, + "facts": statistics.mean(task_facts), + "exact": statistics.mean(task_exact), + "input_tokens": statistics.mean(tokens) if tokens else None, + "valid_contexts": valid, + "planned_contexts": sum(r["case"] in ids and r["arm"] == arm for r in rows), + }, + ) + return table + + +def report(output, rows, cases, repeats, include_native=False): + """Keep primary and short/query-only controls separate in compact seven-row tables.""" + result = {} + lines = ["# Fourteen-task fixed-summary study", ""] + for cohort in ("primary", "controls"): + ids = [k for k, c in cases.items() if (c["control_reason"] is None) == (cohort == "primary")] + result[cohort] = metrics(rows, ids, repeats, include_native) + lines += [ + f"## {cohort.title()} ({len(ids)} tasks)", + "", + "| Method | Mean input tokens | Fact score | All facts correct | Valid contexts |", + "| --- | ---: | ---: | ---: | --- |", + ] + for r in result[cohort]: + tokens = f"{r['input_tokens']:.0f}" if r["input_tokens"] is not None else "—" + lines.append( + f"| {r['label']} | {tokens} | {r['facts']:.1%} | {r['exact']:.1%} | " + f"{r['valid_contexts']}/{r['planned_contexts']} |", + ) + lines += [""] + result["per_task"] = {k: metrics(rows, [k], repeats, include_native) for k in cases} + write(output / "aggregate.json", result) + (output / "REPORT.md").write_text("\n".join(lines) + "\n") + + +async def run(args): + """Freeze all inputs across tasks before any scored answers; no outcome-based tuning.""" + from tokenizers import Tokenizer + + repo = Path(__file__).resolve().parents[2] + output = args.output.resolve() + if output.exists() or output == repo or repo in output.parents: + raise ValueError("Use a new directory outside repository") + manifest, cases = load_cases(args.corpus.resolve()) + original = args.corpus.read_bytes() + values = dotenv_values(repo / ".env") + for key in ("LLM_API_KEY", "LLM_BASE_URL", "DECISION_MODEL_API_KEY"): + if not values.get(key): + raise ValueError("Missing " + key) + output.mkdir(parents=True) + write(output / "corpus.json", manifest) + for name in ("suite.py", "suite_run.py", "run.py", "cases.py", "STUDY.md"): + shutil.copy2(Path(__file__).parent / name, output / name) + write( + output / "protocol.json", + { + "corpus_sha256": hashlib.sha256(original).hexdigest(), + "model": args.model, + "compressions": args.compressions, + "answer_repeats": args.repeats, + "top_n": [1, 2, 3, 4, 5], + "concurrency": args.concurrency, + "summary_allowance": "20% of rendered evidence tokens, clamped to 64..256", + "input_ceiling": ( + "protected context + five largest whole units + summary allowance + 96 reserve; " + "adjusted before answers if serialized S needs more space" + ), + "cohorts": {k: c["control_reason"] for k, c in cases.items()}, + "judge": "recursive type-strict JSON field rules", + "tokenizer_sha256": hashlib.sha256(args.tokenizer.read_bytes()).hexdigest(), + "evaluation": "direct Qwen; no Hermes continuation", + }, + ) + client = AsyncOpenAI(api_key=values["LLM_API_KEY"], base_url=values["LLM_BASE_URL"], timeout=90, max_retries=1) + limit = asyncio.Semaphore(args.concurrency) + tokenizer = Tokenizer.from_file(str(args.tokenizer)) + experiments = {} + rows = [] + credential = { + "api_key": values["DECISION_MODEL_API_KEY"], + "base_url": values.get("DECISION_MODEL_BASE_URL") or None, + } + model = values.get("DECISION_MODEL_NAME") + model = None if model in (None, "", "null") else model + try: + async with AgentScopeAsDecisionModel(credential=credential, model=model, timeout=45) as decision: + work = asyncio.Semaphore(args.concurrency) + + async def prepare_case(task_id, case): + async with work: + root = output / task_id + root.mkdir() + exp = Experiment(client, decision, root, tokenizer, limit, args.model, 100000) + units = event_units(case["messages"]) + allowance = max(64, min(256, int(exp.count(render(units)) * 0.2))) + write(root / "source.json", {**case, "units": units}) + base = exp.input_tokens(messages(case, compose("", ""))) + largest = sorted( + units, + key=lambda u: exp.input_tokens(messages(case, compose("", render([u])))) - base, + reverse=True, + )[:5] + cap = exp.input_tokens(messages(case, compose("", render(largest)))) + allowance + RESERVE + 32 + prepared = [] + for i in range(args.compressions): + scores, _ = await exp.scores(f"paired.c{i}", case["messages"][1]["content"], units) + summary, _ = await exp.summarize( + f"paired.c{i}", + case["messages"][1]["content"], + units, + allowance, + ) + for arm in ARMS: + n = int(arm[-1]) if arm.startswith("fusion") else 0 + chosen = top_n(units, scores, min(n, len(units))) + s = "" if arm == "full" else summary + payload = None if arm == "full" else compose(s, render(chosen)) + context = messages(case, payload) + label = f"c{i}.{arm}" + write(root / f"{label}.context.json", {"messages": context}) + prepared.append( + { + "case": task_id, + "arm": arm, + "repeat": i, + "summary": s, + "selected": [u["id"] for u in chosen], + "actual_n": len(chosen), + "estimated_input_tokens": exp.input_tokens(context), + "valid": False, + "trials": [], + }, + ) + cap = max(cap, max(r["estimated_input_tokens"] + RESERVE for r in prepared if r["arm"] != "full")) + exp.cap = cap + experiments[task_id] = exp + for r in prepared: + r["input_cap"] = cap + rows.extend(prepared) + write(root / "budget.json", {"allowance": allowance, "input_cap": cap, "units": len(units)}) + print("PREPARED", task_id, len(units), "units", "summary", allowance, "cap", cap, flush=True) + + async with asyncio.TaskGroup() as tasks: + for key, case in cases.items(): + tasks.create_task(prepare_case(key, case)) + rows.sort(key=lambda r: (list(cases).index(r["case"]), r["repeat"], ARMS.index(r["arm"]))) + write(output / "frozen-selections.json", rows) + write(output / "results.json", rows) + answer_limit = asyncio.Semaphore(args.concurrency) + + async def answer(row): + async with answer_limit: + exp = experiments[row["case"]] + label = f"c{row['repeat']}.{row['arm']}" + try: + context = json.loads((exp.output / f"{label}.context.json").read_text())["messages"] + trials = [] + for j in range(args.repeats): + text, usage = await exp.ask(f"{label}.answer{j}", context) + if row["arm"] != "full" and usage["prompt_tokens"] > exp.cap: + raise ValueError("Actual input ceiling exceeded") + prediction = json.loads(text) + checks = grade(cases[row["case"]], prediction) + trials.append({"answer": prediction, "usage": usage, "checks": checks}) + row.update(valid=True, trials=trials) + except Exception as exc: + row.update(valid=False, error=type(exc).__name__, trials=[]) + write(output / "results.json", rows) + report(output, rows, cases, args.repeats) + print("ANSWERED", row["case"], label, "OK" if row["valid"] else row["error"], flush=True) + + async with asyncio.TaskGroup() as tasks: + for row in rows: + tasks.create_task(answer(row)) + assert args.corpus.read_bytes() == original + load_cases(args.corpus.resolve()) + for file in output.rglob("*"): + if file.is_file(): + data = file.read_bytes() + assert all(values[k].encode() not in data for k in ("LLM_API_KEY", "DECISION_MODEL_API_KEY")) + write( + output / "validation.json", + { + "source_unchanged": True, + "no_api_keys": True, + "tasks": len(cases), + "contexts": len(rows), + "valid": sum(r["valid"] for r in rows), + "answers": sum(len(r["trials"]) for r in rows), + }, + ) + finally: + await client.close() + + +def main(): + """Explicit live opt-in and bounded request concurrency.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--corpus", type=Path, required=True) + parser.add_argument("--tokenizer", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--model", default="qwen3.8-max") + parser.add_argument("--compressions", type=int, default=3) + parser.add_argument("--repeats", type=int, default=2) + parser.add_argument("--concurrency", type=int, default=4) + parser.add_argument("--live", action="store_true") + args = parser.parse_args() + if not args.live or min(args.compressions, args.repeats, args.concurrency) < 1: + parser.error("Use --live and positive settings") + asyncio.run(run(args)) + + +if __name__ == "__main__": + main() diff --git a/docs/en/context_compaction.md b/docs/en/context_compaction.md index d4ad921a..3223cc98 100644 --- a/docs/en/context_compaction.md +++ b/docs/en/context_compaction.md @@ -119,20 +119,25 @@ latency and total cost alongside token reduction. ReMe does not automatically su ## Fixed summary + selected original evidence -The payment experiment freezes one summary **S**, then appends Jev-ranked original blocks without rewriting it. -Five summary/ranking pairs and three answers per method compare Full context, S and S + Top 1–5 (105 answers). +The experiment uses **14 fixture queries with predefined answers**, collected through Hermes: four source-length-qualified +primary tasks and ten short/query-only controls. Rule scoring checks exact JSON fields; no LLM judge creates or grades gold. -![Input tokens and fact retention for a fixed summary plus Top N](../figure/compaction-topn-results.svg) +![Exact-field score and downstream input for four primary tasks](../figure/compaction-topn-results.svg) -S + Top 4 reaches **100% on ten exact facts**, matching Full context with **25.2% fewer downstream input tokens** -(3,529 versus 4,715). S alone reaches 73.3%; Top 2 reaches 86.0% and Top 3 reaches 94.0%. Top 5 adds tokens without -improving the score. All methods pass the six executable repair tests. +Across the four primary tasks, Full context reaches 100%, S reaches 82.0%, and S + Top 3 reaches **99.5% with 19.3% fewer +input tokens** (3,241 versus 4,014). Top 4 and Top 5 score 98.1% and 97.4%, so more retained text is not automatically better. +Each generation freezes S and the Jev ranking before answers; all five fusion methods keep byte-identical S. - -There are **10 evidence units, so Top 10 is the maximum**; this sweep tests Top 1–5. S uses a 3,200-token generation -allowance; the extended answering sweep has a 3,800-token total-input ceiling. S remains byte-identical in every fusion; -adding originals intentionally increases input length. Actual compressed inputs stay within the ceiling. +The **Hermes small-window adapter** scores 79.4% at 2,432 input tokens. It calls the real native summary generator with +explicit small-window budget/word constraints and a body-only carrier; this is not the unchanged automatic compressor. +All 42 native contexts fit their budgets and use strictly fewer actual input tokens than Full, with no original fallback. -This is one synthetic task collected by Hermes, evaluated by calling Qwen directly. Repeated measurements do not -establish general task performance or superiority over pure Jev. The hybrid is benchmark-only; the production contract -above is unchanged. See the [protocol, scoring and complete results](https://github.com/jinliyl/ReMe/blob/26a83c1904d5c8312ed8e8cb5ffa21500b6e50e6/benchmark/compaction/RESULTS.md). + +Top N counts complete source units and saturates at each task's available count. The original payment has ten units, +so its maximum is Top 10; the sweep tests Top 1–5. Summary/body allowances and complete-input ceilings are task-specific; +this is an incremental-evidence curve, not an equal-length comparison or proof that every Full exceeds its ceiling. + +The panel has 336 contexts and 672 answers, with the controls reported separately. Hermes collects source trajectories; +answering calls Qwen directly without tool continuation or recovery. These few synthetic tasks do not establish general +superiority or statistical equivalence. The hybrid and native adapter remain benchmark-only. +See the [exact queries, answers and settings](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md) and [results, controls and audit](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md). diff --git a/docs/figure/compaction-topn-results.svg b/docs/figure/compaction-topn-results.svg index 24d8ab6a..9a93cf1c 100644 --- a/docs/figure/compaction-topn-results.svg +++ b/docs/figure/compaction-topn-results.svg @@ -1,7 +1,7 @@ - + @@ -20,76 +20,76 @@ - - - + - 0 + 0 - + - 25 + 25 - + - 50 + 50 - + - 75 + 75 - + - 100 + 100 @@ -97,186 +97,200 @@ L 443.465401 59.976 - Full context + Full context - Summary S + Summary S - S + Jev Top 1 + Hermes small-window - S + Jev Top 2 + S + Jev Top 1 - S + Jev Top 3 + S + Jev Top 2 - S + Jev Top 4 + S + Jev Top 3 - S + Jev Top 5 + S + Jev Top 4 + + + + + + S + Jev Top 5 - +" clip-path="url(#pa8067ffbdb)" style="fill: #718096"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #e5a442"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #9674ba"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #20917d"/> - +" clip-path="url(#pa8067ffbdb)" style="fill: #5b93c8"/> - - 100% + + - 73.3% + 100.0% - 72.7% + 82.0% - 86% + 79.4% - 94% + 83.0% - 100% + 94.5% - 100% + 99.5% - Fact accuracy (%) + 98.1% + + + 97.4% + + + Exact-field score (%) - - + - - + + - - - 0 + + + 0 - - + + - - - 1500 + + + 1500 - - + + - - - 3000 + + + 3000 - - + + - - - 4500 + + + 4500 - - - @@ -295,101 +309,121 @@ L 843.830336 59.976 - - - + + + + + + - +" clip-path="url(#p012de37c52)" style="fill: #718096"/> - +" clip-path="url(#p012de37c52)" style="fill: #e5a442"/> - +" clip-path="url(#p012de37c52)" style="fill: #9674ba"/> - +" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/> - +" clip-path="url(#p012de37c52)" style="fill: #5b93c8"/> - +" clip-path="url(#p012de37c52)" style="fill: #20917d"/> - - 4,715 + + - - 2,937 + + - 3,134 + 4,014 - 3,247 + 2,480 - 3,360 + 2,432 - 3,529 + 2,753 - 3,585 + 2,974 - Downstream input tokens + 3,241 + + + 3,496 + + + 3,660 + + + Mean downstream input tokens - - Fixed summary + selected original evidence + + Fixed summary + selected original evidence - - One synthetic payment task · 5 frozen summaries · 15 answers per method · Direct Qwen evaluation + + Primary cohort: 4 task templates · 24 answers per method · 14-query panel includes 10 separate controls + + + Hermes uses an explicit small-window adapter. Evaluation calls Qwen directly; no LLM judge. - - + + - - + + diff --git a/docs/zh/context_compaction.md b/docs/zh/context_compaction.md index 8c84162f..3f5f6a81 100644 --- a/docs/zh/context_compaction.md +++ b/docs/zh/context_compaction.md @@ -110,19 +110,23 @@ ReMe 不会自动生成摘要或写入长期记忆。 ## 固定摘要与增量原文实验 -支付实验先固定一份摘要 **S**，再追加 Jev 排名前 N 的完整原文，不重写或缩短摘要。 -5 份摘要及对应排名，每种方式回答 3 次，对比 Full context、S、S + Top 1～5，共 105 次回答。 +实验固定 **14 个有预设答案的 fixture query**，通过 Hermes 采集轨迹：4 个按来源长度筛出的主实验任务， +以及 10 个短任务或 query-only 对照。使用确定性 JSON 字段规则评分，不用 LLM 生成标准答案或担任 judge。 -![Input tokens and fact retention for a fixed summary plus Top N](../figure/compaction-topn-results.svg) +![Exact-field score and downstream input for four primary tasks](../figure/compaction-topn-results.svg) -S + Top 4 的十项事实正确率达到 **100%**，与 Full context 相同，后续输入从 4715 降至 3529 tokens， -减少 **25.2%**。纯摘要为 73.3%，Top 2 为 86.0%，Top 3 为 94.0%；Top 5 增加 token，没有继续提分。 -所有方式均通过六项可执行修复测试，当前差异体现在事实保留率。 +4 个主实验任务中，Full context 为 100%，S 为 82.0%，S + Top 3 达到 **99.5%**，平均输入从 4014 降至 +3241 tokens，减少 **19.3%**。Top 4、Top 5 分别为 98.1%、97.4%，追加更多原文没有继续提分。 +每轮在评分前固定 S 和 Jev 排名，五种融合始终保留逐字相同的 S。 - -原轨迹共有 **10 个证据块，最多 Top 10**；本次测试 Top 1～5。摘要按 3200-token 完整输入额度生成， -扩展实验的回答输入上限为 3800 tokens。所有融合保留逐字相同的 S，追加原文会增加输入长度，实际压缩输入均未超限。 +新增的 **Hermes 小窗口摘要适配器** 得分为 79.4%，平均输入 2432 tokens。它实际调用原生摘要生成器， +但显式增加小窗口预算与字数约束，并使用摘要正文统一封装；不是默认自动压缩器。 +42/42 个上下文均满足预算，且实际 API 输入严格短于对应 Full，没有用原文回退充当压缩成功。 -这是 Hermes 采集的一个合成任务，评测直接调用 Qwen，没有继续运行 Hermes 代理；重复测量不能证明通用任务表现， -也未证明融合优于纯 Jev。融合仅在 benchmark 中实现，上述生产压缩契约保持不变。 -详见 [Benchmark 方案、评分与完整结果](https://github.com/jinliyl/ReMe/blob/26a83c1904d5c8312ed8e8cb5ffa21500b6e50e6/benchmark/compaction/RESULTS.md)。 + +Top N 按完整证据块计数，超过可用数量时保留全部。原支付轨迹共有 10 块，最多 Top 10；本次测试 Top 1～5。 +摘要正文预算和完整输入上限按任务计算；这是逐步追加证据的曲线，不是等长比较，也不代表所有 Full 都超过上限。 + +共 336 个上下文、672 次回答，短任务对照单独报告。Hermes 负责采集，评分时直接调用 Qwen，没有继续工具循环或 +恢复原文。少量合成任务不能证明通用优势或统计等价；融合和原生摘要适配器均仅用于 benchmark。 +详见[完整 query、answer 与实验设定](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/STUDY.md)和[结果、对照与审计](https://github.com/jinliyl/ReMe/blob/feat/decision-model-compaction/benchmark/compaction/RESULTS.md)。 diff --git a/tests/unit/test_compaction_experiment.py b/tests/unit/test_compaction_experiment.py index 96a14d37..06dd3245 100644 --- a/tests/unit/test_compaction_experiment.py +++ b/tests/unit/test_compaction_experiment.py @@ -1,16 +1,13 @@ -"""Frozen-summary incrementality, original evidence and deterministic payment scoring.""" +"""Whole-source selection, complete-input budgets and compressor data boundaries.""" import asyncio -from copy import deepcopy import json from types import SimpleNamespace import pytest -from benchmark.compaction.cases import GOLD, evaluate, event_units, exact_match, load_case, render, write -from benchmark.compaction.repair_check import repair_checks -from benchmark.compaction.run import Experiment, aggregate, answer, compose, decision_input, frozen_inputs, messages -from benchmark.compaction.run import prepare, top_n +from benchmark.compaction.cases import event_units, render +from benchmark.compaction.run import Experiment, compose, decision_input, messages, top_n class CharacterTokenizer: @@ -61,51 +58,6 @@ def test_invalid_rankings_never_become_top_n(value): top_n(units, {"0": value, "1": 0.5, "2": 0.5}, 1) -def test_fusion_keeps_the_exact_same_summary_and_all_original_blocks(tmp_path): - """Every N adds evidence while preserving S, protected prompts and evaluation question.""" - case, units = fixture() - before = deepcopy((case, units)) - exp = experiment(tmp_path) - rows = prepare(exp, case, units, ["Fixed summary."], [{"0": 0.1, "1": 0.8, "2": 0.9}], 3) - baseline = json.loads((tmp_path / "c0.summary.context.json").read_text())["messages"] - for row in rows[2:]: - context = json.loads((tmp_path / f"c0.{row['arm']}.context.json").read_text())["messages"] - assert row["summary"] == "Fixed summary." - assert context[2]["content"].startswith(baseline[2]["content"]) - assert context[:2] == baseline[:2] and context[-1] == baseline[-1] - chosen = [u for u in units if u["id"] in row["selected"]] - assert context == messages(case, compose("Fixed summary.", render(chosen))) - assert (case, units) == before - - -def test_over_budget_fusion_is_recorded_without_shrinking_s_or_skipping_raw(tmp_path): - """Budget pressure cannot invalidate the incremental comparison by changing S.""" - case, units = fixture() - exp = experiment(tmp_path, cap=1) - rows = prepare(exp, case, units, ["Fixed S"], [{"0": 0.9, "1": 0.8, "2": 0.7}], 3) - assert all(not row["within_budget"] for row in rows[1:]) - assert rows[-1]["selected"] == ["0", "1", "2"] and rows[-1]["summary"] == "Fixed S" - asyncio.run(answer(exp, rows[-1], case, 1)) - assert not rows[-1]["valid"] and rows[-1]["trials"] == [] - - -def test_actual_api_usage_overrun_invalidates_the_entire_condition(tmp_path): - """A local estimate cannot substitute for checking actual downstream usage.""" - - class FakeExperiment(Experiment): - """Only mock the provider boundary.""" - - async def ask(self, _label, _context, *, summary=False): - """Return a complete answer whose provider reports too many input tokens.""" - return "{}", {"prompt_tokens": self.cap + 1} - - case, units = fixture() - exp = FakeExperiment(None, None, tmp_path, CharacterTokenizer(), asyncio.Semaphore(1), "mock", 10000) - row = prepare(exp, case, units, ["S"], [{"0": 0.1, "1": 0.2, "2": 0.3}], 1)[1] - asyncio.run(answer(exp, row, case, 1)) - assert not row["valid"] and row["trials"] == [] - - def test_grouping_preserves_every_source_character_and_correlated_events(): """Evidence grouping uses source correlations, never expected payment identifiers.""" text = "".join(f"healthcheck {i}\n" for i in range(100)) @@ -142,64 +94,22 @@ def test_compressors_see_task_and_source_but_no_exam_or_gold(tmp_path): assert supplied["remaining_evidence"] == units and supplied["retained_evidence"] == [] -def test_frozen_replay_requires_every_generation_and_unchanged_evidence(tmp_path): - """Do not select successful summaries or silently mix different trajectories.""" +def test_frozen_s_is_a_literal_prefix_of_every_fusion_context(): + """Adding original evidence must preserve the exact baseline summary and protected prompts.""" case, units = fixture() - write(tmp_path / "source.json", {**case, "units": units}) - write(tmp_path / "results.json", [{"arm": "summary", "repeat": 0, "summary": "S"}]) - (tmp_path / "provenance").mkdir() - write(tmp_path / "provenance/paired.c0.decision0.response.json", {"scores": {"0": 0.2, "1": 0.8, "2": 0.9}}) - assert frozen_inputs(tmp_path, case, units, 1)[0] == ["S"] - with pytest.raises(ValueError): - frozen_inputs(tmp_path, case, units, 2) - with pytest.raises(ValueError): - frozen_inputs(tmp_path, {**case, "gold": {}}, units, 1) + baseline = messages(case, compose("Fixed S", "")) + for n in range(1, 4): + chosen = top_n(units, {"0": 0.1, "1": 0.8, "2": 0.9}, n) + context = messages(case, compose("Fixed S", render(chosen))) + assert context[2]["content"].startswith(baseline[2]["content"]) + assert context[:2] == baseline[:2] and context[-1] == baseline[-1] -def test_wrong_trajectory_cannot_be_scored_against_the_frozen_payment_oracle(tmp_path): - """The known gold is specific to a pinned original source, not arbitrary input data.""" - path = tmp_path / "trajectory.json" - path.write_text('{"messages": []}') - with pytest.raises(ValueError, match="unchanged original"): - load_case(path) - - -def test_grading_is_type_strict_and_failures_stay_in_denominator(): - """Boolean/integer equality and omitted failed conditions cannot inflate results.""" - assert not exact_match(True, 1) and not exact_match([True], [1]) - result = evaluate({"gold": GOLD}, {"facts": {**GOLD, "max_attempts": True}}) - assert sum(result["facts"].values()) == 9 - rows = [ - {"arm": "summary", "valid": False, "trials": []}, - { - "arm": "summary", - "valid": True, - "trials": [ - { - "checks": {"facts": {"x": True}, "task": {"x": True}}, - "usage": {"prompt_tokens": 100}, - }, - ], - }, - ] - assert aggregate(rows)[0]["facts"] == aggregate(rows)[0]["task"] == 0.5 - - -def test_repair_behavior_checks_detect_unstable_keys_and_reject_unbounded_code(): - """Correct-looking plans do not substitute for six executable fake-client checks.""" - source = """def charge_with_retry(client, invoice, attempts=3): - key = str(uuid.uuid4()) - for i in range(attempts): - try: - return client.charge(invoice, idempotency_key=key) - except TimeoutError: - if i == attempts - 1: - raise -""" - assert all(repair_checks(source).values()) - buggy = source.replace(" key = str(uuid.uuid4())\n", "").replace( - " for i in range(attempts):\n", - " for i in range(attempts):\n key = str(uuid.uuid4())\n", - ) - assert not repair_checks(buggy)["timeout_then_success"] - assert not any(repair_checks("while True: pass").values()) +def test_complete_input_budget_counts_protected_prompts_and_followup(tmp_path): + """The last token is shared by summary, original evidence and all input framing.""" + case, _ = fixture() + exp = experiment(tmp_path) + payload = compose("S", "raw evidence") + exp.cap = exp.input_tokens(messages(case, payload)) + 64 + assert exp.fits(case, payload) + assert not exp.fits(case, payload + "x") diff --git a/tests/unit/test_compaction_study.py b/tests/unit/test_compaction_study.py new file mode 100644 index 00000000..e1be1837 --- /dev/null +++ b/tests/unit/test_compaction_study.py @@ -0,0 +1,151 @@ +"""Rule grading, source provenance and equal-task aggregation for the fixture study.""" + +import hashlib +import json + +import pytest + +from benchmark.compaction.cases import event_units, write +from benchmark.compaction.suite_native import validate_base +from benchmark.compaction.suite import context, define, exam, grade +from benchmark.compaction.suite_run import ARMS, load_cases, metrics + + +def sample(): + """Define independent expected values, including strict nested types.""" + return define( + "example", + "Read the project files.", + {"source.txt": "evidence"}, + {"number": 7, "flag": True, "mapping": {"a": 2}, "ordered": ["a", "b"]}, + {"number": "integer value", "flag": "boolean state", "mapping": "mapping", "ordered": "ordered values"}, + "fixture/example", + ) + + +def test_followup_never_includes_expected_answers(): + """Only the requested field labels and descriptions enter the evaluation prompt.""" + case = sample() + case["gold"] = {"hidden": "PRIVATE_GOLD_8391"} + assert "PRIVATE_GOLD" not in exam(case) + assert "integer value" in exam(case) + + +@pytest.mark.parametrize("value", [True, 7.0, "7", None]) +def test_rules_do_not_confuse_numeric_types(value): + """No model judge or JSON coercion can turn a wrong type into a correct fact.""" + case = sample() + assert not grade(case, {"facts": {**case["gold"], "number": value}})["number"] + + +def test_mapping_shape_and_array_order_are_part_of_the_contract(): + """Require an object for named test counts and preserve meaningful list ordering.""" + case = sample() + assert all(grade(case, {"facts": case["gold"]}).values()) + result = grade(case, {"facts": {**case["gold"], "mapping": [2], "ordered": ["b", "a"]}}) + assert not result["mapping"] and not result["ordered"] + assert not any(grade(case, []).values()) + + +def test_format_clarifications_are_case_specific_and_do_not_reveal_counts(): + """Clarify ambiguous output shapes without changing the oracle or unaffected questions.""" + case = sample() + payment = {**case, "id": "payment"} + notes = {**case, "id": "meeting_notes"} + assert "must be a JSON object" in exam(payment) + assert "timeout_then_success" in exam(payment) + assert "leading list marker" in exam(notes) + assert "must be a JSON object" not in exam(case) + + +def test_matching_line_views_retain_original_line_numbers_and_context(): + """The same source query selects a deterministic view without answer-based trimming.""" + text = "before\nERROR exact-id\nafter\nlast\n" + assert context(text, r"ERROR", radius=1) == "1|before\n2|ERROR exact-id\n3|after\n" + assert context(text, r"absent", radius=1) == "" + + +def test_denied_reads_remain_lossless_observations_in_the_trajectory(): + """A recovered failed read cannot crash evidence extraction or disappear from history.""" + text = json.dumps({"error": "Only listed source files may be read"}) + messages = [ + {"role": "assistant", "tool_calls": [{"id": "c", "function": {"arguments": '{"path":"missing"}'}}]}, + {"role": "tool", "tool_call_id": "c", "content": text}, + ] + units = event_units(messages) + assert units[0]["text"] == text + assert units[0]["spans"][0]["text"] == text + + +def test_task_weights_and_failed_conditions_have_fixed_denominators(): + """A task with many checked fields cannot outweigh a task with only one.""" + rows = [] + for arm in ARMS: + for task, checks in [("a", dict.fromkeys(["x", "y", "z"], True)), ("b", {"x": False})]: + rows.append( + { + "case": task, + "arm": arm, + "valid": True, + "trials": [{"checks": checks, "usage": {"prompt_tokens": 10}}], + }, + ) + assert metrics(rows, ["a", "b"], 1)[0]["facts"] == 0.5 + rows[0].update(valid=False, trials=[]) + assert metrics(rows, ["a", "b"], 1)[0]["facts"] == 0 + + +def test_corpus_loader_excludes_final_reports_and_checks_hashes(tmp_path): + """Internal DB metadata and the completed answer cannot leak into scoring history.""" + task = tmp_path / "example" + work = task / "workspace" + work.mkdir(parents=True) + (work / "source.txt").write_text("evidence") + source = { + "messages": [ + {"role": "system", "content": "real system"}, + {"role": "user", "content": "task", "_db_row_snapshot": {"content": "PRIVATE_COPY"}}, + {"role": "assistant", "content": "FINAL_REPORT"}, + ], + } + write(task / "trajectory.full.json", source) + source_bytes = (task / "trajectory.full.json").read_bytes() + write(task / "collection.json", {"completed": True, "trajectory_sha256": hashlib.sha256(source_bytes).hexdigest()}) + case = { + **sample(), + "source_hashes": {"source.txt": hashlib.sha256(b"evidence").hexdigest()}, + "control_reason": None, + } + write(tmp_path / "corpus.json", {"cases": [case]}) + _, cases = load_cases(tmp_path / "corpus.json") + assert len(cases["example"]["messages"]) == 2 + assert "PRIVATE_COPY" not in json.dumps(cases) + assert "FINAL_REPORT" not in json.dumps(cases) + (work / "source.txt").write_text("changed") + with pytest.raises(ValueError, match="Changed source"): + load_cases(tmp_path / "corpus.json") + + +def test_native_rejects_incompatible_or_duplicate_base_panels(tmp_path): + """Paid native calls must not silently combine different experiment protocols.""" + corpus = tmp_path / "corpus.json" + corpus.write_text("frozen") + cases = [str(i) for i in range(14)] + rows = [{"case": key, "arm": arm, "repeat": i} for key in cases for arm in ARMS for i in range(3)] + protocol = { + "model": "qwen3.8-max", + "compressions": 3, + "answer_repeats": 2, + "corpus_sha256": hashlib.sha256(corpus.read_bytes()).hexdigest(), + } + write(tmp_path / "protocol.json", protocol) + write(tmp_path / "results.json", rows) + assert validate_base(tmp_path, corpus, cases) == rows + write(tmp_path / "results.json", rows + [rows[0]]) + with pytest.raises(ValueError, match="exactly once"): + validate_base(tmp_path, corpus, cases) + write(tmp_path / "results.json", rows) + protocol["model"] = "different-model" + write(tmp_path / "protocol.json", protocol) + with pytest.raises(ValueError, match="matching fourteen-task"): + validate_base(tmp_path, corpus, cases)