mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-23 05:35:30 +00:00
test: add live decision compaction scenarios and strategy guides
This commit is contained in:
parent
ae82e4dde5
commit
4d43236db2
10 changed files with 854 additions and 7 deletions
12
README.md
12
README.md
|
|
@ -49,6 +49,11 @@ users retain control of the durable files.
|
|||
|
||||
## 📰 Latest Updates
|
||||
|
||||
<!-- Keep at most 5 updates expanded; move older entries into the collapsed section below. -->
|
||||
|
||||
- [2026.10] - **[Decision-model context compaction](docs/en/context_compaction.md)**: score old tool calls and results
|
||||
against the next action, preserve protected history, and return a recoverable compaction proposal. Includes an SVG
|
||||
walkthrough and two live-model integration scenarios.
|
||||
- [2026.10] - **[ReMe Multimodal Memory](https://reme.agentscope.io/en/blog_20261009) update**: Auto Memory reads images
|
||||
with their conversation context and preserves image sources; Auto Resource turns image files into searchable daily
|
||||
cards linked to the originals.
|
||||
|
|
@ -67,6 +72,10 @@ users retain control of the durable files.
|
|||
- [2026.09] - **[Hermes Agent memory provider](https://reme.agentscope.io/en/integrations/hermes) available**: choose HTTP or embedded
|
||||
mode for automatic recall before model calls and asynchronous `auto_memory` after completed turns. The integration
|
||||
supports Hermes Agent 0.21+ and includes profile-aware background work.
|
||||
|
||||
<details>
|
||||
<summary>More updates</summary>
|
||||
|
||||
- [2026.09] - **[OpenClaw plugin](https://reme.agentscope.io/en/integrations/openclaw) released**: install it from
|
||||
[ClawHub](https://clawhub.ai/agentscope-ai/plugins/reme-openclaw-plugin) or
|
||||
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-openclaw-plugin) to add native memory recall, automatic
|
||||
|
|
@ -76,9 +85,6 @@ users retain control of the durable files.
|
|||
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-dsh-plugin) for long-term-memory guidance, `reme_search`,
|
||||
automatic memory, Auto Dream, and ReMe Status.
|
||||
|
||||
<details>
|
||||
<summary>More updates</summary>
|
||||
|
||||
- [2026.08] - **ReMe blog published**: the [ReMe blog](https://reme.agentscope.io/en/reme-blog) introduces the
|
||||
local-first memory architecture, self-evolving workflows, hybrid search, proactive discovery, and benchmark results.
|
||||
- [2026.08] - **New ReMe ecosystem plugins**: [Daily Paper](https://reme.agentscope.io/en/plugins/daily-paper)
|
||||
|
|
|
|||
11
README_ZH.md
11
README_ZH.md
|
|
@ -47,6 +47,10 @@
|
|||
|
||||
## 📰 最新动态
|
||||
|
||||
<!-- 最新动态最多展开 5 条;新增条目时,将超出的旧条目移入下方“更多更新”折叠区。 -->
|
||||
|
||||
- [2026.10] - **[基于决策模型的上下文压缩](docs/zh/context_compaction.md)**:根据下一步任务判断旧工具调用与完整输出的保留价值,
|
||||
保护关键历史,并返回可回溯的压缩建议;提供 SVG 流程图和两个真实模型集成场景。
|
||||
- [2026.10] - **[ReMe 多模态记忆更新](https://reme.agentscope.io/zh/blog_20261009)**:Auto Memory 将会话图片和上下文一起
|
||||
写入记忆并保留图片来源;Auto Resource 将图片资料整理为可检索、可回溯原图的 daily 卡片。
|
||||
- [2026.10] - **[ReMe Studio Playground](https://reme.agentscope.io/studio/?lang=zh) 上线**:无需安装或启动后端,
|
||||
|
|
@ -62,6 +66,10 @@
|
|||
可重建 Tag Index 与标签过滤检索。
|
||||
- [2026.09] - **[Hermes Agent 记忆 Provider](https://reme.agentscope.io/zh/integrations/hermes) 已可使用**:支持 HTTP 和 Embedded
|
||||
两种模式,在模型调用前自动召回、每轮对话结束后异步执行 `auto_memory`。集成支持 Hermes Agent 0.21 及以上版本,后台任务也会继承当前 profile 上下文。
|
||||
|
||||
<details>
|
||||
<summary>更多更新</summary>
|
||||
|
||||
- [2026.09] - **[OpenClaw 插件](https://reme.agentscope.io/zh/integrations/openclaw) 发布**:可通过
|
||||
[ClawHub](https://clawhub.ai/agentscope-ai/plugins/reme-openclaw-plugin) 或
|
||||
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-openclaw-plugin) 安装,为 OpenClaw 提供原生记忆召回、自动对话捕获和定时整理能力。
|
||||
|
|
@ -69,9 +77,6 @@
|
|||
[Awesome DSH Plugin](https://awesome-dsh-plugin.com/p/agentscope-ai/ReMe--integrations-dsh/) 或
|
||||
[npm](https://www.npmjs.com/package/@agentscope-ai/reme-dsh-plugin) 安装,提供长期记忆指引、`reme_search`、自动记忆、Auto Dream 和 ReMe Status。
|
||||
|
||||
<details>
|
||||
<summary>更多更新</summary>
|
||||
|
||||
- [2026.08] - **ReMe 博客发布**:[ReMe 博客](https://reme.agentscope.io/zh/reme-blog) 系统介绍了本地优先的记忆架构、
|
||||
自进化工作流、混合检索、主动发现与评测结果。
|
||||
- [2026.08] - **新增 ReMe 生态插件**:[每日论文](https://reme.agentscope.io/zh/plugins/daily-paper) 可自动发现、解析论文并生成文件化简报;
|
||||
|
|
|
|||
|
|
@ -148,6 +148,7 @@ function docsSidebar(language: "zh" | "en"): DefaultTheme.SidebarItem[] {
|
|||
{ text: "Auto Resource", link: `/${language}/auto_resource` },
|
||||
{ text: "Auto Dream", link: `/${language}/auto_dream` },
|
||||
{ text: "Proactive", link: `/${language}/proactive` },
|
||||
{ text: zh ? "决策模型上下文压缩" : "Context Compaction", link: `/${language}/context_compaction` },
|
||||
],
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -209,6 +209,8 @@ terminal with `messages` and any protection or budget options. It uses the same
|
|||
a proposed snapshot; the caller owns archiving and applying it. Request token budgets are estimates with overhead
|
||||
headroom, independent of the host model tokenizer.
|
||||
|
||||
See [Context compaction with decision models](./context_compaction.md) for the strategy, SVG walkthrough, and live integration cases.
|
||||
|
||||
## Embeddings
|
||||
|
||||
Vector retrieval is disabled by default. Credentials alone do not enable it: configure `as_embedding`, `embedding_store`, and connect the store to `file_store`.
|
||||
|
|
|
|||
167
docs/en/context_compaction.md
Normal file
167
docs/en/context_compaction.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# Context compaction with decision models
|
||||
|
||||
Long tasks accumulate source reads, stale searches, repeated diagnostics, and completed side tasks.
|
||||
`compact_context` uses the **next action** to judge whether an old tool call still matters and whether its full result
|
||||
must remain verbatim. It returns a proposed message snapshot and per-call decisions. The host decides whether to apply it.
|
||||
|
||||
The workflow calls the provider-independent `as_decision_model` interface. The current `agentscope` adapter uses
|
||||
AgentScope's TypeSafe/Jev implementation; jobs keep the same interface when another backend is registered.
|
||||
|
||||

|
||||
|
||||
## How one pass works
|
||||
|
||||
1. **The host archives the original history.** Supply an AgentScope 2.x JSON message snapshot, current `goal`, eligible
|
||||
tool names, and protection lists. ReMe neither archives history nor updates a running Agent's state. Apply proposals
|
||||
only when the original record remains recoverable.
|
||||
2. **Deterministic guards select candidates.** Match `tool_call` and `tool_result` by ID. Calls must be `finished`, results
|
||||
must be `success`, tools must be explicitly allowlisted, and outputs must be entirely text. Duplicate IDs, missing or
|
||||
reversed pairs, and invalid JSON inputs are ineligible.
|
||||
3. **Build a bounded observation.** Send the goal, conversation text, tool names, shortened inputs, and result character
|
||||
counts. Tool-result bodies and thinking are omitted. When necessary, shorten inputs and old text, omit old text-only
|
||||
messages, or merge old call records in this view. These changes do not rewrite conversation text in the returned snapshot.
|
||||
4. **Ask two binary questions per candidate.** `call_<id>` asks whether the call and its input still matter for the next
|
||||
action. `result_<id>` asks whether the full output must remain verbatim and later retrieval would be insufficient.
|
||||
Keep each question pair together; batches repeat the same observation and run under concurrency and timeout limits.
|
||||
5. **Rebuild only after every batch succeeds.** Validate answer types and probabilities before applying decisions to a
|
||||
deep copy. Missing answers, invalid probabilities, failed batches, or timeout fail the call without a partial proposal.
|
||||
The host should continue using its saved original snapshot.
|
||||
|
||||
## From probabilities to actions
|
||||
|
||||
With retention threshold `t` (default `0.5`), check the result probability first:
|
||||
|
||||
| Condition | Action | Proposed snapshot |
|
||||
| --- | --- | --- |
|
||||
| `p_result >= t` | `keep` | Keep the call and its complete result, even if `p_call` is lower |
|
||||
| `p_result < t` and `p_call >= t` | `drop_result` | Keep the call; shorten long text output to a prefix plus an omission marker |
|
||||
| Both probabilities `< t` | `drop_call` | Remove both tool blocks; remove a message only if this makes it empty |
|
||||
|
||||
The default prefix is 300 characters. The marker points back to archived history. Replacement occurs only if it makes
|
||||
output shorter, so `drop_result` can leave a short output unchanged. User text, assistant text, thinking, and untouched
|
||||
block fields remain intact.
|
||||
|
||||
Deterministic protection takes precedence over probabilities:
|
||||
|
||||
- The first message, the most recent `preserve_recent_messages` messages, and `protected_message_ids` are protected.
|
||||
Protection of either endpoint protects the complete pair.
|
||||
- `protected_call_ids` and tools outside `eligible_tool_names` are protected. The default allowlist is empty, so no tools
|
||||
are compacted unless the caller explicitly opts them in.
|
||||
- Unfinished or failed calls, non-text outputs, and mismatched tool names remain intact. Duplicate or missing IDs are
|
||||
also retained, but may have no entry in `decisions`.
|
||||
|
||||
The host must choose read-only tools for the allowlist; ReMe cannot infer tool side effects from their names.
|
||||
Explicitly protect indispensable audit records, change receipts, and evidence; semantic probabilities are insufficient
|
||||
as the sole protection mechanism.
|
||||
|
||||
## Configure and call the service
|
||||
|
||||
Install dependencies and configure credentials:
|
||||
|
||||
```bash
|
||||
pip install "reme-ai[core,decision]"
|
||||
export DECISION_MODEL_API_KEY=your-typesafe-api-key
|
||||
export DECISION_MODEL_BACKEND=agentscope
|
||||
# Optional: DECISION_MODEL_NAME and DECISION_MODEL_BASE_URL
|
||||
```
|
||||
|
||||
The CLI also loads `.env`. Migrate old `CLASSIFIER_*` variables to `DECISION_MODEL_*` as described in
|
||||
[configuration](./configuration.md#decision-model-configuration); the backend name is `agentscope`.
|
||||
Omitting the model name or setting it to `null` uses the adapter's SDK default.
|
||||
|
||||
Start the service in one terminal:
|
||||
|
||||
```bash
|
||||
reme start config=decision_compaction
|
||||
```
|
||||
|
||||
Call it from a second terminal. This short example illustrates the request format; the integration scenarios below
|
||||
provide longer histories:
|
||||
|
||||
```bash
|
||||
reme compact_context \
|
||||
goal='Fix payment retries; the completed icon search is irrelevant' \
|
||||
eligible_tool_names='["search_code"]' \
|
||||
options='{"preserve_recent_messages":1}' \
|
||||
messages='[{"id":"m0","role":"user","content":[{"type":"text","text":"Preserve payment source; clean old icon searches"}]},{"id":"m1","role":"assistant","content":[{"type":"tool_call","id":"c1","name":"search_code","input":"{\"pattern\":\"logo\",\"path\":\"web/icons\"}","state":"finished"},{"type":"tool_result","id":"c1","name":"search_code","output":"Icon paths from a completed task","state":"success"}]},{"id":"m2","role":"user","content":[{"type":"text","text":"Now focus on payment retries"}]}]'
|
||||
```
|
||||
|
||||
`decision_compaction` inherits the default service configuration and other jobs, including normal background workflows.
|
||||
For embedding in a host process, assemble only the compaction job and its decision component as the integration tests do.
|
||||
|
||||
The standard Response's `answer` contains:
|
||||
|
||||
- `schema_version` and `messages`: version and proposed snapshot; `changed` and `reason`: whether it changed, with
|
||||
`compacted`, `no_change`, or `no_candidates`.
|
||||
- `decisions`: internal pair ID, original `call_id`, action, reason, and `keep_call` / `keep_result` probabilities for
|
||||
pairs evaluated by the model.
|
||||
- `stats`: message and character counts before and after, candidate/protected counts, estimated observation tokens and
|
||||
fitting stage, request count, provider usage, and elapsed time. The same statistics appear in `response.metadata.compaction`.
|
||||
|
||||
Check `success`, recalculate the final input budget with the host model's tokenizer, and apply, repeat, or reject the
|
||||
proposal. Compaction does not automatically summarize, write long-term memory, or execute historical tool calls.
|
||||
|
||||
## Budgets, concurrency, and timeout
|
||||
|
||||
| `options` field | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `keep_threshold` | `0.5` | Shared retention threshold for both questions |
|
||||
| `preserve_recent_messages` | `6` | Messages protected verbatim, counted as messages rather than turns |
|
||||
| `max_state_tokens` | `25000` | Estimated observation token limit |
|
||||
| `max_request_tokens` | `30000` | Per-batch estimated observation/question budget, including a 256-token overhead reserve |
|
||||
| `truncate_head_chars` | `300` | Text prefix retained by `drop_result` |
|
||||
| `max_concurrency` | `4` | Concurrent decision requests, from 1 to 32 |
|
||||
| `total_timeout_seconds` | `30` | Overall deadline for decision requests and their queueing |
|
||||
|
||||
Component `timeout`, `max_retries`, and `retry_delay` control individual provider requests and retries. The compaction
|
||||
preset uses 10 seconds, one retry, and 0.5 seconds of initial backoff. Local token estimates differ from provider billing
|
||||
and from the host model tokenizer. If the observation cannot fit, or a single question pair cannot fit a request, the
|
||||
call fails and the host retains its original snapshot.
|
||||
|
||||
## Two live-model integration scenarios
|
||||
|
||||
The [integration tests](https://github.com/agentscope-ai/ReMe/blob/main/tests/integration/test_decision_compaction.py)
|
||||
provide two synthetic long-task histories. Historical deploys, rollbacks, and database queries are fixture data and are
|
||||
never executed; only the decision requests call the real TypeSafe/Jev API.
|
||||
|
||||
| Scenario | Current task and important candidates | Distractors and protection boundaries |
|
||||
| --- | --- | --- |
|
||||
| Payment retry patch | Produce an exact patch from previously read source and regression-test lines; keep those candidate results intact | Completed theme CSS, icon searches, and obsolete inventory include removable calls; preserve billing evidence, approval receipts, failed/pending work, duplicate IDs, and recent messages |
|
||||
| Replication incident | Quote exact incident logs and WAL offsets in a root-cause report; keep the verbatim evidence intact | Mix in completed marketing reports, holiday weather, and retired-system load tests; preserve audit approval, rollback receipts, non-text charts, failed queries, and recent messages |
|
||||
|
||||
Both cases check real batched inference, removal of at least one irrelevant call, full retention of critical candidates,
|
||||
protected history and conversation text, unchanged original input and temporary archive, and a reduction in output
|
||||
characters. Semantic assertions can expose model/configuration changes; the tests do not fabricate probabilities to pass.
|
||||
|
||||
A live run on 2026-10-15 against the locally configured TypeSafe/Jev endpoint passed both cases. These are JSON
|
||||
character counts, not token savings:
|
||||
|
||||
| Scenario | Characters before | Characters after | Reduction | Irrelevant calls removed | Decision requests |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Payment retry patch | 39,438 | 12,738 | 67.7% | 3 | 2 |
|
||||
| Replication incident | 51,485 | 12,828 | 75.1% | 4 | 2 |
|
||||
|
||||
Explicitly opt in to live API calls:
|
||||
|
||||
```bash
|
||||
REME_RUN_DECISION_INTEGRATION=1 \
|
||||
pytest tests/integration/test_decision_compaction.py -v -s
|
||||
```
|
||||
|
||||
Tests read the environment or repository-root `.env`, with existing environment variables taking precedence. They skip
|
||||
without opt-in or without an API key. Output contains only synthetic-scenario actions, probabilities, character counts,
|
||||
and usage. Workspaces and original snapshots use pytest temporary directories. Savings depend on history, goal, and
|
||||
model decisions; these two scenarios do not establish a general performance guarantee.
|
||||
|
||||
## Boundaries
|
||||
|
||||
The decision model sees a limited observation and cannot inspect omitted tool-result contents. Incomplete goals or
|
||||
insufficient tool descriptions can hide relevant evidence. The host must supply a clear next-action goal, explicitly
|
||||
protect indispensable information, and retain recoverable original history after applying a proposal.
|
||||
This strategy reduces old tool traces; it does not provide automatic triggering, a final token-budget guarantee,
|
||||
summary generation, or archival storage.
|
||||
|
||||
Contracts and implementation: [compaction schema](https://github.com/agentscope-ai/ReMe/blob/main/reme/schema/compaction.py),
|
||||
[pairing and rebuilding](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/messages.py),
|
||||
[observation and batching](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/state.py), and
|
||||
[job step](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/compact_context.py).
|
||||
64
docs/figure/decision-context-compaction.svg
Normal file
64
docs/figure/decision-context-compaction.svg
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="1080" viewBox="0 0 1280 1080" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Decision-model context compaction</title>
|
||||
<desc id="desc">The caller archives original history. Deterministic guards protect messages and unsafe tool pairs. A bounded observation without tool outputs goes to a decision model, which estimates two probabilities per eligible pair. The result is a proposed snapshot with unchanged protected history; the caller checks its own token budget before applying it.</desc>
|
||||
<defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#516784"/></marker></defs>
|
||||
<style>text{font-family:Inter,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;fill:#24364b}.title{font-size:31px;font-weight:750}.sub{font-size:17px;fill:#526b84}.heading{font-size:21px;font-weight:700}.body{font-size:17px}.small{font-size:15px;fill:#526b84}.mono{font-size:18px;font-family:ui-monospace,SFMono-Regular,Consolas,monospace}.tag{font-size:14px;font-weight:700;letter-spacing:1px}.line{fill:none;stroke:#516784;stroke-width:2.5;marker-end:url(#arrow)}.bypass{fill:none;stroke:#7787a0;stroke-width:2;stroke-dasharray:7 6;marker-end:url(#arrow)}</style>
|
||||
<rect width="1280" height="1080" rx="20" fill="#f4f7fb"/>
|
||||
<text x="40" y="58" class="title">Decision-model context compaction</text>
|
||||
<text x="40" y="91" class="sub">Score old tool history for the next action. Return a proposal; the caller owns the original.</text>
|
||||
<rect x="40" y="140" width="350" height="210" rx="15" fill="#fff" stroke="#d3ddea" stroke-width="1.5"/>
|
||||
<rect x="465" y="140" width="350" height="210" rx="15" fill="#fff" stroke="#d3ddea" stroke-width="1.5"/>
|
||||
<rect x="890" y="140" width="350" height="210" rx="15" fill="#fff" stroke="#d3ddea" stroke-width="1.5"/>
|
||||
<text x="62" y="173" class="tag">1 ORIGINAL SNAPSHOT</text>
|
||||
<text x="62" y="207" class="heading">Caller archives full history</text>
|
||||
<text x="62" y="241" class="body">User / assistant / system messages</text>
|
||||
<text x="62" y="271" class="body">Tool calls + full tool results</text>
|
||||
<text x="62" y="309" class="small">Text, thinking and metadata retained</text>
|
||||
<text x="487" y="173" class="tag">2 DETERMINISTIC GUARDS</text>
|
||||
<text x="487" y="207" class="heading">Choose eligible pairs</text>
|
||||
<text x="487" y="241" class="body">Read-only tool allowlist</text>
|
||||
<text x="487" y="270" class="body">Unique IDs + finished + success</text>
|
||||
<text x="487" y="302" class="small">Protect first / recent / pinned history</text>
|
||||
<text x="912" y="173" class="tag">3 BOUNDED OBSERVATION</text>
|
||||
<text x="912" y="207" class="heading">Send a small decision view</text>
|
||||
<text x="912" y="242" class="body">Goal + text + tool names / inputs</text>
|
||||
<text x="912" y="271" class="body">Result sizes; tool outputs omitted</text>
|
||||
<text x="912" y="309" class="small">Thinking is omitted; inputs may shrink</text>
|
||||
<path d="M 390 245 H 456" class="line"/>
|
||||
<path d="M 815 245 H 881" class="line"/>
|
||||
<rect x="420" y="435" width="440" height="185" rx="15" fill="#eaf2ff" stroke="#9db9e6" stroke-width="1.5"/>
|
||||
<text x="444" y="469" class="tag">4 DECISION MODEL</text>
|
||||
<text x="444" y="505" class="heading">Two binary questions per tool pair</text>
|
||||
<text x="444" y="542" class="mono">p_call Keep call + input?</text>
|
||||
<text x="444" y="578" class="mono">p_result Keep full output verbatim?</text>
|
||||
<path d="M 1065 350 V 397 H 640 V 426" class="line"/>
|
||||
<path d="M 640 350 V 377 H 20 V 858 H 265 V 902" class="bypass"/>
|
||||
<rect x="40" y="425" width="318" height="100" rx="15" fill="#f4f7fb" stroke="#d3ddea" stroke-width="1.5"/>
|
||||
<text x="60" y="459" class="small">Protected pairs bypass scoring</text>
|
||||
<text x="60" y="490" class="small">Their original blocks stay unchanged.</text>
|
||||
<text x="868" y="475" class="small">Same observation in each batch</text>
|
||||
<text x="868" y="506" class="small">Bounded concurrency + deadline</text>
|
||||
<text x="868" y="537" class="small">Any failed batch → no proposal</text>
|
||||
<path d="M 640 620 V 650 H 225 V 683" class="line"/>
|
||||
<path d="M 640 650 V 683" class="line"/>
|
||||
<path d="M 640 650 H 1055 V 683" class="line"/>
|
||||
<rect x="40" y="693" width="370" height="145" rx="15" fill="#e8f6ee" stroke="#99c9ac" stroke-width="1.5"/>
|
||||
<rect x="455" y="693" width="370" height="145" rx="15" fill="#fff4df" stroke="#e5c58b" stroke-width="1.5"/>
|
||||
<rect x="870" y="693" width="370" height="145" rx="15" fill="#f1edfa" stroke="#c5b8df" stroke-width="1.5"/>
|
||||
<text x="62" y="729" class="tag">KEEP</text>
|
||||
<text x="62" y="763" class="mono">p_result ≥ threshold</text>
|
||||
<text x="62" y="800" class="body">Retain the call and full result.</text>
|
||||
<text x="477" y="729" class="tag">DROP_RESULT</text>
|
||||
<text x="477" y="763" class="mono">p_call ≥ t and p_result < t</text>
|
||||
<text x="477" y="800" class="body">Keep call; shorten long text output.</text>
|
||||
<text x="892" y="729" class="tag">DROP_CALL</text>
|
||||
<text x="892" y="763" class="body">Both probabilities < threshold</text>
|
||||
<text x="892" y="800" class="body">Remove both tool blocks together.</text>
|
||||
<path d="M 225 838 V 872 H 640 V 902" class="line"/>
|
||||
<path d="M 640 838 V 902" class="line"/>
|
||||
<path d="M 1055 838 V 872 H 640" class="line"/>
|
||||
<rect x="240" y="912" width="800" height="126" rx="15" fill="#fff" stroke="#9db9e6" stroke-width="1.5"/>
|
||||
<text x="264" y="948" class="tag">5 PROPOSED SNAPSHOT + DECISIONS + STATS</text>
|
||||
<text x="264" y="981" class="body">Original text / thinking / protected blocks remain intact.</text>
|
||||
<text x="264" y="1011" class="small">Caller checks its own model token budget, then applies or rejects the proposal.</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.6 KiB |
|
|
@ -206,6 +206,8 @@ schema 定义问题与答案。接口见 `reme/components/as_decision_model/`。
|
|||
`reme compact_context`,传入 `messages` 以及需要的保护和预算选项。返回值是建议的快照,归档和应用由调用方负责。
|
||||
请求 token 预算使用预留开销的估算值,与宿主模型的 tokenizer 独立。
|
||||
|
||||
完整策略、SVG 图解及真实模型用例见[基于决策模型的上下文压缩](./context_compaction.md)。
|
||||
|
||||
## Embedding 配置
|
||||
|
||||
向量检索默认关闭。只设置 `EMBEDDING_API_KEY` 不会自动启用它;还需要同时启用 `as_embedding`、`embedding_store`,并把它连接到 `file_store`:
|
||||
|
|
|
|||
147
docs/zh/context_compaction.md
Normal file
147
docs/zh/context_compaction.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# 基于决策模型的上下文压缩
|
||||
|
||||
长任务会积累大量工具调用:读过的源码、过期的搜索结果、重复查询的日志、已经结束的支线任务。
|
||||
`compact_context` 根据**下一步要做什么**,判断旧工具调用是否仍然有用,以及它的完整输出是否必须保留。
|
||||
它返回一份建议的消息快照和逐调用决策,由宿主决定是否应用。
|
||||
|
||||
这一流程通过 `as_decision_model` 接口调用概率决策模型。当前 `agentscope` 适配器使用 AgentScope 的
|
||||
TypeSafe/Jev 实现;替换供应商时,Job 和压缩策略仍然使用同一个接口。
|
||||
|
||||

|
||||
|
||||
## 一次压缩如何工作
|
||||
|
||||
1. **宿主先归档原始历史。** 宿主提供 AgentScope 2.x 的 JSON 消息快照、当前 `goal`、允许处理的工具名和保护列表。
|
||||
ReMe 不负责归档,也不会替宿主更新运行中的 Agent 状态。只有原始记录可恢复时,宿主才应应用删减建议。
|
||||
2. **用确定性规则挑选候选。** 按调用 ID 配对 `tool_call` 与 `tool_result`。调用必须为 `finished`,结果必须为
|
||||
`success`,工具必须在显式允许列表中,输出必须全部是文本。重复 ID、缺失配对、反向配对和非法 JSON 输入不会参与决策。
|
||||
3. **建立有限的观测视图。** 发送当前目标、对话文本、工具名、截短后的输入和结果字符数。工具输出正文与 thinking
|
||||
不进入决策请求。观测超预算时,逐步缩短输入和旧文本、移除旧的纯文本消息或合并旧调用记录;返回的会话原文不因此改变。
|
||||
4. **每个候选问两个二分类问题。** `call_<id>` 判断“调用及其输入对下一步是否仍然重要”,`result_<id>` 判断
|
||||
“完整输出是否必须原样保留,之后再检索是否不够”。问题成对分批,每批携带相同观测视图,在并发和总超时约束内执行。
|
||||
5. **全部批次成功后重建快照。** 校验每个答案的类型和概率范围,再在消息的深拷贝上应用决策。缺少答案、非法概率、
|
||||
任意批次失败或超时都会使调用失败,不返回部分压缩结果。宿主应继续使用已保存的原始快照。
|
||||
|
||||
## 两个概率如何变成动作
|
||||
|
||||
设保留阈值为 `t`,默认 `0.5`。先判断结果,再判断调用:
|
||||
|
||||
| 条件 | 动作 | 对返回快照的影响 |
|
||||
| --- | --- | --- |
|
||||
| `p_result >= t` | `keep` | 保留调用及完整结果,即使 `p_call` 较低 |
|
||||
| `p_result < t` 且 `p_call >= t` | `drop_result` | 保留调用,将长文本结果替换为前缀和省略标记 |
|
||||
| 两个概率都 `< t` | `drop_call` | 同时移除调用和结果;仅移除因此变空的消息 |
|
||||
|
||||
`drop_result` 的默认前缀为 300 个字符,标记提示从归档历史恢复全文。只有新文本更短时才替换,所以该动作对短输出可能不产生变化。
|
||||
用户文本、assistant 文本、thinking 和未被修改的块字段保留原样。
|
||||
|
||||
以下内容优先受规则保护,不受概率阈值影响:
|
||||
|
||||
- 第一条消息、最近 `preserve_recent_messages` 条消息,以及 `protected_message_ids` 指定的消息。配对的任一端受保护,整对保留。
|
||||
- `protected_call_ids` 指定的调用,以及不在 `eligible_tool_names` 中的工具。默认允许列表为空,因此默认不会压缩任何工具。
|
||||
- 未完成或失败的调用、非纯文本输出、工具名不一致的配对。重复或缺失 ID 的块也不会删除,但不一定出现在 `decisions` 中。
|
||||
|
||||
允许列表应由宿主限定为安全的只读工具,ReMe 不会根据工具名自动识别副作用。
|
||||
对于必须保留的审计记录、变更回执或关键证据,宿主应显式保护,不能只依赖语义概率。
|
||||
|
||||
## 配置与服务调用
|
||||
|
||||
安装并配置凭证:
|
||||
|
||||
```bash
|
||||
pip install "reme-ai[core,decision]"
|
||||
export DECISION_MODEL_API_KEY=your-typesafe-api-key
|
||||
export DECISION_MODEL_BACKEND=agentscope
|
||||
# 可选:DECISION_MODEL_NAME、DECISION_MODEL_BASE_URL
|
||||
```
|
||||
|
||||
`.env` 也可以配置这些变量。旧的 `CLASSIFIER_*` 名称需按[配置迁移说明](./configuration.md#决策模型配置)改为
|
||||
`DECISION_MODEL_*`;backend 应为 `agentscope`。省略模型名或配置为 `null` 时使用适配器的 SDK 默认模型。
|
||||
|
||||
在一个终端启动服务:
|
||||
|
||||
```bash
|
||||
reme start config=decision_compaction
|
||||
```
|
||||
|
||||
在另一个终端调用。以下短示例展示请求格式;复杂样例见本文后面的集成用例:
|
||||
|
||||
```bash
|
||||
reme compact_context \
|
||||
goal='修复支付重试逻辑,之前完成的图标搜索已无关' \
|
||||
eligible_tool_names='["search_code"]' \
|
||||
options='{"preserve_recent_messages":1}' \
|
||||
messages='[{"id":"m0","role":"user","content":[{"type":"text","text":"保留支付代码,清理旧图标搜索"}]},{"id":"m1","role":"assistant","content":[{"type":"tool_call","id":"c1","name":"search_code","input":"{\"pattern\":\"logo\",\"path\":\"web/icons\"}","state":"finished"},{"type":"tool_result","id":"c1","name":"search_code","output":"旧任务的图标路径列表","state":"success"}]},{"id":"m2","role":"user","content":[{"type":"text","text":"现在只处理支付重试"}]}]'
|
||||
```
|
||||
|
||||
`decision_compaction` 继承默认服务配置和其他 Job,因此常规后台任务也会启动。
|
||||
需要嵌入宿主进程时,可像集成测试一样只装配 `compact_context` Job 和它的决策组件,不启动其他工作流。
|
||||
|
||||
标准 Response 的 `answer` 包含:
|
||||
|
||||
- `schema_version`、`messages`:版本与建议快照;`changed`、`reason`:是否实际改变,以及 `compacted` / `no_change` / `no_candidates`。
|
||||
- `decisions`:内部配对 ID、原始 `call_id`、动作、原因,以及参与推理时的 `keep_call` / `keep_result` 概率。
|
||||
- `stats`:前后消息数和字符数、候选与保护配对数量、观测视图的估算 token 数及缩减阶段、请求数、供应商 usage 和耗时。
|
||||
同一份统计也位于 `response.metadata.compaction`。
|
||||
|
||||
宿主应检查 `success`,使用自己的 tokenizer 重新计算最终上下文预算,再选择应用建议、继续压缩或恢复原始记录。
|
||||
压缩不会自动生成摘要、写入长期记忆或执行历史中的工具。
|
||||
|
||||
## 预算、并发和超时
|
||||
|
||||
| `options` 字段 | 默认值 | 作用 |
|
||||
| --- | --- | --- |
|
||||
| `keep_threshold` | `0.5` | 两个问题共用的保留阈值 |
|
||||
| `preserve_recent_messages` | `6` | 原样保护最近的消息,按消息而非轮次数计数 |
|
||||
| `max_state_tokens` | `25000` | 观测视图的估算 token 上限 |
|
||||
| `max_request_tokens` | `30000` | 每批观测与问题的估算 token 预算,包含 256 token 的开销预留 |
|
||||
| `truncate_head_chars` | `300` | `drop_result` 保留的文本前缀长度 |
|
||||
| `max_concurrency` | `4` | 同时运行的决策请求上限,范围 1–32 |
|
||||
| `total_timeout_seconds` | `30` | 所有决策请求及其排队等待的总超时 |
|
||||
|
||||
组件层的 `timeout`、`max_retries`、`retry_delay` 控制单次供应商请求及重试。
|
||||
内置压缩配置分别为 10 秒、1 次重试、0.5 秒初始退避。预算使用本地估算器,不等于供应商计费 token,也不等于宿主模型的上下文预算。
|
||||
观测无法装入预算,或一对问题也放不进单批请求时,调用失败,由宿主保留原始快照。
|
||||
|
||||
## 两个真实模型集成场景
|
||||
|
||||
[集成测试源码](https://github.com/agentscope-ai/ReMe/blob/main/tests/integration/test_decision_compaction.py)
|
||||
提供两个合成的长任务场景。工具调用及其返回值均为测试数据,历史中的部署、回滚、数据库查询不会实际执行;
|
||||
只有决策请求使用真实的 TypeSafe/Jev API。
|
||||
|
||||
| 场景 | 当前任务与关键候选 | 干扰与保护边界 |
|
||||
| --- | --- | --- |
|
||||
| 支付重试缺陷修复 | 下一步根据已读源码和测试原文生成精确补丁,验证相关源码与测试结果被保留 | 已完成的主题 CSS、图标搜索、过期库存查询应有可淘汰项;显式保护账单证据与审批回执,保留失败、运行中、重复 ID 和最近消息 |
|
||||
| 数据库复制延迟排障 | 下一步引用事件日志及 WAL 位点原文编写根因报告,验证关键日志与数据行被保留 | 混入历史营销报表、旅行天气和退役系统压测;保护审计审批、回滚回执、非文本图表、失败查询和最近消息 |
|
||||
|
||||
两个用例都检查:真实推理按批执行、至少移除一个无关调用、关键候选完整保留、受保护记录和会话文本不变、
|
||||
原始输入与临时归档不变,以及输出字符数实际下降。模型具有不确定性,语义断言可能揭示模型或配置的变化;测试不会伪造概率来保证通过。
|
||||
|
||||
2026-10-15 使用本地配置的 TypeSafe/Jev 端点实测,两例均通过,结果如下。数字按 JSON 字符数统计,不是 token 节省率:
|
||||
|
||||
| 场景 | 压缩前字符数 | 压缩后字符数 | 减少比例 | 被移除的无关调用 | 决策请求数 |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 支付重试缺陷修复 | 39,438 | 12,738 | 67.7% | 3 | 2 |
|
||||
| 数据库复制延迟排障 | 51,485 | 12,828 | 75.1% | 4 | 2 |
|
||||
|
||||
显式授权真实 API 调用后运行:
|
||||
|
||||
```bash
|
||||
REME_RUN_DECISION_INTEGRATION=1 \
|
||||
pytest tests/integration/test_decision_compaction.py -v -s
|
||||
```
|
||||
|
||||
测试读取环境变量或仓库根目录 `.env`,环境变量优先;未显式启用时直接跳过,未配置 API key 时也跳过。
|
||||
输出仅包含合成场景的动作、概率、字符数和 usage;工作区及原始快照放在 pytest 的临时目录中。
|
||||
实际节省比例依赖任务历史、目标和模型判断,不能由这两个用例推导通用收益。
|
||||
|
||||
## 适用边界
|
||||
|
||||
决策模型只能看到有限观测视图,无法核查被省略的工具输出内容。目标写得不完整或工具描述不足时,判断可能遗漏重要信息。
|
||||
宿主需要给出明确的下一步目标,为不可丢失的信息设置保护,并在应用后继续保留可恢复的原始历史。
|
||||
该策略适合减少旧工具轨迹的体积;它不提供自动触发、最终 token 预算保证、摘要生成或归档存储。
|
||||
|
||||
接口与实现分别见 [Compaction schema](https://github.com/agentscope-ai/ReMe/blob/main/reme/schema/compaction.py)、
|
||||
[候选配对与重建](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/messages.py)、
|
||||
[观测与分批](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/state.py) 和
|
||||
[Job Step](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/compaction/compact_context.py)。
|
||||
|
|
@ -7,7 +7,8 @@ LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
|||
# Setting these variables alone does not enable an as_decision_model component.
|
||||
# DECISION_MODEL_API_KEY=your-typesafe-api-key
|
||||
# DECISION_MODEL_BACKEND=agentscope
|
||||
# DECISION_MODEL_NAME=null # Use the adapter default model.
|
||||
# DECISION_MODEL_NAME=null
|
||||
# Omit the model name or use null for the adapter default.
|
||||
# DECISION_MODEL_BASE_URL=null
|
||||
|
||||
# Optional data-source mirrors. Unset variables use the official services.
|
||||
|
|
|
|||
452
tests/integration/test_decision_compaction.py
Normal file
452
tests/integration/test_decision_compaction.py
Normal file
|
|
@ -0,0 +1,452 @@
|
|||
"""Two live decision-model scenarios; explicitly opt in to paid API calls.
|
||||
|
||||
REME_RUN_DECISION_INTEGRATION=1 pytest tests/integration/test_decision_compaction.py -v -s
|
||||
Credentials come from DECISION_MODEL_* in the environment or the repository .env.
|
||||
Only synthetic history is sent. No tools in these histories are actually executed.
|
||||
"""
|
||||
|
||||
# pylint: disable=redefined-outer-name
|
||||
|
||||
import asyncio
|
||||
from collections import Counter
|
||||
from copy import deepcopy
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from reme import Application
|
||||
from reme.config import resolve_app_config
|
||||
from reme.schema.compaction import CompactionRequest
|
||||
from reme.steps.compaction.messages import collect_pairs
|
||||
from reme.steps.compaction.state import estimate_decision_tokens, fit_state
|
||||
from reme.utils.env_utils import load_env
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
os.environ.get("REME_RUN_DECISION_INTEGRATION") != "1",
|
||||
reason="Set REME_RUN_DECISION_INTEGRATION=1 to authorize live decision API calls.",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def live_config():
|
||||
"""Load credentials only after opt-in; never print or persist them."""
|
||||
load_env(Path(__file__).resolve().parents[2] / ".env", override=False)
|
||||
if not os.environ.get("DECISION_MODEL_API_KEY", "").strip():
|
||||
pytest.skip("DECISION_MODEL_API_KEY is required for live decisions.")
|
||||
pytest.importorskip("agentscope.classifier")
|
||||
config = resolve_app_config(config="decision_compaction", log_config=False)
|
||||
component = config["components"]["as_decision_model"]["compaction"]
|
||||
component["timeout"] = 30
|
||||
return {
|
||||
"service": {"backend": "cli"},
|
||||
"jobs": {"compact_context": config["jobs"]["compact_context"]},
|
||||
"components": {"as_decision_model": {"compaction": component}},
|
||||
"enable_logo": False,
|
||||
"log_to_console": False,
|
||||
"log_to_file": False,
|
||||
}
|
||||
|
||||
|
||||
def _rows(prefix, count=100):
|
||||
"""Synthetic, reproducible tool output; no production or personal data."""
|
||||
return "\n".join(f"{prefix} row={i:04d} status=ok sample=synthetic trace=sample-{i:04d}" for i in range(count))
|
||||
|
||||
|
||||
def _pair(history, call_id, tool, arguments, output, note, *, result_state="success", split=False):
|
||||
"""Include realistic metadata and optionally span two messages."""
|
||||
call = {
|
||||
"type": "tool_call",
|
||||
"id": call_id,
|
||||
"name": tool,
|
||||
"input": json.dumps(arguments),
|
||||
"state": "finished",
|
||||
}
|
||||
result = {
|
||||
"type": "tool_result",
|
||||
"id": call_id,
|
||||
"name": tool,
|
||||
"output": output,
|
||||
"state": result_state,
|
||||
"metadata": {"source": "synthetic-fixture"},
|
||||
}
|
||||
history.append(
|
||||
{
|
||||
"id": call_id + "-call",
|
||||
"role": "assistant",
|
||||
"metadata": {"fixture": True},
|
||||
"content": [{"type": "text", "text": note}, call] + ([] if split else [result]),
|
||||
},
|
||||
)
|
||||
if split:
|
||||
history.append({"id": call_id + "-result", "role": "assistant", "content": [result]})
|
||||
|
||||
|
||||
def _blocks(messages, block_type):
|
||||
return [block for message in messages for block in message["content"] if block["type"] == block_type]
|
||||
|
||||
|
||||
async def _run_case(
|
||||
config,
|
||||
tmp_path,
|
||||
*,
|
||||
name,
|
||||
history,
|
||||
goal,
|
||||
eligible,
|
||||
protected_calls,
|
||||
protected_messages,
|
||||
must_keep,
|
||||
irrelevant,
|
||||
):
|
||||
original = deepcopy(history)
|
||||
archive = tmp_path / "original-history.json"
|
||||
archive.write_text(json.dumps(history, ensure_ascii=False), encoding="utf-8")
|
||||
archived_bytes = archive.read_bytes()
|
||||
request = CompactionRequest(
|
||||
messages=history,
|
||||
goal=goal,
|
||||
eligible_tool_names=eligible,
|
||||
protected_call_ids=protected_calls,
|
||||
protected_message_ids=protected_messages,
|
||||
options={"preserve_recent_messages": 2, "max_state_tokens": 6000},
|
||||
)
|
||||
pairs = collect_pairs(request)
|
||||
state, _ = fit_state(request, pairs)
|
||||
# Fit a few question pairs per request, exercising repeated state and concurrency.
|
||||
options = request.options.model_dump()
|
||||
options.update(
|
||||
max_request_tokens=estimate_decision_tokens({"state": state, "questions": {}}) + 750,
|
||||
max_concurrency=2,
|
||||
total_timeout_seconds=90,
|
||||
)
|
||||
async with Application(**config, workspace_dir=str(tmp_path / "workspace")) as app:
|
||||
response = await app.run_job("compact_context", **request.model_dump(exclude={"options"}), options=options)
|
||||
assert response.success, "Live compact_context failed; inspect the configured endpoint and SDK."
|
||||
result = response.answer
|
||||
assert history == original
|
||||
assert archive.read_bytes() == archived_bytes
|
||||
assert _blocks(result["messages"], "text") == _blocks(original, "text")
|
||||
decisions = {item["call_id"]: item for item in result["decisions"]}
|
||||
for call_id in must_keep:
|
||||
assert decisions[call_id]["action"] == "keep", f"Current verbatim evidence was lost: {call_id}"
|
||||
assert any(decisions[call_id]["action"] == "drop_call" for call_id in irrelevant)
|
||||
remaining = {message["id"]: message for message in result["messages"]}
|
||||
protected_ids = {original[0]["id"], *(message["id"] for message in original[-2:]), *protected_messages}
|
||||
for message in original:
|
||||
if message["id"] in protected_ids:
|
||||
assert remaining[message["id"]] == message
|
||||
for pair in pairs:
|
||||
if pair.protected:
|
||||
for mi, bi in (pair.call_location, pair.result_location):
|
||||
message = original[mi]
|
||||
assert message["content"][bi] in remaining[message["id"]]["content"]
|
||||
original_tools = [
|
||||
block for message in original for block in message["content"] if block["type"] in {"tool_call", "tool_result"}
|
||||
]
|
||||
proposed_tools = [
|
||||
block
|
||||
for message in result["messages"]
|
||||
for block in message["content"]
|
||||
if block["type"] in {"tool_call", "tool_result"}
|
||||
]
|
||||
untouched = {
|
||||
block["id"]
|
||||
for block in original_tools
|
||||
if block["id"] not in decisions or decisions[block["id"]]["reason"] == "protected"
|
||||
}
|
||||
for call_id in untouched:
|
||||
assert [block for block in proposed_tools if block["id"] == call_id] == [
|
||||
block for block in original_tools if block["id"] == call_id
|
||||
]
|
||||
for item in decisions.values():
|
||||
if item["reason"] != "protected":
|
||||
assert 0 <= item["keep_call"] <= 1 and 0 <= item["keep_result"] <= 1
|
||||
stats = result["stats"]
|
||||
assert result["changed"] and stats["chars_after"] < stats["chars_before"]
|
||||
assert stats["requests"] >= 2
|
||||
assert stats["usage"], "The live provider should report token usage."
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"case": name,
|
||||
"actions": dict(Counter(item["action"] for item in decisions.values())),
|
||||
"chars_before": stats["chars_before"],
|
||||
"chars_after": stats["chars_after"],
|
||||
"saved_percent": round(100 * (1 - stats["chars_after"] / stats["chars_before"]), 1),
|
||||
"requests": stats["requests"],
|
||||
"usage": stats["usage"],
|
||||
"decisions": {
|
||||
key: {field: value[field] for field in ("action", "keep_call", "keep_result")}
|
||||
for key, value in decisions.items()
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_payment_retry_patch_keeps_source_and_removes_completed_ui_task(live_config, tmp_path):
|
||||
"""Exact next-edit evidence competes with large, completed unrelated work."""
|
||||
history = [
|
||||
{
|
||||
"id": "first",
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Earlier we finished a storefront theme task. "
|
||||
"Now fix duplicate charges after payment retries. "
|
||||
"Keep approval receipts intact; do not deploy anything.",
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
_pair(
|
||||
history,
|
||||
"old_css",
|
||||
"read_file",
|
||||
{"path": "web/theme/archive.css"},
|
||||
_rows("unused-theme"),
|
||||
"The storefront theme task was completed and accepted. This CSS has no payment logic.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"old_icons",
|
||||
"search_code",
|
||||
{"pattern": "logo", "path": "web/icons"},
|
||||
_rows("icon-index"),
|
||||
"Historical icon search from the finished UI task; unrelated to the current retry defect.",
|
||||
split=True,
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"retry_source",
|
||||
"read_file",
|
||||
{"path": "src/payments/retry.py"},
|
||||
"def retry_charge(payment, gateway):\n for attempt in range(3):\n"
|
||||
" try:\n return gateway.charge(payment)\n"
|
||||
" except TimeoutError:\n continue\n" + _rows("source-context", 25),
|
||||
"Current retry source. The next edit must match its exact original lines; no further reads before patching.",
|
||||
split=True,
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"retry_tests",
|
||||
"read_file",
|
||||
{"path": "tests/payments/test_retry.py"},
|
||||
'def test_timeout_reuses_idempotency_key():\n assert keys == ["payment-42"] * 3\n' + _rows("tests", 30),
|
||||
"Current regression-test file. Keep its original lines to place the next patch correctly.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"obsolete_inventory",
|
||||
"query_database",
|
||||
{"table": "catalog.old_inventory", "month": "2024-01"},
|
||||
_rows("old-inventory", 180),
|
||||
"Completed inventory task; these obsolete SKU rows cannot explain payment retries.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"billing_receipt",
|
||||
"query_database",
|
||||
{"table": "billing.events", "payment": "synthetic-42"},
|
||||
_rows("billing-evidence", 35),
|
||||
"Explicitly pinned billing evidence for the incident report.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"approval",
|
||||
"deploy_release",
|
||||
{"release": "synthetic-old-release"},
|
||||
"approved by fixture operator",
|
||||
"Historical approval receipt: a side-effecting tool, never eligible for removal.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"failed_probe",
|
||||
"query_database",
|
||||
{"table": "unavailable_replica"},
|
||||
"replica unreachable",
|
||||
"Failed diagnostic attempt; preserve the failure record.",
|
||||
result_state="error",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"pending_probe",
|
||||
"query_database",
|
||||
{"table": "billing.pending"},
|
||||
"still running",
|
||||
"Unfinished work is protected.",
|
||||
result_state="running",
|
||||
)
|
||||
for _ in range(2):
|
||||
_pair(
|
||||
history,
|
||||
"ambiguous",
|
||||
"read_file",
|
||||
{"path": "unknown-snapshot"},
|
||||
"ambiguous duplicate identifier",
|
||||
"Duplicate call identifiers are intentionally ineligible.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"recent_manifest",
|
||||
"read_file",
|
||||
{"path": "pyproject.toml"},
|
||||
_rows("manifest", 15),
|
||||
"Most recent tool exchange; retain it by the recency rule.",
|
||||
)
|
||||
goal = (
|
||||
"Next action: produce an exact minimal patch to src/payments/retry.py and its regression test, fixing "
|
||||
"idempotency-key reuse after timeouts. You must use the verbatim source and test lines already read, "
|
||||
"without another file read before the patch. The completed storefront theme and obsolete inventory "
|
||||
"tasks are irrelevant. Do not deploy. Preserve pinned billing evidence and all approval receipts."
|
||||
)
|
||||
history.append({"id": "last", "role": "user", "content": [{"type": "text", "text": goal}]})
|
||||
asyncio.run(
|
||||
_run_case(
|
||||
live_config,
|
||||
tmp_path,
|
||||
name="payment_retry_patch",
|
||||
history=history,
|
||||
goal=goal,
|
||||
eligible=["read_file", "search_code", "query_database"],
|
||||
protected_calls=["billing_receipt"],
|
||||
protected_messages=["approval-call"],
|
||||
must_keep=["retry_source", "retry_tests"],
|
||||
irrelevant=["old_css", "old_icons", "obsolete_inventory"],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_replication_incident_keeps_verbatim_evidence_and_removes_unrelated_reports(live_config, tmp_path):
|
||||
"""A forensic report needs exact event records amid mixed operational history."""
|
||||
history = [
|
||||
{
|
||||
"id": "first",
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Investigate replica lag after the synthetic region-A failover. "
|
||||
"Preserve audit evidence and do not roll back.",
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
_pair(
|
||||
history,
|
||||
"old_campaign",
|
||||
"query_database",
|
||||
{"table": "marketing.completed_campaign", "year": 2023},
|
||||
_rows("completed-ad-campaign", 180),
|
||||
"Finished marketing report; no relation to this database incident.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"old_weather",
|
||||
"read_file",
|
||||
{"path": "personal/completed-trip-weather.txt"},
|
||||
_rows("weather", 100),
|
||||
"Historical holiday planning from a completed task; entirely unrelated to database replication.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"old_loadtest",
|
||||
"query_metrics",
|
||||
{"region": "retired-region-Z", "period": "2021-load-test"},
|
||||
_rows("retired-system", 140),
|
||||
"Old load test of a decommissioned system, not the production replica.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"incident_logs",
|
||||
"query_logs",
|
||||
{"region": "region-A", "window": "10:00-10:05", "service": "wal"},
|
||||
"10:00:01 event=F001 leader_switch term=17\n10:00:04 event=W002 apply_paused slot=replica-a\n"
|
||||
"10:04:59 event=L003 lag_bytes=67108864\n" + _rows("wal-log-context", 50),
|
||||
"Current incident log records. The next report must quote the exact event IDs, timestamps and lines; "
|
||||
"summaries or later retrieval cannot replace these records.",
|
||||
split=True,
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"wal_offsets",
|
||||
"query_database",
|
||||
{"table": "replication_slots", "region": "region-A"},
|
||||
"slot=replica-a restart_lsn=0/7000000 confirmed_lsn=0/3000000 active=false\n" + _rows("slot-evidence", 40),
|
||||
"Current WAL-offset evidence. Keep the exact row for a verbatim evidence table in the next answer.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"old_costs",
|
||||
"query_metrics",
|
||||
{"dashboard": "finished-2022-advertising-budget"},
|
||||
_rows("ad-cost", 120),
|
||||
"Completed advertising cost dashboard; unrelated to region-A failover or WAL offsets.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"audit_approval",
|
||||
"read_file",
|
||||
{"path": "audit/synthetic-change-approval.txt"},
|
||||
"ticket=SYN-008 approved_region=region-A action=failover",
|
||||
"Pinned audit record; retain the entire message.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"rollback_receipt",
|
||||
"rollback_release",
|
||||
{"release": "old-synthetic-release"},
|
||||
"old rollback receipt",
|
||||
"Historical side effect, not an eligible read-only query.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"chart",
|
||||
"query_metrics",
|
||||
{"chart": "replication"},
|
||||
[{"type": "data", "source": {"type": "base64", "media_type": "image/png", "data": "AAAA"}}],
|
||||
"Non-text chart output must remain unchanged.",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"failed_metrics",
|
||||
"query_metrics",
|
||||
{"region": "region-A", "metric": "disk"},
|
||||
"query failed",
|
||||
"Failure record must remain intact.",
|
||||
result_state="error",
|
||||
)
|
||||
_pair(
|
||||
history,
|
||||
"recent_alert",
|
||||
"query_logs",
|
||||
{"alert": "synthetic-replica-lag"},
|
||||
_rows("recent-alert", 20),
|
||||
"Newest exchange remains protected by recency.",
|
||||
)
|
||||
goal = (
|
||||
"Next action: write the final region-A failover root-cause report using the exact incident_logs events "
|
||||
"and wal_offsets rows already obtained. Quote every incident event line and the slot row verbatim; "
|
||||
"do not perform additional log, database or metric queries before the report. Correlate the leader switch "
|
||||
"with paused WAL apply and the offset gap. Ignore completed marketing, holiday, retired-system load-test "
|
||||
"and advertising-budget tasks. Preserve the audit approval, chart and failed probes. Do not roll back."
|
||||
)
|
||||
history.append({"id": "last", "role": "user", "content": [{"type": "text", "text": goal}]})
|
||||
asyncio.run(
|
||||
_run_case(
|
||||
live_config,
|
||||
tmp_path,
|
||||
name="replication_incident",
|
||||
history=history,
|
||||
goal=goal,
|
||||
eligible=["read_file", "query_database", "query_logs", "query_metrics"],
|
||||
protected_calls=[],
|
||||
protected_messages=["audit_approval-call"],
|
||||
must_keep=["incident_logs", "wal_offsets"],
|
||||
irrelevant=["old_campaign", "old_weather", "old_loadtest", "old_costs"],
|
||||
),
|
||||
)
|
||||
Loading…
Add table
Reference in a new issue