mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-01 07:03:02 +00:00
refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
This commit is contained in:
sa-buc
parent
db4763aead
commit
0c7893acb0
6 changed files with 60 additions and 122 deletions
|
|
@ -4,10 +4,10 @@ system_prompt: |
|
|||
## Memory System
|
||||
The memory system stores the user's history in three forms:
|
||||
- **Summary notes**: condensed knowledge distilled from past sessions.
|
||||
- **Compressed session chunks**: fragments of session transcripts whose assistant messages have been compressed; user messages remain verbatim, but some assistant-side detail may be shortened or lost.
|
||||
- **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost.
|
||||
- **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files.
|
||||
|
||||
The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice, and each of their lines carries a `L<n>:` prefix giving that message's exact line number in the original session file. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path (guided by the `L<n>` line numbers) to read the original session before drawing conclusions.
|
||||
The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions.
|
||||
|
||||
## Workflow
|
||||
1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient.
|
||||
|
|
|
|||
|
|
@ -1,21 +1,15 @@
|
|||
compress_prompt: |
|
||||
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter while losing zero information.
|
||||
|
||||
## Dialogue Handling
|
||||
|
||||
1. The text is a dialogue with exactly ONE message per line. Each line starts with a `[user @ <timestamp>]` or `[assistant @ <timestamp>]` marker followed by that message's content.
|
||||
2. Preserve this line structure exactly: output exactly one line per input line, in the original order, each keeping its original `[role @ timestamp]` marker. Never merge, split, drop, or reorder lines — the output must have exactly as many lines as the input.
|
||||
3. Only compress the content of assistant lines. Copy every user line verbatim, without any modification.
|
||||
|
||||
## Compression Rules (assistant messages only)
|
||||
## Compression Rules
|
||||
|
||||
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
|
||||
2. You may rewrite, merge, or restructure sentences to make them shorter, but you must NOT lose any information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance present in the original must remain fully recoverable from your output.
|
||||
3. Never delete informative content, and never weaken it into a vaguer statement. Compression must come only from removing zero-information phrasing and from tighter wording.
|
||||
4. Never add new information, comments, guesses, or interpretations of your own.
|
||||
5. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original.
|
||||
6. Keep the original language of the text and preserve the original order of information where practical.
|
||||
7. If an assistant line carries no real information (pure greetings/small talk), keep only its `[assistant @ <timestamp>]` marker on that line.
|
||||
5. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
|
||||
6. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
|
||||
7. If the entire text carries no real information (pure greetings/small talk), output nothing.
|
||||
|
||||
## Output Format
|
||||
|
||||
|
|
@ -27,21 +21,15 @@ compress_prompt: |
|
|||
compress_prompt_zh: |
|
||||
你是一个文本压缩引擎。下面的文本是一段对话记录。请压缩它，使其变短，但信息量零损失。
|
||||
|
||||
## 对话处理
|
||||
|
||||
1. 文本是对话格式，严格每行一条消息。每行以形如 `[user @ <时间戳>]` 或 `[assistant @ <时间戳>]` 的标记开头，后面跟该消息的内容。
|
||||
2. 严格保持行结构：输入有多少行，输出就必须有多少行，顺序不变，每行保留其原有的 `[role @ 时间戳]` 标记。不得合并、拆分、删除或重排任何行。
|
||||
3. 只压缩 assistant 行的内容。所有 user 行必须逐字保留，不做任何修改。
|
||||
|
||||
## 压缩规则（仅针对 assistant 消息）
|
||||
## 压缩规则
|
||||
|
||||
1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。
|
||||
2. 可以改写、合并、调整语句以减小长度，但不得损失任何信息。原文中的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。
|
||||
3. 不得删减信息内容，也不得把信息弱化成更模糊的表述。压缩只能来自删除零信息量的表达和更紧凑的措辞。
|
||||
4. 不得添加任何新信息、评论、猜测或你自己的解读。
|
||||
5. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。
|
||||
6. 保持文本的原始语言，并尽量保持信息的原有顺序。
|
||||
7. 如果某一 assistant 行没有任何实际信息（纯问候/寒暄），该行只保留 `[assistant @ <时间戳>]` 标记。
|
||||
5. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。
|
||||
6. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。
|
||||
7. 如果整段文本没有任何实际信息（纯问候/寒暄），则输出空。
|
||||
|
||||
## 输出格式
|
||||
|
||||
|
|
@ -54,13 +42,7 @@ compress_prompt_zh: |
|
|||
compress_query_prompt: |
|
||||
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter, guided by the queries as a relevance filter, without losing any potentially useful information.
|
||||
|
||||
## Dialogue Handling
|
||||
|
||||
1. The text is a dialogue with exactly ONE message per line. Each line starts with a `[user @ <timestamp>]` or `[assistant @ <timestamp>]` marker followed by that message's content.
|
||||
2. Preserve this line structure exactly: output exactly one line per input line, in the original order, each keeping its original `[role @ timestamp]` marker. Never merge, split, drop, or reorder lines — the output must have exactly as many lines as the input.
|
||||
3. Only compress the content of assistant lines. Copy every user line verbatim, without any modification.
|
||||
|
||||
## Compression Rules (assistant messages only)
|
||||
## Compression Rules
|
||||
|
||||
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
|
||||
2. You may rewrite, merge, or restructure sentences to make them shorter, but the kept content must NOT lose information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance you keep must remain fully recoverable from your output.
|
||||
|
|
@ -68,8 +50,8 @@ compress_query_prompt: |
|
|||
4. You may delete a piece of information ONLY when both conditions hold for EVERY query: it is certainly unrelated to that query, AND it is certainly of no help whatsoever for answering that query.
|
||||
5. If you are unsure whether something can be deleted, KEEP it. When in doubt, always err on the side of keeping.
|
||||
6. Do NOT answer the queries. Do NOT add new information, comments, guesses, or interpretations of your own.
|
||||
7. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original.
|
||||
8. Keep the original language of the text and preserve the original order of information where practical.
|
||||
7. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
|
||||
8. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
|
||||
|
||||
## Output Format
|
||||
|
||||
|
|
@ -85,13 +67,7 @@ compress_query_prompt: |
|
|||
compress_query_prompt_zh: |
|
||||
你是一个文本压缩引擎。下面的文本是一段对话记录。请以下列 query 列表作为相关性过滤条件，压缩文本使其变短，同时不丢失任何可能有用的信息。
|
||||
|
||||
## 对话处理
|
||||
|
||||
1. 文本是对话格式，严格每行一条消息。每行以形如 `[user @ <时间戳>]` 或 `[assistant @ <时间戳>]` 的标记开头，后面跟该消息的内容。
|
||||
2. 严格保持行结构：输入有多少行，输出就必须有多少行，顺序不变，每行保留其原有的 `[role @ 时间戳]` 标记。不得合并、拆分、删除或重排任何行。
|
||||
3. 只压缩 assistant 行的内容。所有 user 行必须逐字保留，不做任何修改。
|
||||
|
||||
## 压缩规则（仅针对 assistant 消息）
|
||||
## 压缩规则
|
||||
|
||||
1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。
|
||||
2. 可以改写、合并、调整语句以减小长度，但保留下来的内容不得损失信息。你保留的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。
|
||||
|
|
@ -99,8 +75,8 @@ compress_query_prompt_zh: |
|
|||
4. 只有对每一个 query 都同时满足以下两个条件时，才可以删除一条信息：它一定与该 query 无关，并且对回答该 query 一定没有任何帮助。
|
||||
5. 对于不确定是否可以删除的信息，一律保留。拿不准时，永远倾向于保留。
|
||||
6. 不要回答这些 query 本身。不得添加任何新信息、评论、猜测或你自己的解读。
|
||||
7. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。
|
||||
8. 保持文本的原始语言，并尽量保持信息的原有顺序。
|
||||
7. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。
|
||||
8. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。
|
||||
|
||||
## 输出格式
|
||||
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@
|
|||
|
||||
Raw session transcripts (``*.jsonl`` under the dialog dir) store one serialized
|
||||
``Msg`` per line. :func:`render_chunk_body` renders those line-aligned — one
|
||||
message per line with a ``L<n>:`` prefix carrying the real line number in the
|
||||
session file; all other chunks keep their raw ``text``. Used by
|
||||
message per line with internal newlines flattened; all other chunks keep their
|
||||
raw ``text``. Used by
|
||||
``search``/``vector_search``/``bm25_search`` so every step renders session hits
|
||||
identically.
|
||||
|
||||
|
|
@ -53,19 +53,17 @@ def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool:
|
|||
|
||||
|
||||
def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str:
|
||||
"""Render a chunk's body; raw session transcripts render line-aligned and numbered.
|
||||
"""Render a chunk's body; raw session transcripts render one message per line.
|
||||
|
||||
Session chunks are jsonl where each line is a serialized ``Msg``. They
|
||||
render via :func:`render_session_chunk_lines` — one message per line with
|
||||
internal newlines flattened — and every line gets a ``L<n>:`` prefix
|
||||
carrying its real line number in the session file, so verbatim and
|
||||
compressed session bodies share the same line-numbered format. All other
|
||||
chunks keep their raw ``text``.
|
||||
internal newlines flattened — so verbatim and compressed session bodies
|
||||
share the same single-line-per-message format. All other chunks keep
|
||||
their raw ``text``.
|
||||
"""
|
||||
if not is_session_chunk(chunk, dialog_dir):
|
||||
return chunk.text
|
||||
lines = render_session_chunk_lines(chunk)
|
||||
return "\n".join(f"L{chunk.start_line + i}: {line}" for i, line in enumerate(lines))
|
||||
return "\n".join(render_session_chunk_lines(chunk))
|
||||
|
||||
|
||||
def render_session_chunk_lines(chunk: FileChunk) -> list[str]:
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ from typing import Final
|
|||
|
||||
from ._dedup import _ToolContextDedupMixin
|
||||
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, is_session_path, join_chunk_entries
|
||||
from ._source_format import merge_session_chunk_intervals, render_chunk_entries, render_session_chunk_lines
|
||||
from ._source_format import merge_session_chunk_intervals, render_chunk_entries
|
||||
from ..base_step import BaseStep
|
||||
from ..file_io import extract_daily_date
|
||||
from ...components import R
|
||||
|
|
@ -211,15 +211,14 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
)
|
||||
|
||||
dialog_dir = self.config_value("dialog_dir")
|
||||
merged_chunks = merge_session_chunk_intervals(fused, dialog_dir)
|
||||
entries = render_chunk_entries(
|
||||
merged_chunks,
|
||||
merge_session_chunk_intervals(fused, dialog_dir),
|
||||
dialog_dir,
|
||||
score_fn=lambda c: self._format_scores(c.scores, hybrid),
|
||||
link_expansion=link_expansion,
|
||||
)
|
||||
if self._session_compress_enabled():
|
||||
await self._compress_session_entries(entries, merged_chunks, query, dialog_dir)
|
||||
await self._compress_session_entries(entries, query, dialog_dir)
|
||||
self.context.response.answer = join_chunk_entries(entries)
|
||||
if not fused:
|
||||
self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE
|
||||
|
|
@ -244,13 +243,7 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
value = (search_cfg.get("_compress") or {}).get("session")
|
||||
return value is True or str(value).strip().lower() == "true"
|
||||
|
||||
async def _compress_session_entries(
|
||||
self,
|
||||
entries: list[dict[str, str]],
|
||||
chunks: list[FileChunk],
|
||||
query: str,
|
||||
dialog_dir: str,
|
||||
) -> None:
|
||||
async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, dialog_dir: str) -> None:
|
||||
"""Compress session-transcript entry bodies in place via the ``compressor`` job.
|
||||
|
||||
Only entries whose ``path`` points at a raw session transcript are
|
||||
|
|
@ -260,16 +253,11 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
the default) it receives the injected ``_search.queries`` plus the
|
||||
current search query.
|
||||
|
||||
The compressor input is the line-aligned rendering of the entry's
|
||||
chunk (one message per line, see :func:`render_session_chunk_lines`),
|
||||
so input line ``i`` maps to file line ``chunk.start_line + i``. The
|
||||
output is adopted only when the compressor succeeded, its line count
|
||||
equals the input's, and it is not longer than the input. Adopted
|
||||
bodies get a ``L<line>:`` prefix per line carrying the true line
|
||||
number in the original session jsonl file, plus a leading
|
||||
``compressed session chunk:`` marker, so downstream consumers can
|
||||
tell them from verbatim transcripts and ``read`` the exact original
|
||||
lines.
|
||||
The compressor receives the already-rendered body (one message per
|
||||
line) stripped. Its output is adopted whenever the compressor
|
||||
succeeded and the result is not longer than the input; adopted bodies
|
||||
get a leading ``compressed session chunk:`` marker so downstream
|
||||
consumers can tell them from verbatim transcripts.
|
||||
"""
|
||||
assert self.context is not None
|
||||
search_cfg: dict = self.context.get("_search") or {}
|
||||
|
|
@ -281,22 +269,12 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
if query and query not in queries:
|
||||
queries.append(query)
|
||||
|
||||
async def compress(entry: dict[str, str], chunk: FileChunk) -> None:
|
||||
async def compress(entry: dict[str, str]) -> None:
|
||||
path = entry.get("path", "")
|
||||
lines = render_session_chunk_lines(chunk)
|
||||
# Trim leading/trailing blank lines but track the offset so the
|
||||
# L-markers stay aligned with the original file line numbers.
|
||||
start = 0
|
||||
end = len(lines)
|
||||
while start < end and not lines[start].strip():
|
||||
start += 1
|
||||
while end > start and not lines[end - 1].strip():
|
||||
end -= 1
|
||||
if start >= end:
|
||||
body = (entry.get("body", "") or "").strip()
|
||||
if not body:
|
||||
return
|
||||
trimmed = lines[start:end]
|
||||
text = "\n".join(trimmed)
|
||||
response = await self.run_job("compressor", text=text, queries=queries)
|
||||
response = await self.run_job("compressor", text=body, queries=queries)
|
||||
compressed = str(response.answer or "").strip()
|
||||
if not response.success or not compressed:
|
||||
self.logger.warning(
|
||||
|
|
@ -304,29 +282,16 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
f"success={response.success} answer={compressed[:100]!r}; keeping original",
|
||||
)
|
||||
return
|
||||
compressed_lines = compressed.splitlines()
|
||||
if len(compressed_lines) != len(trimmed):
|
||||
self.logger.warning(
|
||||
f"[{self.name}] compressed line count mismatch "
|
||||
f"({len(compressed_lines)} != {len(trimmed)}) path={path!r}; keeping original",
|
||||
)
|
||||
return
|
||||
if len(compressed) > len(text):
|
||||
if len(compressed) > len(body):
|
||||
self.logger.info(
|
||||
f"[{self.name}] compressed body longer than original "
|
||||
f"({len(compressed)} > {len(text)}) path={path!r}; keeping original",
|
||||
f"({len(compressed)} > {len(body)}) path={path!r}; keeping original",
|
||||
)
|
||||
return
|
||||
first_line = chunk.start_line + start
|
||||
numbered = "\n".join(f"L{first_line + i}: {line}" for i, line in enumerate(compressed_lines))
|
||||
entry["body"] = f"compressed session chunk:\n{numbered}"
|
||||
entry["body"] = f"compressed session chunk:\n{compressed}"
|
||||
|
||||
targets = [
|
||||
(entry, chunk)
|
||||
for entry, chunk in zip(entries, chunks)
|
||||
if is_session_path(entry.get("path", ""), dialog_dir)
|
||||
]
|
||||
targets = [e for e in entries if is_session_path(e.get("path", ""), dialog_dir)]
|
||||
if not targets:
|
||||
return
|
||||
self.logger.info(f"[{self.name}] compressing {len(targets)} session entries with {len(queries)} queries")
|
||||
await asyncio.gather(*(compress(entry, chunk) for entry, chunk in targets))
|
||||
await asyncio.gather(*(compress(entry) for entry in targets))
|
||||
|
|
|
|||
|
|
@ -1089,9 +1089,9 @@ def test_search_v2_step_keeps_original_body_when_compressed_is_longer():
|
|||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_search_v2_step_compression_strips_input_and_numbers_lines_from_start_line():
|
||||
"""The compressor receives the line-aligned rendering; adopted output is prefixed
|
||||
with ``L<line>:`` markers counted from the chunk's start_line in the session file."""
|
||||
def test_search_v2_step_compression_strips_input_and_adds_marker():
|
||||
"""The compressor receives the stripped rendered body; adopted output is
|
||||
prefixed with the ``compressed session chunk:`` marker only (no line numbers)."""
|
||||
|
||||
async def run():
|
||||
session = FileChunk(
|
||||
|
|
@ -1116,7 +1116,7 @@ def test_search_v2_step_compression_strips_input_and_numbers_lines_from_start_li
|
|||
resp = await step(ctx)
|
||||
|
||||
assert calls[0]["text"] == "first message line\nsecond message line"
|
||||
assert "compressed session chunk:\nL4: first compact\nL5: second compact" in resp.answer
|
||||
assert "compressed session chunk:\nfirst compact\nsecond compact" in resp.answer
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
|
@ -1153,13 +1153,13 @@ def test_search_v2_step_compression_input_is_line_aligned_with_session_jsonl():
|
|||
assert len(sent_lines) == 2
|
||||
assert sent_lines[0].startswith("[user @") and "line one line two" in sent_lines[0]
|
||||
assert sent_lines[1].startswith("[assistant @") and "long answer" in sent_lines[1]
|
||||
assert "compressed session chunk:\nL7: [user @ t] u\nL8: [assistant @ t] a" in resp.answer
|
||||
assert "compressed session chunk:\n[user @ t] u\n[assistant @ t] a" in resp.answer
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_search_v2_step_keeps_original_body_on_compressed_line_count_mismatch():
|
||||
"""Compressed output whose line count differs from the input is rejected."""
|
||||
def test_search_v2_step_adopts_shorter_output_regardless_of_line_count():
|
||||
"""A shorter compressor output is adopted even when its line count differs."""
|
||||
|
||||
async def run():
|
||||
session = FileChunk(
|
||||
|
|
@ -1181,8 +1181,8 @@ def test_search_v2_step_keeps_original_body_on_compressed_line_count_mismatch():
|
|||
|
||||
resp = await step(ctx)
|
||||
|
||||
assert "first message line" in resp.answer
|
||||
assert "merged into one line" not in resp.answer
|
||||
assert "compressed session chunk:\nmerged into one line" in resp.answer
|
||||
assert "first message line" not in resp.answer
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
|
|
|||
|
|
@ -4,9 +4,8 @@ Session chunks (``*.jsonl`` under the dialog dir) whose line ranges overlap,
|
|||
contain one another, or are adjacent are merged into their union by the caller
|
||||
(``merge_session_chunk_intervals``) before rendering with
|
||||
``render_chunk_entries`` + ``join_chunk_entries``. Bodies here are plain
|
||||
(non-``Msg``) text lines, which pass through stripped and get a ``L<n>:``
|
||||
line-number prefix — letting these tests assert the union content, line order,
|
||||
and file-line numbering directly.
|
||||
(non-``Msg``) text lines, which pass through stripped — letting these tests
|
||||
assert the union content and line order directly.
|
||||
"""
|
||||
|
||||
from reme.schema import FileChunk
|
||||
|
|
@ -33,7 +32,7 @@ def test_overlapping_session_chunks_merge_into_union_without_duplicates():
|
|||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\nL4: m4\nL5: m5"
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
||||
def test_contained_session_chunk_is_absorbed_by_the_larger_range():
|
||||
|
|
@ -43,7 +42,7 @@ def test_contained_session_chunk_is_absorbed_by_the_larger_range():
|
|||
|
||||
answer = _render([big, small], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\nL4: m4\nL5: m5"
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
||||
def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start():
|
||||
|
|
@ -53,7 +52,7 @@ def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start():
|
|||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\nL4: m4\nL5: m5\nL6: m6"
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5\nm6"
|
||||
|
||||
|
||||
def test_session_chunks_with_a_gap_are_not_merged():
|
||||
|
|
@ -63,7 +62,7 @@ def test_session_chunks_with_a_gap_are_not_merged():
|
|||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\n\nL5: m5\nL6: m6"
|
||||
assert answer == "m1\nm2\nm3\n\nm5\nm6"
|
||||
|
||||
|
||||
def test_session_chunks_from_different_files_are_not_merged():
|
||||
|
|
@ -73,7 +72,7 @@ def test_session_chunks_from_different_files_are_not_merged():
|
|||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: A1\nL2: A2\nL3: A3\n\nL2: B2\nL3: B3\nL4: B4"
|
||||
assert answer == "A1\nA2\nA3\n\nB2\nB3\nB4"
|
||||
|
||||
|
||||
def test_non_session_chunks_are_never_merged():
|
||||
|
|
@ -94,7 +93,7 @@ def test_merge_preserves_line_order_regardless_of_input_rank_order():
|
|||
# Higher-scored later-range chunk is listed first (as a ranker would).
|
||||
answer = _render([later, earlier], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\nL4: m4\nL5: m5"
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
||||
def test_merged_header_spans_the_union_range_and_keeps_best_score():
|
||||
|
|
@ -117,7 +116,7 @@ def test_separate_intervals_in_same_file_stay_separate():
|
|||
|
||||
answer = _render([a, b, c], _DIALOG_DIR, include_source=False)
|
||||
|
||||
assert answer == "L1: m1\nL2: m2\nL3: m3\nL4: m4\n\nL10: m10\nL11: m11"
|
||||
assert answer == "m1\nm2\nm3\nm4\n\nm10\nm11"
|
||||
|
||||
|
||||
def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank():
|
||||
|
|
@ -133,7 +132,7 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank(
|
|||
|
||||
# s1's two units are adjacent and sorted by start_line (1-3 before 10-12),
|
||||
# placed at s1's earliest rank (0), so the whole s1 block precedes s2.
|
||||
assert answer == "L1: S1a\nL2: S1b\nL3: S1c\n\nL10: S1x\nL11: S1y\nL12: S1z\n\nL1: S2a\nL2: S2b\nL3: S2c"
|
||||
assert answer == "S1a\nS1b\nS1c\n\nS1x\nS1y\nS1z\n\nS2a\nS2b\nS2c"
|
||||
|
||||
|
||||
def test_same_file_units_adjacency_with_source_headers():
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue