ReMe/reme/steps/evolve/compressor.py
xyf2020 5bc46c88b6
feat(benchmark): enhance session memory retrieval and isolate benchmark assets (#409)
* chore(benchmark): isolate dataset/workspaces/results per benchmark

- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py

* feat(benchmark): add read tool to agentic answer and update BEAM results

- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)

* feat(auto_memory): add source line-number markers for note traceability

- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure

* feat(beam): enhance auto_memory with source markers and pilot ingest tooling

* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts

* feat: add CompressorStep and search_v2 dual-mode session compression

- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
  text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
  session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
  join_chunk_entries; session chunks now render line-aligned with
  L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
  per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
  query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths

* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results

* refactor(benchmark): make session compression config-driven via compress_session flag

Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.

* feat(lme/auto_memory): add source attribution markers with line numbers

Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.

* feat: improve agentic answer prompt and update beam 100K results

- Strengthen abstention rule: prohibit extrapolation from related but
  non-direct evidence
- Add multi-angle search after preliminary answer to check for
  conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
  including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
  (+0.140), abstention: 0.500 → 0.525 (+0.025)

* feat(read): add session-aware formatting for read tool and update BEAM eval

- Add truncate_session_output in _file_io.py to render jsonl session
  lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
  job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
  so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
  after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
  1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results

* chore(config): disable expand_links in beam and lme search_v2 configs

* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script

* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions

- Add compressor job and compressor as_llm component to reme/config/beam.yaml
  (aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
  when the compressor job is missing from the active config, log a warning
  and skip compression instead of raising 'Job compressor not found'.
  Skipped when there is no app_context so unit tests mocking run_job still
  drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
  before cd benchmark/beam/dataset (the directory is gitignored and absent
  in a fresh clone)

* fix(steps): guard compressor exceptions and fix ReadStep boolean override

1. search_v2: catch per-entry exceptions from run_job('compressor') inside
   compress() so asyncio.gather never propagates a compressor failure (e.g.
   temporary LLM outage). The failing entry keeps its original body while
   remaining entries are still compressed, preserving already-retrieved
   search results.

2. read: replace 'context_value or yaml_value' with an existence check so
   that a runtime-injected False can explicitly disable a YAML-true
   read_step_format_session flag.

Add focused unit tests for both paths.

* fix(search_v2): use existence check for strict_date_filter boolean override

Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.

* refactor(search): simplify strict_date_filter fallback to truthiness-or

* style(test): rename unused param to satisfy pylint W0613

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
2026-08-05 19:23:42 +08:00

82 lines
3.2 KiB
Python

"""compressor — compress text via a direct LLM call (no agent)."""
from agentscope.message import UserMsg
from agentscope.model import ChatResponse
from ..base_step import BaseStep
from ...components import R
@R.register("compressor_step")
class CompressorStep(BaseStep):
"""Compress ``text`` with a direct LLM call (no agent wrapper).
Inputs (from RuntimeContext):
text (str, required): the text to compress.
queries (list[str], optional, default []): when provided, compression
keeps anything potentially relevant to any of the queries and may
drop content that is certainly irrelevant and certainly unhelpful
for all of them.
Output (written to context.response.answer):
The compressed text.
"""
async def execute(self):
assert self.context is not None
text: str = self.context.get("text", "") or ""
raw_queries = self.context.get("queries") or []
queries: list[str] = [str(q).strip() for q in raw_queries if str(q).strip()]
if not text.strip():
self.context.response.success = False
self.context.response.answer = "Skipped: empty text"
return
if queries:
queries_block = "\n".join(f"- {q}" for q in queries)
user_message = self.prompt_format("compress_query_prompt", queries=queries_block, text=text)
else:
user_message = self.prompt_format("compress_prompt", text=text)
result = await self.as_llm([UserMsg(name="user", content=user_message)])
compressed = await self._response_text(result)
self.logger.info(
f"[{self.name}] compressed {len(text)} -> {len(compressed)} chars (queries={len(queries)})",
)
self.context.response.success = True
self.context.response.answer = compressed
self.context.response.metadata.update(
{
"queries": queries,
"original_length": len(text),
"compressed_length": len(compressed),
},
)
@staticmethod
async def _response_text(result) -> str:
"""Extract text blocks from a streaming or non-streaming ChatResponse.
The async-iteration check is done on the type: ``ChatResponse`` inherits
agentscope's ``DictMixin`` whose instance ``__getattr__`` raises
``KeyError`` (not ``AttributeError``), so ``hasattr(result, ...)``
would crash instead of returning False.
"""
if hasattr(type(result), "__aiter__"):
last: ChatResponse | None = None
async for chunk in result:
last = chunk
result = last
if result is None:
return ""
# Content blocks may be plain dicts or pydantic models (TextBlock/ThinkingBlock).
parts: list[str] = []
for block in result.content or []:
if isinstance(block, dict):
if block.get("type") == "text":
parts.append(str(block.get("text") or ""))
elif getattr(block, "type", None) == "text":
parts.append(str(getattr(block, "text", "") or ""))
return "".join(parts).strip()