* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* Add configurable tag indexing and tag listing
* Add tag-filtered hybrid search with exact candidate ranking
* Remove obsolete generated files
* Rename tag index key to tag_key and reject reserved fields
* Extract automatic tagging into a dedicated step
* Restrict frontmatter updates to authorized keys
* Refine tag filtering and automatic memory tagging
* Require underscore-separated tags in auto-tag prompts
- Forbid spaces in tags and require underscores (e.g. sam_altman) in both
English and Chinese auto_tag prompts, with English examples switched to
English entities (OpenAI, gold)
- Drop prompt-string assertions superseded by the new tagging rule
- Merge construction/runtime tag_key validation tests into one parametrized case
* Make max_tags_per_file configurable in auto-tag step
* Consolidate tag index tests
* Fix search test fixture lint warnings
* Align tag contracts and index health behavior
* Fall back when tag index is unavailable
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* refractor(proactive): upgrade proactive feature with disentangled job and steps
* refactor(proactive): apply audit fixes
- rename read-side job 'proactive' -> 'proactive_read' (less confusing vs the refresh pipeline)
- drop dedicated agent_wrapper.proactive; extraction reuses the default wrapper
- simplify schema: remove unused ProactiveExtractOutput/TopicUpdate, drop resource_paths
- extract no longer scans resource/ directly (daily notes already carry resource content)
- update tests and docs accordingly
* feat(proactive): strict extract-output gate and prompt total budget
- parse_extract_reply now requires a contract section (follow_ups/extends/updates
as a list); non-empty replies with misspelled section names trigger the
existing one-shot retry instead of silently checkpointing changed files
- pack_paths gains max_total_chars; extract packs newest daily material first,
keeps the first file on overflow, and records omitted files in a trailer
(default budget 300000 chars, configurable via max_total_chars)
- tests: schema gate unit, schema-error retry e2e, budget unit + e2e
* feat(proactive): add scenario-card plan step and generative agenda step
* feat(proactive): digest-personal profile personalization and leaner LLM contract
- extract/plan/agenda now draw a user profile block from <digest_dir>/personal/*.md
(frontmatter description + body excerpt, per-file budget, profile.md fallback)
- all daily access honours the configured daily_dir (prompt paths parameterized,
config-driven fallbacks) so workspaces using e.g. memory/ work unchanged
- schema trim: drop dead fields errors/material_paths, carry_forward_all -> count
- shrink LLM output contract: new topics emit title/reason/confidence/paths only;
keywords removed end-to-end, evidence derived from paths[0] (updates keep it)
* fix(proactive): skip checkpoint when extract reply stays unusable after retry
Two consecutive unparseable replies now short-circuit the round without
checkpointing, so the same material is retried next round instead of being
silently consumed (closes the residual audit #1 gap: the structural gate
detected schema-wrong output but a double failure still checkpointed).
* fix(proactive): replace running bool with reference-counted job activity tracker for the idle gate
* refactor(proactive): remove job activity tracking and idle gate, restore job tree to upstream
* fix(proactive): address second audit round (readonly reader, mtime checkpoint, wider fallbacks, profile containment, horizon content, expiry boundary)
* refactor(dream): strip interests.yaml ownership from dream, proactive is now the sole writer
* refactor(dream): separate proactive topic generation
* ci: update renamed auto dream smoke test
* fix(proactive): complete refresh migration and docs
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
