2 commits

Author	SHA1	Message	Date
xyf2020	6b035c6553	feat(evaluation): track job calls and agent token usage in benchmarks (#406) ... * feat(counter): extend counter tree utils and record job call statistics - replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all - record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs - update agentic_answer step and utils exports; add unit tests for job counting and counter utils * feat(evaluation): add check_job_count interface and report search calls in benchmarks - Extract _counter_key from BaseJob._record_call for reusable counter lookup - Add reme.utils.evaluation_interface.check_job_count read-only helper - Track and report average search calls per query in beam and longmemeval benchmarks * job counter * token消耗量统计 * benchmark输出完整token消耗统计 * benchmark统计输出改用标准差 - beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出 - 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring - black 格式化 * fix(evaluation): preserve complete token usage metrics * fix: exclude stream replies from token accounting * Revert "fix: exclude stream replies from token accounting" This reverts commit 85bf32064d. * Reapply "fix: exclude stream replies from token accounting" This reverts commit 6722c24dc5. * support agent scope 2.0.5 * feat: support injection_config to disable runtime state injection in benchmarks - Add InjectionConfig passthrough in AsAgentWrapper.reply() - Disable inject_runtime_state in BaseAgenticAnswerStep to avoid wall-clock time conflicting with benchmark query_time anchors - Disable inject_runtime_state in beam/lme llm_judge calls * feat: agentscope dual-version compat & benchmark improvements - Add version_tuple utility for semantic version comparison - AsAgentWrapper: version-aware InjectionConfig, max_iters doubling, and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5 - Default inject_runtime_state=False in wrapper to avoid benchmark time-anchor conflicts; remove per-callsite injection_config overrides - longmemeval run.py: support question_ids filter in dataset config - Fix unused import in test_evaluation_interface; format fixes * chore: remove temporary flip-test benchmark config * revert: pin agentscope to 2.0.4.post1 and drop dual-version compat * fix(evaluation): clarify usage semantics and atomic counters --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-08-04 11:42:18 +08:00
xyf2020	630f26b119	feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) ... * feat(search): add tool_context-scoped chunk dedup with TTL Introduce _ToolContextDedupMixin shared by search/vector_search/bm25_search to skip already-seen chunks within one agent tool_context. Per-context state lives in app_context.metadata with configurable TTL (default 24h). * feat(search): unify chunk answer rendering with merge and explicit empty messages - Refactor SearchStep/VectorSearchStep/Bm25SearchStep to share format_chunks_answer for consistent source rendering and adjacent session-chunk merging. - Distinguish empty results: ALL_RETURNED_MESSAGE when dedup removes everything vs NO_RESULTS_MESSAGE when nothing matched. - Bump JsonlFileChunker default max_chars to 4000. - Add unit tests for source-format merge and empty-result messages. * refactor(config): reorganize file_chunker components and move jsonl max_chars into config - Register explicit markdown/json/jsonl chunkers in beam.yaml and lme.yaml with markdown options (embed_toc, max_ast_sections, frontmatter handling) and jsonl max_chars=4000. - Restrict default chunker to txt/log extensions. - Revert JsonlFileChunker code default max_chars back to 2000; the 4000 value now lives in config. * chore(benchmark): increase longmemeval num_items from 64 to 500 * refactor(search): split SearchStep into simplified and v2 variants, extract counter utility - Extract global_counter_next from ApplicationContext into reme/utils/counter.py as a standalone function operating on metadata dict with lazy initialization. - Split SearchStep into two variants: - SearchStep (simplified): inline chunk.id dedup, single-branch vector/keyword optimization based on vector_weight, inline answer formatting. - SearchV2Step (full): preserves _ToolContextDedupMixin with interval-subset-aware dedup and format_chunks_answer with session-aware chunk merging. - Update beam.yaml and lme.yaml to use search_v2_step for benchmark jobs. - Rename existing search tests to test_search_v2_step_* and add new test_search_step_* tests covering the simplified variant. * fix: normalise missing trailing newline in _build_union_chunk to prevent line collision * refactor: lazy-init counter tree in ApplicationContext metadata - Remove hardcoded _counter_tree and _counter_tree_lock initialization from ApplicationContext.metadata; rely on lazy initialization in reme.utils.counter.global_counter_next on first call - Set longmemeval num_items back to 500 - Remove obsolete trailing-newline collision tests --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-07-22 17:17:23 +08:00