xyf2020 6b035c6553 feat(evaluation): track job calls and agent token usage in benchmarks (#406) ... * feat(counter): extend counter tree utils and record job call statistics - replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all - record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs - update agentic_answer step and utils exports; add unit tests for job counting and counter utils * feat(evaluation): add check_job_count interface and report search calls in benchmarks - Extract _counter_key from BaseJob._record_call for reusable counter lookup - Add reme.utils.evaluation_interface.check_job_count read-only helper - Track and report average search calls per query in beam and longmemeval benchmarks * job counter * token消耗量统计 * benchmark输出完整token消耗统计 * benchmark统计输出改用标准差 - beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出 - 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring - black 格式化 * fix(evaluation): preserve complete token usage metrics * fix: exclude stream replies from token accounting * Revert "fix: exclude stream replies from token accounting" This reverts commit 85bf32064d. * Reapply "fix: exclude stream replies from token accounting" This reverts commit 6722c24dc5. * support agent scope 2.0.5 * feat: support injection_config to disable runtime state injection in benchmarks - Add InjectionConfig passthrough in AsAgentWrapper.reply() - Disable inject_runtime_state in BaseAgenticAnswerStep to avoid wall-clock time conflicting with benchmark query_time anchors - Disable inject_runtime_state in beam/lme llm_judge calls * feat: agentscope dual-version compat & benchmark improvements - Add version_tuple utility for semantic version comparison - AsAgentWrapper: version-aware InjectionConfig, max_iters doubling, and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5 - Default inject_runtime_state=False in wrapper to avoid benchmark time-anchor conflicts; remove per-callsite injection_config overrides - longmemeval run.py: support question_ids filter in dataset config - Fix unused import in test_evaluation_interface; format fixes * chore: remove temporary flip-test benchmark config * revert: pin agentscope to 2.0.4.post1 and drop dual-version compat * fix(evaluation): clarify usage semantics and atomic counters --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>		2026-08-04 11:42:18 +08:00
..
components	feat(evaluation): track job calls and agent token usage in benchmarks (#406)	2026-08-04 11:42:18 +08:00
config	Revert "feat(plugin): add ReMe integration for Codex (#372)" (#400)	2026-07-29 18:14:05 +08:00
enumeration	feat: add mail component enum (#405)	2026-07-30 14:48:20 +08:00
schema	feat(evaluation): track job calls and agent token usage in benchmarks (#406)	2026-08-04 11:42:18 +08:00
steps	feat(evaluation): track job calls and agent token usage in benchmarks (#406)	2026-08-04 11:42:18 +08:00
utils	feat(evaluation): track job calls and agent token usage in benchmarks (#406)	2026-08-04 11:42:18 +08:00
__init__.py	refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382)	2026-07-20 23:52:14 +08:00
application.py	feat(benchmark): add LongMemEval golden answer validation (#335)	2026-07-13 21:26:27 +08:00
constants.py	feat(index): add file size limits and oversized file handling (#362)	2026-07-15 21:01:18 +08:00
reme.py	refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382)	2026-07-20 23:52:14 +08:00