ReMe/reme/utils
xyf2020 6b035c6553
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
feat(evaluation): track job calls and agent token usage in benchmarks (#406)
* feat(counter): extend counter tree utils and record job call statistics

- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils

* feat(evaluation): add check_job_count interface and report search calls in benchmarks

- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks

* job counter

* token消耗量统计

* benchmark输出完整token消耗统计

* benchmark统计输出改用标准差

- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出
- 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring
- black 格式化

* fix(evaluation): preserve complete token usage metrics

* fix: exclude stream replies from token accounting

* Revert "fix: exclude stream replies from token accounting"

This reverts commit 85bf32064d.

* Reapply "fix: exclude stream replies from token accounting"

This reverts commit 6722c24dc5.

* support agent scope 2.0.5

* feat: support injection_config to disable runtime state injection in benchmarks

- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
  wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls

* feat: agentscope dual-version compat & benchmark improvements

- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
  and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
  time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes

* chore: remove temporary flip-test benchmark config

* revert: pin agentscope to 2.0.4.post1 and drop dual-version compat

* fix(evaluation): clarify usage semantics and atomic counters

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-08-04 11:42:18 +08:00
..
__init__.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
agent_state_io.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
arxiv.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
common_utils.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
counter.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
env_utils.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
evaluation_interface.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
huggingface_papers.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
jsonl_zst.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
link_expansion.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
logger_utils.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
logo_utils.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
service_utils.py feat(core): add shell execution and runtime memory status (#344) 2026-07-14 16:31:41 +08:00
similarity_utils.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
token_utils.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
tushare.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
wikilink_handler.py docs: rename vault_dir to workspace_dir in documentation and examples (#286) 2026-06-22 16:58:57 +08:00