xyf2020
|
6b035c6553
|
feat(evaluation): track job calls and agent token usage in benchmarks (#406)
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* feat(counter): extend counter tree utils and record job call statistics
- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils
* feat(evaluation): add check_job_count interface and report search calls in benchmarks
- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks
* job counter
* token消耗量统计
* benchmark输出完整token消耗统计
* benchmark统计输出改用标准差
- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出
- 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring
- black 格式化
* fix(evaluation): preserve complete token usage metrics
* fix: exclude stream replies from token accounting
* Revert "fix: exclude stream replies from token accounting"
This reverts commit 85bf32064d.
* Reapply "fix: exclude stream replies from token accounting"
This reverts commit 6722c24dc5.
* support agent scope 2.0.5
* feat: support injection_config to disable runtime state injection in benchmarks
- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls
* feat: agentscope dual-version compat & benchmark improvements
- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes
* chore: remove temporary flip-test benchmark config
* revert: pin agentscope to 2.0.4.post1 and drop dual-version compat
* fix(evaluation): clarify usage semantics and atomic counters
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
|
2026-08-04 11:42:18 +08:00 |
|