ReMe/reme/components
xyf2020 6b035c6553
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
feat(evaluation): track job calls and agent token usage in benchmarks (#406)
* feat(counter): extend counter tree utils and record job call statistics

- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils

* feat(evaluation): add check_job_count interface and report search calls in benchmarks

- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks

* job counter

* token消耗量统计

* benchmark输出完整token消耗统计

* benchmark统计输出改用标准差

- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出
- 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring
- black 格式化

* fix(evaluation): preserve complete token usage metrics

* fix: exclude stream replies from token accounting

* Revert "fix: exclude stream replies from token accounting"

This reverts commit 85bf32064d.

* Reapply "fix: exclude stream replies from token accounting"

This reverts commit 6722c24dc5.

* support agent scope 2.0.5

* feat: support injection_config to disable runtime state injection in benchmarks

- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
  wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls

* feat: agentscope dual-version compat & benchmark improvements

- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
  and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
  time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes

* chore: remove temporary flip-test benchmark config

* revert: pin agentscope to 2.0.4.post1 and drop dual-version compat

* fix(evaluation): clarify usage semantics and atomic counters

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-08-04 11:42:18 +08:00
..
agent_wrapper feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
as_embedding refactor(embedding): defer provider construction until first remote call (#361) 2026-07-15 20:46:36 +08:00
as_llm init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
client feat(benchmark): add lme benchmark steps (#326) 2026-07-07 18:53:39 +09:00
embedding_store feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
file_catalog Refactor logging levels and add dream schema definitions (#291) 2026-06-24 16:44:03 +08:00
file_chunker feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
file_graph feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
file_store feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
job feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
keyword_index feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
outbound_proxy feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
service Revert "feat(plugin): add ReMe integration for Codex (#372)" (#400) 2026-07-29 18:14:05 +08:00
tokenizer Refactor logging levels and add dream schema definitions (#291) 2026-06-24 16:44:03 +08:00
__init__.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
application_context.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
base_component.py docs: rename vault_dir to workspace_dir in documentation and examples (#286) 2026-06-22 16:58:57 +08:00
component_registry.py feat(benchmark): add LongMemEval golden answer validation (#335) 2026-07-13 21:26:27 +08:00
prompt_handler.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
runtime_context.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00