scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-10-09 03:20:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
ReMe/tests/unit
History
xyf2020 6b035c6553
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Details
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Details
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Details
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Details
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
Details
feat(evaluation): track job calls and agent token usage in benchmarks (#406) 
* feat(counter): extend counter tree utils and record job call statistics

- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils

* feat(evaluation): add check_job_count interface and report search calls in benchmarks

- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks

* job counter

* token消耗量统计

* benchmark输出完整token消耗统计

* benchmark统计输出改用标准差

- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出
- 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring
- black 格式化

* fix(evaluation): preserve complete token usage metrics

* fix: exclude stream replies from token accounting

* Revert "fix: exclude stream replies from token accounting"

This reverts commit 85bf32064d.

* Reapply "fix: exclude stream replies from token accounting"

This reverts commit 6722c24dc5.

* support agent scope 2.0.5

* feat: support injection_config to disable runtime state injection in benchmarks

- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
  wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls

* feat: agentscope dual-version compat & benchmark improvements

- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
  and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
  time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes

* chore: remove temporary flip-test benchmark config

* revert: pin agentscope to 2.0.4.post1 and drop dual-version compat

* fix(evaluation): clarify usage semantics and atomic counters

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
2026-08-04 11:42:18 +08:00
..
__init__.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_as_embedding_lazy.py refactor(embedding): defer provider construction until first remote call (#361) 2026-07-15 20:46:36 +08:00
test_auto_dream.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_auto_fin.py refactor(auto_fin): normalize data models and selection logic across agents (#396) 2026-07-27 20:15:27 +08:00
test_auto_memory_cc.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_background_steps.py feat(Step tools): add white/black path prefix permission filtering to read, edit, write (#391) 2026-07-27 17:20:21 +08:00
test_base_agent_wrapper.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_base_component.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_cc_agent_wrapper.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_codex_agent_wrapper.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_common_steps.py feat(lme): add cli execution and agentic search tooling (#334) 2026-07-08 17:43:09 +09:00
test_component_registry.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_config_parser.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_crud_steps.py fix(file_io): stop ReadStep small-file path over-counting total lines by 1 (#389) 2026-07-27 11:01:09 +08:00
test_daily_paper.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_daily_steps.py fix(auto-memory): preserve message timestamps (#310) 2026-07-01 17:15:07 +08:00
test_default_file_chunker.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_dingtalk_wait.py refactor(auto_fin): replace similarity with direction classification for historical events (#395) 2026-07-27 11:52:23 +08:00
test_embedded_consumer_compat.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_evaluation_interface.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_evolve_utils.py fix(auto-memory): preserve message timestamps (#310) 2026-07-01 17:15:07 +08:00
test_faiss_index_maintenance.py feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
test_file_catalog.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_file_graph.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_file_store_consistency.py feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
test_http_client.py feat(benchmark): add lme benchmark steps (#326) 2026-07-07 18:53:39 +09:00
test_injected_job_kwargs.py feat(Step tools): add white/black path prefix permission filtering to read, edit, write (#391) 2026-07-27 17:20:21 +08:00
test_job.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_json_file_chunker.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_jsonl_file_chunker.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_keyword_index.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_link_expansion.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_local_embedding_store.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_logging_config.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_markdown_file_chunker.py fix: enforce markdown chunk byte limits (#370) 2026-07-17 22:03:58 +08:00
test_outbound_proxy.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_packaging.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_path_permission.py feat(Step tools): add white/black path prefix permission filtering to read, edit, write (#391) 2026-07-27 17:20:21 +08:00
test_prompt_handler.py feat(benchmark): add lme benchmark steps (#326) 2026-07-07 18:53:39 +09:00
test_read_image_steps.py docs: rename vault_dir to workspace_dir in documentation and examples (#286) 2026-06-22 16:58:57 +08:00
test_read_with_neighbors.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_reme_cli.py feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
test_runtime_context.py feat(benchmark): add lme benchmark steps (#326) 2026-07-07 18:53:39 +09:00
test_search_step.py feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) 2026-07-22 17:17:23 +08:00
test_service.py refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
test_service_utils.py feat(benchmark): add lme benchmark steps (#326) 2026-07-07 18:53:39 +09:00
test_shell_step.py feat(core): add shell execution and runtime memory status (#344) 2026-07-14 16:31:41 +08:00
test_source_format_merge.py feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) 2026-07-22 17:17:23 +08:00
test_status_step.py feat(core): add shell execution and runtime memory status (#344) 2026-07-14 16:31:41 +08:00
test_token_usage.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_tokenizer.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00
test_utils.py feat(evaluation): track job calls and agent token usage in benchmarks (#406) 2026-08-04 11:42:18 +08:00
test_wikilink_utils.py docs: rename vault_dir to workspace_dir in documentation and examples (#286) 2026-06-22 16:58:57 +08:00
test_write_metadata_lock.py init: reme version 0.4.0 (#284) 2026-06-22 15:41:19 +08:00