GitNexus/gitnexus/test
Gergő Magyar f6127c5aee
feat(eval): measure cost drivers in evaluation summaries (#3773)
* feat(eval): add cost diagnostics and trials

* style(eval): format retrieval case fixtures

* fix(eval): address review and CI failures

Refs #3773. Validate locked uv commands, model aliases and oversized budgets. Stop the fake MCP server before releasing its listener and replace the wall-clock staleness assertion with a concurrent result check.

Note: pre-existing failure in test_oracle_assets.py reference-checkout type resolution not addressed by this PR.

* feat(eval): publish detailed run measurements

Collect per-sweep provider ledgers and publish sanitized measurements in the Actions summary and artifacts. Validate experiment tasks with the runner and withhold incomplete token totals. Refs #3773.

Note: pre-existing failure in test_oracle_assets.py reference-checkout type resolution not addressed by this PR.

* fix(eval): validate inputs and quoted commands

Reject non-mapping task YAML before reading fields. Attribute literal quoted CLI queries while preserving Bash attribution for shell operators and expansions. Refs #3773.

* fix(eval): count empty filtered query results

* fix(eval): handle alternate event content

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
2026-10-30 12:58:34 +00:00
..
fixtures feat: add the test scenario map and a Vitest 5 mutation runner (#3686) 2026-10-27 16:46:58 +00:00
helpers feat(ci): rebalance test shards from recorded timings (#3748) 2026-10-28 11:58:29 +00:00
integration feat(ingestion): opt-in indexing of named dot directories (GITNEXUS_INCLUDE_DOT_DIRS) (#3745) 2026-10-29 07:02:45 +00:00
unit feat(eval): measure cost drivers in evaluation summaries (#3773) 2026-10-30 12:58:34 +00:00
utils feat(storage): add configurable index storage and content retention tiers (#3060) 2026-09-12 20:31:55 +00:00