GitNexus/eval/tests
Gergő Magyar 52b06b2642
Some checks failed
CodeQL / Analyze (python) (push) Waiting to run
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
Skill copy sync / shipped skills drift guard (push) Has been cancelled
fix(eval): stop using --bare for arms that need Skill or MCP tools (#2584)
--bare hard-disables the Skill tool and every mcp__* tool by Claude
Code design (confirmed against the pinned 2.1.214 binary; --allowedTools
cannot restore what --bare removes). Every workflow_bench arm except
baseline_nomcp needs Skill and/or GitNexus MCP tools, so every one of
those sessions has been silently unable to invoke gitnexus-plan/work/
review or the CE comparator skills -- the last skill-evolution run
(gen 0) scored 0/3 resolution on both arms across every task with
error_kind "skill-not-invoked", not because the candidate was bad but
because the harness could never invoke either arm's skill at all.

Only baseline_nomcp keeps --bare (it explicitly wants zero Skill/MCP
access anyway). The rest drop --bare and rely on ANTHROPIC_API_KEY
alone; the sandboxed HOME has no OAuth/keychain state to conflict
with it, and there's no committed .claude/settings.json in this repo
for dropping --bare to newly pick up.

Outside --bare the built-in toolset defaults to everything (WebFetch,
Task, subagents, ...), and --allowedTools only pre-approves within
whatever's available -- it doesn't narrow it. Added --tools for
non-bare sessions so the intended tool scope is still enforced instead
of silently widening.


Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 21:26:11 +01:00
..
__init__.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
conftest.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_ce_plugin_runtime.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_errors.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_evolve.py feat(eval): run the skill-evolution loop online (#2571) 2026-07-20 05:37:07 +01:00
test_mcp_bridge.py fix: start MCP bridge correctly when using npx (#1114) 2026-04-27 18:19:02 +01:00
test_oracle_assets.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_parse_run_id.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_process_control.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_promotion_apply.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_property_based.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_proposer_sandbox.py fix(eval): pre-approve proposer tools under Claude Code 2.1.214 ENV_SCRUB hardening (#2576) 2026-07-20 11:25:49 +01:00
test_runner_hardening.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_sanitized_graph.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_task_assets.py fix(eval): size the buffered-fallback budget for the real graph index 2026-07-20 14:42:50 +00:00
test_tool_scripts.py docs: agent development framework, GitHub templates, eval refactor (#479) 2026-03-25 06:48:41 +00:00
test_workflow_bench.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_workflow_bench_evolution.py feat(skills): GitNexus Engineering Tool Kits (#2566) 2026-07-19 15:07:24 +01:00
test_workflow_bench_sessions.py fix(eval): stop using --bare for arms that need Skill or MCP tools (#2584) 2026-07-20 21:26:11 +01:00