Issue #1004 named a real gap: agents leave large mixed diffs, and nothing in the catalog shapes them into commits before they exist. Built independently from public sources. - commit_planner.py groups the working tree in build-safe order (manifest with lockfile, moves, formatting, each change with its tests, docs, CI), holds back .env/keys/tfstate, and flags a move whose callers' import edits sit in another commit, because that commit would not build. A plain mv is detected as a rename. - secret_gate.py blocks staged or pushed secrets, sensitive files and conflict markers with redacted output and reasoned waivers. On 174k lines added to this repo it raised 7 findings, all fake fixtures. - message_linter.py blocks vague subjects, large changes with no why and fixup commits, and enforces Conventional Commits only where the repo already uses them. Standalone plugin at 2.14.5, mirrored into engineering-advanced-skills (bumped to 2.14.8, above the unreleased 2.14.7 on dev). The bundle's G12 baseline grows by the new skill's 279 listing chars. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8KYFJm9vze5Kqnyb5yvFs |
||
|---|---|---|
| .. | ||
| cases | ||
| fixtures | ||
| common.py | ||
| generate_trigger_cases.py | ||
| inventory.py | ||
| README.md | ||
| report.py | ||
| run_evals.py | ||
Skill evals
Do our skills change what Claude does? These scripts answer that with the real Claude Code CLI,
not by reading the SKILL.md files. They grew out of the v2.14 audit harness and run nightly in
.github/workflows/skill-evals.yml.
| Eval | Question | Cases |
|---|---|---|
| Trigger | Does the skill fire on realistic prompts that don't name it, and stay quiet on a near miss? | cases/trigger_cases.json: every installable skill (384 unique, dual-published copies counted once), 2 positive prompts + 1 near miss each |
| Behaviour | With the skill's plugin loaded, does the answer pass more of a checklist than without it? | cases/behaviour_cases.json + fixtures/: 6 cases from the v2.14 audit |
A trigger run is claude -p <prompt> --plugin-dir <the skill's plugin> --max-turns 2 in an empty
folder with a throwaway config dir, so nothing installed on the machine and no CLAUDE.md leaks in.
It counts as fired when the transcript has a Skill call for that skill. A sibling skill or command
from the same plugin firing is counted separately (PLUGIN_ONLY), because it often means the
description is fine and the plugin routed the request elsewhere.
A behaviour run copies the fixture folder, runs the task twice (with the plugin, with none), and a blind grader scores each transcript against the assertions with a JSON schema. The grader does not know which arm it is reading.
Cost and the nightly default
Measured from the v2.14 audit transcripts (Sonnet, Claude Code 2.1.3xx):
| Run | Mean cost |
|---|---|
| Trigger run, only the skill's plugin loaded | $0.08 |
| Trigger run, all 105 plugins loaded | $0.15 |
| Behaviour pair (with + without) | $0.19 |
| Grading a behaviour pair | $0.21 |
A full sweep of 384 skills, 3 prompts each, 2 runs per prompt, is about $185. Running that every
night would be about $5,500 a month, so the default is a rotation: 15 skills a night, 2 runs per
prompt, plus 2 behaviour cases, capped at $10 a night. That is about $8 a night, $240 a month,
and covers the whole library every 26 nights. Every run also passes --max-budget-usd, and no new
run starts once the cap is spent. Raise --count for faster coverage, use --runs 1 to halve
cost (single runs are noisy), or --scenario full to test the skill against all the others.
Running it
python3 scripts/evals/inventory.py # coverage
python3 scripts/evals/run_evals.py --plan # tonight's slice, nothing runs
python3 scripts/evals/run_evals.py --skills engineering/skills/rag-architect --runs 1
python3 scripts/evals/report.py eval-out/results.json --scoreboard board/
run_evals.py writes eval-out/results.json and every transcript under eval-out/transcripts/.
report.py prints a Markdown summary and, with --scoreboard, folds the run into
scoreboard.json and SCOREBOARD.md, flagging a skill that used to fire and now doesn't, a new
near-miss false fire, or a behaviour lift that dropped.
In CI the scoreboard lives on the eval-results branch (one commit per night, with that night's
results under runs/<date>/), and transcripts are kept as a workflow artifact for 30 days.
Enabling the nightly job
- Credentials, either one:
CLAUDE_CODE_OAUTH_TOKEN(recommended for maintainers on a Pro/Max plan): runclaude setup-tokenlocally and store the token as a repo secret.claude.ymlandclaude-code-review.ymlalready use this secret. Runs then draw on the subscription's usage limits instead of a bill, and the dollar figures in this file become a usage meter.ANTHROPIC_API_KEY: pay per token at the prices above; set a spend limit in the Console.
- Set the repo variable
SKILL_EVALS_ENABLEDtotrue. - Optional variables:
SKILL_EVALS_BUDGET_USD(default 10),SKILL_EVALS_MODEL(defaultsonnet),CLAUDE_CODE_VERSION(default2.1.308, the npmstabletag when this was written).
Until step 2, only a manual "Run workflow" starts it. On a subscription, the nightly runs share
the same usage limits as your interactive sessions; the run starts at 03:17 UTC so it lands
outside working hours, and lowering --count or the budget trades coverage for headroom.
Adding cases
- Trigger:
python3 scripts/evals/generate_trigger_cases.py --skills <path>drafts cases for a new skill with Claude (about $0.01 each). Read the draft: the near miss must not be something the skill's description says it does. Hand-written cases carry"origin": "v2.14-audit"; drafted ones"generated". Fixing a generated case means editing it and changing the origin to"reviewed". - Behaviour: add an entry to
cases/behaviour_cases.jsonand a fixture folder. Write assertions a strong model without the skill would plausibly miss. On the current cases the grader often scores both arms full marks, so they show the skill does no harm but can't show a lift; harder cases are where the claim gets made.
Fixtures are deliberately broken inputs (an SQL-injectable route, a settings file with a fake database password); they are test data, not code to run.