scriptoriumadmin/claude-skills
1
0
Fork 0
mirror of https://github.com/alirezarezvani/claude-skills.git synced 2026-10-09 03:17:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
claude-skills/engineering/agenthub/skills/eval/SKILL.md
Claude aecfb8e0bb
feat(skills): wave-3 optimization — domain overhauls per newgen audit 
marketing: context-file unified on .claude/product-marketing-context.md;
ai-seo merged into aeo (2 new cited references, folder deleted); index +
marketing-ops routers rebuilt honestly; 24 orphan scripts wired with exact
CLIs; prompt-engineer-toolkit stub references rewritten with cited content;
Meta 20%-rule + GA4 terminology freshness; 5 zips + 3 planning docs removed

c-level-advisor: role registry 9->14 across all 6 routing surfaces; decision
memory unified on ~/.claude/decisions/{raw,approved}; onboarding schema
canonicalized; 12 phantom commands resolved; index repaired (33/37/68 real
counts); ma-playbook sourced + verification loop; 28 trigger descriptions

engineering(+team): agent-designer 279->76 lines and rag-architect 318->71
lines rebuilt around their tools (stale ada-002/pricing gone); release-manager
merged into changelog-generator (version_bumper + hotfix refs moved, crashing
release_planner dropped); 6 skills' orphan scripts wired; ms365 tools gained
real CLIs; 5 brochure skills de-filled; 4 unreferenced zips removed;
bundle counts trued (25->37, 23->32); 18 trigger descriptions

product/research/compliance/bizops: apple-hig-expert rebuilt around
hig_checker's real CLI with web-verified facts; notebooklm re-verified against
live product; 5 index skills converted to honest routers; research-summarizer
repaired with explicit lane statement; 8 over-1024 descriptions compressed;
9 sub-skills gained fenced CLI examples; GDPR one-month (Art. 12(3)) with
calendar-month deadline math; MDR PSUR table per Art. 86(1); 12 ra-qm zips
removed; 24 trigger descriptions

Verified: check_paths 0 findings; check_dual_publish 0 drifted; smoke 581/581;
check_plugin_json 77 OK; compileall rc=0; all descriptions <=1024 chars

https://claude.ai/code/session_019AJddAL1NADWMXsy1qNPQF
2026-06-11 03:58:41 +00:00

2.4 KiB
Raw Permalink Blame History

name description command
eval Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents. /hub:eval

/hub:eval — Evaluate Agent Results

Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.

Usage

/hub:eval                           # Eval latest session using configured criteria
/hub:eval 20260317-143022           # Eval specific session
/hub:eval --judge                   # Force LLM judge mode (ignore metric config)

What It Does

Metric Mode (eval command configured)

Run the evaluation command in each agent's worktree:

python {skill_path}/scripts/result_ranker.py \
  --session {session-id} \
  --eval-cmd "{eval_cmd}" \
  --metric {metric} --direction {direction}

Output:

RANK  AGENT       METRIC      DELTA      FILES
1     agent-2     142ms       -38ms      2
2     agent-1     165ms       -15ms      3
3     agent-3     190ms       +10ms      1

Winner: agent-2 (142ms)

LLM Judge Mode (no eval command, or --judge flag)

For each agent:

  1. Get the diff: git diff {base_branch}...{agent_branch}
  2. Read the agent's result post from .agenthub/board/results/agent-{i}-result.md
  3. Compare all diffs and rank by:
    • Correctness — Does it solve the task?
    • Simplicity — Fewer lines changed is better (when equal correctness)
    • Quality — Clean execution, good structure, no regressions

Present rankings with justification.

Example LLM judge output for a content task:

RANK  AGENT    VERDICT                               WORD COUNT
1     agent-1  Strong narrative, clear CTA            1480
2     agent-3  Good data points, weak intro           1520
3     agent-2  Generic tone, no differentiation       1350

Winner: agent-1 (strongest narrative arc and call-to-action)

Hybrid Mode

  1. Run metric evaluation first
  2. If top agents are within 10% of each other, use LLM judge to break ties
  3. Present both metric and qualitative rankings

After Eval

  1. Update session state:
python {skill_path}/scripts/session_manager.py --update {session-id} --state evaluating
  1. Tell the user:
    • Ranked results with winner highlighted
    • Next step: /hub:merge to merge the winner
    • Or /hub:merge {session-id} --agent {winner} to be explicit