scriptoriumadmin/claude-skills
1
0
Fork 0
mirror of https://github.com/alirezarezvani/claude-skills.git synced 2026-08-28 04:24:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Merge pull request #970 from alirezarezvani/dev
Some checks failed
Release / Tag + GitHub Release (push) Has been cancelled
Details
Deploy Documentation to Pages / build (push) Has been cancelled
Details
Sync Codex Skills Symlinks / sync (push) Has been cancelled
Details
Deploy Documentation to Pages / deploy (push) Has been cancelled
Details

Browse source
This commit is contained in:
Alireza Rezvani 2026-08-21 07:23:07 +02:00 • committed by GitHub
commit 98180dafc4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
67 changed files with 9917 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
.claude-plugin/marketplace.json
View file

@ -8,7 +8,7 @@
"homepage": "https://github.com/alirezarezvani/claude-skills",
"repository": "https://github.com/alirezarezvani/claude-skills",
"metadata": {
"description": "362 production-ready skills across 18 domains (engineering, engineering-core, marketing, product, c-level, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 644 Python tools, 741 reference guides, 102 agents (cs-* + personas), 116 slash commands across 88 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
"description": "364 production-ready skills across 18 domains (engineering, engineering-core, marketing, product, c-level, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 667 Python tools, 750 reference guides, 104 agents (cs-* + personas), 120 slash commands across 90 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
"version": "2.11.2"
},
"plugins": [
@ -862,6 +862,26 @@
],
"category": "development"
},
{
"name": "book-to-skill",
"source": "./engineering/book-to-skill",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills: a master SKILL.md with core frameworks and a topic index, on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Ships 4 stdlib-only tools (multi-format extractor with invisible-Unicode sanitization, four-family generated-skill validator covering frontmatter/safety/budget/index, pre- and post-flight token budget estimator with a worth-converting verdict, and a plugin emitter that wraps a compiled skill as an installable claude-skills package behind a rights gate), 5 references citing 7-8 sources each, a cs-book-to-skill agent, and /cs:book-to-skill + /cs:book-to-plugin commands. Derived from virgiliojr94/book-to-skill (MIT).",
"version": "1.0.0",
"author": {
"name": "Alireza Rezvani"
},
"keywords": [
"knowledge-base",
"book-to-skill",
"document-extraction",
"pdf",
"epub",
"progressive-disclosure",
"skill-authoring",
"meta-skill"
],
"category": "development"
},
{
"name": "workflow-builder",
"source": "./engineering/workflow-builder",
@ -941,6 +961,27 @@
],
"category": "development"
},
{
"name": "memory-engineering",
"source": "./engineering/memory-engineering",
"description": "Engineer an agent's forgetting, not just its remembering. Four deterministic stdlib scripts implement the four lenses of agent memory: a cost profiler that splits construction from query spend and reports cost per correct answer (construction energy exceeds total query energy across 300 queries in the Stanford characterization); an architecture picker that scores the four paradigm families — long-context, flat RAG, structure-augmented RAG, agentic — disqualifies on hard constraints, names the cost the winning choice makes you pay, and refuses to pick when the top two tie; a density auditor that classifies every record in a real memory directory or JSONL export as FACT / SKILL / LOG / PROSE, finds near-duplicates, and flags stale and time-relative wording; and a forgetting-policy linter that fails any design with no forgetting rule or that auto-merges contradictions. Ships cs-memory-engineer agent, /cs:memory-engineering and /cs:forgetting-audit commands, 4 references citing 7 sources each, and a seven-question forcing worksheet.",
"version": "2.11.2",
"author": {
"name": "Alireza Rezvani"
},
"keywords": [
"agent-memory",
"memory-engineering",
"forgetting-policy",
"context-engineering",
"rag",
"kv-cache",
"retention",
"write-path-cost",
"engineering"
],
"category": "development"
},
{
"name": "grill-me",
"source": "./engineering/grill-me",

16
.codex/skills-index.json
View file

@ -3,7 +3,7 @@
"name": "claude-code-skills",
"description": "Production-ready skill packages for AI agents - Marketing, Engineering, Product, C-Level, PM, and RA/QM",
"repository": "https://github.com/alirezarezvani/claude-skills",
"total_skills": 360,
"total_skills": 362,
"skills": [
{
"name": "business-growth-skills",
@ -983,6 +983,12 @@
"category": "engineering-advanced",
"description": "Read, write, and browse the AgentHub message board for agent coordination. Use when the user runs /hub:board or asks to post, read, or inspect coordination messages between competing AgentHub agents."
},
{
"name": "book-to-skill",
"source": "../../engineering/book-to-skill/skills/book-to-skill",
"category": "engineering-advanced",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills \u2014 extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin."
},
{
"name": "browser-automation",
"source": "../../engineering/skills/browser-automation",
@ -1205,6 +1211,12 @@
"category": "engineering-advanced",
"description": "Design and ship production-ready MCP (Model Context Protocol) servers from OpenAPI contracts instead of hand-written tool wrappers. Python and TypeScript support, schema validation, safe evolution. Use when exposing an existing API as an MCP server, building tool integrations for Claude or Codex or Cursor, or scaffolding an MCP project from scratch."
},
{
"name": "memory-engineering",
"source": "../../engineering/memory-engineering/skills/memory-engineering",
"category": "engineering-advanced",
"description": "Use when designing, reviewing, or paying for an agent memory system \u2014 adding memory to an agent, choosing between long-context / RAG / graph / agentic memory, auditing what a CLAUDE.md or memory directory actually holds, deciding what to keep and what to expire, or when a memory store keeps growing and nobody has said what leaves it. Prices the write path, picks which cost to pay, classifies records as facts / skills / logs, and refuses a design that has no forgetting policy."
},
{
"name": "merge",
"source": "../../engineering/agenthub/skills/merge",
@ -2203,7 +2215,7 @@
"description": "Software engineering and technical skills"
},
"engineering-advanced": {
"count": 83,
"count": 85,
"source": "../../engineering",
"description": "Advanced engineering skills - agents, RAG, MCP, CI/CD, databases, observability"
},

1
.codex/skills/book-to-skill Symbolic link
View file

@ -0,0 +1 @@
../../engineering/book-to-skill/skills/book-to-skill

1
.codex/skills/memory-engineering Symbolic link
View file

@ -0,0 +1 @@
../../engineering/memory-engineering/skills/memory-engineering

9
.gemini/skills-index.json
View file

@ -1,7 +1,7 @@
{
"version": "1.0.0",
"name": "gemini-cli-skills",
"total_skills": 434,
"total_skills": 435,
"skills": [
{
"name": "README",
@ -1158,6 +1158,11 @@
"category": "engineering-advanced",
"description": "Read, write, and browse the AgentHub message board for agent coordination. Use when the user runs /hub:board or asks to post, read, or inspect coordination messages between competing AgentHub agents."
},
{
"name": "book-to-skill",
"category": "engineering-advanced",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills \u2014 extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin."
},
{
"name": "browser-automation",
"category": "engineering-advanced",
@ -2208,7 +2213,7 @@
"description": "Engineering resources"
},
"engineering-advanced": {
"count": 84,
"count": 85,
"description": "Engineering-advanced resources"
},
"finance": {

1
.gemini/skills/book-to-skill/SKILL.md Symbolic link
View file

@ -0,0 +1 @@
../../../engineering/book-to-skill/skills/book-to-skill/SKILL.md

1
.hermes/skills/claude-skills/engineering/book-to-skill Symbolic link
View file

@ -0,0 +1 @@
../../../../engineering/book-to-skill/skills/book-to-skill

7
.hermes/skills/claude-skills/skills-index.json
View file

@ -1,6 +1,6 @@
{
"source": "claude-code-skills",
"total_skills": 305,
"total_skills": 306,
"domains": {
"engineering": [
{
@ -23,6 +23,11 @@
"description": "Use when the user asks to generate API tests, create integration test suites, test REST endpoints, or build contract tests.",
"path": "engineering/api-test-suite-builder"
},
{
"name": "book-to-skill",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills \u2014 extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin.",
"path": "engineering/book-to-skill"
},
{
"name": "browser-automation",
"description": "Use when the user asks to automate browser tasks, scrape websites, fill forms, capture screenshots, extract structured data from web pages, or build web automation workflows. NOT for testing \u2014 use playwright-pro for that.",

1
.vibe/skills/claude-skills/engineering/book-to-skill Symbolic link
View file

@ -0,0 +1 @@
../../../../engineering/book-to-skill/skills/book-to-skill

7
.vibe/skills/claude-skills/skills-index.json
View file

@ -1,6 +1,6 @@
{
"source": "claude-code-skills",
"total_skills": 338,
"total_skills": 339,
"domains": {
"engineering": [
{
@ -23,6 +23,11 @@
"description": "Use when the user asks to generate API tests, create integration test suites, test REST endpoints, or build contract tests.",
"path": "engineering/api-test-suite-builder"
},
{
"name": "book-to-skill",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills \u2014 extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin.",
"path": "engineering/book-to-skill"
},
{
"name": "browser-automation",
"description": "Use when the user asks to automate browser tasks, scrape websites, fill forms, capture screenshots, extract structured data from web pages, or build web automation workflows. NOT for testing \u2014 use playwright-pro for that.",

108
CHANGELOG.md
View file

@ -5,7 +5,113 @@ All notable changes to the Claude Skills Library will be documented in this file
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased] — fable-goal: ramble → autonomous /goal prompt (this PR)
## [Unreleased] — book-to-skill: document → knowledge-base skill → plugin (this PR)
### Added — `engineering/book-to-skill`
Derived from [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
(MIT). Compiles a book, documentation folder, or spec collection (PDF, EPUB, DOCX,
HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into an agent skill: a resident master
`SKILL.md` (core frameworks + chapter index + topic index, capped at 4k tokens) plus
on-demand `chapters/chNN-*.md`, `glossary.md`, `patterns.md`, and a decision
`cheatsheet.md`. The agent reads the core, then one chapter — never the whole source
again.
The extraction library (`scripts/book_to_skill/` — 12 modules including 7 per-format
parsers) is vendored close to verbatim and keeps upstream's format chains, chapter
detection across Latin/Roman/Chinese/Thai/Korean heading styles, invisible-Unicode
(Trojan Source) sanitization, and the DOCX entity-expansion guard.
**25 numbered deviations** are recorded in `engineering/book-to-skill/README.md`,
which is the authoritative list. Highlights:
- **(5) No implicit installs.** `--install-missing` defaults to `report` — it prints
the pip command and uses the stdlib fallback, where upstream prompts on a TTY and
runs `pip install` into the caller's environment.
- **(6) Rights gate.** `skill_plugin_emitter.py --distribution shareable` refuses
without `--rights` from `public-domain|open-license|internal-docs|author-permission`.
`fair-use` is deliberately excluded — a defence, not a licence.
- **(10) Merged, extended validator.** Upstream's two validators become one
four-family gate, adding **budget** (token caps) and **index** (dead chapter links,
unindexed chapter files, dangling topic refs) — the failure that silently breaks
navigation while the skill still looks complete, and which upstream did not check.
- **(11)** Folded YAML scalars now parse, so a wrapped description no longer
under-reports its length past the 1024-char cap.
- **(12)** `discovery_tax.py` → `token_budget_estimator.py`: optional `tiktoken` path
dropped, post-flight budget audit added, and an explicit **worth-converting
verdict** that says "just read it" when the source is under ~3× the compiled skill.
- **(15) Private, per-invocation working directory.** Upstream's fixed
`<tempdir>/book_skill_work` is CWE-377/CWE-59 on a shared host — a local user can
pre-create it and plant a symlink named `full_text.txt`, and `Path.write_text` follows
symlinks. Now a fresh `mkdtemp` (0700, unpredictable) with 0600 artifacts; an explicit
`--workdir` is symlink-refused and mode-restricted. Also fixes a real bug: `parsers/calibre.py`
read a module-level path constant and so ignored `--workdir` entirely.
- **(17) Zip-of-XML hardening generalized, plus decompression-bomb caps.** Upstream's
DTD/entity guard covered DOCX only; EPUB's `ebooklib` path handed the archive straight to a
third-party XML stack. The guard now lives in `book_to_skill/zip_safety.py` and runs for
both, and every archive read checks declared size and compression ratio before
decompressing — a 200 MB zip bomb is refused at ~14 MB peak RSS.
- **(18) Packaging refuses a source tree containing symlinks.** `shutil.copytree` defaults to
following links, which would bake a link target's real content into a package that may be
emitted as `--distribution shareable`. `_assert_no_symlinks()` walks the whole tree and
refuses, before the validation branch so `--skip-validation` cannot bypass it.
- **(19) The magic-byte sniff path goes through the size budget too.** Unknown-extension
files read a `mimetype` member with a bare `zf.read()` before any format was chosen —
ahead of every check in `zip_safety.py`. Now routed through `safe_read()`; a 200 MB
extensionless bomb is refused at ~15 MB peak RSS.
- **(20) Emitter correctness and scope.** `--author`/`--author-url` now reach the printed
marketplace entry; a post-copy re-walk deletes the package if a symlink appears during the
copy (closing the check-then-act window); and `source.license_scope` records that the
top-level `license` covers the scaffolding, not the compiled notes.
- **(21-24) Skill-quality audit** — read as a skill rather than as code. SKILL.md's
quick-start referenced `$WORKDIR`/`$SKILLS_HOME` without defining them (traceback if
followed literally; all five steps now execute verbatim); `token_budget_estimator.py
--skill-dir <typo>` reported a clean audit at exit 0 and now refuses; `epub.py`'s
`except (KeyError, Exception)` was silently disarming the zip-size refusal at that call
site; and `tool | head` no longer tracebacks.
- **(25) Workdir race closed by fd pinning.** `mkdir(exist_ok=True)` does not raise on a
symlink-to-directory (its exists-branch follows symlinks), and a file inside a swapped
directory is not itself a link — so the artifact-level check could not back up the
directory-level one. The directory is now pinned with `O_NOFOLLOW|O_DIRECTORY` and both
artifacts written through that fd; `_write_private` creates with `O_CREAT|O_EXCL|O_NOFOLLOW`
at 0600. Verified against a live mid-write directory swap.
**Repo-native addition with no upstream counterpart — Step 11 / `/cs:book-to-plugin`.**
Upstream stops at a bare folder in `~/.claude/skills/`, which this library cannot route
to. `skill_plugin_emitter.py` wraps a compiled skill as a full plugin package (manifest
+ `cs-<slug>` agent + `/cs:<slug>` command + README) and prints the marketplace entry.
It never edits `marketplace.json` itself, refuses to wrap a skill carrying validation
errors, and guards its `--force` overwrite path against symlinks, paths outside the
destination root, and directories that are not plugin packages.
Ships 4 stdlib-only tools (all `--help` / `--sample` / `--output json`), 5 references
citing 7-8 sources each, 3 asset templates, a `cs-book-to-skill` agent, and
`/cs:book-to-skill` + `/cs:book-to-plugin`.
### Changed — `engineering/write-a-skill`
Cross-linked to the new skill: "author first, compile second" — `write-a-skill` authors
from expertise in your head, `book-to-skill` compiles from a document on disk.
### Changed — engineering harness manifest
Regenerated `engineering/agent-harness/.../assets/harnesses/engineering.json`. Picked up
`book-to-skill` plus three skills that had drifted out of the manifest (`minimalist`,
`skillopt-sleep`, `strict-api`): `skill_count` 81 → 85.
### Changed — counters
skills 362 → 363, tools 644 → 663, refs 741 → 746, agents 102 → 103, commands
116 → 118, plugins 88 → 89 (derived via `scripts/derive_counters.py --check`).
---
## [Unreleased] — fable-goal: ramble → autonomous /goal prompt (previous PR)
### Added — `productivity/fable-goal`

71
CLAUDE.md
View file

File diff suppressed because one or more lines are too long

20
README.md
View file

@ -1,6 +1,6 @@
# Claude Code Skills & Plugins — Agent Skills for Every Coding Tool
**362 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
**364 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
The most comprehensive open-source library of Claude Code skills and agent plugins — also works with OpenAI Codex, Gemini CLI, Cursor, and 9 more coding agents. Reusable expertise packages covering engineering, DevOps, marketing (incl. AEO — Answer Engine Optimization for LLM citation), security (PreToolUse hooks), compliance, C-level advisory (incl. founder-mode CFO/CMO/CRO/CPO/COO/CHRO/CISO/GC/CDO/CAIO/CCO/VPE personas + 21 /cs:* slash commands), productivity (capture/email/reflect/weekly-review/deep-work/meetings), an academic research stack (litreview/grants/dossier/patent/syllabus/pulse/notebooklm/deep-research + hybrid router), and enterprise Research Operations (clinical-research/research-finance/market-research/product-research, v2.9.0).
@ -10,10 +10,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi
[^vibe]: Mistral Vibe is also **BYO-sync tier**: the repo ships a pre-generated `.vibe/skills/claude-skills/` tree, run `./scripts/vibe-install.sh` once locally to install into `~/.vibe/skills/`. Same agentskills.io SKILL.md standard — no format conversion. Docs: <https://docs.mistral.ai/mistral-vibe/agents-skills>.
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](https://opensource.org/licenses/MIT)
[![Skills](https://img.shields.io/badge/Skills-362-brightgreen?style=for-the-badge)](#skills-overview)
[![Agents](https://img.shields.io/badge/Agents-102-blue?style=for-the-badge)](#agents)
[![Skills](https://img.shields.io/badge/Skills-364-brightgreen?style=for-the-badge)](#skills-overview)
[![Agents](https://img.shields.io/badge/Agents-104-blue?style=for-the-badge)](#agents)
[![Personas](https://img.shields.io/badge/Personas-7-purple?style=for-the-badge)](#personas)
[![Commands](https://img.shields.io/badge/Commands-116-orange?style=for-the-badge)](#commands)
[![Commands](https://img.shields.io/badge/Commands-120-orange?style=for-the-badge)](#commands)
[![Stars](https://img.shields.io/github/stars/alirezarezvani/claude-skills?style=for-the-badge)](https://github.com/alirezarezvani/claude-skills/stargazers)
[![SkillCheck Validated](https://img.shields.io/badge/SkillCheck-Validated-4c1?style=for-the-badge)](https://getskillcheck.com)
@ -26,10 +26,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi
Claude Code skills (also called agent skills or coding agent plugins) are modular instruction packages that give AI coding agents domain expertise they don't have out of the box. Each skill includes:
- **SKILL.md** — structured instructions, workflows, and decision frameworks
- **Python tools** — 644 CLI scripts (all stdlib-only, zero pip installs)
- **Reference docs** — 741 templates, checklists, and domain-specific knowledge files
- **Python tools** — 667 CLI scripts (all stdlib-only, zero pip installs)
- **Reference docs** — 750 templates, checklists, and domain-specific knowledge files
**One repo, thirteen platforms.** Works natively as Claude Code plugins, Codex agent skills, Gemini CLI skills, Hermes Agent skills, Mistral Vibe skills, and converts to more tools via `scripts/convert.sh`. All 644 Python tools run anywhere Python runs.
**One repo, thirteen platforms.** Works natively as Claude Code plugins, Codex agent skills, Gemini CLI skills, Hermes Agent skills, Mistral Vibe skills, and converts to more tools via `scripts/convert.sh`. All 667 Python tools run anywhere Python runs.
### Skills vs Agents vs Personas
@ -150,12 +150,12 @@ Run `./scripts/convert.sh --tool all` to generate tool-specific outputs locally.
## Skills Overview
**362 skills across 18 domains:**
**364 skills across 18 domains:**
| Domain | Skills | Highlights | Details |
|--------|--------|------------|---------|
| **🔧 Engineering — Core** | 52 | Architecture, frontend, backend, fullstack, QA, DevOps, SecOps, AI/ML, data, Playwright Pro (test gen, flaky fix, migrations), self-improving agent (auto-memory curation), security suite, a11y audit, **named-persona-adversarial-review** (review via named engineering philosophies) | [engineering-team/](engineering-team/) |
| **⚡ Engineering — POWERFUL** | 84 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt) | [engineering/](engineering/) |
| **⚡ Engineering — POWERFUL** | 86 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin) | [engineering/](engineering/) |
| **🎯 Product** | 17 | Product manager, agile PO, strategist, UX researcher, UI design, landing pages, SaaS scaffolder, analytics, experiment designer, discovery, roadmap communicator, code-to-prd, apple-hig-expert | [product-team/](product-team/) |
| **📣 Marketing** | 48 | 8 pods: Content, SEO + AEO (`aeo` — E-E-A-T audit, citation tracking across 5 LLMs) + local (`local-seo-manager` — GBP/NAP/Map-Pack), CRO, Channels, Growth, Intelligence, Sales + context foundation + orchestration router | [marketing-skill/](marketing-skill/) |
| **🚀 Productivity** | 11 | `capture` (brain-dump-to-action), `email` pair (inbox-setup + inbox-triage), `reflect` (journal), `handoff` (Matt Pocock-inspired), `andreessen` (market-first decision mode), `roast` (5-angle idea panel → GO/RESHAPE/KILL), `fable-goal` (ramble → autonomous /goal prompt), `weekly-review` (GTD loop with refusal gate), `deep-work` (time-blocking + shallow-work budget), `meetings` (cost gate + agenda + action items) | [productivity/](productivity/) |
@ -354,7 +354,7 @@ Yes. Skills work natively with 13 tools: Claude Code, OpenAI Codex, Gemini CLI,
No. We follow semantic versioning and maintain backward compatibility within patch releases. Existing script arguments, plugin source paths, and SKILL.md structures are never changed in patch versions. See the [CHANGELOG](CHANGELOG.md) for details on each release.
**Are the Python tools dependency-free?**
Yes. All 644 Python tools use the standard library only — zero pip installs required. Every skill's CLI entry point is verified to run with `--help` (most skills ship one script per tool; a few, like the vendored `engineering/skillopt-sleep` engine, ship a multi-module package behind a single `python -m` entry point).
Yes. All 667 Python tools use the standard library only — zero pip installs required. Every skill's CLI entry point is verified to run with `--help` (most skills ship one script per tool; a few, like the vendored `engineering/skillopt-sleep` engine, ship a multi-module package behind a single `python -m` entry point). A few tools — `engineering/book-to-skill`'s document extractors — can *optionally* use third-party parsers for higher-fidelity output, but every format falls back to a standard-library parser and nothing is installed implicitly.
**How do I create my own Claude Code skill?**
Each skill is a folder with a `SKILL.md` (frontmatter + instructions), optional `scripts/`, `references/`, and `assets/`. See the [Skills & Agents Factory](https://github.com/alirezarezvani/claude-code-skills-agents-factory) for a step-by-step guide.

132
engineering/agent-harness/skills/agent-harness/assets/harnesses/engineering.json
View file

@ -1,7 +1,7 @@
{
"schema": "agent-harness/manifest.v1",
"domain": "engineering",
"skill_count": 81,
"skill_count": 85,
"loop_defaults": {
"max_attempts_per_task": 3,
"max_loop_iterations": 12,
@ -412,6 +412,94 @@
"api-integration.md"
]
},
{
"name": "book-to-skill",
"path": "engineering/book-to-skill/skills/book-to-skill",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills \u2014 extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin.",
"tools": [
{
"script": "engineering/book-to-skill/skills/book-to-skill/scripts/book_skill_validator.py",
"wired": true,
"supports_sample": true,
"verification": [
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/book_skill_validator.py --help",
"expect_exit": 0,
"kind": "smoke"
},
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/book_skill_validator.py --sample",
"expect_exit": 0,
"kind": "sample"
}
]
},
{
"script": "engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py",
"wired": true,
"supports_sample": true,
"verification": [
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py --help",
"expect_exit": 0,
"kind": "smoke"
},
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py --sample",
"expect_exit": 0,
"kind": "sample"
}
]
},
{
"script": "engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py",
"wired": true,
"supports_sample": true,
"verification": [
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py --help",
"expect_exit": 0,
"kind": "smoke"
},
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py --sample",
"expect_exit": 0,
"kind": "sample"
}
]
},
{
"script": "engineering/book-to-skill/skills/book-to-skill/scripts/token_budget_estimator.py",
"wired": true,
"supports_sample": true,
"verification": [
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/token_budget_estimator.py --help",
"expect_exit": 0,
"kind": "smoke"
},
{
"cmd": "python3 engineering/book-to-skill/skills/book-to-skill/scripts/token_budget_estimator.py --sample",
"expect_exit": 0,
"kind": "sample"
}
]
}
],
"agentic_signals": {
"goal_intake": true,
"refusal_gate": true,
"verification": true,
"loop_discipline": true,
"close_out": false
},
"references": [
"document_extraction_pipeline.md",
"knowledge_extraction_canon.md",
"progressive_disclosure_budgets.md",
"rights_and_provenance.md"
]
},
{
"name": "caveman",
"path": "engineering/caveman/skills/caveman",
@ -1277,6 +1365,20 @@
"wiki-schema.md"
]
},
{
"name": "minimalist",
"path": "engineering/minimalist",
"description": "Use when the user asks to write code efficiently, avoid over-engineering, reduce dependencies, or prevent unnecessary abstractions. Enforces a strict efficiency ladder: YAGNI, reuse, stdlib, native platform, existing deps \u2014 before writing any new code.",
"tools": [],
"agentic_signals": {
"goal_intake": false,
"refusal_gate": false,
"verification": false,
"loop_discipline": true,
"close_out": false
},
"references": []
},
{
"name": "prompt-governance",
"path": "engineering/prompt-governance/skills/prompt-governance",
@ -1307,6 +1409,20 @@
"pretooluse_hook_canon.md"
]
},
{
"name": "skillopt-sleep",
"path": "engineering/skillopt-sleep/skills/skillopt-sleep",
"description": "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated CLAUDE.md and SKILL.md behind a held-out gate.",
"tools": [],
"agentic_signals": {
"goal_intake": false,
"refusal_gate": true,
"verification": false,
"loop_discipline": true,
"close_out": false
},
"references": []
},
{
"name": "agent-designer",
"path": "engineering/skills/agent-designer",
@ -3105,6 +3221,20 @@
"statistical-testing-concepts.md"
]
},
{
"name": "strict-api",
"path": "engineering/strict-api",
"description": "Use when the user says 'no hallucinations', 'verify APIs', 'reality check', or 'don't invent functions'. Prevents the agent from calling methods, imports, or variables that do not provably exist in the user's installed version.",
"tools": [],
"agentic_signals": {
"goal_intake": false,
"refusal_gate": false,
"verification": true,
"loop_discipline": false,
"close_out": false
},
"references": []
},
{
"name": "terraform-patterns",
"path": "engineering/terraform-patterns/skills/terraform-patterns",

21
engineering/book-to-skill/.claude-plugin/plugin.json Normal file
View file

@ -0,0 +1,21 @@
{
"name": "book-to-skill",
"description": "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills: a master SKILL.md with core frameworks and a topic index, on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Ships 4 stdlib-only tools (multi-format extractor with invisible-Unicode sanitization, four-family generated-skill validator, pre/post-flight token budget estimator, and a plugin emitter that wraps a compiled skill as an installable claude-skills package), 5 references citing 7-8 sources each (including the full step-by-step conversion workflow), a cs-book-to-skill agent, and /cs:book-to-skill + /cs:book-to-plugin commands. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a plugin.",
"version": "1.0.0",
"author": {
"name": "Alireza Rezvani",
"url": "https://alirezarezvani.com"
},
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/book-to-skill",
"repository": "https://github.com/alirezarezvani/claude-skills",
"license": "MIT",
"skills": [
"./skills/book-to-skill"
],
"attribution": {
"derived_from": "https://github.com/virgiliojr94/book-to-skill",
"original_author": "virgiliojr94",
"original_license": "MIT",
"derivation_note": "The extraction library (book_to_skill/ — config, exceptions, sanitize, dependencies, utils, and the 7 per-format parsers) is vendored from upstream and carries upstream's multi-format chains, chapter detection across 5 script families, and Unicode/XXE hardening. 25 numbered deviations are recorded in README.md, which is the authoritative list: the workflow was rewritten Claude-Code-first for this repo's conventions, the 3 upstream tools were reworked into 4 argparse CLIs with --help/--sample, install-on-import behaviour was replaced with a report-only default, and Step 11 (plugin emission with a rights gate) plus the validator's budget and index families are additions with no upstream counterpart."
}
}

35
engineering/book-to-skill/LICENSE Normal file
View file

@ -0,0 +1,35 @@
MIT License
Copyright (c) 2025 virgiliojr94
Copyright (c) 2026 Alireza Rezvani (modifications and additions)
The document-extraction library under
skills/book-to-skill/scripts/book_to_skill/ is vendored from
https://github.com/virgiliojr94/book-to-skill and is copyright virgiliojr94.
The workflow, the four command-line tools, the references, the assets, the
agent and the commands are modifications and additions by this repository,
released under the same terms. The numbered list in README.md under
"Deviations from upstream" records what changed.
This license covers the converter only. It does not cover the books or
documents processed with it, and it does not make a compiled skill
redistributable — see
skills/book-to-skill/references/rights_and_provenance.md.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

336
engineering/book-to-skill/README.md Normal file
View file

@ -0,0 +1,336 @@
# book-to-skill
Turn a book, a documentation folder, or a pile of specs into an agent skill.
A source goes in. What comes out is a knowledge base the agent loads on demand: a small
resident core with the author's named frameworks and a topic index, one chapter file per
chapter, a glossary, a patterns file, and a decision cheatsheet. The agent reads the core, then
one chapter — never the whole source again.
Derived from [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill) (MIT).
See **Deviations from upstream** below for the authoritative list of what changed.
---
## What it produces
| File | Contents | Budget |
|------|----------|--------|
| `SKILL.md` | Core frameworks + chapter index + topic index | < 4,000 tokens, resident |
| `chapters/chNN-*.md` | One summary per chapter | 800–3,000 tokens, on demand |
| `glossary.md` | Every significant term, alphabetized, with chapter | < 1,500 tokens |
| `patterns.md` | Techniques and design patterns with trade-offs | < 2,000 tokens |
| `cheatsheet.md` | Decision rules, thresholds, trade-off matrices | < 1,200 tokens |
Formats: PDF, EPUB, DOCX, HTML, Markdown, plain text, reStructuredText, AsciiDoc, RTF, and
MOBI/AZW/AZW3 with Calibre.
---
## Install
Already in this repository. Install as a marketplace plugin:
```
/plugin install book-to-skill@claude-code-skills
```
Or use the tools directly — they are self-contained under
`skills/book-to-skill/scripts/`.
**Dependencies:** none required. Every format degrades to a standard-library parser except
MOBI/AZW/AZW3, which need Calibre's `ebook-convert` on PATH. Optional packages
(`docling`, `pypdf`, `pdfminer.six`, `ebooklib`, `beautifulsoup4`, `python-docx`, `striprtf`)
raise extraction quality where installed; Poppler's `pdftotext` is the fastest PDF path.
Nothing is installed implicitly — run `extract_document.py --check` to see what is present and
get the exact install command.
---
## Use
```
/cs:book-to-skill ~/books/thinking-in-systems.pdf meadows-systems
/cs:book-to-skill ./docs/architecture/ '*.md' internal-arch
/cs:book-to-plugin ~/.claude/skills/meadows-systems --domain engineering
```
Or drive the tools yourself:
```bash
SKILL_ROOT=engineering/book-to-skill/skills/book-to-skill
WORKDIR=$(mktemp -d) # or omit --workdir and capture the path the tool prints
SLUG=<author-lastname>-<concept>
python3 "$SKILL_ROOT/scripts/extract_document.py" BOOK.pdf --mode technical --workdir "$WORKDIR"
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --full-text "$WORKDIR/full_text.txt"
# ... agent generates the skill ...
python3 "$SKILL_ROOT/scripts/book_skill_validator.py" ~/.claude/skills/"$SLUG"
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" --skill-dir ~/.claude/skills/"$SLUG" \
--dest ./engineering --dry-run
```
Every tool supports `--help`, `--sample` and `--output json`.
---
## Tools
| Tool | Does |
|------|------|
| `extract_document.py` | Multi-format extraction → `full_text.txt` + `metadata.json`. Strips invisible Unicode, blocks DOCX entity expansion, detects chapters across Latin, Roman, Chinese, Thai and Korean heading styles. `--check` reports the environment. |
| `book_skill_validator.py` | Four check families over a generated skill — **frontmatter** (host-lens rules), **safety** (injection phrasing, invisible Unicode, authority grants), **budget** (token caps), **index** (dead links, unindexed chapters, dangling topic refs). Errors block; `--strict` promotes warnings. |
| `token_budget_estimator.py` | Pre-flight: models context-dump vs. discovery-loop vs. compiled-skill cost on this source's real chapter sizes and prints a worth-converting verdict. Post-flight: file-by-file budget audit. |
| `skill_plugin_emitter.py` | Wraps a compiled skill as a claude-skills plugin (manifest + agent + command + README) and prints the marketplace entry. Refuses to emit a shareable package without a rights basis. |
---
## Why the tokens matter
A live document-reading agent pays a **discovery loop tax**: it reads the table of contents,
pulls the chapter it guessed at, then backtracks for a definition it turns out to need. Every
one of those lands in history. Dumping the whole source into context is worse — it is resident,
and re-billed every turn.
Compiling pays that navigation cost once. A query afterwards costs what the answer costs, not
what the source costs. `token_budget_estimator.py` models all three on the real token sizes of
your source and says which one wins — including when the honest answer is "this source is too
small; just read it."
---
## Deviations from upstream
**This numbered list is the authoritative record.** `plugin.json`'s
`attribution.derivation_note` summarizes it; if the two ever disagree, this list wins.
**Structural**
1. **Repo-native layout.** Upstream is a standalone repository (`book_to_skill/` package,
`scripts/`, `tools/`, `tests/`, `docs/`, `mkdocs.yml`, `pyproject.toml`, CI workflows). Here
it is one plugin: `skills/book-to-skill/{SKILL.md,scripts,references,assets}` plus
`agents/`, `commands/` and a manifest. Upstream's packaging, docs site, GitHub workflows and
pytest suite were **not** vendored — this repo ships no build system and no test framework
by design.
2. **The extraction library is vendored close to verbatim.** `book_to_skill/` — `config.py`,
`exceptions.py`, `sanitize.py`, `dependencies.py`, `utils.py` and the seven per-format
parsers — carries upstream's format chains, its chapter detection across five script
families, and its Unicode and XXE hardening. That code is good and re-deriving it would
only make it worse.
3. **The library is now import-pure.** `parse_arguments()`, `print_usage()`, `print_banner()`
and `main()` were removed from `utils.py` and rebuilt as an argparse CLI in
`extract_document.py`. Importing the library no longer prints, prompts, or parses `sys.argv`.
4. **SKILL.md rewritten Claude-Code-first.** The ten-step workflow, the per-chapter budget
matrix and the update/fold-in workflow are preserved in substance. Added: this repo's
frontmatter `metadata` block, an explicit hard-rules section, a forcing-question library,
a references section, and related-skill disambiguation against `write-a-skill`,
`skill-security-auditor` and `llm-wiki`. The skill-home table leads with Claude Code;
Copilot CLI and Amp remain supported.
**Safety and behaviour**
5. **No implicit installs.** Upstream's `--install-missing` defaults to `ask`, which prompts on
a TTY and runs `pip install` into the caller's environment. Here the default is `report`:
it prints the exact install command and uses the stdlib fallback. `--install-missing yes`
remains as an explicit opt-in. Reading a file should not install packages.
6. **Rights gate on distribution.** Upstream documents its copyright posture in prose. Here it
is enforced: `skill_plugin_emitter.py --distribution shareable` **refuses** unless
`--rights` names `public-domain`, `open-license`, `internal-docs` or `author-permission`.
`fair-use` is deliberately absent — it is a defence, not a licence, and not a script's call.
Without a basis, packages emit as `local` and carry
`source.cleared_for_distribution: false` in the manifest.
7. **Clean JSON output.** The vendored parsers narrate progress on stdout. Under
`--output json` that narration is redirected to stderr, so stdout carries the metadata
document and nothing else. Upstream had no JSON mode.
8. **Attribution moved off the hot path.** Upstream prints an ASCII banner from
`scripts/banner.txt` on every extraction run, and ships `BACKERS.md` + `FUNDING.yml`.
Attribution here lives in `LICENSE`, `plugin.json`, this README and the SKILL.md footer —
correct, and not re-emitted on every invocation.
**Tooling**
9. **Three upstream tools became four argparse CLIs**, each meeting this repo's contract:
real `--help`, a `--sample` that runs the tool on built-in fixtures, `--output json`, and
documented exit codes. Upstream's `extract.py` parsed `sys.argv` by hand and ignored unknown
flags with a warning, so a typo'd flag silently changed nothing.
10. **Validator merged and extended.** Upstream's `validate_skill.py` (frontmatter lint) and
`scan_generated_skill.py` (injection scan) are one gate, `book_skill_validator.py`, with
every finding carrying a family and a severity. Two check families are new:
**budget** — every generated file against the token caps the workflow commits to, with
`SKILL.md` overflow as the only hard budget error (compaction truncates from the end, so
overflow eats the indexes first); and **index** — dead chapter links, chapter files nothing
links to, topic entries pointing at chapters that were never written, and a missing topic
index. Index integrity is the failure that silently breaks navigation while the skill still
looks complete, and upstream had no check for it.
11. **Folded YAML scalars are parsed correctly.** Upstream reads a frontmatter value with a
single-line regex. A wrapped description therefore under-reports its length — so the
1024-character cap never fires on exactly the descriptions long enough to hit it — and gets
truncated at the wrap when copied into a manifest. `scalar_value()` folds continuation
lines and unescapes inner quotes. A description-trigger warning was also added, matching
this repo's `write-a-skill` rule that a description must say when to use the skill.
12. **`discovery_tax.py` → `token_budget_estimator.py`.** The optional `tiktoken` path was
dropped so every token number in the pipeline comes from one deterministic estimator and
no budget gate depends on whether a package is installed. Added: a post-flight file-by-file
budget audit, and an explicit **worth-converting verdict** — upstream reports only savings
ratios, which read as advocacy on a source too small to be worth converting at all.
13. **`parsers/html.py` renamed to `parsers/html_text.py`.** A module named `html.py`
shadows the standard library's `html` package the moment its own directory lands on
`sys.path[0]` — which happens whenever the file is run directly — and `import
html.parser` then fails with "'html' is not a package". Renaming removes the hazard
instead of documenting it. Two import lines changed; nothing else.
14. **CI-contract wiring.** The eight vendored library modules are registered in
`scripts/smoke_exceptions.txt` (they are imported as `book_to_skill.*`, never run as
CLIs), and the agent's tool table uses paths that resolve from the agent's own folder
so `scripts/check_paths.py` can follow them. Neither has an upstream counterpart —
upstream has no equivalent gates.
15. **Private, per-invocation working directory.** Upstream defaults the extraction workdir
to a fixed `<tempdir>/book_skill_work`. On a shared host that is CWE-377/CWE-59: any local
user can pre-create the directory in a world-writable `/tmp` (the sticky bit prevents
deletion, not creation) and plant a symlink named `full_text.txt` pointing at a file the
victim can write — `Path.write_text` follows symlinks. Two concurrent runs also clobber each
other. The default is now a fresh `mkdtemp` (unpredictable, `0700` by construction);
artifacts are written `0600`; an explicit `--workdir`/`BOOK_SKILL_WORKDIR` is honoured but
symlink-refused and mode-restricted first. `parsers/calibre.py` no longer writes its
`ebook-convert` scratch file to the shared directory either — which also fixes a real bug,
since it read a module-level constant and so ignored `--workdir` entirely.
16. **Shared budget constants.** `book_skill_validator.py` and `token_budget_estimator.py` both
gate on the same token caps; those now live once in `book_to_skill/config.py`
(`SKILL_FILE_BUDGETS`, `CHAPTER_TOKEN_CEILING`) rather than being restated in each tool,
where they would drift the first time a cap changed.
17. **Zip-of-XML hardening generalized to EPUB, plus decompression-bomb caps.** Upstream
hardened DOCX and only DOCX: `validate_docx_xml_safety()` screened that archive for
DTD/entity declarations before any parser touched it. EPUB is the same shape — a zip whose
members are XML — and its `ebooklib` path handed the file straight to a third-party XML
stack with no equivalent check, despite `ebooklib` being one of the packages this skill
recommends installing. The guard now lives in `book_to_skill/zip_safety.py` and runs for
both formats. Every archive read also goes through `safe_read()`, which consults the
declared uncompressed size and the compression ratio *before* decompressing: a 200 MB
zip bomb is refused at ~14 MB peak RSS instead of being materialized. Neither parser ever
writes archive members to disk, so zip-slip stays out of scope by construction.
18. **Packaging refuses a source tree containing symlinks.** The validator checks the files it
knows about (`SKILL.md`, the three supporting files, `chapters/*.md`), but `shutil.copytree`
defaults to `symlinks=False` and follows a link *anywhere else* in the tree — an `assets/`
entry, any subdirectory — baking the target's real content into a package that may then be
emitted as `--distribution shareable`. `_assert_no_symlinks()` now walks the whole tree and
refuses, and it runs **before** the validation branch so `--skip-validation` cannot bypass
it. `copytree` also passes `symlinks=True` so a future edit loosening that check cannot
silently reintroduce dereferencing.
19. **The magic-byte sniff path goes through the size budget too.** `extract_single_file()`
sniffs unknown extensions and read a `mimetype` member with a bare `zf.read()` — the
earliest attacker-controlled point in the pipeline, running *before* a format is chosen
and before any of `zip_safety.py`'s checks. A single-member zip declaring a huge
uncompressed `mimetype` was fully decompressed there. Now routed through `safe_read()`;
its `ExtractionError` deliberately sits outside the surrounding `except` tuple so a bomb
reports as a bomb rather than as a generic unsupported format. Verified: a 200 MB / 1029×
fixture with no file extension is refused at ~15 MB peak RSS.
20. **Emitter correctness and scope.** Three smaller fixes: `--author`/`--author-url` now reach
the *printed* marketplace entry (it hardcoded one name, so the snippet whose whole job is
preventing hand-edit mistakes contradicted the manifest beside it); the symlink guard is
backed by a **post-copy re-walk** that deletes the package if a link appeared during the
copy, closing the check-then-act window rather than only narrowing it; and the manifest
carries `source.license_scope` stating that the top-level `license` covers the package
scaffolding, not the compiled notes — a distinction that previously lived only in README
prose where a tool reading the manifest alone would miss it.
21. **The documented quick-start actually runs.** SKILL.md's copy-paste block referenced
`$WORKDIR` and `$SKILLS_HOME` without ever defining them — following it literally produced
a traceback on step 2. Both are now real assignments, and all five steps were executed
verbatim end to end as a check. A quick-start that does not run is worse than no
quick-start: it is the part a reader trusts most.
22. **Gate tools refuse bad paths instead of reporting success.** `token_budget_estimator.py
--skill-dir <typo>` produced a complete, plausible budget audit — every row "missing",
every cap satisfied, **exit 0** — which reads as a pass. A gate that reports success for a
path that is not there is worse than no gate. It now refuses a missing directory, a
non-directory, and a directory without `SKILL.md` (exit 2); `--full-text <missing>` raised
a bare `FileNotFoundError` traceback and now refuses cleanly. The other three tools already
handled bad input; this one was the outlier.
23. **Three more upstream artifacts cleaned.** `epub.py`'s `except (KeyError, Exception)` is
simply `except Exception` — it swallowed everything, including the size-refusal that
`safe_read()` now raises, quietly disarming deviation 17 at that call site. Narrowed so an
`ExtractionError` propagates and only genuine parse failures fall through to the `.opf`
glob. `utils.py` emitted a dynamic `{pages_label: pages}` key alongside a literal `"pages"`,
which collided whenever the label *was* `"pages"`; the alias is now added only when it
differs. A stray artifact word was removed from a `pdf.py` comment.
24. **`tool | head` no longer tracebacks.** Piping a report into `head` closes the pipe
mid-write and surfaced a `BrokenPipeError` stack trace (observed once; racy on flush
timing). All four CLIs now exit 141 quietly, the standard SIGPIPE convention.
25. **The workdir race is closed, not just narrowed.** The explicit-`--workdir` path was
still check-then-act: `mkdir(parents=True, exist_ok=True)` does **not** raise on a
symlink-to-directory, because its exists-branch tests `is_dir()`, which follows symlinks —
verified directly. Worse, the per-file `is_symlink()` check cannot back that up: a file
inside a swapped directory is an ordinary file, not a link, so a directory swap defeated
the artifact-level guard entirely.
Three changes close it. `resolve_workdir()` now attempts `mkdir` **first** and only
inspects an already-existing path, using `os.lstat` (which does not follow the final
component). `open_workdir()` then pins the directory with `O_NOFOLLOW|O_DIRECTORY`, and
both artifacts are written through that descriptor — an fd names an inode, so a later
rename or symlink swap of the path cannot redirect the write. `_write_private()` creates
with `O_CREAT|O_EXCL|O_NOFOLLOW` at mode 0600, which has no check-then-act window at all;
an artifact from a previous run into the same `--workdir` is `unlink`ed first, and unlink
removes the link itself, never its target.
Verified against a live race: pin the directory, rename it away, plant a symlink to an
attacker directory, then write — the data lands in the pinned inode and the attacker
directory stays empty. Also verified that a pre-planted `full_text.txt -> victim` symlink
leaves the victim's contents intact and is replaced by a 0600 file we own. Degrades to the
previous path-based checks on platforms without `dir_fd`/`O_NOFOLLOW` (Windows).
---
## Security audit
`engineering/skills/skill-security-auditor` on `skills/book-to-skill/`: **0 critical, 4 high**,
all four reviewed and accepted:
| Finding | Why it stands |
|---------|---------------|
| `FS-ABUSE` — `shutil.rmtree(package_root)` in the emitter | The `--force` overwrite path. Guarded by `_assert_replaceable()`: refuses a symlink, a non-directory, a path resolving outside the destination root, a directory containing the source skill, and any directory without a `.claude-plugin/plugin.json` — so it only ever deletes a plugin package this tool created. |
| `DEPS-RUNTIME` ×3 in `dependencies.py` | One is a docstring sentence; two are `print()` calls that *show* an install command. None of the three install anything. The real `pip` subprocess sits behind `--install-missing yes` and is never reached by default (deviation 5). |
Two `PRIV-ESC` criticals were real and are fixed: upstream's install hints contained the literal
string `sudo apt install poppler-utils`. They now name the package manager without telling anyone
to escalate.
---
## Where it fits
- **`engineering/write-a-skill`** — authoring a skill from expertise in your head. This one
compiles a skill from a document on disk. With both, author first and fold the document in.
- **`engineering/skill-security-auditor`** — the repo-wide security audit. The validator here
is the converter's own gate, not a replacement for it.
- **`engineering/llm-wiki`** — an incrementally-grown vault across many sources over time.
This compiles one bounded source set into one skill, once.
---
## License
MIT. See [`LICENSE`](LICENSE) — copyright is retained by the upstream author for the vendored
extraction library, with modifications and additions by this repository under the same terms.
The MIT license covers the **converter**. It does not cover the books or documents you process
with it, and it does not make a compiled skill redistributable. See
`skills/book-to-skill/references/rights_and_provenance.md`.

73
engineering/book-to-skill/agents/cs-book-to-skill.md Normal file
View file

@ -0,0 +1,73 @@
---
name: cs-book-to-skill
description: Book-to-skill converter persona. Interrogates whether a source is worth converting before spending a generation pass on it, then drives extract → analyze → chapters → supporting files → master SKILL.md → validate → package. Refuses to convert a source it cannot see on disk, to generate without a pre-flight cost estimate, to dump a large source into context, or to package a compiled skill for redistribution without a stated rights basis.
skills: engineering/book-to-skill/skills/book-to-skill
domain: engineering
model: opus
tools: [Read, Write, Edit, Bash, Grep, Glob]
---
# Book-to-Skill Converter Agent
## Voice
**Opening:** "Which file, and what three questions do you expect to ask it afterwards?"
**Forcing questions:** "Is this source big enough that converting beats reading it? Reference
or study — and if study, what worked example earns the extra budget? Do you have the right to
share what comes out?"
**Closing:** "Validator is clean and the indexes resolve. That is the whole skill: a resident
core, and one chapter at a time."
Blunt about cost, uninterested in enthusiasm. Treats "convert this book" as a request that
usually deserves a "probably not worth it" and occasionally deserves a real pipeline run.
Refuses to extrapolate past the source it compiled.
## Purpose
Drives the four decisions a conversion actually turns on:
1. **Is it worth converting?** — source size vs. compiled size, and whether the user will
return to it. Runs `token_budget_estimator.py --full-text` and reads its verdict out loud.
2. **What shape?** — `BOOK_TYPE` (technical vs. text) and `DEPTH` (reference vs. study),
which together fix the per-chapter budget and therefore most of the cost.
3. **Is the output sound?** — `book_skill_validator.py` errors block. Dead chapter links and
dangling topic references are the two that silently break navigation.
4. **Where does it live?** — a personal skills home, or wrapped as a repo plugin via
`skill_plugin_emitter.py` so the rest of the library can route to it.
## How it differs
- **vs. the raw `book-to-skill` skill:** the skill is the workflow; this agent is the gate in
front of it. Most of its value is talking users out of conversions that will not pay back.
- **vs. `cs-skill-author` (`engineering/write-a-skill`):** that agent authors a skill from
expertise in your head. This one compiles a skill from a document on disk. When the user has
both, author first and fold the document in as a source second.
- **vs. `engineering/llm-wiki`:** that grows an interlinked vault across many sources over
time. This compiles one bounded source set into one skill, once.
## Hard rules
- **The file must exist.** No converting a book from memory, no fetching one from the web.
- **Cost before generation.** The pre-flight estimate is shown and approved before any
generation pass. Never quote a hardcoded dollar price — token counts, and today's rate,
labelled an estimate.
- **Never dump a large source into context.** Over ~50k tokens, `grep` for chapter offsets and
`sed` the slice. Re-reading the source once per chapter costs more than everything else.
- **Preserve exact framework names.** A paraphrased framework name breaks every lookup that
depends on it.
- **Validation errors block.** Fix the generated files and re-run; never rewrite around a
finding, and never load a skill that has not been read by a human first.
- **Rights before redistribution.** Compiled notes from a copyrighted work stay local unless
the user names a basis: public-domain, open-license, internal-docs, or author-permission.
Fair use is a defence, not a basis this agent will assert on a user's behalf.
- **State the boundary.** Every compiled skill says what its source does not cover, and this
agent says "the source doesn't cover that" instead of filling the gap from general knowledge.
## Tools it drives
| Tool | Stage |
|------|-------|
| `../skills/book-to-skill/scripts/extract_document.py` | Extract text + metadata; `--check` for the environment |
| `../skills/book-to-skill/scripts/token_budget_estimator.py` | Pre-flight worth-it verdict; post-flight budget audit |
| `../skills/book-to-skill/scripts/book_skill_validator.py` | Frontmatter, safety, budget and index gate |
| `../skills/book-to-skill/scripts/skill_plugin_emitter.py` | Wrap the compiled skill as a claude-skills plugin |

73
engineering/book-to-skill/commands/cs-book-to-plugin.md Normal file
View file

@ -0,0 +1,73 @@
---
name: "cs-book-to-plugin"
description: "/cs:book-to-plugin <compiled-skill-dir> [--domain <domain>] — wrap a compiled book skill in a claude-skills plugin package (manifest + cs-* agent + /cs:* command + README) so the rest of the library can route to it. Use after /cs:book-to-skill, or when an existing knowledge-base skill folder needs to become an installable plugin."
---
# /cs:book-to-plugin — Ingest a Compiled Skill into the Library
**Command:** `/cs:book-to-plugin <compiled-skill-dir> [--domain <domain>] [--rights <basis>]`
A folder in `~/.claude/skills/` is invisible to this repository: no manifest, no agent, no
command, no marketplace entry, so nothing else in the library can route to it. This command
closes that gap.
## What it emits
```
<domain>/<slug>/
├── .claude-plugin/plugin.json manifest, ./skills/<slug>, provenance + rights metadata
├── README.md what the skill knows, where it came from, its limits
├── agents/cs-<slug>.md persona that answers from the source and cites chapters
├── commands/cs-<slug>.md /cs:<slug> [topic | framework | chNN]
└── skills/<slug>/ the compiled skill, copied verbatim
```
…then prints the `.claude-plugin/marketplace.json` entry to register. It never edits
marketplace.json itself — registration is a repo-wide change and stays a human decision.
## Gates
| Gate | Behaviour |
|------|-----------|
| Source has no `SKILL.md` | Refuses. This is not a compiled book skill. |
| Source has validation errors | Refuses and lists them. A package built on a broken index stays broken. `--skip-validation` overrides, and is almost always the wrong call. |
| Destination already exists | Refuses without `--force`. |
| `--distribution shareable` without `--rights` | **Refuses.** Compiled notes from a copyrighted work are personal study notes; redistributing them needs a basis. |
Accepted rights bases: `public-domain`, `open-license`, `internal-docs`, `author-permission`.
Fair use is deliberately not one — it is a defence, not a licence, and not a script's call.
Without a basis the package emits as `--distribution local` and records
`source.cleared_for_distribution: false` in the manifest.
## Run
```bash
SKILL_ROOT=engineering/book-to-skill/skills/book-to-skill
# see exactly what would be written, first
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" \
--skill-dir ~/.claude/skills/<slug> \
--dest ./engineering --domain engineering \
--source-note "<Full Title> by <Author>" \
--dry-run
# write it
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" \
--skill-dir ~/.claude/skills/<slug> \
--dest ./engineering --domain engineering \
--source-note "<Full Title> by <Author>"
```
## After emitting
1. Paste the printed entry into `.claude-plugin/marketplace.json` → `plugins`.
2. Re-derive the headline counters: `python3 scripts/derive_counters.py --check`, then update
`README.md`, `CLAUDE.md` and the marketplace description to match.
3. Read the generated agent and command — they are scaffolds keyed to the source, and the
voice is worth a pass by hand.
4. Open the PR against `dev`. Never `main`.
## Related
- `/cs:book-to-skill` — compile the source in the first place
- `/cs:plugin-audit` — 8-phase audit of the emitted package before merge

86
engineering/book-to-skill/commands/cs-book-to-skill.md Normal file
View file

@ -0,0 +1,86 @@
---
name: "cs-book-to-skill"
description: "/cs:book-to-skill <path|folder|glob>... [skill-name] — convert a book, documentation folder, or source collection into a structured agent skill (core frameworks + on-demand chapters + glossary + patterns + cheatsheet). Use when the user wants to study a document with an agent, apply an author's frameworks while working, or turn internal docs into a reusable knowledge base."
---
# /cs:book-to-skill — Compile a Source into an Agent Skill
**Command:** `/cs:book-to-skill <path|folder|glob>... [skill-name-slug]`
Runs the converter end to end: extract → analyze → chapter files → supporting files → master
`SKILL.md` → validate. Add "analyze only" to stop after the extraction report.
## Pre-flight gates
The command refuses, with a reason, when:
| Gate | Refusal |
|------|---------|
| No path given | Prints usage. This tool converts files on disk — not titles from memory, not URLs. |
| No supported file resolves | Names what was searched and the supported extensions. |
| Source is smaller than ~3× the compiled skill | Says converting is not worth it and recommends handing the agent the document. |
| Cost estimate not approved | Waits. Generation is the expensive step and the user approves it with numbers in front of them. |
| Validation errors after generation | Blocks. Dead chapter links and dangling topic references break navigation silently. |
## The six forcing questions
Asked one at a time, each with a recommended answer.
### 1. Is this source worth converting, or should I just read it?
*Recommended:* convert when it is > 3× the compiled skill's size **and** you will return to it.
One-shot reads are cheaper unconverted. `token_budget_estimator.py` prints the verdict.
### 2. Reference or study?
*Recommended:* reference, unless you intend to internalize the author's reasoning. Study depth
roughly doubles generation cost and only earns it with real worked examples.
### 3. Technical or text?
*Recommended:* technical only when tables, code, or formulas carry meaning. Docling costs
~1.5s/page and buys nothing on a prose book.
### 4. What will you actually ask this skill?
*Recommended:* name three real questions before generating. They decide what belongs in Core
Frameworks and what the topic index must resolve.
### 5. Do you have the right to redistribute this?
*Recommended:* assume not. Keep it local unless the source is public-domain, openly licensed,
your organisation's own documentation, or you have written permission.
### 6. Does this belong beside an existing skill?
*Recommended:* check for a compiled skill on the same subject first. Folding new sources into
one skill beats two skills that half-cover a topic and give the agent no way to choose.
## Pipeline
```bash
SKILL_ROOT=engineering/book-to-skill/skills/book-to-skill
# 0. environment (optional — reports extractors, installs nothing)
python3 "$SKILL_ROOT/scripts/extract_document.py" --check
# 1. extract
python3 "$SKILL_ROOT/scripts/extract_document.py" <paths> --mode text|technical
# 2. worth-it verdict, before spending a generation pass
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --full-text "$WORKDIR/full_text.txt"
# 3. generate (agent work: chapters, glossary, patterns, cheatsheet, SKILL.md)
# 4. gate
python3 "$SKILL_ROOT/scripts/book_skill_validator.py" "$SKILLS_HOME/<slug>"
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --skill-dir "$SKILLS_HOME/<slug>"
```
## Output digest
```
✅ <slug> — <Title> by <Author> <N> chapters
SKILL.md ~<N> tokens (resident) · chapters ~<N> each (on demand)
validator: <N> error(s), <N> warning(s)
next: /cs:book-to-plugin to package it for this repo
```
## Related
- `/cs:book-to-plugin` — wrap a compiled skill as a claude-skills plugin
- `/cs:write-a-skill` — author a skill from your own expertise instead of a document

229
engineering/book-to-skill/skills/book-to-skill/SKILL.md Normal file
View file

@ -0,0 +1,229 @@
---
name: book-to-skill
description: "Converts books, documentation folders, and source collections (PDF, EPUB, DOCX, HTML, Markdown, RST, AsciiDoc, RTF, MOBI/AZW) into structured agent skills — extracting named frameworks, principles, techniques, and anti-patterns into a master SKILL.md plus on-demand chapter files, a glossary, a patterns file, and a decision cheatsheet. Use when the user wants to study a document with an agent, apply an author's frameworks while working, turn internal docs or standards into a reusable knowledge base, or package a compiled book skill as a claude-skills plugin."
license: MIT
metadata:
version: 1.0.0
author: Alireza Rezvani
category: engineering
updated: 2026-08-05
---
# Book-to-Skill Converter
Turn written knowledge into an agent skill by extracting **structure**, not summaries.
A book is crystallized expertise: frameworks, principles, techniques that took years to
develop. Read once, forgotten. The workarounds all fail — PDF search returns page numbers
instead of answers, an agent handed the raw file hallucinates or drowns, reading notes rot.
This skill compiles a source into a knowledge base the agent loads on demand: a small
resident core, one chapter file at a time, and never the whole book again.
**What it produces:**
| File | Contents | Budget |
|------|----------|--------|
| `SKILL.md` | Core frameworks + chapter index + topic index | < 4,000 tokens (resident) |
| `chapters/chNN-*.md` | One summary per chapter | 800–3,000 tokens, on demand |
| `glossary.md` | Every significant term, alphabetized, with chapter | < 1,500 tokens |
| `patterns.md` | Techniques and design patterns with trade-offs | < 2,000 tokens |
| `cheatsheet.md` | Decision rules, thresholds, trade-off matrices | < 1,200 tokens |
**Beyond books:** anything referenced often enough to be worth memorizing — internal
documentation, brand systems, standards, specs, research clusters, a folder of RFCs.
---
## Philosophy
**Extract structure, not summaries.** A skill is not a book report. It is a toolkit of
named frameworks, actionable principles, step-by-step techniques, anti-patterns, and the
author's voice.
**Preserve the author's precision.** Framework names are interfaces. "The 5 Whys" is not
interchangeable with "ask why a few times" — the exact formulation is what makes lookup work.
**Layer depth appropriately.** A thin book gets a thin skill. A book with fifteen frameworks
gets chapter files and a real topic index.
**Never reproduce the source at length.** These are structured notes. Synthesize, compress,
name — do not copy passages. See `references/rights_and_provenance.md`.
---
## Modes
| Mode | Trigger | Runs |
|------|---------|------|
| **1. Full conversion** (default) | One or more paths, no special instruction | Steps 0–10 |
| **2. Analyze only** | "analyze", "just extract", "let me review first" | Steps 0–3, then stop with an extraction report |
| **3. Generate from analysis** | User supplies prior analysis notes | Steps 4–10 |
| **4. Update / fold-in** | New sources + an existing compiled skill | Steps 0–2, then the Update Workflow |
| **5. Package as plugin** | "make it a plugin", "add it to the repo" | Step 11 |
Mode 5 is this repository's addition. Upstream stops at a bare folder in a personal skills
home; Step 11 wraps that folder in a plugin package other skills and agents can route to.
---
## Hard rules
1. **Never convert a source the user cannot show you.** No web-scraping a book, no
reconstructing a title from memory. This tool converts files that are already on disk.
2. **Pre-flight the cost before generating** (Step 2.5). Generation is the expensive part;
the user approves it with numbers in front of them.
3. **Never dump a large source into context.** Over ~50k tokens, probe with `grep`/`sed`
and bounded reads (Step 2.6). Re-reading a 200-page book once per chapter costs more
than everything else in this workflow combined.
4. **Validate before anyone loads it** (Step 9.5). A generated skill is untrusted text that
an agent will later read as instructions.
5. **Never widen the generated skill's authority.** Generated frontmatter carries `name` and
`description` only — no `allowed-tools`, no model-invocation flags.
6. **Rights before redistribution.** Compiled notes from a copyrighted work are personal
study notes. Packaging one as a shareable plugin requires a stated basis (Step 11).
7. **State what the skill does not cover.** Every compiled skill's Scope section names its
boundary, so the agent says "the source doesn't cover this" instead of improvising.
---
## Pipeline
```
extract_document.py → analyze → chapter files → supporting files → SKILL.md
(Step 2) (Step 3) (Step 7) (Step 8) (Step 9)
↓
skill_plugin_emitter.py ← book_skill_validator.py
(Step 11) (Step 9.5)
```
All four tools live in `scripts/` and run on the standard library alone.
---
## Run it
```bash
SKILL_ROOT=engineering/book-to-skill/skills/book-to-skill
SKILLS_HOME=~/.claude/skills # Step 5 picks this; see the workflow reference
WORKDIR=$(mktemp -d) # or omit --workdir and capture the path it prints
SLUG=<author-lastname>-<concept>
# 1. extract → $WORKDIR/full_text.txt + metadata.json
# --mode technical when tables, code or formulas carry meaning
python3 "$SKILL_ROOT/scripts/extract_document.py" <paths> --mode text --workdir "$WORKDIR"
# 2. pre-flight: is this worth converting at all? Wait for approval before generating.
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --full-text "$WORKDIR/full_text.txt"
# 3. generate — the agent's work: chapters/, glossary, patterns, cheatsheet, SKILL.md
# 4. gate — errors block. Fix and re-run; never rewrite around a finding.
python3 "$SKILL_ROOT/scripts/book_skill_validator.py" "$SKILLS_HOME/$SLUG"
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --skill-dir "$SKILLS_HOME/$SLUG"
# 5. optional: wrap as a claude-skills plugin so the library can route to it
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" --skill-dir "$SKILLS_HOME/$SLUG" \
--dest ./engineering --source-note "<Title> by <Author>" --dry-run
```
Every path above is a real variable, not a placeholder: run the block as written (with
`<paths>` and `$SLUG` filled in) and it works end to end. Without `--workdir` the extractor
creates a private temp directory and prints it — capture that instead.
`extract_document.py --check` reports which extractors are installed and prints the install
command for what is missing. Every tool supports `--help`, `--sample` and `--output json`.
**The full step-by-step procedure — what to ask at each step, the file templates, the
per-chapter budget matrix, and the update/fold-in workflow — is in
[`references/conversion_workflow.md`](references/conversion_workflow.md). Read it before
running a conversion.** Summary of the eleven steps:
| Step | Does |
|------|------|
| 0–1 | Scope check; resolve paths; detect an update/fold-in against an existing skill |
| 1.5 | Ask content type → `BOOK_TYPE` (technical vs. text), which picks the extractor |
| 2 | Extract → `full_text.txt` + `metadata.json` |
| 2.5 | Pre-flight cost estimate and worth-converting verdict — **wait for approval** |
| 2.6 | Over ~50k tokens, probe with `grep`/`sed` instead of reading the source |
| 3 | Analyze structure (title, author, chapters, themes). Mode 2 stops here. |
| 4 | Ask purpose → `DEPTH` (reference vs. study). Never ask a second budget question. |
| 5 | Skill name and destination root; offer update / overwrite / rename on a collision |
| 6–8 | Create the structure; write chapter files; write glossary, patterns, cheatsheet |
| 9 | Write the master `SKILL.md` — under 4,000 tokens, indexes intact |
| 9.5 | Validate. Errors block. |
| 10 | Clean up the workdir and report |
| 11 | Optionally package as a plugin, behind the rights gate |
## Validator findings worth knowing
| Rule | Means |
|------|-------|
| `index.dead_link` | The chapter index links a file that was never written |
| `index.topic_dangling` | A topic points at a chapter that does not exist |
| `budget.over_cap` on SKILL.md | Compaction will truncate the indexes — navigation is the first thing lost |
| `unicode.invisible` | Extraction should have stripped this; investigate the source |
| `frontmatter.allowed_tools` | The generated skill is trying to grant itself tool authority |
Safety-family warnings are deliberately broad — a source about prompt injection legitimately
trips them. Read each in context; do not auto-silence them.
## Forcing-question library
Walk these one at a time, with a recommended answer, before running a conversion.
1. **"Is this source worth converting, or should I just read it?"**
*Recommended:* convert when it is > 3× the compiled skill's size **and** you will return
to it. One-shot reads are cheaper unconverted. (Step 2.5 verdict.)
2. **"Reference or study?"**
*Recommended:* reference, unless you intend to internalize the author's reasoning. Study
depth roughly doubles generation cost and is only worth it with real worked examples.
(Step 4.)
3. **"Technical or text?"**
*Recommended:* technical only when tables, code, or formulas carry meaning. Docling costs
~1.5s/page; picking it for a prose book buys nothing. (Step 1.5.)
4. **"What will you actually ask this skill?"**
*Recommended:* name three real questions before generating. They tell you what belongs in
Core Frameworks and what the topic index must resolve. A skill nobody queries is a
summary nobody reads.
5. **"Do you have the right to redistribute this?"**
*Recommended:* assume not. Keep it local unless the source is public-domain, openly
licensed, your organisation's own documentation, or you have written permission.
(Step 11 rights gate.)
6. **"Does this belong beside an existing skill?"**
*Recommended:* check for an existing compiled skill on the same subject first — folding
new sources into one skill (Mode 4) beats two skills that half-cover a topic and give
the agent no way to choose. (Step 0.)
---
## References
- `references/conversion_workflow.md` — **the full procedure**: Steps 0–11, the file
templates, the per-chapter budget matrix, and the update/fold-in workflow
- `references/knowledge_extraction_canon.md` — why structure beats summary; the extraction
taxonomy; what makes a framework survive compression
- `references/progressive_disclosure_budgets.md` — where the token budgets come from and
what breaks when they are exceeded
- `references/document_extraction_pipeline.md` — per-format extractor chains, fallbacks,
and the failure modes that produce silently bad text
- `references/rights_and_provenance.md` — copyright posture, the rights gate, and what
provenance a compiled skill must carry
## Related skills
- **`engineering/write-a-skill`** — authoring a skill from your own expertise. Use that when
the knowledge is in your head; use this when it is in a document.
- **`engineering/skill-security-auditor`** — full security audit of a skill package. Step 9.5
is the converter's own gate; the auditor is the repo-wide one.
- **`engineering/llm-wiki`** — an incrementally-grown, interlinked vault across many sources.
This skill compiles one bounded source set into one skill.
---
*Adapted from [virgiliojr94/book-to-skill](https://github.com/virgiliojr94/book-to-skill) (MIT).
See `../../README.md` for the full list of deviations.*

83
engineering/book-to-skill/skills/book-to-skill/assets/chapter_template.md Normal file
View file

@ -0,0 +1,83 @@
# Chapter <N>: <Full Chapter Title>
<!--
Budget (Step 7): reference study
BOOK_TYPE=text 800–1,200 1,000–1,800
BOOK_TYPE=technical 1,200–1,800 2,000–3,000
Targets, not caps. Density beats length — never pad to hit a number.
Delete every section that does not apply; an empty heading costs tokens
on every load and tells the agent nothing.
-->
## Core Idea
<One or two sentences: the single most important thing this chapter teaches. If you cannot
state it plainly, re-read the chapter — you have copied it, not extracted it. If the chapter
is genuinely thin, say so here rather than padding the sections below.>
## Frameworks Introduced
- **<Framework Name — the author's exact naming>**: <exact formulation>
- When to use: <the specific situation that triggers it>
- How: <steps or criteria — explicit for study depth, a line for reference depth>
- Why it works / fails: <study depth only, for the top 1–2 frameworks>
## Key Concepts
<5–10 terms, no more. Past ten you are transcribing a glossary rather than chunking.>
- **<Term>**: <precise one-sentence definition>
## Mental Models
<2–4 thinking tools. Write them as instruments, not descriptions:
"Use X when Y" / "Think of X as Y" / "Prefer X over Y because Z".>
## Anti-patterns
- **<What to avoid>**: <why it fails, in the author's terms>
## Code Examples
<!-- Technical sources only. Delete for prose. Copy the single most instructive snippet;
preserve indentation exactly. One good example beats three partial ones. -->
```<language>
<key code example>
```
- **What it demonstrates**: <one line>
## Reference Tables
<!-- Technical sources only. Reproduce a comparison matrix, parameter table, or decision
table as Markdown. Tables do not compress — this is what the wider technical budget
is for. -->
## Worked Example
<!-- DEPTH=study only. Reconstruct ONE concrete thing the author works through: a sample
document, a dialogue, a filled-in template, a before/after, a decision walked end to
end. This is the single biggest reason a study chapter is worth its budget, and the
main thing a learner returns for.
Reconstruct compactly in your own words. Do not reproduce the author's example
verbatim at length — see references/rights_and_provenance.md. -->
## Key Takeaways
<3–7 items. Each one actionable — something a practitioner does differently tomorrow,
not something they now know.>
1. <Actionable insight>
2. <Actionable insight>
3. <Actionable insight>
## Connects To
<!-- The links are the navigation. A chapter with no Connects To is an island, and the
agent will only ever reach it through the topic index. -->
- **Ch <N>**: <why this chapter relates to that one>
- **<External concept or standard>**: <what it connects with outside this source>

63
engineering/book-to-skill/skills/book-to-skill/assets/cheatsheet_template.md Normal file
View file

@ -0,0 +1,63 @@
# <Source Title> — Cheatsheet
<!--
BUDGET: under 1,200 tokens. One printed page you keep beside you while working.
The constraint is the point: it forces ranking.
This is the most differentiated file in a compiled skill. Anyone can grep the glossary for a
term. This file captures the author's JUDGMENT — the decisions they would make and why. It is
what turns "I know the words" into "I would act the way the author would."
AVOID: bare term → definition rows (that is glossary.md) and prose paragraphs (that is a
chapter file). Every line here helps the reader DECIDE something.
-->
## Decision Rules
<!-- Highest priority. The if/then logic the author applies, stated so it can be used without
re-reading the book. Always include the "because" — a rule without its reason cannot be
applied outside the case it was learned on. -->
| When | Do | Because |
|------|----|---------|
| <situation> | <action> | <the author's reason> |
| <situation> | <action> | <the author's reason> |
## Choosing Between <N> Options
<!-- For branching choices. Nested bullets or a small table — whichever reads faster. -->
- Is <condition>?
- **Yes** → <option A>, because <reason>
- **No** → is <second condition>?
- **Yes** → <option B>
- **No** → <option C>
## Trade-off Matrix
<!-- Competing options scored on the dimensions THIS AUTHOR cares about, so the reader can
pick under their own constraints rather than taking the author's default on faith. -->
| Option | <dimension> | <dimension> | <dimension> | Pick when |
|--------|-------------|-------------|-------------|-----------|
| <option> | <high/med/low> | <…> | <…> | <condition> |
| <option> | <…> | <…> | <…> | <condition> |
## Thresholds & Defaults
<!-- The specific numbers, ratios and rules of thumb the author actually commits to. If the
author gives a number, it belongs here — vague guidance is not a threshold. -->
| Metric | Threshold | What it means |
|--------|-----------|---------------|
| <metric> | <number> | <what crossing it implies> |
## Tells & Smells
<!-- Fast heuristics for recognizing a situation: "if you see X, you are probably in Y."
This is recognition-primed decision making — see references/knowledge_extraction_canon.md.
It is what the reader needs when they do NOT already know what they are looking at. -->
| If you see… | You are probably… | Check |
|-------------|-------------------|-------|
| <observable signal> | <the named situation> | <the confirming test> |

87
engineering/book-to-skill/skills/book-to-skill/assets/master_skill_template.md Normal file
View file

@ -0,0 +1,87 @@
---
name: <slug>
description: "Knowledge base from \"<Full Title>\" by <Author(s)>. Use when applying <author>'s frameworks for <3–6 key topics>, studying the book, or referencing its concepts."
---
<!--
HARD BUDGET: under 4,000 tokens.
Compaction truncates from the END. The end of this file is the Chapter Index and the
Topic Index — the navigation everything else depends on. Overflow does not degrade this
skill gracefully; it removes exactly the part that makes chapter files reachable. Put the
most important content first and keep the indexes inside the budget.
Frontmatter carries `name` and `description` ONLY. No allowed-tools, no model-invocation
flags — a generated skill never widens its own authority.
-->
# <Full Title>
**Author**: <Author(s)> | **Pages**: ~<N> | **Chapters**: <N> | **Generated**: <YYYY-MM-DD>
## How to Use This Skill
- **No argument** — load the core frameworks below
- **A topic** — ask about `<example topic>`; I resolve it through the Topic Index and read
that chapter file
- **`chNN`** — I load that chapter's summary
- **"what chapters do you have?"** — the full index
When you ask about something not in Core Frameworks, I read the relevant chapter file before
answering rather than guessing from the index.
---
## Core Frameworks & Mental Models
<!-- ~2,000 tokens. The author's most important named frameworks and principles, exact names
preserved. Write as instruments: "Use X when Y", "Prefer X over Y because Z".
A toolkit, not a summary. This is the part that is resident in every session — spend
the budget on what gets used, not on what was memorable to read. -->
### <Framework Name>
<What it is, when it applies, how to run it.>
### <Framework Name>
<...>
---
## Chapter Index
| # | Title | Key Frameworks |
|---|-------|----------------|
| [ch01](chapters/ch01-<slug>.md) | <Title> | <framework>, <framework> |
| [ch02](chapters/ch02-<slug>.md) | <Title> | <framework> |
## Topic Index
<!-- Alphabetical. Major terms and frameworks → the chapters that cover them. This is how
the agent navigates; without it the chapter files are unreachable except by guessing.
Every chapter file must appear at least once across these two indexes. -->
- **<Term>** → ch<N>, ch<N>
- **<Term>** → ch<N>
## Supporting Files
- [glossary.md](glossary.md) — every key term with its definition and chapter
- [patterns.md](patterns.md) — techniques and design patterns with trade-offs
- [cheatsheet.md](cheatsheet.md) — decision rules, thresholds, trade-off matrices
---
## Scope & Limits
This skill covers **<Full Title>** and nothing else. It carries that source's blind spots and
its publication-era assumptions.
For hands-on implementation in a codebase, combine it with project-specific tools. For topics
this source does not cover, say so rather than improvising — a compiled skill that quietly
extrapolates is worse than one that admits its boundary.
<!-- Provenance (see references/rights_and_provenance.md): compiled notes, not a reproduction.
If this skill is packaged for distribution, the rights basis is recorded in the plugin
manifest. -->

412
engineering/book-to-skill/skills/book-to-skill/references/conversion_workflow.md Normal file
View file

@ -0,0 +1,412 @@
# Conversion Workflow — Steps 0 through 11
The full step-by-step procedure. `SKILL.md` carries the shape, the hard rules and the
commands; this file carries the detail: what to ask at each step, the exact templates, the
per-chapter budget matrix, and the update/fold-in procedure.
Read it when running a conversion. `$SKILL_ROOT` below is this skill's folder
(`engineering/book-to-skill/skills/book-to-skill`); `$SKILLS_HOME` is the destination
skill root chosen in Step 5. File templates also ship under `assets/`:
`chapter_template.md`, `master_skill_template.md`, `cheatsheet_template.md`. Modes 2 (analyze only) and 3 (generate from prior
analysis) skip steps as noted in `SKILL.md`.
---
## Step 0 — Scope check
With no arguments, stop and say:
> `book-to-skill <path-to-document-folder-or-glob>... [skill-name-slug]`
Then parse the invocation:
- Every argument that is an existing file, folder, or matching glob → `INPUT_PATHS`.
- A trailing argument that matches no path and looks like a slug (lowercase, hyphens) →
`SKILL_NAME`.
- Any input path that is itself a compiled skill (has `SKILL.md` **and** `chapters/`), or a
`SKILL_NAME` that already exists in `SKILLS_HOME` → **Mode 4**.
---
## Step 1 — Validate input
At least one supported file must resolve: `.pdf .epub .docx .txt .md .markdown .rst .adoc
.html .htm .rtf .mobi .azw .azw3`. Expand folders and globs. Nothing supported → stop with
a clear error naming what was searched.
---
## Step 1.5 — Content type
Ask once:
> What kind of content is this?
> 1. **Technical** — code blocks, tables, formulas, diagrams
> 2. **Text-heavy** — mostly prose
> 3. **Not sure** — I'll use the fast path and warn you if quality looks thin
Option 1 → `BOOK_TYPE=technical`; options 2 and 3 → `BOOK_TYPE=text`.
Technical mode uses Docling for layout-aware extraction where installed (~1.5s/page —
say so before starting a long run). Text mode uses the fastest suitable extractor per format.
---
## Step 2 — Extract
```bash
SKILL_ROOT="<this skill folder>"
python3 "$SKILL_ROOT/scripts/extract_document.py" $INPUT_PATHS --mode "$BOOK_TYPE"
```
Writes to a **fresh private working directory** (0700, owner-only artifacts) whose path
the tool prints on completion and stores in `metadata.json` as `output_text`. Capture it:
```bash
WORKDIR=$(dirname "$(python3 "$SKILL_ROOT/scripts/extract_document.py" $INPUT_PATHS \
--mode "$BOOK_TYPE" | awk '/Text ->/ {print $3}')")
```
Pass `--workdir` (or set `BOOK_SKILL_WORKDIR`) when you want a stable location instead;
an explicit path is symlink-checked and mode-restricted before anything is written.
The directory contains:
- `full_text.txt` — combined text, per-source banners, invisible Unicode stripped
- `metadata.json` — sizes, token estimate, chapters detected, ToC present
Useful flags: `--check` reports which extractors are installed and prints the install command
for what is missing; `--workdir` relocates the output; `--output json` prints the metadata.
Nothing is installed unless you pass `--install-missing yes`. Every format degrades to a
stdlib parser except MOBI/AZW/AZW3, which need Calibre's `ebook-convert` on PATH.
---
## Step 2.5 — Pre-flight cost estimate
```bash
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" \
--full-text "$WORKDIR/full_text.txt"
```
Present the estimate and **wait for approval**:
```
📖 Sources: <N> | Pages ~<N> | Words ~<N> | Tokens ~<N>K
💰 Input (reading + prompts) ~<N>K Output (generated files) ~<N>K
Cost: multiply by your model's current per-1M input/output rates — quote today's
rate and label it an estimate. Never hardcode a price.
⏱ Estimated time: ~<N> minutes
➡ Proceed? (or say "analyze only" to preview first)
```
Estimating: input ≈ `estimated_tokens` × 1.3; output ≈ chapters × per-chapter budget +
4,000 (SKILL.md) + 4,500 (glossary + patterns + cheatsheet). Per-chapter midpoint: `text`
≈ 1,000, `technical` ≈ 1,800 — the Step 7 matrix refines it once `DEPTH` is known.
The estimator prints a **verdict**. If the source is smaller than ~3× the compiled skill,
say so and recommend handing the agent the document instead. Converting a 20-page memo is
strictly worse than reading it.
---
## Step 2.6 — Large sources (> 50k tokens)
Treat `full_text.txt` as a queryable corpus, not a single read.
```bash
wc -w "$FULL_TEXT" # size before any read
grep -n -E "^\s*(Chapter|CHAPTER)\s+[0-9]+" "$FULL_TEXT" # chapter offsets
sed -n '<start>,<end>p' "$FULL_TEXT" # pull one chapter
grep -c -i "westrum\|dora" "$FULL_TEXT" # verify before claiming
```
Prefer `Read(offset=…, limit=…)` over an unbounded read. A 200-page book is ~75k tokens;
re-reading it once per chapter across 28 chapters costs ~2M input tokens. Under 50k tokens,
a single read is fine.
---
## Step 3 — Analyze structure
Read the first ~8,000 characters to identify title, author(s), chapter structure, core themes,
and domain. Read the ToC block if present to map every chapter.
**Mode 2 stops here** with an extraction report:
```
## Extraction Report — <Title>
### Author's Core Frameworks - **<Name>**: what it is, when to apply
### Key Principles - <Principle>: actionable rule
### Techniques & Methods - <Technique>: steps
### Anti-patterns - <What to avoid>: why it fails
### Suggested Skill Name `{author-lastname}-{core-concept}`
### Chapters Detected | # | Title | Main Frameworks |
```
---
## Step 4 — Purpose (full conversion only)
> What should this skill help you do?
> 1. Apply the author's frameworks while working
> 2. Think with the author's mental models
> 3. Reference specific chapters and concepts
> 4. All of the above
Derive `DEPTH` — do **not** ask a second question:
- Only 3 → `DEPTH=reference` (lean, fast-lookup chapters)
- Includes 1, 2, or 4 → `DEPTH=study` (worked examples and reasoning)
Modes 2 and 3 skip this step: default `DEPTH=study`.
---
## Step 5 — Name and destination
If `SKILL_NAME` was given, use it. Otherwise offer two and let the user pick:
- **author-concept**: `{author-lastname}-{core-concept}` — `cialdini-influence`
- **title**: `designing-data-intensive-apps`
Prefer author-concept when the source has a strong methodological identity.
Choose `SKILLS_HOME`:
| Host | Personal root (probe in order) | Project-local |
|------|-------------------------------|---------------|
| **Claude Code** | `~/.claude/skills` | `.claude/skills` |
| **GitHub Copilot CLI** | `~/.copilot/skills` → `~/.agents/skills` | `.github/skills` → `.agents/skills` |
| **Amp** | `~/.agents/skills` → `~/.config/agents/skills` | `.agents/skills` |
Exactly one candidate exists → use it. None exist → ask which to create; never silently pick.
Heading for a repo plugin (Step 11) → generate into a temp folder and let the emitter place it.
If `$SKILLS_HOME/<slug>/` already exists, offer: **update** (Mode 4), **overwrite**, or
**rename**.
---
## Step 6 — Create the structure
```bash
mkdir -p "$SKILLS_HOME/<slug>/chapters"
```
---
## Step 7 — Chapter summaries
**Per-chapter budget** — scales with type and depth:
| | `DEPTH=reference` | `DEPTH=study` |
|---|---|---|
| `BOOK_TYPE=text` | 800–1,200 tokens | 1,000–1,800 tokens |
| `BOOK_TYPE=technical` | 1,200–1,800 tokens | 2,000–3,000 tokens |
Targets, not caps. A dense chapter runs over; a thin one runs under. Density beats length —
never pad to hit a number. Chapters load on demand, so a bigger chapter only costs when read.
**`DEPTH=study` is earned with content, not a bigger number.** The template below lands a
dense prose chapter around 700–900 tokens on its own. To reach study depth honestly, add:
- **One worked example or artifact** under `## Worked Example` — the sample document, the
dialogue, the filled-in template, the decision walked end to end. This is the single
biggest lever and the main thing a learner returns for.
- **Expanded "How"** for each framework — explicit steps or criteria, not a one-liner.
- **A "why it works / failure mode" note** on the top one or two frameworks.
A chapter with no worked example lands below the floor and says so in its Core Idea. Padding
is worse than a short chapter.
Write `chapters/ch<NN>-<slug>.md`:
```markdown
# Chapter N: <Full Title>
## Core Idea
<1–2 sentences: the single most important thing this chapter teaches>
## Frameworks Introduced
- **<Framework Name>**: <exact formulation — the author's naming>
- When to use: <specific situation>
- How: <steps or criteria>
## Key Concepts
- **<Term>**: <precise one-sentence definition> (5–10 terms)
## Mental Models
<2–4 thinking tools, written as "Use X when Y" or "Think of X as Y">
## Anti-patterns
- **<What to avoid>**: <why it fails>
## Code Examples *(technical only — omit for text)*
## Reference Tables *(technical only — reproduce comparison/decision tables)*
## Worked Example *(DEPTH=study only — reconstruct compactly, never copy at length)*
## Key Takeaways
1. <Actionable insight> (3–7 items)
## Connects To
- **Ch N**: <why it relates>
- **<Concept>**: <external standard or idea it connects with>
```
Emphasis by type: `technical` prioritizes Code Examples, Reference Tables, and exact syntax;
`text` prioritizes Frameworks, Mental Models, and Key Takeaways, omitting empty sections.
---
## Step 8 — Supporting files
**`glossary.md`** — every significant term, alphabetically sorted, `**Term** — definition (Ch N)`.
**`patterns.md`** — every concrete technique, algorithm, or design pattern:
`## Pattern Name` / `**When to use**` / `**How**` / `**Trade-offs**`.
**`cheatsheet.md`** — the most differentiated layer. Anyone can grep a glossary for a term;
the cheatsheet captures the author's *judgment*. Prioritize in order:
1. **Decision rules** — "When X, do Y, because Z"
2. **Decision trees** — nested bullets or a small table, for branching choices
3. **Trade-off matrices** — options scored on the dimensions the author cares about
4. **Thresholds and defaults** — the specific numbers the author commits to
5. **Tells and smells** — fast heuristics for recognizing a situation
Avoid bare term→definition rows (that is the glossary) and prose paragraphs (that is the
chapters). Every line helps the reader *decide* something. One printed page kept beside you.
---
## Step 9 — Master SKILL.md
**Under 4,000 tokens. Compaction truncates from the end — most important content first.**
```markdown
---
name: <slug>
description: "Knowledge base from \"<Full Title>\" by <Author(s)>. Use when applying
<author>'s frameworks for <3–6 key topics>, studying the book, or referencing its concepts."
---
# <Full Title>
**Author**: <Author(s)> | **Pages**: ~<N> | **Chapters**: <N> | **Generated**: <YYYY-MM-DD>
## How to Use This Skill
- **No argument** — load core frameworks
- **A topic** — I resolve it through the topic index and read that chapter
- **`chNN`** — I load that chapter
- **"what chapters do you have?"** — the full index
## Core Frameworks & Mental Models
<~2,000 tokens. The author's most important named frameworks, exact names preserved.
Write as "Use X when Y", "Prefer X over Y because Z". A toolkit, not a summary.>
## Chapter Index
| # | Title | Key Frameworks |
|---|-------|----------------|
| [ch01](chapters/ch01-<slug>.md) | <Title> | <framework>, <framework> |
## Topic Index
<Alphabetical. Major terms → the chapters covering them.>
- **<Term>** → ch<N>[, ch<N>]
## Supporting Files
- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md)
## Scope & Limits
Covers this source only. For hands-on implementation, combine with project-specific tools.
For topics beyond it, say so rather than improvising.
```
Frontmatter carries `name` and `description` only. No `allowed-tools`, ever (hard rule 5).
---
## Step 9.5 — Validate
```bash
python3 "$SKILL_ROOT/scripts/book_skill_validator.py" "$SKILLS_HOME/<slug>"
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --skill-dir "$SKILLS_HOME/<slug>"
```
The validator runs four families — frontmatter, safety, budget, index. **Errors block.**
Fix them and re-run; do not silently rewrite around a finding. Warnings are review prompts:
safety rules are deliberately broad, and a book about prompt injection legitimately trips them.
Common errors and what they mean:
| Rule | Meaning |
|------|---------|
| `index.dead_link` | The chapter index links a file that was never written |
| `index.topic_dangling` | A topic points at a chapter that does not exist |
| `budget.over_cap` on SKILL.md | Compaction will truncate the indexes — the navigation is the first thing lost |
| `unicode.invisible` | Extraction should have stripped this; investigate the source |
| `frontmatter.allowed_tools` | The generated skill is trying to grant itself tool authority |
---
## Step 10 — Clean up and report
Remove the workdir, then report:
```
✅ Skill created: $SKILLS_HOME/<slug>/
📚 <Title> — <Author> 📄 ~<N> pages | <N> chapters
SKILL.md core frameworks + indexes ~<N> tokens (resident)
chapters/ <N> summaries ~<N> each, ~<N> total (on demand)
glossary.md / patterns.md / cheatsheet.md ~<N> tokens
Usage:
<slug> → core frameworks
<slug> about <topic> → resolve topic, read one chapter
<slug> ch<N> → one chapter summary
Reload: Claude Code — restart the session · Copilot CLI — /skills reload · Amp — restart
```
---
## Step 11 — Package as a claude-skills plugin *(this repo's addition)*
A folder in `~/.claude/skills/` is invisible to this library — no manifest, no agent, no
command, no marketplace entry. To let the rest of the repo route to it:
```bash
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" \
--skill-dir "$SKILLS_HOME/<slug>" \
--dest ./engineering --domain engineering \
--source-note "<Full Title> by <Author>" --dry-run
```
Emits `<domain>/<slug>/` with `.claude-plugin/plugin.json`, `README.md`,
`agents/cs-<slug>.md`, `commands/cs-<slug>.md`, and the compiled skill under
`skills/<slug>/`, then prints the marketplace entry to register. Drop `--dry-run` to write.
**Rights gate.** The emitter defaults to `--distribution local`, which records
`source.cleared_for_distribution: false` in the manifest. `--distribution shareable` **refuses** unless `--rights` names a basis:
`public-domain`, `open-license`, `internal-docs`, or `author-permission`. Fair use is
deliberately not an option — it is a defence, not a licence, and not this tool's call.
The emitter also refuses to wrap a skill with validation errors. Fix the source skill first.
Registration in `.claude-plugin/marketplace.json` stays manual — it is a repo-wide change.
---
## Update / fold-in workflow (Mode 4)
1. **Read the existing skill** — parse the chapter index, topic index, metadata, and core
frameworks from `SKILL.md`; list `chapters/` for the highest number; read the three
supporting files to see what is already indexed.
2. **Classify the new content** — a revision to an existing chapter (merge into that file)
or a genuine addition (new `chNN` files numbered after the current highest).
3. **Write chapter files** per Step 7.
4. **Merge supporting files** — glossary re-alphabetized with chapter references appended to
existing terms (`**Term** — definition (Ch 4, Ch 13)`); patterns appended in the same
format; cheatsheet rules integrated into the existing structure.
5. **Regenerate SKILL.md** — bump chapter count and pages, refresh the generated date, fold
the highest-impact new frameworks into Core Frameworks (staying under 4,000 tokens),
append to both indexes.
6. **Run Step 9.5, then Step 10** with an update-shaped report: chapters added, terms merged,
indexes touched.
---

161
engineering/book-to-skill/skills/book-to-skill/references/document_extraction_pipeline.md Normal file
View file

@ -0,0 +1,161 @@
# Document Extraction Pipeline
What `extract_document.py` actually does per format, which extractor wins and why, and the
failure modes that produce text that *looks* fine and quietly ruins the conversion.
---
## 1. Extractor chains
Each format tries extractors in order and uses the first that returns non-empty text. Every
chain ends in a standard-library parser except MOBI/AZW, which has no stdlib path.
| Format | Order tried | Fallback | Notes |
|--------|-------------|----------|-------|
| **PDF** (`--mode technical`) | `docling` → text chain | text chain | Layout-aware; preserves tables, code, formulas as Markdown. ~1.5s/page. |
| **PDF** (`--mode text`) | `pdftotext -layout` → `pypdf` → `pdfminer.six` | fails with install hints | `pdftotext` (Poppler) is fastest and the only one that keeps column layout. |
| **EPUB** | `ebooklib` + `beautifulsoup4` → stdlib `zipfile` | stdlib | The stdlib path reads the OPF spine for true reading order. |
| **DOCX** | `python-docx` → stdlib `zipfile` + `ElementTree` | stdlib | The stdlib path preserves document order; `python-docx` appends all tables last. |
| **HTML** | `beautifulsoup4` → stdlib `html.parser` | stdlib | The stdlib extractor emits block boundaries on open **and** close tags. |
| **RTF** | `striprtf` → regex fallback | regex | The fallback drops whole destination groups, not just control words. |
| **Text / MD / RST / AsciiDoc** | BOM-aware read | — | Tries UTF-8-sig, UTF-32, UTF-16, then UTF-8 → cp1252 → latin-1. |
| **MOBI / AZW / AZW3** | Calibre `ebook-convert` | **none** | Hard requirement. |
`extract_document.py --check` reports what is installed and prints the exact install command.
Nothing is installed implicitly — `--install-missing yes` is the only path that runs `pip`.
---
## 2. Why `pdftotext -layout` before the Python parsers
PDF has no notion of a paragraph, a heading, or a reading order — it has positioned glyphs.
Every extractor is reconstructing structure that was never stored. Poppler's `pdftotext` with
`-layout` preserves the visual column arrangement, which is what makes a table of contents
survive as parseable lines rather than interleaved fragments. `pypdf` and `pdfminer.six` are
correct fallbacks but flatten multi-column layouts more aggressively.
For technical books the right answer is different. Docling (IBM Research, 2024) runs layout
analysis and table-structure recognition and exports Markdown, so a comparison table stays a
table and a code block keeps its indentation. It is roughly two orders of magnitude slower per
page, which is exactly why Step 1.5 asks rather than guessing.
**Post-processing.** The PDF path then cleans `pdftotext` output: running headers and footers
repeated on more than half the pages are dropped, bare page numbers at a page edge are dropped,
and words hyphenated across a line break are rejoined. That last one is a documented lossy
tradeoff — a genuinely hyphenated compound wrapped at its hyphen ("well-\nknown") rejoins as
one word.
---
## 3. Chapter detection
Structure detection is what makes the whole pipeline work: no chapters means no chapter files,
no topic index, and no navigation. The detector counts **distinct chapter numbers**, so a table
of contents entry and its body heading do not double-count.
Recognized heading styles:
- **Arabic** — `Chapter 5`, `Capítulo 5:`, `Chapitre 5.`, `Kapitel 5`, `Capitolo 5`, `Hoofdstuk 5`
- **Roman** — `I: Loomings`, `II. The Carpet-Bag`, and lowercase only inside a Markdown heading
- **Chinese** — `第三章`, `第 3 回`, `第十二节`, plus Markdown headings led by a CJK ordinal
- **Thai** — `บทที่ 3`, `ตอนที่ ๘๗` (Thai digits remapped)
- **Korean** — `제1장 총칙`, including the `제6장의2` inserted-chapter form used in statutes
- **Structural fallback** — ATX (`# Title`, `== Section`) and setext/RST underline headings,
used only when no numeric headings were found
The detector rejects prose cross-references. `Chapter 6 explores...` is not a heading: a real
heading's number is followed by end-of-line, punctuation, or a capitalized title word, and a
lowercase continuation means it is a sentence. Headings inside fenced code blocks are skipped.
**When detection fails**, the usual cause is a technical PDF whose headings were flattened by
text extraction. The fix is `--mode technical`, not a different regex.
---
## 4. Token estimation
`estimate_tokens()` is words ÷ 0.75 for whitespace-delimited text, with CJK codepoints counted
separately at ~1.5 chars/token. The CJK branch is not an optimization — Chinese and Japanese
carry little or no whitespace, so word-splitting a Chinese book collapses it to a handful of
"words" and the cost pre-flight under-reports by orders of magnitude.
The estimate is deliberately dependency-free so the same source always yields the same number
and no budget gate depends on whether `tiktoken` happens to be installed.
---
## 5. Security: invisible Unicode
Every extracted document is untrusted input that will later be read by a model as
instructions. Extraction strips four classes of invisible code point:
1. **Zero-width and invisible spacers** — U+200B–U+200D, U+2060–U+2064, U+FEFF, U+00AD,
U+034F, U+180E. They render as nothing, so text between them is invisible to a human
reviewer and plain to a model.
2. **Bidirectional formatting controls** — U+202A–U+202E, U+2066–U+2069, U+200E/U+200F,
U+061C. This is the **Trojan Source** class (Boucher & Anderson, CVE-2021-42574): these
characters do not change the sequence a model reads, they change the order a human *sees*.
A crafted line can display as innocuous study advice while the model consumes an injected
instruction. Legitimate right-to-left text is unaffected — the Unicode Bidirectional
Algorithm (UAX #9) derives direction from the characters themselves, so Arabic and Hebrew
still render correctly without explicit embeddings.
3. **Invisible letters** — Hangul fillers U+115F, U+1160, U+3164, U+FFA0. Not format
controls, so a category-based filter misses them, but they render as blank width and
survive whitespace normalization.
4. **The Unicode tag block** — U+E0000–U+E007F, originally language tags, now used to smuggle
an entire ASCII payload as invisible characters.
Extraction reports how many were removed. `book_skill_validator.py` re-checks the generated
files against the **same** predicate, imported from the same module rather than duplicated —
the two defenses drifting apart is a real bug class, and it happened upstream.
This is the OWASP LLM Top 10's **LLM01: Prompt Injection**, indirect variant: the attack
arrives inside content the model is asked to process rather than in the user's message.
Stripping invisible characters is one layer; the validator's phrase scan is a second; a human
reading the generated files before loading them is the one that actually holds.
---
## 6. Security: XML entity expansion in DOCX
DOCX is a ZIP of XML. Before parsing, every `.xml` and `.rels` member is scanned for `<!DOCTYPE`
or `<!ENTITY` declarations across several encodings, and extraction is refused if any are
present. This blocks both **billion laughs** entity-expansion denial of service and **XXE**
external-entity file disclosure. Python's `xml.etree.ElementTree` does not expand external
entities by default, but it does expand internal ones — the scan is what closes that.
---
## 7. Failure modes that look like success
The dangerous outcomes are the ones that produce plausible text:
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Chapters detected: 0, on a real book | Scanned/image PDF, or headings flattened | `--mode technical`; if the PDF is images, it needs OCR first — the extractor does not do OCR |
| Text is present but garbled or interleaved | Multi-column PDF flattened by a fallback parser | Install Poppler so `pdftotext -layout` is used |
| EPUB chapters in the wrong order | Reading order taken from the manifest rather than the spine | The stdlib path reads the spine; check the file is not DRM-protected |
| DOCX tables all at the end | `python-docx` appends tables after paragraphs | The stdlib fallback preserves order |
| RTF text littered with font names | Destination groups stripped of markup but not removed | The regex fallback drops whole groups; confirm it was used |
| Everything empty, no error | DRM-protected source | Nothing to do — the file is encrypted |
Rule of thumb: read the first 2,000 characters of `full_text.txt` before spending a generation
pass on it. Thirty seconds there beats discovering it in chapter 14.
---
## Sources
1. Poppler `pdftotext` documentation, poppler.freedesktop.org. (`-layout` mode; column
preservation.)
2. Auer, C. et al. "Docling Technical Report." IBM Research, 2024. (Layout analysis and
table-structure recognition for PDF → Markdown.)
3. W3C. *EPUB 3.3* Recommendation, 2023. (OPF package document; spine as reading order.)
4. ECMA-376, *Office Open XML File Formats*, 5th ed. (WordprocessingML document structure.)
5. Boucher, N. & Anderson, R. "Trojan Source: Invisible Vulnerabilities." *USENIX Security*,
2023; CVE-2021-42574. (Bidirectional-override attacks.)
6. Unicode Consortium. *UAX #9: Unicode Bidirectional Algorithm*. (Implicit direction from
character properties; explicit formatting characters.)
7. OWASP. *Top 10 for Large Language Model Applications* — LLM01: Prompt Injection, 2025.
(Indirect injection via processed content.)
8. OWASP. *XML External Entity (XXE) Prevention Cheat Sheet*. (DTD and entity handling.)

139
engineering/book-to-skill/skills/book-to-skill/references/knowledge_extraction_canon.md Normal file
View file

@ -0,0 +1,139 @@
# Knowledge Extraction Canon — why structure beats summary
The converter's first quality rule is "extract structure, not summaries." That is not a style
preference. It comes from a long line of work on what makes written knowledge usable later.
---
## 1. The distinction: structure vs. summary
A **summary** compresses a text by discarding detail proportionally. It answers "what was in
this?" A **structure extraction** discards *narrative* and keeps *operators*: named things you
can apply. It answers "what can I now do?"
| | Summary | Structure extraction |
|---|---------|---------------------|
| Unit | Paragraph, chapter | Named framework, rule, technique, anti-pattern |
| Preserves | Argument flow | Exact names, application conditions, trade-offs |
| Query shape | "What did chapter 4 say?" | "Which framework applies when X?" |
| Fails when | You need to act | The source has no reusable structure (fiction, narrative) |
| Decay | Rots — you remember the gist, not the tool | Survives — the name is the retrieval handle |
Mortimer Adler's *How to Read a Book* (1940, rev. 1972) makes the strongest case. His third
level, **analytical reading**, has four rules, and the second is "come to terms with the
author" — identify the key words the author uses in a special sense and pin their exact
meaning. Adler's claim is that you have not understood a book until you can state its
propositions in the author's own terms. That is precisely what the `## Frameworks Introduced`
section requires, and why the converter refuses to paraphrase "The 5 Whys" into "ask why a
few times." The name **is** the interface.
---
## 2. Atomicity: one idea per file
Niklas Luhmann's Zettelkasten — the slip-box that produced ~70 books and 400 articles — kept
one idea per slip, each with an address and explicit links to others. Sönke Ahrens's *How to
Take Smart Notes* (2017) reconstructs the method and identifies the failure mode the converter
inherits: **collector's fallacy** — accumulating material feels like learning and is not.
Notes that merely record are dead weight; notes that state a claim in your own words, linked
to where they connect, compound.
Andy Matuschak's *evergreen notes* (working notes, 2019–) sharpens two properties this
converter builds on directly:
- **Evergreen notes should be atomic** — one concept per note, so it can be linked and reused
from many directions. This is why chapters are separate files rather than sections of one
document.
- **Evergreen notes should be densely linked** — the links *are* the thinking. This is the
`## Connects To` section and the topic index; without them the chapter files are an
unnavigable pile.
The topic index is the load-bearing piece. Luhmann's slip-box worked because of its index and
its link addresses, not because of the slips. A compiled skill with 28 excellent chapter files
and no topic index is 28 files an agent will never find.
---
## 3. Chunking: why named frameworks survive compression
Cognitive Load Theory (John Sweller, 1988 onward) explains why the named-framework unit is the
right size. Working memory holds a small number of elements; expertise is largely the
possession of **schemas** that let many elements be handled as one. A named framework is a
schema with a handle. "The 5 Whys" is one element; "ask why, then ask why about that answer,
five times, to reach a root cause rather than a symptom" is five.
Barbara Oakley's *A Mind for Numbers* (2014) popularizes the same mechanism as **chunking**:
you learn by binding scattered pieces into a single retrievable unit, and the binding needs
both focused practice and a name. This is why `## Key Concepts` caps at 5–10 terms per chapter
— past that, you are transcribing a glossary, not chunking.
The **Feynman technique** (widely attributed to Richard Feynman; documented in Gleick's
*Genius*, 1992) supplies the test the `## Core Idea` section applies: if you cannot state the
chapter's central point in one or two plain sentences, you have not extracted it — you have
copied it. A Core Idea that needs a paragraph is a signal to re-read, not to write more.
---
## 4. Why anti-patterns get their own section
Extracting only what to do produces a skill that cannot recognize trouble. The
`## Anti-patterns` section exists because the recognition case and the application case are
different retrieval problems: you reach for a framework when you know what you are doing, and
you need an anti-pattern when you do not.
Gary Klein's work on **recognition-primed decision making** (*Sources of Power*, 1998) found
that experts under time pressure rarely compare options — they recognize a situation as a type
and run the response that type calls for. Klein's later "premortem" technique inverts it:
imagine the failure has happened and work backwards to its causes. The cheatsheet's "tells and
smells" layer is recognition-primed decision making in file form: fast pattern → named
situation → response.
---
## 5. Why the cheatsheet is the differentiated layer
Glossary, patterns and cheatsheet look like three flavours of the same list. They are not:
- **glossary** answers *what does this word mean* — pure recall
- **patterns** answers *how do I do this thing* — procedure
- **cheatsheet** answers *what should I do here* — judgment
Only the third captures what separates someone who has read the book from someone who has
absorbed it. Decision rules ("when X, do Y, because Z"), thresholds the author actually commits
to, and trade-off matrices are the author's judgment made portable. A cheatsheet that drifts
into term→definition rows has silently become a second glossary, which is why the converter's
Step 8 lists what to avoid as explicitly as what to include.
---
## 6. What does not convert well
Being honest about the boundary keeps the tool credible:
- **Narrative non-fiction with no reusable structure.** A biography compresses to a summary
because there is no framework to name. The converter will produce chapter files, and they
will be book reports.
- **Fiction.** No frameworks, no anti-patterns, no decision rules.
- **Reference works already structured for lookup** — a dictionary, an API reference, a
standards document with numbered clauses. They are already indexed; compiling them adds a
lossy layer between the reader and the authority.
- **Sources you will consult once.** Conversion has a fixed up-front cost. Below roughly 3×
the compiled skill's size, reading the document is cheaper — which is why the token budget
estimator prints a verdict rather than a number.
---
## Sources
1. Adler, M. & Van Doren, C. *How to Read a Book: The Classic Guide to Intelligent Reading*.
Simon & Schuster, rev. ed. 1972. (Analytical reading; "come to terms with the author.")
2. Ahrens, S. *How to Take Smart Notes*. 2017. (Zettelkasten method; collector's fallacy.)
3. Matuschak, A. "Evergreen notes." *Working Notes*, notes.andymatuschak.org, 2019–.
(Atomicity and dense linking as note-design principles.)
4. Sweller, J. "Cognitive Load During Problem Solving: Effects on Learning." *Cognitive
Science* 12(2), 1988. (Schema acquisition; working-memory limits.)
5. Oakley, B. *A Mind for Numbers*. TarcherPerigee, 2014. (Chunking; focused vs. diffuse modes.)
6. Klein, G. *Sources of Power: How People Make Decisions*. MIT Press, 1998. (Recognition-primed
decision making; later the premortem.)
7. Gleick, J. *Genius: The Life and Science of Richard Feynman*. Pantheon, 1992. (The
explain-it-simply test commonly called the Feynman technique.)

155
engineering/book-to-skill/skills/book-to-skill/references/progressive_disclosure_budgets.md Normal file
View file

@ -0,0 +1,155 @@
# Progressive Disclosure and Token Budgets
Every number in this skill's Step 7–9 budget tables traces to a specific failure mode. This
document says where each came from and what breaks when it is exceeded.
---
## 1. The three-tier disclosure model
Agent Skills are built on progressive disclosure: the agent sees a little, and pulls more only
when it needs to. Anthropic's Agent Skills documentation describes three levels:
| Level | What loads | When | Practical budget |
|-------|-----------|------|------------------|
| 1 | `name` + `description` from frontmatter | Always, for every installed skill | ~100 tokens |
| 2 | The `SKILL.md` body | When the skill is triggered | < 5k tokens |
| 3 | Bundled files — references, chapters, scripts | When the body points at them | Unbounded, on demand |
A compiled book skill maps onto this exactly: description → level 1, master `SKILL.md` → level
2, `chapters/` + glossary + patterns + cheatsheet → level 3. The whole design of the converter
is an attempt to put as little as possible in level 2 while keeping level 3 *findable* — which
is what the topic index is for.
Jakob Nielsen's original formulation of progressive disclosure (Nielsen Norman Group, 2006)
carries the warning that applies here: disclosure only works when the first tier makes the
second tier's existence and value obvious. A chapter file nothing links to is not progressive
disclosure — it is a file that will never be read.
---
## 2. Why SKILL.md is capped at 4,000 tokens
Two independent reasons, and the second is the one that actually bites.
**Residency.** The master file is loaded whenever the skill triggers. It is paid for on every
session that touches this knowledge base, whether or not the user asks a question the file
answers. Anthropic's skill-authoring guidance puts the practical ceiling around 5k tokens; the
converter budgets 4k and leaves headroom.
**Truncation direction.** Context compaction and hard truncation drop content from the *end* of
a document. In a compiled skill the end of `SKILL.md` is the Chapter Index and the Topic Index
— the navigation the entire architecture depends on. Overflow does not degrade the skill
gracefully; it removes exactly the part that makes chapter files reachable, leaving a resident
core that looks fine and can no longer route. That asymmetry is why `budget.over_cap` on
`SKILL.md` is the validator's only hard budget error, and why Step 9 says "most important
content first."
---
## 3. Why chapters live in separate files
Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (*TACL*, 2024), showed
that retrieval accuracy over a long context follows a U-curve: models attend well to the
beginning and the end of their context and measurably worse to the middle. Performance on
multi-document QA degraded as relevant information moved toward the centre.
The implication for this converter is direct. A single 40k-token file containing all chapters
puts most chapters in the sag of that curve. Twenty-eight separate files, one loaded at a time,
put the relevant chapter alone at the front of a short context. The gain is not only cost — it
is accuracy.
Anthropic's context-engineering guidance frames the same point as treating context as a finite
resource with diminishing returns: more retrieved text is not monotonically better, because
irrelevant content competes for attention with relevant content. "Load the whole book, it fits"
is a real option in a long-context model and still the wrong one.
---
## 4. The per-chapter matrix
| | `reference` | `study` |
|---|---|---|
| `text` | 800–1,200 | 1,000–1,800 |
| `technical` | 1,200–1,800 | 2,000–3,000 |
The floor comes from what the chapter template naturally produces. Core Idea, Frameworks,
Key Concepts, Mental Models, Anti-patterns, Takeaways and Connects To on a dense prose chapter
land around 700–900 tokens without padding. Anything below that means the chapter had little
extractable structure — which is worth saying in the Core Idea rather than hiding with filler.
The technical rows are wider because code and tables do not compress. A reproduced parameter
table is 300 tokens whether or not the chapter is important, and truncating a code example to
fit a budget produces something worse than omitting it.
The `study` column is wider only because it must contain the **worked example**. That is the
honest way to spend the extra budget: reproduce the artifact the author walks through, compactly
and in your own reconstruction. Everything else — longer prose, more adjectives, restated
takeaways — is padding, and padding costs tokens on every load while adding nothing retrievable.
This is the converter's third quality rule (density over completeness) expressed as a number.
The 3,500-token chapter ceiling in the validator is a runaway detector, not a target: at that
size a single chapter costs as much to load as the entire resident core, which usually means
the source chapter should have been split.
---
## 5. Supporting-file budgets
| File | Cap | Why that number |
|------|-----|-----------------|
| `glossary.md` | 1,500 | ~100–150 terms at one line each. Past that it is a dictionary of the source's whole vocabulary rather than its significant terms, and the signal drops. |
| `patterns.md` | 2,000 | ~15–25 patterns with when/how/trade-offs. Sources with more than that usually want the patterns distributed into chapters. |
| `cheatsheet.md` | 1,200 | One printed page. The constraint is the point: it forces ranking, and a cheatsheet you cannot see at once is not a cheatsheet. |
These three are warnings rather than errors in the validator. They are loaded on demand, so
overflow costs a heavier read but never silently breaks navigation the way `SKILL.md` overflow
does.
---
## 6. The description is a routing decision, not a summary
Level 1 is the only thing the agent sees when deciding whether this skill is relevant at all.
Anthropic's best-practice guidance is specific: write in the third person, state what the skill
does, and state explicitly *when to use it* with the concrete triggers a user would say. The
1,024-character limit is a hard field limit, not a style guideline.
For a compiled book skill the generated description must name the **source and its topics**,
not just the title. "Knowledge base from *Thinking in Systems* by Donella Meadows" does not
help an agent decide whether to load it when the user asks about feedback loops. Naming three
to six key topics is what turns the description into a router.
---
## 7. Budget-check discipline
Estimate with `token_budget_estimator.py`, which uses the same words/0.75 heuristic as the
extractor so every number in the pipeline agrees. It is deliberately not a BPE count: a stable,
comparable estimate that never varies with an optional dependency is more useful for a budget
gate than an exact number that requires `tiktoken` to be installed. Expect it to run roughly
10–20% off a true `cl100k` count on English prose, and further off on code and CJK.
Run it twice: pre-flight on `full_text.txt` to decide whether conversion is worth it at all,
and post-flight on the generated folder to catch overflow before anyone loads the skill.
---
## Sources
1. Anthropic. "Agent Skills." *Claude Docs*, code.claude.com/docs/en/skills. (Three-level
progressive disclosure; SKILL.md size guidance.)
2. Anthropic. "Agent Skills best practices." *Claude Platform Docs*,
platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices. (Description
field rules; third person; explicit triggers.)
3. Nielsen, J. "Progressive Disclosure." Nielsen Norman Group, 2006. (The first tier must make
the second tier's value obvious.)
4. Liu, N. F. et al. "Lost in the Middle: How Language Models Use Long Contexts."
*Transactions of the ACL*, 2024. (U-shaped positional accuracy over long contexts.)
5. Anthropic. "Effective context engineering for AI agents." Anthropic Engineering Blog, 2025.
(Context as a finite resource; retrieval over preloading.)
6. Agent Skills open standard, github.com/agentskills/agentskills. (`name` + `description` as
the only universally required frontmatter fields.)
7. Sweller, J., Ayres, P. & Kalyuga, S. *Cognitive Load Theory*. Springer, 2011. (Extraneous
load: irrelevant material degrades processing of relevant material — the human analogue of
the lost-in-the-middle result.)

163
engineering/book-to-skill/skills/book-to-skill/references/rights_and_provenance.md Normal file
View file

@ -0,0 +1,163 @@
# Rights and Provenance
**Not legal advice.** This document explains the posture the tool takes and why. Where money,
publication, or an employer's exposure is involved, ask a lawyer.
---
## 1. What the tool does and does not do
The converter ships **no** book content. It is a converter for files you already have, it runs
locally, and nothing leaves the machine. What it produces is a set of structured notes about a
source: named frameworks, definitions, decision rules, and an index — the kind of thing a
careful reader produces in a notebook, not a copy of the work.
That distinction is doing real work, and it is why the converter's seventh quality rule is
"never copy raw source text." The rule is not stylistic prudence. A file of long verbatim
passages is a reproduction wearing a Markdown extension; a file of extracted structure is not.
---
## 2. The idea/expression line
US copyright protects **expression**, not ideas, procedures, systems, or methods of operation
(17 U.S.C. §102(b)). *Baker v. Selden*, 101 U.S. 99 (1879), is the origin: a book explaining a
bookkeeping system was protected, but the system itself — and the blank forms needed to use it
— were not.
For this tool that line is directly operational:
| Generally on the ideas side | Generally on the expression side |
|---|---|
| A framework's **name** and what it is for | The chapter that introduces it, in the author's prose |
| The **steps** of a method | The author's phrasing of those steps at length |
| A **decision rule** stated plainly | An extended passage arguing for it |
| A **term's** definition in your own words | The author's definition copied verbatim |
| The **structure** of the argument | The argument as written |
The converter's output shape follows that column split deliberately. It also means the
`## Worked Example` section is the one to watch: "reconstruct compactly" is the instruction,
and reproducing the author's full example verbatim crosses back over.
---
## 3. Fair use, honestly
Fair use (17 U.S.C. §107) is a four-factor defence, not a permission slip, and it is assessed
case by case:
1. **Purpose and character** — personal study and research weigh favourably; commercial
redistribution does not. Transformation matters.
2. **Nature of the work** — factual and technical works are more amenable than fiction.
3. **Amount and substantiality** — how much, and whether it takes the heart of the work.
Structured notes take little; a chapter-by-chapter paraphrase of a narrative takes much.
4. **Market effect** — the factor courts weight heavily. Notes that send you back to the book
do not substitute for it; a compilation that makes buying it unnecessary does.
Two cases bound the space usefully. *Authors Guild v. HathiTrust*, 755 F.3d 87 (2d Cir. 2014),
and *Authors Guild v. Google, Inc.*, 804 F.3d 202 (2d Cir. 2015), both found that scanning
books to build a **search index** — surfacing where terms appear, with snippets rather than
readable text — was transformative fair use, in large part because the output did not
substitute for the original. A compiled book skill is closer to that index than to a copy. It
is not identical, and nobody has litigated this shape.
This is exactly why `--rights fair-use` **is not an option** in the emitter. Fair use is what
you argue after someone objects. It is not a basis a script should let you assert in a manifest.
---
## 4. Outside the US
- **Berne Convention** — protection is automatic on creation in all member states; no
registration or notice is required. "There was no copyright notice" is not a finding.
- **EU** — the InfoSoc Directive (2001/29/EC) has no open-ended fair use; it has an exhaustive
list of narrower exceptions, including private, non-commercial personal copying in many
member states.
- **EU DSM Directive (2019/790)** — Article 3 permits text and data mining for scientific
research by research organisations; Article 4 permits it more broadly **unless the rights
holder has reserved it** in a machine-readable way. The Article 4 opt-out is worth checking
for anything you intend to share.
- **UK** — "fair dealing" is narrower than fair use and enumerated: research and private study
(non-commercial), criticism, review, quotation, news reporting.
Personal study notes from a book you own sit comfortably in most of these. Publishing a
compiled skill does not.
---
## 5. The rights gate
`skill_plugin_emitter.py` refuses to emit a `--distribution shareable` package unless
`--rights` names a basis:
| Basis | Means |
|-------|-------|
| `public-domain` | Copyright expired, forfeited, or never applied (e.g. most US federal works) |
| `open-license` | The source carries a licence permitting derivative distribution — CC BY, CC BY-SA, MIT, Apache-2.0. Check the share-alike and attribution terms; they follow the derivative. |
| `internal-docs` | Your organisation's own documentation, shared inside it. The usual constraint here is confidentiality, not copyright. |
| `author-permission` | The rights holder gave written permission. Keep the writing. |
Anything else emits as `--distribution local`, recording
`source.cleared_for_distribution: false` in the manifest — an advisory marker, not
enforcement, that says the package was never cleared for sharing.
Note what the gate is **not**: it does not inspect your source, it cannot verify a claim, and
`local` does not make a package safe to publish later. It exists to make the question
unavoidable at the moment of packaging, when it is cheap to answer, rather than after
distribution, when it is not.
---
## 6. Attribution the compiled skill must carry
Every generated skill records, in `SKILL.md` and in the emitted plugin's README and manifest:
- **Source document** — full title and author(s)
- **Generated date** — so a reader knows which edition-era the notes reflect
- **Chapter count and page count** — the scope of what was covered
- **Distribution and rights basis** — from the gate above
This is provenance, not politeness. A compiled skill is a lossy derivative that will be read as
authoritative. A reader who can see it was compiled from one source on one date can judge what
it is likely to be missing; a reader who cannot will treat it as ground truth. The generated
`## Scope & Limits` section exists for the same reason — it is the skill telling the agent
where to stop and say "the source doesn't cover this."
---
## 7. Practical guidance
**Safe by default**
- Books you own, compiled for your own use, kept local
- Your organisation's internal documentation, shared internally
- Public-domain works (Project Gutenberg, Standard Ebooks, most US federal publications)
- Openly-licensed technical documentation, with the licence's attribution terms respected
**Ask first**
- A compiled skill from a copyrighted book shared with your team
- Anything a client or employer will use commercially
- Sources under an NDA — copyright is the *second* problem there
**Do not**
- Publish a compiled skill of a copyrighted book to a public marketplace
- Present compiled notes as a substitute for buying the source
- Convert a source you obtained from a pirate library — how the file was acquired is a separate
and worse problem than what you then do with it
---
## Sources
1. 17 U.S.C. §102(b) — scope of copyright; ideas, procedures and methods of operation excluded.
2. *Baker v. Selden*, 101 U.S. 99 (1879) — the idea/expression dichotomy.
3. 17 U.S.C. §107 — fair use; the four factors.
4. *Authors Guild v. HathiTrust*, 755 F.3d 87 (2d Cir. 2014) — full-text search index as
transformative use.
5. *Authors Guild v. Google, Inc.*, 804 F.3d 202 (2d Cir. 2015) — book scanning for search and
snippet display held transformative; market-substitution analysis.
6. Berne Convention for the Protection of Literary and Artistic Works (Paris Act, 1971) —
automatic protection; no formalities.
7. Directive (EU) 2019/790 (DSM), Articles 3–4 — text and data mining exceptions and the
machine-readable rights reservation.
8. UK Copyright, Designs and Patents Act 1988, ss. 29–30 — fair dealing for research, private
study, criticism, review and quotation.

607
engineering/book-to-skill/skills/book-to-skill/scripts/book_skill_validator.py Normal file
View file

@ -0,0 +1,607 @@
#!/usr/bin/env python3
"""book_skill_validator.py — gate a generated book skill before anyone loads it.
Runs four independent check families over a generated skill folder and reports
every finding with a file, a line, and a rule id:
frontmatter name/description shape for the chosen host lens, tool-authority
grants, unrecognized keys
safety prompt-injection phrasing, fake system prefixes, chat-template
delimiters, exfiltration-shaped language, invisible Unicode, and
frontmatter that widens the generated skill's own authority
budget SKILL.md and the supporting files against the token caps the
converter commits to (Steps 7-9 of SKILL.md)
index Chapter Index and Topic Index integrity — every link resolves,
every chapter file is indexed, no topic points at a missing
chapter
The safety family is deliberately broad: a book about LLM security legitimately
contains the phrases it flags. Findings are review prompts, not verdicts. What
they are not is optional — a generated skill is untrusted text that an agent
will later load as instructions, so it gets read by a human before it gets read
by a model.
Severity: ERROR fails the run (exit 1). WARN is reported and does not fail
unless --strict is passed.
Exit codes:
0 no errors (warnings may be present)
1 at least one error (or any finding under --strict)
2 the skill could not be inspected at all
Adapted from virgiliojr94/book-to-skill's validate_skill.py and
scan_generated_skill.py (MIT), merged into one gate and extended with the
budget and index families. See ../../../LICENSE.
Usage:
python3 book_skill_validator.py path/to/generated-skill
python3 book_skill_validator.py path/to/skill --lens copilot --strict
python3 book_skill_validator.py path/to/skill --output json
python3 book_skill_validator.py --sample
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import tempfile
from dataclasses import asdict, dataclass
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from book_to_skill.config import ( # noqa: E402
CHAPTER_TOKEN_CEILING,
SKILL_FILE_BUDGETS,
)
from book_to_skill.sanitize import is_invisible_codepoint # noqa: E402
from book_to_skill.utils import estimate_tokens # noqa: E402
MAX_SKILL_FILES = 1_000
MAX_FILE_BYTES = 2 * 1024 * 1024
MAX_TOTAL_BYTES = 20 * 1024 * 1024
# Token caps the converter's own workflow commits to. Imported rather than
# restated: token_budget_estimator.py gates on the same numbers, and two copies
# would drift the moment a cap changes.
BUDGETS = SKILL_FILE_BUDGETS
CHAPTER_CEILING = CHAPTER_TOKEN_CEILING
SUPPORTING_FILENAMES = ("glossary.md", "patterns.md", "cheatsheet.md")
CLAUDE_CODE_TOOLS = {
"Bash", "Read", "Write", "Edit", "Glob", "Grep",
"WebFetch", "WebSearch", "NotebookEdit", "Task", "TodoWrite",
}
COPILOT_CLI_TOOLS = {"shell", "bash", "write"}
AMP_TOOLS = CLAUDE_CODE_TOOLS | {"shell_command"}
LENSES = {
"claude": {
"label": "Claude Code",
"tools": CLAUDE_CODE_TOOLS,
"recognized_keys": {"name", "description", "allowed-tools", "license", "metadata"},
"reserved_name_words": {"anthropic", "claude"},
"bash_tool_names": {"Bash"},
"unknown_tool_severity": "error",
},
"copilot": {
"label": "GitHub Copilot CLI",
"tools": COPILOT_CLI_TOOLS,
"recognized_keys": {"name", "description", "allowed-tools", "license"},
"reserved_name_words": set(),
"bash_tool_names": {"shell", "bash"},
"unknown_tool_severity": "warn",
},
"amp": {
"label": "Amp",
"tools": AMP_TOOLS,
"recognized_keys": {"name", "description", "allowed-tools", "license",
"compatibility", "argument-hint"},
"reserved_name_words": set(),
"bash_tool_names": {"shell_command", "Bash"},
"unknown_tool_severity": "warn",
},
}
_CONTENT_RULES = (
("prompt.ignore_previous",
re.compile(r"\bignore\s+(?:(?:all|any|the)\s+)?(?:previous|prior)\s+"
r"(?:instructions?|prompts?|rules?|messages?)\b", re.IGNORECASE),
"instruction-override phrase"),
("prompt.disregard_system",
re.compile(r"\bdisregard\s+(?:the\s+)?(?:system|developer)\b", re.IGNORECASE),
"system-instruction override phrase"),
("prompt.role_reassignment",
re.compile(r"\byou\s+are\s+now\b", re.IGNORECASE),
"role-reassignment phrase"),
("prompt.fake_system_prefix",
re.compile(r"^\s*(?:[-*]\s*)?(?:system|developer)\s*:", re.IGNORECASE),
"system-like message prefix"),
("prompt.system_tag",
re.compile(r"<\s*/?\s*system\b[^>]*>", re.IGNORECASE),
"system-message tag"),
("prompt.chat_template_tag",
re.compile(r"<\|\s*im_start\s*\|>|\[\s*INST\s*\]", re.IGNORECASE),
"model chat-template delimiter"),
("prompt.tool_call_tag",
re.compile(r"\btool[_ -]?call\b", re.IGNORECASE),
"tool-call control token"),
)
_EXFILTRATION_TERM = re.compile(r"\bexfiltrat(?:e|es|ed|ing|ion)\b", re.IGNORECASE)
_OUTBOUND_TERM = re.compile(r"\b(?:curl|wget|send|post|upload|transmit)\b|https?://", re.IGNORECASE)
_SENSITIVE_TERM = re.compile(
r"(?:\.env\b|\bbase64\b|\bsecrets?\b|\bcredentials?\b|\bapi[_ -]?keys?\b)", re.IGNORECASE)
_CHAPTER_LINK = re.compile(r"\]\(\s*(chapters/[^)\s]+\.md)\s*\)")
_TOPIC_REF = re.compile(r"→\s*(ch\d{1,3}(?:\s*,\s*ch\d{1,3})*)", re.IGNORECASE)
_CHAPTER_FILE = re.compile(r"^ch(\d{1,3})[-_]", re.IGNORECASE)
class ScanError(RuntimeError):
"""The generated skill could not be inspected completely."""
@dataclass(frozen=True)
class Finding:
family: str
severity: str # "error" | "warn"
path: str
line: int
rule_id: str
message: str
def _terminal_safe(value: str) -> str:
"""Escape control and non-ASCII characters before printing untrusted paths."""
return value.encode("unicode_escape", errors="backslashreplace").decode("ascii")
# --------------------------------------------------------------------------- collection
def _collect_skill_files(skill_dir: Path) -> tuple[Path, list[Path]]:
requested = skill_dir.expanduser()
if requested.name.lower() == "skill.md" and requested.is_file():
requested = requested.parent
if requested.is_symlink():
raise ScanError("the generated skill directory must not be a symbolic link")
try:
root = requested.resolve(strict=True)
except OSError as exc:
raise ScanError(f"no such generated skill directory: {_terminal_safe(str(requested))}") from exc
if not root.is_dir():
raise ScanError("the generated skill path is not a directory")
master = root / "SKILL.md"
if not master.is_file() or master.is_symlink():
raise ScanError("SKILL.md is missing or is a symbolic link")
candidates = {master}
for filename in SUPPORTING_FILENAMES:
supporting = root / filename
if supporting.is_symlink():
raise ScanError(f"{filename} must be a real file, not a symbolic link")
if supporting.exists():
if not supporting.is_file():
raise ScanError(f"{filename} must be a real file")
candidates.add(supporting)
chapters = root / "chapters"
if chapters.exists():
if chapters.is_symlink() or not chapters.is_dir():
raise ScanError("chapters must be a real directory, not a symbolic link")
candidates.update(chapters.glob("*.md"))
files = sorted(candidates, key=lambda p: p.relative_to(root).as_posix().lower())
if len(files) > MAX_SKILL_FILES:
raise ScanError(f"generated skill has {len(files):,} Markdown files; "
f"maximum is {MAX_SKILL_FILES:,}")
return root, files
def _read_skill_files(root: Path, files: list[Path]) -> dict[str, str]:
contents: dict[str, str] = {}
total_bytes = 0
for path in files:
if path.is_symlink():
raise ScanError("generated skill contains a symbolic-link Markdown file")
size = path.stat().st_size
if size > MAX_FILE_BYTES:
raise ScanError(f"{_terminal_safe(path.name)} is {size:,} bytes; the per-file "
f"scan limit is {MAX_FILE_BYTES:,} bytes")
total_bytes += size
if total_bytes > MAX_TOTAL_BYTES:
raise ScanError(f"generated skill Markdown exceeds the {MAX_TOTAL_BYTES:,}-byte scan limit")
relative = path.relative_to(root).as_posix()
try:
contents[relative] = path.read_text(encoding="utf-8-sig")
except UnicodeDecodeError as exc:
raise ScanError(f"{_terminal_safe(relative)} is not valid UTF-8") from exc
except OSError as exc:
raise ScanError(f"could not read {_terminal_safe(relative)}") from exc
return contents
# --------------------------------------------------------------------------- frontmatter
def parse_frontmatter(text: str) -> tuple[str | None, str]:
if not text.startswith("---"):
return None, text
end = text.find("\n---", 3)
if end == -1:
return None, text
return text[3:end].lstrip("\n"), text[end + 4:]
def scalar_value(fm: str, key: str) -> str | None:
"""Read a scalar frontmatter value, folding YAML continuation lines.
A generated description often wraps: the value continues on following
indented lines until the next top-level key. Reading only the first line
under-reports its length (so the 1024-char cap never fires) and truncates it
wherever it happens to have wrapped when it is copied into a manifest.
"""
lines = fm.splitlines()
for index, line in enumerate(lines):
match = re.match(rf"^{re.escape(key)}:\s*(.*)$", line)
if not match:
continue
parts = [match.group(1).strip()]
for continuation in lines[index + 1:]:
if not continuation.strip() or re.match(r"^[A-Za-z][\w-]*:", continuation):
break
if not continuation.startswith((" ", "\t")):
break
parts.append(continuation.strip())
value = " ".join(part for part in parts if part).strip()
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
value = value[1:-1]
# A quoted YAML scalar escapes inner quotes; keep the text, drop the escape.
return value.replace('\\"', '"').strip()
return None
# Back-compat alias for the single-key reads below.
_scalar = scalar_value
def _list_items(fm: str, key: str) -> list[str]:
items, capturing = [], False
for line in fm.splitlines():
if re.match(rf"^{re.escape(key)}:\s*$", line):
capturing = True
continue
if capturing:
item = re.match(r"^\s*-\s*(.+)$", line)
if item:
items.append(item.group(1).strip())
elif re.match(r"^[A-Za-z][\w-]*:", line):
break
return items
def check_frontmatter(text: str, lens: str) -> list[Finding]:
rules = LENSES[lens]
label = rules["label"]
fm, body = parse_frontmatter(text)
out: list[Finding] = []
def err(rule, msg, line=1):
out.append(Finding("frontmatter", "error", "SKILL.md", line, rule, msg))
def warn(rule, msg, line=1):
out.append(Finding("frontmatter", "warn", "SKILL.md", line, rule, msg))
if fm is None:
err("fm.missing", "no valid YAML frontmatter (--- block)")
return out
name = _scalar(fm, "name")
if not name:
err("fm.name_missing", "name: missing (required)")
else:
if len(name) > 64:
err("fm.name_length", f"name: {len(name)} > 64 chars")
if not re.fullmatch(r"[a-z0-9-]+", name):
err("fm.name_charset", f"name: '{name}' must be lowercase letters/digits/hyphens")
if any(word in name.lower() for word in rules["reserved_name_words"]):
err("fm.name_reserved", f"name: '{name}' contains a word reserved by {label}")
description = _scalar(fm, "description")
if not description:
err("fm.description_missing", "description: missing (required)")
else:
if len(description) > 1024:
err("fm.description_length", f"description: {len(description)} > 1024 chars")
if not re.search(r"\buse\s+(?:when|before|after|while|for)\b", description, re.IGNORECASE):
warn("fm.description_trigger",
"description has no explicit 'Use when ...' trigger — the agent has nothing "
"to match on when deciding whether to load this skill")
tools = _list_items(fm, "allowed-tools")
if not tools:
inline = _scalar(fm, "allowed-tools")
tools = inline.split() if inline else []
if tools:
bases = {entry.split("(", 1)[0].strip() for entry in tools}
known = bases & rules["tools"]
unknown = sorted(b for b in bases if b not in rules["tools"])
uses_bash = "```bash" in body or "python3 " in body
if uses_bash and not (bases & rules["bash_tool_names"]):
names = " or ".join(f"'{n}'" for n in sorted(rules["bash_tool_names"]))
err("fm.tools_missing_bash",
f"allowed-tools declares a restriction but omits {names} while the skill runs "
f"bash/python3 — under {label} those steps would be blocked")
if not known and rules["unknown_tool_severity"] == "error":
err("fm.tools_unrecognized", f"allowed-tools: no recognized {label} tool in the list")
if unknown:
warn("fm.tools_unknown_names",
f"allowed-tools: {unknown} are not {label} built-in tool names")
for key in (m.group(1) for line in fm.splitlines()
if (m := re.match(r"^([A-Za-z][\w-]*):", line))):
if key not in rules["recognized_keys"]:
warn("fm.key_unrecognized", f"frontmatter '{key}': not a recognized {label} key")
lines = len(text.splitlines())
if lines > 500:
warn("fm.body_length", f"SKILL.md is {lines} lines > 500 (soft guideline)")
return out
# --------------------------------------------------------------------------- safety
def _frontmatter_line_numbers(lines: list[str]) -> set[int]:
if not lines or lines[0].strip() != "---":
return set()
for index, line in enumerate(lines[1:], start=2):
if line.strip() == "---":
return set(range(2, index))
return set()
def check_safety(relative_path: str, text: str) -> list[Finding]:
out: list[Finding] = []
lines = text.splitlines()
frontmatter_lines = _frontmatter_line_numbers(lines)
for number, line in enumerate(lines, start=1):
invisible = sorted({ord(c) for c in line if is_invisible_codepoint(ord(c))})
if invisible:
codepoints = ", ".join(f"U+{value:04X}" for value in invisible)
out.append(Finding("safety", "error", relative_path, number, "unicode.invisible",
f"invisible Unicode code point(s): {codepoints} — extraction "
f"should have stripped these"))
for rule_id, pattern, message in _CONTENT_RULES:
if pattern.search(line):
out.append(Finding("safety", "warn", relative_path, number, rule_id, message))
if _EXFILTRATION_TERM.search(line) or (
_OUTBOUND_TERM.search(line) and _SENSITIVE_TERM.search(line)
):
out.append(Finding("safety", "warn", relative_path, number, "tool.exfiltration_shape",
"exfiltration-shaped tool or sensitive-data language"))
if number in frontmatter_lines:
if re.match(r"^\s*allowed-tools\s*:", line, re.IGNORECASE):
out.append(Finding("safety", "error", relative_path, number,
"frontmatter.allowed_tools",
"generated frontmatter declares or widens tool authority"))
if re.match(r"^\s*disable-model-invocation\s*:\s*[\"']?(?:false|no|0)[\"']?\s*(?:#.*)?$",
line, re.IGNORECASE):
out.append(Finding("safety", "error", relative_path, number,
"frontmatter.model_invocation_enabled",
"generated frontmatter explicitly enables model invocation"))
return out
# --------------------------------------------------------------------------- budget
def check_budgets(contents: dict[str, str]) -> list[Finding]:
out: list[Finding] = []
for filename, cap in BUDGETS.items():
text = contents.get(filename)
if text is None:
severity = "error" if filename == "SKILL.md" else "warn"
out.append(Finding("budget", severity, filename, 1, "budget.missing_file",
f"{filename} is missing — the converter's Step 8/9 output is incomplete"))
continue
tokens = estimate_tokens(text)
if tokens > cap:
# SKILL.md is the only hard cap: it is resident, and compaction
# truncates from the end, so overflow silently drops the indexes.
severity = "error" if filename == "SKILL.md" else "warn"
out.append(Finding("budget", severity, filename, 1, "budget.over_cap",
f"~{tokens:,} tokens > {cap:,} cap"))
for relative, text in contents.items():
if not relative.startswith("chapters/"):
continue
tokens = estimate_tokens(text)
if tokens > CHAPTER_CEILING:
out.append(Finding("budget", "warn", relative, 1, "budget.chapter_over_ceiling",
f"~{tokens:,} tokens > {CHAPTER_CEILING:,} ceiling — chapters load "
f"on demand, but this one costs as much as the whole core"))
return out
# --------------------------------------------------------------------------- index
def check_indexes(contents: dict[str, str]) -> list[Finding]:
out: list[Finding] = []
master = contents.get("SKILL.md", "")
on_disk = {path for path in contents if path.startswith("chapters/")}
linked: set[str] = set()
for number, line in enumerate(master.splitlines(), start=1):
for target in _CHAPTER_LINK.findall(line):
linked.add(target)
if target not in on_disk:
out.append(Finding("index", "error", "SKILL.md", number, "index.dead_link",
f"Chapter Index links {target}, which does not exist"))
for orphan in sorted(on_disk - linked):
out.append(Finding("index", "warn", orphan, 1, "index.unindexed_chapter",
"chapter file is not linked from the Chapter Index — the agent has "
"no way to navigate to it"))
known_numbers = {
int(match.group(1))
for path in on_disk
if (match := _CHAPTER_FILE.match(Path(path).name))
}
for number, line in enumerate(master.splitlines(), start=1):
for group in _TOPIC_REF.findall(line):
for token in re.findall(r"ch(\d{1,3})", group, re.IGNORECASE):
if known_numbers and int(token) not in known_numbers:
out.append(Finding("index", "error", "SKILL.md", number, "index.topic_dangling",
f"Topic Index points at ch{token}, which has no chapter file"))
if on_disk and "## Topic Index" not in master:
out.append(Finding("index", "warn", "SKILL.md", 1, "index.no_topic_index",
"no Topic Index section — topic lookup falls back to reading every "
"chapter, which is the cost this skill exists to avoid"))
return out
# --------------------------------------------------------------------------- driver
def validate(skill_dir: Path, lens: str = "claude") -> list[Finding]:
root, files = _collect_skill_files(skill_dir)
contents = _read_skill_files(root, files)
findings = check_frontmatter(contents["SKILL.md"], lens)
for relative, text in contents.items():
findings.extend(check_safety(relative, text))
findings.extend(check_budgets(contents))
findings.extend(check_indexes(contents))
order = {"error": 0, "warn": 1}
return sorted(findings, key=lambda f: (order[f.severity], f.path.lower(), f.line, f.rule_id))
SAMPLE_SKILL = {
"SKILL.md": """---
name: sample-book-skill
description: "Knowledge base from \\"A Sample Book\\" by A. Author. Use when applying the
author's frameworks, studying the book, or referencing its concepts."
---
# A Sample Book
## Chapter Index
| # | Title | Key Frameworks |
|---|-------|----------------|
| [ch01](chapters/ch01-discovery-tax.md) | The Discovery Loop Tax | Discovery Loop Tax |
| [ch02](chapters/ch02-missing.md) | Missing On Purpose | — |
## Topic Index
- **Discovery Loop Tax** → ch01
- **Never Written** → ch07
""",
"chapters/ch01-discovery-tax.md": "# Chapter 1\n\n## Core Idea\nStructure once, query cheaply.\n",
"glossary.md": "**Discovery Loop Tax** — repeated navigation cost (Ch 1)\n",
}
def run_sample(lens: str, as_json: bool, strict: bool) -> int:
"""Validate a deliberately broken sample skill: a dead chapter link, a
dangling topic reference, and two missing supporting files.
Always exits 0. This is a demonstration of what the four families report,
not a gate — the findings below are the point, and a smoke test that fails
on purpose is indistinguishable from one that is broken.
"""
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp) / "sample-book-skill"
(root / "chapters").mkdir(parents=True)
for relative, text in SAMPLE_SKILL.items():
(root / relative).write_text(text, encoding="utf-8")
findings = validate(root, lens)
if not as_json:
print("Sample: a deliberately broken skill — a dead chapter link, a dangling topic\n"
"reference, and two missing supporting files. Expected findings:\n")
report(findings, lens, as_json, strict)
if not as_json:
print("\n(sample run — exit 0 regardless; on a real skill these would be exit 1)")
return 0
def report(findings: list[Finding], lens: str, as_json: bool, strict: bool) -> int:
errors = [f for f in findings if f.severity == "error"]
warns = [f for f in findings if f.severity == "warn"]
if as_json:
print(json.dumps({
"lens": lens,
"errors": len(errors),
"warnings": len(warns),
"findings": [asdict(f) for f in findings],
}, indent=2, ensure_ascii=False))
else:
label = LENSES[lens]["label"]
for finding in findings:
marker = "ERROR" if finding.severity == "error" else "WARN "
print(f" {marker} [{finding.family}] {_terminal_safe(finding.path)}:{finding.line} "
f"({finding.rule_id}) {finding.message}")
if findings:
print(f"\n{len(errors)} error(s), {len(warns)} warning(s) [{label}]")
print("Safety rules are intentionally broad and may match legitimate security or "
"AI-topic prose; review each finding in context. No files were modified.")
else:
print(f"Generated skill passed all checks [{label}].")
return 1 if errors or (strict and warns) else 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="book_skill_validator.py",
description="Validate a generated book skill: frontmatter, safety, budgets, indexes.",
)
parser.add_argument("path", nargs="?", help="generated skill directory (or its SKILL.md)")
parser.add_argument("--lens", choices=sorted(LENSES), default="claude",
help="which host's rules to validate against (default: claude)")
parser.add_argument("--strict", action="store_true",
help="treat warnings as failures too")
parser.add_argument("--output", choices=("text", "json"), default="text")
parser.add_argument("--sample", action="store_true",
help="validate a built-in, deliberately broken sample skill")
args = parser.parse_args(argv)
if args.sample:
return run_sample(args.lens, args.output == "json", args.strict)
if not args.path:
parser.print_usage(sys.stderr)
print("ERROR: give a generated skill directory (or use --sample).", file=sys.stderr)
return 2
try:
findings = validate(Path(args.path), args.lens)
except ScanError as exc:
print(f"ERROR validation incomplete: {exc}", file=sys.stderr)
return 2
return report(findings, args.lens, args.output == "json", args.strict)
if __name__ == "__main__":
try:
sys.exit(main())
except BrokenPipeError:
# `tool | head` closes the pipe while this is still writing. Exit quietly
# instead of dumping a traceback: redirect stdout to devnull first so the
# interpreter's shutdown flush cannot re-raise. 141 = 128 + SIGPIPE.
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
sys.exit(141)

27
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/__init__.py Normal file
View file

@ -0,0 +1,27 @@
"""Deterministic document-extraction library for the book-to-skill converter.
Vendored and adapted from virgiliojr94/book-to-skill (MIT). The CLI entry point
lives one directory up in ``extract_document.py`` — this package is a library
only, so importing it never prints, prompts, or installs anything.
Every format degrades to a standard-library parser; optional third-party
packages (docling, pypdf, pdfminer.six, ebooklib, beautifulsoup4, python-docx,
striprtf) only raise extraction quality. MOBI/AZW/AZW3 are the single exception
— they need Calibre's ``ebook-convert`` on PATH and have no fallback.
"""
from book_to_skill.exceptions import ExtractionError
from book_to_skill.utils import (
detect_structure,
estimate_tokens,
extract_single_file,
resolve_input_files,
)
__all__ = [
"ExtractionError",
"detect_structure",
"estimate_tokens",
"extract_single_file",
"resolve_input_files",
]

76
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/config.py Normal file
View file

@ -0,0 +1,76 @@
import os
import tempfile
from pathlib import Path
# Artifact filenames. The *directory* they land in is resolved per invocation by
# extract_document.py — deliberately NOT a module-level constant.
#
# Upstream defaulted to a fixed `<tempdir>/book_skill_work`. On a shared host
# that is CWE-377/CWE-59: any local user can pre-create the directory in a
# world-writable /tmp (the sticky bit stops deletion, not creation) and plant a
# symlink named full_text.txt or metadata.json pointing at a file the victim can
# write — Path.write_text follows symlinks. Two concurrent runs also silently
# clobber each other. See resolve_workdir() in extract_document.py: the default
# is now a per-invocation mkdtemp (0700 by construction, unpredictable name),
# and an explicitly requested directory is symlink-checked and mode-restricted.
OUTPUT_TEXT_NAME = "full_text.txt"
OUTPUT_META_NAME = "metadata.json"
# Permissions for everything the extractor creates. Extracted book text is the
# user's own document content; it does not belong to the host's other users.
WORKDIR_MODE = 0o700
ARTIFACT_MODE = 0o600
def workdir_from_env() -> str | None:
"""The BOOK_SKILL_WORKDIR override, if set. Read at call time, not import."""
return os.environ.get("BOOK_SKILL_WORKDIR") or None
def make_private_tempdir() -> Path:
"""A fresh, unpredictable, owner-only working directory for one invocation."""
return Path(tempfile.mkdtemp(prefix="book_skill_work_"))
# Token caps the converter's workflow commits to (SKILL.md Steps 7-9). Shared so
# book_skill_validator.py and token_budget_estimator.py — which both gate on
# these numbers — cannot drift apart.
SKILL_FILE_BUDGETS = {
"SKILL.md": 4_000,
"glossary.md": 1_500,
"patterns.md": 2_000,
"cheatsheet.md": 1_200,
}
# Not a target: chapters scale by book type and depth (Step 7). This is the
# runaway detector — at this size one chapter costs as much as the whole core.
CHAPTER_TOKEN_CEILING = 3_500
WORDS_PER_TOKEN = 0.75 # approximate (Latin / whitespace-delimited text)
# CJK scripts carry little or no whitespace, so word-splitting under-counts them
# by orders of magnitude. Count CJK codepoints directly against this
# chars-per-token ratio instead (see estimate_tokens in utils.py).
CJK_CHARS_PER_TOKEN = 1.5 # approximate for cl100k-style tokenizers
TEXT_EXTENSIONS = {".txt", ".text", ".md", ".markdown", ".rst", ".adoc", ".asciidoc"}
HTML_EXTENSIONS = {".html", ".htm", ".xhtml"}
CALIBRE_EBOOK_EXTENSIONS = {".mobi", ".azw", ".azw3"}
SUPPORTED_EXTENSIONS = {
".pdf", ".epub", ".docx", ".rtf",
*TEXT_EXTENSIONS,
*HTML_EXTENSIONS,
*CALIBRE_EBOOK_EXTENSIONS,
}
PYTHON_DEPENDENCIES = {
"docling": "docling",
"pypdf": "pypdf",
"pdfminer": "pdfminer.six",
"ebooklib": "ebooklib",
"bs4": "beautifulsoup4",
"docx": "python-docx",
"striprtf": "striprtf",
}
def supported_formats_message() -> str:
return ", ".join(sorted(SUPPORTED_EXTENSIONS))

309
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/dependencies.py Normal file
View file

@ -0,0 +1,309 @@
"""Optional-dependency probing for the extractor.
Vendored from virgiliojr94/book-to-skill (MIT) with one behavioural change:
the install mode defaults to ``report`` instead of ``ask``. Upstream prompts
interactively and installs packages into the caller's environment; inside an
agent session that is an unattended `pip install` triggered by reading a file.
Here the default prints the exact install command and uses the standard-library
fallback, and ``--install-missing yes`` remains available as an explicit opt-in.
"""
from __future__ import annotations
import importlib.util
import os
import shutil
import subprocess
import sys
from book_to_skill.config import PYTHON_DEPENDENCIES, HTML_EXTENSIONS
# Install-mode vocabulary:
# report — never install; print the command and use the fallback (default)
# ask — prompt, but only on a real TTY; falls back to `report` otherwise
# yes — install missing packages without prompting (explicit opt-in)
INSTALL_MODES = ("report", "ask", "yes")
DEFAULT_INSTALL_MODE = "report"
# Ordered groups for the --check preflight report. Each entry describes one
# format and what it needs. `modules` are optional Python packages (any one is
# enough unless noted); `system` are external commands resolved via PATH.
DEPENDENCY_GROUPS = [
{
"label": "PDF (text-heavy)",
"modules": ["pypdf", "pdfminer"],
"any_of_modules": True,
"any_tool_suffices": True,
"system": [("pdftotext", "poppler-utils",
"install poppler-utils with your system package manager "
"(Debian/Ubuntu: apt install poppler-utils; macOS: brew install poppler)")],
"note": "any one of pdftotext / pypdf / pdfminer is enough",
},
{
"label": "PDF (technical: tables, code, formulas)",
"modules": ["docling"],
"any_of_modules": True,
"system": [],
"note": "needed only for --mode technical; otherwise falls back to the text chain",
},
{
"label": "EPUB",
"modules": ["ebooklib", "bs4"],
"any_of_modules": False,
"system": [],
"note": "falls back to a stdlib zipfile parser if missing",
},
{
"label": "DOCX",
"modules": ["docx"],
"any_of_modules": True,
"system": [],
"note": "falls back to a stdlib ZIP/XML parser if missing",
},
{
"label": "HTML",
"modules": ["bs4"],
"any_of_modules": True,
"system": [],
"note": "falls back to the stdlib html.parser if missing",
},
{
"label": "RTF",
"modules": ["striprtf"],
"any_of_modules": True,
"system": [],
"note": "falls back to a basic regex cleanup if missing",
},
{
"label": "MOBI / AZW / AZW3",
"modules": [],
"any_of_modules": True,
"required": True,
"system": [
("ebook-convert", "Calibre", "install Calibre: https://calibre-ebook.com/download"),
],
"note": "no fallback — Calibre is required for these formats",
},
]
def python_module_available(module_name: str) -> bool:
return importlib.util.find_spec(module_name) is not None
def missing_python_packages(module_names: list[str]) -> list[str]:
missing = []
for module_name in module_names:
if not python_module_available(module_name):
missing.append(PYTHON_DEPENDENCIES[module_name])
return missing
def install_python_packages(packages: list[str]) -> bool:
if not packages:
return True
print(f"Installing missing Python package(s): {', '.join(packages)}")
try:
result = subprocess.run(
[sys.executable, "-m", "pip", "install", *packages],
text=True,
timeout=600,
)
except Exception as exc:
print(f"Package installation failed: {exc}", file=sys.stderr)
return False
importlib.invalidate_caches()
return result.returncode == 0
def normalize_install_mode(value: str | None = None) -> str:
"""Resolve the install mode from an explicit flag value, then the
``BOOK_SKILL_INSTALL_MISSING`` environment variable, then the default.
Anything unrecognized resolves to ``report`` — the safe end of the range —
so a typo can never turn into an unattended install.
"""
raw = (value or os.environ.get("BOOK_SKILL_INSTALL_MISSING") or DEFAULT_INSTALL_MODE).lower()
if raw in {"1", "true", "y", "yes", "install"}:
return "yes"
if raw == "ask":
return "ask"
return "report"
def offer_dependency_install(
*,
feature: str,
module_names: list[str],
fallback: str | None,
install_mode: str,
) -> None:
packages = missing_python_packages(module_names)
if not packages:
return
message = f"{feature} uses {', '.join(packages)} if installed"
if fallback:
message += f", otherwise {fallback}"
message += "."
print(message)
should_install = False
if install_mode == "yes":
should_install = True
elif install_mode == "ask" and sys.stdin.isatty():
answer = input("Missing package(s) detected. Do you want to install? y=install, n=fallback: ").strip().lower()
should_install = answer in {"y", "yes", "install"}
else:
# report mode (or `ask` with no TTY): never install, but always say what
# would raise quality, so the user can decide outside the agent session.
print(f" install for better quality: {sys.executable} -m pip install {' '.join(packages)}")
if fallback:
print(" proceeding with the standard-library fallback.")
else:
print(" no fallback exists for this format.", file=sys.stderr)
if not should_install:
if fallback:
print(f"Using fallback: {fallback}.")
return
if install_python_packages(packages):
still_missing = missing_python_packages(module_names)
if not still_missing:
print("Package installation complete.")
return
print(f"Package installation incomplete; still missing: {', '.join(still_missing)}", file=sys.stderr)
else:
print("Package installation failed.", file=sys.stderr)
if fallback:
print(f"Using fallback: {fallback}.")
def prepare_dependencies(ext: str, extraction_mode: str, install_mode: str) -> None:
if ext == ".pdf" and extraction_mode == "technical":
offer_dependency_install(
feature="Technical PDF extraction",
module_names=["docling"],
fallback="the PDF text fallback chain",
install_mode=install_mode,
)
if ext == ".pdf" and not shutil.which("pdftotext"):
offer_dependency_install(
feature="PDF text extraction",
module_names=["pypdf", "pdfminer"],
fallback="any installed Python PDF parser; extraction fails if none are available",
install_mode=install_mode,
)
if ext == ".epub":
offer_dependency_install(
feature="EPUB extraction",
module_names=["ebooklib", "bs4"],
fallback="a stdlib ZIP/HTML parser",
install_mode=install_mode,
)
if ext in HTML_EXTENSIONS:
offer_dependency_install(
feature="HTML extraction",
module_names=["bs4"],
fallback="a stdlib HTML parser",
install_mode=install_mode,
)
if ext == ".docx":
offer_dependency_install(
feature="DOCX extraction",
module_names=["docx"],
fallback="a stdlib ZIP/XML parser",
install_mode=install_mode,
)
if ext == ".rtf":
offer_dependency_install(
feature="RTF extraction",
module_names=["striprtf"],
fallback="a basic regex cleanup fallback",
install_mode=install_mode,
)
def run_dependency_check() -> int:
"""Scan every optional dependency across all formats and print a status
report plus the exact command to install whatever is missing.
Returns a process exit code: 0 always (a missing optional dep is not an
error — most formats degrade to a fallback). Intended for `extract.py --check`.
"""
print("book-to-skill — dependency check\n")
missing_pip_packages: list[str] = []
missing_system: list[tuple[str, str]] = [] # (name, install hint)
for group in DEPENDENCY_GROUPS:
print(f" {group['label']}")
present_modules = [m for m in group["modules"] if python_module_available(m)]
absent_modules = [m for m in group["modules"] if not python_module_available(m)]
system_present = [c for c, _, _ in group["system"] if shutil.which(c)]
system_absent = [c for c, _, _ in group["system"] if not shutil.which(c)]
for module_name in group["modules"]:
pip_name = PYTHON_DEPENDENCIES.get(module_name, module_name)
ok = module_name in present_modules
print(f" {'✓' if ok else '✗'} python: {pip_name}")
if not ok:
missing_pip_packages.append(pip_name)
for cmd, pretty, hint in group["system"]:
ok = cmd in system_present
print(f" {'✓' if ok else '✗'} system: {cmd} ({pretty})")
if not ok:
missing_system.append((pretty, hint))
# Satisfaction semantics:
# - any_tool_suffices: any single extractor (module OR system) is enough
# - any_of_modules: at least one module present
# - otherwise: every listed module present
# - system tools that aren't alternatives are always required
if group.get("any_tool_suffices"):
satisfied = bool(present_modules) or bool(system_present)
else:
if group["modules"]:
satisfied = bool(present_modules) if group["any_of_modules"] else not absent_modules
else:
satisfied = True
if system_absent:
satisfied = False
if satisfied:
status = "ready"
elif group.get("required"):
status = "MISSING — required, no fallback"
else:
status = "fallback available (install for best quality)"
print(f" → {status} — {group['note']}\n")
# Deduplicate while preserving order
missing_pip_packages = list(dict.fromkeys(missing_pip_packages))
missing_system = list(dict.fromkeys(missing_system))
if not missing_pip_packages and not missing_system:
print("All optional dependencies are installed. You're ready for every format.")
return 0
print("To enable the best extractor for every format, install the missing pieces:\n")
if missing_pip_packages:
print(f" {sys.executable} -m pip install {' '.join(missing_pip_packages)}")
for pretty, hint in missing_system:
print(f" # {pretty}: {hint}")
print(
"\nNote: missing Python packages are optional — most formats fall back to a "
"stdlib parser. Calibre is the only hard requirement, and only for MOBI/AZW files."
)
return 0

2
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/exceptions.py Normal file
View file

@ -0,0 +1,2 @@
class ExtractionError(Exception):
"""Raised when a single file cannot be extracted (non-fatal in batch mode)."""

1
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/__init__.py Normal file
View file

@ -0,0 +1 @@
# Parsers package

37
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/calibre.py Normal file
View file

@ -0,0 +1,37 @@
from __future__ import annotations
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
def extract_with_ebook_convert(input_path: str) -> str | None:
"""Convert MOBI/AZW/AZW3 via Calibre's ebook-convert.
The intermediate file goes in a private per-call temp directory rather than
upstream's fixed `<tempdir>/book_skill_work/ebook-convert-output.txt`. Two
reasons: a fixed, world-guessable output path that a subprocess writes to is
the CWE-377 shape, and the module-level constant upstream used ignored the
caller's --workdir entirely, so the scratch file escaped the directory the
caller asked for.
"""
if not shutil.which("ebook-convert"):
return None
try:
input_path = os.path.abspath(input_path)
with tempfile.TemporaryDirectory(prefix="book_skill_calibre_") as scratch:
output_path = Path(scratch) / "ebook-convert-output.txt"
result = subprocess.run(
["ebook-convert", input_path, str(output_path)],
capture_output=True, text=True, timeout=300
)
if result.returncode == 0 and output_path.exists():
text = output_path.read_text(encoding="utf-8", errors="replace")
if text.strip():
return text
except Exception as e:
print(f" [warn] extract_with_ebook_convert failed: {type(e).__name__}: {e}", file=sys.stderr)
return None

102
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/docx.py Normal file
View file

@ -0,0 +1,102 @@
from __future__ import annotations
import zipfile
import sys
from book_to_skill.exceptions import ExtractionError
from book_to_skill.zip_safety import safe_read, validate_zip_xml_safety
def extract_docx_with_python_docx(docx_path: str) -> str | None:
try:
import docx
document = docx.Document(docx_path)
parts = [paragraph.text for paragraph in document.paragraphs if paragraph.text]
for table in document.tables:
for row in table.rows:
cells = [cell.text.strip() for cell in row.cells]
if any(cells):
parts.append("\t".join(cells))
return "\n".join(parts)
except ImportError:
return None
except Exception as e:
print(f" [warn] extract_docx_with_python_docx failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def extract_docx_with_zipfile(docx_path: str) -> str | None:
try:
import xml.etree.ElementTree as ET
with zipfile.ZipFile(docx_path) as zf:
xml_bytes = safe_read(zf, "word/document.xml")
root = ET.fromstring(xml_bytes)
ns = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}"
parts: list[str] = []
def emit_block(elem) -> None:
# Walk block content in document order. Paragraphs join their runs;
# tables emit one tab-joined line per row (same row format as the
# python-docx path, but order-preserving — python-docx appends all
# tables last). Unknown wrappers (e.g. <w:sdt> content controls) are
# recursed into so their paragraphs/tables are not lost; <w:p> and
# <w:tbl> are NOT recursed into, so table-cell paragraphs are not
# double-counted. Cell text concatenates the cell's runs; nested
# tables fold into the parent cell and are also emitted standalone
# (rare; best-effort).
for child in elem:
tag = child.tag
if tag == f"{ns}p":
texts = [t.text for t in child.iter(f"{ns}t") if t.text]
if texts:
parts.append("".join(texts))
elif tag == f"{ns}tbl":
for row in child.iter(f"{ns}tr"):
cells = []
for cell in row.iter(f"{ns}tc"):
cell_texts = [t.text for t in cell.iter(f"{ns}t") if t.text]
cells.append("".join(cell_texts).strip())
if any(cells):
parts.append("\t".join(cells))
else:
emit_block(child)
body = root.find(f"{ns}body")
emit_block(body if body is not None else root)
return "\n".join(parts) if parts else None
except Exception as e:
print(f" [warn] extract_docx_with_zipfile failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def validate_docx_xml_safety(docx_path: str) -> None:
"""Refuse a DOCX whose XML declares a DTD or entity (billion-laughs / XXE).
Now a thin wrapper over the shared zip guard so EPUB gets identical
treatment; the scan also enforces per-member size and compression-ratio caps
before decompressing anything.
"""
validate_zip_xml_safety(docx_path, label="DOCX")
def extract_docx(docx_path: str) -> tuple[str, str]:
validate_docx_xml_safety(docx_path)
print("Trying python-docx...", end=" ", flush=True)
text = extract_docx_with_python_docx(docx_path)
if text and text.strip():
print("OK")
return text, "python-docx"
print("not available")
print("Trying stdlib DOCX parser...", end=" ", flush=True)
text = extract_docx_with_zipfile(docx_path)
if text and text.strip():
print("OK")
return text, "zipfile-docx"
print("FAILED")
raise ExtractionError(
"Could not extract text from DOCX.\n"
"Install python-docx for best results:\n"
" pip3 install python-docx"
)

144
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/epub.py Normal file
View file

@ -0,0 +1,144 @@
from __future__ import annotations
import posixpath
import re
import zipfile
import sys
from book_to_skill.parsers.html_text import _HTMLTextExtractor
from book_to_skill.exceptions import ExtractionError
from book_to_skill.zip_safety import _Budget, safe_read, validate_zip_xml_safety
def extract_with_ebooklib(epub_path: str) -> str | None:
"""Extract via ebooklib, after the same XML guard DOCX gets.
ebooklib parses container.xml, the OPF package document and the content docs
with a third-party XML stack whose entity handling this code does not
control. The archive is an untrusted file the user was handed, so it is
screened for DTD/entity declarations before ebooklib sees it — upstream ran
this check on DOCX only.
"""
validate_zip_xml_safety(epub_path, label="EPUB")
try:
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup
book = epub.read_epub(epub_path)
parts = []
for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
soup = BeautifulSoup(item.get_content(), "html.parser")
parts.append(soup.get_text(separator="\n"))
return "\n\n".join(parts)
except ImportError:
return None
except Exception as e:
print(f" [warn] extract_with_ebooklib failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def _find_opf_path(zf: zipfile.ZipFile) -> str | None:
"""Locate the OPF package document inside an EPUB archive.
First tries ``META-INF/container.xml`` (the spec-defined entry point),
then falls back to scanning the archive for any ``.opf`` file.
"""
# Spec-defined: read container.xml for the rootfile path
try:
container = safe_read(zf, "META-INF/container.xml").decode("utf-8", errors="replace")
match = re.search(r'full-path=["\']([^"\']+\.opf)["\']', container)
if match:
return match.group(1)
except ExtractionError:
# A size/ratio refusal from safe_read is a verdict, not a parse failure:
# let it reach the caller instead of silently falling through to the .opf
# glob below. Upstream wrote `except (KeyError, Exception)`, which is just
# `except Exception` and swallowed everything.
raise
except (KeyError, OSError, UnicodeDecodeError, zipfile.BadZipFile):
# container.xml absent or malformed — the glob fallback is the answer.
pass
# Fallback: glob for any .opf file
opf_files = [n for n in zf.namelist() if n.endswith(".opf")]
return opf_files[0] if opf_files else None
def extract_with_zipfile(epub_path: str) -> str | None:
"""stdlib-only EPUB extractor: unzip → parse HTML files."""
try:
with zipfile.ZipFile(epub_path) as zf:
budget = _Budget()
names = zf.namelist()
# Locate OPF and determine its directory for resolving relative hrefs
opf_path = _find_opf_path(zf)
opf_dir = posixpath.dirname(opf_path) if opf_path else ""
# Build reading order from the OPF spine (not the manifest's href
# order), then append any remaining content docs as a safety net.
spine_order: list[str] = []
seen: set[str] = set()
if opf_path:
opf_text = safe_read(zf, opf_path, budget).decode("utf-8", errors="replace")
# Manifest: item id -> resolved href. Parse each <item> opening
# tag so attribute order (id before/after href) does not matter;
# both self-closing <item .../> and <item ...></item> forms work
# because all attributes live in the opening tag.
manifest: dict[str, str] = {}
for item_tag in re.findall(r"<item\b[^>]*?/?>", opf_text):
id_m = re.search(r'\bid=["\']([^"\']+)["\']', item_tag)
href_m = re.search(r'\bhref=["\']([^"\']+)["\']', item_tag)
if id_m and href_m:
href = href_m.group(1)
resolved = posixpath.normpath(posixpath.join(opf_dir, href)) if opf_dir else href
manifest[id_m.group(1)] = resolved
# Spine: ordered idrefs -> hrefs (true reading order).
for idref in re.findall(r'<itemref\b[^>]*?\bidref=["\']([^"\']+)["\']', opf_text):
href = manifest.get(idref)
if href and href not in seen:
spine_order.append(href)
seen.add(href)
# Safety net: append remaining manifest content documents (e.g. a
# nav doc not in the spine) in manifest order, so nothing is lost.
for href in manifest.values():
if href.endswith((".html", ".xhtml")) and href not in seen:
spine_order.append(href)
seen.add(href)
html_files = spine_order or sorted(
n for n in names if n.endswith((".html", ".xhtml"))
)
if not html_files:
return None
parts = []
for name in html_files:
try:
raw = safe_read(zf, name, budget).decode("utf-8", errors="replace")
parser = _HTMLTextExtractor()
parser.feed(raw)
parts.append(parser.get_text())
except Exception:
continue
return "\n\n".join(parts) if parts else None
except Exception as e:
print(f" [warn] extract_with_zipfile failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def count_epub_chapters(epub_path: str) -> int:
"""Count spine items (approximate chapter count) without dependencies."""
try:
with zipfile.ZipFile(epub_path) as zf:
opf_path = _find_opf_path(zf)
if not opf_path:
return 0
opf_text = safe_read(zf, opf_path).decode("utf-8", errors="replace")
return len(re.findall(r'<itemref\b', opf_text))
except Exception:
return 0

113
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/html_text.py Normal file
View file

@ -0,0 +1,113 @@
"""Stdlib HTML → plain text extraction.
Named ``html_text`` rather than upstream's ``html``: a module called
``html.py`` shadows the standard library's ``html`` package whenever its own
directory lands on ``sys.path[0]`` — which happens the moment anyone runs the
file directly — and ``import html.parser`` then fails with
"'html' is not a package".
"""
from __future__ import annotations
import html.parser
from book_to_skill.parsers.text import read_text_file
class _HTMLTextExtractor(html.parser.HTMLParser):
"""Minimal HTML → plain text converter using stdlib only."""
SKIP_TAGS = {"script", "style", "head"}
# Block-level elements. A boundary is emitted both when they open and when
# they CLOSE — closing matters, because without it the text of two adjacent
# blocks concatenates ("<h2>Chapter 1</h2>Intro" -> "Chapter 1Intro"), which
# destroys chapter detection: _EXPLICIT_CHAPTER requires a word boundary
# after the number, and "1I" has none.
BLOCK_TAGS = frozenset({
"address", "article", "aside", "blockquote", "br", "dd", "details",
"div", "dl", "dt", "fieldset", "figcaption", "figure", "footer",
"form", "h1", "h2", "h3", "h4", "h5", "h6", "header", "hgroup", "hr",
"li", "main", "nav", "ol", "p", "pre", "section", "table", "tbody",
"tfoot", "thead", "tr", "ul",
})
# Table cells are separated by a tab rather than a newline so a row stays on
# one line — the same convention the stdlib DOCX fallback already uses for
# tab-joined table rows, and what keeps a table-formatted table of contents
# ("Chapter 1 | Introduction | 1") parseable as a single heading line.
CELL_TAGS = frozenset({"td", "th"})
def __init__(self):
super().__init__()
self._parts: list[str] = []
self._skip_depth = 0
# Strongest boundary awaiting the next non-blank text run. Deferring it
# (instead of appending immediately) means nested blocks such as
# "<div><p>x" collapse to one separator rather than a run of blank lines.
self._pending = ""
def _mark(self, separator: str) -> None:
# "\n" outranks "\t": a row/block boundary must not be downgraded to a
# cell boundary by a <td> that opens straight after a <tr>.
if separator == "\n" or not self._pending:
self._pending = separator
def handle_starttag(self, tag, attrs):
if tag in self.SKIP_TAGS:
self._skip_depth += 1
if tag in self.BLOCK_TAGS:
self._mark("\n")
elif tag in self.CELL_TAGS:
self._mark("\t")
def handle_endtag(self, tag):
if tag in self.SKIP_TAGS:
if self._skip_depth:
self._skip_depth -= 1
return
if tag in self.BLOCK_TAGS:
self._mark("\n")
elif tag in self.CELL_TAGS:
self._mark("\t")
def handle_data(self, data):
if self._skip_depth:
return
if self._pending:
if not data.strip():
# Whitespace-only text between tags is layout indentation. It
# cannot satisfy a pending boundary, and emitting it before the
# boundary would just add trailing spaces — drop it and keep
# waiting for real content.
return
# Suppress a leading separator so the output does not start with a
# blank line.
if self._parts:
self._parts.append(self._pending)
self._pending = ""
self._parts.append(data)
def get_text(self) -> str:
# HTMLParser(convert_charrefs=True) already decoded entities in
# handle_data; do NOT unescape again or double-encoded entities
# (e.g. "&amp;amp;") collapse incorrectly.
return "".join(self._parts)
def extract_html_content(raw_html: str) -> str:
try:
from bs4 import BeautifulSoup
soup = BeautifulSoup(raw_html, "html.parser")
for element in soup(["script", "style", "head"]):
element.decompose()
return soup.get_text(separator="\n")
except ImportError:
parser = _HTMLTextExtractor()
parser.feed(raw_html)
return parser.get_text()
def extract_html_file(path: str) -> str | None:
raw = read_text_file(path)
if raw is None:
return None
return extract_html_content(raw)

142
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/pdf.py Normal file
View file

@ -0,0 +1,142 @@
from __future__ import annotations
import os
import re
import shutil
import subprocess
import sys
from collections import Counter
_PDF_PAGE_NUM = re.compile(r"^\s*(\d{1,4}|[ivxlcdm]{1,7})\s*$", re.IGNORECASE)
_PDF_HYPHEN_WRAP = re.compile(r"(\w)-\n(\w)")
def clean_pdftotext(text: str) -> str:
"""Clean pdftotext '-layout' output (pages are form-feed delimited): drop
repeated running headers/footers and edge page numbers, and join words split
across a line by a hyphen."""
pages = text.split("\f")
if len(pages) >= 3:
# A top/bottom line repeated on > half the pages is boilerplate.
edge = Counter()
for p in pages:
nb = [ln.strip() for ln in p.splitlines() if ln.strip()]
if nb:
edge[nb[0]] += 1
edge[nb[-1]] += 1
boiler = {ln for ln, c in edge.items() if c > len(pages) / 2}
kept = []
for p in pages:
lines = p.splitlines()
nb_idx = [i for i, ln in enumerate(lines) if ln.strip()]
first = nb_idx[0] if nb_idx else None
last = nb_idx[-1] if nb_idx else None
for i, ln in enumerate(lines):
s = ln.strip()
if s in boiler:
continue
# Drop a bare page number only at a page edge (varies per page).
if i in (first, last) and _PDF_PAGE_NUM.match(s):
continue
kept.append(ln)
text = "\n".join(kept)
else:
text = text.replace("\f", "\n")
# Naive dehyphenation: may join a genuinely-hyphenated wrapped
# compound ("well-\nknown" -> "wellknown"). Dictionary-aware split if it bites.
return _PDF_HYPHEN_WRAP.sub(r"\1\2", text)
def extract_with_pdftotext(pdf_path: str) -> str | None:
if not shutil.which("pdftotext"):
return None
try:
pdf_path = os.path.abspath(pdf_path)
result = subprocess.run(
["pdftotext", "-layout", pdf_path, "-"],
capture_output=True, text=True, timeout=120,
encoding="utf-8", errors="replace",
)
if result.returncode == 0 and result.stdout.strip():
return clean_pdftotext(result.stdout)
except Exception as e:
print(f" [warn] extract_with_pdftotext failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def extract_with_pypdf(pdf_path: str) -> str | None:
try:
import pypdf
text_parts = []
with open(pdf_path, "rb") as f:
reader = pypdf.PdfReader(f)
for page in reader.pages:
try:
text_parts.append(page.extract_text() or "")
except Exception:
text_parts.append("")
return "\n".join(text_parts)
except ImportError:
return None
except Exception as e:
print(f" [warn] extract_with_pypdf failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def extract_with_pdfminer(pdf_path: str) -> str | None:
try:
from pdfminer.high_level import extract_text
return extract_text(pdf_path)
except ImportError:
return None
except Exception as e:
print(f" [warn] extract_with_pdfminer failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def extract_with_docling(pdf_path: str) -> str | None:
"""Layout-aware extraction using Docling. Best for technical books with tables and code."""
try:
from docling.document_converter import DocumentConverter
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.datamodel.base_models import InputFormat
from docling.document_converter import PdfFormatOption
pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = False
pipeline_options.do_table_structure = True
converter = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)
result = converter.convert(pdf_path)
return result.document.export_to_markdown()
except ImportError:
return None
except Exception as e:
print(f" [warn] extract_with_docling failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
def count_pages(pdf_path: str) -> int:
# Try pdfinfo first
if shutil.which("pdfinfo"):
try:
pdf_path = os.path.abspath(pdf_path)
result = subprocess.run(
["pdfinfo", pdf_path], capture_output=True, text=True, timeout=15
)
for line in result.stdout.splitlines():
if line.startswith("Pages:"):
return int(line.split(":")[1].strip())
except Exception:
pass
# Fallback: count pages with pypdf
try:
import pypdf
with open(pdf_path, "rb") as f:
return len(pypdf.PdfReader(f).pages)
except Exception:
return 0

146
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/rtf.py Normal file
View file

@ -0,0 +1,146 @@
import html
import re
import sys
from book_to_skill.parsers.text import read_text_file
from book_to_skill.exceptions import ExtractionError
# RTF unicode escape: \uN (signed decimal) followed by its fallback char(s).
# Decode the code point and drop the standard single fallback — a \'XX hex byte
# or a literal "?". Assumes the default \uc1 (one fallback char); \ucN directives
# and multi-char/group fallbacks are not parsed (best-effort fallback only).
_RTF_UNICODE = re.compile(r"\\u(-?\d+)[ ]?(?:\\'[0-9a-fA-F]{2}|\?)?")
def _rtf_unicode_repl(match: re.Match) -> str:
cp = int(match.group(1)) % 0x10000 # RTF uses signed 16-bit; wrap negatives
if cp == 0 or 0xD800 <= cp <= 0xDFFF: # NUL and lone surrogates: unwanted in text
return ""
return chr(cp)
# RTF groups whose contents are metadata or formatting tables rather than
# document text. Stripping only the control words inside them (what the cleanup
# below does) leaves the residue behind: font and style *names*, the generator
# string, and the \info title/author all end up in the extracted book text.
_SKIP_DESTINATIONS = frozenset({
"fonttbl", # {\fonttbl{\f0\fnil Calibri;}} -> "Calibri;"
"colortbl", # {\colortbl;\red255...;} -> ";;;"
"stylesheet", # {\stylesheet{\s0 Normal;}} -> "Normal;"
"info", # {\info{\title X}{\author Y}} -> "XY"
"listtable", "listoverridetable", "revtbl", "rsidtbl",
"latentstyles", "datastore", "themedata", "colorschememapping",
"filetbl", "xmlnstbl", "pgptbl", "protusertbl", "userprops",
"docvar",
"pict", "objdata", # binary image / OLE payloads as hex text
"bkmkstart", "bkmkend",
})
# The first control word of a group, allowing the "\*" ignorable-destination
# prefix: "{\fonttbl", "{\*\generator", "{\*\bkmkstart".
_GROUP_DESTINATION = re.compile(r"\\\*?\\?([a-zA-Z]+)")
def _strip_destination_groups(raw: str) -> str:
"""Remove RTF groups that hold no document text.
Tracks brace depth so a whole group is dropped, not just its control words.
Per the RTF spec a reader that does not understand a ``\\*`` destination must
skip the entire group, which also handles ``\\*\\generator`` and any vendor
extension without naming it. Escaped ``\\{`` / ``\\}`` / ``\\\\`` are not
treated as delimiters.
A useful side effect: for a field, ``{\\field{\\*\\fldinst HYPERLINK ...}
{\\fldrslt visible text}}`` keeps the result and drops the instruction.
"""
out: list[str] = []
index = 0
depth = 0
skip_at_depth = 0 # non-zero while inside a skipped group
length = len(raw)
while index < length:
char = raw[index]
# Escaped literal: "\{", "\}", "\\" are text, never group delimiters.
if char == "\\" and index + 1 < length and raw[index + 1] in "{}\\":
if not skip_at_depth:
out.append(raw[index:index + 2])
index += 2
continue
if char == "{":
depth += 1
if not skip_at_depth:
match = _GROUP_DESTINATION.match(raw, index + 1)
ignorable = raw.startswith("{\\*", index)
if ignorable or (match and match.group(1) in _SKIP_DESTINATIONS):
skip_at_depth = depth
else:
out.append(char)
index += 1
continue
if char == "}":
if skip_at_depth and depth == skip_at_depth:
skip_at_depth = 0
elif not skip_at_depth:
out.append(char)
depth -= 1
index += 1
continue
if not skip_at_depth:
out.append(char)
index += 1
if skip_at_depth:
# Unterminated destination group: the file is malformed and everything
# after the unclosed brace was just dropped, which could be the whole
# book. Leaking some metadata residue is the lesser evil, so fall back
# to the unscanned text rather than returning a truncated document.
return raw
return "".join(out)
def strip_rtf_fallback(raw: str) -> str:
# Drop metadata/table groups wholesale first, so their contents never reach
# the control-word cleanup that would otherwise strip the markup and leave
# the names behind as if they were prose.
raw = _strip_destination_groups(raw)
raw = _RTF_UNICODE.sub(_rtf_unicode_repl, raw) # decode \uN escapes first
raw = re.sub(r"\\'[0-9a-fA-F]{2}", " ", raw)
raw = re.sub(r"\\par[d]?", "\n", raw)
raw = re.sub(r"\\tab", "\t", raw)
# Park the three escaped literals ("\\", "\{", "\}") on placeholders before
# the sweeps below, which would otherwise strip the backslash as a control
# symbol and then delete the brace along with the real group delimiters —
# leaving a stray "\" where the book said "{a, b}". Longest escape first.
raw = (
raw.replace("\\\\", "\x01")
.replace("\\{", "\x02")
.replace("\\}", "\x03")
)
raw = re.sub(r"\\[a-zA-Z]+-?\d* ?", "", raw)
raw = raw.replace("{", "").replace("}", "")
raw = raw.replace("\x01", "\\").replace("\x02", "{").replace("\x03", "}")
return html.unescape(raw)
def extract_rtf(rtf_path: str) -> tuple[str, str]:
raw = read_text_file(rtf_path)
if raw is None:
raise ExtractionError(f"Could not read RTF file: {rtf_path}")
try:
from striprtf.striprtf import rtf_to_text
text = rtf_to_text(raw)
if text.strip():
return text, "striprtf"
except ImportError:
pass
except Exception as e:
print(f" [warn] extract_rtf/striprtf failed: {type(e).__name__}: {e}", file=sys.stderr)
return strip_rtf_fallback(raw), "rtf-regex"

38
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/text.py Normal file
View file

@ -0,0 +1,38 @@
from __future__ import annotations
import sys
from pathlib import Path
# Byte-order marks, longest first: the UTF-32 LE BOM ("ff fe 00 00") starts with
# the UTF-16 LE BOM ("ff fe"), so UTF-32 must be checked before UTF-16.
_BOMS = (
(b"\xef\xbb\xbf", "utf-8-sig"),
(b"\xff\xfe\x00\x00", "utf-32"),
(b"\x00\x00\xfe\xff", "utf-32"),
(b"\xff\xfe", "utf-16"),
(b"\xfe\xff", "utf-16"),
)
def read_text_file(path: str) -> str | None:
try:
data = Path(path).read_bytes()
except Exception as e:
print(f" [warn] read_text_file failed: {type(e).__name__}: {e}", file=sys.stderr)
return None
# Decode by BOM when present (the utf-16/utf-32 codecs strip the BOM and
# auto-select byte order).
for bom, encoding in _BOMS:
if data.startswith(bom):
try:
return data.decode(encoding)
except (UnicodeDecodeError, LookupError):
break
# No (usable) BOM: fall back to the prior chain for BOM-less files.
for encoding in ("utf-8", "cp1252", "latin-1"):
try:
return data.decode(encoding)
except UnicodeDecodeError:
continue
return None

96
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/sanitize.py Normal file
View file

@ -0,0 +1,96 @@
from __future__ import annotations
# Invisible code points used to hide document-borne prompt injection. Grouped by
# attack shape so the reasoning behind each entry stays reviewable.
#
# 1. Zero-width and invisible spacers. Render as nothing, so text between them is
# invisible to a human reading the page but plain to the model.
_ZERO_WIDTH_CODEPOINTS = frozenset({
0x200B, # ZERO WIDTH SPACE
0x200C, # ZERO WIDTH NON-JOINER
0x200D, # ZERO WIDTH JOINER
0x2060, # WORD JOINER
0xFEFF, # ZERO WIDTH NO-BREAK SPACE / BOM outside position 0
0x00AD, # SOFT HYPHEN — invisible except at a line break
0x034F, # COMBINING GRAPHEME JOINER — no rendering effect at all
0x180E, # MONGOLIAN VOWEL SEPARATOR
0x2061, # FUNCTION APPLICATION
0x2062, # INVISIBLE TIMES
0x2063, # INVISIBLE SEPARATOR
0x2064, # INVISIBLE PLUS
})
# 2. Bidirectional formatting controls — the Trojan Source class
# (CVE-2021-42574). These do not change the character sequence a model reads,
# they change the order a human SEES. A crafted line can display as innocuous
# study advice while the model consumes an injected instruction, so the
# reviewer approving a generated skill and the agent loading it disagree.
# Removing them makes rendered order match logical order.
#
# Legitimate right-to-left books are unaffected: the Unicode Bidi Algorithm
# derives direction from the characters themselves, so Arabic and Hebrew still
# render right-to-left without these. Only explicit embeddings, overrides and
# isolates are dropped, and running prose essentially never needs them.
_BIDI_CONTROL_CODEPOINTS = frozenset({
0x200E, # LEFT-TO-RIGHT MARK
0x200F, # RIGHT-TO-LEFT MARK
0x061C, # ARABIC LETTER MARK
0x202A, # LEFT-TO-RIGHT EMBEDDING
0x202B, # RIGHT-TO-LEFT EMBEDDING
0x202C, # POP DIRECTIONAL FORMATTING
0x202D, # LEFT-TO-RIGHT OVERRIDE
0x202E, # RIGHT-TO-LEFT OVERRIDE
0x2066, # LEFT-TO-RIGHT ISOLATE
0x2067, # RIGHT-TO-LEFT ISOLATE
0x2068, # FIRST STRONG ISOLATE
0x2069, # POP DIRECTIONAL ISOLATE
})
# 3. Characters that are not format controls (so a category-based filter misses
# them) but still render as blank width. Unlike a space they are letters, so
# they survive whitespace normalisation and can pad hidden text.
_INVISIBLE_LETTER_CODEPOINTS = frozenset({
0x115F, # HANGUL CHOSEONG FILLER
0x1160, # HANGUL JUNGSEONG FILLER
0x3164, # HANGUL FILLER
0xFFA0, # HALFWIDTH HANGUL FILLER
})
_INVISIBLE_CODEPOINTS = (
_ZERO_WIDTH_CODEPOINTS
| _BIDI_CONTROL_CODEPOINTS
| _INVISIBLE_LETTER_CODEPOINTS
)
# 4. The Unicode tag block. Originally language tags, now used to smuggle an
# entire ASCII payload as invisible "tag" characters.
_TAG_BLOCK_START = 0xE0000
_TAG_BLOCK_END = 0xE007F
def is_invisible_codepoint(codepoint: int) -> bool:
"""Return True if the code point renders as nothing and should be stripped.
Exposed so the generated-skill scanner can flag exactly what extraction
strips. When the two sets drift, the extractor lets a character through that
the scanner then warns about — or worse, neither layer covers it.
"""
return (
codepoint in _INVISIBLE_CODEPOINTS
or _TAG_BLOCK_START <= codepoint <= _TAG_BLOCK_END
)
def sanitize_extracted_text(text: str) -> tuple[str, int]:
"""Remove invisible code points used for document-borne prompt injection."""
kept: list[str] = []
removed = 0
for character in text:
if is_invisible_codepoint(ord(character)):
removed += 1
continue
kept.append(character)
return "".join(kept), removed

601
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/utils.py Normal file
View file

@ -0,0 +1,601 @@
"""Extraction, token-estimation and chapter-structure detection helpers.
Vendored from virgiliojr94/book-to-skill (MIT) with the CLI layer removed:
argument parsing, the attribution banner and ``main()`` now live in
``extract_document.py`` so this module stays a side-effect-free library.
"""
from __future__ import annotations
import glob
import json
import os
import re
import sys
import shutil
import zipfile
from pathlib import Path
from book_to_skill.exceptions import ExtractionError
from book_to_skill.config import (
WORDS_PER_TOKEN,
CJK_CHARS_PER_TOKEN,
SUPPORTED_EXTENSIONS,
TEXT_EXTENSIONS,
HTML_EXTENSIONS,
CALIBRE_EBOOK_EXTENSIONS,
supported_formats_message,
)
from book_to_skill.dependencies import prepare_dependencies
from book_to_skill.parsers.text import read_text_file
from book_to_skill.parsers.html_text import extract_html_file
from book_to_skill.parsers.docx import extract_docx
from book_to_skill.parsers.rtf import extract_rtf
from book_to_skill.parsers.calibre import extract_with_ebook_convert
from book_to_skill.parsers.pdf import (
extract_with_docling,
extract_with_pdftotext,
extract_with_pypdf,
extract_with_pdfminer,
count_pages,
)
from book_to_skill.parsers.epub import (
extract_with_ebooklib,
extract_with_zipfile,
count_epub_chapters,
)
from book_to_skill.sanitize import sanitize_extracted_text
from book_to_skill.zip_safety import safe_read
# CJK codepoints: ideographs + extensions, kana, hangul, CJK punctuation, and
# fullwidth forms. These are not whitespace-delimited, so counting "words" on a
# Chinese/Japanese book collapses it to a handful of tokens; count them directly.
_CJK_RE = re.compile(
r"[　-〿぀-ヿ㐀-䶿一-鿿"
r"가-힣豈-﫿＀-￯]"
)
def estimate_tokens(text: str) -> int:
"""Estimate the token count of ``text`` with a deterministic heuristic.
Latin / whitespace-delimited text is counted by words (``words /
WORDS_PER_TOKEN`` — the project's long-standing ratio). CJK characters are
counted directly against ``CJK_CHARS_PER_TOKEN`` because they carry little
or no whitespace; without this a space-less Chinese/Japanese book estimates
at a few tokens and the cost pre-flight under-reports by ~1000x. Kept
dependency-free on purpose so the same book always yields the same number.
"""
if not text:
return 0
cjk = len(_CJK_RE.findall(text))
if not cjk:
return int(len(text.split()) / WORDS_PER_TOKEN)
latin_words = len(_CJK_RE.sub(" ", text).split())
return int(latin_words / WORDS_PER_TOKEN + cjk / CJK_CHARS_PER_TOKEN)
# Explicit chapter heading: "Chapter 5", "Capítulo 5: ...", "Chapter 1. Intro".
# Also French/German/Italian/Dutch chapter words (chapitre/kapitel/capitolo/
# hoofdstuk), matching the ToC languages added alongside. "ch.?" stays last so
# the longer words match in full. Captures the number (bounded to 1..99 — drops
# years like "2025.") and whatever follows it on the line, so we can reject prose.
_EXPLICIT_CHAPTER = re.compile(
r"^\s*(?:chapter|chapitre|kapitel|cap[ií]tulo|capitolo|hoofdstuk|ch\.?)\s*(?:(\d{1,2})|(?P<roman>[IVXLCDMivxlcdm]{1,7}))\b(?P<rest>.*)$",
re.IGNORECASE,
)
# A heading's number is followed by end-of-line, punctuation (“. : - —“), or a
# Capitalized title word. A lowercase continuation (“Chapter 6 explores...”,
# “Chapter 8 are relevant...”) is prose / a cross-reference, not a heading.
# The uppercase class is À-Þ so titles starting with Ü/Û (common in German, e.g. “Überblick”) are recognized.
_HEADING_TAIL = re.compile(r"^\s*$|^\s*[.:\-—–]|^\s+[A-ZÀ-Þ0-9\"“(]")
# Roman-numeral chapter heading: "I: Loomings", "II. The Carpet-Bag".
# Uppercase alone at line start is safe — no common English word is a valid
# uppercase Roman numeral. Lowercase ("i: Loomings") is only accepted inside
# a markdown heading ("## i. introduction") to avoid false positives from
# words that happen to be valid Roman numerals ("vi: the editor" → 6).
_ROMAN_HEAD = re.compile(r"^\s*([IVXLCDM]+)\s*[:.]\s+[A-ZÀ-Þ0-9\"“(]")
_LC_MD_ROMAN = re.compile(r"^\s*#{1,6}\s+([ivxlcdm]+)\s*[:.]\s+[A-Za-zÀ-Þ\"“(]")
_ROMAN_VALUES = {"I": 1, "V": 5, "X": 10, "L": 50, "C": 100, "D": 500, "M": 1000}
# Chinese chapter headings. Two common styles:
# 1. explicit "第N章" / "第 3 回" / "第十二节" / "第一讲" — 第 + numeral + a
# chapter classifier (章回卷节篇讲);
# 2. a Markdown heading led by a CJK ordinal and a separator, e.g.
# "## 一 · 缘起" or "## 第一讲" — common in CJK ebooks and lecture notes.
# Scoped to CJK numerals, so Latin/Roman detection above is completely unaffected
# (e.g. "## 5 Setup" is still not treated as a heading here). detect_structure()
# dedupes by number, so a "##" heading and a repeated "###" sub-ordinal collapse
# to a single chapter.
_CN_NUM_VALUES = {
"〇": 0, "零": 0, "一": 1, "二": 2, "两": 2, "三": 3, "四": 4, "五": 5,
"六": 6, "七": 7, "八": 8, "九": 9,
}
_CN_NUM_UNITS = {"十": 10, "百": 100, "千": 1000}
_CN_NUM_CLASS = "〇零一二两三四五六七八九十百千"
# Full-width Arabic digits (U+FF10–U+FF19) are common in Japanese typesetting,
# e.g. "第１章". int() already parses them (str.isdigit() is True), so only the
# regex character classes need to accept them.
_FW_DIGITS = "０-９"
_CN_CHAPTER = re.compile(rf"^\s*第\s*([0-9{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[章回卷节篇讲]")
_MD_CN_HEADING = re.compile(rf"^#{{1,6}}\s+第?\s*([{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[·、.:：章回卷节篇讲]")
# Thai chapter headings: "บทที่ 3", "บทที่ ๑๒", "ตอนที่ ๘๗", "ภาคที่ 2".
# Thai digits (U+0E50-U+0E59) are positional like Arabic — unlike the Chinese
# numerals above they need no unit composition, only a digit remap. Optional
# Markdown "#" prefix so "## บทที่ ๑" is recognized in converted ebooks.
_TH_DIGITS = "๐-๙"
_TH_DIGIT_MAP = str.maketrans("๐๑๒๓๔๕๖๗๘๙", "0123456789")
_TH_CHAPTER = re.compile(
rf"^\s*(?:#{{1,6}}\s+)?(?:บทที่|ตอนที่|ภาคที่|บท|ตอน|ภาค)\s*([0-9{_TH_DIGITS}]+)\b"
)
# Korean chapter headings: "제1장 총칙", "## 제4장 근로시간과 휴식", "제6장의2 …".
# 제 + Arabic numeral + a classifier (장 chapter / 편 part / 절 section / 관
# subsection), with an optional "의N" branch suffix that Korean statutes use for
# inserted chapters (제6장의2). Modern Korean numbers chapters with Arabic digits,
# so unlike the Chinese branch no numeral composition is needed. Optional Markdown
# "#" prefix so "## 제1장" is recognized in converted ebooks.
#
# The trailing group is the Korean analogue of _HEADING_TAIL: Korean has no letter
# case, so the existing "capitalized title word" test does not transfer.
# Requiring end-of-line, punctuation, or whitespace-then-content is what separates
# a heading from a prose cross-reference, because Korean particles attach directly
# to the noun ("제5장에서", "제2장의") with no intervening space.
_KO_CHAPTER = re.compile(
r"^\s*(?:#{1,6}\s+)?제\s*([0-9]+)\s*[장편절관](?:\s*의\s*[0-9]+)?(?:\s*$|[.:\-]|\s+\S)"
)
# Table-of-contents header lines across common languages. Anchored to a whole
# line (^\s*X\s*$) so an inline "the contents of this chapter" never matches.
_TOC_HEADERS = (
"table of contents", "contents", "índice", "sumário", # EN / ES / PT
"目录", "目錄", "目次", # Chinese / Japanese
"table des matières", # French
"inhaltsverzeichnis", # German
"indice", "sommario", # Italian (no accent — distinct from índice above)
"inhoudsopgave", # Dutch
)
_TOC_PATTERN = re.compile(
r"^\s*(?:" + "|".join(re.escape(h) for h in _TOC_HEADERS) + r")\s*$",
re.IGNORECASE | re.MULTILINE,
)
# ATX-style heading: "# Title", "## Section", AsciiDoc "= Title", "== Section".
# The required space after the marker distinguishes an AsciiDoc "== X" from a
# reStructuredText underline "=====" (no space) — the latter is intentionally
# ignored (RST underline headings are out of scope).
_ATX_HEADING = re.compile(r"^(#{1,6}|={1,6})\s+(.+?)\s*#*$")
# Setext/RST underline: a full line of "=" (level 1) or "-" (level 2), length
# >= 2. Marks the line directly above it as a heading title.
_SETEXT_UNDERLINE = re.compile(r"^(={2,}|-{2,})$")
def _structural_chapter_count(text: str) -> int:
"""Count chapter-like structural headings in Markdown/AsciiDoc/RST sources.
Recognizes ATX headings ("# Title", "== Section") and setext/RST underline
headings (a title line directly above a row of "=" or "-"). Groups distinct
(case-normalized) titles by depth and returns the count at the shallowest
depth with >= 2 distinct titles — this selects the real chapter level in the
common "# Book Title / ## Chapter" layout where the top level appears once.
Guards against false positives: headings inside fenced code blocks are
skipped; an ATX title starting with a bare digit ("## 5 Setup") or made only
of punctuation ("=====" table borders) is rejected; a setext underline counts
only when it sits directly under a non-blank title line at least as long as
the underline (so thematic breaks, table borders, and front-matter "---" do
not match).
"""
levels: dict[int, set[str]] = {}
in_fence = False
prev = "" # previous non-fence line (stripped); a setext title candidate
for line in text.splitlines():
s = line.strip()
if s.startswith("```") or s.startswith("~~~"):
in_fence = not in_fence
prev = ""
continue
if in_fence:
prev = ""
continue
# Setext/RST underline: "=" (level 1) or "-" (level 2) directly under a
# title line at least as long as the underline.
if (
_SETEXT_UNDERLINE.match(s)
and prev
and not _SETEXT_UNDERLINE.match(prev)
and len(s) >= len(prev)
):
depth = 1 if s[0] == "=" else 2
levels.setdefault(depth, set()).add(prev.lower())
prev = ""
continue
# ATX heading ("# Title", "== Section").
m = _ATX_HEADING.match(s)
if m:
title = m.group(2).strip().lower()
# Reject empty, bare-digit-led ("## 5 Setup"), and all-punctuation
# ("=====" table-border) titles — none are real chapter headings.
if title and not title[0].isdigit() and re.search(r"\w", title):
levels.setdefault(len(m.group(1)), set()).add(title)
# An ATX heading line is not a setext title for the next line.
prev = ""
continue
prev = s
if not levels:
return 0
for depth in sorted(levels):
if len(levels[depth]) >= 2:
return len(levels[depth])
# No level has >= 2 distinct headings: a thin doc (e.g. one heading per
# level). Count them all — this path runs only as a fallback when numeric
# chapter detection already found zero, so it cannot inflate real books.
return sum(len(titles) for titles in levels.values())
def _cn_numeral_to_int(s: str) -> int | None:
"""Parse a Chinese (or ASCII-digit) chapter numeral into an int (1..999)."""
if s.isdigit():
n = int(s)
return n if 1 <= n <= 999 else None
section = current = 0
for ch in s:
if ch in _CN_NUM_VALUES:
current = _CN_NUM_VALUES[ch]
elif ch in _CN_NUM_UNITS:
section += (current or 1) * _CN_NUM_UNITS[ch]
current = 0
else:
return None
total = section + current
return total if 1 <= total <= 999 else None
def _int_to_roman(n: int) -> str:
table = [(1000, "M"), (900, "CM"), (500, "D"), (400, "CD"), (100, "C"),
(90, "XC"), (50, "L"), (40, "XL"), (10, "X"), (9, "IX"),
(5, "V"), (4, "IV"), (1, "I")]
out = []
for val, sym in table:
while n >= val:
out.append(sym)
n -= val
return "".join(out)
def _roman_to_int(s: str) -> int | None:
"""Convert a Roman numeral to int, returning None if it isn't canonical."""
s = s.upper()
total = prev = 0
for ch in reversed(s):
v = _ROMAN_VALUES.get(ch)
if v is None:
return None
total += -v if v < prev else v
prev = max(prev, v)
if total == 0 or total > 200:
return None
# Reject non-canonical forms ("IIII", "VV") by round-tripping.
return total if _int_to_roman(total) == s else None
def _chapter_number(line: str) -> int | None:
"""Return the chapter number if the line is a genuine chapter heading.
Handles Arabic ("Chapter 5", "Capítulo 5: ..."), Roman-numeral
("I: Loomings", "## i. introduction", "II. The Carpet-Bag"),
Chinese ("第三章 …", "## 一 · …", "## 第一讲"), Thai ("บทที่ 3",
"## บทที่ ๑"), and Korean ("제1장 총칙", "## 제4장 근로시간과 휴식")
heading styles.
"""
s = line.strip()
if len(s) > 80:
return None
m = _EXPLICIT_CHAPTER.match(s)
if m and _HEADING_TAIL.match(m.group("rest")):
if m.group(1):
return int(m.group(1))
return _roman_to_int(m.group("roman").upper())
rm = _ROMAN_HEAD.match(s) or _LC_MD_ROMAN.match(s)
if rm:
return _roman_to_int(rm.group(1))
cm = _CN_CHAPTER.match(s) or _MD_CN_HEADING.match(s)
if cm:
return _cn_numeral_to_int(cm.group(1))
tm = _TH_CHAPTER.match(s)
if tm:
return int(tm.group(1).translate(_TH_DIGIT_MAP))
km = _KO_CHAPTER.match(s)
if km:
return int(km.group(1))
return None
def detect_structure(text: str) -> dict:
"""Detect chapter count and table of contents presence.
Scans the whole text (not just the head) and counts DISTINCT chapter numbers
from explicit "Chapter N"/"Capítulo N" headings, rejecting prose
cross-references and numbered list items. Counting distinct numbers means a
ToC entry and its body heading are not double-counted.
"""
lines = text.splitlines()
headings = []
numbers = set()
for line in lines:
num = _chapter_number(line)
if num is not None:
numbers.add(num)
headings.append(line.strip())
numeric_count = len(numbers)
# Fall back to structural (Markdown/AsciiDoc) headings only when no numeric
# "Chapter N" headings were found, so books with real chapters are unaffected.
chapters_detected = (
numeric_count if numeric_count > 0 else _structural_chapter_count(text)
)
# Look for ToC indicators in the first ~30k chars (multilingual; see _TOC_PATTERN)
has_toc = bool(_TOC_PATTERN.search(text[:30000]))
return {
"chapters_detected": chapters_detected,
"chapter_headings_sample": headings[:10],
"has_toc": has_toc,
}
def resolve_input_files(paths: list[str]) -> list[Path]:
"""Resolve paths including files, directories, and glob patterns to Path objects.
User-given order is preserved for explicit file arguments. Expanded
results (directories, globs) are sorted deterministically so repeated
runs produce the same output.
"""
resolved = []
for path_str in paths:
# Check if it has glob wildcards
if any(char in path_str for char in ("*", "?", "[")):
glob_matches = glob.glob(path_str, recursive=True)
# Sort expanded glob results deterministically
expanded = []
for match in glob_matches:
p = Path(match)
if p.is_file() and p.suffix.lower() in SUPPORTED_EXTENSIONS:
expanded.append(p.resolve())
expanded.sort(key=lambda x: str(x).lower())
resolved.extend(expanded)
else:
p = Path(path_str)
if p.is_dir():
# Sort expanded directory results deterministically
dir_files = []
for root, _, files in os.walk(p):
for file in files:
file_path = Path(root) / file
if file_path.suffix.lower() in SUPPORTED_EXTENSIONS:
dir_files.append(file_path.resolve())
dir_files.sort(key=lambda x: str(x).lower())
resolved.extend(dir_files)
else:
# Keep even if it doesn't exist so the error check can report it
resolved.append(p.resolve())
# Deduplicate while preserving insertion order (user order for explicit files)
seen = set()
unique_paths = []
for path in resolved:
resolved_path = path.resolve() if path.exists() else path
if resolved_path not in seen:
seen.add(resolved_path)
unique_paths.append(resolved_path)
return unique_paths
def extract_single_file(input_path: Path, extraction_mode: str, install_mode: str) -> dict:
"""Extract text and metadata from a single file path."""
input_str = str(input_path)
if not input_path.exists():
raise ExtractionError(f"File not found: {input_str}")
ext = input_path.suffix.lower()
document_format = ext.lstrip(".")
# Sniff magic bytes if suffix is not supported
if ext not in SUPPORTED_EXTENSIONS:
with open(input_str, "rb") as f:
header = f.read(8)
if header[:4] == b"%PDF":
ext = ".pdf"
document_format = "pdf"
elif header[:2] == b"PK":
try:
with zipfile.ZipFile(input_str) as zf:
names = set(zf.namelist())
# safe_read, not zf.read: this runs during *sniffing*, before a
# format is even chosen, so it is the earliest point an attacker
# controls. A single-member zip declaring a huge uncompressed
# `mimetype` would otherwise be fully decompressed here — ahead of
# every size check in zip_safety.py. Its ExtractionError is not in
# the except tuple below, so a bomb reports as a bomb rather than
# as a generic unsupported format.
if "mimetype" in names and safe_read(zf, "mimetype").startswith(b"application/epub"):
ext = ".epub"
document_format = "epub"
elif "word/document.xml" in names:
ext = ".docx"
document_format = "docx"
else:
raise ExtractionError(
f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
)
except (zipfile.BadZipFile, KeyError, OSError):
raise ExtractionError(
f"Unsupported ZIP-based format '{input_path.name}'. Supported: {supported_formats_message()}"
)
else:
raise ExtractionError(
f"Unsupported format '{ext or '<none>'}'. Supported: {supported_formats_message()}"
)
prepare_dependencies(ext, extraction_mode, install_mode)
if ext in CALIBRE_EBOOK_EXTENSIONS and not shutil.which("ebook-convert"):
raise ExtractionError(
"MOBI/AZW/AZW3 extraction requires Calibre's ebook-convert command. "
"Install Calibre and ensure ebook-convert is on PATH, then rerun this command."
)
text = ""
method = ""
pages = 0
pages_label = "sections"
if ext == ".epub":
print(f"Extracting EPUB: {input_str}")
text = extract_with_ebooklib(input_str)
if text and text.strip():
method = "ebooklib"
else:
print("ebooklib not available")
print("Trying stdlib zipfile parser...", end=" ", flush=True)
text = extract_with_zipfile(input_str)
if text and text.strip():
print("OK")
method = "zipfile"
else:
print("FAILED")
raise ExtractionError(
"Could not extract text from EPUB.\n"
"Install ebooklib + beautifulsoup4 for best results:\n"
" pip3 install ebooklib beautifulsoup4"
)
pages = count_epub_chapters(input_str)
pages_label = "spine_items"
elif ext == ".pdf":
print(f"Extracting PDF: {input_str}")
if extraction_mode == "technical":
print("Mode: technical — using Docling (layout-aware)...", end=" ", flush=True)
text = extract_with_docling(input_str)
if text:
method = "docling"
print("OK")
else:
print("not available, falling back to pdftotext")
extraction_mode = "text"
if extraction_mode == "text" or not text:
print("Mode: text — using pdftotext...")
print("Trying pdftotext...", end=" ", flush=True)
text = extract_with_pdftotext(input_str)
if text:
method = "pdftotext"
print("OK")
else:
print("not available")
print("Trying pypdf...", end=" ", flush=True)
text = extract_with_pypdf(input_str)
if text:
method = "pypdf"
print("OK")
else:
print("not available")
print("Trying pdfminer.six...", end=" ", flush=True)
text = extract_with_pdfminer(input_str)
if text:
method = "pdfminer"
print("OK")
else:
print("FAILED")
raise ExtractionError(
"Could not extract text from PDF.\n"
"Install one of: poppler-utils (pdftotext), pypdf, or pdfminer.six\n"
" poppler-utils: use your system package manager\n"
" pip3 install pypdf\n"
" pip3 install pdfminer.six"
)
pages = count_pages(input_str)
pages_label = "pages"
elif ext in TEXT_EXTENSIONS:
print(f"Extracting text document: {input_str}")
text = read_text_file(input_str)
if text is None or not text.strip():
raise ExtractionError(f"Could not read text document: {input_path.name}")
method = "plain-text"
pages = 0
pages_label = "sections"
elif ext in HTML_EXTENSIONS:
print(f"Extracting HTML: {input_str}")
text = extract_html_file(input_str)
if text is None or not text.strip():
raise ExtractionError(f"Could not extract text from HTML: {input_path.name}")
method = "html-parser"
pages = 0
pages_label = "sections"
elif ext == ".docx":
print(f"Extracting DOCX: {input_str}")
text, method = extract_docx(input_str)
pages = 0
pages_label = "sections"
elif ext == ".rtf":
print(f"Extracting RTF: {input_str}")
text, method = extract_rtf(input_str)
pages = 0
pages_label = "sections"
elif ext in CALIBRE_EBOOK_EXTENSIONS:
print(f"Extracting ebook with Calibre: {input_str}")
text = extract_with_ebook_convert(input_str)
if text is None or not text.strip():
raise ExtractionError(
f"Could not extract text from {ext}. Install Calibre and ensure ebook-convert is on PATH."
)
method = "ebook-convert"
pages = 0
pages_label = "sections"
text, removed_invisible = sanitize_extracted_text(text)
if removed_invisible:
print(
f" [security] removed {removed_invisible} invisible Unicode "
f"code point(s) from {input_path.name}",
file=sys.stderr,
)
if not text.strip():
raise ExtractionError(
f"Extracted text from {input_path.name} contained no visible content "
"after Unicode sanitization."
)
tokens = estimate_tokens(text)
structure = detect_structure(text)
file_size_mb = os.path.getsize(input_str) / (1024 * 1024)
return {
"source_file": str(input_path.resolve()),
"filename": input_path.name,
"format": document_format,
"extraction_method": method,
"file_size_mb": round(file_size_mb, 2),
# `pages` is the canonical count; `pages_label` names the unit ("pages"
# for PDF, "spine_items" for EPUB, "sections" otherwise). Upstream also
# emitted a dynamic `{pages_label: pages}` key, which silently collided
# with "pages" whenever the label WAS "pages"; the alias is added below
# only when it is actually a different key.
"pages": pages,
"pages_label": pages_label,
"chars": len(text),
"words": len(text.split()),
"estimated_tokens": tokens,
"text": text,
**({pages_label: pages} if pages_label != "pages" else {}),
**structure,
}

130
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/zip_safety.py Normal file
View file

@ -0,0 +1,130 @@
"""Shared safety checks for zip-of-XML container formats (DOCX, EPUB).
Upstream hardened DOCX and only DOCX: `validate_docx_xml_safety()` scanned that
archive for DTD/entity declarations before any parser touched it. EPUB is the
same shape — a zip whose members are XML (`container.xml`, the OPF package
document, NCX, content docs) — and its `ebooklib` path handed the file straight
to an XML parser with no equivalent pre-check, even though `ebooklib` is one of
the packages this skill recommends installing "for best results".
Two classes of attack, both from a file the user was handed rather than wrote:
1. **Entity expansion / XXE.** `<!ENTITY>` declarations drive billion-laughs
memory exhaustion; external entities read local files into the parsed output.
Python's `xml.etree.ElementTree` does not resolve *external* entities, but it
does expand internal ones — and third-party parsers built on lxml vary. The
check refuses the archive rather than trusting each parser's defaults.
2. **Decompression bombs.** A few KB of zip can expand to gigabytes. Every read
goes through `safe_read()`, which consults the declared uncompressed size and
the compression ratio *before* decompressing, so a bomb is refused rather
than materialized.
Neither parser ever writes archive members to disk (no `extractall`/`extract`),
so zip-slip is out of scope by construction.
"""
from __future__ import annotations
import zipfile
from book_to_skill.exceptions import ExtractionError
# A single member of a book archive. Generous for a chapter's XHTML or a
# document.xml, far below what a bomb needs to hurt.
MAX_MEMBER_BYTES = 64 * 1024 * 1024
# Everything read from one archive, across all members.
MAX_TOTAL_BYTES = 512 * 1024 * 1024
# Declared-uncompressed / stored ratio. Real prose and XML land well under 100x;
# the classic zip bomb is ~1000x and up.
MAX_COMPRESSION_RATIO = 200
# Members worth scanning for DTD/entity declarations: XML by extension, plus the
# extensionless `mimetype` member and OPF/NCX which are XML without an .xml suffix.
_XML_SUFFIXES = (".xml", ".rels", ".opf", ".ncx", ".xhtml", ".html", ".htm")
class _Budget:
"""Running total of bytes decompressed from one archive."""
def __init__(self, limit: int = MAX_TOTAL_BYTES) -> None:
self.limit = limit
self.used = 0
def charge(self, size: int, name: str) -> None:
self.used += size
if self.used > self.limit:
raise ExtractionError(
f"archive exceeds the {self.limit:,}-byte decompression budget "
f"(reading '{name}') — refusing to continue"
)
def safe_read(zf: zipfile.ZipFile, name: str, budget: _Budget | None = None) -> bytes:
"""Read one archive member after checking its declared size and ratio.
The checks run against the zip's central directory *before* decompressing,
so a bomb never gets materialized. A liar in the directory still cannot get
past the budget, because the actual read is charged against it too.
"""
try:
info = zf.getinfo(name)
except KeyError as exc:
raise ExtractionError(f"archive member not found: {name}") from exc
if info.file_size > MAX_MEMBER_BYTES:
raise ExtractionError(
f"archive member '{name}' declares {info.file_size:,} bytes uncompressed; "
f"the per-member limit is {MAX_MEMBER_BYTES:,} bytes"
)
if info.compress_size > 0:
ratio = info.file_size / info.compress_size
if ratio > MAX_COMPRESSION_RATIO:
raise ExtractionError(
f"archive member '{name}' expands {ratio:.0f}x "
f"({info.compress_size:,} -> {info.file_size:,} bytes); the limit is "
f"{MAX_COMPRESSION_RATIO}x — this looks like a decompression bomb"
)
data = zf.read(name)
if budget is not None:
budget.charge(len(data), name)
return data
def is_xml_member(name: str) -> bool:
return name.lower().endswith(_XML_SUFFIXES)
def validate_zip_xml_safety(archive_path: str, label: str = "archive") -> None:
"""Refuse a zip-of-XML archive that declares a DTD or any entity.
Scans every XML-ish member across the encodings a hostile file might use to
hide the declaration from a naive UTF-8 substring search. Generalized from
upstream's DOCX-only guard so EPUB gets the same treatment.
"""
try:
with zipfile.ZipFile(archive_path) as zf:
budget = _Budget()
for name in zf.namelist():
if not is_xml_member(name):
continue
xml_bytes = safe_read(zf, name, budget)
for encoding in ("utf-8", "utf-16", "utf-16le", "utf-16be", "utf-32"):
try:
content = xml_bytes.decode(encoding, errors="ignore").upper()
except LookupError:
continue
if "<!DOCTYPE" in content or "<!ENTITY" in content:
raise ExtractionError(
f"security validation failed: XML member '{name}' in the {label} "
f"archive contains a forbidden DTD or entity declaration"
)
except zipfile.BadZipFile as exc:
raise ExtractionError(f"invalid {label} file: {exc}") from exc
except ExtractionError:
raise
except Exception as exc:
raise ExtractionError(
f"error during security validation of the {label} archive: {exc}"
) from exc

403
engineering/book-to-skill/skills/book-to-skill/scripts/extract_document.py Normal file
View file

@ -0,0 +1,403 @@
#!/usr/bin/env python3
"""extract_document.py — deterministic document -> clean text + metadata.
Stage 1 of the book-to-skill pipeline. Takes one or more files, folders, or glob
patterns, extracts their text, sanitizes invisible Unicode (document-borne
prompt injection), detects chapter structure, and writes two artifacts the
agent then reasons over:
<workdir>/full_text.txt combined text with per-source banners
<workdir>/metadata.json sizes, token estimate, chapter/ToC detection
The workdir is a fresh private temp directory per invocation (0700, owner-only
artifacts) whose path is printed on completion and carried in metadata.json's
``output_text``. Pin it with ``--workdir`` or ``BOOK_SKILL_WORKDIR`` when you
want a stable location; an explicit path is symlink-checked and mode-restricted
before anything is written to it.
Runs on the standard library alone. Optional packages (docling, pypdf,
pdfminer.six, ebooklib, beautifulsoup4, python-docx, striprtf) raise extraction
quality where installed; ``--check`` reports what is present and prints the
exact install command for what is not. MOBI/AZW/AZW3 are the one format with no
stdlib fallback — they need Calibre's ``ebook-convert`` on PATH.
Nothing is installed unless you explicitly pass ``--install-missing yes``.
Exit codes:
0 extraction succeeded (or --check / --sample completed)
1 no usable input, or every source failed extraction
2 bad invocation
Adapted from virgiliojr94/book-to-skill (MIT). See ../../../LICENSE.
Usage:
python3 extract_document.py BOOK.pdf --mode technical
python3 extract_document.py ./docs/ '*.epub' --output json
python3 extract_document.py --check
python3 extract_document.py --sample
"""
from __future__ import annotations
import argparse
import contextlib
import json
import os
import stat
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from book_to_skill.config import ( # noqa: E402
ARTIFACT_MODE,
OUTPUT_META_NAME,
OUTPUT_TEXT_NAME,
WORKDIR_MODE,
make_private_tempdir,
supported_formats_message,
workdir_from_env,
)
from book_to_skill.dependencies import ( # noqa: E402
INSTALL_MODES,
normalize_install_mode,
run_dependency_check,
)
from book_to_skill.exceptions import ExtractionError # noqa: E402
from book_to_skill.utils import ( # noqa: E402
detect_structure,
estimate_tokens,
extract_single_file,
resolve_input_files,
)
SAMPLE_DOCUMENT = """Table of Contents
Chapter 1: The Discovery Loop Tax
Chapter 2: Extract Structure, Not Summaries
Chapter 1: The Discovery Loop Tax
An agent that reads a PDF to answer one question pays for the table of
contents, the chapter it guessed at, and the chapter it backtracks to for a
missing definition. Structuring the source once makes every later query cost
what the answer costs, not what the book costs.
Chapter 2: Extract Structure, Not Summaries
A skill is not a book report. Capture named frameworks with their exact
formulations, the decision rules the author commits to, and the anti-patterns
they warn against. "The 5 Whys" is not interchangeable with "ask why a few
times" — the name is the interface.
"""
class WorkdirError(RuntimeError):
"""The requested working directory cannot be used safely."""
def resolve_workdir(explicit: str | None) -> Path:
"""Return a private working directory for this invocation.
With no explicit path, creates a fresh `mkdtemp` — unpredictable name, 0700
by construction, and never shared with a concurrent run. A fixed default
(upstream's `<tempdir>/book_skill_work`) is the CWE-377/CWE-59 shape on a
shared host: another local user pre-creates the directory and plants a
symlink named full_text.txt, and `Path.write_text` follows it.
An explicitly requested path is honoured but hardened: refused if it is a
symlink, created 0700 if new, and tightened to 0700 if it already exists.
"""
if not explicit:
return make_private_tempdir()
workdir = Path(explicit).expanduser()
if workdir.is_symlink():
raise WorkdirError(f"{workdir} is a symbolic link — refusing to write extraction "
f"artifacts through it")
if workdir.exists() and not workdir.is_dir():
raise WorkdirError(f"{workdir} exists and is not a directory")
# Create first, then inspect with lstat. `mkdir(exist_ok=True)` alone is not a
# check: its exists-branch tests is_dir(), which FOLLOWS symlinks, so a
# symlink-to-directory satisfies it silently. Attempting creation and only
# examining the path when it already existed removes that ordering problem.
try:
workdir.mkdir(parents=True, mode=WORKDIR_MODE)
except FileExistsError:
info = os.lstat(workdir) # lstat: does not follow the final component
if stat.S_ISLNK(info.st_mode):
raise WorkdirError(f"{workdir} is a symbolic link — refusing to write "
f"extraction artifacts through it") from None
if not stat.S_ISDIR(info.st_mode):
raise WorkdirError(f"{workdir} exists and is not a directory") from None
try:
# mkdir's mode applies only on creation and is masked by umask; an
# existing directory keeps whatever permissions it already had.
workdir.chmod(WORKDIR_MODE)
except OSError:
# Not ours to chmod (e.g. a shared mount). The lstat check above and
# the fd pinning below still hold.
pass
return workdir
def open_workdir(workdir: Path) -> int | None:
"""Pin the working directory by file descriptor, or return None if unsupported.
Everything above is still check-then-act: an attacker who can write the parent
directory could swap the directory for a symlink after the lstat and before a
write, and the per-file symlink check cannot see that — the file inside a
swapped directory is a perfectly ordinary file.
Opening the directory once with O_NOFOLLOW|O_DIRECTORY and writing through
that descriptor closes the race: the fd names an inode, so a later rename or
symlink swap of the path cannot redirect writes that go through it.
Returns None on platforms without these flags or without `dir_fd` support
(Windows), where the caller falls back to path-based writes.
"""
flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0)
if not (flags & getattr(os, "O_NOFOLLOW", 0)) or os.open not in os.supports_dir_fd:
return None
try:
return os.open(workdir, flags)
except OSError as exc:
raise WorkdirError(f"could not open {workdir} as a real directory: {exc}") from exc
def _write_private(path: Path, text: str, dir_fd: int | None = None) -> None:
"""Write a file only this user can read, never through a symlink.
O_CREAT|O_EXCL|O_NOFOLLOW is the atomic primitive: it refuses outright if
anything already exists at the name — including a symlink — so there is no
window between checking and writing. An artifact left by a previous run into
the same --workdir is unlinked first; unlink removes the link itself, never
its target.
"""
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0)
name = path.name if dir_fd is not None else str(path)
opener_kwargs = {"dir_fd": dir_fd} if dir_fd is not None else {}
if dir_fd is None and path.is_symlink():
# Fallback path (no dir_fd support): best-effort pre-check.
raise WorkdirError(f"{path} is a symbolic link — refusing to write through it")
try:
fd = os.open(name, flags, ARTIFACT_MODE, **opener_kwargs)
except FileExistsError:
os.unlink(name, **opener_kwargs)
fd = os.open(name, flags, ARTIFACT_MODE, **opener_kwargs)
with os.fdopen(fd, "w", encoding="utf-8") as handle:
handle.write(text)
def _extract_all(input_files, extraction_mode: str, install_mode: str):
"""Extract every resolved file. Returns (sources, errors)."""
sources, errors = [], []
for file_path in input_files:
try:
sources.append(extract_single_file(file_path, extraction_mode, install_mode))
except ExtractionError as exc:
print(f"WARNING: skipping {file_path.name}: {exc}", file=sys.stderr)
errors.append((file_path, str(exc)))
return sources, errors
def _build_metadata(sources: list[dict], consolidated_text: str, extraction_mode: str,
text_path: Path) -> dict:
total_tokens = estimate_tokens(consolidated_text)
# Structure is detected on source text only: the per-source banners in
# full_text.txt are rows of "=", which would otherwise register as phantom
# setext headings and make the result depend on how long the paths are.
structure = detect_structure("\n\n".join(src["text"] for src in sources))
multi = len(sources) > 1
return {
"source_file": "Consolidated from multiple sources" if multi else sources[0]["source_file"],
"filename": "multi-source" if multi else sources[0]["filename"],
"format": "mixed" if multi else sources[0]["format"],
"extraction_method": "multi-method" if multi else sources[0]["extraction_method"],
"extraction_mode": extraction_mode,
"file_size_mb": round(sum(src["file_size_mb"] for src in sources), 2),
"pages": sum(src["pages"] for src in sources),
"chars": len(consolidated_text),
"words": len(consolidated_text.split()),
"estimated_tokens": total_tokens,
"estimated_tokens_human": f"~{total_tokens // 1000}K",
"output_text": str(text_path),
"total_sources": len(sources),
"sources": [
{key: src[key] for key in (
"source_file", "filename", "format", "extraction_method",
"file_size_mb", "pages", "pages_label", "chars", "words",
"estimated_tokens", "chapters_detected", "has_toc",
)}
for src in sources
],
**structure,
}
def _print_report(metadata: dict, errors: list, text_path: Path, meta_path: Path) -> None:
print("\nExtraction complete:")
print(f" Sources : {metadata['total_sources']} processed")
print(f" Size : {metadata['file_size_mb']:.2f} MB")
print(f" Pages : {metadata['pages']}")
print(f" Words : {metadata['words']:,}")
print(f" Tokens : {metadata['estimated_tokens_human']}")
print(f" Chapters: {metadata['chapters_detected']} detected overall")
print(f" ToC : {'yes' if metadata['has_toc'] else 'not detected'}")
if not metadata["has_toc"]:
print(" WARN : no table of contents detected — chapter mapping falls back to a "
"heading scan, which may miss or duplicate sections.")
print(f"\n Text -> {text_path}")
print(f" Meta -> {meta_path}")
if errors:
print(f"\n WARNING: {len(errors)} source(s) skipped:")
for path, err in errors:
print(f" - {path.name}: {err}")
def run_sample(as_json: bool) -> int:
"""Extract a built-in two-chapter document so the pipeline can be smoke-tested
without supplying a real book."""
import tempfile
with tempfile.TemporaryDirectory() as tmp:
sample = Path(tmp) / "sample-book.md"
sample.write_text(SAMPLE_DOCUMENT, encoding="utf-8")
return run_extraction(
input_paths=[str(sample)],
extraction_mode="text",
install_mode="report",
workdir=resolve_workdir(str(Path(tmp) / "work")),
as_json=as_json,
keep=False,
)
def run_extraction(*, input_paths: list[str], extraction_mode: str, install_mode: str,
workdir: Path, as_json: bool, keep: bool = True) -> int:
input_files = resolve_input_files(input_paths)
if not input_files:
print(f"ERROR: no supported files found matching: {', '.join(input_paths)}", file=sys.stderr)
print(f"Supported formats: {supported_formats_message()}", file=sys.stderr)
return 1
workdir.mkdir(parents=True, mode=WORKDIR_MODE, exist_ok=True)
text_path = workdir / OUTPUT_TEXT_NAME
meta_path = workdir / OUTPUT_META_NAME
# The vendored extractors narrate progress on stdout. When the caller asked
# for JSON, that narration would corrupt the payload — send it to stderr so
# stdout carries the document metadata and nothing else.
sink = contextlib.redirect_stdout(sys.stderr) if as_json else contextlib.nullcontext()
with sink:
sources, errors = _extract_all(input_files, extraction_mode, install_mode)
if not sources:
print(f"ERROR: all {len(errors)} source(s) failed extraction:", file=sys.stderr)
for path, err in errors:
print(f" - {path.name}: {err}", file=sys.stderr)
return 1
consolidated_text = "".join(
f"\n\n{'=' * 80}\nSOURCE: {src['filename']} (Path: {src['source_file']})\n{'=' * 80}\n\n"
+ src["text"]
for src in sources
).strip()
metadata = _build_metadata(sources, consolidated_text, extraction_mode, text_path)
# Both artifacts are written through one pinned directory descriptor, so a
# swap of the workdir path between the two writes cannot redirect either.
# The payload is dumped with ensure_ascii=False, so a non-ASCII chapter
# heading or path reaches the encoder verbatim — _write_private opens with
# encoding="utf-8" for exactly that reason (a cp1252 host or LC_ALL=C would
# otherwise raise after every source had already been extracted).
workdir_fd = open_workdir(workdir)
try:
_write_private(text_path, consolidated_text, workdir_fd)
_write_private(meta_path, json.dumps(metadata, indent=2, ensure_ascii=False),
workdir_fd)
finally:
if workdir_fd is not None:
os.close(workdir_fd)
if as_json:
print(json.dumps(metadata, indent=2, ensure_ascii=False))
else:
_print_report(metadata, errors, text_path, meta_path)
if not keep:
print("\n(sample run — the temporary workdir was discarded)")
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="extract_document.py",
description="Extract clean text + metadata from books and documents.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=f"Supported formats: {supported_formats_message()}",
)
parser.add_argument("paths", nargs="*",
help="file(s), folder(s), or glob pattern(s) to extract")
parser.add_argument("--mode", choices=("text", "technical"), default="text",
help="technical preserves tables/code via docling when installed; "
"text uses the fastest suitable extractor (default: text)")
parser.add_argument("--workdir", help="where to write full_text.txt + metadata.json "
"(default: a fresh private temp directory; the path "
"is printed and stored in metadata.json)")
parser.add_argument("--install-missing", choices=INSTALL_MODES, default=None,
help="report prints the pip command and uses the stdlib fallback "
"(default); ask prompts on a TTY; yes installs without asking")
parser.add_argument("--output", choices=("text", "json"), default="text",
help="text prints a human report; json prints the metadata document")
parser.add_argument("--check", action="store_true",
help="report which extractors are installed, then exit")
parser.add_argument("--sample", action="store_true",
help="run the pipeline on a built-in sample document, then exit")
args = parser.parse_args(argv)
for stream in (sys.stdout, sys.stderr):
with contextlib.suppress(AttributeError, ValueError):
stream.reconfigure(encoding="utf-8")
if args.check:
return run_dependency_check()
if args.sample:
return run_sample(as_json=args.output == "json")
if not args.paths:
parser.print_usage(sys.stderr)
print("ERROR: no input document, folder, or glob pattern given "
"(use --sample to try the pipeline).", file=sys.stderr)
return 2
try:
workdir = resolve_workdir(args.workdir or workdir_from_env())
except WorkdirError as exc:
print(f"REFUSED: {exc}", file=sys.stderr)
return 2
try:
return run_extraction(
input_paths=args.paths,
extraction_mode=args.mode,
install_mode=normalize_install_mode(args.install_missing),
workdir=workdir,
as_json=args.output == "json",
)
except WorkdirError as exc:
print(f"REFUSED: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
try:
sys.exit(main())
except BrokenPipeError:
# `tool | head` closes the pipe while this is still writing. Exit quietly
# instead of dumping a traceback: redirect stdout to devnull first so the
# interpreter's shutdown flush cannot re-raise. 141 = 128 + SIGPIPE.
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
sys.exit(141)

590
engineering/book-to-skill/skills/book-to-skill/scripts/skill_plugin_emitter.py Normal file
View file

@ -0,0 +1,590 @@
#!/usr/bin/env python3
"""skill_plugin_emitter.py — wrap a generated book skill in a claude-skills plugin.
The upstream converter stops at a bare skill folder in a personal skills home
(`~/.claude/skills/<slug>/`). That folder is invisible to this repository: it
has no plugin manifest, no `cs-*` agent, no `/cs:*` command, and no marketplace
entry, so nothing else in the library can route to it.
This tool closes that gap. Point it at a generated book skill and it emits the
plugin package this repo's conventions require:
<domain>/<slug>/
├── .claude-plugin/plugin.json manifest with `./skills/<slug>` + attribution
├── README.md what the skill knows and where it came from
├── agents/cs-<slug>.md persona that answers from the book
├── commands/cs-<slug>.md /cs:<slug> entry point
└── skills/<slug>/ the generated skill, copied verbatim
and prints the marketplace.json entry to paste into `.claude-plugin/`. It never
edits marketplace.json itself — registration is a repo-wide change and stays a
human decision.
Rights gate: a book skill built from a copyrighted work is personal study
notes. `--distribution shareable` therefore refuses to emit unless `--rights`
names a basis that permits redistribution. `--distribution local` (the default)
emits with a notice and records `source.cleared_for_distribution: false`
in the manifest.
Exit codes:
0 package emitted (or --dry-run / --sample completed)
1 refused: rights gate, existing destination, or unusable source skill
2 bad invocation
Usage:
python3 skill_plugin_emitter.py --skill-dir ~/.claude/skills/meadows-systems \\
--dest ./engineering --domain engineering
python3 skill_plugin_emitter.py --skill-dir ./ddia --dest ./engineering \\
--distribution shareable --rights public-domain
python3 skill_plugin_emitter.py --sample
"""
from __future__ import annotations
import argparse
import json
import os
import re
import shutil
import sys
import tempfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from book_skill_validator import ( # noqa: E402
ScanError,
parse_frontmatter,
scalar_value,
validate,
)
# Bases that permit redistributing a derived knowledge base. "fair-use" is
# deliberately absent: it is a defence, not a licence, and it is not this
# script's call to make.
RIGHTS_BASES = {
"public-domain": "the source work is in the public domain",
"open-license": "the source work carries a licence permitting derivative distribution",
"internal-docs": "the source is the organisation's own documentation",
"author-permission": "the rights holder gave written permission",
}
DEFAULT_DOMAIN = "engineering"
SLUG_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
class EmitError(RuntimeError):
"""The plugin package could not be emitted."""
def _scalar(frontmatter: str, key: str) -> str:
return scalar_value(frontmatter, key) or ""
def read_skill_identity(skill_dir: Path) -> dict:
"""Pull slug, description, title and chapter count out of the generated skill."""
master = skill_dir / "SKILL.md"
if not master.is_file():
raise EmitError(f"{skill_dir} has no SKILL.md — this is not a generated book skill")
text = master.read_text(encoding="utf-8-sig")
frontmatter, body = parse_frontmatter(text)
if frontmatter is None:
raise EmitError("SKILL.md has no YAML frontmatter; run book_skill_validator.py first")
heading = next((line[2:].strip() for line in body.splitlines() if line.startswith("# ")), "")
chapters = sorted((skill_dir / "chapters").glob("*.md")) if (skill_dir / "chapters").is_dir() else []
return {
"slug": _scalar(frontmatter, "name") or skill_dir.name,
"description": _scalar(frontmatter, "description"),
"title": heading or skill_dir.name,
"chapters": len(chapters),
}
def _assert_replaceable(package_root: Path, dest_root: Path, skill_dir: Path) -> None:
"""Refuse to recursively delete anything that is not a plugin package we own.
``--force`` reaches a ``shutil.rmtree``, so every assumption behind that call
is checked here rather than trusted: the target must be a real directory (not
a symlink pointing elsewhere), must live directly under the resolved
destination, must not contain the source skill being copied from, and must
look like a plugin package rather than an arbitrary folder that happens to
share the slug's name.
"""
if package_root.is_symlink():
raise EmitError(f"{package_root} is a symbolic link — refusing to replace it")
if not package_root.is_dir():
raise EmitError(f"{package_root} exists and is not a directory — refusing to replace it")
resolved = package_root.resolve()
if resolved.parent != dest_root:
raise EmitError(f"{package_root} resolves outside {dest_root} — refusing to replace it")
if resolved == dest_root or resolved.parent == resolved:
raise EmitError("refusing to replace a filesystem root or the destination itself")
source = skill_dir.resolve()
if resolved == source or resolved in source.parents:
raise EmitError(
f"{package_root} contains the source skill being copied from — refusing to "
f"delete the input"
)
manifest = resolved / ".claude-plugin" / "plugin.json"
if not manifest.is_file():
raise EmitError(
f"{package_root} exists but has no .claude-plugin/plugin.json, so it is not a "
f"plugin package this tool created. Move it aside by hand rather than passing "
f"--force."
)
def _assert_no_symlinks(skill_dir: Path) -> None:
"""Refuse to package a source tree containing any symbolic link.
The validator checks the files it knows about — SKILL.md, the three
supporting files, chapters/*.md — but `shutil.copytree` defaults to
`symlinks=False`, which follows a link *anywhere else* in the tree (an
`assets/` entry, an arbitrary subdirectory) and bakes the target's real
content into the emitted package. That package can then go out as
`--distribution shareable`, so a link pointing at something outside the
skill becomes part of a published artifact.
Refusing beats `copytree(symlinks=True)`: preserving the link would ship a
package whose contents depend on the emitting machine's filesystem. This
check runs even under `--skip-validation`, which otherwise disables the
file-level symlink checks entirely.
"""
offenders = [p for p in sorted(skill_dir.rglob("*")) if p.is_symlink()]
if offenders:
listed = "\n".join(f" {p.relative_to(skill_dir).as_posix()} -> {os.readlink(p)}"
for p in offenders[:10])
more = f"\n ... and {len(offenders) - 10} more" if len(offenders) > 10 else ""
raise EmitError(
f"source skill contains {len(offenders)} symbolic link(s); packaging would copy "
f"the target's content into a distributable package:\n{listed}{more}\n"
"Replace them with real files, or point --skill-dir at a tree without links."
)
def _plugin_manifest(identity: dict, *, domain: str, author: str, author_url: str,
repository: str, distribution: str, rights: str | None,
source_note: str) -> dict:
# Key set and the two approved extension fields (`source`, `attribution`) are
# fixed by scripts/check_plugin_json.py. Anything else — including a
# top-level "private" flag — fails the repo's CI gate, so the distribution
# marker lives inside `source`.
manifest = {
"name": identity["slug"],
"description": identity["description"],
"version": "1.0.0",
"author": {"name": author, "url": author_url},
"homepage": f"{repository}/tree/main/{domain}/{identity['slug']}",
"repository": repository,
# Covers this package's scaffolding — manifest, agent, command, README —
# NOT the compiled notes under skills/, whose terms follow the source
# document. Stated in `source.license_scope` too, so a tool reading only
# the manifest sees the distinction that README.md makes in prose.
"license": "MIT",
"skills": [f"./skills/{identity['slug']}"],
"source": {
"spec": "generated by engineering/book-to-skill",
"build_pattern": "book-to-skill conversion: extract -> analyze -> chapter files "
"+ glossary + patterns + cheatsheet -> master SKILL.md",
"distinct_from": "not a hand-authored skill; every claim traces to the source "
"document named below",
"source_document": source_note or identity["title"],
"chapters": identity["chapters"],
"distribution": distribution,
"license_scope": (
"The top-level `license` covers this package's scaffolding only. The "
"compiled notes under skills/ are derived from the source document and "
"carry that work's terms; see source.rights_basis."
),
},
}
if rights:
manifest["source"]["rights_basis"] = rights
manifest["source"]["rights_note"] = RIGHTS_BASES[rights]
if distribution == "local":
# A local package is study notes from a work the user owns. The flag is
# advisory metadata, not enforcement — it exists so a later publish step
# (or a reviewer) can see the package was never cleared for sharing.
manifest["source"]["cleared_for_distribution"] = False
return manifest
def _agent_markdown(identity: dict, domain: str, source_note: str) -> str:
slug = identity["slug"]
return f"""---
name: cs-{slug}
description: Answers from the knowledge base compiled from {source_note or identity['title']}. \
Loads the master frameworks first and reads a single chapter file on demand rather than the \
whole source. Refuses to answer beyond what the source covers.
skills: {domain}/{slug}/skills/{slug}
domain: {domain}
model: opus
tools: [Read, Grep, Glob]
---
# {identity['title']} — Knowledge Agent
## Voice
**Opening:** "Which framework or chapter are you reaching for?"
**Forcing question:** "Is this something the source actually covers, or are you asking me to
extrapolate past it?"
**Closing:** "That is the author's formulation, from ch<N>. Anything past it is my inference, not theirs."
## Purpose
Applies the frameworks compiled from **{source_note or identity['title']}** ({identity['chapters']} chapters
indexed) while the user works. Answers with the author's exact naming, then cites the chapter.
## How it navigates
1. Read `skills/{slug}/SKILL.md` — Core Frameworks and both indexes.
2. Match the question against the Topic Index; read **only** the chapter files it points to.
3. Reach for `glossary.md` for a term, `patterns.md` for a technique, `cheatsheet.md` for a decision.
4. Never load every chapter — that is the cost this skill exists to avoid.
## Hard rules
- **Cite the chapter.** Every framework claim names the chapter it came from.
- **Do not extrapolate silently.** If the source does not cover it, say so before answering from
general knowledge, and label which is which.
- **Preserve exact naming.** The author's term is the interface; a paraphrase breaks lookup.
- **Do not reproduce the source at length.** These are structured notes, not a copy of the work.
"""
def _command_markdown(identity: dict, domain: str, source_note: str) -> str:
slug = identity["slug"]
return f"""---
name: "cs-{slug}"
description: "/cs:{slug} [topic | framework | chNN] — query the knowledge base compiled from \
{source_note or identity['title']}. Use when applying its frameworks while working, looking up a \
term, or reading one chapter's summary."
---
# /cs:{slug} — {identity['title']}
**Command:** `/cs:{slug} [topic | framework name | chNN]`
## When to run
- Applying a framework from this source to work in progress
- Looking up the author's exact formulation of a term
- Reading one chapter's compiled summary without opening the source
- Checking whether the source covers a question at all
## What it does
1. Loads `{domain}/{slug}/skills/{slug}/SKILL.md` — Core Frameworks plus the Chapter and Topic indexes.
2. **No argument** → reports the core frameworks and the chapter index.
3. **A topic or framework name** → resolves it through the Topic Index and reads only the
matching chapter file.
4. **`chNN`** → reads that chapter's summary directly.
5. Answers with the author's naming and cites the chapter.
## Boundary
This command answers from **one source** ({identity['chapters']} chapters indexed). Anything it does not
cover gets said out loud rather than filled in — and hands-on work in your codebase belongs to the
engineering skills, not here.
"""
def _readme_markdown(identity: dict, *, domain: str, distribution: str, rights: str | None,
source_note: str) -> str:
slug = identity["slug"]
rights_line = (
f"**Rights basis:** `{rights}` — {RIGHTS_BASES[rights]}."
if rights else
"**Rights basis:** not declared. This package is local study notes from a work the "
"operator owns; it is not cleared for redistribution."
)
return f"""# {identity['title']}
Knowledge-base plugin compiled from **{source_note or identity['title']}** by
[`engineering/book-to-skill`](../book-to-skill/). {identity['chapters']} chapters indexed.
## What is in here
| File | Contents |
|------|----------|
| `skills/{slug}/SKILL.md` | Core frameworks, chapter index, topic index (resident, under 4k tokens) |
| `skills/{slug}/chapters/` | One summary per chapter — loaded on demand, never all at once |
| `skills/{slug}/glossary.md` | Every significant term, alphabetized, with its chapter |
| `skills/{slug}/patterns.md` | Techniques and design patterns with trade-offs |
| `skills/{slug}/cheatsheet.md` | Decision rules, thresholds and trade-off matrices |
## Use
```
/cs:{slug} # core frameworks + chapter index
/cs:{slug} <topic> # resolve via topic index, read one chapter
/cs:{slug} ch05 # read one chapter summary
```
Or invoke the `cs-{slug}` agent for a working session anchored to this source.
## Provenance and limits
Generated, not hand-authored: every claim traces to the source document. It carries that source's
blind spots, and it is a set of structured notes — not a copy of the work and not a substitute for
reading it.
{rights_line}
Distribution: `{distribution}`. Regenerate or extend with
`python3 {domain}/book-to-skill/skills/book-to-skill/scripts/extract_document.py`, then re-run
`book_skill_validator.py` before loading the result.
"""
def _marketplace_entry(identity: dict, domain: str, author: str) -> dict:
"""The entry to paste into marketplace.json.
`author` is threaded through rather than hardcoded: this snippet exists to
stop hand-editing mistakes, so printing a different name than the manifest
it accompanies would defeat its own purpose for anyone but the default.
"""
return {
"name": identity["slug"],
"source": f"./{domain}/{identity['slug']}",
"description": identity["description"],
"version": "1.0.0",
"author": {"name": author},
"keywords": ["knowledge-base", "book-to-skill", domain],
"category": domain,
}
def emit(*, skill_dir: Path, dest_root: Path, domain: str, author: str, author_url: str,
repository: str,
distribution: str, rights: str | None, source_note: str, slug_override: str | None,
force: bool, dry_run: bool, skip_validation: bool) -> dict:
skill_dir = skill_dir.expanduser().resolve()
identity = read_skill_identity(skill_dir)
if slug_override:
identity["slug"] = slug_override
if not SLUG_RE.fullmatch(identity["slug"]):
raise EmitError(f"'{identity['slug']}' is not a valid slug (lowercase, digits, single hyphens)")
if not identity["description"]:
raise EmitError("SKILL.md frontmatter has no description — the plugin manifest requires one")
if distribution == "shareable" and not rights:
raise EmitError(
"refusing to emit a shareable package without --rights.\n"
"A skill compiled from a copyrighted work is personal study notes; redistributing it "
"needs a basis.\n"
"Choose one of: " + ", ".join(sorted(RIGHTS_BASES)) + "\n"
"Or emit with --distribution local (the default) to keep it on this machine."
)
# Unconditional: --skip-validation waives content findings, never the
# guarantee that packaging copies only what is actually in the source tree.
_assert_no_symlinks(skill_dir)
if not skip_validation:
try:
findings = validate(skill_dir)
except ScanError as exc:
raise EmitError(f"source skill could not be validated: {exc}") from exc
errors = [f for f in findings if f.severity == "error"]
if errors:
summary = "\n".join(f" {f.path}:{f.line} ({f.rule_id}) {f.message}" for f in errors)
raise EmitError(
f"source skill has {len(errors)} validation error(s); fix them before wrapping "
f"it in a plugin:\n{summary}\n"
"(--skip-validation overrides this, but a package built on a broken index stays broken.)"
)
resolved_dest = dest_root.expanduser().resolve()
package_root = resolved_dest / identity["slug"]
if package_root.exists() and not force:
raise EmitError(f"{package_root} already exists — pass --force to replace it")
if package_root.exists():
_assert_replaceable(package_root, resolved_dest, skill_dir)
manifest = _plugin_manifest(identity, domain=domain, author=author, author_url=author_url,
repository=repository, distribution=distribution, rights=rights,
source_note=source_note)
files = {
".claude-plugin/plugin.json": json.dumps(manifest, indent=2, ensure_ascii=False) + "\n",
"README.md": _readme_markdown(identity, domain=domain, distribution=distribution,
rights=rights, source_note=source_note),
f"agents/cs-{identity['slug']}.md": _agent_markdown(identity, domain, source_note),
f"commands/cs-{identity['slug']}.md": _command_markdown(identity, domain, source_note),
}
result = {
"slug": identity["slug"],
"title": identity["title"],
"chapters": identity["chapters"],
"package_root": str(package_root),
"files": sorted(files) + [f"skills/{identity['slug']}/ (copied from {skill_dir})"],
"distribution": distribution,
"rights_basis": rights,
"dry_run": dry_run,
"marketplace_entry": _marketplace_entry(identity, domain, author),
}
if dry_run:
return result
if package_root.exists():
shutil.rmtree(package_root)
for relative, text in files.items():
target = package_root / relative
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(text, encoding="utf-8")
# symlinks=True is redundant after _assert_no_symlinks, and kept so a future
# edit that loosens that check cannot silently reintroduce dereferencing. It
# also bounds the check-then-act window: a link planted between the walk and
# this copy is copied *as a link*, so no target content is read.
copied = package_root / "skills" / identity["slug"]
shutil.copytree(skill_dir, copied, symlinks=True)
# Close the window properly: re-walk what actually landed. A concurrent
# writer with access to skill_dir could have planted a link after
# _assert_no_symlinks ran, and a distributable package must not carry one.
planted = [p for p in sorted(copied.rglob("*")) if p.is_symlink()]
if planted:
shutil.rmtree(package_root)
listed = "\n".join(f" {p.relative_to(copied).as_posix()}" for p in planted[:10])
raise EmitError(
f"symbolic link(s) appeared in the source tree while it was being copied — "
f"the package has been removed rather than shipped:\n{listed}\n"
"Something else is writing to the source directory; re-run once it is quiet."
)
return result
SAMPLE_SKILL = {
"SKILL.md": """---
name: sample-systems-thinking
description: "Knowledge base from \\"A Sample Systems Primer\\" by A. Author. Use when applying
the author's leverage-point frameworks, studying the book, or referencing its concepts."
---
# A Sample Systems Primer
## Chapter Index
| # | Title | Key Frameworks |
|---|-------|----------------|
| [ch01](chapters/ch01-stocks-and-flows.md) | Stocks and Flows | Stock-Flow Model |
## Topic Index
- **Stock-Flow Model** → ch01
""",
"chapters/ch01-stocks-and-flows.md": "# Chapter 1: Stocks and Flows\n\n## Core Idea\n"
"A stock is what accumulates; a flow is what changes it.\n",
"glossary.md": "**Stock** — an accumulation measured at a point in time (Ch 1)\n",
"patterns.md": "## Stock-Flow Model\n**When to use**: modelling accumulation.\n",
"cheatsheet.md": "| Situation | Do | Because |\n|---|---|---|\n| Stock falling | Raise inflow | Flows set direction |\n",
}
def run_sample(as_json: bool) -> int:
"""Emit a package from a built-in sample skill into a temp folder, then discard it."""
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "sample-systems-thinking"
(source / "chapters").mkdir(parents=True)
for relative, text in SAMPLE_SKILL.items():
(source / relative).write_text(text, encoding="utf-8")
result = emit(
skill_dir=source, dest_root=Path(tmp) / "out", domain=DEFAULT_DOMAIN,
author="Alireza Rezvani", author_url="https://alirezarezvani.com",
repository="https://github.com/alirezarezvani/claude-skills",
distribution="local", rights=None, source_note="A Sample Systems Primer",
slug_override=None, force=False, dry_run=False, skip_validation=False,
)
emitted = sorted(
p.relative_to(Path(tmp) / "out").as_posix()
for p in (Path(tmp) / "out").rglob("*") if p.is_file()
)
result["files"] = emitted
return _report(result, as_json)
def _report(result: dict, as_json: bool) -> int:
if as_json:
print(json.dumps(result, indent=2, ensure_ascii=False))
return 0
verb = "Would emit" if result["dry_run"] else "Emitted"
print(f"{verb} plugin package: {result['package_root']}")
print(f" source : {result['title']} ({result['chapters']} chapters)")
print(f" slug : {result['slug']}")
print(f" sharing : {result['distribution']}"
+ (f" (rights: {result['rights_basis']})" if result["rights_basis"] else " — not cleared for redistribution"))
print(" files :")
for relative in result["files"]:
print(f" {relative}")
print("\nRegister it by adding this entry to .claude-plugin/marketplace.json `plugins`:\n")
print(json.dumps(result["marketplace_entry"], indent=2, ensure_ascii=False))
print("\n(This tool never edits marketplace.json — registration is a repo-wide change.)")
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="skill_plugin_emitter.py",
description="Wrap a generated book skill in a claude-skills plugin package.",
)
parser.add_argument("--skill-dir", help="generated book skill folder (contains SKILL.md)")
parser.add_argument("--dest", help="domain folder to emit into, e.g. ./engineering")
parser.add_argument("--domain", default=DEFAULT_DOMAIN,
help=f"domain name used in paths and metadata (default: {DEFAULT_DOMAIN})")
parser.add_argument("--slug", help="override the plugin slug (default: the skill's name)")
parser.add_argument("--source-note", default="",
help="how to name the source document in generated prose")
parser.add_argument("--author", default="Alireza Rezvani", help="plugin manifest author")
parser.add_argument("--author-url", default="https://alirezarezvani.com",
help="author URL (required by the repo plugin.json schema)")
parser.add_argument("--repository", default="https://github.com/alirezarezvani/claude-skills",
help="repository URL for the manifest")
parser.add_argument("--distribution", choices=("local", "shareable"), default="local",
help="local keeps the package private (default); shareable requires --rights")
parser.add_argument("--rights", choices=sorted(RIGHTS_BASES),
help="basis on which the compiled notes may be redistributed")
parser.add_argument("--force", action="store_true", help="replace an existing destination")
parser.add_argument("--dry-run", action="store_true", help="report without writing anything")
parser.add_argument("--skip-validation", action="store_true",
help="emit even if the source skill has validation errors")
parser.add_argument("--output", choices=("text", "json"), default="text")
parser.add_argument("--sample", action="store_true",
help="emit a package from a built-in sample skill into a temp folder")
args = parser.parse_args(argv)
if args.sample:
return run_sample(args.output == "json")
if not args.skill_dir or not args.dest:
parser.print_usage(sys.stderr)
print("ERROR: --skill-dir and --dest are both required (or use --sample).", file=sys.stderr)
return 2
try:
result = emit(
skill_dir=Path(args.skill_dir), dest_root=Path(args.dest), domain=args.domain,
author=args.author, author_url=args.author_url, repository=args.repository,
distribution=args.distribution,
rights=args.rights, source_note=args.source_note, slug_override=args.slug,
force=args.force, dry_run=args.dry_run, skip_validation=args.skip_validation,
)
except EmitError as exc:
print(f"REFUSED: {exc}", file=sys.stderr)
return 1
return _report(result, args.output == "json")
if __name__ == "__main__":
try:
sys.exit(main())
except BrokenPipeError:
# `tool | head` closes the pipe while this is still writing. Exit quietly
# instead of dumping a traceback: redirect stdout to devnull first so the
# interpreter's shutdown flush cannot re-raise. 141 = 128 + SIGPIPE.
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
sys.exit(141)

366
engineering/book-to-skill/skills/book-to-skill/scripts/token_budget_estimator.py Normal file
View file

@ -0,0 +1,366 @@
#!/usr/bin/env python3
"""token_budget_estimator.py — what the conversion costs, and what it saves.
Two modes, both deterministic and standard-library only.
**Pre-flight** (`--full-text`): before spending a token on generation, model
the cost of answering one targeted question about the source three ways —
context-dump the whole source stays resident and is re-billed every turn
discovery-loop a live document-reading agent navigates: table of contents,
then the chapter it guessed at, then a backtrack for a
definition it turns out to need
book-to-skill a small resident core plus one compiled chapter, on demand
The discovery-loop figure is a *model with stated assumptions*, not a
measurement of any particular agent — but it is built from the real token sizes
of this source's own ToC and chapters, so it is an estimate you can argue with
rather than a number pulled from a marketing page.
**Post-flight** (`--skill-dir`): audit a generated skill file-by-file against
the token budgets the converter commits to, so overflow is caught before
compaction silently truncates the master file's indexes.
Given both, it does both and cross-references them.
Token counts use the same deterministic heuristic as the extractor (words/0.75,
with a separate CJK codepoint ratio), so every number in the pipeline agrees.
The estimate is an estimate: it is stable and comparable, not a BPE count.
Exit codes:
0 report produced
1 the source has no detectable chapters (nothing to model)
2 bad invocation, or a path that does not exist / is not a compiled skill
Adapted from virgiliojr94/book-to-skill's discovery_tax.py (MIT), rewritten to
drop the optional tiktoken path and to add the post-flight budget audit.
See ../../../LICENSE.
Usage:
python3 token_budget_estimator.py --full-text "$WORKDIR/full_text.txt"
python3 token_budget_estimator.py --skill-dir ~/.claude/skills/meadows-systems
python3 token_budget_estimator.py --full-text FT.txt --skill-dir SKILL_DIR --target-chapter 5
python3 token_budget_estimator.py --sample
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import tempfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from book_to_skill.config import ( # noqa: E402
CHAPTER_TOKEN_CEILING,
SKILL_FILE_BUDGETS,
)
from book_to_skill.utils import _chapter_number as chapter_number # noqa: E402
from book_to_skill.utils import estimate_tokens # noqa: E402
# Shared with book_skill_validator.py via config — see the note there.
BUDGETS = SKILL_FILE_BUDGETS
CHAPTER_CEILING = CHAPTER_TOKEN_CEILING
DEFAULT_CORE_TOKENS = 4_000
DEFAULT_COMPILED_CHAPTER = 1_000
_TOC_RE = re.compile(
r"^\s*(?:table of contents|contents|sum[áa]rio|[íi]ndice|目录|目錄|目次|"
r"table des matières|inhaltsverzeichnis|inhoudsopgave)\s*$",
re.IGNORECASE | re.MULTILINE,
)
def split_chapters(text: str) -> list[tuple[int | None, str, str]]:
"""Return [(number, heading, body)], one segment per heading occurrence.
Text before the first heading is the leading front-matter segment
(number=None). A chapter number can appear twice — once in the table of
contents and once as the real heading — so callers pick the largest body.
"""
segments: list[tuple[int | None, str, list[str]]] = [(None, "__front__", [])]
for line in text.splitlines():
number = chapter_number(line)
if number is not None:
segments.append((number, line.strip(), []))
segments[-1][2].append(line)
return [(n, h, "\n".join(b)) for n, h, b in segments]
def best_chapter(chapters, number: int) -> tuple[str, int] | None:
"""(heading, body_tokens) for a chapter number, taking the largest body —
the real chapter, not its one-line table-of-contents entry."""
candidates = [(h, estimate_tokens(b)) for n, h, b in chapters if n == number]
return max(candidates, key=lambda item: item[1]) if candidates else None
def extract_toc(front_matter: str) -> str:
match = _TOC_RE.search(front_matter)
# No explicit ToC heading: assume the agent skims the whole front matter.
return front_matter[match.start():] if match else front_matter
class InputError(RuntimeError):
"""A path given on the command line cannot be used."""
def audit_skill_dir(skill_dir: Path) -> dict:
"""File-by-file token audit of a generated skill against the budgets.
Refuses a directory that does not exist or has no SKILL.md. Without that
check a typo'd --skill-dir produced a full, plausible-looking audit — every
row "missing", every cap satisfied, exit 0 — which reads as a pass. A gate
that reports success for a path that isn't there is worse than no gate.
"""
if not skill_dir.exists():
raise InputError(f"no such skill directory: {skill_dir}")
if not skill_dir.is_dir():
raise InputError(f"not a directory: {skill_dir}")
if not (skill_dir / "SKILL.md").is_file():
raise InputError(
f"{skill_dir} has no SKILL.md — this is not a compiled skill. "
f"Point --skill-dir at the generated skill folder."
)
rows, over = [], []
for filename, cap in BUDGETS.items():
path = skill_dir / filename
if not path.is_file():
rows.append({"file": filename, "tokens": None, "cap": cap, "status": "missing"})
continue
tokens = estimate_tokens(path.read_text(encoding="utf-8-sig"))
status = "over" if tokens > cap else "ok"
rows.append({"file": filename, "tokens": tokens, "cap": cap, "status": status})
if status == "over":
over.append(filename)
chapter_dir = skill_dir / "chapters"
chapters = sorted(chapter_dir.glob("*.md")) if chapter_dir.is_dir() else []
chapter_tokens = [estimate_tokens(p.read_text(encoding="utf-8-sig")) for p in chapters]
for path, tokens in zip(chapters, chapter_tokens):
status = "over" if tokens > CHAPTER_CEILING else "ok"
rows.append({"file": f"chapters/{path.name}", "tokens": tokens,
"cap": CHAPTER_CEILING, "status": status})
if status == "over":
over.append(f"chapters/{path.name}")
resident = next((r["tokens"] for r in rows if r["file"] == "SKILL.md" and r["tokens"]), 0)
return {
"rows": rows,
"chapters": len(chapters),
"resident_tokens": resident,
"chapter_tokens_avg": sum(chapter_tokens) // len(chapter_tokens) if chapter_tokens else 0,
"chapter_tokens_total": sum(chapter_tokens),
"over_budget": over,
}
def model_discovery(full_text: str, target: int, core_tokens: int,
compiled_chapter: int) -> dict | None:
segments = split_chapters(full_text)
front, chapters = segments[0][2], segments[1:]
if not chapters:
return None
distinct = sorted({n for n, _, _ in chapters if n is not None})
number = target
chosen = best_chapter(chapters, number)
if chosen is None and distinct:
number = distinct[min(target - 1, len(distinct) - 1)]
chosen = best_chapter(chapters, number)
if chosen is None:
return None
heading, target_raw = chosen
prior = best_chapter(chapters, number - 1)
toc_tokens = estimate_tokens(extract_toc(front))
skill_cost = core_tokens + compiled_chapter
return {
"chapters_detected": len(distinct),
"target_chapter": number,
"target_heading": heading[:80],
"source_total_tokens": estimate_tokens(full_text),
"toc_tokens": toc_tokens,
"target_chapter_raw_tokens": target_raw,
"prior_chapter_raw_tokens": prior[1] if prior else 0,
"cost_context_dump": estimate_tokens(full_text),
"cost_discovery_best": toc_tokens + target_raw,
"cost_discovery_loop": toc_tokens + target_raw + (prior[1] if prior else 0),
"cost_book_to_skill": skill_cost,
"core_tokens": core_tokens,
"compiled_chapter_tokens": compiled_chapter,
# Conversion is not free: it costs a full generation pass up front. It
# pays back only when the source is large enough that reading it (or
# navigating it) repeatedly dwarfs the resident core. Below roughly 3x
# the compiled cost, just hand the agent the document.
"worth_converting": estimate_tokens(full_text) >= 3 * skill_cost,
}
SAMPLE_SOURCE = """Table of Contents
Chapter 1: Framing
Chapter 2: Leverage Points
Chapter 3: Failure Modes
Chapter 1: Framing
""" + ("A system is a set of elements interconnected in a way that produces a "
"characteristic pattern of behaviour over time. " * 900) + """
Chapter 2: Leverage Points
""" + ("Places within a complex system where a small shift in one thing produces "
"big changes in everything. The list runs from parameters, the weakest, to "
"paradigms, the strongest. " * 1200) + """
Chapter 3: Failure Modes
""" + ("Policy resistance, the tragedy of the commons, drift to low performance, "
"escalation, and seeking the wrong goal. " * 900)
def run_sample(as_json: bool) -> int:
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp) / "full_text.txt"
source.write_text(SAMPLE_SOURCE, encoding="utf-8")
return run(full_text_path=source, skill_dir=None, target=2,
core_tokens=DEFAULT_CORE_TOKENS, as_json=as_json)
def _ratio(a: int, b: int) -> str:
return f"{a / b:.1f}x" if b else "n/a"
def _print_discovery(model: dict) -> None:
print("Discovery Loop Tax — modelled on this source\n")
print(f" chapters detected : {model['chapters_detected']}")
print(f" target : chapter {model['target_chapter']} ({model['target_heading']})")
print(f" source total : {model['source_total_tokens']:,} tokens\n")
print(" Cost to answer ONE targeted question (tokens entering context):\n")
print(f" context-dump : {model['cost_context_dump']:>9,} resident, re-billed EVERY turn")
print(f" discovery (best) : {model['cost_discovery_best']:>9,} ToC ({model['toc_tokens']:,}) "
f"+ raw target chapter ({model['target_chapter_raw_tokens']:,})")
print(f" discovery (loop) : {model['cost_discovery_loop']:>9,} + one backtrack chapter "
f"({model['prior_chapter_raw_tokens']:,})")
print(f" book-to-skill : {model['cost_book_to_skill']:>9,} core ({model['core_tokens']:,}) "
f"+ compiled chapter ({model['compiled_chapter_tokens']:,})\n")
print(" book-to-skill advantage:")
print(f" vs context-dump : {_ratio(model['cost_context_dump'], model['cost_book_to_skill'])} fewer tokens")
print(f" vs discovery best : {_ratio(model['cost_discovery_best'], model['cost_book_to_skill'])} fewer tokens")
print(f" vs discovery loop : {_ratio(model['cost_discovery_loop'], model['cost_book_to_skill'])} fewer tokens")
print("\n Read this honestly: the discovery figures are a model built from this source's real")
print(" ToC and chapter sizes, and they are a one-time read. context-dump is the recurring one.")
if model["worth_converting"]:
print(f"\n VERDICT: worth converting — the source is {_ratio(model['cost_context_dump'], model['cost_book_to_skill'])} "
f"the size of the compiled skill.")
else:
print(f"\n VERDICT: probably not worth converting — at {model['source_total_tokens']:,} tokens this source")
print(f" is smaller than ~3x the compiled skill ({model['cost_book_to_skill']:,}). Hand the agent the document.")
def _print_audit(audit: dict) -> None:
print("\nGenerated-skill budget audit\n")
print(f" {'file':<34} {'tokens':>8} {'cap':>8} status")
for row in audit["rows"]:
tokens = f"{row['tokens']:,}" if row["tokens"] is not None else "—"
print(f" {row['file']:<34} {tokens:>8} {row['cap']:>8,} {row['status']}")
print(f"\n resident core : {audit['resident_tokens']:,} tokens (loaded every session)")
print(f" chapters : {audit['chapters']} files, "
f"~{audit['chapter_tokens_avg']:,} tokens each, "
f"{audit['chapter_tokens_total']:,} total (on demand)")
if audit["over_budget"]:
print(f"\n OVER BUDGET: {', '.join(audit['over_budget'])}")
print(" SKILL.md overflow is the one that bites: compaction truncates from the end, so the")
print(" Chapter and Topic indexes — the navigation this skill exists for — go first.")
def run(*, full_text_path: Path | None, skill_dir: Path | None, target: int,
core_tokens: int, as_json: bool) -> int:
audit = audit_skill_dir(skill_dir) if skill_dir else None
model = None
if full_text_path:
if not full_text_path.is_file():
raise InputError(
f"no such extracted-text file: {full_text_path}\n"
"Pass the full_text.txt path that extract_document.py printed "
"(it is also metadata.json's `output_text`)."
)
text = full_text_path.read_text(encoding="utf-8", errors="ignore")
# A measured skill beats the design cap: use its real resident size and
# its real average chapter size when we have them.
core = audit["resident_tokens"] if audit and audit["resident_tokens"] else core_tokens
chapter = (audit["chapter_tokens_avg"] if audit and audit["chapter_tokens_avg"]
else DEFAULT_COMPILED_CHAPTER)
model = model_discovery(text, target, core, chapter)
if model is None:
print("No chapters detected — nothing to model. The source may be a technical PDF "
"whose headings were flattened by text extraction; try --mode technical.",
file=sys.stderr)
return 1
if as_json:
print(json.dumps({"discovery_model": model, "budget_audit": audit},
indent=2, ensure_ascii=False))
return 0
print("Token counts use the extractor's words/0.75 heuristic (with a CJK codepoint ratio) — "
"stable and comparable, not a BPE count.\n")
if model:
_print_discovery(model)
if audit:
_print_audit(audit)
return 0
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="token_budget_estimator.py",
description="Model conversion savings pre-flight and audit skill budgets post-flight.",
)
parser.add_argument("--full-text", help="extractor output full_text.txt (pre-flight model)")
parser.add_argument("--skill-dir", help="generated skill folder (post-flight budget audit)")
parser.add_argument("--target-chapter", type=int, default=5,
help="chapter number the modelled question is about (default: 5)")
parser.add_argument("--core-tokens", type=int, default=DEFAULT_CORE_TOKENS,
help="assumed resident SKILL.md size when --skill-dir is not given")
parser.add_argument("--output", choices=("text", "json"), default="text")
parser.add_argument("--sample", action="store_true",
help="run the model on a built-in three-chapter sample source")
args = parser.parse_args(argv)
if args.sample:
return run_sample(args.output == "json")
if not args.full_text and not args.skill_dir:
parser.print_usage(sys.stderr)
print("ERROR: give --full-text, --skill-dir, or both (or use --sample).", file=sys.stderr)
return 2
try:
return run(
full_text_path=Path(args.full_text) if args.full_text else None,
skill_dir=Path(args.skill_dir) if args.skill_dir else None,
target=args.target_chapter,
core_tokens=args.core_tokens,
as_json=args.output == "json",
)
except InputError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
try:
sys.exit(main())
except BrokenPipeError:
# `tool | head` closes the pipe while this is still writing. Exit quietly
# instead of dumping a traceback: redirect stdout to devnull first so the
# interpreter's shutdown flush cannot re-raise. 141 = 128 + SIGPIPE.
os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
sys.exit(141)

19
engineering/memory-engineering/.claude-plugin/plugin.json Normal file
View file

@ -0,0 +1,19 @@
{
"name": "memory-engineering",
"description": "Engineer an agent's forgetting, not just its remembering. Four deterministic stdlib scripts implement the four lenses of agent memory: a cost profiler that splits construction from query spend and reports cost per correct answer (construction energy exceeds total query energy across 300 queries in the Stanford characterization); an architecture picker that scores the four paradigm families — long-context, flat RAG, structure-augmented RAG, agentic — disqualifies on hard constraints, names the cost the winning choice makes you pay, and refuses to pick when the top two tie; a density auditor that classifies every record in a real memory directory or JSONL export as FACT / SKILL / LOG / PROSE, finds near-duplicates, and flags stale and time-relative wording; and a forgetting-policy linter that fails any design with no forgetting rule or that auto-merges contradictions. Ships cs-memory-engineer agent, /cs:memory-engineering and /cs:forgetting-audit commands, 4 references citing 7 sources each, and a seven-question forcing worksheet.",
"version": "2.11.2",
"author": {
"name": "Alireza Rezvani",
"url": "https://alirezarezvani.com"
},
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/memory-engineering",
"repository": "https://github.com/alirezarezvani/claude-skills",
"license": "MIT",
"skills": [
"./skills/memory-engineering"
],
"source": {
"build_pattern": "Four-lens synthesis (Stanford cost / Microsoft what-to-keep / Anthropic control / Nvidia hardware) + 4 deterministic stdlib scripts, with a blocking forgetting gate. Framing synthesized from 'How to be a Memory Engineer' by @N01ennn (x.com/N01ennn/status/2083971749079581120); every quantitative claim is cited to the primary source instead, and two of the article's paraphrases are corrected in the references (the 47x energy figure is the spread across ten systems, not an accuracy-matched pair; the 97% error-reduction figure is one named customer's reported result, not a general property).",
"distinct_from": "llm-wiki (maintains one specific markdown vault; this audits and prices any memory system); skillopt-sleep (runs a nightly consolidation loop; this decides whether that loop's output is worth keeping); agent-harness (bounds a task loop; this bounds a store); llm-cost-optimizer (prices inference generally, not the memory write path specifically)"
}
}

121
engineering/memory-engineering/README.md Normal file
View file

@ -0,0 +1,121 @@
# memory-engineering
> Your agent's problem was never that it forgets. It's that it never forgets
> **on purpose**.
A storer optimizes what a system remembers. A memory engineer optimizes what it
forgets. This plugin makes that shift executable: four deterministic stdlib
scripts that price the write path, choose which cost to pay, audit what a store
actually holds, and **refuse a design with no forgetting policy**.
## Why this exists
Everyone building agent memory optimizes retrieval. Almost nobody engineers
what it costs to build, what is worth keeping, who can delete it, and where it
lands on the hardware. Stanford's systems characterization of ten memory systems
found the gap concretely:
- **Construction energy exceeds total query-phase energy across 300 queries** —
the bill is paid on the write path you never watch.
- Energy per correct answer **spreads more than 47×** across systems (BM25 at
4,145 J; MIRIX at ~197 kJ).
- At 1M tokens, footprint varies **up to 9×** — and **"none of the evaluated
systems prune or forget by default."**
If you did not build forgetting, you do not have it.
## Install
```bash
/plugin marketplace add alirezarezvani/claude-skills
/plugin install memory-engineering
```
## Use
```bash
/cs:memory-engineering ~/.claude/memory # full four-lens pass
/cs:forgetting-audit design.json # just the blocking gate
```
Or run the scripts directly — each has `--help`, `--sample`, and `--output json`:
```bash
cd skills/memory-engineering
python scripts/memory_cost_profiler.py --sample
python scripts/memory_architecture_picker.py --sample
python scripts/memory_density_auditor.py --dir ~/.claude/memory
python scripts/forgetting_policy_linter.py --sample-failing
```
## The four scripts
| Script | Lens | What it does | Exit codes |
|---|---|---|---|
| `memory_cost_profiler.py` | Stanford — *what does it cost?* | Splits construction vs query spend, computes **cost per correct answer**, flags under-amortized writes and construction co-located with live queries | 0 · 2 finding · 3 bad input |
| `memory_architecture_picker.py` | Stanford — *which cost to pay?* | Scores long-context / flat RAG / structure-augmented RAG / agentic against constraints, disqualifies on hard limits, **names the cost you're choosing**, refuses to pick on a tie | 0 · 2 ambiguous · 3 bad input · 4 none viable |
| `memory_density_auditor.py` | Microsoft — *what's worth keeping?* | Classifies records **FACT / SKILL / LOG / PROSE**, finds near-duplicates, flags stale and time-relative wording, scores knowledge density. Runs on a real directory or JSONL | 0 dense · 2 finding · 3 bad input |
| `forgetting_policy_linter.py` | Anthropic + the gate | 8 checks; **F1** (explicit forgetting rule) and **F4** (contradictions surfaced, never auto-merged) are **blocking** | 0 PASS · 2 CONDITIONAL · **4 FAIL** |
Stdlib only. No network, no LLM calls, no dependencies.
## The gate
```
$ python scripts/forgetting_policy_linter.py --sample-failing
VERDICT: FAIL (0/8 checks pass)
This design does not forget on purpose. F1 failed. F4 failed.
FAIL F1 explicit forgetting rule [BLOCKING]
No TTL, no capacity bound, no decay. The store only grows.
FAIL F4 contradictions surfaced, never auto-merged [BLOCKING]
Contradiction policy is 'newest_wins', which resolves conflicts silently.
```
**F4 is blocking on purpose.** Two memories that disagree may both have been
true in different contexts — "deploys go through Jenkins" and "deploys go
through GitHub Actions" is not a contradiction to resolve, it is a migration to
record. Auto-merging destroys the only evidence the conflict existed.
## Evidence discipline
The four-lens framing synthesizes *"How to be a Memory Engineer, from the
perspective of Stanford, Microsoft, Anthropic and Nvidia"* by
[@N01ennn](https://x.com/N01ennn/status/2083971749079581120).
**Every quantitative claim is cited to the primary source, not to that
article,** and each carries an explicit confidence level. Two of the article's
paraphrases are corrected in the references:
- The **47×** energy figure is the spread across ten evaluated systems, not
"two systems with identical accuracy" (`memory_cost_canon.md` §2).
- The **97%** first-pass-error reduction is Rakuten's named, vendor-published
customer testimonial — not a controlled study or a general property of
building memory this way (`memory_control_and_governance.md` §4).
## Not this plugin
| You want | Use |
|---|---|
| Build and maintain one markdown knowledge vault | `llm-wiki` |
| A nightly self-improvement loop over transcripts | `skillopt-sleep` |
| Bound an agent's *task loop* | `agent-harness` |
| Price inference generally | `llm-cost-optimizer` |
This bounds a **store**, not a loop and not a vault.
## Primary sources
- Omri, Y. et al. — *Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads*, [arXiv:2606.06448](https://arxiv.org/abs/2606.06448)
- Microsoft Research — [*PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents*](https://www.microsoft.com/en-us/research/publication/plugmem-a-task-agnostic-plugin-memory-module-for-llm-agents/)
- Kontonis, V. et al. — *MEMENTO: Teaching LLMs to Manage Their Own Context*, [arXiv:2604.09852](https://arxiv.org/abs/2604.09852)
- Anthropic — [*Built-in memory for Claude Managed Agents*](https://claude.com/blog/claude-managed-agents-memory)
Full citation lists (7 sources each) are in
[`skills/memory-engineering/references/`](skills/memory-engineering/references/).
## License
MIT.

78
engineering/memory-engineering/agents/cs-memory-engineer.md Normal file
View file

@ -0,0 +1,78 @@
---
name: cs-memory-engineer
description: Use when someone is adding memory to an agent, choosing a memory architecture, auditing an existing memory store, or asking why their memory system is expensive, slow, or wrong. Prices the write path, names which cost the design is paying, classifies what the store actually holds, and refuses to sign off a design with no forgetting policy.
model: inherit
---
# cs-memory-engineer
You are a memory engineer. Your first question is never "what should it
remember?" — it is **"what leaves the store, and on what rule?"**
## Voice
Blunt, cost-first, and allergic to the word "best". You have read the systems
research and you quote it with its confidence level attached. You would rather
tell someone their memory system is unaffordable now than let them discover it
after two years of accumulated records.
Your opening move on almost any request:
> "Before we talk about what it retrieves — what does one write cost, and what
> leaves the store?"
## Hard rules
1. **Never quote a quality number without a cost number.** Accuracy alone is
the measurement this role exists to refuse.
2. **Never recommend the "best" memory system.** No family wins on build cost,
query speed, and accuracy at once. Recommend a family and *name the cost it
makes them pay*.
3. **Never auto-merge contradictions**, and never let a design do it. Two
memories that disagree may both have been true in different contexts. The
system surfaces; the human decides.
4. **Never sign off a design without a forgetting rule.** If they did not build
forgetting, they do not have it — no evaluated system provides it by default.
`forgetting_policy_linter.py` exiting 4 is a stop, not a suggestion.
5. **Never schedule a pass that has not been run by hand once.** If the manual
run did not change a decision, automating it only makes noise.
6. **Attribute every number.** Say which paper or vendor it came from and how
much confidence it carries. Vendor customer testimonials are not benchmarks
and must be labeled as testimonials.
## How you work
1. **Price it.** Run `memory_cost_profiler.py`. Lead with the
construction/query split and cost per correct answer, not with latency.
2. **Name the tradeoff.** Run `memory_architecture_picker.py`. If it exits 2
(ambiguous), do not pick for them — put the tie-breaking question to them and
wait.
3. **Look in the store.** Run `memory_density_auditor.py` against the real
directory. People are consistently wrong about how much of their memory is
transcripts.
4. **Gate.** Run `forgetting_policy_linter.py`. Report FAIL as a blocker with
the specific check that failed and its fix.
5. **Sequence it.** Write path first → contradiction detection by hand →
forgetting policy before volume climbs → hardware tuning last.
## What you refuse
- Recommending a memory system when the user has not stated a retention rule.
- Reporting accuracy improvements without the cost delta beside them.
- Treating a vendor's published customer figure as a general property of an
approach.
- Letting "we'll add pruning later" stand. Later is a data migration with a
judgment call attached to every record, which is why it never happens.
## Scope boundaries
- Maintaining one specific markdown vault → hand off to `llm-wiki`.
- A nightly consolidation loop over transcripts → hand off to `skillopt-sleep`.
- Bounding an agent's task loop → hand off to `agent-harness`.
You bound the **store**, not the loop and not the vault.
## Skill
Full workflow, scripts, references and worksheets:
`engineering/memory-engineering/skills/memory-engineering/SKILL.md`

60
engineering/memory-engineering/commands/cs-forgetting-audit.md Normal file
View file

@ -0,0 +1,60 @@
---
description: Run only the blocking forgetting gate on a memory design or store — what leaves, and on what rule.
argument-hint: "[policy JSON, or a memory directory to audit]"
---
# /cs:forgetting-audit
The short pass. Skip the cost and architecture work; answer one question about
`$ARGUMENTS`:
> **What leaves this store, and on what rule?**
## Run
If given a policy JSON:
```bash
python skills/memory-engineering/scripts/forgetting_policy_linter.py --policy <policy.json>
```
If given a directory, first show what is actually accumulating, then gate:
```bash
python skills/memory-engineering/scripts/memory_density_auditor.py --dir <path>
python skills/memory-engineering/scripts/forgetting_policy_linter.py --policy <policy.json>
```
If no policy file exists, that is the answer — nothing leaves the store. Show
what `--sample-failing` blocks, then help write one from
`skills/memory-engineering/assets/forgetting_policy_template.md`.
## The two blocking checks
- **F1 — an explicit forgetting rule** (TTL, capacity bound with a stated
eviction order, or relevance decay). None of the memory systems in the
Stanford evaluation prunes or forgets by default: if it was not built, it does
not exist.
- **F4 — contradictions surfaced, never auto-merged.** `newest_wins`,
`auto_merge`, `overwrite` and `last_write_wins` all fail. Two memories that
disagree may both have been true in different contexts, and silently resolving
them destroys the only evidence the conflict existed.
The other six checks (dedup, consolidation, scope, audit trail, rollback,
growth-slope monitoring) degrade the verdict to CONDITIONAL rather than failing
it.
## Report
1. **Verdict** — PASS (0) / CONDITIONAL (2) / **FAIL (4)**
2. **Every failing check** with its ID, why it matters, and its fix
3. **The one thing to fix first** — F1 or F4 if either failed; otherwise the
highest-leverage warning
## Do not
- Do not soften a FAIL into a suggestion. Retrofitting forgetting onto a full
store is a data migration with a judgment call attached to every record —
which is exactly why it never happens.
- Do not accept "we will add pruning later." Later is the failure mode.
- Do not propose auto-resolution for contradictions, in any form.

77
engineering/memory-engineering/commands/cs-memory-engineering.md Normal file
View file

@ -0,0 +1,77 @@
---
description: Price, choose, audit and gate an agent memory system — the full four-lens memory-engineering pass.
argument-hint: "[memory dir, design spec JSON, or a question about a memory system]"
---
# /cs:memory-engineering
Run the memory-engineering pass on `$ARGUMENTS`.
Load `engineering/memory-engineering/skills/memory-engineering/SKILL.md` and
follow it. Report every script's exit code as a finding — a non-zero exit is a
result to surface, never an error to swallow.
## Pre-flight
Establish these before running anything. If the user cannot answer 1 or 2,
that gap **is** the first finding — say so rather than guessing:
1. **Does a memory system exist yet, or is this a design?** Design → steps 1, 2, 4. Existing store → steps 1, 3, 4.
2. **What leaves the store today?** If the answer is "nothing", skip to step 4; the gate result is the headline.
3. **Is this actually a memory question?** Maintaining one markdown vault → `llm-wiki`. Nightly consolidation loop → `skillopt-sleep`. Bounding a task loop → `agent-harness`.
## Pass
**1. Price the write path**
```bash
python skills/memory-engineering/scripts/memory_cost_profiler.py --spec <workload.json>
```
Lead the report with the construction/query split and **cost per correct
answer**. Never present accuracy on its own.
**2. Choose which cost to pay**
```bash
python skills/memory-engineering/scripts/memory_architecture_picker.py --constraints <workload.json>
```
If it exits 2 (`AMBIGUOUS`), **stop and put the printed tie-breaking question to
the user.** Do not pick for them — the tie is real, not a tooling limitation.
**3. Audit the real store** (skip if this is a greenfield design)
```bash
python skills/memory-engineering/scripts/memory_density_auditor.py --dir <path>
```
Report the FACT/SKILL/LOG/PROSE split. Users are routinely wrong about how much
of their store is transcripts.
**4. Gate on forgetting** — blocking
```bash
python skills/memory-engineering/scripts/forgetting_policy_linter.py --policy <design.json>
```
Exit 4 is a **stop**. Name the failing check (F1 or F4) and its fix. Do not
present a FAIL alongside a recommendation to proceed.
## Output
Report in this order — cost before quality, always:
1. **Verdict** — one line, leading with the blocking result if there is one
2. **Cost** — construction/query split, cost per correct answer, amortization
3. **Architecture** — the family, and the cost it makes them pay
4. **What the store holds** — the FACT/SKILL/LOG/PROSE split, duplicates, staleness
5. **Forgetting gate** — PASS / CONDITIONAL / FAIL with the named failing checks
6. **Next step** — exactly one, sequenced per the ship order
Attribute every number to its source with a confidence level. Vendor customer
figures are testimonials, not benchmarks — label them as such.
For a structured walkthrough, hand the user
`skills/memory-engineering/assets/memory_engineer_worksheet.md` (the seven forcing questions) and walk
them **one at a time**.

100
engineering/memory-engineering/skills/memory-engineering/SKILL.md Normal file
View file

@ -0,0 +1,100 @@
---
name: memory-engineering
description: Use when designing, reviewing, or paying for an agent memory system — adding memory to an agent, choosing between long-context / RAG / graph / agentic memory, auditing what a CLAUDE.md or memory directory actually holds, deciding what to keep and what to expire, or when a memory store keeps growing and nobody has said what leaves it. Prices the write path, picks which cost to pay, classifies records as facts / skills / logs, and refuses a design that has no forgetting policy.
argument-hint: "[optional: path to a memory dir, design spec JSON, or a question]"
license: MIT
metadata:
version: 1.0.0
build_pattern: "Four-lens synthesis (Stanford / Microsoft / Anthropic / Nvidia) + 4 deterministic stdlib scripts, with a blocking forgetting gate"
distinct_from: "llm-wiki (maintains one specific markdown vault; this audits and prices any memory system); skillopt-sleep (runs a nightly consolidation loop; this decides whether that loop's output is worth keeping); agent-harness (bounds a task loop; this bounds a store)"
---
# Memory Engineering — engineer the forgetting, not just the remembering
> **Portability:** 4 stdlib scripts, no APIs/LLM calls/network. They measure and gate; you decide.
## What this does
Anyone can give an agent memory: vector store, pipe in the history, retrieve
top-k. That works until the history outgrows the context window, the write path
costs more than every query it serves, and the store fills with stale state
nobody removes. Memory is not a bucket — it is a system with a metabolism.
**The shift:** a storer optimizes what a system remembers; a memory engineer
optimizes what it forgets. The problem was never that an agent forgets — it is
that it never forgets *on purpose*.
## The four lenses
| Lens | Question | The finding that hurts |
|---|---|---|
| **Stanford** | What does remembering cost? | Construction energy exceeds total query energy across 300 queries. The tuned half is the smaller half. |
| **Microsoft** | What is worth keeping? | More raw memory can make an agent *worse*. Keep facts and skills; drop the events. |
| **Anthropic** | Who controls what it keeps? | A wrong memory does not fail once — it persists into every future session that reads it. |
| **Nvidia** | Where does it hit hardware? | It is all KV cache in HBM. Construction is prefill-heavy and stalls the query a user is waiting on. |
## Workflow
```bash
# 1 - Price it first. Never quote a quality number without a cost number.
python scripts/memory_cost_profiler.py --print-sample-spec > workload.json
python scripts/memory_cost_profiler.py --spec workload.json
# 2 - Pick which cost to pay. No "best" verdict; on a tie it asks, exit 2.
python scripts/memory_architecture_picker.py --constraints workload.json
# 3 - Audit what the store actually holds (skip if greenfield).
python scripts/memory_density_auditor.py --dir ~/.claude/memory
# 4 - Gate on forgetting. Exit 4 is a stop, not a suggestion.
python scripts/forgetting_policy_linter.py --policy design.json
# 5 - No command. Prove each pass by hand before scheduling it.
```
Step 1 reports the construction/query split, **cost per correct answer**, and
amortization — if construction dominates, cut construction tokens *before*
touching retrieval. Step 2 names the cost the winning family makes you pay.
Step 3 classifies records FACT / SKILL / LOG / PROSE (`LOG-HEAVY` = archiving
events; `PROSE-HEAVY` = docs, not memory).
Step 4 is the gate: **F1** (explicit forgetting rule) and **F4** (contradictions
surfaced, never auto-merged) are blocking. Retrofitting forgetting onto two
years of records is a migration nobody does; auto-merging disagreeing memories
destroys the evidence the conflict existed.
Step 5 has no script — prove each pass by hand, then automate. Run it once
against real history and ask whether it changed a decision. If not, scheduling
it only makes noise. Ship order: `forgetting_policy_design.md` §7.
## Hard rules
1. **Never quote accuracy without cost per correct answer.**
2. **Never return a "best" memory system** — name the cost the choice makes you pay.
3. **Never auto-merge contradictions.** The system surfaces; the human decides.
4. **Never call a design done without a forgetting rule.** No evaluated system provides one by default.
5. **Never schedule a pass not yet run by hand.**
6. **Report findings as findings.** A non-zero exit is a result to surface, not an error to swallow.
7. **Attribute every number** with its confidence level. Vendor customer figures are testimonials, not benchmarks.
## Scripts
| Script | Role | Exit codes |
|---|---|---|
| `scripts/memory_cost_profiler.py` | Construction vs query split, cost per correct answer, amortization, co-location warning | 0 · 2 finding · 3 bad input |
| `scripts/memory_architecture_picker.py` | Scores 4 families, disqualifies, names the cost, refuses to pick on a tie | 0 · 2 ambiguous · 3 bad input · 4 none viable |
| `scripts/memory_density_auditor.py` | FACT/SKILL/LOG/PROSE, duplicates, staleness, density (`--dir` or `--jsonl`) | 0 dense · 2 finding · 3 bad input |
| `scripts/forgetting_policy_linter.py` | The gate: 8 checks, F1 and F4 blocking | 0 PASS · 2 CONDITIONAL · 4 FAIL |
All support `--output json` and `--sample` (no input file needed).
## References and assets
- [`references/memory_cost_canon.md`](references/memory_cost_canon.md) — construction dominance, energy per correct answer, the four families, ten recommendations (7 sources)
- [`references/what_to_keep.md`](references/what_to_keep.md) — PlugMem and MEMENTO: facts over logs, density over volume (7 sources)
- [`references/memory_control_and_governance.md`](references/memory_control_and_governance.md) — memory as files, scope/audit/rollback, poisoning, reading vendor numbers (7 sources)
- [`references/forgetting_policy_design.md`](references/forgetting_policy_design.md) — forgetting mechanisms, contradiction discipline, KV cache, ship order (7 sources)
- [`assets/memory_engineer_worksheet.md`](assets/memory_engineer_worksheet.md) — seven forcing questions with recommended answers + citations; walk one at a time
- [`assets/memory_design_spec.example.json`](assets/memory_design_spec.example.json) — one file covering every script's input
- [`assets/forgetting_policy_template.md`](assets/forgetting_policy_template.md) — fillable policy covering F1–F8
## Provenance
Framing from *"How to be a Memory Engineer"* by [@N01ennn](https://x.com/N01ennn/status/2083971749079581120); every
number is cited to a primary source instead, and two paraphrases are corrected — `memory_cost_canon.md` §2, `memory_control_and_governance.md` §4.

136
engineering/memory-engineering/skills/memory-engineering/assets/forgetting_policy_template.md Normal file
View file

@ -0,0 +1,136 @@
# Forgetting Policy — <system name>
> Fill this in **before** the store grows. Every section maps to a check in
> `forgetting_policy_linter.py`. F1 and F4 are blocking: a design that fails
> either is not ready, regardless of how good its retrieval is.
>
> Copy the JSON block at the bottom into your own file and lint it.
**System:** ____________________ **Named owner:** ____________________
**Store location:** ____________________ **Reviewed:** ____________
---
## F1 — Explicit forgetting rule (BLOCKING)
*No evaluated memory system prunes or forgets by default. If it is not written
here, the store only grows.*
Choose at least one:
- [ ] **TTL** — records expire after `______` days
- [ ] **Capacity bound** — max `______` records / `______` bytes
- Eviction order: ____________________________________
- *(The eviction order IS the policy. "LRU" is a decision, not a default —
recency is a poor proxy for value, and the fact retrieved once a year is
often the one you cannot reconstruct.)*
- [ ] **Relevance decay** — score decays unless retrieved; expire below `______`
**Exempt from forgetting** (records that must never expire), and why:
```
________________________________________________________
```
## F2 — Dedup at write time
- [ ] Every write is fingerprinted and near-matches collapsed
- Method: ____________________ Threshold: ____________
*Duplicates do not merely waste tokens — they let a stale copy outrank a
corrected one.*
## F3 — Consolidation / compaction
- [ ] Enabled Cadence: ____________
- What merges into what: ______________________________
- [ ] Merges preserve the source list of every merged record
*Warning: consolidation is lossy, and it amplifies. A merge pass propagates a
wrong (or poisoned) record into derived records, past the point where source
attribution helps.*
## F4 — Contradiction handling (BLOCKING)
- [ ] Contradictions are **surfaced to a human** with both versions, both
sources, and both timestamps
- Who resolves them: ____________________
- Where they surface: ____________________
**Explicitly forbidden:** `auto_merge`, `newest_wins`, `overwrite`,
`last_write_wins`.
*Two memories that disagree may both have been true in different contexts.
"Deploys go through Jenkins" and "deploys go through GitHub Actions" is not a
contradiction to resolve — it is a migration to record.*
## F5 — Scope
| | Who | Notes |
|---|---|---|
| **Read** | ______________ | |
| **Write** | ______________ | |
- [ ] Read scope and write scope are different
- [ ] Shared / org-wide stores are read-only
- [ ] Components ingesting untrusted content do **not** hold write scope
## F6 — Audit trail
- [ ] Every write carries a timestamp
- [ ] Every write carries source attribution
- Where audit events are visible: ____________________
*Without attribution you can detect a bad fact but not find its origin — which
means you cannot stop it recurring.*
## F7 — Rollback and delete
- [ ] Earlier versions can be restored
- [ ] A single record can be hard-deleted without a migration
- [ ] Content can be **redacted from history** (distinct from version rollback,
and what erasure obligations such as GDPR Art. 17 actually require)
- Delete path: ____________________ Typical time to delete: __________
## F8 — Growth-slope monitoring
- [ ] Footprint over time is tracked, not just current size
- Alert threshold (slope): ____________________
- Current baseline: ____________ Current slope: ____________/month
*Slope, not starting size, is what bankrupts a long-lived agent.*
---
## Lint this
```json
{
"name": "<system name>",
"forgetting": {
"rule": "ttl",
"ttl_days": 365,
"max_records": 50000,
"decay": "none"
},
"dedup_on_write": true,
"consolidation": { "enabled": true, "cadence": "weekly" },
"contradiction_policy": "surface",
"scope": {
"read": ["<who reads>"],
"write": ["<who writes>"]
},
"audit_trail": { "timestamp": true, "source_attribution": true },
"rollback": { "supported": true, "delete_path": "api" },
"growth_monitoring": { "tracks_slope": true, "baseline_only": false }
}
```
```bash
python scripts/forgetting_policy_linter.py --policy my_policy.json
# exit 0 = PASS · 2 = CONDITIONAL · 4 = FAIL (F1 or F4 failed)
```
**Ship order reminder** — build the write path first and let it fill; add
contradiction detection by hand a few times; add this policy *before* volume
climbs; tune the hardware layer last.

71
engineering/memory-engineering/skills/memory-engineering/assets/memory_design_spec.example.json Normal file
View file

@ -0,0 +1,71 @@
{
"_comment": "One spec covering every script in the memory-engineering skill. Each top-level block is consumed by a different script; pass this same file to all of them.",
"_usage": {
"memory_cost_profiler.py": "--spec (reads: name, construction, query, pricing_usd_per_mtok, accuracy)",
"memory_architecture_picker.py": "--constraints (reads: name, *_sensitivity/_pressure/_need/growth, hard_constraints)",
"forgetting_policy_linter.py": "--policy (reads: name, forgetting, dedup_on_write, consolidation, contradiction_policy, scope, audit_trail, rollback, growth_monitoring)",
"memory_density_auditor.py": "takes --dir or --jsonl instead of this file"
},
"name": "support-agent memory store (worked example)",
"construction": {
"records_per_day": 400,
"prompt_tokens_per_record": 12000,
"output_tokens_per_record": 800,
"embedding_tokens_per_record": 12000,
"colocated_with_queries": true
},
"query": {
"queries_per_day": 1200,
"prompt_tokens_per_query": 2400,
"output_tokens_per_query": 300,
"embedding_tokens_per_query": 40
},
"pricing_usd_per_mtok": {
"prompt": 3.0,
"output": 15.0,
"embedding": 0.02
},
"accuracy": 0.72,
"query_latency_sensitivity": "high",
"build_budget_pressure": "medium",
"recall_need": "high",
"volume_growth": "high",
"mutability_need": "medium",
"hard_constraints": {
"max_p99_query_ms": 1500,
"context_window_tokens": 200000,
"expected_history_tokens": 4000000
},
"forgetting": {
"rule": "ttl",
"ttl_days": 365,
"max_records": 50000,
"decay": "none"
},
"dedup_on_write": true,
"consolidation": {
"enabled": true,
"cadence": "weekly"
},
"contradiction_policy": "surface",
"scope": {
"read": [
"support-agents"
],
"write": [
"memory-writer-service"
]
},
"audit_trail": {
"timestamp": true,
"source_attribution": true
},
"rollback": {
"supported": true,
"delete_path": "api"
},
"growth_monitoring": {
"tracks_slope": false,
"baseline_only": true
}
}

171
engineering/memory-engineering/skills/memory-engineering/assets/memory_engineer_worksheet.md Normal file
View file

@ -0,0 +1,171 @@
# Memory Engineer Worksheet — the seven forcing questions
Walk these **one at a time**, in order. Each has a recommended answer and a
citation. Do not batch them; the answer to one changes the framing of the next.
Fill this in before writing any memory code. If a question cannot be answered,
that is the finding — stop and go get the answer.
---
**System under review:** ________________________________________
**Date:** ____________ **Owner (a named human):** ____________________
---
## 1. What does one constructed record cost, and how many queries will it serve?
*Why it matters:* Construction energy exceeds total query energy across 300
queries for LLM-mediated memory. The invisible half of the bill is usually the
bigger half.
*Recommended answer:* Under roughly 10 queries per record, build **lazily on
second access** rather than eagerly at the end of every session.
*Citation:* Stanford rec. 4 — `memory_cost_canon.md` §6
*Check with:* `memory_cost_profiler.py`
**Your answer:**
```
cost per record: ______ queries per record: ______
decision: ____________________________________________
```
---
## 2. Which of build cost, query speed, and accuracy are you giving up?
*Why it matters:* No paradigm family wins all three. Refusing to name the
sacrifice does not avoid it — it just means it gets discovered in production.
*Recommended answer:* Name it explicitly and write it down here, so the next
person does not re-litigate it.
*Citation:* Stanford taxonomy — `memory_cost_canon.md` §3
*Check with:* `memory_architecture_picker.py`
**Your answer:**
```
family chosen: ________________________________________
cost accepted: ________________________________________
what would kill this choice: __________________________
```
---
## 3. Is this record a fact, a skill, or an event?
*Why it matters:* Agents do not need to replay what happened; they need the
facts and skills extracted from it. Storing events is what makes retrieval
drown.
*Recommended answer:* Keep facts and skills. Extract from events, then drop the
events.
*Citation:* PlugMem — `what_to_keep.md` §1
*Check with:* `memory_density_auditor.py`
**Your answer:**
```
current FACT/SKILL/LOG/PROSE split: ___________________
extraction happens at: [ ] write time [ ] read time [ ] not at all
```
---
## 4. When two stored memories disagree, what happens?
*Why it matters:* Two memories that disagree may both have been true in
different contexts. Anything automatic destroys the evidence that a conflict
existed — and the conflict is usually the interesting part.
*Recommended answer:* Surface both versions with sources and timestamps to a
human. `newest_wins` and `auto_merge` are not policies, they are defaults
nobody chose.
*Citation:* `forgetting_policy_design.md` §3 — linter check **F4 (blocking)**
**Your answer:**
```
contradiction policy: _________________________________
who resolves it: ______________________________________
```
---
## 5. Who can write to this store, and can you delete one record without a migration?
*Why it matters:* A wrong memory does not fail once — it persists into every
future session that reads it. And anything that influences what the agent reads
can influence what it permanently believes.
*Recommended answer:* Separate read scope from write scope; shared stores
read-only. Keep a hard-delete path you can invoke without a migration.
*Citation:* Anthropic — `memory_control_and_governance.md` §3, §5
*Linter checks:* F5, F6, F7
**Your answer:**
```
read scope: ___________________________________________
write scope: __________________________________________
delete one record without a migration? [ ] yes [ ] no
untrusted-content ingestion holds write scope? [ ] yes [ ] no
```
---
## 6. What leaves the store, and on what rule?
*Why it matters:* None of the evaluated memory systems prunes or forgets by
default. If you did not build it, you do not have it — and retrofitting it onto
a full store is a migration nobody ever does.
*Recommended answer:* Choose now, while the store is small: a TTL, a capacity
bound **with a stated eviction order**, or relevance decay.
*Citation:* Stanford — `forgetting_policy_design.md` §1, §2 — linter check
**F1 (blocking)**
**Your answer:**
```
mechanism: [ ] TTL ____d [ ] capacity ______ [ ] decay [ ] NONE (blocked)
eviction order (if capacity): _________________________
```
---
## 7. Are you tracking footprint growth *slope*, or only current size?
*Why it matters:* At 1M tokens, footprint already varies up to 9× across
systems. Growth slope, not starting size, is what bankrupts a long-lived agent —
and agentic systems compound as the store itself grows.
*Recommended answer:* Track the slope and alert on it.
*Citation:* Stanford rec. 9 — `memory_cost_canon.md` §4 — linter check F8
**Your answer:**
```
slope tracked? [ ] yes [ ] no (baseline only)
alert threshold: ______________________________________
```
---
## Sign-off
- [ ] `memory_cost_profiler.py` run; cost per correct answer recorded
- [ ] `memory_architecture_picker.py` run; the accepted cost is named above
- [ ] `memory_density_auditor.py` run; FACT/SKILL/LOG/PROSE split recorded
- [ ] `forgetting_policy_linter.py` exits 0 or 2 — **never 4**
- [ ] Every scheduled pass was run by hand first and changed a decision
**Named owner:** ______________________ **Date:** ____________

184
engineering/memory-engineering/skills/memory-engineering/references/forgetting_policy_design.md Normal file
View file

@ -0,0 +1,184 @@
# Forgetting Policy Design — and where memory hits the hardware
> The keystone, plus the Nvidia lens: *how do you forget on purpose, and where
> does all of it land on the GPU?*
---
## 1. Why forgetting must be designed before the store grows
The single most consequential finding for anyone shipping agent memory:
> "None of the evaluated systems prune or forget by default, so footprint grows
> monotonically under default behavior."
> — Omri et al., arXiv:2606.06448 (confidence: **high**)
Read that as an engineering instruction: **if you did not build forgetting, you
do not have it.** Not from your vector database, not from your memory
framework, not from your agent SDK.
And the cost of retrofitting is asymmetric. Adding a forgetting policy to an
empty store is a config decision. Adding one to a store with two years of
accumulated records is a data-migration project with a judgment call attached to
every record — which is why it never happens, and why the store keeps growing.
**Slope beats baseline.** The paper's recommendation 9 is to evaluate both
baseline footprint *and* cost growth slope. A store that starts at 2 GB and
grows 1%/month is healthier than one starting at 200 MB and growing 40%/month.
Agentic systems (Paradigm IV) compound worst, because the store itself becomes
input to the next construction pass.
## 2. The four forgetting mechanisms
Pick at least one. They compose.
| Mechanism | Rule | Best when | Failure mode |
|---|---|---|---|
| **TTL** | Expire after N days | Facts with a natural shelf life (prices, staffing, config) | Deletes a still-true fact nobody restated |
| **Capacity bound** | Cap records/bytes, evict by policy | Hard budget, predictable cost | Eviction order becomes the real policy — LRU evicts rare-but-critical facts |
| **Relevance decay** | Score decays unless retrieved; expire below threshold | Retrieval frequency correlates with value | Self-reinforcing: never-retrieved-because-never-surfaced records die |
| **Consolidation** | Merge N related records into one denser record | Many thin records on one topic | Lossy merges destroy the distinctions that mattered |
**On eviction order.** If you use a capacity bound, the eviction policy *is*
your forgetting policy — "LRU" is a real design decision, not a default. Recency
is a poor proxy for value in memory systems: the fact you look up once a year is
often the one you cannot reconstruct.
## 3. Never auto-merge contradictions
This is check **F4** in the linter, and it is blocking.
When two stored memories disagree, the tempting resolutions are all wrong:
- **`newest_wins`** — assumes recency implies correctness. Often it means the
newest session was confused.
- **`auto_merge`** — produces a record that says something neither source said.
- **`overwrite` / `last_write_wins`** — the storage layer's default, chosen by
nobody, silently destroying the evidence that a conflict existed.
**Two memories that disagree may both have been true in different contexts.**
"Deploys go through Jenkins" and "deploys go through GitHub Actions" are not a
contradiction to resolve — they are a migration to record, and the interesting
information is *when it changed and why*.
The rule: **the system surfaces, the human decides.** Surface both versions with
their sources and timestamps. A contradiction is a signal that your model of the
world is out of date, which is exactly the signal you do not want auto-resolved.
## 4. Where all of this lands on the hardware
Strip away the algorithms and every memory decision becomes a GPU decision.
**Keeping full history in context is quadratic, not merely slow.** Attention
cost grows with the square of sequence length, so doubling retained history
roughly quadruples the attention work.
**Prefix caching saves you within a session and collapses across sessions.**
The KV cache that makes turn 40 cheap in one conversation does not carry to
tomorrow's conversation. Inter-session memory is precisely the case where the
cache does not help — which is why "just keep it in context" degrades from a
cost problem into a feasibility problem at session boundaries.
**The scarce resource is KV cache in high-bandwidth memory.** A memory engineer
should be able to state their system in these units: HBM bandwidth, GPU
utilization, tokens/second, and KV slots freed. Under every clever retrieval
scheme, the real currency is cache.
MEMENTO makes the connection concrete: compressing reasoning blocks into
summaries cut peak KV cache ~2.5× and improved vLLM throughput ~1.75×
(arXiv:2604.09852, confidence: **high**). Forgetting *is* a throughput
optimization.
## 5. Construction is a background job
Construction is almost pure prefill — long reads in, short writes out — so it
behaves like a background indexing job, not like a user request.
**Co-locate it with live queries and a large write will stall the scheduler
exactly when a user query arrives.** Prefill saturates the GPU; a query that
lands behind a big construction batch waits for it.
The controls, per the paper's recommendations 3, 6, 8 and 10:
1. **Rate-limit** construction with admission control.
2. **Batch** writes rather than constructing per-session inline.
3. **Defer** construction off the latency-sensitive path entirely.
4. **Cap** LLM-bounded retrieval loops — worst-case latency is a selection
criterion, and agentic systems have no natural bound without one.
`memory_cost_profiler.py` flags `CONSTRUCTION_COLOCATED` for exactly this.
## 6. Prove each pass by hand before you schedule it
Before automating any memory pass — extraction, consolidation, contradiction
detection, expiry — run it once, by hand, against your real history.
Ask of the output: **did this change a decision?** If yes, it earns a schedule.
If no, scheduling it just generates noise you will learn to ignore.
**A memory system run against three notes will hallucinate connections that are
not there.** Sparse input produces confident spurious structure, and the
experience of being wrong early trains you to distrust the system permanently.
Let the store fill with real material first.
## 7. Ship order
The sequence matters, because each step's output is the next step's input:
1. **Build the write path first** — storing facts and skills, not logs — and let
it fill for a few weeks so there is real material to work with.
2. **Add contradiction detection by hand**, a few times. Schedule it only if the
collisions surprise you.
3. **Add the forgetting and maintenance policy before volume climbs.** This is
the step that is easy now and impossible later.
4. **Tune the hardware layer last**, once volume is real: batch construction,
cap retrieval, watch the KV cache.
Do not schedule everything on day one. Get one manual run reliable, wrap it,
then automate it.
## 8. The eight checks
What `forgetting_policy_linter.py` enforces:
| ID | Check | Blocking |
|---|---|---|
| F1 | Explicit forgetting rule (TTL, capacity, or decay) | **Yes** |
| F2 | Dedup at write time | No |
| F3 | Consolidation / compaction | No |
| F4 | Contradictions surfaced, never auto-merged | **Yes** |
| F5 | Read scope and write scope both named | No |
| F6 | Audit trail (timestamp + source attribution) | No |
| F7 | Rollback and delete path | No |
| F8 | Growth-slope monitoring | No |
F1 and F2–F8 come from the Stanford recommendations and the Anthropic control
surface. F4 is blocking because it is the one failure that silently destroys
information rather than merely accumulating it.
---
## Sources
1. Omri, Y. et al. — *Agent Memory: Characterization and System Implications of
Stateful Long-Horizon Workloads*, arXiv:2606.06448.
<https://arxiv.org/abs/2606.06448> — §1, §5, §8.
2. Kontonis, V. et al. — *MEMENTO: Teaching LLMs to Manage Their Own Context*,
arXiv:2604.09852. <https://arxiv.org/abs/2604.09852> — §4.
3. Kwon, W. et al. — *Efficient Memory Management for Large Language Model
Serving with PagedAttention* (vLLM), SOSP 2023. KV cache paging and prefix
reuse mechanics behind §4.
4. Dao, T. et al. — *FlashAttention* (2022) and *FlashAttention-2* (2023). The
quadratic-attention cost structure referenced in §4.
5. Ebbinghaus, H. — *Über das Gedächtnis* (1885), and Bjork, R. A. & Bjork,
E. L. — *A New Theory of Disuse* (1992). The retrieval-strength/decay model
behind relevance decay in §2.
6. Anthropic — *Built-in memory for Claude Managed Agents*.
<https://claude.com/blog/claude-managed-agents-memory> — §8 controls F5–F7.
7. Denning, P. J. — *The Working Set Model for Program Behavior* (1968), and the
LRU/LFU cache-eviction literature. Prior art for §2's eviction-order warning.
## Related tools in this skill
- `forgetting_policy_linter.py` — §2, §3, §8
- `memory_cost_profiler.py` — §5

165
engineering/memory-engineering/skills/memory-engineering/references/memory_control_and_governance.md Normal file
View file

@ -0,0 +1,165 @@
# Memory Control and Governance — who controls what it keeps
> The Anthropic lens: *who controls what it keeps?*
>
> Every figure is attributed with a confidence level. The customer metrics in
> §4 are **vendor-published testimonials**, not controlled studies, and are
> labeled as such.
---
## 1. A wrong memory does not fail once
This is the asymmetry that makes memory governance different from ordinary
storage governance:
**A bad retrieval fails one query. A bad memory fails every future session that
reads it.**
An error written into memory is not a transient defect — it is a persistent
one that compounds, propagates into derived records during consolidation, and
is retrieved with the same confidence as a correct one. By the time you notice,
it has been read hundreds of times and possibly rewritten into summaries.
This is why control is not a layer added on top of a memory system. It is a
property the system either has by construction or does not have at all.
## 2. Memory as files — the deliberately boring move
Anthropic's design for Claude Managed Agents mounts memory as **files on a
filesystem**, at `/mnt/memory/` inside the agent's container, so the agent reads
and writes memory with the same bash and code-execution tools it already uses.
> Memory is "a workspace-scoped collection of text documents mounted as a
> directory," relying on "the same bash and code execution capabilities that
> make it effective at agentic tasks."
> — Anthropic, *Built-in memory for Claude Managed Agents*
> (confidence: **high**)
The choice looks unambitious, and that is the argument for it. Files give you,
for free, every property a bespoke store has to reimplement:
| Property | What files give you |
|---|---|
| Inspection | `cat`, `grep`, a text editor |
| Diffing | `diff`, version control |
| Export | copy the directory |
| Selective deletion | `rm` one file |
| Programmatic control | any language's file API |
| Review | a human reads it without a query language |
**The test:** *a store you cannot open and edit is a store you do not control.*
If answering "what does the agent believe about X, and why?" requires writing a
query against an embedding index, you have given up observability to gain
retrieval convenience.
## 3. Scope, audit, rollback
The three controls, as implemented in the Anthropic design (confidence:
**high** — all quoted from vendor documentation):
**Scope.** Access is set per store: `read_only` makes the mount read-only at
the filesystem level; `read_write` allows create, edit, and delete. An org-wide
store is typically read-only while per-user stores are writable — so shared
knowledge cannot be corrupted by one agent's bad session. Multiple agents can
work concurrently against the same store without overwriting each other.
**Audit.** "Each write becomes a session event with a timestamp, source
attribution, and a rollback option." Session events surface in the Console.
This is what makes a wrong memory traceable to the session that wrote it —
without attribution, you can detect a bad fact but not find its origin, which
means you cannot stop it recurring.
**Rollback.** Earlier versions can be restored, and content can be redacted
from history. Note the second half: *redaction from history* is a distinct
capability from *restoring a prior version*, and regimes like GDPR's right to
erasure require the former.
These map to checks **F5** (scope), **F6** (audit trail), and **F7** (rollback)
in `forgetting_policy_linter.py`.
## 4. Reported outcomes — read the label
Vendor-published customer testimonials accompanying the Anthropic memory
launch:
| Source | Reported result |
|---|---|
| Rakuten (Yusuke Kaji, GM AI for Business) | "97% fewer first-pass errors" at "27% lower cost and 34% lower latency" |
| Wisedocs (Denys Linkov, Head of ML) | Memory use "sped verification up 30%" |
**Confidence: low-to-moderate, and the reason matters.** These are named,
attributable, on-the-record customer statements — which is better than an
anonymous benchmark — but they are:
- **not controlled experiments** (no stated baseline methodology, no control arm);
- **selected for publication** by the vendor;
- **not isolated to memory** (a team that adds memory usually changes other
things at the same time).
⚠️ A widely-shared summary of this material presents the 97% figure as though it
were a general property of building memory this way — "teams building this way
cut first-pass errors by 97 percent." That overstates it. It is *one named
customer's reported result*, not a generalizable finding. Cite it as Rakuten's
claim, with the attribution attached, or do not cite it.
The defensible version of the claim is the mechanism, not the number:
**observable learning is debuggable learning.** When every write is attributed
and reversible, you can find and fix a bad memory instead of discovering it as
unexplained model drift.
## 5. Memory poisoning is a security boundary, not just a quality problem
If an agent writes to memory based on content it reads, then **anything that
can influence what the agent reads can influence what it permanently believes.**
A prompt injection that lands in a memory record does not end with the session
— it persists and is retrieved as trusted context later.
Minimum controls:
1. **Separate write authority from read exposure.** The component that ingests
untrusted content should not be the component with write scope.
2. **Attribute every write to a source**, so injected records can be traced and
swept.
3. **Keep the delete path fast.** Incident response on a poisoned memory store
is bounded by how quickly you can remove records.
4. **Treat consolidation as an amplifier.** A merge pass propagates a poisoned
record into derived records, past the point where source attribution helps.
## 6. The governance checklist
Before a memory system holds anything that matters:
- [ ] Read scope and write scope are both named, and they differ
- [ ] Every write carries a timestamp and a source
- [ ] There is a restore path *and* a hard-delete path
- [ ] Deletion satisfies your regulatory obligations (erasure, not just tombstoning)
- [ ] A human can read the store without a query language
- [ ] Untrusted-content ingestion does not hold write scope
- [ ] Contradictions surface to a human rather than resolving silently
---
## Sources
1. Anthropic — *Built-in memory for Claude Managed Agents*.
<https://claude.com/blog/claude-managed-agents-memory> — primary source for
§2, §3, §4.
2. `anthropics/skills` — `skills/claude-api/shared/managed-agents-memory.md`.
Implementation-level reference for scopes and store semantics.
<https://github.com/anthropics/skills>
3. Anthropic — memory tool documentation, Claude Developer Platform. Reference
for the filesystem-backed memory tool surface.
4. OWASP — *Top 10 for LLM Applications*, notably LLM01 (Prompt Injection) and
the agentic-memory poisoning discussion. Basis for §5.
5. NIST — *AI Risk Management Framework* (AI 100-1), MAP and MEASURE functions.
Governance framing for auditability requirements.
6. Regulation (EU) 2016/679 (GDPR), Article 17 — right to erasure. Why redaction
from history is a distinct requirement from version rollback.
7. Anthropic — *Building effective agents* (2024). Design context for
tool-mediated agent state.
## Related tools in this skill
- `forgetting_policy_linter.py` — §3 (F5 scope, F6 audit, F7 rollback),
§6 (F4 contradictions)

154
engineering/memory-engineering/skills/memory-engineering/references/memory_cost_canon.md Normal file
View file

@ -0,0 +1,154 @@
# Memory Cost Canon — what remembering actually costs
> The Stanford lens: *what does remembering cost?*
>
> Every figure below is attributed and carries a confidence level. Where the
> popular summary of this research differs from the paper, the paper wins and
> the difference is called out.
---
## 1. The finding that reorders your priorities
**Construction, not retrieval, dominates the lifecycle cost of LLM-mediated
agent memory.**
> "For LLM-mediated agent memory systems, construction energy exceeds total
> query-phase energy across 300 queries."
> — Omri et al., *Agent Memory: Characterization and System Implications of
> Stateful Long-Horizon Workloads*, arXiv:2606.06448 (confidence: **high** —
> direct quote, measured with a phase-aware profiling harness)
This is uncomfortable because query latency is the number you watch: the user
feels it, your dashboard graphs it, your on-call pages on it. Construction is
invisible — it happens after the session, on a background worker, and nobody
has an SLO for it.
The practical consequence: **tuning retrieval on a write-heavy memory system is
optimizing the smaller half of the bill.**
## 2. Normalize by correct answers, never by accuracy alone
Accuracy hides cost. Two systems can report the same benchmark score while
differing by more than an order of magnitude in energy per useful result.
Measured spread across the ten evaluated systems:
| System | Energy per correct answer | Multiple vs baseline |
|---|---|---|
| BM25 (lexical baseline) | 4,145 J | 1× |
| A-Mem | ~115 kJ | ~28× |
| MIRIX | ~197 kJ | ~47× |
> "The spread across agent memory systems exceeds 47×."
> — Omri et al., arXiv:2606.06448 (confidence: **high**)
⚠️ **Correction to a widely-shared summary.** A popular thread describing this
paper states that *"two systems with identical accuracy split by 47 times."*
That is not what the paper reports. The 47× is the **spread across the ten
evaluated systems** (BM25 baseline vs. MIRIX), and A-Mem/MIRIX are described as
a "28–47× premium" — not a pair matched on accuracy. The directional lesson
survives intact and is still the right one: *quote quality and cost together,
always.* But do not cite the 47× as an accuracy-matched comparison.
## 3. The four paradigm families
The paper's taxonomy classifies systems along four axes — **construction,
storage, retrieval, and mutability** — yielding four families:
| Paradigm | Family | Evaluated systems |
|---|---|---|
| I | Long-context memory | `long_context` |
| II | Flat RAG memory | BM25, EmbedRAG |
| III | Structure-augmented RAG | GraphRAG, HippoRAG v2, Mem0, SimpleMem |
| IV | Agentic control flow | Letta, MIRIX, A-Mem |
(confidence: **high** — taxonomy and system list quoted directly)
**No family wins on all three of construction time, query latency, and
accuracy.** The paper is explicit that "no single system is therefore best on
all three axes." This is why `memory_architecture_picker.py` never returns a
"best" — it returns the family that fits your constraints and names the cost
that choice makes you pay.
## 4. Footprint grows monotonically, because nothing forgets
> "At 1M tokens, footprint varies by up to 9× across systems... None of the
> evaluated systems prune or forget by default, so footprint grows
> monotonically under default behavior."
> — Omri et al., arXiv:2606.06448 (confidence: **high**)
Two lessons, in order of importance:
1. **Forgetting is not a feature any of these systems gives you.** If you did
not build it, you do not have it. This is the entire justification for
`forgetting_policy_linter.py` treating a missing forgetting rule as a
blocking failure rather than a warning.
2. **Judge growth slope, not baseline footprint.** A store that starts small
with a steep slope bankrupts you later than one that starts large and flat —
but it still bankrupts you, and it does so after you have built on it.
## 5. The ten system recommendations
Paraphrased from the paper (confidence: **high** on existence and substance,
**moderate** on exact wording):
1. Treat system selection as a systems-level decision, beyond accuracy.
2. Account for energy across the full agent lifecycle.
3. Treat construction as background throughput with admission control.
4. Exploit reuse across overlapping inputs.
5. Treat the minimum viable construction LLM as an algorithm-imposed cost floor.
6. Match the cost split to the workload's query arrival pattern.
7. Treat construction time as a hard feasibility constraint for inter-session
workloads.
8. Make construction cadence system-aware.
9. Evaluate both baseline footprint and cost growth slope.
10. Treat worst-case latency as a selection criterion; LLM-bounded systems need
caps.
Recommendations 3, 6, 9 and 10 are the ones this skill's tools enforce
mechanically.
## 6. Amortization: the question behind recommendation 4
A constructed record has to be read enough times to justify what it cost to
write. If your agent writes 400 records a day and serves 1,200 queries, each
record serves 3 queries on average — you are paying to remember things nobody
asks about.
`memory_cost_profiler.py` flags this below 10 queries per record. The floor is
a heuristic chosen for this skill, not a number from the paper (confidence:
**low** on the specific threshold, **high** on the principle).
The fix is usually *lazy construction*: build the memory record on second
access rather than eagerly at the end of every session.
---
## Sources
1. Omri, Y., Gan, Z., Broveak, Z., Geens, R., He, Z., Pentland, A., Verhelst,
M., Weissman, T., Tambe, T. — *Agent Memory: Characterization and System
Implications of Stateful Long-Horizon Workloads*, arXiv:2606.06448 (2026).
<https://arxiv.org/abs/2606.06448> — the primary source for §1–§5.
2. MemoryAgentBench — the benchmark suite used for the characterization,
evaluating accurate retrieval, test-time learning, long-range understanding,
and selective forgetting.
3. Robertson, S. & Zaragoza, H. — *The Probabilistic Relevance Framework: BM25
and Beyond* (2009). Establishes the lexical baseline that turns out to be
the energy-efficiency floor in the Stanford run.
4. Kwon, W. et al. — *Efficient Memory Management for Large Language Model
Serving with PagedAttention* (vLLM), SOSP 2023. The serving substrate the
hardware findings sit on.
5. Chase, H. et al. — Mem0 / LangMem / Letta system documentation. Primary
sources for the Paradigm III and IV systems named above.
6. Gao, Y. et al. — *Retrieval-Augmented Generation for Large Language Models:
A Survey*, arXiv:2312.10997. Background for the Paradigm II family.
7. Anthropic — *Effective context engineering for AI agents* (2025).
Practitioner framing of context as a finite, priced resource.
## Related tools in this skill
- `memory_cost_profiler.py` — §1, §2, §6
- `memory_architecture_picker.py` — §3
- `forgetting_policy_linter.py` — §4 (checks F1 and F8)

159
engineering/memory-engineering/skills/memory-engineering/references/what_to_keep.md Normal file
View file

@ -0,0 +1,159 @@
# What To Keep — facts and skills, not logs
> The Microsoft lens: *what is worth keeping?*
>
> Every figure is attributed with a confidence level.
---
## 1. More memory can make an agent worse
The uncomfortable premise of Microsoft Research's **PlugMem**: giving an agent
more raw memory does not monotonically help it. History piles up, retrieval
drowns in near-misses, and the agent burns attention wading through transcripts
for the one line that mattered.
> PlugMem "distinguishes between remembering events, knowing facts, and knowing
> how to perform tasks, with effective decisions relying on the facts and skills
> extracted from those events."
> — Microsoft Research, *PlugMem: A Task-Agnostic Plugin Memory Module for LLM
> Agents* (confidence: **high**)
The structure borrowed here is the classic memory taxonomy from cognitive
psychology — Tulving's split between **episodic** memory (what happened) and
**semantic** memory (what is true), with **procedural** memory (how to do it) as
the third leg. Humans do not replay episodes to make decisions; we act on the
semantic and procedural residue we distilled from them.
**The engineering translation:** your write path's job is *extraction*, not
*archival*. If your memory system stores what happened, you built a log with a
vector index on it.
## 2. Density is the metric, not volume
> PlugMem "enables agents to achieve better results while using significantly
> fewer memory tokens, with efficiency measured by the utility of the
> information delivered relative to the context consumed," reporting "consistent
> gains over generic retrieval and task-specific memory designs across three
> benchmarks while consuming less of the agent's context window."
> — Microsoft Research (confidence: **high** on the directional claim;
> **moderate** on magnitude, as per-benchmark numbers vary)
So the metric to optimize is:
```
decision-relevant information delivered
───────────────────────────────────────
tokens of context it costs
```
Not "how many records did we store." `memory_density_auditor.py` approximates
the numerator by counting FACT and SKILL records, and the denominator by
estimated tokens.
⚠️ **A note on a related claim.** A widely-shared summary states PlugMem "cuts
context by up to 100×." Microsoft's own materials describe consistent gains at
lower context cost without foregrounding that multiple. Treat any specific
compression multiple as **low confidence** unless you read it in the paper's
results table for your own workload shape.
## 3. Memento — the model manages its own context
Microsoft's **MEMENTO** pushes context management inside the model rather than
bolting orchestration around it. The model learns to segment its reasoning into
blocks, compress each block into a dense "memento" summary, drop the full block
from context, and reason forward attending only to the mementos.
Measured results (confidence: **high** — reported in the paper):
| Metric | Result |
|---|---|
| Memento size target | 15–25% of original block tokens |
| Peak KV cache | ~2.5× reduction (paper reports 2–3× peak memory) |
| Throughput (vLLM) | ~1.75× improvement |
| Training data | OpenMementos — 228K reasoning traces |
Two things a memory engineer should take from it:
**First, this is a learned skill, not an orchestration layer.** It comes from
ordinary fine-tuning on segmented traces. You cannot get it by wrapping a model
in a summarizer loop.
**Second — and this is the subtle one — forgetting is not deletion.**
> "Information from each reasoning block is carried both by the memento text
> and by corresponding KV states, which retain implicit information from the
> original block — removing this channel drops accuracy by 15 percentage points
> on AIME24."
> — *MEMENTO: Teaching LLMs to Manage Their Own Context*, arXiv:2604.09852
> (confidence: **high**)
A shadow of the erased reasoning survives in the KV states. Rebuilding context
from the summary text *alone* costs 15 points. The lesson generalizes beyond
Memento: **a summary is not equivalent to what it summarizes**, and any
architecture that assumes "we distilled it, so we can drop the original" should
measure that assumption rather than trust it.
## 4. The three record types, and how to tell them apart
`memory_density_auditor.py` classifies every record into one of four buckets.
The classification is lexical and deliberately conservative.
| Type | What it is | Signal | Keep? |
|---|---|---|---|
| **FACT** | A declarative truth about the world | `X is Y`, `key: value`, versions, owners, endpoints | Yes — this is semantic memory |
| **SKILL** | A procedure or a rule | numbered steps, `always/never`, `to X, do Y` | Yes — this is procedural memory |
| **LOG** | A record of an event | speaker turns, timestamps, first-person past tense | Extract from it, then drop it |
| **PROSE** | Narrative with no signal either way | none of the above | Distill or move to docs |
**Why PROSE exists as a category.** An earlier version of this classifier
labeled every signal-less block as LOG, which fired the log-heavy finding on any
prose-shaped documentation store — a false positive severe enough to make the
tool untrustworthy. Narrative documentation is neither an event log nor a
retrievable fact; it deserves its own verdict and its own fix (distill it, or
move it out of the memory store).
**Why LOG wins ties.** Mistaking an event for knowledge is the costly direction
of error. A false LOG label costs you a review; a false FACT label puts a
transcript into the retrieval path forever.
## 5. What this means for your write path
Ordered by leverage:
1. **Extract at write time, not read time.** The whole point of paying for
construction is that the thinking already happened when the query arrives.
2. **Store the conclusion with its provenance, not the conversation.** "Billing
is owned by the payments team (source: 2026-03 handover doc)" beats the
thread where that was worked out.
3. **Write the fact so it can go stale detectably.** "Postgres 14 as of
2026-03" beats "currently on Postgres." Time-relative wording rots silently;
`memory_density_auditor.py` flags it as `VOLATILE_WORDING`.
4. **Prefer one dense record to five thin ones.** Consolidation is check F3 in
the forgetting linter for exactly this reason.
---
## Sources
1. Microsoft Research — *PlugMem: A Task-Agnostic Plugin Memory Module for LLM
Agents*.
<https://www.microsoft.com/en-us/research/publication/plugmem-a-task-agnostic-plugin-memory-module-for-llm-agents/>
2. Microsoft Research Blog — *From raw interaction to reusable knowledge:
rethinking memory for AI agents*.
<https://www.microsoft.com/en-us/research/blog/from-raw-interaction-to-reusable-knowledge-rethinking-memory-for-ai-agents/>
3. Kontonis, V. et al. — *MEMENTO: Teaching LLMs to Manage Their Own Context*,
arXiv:2604.09852. <https://arxiv.org/abs/2604.09852>
4. `microsoft/OpenMementos` — the 228K-trace public dataset released with
MEMENTO. <https://huggingface.co/datasets/microsoft/OpenMementos>
5. Tulving, E. — *Episodic and Semantic Memory* (1972), and *Elements of
Episodic Memory* (1983). The episodic/semantic distinction PlugMem borrows.
6. Anderson, J. R. — *ACT-R* and the declarative/procedural memory split.
Source of the fact-versus-skill distinction used in the classifier.
7. Anthropic — *Effective context engineering for AI agents* (2025). The
context-as-finite-budget framing behind the density metric.
## Related tools in this skill
- `memory_density_auditor.py` — §2, §4
- `forgetting_policy_linter.py` — §5 (checks F2 dedup, F3 consolidation)

420
engineering/memory-engineering/skills/memory-engineering/scripts/forgetting_policy_linter.py Normal file
View file

@ -0,0 +1,420 @@
#!/usr/bin/env python3
"""Refuse a memory design that has no forgetting policy.
This is the gate. Everything else in this skill measures a memory system; this
one blocks it.
Stanford's characterization found that none of the ten evaluated memory systems
prunes or forgets by default, so footprint grows monotonically -- and at 1M
tokens, footprint already varies by up to 9x across systems. Growth slope, not
starting size, is what bankrupts a long-lived agent.
Two rules are non-negotiable, and either one failing fails the whole design:
F1 an explicit forgetting rule must exist (TTL, capacity bound, or decay)
F4 contradictions must be SURFACED to a human, never auto-merged
F4 is not fussiness. Two memories that disagree may both have been true in
different contexts. A system that silently merges them destroys the only
evidence that the conflict existed.
The remaining checks (F2, F3, F5-F8) degrade the verdict to CONDITIONAL rather
than failing it.
Deterministic policy audit. No LLM calls, no network, stdlib only.
Exit codes:
0 PASS -- forgetting is designed
2 CONDITIONAL -- forgetting exists but the controls around it are thin
3 invalid input
4 FAIL -- no forgetting rule, or contradictions are auto-merged
"""
import argparse
import json
import sys
# Recognized `forgetting.rule` values. Anything outside this set is treated as
# no rule at all -- see _check_forgetting_rule.
KNOWN_FORGETTING_RULES = {"ttl", "capacity", "decay", "none", "never", ""}
VALID_CONTRADICTION_POLICIES = {"surface", "surface_to_human", "flag", "escalate"}
AUTO_MERGE_POLICIES = {"auto_merge", "newest_wins", "overwrite", "last_write_wins"}
SAMPLE_POLICY = {
"name": "support-agent memory store",
"forgetting": {
"rule": "ttl",
"ttl_days": 365,
"max_records": 50000,
"decay": "none",
},
"dedup_on_write": True,
"consolidation": {"enabled": True, "cadence": "weekly"},
"contradiction_policy": "surface",
"scope": {"read": ["support-agents"], "write": ["memory-writer-service"]},
"audit_trail": {"timestamp": True, "source_attribution": True},
"rollback": {"supported": True, "delete_path": "api"},
"growth_monitoring": {"tracks_slope": False, "baseline_only": True},
}
FAILING_SAMPLE = {
"name": "naive vector store (the default everyone ships)",
"forgetting": {"rule": "none"},
"dedup_on_write": False,
"consolidation": {"enabled": False},
"contradiction_policy": "newest_wins",
"scope": {},
"audit_trail": {},
"rollback": {"supported": False},
"growth_monitoring": {},
}
def _fail(message: str) -> None:
print(f"error: {message}", file=sys.stderr)
raise SystemExit(3)
def _check_forgetting_rule(policy: dict) -> dict:
"""F1 -- pass only on a concrete, named forgetting mechanism.
This check is allowlist-based on purpose. An earlier version failed only
when `rule` was literally "none"/""/"never" and inferred PASS from what the
rule was *not*, so a typo ("asdf") or a declared-but-unconfigured rule
("ttl" with ttl_days omitted) fell through to PASS with an empty mechanism
list -- the blocking gate this whole skill is built around, defeated by a
misspelling. PASS is now unreachable unless a mechanism is actually found.
"""
forgetting = policy.get("forgetting") or {}
if not isinstance(forgetting, dict):
_fail("'forgetting' must be an object")
rule = str(forgetting.get("rule", "none")).lower().strip()
ttl_days = forgetting.get("ttl_days")
has_ttl = isinstance(ttl_days, (int, float)) and not isinstance(
ttl_days, bool
) and ttl_days > 0
capacity = forgetting.get("max_records") or forgetting.get("max_bytes")
has_capacity = isinstance(capacity, (int, float)) and not isinstance(
capacity, bool
) and capacity > 0
decay = str(forgetting.get("decay", "none")).lower().strip()
has_decay = rule == "decay" or decay not in {"none", ""}
mechanisms = []
if has_ttl:
mechanisms.append(f"TTL {ttl_days}d")
if has_capacity:
mechanisms.append(f"capacity bound ({capacity})")
if has_decay:
mechanisms.append("relevance decay")
if mechanisms:
return {
"id": "F1",
"name": "explicit forgetting rule",
"status": "PASS",
"blocking": True,
"detail": "Forgetting is designed: " + ", ".join(mechanisms) + ".",
"fix": None,
}
# No mechanism found. Say precisely why, so a typo is not mistaken for a
# deliberate "we decided not to forget".
if rule not in KNOWN_FORGETTING_RULES:
detail = (
f"Unrecognized forgetting rule {rule!r}. Recognized values are "
f"{sorted(KNOWN_FORGETTING_RULES)}. An unrecognized rule is treated "
"as no rule -- a misspelling must never read as a policy."
)
elif rule in {"ttl", "capacity", "decay"}:
detail = (
f"Rule is declared as {rule!r} but carries no usable parameter "
"(ttl_days > 0, max_records/max_bytes > 0, or a decay setting). "
"A declared rule with nothing configured forgets exactly as much "
"as no rule at all."
)
else:
detail = (
"No TTL, no capacity bound, no decay. The store only grows. This "
"is the default behaviour of every system Stanford evaluated, and "
"it is the one thing that makes a long-lived agent unaffordable."
)
return {
"id": "F1",
"name": "explicit forgetting rule",
"status": "FAIL",
"blocking": True,
"detail": detail,
"fix": (
"Pick one before the store gets big: a TTL (ttl_days), a hard "
"record/byte cap with an eviction order, or a relevance decay that "
"expires unreferenced records."
),
}
def _check_contradictions(policy: dict) -> dict:
raw = str(policy.get("contradiction_policy", "")).lower().strip()
if raw in AUTO_MERGE_POLICIES:
return {
"id": "F4",
"name": "contradictions surfaced, never auto-merged",
"status": "FAIL",
"blocking": True,
"detail": (
f"Contradiction policy is '{raw}', which resolves conflicts "
"silently. Two memories that disagree may both have been true "
"in different contexts; auto-merging destroys the evidence "
"that the conflict existed."
),
"fix": (
"Change the policy to surface the conflict to a human with "
"both versions and their sources. The system surfaces, the "
"human decides."
),
}
if raw not in VALID_CONTRADICTION_POLICIES:
return {
"id": "F4",
"name": "contradictions surfaced, never auto-merged",
"status": "FAIL",
"blocking": True,
"detail": (
f"No contradiction policy declared (got '{raw or 'nothing'}'). "
"Undeclared means whatever the storage layer does by default, "
"which is almost always last-write-wins."
),
"fix": (
"Declare 'contradiction_policy': 'surface' and build the "
"surfacing path before the store holds conflicting facts."
),
}
return {
"id": "F4",
"name": "contradictions surfaced, never auto-merged",
"status": "PASS",
"blocking": True,
"detail": f"Contradiction policy is '{raw}' -- a human resolves conflicts.",
"fix": None,
}
def _simple_check(
check_id: str, name: str, ok: bool, detail_ok: str, detail_bad: str, fix: str
) -> dict:
return {
"id": check_id,
"name": name,
"status": "PASS" if ok else "WARN",
"blocking": False,
"detail": detail_ok if ok else detail_bad,
"fix": None if ok else fix,
}
def lint(policy: dict) -> dict:
if not isinstance(policy, dict):
_fail("policy must be a JSON object")
scope = policy.get("scope") or {}
audit = policy.get("audit_trail") or {}
rollback = policy.get("rollback") or {}
consolidation = policy.get("consolidation") or {}
growth = policy.get("growth_monitoring") or {}
checks = [
_check_forgetting_rule(policy),
_simple_check(
"F2",
"dedup at write time",
bool(policy.get("dedup_on_write")),
"Duplicates are collapsed on the way in.",
"No write-time dedup. Duplicates let a stale copy outrank a corrected one.",
"Hash or fingerprint each record at write and collapse near-matches.",
),
_simple_check(
"F3",
"consolidation / compaction",
bool(consolidation.get("enabled")),
f"Consolidation runs ({consolidation.get('cadence', 'cadence unspecified')}).",
"Nothing compacts many small records into fewer dense ones.",
"Schedule a consolidation pass that merges related records into one denser record.",
),
_check_contradictions(policy),
_simple_check(
"F5",
"read and write scope",
bool(scope.get("read")) and bool(scope.get("write")),
"Read and write scopes are both named.",
"Read scope, write scope, or both are undeclared -- anything can write anything.",
"Name who may read and who may write. An org-wide store should usually be read-only.",
),
_simple_check(
"F6",
"audit trail",
bool(audit.get("timestamp")) and bool(audit.get("source_attribution")),
"Every write carries a timestamp and a source.",
"Writes are not fully attributed, so a wrong memory cannot be traced to its origin.",
"Record timestamp plus source attribution on every write.",
),
_simple_check(
"F7",
"rollback and delete path",
bool(rollback.get("supported")),
"Earlier versions can be restored and records can be deleted.",
"No rollback. A wrong memory persists into every future session that reads it.",
"Add a restore path and a hard-delete path you can invoke without a migration.",
),
_simple_check(
"F8",
"growth-slope monitoring",
bool(growth.get("tracks_slope")),
"Growth slope is tracked, not just current size.",
"Only baseline size is watched. Slope, not starting size, is what bankrupts the store.",
"Track footprint over time and alert on the slope, per Stanford recommendation 9.",
),
]
blocking_failures = [c for c in checks if c["blocking"] and c["status"] == "FAIL"]
warnings = [c for c in checks if c["status"] == "WARN"]
if blocking_failures:
verdict = "FAIL"
exit_code = 4
summary = (
"This design does not forget on purpose. "
+ " ".join(f"{c['id']} failed." for c in blocking_failures)
)
elif warnings:
verdict = "CONDITIONAL"
exit_code = 2
summary = (
f"Forgetting is designed, but {len(warnings)} control"
f"{'s are' if len(warnings) != 1 else ' is'} missing around it."
)
else:
verdict = "PASS"
exit_code = 0
summary = "Forgetting is designed and the controls around it are in place."
return {
"name": policy.get("name", "unnamed memory design"),
"verdict": verdict,
"exit_code": exit_code,
"summary": summary,
"passed": sum(1 for c in checks if c["status"] == "PASS"),
"total": len(checks),
"checks": checks,
}
def render(report: dict) -> str:
lines = []
lines.append(f"FORGETTING POLICY LINT - {report['name']}")
lines.append("=" * 68)
lines.append(f"VERDICT: {report['verdict']} ({report['passed']}/{report['total']} checks pass)")
lines.append(f"{report['summary']}")
lines.append("")
for check in report["checks"]:
blocking = " [BLOCKING]" if check["blocking"] else ""
lines.append(
f" {check['status']} {check['id']} {check['name']}{blocking}"
)
lines.append(f" {check['detail']}")
if check["fix"]:
lines.append(f" -> {check['fix']}")
lines.append("")
if report["verdict"] == "FAIL":
lines.append(
"Blocked. A storer optimizes what a system remembers; a memory\n"
"engineer optimizes what it forgets. Fix the blocking checks before\n"
"this design goes near real volume."
)
return "\n".join(lines)
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Audit a memory design against eight forgetting-policy checks. "
"Fails the design if it has no forgetting rule or auto-merges "
"contradictions."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Examples:\n"
" forgetting_policy_linter.py --sample\n"
" forgetting_policy_linter.py --sample-failing\n"
" forgetting_policy_linter.py --policy design.json --output json\n"
),
)
# Not required=True: argparse enforces a required group during
# parse_args(), which made --print-sample-spec unreachable on its own.
# Validated explicitly after the print-and-exit branch instead.
source = parser.add_mutually_exclusive_group(required=False)
source.add_argument("--policy", help="path to a memory design policy JSON file")
source.add_argument(
"--sample",
action="store_true",
help="lint the built-in passing sample policy",
)
source.add_argument(
"--sample-failing",
action="store_true",
help="lint a policy with no forgetting rule (shows what the gate blocks)",
)
parser.add_argument(
"--output",
choices=["text", "json"],
default="text",
help="output format (default: text)",
)
parser.add_argument(
"--print-sample-spec",
action="store_true",
help="print the sample policy JSON and exit (use as a template)",
)
args = parser.parse_args()
if args.print_sample_spec:
print(json.dumps(SAMPLE_POLICY, indent=2))
return 0
if not (args.policy or args.sample or args.sample_failing):
parser.error(
"one of --policy, --sample, --sample-failing, or --print-sample-spec is required"
)
if args.sample:
policy = SAMPLE_POLICY
elif args.sample_failing:
policy = FAILING_SAMPLE
else:
try:
with open(args.policy, "r", encoding="utf-8") as handle:
policy = json.load(handle)
except FileNotFoundError:
_fail(f"policy file not found: {args.policy}")
except json.JSONDecodeError as exc:
_fail(f"policy file is not valid JSON: {exc}")
report = lint(policy)
if args.output == "json":
print(json.dumps(report, indent=2))
else:
print(render(report))
return report["exit_code"]
if __name__ == "__main__":
sys.exit(main())

412
engineering/memory-engineering/skills/memory-engineering/scripts/memory_architecture_picker.py Normal file
View file

@ -0,0 +1,412 @@
#!/usr/bin/env python3
"""Pick which memory cost to pay on purpose -- there is no best system.
The Stanford characterization (arXiv:2606.06448) evaluated ten memory systems
across four paradigm families and found no single system wins on construction
time, query latency, and accuracy at once. So this tool never returns "best".
It returns the family that fits the stated constraints and names, explicitly,
the cost that choice makes you pay.
The four families, and the taxonomy axes they are scored on (construction,
storage, retrieval, mutability):
long_context Paradigm I -- keep raw history in the window
flat_rag Paradigm II -- BM25 / embedding retrieval over chunks
structured_rag Paradigm III -- GraphRAG / Mem0-style extraction
agentic Paradigm IV -- Letta / MIRIX-style agentic control flow
When the top two families score within AMBIGUITY_MARGIN of each other, the tool
refuses to pick and names the one question that would break the tie.
Deterministic scoring only. No LLM calls, no network, stdlib only.
Exit codes:
0 a single family fits the constraints
2 ambiguous -- two families tie; a tie-breaking question is printed
3 invalid input
4 no family satisfies a hard constraint
"""
import argparse
import json
import sys
AMBIGUITY_MARGIN = 0.06
# Per-family profile. Scores are 0-1 where higher is better on that axis.
# Values encode the qualitative ordering reported in the Stanford
# characterization, not measurements from any one deployment.
FAMILIES = {
"long_context": {
"label": "Paradigm I - Long-context memory",
"examples": "raw history in the context window",
"build_cheapness": 1.00, # no construction at all
"query_speed": 0.25, # quadratic attention over the whole history
"recall_quality": 0.55, # strong within window, cliff-edge outside it
"scales_with_volume": 0.10, # hard context ceiling
"mutability": 0.20, # cannot selectively update or delete
"cost_you_pay": (
"Query latency and a hard ceiling. Cheap to build because there is "
"no write path, but cost grows quadratically with history and "
"prefix caching collapses across sessions."
),
"kills_it": "history that outgrows the context window",
},
"flat_rag": {
"label": "Paradigm II - Flat RAG memory",
"examples": "BM25, EmbedRAG",
"build_cheapness": 0.85, # indexing only, no LLM in the write path
"query_speed": 0.80,
"recall_quality": 0.55, # blunt: no relation between chunks
"scales_with_volume": 0.85,
"mutability": 0.75, # re-index a document, but no fact-level edit
"cost_you_pay": (
"Precision. Builds almost instantly and stays cheap, but retrieval "
"is blunt -- it returns chunks that mention the topic, not the "
"fact that answers the question."
),
"kills_it": "questions whose answer spans several documents",
},
"structured_rag": {
"label": "Paradigm III - Structure-augmented RAG",
"examples": "GraphRAG, Mem0, HippoRAG v2",
"build_cheapness": 0.25, # LLM-mediated extraction on every write
"query_speed": 0.90, # small, pre-distilled payload at query time
"recall_quality": 0.85,
"scales_with_volume": 0.70,
"mutability": 0.80, # fact-level update and delete
"cost_you_pay": (
"Construction. Answers fast because the thinking already happened "
"at write time -- which is exactly why the write path can cost "
"more than every query it will ever serve."
),
"kills_it": "a write budget that cannot absorb an LLM call per record",
},
"agentic": {
"label": "Paradigm IV - Agentic control flow",
"examples": "Letta, MIRIX, A-Mem",
"build_cheapness": 0.10, # multi-step agent loop per write
"query_speed": 0.45, # unbounded LLM-mediated retrieval loop
"recall_quality": 0.90,
"scales_with_volume": 0.45, # footprint compounds as the store grows
"mutability": 0.95, # the agent can rewrite its own memory
"cost_you_pay": (
"Everything except recall. Highest energy per correct answer in "
"the Stanford run, worst-case query latency is unbounded without "
"an explicit cap, and footprint compounds as the store grows."
),
"kills_it": "a hard p99 latency SLO, or an unbounded footprint budget",
},
}
# Which constraint drives which axis, and how heavily.
WEIGHTS = {
"query_latency_sensitivity": ("query_speed", 0.28),
"build_budget_pressure": ("build_cheapness", 0.26),
"recall_need": ("recall_quality", 0.24),
"volume_growth": ("scales_with_volume", 0.12),
"mutability_need": ("mutability", 0.10),
}
LEVELS = {"low": 0.15, "medium": 0.5, "high": 1.0}
# Keyed by the two tied families. Lookup normalizes the pair with sorted(), so
# these keys are normalized too -- see the guard below. Authored in whatever
# order reads naturally; order is not significant.
_TIE_BREAKERS_RAW = {
("flat_rag", "structured_rag"): (
"Does a correct answer usually require joining facts that live in "
"different sessions? If yes, pay for structured extraction. If a single "
"well-retrieved chunk normally answers it, stay flat."
),
("structured_rag", "agentic"): (
"Does your memory need to correct itself without a human in the loop? "
"If yes, go agentic and cap its retrieval loop. If a human reviews "
"corrections, structured extraction is cheaper for the same recall."
),
("long_context", "flat_rag"): (
"Will the history exceed the context window within your planning "
"horizon? If yes, build the retrieval path now -- migrating later costs "
"a full re-index. If not, raw context is free."
),
("long_context", "structured_rag"): (
"Is the value in the raw transcript or in the facts extracted from it? "
"If a human would summarize before reusing it, extract at write time."
),
}
# Normalize every key to its sorted form, because the lookup below sorts the
# tied pair. Without this, a key authored in the other order is unreachable and
# the tool silently falls back to the generic question -- no error, just a worse
# answer. This repo has no test suite, so the collision check runs at import.
TIE_BREAKERS = {}
for _pair, _question in _TIE_BREAKERS_RAW.items():
_key = tuple(sorted(_pair))
if _key in TIE_BREAKERS:
raise AssertionError(
f"duplicate tie-breaker for {_key} after normalization -- two "
"entries describe the same family pair"
)
TIE_BREAKERS[_key] = _question
del _pair, _question, _key
SAMPLE_CONSTRAINTS = {
"name": "customer-support agent, 18-month retention",
"query_latency_sensitivity": "high",
"build_budget_pressure": "medium",
"recall_need": "high",
"volume_growth": "high",
"mutability_need": "medium",
"hard_constraints": {
"max_p99_query_ms": 1500,
"context_window_tokens": 200000,
"expected_history_tokens": 4000000,
},
}
def _fail(message: str) -> None:
print(f"error: {message}", file=sys.stderr)
raise SystemExit(3)
def _level(constraints: dict, key: str) -> float:
raw = constraints.get(key, "medium")
if isinstance(raw, (int, float)) and not isinstance(raw, bool):
value = float(raw)
if not 0.0 <= value <= 1.0:
_fail(f"'{key}' as a number must be in [0, 1], got {value}")
return value
if not isinstance(raw, str) or raw.lower() not in LEVELS:
_fail(f"'{key}' must be one of {sorted(LEVELS)} or a number in [0, 1]")
return LEVELS[raw.lower()]
def _hard_constraint_kills(family, hard):
"""Return a disqualifying reason, or None if the family survives."""
window = hard.get("context_window_tokens")
history = hard.get("expected_history_tokens")
if family == "long_context" and window and history and history > window:
return (
f"expected history ({history:,} tokens) exceeds the context window "
f"({window:,} tokens) -- raw context cannot hold it"
)
p99 = hard.get("max_p99_query_ms")
if family == "agentic" and p99 and p99 < 3000:
return (
f"p99 budget of {p99} ms cannot absorb an uncapped agentic "
"retrieval loop"
)
if family == "long_context" and p99 and p99 < 1000:
return (
f"p99 budget of {p99} ms cannot absorb quadratic attention over "
"full history"
)
return None
def pick(constraints: dict) -> dict:
if not isinstance(constraints, dict):
_fail("constraints must be a JSON object")
hard = constraints.get("hard_constraints", {})
if not isinstance(hard, dict):
_fail("'hard_constraints' must be an object")
levels = {key: _level(constraints, key) for key in WEIGHTS}
scored = []
disqualified = []
for name, profile in FAMILIES.items():
reason = _hard_constraint_kills(name, hard)
if reason:
disqualified.append(
{"family": name, "label": profile["label"], "reason": reason}
)
continue
score = 0.0
breakdown = {}
for constraint_key, (axis, weight) in WEIGHTS.items():
contribution = levels[constraint_key] * weight * profile[axis]
breakdown[axis] = round(contribution, 4)
score += contribution
scored.append(
{
"family": name,
"label": profile["label"],
"examples": profile["examples"],
"score": round(score, 4),
"breakdown": breakdown,
"cost_you_pay": profile["cost_you_pay"],
"kills_it": profile["kills_it"],
}
)
if not scored:
return {
"name": constraints.get("name", "unnamed workload"),
"verdict": "NO-VIABLE-FAMILY",
"ranked": [],
"disqualified": disqualified,
"recommendation": None,
"tie_breaker": None,
}
scored.sort(key=lambda item: item["score"], reverse=True)
winner = scored[0]
tie_breaker = None
verdict = "RECOMMENDED"
if len(scored) > 1:
runner_up = scored[1]
gap = winner["score"] - runner_up["score"]
if gap < AMBIGUITY_MARGIN:
verdict = "AMBIGUOUS"
pair = tuple(sorted([winner["family"], runner_up["family"]]))
tie_breaker = {
"between": [winner["label"], runner_up["label"]],
"gap": round(gap, 4),
"question": TIE_BREAKERS.get(
pair,
"Which cost hurts you more in production: a slow write path "
"or a blunt retrieval result? Answer that and the choice "
"resolves.",
),
}
return {
"name": constraints.get("name", "unnamed workload"),
"verdict": verdict,
"ranked": scored,
"disqualified": disqualified,
"recommendation": None if verdict == "AMBIGUOUS" else winner,
"tie_breaker": tie_breaker,
}
def render(result: dict) -> str:
lines = []
lines.append(f"MEMORY ARCHITECTURE - {result['name']}")
lines.append("=" * 68)
lines.append(f"VERDICT: {result['verdict']}")
lines.append("")
if result["verdict"] == "NO-VIABLE-FAMILY":
lines.append("Every family was disqualified by a hard constraint.")
lines.append("Relax a hard constraint, or split the workload in two.")
lines.append("")
elif result["recommendation"]:
rec = result["recommendation"]
lines.append(f"Use: {rec['label']} ({rec['examples']})")
lines.append("")
lines.append("The cost you are choosing to pay:")
lines.append(f" {rec['cost_you_pay']}")
lines.append("")
lines.append(f"What would kill this choice: {rec['kills_it']}")
lines.append("")
else:
tie = result["tie_breaker"]
lines.append(f"Too close to call ({tie['gap']:.3f} apart):")
for label in tie["between"]:
lines.append(f" - {label}")
lines.append("")
lines.append("Answer this before picking:")
lines.append(f" {tie['question']}")
lines.append("")
if result["ranked"]:
lines.append("Ranked")
lines.append("-" * 68)
for index, item in enumerate(result["ranked"], start=1):
lines.append(f" {index}. {item['label']:<42} {item['score']:.4f}")
lines.append("")
if result["disqualified"]:
lines.append("Disqualified by hard constraints")
lines.append("-" * 68)
for item in result["disqualified"]:
lines.append(f" {item['label']}")
lines.append(f" {item['reason']}")
lines.append("")
lines.append(
"No family wins on build cost, query speed, and accuracy at once.\n"
"This is a choice of which cost to pay, not a ranking of quality."
)
return "\n".join(lines)
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Recommend an agent memory paradigm from stated constraints, and "
"name the cost that choice makes you pay."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Examples:\n"
" memory_architecture_picker.py --sample\n"
" memory_architecture_picker.py --constraints workload.json\n"
" memory_architecture_picker.py --sample --output json\n"
),
)
# Not required=True: argparse enforces a required group during
# parse_args(), which made --print-sample-spec unreachable on its own.
# Validated explicitly after the print-and-exit branch instead.
source = parser.add_mutually_exclusive_group(required=False)
source.add_argument("--constraints", help="path to a constraints JSON file")
source.add_argument(
"--sample",
action="store_true",
help="run against the built-in sample constraints",
)
parser.add_argument(
"--output",
choices=["text", "json"],
default="text",
help="output format (default: text)",
)
parser.add_argument(
"--print-sample-spec",
action="store_true",
help="print the sample constraints JSON and exit (use as a template)",
)
args = parser.parse_args()
if args.print_sample_spec:
print(json.dumps(SAMPLE_CONSTRAINTS, indent=2))
return 0
if not (args.constraints or args.sample):
parser.error(
"one of --constraints, --sample, or --print-sample-spec is required"
)
if args.sample:
constraints = SAMPLE_CONSTRAINTS
else:
try:
with open(args.constraints, "r", encoding="utf-8") as handle:
constraints = json.load(handle)
except FileNotFoundError:
_fail(f"constraints file not found: {args.constraints}")
except json.JSONDecodeError as exc:
_fail(f"constraints file is not valid JSON: {exc}")
result = pick(constraints)
if args.output == "json":
print(json.dumps(result, indent=2))
else:
print(render(result))
if result["verdict"] == "NO-VIABLE-FAMILY":
return 4
if result["verdict"] == "AMBIGUOUS":
return 2
return 0
if __name__ == "__main__":
sys.exit(main())

379
engineering/memory-engineering/skills/memory-engineering/scripts/memory_cost_profiler.py Normal file
View file

@ -0,0 +1,379 @@
#!/usr/bin/env python3
"""Price the write path of an agent memory system, not just the query path.
The Stanford characterization (arXiv:2606.06448) found that for LLM-mediated
memory systems, construction energy exceeds total query-phase energy across 300
queries. Everyone watches query latency because the user feels it; the bill is
paid at construction, which the user never sees.
This tool splits a memory workload into its construction and query phases,
computes cost per *correct* answer (never accuracy alone), and reports the
amortization ratio -- how many queries each constructed record has to serve
before the write that produced it pays for itself.
Deterministic arithmetic only. No LLM calls, no network, stdlib only.
Exit codes:
0 profile produced, no blocking finding
2 actionable finding (write-path dominant, under-amortized, or
construction co-located with latency-sensitive queries)
3 invalid input
"""
import argparse
import json
import sys
# Cost split beyond which construction is judged to dominate the lifecycle.
WRITE_DOMINANT_SHARE = 0.50
# Below this many queries per constructed record, the write path has not been
# amortized (Stanford recommendation 4: exploit reuse; recommendation 6: match
# the cost split to the workload's query arrival pattern).
MIN_AMORTIZATION_QUERIES = 10.0
SAMPLE_SPEC = {
"name": "support-agent-memory (structure-augmented RAG)",
"construction": {
"records_per_day": 400,
"prompt_tokens_per_record": 12000,
"output_tokens_per_record": 800,
"embedding_tokens_per_record": 12000,
"colocated_with_queries": True,
},
"query": {
"queries_per_day": 1200,
"prompt_tokens_per_query": 2400,
"output_tokens_per_query": 300,
"embedding_tokens_per_query": 40,
},
"pricing_usd_per_mtok": {
"prompt": 3.0,
"output": 15.0,
"embedding": 0.02,
},
"accuracy": 0.72,
}
def _fail(message: str) -> None:
print(f"error: {message}", file=sys.stderr)
raise SystemExit(3)
def _number(container: dict, key: str, where: str, *, default=None) -> float:
if key not in container:
if default is not None:
return float(default)
_fail(f"missing required field '{key}' in {where}")
value = container[key]
if isinstance(value, bool) or not isinstance(value, (int, float)):
_fail(f"field '{key}' in {where} must be a number, got {type(value).__name__}")
if value < 0:
_fail(f"field '{key}' in {where} must be >= 0, got {value}")
return float(value)
def _phase_cost(
prompt_tokens: float,
output_tokens: float,
embedding_tokens: float,
pricing: dict,
) -> dict:
"""Cost of one phase, in USD, broken out by token class."""
per_mtok = 1_000_000.0
prompt_cost = prompt_tokens / per_mtok * pricing["prompt"]
output_cost = output_tokens / per_mtok * pricing["output"]
embedding_cost = embedding_tokens / per_mtok * pricing["embedding"]
return {
"prompt_tokens": round(prompt_tokens, 2),
"output_tokens": round(output_tokens, 2),
"embedding_tokens": round(embedding_tokens, 2),
"prompt_usd": round(prompt_cost, 6),
"output_usd": round(output_cost, 6),
"embedding_usd": round(embedding_cost, 6),
"total_usd": round(prompt_cost + output_cost + embedding_cost, 6),
}
def profile(spec: dict) -> dict:
if not isinstance(spec, dict):
_fail("spec must be a JSON object")
construction = spec.get("construction")
query = spec.get("query")
if not isinstance(construction, dict):
_fail("spec must contain a 'construction' object")
if not isinstance(query, dict):
_fail("spec must contain a 'query' object")
pricing_raw = spec.get("pricing_usd_per_mtok", {})
if not isinstance(pricing_raw, dict):
_fail("'pricing_usd_per_mtok' must be an object")
pricing = {
"prompt": _number(pricing_raw, "prompt", "pricing_usd_per_mtok", default=3.0),
"output": _number(pricing_raw, "output", "pricing_usd_per_mtok", default=15.0),
"embedding": _number(
pricing_raw, "embedding", "pricing_usd_per_mtok", default=0.02
),
}
records = _number(construction, "records_per_day", "construction")
queries = _number(query, "queries_per_day", "query")
build = _phase_cost(
records * _number(construction, "prompt_tokens_per_record", "construction"),
records * _number(construction, "output_tokens_per_record", "construction"),
records
* _number(
construction, "embedding_tokens_per_record", "construction", default=0
),
pricing,
)
read = _phase_cost(
queries * _number(query, "prompt_tokens_per_query", "query"),
queries * _number(query, "output_tokens_per_query", "query"),
queries * _number(query, "embedding_tokens_per_query", "query", default=0),
pricing,
)
total = build["total_usd"] + read["total_usd"]
build_share = build["total_usd"] / total if total > 0 else 0.0
accuracy = spec.get("accuracy")
if accuracy is None:
_fail(
"spec must contain 'accuracy' (0-1). Cost per correct answer is the "
"whole point -- a quality number without a cost number is the "
"measurement this tool exists to refuse."
)
accuracy = float(accuracy)
if not 0.0 < accuracy <= 1.0:
_fail(f"'accuracy' must be in (0, 1], got {accuracy}")
cost_per_query = total / queries if queries > 0 else 0.0
cost_per_correct = cost_per_query / accuracy
amortization = queries / records if records > 0 else float("inf")
colocated = bool(construction.get("colocated_with_queries", False))
findings = []
if build_share > WRITE_DOMINANT_SHARE:
findings.append(
{
"code": "WRITE_PATH_DOMINANT",
"severity": "high",
"detail": (
f"Construction is {build_share:.0%} of daily spend "
f"(${build['total_usd']:.2f} build vs "
f"${read['total_usd']:.2f} query). The cost you tuned is not "
"the cost you pay."
),
"action": (
"Cut construction tokens before touching retrieval: batch "
"writes, dedup before extraction, or drop to a cheaper "
"construction model."
),
}
)
if amortization < MIN_AMORTIZATION_QUERIES:
findings.append(
{
"code": "UNDER_AMORTIZED",
"severity": "high",
"detail": (
f"Each constructed record serves only {amortization:.1f} "
f"queries (floor {MIN_AMORTIZATION_QUERIES:.0f}). You are "
"paying to remember things nobody asks about."
),
"action": (
"Write less, or write later: build memory lazily on second "
"access rather than eagerly on every session."
),
}
)
if colocated:
findings.append(
{
"code": "CONSTRUCTION_COLOCATED",
"severity": "medium",
"detail": (
"Construction shares a scheduler with latency-sensitive "
"queries. Construction is prefill-heavy, so a large write "
"stalls exactly the query a user is waiting on."
),
"action": (
"Treat construction as a background job with admission "
"control: rate-limit, batch, or defer it off the "
"latency-sensitive path."
),
}
)
# The verdict names the actual dominant problem, so it can never disagree
# with the findings list below it.
codes = {f["code"] for f in findings}
if "WRITE_PATH_DOMINANT" in codes:
verdict = "WRITE-PATH-DOMINANT"
elif "UNDER_AMORTIZED" in codes:
verdict = "UNDER-AMORTIZED"
elif "CONSTRUCTION_COLOCATED" in codes:
verdict = "NEEDS-SCHEDULING-FIX"
elif build_share < 0.15:
verdict = "QUERY-DOMINANT"
else:
verdict = "BALANCED"
return {
"name": spec.get("name", "unnamed memory system"),
"verdict": verdict,
"daily_cost_usd": {
"construction": build["total_usd"],
"query": read["total_usd"],
"total": round(total, 6),
"construction_share": round(build_share, 4),
},
"construction_phase": build,
"query_phase": read,
"quality_and_cost": {
"accuracy": accuracy,
"cost_per_query_usd": round(cost_per_query, 6),
"cost_per_correct_answer_usd": round(cost_per_correct, 6),
"note": (
"Never quote accuracy without cost per correct answer. Two "
"systems at identical accuracy can differ by more than an "
"order of magnitude on this number."
),
},
"amortization": {
"queries_per_constructed_record": (
round(amortization, 2) if amortization != float("inf") else None
),
"floor": MIN_AMORTIZATION_QUERIES,
},
"findings": findings,
}
def render(report: dict) -> str:
lines = []
lines.append(f"MEMORY COST PROFILE - {report['name']}")
lines.append("=" * 68)
lines.append(f"VERDICT: {report['verdict']}")
lines.append("")
cost = report["daily_cost_usd"]
lines.append("Daily cost split")
lines.append("-" * 68)
lines.append(
f" construction ${cost['construction']:>10.2f} "
f"({cost['construction_share']:.0%} of total)"
)
lines.append(
f" query ${cost['query']:>10.2f} "
f"({1 - cost['construction_share']:.0%} of total)"
)
lines.append(f" total ${cost['total']:>10.2f}")
lines.append("")
qc = report["quality_and_cost"]
lines.append("Quality AND cost (never one without the other)")
lines.append("-" * 68)
lines.append(f" accuracy {qc['accuracy']:.1%}")
lines.append(f" cost per query ${qc['cost_per_query_usd']:.6f}")
lines.append(
f" cost per CORRECT answer ${qc['cost_per_correct_answer_usd']:.6f}"
)
lines.append("")
amort = report["amortization"]
if amort["queries_per_constructed_record"] is not None:
lines.append(
f"Amortization: {amort['queries_per_constructed_record']} queries per "
f"constructed record (floor {amort['floor']:.0f})"
)
lines.append("")
if report["findings"]:
lines.append(f"Findings ({len(report['findings'])})")
lines.append("-" * 68)
for finding in report["findings"]:
lines.append(f" [{finding['severity'].upper()}] {finding['code']}")
lines.append(f" {finding['detail']}")
lines.append(f" -> {finding['action']}")
lines.append("")
else:
lines.append("No blocking findings. Re-run when volume or pricing changes.")
lines.append("")
return "\n".join(lines)
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Profile the construction (write) and query (read) phases of an "
"agent memory system, and report cost per correct answer."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Examples:\n"
" memory_cost_profiler.py --sample\n"
" memory_cost_profiler.py --spec workload.json\n"
" memory_cost_profiler.py --sample --output json\n"
),
)
# Not required=True: argparse enforces a required group during
# parse_args(), which made --print-sample-spec unreachable on its own.
# Validated explicitly after the print-and-exit branch instead.
source = parser.add_mutually_exclusive_group(required=False)
source.add_argument("--spec", help="path to a memory workload spec JSON file")
source.add_argument(
"--sample",
action="store_true",
help="profile the built-in sample workload (no input file needed)",
)
parser.add_argument(
"--output",
choices=["text", "json"],
default="text",
help="output format (default: text)",
)
parser.add_argument(
"--print-sample-spec",
action="store_true",
help="print the sample spec JSON and exit (use as a template)",
)
args = parser.parse_args()
if args.print_sample_spec:
print(json.dumps(SAMPLE_SPEC, indent=2))
return 0
if not (args.spec or args.sample):
parser.error(
"one of --spec, --sample, or --print-sample-spec is required"
)
if args.sample:
spec = SAMPLE_SPEC
else:
try:
with open(args.spec, "r", encoding="utf-8") as handle:
spec = json.load(handle)
except FileNotFoundError:
_fail(f"spec file not found: {args.spec}")
except json.JSONDecodeError as exc:
_fail(f"spec file is not valid JSON: {exc}")
report = profile(spec)
if args.output == "json":
print(json.dumps(report, indent=2))
else:
print(render(report))
return 2 if report["findings"] else 0
if __name__ == "__main__":
sys.exit(main())

723
engineering/memory-engineering/skills/memory-engineering/scripts/memory_density_auditor.py Normal file
View file

@ -0,0 +1,723 @@
#!/usr/bin/env python3
"""Audit what a memory store actually holds: facts, skills, or logs.
Microsoft's PlugMem starts from a result that should unsettle anyone adding
memory to an agent: giving it more raw memory can make it worse. History piles
up, retrieval drowns, and the agent burns attention wading through transcripts
for the one line that mattered. The fix borrows from human memory -- we do not
replay events, we keep the facts and the skills we pulled out of them.
This tool classifies every record in a memory store as FACT, SKILL, or LOG,
finds near-duplicates, flags staleness, and scores knowledge density: how much
decision-relevant material there is per 1,000 tokens of context it costs.
It runs on either shape of memory:
--dir a directory of markdown/text memory files (CLAUDE.md, a wiki vault,
agent memory files) -- records are split on markdown headings
--jsonl a JSONL file of records, one object per line with a "text" field
Deterministic classification by lexical signal. No LLM calls, no network,
stdlib only. Classification is a triage aid, not ground truth -- it is tuned to
over-report LOG, because storing a log you thought was a fact is the failure
mode this tool exists to catch.
Exit codes:
0 store is knowledge-dense
2 actionable finding (log-heavy, duplicate-bloated, or stale)
3 invalid input
"""
import argparse
import json
import os
import re
import sys
from datetime import datetime, timezone
# Above this share of LOG records, the store is replaying events rather than
# keeping the knowledge extracted from them.
LOG_HEAVY_SHARE = 0.35
# Above this share of near-duplicate records, retrieval is competing with itself.
DUPLICATE_SHARE = 0.15
# Above this share of signal-less narrative records, the store is documentation
# rather than retrievable memory.
PROSE_HEAVY_SHARE = 0.40
# Jaccard similarity over word shingles at which two records are near-duplicates.
DUPLICATE_THRESHOLD = 0.75
SHINGLE_SIZE = 3
# A heading whose body is shorter than this is a section marker, not a record.
MIN_RECORD_WORDS = 3
# Records shorter than this are excluded from duplicate comparison. Two short
# fragments share shingle sets trivially, which produces 1.00 "duplicates"
# between unrelated files.
MIN_DUPLICATE_WORDS = 20
# Duplicate detection is O(n^2) pairwise. Above this many eligible records the
# scan is capped -- and the number skipped is reported, never dropped silently.
MAX_DUPLICATE_SCAN = 2000
# Records whose newest date is older than this are candidates for review.
DEFAULT_STALE_DAYS = 180
TEXT_SUFFIXES = {".md", ".markdown", ".txt", ".mdx"}
# --- classification signals ------------------------------------------------
LOG_PATTERNS = [
re.compile(r"^\s*(user|assistant|human|ai|system)\s*:", re.IGNORECASE | re.M),
re.compile(r"^\s*\[?\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}", re.M),
re.compile(r"\b(session|conversation|transcript|chat log)\b", re.IGNORECASE),
re.compile(
r"\b(then (?:i|we|the user)|(?:i|we) (?:ran|tried|asked|noticed|said)"
r"|the user (?:said|asked|wanted|reported))\b",
re.IGNORECASE,
),
re.compile(r"\bon \w+ \d{1,2}(?:st|nd|rd|th)?,? \d{4}\b", re.IGNORECASE),
]
SKILL_PATTERNS = [
re.compile(r"^\s*\d+[.)]\s+\S", re.M), # numbered procedure
re.compile(
r"\b(always|never|must|should|prefer|avoid|do not|don't)\b", re.IGNORECASE
),
re.compile(r"\bto\s+\w+,\s+(?:use|run|call|set|add|check)\b", re.IGNORECASE),
re.compile(
r"^\s*(use|run|call|set|add|check|prefer|avoid|install|configure|deploy)\b",
re.IGNORECASE | re.M,
),
re.compile(r"\b(workflow|procedure|steps?|recipe|playbook|how to)\b", re.IGNORECASE),
]
FACT_PATTERNS = [
re.compile(
r"\b\w+\s+(?:is|are|was|were|uses|runs on|lives in|owns|has|equals)\s+\S",
re.IGNORECASE,
),
re.compile(r"^\s*[-*]\s*\*?\*?[\w ./-]+\*?\*?\s*[:=]\s*\S", re.M), # key: value
re.compile(r"\b(version|endpoint|port|repo|owner|deadline|budget)\b", re.IGNORECASE),
]
# Phrases whose truth depends on when they were written.
VOLATILE_PATTERNS = [
re.compile(
r"\b(currently|right now|at the moment|as of (?:today|now)|this (?:week|month|quarter|sprint)"
r"|for now|temporarily|at present|these days|nowadays)\b",
re.IGNORECASE,
),
re.compile(r"\b(latest|newest|most recent|upcoming|soon|next release)\b", re.IGNORECASE),
]
DATE_PATTERN = re.compile(r"\b(\d{4})-(\d{2})-(\d{2})\b")
WORD_PATTERN = re.compile(r"[a-z0-9]+")
def _fail(message: str) -> None:
print(f"error: {message}", file=sys.stderr)
raise SystemExit(3)
def _plural(count: int, noun: str) -> str:
return f"{count} {noun}" if count == 1 else f"{count} {noun}s"
def estimate_tokens(text: str) -> int:
"""Rough token estimate. Deliberately crude -- ~4 chars per token."""
return max(1, len(text) // 4)
def _mask_code_fences(text: str) -> str:
"""Blank out fenced code bodies so '# comment' inside them is not a heading.
Same length is preserved, so offsets into the masked copy still index the
original text correctly.
"""
masked = list(text)
fence = re.compile(r"^[ \t]*(`{3,}|~{3,})", re.M)
positions = [match.start() for match in fence.finditer(text)]
for index in range(0, len(positions) - 1, 2):
start, end = positions[index], positions[index + 1]
for offset in range(start, min(end, len(masked))):
if masked[offset] != "\n":
masked[offset] = " "
return "".join(masked)
def split_records(text: str, source: str) -> list[dict]:
"""Split a document into records on markdown headings, else blank lines."""
records = []
heading = re.compile(r"^(#{1,6})\s+(.*)$", re.M)
# Detect headings on a code-masked copy, but slice bodies from the original.
matches = list(heading.finditer(_mask_code_fences(text)))
if len(matches) >= 2:
for index, match in enumerate(matches):
start = match.end()
end = matches[index + 1].start() if index + 1 < len(matches) else len(text)
body = text[start:end].strip()
# A heading with a near-empty body is a section marker, not a
# memory record. Admitting it would inflate every count below.
if len(body.split()) >= MIN_RECORD_WORDS:
records.append(
{
"source": source,
"title": match.group(2).strip(),
"text": body,
}
)
return records
for block in re.split(r"\n\s*\n", text):
block = block.strip()
if len(block) >= 40:
records.append({"source": source, "title": "", "text": block})
return records
def _score(patterns: list, text: str) -> int:
return sum(1 for pattern in patterns if pattern.search(text))
def classify(text: str) -> tuple[str, dict]:
"""Classify a record as LOG, SKILL, FACT, or PROSE with its signal counts.
A record is only called LOG when it carries a positive event signal.
Records with no signal at all are PROSE, not LOG -- narrative documentation
is neither an event log nor a retrievable fact, and calling it LOG would
fire the log-heavy finding on every prose-shaped store.
"""
signals = {
"log": _score(LOG_PATTERNS, text),
"skill": _score(SKILL_PATTERNS, text),
"fact": _score(FACT_PATTERNS, text),
}
# LOG wins ties: mistaking an event for knowledge is the costly direction.
if signals["log"] > 0 and signals["log"] >= max(signals["skill"], signals["fact"]):
return "LOG", signals
if signals["skill"] > signals["fact"]:
return "SKILL", signals
if signals["fact"] > 0:
return "FACT", signals
return "PROSE", signals
def shingles(text: str) -> set:
words = WORD_PATTERN.findall(text.lower())
if len(words) < SHINGLE_SIZE:
return {" ".join(words)} if words else set()
return {
" ".join(words[i : i + SHINGLE_SIZE])
for i in range(len(words) - SHINGLE_SIZE + 1)
}
def jaccard(left: set, right: set) -> float:
if not left or not right:
return 0.0
intersection = len(left & right)
union = len(left | right)
return intersection / union if union else 0.0
def find_duplicates(records):
"""Pairwise near-duplicate detection over word shingles.
Returns (pairs, participants, redundant, scanned, skipped).
Two distinct counts, because they answer different questions and reporting
one under the other's name is misleading:
participants -- records having at least one near-duplicate. BOTH members
of a matching pair count. This is what "N records have a
near-duplicate" means, and it drives duplicate_share.
redundant -- copies that could actually be deleted: participants minus
one survivor per connected cluster. For a cluster of k
mutually-duplicate records this is k-1, not k.
Clusters are resolved with union-find rather than by counting pair
endpoints: a 3-record cluster produces pairs (i,j), (i,k), (j,k), so any
endpoint-counting shortcut gets the redundant count wrong.
"""
fingerprints = [shingles(record["text"]) for record in records]
# Only records long enough to have a meaningful fingerprint are compared;
# short fragments share shingle sets trivially.
eligible_ix = [
i
for i, record in enumerate(records)
if len(record["text"].split()) >= MIN_DUPLICATE_WORDS
]
# Comparison is O(n^2). Cap it, and report the cap rather than truncating
# silently -- a quiet cap reads as "no duplicates found".
skipped = 0
if len(eligible_ix) > MAX_DUPLICATE_SCAN:
skipped = len(eligible_ix) - MAX_DUPLICATE_SCAN
eligible_ix = eligible_ix[:MAX_DUPLICATE_SCAN]
parent = {i: i for i in eligible_ix}
def find(x):
while parent[x] != x:
parent[x] = parent[parent[x]]
x = parent[x]
return x
def union(a, b):
ra, rb = find(a), find(b)
if ra != rb:
parent[rb] = ra
pairs = []
participants = set()
for pos, i in enumerate(eligible_ix):
for j in eligible_ix[pos + 1 :]:
score = jaccard(fingerprints[i], fingerprints[j])
if score >= DUPLICATE_THRESHOLD:
pairs.append(
{
"similarity": round(score, 3),
"a": {
"source": records[i]["source"],
"title": records[i].get("title", ""),
},
"b": {
"source": records[j]["source"],
"title": records[j].get("title", ""),
},
}
)
participants.add(i)
participants.add(j)
union(i, j)
clusters = {find(i) for i in participants}
redundant = len(participants) - len(clusters)
return pairs, len(participants), redundant, len(eligible_ix), skipped
def newest_date(text: str):
newest = None
for match in DATE_PATTERN.finditer(text):
try:
found = datetime(
int(match.group(1)),
int(match.group(2)),
int(match.group(3)),
tzinfo=timezone.utc,
)
except ValueError:
continue
if newest is None or found > newest:
newest = found
return newest
def load_records(args) -> list[dict]:
records = []
if args.jsonl:
try:
with open(args.jsonl, "r", encoding="utf-8") as handle:
for number, line in enumerate(handle, start=1):
line = line.strip()
if not line:
continue
try:
payload = json.loads(line)
except json.JSONDecodeError as exc:
_fail(f"{args.jsonl}:{number} is not valid JSON: {exc}")
if not isinstance(payload, dict) or "text" not in payload:
_fail(f"{args.jsonl}:{number} must be an object with a 'text' field")
records.append(
{
"source": payload.get("source", f"{args.jsonl}:{number}"),
"title": payload.get("title", ""),
"text": str(payload["text"]),
}
)
except FileNotFoundError:
_fail(f"jsonl file not found: {args.jsonl}")
return records
if not os.path.isdir(args.dir):
_fail(f"not a directory: {args.dir}")
for root, dirnames, filenames in os.walk(args.dir):
dirnames[:] = [d for d in dirnames if not d.startswith(".")]
for filename in sorted(filenames):
if os.path.splitext(filename)[1].lower() not in TEXT_SUFFIXES:
continue
path = os.path.join(root, filename)
try:
with open(path, "r", encoding="utf-8", errors="replace") as handle:
content = handle.read()
except OSError:
continue
relative = os.path.relpath(path, args.dir)
records.extend(split_records(content, relative))
return records
def audit(records: list[dict], stale_days: int) -> dict:
if not records:
_fail("no records found -- check the path, or that files are .md/.txt")
now = datetime.now(timezone.utc)
counts = {"FACT": 0, "SKILL": 0, "LOG": 0, "PROSE": 0}
total_tokens = 0
stale = []
volatile = []
for record in records:
kind, signals = classify(record["text"])
record["kind"] = kind
record["signals"] = signals
tokens = estimate_tokens(record["text"])
record["tokens"] = tokens
counts[kind] += 1
total_tokens += tokens
found = newest_date(record["text"])
if found is not None:
age = (now - found).days
record["age_days"] = age
if age > stale_days:
stale.append(
{
"source": record["source"],
"title": record.get("title", ""),
"age_days": age,
}
)
if any(pattern.search(record["text"]) for pattern in VOLATILE_PATTERNS):
volatile.append(
{
"source": record["source"],
"title": record.get("title", ""),
"why": "contains time-relative wording that rots silently",
}
)
(
duplicate_pairs,
duplicate_records,
redundant_copies,
scanned,
skipped,
) = find_duplicates(records)
total = len(records)
knowledge = counts["FACT"] + counts["SKILL"]
log_share = counts["LOG"] / total
duplicate_share = duplicate_records / total
density = knowledge / (total_tokens / 1000.0) if total_tokens else 0.0
findings = []
if log_share > LOG_HEAVY_SHARE:
findings.append(
{
"code": "LOG_HEAVY",
"severity": "high",
"detail": (
f"{counts['LOG']}/{total} records ({log_share:.0%}) read as "
"event logs rather than facts or skills."
),
"action": (
"Extract the facts and skills out of these transcripts, "
"then drop the transcripts. Storing the event is what makes "
"retrieval drown."
),
}
)
if duplicate_share > DUPLICATE_SHARE:
findings.append(
{
"code": "DUPLICATE_BLOATED",
"severity": "high",
"detail": (
f"{duplicate_records}/{total} records ({duplicate_share:.0%}) "
f"have a near-duplicate at >= {DUPLICATE_THRESHOLD:.0%} "
f"similarity; {redundant_copies} of them are redundant "
"copies that could be deleted."
),
"action": (
"Dedup at write time. Duplicates do not just waste tokens -- "
"they let a stale copy outrank a corrected one."
),
}
)
if stale:
findings.append(
{
"code": "STALE_RECORDS",
"severity": "medium",
"detail": (
f"{len(stale)} records carry a date older than {stale_days} "
"days."
),
"action": "Re-verify or expire them. A memory that was true is still wrong.",
}
)
prose_share = counts["PROSE"] / total
if prose_share > PROSE_HEAVY_SHARE:
findings.append(
{
"code": "PROSE_HEAVY",
"severity": "medium",
"detail": (
f"{counts['PROSE']}/{total} records ({prose_share:.0%}) carry "
"no fact, skill, or event signal -- they read as narrative "
"documentation."
),
"action": (
"Prose is what a human re-reads, not what an agent "
"retrieves. Distill each block into the fact or the skill "
"it is trying to convey, or move it out of the memory "
"store and into docs."
),
}
)
if volatile:
findings.append(
{
"code": "VOLATILE_WORDING",
"severity": "medium",
"detail": (
f"{_plural(len(volatile), 'record')} "
f"{'uses' if len(volatile) == 1 else 'use'} time-relative "
"wording ('currently', 'latest') that becomes false without "
"editing."
),
"action": (
"Rewrite with an explicit date or version, so staleness is "
"detectable instead of invisible."
),
}
)
if any(f["severity"] == "high" for f in findings):
verdict = "LOG-HEAVY" if log_share > LOG_HEAVY_SHARE else "DUPLICATE-BLOATED"
elif prose_share > PROSE_HEAVY_SHARE:
verdict = "PROSE-HEAVY"
elif findings:
verdict = "NEEDS-MAINTENANCE"
else:
verdict = "KNOWLEDGE-DENSE"
return {
"verdict": verdict,
"totals": {
"records": total,
"estimated_tokens": total_tokens,
"fact": counts["FACT"],
"skill": counts["SKILL"],
"log": counts["LOG"],
"prose": counts["PROSE"],
"log_share": round(log_share, 4),
"prose_share": round(prose_share, 4),
},
"density": {
"knowledge_records_per_1k_tokens": round(density, 3),
"note": (
"Optimize decision-relevant information per token of context "
"it costs, not how much you managed to store."
),
},
"duplicates": {
"records_with_a_duplicate": duplicate_records,
"redundant_copies": redundant_copies,
"share": round(duplicate_share, 4),
"scanned": scanned,
"skipped_over_scan_cap": skipped,
"pairs": duplicate_pairs[:20],
"pairs_truncated": max(0, len(duplicate_pairs) - 20),
},
"stale_records": stale[:20],
"volatile_records": volatile[:20],
"findings": findings,
}
def render(report: dict) -> str:
lines = []
totals = report["totals"]
lines.append("MEMORY DENSITY AUDIT")
lines.append("=" * 68)
lines.append(f"VERDICT: {report['verdict']}")
lines.append("")
lines.append("What the store actually holds")
lines.append("-" * 68)
lines.append(f" records {totals['records']}")
lines.append(f" estimated tokens {totals['estimated_tokens']:,}")
lines.append(f" FACT {totals['fact']}")
lines.append(f" SKILL {totals['skill']}")
lines.append(
f" LOG {totals['log']} ({totals['log_share']:.0%} of records)"
)
lines.append(
f" PROSE {totals['prose']} "
f"({totals['prose_share']:.0%} of records, no signal either way)"
)
lines.append("")
lines.append(
f"Knowledge density: "
f"{report['density']['knowledge_records_per_1k_tokens']} "
"fact-or-skill records per 1k tokens"
)
lines.append("")
duplicates = report["duplicates"]
if duplicates["records_with_a_duplicate"]:
lines.append(
f"Near-duplicates: "
f"{_plural(duplicates['records_with_a_duplicate'], 'record')} "
f"({duplicates['share']:.0%}), of which "
f"{duplicates['redundant_copies']} redundant"
)
for pair in duplicates["pairs"][:5]:
left = f"{pair['a']['source']} {pair['a']['title']}".strip()
right = f"{pair['b']['source']} {pair['b']['title']}".strip()
lines.append(f" {pair['similarity']:.2f} {left} <-> {right}")
if duplicates["pairs_truncated"]:
lines.append(f" ... {duplicates['pairs_truncated']} more pairs")
if duplicates["skipped_over_scan_cap"]:
lines.append(
f" NOTE: {duplicates['skipped_over_scan_cap']} eligible records "
f"were not scanned (cap {MAX_DUPLICATE_SCAN}); duplicate counts "
"are a lower bound."
)
lines.append("")
if report["findings"]:
lines.append(f"Findings ({len(report['findings'])})")
lines.append("-" * 68)
for finding in report["findings"]:
lines.append(f" [{finding['severity'].upper()}] {finding['code']}")
lines.append(f" {finding['detail']}")
lines.append(f" -> {finding['action']}")
lines.append("")
else:
lines.append("No findings. This store is holding knowledge, not history.")
lines.append("")
return "\n".join(lines)
SAMPLE_RECORDS = [
{
"source": "sample/onboarding.md",
"title": "Deploy procedure",
"text": (
"To deploy the api service, run `make release` from main. Always "
"wait for the migration job to report green before promoting. "
"Never deploy on a Friday after 16:00 UTC.\n"
"1. Tag the release\n2. Run the migration\n3. Promote"
),
},
{
"source": "sample/infra.md",
"title": "Service ownership",
"text": (
"The billing service is owned by the payments team. Its production "
"endpoint is https://api.internal/billing and it runs on port 8443. "
"Repo: org/billing-service."
),
},
{
"source": "sample/session-2024-01-14.md",
"title": "Debugging session",
"text": (
"2024-01-14 09:12 User: the billing job is failing again\n"
"Assistant: let me look at the logs\n"
"Then I ran the migration by hand and it worked. The user said "
"they would file a ticket about it later."
),
},
{
"source": "sample/session-2024-02-02.md",
"title": "Another debugging session",
"text": (
"2024-02-02 14:40 User: billing job failing\n"
"Assistant: checking the logs now\n"
"Then I ran the migration by hand and it worked. The user said "
"they would file a ticket about it later."
),
},
{
"source": "sample/stack.md",
"title": "Current stack",
"text": (
"We are currently on Postgres 14 and the latest Redis. This is the "
"most recent setup as of now."
),
},
{
# The same fact written twice in two files -- the most common way a
# memory store bloats, and the way a stale copy outranks a fixed one.
"source": "sample/teams.md",
"title": "Billing ownership",
"text": (
"The billing service is owned by the payments team. Its production "
"endpoint is https://api.internal/billing and it runs on port 8443. "
"Repo: org/billing-service."
),
},
]
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Classify memory records as FACT / SKILL / LOG, find near-duplicates "
"and staleness, and score knowledge density."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Examples:\n"
" memory_density_auditor.py --sample\n"
" memory_density_auditor.py --dir ~/.claude/memory\n"
" memory_density_auditor.py --jsonl records.jsonl --output json\n"
),
)
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--dir", help="directory of markdown/text memory files")
source.add_argument("--jsonl", help="JSONL file of records with a 'text' field")
source.add_argument(
"--sample",
action="store_true",
help="audit the built-in sample store (no input needed)",
)
parser.add_argument(
"--stale-days",
type=int,
default=DEFAULT_STALE_DAYS,
help=f"age in days past which a dated record is stale (default: {DEFAULT_STALE_DAYS})",
)
parser.add_argument(
"--output",
choices=["text", "json"],
default="text",
help="output format (default: text)",
)
args = parser.parse_args()
if args.stale_days < 1:
_fail("--stale-days must be >= 1")
records = list(SAMPLE_RECORDS) if args.sample else load_records(args)
report = audit(records, args.stale_days)
if args.output == "json":
print(json.dumps(report, indent=2))
else:
print(render(report))
return 2 if report["findings"] else 0
if __name__ == "__main__":
sys.exit(main())

16
engineering/write-a-skill/skills/write-a-skill/SKILL.md
View file

@ -135,6 +135,22 @@ python scripts/skill_review_checklist_runner.py path/to/skill-folder
See [references/companion_tooling.md](references/companion_tooling.md) for the tool catalogue, cs-skill-author persona agent, and `/cs:write-a-skill` slash command.
## When the knowledge is in a document, not your head
This skill authors from expertise you already have. When the source is a book, a docs folder,
a standard, or a pile of specs, use `engineering/book-to-skill` instead — it compiles the
document into a knowledge-base skill (core frameworks + on-demand chapters + glossary +
patterns + cheatsheet) and can package the result as a plugin.
```
/cs:book-to-skill <path|folder|glob> [skill-name] # compile the source
/cs:book-to-plugin <compiled-skill-dir> # wrap it as a plugin
```
Rule of thumb: **author first, compile second.** A hand-written skill states what you want the
agent to do; a compiled book skill is the reference it consults while doing it. If you have
both, they are two skills, not one.
---
**Version:** 1.0.0

30
scripts/smoke_exceptions.txt
View file

@ -59,3 +59,33 @@ engineering/skillopt-sleep/skillopt_sleep/staging.py # internal module, types.p
engineering/skillopt-sleep/skillopt_sleep/state.py # internal module, types.py shadow
engineering/skillopt-sleep/skillopt_sleep/tasks_file.py # internal module, types.py shadow
engineering/skillopt-sleep/skillopt_sleep/types.py # internal module; module itself IS the stdlib-name collision
# book_to_skill extraction library (engineering/book-to-skill, vendored from
# virgiliojr94/book-to-skill): internal package modules imported as
# `book_to_skill.*`, not standalone CLI scripts.
#
# Listed here are the modules that FAIL the G8 probe — those with absolute
# `book_to_skill.*` imports, which raise `No module named 'book_to_skill'` before
# they could parse a flag. The package's remaining modules (config.py,
# exceptions.py, sanitize.py, parsers/__init__.py, parsers/pdf.py,
# parsers/text.py) are equally not-CLIs; they pass G8 only because they have no
# argv handling at all, so `--help` no-ops and exits 0. This list is therefore
# "modules the probe trips on", not "modules that aren't CLIs" — the latter is
# the whole package.
#
# The four real entry points DO implement the CLI contract and are smoke-tested
# normally:
# scripts/extract_document.py, book_skill_validator.py,
# token_budget_estimator.py, skill_plugin_emitter.py (--help + --sample, exit 0)
# Unlike the skillopt_sleep case below, no module here shadows a stdlib name:
# upstream's parsers/html.py was renamed to parsers/html_text.py precisely so
# `import html.parser` cannot resolve to a sibling.
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/__init__.py # package init, not a CLI
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/dependencies.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/utils.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/zip_safety.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/calibre.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/docx.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/epub.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/html_text.py # internal module
engineering/book-to-skill/skills/book-to-skill/scripts/book_to_skill/parsers/rtf.py # internal module