claude-skills/engineering/book-to-skill

book-to-skill

Turn a book, a documentation folder, or a pile of specs into an agent skill.

A source goes in. What comes out is a knowledge base the agent loads on demand: a small resident core with the author's named frameworks and a topic index, one chapter file per chapter, a glossary, a patterns file, and a decision cheatsheet. The agent reads the core, then one chapter — never the whole source again.

Derived from virgiliojr94/book-to-skill (MIT). See Deviations from upstream below for the authoritative list of what changed.

---

What it produces

File	Contents	Budget
SKILL.md	Core frameworks + chapter index + topic index	< 4,000 tokens, resident
chapters/chNN-*.md	One summary per chapter	800–3,000 tokens, on demand
glossary.md	Every significant term, alphabetized, with chapter	< 1,500 tokens
patterns.md	Techniques and design patterns with trade-offs	< 2,000 tokens
cheatsheet.md	Decision rules, thresholds, trade-off matrices	< 1,200 tokens

Formats: PDF, EPUB, DOCX, HTML, Markdown, plain text, reStructuredText, AsciiDoc, RTF, and MOBI/AZW/AZW3 with Calibre.

---

Install

Already in this repository. Install as a marketplace plugin:

/plugin install book-to-skill@claude-code-skills

Or use the tools directly — they are self-contained under skills/book-to-skill/scripts/.

Dependencies: none required. Every format degrades to a standard-library parser except MOBI/AZW/AZW3, which need Calibre's ebook-convert on PATH. Optional packages (docling, pypdf, pdfminer.six, ebooklib, beautifulsoup4, python-docx, striprtf) raise extraction quality where installed; Poppler's pdftotext is the fastest PDF path. Nothing is installed implicitly — run extract_document.py --check to see what is present and get the exact install command.

---

Use

/cs:book-to-skill ~/books/thinking-in-systems.pdf meadows-systems
/cs:book-to-skill ./docs/architecture/ '*.md' internal-arch
/cs:book-to-plugin ~/.claude/skills/meadows-systems --domain engineering

Or drive the tools yourself:

SKILL_ROOT=engineering/book-to-skill/skills/book-to-skill
WORKDIR=$(mktemp -d)     # or omit --workdir and capture the path the tool prints
SLUG=<author-lastname>-<concept>

python3 "$SKILL_ROOT/scripts/extract_document.py" BOOK.pdf --mode technical --workdir "$WORKDIR"
python3 "$SKILL_ROOT/scripts/token_budget_estimator.py" --full-text "$WORKDIR/full_text.txt"
# ... agent generates the skill ...
python3 "$SKILL_ROOT/scripts/book_skill_validator.py" ~/.claude/skills/"$SLUG"
python3 "$SKILL_ROOT/scripts/skill_plugin_emitter.py" --skill-dir ~/.claude/skills/"$SLUG" \
    --dest ./engineering --dry-run

Every tool supports --help, --sample and --output json.

---

Tools

Tool	Does
extract_document.py	Multi-format extraction → full_text.txt + metadata.json. Strips invisible Unicode, blocks DOCX entity expansion, detects chapters across Latin, Roman, Chinese, Thai and Korean heading styles. --check reports the environment.
book_skill_validator.py	Four check families over a generated skill — frontmatter (host-lens rules), safety (injection phrasing, invisible Unicode, authority grants), budget (token caps), index (dead links, unindexed chapters, dangling topic refs). Errors block; --strict promotes warnings.
token_budget_estimator.py	Pre-flight: models context-dump vs. discovery-loop vs. compiled-skill cost on this source's real chapter sizes and prints a worth-converting verdict. Post-flight: file-by-file budget audit.
skill_plugin_emitter.py	Wraps a compiled skill as a claude-skills plugin (manifest + agent + command + README) and prints the marketplace entry. Refuses to emit a shareable package without a rights basis.

---

Why the tokens matter

A live document-reading agent pays a discovery loop tax: it reads the table of contents, pulls the chapter it guessed at, then backtracks for a definition it turns out to need. Every one of those lands in history. Dumping the whole source into context is worse — it is resident, and re-billed every turn.

Compiling pays that navigation cost once. A query afterwards costs what the answer costs, not what the source costs. token_budget_estimator.py models all three on the real token sizes of your source and says which one wins — including when the honest answer is "this source is too small; just read it."

---

Deviations from upstream

This numbered list is the authoritative record. .claude-plugin/authoring-notes.json's attribution.derivation_note summarizes it; if the two ever disagree, this list wins.

Structural

1. Repo-native layout. Upstream is a standalone repository (book_to_skill/ package, scripts/, tools/, tests/, docs/, mkdocs.yml, pyproject.toml, CI workflows). Here it is one plugin: skills/book-to-skill/{SKILL.md,scripts,references,assets} plus agents/, commands/ and a manifest. Upstream's packaging, docs site, GitHub workflows and pytest suite were not vendored — this repo ships no build system and no test framework by design.

2. The extraction library is vendored close to verbatim. book_to_skill/ — config.py, exceptions.py, sanitize.py, dependencies.py, utils.py and the seven per-format parsers — carries upstream's format chains, its chapter detection across five script families, and its Unicode and XXE hardening. That code is good and re-deriving it would only make it worse.

3. The library is now import-pure. parse_arguments(), print_usage(), print_banner() and main() were removed from utils.py and rebuilt as an argparse CLI in extract_document.py. Importing the library no longer prints, prompts, or parses sys.argv.

4. SKILL.md rewritten Claude-Code-first. The ten-step workflow, the per-chapter budget matrix and the update/fold-in workflow are preserved in substance. Added: this repo's frontmatter metadata block, an explicit hard-rules section, a forcing-question library, a references section, and related-skill disambiguation against write-a-skill, skill-security-auditor and llm-wiki. The skill-home table leads with Claude Code; Copilot CLI and Amp remain supported.

Safety and behaviour

5. No implicit installs. Upstream's --install-missing defaults to ask, which prompts on a TTY and runs pip install into the caller's environment. Here the default is report: it prints the exact install command and uses the stdlib fallback. --install-missing yes remains as an explicit opt-in. Reading a file should not install packages.

6. Rights gate on distribution. Upstream documents its copyright posture in prose. Here it is enforced: skill_plugin_emitter.py --distribution shareable refuses unless --rights names public-domain, open-license, internal-docs or author-permission. fair-use is deliberately absent — it is a defence, not a licence, and not a script's call. Without a basis, packages emit as local and carry source.cleared_for_distribution: false in the manifest.

7. Clean JSON output. The vendored parsers narrate progress on stdout. Under --output json that narration is redirected to stderr, so stdout carries the metadata document and nothing else. Upstream had no JSON mode.

8. Attribution moved off the hot path. Upstream prints an ASCII banner from scripts/banner.txt on every extraction run, and ships BACKERS.md + FUNDING.yml. Attribution here lives in LICENSE, plugin.json, this README and the SKILL.md footer — correct, and not re-emitted on every invocation.

Tooling

9. Three upstream tools became four argparse CLIs, each meeting this repo's contract: real --help, a --sample that runs the tool on built-in fixtures, --output json, and documented exit codes. Upstream's extract.py parsed sys.argv by hand and ignored unknown flags with a warning, so a typo'd flag silently changed nothing.

10. Validator merged and extended. Upstream's validate_skill.py (frontmatter lint) and scan_generated_skill.py (injection scan) are one gate, book_skill_validator.py, with every finding carrying a family and a severity. Two check families are new: budget — every generated file against the token caps the workflow commits to, with SKILL.md overflow as the only hard budget error (compaction truncates from the end, so overflow eats the indexes first); and index — dead chapter links, chapter files nothing links to, topic entries pointing at chapters that were never written, and a missing topic index. Index integrity is the failure that silently breaks navigation while the skill still looks complete, and upstream had no check for it.

11. Folded YAML scalars are parsed correctly. Upstream reads a frontmatter value with a single-line regex. A wrapped description therefore under-reports its length — so the 1024-character cap never fires on exactly the descriptions long enough to hit it — and gets truncated at the wrap when copied into a manifest. scalar_value() folds continuation lines and unescapes inner quotes. A description-trigger warning was also added, matching this repo's write-a-skill rule that a description must say when to use the skill.

12. discovery_tax.py → token_budget_estimator.py. The optional tiktoken path was dropped so every token number in the pipeline comes from one deterministic estimator and no budget gate depends on whether a package is installed. Added: a post-flight file-by-file budget audit, and an explicit worth-converting verdict — upstream reports only savings ratios, which read as advocacy on a source too small to be worth converting at all.

13. parsers/html.py renamed to parsers/html_text.py. A module named html.py shadows the standard library's html package the moment its own directory lands on sys.path[0] — which happens whenever the file is run directly — and import html.parser then fails with "'html' is not a package". Renaming removes the hazard instead of documenting it. Two import lines changed; nothing else.

14. CI-contract wiring. The eight vendored library modules are registered in scripts/smoke_exceptions.txt (they are imported as book_to_skill.*, never run as CLIs), and the agent's tool table uses paths that resolve from the agent's own folder so scripts/check_paths.py can follow them. Neither has an upstream counterpart — upstream has no equivalent gates.

15. Private, per-invocation working directory. Upstream defaults the extraction workdir to a fixed <tempdir>/book_skill_work. On a shared host that is CWE-377/CWE-59: any local user can pre-create the directory in a world-writable /tmp (the sticky bit prevents deletion, not creation) and plant a symlink named full_text.txt pointing at a file the victim can write — Path.write_text follows symlinks. Two concurrent runs also clobber each other. The default is now a fresh mkdtemp (unpredictable, 0700 by construction); artifacts are written 0600; an explicit --workdir/BOOK_SKILL_WORKDIR is honoured but symlink-refused and mode-restricted first. parsers/calibre.py no longer writes its ebook-convert scratch file to the shared directory either — which also fixes a real bug, since it read a module-level constant and so ignored --workdir entirely.

16. Shared budget constants. book_skill_validator.py and token_budget_estimator.py both gate on the same token caps; those now live once in book_to_skill/config.py (SKILL_FILE_BUDGETS, CHAPTER_TOKEN_CEILING) rather than being restated in each tool, where they would drift the first time a cap changed.

17. Zip-of-XML hardening generalized to EPUB, plus decompression-bomb caps. Upstream hardened DOCX and only DOCX: validate_docx_xml_safety() screened that archive for DTD/entity declarations before any parser touched it. EPUB is the same shape — a zip whose members are XML — and its ebooklib path handed the file straight to a third-party XML stack with no equivalent check, despite ebooklib being one of the packages this skill recommends installing. The guard now lives in book_to_skill/zip_safety.py and runs for both formats. Every archive read also goes through safe_read(), which consults the declared uncompressed size and the compression ratio before decompressing: a 200 MB zip bomb is refused at ~14 MB peak RSS instead of being materialized. Neither parser ever writes archive members to disk, so zip-slip stays out of scope by construction.

18. Packaging refuses a source tree containing symlinks. The validator checks the files it knows about (SKILL.md, the three supporting files, chapters/*.md), but shutil.copytree defaults to symlinks=False and follows a link anywhere else in the tree — an assets/ entry, any subdirectory — baking the target's real content into a package that may then be emitted as --distribution shareable. _assert_no_symlinks() now walks the whole tree and refuses, and it runs before the validation branch so --skip-validation cannot bypass it. copytree also passes symlinks=True so a future edit loosening that check cannot silently reintroduce dereferencing.

19. The magic-byte sniff path goes through the size budget too. extract_single_file() sniffs unknown extensions and read a mimetype member with a bare zf.read() — the earliest attacker-controlled point in the pipeline, running before a format is chosen and before any of zip_safety.py's checks. A single-member zip declaring a huge uncompressed mimetype was fully decompressed there. Now routed through safe_read(); its ExtractionError deliberately sits outside the surrounding except tuple so a bomb reports as a bomb rather than as a generic unsupported format. Verified: a 200 MB / 1029× fixture with no file extension is refused at ~15 MB peak RSS.

20. Emitter correctness and scope. Three smaller fixes: --author/--author-url now reach the printed marketplace entry (it hardcoded one name, so the snippet whose whole job is preventing hand-edit mistakes contradicted the manifest beside it); the symlink guard is backed by a post-copy re-walk that deletes the package if a link appeared during the copy, closing the check-then-act window rather than only narrowing it; and the manifest emits source.license_scope stating that the top-level license covers the package scaffolding, not the compiled notes — a distinction that previously lived only in README prose where a tool reading the metadata alone would miss it. (Deviation 26 later moved that whole source block out of plugin.json and into the sidecar.)

21. The documented quick-start actually runs. SKILL.md's copy-paste block referenced $WORKDIR and $SKILLS_HOME without ever defining them — following it literally produced a traceback on step 2. Both are now real assignments, and all five steps were executed verbatim end to end as a check. A quick-start that does not run is worse than no quick-start: it is the part a reader trusts most.

22. Gate tools refuse bad paths instead of reporting success. token_budget_estimator.py --skill-dir <typo> produced a complete, plausible budget audit — every row "missing", every cap satisfied, exit 0 — which reads as a pass. A gate that reports success for a path that is not there is worse than no gate. It now refuses a missing directory, a non-directory, and a directory without SKILL.md (exit 2); --full-text <missing> raised a bare FileNotFoundError traceback and now refuses cleanly. The other three tools already handled bad input; this one was the outlier.

23. Three more upstream artifacts cleaned. epub.py's except (KeyError, Exception) is simply except Exception — it swallowed everything, including the size-refusal that safe_read() now raises, quietly disarming deviation 17 at that call site. Narrowed so an ExtractionError propagates and only genuine parse failures fall through to the .opf glob. utils.py emitted a dynamic {pages_label: pages} key alongside a literal "pages", which collided whenever the label was "pages"; the alias is now added only when it differs. A stray artifact word was removed from a pdf.py comment.

24. tool | head no longer tracebacks. Piping a report into head closes the pipe mid-write and surfaced a BrokenPipeError stack trace (observed once; racy on flush timing). All four CLIs now exit 141 quietly, the standard SIGPIPE convention.

25. The workdir race is closed, not just narrowed. The explicit---workdir path was still check-then-act: mkdir(parents=True, exist_ok=True) does not raise on a symlink-to-directory, because its exists-branch tests is_dir(), which follows symlinks — verified directly. Worse, the per-file is_symlink() check cannot back that up: a file inside a swapped directory is an ordinary file, not a link, so a directory swap defeated the artifact-level guard entirely.

    Three changes close it. resolve_workdir() now attempts mkdir first and only inspects an already-existing path, using os.lstat (which does not follow the final component). open_workdir() then pins the directory with O_NOFOLLOW|O_DIRECTORY, and both artifacts are written through that descriptor — an fd names an inode, so a later rename or symlink swap of the path cannot redirect the write. _write_private() creates with O_CREAT|O_EXCL|O_NOFOLLOW at mode 0600, which has no check-then-act window at all; an artifact from a previous run into the same --workdir is unlinked first, and unlink removes the link itself, never its target.

    Verified against a live race: pin the directory, rename it away, plant a symlink to an attacker directory, then write — the data lands in the pinned inode and the attacker directory stays empty. Also verified that a pre-planted full_text.txt -> victim symlink leaves the victim's contents intact and is replaced by a 0600 file we own. Degrades to the previous path-based checks on platforms without dir_fd/O_NOFOLLOW (Windows).

26. Provenance moved out of plugin.json into the sidecar the repo actually allows. The manifest builder wrote its whole source block (spec, build pattern, source document, chapter count, distribution, license_scope, rights_basis) into plugin.json, and an inline comment asserted that source and attribution were "approved extension fields." That had been true and no longer was: Claude Code rejects an entire manifest on any unrecognized key (issue #954), and this repo's own scripts/check_plugin_json.py hard-fails a plugin.json carrying either field, pointing at .claude-plugin/authoring-notes.json instead. So every package the emitter produced failed the repo's blocking CI gate the moment it was committed — a defect that only surfaces at the very last step of the pipeline, which is why it survived. _plugin_manifest() now emits spec fields only, and a new _authoring_notes() writes the source block to .claude-plugin/authoring-notes.json. Note that source remains a valid key in a marketplace.json plugins[] entry, which is how it leaked into the manifest in the first place; the printed marketplace snippet is unchanged and still correct.

---

Security audit

engineering/skills/skill-security-auditor on skills/book-to-skill/: 0 critical, 4 high, all four reviewed and accepted:

Finding	Why it stands
FS-ABUSE — shutil.rmtree(package_root) in the emitter	The --force overwrite path. Guarded by _assert_replaceable(): refuses a symlink, a non-directory, a path resolving outside the destination root, a directory containing the source skill, and any directory without a .claude-plugin/plugin.json — so it only ever deletes a plugin package this tool created.
DEPS-RUNTIME ×3 in dependencies.py	One is a docstring sentence; two are print() calls that show an install command. None of the three install anything. The real pip subprocess sits behind --install-missing yes and is never reached by default (deviation 5).

Two PRIV-ESC criticals were real and are fixed: upstream's install hints contained the literal string sudo apt install poppler-utils. They now name the package manager without telling anyone to escalate.

---

Where it fits

• engineering/write-a-skill — authoring a skill from expertise in your head. This one compiles a skill from a document on disk. With both, author first and fold the document in.
• engineering/skill-security-auditor — the repo-wide security audit. The validator here is the converter's own gate, not a replacement for it.
• engineering/llm-wiki — an incrementally-grown vault across many sources over time. This compiles one bounded source set into one skill, once.

---

License

MIT. See LICENSE — copyright is retained by the upstream author for the vendored extraction library, with modifications and additions by this repository under the same terms.

The MIT license covers the converter. It does not cover the books or documents you process with it, and it does not make a compiled skill redistributable. See skills/book-to-skill/references/rights_and_provenance.md.