mirror of
https://github.com/alirezarezvani/claude-skills.git
synced 2026-08-28 04:24:58 +00:00
feat(engineering): add deep-learning-book — companion skill for the free Goodfellow/Bengio/Courville textbook
Requested as "convert deeplearningbook.org into a skill". Built as a companion rather than a compilation, because the repo's own rights gate forbids the latter for a public plugin. Why not book-to-skill: its emitter refuses a shareable package without public-domain / open-license / internal-docs / author-permission, none of which applies to an MIT Press title whose site states its HTML-only format exists as a friction against copying under the authors' contract. Its rights reference lists publishing a compiled skill of a copyrighted book to a public marketplace under "Do not", and its hard rule 1 forbids scraping a book from the web, so the pipeline could not have run against a URL either. What shipped instead: the compiled-skill shape (master SKILL.md ~2.0k tokens with chapter and topic indexes, chapters/ch01..ch20, glossary, patterns, cheatsheet) filled with original synthesis and linking to the official free chapters. No passages, figures, or per-paragraph paraphrase. Passes book-to-skill's own book_skill_validator.py clean, with every file inside token_budget_estimator's caps. The differentiator is the delta layer. A compilation freezes a source at its publication date; this one dates it. Every chapter carries "What changed after 2016", and references/book_to_2026_delta.md gives five corrections with primary citations and per-claim confidence levels: double descent qualifying Ch 5's U-curve, AdamW splitting weight decay from L2, transformers displacing Ch 10's recurrence, diffusion growing out of Ch 18's score matching, and self-supervised learning vindicating Ch 15 while replacing its methods. Two claims are marked contested rather than propagated, two named as folklore. Four stdlib-only tools, each with a real refusal: - reading_path_planner.py — prerequisite closure over the book's actual dependency graph; exit 3 for a goal the book does not cover, exit 4 with forcing questions - training_diagnostics.py — Ch 11's rules in priority order, so a NaN is never reported as overfitting; exit 4 rather than diagnosing with no instruments - capacity_planner.py — regularization ladder in cost order, "shrink the model" ranked last in the overparameterized regime; exit 4 on a val-below-train split - model_arithmetic.py — params/FLOPs/activation memory for conv, linear, MHA and LSTM/GRU stacks; exit 5 naming the layer whose shapes do not connect Also: 4 references citing 7-8 sources each, 3 assets, cs-deep-learning-tutor agent, /cs:deep-learning + /cs:dl-reading-path + /cs:dl-diagnose. Counters: skills 386 to 387, tools 723 to 727, refs 838 to 842, agents 116 to 117, commands 146 to 149, plugins 97 to 98 (verified by derive_counters.py --check). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BswsZp5zrJWFAGU6KWNA1s
This commit is contained in:
Claude
parent
f2bac0a8f2
commit
1a8d782872
45 changed files with 3973 additions and 11 deletions
|
|
@ -8,7 +8,7 @@
|
|||
"homepage": "https://github.com/alirezarezvani/claude-skills",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"metadata": {
|
||||
"description": "386 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 723 Python tools, 838 reference guides, 116 agents (cs-* + personas), 146 slash commands across 97 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"description": "387 production-ready skills across 20 domains (engineering, engineering-core, marketing, product, c-level, c-level-agents, compliance-os, project management, RA/QM, business growth, finance, productivity, marketing top-level, research, research-ops, business-operations, commercial, markdown-html, loop-library, plus standards). 727 Python tools, 842 reference guides, 117 agents (cs-* + personas), 149 slash commands across 98 marketplace plugins. v2.11.2 vendors engineering/skillopt-sleep — a verbatim copy of microsoft/SkillOpt's stdlib-only skillopt_sleep engine + Claude Code plugin surface, giving a local agent a nightly gated self-improvement cycle (read-only session harvest -> mine -> offline replay -> held-out-gated CLAUDE.md/SKILL.md edits -> staged for explicit /skillopt-sleep adopt). productivity/fable-goal (unreleased, post-v2.11.1) converts a rambling description of a desired outcome into one polished /goal prompt for a fresh autonomous session. v2.11.1 turns product-team and project-management into agent-harness domains: fork-orchestrators with deterministic goal routers, a Jira MCP snapshot bridge (Kanban flow metrics + Monte Carlo forecasting), a delegation-governance loop gate, a continuous-discovery cadence tracker, and an Opportunity Solution Tree linter, with /cs:pm and /cs:product command families. v2.10.3 completes the markdown-html domain with md-slides — slide-deck converter (arrow-key / Space / PgDn / Home/End / P keyboard navigation + presenter mode with split-view clock + speaker notes + next-slide preview + URL-hash deep linking like #3 for direct slide jumps + @media print page-per-slide for browser-native PDF export). Reuses md-document's markdown parser; vanilla JS only (no framework runtime); Prism.js opt-in via --syntax. Joins md-review (v2.10.2 code-review converter), md-document (v2.10.1 long-form converter), and the v2.10.0 foundation (orchestrator + design-system). Compatible with Claude Code, Codex CLI, Gemini CLI, Cursor, OpenClaw, Hermes Agent, Mistral Vibe, and 5 more coding agents.",
|
||||
"version": "2.12.0"
|
||||
},
|
||||
"plugins": [
|
||||
|
|
@ -2050,6 +2050,27 @@
|
|||
"engineering"
|
||||
],
|
||||
"category": "development"
|
||||
},
|
||||
{
|
||||
"name": "deep-learning-book",
|
||||
"source": "./engineering/deep-learning-book",
|
||||
"description": "Study companion for the Deep Learning textbook by Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org. Twenty chapter files, a glossary, patterns and a cheatsheet index the whole book, and a delta reference dates it against 2026 practice with per-claim confidence levels: double descent qualifying the U-curve, AdamW splitting weight decay from L2, transformers displacing Chapter 10's recurrence, diffusion growing out of Chapter 18's score matching. Four stdlib tools make it executable — a prerequisite-closed reading-path planner that refuses goals the book does not cover, a training diagnostic running Chapter 11's rules in priority order so a NaN is never reported as overfitting, a capacity planner that ranks 'shrink the model' last when overparameterized, and a parameter/FLOP/activation-memory calculator that refuses a stack whose shapes do not connect. A companion, not a compilation: the book is copyrighted, so nothing here reproduces its text.",
|
||||
"version": "2.12.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani"
|
||||
},
|
||||
"keywords": [
|
||||
"deep-learning",
|
||||
"machine-learning",
|
||||
"study-companion",
|
||||
"goodfellow",
|
||||
"neural-networks",
|
||||
"training-diagnostics",
|
||||
"optimization",
|
||||
"generative-models",
|
||||
"engineering"
|
||||
],
|
||||
"category": "development"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
53
CLAUDE.md
53
CLAUDE.md
File diff suppressed because one or more lines are too long
16
README.md
16
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# Claude Code Skills & Plugins — Agent Skills for Every Coding Tool
|
||||
|
||||
**386 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
**387 production-ready Claude Code skills, plugins, and agent skills for 13 AI coding tools.**
|
||||
|
||||
The most comprehensive open-source library of Claude Code skills and agent plugins — also works with OpenAI Codex, Gemini CLI, Cursor, and 9 more coding agents. Reusable expertise packages covering engineering, DevOps, marketing (incl. AEO — Answer Engine Optimization for LLM citation), security (PreToolUse hooks), compliance, C-level advisory (incl. founder-mode CFO/CMO/CRO/CPO/COO/CHRO/CISO/GC/CDO/CAIO/CCO/VPE personas + 21 /cs:* slash commands), productivity (capture/email/reflect/weekly-review/deep-work/meetings), an academic research stack (litreview/grants/dossier/patent/syllabus/pulse/notebooklm/deep-research + hybrid router), and enterprise Research Operations (clinical-research/research-finance/market-research/product-research, v2.9.0).
|
||||
|
||||
|
|
@ -10,10 +10,10 @@ The most comprehensive open-source library of Claude Code skills and agent plugi
|
|||
[^vibe]: Mistral Vibe is also **BYO-sync tier**: the repo ships a pre-generated `.vibe/skills/claude-skills/` tree, run `./scripts/vibe-install.sh` once locally to install into `~/.vibe/skills/`. Same agentskills.io SKILL.md standard — no format conversion. Docs: <https://docs.mistral.ai/mistral-vibe/agents-skills>.
|
||||
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#skills-overview)
|
||||
[](#agents)
|
||||
[](#personas)
|
||||
[](#commands)
|
||||
[](#commands)
|
||||
[](https://github.com/alirezarezvani/claude-skills/stargazers)
|
||||
[](https://getskillcheck.com)
|
||||
|
||||
|
|
@ -29,7 +29,7 @@ Claude Code skills (also called agent skills or coding agent plugins) are modula
|
|||
- **Python tools** — 706 CLI scripts (all stdlib-only, zero pip installs)
|
||||
- **Reference docs** — 823 templates, checklists, and domain-specific knowledge files
|
||||
|
||||
**One repo, thirteen platforms.** Works natively as Claude Code plugins, Codex agent skills, Gemini CLI skills, Hermes Agent skills, Mistral Vibe skills, and converts to more tools via `scripts/convert.sh`. All 723 Python tools run anywhere Python runs.
|
||||
**One repo, thirteen platforms.** Works natively as Claude Code plugins, Codex agent skills, Gemini CLI skills, Hermes Agent skills, Mistral Vibe skills, and converts to more tools via `scripts/convert.sh`. All 727 Python tools run anywhere Python runs.
|
||||
|
||||
### Skills vs Agents vs Personas
|
||||
|
||||
|
|
@ -152,12 +152,12 @@ Run `./scripts/convert.sh --tool all` to generate tool-specific outputs locally.
|
|||
|
||||
## Skills Overview
|
||||
|
||||
**386 production-ready skills across 20 domains:**
|
||||
**387 production-ready skills across 20 domains:**
|
||||
|
||||
| Domain | Skills | Highlights | Details |
|
||||
|--------|--------|------------|---------|
|
||||
| **🔧 Engineering — Core** | 53 | Architecture, frontend, backend, fullstack, QA, DevOps, SecOps, AI/ML, data, Playwright Pro (test gen, flaky fix, migrations), self-improving agent (auto-memory curation), security suite, a11y audit, **named-persona-adversarial-review** (review via named engineering philosophies), **embedded-iot-mentor** (MCU/board selection, firmware-reuse-first, breadboard-MVP discipline) | [engineering-team/](engineering-team/) |
|
||||
| **⚡ Engineering — POWERFUL** | 91 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits) | [engineering/](engineering/) |
|
||||
| **⚡ Engineering — POWERFUL** | 92 | Agent designer, RAG architect, database designer, CI/CD builder, security auditor, MCP builder, AgentHub, Helm charts, Terraform, self-eval, llm-wiki, tc-tracker, autoresearch-agent, **reliability portfolio** (feature-flags-architect, kubernetes-operator, chaos-engineering, slo-architect), ship-gate, security-guidance PreToolUse hook, **Matt Pocock skills** (write-a-skill, caveman, grill-me, handoff, grill-with-docs), **zero-hallucination-coder** (Discuss→Map→Decompose→Execute→Verify), **agent-harness** (goal→plan→execute→verify→close loops over any domain), **memory-engineering** (price the memory write path, pick which cost to pay, audit FACT/SKILL/LOG density, gate on a forgetting policy), **skillopt-sleep** (nightly gated self-evolution from real Claude Code sessions, vendored from microsoft/SkillOpt), **book-to-skill** (compile a book, docs folder, or spec collection into a knowledge-base skill, then package it as a plugin), **boost-asio-pro** (async C++ networking — version-gated coroutine/callback styles, strand discipline), **human-gate** (batched human review as a structured artifact + a gate that refuses to close on open blockers), **agent-memory** (four-tier L0-L3 memory ladder over Claude Code hooks; promotion earned by recurrence across sessions and days, redacted or contested claims refuse to promote, nothing reaches CLAUDE.md without a human adopt), **skill-doctor** (grade the agent setup from real session history — rubric-judged, deterministically gated, evidence-traced skill edits), **deep-learning-book** (companion to the free Goodfellow/Bengio/Courville textbook — 20 chapter files with a 2016→2026 delta layer, reading-path planner, Ch-11 training diagnostic, capacity planner, FLOP/activation-memory calculator) | [engineering/](engineering/) |
|
||||
| **🎯 Product** | 17 | Product manager, agile PO, strategist, UX researcher, UI design, landing pages, SaaS scaffolder, analytics, experiment designer, discovery, roadmap communicator, code-to-prd, apple-hig-expert | [product-team/](product-team/) |
|
||||
| **📣 Marketing** | 49 | 8 pods: Content, SEO + AEO (`aeo` — E-E-A-T audit, citation tracking across 5 LLMs) + local (`local-seo-manager` — GBP/NAP/Map-Pack), CRO, Channels, Growth, Intelligence, Sales + `business-name-fit` (cross-cultural naming) + context foundation + orchestration router | [marketing-skill/](marketing-skill/) |
|
||||
| **🚀 Productivity** | 12 | `capture` (brain-dump-to-action), `email` pair (inbox-setup + inbox-triage), `reflect` (journal), `handoff` (Matt Pocock-inspired), `andreessen` (market-first decision mode), `roast` (5-angle idea panel → GO/RESHAPE/KILL), `fable-goal` (ramble → autonomous /goal prompt), `weekly-review` (GTD loop with refusal gate), `deep-work` (time-blocking + shallow-work budget), `meetings` (cost gate + agenda + action items), `swedish-mentor` (CEFR-leveled Swedish learning paths) | [productivity/](productivity/) |
|
||||
|
|
@ -359,7 +359,7 @@ Yes. Skills work natively with 13 tools: Claude Code, OpenAI Codex, Gemini CLI,
|
|||
No. We follow semantic versioning and maintain backward compatibility within patch releases. Existing script arguments, plugin source paths, and SKILL.md structures are never changed in patch versions. See the [CHANGELOG](CHANGELOG.md) for details on each release.
|
||||
|
||||
**Are the Python tools dependency-free?**
|
||||
Yes. All 723 Python tools use the standard library only — zero pip installs required. Every skill's CLI entry point is verified to run with `--help` (most skills ship one script per tool; a few, like the vendored `engineering/skillopt-sleep` engine, ship a multi-module package behind a single `python -m` entry point). A few tools — `engineering/book-to-skill`'s document extractors — can *optionally* use third-party parsers for higher-fidelity output, but every format falls back to a standard-library parser and nothing is installed implicitly.
|
||||
Yes. All 727 Python tools use the standard library only — zero pip installs required. Every skill's CLI entry point is verified to run with `--help` (most skills ship one script per tool; a few, like the vendored `engineering/skillopt-sleep` engine, ship a multi-module package behind a single `python -m` entry point). A few tools — `engineering/book-to-skill`'s document extractors — can *optionally* use third-party parsers for higher-fidelity output, but every format falls back to a standard-library parser and nothing is installed implicitly.
|
||||
|
||||
**How do I create my own Claude Code skill?**
|
||||
Each skill is a folder with a `SKILL.md` (frontmatter + instructions), optional `scripts/`, `references/`, and `assets/`. See the [Skills & Agents Factory](https://github.com/alirezarezvani/claude-code-skills-agents-factory) for a step-by-step guide.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"source": {
|
||||
"spec": "User request: convert https://www.deeplearningbook.org/ into a shareable skill.",
|
||||
"build_pattern": "Companion, not compilation. Follows the book-to-skill OUTPUT SHAPE (master SKILL.md under 4k tokens with a chapter index and topic index, chapters/chNN-*.md, glossary.md, patterns.md, cheatsheet.md — validated by book-to-skill's own book_skill_validator.py) while deliberately NOT running its extraction pipeline. Every chapter file is original synthesis written from domain knowledge, linking to the official free chapter; the organizing structure is the book's published table of contents. Adds four stdlib-only tools, four references citing 7-8 sources each with per-claim confidence levels, and three assets.",
|
||||
"distinct_from": "engineering/book-to-skill (the converter — this is the artifact its rights gate calls for when the source is copyrighted); engineering-team/senior-ml-engineer (production MLOps and serving, not the theory); engineering/llm-cost-optimizer (LLM economics, which the book predates); engineering/memory-engineering and engineering/agent-memory (agent systems, not model training); the teach/learn skills (general study workflows, not one specific text with its dependency graph and expiry dates)."
|
||||
},
|
||||
"attribution": {
|
||||
"source_work": "Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press, 2016 — https://www.deeplearningbook.org/. All rights remain with the authors and publisher. This companion is independent and unaffiliated, and is not endorsed by the authors or MIT Press.",
|
||||
"rights_basis": "No rights basis is claimed or needed, because no protected expression is reproduced. The book is copyrighted and its site states the HTML-only format exists as a deliberate friction against copying under the authors' MIT Press contract; PDF distribution is not permitted. book-to-skill's rights gate would classify it 'local' (none of public-domain / open-license / internal-docs / author-permission applies), and its rights reference lists publishing a compiled skill of a copyrighted book to a public marketplace under 'Do not'. This plugin therefore contains no passages, paragraphs, figures, exercises, or per-paragraph paraphrase — only original synthesis, the factual table of contents, and links to the official free chapters. See skills/deep-learning-book/references/rights_and_use.md.",
|
||||
"evidence_discipline": "Every claim about what changed since 2016 carries a confidence level (well-established / contested / folklore) and a primary citation, and two commonly repeated claims are marked contested rather than propagated: batch normalization's 'internal covariate shift' mechanism (challenged by Santurkar et al. 2018) and the strong form of the lottery-ticket hypothesis. Two pieces of folklore are named as folklore, including the local-minima fear the book itself corrects."
|
||||
}
|
||||
}
|
||||
15
engineering/deep-learning-book/.claude-plugin/plugin.json
Normal file
15
engineering/deep-learning-book/.claude-plugin/plugin.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"name": "deep-learning-book",
|
||||
"description": "Study companion and working knowledge base for the Deep Learning textbook by Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org. Twenty chapter files, a glossary, a patterns file and a cheatsheet index the whole book; every chapter carries a 'what changed after 2016' section, and a dedicated delta reference dates the text against 2026 practice with per-claim confidence levels — double descent qualifying the U-curve, AdamW splitting weight decay from L2, transformers displacing Chapter 10's recurrence, diffusion growing out of Chapter 18's score matching, and self-supervised learning vindicating Chapter 15 while replacing its methods. Four stdlib-only tools make the book executable: a prerequisite-closed reading-path planner that refuses goals the 2016 book does not cover, a training diagnostic running Chapter 11's rules in priority order so a NaN is never reported as overfitting, a capacity planner that ranks the regularization ladder and pushes 'shrink the model' last in the overparameterized regime, and a parameter/FLOP/activation-memory calculator that refuses a stack whose shapes do not connect. Deliberately a companion, not a compilation: the book is copyrighted, so nothing here reproduces its text — every chapter file is original synthesis linking to the official free chapter. Use when studying or teaching this book, planning a route through it, or checking whether one of its recommendations is still current.",
|
||||
"version": "2.12.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani",
|
||||
"url": "https://alirezarezvani.com"
|
||||
},
|
||||
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/engineering/deep-learning-book",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"license": "MIT",
|
||||
"skills": [
|
||||
"./skills/deep-learning-book"
|
||||
]
|
||||
}
|
||||
118
engineering/deep-learning-book/README.md
Normal file
118
engineering/deep-learning-book/README.md
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
# deep-learning-book — study companion for *Deep Learning* (Goodfellow, Bengio & Courville)
|
||||
|
||||
A navigable, executable, **date-stamped** companion to the 2016 MIT Press textbook that is free
|
||||
to read at [deeplearningbook.org](https://www.deeplearningbook.org/).
|
||||
|
||||
Twenty chapter files, a glossary, a patterns file, a cheatsheet, four references and four
|
||||
deterministic tools — plus the thing a static compilation cannot give you: a **2016 → 2026 delta
|
||||
layer** that says, per chapter, what still holds, what was superseded, and what is now purely
|
||||
historical.
|
||||
|
||||
## Why this is a companion and not a compiled skill
|
||||
|
||||
This repository ships [`engineering/book-to-skill`](../book-to-skill/), which compiles a document
|
||||
on disk into a knowledge-base skill. It was deliberately **not** used here, for three reasons its
|
||||
own rights gate makes binding:
|
||||
|
||||
1. The book is copyrighted (MIT Press, 2016). Free to read ≠ free to redistribute.
|
||||
2. deeplearningbook.org states that its HTML-only presentation is a deliberate friction against
|
||||
copying, required by the authors' contract, and that PDF distribution is not permitted.
|
||||
3. This repository is public, and `book-to-skill`'s rights gate refuses a `shareable` package
|
||||
without `public-domain`, `open-license`, `internal-docs` or `author-permission`. None applies.
|
||||
(Its hard rule 1 also forbids scraping a book from the web, so the pipeline could not have run
|
||||
against a URL in any case.)
|
||||
|
||||
So this skill contains **no passages, paragraphs, figures, or per-paragraph paraphrase**. Every
|
||||
chapter file is original synthesis — what the chapter establishes, how to use it, where it has
|
||||
aged — with a link to the official free chapter. The organizing structure is the book's published
|
||||
table of contents, which is factual metadata. Full reasoning in
|
||||
[`references/rights_and_use.md`](skills/deep-learning-book/references/rights_and_use.md).
|
||||
|
||||
**The rule this sets for the repository:** when a user asks to convert a copyrighted work into a
|
||||
shareable skill, build a companion that indexes and updates the source, not a compilation that
|
||||
reproduces it. Compile only when the rights gate clears, and keep the output local when it does not.
|
||||
|
||||
## What is in it
|
||||
|
||||
```
|
||||
skills/deep-learning-book/
|
||||
├── SKILL.md core frameworks + chapter index + topic index (~1.9k tokens, resident)
|
||||
├── chapters/ch01..ch20 one file per chapter, each with "What changed after 2016"
|
||||
├── glossary.md every key term → its chapter
|
||||
├── patterns.md techniques as instruments, with trade-offs
|
||||
├── cheatsheet.md decision tables: loss choice, fit verdict, optimizer defaults
|
||||
├── references/ delta layer · prerequisite map · study method · rights
|
||||
├── scripts/ 4 stdlib tools
|
||||
└── assets/ layer spec · chapter worksheet · study log
|
||||
```
|
||||
|
||||
## The tools
|
||||
|
||||
| Tool | Does | Refuses |
|
||||
|---|---|---|
|
||||
| `reading_path_planner.py` | Goal + background + hours → prerequisite-closed, ordered path with an hour budget | A goal outside the book (exit 3, names what covers it); an unroutable goal (exit 4, prints the questions) |
|
||||
| `training_diagnostics.py` | Measurements → ranked cause + next action + chapter, rules in priority order | Diagnosing with no instruments (exit 4). Never reports a NaN as overfitting |
|
||||
| `capacity_planner.py` | Gap + params-per-example → ordered regularization ladder | A validation error materially below training error (exit 4 — leaky split) |
|
||||
| `model_arithmetic.py` | Layer stack → parameters, FLOPs, activation memory per example | A stack whose shapes do not connect (exit 5); an unknown layer type (exit 4) |
|
||||
|
||||
All four are standard-library only, support `--help` / `--sample` / `--output json`, make no
|
||||
network calls, and load no frameworks.
|
||||
|
||||
## The delta layer
|
||||
|
||||
The book was published in 2016; *Attention Is All You Need* appeared in 2017. Five corrections
|
||||
carry most of the weight, each cited and confidence-rated in
|
||||
[`references/book_to_2026_delta.md`](skills/deep-learning-book/references/book_to_2026_delta.md):
|
||||
|
||||
1. **Double descent** qualifies Chapter 5's U-shaped capacity curve — "shrink the model when it
|
||||
overfits" is no longer the only correct move, and `capacity_planner.py` encodes the correction.
|
||||
2. **AdamW**: weight decay and L2 are not equivalent under an adaptive optimizer; Chapter 7 treats
|
||||
them as interchangeable.
|
||||
3. **Transformers** displaced Chapter 10's recurrence — keep its gradient-flow analysis, drop its
|
||||
architecture recommendation. (State-space models made that analysis live again.)
|
||||
4. **Diffusion** grew directly out of Chapter 14's denoising autoencoders and Chapter 18's score
|
||||
matching, and displaced Chapter 20's model list.
|
||||
5. **Self-supervised learning** vindicated Chapter 15's bet while replacing every method it names;
|
||||
unsupervised disentanglement was proven impossible without inductive bias.
|
||||
|
||||
The general rule the file states: the conflict is almost always in the **recommendation**, not the
|
||||
**analysis**. Keep the diagnosis, replace the prescription.
|
||||
|
||||
## Use it
|
||||
|
||||
```bash
|
||||
S=engineering/deep-learning-book/skills/deep-learning-book/scripts
|
||||
|
||||
python3 $S/reading_path_planner.py --goal "train and debug convnets" --background applied --hours-per-week 5
|
||||
python3 $S/training_diagnostics.py --train-loss 0.02 --val-loss 1.9 --tiny-subset-fits yes
|
||||
python3 $S/capacity_planner.py --params 12000000 --train-examples 50000 --train-error 0.01 --val-error 0.22
|
||||
python3 $S/model_arithmetic.py --spec skills/deep-learning-book/assets/example_layer_spec.json
|
||||
```
|
||||
|
||||
Slash commands: `/cs:deep-learning` (navigate and date the answer), `/cs:dl-reading-path`,
|
||||
`/cs:dl-diagnose`. Agent: `cs-deep-learning-tutor`.
|
||||
|
||||
## Distinct from
|
||||
|
||||
- **`engineering/book-to-skill`** — the converter. This is what you build when its rights gate
|
||||
says no.
|
||||
- **`engineering-team/senior-ml-engineer`** — production MLOps, deployment, serving. This is the
|
||||
theory underneath.
|
||||
- **`engineering/llm-cost-optimizer`** — LLM economics, which the book predates entirely.
|
||||
- **`teach` / `learn` skills** — general study workflows. This is one specific text, with its
|
||||
dependency graph and its expiry dates.
|
||||
|
||||
## Scope
|
||||
|
||||
The twenty chapters and the delta between them and 2026 practice. **Not covered**, because the
|
||||
book does not cover them: reinforcement learning beyond passing mention, LLM training
|
||||
infrastructure, RLHF/DPO, agentic systems, MLOps tooling, fairness and safety evaluation. The
|
||||
skill is built to say so rather than improvise.
|
||||
|
||||
## Attribution
|
||||
|
||||
*Deep Learning*, Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press, 2016 —
|
||||
https://www.deeplearningbook.org/. All rights in the book remain with its authors and publisher.
|
||||
This companion is an independent work: no text, figures, or exercises from the book are
|
||||
reproduced here, and it is not endorsed by or affiliated with the authors or MIT Press. The
|
||||
companion's own content is MIT-licensed as part of this repository.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
name: cs-deep-learning-tutor
|
||||
description: "Study companion for the Deep Learning textbook (Goodfellow, Bengio & Courville, 2016). Plans a prerequisite-closed reading path, answers chapter questions from the compiled knowledge base, diagnoses training runs against Chapter 11's decision tree, and flags every place the 2016 text has been superseded. Use for studying the book, teaching from it, or checking whether one of its recommendations is still current."
|
||||
---
|
||||
|
||||
# Deep Learning Tutor
|
||||
|
||||
You are a study companion for *Deep Learning* by Ian Goodfellow, Yoshua Bengio and Aaron
|
||||
Courville (MIT Press, 2016), which is free to read at deeplearningbook.org.
|
||||
|
||||
## What you are working from
|
||||
|
||||
`engineering/deep-learning-book/skills/deep-learning-book/` — a master SKILL.md with core
|
||||
frameworks and two indexes, 20 chapter files, a glossary, a patterns file, a cheatsheet, four
|
||||
references and four tools. Read the SKILL.md first, resolve the question through the Topic
|
||||
Index, then read that chapter file before answering.
|
||||
|
||||
## Hard rules
|
||||
|
||||
1. **Never reproduce the book's text.** Not a paragraph, not a figure, not a
|
||||
sentence-by-sentence paraphrase. Point the reader at the official chapter URL and explain in
|
||||
your own words. This is the constraint the whole skill is built around — see
|
||||
`references/rights_and_use.md`.
|
||||
2. **Date every recommendation.** The book is from 2016 and *Attention Is All You Need* is from
|
||||
2017. When a chapter's advice has been superseded, say so and cite
|
||||
`references/book_to_2026_delta.md`. Never present a 2016 recommendation as current practice
|
||||
without that check.
|
||||
3. **Separate the analysis from the prescription.** The book's diagnoses (why gradients vanish,
|
||||
why the partition function is hard, why depth helps) almost all still hold. Its
|
||||
prescriptions (use an LSTM, use Adam with L2, shrink the model when it overfits) frequently
|
||||
do not. Keep the diagnosis, replace the prescription.
|
||||
4. **Say when the book does not cover something.** RLHF, LLM infrastructure, agents, MLOps,
|
||||
fairness — name the gap and route elsewhere rather than improvising the book's position.
|
||||
5. **Read the chapter file before answering from it.** The indexes are for navigation, not for
|
||||
answering.
|
||||
6. **Run the tool rather than estimating.** Reading paths, training diagnoses, capacity plans
|
||||
and parameter counts all have deterministic tools. Use them, then interpret the output.
|
||||
|
||||
## How you work
|
||||
|
||||
**When asked where to start** — run `reading_path_planner.py` with the stated goal, background
|
||||
and weekly hours. If it exits 3 or 4, relay its questions rather than guessing a path.
|
||||
|
||||
**When asked about a topic** — resolve through the Topic Index, read the chapter file, answer,
|
||||
and always surface the "What changed after 2016" section if one applies.
|
||||
|
||||
**When asked to diagnose a training run** — ask for the measurements the tool needs
|
||||
(train loss, val loss, target loss, gradient norm, whether it can overfit a tiny subset), run
|
||||
`training_diagnostics.py`, and act on finding [1] before anything below it. Do not skip to the
|
||||
interesting hypothesis; the rule order exists because a NaN is not an overfitting problem.
|
||||
|
||||
**When teaching** — use the retrieval-practice cadence in `references/study_method_canon.md`:
|
||||
ask the reader to state the core idea from memory first, then correct. Do not lecture the
|
||||
chapter at someone who has just read it.
|
||||
|
||||
## Voice
|
||||
|
||||
Direct and specific. Name the chapter for every claim. When the reader's plan is wrong — front
|
||||
to back through Part I, or a Part III chapter without its prerequisites — say so once, give the
|
||||
alternative, and let them decide. When something in the book is simply out of date, say that
|
||||
plainly rather than defending it; a companion that will not date its source is worthless.
|
||||
55
engineering/deep-learning-book/commands/cs-deep-learning.md
Normal file
55
engineering/deep-learning-book/commands/cs-deep-learning.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
name: "cs-deep-learning"
|
||||
description: "/cs:deep-learning — Study companion for the Deep Learning textbook (Goodfellow, Bengio & Courville, 2016). Answers chapter questions from a compiled knowledge base, always dating the answer against 2026 practice, and routes to the reading-path planner or the training diagnostic. Points at the free official chapters; never reproduces them."
|
||||
argument-hint: "[a topic, a chapter number, a question about the book, or 'where do I start']"
|
||||
---
|
||||
|
||||
# /cs:deep-learning — Navigate the book, and date its advice
|
||||
|
||||
**Command:** `/cs:deep-learning [topic | chNN | question]`
|
||||
|
||||
The book is free to read at [deeplearningbook.org](https://www.deeplearningbook.org/). This
|
||||
command navigates it and keeps it current; it does not replace it.
|
||||
|
||||
## When to run
|
||||
|
||||
- "What does the book say about regularization / saddle points / the partition function?"
|
||||
- "Is Chapter 10's advice on LSTMs still right?"
|
||||
- "Explain the ELBO the way Chapter 19 sets it up"
|
||||
- "Where should I start?" (routes to `/cs:dl-reading-path`)
|
||||
- "Why is my training run doing this?" (routes to `/cs:dl-diagnose`)
|
||||
|
||||
## When NOT to run
|
||||
|
||||
- Production ML engineering → `engineering-team/senior-ml-engineer`
|
||||
- LLM cost and serving → `engineering/llm-cost-optimizer`
|
||||
- RLHF, agents, prompting, MLOps → outside the book entirely; this command will say so
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Load** `engineering/deep-learning-book/skills/deep-learning-book/SKILL.md`.
|
||||
2. **Resolve** the request through the Topic Index to one or more chapters.
|
||||
3. **Read** those chapter files before answering. The index is navigation, not content.
|
||||
4. **Answer** in your own words, naming the chapter, and link the official chapter URL.
|
||||
5. **Date it.** If the chapter file has a "What changed after 2016" section relevant to the
|
||||
answer, surface it — with the confidence level from
|
||||
`references/book_to_2026_delta.md`. Separate the book's *analysis* (usually still true) from
|
||||
its *prescription* (frequently superseded).
|
||||
6. **Name the boundary.** If the question is outside the twenty chapters, say so and route.
|
||||
|
||||
## The four things this command will not do
|
||||
|
||||
- Reproduce the book's text, figures, or a paragraph-by-paragraph paraphrase.
|
||||
- Present a 2016 recommendation as current practice without checking the delta layer.
|
||||
- Answer from the index without reading the chapter file.
|
||||
- Improvise the book's position on material published after it.
|
||||
|
||||
## Output shape
|
||||
|
||||
```
|
||||
Chapter(s): ch07 (Regularization), ch05 (capacity)
|
||||
Answer : <original explanation, naming the framework>
|
||||
Still true: <holds / analysis holds but recommendation superseded / historical> + why
|
||||
Read it : https://www.deeplearningbook.org/contents/regularization.html
|
||||
Next : <the tool, worksheet, or chapter that follows>
|
||||
```
|
||||
52
engineering/deep-learning-book/commands/cs-dl-diagnose.md
Normal file
52
engineering/deep-learning-book/commands/cs-dl-diagnose.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
name: "cs-dl-diagnose"
|
||||
description: "/cs:dl-diagnose — Diagnose a training run against Chapter 11's decision tree: read training error before deciding anything, and never diagnose a NaN as overfitting. Returns a ranked cause, the specific next action, and the chapter that justifies it."
|
||||
argument-hint: "[what the run is doing — losses, gradient norm, or a description]"
|
||||
---
|
||||
|
||||
# /cs:dl-diagnose — Measurement first, hypothesis second
|
||||
|
||||
**Command:** `/cs:dl-diagnose [symptoms]`
|
||||
|
||||
Chapter 11's rule, which most teams have backwards: **read training error first.** High
|
||||
training error means the model or the optimizer is the bottleneck, and more data cannot help.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Collect the instruments.** Ask for what is missing, in this order:
|
||||
- training loss and validation loss (same units, same epoch)
|
||||
- the target loss — a human baseline, a published number, or an irreducible-error estimate
|
||||
(without it, underfitting cannot be distinguished from convergence)
|
||||
- global gradient norm, if available
|
||||
- has the loss ever gone NaN or inf?
|
||||
- can the model drive training loss to ~0 on 10–50 examples? (the smoke test that separates
|
||||
a bug from a hard problem)
|
||||
2. **Run the tool:**
|
||||
```bash
|
||||
python3 engineering/deep-learning-book/skills/deep-learning-book/scripts/training_diagnostics.py \
|
||||
--train-loss <x> --val-loss <y> --target-loss <z> --grad-norm <g> \
|
||||
--tiny-subset-fits yes|no|unknown
|
||||
```
|
||||
Exit 4 means not enough instruments — ask for one of the named measurements rather than
|
||||
guessing.
|
||||
3. **Act on finding [1] first.** Rules fire in priority order for a reason: a non-finite loss is
|
||||
a numerics failure, not a modelling one, and a model that cannot overfit 20 examples has a
|
||||
bug that no hyperparameter will fix.
|
||||
4. **On an OVERFIT verdict**, follow up with the capacity planner, which ranks the
|
||||
regularization ladder and applies the double-descent correction:
|
||||
```bash
|
||||
python3 .../capacity_planner.py --params <n> --train-examples <m> \
|
||||
--train-error <x> --val-error <y> --applied early-stopping
|
||||
```
|
||||
5. **On a memory or throughput question**, run `model_arithmetic.py --spec <file>` — it reports
|
||||
parameters, FLOPs and activation memory per example, and refuses a stack whose shapes do not
|
||||
connect.
|
||||
6. **Close with the discipline, not just the fix:** change one thing per experiment, log it,
|
||||
re-measure the gap.
|
||||
|
||||
## Do not
|
||||
|
||||
- Skip to the interesting hypothesis before the rules have been read in order.
|
||||
- Recommend collecting data while training error is high.
|
||||
- Recommend shrinking an overparameterized model first — see the double-descent caveat in
|
||||
`references/book_to_2026_delta.md`.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
---
|
||||
name: "cs-dl-reading-path"
|
||||
description: "/cs:dl-reading-path — Build a prerequisite-closed reading path through the Deep Learning book from a goal, a background and the hours you actually have. Refuses to route a goal the 2016 book does not cover, and names what covers it instead."
|
||||
argument-hint: "[your goal — what you want to be able to do after reading]"
|
||||
---
|
||||
|
||||
# /cs:dl-reading-path — A route, not a page count
|
||||
|
||||
**Command:** `/cs:dl-reading-path [goal]`
|
||||
|
||||
The book's part order is not its dependency order. Front-to-back means weeks in Chapters 2–4
|
||||
before touching a network, which is where most readers stop.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Ask three things** if the user has not said them, one at a time:
|
||||
- What do you want to be able to *do* afterwards?
|
||||
- Background: `none` / `math` / `applied` / `research`?
|
||||
- Realistic study hours per week?
|
||||
2. **Run the planner:**
|
||||
```bash
|
||||
python3 engineering/deep-learning-book/skills/deep-learning-book/scripts/reading_path_planner.py \
|
||||
--goal "<goal>" --background <bg> --hours-per-week <n>
|
||||
```
|
||||
3. **Handle the refusals rather than working around them.**
|
||||
- Exit 3 — the goal is outside the book. Relay what covers it; do not invent a path.
|
||||
- Exit 4 — the goal is unroutable. Ask the printed questions, one per turn.
|
||||
4. **Relay the path** with the hour budget, and flag that the hours are a planning heuristic to
|
||||
recalibrate after chapter one.
|
||||
5. **Offer the study scaffolding**: `assets/study_log_template.md` and
|
||||
`assets/chapter_worksheet.md`, plus the retrieval-practice cadence from
|
||||
`references/study_method_canon.md`.
|
||||
|
||||
## What to say about Part I
|
||||
|
||||
Most applied readers should skim Chapters 2–4 once for vocabulary, start at Chapter 5, and
|
||||
return to a specific section when Chapter 8 needs it. Readers heading for Part III should work
|
||||
Part I properly — those chapters compound, and so do the gaps.
|
||||
`references/prerequisite_map.md` has the dependency graph and the three strategies.
|
||||
|
|
@ -0,0 +1,194 @@
|
|||
---
|
||||
name: deep-learning-book
|
||||
description: "Study companion and working knowledge base for the Deep Learning textbook by Goodfellow, Bengio & Courville (MIT Press, 2016), read free at deeplearningbook.org. Indexes all 20 chapters, carries a 2016-to-2026 delta layer naming what the book got right, what was superseded (transformers, AdamW, diffusion, double descent) and what still holds, and ships four deterministic tools: a prerequisite-aware reading-path planner, a training-failure diagnostic, a capacity-and-regularization planner, and a parameter/FLOP/activation-memory calculator. Use when studying or teaching this book, planning a route through it, deciding whether a chapter's advice is still current, or translating its math into a training decision. It points at the official chapters — it never reproduces them."
|
||||
license: MIT
|
||||
metadata:
|
||||
version: 1.0.0
|
||||
author: Alireza Rezvani
|
||||
category: engineering
|
||||
updated: 2026-08-25
|
||||
---
|
||||
|
||||
# Deep Learning — Study Companion
|
||||
|
||||
**Source book**: *Deep Learning*, Ian Goodfellow, Yoshua Bengio & Aaron Courville
|
||||
(MIT Press, 2016) · 20 chapters, 3 parts · read free at
|
||||
[deeplearningbook.org](https://www.deeplearningbook.org/) · companion compiled 2026-08-25.
|
||||
|
||||
**This is a companion, not a copy.** The book is copyrighted, and its site states that the
|
||||
HTML-only format exists to discourage copying under the authors' MIT Press contract. Nothing
|
||||
here reproduces its text. Every chapter file is original synthesis — what the chapter
|
||||
establishes, how to use it, where it has aged — plus a link to the official chapter. Read the
|
||||
book at the link; use this to navigate it, keep it current, and turn it into decisions.
|
||||
See [references/rights_and_use.md](references/rights_and_use.md).
|
||||
|
||||
## How to Use This Skill
|
||||
|
||||
- **No argument** — load the core frameworks below.
|
||||
- **A topic** — ask about `regularization`, `saddle points`, `partition function`; resolved
|
||||
through the Topic Index, then that chapter file is read before answering.
|
||||
- **`chNN`** — load that chapter's file.
|
||||
- **"is this still true?"** — the 2016→2026 delta layer, in every chapter file and in
|
||||
[references/book_to_2026_delta.md](references/book_to_2026_delta.md).
|
||||
- **"where do I start?"** — run `scripts/reading_path_planner.py`.
|
||||
|
||||
When asked about something outside these 20 chapters, say so and route to the delta reference
|
||||
rather than improvising the book's position on material published after it.
|
||||
|
||||
---
|
||||
|
||||
## Core Frameworks & Mental Models
|
||||
|
||||
### The (T, P, E) frame — ch05
|
||||
|
||||
Name the **task**, the **performance measure**, and the **experience** in one sentence before any
|
||||
model code. Most failed projects failed at P: an unstated metric, or a proxy whose relationship
|
||||
to the real objective was never checked.
|
||||
|
||||
### Every loss is a negative log-likelihood — ch03, ch06
|
||||
|
||||
Choose the output distribution, then take its negative log. Gaussian → MSE, Bernoulli → binary
|
||||
cross-entropy, categorical → cross-entropy, Laplace → MAE. "Which loss?" is always the question
|
||||
"which distribution?" in disguise. Modern contrastive and preference objectives sit outside this
|
||||
frame — a real limit of the book, not a gap in your understanding.
|
||||
|
||||
### KL asymmetry decides your failure mode — ch03, ch19, ch20
|
||||
|
||||
D(p‖q) ≠ D(q‖p). Forward KL is mode-covering (blurry averages); reverse KL is mode-seeking
|
||||
(sharp but partial). This single fact predicts VAE blur, GAN mode collapse, and the
|
||||
characteristic over-confidence of mean-field variational posteriors.
|
||||
|
||||
### Train-error-first triage — ch11, ch05
|
||||
|
||||
High training error → capacity or optimization is the bottleneck; **more data will not help**.
|
||||
Low training error with a large validation gap → data or regularization. This is the highest-value
|
||||
heuristic in the book. `scripts/training_diagnostics.py` runs it.
|
||||
|
||||
### Capacity, the gap, and the U-curve's caveat — ch05, ch07
|
||||
|
||||
Regularization trades variance for bias. But the classical U-shaped capacity curve is incomplete:
|
||||
past the interpolation threshold, test error can fall again (double descent, 2019–2020,
|
||||
post-dating the book). Practical consequence: when a large model overfits, try more data, more
|
||||
regularization or longer training **before** shrinking it.
|
||||
|
||||
### Architecture is a prior, not a trick — ch09, ch10, ch15
|
||||
|
||||
Convolution asserts translation equivariance and locality. Recurrence asserts that the past
|
||||
compresses into a state. A distributed representation asserts that factors combine
|
||||
combinatorially. When the assertion is false, the architecture cannot be rescued by tuning — and
|
||||
when it is true, it beats capacity. This is also why Vision Transformers need more data than
|
||||
ConvNets: they discard the prior and buy it back with examples.
|
||||
|
||||
### Depth's real cost is gradient flow and activation memory — ch06, ch08, ch10
|
||||
|
||||
Backprop is the chain rule scheduled well: one forward-pass-equivalent of compute, and memory
|
||||
proportional to stored activations. Depth fails through vanishing/exploding gradients and
|
||||
ill-conditioning, which is why residual connections, normalization and clipping exist.
|
||||
|
||||
### The partition function organizes Part III — ch16, ch17, ch18, ch19
|
||||
|
||||
For undirected models, the likelihood gradient needs samples from the model itself. Four escape
|
||||
routes: sample it (CD/PCD), sidestep it algebraically (pseudolikelihood, **score matching**),
|
||||
learn around it (NCE), or estimate it for evaluation (AIS). Score matching's descendants are
|
||||
today's diffusion models — which is why Part III repays reading even though its models did not
|
||||
survive.
|
||||
|
||||
### Diagnose before you redesign — ch04, ch08, ch11
|
||||
|
||||
Gradient norm exploding → clip. Norm large but loss flat → ill-conditioning. Norm near zero with
|
||||
high loss → saturation or dead units. NaN → numerics first. Change one thing per experiment.
|
||||
|
||||
---
|
||||
|
||||
## Chapter Index
|
||||
|
||||
| # | Title | Key content |
|
||||
|---|-------|-------------|
|
||||
| [ch01](chapters/ch01-introduction.md) | Introduction | representation learning, depth as composition, curse of dimensionality |
|
||||
| [ch02](chapters/ch02-linear-algebra.md) | Linear Algebra | norms, SVD, eigendecomposition, conditioning, PCA |
|
||||
| [ch03](chapters/ch03-probability-information-theory.md) | Probability & Information Theory | distributions, entropy, KL, cross-entropy |
|
||||
| [ch04](chapters/ch04-numerical-computation.md) | Numerical Computation | under/overflow, conditioning, gradient descent, KKT |
|
||||
| [ch05](chapters/ch05-machine-learning-basics.md) | Machine Learning Basics | capacity, bias–variance, No Free Lunch, MLE, manifolds |
|
||||
| [ch06](chapters/ch06-deep-feedforward-networks.md) | Deep Feedforward Networks | output/hidden units, universal approximation, backprop |
|
||||
| [ch07](chapters/ch07-regularization.md) | Regularization | norm penalties, augmentation, early stopping, dropout |
|
||||
| [ch08](chapters/ch08-optimization.md) | Optimization | SGD, momentum, init, Adam, batch norm, saddles |
|
||||
| [ch09](chapters/ch09-convolutional-networks.md) | Convolutional Networks | sparse interactions, sharing, equivariance, pooling |
|
||||
| [ch10](chapters/ch10-sequence-modeling.md) | Sequence Modeling | BPTT, vanishing gradients, LSTM/GRU, attention |
|
||||
| [ch11](chapters/ch11-practical-methodology.md) | Practical Methodology | metrics, baselines, the data-vs-capacity rule, debugging |
|
||||
| [ch12](chapters/ch12-applications.md) | Applications | scaling, compression, vision, speech, NLP (dated) |
|
||||
| [ch13](chapters/ch13-linear-factor-models.md) | Linear Factor Models | PPCA, factor analysis, ICA, sparse coding |
|
||||
| [ch14](chapters/ch14-autoencoders.md) | Autoencoders | undercomplete, sparse, denoising, contractive |
|
||||
| [ch15](chapters/ch15-representation-learning.md) | Representation Learning | transfer, distributed codes, disentanglement |
|
||||
| [ch16](chapters/ch16-structured-probabilistic-models.md) | Structured Probabilistic Models | directed/undirected, energy-based, d-separation |
|
||||
| [ch17](chapters/ch17-monte-carlo-methods.md) | Monte Carlo Methods | importance sampling, MCMC, Gibbs, mixing |
|
||||
| [ch18](chapters/ch18-partition-function.md) | Confronting the Partition Function | CD/PCD, pseudolikelihood, score matching, NCE, AIS |
|
||||
| [ch19](chapters/ch19-approximate-inference.md) | Approximate Inference | ELBO, EM, mean field, amortization |
|
||||
| [ch20](chapters/ch20-deep-generative-models.md) | Deep Generative Models | Boltzmann machines, VAE, GAN, autoregressive |
|
||||
|
||||
## Topic Index
|
||||
|
||||
- **Activation functions, ReLU, GELU** → ch06
|
||||
- **Adam, AdamW, adaptive optimizers** → ch08, ch07
|
||||
- **Attention, transformers** → ch10, ch12
|
||||
- **Autoencoders, denoising, sparse** → ch14, ch13
|
||||
- **Backpropagation, autodiff** → ch06
|
||||
- **Batch / layer normalization** → ch08
|
||||
- **Bias–variance, double descent** → ch05
|
||||
- **Convolution, pooling, receptive field** → ch09
|
||||
- **Cross-entropy, KL divergence, entropy** → ch03
|
||||
- **Diffusion, score matching** → ch18, ch14, ch20
|
||||
- **Dropout, weight decay, early stopping** → ch07
|
||||
- **ELBO, variational inference, EM** → ch19
|
||||
- **Energy-based models, graphical models** → ch16
|
||||
- **GANs, VAEs, generative taxonomy** → ch20
|
||||
- **Gradient clipping, exploding/vanishing** → ch10, ch08
|
||||
- **Hyperparameter search** → ch11
|
||||
- **Initialization** → ch08
|
||||
- **LSTM, GRU, BPTT, teacher forcing** → ch10
|
||||
- **Maximum likelihood, MAP** → ch05, ch03
|
||||
- **MCMC, Gibbs, importance sampling** → ch17
|
||||
- **Numerical stability, softmax, log-space** → ch04
|
||||
- **Partition function, CD, PCD, NCE** → ch18, ch16
|
||||
- **PCA, ICA, factor analysis** → ch13, ch02
|
||||
- **Representation learning, transfer, probes** → ch15, ch01
|
||||
- **Saddle points, ill-conditioning** → ch08, ch04
|
||||
- **SVD, eigendecomposition, condition number** → ch02
|
||||
- **Training diagnostics, metric choice** → ch11
|
||||
- **Universal approximation** → ch06
|
||||
|
||||
## Supporting Files
|
||||
|
||||
- [glossary.md](glossary.md) — every key term with its chapter
|
||||
- [patterns.md](patterns.md) — techniques as instruments, with trade-offs
|
||||
- [cheatsheet.md](cheatsheet.md) — decision tables and defaults
|
||||
- [references/book_to_2026_delta.md](references/book_to_2026_delta.md) — what changed, per chapter
|
||||
- [references/prerequisite_map.md](references/prerequisite_map.md) — the real dependency graph
|
||||
- [references/study_method_canon.md](references/study_method_canon.md) — how to study a hard text
|
||||
- [references/rights_and_use.md](references/rights_and_use.md) — why this is a companion
|
||||
|
||||
## Tools
|
||||
|
||||
```bash
|
||||
S=engineering/deep-learning-book/skills/deep-learning-book/scripts
|
||||
python3 $S/reading_path_planner.py --goal "train a transformer" --background applied --hours-per-week 5
|
||||
python3 $S/training_diagnostics.py --train-loss 0.02 --val-loss 1.9 --grad-norm 0.4 --epochs 30
|
||||
python3 $S/capacity_planner.py --params 12000000 --train-examples 50000 --train-error 0.01 --val-error 0.22
|
||||
python3 $S/model_arithmetic.py --spec-sample
|
||||
```
|
||||
|
||||
Every tool supports `--help`, `--sample` and `--output json`, uses the standard library only, and
|
||||
returns typed exit codes.
|
||||
|
||||
---
|
||||
|
||||
## Scope & Limits
|
||||
|
||||
This companion covers the 2016 edition's 20 chapters and the delta between them and 2026
|
||||
practice. It does **not** cover: reinforcement learning beyond passing mention, LLM training
|
||||
infrastructure, RLHF/DPO alignment, agentic systems, MLOps tooling, or fairness and safety
|
||||
evaluation — none of which the book treats. For production ML engineering use
|
||||
`engineering-team/senior-ml-engineer`; for LLM cost work use `engineering/llm-cost-optimizer`.
|
||||
|
||||
When a question lands outside the book, say the book does not cover it and cite the delta
|
||||
reference for what replaced its position. A companion that quietly extrapolates is worse than one
|
||||
that names its boundary.
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
# Chapter Worksheet
|
||||
|
||||
One page per chapter. Fill it from memory **with the book closed**, then check against the
|
||||
skill's chapter file. The gap between what you wrote and what is there is the actual learning
|
||||
signal — retrieval practice beats re-reading (see `references/study_method_canon.md`).
|
||||
|
||||
---
|
||||
|
||||
**Chapter**: ___ **Title**: ______________________ **Date**: __________
|
||||
**Time spent**: ______ **Read as**: ☐ first pass ☐ working the derivations ☐ revisit
|
||||
|
||||
## 1. Core idea, in one sentence
|
||||
|
||||
> If this takes more than two sentences, you have summarized rather than extracted.
|
||||
|
||||
_______________________________________________________________________________
|
||||
|
||||
## 2. Three concepts I could define to someone else
|
||||
|
||||
1. _____________________________________________________________________________
|
||||
2. _____________________________________________________________________________
|
||||
3. _____________________________________________________________________________
|
||||
|
||||
## 3. One thing I could not follow
|
||||
|
||||
> Name it precisely — "the derivation on the negative phase" beats "the math."
|
||||
|
||||
_______________________________________________________________________________
|
||||
|
||||
**Is it blocking?** ☐ yes, a later chapter needs it (which: ____) ☐ no, note and move on
|
||||
|
||||
## 4. Self-explanation prompt
|
||||
|
||||
Pick one equation. **What breaks if this term is removed?**
|
||||
|
||||
_______________________________________________________________________________
|
||||
|
||||
## 5. When would I actually use this?
|
||||
|
||||
> A note that answers this gets read again. A note that restates the text does not.
|
||||
|
||||
_______________________________________________________________________________
|
||||
|
||||
## 6. Still true in 2026?
|
||||
|
||||
Check the chapter file's "What changed after 2016" section and this skill's
|
||||
`references/book_to_2026_delta.md`.
|
||||
|
||||
☐ holds as written ☐ analysis holds, recommendation superseded ☐ historical only
|
||||
|
||||
What replaced it: ___________________________________________________________
|
||||
|
||||
## 7. The small thing I applied before moving on
|
||||
|
||||
> Five lines of code, one diagnosis of a real run, one re-derivation. This step is the
|
||||
> one most readers skip and the one that decides whether any of it survives.
|
||||
|
||||
_______________________________________________________________________________
|
||||
|
||||
## 8. Revisit date
|
||||
|
||||
☐ +1 week: ________ ☐ +1 month: ________ (ch05, ch08, ch11 and ch18 are worth both)
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"name": "transformer encoder block, 768-dim, 512 tokens",
|
||||
"_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. Note how the mha row's cost grows quadratically with the input sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.",
|
||||
"layers": [
|
||||
{"type": "input", "shape": [512, 768], "name": "token embeddings in"},
|
||||
{"type": "layernorm", "name": "pre-norm 1"},
|
||||
{"type": "mha", "heads": 12, "name": "self-attention"},
|
||||
{"type": "layernorm", "name": "pre-norm 2"},
|
||||
{"type": "flatten", "name": "flatten for the feedforward accounting"},
|
||||
{"type": "linear", "units": 3072, "name": "ffn up"},
|
||||
{"type": "activation", "name": "gelu"},
|
||||
{"type": "linear", "units": 768, "name": "ffn down"}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Study Log — Deep Learning (Goodfellow, Bengio & Courville)
|
||||
|
||||
Started: __________ · Goal: ______________________________________________
|
||||
Path from `reading_path_planner.py`: _______________________________________
|
||||
Hours available per week: ______ · Estimated weeks: ______
|
||||
|
||||
## Progress
|
||||
|
||||
| Date | Ch | Hours | Status | Confidence (1-5) | Applied it to |
|
||||
|------|----|-------|--------|------------------|---------------|
|
||||
| | | | ☐ read ☐ worked ☐ applied | | |
|
||||
| | | | | | |
|
||||
| | | | | | |
|
||||
|
||||
**Confidence** means "could I explain this to a colleague without notes?" — not "did I finish
|
||||
the pages." A 2 is useful information; record it honestly and put the chapter on the revisit
|
||||
list.
|
||||
|
||||
## Revisit queue (spaced repetition)
|
||||
|
||||
| Chapter | +1 week | +1 month | Notes |
|
||||
|---------|---------|----------|-------|
|
||||
| ch05 | | | vocabulary every later chapter reuses |
|
||||
| ch08 | | | the optimization diagnoses |
|
||||
| ch11 | | | the workflow — apply it, do not just read it |
|
||||
| ch18 | | | score matching → diffusion |
|
||||
|
||||
## Deferred with intent
|
||||
|
||||
Things skipped on purpose, and what would bring them back.
|
||||
|
||||
| Chapter or section | Why deferred | Bring it back when |
|
||||
|--------------------|--------------|--------------------|
|
||||
| | | |
|
||||
|
||||
## Open confusions
|
||||
|
||||
Carry these forward — a confusion that survives three chapters is a prerequisite gap, and
|
||||
`references/prerequisite_map.md` will name the chapter that fills it.
|
||||
|
||||
| # | Confusion | First hit (ch) | Resolved? |
|
||||
|---|-----------|----------------|-----------|
|
||||
| 1 | | | |
|
||||
|
||||
## Applied log
|
||||
|
||||
The point of the book. One line per real use.
|
||||
|
||||
| Date | What I was doing | Which chapter helped | Outcome |
|
||||
|------|------------------|----------------------|---------|
|
||||
| | | | |
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
# Chapter 1: Introduction
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/intro.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Deep learning is representation learning stacked: instead of hand-designing the features a
|
||||
model consumes, you let the model learn a hierarchy of representations, each layer expressed
|
||||
in terms of the one below it. Depth is the mechanism that makes composition cheap.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Representation learning**: learning the features, not just the mapping from features to output.
|
||||
- **Depth as composition**: a deep model reuses sub-computations, so a function needing
|
||||
exponentially many units at depth 1 may need polynomially many at depth k.
|
||||
- **The curse of dimensionality**: as input dimension grows, the number of distinguishable
|
||||
configurations grows exponentially, so local-interpolation methods run out of neighbours.
|
||||
- **Prior of compositionality**: deep learning bets that the data was generated by a composition
|
||||
of simple factors. This is a prior, not a theorem — where the bet is wrong, depth does not help.
|
||||
- **AI winters and rebrands**: cybernetics → connectionism → deep learning; the ideas outlived
|
||||
three funding cycles because the constraint was data and compute, not concept.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Think of each layer as **re-coordinatizing** the input so the next layer's job gets easier —
|
||||
not as "extracting meaning."
|
||||
- Prefer depth over width when the target function plausibly **factors**; prefer width when it
|
||||
does not. Depth is a structural prior, and priors have to be earned.
|
||||
- Treat "the model learns features automatically" as **shifting where the human judgment goes**
|
||||
(architecture, objective, data curation), not eliminating it.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Reading this chapter as history and skipping it**: the three growth curves it draws
|
||||
(dataset size, model size, benchmark accuracy) are the argument for why anything else works.
|
||||
- **Assuming depth is always better**: without the compositional structure, extra depth buys
|
||||
optimization difficulty and nothing else.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The chapter's central claim — capability tracks dataset and model scale — held, and was made
|
||||
quantitative by neural scaling laws (Hestness et al. 2017; Kaplan et al. 2020; Hoffmann et al.
|
||||
2022 "Chinchilla"), which the 2016 text predates. Its examples of "what deep learning can do"
|
||||
are now badly dated; its explanation of *why* depth helps is not. **Confidence: high** — the
|
||||
scaling-law literature is public and replicated.
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Before adding depth, state the compositional structure you believe the data has.
|
||||
2. Use the curse of dimensionality as the test for whether a nearest-neighbour-style baseline
|
||||
is even viable on your input.
|
||||
3. Read the growth curves as a resource argument: most 2016-era "deep learning doesn't work
|
||||
here" results were data or compute verdicts, not architecture verdicts.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 5**: the formal learning-theory machinery behind the claims made informally here.
|
||||
- **Ch 15**: representation learning, promised here and delivered there.
|
||||
- **references/book_to_2026_delta.md**: the scaling-law layer this chapter now needs.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
# Chapter 2: Linear Algebra
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/linear_algebra.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
The minimum linear algebra a deep learning practitioner needs, chosen for what appears later:
|
||||
tensors and broadcasting, norms, eigendecomposition, SVD, the pseudoinverse, and PCA derived
|
||||
from scratch. This is a filter, not a course — it teaches what Chapters 4, 5, 8 and 13 will use.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Tensor / broadcasting**: an n-dimensional array plus the rules for combining shapes. Most
|
||||
real "model bugs" are shape bugs.
|
||||
- **Norms**: L2 (Euclidean), L1 (sparsity-friendly, non-differentiable at 0), max-norm,
|
||||
Frobenius. The choice of norm *is* the choice of what "small" means in a regularizer.
|
||||
- **Eigendecomposition**: A = V diag(λ) V⁻¹ for square matrices; the eigenvalues tell you the
|
||||
local curvature story later in Ch 4 and 8.
|
||||
- **Singular value decomposition (SVD)**: A = U D Vᵀ, defined for *any* matrix. The general tool
|
||||
where eigendecomposition needs square and diagonalizable.
|
||||
- **Moore–Penrose pseudoinverse**: the least-squares / minimum-norm solution when a system is
|
||||
over- or under-determined.
|
||||
- **Condition number**: ratio of largest to smallest singular value; large means small input
|
||||
perturbations produce large output changes — the numerical fragility Ch 4 confronts.
|
||||
- **PCA**: derived here as the linear encoder/decoder minimizing L2 reconstruction error, which
|
||||
is exactly the framing Ch 13–14 generalize.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read a matrix as a **function on space**, and its singular values as how much it stretches
|
||||
each orthogonal direction. Condition number = worst stretch / least stretch.
|
||||
- Use SVD as the **default** decomposition and reach for eigendecomposition only when symmetry
|
||||
buys you something (it does for Hessians).
|
||||
- Treat a regularizer's norm as a **prior over parameter space**: L2 says "small and spread",
|
||||
L1 says "mostly zero."
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Skipping to Chapter 6.** Ch 4 and 8's discussion of ill-conditioning is unreadable without
|
||||
eigenvalues and condition number, and readers who skip typically bounce off Ch 8.
|
||||
- **Memorizing decompositions as identities** instead of as geometry — the geometry is what
|
||||
transfers to optimization.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Nothing in the mathematics. What changed is the practice: in half-precision training, condition
|
||||
number stopped being a theoretical concern and became an operational one (loss scaling, bf16 over
|
||||
fp16 precisely because of dynamic range). **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Track shapes explicitly; treat a shape mismatch as a modelling error, not a typing error.
|
||||
2. Pick your norm deliberately when you regularize — you are choosing the shape of the prior.
|
||||
3. Learn PCA in this chapter's form (encoder/decoder minimizing reconstruction) so Ch 13–14 read
|
||||
as generalizations rather than new material.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 4**: conditioning and numerical stability use exactly these quantities.
|
||||
- **Ch 13**: PCA reappears as a linear factor model with an explicit probabilistic story.
|
||||
- **Ch 8**: Hessian eigenvalues explain why gradient descent stalls.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
# Chapter 3: Probability and Information Theory
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/prob.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Deep learning models are probability distributions, and losses are log-likelihoods. This chapter
|
||||
installs the vocabulary — random variables, the common distributions, expectation, Bayes' rule,
|
||||
entropy, KL divergence, cross-entropy — that makes "why this loss?" answerable rather than
|
||||
conventional.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Frequentist vs Bayesian probability**: rates of events vs degrees of belief. Deep learning
|
||||
uses the Bayesian reading for parameters and the frequentist one for evaluation, mostly
|
||||
without saying so.
|
||||
- **Marginal / conditional / chain rule**: the algebra every graphical model in Ch 16 runs on.
|
||||
- **Common distributions**: Bernoulli, categorical, Gaussian, exponential, Laplace, Dirac,
|
||||
empirical, and mixtures. Each corresponds to an output layer you will actually build.
|
||||
- **Self-information and Shannon entropy**: surprise, and expected surprise.
|
||||
- **KL divergence**: asymmetric. D(p‖q) ≠ D(q‖p), and the asymmetry decides whether your fitted
|
||||
model covers all modes or concentrates on one.
|
||||
- **Cross-entropy**: H(p,q) = H(p) + D(p‖q). Minimizing cross-entropy over q is minimizing KL,
|
||||
because H(p) is constant in q — this is why classification uses it.
|
||||
- **Structured probabilistic models**: factorization of a joint into conditionals over a graph.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read every loss as a **negative log-likelihood** under an assumed output distribution: MSE is
|
||||
a Gaussian with fixed variance, cross-entropy is a categorical, MAE is a Laplace. If you know
|
||||
the assumed distribution, you know when the loss is wrong for your data.
|
||||
- Use KL **direction** as a design lever: forward KL (data ‖ model) is mode-covering and gives
|
||||
blurry averages; reverse KL (model ‖ data) is mode-seeking and gives sharp but partial fits.
|
||||
This one fact explains most of Ch 19 and 20.
|
||||
- Treat softmax as **exp-then-normalize on logits**, and remember from Ch 4 that it must be
|
||||
computed in a shift-stabilized form.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Choosing MSE for a bounded or count-valued target** — you have assumed a Gaussian on data
|
||||
that is not Gaussian, and the residual structure will tell you so.
|
||||
- **Reading KL as a distance**: it is not symmetric and does not satisfy the triangle inequality.
|
||||
- **Interpreting softmax outputs as calibrated probabilities** without checking calibration —
|
||||
the chapter's math does not promise calibration, and modern networks are typically
|
||||
overconfident (Guo et al. 2017, post-dating the book).
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The probabilistic core is unchanged. Two additions matter: the calibration literature (deep
|
||||
networks are systematically overconfident; temperature scaling is the cheap fix), and the rise
|
||||
of losses that are *not* clean log-likelihoods — contrastive/InfoNCE objectives, and preference
|
||||
losses such as DPO. The chapter's "every loss is a likelihood" framing needs that caveat now.
|
||||
**Confidence: high** for calibration; **high** for the contrastive family being outside the
|
||||
chapter's scope.
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. State the output distribution before choosing the loss; the loss follows from it.
|
||||
2. When a generative model looks blurry, suspect forward KL; when it looks mode-collapsed,
|
||||
suspect reverse KL.
|
||||
3. Check calibration separately from accuracy — the book does not, and the gap is real.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 5**: maximum likelihood as the estimator that justifies these losses.
|
||||
- **Ch 16–19**: graphical models, sampling, and variational inference all run on this algebra.
|
||||
- **Ch 20**: the KL asymmetry decides GAN vs VAE failure modes.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# Chapter 4: Numerical Computation
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/numerical.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Real arithmetic on finite hardware fails in specific, predictable ways — underflow, overflow,
|
||||
ill-conditioning — and gradient-based optimization inherits every one of them. This is the
|
||||
shortest chapter with the highest debugging payoff.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Underflow / overflow**: numbers rounding to zero (then dividing) or exceeding representable
|
||||
range (then becoming inf/NaN).
|
||||
- **Stabilized softmax**: subtract the max logit before exponentiating; mathematically identical,
|
||||
numerically survivable. `log_softmax` exists for the same reason.
|
||||
- **Conditioning**: how much a function's output moves for small input moves; for a matrix, the
|
||||
condition number from Ch 2.
|
||||
- **Gradient descent** and the first-order Taylor picture; **critical points**: minima, maxima,
|
||||
saddles.
|
||||
- **Jacobian and Hessian**; second-order Taylor expansion, and the optimal step size implied by
|
||||
curvature.
|
||||
- **Newton's method** and why it is not the default in deep learning: the Hessian is n×n in the
|
||||
parameter count.
|
||||
- **Constrained optimization / KKT**: the framing that makes Ch 7's norm penalties readable as
|
||||
constraints.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read a **NaN in the loss** as a numerics report first and a modelling bug second: check
|
||||
log(0), division by a near-zero denominator, exp of a large logit, and an exploding gradient
|
||||
in that order.
|
||||
- Think of the Hessian's **eigenvalue spread as the terrain**: a large condition number is a
|
||||
narrow ravine, and the largest safe step size is set by the largest eigenvalue while progress
|
||||
is set by the smallest.
|
||||
- Prefer the **log-domain** whenever probabilities are multiplied — this is the single highest
|
||||
yield habit in this chapter.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Hand-rolling softmax or cross-entropy** in a training loop instead of using the fused,
|
||||
stabilized primitive.
|
||||
- **Blaming the learning rate for every divergence**: ill-conditioning produces the same symptom
|
||||
and does not respond to the same fix.
|
||||
- **Reaching for a second-order optimizer** in a model with millions of parameters without
|
||||
understanding the memory cost.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Mixed-precision training (fp16 with loss scaling; then bf16, which trades mantissa bits for
|
||||
exponent range specifically to avoid these failures) made this chapter's content a daily
|
||||
operational concern rather than a background caution. Gradient clipping became standard practice
|
||||
for transformer training. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Work in log-space by default for anything probabilistic.
|
||||
2. When training diverges, separate "step too large" from "problem ill-conditioned" — they need
|
||||
different fixes.
|
||||
3. Understand KKT well enough to read weight decay as a constraint; Ch 7 assumes it.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 8**: every optimization difficulty named here recurs there at scale.
|
||||
- **Ch 7**: the constrained-optimization view of regularization.
|
||||
- **scripts/training_diagnostics.py**: mechanizes the NaN/divergence triage above.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# Chapter 5: Machine Learning Basics
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/ml.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Everything after this chapter is a special case of it: a task, a performance measure, an
|
||||
experience, a capacity choice, and the generalization gap that capacity choice produces. If you
|
||||
read one chapter of Part I, read this one.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Task / performance measure / experience (T, P, E)**: the definition that forces you to name
|
||||
the metric before the model.
|
||||
- **Capacity**: the range of functions a learner can express. Representational capacity vs
|
||||
*effective* capacity (what the optimizer actually reaches).
|
||||
- **Underfitting / overfitting** and the classical **U-shaped** generalization-error curve.
|
||||
- **Bias–variance decomposition**: expected error splits into bias², variance, and irreducible
|
||||
noise. Regularization trades variance for bias.
|
||||
- **No Free Lunch theorem**: averaged over *all* data-generating distributions, every algorithm
|
||||
ties. Therefore all progress comes from priors matched to the distributions we actually face.
|
||||
- **Regularization**: any change intended to reduce generalization error but not training error.
|
||||
- **Maximum likelihood estimation** and its **MAP** counterpart; consistency and efficiency.
|
||||
- **Hyperparameters and validation sets**: never tune on test.
|
||||
- **Curse of dimensionality, local constancy, manifold hypothesis**: why nearest-neighbour-style
|
||||
priors fail and why deep models assume data concentrates near a low-dimensional manifold.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Ask "**what is the prior?**" before "what is the model?" — No Free Lunch says the prior is
|
||||
where all the leverage is.
|
||||
- Read regularization as **moving mass in the bias–variance budget**, so that "add dropout" and
|
||||
"get more data" are alternative purchases of the same thing.
|
||||
- Treat the **train/val gap** as your primary instrument: gap small + error high = underfit
|
||||
(capacity or optimization); gap large = overfit (regularization or data).
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Tuning against the test set** — including "just peeking once."
|
||||
- **Comparing models on different splits**, or reporting a single seed for a small dataset.
|
||||
- **Believing the U-curve unconditionally.** See below: this is the one place where the 2016
|
||||
text is now known to be incomplete.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
**This is the chapter's one substantive correction.** The classical U-shaped capacity curve is
|
||||
not the whole picture. In the heavily overparameterized regime, test error can fall again past
|
||||
the interpolation threshold — "double descent" (Belkin et al. 2019; Nakkiran et al. 2020), which
|
||||
post-dates the book. Modern large models routinely sit in a regime the 2016 framing predicts
|
||||
should overfit catastrophically and does not. The bias–variance decomposition remains correct as
|
||||
algebra; the *managerial advice* "reduce capacity when you overfit" is no longer the only right
|
||||
move. **Confidence: high** — double descent is widely replicated. Practical consequence: try
|
||||
"more data / more regularization / train longer" before "smaller model."
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Write down T, P, E in one sentence before writing any model code.
|
||||
2. Use the train/val gap to choose your next action; do not guess.
|
||||
3. Hold the U-curve loosely — check whether you are past the interpolation threshold before
|
||||
shrinking a model that overfits.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 7**: every regularizer named here, in depth.
|
||||
- **Ch 11**: this chapter's diagnostics turned into a workflow.
|
||||
- **scripts/capacity_planner.py**: mechanizes the gap-to-action rule, double descent included.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
# Chapter 6: Deep Feedforward Networks
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/mlp.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
An MLP is a composed chain of parametric functions trained by gradient descent, where the choice
|
||||
of output unit follows from the assumed output distribution and the choice of hidden unit follows
|
||||
from gradient behaviour. Backpropagation is not learning; it is the chain rule scheduled well.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Universal approximation**: one sufficiently wide hidden layer can approximate any continuous
|
||||
function on a compact set. It says nothing about learnability, sample efficiency, or the size
|
||||
required — depth is how you get the same function with far fewer units.
|
||||
- **Output units follow the distribution**: linear + MSE → Gaussian; sigmoid + binary
|
||||
cross-entropy → Bernoulli; softmax + cross-entropy → categorical; mixture density → multimodal.
|
||||
- **Hidden units**: ReLU as the sane default; leaky/parametric ReLU, ELU, GELU/Swish (post-2016)
|
||||
for smoothness; sigmoid/tanh only where saturation is desired (gates).
|
||||
- **Saturation**: sigmoid/tanh gradients vanish in the tails, which is why they are poor hidden
|
||||
units and fine as gates.
|
||||
- **Backpropagation**: reverse-mode automatic differentiation over the computation graph. Cost is
|
||||
roughly one forward pass, memory is the stored activations.
|
||||
- **Computation graph / autodiff**: the abstraction every framework implements. Forward mode is
|
||||
cheap in inputs; reverse mode is cheap in outputs — losses are scalar, hence reverse.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Pick the output unit from the **likelihood** (Ch 3), then pick the loss as its negative log.
|
||||
Every "which loss?" question reduces to "which distribution?"
|
||||
- Read **ReLU's advantage as gradient preservation**, not nonlinearity per se: it is piecewise
|
||||
linear, so the gradient through an active unit is exactly 1.
|
||||
- Treat activation memory as the **real** cost of depth in training: the backward pass needs the
|
||||
forward activations, which is why checkpointing trades compute for memory.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Citing universal approximation to justify a shallow model** — the theorem allows an
|
||||
exponentially wide layer, which is not an engineering plan.
|
||||
- **Sigmoid hidden layers** in a deep stack.
|
||||
- **Pairing a sigmoid output with MSE**: gradients vanish exactly where the model is most wrong.
|
||||
Use cross-entropy.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Smooth activations (GELU, SiLU/Swish) became the default in transformers; gated variants
|
||||
(GLU, SwiGLU) are now standard in large language model feedforward blocks. Residual connections
|
||||
(He et al. 2015) are mentioned here only in passing but became the structural default for every
|
||||
deep stack. Reverse-mode autodiff is unchanged. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Derive the loss from the output distribution rather than picking it by habit.
|
||||
2. Default to ReLU-family hidden units; reserve saturating units for gates.
|
||||
3. Budget activation memory as a first-class constraint, not an implementation detail.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 3**: the distributions that determine output units.
|
||||
- **Ch 8**: why gradient preservation matters at depth.
|
||||
- **scripts/model_arithmetic.py**: parameter, FLOP and activation-memory accounting for a stack.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# Chapter 7: Regularization for Deep Learning
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/regularization.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Regularization is anything that trades training error for generalization error, and this chapter
|
||||
is the catalogue: norm penalties, data augmentation, noise, early stopping, parameter sharing,
|
||||
sparsity, ensembling, dropout, and adversarial training — with their equivalences made explicit.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **L2 / weight decay**: shrinks parameters along low-curvature Hessian directions. Under a
|
||||
quadratic approximation, it damps each eigen-direction by λᵢ/(λᵢ+α) — directions the loss does
|
||||
not care about get pulled to zero.
|
||||
- **L1**: yields genuine sparsity; equivalent to a Laplace prior on parameters.
|
||||
- **Norm penalty as constraint**: via KKT (Ch 4), a penalty is a soft version of a norm ball.
|
||||
Explicit projection (max-norm) is sometimes better behaved.
|
||||
- **Dataset augmentation**: the most reliable regularizer when the invariance is real. Choose
|
||||
transformations that preserve the label — and check that they do.
|
||||
- **Noise injection**: on inputs (≈ L2 penalty for some models), on weights (encourages flat
|
||||
minima), on labels (label smoothing).
|
||||
- **Early stopping**: the cheapest regularizer; under a quadratic approximation it is
|
||||
approximately equivalent to L2 with a strength set by the number of steps.
|
||||
- **Parameter tying / sharing**: the strongest form — it removes parameters instead of penalizing
|
||||
them. Convolution is parameter sharing (Ch 9).
|
||||
- **Bagging and ensembles**: variance reduction by averaging independently trained models.
|
||||
- **Dropout**: approximate ensembling over exponentially many sub-networks at the cost of one.
|
||||
- **Adversarial training**: penalizing sensitivity to worst-case local perturbation.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Order regularizers by **cost per unit of gap closed**: more real data > augmentation > early
|
||||
stopping > weight decay > dropout > architecture surgery.
|
||||
- Read weight decay as **"forget the directions the data does not constrain"** — that is exactly
|
||||
what the eigen-analysis says.
|
||||
- Treat dropout as an **ensemble**, so its interaction with batch normalization (train/test
|
||||
statistics mismatch) is expected rather than surprising.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Stacking every regularizer at once** and then tuning — you cannot attribute the effect.
|
||||
- **Augmenting with label-destroying transforms** (horizontal flip on digits, aggressive crops on
|
||||
fine-grained classes).
|
||||
- **Dropout inside a residual transformer block plus batch norm plus heavy weight decay** without
|
||||
measuring; modern stacks regularize far more lightly than 2016 practice.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Weight decay and L2 were shown to be *not* equivalent under adaptive optimizers, which is why
|
||||
**AdamW** (Loshchilov & Hutter 2017/2019) decouples them — this post-dates the book and is now
|
||||
the default. Dropout largely left large-scale vision and language models, displaced by
|
||||
normalization, augmentation and sheer data volume. Label smoothing and stochastic depth became
|
||||
common. **Confidence: high** for AdamW; **high** for the decline of dropout at scale.
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Add regularizers one at a time and measure the gap after each.
|
||||
2. Use decoupled weight decay (AdamW) rather than L2-in-the-loss with an adaptive optimizer.
|
||||
3. Prefer parameter sharing over penalties when a real invariance exists.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 5**: the bias–variance budget these all spend from.
|
||||
- **Ch 8**: optimizer choice changes what weight decay means.
|
||||
- **Ch 11**: which regularizer to reach for, given a measured gap.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
# Chapter 8: Optimization for Training Deep Models
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/optimization.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Training is not pure optimization: you minimize an empirical surrogate to reduce a risk you
|
||||
cannot measure, on a non-convex landscape, with noisy gradients. The chapter separates the
|
||||
difficulties (ill-conditioning, saddles, cliffs, long-term dependencies, poor correspondence
|
||||
between local and global structure) from the algorithms that address them.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Empirical risk minimization and its surrogates**: you optimize a differentiable proxy, not
|
||||
the metric you report.
|
||||
- **Minibatch SGD**: gradient noise scales roughly with 1/√batch, so bigger batches buy accuracy
|
||||
in the gradient at linear cost — a poor trade past a point.
|
||||
- **Ill-conditioning**: the dominant obstacle in practice; gradient norm can *grow* while the loss
|
||||
stalls.
|
||||
- **Saddle points, not local minima**: in high dimensions, critical points are overwhelmingly
|
||||
saddles. This reframes the folklore fear of local minima.
|
||||
- **Cliffs and exploding gradients** → **gradient clipping**.
|
||||
- **Momentum** and **Nesterov momentum**: accumulate a velocity to cross ravines.
|
||||
- **Initialization**: Xavier/Glorot and He scaling to keep activation and gradient variance
|
||||
roughly constant with depth. Initialization is an algorithm, not a detail.
|
||||
- **Adaptive methods**: AdaGrad (decaying), RMSProp, **Adam** (RMSProp + momentum + bias
|
||||
correction).
|
||||
- **Second-order methods**: Newton, conjugate gradient, BFGS/L-BFGS — and why they rarely survive
|
||||
contact with minibatch noise and parameter counts.
|
||||
- **Batch normalization**: reparameterization that stabilizes the scale of layer inputs.
|
||||
- **Curriculum learning / coordinate descent / Polyak averaging**: the meta-strategies.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Diagnose by **what the gradient is doing**: norm exploding = cliff, clip. Norm large but loss
|
||||
flat = ill-conditioning, use momentum/adaptive/normalization. Norm ~0 with high loss = dead
|
||||
units or saturation.
|
||||
- Treat **learning-rate schedule as a first-class hyperparameter** — usually more important than
|
||||
the choice among SGD/Adam variants.
|
||||
- Read batch norm as **making the loss surface better conditioned**; whether it does so by
|
||||
reducing "internal covariate shift" is contested (Santurkar et al. 2018, post-2016).
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Fearing local minima**: spend the debugging effort on conditioning and learning rate.
|
||||
- **Tuning the optimizer before the learning rate.**
|
||||
- **Batch norm with tiny batches** — the batch statistics become noise. Use group/layer norm.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The largest delta in Part II. **AdamW** replaced Adam+L2. Learning-rate **warmup plus cosine
|
||||
decay** became standard for transformers. **LayerNorm** (and RMSNorm) displaced batch norm in
|
||||
sequence models; pre-norm residual placement displaced post-norm for deep stacks. Large-batch
|
||||
training with LARS/LAMB, and the linear-scaling-rule literature, arrived after the book.
|
||||
**Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Tune learning rate and schedule first; optimizer family second.
|
||||
2. Read gradient-norm behaviour before changing the architecture.
|
||||
3. Use He/Xavier-style initialization deliberately — bad init is a common silent failure.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 4**: conditioning, Hessians, and step size.
|
||||
- **Ch 10**: exploding/vanishing gradients as the sequence-model version of these problems.
|
||||
- **scripts/training_diagnostics.py**: this chapter's triage as an executable decision tree.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# Chapter 9: Convolutional Networks
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/convnets.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Convolution is three ideas at once — sparse interactions, parameter sharing, and equivariance to
|
||||
translation — and each is a prior about grid-structured data. Pooling adds approximate invariance
|
||||
to small translations. The architecture is a statement about the data, not a trick.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Sparse interactions**: each output depends on a small receptive field, so cost drops from
|
||||
O(m·n) to O(k·n).
|
||||
- **Parameter sharing**: one kernel is reused at every position — the strongest regularizer in
|
||||
Ch 7's catalogue, applied structurally.
|
||||
- **Equivariance to translation**: shift the input, the feature map shifts. Convolution is *not*
|
||||
equivariant to rotation or scale — a fact that motivates augmentation.
|
||||
- **Pooling**: max/average pooling gives local invariance and downsampling; it is a prior that
|
||||
small position changes should not matter.
|
||||
- **Variants**: valid/same/full padding, strided convolution, dilated convolution, tiled
|
||||
convolution, locally-connected layers (no sharing), transposed convolution for upsampling.
|
||||
- **Convolution as an infinitely strong prior**: the chapter's sharpest framing — a conv layer is
|
||||
a fully-connected layer with hard constraints on its weights.
|
||||
- **Structured outputs**: dense prediction (segmentation) rather than one label per image.
|
||||
- **Efficiency**: FFT-based and separable convolutions.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Ask "**is the statistic I need position-invariant?**" If yes, convolve. If not (a face-aligned
|
||||
dataset, a tabular grid with meaningful coordinates), the prior is wrong and locally-connected
|
||||
or attention layers may fit better.
|
||||
- Compute **receptive field** deliberately: it must cover the evidence needed for the decision.
|
||||
Depth, stride and dilation are three ways to buy it, with different costs.
|
||||
- Treat pooling as **throwing away location on purpose**; when location is the answer
|
||||
(segmentation, detection), pool less and use dilation or skip connections.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Using convolution on data with no spatial/temporal locality** (arbitrary tabular columns) —
|
||||
the sharing prior is simply false there.
|
||||
- **Ignoring receptive field** and then adding parameters to fix an underfitting model.
|
||||
- **Aggressive pooling in a dense-prediction task.**
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Residual networks (2015) are treated only briefly here but became the default. Since then:
|
||||
depthwise-separable convolutions (MobileNet/Xception), EfficientNet-style compound scaling,
|
||||
and — most importantly — **Vision Transformers** (Dosovitskiy et al. 2020), which discard the
|
||||
convolutional prior in favour of data plus attention, and win at scale while ConvNets remain
|
||||
competitive at smaller data sizes (ConvNeXt, 2022, closed much of the gap). The chapter's
|
||||
argument that the conv prior is a *bet on the data* is exactly what ViT's data-hunger confirms.
|
||||
**Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Justify convolution by the invariance you actually believe in.
|
||||
2. Size the receptive field before adding depth for its own sake.
|
||||
3. Expect the conv prior to pay off most when data is limited — this is the modern boundary
|
||||
between ConvNets and ViTs.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 7**: parameter sharing as structural regularization.
|
||||
- **Ch 12**: computer-vision applications built on this.
|
||||
- **references/book_to_2026_delta.md**: the ViT/ConvNeXt line.
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
# Chapter 10: Sequence Modeling — Recurrent and Recursive Nets
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/rnn.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Recurrence shares parameters across time the way convolution shares them across space, which lets
|
||||
one model handle variable-length sequences. The price is that gradients must traverse many
|
||||
multiplicative steps, producing the vanishing/exploding gradient problem — and gated
|
||||
architectures exist to pay it.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Unfolding the computational graph**: a recurrent definition becomes a deep feedforward graph
|
||||
with tied weights.
|
||||
- **BPTT (backpropagation through time)** and **truncated BPTT**.
|
||||
- **Teacher forcing**: train on ground-truth previous tokens; note the train/inference mismatch
|
||||
(exposure bias) it creates.
|
||||
- **Vanishing and exploding gradients**: repeated multiplication by the recurrent Jacobian; the
|
||||
spectral radius decides which.
|
||||
- **Gradient clipping** for the exploding half; **gating** for the vanishing half.
|
||||
- **LSTM** (input/forget/output gates plus a cell with an additive path) and **GRU** (a
|
||||
two-gate simplification). The additive cell path is the mechanism — it makes the gradient path
|
||||
through time approximately linear.
|
||||
- **Bidirectional RNNs**; **encoder–decoder / sequence-to-sequence** with a fixed-size context.
|
||||
- **Attention** appears here as the fix for the encoder–decoder bottleneck.
|
||||
- **Deep RNNs, recursive (tree-structured) nets, echo state networks, leaky units, skip
|
||||
connections through time.**
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read gating as **learned, data-dependent memory management**: the forget gate decides how long
|
||||
the additive highway stays open.
|
||||
- Treat the fixed-size context vector in vanilla seq2seq as a **bottleneck** — the whole reason
|
||||
attention was invented.
|
||||
- Expect **exposure bias** whenever you teacher-force: the model has never seen its own mistakes
|
||||
during training.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Reaching for an RNN by default in 2026** for a task where a transformer is the standard
|
||||
baseline and the sequence fits in context.
|
||||
- **Training a long-sequence RNN without gradient clipping.**
|
||||
- **Ignoring the train/inference mismatch** in autoregressive generation.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
**The largest single delta in the book.** Attention, described here as an enhancement to
|
||||
recurrent seq2seq, became the whole architecture with *Attention Is All You Need* (Vaswani et al.
|
||||
2017) — one year after publication. Transformers replaced RNNs for essentially all large-scale
|
||||
sequence work: parallel training over sequence positions, direct O(1) path length between any two
|
||||
tokens, and much better scaling. What survived: gradient clipping, teacher forcing, exposure bias,
|
||||
and the vanishing/exploding analysis. What returned: linear-time recurrent architectures
|
||||
(S4/Mamba-style state-space models, 2021–2024) as a long-context alternative, which makes this
|
||||
chapter's material relevant again rather than obsolete. **Confidence: high.**
|
||||
|
||||
**Read this chapter for the gradient-flow analysis, not for the architecture recommendation.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Learn the vanishing/exploding analysis here — it explains residual connections, LayerNorm
|
||||
placement, and state-space models alike.
|
||||
2. Clip gradients whenever a recurrence is in the graph.
|
||||
3. Treat this chapter's architecture advice as historical; treat its diagnosis as current.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 8**: the same optimization pathologies, in the time dimension.
|
||||
- **Ch 12**: NLP and speech applications built on these.
|
||||
- **references/book_to_2026_delta.md**: the transformer displacement, in detail.
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
# Chapter 11: Practical Methodology
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/guidelines.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Knowing many algorithms matters less than knowing which one to reach for given a measurement.
|
||||
This chapter is a workflow: pick a metric, build an end-to-end baseline fast, then let
|
||||
instrumentation — not intuition — decide whether to add data, capacity, or regularization.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **The design process**: (1) determine goals — the error metric and the target value; (2) build
|
||||
a working end-to-end pipeline early; (3) instrument it to find bottlenecks; (4) change one
|
||||
thing at a time based on measurement.
|
||||
- **Choose the metric before the model**: accuracy, precision/recall, F-score, PR/ROC curves,
|
||||
coverage. Name the target value and where it came from.
|
||||
- **Sensible baselines**: pick the standard architecture and optimizer for the data type before
|
||||
inventing anything.
|
||||
- **More data or a bigger model?** — the decision procedure: if training error is high, the model
|
||||
or the optimization is the bottleneck (data will not help). If training error is low and test
|
||||
error is high, gather more data or regularize.
|
||||
- **Hyperparameter tuning**: manual (understand what each knob does to effective capacity),
|
||||
**grid search** (poor scaling in dimensions), **random search** (better — it does not waste
|
||||
trials on unimportant dimensions; Bergstra & Bengio 2012), and model-based/Bayesian.
|
||||
- **Debugging strategies**: visualize the model's actual predictions and worst cases; fit a tiny
|
||||
subset to zero training error; compare backprop against numerical derivatives; monitor
|
||||
activation and gradient histograms.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Use the **train-error-first rule** as the single most valuable heuristic in the book: high
|
||||
training error means "do not collect data yet."
|
||||
- Treat "**can it overfit 20 examples?**" as the smoke test that separates a bug from a modelling
|
||||
limitation. If it cannot, you have a bug.
|
||||
- Prefer **random search over grid search** whenever the number of hyperparameters exceeds ~2.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Tuning many things per experiment** — you learn nothing attributable.
|
||||
- **Collecting more data to fix underfitting.**
|
||||
- **Optimizing a proxy metric** whose relationship to the real objective was never checked.
|
||||
- **Skipping the end-to-end pipeline** in favour of perfecting one component.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The workflow is the most durable material in the book and is now the backbone of MLOps practice.
|
||||
Additions since: experiment tracking as standard tooling, seeded reproducibility expectations,
|
||||
Hyperband/ASHA for early-stopping-based search (post-2016), and — for large models — scaling laws
|
||||
used to *predict* the return on more data or parameters instead of testing empirically at full
|
||||
size. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Write the metric and its target number before writing model code.
|
||||
2. Ask "is training error high?" before every data or capacity decision.
|
||||
3. Change one thing per experiment and log it.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 5**: the underfit/overfit framing this operationalizes.
|
||||
- **Ch 7 / Ch 8**: the two toolboxes the workflow selects from.
|
||||
- **scripts/training_diagnostics.py**: this decision tree, executable.
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
# Chapter 12: Applications
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/applications.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
How deep learning was actually deployed circa 2016: large-scale implementation (GPUs,
|
||||
distributed training, model compression), then computer vision, speech recognition, and NLP.
|
||||
This is the most time-dated chapter in the book — read the *systems* half, discount the
|
||||
*state-of-the-art* half.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Large-scale implementation**: GPU data parallelism, model parallelism, asynchronous SGD
|
||||
(Hogwild-style), parameter servers.
|
||||
- **Model compression**: distillation, quantization, pruning — for inference cost.
|
||||
- **Dynamic structure / conditional computation**: cascades and gating so that not every input
|
||||
pays the full cost.
|
||||
- **Preprocessing in vision**: contrast normalization, whitening, dataset augmentation.
|
||||
- **Speech**: the acoustic-model pipeline and its shift from GMM-HMM to deep networks.
|
||||
- **NLP**: n-gram models, word embeddings and the curse of dimensionality over vocabularies,
|
||||
hierarchical softmax and sampling-based approximations for large output vocabularies,
|
||||
neural machine translation.
|
||||
- **Recommender systems** and the exploration/exploitation problem.
|
||||
- **Knowledge representation and relational reasoning.**
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Separate **capability claims** (perishable) from **systems constraints** (durable). Memory
|
||||
bandwidth, batch efficiency, and inference cost still shape architecture choices.
|
||||
- Read the large-vocabulary softmax section as an instance of the general pattern: **when the
|
||||
normalizing sum is expensive, approximate it** — the same problem Ch 18 attacks head-on.
|
||||
- Treat conditional computation as the ancestor of **Mixture-of-Experts** routing.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Citing this chapter's benchmark numbers or SOTA claims** — they are a decade old.
|
||||
- **Copying its NLP pipeline**: subword tokenization (BPE/SentencePiece) and pretrained
|
||||
transformers replaced nearly all of it.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Almost everything at the application layer. Speech moved to end-to-end CTC/attention models and
|
||||
then to large self-supervised encoders. NLP moved to pretrained transformers (BERT 2018, GPT
|
||||
family), with subword tokenization and full-softmax over ~30k–200k subwords making hierarchical
|
||||
softmax largely unnecessary. Vision moved to self-supervised pretraining and ViTs. Distillation,
|
||||
quantization and pruning grew into a mature inference-optimization discipline. Conditional
|
||||
computation matured into sparse MoE. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Read this chapter for the systems reasoning and the approximation techniques, not for what is
|
||||
state of the art.
|
||||
2. When your output vocabulary or normalizing constant is huge, recognize it as the recurring
|
||||
partition-function problem.
|
||||
3. Plan inference cost as an architecture constraint from the start.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 18**: the partition-function problem in its general form.
|
||||
- **Ch 9 / Ch 10**: the architectures being applied here.
|
||||
- **references/book_to_2026_delta.md**: what replaced each application pipeline.
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
# Chapter 13: Linear Factor Models
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/linear_factors.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
The simplest generative models: sample latent factors from a simple prior, map them linearly to
|
||||
observations, add noise. Probabilistic PCA, factor analysis, ICA, sparse coding and slow feature
|
||||
analysis are all this one template with different priors — and they are the scaffolding for
|
||||
everything in Chapters 14–20.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **The template**: h ~ p(h); x = Wh + b + noise. Change p(h) and the noise model, get a
|
||||
different named method.
|
||||
- **Factor analysis**: Gaussian latent prior, diagonal (per-feature) observation noise.
|
||||
- **Probabilistic PCA**: factor analysis with isotropic noise; recovers PCA as noise → 0.
|
||||
- **Independent component analysis (ICA)**: non-Gaussian independent latents — this is what makes
|
||||
the factors identifiable, which Gaussian models cannot be (any rotation fits equally well).
|
||||
- **Sparse coding**: a heavy-tailed (Laplace/Cauchy) prior; inference is an optimization, not a
|
||||
closed form, which makes encoding expensive.
|
||||
- **Slow feature analysis**: a prior that useful factors change slowly over time.
|
||||
- **Manifold interpretation of PCA**: the model concentrates probability near a linear subspace.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read each model as **a prior choice**, and read the prior as the assumption that buys
|
||||
identifiability. Gaussian latents are rotation-invariant, so a Gaussian model cannot tell you
|
||||
*which* factors — only which subspace.
|
||||
- Treat sparse coding as the moment where **inference becomes iterative**; that cost is precisely
|
||||
what autoencoders (Ch 14) amortize with a learned encoder.
|
||||
- Use these as the **linear baseline** for any representation-learning claim: if a linear factor
|
||||
model matches your deep encoder, the depth is not earning its cost.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Skipping this chapter and starting at Ch 20** — VAEs read as arbitrary machinery without the
|
||||
latent-variable template established here.
|
||||
- **Expecting interpretable factors from a Gaussian-latent model.**
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The template survived; the emphasis moved. Nonlinear ICA identifiability results (Hyvärinen et
|
||||
al., 2016–2020) clarified when latent factors are recoverable at all, and the disentanglement
|
||||
literature — notably Locatello et al. (2019) — showed that unsupervised disentanglement is
|
||||
impossible without inductive biases or supervision, which is a formal statement of this
|
||||
chapter's identifiability point. Sparse coding returned as a tool for interpreting neural
|
||||
networks (sparse autoencoders over LLM activations, 2023–2024). **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Name the latent prior when you propose any generative model; it determines what is learnable.
|
||||
2. Use a linear factor model as the baseline before claiming a deep representation helps.
|
||||
3. Remember that identifiability, not fit quality, is what non-Gaussian priors buy.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 14**: autoencoders amortize the inference these models do by optimization.
|
||||
- **Ch 15**: what makes a representation good.
|
||||
- **Ch 20**: the VAE is this template with a nonlinear decoder and amortized inference.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
# Chapter 14: Autoencoders
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/autoencoders.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
An autoencoder learns to copy its input imperfectly, and the constraint that prevents perfect
|
||||
copying is where the learning happens. Undercompleteness, sparsity, denoising and contraction are
|
||||
four different ways to impose that constraint — each defines a different notion of "useful".
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Undercomplete autoencoder**: bottleneck smaller than the input. With linear units and MSE it
|
||||
recovers PCA's subspace.
|
||||
- **Regularized autoencoders**: capacity can exceed the input dimension as long as something else
|
||||
prevents identity — this is the chapter's key move.
|
||||
- **Sparse autoencoder**: penalize code activation; interpretable as a latent prior (Ch 13).
|
||||
- **Denoising autoencoder (DAE)**: corrupt the input, reconstruct the clean version. The learned
|
||||
map estimates the *score* — it points back toward the data manifold, which is the direct
|
||||
ancestor of score-based diffusion models.
|
||||
- **Contractive autoencoder (CAE)**: penalize the Jacobian norm of the encoder, so the
|
||||
representation resists input perturbation except along the manifold.
|
||||
- **Manifold learning view**: the encoder is sensitive along tangent directions of the data
|
||||
manifold and insensitive orthogonal to it.
|
||||
- **Stochastic encoders and decoders**: autoencoders as p(h|x) and p(x|h).
|
||||
- **Predictive sparse decomposition**; **applications**: dimensionality reduction, semantic
|
||||
hashing, pretraining.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Ask "**what stops it from learning the identity?**" — the answer names the inductive bias, and
|
||||
a model with no answer learns nothing useful.
|
||||
- Read a DAE's learned vector field as **pointing uphill in density**. Once you see that,
|
||||
diffusion models are the same idea run at many noise levels.
|
||||
- Treat the encoder as **amortized inference**: it replaces the per-example optimization of
|
||||
sparse coding with one forward pass.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **An overcomplete autoencoder with no regularizer** — it can and will learn a copy.
|
||||
- **Judging representation quality by reconstruction error**: low reconstruction error can mean
|
||||
the code memorized the input, which is the opposite of useful.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The denoising idea became the foundation of modern generative modelling: denoising score matching
|
||||
(Vincent 2011) → score-based models (Song & Ermon 2019) → DDPM (Ho et al. 2020) → the diffusion
|
||||
family. Greedy layerwise autoencoder pretraining, presented here as a live technique, disappeared
|
||||
— displaced first by better initialization/normalization and then by large-scale supervised and
|
||||
self-supervised pretraining. The VAE overshadowed the deterministic autoencoders for generation,
|
||||
while masked autoencoders (He et al. 2021) revived the corruption idea for vision pretraining.
|
||||
**Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Always name the constraint that prevents identity before training an autoencoder.
|
||||
2. Do not use reconstruction error as a representation-quality metric; evaluate downstream.
|
||||
3. Learn the denoising/score connection here — it is the cheapest on-ramp to diffusion models.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 13**: sparse coding, amortized.
|
||||
- **Ch 15**: what a "good" representation means.
|
||||
- **Ch 20**: VAEs, and the diffusion line that grew from denoising.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
# Chapter 15: Representation Learning
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/representation.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
What makes one representation better than another, and how do you get one without labels? The
|
||||
chapter names the properties worth wanting — distributed, disentangled, capturing causal factors
|
||||
— and the transfer mechanisms that let a representation earned on one task pay off on another.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Greedy layer-wise unsupervised pretraining**: the historically important technique that
|
||||
restarted the field in 2006; the chapter is already candid that its value had shrunk by 2016.
|
||||
- **Transfer learning and domain adaptation**: reuse a representation across tasks or
|
||||
distributions; **one-shot / zero-shot learning** as the limiting cases.
|
||||
- **Distributed representations**: n binary features describe 2ⁿ regions, versus n regions for a
|
||||
one-hot/symbolic code. This exponential advantage is the central argument for learned features.
|
||||
- **Disentangled factors**: separate underlying causes onto separate directions.
|
||||
- **Exponential gains from depth**: deeper composition of distributed features again multiplies
|
||||
expressible structure.
|
||||
- **Causal factors and semi-supervised learning**: unsupervised learning helps supervised learning
|
||||
exactly when p(x) and p(y|x) share structure — if the factors generating x include y's causes.
|
||||
- **Regularization priors that define good representations**: smoothness, linearity, multiple
|
||||
explanatory factors, hierarchy, sparsity, simplicity of factor dependencies, shared factors
|
||||
across tasks, manifolds, temporal/spatial coherence.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Judge a representation by **what becomes linearly separable** in it — that is the operational
|
||||
version of "good features."
|
||||
- Use the **shared-cause test** to predict whether unlabeled data will help: if p(x) tells you
|
||||
nothing about p(y|x), self-supervision will not rescue a supervised task.
|
||||
- Read distributed-vs-symbolic as **the reason embeddings beat lookup tables**, and note that the
|
||||
advantage is combinatorial, not merely empirical.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Expecting unsupervised pretraining to help unconditionally** — the chapter itself is careful
|
||||
here, and the 2016-era conclusion (it often does not, for large labeled datasets) was correct
|
||||
for the methods then available.
|
||||
- **Claiming disentanglement without a metric or an intervention.**
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
This chapter aged into relevance rather than out of it. Self-supervised learning became the
|
||||
dominant paradigm — contrastive methods (SimCLR, MoCo, CLIP), masked prediction (BERT, MAE), and
|
||||
next-token prediction at scale — vindicating the chapter's core bet while replacing every
|
||||
specific method it lists. Two corrections: greedy layer-wise pretraining is now purely
|
||||
historical, and unsupervised disentanglement was proven impossible without inductive bias
|
||||
(Locatello et al. 2019). Linear-probe evaluation became the standard test of representation
|
||||
quality. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Evaluate representations with linear probes and downstream transfer, not reconstruction.
|
||||
2. Before investing in self-supervision, argue that p(x) and p(y|x) share causes.
|
||||
3. Treat "disentangled" as a claim requiring an intervention-based test.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 14**: autoencoders as one route to a representation.
|
||||
- **Ch 1**: the promise made in the introduction, cashed out here.
|
||||
- **references/book_to_2026_delta.md**: the self-supervised learning line.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
# Chapter 16: Structured Probabilistic Models for Deep Learning
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/graphical_models.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
A joint distribution over n variables is exponentially large; a graph that encodes conditional
|
||||
independence makes it tractable. Directed models factor into conditionals, undirected models into
|
||||
unnormalized potentials divided by a partition function — and that partition function is the bill
|
||||
Chapters 17–19 spend their pages paying.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **The challenge of unstructured modelling**: memory, statistical efficiency, and inference cost
|
||||
all scale with the number of parameters in a full joint.
|
||||
- **Directed models (Bayesian networks)**: p(x) = ∏ p(xᵢ | parents(xᵢ)). Normalized by
|
||||
construction — no partition function.
|
||||
- **Undirected models (Markov random fields)**: p(x) = (1/Z) ∏ φ_c(x_c) over cliques. Z is the
|
||||
partition function, and it is a sum over all configurations.
|
||||
- **Energy-based models**: p(x) ∝ exp(−E(x)). Any positive distribution can be written this way.
|
||||
- **Separation / d-separation**: reading conditional independence off the graph.
|
||||
- **Converting between graph types; factor graphs** for disambiguating factorization.
|
||||
- **Sampling from graphical models**: ancestral sampling (easy, directed) vs Gibbs sampling
|
||||
(needed for undirected).
|
||||
- **Structure learning and latent variables**; the **restricted Boltzmann machine (RBM)** as the
|
||||
worked example.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Use the rule of thumb: **directed = easy sampling, harder inference with explaining-away;
|
||||
undirected = natural for mutual constraints, hard normalization.**
|
||||
- Read an energy function as a **soft constraint set**: low energy where constraints are
|
||||
satisfied. Design E, and p follows.
|
||||
- Remember that the partition function is not an inconvenience — it is **the** obstacle that
|
||||
organizes all of Part III.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Writing an undirected model and ignoring Z** until training refuses to work.
|
||||
- **Assuming a graph's missing edge means independence in the data** — it means the *model*
|
||||
asserts independence.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Deep undirected graphical models (deep Boltzmann machines, RBM stacks) are now largely historical
|
||||
as generative workhorses. But energy-based modelling did not die: it returned via score matching
|
||||
and diffusion (which sidestep Z entirely by learning ∇ log p rather than p), and via
|
||||
energy-based reinterpretations of contrastive learning. Autoregressive factorization — the
|
||||
directed side of this chapter — became the dominant paradigm through transformer language models,
|
||||
which are exactly chain-rule factorizations with a huge neural conditional. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Classify any generative proposal as directed or undirected first; it predicts which problems
|
||||
you will have.
|
||||
2. When Z is intractable, look for a formulation that never needs it (score, ratio, or
|
||||
autoregressive).
|
||||
3. Read modern LLMs as ancestral sampling from a directed chain — the framing here still applies.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 17–19**: the three responses to intractability (sample it, approximate Z, approximate the
|
||||
posterior).
|
||||
- **Ch 20**: the models built on these foundations.
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
# Chapter 17: Monte Carlo Methods
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/monte_carlo.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
When a sum or integral is intractable, estimate it with samples. The estimator is unbiased and
|
||||
its error shrinks as 1/√n regardless of dimension — which is why sampling survives where
|
||||
quadrature does not. The difficulty moves from computing the integral to obtaining the samples.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Monte Carlo estimation**: approximate E_p[f(x)] by the sample mean; variance falls as 1/n,
|
||||
standard error as 1/√n.
|
||||
- **Importance sampling**: sample from a tractable q and reweight by p/q. Variance depends
|
||||
entirely on how well q matches p·f — a bad proposal gives an estimator that is unbiased and
|
||||
useless.
|
||||
- **Markov chain Monte Carlo (MCMC)**: build a chain whose stationary distribution is the target,
|
||||
then use its states as (correlated) samples.
|
||||
- **Gibbs sampling**: resample each variable from its conditional given the rest; the natural
|
||||
MCMC scheme for graphical models.
|
||||
- **Burn-in / mixing time**: the chain needs time to forget its initialization and to move
|
||||
between modes.
|
||||
- **The mixing problem between separated modes**: with well-separated modes, transition
|
||||
probability between them is tiny and the chain reports a single mode as if it were everything.
|
||||
- **Tempering / annealing** as remedies: flatten the distribution so the chain can travel.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Treat a Monte Carlo estimate as a **measurement with error bars**; report the standard error,
|
||||
because "the estimate is unbiased" says nothing about whether n was large enough.
|
||||
- Diagnose a suspiciously confident sampler as a **mixing failure**, not a modelling success.
|
||||
- Read importance sampling's variance condition as: **you must already know roughly where the
|
||||
mass is** — this is why proposals matter more than sample counts.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Reporting MCMC samples without a mixing diagnostic.**
|
||||
- **Trusting importance weights with huge dynamic range** — an effective sample size of ~1 is
|
||||
common and invisible unless measured.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The chapter's methods remain textbook-correct, but their role in deep generative modelling
|
||||
shrank: diffusion models replaced slow MCMC-based sampling with a fixed, finite denoising chain,
|
||||
and modern generation is dominated by ancestral sampling from autoregressive models. Where MCMC
|
||||
is still used — Bayesian deep learning, some energy-based models — HMC/NUTS variants dominate
|
||||
over plain Gibbs. **Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Always pair a Monte Carlo estimate with its standard error.
|
||||
2. Check mixing before believing anything an MCMC sampler tells you about multimodality.
|
||||
3. Recognize the pattern "intractable expectation → sample it" — it recurs throughout Ch 18–20.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 16**: the undirected models that need sampling in the first place.
|
||||
- **Ch 18**: sampling as an ingredient in partition-function estimation.
|
||||
- **Ch 20**: sampling as generation.
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
# Chapter 18: Confronting the Partition Function
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/partition.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
For undirected models, the log-likelihood gradient splits into a positive phase (push energy down
|
||||
on data) and a negative phase (push energy up everywhere the model believes) — and the negative
|
||||
phase requires samples from the model itself. Every technique in this chapter is a way to afford
|
||||
that negative phase, or to avoid needing Z at all.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Positive and negative phase**: the gradient of log Z is an expectation under the *model*.
|
||||
This is the structural reason undirected models are expensive.
|
||||
- **Contrastive divergence (CD-k)**: initialize the negative chain at the data and run k steps.
|
||||
Fast; biased; produces spurious modes in regions the short chain never visits.
|
||||
- **Stochastic maximum likelihood / persistent CD (PCD)**: keep the chain's state across parameter
|
||||
updates so it has effectively run for a long time. Better mixing, at the cost of a chain that
|
||||
can fall behind fast-moving parameters.
|
||||
- **Pseudolikelihood**: replace the joint with a product of conditionals — Z cancels. Cheap; a
|
||||
different objective, so it optimizes for a different thing.
|
||||
- **Score matching**: match ∇_x log p instead of p, which eliminates Z because the gradient of
|
||||
log Z with respect to x is zero. **Ratio matching** and **denoising score matching** are the
|
||||
variants.
|
||||
- **Noise-contrastive estimation (NCE)**: turn density estimation into a classification problem —
|
||||
real data versus noise — and treat the normalizer as a learned parameter.
|
||||
- **Annealed importance sampling (AIS)** and **bridge sampling**: estimate Z itself, mainly for
|
||||
evaluation.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read the whole chapter as **four escape routes from Z**: sample it (CD/PCD), sidestep it
|
||||
(pseudolikelihood, score matching), learn around it (NCE), or estimate it for evaluation only
|
||||
(AIS).
|
||||
- Treat score matching's trick — **differentiate with respect to x, not θ, so Z vanishes** — as
|
||||
the single most consequential idea in Part III for modern practice.
|
||||
- Expect **spurious modes** from short-chain training; they are a predictable artifact, not a bug
|
||||
in your code.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Comparing likelihoods across undirected models without estimating Z** — the numbers are not
|
||||
comparable.
|
||||
- **Using CD-1 and reporting it as maximum likelihood.**
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Score matching's downstream career is the story here. Denoising score matching plus a noise
|
||||
schedule became score-based generative modelling (Song & Ermon 2019) and DDPM (Ho et al. 2020) —
|
||||
the diffusion family that now dominates image, audio and video generation. NCE became the
|
||||
backbone of contrastive representation learning (word2vec's negative sampling, then InfoNCE and
|
||||
CLIP). AIS remains the standard tool for evaluating likelihoods when Z is unknown. Contrastive
|
||||
divergence and PCD are largely historical. **Confidence: high.**
|
||||
|
||||
**This is the chapter whose ideas travelled furthest — read it even if you never train a
|
||||
Boltzmann machine.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. When a normalizer blocks you, ask which of the four escape routes fits your objective.
|
||||
2. Learn score matching properly; diffusion models are unintelligible without it.
|
||||
3. Never compare unnormalized likelihoods.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 17**: the sampling machinery the negative phase depends on.
|
||||
- **Ch 14**: denoising autoencoders as score estimators.
|
||||
- **Ch 20**: the generative models built from these objectives.
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
# Chapter 19: Approximate Inference
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/inference.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
Computing p(h|x) — the posterior over latents — is intractable for most interesting models. The
|
||||
fix is to reframe inference as optimization: choose a tractable family q, and maximize a lower
|
||||
bound on the log-likelihood. The gap between the bound and the truth is exactly a KL divergence.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Inference as optimization**: pick q(h) to maximize the **evidence lower bound (ELBO)**,
|
||||
L(q) = log p(x) − D_KL(q(h) ‖ p(h|x)). Maximizing L both fits the model and tightens the bound.
|
||||
- **Expectation maximization (EM)**: alternate between setting q to the current posterior
|
||||
(E-step) and maximizing with respect to parameters (M-step).
|
||||
- **MAP inference and sparse coding**: taking a point estimate of h is a degenerate q (a Dirac);
|
||||
sparse coding's inference step is exactly MAP inference.
|
||||
- **Variational inference and mean field**: restrict q to a factorized family, q(h) = ∏ q(hᵢ),
|
||||
and derive fixed-point updates. Tractability is bought with an independence assumption.
|
||||
- **Reverse-KL consequences**: the ELBO uses D(q‖p), which is mode-seeking — variational
|
||||
posteriors are characteristically **too narrow**, and underestimate uncertainty.
|
||||
- **Learned approximate inference / amortization**: train a network to output q's parameters
|
||||
directly, replacing per-example optimization with one forward pass.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Read the ELBO as **"log-likelihood minus the cost of your approximation"** — improving q and
|
||||
improving the model are the same optimization.
|
||||
- Expect **underestimated variance** from mean-field posteriors, and never report variational
|
||||
uncertainty as calibrated without checking.
|
||||
- Treat amortization as the **encoder** of Chapter 14: the VAE is exactly this idea plus the
|
||||
reparameterization trick.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Reporting a mean-field posterior's credible intervals as if they were exact.**
|
||||
- **Blaming the model for a poor fit** that is actually a too-restrictive q — diagnose the bound
|
||||
before the model.
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
Amortized variational inference became routine (VAEs and descendants); normalizing flows and
|
||||
importance-weighted bounds (IWAE) gave tighter, more expressive posteriors than mean field. But
|
||||
the strategic picture changed more: modern large generative models largely **avoid latent-variable
|
||||
posteriors altogether** — autoregressive transformers have no posterior to infer, and diffusion
|
||||
models use a fixed forward process, so their "inference" is trivial by construction. Variational
|
||||
inference remains central in Bayesian deep learning and in structured latent-variable modelling.
|
||||
**Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Write the ELBO down for any latent-variable model you train; it tells you what you are
|
||||
actually optimizing.
|
||||
2. Assume a mean-field posterior is over-confident until proven otherwise.
|
||||
3. Ask whether your problem needs a latent posterior at all — many modern designs are structured
|
||||
to avoid one.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 13 / Ch 14**: the latent-variable models needing inference, and amortization.
|
||||
- **Ch 20**: the VAE, assembled from this chapter plus the reparameterization trick.
|
||||
- **Ch 3**: the KL asymmetry that dictates the narrow-posterior failure mode.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
# Chapter 20: Deep Generative Models
|
||||
|
||||
**Source chapter (free, official):** https://www.deeplearningbook.org/contents/generative_models.html
|
||||
|
||||
## Core Idea
|
||||
|
||||
The book's capstone: every generative model family circa 2016, organized by how it handles the
|
||||
intractable quantity. Boltzmann machines and their deep variants pay for the partition function;
|
||||
VAEs bound the likelihood; GANs avoid likelihood entirely; autoregressive models factor it away.
|
||||
That taxonomy — **not** the specific models — is what still holds.
|
||||
|
||||
## Frameworks Introduced
|
||||
|
||||
- **Boltzmann machines, RBMs, deep belief networks, deep Boltzmann machines**: energy-based
|
||||
models trained with the Ch 18 machinery. Historically pivotal; now largely superseded.
|
||||
- **Variational autoencoder (VAE)**: an encoder produces q(h|x), a decoder produces p(x|h), and
|
||||
the **reparameterization trick** makes the sampling step differentiable so the ELBO can be
|
||||
optimized by backprop. Blurry samples are the predictable consequence of the likelihood
|
||||
objective plus a limited posterior family.
|
||||
- **Generative adversarial networks (GANs)**: a generator and a discriminator in a minimax game;
|
||||
no explicit likelihood, sharp samples, unstable training, and mode collapse as the
|
||||
characteristic failure.
|
||||
- **Autoregressive / fully-visible belief networks**: factor p(x) by the chain rule and model each
|
||||
conditional. Exact likelihood, sequential sampling. (NADE, PixelRNN/PixelCNN, WaveNet era.)
|
||||
- **Generative stochastic networks; denoising-based generation**: learning a transition operator
|
||||
rather than a distribution — the direct ancestor of diffusion.
|
||||
- **Evaluating generative models**: the chapter's warning that likelihood, sample quality and
|
||||
downstream usefulness are **three different axes** that routinely disagree.
|
||||
|
||||
## Mental Models
|
||||
|
||||
- Classify any new generative model by **what it does about the intractable term**: bound it,
|
||||
avoid it, factor it away, or learn a sampler directly. New families are new answers to that one
|
||||
question.
|
||||
- Read blurriness vs mode collapse as **the KL asymmetry from Ch 3 made visible**: likelihood-based
|
||||
models cover modes and blur; adversarial models sharpen and drop modes.
|
||||
- Never accept a **single** evaluation number for a generative model.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Comparing FID across papers with different preprocessing** — the metric is not portable.
|
||||
- **Treating sample quality as evidence of density estimation quality**, or vice versa.
|
||||
- **Starting a new project with a Boltzmann machine.**
|
||||
|
||||
## What changed after 2016
|
||||
|
||||
The most-superseded chapter in the book, and worth reading anyway for its taxonomy.
|
||||
**Diffusion models** (Ho et al. 2020, built on Ch 18's denoising score matching and Ch 14's
|
||||
denoising autoencoders) displaced GANs for image, audio and video generation. **Autoregressive
|
||||
transformers** became the dominant generative model overall — the chain-rule factorization
|
||||
described here, scaled. VAEs persist mainly as latent-space compressors inside latent diffusion
|
||||
pipelines rather than as end-user generators. GANs remain useful for fast, low-step generation.
|
||||
Boltzmann machines are history. Evaluation remains unsolved, exactly as the chapter warned.
|
||||
**Confidence: high.**
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. Use the taxonomy, not the model list — it classifies architectures invented since.
|
||||
2. Expect blurring from likelihood objectives and mode-dropping from adversarial ones; pick your
|
||||
failure mode deliberately.
|
||||
3. Evaluate generative models on at least likelihood, sample quality, and downstream use.
|
||||
|
||||
## Connects To
|
||||
|
||||
- **Ch 18**: score matching, whose descendants replaced most of this chapter.
|
||||
- **Ch 19**: the ELBO the VAE optimizes.
|
||||
- **references/book_to_2026_delta.md**: the diffusion and autoregressive-transformer lines.
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
# Cheatsheet
|
||||
|
||||
Decision rules and thresholds. One line each; the chapter has the reasoning.
|
||||
|
||||
## Loss selection (ch03, ch06)
|
||||
|
||||
| Target | Distribution | Output unit | Loss |
|
||||
|---|---|---|---|
|
||||
| Real, unbounded | Gaussian | linear | MSE |
|
||||
| Binary | Bernoulli | sigmoid | binary cross-entropy |
|
||||
| One of K | Categorical | softmax | cross-entropy |
|
||||
| Real, heavy-tailed | Laplace | linear | MAE |
|
||||
| Multimodal real | Mixture | mixture density | NLL of the mixture |
|
||||
|
||||
Never pair sigmoid output with MSE — the gradient vanishes where the model is most wrong.
|
||||
|
||||
## Underfit vs overfit (ch05, ch11)
|
||||
|
||||
| Train error | Val error | Verdict | First action |
|
||||
|---|---|---|---|
|
||||
| High | High | Underfit / optimization | Check LR, init, capacity — **not** more data |
|
||||
| Low | High | Overfit | More data → augmentation → regularization |
|
||||
| Low | Low | Working | Stop; check the metric is the right one |
|
||||
| ~0 on 20 examples: fails | — | Bug | Fix the pipeline before anything else |
|
||||
|
||||
Past the interpolation threshold, prefer more data / longer training / more regularization over
|
||||
shrinking the model (double descent, ch05).
|
||||
|
||||
## Optimizer defaults (ch08)
|
||||
|
||||
| Situation | Reach for |
|
||||
|---|---|
|
||||
| Any starting point | AdamW, LR ~1e-3 (small nets) / ~1e-4 (large), warmup + cosine decay |
|
||||
| Recurrence in the graph | Add gradient clipping, always |
|
||||
| Tiny batches | Group/layer norm, not batch norm |
|
||||
| Loss flat, grad norm large | Ill-conditioning — momentum, adaptive, or normalization |
|
||||
| Loss NaN | Numerics first: log(0), div by ~0, exp overflow, exploding grad |
|
||||
|
||||
## Regularization ladder (ch07)
|
||||
|
||||
More real data → label-preserving augmentation → early stopping → decoupled weight decay →
|
||||
label smoothing → dropout → smaller model. One at a time; measure the gap after each.
|
||||
|
||||
## Architecture prior (ch09, ch10)
|
||||
|
||||
| Data | Prior that fits | Note |
|
||||
|---|---|---|
|
||||
| Grid, translation-invariant statistics | Convolution | Cheapest when data is limited |
|
||||
| Sequence, long-range dependence | Attention/transformer | The 2016 RNN advice is superseded |
|
||||
| Sequence, very long context | State-space / linear recurrence | Post-2016; ch10's gradient analysis applies |
|
||||
| Arbitrary tabular columns | Neither | Locality prior is false |
|
||||
|
||||
## Generative model taxonomy (ch20)
|
||||
|
||||
| Family | Handles Z by | Characteristic failure |
|
||||
|---|---|---|
|
||||
| Autoregressive | Chain rule — no Z | Slow sequential sampling |
|
||||
| VAE | Bounding likelihood (ELBO) | Blurry samples |
|
||||
| GAN | Avoiding likelihood | Mode collapse, unstable training |
|
||||
| Diffusion / score-based | Learning ∇ log p | Many sampling steps (mitigable) |
|
||||
| Boltzmann machines | Sampling the negative phase | Historical; do not start here |
|
||||
|
||||
## Numerical hygiene (ch04)
|
||||
|
||||
Log-space for probability products · stabilized/fused softmax and cross-entropy · clip gradients
|
||||
with recurrence · bf16 over fp16 when range matters · check condition number before blaming LR.
|
||||
|
||||
## Study-order rule (ch01–ch20)
|
||||
|
||||
Ch 5 and Ch 11 are the highest-value chapters for a practitioner and can be read early.
|
||||
Ch 2–4 are prerequisites for Ch 8 specifically. Ch 13→14→19→20 is the only strict chain in
|
||||
Part III. Ch 18 is worth reading even if you never train an undirected model — score matching
|
||||
became diffusion.
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
# Glossary
|
||||
|
||||
Terms as this skill uses them, with the chapter that develops each. Definitions are written for
|
||||
lookup, not for teaching — read the chapter file for the reasoning.
|
||||
|
||||
- **Adam / AdamW** (ch08, ch07) — adaptive optimizer combining momentum with per-parameter scaling. AdamW decouples weight decay from the gradient, which Adam+L2 does not do correctly.
|
||||
- **Amortized inference** (ch19, ch14) — training a network to predict a posterior's parameters instead of optimizing per example.
|
||||
- **Ancestral sampling** (ch16) — sampling a directed model by drawing each variable after its parents.
|
||||
- **Attention** (ch10) — a learned weighted read over a set of positions; introduced here as a fix for the seq2seq bottleneck, later the whole architecture.
|
||||
- **Backpropagation** (ch06) — reverse-mode automatic differentiation over the computation graph.
|
||||
- **Batch normalization** (ch08) — reparameterization normalizing layer inputs by batch statistics.
|
||||
- **Bias–variance decomposition** (ch05) — expected error as bias² + variance + irreducible noise.
|
||||
- **Capacity, effective vs representational** (ch05) — functions reachable by the optimizer vs functions expressible by the architecture.
|
||||
- **Condition number** (ch02, ch04) — ratio of largest to smallest singular value; how amplified small perturbations become.
|
||||
- **Contrastive divergence (CD-k)** (ch18) — biased approximation of the negative phase using a short chain started at the data.
|
||||
- **Cross-entropy** (ch03) — H(p,q) = H(p) + D_KL(p‖q); minimizing it over q minimizes KL.
|
||||
- **Curse of dimensionality** (ch05, ch01) — configurations grow exponentially with dimension, defeating local-interpolation methods.
|
||||
- **Denoising autoencoder (DAE)** (ch14) — reconstructs clean input from a corrupted copy; its learned map estimates the score.
|
||||
- **Disentangled representation** (ch15) — separate generative factors on separate directions.
|
||||
- **Distributed representation** (ch15) — n features describing exponentially many regions, versus one-hot's n.
|
||||
- **d-separation** (ch16) — reading conditional independence off a directed graph.
|
||||
- **Double descent** (ch05) — test error falling again past the interpolation threshold; post-dates the book and qualifies its U-curve.
|
||||
- **ELBO** (ch19) — evidence lower bound; log p(x) minus KL(q ‖ posterior).
|
||||
- **Empirical risk minimization** (ch08) — minimizing training loss as a surrogate for unmeasurable true risk.
|
||||
- **Energy-based model** (ch16) — p(x) ∝ exp(−E(x)).
|
||||
- **Equivariance** (ch09) — a shift in input produces the corresponding shift in output; convolution's defining property.
|
||||
- **Exposure bias** (ch10) — train/inference mismatch from teacher forcing.
|
||||
- **Gradient clipping** (ch08, ch10) — rescaling gradients above a norm threshold to survive cliffs.
|
||||
- **Ill-conditioning** (ch04, ch08) — wide Hessian eigenvalue spread; loss stalls while gradient norm stays large.
|
||||
- **Importance sampling** (ch17) — sampling from q and reweighting by p/q.
|
||||
- **KL divergence** (ch03) — asymmetric divergence between distributions; direction decides mode-covering vs mode-seeking.
|
||||
- **Manifold hypothesis** (ch05, ch14) — real data concentrates near a low-dimensional manifold.
|
||||
- **Maximum likelihood estimation (MLE)** (ch05) — parameters maximizing data probability; the justification for standard losses.
|
||||
- **Mean field** (ch19) — factorized variational family; buys tractability with an independence assumption.
|
||||
- **Mixing time / burn-in** (ch17) — steps an MCMC chain needs before its samples are usable.
|
||||
- **Mode collapse** (ch20) — a generator covering only part of the data distribution.
|
||||
- **Momentum / Nesterov momentum** (ch08) — velocity accumulation across gradient steps.
|
||||
- **No Free Lunch theorem** (ch05) — averaged over all distributions, all learners tie; priors carry all leverage.
|
||||
- **Noise-contrastive estimation (NCE)** (ch18) — density estimation recast as data-versus-noise classification.
|
||||
- **Partition function (Z)** (ch16, ch18) — the normalizer of an undirected model; the central intractability of Part III.
|
||||
- **Persistent contrastive divergence (PCD)** (ch18) — CD with a chain persisted across updates.
|
||||
- **Pooling** (ch09) — local aggregation giving approximate invariance to small translations.
|
||||
- **Positive / negative phase** (ch18) — the two terms of an undirected model's likelihood gradient.
|
||||
- **Pseudolikelihood** (ch18) — product of conditionals, in which Z cancels.
|
||||
- **Receptive field** (ch09) — the input region influencing one output unit.
|
||||
- **Regularization** (ch07, ch05) — any change intended to reduce generalization error but not training error.
|
||||
- **Reparameterization trick** (ch20) — expressing a sample as a differentiable function of noise, making the ELBO backprop-able.
|
||||
- **Representation learning** (ch15, ch01) — learning features rather than hand-designing them.
|
||||
- **Saddle point** (ch08) — critical point that is a minimum in some directions and a maximum in others; the dominant critical point in high dimensions.
|
||||
- **Score matching** (ch18) — matching ∇_x log p instead of p, so Z vanishes; the basis of diffusion models.
|
||||
- **SGD, minibatch** (ch08) — gradient descent on noisy gradients from data subsets.
|
||||
- **Singular value decomposition (SVD)** (ch02) — A = U D Vᵀ for any matrix.
|
||||
- **Sparse coding** (ch13) — heavy-tailed latent prior with inference by optimization.
|
||||
- **Teacher forcing** (ch10) — training an autoregressive model on ground-truth history.
|
||||
- **Universal approximation theorem** (ch06) — one wide hidden layer suffices in principle; silent on width, learnability and sample cost.
|
||||
- **Vanishing / exploding gradients** (ch10, ch08) — repeated Jacobian multiplication shrinking or growing gradients through depth or time.
|
||||
- **Variational autoencoder (VAE)** (ch20, ch19) — amortized variational inference plus the reparameterization trick.
|
||||
- **Weight decay** (ch07) — pulling parameters toward zero; equivalent to L2 for SGD, not for Adam.
|
||||
|
|
@ -0,0 +1,92 @@
|
|||
# Patterns
|
||||
|
||||
Recurring techniques from the book, stated as instruments with their trade-offs. Each names the
|
||||
chapter that develops it.
|
||||
|
||||
## Derive the loss from the output distribution (ch03, ch06)
|
||||
|
||||
Choose what p(y|x) is, then take its negative log. Gaussian → MSE, Bernoulli → binary
|
||||
cross-entropy, categorical → cross-entropy, Laplace → MAE, mixture → mixture density loss.
|
||||
**Trade-off:** none — this is strictly better than picking a loss by habit. **Failure mode it
|
||||
prevents:** MSE on bounded, count-valued or heavily skewed targets.
|
||||
|
||||
## Work in log-space (ch04)
|
||||
|
||||
Sums of logs instead of products of probabilities; stabilized softmax and `log_softmax` rather
|
||||
than hand-rolled exp/normalize. **Trade-off:** slightly more care at implementation time.
|
||||
**Prevents:** silent underflow, NaN losses, and inf logits.
|
||||
|
||||
## Train-error-first triage (ch11, ch05)
|
||||
|
||||
Read training error before deciding anything. High training error → capacity or optimization is
|
||||
the bottleneck; more data will not help. Low training error with high validation error → data or
|
||||
regularization. **Trade-off:** requires honest, non-leaky splits. **Prevents:** the most expensive
|
||||
common mistake, which is collecting data to fix underfitting.
|
||||
|
||||
## Overfit a tiny subset as a smoke test (ch11)
|
||||
|
||||
Take 10–50 examples and drive training loss to ~0. If you cannot, you have a bug — not a hard
|
||||
problem. **Trade-off:** minutes. **Prevents:** weeks of tuning around a broken data pipeline,
|
||||
a wrong loss reduction, or a detached gradient.
|
||||
|
||||
## Random search over grid search (ch11)
|
||||
|
||||
With more than about two hyperparameters, random search finds better configurations for the same
|
||||
budget because it does not spend trials re-testing unimportant dimensions. **Trade-off:** results
|
||||
are less tidy to tabulate. **Prevents:** exponential waste in dimensions that do not matter.
|
||||
|
||||
## Regularize in cost order (ch07)
|
||||
|
||||
More real data → augmentation with label-preserving transforms → early stopping → weight decay →
|
||||
dropout → architecture change. Add one at a time and measure the train/val gap after each.
|
||||
**Trade-off:** slower than stacking everything. **Prevents:** an unattributable result you cannot
|
||||
tune.
|
||||
|
||||
## Parameter sharing over parameter penalties (ch07, ch09)
|
||||
|
||||
When a real invariance exists, encode it structurally (convolution, weight tying) rather than
|
||||
penalizing a free parameter into behaving. **Trade-off:** the prior is hard — wrong invariance
|
||||
means a wrong model, with no way for data to override it. **Prevents:** paying for capacity you
|
||||
then have to regularize away.
|
||||
|
||||
## Diagnose by gradient behaviour (ch08, ch04)
|
||||
|
||||
Gradient norm exploding → clip. Norm large, loss flat → ill-conditioning; use momentum, an
|
||||
adaptive optimizer, or normalization. Norm near zero with high loss → saturation or dead units;
|
||||
check initialization and activations. Loss NaN → numerics before modelling. **Trade-off:**
|
||||
requires instrumentation. **Prevents:** changing the architecture to fix an optimizer problem.
|
||||
|
||||
## Tune learning rate and schedule before optimizer family (ch08)
|
||||
|
||||
The schedule usually dominates the choice among SGD/Adam variants. **Trade-off:** none. **Prevents:**
|
||||
optimizer-shopping while the real problem is a step size two orders of magnitude off.
|
||||
|
||||
## Name what prevents the identity map (ch14)
|
||||
|
||||
Every autoencoder needs an answer: bottleneck, sparsity penalty, input corruption, or Jacobian
|
||||
contraction. **Trade-off:** each constraint defines a different notion of useful.
|
||||
**Prevents:** an overcomplete autoencoder that learns a copy and reports a low loss.
|
||||
|
||||
## Escape the partition function deliberately (ch18, ch16)
|
||||
|
||||
Four routes: sample the negative phase (CD/PCD), sidestep Z algebraically (pseudolikelihood,
|
||||
score matching), learn around it (NCE), or estimate it for evaluation only (AIS). **Trade-off:**
|
||||
each optimizes a different objective, so they are not interchangeable. **Prevents:** discovering
|
||||
mid-project that your undirected model cannot be trained or compared.
|
||||
|
||||
## Pick your generative failure mode (ch20, ch03)
|
||||
|
||||
Likelihood-based objectives cover modes and blur. Adversarial objectives sharpen and drop modes.
|
||||
This follows from KL direction, so it is a design choice, not bad luck. **Trade-off:** you must
|
||||
decide which error your application tolerates. **Prevents:** treating blurriness as a bug to be
|
||||
tuned away.
|
||||
|
||||
## Evaluate representations by transfer, not reconstruction (ch15, ch14)
|
||||
|
||||
Linear probes and downstream task performance. **Trade-off:** needs a downstream task.
|
||||
**Prevents:** optimizing reconstruction error into a code that memorized the input.
|
||||
|
||||
## Report Monte Carlo estimates with error bars (ch17)
|
||||
|
||||
Standard error and a mixing diagnostic, always. **Trade-off:** more reporting.
|
||||
**Prevents:** a confidently unimodal answer from a chain that never left its starting mode.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
# The 2016 → 2026 Delta
|
||||
|
||||
The book was published in 2016. *Attention Is All You Need* appeared in 2017. Everything below
|
||||
tracks what a reader should hold, discount, or replace — per chapter, with a confidence level on
|
||||
each claim.
|
||||
|
||||
Confidence levels: 🟢 well-established and replicated · 🟡 supported but contested or
|
||||
context-dependent · 🔴 folklore, named as folklore.
|
||||
|
||||
---
|
||||
|
||||
## Chapters that aged well (read as current)
|
||||
|
||||
| Ch | Why it holds |
|
||||
|---|---|
|
||||
| 2, 3, 4 | Mathematics does not age. 🟢 |
|
||||
| 5 | The learning-theory frame is current — with one correction (double descent, below). 🟢 |
|
||||
| 11 | The practical workflow became the backbone of MLOps practice. 🟢 |
|
||||
| 16 | The directed/undirected taxonomy classifies models invented since, including transformers as chain-rule factorizations. 🟢 |
|
||||
| 18 | Score matching became diffusion; NCE became contrastive learning. The chapter's ideas travelled further than its models. 🟢 |
|
||||
|
||||
## The five corrections that matter
|
||||
|
||||
### 1. Double descent qualifies the U-curve (ch05)
|
||||
|
||||
The classical capacity curve — error falls, then rises past the optimum — is incomplete. Past the
|
||||
interpolation threshold, test error can descend a second time. Belkin et al. (2019) named it;
|
||||
Nakkiran et al. (2020) showed model-wise, epoch-wise and sample-wise variants in deep networks.
|
||||
The bias–variance algebra is unchanged; the *advice* "shrink the model when it overfits" is no
|
||||
longer the only correct move. 🟢
|
||||
|
||||
### 2. AdamW: weight decay ≠ L2 under adaptive optimizers (ch07, ch08)
|
||||
|
||||
Adding an L2 term to the loss and applying weight decay to the update are equivalent for plain
|
||||
SGD and **not** equivalent for Adam, because the adaptive denominator rescales the penalty.
|
||||
Loshchilov & Hutter (2017/2019) decoupled them; AdamW is now the default. The book's Ch 7 treats
|
||||
the two as interchangeable. 🟢
|
||||
|
||||
### 3. Transformers displaced recurrence (ch10, ch12)
|
||||
|
||||
Vaswani et al. (2017) removed recurrence entirely: attention gives an O(1) path between any two
|
||||
positions and parallelizes over sequence length. What survives from Ch 10: the vanishing/exploding
|
||||
gradient analysis, gradient clipping, teacher forcing, exposure bias. What is superseded: the
|
||||
architecture recommendation. Note the return of linear-time recurrence in state-space models
|
||||
(S4, 2021; Mamba, 2023) for long context — which makes Ch 10's analysis live again. 🟢
|
||||
|
||||
### 4. Diffusion displaced the Part III generative models (ch14, ch18, ch20)
|
||||
|
||||
The line runs directly through the book: denoising autoencoders (Ch 14) → denoising score
|
||||
matching (Ch 18) → score-based generative models (Song & Ermon, 2019) → DDPM (Ho et al., 2020).
|
||||
Diffusion now dominates image, audio and video generation; autoregressive transformers dominate
|
||||
text. VAEs survive mainly as latent-space compressors inside latent-diffusion pipelines. GANs
|
||||
remain useful for few-step generation. Boltzmann machines are historical. 🟢
|
||||
|
||||
### 5. Self-supervised learning vindicated Ch 15 while replacing its methods
|
||||
|
||||
The chapter's bet — that unsupervised representation learning would matter — was right, and every
|
||||
specific method it lists was replaced. Contrastive (SimCLR, MoCo, CLIP), masked prediction (BERT,
|
||||
MAE) and next-token prediction at scale are the modern routes. Two corrections: greedy layer-wise
|
||||
pretraining is now purely historical 🟢, and unsupervised disentanglement was shown impossible
|
||||
without inductive biases or supervision (Locatello et al., 2019) 🟢.
|
||||
|
||||
## Additions the book has no chapter for
|
||||
|
||||
| Topic | Status |
|
||||
|---|---|
|
||||
| **Neural scaling laws** (Kaplan 2020; Hoffmann 2022) | Quantifies Ch 1's "scale matters" and Ch 11's data-vs-capacity decision. 🟢 |
|
||||
| **Normalization placement** — pre-norm vs post-norm residual, LayerNorm/RMSNorm over BatchNorm | Standard for deep sequence stacks. 🟢 |
|
||||
| **Warmup + cosine decay schedules** | Standard for transformer training. 🟢 |
|
||||
| **Calibration** (Guo et al., 2017) | Modern networks are systematically overconfident; temperature scaling is the cheap fix. Ch 3 does not mention it. 🟢 |
|
||||
| **RLHF / DPO alignment** | Entirely outside the book. 🟢 |
|
||||
| **Mixture-of-Experts** | The mature form of Ch 12's conditional computation. 🟢 |
|
||||
| **Mixed precision (fp16 loss scaling, bf16)** | Made Ch 4's numerics an operational daily concern. 🟢 |
|
||||
| **Lottery-ticket / pruning theory** (Frankle & Carbin, 2019) | Active; the strong form remains contested. 🟡 |
|
||||
| **"Batch norm works by reducing internal covariate shift"** | The original explanation; challenged by Santurkar et al. (2018), who attribute the effect to smoothing the loss landscape. Treat the mechanism as unsettled. 🟡 |
|
||||
| **"Local minima are the problem in deep nets"** | 🔴 Folklore the book itself corrects: high-dimensional critical points are overwhelmingly saddles (Ch 8). |
|
||||
| **"You need a GPU cluster to learn deep learning"** | 🔴 Folklore. Every mechanism in Parts I–II is observable on a laptop-scale model. |
|
||||
|
||||
## How to use this file
|
||||
|
||||
When a chapter's advice conflicts with current practice, the conflict is almost always in the
|
||||
**recommendation**, not the **analysis**. The book explains why things fail; the field has changed
|
||||
what it reaches for. Keep the diagnosis, replace the prescription.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Vaswani et al., "Attention Is All You Need," NeurIPS 2017 — arXiv:1706.03762.
|
||||
2. Loshchilov & Hutter, "Decoupled Weight Decay Regularization," ICLR 2019 — arXiv:1711.05101.
|
||||
3. Belkin, Hsu, Ma & Mandal, "Reconciling modern machine-learning practice and the classical
|
||||
bias–variance trade-off," PNAS 116(32), 2019; Nakkiran et al., "Deep Double Descent," ICLR 2020.
|
||||
4. Ho, Jain & Abbeel, "Denoising Diffusion Probabilistic Models," NeurIPS 2020 — arXiv:2006.11239;
|
||||
Song & Ermon, "Generative Modeling by Estimating Gradients of the Data Distribution,"
|
||||
NeurIPS 2019.
|
||||
5. Kaplan et al., "Scaling Laws for Neural Language Models," 2020 — arXiv:2001.08361;
|
||||
Hoffmann et al., "Training Compute-Optimal Large Language Models," 2022 — arXiv:2203.15556.
|
||||
6. Locatello et al., "Challenging Common Assumptions in the Unsupervised Learning of Disentangled
|
||||
Representations," ICML 2019 (best paper) — arXiv:1811.12359.
|
||||
7. Guo, Pleiss, Sun & Weinberger, "On Calibration of Modern Neural Networks," ICML 2017;
|
||||
Santurkar et al., "How Does Batch Normalization Help Optimization?," NeurIPS 2018.
|
||||
8. Dosovitskiy et al., "An Image is Worth 16x16 Words" (ViT), ICLR 2021 — arXiv:2010.11929;
|
||||
Liu et al., "A ConvNet for the 2020s" (ConvNeXt), CVPR 2022.
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
# Prerequisite Map
|
||||
|
||||
The book's part structure is not its dependency structure. Readers who go strictly front-to-back
|
||||
spend weeks in Chapters 2–4 before touching a network, and a large fraction stop there. This file
|
||||
gives the actual graph.
|
||||
|
||||
---
|
||||
|
||||
## The real dependencies
|
||||
|
||||
```
|
||||
ch01 ──────────────────────────────────► (context only; no hard dependents)
|
||||
|
||||
ch02 (linear algebra) ──┬──► ch04 ──┬──► ch08 ──► ch09, ch10
|
||||
│ │
|
||||
ch03 (probability) ─────┴──► ch05 ──┴──► ch06 ──► ch07 ──► ch11
|
||||
│
|
||||
└──► ch13 ──► ch14 ──► ch15
|
||||
│
|
||||
ch03 ──► ch16 ──► ch17 ──► ch18 ──► ch19 ──► ch20
|
||||
```
|
||||
|
||||
**Hard prerequisites** (skipping these makes the target chapter unreadable, not merely harder):
|
||||
|
||||
| To read | You need | Specifically |
|
||||
|---|---|---|
|
||||
| ch08 | ch02, ch04 | eigenvalues, condition number, Hessian, Taylor expansion |
|
||||
| ch06 | ch03 | the output distributions that determine output units and losses |
|
||||
| ch07 | ch05, ch04 | bias–variance; KKT for the constraint view of penalties |
|
||||
| ch18 | ch16, ch17 | the partition function; sampling |
|
||||
| ch19 | ch03 | KL divergence, and its asymmetry |
|
||||
| ch20 | ch13, ch19 | the latent-variable template; the ELBO |
|
||||
|
||||
**Soft prerequisites** — helpful, not blocking: ch02 before ch13 (PCA), ch09 before ch12,
|
||||
ch10 before ch12.
|
||||
|
||||
## Chapters you can read early, out of order
|
||||
|
||||
- **ch05** — the single highest-value chapter for a practitioner. Needs only basic probability.
|
||||
- **ch11** — readable on day one and immediately actionable. Depends on ch05's vocabulary only.
|
||||
- **ch01** — context; skim it.
|
||||
- **ch09** — comprehensible with ch06 alone if you accept the optimization details on faith.
|
||||
|
||||
## Chapters you can defer or skip, by goal
|
||||
|
||||
| Goal | Skip or defer |
|
||||
|---|---|
|
||||
| Applied practitioner shipping models | Part III entirely (ch13–20), except ch14's denoising section |
|
||||
| Preparing for modern generative modelling | Nothing in Part III — but read ch18 before ch20 |
|
||||
| Interview / fundamentals refresher | ch12 (dated), ch17–19 (unless the role is probabilistic ML) |
|
||||
| Understanding transformers | ch10 for the gradient analysis; then leave the book for the 2017 paper |
|
||||
| Research in probabilistic ML | Everything; Part III is the reason this book has no substitute |
|
||||
|
||||
## The Part I wall, and how to get through it
|
||||
|
||||
Chapters 2–4 are compressed reference material, not pedagogy. Three viable strategies:
|
||||
|
||||
1. **Read-on-demand** (recommended for applied readers): skim ch2–4 once for vocabulary, start at
|
||||
ch05, and return to a specific section when ch08 uses it. The prerequisite table above tells
|
||||
you exactly which section.
|
||||
2. **Front-load** (recommended if you intend to read Part III): work ch2–4 properly with a
|
||||
separate linear algebra source alongside. Budget 2–3× the page count in time.
|
||||
3. **Substitute**: use a dedicated linear algebra or probability text for Part I and treat these
|
||||
chapters purely as a notation reference for the rest of the book.
|
||||
|
||||
Strategy 1 is right for most readers and is what `scripts/reading_path_planner.py` recommends by
|
||||
default. Strategy 2 is right when Part III is the destination — its chapters compound, and gaps
|
||||
compound with them.
|
||||
|
||||
## Time budgeting
|
||||
|
||||
Reported reading times vary by an order of magnitude, so treat any figure as a planning
|
||||
assumption, not a fact. As a planning heuristic used by the reading-path planner: a Part I or
|
||||
Part III chapter is roughly 1.5–2× the time of a Part II chapter of the same length, because the
|
||||
derivation density is higher. The planner exposes its per-chapter assumptions in
|
||||
`--output json` so you can recalibrate them against your own first chapter.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Goodfellow, Bengio & Courville, *Deep Learning*, MIT Press 2016 — table of contents and part
|
||||
structure: https://www.deeplearningbook.org/
|
||||
2. Sweller, van Merriënboer & Paas, "Cognitive Architecture and Instructional Design," *Educational
|
||||
Psychology Review* 10(3), 1998 — element interactivity and intrinsic load, which is what makes
|
||||
ch2–4 expensive.
|
||||
3. Sweller & Cooper, "The Use of Worked Examples as a Substitute for Problem Solving," *Cognition
|
||||
and Instruction* 2(1), 1985 — the worked-example effect for novices.
|
||||
4. Bjork & Bjork, "Making Things Hard on Yourself, But in a Good Way: Creating Desirable
|
||||
Difficulties to Enhance Learning," 2011.
|
||||
5. Roediger & Karpicke, "Test-Enhanced Learning," *Psychological Science* 17(3), 2006.
|
||||
6. Ericsson, Krampe & Tesch-Römer, "The Role of Deliberate Practice in the Acquisition of Expert
|
||||
Performance," *Psychological Review* 100(3), 1993.
|
||||
7. Chi, Bassok, Lewis, Reimann & Glaser, "Self-Explanations: How Students Study and Use Examples
|
||||
in Learning to Solve Problems," *Cognitive Science* 13(2), 1989.
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
# Why This Is a Companion, Not a Compiled Copy
|
||||
|
||||
This repository ships `engineering/book-to-skill`, which compiles a document on disk into a
|
||||
knowledge-base skill. This skill was **not** produced that way, and the reason is worth stating
|
||||
plainly, because the same reasoning applies to every future request of the form "turn this book
|
||||
into a skill."
|
||||
|
||||
---
|
||||
|
||||
## The three facts that decided it
|
||||
|
||||
1. **The book is copyrighted.** *Deep Learning* (Goodfellow, Bengio & Courville) is published by
|
||||
MIT Press, 2016. Free-to-read is not free-to-redistribute; the two are unrelated.
|
||||
2. **The publisher's position is explicit.** deeplearningbook.org states that the HTML-only
|
||||
presentation exists as a deliberate friction against copying and editing, required by the
|
||||
authors' contract with MIT Press, and that PDF distribution is not permitted.
|
||||
3. **This repository is public.** `book-to-skill`'s own rights gate refuses a `shareable`
|
||||
distribution without one of `public-domain`, `open-license`, `internal-docs`, or
|
||||
`author-permission`. None applies here. Its `references/rights_and_provenance.md` lists
|
||||
"publish a compiled skill of a copyrighted book to a public marketplace" under **Do not**.
|
||||
|
||||
`book-to-skill` also has a hard rule that it converts files already on disk and never scrapes a
|
||||
book from the web — so the requested pipeline could not have been run against a URL regardless.
|
||||
|
||||
## What was built instead
|
||||
|
||||
An **original companion**: every chapter file is written from domain knowledge, states what the
|
||||
chapter establishes, how to use it, and where it has aged, and links to the official free
|
||||
chapter. The organizing structure — 20 chapters in 3 parts — is the book's published table of
|
||||
contents, which is factual metadata, not expression.
|
||||
|
||||
Concretely, this skill contains:
|
||||
|
||||
- **No passages, paragraphs, sentences, or figures from the book.**
|
||||
- **No paraphrase-per-paragraph** — the compression ratio is roughly a whole chapter to a page,
|
||||
which is a synthesis, not a substitute.
|
||||
- **Links to the official chapters**, so the skill sends readers *to* the book rather than
|
||||
replacing it.
|
||||
- **Original material the book does not contain**: the 2016→2026 delta layer, the prerequisite
|
||||
graph, the study method, and four executable tools.
|
||||
|
||||
Under the idea/expression line (17 U.S.C. §102(b); *Baker v. Selden*), what this skill carries —
|
||||
the names of methods, the structure of an argument, decision rules stated plainly, and terms
|
||||
defined in other words — sits on the ideas side. The chapters themselves, in the authors' prose,
|
||||
sit on the expression side, and stay at deeplearningbook.org.
|
||||
|
||||
**Not legal advice.** This is the posture and its reasoning. Where money or publication is
|
||||
involved, ask a lawyer.
|
||||
|
||||
## The rule this establishes for the repository
|
||||
|
||||
> When a user asks to convert a copyrighted work into a shareable skill, build a **companion**
|
||||
> that indexes and updates the source, not a **compilation** that reproduces it. Compile only
|
||||
> when the rights gate clears — and keep compiled output local when it does not.
|
||||
|
||||
A companion is often the better artifact anyway. A compilation freezes a source at its
|
||||
publication date; a companion can say which parts of a ten-year-old text are still true, which
|
||||
is precisely what a reader of a 2016 deep learning book needs most.
|
||||
|
||||
## Use this skill correctly
|
||||
|
||||
- **Do** read the book at deeplearningbook.org; this navigates it.
|
||||
- **Do** use the delta reference before acting on any 2016-era recommendation.
|
||||
- **Don't** treat the chapter files as a substitute for reading the chapters — they are the
|
||||
answer key for retrieval practice, not the material.
|
||||
- **Don't** extend this skill by pasting book text into it. That converts a companion into the
|
||||
thing this file exists to avoid.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Goodfellow, Bengio & Courville, *Deep Learning*, MIT Press, 2016 — https://www.deeplearningbook.org/
|
||||
2. deeplearningbook.org — the site's own statement on its HTML-only format and PDF distribution.
|
||||
3. 17 U.S.C. §102(b) — ideas, procedures, processes and methods of operation are outside
|
||||
copyright.
|
||||
4. *Baker v. Selden*, 101 U.S. 99 (1879) — the idea/expression dichotomy.
|
||||
5. 17 U.S.C. §107 — fair use as a four-factor defence, assessed case by case.
|
||||
6. *Authors Guild v. Google, Inc.*, 804 F.3d 202 (2d Cir. 2015) — indexing that does not
|
||||
substitute for the original held transformative.
|
||||
7. This repository: `engineering/book-to-skill/skills/book-to-skill/references/rights_and_provenance.md`
|
||||
(the rights gate) and its SKILL.md hard rules 1 and 6.
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
# Studying a Hard Technical Text
|
||||
|
||||
This book defeats more readers than it teaches, and the reason is rarely ability. It is a
|
||||
reference text with textbook page counts, high element interactivity, and no exercises. The
|
||||
methods below are the evidence-based ones that apply specifically to that shape of material.
|
||||
|
||||
---
|
||||
|
||||
## The five that matter here
|
||||
|
||||
### 1. Retrieval practice beats re-reading
|
||||
|
||||
Testing yourself on material produces markedly better long-term retention than reviewing it,
|
||||
even when re-reading feels more productive at the time (Roediger & Karpicke, 2006). For this
|
||||
book, the cheap version: after each chapter, close it and write the chapter's core idea, three
|
||||
concepts, and one thing you could not explain. The chapter files in this skill are structured to
|
||||
be *checked against*, not read first — use them as the answer key.
|
||||
|
||||
### 2. Spacing beats massing
|
||||
|
||||
Distributed practice outperforms cramming for durable retention (Cepeda et al., 2006). Practical
|
||||
rule: revisit ch05 and ch11 at one week and one month. They are the chapters whose vocabulary
|
||||
every later chapter reuses, so spaced review of those two pays across the whole book.
|
||||
|
||||
### 3. Worked examples before problem-solving, for novices
|
||||
|
||||
For learners without domain schemas, studying worked examples is more effective than solving
|
||||
equivalent problems — the expertise-reversal effect means this flips once you *have* the schemas
|
||||
(Sweller & Cooper, 1985; Kalyuga et al., 2003). Consequence for this book: in Ch 8 and Ch 18, work
|
||||
the derivations the text presents before attempting your own; in Ch 11, do the opposite — you
|
||||
already have the schema, so run the workflow on a real project.
|
||||
|
||||
### 4. Self-explanation
|
||||
|
||||
Students who explain each step to themselves while studying examples learn substantially more than
|
||||
those who do not (Chi et al., 1989). The highest-yield prompt for this book: after every
|
||||
equation, answer "what would change if this term were removed?" It converts symbol-tracking into
|
||||
understanding, and it is exactly how the book's own arguments are structured.
|
||||
|
||||
### 5. Desirable difficulties
|
||||
|
||||
Conditions that slow acquisition often improve retention and transfer (Bjork & Bjork, 2011).
|
||||
Interleaving Part II chapters with attempts to apply them beats reading Part II straight through,
|
||||
even though it feels slower. This is the empirical case for the reading-path planner's
|
||||
"read one chapter, then implement something" cadence.
|
||||
|
||||
---
|
||||
|
||||
## Managing cognitive load in Chapters 2–4 and 16–20
|
||||
|
||||
High **element interactivity** — where symbols cannot be understood one at a time because they
|
||||
only make sense in relation to each other — is what makes these chapters expensive (Sweller, van
|
||||
Merriënboer & Paas, 1998). Intrinsic load cannot be reduced by better presentation; it can only be
|
||||
sequenced. Two consequences:
|
||||
|
||||
- **Do not skim a high-interactivity derivation.** Skimming works for prose and fails for
|
||||
Chapter 18. Either work it or defer it explicitly.
|
||||
- **Chunk by named object.** "Score matching" as one unit is cheaper than seven equations. Every
|
||||
chapter file in this skill names its objects for exactly this reason.
|
||||
|
||||
## A cadence that works for this book
|
||||
|
||||
1. **Skim** the chapter's section headings and the summary. Five minutes.
|
||||
2. **Read** with a pen, working every derivation you intend to keep.
|
||||
3. **Close the book** and write, from memory: the core idea, three concepts, one confusion.
|
||||
4. **Check** against this skill's chapter file — it is deliberately written as a synthesis, so a
|
||||
mismatch is informative.
|
||||
5. **Apply** something small before moving on (a five-line implementation, a diagnosis of a real
|
||||
training run, a re-derivation).
|
||||
6. **Revisit** at one week for ch05, ch08, ch11, ch18.
|
||||
|
||||
Step 5 is the one most readers skip and the one that decides whether any of it survives.
|
||||
|
||||
## What not to do
|
||||
|
||||
- **Reading front-to-back through Part I as a first pass.** See `prerequisite_map.md`.
|
||||
- **Highlighting.** It produces a strong feeling of learning and among the weakest measured
|
||||
effects of the common study techniques (Dunlosky et al., 2013).
|
||||
- **Taking notes that restate the text.** Restatement is transcription. Write the answer to
|
||||
"when would I use this?" instead — that is the note that gets read again.
|
||||
- **Waiting to "finish the math" before touching a model.** The book's own Ch 11 argues against
|
||||
it: build the end-to-end pipeline early.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Roediger & Karpicke, "Test-Enhanced Learning: Taking Memory Tests Improves Long-Term
|
||||
Retention," *Psychological Science* 17(3), 2006.
|
||||
2. Cepeda, Pashler, Vul, Wixted & Rohrer, "Distributed Practice in Verbal Recall Tasks: A Review
|
||||
and Quantitative Synthesis," *Psychological Bulletin* 132(3), 2006.
|
||||
3. Sweller & Cooper, "The Use of Worked Examples as a Substitute for Problem Solving in Learning
|
||||
Algebra," *Cognition and Instruction* 2(1), 1985; Kalyuga, Ayres, Chandler & Sweller, "The
|
||||
Expertise Reversal Effect," *Educational Psychologist* 38(1), 2003.
|
||||
4. Chi, Bassok, Lewis, Reimann & Glaser, "Self-Explanations," *Cognitive Science* 13(2), 1989.
|
||||
5. Bjork & Bjork, "Making Things Hard on Yourself, But in a Good Way," in *Psychology and the Real
|
||||
World*, 2011.
|
||||
6. Sweller, van Merriënboer & Paas, "Cognitive Architecture and Instructional Design,"
|
||||
*Educational Psychology Review* 10(3), 1998.
|
||||
7. Dunlosky, Rawson, Marsh, Nathan & Willingham, "Improving Students' Learning With Effective
|
||||
Learning Techniques," *Psychological Science in the Public Interest* 14(1), 2013.
|
||||
|
|
@ -0,0 +1,261 @@
|
|||
#!/usr/bin/env python3
|
||||
"""capacity_planner.py — capacity, the generalization gap, and what to spend next.
|
||||
|
||||
Chapter 5 gives the bias-variance budget and the classical U-shaped capacity curve.
|
||||
Chapter 7 gives the regularizers you spend from that budget. This tool combines them
|
||||
into one recommendation: given a measured gap and a parameters-per-example ratio, what
|
||||
is the cheapest next purchase?
|
||||
|
||||
It carries one correction the 2016 text predates. The classical advice "shrink the
|
||||
model when it overfits" assumed the underparameterized side of the U-curve. Past the
|
||||
interpolation threshold, test error can fall again (double descent; Belkin et al. 2019,
|
||||
Nakkiran et al. 2020), so in the overparameterized regime this tool ranks *more data,
|
||||
more regularization, longer training* above *smaller model* and says why.
|
||||
|
||||
Standard library only. No network calls.
|
||||
|
||||
Exit codes:
|
||||
0 a plan was produced and the fit looks balanced
|
||||
1 a plan was produced and an action is recommended
|
||||
2 usage error (argparse)
|
||||
4 inputs are inconsistent (e.g. validation error below training error by a wide
|
||||
margin, which usually means a leaky split rather than a good model)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
|
||||
# Regularization ladder in cost order (ch07). Cost is effort+risk, not compute.
|
||||
LADDER = [
|
||||
("more-real-data", "Collect more real labelled data",
|
||||
"The only intervention that reduces variance without adding bias. Everything below "
|
||||
"is a substitute for it.", "ch05, ch07"),
|
||||
("augmentation", "Label-preserving data augmentation",
|
||||
"Cheapest substitute for real data where a real invariance exists. Verify the "
|
||||
"transform preserves the label — a flip destroys the label on digits.", "ch07"),
|
||||
("early-stopping", "Early stopping on a validation metric",
|
||||
"Approximately equivalent to L2 under a quadratic approximation, at near-zero cost. "
|
||||
"Usually the first thing to try.", "ch07"),
|
||||
("weight-decay", "Decoupled weight decay (AdamW)",
|
||||
"Damps directions the data does not constrain. Use AdamW, not Adam with an L2 term "
|
||||
"in the loss — they are not equivalent under an adaptive optimizer.", "ch07, ch08"),
|
||||
("label-smoothing", "Label smoothing",
|
||||
"Reduces over-confidence on the training targets; cheap and usually harmless.",
|
||||
"ch07 (noise injection on labels)"),
|
||||
("dropout", "Dropout",
|
||||
"Approximate ensembling. Note the interaction with batch norm, and that modern "
|
||||
"large stacks regularize far more lightly than 2016 practice.", "ch07"),
|
||||
("parameter-sharing", "Structural parameter sharing",
|
||||
"The strongest regularizer: remove parameters rather than penalize them. Only valid "
|
||||
"when a real invariance exists (convolution, weight tying).", "ch07, ch09"),
|
||||
("smaller-model", "Reduce model capacity",
|
||||
"Classical advice from the underparameterized side of the U-curve. In the "
|
||||
"overparameterized regime this is ranked last, not first.", "ch05"),
|
||||
]
|
||||
|
||||
# Parameters-per-example ratio bands. Approximate regime markers, not a threshold
|
||||
# theorem — the interpolation threshold depends on the task, the architecture and the
|
||||
# label noise, and cannot be read off a parameter count alone.
|
||||
UNDERPARAMETERIZED_MAX = 1.0
|
||||
OVERPARAMETERIZED_MIN = 10.0
|
||||
|
||||
|
||||
def classify_regime(params: int, examples: int) -> tuple[str, float, str]:
|
||||
ratio = params / examples
|
||||
if ratio < UNDERPARAMETERIZED_MAX:
|
||||
regime = "underparameterized"
|
||||
note = ("Fewer parameters than training examples. The classical U-curve applies "
|
||||
"directly here: reducing capacity is a legitimate response to overfitting.")
|
||||
elif ratio < OVERPARAMETERIZED_MIN:
|
||||
regime = "near-interpolation"
|
||||
note = ("Roughly at the interpolation threshold, where the classical curve peaks "
|
||||
"and double descent begins. This is the worst place to sit: both more "
|
||||
"capacity and less capacity can improve test error, so measure rather "
|
||||
"than reason.")
|
||||
else:
|
||||
regime = "overparameterized"
|
||||
note = ("Many more parameters than examples. The classical 'shrink the model' "
|
||||
"advice is not reliable here — double descent means a larger model with "
|
||||
"more data or stronger regularization often generalizes better.")
|
||||
return regime, ratio, note
|
||||
|
||||
|
||||
def plan(params: int, examples: int, train_error: float, val_error: float,
|
||||
applied: set[str], target_error: float | None,
|
||||
overfit_rel_gap: float) -> dict:
|
||||
regime, ratio, regime_note = classify_regime(params, examples)
|
||||
denom = max(abs(train_error), 1e-6)
|
||||
gap = val_error - train_error
|
||||
rel_gap = gap / denom
|
||||
|
||||
if target_error is not None and train_error > target_error * 1.15:
|
||||
verdict = "UNDERFIT"
|
||||
headline = ("Training error is above target — capacity or optimization is the "
|
||||
"bottleneck. More data cannot help yet.")
|
||||
actions = [
|
||||
("tune-lr-schedule", "Tune learning rate and schedule first",
|
||||
"Usually dominates the choice of optimizer family. Warmup plus cosine decay "
|
||||
"is the modern default for deep stacks.", "ch08"),
|
||||
("check-init", "Check initialization scale (He / Xavier)",
|
||||
"Bad initialization is a common silent cause of a model that will not fit.",
|
||||
"ch08"),
|
||||
("remove-regularization", "Remove regularization you already added",
|
||||
"Every regularizer you applied is buying variance reduction you cannot "
|
||||
"currently afford.", "ch07"),
|
||||
("add-capacity", "Add capacity (width or depth)",
|
||||
"Only after the three above — an optimization problem does not respond to "
|
||||
"more parameters.", "ch05, ch06"),
|
||||
]
|
||||
elif rel_gap > overfit_rel_gap:
|
||||
verdict = "OVERFIT"
|
||||
headline = (f"Validation error exceeds training error by {rel_gap:.0%} of the "
|
||||
"training error — spend from the regularization budget.")
|
||||
ladder = [item for item in LADDER if item[0] not in applied]
|
||||
if regime == "underparameterized":
|
||||
actions = ladder
|
||||
else:
|
||||
# Push smaller-model to the end and say why.
|
||||
actions = ([item for item in ladder if item[0] != "smaller-model"]
|
||||
+ [item for item in ladder if item[0] == "smaller-model"])
|
||||
else:
|
||||
verdict = "BALANCED"
|
||||
headline = ("The gap is within tolerance. Neither more capacity nor more "
|
||||
"regularization is indicated by these numbers.")
|
||||
actions = [
|
||||
("verify-metric", "Verify the metric is the one you care about",
|
||||
"A balanced fit on the wrong proxy is still the wrong model.", "ch11"),
|
||||
("check-splits", "Confirm the split is clean and the result holds across seeds",
|
||||
"A small dataset with one seed is not a measurement.", "ch05, ch11"),
|
||||
]
|
||||
|
||||
return {
|
||||
"verdict": verdict,
|
||||
"headline": headline,
|
||||
"regime": regime,
|
||||
"regime_note": regime_note,
|
||||
"params": params,
|
||||
"train_examples": examples,
|
||||
"params_per_example": round(ratio, 3),
|
||||
"train_error": train_error,
|
||||
"val_error": val_error,
|
||||
"gap": round(gap, 6),
|
||||
"relative_gap": round(rel_gap, 4),
|
||||
"target_error": target_error,
|
||||
"already_applied": sorted(applied),
|
||||
"actions": [
|
||||
{"id": a[0], "action": a[1], "why": a[2], "chapter": a[3]}
|
||||
for a in actions
|
||||
],
|
||||
"double_descent_caveat": (
|
||||
regime != "underparameterized" and verdict == "OVERFIT"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def render(result: dict) -> str:
|
||||
lines = [
|
||||
"CAPACITY & REGULARIZATION PLAN",
|
||||
"=" * 70,
|
||||
f"Verdict : {result['verdict']}",
|
||||
f" {result['headline']}",
|
||||
"",
|
||||
f"Regime : {result['regime']} "
|
||||
f"({result['params_per_example']} params per training example)",
|
||||
f" {result['regime_note']}",
|
||||
"",
|
||||
f"Train err : {result['train_error']} Val err: {result['val_error']} "
|
||||
f"gap: {result['gap']} ({result['relative_gap']:.0%} of train error)",
|
||||
]
|
||||
if result["already_applied"]:
|
||||
lines.append(f"Applied : {', '.join(result['already_applied'])} (excluded below)")
|
||||
lines += ["", "Do these in order:", "-" * 70]
|
||||
for index, action in enumerate(result["actions"], start=1):
|
||||
lines.append(f"{index}. {action['action']} [{action['chapter']}]")
|
||||
lines.append(f" {action['why']}")
|
||||
lines.append("-" * 70)
|
||||
if result["double_descent_caveat"]:
|
||||
lines.append(
|
||||
"Double-descent caveat: 'reduce capacity' is ranked LAST here because this "
|
||||
"model is at or past the interpolation threshold, where the classical "
|
||||
"U-curve advice is unreliable (Belkin 2019, Nakkiran 2020 — both post-date "
|
||||
"the book). See references/book_to_2026_delta.md."
|
||||
)
|
||||
lines.append("Change one thing per experiment and re-measure the gap. (ch11)")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Turn a measured generalization gap into an ordered action plan.",
|
||||
epilog="Exit codes: 0 balanced · 1 action recommended · 4 inconsistent inputs.",
|
||||
)
|
||||
parser.add_argument("--params", type=int, help="trainable parameter count")
|
||||
parser.add_argument("--train-examples", type=int, help="number of training examples")
|
||||
parser.add_argument("--train-error", type=float,
|
||||
help="training error or loss (same units as --val-error)")
|
||||
parser.add_argument("--val-error", type=float, help="validation error or loss")
|
||||
parser.add_argument("--target-error", type=float,
|
||||
help="the value that would count as success; without it, "
|
||||
"underfitting cannot be distinguished from convergence")
|
||||
parser.add_argument("--applied", default="",
|
||||
help="comma-separated regularizers already in use, from: "
|
||||
+ ",".join(item[0] for item in LADDER))
|
||||
parser.add_argument("--overfit-rel-gap", type=float, default=0.30,
|
||||
help="relative gap above which the fit counts as overfitting "
|
||||
"(default: 0.30)")
|
||||
parser.add_argument("--output", choices=("text", "json"), default="text")
|
||||
parser.add_argument("--sample", action="store_true",
|
||||
help="run against a built-in overparameterized example")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
args.params, args.train_examples = 12_000_000, 50_000
|
||||
args.train_error, args.val_error = 0.01, 0.22
|
||||
args.applied = "early-stopping"
|
||||
|
||||
required = (args.params, args.train_examples, args.train_error, args.val_error)
|
||||
if any(value is None for value in required):
|
||||
parser.error("--params, --train-examples, --train-error and --val-error are all "
|
||||
"required (or use --sample)")
|
||||
if args.params <= 0 or args.train_examples <= 0:
|
||||
parser.error("--params and --train-examples must be positive")
|
||||
|
||||
known = {item[0] for item in LADDER}
|
||||
applied = {token.strip() for token in args.applied.split(",") if token.strip()}
|
||||
unknown = applied - known
|
||||
if unknown:
|
||||
parser.error(f"unknown --applied value(s): {', '.join(sorted(unknown))}; "
|
||||
f"choose from {', '.join(sorted(known))}")
|
||||
|
||||
if args.val_error < args.train_error - 0.05 * max(abs(args.train_error), 1e-6):
|
||||
payload = {
|
||||
"status": "inconsistent_input",
|
||||
"reason": "validation error is materially below training error",
|
||||
"explanation": "This usually means a leaky split, a validation set that is "
|
||||
"easier than the training set, or regularization active at "
|
||||
"train time but not at eval (dropout, augmentation). Fix the "
|
||||
"measurement before acting on it.",
|
||||
"chapter": "ch05, ch11",
|
||||
}
|
||||
if args.output == "json":
|
||||
print(json.dumps(payload, indent=2))
|
||||
else:
|
||||
print("INCONSISTENT INPUT — " + payload["reason"])
|
||||
print(payload["explanation"])
|
||||
return 4
|
||||
|
||||
result = plan(args.params, args.train_examples, args.train_error, args.val_error,
|
||||
applied, args.target_error, args.overfit_rel_gap)
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(render(result))
|
||||
return 0 if result["verdict"] == "BALANCED" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,352 @@
|
|||
#!/usr/bin/env python3
|
||||
"""model_arithmetic.py — parameters, FLOPs and activation memory for a layer stack.
|
||||
|
||||
Chapter 6 makes the point that the real cost of depth in training is activation memory:
|
||||
the backward pass needs the forward activations, which is why gradient checkpointing
|
||||
trades compute for memory. Chapter 9 makes the receptive-field and parameter-sharing
|
||||
arithmetic explicit. This tool does that arithmetic for a declared stack, and — the part
|
||||
that catches real bugs — refuses to continue when one layer's output shape cannot feed
|
||||
the next.
|
||||
|
||||
Shapes are per example, excluding the batch dimension. FLOPs are per example, per
|
||||
forward pass; a training step costs roughly 3x a forward pass (forward + backward).
|
||||
|
||||
Layer types: input, linear, conv2d, pool2d, flatten, embedding, layernorm, activation,
|
||||
dropout, mha (multi-head self-attention), lstm, gru.
|
||||
|
||||
Standard library only. No frameworks, no network calls.
|
||||
|
||||
Exit codes:
|
||||
0 the stack is consistent and the report was produced
|
||||
2 usage error (argparse)
|
||||
4 the spec could not be parsed (bad JSON, unknown layer type, missing field)
|
||||
5 shape mismatch between consecutive layers — the offending layer is named
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
|
||||
BYTES_PER_ELEMENT = {"fp32": 4, "tf32": 4, "fp16": 2, "bf16": 2, "fp8": 1}
|
||||
|
||||
# A multiply-accumulate is counted as 2 FLOPs. --convention mac reports MACs instead.
|
||||
FLOPS_PER_MAC = 2
|
||||
|
||||
|
||||
class SpecError(ValueError):
|
||||
"""The spec is malformed: bad JSON, unknown layer, or a missing field."""
|
||||
|
||||
|
||||
class ShapeError(ValueError):
|
||||
"""A layer cannot consume the shape the previous layer produced."""
|
||||
|
||||
|
||||
def _require(layer: dict, key: str, index: int):
|
||||
if key not in layer:
|
||||
raise SpecError(f"layer {index} ({layer.get('type', '?')}) is missing '{key}'")
|
||||
return layer[key]
|
||||
|
||||
|
||||
def _prod(shape: tuple[int, ...]) -> int:
|
||||
total = 1
|
||||
for dim in shape:
|
||||
total *= dim
|
||||
return total
|
||||
|
||||
|
||||
def step(layer: dict, shape: tuple[int, ...], index: int) -> tuple[tuple[int, ...], int, int]:
|
||||
"""Return (output_shape, parameters, macs) for one layer given its input shape."""
|
||||
kind = _require(layer, "type", index)
|
||||
|
||||
if kind == "input":
|
||||
return tuple(_require(layer, "shape", index)), 0, 0
|
||||
|
||||
if kind == "linear":
|
||||
units = int(_require(layer, "units", index))
|
||||
if len(shape) != 1:
|
||||
raise ShapeError(
|
||||
f"layer {index} (linear) needs a 1-D input, got {shape}. "
|
||||
"Insert a flatten layer, or use a per-token linear on a 2-D sequence "
|
||||
"by declaring the shape as [features]."
|
||||
)
|
||||
bias = bool(layer.get("bias", True))
|
||||
params = shape[0] * units + (units if bias else 0)
|
||||
return (units,), params, shape[0] * units
|
||||
|
||||
if kind == "conv2d":
|
||||
filters = int(_require(layer, "filters", index))
|
||||
kernel = int(_require(layer, "kernel", index))
|
||||
stride = int(layer.get("stride", 1))
|
||||
padding = layer.get("padding", "same")
|
||||
if len(shape) != 3:
|
||||
raise ShapeError(
|
||||
f"layer {index} (conv2d) needs a 3-D input (C, H, W), got {shape}"
|
||||
)
|
||||
channels, height, width = shape
|
||||
if padding == "same":
|
||||
out_h, out_w = math.ceil(height / stride), math.ceil(width / stride)
|
||||
elif padding == "valid":
|
||||
out_h = (height - kernel) // stride + 1
|
||||
out_w = (width - kernel) // stride + 1
|
||||
else:
|
||||
raise SpecError(f"layer {index} (conv2d): padding must be 'same' or 'valid'")
|
||||
if out_h <= 0 or out_w <= 0:
|
||||
raise ShapeError(
|
||||
f"layer {index} (conv2d) with kernel {kernel} and stride {stride} "
|
||||
f"reduces {height}x{width} to {out_h}x{out_w} — the kernel is larger "
|
||||
"than the feature map."
|
||||
)
|
||||
groups = int(layer.get("groups", 1))
|
||||
if channels % groups or filters % groups:
|
||||
raise SpecError(
|
||||
f"layer {index} (conv2d): groups={groups} does not divide "
|
||||
f"in_channels={channels} and filters={filters}"
|
||||
)
|
||||
bias = bool(layer.get("bias", True))
|
||||
params = (kernel * kernel * (channels // groups) * filters) + (filters if bias else 0)
|
||||
macs = kernel * kernel * (channels // groups) * filters * out_h * out_w
|
||||
return (filters, out_h, out_w), params, macs
|
||||
|
||||
if kind == "pool2d":
|
||||
size = int(layer.get("size", 2))
|
||||
stride = int(layer.get("stride", size))
|
||||
if len(shape) != 3:
|
||||
raise ShapeError(f"layer {index} (pool2d) needs a 3-D input, got {shape}")
|
||||
channels, height, width = shape
|
||||
out_h = (height - size) // stride + 1
|
||||
out_w = (width - size) // stride + 1
|
||||
if out_h <= 0 or out_w <= 0:
|
||||
raise ShapeError(
|
||||
f"layer {index} (pool2d) reduces {height}x{width} to {out_h}x{out_w}"
|
||||
)
|
||||
return (channels, out_h, out_w), 0, 0
|
||||
|
||||
if kind == "flatten":
|
||||
return (_prod(shape),), 0, 0
|
||||
|
||||
if kind == "embedding":
|
||||
vocab = int(_require(layer, "vocab", index))
|
||||
dim = int(_require(layer, "dim", index))
|
||||
seq = int(layer.get("seq_len", shape[0] if shape else 1))
|
||||
return (seq, dim), vocab * dim, 0 # a lookup, not a matmul
|
||||
|
||||
if kind == "layernorm":
|
||||
features = shape[-1]
|
||||
return shape, 2 * features, 0
|
||||
|
||||
if kind in ("activation", "dropout"):
|
||||
return shape, 0, 0
|
||||
|
||||
if kind == "mha":
|
||||
# Multi-head self-attention over a (seq, d_model) input.
|
||||
if len(shape) != 2:
|
||||
raise ShapeError(
|
||||
f"layer {index} (mha) needs a 2-D input (seq_len, d_model), got {shape}"
|
||||
)
|
||||
seq, d_model = shape
|
||||
heads = int(layer.get("heads", 8))
|
||||
if d_model % heads:
|
||||
raise SpecError(
|
||||
f"layer {index} (mha): d_model={d_model} is not divisible by heads={heads}"
|
||||
)
|
||||
# 4 projections (Q, K, V, O), each d_model x d_model.
|
||||
params = 4 * d_model * d_model + (4 * d_model if layer.get("bias", True) else 0)
|
||||
proj_macs = 4 * seq * d_model * d_model
|
||||
# Scores (seq x seq x d_model) and the weighted value sum, both quadratic in seq.
|
||||
attn_macs = 2 * seq * seq * d_model
|
||||
return shape, params, proj_macs + attn_macs
|
||||
|
||||
if kind in ("lstm", "gru"):
|
||||
if len(shape) != 2:
|
||||
raise ShapeError(
|
||||
f"layer {index} ({kind}) needs a 2-D input (seq_len, features), got {shape}"
|
||||
)
|
||||
seq, features = shape
|
||||
units = int(_require(layer, "units", index))
|
||||
gates = 4 if kind == "lstm" else 3
|
||||
params = gates * (features * units + units * units + 2 * units)
|
||||
macs = seq * gates * (features * units + units * units)
|
||||
out = (seq, units) if layer.get("return_sequences", True) else (units,)
|
||||
return out, params, macs
|
||||
|
||||
raise SpecError(f"layer {index}: unknown layer type {kind!r}")
|
||||
|
||||
|
||||
def analyse(spec: dict, dtype: str, convention: str) -> dict:
|
||||
layers = spec.get("layers")
|
||||
if not isinstance(layers, list) or not layers:
|
||||
raise SpecError("spec must contain a non-empty 'layers' list")
|
||||
if layers[0].get("type") != "input":
|
||||
raise SpecError("the first layer must be of type 'input'")
|
||||
|
||||
width = BYTES_PER_ELEMENT[dtype]
|
||||
shape: tuple[int, ...] = ()
|
||||
rows = []
|
||||
total_params = 0
|
||||
total_macs = 0
|
||||
total_activations = 0
|
||||
|
||||
for index, layer in enumerate(layers):
|
||||
shape, params, macs = step(layer, shape, index)
|
||||
activations = _prod(shape)
|
||||
total_params += params
|
||||
total_macs += macs
|
||||
# The input layer's tensor is not a stored intermediate activation.
|
||||
if index > 0:
|
||||
total_activations += activations
|
||||
rows.append({
|
||||
"index": index,
|
||||
"type": layer["type"],
|
||||
"name": layer.get("name", layer["type"]),
|
||||
"output_shape": list(shape),
|
||||
"parameters": params,
|
||||
"macs": macs,
|
||||
"flops": macs * FLOPS_PER_MAC,
|
||||
"activation_elements": activations,
|
||||
})
|
||||
|
||||
compute = total_macs if convention == "mac" else total_macs * FLOPS_PER_MAC
|
||||
return {
|
||||
"dtype": dtype,
|
||||
"convention": convention,
|
||||
"layers": rows,
|
||||
"totals": {
|
||||
"parameters": total_params,
|
||||
"parameter_bytes": total_params * width,
|
||||
"forward_macs": total_macs,
|
||||
"forward_flops": total_macs * FLOPS_PER_MAC,
|
||||
"reported_compute": compute,
|
||||
"activation_elements_per_example": total_activations,
|
||||
"activation_bytes_per_example": total_activations * width,
|
||||
},
|
||||
"notes": [
|
||||
"Shapes and costs are per example; the batch dimension is excluded.",
|
||||
"A training step costs roughly 3x the forward FLOPs (forward + backward).",
|
||||
"Activation memory is the training-time cost of depth (ch06). Gradient "
|
||||
"checkpointing trades compute for it.",
|
||||
"Optimizer state is extra: Adam/AdamW holds two moments per parameter, so "
|
||||
"budget ~3x parameter bytes for weights plus state in fp32.",
|
||||
"Attention cost is quadratic in sequence length — visible in the mha row.",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
SAMPLE_SPEC = {
|
||||
"name": "small convnet, CIFAR-shaped",
|
||||
"layers": [
|
||||
{"type": "input", "shape": [3, 32, 32]},
|
||||
{"type": "conv2d", "filters": 32, "kernel": 3, "padding": "same"},
|
||||
{"type": "activation", "name": "relu"},
|
||||
{"type": "pool2d", "size": 2},
|
||||
{"type": "conv2d", "filters": 64, "kernel": 3, "padding": "same"},
|
||||
{"type": "activation", "name": "relu"},
|
||||
{"type": "pool2d", "size": 2},
|
||||
{"type": "flatten"},
|
||||
{"type": "linear", "units": 128},
|
||||
{"type": "activation", "name": "relu"},
|
||||
{"type": "dropout"},
|
||||
{"type": "linear", "units": 10},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def human(value: int) -> str:
|
||||
for unit, scale in (("G", 1e9), ("M", 1e6), ("K", 1e3)):
|
||||
if value >= scale:
|
||||
return f"{value / scale:.2f}{unit}"
|
||||
return str(value)
|
||||
|
||||
|
||||
def render(result: dict) -> str:
|
||||
lines = [
|
||||
"MODEL ARITHMETIC",
|
||||
"=" * 78,
|
||||
f"{'#':>2} {'layer':<12} {'output shape':<18} {'params':>12} "
|
||||
f"{'MFLOPs':>10} {'act.elts':>10}",
|
||||
"-" * 78,
|
||||
]
|
||||
for row in result["layers"]:
|
||||
shape = "x".join(str(dim) for dim in row["output_shape"])
|
||||
lines.append(
|
||||
f"{row['index']:>2} {row['type']:<12} {shape:<18} "
|
||||
f"{row['parameters']:>12,} {row['flops'] / 1e6:>10.2f} "
|
||||
f"{row['activation_elements']:>10,}"
|
||||
)
|
||||
totals = result["totals"]
|
||||
lines += [
|
||||
"-" * 78,
|
||||
f"Parameters : {totals['parameters']:,} "
|
||||
f"({human(totals['parameter_bytes'])}B at {result['dtype']})",
|
||||
f"Forward FLOPs/ex : {human(totals['forward_flops'])} "
|
||||
f"({human(totals['forward_macs'])} MACs)",
|
||||
f"Training step (~3x) : {human(totals['forward_flops'] * 3)} FLOPs/ex",
|
||||
f"Activations/ex : {totals['activation_elements_per_example']:,} elements "
|
||||
f"({human(totals['activation_bytes_per_example'])}B at {result['dtype']})",
|
||||
"",
|
||||
]
|
||||
lines.append(f"At batch 128: activations ≈ "
|
||||
f"{human(totals['activation_bytes_per_example'] * 128)}B")
|
||||
lines.append("")
|
||||
for note in result["notes"]:
|
||||
lines.append(f"· {note}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compute parameters, FLOPs and activation memory for a layer stack.",
|
||||
epilog="Exit codes: 0 ok · 4 bad spec · 5 shape mismatch.",
|
||||
)
|
||||
parser.add_argument("--spec", help="path to a JSON layer spec")
|
||||
parser.add_argument("--spec-sample", action="store_true",
|
||||
help="print the built-in sample spec and exit")
|
||||
parser.add_argument("--dtype", choices=sorted(BYTES_PER_ELEMENT), default="fp32")
|
||||
parser.add_argument("--convention", choices=("flop", "mac"), default="flop",
|
||||
help="report FLOPs (1 MAC = 2 FLOPs) or raw MACs")
|
||||
parser.add_argument("--output", choices=("text", "json"), default="text")
|
||||
parser.add_argument("--sample", action="store_true",
|
||||
help="analyse the built-in sample spec")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.spec_sample:
|
||||
print(json.dumps(SAMPLE_SPEC, indent=2))
|
||||
return 0
|
||||
|
||||
if args.sample:
|
||||
spec = SAMPLE_SPEC
|
||||
elif args.spec:
|
||||
try:
|
||||
with open(args.spec, encoding="utf-8") as handle:
|
||||
spec = json.load(handle)
|
||||
except OSError as error:
|
||||
print(f"cannot read spec: {error}", file=sys.stderr)
|
||||
return 4
|
||||
except json.JSONDecodeError as error:
|
||||
print(f"spec is not valid JSON: {error}", file=sys.stderr)
|
||||
return 4
|
||||
else:
|
||||
parser.error("--spec is required (or use --sample / --spec-sample)")
|
||||
|
||||
try:
|
||||
result = analyse(spec, args.dtype, args.convention)
|
||||
except ShapeError as error:
|
||||
print(f"SHAPE MISMATCH — {error}", file=sys.stderr)
|
||||
print("A shape mismatch is a modelling error, not a typing error (ch02).",
|
||||
file=sys.stderr)
|
||||
return 5
|
||||
except SpecError as error:
|
||||
print(f"BAD SPEC — {error}", file=sys.stderr)
|
||||
return 4
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(render(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,419 @@
|
|||
#!/usr/bin/env python3
|
||||
"""reading_path_planner.py — turn a goal into an ordered, prerequisite-closed reading path.
|
||||
|
||||
The book's part order is not its dependency order. Read front-to-back and you spend
|
||||
weeks in Chapters 2-4 before touching a network; skip Part I entirely and Chapter 8
|
||||
becomes unreadable. This tool takes a goal and a background, resolves the *hard*
|
||||
prerequisites of the chapters that goal needs, orders them, and prices the result
|
||||
in weeks at your stated pace.
|
||||
|
||||
It refuses two things rather than guessing:
|
||||
* a goal whose subject the 2016 book does not cover at all (exit 3) — it names what
|
||||
covers it instead, rather than inventing the book's position on RLHF;
|
||||
* a goal too vague to route (exit 4) — it prints the questions that would fix it.
|
||||
|
||||
Deterministic keyword scoring. Standard library only. No network calls.
|
||||
|
||||
Exit codes:
|
||||
0 a path was produced
|
||||
2 usage error (argparse)
|
||||
3 goal is outside the book's scope
|
||||
4 goal could not be routed — forcing questions printed
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
|
||||
# --------------------------------------------------------------------------- data
|
||||
|
||||
# Hard prerequisites only: skipping one makes the target unreadable, not merely
|
||||
# harder. Soft prerequisites live in references/prerequisite_map.md and are
|
||||
# deliberately not enforced here.
|
||||
PREREQS: dict[int, tuple[int, ...]] = {
|
||||
1: (),
|
||||
2: (),
|
||||
3: (),
|
||||
4: (2,),
|
||||
5: (3,),
|
||||
6: (3,),
|
||||
7: (4, 5),
|
||||
8: (2, 4),
|
||||
9: (6,),
|
||||
10: (6,),
|
||||
11: (5,),
|
||||
12: (6,),
|
||||
13: (5,),
|
||||
14: (13,),
|
||||
15: (14,),
|
||||
16: (3,),
|
||||
17: (16,),
|
||||
18: (16, 17),
|
||||
19: (3,),
|
||||
20: (13, 19),
|
||||
}
|
||||
|
||||
TITLES: dict[int, str] = {
|
||||
1: "Introduction",
|
||||
2: "Linear Algebra",
|
||||
3: "Probability and Information Theory",
|
||||
4: "Numerical Computation",
|
||||
5: "Machine Learning Basics",
|
||||
6: "Deep Feedforward Networks",
|
||||
7: "Regularization for Deep Learning",
|
||||
8: "Optimization for Training Deep Models",
|
||||
9: "Convolutional Networks",
|
||||
10: "Sequence Modeling: Recurrent and Recursive Nets",
|
||||
11: "Practical Methodology",
|
||||
12: "Applications",
|
||||
13: "Linear Factor Models",
|
||||
14: "Autoencoders",
|
||||
15: "Representation Learning",
|
||||
16: "Structured Probabilistic Models for Deep Learning",
|
||||
17: "Monte Carlo Methods",
|
||||
18: "Confronting the Partition Function",
|
||||
19: "Approximate Inference",
|
||||
20: "Deep Generative Models",
|
||||
}
|
||||
|
||||
# Planning assumptions, not measurements. Part I and Part III chapters carry higher
|
||||
# derivation density per page, which is why they cost more than their length suggests.
|
||||
# Exposed in --output json so a reader can recalibrate against their own first chapter.
|
||||
BASE_HOURS: dict[int, float] = {
|
||||
1: 1.5, 2: 5.0, 3: 6.0, 4: 4.5, 5: 9.0,
|
||||
6: 6.0, 7: 6.0, 8: 8.0, 9: 6.0, 10: 7.0, 11: 4.0, 12: 5.0,
|
||||
13: 4.0, 14: 4.5, 15: 5.0, 16: 6.0, 17: 5.0, 18: 7.0, 19: 6.0, 20: 8.0,
|
||||
}
|
||||
|
||||
# Background multipliers on the hour estimate.
|
||||
BACKGROUNDS: dict[str, tuple[float, str]] = {
|
||||
"none": (1.6, "little linear algebra or probability — Part I is real work, not review"),
|
||||
"math": (0.8, "comfortable with linear algebra and probability, new to ML"),
|
||||
"applied": (1.0, "ships models, wants the theory underneath"),
|
||||
"research": (0.7, "reads papers in the field; the book is filling gaps"),
|
||||
}
|
||||
|
||||
# Destination lanes. Each is (label, target chapters, note). Scored by keyword hits.
|
||||
LANES: dict[str, dict] = {
|
||||
"practitioner": {
|
||||
"label": "Applied practitioner — train models that work",
|
||||
"targets": (5, 6, 7, 8, 11),
|
||||
"keywords": ("train", "training", "practical", "ship", "production", "apply",
|
||||
"applied", "practitioner", "debug", "tune", "tuning", "improve",
|
||||
"overfit", "underfit", "hyperparameter", "baseline", "workflow"),
|
||||
"note": "Chapters 5 and 11 first; they pay off before you finish Part I.",
|
||||
},
|
||||
"vision": {
|
||||
"label": "Computer vision — convolutional models",
|
||||
"targets": (6, 7, 8, 9, 12),
|
||||
"keywords": ("vision", "image", "convolution", "convolutional", "cnn", "conv",
|
||||
"segmentation", "detection", "pooling", "receptive"),
|
||||
"note": "Ch 9's prior-vs-data argument is what explains ViT's data hunger later.",
|
||||
},
|
||||
"sequence": {
|
||||
"label": "Sequence modelling — recurrence, gradients through time, attention",
|
||||
"targets": (6, 8, 10, 12),
|
||||
"keywords": ("sequence", "rnn", "lstm", "gru", "recurrent", "time series",
|
||||
"timeseries", "nlp", "language", "text", "speech", "translation",
|
||||
"transformer", "attention"),
|
||||
"note": "Read Ch 10 for the gradient analysis, not the architecture advice — "
|
||||
"see references/book_to_2026_delta.md before applying it.",
|
||||
},
|
||||
"generative": {
|
||||
"label": "Generative modelling — the Part III chain",
|
||||
"targets": (13, 14, 16, 17, 18, 19, 20),
|
||||
"keywords": ("generative", "vae", "gan", "diffusion", "autoencoder", "sampling",
|
||||
"mcmc", "boltzmann", "latent", "variational", "elbo", "density",
|
||||
"score matching", "partition"),
|
||||
"note": "Ch 18 is the chapter whose ideas became diffusion. Do not skip it to "
|
||||
"reach Ch 20.",
|
||||
},
|
||||
"representation": {
|
||||
"label": "Representation and self-supervised learning",
|
||||
"targets": (5, 13, 14, 15),
|
||||
"keywords": ("representation", "embedding", "feature", "features", "transfer",
|
||||
"pretrain", "pretraining", "self-supervised", "unsupervised",
|
||||
"disentangle", "disentangled", "probe"),
|
||||
"note": "The bet in Ch 15 was right; every method it lists was replaced. "
|
||||
"Read it with the delta reference open.",
|
||||
},
|
||||
"foundations": {
|
||||
"label": "Mathematical foundations — the machinery under everything else",
|
||||
"targets": (2, 3, 4, 5),
|
||||
"keywords": ("math", "mathematics", "foundation", "foundations", "linear algebra",
|
||||
"probability", "statistics", "theory", "fundamentals", "basics",
|
||||
"prerequisite", "prerequisites", "refresher", "interview"),
|
||||
"note": "Pair with a dedicated linear algebra text; Part I is reference "
|
||||
"material, not pedagogy.",
|
||||
},
|
||||
"optimization": {
|
||||
"label": "Optimization — why training stalls, diverges, or crawls",
|
||||
"targets": (2, 4, 8, 11),
|
||||
"keywords": ("optimization", "optimizer", "sgd", "adam", "momentum", "gradient",
|
||||
"converge", "convergence", "diverge", "learning rate", "saddle",
|
||||
"initialization", "batch norm", "normalization", "clipping"),
|
||||
"note": "Ch 8 is unreadable without Ch 2 and 4. This is the one place the "
|
||||
"prerequisite is genuinely hard.",
|
||||
},
|
||||
"complete": {
|
||||
"label": "Complete read — all twenty chapters",
|
||||
"targets": tuple(range(1, 21)),
|
||||
"keywords": ("everything", "whole book", "entire book", "all chapters",
|
||||
"cover to cover", "complete", "full read", "read the book",
|
||||
"start to finish"),
|
||||
"note": "Even here, read Ch 5 and Ch 11 early rather than in numeric order.",
|
||||
},
|
||||
}
|
||||
|
||||
# Subjects the 2016 book does not cover. Naming these beats improvising its position.
|
||||
OUT_OF_SCOPE: dict[str, str] = {
|
||||
"rlhf": "RLHF / preference tuning — published 2017+; nothing in this book covers it.",
|
||||
"dpo": "Direct preference optimization — 2023; outside the book entirely.",
|
||||
"llm": "Large language model training and serving — the book predates it; see "
|
||||
"engineering/llm-cost-optimizer and references/book_to_2026_delta.md.",
|
||||
"prompt": "Prompting and in-context learning — post-dates the book.",
|
||||
"agent": "Agentic systems — outside the book; see engineering/agent-harness.",
|
||||
"mlops": "MLOps tooling and deployment — see engineering-team/senior-ml-engineer.",
|
||||
"fine-tun": "Fine-tuning of pretrained foundation models — the book's transfer "
|
||||
"learning section (ch15) is the nearest thing, and it is not the same.",
|
||||
"lora": "Parameter-efficient fine-tuning (LoRA and relatives) — 2021+.",
|
||||
"rag": "Retrieval-augmented generation — outside the book.",
|
||||
"mamba": "State-space models — 2021+; ch10's gradient analysis is the relevant "
|
||||
"background the book does provide.",
|
||||
"fairness": "Fairness, bias auditing and model governance — not treated.",
|
||||
"reinforcement": "Reinforcement learning — mentioned only in passing (ch12).",
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- logic
|
||||
|
||||
|
||||
def close_prerequisites(targets: tuple[int, ...]) -> list[int]:
|
||||
"""Return targets plus every hard prerequisite, in ascending chapter order."""
|
||||
needed: set[int] = set()
|
||||
stack = list(targets)
|
||||
while stack:
|
||||
chapter = stack.pop()
|
||||
if chapter in needed:
|
||||
continue
|
||||
needed.add(chapter)
|
||||
stack.extend(PREREQS.get(chapter, ()))
|
||||
return sorted(needed)
|
||||
|
||||
|
||||
def order_path(chapters: list[int]) -> list[int]:
|
||||
"""Order chapters so every hard prerequisite precedes its dependent.
|
||||
|
||||
Among chapters whose prerequisites are already satisfied, the cheapest
|
||||
high-value chapter goes first: ch05 and ch11 are promoted because their
|
||||
vocabulary is reused everywhere and they are actionable immediately.
|
||||
"""
|
||||
promoted = {5: -2, 11: -1}
|
||||
remaining = set(chapters)
|
||||
placed: list[int] = []
|
||||
while remaining:
|
||||
ready = [c for c in remaining if all(p in placed for p in PREREQS.get(c, ()))]
|
||||
if not ready: # unreachable with the current acyclic table; fail loudly if it changes
|
||||
raise RuntimeError("prerequisite cycle in PREREQS")
|
||||
ready.sort(key=lambda c: (promoted.get(c, 0), c))
|
||||
nxt = ready[0]
|
||||
placed.append(nxt)
|
||||
remaining.discard(nxt)
|
||||
return placed
|
||||
|
||||
|
||||
def score_lanes(goal: str) -> list[tuple[str, int]]:
|
||||
"""Score every lane by keyword hits in the goal text, best first."""
|
||||
text = goal.lower()
|
||||
scored = []
|
||||
for key, lane in LANES.items():
|
||||
hits = sum(1 for kw in lane["keywords"] if kw in text)
|
||||
if hits:
|
||||
scored.append((key, hits))
|
||||
scored.sort(key=lambda pair: (-pair[1], pair[0]))
|
||||
return scored
|
||||
|
||||
|
||||
def out_of_scope_hits(goal: str) -> list[str]:
|
||||
text = goal.lower()
|
||||
return [note for token, note in OUT_OF_SCOPE.items() if token in text]
|
||||
|
||||
|
||||
def plan(goal: str, background: str, hours_per_week: float,
|
||||
include_optional: bool) -> dict:
|
||||
lane_key, _ = score_lanes(goal)[0]
|
||||
lane = LANES[lane_key]
|
||||
targets = lane["targets"]
|
||||
chapters = close_prerequisites(targets)
|
||||
if not include_optional and lane_key != "complete":
|
||||
# ch01 is context; keep it only when the reader asked for everything.
|
||||
chapters = [c for c in chapters if c != 1]
|
||||
ordered = order_path(chapters)
|
||||
|
||||
multiplier, background_note = BACKGROUNDS[background]
|
||||
entries = []
|
||||
for chapter in ordered:
|
||||
hours = round(BASE_HOURS[chapter] * multiplier, 1)
|
||||
entries.append({
|
||||
"chapter": chapter,
|
||||
"title": TITLES[chapter],
|
||||
"role": "target" if chapter in targets else "prerequisite",
|
||||
"hours": hours,
|
||||
"file": f"chapters/ch{chapter:02d}-*.md",
|
||||
"url": "https://www.deeplearningbook.org/",
|
||||
})
|
||||
|
||||
total_hours = round(sum(e["hours"] for e in entries), 1)
|
||||
weeks = round(total_hours / hours_per_week, 1) if hours_per_week > 0 else None
|
||||
skipped = [c for c in range(1, 21) if c not in ordered]
|
||||
|
||||
return {
|
||||
"goal": goal,
|
||||
"lane": lane_key,
|
||||
"lane_label": lane["label"],
|
||||
"background": background,
|
||||
"background_note": background_note,
|
||||
"hours_per_week": hours_per_week,
|
||||
"path": entries,
|
||||
"total_hours": total_hours,
|
||||
"estimated_weeks": weeks,
|
||||
"skipped_chapters": skipped,
|
||||
"note": lane["note"],
|
||||
"assumptions": {
|
||||
"base_hours_per_chapter": BASE_HOURS,
|
||||
"background_multiplier": multiplier,
|
||||
"basis": "planning heuristic, not measurement — recalibrate after chapter one",
|
||||
},
|
||||
"hard_prerequisites_applied": {
|
||||
str(c): list(PREREQS[c]) for c in ordered if PREREQS.get(c)
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- output
|
||||
|
||||
|
||||
def render(result: dict) -> str:
|
||||
lines = [
|
||||
"READING PATH",
|
||||
"=" * 64,
|
||||
f"Goal : {result['goal']}",
|
||||
f"Lane : {result['lane_label']}",
|
||||
f"Background : {result['background']} — {result['background_note']}",
|
||||
f"Budget : {result['total_hours']} h at {result['hours_per_week']} h/week"
|
||||
f" ≈ {result['estimated_weeks']} weeks",
|
||||
"",
|
||||
f"{'#':>3} {'ch':>4} {'hrs':>5} role title",
|
||||
"-" * 64,
|
||||
]
|
||||
for index, entry in enumerate(result["path"], start=1):
|
||||
lines.append(
|
||||
f"{index:>3} ch{entry['chapter']:02d} {entry['hours']:>5} "
|
||||
f"{entry['role']:<12} {entry['title']}"
|
||||
)
|
||||
lines.append("-" * 64)
|
||||
if result["hard_prerequisites_applied"]:
|
||||
lines.append("Hard prerequisites pulled in:")
|
||||
for chapter, prereqs in result["hard_prerequisites_applied"].items():
|
||||
names = ", ".join(f"ch{p:02d}" for p in prereqs)
|
||||
lines.append(f" ch{int(chapter):02d} needs {names}")
|
||||
if result["skipped_chapters"]:
|
||||
skipped = ", ".join(f"ch{c:02d}" for c in result["skipped_chapters"])
|
||||
lines.append(f"Not in this path: {skipped}")
|
||||
lines.append("")
|
||||
lines.append(f"Note: {result['note']}")
|
||||
lines.append("Hours are a planning heuristic. Recalibrate after your first chapter.")
|
||||
lines.append("Read the chapters free at https://www.deeplearningbook.org/")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
SAMPLE_GOAL = "I want to train convolutional models for image classification and debug them"
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Plan a prerequisite-closed reading path through the Deep Learning book.",
|
||||
epilog="Exit codes: 0 ok · 3 goal outside the book · 4 goal too vague to route.",
|
||||
)
|
||||
parser.add_argument("--goal", help="what you want to be able to do afterwards")
|
||||
parser.add_argument("--background", default="applied", choices=sorted(BACKGROUNDS),
|
||||
help="your starting point (default: applied)")
|
||||
parser.add_argument("--hours-per-week", type=float, default=5.0,
|
||||
help="study hours available per week (default: 5)")
|
||||
parser.add_argument("--include-intro", action="store_true",
|
||||
help="keep ch01, which is context rather than content")
|
||||
parser.add_argument("--output", choices=("text", "json"), default="text")
|
||||
parser.add_argument("--sample", action="store_true",
|
||||
help="run against a built-in example goal")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
goal = SAMPLE_GOAL if args.sample else args.goal
|
||||
if not goal:
|
||||
parser.error("--goal is required (or use --sample)")
|
||||
if args.hours_per_week <= 0:
|
||||
parser.error("--hours-per-week must be positive")
|
||||
|
||||
scoped_out = out_of_scope_hits(goal)
|
||||
lane_scores = score_lanes(goal)
|
||||
top_score = lane_scores[0][1] if lane_scores else 0
|
||||
# An out-of-scope subject wins over a weak lane match: "LoRA fine-tuning" hits the
|
||||
# practitioner lane on the word "tuning" while being entirely outside the book.
|
||||
if scoped_out and (len(scoped_out) >= 2 or top_score < 2):
|
||||
payload = {
|
||||
"status": "out_of_scope",
|
||||
"goal": goal,
|
||||
"reasons": scoped_out,
|
||||
"pointer": "references/book_to_2026_delta.md",
|
||||
}
|
||||
if args.output == "json":
|
||||
print(json.dumps(payload, indent=2))
|
||||
else:
|
||||
print("OUT OF SCOPE — the 2016 book does not cover this goal.\n")
|
||||
for reason in scoped_out:
|
||||
print(f" - {reason}")
|
||||
print("\nSee references/book_to_2026_delta.md for what replaced the book's "
|
||||
"position, and route to the skills named above.")
|
||||
return 3
|
||||
|
||||
if not lane_scores:
|
||||
payload = {
|
||||
"status": "unroutable",
|
||||
"goal": goal,
|
||||
"questions": [
|
||||
"What do you want to be able to DO afterwards — train, diagnose, "
|
||||
"derive, or evaluate?",
|
||||
"Which data type: images, sequences, tabular, or generative modelling?",
|
||||
"Is this a refresher over known material or a first pass?",
|
||||
],
|
||||
"lanes": {key: lane["label"] for key, lane in LANES.items()},
|
||||
}
|
||||
if args.output == "json":
|
||||
print(json.dumps(payload, indent=2))
|
||||
else:
|
||||
print("CANNOT ROUTE — the goal does not name a subject in the book.\n")
|
||||
print("Answer one of these and re-run:")
|
||||
for question in payload["questions"]:
|
||||
print(f" - {question}")
|
||||
print("\nOr name a lane directly:")
|
||||
for key, label in payload["lanes"].items():
|
||||
print(f" {key:<15} {label}")
|
||||
return 4
|
||||
|
||||
result = plan(goal, args.background, args.hours_per_week, args.include_intro)
|
||||
if scoped_out:
|
||||
result["scope_warnings"] = scoped_out
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(render(result))
|
||||
if scoped_out:
|
||||
print("\nPartly outside the book — these parts are not covered:")
|
||||
for reason in scoped_out:
|
||||
print(f" - {reason}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,306 @@
|
|||
#!/usr/bin/env python3
|
||||
"""training_diagnostics.py — Chapter 11's triage as an executable decision tree.
|
||||
|
||||
Chapter 11 argues that knowing many algorithms matters less than knowing which one a
|
||||
measurement calls for, and gives the rule most teams get backwards: *read training
|
||||
error first*. High training error means the model or the optimizer is the bottleneck,
|
||||
and collecting more data will not help.
|
||||
|
||||
This tool takes the measurements you already have from a run and returns a ranked
|
||||
diagnosis, the specific next action, and the chapter that justifies it. Rules fire in
|
||||
priority order, so a NaN is never diagnosed as overfitting.
|
||||
|
||||
The thresholds are documented defaults, not physics — every one is overridable, and
|
||||
--output json prints the values actually used.
|
||||
|
||||
Standard library only. No network calls, no model loading.
|
||||
|
||||
Exit codes:
|
||||
0 a diagnosis was produced and training looks healthy
|
||||
1 a diagnosis was produced and something needs fixing
|
||||
2 usage error (argparse)
|
||||
4 not enough measurements to diagnose — the missing ones are named
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
|
||||
# Documented defaults. Every one is overridable from the command line.
|
||||
DEFAULTS = {
|
||||
# Relative gap = (val - train) / max(train, floor). Above this, overfitting.
|
||||
"overfit_rel_gap": 0.30,
|
||||
# Below this relative gap, the run is not overfitting in any actionable sense.
|
||||
"healthy_rel_gap": 0.10,
|
||||
# Gradient norm above this is a cliff (ch08); clip before anything else.
|
||||
"grad_explode": 100.0,
|
||||
# Gradient norm below this with non-trivial loss means saturation or dead units.
|
||||
"grad_vanish": 1e-6,
|
||||
# Divisor floor so a near-zero training loss does not produce an infinite ratio.
|
||||
"loss_floor": 1e-6,
|
||||
# Training loss above target * this multiple counts as underfitting.
|
||||
"underfit_multiple": 1.15,
|
||||
}
|
||||
|
||||
|
||||
class Finding:
|
||||
def __init__(self, rank: int, code: str, verdict: str, evidence: str,
|
||||
action: str, chapter: str) -> None:
|
||||
self.rank = rank
|
||||
self.code = code
|
||||
self.verdict = verdict
|
||||
self.evidence = evidence
|
||||
self.action = action
|
||||
self.chapter = chapter
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return {
|
||||
"rank": self.rank,
|
||||
"code": self.code,
|
||||
"verdict": self.verdict,
|
||||
"evidence": self.evidence,
|
||||
"action": self.action,
|
||||
"chapter": self.chapter,
|
||||
}
|
||||
|
||||
|
||||
def diagnose(train_loss: float | None, val_loss: float | None,
|
||||
grad_norm: float | None, target_loss: float | None,
|
||||
tiny_subset_fits: str, nan_seen: bool,
|
||||
epochs: int | None, thresholds: dict) -> tuple[list[Finding], dict]:
|
||||
"""Return (findings, metrics). Rules fire in priority order; rank 1 acts first."""
|
||||
findings: list[Finding] = []
|
||||
metrics: dict = {}
|
||||
|
||||
non_finite = (
|
||||
nan_seen
|
||||
or (train_loss is not None and not math.isfinite(train_loss))
|
||||
or (val_loss is not None and not math.isfinite(val_loss))
|
||||
)
|
||||
|
||||
# --- Rule 1: numerics before modelling (ch04) -------------------------------
|
||||
if non_finite:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "NUMERICS",
|
||||
"Loss is NaN or infinite — this is a numerics failure, not a modelling one",
|
||||
"a non-finite loss was reported",
|
||||
"Check in this order: log(0) or log of a negative, division by a near-zero "
|
||||
"denominator, exp of a large logit, then an exploding gradient. Use fused "
|
||||
"log_softmax / cross-entropy rather than hand-rolled exp-then-normalize, and "
|
||||
"work in log-space for probability products.",
|
||||
"ch04 (Numerical Computation)",
|
||||
))
|
||||
# A non-finite loss makes every downstream ratio meaningless.
|
||||
return findings, metrics
|
||||
|
||||
# --- Rule 2: is it a bug at all? (ch11) -------------------------------------
|
||||
if tiny_subset_fits == "no":
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "BUG",
|
||||
"The model cannot overfit a tiny subset — this is a bug, not a hard problem",
|
||||
"--tiny-subset-fits no",
|
||||
"Stop tuning. Check the data pipeline (labels aligned with inputs?), the loss "
|
||||
"reduction, whether gradients actually reach the parameters (a detached tensor "
|
||||
"or a frozen module), and the learning rate. Compare backprop against numerical "
|
||||
"derivatives on one layer if it is still unclear.",
|
||||
"ch11 (Practical Methodology — debugging strategies)",
|
||||
))
|
||||
|
||||
# --- Rule 3: gradient behaviour (ch08, ch10, ch06) --------------------------
|
||||
if grad_norm is not None:
|
||||
metrics["grad_norm"] = grad_norm
|
||||
if grad_norm > thresholds["grad_explode"]:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "EXPLODING_GRADIENT",
|
||||
"Gradient norm is in cliff territory",
|
||||
f"grad_norm {grad_norm:g} > {thresholds['grad_explode']:g}",
|
||||
"Clip gradients by global norm before changing anything else. If a "
|
||||
"recurrence is in the graph, clipping is not optional. Then re-check the "
|
||||
"learning rate and the initialization scale.",
|
||||
"ch08 (Optimization — cliffs), ch10 (exploding gradients through time)",
|
||||
))
|
||||
elif grad_norm < thresholds["grad_vanish"]:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "VANISHING_GRADIENT",
|
||||
"Gradient norm is effectively zero — units are saturated or dead",
|
||||
f"grad_norm {grad_norm:g} < {thresholds['grad_vanish']:g}",
|
||||
"Check for saturating hidden units (sigmoid/tanh in a deep stack), dead "
|
||||
"ReLUs from a too-large learning rate, and initialization scale "
|
||||
"(He/Xavier). Add residual connections or normalization if the stack is "
|
||||
"deep.",
|
||||
"ch08 (initialization), ch06 (hidden units), ch10 (vanishing gradients)",
|
||||
))
|
||||
|
||||
# --- Rule 4/5: the fit verdict (ch05, ch07, ch11) ---------------------------
|
||||
if train_loss is not None and val_loss is not None:
|
||||
denom = max(abs(train_loss), thresholds["loss_floor"])
|
||||
gap = val_loss - train_loss
|
||||
rel_gap = gap / denom
|
||||
metrics.update({
|
||||
"train_loss": train_loss,
|
||||
"val_loss": val_loss,
|
||||
"gap": round(gap, 6),
|
||||
"relative_gap": round(rel_gap, 4),
|
||||
})
|
||||
|
||||
underfitting = None
|
||||
if target_loss is not None:
|
||||
metrics["target_loss"] = target_loss
|
||||
underfitting = train_loss > target_loss * thresholds["underfit_multiple"]
|
||||
|
||||
if underfitting:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "UNDERFIT",
|
||||
"Training error is above target — the bottleneck is capacity or optimization",
|
||||
f"train_loss {train_loss:g} > target {target_loss:g} × "
|
||||
f"{thresholds['underfit_multiple']}",
|
||||
"Do NOT collect more data — it cannot help while training error is high. "
|
||||
"In order: tune learning rate and schedule, check initialization, add "
|
||||
"capacity, remove regularization you added earlier, train longer.",
|
||||
"ch11 (the data-vs-capacity rule), ch08 (Optimization), ch05 (capacity)",
|
||||
))
|
||||
elif rel_gap > thresholds["overfit_rel_gap"]:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "OVERFIT",
|
||||
"Validation error substantially exceeds training error",
|
||||
f"relative gap {rel_gap:.2f} > {thresholds['overfit_rel_gap']}",
|
||||
"Work the regularization ladder in cost order, one change at a time: more "
|
||||
"real data → label-preserving augmentation → early stopping → decoupled "
|
||||
"weight decay (AdamW, not Adam+L2) → label smoothing → dropout. Shrink the "
|
||||
"model LAST: past the interpolation threshold, double descent means a "
|
||||
"bigger model with more data can generalize better.",
|
||||
"ch07 (Regularization), ch05 (capacity and the U-curve's caveat)",
|
||||
))
|
||||
elif rel_gap < thresholds["healthy_rel_gap"] and target_loss is None:
|
||||
findings.append(Finding(
|
||||
len(findings) + 1, "GAP_SMALL_TARGET_UNKNOWN",
|
||||
"The train/val gap is small — but without a target loss this cannot "
|
||||
"distinguish 'converged' from 'underfitting equally on both splits'",
|
||||
f"relative gap {rel_gap:.2f} < {thresholds['healthy_rel_gap']}, "
|
||||
"no --target-loss supplied",
|
||||
"Supply --target-loss (a human baseline, a published number, or the "
|
||||
"irreducible-error estimate for the task) and re-run. Chapter 11's first "
|
||||
"step is naming the metric and its target value, and this is why.",
|
||||
"ch11 (determine goals: error metric and target value)",
|
||||
))
|
||||
|
||||
if not findings:
|
||||
findings.append(Finding(
|
||||
1, "HEALTHY",
|
||||
"No rule fired — the measurements supplied look healthy",
|
||||
"; ".join(f"{k}={v}" for k, v in metrics.items()) or "measurements within thresholds",
|
||||
"Confirm the metric you are optimizing is the one you care about, then change "
|
||||
"one thing per experiment and keep the log.",
|
||||
"ch11 (Practical Methodology)",
|
||||
))
|
||||
|
||||
if epochs is not None:
|
||||
metrics["epochs"] = epochs
|
||||
return findings, metrics
|
||||
|
||||
|
||||
def render(findings: list[Finding], metrics: dict, thresholds: dict) -> str:
|
||||
lines = ["TRAINING DIAGNOSIS", "=" * 68]
|
||||
if metrics:
|
||||
lines.append("Measurements: " + " ".join(f"{k}={v}" for k, v in metrics.items()))
|
||||
lines.append("")
|
||||
for finding in findings:
|
||||
lines.append(f"[{finding.rank}] {finding.code} — {finding.verdict}")
|
||||
lines.append(f" evidence : {finding.evidence}")
|
||||
lines.append(f" action : {finding.action}")
|
||||
lines.append(f" chapter : {finding.chapter}")
|
||||
lines.append("")
|
||||
lines.append("Thresholds used: " + ", ".join(f"{k}={v}" for k, v in thresholds.items()))
|
||||
lines.append("Rules fire in priority order — act on [1] before anything below it.")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Diagnose a training run against Chapter 11's decision tree.",
|
||||
epilog="Exit codes: 0 healthy · 1 something needs fixing · 4 not enough input.",
|
||||
)
|
||||
parser.add_argument("--train-loss", type=float, help="final or current training loss")
|
||||
parser.add_argument("--val-loss", type=float, help="matching validation loss")
|
||||
parser.add_argument("--target-loss", type=float,
|
||||
help="the loss value that would count as success — a human "
|
||||
"baseline, a published number, or an irreducible-error "
|
||||
"estimate. Without it, underfitting cannot be detected.")
|
||||
parser.add_argument("--grad-norm", type=float, help="observed global gradient norm")
|
||||
parser.add_argument("--epochs", type=int, help="epochs trained so far (recorded only)")
|
||||
parser.add_argument("--nan", action="store_true",
|
||||
help="the loss went NaN or inf at any point")
|
||||
parser.add_argument("--tiny-subset-fits", choices=("yes", "no", "unknown"),
|
||||
default="unknown",
|
||||
help="can the model drive training loss to ~0 on 10-50 examples? "
|
||||
"(ch11's smoke test; default: unknown)")
|
||||
parser.add_argument("--overfit-rel-gap", type=float, default=DEFAULTS["overfit_rel_gap"])
|
||||
parser.add_argument("--healthy-rel-gap", type=float, default=DEFAULTS["healthy_rel_gap"])
|
||||
parser.add_argument("--grad-explode", type=float, default=DEFAULTS["grad_explode"])
|
||||
parser.add_argument("--grad-vanish", type=float, default=DEFAULTS["grad_vanish"])
|
||||
parser.add_argument("--underfit-multiple", type=float,
|
||||
default=DEFAULTS["underfit_multiple"])
|
||||
parser.add_argument("--output", choices=("text", "json"), default="text")
|
||||
parser.add_argument("--sample", action="store_true",
|
||||
help="run against a built-in overfitting example")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
args.train_loss, args.val_loss = 0.02, 1.90
|
||||
args.grad_norm, args.epochs = 0.4, 30
|
||||
args.tiny_subset_fits = "yes"
|
||||
|
||||
thresholds = {
|
||||
"overfit_rel_gap": args.overfit_rel_gap,
|
||||
"healthy_rel_gap": args.healthy_rel_gap,
|
||||
"grad_explode": args.grad_explode,
|
||||
"grad_vanish": args.grad_vanish,
|
||||
"loss_floor": DEFAULTS["loss_floor"],
|
||||
"underfit_multiple": args.underfit_multiple,
|
||||
}
|
||||
|
||||
have_losses = args.train_loss is not None and args.val_loss is not None
|
||||
if not have_losses and not args.nan and args.grad_norm is None \
|
||||
and args.tiny_subset_fits == "unknown":
|
||||
missing = {
|
||||
"status": "insufficient_input",
|
||||
"need_at_least_one_of": [
|
||||
"--train-loss with --val-loss",
|
||||
"--grad-norm",
|
||||
"--nan",
|
||||
"--tiny-subset-fits yes|no",
|
||||
],
|
||||
"note": "Chapter 11's first instruction is to instrument the run. This tool "
|
||||
"reads instruments; it does not guess.",
|
||||
}
|
||||
if args.output == "json":
|
||||
print(json.dumps(missing, indent=2))
|
||||
else:
|
||||
print("NOT ENOUGH INPUT — supply at least one of:")
|
||||
for item in missing["need_at_least_one_of"]:
|
||||
print(f" {item}")
|
||||
print(f"\n{missing['note']}")
|
||||
return 4
|
||||
|
||||
findings, metrics = diagnose(
|
||||
args.train_loss, args.val_loss, args.grad_norm, args.target_loss,
|
||||
args.tiny_subset_fits, args.nan, args.epochs, thresholds,
|
||||
)
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps({
|
||||
"findings": [f.as_dict() for f in findings],
|
||||
"metrics": metrics,
|
||||
"thresholds": thresholds,
|
||||
}, indent=2))
|
||||
else:
|
||||
print(render(findings, metrics, thresholds))
|
||||
|
||||
return 0 if findings[0].code == "HEALTHY" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Reference in a new issue