diff --git a/CHANGELOG.md b/CHANGELOG.md index 886cabf6..09ab4a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added — engineering/deep-learning-book: a companion to the free Deep Learning textbook + +New `engineering/deep-learning-book/` plugin: a study companion for *Deep Learning* by +Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org. +One skill, 4 stdlib-only tools, 4 references, 3 assets, 1 agent, 3 commands. + +- **Companion, not compilation — and that was the design decision.** `book-to-skill`'s + rights gate refuses a `shareable` package without `public-domain` / `open-license` / + `internal-docs` / `author-permission`, none of which applies to an MIT Press title whose + own site states the HTML-only format exists as a friction against copying under the + authors' contract; its rights reference lists publishing a compiled skill of a copyrighted + book to a public marketplace under **Do not**, and its hard rule 1 forbids scraping a book + from the web. So nothing here reproduces the book: every chapter file is original + synthesis linking to the official free chapter, and the organizing structure is the + published table of contents. **The rule this sets:** convert a copyrighted work into a + companion that indexes and updates the source, never a compilation that reproduces it. +- **The compiled-skill shape, validated by the compiler's own gate.** Master `SKILL.md` + (~2.0k tokens, chapter index + topic index), `chapters/ch01..ch20`, `glossary.md`, + `patterns.md`, `cheatsheet.md` — passes `book_skill_validator.py` clean with every file + inside `token_budget_estimator.py`'s caps. +- **The 2016→2026 delta layer is the differentiator.** A compilation freezes a source at its + publication date; this one dates it. Every chapter carries "What changed after 2016", and + `references/book_to_2026_delta.md` gives five corrections with primary citations and + per-claim confidence: double descent qualifying Ch 5's U-curve, AdamW splitting weight + decay from L2, transformers displacing Ch 10's recurrence, diffusion growing out of Ch 18's + score matching, and self-supervised learning vindicating Ch 15 while replacing its methods. + Two contested claims are marked contested rather than propagated; two named as folklore. + Stated rule: **the conflict is almost always in the recommendation, not the analysis.** +- **Four tools, each with a real refusal.** `reading_path_planner.py` (prerequisite closure + over the book's actual dependency graph, priced in weeks; exit 3 naming what covers an + out-of-scope goal, exit 4 with forcing questions when unroutable; ties break on keyword + specificity, not alphabetically); `training_diagnostics.py` (Ch 11's rules in priority + order, so a NaN is never reported as overfitting; exit 4 rather than diagnosing with no + instruments); `capacity_planner.py` (regularization ladder in cost order with "shrink the + model" ranked **last** in the overparameterized regime; exit 4 on a val-below-train split); + `model_arithmetic.py` (params/FLOPs/activation memory for conv, linear, position-wise + linear, MHA and LSTM/GRU stacks; exit 5 naming the layer whose shapes do not connect). +- `cs-deep-learning-tutor` agent; `/cs:deep-learning`, `/cs:dl-reading-path`, + `/cs:dl-diagnose`. **Counters:** skills 386 → 387; tools 723 → 727; refs 838 → 842; + agents 116 → 117; commands 146 → 149; plugins 97 → 98. + ### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code New `marketing/linkedin/` plugin, answering diff --git a/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json b/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json index 9e2b5aa5..ea91f7ab 100644 --- a/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json +++ b/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json @@ -1,14 +1,41 @@ { "name": "transformer encoder block, 768-dim, 512 tokens", - "_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. Note how the mha row's cost grows quadratically with the input sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.", + "_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. The two feedforward linears run position-wise over the (512, 768) sequence: one weight matrix shared across all 512 tokens, which is what a transformer FFN actually is. Do NOT insert a flatten before them \u2014 that models a dense layer over the whole flattened sequence and inflates the parameter count by 512x. Note how the mha row's cost grows quadratically with sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.", "layers": [ - {"type": "input", "shape": [512, 768], "name": "token embeddings in"}, - {"type": "layernorm", "name": "pre-norm 1"}, - {"type": "mha", "heads": 12, "name": "self-attention"}, - {"type": "layernorm", "name": "pre-norm 2"}, - {"type": "flatten", "name": "flatten for the feedforward accounting"}, - {"type": "linear", "units": 3072, "name": "ffn up"}, - {"type": "activation", "name": "gelu"}, - {"type": "linear", "units": 768, "name": "ffn down"} + { + "type": "input", + "shape": [ + 512, + 768 + ], + "name": "token embeddings in" + }, + { + "type": "layernorm", + "name": "pre-norm 1" + }, + { + "type": "mha", + "heads": 12, + "name": "self-attention" + }, + { + "type": "layernorm", + "name": "pre-norm 2" + }, + { + "type": "linear", + "units": 3072, + "name": "ffn up (position-wise)" + }, + { + "type": "activation", + "name": "gelu" + }, + { + "type": "linear", + "units": 768, + "name": "ffn down (position-wise)" + } ] } diff --git a/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py b/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py index b3cd9cb7..3dba350b 100644 --- a/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py +++ b/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py @@ -14,6 +14,12 @@ forward pass; a training step costs roughly 3x a forward pass (forward + backwar Layer types: input, linear, conv2d, pool2d, flatten, embedding, layernorm, activation, dropout, mha (multi-head self-attention), lstm, gru. +A linear layer on a 2-D (seq, features) input is treated as position-wise: one weight +matrix shared across all positions, as in a transformer feedforward block. Parameters do +not scale with sequence length; compute does. Flatten first only when you really mean a +dense layer over the whole flattened sequence — that is a different layer, and its +parameter count is seq_len times larger. + conv2d "same" padding follows TensorFlow/Keras SAME: output is ceil(H / stride), with any needed padding split across the two sides (and the extra pixel going to the bottom and right at even kernel sizes). PyTorch's padding='same' is symmetric-only and rejects @@ -72,13 +78,21 @@ def step(layer: dict, shape: tuple[int, ...], index: int) -> tuple[tuple[int, .. if kind == "linear": units = int(_require(layer, "units", index)) + bias = bool(layer.get("bias", True)) + if len(shape) == 2: + # Per-token (position-wise) linear over a (seq, features) sequence: one + # weight matrix shared across positions, exactly like a transformer FFN + # projection. Parameters do NOT scale with sequence length; compute does. + # Flattening instead would multiply the parameter count by seq_len, which + # is a different layer and almost never the intended one. + seq, features = shape + params = features * units + (units if bias else 0) + return (seq, units), params, seq * features * units if len(shape) != 1: raise ShapeError( - f"layer {index} (linear) needs a 1-D input, got {shape}. " - "Insert a flatten layer, or use a per-token linear on a 2-D sequence " - "by declaring the shape as [features]." + f"layer {index} (linear) needs a 1-D or 2-D input, got {shape}. " + "Insert a flatten layer to collapse a feature map into one vector." ) - bias = bool(layer.get("bias", True)) params = shape[0] * units + (units if bias else 0) return (units,), params, shape[0] * units