From eeb3cb9ad66552d8d19a7dbf6f545185c6878057 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 19:17:50 +0000 Subject: [PATCH] fix(deep-learning-book): position-wise linear, correcting a 512x parameter error in the example asset MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third review on PR #994 found a real bug in the flagship example asset. Verified before fixing: assets/example_layer_spec.json reported 1,207,962,624 parameters for the feedforward up-projection instead of 2,362,368 — off by exactly 512x, the sequence length — putting the block's total at ~1.21B instead of ~7.09M. Root cause was in the tool, not only the asset. model_arithmetic.py's mha layer emits (seq, d_model) but linear refused any 2-D input, so the only way to attach a feedforward block was to flatten first. Flattening collapses all 512 positions into one 393,216-element vector, which models a dense layer over the whole sequence — a different layer, with seq_len times the parameters. A transformer FFN was therefore not expressible at all, and the shipped example walked straight into it. Clean exit is not correct numbers, which is why --sample exit-code testing never caught it. Fixed the cause: linear on a 2-D (seq, features) input is now position-wise — one weight matrix shared across positions, parameters independent of sequence length, compute linear in it. Documented in the module docstring; the 3-D path still refuses with an updated message pointing at flatten. Removed the flatten from the example asset and recorded in its comment why it must not come back. Verified: the corrected block reports 7,087,872 parameters, matching a hand-check of 2*(2*768) + (4*768^2+4*768) + (768*3072+3072) + (3072*768+768) exactly, and the size of a BERT-base encoder layer. The convnet --sample is unchanged at 545,098, and linear on 3-D input still exits 5. Also adds the missing CHANGELOG.md [Unreleased] entry, which the same review noted: CLAUDE.md, README.md and marketplace.json carried the new skill and its counter deltas but CHANGELOG.md did not. Gates green: compileall, check_paths, check_frontmatter, check_dual_publish, check_model_freshness, smoke_scripts (696 passed), derive_counters --check, check_skill_names, check_plugin_json, book_skill_validator, and --help + --sample --output json on all four tools. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01BswsZp5zrJWFAGU6KWNA1s --- CHANGELOG.md | 41 +++++++++++++++++ .../assets/example_layer_spec.json | 45 +++++++++++++++---- .../scripts/model_arithmetic.py | 22 +++++++-- 3 files changed, 95 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 886cabf6..09ab4a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added — engineering/deep-learning-book: a companion to the free Deep Learning textbook + +New `engineering/deep-learning-book/` plugin: a study companion for *Deep Learning* by +Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org. +One skill, 4 stdlib-only tools, 4 references, 3 assets, 1 agent, 3 commands. + +- **Companion, not compilation — and that was the design decision.** `book-to-skill`'s + rights gate refuses a `shareable` package without `public-domain` / `open-license` / + `internal-docs` / `author-permission`, none of which applies to an MIT Press title whose + own site states the HTML-only format exists as a friction against copying under the + authors' contract; its rights reference lists publishing a compiled skill of a copyrighted + book to a public marketplace under **Do not**, and its hard rule 1 forbids scraping a book + from the web. So nothing here reproduces the book: every chapter file is original + synthesis linking to the official free chapter, and the organizing structure is the + published table of contents. **The rule this sets:** convert a copyrighted work into a + companion that indexes and updates the source, never a compilation that reproduces it. +- **The compiled-skill shape, validated by the compiler's own gate.** Master `SKILL.md` + (~2.0k tokens, chapter index + topic index), `chapters/ch01..ch20`, `glossary.md`, + `patterns.md`, `cheatsheet.md` — passes `book_skill_validator.py` clean with every file + inside `token_budget_estimator.py`'s caps. +- **The 2016→2026 delta layer is the differentiator.** A compilation freezes a source at its + publication date; this one dates it. Every chapter carries "What changed after 2016", and + `references/book_to_2026_delta.md` gives five corrections with primary citations and + per-claim confidence: double descent qualifying Ch 5's U-curve, AdamW splitting weight + decay from L2, transformers displacing Ch 10's recurrence, diffusion growing out of Ch 18's + score matching, and self-supervised learning vindicating Ch 15 while replacing its methods. + Two contested claims are marked contested rather than propagated; two named as folklore. + Stated rule: **the conflict is almost always in the recommendation, not the analysis.** +- **Four tools, each with a real refusal.** `reading_path_planner.py` (prerequisite closure + over the book's actual dependency graph, priced in weeks; exit 3 naming what covers an + out-of-scope goal, exit 4 with forcing questions when unroutable; ties break on keyword + specificity, not alphabetically); `training_diagnostics.py` (Ch 11's rules in priority + order, so a NaN is never reported as overfitting; exit 4 rather than diagnosing with no + instruments); `capacity_planner.py` (regularization ladder in cost order with "shrink the + model" ranked **last** in the overparameterized regime; exit 4 on a val-below-train split); + `model_arithmetic.py` (params/FLOPs/activation memory for conv, linear, position-wise + linear, MHA and LSTM/GRU stacks; exit 5 naming the layer whose shapes do not connect). +- `cs-deep-learning-tutor` agent; `/cs:deep-learning`, `/cs:dl-reading-path`, + `/cs:dl-diagnose`. **Counters:** skills 386 → 387; tools 723 → 727; refs 838 → 842; + agents 116 → 117; commands 146 → 149; plugins 97 → 98. + ### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code New `marketing/linkedin/` plugin, answering diff --git a/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json b/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json index 9e2b5aa5..ea91f7ab 100644 --- a/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json +++ b/engineering/deep-learning-book/skills/deep-learning-book/assets/example_layer_spec.json @@ -1,14 +1,41 @@ { "name": "transformer encoder block, 768-dim, 512 tokens", - "_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. Note how the mha row's cost grows quadratically with the input sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.", + "_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. The two feedforward linears run position-wise over the (512, 768) sequence: one weight matrix shared across all 512 tokens, which is what a transformer FFN actually is. Do NOT insert a flatten before them \u2014 that models a dense layer over the whole flattened sequence and inflates the parameter count by 512x. Note how the mha row's cost grows quadratically with sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.", "layers": [ - {"type": "input", "shape": [512, 768], "name": "token embeddings in"}, - {"type": "layernorm", "name": "pre-norm 1"}, - {"type": "mha", "heads": 12, "name": "self-attention"}, - {"type": "layernorm", "name": "pre-norm 2"}, - {"type": "flatten", "name": "flatten for the feedforward accounting"}, - {"type": "linear", "units": 3072, "name": "ffn up"}, - {"type": "activation", "name": "gelu"}, - {"type": "linear", "units": 768, "name": "ffn down"} + { + "type": "input", + "shape": [ + 512, + 768 + ], + "name": "token embeddings in" + }, + { + "type": "layernorm", + "name": "pre-norm 1" + }, + { + "type": "mha", + "heads": 12, + "name": "self-attention" + }, + { + "type": "layernorm", + "name": "pre-norm 2" + }, + { + "type": "linear", + "units": 3072, + "name": "ffn up (position-wise)" + }, + { + "type": "activation", + "name": "gelu" + }, + { + "type": "linear", + "units": 768, + "name": "ffn down (position-wise)" + } ] } diff --git a/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py b/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py index b3cd9cb7..3dba350b 100644 --- a/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py +++ b/engineering/deep-learning-book/skills/deep-learning-book/scripts/model_arithmetic.py @@ -14,6 +14,12 @@ forward pass; a training step costs roughly 3x a forward pass (forward + backwar Layer types: input, linear, conv2d, pool2d, flatten, embedding, layernorm, activation, dropout, mha (multi-head self-attention), lstm, gru. +A linear layer on a 2-D (seq, features) input is treated as position-wise: one weight +matrix shared across all positions, as in a transformer feedforward block. Parameters do +not scale with sequence length; compute does. Flatten first only when you really mean a +dense layer over the whole flattened sequence — that is a different layer, and its +parameter count is seq_len times larger. + conv2d "same" padding follows TensorFlow/Keras SAME: output is ceil(H / stride), with any needed padding split across the two sides (and the extra pixel going to the bottom and right at even kernel sizes). PyTorch's padding='same' is symmetric-only and rejects @@ -72,13 +78,21 @@ def step(layer: dict, shape: tuple[int, ...], index: int) -> tuple[tuple[int, .. if kind == "linear": units = int(_require(layer, "units", index)) + bias = bool(layer.get("bias", True)) + if len(shape) == 2: + # Per-token (position-wise) linear over a (seq, features) sequence: one + # weight matrix shared across positions, exactly like a transformer FFN + # projection. Parameters do NOT scale with sequence length; compute does. + # Flattening instead would multiply the parameter count by seq_len, which + # is a different layer and almost never the intended one. + seq, features = shape + params = features * units + (units if bias else 0) + return (seq, units), params, seq * features * units if len(shape) != 1: raise ShapeError( - f"layer {index} (linear) needs a 1-D input, got {shape}. " - "Insert a flatten layer, or use a per-token linear on a 2-D sequence " - "by declaring the shape as [features]." + f"layer {index} (linear) needs a 1-D or 2-D input, got {shape}. " + "Insert a flatten layer to collapse a feature map into one vector." ) - bias = bool(layer.get("bias", True)) params = shape[0] * units + (units if bias else 0) return (units,), params, shape[0] * units