fix(deep-learning-book): position-wise linear, correcting a 512x parameter error in the example asset

Third review on PR #994 found a real bug in the flagship example asset. Verified
before fixing: assets/example_layer_spec.json reported 1,207,962,624 parameters
for the feedforward up-projection instead of 2,362,368 — off by exactly 512x, the
sequence length — putting the block's total at ~1.21B instead of ~7.09M.

Root cause was in the tool, not only the asset. model_arithmetic.py's mha layer
emits (seq, d_model) but linear refused any 2-D input, so the only way to attach a
feedforward block was to flatten first. Flattening collapses all 512 positions into
one 393,216-element vector, which models a dense layer over the whole sequence — a
different layer, with seq_len times the parameters. A transformer FFN was therefore
not expressible at all, and the shipped example walked straight into it. Clean exit
is not correct numbers, which is why --sample exit-code testing never caught it.

Fixed the cause: linear on a 2-D (seq, features) input is now position-wise — one
weight matrix shared across positions, parameters independent of sequence length,
compute linear in it. Documented in the module docstring; the 3-D path still refuses
with an updated message pointing at flatten. Removed the flatten from the example
asset and recorded in its comment why it must not come back.

Verified: the corrected block reports 7,087,872 parameters, matching a hand-check of
2*(2*768) + (4*768^2+4*768) + (768*3072+3072) + (3072*768+768) exactly, and the size
of a BERT-base encoder layer. The convnet --sample is unchanged at 545,098, and
linear on 3-D input still exits 5.

Also adds the missing CHANGELOG.md [Unreleased] entry, which the same review noted:
CLAUDE.md, README.md and marketplace.json carried the new skill and its counter
deltas but CHANGELOG.md did not.

Gates green: compileall, check_paths, check_frontmatter, check_dual_publish,
check_model_freshness, smoke_scripts (696 passed), derive_counters --check,
check_skill_names, check_plugin_json, book_skill_validator, and --help +
--sample --output json on all four tools.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BswsZp5zrJWFAGU6KWNA1s
This commit is contained in:
Claude 2026-08-25 19:17:50 +00:00
parent 1366714fdc
commit eeb3cb9ad6
No known key found for this signature in database
3 changed files with 95 additions and 13 deletions

View file

@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added — engineering/deep-learning-book: a companion to the free Deep Learning textbook
New `engineering/deep-learning-book/` plugin: a study companion for *Deep Learning* by
Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org.
One skill, 4 stdlib-only tools, 4 references, 3 assets, 1 agent, 3 commands.
- **Companion, not compilation — and that was the design decision.** `book-to-skill`'s
rights gate refuses a `shareable` package without `public-domain` / `open-license` /
`internal-docs` / `author-permission`, none of which applies to an MIT Press title whose
own site states the HTML-only format exists as a friction against copying under the
authors' contract; its rights reference lists publishing a compiled skill of a copyrighted
book to a public marketplace under **Do not**, and its hard rule 1 forbids scraping a book
from the web. So nothing here reproduces the book: every chapter file is original
synthesis linking to the official free chapter, and the organizing structure is the
published table of contents. **The rule this sets:** convert a copyrighted work into a
companion that indexes and updates the source, never a compilation that reproduces it.
- **The compiled-skill shape, validated by the compiler's own gate.** Master `SKILL.md`
(~2.0k tokens, chapter index + topic index), `chapters/ch01..ch20`, `glossary.md`,
`patterns.md`, `cheatsheet.md` — passes `book_skill_validator.py` clean with every file
inside `token_budget_estimator.py`'s caps.
- **The 2016→2026 delta layer is the differentiator.** A compilation freezes a source at its
publication date; this one dates it. Every chapter carries "What changed after 2016", and
`references/book_to_2026_delta.md` gives five corrections with primary citations and
per-claim confidence: double descent qualifying Ch 5's U-curve, AdamW splitting weight
decay from L2, transformers displacing Ch 10's recurrence, diffusion growing out of Ch 18's
score matching, and self-supervised learning vindicating Ch 15 while replacing its methods.
Two contested claims are marked contested rather than propagated; two named as folklore.
Stated rule: **the conflict is almost always in the recommendation, not the analysis.**
- **Four tools, each with a real refusal.** `reading_path_planner.py` (prerequisite closure
over the book's actual dependency graph, priced in weeks; exit 3 naming what covers an
out-of-scope goal, exit 4 with forcing questions when unroutable; ties break on keyword
specificity, not alphabetically); `training_diagnostics.py` (Ch 11's rules in priority
order, so a NaN is never reported as overfitting; exit 4 rather than diagnosing with no
instruments); `capacity_planner.py` (regularization ladder in cost order with "shrink the
model" ranked **last** in the overparameterized regime; exit 4 on a val-below-train split);
`model_arithmetic.py` (params/FLOPs/activation memory for conv, linear, position-wise
linear, MHA and LSTM/GRU stacks; exit 5 naming the layer whose shapes do not connect).
- `cs-deep-learning-tutor` agent; `/cs:deep-learning`, `/cs:dl-reading-path`,
`/cs:dl-diagnose`. **Counters:** skills 386 → 387; tools 723 → 727; refs 838 → 842;
agents 116 → 117; commands 146 → 149; plugins 97 → 98.
### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code
New `marketing/linkedin/` plugin, answering

View file

@ -1,14 +1,41 @@
{
"name": "transformer encoder block, 768-dim, 512 tokens",
"_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. Note how the mha row's cost grows quadratically with the input sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.",
"_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. The two feedforward linears run position-wise over the (512, 768) sequence: one weight matrix shared across all 512 tokens, which is what a transformer FFN actually is. Do NOT insert a flatten before them \u2014 that models a dense layer over the whole flattened sequence and inflates the parameter count by 512x. Note how the mha row's cost grows quadratically with sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.",
"layers": [
{"type": "input", "shape": [512, 768], "name": "token embeddings in"},
{"type": "layernorm", "name": "pre-norm 1"},
{"type": "mha", "heads": 12, "name": "self-attention"},
{"type": "layernorm", "name": "pre-norm 2"},
{"type": "flatten", "name": "flatten for the feedforward accounting"},
{"type": "linear", "units": 3072, "name": "ffn up"},
{"type": "activation", "name": "gelu"},
{"type": "linear", "units": 768, "name": "ffn down"}
{
"type": "input",
"shape": [
512,
768
],
"name": "token embeddings in"
},
{
"type": "layernorm",
"name": "pre-norm 1"
},
{
"type": "mha",
"heads": 12,
"name": "self-attention"
},
{
"type": "layernorm",
"name": "pre-norm 2"
},
{
"type": "linear",
"units": 3072,
"name": "ffn up (position-wise)"
},
{
"type": "activation",
"name": "gelu"
},
{
"type": "linear",
"units": 768,
"name": "ffn down (position-wise)"
}
]
}

View file

@ -14,6 +14,12 @@ forward pass; a training step costs roughly 3x a forward pass (forward + backwar
Layer types: input, linear, conv2d, pool2d, flatten, embedding, layernorm, activation,
dropout, mha (multi-head self-attention), lstm, gru.
A linear layer on a 2-D (seq, features) input is treated as position-wise: one weight
matrix shared across all positions, as in a transformer feedforward block. Parameters do
not scale with sequence length; compute does. Flatten first only when you really mean a
dense layer over the whole flattened sequence that is a different layer, and its
parameter count is seq_len times larger.
conv2d "same" padding follows TensorFlow/Keras SAME: output is ceil(H / stride), with
any needed padding split across the two sides (and the extra pixel going to the bottom
and right at even kernel sizes). PyTorch's padding='same' is symmetric-only and rejects
@ -72,13 +78,21 @@ def step(layer: dict, shape: tuple[int, ...], index: int) -> tuple[tuple[int, ..
if kind == "linear":
units = int(_require(layer, "units", index))
bias = bool(layer.get("bias", True))
if len(shape) == 2:
# Per-token (position-wise) linear over a (seq, features) sequence: one
# weight matrix shared across positions, exactly like a transformer FFN
# projection. Parameters do NOT scale with sequence length; compute does.
# Flattening instead would multiply the parameter count by seq_len, which
# is a different layer and almost never the intended one.
seq, features = shape
params = features * units + (units if bias else 0)
return (seq, units), params, seq * features * units
if len(shape) != 1:
raise ShapeError(
f"layer {index} (linear) needs a 1-D input, got {shape}. "
"Insert a flatten layer, or use a per-token linear on a 2-D sequence "
"by declaring the shape as [features]."
f"layer {index} (linear) needs a 1-D or 2-D input, got {shape}. "
"Insert a flatten layer to collapse a feature map into one vector."
)
bias = bool(layer.get("bias", True))
params = shape[0] * units + (units if bias else 0)
return (units,), params, shape[0] * units