mirror of
https://github.com/alirezarezvani/claude-skills.git
synced 2026-08-28 04:24:58 +00:00
fix(deep-learning-book): position-wise linear, correcting a 512x parameter error in the example asset
Third review on PR #994 found a real bug in the flagship example asset. Verified before fixing: assets/example_layer_spec.json reported 1,207,962,624 parameters for the feedforward up-projection instead of 2,362,368 — off by exactly 512x, the sequence length — putting the block's total at ~1.21B instead of ~7.09M. Root cause was in the tool, not only the asset. model_arithmetic.py's mha layer emits (seq, d_model) but linear refused any 2-D input, so the only way to attach a feedforward block was to flatten first. Flattening collapses all 512 positions into one 393,216-element vector, which models a dense layer over the whole sequence — a different layer, with seq_len times the parameters. A transformer FFN was therefore not expressible at all, and the shipped example walked straight into it. Clean exit is not correct numbers, which is why --sample exit-code testing never caught it. Fixed the cause: linear on a 2-D (seq, features) input is now position-wise — one weight matrix shared across positions, parameters independent of sequence length, compute linear in it. Documented in the module docstring; the 3-D path still refuses with an updated message pointing at flatten. Removed the flatten from the example asset and recorded in its comment why it must not come back. Verified: the corrected block reports 7,087,872 parameters, matching a hand-check of 2*(2*768) + (4*768^2+4*768) + (768*3072+3072) + (3072*768+768) exactly, and the size of a BERT-base encoder layer. The convnet --sample is unchanged at 545,098, and linear on 3-D input still exits 5. Also adds the missing CHANGELOG.md [Unreleased] entry, which the same review noted: CLAUDE.md, README.md and marketplace.json carried the new skill and its counter deltas but CHANGELOG.md did not. Gates green: compileall, check_paths, check_frontmatter, check_dual_publish, check_model_freshness, smoke_scripts (696 passed), derive_counters --check, check_skill_names, check_plugin_json, book_skill_validator, and --help + --sample --output json on all four tools. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BswsZp5zrJWFAGU6KWNA1s
This commit is contained in:
parent
1366714fdc
commit
eeb3cb9ad6
3 changed files with 95 additions and 13 deletions
41
CHANGELOG.md
41
CHANGELOG.md
|
|
@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Added — engineering/deep-learning-book: a companion to the free Deep Learning textbook
|
||||
|
||||
New `engineering/deep-learning-book/` plugin: a study companion for *Deep Learning* by
|
||||
Goodfellow, Bengio & Courville (MIT Press, 2016), free to read at deeplearningbook.org.
|
||||
One skill, 4 stdlib-only tools, 4 references, 3 assets, 1 agent, 3 commands.
|
||||
|
||||
- **Companion, not compilation — and that was the design decision.** `book-to-skill`'s
|
||||
rights gate refuses a `shareable` package without `public-domain` / `open-license` /
|
||||
`internal-docs` / `author-permission`, none of which applies to an MIT Press title whose
|
||||
own site states the HTML-only format exists as a friction against copying under the
|
||||
authors' contract; its rights reference lists publishing a compiled skill of a copyrighted
|
||||
book to a public marketplace under **Do not**, and its hard rule 1 forbids scraping a book
|
||||
from the web. So nothing here reproduces the book: every chapter file is original
|
||||
synthesis linking to the official free chapter, and the organizing structure is the
|
||||
published table of contents. **The rule this sets:** convert a copyrighted work into a
|
||||
companion that indexes and updates the source, never a compilation that reproduces it.
|
||||
- **The compiled-skill shape, validated by the compiler's own gate.** Master `SKILL.md`
|
||||
(~2.0k tokens, chapter index + topic index), `chapters/ch01..ch20`, `glossary.md`,
|
||||
`patterns.md`, `cheatsheet.md` — passes `book_skill_validator.py` clean with every file
|
||||
inside `token_budget_estimator.py`'s caps.
|
||||
- **The 2016→2026 delta layer is the differentiator.** A compilation freezes a source at its
|
||||
publication date; this one dates it. Every chapter carries "What changed after 2016", and
|
||||
`references/book_to_2026_delta.md` gives five corrections with primary citations and
|
||||
per-claim confidence: double descent qualifying Ch 5's U-curve, AdamW splitting weight
|
||||
decay from L2, transformers displacing Ch 10's recurrence, diffusion growing out of Ch 18's
|
||||
score matching, and self-supervised learning vindicating Ch 15 while replacing its methods.
|
||||
Two contested claims are marked contested rather than propagated; two named as folklore.
|
||||
Stated rule: **the conflict is almost always in the recommendation, not the analysis.**
|
||||
- **Four tools, each with a real refusal.** `reading_path_planner.py` (prerequisite closure
|
||||
over the book's actual dependency graph, priced in weeks; exit 3 naming what covers an
|
||||
out-of-scope goal, exit 4 with forcing questions when unroutable; ties break on keyword
|
||||
specificity, not alphabetically); `training_diagnostics.py` (Ch 11's rules in priority
|
||||
order, so a NaN is never reported as overfitting; exit 4 rather than diagnosing with no
|
||||
instruments); `capacity_planner.py` (regularization ladder in cost order with "shrink the
|
||||
model" ranked **last** in the overparameterized regime; exit 4 on a val-below-train split);
|
||||
`model_arithmetic.py` (params/FLOPs/activation memory for conv, linear, position-wise
|
||||
linear, MHA and LSTM/GRU stacks; exit 5 naming the layer whose shapes do not connect).
|
||||
- `cs-deep-learning-tutor` agent; `/cs:deep-learning`, `/cs:dl-reading-path`,
|
||||
`/cs:dl-diagnose`. **Counters:** skills 386 → 387; tools 723 → 727; refs 838 → 842;
|
||||
agents 116 → 117; commands 146 → 149; plugins 97 → 98.
|
||||
|
||||
### Added — marketing/linkedin: organic LinkedIn presence with the platform rules in code
|
||||
|
||||
New `marketing/linkedin/` plugin, answering
|
||||
|
|
|
|||
|
|
@ -1,14 +1,41 @@
|
|||
{
|
||||
"name": "transformer encoder block, 768-dim, 512 tokens",
|
||||
"_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. Note how the mha row's cost grows quadratically with the input sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.",
|
||||
"_comment": "Feed to model_arithmetic.py --spec. Shapes are per example, no batch dim. The two feedforward linears run position-wise over the (512, 768) sequence: one weight matrix shared across all 512 tokens, which is what a transformer FFN actually is. Do NOT insert a flatten before them \u2014 that models a dense layer over the whole flattened sequence and inflates the parameter count by 512x. Note how the mha row's cost grows quadratically with sequence length: double 512 to 1024 and the attention term quadruples while the projections only double.",
|
||||
"layers": [
|
||||
{"type": "input", "shape": [512, 768], "name": "token embeddings in"},
|
||||
{"type": "layernorm", "name": "pre-norm 1"},
|
||||
{"type": "mha", "heads": 12, "name": "self-attention"},
|
||||
{"type": "layernorm", "name": "pre-norm 2"},
|
||||
{"type": "flatten", "name": "flatten for the feedforward accounting"},
|
||||
{"type": "linear", "units": 3072, "name": "ffn up"},
|
||||
{"type": "activation", "name": "gelu"},
|
||||
{"type": "linear", "units": 768, "name": "ffn down"}
|
||||
{
|
||||
"type": "input",
|
||||
"shape": [
|
||||
512,
|
||||
768
|
||||
],
|
||||
"name": "token embeddings in"
|
||||
},
|
||||
{
|
||||
"type": "layernorm",
|
||||
"name": "pre-norm 1"
|
||||
},
|
||||
{
|
||||
"type": "mha",
|
||||
"heads": 12,
|
||||
"name": "self-attention"
|
||||
},
|
||||
{
|
||||
"type": "layernorm",
|
||||
"name": "pre-norm 2"
|
||||
},
|
||||
{
|
||||
"type": "linear",
|
||||
"units": 3072,
|
||||
"name": "ffn up (position-wise)"
|
||||
},
|
||||
{
|
||||
"type": "activation",
|
||||
"name": "gelu"
|
||||
},
|
||||
{
|
||||
"type": "linear",
|
||||
"units": 768,
|
||||
"name": "ffn down (position-wise)"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,6 +14,12 @@ forward pass; a training step costs roughly 3x a forward pass (forward + backwar
|
|||
Layer types: input, linear, conv2d, pool2d, flatten, embedding, layernorm, activation,
|
||||
dropout, mha (multi-head self-attention), lstm, gru.
|
||||
|
||||
A linear layer on a 2-D (seq, features) input is treated as position-wise: one weight
|
||||
matrix shared across all positions, as in a transformer feedforward block. Parameters do
|
||||
not scale with sequence length; compute does. Flatten first only when you really mean a
|
||||
dense layer over the whole flattened sequence — that is a different layer, and its
|
||||
parameter count is seq_len times larger.
|
||||
|
||||
conv2d "same" padding follows TensorFlow/Keras SAME: output is ceil(H / stride), with
|
||||
any needed padding split across the two sides (and the extra pixel going to the bottom
|
||||
and right at even kernel sizes). PyTorch's padding='same' is symmetric-only and rejects
|
||||
|
|
@ -72,13 +78,21 @@ def step(layer: dict, shape: tuple[int, ...], index: int) -> tuple[tuple[int, ..
|
|||
|
||||
if kind == "linear":
|
||||
units = int(_require(layer, "units", index))
|
||||
bias = bool(layer.get("bias", True))
|
||||
if len(shape) == 2:
|
||||
# Per-token (position-wise) linear over a (seq, features) sequence: one
|
||||
# weight matrix shared across positions, exactly like a transformer FFN
|
||||
# projection. Parameters do NOT scale with sequence length; compute does.
|
||||
# Flattening instead would multiply the parameter count by seq_len, which
|
||||
# is a different layer and almost never the intended one.
|
||||
seq, features = shape
|
||||
params = features * units + (units if bias else 0)
|
||||
return (seq, units), params, seq * features * units
|
||||
if len(shape) != 1:
|
||||
raise ShapeError(
|
||||
f"layer {index} (linear) needs a 1-D input, got {shape}. "
|
||||
"Insert a flatten layer, or use a per-token linear on a 2-D sequence "
|
||||
"by declaring the shape as [features]."
|
||||
f"layer {index} (linear) needs a 1-D or 2-D input, got {shape}. "
|
||||
"Insert a flatten layer to collapse a feature map into one vector."
|
||||
)
|
||||
bias = bool(layer.get("bias", True))
|
||||
params = shape[0] * units + (units if bias else 0)
|
||||
return (units,), params, shape[0] * units
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue