# Glossary

Terms as this skill uses them, with the chapter that develops each. Definitions are written for
lookup, not for teaching — read the chapter file for the reasoning.

- **Adam / AdamW** (ch08, ch07) — adaptive optimizer combining momentum with per-parameter scaling. AdamW decouples weight decay from the gradient, which Adam+L2 does not do correctly.
- **Amortized inference** (ch19, ch14) — training a network to predict a posterior's parameters instead of optimizing per example.
- **Ancestral sampling** (ch16) — sampling a directed model by drawing each variable after its parents.
- **Attention** (ch10) — a learned weighted read over a set of positions; introduced here as a fix for the seq2seq bottleneck, later the whole architecture.
- **Backpropagation** (ch06) — reverse-mode automatic differentiation over the computation graph.
- **Batch normalization** (ch08) — reparameterization normalizing layer inputs by batch statistics.
- **Bias–variance decomposition** (ch05) — expected error as bias² + variance + irreducible noise.
- **Capacity, effective vs representational** (ch05) — functions reachable by the optimizer vs functions expressible by the architecture.
- **Condition number** (ch02, ch04) — ratio of largest to smallest singular value; how amplified small perturbations become.
- **Contrastive divergence (CD-k)** (ch18) — biased approximation of the negative phase using a short chain started at the data.
- **Cross-entropy** (ch03) — H(p,q) = H(p) + D_KL(p‖q); minimizing it over q minimizes KL.
- **Curse of dimensionality** (ch05, ch01) — configurations grow exponentially with dimension, defeating local-interpolation methods.
- **Denoising autoencoder (DAE)** (ch14) — reconstructs clean input from a corrupted copy; its learned map estimates the score.
- **Disentangled representation** (ch15) — separate generative factors on separate directions.
- **Distributed representation** (ch15) — n features describing exponentially many regions, versus one-hot's n.
- **d-separation** (ch16) — reading conditional independence off a directed graph.
- **Double descent** (ch05) — test error falling again past the interpolation threshold; post-dates the book and qualifies its U-curve.
- **ELBO** (ch19) — evidence lower bound; log p(x) minus KL(q ‖ posterior).
- **Empirical risk minimization** (ch08) — minimizing training loss as a surrogate for unmeasurable true risk.
- **Energy-based model** (ch16) — p(x) ∝ exp(−E(x)).
- **Equivariance** (ch09) — a shift in input produces the corresponding shift in output; convolution's defining property.
- **Exposure bias** (ch10) — train/inference mismatch from teacher forcing.
- **Gradient clipping** (ch08, ch10) — rescaling gradients above a norm threshold to survive cliffs.
- **Ill-conditioning** (ch04, ch08) — wide Hessian eigenvalue spread; loss stalls while gradient norm stays large.
- **Importance sampling** (ch17) — sampling from q and reweighting by p/q.
- **KL divergence** (ch03) — asymmetric divergence between distributions; direction decides mode-covering vs mode-seeking.
- **Manifold hypothesis** (ch05, ch14) — real data concentrates near a low-dimensional manifold.
- **Maximum likelihood estimation (MLE)** (ch05) — parameters maximizing data probability; the justification for standard losses.
- **Mean field** (ch19) — factorized variational family; buys tractability with an independence assumption.
- **Mixing time / burn-in** (ch17) — steps an MCMC chain needs before its samples are usable.
- **Mode collapse** (ch20) — a generator covering only part of the data distribution.
- **Momentum / Nesterov momentum** (ch08) — velocity accumulation across gradient steps.
- **No Free Lunch theorem** (ch05) — averaged over all distributions, all learners tie; priors carry all leverage.
- **Noise-contrastive estimation (NCE)** (ch18) — density estimation recast as data-versus-noise classification.
- **Partition function (Z)** (ch16, ch18) — the normalizer of an undirected model; the central intractability of Part III.
- **Persistent contrastive divergence (PCD)** (ch18) — CD with a chain persisted across updates.
- **Pooling** (ch09) — local aggregation giving approximate invariance to small translations.
- **Positive / negative phase** (ch18) — the two terms of an undirected model's likelihood gradient.
- **Pseudolikelihood** (ch18) — product of conditionals, in which Z cancels.
- **Receptive field** (ch09) — the input region influencing one output unit.
- **Regularization** (ch07, ch05) — any change intended to reduce generalization error but not training error.
- **Reparameterization trick** (ch20) — expressing a sample as a differentiable function of noise, making the ELBO backprop-able.
- **Representation learning** (ch15, ch01) — learning features rather than hand-designing them.
- **Saddle point** (ch08) — critical point that is a minimum in some directions and a maximum in others; the dominant critical point in high dimensions.
- **Score matching** (ch18) — matching ∇_x log p instead of p, so Z vanishes; the basis of diffusion models.
- **SGD, minibatch** (ch08) — gradient descent on noisy gradients from data subsets.
- **Singular value decomposition (SVD)** (ch02) — A = U D Vᵀ for any matrix.
- **Sparse coding** (ch13) — heavy-tailed latent prior with inference by optimization.
- **Teacher forcing** (ch10) — training an autoregressive model on ground-truth history.
- **Universal approximation theorem** (ch06) — one wide hidden layer suffices in principle; silent on width, learnability and sample cost.
- **Vanishing / exploding gradients** (ch10, ch08) — repeated Jacobian multiplication shrinking or growing gradients through depth or time.
- **Variational autoencoder (VAE)** (ch20, ch19) — amortized variational inference plus the reparameterization trick.
- **Weight decay** (ch07) — pulling parameters toward zero; equivalent to L2 for SGD, not for Adam.
