# Glossary

**Action space** — the set of all valid actions in an environment; discrete or continuous (Ch 7)
**Action-value function** — see Q-function (Ch 7)
**Advantage function** — `A^pi(s,a) = Q^pi(s,a) - V^pi(s)`; how much better a specific action is than randomly selecting one per pi (Ch 7, Ch 9)
**Agent** — the learner and decider; often used interchangeably with "policy" (Ch 7)
**Backtracking line search** — shrink the step by `alpha^j` until the exact KL constraint holds and surrogate advantage is positive (Ch 15)
**Baseline** — any state-only function `b(s)` subtractable from the policy gradient weight without changing its expectation (Ch 9)
**Bellman backup** — the right-hand side of a Bellman equation: reward plus next value (Ch 7)
**Bellman equation** — self-consistency condition: the value of your starting point is the reward you expect from being there plus the value of where you land next (Ch 7)
**Categorical policy** — stochastic policy for discrete actions, built like a classifier: logits then softmax (Ch 7)
**Clipped double-Q learning** — learn two Q-functions and use the smaller value to form the target (Ch 18, Ch 19)
**Deadly triad** — function approximation + bootstrapping + off-policy data, jointly causing instability in value learning (Ch 8)
**Diagonal Gaussian policy** — stochastic policy for continuous actions; mean network plus a diagonal covariance represented as a vector of log stds (Ch 7)
**Discount factor (gamma)** — in (0,1); makes an infinite reward sum converge and encodes "cash now beats cash later" (Ch 7)
**EGLP lemma** — Expected Grad-Log-Prob: `E_{x~P_theta}[grad log P_theta(x)] = 0` (Ch 9)
**Entropy** — `H(P) = E_{x~P}[-log P(x)]`; how random a random variable is (Ch 19)
**Entropy regularization** — adding an entropy bonus to the reward at each timestep, scaled by `alpha` (Ch 19)
**Epoch** — one experience-collection phase plus one policy update (Ch 9)
**ExperimentGrid** — Spinning Up's hyperparameter ablation tool; backs the CLI (Ch 4, Ch 20)
**Fully / partially observed** — whether the agent sees the complete state or only part of it (Ch 7)
**GAE** — Generalized Advantage Estimation; used by Spinning Up's VPG, TRPO and PPO (Ch 9)
**Log-derivative trick** — `grad P = P * grad log P` (Ch 9)
**Markov property** — transitions depend only on the most recent state and action (Ch 7)
**MDP** — Markov Decision Process, the 5-tuple `<S, A, R, P, rho_0>` (Ch 7)
**Model** — a function predicting state transitions and rewards (Ch 8)
**Model-based / model-free** — whether the agent has or learns a model (Ch 8)
**MPC** — model-predictive control: plan over a window, execute the first action, discard the rest, re-plan (Ch 8)
**MSBE** — Mean-Squared Bellman Error, the loss deep Q-learning minimizes (Ch 17)
**Natural Policy Gradient** — TRPO's analytic solution before the line search is added (Ch 15)
**Observation** — a partial description of a state, possibly omitting information (Ch 7)
**Off-policy** — updates may use data collected at any point in training (Ch 3, Ch 8)
**On-policy** — updates use only data from the most recent policy (Ch 3, Ch 8)
**Optimal policy (pi\*)** — `argmax_pi J(pi)` (Ch 7)
**Parameterized policy** — a policy whose output depends on adjustable parameters, e.g. network weights (Ch 7)
**Performance** — the plotter's family-aware alias, resolving to `AverageEpRet` (on-policy) or `AverageTestEpRet` (off-policy) (Ch 6)
**Policy** — the rule for choosing actions; deterministic `mu(s)` or stochastic `pi(.|s)` (Ch 7)
**Policy delay** — TD3's ratio of Q-updates to policy updates; recommended 2 (Ch 18)
**Policy gradient** — `grad_theta J(pi_theta)` (Ch 9)
**Polyak averaging** — target update `phi_targ <- rho*phi_targ + (1-rho)*phi`; `polyak` in the code (Ch 17)
**Q-function** — `Q^pi(s,a)`, expected return from taking a in s then following pi (Ch 7)
**Replay buffer** — the stored set of past transitions that off-policy algorithms sample from (Ch 17)
**Reparameterization trick** — sample by a deterministic function of state, parameters and independent noise (Ch 19)
**Return** — cumulative reward over a trajectory; finite-horizon undiscounted or infinite-horizon discounted (Ch 7)
**Reward-to-go** — `Rhat_t = sum_{t'=t}^T R(s_t', a_t', s_{t'+1})` (Ch 9)
**Silent failure** — broken RL code that runs correctly and simply never learns (Ch 10, Ch 12)
**Squashed Gaussian policy** — a Gaussian passed through `tanh` to bound actions; SAC's policy (Ch 19)
**start_steps** — initial phase of uniform-random actions used by DDPG, TD3 and SAC (Ch 17)
**State** — a complete description of the world, with nothing hidden (Ch 7)
**Surrogate advantage** — importance-weighted advantage measuring new-policy performance on old-policy data (Ch 15)
**Surrogate objective** — a local approximation of `J(pi_theta)` that is safe to maximize (Ch 8, Ch 16)
**Target network** — a time-lagged copy of a network used to compute stable Bellman targets (Ch 17)
**Target policy smoothing** — clipped noise added to the target action to prevent exploiting sharp Q peaks (Ch 18)
**Trajectory (tau)** — a sequence of states and actions; also called episode or rollout (Ch 7)
**Trust region** — a KL-divergence bound on how far the new policy may move from the old (Ch 15)
**Value function** — `V^pi(s)`, expected return from s following pi (Ch 7)
